Jorge Nocedal

Noise-Tolerant Optimization Methods for the Solution of a Robust Design Problem

  • Yuchen Lou
  • Shigeng Sun
  • Jorge Nocedal
    • Categories Nonlinear Optimization, Optimization of Systems modeled by PDEs, Stochastic Programming

    The development of nonlinear optimization algorithms capable of performing reliably in the presence of noise has garnered considerable attention lately. This paper advocates for strategies to create noise-tolerant nonlinear optimization algorithms by adapting classical deterministic methods. These adaptations follow certain design guidelines described here, which make use of estimates of the noise level in the … Read more

    A Trust Region Method for the Optimization of Noisy Functions

  • Shigeng Sun
  • Jorge Nocedal
    • Categories Nonlinear Optimization Tags , ,

    Classical trust region methods were designed to solve problems in which function and gradient information are exact. This paper considers the case when there are bounded errors (or noise) in the above computations and proposes a simple modification of the trust region method to cope with these errors. The new algorithm only requires information about … Read more

    Adaptive Finite-Difference Interval Estimation for Noisy Derivative-Free Optimization

  • Hao-Jun Shi
  • Yuchen Xie
  • Melody Xuan
  • Jorge Nocedal
    • Categories Nonlinear Optimization Tags , , , ,

    A common approach for minimizing a smooth nonlinear function is to employ finite-difference approximations to the gradient. While this can be easily performed when no error is present within the function evaluations, when the function is noisy, the optimal choice requires information about the noise level and higher-order derivatives of the function, which is often … Read more

    Constrained Optimization in the Presence of Noise

  • Jorge Nocedal
  • Richard H. Byrd
  • Figen Oztoprak
    • Categories Constrained Nonlinear Optimization Tags , ,

    The problem of interest is the minimization of a nonlinear function subject to nonlinear equality constraints using a sequential quadratic programming (SQP) method. The minimization must be performed while observing only noisy evaluations of the objective and constraint functions. In order to obtain stability, the classical SQP method is modified by relaxing the standard Armijo … Read more

    On the Numerical Performance of Derivative-Free Optimization Methods Based on Finite-Difference Approximations

  • Jorge Nocedal
  • Hao-Jun Shi
  • Melody Xuan
  • Figen Oztoprak
    • Categories Nonlinear Optimization Tags , , , ,

    The goal of this paper is to investigate an approach for derivative-free optimization that has not received sufficient attention in the literature and is yet one of the simplest to implement and parallelize. It consists of computing gradients of a smoothed approximation of the objective function (and constraints), and employing them within established codes. These … Read more

    Constrained and Composite Optimization via Adaptive Sampling Methods

  • Raghu Bollapragada
  • Richard H. Byrd
  • Jorge Nocedal
  • Yuchen Xie
    • Categories Convex and Nonsmooth Optimization, Nonlinear Optimization, Stochastic Programming

    The motivation for this paper stems from the desire to develop an adaptive sampling method for solving constrained optimization problems in which the objective function is stochastic and the constraints are deterministic. The method proposed in this paper is a proximal gradient method that can also be applied to the composite optimization problem min f(x) … Read more

    A Noise-Tolerant Quasi-Newton Method for Unconstrained Optimization

  • Richard H. Byrd
  • Jorge Nocedal
  • Hao-Jun Shi
  • Yuchen Xie
    • Categories Unconstrained Optimization Tags , , ,

    This paper describes an extension of the BFGS and L-BFGS methods for the minimization of a nonlinear function subject to errors. This work is motivated by applications that contain computational noise, employ low-precision arithmetic, or are subject to statistical noise. The classical BFGS and L-BFGS methods can fail in such circumstances because the updating procedure … Read more

    Analysis of the BFGS Method with Errors

  • Richard H. Byrd
  • Jorge Nocedal
  • Yuchen Xie
    • Categories Unconstrained Optimization Tags ,

    The classical convergence analysis of quasi-Newton methods assumes that the function and gradients employed at each iteration are exact. In this paper, we consider the case when there are (bounded) errors in both computations and establish conditions under which a slight modification of the BFGS algorithm with an Armijo-Wolfe line search converges to a neighborhood … Read more

    Derivative-Free Optimization of Noisy Functions via Quasi-Newton Methods

  • Albert S. Berahas
  • Richard H. Byrd
  • Jorge Nocedal
    • Categories Nonlinear Optimization, Unconstrained Optimization Tags , ,

    This paper presents a finite difference quasi-Newton method for the minimization of noisy functions. The method takes advantage of the scalability and power of BFGS updating, and employs an adaptive procedure for choosing the differencing interval h based on the noise estimation techniques of Hamming (2012) and Moré and Wild (2011). This noise estimation procedure … Read more

    A Progressive Batching L-BFGS Method for Machine Learning

  • Raghu Bollapragada
  • Jorge Nocedal
  • Hao-Jun Shi
  • Ping Tak Peter Tang
  • Dheevatsa Mudigere
    • Categories Convex Optimization, Nonlinear Optimization, Stochastic Programming Tags , , ,

    The standard L-BFGS method relies on gradient approximations that are not dominated by noise, so that search directions are descent directions, the line search is reliable, and quasi-Newton updating yields useful quadratic models of the objective function. All of this appears to call for a full batch approach, but since small batch sizes give rise … Read more