Optimization Online


On the asymptotic convergence and acceleration of gradient methods

Yakui Huang(huangyakui2006***at***gmail.com)
Yu-Hong Dai(dyh***at***lsec.cc.ac.cn)
Xin-Wei Liu(mathlxw***at***hebut.edu.cn)
Hongchao Zhang(hozhang***at***math.lsu.edu )

Abstract: We consider the asymptotic behavior of a family of gradient methods, which include the steepest descent and minimal gradient methods as special instances. It is proved that each method in the family will asymptotically zigzag between two directions. Asymptotic convergence results of the objective value, gradient norm, and stepsize are presented as well. To accelerate the family of gradient methods, we further exploit spectral properties of stepsizes to break the zigzagging pattern. In particular, a new stepsize is derived by imposing finite termination on minimizing two-dimensional strictly convex quadratic function. It is shown that, for the general quadratic function, the proposed stepsize asymptotically converges to the reciprocal of the largest eigenvalue of the Hessian. Furthermore, based on this spectral property, we propose a periodic gradient method by incorporating the Barzilai-Borwein method. Numerical comparisons with some recent successful gradient methods show that our new method is very promising.

Keywords: gradient methods, asymptotic convergence, spectral property, acceleration of gradient methods, Barzilai-Borwein method, unconstrained optimization, quadratic optimization

Category 1: Convex and Nonsmooth Optimization (Convex Optimization )

Category 2: Nonlinear Optimization (Unconstrained Optimization )

Category 3: Nonlinear Optimization (Quadratic Programming )


Download: [PDF]

Entry Submitted: 08/19/2019
Entry Accepted: 08/19/2019
Entry Last Modified: 08/19/2019

Modify/Update this entry

  Visitors Authors More about us Links
  Subscribe, Unsubscribe
Digest Archive
Search, Browse the Repository


Coordinator's Board
Classification Scheme
Give us feedback
Optimization Journals, Sites, Societies
Mathematical Optimization Society