Optimization Online


Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization

Xiao Wang (wangxiao***at***ucas.ac.cn)
Shiqian Ma (sqma***at***se.cuhk.edu.hk)
Wei Liu (weiliu***at***us.ibm.com)

Abstract: In this paper we study stochastic quasi-Newton methods for nonconvex stochastic optimization, where we assume that only stochastic information of the gradients of the objective function is available via a stochastic first-order oracle (SFO). Firstly, we propose a general framework of stochastic quasi-Newton methods for solving nonconvex stochastic optimization. The proposed framework extends the classic quasi-Newton methods working in deterministic settings to stochastic settings, and we prove its almost sure convergence to stationary points. Secondly, we propose a general framework for a class of randomized stochastic quasi-Newton methods, in which the number of iterations conducted by the algorithm is a random variable. The worst-case SFO-calls complexities of this class of methods are analyzed. Thirdly, we present two specific methods that fall into this framework, namely stochastic damped-BFGS method and stochastic cyclic Barzilai-Borwein method. Finally, we report numerical results to demonstrate the efficiency of the proposed methods.

Keywords: Stochastic Optimization, Nonconvex Optimization, Stochastic Approximation, Quasi-Newton Method, BFGS Method, Barzilai-Borwein Method, Complexity

Category 1: Nonlinear Optimization

Category 2: Stochastic Programming


Download: [PDF]

Entry Submitted: 12/02/2014
Entry Accepted: 12/03/2014
Entry Last Modified: 12/03/2014

Modify/Update this entry

  Visitors Authors More about us Links
  Subscribe, Unsubscribe
Digest Archive
Search, Browse the Repository


Coordinator's Board
Classification Scheme
Give us feedback
Optimization Journals, Sites, Societies
Mathematical Optimization Society