Optimization Online


On the convergence of stochastic bi-level gradient methods

Nicolas Couellan (nicolas.couellan***at***math.univ-toulouse.fr)
Wenjuan Wang (wenjuan.wang***at***math.univ-toulouse.fr)

Abstract: We analyze the convergence of stochastic gradient methods for bi-level optimization problems. We address two specific cases: first when the outer objective function can be expressed as a finite sum of independent terms, and next when both the outer and inner objective functions can be expressed as finite sums of independent terms. We assume Lipschitz continuity and differentiability of both objectives as well as convexity of the inner objective and consider diminishing steps sizes. We show that, under these conditions and some other assumptions on the implicit function and the variance of the gradient errors, both methods converge in expectation to a stationary point of the problem. We also discuss the satisfaction of our assumptions in machine learning problems where these methods can be nicely applied to automatically tune hyperparameters when the loss functions are very large sums of error terms.

Keywords: Bi-level optimization, stochastic gradient, gradient approximation, empirical risk minimization

Category 1: Nonlinear Optimization (Other )

Category 2: Applications -- Science and Engineering (Data-Mining )


Download: [PDF]

Entry Submitted: 02/15/2016
Entry Accepted: 02/15/2016
Entry Last Modified: 02/15/2016

Modify/Update this entry

  Visitors Authors More about us Links
  Subscribe, Unsubscribe
Digest Archive
Search, Browse the Repository


Coordinator's Board
Classification Scheme
Give us feedback
Optimization Journals, Sites, Societies
Mathematical Optimization Society