Optimization Online


Inexact Variable Metric Stochastic Block-Coordinate Descent for Regularized Optimization

Ching-pei Lee(ching-pei***at***cs.wisc.edu)
Stephen Wright(swright***at***cs.wisc.edu)

Abstract: Block-coordinate descent (BCD) is a popular method for large-scale regularized optimization problems with block-separable structure. However, existing analyses require either a fixed second-order approximation of the smooth part, or restrictions on the subproblem solutions, such as exactness or termination conditions that are difficult to verify except in simple cases. These assumptions essentially confine the quadratic term in the approximation of the smooth part to being diagonal, so the benefits of second-order approximation are mostly lost. Moreover, in contrast to the smooth case, non-uniform sampling has not yet been shown to improve the convergence rate for regularized problems. In this work, we propose an inexact randomized BCD method based on a regularized quadratic subproblem, in which the quadratic term can vary from iteration to iteration (and is thus known as a ``variable metric''). We provide a detailed convergence analysis. When specialized to the non-regularized case, Nesterov's proposal to improve convergence rate by sampling proportional to the blockwise Lipschitz constants is covered in our framework. Empirical results also show that significant benefits are attainable when a variable quadratic term is used in the subproblem, rather than a fixed term.

Keywords: block-coordinate descent; regularized optimization; nonsmooth optimization; inexact method; arbitrary sampling

Category 1: Convex and Nonsmooth Optimization (Nonsmooth Optimization )

Category 2: Convex and Nonsmooth Optimization (Convex Optimization )


Download: [PDF]

Entry Submitted: 08/01/2018
Entry Accepted: 08/01/2018
Entry Last Modified: 08/01/2018

Modify/Update this entry

  Visitors Authors More about us Links
  Subscribe, Unsubscribe
Digest Archive
Search, Browse the Repository


Coordinator's Board
Classification Scheme
Give us feedback
Optimization Journals, Sites, Societies
Mathematical Optimization Society