  


Fast global convergence of gradient methods for highdimensional statistical recovery
Alekh Agarwal(alekhcs.berkeley.edu) Abstract: Many statistical $M$estimators are based on convex optimization problems formed by the combination of a datadependent loss function with a normbased regularizer. We analyze the convergence rates of projected gradient and composite gradient methods for solving such problems, working within a highdimensional framework that allows the data dimension $\pdim$ to grow with (and possibly exceed) the sample size $\numobs$. This highdimensional structure precludes the usual global assumptionsnamely, strong convexity and smoothness conditionsthat underlie much of classical optimization analysis. We define appropriately restricted versions of these conditions, and show that they are satisfied with high probability for various statistical models. Under these conditions, our theory guarantees that projected gradient descent has a globally geometric rate of convergence up to the \emph{statistical precision} of the model, meaning the typical distance between the true unknown parameter $\theta^*$ and an optimal solution $\widehat{\theta}$. This result is substantially sharper than previous convergence results, which yielded sublinear convergence, or linear convergence only up to the noise level. Our analysis applies to a wide range of $M$estimators and statistical models, including sparse linear regression using Lasso ($\ell_1$regularized regression); group Lasso for block sparsity; loglinear models with regularization; lowrank matrix recovery using nuclear norm regularization; and matrix decomposition. Overall, our analysis reveals interesting connections between statistical precision and computational efficiency in highdimensional estimation. Keywords: Convex optimization, firstorder methods, sparsity Category 1: Convex and Nonsmooth Optimization (Convex Optimization ) Category 2: Applications  Science and Engineering (Statistics ) Citation: To appear in The Annals of Statistics Download: [PDF] Entry Submitted: 08/23/2012 Modify/Update this entry  
Visitors  Authors  More about us  Links  
Subscribe, Unsubscribe Digest Archive Search, Browse the Repository

Submit Update Policies 
Coordinator's Board Classification Scheme Credits Give us feedback 
Optimization Journals, Sites, Societies  