Stochastic generalized gradient methods for training nonconvex nonsmooth neural networks
Vladimir I. Norkin(vladimir.norkingmail.com)
Abstract: The paper observes a similarity between the stochastic optimal control of discrete dynamical systems and the learning multilayer neural networks. It focuses on contemporary deep networks with nonconvex nonsmooth loss and activation functions. The machine learning problems are treated as nonconvex nonsmooth stochastic optimization problems. As a model of nonsmooth nonconvex dependences, the so-called generalized differentiable functions are used. The backpropagation method for calculating stochastic generalized gradients of the learning quality functional for such systems is substantiated basing on Hamilton-Pontryagin formalism. Stochastic generalized gradient learning algorithms are extended for training nonconvex nonsmooth neural networks. The performance of a stochastic generalized gradient algorithm is illustrated in the linear multiclass classification problem.
Keywords: Machine learning, deep learning, multilayer neural networks, nonsmooth nonconvex optimization, stochastic optimization, stochastic generalized gradient.
Category 1: Stochastic Programming
Category 2: Convex and Nonsmooth Optimization (Nonsmooth Optimization )
Category 3: Nonlinear Optimization
Citation: Preprint 30.09.2019. V.M.Glushkov Institute of Cybernetics of the National Academy of Sciences of Ukraine, Kyiv, September 2019. To appear in "Cybernetics and Systems Analysis".
Entry Submitted: 09/29/2019
Modify/Update this entry
|Visitors||Authors||More about us||Links|
Search, Browse the Repository
Give us feedback
|Optimization Journals, Sites, Societies|