-

 

 

 




Optimization Online





 

Efficient and Robust Mixed-Integer Optimization Methods for Training Binarized Deep Neural Networks

Jannis Kurtz (jannis.kurtz***at***uni-siegen.de)
Bubacarr Bah (bubacarr***at***aims.ac.za)

Abstract: Compared to classical deep neural networks its binarized versions can be useful for applications on resource-limited devices due to their reduction in memory consumption and computational demands. In this work we study deep neural networks with binary activation functions and continuous or integer weights (BDNN). We show that the BDNN can be reformulated as a mixed-integer linear program with bounded weight space which can be solved to global optimality by classical mixed-integer programming solvers. Additionally, a local search heuristic is presented to calculate locally optimal networks. Furthermore to improve efficiency we present an iterative data-splitting heuristic which iteratively splits the training set into smaller subsets by using the k-mean method. Afterwards all data points in a given subset are forced to follow the same activation pattern, which leads to a much smaller number of integer variables in the mixed-integer programming formulation and therefore to computational improvements. Finally for the first time a robust model is presented which enforces robustness of the BDNN during training. All methods are tested on random and real datasets and our results indicate that all models can often compete with or even outperform classical DNNs on small network architectures confirming the viability for applications having restricted memory or computing power.

Keywords: Mixed-Integer Programming, Binarized Neural Networks

Category 1: Applications -- OR and Management Sciences (Other )

Category 2: Integer Programming

Category 3: Robust Optimization

Citation:

Download: [PDF]

Entry Submitted: 10/21/2021
Entry Accepted: 10/21/2021
Entry Last Modified: 10/22/2021

Modify/Update this entry


  Visitors Authors More about us Links
  Subscribe, Unsubscribe
Digest Archive
Search, Browse the Repository

 

Submit
Update
Policies
Coordinator's Board
Classification Scheme
Credits
Give us feedback
Optimization Journals, Sites, Societies
Mathematical Optimization Society