EURO 2024 Copenhagen
Abstract Submission

EURO-Online login

2333. Simultaneous Training and Optimization of Few-Bit Neural Networks through a Multi-Objective MILP

Invited abstract in session MC-26: Combinatorial Optimization for Machine Learning, stream Combinatorial Optimization.

Monday, 12:30-14:00
Room: 012 (building: 208)

Authors (first author is the speaker)

1. Simone Milanesi
2. Ambrogio Maria Bernardelli
Dipartimento di Matematica "Felice Casorati", Università degli Studi di Pavia
3. Stefano Gualandi
Pavia, Department of Mathematics, "Felice Casorati"
4. Hoong Chuin Lau
School of Information Systems, Singapore Management University
5. Neil Yorke-Smith
Algorithmics, Delft University of Technology

Abstract

Training neural networks using combinatorial optimization solvers has gained attention, especially in low-data scenarios. Utilizing advanced solvers like mixed integer linear programming can precisely train networks without relying on intensive GPU-based methods. We focus on few-bit discrete-valued neural networks, including Binarized Neural Networks (BNNs) and Integer Neural Networks (INNs). These lightweight architectures are notable for their ability to operate on low-power devices, for example, by being implemented using boolean operations. Our proposed method involves training a network for each class pair of the classification problem and using a majority voting scheme for the output prediction. The optimization process adheres to the principles of robustness and sparsity by employing a multi-objective function. We compare this approach, named BeMi, to existing solver-based and gradient-based methods, particularly in few-shot learning contexts with BNNs. We also assess the trade-offs between INNs and BNNs. Empirical results on the MNIST dataset show that BeMi achieves higher accuracy (up to 81.8%) with a significant reduction in active weights, outperforming previous methods.

Keywords

Status: accepted

Back to the list of papers