123. prunAdag: an adaptive pruning-aware gradient method
Invited abstract in session TB-3: Theoretical and algorithmic advances in large scale nonlinear optimization and applications Part 1, stream Large scale optimization: methods and algorithms.
Tuesday, 10:30-12:30Room: B100/4011
Authors (first author is the speaker)
| 1. | Giovanni Seraghiti
|
| Umons | |
| 2. | Margherita Porcelli
|
| Dipartimento di Ingegneria Industriale, Università degli Studi di Firenze | |
| 3. | Philippe L. Toint
|
| Mathematics, FUNDP |
Abstract
In this talk, we will discuss Objective Function Free Optimization (OFFO) in the context of pruning the parameter of a given model. OFFO algorithms are methods where the objective function is never computed; instead, they rely only on derivative information, thus on the gradient in the first-order case. One prominent example of OFFO methods are adaptive gradient algorithms.
Pruning emerges as an alternative compression technique, mainly for neural networks, to matrix and tensor factorization or quantization. We will focus on pruning-aware methods that use specific rules to classify parameters as relevant or irrelevant at each iteration, promoting the former and penalizing the latter. The final purpose is to enhance convergence to a solution of the problem at hand, which is robust to pruning irrelevant parameters after training.
Our contribution is a novel deterministic algorithm, termed prunAdag, which is both adaptive and pruning-aware. We develop a new adaptive strategy to separately update parameters that extends the distinction between relevant and irrelevant. Specifically, we introduce the concepts of optimisable and decreasable parameters. prunAdag uses the Adagrad step for updating the optimisable parameters and an Adagrad-like trust-region framework to gradually decrease the magnitude of the decreasable ones. We will also discuss the convergence properties of the algorithm, and we will illustrate preliminary results on different applications.
Keywords
- First-order optimization
- Large-scale optimization
- Optimization for learning and data analysis
Status: accepted
Back to the list of papers