322. Diagonal linear networks and the regularization path of the LASSO
Invited abstract in session WC-8: Advances in non-convex optimization, stream Optimization for machine learning.
Wednesday, 14:00-16:00Room: B100/7007
Authors (first author is the speaker)
| 1. | Raphael Berthier
|
| Inria Sorbonne Université |
Abstract
Diagonal linear networks is a crude model of neural networks in which we can analyze rigorously the effect of the composition of layers on the optimization trajectory. In the overparametrized setting, diagonal linear networks enjoy an implicit regularization: they converge to the interpolator of the data with minimum l1 norm, even without any explicit regularization on the network weights. In this talk, I will discuss the implicit regularization of diagonal linear networks when early stopped. I will show that the full optimization trajectory of diagonal linear networks is closely related to the regularization path of the LASSO, where the iteration number plays the role of the inverse regularization parameter.
Keywords
- Optimization for learning and data analysis
Status: accepted
Back to the list of papers