EURO 2024 Copenhagen
Abstract Submission

EURO-Online login

1686. Stochastic gradient descent with momentum and line-searches

Invited abstract in session MB-32: Algorithmic Advances in Large Scale Nonconvex Optimization, stream Advances in large scale nonlinear optimization.

Monday, 10:30-12:00
Room: 41 (building: 303A)

Authors (first author is the speaker)

1. Davide Pucci
Department of Information Engineering, University of Florence
2. Matteo Lapucci
Department of Information Engineering, University of Florence

Abstract

In recent years, the adoption of line-search techniques within incremental gradient-based methods for finite-sums problems has gathered considerable interest among researchers.
These techniques enable the use of substantially larger step-sizes while maintaining robust convergence properties and empirically showing a reduction in the number of iterations required to achieve high-quality solutions. Most recent research focused on incorporating line-searches within Stochastic Gradient Descent (SGD), as the usage of a descent direction for the mini-batch objective is essential to ensure the line-search terminates in a finite number of steps.

In this talk we discuss the nontrivial integration of these line-search techniques within algorithmic frameworks employing Polyak's momentum terms. In particular, we are interested in structured ways to combine stochastic gradients and momentum to obtain a suitable search direction together with a proper initial step-size for the line search.
We then present a computational comparison, carried out on both convex and nonconvex test problems, concerning different viable options to achieve this goal. We thus offer insights into the potential strengths and drawbacks of the considered approaches, with respect to other state-of-the-art methods.

Keywords

Status: accepted

Back to the list of papers