2372. Stochastic first-order methods can leverage arbitrarily higher-order smoothness for acceleration
Invited abstract in session WE-12: AI in Optimization Heuristics, stream Artificial Intelligence, Machine Learning and Optimization.
Wednesday, 16:30-18:00Room: H10
Authors (first author is the speaker)
| 1. | Chuan He
|
| Linköping University |
Abstract
In many emerging applications---particularly in machine learning and related fields---optimization problems are often large- or even huge-scale, which poses significant challenges to classical first-order methods due to the high cost of computing the exact derivative. To address this issue, stochastic first-order methods (SFOMs) have been extensively studied, as they employ stochastic estimators of the gradient that are typically much cheaper to compute.
In this work, we propose an SFOM with multi-extrapolated momentum for nonconvex unconstrained optimization, where multiple extrapolations are performed in each iteration, followed by a momentum update based on these extrapolations. We show that the proposed SFOM can accelerate optimization by exploiting the higher-order smoothness of the objective function. To the best of our knowledge, this is the first SFOM to leverage the Lipschitz continuity of arbitrarily higher-order derivatives of the objective function for acceleration. Preliminary numerical experiments validate the practical performance of our method and support our theoretical results.
Keywords
- Computational Complexity
- Machine Learning
- Large Scale Optimization
Status: accepted
Back to the list of papers