160. Almost sure convergence of stochastic Hamiltonian descent methods
Invited abstract in session TD-6: Stochastic methods, stream Methods for non-/monotone inclusions and their applications.
Thursday, 14:10 - 15:50Room: M:H
Authors (first author is the speaker)
| 1. | Måns Williamson
|
| Centre for Mathematical Sciences, Lund University |
Abstract
Gradient normalization and soft clipping are two popular techniques for tackling instability issues and improving convergence of stochastic optimization methods.
In this talk, we study these types of methods through the lens of dissipative Hamiltonian systems. Gradient normalization and certain types of soft clipping algorithms can be seen as (stochastic) implicit-explicit Euler discretizations of dissipative Hamiltonian systems, where the kinetic energy function determines the type of clipping that is applied.
We make use of unified theory from dynamical systems to show that all of these schemes converge almost surely to stationary points of the objective function.
Keywords
- Optimization for learning and data analysis
- Optimization under uncertainty and applications
- Global optimization
Status: accepted
Back to the list of papers