35. Stochastic gradient methods and tame geometry
Invited abstract in session WF-6: Stochastic Gradient Methods: Bridging Theory and Practice, stream Challenges in nonlinear programming.
Wednesday, 16:20 - 18:00Room: M:H
Authors (first author is the speaker)
| 1. | Johannes Aspman
|
| Computer Science, Czech Technical University | |
| 2. | Jiri Nemecek
|
| Czech Technical University in Prague | |
| 3. | Vyacheslav Kungurtsev
|
| Czech Technical University | |
| 4. | Fabio V. Difonzo
|
| IAC, CNR | |
| 5. | Jakub Marecek
|
| Dept. of Computer Science, Czech Technical University |
Abstract
Stochastic differential equations of Langevin-diffusion form have received significant attention, thanks to their foundational role in both Bayesian sampling algorithms and optimization in machine learning. In the latter, they serve as a conceptual model of the stochastic gradient flow in training over-parameterized models. However, the literature typically assumes smoothness of the potential, whose gradient is the drift term. Nevertheless, there are many problems for which the potential function is not continuously differentiable, and hence the drift is not Lipschitz continuous everywhere. This is exemplified by robust losses and Rectified Linear Units in regression problems. In arXiv:2206.11533, we show some foundational results regarding the flow and asymptotic properties of Langevin-type Stochastic Differential Inclusions under assumptions from tame geometry. In arXiv:2302.00709 we extend some of these results for manifold constrained optimization. In arXiv:2311.13544 we further show how to approximate the stratification of tame functions, which makes the above results practical.
Keywords
- SS - Semidefinite Optimization
Status: accepted
Back to the list of papers