323. SPAM: Stochastic Proximal Point Method with Momentum Variance Reduction for Non-convex Cross-Device Federated Learning
Invited abstract in session TC-5: Randomized Optimization algorithms II, stream Optimization for machine learning.
Tuesday, 14:00-16:00Room: B100/4013
Authors (first author is the speaker)
| 1. | Avetik Karagulyan
|
| CNRS |
Abstract
Cross-device training is a crucial subfield of federated learning, where the number of clients can reach the billions. Standard approaches and local methods are prone to client drift and insensitivity to data similarities. We propose a novel algorithm (SPAM) for cross-device federated learning with non-convex and non-smooth losses. We provide a sharp analysis under second-order (Hessian) similarity, a condition satisfied by various machine learning problems in practice. Additionally, we extend our results to the partial participation setting, where a cohort of selected clients communicate with the server at each communication round. We then conduct a complexity analysis of our convergence results, showing the improvement of our methods upon prior work.
Keywords
- First-order optimization
- Optimization for learning and data analysis
- Stochastic optimization
Status: accepted
Back to the list of papers