285. Distributed Optimization with Communication Compression
Invited abstract in session TB-5: Randomized Optimization algorithms I, stream Optimization for machine learning.
Tuesday, 10:30-12:30Room: B100/4013
Authors (first author is the speaker)
| 1. | Yuan Gao
|
| 2. | Sebastian Stich
|
| CISPA Helmholtz Center for Information Security |
Abstract
Modern machine learning tasks often involve massive datasets and models, necessitating distributed optimization algorithms with reduced communication overhead. Communication compression, where clients transmit compressed updates to a central server, has emerged as a key technique to mitigate communication bottlenecks. In this work, we consider the theory of distributed optimization with contractive communication compression. The naive implementation of distributed optimization algorithm with contractive compression often leads to unstable convergence or even divergence. We discuss how recent variants of the popular and practical Error Feedback mechanism help to mitigate the aforementioned issues, and obtain provable convergence under standard assumptions. We further extend these theories and consider the decentralised setting or the incorporation of Nesterov's acceleration.
Keywords
- Distributed optimization
- Stochastic optimization
Status: accepted
Back to the list of papers