Timezone: »

 
Spotlight
Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guarantees
Aleksandr Beznosikov · Peter Richtarik · Michael Diskin · Max Ryabinin · Alexander Gasnikov

Wed Dec 07 05:00 PM -- 07:00 PM (PST) @
in Lightning Talks 4B-1 »
Variational inequalities in general and saddle point problems in particular are increasingly relevant in machine learning applications, including adversarial learning, GANs, transport and robust optimization. With increasing data and problem sizes necessary to train high performing models across various applications, we need to rely on parallel and distributed computing. However, in distributed training, communication among the compute nodes is a key bottleneck during training, and this problem is exacerbated for high dimensional and over-parameterized models. Due to these considerations, it is important to equip existing methods with strategies that would allow to reduce the volume of transmitted information during training while obtaining a model of comparable quality. In this paper, we present the first theoretically grounded distributed methods for solving variational inequalities and saddle point problems using compressed communication: MASHA1 and MASHA2. Our theory and methods allow for the use of both unbiased (such as Rand$k$; MASHA1) and contractive (such as Top$k$; MASHA2) compressors. New algorithms support bidirectional compressions, and also can be modified for stochastic setting with batches and for federated learning with partial participation of clients. We empirically validated our conclusions using two experimental setups: a standard bilinear min-max problem, and large-scale distributed adversarial training of transformers.
Keywords: Convex Optimization · compression · saddle point problems · variational inequalities

Author Information

Aleksandr Beznosikov (Moscow Institute of Physics and Technology)
Peter Richtarik (KAUST)
Michael Diskin (Yandex, Higher School of Economics)
Max Ryabinin (Yandex, HSE University)
Alexander Gasnikov (Moscow Institute of Physics and Technology)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors