Layer-wise Compression for Variation Inequalities

We develop a general layer-wise and adaptive compression framework with applications to solving variational inequality problems (VI) in a large-scale and distributed setting where multiple nodes have access to local stochastic dual vectors. We establish tight error bounds and code-length bounds for adaptive layer-wise quantization that generalize previous bounds for global quantization. We also propose Quantized and Generalized Optimistic Dual Averaging (QODA) with adaptive learning rates, which achieves optimal rate of convergence for distributed monotone VIs. We empirically show that the adaptive layer-wise compression achieves up to a $150\%$ speedup in end-to-end training time for training Wasserstein GAN on $12+$ GPUs.