Timezone: »

 
Spotlight
Communication-Efficient Stochastic Gradient Descent, with Applications to Neural Networks
Dan Alistarh · Demjan Grubic · Jerry Li · Ryota Tomioka · Milan Vojnovic

Tue Dec 05 11:30 AM -- 11:35 AM (PST) @ Hall A
in Algorithms »

Parallel implementations of stochastic gradient descent (SGD) have received significant research attention, thanks to its excellent scalability properties. A fundamental barrier when parallelizing SGD is the high bandwidth cost of communicating gradient updates between nodes; consequently, several lossy compresion heuristics have been proposed, by which nodes only communicate quantized gradients. Although effective in practice, these heuristics do not always guarantee convergence, and it is not clear whether they can be improved. In this paper, we propose Quantized SGD (QSGD), a family of compression schemes for gradient updates which provides convergence guarantees. QSGD allows the user to smoothly trade off \emph{communication bandwidth} and \emph{convergence time}: nodes can adjust the number of bits sent per iteration, at the cost of possibly higher variance. We show that this trade-off is inherent, in the sense that improving it past some threshold would violate information-theoretic lower bounds. QSGD guarantees convergence for convex and non-convex objectives, under asynchrony, and can be extended to stochastic variance-reduced techniques. When applied to training deep neural networks for image classification and automated speech recognition, QSGD leads to significant reductions in end-to-end training time. For example, on 16GPUs, we can train the ResNet152 network to full accuracy on ImageNet 1.8x faster than the full-precision variant.

Author Information

Dan Alistarh (IST Austria)
Demjan Grubic (ETH Zurich / Google)
Jerry Li (Berkeley)
Ryota Tomioka (Microsoft Research Cambridge)
Milan Vojnovic (London School of Economics (LSE))

Milan Vojnovic is Professor, Chair in Data Science, with the Department of Statistics, at London School of Economics and Political Science (LSE), where he is also director of MSc in Data Science program. Prior to this, he worked for 13 years in a corporate research laboratory environment, from 2004 until 2016, as a researcher with Microsoft Research, Cambridge, United Kingdom. He received his Ph.D. degree in Technical Sciences from Ecole Polytechnique Federale de Lausanne (EPFL), Switzerland, in 2003, and both M.Sc. and B.Sc. degrees in Electrical Engineering from the University of Split, Croatia, in 1995 and 1998, respectively. He undertook an internship with Mathematical Research Center, Bell Laboratories, Murray Hill, New Jersey, in 2001. From 2005 to 2014, he was a visiting professor with the University of Split, Croatia, and from 2014 to 2016, he was an affiliated lecturer with the Statistical Laboratory, at the University of Cambridge.    His research interests are in data science, machine learning, artificial intelligence, game theory, multi-agent systems, and information networks, with applications in the broad area of information systems and networks. He has made contributions to the theory and the design of computation platforms for processing large-scale data, and to the performance evaluation of computer systems and networks, in particular, in the areas of incentives and online services, distributed computing, network resource allocation, transport control protocols, and peer-to-peer networks.   He received several prizes for his work. In 2010, he was awarded the ACM Sigmetrics Rising Star Researcher award, and, in 2005, the ERCIM Cor Baayen Award. He received the IEEE IWQoS 2007 Best Student Paper Award (with Shao Liu and Dinan Gunawardena), the IEEE Infocom 2005 Best Paper Award (with Jean-Yves Le Boudec), the ACM Sigmetrics 2005 Best Paper Award (with Laurent Massoulie), and the ITC 2001 Best Student Paper Award (with Jean-Yves Le Boudec).   He delivered numerous lectures and seminars in both academia and industry. He taught several editions of a computer networking course within the undergraduate computer science program at the University of Split. He taught two editions of a course on contest theory within Part III of Mathematical Tripos (master program in mathematics) at the University of Cambridge. He authored the book “Contest Theory: Incentive Mechanisms and Ranking Methods,” Cambridge University Press, 2016.

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors