Timezone: »

TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
Wei Wen · Cong Xu · Feng Yan · Chunpeng Wu · Yandan Wang · Yiran Chen · Hai Li

Wed Dec 06 06:30 PM -- 10:30 PM (PST) @ Pacific Ballroom #127

High network communication cost for synchronizing gradients and parameters is the well-known bottleneck of distributed training. In this work, we propose TernGrad that uses ternary gradients to accelerate distributed deep learning in data parallelism. Our approach requires only three numerical levels {-1,0,1}, which can aggressively reduce the communication time. We mathematically prove the convergence of TernGrad under the assumption of a bound on gradients. Guided by the bound, we propose layer-wise ternarizing and gradient clipping to improve its convergence. Our experiments show that applying TernGrad on AlexNet does not incur any accuracy loss and can even improve accuracy. The accuracy loss of GoogLeNet induced by TernGrad is less than 2% on average. Finally, a performance model is proposed to study the scalability of TernGrad. Experiments show significant speed gains for various deep neural networks. Our source code is available.

Author Information

Wei Wen (Duke University)

I’m a Ph.D. student in Duke University. My research interests include scalable deep learning, model compression, structure learning and deep neural network understanding.

Cong Xu (Hewlett Packard Labs)
Feng Yan (University of Nevada, Reno)
Chunpeng Wu (Duke University)
Yandan Wang (University of Pittsburgh)
Yiran Chen (Duke University)
Hai Li (Duke University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors