Poster
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Yanping Huang · Youlong Cheng · Ankur Bapna · Orhan Firat · Dehao Chen · Mia Chen · HyoukJoong Lee · Jiquan Ngiam · Quoc V Le · Yonghui Wu · zhifeng Chen

Tue Dec 10th 05:30 -- 07:30 PM @ East Exhibition Hall B + C #40
in Algorithms -- Large Scale Learning »

Scaling up deep neural network capacity has been known as an effective approach to improving model quality for several different machine learning tasks. In many cases, increasing model capacity beyond the memory limit of a single accelerator has required developing special algorithms or infrastructure. These solutions are often architecture-specific and do not transfer to other machine learning tasks. To address the need for efficient and task-independent model parallelism, we introduce TensorPipe, a pipeline parallelism library that allows scaling any network that can be expressed as a sequence of layers. By pipelining different sub-sequences of layers on separate accelerators, TensorPipe provides the flexibility of scaling a variety of different networks to gigantic sizes efficiently. Moreover, TensorPipe utilizes a novel batch-splitting pipelining algorithm, resulting in almost linear speedup when a model is partitioned across multiple accelerators. We demonstrate the advantages of TensorPipe by training large-scale neural networks on two different tasks with distinct network architectures: (i)Image Classification: We train a 557-million-parameter AmoebaNet model and attain a top-1 accuracy of 84.4% on ImageNet-2012, (ii)Multilingual Neural Machine Translation: We train a single 6-billion-parameter, 128-layer Transformer model on a corpus spanning over 100 languages and achieve better quality than all bilingual models.

Keywords: Data, Challenges, Implementations, and Software -> Software Toolkits · Algorithms -> Large Scale Learning; Applications -> Natural Language Processing; Applications -> Object Recognition

Author Information

Yanping Huang (Google Brain)
Youlong Cheng (Google)

Software engineer in Tensorflow.

Ankur Bapna (Google)
Orhan Firat (Google)
Dehao Chen (Google)
Mia Chen (Google Brain)
HyoukJoong Lee (Google)
Jiquan Ngiam (Google Brain)
Quoc V Le (Google)
Yonghui Wu (Google)
ZF Chen (Google Brain)

More from the Same Authors