Timezone: »

 
Poster
An empirical analysis of compute-optimal large language model training
Jordan Hoffmann · Sebastian Borgeaud · Arthur Mensch · Elena Buchatskaya · Trevor Cai · Eliza Rutherford · Diego de Las Casas · Lisa Anne Hendricks · Johannes Welbl · Aidan Clark · Thomas Hennigan · Eric Noland · Katherine Millican · George van den Driessche · Bogdan Damoc · Aurelia Guy · Simon Osindero · Karén Simonyan · Erich Elsen · Oriol Vinyals · Jack Rae · Laurent Sifre

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #639
in Poster Session 4 »
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more data. Chinchilla uniformly and significantly outperformsGopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, a 7% improvement over Gopher.
Keywords: Deep Learning · Large language models · NLP

Author Information

Jordan Hoffmann (Inflection)
Sebastian Borgeaud (DeepMind)
Arthur Mensch (DeepMind)
Elena Buchatskaya (DeepMind)
Trevor Cai (OpenAI)
Eliza Rutherford (University of Oxford)
Diego de Las Casas (Deepmind)
Lisa Anne Hendricks (DeepMind)
Johannes Welbl (Google)
Aidan Clark (DeepMind)
Thomas Hennigan (DeepMind)
Eric Noland
Katherine Millican (DeepMind)
George van den Driessche (DeepMind)
Bogdan Damoc (INSA de Lyon)
Aurelia Guy (University of California Berkeley)
Simon Osindero (DeepMind)
Karén Simonyan (Inflection AI)
Erich Elsen (Royal Caliber)
Oriol Vinyals (DeepMind)

Oriol Vinyals is a Research Scientist at Google. He works in deep learning with the Google Brain team. Oriol holds a Ph.D. in EECS from University of California, Berkeley, and a Masters degree from University of California, San Diego. He is a recipient of the 2011 Microsoft Research PhD Fellowship. He was an early adopter of the new deep learning wave at Berkeley, and in his thesis he focused on non-convex optimization and recurrent neural networks. At Google Brain he continues working on his areas of interest, which include artificial intelligence, with particular emphasis on machine learning, language, and vision.

Jack Rae (DeepMind, UCL)
Laurent Sifre (Google DeepMind)

More from the Same Authors