Timezone: »

 
Poster
Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Ge Yang · Edward Hu · Igor Babuschkin · Szymon Sidor · Xiaodong Liu · David Farhi · Nick Ryder · Jakub Pachocki · Weizhu Chen · Jianfeng Gao

Tue Dec 07 04:30 PM -- 06:00 PM (PST) @
in Poster Session 2 »
Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters.We show that, in the recently discovered Maximal Update Parametrization ($\mu$P), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call *$\mu$Transfer*: parametrize the target model in $\mu$P, tune the HP indirectly on a smaller model, and *zero-shot transfer* them to the full-sized model, i.e., without directly tuning the latter at all.We verify $\mu$Transfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at github.com/microsoft/mup. See arxiv.org for the full, up-to-date version of this work.
Keywords: Deep Learning · Transformers

Author Information

Ge Yang (Microsoft Research)
Edward Hu (Microsoft Research)
Igor Babuschkin (DeepMind)
Szymon Sidor
Xiaodong Liu (Microsoft)
David Farhi (OpenAI)
Nick Ryder
Jakub Pachocki (OpenAI)
Weizhu Chen (Microsoft Dynamics 365 AI)
Jianfeng Gao (Microsoft Research, Redmond, WA)

More from the Same Authors