Poster
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
Guodong Zhang · Lala Li · Zachary Nado · James Martens · Sushant Sachdeva · George Dahl · Chris Shallue · Roger Grosse

Thu Dec 12th 10:45 AM -- 12:45 PM @ East Exhibition Hall B + C #174
in Deep Learning -- Optimization for Deep Networks »

Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns. In this work, we study how the critical batch size changes based on properties of the optimization algorithm, including acceleration and preconditioning, through two different lenses: large scale experiments and analysis using a simple noisy quadratic model (NQM). We experimentally demonstrate that optimization algorithms that employ preconditioning, specifically Adam and K-FAC, result in much larger critical batch sizes than stochastic gradient descent with momentum. We also demonstrate that the NQM captures many of the essential features of real neural network training, despite being drastically simpler to work with. The NQM predicts our results with preconditioned optimizers, previous results with accelerated gradient descent, and other results around optimal learning rates and large batch training, making it a useful tool to generate testable predictions about neural network optimization. We demonstrate empirically that the simple noisy quadratic model (NQM) displays many similarities to neural networks in terms of large-batch training. We prove analytical convergence results for the NQM model that predict such behavior and hence provide possible explanations and a better understanding for many large-batch training phenomena.

Keywords: Deep Learning -> Optimization for Deep Networks · Deep Learning -> Efficient Training Methods; Optimization -> Stochastic Optimization

Author Information

Guodong Zhang (University of Toronto)
Lala Li (Google)
Zachary Nado (Google Inc.)
James Martens (DeepMind)
Sushant Sachdeva (University of Toronto)
George Dahl (Google Brain)
Chris Shallue (Google Brain)
Roger Grosse (University of Toronto)

More from the Same Authors