`

Timezone: »

 
Oral
How Does Batch Normalization Help Optimization?
Shibani Santurkar · Dimitris Tsipras · Andrew Ilyas · Aleksander Madry

Tue Dec 04 01:25 PM -- 01:40 PM (PST) @ Room 220 E
in Tue Track 2 -- Session 2 »

Batch Normalization (BatchNorm) is a widely adopted technique that enables faster and more stable training of deep neural networks (DNNs). Despite its pervasiveness, the exact reasons for BatchNorm's effectiveness are still poorly understood. The popular belief is that this effectiveness stems from controlling the change of the layers' input distributions during training to reduce the so-called "internal covariate shift". In this work, we demonstrate that such distributional stability of layer inputs has little to do with the success of BatchNorm. Instead, we uncover a more fundamental impact of BatchNorm on the training process: it makes the optimization landscape significantly smoother. This smoothness induces a more predictive and stable behavior of the gradients, allowing for faster training.

[ Video] 

Author Information

Shibani Santurkar (MIT)
Dimitris Tsipras (MIT)
Andrew Ilyas (MIT)
Aleksander Madry (MIT)

Aleksander Madry is the NBX Associate Professor of Computer Science in the MIT EECS Department and a principal investigator in the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL). He received his PhD from MIT in 2011 and, prior to joining the MIT faculty, he spent some time at Microsoft Research New England and on the faculty of EPFL. Aleksander's research interests span algorithms, continuous optimization, science of deep learning and understanding machine learning from a robustness perspective. His work has been recognized with a number of awards, including an NSF CAREER Award, an Alfred P. Sloan Research Fellowship, an ACM Doctoral Dissertation Award Honorable Mention, and 2018 Presburger Award.

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors