Timezone: »

 
Poster
Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction
Kaifeng Lyu · Zhiyuan Li · Sanjeev Arora

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #535
in Poster Session 2 »

Normalization layers (e.g., Batch Normalization, Layer Normalization) were introduced to help with optimization difficulties in very deep nets, but they clearly also help generalization, even in not-so-deep nets. Motivated by the long-held belief that flatter minima lead to better generalization, this paper gives mathematical analysis and supporting experiments suggesting that normalization (together with accompanying weight-decay) encourages GD to reduce the sharpness of loss surface. Here ``sharpness'' is carefully defined given that the loss is scale-invariant, a known consequence of normalization. Specifically, for a fairly broad class of neural nets with normalization, our theory explains how GD with a finite learning rate enters the so-called Edge of Stability (EoS) regime, and characterizes the trajectory of GD in this regime via a continuous sharpness-reduction flow.

Keywords: Gradient Descent · edge of stability · sharpness · normalization · theoretical analysis · scale-invariance
[ Paper]  [ Slides]  [ Poster]  [ OpenReview] 

Author Information

Kaifeng Lyu (Princeton University)
Zhiyuan Li (Stanford University)
Sanjeev Arora (Princeton University)

More from the Same Authors