Timezone: »

 
Poster
The Generalization-Stability Tradeoff In Neural Network Pruning
Brian Bartoldson · Ari Morcos · Adrian Barbu · Gordon Erlebacher

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #293
in Poster Session 2 »

Pruning neural network parameters is often viewed as a means to compress models, but pruning has also been motivated by the desire to prevent overfitting. This motivation is particularly relevant given the perhaps surprising observation that a wide variety of pruning approaches increase test accuracy despite sometimes massive reductions in parameter counts. To better understand this phenomenon, we analyze the behavior of pruning over the course of training, finding that pruning's benefit to generalization increases with pruning's instability (defined as the drop in test accuracy immediately following pruning). We demonstrate that this "generalization-stability tradeoff'' is present across a wide variety of pruning settings and propose a mechanism for its cause: pruning regularizes similarly to noise injection. Supporting this, we find less pruning stability leads to more model flatness and the benefits of pruning do not depend on permanent parameter removal. These results explain the compatibility of pruning-based generalization improvements and the high generalization recently observed in overparameterized networks.

Author Information

Brian Bartoldson (Lawrence Livermore National Laboratory)
Ari Morcos (Facebook AI Research)
Adrian Barbu (Florida State University, USA)
Gordon Erlebacher (Florida State University)

More from the Same Authors