Timezone: »

 
Poster
Redundant representations help generalization in wide neural networks
Diego Doimo · Aldo Glielmo · Sebastian Goldt · Alessandro Laio

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #614
in Poster Session 5 »

Deep neural networks (DNNs) defy the classical bias-variance trade-off: adding parameters to a DNN that interpolates its training data will typically improve its generalization performance. Explaining the mechanism behind this ``benign overfitting'' in deep networks remains an outstanding challenge. Here, we study the last hidden layer representations of various state-of-the-art convolutional neural networks and find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information and differ from each other only by statistically independent noise. The number of such groups increases linearly with the width of the layer, but only if the width is above a critical value. We show that redundant neurons appear only when the training is regularized and the training error is zero.

Keywords: generalization · wide neural networks
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Diego Doimo (International School for Advanced Studies (SISSA))
Aldo Glielmo (Banca d'Italia)
Sebastian Goldt (SISSA, Trieste, Italy)
Alessandro Laio (International School for Advanced Studies (SISSA))

More from the Same Authors