Timezone: »

 
Spotlight
What Do Neural Networks Learn When Trained With Random Labels?
Hartmut Maennel · Ibrahim Alabdulmohsin · Ilya Tolstikhin · Robert Baldock · Olivier Bousquet · Sylvain Gelly · Daniel Keysers

Tue Dec 08 07:30 AM -- 07:40 AM (PST) @ Orals & Spotlights: Deep Learning
in Orals & Spotlights Track 08: Deep Learning »

We study deep neural networks (DNNs) trained on natural image data with entirely random labels. Despite its popularity in the literature, where it is often used to study memorization, generalization, and other phenomena, little is known about what DNNs learn in this setting. In this paper, we show analytically for convolutional and fully connected networks that an alignment between the principal components of network parameters and data takes place when training with random labels. We study this alignment effect by investigating neural networks pre-trained on randomly labelled image data and subsequently fine-tuned on disjoint datasets with random or real labels. We show how this alignment produces a positive transfer: networks pre-trained with random labels train faster downstream compared to training from scratch even after accounting for simple effects, such as weight scaling. We analyze how competing effects, such as specialization at later layers, may hide the positive transfer. These effects are studied in several network architectures, including VGG16 and ResNet18, on CIFAR10 and ImageNet.

[ Paper ]

Author Information

Hartmut Maennel (Google)
Ibrahim Alabdulmohsin (Google Research)
Ilya Tolstikhin (Google, Brain Team, Zurich)
Robert Baldock (Google)
Olivier Bousquet (Google Brain (Zurich))
Sylvain Gelly (Google Brain (Zurich))
Daniel Keysers (Google Research, Brain Team)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors