Timezone: »

 
Poster
Does Knowledge Distillation Really Work?
Samuel Stanton · Pavel Izmailov · Polina Kirichenko · Alexander Alemi · Andrew Wilson

Tue Dec 07 04:30 PM -- 06:00 PM (PST) @
in Poster Session 2 »

Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood: there often remains a surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher. We identify difficulties in optimization as a key reason for why the student is unable to match the teacher. We also show how the details of the dataset used for distillation play a role in how closely the student matches the teacher --- and that more closely matching the teacher paradoxically does not always lead to better student generalization.

Keywords: Optimization

Author Information

Samuel Stanton (New York University)

ML Scientist at Genentech Early Research and Development (gRED). Building ML systems for scientific discovery in biotech.

Pavel Izmailov (New York University)
Polina Kirichenko (New York University)
Alexander Alemi (Google)
Andrew Wilson (New York University)
Andrew Wilson

I am a professor of machine learning at New York University.

More from the Same Authors