Timezone: »

 
Spotlight
Efficient Knowledge Distillation from Model Checkpoints
Chaofei Wang · Qisen Yang · Rui Huang · Shiji Song · Gao Huang

Tue Dec 06 09:00 AM -- 11:00 AM (PST) @
in Lightning Talks 1B-3 »

Knowledge distillation is an effective approach to learn compact models (students) with the supervision of large and strong models (teachers). As empirically there exists a strong correlation between the performance of teacher and student models, it is commonly believed that a high performing teacher is preferred. Consequently, practitioners tend to use a well trained network or an ensemble of them as the teacher. In this paper, we make an intriguing observation that an intermediate model, i.e., a checkpoint in the middle of the training procedure, often serves as a better teacher compared to the fully converged model, although the former has much lower accuracy. More surprisingly, a weak snapshot ensemble of several intermediate models from a same training trajectory can outperform a strong ensemble of independently trained and fully converged models, when they are used as teachers. We show that this phenomenon can be partially explained by the information bottleneck principle: the feature representations of intermediate models can have higher mutual information regarding the input, and thus contain more "dark knowledge'' for effective distillation. We further propose an optimal intermediate teacher selection algorithm based on maximizing the total task-related mutual information. Experiments verify its effectiveness and applicability.

Keywords: Deep Learning · knowledge distillation · image classification · intermediate model · mutual information.

Author Information

Chaofei Wang (Tsinghua University)
Qisen Yang (Department of Automation, Tsinghua University)
Rui Huang (Tsinghua University, Tsinghua University)
Shiji Song (Department of Automation, Tsinghua University)
Gao Huang (Cornell University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors