Timezone: »

 
Poster
Meta-Learning with Self-Improving Momentum Target
Jihoon Tack · Jongjin Park · Hankook Lee · Jaeho Lee · Jinwoo Shin

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #138
in Poster Session 3 »

The idea of using a separately trained target model (or teacher) to improve the performance of the student model has been increasingly popular in various machine learning domains, and meta-learning is no exception; a recent discovery shows that utilizing task-wise target models can significantly boost the generalization performance. However, obtaining a target model for each task can be highly expensive, especially when the number of tasks for meta-learning is large. To tackle this issue, we propose a simple yet effective method, coined Self-improving Momentum Target (SiMT). SiMT generates the target model by adapting from the temporal ensemble of the meta-learner, i.e., the momentum network. This momentum network and its task-specific adaptations enjoy a favorable generalization performance, enabling self-improving of the meta-learner through knowledge distillation. Moreover, we found that perturbing parameters of the meta-learner, e.g., dropout, further stabilize this self-improving process by preventing fast convergence of the distillation loss during meta-training. Our experimental results demonstrate that SiMT brings a significant performance gain when combined with a wide range of meta-learning methods under various applications, including few-shot regression, few-shot classification, and meta-reinforcement learning. Code is available at https://github.com/jihoontack/SiMT.

Keywords: Meta-Learning · knowledge distillation · momentum network
[ Paper]  [ Slides]  [ Poster]  [ OpenReview] 

Author Information

Jihoon Tack (KAIST)
Jongjin Park (KAIST)
Hankook Lee (Korea Advanced Institute of Science and Technology)
Jaeho Lee (POSTECH)
Jinwoo Shin (KAIST)

More from the Same Authors