Knowledge distillation is to transfer the knowledge of a large neural network into a smaller one and has been shown to be effective especially when the amount of training data is limited or the size of the student model is very small. To transfer the knowledge, it is essential to observe the data that have been used to train the network since its knowledge is concentrated on a narrow manifold rather than the whole input space. However, the data are not accessible in many cases due to the privacy or confidentiality issues in medical, industrial, and military domains. To the best of our knowledge, there has been no approach that distills the knowledge of a neural network when no data are observable. In this work, we propose KegNet (Knowledge Extraction with Generative Networks), a novel approach to extract the knowledge of a trained deep neural network and to generate artificial data points that replace the missing training data in knowledge distillation. Experiments show that KegNet outperforms all baselines for data-free knowledge distillation. We provide the source code of our paper in https://github.com/snudatalab/KegNet.
Jaemin Yoo (Seoul National University)
Minyong Cho (Seoul National University)
Taebum Kim (Seoul National University)
U Kang (Seoul National University)
More from the Same Authors
2021 Poster: Terra: Imperative-Symbolic Co-Execution of Imperative Deep Learning Programs »
Taebum Kim · Eunji Jeong · Geon-Woo Kim · Yunmo Koo · Sehoon Kim · Gyeongin Yu · Byung-Gon Chun