Timezone: »

 
Spotlight
Distilling Representations from GAN Generator via Squeeze and Span
Yu Yang · Xiaotian Cheng · Chang Liu · Hakan Bilen · Xiangyang Ji

Tue Dec 06 05:00 PM -- 07:00 PM (PST) @
in Lightning Talks 2A-3 »

In recent years, generative adversarial networks (GANs) have been an actively studied topic and shown to successfully produce high-quality realistic images in various domains. The controllable synthesis ability of GAN generators suggests that they maintain informative, disentangled, and explainable image representations, but leveraging and transferring their representations to downstream tasks is largely unexplored. In this paper, we propose to distill knowledge from GAN generators by squeezing and spanning their representations. We \emph{squeeze} the generator features into representations that are invariant to semantic-preserving transformations through a network before they are distilled into the student network. We \emph{span} the distilled representation of the synthetic domain to the real domain by also using real training data to remedy the mode collapse of GANs and boost the student network performance in a real domain. Experiments justify the efficacy of our method and reveal its great significance in self-supervised representation learning. Code is available at https://github.com/yangyu12/squeeze-and-span.

Keywords: Representation Learning · Self-supervised learning · knowledge distillation · Generative Adversarial Network

Author Information

Yu Yang (Tsinghua University, Tsinghua University)
Xiaotian Cheng (Tsinghua University, Tsinghua University)
Chang Liu (Tsinghua University, Tsinghua University)
Hakan Bilen (University of Edinburgh)
Xiangyang Ji (Tsinghua University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors