Timezone: »

CompRess: Self-Supervised Learning by Compressing Representations
Soroush Abbasi Koohpayegani · Ajinkya Tejankar · Hamed Pirsiavash

Wed Dec 09 09:00 AM -- 11:00 AM (PST) @ Poster Session 3 #1010

Self-supervised learning aims to learn good representations with unlabeled data. Recent works have shown that larger models benefit more from self-supervised learning than smaller models. As a result, the gap between supervised and self-supervised learning has been greatly reduced for larger models. In this work, instead of designing a new pseudo task for self-supervised learning, we develop a model compression method to compress an already learned, deep self-supervised model (teacher) to a smaller one (student). We train the student model so that it mimics the relative similarity between the datapoints in the teacher's embedding space. For AlexNet, our method outperforms all previous methods including the fully supervised model on ImageNet linear evaluation (59.0% compared to 56.5%) and on nearest neighbor evaluation (50.7% compared to 41.4%). To the best of our knowledge, this is the first time a self-supervised AlexNet has outperformed supervised one on ImageNet classification. Our code is available here: https://github.com/UMBCvision/CompRess

Author Information

Soroush Abbasi Koohpayegani (University of Maryland Baltimore County)
Ajinkya Tejankar (University of Maryland Baltimore County)
Hamed Pirsiavash (University of Maryland, Baltimore County)

More from the Same Authors

  • 2020 Poster: COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning »
    Simon Ging · Mohammadreza Zolfaghari · Hamed Pirsiavash · Thomas Brox
  • 2018 : Accepted papers »
    Sven Gowal · Bogdan Kulynych · Marius Mosbach · Nicholas Frosst · Phil Roth · Utku Ozbulak · Simral Chaudhary · Toshiki Shibahara · Salome Viljoen · Nikita Samarin · Briland Hitaj · Rohan Taori · Emanuel Moss · Melody Guan · Lukas Schott · Angus Galloway · Anna Golubeva · Xiaomeng Jin · Felix Kreuk · Akshayvarun Subramanya · Vipin Pillai · Hamed Pirsiavash · Giuseppe Ateniese · Ankita Kalra · Logan Engstrom · Anish Athalye
  • 2016 Poster: Generating Videos with Scene Dynamics »
    Carl Vondrick · Hamed Pirsiavash · Antonio Torralba