Timezone: »
A desirable objective in self-supervised learning (SSL) is to avoid feature collapse. Whitening loss guarantees collapse avoidance by minimizing the distance between embeddings of positive pairs under the conditioning that the embeddings from different views are whitened. In this paper, we propose a framework with an informative indicator to analyze whitening loss, which provides a clue to demystify several interesting phenomena as well as a pivoting point connecting to other SSL methods. We reveal that batch whitening (BW) based methods do not impose whitening constraints on the embedding, but they only require the embedding to be full-rank. This full-rank constraint is also sufficient to avoid dimensional collapse. Based on our analysis, we propose channel whitening with random group partition (CW-RGP), which exploits the advantages of BW-based methods in preventing collapse and avoids their disadvantages requiring large batch size. Experimental results on ImageNet classification and COCO object detection reveal that the proposed CW-RGP possesses a promising potential for learning good representations. The code is available at https://github.com/winci-ai/CW-RGP.
Author Information
Xi Weng (Beijing University of Aeronautics and Astronautics)
Lei Huang (Beihang University)
Lei Huang received his BSc and PhD degrees under supervision of Prof. Wei Li, respectively in 2010 and 2018, at the School of Computer Science and Engineering, Beihang University, China. From 2015 to 2016, he visited the Vision and Learning Lab, University of Michigan, Ann Arbor, as a joint PhD student supervised by Prof. Jia Deng. During 2018 to 2020, he was a research scientist in Inception Institute of Artificial Intelligence (IIAI), UAE. His current research mainly focuses on normalization techniques (involving methods, theories and applications) in training DNNs. He also has wide interests in deep learning theory (representation & optimization) and computers vision tasks. He serves as a reviewer for the top conferences and journals such as CVPR, ICML, ICCV, ECCV, NeurIPS, AAAI, JMLR, TPAMI, IJCV, TNNLS, etc.
Lei Zhao (TalkingData)
Rao Anwer (Mohamed bin Zayed University of Artificial Intelligence)
Salman Khan (MBZ University of AI)
Fahad Shahbaz Khan (Inception Institute of Artificial Intelligence)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: An Investigation into Whitening Loss for Self-supervised Learning »
Dates n/a. Room
More from the Same Authors
-
2021 Spotlight: Intriguing Properties of Vision Transformers »
Muhammad Muzammal Naseer · Kanchana Ranasinghe · Salman H Khan · Munawar Hayat · Fahad Shahbaz Khan · Ming-Hsuan Yang -
2022 Poster: Understanding the Failure of Batch Normalization for Transformers in NLP »
Jiaxi Wang · Ji Wu · Lei Huang -
2023 Poster: Hardware Resilience Properties of Text-Guided Image Classifiers »
Syed Talal Wasim · Kabila Haile Soboka · Abdulrahman Mahmoud · Salman Khan · David Brooks · Gu-Yeon Wei -
2023 Poster: PromptIR: Prompting for All-in-One Image Restoration »
Vaishnav Potlapalli · Syed Waqas Zamir · Salman Khan · Fahad Shahbaz Khan -
2023 Poster: Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization »
Jameel Abdul Samadh · Mohammad Hanan Gani · Noor Hussein · Muhammad Uzair Khattak · Muzammal Naseer · Salman Khan · Fahad Shahbaz Khan -
2023 Poster: 3D Indoor Instance Segmentation in an Open-World »
Mohamed Boudjoghra · Salwa Al Khatib · Jean Lahoud · Hisham Cholakkal · Rao Anwer · Salman Khan · Fahad Shahbaz Khan -
2023 Poster: Cal-DETR: Calibrated Detection Transformer »
Muhammad Akhtar Munir · Salman Khan · Muhammad Haris Khan · Mohsen Ali · Fahad Shahbaz Khan -
2022 Workshop: Vision Transformers: Theory and applications »
Fahad Shahbaz Khan · Gul Varol · Salman Khan · Ping Luo · Rao Anwer · Ashish Vaswani · Hisham Cholakkal · Niki Parmar · Joost van de Weijer · Mubarak Shah -
2022 Spotlight: Lightning Talks 3B-3 »
Sitao Luan · Zhiyuan You · Ruofan Liu · Linhao Qu · Yuwei Fu · Jiaxi Wang · Chunyu Wei · Jian Liang · xiaoyuan luo · Di Wu · Yun Lin · Lei Cui · Ji Wu · Chenqing Hua · Yujun Shen · Qincheng Lu · XIANGLIN YANG · Benoit Boulet · Manning Wang · Di Liu · Lei Huang · Fei Wang · Kai Yang · Jiaqi Zhu · Jin Song Dong · Zhijian Song · Xin Lu · Mingde Zhao · Shuyuan Zhang · Yu Zheng · Xiao-Wen Chang · Xinyi Le · Doina Precup -
2022 Spotlight: Understanding the Failure of Batch Normalization for Transformers in NLP »
Jiaxi Wang · Ji Wu · Lei Huang -
2022 Spotlight: Lightning Talks 1B-3 »
Chaofei Wang · Qixun Wang · Jing Xu · Long-Kai Huang · Xi Weng · Fei Ye · Harsh Rangwani · shrinivas ramasubramanian · Yifei Wang · Qisen Yang · Xu Luo · Lei Huang · Adrian G. Bors · Ying Wei · Xinglin Pan · Sho Takemori · Hong Zhu · Rui Huang · Lei Zhao · Yisen Wang · Kato Takashi · Shiji Song · Yanan Li · Rao Anwer · Yuhei Umeda · Salman Khan · Gao Huang · Wenjie Pei · Fahad Shahbaz Khan · Venkatesh Babu R · Zenglin Xu -
2022 Poster: Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection »
Hanoona Bangalath · Muhammad Maaz · Muhammad Uzair Khattak · Salman Khan · Fahad Shahbaz Khan -
2021 Poster: Intriguing Properties of Vision Transformers »
Muhammad Muzammal Naseer · Kanchana Ranasinghe · Salman H Khan · Munawar Hayat · Fahad Shahbaz Khan · Ming-Hsuan Yang -
2021 Poster: Rot-Pro: Modeling Transitivity by Projection in Knowledge Graph Embedding »
Tengwei Song · Jie Luo · Lei Huang -
2019 Poster: Cross-Domain Transferability of Adversarial Perturbations »
Muhammad Muzammal Naseer · Salman H Khan · Muhammad Haris Khan · Fahad Shahbaz Khan · Fatih Porikli