Timezone: »
Real-world datasets exhibit imbalances of varying types and degrees. Several techniques based on re-weighting and margin adjustment of loss are often used to enhance the performance of neural networks, particularly on minority classes. In this work, we analyze the class-imbalanced learning problem by examining the loss landscape of neural networks trained with re-weighting and margin based techniques. Specifically, we examine the spectral density of Hessian of class-wise loss, through which we observe that the network weights converges to a saddle point in the loss landscapes of minority classes. Following this observation, we also find that optimization methods designed to escape from saddle points can be effectively used to improve generalization on minority classes. We further theoretically and empirically demonstrate that Sharpness-Aware Minimization (SAM), a recent technique that encourages convergence to a flat minima, can be effectively used to escape saddle points for minority classes. Using SAM results in a 6.2\% increase in accuracy on the minority classes over the state-of-the-art Vector Scaling Loss, leading to an overall average increase of 4\% across imbalanced datasets. The code is available at https://github.com/val-iisc/Saddle-LongTail.
Author Information
Harsh Rangwani (Indian Institute of Science)
Sumukh K Aithal (-)
Domains of interest : Generalization in Deep Learning , Domain Adaptation, Shape/Texture Bias of CNN, Explainable AI. Undergraduate student passionate about research in deep learning. Always eager to learn more.
Mayank Mishra (Indian Institute of Science)
Venkatesh Babu R (Indian Institute of Science)
More from the Same Authors
-
2022 : Learning an Invertible Output Mapping Can Mitigate Simplicity Bias in Neural Networks »
Sravanti Addepalli · Anshul Nasery · Venkatesh Babu R · Praneeth Netrapalli · Prateek Jain -
2022 Spotlight: Lightning Talks 6A-3 »
Junyu Xie · Chengliang Zhong · Ali Ayub · Sravanti Addepalli · Harsh Rangwani · Jiapeng Tang · Yuchen Rao · Zhiying Jiang · Yuqi Wang · Xingzhe He · Gene Chou · Ilya Chugunov · Samyak Jain · Yuntao Chen · Weidi Xie · Sumukh K Aithal · Carter Fendley · Lev Markhasin · Yiqin Dai · Peixing You · Bastian Wandt · Yinyu Nie · Helge Rhodin · Felix Heide · Ji Xin · Angela Dai · Andrew Zisserman · Bi Wang · Xiaoxue Chen · Mayank Mishra · ZHAO-XIANG ZHANG · Venkatesh Babu R · Justus Thies · Ming Li · Hao Zhao · Venkatesh Babu R · Jimmy Lin · Fuchun Sun · Matthias Niessner · Guyue Zhou · Xiaodong Mu · Chuang Gan · Wenbing Huang -
2022 Spotlight: Efficient and Effective Augmentation Strategy for Adversarial Training »
Sravanti Addepalli · Samyak Jain · Venkatesh Babu R -
2022 Spotlight: Escaping Saddle Points for Effective Generalization on Class-Imbalanced Data »
Harsh Rangwani · Sumukh K Aithal · Mayank Mishra · Venkatesh Babu R -
2022 Spotlight: Lightning Talks 1B-3 »
Chaofei Wang · Qixun Wang · Jing Xu · Long-Kai Huang · Xi Weng · Fei Ye · Harsh Rangwani · shrinivas ramasubramanian · Yifei Wang · Qisen Yang · Xu Luo · Lei Huang · Adrian G. Bors · Ying Wei · Xinglin Pan · Sho Takemori · Hong Zhu · Rui Huang · Lei Zhao · Yisen Wang · Kato Takashi · Shiji Song · Yanan Li · Rao Anwer · Yuhei Umeda · Salman Khan · Gao Huang · Wenjie Pei · Fahad Shahbaz Khan · Venkatesh Babu R · Zenglin Xu -
2022 Spotlight: Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metrics »
Harsh Rangwani · shrinivas ramasubramanian · Sho Takemori · Kato Takashi · Yuhei Umeda · Venkatesh Babu R -
2022 Poster: Efficient and Effective Augmentation Strategy for Adversarial Training »
Sravanti Addepalli · Samyak Jain · Venkatesh Babu R -
2022 Poster: Subsidiary Prototype Alignment for Universal Domain Adaptation »
Jogendra Nath Kundu · Suvaansh Bhambri · Akshay R Kulkarni · Hiran Sarkar · Varun Jampani · Venkatesh Babu R -
2022 Poster: Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metrics »
Harsh Rangwani · shrinivas ramasubramanian · Sho Takemori · Kato Takashi · Yuhei Umeda · Venkatesh Babu R -
2021 Poster: Towards Efficient and Effective Adversarial Training »
Gaurang Sriramanan · Sravanti Addepalli · Arya Baburaj · Venkatesh Babu R -
2021 Poster: Non-local Latent Relation Distillation for Self-Adaptive 3D Human Pose Estimation »
Jogendra Nath Kundu · Siddharth Seth · Anirudh Jamkhandi · Pradyumna YM · Varun Jampani · Anirban Chakraborty · Venkatesh Babu R -
2021 Poster: Aligning Silhouette Topology for Self-Adaptive 3D Human Pose Recovery »
Ramesha Rakesh Mugaludi · Jogendra Nath Kundu · Varun Jampani · Venkatesh Babu R -
2020 : Runner up presentation: Robustness to Augmentations as a Generalization Metric (PGDL) »
Sumukh K Aithal -
2020 Poster: Your Classifier can Secretly Suffice Multi-Source Domain Adaptation »
Naveen Venkat · Jogendra Nath Kundu · Durgesh Singh · Ambareesh Revanur · Venkatesh Babu R -
2020 Poster: Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses »
Gaurang Sriramanan · Sravanti Addepalli · Arya Baburaj · Venkatesh Babu R -
2020 Spotlight: Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses »
Gaurang Sriramanan · Sravanti Addepalli · Arya Baburaj · Venkatesh Babu R