Timezone: »

 
Poster
Escaping Saddle Points for Effective Generalization on Class-Imbalanced Data
Harsh Rangwani · Sumukh K Aithal · Mayank Mishra · Venkatesh Babu R

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #101
in Poster Session 4 »

Real-world datasets exhibit imbalances of varying types and degrees. Several techniques based on re-weighting and margin adjustment of loss are often used to enhance the performance of neural networks, particularly on minority classes. In this work, we analyze the class-imbalanced learning problem by examining the loss landscape of neural networks trained with re-weighting and margin based techniques. Specifically, we examine the spectral density of Hessian of class-wise loss, through which we observe that the network weights converges to a saddle point in the loss landscapes of minority classes. Following this observation, we also find that optimization methods designed to escape from saddle points can be effectively used to improve generalization on minority classes. We further theoretically and empirically demonstrate that Sharpness-Aware Minimization (SAM), a recent technique that encourages convergence to a flat minima, can be effectively used to escape saddle points for minority classes. Using SAM results in a 6.2\% increase in accuracy on the minority classes over the state-of-the-art Vector Scaling Loss, leading to an overall average increase of 4\% across imbalanced datasets. The code is available at https://github.com/val-iisc/Saddle-LongTail.

Keywords: Long-tailed learning · Saddle Points · Class-Imbalanced Learning
[ Paper]  [ Slides]  [ Poster]  [ OpenReview] 

Author Information

Harsh Rangwani (Indian Institute of Science)
Sumukh K Aithal (-)

Domains of interest : Generalization in Deep Learning , Domain Adaptation, Shape/Texture Bias of CNN, Explainable AI. Undergraduate student passionate about research in deep learning. Always eager to learn more.

Mayank Mishra (Indian Institute of Science)
Venkatesh Babu R (Indian Institute of Science)

More from the Same Authors