Superclass-Guided Representation Disentanglement for Spurious Correlation Mitigation

To enhance group robustness to spurious correlations, prior work often relies on auxiliary groups or features annotations and assumes identical sets of groups across training and test domains. To overcome these limitations, we propose a method that leverages the semantic structure inherent in class labels—specifically, superclass information—to naturally reduce reliance on spurious features. Our model employs gradient-based attention from a pretrained vision-language model to disentangle superclass-relevant and irrelevant features. Then, by promoting the use of all superclass-relevant features for prediction, our approach achieves robustness to more complex spurious correlations without annotating any training samples. Experiments across diverse datasets demonstrate that our method significantly outperforms baselines in domain generalization tasks, with clear improvements in both quantitative metrics and qualitative visualizations.