PURGE: Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models

Large Vision-Language Models (LVLMs) often exploit spurious object--background correlations, producing predictions driven by contextual shortcuts rather than visual evidence. Existing evaluations provide limited control for separating object-grounded from background-induced failures. We introduce PURGE (Partition-aware Unlearning for Removing spurious-correlation Generated Errors), which constructs structured retain/forget partitions and selectively unlearns spurious associations while preserving object-grounded reasoning. Across multiple LVLMs and benchmarks, PURGE consistently reduces hallucination and spurious-correlation errors while maintaining overall performance.