The Pitfalls of Memorization: When Memorization Hinders Generalization

Neural networks often learn simple explanations that fit the majority of the data while memorizing exceptions that deviate from these explanations. This leads to poor generalization when the learned explanations are spurious. In this work, we formalize $\textit{the interplay between memorization and generalization}$, showing that spurious correlations, when combined with memorization, can reduce the training loss to zero, leaving no incentive to learn robust, generalizable patterns. To address this issue, we introduce $\textit{memorization-aware training}$ (MAT). MAT leverages the flip side of memorization by using held-out predictions to adjust a model's logits, guiding it towards learning robust patterns that remain invariant from training to test, thereby enhancing generalization under distribution shifts.