Beyond Accuracy: Distilling Foundation Models into Interpretable and Stable Predictors

Interpretable models such as Generalized Additive Models (GAMs) are increasingly deployed in high-stakes domains where transparency and accountability are paramount. However, these models suffer from model multiplicity—the Rashomon effect—where different instantiations achieve comparable accuracy yet produce contradictory predictions and inconsistent explanations. We propose a knowledge distillation framework that leverages Tabular Foundation Models to guide the training of interpretable GAMs, replacing binary hard labels with soft labels derived from foundation model probabilities. Across two datasets and multiple architectures, our approach achieves three simultaneous gains: up to a 97 percent reduction in predictive ambiguity, significantly lower explanation multiplicity, and preserved or improved AUROC. Our framework advances the deployment of interpretable models that are not only transparent but also stable and reliable at population scale—a necessary condition for AI systems to serve the public good.