Conformal Prediction Adaptive to Unknown Subpopulation Shifts

Conformal prediction is widely used to provide uncertainty quantification for black-box machine learning models with formal coverage guarantees. However, these guarantees typically break down in the presence of distribution shifts, where the data distribution at test time differs from the training distribution. In this work, we address subpopulation shifts, where the test environment exhibits an unknown and differing mixture of subpopulations compared to the training data. We propose a method to adapt conformal prediction to these unknown subpopulation shifts. Our results demonstrate that the proposed algorithms maintain coverage in test domains where standard conformal prediction fails.