Abstract: Fine-tuning zero-shot foundation models often compromises their robustness to distribution shifts. Existing approaches aim to preserve the pre-trained features in fine-tuned models but lack a clear goal of what features to preserve. This shows to be suboptimal as not all pre-trained features are equally robust. To address the issue, we propose dual risk minimization (DRM), a novel approach that combines empirical risk minimization with worst-case risk minimization to better preserve the core features of downstream tasks. In particular, we utilize core-feature descriptions generated by LLMs to induce core-based zero-shot predictions which then serve as proxies to estimate the worst-case risk. DRM balances two crucial aspects of model robustness: expected performance and worst-case performance, establishing a new state of the art on various real-world benchmarks. DRM significantly improves the out-of-distribution performance of CLIP ViT-L/14@336 on ImageNet (75.9$\to$77.1), WILDS-iWildCam (47.1$\to$51.8), and WILDS-FMoW (51.0$\to$53.1); opening up new avenues for achieving next-level robustness in fine-tuning zero-shot models.