Timezone: »
Machine learning models often encounter distribution shifts when deployed in the real world. In this paper, we focus on adaptation to label distribution shift in the online setting, where the test-time label distribution is continually changing and the model must dynamically adapt to it without observing the true label. This setting is common in many real world scenarios such as medical diagnosis, where disease prevalences can vary substantially at different times of the year. Leveraging a novel analysis, we show that the lack of true label does not hinder estimation of the expected test loss, which enables the reduction of online label shift adaptation to conventional online learning. Informed by this observation, we propose adaptation algorithms inspired by classical online learning techniques such as Follow The Leader (FTL) and Online Gradient Descent (OGD) and derive their regret bounds. We empirically verify our findings under both simulated and real world label distribution shifts and show that OGD is particularly effective and robust to a variety of challenging label shift scenarios.
Author Information
Ruihan Wu (Cornell University)
Chuan Guo (Facebook AI Research)
Yi Su (Cornell University)
Kilian Weinberger (Cornell University)
More from the Same Authors
-
2022 Poster: Data-Driven Model-Based Optimization via Invariant Representation Learning »
Han Qi · Yi Su · Aviral Kumar · Sergey Levine -
2021 Poster: Towards Deeper Deep Reinforcement Learning with Spectral Normalization »
Nils Bjorck · Carla Gomes · Kilian Weinberger -
2021 Poster: Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems »
Ruihan Wu · Chuan Guo · Awni Hannun · Laurens van der Maaten -
2021 Poster: ReAct: Out-of-distribution Detection With Rectified Activations »
Yiyou Sun · Chuan Guo · Yixuan Li -
2021 Poster: BulletTrain: Accelerating Robust Neural Network Training via Boundary Example Mining »
Weizhe Hua · Yichi Zhang · Chuan Guo · Zhiru Zhang · G. Edward Suh -
2019 Poster: Breaking the Glass Ceiling for Embedding-Based Classifiers for Large Output Spaces »
Chuan Guo · Ali Mousavi · Xiang Wu · Daniel Holtmann-Rice · Satyen Kale · Sashank Reddi · Sanjiv Kumar -
2019 Poster: A New Defense Against Adversarial Images: Turning a Weakness into a Strength »
Shengyuan Hu · Tao Yu · Chuan Guo · Wei-Lun Chao · Kilian Weinberger -
2018 : CAB: Continuous Adaptive Blending Estimator for Policy Evaluation and Learning »
Yi Su · Lequn Wang -
2018 Poster: Understanding Batch Normalization »
Johan Bjorck · Carla Gomes · Bart Selman · Kilian Weinberger -
2016 Poster: Supervised Word Mover's Distance »
Gao Huang · Chuan Guo · Matt J Kusner · Yu Sun · Fei Sha · Kilian Weinberger -
2016 Oral: Supervised Word Mover's Distance »
Gao Huang · Chuan Guo · Matt J Kusner · Yu Sun · Fei Sha · Kilian Weinberger