Timezone: »
The performance of machine learning models under distribution shift has been the focus of the community in recent years. Most of current methods have been proposed to improve the robustness to distribution shift from the algorithmic perspective, i.e., designing better training algorithms to help the generalization in shifted test distributions. This paper studies the distribution shift problem from the perspective of pre-training and data augmentation, two important factors in the practice of deep learning that have not been systematically investigated by existing work. By evaluating seven pre-trained models, including ResNets and ViT's with self-supervision and supervision mode, on five important distribution-shift datasets, from WILDS and DomainBed benchmarks, with five different learning algorithms, we provide the first comprehensive empirical study focusing on pre-training and data augmentation. With our empirical result obtained from 1,330 models, we provide the following main observations: 1) ERM combined with data augmentation can achieve state-of-the-art performance if we choose a proper pre-trained model respecting the data property; 2) specialized algorithms further improve the robustness on top of ERM when handling a specific type of distribution shift, e.g., GroupDRO for spurious correlation and CORAL for large-scale out-of-distribution data; 3) Comparing different pre-training modes, architectures and data sizes, we provide novel observations about pre-training on distribution shift, which sheds light on designing or selecting pre-training strategy for different kinds of distribution shifts. In summary, our empirical study provides a comprehensive baseline for a wide range of pre-training models fine-tuned with data augmentation, which potentially inspires research exploiting the power of pre-training and data augmentation in the future of distribution shift study.
Author Information
Ziquan Liu (City University of Hong Kong)
Yi Xu (Alibaba Group U.S. Inc.)
Yuanhong Xu
Qi Qian (Alibaba Group)
Hao Li (alibaba group)
Rong Jin
Xiangyang Ji (Tsinghua University)
Antoni Chan (City University of Hong Kong)
More from the Same Authors
-
2022 Poster: Near-Optimal Regret Bounds for Multi-batch Reinforcement Learning »
Zihan Zhang · Yuhang Jiang · Yuan Zhou · Xiangyang Ji -
2022 Poster: Self-Organized Group for Cooperative Multi-agent Reinforcement Learning »
Jianzhun Shao · Zhiqiang Lou · Hongchang Zhang · Yuhang Jiang · Shuncheng He · Xiangyang Ji -
2022 Poster: VTC-LFC: Vision Transformer Compression with Low-Frequency Components »
Zhenyu Wang · Hao Luo · Pichao WANG · Feng Ding · Fan Wang · Hao Li -
2022 Poster: SPD: Synergy Pattern Diversifying Oriented Unsupervised Multi-agent Reinforcement Learning »
Yuhang Jiang · Jianzhun Shao · Shuncheng He · Hongchang Zhang · Xiangyang Ji -
2022 : GLINKX: A Unified Framework for Large-scale Homophilous and Heterophilous Graphs »
Marios Papachristou · Rishab Goel · Frank Portman · Matthew Miller · Rong Jin -
2022 : Precise Augmentation and Counting of Helicobacter Pylori in Histology Image »
Yufei CUI · Yixin Chen · Zhifeng Shuai · Fang Peng · Yanbo Lv · Luoning Zheng · Xue (Steve) Liu · Antoni Chan · Tei-Wei Kuo · Chun Jason XUE -
2022 : A Comparative Survey of Deep Active Learning »
Xueying Zhan · Qingzhong Wang · Kuan-Hao Huang · Haoyi Xiong · Dejing Dou · Antoni Chan -
2023 : Towards the next generation explainable AI that promotes AI-human mutual understanding »
Janet Hsiao · Antoni Chan -
2023 Poster: Retrieval-Augmented Multiple Instance Learning »
Yufei CUI · Ziquan Liu · Yixin Chen · Yuchen Lu · Xinyue Yu · Xue (Steve) Liu · Tei-Wei Kuo · Miguel Rodrigues · Chun Jason XUE · Antoni Chan -
2023 Poster: Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarks »
Yun Qu · Boyuan Wang · Jianzhun Shao · Yuhang Jiang · Chen Chen · Zhenbin Ye · Liu Linc · Yang Feng · Lin Lai · Hongyang Qin · Minwen Deng · Juchao Zhuo · Deheng Ye · Qiang Fu · YANG GUANG · Wei Yang · Lanxiao Huang · Xiangyang Ji -
2023 Poster: Not All Out-of-Distribution Data Are Harmful to Open-Set Active Learning »
Yang Yang · Yuxuan Zhang · XIN SONG · Yi Xu -
2023 Poster: Supported Value Regularization for Offline Reinforcement Learning »
Yixiu Mao · Hongchang Zhang · Chen Chen · Yi Xu · Xiangyang Ji -
2023 Poster: OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensembling »
yifan zhang · Qingsong Wen · xue wang · Weiqi Chen · Liang Sun · Zhang Zhang · Liang Wang · Rong Jin · Tieniu Tan -
2023 Poster: Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP »
Qi Qian · Yuanhong Xu · Juhua Hu -
2023 Poster: DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Field »
Chenyangguang Zhang · Yan Di · Ruida Zhang · Guangyao Zhai · Fabian Manhardt · Federico Tombari · Xiangyang Ji -
2023 Poster: Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning »
Jianzhun Shao · Yun Qu · Chen Chen · Hongchang Zhang · Xiangyang Ji -
2022 Spotlight: Lightning Talks 6B-3 »
Lingfeng Yang · Yao Lai · Zizheng Pan · Zhenyu Wang · Weicong Liang · Chuanyang Zheng · Jian-Wei Zhang · Peng Jin · Jing Liu · Xiuying Wei · Yao Mu · Xiang Li · YUHUI YUAN · Zizheng Pan · Yifan Sun · Yunchen Zhang · Jianfei Cai · Hao Luo · zheyang li · Jinfa Huang · Haoyu He · Yi Yang · Ping Luo · Fenglin Liu · Henghui Ding · Borui Zhao · Xiangguo Zhang · Kai Zhang · Pichao WANG · Bohan Zhuang · Wei Chen · Ruihao Gong · Zhi Yang · Xian Wu · Feng Ding · Jianfei Cai · Xiao Luo · Renjie Song · Weihong Lin · Jian Yang · Wenming Tan · Bohan Zhuang · Shanghang Zhang · Shen Ge · Fan Wang · Qi Zhang · Guoli Song · Jun Xiao · Hao Li · Ding Jia · David Clifton · Ye Ren · Fengwei Yu · Zheng Zhang · Jie Chen · Shiliang Pu · Xianglong Liu · Chao Zhang · Han Hu -
2022 Spotlight: VTC-LFC: Vision Transformer Compression with Low-Frequency Components »
Zhenyu Wang · Hao Luo · Pichao WANG · Feng Ding · Fan Wang · Hao Li -
2022 Spotlight: Lightning Talks 5A-3 »
Minting Pan · Xiang Chen · Wenhan Huang · Can Chang · Zhecheng Yuan · Jianzhun Shao · Yushi Cao · Peihao Chen · Ke Xue · Zhengrong Xue · Zhiqiang Lou · Xiangming Zhu · Lei Li · Zhiming Li · Kai Li · Jiacheng Xu · Dongyu Ji · Ni Mu · Kun Shao · Tianpei Yang · Kunyang Lin · Ningyu Zhang · Yunbo Wang · Lei Yuan · Bo Yuan · Hongchang Zhang · Jiajun Wu · Tianze Zhou · Xueqian Wang · Ling Pan · Yuhang Jiang · Xiaokang Yang · Xiaozhuan Liang · Hao Zhang · Weiwen Hu · Miqing Li · YAN ZHENG · Matthew Taylor · Huazhe Xu · Shumin Deng · Chao Qian · YI WU · Shuncheng He · Wenbing Huang · Chuanqi Tan · Zongzhang Zhang · Yang Gao · Jun Luo · Yi Li · Xiangyang Ji · Thomas Li · Mingkui Tan · Fei Huang · Yang Yu · Huazhe Xu · Dongge Wang · Jianye Hao · Chuang Gan · Yang Liu · Luo Si · Hangyu Mao · Huajun Chen · Jianye Hao · Jun Wang · Xiaotie Deng -
2022 Spotlight: Self-Organized Group for Cooperative Multi-agent Reinforcement Learning »
Jianzhun Shao · Zhiqiang Lou · Hongchang Zhang · Yuhang Jiang · Shuncheng He · Xiangyang Ji -
2022 Spotlight: Lightning Talks 2A-3 »
David Buterez · Chengan He · Xuan Kan · Yutong Lin · Konstantin Schürholt · Yu Yang · Louis Annabi · Wei Dai · Xiaotian Cheng · Alexandre Pitti · Ze Liu · Jon Paul Janet · Jun Saito · Boris Knyazev · Mathias Quoy · Zheng Zhang · James Zachary · Steven J Kiddle · Xavier Giro-i-Nieto · Chang Liu · Hejie Cui · Zilong Zhang · Hakan Bilen · Damian Borth · Dino Oglic · Holly Rushmeier · Han Hu · Xiangyang Ji · Yi Zhou · Nanning Zheng · Ying Guo · Pietro Liò · Stephen Lin · Carl Yang · Yue Cao -
2022 Spotlight: Distilling Representations from GAN Generator via Squeeze and Span »
Yu Yang · Xiaotian Cheng · Chang Liu · Hakan Bilen · Xiangyang Ji -
2022 Spotlight: Lightning Talks 1B-4 »
Andrei Atanov · Shiqi Yang · Wanshan Li · Yongchang Hao · Ziquan Liu · Jiaxin Shi · Anton Plaksin · Jiaxiang Chen · Ziqi Pan · yaxing wang · Yuxin Liu · Stepan Martyanov · Alessandro Rinaldo · Yuhao Zhou · Li Niu · Qingyuan Yang · Andrei Filatov · Yi Xu · Liqing Zhang · Lili Mou · Ruomin Huang · Teresa Yeo · kai wang · Daren Wang · Jessica Hwang · Yuanhong Xu · Qi Qian · Hu Ding · Michalis Titsias · Shangling Jui · Ajay Sohmshetty · Lester Mackey · Joost van de Weijer · Hao Li · Amir Zamir · Xiangyang Ji · Antoni Chan · Rong Jin -
2022 Spotlight: Improved Fine-Tuning by Better Leveraging Pre-Training Data »
Ziquan Liu · Yi Xu · Yuanhong Xu · Qi Qian · Hao Li · Xiangyang Ji · Antoni Chan · Rong Jin -
2022 Poster: Entropy-Driven Mixed-Precision Quantization for Deep Network Design »
Zhenhong Sun · Ce Ge · Junyan Wang · Ming Lin · Hesen Chen · Hao Li · Xiuyu Sun -
2022 Poster: Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks »
Yunwen Lei · Rong Jin · Yiming Ying -
2022 Poster: Distilling Representations from GAN Generator via Squeeze and Span »
Yu Yang · Xiaotian Cheng · Chang Liu · Hakan Bilen · Xiangyang Ji -
2022 Poster: Improved Fine-Tuning by Better Leveraging Pre-Training Data »
Ziquan Liu · Yi Xu · Yuanhong Xu · Qi Qian · Hao Li · Xiangyang Ji · Antoni Chan · Rong Jin -
2021 Poster: Improved Variance-Aware Confidence Sets for Linear Bandits and Linear Mixture MDP »
Zihan Zhang · Jiaqi Yang · Xiangyang Ji · Simon Du -
2021 Poster: HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning »
Shiming Chen · Guosen Xie · Yang Liu · Qinmu Peng · Baigui Sun · Hao Li · Xinge You · Ling Shao -
2021 Poster: TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification »
Zhuchen Shao · Hao Bian · Yang Chen · Yifeng Wang · Jian Zhang · Xiangyang Ji · yongbing zhang -
2021 Poster: An Online Method for A Class of Distributionally Robust Optimization with Non-convex Objectives »
Qi Qi · Zhishuai Guo · Yi Xu · Rong Jin · Tianbao Yang -
2020 Poster: Modeling Noisy Annotations for Crowd Counting »
Jia Wan · Antoni Chan -
2020 Poster: Almost Optimal Model-Free Reinforcement Learningvia Reference-Advantage Decomposition »
Zihan Zhang · Yuan Zhou · Xiangyang Ji -
2020 Poster: Optimal Epoch Stochastic Gradient Descent Ascent Methods for Min-Max Optimization »
Yan Yan · Yi Xu · Qihang Lin · Wei Liu · Tianbao Yang -
2019 Poster: Regret Minimization for Reinforcement Learning by Evaluating the Optimal Bias Function »
Zihan Zhang · Xiangyang Ji -
2017 Poster: Incorporating Side Information by Adaptive Convolution »
Di Kang · Debarun Dhar · Antoni Chan -
2012 Poster: The variational hierarchical EM algorithm for clustering hidden Markov models. »
Emanuele Coviello · Antoni Chan · Gert Lanckriet