Timezone: »

 
Poster
The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift
Jingfeng Wu · Difan Zou · Vladimir Braverman · Quanquan Gu · Sham Kakade

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #335
in Poster Session 3 »
We study linear regression under covariate shift, where the marginal distribution over the input covariates differs in the source and the target domains, while the conditional distribution of the output given the input covariates is similar across the two domains. We investigate a transfer learning approach with pretraining on the source data and finetuning based on the target data (both conducted by online SGD) for this problem. We establish sharp instance-dependent excess risk upper and lower bounds for this approach. Our bounds suggest that for a large class of linear regression instances, transfer learning with $O(N^2)$ source data (and scarce or no target data) is as effective as supervised learning with $N$ target data. In addition, we show that finetuning, even with only a small amount of target data, could drastically reduce the amount of source data required by pretraining. Our theory sheds light on the effectiveness and limitation of pretraining as well as the benefits of finetuning for tackling covariate shift problems.
Keywords: covariate shift · linear regression · finetuning · risk bound · pretraining
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Jingfeng Wu (Johns Hopkins University)
Difan Zou (University of Hong Kong)
Vladimir Braverman (Rice University)
Quanquan Gu (UCLA)
Sham Kakade (Harvard University & Amazon)

More from the Same Authors