Timezone: »

 
Poster
Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations
Jeff Z. HaoChen · Colin Wei · Ananya Kumar · Tengyu Ma

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #920
in Poster Session 3 »

Contrastive learning is a highly effective method for learning representations from unlabeled data. Recent works show that contrastive representations can transfer across domains, leading to simple state-of-the-art algorithms for unsupervised domain adaptation. In particular, a linear classifier trained to separate the representations on the source domain can also predict classes on the target domain accurately, even though the representations of the two domains are far from each other. We refer to this phenomenon as linear transferability. This paper analyzes when and why contrastive representations exhibit linear transferability in a general unsupervised domain adaptation setting. We prove that linear transferability can occur when data from the same class in different domains (e.g., photo dogs and cartoon dogs) are more related with each other than data from different classes in different domains (e.g., photo dogs and cartoon cats) are. Our analyses are in a realistic regime where the source and target domains can have unbounded density ratios and be weakly related, and they have distant representations across domains.

Keywords: Deep Learning Theory · representation learning theory · domain adaptation theory · self-supervised learning theory · contrastive learning theory
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Jeff Z. HaoChen (Stanford University)
Colin Wei (Stanford University)
Ananya Kumar (Stanford University)
Tengyu Ma (Stanford University)

More from the Same Authors