Why We Want Contrastive Learning in Language Models

Contrastive learning aims to learn representations such that similar samples stay close to each other while dissimilar ones are far apart. Recently, it has achieved great success in self-supervised learning of visual representations and even surpassed its supervised counterparts. In this talk, I will argue why contrastive learning may provide new solutions in language model pre-training and fine-tuning. I will first describe our recent work SimCSE on how contrastive learning can be used with pre-trained language models to produce universal sentence representations. And then, I will discuss why contrastive learning can potentially lead to better pre-trained representations. I hope this talk can shed light on some limitations of pre-trained language representations as well as why contrastive learning is a great idea to tackle these problems.