Timezone: »

 
Spotlight
CoNT: Contrastive Neural Text Generation
Chenxin An · Jiangtao Feng · Kai Lv · Lingpeng Kong · Xipeng Qiu · Xuanjing Huang

Thu Dec 08 09:00 AM -- 11:00 AM (PST) @
in Lightning Talks 5B-2 »

Recently, contrastive learning attracts increasing interests in neural text generation as a new solution to alleviate the exposure bias problem. It introduces a sequence-level training signal which is crucial to generation tasks that always rely on auto-regressive decoding. However, previous methods using contrastive learning in neural text generation usually lead to inferior performance. In this paper, we analyse the underlying reasons and propose a new Contrastive Neural Text generation framework, CoNT. CoNT addresses bottlenecks that prevent contrastive learning from being widely adopted in generation tasks from three aspects -- the construction of contrastive examples, the choice of the contrastive loss, and the strategy in decoding. We validate CoNT on five generation tasks with ten benchmarks, including machine translation, summarization, code comment generation, data-to-text generation and commonsense generation. Experimental results show that CoNT clearly outperforms its baseline on all the ten benchmarks with a convincing margin. Especially, CoNT surpasses previous the most competitive contrastive learning method for text generation, by 1.50 BLEU on machine translation and 1.77 ROUGE-1 on summarization, respectively. It achieves new state-of-the-art on summarization, code comment generation (without external data) and data-to-text generation.

Author Information

Chenxin An (Fudan University)
Jiangtao Feng (Shanghai AI Lab)
Kai Lv (Fudan University)
Lingpeng Kong (Department of Computer Science, The University of Hong Kong)
Xipeng Qiu (Fudan University)
Xuanjing Huang (Fudan University)

Xuanjing Huang is a Professor of the School of Computer Science, Fudan University, Shanghai, China. She received her PhD degree in Computer Science from Fudan University in 1998. From 2008 to 2009, she is a visiting scholar in CIIR, UMass Amherst. Her research interest includes text retrieval, natural language processing, and data intensive computing. She has published dozens of papers in several major conferences including SIGIR, ACL, ICML, IJCAI, AAAI, CIKM, ISWC, EMNLP, WSDM and COLING. She has also translated the second version of “Modern Information Retrieval” to Chinese. In the research community, she served as the organizer of WSDM 2015, competition chair of CIKM 2014, tutorial chair of COLING 2010, SPC or PC member of past WSDM, SIGIR, WWW, CIKM, ACL, IJCAI, EMNLP and many other conferences.

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors