Timezone: »

Cross-lingual Retrieval for Iterative Self-Supervised Training
Chau Tran · Yuqing Tang · Xian Li · Jiatao Gu

Mon Dec 07 07:50 PM -- 08:00 PM (PST) @ Orals & Spotlights: Language/Audio Applications

Recent studies have demonstrated the cross-lingual alignment ability of multilingual pretrained language models. In this work, we found that the cross-lingual alignment can be further improved by training seq2seq models on sentence pairs mined using their own encoder outputs. We utilized these findings to develop a new approach --- cross-lingual retrieval for iterative self-supervised training (CRISS), where mining and training processes are applied iteratively, improving cross-lingual alignment and translation ability at the same time. Using this method, we achieved state-of-the-art unsupervised machine translation results on 9 language directions with an average improvement of 2.4 BLEU, and on the Tatoeba sentence retrieval task in the XTREME benchmark on 16 languages with an average improvement of 21.5% in absolute accuracy. Furthermore, CRISS also brings an additional 1.8 BLEU improvement on average compared to mBART, when finetuned on supervised machine translation downstream tasks.

Author Information

Chau Tran (Facebook AI)
Yuqing Tang (Facebook AI)
Xian Li (Facebook AI)
Jiatao Gu (Facebook AI Research)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors