Cross-lingual pretraining sets new state of the art for natural language understanding

What the research is:

This research offers a new approach to pretraining cross-lingual models for natural language processing (NLP) tasks. Our method delivers a significant improvement over the previous state of the art in both supervised and unsupervised machine translation, as well as in cross-lingual text classification of low-resource languages. For example, we set new accuracy marks for unsupervised German-English machine translation and supervised Romanian-English translation. Specifically, this pretraining technique pushes the performance of unsupervised machine translation systems closer to that of supervised ones and even surpasses them on low-resource language pairs, such as Romanian-English.

We are providing code and pretrained models to the community so that others can use and build upon this work.

How it works:

We developed a translation language modeling (TLM) method that is an extension of masked language modeling (MLM), a popular and successful technique that trains NLP systems by making the model deduce a randomly hidden or masked word from the other words in the sentence. In our work, we concatenate parallel sentences in a corpora consisting of two different languages. Then the model learns by analyzing text in either language or in both of them to infer the masked word.

This approach has allowed us to improve performance for zero-shot cross-lingual classification, to improve initialization of supervised and unsupervised neural machine translation (NMT) systems, and to create unsupervised cross-lingual word embeddings. The diagram above illustrates how our model works.

Why it matters:

Pretraining — using unlabeled datasets to help an NLP model perform a particular task — has been shown to be very effective in improving results. This work is the first to demonstrate how cross-lingual pretraining can help produce even greater improvements. Our approach is a significant step for cross-lingual understanding, which is essential for providing better NLP systems for low-resource languages. It has also shown improvements in a wide range of NLP tasks. For instance, our model improves the performance of unsupervised machine translation systems by up to 9 BLEU (bilingual evaluation understudy, a common machine translation metric), even surpassing the performance of supervised approaches on language pairs with few translation resources. In cross-lingual classification (XNLI), our approach beats the state of the art across all languages by an average of 5 percent. It is also the first time that a cross-lingual model has outperformed monolingual models on high-resource languages.

This work follows a series of recent achievements by FAIR on cross-lingual text representations (including MUSE, UMT, LASER, and XNLI).

Read the full paper:

Cross-Lingual Language Model Pretraining

The code for our approach is publicly available here.