The world we live in has never been more interconnected—the global proliferation of the internet, mobile devices, social media, and communication platforms gives people access to more multilingual content than ever before. In such a context, having an on-demand ability to communicate and understand information in any language becomes increasingly important. While such a capability has long been dreamed of in science fiction, AI is on the verge of bringing this vision into technical reality.

In keeping with our approach to open science, we’re publicly releasing SeamlessM4T under CC BY-NC 4.0 to allow researchers and developers to build on this work. We’re also releasing the metadata of SeamlessAlign, the biggest open multimodal translation dataset to date, totaling 470,000 hours of mined speech and text alignments. We make it easy for the community to perform mining on their own monolingual datasets with SONAR, a complete suite of speech and text sentence encoders, and stopes, our library for multimodal data processing and parallel data mining. All research advancements are supported by fairseq2, our next-generation sequence modeling library.

Building a universal language translator, like the fictional Babel Fish in The Hitchhiker’s Guide to the Galaxy, is challenging because existing speech-to-speech and speech-to-text systems only cover a small fraction of the world’s languages. SeamlessM4T represents a significant breakthrough in the field of speech-to-speech and speech-to-text by addressing the challenges of limited language coverage and a reliance on separate systems, which divide the task of speech-to-speech translation into multiple stages across subsystems. These systems can leverage large amounts of data and generally perform well for only one modality. Our challenge was to create a unified multilingual model that could do it all.

We believe the work we’re announcing today is a significant step forward in this journey. Our single model provides on-demand translations that enable people who speak different languages to communicate more effectively. We significantly improve performance for the low and mid-resource languages we support. These are languages that have smaller digital linguistic footprints. We also maintain strong performance on high-resource languages, such as English, Spanish, and German. SeamlessM4T implicitly recognizes the source languages, without the need for a separate language identification model.

Building a unified model requires a sequence modeling toolkit that is lightweight and easily composable with other modern PyTorch ecosystem libraries. We redesigned fairseq, our original sequence modeling toolkit. With more efficient modeling and data loader APIs, fairseq2 helps power the modeling behind SeamlessM4T.

For the model, we use the multitask UnitY model architecture, which is capable of directly generating translated text and speech. This new architecture also supports automatic speech recognition, text-to-text, text-to-speech, speech-to text, and speech-to-speech translations that are already a part of the vanilla UnitY model. The multitask UnitY model consists of three main sequential components. Text and speech encoders have the task of recognizing speech input in nearly 100 languages. The text decoder then transfers that meaning into nearly 100 languages for text followed by a text-to-unit model to decode into discrete acoustic units for 36 speech languages. The self-supervised encoder, speech-to-text, text-to-text translation components, and text-to-unit model are pre-trained to improve the quality of the model and for training stability The decoded discrete units are then converted into speech using a multilingual HiFi-GAN unit vocoder.

Data-driven models like SeamlessM4T usually benefit from large amounts of high-quality end-to-end data, namely speech-to-text and speech-to-speech data. Relying only on human transcribed and translated speech does not scale to tackle the challenging task of speech translation for 100 languages. We build upon our pioneering work on text-to-text mining using a similarity measure in a joint embedding space, and initial work in speech mining to create additional resources to train the SeamlessM4T model.

First, we build a new massively multilingual and -modal text embedding space for 200 languages, named SONAR (Sentence-level mOdality- and laNguage-Agnostic Representations), which substantially outperforms existing approaches like LASER3 or LaBSE in multilingual similarity search. We then apply a teacher-student approach to extend this embedding space to the speech modality and currently cover 36 languages. Mining is performed in data from publicly available repositories of web data (tens of billions of sentences) and speech (4 million hours). In total, we were able to automatically align more than 443,000 hours of speech with texts and create about 29,000 hours of speech-to-speech alignments. This corpus, dubbed SeamlessAlign, is the largest open speech/speech and speech/text parallel corpus in terms of total volume and language coverage to date.

To more accurately evaluate the system without depending on text-based metrics, we extended our text-less metric into BLASER 2.0, which now enables evaluation across speech and text units with similar accuracy compared to its predecessor. When tested for robustness, our system performs better against background noises and speaker variations in speech-to-text tasks (average improvements of 37% and 48%, respectively) compared to the current state-of-the-art model.

SeamlessM4T also outperforms previous state-of-the-art competitors.

It is important that translation systems are accurate. As with all AI systems, there are inherent risks that the model could mistranscribe what a person wants to say or generate outputs that are toxic or inaccurate.

At Meta, our AI research and development follows a responsible framework that is guided by our five pillars of Responsible AI. In line with our commitment to responsible AI, we conducted research on toxicity and bias to help us understand which areas of the model might be sensitive. For toxicity, we extended our highly multilingual toxicity classifier to speech to help identify toxic words from speech inputs and outputs. We filtered unbalanced toxicity in training data. If input or output contained different amounts of toxicity, we removed that training pair.

The demo we’re releasing today showcases the capabilities of SeamlessM4T and is an important part of the research. We detect toxicity in both the input and the output for the demo. If toxicity is only detected in the output, it means that toxicity is added. In this case, we include a warning and do not show the output. When comparing our models to the state of the art, we significantly reduce added toxicity on both speech-to-speech and speech-to-text translation.

Gender bias, where the results unfairly favor a gender and sometimes default to gender stereotypes, is another area we are beginning to evaluate in languages at scale. We are now able to quantify gender bias in dozens of speech translation directions by extending our previously designed Multilingual HolisticBias dataset to speech.

Our work around safety and security is an ongoing effort. We’ll continue to research and take action in this area to continuously improve SeamlessM4T and reduce any instances of toxicity we see in the model.

With state-of-the-art results, we believe SeamlessM4T is an important breakthrough in the AI community’s quest to create universal multitask systems. In keeping with our approach to open science, we’re excited to share our model publicly to allow researchers and developers to build on this technology.

This is only the latest step in our ongoing effort to build AI-powered technology that helps connect people across languages. In the future, we want to explore how this foundational model can enable new communication capabilities—ultimately bringing us closer to a world where everyone can be understood.