• Update: 12/11/2023: Audiobox's interactive demo and research paper are now available.
• Audiobox is Meta’s new foundation research model for audio generation.
• Audiobox can generate voices and sound effects using a combination of voice inputs and natural language text prompts — making it easy to create custom audio for a wide range of use cases.
• As part of our ongoing commitment to responsible research conduct, we're inviting researchers and academic institutions to apply for a grant to conduct safety and responsibility research with Audiobox.

Audio plays a fundamental role in many forms of media, from movies to podcasts, audiobooks, and video games. But producing quality audio can often be a challenging process that requires access to extensive sound libraries as well as deep domain expertise (sound engineering, foley, voice acting, etc.) to yield optimal results — expertise that the public, or even hobbyists, may not possess.

We’re releasing Audiobox to a hand-selected group of researchers and academic institutions with a track record in speech research to help further the state of the art in this research area and ensure we have a diverse set of partners to tackle the responsible AI aspects of this work. In the future, we believe research breakthroughs like Audiobox will lower the barrier of accessibility for audio creation and make it easy for anyone to become an audio content creator. Creators could use models like Audiobox to generate soundscapes for videos or podcasts, custom sound effects for games, or any of a number of other use cases.

Being able to use text and voice inputs also greatly enhances Audiobox’s controllability compared to Voicebox. Audiobox users can use text description prompts to specify the style of speech and sound effects, a feature that was not supported in Voicebox. When a voice input and text prompt are used together, the voice input anchors the timbre, and the text prompt can be used to change other aspects.

Audiobox inherits Voicebox’s guided audio generation training objective and flow-matching modeling method to allow for audio infilling. With infilling, users can also use the model to polish sound effects (adding different thunder sounds into a raining soundscape, for example).

AI for audio generation has made significant progress over the past year. But, as with all AI innovations, we must work to help ensure responsible use. The known issues with AI cannot be addressed by any individual or single organization alone. That’s why collaboration with the research community on state-of-the-art models is more important now than ever.

For these tools to be better and safer for everyone, the AI community must be empowered to build on top of our work and continue to develop these innovations responsibly. But access must be shared in the right way. To honor this and our ongoing commitment to open science, we’re releasing Audiobox under a research-only license to a limited number of hand-selected researchers and institutions.

We’re inviting researchers and institutions who have been previously involved in speech research, and who want to pursue responsibility and safety research on the latest Audiobox models, to apply.

We've also released an interactive demo that showcases Audiobox’s capabilities.

Tools like Audiobox can raise concerns about voice impersonation or other abuses. As part of our commitment to building generative AI features responsibly, we’ve implemented new technologies to help address these issues.

Both the Audiobox model and our interactive demo feature automatic audio watermarking so any audio created with Audiobox can be accurately traced to its origin. Our watermarking method embeds a signal into the audio that’s imperceptible to the human ear but can be detected all the way down to the frame level using a model capable of finding AI-generated segments in audio.

We’ve tested this method against a broad range of attacks and found it to be more robust than even current state-of-the-art solutions, making it extremely difficult for bad actors to try and bypass detection by modifying the AI-generated audio.

Additionally, similar to how websites use CAPTCHAs to deter bots and spam, our interactive demo includes a voice authentication feature to safeguard against impersonation. Anyone who wants to add a voice to the Audiobox demo will have to speak a voice prompt using their own voice. The prompt changes at regular, rapid intervals and makes it extremely difficult to add someone else’s voice with pre-recorded audio.

To help ensure robustness across different groups of speakers, we tested the performance of Audiobox on speakers of different genders and with different native languages and verified that the performance is close across all groups of speakers.

In the long term, it will be crucial to move from building specialized audio generative models that can only generate one type of audio (such as speech or sound) to building generalized audio generative models that can generate any audio. With such models we can perform any generative audio task that requires understanding beyond a single modality. This will make it simpler for developers to build towards a more dynamic and wide range of use cases.

Audiobox is an important step toward democratizing audio generation. We envision a future where everyone can more easily and efficiently create audio that is tailored to their use cases. Our hope is that we can see the same creativity sparked by advancements in text and image generation happen for audio as well, for both professionals and hobbyists. Content creation, narration, sound editing, game development, and even AI chatbots can all benefit from the capabilities of audio generation models.