SPEECH & AUDIO

NLP

Cocktail HuBERT: Generalized Self-Supervised Pre-Training for Mixture and Single-Source Speech

March 27, 2023

Abstract

Self-supervised learning leverages unlabeled data effectively, improving label efficiency and generalization to domains without labeled data. While recent work has studied generalization to more acoustic/linguistic domains, languages, and modalities, these investigations are limited to single-source speech with one primary speaker in the recording. This paper presents Cocktail HuBERT, a self-supervised learning framework that generalizes to mixture speech using a masked pseudo source separation objective. This objective encourages the model to identify the number of sources, separate and understand the context, and infer the content of masked regions represented as discovered units. Cocktail HuBERT outperforms state-of-the-art results with 69% lower WER on multispeaker ASR, 31% lower DER on diarization, and is competitive on single- and multi-speaker tasks from SUPERB.

Download the Paper

Related Publications

November 13, 2026

NLP

COMPUTER VISION

Unbundling Multimodal Vision Encoding

Ellis Brown, Chris Hoang, Shang-Wen Li, Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer, Mengye Ren, Jakob Verbeek, Saining Xie, Hu Xu

November 13, 2026

November 04, 2026

SPEECH & AUDIO

GSLM2: Multi-Stream Speech Representation Learning and Generative Spoken Language Modeling

Yosuke Higuchi, Tom Fizycki, Mahi Luthra, Charles-Éric Saint-James, Azeez Syed, Angel Villar Corrales, Mido Assran, Nicolas Ballas, Jiayi Shen, Dongyan Lin, Phillip Rust, Surya Parimi, Vanessa Stark, Alexandre Allauzen, Balázs Kegl, Juan Pino, Emmanuel Dupoux

November 04, 2026

October 20, 2026

REINFORCEMENT LEARNING

NLP

Learning to Program Retrievals: Composing Search Primitives via Reinforcement Learning

Kyuyoung Kim, Zilin Xiao, Zhiyuan Wang, Avinash Atreya, Ke Yang, Jinwoo Shin, Kecheng Hao

October 20, 2026

October 19, 2026

RESEARCH

NLP

Muse Glimmer 30B: An On-Device Agentic Model

Muse Glimmer team

October 19, 2026

Help Us Pioneer The Future of AI

We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.