Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis

Abstract

We present Alignment-Free Text-Audiobox (Text-AB), a unified framework for high-quality voice dubbing and full-duplex dialogue synthesis. Building on a Diffusion Transformer backbone trained with a flow-matching objective, Text-AB departs from prior Audiobox system along three key dimensions. First, it operates in a latent diffusion framework using DAC-VAE features that encode 48 kHz waveforms into a 25 Hz low-rate latent sequence, providing over 10× higher compression than previous EnCodec representations while improving resynthesis quality. Second, Text-AB is alignment-free: it consumes raw text via an off-the-shelf text encoder and learns text–speech alignment through cross-attention, removing the need for forced alignment and explicit duration prediction. Third, we scale the model and data substantially, pretraining a 3B-parameter model on 480k hours of monolingual speech, followed by supervised fine-tuning on three downstream tasks—cross-lingual voice dubbing, full-duplex dialogue synthesis, and emotional full-duplex dialogue synthesis. At inference time, Text-AB supports both one-shot generation for up to ∼1 min speech and arbitrarily long-form generation via a multi-diffusion scheme. We further incorporate a multi-stage reranking strategy to effectively enhance generation quality based on automated metrics. On a real-world dubbing benchmark, Text-AB delivers a step-change improvement over the latest internal dubbing system, with large gains in prosody similarity, voice similarity, naturalness, and shareability. For full-duplex dialogue synthesis, Text-AB approaches human recordings on short-form conversations and substantially outperforms the latest internal model on long-form human-likeness and expressivity, while natively modeling turn-taking, back-channeling, and emotional dynamics. For emotional dialogue synthesis, Text-AB with emotion conditioning significantly improves emotion alignment and emotional interaction quality over the baseline without emotion conditioning.