Unified Vision–Language Modeling via Concept Space Alignment

Abstract

We introduce v-Sonar, a vision–language embedding space extended from the text-only embedding space Sonar (Omnilingual Embeddings Team et al., 2026), which supports 1500 text languages and 177 speech languages. To construct v-Sonar, we propose a post-hoc alignment pipeline that maps the representations of an existing vision encoder into the Sonar space. We thoroughly evaluate v-Sonar and show that its embeddings achieve competitive performance on text-to-video retrieval. Equipped with the Sonar text decoder, v-Sonar further surpasses state-of-the-art vision–language models on video captioning tasks, including Dream-1k (Bleu 24.3 vs. 19.6) and Vatex (Bleu 45.0 vs. 41.5). Leveraging v-Sonar, we first demonstrate that the Large Concept Model (LCM; LCM team et al. 2024) operating in Sonar and trained with English text only, can perform both single- and multi-visual concept understanding in a zero-shot manner. Finally, we introduce v-LCM, which extends the LCM with vision–language instruction tuning. v-LCM encodes vision and language inputs into an unified sequence of latent embeddings via v-Sonar and Sonar, and it is trained with the same latent diffusion objective for next-embedding prediction as in LCM’s text-only pre-training. Experiments on a large-scale multilingual and -modal instruction–tuning data mixture highlight the potential of v-LCM: v-LCM matches state-of-the-art vision-language models on tasks covering image/video captioning and question answering, while significantly outperforming them across 61 rich- to low-resource languages out of all 62 tested languages.