RESEARCH

NLP

Unbundling Multimodal Vision Encoding

November 13, 2026

Abstract

How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and compute, comparing them cannot isolate the effect of each choice. We disentangle these choices via a controlled framework where we pretrain all models from scratch on common data and compare them on visual question answering and language benchmarks at matched lifetime compute. We find that the visual prior is not necessary; an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage. What matters instead is dedicated vision capacity: an encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder's parameter count. Yet that capacity need not precede multimodal fusion, as vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token. These results also hold across backbone sizes, when encoder pretraining is treated as free, and when starting from a pretrained language model. Overall, our findings favor dedicated vision capacity trained jointly with the backbone and placed early within it.

Download the Paper

AUTHORS

Written by

Ellis Brown

Chris Hoang

Shang-Wen Li

Mike Lewis

Marjan Ghazvininejad

Luke Zettlemoyer

Lucas Beyer

Mengye Ren

Jakob Verbeek

Saining Xie

Hu Xu

Publisher

arXiv

Research Topics

Natural Language Processing (NLP)

Computer Vision

Core Machine Learning

Related Publications

October 20, 2026

REINFORCEMENT LEARNING

NLP

Learning to Program Retrievals: Composing Search Primitives via Reinforcement Learning

Kyuyoung Kim, Zilin Xiao, Zhiyuan Wang, Avinash Atreya, Ke Yang, Jinwoo Shin, Kecheng Hao

October 20, 2026

October 19, 2026

RESEARCH

NLP

Muse Glimmer 30B: An On-Device Agentic Model

Muse Glimmer team

October 19, 2026

October 14, 2026

RESEARCH

Human-Guided Agentic AI for End-to-End EHR Research: Sequential Antibody-Drug Conjugate Pathways in Breast Cancer

Arturo Loaiza-Bonilla

October 14, 2026

October 14, 2026

RESEARCH

Pickleball injuries, search interest, and participation in the United States, 2014-2023: an ecological time series study

Daniel Kim

October 14, 2026

Help Us Pioneer The Future of AI

We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.