Disentangling the Factors of Convergence between Brains and Computer Vision Models

Abstract

Many AI models trained on natural images develop representations that resemble those of the human brain. However, the exact factors that drive this brain-model similarity remain poorly understood. In order to disentangle how the model architecture, training recipe and data type independently lead a neural network to develop brain-like representations, we trained a family of self-supervised vision transformers (DINOv3) that systematically varied these different factors. We compare their representations of natural images to those of the human brain recorded with both ultra-high field functional magnetic resonance imaging (fMRI) and magneto-encephalography (MEG), providing high resolution in spatial and temporal analyses. We assess the brain-model similarity with three complementary metrics focusing on overall representational similarity, topographical organization, and temporal dynamics. We show that all three factors – model size, training amount, and image type – independently and interactively impact each of these brain similarity metrics. In particular, the largest DINOv3 models trained with the largest amount of human-centric images reach the highest brain-similarity scores. Importantly, this emergence of brain-like representations in AI models follows a specific chronology during training: models first align with the early representations of the sensory cortices, and only align with the late and prefrontal representations of the brain with considerably more training data. Finally, this developmental trajectory is indexed by both structural and functional properties of the human cortex: the representations that are acquired last by the models specifically align with the cortical areas with the largest developmental expansion, the largest thickness, the least myelination, and the slowest timescales. Overall, these findings disentangle the interplay between architecture and experience in shaping how artificial neural networks come to see the world as humans do, thus offering a promising framework to understand how the human brain comes to represent its visual world.