HUMAN & MACHINE INTELLIGENCE

Communication breakdown: On the low mutual intelligibility between human and neural captioning

October 27, 2022

Abstract

We compare the 0-shot performance of a neural caption-based image retriever when given as input either human-produced captions or captions generated by a neural captioner. We conduct this comparison on the recently introduced IMAGECODE data-set (Krojer et al., 2022), which contains hard distractors nearly identical to the images to be retrieved. We find that the neural retriever has much higher performance when fed neural rather than human captions, despite the fact that the former, unlike the latter, were generated without awareness of the distractors that make the task hard. Even more remarkably, when the same neural captions are given to human subjects, their retrieval performance is almost at chance level. Our results thus add to the growing body of evidence that, even when the “language” of neural models resembles English, this superficial resemblance might be deeply misleading.

Download the Paper

AUTHORS

Written by

Roberto Dessì

Eleonora Gualdoni

Francesca Franzon

Gemma Boleda

Marco Baroni

Publisher

EMNLP

Related Publications

June 17, 2024

HUMAN & MACHINE INTELLIGENCE

COMPUTER VISION

D-Flow: Differentiating through Flows for Controlled Generation

Heli Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, Yaron Lipman

June 17, 2024

October 04, 2023

HUMAN & MACHINE INTELLIGENCE

SPEECH & AUDIO

Decoding speech perception from non-invasive brain recordings

Alexandre Defossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, Jean Remi King

October 04, 2023

July 10, 2023

HUMAN & MACHINE INTELLIGENCE

NLP

Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks

Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, Douwe Kiela

July 10, 2023

July 06, 2023

HUMAN & MACHINE INTELLIGENCE

REINFORCEMENT LEARNING

Augmented Language Models: a Survey

Gregoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, Thomas Scialom

July 06, 2023

Help Us Pioneer The Future of AI

We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.