HUMAN & MACHINE INTELLIGENCE

Communication breakdown: On the low mutual intelligibility between human and neural captioning

October 27, 2022

Abstract

We compare the 0-shot performance of a neural caption-based image retriever when given as input either human-produced captions or captions generated by a neural captioner. We conduct this comparison on the recently introduced IMAGECODE data-set (Krojer et al., 2022), which contains hard distractors nearly identical to the images to be retrieved. We find that the neural retriever has much higher performance when fed neural rather than human captions, despite the fact that the former, unlike the latter, were generated without awareness of the distractors that make the task hard. Even more remarkably, when the same neural captions are given to human subjects, their retrieval performance is almost at chance level. Our results thus add to the growing body of evidence that, even when the “language” of neural models resembles English, this superficial resemblance might be deeply misleading.

Download the Paper

AUTHORS

Written by

Roberto Dessì

Eleonora Gualdoni

Francesca Franzon

Gemma Boleda

Marco Baroni

Publisher

EMNLP

Related Publications

October 04, 2023

HUMAN & MACHINE INTELLIGENCE

SPEECH & AUDIO

Decoding speech perception from non-invasive brain recordings

Alexandre Defossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, Jean Remi King

October 04, 2023

July 10, 2023

HUMAN & MACHINE INTELLIGENCE

NLP

Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks

Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, Douwe Kiela

July 10, 2023

June 21, 2023

HUMAN & MACHINE INTELLIGENCE

NLP

Benchmarking Compositionality with Formal Languages

Josef Valvoda, Naomi Saphra, Jonathan Rawski, Adina Williams, Ryan Cotterell

June 21, 2023

November 30, 2022

HUMAN & MACHINE INTELLIGENCE

NLP

ToKen: Task Decomposition and Knowledge Infusion for Few-Shot Hate Speech Detection

Badr Alkhamissy, Asli Celikyilmaz, Lambert Mathias, Mona Diab, Pascale Fung, Srini Iyer, Ves Stoyanov, Xian Li, Zornitsa Kozareva, Faisal Ladhak

November 30, 2022

Help Us Pioneer The Future of AI

We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.