Research
Adaptive Attention Span in Transformers
July 27, 2019
Abstract
We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining control over their memory footprint and computational time. We show the effectiveness of our approach on the task of character level language modeling, where we achieve state-of-the-art performances on text8 and enwiki8 by using a maximum context of 8k characters.
Related Publications
April 17, 2025
Human & Machine Intelligence
Conversational AI
Collaborative Reasoner: Self-improving Social Agents with Synthetic Conversations
Ansong Ni, Ruta Desai, Yang Li, Xinjie Lei, Dong Wang, Ramya Raghavendra, Gargi Ghosh, Daniel Li (FAIR), Asli Celikyilmaz
April 17, 2025
April 16, 2025
Robotics
Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Paul McVay, Sergio Arnaud, Ada Martin, Arjun Majumdar, Krishna Murthy Jatavallabhula, Phillip Thomas, Ruslan Partsey, Daniel Dugas, Abha Gejji, Alexander Sax, Vincent-Pierre Berges, Mikael Henaff, Ayush Jain, Ang Cao, Ishita Prasad, Mrinal Kalakrishnan, Mike Rabbat, Nicolas Ballas, Mido Assran, Oleksandr Maksymets, Aravind Rajeswaran, Franziska Meier
April 16, 2025
April 14, 2025
Graphics
Autoregressive Distillation of Diffusion Transformers
Yeongmin Kim, Sotiris Anagnostidis, Yuming Du, Edgar Schoenfeld, Jonas Kohler, Markos Georgopoulos, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu
April 14, 2025
March 24, 2025
Integrity
Data Taggants: Dataset Ownership Verification Via Harmless Targeted Data Poisoning
Wassim (Wes) Bouaziz, Nicolas Usunier, El Mahdi El Mhamdi
March 24, 2025
April 08, 2021
Responsible AI
Integrity
Towards measuring fairness in AI: the Casual Conversations dataset
Caner Hazirbas, Joanna Bitton, Brian Dolhansky, Jacqueline Pan, Albert Gordo, Cristian Canton Ferrer
April 08, 2021
April 30, 2018
The Role of Minimal Complexity Functions in Unsupervised Learning of Semantic Mappings | Facebook AI Research
Tomer Galanti, Lior Wolf, Sagie Benaim
April 30, 2018
April 30, 2018
Computer Vision
NAM – Unsupervised Cross-Domain Image Mapping without Cycles or GANs | Facebook AI Research
Yedid Hoshen, Lior Wolf
April 30, 2018
December 11, 2019
Speech & Audio
Computer Vision
Hyper-Graph-Network Decoders for Block Codes | Facebook AI Research
Eliya Nachmani, Lior Wolf
December 11, 2019
Help Us Pioneer The Future of AI
We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.
Foundational models
Our approach
Latest news
Foundational models
Meta © 2025