Research
Fairness On The Ground: Applying Algorithmic Fairness Approaches To Production Systems
March 11, 2021
Abstract
Many technical approaches have been proposed for ensuring that decisions made by machine learning systems are fair, but few of these proposals have been stress-tested in real-world systems. This paper presents an example of one team’s approach to the challenge of applying algorithmic fairness approaches to complex production systems within the context of a large technology company. We discuss how we disentangle normative questions of product and policy design (like, “how should the system trade off between different stakeholders’ interests and needs?”) from empirical questions of system implementation (like, “is the system achieving the desired tradeoff in practice?”). We also present an approach for answering questions of the latter sort, which allows us to measure how machine learning systems and human labelers are making these tradeoffs across different relevant groups. We hope our experience integrating fairness tools and approaches into large-scale and complex production systems will be useful to other practitioners facing similar challenges, and illuminating to academics and researchers looking to better address the needs of practitioners.
AUTHORS
Written by
Chloé Bakalar
Renata Barreto
Stevie Bergman
Miranda Bogen
Bobbie Chern
Sam Corbett-Davies
Melissa Hall
Isabel Kloumann
Michelle Lam
Manish Raghavan
Joshua Simons
Jonathan Tannen
Edmund Tong
Kate Vredenburgh
Jiejing Zhao
Related Publications
May 14, 2025
Core Machine Learning
UMA: A Family of Universal Models for Atoms
Brandon M. Wood, Misko Dzamba, Xiang Fu, Meng Gao, Muhammed Shuaibi, Luis Barroso-Luque, Kareem Abdelmaqsoud, Vahe Gharakhanyan, John R. Kitchin, Daniel S. Levine, Kyle Michel, Anuroop Sriram, Taco Cohen, Abhishek Das, Ammar Rizvi, Sushree Jagriti Sahoo, Zachary W. Ulissi, C. Lawrence Zitnick
May 14, 2025
May 13, 2025
Human & Machine Intelligence
Speech & Audio
Emergence of Language in the Developing Brain
Linnea Evanson, Christine Bulteau, Mathilde Chipaux, Georg Dorfmüller, Sarah Ferrand-Sorbets, Emmanuel Raffo, Sarah Rosenberg, Pierre Bourdillon, Jean Remi King
May 13, 2025
May 13, 2025
Human & Machine Intelligence
Dynadiff: Single-stage Decoding of Images from Continuously Evolving fMRI
Marlène Careil, Yohann Benchetrit, Jean-Rémi King
May 13, 2025
April 28, 2025
LlamaFirewall: An open source guardrail system for building secure AI agents
Sahana Chennabasappa, Cyrus Nikolaidis, Daniel Song, Stephanie Ding, Shengye Wan, Rashnil Chaturvedi, James Crnkovich, Beto de Paola, Lauren Deason, Nicholas Doucette, Dominik Gabi, Alekhya Gampa, Kat He, David Molnar, Abraham Montilla, Jean-Christophe Testud, Spencer Whitman, Joshua Saxe
April 28, 2025
April 08, 2021
Responsible AI
Integrity
Towards measuring fairness in AI: the Casual Conversations dataset
Caner Hazirbas, Joanna Bitton, Brian Dolhansky, Jacqueline Pan, Albert Gordo, Cristian Canton Ferrer
April 08, 2021
November 16, 2021
Computer Vision
How Meta is working to assess fairness in relation to race in the U.S. across its products and systems
Rachad Alao, Miranda Bogen, Jingang Miao Ilya Mironov, Jonathan Tannen
November 16, 2021
October 12, 2021
Computer Vision
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Pingchuan Ma, Rodrigo Mira, Stavros Petridis, Bjorn W. Schuller,Maja Pantic
October 12, 2021
October 14, 2021
Graphics
Computer Vision
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Ramakrishnan, Fiona Ryan,Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, …
October 14, 2021
Help Us Pioneer The Future of AI
We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.
Our approach
Latest news
Foundational models
Meta © 2025