A new system to help ensure ads are delivered fairly to different demographic groups

Advocates, researchers, and others have long been focused on the relationship between advertising and economic mobility, especially when it comes to ads that pertain to housing, employment, and credit opportunities. People can more easily pursue a new job or consider moving to a new home when they’re aware of the options out there, but the enduring effects of historically unequal treatment in these areas still shape the economic opportunities of too many. Across both the tech industry and the AI research community, approaches to fairness are still evolving, including in the realm of personalized, auction-based advertising systems. But we know we can’t wait for a consensus to make progress in addressing these important concerns, so today we’re sharing early but significant changes Meta has made to our advertising system that build on and reflect the evolution of our approach to ad fairness.

After more than a year of collaboration with the DOJ and HUD, Meta has developed a new method called the Variance Reduction System (VRS) to help ensure an equitable distribution of ads on our services. Our policies already prohibit advertisers from using our ad products to discriminate against individuals or groups of people, and we’ve implemented additional safeguards, such as disallowing the use of gender, age, or zip code targeting for certain ads. But even without these sorts of targeting options, factors such as people’s interests or activity on a service could affect how ads are distributed to different demographic groups. The goal of the VRS is to help ensure that the audience that ends up seeing a housing, employment, or credit ad more closely reflects the eligible targeted audience for that ad. We’ll do this by regularly measuring the actual audience for a particular ad to see how it compares with the demographic distribution — age, gender, and estimated race or ethnicity — of the audience the advertiser has selected. (Our initial launch will focus on gender and estimated race.) To implement this system in a way that respects people’s privacy, the system relies on aggregate measurements as well as privacy-enhancing approaches to measure race and ethnicity at the aggregate level. We will begin by applying the VRS to housing ads in the United States, and will expand it to employment and credit ads in the United States over the next year.

This system complements and extends our other longstanding efforts to help advance fairness in our ads system, such as limiting the targeting options available to advertisers running housing ads on our services, to make progress toward a more equitable distribution of ads through our ad delivery process.

In addition to sharing this blog post summarizing how the VRS works, we’re publishing technical details in a new white paper, available here:

How the VRS works

The VRS is an offline reinforcement learning framework with the explicit goal of minimizing the variance in number of ad views between people who have seen the ad and the broader eligible audience of people who could have seen the ad. Reinforcement learning is a type of machine learning that learns from trial and error to optimize toward a predefined outcome — in this case, to minimize ad impression variance across demographic subgroups no matter the cause of that variance. Importantly, the VRS will not be provided with individual-level age, gender, or estimated race/ethnicity to make these determinations. The system will instead receive aggregate measurements of variance across these demographics.

The system starts by measuring the demographic distribution of the baseline or eligible ratio of the age, gender, and estimated racial/ethnic distribution of the population of users to whom an advertiser has indicated they would like their ad to be displayed. As the ad is being delivered, we periodically measure the delivery ratio, or the demographic distribution of the impressions of an ad.

The VRS relies on a controller that has the ability to change one of the values used to calculate an ad’s total value in our ad auction, which will influence the probability that a given ad will win an auction and be shown to a user.

When there is an opportunity to show an ad to someone, all the ads that are eligible to be shown to that person are narrowed down through the ad auction, where the total value of each ad is calculated and compared. There are three key components of total value: the advertiser bid (how much they are willing to pay), the estimated action rate, and ad quality. We use AI models to predict estimated action rate, or a person’s likelihood of taking the advertiser’s desired action (such as visiting a website, watching a video, or completing a purchase). This prediction is based on ad delivery inputs, such as clicking an ad, engaging with a Page, or installing an app.

Then a process in our system called pacing adjusts the ad’s total value by adjusting what’s called the pacing multiplier. Pacing is already used in ad auctions to help ensure that an advertiser’s entire campaign budget is not spent in just a few days, and now it will help the VRS accomplish its goal.

The VRS begins when an ad for housing, employment, or credit wins the auction and starts being shown to people. After the ad has been shown to a large enough group of people, the system measures the aggregate age, gender, and estimated race/ethnicity distribution of those who have seen the ad. (To measure the estimated race/ethnicity distribution of these groups, VRS relies on a widely used method of measurement called Bayesian Improved Surname Geocoding, which we built with added privacy enhancements, including differential privacy.) These measurements are compared with measurements of the population of people who are more broadly eligible to see the ad, and if there is a difference in distributions, the system is instructed to adjust pacing multipliers.

The VRS remeasures the audience’s demographic distribution and updates the pacing of ads throughout the campaign, working to reduce variance between the audiences. When there’s a new chance to show someone an ad, the system uses the latest demographic measurements, along with limited information about that person, to determine how to best adjust the pacing of the bid in order to encourage the ad to be distributed to an audience that more closely reflects the ad’s eligible targeted audience.

Reinforcement learning helps the VRS learn how to do this effectively before it is used in our ad auction. With reinforcement learning, the system has been trained offline to reduce the difference between the distribution of people who have seen the ad and the distribution of people who could have seen the ad, based on these demographic measurements.

Privacy safeguards

One of the key priorities of the VRS is to reduce variance for ads delivery in a privacy-preserving way. In particular, we aim to avoid demographic information leaking into the VRS or being discernable to human analysts reviewing the VRS or its outputs. To implement the VRS while also taking into account people’s privacy, we use the following privacy-preserving approaches:

• The VRS will not have access to individuals’ age, gender, or estimated race/ethnicity.

• Estimated race/ethnicity will be measured using Meta’s privacy-enhanced implementation of Bayesian Improved Surname Geocoding.

• Aggregate demographic measurements that are generated and used by the VRS will include differential privacy noise to help prevent the system from learning and subsequently acting on individual-level demographic information with high fidelity.

We are also exploring ways to obfuscate the generation of user summary vectors by randomly rotating these summary vectors using a private rotation matrix, which could help prevent adversarial actors from reconstructing the user summarization process.

Challenges and limitations

In developing this new method, we encountered a number of challenges and questions. We hope that by highlighting some of the questions and tensions we have navigated or are still working through, we can help inform conversations across industry and with experts in civil society, academia, and policymaking who have similar goals of advancing the equitable distribution of ads. Ultimately, we believe our collective progress will be fastest if we work together to chart a responsible path forward. Some of the technical challenges include:

• Availability of demographic data: Gender and age are data more readily available to tech services since they are commonly collected at account creation. But research has shown that for many companies, the absence of labeled demographic data, particularly race and ethnicity, has raised significant barriers to systematically investigating the potential for differences across those protected characteristics, or to monitoring the progress of equity efforts over time.

• Multiobjective learning: The VRS simultaneously prioritizes multiple objectives: helping reduce variance for gender, for age, and for estimated race/ethnicity. To do so for all of these effectively, it must not consistently prioritize any one of those goals over the others. Like most societal challenges, navigating multiple objectives and potential trade-offs will likely mean that multiobjective systems are unlikely to be able to achieve every component goal perfectly.

• Low-volume ads: The number of ad impressions in a given ad’s run relate directly to how many opportunities the system has to find successful strategies to reduce variance.

• System latency: Because the VRS works based on iterative measurements that rely on several pieces of technical infrastructure, there will be latency between when measurements are taken and when those measurements are shared back to the VRS.

Important questions also arise in terms of how services should continue navigating trade-offs between privacy and fairness, including whether to apply similar tools to other demographic categories for which where privacy-preserving measurement methods have not yet been developed. Additionally, we expect important discussions around how baselines for such systems are selected and measured, and how to ensure that fairness goals are sustained beyond individual services, since people might use multiple online and offline sources to find information about housing, jobs, or financial tools.

Evolving the field of AI fairness

Fairness in AI is a dynamic and evolving field. The changes described in our paper were informed by substantial consultation with a broad array of stakeholders and represent a significant technological advancement in how AI can be responsibly used to deliver personalized ads. We are excited to pioneer this effort, and we hope that our sharing details on this work will help other AI and digital advertising practitioners make progress to advance fairness and equity and avoid amplifying societal biases, whose impact extends far beyond any one service. We know that our ongoing progress — in both ad fairness and broader civil rights initiatives — will be determined not just by our commitment to this work but also by the concrete changes we make in our products. With the deployment of the VRS, we’re pleased to be making a tangible impact on Meta’s ad-serving systems.