Testing Quality in the Meta Ads Auction: Building the UQL Testing Framework

When advertisers think about Meta’s ads auction, they often focus on bids and budgets. But inside the system, quality is just as important as how much an advertiser is willing to pay. Meta ranks ads using a “total value” calculation that combines the advertiser’s bid, the estimated probability a user will take the desired action, and an assessment of ad quality or user value. To keep the experience healthy for people, quality is tracked through signals like engagement, negative feedback, and post‑click behavior, and surfaced back to advertisers through “Ad Relevance Diagnostics” (Quality Ranking, Engagement Rate Ranking, Conversion Rate Ranking).

Inside Meta, the Unified Quality Library (UQL) is one of the core engineering levers that shapes this quality side of the auction. This post is about how we moved from slow, manual, visual checks of UQL-based components to a proper, scalable testing framework: the UQL Testing Framework (UTF).

$\text{Total bid} = \text{Advertiser bid} + \text{multiplicative bid components}$

$\text{Organic bid} = \sum(\text{additive bid components}) + \pi(\text{multiplicative bid components})$

Highlevel diagram of the Ads Serving Stack at Meta

Meta's complex ads serving stack is made up of different services with various ranking and auction phases.
The details of the stack is not within the scope of this article, but a high-level extremely simplified version of it can be viewed below.

What UQL Is

UQL (Unified Quality Library) is an internal library and domain‑specific language used to describe quality bid components—the pieces of logic that determine the “quality” part of an ad’s total bid in Meta’s auction.

At a high level:

• The total value of an ad impression is a blend of:

  • Bid – how much the advertiser is willing to pay.
  • Estimated action rate – how likely a user is to take the goal action (click, convert, etc.).
  • Ad quality / user value – how relevant and high‑quality the ad experience is for that user.

• UQL is the engineering tool that lets us define the quality side of this equation in a structured, composable way. Each quality bid component encapsulates some notion of value or risk (for example, user experience, creative quality, engagement predictions, or policy‑related risk) and how that should impact the final quality score.

• These components are composed into a directed acyclic graph (DAG) that runs across tens of serving stages and surfaces (Feeds, Stories, Reels, etc.), ultimately producing the quality contribution that influences an ad’s rank and cost.

In other words, UQL is how we describe what “quality” means for the auction, in a way that can be reasoned about, versioned, and evolved safely over time.

A Fictional Example DAG

Note: The following example is entirely fictional, for explanation purposes only.
It does not reflect any real model, signal, or production logic at Meta.

To make the idea of a quality bid component concrete, imagine a toy component called Ad Creative Richness Bid (ACR). In this fictional world, the richer or more polished an ad’s creative is, the more it might help the ad in the quality side of the auction, with the effect scaled by how much a given age group cares about creative polish.

We could sketch the structure of such a component as:

Loading diagram…

Read left‑to‑right:

• Ad Creative Richeness Bid flows into an addition node + (ADD1) that combines:
  • An age‑dependent scalar (Age group Scalar, AG) capturing how much this age group cares about creative richness.
  • A creative‑specific scalar (ACR scalar, ACS) capturing how rich the ad creative is.
  • An offset term (OFS) that could vary by surface or serving stage.

Again, this is not a real Meta component—just a simple DAG to illustrate how UQL‑style building blocks can be composed into a quality bid.

Imagine that this bid was used on 10 surfaces; (FB feed, FB story, FB feed, IG reel, Messenger, IG feed, IG Explore, IG story, IG reel and threads), across two of the three serving stages in ads (AdFinder & AdRanker).
Testing the component would quickly exlode to 20 different variations, potentially with different offsets (as normally those are surface/stage-dependable), to test visually before any change.

Typical UQL authoring user flow

Black box voodoo in the figure above is a custom compiler that takes configerator DSL-written “python files” and process them into JSON files representing the exported thrift structs.
This is the essence of why verifying the correctness and ensuring the stability and integrity of UQL-authored bids was very hard.

Limitations Before UTF

Before the UQL Testing Framework, testing UQL components was almost entirely visual and manual:

• Primitive DAG visualization as the main safety net
  The primary way to verify a change was to render the UQL DAG in an internal visualizer across many serving stages and ad surfaces, then manually inspect the resulting graph to see if anything looked “off.” This process could easily take hours for non‑trivial changes, and it didn’t scale well to the complexity of modern quality logic.

• High cognitive load and review overhead
  Reviewers had to mentally diff complex graphs, remember expected shapes and invariants, and spot subtle structural regressions just by eyeballing nodes and edges. Under time pressure, it was easy for both authors and reviewers to miss small but impactful changes.

• No programmatic regression guarantees
  There was no unit‑test‑style, machine‑enforced contract for how a UQL component should look or behave. If a refactor or a new experiment unintentionally dropped a signal, changed a weight, or rewired a subgraph, the only protection was whether a human happened to notice it in the DAG viewer.

• Difficulty scaling to 100+ engineers and many surfaces
  As more teams and surfaces adopted UQL, the manual workflow became increasingly brittle. Each change demanded bespoke visual inspection, and there was no reusable, composable test abstraction that could encode “this is what our quality bid should look like for this stage/surface.”

The result was a system where UQL itself was powerful, but our ability to safely iterate on it lagged behind. We were shipping changes that influenced a space responsible for a significant share of Meta’s ads revenue, while relying on tooling that made some regressions almost inevitable.

What the UQL Testing Framework Is

The UQL Testing Framework (UTF) was built to bring unit‑test discipline and CI‑friendly automation to UQL quality components, without compromising on realism. Instead of testing an approximation, UTF evaluates components with the same engine that runs in production.

Core idea: reuse the production evaluation engine

The first design decision was to refactor the UQL compute library so that the evaluation logic could be cleanly extracted into a standalone module. UTF then reuses this evaluation module to run tests, which means:

• Tests are evaluated by the exact same engine that scores quality bids in serving.
• There is no separate “test harness” implementation to keep in sync.
• A test that passes under UTF is genuinely exercising the production semantics of the UQL component.

This avoids an entire class of “tests are green but prod is broken” failures that come from duplicated logic.

A unit-test-style API on top of the DSL

UTF exposes a pragmatic, unit‑test‑style API that sits directly on top of the UQL DSL:

• Engineers write tests in the same language and abstractions they use for authoring components.
• A single test class can generate tens of tests that span multiple serving stages and ad surfaces by parameterizing over inputs like:
  • Placement or surface (Feed, Reels, Stories, etc.).
  • Serving stage (early filtering vs. late ranking).
  • Treatment/experiment variants.

This pattern lets teams encode “the shape of our quality bid” once, and have UTF automatically fan it out into many concrete test cases.

Structure of the framework

At a high level, UTF is structured as:

• Evaluation module – the extracted UQL compute engine, reused as‑is in tests.
• Test specification layer – a DSL‑aligned API for expressing expectations about the shape and structure of a bid (e.g., “this subgraph must always include signal X,” “this weight must remain within a range,” “this path must exist for surface Y but never for surface Z”).
• Runner and CLI – a sleek command‑line interface that lets developers:
  • Discover tests associated with their components.
  • Filter by owner, surface, stage, or experiment.
  • Run suites locally or in bulk (for example, as part of pre‑submit workflows).
• CI and infra integration – UTF is wired into Meta’s CI/CD system (Sandcastle) and the broader Testing Infrastructure so that:
  • Tests run automatically at diff time and can gate changes.
  • Failures show up with rich metadata (coverage, flakiness, health signals).
  • Automated tasks are filed against authors or on‑call when tests are broken in production.

Together, these pieces make UTF feel like a first‑class testing framework rather than a one‑off script: engineers can write, list, filter, and run tests using familiar patterns, and the framework plugs directly into the company’s standard reliability surface.

Going back to the fictional ACR bid from earlier, a UTF test for it would look something like this — a single base class that encodes the expected behaviour, automatically fanned out to both serving stages:

class ACRBidTest(UqlUnitTest, abc.ABC):
    def pack(self) -> UqlUnitTestThriftStruct:
        self.pages(
            [
                UqlUnitTestPage.FB_FEED,
                UqlUnitTestPage.FB_STORY,
                UqlUnitTestPage.IG_REEL,
                UqlUnitTestPage.MESSENGER,
                UqlUnitTestPage.IG_FEED,
                UqlUnitTestPage.IG_EXPLORE,
                UqlUnitTestPage.IG_STORY,
                UqlUnitTestPage.IG_REEL,
                UqlUnitTestPage.THREADS,
            ]
        )
        self.override(
            (Signal.AgeGroupScalar, 0.23),
            (Signal.ACRScalar, 2.9),
            (Signal.offset, 0.03),
        )
        # ACRBid = AgeGroupScalar * ACRScalar + offset
        # 0.23*2.9+0.03 = 0.697
        self.expectEquals(Signal.ACRBid, 0.697)
        return super.pack()


class ACRBidAFTest(ACRBidTest):
    def pack(self) -> UqlUnitTestThriftStruct:
        self.name("ACRBidAFTest")
        self.stages([UqlUnitTestStage.AF])
        return super().pack()


class ACRBidARTest(ACRBidTest):
    def pack(self) -> UqlUnitTestThriftStruct:
        self.name("ACRBidARTest")
        self.stages([UqlUnitTestStage.AR])
        return super().pack()

Two concrete test classes, 20 lines of code, and UTF fans this out to 18 test cases (9 surfaces × 2 stages) — each run against the production evaluation engine.

What UTF Achieved

By giving UQL components a first‑class, realistic testing framework, UTF materially changed how the Ads Quality organization shipped changes.

1. From hours of visual diffing to fast, repeatable checks

Previously, validating a complex change could mean:

• Rendering the DAG in a visualizer for multiple stages and surfaces.
• Manually scanning for structural differences and hoping nothing subtle slipped through.

With UTF:

• Engineers encode structural invariants once in tests.
• Those invariants are re‑checked automatically on every relevant change, both locally and in CI.
• The feedback loop shrinks from hours of manual review to seconds or minutes of automated test runs.

This reduced the cognitive tax on both authors and reviewers and made it much cheaper to iterate on complex quality logic.

2. Real regression protection, not just “looks OK”

Because UTF is backed by the production evaluation engine, a UTF test failure is a strong signal that the shipped component would behave differently in serving. That unlocks:

• True regression tests for quality bids—protecting against accidental changes in structure, signal usage, or weighting.
• Confidence for refactors—teams can aggressively simplify or reorganize UQL graphs knowing that any behavior‑changing difference will be caught at diff time, not in production.

Given how strongly ad quality and user experience influence Meta’s auction outcomes, this kind of guardrail is critical.

3. Scale across 100+ engineers and many surfaces

UTF was deliberately designed to scale:

• A single well‑designed test class can generate a large suite of tests spanning multiple stages, surfaces, and experiment variants, which is important in a codebase touched by more than a hundred engineers.
• The CLI and metadata‑rich integration with Testing Infrastructure make it easy to:
  • See which parts of the UQL graph are covered by tests.
  • Track flakiness and test health over time.
  • Route breakages to the right owners automatically.

As coverage grew, UTF became the default way to validate quality bid changes, rather than an optional extra.

4. Quantitative impact

At the time it was authored and rolled out, UTF had a clear and measurable impact:

• It saved about 1.8 engineering years of effort across a roughly 100‑engineer area by replacing slow, manual DAG inspection with fast, programmatic tests.
• It helped prevent dozens of production incidents that previously would have slipped through manual review, by catching structural regressions before they reached serving.
• Because the Ads Quality space influences on the order of a fifth of Meta’s annual ads revenue, avoiding even a small number of quality regressions translates into millions of dollars of protected or incremental revenue, as better quality scores allow strong ads to win auctions more efficiently while preserving user experience.

In short, UTF turned UQL from “a powerful but fragile lever on a critical revenue stream” into “a powerful, testable lever with genuine regression guarantees.”