Mathematical Foundations of AI Alignment, Lecture 10 (Groupwise Welfare Distortion)

Overview

Aaron Roth develops a contextual policy for preference alignment that protects overlapping user groups from welfare failures hidden by population averages. Using Blackwell approachability, antisymmetric pairwise-comparison margins, and one comparison per training round, the method learns lotteries that are approximately maximal for every specified group and yields group-level welfare guarantees without observing utilities.

Key takeaways

A pooled maximal lottery can be arbitrarily poor for a subgroup: with two completions, opposing group preferences can make every population lottery maximal even though always choosing A loses three-quarters of the time in one group.
For K overlapping groups, splitting users into every intersection can require up to 2^K separate categories; a contextual policy instead handles intersections through each user's observed group memberships.
Groupwise maximality compares the policy's group-average performance against every fixed challenger lottery, while allowing the policy to personalize its lottery for each user in that group.
Blackwell approachability turns one violation coordinate per group–completion pair into a learning objective; positive cumulative violations become weights that focus the policy on persistently losing comparisons.
Antisymmetry is the key algebraic property: the constructed weighted lottery plays against itself with exactly zero expected margin, satisfying the approachability condition.
With Bradley–Terry preferences, pairwise-margin guarantees translate into group welfare guarantees against utility-optimal lotteries, using pairwise labels without observing the underlying utilities.

Chapters

0:00 Why Population-Average Alignment Can Fail for User Groups

The task is to choose a lottery over a finite set of LLM completions that serves users with different preferences.
A policy that performs well for the pooled population can still produce poor outcomes for particular groups.
The goal is a contextual policy that uses attributes such as age or location to choose outcomes and protect each specified group.

3:31 Two-Completion Example of Hidden Groupwise Failure

With completions A and B, two groups can have opposing preferences even when the population prefers A over B with probability one-half.
In that pooled case, every lottery over A and B is population-maximal, so the aggregate guarantee hides group-specific losses.
Always choosing A loses to B three-quarters of the time in group two; group one requires choosing A, while group two requires choosing B.

5:22 Overlapping Groups Require One Contextual Policy

Separate maximal lotteries work when groups are disjoint, but users commonly belong to several groups, such as age and gender categories.
Turning K overlapping groups into disjoint intersections can create as many as 2^K patterns, greatly increasing data requirements.
The desired policy maps each user's context to a lottery while evaluating each group separately, even when groups overlap.

8:34 Pairwise Margin Matrices and Groupwise Maximality

For context X, the margin entry M_AB(X) is 2 times the probability the user prefers A to B, minus 1.
Pairwise margins are antisymmetric: M_AB equals -M_BA, and a lottery played against itself has expected margin zero.
A policy is epsilon groupwise maximal when, for every group and fixed completion, its group-average margin is at least -epsilon.

13:01 Comparing Contextual Policies with Fixed Group Lotteries

A contextual policy may give different users in one group different lotteries; its group performance is averaged over those users.
The challenger is a single fixed lottery for that group, and the guarantee must hold for every completion and therefore every mixture of completions.
This comparison lets overlapping groups be assessed independently without requiring a separate policy for every intersection.

19:41 Bradley–Terry Preferences Connect Margins to Welfare

Under the Bradley–Terry model, a user's probability of preferring A to B depends on the utility difference scaled by inverse temperature beta.
The deployed policy observes context and learns from pairwise comparisons, not users' underlying utility vectors.
A bound translating pairwise margins into utility differences converts groupwise maximality into a comparison with the utility-optimal lottery for each group.

27:14 Blackwell Approachability as the Learning Framework

The approachability setup treats contextual policy choices as actions and comparison outcomes as environment actions that generate violation vectors.
The target is a closed convex set that the running average of those vectors should approach.
Blackwell's condition requires a response whose expected violation points toward the target for every environment distribution.

31:12 Encoding Every Group–Completion Failure as a Coordinate

The violation vector has one coordinate for each of K groups and each of the m possible challenger completions.
The target set is the negative orthant: a nonpositive coordinate means the policy is not beaten by that challenger in that group.
Projection onto this target is simple: retain negative coordinates and set positive coordinates to zero.

34:00 Positive Cumulative Violations Set the Policy Weights

The algorithm tracks cumulative violations for every group–completion pair and uses their positive parts as nonnegative weights.
Large weights identify group–challenger pairs where the policy has repeatedly lost, so those comparisons receive greater emphasis.
For each arriving context, the active groups' weights are combined by challenger completion to define the next lottery.

38:44 Antisymmetry Makes the Blackwell Response Work

The policy assigns probability to each completion in proportion to the combined weight of that completion across active groups.
If all active weights sum to zero, the algorithm uses the uniform lottery as a fallback.
Against any antisymmetric margin matrix, the weighted response plays against itself, giving exactly zero expected margin and satisfying Blackwell's half-space condition.

49:09 Estimating Preferences from One Pairwise Label per Round

Each training round samples a user, observes their context, and presents a pair of completions; the user reports which one they prefer.
The estimator places nonzero entries only on the sampled pair, assigning opposite signs to the two matrix directions.
Scaling the sparse estimate appropriately makes it unbiased for the true antisymmetric pairwise-margin matrix without constructing a full matrix.

54:17 Per-Round Training and the Deployed Mixture Policy

At round t, the algorithm converts the previous cumulative violation into weights, builds a contextual lottery, gathers one comparison, and updates the violation vector.
The observed comparison produces a sparse antisymmetric estimate, preserving the exact zero inner-product property needed by Blackwell approachability.
Deployment samples a training round and uses that round's contextual policy for a fresh user's context.

56:06 Approachability Error Shrinks with Training Rounds

The analysis assumes each relevant group has positive population mass, with the smallest group probability bounded below by gamma.
The per-round violation vector has bounded norm, with the bound depending on the number of groups and completions.
Blackwell's guarantee makes average positive violations decrease at a rate proportional to 1 divided by the square root of the number of rounds.

1:04:32 Transfer Training Violations to Fresh Users

For each round, the sparse comparison estimate is unbiased, so its expected violation equals the population violation of that round's policy.
The difference between observed and population violations has conditional expectation zero; cross-round error terms therefore vanish in expectation.
A concentration-style argument bounds the average population violation, with the smallest group mass gamma accounting for a factor of 1/gamma.

1:12:05 Group-Level Welfare Guarantees and Lecture Takeaways

Combining the learned groupwise maximality bound with the Bradley–Terry utility conversion gives each group a welfare guarantee against its utility-optimal lottery, up to learning error.
The result uses pairwise preference labels alone and does not require observing utilities or building group-specific preference matrices.
Roth concludes that contextual policies prevent pooled welfare guarantees from concealing group failures, including for overlapping groups.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Aaron Roth.