Mathematical Foundations of AI Alignment, Lecture 6 (Welfare Distortion)

Overview

Aaron Roth studies how well social welfare can be optimized when the available data consists only of pairwise preference comparisons, not cardinal utilities. Under a shared-parameter Bradley-Terry model, he proves that the maximal lottery computed from the comparison matrix achieves worst-case distortion γ(β) = β / [4(σ(β) - 1/2)], and constructs indistinguishable instances showing no method can guarantee a smaller distortion.

Key takeaways

Pairwise preference majorities do not determine social welfare: Bradley-Terry aggregation observes E[σ(βΔu)], while welfare depends on E[Δu], and nonlinearity can even reverse the sign of the comparison.
For users sharing a Bradley-Terry parameter β, a maximum lottery guarantees welfare at least 1/γ(β) of optimum, where γ(β) = β / [4(σ(β) - 1/2)].
The maximum lottery is minimax optimal under the model: a family of m indistinguishable instances with all comparison probabilities equal to 1/2 forces every algorithm to incur distortion at least γ(β) on some instance.
The distortion bound approaches 1 as β approaches zero from above and grows like β/2 for large β; exactly β = 0 is a separate no-signal case.
The result depends on individual users following Bradley-Terry choices with a common β; it does not require the population's aggregate preferences themselves to follow Bradley-Terry.

Chapters

0:00 From Pairwise Preferences to Social Welfare

Earlier results address ordinal preferences from pairwise comparisons, including the maximum-lottery approach and aggregation impossibilities.
Cardinal utility functions encode preference strength, making their sum a possible social-welfare objective.
The central question is how closely pairwise data alone can guide welfare optimization when utilities are not observed.

6:00 Maximum Lotteries Are Strong Ordinal Choices, Not Welfare Guarantees

A maximum lottery is a Nash equilibrium strategy in a zero-sum comparison game: no challenger lottery has positive expected margin against it.
Its guarantee concerns how many users prefer one option over another, not how strongly they prefer it.
A narrow majority with weak preferences can outweigh a smaller group of strong supporters in welfare, so ordinal victory alone does not imply higher social welfare.

15:30 Defining Population Welfare for Anonymous Service

Each completion receives a utility between 0 and 1 for each user; the population is modeled as a distribution over utility functions.
The welfare of a completion is its expected utility for a randomly selected user, and a lottery's welfare is the probability-weighted average of its completions' welfare.
If the utility distribution were known, linearity of expectation means a welfare-optimal lottery could place all its weight on a single highest-welfare completion.
The anonymous-policy assumption gives everyone the same policy; personalization based on user information is deferred for later study.

21:00 The Shared-Parameter Bradley-Terry Observation Model

For a user with utilities u(A) and u(B), the probability of choosing A is σ(β[u(A) - u(B)]), where σ is the logistic sigmoid.
The parameter β controls choice sharpness: small β makes choices noisier, while large β approaches deterministic selection of the higher-utility option.
The model applies to individual users, not necessarily to the population mixture, which can still produce cyclic aggregate preferences.
The welfare guarantee depends on assuming users share the same β and follow this comparison model.

25:00 Worst-Case Distortion Measures Welfare Loss from Missing Utilities

The observable comparison matrix gives the population probability that A is preferred to B, but does not directly reveal cardinal utilities.
A policy's distortion is optimal population welfare divided by the welfare it achieves, with the worst case taken over utility distributions consistent with the data.
Distortion 1 would mean attaining the welfare optimum; the goal is to minimize the largest possible ratio despite incomplete identification.

31:00 Why Bradley-Terry Mixtures Do Not Identify Welfare Differences

With a linear choice model, averaging pairwise responses would expose average utility differences and identify the welfare-maximizing completion.
Bradley-Terry instead averages σ(βΔu), so the expectation is outside a nonlinear function of each user's utility gap.
Inverting the aggregate sigmoid cannot recover the average utility gap, and even the sign of aggregate preference can differ from the sign of the welfare difference.

35:00 Bounding the Logistic Margin with Linear Functions

Roth defines h(z) = σ(βz) - 1/2, the individual comparison margin associated with utility gap z.
Concavity on the relevant nonnegative range gives a tight tangent upper bound with slope β/4 and a chord lower bound with slope σ(β) - 1/2.
These linear bounds make it possible to take expectations and connect comparison margins to average utilities.

40:00 The Wβ Weight Explains What Comparison Data Emphasizes

The normalized margin can be written as a utility gap multiplied by a weight depending on the gap's absolute size.
Because the Bradley-Terry sigmoid is nonlinear, large and small utility gaps receive different effective weights in aggregate comparisons.
The data therefore contains a weighted average of utility differences, while welfare depends on the unweighted average.

45:00 Turning User-Level Choice Margins into Population Moments

For each user, the A-versus-B margin is h(u(A) - u(B)); antisymmetry handles negative as well as positive utility gaps.
Averaging over users gives the observed entry of the comparison matrix as an expectation of these nonlinear margins.
The desired welfare gap is instead E[u(A) - u(B)], motivating bounds that relate the observed margin to welfare.

49:00 Apply Linear Margin Bounds to Any Pair of Completions

For utility values x and y, the margin h(x - y) is bounded above by β times the upper-bound slope on x minus the lower-bound slope on y.
A matching lower inequality follows by considering the reversed comparison and using h(-z) = -h(z).
These inequalities work for every pair of utility values, so they remain valid after averaging over users and completion lotteries.

55:00 Aggregate the Bounds into Lottery Welfare Comparisons

Averaging over users converts expected utilities of completions into population welfare; averaging over a lottery converts them into that lottery's welfare.
The expected comparison margin between lotteries can therefore be bounded using weighted differences between their welfare values.
The bounds supply the bridge needed to translate a maximum lottery's ordinal no-challenger condition into a welfare guarantee.

59:00 Maximum Lotteries Achieve the γ(β) Distortion Bound

Define γ(β) as the ratio of the best linear upper-bound slope to the lower-bound slope: β / [4(σ(β) - 1/2)].
A maximum lottery has nonnegative expected comparison margin against every challenger, including the population's welfare-optimal lottery.
Combining that condition with the upper linear bound yields welfare(maximum lottery) ≥ welfare(optimum) / γ(β).
The calculation does not require knowing the welfare-optimal lottery; it is used only to prove the guarantee.

1:02:00 How β Controls the Tightness of the Welfare Guarantee

As β approaches zero from above, the sigmoid is nearly linear around the comparison range and γ(β) approaches 1.
For large β, γ(β) grows approximately as β/2, so the worst-case welfare guarantee deteriorates as choices become sharper.
At β = 0, comparisons contain no preference signal; the limiting guarantee as β approaches zero does not mean the exactly zero-signal case identifies welfare.
The interpretation of β as inverse noise scale matters: lower β means noisier choices and a better bound under this model.

1:08:00 Why More Inference Cannot Beat the Bound in the Worst Case

A more sophisticated algorithm might try to infer utility parameters or optimize over feasible utility distributions, but the theorem's lower bound rules out a better worst-case guarantee.
The construction hides which completion is welfare-optimal while making all candidate instances generate the same observed comparison matrix.
Since an algorithm receives identical data on these instances, it must choose the same lottery even when different choices would be needed for optimal welfare.

1:12:00 Constructing Indistinguishable Bradley-Terry Populations

The lower bound uses m completions, with one hidden special completion A and m - 1 alternatives B₁ through Bₘ₋₁.
Type-A users, present with probability α, value A at 1 and every alternative at 0; type-B users value A at 0 and each alternative at ε.
The welfare of A is α, while the welfare of any alternative is (1 - α)ε.
Choose α to balance type-A and type-B comparison margins, making every pairwise preference probability exactly 1/2.

1:18:00 The Lower Bound Matches the Maximum-Lottery Guarantee

Because all m choices of the hidden special completion yield the same all-ties comparison matrix, any algorithm must output the same lottery on every instance.
At least one completion receives lottery probability at most 1/m; the construction makes that least-weight completion the welfare-optimal special option.
As ε approaches zero, the optimal-to-alternative welfare ratio approaches β / [4(σ(β) - 1/2)], exactly γ(β).
Thus no mapping from the comparison matrix to a lottery can improve on the maximum lottery's worst-case distortion under the shared-β model.

1:24:00 Conclusion: Ordinal Data Has a Sharp Welfare Limit

The maximum lottery is computable directly from pairwise comparisons and achieves distortion γ(β) under the Bradley-Terry assumption.
The matching lower bound shows that this guarantee is minimax optimal, not merely a loose analysis of the maximum lottery.
The result is conditional on the behavioral model, especially its shared β; alternative models or heterogeneous β values remain open extensions.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Aaron Roth.