Mathematical Foundations of AI Alignment, Lecture 3 (The VnM axioms and Harsanyi's Theorem.)
Watch on YouTube →
Overview
Aaron Roth derives the von Neumann–Morgenstern (VNM) expected-utility representation: an agent’s preferences over lotteries can be represented by a utility function exactly when they satisfy completeness, transitivity, continuity, and independence. He then presents Harsanyi’s social aggregation result: when individual and social preferences satisfy the relevant VNM conditions and weak Pareto, social utility must be a nonnegative weighted sum of individual utilities.
Key takeaways
- An agent’s preferences over lotteries admit an expected-utility representation if and only if they satisfy completeness, transitivity, continuity, and independence.
- VNM utility is unique only up to positive affine transformations; normalizing the best outcome to 1 and the worst to 0 fixes the scale.
- Independence is the key risk-related assumption: it requires that adding the same lottery branch to two options preserve their ranking.
- Harsanyi’s theorem says that VNM-consistent social preferences satisfying weak Pareto must be representable as a nonnegative weighted sum of individual utilities.
- Lottery comparisons reveal preference intensity that a ranking such as A above B above C cannot supply, enabling cardinal social-welfare rules that ordinal aggregation cannot express.
Chapters
- Pairwise preferences such as choosing A over B show order but not how strongly an agent prefers A.
- Aaron Roth motivates utility models for evaluating stochastic language-model outputs and aggregating the goals of multiple people.
- The lecture connects individual expected utility to social welfare, often represented as a sum of agent utilities.
- A question about ratings from 1 to 10 assumes agents can report cardinal values and evaluate randomness using those values.
- The VNM framework instead asks agents to compare lotteries directly, providing information about preference strength.
- These comparisons can support aggregation methods that ordinal rankings alone cannot justify.
- For a fixed prompt, a stochastic language model defines a probability distribution over possible completions.
- A lottery is a vector in the probability simplex: each completion receives a nonnegative probability and all probabilities sum to one.
- A deterministic completion is represented by a unit vector assigning probability one to that completion.
- Knowing that an agent ranks A above B above C does not reveal whether B is better than a lottery mixing A and C.
- If A is far better than B while B and C are similar, a lottery over A and C may beat B; different preference strengths can reverse that comparison.
- The VNM axioms make it possible to infer cardinal utility from lottery preferences under specified assumptions.
- Completeness requires an agent to compare every pair of lotteries, allowing either weak preference direction or indifference.
- Indifference means weak preference in both directions; strict preference means one direction holds but the reverse does not.
- Transitivity rules out an individual cycle such as preferring P to Q, Q to R, and R to P.
- Continuity says that when P is preferred to Q and Q to R, some mixture of P and R must be indifferent to Q.
- Independence says that mixing P or Q with the same third lottery R preserves their original preference ordering.
- Independence allows an identical branch of two lotteries to be canceled when comparing the branches that differ.
- The VNM theorem states that a preference relation satisfying all four axioms can be represented by maximizing expected utility.
- For a lottery P, expected utility is the inner product of the completion-utility vector and P’s probability vector.
- The utility representation is unique up to a positive affine transformation, so adding a constant or multiplying by a positive scale preserves preferences.
- Independence is the substantive assumption: it requires an agent’s comparison to remain unchanged when both options receive the same lottery branch.
- An agent who cares about variance or other distributional features may violate independence and therefore may not be an expected-utility maximizer.
- Roth distinguishes individual lottery independence from Arrow’s independence of irrelevant alternatives, which constrains an aggregation rule rather than one person’s preferences.
- Completeness and transitivity follow from the completeness and transitivity of numerical comparisons between expected utilities.
- Continuity follows by choosing a mixture probability that makes the expected utility of a P–R lottery equal to Q’s.
- Independence follows from linearity of expectation: the common R component cancels when the two mixed lotteries are compared.
- The harder direction starts with preferences satisfying the VNM axioms and constructs a utility function that explains them.
- Independence permits replacing a lottery component S with an indifferent lottery S′ without changing the overall preference.
- This substitution rule is used repeatedly to reduce arbitrary lotteries to simpler mixtures of a best and a worst outcome.
- If an agent is indifferent among every outcome, the constant-zero utility function represents those preferences.
- Otherwise, choose a best completion J+ and a worst completion J−, with J+ strictly preferred to J−.
- Define L(α) as a lottery giving J+ with probability α and J− with probability 1−α; independence implies that larger α is strictly preferred.
- Normalize the best outcome’s utility to 1 and the worst outcome’s utility to 0.
- Continuity guarantees that every completion J is indifferent to some lottery L(αJ) between J+ and J−.
- Strict ordering of L(α) across probabilities makes αJ unique, defining the utility assigned to completion J.
- For each completion J, replace it with its indifferent best–worst lottery L(αJ), using the independence-based substitution rule.
- A lottery P becomes indifferent to a best–worst lottery whose probability on J+ is the sum of P(J)αJ over all completions.
- That sum is the inner product of P and the constructed utility vector, so comparing lotteries is equivalent to comparing expected utilities.
- When an agent’s lottery preferences satisfy the four VNM axioms, expected utility is an exact representation, not merely a convenient approximation.
- A positive affine transformation changes the utility scale but not preferences; fixing best utility at 1 and worst at 0 selects a normalization.
- The axioms can be investigated empirically, but adopting them commits the model to a particular treatment of risk.
- Ordinal rankings alone provide no measure of how much one person prefers an outcome over another, limiting the available aggregation rules.
- With cardinal utility functions, a social designer can define welfare using a sum or a weighted sum of individual utilities.
- Weak Pareto requires the aggregate to prefer P to Q whenever every individual strictly prefers P to Q.
- Harsanyi’s theorem characterizes social utility when individual and aggregate preferences satisfy VNM-style expected-utility conditions and weak Pareto.
- The social utility must be a nonnegative weighted sum of individual utility functions.
- The result permits different social weights, including zero weights, but rules out other aggregation forms if unanimous improvements must not be reversed.
- Represent the difference between lotteries P and Q by a zero-sum vector Z, since both probability vectors sum to one.
- A linear program searches for a Z that every individual values positively but the proposed social utility values negatively, which would violate weak Pareto.
- If no violation exists, dual variables yield nonnegative coefficients expressing social utility as a linear combination of individual utilities.
- Ordinal preference collection can trigger Arrow-style limits because it omits the magnitude of individual preferences.
- VNM axioms supply a principled route from lottery comparisons to cardinal utility; weak Pareto then constrains social aggregation to weighted utility sums.
- Roth previews analysis of what standard reinforcement learning from human feedback optimizes with ordinal comparisons, including the Borda rule, versus expected-utility objectives.
Transcript excerpt
0:02 Last class we sort of talked about the real sort of foundations of preferences. Preference we talked about sort of ordinal preferences. Preferences where people um specify which of various options they prefer. We started with that because that's kind of the um model in which the the standard RHF pipeline collects preferences. It asks, do you prefer A or B? But we we kind of noted there were some there was some important information about what people might want that is missing from preferences like that. Like you don't that doesn't specify if you prefer um A over B, whether you prefer A over B by a whole lot or only by a little bit. And these
0:51 kinds of things start to really matter when you're trying to aggregate preferences over a bunch of people or if you're even trying to figure out how people will sort of evaluate distributions over outcomes, which are ultimately the things that language models produce. And so for a lot of the class, we're going to be um imagining that agents of various sorts, either people or AI agents, have utility functions that they're trying to maximize. And you know, we might set as our goals to maximize social welfare, the sum [clears throat] of age utilities. And so um like what I want to do today is dive into sort of another basic foundational topic just to understand when we are justified in modeling agents either human agents or
1:40 AI agents in this classic economic way like when when can I think about someone's goals as being expressible as maximizing a utility function and when is it reasonable to set my goals as maybe sort of the the social designer to maximize social welfare, the sum of a utility cost. Okay. So, so today is
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Aaron Roth.