Mathematical Foundations of AI Alignment, Lecture 1

Overview

Aaron Roth introduces a research-focused course on mathematical foundations of AI alignment, organized around preferences, decision-useful predictions, and mechanisms. He previews tools including Arrow's impossibility theorem, maximal lotteries and Nash learning from human feedback, calibration, Aumann's agreement theorem, AI debate, market alignment, and delegated approval, while urging students to pursue ambitious semester-long research projects.

Key takeaways

Pairwise human ratings used in RLHF reveal preference order but not preference strength, so they cannot by themselves identify the utility magnitudes needed for social-welfare maximization.
Arrow's impossibility theorem shows that Pareto efficiency and independence of irrelevant alternatives can force preference aggregation into dictatorship, exposing a basic limitation of single-order reward models.
A maximal lottery can defeat or tie every competing completion distribution, and can optimize worst-case welfare distortion when utility functions are unknown but pairwise preferences are available.
Calibration is a distribution-free interface for decision-making: for any utility function, best-responding to a calibrated forecast is optimal among policies using that forecast, even when the forecast is not the true conditional distribution.
Conversation calibration can support accuracy-improving agreement between people and AI without requiring the implausible common-prior and full Bayesian reasoning assumptions of Aumann's theorem.
Mechanisms can offer weaker alternatives to perfect alignment: market alignment relies on provider competition, while delegated approval can provide baseline-relative safety when the user's utility lies in reviewers' nonnegative span.

Chapters

0:00 Aaron Roth Sets the Course Goal and Research-Frontier Focus

The course aims to bring students to, or beyond, the research frontier in the underdeveloped field of mathematical foundations of AI alignment.
Aaron Roth directs students to the course website for lecture notes and project ideas, and to Slack for announcements and course slides.
Because the field lacks a canonical body of knowledge, the course plan is provisional and students can help shape it.

6:41 Project-Based Grading, Student Lectures, and Research Expectations

There are no problem sets or exams; attendance and active engagement account for 20% of the grade.
A one-to-two-page mid-semester project proposal counts for 15%, a final class presentation for 15%, and the research-paper-style write-up makes up most of the remaining grade.
Aaron Roth recommends ambitious projects, ideally in groups of three or four, and says AI tools such as Codex and Claude Code are welcome when they improve the research rather than produce low-quality work.
Students can claim a lecture or lead a paper discussion, individually or in a group, and are encouraged to begin project planning early.

13:11 Three Course Themes: Preferences, Integration, and Mechanisms

The first theme asks what human preferences are, when utility functions can represent them, and what can be learned from elicited comparisons.
The second asks how to make predictions useful to downstream decision-makers with different goals, with or without knowing their preferences.
The third uses mechanism design and institutions to ask whether interactions among models or organizations can produce safe outcomes even when individual components are misaligned.

17:14 Pairwise Feedback, RLHF, and the Limits of Preference Data

Commercial systems such as ChatGPT, Gemini, and Claude often ask users to compare two responses, a simple form of preference elicitation.
A binary choice cannot reveal preference strength: preferring response A over B does not show whether A is slightly better or B is unacceptable.
In the standard RLHF pipeline, pairwise human comparisons train a reward model that assigns scores to prompt-completion pairs before those scores guide language-model training.
The reward model compresses the differing preferences of many labelers into a single ordering of possible completions.

24:01 Arrow's Impossibility Theorem Challenges Reward-Model Aggregation

Arrow's social-choice framework treats each labeler's preferences as an ordering and asks how to combine many orderings into one.
Pareto efficiency requires the aggregate to rank A above B when every person ranks A above B; independence of irrelevant alternatives requires changes to preferences about C not to change the aggregate ranking of A versus B.
Under Arrow's theorem, satisfying these conditions leads to dictatorship: the aggregate ordering follows one preselected person's preferences rather than meaningfully combining everyone's.
The result highlights a structural limitation for reward models that turn diverse human feedback into one fixed ranking.

34:49 Maximal Lotteries and Nash Learning from Human Feedback

Instead of choosing one completion, a language model can be treated as a distribution over completions, and two distributions can be compared by sampling one completion from each and asking a random user which they prefer.
A distribution defeats a rival when it is more likely to win this comparison; a maximal lottery defeats or ties every challenger, even when pairwise preferences contain cycles.
Maximal lotteries, also called von Neumann winners in some AI alignment work, can be formulated as equilibrium strategies in a zero-sum game.
Nash learning from human feedback targets this distribution directly, avoiding the intermediate step of fitting a single total-order reward model.

42:27 Von Neumann-Morgenstern Utility and Preference Axioms

A utility function represents not only whether someone prefers one outcome to another, but also the strength of that preference, including over lotteries.
The von Neumann-Morgenstern framework uses axioms including transitivity, continuity, and independence to characterize when lottery preferences admit a utility representation.
Transitivity rules out preferences such as P over Q and Q over R but R over P; continuity requires an appropriate lottery between preferred and less-preferred outcomes to create indifference.
Under the stated rationality assumptions, maximizing expected utility is a coherent model of an agent's choices.

48:46 Social Welfare, Utility Weights, and Missing Preference Strength

If social preferences satisfy the relevant utility axioms and the Pareto condition, the resulting social objective is a sum of individual utilities, possibly with different weights.
Pairwise comparisons over single completions reveal ordering but not utility magnitudes, so they cannot identify the utility functions required to calculate social welfare.
Roth frames the gap as a worst-case distortion problem: compare the welfare of a chosen lottery with the welfare achievable if the unknown utility functions were known.
A maximal lottery can optimize this worst-case objective using only pairwise preference information, connecting Nash learning from human feedback to welfare under uncertainty.

52:34 Decision-Relevant Predictions for Users with Different Utilities

Users consult an AI system to make decisions, such as choosing a college or selecting a drug for a patient, where the best action depends on an unknown state of the world.
Different users may have different utility functions, so a shared prediction system should support varied downstream decisions rather than assume one user's goals.
Knowing the true conditional distribution of outcomes would make the prediction sufficient for optimal decisions, but learning that distribution perfectly is often unrealistic.
The course therefore considers weaker guarantees that make predictions useful without requiring perfect statistical learning.

55:01 Calibration as a Decision-Making Interface

A forecast is calibrated when outcomes match its predictions on average conditional on the prediction itself; for example, events forecast at 20% should occur about 20% of the time in that forecast group.
For any downstream utility function, calibration is necessary and sufficient for best-responding to the forecast to be optimal among policies that map that forecast to actions.
This guarantee does not say the forecast is the true conditional distribution, or that users should ignore information they have beyond the forecast.
Roth presents calibration as an interface between prediction and decision-making that can serve many users without requiring the predictor to know their goals.

1:00:20 Calibration, Correlated Equilibrium, and Information Limits

In response to a question about correlated equilibrium, Roth clarifies that best response is relative to the information signal and the resulting beliefs about other players' actions.
A calibrated forecast supports optimal action choice only among policies based on that forecast, not policies using additional private or external information.
Stronger ideas such as multicalibration can condition guarantees on observable group or contextual information in addition to the forecast.
Precision calibration extends the discussion beyond binary events, including high-dimensional outcomes such as a hospital diagnosis chosen from many diagnostic codes.

1:04:05 Adversarial Calibration and Decision-Pipeline Guarantees

Calibration can be achieved in settings where outcomes shift arbitrarily over time, without assuming training and test data come from one fixed distribution.
No-regret learning and Blackwell approachability provide techniques for producing calibrated sequences of forecasts even when events are adversarial.
In a setting without an underlying probability distribution, calibration offers a useful consistency guarantee rather than a claim of conventional statistical correctness.
These properties matter when one prediction component is reused in multiple decision pipelines whose downstream objectives may be unknown.

1:09:52 Conversation Calibration and Challenges to Utility Theory

Conversation calibration offers a communication-efficient way to aggregate beliefs by exchanging beliefs rather than the observations that produced them.
Roth distinguishes Aumann's original fixed-point result about common knowledge from later work on convergence through an ongoing conversation.
Asked whether preference theory can misrepresent human behavior, Roth acknowledges behavioral economics and says utility maximization is a useful but contestable foundation.
He invites students to develop better foundations for reasoning about AI and human goals if consistent utility-maximizing preferences prove inadequate.

1:12:39 AI Debate Uses Competing Models for Scalable Oversight

AI debate pairs models acting like a prosecutor and defense attorney, with one supporting a claim and the other searching for flaws that a human adjudicator can assess.
The goal is for a limited human reviewer to evaluate bite-sized claims without independently solving or verifying a large problem, such as a lengthy proposal to address world hunger.
In the idealized framework, optimally acting models and a polynomial-time adjudicator can achieve sound and complete verification for problems in PSPACE.
Debate is presented as a scalable-oversight approach for claims that may be difficult for humans to check directly.

1:17:42 Market Alignment: How Competition Can Protect User Interests

An AI provider may align its model with its own goals, such as influencing purchases or voting, rather than with the user's preferences.
When users can choose among competing providers, organizations seeking influence must still make their systems attractive enough for users to select them.
Roth introduces market alignment as a weaker alternative to perfect alignment: the user's utility need not match any provider's utility, but can lie in the convex hull of provider utilities.
The proposed condition aims to guarantee user-beneficial outcomes across equilibria of the providers' strategic competition.

1:20:26 Delegated Approval for Long-Running AI Agents

Requiring human approval for every hard-to-reverse action can stall long-running agents, while unrestricted computer access creates risks such as deleting a hard drive.
Codex's approval-agent setup delegates permission checks to another model, raising the question of whether the reviewer genuinely protects the user's interests or simply moves the trust problem.
Roth models a driver agent proposing actions in a stateful Markov decision process, with a panel of reviewer agents applying an authorization rule before actions execute.
A safety guarantee relative to a baseline policy can be characterized when the user's utility lies in the reviewers' nonnegative span, a weaker condition than requiring every reviewer to be fully aligned.

1:24:45 Course Wrap-Up and Invitation to Develop the Field

Roth describes the listed topics as a partial selection of areas where formal mathematical work exists, not a complete map of AI alignment research.
He emphasizes that many important alignment topics are empirical and lack mathematical foundations, leaving substantial gaps for student projects.
Students are again encouraged to claim lectures, lead discussions, and pursue research that helps define this emerging field.
The class concludes around 86:00; the remaining transcript is incidental post-lecture conversation rather than additional course content.

Transcript excerpt

0:00 in you know particularly much depth like next class will be the first like actual like technical lecture where we like start you know giving definitions and proving things today is just you know like an overview and um so all right so so basic um parameters I guess I'm Aaron there's a course website um haven't found it maybe the best way to get to it is first to try to find my website you can find my website by typing my name into Google and then like scroll down on my website like list of things and this is what I'm teaching this semester and there will be like a link that goes here and that's where things like um lecture notes and eventually

0:49 project ideas you need to know for the class will be found on that website. Other thing that is on the website is a sign up link for the course slides. So the way we're gonna I'm not going to use like Blackboard or Canvas or anything. We're just we just have a Slack like everyone please sign up for it. If there's any announcements it'll um and oh and like so what is like what are we what are we all doing class? um it's to bring all of us including me up to and hopefully beyond the research frontier for what is I think um a new and important but very much underdeveloped area. So the goal is that at the end of this class all of you guys

1:38 will be sort of at or you know like beyond contributing to the research frontier for yeah the mathematical foundations of AI alignment whatever that you guys have an opportunity to help define what that is. Um and so like what does that mean? Well, um, there's

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Aaron Roth.