Mathematical Foundations of AI Alignment, Lecture 11.1 (Aumann's Agreement Theorem)

Overview

Aaron Roth introduces Aumann’s Agreement Theorem: two Bayesian agents with a common prior cannot retain different probabilities for an event once their beliefs are common knowledge, even if they never exchange their private observations. He derives the result using information partitions and a common-knowledge graph, then motivates a more practical extension based on approximate agreement, conversation calibration, and Blackwell approachability—an extension left for the next lecture to prove.

Key takeaways

With a common prior, two Bayesian agents who commonly know their respective probabilities for the same event must assign it the same probability, even if they never share their private observations.
Common knowledge can be represented as the connected component of the true state in a graph that links states indistinguishable to either agent.
In a finite state space, alternating truthful posterior announcements eventually stabilize the publicly possible states and make the agents’ current beliefs common knowledge, though the number of rounds may be as large as the state space.
Marginal calibration alone does not ensure that agents respond to each other; requiring calibration conditional on the other agent’s previous report detects failures that aggregate calibration misses.
Conditioning on every conversation history is impractical because the number of histories grows exponentially with conversation length; conversation calibration instead conditions on the current report and the immediately preceding report.
The lecture motivates using Blackwell approachability to enforce a small, tractable set of calibration constraints, but does not yet prove the resulting fast-agreement theorem.

Chapters

0:00 From Model-to-Human Calibration to Two-Way Information Sharing

Earlier lectures treated information as flowing from a model to a human decision-maker through calibration and omniprediction.
Aaron Roth motivates a two-way setting: a doctor has direct access to a patient, while an LLM may have absorbed information from medical journals.
The goal is to combine complementary information through predictions rather than transmitting every observation.

6:00 Bayesian State Spaces and Agents’ Information Partitions

The classical setup has a finite set of world states and a common prior shared by two agents, such as an AI and a human.
Each agent observes an information partition: the cell containing the true state, rather than the state itself.
Different partitions allow agents with the same prior to form different posteriors.

10:50 Computing Event Beliefs from Private Observations

An event E is a subset of world states; each agent’s posterior is the prior probability of E conditioned on the agent’s information cell.
The posterior is computed as the probability of the event intersected with the information cell, divided by the probability of that cell.
An information cell may contain both event and non-event states, leaving the agent uncertain about whether E occurred.

13:20 Two-Bit Example: Predictions Reveal Complementary Evidence

The state consists of an AI bit and a human bit, with all four states equally likely; E occurs only when both bits are 1.
If the AI observes 0, its posterior probability of E is 0; if it observes 1, its posterior is 1/2. The human’s posteriors are symmetric.
Exchanging predictions reveals information: a report of 0 identifies a zero bit, while successive reports of 1/2 can let both agents infer the state is (1,1).

16:00 Knowledge and the Infinite Recursion of Common Knowledge

An agent knows an event at state ω when every state in the agent’s information cell containing ω lies in that event.
Both agents knowing E is weaker than common knowledge: common knowledge includes knowing that the other knows, and every further iteration.
The recursive definition captures why one announcement does not automatically make current beliefs mutually known.

18:20 Common Knowledge as Connectivity in an Indistinguishability Graph

Construct a graph whose vertices are world states, connecting states whenever either agent cannot distinguish between them.
Starting from the true state, alternating through each agent’s indistinguishable states traces the possible-belief chain behind “I know that you know.”
The connected component containing the true state is the common-knowledge set; an event is common knowledge if the whole component lies inside it.

24:00 Information Cells and the Structure of Common-Knowledge Sets

In the two-bit example, blue edges connect states sharing the AI’s bit and red edges connect states sharing the human’s bit.
The graph’s connected component can include the entire state space, in which case little beyond prior knowledge is common knowledge.
Any common-knowledge set is a union of whole information cells for each agent, a property used in the theorem’s proof.

29:00 Aumann’s Agreement Theorem: Common Knowledge Forces Equal Beliefs

Assume two Bayesian agents share a common prior but have different information partitions.
If it is common knowledge that one agent assigns event E probability α and the other assigns it probability β, then α must equal β.
The theorem’s conclusion is about agreement on a fixed event’s probability, not recovery of all private information.

31:30 The Agreement Theorem’s Proof by Conditional Probability

Let D be the common-knowledge connected component; it can be decomposed into whole information cells for either agent.
Apply the law of total probability to P(E | D), first summing over the AI’s cells, where every posterior is α; this gives P(E | D) = α.
Repeating the calculation over the human’s cells gives P(E | D) = β, so the shared conditional probability forces α = β.

37:00 Why the Classical Result Is Hard to Use Directly

Aumann’s theorem assumes perfect Bayesian agents and a common prior over the world, both agents’ observations, and the outcome.
Obtaining such a prior and computing the required Bayesian updates can be epistemically and computationally implausible.
The theorem alone is static: it says what common knowledge implies, but not how agents reach common knowledge.

45:00 Alternating Belief Reports Eventually Create Common Knowledge

Agents take turns announcing their current posterior beliefs, updating after each announcement without directly sharing observations.
Let Ωⱼ be the set of states consistent with the first j public announcements; each report can only shrink this set.
Because the state space is finite, the sequence must stabilize; at stabilization, both agents’ current beliefs are constant across the remaining states and therefore common knowledge.

59:00 Why Finite Bayesian Agreement Is Not Yet a Practical Guarantee

The finite-state argument bounds the number of rounds by the number of states, which may be enormous.
The argument still requires perfect Bayesian updating and a correct common prior.
Roth proposes relaxing exact agreement to ε-agreement, where reports within ε count as agreement.

1:04:00 An Online Protocol for Repeated Prediction Conversations

The proposed setting has repeated days, such as successive patients arriving for diagnosis, with a new event probability to discuss each day.
The AI and human can make arbitrary opening predictions, then alternate reports; the conversation halts when adjacent reports differ by less than ε, subject to a round cap.
After each day, the binary outcome Yₜ is revealed, allowing accuracy to be evaluated without assuming a fixed distribution or common prior.

1:07:00 Why Ordinary Calibration Does Not Ensure Useful Agreement

A perfectly informed human can report 0% or 100% while an uninformative AI reports 50% every day; both can be marginally calibrated without engaging with each other.
Conditioning the AI’s forecast on the human’s previous 0% report exposes the failure: a 50% forecast is not calibrated on that subset.
The example motivates conversation calibration, which requires each report to be unbiased conditional on its own value and the other agent’s immediately preceding report.

1:10:00 Conversation Calibration, Orthogonality, and the Next Theorem

A Bayesian posterior is calibrated conditional on everything the agent knows, but enforcing calibration over every conversation history would require data for exponentially many histories.
Conversation calibration uses only the agent’s own report and the most recent report from the other agent, reducing the number of constraints.
Roth frames calibration as residual orthogonality to selected functions and says Blackwell approachability can enforce a tractable subset; the fast-agreement result is reserved for the next lecture.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Aaron Roth.