Mathematical Foundations of AI Alignment, Lecture 15 (Bayesian Persuasion)

Overview

Aaron Roth develops Bayesian persuasion as a model in which an informed sender commits to a signaling scheme to influence a rational receiver whose goals differ. In the safety example, the receiver deploys only when the posterior probability of safety reaches 4/5; with a prior safety probability of 2/5, a randomized scheme can induce deployment half the time, more than full disclosure’s 2/5. Roth formalizes the design problem through the splitting lemma and shows that the sender’s optimal value is the concave envelope of its payoff from a fixed posterior, yielding a general maximum deployment probability of P/Q.

Key takeaways

With deployment payoffs of +1 when safe and −4 when unsafe, the receiver deploys only when its posterior safety probability exceeds 4/5.
For a prior safety probability of 2/5, recommending deployment in every safe state and in 1/6 of unsafe states makes the deployment posterior exactly 4/5 and induces deployment half the time.
The splitting lemma says a signaling scheme is fully characterized, for persuasion outcomes, by its message probabilities and induced posteriors, whose weighted average must equal the prior.
The sender’s optimal value function is the concave envelope of its payoff from a fixed posterior, converting a signaling design problem into a geometric optimization problem.
For a binary safety decision with prior P below deployment threshold Q, the maximum deployment probability is P/Q, achievable with two posterior signals: 0 and Q.
A rational receiver cannot be pushed below the no-information utility baseline because ignoring the sender’s signal remains an available strategy.

Chapters

0:00 Bayesian Persuasion: An Informed Sender and an Acting Receiver

Aaron Roth contrasts decision-calibrated communication, where sender and receiver share objectives, with persuasion, where their utilities differ.
The sender observes relevant information but cannot act directly; the receiver chooses actions after interpreting the sender’s message.
The central design question is how a sender can influence a rational receiver without being able to dictate the receiver’s action.

5:00 The Receiver’s Deployment Threshold Is 4/5

The receiver gets +1 for deploying safe technology, −4 for deploying unsafe technology, and 0 for holding.
If the posterior safety probability is x, deployment has expected utility x − 4(1 − x), so the receiver deploys when x exceeds 4/5.
The sender knows the receiver’s utility and decision rule, allowing it to design messages around the threshold Q.

10:00 Prior Beliefs and the Limits of Full Disclosure

The shared prior assigns safety probability 2/5, below the receiver’s 4/5 deployment threshold, making persuasion necessary.
With no information, the receiver holds and the sender earns 0; with full disclosure, deployment occurs only when the technology is safe, earning the sender 2/5.
Always claiming the technology is safe does not persuade a Bayesian receiver if the sender’s policy is known, because an unvarying message carries no information.

14:00 Randomized Signaling Schemes and Commitment

The sender can commit to a mapping from each observed state to a probability distribution over messages, rather than choosing only full or zero disclosure.
After observing the message, the receiver conditions on both the shared prior and the known signaling policy to calculate a posterior.
Commitment is essential: the receiver must know the sender’s policy before the state is drawn to interpret the signal correctly.

17:30 Why Commitment Matters for LLM Communication

Roth treats commitment as a stylized but potentially useful approximation for LLMs, since model updates may occur less frequently than users change their behavior.
Users can learn how to interpret a deployed model’s outputs and switch between systems such as Claude and GPT between model updates.
Without commitment, a sender could change its message after seeing the state; Roth notes that the alternative model of cheap talk produces different results.

23:00 Using Randomization to Waste Fewer Safe-State Votes

Full disclosure produces posterior beliefs of 100% or 0%, even though the receiver needs only 80% confidence to deploy.
The sender can combine safe and unsafe states in a deployment message, while reserving an unambiguous non-deployment message for some unsafe cases.
The strategy resembles electoral gerrymandering: aim to cross the receiver’s decision threshold by the smallest possible margin instead of overshooting it.

27:00 The Optimal Toy Scheme: Lie on Unsafe States 1/6 of the Time

The sender recommends deployment whenever the technology is safe and recommends it with probability R when it is unsafe.
Setting the posterior safety probability after a deployment message to exactly 4/5 gives R = 1/6 under the 2/5 safety prior.
The resulting deployment probability is 1/2, higher than the 2/5 achieved by full disclosure; unsafe technology receives a deployment recommendation only one-sixth of the time.

34:00 Strategic Persuasion Can Reduce Receiver Utility to Its Baseline

The sender’s randomized policy makes the receiver indifferent between deploying and holding after a deployment message, so the receiver’s expected utility is 0.
The receiver cannot be made worse off than under no information, because ignoring the signal remains an available option.
The example shows how a misaligned sender can improve its own payoff by strategically withholding information while reducing the receiver’s gains.

36:00 Generalizing to Finite States, Actions, and Priors

The general model allows any finite set of states and actions, a common prior B0, and separate sender and receiver utility functions.
The sender observes the realized state and selects a message using a committed, possibly randomized signaling scheme.
States with zero prior probability can be discarded, since no signaling policy can make them occur.

41:00 Posteriors Summarize What a Signaling Scheme Does

For each message s, its probability α(s) is the prior-weighted sum of the probabilities of sending s in each state.
Bayes’ rule gives the receiver’s posterior for state θ as the prior probability of θ times the likelihood of s given θ, divided by α(s).
The sender’s and receiver’s outcomes depend on the induced posterior beliefs and their message probabilities, not on the signaling scheme’s other details.

45:00 The Splitting Lemma and Bayes-Plausible Posteriors

A collection of posterior beliefs and message probabilities is Bayes-plausible when their probability-weighted average equals the prior.
The splitting lemma states that every signaling scheme induces such a split, and every Bayes-plausible split can be implemented by a signaling scheme.
This equivalence replaces optimization over communication policies with optimization over weighted posterior distributions.

55:00 Turning Sender Optimization into a Posterior-Split Problem

For a posterior B, the receiver selects the action maximizing its expected utility under B, with ties assumed to break in favor of the sender.
The sender’s payoff V(B) is its expected utility when the receiver takes that best-response action.
The sender’s total expected payoff is the probability-weighted sum of V over induced posteriors, optimized subject to the Bayes-plausibility constraint.

1:00:00 Visualizing the Safety Example as a Step-Function Payoff

With two states, a posterior is summarized by one number: the probability that the technology is safe.
The sender’s payoff is a step function: 0 below the receiver’s 4/5 threshold and 1 at or above it.
The successful signaling scheme combines a posterior of 0, which yields no deployment, with a posterior of 4/5, which induces deployment.

1:05:00 Concave Envelopes Bound Every Feasible Signaling Scheme

A concave function lies above the chord between any two points; Jensen’s inequality extends this property to weighted averages.
The concave envelope of a payoff function is the smallest concave function that upper-bounds it.
For any concave upper bound on the sender’s fixed-posterior payoff, concavity and Bayes plausibility ensure it also upper-bounds the sender’s expected payoff under every signaling scheme.

1:11:00 Why the Sender’s Optimal Value Is the Concave Envelope

Roth defines the sender’s optimal value at prior B as the best expected payoff over all Bayes-plausible posterior splits of B.
This optimal-value function is concave: combining a split of B and a split of C with weights t and 1 − t gives a valid split of tB + (1 − t)C.
Because the optimal value is concave and upper-bounds the fixed-posterior payoff, it is exactly that payoff’s smallest concave upper bound.

1:20:00 The Safety Example’s Concave Bound Certifies Optimality

For the step payoff that jumps from 0 to 1 at safety probability Q, the concave envelope gives an upper bound on achievable deployment probability.
With prior safety probability P = 2/5 and threshold Q = 4/5, the bound is P/Q = 1/2, matching the randomized scheme.
Two messages suffice: one induces posterior safety 0, and the other induces exactly Q, the minimum belief that triggers deployment.

1:25:00 Bayesian Persuasion’s Takeaway and the Role of Competition

The sender’s power comes from shaping rational posterior beliefs, not from forcing the receiver to take an action.
Strategic information control can benefit a misaligned sender, but a Bayesian receiver can always ignore the signal and retain the no-information baseline.
Roth previews a later model with competing information providers, where users choose whom to listen to and competition may improve outcomes even when every sender remains misaligned.

Transcript excerpt

0:00 So, some weeks ago we sort of talked about um calibration and and things like decision calibration as um something like decision alignment in a communication setting. I am I'm not empowered directly to take actions. You're the one empowered to take actions, but I can talk to you. I can give you um information that's hopefully relevant to your sort of future relevant state. I'm like informative information. So, so okay, if I know your utility function and I want to help you optimize your utility function, um I can make forecasts that are, for

0:48 example, decision calibrated relative to that utility function, maybe also relative to various benchmark policies you could have used, and if you then trust the forecasts and act accordingly, uh you will inherit nice downstream guarantees. What I want to talk about um in this lecture is sort of a a similar communication setting. So, think about maybe a model that is not empowered to take actions. All it can do is try to inform you, and you get to take actions. But I want to think about a a similar model in which party communicating with you does not necessarily have the same objectives as you do. So, you know, it knows that you're the one

1:36 empowered to take actions, but because it has different objective function than you, it's trying to convince you to take actions that would ultimately be beneficial for, say, its utility function. And that'll sort of be this this sort of classical model of Bayesian persuasion. So, just to have maybe like a toy scenario um in our mind, you know, we'll see this this more general framework. But, let's suppose uh there's some, you know, new technology and we're trying to decide whether we want to deploy it or

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Aaron Roth.