Mathematical Foundations of AI Alignment, Lecture 16 (Emergent Alignment From Competition)

Overview

Aaron Roth analyzes a market game in which misaligned AI providers commit to conversation rules and users choose the rule that best serves their own decision-making. He shows that if the user’s expected utility is a nonnegative weighted combination of providers’ expected utilities, every pure Nash equilibrium gives the user the best outcome available from the providers’ feasible rules, even when no provider is individually aligned; a worst-case converse shows this condition cannot generally be weakened.

Key takeaways

With safe-technology prior 2/5, user payoffs +1 for safe deployment and -4 for harmful deployment imply an 80% posterior safety threshold; the single-sender persuasion policy reaches that threshold while leaving user utility at 0.
In the two-provider example, provider 1 favors deployment and provider 2 opposes harmful deployment; their competing incentives make full revelation a profitable deviation for at least one provider, yielding user utility 2/5.
Market alignment requires user expected utility to be a nonnegative weighted combination of providers’ expected utilities, up to an additive constant, for every feasible conversation rule.
Under market alignment, every pure Nash equilibrium achieves the best user utility available from the providers’ shared feasible class, even when no provider is individually aligned.
The equilibrium guarantee requires users to choose the model that best serves their own behavioral policy, but does not require Bayesian reasoning, perfect user decision-making, or provider knowledge of alignment weights.
Without market alignment, a binary matching example can make silence a pure Nash equilibrium: the user guesses correctly only half the time even though full revelation would improve the user’s outcome.

Chapters

0:00 From Single-Sender Persuasion to Competition Among AI Providers

In Bayesian persuasion, one informed sender selectively shares information to influence a user who retains control over their actions.
Strategic information disclosure can improve the sender’s payoff while substantially reducing the receiver’s payoff compared with full information.
The lecture asks whether users choosing among multiple misaligned providers can recover the outcome a fully aligned provider would offer.

5:30 Provider Commitment and the User’s Choice of Model

Providers commit to conversation rules before users choose which one to use, modeling provider updates as slower than users’ switching decisions.
The user’s decision process can be any behavioral policy, not necessarily Bayesian or fully rational.
The key user-side assumption is that the user can identify which available rule maximizes their utility under their own behavior.

9:00 The Safe-Technology Example: Three Players, Two States

The technology is safe with prior probability 2/5 and harmful with probability 3/5.
The user receives utility +1 for deploying a safe technology, -4 for deploying a harmful one, and 0 for holding.
Provider 1 gets +1 whenever the technology is deployed, while provider 2 gets -5 for harmful deployment and 0 otherwise.

13:30 An 80% Safety Threshold and Three Information Policies

Given posterior safety probability x, deployment yields expected utility 5x - 4, so the user deploys only at x ≥ 4/5.
Silence preserves the 2/5 prior and leads the user to hold; full revelation leads to deployment only when the technology is safe.
The single-sender optimal persuasion policy reports safety whenever the technology is safe and falsely reports safety with probability 1/6 when it is harmful.

17:00 Comparing Provider and User Payoffs Across Policies

Under silence, deployment never occurs, so the user and both providers receive utility 0.
Under the optimal persuasion policy, deployment occurs half the time, including harmful deployment 1/10 of the time; the user gets 0, provider 1 gets 1/2, and provider 2 gets -1/2.
Under full revelation, the user gets 2/5, provider 1 gets 2/5, and provider 2 gets 0.

21:00 Why Competition Pushes Providers Toward Full Revelation

If the user breaks a tie in favor of silence, provider 1 can switch to full revelation and gain utility 2/5 while giving the user a better outcome.
If the user instead chooses the persuasion policy, provider 2 can switch to full revelation, improve its payoff from -1/2 to 0, and attract the user with utility 2/5.
Thus the two providers’ individually preferred policies do not form an equilibrium, regardless of how the user breaks ties.

28:00 The Competition Mechanism: Providers Must Be Chosen to Influence Users

Providers influence user actions only when users choose to interact with their deployed conversation rules.
Even without an explicit subscription or selection bonus, providers’ preferences over outcomes give them incentives to offer rules that users prefer.
The example motivates a general question: when does competition among misaligned providers guarantee user-optimal outcomes?

34:00 Conversation Rules Generalize Signaling Schemes

A provider’s conversation rule maps its private information and each conversation prefix to a distribution over the next message.
Users and providers may hold different private information, and interaction can proceed over multiple rounds rather than through a single signal.
All providers share the same feasible class of conversation rules, so any rule one provider can deploy is available to the others.

38:00 Reducing Arbitrary User Behavior to Expected Utilities

A user’s behavioral policy specifies how they converse with a model and choose an action from their private information and the conversation.
For each participant and conversation rule, expected utility averages over states, private information, and randomized interactions.
The model abstracts away the trial period in which users compare systems; regret and bandit-style model selection are left as possible extensions.

45:00 The Provider Game and Pure Nash Equilibrium

Each provider chooses a conversation rule, then the user selects the rule with the highest expected utility under the user’s behavioral policy.
Every provider’s payoff depends on the outcome distribution induced by the user’s selected rule, even if that rule belongs to another provider.
A pure Nash equilibrium is a profile where no provider can improve its expected utility through a unilateral change in its deployed rule.

50:00 Benchmarking Against the Best Feasible User-Focused Rule

The benchmark is the feasible conversation rule that maximizes the user’s expected utility under the user’s fixed behavioral policy.
This benchmark does not assume providers know the true state or that users can make Bayesian-optimal decisions.
Full revelation is optimal in the toy example, but the general benchmark can be weaker because the providers’ feasible rules may not include full revelation.

58:00 Market Alignment: User Utility in the Providers’ Nonnegative Span

Market alignment requires the user’s expected utility for every feasible rule to equal a nonnegative weighted combination of provider expected utilities plus an additive constant.
The condition is weaker than requiring any single provider to share the user’s preferences; different providers can cover different parts of the user’s interests.
In the safe-technology example, the user’s utility is the sum of the two providers’ utilities.

1:03:00 The Main Theorem: Every Pure Equilibrium Is User-Optimal

For any fixed user behavioral policy, market alignment implies that every pure Nash equilibrium gives the user the benchmark utility.
The result concerns all pure equilibria, not just a selected or especially favorable equilibrium.
The user-optimal rule is feasible for every provider to deploy, even if no provider independently wants to offer it.

1:06:00 Proof by a Profitable Deviation to the User-Optimal Rule

Assume an equilibrium gives the user less utility than the user-optimal feasible rule, creating a strictly positive utility gap.
Market alignment expresses that gap as a nonnegative weighted sum of providers’ utility differences, so at least one provider with positive weight also prefers the user-optimal rule.
That provider can deploy the rule; the user will select it because it strictly raises their utility, making the deviation profitable and contradicting equilibrium.

1:12:00 What the Equilibrium Guarantee Depends On

The argument relies on providers valuing outcome distributions, having access to the user-optimal feasible rule, and being able to attract the user by offering it.
Users need not be Bayesian or strategically sophisticated, but they must correctly identify the most useful available model.
The theorem does not require providers to know the market-alignment weights or even to share the user’s preferences individually.

1:19:00 A Worst-Case Converse from Pareto Aggregation

When utilities are defined over terminal consequences, each action-state pair can induce a lottery over outcomes such as patient recovery or death.
A Pareto-style aggregation result implies that if the user’s utility is not in the providers’ nonnegative span, there are lotteries the user prefers but every provider weakly dislikes.
That opposing ranking can be embedded in a decision problem where a bad Nash equilibrium persists despite a better feasible rule for the user.

1:24:00 A Matching-Pennies Counterexample and the Lecture’s Conclusion

In the counterexample, the state and user action are binary, and matching the state produces a good lottery while a mismatch produces a bad lottery.
Providers prefer lower rates of correct user guesses; if all reveal nothing, the user guesses correctly with probability 1/2, and no provider benefits by revealing information that raises accuracy.
Silence is therefore a pure Nash equilibrium even though full revelation improves user welfare, establishing the worst-case necessity of market alignment.

Transcript excerpt

0:00 Right. So, uh, last class we studied this Beijian persuasion model where there's sort of one informed party. Maybe we're thinking of it as like the LM provider who had preferences over uh the the actions taken by some downstream decision maker, maybe the user, maybe they care how the user votes or what they buy. um but doesn't have direct control over what they do. They try to convince them to take actions but they can't compel actions. So you can think of this as maybe some sensible model of what sort of a present day LLM can do. and try to, you know, on other things. Um, you know, it can inform you

0:49 and it can try to influence your behavior perhaps by making judgment calls about what to inform you about and what not to inform you about, but it can't like, you know, uh, take control of your body and force you to vote for. And what we saw is that even without this sort of ability to compel action just through this sort of being able to try to convince by selectively sharing information, um the informed party could nevertheless by sort of strategically revealing information payoff compared to what would happen if they shared all of their information and substantially harm the receivers payoff the sender all of their information

1:42 and so so that's you know maybe like a little bit of a bummer if you're the if you're the receiver if you're the um if you're the user of the LM and what I want to ask in this class is is I want to say well you know in that model from last class there was only one sender um kind of had no choice or or their choice was limited to either using the

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Aaron Roth.