Mathematical Foundations of AI Alignment, Lecture 17 (Safe Delegated Approval to Misaligned Agents)
Watch on YouTube →
Overview
Aaron Roth shows how a panel of individually misaligned AI reviewers can safely approve actions proposed by an agent when the principal’s utility is a non-negative weighted sum of the reviewers’ utilities in every winning coalition. The lecture gives an if-and-only-if safety condition for monotone approval rules, explains why unanimity can preserve safety even with strategic reviewers, and proves that binary votes cannot achieve both safety and completeness unless at least one reviewer is perfectly aligned with the principal.
Key takeaways
- A monotone approval rule is sound exactly when every winning coalition can represent the principal’s utility as a non-negative weighted sum of its members’ utilities.
- Unanimous approval provides safety when the full reviewer panel covers the principal, and gives each reviewer veto power that makes sincere voting the only Nash equilibrium.
- More permissive threshold rules need stronger, deletion-robust coverage: a Q-vote rule requires every Q-member subset to cover the principal.
- Binary approval votes lose magnitude information, so proposals that are beneficial and harmful to the principal can generate identical vote patterns.
- With binary votes, perfect soundness and completeness are simultaneously possible for every proposal only if at least one reviewer is perfectly aligned with the principal.
Chapters
- Last lecture’s market alignment result relied on LLM providers offering advice while users retained control over real-world actions.
- Agentic systems can directly use tools, such as sending email or changing files, so user attention competition no longer ensures aligned outcomes.
- Aaron Roth asks whether diverse misaligned components can be arranged to provide safety guarantees for autonomous agents.
- Requiring human permission for every tool call is conservative, but can interrupt long-running work, such as installing NumPy during an experiment.
- Unrestricted permission lets an agent act without review, which can lead to destructive behavior such as deleting files.
- Codex and Claude Code style reviewer or auto-review modes delegate approvals to another model, raising the question of whether that reviewer can be trusted.
- A principal delegates a long-running task to a driver agent but wants proposed actions reviewed before execution.
- A single reviewer does not solve the alignment problem if its goals may differ from the principal’s.
- The proposed alternative is a panel whose members may be misaligned in different ways, with safety arising from their combined judgments.
- The principal values a software change by its performance gain D1 plus its privacy gain D2.
- Reviewer 1 values only performance, D1, while reviewer 2 values only privacy, D2.
- Requiring both reviewers to approve certifies that D1 and D2 are each non-negative, so their sum cannot harm the principal.
- In the performance-and-privacy example, unanimous approval accepts only proposals in the quadrant where both D1 and D2 are non-negative.
- A beneficial proposal such as (2, -1) is rejected, because its vote pattern matches that of the harmful proposal (1, -2).
- Approving when either reviewer says yes is unsafe, since a performance gain can outweigh a much larger privacy loss.
- A proposal is modeled as a change from a baseline distribution over actions, and the principal evaluates its expected utility relative to that baseline.
- The utility change is the inner product of the principal’s utility vector and the proposal vector.
- The vector model covers both continuous attributes, such as performance and privacy, and ordinary expected utility over discrete actions.
- The principal is participant 0, and the panel contains N reviewers with utility functions U1 through UN.
- A reviewer sincerely approves proposal D exactly when their utility for D is non-negative.
- An authorization rule is specified by its winning coalitions: any proposal whose approving reviewers contain a winning coalition is executed.
- A monotone rule remains approving when more reviewers vote yes, avoiding the counterintuitive result that extra approvals reverse execution.
- The framework excludes a useless rule that never executes anything by requiring that unanimous approval be sufficient.
- Soundness means every executed proposal gives the principal non-negative utility, though a sound rule may reject beneficial proposals.
- A reviewer coalition covers the principal when the principal’s utility is a non-negative linear combination of the utilities of reviewers in that coalition.
- The condition applies to each coalition separately, and different covering coalitions may use different weights.
- In the performance-and-privacy example, the two reviewers jointly cover the principal because their utilities sum to the principal’s utility.
- A monotone approval rule is sound if and only if every winning coalition covers the principal.
- For the forward direction, approving reviewers each have non-negative utility, so any non-negative weighted sum of their utilities is also non-negative for the principal.
- The reverse direction uses monotonicity: if a winning coalition fails coverage, a harmful proposal approved by that coalition remains approved even if additional reviewers also vote yes.
- Harsanyi’s aggregation theorem connects agreement on non-negative outcomes to a social utility represented as a non-negative weighted sum of individual utilities.
- A linear program searches for a proposal all coalition members approve while minimizing the principal’s utility.
- If that program has no harmful proposal, its dual variables provide the non-negative weights certifying coalitional coverage.
- If a winning coalition does not cover the principal, linear programming can construct a proposal that coalition approves but that harms the principal.
- The reverse implication depends on monotonicity, while the direct proof from coverage to safety does not.
- For a monotone rule, it is enough to check minimal winning coalitions, because adding reviewers permits zero weights and preserves coverage.
- Unanimous approval has one winning coalition, the full panel, and is safe exactly when that panel covers the principal.
- A threshold rule that executes with Q approving reviewers is safe exactly when every subset of Q reviewers covers the principal.
- More permissive thresholds require more robust coverage, because safety must survive deletion of up to N minus Q reviewers.
- Under any monotone authorization rule, a reviewer’s vote can either have no effect or change the outcome from rejection to execution.
- Because execution gives the reviewer the proposal’s utility while rejection gives the baseline utility, sincere voting is weakly dominant.
- Sincere voting is therefore a Nash equilibrium, though other equilibria can exist when reviewers lack the power to change the outcome.
- With three aligned reviewers using majority vote, all three approving a proposal worth -1 to everyone is still a Nash equilibrium because no single reviewer is pivotal.
- Under unanimity, every reviewer can veto, so an equilibrium with execution probability ρ requires each reviewer’s expected utility ρ times their proposal utility to be non-negative.
- If the full panel covers the principal, those non-negative reviewer utilities imply that every harmful proposal is rejected with certainty; sincere voting is the only Nash equilibrium.
- Completeness requires every proposal that benefits the principal to be executed; together with soundness, it would reproduce the principal’s decisions exactly.
- In the performance-and-privacy example, proposals (2, -1) and (1, -2) produce the same votes but have opposite effects on the principal.
- With only binary votes, no approval rule can distinguish such proposals, so safety and completeness cannot generally both be achieved.
- A binary-vote rule can be both sound and complete for all proposals if and only if at least one reviewer is perfectly aligned with the principal.
- Numeric utility reports could support exact reconstruction when the principal’s utility lies in the reviewers’ span and the aggregation weights are known.
- Numeric reporting requires stronger modeling and known weights, and may undermine truthful voting; binary panels instead provide safety through coalitional coverage.
Transcript excerpt
0:01 So, last class, um, we sort of studied how we might get aligned behavior even if sort of all of the ML providers in the system are misaligned. LLM providers that wanted different things from the user, uh, and yet because they were sort of reliant on the user to actually take action, so [clears throat] all of all the LLM providers could do was provide information and advice, they were sort of in the position of needing to compete for the user's attention. And that mechanism of competition we showed under this weaker market level condition that we called, uh, market alignment was enough to provide aligned
0:50 outcomes in the equilibrium of this game without needing like actually aligned agents. Now, what was driving that competition was really the fact that the AI providers could not directly act in the world themselves. Like only the user could act. This was modeling sort of LLMs that can talk to you Of course, the situation is quite different if we have sort of agentic AI systems that can actually take actions themselves. They don't necessarily need to convince you of anything. They can just do the thing. And so, what I want to think about today is whether there's some way to build agentic AI systems. You know, we we get to design these things.
1:38 Um, that can get safety guarantees from the same kind of emergent alignment phenomena that we saw, like, um, yeah, we want we want to like build a system that might be composed of misaligned components, maybe because we don't know how to construct aligned components, but we want to arrange them such that we get somehow the same kind of emergent alignment we saw last time, but now in in agentic systems. So, um if you've used I mean if you of course
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Aaron Roth.