Save this video free

Mathematical Foundations of AI Alignment, Lecture 20 (Program Equilibria)

Aaron Roth · 1:25:47 · Watch on YouTube

Part of University of Pennsylvania CIS 7000. See all 22 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Aaron Roth explains program equilibria as Nash equilibria in a higher-level game where players choose inspectable or simulatable programs rather than actions. In the Prisoner’s Dilemma, he shows how programs can support cooperation through exact code matching or recursive simulation with an escape probability ε ≤ 1/2, then connects joint simulation to discounted repeated games, where grim trigger sustains cooperation when the discount factor δ is at least 1/2.

Key takeaways

Chapters

0:00 Program Equilibria: Choosing Agents Instead of Actions
3:00 The Prisoner’s Dilemma and the Incentive to Defect
9:00 Moving the Strategy Space from Actions to Programs
12:00 Defining Valid Programs and Avoiding Nontermination
18:00 Program Equilibrium as Nash Equilibrium in a Meta-Game
20:00 Exact Source-Code Matching Can Sustain Cooperation
25:00 Why Source-Code Matching Is Brittle
29:00 Faithful Simulations and Recursive Agent Calls
33:00 The Naive Simulate-and-Copy Strategy Fails to Halt
37:00 Adding an ε Escape Probability to Recursive Simulation
42:00 Proving Cooperation Is a Program Equilibrium for ε ≤ 1/2
50:00 Joint Simulation Connects One-Shot Play to Repeated Games
58:00 How the Joint Simulator Builds Histories
1:05:00 Geometric Simulation Depth and the Unknown Real Round
1:10:00 Joint Simulation Is Payoff-Equivalent to Discounting
1:17:00 Grim Trigger Sustains Cooperation When δ ≥ 1/2
1:23:00 More Program Equilibria Mean More Cooperation and More Collusion

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 All right, I want to talk about a program equilibria. Just sort of an interesting idea for how you should think about strategic interactions in a world where principle might not be like directly acting um their own, but rather might be deploying say a software agent to act on their behalf. And the general idea is that in sort of traditional game theory when we study equilibrium, the premise is that what we have the ability to unilaterally control and hence the sort of basis on which we define the stability of equilibria is that we can sort of unilaterally

0:50 change the action that we take in the underlying game. And by unilateral, you know, what we mean is I can change my action, but like I can't change your action. And the, you know, what equilibrium requires is that play be stable in the sense that nobody should be able to improve their own payoff by changing their own action, holding everyone else's fixed. But you know in a world in which we have you know maybe like AI agents perhaps we're not like directly choosing the action we take rather the part of the game that we play is in deciding what agent to deploy. Like maybe what we're doing is we're deciding what piece of software to act on our behalf. And the thing about a piece of software is that

1:40 um it can be inspected. It can be simulated and this potentially opens up like a channel for cooperation even in one shy games that doesn't exist if if the premise is that you're playing in the game um yourselves just because you are much more opaque than the piece of software I can't simulate you and see what you would do in different situations. The idea of program

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Aaron Roth.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.