Mathematical Foundations of AI Alignment, Lecture 19. (Game Theory for Pacing the Frontier)

Overview

Aaron Roth models competition between two AI providers as a continuous-time game: each firm's capabilities rise until it stops, profits depend exponentially on the capability gap, and catastrophe arrives at hazard rate 2 while either firm keeps developing. Simultaneous stopping yields payoff 1 per firm, compared with 1/3 from racing indefinitely; with exponential monitoring rate ω, stopping first pays (1 + ω)/(4 + ω), so the race equilibrium disappears when ω > 1/2. The model's central lesson is that credible, sufficiently fast monitoring can make safe coordination incentive-compatible, though subgame-perfect equilibria need not require immediate stopping or eliminate all catastrophe risk.

Key takeaways

In the two-provider model, simultaneous stopping gives each firm payoff 1, while matched firms that keep developing face catastrophe hazard 2 and earn only 1/3 in expected discounted profit.
With monitoring delay distributed exponentially at rate ω, stopping first yields payoff (1 + ω)/(4 + ω); it beats the race payoff of 1/3 precisely when ω > 1/2.
Fast monitoring makes unilateral stopping sustainable because a rival can verify the stop and has an incentive to stop too, preventing the first firm from remaining behind while danger continues.
A race-to-catastrophe Nash equilibrium can survive even under instantaneous monitoring if firms threaten to continue after a rival stops, but subgame-perfect equilibrium excludes that noncredible behavior.
For ω > 1/2, every subgame-perfect equilibrium has finite planned stopping times with probability 1, but the result does not guarantee immediate stopping or eliminate all catastrophe risk.
For AI safety coordination, credible verification that training has ended can matter more than a public pledge; reducing monitoring delays helps align market incentives without requiring instantaneous oversight.

Chapters

0:00 AI Frontier Pacing as a Provider Coordination Problem

Aaron Roth shifts from learning-system mechanics to the incentives of AI providers and whether market competition blocks joint safety goals.
The motivating concern is that developing superintelligent AI too quickly could create existential risk, even when companies and employees recognize the danger.
The central question is whether a provider can rationally stop development first without being disadvantaged by a competitor that continues.

5:00 Why a Mutually Beneficial Pause Is Hard to Coordinate

With only two providers, such as Anthropic and OpenAI, both might prefer to stop training, sell existing systems, and avoid further training costs and catastrophe risk.
Explicitly agreeing not to improve products could raise antitrust concerns, so the relevant issue is unilateral incentives rather than a formal pact.
Roth identifies monitoring speed as crucial: a firm needs to know how quickly its competitor can verify that it has actually stopped.

10:00 A Toy Market Model of Relative AI Capability

The model assumes two firms whose revenues depend on their models' relative capabilities, rather than capability growth alone.
If both firms improve equally, their capability gap stays zero and the model gives them equal profits; a firm with a capability lead gains a strong advantage.
Roth cautions that the market assumptions are stylized, including the idea that equally capable models compete prices down while a capability lead yields market power.

15:00 Exponential Profits, Discounting, and Catastrophe Risk

Firm i's instantaneous profit is modeled as exp(c_i(t) - c_j(t)), the exponential of its capability advantage over its rival.
Future profits are discounted exponentially with discount rate 1, keeping lifetime returns finite in the continuous-time model.
As long as either firm develops AI, the world faces a constant catastrophe hazard; both firms must stop for that risk to disappear.

20:00 Capability Growth and the Catastrophe Hazard

Each firm's capability begins at zero and grows linearly at rate 1 until its chosen stopping time, then remains fixed: c_i(t) = min(t, t_i).
Stopping means ending model development, not closing the company; firms can continue serving and selling their existing systems.
The model sets catastrophe hazard λ = 2, so the probability of surviving s time units while development continues is exp(-2s).

25:00 Why Simultaneous Stopping Pays More Than Racing

When both firms develop at the same rate, their capability gap remains zero and each earns instantaneous profit 1.
A firm that stops before its rival loses relative capability, while the firm that continues gains an exponential profit advantage.
With both stopping immediately, each firm's discounted lifetime payoff is 1; with both racing indefinitely under hazard 2, each payoff is 1/3.

30:00 Exponential Monitoring Delay and the Parameter ω

After a firm stops, its rival receives a conclusive report after an exponential delay with rate ω.
The probability that the rival has not yet received the report after s time units is exp(-ωs), and the mean delay is 1/ω.
A larger ω means faster verification, which reduces the time a first-moving firm remains behind while catastrophe risk continues.

35:00 The Race Strategy and Its First-Mover Risk

Roth considers a strategy of continuing development while no report says the rival has stopped, then stopping immediately upon receiving that report.
If both firms use this rule and neither stops first, both continue developing until catastrophe, earning expected payoff 1/3 each.
The key deviation to test is whether either firm should stop unilaterally before receiving evidence that its rival has stopped.

40:00 Why Stopping After a Verified Rival Stop Is Optimal

At the moment a report arrives, let the continuing firm's current instantaneous profit be B; stopping locks in B while eliminating future catastrophe exposure.
Continuing for h more time units raises capability and profit, but the model's catastrophe survival probability falls during that period.
Under the chosen exponential functions, waiting to stop produces a pointwise lower expected-profit integrand than stopping immediately, for every B and every h.

45:00 First-Mover Payoff and the ω = 1/2 Threshold

Stopping first creates two phases: before the report, the firm falls behind while catastrophe risk remains; after the report, both firms stop and the capability gap freezes.
Integrating the discounted profits gives a first-stopper payoff of (1 + ω)/(4 + ω), compared with 1/3 from continuing the race.
The first-stopper payoff exceeds 1/3 exactly when ω > 1/2; at slower monitoring, stopping first is not a profitable deviation from the race.

50:00 Why Delayed or Randomized Stopping Does Not Improve the Deviation

Stopping at a future deterministic time adds no advantage when no new information arrives before the rival stops.
The exponential model's memoryless property makes the payoff difference at a later stopping time the same as the difference from stopping immediately, aside from a positive discount and survival multiplier.
Randomizing over stopping times cannot help when each possible deterministic stopping time has no profitable deviation.

55:00 Multiple Equilibria and the Implausible Never-Stop Threat

The model permits both a safe equilibrium, where firms stop together for payoff 1, and a race equilibrium, where they continue until catastrophe for payoff 1/3.
Even with very fast monitoring, a Nash equilibrium can prescribe that each firm continues forever even after learning the other has stopped.
That threat is not credible: once a firm learns its rival has stopped, stopping itself is a best response.

1:00:00 Subgame-Perfect Equilibrium Removes Noncredible Threats

Subgame-perfect equilibrium requires strategies to remain best responses at every information state, including after a firm receives verified evidence of a rival's stop.
This refinement rules out the strategy of continuing forever after the rival has stopped, because the continuing firm would then be acting against its own interests.
For monitoring rate ω > 1/2, every subgame-perfect equilibrium has finite planned stopping times with probability 1.

1:05:00 Proof Setup: Firms' Planned Stopping Times

Let each firm's planned stopping time be the time it would stop if it received no report that its rival had stopped; strategies may randomize over these times.
The proof aims to rule out positive probability Q that neither firm plans to stop first, the event that could sustain a race until certain catastrophe.
For finite planned stopping times, the probability of stopping later than t approaches zero as t grows, because the tail of any probability distribution vanishes.

1:10:00 A Late-Stopping Deviation Rules Out Certain Catastrophe

Suppose Q is positive; a firm can follow its equilibrium strategy until a chosen finite time t and then stop even if it has received no report.
If both firms were planning never to stop, stopping at t gains the previously established positive payoff difference between stopping first and racing.
The possible loss when the rival had instead planned to stop at a finite later time is bounded, and its probability shrinks as t grows, so a sufficiently large finite t makes the deviation profitable.

1:17:30 Policy Lesson: Make Safety Commitments Verifiable

The proof implies that with ω > 1/2, subgame-perfect equilibrium rules out a probability-one race to catastrophe, though it does not guarantee immediate stopping or zero catastrophe risk.
Roth emphasizes credible monitoring over press releases: a rival must be able to verify that training has actually stopped, not merely hear a public claim.
Market or policy design can improve coordination by shortening verification delays, even without instantaneous monitoring or explicit agreements between competitors.

Transcript excerpt

0:00 Okay. So, uh what I want to talk about today, um is maybe a little bit of a departure from sort of the technical details of of you know, learning systems that we've spent much of the class on. And [clears throat] I want to think about the incentives of AI providers themselves and whether they can um cooperate on joint safety goals or whether sort of the uh competitive pressures of the market stop that coordination even when it would be mutually beneficial. So um you know like the maybe like the underlying

0:46 picture you should have in your mind are these calls to pace the frontier. So like the premise is that maybe by developing you know super intelligent AI as fast as we can we're kind of h hurtling towards civilizational disaster like you know we don't know exactly when it's going to happen but you know we get these models that are sort of uh smarter than we know how to control and at some point um it's kind of like the world ends and that is bad not just for us but also for the profits of the AI companies. And it might even be that sort of um AI companies and their employees kind of

1:37 agree on this picture. Like we've we've seen for example high-profile resignations of of sort of highly placed employees at profic and open AI kind of making exactly this case. they sort of say, oh, you know, what are we doing? Like we are working towards the end of the world. Like we need to we need to um so like maybe like the people in charge of these companies are are kind of like even on board with this premise. I think you could make the case that this is maybe true of

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Aaron Roth.