Could an international agreement protect us from dangerous AI? (with Malo Bourgon)

Overview

Malo Bourgon argues that building superintelligence before we can understand or reliably govern it creates an unacceptable risk of catastrophic loss of control, even though safely aligned AI could enable extraordinary human flourishing. He outlines a path toward a US-China-led international agreement that limits frontier training through compute tracking, data-center monitoring, chip verification, and incremental rules on dangerous capabilities, while drawing lessons from nuclear arms control and acknowledging major verification and political-will challenges.

Key takeaways

Bourgon argues that a 25% to 75% chance of catastrophic AI failure is unacceptable when the potential consequence is the loss of human civilization, even if the exact probability is uncertain.
The proposed Machine Intelligence Research Institute agreement would prohibit AI training runs above 10^24 FLOPs, monitor runs from 10^22 to 10^24 FLOPs, and register clusters of 16 or more connected H100-equivalent chips.
AI safety requires more than teaching a system to follow instructions: researchers also need to understand its internal reasoning and learned objectives, prevent unsafe competitors from building systems, and decide who governs concentrated AI power.
International AI controls could begin with mutually beneficial, verifiable steps such as restricting the open release of powerful cyber-capable models, then advance toward chip tracking, data-center inspections, and training notifications.
AI compute is comparatively concentrated in chip manufacturing and specialized equipment, but open-source models and algorithmic efficiency could erode hardware-based controls, making any training pause a limited window for solving deeper governance problems.

Chapters

0:00 What AI Companies Mean by Superintelligence

Bourgon describes superintelligence as an AI system better at virtually every cognitive task than any human, possibly even all humans combined.
He compares the capability gap not just to Einstein versus an average person, but potentially to humans versus ants.
The stated promise is to use radically greater intelligence to solve problems and enable abundance, if the system can be safely steered.

3:00 Superintelligence Could Concentrate Power at Unprecedented Scale

A company, individual, or state that controls superintelligence could gain decisive advantages through new technologies, strategy, persuasion, and manipulation.
Bourgon compares technological asymmetry to encounters in which firearms gave colonizers overwhelming advantages over people with spears.
AI could remove human labor and information-processing bottlenecks by operating as a parallelized workforce of highly capable agents.

9:00 Why AI Capability May Generalize Beyond Human IQ

Human abilities such as mathematical reasoning and marketing often differ, but Bourgon argues AI systems could be trained across many capabilities using general-purpose learning and broad training data.
Modern systems can perform tasks ranging from image generation and poetry to legal work, illustrating a move beyond narrow, single-task AI.
Large language models may learn abstractions that support broad generalization rather than merely memorizing training text.

16:00 Language Models May Build Internal Models of Emotion

Bourgon discusses research finding that parts of language models appear to represent emotion-related features and change in expected ways with emotional context.
He compares this to models learning a rudimentary Python interpreter by predicting program outputs from examples.
The findings suggest that generating emotionally appropriate text may involve internal abstractions, though researchers do not yet know how closely these resemble human emotions.

20:00 Why AI Leaders’ Catastrophic-Risk Warnings Deserve Attention

Bourgon rejects the claim that company leaders discuss extinction risk only to generate hype, noting that some expressed similar concerns before becoming CEOs.
He cites Sam Altman’s warning that failure could mean lights out for everyone and Dario Amodei’s reported estimate of a 25% chance of a very bad outcome, potentially including human extinction.
He points to leaders including Demis Hassabis and Elon Musk saying they would prefer slower development but feel constrained by competition.

26:00 Why Bourgon Wants a Cautious Path, Not an AI Ban

Bourgon says he wants the benefits of safely aligned superintelligence and does not argue that current chatbots, self-driving cars, or AlphaFold-like systems are existential threats.
His concern is the race toward broadly capable, superintelligent systems before safety and control problems are solved.
He says his estimates of catastrophic risk vary, sometimes from about 25% to 75%, but even a double-digit risk to human civilization is unacceptable.

30:00 Interpretability and Alignment Are Only the First Safety Challenges

Researchers lack a sufficiently deep understanding of how advanced AI systems reason or which goals and incentives training causes them to internalize.
Behavioral testing can mask underlying problems: a system appearing safe does not establish that its core risks have been addressed.
Even successful alignment would leave questions about which values to instill, how to prevent other actors from building unsafe systems, and who should govern concentrated AI power.

35:00 Bourgon’s Proposal: A US-China Agreement to Limit Frontier Training

Bourgon argues that company-level restraint or a unilateral US pause would fail if competitors or other countries continued, making international coordination essential.
His proposed starting point is a US-China agreement to restrict training of increasingly capable general AI, with allies joining a broader coalition.
The agreement outlined by the Machine Intelligence Research Institute would prohibit training above 10^24 FLOPs and monitor runs between 10^22 and 10^24 FLOPs.

42:00 Why AI Governance Must Be Led by States, Not Just Companies

Bourgon says governments, rather than private firms, must make durable international agreements and enforce them within their jurisdictions.
AI companies may be legally constrained from coordinating on safety rules because such cooperation can be treated as anticompetitive.
Private commitments could help, but Bourgon argues they cannot substitute for public authority when the technology could have effects comparable to nuclear weapons.

47:00 Building Political Will Through the Nuclear Arms-Control Example

Bourgon says many leaders have not yet grappled with AI loss-of-control arguments, but policymakers can respond when the risks and current development plans are explained clearly.
He compares today’s fatalism to nuclear fears in the 1950s, when many people expected nuclear proliferation to end in catastrophe.
The 1983 television film The Day After reportedly affected Ronald Reagan, helping him recognize nuclear dangers and engage with Mikhail Gorbachev; Bourgon sees this as evidence that leaders can change course.

56:00 Social Taboos Can Reinforce Formal Technology Controls

Bourgon acknowledges that surveillance-heavy restrictions could create an undesirable international police state, but argues governance need not depend on catching every violator.
He points to the scientific taboo against human germline engineering, where researchers who violate norms can face ostracism and loss of funding.
Compute restrictions and formal rules could be reinforced by broad social agreement that developing superintelligence prematurely is unacceptable.

59:00 Start US-China AI Talks With Verifiable Dual-Use Rules

Bourgon recommends establishing regular government-to-government US-China discussions and building toward tougher agreements through smaller steps.
A near-term candidate is limiting open release of models with dangerous dual-use capabilities, such as advanced cyber vulnerability discovery and exploitation.
He argues that rules on model access, safeguards, chip accounting, and advance notice of training runs could create trust and procedures for later agreements.

1:05:00 Verify AI Restrictions With Chip and Data-Center Monitoring

A basic agreement could use physical inspectors and data-center monitoring; more precise controls could verify workloads directly through AI hardware.
Bourgon says current chips already support some location and workload verification, but tamper resistance is the main obstacle to trusting those safeguards across borders.
One proposed alternative is a tamper-resistant enclosure around a chip that destroys it if someone attempts to interfere with the protections.

1:12:00 Borrow Treaty Tools From Nuclear and Chemical Weapons Agreements

Bourgon says the proposed AI agreement draws on the Nuclear Non-Proliferation Treaty’s neutral inspection mechanisms and the START treaty’s challenge inspections.
It also adapts the Chemical Weapons Convention’s system of declared, monitored facilities to AI data centers.
He cautions that AI differs from nuclear weapons because its catastrophic risks are less visibly demonstrated and it offers major economic benefits that encourage development.

1:19:00 AI Compute Is a Bottleneck, but Efficiency and Open Models Complicate Controls

Bourgon argues that large AI data centers may be easier to monitor than uranium enrichment because advanced chips and chipmaking equipment depend on a concentrated supply chain, including TSMC and a Dutch equipment maker.
Open-source models and small clusters such as four Mac Studios make control of already-released systems much harder than control of frontier training.
Algorithmic efficiency could let smaller hardware train more capable models, so a compute agreement would be a temporary bridge, perhaps buying 10 to 20 years rather than stopping superintelligence forever.

1:22:00 Consciousness, the Orthogonality Thesis, and Values for AI

Bourgon says an AI could pose existential risks without being conscious; whether AI can be conscious is a separate unresolved question with its own moral implications.
The orthogonality thesis makes the narrow claim that high intelligence can be paired with many different goals, not that every AI will inevitably pursue arbitrary or harmful goals.
For a flourishing future, Bourgon discusses coherent extrapolated volition, where AI helps determine what people and society would want after reflection and protects that outcome from interference.

1:26:00 Why Bourgon Calls for Action Rather Than Resignation

Bourgon says many people treat a dangerous AI race as inevitable, but coordinated public engagement can change political incentives.
He invokes past nuclear diplomacy to argue that millions of individual efforts to inform policymakers and develop workable proposals can make leadership agreements possible.
He reframes cautious advocates as optimists who believe society can govern AI wisely and preserve the possibility of a flourishing future.

Transcript excerpt

0:00 There's this whole notion [music] of Pdoom and people have different numbers or something. Some people are like, "Oh, my PDoom is 10%." Some people are like, "It's [music] 90% plus." Your risks seem deep enough, fundamental technical challenges seem hard enough and the probability seems high enough that it seems unacceptable. All of those are [music] like unacceptably high for many things. I think humans should be allowed to make decisions and take risk into their own hands. When we're talking about all of [music] human civilization, I think anything approaching double-digit probabilities [music] is just unacceptable. The general space seems like so fraught and so reckless that the answers going poorly are just unacceptably [music] high. >> Malo, welcome to the Clear Thinking podcast. >> Hey man, thanks for having me. >> What is it that the biggest AI companies claim they're trying to build? I mean, I

0:49 think most of them are on record saying that they're trying to build super intelligence. Um, you know, we could talk about what that means, but uh, some of them don't use that word specifically. Many of them do, but I think that's the explicit goal. And, you know, the thing they're doing now, chatbots are just a stepping stone to that kind of ultimate goal. >> Yeah. So, what do you mean by super intelligence? >> So, it's one of these things where a bunch of people, you know, want a crisp definition. Um, and I don't think there's kind of like one. >> Well, what do they mean? What are they what are they trying to build, right, when they >> Well, yeah. I I mean, I think the thing that I hear when they talk about it is that there and it's similar to a definition that I have, which is

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Clearer Thinking with Spencer Greenberg.