Mathematical Foundations of AI Alignment, Lecture 21 (AI Auditing and Forecast Testing)
Overview
Aaron Roth formalizes auditing a forecasting model as distinguishing a forecaster that reports the true conditional probabilities from one that knows only the test. For any fixed finite deadline, if a test accepts every correctly informed forecaster with probability at least 1 minus α across all outcome distributions, the minimax theorem implies a test-aware randomized forecaster can pass with that probability on every outcome sequence. An unbounded-horizon tail test can evade that specific impossibility, but does not guarantee useful power on outcomes drawn from the real process; assumptions such as product-distributed outcomes can also change what is testable.
Key takeaways
- Calibration is not evidence of mechanistic understanding: a constant 50% forecast is calibrated on the alternating sequence 0, 1, 0, 1.
- For a fixed finite horizon, any test that accepts every correctly informed forecaster with probability at least 1 minus α can be passed with the same guarantee by a test-specific randomized forecaster on every outcome sequence.
- The finite-horizon impossibility proof uses minimax after reducing forecasting rules to at most 2^(2^T) pass/fail patterns over the 2^T binary sequences.
- An unbounded-horizon test can reject when the first event falls outside the forecaster's own α-tail, controlling false negatives while still lacking guaranteed power on the real data-generating process.
- Knowing that outcomes follow a product distribution can make forecasts invariant across days and enable tests for changing forecasts or persistent bias; the restriction matters because product distributions are not convex under mixtures.
- Computational limits are distinct from statistical impossibility: if true conditional probabilities are efficiently computable, an efficient algorithm can pass any test, while some test-specific strategies may otherwise be computationally intractable.
Chapters
- The audit goal is to distinguish a forecaster reporting true conditional event probabilities from a charlatan who knows only how the test works.
- Observations consist of probabilities, such as a 2% event forecast, followed by whether the event occurs.
- The motivating question is whether any test can reveal genuine understanding of the underlying process.
- Calibration checks whether events occur at the predicted frequency, such as rain on roughly 23% of days forecast at 23%.
- For an alternating sequence 0, 1, 0, 1, a constant 50% forecast is perfectly calibrated.
- A forecaster who knows calibration is the test can pass it without learning the deterministic pattern.
- The audit can inspect the forecasting rule and query predictions after histories that did not actually occur.
- Ordinary calibration uses only forecasts and outcomes along the realized sequence, so it ignores this additional access.
- The model may also condition on daily features, without changing the core argument.
- An outcome distribution induces the true conditional probability of the next event after every history.
- The informed forecasting rule reports that conditional probability for each history.
- A test may be any function of the published rule and one realized outcome sequence, including off-path model behavior.
- Acceptance should mean the audit cannot rule out that the forecaster is informed.
- For every possible outcome law, an informed forecaster must be accepted with probability at least 1 minus α.
- The error probability is over the random sequence drawn from the true law, and the same test must work across all laws.
- A test must issue accept or reject after a finite, fixed horizon T, such as 1,000 or 1,000,000 days.
- Waiting until T can include any sequential rejection rule, so it does not weaken the test.
- Without a fixed deadline, a test can keep the option to reject open indefinitely, changing the result.
- Repeatedly querying the same history should not expose fresh random answers from an online calibration algorithm.
- Model random choices as drawn at day zero and embedded in a deterministic rule that the auditor can inspect.
- The forecasting rule can still adapt to previous outcomes, as the correct conditional forecast may change with history.
- If a test's false-negative rate is at most α for every informed forecaster, a test-aware strategy can pass with probability at least 1 minus α.
- The strategy can be randomized and can achieve that pass probability for every possible outcome sequence.
- Thus, under this setup, the test has no guaranteed distinguishing power between informed and uninformed forecasting.
- The learner chooses a forecasting rule to maximize the chance of passing the test; nature chooses an outcome sequence to minimize it.
- Mixed strategies represent distributions over forecasting rules and distributions over outcome sequences.
- The minimax theorem will equate the value when nature moves first with the value when the learner moves first.
- With T binary outcomes, nature has 2^T pure outcome sequences.
- Although forecasts range over real probabilities, a rule can be represented by its pass-or-fail result on each of the 2^T sequences.
- There are at most 2^(2^T) distinct such test patterns, so the game can be reduced to finite payoff-equivalent strategies.
- When nature moves first and commits to an outcome distribution, the learner can best respond with its true conditional forecasts.
- The test's false-negative guarantee means this informed response passes with probability at least 1 minus α.
- By minimax, a learner that commits first to a test-specific randomized forecasting rule can guarantee the same pass probability against every sequence.
- The finite-game proof depends on a fixed T; allowing monitoring for arbitrarily long periods removes the finite sequence bound.
- A variable rejection time can depend on the forecasting rule, so no single cutoff works for every rule.
- This changes the order of quantifiers that enabled the minimax argument.
- A forecasting rule implies a distribution over outcome sequences and therefore over the day of the first 1.
- The audit finds a finite day N after which the forecast assigns at most α probability to the first event occurring.
- It rejects if the first event occurs after N, using the forecast's own tail as the rejection region.
- If the forecast distribution is correct, the probability that the first event lands in its α-tail is at most α.
- For every fixed forecasting distribution, an outcome sequence can place the first event beyond a sufficiently large cutoff and cause rejection.
- For randomized forecasts, a cutoff can be chosen to make rejection highly likely over the forecaster's random choice, but this does not establish power under the real outcome law.
- After seeing a forecast, nature can choose a later first-event time than the forecast's tail cutoff.
- This resembles the unbounded choose-the-bigger-number game: whichever side names a finite integer first can be outdone.
- The sequential test only defeats the theorem's guarantee of passing every sequence; it does not make auditing generally easy.
- If outcomes are known to follow a product distribution, the correct forecast is constant across days because observations are independent.
- An audit could reject changing forecasts and check whether their long-run bias approaches zero.
- Product distributions are not convex under mixtures, so restricting nature to them breaks a condition used by the minimax theorem.
- Online calibration algorithms provide efficient strategies for passing calibration tests, while the general minimax construction may be computationally expensive.
- Some efficiently computable tests may resist efficient uninformed strategies, but can also be hard for informed forecasters if true conditional probabilities are hard to compute.
- If those conditional probabilities are efficiently computable, an efficient strategy can pass any test; Roth says this question continues in the next lecture.
Transcript excerpt
0:00 Right. I want to today think about an auditing or testing. Imagine that there is some phenomena that you would like to predict. You don't know the physical process underlying this phenomena. Think about I don't know, weather on Pluto or you know, you're trying to predict somehow relevant event. Point is like you know, what you're going to get to observe is a sequence of probabilistic forecasts things like you know, I think there's a
0:48 2% chance that the event happens today. Followed by realization of whether the event actually happened today or not. And you've got this forecasting technology that is feeding you these forecasts, but like what you're trying to audit for is whether this forecasting technology actually understands the underlying physical process governing probabilistic nature of these events. That is whether it is actually telling you the true conditional outcome probability given everything it has. Or whether instead it's not. It's somehow you know, doing something that is just trying to fool you into
1:36 thinking that it knows the underlying ground truth. So like on the one you're trying to sort of distinguish on the one hand like a fully informed forecaster that knows every day the true conditional label probabilities and tells them to you honestly. On the other hand, maybe like a charlatan, like a completely uninformed forecaster that knows the test it's trying pass, that knows how your auditing it. But otherwise um you know like knows nothing at all about the underlying physical process. And the question is like to what extent can we design tests that have some distinguishing power, that have some ability to tell the difference between like a forecasting technology that is correct versus one that is kind
2:27 of just trying to game the test. So um just to sort of
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Aaron Roth.