Mathematical Foundations of AI Alignment, Lecture 13 (Hallucination and Calibration)

Overview

Aaron Roth models factual hallucination using independent, arbitrary facts: each prompt has one correct answer among K+1 possibilities, and training data contains only correct answers but may leave substantial missing mass. He proves that hallucination rate plus a coarse calibration discrepancy is at least missing mass minus 2/K; because maximum-likelihood training favors calibration, it can favor hallucination over abstention when the model has not seen many prompts. He closes with the Good-Turing singleton estimator, which estimates missing mass from the fraction of training observations seen exactly once, with bias at most 1/n for a sample of size n.

Key takeaways

With K incorrect answers and one correct answer per prompt, an unseen fact has only 1/(K+1) chance of being guessed correctly, regardless of the training algorithm.
A lookup-table model that abstains on every unseen fact can have zero hallucination, but its calibration discrepancy can be as large as the missing mass.
For any generator in the toy model, hallucination rate plus coarse calibration discrepancy is at least missing mass − 2/K.
Maximum-likelihood training drives the coarse calibration discrepancy to zero when the model family can rescale high-confidence probabilities, leaving hallucination as the cost of substantial missing mass.
The Good-Turing singleton fraction estimates expected missing mass for a sample of size n−1; for a size-n sample its bias is at most 1/n.
The results apply to independent, arbitrary facts such as birthdays, not to questions whose answers can be deduced from learned rules, such as arithmetic.

Chapters

0:00 Calibration, Distribution Matching, and Hallucination

Aaron Roth connects calibration to earlier results on agreement, including Aumann’s agreement theorem.
The lecture asks why models trained to match a distribution may confidently state false facts instead of saying “I don’t know.”
The proposed explanation uses only correct training facts, so hallucinations cannot be blamed on errors in the dataset.

4:00 Why Independent Factoids Cannot Be Deduced

The toy setting focuses on arbitrary facts, such as a person’s birthday, that cannot be inferred from unrelated facts.
A model may learn mathematical rules and solve unseen arithmetic questions, but a birthday absent from its data has no such inferential route.
For an unobserved birthday, the model has no evidence favoring one of the 365 month-and-day answers.

8:32 The K+1-Answer Prior and Missing Mass

Each prompt has K incorrect answers and one correct answer, for K+1 possible answers total.
Nature selects each prompt’s correct answer uniformly and independently, so observing other prompts gives no information about it.
Training consists of n independent prompt-answer samples; missing mass is the probability that a fresh prompt was not observed in training.

10:00 The Error Floor When a Model Must Always Guess

For an unseen prompt, any predictor’s chance of choosing the correct answer is only 1/(K+1), because the true answer is independent of its training data.
A system required to answer every prompt has error at least (1 − 1/(K+1)) times the missing mass.
With birthdays, the unseen-prompt error is 364/365, multiplied by the fraction of people whose birthdays the model does not know.

14:00 Abstention Makes Zero Hallucination Possible

Allowing an “I don’t know” response lets a model answer known prompts from a lookup table and abstain on every unseen prompt.
That lookup-table strategy can avoid all hallucinations, even though it cannot answer questions outside its training sample.
The central issue becomes whether pre-training’s distribution-matching objective selects abstention or some other behavior.

18:02 Defining Nature’s Distribution and Hallucination Rate

Nature’s distribution assigns probability 1 to the correct factual answer and 0 to incorrect answers or “I don’t know.”
A generator produces an answer distribution conditional on each prompt, including the option to abstain.
Hallucination rate is the probability that the generator returns an incorrect factual answer; both a correct answer and “I don’t know” are non-hallucinations.

25:00 Why Guessing and Abstaining Behave Differently

The lecture distinguishes a model that guesses uniformly on unseen prompts from one that says “I don’t know.”
The guesser’s hallucination rate is (1 − 1/(K+1)) times missing mass, while the abstainer’s rate is zero.
Reporting uncertainty is a possible alternative, but Roth notes challenges of usability and defining what an uncertainty estimate means for arbitrary factual queries.

27:00 Semantic Calibration as Matching Confidence-Level Mass

Nature and the generator share the same prompt distribution; they differ in their answer distributions conditional on each prompt.
For each confidence value, semantic calibration compares the mass of question-answer pairs at that model confidence under the generator and nature.
A model can be calibrated on each confidence level even when its factual behavior on unseen prompts is undesirable.

32:00 The Guesser Is Calibrated; the Abstainer Is Not

The guessing model assigns probability 1 to memorized answers and 1/(K+1) to each answer on an unseen prompt.
Its confidence-level masses match nature’s: known prompts have mass 1 minus missing mass, and unseen answers collectively have mass equal to missing mass.
The abstainer assigns probability 1 to either a memorized answer or “I don’t know”; because nature never abstains, its calibration discrepancy can equal the missing mass.

36:34 A Two-Level Calibration Test and the Tradeoff Theorem

Roth coarsens confidence into a high-confidence set, where answer probability exceeds 1/K, and its low-confidence complement.
The coarse calibration discrepancy is the absolute difference between nature’s and the generator’s mass on the high-confidence set.
For any generator, hallucination rate plus this discrepancy is at least missing mass minus 2/K.

41:31 Reducing the Tradeoff to a Binary Classification Problem

The proof imagines a classifier whose examples are prompt-answer pairs, with a fair coin choosing positive or negative labels.
A positive example pairs a prompt with its true answer; a negative example pairs it with a uniformly selected incorrect answer among K choices.
On an unseen prompt, the label provides no information about which answer is correct, so every classifier has error at least 1/2 there.

47:06 Turning the Generator into a Confidence-Threshold Classifier

The constructed classifier accepts a prompt-answer pair as positive exactly when the generator assigns it probability greater than 1/K.
That rule is equivalent to classifying the generator’s high-confidence set as valid and the low-confidence set as invalid.
Since unseen prompts have classification error 1/2, the constructed classifier’s total error is at least half the missing mass.

50:55 Bounding Classifier Errors by Hallucinations and Calibration

False positives are incorrect pairs in the high-confidence set; because their generator probability exceeds 1/K, their generated mass is at least twice their classification mass.
False negatives are correct pairs in the low-confidence set; the low-set mass is bounded using the classifier’s false-negative rate and the coarse calibration discrepancy.
At most 2/K mass in the low-confidence set can be assigned to the two non-hallucination outcomes: the correct answer and “I don’t know.”
Combining the bounds yields hallucination rate at least twice classifier error minus calibration discrepancy minus 2/K, giving the tradeoff theorem.

1:07:42 Why Likelihood Training Favors Calibration

The tradeoff leaves two options when missing mass is large: accept substantial calibration error or incur a substantial hallucination rate.
Roth argues that minimizing negative log-likelihood naturally selects low calibration error when the model family can adjust probability mass across confidence sets.
Thus, likelihood-trained models can favor guessing over abstention on unseen factoids.

1:10:18 Rescaling High-Confidence Answers Changes Log-Likelihood

The proof multiplies generator probabilities in the high-confidence set by a factor s, leaves other weights unchanged, then renormalizes.
At s = 1 the model is unchanged; increasing or decreasing s shifts probability mass toward or away from high-confidence answers.
The derivative of negative log-likelihood at s = 1 equals the difference between generator and nature mass on the high-confidence set.
If the model is an interior likelihood optimum and can make this adjustment, the derivative must be zero, implying zero coarse calibration discrepancy.

1:19:29 Estimating Missing Mass with the Good-Turing Method

Missing mass is unknown by definition, but the number of prompts observed exactly once—singletons—provides a practical estimate.
The Good-Turing estimator, historically used during World War II to estimate German tank production, uses the fraction of sample observations that are singletons.
The singleton fraction is an unbiased estimate of expected missing mass for a sample of size n−1.
Adding one observation can reduce missing mass by at most 1/n in expectation, so the estimator’s bias for a sample of size n is at most 1/n.

1:27:44 Lecture Takeaway: Missing Facts Make Calibration Costly

For independent factoids, a model cannot avoid both hallucination and miscalibration when substantial probability mass remains unseen.
Log-likelihood minimization favors calibration, so the theorem implies higher hallucination rates when missing mass is large.
The Good-Turing singleton estimate offers a data-based way to assess whether the missing-mass regime is substantial.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Aaron Roth.