Lecture 3b

Overview

The lecture reviews polynomial-model selection and cross-validation, showing how training error favors overly complex models while held-out error helps balance underfitting and overfitting. It then reviews probability tools needed for statistical learning, including random variables, expectation, variance, and conditional expectation, and sets up a model in which training data makes fitted coefficients random and test MSE is averaged over test noise and coefficient variability.

Key takeaways

Training RSS alone favors increasingly high polynomial orders, so held-out cross-validation is needed to identify a model that generalizes rather than fits training noise.
The one-standard-error rule selects the simplest model whose validation error is within one standard error of the minimum; in the lecture's example it recovers the true cubic order 3 despite minimum average error occurring at order 7.
Training, tuning, and final evaluation should be kept conceptually separate: repeatedly tuning against the same held-out data can overfit it, motivating a separately reserved final evaluation set.
A discrete random variable is described by a PMF that sums to 1, while a continuous variable is described by a nonnegative PDF that integrates to 1.
Independence enables useful reductions: expectations of products factor, conditional expectation becomes unconditional when the conditioning variable is independent, and variances of independent sums add.
When the training set is treated as random, fitted coefficients are random too; conditional test MSE therefore averages across both test noise and possible training outcomes while holding test features fixed.

Chapters

0:00 Polynomial Overfitting, Cross-Validation, and the One-Standard-Error Rule

A linear model underfits data generated by a cubic polynomial, while a 15th-order polynomial overfits noise and incidental patterns.
Training RSS generally decreases as polynomial order increases, so training fit alone does not identify the right complexity.
Two-fold and k-fold cross-validation estimate held-out error; with 10 folds, each fit uses 90% of the data and errors are averaged across 10 test folds.
The one-standard-error rule chooses the simplest model within one standard error of minimum validation error; in the example it selects order 3 instead of the minimum-error order 7.

9:46 Training, Test, and Validation Data Have Different Jobs

Training data estimates model parameters such as polynomial coefficients β, while a held-out test split is used to tune hyperparameters such as order d.
Repeatedly choosing models based on the same test data can overfit that test set, even though it was excluded from parameter fitting.
Validation data is reserved for evaluation after design is complete; competition organizers may keep it inaccessible to participants.
Terminology varies across machine-learning sources: some use “validation” for hyperparameter tuning and “test” for final evaluation.

14:43 From Polynomial Order to the Bias-Variance Trade-Off

Polynomial degree is one example of a model-complexity hyperparameter; neural-network depth and width are other examples.
The lecture distinguishes sample MSE, a numerical average over observed data, from statistical MSE, which uses probability distributions and expectations.
The upcoming statistical MSE decomposition will separate squared bias, variance, and irreducible error.
Larger training sets are expected to reduce variance and improve test MSE, while model complexity trades off bias against variance.

18:36 Random Variables, Realizations, PMFs, and PDFs

A random variable represents uncertainty; the marble-bag analogy distinguishes the full distribution from an individual realization drawn from it.
Discrete random variables take values from a countable set, such as {0, 1} or the integers, and are described by a probability mass function whose entries sum to 1.
Continuous random variables have uncountably many possible values and are described using a cumulative distribution function or its derivative, the probability density function.
A PDF must be nonnegative and integrate to 1; probabilities over ranges are obtained by integrating the density.

26:40 Expectation and Its Linearity and Independence Rules

Expectation is the statistical mean: for discrete variables it sums function values weighted by their probabilities, and for continuous variables it integrates against the PDF.
Linearity allows deterministic constants to come outside expectation and turns the expectation of a sum into the sum of expectations.
For independent random variables A and B, the expectation of f(A)g(B) factors into E[f(A)]E[g(B)].
Independence means the joint PMF or density factors into the product of the separate distributions; knowing one variable gives no information about the other.

33:12 Statistical Variance and Conditional Expectation

Statistical variance is E[(A − E[A])²], with the equivalent form E[A²] − E[A]² often useful in derivations.
Conditional expectation E[f(A) | B=b] averages over A using its distribution conditioned on B taking the value b.
The rain-and-blue-sky example illustrates that the chance of rain changes when the sky condition is known.
When conditioning on A, terms that depend only on A can be treated as fixed and pulled outside the conditional expectation.

40:41 Independence Shortcuts and the Law of Total Expectation

If A and B are independent, E[f(A) | B] reduces to E[f(A)], because B provides no information about A.
For independent variables, Var(A + B) = Var(A) + Var(B), avoiding covariance terms.
The law of total expectation averages a conditional expectation over the conditioning variable to recover the unconditional expectation.
The lecture frames these identities as core tools for the upcoming machine-learning error derivation.

46:03 The True Data-Generating Model and Its Noise Assumptions

The true model maps a feature vector X through an unknown function and adds random noise to produce a target.
The noise distribution need not be specified, but it is assumed to have mean zero and finite variance σ².
Test features are treated as random and statistically independent of test noise; corresponding training features and noises are also modeled as random.
Training and test quantities are assumed independent, supporting generalization analysis beyond one particular dataset.

49:29 Why Fitted Coefficients Are Random Variables

A prediction model f-hat uses trained coefficients β-hat, which are outputs of an algorithm applied to the training data.
Because the training data is random, β-hat is random too, even though a practitioner observes only one fitted coefficient vector.
Theoretical analysis considers many possible training datasets, like repeated samples across hypothetical parallel cases, to describe average behavior.
Independence between training and test data makes β-hat independent of test features and test noise.

51:55 Conditional Statistical MSE: Averaging Over Noise and Training

The lecture defines statistical MSE as the expectation of the squared difference between the target Y and prediction Ŷ, conditioned on a fixed test feature vector x.
This conditional error averages over test-noise realizations and over randomness in the fitted coefficients β-hat.
The test features remain fixed in this measure, so the result can vary with x rather than collapsing immediately to one overall scalar.
The setup prepares the next derivation of the bias-squared, variance, and irreducible-error decomposition; that derivation is not completed here.

Transcript excerpt

0:02 Okay. Uh hope you guys had a good weekend. Uh let's get started by reviewing what we did last time. So we were talking about [clears throat] sort of motivated by this problem of polomial modeling. We have an option for how uh we choose the polomial order. We can choose a low order um which gives us a simple model like if we choose d equals 1 that gives us a linear model or we can choose a high order such as like 15th order polinomial and in this particular um example the data comes from a cubic model and so when we choose a linear model which is simpler than the true cubic data generating model we are underfitting which means means we're using an overly

0:50 simplistic model relative to the truth. When we use D equ= 15, we overfit, [clears throat] which means our prediction model is a lot more complicated than it needs to be. And when we underfit, uh we cannot really capture the true shape of the of the data. um when we overfit we end up capturing things about the data that are more like noise and other random aspects of it rather than the true data generating model which is this cubic uh polomial here. [clears throat] So um this is just sort of some extremes. In general we want to choose we want to somehow determine from the data itself the right complexity of the model the model that comes closest to uh

1:39 the complexity of the true data generating model. [clears throat] So uh we said that the way to do that is to use something called cross validation. uh because if you um just look at the training data alone and you look at a training data metric something like RSS

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof. Schniter.