Lecture 3d / 4a
Watch on YouTube →
Overview
The lecture connects the bias-variance tradeoff and estimator accuracy to feature selection, then compares exhaustive search, stepwise methods, feature ranking, and regularization. Using a 97-person prostate-specific antigen (PSA) dataset with eight biomarkers, it demonstrates cross-validation in scikit-learn and introduces ridge, lasso, and feature standardization.
Key takeaways
- The sample mean is unbiased, with variance v/n and standard error √v/√n, so increasing the sample size improves its precision.
- The sample variance using denominator n is biased downward by a factor of (n−1)/n; using n−1 removes that bias, although the biased estimator can still have lower MSE.
- Exhaustive feature selection requires checking 2ᴰ subsets, whereas forward or backward stepwise selection requires roughly D²/2 evaluations but can miss useful feature combinations.
- Univariate feature ranking followed by cross-validation can fail when top-ranked predictors are redundant, because individual scores do not account for relationships among features.
- Ridge penalizes squared coefficient magnitudes, while lasso penalizes absolute magnitudes; both add a term to RSS, exclude the intercept, and use α to set regularization strength.
- Standardizing training features and targets by subtracting their means and dividing by their standard deviations prevents regularization from unfairly penalizing coefficients merely because features use different units.
Chapters
- Test mean squared error decomposes into irreducible target-noise error, squared bias, and prediction variance.
- Bias measures how far the average prediction is from the true target; variance measures how much predictions vary across training samples.
- Increasing model complexity generally reduces bias and raises variance, so the best model order balances both.
- For independent samples Z₁ through Zₙ with mean μ and variance v, the sample mean is Z̄ = (1/n)ΣZᵢ.
- Linearity of expectation gives E[Z̄] = μ, so the sample mean is unbiased even though any particular estimate may differ from μ.
- Estimating a population quantity, such as average ECE graduate salary, requires reasoning over repeated samples rather than one observed list.
- For independent, identically distributed samples, Var(Z̄) = v/n, so using more observations reduces estimate variability.
- The standard deviation of the sample mean is √v/√n, which is the standard error.
- As n grows, the sample mean converges toward the population mean, and its variance approaches zero.
- Model-order selection changes how many features are used, while feature selection chooses which specific features to retain.
- With D features, each feature can be represented by a binary include-or-exclude decision, yielding 2ᴰ possible subsets.
- Exhaustive cross-validation can identify the best subset when D is small, but its exponential search becomes impractical as D grows.
- The example dataset contains PSA measurements from 97 men and eight biomarkers, including age and prostate measurements; the target is log PSA.
- Multiple linear regression predicts the target as an intercept plus weighted biomarker features, and coefficient magnitudes can aid interpretation when feature scales are comparable.
- scikit-learn's cross_val_score evaluates a model using repeated K-fold cross-validation, avoiding a hand-written loop.
- Because scikit-learn's scoring convention favors larger scores, mean squared error is returned as negative MSE; negating and averaging fold scores gives an estimated MSE of about 0.4.
- Forward selection evaluates each of D single-feature models with cross-validation, then retains the best feature.
- At each subsequent step, it tests each remaining feature alongside those already selected and adds the best candidate.
- The search requires roughly D + (D−1) + … + 1 evaluations, or about D²/2, rather than the 2ᴰ evaluations required for exhaustive search.
- Cross-validation error may eventually rise as added features cause overfitting, giving a stopping point.
- Backward elimination begins with all D features, then cross-validates models formed by removing each feature in turn.
- It repeatedly removes the feature whose exclusion gives the best performance, which suits problems expected to need most features.
- Removing redundant features can reduce variance at first, but removing too many eventually raises bias and causes underfitting.
- Forward and backward selection are greedy and can miss feature combinations that work well together; scikit-learn provides a SequentialFeatureSelector implementation.
- A ranking method scores each feature individually, often by its correlation or covariance with the target, then sorts features from strongest to weakest.
- Cross-validation selects the model order by comparing the top one, top two, and subsequent ranked features.
- Other possible scores include mutual information, F-values, p-values, and permutation importance.
- Univariate ranking can select redundant features: two nearly identical top-ranked predictors may add little bias reduction while increasing overfitting risk.
- Ordinary least squares chooses coefficients β̂ that minimize residual sum of squares (RSS); arg min identifies the coefficients attaining the minimum.
- Regularization adds a penalty to RSS so a feature is used only when its improvement in fit justifies the penalty.
- Ridge uses the L2 penalty αΣⱼβⱼ², which penalizes large coefficients more strongly than small nonzero ones.
- Lasso uses the L1 penalty αΣⱼ|βⱼ|, which penalizes coefficients immediately as they move away from zero; α controls penalty strength.
- Both penalties exclude the intercept by summing from feature index 1, and combining L1 and L2 penalties produces elastic net.
- Regularization treats coefficient magnitudes comparably, but different feature units can require very different coefficients even when the features are equally useful.
- A feature measured in hundreds may need a small coefficient, while one measured below one may need a large coefficient; penalizing these raw coefficients can unfairly favor one feature.
- Standardization centers each feature and target by its training-set sample mean and divides by its sample standard deviation.
- Centering also makes the fitted intercept zero in the centered representation, while scaling features to comparable variation makes coefficient penalties more meaningful.
Transcript excerpt
0:01 Okay, happy Friday. Let's get started. So last time we discussed uh this important equation which says that the mean squed error or test mean square error um test mean square error is going to be some irreducible error that comes from the noise in our test targets which can't be predicted plus the um squared bias and the bias is basically measures um if the bias is zero it means that on average your predicted target equals your true true target. uh that depends it can depend on your
0:51 test features and so this expectation is over those test features. And then we have the last term which is the variance in your predicted test targets and this measures how far they are away from their average value. Okay, so this one links the targets to the truth. This is just how far the targets are from their average. [clears throat] And what you see is as models get more complex uh the bias term tends to decrease. [clears throat] Um if you have a powerful enough model your bias will go to zero. That's the case for a lot of very sophisticated um giant neural networks and things like that that we
1:39 have today. >> [clears throat] >> you have um variance. Um this increases as your model gets more complex. And what you really care about is the sum of those two things. So you want to be somewhere in the middle where your bias is not too large and your variance is not too large. And that's going to be the optimal model complexity, the optimal model order. [clears throat] Um, so this is a super important concept and we'll be referring to this a lot as the semester goes on. Any questions on any of these ideas?
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof. Schniter.