Lecture 4b
Watch on YouTube →
Overview
The lecture develops regularization for feature selection, contrasting ridge regression's L2 penalty, which stabilizes estimates when features are correlated, with lasso's L1 penalty, which sets some coefficients exactly to zero. It also explains training-only standardization, cross-validation for choosing regularization strength, and how to combine lasso feature selection with least squares in scikit-learn to reduce coefficient bias without leaking validation data.
Key takeaways
- Standardize features using training-set means and standard deviations, then apply those same statistics to test data to keep preprocessing consistent and avoid leakage.
- Ridge regression uses β̂ = (XᵀX + αI)⁻¹Xᵀy; for α > 0, the added identity term stabilizes estimation even when correlated or overparameterized features make XᵀX singular.
- Ridge shrinks coefficients but usually leaves them nonzero, whereas lasso's L1 penalty can set coefficients exactly to zero and therefore select features.
- Lasso's selected features can be useful even when its coefficient values are biased: refitting least squares on the selected columns can reduce shrinkage bias.
- Tune α based on cross-validation performance of the full lasso-selection-plus-least-squares procedure; a scikit-learn Pipeline ensures each fold performs feature selection without using held-out data.
Chapters
- Using all D features can raise error variance, while dropping useful features can raise bias.
- Exhaustively testing 2^D feature subsets with cross-validation becomes computationally impractical as D grows.
- Regularization adds a penalty to residual sum of squares (RSS) to balance model fit against coefficient complexity.
- L1 and L2 penalties treat coefficients uniformly, so different feature scales can make the penalty unfair.
- Standardization subtracts each feature's training mean and divides by its training standard deviation, producing zero-mean, unit-variance columns.
- Centering features and targets makes the intercept beta₀ zero, so it need not be included in the coefficient vector.
- Compute means and standard deviations from the training set, then reuse those exact statistics to transform test samples.
- Computing test-set statistics can be impossible with a single sample and causes a mismatch between training and inference preprocessing.
- After centering, the feature matrix X excludes the intercept column; ridge and lasso combine RSS with L2 and L1 penalties, respectively.
- Ridge minimizes ||y − Xβ||₂² + α||β||₂², where α controls the L2 penalty.
- Lasso minimizes ||y − Xβ||₂² + α||β||₁, using the sum of absolute coefficient values.
- The L1 penalty is more influential near zero than the squared L2 penalty, which leads to different coefficient behavior.
- Because both terms in the ridge objective are quadratic, setting the gradient to zero gives β̂ = (XᵀX + αI)⁻¹Xᵀy.
- When α = 0, the solution reduces to ordinary least squares, (XᵀX)⁻¹Xᵀy, when that inverse exists.
- For any α > 0, adding αI makes XᵀX + αI invertible, even when the unregularized system is singular or overparameterized.
- A ridge coefficient path shows coefficients approaching the least-squares estimates at tiny α and shrinking toward zero as α grows.
- Ridge generally shrinks coefficients without setting them exactly to zero, so it does not perform feature selection.
- Highly correlated features can create near-zero eigenvalues in XᵀX; inversion amplifies them, producing large coefficients that are sensitive to test-data noise.
- Ridge controls coefficient magnitude, reducing the instability caused by correlated predictors and helping prevent overfitting.
- Choose α by evaluating a grid of candidate values with cross-validation, optionally using the one-standard-error rule.
- In the synthetic example, correlated features with true coefficients 1 and −1 produced least-squares estimates near −30 and 30; an appropriate ridge α brought estimates close to the truth.
- The absolute-value penalty is nondifferentiable at zero, so lasso lacks ridge's simple closed-form solution and requires numerical convex optimization.
- As α increases, lasso coefficients reach exactly zero and remain there, turning off features; the coefficient path reveals the order in which features are removed.
- Lasso also shrinks its nonzero coefficients toward zero, introducing bias even for features it keeps.
- Use lasso to identify the indices of nonzero coefficients, then discard its coefficient values and refit ordinary least squares using only those features.
- This selected-feature least-squares fit can reduce lasso's shrinkage bias when the selected set includes the relevant predictors and enough data are available.
- When combining L1 and L2 regularization, tune both penalty strengths jointly over a two-dimensional grid rather than assuming one should always precede the other.
- A basic lasso search uses nested loops over cross-validation folds and candidate α values, typically on a logarithmic grid spanning several orders of magnitude.
- Scikit-learn's GridSearchCV automates estimator fitting, parameter-grid evaluation, fold scoring, and collection of mean test scores.
- For lasso, larger α means fewer selected features and a simpler model, so the one-standard-error rule favors values to the right of the minimum; ridge has no comparable feature-count ordering.
- Selecting α by the prediction error of lasso's own coefficients optimizes a biased estimator, not the desired lasso-selected least-squares model.
- A better approach evaluates each α by running lasso for feature selection and then least squares on the selected columns; the error curve can have steps when the selected set changes.
- Fitting lasso on all data before cross-validation leaks validation information; a scikit-learn Pipeline keeps selection and least squares inside each fold, and GridSearchCV can tune α on the complete procedure.
Transcript excerpt
0:01 Welcome back everyone. So let's uh get started. Um last time we were talking about feature selection. So the [clears throat] idea there is that you select some subset of your D possible features. Uh the motivation is that if you use all your features um there may be too many of them. Some of them may not be very useful and so you'll see a large error variance. um we would like to use fewer than all of them but it's not clear which of them to use. If you leave out the wrong features um your model will have a high bearing uh bias. So the question is exactly which one should we be using? And um
0:49 [clears throat] we could use cross validation to go through every one of the possible subsets of features and find which one is best according to our our test error. But the problem is that um there's two to the power d of them. And so when d is um more than a small number this becomes captationally impossible. So [clears throat] typically people focus on suboptimal methods that are a lot cheaper than exponential and d complexity although they may not work quite as well. So uh one that we talked about is stepwise selection. We also talked about feature ranking based on some statistics that correlate features and target
1:37 individually. And [clears throat] then we just started to talk about regularizationbased methods and we'll continue that discussion today. So let's go to okay for the regularizationbased methods um [clears throat] the idea is to um okay so previously when we did this back in unit 2 uh when we designed our beta coefficients we chose them to minimize residual sum of squares which we know is defined this way um and that was [clears throat] a
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof. Schniter.