CSci574 / AI 520 Machine Learning: Classroom Session 09/29/2026

Overview

Derek Harter explains how bias, variance, and learning curves reveal underfitting and overfitting, then demonstrates ridge (L2), lasso (L1), elastic net, and early stopping as ways to control model complexity. Using degree-25 polynomial regression on noisy quadratic data, he shows how alpha penalizes large coefficients, why lasso can zero out irrelevant features, and how these ideas connect to Assignment 4’s polynomial-model experiments.

Key takeaways

A degree-1 model underfits data generated by a quadratic because it cannot represent the true relationship; a degree-25 model can overfit by fitting noise and behaving erratically on unseen inputs.
Ridge regression adds alpha times the sum of squared coefficients, excluding the intercept; larger alpha shrinks coefficients smoothly, while alpha that is too large can reduce the model to an almost constant prediction.
Lasso uses the sum of absolute coefficient values and can set irrelevant coefficients exactly to zero, making it useful for feature selection as well as overfitting control.
For Harter’s noisy quadratic example, ridge with alpha around 1 and lasso with alpha around 0.01 produced useful fits, but the appropriate alpha depends on the model and data.
A widening gap between training and validation error indicates overfitting; in epoch-based training, stopping when validation error stops improving can preserve better generalization.
Assignment 4 applies underfit/overfit comparisons, learning curves, and ridge/lasso alpha exploration to a single-feature noisy polynomial dataset, with written interpretation and clear plot limits.

Chapters

0:00 Session Plan, Assignment 4, and Fall Break Schedule

Derek Harter says the class will focus on regularization and preview Assignment 4.
The assignment applies polynomial regression and regularization to fight overfitting.
The class will meet once during the following week and skip the fall-break Thursday session.

2:00 Bias, Variance, and Polynomial Underfitting

The example data comes from a degree-2 polynomial, so a degree-1 linear model lacks enough capacity and underfits.
Underfitting produces bias error because the model cannot represent the true relationship.
An overly powerful polynomial can follow noisy training points, creating high variance and unreliable predictions on unseen inputs.

8:00 Learning Curves for Diagnosing Model Fit

Training-set-size curves add data points incrementally, from one point up to the training split, which contains 80% of the data.
An underpowered model plateaus at higher error as more data is added; a suitable degree-2 model approaches the data’s irreducible noise level.
A large gap between training and validation performance signals overfitting, although sufficiently large datasets can sometimes help close that gap.

14:00 Choosing Model Capacity Before Regularizing

For an unfamiliar dataset, Harter recommends first finding a model that converges and comparing performance on training and validation data.
Testing a deliberately powerful model can reveal overfitting through a validation-performance gap.
When reducing polynomial degree is not practical, regularization can constrain a model while retaining its features.

15:30 Ridge Regression Adds an L2 Coefficient Penalty

Ridge regression adds alpha times the sum of squared parameters to the prediction-error cost.
The penalty applies to theta 1 through theta M, excluding theta 0, the intercept or bias term.
Alpha equal to zero removes regularization; increasing alpha favors smaller coefficients, but an excessively large value sacrifices predictive fit.

21:00 Ridge Regression Smooths a Degree-25 Polynomial

Harter fits a degree-25 polynomial to noisy quadratic data whose true coefficients are 3, 3/4, and 1/2, including the intercept.
With alpha zero, large higher-order coefficients produce a wavy fit that models noise; alpha around 1 gives a smoother, closer approximation.
At alpha 100, most coefficients shrink toward zero and the fit approaches a horizontal line because the intercept is unpenalized.
Ridge’s squared-coefficient penalty is also called L2 regularization in tools such as scikit-learn’s SGDRegressor.

27:00 Lasso’s L1 Penalty Enables Feature Selection

Lasso replaces the sum of squared coefficients with the sum of their absolute values, penalizing coefficient magnitude without sign cancellation.
Unlike ridge, L1 regularization can set unhelpful coefficients exactly to zero, producing a sparse model and supporting feature selection.
The right alpha retains useful predictors while eliminating features that do not improve the cost; lasso is also known as L1 regularization.

33:00 Lasso Results on the Noisy Quadratic Example

For the degree-25 fit, an alpha near 0.01 gives a close approximation to the underlying quadratic; alpha values suitable for ridge are not directly transferable.
Excessively large alpha drives non-intercept coefficients to zero, leaving an approximately constant prediction near the data average of 3.04.
With a balanced alpha, lasso retains the intercept, linear, and quadratic terms while removing most unnecessary higher-order polynomial features.

40:00 Why L1 Produces Sparse Models but L2 Usually Does Not

The textbook’s two-parameter diagram contrasts L1’s diamond-shaped penalty contours with L2’s circular contours.
L1’s corners make solutions with individual coefficients at zero more likely, explaining its feature-selection effect.
L2’s smoother squared penalty generally shrinks coefficients without forcing the less useful ones exactly to zero.

43:50 Elastic Net Mixes L1 and L2 Regularization

Elastic net combines absolute-value and squared-coefficient penalties, with alpha controlling overall regularization.
The scikit-learn parameter l1_ratio controls the mix: 1 uses L1 only, 0 uses L2 only, and 0.5 combines them evenly.
Harter notes that his Assignment 4 asks for ridge and lasso, not elastic net, and flags a likely theta-index typo in the displayed formula.

47:20 Early Stopping Uses Validation Error to Limit Training

For epoch-based training, such as neural-network optimization, both training and validation error may fall at first.
After overfitting begins, training error can keep decreasing while validation error rises; the textbook example marks this turning point around epoch 250.
Monitor unseen validation performance and stop at its best point to retain a model that generalizes better.

52:02 Assignment 4: Test Polynomial Degree and Regularization Strength

The assignment uses a single-feature dataset generated from a noisy polynomial, with its degree described as roughly between 4 and 8.
Students will compare underfit and overfit polynomial models, create learning curves, and test ridge and lasso with alpha values that are too low, suitable, and too high.
Harter asks students to set useful plot limits, especially on the y-axis, and discuss what alpha works, what polynomial degree seems plausible, and what lasso selects.
Harter says he will explain the assignment further next week and plans to revisit logistic regression.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Derek Harter.