CSci574 / AI 520 Machine Learning: Classroom Session 09/29/2026

Overview

Derek Harter reviews project and Assignment 3 expectations, then explains how polynomial feature expansion lets linear regression fit nonlinear relationships and how learning curves reveal underfitting and overfitting. Using noisy quadratic data, NumPy, and scikit-learn, he compares model degrees, train-versus-validation error, and the bias–variance tradeoff as preparation for Assignment 4 and regularization.

Key takeaways

Polynomial regression can use ordinary linear-regression optimization: expand x into columns such as x and x², then fit a linear model over the expanded feature space.
Scikit-learn’s `PolynomialFeatures` creates both powers and cross-feature interactions, such as a²b, which becomes cumbersome to construct manually for multiple inputs.
A high-degree polynomial may minimize training error by fitting noise, yet produce unstable predictions for unseen inputs, especially near the boundaries of the observed range.
Learning curves compare error on progressively larger training subsets with error on held-out validation data; a persistent gap or rising validation error is evidence of overfitting.
Underfitting is associated with high bias and insufficient model capacity, while overfitting is associated with high variance and poor generalization; model selection should consider both training and validation performance.

Chapters

0:00 Class Project Dataset Selection and Sunday Repository Checkpoint

By Sunday, students should accept the class-project assignment, clone its repository, and edit the top-level README.
The README should name and link the dataset, describe its features and labels, identify whether the task is classification or regression, and sketch a planned analysis.
Derek Harter uses the submissions to check for duplicate datasets and datasets unsuitable for the project.

4:00 Assignment 3: Commit History and Required Implementation

Assignment 3 implementations of the loss gradient, prediction function, and fixed-iteration gradient descent loop were generally correct.
Students must divide work into multiple Git commits rather than submitting the entire assignment in one commit.
Commit history helps Derek Harter understand students’ progress and estimate time spent on individual tasks.

8:00 Python Docstrings, Autograder Checks, and Complete Notebooks

Python functions should include NumPy-style docstrings describing each argument, its type, and the return value.
Students should inspect the autograder after each commit and compare its test results with local runs; hard-coded paths can pass locally but fail in the grading environment.
Notebook cells must contain the requested work, not merely pass execution checks; restart the kernel and run all cells to verify the notebook runs cleanly.

13:00 Notebook Grading Workflow and Upcoming Course Schedule

A committed notebook with saved outputs lets Derek Harter inspect rendered results in JupyterHub and spot-check figures without cloning and rerunning the assignment.
The expected submission usually modifies the source task file and notebook; the autograder does not automatically save notebook outputs back into the repository.
Assignment 4 and the first test are due within roughly two weeks; the next assignment focuses on fitting nonlinear data with polynomial regression.

16:00 Quadratic Data Shows Why Linear Models Can Be Too Limited

Harter generates 100 noisy samples from a quadratic relationship containing an intercept, an x term, and an x² term.
Without noise, the generating parabola could be recovered exactly; noise makes the best-fit parameters differ from the original values.
Real datasets may follow polynomial, exponential, or other nonlinear relationships, even when their underlying form is not known.

20:00 NumPy polyfit Estimates a Chosen Polynomial Degree

NumPy’s `polyfit` requires the user to specify a degree, such as degree 2; it does not automatically choose the polynomial complexity.
Because the example data includes noise, the fitted coefficients approximate rather than exactly recover the generating parameters.
`polyfit` returns coefficients from the highest power down to the constant term, unlike scikit-learn’s usual intercept-and-coefficient arrangement.

25:00 Polynomial Features Turn Nonlinear Regression into Linear Regression

A quadratic model can be fit with the same linear-regression optimizer by treating x and x² as separate features, with an intercept for the constant term.
Scikit-learn’s `PolynomialFeatures(degree=2, include_bias=False)` converts a single input feature into columns for x and x².
A scikit-learn `LinearRegression` estimator then fits the expanded features and supplies its own intercept.

29:00 PolynomialFeatures Builds Interactions for Multiple Inputs

With multiple input features, nonlinear models often need interaction terms as well as separate powers of each feature.
For features a and b, degree-three expansion can include terms such as a²b and ab² alongside their individual powers.
`PolynomialFeatures` generates these combinations automatically, avoiding manual construction of an increasingly large feature matrix.

35:00 Degree 300 Polynomial Illustrates Overfitting

Harter compares degree-one, degree-two, and degree-300 fits to data generated from a quadratic relationship.
The degree-300 curve wiggles to follow noisy training points, showing how adding many parameters can fit noise as well as the underlying pattern.
Such a curve can make wildly different predictions for nearby unseen inputs, especially near the edges of the training range.

39:00 Training Error and Validation Error Diagnose Model Fit

An underpowered model tends to retain high training error because it cannot represent the data’s underlying relationship.
An overfit model can achieve low training error while performing poorly on held-out data that was not used during fitting.
The desired model balances representation power and generalization; training performance alone cannot establish that a model is useful.

46:00 Learning Curves Track Error as Training Data Grows

The learning-curve routine repeatedly fits a model using progressively larger subsets of training data and measures mean squared error.
It plots training error alongside error on held-back validation data, which remains unseen during each fit.
The example has 10,000 samples, with about 80% assigned to training; plotting only the first 150 training-set sizes can make early behavior easier to inspect.

53:00 Overfit Learning Curves and Scikit-learn Pipelines

For an overfit model, training error can keep falling while validation error remains much higher or worsens, so the curves fail to converge.
A high-degree polynomial pipeline combines `PolynomialFeatures` with `LinearRegression` and behaves like a single estimator with `fit` and `predict` methods.
Learning curves for thousands of training-set sizes can be slow; axis limits help reveal validation-error growth that might otherwise be compressed visually.

1:00:00 Bias–Variance Tradeoff and Regularization Preview

High bias corresponds to underfitting from an overly restrictive model; high variance corresponds to overfitting noise and unstable predictions on unseen data.
With the known quadratic example, the degree-two model reaches lower error than the degree-one model while its training and validation curves converge.
The best polynomial degree can be assessed by comparing training and validation error across degrees; the next class session will introduce regularization techniques to reduce overfitting.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Derek Harter.