CSci574 / AI 520 Machine Learning: Classroom Session 09/24/2026

Overview

Derek Harter reviews course deadlines and explains how gradient descent choices affect optimization, from full-batch updates to stochastic and mini-batch methods. He also uses scaled housing data and synthetic linear data with noise to show why feature scaling matters, why fitted parameters may differ from the true relationship, and how repeated fits reveal uncertainty.

Key takeaways

Full-batch gradient descent uses every training example for each update and gives a smoother cost signal, but its computation and memory requirements become impractical for very large datasets.
Stochastic gradient descent updates from one randomly selected sample, which can make training faster on large data but creates noisy learning curves and run-to-run variation.
Mini-batches—such as batches of 10 examples—balance update cost and stability by reducing the noise of single-sample updates without processing the entire dataset each time.
Feature-scale mismatches can obstruct gradient descent: scaling the 47-sample housing data’s size and price features allowed the optimizer to approach the fit from scikit-learn and polyfit.
Noise prevents a fitted model from exactly recovering the generating parameters: repeated fits to noisy samples produce a distribution of slopes and intercepts rather than one certain answer.
A learning-rate schedule can use larger steps early and smaller steps near an optimum, while a tolerance criterion can stop iterative optimization when cost improvement becomes negligible.

Chapters

0:00 Project Dataset Selection and Upcoming Deadlines

Assignment 3 is due Sunday at midnight; students should accept its repository assignment and begin work.
The class project requires choosing a dataset and updating the top-level README with the choice and an initial plan.
Dataset choices are first come, first served; avoid commonly reused datasets such as Iris, Wine Quality, and Breast Cancer.
Projects must use data that can be shared publicly, though students may adapt research or work-related datasets if sharing is permitted.

7:34 Online Midterm Format and Individual-Work Rules

The online midterm will use MyLeo Online, an empty Jupyter notebook, and a timed submission process.
Expect roughly two or three coding questions, such as fitting linear or polynomial regression or applying regularization.
The midterm is open-book and internet-enabled, but assignments and tests must be completed individually without collaboration or past solutions.
A final exam is also planned, with its timing still under consideration because of fall break.

10:33 Where to Find a Suitable Machine-Learning Dataset

Derek Harter recommends searching the UCI Machine Learning Repository for regression and classification datasets.
Kaggle competitions and their associated datasets are another source of project material.
A linked NSF-funded dataset resource aggregates other repositories and may offer more realistic data suitable for portfolio or job-hunting projects.

12:18 Assignment 3 Implements Full-Batch Gradient Descent

Assignment 3 asks students to calculate mean-squared-error gradients and implement a basic batch gradient-descent optimizer.
Full-batch gradient descent calculates cost and gradients using every training sample for each update.
The implementation should keep a history of optimization results so the cost can be plotted across iterations.
A cost-versus-iteration plot acts as a learning curve, showing whether optimization is progressing.

15:50 Synthetic Data Shows Learning-Rate and Convergence Behavior

A generated dataset has a known intercept of 4 and slope of 3, with normally distributed noise added to the targets.
With a learning rate of 0.1, 10 iterations do not reach the known parameters; simply increasing iterations can produce little improvement if the rate is unsuitable.
Long runs can make early cost reductions hard to see on a linear plot, so a logarithmic axis may help display optimization progress.

19:39 Feature Scaling Improves Gradient Descent on Housing Data

The 47-sample housing dataset has an approximate best-fit intercept of 71 and slope of 0.13, but unscaled gradient descent struggles to recover them.
Housing size spans roughly 800–4,500 square feet while prices are measured in thousands of dollars, creating a substantial feature-scale mismatch.
Applying min-max scaling allows the manual gradient-descent implementation to approach the scikit-learn or NumPy polyfit result after many iterations.
Scaling changes the parameter values, so the fitted model must be transformed back to the original units before interpreting its slope and intercept.

26:50 Why Full-Batch Optimization Becomes Expensive

Full-batch gradient descent computes cost and gradients over the entire dataset for every update.
With millions of samples, each full-batch update can be slow; genuinely large datasets may not fit in 16, 64, or even 128 GB of memory.
Processing subsets of data can reduce computation and memory demands, motivating stochastic and mini-batch approaches.

30:00 Stochastic Gradient Descent Uses One Sample per Update

Stochastic gradient descent calculates each cost-and-gradient update from a single randomly selected training example.
Selecting samples independently at random can repeat some examples and omit others during an epoch.
Single-sample updates make optimization curves noisier than full-batch curves; individual epochs may show cost increases even when progress trends downward overall.
The randomness of sample order can produce different learning curves across runs.

36:50 Shuffle Every Sample Once and Schedule the Learning Rate

A common implementation creates indices from 0 to m−1, shuffles them each epoch, and visits each of the m samples exactly once.
A learning-rate schedule starts with larger updates and reduces the step size as optimization approaches a solution.
Derek Harter’s example derives the rate from the epoch and within-epoch iteration, decreasing it over time rather than using one fixed value.
Shuffling and scheduled learning rates can reduce some instability, but stochastic results remain dependent on sample order.

43:28 Exact Linear Regression, Early Stopping, and Noisy Targets

Scikit-learn’s basic LinearRegression uses an exact linear-algebra approach, such as singular-value decomposition, rather than the hand-coded gradient-descent procedure.
Exact methods and iterative optimizers have different performance tradeoffs depending on dataset size and feature count.
Iterative estimators may stop when cost improvement falls below a tolerance; lowering the tolerance can allow more iterations.
Because the synthetic targets contain noise, a fitted intercept and slope need not equal the generating values of 4 and 3.

48:16 Mini-Batch Gradient Descent Balances Speed and Stability

Mini-batch gradient descent uses several samples per update rather than one sample or the entire dataset.
The implementation shuffles sample indices and slices them into batches, such as groups of 10.
A batch size of 10 produces a less noisy learning curve than single-sample updates while avoiding full-dataset calculations.
Choosing batch size balances computational performance against the noise that can obscure whether optimization is working.

51:50 Noise Hides the True Relationship in Synthetic Data

The synthetic example generates inputs between approximately 0 and 2 from 100 samples using a known line with intercept 4 and slope 3.
Without added noise, every point lies exactly on the generating line; adding normal noise with standard deviation 1 obscures that relationship.
Reducing the noise scale to 0.5 makes points cluster more closely around the true line, while larger noise makes the relationship harder to identify.
For real datasets, the underlying relationship and noise level are unknown, so a fitted linear model is a hypothesis rather than a guaranteed recovery of the truth.

1:01:00 Repeated Fits Illustrate Parameter Uncertainty

The normal-equation-style solution, scikit-learn LinearRegression, and polyfit produce matching fitted results for the same noisy dataset, even though those results differ from the true parameters.
Repeatedly fitting 100-sample datasets generated from the same line but with different noise yields a range of slopes and intercepts.
That spread motivates confidence intervals: statistical analysis can estimate how tightly fitted parameters may be expected to approximate the unknown true relationship.
Derek Harter defers polynomial regression and regularization until the following week, noting that noise and model uncertainty will matter for both.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Derek Harter.