CSci574 / AI 520 Machine Learning: Classroom Session 09/22/2026
Overview
Derek Harter reviews the implementation requirements for Assignment 3, then explains how linear regression minimizes mean squared error using the normal equation and gradient descent. He connects the gradient formula to NumPy vectorization, illustrates parameter updates with one- and two-parameter cost surfaces, and explains why learning-rate choice matters.
Key takeaways
- The normal equation, theta = (XᵀX)⁻¹Xᵀy, gives the analytic linear regression minimum, but matrix inversion becomes costly as the number of features grows.
- For mean squared error, the gradient can be computed as (2/n)Xᵀ(Xθ - y), allowing NumPy to update every model parameter in one vectorized operation.
- Gradient descent updates parameters by subtracting the learning rate times the gradient; the learning rate must be small enough to avoid overshooting or divergence.
- Adding a column of ones as the first feature lets linear regression learn an intercept alongside the feature weights through the same matrix operations.
- The Assignment 3 implementation is designed to generalize from one feature to multiple features without changing the core prediction and gradient calculations.
Chapters
- The `add_dummy_feature` function takes a NumPy array and returns it with a column of ones in column zero, representing the intercept or bias.
- Students need to add the function to the assignment task module and uncomment the corresponding `from` import in the notebook.
- A stub that returns the expected type can help confirm that imports and notebook code work before implementing the function.
- Task 2 implements prediction by multiplying the input features, including the bias column, by the model parameters theta.
- Task 3 implements mean squared error and a combined loss-and-gradient function; Task 4 implements gradient descent.
- Task 5 extends the work to multiple features through notebook code and plots, without requiring another function.
- Derek Harter recommends separate commits for the five tasks and notes that assignments are generally due Sunday at midnight.
- For a given theta, predictions are computed from the feature matrix and compared with the true labels to produce an error for each example.
- Mean squared error squares the prediction errors, so errors above and below the target both contribute according to their magnitude.
- Squaring the errors gives a cost function with an analytically calculable derivative, which supports optimization with gradient descent.
- For a quadratic cost function, setting the derivative to zero identifies the minimum; the matrix equivalent gives the normal equation.
- The solution is theta = (XᵀX)⁻¹Xᵀy, using the feature matrix X and true labels y.
- On the 47-point housing example, the fitted line has an intercept of about 71.2 and a slope of about 0.13.
- NumPy's normal-equation result matches the line from `polyfit`; the same quadratic objective has a single global minimum.
- Matrix transposes and inverses in the normal equation can become computationally expensive when a model has hundreds or more features.
- Gradient descent approximates the minimizing parameters through repeated updates and can be more efficient for large datasets.
- The notebook's Matplotlib animation requires the external FFmpeg executable; on Debian-based environments, Derek Harter demonstrates installing it with `sudo apt install ffmpeg`.
- An animation cell still encounters a problem after installation, which Derek Harter leaves to investigate later.
- A toy dataset with points (0,0), (1,1), (2,2), and (3,3) has a best-fit line with slope 1 and intercept fixed at zero.
- With the correct slope, the sum of squared errors is zero; slopes such as 1.1 produce a positive cost.
- Plotting cost against candidate slopes produces a parabola with its minimum at slope 1, illustrating the shape of the optimization objective.
- Starting from a slope below the minimum, a negative derivative indicates that increasing the parameter moves toward lower cost.
- At a slope above the minimum, a positive derivative indicates that decreasing the parameter moves toward lower cost.
- The update direction follows the negative gradient, but an excessively large step can overshoot the minimum and cause divergence.
- When both intercept theta 0 and slope theta 1 vary, the cost is visualized as a two-parameter surface rather than a single parabola.
- Contour colors, contour lines, and a 3D plot show the cost falling toward its minimum.
- For the toy example, the minimum occurs at theta 0 = 1 and theta 1 = 1, where the line fits the data exactly.
- The 47-point housing dataset has no line that fits every point, but its cost surface still has a minimum.
- The contour visualization contrasts a poor initial model, with an intercept near 1,600 and a negative slope, against the fitted values near 71.2 and 0.13.
- The gradient components indicate which direction to adjust the intercept and slope to move toward lower-cost contours.
- For parameter theta j, the mean squared error derivative combines the prediction error with feature column j and the factor 2/n.
- The error term is the prediction, Xθ, minus the true labels y; multiplying it by the features produces parameter gradients.
- A vectorized matrix operation can calculate gradients for the intercept, slope, and additional feature parameters without a Python loop.
- Assignment 3 initializes the theta parameters at zero and updates them using theta := theta - alpha times the gradient.
- The learning rate alpha controls step size: a very small value slows progress, while a value that is too large can overshoot or diverge.
- Derek Harter explains that subtracting the gradient moves each parameter toward lower cost when the step size is suitable.
- The next class will cover batch gradient descent and mini-batches.
Transcript excerpt
0:05 Okay. Uh, let me try restarting that again. Okay. Um, so I think I'm back. Can you guys hear me again? Is my audio okay on the Zoom?
0:58 >> Yeah, we can hear you. >> All right, great. Uh, all right. So, sorry about that. I hope it doesn't do it again. I I'm not certain why I'm having these issues with freezing. So, uh, where I where I left off, um, I was I was going to kind of bookend the class talking about assignment two. Uh, so there's a little bit of a difference in what you need to do for the workflow. So I started saying that um so like for the first task you're supposed to write this add dummy features um function but um I didn't give the kind of like the stubs to you. So you have to do like uh find the from import in the notebook um and uncomment that uh and in order to use it. Um so I think I normally have some instructions about you shouldn't modify uh these cells
1:46 where we're calling the uh pi tests but uh in this case uh you should uh actually actually do modify them to to uh open the code here. So like in this case um it's probably described in here but uh when you're implementing this function it's going to take a like an array as input uh and return an array as a result. Um so as I started saying uh um you should have some PID documentation of the general form um descriptions I just do this real quickly. I won't type in everything here, but um like uh in this case, we're taking
2:34 x, which is a numpy array um and we're going to return the same array
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Derek Harter.