ES.1802, October 14, 2026

Overview

The lecture connects linear regression to linear algebra: when data points cannot all lie on a line, the equations for the line parameters have no exact solution, so the best fit is found by projecting onto the design matrix’s column space. For a predictor y = c + dt, it defines the half-squared-error objective over observations and outlines minimizing it with respect to c and d by setting the derivatives to zero.

Key takeaways

For a line y = c + dt, n observations create n equations but only two unknown parameters, so arbitrary data usually cannot be fit exactly.
The regression design matrix has one row [1, tᵢ] per observation; its columns encode the intercept and the input values.
Least squares chooses c and d to minimize the sum of squared residuals, preventing positive and negative prediction errors from canceling.
The least-squares fit is equivalent to projecting the observation vector onto the design matrix’s column space.
Setting the error’s partial derivatives with respect to c and d to zero yields the minimizing parameters and connects the solution to means, variance, and covariance.

Chapters

0:00 Projection Recap: When a Vector Is Outside a Matrix’s Column Space

The lecture recalls projecting a vector B onto the column space of a matrix A.
The projection matrix is given as A(AᵀA)⁻¹Aᵀ, which produces the closest vector in that subspace.
This setup motivates what to do when a data vector cannot be represented exactly by the columns of A.

2:40 Modeling Age and Height with a Line

Age and height serve as an example dataset, with points such as age 10 and height 48 inches.
The proposed linear predictor is y = c + dt, where c is the intercept and d is the slope.
Because no single line passes through every plotted point, fit quality is judged by the prediction errors between observed y and predicted y*.
The lecture chooses half the squared error, ½(y − y*)², so positive and negative errors do not cancel and differentiation is simpler.

10:30 Writing the Regression Data as an Inconsistent Linear System

For four observations, the exact-fit equations are yᵢ = c + dtᵢ for i = 1, 2, 3, 4.
In matrix form, the design matrix has rows [1, tᵢ], the parameter vector is [c, d]ᵀ, and the observations form the vector of y-values.
Four equations generally cannot all be satisfied by only two parameters when the points are not collinear.
An unsolvable system means the observation vector is outside the design matrix’s column space, linking regression back to projection.

15:30 Minimizing Total Squared Error with Derivatives

The least-squares choice of c and d minimizes the sum of half-squared residuals across all data points.
The observed tᵢ and yᵢ values are fixed; only the line parameters c and d are adjusted.
To find the minimum, differentiate the error with respect to c and d, then set both derivatives to zero.
Solving these conditions produces relationships involving means, variance, and covariance, and the fitted prediction is the projection onto the design matrix’s column space.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, ChristianAltoSax.