15. Projections onto Subspaces

Overview

The lecture derives orthogonal projection first onto a line and then onto the column space of a matrix, showing that the nearest point is characterized by an error perpendicular to the subspace. It develops the normal equations A^T A x̂ = A^T b and the projection matrix A(A^T A)^-1A^T, then applies the setup to fitting a line to three data points that cannot all lie on one line.

Key takeaways

The closest point on a line through a nonzero vector a is a(a^T b)/(a^T a), obtained by requiring the residual b - P to be perpendicular to a.
An orthogonal projection matrix is symmetric and idempotent: P^T = P and P^2 = P, since projecting twice has the same result as projecting once.
For a full-column-rank matrix A, the least-squares coefficients satisfy A^T A x̂ = A^T b and equal (A^T A)^-1 A^T b.
The residual e = b - A x̂ lies in N(A^T), which is orthogonal to Col(A); this algebraic condition is exactly the geometric requirement for the nearest-point projection.
The projection onto Col(A) is A(A^T A)^-1A^T, and A itself cannot generally be inverted or canceled because it is rectangular.
Fitting b = C + Dt to (1,1), (2,2), and (3,2) creates a 3-equation, 2-unknown inconsistent system; least squares finds the line minimizing the residual through the normal equations.

Chapters

0:00 Deriving the Projection of b onto a Line Through a

The closest point P on the line through a has the form P = ax, so the unknown is the scalar coefficient x.
The error e = b - ax must be perpendicular to a, giving a^T(b - ax) = 0.
Solving yields x = (a^T b)/(a^T a) and P = a(a^T b)/(a^T a), assuming a is nonzero.

5:34 The Rank-One Projection Matrix and Its Two Defining Properties

Writing P as a matrix acting on b gives the projection matrix aa^T/(a^T a).
The matrix has rank one and column space equal to the line spanned by a.
It is symmetric, P^T = P, and idempotent, P^2 = P, because projecting a point already on the line leaves it unchanged.
Doubling b doubles its projection, while scaling a does not change the line or the projection.

15:20 Why Projections Solve Inconsistent Systems Ax = b

When b is outside the column space of A, the equation Ax = b has no solution.
Instead, choose the closest vector in Col(A), the projection P of b, and solve Ax̂ = P.
The least-squares solution x̂ gives the coefficients of the best approximation to b using the columns of A.

19:42 Representing a Plane with the Columns of A

In the three-dimensional example, independent vectors a₁ and a₂ form a basis for a plane.
Putting a₁ and a₂ into the columns of A makes that plane equal to Col(A); the same setup extends to n columns.
The projected point is P = A x̂, and the residual e = b - A x̂ is perpendicular to the plane.

24:28 Perpendicular Residuals Produce the Normal Equations

For a plane spanned by a₁ and a₂, the residual must satisfy a₁^T e = 0 and a₂^T e = 0.
Combining those conditions gives A^T(b - A x̂) = 0.
The equation is equivalent to A^T A x̂ = A^T b, with one equation for each column coefficient.

28:00 Connecting the Residual to the Four Fundamental Subspaces

The condition A^T e = 0 places the residual e in the null space of A^T.
The null space of A^T is orthogonal to Col(A), matching the geometric requirement that the error be perpendicular to the projected subspace.
The line-projection equation is the one-column special case of the same normal-equation framework.

32:27 Least-Squares Coefficients and the General Projection Formula

When A has linearly independent columns, A^T A is invertible and x̂ = (A^T A)^-1 A^T b.
Substituting x̂ into P = A x̂ gives P = A(A^T A)^-1A^T b.
The projection matrix is therefore A(A^T A)^-1A^T, extending aa^T/(a^T a) from a line to a higher-dimensional column space.

35:49 Why the Rectangular Matrix A Cannot Be Inverted

A is generally rectangular when there are more equations than unknowns, so A has no ordinary inverse and the factors in (A^T A)^-1 cannot be split apart.
If A is square and invertible, Col(A) is all of R^n, so projection onto that space is the identity matrix.
The higher-dimensional projection matrix remains symmetric and idempotent, as confirmed by transposing it and simplifying its square.

42:00 Setting Up a Least-Squares Line Fit for Three Data Points

The example fits b = C + Dt to the points (1,1), (2,2), and (3,2), which do not lie on a single line.
The desired equations are C + D = 1, C + 2D = 2, and C + 3D = 2, giving A = [[1,1],[1,2],[1,3]], x = [C,D]^T, and b = [1,2,2]^T.
Because the system has three equations and two unknowns, it is inconsistent; the next step is to solve A^T A x̂ = A^T b for the best-fit line.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT OpenCourseWare.