17. Orthogonal Matrices and Gram-Schmidt

Overview

An orthonormal matrix Q has mutually perpendicular unit columns, so QᵀQ = I; when Q is square, Qᵀ is also its inverse. The lecture shows how this simplifies projections and least-squares calculations, then derives Gram-Schmidt orthogonalization and its matrix form, the QR factorization A = QR, where R is upper triangular.

Key takeaways

For any matrix Q with orthonormal columns, QᵀQ = I; if Q is square, this also means Q⁻¹ = Qᵀ.
The projection matrix onto the columns of an orthonormal Q is QQᵀ, which is symmetric and idempotent.
Least-squares fitting with orthonormal columns requires no inversion: the coefficient vector is x̂ = Qᵀb.
Gram-Schmidt orthogonalizes a new vector by subtracting its projections onto all previously constructed orthogonal vectors, then normalization makes the result unit length.
For a = (1,1,1) and b = (1,0,2), Gram-Schmidt gives orthogonal vectors (1,1,1) and (0,−1,1), which normalize using √3 and √2.
The QR factorization A = QR preserves A's column space, with Q orthonormal and R upper triangular.

Chapters

0:00 Orthogonal Bases and the Meaning of Orthonormal

An orthogonal basis consists of vectors q₁ through qₙ whose pairwise dot products are zero.
Normalizing each vector to length 1 makes the basis orthonormal.
Writing the basis vectors as columns of Q sets up the central identity QᵀQ = I.

3:17 Why Orthonormal Columns Give QᵀQ = I

Each entry of QᵀQ is a dot product between two columns of Q.
Orthonormal columns produce 1s on the diagonal and 0s off the diagonal, giving the identity matrix.
Q can be rectangular when its columns form an orthonormal set without spanning the entire ambient space.
For square Q, QᵀQ = I implies Qᵀ = Q⁻¹.

8:04 Permutation, Rotation, and Hadamard Matrix Examples

Permutation matrices have standard unit vectors as columns, so they are orthogonal matrices.
A 2D rotation matrix with columns (cos θ, sin θ) and (−sin θ, cos θ) has orthonormal columns.
The matrix with rows (1, 1) and (1, −1) becomes orthogonal after dividing by √2.
A 4-by-4 matrix of 1s and −1s can be normalized by dividing by 2; such constructions are known as Hadamard matrices.

14:02 Rectangular Orthonormal Matrices and Column-Space Bases

A rectangular Q can have two orthonormal columns in four-dimensional space, spanning a two-dimensional subspace.
The lecture constructs vectors with entries 1, 2, and 2, whose length is 3, illustrating normalization by dividing by 3.
Orthonormal vectors are automatically linearly independent because perpendicular directions cannot cancel in a nontrivial combination.
A rectangular orthonormal matrix provides a convenient basis for its column space.

17:42 Projection onto a Subspace with Q Qᵀ

The general projection formula simplifies for orthonormal columns: the projection onto Col(Q) is QQᵀ.
When Q is square, its columns span the whole space, so QQᵀ = I.
For rectangular Q, QQᵀ is symmetric and idempotent: (QQᵀ)² = QQᵀ.
Idempotence follows directly from the middle factor QᵀQ = I.

23:02 Orthonormal Bases Simplify Least-Squares Equations

Replacing A with an orthonormal-column matrix Q turns the normal equation AᵀA x̂ = Aᵀb into QᵀQ x̂ = Qᵀb.
Since QᵀQ = I, the least-squares coefficients are x̂ = Qᵀb, with no matrix inversion required.
Each coefficient is a dot product qᵢᵀb, the component of b along the corresponding orthonormal basis vector.

25:04 Gram-Schmidt Removes the Projection onto Earlier Vectors

Starting with independent vectors a and b, keep a as the first orthogonal vector and subtract b's projection onto a.
The second vector is b̃ = b − [(aᵀb)/(aᵀa)]a.
Taking the dot product with a verifies the result: aᵀb̃ = 0.
Normalize the resulting orthogonal vectors by dividing each by its length to obtain q₁ and q₂.

34:05 Extending Gram-Schmidt to a Third Vector

For independent a, b, and c, first form orthogonal vectors A and B from a and b.
Construct C by subtracting c's projection onto both established directions: C = c − [(Aᵀc)/(AᵀA)]A − [(Bᵀc)/(BᵀB)]B.
The resulting C is perpendicular to A and B; dividing A, B, and C by their lengths gives an orthonormal set.
The projection coefficients use the orthogonal vectors already constructed, not the original nonorthogonal inputs.

38:10 Gram-Schmidt Example with (1,1,1) and (1,0,2)

For a = (1,1,1) and b = (1,0,2), the projection coefficient is (aᵀb)/(aᵀa) = 3/3 = 1.
Subtracting a from b gives the orthogonal vector b̃ = (0,−1,1), which has zero dot product with a.
Normalizing gives q₁ = (1,1,1)/√3 and q₂ = (0,−1,1)/√2.
The original vectors and the orthonormal vectors span the same plane, so they have the same column space.

44:09 QR Factorization Encodes Gram-Schmidt in Matrix Form

Gram-Schmidt applied to a matrix A with independent columns produces A = QR.
Q contains the orthonormal columns, while R records the coefficients expressing A's columns in that basis.
R is upper triangular because each later q vector is constructed perpendicular to earlier input columns.
The factorization preserves the column space: Col(A) = Col(Q).

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT OpenCourseWare.