14. Orthogonal Vectors and Subspaces

Overview

Orthogonality is defined through the dot product: real vectors x and y are perpendicular exactly when xᵀy = 0, a condition connected to the Pythagorean theorem. The lecture applies this definition to subspaces, showing that the row space and null space of A are orthogonal complements, then introduces AᵀA as the symmetric matrix behind the next lecture’s approach to inconsistent systems and least squares.

Key takeaways

For real vectors, x and y are orthogonal exactly when xᵀy = 0; expanding the squared length of x + y links this test directly to the Pythagorean theorem.
Two subspaces are orthogonal only when every vector in one is perpendicular to every vector in the other; sharing any nonzero vector rules out orthogonality.
The null space of A is the orthogonal complement of its row space: Ax = 0 says that x is perpendicular to every row and therefore to every linear combination of rows.
The row space and null space have dimensions r and n − r, so they form complementary subspaces of Rⁿ; for a rank-one row direction in R³, the null space is a perpendicular plane.
For inconsistent or noisy measurements, the normal equations AᵀAx̂ = Aᵀb provide the lecture’s proposed route to a best-fit solution using all equations.
AᵀA is symmetric, has the same rank and null space as A, and is invertible exactly when A has independent columns.

Chapters

0:00 Orthogonality and the Pythagorean Test for Vectors

Orthogonal vectors meet at a 90-degree angle, and the Pythagorean theorem connects that geometry to their lengths.
The algebraic test is xᵀy = 0, where the dot product sums corresponding component products.
For real vectors, expanding (x + y)ᵀ(x + y) shows that the Pythagorean identity holds exactly when the cross terms xᵀy vanish.

5:30 Vector Lengths, Dot Products, and a Three-Dimensional Example

A vector’s squared length is xᵀx = x₁² + … + xₙ²; for x = (1, 2, 3), the squared length is 14.
The example uses y = (2, -1, 0), for which xᵀy = 0 and yᵀy = 5.
Since x + y = (3, 1, 3) has squared length 19, the example verifies 14 + 5 = 19 for perpendicular vectors.

13:20 Defining Orthogonal Subspaces with Every Vector Pair

Subspaces S and T are orthogonal only if every vector in S is orthogonal to every vector in T.
The blackboard plane and floor plane in R³ are not orthogonal: their intersection contains a nonzero direction along the crack.
In R², two distinct lines through the origin are orthogonal when they meet at 90 degrees; every subspace is orthogonal to the zero subspace.

20:40 Why the Row Space of A Is Orthogonal to Its Null Space

If x is in the null space, Ax = 0; each row of A therefore has dot product zero with x.
Every vector in the row space is a linear combination of A’s rows, so it also has dot product zero with x.
This establishes that the row space and null space of A are orthogonal subspaces.

27:00 Orthogonal Complements and the Dimensions of A’s Subspaces

The same row-space and null-space argument applied to Aᵀ establishes orthogonality between the column space and left null space.
The row space and null space have dimensions r and n − r, so together they account for all of Rⁿ.
For a rank-one matrix with row direction (1, 2, 5), the null space is the two-dimensional plane perpendicular to that row direction.

34:10 Why Inconsistent Ax = b Systems Still Need a Best Solution

When b is outside the column space of A, the system Ax = b has no exact solution.
Overdetermined measurements, such as 1,000 satellite observations used to estimate roughly six parameters, generally cannot all be fit exactly.
Measurement noise creates inconsistent right-hand sides, so the goal is to use all observations rather than discard equations to force a square system.

39:25 AᵀA and the Normal Equations for an Approximate Solution

For an m by n matrix A, AᵀA is an n by n square matrix and is symmetric.
The proposed equation for the best-fit estimate x̂ is AᵀAx̂ = Aᵀb, obtained by multiplying Ax = b by Aᵀ.
The lecture identifies AᵀA as the key matrix for the next chapter’s treatment of systems with no exact solution.

43:05 Computing AᵀA for a Full-Column-Rank Example

For A with rows (1, 1), (1, 2), and (1, 5), the matrix has three equations and two unknowns, and its columns are independent.
Its Gram matrix is AᵀA = [[3, 8], [8, 30]], which is symmetric and invertible.
An exact solution Ax = b exists only when b lies in A’s two-dimensional column space in R³, which is not true for a typical b.

45:55 When AᵀA Is Invertible: Independent Columns

The lecture states that AᵀA and A have the same rank and the same null space.
AᵀA is invertible exactly when its null space contains only the zero vector, which occurs exactly when A’s columns are independent.
A rank-one A produces a singular AᵀA, illustrating why full column rank is necessary for an invertible normal-equation matrix.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT OpenCourseWare.