29. Singular Value Decomposition

Overview

Gilbert Strang develops the singular value decomposition, A = UΣV^T, as a factorization of any matrix into two orthogonal changes of basis and a diagonal matrix of nonnegative stretches. He derives right singular vectors from A^T A and left singular vectors from AA^T, then works through a full-rank 2 by 2 example and a rank-one example to connect the SVD to the row space, column space, null space, and left null space.

Key takeaways

Every matrix A can be factored as UΣV^T, where U and V are orthogonal and Σ contains the singular values; the factors generally use different bases.
The eigenvectors of A^T A are the right singular vectors, while the eigenvectors of AA^T are the left singular vectors.
Singular values are nonnegative square roots of the eigenvalues of A^T A, and the number of nonzero singular values equals the rank of A.
For A = [[4, 4], [-3, 3]], the singular values are √32 and √18; the second left singular vector must be -e_2 to reconstruct A with positive singular values.
For the rank-one matrix [[4, 3], [8, 6]], the sole nonzero singular value is 5√5, and its zero singular value represents the null-space direction.
The SVD supplies orthonormal bases for all four fundamental subspaces: the row space, column space, null space, and null space of A^T.

Chapters

0:00 SVD: Two Orthogonal Bases Around a Diagonal Matrix

The singular value decomposition writes a matrix as A = UΣV^T, with orthogonal matrices U and V around diagonal Σ.
Unlike symmetric eigendecomposition, the SVD applies to any matrix and generally requires different bases on the two sides.
For a symmetric positive definite matrix, the two orthogonal factors can be the same eigenvector matrix, and the diagonal entries are positive.

3:00 Choosing Row-Space and Column-Space Bases

The geometric goal is to find orthonormal row-space vectors v_i whose images Av_i point along orthonormal column-space vectors u_i.
Each paired basis vector satisfies Av_i = σ_i u_i, where σ_i measures the stretch produced by A.
Gram-Schmidt can make a basis orthonormal, but it does not by itself ensure that applying A maps one orthogonal basis to another.

8:00 Building the SVD and Accounting for Null Spaces

Collecting the equations Av_i = σ_i u_i gives the matrix factorization A = UΣV^T.
For rank r, the first r vectors in V span the row space and the first r vectors in U span the column space.
Complete V with an orthonormal basis for the null space and U with one for the null space of A^T; the added directions correspond to zero singular values.

13:30 Deriving Right Singular Vectors from A^T A

Taking A^T A cancels U because U^T U = I, leaving A^T A = VΣ^TΣV^T.
The columns of V are eigenvectors of the symmetric positive semidefinite matrix A^T A.
The singular values σ_i are the nonnegative square roots of the eigenvalues of A^T A; AA^T similarly identifies the columns of U.

17:00 Computing Singular Values for a Full-Rank 2 by 2 Matrix

For A = [[4, 4], [-3, 3]], Strang computes A^T A = [[25, 7], [7, 25]].
The normalized eigenvectors (1, 1)/√2 and (1, -1)/√2 have eigenvalues 32 and 18, giving singular values √32 and √18.
The matrix AA^T is diagonal with eigenvalues 32 and 18, so its eigenvectors provide a convenient basis for U.

22:00 Resolving the Sign Choice in the 2 by 2 SVD

Using U = I with the chosen V produces the wrong signs in the reconstructed matrix: the second image is in the negative e_2 direction.
The correct second left singular vector is -e_2, since A(1, -1)/√2 = -√18 e_2.
Eigenvectors are defined up to sign, so U must use a consistent sign choice with V and positive diagonal singular values.

28:00 Rank-One SVD with a Zero Singular Value

For A = [[4, 3], [8, 6]], the rank is one, the row space is spanned by (4, 3), and the column space is spanned by (1, 2).
The unit vectors v_1 = (4/5, 3/5) and u_1 = (1, 2)/√5 represent the nonzero row-space and column-space directions.
A^T A has eigenvalues 125 and 0, so Σ has singular values √125 = 5√5 and 0.
Completing the bases with perpendicular vectors yields orthogonal U and V, while the zero singular value accounts for the null-space direction.

37:00 SVD as Orthonormal Bases for the Four Fundamental Subspaces

The first r right singular vectors form an orthonormal basis for the row space, and the first r left singular vectors form one for the column space.
The remaining n-r vectors in V span the null space; the remaining m-r vectors in U span the null space of A^T.
The dimensions match the rank-nullity relationships: row and column spaces have dimension r, while the two null spaces have dimensions n-r and m-r.
In these bases, A acts independently on each paired direction through a singular value, with null-space directions mapped to zero.

Transcript excerpt

0:14 Okay. This is the lecture on the singular value decomposition. But everybody calls it the SVD. So this is the final and best factorization of a matrix. Let me tell you what's coming. The factors will be, orthogonal matrix, diagonal matrix, orthogonal matrix. So it's things that we've seen before, these special good matrices, orthogonal diagonal. The new point is that we need two orthogonal matrices. A can be any matrix whatsoever. Any matrix whatsoever has this singular value decomposition,

1:07 so a diagonal one in the middle, but I need two different -- probably different orthogonal matrices to be able to do this. Okay. And this factorization has jumped into importance and is properly, I think, maybe the bringing together of everything in this course. One thing we'll bring together is the very good family of matrices that we just studied, symmetric, positive, definite. Do you remember the stories with those guys? Because they were symmetric, their eigenvectors were orthogonal, so I could produce an orthogonal matrix -- this is my usual one.

1:56 My usual one is the eigenvectors and eigenvalues In the symmetric case, the eigenvectors are orthogonal, so I've got the good -- my ordinary s has become an especially good Q. And positive definite, my ordinary lambda has become a positive lambda. So that's the singular value decomposition in case our matrix is symmetric positive definite -- in that case, I don't need two -- U and a V -- one orthogonal matrix will do for both sides. So this would be no good in general, because usually the eigenvector matrix isn't orthogonal.

2:45 So that's not what I'm after. I'm looking for orthogonal times diagonal times orthogonal. And let me show you what that means and where it comes from. Okay. What does it mean? You remember the picture of any linear transformation.

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT OpenCourseWare.