Linear Algebra for Engineers lecture 33: More on orthogonality
Overview
Alex Townsend reviews Gram-Schmidt orthogonalization and develops the QR factorization, showing how an independent set of columns becomes an orthonormal matrix Q times an upper-triangular matrix R. He also derives the projection formula for a subspace with an orthonormal basis, connects it to projection onto a line, and distinguishes the scalar u^T u from the rank-one matrix uu^T.
Key takeaways
- Gram-Schmidt converts linearly independent columns into orthogonal columns, and normalizing those columns supplies Q in A = QR.
- For an m by n matrix A with independent columns, Q is m by n with orthonormal columns and R is n by n upper triangular.
- QR factors are not inherently unique: flipping a column of Q and the corresponding row of R leaves their product unchanged.
- Projection onto the line spanned by u is u(u^T v)/(u^T u); for an orthonormal basis matrix U, projection onto its span simplifies to UU^T v.
- The matrix uu^T is rank one and n by n, whereas u^T u is a scalar; confusing these products leads to invalid cancellation.
- Orthogonal projection is the central mechanism behind least-squares regression, the application Townsend previews for the following lecture.
Chapters
- The final exam is scheduled for December 13, a Sunday, with 50 minutes to work and an additional 30 minutes for submission.
- The exam has three questions: one true-or-false question followed by two standard problems.
- Exam emphasis follows lecture emphasis; practice exams include other topics, so students should select relevant problems on Gram-Schmidt, QR, eigenvalues and eigenvectors, and differential equations.
- Gram-Schmidt processes the basis from left to right, retaining the first vector unchanged and removing its component from each later vector.
- For a1 = (1, 1, 0, 0) and a2 = (0, 1, 1, 0), the projection coefficient is (a1^T a2)/(a1^T a1) = 1/2.
- The resulting second vector is u2 = (-1/2, 1/2, 1, 0), which is orthogonal to u1 = a1; their dot product is zero.
- Townsend compares QR to elimination's PA = LU: both factorizations store a procedure in matrices for later use.
- For an m by n matrix A with linearly independent columns, QR writes A = QR, with Q having orthonormal columns and R an n by n upper-triangular matrix.
- Q need not be square when m differs from n, so it has orthonormal columns without necessarily being an orthogonal matrix.
- The Gram-Schmidt output columns are orthogonal but not yet unit length; dividing each by its norm produces the orthonormal columns of Q.
- For the example, the orthogonal columns are (1, 1, 0, 0) and (-1/2, 1/2, 1, 0), with lengths √2 and √(3/2).
- The coefficients expressing the original columns in the orthonormal basis form R; the resulting 2 by 2 matrix is upper triangular.
- Even a simple 4 by 2 example involves square roots and substantial arithmetic, so larger hand-calculated QR problems quickly become cumbersome.
- Every matrix with linearly independent columns has a QR factorization: Gram-Schmidt creates orthogonal columns, then normalization makes them orthonormal.
- Dependent columns can make Gram-Schmidt fail because a new residual can be zero, preventing the normalization step.
- QR is not unique: negating a column of Q and the corresponding row of R preserves A = QR.
- Applying Gram-Schmidt in the fixed input-column order yields a consistent factorization under a fixed sign convention.
- R records the linear combinations that reconstruct the original columns from the orthonormal columns of Q.
- Townsend explains R as undoing the transformations of Gram-Schmidt and the subsequent column scaling, much like elimination factors encode row operations.
- A computer computes QR with an algorithm related to Gram-Schmidt, though practical implementations use a numerically better method.
- QR belongs to a broader pattern of factorizations: PA = LU comes from elimination, A = QR from orthogonalization, and A = PDP⁻¹ from eigenvalues and eigenvectors.
- For a subspace W with orthonormal basis columns u1 through un, place those vectors in a matrix U.
- The orthogonal projection of v onto W is UU^T v, which computes the component of v lying in the subspace.
- For n = 2, W is a plane spanned by two perpendicular unit vectors; the projection represents the geometric shadow of v on that plane.
- Projection onto a line spanned by a non-unit vector u is u(u^T v)/(u^T u).
- Rearranging the scalar factors gives u(u^T/(u^T u))v; when u has unit length, u^T u = 1 and the expression becomes uu^T v.
- For a basis matrix U, U^T U = I when its columns are orthonormal, so the projection generalizes to UU^T v.
- The products u^T u and uu^T have different shapes: the former is a scalar, while the latter is an n by n rank-one matrix.
- Townsend identifies Gram-Schmidt, QR, and the orthonormal-basis projection formula as the final examinable material.
- The next lecture will apply projection to regression, especially the least-squares method.
- The course connects orthogonal projection to fitting data by finding the closest vector in a chosen subspace.
Transcript excerpt
0:00 Please restart my computer. Anyway, I think we have it working. Okay, so uh the joy and excitement is lecture uh 32, but we swapped that out for 33, right? So, it's just a direct translation from that. Let me pin. Okay, so um right, I wanted to talk about the final exam too. So, the Okay, the final exam is on the 13th of December. Uh that's a Sunday. It's at the morning and the evening session is here. They last 50 minutes but in addition to the 50 minutes you still get the 30 minute submission time. Okay. So there are uh three questions and um there will be one true false question which is the first question
0:46 then two normal questions. Um they're based on one of the lectures these lectures and these sections of the book. the emphasis on the exam matches the emphasis in the lectures. So um right so there's like section 5.6 six. We didn't we it's still useful. It's difference equations, but we haven't emphasized it a lot in lecture. So, um our emphasis matches the emphasis on the electric. Okay. Do you guys have any questions about the final that I can answer now? It's exactly the same as uh the the morning. Yeah. So today today is the last day for content that count that will be examined
1:37 on the final exam. So we're we're six days out, right? So yeah. Yeah. Today's the last day after today that that material is only tested on the homework. The homework will be due on the last day of class on the Wednesday. That's the best we got. Okay. Um yes um the semi-final results I'm hoping to release them today. Uh there's one question left to grade uh left that's partly graded but needs to be completely
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Alex Townsend.