Linear Algebra for Engineers lecture 30: Orthogonality and Dot Products
Overview
Alex Townsend reviews how diagonalization solves linear differential systems, then develops dot products as a way to define orthogonality and vector length. He proves the higher-dimensional Pythagorean theorem, shows that nonzero orthogonal vectors are linearly independent, and derives a formula for coordinates in an orthogonal basis that avoids solving a general linear system.
Key takeaways
- For any vector u, the dot product with itself gives its squared length: uᵀu = ‖u‖² = Σᵢuᵢ².
- Orthogonal vectors satisfy the higher-dimensional Pythagorean identity ‖u + v‖² = ‖u‖² + ‖v‖² because both cross terms are zero.
- Any set of mutually orthogonal, nonzero vectors is linearly independent: dotting a zero linear combination with uᵢ isolates its coefficient through the positive value ‖uᵢ‖².
- For an orthogonal basis, the coefficient of uᵢ in b is aᵢ = (uᵢᵀb)/‖uᵢ‖², so coordinates can be found without solving a general linear system.
- With m-entry basis vectors and n coefficients, the orthogonal-basis calculation takes n(4m − 1) basic operations, using two dot products and one division per coefficient.
- The zero vector is orthogonal to every vector, but it must be excluded from an orthogonal basis because its presence prevents linear independence.
Chapters
- For a system x'(t) = Ax(t), diagonalizing A changes to coordinates where the differential equations decouple.
- The solution combines the eigenvector matrix, exponentials of eigenvalues, and the initial condition.
- Real eigenvalues produce growing, shrinking, or constant exponential behavior; complex-conjugate eigenvalues can produce oscillations while the solution remains real.
- For vectors u and v in R^n, the dot product is u₁v₁ + … + uₙvₙ.
- The matrix expression uᵀv produces the same scalar as the componentwise sum, and uᵀv = vᵀu.
- A zero dot product means the vectors are orthogonal, or perpendicular.
- For u = (u₁, …, uₙ), its length is ‖u‖ = √(u₁² + … + uₙ²).
- The same length formula is ‖u‖ = √(uᵀu), so the dot product generalizes the Pythagorean calculation to any dimension.
- The squared length uᵀu is zero only for the zero vector.
- When uᵀv = 0, placing u and v tip to tail forms a right-angle triangle, even when the vectors lie in a higher-dimensional space.
- Expanding (u + v)ᵀ(u + v) gives uᵀu + uᵀv + vᵀu + vᵀv.
- The cross terms vanish for orthogonal vectors, yielding ‖u + v‖² = ‖u‖² + ‖v‖².
- A set {u₁, …, uₙ} is orthogonal when uᵢᵀuⱼ = 0 for every pair of distinct indices i and j.
- The zero vector is orthogonal to every vector, so an orthogonal set can contain zero vectors.
- For basis construction, Townsend excludes zero vectors because they cannot contribute to a linearly independent set.
- Assume a linear combination c₁u₁ + … + cₙuₙ equals zero, with the uᵢ nonzero and mutually orthogonal.
- Take the dot product with a chosen uᵢ; orthogonality eliminates every term except cᵢuᵢᵀuᵢ.
- Since uᵢᵀuᵢ = ‖uᵢ‖² > 0, cᵢ must be zero; repeating for each index proves linear independence.
- A nonzero orthogonal set that spans a vector space V is automatically a basis because it is already linearly independent.
- For b in V, finding coefficients in b = a₁u₁ + … + aₙuₙ is a general linear-system problem when the basis has no useful structure.
- Townsend contrasts this with the standard basis, where the coordinates can be read directly from the vector.
- Take the dot product of b = Σ aⱼuⱼ with uᵢ; mutual orthogonality removes every term except aᵢuᵢᵀuᵢ.
- The coordinate is aᵢ = (uᵢᵀb)/(uᵢᵀuᵢ) = (uᵢᵀb)/‖uᵢ‖².
- Both the numerator and denominator must first be computed as scalars; vector or matrix factors cannot be canceled like ordinary numbers.
- For vectors with m entries, one dot product takes 2m − 1 operations: m multiplications and m − 1 additions.
- Computing one coefficient uses two dot products and one division, for 4m − 1 operations; computing n coefficients costs about n(4m − 1).
- The coordinate formula avoids row reduction of a general system, a major savings for large dimensions and applications such as machine learning.
- In the discussion, Townsend clarifies that V can be an n-dimensional subspace of R^m.
- He notes that orthogonality ideas can also apply to polynomials, while polynomial vectors are unlikely to return as a major topic.
- Townsend explains that the 4m − 1 count adds the two dot-product costs and one division.
Transcript excerpt
0:02 Um okay so the joy and excitement today is uh in lecture 30 section 6.1 of the book. Uh it's actually I actually think this chapter is relatively easy though it's more like on the ab slight abstract size but they're easy concepts. So you'll see anyway I just wanted to recap this formula because I thought it was kind of nice to do that. So we've been we have looked at in the last before the semi-finals we were looking at uh systems of uh rates of change right on the left here are some quantities and there's the rate of change of those quantities over time and those rates of change are connected to the quantities themselves by this matrix A. So this a
0:50 right does a linear combination of these everything gets combined together with this a multiplying by this vector and it makes up the rate of change. We have some initial conditions telling us where these quantities are initially right you might think about the steroid example where n was two we had steroid in the muscle steroid in the blood and they were interacting through a matrix and our initial condition was how much steroid was in the muscle and the blood at time zero. Okay, so if a is diagonalizable then we know a formula. We can basically instantly write down once we have the igen value decomposition a formula for the solution. So this is the formula right the the nice thing about diagonalization is it decouples the all the x's right it
1:38 decouples these variables. So we change basis to a basis that depends on vectors. By changing bases, we can solve some decoupled differential equations and they end up being exponentials like this. These are the solutions to these decoupled problem this decoupled
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Alex Townsend.