Linear Algebra - Lecture 20 - Fall, 2026
Overview
The lecture develops least-squares solutions for possibly inconsistent systems \(Ax=b\) by projecting \(b\) onto the column space of \(A\). It proves that the minimizing vectors \(x\) are exactly the solutions of the normal equations \(A^T A x=A^T b\), then works through a three-equation, two-variable example whose least-squares solution is \((178/101,47/101)\).
Key takeaways
- For an inconsistent system \(Ax=b\), least squares minimizes the Euclidean norm \(\|b-Ax\|\), or equivalently the sum of squared residual coordinates.
- All attainable outputs \(Ax\) form the column space of \(A\), so the best approximation to \(b\) is its orthogonal projection onto that space.
- The projection residual \(b-Ax\) is orthogonal to every column of \(A\), which yields the normal equations \(A^T A x=A^T b\).
- The projected vector \(Ax\) is unique, but the minimizing \(x\) need not be unique if different input vectors produce the same projection.
- For the example with three equations and two unknowns, the exact system is inconsistent, while its normal equations give \(x=(178/101,47/101)^T\).
Chapters
- A homogeneous system always has the zero solution, but an inhomogeneous system can be inconsistent.
- The equations \(x_1+x_2=1\) and \(x_1+x_2=2\) have no common solution.
- For a candidate \(x\), define the residual as \(b-Ax\) and seek to minimize its Euclidean length, equivalently its sum of squared coordinate errors.
- For an \(m\times n\) matrix \(A\), every vector \(Ax\) lies in \(\mathbb{R}^m\).
- Writing \(Ax\) as \(x_1\) times column 1 through \(x_n\) times column \(n\) shows that its possible values form the column space of \(A\).
- The least-squares task is to find the vector in that column space closest to the target \(b\).
- The stated characterization says \(x\) is a least-squares solution when \(Ax\) is the projection of \(b\) onto the column space \(W\) of \(A\).
- The same set of solutions is described computationally by \(A^T A x=A^T b\).
- There may be multiple minimizing vectors \(x\), even though the closest vector \(Ax\) in the column space is unique.
- Transposing an \(m\times n\) matrix interchanges rows and columns, producing an \(n\times m\) matrix.
- For example, the rows \((1,2,3)\) and \((4,5,6)\) become the columns of the transpose.
- A column vector transposes to a row vector, and transposing twice returns the original matrix.
- For a finite-dimensional subspace \(W\) of an inner-product space, each vector \(v\) has a unique closest vector \(w_0\in W\).
- An orthonormal basis \(e_1,\ldots,e_k\) of \(W\), obtainable using Gram-Schmidt, gives the projection \(w_0=\sum_{i=1}^k\langle v,e_i\rangle e_i\).
- The residual \(v-w_0\) is orthogonal to every basis vector and therefore lies in \(W^\perp\).
- For any other \(w\in W\), the difference \(w-w_0\) lies in \(W\), while \(v-w_0\) lies in \(W^\perp\).
- The Pythagorean theorem gives \(\|v-w\|^2=\|v-w_0\|^2+\|w-w_0\|^2\).
- If \(w\ne w_0\), the second term is positive, proving that \(w_0\) is uniquely closest to \(v\).
- For a least-squares fit, \(\hat b=Ax\) is the projection of \(b\) onto the column space of \(A\), so \(b-\hat b\in W^\perp\).
- Every column of \(A\) lies in \(W\), making its inner product with \(b-Ax\) zero.
- Collecting those column-wise conditions gives \(A^T(b-Ax)=0\), or \(A^T A x=A^T b\).
- The example uses \(3x_1+x_2=7\), \(x_1-2x_2=0\), and \(2x_1+2x_2=3\).
- The first two equations force \((x_1,x_2)=(2,1)\), which makes the third left-hand side \(6\), not \(3\).
- The coefficient matrix is \(A=\begin{bmatrix}3&1\\1&-2\\2&2\end{bmatrix}\) and \(b=(7,0,3)^T\).
- The example yields \(A^T A=\begin{bmatrix}14&5\\5&9\end{bmatrix}\) and \(A^T b=(27,13)^T\).
- Solving the resulting two-equation system gives the least-squares vector \(x=(178/101,47/101)^T\).
- Multiplying by \(A\) gives the closest attainable right-hand-side vector \(Ax=(581/101,84/101,450/101)^T\), which lies in the column space of \(A\).
- The lecture closes the inner-product-spaces chapter and notes that determinants and eigenvalues are upcoming topics.
Transcript excerpt
0:01 So this is lecture 20 on uh what is called best approximation and least squares. So the fundamental problem in linear algebra is solving systems of linear equations. So if you have a system of m equations in n variables a11 x1 up to a1 n xn = b sub1 that's one equation in n variables down to the mth equation a m1 x1 plus a m n xn = b subm. So we have m equations
0:51 in n variables. If the system is homogeneous, which means all the right hand sides are zero, then there is always a solution. Namely, there's the zero solution when you choose each of the n variables. to be zero. But if the system is inhomogeneous,
1:40 which means not homogeneous. That means B subi is different from zero for at least one of the right hand sides. I then the system might have no solution. For example, if you have the equation the system x1 + x2 = 1 and x1 + x2 = 2. There's no solution to this system of equations because [clears throat] no matter what values you pick for x1 and
2:27 x2, x1 + x2 cannot be equal to 1 and 2 simultaneously. So this is an example of a system that is inconsistent. A system with no solution is inconsistent. Nonetheless, nonetheless, you might try to find a set of values for the variables x1 up to xn that is as close as possible to a solution. So, let me say what I mean by that. So for every
3:18 vector x x1 up to xn in Rn, let when you plug these variables into the equations that a11 x1 up to a1 n xn, it's going to be equal to something. Let's say it's equal to some number b1 hat. And for the second
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Nathanson math lectures.