CPBS 7620 - Lecture 10 - Pseudoinverse

Overview

Nicholas Dwork explains the pseudoinverse as a way to solve least-squares problems when a matrix is not invertible: project B onto the image of A, then choose the minimum-norm X that maps to that projection. He derives the full-rank formulas for tall and skinny and short and fat matrices, recommends numerically stable software such as MATLAB's lsqminnorm and NumPy's numpy.linalg.lstsq, and applies least squares to warfarin-dose prediction.

Key takeaways

Least squares finds the point in image(A) closest to B, so its residual is orthogonal to the columns of A.
When A has a nontrivial null space, adding any vector from null(A) preserves AX; the pseudoinverse resolves this ambiguity by choosing the solution with the smallest L2 norm.
For full-column-rank tall matrices, A⁺ = (AᵀA)⁻¹Aᵀ; for full-row-rank wide matrices, A⁺ = Aᵀ(AAᵀ)⁻¹.
The normal-equation formulas are useful for derivation but can be numerically unstable; MATLAB's lsqminnorm and NumPy's numpy.linalg.lstsq provide more stable computational methods.
Warfarin-dose regression represents patient characteristics as numerical features and fits a coefficient vector, with predicting the square root of dosage working better than predicting dosage directly.

Chapters

0:00 Least Squares as Projection onto the Image of A

For invertible A, the solution is X = A⁻¹B, which makes the residual AX − B equal zero.
For non-invertible A, least squares minimizes the L2 norm of AX − B by projecting B onto the image of A.
If A has a nontrivial null space, adding any null-space vector to a minimizer leaves AX unchanged, so infinitely many least-squares solutions can exist.
The pseudoinverse selects the minimum-norm solution, a useful choice when large inputs have a cost such as spacecraft thruster fuel.

5:33 Pseudoinverse Formulas for Tall and Wide Full-Rank Matrices

For tall, skinny, full-column-rank A, the residual B − AX* lies in the null space of Aᵀ, yielding the normal equations AᵀAX* = AᵀB.
The tall-matrix pseudoinverse is A⁺ = (AᵀA)⁻¹Aᵀ, so X* = A⁺B.
For short, fat, full-row-rank A, every B is reachable, but the null space of A creates multiple solutions; orthogonality between image(Aᵀ) and null(A) makes the minimum-norm solution lie in image(Aᵀ).
The short-matrix pseudoinverse is A⁺ = Aᵀ(AAᵀ)⁻¹, giving the minimum-norm exact solution X* = A⁺B.

11:35 Stable Pseudoinverse Software and Warfarin-Dose Regression

Although full-rank formulas are analytic, Dwork warns that they are numerically unstable; he names MATLAB's lsqminnorm and Python's numpy.linalg.lstsq as practical alternatives.
The lecture identifies linear regression, system identification, experiment planning, and data fusion as applications of least squares.
A warfarin study encodes patient age, height, weight, gene variants, race, and medication use as features, then fits coefficients to predict stable dosage.
The initial linear model predicted dose poorly; changing the target to the square root of dose improved the fit, allowing dosage estimates from a new patient's feature vector.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Nicholas Dwork.