CPBS 7620 - Lecture 15 - More on the Frechet Derivative

Overview

Nicholas Dwork derives Fréchet derivatives for linear maps and quadratic forms, showing that the gradient of f(x) = xᵀAx is (A + Aᵀ)x and simplifies to 2Ax when A is symmetric. He applies these results to minimize ||Ax − b||², deriving the normal equations and the full-column-rank least-squares solution x = (AᵀA)⁻¹Aᵀb.

Key takeaways

The Fréchet derivative of f(x) = aᵀx is the Jacobian row aᵀ, while its gradient is the column vector a.
A linear map f(x) = Ax has constant Fréchet derivative A, independent of x.
For f(x) = xᵀAx, the gradient is (A + Aᵀ)x; symmetry of A gives the familiar 2Ax formula.
The quadratic remainder δxᵀAδx vanishes relative to ||δx|| because its magnitude is bounded by ||A||·||δx||².
Minimizing ||Ax − b||² produces the normal equations AᵀAx = Aᵀb and, for full-column-rank A, the pseudoinverse solution (AᵀA)⁻¹Aᵀb.

Chapters

0:00 Fréchet Derivatives of Linear Maps and Quadratic Forms

For f(x) = aᵀx from ℝⁿ to ℝ, the derivative is the row vector aᵀ and the gradient is a.
For f(x) = Ax from ℝⁿ to ℝᵐ, the Fréchet derivative is the matrix A.
For f(x) = xᵀAx, expanding around x shows the linear terms give derivative xᵀA + xᵀAᵀ; the remaining term δxᵀAδx is quadratic in δx.

1:40 Verifying the Quadratic Gradient with a Fréchet Limit

The quadratic-form remainder is δxᵀAδx; Cauchy–Schwarz bounds its magnitude by ||δx||·||Aδx||.
Using ||Aδx|| ≤ ||A||·||δx|| makes the remainder ratio at most ||A||·||δx||, which tends to zero.
Transposing the derivative gives ∇f(x) = (A + Aᵀ)x; for symmetric A, this reduces to 2Ax.

4:20 Least-Squares Critical Point and the Normal Equations

For f(x) = ||Ax − b||² with tall, skinny, full-column-rank A, expansion gives xᵀAᵀAx − 2bᵀAx + bᵀb.
Differentiation yields ∇f(x) = 2AᵀAx − 2Aᵀb; setting it to zero gives the normal equations AᵀAx = Aᵀb.
Since A has full column rank, the minimizer is x = (AᵀA)⁻¹Aᵀb = A†b, and convexity makes this critical point a minimum.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Nicholas Dwork.