CPBS 7620 - Lecture 14 - Frechet Derivative

Overview

The Fréchet derivative generalizes the scalar derivative to functions from ℝⁿ to ℝᵐ: it is the Jacobian matrix whose linear approximation error, divided by the input-change norm, tends to zero. The lecture connects Jacobian entries to partial derivatives and shows how gradients encode directional change, steepest ascent, and the update rule for gradient descent.

Key takeaways

The Fréchet derivative A of f: ℝⁿ → ℝᵐ is characterized by a residual error that vanishes relative to ‖Δx‖, not merely by requiring the approximation error to approach zero.
The Jacobian has m rows and n columns, with entry (i, j) equal to the partial derivative of output component fᵢ with respect to input coordinate xⱼ.
A directional derivative is the ordinary one-dimensional derivative of the restricted curve g(t) = f(x + tu) at t = 0.
For a scalar-valued function and a unit direction u, the rate of change is ∇f(x) · u; it is maximized when u points along the gradient.
Gradient descent uses xₖ₊₁ = xₖ − α∇f(xₖ), and an overly large step size can impair convergence, motivating backtracking line search.

Chapters

0:00 From Scalar Derivatives to the Fréchet Derivative

For a scalar function, the derivative gives a local linear approximation: f(x + Δx) ≈ f(x) + f′(x)Δx.
For f: ℝⁿ → ℝᵐ, the linear approximation must multiply an n-dimensional input change and return an m-dimensional output change, so its matrix has shape m × n.
A direct vector analogue of the scalar difference quotient fails because a vector in ℝᵐ cannot be divided by a vector in ℝⁿ.
The Fréchet derivative is a matrix A for which ‖f(x + Δx) − f(x) − AΔx‖ / ‖Δx‖ tends to zero as Δx tends to zero; this stricter condition rules out arbitrary matrices.

3:09 Jacobian Entries, Partial Derivatives, and Directional Change

For f: ℝⁿ → ℝᵐ, the Fréchet derivative is the Jacobian matrix; for a scalar-valued function, its derivative is represented by the transposed gradient.
Restricting f to a line through x in direction u gives g(t) = f(x + tu); g′(0) is the directional derivative of f at x along u.
When u is a standard basis vector eᵢ, the directional derivative becomes the partial derivative with respect to coordinate xᵢ.
The Jacobian entry in row i and column j is ∂fᵢ/∂xⱼ, describing how output component i changes when input coordinate j changes.

6:30 Gradient Geometry and the Gradient Descent Update

For scalar-valued f, the directional derivative along u equals ∇f(x) · u; for a unit direction, it is ‖∇f(x)‖ cos θ.
The gradient points in the direction of steepest ascent, and its norm gives the maximum directional derivative over unit directions.
Gradient descent moves opposite the gradient using xₖ₊₁ = xₖ − α∇f(xₖ), where α > 0 is the step size.
An excessively large α can prevent convergence; backtracking line search, such as the Armijo rule, can adapt the step size.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Nicholas Dwork.