Lecture 11: Hessian matrix; Steepest descent

Overview

Burton Ma explains how Hessian eigenvalues help classify critical points, including cases where a zero Hessian leaves the classification uncertain, then derives steepest descent as movement along the negative gradient. He compares fixed-step gradient descent with Armijo backtracking on a paraboloid, an inverted Gaussian, and the Rosenbrock function, and connects gradient evaluation costs to training neural networks with millions or billions of parameters.

Key takeaways

A positive semidefinite Hessian with eigenvalues 4 and 0 can correspond to an entire line of minima, as in the function whose gradient is 2(w1 + w2) in both coordinates.
A zero Hessian at a critical point is inconclusive: the monkey saddle is a saddle, w1⁴ + w2⁴ has a minimum, and its negative has a maximum.
In Euclidean space, the steepest unit descent direction is the normalized negative gradient, because it makes the directional derivative most negative.
Fixed-step gradient descent can zigzag across narrow valleys; on the Rosenbrock example, plain descent fails to converge within 100 iterations while Armijo backtracking converges in 33.
Armijo backtracking can reduce the number of gradient evaluations because it reuses the direction during step-size trials, but it pays for repeated objective evaluations.
Gradient computation is central to AI model training, where objectives may have millions or billions of parameters and are defined by neural networks rather than simple closed-form equations.

Chapters

0:00 The Taco Function Has a Line of Minima

For the function shaped like a folded sheet, both gradient components are 2(w1 + w2), so stationary points lie on the line w1 = -w2.
Its Hessian has every entry equal to 2, with eigenvalues 4 and 0; the zero eigenvalue makes it positive semidefinite but not invertible.
Every point on the stationary line is a minimum, illustrating that a positive semidefinite Hessian can accompany infinitely many minimizers rather than one isolated point.

3:30 Hessian Eigenvalues, Saddle Points, and Zero-Hessian Ambiguity

For the example with diagonal Hessian entries 1 and -2w2, the stationary points are (0, -1) and (0, 1): the first has a positive definite Hessian and is a local minimum, while the second has an indefinite Hessian and is a saddle.
The function w1² - w2² has a stationary point at (0, 0) with one positive and one negative Hessian eigenvalue, producing a clear saddle.
The monkey saddle has zero gradient and a zero Hessian at the origin, yet is a saddle; the quartic function w1⁴ + w2⁴ has the same zero-Hessian test result but a global minimum there.
A zero Hessian does not determine whether a critical point is a minimum, maximum, or saddle; further analysis is needed, as shown by also negating the quartic to create a maximum.

14:52 Gradient Descent as Iterative Minimization

Gradient descent extends one-dimensional derivative-based line search to vector inputs, repeatedly updating an estimate of the minimizer using a local model of the objective.
A descent direction has a negative directional derivative, which guarantees a sufficiently small step decreases the function; an overly large step can overshoot and cause oscillation.
Momentum-based methods can deliberately continue moving through shallow local minima, much like a rolling ball carrying momentum, though they are outside this course's main treatment.
An algorithm can use backtracking to choose a step size, adding a small step-size optimization within the broader minimization procedure.

22:43 Why the Negative Gradient Is the Steepest Descent Direction

In Euclidean space, the directional derivative is the gradient dotted with a unit direction, or the gradient magnitude multiplied by the cosine of the angle between them.
The derivative is most negative when the direction points opposite the gradient, so the unit steepest-descent direction is -∇f/||∇f||; implementations can use the unnormalized negative gradient instead.
The one-dimensional line-search algorithm generalizes by replacing the scalar derivative with the gradient and the scalar variable with a vector.
Armijo backtracking also generalizes to multiple dimensions; the squared direction term becomes the quadratic form dᵀd.

28:53 Fixed-Step Gradient Descent Oscillates on a Paraboloid

For the example objective with gradient components w1/2 and 2w2, the level curves are elliptical and the minimizer is at (0, 0).
The negative gradient is perpendicular to each contour, not necessarily pointed directly toward the minimizer, so fixed-step updates can zigzag across the valley.
A large fixed step can overshoot and oscillate; a smaller step reduces oscillation but needs many iterations.
Armijo backtracking reaches the paraboloid minimum in 9 iterations in the demonstrated run, though repeated objective evaluations also have a cost.

32:54 Step Sizes on the Inverted Gaussian

The inverted Gaussian has elliptical contours similar to the paraboloid, but its flatter region produces small gradient magnitudes and slow progress.
With the illustrated fixed-step settings, gradient descent does not converge within 100 iterations.
Reducing the step size yields convergence in 17 iterations, while Armijo backtracking adapts the step and converges in 5 iterations.
Backtracking can take larger acceptable steps farther from the minimum and adjust step size as the iterates approach it.

35:06 Rosenbrock's Long, Shallow Valley Challenges Gradient Descent

The Rosenbrock function is a fourth-order polynomial with a long valley that is steep across its walls and shallow along the route to its minimum near (1, 1).
Plain gradient descent can move quickly into the valley, then make slow progress along it and bounce between its steep walls; the illustrated run does not reach the minimum within 100 iterations.
Armijo backtracking smooths the path and converges in 33 iterations, although the shallow valley still makes progress slow.
The contour plot uses a logarithmic scale, so neighboring contour lines represent orders of magnitude rather than equal linear increments.

39:56 Gradient Costs in AI Training and a Preview of Newton's Method

Training large AI models involves minimizing objectives with millions or billions of parameters, making each gradient evaluation computationally expensive.
Armijo backtracking tests step sizes without recomputing the gradient at every inner-loop attempt, but it does require repeated objective-function evaluations.
Neural-network objectives are not usually available as simple equations, so efficient methods for computing their gradients are essential to model training.
The next lecture introduces Newton's method, which can find the minimum of a paraboloid in one iteration when it works.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Burton Ma.