Lecture 5: Line search

Overview

Burton Ma explains bracket-based dichotomous line search and derivative-driven one-dimensional searches, showing how step size affects evaluation cost, accuracy, and convergence. He compares dichotomous search with Fibonacci and golden-section variants, then demonstrates fixed-step and derivative-scaled updates, stopping criteria, and why an oversized step can cause divergence.

Key takeaways

Dichotomous line search narrows a bracket by comparing two nearby interior points, while Fibonacci search can reduce a bracket to 1% of its initial width in 10 function evaluations.
A fixed-step search using only derivative signs brackets the minimum when the sign flips; taking the midpoint of the crossing points yields an error bound of half the step size.
Larger fixed steps save derivative evaluations but worsen the possible minimizer error, while smaller steps improve precision at the cost of more evaluations.
Scaling steps by derivative magnitude can make progress faster when far from a minimum, but a large scale can cause oscillation or divergence.
A practical stopping rule can combine a derivative-magnitude threshold such as |f′(t)| < ε with a maximum iteration count, illustrated with a cap of 100.
Starting point and step-size choices are consequential: an oversized update can increase the objective repeatedly and may lead a search to a different local minimum.

Chapters

0:00 Dichotomous Line Search: Narrowing a Bracket Around the Minimum

Burton Ma illustrates dichotomous search on a parabola with a known minimum at t = 0.
The algorithm evaluates two nearby points near the interval midpoint, then removes the side that cannot contain the minimum.
The demonstration repeats interval reduction until the bracket reaches the desired width; Jupyter with NumPy and MATLAB implementations are mentioned.

4:08 Fibonacci and Golden-Section Search Reduce Function Evaluations

Fibonacci and golden-section line searches retain one interior function evaluation between iterations, unlike dichotomous search.
Burton Ma gives Fibonacci search as requiring 9 iterations and 10 total evaluations to reduce the bracket to 1% of its initial width.
Dichotomous search would need about 14 evaluations for that reduction, while quadratic interpolation generally needs fewer evaluations but requires solving a system of equations.

5:36 Using Derivative Signs to Choose a Search Direction

A derivative sign can replace bracketing information: a negative derivative means move right, while a positive derivative means move left.
The search direction combines a sign with a step magnitude; Burton Ma uses the sign function to choose whether to move left or right.
The approach requires a derivative or a numerical approximation of one and assumes a starting point reasonably close to the minimum.

7:44 Fixed-Step Search: Accuracy Depends on Step Size

The fixed-step method repeatedly moves by a constant positive step size in the direction opposite the derivative sign.
A derivative sign change indicates that the minimum has been crossed; the midpoint of the two points gives an estimate with error no greater than half the step size.
Large steps reduce derivative evaluations but can produce a poor estimate: a step of 3 in the example returns 1 instead of the true minimum at 0.
Small steps can improve accuracy but require more derivative evaluations.

14:21 Scaling Fixed Steps by the Derivative Can Overshoot

Instead of using only the derivative sign, the update can scale a fixed parameter by the derivative magnitude: larger slopes produce larger steps.
For f(t) = t² + 1, the derivative is 2t; starting at t = 2.5 gives derivative 5, so a scale of 0.75 produces an oversized initial move.
Derivative-scaled steps may shrink near a minimum, but this behavior is not guaranteed for every function or starting point.
In the example, updates oscillate around t = 0 before approaching it, showing why overshoot and stopping rules matter.

21:00 Stopping Rules and MATLAB Implementation Choices

Burton Ma's MATLAB example stops when the derivative magnitude falls below epsilon and also imposes a maximum of 100 iterations.
Other possible stopping rules include detecting a derivative sign change, limiting iterations, or checking whether the objective value changes only slightly.
A sign change alone may stop too early after an overshoot; some one-dimensional searches used inside higher-dimensional optimization need only an approximate minimizer.
The example needs a derivative, starting estimate, step size, and convergence settings, though the objective function itself is not required by that implementation.

23:12 Oversized Steps Cause Divergence and Make Tuning Necessary

With f(t) = ½t², the derivative is t and a derivative-scaled update is tₖ₊₁ = tₖ − s₀tₖ; a step parameter that is too large can move away from the minimum.
For a symmetric function, an overshoot can increase the objective value at each iteration and rapidly drive the search farther from the minimum.
The algorithm’s outcome depends on the starting point, step size, and convergence parameters; Burton Ma emphasizes that convergence is not guaranteed.
On functions with multiple minima, a diverging trajectory may enter another basin and converge to a different local minimum.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Burton Ma.