Queen's University CISC 371
Professor Ma · Queen's University · 9 lectures with notes
Students in this class: ask your lecturer for the class code, and these lectures will already be in your library when you sign up.
Lecture 1: Introduction
Burton Ma connects the history and mathematics of optimization to machine learning, then demonstrates grid search in MATLAB.
Lecture 2: Optimization via quadratic approximation
Quadratic fitting turns three function evaluations into an iterative line search that narrows a bracket around a local minimum.
Lecture 3: Optimization via quadratic approximation (cont); Stationarity
Quadratic interpolation narrows a minimum bracket; derivative tests distinguish minima from other stationary points.
Lecture 4: Stationarity (cont); Line search algorithms
Convexity supports reliable line searches, from finding a minimum-containing bracket to shrinking it efficiently with dichotomous search.
Lecture 5: Line search
Derivative-driven line searches trade fewer iterations for sensitivity to step size and convergence settings.
Lecture 6: Armijo condition
Armijo backtracking automatically shrinks a step until it achieves a sufficient decrease in the objective function.
Lecture 7: Multivariate calculus
Burton Ma develops the vector, partial-derivative, and directional-derivative notation required for high-dimensional optimization.
Lecture 8: Multivariate calculus (cont); Proof of second-order necessary condition for optimality
A little-o Taylor remainder shows why a smooth local minimum with zero slope must have nonnegative second derivative.
Lecture 10: Stationarity (multivariate)
A zero gradient identifies candidate extrema; the Hessian’s definiteness determines whether they are minima, maxima, or saddle points.