CPBS 7620 - Lecture 4 - The Derivative

Overview

Nicholas Dwork develops the derivative as the slope of a local linear approximation, emphasizing the formula f(x + delta x) approximately equals f(x) + f'(x) delta x as a foundation for applied mathematics. He connects derivatives to convexity and extrema, then demonstrates numerical methods including derivative descent with line search, Newton's method for finding roots and extrema, and bracketed Newton with bisection as a safeguard.

Key takeaways

The derivative is the limiting slope of nearby secant lines and supports the local approximation f(x + delta x) approximately equals f(x) + f'(x) delta x.
Relative difference accounts for scale: an absolute error of 1 has very different significance when the reference value is 1 versus 1,000,000.
For differentiable functions, interior extrema are found among the roots of f'; on a closed interval, endpoints must also be checked.
Derivative descent moves opposite the derivative using x <- x - alpha f'(x); backtracking and Armijo line searches help select steps that decrease the objective.
Newton's method can converge rapidly near a sufficiently smooth root, but the example f(x) = x^(1/3) shows that poor differentiability can make iterations diverge.
Bracketed Newton combines Newton's fast local updates with bisection's interval guarantee, falling back to bisection when a Newton step leaves the known root interval.

Chapters

0:00 Relative Difference, Slope, and the Derivative as a Local Linear Approximation

Absolute difference alone can mislead: a difference of 1 is small relative to a reference of 1,000,000 but large relative to a reference of 1.
Slope measures steepness as rise over run, delta y divided by delta x; a line has the form f(x) = mx + b.
Linear interpolation estimates an unknown function value by connecting nearby samples, with slope (f(x2) - f(x1)) / (x2 - x1).
Taking the limit of that slope as delta x approaches zero defines f'(x), yielding the key approximation f(x + delta x) approximately equals f(x) + f'(x) delta x.

7:16 Differentiability, Convexity, and Local and Global Extrema

A function is differentiable where the derivative limit exists; C1 means the first derivative is continuous, Ck means k continuous derivatives, and C-infinity means smooth.
The epigraph includes the graph and all points above it; a function is convex when its epigraph is convex, equivalently when chords between graph points lie above the graph.
The indicator function I_C equals 0 inside a set C and infinity outside it; it is convex whenever C is convex.
For twice-differentiable functions, positive second derivative indicates convexity and negative second derivative indicates concavity; a sign change marks an inflection point.

11:00 Finding Extrema with Derivative Descent and Line Search

Interior local extrema of a differentiable function occur where f'(x) = 0; for f(x) = x^3 - 3x, solving 3x^2 - 3 = 0 gives candidates x = -1 and x = 1.
To find extrema on a closed interval [a, b], evaluate the function at every interior critical point and at both endpoints, since an endpoint can be the maximum or minimum.
Derivative descent updates x to x - alpha f'(x), stopping when the derivative magnitude is below a tolerance epsilon; alpha must be chosen carefully to avoid slow progress or divergence.
A backtracking line search repeatedly shrinks alpha by a factor beta until the function decreases; the Armijo condition accepts a step when f(x + delta x) is at most f(x) + gamma f'(x) delta x.

19:31 Newton's Method for Roots: Updates, Stopping Rules, and Failure Cases

Newton's method repeatedly linearizes f at the current estimate and uses the tangent line's root, producing x_(n+1) = x_n - f(x_n) / f'(x_n).
Stopping options include a fixed iteration cap, requiring |f(x)| <= epsilon, or requiring the relative iterate change |x_(n+1) - x_n| / (1 + |x_n|) to be small.
Convergence requires a suitable starting point and sufficient smoothness; for f(x) = x^(1/3), the derivative becomes unbounded near zero and Newton iterations can move farther from the root.
A maximum-iteration limit prevents a runaway loop, but reaching it does not establish that the estimate is close to a root.

23:10 Newton Updates for Extrema and Bracketed Newton with Bisection

To find an extremum with Newton's method, solve f'(x) = 0 using the update x_(n+1) = x_n - f'(x_n) / f''(x_n), which requires a twice-differentiable function.
A root of f' is only an extremum candidate; the sign of f'' helps determine whether it is a local minimum or maximum.
Bracketed Newton assumes a closed interval [a, b] containing the root, takes a Newton step when it stays inside the interval, and uses bisection when it does not.
The interval safeguard preserves progress by eliminating half the remaining bracket when a Newton step is excessively large or unreliable.

Transcript excerpt

0:00 Hi, I'm Nicholas Dwork and today we're going to discuss the derivative. First, we need a metric of difference. A simple metric is just x - x reference and very related to that is the absolute difference which is the magnitude of x - x reference. So if x is 10 and x reference is 11, then the absolute difference would be one. The issue with absolute difference as our metric is that one doesn't tell the whole story. If I tell you the absolute difference is one and x reference is a million, that seems pretty close. But if I tell you that the absolute difference is one and x reference is one, that seems much farther away. So instead, we use relative difference where we take the absolute difference and divide by the magnitude of x reference. When the

0:47 reference is large, you need a much larger difference to get the same relative difference. And this is often expressed as a percentage. Now, let's talk about mountains. Any hiker will tell you it's a lot harder to traverse a mile horizontally when the mountain is steeper than when it's shallow. And so, we want some number that represents how steep the mountain is. How hard is it going to be to climb that mountain? And the metric that we come up with is the slope. We look at how tall the mountain is. We look at how far we have to travel horizontally in order to move that height. [snorts] And we take the ratio of those two values. So the slope is delta y over delta x and it's commonly described as rise over run. We can impose these lines on a coordinate system. And now we can measure delta y and delta x and m becomes a number. We

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Nicholas Dwork.