Lecture 14: Non-linear least-squares
Watch on YouTube →
Overview
Burton Ma develops nonlinear least squares from linear least squares, first showing why normal equations can fail numerically and why squared-error estimates are vulnerable to outliers. He derives Gauss-Newton by linearizing residuals with a first-order Taylor approximation and using the Jacobian to solve and repeat a local least-squares problem, then introduces Levenberg-Marquardt and a landmark-based localization problem.
Key takeaways
- Explicitly inverting X transpose times X can be numerically unstable; MATLAB left division or an SVD-based Moore-Penrose pseudoinverse is generally more robust.
- Least squares and the mean have breakdown point 1/n, so one extreme observation can distort their estimates, while the median can tolerate corruption of up to half the data.
- A degree-14 polynomial design matrix can be poorly conditioned because its power columns have widely different scales, causing normal equations to produce a badly incorrect curve fit.
- Gauss-Newton makes nonlinear least squares tractable by approximating residuals with a first-order Taylor expansion, solving the resulting Jacobian-based linear least-squares problem, and iterating.
- Levenberg-Marquardt adds damping controlled by lambda to interpolate between fast Newton-like updates and more cautious gradient-descent-like steps.
- Distance measurements to known landmarks turn localization into a least-squares problem when noisy circles fail to intersect; GPS uses the related sphere-intersection principle and needs at least four visible satellites.
Chapters
- The linear model uses a design matrix X, observations Y, and unknown parameter vector W.
- For a single unknown height measured multiple times, the least-squares estimate is the average of the observations.
- For fitting a line, X contains the measured x-values and a column of ones for the intercept.
- Burton Ma's linearLS function checks that X and Y have compatible dimensions before solving.
- The normal-equation formula explicitly computes an inverse of X transpose times X, but MATLAB's left-division operator is a more numerically robust choice.
- The Moore-Penrose pseudoinverse, computed using singular value decomposition, is another robust option, though slower than MATLAB left division.
- A single distant point can dominate the sum of squared residuals and pull a fitted line far from the true relationship.
- Least-squares estimates, including the mean, have a breakdown point of 1/n: one corrupted measurement can distort the estimate arbitrarily.
- For ten values, changing one value to 100 shifts the mean sharply, while the median remains near the center until more than half the observations are corrupted.
- The median's breakdown point is 1/2, making it a more robust measure of center than the mean.
- A degree-14 polynomial fit creates design-matrix columns with powers of x, producing values with very different scales.
- The example fits e to the sine of 4t and shows the normal-equation result diverging badly from the expected curve.
- MATLAB's left-division solution and the singular-value-decomposition solution closely agree and give a much more reasonable fit.
- The example is deliberately extreme; fitting a lower-degree polynomial reduces the conditioning problem.
- The nonlinear model replaces the linear prediction XW with a function F(x, W), while still minimizing the sum of squared residuals.
- Unlike linear least squares, nonlinear least squares generally has no analytic solution and requires an optimization algorithm.
- The gradient of the sum of squared residuals can be expressed using the Jacobian of F and the residual vector Y minus F.
- The Jacobian collects the model's first-order partial derivatives and can support gradient-based descent or Newton-style methods.
- Burton Ma approximates each residual around a current parameter estimate W0 using a first-order Taylor series.
- The linearized residuals form a linear least-squares problem, with the Jacobian taking the role of the design matrix.
- Solving that local problem yields a parameter update; repeating the linearization and solve produces the Gauss-Newton method.
- The central transformation is to replace a difficult nonlinear problem with a sequence of locally linear least-squares problems.
- Each iteration evaluates the residual vector and Jacobian, then solves the linearized least-squares problem to move from W0 to a new estimate.
- Burton Ma's MATLAB implementation is called GNLS and takes the data X and Y, a model function, its Jacobian, and an initial parameter estimate.
- The stopping rules include a maximum iteration count or a parameter change whose norm is at most epsilon.
- The key conceptual takeaway is the linearize-and-iterate strategy, not memorizing the full derivation for a test.
- Levenberg-Marquardt adds a scaled identity matrix to the Gauss-Newton system to address cases where the undamped matrix is not positive definite.
- The damping parameter lambda tunes the update: suitable settings give Newton-like fast convergence, while larger damping makes the behavior more like small-step gradient descent.
- A Fletcher modification uses the diagonal of the Jacobian transpose times the Jacobian, evaluated at the starting point, in place of the identity matrix.
- In a two-dimensional map, one measured distance to a known landmark constrains a robot's position to a circle; two circles usually leave two candidate locations.
- Three landmark distances can identify a common intersection through trilateration, while using more landmarks is called multilateration.
- GPS applies the same distance-based idea in three dimensions and requires at least four visible satellites for a unique location.
- With imprecise measurements, the circles may not intersect at one point; least squares can estimate the position that best fits the measured distances.
Transcript excerpt
0:04 Okay, good morning everybody. Uh so, to remind you where we were at the end of last day, we were talking about linearly squares. Uh and we ended up deriving the normal equation, which is right there. And remember, X is our design matrix, uh Y are our measured or our observation our measured values or our observations, uh and our W star is the thing we're after. Those are model parameters that we're trying to estimate. Uh so, for linearly squares, there's actually an analytic solution. All you have to do is multiply some matrices, take their inverse, uh and you uh end up with a solution. Uh so, we discussed talking about how do you estimate a single value from a direct observation? So, this is was the height example where you're measuring someone's height. Oops, sorry. And we plugged it into this
0:52 equation here, and you saw that to estimate someone's height from multiple observations, the estimate is just the average of the observations. Um which was nice. Uh now, estimating the line of best fit. So, for the line of best fit, uh we know what the matrix X needs to look like. It looks like um the matrix X ends up looking like uh X1, X2, dot dot dot dot dot XM. Uh and then all ones down here. So, you just plug that thing in into the normal equation, uh and you can get an answer. Uh however, uh you shouldn't do it that way. So, to implement this thing, um uh I've written this little function here called linear LS, so linearly
1:39 squares. Right? It takes in our design matrix X. It takes in our vector of observations. Uh and this one takes in an a third parameter, uh which is how do you solve
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Burton Ma.