Lecture 4c

Overview

The lecture reviews feature-selection methods on prostate data, then derives maximum-likelihood and maximum-a-posteriori estimation from probabilistic models. It shows that least squares is the maximum-likelihood solution for a linear model with independent Gaussian noise, while ridge and lasso correspond to MAP estimation with Gaussian and Laplace priors on the coefficients.

Key takeaways

On the eight-feature prostate dataset, exhaustive search evaluates 256 subsets, while forward or backward selection takes roughly d²/2 tests; sequential selection matched exhaustive search in the demonstrated run.
A feature-ranking method based on mutual information can be computationally inexpensive, but choosing how many top-ranked features to keep remains a separate validation problem and performed less well in this example.
Least squares is not merely a standalone optimization choice: it is the maximum-likelihood estimator for linear regression under independent, zero-mean Gaussian noise with fixed variance.
The noise distribution determines the residual loss: Gaussian noise yields squared error, while Laplacian noise yields an L1 residual objective that is more appropriate for some outlier-prone datasets.
MAP estimation adds a coefficient prior to the likelihood; an IID zero-mean Gaussian prior yields ridge's L2 penalty, while an IID Laplace prior yields lasso's L1 penalty.
For K regression targets sharing the same features, use a coefficient matrix with one column per target; scikit-learn accepts a target matrix and returns the corresponding coefficient matrix.

Chapters

0:00 Exhaustive and Sequential Feature Selection on Prostate Data

Ordinary least squares uses every feature, giving low bias but potentially high variance when unproductive features are included.
Exhaustive search tests all 2^d subsets; with eight prostate-data features, its 256 candidate subsets were practical to evaluate by cross-validation.
Forward selection adds one feature at a time and backward selection removes one at a time, requiring roughly d²/2 tests and matching exhaustive search in this demonstration.

4:17 Ranking, Ridge, and Lasso: Feature Selection Trade-offs

The ranking approach orders features by their mutual information with the target, then cross-validates how many top-ranked features to retain; it performed worse than exhaustive search here.
Ridge adds an L2 penalty and handles correlated features, but does not directly select features; it helped little on this dataset because feature correlation was limited.
Lasso's L1 penalty can zero coefficients, but also shrinks retained coefficients and introduces bias; selecting features with lasso and refitting least squares can reduce that bias.
Cross-validating the complete lasso-selection-plus-least-squares pipeline performed nearly as well as exhaustive search; exact results varied with the cross-validation shuffle seed.

8:29 From Feature Selection to Probabilistic Models

The lecture shifts from practical feature-selection methods to statistical estimation theory, a framework used throughout the rest of the semester.
Machine learning is framed as estimating unknown parameters in a probabilistic model that describes how targets are generated from features.
Maximum likelihood and maximum a posteriori estimation provide general procedures for fitting model parameters to observed data.

10:36 Gaussian Parameters, IID Samples, and the Likelihood Setup

A Gaussian distribution is parameterized by its mean μ and variance σ²; μ sets the bell curve's center and σ² determines its spread.
Observed values y₁ through yₙ are treated as realizations of independent, identically distributed random variables, or IID samples.
Independence makes the joint probability density a product of individual densities, with each sample governed by the same unknown parameter vector θ.

20:00 Maximum Likelihood and the Negative Log Objective

The likelihood treats collected data y as fixed and views the probability density as a function of unknown parameters θ.
Maximum likelihood selects the θ that maximizes the likelihood, meaning the model parameters that best explain the observed samples under the assumed distribution.
Taking the natural logarithm preserves the maximizer while converting products into sums; minimizing negative log likelihood is often the cleanest equivalent form.
Maximum likelihood uses the likelihood alone and makes no prior assumption about plausible parameter values.

29:17 Visualizing a Gaussian Mean Estimate

For samples from a Gaussian with known unit variance and unknown mean μ, each candidate μ defines a shifted bell curve.
With two fixed observations, the likelihood at a candidate μ is the product of the two Gaussian density heights at those observations.
The best estimate is the μ that maximizes that product; changing the assumed data distribution would change the curves but not the estimation procedure.

37:05 The Linear Gaussian Regression Model

The regression model is y = Xβ + ε, with standardized data and no intercept term in the displayed formulation.
Each noise sample εᵢ is assumed independent and identically distributed as a zero-mean Gaussian with variance σ².
For a fixed feature row xᵢ and coefficient vector β, the target yᵢ is Gaussian with mean xᵢᵀβ and variance σ².

44:11 Deriving Least Squares from Gaussian Maximum Likelihood

The joint likelihood multiplies the Gaussian densities for all n samples, producing an exponential whose exponent contains the sum of squared residuals.
Taking the negative log likelihood reduces the parameter-dependent objective to a scaled squared norm, ||y − Xβ||².
Terms that do not depend on β, including the Gaussian normalization and the positive scale factor, do not change the optimizer.
Thus, least squares is maximum-likelihood estimation when the linear regression errors are modeled as IID Gaussian noise.

49:38 Choosing a Noise Model for Outlier-Prone Data

Gaussian noise assigns a poor fit to data containing severely corrupted outliers, helping explain why least squares can perform badly in that setting.
The lecture contrasts Gaussian assumptions with Laplacian noise, which leads to minimizing the L1 norm of residuals.
Changing the assumed noise distribution changes the negative log-likelihood objective and can produce a procedure better suited to the data.

51:47 Linear Regression with Multiple Target Variables

For K simultaneous targets, each target receives its own coefficient vector, and the vectors are stacked as columns of a coefficient matrix B.
With d features and an intercept, B has d + 1 rows and K columns.
The automobile example predicts fuel efficiency and maximum speed from shared features; scikit-learn can accept a target matrix and return a coefficient matrix.

54:27 Bayes' Rule and Maximum A Posteriori Estimation

MAP estimates the coefficient vector β with the highest posterior probability given the observed targets and features.
Bayes' rule combines the likelihood P(y | β, X) with a prior P(β); the evidence P(y | X) can be dropped when optimizing over β because it is constant with respect to β.
Unlike maximum likelihood, MAP incorporates assumptions about plausible parameter values before observing the data.

1:00:06 Gaussian and Laplace Priors Recover Ridge and Lasso

Taking the negative logarithm of the MAP objective adds a prior-derived penalty to the negative log likelihood.
For linear Gaussian regression, the data-fit term is squared residual error, and the prior contributes a regularization term on β.
IID zero-mean Gaussian coefficient priors produce an L2 penalty and ridge regression; IID Laplace coefficient priors produce an L1 penalty and lasso.
The regularization weight depends on the noise variance σ² and the prior's scale or variance.

Transcript excerpt

0:02 Okay, let's get started. Um, so we've been talking about different feature selection techniques and um we've gone through all the different techniques we're going to talk about and then uh we can sort of review them through their performance in the demo. So this is the demo with the prostate data. Um and so first of all we talked about plain le squares no feature selection. Um we know that that um has the issue of possibly overfitting the data in terms of using features that are not um very productive and so you're going to see a penalty in the variance. So um it has low bias but high variance and when we

0:52 look at the cross validation le squares uh that has um I think the highest of all the other techniques we'll look at. We talked about the optimal thing to do um the optimal thing to do is to consider every possible subset of features for each one. You design the le squares coefficients then you use cross validation to test that and you see which of those subsets works best and that's possible with few features because you have two to the power d different subsets to test. So in this uh demo there was only I think eight features and so two to the^ eight is something we can handle and we're able to implement this method and not

1:40 surprisingly uh when we look at the cross validation mean squareed error over here this gives us the best performances and it is the optimal one. [clears throat] Um next we talked about some suboptimal techniques. One of the ones we talked about was a sequential uh feature selection. So let me just put this is the this is the sequential method. So we had

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof. Schniter.