Lecture 8b
Overview
The lecture explains stacking as a two-stage ensemble method: cross-validated predictions from diverse base estimators become meta-features for a trained combiner, often ridge regression or classification in scikit-learn. It then develops decision trees, covering recursive feature-threshold splits, regression and classification impurity measures, overfitting controls, and random forests, which reduce tree variance by training trees on randomized data and feature subsets.
Key takeaways
- Stacking avoids training its combiner on in-sample base predictions by using k-fold out-of-fold predictions as meta-features, then retrains each base estimator on the full dataset.
- Ridge regularization is well suited to stacking when base predictions are correlated, and scikit-learn supplies RidgeCV, RidgeClassifierCV, StackingRegressor, and StackingClassifier.
- For regression trees, evaluate candidate splits using sample-count-weighted variance or absolute error; absolute error is less sensitive to outliers.
- For classification trees, Gini impurity equals 1 - Σpₖ², ranges from zero for a single-class subset to 1 - 1/K for a uniform K-class subset, and is aggregated across children by subset size.
- Decision trees need no feature standardization and can split categorical features directly, but depth limits and minimum leaf sizes help counter their tendency to overfit and change with small data perturbations.
- Random forests address individual trees' high variance by training many trees with randomized data and feature subsets, then aggregating predictions through voting or averaging.
Chapters
- Stacking trains different base estimators on the same features and data, then learns a separate combiner from their predictions.
- Unlike fixed voting or averaging, the combiner is trained; it can optionally receive the original features as well as the base predictions.
- Because base predictions may be correlated, ridge regularization is a useful combiner choice; scikit-learn provides RidgeCV and RidgeClassifierCV.
- For each base estimator and each fold, train on all other folds and predict the held-out samples.
- Collect the out-of-fold predictions into a matrix whose columns represent base estimators and whose rows represent training samples.
- Train the second-stage combiner on this meta-feature matrix and the true labels, using cross-validation such as RidgeCV to select regularization.
- After fitting the combiner, retrain each base estimator on the full dataset; scikit-learn's StackingClassifier and StackingRegressor implement this workflow.
- A decision tree divides the D-dimensional feature space into L regions and assigns an output value Z for each region.
- For regression, a region can predict the mean of its training targets; for classification, it can predict the most common label.
- Recursive thresholding creates the tree structure: the initial feature domain is the root, and the terminal regions are leaves.
- At each node, choose a feature and split its samples so that labels within each resulting subset are as homogeneous as possible.
- Ordinal features use thresholds between observed feature values, so only finitely many distinct splits need consideration.
- Categorical features can be divided into category groups, such as A or C versus B or D.
- Stop splitting when labels are identical or when constraints such as minimum samples per leaf or maximum depth are reached, limiting overfitting.
- The example predicts high or low insurance risk from driver age and car type.
- A first age threshold at 27.5 separates drivers aged 17, 20, and 23 from the older examples, producing a homogeneous high-risk group.
- For ages above 27.5, a car-type split separates sports cars, labeled high risk, from other cars, labeled low risk.
- The resulting three regions classify all six training examples correctly, illustrating both recursive partitioning and possible overfitting.
- For a candidate feature-threshold pair, compute the sample mean and variance of the targets in each resulting subset.
- Choose the split minimizing the subset variances weighted by their sample counts.
- Absolute error is an alternative homogeneity measure that is more robust to outliers than squared deviations.
- Represent labels in each subset with an empirical probability mass function over K classes.
- Gini impurity is 1 minus the sum of squared class probabilities, and split impurity is weighted by subset sizes.
- A uniform class distribution has maximum impurity, 1 - 1/K; a subset containing only one class has impurity zero.
- Entropy, misclassification error, and chi-squared measures are alternatives to Gini impurity.
- A tree can be designed to minimize a chosen loss by searching feature-threshold pairs, selecting the best split, and repeating recursively.
- Trees handle ordinal and categorical features without standardization, and their explicit split rules make them interpretable white-box models.
- Prediction follows a path through the tree and is fast; a balanced tree can reach individual samples in roughly log₂N levels.
- Training can be expensive, and a small training-data change can alter the tree; common controls include maximum depth and minimum leaf size.
- Individual trees can have low bias but high variance because deep trees fit training data closely and are sensitive to sample changes.
- A random forest trains many trees using randomized data subsets and random feature subsets at each split.
- Combine tree outputs with voting or averaging; the randomized trees differ in structure, and their ensemble can perform better than individual trees.
- The next topic is sequential tree ensembles, contrasting with random forests' parallel tree training.
Transcript excerpt
0:01 Okay, next we'll talk about stacking. Um so in stacking uh what you do is you take different base estimators. So here we're getting diversity using different estimators. Um and we basically give them the same features and the same data for training. So no diversity from those perspectives. Um >> [clears throat] >> what we do is we we train these different base estimators to get these predictions. Uh we'll call them Y hat one through L. And then we um we have a separate combiner function here that is also something we train. So that's a little bit different than voting or averaging
0:50 where this is the combiner would just be a fixed deterministic procedure, but here we're actually going to going to train this. >> [clears throat] >> And the combiner um takes those L different predictions, one for each base estimator, and from them uh learns to predict the the final target. >> [clears throat] >> Now you could also sort of as an option you can also give the combiner uh the features as well. Uh but here uh in this picture we don't do that. Oh [clears throat] actually here yeah, so here's the optional. Um Now of course um because these predictors are using the same data and the same features, uh these
1:36 um predictions can be very correlated. And so a popular choice for the combiner is to use um ridge regularization. So you know, ridge if you you know, you can do this with classification or regression. And um in fact there's some uh nice um scikit-learn uh that include cross validation and ridge. So, you have ridge ridge CV for regression and ridge classifier CV for classification. Okay, so let's get into a little bit more detail about how we do this because
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof. Schniter.