Lecture 8c

Overview

Prof. Schniter explains sequential boosting through AdaBoost, gradient boosting, and XGBoost, contrasting them with parallel random forests. He derives how AdaBoost reweights misclassified examples, how gradient boosting fits negative loss gradients, and how XGBoost adds second-order loss approximations and tree regularization, then covers implementation, feature importance, and course guidance.

Key takeaways

AdaBoost emphasizes examples that earlier classifiers misclassified, and its weighted error rate determines how strongly each base classifier contributes to the final ensemble.
Gradient boosting works with general losses by fitting base predictors to per-example negative gradients, then scaling each update with a learning rate such as scikit-learn's default 0.1.
XGBoost combines first- and second-order loss information with penalties on leaf values and leaf count to guide both tree predictions and tree growth.
Scikit-learn's default AdaBoost classifier uses 50 depth-1 decision trees, while its gradient boosting classifier and regressor default to 100 depth-3 trees.
Random forest MDI can overvalue high-cardinality features, while permutation importance offers a model-agnostic, no-retraining diagnostic by measuring performance after shuffling one feature.
Prof. Schniter recommends prioritizing XGBoost and random forests for the tree portion of the final project.

Chapters

0:00 Random Forests Versus Sequential Boosting

Random forests train trees in parallel using random subsets of training data and features at each split.
Boosting trains base predictors sequentially, adjusting later training to address earlier errors.
The lecture introduces AdaBoost, gradient boosting, and XGBoost as three boosting methods.

2:54 AdaBoost Builds a Weighted Ensemble for Binary Classification

AdaBoost trains base predictors one at a time, using sample weights to emphasize examples with larger losses.
For binary classification, labels and base classifier outputs are restricted to +1 and -1.
The stage-wise ensemble adds each classifier with a positive weight alpha, then uses the weighted sum for prediction.

7:30 AdaBoost Derives Its Weighted Misclassification Objective

The exponential loss penalizes predictions that disagree in sign with a label more than predictions that agree.
At each stage, the loss reduces to a weighted misclassification objective for selecting the next classifier.
The weighted error rate epsilon determines the classifier weight alpha, while examples are reweighted for the next stage.

20:00 AdaBoost Stumps and Scikit-learn Defaults

Scikit-learn provides AdaBoostClassifier and AdaBoostRegressor; the classifier defaults to 50 decision stumps.
In a two-feature illustration, later stumps prioritize examples misclassified by earlier stumps, shown by increasing their weights.
Combining three simple stumps with learned alpha weights produces a more complex classifier with six decision regions.

24:36 Gradient Boosting Fits Negative Loss Gradients

Gradient boosting generalizes the ensemble objective to arbitrary loss functions and base-predictor classes.
An exact joint optimization over the next predictor and its coefficient can be difficult, so the method approximates a gradient-descent step.
It computes per-example negative gradients, or pseudo-residuals, then trains a base predictor to approximate them using quadratic loss.

35:00 Gradient Boosting Learning Rates and Regression Trees

A learning-rate parameter mu, typically between 0 and 1, scales each gradient boosting update.
Scikit-learn's GradientBoostingClassifier and GradientBoostingRegressor default to 100 trees of depth 3 and mu of 0.1.
The regression illustration compares one depth-1 stump, a depth-3 tree with up to eight leaves, and up to 2,000 boosted depth-1 trees.

38:30 XGBoost Adds Second-Order Updates and Regularization

XGBoost extends gradient boosting with a second-order Taylor approximation and an explicit regularization term.
Its stage-wise formulation adds base trees without separate alpha coefficients, effectively using a coefficient of one.
The package is separate from scikit-learn but provides a similar Python interface and often performs well with default settings.

42:30 XGBoost Taylor Expansion Produces a Quadratic Objective

The second-order expansion includes both the loss gradient and its second derivative around the previous ensemble prediction.
Approximating the loss this way makes the new tree's objective quadratic in its leaf outputs.
Each training example is assigned to a leaf, whose output value becomes the tree's prediction for that example.

46:00 XGBoost Regularizes Leaf Values and Tree Size

XGBoost's tree penalty combines a lambda-weighted squared-leaf-output term with a gamma penalty for the number of leaves.
For each leaf, the optimal output is found by differentiating the quadratic objective and setting the result to zero.
The resulting objective lets tree construction compare candidate splits by their reduction in regularized loss.

50:00 XGBoost Selects Splits by Regularized Loss Reduction

Tree construction searches for a feature and threshold that divide a leaf's samples into two subsets.
Each candidate split is evaluated by comparing the parent leaf's loss with the combined loss of its two children.
If no candidate split improves the objective, the tree stops growing that branch.

53:40 XGBoost Python API and Tree-Model Tuning

The example imports XGBClassifier, creates a model, and calls fit and predict using a scikit-learn-like interface.
For the course project, Prof. Schniter recommends focusing on XGBoost and random forests rather than AdaBoost or gradient boosting.
Hyperparameter tuning resources are provided because some parameters should be tuned jointly and others can be adjusted individually.

55:52 Random Forest and XGBoost Feature Importance

Random forests report mean decrease in impurity, or MDI, based on how splits using a feature reduce Gini impurity.
MDI can favor high-cardinality categorical features, so a large importance score is not always reliable evidence of predictive value.
XGBoost provides gain, based on objective improvement from feature splits, and cover, related to the amount of data or Hessian mass affected.

59:47 Permutation Importance and Course Wrap-Up

Permutation importance shuffles one feature's sample values and measures the resulting performance drop without retraining the model.
A large performance decrease indicates the feature is useful; little change suggests it may contribute little to predictions.
Retraining after permutation or removing a feature can yield more accurate selection at greater computational cost.
The course transitions away from homework and labs toward the final project; the final exam is described as knowledge-based rather than derivation-focused.

Transcript excerpt

0:01 Okay, we're going to continue our discussion of ensemble methods and trees. So, last time we talked about um when we talked about ensembling, we talked about decision trees, and then we put them together to create random forests. So, random forest, if you remember, is an ensemble of trees um where you use a random subset of data to train each tree and a random subset of features at each split in each trees in each tree, and then um the tree predictions are all combined or the you know, those those weak learners are combined. Um and basically, we can think of this as a parallel method because you can train each tree simultaneously

0:50 um all in parallel and then just combine them when you're done. So, what we're going to talk about now is boosting. And boosting, instead of a parallel approach, is a sequential approach. Um >> [clears throat] >> These are ensemble methods um typically done with trees uh also called boosting. So, um we're going to talk about three um particular types of boosting methods. One is called AdaBoost. The next, gradient boosting. And finally, um more recent method called extreme gradient boosting or XGBoost. The overall um

1:39 idea behind >> [snorts] >> these methods can be shown here. So, you start with a data set. You train your model, in this case, some sort of a tree. Uh you look at how um you know, how well it it's done according to some uh training errors and wherever you um see errors, you increase the weight on those errors and you say, "Okay, now let's retrain this this model, this tree um with a higher weight on those errors. So, let's make sure that it doesn't make the same mistakes again." And then, you know, you test that

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof. Schniter.