Lecture 8a

Overview

Ensemble methods improve prediction by combining multiple weak learners: under the assumption of independent errors, majority voting across 31 classifiers that are each 70% accurate can reach about 99% accuracy. The lecture compares hard and soft voting, explains how diversity can come from different training samples, features, or model types, and details pasting, bootstrap aggregating (bagging), random patches, and random subspaces, including their scikit-learn options.

Key takeaways

With independent 70%-accurate classifiers, majority voting can raise accuracy to about 99% with 31 predictors; correlated errors reduce that benefit.
Soft voting averages predicted probabilities or class PMFs before making a final decision and usually outperforms voting over hard class labels.
Pasting samples without replacement, whereas bagging samples with replacement and can include duplicate training examples in each predictor's dataset.
A bagged predictor trained from n bootstrap draws of n examples leaves about 37% of examples out on average, and those out-of-bag examples can estimate performance.
Random patches combine variation in training examples and features; random subspaces vary features while using the full dataset.

Chapters

0:00 Independent Weak Learners Turn Majority Voting into a Strong Ensemble

A biased coin with a 70% chance of heads illustrates the benefit of majority voting: 11 flips identify the heavy side with 92% probability, 21 with 97%, and 31 with 99%.
In binary classification, 31 weak learners that are each 70% accurate can reach about 99% ensemble accuracy when their outcomes are statistically independent.
Correlated classifiers provide less improvement; if all 31 always make the same prediction, voting remains at 70% accuracy.
Randomization helps create diverse, less-correlated predictors, which is essential for ensemble gains.

4:06 Soft Voting and Three Ways to Create Predictor Diversity

Hard voting combines class labels, while soft voting averages predicted probabilities or class PMFs and then chooses a threshold or peak; soft voting usually performs better.
Scikit-learn's VotingClassifier supports both hard and soft voting, and regression ensembles can average their numeric predictions.
Predictor diversity can come from training on different data samples, using different feature subsets, or combining model types such as logistic regression, SVMs, and neural networks.

8:20 Pasting, Bagging, and Feature-Based Randomization in scikit-learn

Pasting samples training examples without replacement, assigning each example to one predictor; bagging samples with replacement, so examples may repeat or be omitted.
Bagging draws n samples for each predictor from n training examples; for large n, about 37% of examples are left out of a predictor's bootstrap sample.
Those out-of-bag samples can support cross-validation through scikit-learn's BaggingClassifier or BaggingRegressor and the OOB score option; bootstrap=True selects bagging, while bootstrap=False selects pasting.
Feature sampling suits datasets with many features: random patches vary both samples and features, while random subspaces use all samples but different feature subsets; max_features controls feature selection.

Transcript excerpt

0:01 Hello everyone, we're going to start a new unit uh unit eight on ensemble methods and decision trees. So uh for intuition about what these [clears throat] ensemble methods are is that suppose that you want to determine the heavy side of a biased coin. Um you want to know if it's heads or tails. And although you don't know this, uh let's say that heads occurs 70% of the time and tails occurs 30% of the time. So [clears throat] to predict the heavy side of the coin you would flip it n times and choose the majority vote. So let's say you just flipped it once because heads occurs

0:48 70% of the time uh the majority vote would be, you know, whatever whatever you got and so you'd be correct 70% of the time. But if you had um flipped it 11 times and chosen [snorts] the majority vote then you'd find you'd be correct with 92% probability. And if you flipped it 21 times you'd be uh correct 97% of the time and if you flipped it 31 times, took the majority vote, you'd be correct 99% of the time. >> [clears throat] >> Okay, so these numbers can be computed from the CDF of the binomial distribution. Um but basically the trend that you see here is if you um choose the majority vote from, you know,

1:38 more and more coin flips you would um get a steadily increasing probability of correctly identifying the heavy side of this biased coin. Okay, so um even though this is about you know, coin flips, the analogy in machine learning is binary classification. So, let's say we have 31 binary classifiers, and let's say that each one of these binary classifiers is correct only 70% of the time. So, you know,

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof. Schniter.