CSci574 / AI 520 Machine Learning: Classroom Session 11/05/2026
Overview
Derek Harter explains how ensemble learning combines diverse classifiers through hard or soft voting, then introduces bagging, pasting, and out-of-bag evaluation as ways to diversify models trained with the same algorithm. Using scikit-learn examples on the make_moons dataset, he connects these methods to random forests and previews an assignment using the Magic Telescope dataset, five base classifiers, extra trees, and stacking.
Key takeaways
- Ensembling helps most when component classifiers make different errors; combining nearly identical models offers little benefit.
- Hard voting combines predicted class labels, while soft voting uses predict_proba estimates and requires probability-capable or calibrated estimators.
- Bagging trains models on bootstrap samples drawn with replacement, while pasting uses samples drawn without replacement.
- Bagging can reduce prediction variance: individual sampled trees may be less stable, but their aggregated decision boundary is smoother.
- Out-of-bag examples are training instances omitted from a particular bootstrap sample, allowing an internal estimate of generalization performance.
- Stacking adds a second-level classifier that learns how to combine base-model outputs instead of relying on a fixed majority or probability vote.
Chapters
- Derek Harter introduces ensemble learning after wrapping up the main decision-tree material from chapter 6.
- The session focuses on voting classifiers, with random forests and stacking reserved mainly for the following week.
- Assignment 6 asks students to build voting, random-forest-style, and stacking ensembles.
- The wisdom-of-the-crowd idea works best when classifiers make different errors or perform well in different regions of the input space.
- Combining models trained on identical data with the same algorithm may add little if their predictions are nearly identical.
- Random forests create diversity among decision trees by training them on different samples or feature subsets.
- A textbook demonstration compares 10 coins that each land heads 51% of the time with the aggregate result across the coins.
- Individual coin estimates fluctuate, while their combined estimate tends to approach the true 0.51 probability more quickly.
- Harter cautions that averaging coin flips is an imperfect analogy for classifier ensembles, but it illustrates how aggregation can reduce variation.
- Hard voting selects the class receiving the most predictions, with ties requiring a separate rule in multiclass settings.
- The scikit-learn example combines logistic regression, a decision tree, and a support vector classifier on 2,500 make_moons samples.
- The nonlinear moon-shaped classes challenge linear decision boundaries, while a decision tree can create nonlinear boundaries.
- VotingClassifier accepts named estimator tuples and behaves like other scikit-learn estimators, including supporting fit and predict.
- Calling fit on the ensemble trains each included classifier on the training data.
- The ensemble can be passed to a decision-boundary plotting function that queries predict across a grid.
- In the make_moons example, logistic regression performs several percentage points below the decision tree and support vector classifier.
- The ensemble is around 0.89 accuracy in the demonstrated run, but does not produce a dramatic improvement over the strongest individual models.
- Harter notes that results can vary between runs and recommends a larger dataset or cross-validation for a more reliable estimate.
- Hard voting combines predicted labels, while soft voting combines class-probability estimates from each classifier.
- A model assigning a higher probability to a class can contribute more strongly than one whose estimate is close to 0.5.
- Every estimator in a soft-voting ensemble must support predict_proba; SVC may require probability calibration, and scikit-learn's API is changing toward CalibratedClassifierCV.
- A diverse voting ensemble can combine independently developed approaches such as support vector machines and neural networks.
- Machine-learning competition teams may ensemble their best separately tuned models to gain a few percentage points.
- The improvement is problem-dependent and may be small or absent, even when the models differ.
- Instead of combining different algorithms, an ensemble can use one algorithm trained on different random subsets.
- Sampling can vary training instances, feature subsets, or both; decision trees are a natural fit for this approach.
- Bagging samples with replacement, while pasting samples without replacement, creating variation among otherwise similar models.
- Independently sampled models can be trained in parallel, with separate CPUs handling separate ensemble members.
- Each model trained on a subset may have more bias than one trained on all data, while aggregation can reduce variance.
- Once trained, sampled models can be combined with the same hard- or soft-voting methods used for heterogeneous classifiers.
- BaggingClassifier can create hundreds of estimators, such as 500 decision trees, and train them on sampled data.
- Setting bootstrap=True selects samples with replacement, the bagging approach; setting it to false uses sampling without replacement.
- The make_moons example produces a smoother ensemble boundary than an individual decision tree and reaches roughly 0.88 to 0.89 accuracy in one run.
- A single decision tree forms axis-aligned boundaries and can produce a jagged fit, especially when it is left unregularized.
- The bagged ensemble has a smoother boundary, illustrating how combining trees can reduce variance.
- One demonstrated ensemble score reaches about 0.91 to 0.92, though Harter warns that run-to-run variation makes the exact result uncertain.
- With sampling with replacement, some training examples are omitted from each individual model's bootstrap sample.
- Those omitted examples are that estimator's out-of-bag instances, and the sets differ across the 500 trees in the example.
- An out-of-bag score estimates performance on unseen data and can support preliminary evaluation without a separate validation split.
- Students fetch the binary Magic Telescope dataset from OpenML using scikit-learn rather than loading a bundled local data file.
- The voting task uses five classifiers, including logistic regression, K-nearest neighbors, decision trees, and a multilayer perceptron, and compares hard with soft voting.
- Later tasks use a random forest or extra trees and a stacking ensemble; stacking trains a second classifier to learn how to use base-model outputs.
- The stacking workflow requires three data splits, which can be created by calling scikit-learn's train_test_split twice.
Transcript excerpt
0:50 All right. Good afternoon. Excuse me. Let's go and get started here. Got a few online, looks like. Um So, let me know if you guys can't hear me. Hopefully hopefully I got everything going here. Um So, yeah, to begin with, so don't forget I'm I'm going to be talking about our assignment on ensemble learning, but that that's you still have a little bit of time on that, but if I get a chance here at the end, I'll maybe bring that up. Make sure that the link is working. I haven't checked if people have accepted the assignment yet and started working on it. So, yeah, I mean, my plan today is we we kind of got through most all the decision tree stuff, so that's actually the uh
1:38 uh the chapter six kinds of materials, at least the main stuff. Um so, yeah, I'm going to start with talking about ensemble learning and forest a bit here today. So, you're I mean, base basically that's what you're going to be doing on the next assignment that you guys should be working on right now, the assignment six. Um it's kind of doing a bit building some ensembles by hand, which is sort of what goes on in the first part of this uh chapter here, and then talking a little bit uh and then the assignment has a little bit about using some random forests, and then doing some of this stuff with like the stacking sorts of things in the assignment. So, Okay doc. Let's Let's go ahead.
2:29 Um and uh get started here. Um So, I I just I I I'm planning today. I think I'm mostly just going to talk about uh ensemble uh like voting classifiers, and kind of try and stop
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Derek Harter.