CSci574 / AI 520 Machine Learning: Classroom Session 11/10/2026
Overview
Derek Harter reviews ensemble learning methods in CSci574 / AI 520, including bagging, random forests, Extra Trees, boosting, and stacking. He explains how scikit-learn supports parallel random-forest training and feature-importance estimates, then outlines the three-way data split and probability outputs needed to train a stacking blender without leakage in Assignment 6.
Key takeaways
- Random forests create an ensemble by fitting many decision trees on varied training samples and combining their predictions; the example uses 500 trees and reports about 88% accuracy on the moons dataset.
- Random-forest trees can be fitted independently, so scikit-learn's n_jobs=-1 can use all available CPU cores when parallel training is supported.
- Extra Trees increase diversity by adding randomness to decision-tree split selection, which can help when ordinary random forests produce overly similar trees.
- Forest-based feature importance can support data exploration: the Iris example ranks petal width and petal length above sepal measurements, while MNIST importance concentrates on central pixels.
- A stacking blender should be trained on predictions from base models applied to a separate validation partition, not on their training examples, to avoid information leakage.
- Using base-model probabilities instead of binary class labels gives the stacking blender more information about each model's confidence.
Chapters
- Assignment 6 is due later in the week and focuses in part on implementing a stacking ensemble.
- The assignment dataset is obtained through scikit-learn's fetch_openml interface to the OpenML repository, so students may need to check its documentation.
- Derek Harter invites questions and encourages students to accept and inspect the assignment promptly.
- Voting ensembles work best when their classifiers make diverse, non-overlapping errors; soft voting can weight predictions by confidence.
- Bagging and pasting build ensembles from different samples of training examples, while feature sampling can introduce additional diversity.
- A bagging example uses 500 decision trees, each trained on a sample of up to 100 examples from the moons dataset.
- A random forest combines many decision trees trained on varied samples and aggregates their predictions, commonly using class probabilities.
- The random-forest example reaches about 88% accuracy on the moons dataset, comparable to earlier voting-ensemble results without hyperparameter tuning.
- Because each tree can be fitted independently, random-forest training is embarrassingly parallel across CPU cores.
- In scikit-learn, n_jobs=-1 requests all available processors; other estimators may not support parallel execution in the same way.
- Tree controls such as maximum depth and leaf constraints can be tuned to change model complexity.
- CART decision trees greedily choose splits that reduce impurity, which can make trees similar even when trained on different samples.
- Extra Trees, short for Extremely Randomized Trees, add randomness to the tree-building split decisions to increase ensemble diversity.
- Harter recommends starting with a random forest and comparing Extra Trees or stronger randomization when additional diversity may improve performance.
- Feature-importance estimates can guide data exploration, feature selection, and feature engineering, alongside measures such as Pearson correlation.
- Across a forest, the way features contribute to impurity reduction provides an estimate of their relative importance; scikit-learn exposes a feature_importances_ attribute.
- On the Iris dataset, petal width and petal length rank as more informative than the two sepal measurements.
- For MNIST's 28 by 28 pixel images, a feature-importance heat map emphasizes central pixels, where the centered handwritten digits contain useful information.
- Boosting builds a sequence of predictors rather than fitting every ensemble member independently.
- A later classifier can focus on training examples that an earlier classifier misclassified, and successive predictors can be combined for new predictions.
- Unlike bagging-based forests, this sequential dependency makes boosting difficult to parallelize across learners.
- AdaBoost is mentioned as a scikit-learn method discussed in the textbook; Harter skips the details of gradient boosting.
- Stacking feeds the predictions of first-layer classifiers into a second classifier, called a blender, that learns how to combine them.
- The first-layer models can use different machine-learning algorithms, while a voting ensemble combines their outputs with a fixed rule such as majority vote.
- The blender can learn model strengths and weaknesses for the dataset rather than applying a simple unweighted vote.
- Assignment 6 uses three data partitions: one to train first-layer models, a second to generate their predictions and train the blender, and a third for final evaluation.
- Training the blender on predictions made for the same examples used to fit the base models would leak information and yield an unreliable evaluation.
- The validation partition is passed through the already-trained base predictors; their outputs become the blender's training features.
- For binary Magic Telescope classification, the assignment uses predicted probabilities rather than only yes-or-no class labels to give the blender richer inputs.
- Stacking could be extended to multiple layers, though Harter notes that deeper stacks begin to resemble neural networks.
- Students are reminded to keep making regular class-project commits, with one or two commits per week suggested.
- Harter says he will announce whether a Thursday meeting will take place and closes by reminding students to work on Assignment 6.
Transcript excerpt
0:08 Okay. Um, good afternoon. Um, might be having some technical problems. Um, but, uh, let's go and let me go ahead and get started. Um, so also might be a little bit of a short day today. We'll see. Um, I just, you know, want to give the usual reminders, u, make sure everybody knows where we're at here. So, uh, you still have, uh, the rest of this week for the assignment six. Um, so maybe I'll bring that one up first here. Um, but, um, we mostly went through the, uh, the trees and stuff and even did kind of voting ensembles last week. Uh, I have a few more things to say about uh about uh
0:57 boosting and stacking for the most part here. Um, so I'll probably go ahead and go over that. Uh, but first let me bring up um so I don't have too many people on yet, but um uh let me know if anybody has any questions right now they want to ask about the uh assignment. Uh I did I did talk about a little bit at least uh the first three or four tasks uh last time. So so I won't I'll resist the urge to go into too much detail. Uh but yeah there is a uh we are using this data set. Uh but we didn't give that to you in the uh uh in the the data this time. And it's actually going to be loaded from from uh from scikitlearn's uh this from it has some functions that
1:47 uh that uh interface with the open machine learning uh repository. So uh uh so I can't remember you might have to you might have to I might might not have given you everything that you need to
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Derek Harter.