CSci574 / AI 520 Machine Learning: Classroom Session 11/03/2026

Overview

Derek Harter introduces decision trees through the Iris dataset, explaining how binary feature-threshold splits create axis-aligned boundaries and how scikit-learn’s CART algorithm selects splits by minimizing weighted Gini impurity. He emphasizes that unconstrained trees can memorize training data, while depth and sample-count limits regularize them; individual trees are interpretable but often less effective than ensembles such as random forests.

Key takeaways

CART scores candidate splits by the sample-weighted average of the left and right child impurities, selecting the feature and threshold that minimize that cost.
Gini impurity equals 1 minus the sum of squared class proportions: a pure leaf scores 0, while a leaf with balanced classes is more impure.
A decision-tree classifier can perfectly fit its training set by growing until leaves are pure, so controls such as max_depth and minimum leaf samples are essential for generalization.
Decision-tree boundaries are axis-aligned because each internal node tests one feature at a time; data with rotated or oblique boundaries may require a transformation or a different model.
Tree leaves provide interpretable class predictions and class proportions that can be used as probability estimates, while tree inference follows only the tests along one branch.
Individual trees can be weak classifiers, but ensembles such as random forests combine multiple differently constrained trees and can achieve much stronger performance.

Chapters

0:00 Assignment 5 Feedback and the Decision-Tree Unit

Derek Harter reports that most students submitted Assignment 5 successfully and that grades were returned during class.
Assignment 6 covers decision trees and ensembles, with work scheduled across the current and following week.
The course is finishing supervised learning before moving to unsupervised learning for roughly the final three weeks.

3:00 Portable Data Paths for pytest and Notebooks

pytest runs from the project root, while notebooks may use a different working directory, causing relative data paths to fail.
The class utilities provide a project-root path based on the utility file’s location, avoiding fragile try/except searches.
Students should run notebooks through completion before submitting, since some Assignment 5 notebooks were incomplete or had errors.

7:00 Assignment 5: SVM Regularization and Gaussian Kernels

The SVM parameter C controls the penalty for classification errors: a higher C can force the boundary to accommodate an outlier.
The assignment compared a Gaussian kernel implemented by hand with scikit-learn’s built-in Gaussian kernel.
Both kernel implementations should produce the same decision boundary when applied with the same settings.

11:00 Decision-Tree Materials and Graphviz Setup

The upcoming course materials are Chapter 6 on decision trees and Chapter 7 on ensemble learning.
Rendering tree diagrams may require both the Graphviz system package and its Python package.
In the class development container, the system dependency can be installed with apt and the Python front end with pip.

16:00 Decision Trees as Binary Feature Tests

A decision tree uses binary tests at internal nodes, usually comparing one feature against a threshold, and assigns a prediction at a leaf.
The Iris example uses petal length and petal width to classify Setosa, Versicolor, and Virginica.
Manually created expert-system rules resemble decision trees, but CART automates feature and threshold selection.

20:00 Iris Decision Boundaries and Tree Depth

Each decision tests one feature, so the resulting regions are axis-aligned, with boundaries perpendicular to a feature axis.
A shallow tree can separate Setosa using petal length, while additional splits help distinguish Versicolor from Virginica.
Without stopping rules, a classifier can continue splitting until it perfectly classifies its training examples.

23:00 Reading the Iris Tree’s Thresholds and Class Counts

A scikit-learn tree with max_depth=2 first splits at a petal-length threshold of 2.45 cm.
The left branch contains 50 Setosa examples; the right branch contains 50 Versicolor and 50 Virginica examples.
A further split at petal width 1.75 leaves a small number of misclassified Iris samples, illustrating that the classes are not perfectly separable at that depth.

29:00 Gini Impurity Measures Class Mixing

For class proportions pᵢ, Gini impurity is 1 minus the sum of pᵢ squared; a pure node scores 0.
A node with class counts 49 and 5 out of 54 has low impurity because one class dominates.
More evenly distributed class counts produce higher impurity, with the maximum depending on the number of classes.

38:00 CART Chooses Splits by Weighted Impurity Reduction

For each candidate feature and threshold, CART calculates the impurity of the left and right child nodes.
The split cost weights each child’s Gini score by the fraction of samples sent to that child, rather than taking a simple average.
The algorithm selects the feature-threshold split with the lowest weighted impurity.

43:00 Greedy CART Search and Training Complexity

Finding a globally optimal tree is computationally impractical as the number of training examples grows, so CART uses a greedy strategy.
CART selects the best current split, then repeats the search independently within each resulting branch until a stopping condition is met.
Prediction follows a path through the tree and is fast when the tree is shallow; training takes more work because it searches over samples and features.

47:00 Entropy as an Alternative to Gini Impurity

Entropy uses class proportions multiplied by their logarithms to quantify class mixing.
Like Gini impurity, lower entropy indicates purer nodes and can guide split selection.
Harter notes that Gini and entropy commonly produce similar trees, though particular datasets may respond differently.

55:00 Regularizing Trees with Depth and Sample Constraints

An unrestricted tree can reach pure leaves and achieve 100% accuracy on its training set while performing poorly on unseen data.
max_depth limits the number of decision levels; other controls include minimum samples for a split or leaf and a maximum leaf count.
A parameter search can compare these constraints to find a tree that generalizes better than one controlled only by depth.

1:05:00 Interpretable Leaves, Class Probabilities, and Regression Trees

Decision trees are white-box models: their tests and leaf assignments make the classification process directly inspectable.
A leaf’s class proportions can serve as estimated class probabilities, expressing confidence based on the training samples that reached that leaf.
DecisionTreeRegressor applies threshold splits to ranges of feature values; deeper trees fit the training targets more closely and can overfit.

1:09:00 Axis-Aligned Limits and the Case for Random Forests

Trees can struggle when the natural decision boundary is not aligned with feature axes; rotating the feature space may improve their fit.
Individual decision trees are often not state-of-the-art classifiers, but collections of trees can be highly effective.
Random forests and other ensembles combine trees built under different constraints, the next topic in Chapter 7.

Transcript excerpt

0:53 Okay, good afternoon. I think I'm about ready to start. Uh, I got a few people here. All right. All right. Um, let's go ahead and start. Um, so, um, yeah. So, sorry about, uh, last week. I hope I went too late on posting that. Something came up again. I've been having a few conflicts. Uh, but, um, all right. Um, almost everybody got in the, uh, assignment five. Um, and I did actually return those uh, today, just now. Um, so you might not have seen it yet, but uh, if not, you know, you might want to check just make certain. Uh most everybody did good except for one or two

1:39 people. Um um I might I was gonna start off reviewing assignment five a little bit. Um but yeah before I get to that so we are working we're up to actually kind of our last unit uh about supervised learning here. Um so I my I do plan today to talk about the some materials from decision trees. Um uh so uh as usual there is uh the next assignment I I also just posted the next assignment so you wouldn't have been able to see it before now but um the uh the invitation link for assignment six on decision trees and ensembles is now um clickable although I haven't tested it yet. So but hopefully it's it's uh working all right. So, um but uh but yeah, most of the the next assignment

2:28 you guys will be working on uh is really more of the uh the chapter and the materials on the ensemble learning. Um so, uh I might not specifically talk about uh that one until next week. So, we'll spend all of our time today talking about decision trees uh for the

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Derek Harter.