CSci574 / AI 520 Machine Learning: Classroom Session 10/22/2026
Overview
Derek Harter explains how support vector machines use feature transformations and kernels, especially the Gaussian RBF kernel, to create nonlinear decision boundaries. He connects kernel parameters C, gamma, and sigma to model behavior, explains the kernel trick and its computational tradeoffs, and previews Assignment 5, which has three tasks involving SVM fitting and a hand-written Gaussian kernel.
Key takeaways
- A linear SVM can produce a nonlinear decision boundary after features are transformed, for example by adding degree-3 polynomial terms or RBF landmark similarities.
- For the RBF kernel exp(-gamma ||x-L||^2), larger gamma creates narrower, more localized similarities; Andrew Ng's sigma formulation converts through gamma = 1/(2 sigma^2).
- Small C allows wider margins and more training violations, whereas large C penalizes violations more heavily and can encourage overfitting.
- A standard RBF SVM can use every training sample as a landmark, so its implicit feature-space dimension can grow with the number of samples.
- The kernel trick computes transformed-space inner products using kernel functions, avoiding explicit feature expansion, but kernel SVM training can still scale between quadratic and cubic in sample count.
- Assignment 5 asks students to implement a Gaussian kernel and verify that a custom callable matches scikit-learn's built-in RBF kernel when sigma and gamma are converted consistently.
Chapters
- Derek Harter reminds students to accept and start Assignment 5, which covers support vector machines and is due the following weekend, not the upcoming one.
- He notes that students have mainly committed a README identifying their final-project dataset and encourages them to begin substantive work.
- Harter recommends committing project progress weekly rather than leaving all work until the final week.
- Organize notebooks by purpose, such as data exploration, data cleaning, and model experiments, instead of putting everything in one notebook.
- Move reusable Python functions, including data-loading and cleaning code, into project modules or a source directory.
- Include an environment.yml and README instructions so others can recreate the environment and reproduce project results.
- Harter recaps that support vector machines are also called large-margin or soft-margin classifiers.
- Scikit-learn's SVC, SVR, LinearSVC, and LinearSVR use C to control the tradeoff between fitting training data and regularization.
- C is inversely related to regularization parameters called alpha or lambda in other course materials: smaller C imposes stronger regularization.
- Using scikit-learn's make_moons dataset, Harter demonstrates classification data that cannot be separated by a straight line in its original two-feature space.
- Adding polynomial features through degree 3 gives a linear SVM a richer feature space and produces a nonlinear decision boundary in the original space.
- SVMs are sensitive to feature scales, so Harter recommends scaling features before fitting them.
- Scikit-learn's SVC supports linear, polynomial, RBF, and sigmoid kernels, and can also accept a custom callable kernel.
- The degree parameter applies to polynomial kernels, while coef0 adjusts the contribution of lower- versus higher-degree terms.
- A small C permits a wider margin and more margin violations, while a large C pushes the fit toward fewer violations and can overfit.
- Harter presents the scikit-learn RBF formula: exp(-gamma times the squared Euclidean distance between a feature vector and a landmark).
- The RBF output ranges from 0 to 1: it equals 1 at the landmark and decreases as the feature vector moves away.
- Andrew Ng's lectures describe the same function using sigma, with gamma = 1/(2 sigma^2).
- A large gamma makes RBF similarity fall off quickly, creating a narrow, localized response around each landmark.
- A small gamma makes similarity decrease more gradually, so distant points retain higher similarity values.
- Scikit-learn's gamma parameter controls this width for the RBF kernel; it is also used by polynomial and sigmoid kernels.
- A kernel can map original features into a space with a different number of dimensions, just as polynomial features do.
- In Harter's one-dimensional example, two landmarks at -2 and 1 create two new features measuring similarity to each landmark.
- The transformed points become linearly separable, allowing a linear classifier to create a nonlinear boundary in the original space.
- A standard RBF SVM effectively uses each training sample as a landmark, so 100 samples can yield 100 similarity features.
- This can expand a low-dimensional input into a space with one dimension per training example.
- That expansion can become impractical for datasets with millions of samples, although the kernel trick avoids explicitly constructing all transformed features.
- Harter recommends trying an RBF kernel first for many nonlinear problems, then considering specialized kernels when the data or task calls for them.
- Gamma controls the locality of RBF similarity, while C controls the penalty for margin violations and therefore the margin-versus-fit tradeoff.
- Small gamma produces smoother, less localized effects; large gamma allows the boundary to respond to finer details and can increase overfitting risk.
- Harter compares combinations of C and gamma on make_moons data to show how both hyperparameters shape the decision boundary.
- LinearSVC scales roughly linearly with dataset size and feature count, making it a better choice for very large datasets when a linear boundary suffices.
- Kernel SVM training can scale between the square and cube of the number of samples, making it less suitable for truly large datasets.
- The kernel trick computes inner products in transformed feature space through a kernel function, avoiding explicit construction of potentially high-dimensional features.
- This shortcut makes polynomial and RBF SVMs practical on medium-sized datasets, but does not eliminate their scaling costs.
- SVR adapts the large-margin objective for regression by seeking a narrow margin that contains as many target values as possible.
- A linear SVR can produce a line similar to linear regression, though its slope and intercept need not match ordinary least squares.
- Applying a polynomial kernel to SVR produces a nonlinear regression fit using the same kernel-based feature transformation idea.
- Assignment 5 has three tasks: examine how C affects an SVM fit and outliers, implement a Gaussian kernel using sigma, and fit a nonlinear boundary with built-in and custom kernels.
- The final task compares scikit-learn's built-in RBF kernel with a hand-written callable; converting sigma to gamma correctly should yield the same decision boundary.
- Harter previews next week's k-nearest neighbors and naive Bayes, two approaches that differ from the parametric models covered so far.
- In the closing discussion, a student asks about changing their final-project dataset and waits for Harter's approval before committing to the replacement.
Transcript excerpt
0:07 Good. Okay. Um Yes, let's go ahead and get started. Um So, Um So, my plan for today, I'm going to go ahead and finish up talking about a few things on support vector machines. Um but uh before uh I get back into support vector machines, uh just a quick reminder about um activities. So, uh for one, um Uh let me check here, make certain that the uh the uh um assignment five is actually available here. I I don't think I've checked to make certain that uh uh
0:52 or look to see if people have accepted yet. So, you know, as usual, I do always um encourage you to, you know, get started early on assignments. Uh if you haven't accepted the assignment yet, you might want to go ahead and make it a uh um >> [snorts] >> make a point to at least accept it and start looking through it. Um So, so yeah, it should be available. Though, you do still have all of next week uh for the uh assignment five on support vector machines. Make certain that um it's actually working here if I clone it from the start. >> [snorts] >> So, I'll go ahead and open it up and
1:40 reopen the container here just to check that uh everything's working. So, uh >> [clears throat] >> um >> [snorts] >> So, yeah, uh maybe I'll I'll probably won't wait for that to finish up here, but yeah, do get started on assignment five. Maybe at the end once I'm done talking about support vector machines, I might go back and at least just look at the questions here. There's kind of like two different two or three different tasks on this one with some subtasks for each one. So most people hopefully won't find the
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Derek Harter.