CSci574 / AI 520 Machine Learning: Classroom Session 10/27/2026
Watch on YouTube →
Overview
Derek Harter introduces K-nearest neighbors (KNN) as a supervised, non-parametric method for classification and regression, contrasting its example-based predictions with parametric models such as linear and logistic regression. He explains distance metrics, choosing K, feature scaling, computational costs, evaluation, and scikit-learn usage, then reviews the three tasks in Support Vector Machine Assignment 5 and previews Naive Bayes.
Key takeaways
- KNN predicts by comparing a new example with stored labeled examples: classification typically uses a vote, while regression typically averages neighboring target values.
- KNN has almost no substantive training phase, but direct prediction requires distance comparisons against stored examples, so inference can become expensive as the dataset grows.
- Because KNN relies on feature distances, features with larger numeric ranges can dominate; split the data first, then fit scaling only on training data to avoid test leakage.
- The choice of K affects bias and performance, so tune it using an appropriate validation metric such as regression error or classification F1 rather than assuming one value works for every dataset.
- Scikit-learn's Minkowski distance with p = 2 is Euclidean distance, and p = 1 is Manhattan distance; KNeighborsClassifier and KNeighborsRegressor provide the task-specific estimators.
- KNN can serve as a useful non-parametric baseline alongside linear and logistic regression, but high dimensionality and large dataset size can limit its effectiveness.
Chapters
- Harter places KNN and Naive Bayes among supplemental algorithms not covered in detail by the course textbook.
- The upcoming course sequence includes decision trees and random forests before unsupervised learning.
- Support Vector Machine Assignment 5 has three tasks and is due at the end of the week; students are encouraged to accept and review it.
- KNN supports both classification and regression without assuming a linear relationship between features and outcomes.
- Unlike linear or logistic regression, KNN does not learn a compact set of parameters such as feature weights.
- Its intuitive approach resembles estimating an apartment's rent by comparing similar nearby apartments.
- For a new example, KNN measures its similarity to labeled examples and selects the K closest neighbors.
- With K = 3 in a two-class illustration, the predicted class comes from a majority vote among the three neighbors.
- For regression, the simplest prediction is the average of neighboring target values; distance-weighted averages give closer examples more influence.
- KNN is called lazy learning because it does no substantive model fitting in advance; prediction requires comparing a query with stored examples.
- Prediction cost grows with the number of stored examples, making a direct scan burdensome for datasets with millions of rows.
- Euclidean distance can become uninformative with many features, so high-dimensional data may require another similarity measure or approach.
- K is a hyperparameter that should be tuned against validation performance for the specific classification or regression task.
- KNN is sensitive to feature scales: a large-range feature can dominate Euclidean distance and obscure smaller-range features.
- StandardScaler or another suitable scaling method should be used so features contribute comparably to distance calculations.
- KNN is useful as a non-parametric comparison model, even though it may be impractical for very large datasets.
- A hand-built implementation calculates Euclidean distances from one query to every stored example, then selects the K smallest distances.
- The example sorts distance-example pairs and slices out the nearest neighbors; sorting adds work beyond calculating distances.
- Classification counts neighbor labels and returns the most common class, while regression averages neighbor targets.
- A production-quality implementation could vectorize distance calculations with NumPy and explicitly handle ties or distance weighting.
- Scikit-learn provides separate KNeighborsClassifier and KNeighborsRegressor estimators because their prediction rules differ.
- Calling fit for KNN mainly stores the training examples and their labels for later neighbor searches.
- Split training and test data before fitting StandardScaler; scaling the full dataset leaks test-set statistics into preprocessing.
- The Iris dataset is used in the course notebook, while the recommended reading also demonstrates California housing data.
- Scikit-learn's Minkowski metric with p = 2 corresponds to Euclidean distance, while p = 1 gives Manhattan distance.
- Regression performance can be compared with mean absolute error, mean squared error, root mean squared error, or R-squared.
- A California housing example illustrates tuning K by comparing errors across neighbor counts, with intermediate values outperforming values that are too small or large.
- Classification KNN predicts a label by voting among neighbors; scikit-learn uses KNeighborsClassifier for this task.
- Accuracy can be misleading, so precision, recall, and F1 may be more useful when comparing classifiers or tuning K.
- An example bins house values into four categories and evaluates F1 across different K values, with a value near 15 performing well in that dataset.
- KNN and K-means both require choosing K, but KNN uses neighbors for supervised prediction whereas K-means uses K to specify clusters.
- Assignment 5 Task 1 plots a dataset containing outliers and examines how SVM regularization changes the decision boundary.
- Task 2 implements Gaussian kernel similarity, which maps distance to a value between zero and one; scikit-learn parameterizes the kernel with gamma rather than sigma.
- Task 3 uses a second, more complex dataset to fit a nonlinear SVM decision boundary with a kernel function.
- Harter closes by previewing a Naive Bayes discussion for the next class session.
Transcript excerpt
0:07 All right. >> Uh, it's about that time. I'm going to get started. So, uh, for my online people, are you guys able to hear me okay? Or not. Anybody? Yeah? Okay. I'll take that as yes. Yeah, I got somebody thumbs up. Okay. All right. All right. Yeah, good afternoon. So,
0:54 I'll get going and get started. So, kind of reminders where we're at here. This is sort of just a few additional machine learning algorithms for this week that I kind of put up there. We don't really have these in our hands-on machine learning textbook, our official course textbook. So, I'm just going to talk a little bit about the KNN and naive Bayes and I don't know. I mean, you know, I might give a question on this on the test two or maybe not, but just for some examples cuz yeah, if you look at the textbook, I should have brought my textbook here. Um, um, oops. Let's bring up our textbook real quickly here. Um, you know, so about the other
1:43 about the last half of it is really neural networks and oh, and you know, kind of a plug. Um, so we'll probably be doing our neural networks course in the spring or maybe summer. That's a good it to take after you do machine learning, but but yeah really we only look at kind of the first half of this book and and on our hands on machine learning textbook kind of really we only specific he only specifically covers you know linear and logistic regression and support vector machines but then after this week we are going to get into some other non-parametric models so looking at decision trees and random forests before we get into unsupervised learning. But but yeah I always like to throw in a
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Derek Harter.