CSci574 / AI 520 Machine Learning: Classroom Session 11/19/2026
Overview
Derek Harter explains K-means clustering as an unsupervised learning method, showing how it alternates between assigning each data point to its nearest centroid by Euclidean distance and recomputing centroids as cluster means. He connects the algorithm to Assignment 7, which also requires implementing PCA with singular value decomposition, and explains initialization sensitivity, repeated runs, inertia, and the elbow method for choosing K.
Key takeaways
- K-means alternates two operations: assign each observation to its nearest centroid using Euclidean distance, then replace each centroid with the mean of its assigned observations.
- K-means converges, but random centroid initialization can lead the same dataset and K to different local solutions; multiple runs and score comparison can improve the result.
- The inertia measure summarizes within-cluster squared distances and tends to decrease as K grows, so the elbow in an inertia-versus-K plot can help balance fit against unnecessary clusters.
- Assignment 7 requires from-scratch implementations of K-means and PCA, including PCA normalization, singular value decomposition, dimensionality reduction, and reconstruction.
- The announced Assignment 7 deadline is Sunday, November 29, and Test 2 is planned for the final face-to-face class week, covering topics such as SVMs, decision trees, and unsupervised learning.
Chapters
- Assignment 7 is available, and Assignment 6 evaluations have been returned with limited additional feedback.
- The Assignment 7 deadline remains Sunday, November 29, despite the shortened Thanksgiving week; students with holiday plans are encouraged to start early.
- Test 2 is planned for the final week of face-to-face classes, with the same format as Test 1 and material including SVMs, decision trees, and unsupervised learning.
- The assignment asks students to implement K-means clustering and principal component analysis rather than only call library routines.
- K-means work includes finding each point's closest centroid, recomputing centroids from assigned points, and combining those steps in an iterative algorithm.
- The PCA portion includes feature normalization, singular value decomposition, projecting to fewer dimensions, and reconstructing data from its reduced representation.
- The Hands-On Machine Learning textbook's GitHub repository contains IPython notebooks that reproduce many figures and code examples.
- Harter shifts the class focus from PCA to K-means because Assignment 7's first task concerns clustering.
- The class plans to return to PCA at the next Tuesday meeting before Thanksgiving.
- The Iris dataset illustrates the difference between data with known classes and unlabeled observations.
- Because much real-world data lacks labels, clustering can provide an initial estimate of possible classes before a supervised classification task.
- Clustering also supports anomaly or outlier detection, including potential fraud detection, by identifying observations unlike normal cases.
- A two-feature synthetic dataset makes clusters visually apparent, including three compact groups and two less clearly separated groups.
- K is a meta-parameter specifying the desired number of clusters, so users must choose it before running K-means.
- When clusters cannot be judged visually, trying several K values and comparing a clustering-quality measure can help identify a useful number.
- The example uses scikit-learn's make_blobs to generate 2,000 observations and requests five clusters.
- K-means repeatedly improves an initial set of five centroids, but different random starting points can lead to different final clusterings.
- For the assignment, one suggested initialization is to select K existing data points at random as the initial centroids; another is to sample locations within feature ranges.
- Changing the random seed changed whether the example found the visually expected five groups.
- For each observation, calculate its Euclidean distance to every centroid and assign it the integer label of the nearest one.
- Recompute each centroid by taking the feature-wise average of all points assigned to its cluster.
- Repeat assignment and centroid updates so points can switch clusters as the centroids move.
- The assignment's find-closest-centroids and compute-centroids functions implement these two core steps.
- K-means eventually converges when its centroids stop changing, although the final clustering is not guaranteed to be globally optimal.
- The assignment can use a fixed number of iterations, with roughly five or ten often sufficient for its examples.
- A convergence check can compare successive centroid arrays and stop when changes fall below a tolerance such as 10^-4.
- Different initial centroid selections can converge to different local solutions for the same dataset and K.
- Scikit-learn can run K-means multiple times with different initializations and select the run with the best score.
- The score is based on negative inertia, so lower inertia corresponds to a better fit under the clustering objective.
- The scratch implementation in Assignment 7 does not include these repeated runs, which students would need to add themselves to compare initializations.
- Choosing K data points at random can produce suboptimal starting centroids.
- Scikit-learn uses a more deliberate initialization strategy intended to place initial centroids farther apart.
- Better-spaced starting points can reduce the chance that a single run settles on a poor local solution.
- Inertia measures within-cluster squared distances between observations and their assigned centroids; smaller values indicate tighter clusters.
- Inertia generally decreases as K increases, since additional centroids can bring observations closer to a center.
- Plotting inertia for several K values can reveal an elbow, where the improvement begins to diminish and adding more clusters may offer little benefit.
- The closest-centroid and centroid-recomputation functions are small components, and a loop over centroids is an acceptable implementation approach.
- Students can implement the functions iteratively first, then consider vectorization as an optimization.
- Harter closes by inviting questions and notes that the class meets once more before Thanksgiving, when he expects to revisit PCA.
Transcript excerpt
0:44 Okay. Um I'm going to go ahead and get started. Don't have any too many people yet here, but um I am recording this, so I'll post afterwards. Hopefully I won't have as many problems as we did on Tuesday. So uh first kind of announcements looking ahead. Um I hadn't had the assignment seven uh available until today, but that's available now. So uh and I did get back the assignment sixes. I didn't I don't think I have too much to say on the assignment six, but um uh you can go check your evaluation. Um I didn't have a whole lot of extra feedback from most people. Uh but looking for the assignment seven, so hopefully everybody's aware that we have Thanksgiving uh next week, kind of only
1:33 a half week. Uh but I am going to keep the assignment seven due date for the Sunday 11/29, uh which is actually the, you know, Sunday kind of at the at the end of the uh the weekend for the Thanksgiving break just because we've actually only got one week of face-to-face classes and then a finals week after that. So so for anybody that's listening to this, uh the plan is we're to finish up the supervised, unsupervised learning materials and the assignment seven, you know, over this week and over kind of the half Thanksgiving week with the assignment seven is still going to be due on Sunday, although it might be okay. Uh you know, I mean I would encourage people if you especially if you have Thanksgiving plans to go ahead and start, you know, get the assignment seven started at least like this week,
2:22 this weekend here. [snorts] Uh but but it might be okay if if some people need an extra day or two
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Derek Harter.