CSci574 / AI 520 Machine Learning: Classroom Session 09/10/2026
Overview
Derek Harter explains Assignment 2 implementation requirements for linear regression and binary logistic classification using scikit-learn and statsmodels, including weather-feature encoding, mean imputation, and plotting a learned decision boundary. He then introduces classification evaluation with MNIST examples, five-fold concepts for cross-validation, class-imbalance baselines, confusion matrices, precision and recall, and ROC or precision-recall curves.
Key takeaways
- For Assignment 2, explicitly encode RainTomorrow as No = 0 and Yes = 1, and mean-impute missing Sunshine and Pressure3pm values before fitting the classifier.
- A logistic regression with Sunshine and Pressure3pm has two feature coefficients and an intercept; these coefficients can be rearranged to calculate and plot the linear decision boundary.
- Always predicting not-five on MNIST can exceed 90 percent accuracy because fives account for slightly less than 10 percent of the examples, making accuracy an inadequate standalone metric.
- Three-fold cross-validation estimates generalization by rotating the held-out subset, while a separate test set can be reserved for final evaluation after choosing a model.
- A confusion matrix separates true positives, true negatives, false positives, and false negatives, revealing mistakes that aggregate accuracy conceals.
- Adjusting a classifier's decision threshold trades precision against recall; ROC and precision-recall curves help compare models across that tradeoff, while application-specific error costs determine the preferred balance.
Chapters
- Implement the Task 1 regression functions in the assignment source files, fitting one model with scikit-learn and another with statsmodels.
- Return the fitted model and results such as the intercept, slope, R-squared, and mean squared error.
- The function argument order differs: the scikit-learn version takes X then y, while the statsmodels version takes y then X.
- Commit work separately for functions such as the two Task 1 implementations, and commit notebook progress incrementally.
- Task 2 uses logistic regression in scikit-learn and the Logit model in statsmodels to predict the binary RainTomorrow label.
- The classifier uses Sunshine and Pressure3pm as its two input features, so the fitted model has two feature coefficients plus an intercept.
- Encode RainTomorrow as a binary target, explicitly mapping No to 0 and Yes to 1.
- Use scikit-learn preprocessing tools to identify missing Sunshine and Pressure3pm values and impute each feature with its own mean.
- Check encoder label ordering rather than assuming a categorical encoder will map No and Yes to the intended values.
- Plot Sunshine against Pressure3pm and use point color or shape to distinguish RainTomorrow values.
- A two-feature logistic regression has an intercept and two coefficients; rearranging the coefficient equation gives the boundary value of one feature for each value of the other.
- Plotting the linear boundary against the labeled samples helps assess whether the weather classes appear linearly separable.
- MNIST contains 70,000 handwritten digit images, each represented by 784 pixel values from a 28 by 28 image.
- The dataset is a widely used machine-learning benchmark; the class previously inspected a smaller 8 by 8 digit example.
- Predicting digits 0 through 9 is a multiclass task, though some classifiers require multiple binary classifiers to handle it.
- Scikit-learn can handle multiclass tasks for some estimators by combining binary classifiers, such as digit-versus-not-digit models.
- MNIST is already shuffled, but a manual train-test split can fail if samples are ordered by class, leaving some classes out of one split.
- Turning MNIST into a five-versus-not-five task creates an imbalanced binary dataset because fives make up slightly less than 10 percent of examples.
- Create binary labels with vectorized NumPy operations, assigning 1 to digit fives and 0 to all other digits.
- Fit scikit-learn's SGDClassifier to the training images and binary labels; the estimator uses the supplied labels to determine the classification task.
- Use the classifier's predict method on images in the same feature format as the training data and compare predicted labels with the known labels.
- K-fold cross-validation divides training data into K parts, trains on K minus 1 parts, and evaluates on the held-out part in turn.
- The classroom example uses three-fold cross-validation and reports an average accuracy close to 96 percent.
- Because fewer than 10 percent of the binary examples are fives, always predicting not-five already exceeds 90 percent accuracy, so raw accuracy alone can exaggerate model quality.
- In scikit-learn's confusion matrix, rows represent true labels and columns represent predicted labels.
- For five detection, the diagonal contains true negatives and true positives; off-diagonal cells contain false negatives and false positives.
- A ten-digit MNIST confusion matrix has 10 rows and 10 columns, and normalizing counts into percentages can make class-level errors easier to interpret.
- Precision is the fraction of positive predictions that are true positives, so it falls when the model produces false positives.
- Recall is the fraction of actual positives correctly found, so it falls when the model produces false negatives.
- The F1 score combines precision and recall into one measure when a single summary is preferred.
- The cost of false positives and false negatives depends on the application; in cancer screening, missing a true case can be more harmful than triggering an unnecessary follow-up test.
- Moving a classifier's decision threshold changes its precision-recall balance and the rates of false positives and false negatives.
- ROC and precision-recall curves compare performance across thresholds; a stronger model, such as the discussed Random Forest, can outperform an SGD classifier across the curve.
Transcript excerpt
0:06 Hey. Uh, well, I'm back. My computer's responsive again. So, let's try this again here. Uh, let me share my screen. All right. Um, well, so just jumping back to where I was. Um, I was just mostly talking about, you know, the high stuff. Um, so let me know. Don't know if
0:55 everybody got back on here or not. I got a lot of people uh online at the moment. So um but again I encourage you you know if you have questions I'm mostly just doing this to see if anybody wants to ask a question about uh anything. So uh but um um in general on this assignment basically you have to implement uh like a linear regression first of all. So you're going to put that in your scikit into the task one sklearn linear regression function uh which will be in the same place that you had to do stuff on the first assignment. So when you're uh source assignment tasks uh you should have uh all these functions here. Um,
1:49 so I think I must have had a solution there for for the uh the the um for the scikitle learn one. Uh I don't know why that one's in there, but um um and then you know I pretty much ask you to do the same thing uh but using stats model. Uh so you know you might have to go through and look um uh you know so you might need to do some of these you might be able to reuse like scikitle learn stuff uh you have to create the model using stats model for the second part but um uh how you pull out the information like what the resulting intercept and slope were uh will be different you'll have to get
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Derek Harter.