STAT 4051 Fall 2026: Sparse PCA in Statistical learning

Overview

Kazeem Adepoju explains how sparse PCA modifies conventional PCA to make components easier to interpret by shrinking some feature loadings to zero, trading off some explained variance. The lecture connects PCA to SVD and low-rank reconstruction, demonstrates image compression with truncated singular values, and describes sparse PCA’s reconstruction objective, ridge and lasso penalties, alternating updates, and applications such as genomics and finance.

Key takeaways

PCA loadings describe how original features contribute to a component, while eigenvalues quantify the variance captured by that component.
Truncated SVD compresses an image by retaining only its K largest singular values; increasing K improves reconstruction quality but reduces storage savings.
Sparse PCA improves interpretability by shrinking some loadings to zero, but generally sacrifices some of conventional PCA’s maximum-variance objective.
Lasso encourages exact zero loadings, ridge shrinks loadings without typically eliminating them, and elastic net combines both behaviors for correlated features.
Sparse PCA can be useful for feature selection in high-dimensional datasets, such as 200 patients measured across 10,000 genes, but selected loadings do not prove causal relationships.
Sparse PCA optimization can alternate updates to its component-related matrices until the reconstruction-and-penalty objective stabilizes.

Chapters

0:00 PCA Foundations: Components, Eigenvalues, and Feature Loadings

The lecture frames sparse PCA as a way to make dimension-reduced representations easier to interpret.
Each principal component is a linear combination of the original features, with loadings indicating each feature’s contribution.
Eigenvalues represent the variance associated with each principal component.

7:00 Connecting PCA to Singular Value Decomposition

SVD decomposes a centered data matrix into left singular vectors, singular values, and right singular vectors.
The singular vectors and values provide the directions and variance-related quantities needed to construct principal components.
The decomposition offers a matrix-based route to PCA’s component scores and loadings.

14:30 PCA Analysis, Synthesis, and Low-Rank Reconstruction

Analysis projects observations from the original feature space into a lower-dimensional space using principal-component scores.
Synthesis combines component scores and loadings to estimate the original features.
Keeping only K components produces a low-rank reconstruction that retains major variation while discarding other components.

19:18 Image Compression with Truncated SVD

An image can be represented as a matrix whose entries encode pixel intensities; a 1,000 by 1,000 image has one million entries.
Truncated SVD reconstructs the image using only the K largest singular values, reducing the amount of information stored.
The University of Minnesota logo illustrates the quality trade-off: reconstructions improve as more singular values are retained, with examples at 1, 5, 20, and 50.
Compression introduces reconstruction error, so K should balance visual quality against storage savings.

28:10 Why Dense PCA Loadings Are Hard to Interpret

Conventional PCA can assign nonzero loadings to nearly every variable, making it difficult to identify which features drive a component.
The USA arrests example contains 50 states and four variables, illustrating how dense loadings complicate interpretation.
Sparse PCA aims to set some loadings exactly to zero so a component can be associated with a smaller subset of features.

35:22 Sparse PCA Trades Explained Variance for Interpretability

Conventional PCA maximizes variance, while sparse PCA adds constraints or penalties that encourage zero loadings.
The sparsity penalty can remove less influential features, making components simpler to describe.
That simplicity comes at a cost: sparse components may capture less variance than unconstrained PCA components.

39:41 Lasso, Ridge, and Elastic Net Penalties for Sparse PCA

Lasso regularization can shrink loadings to exactly zero, producing feature selection in the unsupervised PCA setting.
Ridge regularization shrinks coefficients without generally eliminating them, which can help retain correlated variables.
Elastic net combines lasso and ridge penalties to balance sparsity with the treatment of correlated features.
The regression-style sparse PCA objective minimizes reconstruction error while applying penalties to the component representation.

45:09 Alternating Updates and the Sparse PCA Trade-Off

The described optimization initializes from an ordinary PCA direction, then alternates updates to matrices A and B until the objective stabilizes.
Sparse PCA can produce components with fewer active variables, but their explained variance may be below that of ordinary PCA.
Sparse components are not necessarily mutually orthogonal, unlike the conventional PCA directions emphasized in the lecture.
A covariance-matrix example shows that forcing only one nonzero coordinate can substantially reduce projected variance.

48:42 Choosing Sparsity and Applying Sparse PCA

The number of components and the sparsity level must be chosen together, with reconstruction error as one criterion.
For 200 patients measured across 10,000 genes, sparse PCA could reduce a component to a much smaller set of relevant genes.
Other proposed applications include finance, medical imaging, and high-dimensional customer or sensor data.
Sparse loadings support exploratory variable selection, but they do not establish causality.

51:13 Next Topic: Completing Matrices with Missing Data

PCA and related algorithms generally require a complete data matrix, but real datasets may contain missing entries.
Matrix completion will address how to estimate missing values before applying methods such as PCA or sparse PCA.
The lecture identifies matrix completion as the topic planned for the next class.

Transcript excerpt

5:21 Okay. Uh good morning everyone. I want to welcome you to another statistical machine learning task. Of course, uh the focus today will be on sparse principle component analysis. Uh before now we started principal component analysis and of course uh we don't need to spar PCA. So um today I'm actually going to justify the reason why we need sparse PCA. So in the outline I'm actually um going to explain the

6:10 dense PCA loadings and why sparse representation can improve interpretation. Not only that, I'm also going to discuss cardality constraints that will be involved and I'm going to walk you through the optimization objective of sparse PCA. Then I'm also going to walk you through um what how that connect to singular value de composition to image uh compression missing data and matrix completion. So let's get started. You know I wanted to pay attention.

7:00 I remember on Monday I said I was going to go over the singular value de composition how it connect to PCA. I think I first start from there before into before going into sparse PCA. Now we with principal component analysis of course uh our intention is actually um to find a way uh to reduce the dimension of in our data without a lot of features and is there a way we can artificially fabricate

7:48 dimensions fewer dimensions that represent them. And of course to be able to do this we need a way to figure out the principal component in the analysis of uh principal component. We need uh two important things. Number one is the eing values. Number two is actually the eing vectors. The vectors are the loadings. The IG values you know we give the variance of each principal component but the vector will actually represent

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof Kazeem Adepoju.