STAT 4051 Fall 2026: Factor Analysis in Statistical Learning
Overview
Prof Kazeem Adepoju explains factor analysis as a way to identify latent factors behind correlations among observed variables, modeling covariance as shared factor variance plus variable-specific uniqueness. He compares principal component and principal factor extraction with maximum likelihood, then uses an R example with 112 students and six ability measures to select two factors, interpret loadings, and improve interpretation with promax rotation.
Key takeaways
- Factor analysis models a P-variable covariance structure through shared latent-factor variation and uniqueness, the variance specific to each observed variable.
- For six observed variables, the lecture's example has four model degrees of freedom with two factors, zero with three factors, and an overparameterized case with four factors.
- In the 112-subject, six-measure R example, a two-factor model is not rejected at the 0.05 level with p=0.191, while the one-factor model is rejected.
- Promax rotation improves interpretation by creating a more distinct loading pattern while preserving the model's covariance structure.
- Bartlett weighted least squares and regression are two methods for estimating factor scores for individual observations.
Chapters
- Prof Kazeem Adepoju recaps factor analysis as a method for finding hidden factors that may explain relationships among observed variables.
- The lecture continues a previous class session, which introduced a statistical model for decomposing covariance into shared and variable-specific components.
- Adepoju notes that the class is remote because he has the flu and says he plans to return in person Friday with exam booklets.
- The model represents covariance using shared variance from factor loadings and uniqueness, the variable-specific variance.
- For P observed variables, a symmetric covariance matrix contains P(P+1)/2 distinct entries; with P=5, that gives 15.
- Estimating factor loadings and uniqueness lets the model reconstruct the observed covariance structure.
- Degrees of freedom measure how much covariance information remains after estimating factor-model parameters.
- Adepoju uses degrees of freedom to assess whether a model is estimable, whether it uses too many factors, and whether its fit can be statistically tested.
- A negative degree of freedom indicates an overparameterized model; zero leaves no residual information for a conventional model-fit test.
- Factor analysis seeks K<P, retaining fewer factors than observed variables and preserving nonzero uniqueness.
- The sample covariance matrix is the starting point for estimating factor loadings and uniqueness.
- If the covariance matrix is approximately diagonal, its off-diagonal entries indicate little relationship to explain, so factor analysis offers little value.
- The goal is to make model-implied covariance close to sample covariance, minimizing their discrepancy, described using the Frobenius norm.
- The principal component method applies eigen decomposition to the sample covariance matrix and retains the K largest eigenvalues and corresponding eigenvectors.
- Retaining fewer than all P components leaves nonzero uniqueness, unlike using all components, which reproduces the covariance without a specific-variance component.
- The principal factor method begins with an estimate of error or uniqueness, performs an eigen-based extraction, updates the estimate, and iterates until results stabilize.
- The statistical approach assumes multivariate Gaussian factors and errors, then estimates loadings and uniqueness by maximizing the likelihood.
- Taking the logarithm of the likelihood makes the objective more workable for optimization.
- Adepoju identifies the expectation-maximization algorithm as a way to obtain factor-loading and uniqueness estimates.
- Unlike principal component analysis, factor count is not chosen simply by a percentage-of-variance-explained threshold in this factor-analysis procedure.
- The likelihood-based test compares covariance implied by the model with covariance estimated from the data, using a chi-squared reference distribution and model degrees of freedom.
- The selection process tests candidate factor counts and looks for a model that is not rejected while retaining positive degrees of freedom.
- The example uses 112 subjects and six measures: general ability, picture, block, maze, reading, and vocabulary.
- The sample covariance and correlation matrices have nonzero off-diagonal entries, indicating relationships among the six measures.
- Four factors produce an error in the example; three factors have zero degrees of freedom, leaving no information for the stated fit test.
- With two factors, the model has four degrees of freedom and a reported p-value of 0.191, so it is not rejected at the 0.05 significance level.
- A one-factor model has a p-value below 0.05 in the R example, so it is rejected as inadequate for reproducing the observed covariance.
- The two-factor output reports factor loadings and uniqueness values, separating shared variance from variable-specific variance.
- Unrotated loadings are difficult to interpret when variables load highly on multiple factors; rotation seeks a clearer pattern without changing the covariance structure.
- With promax rotation, factor one is associated with general ability, reading, and vocabulary, while factor two is associated with general ability, picture, blocks, and maze.
- Factor scores assign each observation coordinates along the estimated factor directions, analogous to principal component scores in PCA.
- Adepoju names Bartlett weighted least squares and regression as factor-score methods.
- He briefly describes a Thomson scoring approach using a normal prior for factors and a posterior distribution, noting that the details are beyond the class scope.
- Adepoju closes by previewing probabilistic graphical models for Friday and reminding students to collect their exam booklets.
Transcript excerpt
0:00 Are you able to see? >> you. >> Oh, thank you so much. Oh, thank you for confirming that. Um I want to welcome every one of you to my lecture today. Of course, uh today uh I pick uh a flu, so that's why we are having um this lecture remotely. And of course, I'm going to be back in person on Friday. I'm actually going to be back in person on Friday. And on Friday, I'm going to come to class with your exam booklets. I'm going to uh you know, return your exam booklets so that you actually going to see um you know, we are I mean, what's how how
0:50 your grade goes. Uh I remember on Monday, uh we were we started factor analysis. And when we get to this point, we realized that the the temperature was too high, was no longer comfortable. So, uh because of that, I'm actually going to start from where we stopped. But you know what? Um alpha force uh have a recap on this. Okay, so um when you're talking about factor analysis, we actually um want to figure out we want to
1:38 identify um some meeting factors, you know, that's maybe responsible uh for reason why variables correlate. I'm going to say I'm going to say this again. Um uh let me see. Um I'm going to I me reshare this. I think I lost that. I'm actually, you know, reshare that. One second. Okay, can you all confirm if you can see what I'm sharing right now? Is everybody seeing that? Is every If everybody's seeing my slide? >> Yes, you can see it.
2:26 >> Thank you so much. Thank you for confirming that. So, the the going factor analysis is if relationship exists among variables,
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof Kazeem Adepoju.