Lecture 2: Probability Background (KAIST AIC313/CS378, Fall 2026)
Overview
Minhyuk Sung reviews the probability and statistics foundations used throughout KAIST AIC313/CS378, including random variables, probability distributions, Bayes' rule, and likelihood. He connects these tools to generative modeling through maximum-likelihood estimation for categorical and Gaussian data, then motivates autoregressive models by showing why sequence distributions become too large to model directly.
Key takeaways
- A continuous PDF value is a density, not a probability; probability for an interval comes from integrating the density, while the probability of any exact point is zero.
- Bayes' rule updates a latent-variable prior p(z) into a posterior p(z | x) using likelihood p(x | z) and evidence p(x), a framework that supports sampling latent variables and generating data.
- Maximum-likelihood estimation for categorical data assigns each class its observed frequency, π_k = n_k / n.
- For IID Gaussian samples, the maximum-likelihood mean is the sample average and the covariance uses deviations from that mean with normalization 1/n.
- Maximizing likelihood is equivalent to minimizing negative log-likelihood; the logarithm converts a product over samples into a sum suitable for optimization.
- A length-n token sequence over a vocabulary of K items has K^n possible combinations, motivating autoregressive factorization into manageable conditional predictions.
Chapters
- The course goal is to learn an unknown data distribution from training samples.
- Likelihood measures how compatible observed data are with a proposed distribution model.
- The lecture reviews probability concepts used across machine learning, including random variables, distributions, and maximum-likelihood estimation.
- A random variable is a function mapping outcomes in a sample space to values, not the value itself.
- Counting how many times a die shows 1 in three rolls gives a discrete random variable; next-token IDs in language models are also discrete.
- A probability mass function assigns probabilities to discrete values, with nonnegative entries summing to 1.
- A probability density function describes continuous variables: probabilities come from integrating over intervals, and the density at one exact value is not itself a probability.
- Expectation is a probability-weighted average, computed by summation for discrete variables and integration for continuous variables; mean and variance build on it.
- Maurice Allais's choice example compares a guaranteed $1 million with an 89% chance of $1 million, 10% chance of $5 million, and 1% chance of zero.
- A second choice contrasts an 11% chance of $1 million with a 10% chance of $5 million; the differing preferences illustrate the Allais paradox and challenges to expected-utility consistency.
- A generative model must both represent a distribution and make it possible to sample from that distribution.
- A joint distribution describes how two random variables vary together, using a probability table for discrete variables or a density over a product space for continuous ones.
- Marginalization sums or integrates over one variable; the lecture's table exercise obtains P(X = 1) = 0.5 and an expected value of X equal to 1.7.
- A conditional distribution gives the probabilities of one variable after fixing a value of another.
- For discrete variables, compute P(X | Z) by dividing the relevant joint probability by the marginal probability of Z.
- The joint distribution can be recovered by multiplying a marginal distribution by the corresponding conditional distribution.
- Bayes' rule relates the posterior p(z | x) to the prior p(z), likelihood p(x | z), and evidence p(x).
- The prior describes beliefs about Z before observing X; the posterior updates those probabilities after observing X.
- Many generative models introduce a latent variable Z, sample it from a simple prior such as a standard Gaussian, then generate data X conditionally.
- The evidence normalizes the posterior and can be difficult to compute, making these relationships important in model design.
- Two variables are independent when their joint distribution factors into the product of their marginal distributions.
- Equivalently, conditioning on one independent variable does not change the distribution of the other.
- Conditional independence allows X and Y to become independent after observing a third variable Z, a structure used in probabilistic models.
- A categorical distribution over K classes uses probability parameters π whose entries sum to 1; language models predict a categorical distribution over next-token IDs.
- A Gaussian distribution is specified by its mean and variance, while a standard Gaussian has mean zero and covariance equal to the identity matrix.
- A multivariate Gaussian uses a covariance matrix; a nonsingular full-dimensional Gaussian requires that matrix to be symmetric and positive definite.
- Isotropic or diagonal covariance matrices simplify modeling, while covariance eigenvectors identify principal directions.
- Likelihood treats the probability or density of observed samples as a function of model parameters, such as categorical probabilities or Gaussian mean and variance.
- For independent, identically distributed samples, the likelihood is the product of each sample's probability or density.
- Taking the logarithm changes the product into a sum; maximizing log-likelihood is equivalent to minimizing negative log-likelihood.
- For complex models, minimizing average negative log-likelihood becomes a loss optimized with methods such as gradient descent.
- For K categories and n observations, let n_k count the samples in category k.
- The maximum-likelihood estimate sets each category probability to π_k = n_k / n.
- This estimate fits the categorical distribution to observed frequencies and can then be used to sample new category values.
- Given IID samples from a Gaussian with unknown parameters, maximum likelihood estimates the mean with the sample average.
- The covariance estimate is the average outer product of deviations from that mean, using the maximum-likelihood normalization of 1/n.
- These closed-form estimates illustrate that familiar statistics arise from maximizing the probability density assigned to the observed samples.
- Categorical and Gaussian maximum-likelihood models assume a known, simple distribution family, but images, videos, and proteins generally do not follow a single Gaussian.
- A sequence of n tokens from a vocabulary of K items has K^n possible outcomes, making direct joint-distribution modeling impractical.
- Autoregressive models address this scale problem by decomposing a sequence distribution into conditional distributions over successive tokens.
- The recap emphasizes the distinction between discrete probability mass and continuous probability density, plus marginalization, conditioning, and independence.
- Prior, likelihood, evidence, posterior, and negative log-likelihood are identified as recurring concepts for the semester's generative models.
- Minhyuk Sung previews autoregressive models for the next lecture and notes that future lectures may include quizzes submitted through KMS.
Transcript excerpt
0:03 Okay. So, welcome to the second uh lecture of the uh introduction to the model modeling. So today um we are going to have some kind of review of the background of the probability and the statistics and how uh some of the other people who took the machine learning course or the deep learning course was there some kind of the some those kind of the review of the some background of the probably in the statistics. Yes. Anyone who took the introduction to machine learning course or introduction to AI was there any kind of some review for the probably the statistics? Yes. Yeah. Then there might be lots of some overlap in terms of dealing with those kind of things. Actually the reason like why we want to cover those kind of the
0:51 basic things is that actually not only uh for the generative modeling but for the many aspects in the machine learning like everything is basically based on the probability and also the statistics. So as we also discussed last time uh basically what we want to do with the uh in the general modeling is that we want to basically learn kind of the distribution of the the data right so that's basically the case that we only have some kind of the samples which are basically our training data which are coming from some kind of underlying distribution that we want to know which is basically not known. Uh so our goal in this modeling is to learn that distribution uh in a way that the we can see that some kind of the the samples that are basically given as kind of
1:38 training data are basically compatible with the basically our model for that distribution. So we're going to keep uh you know discussing those kind of the
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Minhyuk Sung.