UUtah CS 6340 NLP | Fall 2026 | Language modeling
Overview
The lecture introduces language modeling as next-token prediction, showing how the chain rule turns token probabilities into sequence probabilities and how n-gram models estimate them from corpus counts. It explains why sparse counts and long-range dependencies limit n-grams, introduces smoothing and perplexity, then begins deriving a recurrent neural network language model, including embeddings, hidden states, softmax outputs, and the distinction between training inputs and generated tokens.
Key takeaways
- The chain rule makes next-token prediction a method for assigning probability to a complete sequence: multiply each token’s probability conditioned on its preceding context.
- N-gram models trade context for count reliability: a bigram uses only one preceding token, while longer histories create sparse counts and many zero-probability continuations.
- Smoothing gives unseen events nonzero probability, but it cannot restore long-range relationships such as a subject and verb separated by several words.
- Perplexity is a length-normalized measure of held-out prediction quality, with lower values indicating better fit; valid comparisons require identical token vocabularies.
- An RNN carries a hidden state forward to summarize sequence history, combining that state with each new token embedding before producing vocabulary probabilities.
- RNN training commonly uses observed next tokens as subsequent inputs, while generation feeds each sampled token back into the model.
Chapters
- The instructor will review a potentially ambiguous first quiz question and asks students to submit regrade requests for substantial concerns.
- Canvas grades may not reflect bonuses; bonus points will be considered holistically around fall break and again at the end of the course.
- Each practicum is scored out of 100, with practicum parts weighted equally; quiz weighting may vary with coverage and class-wide performance.
- Discriminative models separate classes, while generative models estimate probability distributions over structures such as token sequences.
- A language model predicts the next token from its preceding context, which also provides a way to assign probability to a full sequence.
- Statistical language modeling predates modern LLMs: Frederick Jelinek and IBM colleagues developed n-gram approaches for speech recognition in the 1970s.
- The joint probability of tokens w1 through wn factors into conditional probabilities, each predicting one token from its preceding history.
- A sequence is more likely when its component next-token predictions are likely, making next-token prediction equivalent to modeling the full sequence.
- Sequence probabilities can support speech recognition and spelling correction by favoring plausible or grammatical text.
- An n-gram is a consecutive sequence of n tokens: one token is a unigram, two form a bigram, and three form a trigram.
- A count-based estimate for a continuation divides the count of the history plus candidate token by the count of the history.
- Even a vocabulary of 40,000 tokens yields 40,000^n possible n-token sequences, making reliable counts for long contexts impractical.
- A first-order Markov assumption approximates the next-token probability using only the immediately preceding token.
- An n-gram model retains n minus one preceding tokens, so a bigram uses one-token context and a trigram uses two.
- Shorter histories make counting feasible, but discard information that may be relevant to a continuation.
- A bigram estimate counts ordered pairs such as “cat sat” and normalizes by how often the preceding token “cat” occurs.
- Token order matters: the count for “cat sat” is distinct from the count for “sat cat.”
- Generation starts with a first-token distribution and then repeatedly predicts from the latest token in a bigram model.
- Pure language-model generation can start from a beginning-of-sequence token and produce an entire sequence without a user prompt.
- Prompted generation instead conditions on an existing sequence before predicting the next token, a common practical setup.
- Greedy decoding always selects the most likely token and can produce repetitive text; top-p sampling considers a set of tokens covering a chosen probability mass, such as 98%.
- Unigram and trigram models trained on Shakespeare produce increasingly structured but still unreliable text; the trigram sample retains a Shakespearean tone.
- A model trained on 38 million words from the Wall Street Journal better reflects that domain, but longer generated passages can still become incoherent.
- An unseen n-gram receives zero probability under unsmoothed count estimates, preventing the model from generating that continuation.
- Smoothing assigns probability to events with zero observed counts, typically by adding a small amount of probability mass and renormalizing.
- One described approach adds a small value to each vocabulary item; another redistributes probability from frequent events to less frequent ones.
- Smoothing reduces zero-count failures but does not solve the broader limitations of count-based models.
- In “Gorillas always like to groom their friends,” the pronoun “their” depends on “gorillas,” several tokens earlier than a bigram model can represent.
- In “The computer that’s on the third floor of our office building crashed,” the subject “computer” is separated from its verb by a long phrase.
- Increasing n-gram context length captures more history but makes sequences increasingly unique and their counts increasingly sparse.
- Intrinsic evaluation tests a model on held-out text and checks whether the model assigns it high likelihood after estimating parameters on training data.
- Perplexity normalizes sequence likelihood for length using an inverse geometric mean, so lower perplexity indicates better predictions.
- Perplexity is common in research and scaling-law experiments, but task-specific measures remain necessary for outcomes such as translation or summarization.
- Perplexity comparisons require the same vocabulary because each model’s probability distribution must be defined over the same token set.
- Different tokenizers, BPE merge counts, hyperparameters, or random tie-breaking can produce different vocabularies; comparing models such as Llama and Qwen by raw perplexity is therefore not meaningful.
- The lecture’s Wall Street Journal example reports perplexity around 962 for a unigram model, around 200 for later n-gram models, and about 20 for GPT-3.
- A student asks how the relative-frequency approach estimates the first token when no preceding word is available.
- The instructor describes counting a token’s frequency in the corpus and suggests adding a beginning-of-sequence token to model sentence starts explicitly.
- The recording contains an extended break and mostly indistinct discussion before the lecture resumes.
- Recurrent neural networks offer an alternative to truncating history to a fixed n-gram window by maintaining a hidden representation across positions.
- The course will next add attention to recurrent models and then move to Transformers with causal attention.
- The training corpus can be viewed as supervised examples: use x1 to predict x2, use x1 and x2 to predict x3, and continue across each sequence.
- At each position, the RNN combines the current token’s embedding with a hidden state summarizing earlier tokens.
- The updated hidden state represents both the current input and accumulated history, and is passed to the next sequence position.
- LSTMs and GRUs are related RNN variants, but the lecture focuses on the basic recurrent architecture.
- The first input can be a beginning-of-sequence token, whose embedding is retrieved as a row from an embedding matrix with vocabulary-size dimension V.
- The initial hidden state h0 is commonly initialized to zeros; the next state combines linear projections of the embedding and prior hidden state, plus a bias, followed by a nonlinearity such as sigmoid or ReLU.
- An output matrix projects the hidden vector to V logits, and softmax converts them into a probability distribution over vocabulary tokens.
- At inference time, a sampled token becomes the input to the next RNN position, allowing generation to proceed autoregressively.
- During training, the next observed corpus token is commonly supplied instead of the model’s own prediction, avoiding chains of early random errors.
- The lecture ends before deriving the loss function or fully explaining generation; those topics are deferred to the next class.
Transcript excerpt
0:00 Let's get started. Good afternoon uh everyone. Um I'm excited we are jumping into language modeling which is kind of what this whole course is going to eventually be about. Uh before we do start with this new topic. Um I want to acknowledge that someone did reach out about the ambiguity in the first question in the quiz and I'll look into it as soon as I have more time on my plate. Uh definitely sometime this this week. I agree that something might have been lost uh in my wording uh of the of the question. So I will look into that and then uh yeah I have a huge limitation of not knowing how to deal with canvas. So I personally put all these course in the external sheet and uh I will give you a
0:49 little way of kind of uh thinking about what your grade in the course is. do not rely on canvas uh grades because I do not know how to set up the bonuses there. Uh I mean I didn't even yet decided how I will uh consider them. As I said um from my experience I kind of tend to overinflate the grades with how I I gave bonuses before. So I want to be mindful and cautious of that and likely once uh around the fall break I will apply them and then again at the end kind of looking holistically at uh all of this course which basically for you means um you know study practice and don't overfocus on these bonuses. Uh they might be like a nice pleasant
1:37 surprise but don't over over rely on uh on them. Any questions before we move forward? Yes. Um, one was what was the bonus question and then secondly speaking of ambiguity I felt like for
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.