UUtah CS 6340 NLP | Fall 2026 | RNN LMs
Overview
The lecture develops recurrent neural network language models from next-token prediction and negative log-likelihood through teacher forcing, sequence batching, and decoding methods such as beam search, temperature, top-k, and nucleus sampling. It then connects language modeling to machine translation and encoder-decoder sequence-to-sequence systems, while clarifying that teacher forcing does not let an RNN compute across time in parallel because each hidden state depends on the previous one.
Key takeaways
- RNN language-model training predicts the next token at every position, indexes the target probability by vocabulary ID, and aggregates token losses into sequence and batch losses.
- Teacher forcing feeds the observed previous token rather than the model's sampled prediction, but it does not parallelize RNN computation across time because each hidden state depends on its predecessor.
- Start-of-sequence and end-of-sequence tokens teach a language model how to begin and stop; inference also needs a maximum output length as a safeguard.
- Nucleus sampling selects a variable-sized set of top-ranked tokens until their cumulative probability reaches a chosen threshold, avoiding the fixed candidate count used by top-k sampling.
- Encoder-decoder sequence-to-sequence models represent a source sequence with an encoder and condition a decoder on that representation to generate a target sequence, as in machine translation.
Chapters
- The lecture reviews neural language modeling after n-gram models, whose limited history makes them weak at long-term dependencies.
- An RNN combines the current token embedding with the prior hidden state to represent the history seen so far.
- A linear output layer and softmax produce probabilities over the vocabulary for the next-token prediction.
- At position i, the model predicts the token at position i+1 using the context through position i.
- The loss is the negative log of the probability assigned to the observed next token, so minimizing it maximizes that token's likelihood.
- The target probability is selected from the softmax vector using the target token's vocabulary ID, not its position in the sentence.
- Rows in the output weight matrix correspond to vocabulary items in the same index order used by the embedding matrix.
- A sequence has a next-token loss at each prediction position; these token losses are summed to form an instance loss.
- Training then combines or averages instance losses across a batch before backpropagation, rather than updating weights separately for every token.
- Teacher forcing feeds the actual token from the training sequence into the next step instead of a sampled model prediction.
- Using observed tokens avoids compounding errors from a randomly initialized or still-improving model.
- Scheduled sampling, introduced as a training strategy around 2015, gradually mixes model predictions with observed tokens to reduce the gap between training contexts and inference contexts.
- For a token sequence, inputs are all tokens except the last and targets are the same sequence shifted one position forward.
- A start-of-sequence token gives the model a context from which to predict the first real token.
- An end-of-sequence token teaches the model to stop, while inference also needs a maximum generation length in case the model does not emit it.
- Batches require a common sequence length, so shorter examples are padded to the batch's chosen maximum length.
- The padding token has its own vocabulary ID and embedding, but padded targets should not affect the training objective.
- PyTorch negative log-likelihood loss supports an ignore index, commonly -100, so padding positions can be excluded from gradient calculations.
- Backpropagation through a long RNN sequence repeatedly applies the chain rule across recurrent steps.
- Repeatedly small derivatives can cause vanishing gradients, leaving early-token dependencies with little influence on weight updates.
- Large derivatives can instead produce exploding gradients, causing oversized updates and unstable movement during gradient descent.
- An RNN can be stacked by feeding one layer's hidden representation into another layer with its own weight matrices.
- The lecture uses a single-layer RNN for clarity, but the number of nonlinear layers can be increased.
- Decoder-based Transformers reuse the language-modeling setup, while replacing the RNN's hidden-state computation with a different architecture.
- Greedy decoding selects the highest-probability next token at every step, even when a less likely early choice could lead to a better full sequence.
- Beam search keeps multiple partial sequences, expands them with candidate next tokens, and ranks the resulting continuations.
- With beam size 2, the decoder retains the two best-scoring continuations after considering their next-token probabilities.
- Beam search can produce repetitive text, such as repeating the same word or phrase across a continuation.
- N-gram penalties discourage repeating unigrams or multi-token sequences, sometimes by setting the probability of a repeated continuation to zero.
- Hard repetition bans can damage valid output, such as a New York travel guide that naturally needs to mention “New York” more than once.
- Beam search tends to favor common, high-probability words, which can make generated text readable but less distinctive.
- Sampling draws tokens from the model's probability distribution, allowing lower-probability options to appear while favoring more likely ones.
- A temperature parameter divides logits before softmax: higher temperatures flatten the distribution, while lower temperatures make it sharper.
- Top-k sampling keeps a fixed number of the highest-probability tokens and renormalizes their probabilities before sampling.
- A fixed k can include too many weak candidates for a peaked distribution or too few useful candidates for a flatter one.
- Nucleus, or top-p, sampling dynamically keeps the smallest ranked set whose cumulative probability reaches a threshold, then renormalizes; the threshold can be adjusted for conservative or creative generation.
- Machine translation must handle changes in word order, context-dependent meanings, and cases where one word in one language corresponds to a phrase in another.
- Croatian examples illustrate that literal translations of greetings and word order can sound unnatural or convey the wrong meaning.
- Earlier systems used word alignment and expectation-maximization methods, then phrase-based approaches to represent mappings such as one word aligned with several words.
- Neural machine translation shifted the task toward sequence-to-sequence learning, also described as conditional text generation.
- An encoder reads a variable-length source sequence and compresses it into a hidden representation; a decoder conditions on that representation to generate the target sequence.
- The 2014 sequence-to-sequence work helped establish an architecture that later attention-based systems and foundation models built upon.
- Machine translation training uses parallel corpora containing source-language text paired with human translations.
- A human-written translation is called a reference, while text generated by a model is a hypothesis.
- In the RNN encoder-decoder illustration, the encoder's final hidden state represents the source sequence and initializes the decoder's language-modeling process.
Transcript excerpt
0:00 Okay, good afternoon everyone. How are you? How is the semester going? Going okay. It's kind of going fast for me. I feel like the fall break is just around the corner. Uh so hopefully hopefully I speed up too. Um I should turn on the project. Okay, this is always taking a few seconds. So uh let's recap what we talked about last time on uh Wednesday. We started finally to define what language modeling is and what the text generation uh is. And to kind of give you some background, I started with engram language models that are enabling generation of next token from just data
0:49 statistics alone. we were counting how many uh sequences of tokens which we termed engrams appear in a given corpus and then we use that to generate new sequences. Um and then the kind of main point I was trying to drive by the end of the lecture is that um uh engram language models are very dependent on approximation of history and approximations of history always come with some uh you know um negative impact on the performance of language modeling because language is simply has a lot of long-term dependencies right um so we started to talk about language modeling using an architecture called RNN as a as a you know um architecture of a choice. Uh although that's not
1:38 something we are going to use these days. We are going to use transformer which is going to be our next topic. But I did want to bring up a simpler architecture to kind of get going with what is neural language modeling. And basically the bits and pieces are going to stay the same except when we compute those hidden representations is going to get a little bit more complicated with
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.