Save this video free

UUtah CS 6340 NLP | Fall 2026 | RNN LMs

UofU Data Science · 1:19:16 · Watch on YouTube

Part of University of Utah CS 6340. See all 19 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

The lecture develops recurrent neural network language models from next-token prediction and negative log-likelihood through teacher forcing, sequence batching, and decoding methods such as beam search, temperature, top-k, and nucleus sampling. It then connects language modeling to machine translation and encoder-decoder sequence-to-sequence systems, while clarifying that teacher forcing does not let an RNN compute across time in parallel because each hidden state depends on the previous one.

Key takeaways

Chapters

0:00 RNN Language Modeling Recap: Hidden States Summarize Token History
7:30 Negative Log-Likelihood Selects the Observed Next Token
19:00 Vocabulary Indices, Token-Level Losses, and Batch Loss
25:00 Teacher Forcing Supplies Gold Tokens During RNN Training
35:00 Shifted Inputs and Targets, Plus Start and End Tokens
40:00 Padding Variable-Length Sequences for PyTorch Batches
43:00 Why RNNs Struggle: Vanishing and Exploding Gradients
50:10 Stacked RNN Layers and the Reusable Language-Modeling Pipeline
52:00 Greedy Decoding Can Miss Better Token Sequences
55:00 Beam Search Repetition and the Limits of N-Gram Penalties
59:00 Sampling for More Varied, Human-Sounding Text
1:02:00 Top-k and Nucleus Sampling Control Candidate Tokens
1:06:00 Machine Translation Shows Why Sequence-to-Sequence Models Matter
1:11:45 Neural Machine Translation Introduces the Encoder-Decoder
1:14:30 Parallel Corpora, Translation References, and Decoder Conditioning

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 Okay, good afternoon everyone. How are you? How is the semester going? Going okay. It's kind of going fast for me. I feel like the fall break is just around the corner. Uh so hopefully hopefully I speed up too. Um I should turn on the project. Okay, this is always taking a few seconds. So uh let's recap what we talked about last time on uh Wednesday. We started finally to define what language modeling is and what the text generation uh is. And to kind of give you some background, I started with engram language models that are enabling generation of next token from just data

0:49 statistics alone. we were counting how many uh sequences of tokens which we termed engrams appear in a given corpus and then we use that to generate new sequences. Um and then the kind of main point I was trying to drive by the end of the lecture is that um uh engram language models are very dependent on approximation of history and approximations of history always come with some uh you know um negative impact on the performance of language modeling because language is simply has a lot of long-term dependencies right um so we started to talk about language modeling using an architecture called RNN as a as a you know um architecture of a choice. Uh although that's not

1:38 something we are going to use these days. We are going to use transformer which is going to be our next topic. But I did want to bring up a simpler architecture to kind of get going with what is neural language modeling. And basically the bits and pieces are going to stay the same except when we compute those hidden representations is going to get a little bit more complicated with

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.