Save this video free

UUtah CS 6340 NLP | Fall 2026 | Language modeling

UofU Data Science · 1:22:35 · Watch on YouTube

Part of University of Utah CS 6340. See all 19 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

The lecture introduces language modeling as next-token prediction, showing how the chain rule turns token probabilities into sequence probabilities and how n-gram models estimate them from corpus counts. It explains why sparse counts and long-range dependencies limit n-grams, introduces smoothing and perplexity, then begins deriving a recurrent neural network language model, including embeddings, hidden states, softmax outputs, and the distinction between training inputs and generated tokens.

Key takeaways

Chapters

0:00 Course Updates: Quiz Regrading, Bonuses, and Practicum Scores
6:57 Language Modeling as a Generative Machine Learning Task
10:19 The Chain Rule Connects Next-Token and Sequence Probabilities
15:53 N-Grams Estimate Continuations from Corpus Counts
19:47 Markov Assumptions Limit the Context of N-Gram Models
22:20 Bigram Counts and Sequential Text Generation
28:50 From Empty-Context Generation to Prompted Continuation
32:22 N-Gram Samples Reveal Data and Context Limitations
37:19 Smoothing Addresses Zero-Probability N-Grams
38:05 Long-Range Dependencies Expose N-Gram Weaknesses
45:08 Perplexity Evaluates Held-Out Language Modeling
51:54 Tokenizer Differences Make Perplexity Comparisons Invalid
58:32 Break Discussion: Estimating First-Token Probabilities
1:04:50 RNNs Preserve Context for Next-Token Prediction
1:10:17 RNN Inputs Combine Token Embeddings with Hidden History
1:12:27 RNN Parameter Shapes, Hidden States, and Softmax Outputs
1:20:10 Teacher Forcing Separates RNN Training from Generation

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 Let's get started. Good afternoon uh everyone. Um I'm excited we are jumping into language modeling which is kind of what this whole course is going to eventually be about. Uh before we do start with this new topic. Um I want to acknowledge that someone did reach out about the ambiguity in the first question in the quiz and I'll look into it as soon as I have more time on my plate. Uh definitely sometime this this week. I agree that something might have been lost uh in my wording uh of the of the question. So I will look into that and then uh yeah I have a huge limitation of not knowing how to deal with canvas. So I personally put all these course in the external sheet and uh I will give you a

0:49 little way of kind of uh thinking about what your grade in the course is. do not rely on canvas uh grades because I do not know how to set up the bonuses there. Uh I mean I didn't even yet decided how I will uh consider them. As I said um from my experience I kind of tend to overinflate the grades with how I I gave bonuses before. So I want to be mindful and cautious of that and likely once uh around the fall break I will apply them and then again at the end kind of looking holistically at uh all of this course which basically for you means um you know study practice and don't overfocus on these bonuses. Uh they might be like a nice pleasant

1:37 surprise but don't over over rely on uh on them. Any questions before we move forward? Yes. Um, one was what was the bonus question and then secondly speaking of ambiguity I felt like for

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.