Save this video free

1786 lecture10 Nov17

Jonathan Rose · 1:26:48 · Watch on YouTube

Part of University of Toronto ECE 1786. See all 10 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Jonathan Rose explains how a transformer language model turns next-token probabilities into a complete response, comparing greedy decoding, beam search, and the sampling methods used most often in practice. He connects decoding choices to sequence quality, randomness, temperature, and hallucinations, then distinguishes pre-training on next-token prediction from post-training for instruction following, safety, and reasoning.

Key takeaways

Chapters

0:00 The Autoregressive Loop Turns Predictions into Text
6:35 GPT-2 Completions Show Early Language Generation
8:30 The Language Head Produces a Probability for Every Token
11:00 A Good Completion Is More Than a Likely Next Token
13:20 Greedy Decoding Can Produce Boring or Repetitive Text
18:30 Sequence Probability Requires Looking Beyond the Current Token
23:10 Vocabulary Size Makes Exhaustive Completion Search Impossible
26:30 Token Choices Form a Probability Tree
30:00 Tree Paths Can Outrank the Greedy First Choice
39:10 Beam Search Keeps Multiple Candidate Paths Alive
46:30 Sampling Selects Tokens According to Their Probabilities
51:40 Random Sampling Explains Variation and Can Contribute to Hallucinations
59:55 Top-K and Top-P Sampling Limit the Candidate Set
1:04:30 Temperature Controls Token Diversity
1:10:00 Softmax Logits Explain Temperature and Repetition Penalties
1:14:30 LM Studio and OpenAI Tools Expose Decoding Controls
1:21:35 Pre-Training and Post-Training Serve Different Purposes

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:03 All right, welcome to ECE 1786 lecture number 10. We are a long way into this course. Two big things in flight right now, the progress report for the projects due tonight at 9 I hope. I hope you've done lots of work on that. The real purpose of that progress report is to make sure that you don't leave it all all the work to the end. But also I gave you targets to shoot at and and get you going. Some some people were very late in the week asking for access to their repository. So I hope you've been actively committing your work to that repository so I can see that you're working, both of you. And I look forward to reading reading those reports. Um assignment four is due next Monday at in the evening. And so

0:52 that assignment covers material from the last two lectures and and this lecture. And it is about the core operation of a large language model, which is one of the key things I wanted you to learn in this course. Uh last day I covered the core mechanisms of the transformer block. You'll have to remember that the large language models we've all been using have many transformer blocks all cascading together. And the last lecture was well what's happening in each one of those blocks uh as best explained as best as I could cook it up. Um can't remember if I said this but I I'd be willing to bet that within a decade that won't be the main mechanism anymore. I think they'll have come up

1:39 with much better ways to predict the next token. But that's the way it's being done now. Today I want to go and hit the kind of the final key piece of

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Jonathan Rose.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.