Save this video free

1786 lecture8 Nov3

Jonathan Rose · 1:37:46 · Watch on YouTube

Part of University of Toronto ECE 1786. See all 10 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Jonathan Rose connects ECE 786’s embedding and classification assignments to language models, explaining next-token prediction, training data, cross-entropy loss, and perplexity before outlining the transformer architecture. He emphasizes that LLMs learn to predict tokens from large text collections, then describes transformer blocks, context size, positional embeddings, the final-token language head, layer normalization, and residual connections.

Key takeaways

Chapters

0:00 ECE 786 Project Deadlines and Assignment Four
3:00 Two Course Tracks and the Risks of Tool-Using Local Models
6:20 Course Recap: Embeddings, Word Prediction, and Classification
10:16 Interpreting Assignment Three’s Learned Vectors and Kernels
17:11 Language Models Predict the Next Word
22:41 What Makes a Next Word ‘Good’?
28:33 Formalizing Next-Token Probabilities and Vocabulary
32:21 Computing the Probability of a Whole Sequence
35:01 Perplexity Compares Models on the Same Text
50:11 From Classic Language Models to Neural Next-Token Prediction
51:13 Training Token Embeddings Alongside the Network
53:55 Training Data and the Legal Right to Use Text
57:58 Next-Token Labels Come Free from Existing Text
1:01:17 Why Next-Token Prediction Matters More Than Starting with Attention
1:09:40 Scaling Training and the Meaning of Model ‘Goodness’
1:14:03 Transformer Architecture: Embeddings Through Stacked Blocks
1:21:20 Context Size, Language Heads, and Shared Block Parameters
1:27:16 Positional Embeddings and the Order-Information Puzzle
1:31:23 The Final Token, Layer Normalization, and Residual Connections

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:03 All right. Good morning, everyone, and welcome to ECE 786 lecture number eight. The work that is in flight for you is principally the project. The progress report is due on November 17th. It feels like it was a pretty big hurly-burly, if you know that that word, uh, to do the proposals and get you your feedback. And some some people responding to that feedback and and and all that it is it is a major operation to to actually do all that stuff that you participated in at least two of the three events, but for us it's uh Yeah, a large operation. I don't know if you Yeah, I don't know if you care about our perspective. Uh Um

0:51 The other thing that you have in flight, but I was going to release today that I changed my mind on is assignment four. I'm rethinking assignment four to I want to try to modernize it. Uh to to uh something that's better than what I have had I've done before. So, I'm going to try to rethink it this week, but I plan to have it due on November 23rd. Which is, you know, a lot of work in a small amount of time. Um All of that makes me kind of wake up and recognize This is a strange course. And I have reorganized it this year. And it makes it feel even a little stranger. And it's The course is strange because there's two tracks in it. Track one is how does a large language model work? What is What What is the nitty-gritty of of the design of of it and the theory

1:40 around it? And track two is how do we use these things? And that part is just kind of exploding in the outside world about how you use

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Jonathan Rose.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.