Save this video free

1786 lecture9 Nov10

Jonathan Rose · 1:19:09 · Watch on YouTube

Part of University of Toronto ECE 1786. See all 10 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Jonathan Rose explains how transformer language models learn to predict the next token from large text corpora, then develops the mechanics of self-attention: contextual embeddings, dot-product scores, softmax weights, and learned query, key, and value projections. He connects multi-head attention to residual connections, feed-forward networks, parallel computation, and KV caching, then previews Assignment 4, which uses Andrej Karpathy’s minGPT and GPT-2.

Key takeaways

Chapters

0:00 ECE 1786 Project Milestones and Assignment Deadlines
2:11 Next-Token Prediction Turns Text into Training Data
6:42 Transformer Architecture: Tokens, Embeddings, and Repeated Blocks
9:06 Inside a Transformer Block: Attention, MLP, and Skip Paths
14:25 Context Resolves Ambiguous Words Like “Bank”
19:09 Self-Attention Compares Each Token with Its Prefix
24:28 Dot Products Produce Attention Similarity Scores
29:48 Softmax Converts Scores into Attention Weights
34:33 Weighted Context Mixtures Create Updated Embeddings
36:36 Learnable Matrices Make Attention Trainable
42:04 Query, Key, and Value Projections Define Attention
53:03 Scaled Dot-Product Attention and Multiple Heads
1:02:47 Residual Connections Preserve Information and Gradients
1:07:16 Feed-Forward Layers, GPU Parallelism, and KV Caches
1:12:00 Assignment 4, GPT-2 Fine-Tuning, and Positional Information

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:03 Okay, welcome to ECE 1786 lecture number nine. I have written a lot on the board because I'm reprising the two things I the two big pictures I had last day and I needed it needed it there. They're in the notes so I don't know if you have to write them all down but let me tell you the work that I'm I've put you put you towards in this course. two big things. Uh the project which are largely underway and people are thinking carefully about you need to submit a a progress report. In my experience as a teacher, you need to have these kind of milestones where people you have some impetus towards doing the work and not leaving it all towards the end. So there's a progress report that has very specific specifications including a hard word limit like usual. Uh in the circus assignment, which I just released, I

0:50 realized I didn't I should have put it out a little earlier. tells you what you need to deliver next week. Uh, and I also noticed that I would have normally made the due on the Monday at 9:00 p.m., but I've been saying it's November 17th, so it's actually due on Tuesday, November 17th at 9:00 p.m. Uh, um, and the other key point is I've made repositories for everyone, and I expect to see code in those repositories submitted by each member of the team. You're not allowed to let your teammate do the commits for you. you must commit your own work so that I can see that you did some work. Um, I did release belatedly assignment 4. I was hoping to change it but uh didn't. It's still the same as previously. Uh, which is about

1:39 the subject of the previous lecture in

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Jonathan Rose.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.