Save this video free

Lecture 9d

Prof. Schniter · 56:41 · Watch on YouTube

Part of The Ohio State University ECE 5307. See all 30 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

The lecture connects transformer mechanics to the ways language models represent meaning: residual connections preserve information, attention contextualizes token embeddings, and MLPs may store factual associations. It also explains image tokenization through Vision Transformers and vector-quantized autoencoders, then traces the 2017 Transformer to 2018 GPT, whose 117 million parameters were trained on BookCorpus using next-token prediction before task-specific fine-tuning.

Key takeaways

Chapters

0:00 Causal Transformers, Embeddings, and Classification
7:54 Why Transformer Blocks Use Residual Connections and Layer Norm
14:36 GloVe Embeddings Encode Similarity and Word Relationships
19:51 Self-Attention Turns Static Token Embeddings into Contextual Meaning
25:09 How Transformer MLPs May Store Facts
32:39 Johnson-Lindenstrauss Geometry and MLP Fact Capacity
37:50 Vision Transformer Patch Embeddings
43:32 Vector-Quantized Autoencoders Convert Images into Tokens
47:01 The 2017 Transformer: Encoder-Decoder Translation
52:24 2018 GPT: Unlabeled Pretraining Followed by Fine-Tuning
56:19 From GPT's Baseline to Rapid Model Scaling

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:01 Okay, hope you guys had a good weekend. Uh, so we're going to get back into our discussion about transformers and large language models and all that. So, uh, to sort of summarize some of what we've seen before, we have this picture here which shows how we build this text generating transformer. [snorts] Um, in this case, we're just predicting forward one character at a time, but um, as we'll talk about more today, it's it's typically like one word at a time or, you know, perhaps even something different when you have models that use words and images and language together. But anyway, um, [clears throat] these um, these inputs, characters in this case, are converted to um, integers. Um, and each of those integers is then

0:49 used to pick out the an element within a code book of what are called embeddings. Embeddings are these vectors. So these are the vectors that correspond to those integers there. And [clears throat] um those go into the uh serial concatenation of these transformer blocks. Um, we said that in addition to those embeddings, we also would like the transformer to know which is the first and the second and the third. And so with to do that, we add these additional embeddings called um positional embeddings. And they're literally vectors of the same size as our regular embeddings that are just added element-wise like that. [clears throat]

1:36 So these go through the sets of transformers to eventually learn an output sequence that is also um just in this embedding space. And then if we want to come back to the um [clears throat] the world of characters, we first have to convert those embeddings to uh probabilities where the pro this vector this PMF vector of probabilities is telling us how probable is like the

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof. Schniter.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.