Lecture 9e
Overview
Prof. Schniter traces language models from GPT-1's 117 million parameters and BERT's masked-word training through GPT-3's 175 billion parameters, multimodal systems, reasoning models, and AI agents. The central lesson is that progress has come from scaling data and compute as well as refining training: pretraining is followed by supervised fine-tuning and reinforcement learning, with verifiable rewards especially effective for math and code. The lecture also demonstrates zero-shot classification and text embeddings using scikit-llm and dimensionality reduction.
Key takeaways
- GPT-1 combined next-token pretraining with downstream fine-tuning at 117 million parameters, while GPT-3 reached 175 billion parameters and required 10,000 NVIDIA V100 GPUs.
- A pretrained language model is not automatically a useful assistant: supervised fine-tuning supplies conversation examples and role tokens to shape its behavior.
- For problems with checkable answers, reinforcement learning with verifiable rewards can scale better than subjective human ratings because candidate solutions can be scored as correct or incorrect.
- Reasoning models use additional generated tokens to distribute computation across intermediate steps, which can improve performance on complex math and coding tasks.
- Tools address limits that extra text training alone may not solve: calculators support precise arithmetic, and web search provides information beyond a model's training data.
- Text embeddings turn whole passages into vectors, and PCA can project those vectors into two dimensions to inspect semantic groupings such as gender, nobility, or food-country associations.
Chapters
- GPT-1 introduced next-token pretraining followed by task-specific fine-tuning, using 117 million parameters, 12 Transformer blocks, and a 512-token context.
- GPT-1 trained on the BookCorpus, about 7,000 unpublished fiction books and one billion words, at an estimated cost of $50,000.
- BERT instead trained to predict masked words and whether one sentence followed another, with 24 Transformer blocks and roughly twice GPT-1's parameters.
- BERT's training used books and English Wikipedia; its greater compute cost coincided with lower dollar cost as hardware improved.
- GPT-2 moved layer normalization before attention and feed-forward blocks, a pre-norm arrangement that became common in later Transformers.
- The largest GPT-2 version had 48 blocks, 25 attention heads, a 1,600-dimensional embedding, a 1,024-token context, and about 1.5 billion parameters.
- GPT-3 expanded to 96 blocks and 175 billion parameters, trained on web crawls, Wikipedia, and other text using 10,000 NVIDIA V100 GPUs.
- The lecture estimates GPT-3 training at $5 million to $15 million; its GPT-3.5 update used reinforcement learning from human feedback and became ChatGPT in November 2022.
- A GPT-3 parameter breakdown assigns substantial capacity to Transformer attention and especially feed-forward networks, which the lecture describes as storing many learned facts.
- The lecture characterizes GPT-4 as multimodal and using a mixture-of-experts design, with expert selection allowing inference to use only some components rather than the entire model.
- It gives an estimated GPT-4 total of about 1.8 trillion parameters and a training cost near $70 million, while noting that details were proprietary.
- The timeline proceeds through GPT-4o's longer context and multimodal inputs, GPT-4.5, and reasoning models such as o1 and o3, which spend more tokens working through complex tasks.
- The lecture contrasts early published model details with later proprietary systems whose training data and architecture are less transparent.
- Meta's Llama models popularized refinements including SwiGLU activations, rotary positional embeddings (RoPE), and RMSNorm.
- DeepSeek-R1 drew attention for open weights, reported performance comparable to leading proprietary models, and a much lower training cost.
- DeepSeek-R1 documentation showed chain-of-thought reasoning emerging through reinforcement learning, while open-weight models can be cheaper to run locally.
- Unlike a one-shot chatbot response, an agent cycles through actions that change an external environment, observations, and decisions about what to do next.
- A task such as booking Super Bowl flights requires multi-step planning, APIs, persistent memory, result verification, and error recovery.
- Examples discussed include OpenAI Operator for computer use, Codex for coding, and Anthropic's Claude computer-use and Claude Code products.
- Agent systems may manage subgoals, subagents, and memory outside the language model while operating with limited human oversight.
- Next-token pretraining can produce memorization, in-context learning, and hallucinations: a model may reproduce familiar text but invent details about material absent from training.
- Supervised fine-tuning adapts a pretrained model to tasks such as chat, translation, or coding using example conversations and system, user, and assistant role tokens.
- Fine-tuning can teach helpful, truthful, and harmless behavior, including how to answer questions about model identity.
- Hallucinations can be reduced by training on cases where the correct response is 'I don't know' or teaching the model to use tools such as calculators and web search.
- Reasoning models distribute hard problems across many generated tokens because each token has limited computation from the model's finite layers and attention heads.
- Breaking a task into gradual steps can improve performance on coding, mathematics, and science problems; asking for an immediate answer can discourage that process.
- The lecture calls uneven competence the Swiss-cheese phenomenon: a model can handle sophisticated tasks yet fail simple character or counting problems.
- Tokenization can explain some failures: a model may represent 'ubiquitous' as a token rather than reasoning directly over its letters, and may misread comparisons such as 9.9 versus 9.11.
- Reinforcement learning trains an agent to select actions in an environment, observe the resulting state, and use rewards to improve future decisions.
- Chess illustrates the loop: a move changes the board, and an apparently costly action such as sacrificing a pawn may improve the chance of winning later.
- Effective policies must consider long-term reward rather than only immediate gains.
- Exploration versus exploitation is a central tradeoff: an agent can choose a known short-term benefit or investigate a strategy that may pay off later.
- For math and coding, reinforcement learning with verifiable rewards generates candidate solutions, checks which are correct, and makes successful reasoning paths more likely.
- The training analogy is pretraining as reading textbooks, supervised fine-tuning as studying worked examples, and verifiable-reward reinforcement learning as solving practice problems with an answer key.
- For subjective goals such as writing funny jokes, human ratings train a reward model, which then supplies scalable feedback in reinforcement learning from human feedback (RLHF).
- The lecture notes that imperfect reward models can be gamed; it also cites AlphaGo, AlphaZero, MuZero, AlphaFold, and AlphaDev as examples of systems exceeding human performance in specialized domains.
- The scikit-llm library provides interfaces to language models for tasks including text classification, text-to-text modeling, tagging, and vectorization.
- Zero-shot classification supplies class names and a text sample without labeled training examples, relying on the model's broad pretraining.
- A movie review can be classified as positive, negative, or neutral by sending instructions and the review to a model such as GPT-5.1.
- Multi-label classification can assign several genres to one movie; the example uses a 0.5 probability threshold to report each qualifying class.
- Text vectorization pools representations from a sequence of tokens into one vector for an arbitrary-length input such as a sentence, paragraph, or book.
- Embedding models such as text-embedding-3-large produce high-dimensional vectors; the lecture's examples use 3,000 dimensions.
- Principal component analysis (PCA) compresses those vectors into two dimensions for visualization.
- The plotted examples organize phrases by gender and nobility, and foods by their associations with France, Germany, Italy, and Russia.
Transcript excerpt
0:02 All right, welcome back, everyone. Hope you had a good weekend. Um so what we're going to do today is we're going to move um chronologically through a number of the famous um large language models, um focusing on OpenAI because they were sort of there uh from the beginning. And um yeah, so we'll I guess we started with this Transformer paper last time. Um but let's focus on GPT-1. So [snorts] what's special about GPT-1 is that it this was the first model that proposed to pre-train on next token prediction, which is what we sort of focused on for most of this unit. [snorts] Um and then fine-tune on a desired target task. We haven't really talked much about that, um but maybe
0:50 we'll get into that later today. And um just wanted to pay attention to a few of the numbers here and see how they change over time. So uh this model used this byte pair encoding tokenization with 50,000 tokens. Um it had 10 or sorry, 12 Transformer blocks, 12 attention heads per block. The embedding dimension was 768. The context [snorts] length 512 and 117 million parameters. Uh pre-training was done on this book corpus dataset, 7,000 unpublished fiction books, about a billion words, taking one petaflops per second days and costing
1:38 about $50,000 at the time. Um just to show you that this is sort of not the only thing that works, pre-training on next token prediction, um Google uh very similar time, so this is June 2018, Google and October 2018 proposed this BERT model. Um so instead of pre-train next token prediction, [snorts] BERT pre-trained to number one, predict a random masked word in the sentence and predict whether one sentence follows another or not. [clears throat] And so once they
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof. Schniter.