CMU Introduction To Deep Learning 11-785, Fall 2026: Lecture 20
Overview
Large language models gain broad capabilities through web-scale next-token pretraining, then become useful assistants through carefully designed supervised fine-tuning and reinforcement learning. The account covers scaling laws and Chinchilla-style compute allocation, the limits of perplexity and benchmark evaluation, and the shift toward verifiable rewards and reasoning models that spend additional test-time computation on problems such as math and coding.
Key takeaways
- Next-token pretraining can develop facts, syntax, sentiment, code, and problem-solving patterns because predicting text well requires modeling the processes that produced it.
- Data curation is part of model design: web crawls require spam filtering and markup cleanup, and training mixtures can shift toward higher-quality code and math as training progresses.
- Perplexity tracks broad predictive quality but cannot establish performance on long-horizon reasoning, tool use, or instruction following; benchmarks add specificity while creating contamination and optimization risks.
- Scaling laws offer empirical guidance for allocating compute across model parameters and data, but power-law returns diminish and increasing parameter count alone does not guarantee new capabilities.
- Supervised fine-tuning examples can generalize beyond their literal content, so a narrow annotation mistake or reward-hacking demonstration may teach a broad and unwanted behavior.
- Verifiable rewards and reasoning models combine precise checks, such as math-answer validation or code tests, with extra test-time computation, enabling progress that preference-based reward models may not reliably measure.
Chapters
- Traditional task-specific deep learning starts with labeled data or simulations, an architecture, and training and validation tailored to task T.
- Foundation models let users specify tasks in natural language, as with ChatGPT or Claude, instead of building a separate model for each task.
- Desired capabilities include language understanding, world knowledge, instruction following, safety, tool use, self-correction, and extended reasoning.
- Transformers provide a flexible architecture based on token representations and self-attention, with relatively few hand-coded language rules.
- Large datasets can teach useful inductive biases that engineers cannot reliably specify in advance.
- Web data choices change with desired capabilities: code was sometimes filtered out in 2019 and 2020, while coding-agent training now prioritizes it.
- Self-supervised pretraining usually trains a language model to predict the next token from large text collections.
- Examples such as the Eiffel Tower being in France illustrate how prediction can encode factual associations and uncertainty.
- Predicting pronouns, sarcasm, math, prompts, and code requires learning patterns beyond surface grammar, including semantics and task structure.
- A web crawl contains spam, gibberish, and page markup, so training data requires selection, cleaning, and filtering rather than a simple download.
- PageRank can help identify valuable pages through their links to other trusted pages.
- Language models can also help classify data quality, while HTML, CSS, JavaScript, and boilerplate need processing before text is used for training.
- An open Hugging Face training recipe illustrates staged data mixtures, beginning with broad data and later emphasizing higher-quality code, math, and reasoning examples.
- Architecture matters, including mixture-of-experts and hybrid Transformer designs, but data composition is a major determinant of model behavior.
- Training runs require active monitoring of loss, gradient norms, stability, and evaluation metrics, especially with low-precision training and distributed systems.
- Validation cross-entropy measures average next-token prediction loss; exponentiating it gives perplexity, an intuitive measure of predictive uncertainty.
- Perplexity is useful because it is smooth and broadly applicable, without requiring a hand-picked list of capabilities.
- A low perplexity does not directly establish competence at long-horizon reasoning, instruction following, coding, or tool use.
- Capability evaluations range from language-understanding questions and world knowledge to coding, tool use, computer use, and live user preferences.
- Human preference ratings can reward longer or more flattering answers and may fail to catch factual errors or sycophancy.
- Web-scale training makes benchmark contamination difficult to rule out, while repeatedly optimizing against evaluation sets can invalidate them.
- An OpenAI math-data example added 4,500 of a dataset's 5,000 test examples to training, leaving only 500 for evaluation.
- With a large compute budget, teams must choose model size, token count, data mixture, architecture, optimizer, depth, width, and learning rate.
- Training many equal-budget models wastes most of the budget on experiments, while tuning small models and launching one large run risks hyperparameters not transferring.
- Empirical scaling laws relate compute and validation loss, letting smaller runs inform estimates for larger training runs.
- Early scaling-law experiments varied parameters and data across roughly thousandfold ranges and found power-law relationships with validation loss.
- Chinchilla-style fits model loss using parameter count, data quantity, and an irreducible-loss term, with exponents determined by the training recipe.
- The reported parameter and data exponents were roughly equal, suggesting that doubling model size should be accompanied by roughly doubling training data for the fitted optimal allocation.
- Kaplan and colleagues initially found loss more sensitive to parameter count than to data, encouraging a period of model-heavy scaling.
- Later practice emphasized scaling data more rapidly, since parameter growth alone does not deliver every desired capability.
- Power-law loss improvements diminish as compute rises, making trillion-fold budgets hard to justify from loss alone.
- A GSM8K extrapolation once suggested that 10^16 parameters might be needed for an 80% solve rate, far beyond what later training approaches required.
- Broad pretraining supplies language ability, world knowledge, and useful priors that make later learning and exploration more effective.
- Training recipes can move from large, imperfect corpora toward smaller, curated datasets as capabilities are refined.
- Conversation templates distinguish system instructions, user messages, and assistant responses, preventing a model from simply continuing or completing the user's text.
- Supervised fine-tuning uses a pretrained model and human-written examples of useful answers, training next-token prediction primarily on assistant output tokens.
- Math questions can include either a final numerical answer or solution steps, while other tasks may draw on deployed interactions, data vendors, or synthetic examples.
- InstructGPT, published in early 2022, helped establish the modern approach to instruction-following fine-tuning.
- Because SFT datasets are much smaller than pretraining corpora, each example can exert substantial influence on model behavior.
- A correct population answer for one town may teach a model to produce precise-looking estimates for places it does not know, rather than grounded facts.
- Annotators can also penalize valid answers by imposing an unnecessarily specific response format.
- Models generalize compositionally from fine-tuning examples, so apparently narrow data can create broader behaviors.
- The School of Reward Hacks study found that training on low-stakes reward-hacking demonstrations could generalize to higher-stakes settings, making careful data selection essential.
- Reinforcement learning replaces a single prescribed answer with a reward function that scores a model response to a prompt.
- In RLHF, annotators rank sampled model responses, and those comparisons train a reward model that scores prompt-response pairs.
- A policy such as an SFT model can then generate responses, receive reward-model scores, and update its token probabilities using methods such as PPO.
- Preference comparisons are imperfect: annotators may not know the correct facts and may favor superficial features such as shorter answers or tables.
- Reinforcement learning from verifiable rewards uses high-precision checks, such as comparing a math answer or running code against tests.
- Coding rewards can rely on execution, runtime limits, and autograder test results rather than exact string matching.
- Formal proof systems such as Lean can verify proofs even when no human has supplied the solution in advance.
- OpenAI o1 and DeepSeek-R1 helped popularize reasoning models that can generate additional tokens before returning an answer to the user or verifier.
- Rejection fine-tuning samples K candidate responses per prompt, keeps the highest-reward response, and trains the model to imitate it.
- Online sampling repeats the process after each update, but retaining only the best response discards information from failures and can reinforce mediocre answers.
- Policy-gradient methods optimize expected reward using sampled rollouts and the gradient of response log-probability weighted by reward.
- GRPO groups rollouts and normalizes their rewards, adding a relative comparison to the basic policy-gradient loop.
- Long-horizon tool-using agents make language-model reinforcement learning more like conventional RL than math tasks with a single final reward.
- RL requires substantial sampling and infrastructure, and asynchronous updates can make the sampled policy stale relative to the policy being optimized.
- Narrow rewards can cause mode collapse, repetitive jokes, and less creative writing even as measured task scores improve.
- General-purpose pretraining remains valuable because optimizing specific rewards can sacrifice broader capabilities.
- The closing argument is that simple ideas such as Transformer pretraining and SFT depend on extensive data, systems engineering, evaluation, and empirical iteration.
- The Q&A describes a spectrum between supervised imitation and reinforcement learning, rather than treating them as the only two ways to teach an answer.
- Verification can be easier than generation when a checker has privileged information, such as a known math answer or executable test cases.
- For formal proofs or citations, models may need to produce computer-checkable proofs or source-backed claims, but weak proxies can still invite reward hacking.
Transcript excerpt
0:00 You folks see most >> uh can you folks see the slide? >> Yeah, we can. >> We can. Yeah. >> Okay. Is the quality good? Sometimes there's issues with that. It's normal. Okay, cool. Um, so yeah. Uh, thank you, Professor Ba. Thank you for uh for having me here today. Um it is always great uh to to meet the to meet the class. Like Professor Ba said, I I was a student in this this course u almost a decade ago and I was a TA after that. Um and you know this course is a big part of
0:48 why I work on what I work on today. So um you know it's always an honor to be back. So uh this is going to be a lecture on large language models. Um please uh you know interrupt me at any time. Uh I'll try to monitor the chat but also feel free to you know interrupt and uh and ask questions. Um so we'll try to cover at a high level the stages of building these models and some you know useful ideas along the way. uh we'll talk about pre-training and this will build on what you guys um have been learning in terms of the architecture for a transformer and in terms of language modeling as an objective. Um we will then talk about evaluation and scaling laws and we'll discuss you
1:38 know how we evaluate these models and um how and why um scaling them is is is its own art and science. Um, and then we'll talk about post- training and basically how we adapt them to downstream use and the behaviors that you expect to see in a chatbot or in a coding agent and in these types of uh, you know, more polished uh, form.
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Carnegie Mellon University Deep Learning.