1786 lecture10 Nov17
Overview
Jonathan Rose explains how a transformer language model turns next-token probabilities into a complete response, comparing greedy decoding, beam search, and the sampling methods used most often in practice. He connects decoding choices to sequence quality, randomness, temperature, and hallucinations, then distinguishes pre-training on next-token prediction from post-training for instruction following, safety, and reasoning.
Key takeaways
- Autoregressive generation repeatedly appends one selected token to the input and runs the transformer again, so producing a long answer requires many expensive inference steps.
- Greedy decoding optimizes the next-token choice locally, not the full completion; a lower-probability first token can lead to a higher-probability sequence.
- A vocabulary of 50,000 tokens creates 50,000^G possible sequences for a G-token completion, making exhaustive search impractical.
- Sampling uses a random draw weighted by token probabilities, which creates response diversity but can also send a completion down a low-probability, unreliable path.
- Top-K limits sampling to a fixed number of likely tokens, while top-P retains tokens up to a cumulative probability threshold; temperature separately adjusts how concentrated the distribution is.
- Pre-training teaches next-token prediction, whereas post-training adds behaviors such as instruction following, safety constraints, and reasoning.
Chapters
- Assignment 4 covers the core operation of a large language model, building on the previous lecture's transformer-block material.
- An LLM predicts a distribution over next tokens, then appends the selected token to its input and runs another inference.
- This autoregressive loop repeats token by token through many transformer blocks, making generation computationally expensive.
- The early GPT terminology called generated text a completion because it continued an input such as “The sky is blue.”
- A GPT-2 completion of “The clean river flowed” produced fluent prose about currents, fish, and people along the river.
- GPT-2's output demonstrated coherent wording, but fluency did not guarantee factual grounding.
- After token embeddings pass through transformer blocks, the language head produces probabilities for the model's vocabulary.
- The vocabulary is represented as M possible tokens, with probabilities indicating which token could come next.
- Choosing a token from that probability distribution is called decoding, distinct from a transformer decoder architecture.
- The goal is the best completion, a sequence of tokens, rather than simply the highest-probability next token.
- A useful answer should be grammatical, coherent, and responsive, while also meeting expectations for knowledge and avoiding harmful content.
- Rose frames the challenge as producing capable, helpful responses without enabling dangerous requests, a topic for the next lecture.
- Greedy decoding selects the single highest-probability token at every generation step.
- This local choice can lead to obvious, uninteresting completions or repetitive output, especially in earlier language models.
- The most likely token at each step does not necessarily form the most likely or best overall sequence.
- A completion can be evaluated by combining the probabilities of its successive tokens, conditioned on the preceding context.
- When selecting token Yᵢ, an autoregressive model cannot know the probabilities of future tokens in the sequence.
- Comparing completions of different lengths also complicates the simple product-of-probabilities objective.
- With M possible tokens at each step, a completion of G tokens has M^G possible sequences.
- Even a vocabulary of 50,000 tokens and a 20-token completion creates an enormous search space.
- Evaluating every candidate would require an impractical number of costly model inferences and substantial energy.
- A probability tree represents each possible next token as a branch, with another model inference at every subsequent node.
- For “The clean river flowed,” example first-token choices include “in,” “through,” “to,” and “while,” each with an assigned probability.
- Assignment 4 asks students to generate such trees, making the branching structure and search difficulty visible.
- The classroom example expands several branches for three generated tokens and multiplies their token probabilities for comparison.
- A branch beginning with a less-probable first token can yield a higher-probability full sequence than the greedy branch.
- This illustrates why local optimization can get stuck in a local optimum instead of finding a stronger completion.
- Beam search traverses part of the probability tree while maintaining K candidate paths at each layer.
- For each path, it considers V downstream tokens, then prunes the combined K × V candidates back to the top K paths.
- Beam search requires multiple model inferences per layer and is often too costly for large modern models, despite its use as a standard search concept.
- Sampling draws a token randomly so that each token's chance of selection matches its probability from the model.
- For probabilities 0.5, 0.3, and 0.2, a uniform random number from 0 to 1 can be divided into intervals for three choices.
- Unlike greedy decoding, sampling can select a lower-probability token and send the subsequent completion in an unexpected direction.
- Repeated runs can produce different answers because the random draw may select different tokens, even for the same prompt.
- Many questions have multiple valid answers, so variation is not inherently a flaw; a low-probability choice can nevertheless start an unreliable path.
- If a response is unsatisfactory, rerunning the prompt may yield another answer because the sampling outcome changes.
- Top-K sampling restricts random selection to a fixed number of high-probability tokens, such as 20 to 50 candidates.
- Top-P sampling instead keeps the highest-probability tokens whose cumulative probability reaches a threshold, often around 0.8.
- Both methods reduce the chance of selecting an extremely unlikely token while preserving some randomness.
- Temperature T adjusts the token probabilities before sampling: T = 1 leaves them unchanged, while T > 1 makes less likely tokens relatively more likely.
- A lower temperature concentrates probability on likely tokens, and the limit as T approaches zero behaves like greedy decoding.
- Higher temperature can increase output diversity and creativity, while Rose notes that later training may make modern models' behavior less visibly variable.
- The language head produces logits, and softmax converts them into probabilities using exp(Lᵢ) divided by the sum of exp(Lⱼ).
- Temperature divides each logit by T before softmax, reducing the dominance of large logits when T is greater than one.
- A repetition penalty can reduce the likelihood of tokens already generated; values above one make repeated tokens less likely.
- LM Studio exposes settings including temperature and top-P sampling, though Rose found modern model outputs harder to use for a simple high-temperature demonstration.
- The OpenAI Playground's older GPT-4o interface showed temperature, top P, and maximum-token controls.
- OpenAI's tokenizer view colors token boundaries, while Assignment 4 uses API token probabilities to build a real probability tree.
- Pre-training teaches a model to predict the next token, while post-training adapts the base model for instruction following and other desired behaviors.
- Post-training can include reinforcement learning from human feedback, direct preference optimization, safety training, and reasoning-focused training.
- Instruction following and safety can conflict when a user requests harmful assistance, so post-training must balance competing objectives.
Transcript excerpt
0:03 All right, welcome to ECE 1786 lecture number 10. We are a long way into this course. Two big things in flight right now, the progress report for the projects due tonight at 9 I hope. I hope you've done lots of work on that. The real purpose of that progress report is to make sure that you don't leave it all all the work to the end. But also I gave you targets to shoot at and and get you going. Some some people were very late in the week asking for access to their repository. So I hope you've been actively committing your work to that repository so I can see that you're working, both of you. And I look forward to reading reading those reports. Um assignment four is due next Monday at in the evening. And so
0:52 that assignment covers material from the last two lectures and and this lecture. And it is about the core operation of a large language model, which is one of the key things I wanted you to learn in this course. Uh last day I covered the core mechanisms of the transformer block. You'll have to remember that the large language models we've all been using have many transformer blocks all cascading together. And the last lecture was well what's happening in each one of those blocks uh as best explained as best as I could cook it up. Um can't remember if I said this but I I'd be willing to bet that within a decade that won't be the main mechanism anymore. I think they'll have come up
1:39 with much better ways to predict the next token. But that's the way it's being done now. Today I want to go and hit the kind of the final key piece of
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Jonathan Rose.