CMU Introduction To Deep Learning 11-785, Fall 2026: Lecture 17
Watch on YouTube →
Overview
The lecture builds from language modeling to sequence-to-sequence generation: next-token probabilities make it practical to model complete sequences, while learned embeddings replace sparse one-hot inputs. It then develops an encoder-decoder RNN for conditional generation, explains autoregressive decoding, beam search, and teacher forcing, and shows applications in translation, dialogue, speech recognition, and image captioning before identifying the encoder's final-state bottleneck.
Key takeaways
- A language model represents probabilities of complete sequences by applying the chain rule and predicting each next token conditioned on the preceding context.
- One-hot vectors avoid imposing arbitrary word similarities, but a learned projection maps them into compact embeddings whose distances can reflect semantic relationships.
- An autoregressive encoder-decoder conditions each output token on both the encoded input and the previously generated output prefix.
- Greedy decoding can make a locally likely choice that damages later predictions; beam search preserves several high-probability paths but remains approximate and can favor short outputs.
- Teacher forcing trains a decoder by supplying the true target prefix at each step, allowing it to learn next-token probabilities before it can reliably generate its own prefixes.
- A single final encoder state must summarize the whole input in a basic seq2seq model, creating an information bottleneck for long or complex sequences.
Chapters
- The recording opens with room noise and informal conversation before the class begins at about 3:23.
- At 5:21, students are asked to close their laptops before the lecture starts.
- The prior lecture covered sequence-to-sequence tasks with order correspondence between input and output elements.
- This lecture turns to tasks such as translation and dialogue, where input and output words need not align directly.
- A small, three-layer LSTM trained on Linux source code generated code-like text with consistent braces and variables.
- A language model represents the probability distribution over token sequences, enabling both sequence scoring and text generation.
- The chain rule factors a sequence probability into conditional probabilities for each token given the preceding context.
- A model can therefore predict one token from a finite vocabulary instead of assigning probabilities directly to an effectively unbounded set of sentences.
- The example “four score and seven years” favors “ago” because the preceding context changes the next-token probabilities.
- A vocabulary with a fixed ordering maps each word to a one-hot vector, with one active position per token.
- For a five-word context, six-token windows provide training examples: the first five tokens are inputs and the sixth is the target.
- Overlapping windows from ordinary text create many examples that can be trained with backpropagation.
- A 400,000-word vocabulary requires 400,000-dimensional one-hot vectors, which are sparse and wasteful.
- One-hot vectors have equal pairwise distance, so they impose no prior claim that words such as “computer” and “phone” are similar.
- A learned projection matrix maps one-hot vectors into a smaller continuous space whose geometry can encode useful word relationships.
- A sliding-context predictor is a time-delay neural network, or TDNN, that reuses its parameters across sequence positions.
- Skip-gram and related bag-of-words objectives learn the projection matrix rather than treating next-word prediction as the sole goal.
- Word2Vec examples from Thomas Mikolov's 2013 work show similar capital-minus-country vectors and the relation “king minus man plus woman” approximating “queen.”
- An RNN language model consumes embedded tokens and produces a probability distribution over the vocabulary at each step.
- Sampling a token and feeding it back lets the model generate text incrementally, conditioning each prediction on the generated prefix.
- Start-of-sentence (SOS) and end-of-sentence (EOS) markers identify sentence boundaries and tell generation when to stop.
- Translation can reorder or split meaning across words, as illustrated by the English input “I ate an apple” and a Japanese rendering.
- The encoder reads the complete input sequence and produces a hidden representation; a decoder uses that representation to generate output.
- The initial decoder sketch lacks feedback from each sampled output token, so later output choices do not depend on earlier ones.
- Without feeding sampled tokens back, the decoder can choose an implausible continuation without conditioning on its previous choice.
- The decoder should generate one token at a time, passing each sampled token into the next recurrent step.
- Token-level generation keeps each prediction over a finite vocabulary, unlike direct prediction over the effectively unbounded set of full sentences.
- The encoder processes input through its EOS marker, and its final hidden state initializes the decoder.
- The decoder begins with SOS, predicts a token distribution, feeds the selected token back, and continues until it emits EOS.
- The encoder and decoder can each contain more complex components, including multilayer LSTMs.
- The decoder models output-token probabilities conditioned on both the encoded input and all previously generated output tokens.
- Multiplying the stepwise conditional probabilities yields the probability of the complete output sequence given the input.
- For translation, the model should assign high probability to target sequences that preserve the input meaning and low probability to unrelated sentences.
- Greedy decoding selects the highest-probability token at each step, but that local choice may lead to a poor continuation.
- The speech-recognition example contrasts “he knows” with a competing sound-alike path that can produce a more plausible full sentence.
- Multiplying token probabilities can also favor short outputs, so sequence scoring may need length-related heuristics.
- Random sampling chooses each next token according to the model's probability distribution and can avoid some greedy failures.
- Sampling is not guaranteed to find the most probable complete sequence, because each choice changes later conditional distributions.
- Keeping every possible continuation would grow exponentially with vocabulary size and sequence length.
- Beam search retains a fixed number of candidate paths, such as the best 10, instead of expanding every vocabulary option.
- Candidates are ranked by cumulative sequence probability, the product of conditional probabilities along each path.
- Beam search is approximate: a pruned path could later have become the best, and short-sequence bias may require heuristics such as delaying EOS.
- During training, the known target sequence is fed into the decoder so it can compute the probability assigned to each correct next token.
- This procedure is called teacher forcing: it supplies the correct preceding target tokens rather than relying on the model's early, often poor predictions.
- Training compares output distributions with target tokens, computes divergence, and updates parameters with stochastic gradient descent.
- Encoder-decoder models apply to machine translation, speech recognition, dialogue, image captioning, and video captioning.
- Early image captioning used a CNN encoder, including an Inception model, to represent images before an RNN decoder generated captions.
- Visualizations of sentence representations showed semantic structure, including changes when subject and object were swapped.
- Compressing the entire input into the encoder's final hidden state overloads that representation, motivating the next lecture's discussion of the bottleneck.
Transcript excerpt
0:07 I'm scared. My dad's birthday. [clears throat] Does this administration have so many focus? every single thing. If there's a right choice and a wrong choice, they do it on purpose. He started
0:58 but it's not him. >> [clears throat] >> Can you come back? I'm hoping to say I don't think bad things have happened this happened the Yeah. >> [clears throat]
2:05 >> Not feeling great today. It's one of those days I have cycles. Aisha is in bad shape. >> So not just migraines have she's had maxilla facial surgery. Her jaw was she had a very huge overbite. So they moved her jaw front and whatever they've done is now causing problems. She can't open her mouth without it clicking. Mhm. So now she's we're going to got to find a doctor and take her. She may need something,
2:54 >> you know, if she if if you're lucky, it's just some therapy. If not, it's going to be something worse. >> She's not here. just suffering from severe migraines. Okay, good morning everybody. >> Let us hammer Good morning, Lucas.
4:08 >> [clears throat] >> You bicycle here. >> Yeah. >> You can make out they're wearing gloves. >> Did you bicycle here? >> The gloves. What's this? All right, guys. Everybody laptop shut.
5:24 The usual story. and let's begin somebody. Okay, which case do we have the Pokemons? >> Okay. Uh what was the last topic we covered in class? >> Sequence to sequence models and specifically what kind of uh sequence to sequence problems did we look at? >> Ah yeah aligned but not necessarily time aligned. Right. So, uh this these were the this was the class of problems we looking at. A sequence goes in, a different sequence comes out like speech recognition speech
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Carnegie Mellon University Deep Learning.