10/28/26: Lect-09
Watch on YouTube →
Overview
Joseph Zambreno explains why ordered data such as text, sensor streams, and video requires sequence models: shuffling destroys useful relationships, while fixed-window MLP regression loses older context and struggles with multi-step forecasts. He develops recurrent neural networks as a way to carry a fixed-size hidden summary forward, then connects that idea to language modeling through tokenization, embeddings, teacher forcing, and Andrej Karpathy’s Shakespeare character-RNN demonstration.
Key takeaways
- Sequence order is part of the data’s meaning: shuffling text, sensor readings, or video frames removes dependencies that ordinary independent-sample MLP and CNN setups do not represent.
- A fixed n-gram window makes autoregression compatible with an MLP, but irreversibly excludes context older than the chosen window size.
- One-step autoregressive prediction can work on smooth, autocorrelated signals with modest noise, but iterative multi-step forecasts compound errors because predicted values replace unknown future observations.
- An RNN’s recurrent connection updates a fixed-size hidden state from both the current input and prior state, preserving a learned summary of sequence history across arbitrary input lengths.
- Teacher forcing trains a language model efficiently by pairing each known token with the next corpus token, but inference must use generated predictions because the true continuation is unavailable.
- A character-level Shakespeare model can learn plausible local spelling and style from about 40,000 lines, yet its fluent-sounding output may still contain invented words and incoherent story references.
Chapters
- Earlier MLP and CNN tasks treated samples as independent, so training and test examples could be shuffled.
- In text, sensor streams, financial data, and video, neighboring observations carry information; randomizing video frames destroys motion context.
- Sequence tasks vary by where the sequence appears: document classification is many-to-one, image captioning is one-to-many, and translation is many-to-many.
- An autoregressive model predicts the next value from earlier values, but a growing history creates a variable-length input that standard MLPs do not accept.
- MLP weights attach to absolute input positions, even though sequence meaning depends on relative order and positions shift as new data arrives.
- A sliding window or n-gram model fixes input size, for example using the last four values, but discards all information older than the window.
- The example generates 1,000 sine-wave samples with noise scaled to 20% of the signal and uses a window of four prior samples.
- The feature vectors are sliding sequences such as x[0:4] and x[1:5], with each target being the next value.
- Training uses the first 600 sequences, and a basic linear regression model learns one-step-ahead predictions from the four-value windows.
- One-step prediction can work when the signal has autocorrelation, seasonality, or a smooth trend, and noise is small relative to the signal.
- The model must have timely access to prior observations, and its learned relationship should remain reasonably stable over time.
- For multi-step forecasts, each prediction becomes input to the next; errors compound, and forecasts at 4, 16, or 64 steps deteriorate.
- A noisy random-walk stock series offers little predictive signal, so a simple model may do no better than repeating the latest price.
- Instead of feeding a growing history into an MLP, a latent autoregressive model maintains a fixed-size hidden state, hₜ.
- The hidden state is updated as new observations arrive and is intended to summarize information from the full sequence.
- Predictions use both the current hidden summary and new input, allowing the model to carry context older than a fixed sliding window.
- A recurrent neural network updates hidden state from both the current input xₜ and prior state hₜ₋₁, using learned input and recurrent weight matrices.
- The updated hidden state feeds an output layer, much like an MLP, but the recurrent connection carries information across sequence positions.
- The recurrent loop makes input order matter; without it, samples remain independent and can be shuffled as in ordinary MLP training.
- Inference generally needs the latest hidden state, while backpropagation through time needs intermediate states to update weights.
- Natural-language tasks commonly use RNNs alongside input preprocessing and output layers, rather than treating the recurrent layer as a complete classifier.
- A tokenizer turns a text corpus into sequence values; whole words and individual characters are possible choices, each with different tradeoffs.
- Embeddings map arbitrary token IDs into learned multidimensional representations, avoiding the false assumption that numeric token order conveys linguistic meaning.
- Text generation can be self-supervised by using each corpus token as the target for the preceding context; translation instead requires paired language examples.
- Karpathy’s character-RNN trains on roughly 40,000 lines of Shakespeare, tokenizing the corpus one character at a time.
- Teacher forcing shifts known text by one position: the model predicts the next character, receives a loss, then uses the true character for the next training step.
- This training setup is more efficient and stable than feeding the model’s own predictions back during training, but it is unavailable during generation.
- A from-scratch RNN uses separate transformations for the current input and previous hidden state, then iterates through sequence positions in a loop.
- The same recurrent weights are reused at every position, enabling patterns learned at one text position to apply elsewhere and supporting variable-length sequences.
- Long sequences make backpropagation through time vulnerable to vanishing or exploding gradients; the lecture introduces tanh, bounded between -1 and 1, as a relevant activation.
- The corpus vocabulary is built by counting characters; spaces are most frequent, followed by letters including E, T, O, and A.
- Training examples use 100-character input windows with targets shifted ahead by one character, implementing teacher forcing.
- Karpathy’s example uses a built-in PyTorch LSTM, a learned token embedding, a recurrent layer, and an output layer that predicts vocabulary tokens.
- Cross-entropy measures token predictions, connecting the language model’s training objective to standard classification.
- At generation time, the model samples or greedily selects a next token, appends it to the prompt, and repeats without teacher forcing.
- Because each generated character becomes later input, prediction errors can compound; a King Lear prompt produced Shakespeare-like phrasing but also malformed words and incoherent references.
- The demonstration shows that a recurrent hidden state captures sequence patterns that a basic MLP cannot, while remaining far below a reliable Shakespeare text generator.
- Zambreno closes by previewing further RNN practice, LSTM and GRU models, and later attention and transformer topics.
Transcript excerpt
0:00 But let's let's go ahead and you know officially get started. So this is our next major topic in the course. This is sequence modeling and you've done some of this already in lab. Um it's okay that the lab happened first. You know for earlier labs the labs happen a little bit late and now for later labs the labs are happening first and that that was always going to happen. Um but my hope is okay you've done it. You've written some code. You've gotten some things to work or not work to some degree in the lab. But now let's talk about what this means you know in reality and really kind of cover more of the appropriate background for that and that's the hope. So you tried it made some progress now there should be a lot more of okay I got I understand now why we did what we did for that particular part uh in lab. Um I also like this quote I don't like all the quotes equally but I like this one that prediction is very difficult especially about the future. I think you'd see it attributed to Yogi Barra more often than
0:48 Neils Boore, but no one really knows who originally uh created uh that line and it's appropriate because that that's kind of what we're trying to do when we talk about sequence models. It is really about attempting to predict what comes next, attempting to predict the future. So my my hope is today uh certainly we'll spend some time on the justification for sequence models. It's not that much more than what I just said. Um we'll we'll talk about tokenization, language models, and perplexity. We'll definitely get to that today. We will talk about RNN's and we'll kind of bridge that topic between
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Joseph Zambreno.