UUtah CS 6340 NLP | Fall 2026 | seq2seq & attention

Overview

The lecture derives attention for recurrent sequence-to-sequence models, showing how decoder steps can retrieve relevant encoder states instead of relying on a single fixed-size source representation. It formalizes dot-product scores, softmax weights, weighted context vectors, and vocabulary prediction, then connects the architecture to supervised negative log-likelihood training and explains why attention visualizations are not definitive explanations.

Key takeaways

In RNN sequence-to-sequence attention, each decoder state scores every encoder state, normalizes those scores with softmax, and uses them to form a position-specific weighted context.
The basic attention-based decoder combines its current hidden state with the weighted source context before projecting to vocabulary logits.
Attention addresses the single-vector bottleneck by letting decoder steps revisit individual source representations, which is especially useful for long inputs.
During supervised translation training, negative log-likelihood rewards probability assigned to the human-written next token; teacher forcing supplies target tokens, while inference uses generated tokens.
Attention heatmaps can show learned source-target associations, including phrase-level off-diagonal patterns, but they do not establish causal explanations of model behavior.
Attention is a general query-and-values operation: a query determines a weighted sum of value vectors, yielding a fixed-size context tailored to that query.

Chapters

0:00 From RNN Sequence-to-Sequence Models to Attention

The lecture builds on RNN language modeling and sequence-to-sequence tasks such as machine translation.
The encoder-decoder structure used by RNNs also provides a foundation for understanding the Transformer.
Attention is introduced as a key Transformer concept, though the basic mechanism here is not the Transformer’s exact implementation.

2:28 Why a Single Encoder Vector Becomes a Bottleneck

Vanilla sequence-to-sequence models compress the entire source into one vector used to condition every decoder step.
Long inputs, such as 100 documents of 1,000 tokens each, make it difficult for one representation to preserve all useful information.
Attention gives the decoder access to source positions dynamically, according to what is relevant for the next token.

5:56 Computing Attention for Each Decoder Step

The encoder produces a hidden representation for every source token; unlike the decoder, it does not predict next tokens.
The decoder starts from the encoder’s final hidden state and compares its current state with every source hidden state using dot products.
A softmax converts the unbounded scores into weights between 0 and 1 that sum to 1.
The decoder forms a source context vector by taking the weighted sum of encoder hidden states, then concatenates it with its current hidden state before vocabulary prediction.

13:30 Reading Attention Alignments—and Their Limits

Attention maps can display a score for each source-token and target-token pairing, often revealing diagonal patterns for literal translations.
Off-diagonal weights can indicate that a target word corresponds to a source phrase rather than one source word.
The model learns these weights during training without requiring explicit word-alignment labels.
Attention maps can aid inspection, but the lecture cautions against treating them as causal explanations of model behavior.

18:25 Setting Up the Encoder-Decoder Training Equations

The source sequence has length L and the target sequence length K; the two lengths need not match.
The encoder initializes its hidden state with a zero vector and computes a representation for each of the L source tokens.
The decoder’s initial hidden state is the encoder’s final state, linking the two networks.
Encoder and decoder use separate learned weights, and the encoder does not need a next-token softmax.

25:25 Formalizing Dot-Product Scores and Context Vectors

For decoder position i, a dot product between its hidden state and source state j gives a scalar attention score for that source token.
The scores across the L source positions are collected and passed through softmax to produce normalized attention weights.
The context vector is the sum of each source hidden state multiplied by its attention weight, creating a source representation specialized to decoder position i.
The decoder combines this context with its current hidden state before projecting to vocabulary logits and applying softmax.

32:30 How Context Vectors Influence Predictions and Gradients

A high attention weight makes the corresponding source hidden state contribute more strongly to the context vector.
For literal translation, the first decoding step may assign greater weight to the first source token when predicting the first translated word.
The encoder remains trainable: its states affect decoder initialization and attention contexts, so backpropagation sends gradients through those connections.
The context is a selective, decoder-dependent summary—not a replacement for the encoder’s individual source representations.

38:12 Training with Human Translations and Negative Log-Likelihood

Supervised machine translation uses human-written source-target sentence pairs, typically from a parallel corpus.
With teacher forcing, the decoder receives the appropriate target tokens during training; at inference, it conditions on tokens generated in earlier steps.
Training maximizes the probability of the human-written target sequence given its source, commonly by minimizing negative log-likelihood.
At each position, the loss uses the softmax probability assigned to the correct target token; PyTorch loss functions differ in whether they expect logits or probabilities.

43:46 Combining Decoder and Context Representations

The attention context and current decoder state are concatenated to form an enhanced decoder representation.
That concatenated vector is passed through an output projection to produce logits over the vocabulary.
Concatenation is a basic design choice; the lecture notes that an MLP or other transformations could also combine the representations.
The tilde on a hidden-state symbol is simply notation for the enhanced representation, not a separate operation by itself.

47:00 Output Weights, Model Parameters, and Initialization

The output matrix maps the combined representation to vocabulary-sized logits; its name reflects its role in producing model output.
The learned parameters include separate encoder and decoder recurrent weights, biases, embeddings, and the output projection.
Parameters have defined matrix shapes and are initialized before training; Xavier initialization is cited as a common approach.
The output softmax turns logits into token probabilities used for prediction and the training loss.

51:11 Attention’s 2015 Impact and General Definition

Attention emerged in 2015 and substantially improved neural machine translation while addressing the fixed-vector bottleneck.
The lecture also credits attention with helping mitigate vanishing-gradient problems in sequence modeling.
More generally, attention uses a query to compute a weighted sum of value vectors; here, the decoder state is the query and encoder states are the values.
The resulting context can be understood as a fixed-size, query-dependent selective summary of an arbitrary set of representations.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.