UUtah CS 6340 NLP | Fall 2026 | Transformers: Part 2/2

Overview

The lecture completes the original Transformer architecture and explains how modern language models adapt it: learned and sinusoidal positional encodings give way to RoPE and related long-context techniques, while encoder and decoder blocks combine attention, feed-forward layers, residual connections, and normalization. It then derives teacher-forced causal attention, cross-attention, and next-token loss, highlighting practical constraints such as quadratic attention memory and the importance of masking future tokens.

Key takeaways

RoPE encodes position by rotating query and key vectors inside attention, avoiding the direct addition of positional vectors to token embeddings; however, it does not by itself ensure strong performance beyond the context lengths used in training.
Standard attention has quadratic memory growth in sequence length: increasing context from 4,000 to 128,000 tokens can require about 1,000 times more attention memory, so context length is constrained by both GPU capacity and generalization.
Teacher forcing shifts the known target sequence right by one position, allowing all next-token predictions to be trained in parallel when causal attention blocks access to future tokens.
Masking attention scores above the diagonal with negative infinity before softmax produces zero future-token weights while preserving normalized attention over the allowed past and current positions.
A standard encoder–decoder block uses unrestricted encoder self-attention, masked decoder self-attention, and cross-attention; cross-attention queries come from decoder states and keys and values come from encoder outputs.
Tying the embedding and output-projection weights can save a substantial number of parameters in smaller models, while larger models can afford separate matrices and may perform better without tying.

Chapters

0:00 Course Announcements: Practicum, CHPC GPUs, and Recorded Lecture

The second practice assignment is available for the practicum scheduled for the Monday after fall break; the autograder still needs fixes.
Monday includes the usual in-person quiz, while the lecture will be provided as a recording because the instructor is attending a language-modeling conference.
Wednesday's TA tutorial covers fine-tuning with the Hugging Face Transformers library and running GPU jobs on the University of Utah CHPC; students need to create CHPC accounts beforehand.

3:29 Transformer Inputs: Vocabulary IDs and Token Embeddings

The original 2017 Transformer is an encoder–decoder architecture: the encoder represents source tokens, and the decoder predicts target tokens one by one.
Tokenization maps each input token to a vocabulary ID, which indexes a row in the token-embedding matrix.
The original paper used a 512-dimensional model representation; modern open-weight models commonly use dimensions in the thousands.

7:48 Learned Positional Embeddings and Their Limits

A learned position-embedding matrix assigns a vector of the model dimension—512 in the original example—to every position up to a chosen maximum sequence length.
The position vector is added to the corresponding token embedding, mixing positional and semantic information.
A fixed maximum creates an extrapolation problem: positions beyond the training limit have no learned embedding, and a large position table adds parameters.

13:35 The 2017 Transformer’s Fixed Sinusoidal Position Encoding

The original Transformer paper replaced learned position vectors with fixed sine and cosine values, using sine for even vector indices and cosine for odd indices.
These deterministic encodings add no learned parameters and keep values between −1 and 1.
Different sinusoidal frequencies provide a structured signal in which nearby positions have related encodings, while the method still adds position information directly to token representations.

20:57 RoPE: Encoding Relative Position in Query–Key Attention

Rotary position embeddings (RoPE) apply position-dependent rotations to query and key vectors rather than adding position vectors to input token embeddings.
Because attention scores come from query–key dot products, rotations can make the score reflect relative token positions and favor useful local relationships.
RoPE is a core positional technique in modern LLMs, often combined with additional changes intended to improve efficiency or generalization.

23:24 Long-Context Limits: Attention Memory and NoPE Layers

Standard attention forms a sequence-length-by-sequence-length score matrix, so increasing context from 4,000 to 128,000 tokens can require roughly 1,000 times as much attention memory.
RoPE remains computable beyond the training context, but that does not guarantee good performance on longer sequences; context limits must also fit available GPU memory.
The lecture notes that million-token contexts are increasingly available, but long codebases and generated text can still exceed a model's trained context and degrade output quality.
NoPE layers omit positional encoding in some attention layers, encouraging the model to infer order from causal structure; this is presented as a research direction rather than a settled scaling result.

31:57 RoPE’s Continued Use and the Limits of Long-Sequence Generalization

RoPE and variants remain common in newer open-weight models; papers often add techniques specifically to extend its behavior to longer sequences.
The lecture distinguishes positional generalization from implementation: RoPE can rotate vectors at new positions even when model quality still deteriorates beyond the training length.
Long-context improvements often connect memory efficiency with better positional generalization, making both constraints important when designing a model.

35:05 Transformer Feed-Forward Networks: Expansion, Activation, Projection

After attention contextualizes each token, a feed-forward network transforms each token representation independently using an up-projection, a nonlinearity, and a down-projection.
In the original scale, a 512-dimensional representation might expand to roughly 2,000 dimensions; the lecture gives modern examples around 3,000 dimensions expanding to 14,000.
Activations evolved from ReLU to GELU and, in modern LLMs, gated units such as SwiGLU.

40:01 Residual Connections Preserve Token Information

A residual connection adds a block's input representation to its transformed output, preserving a direct path for the original information.
The addition is unweighted: the input and transformed output are summed rather than mixed with a learned 50/50 coefficient.
Attention and feed-forward sublayers both use residual connections so repeated processing does not have to discard earlier token information.

44:09 LayerNorm, RMSNorm, and Pre-Norm Transformer Blocks

Layer normalization stabilizes training by normalizing each token vector; learned scale and shift parameters are commonly called gamma and beta.
RMSNorm simplifies normalization by removing the learned beta parameter and is widely used in modern LLMs.
Pre-norm applies normalization before attention and feed-forward sublayers, whereas the original Transformer used post-norm; the lecture connects pre-norm's adoption to healthier gradients.
A pre-norm model commonly applies a final normalization before producing vocabulary logits; a short break then precedes the decoder discussion.

51:21 Decoder Training with Teacher Forcing

The decoder predicts a target sequence autoregressively, with the sequence probability factored into next-token probabilities by the chain rule.
During training, teacher forcing feeds the known target sequence shifted right by one position, beginning with a special start token.
Unlike inference, where generated tokens must be produced sequentially, teacher forcing makes it possible to compute training predictions for all positions in one forward pass.

55:00 Causal Attention Masks Future Tokens During Training

Teacher forcing exposes later ground-truth tokens in the input, so causal attention must prevent a position from using information about future target tokens.
The attention-score matrix is masked above its main diagonal; setting those scores to negative infinity before softmax makes their attention weights zero.
This mask preserves normalized attention over permitted positions and enables parallel loss computation across the sequence in a single forward pass.
Decoder-only LLMs rely on causal or masked self-attention rather than unrestricted encoder-style self-attention.

1:03:44 Decoder Cross-Attention Connects Target States to Encoder Outputs

After causal self-attention, an encoder–decoder Transformer uses cross-attention to connect decoder representations to the source representations produced by the final encoder block.
In standard cross-attention, decoder states supply queries, while encoder outputs supply keys and values; the resulting scores determine how each target position uses source information.
The decoder block combines causal attention, cross-attention, and a feed-forward network, with residual connections and normalization around the sublayers.
Unlike causal attention over target tokens, cross-attention to the fully available source sequence does not need a future-token mask.

1:10:16 Language-Modeling Head, Next-Token Loss, and Weight Tying

A linear projection maps each decoder state to vocabulary-sized logits, and softmax converts those logits into next-token probabilities.
Training uses the probability assigned to the ground-truth next token to calculate negative log-likelihood; it does not sample a token at each position.
Tied weights reuse the token-embedding matrix for the output projection, reducing parameters in smaller models.
For larger models, embedding and output weights are more often untied because the extra matrix is a smaller fraction of total parameters and tying may reduce performance.

1:16:05 Encoder-Only, Encoder–Decoder, and Decoder-Only Transformer Models

Encoder-only models became useful for classification: token representations can be pooled and passed to a classifier instead of a language-modeling head.
Encoder–decoder models such as T5 support sequence-to-sequence tasks, while decoder-only models became the dominant LLM architecture for generation.
A decoder-only model can process source text and then continue generating, avoiding a separate encoder even for sequence-to-sequence tasks.
Architecture trends are not permanent: encoder–decoder models continue to appear, so a new model should not be treated as evidence that the field has universally switched designs.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.