Building makemore Part 4: Becoming a Backprop Ninja

Overview

Andrej Karpathy's "Building makemore Part 4" details the manual implementation of backpropagation for a two-layer neural network, moving beyond PyTorch's autograd. This "Backprop Ninja" exercise covers deriving gradients for operations like cross-entropy loss, batch normalization, linear layers, and embeddings, emphasizing the importance of understanding internals for debugging and optimization. The lecture culminates in a fully manual training loop, demonstrating that complex neural networks can be trained without relying on automatic differentiation frameworks.

Key takeaways

Implementing backpropagation manually, even for simple networks, demystifies gradient flow and reveals potential issues masked by autograd.
The derivative of a sum in the forward pass becomes a replication/broadcasting in the backward pass, and vice-versa for replication/broadcasting.
Deriving gradients for complex operations like batch normalization requires careful application of the chain rule and understanding of vector calculus.
Simplified analytical gradients for operations like cross-entropy and batch normalization significantly improve computational efficiency compared to breaking them into atomic steps.
By manually implementing backpropagation, Andrej Karpathy demonstrates that achieving state-of-the-art results is possible without relying on automatic differentiation frameworks, fostering a deeper understanding of neural network mechanics.

Chapters

0:00 Motivation: The Need for Manual Backpropagation

Current implementation relies on PyTorch's `loss.backward()` for gradient calculation.
Manual backpropagation provides deeper understanding, aids debugging, and prevents "leaky abstraction" issues.
Historical context: manual backprop was standard practice ~10 years ago.

0:37 Why Manual Backprop is Crucial

Backpropagation is not magic; understanding internals is key to avoiding subtle bugs.
Examples of issues: dead neurons, exploding/vanishing gradients, incorrect gradient clipping.
Frameworks like PyTorch's autograd can mask fundamental misunderstandings.

5:00 Historical Examples of Manual Backpropagation

1990s/2000s: manual backprop was pervasive, even in Matlab.
Jeff Hinton's 2006 paper on Restricted Boltzmann Machines.
Karpathy's 2010 Matlab library for RBMs and 2014 Numpy implementation for image/text alignment.

11:46 Setting Up the Manual Backprop Notebook

Reusing the two-layer MLP architecture with batch normalization.
Removing `loss.backward()` and implementing manual backward pass.
Introducing a utility function `CNP` for gradient comparison with PyTorch.

13:18 Parameter Initialization for Debugging

Biases initialized to small random numbers instead of zero to unmask potential gradient errors.
Including a bias in the first layer despite batch normalization for exercise.
Retaining gradients for intermediate tensors using `.retain_grad()`.

15:05 Manual Backpropagation: D_log_probs

Calculating the derivative of the loss with respect to `log_probs`.
Loss is the negative mean of `log_probs` indexed by correct labels (`yb`).
Derivative is -1/N for participating elements, 0 for others.

31:46 Manual Backpropagation: D_probs

Applying the chain rule: `d_probs = d_log_probs * d(log)/d(probs)`.
Local derivative of log is `1/probs`.
Gradient is boosted for low-probability correct characters.

34:19 Manual Backpropagation: D_counts

Breaking down the softmax normalization step.
Differentiating through `counts = exp(logits - max_logit)`.
Gradient calculation involves `counts` and `d_probs`.

35:41 Manual Backpropagation: D_count_sum_inv

Differentiating through the inverse of the sum of counts (`1 / count_sum`).
Local derivative is `- (count_sum)^-2`.
Gradient is `d_count_sum_inv * local_derivative`.

37:04 Manual Backpropagation: D_count_sum

Differentiating through the sum of counts (`count.sum(dim=1, keepdim=True)`).
Summation in forward pass becomes broadcasting/replication in backward pass.
Gradient `d_count_sum` is `d_count_sum_inv` replicated across columns.

43:46 Manual Backpropagation: D_counts (Second Pass)

Differentiating `counts = exp(logits - max_logit)` with respect to `counts`.
Local derivative is `exp(logits - max_logit)` (i.e., `counts`).
Gradient `d_counts` is `d_count_sum * counts` (with broadcasting).

53:49 Manual Backpropagation: D_normal_logits

Differentiating `normal_logits = counts` (element-wise exponential).
Local derivative of `exp(x)` is `exp(x)`.
Gradient `d_normal_logits` is `d_counts * counts`.

55:14 Manual Backpropagation: D_logits (Softmax Branch)

Differentiating `logits = normal_logits - logit_maxes`.
Handles broadcasting of `logit_maxes`.
Gradient `d_logits` is `d_normal_logits` (with summation for broadcasted `logit_maxes`).

1:03:56 Understanding Logit Maxima's Role

Subtracting `logit_maxes` is for numerical stability in softmax.
Ideally, gradients w.r.t. `logit_maxes` should be near zero.
Small non-zero gradients due to floating-point arithmetic are observed.

1:05:08 Manual Backpropagation: D_logits (Max Index Branch)

Differentiating through the `max` operation using indices.
Gradient `d_logits` is scattered from `d_logit_maxes` to the max positions.
Uses `torch.nn.functional.one_hot` for scattering.

1:09:04 Manual Backpropagation: Linear Layer (Layer 2)

Differentiating `logits = h @ W2 + b2`.
Derivatives for `h`, `W2`, and `b2` calculated using matrix calculus principles.
Shape matching and broadcasting are key for correct gradient calculation.

1:28:56 Manual Backpropagation: Tanh Activation

Differentiating the `tanh` activation function.
Local derivative is `1 - tanh(z)^2` or `1 - h^2`.
Gradient `d_h_preact` is `d_h * (1 - h^2)`.

1:31:59 Manual Backpropagation: Batch Normalization

Breaking down batch norm into intermediate steps: mean, variance, standardization, scaling, shifting.
Differentiating `BN_gain * (x - mu) / sigma + BN_bias`.
Complex derivatives for `BN_gain`, `BN_bias`, `mu`, `sigma^2`, and `x`.

1:48:40 Batch Normalization: Bias Correction Discussion

Debate on using biased (1/N) vs. unbiased (1/(N-1)) variance estimation.
Paper uses biased during training, unbiased during inference (train-test mismatch).
Karpathy prefers consistent unbiased estimation using `N-1`.

1:55:03 Manual Backpropagation: Linear Layer (Layer 1)

Differentiating `h_bn = m @ W1 + b1`.
Derivatives `d_m`, `dW1`, `db1` calculated based on shapes and matrix multiplication rules.
Summation for `db1` due to broadcasting of `b1`.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Andrej Karpathy.