Lecture 7e

Overview

Prof. Schniter moves from a PyTorch training-loop recap to practical methods for making deep networks train effectively, including activation choices, weight initialization, batch normalization, dropout, and learning-rate schedules. The lecture also explains why very wide networks can exhibit double descent rather than classical overfitting, and emphasizes using validation results and computational limits to choose model size.

Key takeaways

PyTorch's loss.backward() computes gradients from the model's forward pass, but output representation and loss must match: use logits with BCEWithLogitsLoss or CrossEntropyLoss, and probabilities with BCELoss or NLLLoss.
Sigmoid and tanh can cause vanishing gradients because their derivatives shrink across layers; ReLU preserves gradients for positive inputs, while GELU, Swish, and Mish offer smoother nonlinear alternatives.
Batch normalization standardizes each feature within a mini-batch, learns gamma and beta, and uses running statistics at inference, often permitting faster training with higher learning rates.
Inverted dropout zeros activations with probability p and scales survivors by 1/(1-p) during training; inference then uses the complete network, approximating an ensemble of subnetworks.
Learning-rate schedules balance exploration and convergence: the one-cycle method ramps from a moderate rate to about 10 times higher before cooling nearly to zero.
Neural-network test error can follow double descent rather than a simple U-shaped bias-variance curve, so width and depth should be selected using validation performance and available compute.

Chapters

0:00 PyTorch Training Loops and Matching Outputs to Losses

A typical PyTorch workflow defines a dataset, model forward pass, optimizer, and nested epoch and mini-batch loops.
Within each batch, evaluate the loss, call loss.backward() for gradients, and use optimizer.step() to update parameters.
Track training loss and metrics such as classification accuracy, and compare them with held-out test performance to monitor generalization.
For binary classification, pair logits with BCEWithLogitsLoss or sigmoid activations with BCELoss; for multiclass tasks, pair logits with CrossEntropyLoss or probabilities with NLLLoss.

6:31 Why Deep Networks Beat Impractically Wide Two-Layer Models

The universal approximation theorem says a two-layer network with enough hidden units and a non-polynomial activation such as ReLU or sigmoid can approximate continuous functions.
Achieving that capacity through width alone can require impractically many hidden units, training examples, and resources, and can make optimization difficult.
Adding layers provides a practical way to increase model depth while keeping each layer at a manageable width.
Early deep learning faced limited data and computing power, as well as the vanishing-gradient problem.

13:07 Vanishing Gradients, ReLU, and Smoother Activations

Sigmoid and tanh derivatives are generally below 1, so multiplying gradients across many layers can leave early-layer parameters with tiny updates.
ReLU has slope 1 for positive inputs, helping gradients pass through active units, but units stuck at zero can become dead ReLUs.
Leaky ReLU uses a small negative-side slope, commonly 0.01, while PReLU learns that slope as a parameter.
GELU, Swish, and Mish provide smoother alternatives to ReLU; their activation inputs should remain in a useful range rather than saturating far from zero.

18:22 Weight Initialization and Layer-Specific Scaling

Weights are commonly initialized independently from a zero-mean uniform or Gaussian distribution, with the standard deviation selected deliberately.
A standard deviation proportional to gain divided by the square root of the layer's input width helps keep signals and gradient scales controlled across layers.
The appropriate gain depends on the activation, so a generic linear-layer default may not be ideal for a network using ReLU or another nonlinearity.
PyTorch's torch.nn.init utilities allow model builders to set initialization schemes before training; Prof. Schniter recommends Andrej Karpathy's educational material for practical PyTorch guidance.

23:34 Batch Normalization: Mini-Batch Statistics and Learned Scale

Batch normalization standardizes each feature using its mean and variance across the current mini-batch.
A small epsilon is added to the variance-based denominator to prevent division by zero.
Learned parameters gamma and beta rescale and shift each standardized feature so the network can choose useful output statistics.
During inference, running averages accumulated during training replace mini-batch statistics, which may not be available for a single test example.

28:00 Batch Normalization Speeds Inception Training

The cited Inception experiment on ImageNet compared classification accuracy over training iterations, with the original network requiring about 30 million iterations to reach a reference accuracy.
Adding batch normalization improved convergence at the same learning rate; increasing the learning rate by 5 times or 30 times changed both convergence speed and final accuracy.
Batch normalization helps keep layer and feature scales more consistent, enabling more aggressive learning rates and smoothing the optimization landscape.
The experiment also showed that a sigmoid network with batch normalization could train, although it did not match the ReLU-based result.

32:09 Dropout Masks Activations During Training

Dropout randomly sets a fraction of layer activations to zero, creating a different sparse subnetwork for each mini-batch.
With drop probability p, a surviving activation is scaled by 1/(1-p), preserving its expected value; for p = 0.5, about half the activations are removed on average.
Dropout is applied during training, while inference uses the full network.
The resulting training procedure resembles training many related subnetworks that share parameters.

35:39 Dropout as Ensembling and Its Training Trade-Offs

Dropout can improve generalization by combining the effects of many sparse subnetworks, an effect related to ensembling.
Training often takes longer because each batch uses a weakened network, and dropout can make training performance look worse than test performance.
The lecture recommends starting with an adequately powerful network so the individual subnetworks can still learn useful behavior.
Dropout is more commonly applied before dense layers than convolutional layers; modern image networks such as ResNet may have few dense layers where it can be used.

40:28 Why Nonconvex Networks Need Learning-Rate Schedules

In convex optimization, a learning rate that is too small mainly slows convergence, while one that is too large can cause persistent oscillation.
For nonconvex neural-network losses, a small learning rate may trap gradient descent in a nearby poor local minimum.
A large learning rate can jump across shallow barriers, but it may also prevent the optimizer from settling near the bottom of a good minimum.
A common strategy is to use a larger rate early and reduce it over epochs or iterations; schedules can be stepwise, exponential, cosine, or cyclic.

45:32 One-Cycle Learning Rates and Range Tests

The one-cycle schedule begins at a moderate rate, increases to roughly 10 times that value, decreases back, and then cools toward nearly zero.
The initial moderate phase lets weights become reasonable before a large learning rate helps move out of shallow minima; the final phase supports fine convergence.
A learning-rate range test runs a small number of iterations from the same initialization across rates such as 10^-7 to 10^1, then compares losses.
The lecture identifies a rate near the bottom of the loss-versus-rate curve as a useful initial learning rate and notes that one-cycle training can converge quickly.

50:12 Momentum, Adaptive Solvers, and Double Descent

Momentum combines the current gradient with previous gradient information, helping updates continue in a consistent direction and move through shallow regions.
Adaptive step sizes adjust learning rates during optimization; momentum and adaptivity are ingredients in methods such as AdamW.
A 2019-era ResNet-18 width experiment illustrated double descent: test error can rise with model size, then fall again as the network becomes wide enough.
This behavior depends on factors including training duration and regularization, so the classical bias-variance curve does not describe every neural-network regime.

55:30 Choosing Width, Depth, and Dropout for Image Networks

For width selection, compare validation or cross-validation error across models and stop increasing width when improvements become negligible.
Depth has no universal formula: gradients, architecture, learning-rate choices, data, and compute all affect whether deeper models help.
Residual networks use skip connections to support very deep designs, potentially reaching depths around 1,000 layers, though the best depth still depends on the task.
For image classification, convolutional layers increasingly replaced dense layers; Prof. Schniter postpones the promised image-processing introduction to the next class.

Transcript excerpt

0:01 Okay. Uh, happy Friday everyone. So, last time we were talking about how to [clears throat] code these deep networks in PyTorch. Um, what PyTorch does for us essentially is it does all the back propagation without us deriving a single thing. Um, it makes extremely complicated networks and models um, essentially trivial to train. So we talked about a recipe of how how you organize how you write your pie torch code. Um first you basically describe your data set in a [snorts] way that it um it can build the computation graph and distribute it across possibly many

0:50 machines and give you many batches very easily. Then you program the the model the deep network in this case. Essentially what you do is you program the forward pass and from that it's able to figure out the backward pass. You tell it which kind of optimization algorithm you want to use. They're all variants of stochastic gradient descent but with um some improvements to avoid local minima and converge faster and so on. Um and then you write your main loop that has an outer or your main piece of code that has an outer loop over epochs and inner loop of batches within that epoch. And in the inner loop you do a

1:39 forward pass. you um you evaluate [clears throat] your loss. Then you um just with loss.backward that essentially asks PyTorch to do the backward pass to compute the gradients. And then the opt step takes one step of the optimizer using those computed gradients. And then you can do whatever other things you want to do to keep track of how your network is training. So you may want to um save the loss versus epochs. You may want to save other performance metrics like classification if you're

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof. Schniter.