CMU Introduction To Deep Learning 11-785, Fall 2026: Lecture 3

Overview

The lecture explains how learning turns a neural network’s representational capacity into a fitted model: parameters are estimated from labeled examples by minimizing empirical loss rather than by hand-writing the target function. It derives the perceptron update for linearly separable data, shows why hidden-neuron labels make that rule impractical for multilayer perceptrons, and motivates differentiable activations and divergence-based empirical risk minimization.

Key takeaways

The perceptron update w ← w + yx corrects a misclassified example for y ∈ {−1, +1}, and converges in finite time when the training classes are linearly separable.
Directly applying perceptron learning to an MLP requires hidden-unit labels that are absent from the training data; trying 2ⁿ labels for one neuron already creates an impractical search over n examples.
Threshold activations block useful local learning signals because their output is flat almost everywhere, so changing weights may not change the counted classification error.
Smooth activations such as sigmoid make output changes measurable, allowing the chain rule to connect changes in early-layer weights to changes in the final prediction.
Training minimizes average divergence on sampled examples as an empirical estimate of expected error under P(x), rather than directly minimizing error over every possible input.

Chapters

0:00 Classroom Setup and Lecture 3 Roadmap

The opening includes a clicker malfunction before the lecture begins.
Students are asked to close laptops, make name tags, and respond to attendance poll 139.
The agenda is learning neural-network parameters, the perceptron learning rule, greedy approaches, and empirical risk minimization.

7:00 Recap: MLPs as Universal Function Approximators

A multilayer perceptron can approximate Boolean functions, classifiers, and continuous-valued functions to arbitrary accuracy when its architecture has sufficient width and depth.
Universal approximation describes what some appropriately designed MLP can represent—not what every network architecture can represent.
The lecture shifts from representation to the practical question of how to find the network’s parameters.

10:00 Functions, Perceptrons, and Connectionist Networks

A function maps each input to one corresponding output; computer implementations represent both as numbers.
A perceptron computes a weighted sum plus a bias, then applies an activation such as a threshold, sigmoid, or ReLU.
A connectionist machine uses simple computational units whose overall capacity comes from their connections; the networks discussed here are feed-forward, without loops.

14:15 Network Architecture and the Parameter Vector

A layer is a set of nodes at the same depth from the input, and architecture specifies layer count and connectivity.
The network can be written as f(x; w): x is the input, while w collects the weights and biases as parameters.
A bias can be represented explicitly or absorbed into the weights by appending a constant input of 1; every perceptron, including hidden units, has a bias.

16:15 From Hand-Crafted Boundaries to Learning Parameters

A simple region such as a diamond can be implemented by hand using perceptrons for its edges.
For a complicated target function, manually deriving weights is not practical even when an MLP with enough capacity can represent it.
Learning is framed as finding w so the network function f(x; w) matches the desired function g(x).

21:00 Training Samples Approximate an Unknown Target Function

The ideal objective would minimize the divergence between network output and target output across the entire input domain.
In practice, the target function is unknown or lacks a usable formula, so training uses sampled input-output pairs such as images paired with captions.
The average error measured on observed examples is empirical error: a practical proxy for the unobserved, domain-wide discrepancy.

28:00 Binary Classification and Empirical Error

For binary classification, each training input has a label of 0 or 1, and empirical error is the fraction of examples the model labels incorrectly.
The example classifier makes errors on 40% of the displayed instances, illustrating the goal of reducing training-set mistakes.
Fitting the training instances exactly is only a hope for generalization; it can fail when training examples do not represent the domain of interest.

30:10 A Perceptron’s Hyperplane and Bias Augmentation

A threshold perceptron is a linear classifier: the boundary is the hyperplane where the weighted sum plus bias equals zero.
Adding a constant input of 1 folds the bias into the weight vector, turning the affine boundary equation into a linear one in an augmented space.
With the boundary written wᵀx = 0, w is normal to the hyperplane; the sign of wᵀx identifies which side contains a point.

38:30 Rosenblatt’s Perceptron Update Rule

For labels y ∈ {−1, +1}, an incorrectly classified training example updates the weights as w ← w + yx.
A misclassified positive example pulls w toward x; a misclassified negative example pulls it toward −x.
For linearly separable data, the perceptron algorithm converges in a finite number of updates; if the data are not separable, the updates can continue indefinitely.

45:00 Why the Perceptron Rule Does Not Directly Train an MLP

A multilayer network can combine first-layer linear boundaries to represent a nonlinear region such as a double pentagon.
The perceptron rule needs labels for each unit, but the training set provides labels only for the network’s final output.
Even when all other units are known, the remaining hidden perceptron may receive labels that are not linearly separable.

49:30 Hidden-Unit Relabeling Creates Combinatorial Search

One proposed workaround is to try possible hidden-unit labels, train a linear separator, and check whether the whole MLP produces the correct final outputs.
For n examples, one hidden unit has 2ⁿ possible binary relabelings, many of which may be linearly separable but yield the wrong network output.
Repeating the search across multiple neurons makes direct threshold-MLP training infeasible because the hidden labels must jointly be correct.

54:00 Greedy Alternatives and the ADALINE Delta Rule

The hidden-label problem stalled neural-network training efforts for more than a decade, motivating approximate and greedy methods.
Bernard Widrow’s ADALINE and MADALINE work introduced the delta rule, which adjusts parameters using error measured before the activation.
The lecture turns toward smooth objectives as a way to guide parameter updates without enumerating hidden-unit labels.

57:00 Why Threshold Activations Give No Useful Gradient

A threshold classifier can misclassify examples while small changes to its threshold leave the error count unchanged.
Its derivative is zero almost everywhere and undefined at the threshold, so local parameter perturbations provide no reliable direction for improvement.
In higher dimensions, a decision boundary can move substantially without crossing any training sample, leaving the counted classification error flat.

1:01:30 Smooth Activations and Continuous Error Signals

Replacing a threshold with a graded activation lets small changes in weights produce measurable changes in neuron output.
Instead of counting only incorrect labels, the lecture measures the difference between each target and the network’s continuous output.
This continuous error supplies a directional signal even when the final 0.5 classification threshold has not crossed a training point.

1:11:00 Sigmoid Outputs, Differentiability, and Backpropagated Effects

The sigmoid σ(z) = 1/(1 + e⁻ᶻ) varies smoothly between values near 0 and 1 and can be interpreted as a class-1 probability.
A differentiable activation allows the chain rule to relate a small change in any weight to a change in the network output, including weights in earlier layers.
ReLU is differentiable almost everywhere; at its corner, a subgradient can still provide an update direction.

1:17:00 Divergence Functions and Expected Risk

A divergence measures discrepancy between target and predicted outputs, is zero when they match, and is positive otherwise so opposite-signed errors cannot cancel.
Squared error, cross-entropy, and KL divergence are examples of different discrepancy measures; the lecture does not assume outputs are probability distributions in every setting.
The desired objective weights divergence by the input distribution P(x), giving greater importance to inputs that occur more often.

1:23:00 Empirical Risk Minimization Defines the Training Loss

A finite training set sampled from P(x) approximates expected divergence by the average divergence over observed examples, subject to generalization caveats.
With training inputs and targets fixed, this average is a function of the network parameters alone and is called the loss or empirical risk.
Empirical risk minimization trains the network by choosing parameters that minimize that loss.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Carnegie Mellon University Deep Learning.