Backpropagation & Optimization
The chain rule at scale, SGD, momentum and Adam.
Training a network means computing the gradient of one number, the loss, with respect to millions of weights, thousands of times over. Backpropagation does it for about twice the cost of the forward pass. Everything else in this lesson (initialization, momentum, Adam, schedules, normalization) exists to make those gradients useful.
What you’ll learn
- Derive the backpropagation equations for a multilayer perceptron in matrix form, for one example and for a mini-batch, and count their time and memory cost.
- Explain vanishing and exploding gradients as products of layer Jacobians, and derive the LeCun, Glorot and He initializations from variance propagation.
- Implement mini-batch SGD with momentum and Adam, including Adam's bias correction, and explain why Adam's steps are scale-invariant.
- Choose a learning-rate schedule with warmup, and state what batch normalization and layer normalization compute at training and at test time.
Backpropagation in matrix form
Take an MLP with layers : pre-activations , activations , input , and a loss computed from the output . The network is a computational graph, and backpropagation is the chain rule applied to it from the loss backward. Its bookkeeping device is each layer's error signal , a column vector with one entry per unit.
The Jacobian of one layer's pre-activations with respect to the previous layer's is a weight matrix times a diagonal matrix of activation slopes:
For a scalar loss, the chain rule multiplies the gradient by the transposed Jacobian. The diagonal matrix acts elementwise:
Each weight enters only , with coefficient . So the weight gradient is an outer product, and the bias gradient is the error signal itself:
The recursion starts at the output. For a sigmoid or softmax output with the cross-entropy loss it is "prediction minus target" (derived in From Neurons to Networks):
For a mini-batch of examples, stack them as rows, and , with the batch mean, so that . Every step becomes a matrix product:
import numpy as np
def forward_backward(X, Y, Ws, bs):
"""X: (B, n0) inputs, Y: (B, K) one-hot labels, Ws[l]: (n_{l+1}, n_l).
tanh hidden layers, softmax output, mean cross-entropy."""
Hs = [X] # the "tape": every layer's input
for W, b in zip(Ws[:-1], bs[:-1]):
Hs.append(np.tanh(Hs[-1] @ W.T + b))
Z = Hs[-1] @ Ws[-1].T + bs[-1]
P = np.exp(Z - Z.max(axis=1, keepdims=True))
P /= P.sum(axis=1, keepdims=True)
loss = -np.mean(np.sum(Y * np.log(P + 1e-12), axis=1))
D = (P - Y) / len(X) # Delta at the output
gWs, gbs = [], []
for l in range(len(Ws) - 1, -1, -1):
gWs.insert(0, D.T @ Hs[l]) # same shape as Ws[l]
gbs.insert(0, D.sum(axis=0))
if l > 0:
D = (D @ Ws[l]) * (1 - Hs[l] ** 2) # tanh'(z) = 1 - tanh(z)^2
return loss, gWs, gbsThe backward loop mirrors the forward loop in reverse, reusing every stored activation. Production frameworks build this loop automatically from the forward code, by reverse-mode automatic differentiation.
Cost. The forward pass does one matrix product per layer, about multiply-adds for weights. The backward pass does two per layer, for the weights and for the next error signal, so the whole gradient costs about three forward passes, against the passes of finite differences (the comparison in A Short History of AI). The price is memory: every must be kept until the backward pass reaches it, numbers, often more than the weights themselves.
In the widget, , the weight from to the output is , and . What is ?
Show the answer
C.
. The error signal passes back through the weight and the activation's slope, and since , every sigmoid layer shrinks it.
Vanishing and exploding gradients, and initialization
Unroll the recursion and the error signal at layer is a product of layer Jacobians:
Long products of matrices behave like powers. If the factors typically shrink vectors, decays geometrically toward the input (vanishing gradients); if they stretch them, it grows geometrically (exploding gradients). From Neurons to Networks bounded the sigmoid case by . Since , early layers then learn far more slowly, or more erratically, than late ones.
The remedy for explosions is blunt: gradient clipping rescales the gradient whenever its norm exceeds a threshold , (Pascanu, Mikolov & Bengio, 2013). The remedy for both is to choose weight scales that make the typical factor close to 1, which is what a careful initialization does.
Draw the weights of layer independently with mean 0 and variance , independent of the layer's inputs. A pre-activation is a sum of independent terms with mean 0, so its variance is
For tanh near the origin , so , and the forward variance is preserved when . The same computation for the backward recursion sums over terms and asks for :
A ReLU zeroes the negative half of a symmetric , so . Each layer then halves the variance unless the weights supply the missing factor 2:
Glorot's uniform version draws from with , because . Biases start at zero. All of this fixes the scale only at initialization; normalization layers (step 5) keep it in check during training.
A deep ReLU network is initialized with variance , right for tanh but missing He's factor of 2. After how many layers has the variance of the pre-activations fallen below of its value at the first layer?
Show the answer
10
Each layer multiplies the variance by , and . After 50 layers the factor is . He et al. (2015) reported exactly this failure: their 30-layer ReLU network trained with the factor 2 but stalled completely with the Glorot scale.
Mini-batch SGD and momentum
The training loss is an average over examples, . A mini-batch drawn uniformly at random gives an unbiased estimate of its gradient, and the estimate's noise shrinks with the batch size:
function SGD-MOMENTUM(θ, learning rate η, momentum β ∈ [0, 1), batch size b, epochs E):
v ← 0
for epoch = 1 to E:
shuffle the training set and split it into batches of size b
for each batch 𝓑:
g ← (1/b) Σ_{i ∈ 𝓑} ∇ℓ_i(θ) # one forward and one backward pass
v ← β·v + g # heavy-ball velocity
θ ← θ − η·vOne pass over the data is an epoch. With this is plain SGD. Deep-learning libraries use this form of momentum; the variant is the same method with rescaled by .
Why momentum helps: unroll , an exponentially weighted sum of past gradients. Components that keep their sign add up, to for a constant gradient, a ten times larger step at . Components that flip sign from step to step, like the zig-zag across a ravine, cancel. The ravine, the condition number and Nesterov's accelerated method are treated in The Math Toolkit of AI and Optimization & Convexity. Noise matters as well: smaller batches and larger steps make SGD noisier, and that noise is widely believed to influence which minimum training ends in (see Loss Landscapes & Initialization).
Think first You raise the batch size from 32 to 512 and keep the learning rate. By what factor does the standard deviation of the gradient noise shrink, and what does that suggest about the learning rate?
The covariance scales as , so the standard deviation shrinks by . Each step now costs 16 times as much but is only 4 times less noisy, so at the same an epoch makes 16 times fewer, only slightly better steps. The linear scaling rule (Goyal et al., 2017) raises in proportion to the batch size, with a warmup, and works up to a problem-dependent critical batch size beyond which larger batches stop paying off.
Adaptive step sizes: Adam
One global learning rate fits badly when gradient scales differ across parameters, as they do between layers, or between the embedding of a common word and that of a rare one. AdaGrad (Duchi, Hazan & Singer, 2011) divides each coordinate's step by the root of its summed squared gradients. RMSProp (Tieleman & Hinton, 2012) uses an exponential moving average instead, so old gradients are forgotten. Adam (Kingma & Ba, 2015) adds momentum and corrects both averages for starting at zero.
function ADAM(θ, η = 10⁻³, β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸):
m ← 0; v ← 0 # first and second moment estimates
for t = 1, 2, …:
g ← stochastic gradient at θ
m ← β₁·m + (1 − β₁)·g
v ← β₂·v + (1 − β₂)·g⊙g # elementwise square
m̂ ← m / (1 − β₁ᵗ); v̂ ← v / (1 − β₂ᵗ) # bias correction
θ ← θ − η · m̂ / (√v̂ + ε) # elementwiseThe defaults are Kingma and Ba's. The state is two extra numbers per parameter, which for large models is a significant share of training memory.
Unroll the first-moment average from :
If the gradients have a constant mean, the weights sum to , so is biased toward 0 early on:
The same holds for . At the corrected ratio is exactly the sign of the gradient, so the first step moves every coordinate by . Without either correction it would be times larger:
Two properties follow from the ratio . It is scale-invariant: multiplying the loss by any leaves Adam's steps unchanged (up to ), while SGD's steps grow by . And it is roughly bounded: when a coordinate's gradient keeps its sign, its step is about whatever the gradient's size, so sets how far each weight moves per step rather than multiplying the gradient.
You multiply the training loss by 100, a change of units. What happens to the parameter updates of SGD and of Adam (ignoring )?
- Both become 100 times larger
- SGD's become 100 times larger; Adam's are unchanged
- SGD's are unchanged; Adam's become 10 times larger
- Both are unchanged, since the minimizer is the same
Show the answer
B. SGD's become 100 times larger; Adam's are unchanged
SGD's step is , so it scales with the loss. Adam's is : grows by 100 and by , so the ratio is unchanged. This is one reason Adam's default transfers across problems far better than any fixed SGD learning rate.
Schedules, warmup and normalization layers
With a constant learning rate, SGD never settles: near a minimum, gradient noise keeps it bouncing in a region whose size grows with (Optimizers in Depth derives this). So is decayed during training. Step decay divides it by 10 at fixed epochs; cosine decay lowers it smoothly to near zero (Loshchilov & Hutter, 2017); linear decay is common for transformers. Warmup ramps up from near zero over the first hundreds or thousands of steps, because early on Adam's second-moment estimates rest on few gradients and the network can sit in a sharp region where a full-size step diverges. Warmup is standard for transformers and for large-batch training (Goyal et al., 2017).
Batch normalization (Ioffe & Szegedy, 2015) attacks the scale problem directly. For each feature, it standardizes the pre-activations over the mini-batch, then applies a learned scale and shift:
At test time there may be no batch, so BN switches to running averages of and collected during training, and each example's output stops depending on the others. Layer normalization (Ba, Kiros & Hinton, 2016) computes the same statistics over the features of each example instead. It behaves identically in training and testing, which suits recurrent networks and transformers.
| Property | Batch norm | Layer norm |
|---|---|---|
| Statistics over | the batch, separately per feature | the features, separately per example |
| Training vs test | batch statistics vs running averages | identical |
| Small batches | noisy statistics, worse results | unaffected |
| Typical home | convolutional networks | transformers and RNNs |
Key takeaways
- Backpropagation runs the chain rule backward: and . In mini-batch form that is two matrix products per layer, about twice the forward cost, paid for in activation memory.
- Gradients are products of layer Jacobians, so they vanish or explode geometrically with depth. LeCun (), Glorot () and He (, for ReLU) initializations keep the variance of signals constant across layers.
- Mini-batch SGD follows unbiased gradient estimates whose noise falls as ; momentum sums past gradients, amplifying consistent directions by and cancelling oscillations.
- Adam combines momentum with per-coordinate scaling and bias correction. Its steps are scale-invariant and about in size, which makes it robust to the learning rate.
- Schedules decay to shrink SGD's noise floor and warmup protects the first steps; batch norm (over the batch, with running averages at test time) and layer norm (over features, identical at test time) keep activations at a stable scale.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
For a mini-batch with error signals (the of the mean loss included) and layer inputs , which expression is the gradient with respect to ?
Show the answer
B.
, the sum of the per-example outer products, with shape like .
A ReLU layer has 512 inputs and 256 outputs. Which weight variance does He initialization prescribe?
Show the answer
C.
He's rule is : the factor 2 compensates for ReLU zeroing half of a symmetric input, since . The standard deviation is .
An implementation of Adam corrects for its bias but forgets to correct . With , what happens on the first step?
- Nothing noticeable: bias correction only matters after many steps
- The first step is about 1,000 times too small
- Each coordinate moves about instead of
- The step points uphill
Show the answer
C. Each coordinate moves about instead of
but , so the step is . The uncorrected is tiny early on, and dividing by its root inflates the first steps: exactly the instability the correction prevents.
SGD with heavy-ball momentum, and , sees the same gradient at every step. Which step does it settle into for ?
Show the answer
C.
The velocity converges to the fixed point of , which is . Along directions where the gradient is consistent, momentum multiplies the effective learning rate by , while oscillating components cancel.
An MLP with batch normalization after each linear layer is evaluated one example at a time, but it was accidentally left in training mode. What goes wrong?
- Nothing: batch norm computes the same function in both modes
- The running averages are updated, but predictions are unaffected
- The learned and are reset to 1 and 0
- Each example is normalized by its own statistics, so every normalized value is 0 and the output no longer depends on the input
Show the answer
D. Each example is normalized by its own statistics, so every normalized value is 0 and the output no longer depends on the input
With a batch of one, and , so and for every input. In evaluation mode BN uses the running averages from training and acts as a fixed affine map. Layer norm has no such mode switch.
Go deeper
- Automatic Differentiation · Forward and reverse mode, and computational graphs.
- Optimizers in Depth · Adam, schedules, warmup and why they work.
- Loss Landscapes & Initialization · Saddle points, flat minima and signal propagation.
Further reading
- I. Goodfellow, Y. Bengio & A. Courville, Deep Learning, §6.5 and ch. 8 (MIT Press, 2016).
- X. Glorot & Y. Bengio, "Understanding the difficulty of training deep feedforward neural networks," AISTATS (2010).
- D. P. Kingma & J. Ba, "Adam: A method for stochastic optimization," ICLR (2015).
- S. Ioffe & C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," ICML (2015).