Simple Algorithm
Artificial Intelligence · Hidden Layer · 24 min

Backpropagation & Optimization

The chain rule at scale, SGD, momentum and Adam.

Training a network means computing the gradient of one number, the loss, with respect to millions of weights, thousands of times over. Backpropagation does it for about twice the cost of the forward pass. Everything else in this lesson (initialization, momentum, Adam, schedules, normalization) exists to make those gradients useful.

What you’ll learn

  • Derive the backpropagation equations for a multilayer perceptron in matrix form, for one example and for a mini-batch, and count their time and memory cost.
  • Explain vanishing and exploding gradients as products of layer Jacobians, and derive the LeCun, Glorot and He initializations from variance propagation.
  • Implement mini-batch SGD with momentum and Adam, including Adam's bias correction, and explain why Adam's steps are scale-invariant.
  • Choose a learning-rate schedule with warmup, and state what batch normalization and layer normalization compute at training and at test time.

Backpropagation in matrix form

Take an MLP with layers l=1,…,Ll=1,\dots,L: pre-activations z(l)=W(l)h(l−1)+b(l)\mathbf z^{(l)}=\mathbf W^{(l)}\mathbf h^{(l-1)}+\mathbf b^{(l)}, activations h(l)=φ(z(l))\mathbf h^{(l)}=\varphi(\mathbf z^{(l)}), input h(0)=x\mathbf h^{(0)}=\mathbf x, and a loss ℓ\ell computed from the output z(L)\mathbf z^{(L)}. The network is a computational graph, and backpropagation is the chain rule applied to it from the loss backward. Its bookkeeping device is each layer's error signal δ(l)=∂ℓ/∂z(l)\boldsymbol\delta^{(l)}=\partial\ell/\partial\mathbf z^{(l)}, a column vector with one entry per unit.

DerivationThe backpropagation equations
  1. ∂z(l+1)∂z(l)=∂z(l+1)∂h(l) ∂h(l)∂z(l)=W(l+1)diag⁡(φ′(z(l)))\frac{\partial\mathbf z^{(l+1)}}{\partial\mathbf z^{(l)}}=\frac{\partial\mathbf z^{(l+1)}}{\partial\mathbf h^{(l)}}\,\frac{\partial\mathbf h^{(l)}}{\partial\mathbf z^{(l)}}=\mathbf W^{(l+1)}\operatorname{diag}\big(\varphi'(\mathbf z^{(l)})\big)

    The Jacobian of one layer's pre-activations with respect to the previous layer's is a weight matrix times a diagonal matrix of activation slopes:

  2. δ(l)=(∂z(l+1)∂z(l)) ⁣⊤δ(l+1)=φ′(z(l))⊙(W(l+1)⊤δ(l+1))\boldsymbol\delta^{(l)}=\Big(\frac{\partial\mathbf z^{(l+1)}}{\partial\mathbf z^{(l)}}\Big)^{\!\top}\boldsymbol\delta^{(l+1)}=\varphi'(\mathbf z^{(l)})\odot\big(\mathbf W^{(l+1)\top}\boldsymbol\delta^{(l+1)}\big)

    For a scalar loss, the chain rule multiplies the gradient by the transposed Jacobian. The diagonal matrix acts elementwise:

  3. ∂ℓ∂Wij(l)=δi(l) hj(l−1)⟹∂ℓ∂W(l)=δ(l) h(l−1)⊤,∂ℓ∂b(l)=δ(l)\frac{\partial\ell}{\partial W^{(l)}_{ij}}=\delta^{(l)}_i\,h^{(l-1)}_j\quad\Longrightarrow\quad\frac{\partial\ell}{\partial\mathbf W^{(l)}}=\boldsymbol\delta^{(l)}\,\mathbf h^{(l-1)\top},\qquad\frac{\partial\ell}{\partial\mathbf b^{(l)}}=\boldsymbol\delta^{(l)}

    Each weight Wij(l)W^{(l)}_{ij} enters only zi(l)z^{(l)}_i, with coefficient hj(l−1)h^{(l-1)}_j. So the weight gradient is an outer product, and the bias gradient is the error signal itself:

  4. δ(L)=p−y\boldsymbol\delta^{(L)}=\mathbf p-\mathbf y

    The recursion starts at the output. For a sigmoid or softmax output with the cross-entropy loss it is "prediction minus target" (derived in From Neurons to Networks):

  5. Δ(l)=φ′(Z(l))⊙(Δ(l+1)W(l+1)),∂ℓ∂W(l)=Δ(l)⊤H(l−1),∂ℓ∂b(l)=Δ(l)⊤1\Delta^{(l)}=\varphi'(Z^{(l)})\odot\big(\Delta^{(l+1)}\mathbf W^{(l+1)}\big),\qquad\frac{\partial\ell}{\partial\mathbf W^{(l)}}=\Delta^{(l)\top}H^{(l-1)},\qquad\frac{\partial\ell}{\partial\mathbf b^{(l)}}=\Delta^{(l)\top}\mathbf 1

    For a mini-batch of BB examples, stack them as rows, H(l)∈RB×nlH^{(l)}\in\R^{B\times n_l} and Δ(l)=∂ℓ/∂Z(l)\Delta^{(l)}=\partial\ell/\partial Z^{(l)}, with ℓ\ell the batch mean, so that Δ(L)=(P−Y)/B\Delta^{(L)}=(P-Y)/B. Every step becomes a matrix product:

python
import numpy as np

def forward_backward(X, Y, Ws, bs):
    """X: (B, n0) inputs, Y: (B, K) one-hot labels, Ws[l]: (n_{l+1}, n_l).
    tanh hidden layers, softmax output, mean cross-entropy."""
    Hs = [X]                                  # the "tape": every layer's input
    for W, b in zip(Ws[:-1], bs[:-1]):
        Hs.append(np.tanh(Hs[-1] @ W.T + b))
    Z = Hs[-1] @ Ws[-1].T + bs[-1]
    P = np.exp(Z - Z.max(axis=1, keepdims=True))
    P /= P.sum(axis=1, keepdims=True)
    loss = -np.mean(np.sum(Y * np.log(P + 1e-12), axis=1))
    D = (P - Y) / len(X)                      # Delta at the output
    gWs, gbs = [], []
    for l in range(len(Ws) - 1, -1, -1):
        gWs.insert(0, D.T @ Hs[l])              # same shape as Ws[l]
        gbs.insert(0, D.sum(axis=0))
        if l > 0:
            D = (D @ Ws[l]) * (1 - Hs[l] ** 2)  # tanh'(z) = 1 - tanh(z)^2
    return loss, gWs, gbs

The backward loop mirrors the forward loop in reverse, reusing every stored activation. Production frameworks build this loop automatically from the forward code, by reverse-mode automatic differentiation.

Cost. The forward pass does one matrix product per layer, about BPBP multiply-adds for PP weights. The backward pass does two per layer, Δ(l)⊤H(l−1)\Delta^{(l)\top}H^{(l-1)} for the weights and Δ(l+1)W(l+1)\Delta^{(l+1)}\mathbf W^{(l+1)} for the next error signal, so the whole gradient costs about three forward passes, against the P+1P+1 passes of finite differences (the comparison in A Short History of AI). The price is memory: every H(l)H^{(l)} must be kept until the backward pass reaches it, O(B∑lnl)\order{B\sum_ln_l} numbers, often more than the weights themselves.

Check your understanding

In the widget, ∂L/∂z3=−0.197\partial L/\partial z_3=-0.197, the weight from h1h_1 to the output is v1=0.7v_1=0.7, and h1=σ(z1)=0.711h_1=\sigma(z_1)=0.711. What is ∂L/∂z1\partial L/\partial z_1?

  1. −0.138-0.138
  2. −0.140-0.140
  3. −0.028-0.028
  4. −0.197-0.197
Show the answer

C. −0.028-0.028

δ1=σ′(z1) v1 δ3=h1(1−h1)⋅0.7⋅(−0.197)≈0.2055⋅0.7⋅(−0.197)≈−0.028\delta_1=\sigma'(z_1)\,v_1\,\delta_3=h_1(1-h_1)\cdot0.7\cdot(-0.197)\approx0.2055\cdot0.7\cdot(-0.197)\approx-0.028. The error signal passes back through the weight and the activation's slope, and since σ′≤14\sigma'\le\tfrac14, every sigmoid layer shrinks it.

Vanishing and exploding gradients, and initialization

Unroll the recursion and the error signal at layer ll is a product of L−lL-l layer Jacobians:

δ(l)=D(l)W(l+1)⊤D(l+1)W(l+2)⊤⋯D(L−1)W(L)⊤δ(L),D(k)=diag⁡(φ′(z(k))).\boldsymbol\delta^{(l)}=D^{(l)}\mathbf W^{(l+1)\top}D^{(l+1)}\mathbf W^{(l+2)\top}\cdots D^{(L-1)}\mathbf W^{(L)\top}\boldsymbol\delta^{(L)},\qquad D^{(k)}=\operatorname{diag}\big(\varphi'(\mathbf z^{(k)})\big).

Long products of matrices behave like powers. If the factors typically shrink vectors, δ(l)\boldsymbol\delta^{(l)} decays geometrically toward the input (vanishing gradients); if they stretch them, it grows geometrically (exploding gradients). From Neurons to Networks bounded the sigmoid case by (∥W∥/4)L−1(\norm{\mathbf W}/4)^{L-1}. Since ∂ℓ/∂W(l)=δ(l)h(l−1)⊤\partial\ell/\partial\mathbf W^{(l)}=\boldsymbol\delta^{(l)}\mathbf h^{(l-1)\top}, early layers then learn far more slowly, or more erratically, than late ones.

The remedy for explosions is blunt: gradient clipping rescales the gradient whenever its norm exceeds a threshold cc, g←g⋅min⁡(1,c/∥g∥)\mathbf g\leftarrow\mathbf g\cdot\min(1,c/\norm{\mathbf g}) (Pascanu, Mikolov & Bengio, 2013). The remedy for both is to choose weight scales that make the typical factor close to 1, which is what a careful initialization does.

DerivationVariance propagation (Glorot & Bengio, 2010; He et al., 2015)
  1. Var⁡(zi(l))=∑j=1nl−1Var⁡(Wij(l)hj(l−1))=nl−1 vl E[(hj(l−1))2]\Var\big(z^{(l)}_i\big)=\sum_{j=1}^{n_{l-1}}\Var\big(W^{(l)}_{ij}h^{(l-1)}_j\big)=n_{l-1}\,v_l\,\E\big[(h^{(l-1)}_j)^2\big]

    Draw the weights of layer ll independently with mean 0 and variance vlv_l, independent of the layer's inputs. A pre-activation is a sum of nl−1n_{l-1} independent terms with mean 0, so its variance is

  2. forward: vl=1nl−1 (LeCun),backward: vl=1nl,compromise: vl=2nl−1+nl (Glorot)\text{forward: }v_l=\frac1{n_{l-1}}\ \text{(LeCun)},\qquad\text{backward: }v_l=\frac1{n_l},\qquad\text{compromise: }v_l=\frac2{n_{l-1}+n_l}\ \text{(Glorot)}

    For tanh near the origin h≈zh\approx z, so E[h2]≈Var⁡(z(l−1))\E[h^2]\approx\Var(z^{(l-1)}), and the forward variance is preserved when nl−1vl=1n_{l-1}v_l=1. The same computation for the backward recursion δ(l−1)=φ′⊙W(l)⊤δ(l)\boldsymbol\delta^{(l-1)}=\varphi'\odot\mathbf W^{(l)\top}\boldsymbol\delta^{(l)} sums over nln_l terms and asks for nlvl=1n_lv_l=1:

  3. Var⁡(z(l))=nl−1vl2 Var⁡(z(l−1))⟹vl=2nl−1 (He)\Var\big(z^{(l)}\big)=\frac{n_{l-1}v_l}{2}\,\Var\big(z^{(l-1)}\big)\quad\Longrightarrow\quad v_l=\frac{2}{n_{l-1}}\ \text{(He)}

    A ReLU zeroes the negative half of a symmetric zz, so E[ReLU(z)2]=12Var⁡(z)\E[\mathrm{ReLU}(z)^2]=\tfrac12\Var(z). Each layer then halves the variance unless the weights supply the missing factor 2:

  4. Wij(l)∼U[−6nl−1+nl, 6nl−1+nl ]orWij(l)∼N(0, 2nl−1)W^{(l)}_{ij}\sim U\Big[-\sqrt{\tfrac{6}{n_{l-1}+n_l}},\ \sqrt{\tfrac{6}{n_{l-1}+n_l}}\,\Big]\qquad\text{or}\qquad W^{(l)}_{ij}\sim\mathcal N\Big(0,\ \tfrac{2}{n_{l-1}}\Big)

    Glorot's uniform version draws from U[−a,a]U[-a,a] with a=6/(nl−1+nl)a=\sqrt{6/(n_{l-1}+n_l)}, because Var⁡ U[−a,a]=a2/3\Var\,U[-a,a]=a^2/3. Biases start at zero. All of this fixes the scale only at initialization; normalization layers (step 5) keep it in check during training.

Predict before you look

A deep ReLU network is initialized with variance 1/nl−11/n_{l-1}, right for tanh but missing He's factor of 2. After how many layers has the variance of the pre-activations fallen below 1/10001/1000 of its value at the first layer?

Show the answer

10

Each layer multiplies the variance by nl−1vl/2=12n_{l-1}v_l/2=\tfrac12, and 2−10=1/1024<1/10002^{-10}=1/1024<1/1000. After 50 layers the factor is 2−50≈10−152^{-50}\approx10^{-15}. He et al. (2015) reported exactly this failure: their 30-layer ReLU network trained with the factor 2 but stalled completely with the Glorot scale.

Mini-batch SGD and momentum

The training loss is an average over nn examples, L(θ)=1n∑iℓi(θ)L(\boldsymbol\theta)=\frac1n\sum_i\ell_i(\boldsymbol\theta). A mini-batch B\mathcal B drawn uniformly at random gives an unbiased estimate of its gradient, and the estimate's noise shrinks with the batch size:

g=1∣B∣∑i∈B∇ℓi(θ),E[g]=∇L(θ),Cov⁡(g)≈Σ(θ)∣B∣\mathbf g=\frac1{\abs{\mathcal B}}\sum_{i\in\mathcal B}\nabla\ell_i(\boldsymbol\theta),\qquad\E[\mathbf g]=\nabla L(\boldsymbol\theta),\qquad\Cov(\mathbf g)\approx\frac{\Sigma(\boldsymbol\theta)}{\abs{\mathcal B}}
Σ\Sigma is the covariance of the per-example gradients. Sampling without replacement multiplies the covariance by (n−∣B∣)/(n−1)(n-\abs{\mathcal B})/(n-1). Halving the noise's standard deviation costs four times the batch.
text
function SGD-MOMENTUM(θ, learning rate η, momentum β ∈ [0, 1), batch size b, epochs E):
    v ← 0
    for epoch = 1 to E:
        shuffle the training set and split it into batches of size b
        for each batch 𝓑:
            g ← (1/b) Σ_{i ∈ 𝓑} ∇ℓ_i(θ)          # one forward and one backward pass
            v ← β·v + g                          # heavy-ball velocity
            θ ← θ − η·v

One pass over the data is an epoch. With β=0\beta=0 this is plain SGD. Deep-learning libraries use this form of momentum; the variant v←βv+(1−β)g\mathbf v\leftarrow\beta\mathbf v+(1-\beta)\mathbf g is the same method with η\eta rescaled by 1−β1-\beta.

Why momentum helps: unroll vt=∑k≤tβt−kgk\mathbf v_t=\sum_{k\le t}\beta^{t-k}\mathbf g_k, an exponentially weighted sum of past gradients. Components that keep their sign add up, to g/(1−β)\mathbf g/(1-\beta) for a constant gradient, a ten times larger step at β=0.9\beta=0.9. Components that flip sign from step to step, like the zig-zag across a ravine, cancel. The ravine, the condition number and Nesterov's accelerated method are treated in The Math Toolkit of AI and Optimization & Convexity. Noise matters as well: smaller batches and larger steps make SGD noisier, and that noise is widely believed to influence which minimum training ends in (see Loss Landscapes & Initialization).

Think first You raise the batch size from 32 to 512 and keep the learning rate. By what factor does the standard deviation of the gradient noise shrink, and what does that suggest about the learning rate?

The covariance scales as 1/∣B∣1/\abs{\mathcal B}, so the standard deviation shrinks by 512/32=4\sqrt{512/32}=4. Each step now costs 16 times as much but is only 4 times less noisy, so at the same η\eta an epoch makes 16 times fewer, only slightly better steps. The linear scaling rule (Goyal et al., 2017) raises η\eta in proportion to the batch size, with a warmup, and works up to a problem-dependent critical batch size beyond which larger batches stop paying off.

Adaptive step sizes: Adam

One global learning rate fits badly when gradient scales differ across parameters, as they do between layers, or between the embedding of a common word and that of a rare one. AdaGrad (Duchi, Hazan & Singer, 2011) divides each coordinate's step by the root of its summed squared gradients. RMSProp (Tieleman & Hinton, 2012) uses an exponential moving average instead, so old gradients are forgotten. Adam (Kingma & Ba, 2015) adds momentum and corrects both averages for starting at zero.

text
function ADAM(θ, η = 10⁻³, β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸):
    m ← 0;  v ← 0                                 # first and second moment estimates
    for t = 1, 2, …:
        g ← stochastic gradient at θ
        m ← β₁·m + (1 − β₁)·g
        v ← β₂·v + (1 − β₂)·g⊙g                    # elementwise square
        m̂ ← m / (1 − β₁ᵗ);   v̂ ← v / (1 − β₂ᵗ)       # bias correction
        θ ← θ − η · m̂ / (√v̂ + ε)                   # elementwise

The defaults are Kingma and Ba's. The state is two extra numbers per parameter, which for large models is a significant share of training memory.

DerivationWhy the bias correction
  1. mt=(1−β1)∑k=1tβ1t−k gk\mathbf m_t=(1-\beta_1)\sum_{k=1}^t\beta_1^{t-k}\,\mathbf g_k

    Unroll the first-moment average from m0=0\mathbf m_0=\mathbf 0:

  2. E[mt]=(1−β1t) E[g]⟹m^t=mt1−β1t is unbiased\E[\mathbf m_t]=\big(1-\beta_1^t\big)\,\E[\mathbf g]\quad\Longrightarrow\quad\hat{\mathbf m}_t=\frac{\mathbf m_t}{1-\beta_1^t}\ \text{is unbiased}

    If the gradients have a constant mean, the weights sum to (1−β1)∑k=1tβ1t−k=1−β1t(1-\beta_1)\sum_{k=1}^t\beta_1^{t-k}=1-\beta_1^t, so mt\mathbf m_t is biased toward 0 early on:

  3. m^1v^1=g1∣g1∣=sign⁡(g1),m1v1=0.1 g10.001 ∣g1∣≈3.16 sign⁡(g1)\frac{\hat m_1}{\sqrt{\hat v_1}}=\frac{g_1}{\abs{g_1}}=\sign(g_1),\qquad\frac{m_1}{\sqrt{v_1}}=\frac{0.1\,g_1}{\sqrt{0.001}\,\abs{g_1}}\approx3.16\,\sign(g_1)

    The same holds for vt\mathbf v_t. At t=1t=1 the corrected ratio is exactly the sign of the gradient, so the first step moves every coordinate by η\eta. Without either correction it would be (1−β1)/1−β2≈3.2(1-\beta_1)/\sqrt{1-\beta_2}\approx3.2 times larger:

Two properties follow from the ratio m^/v^\hat{\mathbf m}/\sqrt{\hat{\mathbf v}}. It is scale-invariant: multiplying the loss by any c>0c>0 leaves Adam's steps unchanged (up to ε\varepsilon), while SGD's steps grow by cc. And it is roughly bounded: when a coordinate's gradient keeps its sign, its step is about η\eta whatever the gradient's size, so η\eta sets how far each weight moves per step rather than multiplying the gradient.

Check your understanding

You multiply the training loss by 100, a change of units. What happens to the parameter updates of SGD and of Adam (ignoring ε\varepsilon)?

  1. Both become 100 times larger
  2. SGD's become 100 times larger; Adam's are unchanged
  3. SGD's are unchanged; Adam's become 10 times larger
  4. Both are unchanged, since the minimizer is the same
Show the answer

B. SGD's become 100 times larger; Adam's are unchanged

SGD's step is ηg\eta\mathbf g, so it scales with the loss. Adam's is η m^/v^\eta\,\hat{\mathbf m}/\sqrt{\hat{\mathbf v}}: m^\hat{\mathbf m} grows by 100 and v^\sqrt{\hat{\mathbf v}} by 1002=100\sqrt{100^2}=100, so the ratio is unchanged. This is one reason Adam's default η=10−3\eta=10^{-3} transfers across problems far better than any fixed SGD learning rate.

Schedules, warmup and normalization layers

With a constant learning rate, SGD never settles: near a minimum, gradient noise keeps it bouncing in a region whose size grows with η\eta (Optimizers in Depth derives this). So η\eta is decayed during training. Step decay divides it by 10 at fixed epochs; cosine decay lowers it smoothly to near zero (Loshchilov & Hutter, 2017); linear decay is common for transformers. Warmup ramps η\eta up from near zero over the first hundreds or thousands of steps, because early on Adam's second-moment estimates rest on few gradients and the network can sit in a sharp region where a full-size step diverges. Warmup is standard for transformers and for large-batch training (Goyal et al., 2017).

ηt={ηmax⁡ tTw,t<Tw(linear warmup)ηmin⁡+ηmax⁡−ηmin⁡2(1+cos⁡π(t−Tw)T−Tw),Tw≤t≤T(cosine decay)\eta_t=\begin{cases}\eta_{\max}\,\dfrac{t}{T_w}, & t<T_w\quad\text{(linear warmup)}\\[8pt] \eta_{\min}+\dfrac{\eta_{\max}-\eta_{\min}}{2}\Big(1+\cos\dfrac{\pi(t-T_w)}{T-T_w}\Big), & T_w\le t\le T\quad\text{(cosine decay)}\end{cases}
A common recipe for TT steps with TwT_w warmup steps. The peak ηmax⁡\eta_{\max} is usually the single most important hyperparameter of a training run.

Batch normalization (Ioffe & Szegedy, 2015) attacks the scale problem directly. For each feature, it standardizes the pre-activations over the mini-batch, then applies a learned scale and shift:

z^i=zi−μBσB2+ϵ,BN(zi)=γ z^i+β,μB=1∣B∣∑i∈Bzi,σB2=1∣B∣∑i∈B(zi−μB)2\hat z_i=\frac{z_i-\mu_{\mathcal B}}{\sqrt{\sigma_{\mathcal B}^2+\epsilon}},\qquad\mathrm{BN}(z_i)=\gamma\,\hat z_i+\beta,\qquad\mu_{\mathcal B}=\frac1{\abs{\mathcal B}}\sum_{i\in\mathcal B}z_i,\quad\sigma_{\mathcal B}^2=\frac1{\abs{\mathcal B}}\sum_{i\in\mathcal B}(z_i-\mu_{\mathcal B})^2
γ\gamma and β\beta are learned per feature, so the layer can still produce any scale. Backpropagation runs through μB\mu_{\mathcal B} and σB\sigma_{\mathcal B} as well, which couples the examples of a batch.

At test time there may be no batch, so BN switches to running averages of μ\mu and σ2\sigma^2 collected during training, and each example's output stops depending on the others. Layer normalization (Ba, Kiros & Hinton, 2016) computes the same statistics over the features of each example instead. It behaves identically in training and testing, which suits recurrent networks and transformers.

PropertyBatch normLayer norm
Statistics overthe batch, separately per featurethe features, separately per example
Training vs testbatch statistics vs running averagesidentical
Small batchesnoisy statistics, worse resultsunaffected
Typical homeconvolutional networkstransformers and RNNs

Key takeaways

  • Backpropagation runs the chain rule backward: δ(l)=φ′(z(l))⊙W(l+1)⊤δ(l+1)\boldsymbol\delta^{(l)}=\varphi'(\mathbf z^{(l)})\odot\mathbf W^{(l+1)\top}\boldsymbol\delta^{(l+1)} and ∂ℓ/∂W(l)=δ(l)h(l−1)⊤\partial\ell/\partial\mathbf W^{(l)}=\boldsymbol\delta^{(l)}\mathbf h^{(l-1)\top}. In mini-batch form that is two matrix products per layer, about twice the forward cost, paid for in activation memory.
  • Gradients are products of layer Jacobians, so they vanish or explode geometrically with depth. LeCun (1/nin1/n_{\text{in}}), Glorot (2/(nin+nout)2/(n_{\text{in}}+n_{\text{out}})) and He (2/nin2/n_{\text{in}}, for ReLU) initializations keep the variance of signals constant across layers.
  • Mini-batch SGD follows unbiased gradient estimates whose noise falls as 1/∣B∣1/\sqrt{\abs{\mathcal B}}; momentum sums past gradients, amplifying consistent directions by 1/(1−β)1/(1-\beta) and cancelling oscillations.
  • Adam combines momentum with per-coordinate scaling and bias correction. Its steps are scale-invariant and about η\eta in size, which makes it robust to the learning rate.
  • Schedules decay η\eta to shrink SGD's noise floor and warmup protects the first steps; batch norm (over the batch, with running averages at test time) and layer norm (over features, identical at test time) keep activations at a stable scale.

Check yourself

In the interactive lesson these are scored with your confidence, and what you miss comes back for review.

Question 1

For a mini-batch with error signals Δ(l)∈RB×nl\Delta^{(l)}\in\R^{B\times n_l} (the 1/B1/B of the mean loss included) and layer inputs H(l−1)∈RB×nl−1H^{(l-1)}\in\R^{B\times n_{l-1}}, which expression is the gradient with respect to W(l)∈Rnl×nl−1\mathbf W^{(l)}\in\R^{n_l\times n_{l-1}}?

  1. H(l−1)⊤Δ(l)H^{(l-1)\top}\Delta^{(l)}
  2. Δ(l)⊤H(l−1)\Delta^{(l)\top}H^{(l-1)}
  3. Δ(l)H(l−1)⊤\Delta^{(l)}H^{(l-1)\top}
  4. Δ(l)⊤H(l)\Delta^{(l)\top}H^{(l)}
Show the answer

B. Δ(l)⊤H(l−1)\Delta^{(l)\top}H^{(l-1)}

Δ(l)⊤H(l−1)=∑bδb(l)hb(l−1)⊤\Delta^{(l)\top}H^{(l-1)}=\sum_b\boldsymbol\delta^{(l)}_b\mathbf h^{(l-1)\top}_b, the sum of the per-example outer products, with shape nl×nl−1n_l\times n_{l-1} like W(l)\mathbf W^{(l)}.

Question 2

A ReLU layer has 512 inputs and 256 outputs. Which weight variance does He initialization prescribe?

  1. 1/5121/512
  2. 2/7682/768
  3. 2/5122/512
  4. 1/2561/256
Show the answer

C. 2/5122/512

He's rule is v=2/ninv=2/n_{\text{in}}: the factor 2 compensates for ReLU zeroing half of a symmetric input, since E[ReLU(z)2]=Var⁡(z)/2\E[\mathrm{ReLU}(z)^2]=\Var(z)/2. The standard deviation is 2/512=1/16\sqrt{2/512}=1/16.

Question 3

An implementation of Adam corrects m\mathbf m for its bias but forgets to correct v\mathbf v. With β2=0.999\beta_2=0.999, what happens on the first step?

  1. Nothing noticeable: bias correction only matters after many steps
  2. The first step is about 1,000 times too small
  3. Each coordinate moves about η/1−β2≈32η\eta/\sqrt{1-\beta_2}\approx32\eta instead of η\eta
  4. The step points uphill
Show the answer

C. Each coordinate moves about η/1−β2≈32η\eta/\sqrt{1-\beta_2}\approx32\eta instead of η\eta

m^1=g1\hat m_1=g_1 but v1=(1−β2)g12=0.001 g12v_1=(1-\beta_2)g_1^2=0.001\,g_1^2, so the step is η g1/0.001 g12≈31.6 η sign⁡(g1)\eta\,g_1/\sqrt{0.001\,g_1^2}\approx31.6\,\eta\,\sign(g_1). The uncorrected v\mathbf v is tiny early on, and dividing by its root inflates the first steps: exactly the instability the correction prevents.

Question 4

SGD with heavy-ball momentum, v←βv+g\mathbf v\leftarrow\beta\mathbf v+\mathbf g and θ←θ−ηv\boldsymbol\theta\leftarrow\boldsymbol\theta-\eta\mathbf v, sees the same gradient g\mathbf g at every step. Which step does it settle into for β=0.9\beta=0.9?

  1. ηg\eta\mathbf g
  2. 0.9 ηg0.9\,\eta\mathbf g
  3. 10 ηg10\,\eta\mathbf g
  4. 0.1 ηg0.1\,\eta\mathbf g
Show the answer

C. 10 ηg10\,\eta\mathbf g

The velocity converges to the fixed point of v=βv+g\mathbf v=\beta\mathbf v+\mathbf g, which is v=g/(1−β)=10g\mathbf v=\mathbf g/(1-\beta)=10\mathbf g. Along directions where the gradient is consistent, momentum multiplies the effective learning rate by 1/(1−β)1/(1-\beta), while oscillating components cancel.

Question 5

An MLP with batch normalization after each linear layer is evaluated one example at a time, but it was accidentally left in training mode. What goes wrong?

  1. Nothing: batch norm computes the same function in both modes
  2. The running averages are updated, but predictions are unaffected
  3. The learned γ\gamma and β\beta are reset to 1 and 0
  4. Each example is normalized by its own statistics, so every normalized value is 0 and the output no longer depends on the input
Show the answer

D. Each example is normalized by its own statistics, so every normalized value is 0 and the output no longer depends on the input

With a batch of one, μB=z\mu_{\mathcal B}=z and σB2=0\sigma_{\mathcal B}^2=0, so z^=0\hat z=0 and BN(z)=β\mathrm{BN}(z)=\beta for every input. In evaluation mode BN uses the running averages from training and acts as a fixed affine map. Layer norm has no such mode switch.

Go deeper

Further reading

  • I. Goodfellow, Y. Bengio & A. Courville, Deep Learning, §6.5 and ch. 8 (MIT Press, 2016).
  • X. Glorot & Y. Bengio, "Understanding the difficulty of training deep feedforward neural networks," AISTATS (2010).
  • D. P. Kingma & J. Ba, "Adam: A method for stochastic optimization," ICLR (2015).
  • S. Ioffe & C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," ICML (2015).