Normalization & Residual Connections
BatchNorm, LayerNorm and very deep networks.
A residual network is at once a sum over exponentially many paths, a discretized differential equation, and a stack of normalized blocks whose gradients would otherwise shatter into noise. Each view explains something the others miss: why deleting a block barely hurts, why depth can be traded for an ODE solver, and why modern architectures pair skip connections with normalization.
What you’ll learn
- Derive the gradient of a pre-activation residual network and explain why the shortcut must be an exact identity (He et al., 2016).
- Interpret residual networks as ensembles of paths (Veit et al., 2016) and as Euler discretizations of ODEs, and derive the adjoint method of neural ODEs (Chen et al., 2018).
- Define batch, layer, group and RMS normalization, and prove that normalization makes the loss scale-invariant in the preceding weights, with the consequences for effective learning rates.
- Analyze signal propagation in residual networks, the role of branch scaling and normalization, and the shattered-gradients problem (Balduzzi et al., 2017).
Identity mappings
The first ResNet applied a ReLU after each residual addition, . Its successor (He, Zhang, Ren & Sun, 2016, "Identity mappings in deep residual networks") moved batch normalization and ReLU inside the branch, the pre-activation block, so that the shortcut is an exact identity: with .
Unrolled, each block's output is the input of block plus a sum of residuals, and the network's output contains its input unchanged:
The gradient at any block is the top gradient plus terms through the branches; it can vanish only if the branch terms cancel the identity exactly:
Now scale the shortcut, . The identity term becomes a power of , which vanishes or explodes with depth unless . Gating the shortcut, as highway networks do, has the same effect wherever the gates are not fully open:
The experiments agreed. On CIFAR-10, a 110-layer network whose shortcuts were scaled by 0.5, or gated, trained markedly worse than one with clean identities, and the pre-activation design trained a 1,001-layer network to a lower test error than the original block. The lesson, keep the path from the loss to every layer free of multiplications and nonlinearities, carried over to transformers: the Pre-LN block trains more stably than the original Post-LN block , which puts a normalization on the main path (Xiong et al., 2020).
Two lenses: ensembles and ODEs
Residual networks as ensembles (Veit, Wilber & Belongie, 2016). Expanding over blocks gives paths, one for each subset of blocks the signal passes through, and of them have length , so path lengths are binomially distributed around . Three experiments support reading the network as an ensemble of these paths. Deleting a single block at test time barely changes a ResNet's error but wrecks a VGG network; deleting several blocks degrades the error smoothly; and the gradient is carried mostly by short paths, which in a 110-layer ResNet span only 10 to 34 layers.
Residual networks as ODEs (E, 2017; Haber & Ruthotto, 2017; Chen et al., 2018). Write a block as : one forward-Euler step, of size , of the ODE . Neural ODEs take the limit literally. The layer stack becomes an ODE solver and depth becomes integration time, and the gradient comes from integrating an adjoint equation backward, with memory independent of the number of solver steps.
Let solve on , with loss , and define the adjoint . Since , the chain rule over one short step backward gives:
The parameters collect contributions from every instant, just as shared weights in backpropagation through time collect them from every step:
Solve for forward; then integrate , and the parameter gradient backward together from , recomputing instead of storing it. Memory is constant in the number of steps. The price is extra computation and possible numerical drift in the reconstructed , since the dynamics need not be stable in reverse:
The ODE view imports ideas from numerical analysis: stable discretizations, reversible and symplectic blocks, adaptive depth. Whether trained residual networks actually approximate a continuous limit depends on how their branches are scaled and how the weights vary from layer to layer, and is the subject of ongoing work. In practice, neural ODEs found their niche in continuous-time and physical modeling and in continuous normalizing flows, rather than as replacements for discrete deep networks.
Normalization and the scale invariance it creates
| Method | Statistics computed over | Typical use |
|---|---|---|
| BatchNorm (Ioffe & Szegedy, 2015) | the batch and spatial positions, per channel | CNNs trained with large batches |
| LayerNorm (Ba, Kiros & Hinton, 2016) | all features of one example | transformers and RNNs |
| GroupNorm (Wu & He, 2018) | groups of channels, per example | detection and segmentation with small batches |
| RMSNorm (Zhang & Sennrich, 2019) | the features of one example, without centering: | large language models |
Whatever their statistics, these layers share one algebraic property. If a linear layer with weights is followed by normalization, scaling by scales the pre-activations and their standard deviation alike, so the normalized output, and hence the loss, is unchanged.
Differentiate in at (Euler's theorem for functions homogeneous of degree 0), and in for fixed :
The gradient is orthogonal to , so each update adds an orthogonal vector, and by Pythagoras the norm grows:
Only the direction matters to the loss. By (ii) the gradient at is , and a displacement of that size turns the direction by times as much again:
So with normalization the raw learning rate is not the real one. Without weight decay, grows and the effective step shrinks on its own, which makes training robust to the choice of (Arora, Li & Lyu, 2019). Weight decay shrinks , the two forces balance at an equilibrium norm, and the decay then acts mainly by keeping the effective learning rate up rather than by regularizing the function (van Laarhoven, 2017; Zhang et al., 2019). BN has further costs: its batch dependence complicates small-batch, distributed and sequence training, and its running statistics can mismatch the test distribution. Why BN helps at all is still debated; Santurkar et al. (2018) argued that it smooths the loss landscape rather than reducing internal covariate shift.
A weight vector feeds a batch-normalized layer, so the loss is invariant to rescaling it. You train with plain gradient descent and no weight decay. What happens to ?
- It shrinks toward zero
- It stays exactly constant
- It grows at every step, by
- It oscillates
Show the answer
C. It grows at every step, by
The gradient of a scale-invariant loss is orthogonal to , so each step adds an orthogonal vector and the norm grows by Pythagoras. The effective learning rate on the direction, , therefore decays by itself; weight decay counteracts the growth.
Signal propagation in very deep residual networks
Residual connections fix the backward flow but create a forward problem: when a branch's output is uncorrelated with its input, as at initialization, variances add.
With , and each branch adding a fraction of the stream's variance:
Scaling each branch's output by keeps the total bounded, since . Fixup (Zhang, Dauphin & Ma, 2019), SkipInit (De & Smith, 2020) and ReZero (Bachlechner et al., 2021) go further, initializing the last layer of each branch, or a scalar gate on it, at zero, so that every block starts as the identity:
Normalization inside the branch does this automatically. With BN in the branch, stays near 1 while the stream's variance grows linearly, so block adds only about of the stream's variance. Deep blocks start near the identity, and the network is effectively shallow at initialization (De & Smith, 2020):
A residual network has 64 blocks. Unscaled, each branch adds half of the stream's variance, so the variance grows by . If each branch is scaled by , so that it adds of the stream's variance, by what factor does the variance grow over all 64 blocks?
Show the answer
1.65
, essentially with . The same 64 blocks grow the variance by a factor of about 1.6 instead of : branch scaling turns exponential growth in depth into a constant.
Shattered gradients (Balduzzi et al., 2017) are the corresponding problem for gradients. In a deep feedforward ReLU network, the gradients with respect to the input at two nearby inputs decorrelate as depth grows, their correlation decaying exponentially, like : the gradient comes to resemble white noise and carries little local information. In residual networks with batch normalization the correlation decays only sublinearly, like , which Balduzzi et al. proposed as a deeper reason skip connections help; their "looks linear" initialization lets very deep plain networks train too.
Key takeaways
- With exact identity shortcuts (pre-activation blocks), : the loss reaches every block directly, while scaling or gating the shortcut reintroduces products that vanish or explode.
- A residual network is an ensemble of paths of binomially distributed lengths, its gradient carried mostly by short ones, and also a forward-Euler discretization of an ODE; neural ODEs train such flows with the adjoint method in constant memory.
- Batch, layer, group and RMS normalization make the loss scale-invariant in the preceding weights: the gradient is orthogonal to , the norm grows under gradient descent, and the effective learning rate is , which weight decay then controls.
- Residual variances compound as unless branches are scaled (Fixup, SkipInit, ReZero) or normalized; normalized residual networks start effectively shallow and avoid the exponentially shattered gradients of deep plain networks.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
In with and 100 blocks, how large is the identity-path term in , relative to ?
- , as in a standard ResNet
Show the answer
B.
Unrolling multiplies the shortcut factors, , so the identity path contributes and the gradient must flow through the branches, as in a plain network. He et al. found that such scaled shortcuts trained much worse: the identity, , is special.
Veit et al. (2016) deleted single residual blocks from a trained ResNet at test time. What did they observe, and how did they interpret it?
- The error exploded, showing that every block is essential
- The error dropped, showing that the blocks were overfitting
- The output became constant
- The error barely changed, unlike in a VGG network, consistent with the network behaving like an ensemble of many paths of different lengths
Show the answer
D. The error barely changed, unlike in a VGG network, consistent with the network behaving like an ensemble of many paths of different lengths
With blocks there are paths, and deleting one block removes only the half of them that pass through it, leaving many short paths intact; the error degrades smoothly as more blocks go. In a VGG network every input travels the single path through all layers, so deleting any layer is catastrophic.
What does the adjoint method buy neural ODEs, and at what cost?
- Exact gradients at no extra cost
- Memory independent of the number of solver steps, paid for by integrating the state backward again, with extra computation and possible numerical error
- Gradients without any Jacobian–vector products
- It makes the ODE linear
Show the answer
B. Memory independent of the number of solver steps, paid for by integrating the state backward again, with extra computation and possible numerical error
Integrating backward together with means no intermediate states are stored. Each backward step needs a vector–Jacobian product of , the state is recomputed by backward integration, and if the dynamics are unstable in reverse, the reconstructed trajectory, and hence the gradient, can be inaccurate.
Why does batch normalization inside the residual branches make a deep ResNet behave like a shallow one at initialization (De & Smith, 2020)?
- Because BN sets every branch output to zero
- Because BN removes the shortcut connections
- Because the branch output's variance stays near 1 while the stream's variance grows linearly with depth, so block contributes only about of it
- Because BN makes the weight matrices orthogonal
Show the answer
C. Because the branch output's variance stays near 1 while the stream's variance grows linearly with depth, so block contributes only about of it
Each branch adds roughly unit variance, so after blocks the stream has variance about and the next branch is a small relative perturbation. Deep blocks start close to the identity, which is what Fixup, SkipInit and ReZero arrange explicitly by initializing the branches at or near zero.
Further reading
- K. He, X. Zhang, S. Ren & J. Sun, "Identity mappings in deep residual networks," ECCV (2016).
- A. Veit, M. Wilber & S. Belongie, "Residual networks behave like ensembles of relatively shallow networks," NeurIPS (2016).
- R. T. Q. Chen, Y. Rubanova, J. Bettencourt & D. Duvenaud, "Neural ordinary differential equations," NeurIPS (2018).
- D. Balduzzi, M. Frean, L. Leary, J. P. Lewis, K. W.-D. Ma & B. McWilliams, "The shattered gradients problem: If resnets are the answer, then what is the question?," ICML (2017).
- S. J. D. Prince, Understanding Deep Learning (MIT Press, 2023).