CNNs, RNNs & Residual Networks
Architectures that bake in structure.
To a fully connected network, a photo is a list of unrelated numbers: shift the cat by one pixel and every input changes. The architectures that made deep learning work build in what is known about the data: images look alike everywhere, sequences unfold in time, and deep functions are small corrections to the identity. They learn from far less data and train far deeper.
What you’ll learn
- Compute a convolutional layer's output (cross-correlation with stride and padding), its output size and parameter count, and prove that convolution is translation equivariant.
- Compute the receptive fields of stacked convolutions and pooling, and trace the LeNet, AlexNet, VGG and ResNet lineage.
- Derive backpropagation through time for a recurrent network and explain why its gradients vanish or explode over long sequences.
- Write the LSTM, GRU and residual-block equations, and explain how their additive paths keep gradients alive.
Convolution: sharing weights across space
A fully connected layer from a image to a feature map of the same size with 64 channels would need about weights. A convolutional layer instead slides small kernels across the image and reuses them at every position. With input channels, output channels, kernels, stride and zero padding :
Two assumptions are built in. Locality: each output depends on a small patch. Weight sharing: the same detector is applied everywhere. Together they give the layer its defining symmetry.
Suppose the widget computed a true convolution, rotating each kernel by before sliding it. What would change?
- Nothing, for any kernel
- Every feature map would shift by one pixel
- The Sobel maps would change sign everywhere, while blur, sharpen and edge would be unchanged
- Only the blur map would change
Show the answer
C. The Sobel maps would change sign everywhere, while blur, sharpen and edge would be unchanged
Rotating Sobel-y by gives top row minus bottom row, exactly Sobel-y, and likewise for Sobel-x. Blur, sharpen and edge are symmetric under the rotation, so for them convolution and cross-correlation agree. In a network the kernels are learned, so it would simply learn the rotated kernels.
Pooling, receptive fields and the classic CNNs
Pooling summarizes each window, usually by its maximum over blocks with stride 2. It halves the resolution, has no parameters, and makes the output insensitive to small shifts within a block. A unit's receptive field is the input region it depends on. Track its width and the jump , the input distance between adjacent units, through layers with kernel size and stride :
Three convolutions with stride 1, then max pooling with stride 2, then two more convolutions. How wide is the receptive field of one final unit, in input pixels?
Show the answer
16
The convolutions give ; pooling gives and doubles the jump to ; each later layer adds , so . Downsampling is what lets receptive fields cover a whole object after a few layers.
The VGG networks (Simonyan & Zisserman, 2015) used only kernels, because a stack of small kernels is cheaper and more expressive than one large kernel with the same field. With channels in and out, two layers see with weights instead of , three see with instead of , and every extra layer adds a nonlinearity.
| Network | Year | Depth and size | What it introduced or showed |
|---|---|---|---|
| LeNet-5 | 1998 | 7 layers, about 60,000 parameters | convolution, subsampling and end-to-end gradient training for reading digits and documents (LeCun et al.) |
| AlexNet | 2012 | 8 learned layers, about 60 million parameters | ReLU, dropout, data augmentation and two GPUs; won ImageNet with 15.3% top-5 error against 26.2% (Krizhevsky, Sutskever & Hinton) |
| VGG-16 | 2014 | 16 weight layers, about 138 million parameters | depth from uniform stacks (Simonyan & Zisserman) |
| ResNet-152 | 2015 | 152 layers, about 60 million parameters | residual connections; won ImageNet 2015 with 3.57% top-5 error (He et al.) |
Recurrent networks and backpropagation through time
A recurrent network reads a sequence one element at a time and carries a hidden state forward, using the same weights at every step: weight sharing across time instead of space.
Backpropagate through the unrolled graph. The state affects the loss through its own output and through . With , the slope of tanh:
A shared weight receives the sum of its contributions at every step:
Unrolling the recursion, the influence of on is a product of Jacobians. Since , its norm is at most a power of :
If , the gradient that a loss at time sends to step shrinks exponentially in , so long-range dependencies cannot be learned from it (Hochreiter, 1991; Bengio, Simard & Frasconi, 1994). If the largest singular value exceeds 1, the product can instead grow exponentially, and occasional huge gradients wreck training:
Exploding gradients are tamed by gradient clipping. Memory and computation grow with , so long sequences use truncated BPTT, backpropagating only steps into the past; that caps the cost, but also the span of dependencies the gradient can teach. Vanishing gradients need a change of architecture.
With , after how many steps back in time does the bound on a hidden state's influence first fall below 1%?
Show the answer
44
needs , so 44 steps. A dependency 100 steps back is attenuated by up to : its gradient signal is buried under those of recent steps.
Gated memory: LSTM and GRU
The long short-term memory (Hochreiter & Schmidhuber, 1997) adds a cell state that is updated mostly by addition, with sigmoid gates deciding what to forget, what to write and what to reveal. With stacked into one vector:
Along the cell path, , plus terms through the gates. Error flowing back steps along this path is multiplied by a product of forget gates, which stays near 1 whenever they are open: there is no repeated multiplication by and no squashing derivative. Hochreiter and Schmidhuber called the original gate-free path the constant error carousel. Initializing the forget-gate bias near 1 keeps it running early in training (Gers et al., 2000; Jozefowicz, Zaremba & Sutskever, 2015).
The GRU (Cho et al., 2014) merges cell and hidden state and uses two gates, an update gate and a reset gate :
| Cell | Gates | Parameters (input size , state size ) | Memory path |
|---|---|---|---|
| Vanilla RNN | none | multiplied by at every step | |
| LSTM | forget, input, output | cell , scaled by | |
| GRU | update, reset | state , scaled by |
Residual connections and very deep networks
Depth should not hurt: a deeper network can copy a shallower one and set its extra layers to the identity. Yet He et al. (2016) found that a plain 56-layer CNN had higher training error than a 20-layer one on CIFAR-10. That is not overfitting; the optimizer fails to find the deeper network's better solutions, the degradation problem. Their fix, the residual connection, makes the identity the default:
In the pre-activation form, where nothing follows the addition, unroll the recursion from block to the last block :
Differentiate: the gradient at block is the gradient at the top plus correction terms.
The identity term carries to every block unchanged, so the gradient cannot vanish through a product of small factors, as it can in a plain network, where it is a product of layer Jacobians . Expanding the residual product writes it as a sum over paths of every length:
With residual connections He et al. trained 152-layer networks on ImageNet and networks of more than 1,000 layers on CIFAR-10. Residual connections, together with batch or layer normalization (Backpropagation & Optimization), are now part of nearly every deep architecture: transformers are stacks of residual attention and feed-forward blocks. The LSTM's cell update is a gated residual connection through time, a link that highway networks made explicit across depth (Srivastava, Greff & Schmidhuber, 2015).
Key takeaways
- A convolutional layer applies shared kernels at every position (cross-correlation, in practice): parameters whatever the image size, output size , and exact translation equivariance away from borders and strides.
- Receptive fields grow by per layer, with strides and pooling multiplying the jump; stacks of kernels (VGG) see as far as large kernels with fewer weights.
- An RNN shares weights across time. BPTT sums gradients over steps, and the products of Jacobians make long-range gradients vanish or explode.
- LSTMs and GRUs add gated, additive memory paths whose Jacobians are diagonal gates near 1; residual connections do the same across depth, adding an identity term to every gradient.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
A convolutional layer maps 64 input channels to 128 output channels with kernels and biases. How many parameters does it have, and how does that change if the input image doubles in width and height?
- , unchanged when the image grows
- , four times as many when the image grows
- , unchanged
- , unchanged
Show the answer
A. , unchanged when the image grows
. Weight sharing makes the count independent of the image size; only the computation and the activation memory grow, here fourfold.
A input passes through a convolution with stride 2 and padding 2. What is the spatial size of the output?
Show the answer
B.
. Padding would preserve the size at stride 1; stride 2 halves it.
Why do vanilla RNNs struggle to learn dependencies that span hundreds of time steps?
- They have too few parameters to store long sequences
- Backpropagation through time is not exact for recurrent networks
- The gradient linking distant steps is a product of many Jacobians , which typically shrinks (or grows) exponentially with the time gap
- The tanh activation cannot represent long-range functions
Show the answer
C. The gradient linking distant steps is a product of many Jacobians , which typically shrinks (or grows) exponentially with the time gap
, bounded by since . With the contributions of distant steps vanish exponentially; with large singular values they can explode. Gating (LSTM, GRU) adds a path along which the factors stay near 1.
In an LSTM, what is along the cell-state path, ignoring the paths through the gates?
Show the answer
B.
, so the direct dependence is elementwise multiplication by the forget gate. Over steps the error is scaled by a product of forget gates, which stays near 1 when they are open, with no weight matrix and no squashing derivative in the product.
A plain 56-layer CNN reaches a higher training error than a plain 20-layer CNN on the same data. What is the most likely explanation?
- The deeper network overfits
- The deeper network cannot represent the shallower one's function
- The training set is too small for 56 layers
- Optimization: gradient methods fail to find the deeper network's good solutions, which residual connections fix by making identity blocks easy
Show the answer
D. Optimization: gradient methods fail to find the deeper network's good solutions, which residual connections fix by making identity blocks easy
Overfitting would raise test error, not training error. The deep network can represent everything the shallow one can, with the extra layers set to the identity, so the gap is an optimization failure, the degradation problem. With blocks, He et al. trained deeper networks to lower training and test error.
Go deeper
- Convolutional Networks in Depth · Weight sharing, receptive fields and equivariance.
- Recurrent Networks & LSTMs · Memory over time and the vanishing gradient problem.
- Normalization & Residual Connections · BatchNorm, LayerNorm and very deep networks.
Further reading
- Y. LeCun, L. Bottou, Y. Bengio & P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE 86(11) (1998).
- S. Hochreiter & J. Schmidhuber, "Long short-term memory," Neural Computation 9(8) (1997).
- K. He, X. Zhang, S. Ren & J. Sun, "Deep residual learning for image recognition," CVPR (2016).
- I. Goodfellow, Y. Bengio & A. Courville, Deep Learning, chs. 9–10 (MIT Press, 2016).