Neural Networks & Deep Learning
Layers of simple units that learn rich representations.
An artificial neuron does almost nothing: it multiplies, adds and checks a threshold. Wire thousands of them in layers, let data tune the connections, and the same humble unit learns to read handwriting, recognize faces and translate languages.
What you’ll learn
- Explain what a single artificial neuron computes and why its decision boundary is a straight line.
- Show why XOR defeats one perceptron and how a hidden layer fixes it.
- Describe how gradient descent and backpropagation train a network's weights.
- Name the main architectures (CNNs, RNNs, residual networks) and the idea of learned representations.
One neuron, one line
What is the simplest machine that can learn? In 1958 Frank Rosenblatt proposed the perceptron, an artificial neuron that multiplies each input by a weight, adds a bias , and fires (outputs 1) if the total is positive:
The weights tilt the line and the bias slides it. Rosenblatt also gave a learning rule: whenever a point lands on the wrong side, nudge the weights to fix it, . If any line separates the two classes, this rule is guaranteed to find one after finitely many updates.
The XOR wall
Now try a function a child could state: XOR ("exclusive or") outputs 1 when exactly one input is 1. Its four points sit at the corners of a square, the 1s on one diagonal and the 0s on the other. Can you place a line with both 1s on one side and both 0s on the other?
Why can no single perceptron compute XOR?
- The learning rule is too slow; with enough epochs it would get there
- No straight line puts XOR's two 1s on one side and its two 0s on the other
- It would need negative weights, and perceptron weights must be positive
- XOR needs more than two inputs
Show the answer
B. No straight line puts XOR's two 1s on one side and its two 0s on the other
A perceptron's boundary is always a line, and XOR's classes are not linearly separable. The convergence guarantee only applies when a separating line exists, so here the rule cycles forever.
Hidden layers bend the boundary
The fix is to stack neurons. XOR is "OR, but not AND", so one neuron can compute OR, a second NAND, and an output neuron can AND their answers. You just built all three gates with single lines. Networks like this, where each layer's outputs feed the next, are multilayer perceptrons; the layers between input and output are hidden.
To make them trainable, the hard threshold is replaced by a smooth activation function such as tanh or the ReLU . Each hidden neuron still draws one soft line; the next layer combines the lines into curves.
You build a 10-layer network but forget the activation functions, so every layer only computes weighted sums. What can it represent?
- Anything a deep network with activations can; it just trains more slowly
- Smooth curved boundaries, but no sharp corners
- Only what a single linear layer can: its boundary is still one straight line
- Nothing: without activations every output is zero
Show the answer
C. Only what a single linear layer can: its boundary is still one straight line
A weighted sum of weighted sums is another weighted sum: . Ten linear layers multiply out to one matrix. The nonlinearity between layers is what lets depth bend the boundary.
Learning means rolling downhill
Who sets millions of weights? Nobody, by hand. Training defines a loss, one number measuring how wrong the network is on the training data, then repeatedly nudges every weight in the direction that lowers it. That direction is minus the gradient, and the procedure is gradient descent:
where is the learning rate, the step size.
Backpropagation, popularized for neural networks by Rumelhart, Hinton and Williams in 1986, gets the gradient for every weight in one backward sweep by applying the chain rule from the loss back toward the inputs. It costs only a small multiple of one forward pass, however many weights there are.
In practice each gradient is estimated on a small random mini-batch of examples (stochastic gradient descent). Plain steps zig-zag in narrow ravines of the loss, so momentum averages past gradients to cancel the zig-zags, and Adam also gives each weight its own step size.
Gradient descent on the loss starts at with learning rate . The slope there is . Where is after one step?
Show the answer
1.5
. Each step multiplies by , so it halves and slides toward the minimum at . With the factor would be : every step would overshoot and the error would grow without bound.
Think first Why not skip gradients and just try random changes to the weights, keeping the ones that lower the loss?
With a million weights, a random change is almost never usefully downhill, and each trial reveals only one number. One backward pass gives a downhill direction for all weights at once, which is why gradient-based training scales and random search does not.
Architectures and learned features
A plain network connects every input to every neuron of the next layer. The most successful designs instead bake in structure that suits the data:
- A convolutional network slides one small filter across an image, so an edge detector learned in one corner works everywhere.
- A recurrent network reads a sequence step by step, carrying a memory forward; LSTMs add gates that make long memories trainable.
- Residual connections let each block add a correction to its input, , which made networks with over a hundred layers trainable (ResNet, 2016).
- Transformers, built on attention, now dominate language; they get a topic of their own.
What do the hidden layers learn? In trained image networks, early layers respond to edges and color blobs, middle layers to textures and parts, later layers to object-like patterns such as faces or wheels. Nobody designed those features: they are learned representations.
In the Hidden Layer you'll open up each piece: From Neurons to Networks (perceptrons, multilayer networks and activation functions), Backpropagation & Optimization (the chain rule at scale, SGD, momentum and Adam), CNNs, RNNs & Residual Networks, and Representation Learning (embeddings, autoencoders and self-supervision). The Deep Layer goes further still: perceptron convergence proofs, universal approximation and why depth helps.
Key takeaways
- A neuron computes a weighted sum plus a bias and applies an activation; on its own it draws one straight boundary.
- XOR is not linearly separable, so one perceptron fails; a hidden layer that combines several lines succeeds.
- Nonlinear activations are essential: without them, any stack of layers collapses to one linear map.
- Training minimizes a loss by gradient descent, and backpropagation gets every weight's gradient in one backward pass via the chain rule.
- Architectures such as CNNs, RNNs, residual networks and transformers bake in structure, and their hidden layers learn their own features.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
A perceptron has weights , and bias . On inputs from , which inputs make it fire?
- and only: it computes XOR
- Every input except : it computes OR
- Only : it computes AND
- None: a negative bias means it never fires
Show the answer
C. Only : it computes AND
, but an input with a single 1 gives . Only fires, so this is AND. The bias sets how much evidence the neuron needs.
What does backpropagation compute?
- The gradient of the loss with respect to every weight, in one backward pass using the chain rule
- The best possible weights, by searching all combinations
- The network's prediction for a new input
- A random change to each weight, kept only if the loss goes down
Show the answer
A. The gradient of the loss with respect to every weight, in one backward pass using the chain rule
Backprop is bookkeeping for the chain rule: it reuses each layer's upstream gradient to get every weight's derivative cheaply. An optimizer such as SGD or Adam then uses those gradients to update the weights.
Your training loss bounces around and then shoots up to huge values within a few steps. What is the most likely culprit?
- The network has too few layers
- The learning rate is too small
- The training set is too large
- The learning rate is too large, so each step overshoots the valley
Show the answer
D. The learning rate is too large, so each step overshoots the valley
A step that is too big jumps past the minimum to a steeper spot, and the next step overshoots even more: divergence. Lower the learning rate, or use an optimizer that adapts its step sizes.
A convolutional layer applies the same small filter at every position of an image. What does this weight sharing buy?
- It makes the layer fully connected
- Far fewer parameters, and a feature learned in one place is detected everywhere
- It guarantees the network cannot overfit
- It removes the need for training
Show the answer
B. Far fewer parameters, and a feature learned in one place is detected everywhere
A filter has 9 weights whether the image is 28 or 2,000 pixels wide. Sharing them means an edge detector learned in one corner works in every corner, and the network needs far less data to learn it.
Go deeper
- From Neurons to Networks · Perceptrons, multilayer networks and activation functions.
- Backpropagation & Optimization · The chain rule at scale, SGD, momentum and Adam.
- CNNs, RNNs & Residual Networks · Architectures that bake in structure.
- Representation Learning · Embeddings, autoencoders and self-supervision.
Further reading
- I. Goodfellow, Y. Bengio & A. Courville, Deep Learning, ch. 6 "Deep Feedforward Networks" (MIT Press, 2016), free at deeplearningbook.org.
- D. E. Rumelhart, G. E. Hinton & R. J. Williams, "Learning representations by back-propagating errors", Nature (1986).