The Scaling Era & the Bitter Lesson
Compute, data and why general methods keep winning.
In 2019 Rich Sutton summed up seventy years of AI in one uncomfortable sentence: general methods that leverage computation beat methods that leverage human knowledge, and by a large margin. This lesson weighs the evidence (compute trends, the track record from chess to language, scaling laws) against a theorem that says no method can win everywhere.
What you’ll learn
- Quantify the growth of training compute across three eras, and estimate the compute of a training run with .
- State the bitter lesson precisely, with its historical cases and its strongest counterarguments.
- Prove the no-free-lunch theorem for off-training-set error, and explain what it implies for the bitter lesson.
- Derive compute-optimal model and data sizes from a parametric scaling law, and contrast Kaplan et al. (2020) with Hoffmann et al. (2022).
Three eras of compute
Sevilla et al. (2022) compiled the training compute of over a hundred milestone systems and found three eras. Before about 2010, compute doubled roughly every 20 months, in line with Moore's law. From 2010 to 2022, the deep learning era, it doubled about every 6 months. And from late 2015 a large-scale trend appeared, of systems trained with 10 to 100 times more compute than the main trend.
Hardware price-performance improved far more slowly than total compute, so most of the growth came from spending: more chips, for longer.
In a dense transformer with parameters, each parameter takes part in about one multiply-add (2 floating-point operations) per token in the forward pass:
The backward pass computes gradients with respect to both activations and weights, about twice the forward cost:
Training on tokens therefore costs about
For GPT-3 (2020), parameters and tokens, which matches the roughly FLOPs its authors reported.
Chinchilla (2022) had parameters and was trained on tokens. Using , what was its training compute, in units of FLOPs?
Show the answer
5.9
: almost twice GPT-3's compute, spent on a model 2.5 times smaller trained on nearly five times as many tokens.
The bitter lesson
| Domain | Knowledge-based approach | General method that won |
|---|---|---|
| Chess | encoded chess expertise and selective search | massive alpha–beta search on custom hardware (Deep Blue, 1997); later, evaluation learned by self-play (AlphaZero, 2017) |
| Go | hand-crafted patterns and proverbs | Monte Carlo tree search (from 2006), then deep networks trained by reinforcement learning plus search (AlphaGo, 2016; AlphaGo Zero, 2017, with no human games) |
| Speech | phonetic and linguistic rules | statistical HMMs (1970s–80s), then deep networks trained end to end (2010s) |
| Vision | hand-designed features (edges, SIFT, HOG) | convolutional networks learning their features from data (AlexNet, 2012) |
| Language | grammars and hand-built lexicons | statistical translation (1990s), then transformers pretrained on web-scale text (from 2017) |
The strongest objections deserve equal space:
- Inductive biases are knowledge too. Convolution builds in translation equivariance, attention and residual connections are human designs, and alpha–beta and MCTS are clever algorithms. The real question is which knowledge to build in: general structure rather than domain specifics. Rodney Brooks made this point within days, in a reply titled "A Better Lesson".
- Data is finite. Projections suggest that the stock of public human-written text could be used up for training somewhere between the mid-2020s and the early 2030s, if current trends continue.
- Efficiency matters. Children acquire language from at most a few hundred million words, while language models train on trillions of tokens, and compute has financial and energy costs.
- Some domains have little data, and there priors and structure still decide.
No free lunch
If general methods keep winning, is there a universally best learner? A theorem says no, and it sharpens what the bitter lesson can mean.
Fix the training inputs and an input outside them. Under the uniform distribution over , the labels at different inputs are independent and uniform, so is independent of everything in :
depends only on , so whatever it predicts is right with probability :
Average over the unseen inputs and over training sets. The learner never entered the answer.
The PAC setting has its own version of the no-free-lunch theorem: for any learner and any training size , some distribution makes a constant classifier perfect while the learner errs at least of the time with probability at least (Shalev-Shwartz and Ben-David, 2014).
So the bitter lesson cannot be a theorem about all problems. It is an empirical claim about the problems we care about: they are overwhelmingly structured (smooth, compositional, local, symmetric, with heavy-tailed statistics), and flexible learners with the right mild biases, given enough data and compute, find that structure more reliably than people can write it down.
What does the no-free-lunch theorem imply about deep learning's success?
- That the success must be an illusion caused by flawed benchmarks
- That neural networks are universal learners that beat all others on every problem
- That the advantage must come from assumptions, such as inductive biases, that match the structure of the problems that actually occur
- Nothing, because the theorem applies only to finite input spaces
Show the answer
C. That the advantage must come from assumptions, such as inductive biases, that match the structure of the problems that actually occur
Averaged over all targets, no learner beats chance off the training set. Real tasks are a tiny, highly structured subset, and a learner wins exactly when its biases fit that subset.
Scaling laws and compute-optimal training
Scaling laws turned the bitter lesson into engineering. Hestness et al. (2017) found power laws relating error to data size across several domains. Kaplan et al. (2020) found that the test loss of transformer language models falls as a power law in parameters, data and compute over many orders of magnitude, for example in the number of non-embedding parameters, and concluded that extra compute should go mostly into bigger models, . Hoffmann et al. (2022) fit a joint law and reached a different conclusion.
Minimize subject to the budget . Substitute :
Set the derivative to zero. At the optimum, the two penalty terms, weighted by their exponents, balance:
Solving gives power laws in compute whose exponents sum to 1:
So parameters and data should grow roughly equally, about 20 tokens per parameter at the scales studied. Chinchilla (70 billion parameters, 1.4 trillion tokens) beat the 280-billion-parameter Gopher, trained on 300 billion tokens with about the same compute. Later work traced much of the disagreement with Kaplan et al. to details such as learning-rate schedules, how parameters are counted and the scale of the experiments.
Key takeaways
- Training compute doubled about every 20 months before 2010 and about every 6 months from 2010 to 2022, mostly through spending. A dense transformer costs to train; GPT-3 took about FLOPs.
- Sutton's bitter lesson: search and learning that scale with compute have repeatedly overtaken built-in knowledge, in chess, Go, speech, vision and language. The counterarguments: inductive biases are knowledge too, data is finite, and efficiency matters.
- No free lunch: averaged over all targets, every learner has off-training-set error . The bitter lesson is therefore an empirical claim about the structured problems we actually face, not a theorem.
- With and , the optimum balances , giving (Chinchilla), rather than putting most compute into parameters (Kaplan et al.).
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
About how many floating-point operations does it take to train a dense transformer with parameters on tokens?
Show the answer
C.
About per token in the forward pass and in the backward pass, so in total.
Under Chinchilla-style compute-optimal scaling, the compute budget grows 100-fold. Roughly how should the model size change?
- About 10 times larger, trained on about 10 times more data
- About 100 times larger, trained on the same data
- About 29 times larger, trained on about 3.5 times more data
- Unchanged, trained on 100 times more data
Show the answer
A. About 10 times larger, trained on about 10 times more data
With exponents near each, for both parameters and tokens. The third option is Kaplan et al.'s earlier prescription, .
What does the no-free-lunch theorem say?
- Deep learning cannot generalize
- Averaged uniformly over all target functions, every learner has the same off-training-set error
- Larger models always generalize better
- Learning is impossible without infinite data
Show the answer
B. Averaged uniformly over all target functions, every learner has the same off-training-set error
Under a uniform prior over targets, unseen labels are independent of the training data, so no learner beats chance on them. Any real advantage comes from assumptions that match the problems that actually occur.
Which is the strongest form of the objection that Sutton's lesson undervalues human knowledge?
- Search and learning never worked well in practice
- Hand-coded expert systems still beat neural networks at most tasks
- Computation has stopped getting cheaper
- Successful general methods themselves embed human-designed inductive biases (convolution, attention, search algorithms), so the real question is which knowledge to build in
Show the answer
D. Successful general methods themselves embed human-designed inductive biases (convolution, attention, search algorithms), so the real question is which knowledge to build in
The objection accepts the track record but reframes it: what lost was domain-specific knowledge, and what won was general structure, also designed by people.
Further reading
- R. S. Sutton, "The Bitter Lesson" (essay, 2019).
- J. Sevilla et al., "Compute trends across three eras of machine learning," arXiv:2202.05924 (2022).
- J. Kaplan et al., "Scaling laws for neural language models," arXiv:2001.08361 (2020).
- J. Hoffmann et al., "Training compute-optimal large language models," NeurIPS (2022).
- S. Shalev-Shwartz & S. Ben-David, Understanding Machine Learning: From Theory to Algorithms (Cambridge University Press, 2014), chapter 5.