Simple Algorithm
Artificial Intelligence · Deep Layer · 26 min

The Scaling Era & the Bitter Lesson

Compute, data and why general methods keep winning.

In 2019 Rich Sutton summed up seventy years of AI in one uncomfortable sentence: general methods that leverage computation beat methods that leverage human knowledge, and by a large margin. This lesson weighs the evidence (compute trends, the track record from chess to language, scaling laws) against a theorem that says no method can win everywhere.

What you’ll learn

  • Quantify the growth of training compute across three eras, and estimate the compute of a training run with C≈6NDC\approx6ND.
  • State the bitter lesson precisely, with its historical cases and its strongest counterarguments.
  • Prove the no-free-lunch theorem for off-training-set error, and explain what it implies for the bitter lesson.
  • Derive compute-optimal model and data sizes from a parametric scaling law, and contrast Kaplan et al. (2020) with Hoffmann et al. (2022).

Three eras of compute

Sevilla et al. (2022) compiled the training compute of over a hundred milestone systems and found three eras. Before about 2010, compute doubled roughly every 20 months, in line with Moore's law. From 2010 to 2022, the deep learning era, it doubled about every 6 months. And from late 2015 a large-scale trend appeared, of systems trained with 10 to 100 times more compute than the main trend.

Hardware price-performance improved far more slowly than total compute, so most of the growth came from spending: more chips, for longer.

DerivationCounting the FLOPs of a training run
  1. Cforward≈2N FLOPs per tokenC_{\text{forward}}\approx2N\ \text{FLOPs per token}

    In a dense transformer with NN parameters, each parameter takes part in about one multiply-add (2 floating-point operations) per token in the forward pass:

  2. Cbackward≈4N FLOPs per tokenC_{\text{backward}}\approx4N\ \text{FLOPs per token}

    The backward pass computes gradients with respect to both activations and weights, about twice the forward cost:

  3. C≈6NDC\approx6ND

    Training on DD tokens therefore costs about

  4. 6×(1.75×1011)×(3×1011)≈3.2×1023 FLOPs6\times\big(1.75\times10^{11}\big)\times\big(3\times10^{11}\big)\approx3.2\times10^{23}\ \text{FLOPs}

    For GPT-3 (2020), N=1.75×1011N=1.75\times10^{11} parameters and D=3×1011D=3\times10^{11} tokens, which matches the roughly 3×10233\times10^{23} FLOPs its authors reported.

Predict before you look

Chinchilla (2022) had 7×10107\times10^{10} parameters and was trained on 1.4×10121.4\times10^{12} tokens. Using C≈6NDC\approx6ND, what was its training compute, in units of 102310^{23} FLOPs?

Show the answer

5.9

6×(7×1010)×(1.4×1012)=5.88×10236\times(7\times10^{10})\times(1.4\times10^{12})=5.88\times10^{23}: almost twice GPT-3's compute, spent on a model 2.5 times smaller trained on nearly five times as many tokens.

The bitter lesson

DomainKnowledge-based approachGeneral method that won
Chessencoded chess expertise and selective searchmassive alpha–beta search on custom hardware (Deep Blue, 1997); later, evaluation learned by self-play (AlphaZero, 2017)
Gohand-crafted patterns and proverbsMonte Carlo tree search (from 2006), then deep networks trained by reinforcement learning plus search (AlphaGo, 2016; AlphaGo Zero, 2017, with no human games)
Speechphonetic and linguistic rulesstatistical HMMs (1970s–80s), then deep networks trained end to end (2010s)
Visionhand-designed features (edges, SIFT, HOG)convolutional networks learning their features from data (AlexNet, 2012)
Languagegrammars and hand-built lexiconsstatistical translation (1990s), then transformers pretrained on web-scale text (from 2017)
Sutton's essay cites chess, Go, speech and vision; language has been the most dramatic case since.

The strongest objections deserve equal space:

  • Inductive biases are knowledge too. Convolution builds in translation equivariance, attention and residual connections are human designs, and alpha–beta and MCTS are clever algorithms. The real question is which knowledge to build in: general structure rather than domain specifics. Rodney Brooks made this point within days, in a reply titled "A Better Lesson".
  • Data is finite. Projections suggest that the stock of public human-written text could be used up for training somewhere between the mid-2020s and the early 2030s, if current trends continue.
  • Efficiency matters. Children acquire language from at most a few hundred million words, while language models train on trillions of tokens, and compute has financial and energy costs.
  • Some domains have little data, and there priors and structure still decide.

No free lunch

If general methods keep winning, is there a universally best learner? A theorem says no, and it sharpens what the bitter lesson can mean.

DerivationProof
  1. Pr⁡f(f(x)=y∣D)=1kfor every y∈Y\Pr_f\big(f(x)=y\mid D\big)=\frac1k\quad\text{for every }y\in Y

    Fix the training inputs and an input xx outside them. Under the uniform distribution over ff, the labels at different inputs are independent and uniform, so f(x)f(x) is independent of everything in DD:

  2. Pr⁡f(AD(x)≠f(x)∣D)=1−1k\Pr_f\big(A_D(x)\ne f(x)\mid D\big)=1-\frac1k

    AD(x)A_D(x) depends only on DD, so whatever it predicts is right with probability 1/k1/k:

  3. Ef[errOTS(A,f)]=1−1kfor every learner A\E_f\big[\mathrm{err}_{\mathrm{OTS}}(A,f)\big]=1-\frac1k\quad\text{for every learner }A

    Average over the unseen inputs and over training sets. The learner never entered the answer.

The PAC setting has its own version of the no-free-lunch theorem: for any learner and any training size m<∣X∣/2m<\abs X/2, some distribution makes a constant classifier perfect while the learner errs at least 1/81/8 of the time with probability at least 1/71/7 (Shalev-Shwartz and Ben-David, 2014).

So the bitter lesson cannot be a theorem about all problems. It is an empirical claim about the problems we care about: they are overwhelmingly structured (smooth, compositional, local, symmetric, with heavy-tailed statistics), and flexible learners with the right mild biases, given enough data and compute, find that structure more reliably than people can write it down.

Check your understanding

What does the no-free-lunch theorem imply about deep learning's success?

  1. That the success must be an illusion caused by flawed benchmarks
  2. That neural networks are universal learners that beat all others on every problem
  3. That the advantage must come from assumptions, such as inductive biases, that match the structure of the problems that actually occur
  4. Nothing, because the theorem applies only to finite input spaces
Show the answer

C. That the advantage must come from assumptions, such as inductive biases, that match the structure of the problems that actually occur

Averaged over all targets, no learner beats chance off the training set. Real tasks are a tiny, highly structured subset, and a learner wins exactly when its biases fit that subset.

Scaling laws and compute-optimal training

Scaling laws turned the bitter lesson into engineering. Hestness et al. (2017) found power laws relating error to data size across several domains. Kaplan et al. (2020) found that the test loss of transformer language models falls as a power law in parameters, data and compute over many orders of magnitude, for example L(N)≈(Nc/N)0.076L(N)\approx(N_c/N)^{0.076} in the number of non-embedding parameters, and concluded that extra compute should go mostly into bigger models, Nopt∝C0.73N_{\mathrm{opt}}\propto C^{0.73}. Hoffmann et al. (2022) fit a joint law and reached a different conclusion.

L(N,D)=E+ANα+BDβ,α≈0.34,β≈0.28L(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}},\qquad\alpha\approx0.34,\quad\beta\approx0.28
EE is the irreducible loss (the entropy of text itself); the other two terms are the penalties for a finite model and for finite data.
DerivationThe compute-optimal allocation
  1. L(N)=E+AN−α+B(6NC)βL(N)=E+AN^{-\alpha}+B\Big(\frac{6N}{C}\Big)^{\beta}

    Minimize LL subject to the budget 6ND=C6ND=C. Substitute D=C/(6N)D=C/(6N):

  2. −αAN−α−1+βB(6C)βNβ−1=0 ⟺ αAN−α=βBD−β-\alpha AN^{-\alpha-1}+\beta B\Big(\frac{6}{C}\Big)^{\beta}N^{\beta-1}=0\ \Longleftrightarrow\ \alpha AN^{-\alpha}=\beta BD^{-\beta}

    Set the derivative to zero. At the optimum, the two penalty terms, weighted by their exponents, balance:

  3. Nopt=G(C6)βα+β∝C0.45,Dopt=G−1(C6)αα+β∝C0.55,G=(αAβB)1α+βN_{\mathrm{opt}}=G\Big(\frac C6\Big)^{\frac{\beta}{\alpha+\beta}}\propto C^{0.45},\qquad D_{\mathrm{opt}}=G^{-1}\Big(\frac C6\Big)^{\frac{\alpha}{\alpha+\beta}}\propto C^{0.55},\qquad G=\Big(\frac{\alpha A}{\beta B}\Big)^{\frac{1}{\alpha+\beta}}

    Solving gives power laws in compute whose exponents sum to 1:

  4. C×100 ⟹ Nopt×10,Dopt×10C\times100\ \Longrightarrow\ N_{\mathrm{opt}}\times10,\quad D_{\mathrm{opt}}\times10

    So parameters and data should grow roughly equally, about 20 tokens per parameter at the scales studied. Chinchilla (70 billion parameters, 1.4 trillion tokens) beat the 280-billion-parameter Gopher, trained on 300 billion tokens with about the same compute. Later work traced much of the disagreement with Kaplan et al. to details such as learning-rate schedules, how parameters are counted and the scale of the experiments.

Key takeaways

  • Training compute doubled about every 20 months before 2010 and about every 6 months from 2010 to 2022, mostly through spending. A dense transformer costs C≈6NDC\approx6ND to train; GPT-3 took about 3×10233\times10^{23} FLOPs.
  • Sutton's bitter lesson: search and learning that scale with compute have repeatedly overtaken built-in knowledge, in chess, Go, speech, vision and language. The counterarguments: inductive biases are knowledge too, data is finite, and efficiency matters.
  • No free lunch: averaged over all targets, every learner has off-training-set error 1−1/k1-1/k. The bitter lesson is therefore an empirical claim about the structured problems we actually face, not a theorem.
  • With L=E+A/Nα+B/DβL=E+A/N^\alpha+B/D^\beta and C=6NDC=6ND, the optimum balances αAN−α=βBD−β\alpha AN^{-\alpha}=\beta BD^{-\beta}, giving N,D∝C≈0.5N,D\propto C^{\approx0.5} (Chinchilla), rather than putting most compute into parameters (Kaplan et al.).

Check yourself

In the interactive lesson these are scored with your confidence, and what you miss comes back for review.

Question 1

About how many floating-point operations does it take to train a dense transformer with NN parameters on DD tokens?

  1. NDND
  2. 2ND2ND
  3. 6ND6ND
  4. N2DN^2D
Show the answer

C. 6ND6ND

About 2N2N per token in the forward pass and 4N4N in the backward pass, so 6ND6ND in total.

Question 2

Under Chinchilla-style compute-optimal scaling, the compute budget grows 100-fold. Roughly how should the model size change?

  1. About 10 times larger, trained on about 10 times more data
  2. About 100 times larger, trained on the same data
  3. About 29 times larger, trained on about 3.5 times more data
  4. Unchanged, trained on 100 times more data
Show the answer

A. About 10 times larger, trained on about 10 times more data

With exponents near 0.50.5 each, 1000.5=10100^{0.5}=10 for both parameters and tokens. The third option is Kaplan et al.'s earlier prescription, N∝C0.73N\propto C^{0.73}.

Question 3

What does the no-free-lunch theorem say?

  1. Deep learning cannot generalize
  2. Averaged uniformly over all target functions, every learner has the same off-training-set error
  3. Larger models always generalize better
  4. Learning is impossible without infinite data
Show the answer

B. Averaged uniformly over all target functions, every learner has the same off-training-set error

Under a uniform prior over targets, unseen labels are independent of the training data, so no learner beats chance on them. Any real advantage comes from assumptions that match the problems that actually occur.

Question 4

Which is the strongest form of the objection that Sutton's lesson undervalues human knowledge?

  1. Search and learning never worked well in practice
  2. Hand-coded expert systems still beat neural networks at most tasks
  3. Computation has stopped getting cheaper
  4. Successful general methods themselves embed human-designed inductive biases (convolution, attention, search algorithms), so the real question is which knowledge to build in
Show the answer

D. Successful general methods themselves embed human-designed inductive biases (convolution, attention, search algorithms), so the real question is which knowledge to build in

The objection accepts the track record but reframes it: what lost was domain-specific knowledge, and what won was general structure, also designed by people.

Further reading

  • R. S. Sutton, "The Bitter Lesson" (essay, 2019).
  • J. Sevilla et al., "Compute trends across three eras of machine learning," arXiv:2202.05924 (2022).
  • J. Kaplan et al., "Scaling laws for neural language models," arXiv:2001.08361 (2020).
  • J. Hoffmann et al., "Training compute-optimal large language models," NeurIPS (2022).
  • S. Shalev-Shwartz & S. Ben-David, Understanding Machine Learning: From Theory to Algorithms (Cambridge University Press, 2014), chapter 5.