Simple Algorithm
Artificial Intelligence · Deep Layer · 26 min

Measuring Intelligence & AGI

Benchmarks, the ARC challenge and formal definitions of intelligence.

Every few months an AI system "beats humans" on another benchmark, yet there is no agreement on whether machines are becoming generally intelligent. Much of the confusion comes from never saying what is being measured. This lesson examines a formal definition of intelligence, the ARC challenge, the statistics of leaderboards, and what claims about AGI can and cannot mean.

What you’ll learn

  • Define Legg–Hutter universal intelligence, prove that it is well defined, and explain why it is incomputable and depends on a reference machine.
  • Distinguish skill from skill-acquisition efficiency, and explain how ARC is designed to measure the latter.
  • Derive confidence bounds for benchmark scores, and explain how adaptive reuse of a test set breaks them.
  • Evaluate claims about AGI against explicit definitions, and present the main positions in the debate fairly.

A formal definition of intelligence

Legg and Hutter (2007) collected some seventy informal definitions of intelligence and distilled their common core: intelligence measures an agent's ability to achieve goals in a wide range of environments. Their formal version averages an agent's performance over every computable environment, weighting simple environments more heavily: Occam's razor, applied to worlds.

DerivationWell defined, bounded, and incomputable
  1. ∑μ∈E2−K(μ) ≤ ∑p ∈ a prefix-free set2−ℓ(p) ≤ 1\sum_{\mu\in E}2^{-K(\mu)}\ \le\ \sum_{p\ \in\ \text{a prefix-free set}}2^{-\ell(p)}\ \le\ 1

    Distinct environments have distinct shortest programs, and the shortest programs form a prefix-free set, so Kraft's inequality bounds the total weight:

  2. 0 ≤ Υ(π) ≤ 10\ \le\ \Upsilon(\pi)\ \le\ 1

    Since 0≤Vμπ≤10\le V^\pi_\mu\le1, the series converges and Υ(π)∈[0,1]\Upsilon(\pi)\in[0,1]. A uniform average would be impossible: there is no uniform distribution on a countably infinite set.

  3. 2−202−40=220≈106\frac{2^{-20}}{2^{-40}}=2^{20}\approx10^6

    The weighting is steep. An environment describable in 20 bits counts about a million times more than one that needs 40 bits, so Υ\Upsilon is dominated by simple worlds.

  4. ξ=∑μ∈E2−K(μ) μ,Υ(π)=Vξπ,πAIXI∈arg max⁡πVξπ\xi=\sum_{\mu\in E}2^{-K(\mu)}\,\mu,\qquad\Upsilon(\pi)=V^\pi_\xi,\qquad\pi^{\mathrm{AIXI}}\in\argmax_\pi V^\pi_\xi

    KK is incomputable, and so is Υ\Upsilon. Hutter's agent AIXI acts optimally in the mixture ξ\xi of all computable environments weighted the same way. For a fixed policy, the expected reward in a mixture is the weighted average of the expected rewards, so Υ(π)=Vξπ\Upsilon(\pi)=V^\pi_\xi, and AIXI maximizes Υ\Upsilon by construction. It too is incomputable: a definition of an ideal, not a program.

Think first If the constant in the invariance theorem is harmless for KK, why can it wreck Υ\Upsilon?

A change of machine can multiply each weight 2−K(μ)2^{-K(\mu)} by a factor as large as 2c2^{c}, and cc can be enormous. A machine with a very short program for one "trap" environment, say one that punishes any deviation from a particular policy, can give that environment most of the total weight, and then Υ\Upsilon mostly measures performance in that single world. Invariance up to a constant says nothing about which environments dominate a finite sum.

Skill is not intelligence: Chollet and ARC

Chollet (2019) argues that measuring skill at fixed tasks says little about intelligence, because skill can be bought: with built-in priors (a hand-coded chess engine) or with unlimited experience (a network trained on millions of games). Neither shows an ability to handle what neither the system nor its developers anticipated. His definition: "The intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty."

He formalizes it with algorithmic information theory: skill achieved, discounted by the information supplied through priors and experience, and weighted by how far the test tasks lie from anything the system was given.

The Abstraction and Reasoning Corpus (ARC) puts this into practice. Each task shows a few input–output pairs of colored grids (up to 30×3030\times30 cells, ten colors) and asks for the exact output grid of a new input, within two or three attempts. Tasks assume only "core knowledge" priors that young children share (objectness, goal-directedness, counting, basic geometry and topology, after the developmental psychologist Elizabeth Spelke), and the hidden test tasks are designed to differ from the public ones, so that developers cannot anticipate them.

YearWhat happened
2019ARC released with Chollet's paper. Most tasks are easy for people and very hard for deep learning systems.
2020First Kaggle competition. The winning entry, a search over a hand-written library of grid transformations, solved roughly a fifth of the hidden tasks.
2024ARC Prize launched, with over a million dollars in prizes. By the end of the year, systems combining large language models with heavy test-time computation (search, sampling, or training on each task's examples) had made large gains, at a high cost per task.
2025ARC-AGI-2 released: harder tasks, each one solved by people in calibration studies, aimed at the weaknesses of those systems.
A benchmark built to resist memorization still saw rapid progress once methods began spending computation per task. Whether that progress reflects efficient skill acquisition or very efficient search over learned priors is exactly what Chollet's definition asks you to account for.
Think first Why does ARC give only a handful of demonstrations per task, instead of thousands?

Because the quantity of interest is efficiency. With thousands of examples, a generic learner could acquire each skill by brute-force fitting, and the score would measure data rather than generalization. A few examples, plus test tasks unlike anything public, force the solver to generalize from the priors it already has.

What a benchmark score can tell you

DerivationProof and numbers
  1. Pr⁡(∃j≤k: ∣err^(hj)−err(hj)∣>ε) ≤ 2k e−2nε2\Pr\big(\exists j\le k:\ |\widehat{\mathrm{err}}(h_j)-\mathrm{err}(h_j)|>\varepsilon\big)\ \le\ 2k\,e^{-2n\varepsilon^2}

    The test error is an average of nn independent indicators in [0,1][0,1] with mean err(h)\mathrm{err}(h), so Hoeffding's inequality gives the first bound. A union bound over kk classifiers multiplies the failure probability by kk:

  2. ε(k,n,δ)=ln⁡(2k/δ)2n\varepsilon(k,n,\delta)=\sqrt{\frac{\ln(2k/\delta)}{2n}}

    Set the right-hand side to δ\delta and solve for ε\varepsilon:

  3. ε(1)≈0.014,ε(1000)≈0.023\varepsilon(1)\approx0.014,\qquad\varepsilon(1000)\approx0.023

    With n=10,000n=10{,}000 and δ=0.05\delta=0.05, one model is pinned down to about ±1.4\pm1.4 points, and even a thousand pre-registered models to about ±2.3\pm2.3. Comparing many models costs only logarithmically, if they were chosen before anyone saw a score.

Predict before you look

A single model, fixed in advance, is scored on n=2,500n=2{,}500 test examples. What is the half-width ε\varepsilon of its 95% Hoeffding interval, in percentage points?

Show the answer

2.7

ln⁡(2/0.05)/(2⋅2500)=3.69/5000≈0.027\sqrt{\ln(2/0.05)/(2\cdot2500)}=\sqrt{3.69/5000}\approx0.027. A test set four times smaller than 10,000 doubles the width, from 1.4 to 2.7 points: precision scales like 1/n1/\sqrt n.

Public leaderboards break the "fixed in advance" assumption, because each submission can depend on the scores of earlier ones. Blum and Hardt (2015) give a simple attack on a binary task: submit kk random labelings of the test set, keep those that score above 50%, and submit their majority vote. Its leaderboard accuracy is 12+Ω(k/n)\tfrac12+\Omega\big(\sqrt{k/n}\big) while its true accuracy is exactly 12\tfrac12: adaptivity turns a logarithmic penalty into a polynomial one. Remedies include their Ladder mechanism, a "reusable holdout" built on differential privacy (Dwork et al., 2015), and fresh test sets.

How much has this mattered in practice? Recht et al. (2019) rebuilt the CIFAR-10 and ImageNet test sets following the original procedures. Accuracy fell by 3–15 points on CIFAR-10 and 11–14 points on ImageNet, yet model rankings were largely preserved and gains on the old test sets carried over. The drop reflects a small distribution shift, not years of adaptive overfitting. For models trained on web-scale text, a different threat dominates: contamination, test items present in the training data.

AGI: definitions, levels and the debate

The phrase "artificial general intelligence" appeared in the late 1990s and spread in the 2000s, through Ben Goertzel, Shane Legg and others, as a name for the field's original ambition, in contrast with narrow AI. There is still no agreed definition. Proposals differ on the yardstick (humans, economic value, an ideal), on the breadth required, and on whether learning efficiency or final skill is what counts.

ProposalAGI means…Measured by
Turing (1950)indistinguishable from a person in open conversationthe imitation game
Legg & Hutter (2007)high universal intelligenceΥ\Upsilon (incomputable, so only approximations)
Chollet (2019)human-level skill-acquisition efficiency on novel tasksARC-style few-shot tasks
Morris et al. (2023)a cell in a matrix of performance (emerging, competent, expert, virtuoso, superhuman: from unskilled-human level to beyond every human) by generality (narrow or general)batteries of tasks; they rated the chatbots of 2023 "emerging AGI"
Economic definitionssystems that can do most economically valuable workautomation of real tasks and jobs
Each definition makes different systems look close to, or far from, AGI. A claim that "AGI is near" is meaningless until you say which definition it uses.

Newer measurements track trends instead of thresholds. METR (2025) measured the length of software tasks, in human working time, that AI agents complete with 50% reliability, and reported that this horizon doubled roughly every seven months from 2019 to 2025. That is one family of tasks, and extrapolating any trend is a forecast, not a measurement.

The debate over whether current methods lead to general intelligence has three main camps. Scaling optimists point to predictable gains with compute and data, and to a breadth no earlier system had. Missing-ingredient critics (citing world models, persistent memory, grounding, reliable reasoning or program synthesis) point to brittleness out of distribution, hallucinations and tasks like ARC. Concept skeptics argue that intelligence is multidimensional, so "general" has no single scale. The evidence so far fits parts of all three; what would discriminate between them is clean measurement on tasks that are genuinely novel to the system.

Key takeaways

  • Legg and Hutter's Υ(π)=∑μ2−K(μ)Vμπ\Upsilon(\pi)=\sum_\mu2^{-K(\mu)}V^\pi_\mu formalizes "achieving goals in a wide range of environments". Kraft's inequality makes it well defined; it is incomputable, and Leike and Hutter showed it depends heavily on the reference machine.
  • Chollet defines intelligence as skill-acquisition efficiency relative to priors and experience; ARC tests few-shot generalization to novel grid tasks using only core-knowledge priors.
  • A pre-registered model's test score is accurate to about ln⁡(2k/δ)/(2n)\sqrt{\ln(2k/\delta)/(2n)}, but adaptive reuse of a test set can bias scores by order k/n\sqrt{k/n}. ImageNet's replication found distribution shift rather than adaptive overfitting.
  • "AGI" has competing definitions (human indistinguishability, universal intelligence, learning efficiency, levels of performance and generality, economic value); a claim about AGI means little until you say which.

Check yourself

In the interactive lesson these are scored with your confidence, and what you miss comes back for review.

Question 1

Why does Legg–Hutter intelligence weight environments by 2−K(μ)2^{-K(\mu)} instead of averaging uniformly?

  1. Complex environments matter less to real agents
  2. No uniform average over countably many environments exists; complexity weights encode Occam's razor and sum to at most 1 by Kraft's inequality
  3. It makes Υ\Upsilon computable
  4. It guarantees that every reference machine gives the same scores
Show the answer

B. No uniform average over countably many environments exists; complexity weights encode Occam's razor and sum to at most 1 by Kraft's inequality

There is no uniform probability distribution on a countably infinite set. Weights 2−K(μ)2^{-K(\mu)} form a natural prior that favors simple environments and sums to at most 1.

Question 2

A team never trains on test labels, but it submits 500 model variants to a public leaderboard and keeps whichever scores best. Why might its reported accuracy be optimistic?

  1. Leaderboards round their scores
  2. Hoeffding's inequality fails for large numbers of models
  3. Choosing among many noisy scores, each submission informed by earlier ones, selects for test-set noise; adaptivity can bias results by order k/n\sqrt{k/n}
  4. It cannot be: without training on test labels there is no overfitting
Show the answer

C. Choosing among many noisy scores, each submission informed by earlier ones, selects for test-set noise; adaptivity can bias results by order k/n\sqrt{k/n}

Selection after observation is a form of fitting. With kk models fixed in advance the union bound costs only a ln⁡k\sqrt{\ln k} factor, but adaptive choices extract information about the test labels from the scores themselves.

Question 3

Recht et al. (2019) found that ImageNet models lost 11–14 points of accuracy on a freshly collected test set, but largely kept their relative order. What does this suggest?

  1. The drop mostly reflects a small distribution shift between the old and new test sets, not adaptive overfitting to the old one
  2. A decade of research had overfit the original ImageNet test set
  3. The new test set was mislabeled
  4. Benchmark rankings are meaningless
Show the answer

A. The drop mostly reflects a small distribution shift between the old and new test sets, not adaptive overfitting to the old one

Adaptive overfitting would reward the models most tuned to the old test set and scramble the rankings. Instead, gains on the old set carried over to the new one, pointing to slightly harder images in the new set.

Question 4

According to Chollet, what should a test of intelligence measure?

  1. Peak skill on a fixed set of tasks
  2. How much training data a system can absorb
  3. Performance on tasks known during development
  4. Skill-acquisition efficiency on novel tasks, relative to the priors and experience the system was given
Show the answer

D. Skill-acquisition efficiency on novel tasks, relative to the priors and experience the system was given

Skill can be bought with priors or data. Intelligence, on this view, is how efficiently a system turns limited priors and experience into skill on tasks it has not seen, which is what ARC's few-shot, novel tasks target.

Further reading

  • S. Legg & M. Hutter, "Universal intelligence: A definition of machine intelligence," Minds and Machines 17, 391–444 (2007).
  • F. Chollet, "On the measure of intelligence," arXiv:1911.01547 (2019).
  • A. Blum & M. Hardt, "The Ladder: A reliable leaderboard for machine learning competitions," ICML (2015).
  • B. Recht, R. Roelofs, L. Schmidt & V. Shankar, "Do ImageNet classifiers generalize to ImageNet?," ICML (2019).
  • J. Hernández-Orallo, The Measure of All Minds: Evaluating Natural and Artificial Intelligence (Cambridge University Press, 2017).