Experimental Design & Statistical Testing
Significance, multiple comparisons and honest benchmarks.
A new model beats the old one by 0.4 points. Is it better, or did it draw a luckier seed, a friendlier test set, or more tuning runs? Many machine-learning comparisons never ask. This lesson builds the statistics: what varies, which tests respect the pairing created by a shared test set, what Dietterich's 1998 study found about popular tests, how many examples a comparison needs, and how to keep many comparisons honest.
What you’ll learn
- Separate the sources of variance in a comparison, and derive the variance of a paired accuracy difference and the sample size it implies.
- Prove McNemar's null distribution, and apply the exact test and the paired bootstrap.
- Summarize Dietterich's (1998) findings on resampled t-tests, McNemar's test and the 5×2cv test.
- Prove that Bonferroni and Holm control the family-wise error rate, and state the Benjamini–Hochberg guarantee for the false discovery rate.
What varies, and the paired advantage
First fix the question. (A) Is this trained model better than on this distribution? Then only the test sample is random. (B) Is learning algorithm better than on datasets of this kind and size? Then the training set, initialization, data order and hyperparameter search are random too. Bouthillier et al. (2021) measured these sources in deep-learning benchmarks, found data sampling among the largest, and recommended randomizing as many sources as possible. Question (B) needs repeated training runs; question (A) needs a good test of a single split.
On a shared test set let , with and , the two disagreement probabilities. The accuracy difference is the mean of the :
So the paired estimate has variance driven by how often the models disagree, not by their accuracy. Treating the two accuracies as independent proportions gives instead
With versus , 4% disagreement and : standard errors of 0.20 points paired against 0.41 unpaired. For a two-sided level- test with power against a difference , the required size is
Two models near 90% accuracy disagree on 4% of examples. With a paired test at and 80% power, roughly how many test examples do you need to detect a true difference of 1 point? And how many if the models disagreed on 10%?
Show the answer
3100
. At 10% disagreement, and . Card et al. (2020) found that many NLP comparisons on standard test sets are underpowered for the small gains they report.
Paired tests: McNemar and the bootstrap
Cross-tabulate the two models on the test set: examples where only is right, where only is right, and the rest where they agree. Agreements say nothing about which model is better; the test lives on the discordant pairs.
Each example falls independently into one of four cells, with probabilities . Given that an example is discordant, it favors with probability
Independence across examples makes the discordant examples' directions i.i.d. coin flips, whichever examples turn out to be discordant:
import numpy as np
def paired_bootstrap_ci(correct_f, correct_g, B=10_000, level=0.95, seed=0):
"""Percentile CI for acc(f) - acc(g), resampling test examples jointly."""
d = np.asarray(correct_f, float) - np.asarray(correct_g, float)
rng = np.random.default_rng(seed)
idx = rng.integers(0, len(d), size=(B, len(d))) # B resampled test sets
means = d[idx].mean(axis=1)
lo, hi = np.quantile(means, [(1 - level) / 2, (1 + level) / 2])
return d.mean(), (lo, hi)Resample examples, keeping both models' outcomes together, so the pairing survives. For , the interval is about , in line with McNemar. With very few discordant pairs the percentile bootstrap is anticonservative: for , it gives about , excluding zero, while the exact test gives .
| Test (Dietterich, 1998) | What it does | Finding |
|---|---|---|
| Difference of two proportions | -test treating the two accuracies as independent | high type I error in some settings; avoid |
| Resampled paired -test | -test over repeated random train/test splits | high type I error: overlapping splits make the differences correlated, so the variance is underestimated |
| 10-fold CV paired -test | -test over the ten fold differences | somewhat elevated type I error; the most power |
| McNemar | one split, discordant pairs | low type I error; the choice when an algorithm can be trained only once |
| 5×2cv paired -test | five replications of 2-fold CV | acceptable type I error, slightly more power than McNemar; recommended when ten training runs are affordable |
Many comparisons
Each test controls its own false-positive rate at . Run of them and the probability of at least one false positive, the family-wise error rate (FWER), can approach . Benchmark tables compare many models, datasets, metrics and ablations at once, and leaderboards thousands.
Let be the true nulls; under , . Bonferroni by the union bound:
Holm: let be the rank of the smallest true-null p-value. Only false nulls precede it, so . A false rejection requires reaching step and passing it:
For large families, controlling the false discovery rate, with false and total rejections, is often more useful. Benjamini and Hochberg (1995) reject the smallest p-values, where
Five comparisons give p-values . At , how many hypotheses does Holm's procedure reject?
- 1
- 2
- 3
- 4
Show the answer
C. 3
Holm's thresholds are , that is . The first three p-values pass; stops the procedure. Bonferroni (all thresholds ) rejects 2, and Benjamini–Hochberg (thresholds ) rejects 4.
Across datasets, Demšar (2006) recommends the Wilcoxon signed-rank test on per-dataset differences for two classifiers, rather than a -test over incommensurable scores or a count of wins, and for several classifiers the Friedman test on ranks followed by a post-hoc test. Implicit comparisons count too: trying many variants and reporting the one that worked is a multiple comparison even if only one test is ever run, Gelman and Loken's "garden of forking paths".
Honest benchmarks
Tuning effort is a hidden variable. Melis, Dyer and Blunsom (2018) found that carefully tuned standard LSTMs outperformed more recent architectures on language-modeling benchmarks. Lucic et al. (2018) found that, with equal hyperparameter budgets and enough random restarts, most GAN variants reached similar scores. Henderson et al. (2018) showed that deep reinforcement-learning results swing with random seeds and implementation details, and Agarwal et al. (2021) proposed reporting interquartile means with stratified bootstrap intervals when only a few runs are affordable. How adaptive reuse of a test set biases leaderboards is covered in Measuring Intelligence & AGI, and contamination of language-model benchmarks in Evaluating Language Models.
Key takeaways
- Decide whether you are comparing trained models (only the test sample varies) or algorithms (training data, seeds and tuning vary too).
- A paired difference has variance , set by disagreement; detecting 1 point at 4% disagreement takes about 3,100 test examples, against 13,500 unpaired.
- McNemar's exact test conditions on the discordant pairs ( under ); the paired bootstrap resamples examples jointly and is unreliable when is tiny.
- Dietterich (1998): avoid the resampled -test and independent-proportions tests; use McNemar for a single run or 5×2cv when ten runs are affordable.
- Bonferroni and Holm control the FWER under any dependence (union bound); Benjamini–Hochberg controls the FDR at .
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
Why is a paired comparison of two models on one test set much more sensitive than comparing their accuracies as independent proportions?
- Pairing doubles the effective number of test examples
- Both models get most examples right or wrong together; pairing cancels that shared variation, leaving variance set by how often they disagree
- The paired test assumes the models are independent
- Pairing removes the need for a significance level
Show the answer
B. Both models get most examples right or wrong together; pairing cancels that shared variation, leaving variance set by how often they disagree
depends only on disagreements. Highly correlated models disagree rarely, so the paired standard error is small even when each accuracy alone is noisy.
Models and disagree on 12 test examples: is right on 9 of them and on 3. What is the exact two-sided McNemar p-value?
- About 0.07
- It cannot be computed without the number of examples on which they agree
- About 0.15
- About 0.0005
Show the answer
C. About 0.15
Under , : . Not significant at 5%, despite a 3-to-1 split.
Why did Dietterich (1998) find that the paired -test over repeated random train/test splits rejects true null hypotheses too often?
- The splits overlap, so the differences are positively correlated and their sample variance underestimates the true variance
- The distribution has heavier tails than the normal
- Random splits always favor the more complex algorithm
- It uses too few splits to be valid
Show the answer
A. The splits overlap, so the differences are positively correlated and their sample variance underestimates the true variance
The -test assumes independent differences. Training and test sets reused across splits make them dependent, so the variance is underestimated and the statistic is inflated. Nadeau and Bengio's correction inflates the variance accordingly.
A screen of 200 ablations is analyzed with the Benjamini–Hochberg procedure at 5%. What does this guarantee (for independent p-values)?
- The probability of any false discovery is at most 5%
- Each rejected hypothesis has p-value at most 0.05/200
- On average, at most 5% of the rejected hypotheses are false discoveries
- At least 95% of the real effects are detected
Show the answer
C. On average, at most 5% of the rejected hypotheses are false discoveries
FDR is the expected fraction of false discoveries among the rejections. It allows some false positives in exchange for much more power than FWER control when the family is large.
Further reading
- T. G. Dietterich, "Approximate statistical tests for comparing supervised classification learning algorithms," Neural Computation 10, 1895–1923 (1998).
- J. Demšar, "Statistical comparisons of classifiers over multiple data sets," Journal of Machine Learning Research 7, 1–30 (2006).
- C. Nadeau & Y. Bengio, "Inference for the generalization error," Machine Learning 52, 239–281 (2003).
- Y. Benjamini & Y. Hochberg, "Controlling the false discovery rate: a practical and powerful approach to multiple testing," Journal of the Royal Statistical Society B 57, 289–300 (1995).
- B. Efron & R. J. Tibshirani, An Introduction to the Bootstrap (Chapman & Hall, 1993).