Double Descent & Modern Generalization
Why huge models can still generalize.
Classical theory says a model that fits noisy training data perfectly should generalize badly. Modern networks do it routinely and generalize well. This lesson computes test-error curves exactly for minimum-norm interpolation, shows where the infamous peak comes from and what removes it, proves when overfitting is benign, and maps what is still unexplained about deep networks.
What you’ll learn
- Describe model-wise, epoch-wise and sample-wise double descent, and the experiments that established them.
- Derive the exact risk of minimum-norm least squares with isotropic Gaussian features on both sides of the interpolation threshold, and explain the peak.
- State the Bartlett–Long–Lugosi–Tsigler characterization of benign overfitting through effective ranks, and sketch why the variance splits into .
- Weigh the explanations proposed for deep networks (kernel regimes, norms and margins, PAC-Bayes, implicit bias) against what they fail to explain.
The puzzle and the curve
Zhang, Bengio, Hardt, Recht & Vinyals (2017) trained standard image classifiers on CIFAR-10 with the labels replaced by random ones. The networks still reached zero training error, only somewhat more slowly, while the same architectures generalize well on the true labels. A class that can fit noise has maximal Rademacher complexity, so every bound that depends only on the hypothesis class (VC dimension, Rademacher complexity of the class) is vacuous for these models. Explicit regularization helped, but it was neither necessary nor sufficient for good generalization.
Belkin, Hsu, Ma & Mandal (2019) then traced the full curve. Fitting random Fourier features by minimum-norm least squares, they found test error falling, peaking where the number of features equals the number of training points , then falling again as grows past : double descent. Nakkiran et al. (2020) found it in deep networks along three axes: model-wise (width), epoch-wise (training time) and sample-wise, where adding data to a fixed model can hurt. Label noise sharpens the peak, which sits where the training procedure can just barely fit the training set.
Think first A network architecture reaches zero training error on CIFAR-10 both with the true labels and with uniformly random labels. What follows for generalization bounds that depend only on the architecture's hypothesis class?
They must be vacuous for it. On random labels the same class reaches zero training error and 90% test error (10 classes), and a class-only bound has to cover that case too, so it cannot certify anything better on real labels. Nor does this mean the network cannot generalize (it does, on true labels), or that its VC dimension is small (fitting random labelings of 50,000 points is strong evidence that it is huge). Whatever explains generalization must involve the data distribution and the algorithm: which of the many interpolants training actually finds. Data- and algorithm-dependent bounds (PAC-Bayes, compression, margins) remain possible.
Exact curves for minimum-norm least squares
The cleanest model where everything can be computed: inputs , responses with and noise variance , and samples stacked into . The estimator is : ordinary least squares when , the minimum-norm interpolant when . Its excess risk is .
Underparameterized. , so the error is pure variance. is Wishart with degrees of freedom, and the mean of its inverse is :
Overparameterized. , where projects onto the row space of . The cross term has mean zero:
Bias. By rotational invariance the row space is a uniformly random -dimensional subspace of , which captures on average a fraction of . Variance. is an Wishart matrix with degrees of freedom:
Both variance terms diverge as . The smallest singular value of a Gaussian matrix is about , so near the threshold has a huge norm and amplifies the noise: the peak is a variance explosion, not a bias effect.
Isotropic Gaussian features, samples, features, ordinary least squares. The excess risk is how many times the noise variance ?
Show the answer
10 × σ²
. With 50 features it would be ; with 99 features the expectation is infinite. The textbook advice of "a few samples per parameter" is about exactly this blow-up.
Three refinements matter. First, the peak belongs to the ridgeless estimator: for isotropic Gaussian features, optimally tuned ridge has risk monotone in both and (Nakkiran, Venkat, Kakade & Ma, 2021), as the widget's flat curve at λ ≈ 0.05 shows. Second, past the threshold the interpolant's norm falls: adding features enlarges the set of interpolants, so the minimum norm can only go down, and here its expectation is , decreasing in . Third, in this well-specified model the risk tends to , the risk of predicting zero, as ; when the learner sees only some of many relevant features (a misspecified model), extra features also cut the approximation error, and the global minimum can lie beyond the threshold (Belkin, Hsu & Xu, 2020; Hastie et al., 2022).
Benign overfitting
Isotropic features are the worst case for interpolation: every direction is as important as every other, so noise fitted anywhere hurts. Real data have decaying spectra. Bartlett, Long, Lugosi & Tsigler (2020) found exactly which covariances let the minimum-norm interpolant fit pure noise and still predict almost optimally.
The noise contributes with . In the eigenbasis of , with i.i.d. standardized vectors (one per direction), and
The tail acts as a ridge. If , the tail is a sum of many comparable rank-one terms, and with high probability all its eigenvalues are within constant factors of . On the top directions it behaves like a ridge penalty of that size:
Tail directions () each see . Their total is small when the tail has many more than comparable directions: the noise is spread thinly over directions that barely affect predictions.
Top directions (), each strong compared with the implicit ridge, contribute about each, as in least squares on well-conditioned features (a Sherman–Morrison computation). Taking gives the noise term of the theorem.
So the interpolant is a hybrid: on the top directions it behaves like ridge regression with implicit penalty , and the tail acts as a sponge that absorbs the noise. Benign examples include with , and a few strong directions plus a flat tail spread over many more than dimensions with small total variance. Power laws with decay too fast ( stays comparable to ), and isotropic features in dimension have no sponge at all; both overfit harmfully.
For which covariance spectrum (infinite-dimensional, ) is minimum-norm interpolation of noisy data benign?
- Isotropic features in dimension
Show the answer
D.
This spectrum is summable, so is finite, yet it decays slowly enough that the tail beyond any has effective rank far larger than , making . Bartlett et al. show is benign exactly when and .
Deep networks: what is and is not explained
Kernel regime. As width grows with standard scaling and small steps, training linearizes around the initialization: the network behaves like kernel regression with the neural tangent kernel (Jacot, Gabriel & Hongler, 2018), and gradient descent converges to the minimum-RKHS-norm interpolant relative to the initial function, so the linear theory above applies with the kernel's eigenvalues as the spectrum. Random-features models give exact double-descent curves (Mei & Montanari, 2022). But real networks move far from their initialization and learn features, and networks kept in this "lazy" regime usually perform worse (Chizat, Oyallon & Bach, 2019).
Beyond kernels. Grokking (Power et al., 2022) shows training accuracy perfect long before test accuracy jumps, on small algorithmic tasks, with weight decay playing a key role. In practice, with tuned regularization and early stopping, double-descent peaks are often small or absent; the durable lesson is that parameter count is a poor measure of complexity.
| Explanation | Core idea | Captures | Limits |
|---|---|---|---|
| Class capacity (VC, Rademacher of the class) | count what the class can fit | classical U-curve | vacuous: the same networks fit random labels |
| Implicit bias + benign overfitting | the algorithm picks low-norm interpolants; noise hides in weak directions | exact for linear and kernel models, so for the lazy regime | feature learning lies outside it |
| Norm and margin bounds | scale-sensitive complexity of the trained weights (e.g. Bartlett, Foster & Telgarsky, 2017) | trends with data and label noise | numerically loose; uniform convergence can provably fail (Nagarajan & Kolter, 2019) |
| PAC-Bayes and compression | a trained predictor close to a prior, or describable in few bits | non-vacuous numbers for some real networks (Dziugaite & Roy, 2017) | needs stochastic or compressed predictors; weakly predictive for design choices |
| Scaling laws | empirical power laws in parameters, data and compute | forecasts at scale (Scaling Laws) | descriptive, not explanatory |
Key takeaways
- Networks that fit random labels make class-only bounds vacuous. Double descent, a test-error peak at the interpolation threshold followed by a second descent, appears model-wise, epoch-wise and sample-wise.
- For isotropic Gaussian features the minimum-norm estimator's excess risk is below the threshold and above: the peak is noise amplified by the smallest singular values of , and tuned ridge removes it.
- Past the threshold the minimum-norm interpolant's norm falls as features are added: more parameters can mean a simpler interpolant, so parameter count is a poor measure of complexity.
- Benign overfitting: interpolation is consistent when , and ; the tail acts as an implicit ridge and a sponge for noise.
- For deep networks, kernel limits, norm and margin bounds, PAC-Bayes, compression and scaling laws each explain part of the picture; a predictive theory of generalization with feature learning is still open.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
For minimum-norm least squares with samples and isotropic Gaussian features, where does the expected test error peak as the number of features varies, and why?
- At , where bias and variance are equal
- At , because the smallest singular value of approaches zero and amplifies the noise
- At , because more parameters always mean more variance
- Nowhere: the error decreases monotonically in
Show the answer
B. At , because the smallest singular value of approaches zero and amplifies the noise
Both variance formulas, and , diverge at , where . Far past the threshold, the variance falls again as .
In the isotropic Gaussian model, what does optimally tuned ridge regularization do to the double-descent curve?
- Shifts the peak to larger
- Doubles the height of the peak
- Removes the peak: the optimally regularized risk is monotone in and in
- Nothing, because the peak is caused by bias
Show the answer
C. Removes the peak: the optimally regularized risk is monotone in and in
The peak is variance from tiny singular values, exactly what ridge's filter suppresses. Nakkiran, Venkat, Kakade & Ma (2021) proved monotonicity for optimally tuned ridge in this model; in the widget, λ ≈ 0.05 flattens the curve to between 0.056 and 0.070 at every degree from 3 to 15.
Isotropic features, , minimum-norm estimator. You grow the training set from to . What happens to the variance term ?
- It rises from about to about : more data hurts here
- It falls, because more data always reduces variance
- It stays the same, because did not change
- It falls to zero, because the model now interpolates
Show the answer
A. It rises from about to about : more data hurts here
and . Moving toward moves the problem onto the interpolation threshold: this is sample-wise double descent (Nakkiran et al., 2020). With optimally tuned ridge, more data would help, as one expects.
In the Bartlett–Long–Lugosi–Tsigler theory, what makes fitting noise harmless?
- A small number of features, so the noise has nowhere to go
- Isotropic features, so the noise is spread evenly
- Early stopping before the interpolation point
- A tail of many low-variance directions, with effective rank , that absorbs the noise without affecting predictions much
Show the answer
D. A tail of many low-variance directions, with effective rank , that absorbs the noise without affecting predictions much
The noise term is : few important directions (), and a tail with many more than comparable weak directions. The fitted noise then lives in directions that contribute little to the prediction for a new input.
Further reading
- M. Belkin, D. Hsu, S. Ma & S. Mandal, "Reconciling modern machine-learning practice and the classical bias–variance trade-off," Proceedings of the National Academy of Sciences 116(32), 15849–15854 (2019).
- P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak & I. Sutskever, "Deep double descent: where bigger models and more data hurt," ICLR (2020).
- P. L. Bartlett, P. M. Long, G. Lugosi & A. Tsigler, "Benign overfitting in linear regression," Proceedings of the National Academy of Sciences 117(48), 30063–30070 (2020).
- T. Hastie, A. Montanari, S. Rosset & R. J. Tibshirani, "Surprises in high-dimensional ridgeless least squares interpolation," Annals of Statistics 50(2), 949–986 (2022).
- P. L. Bartlett, A. Montanari & A. Rakhlin, "Deep learning: a statistical viewpoint," Acta Numerica 30, 87–201 (2021).