Distribution Shift & Robustness
When the test world differs from the training world.
Every guarantee in this topic assumed that test data come from the training distribution. Deployment breaks that assumption: a diagnostic model moves to a hospital with different patients, an epidemic changes disease prevalence, spammers change tactics. This lesson classifies the ways a distribution can shift, derives what can be corrected (importance weighting for covariate shift, prior correction and estimation for label shift), proves the bound of Ben-David et al. that frames domain adaptation, and surveys what robustness benchmarks show.
What you’ll learn
- Define covariate, label and concept shift by which factor of the joint distribution changes, and judge which assumption fits a problem.
- Derive importance weighting, its variance and effective sample size, and estimate density ratios with a domain classifier.
- Correct calibrated probabilities for a new prevalence, and estimate that prevalence from unlabeled data.
- Prove the Ben-David et al. (2010) domain-adaptation bound and explain what it does and does not justify.
A taxonomy of shift
A source distribution produced the training data; a target will produce the inputs the model meets. With no assumption linking them, the source says nothing about the target. Each standard kind of distribution shift keeps one factor of the joint distribution fixed:
| Shift | Changes | Stays fixed | Typical setting |
|---|---|---|---|
| Covariate shift | new sites or sensors; selection of who gets labeled | ||
| Label (prior) shift | an epidemic changes prevalence; fraud rates vary by season | ||
| Concept shift | possibly | adversaries adapt; definitions or policies change | |
| Subpopulation shift | group proportions | each group's distribution | a minority group is common at deployment |
Causal direction suggests which assumption is plausible (Schölkopf et al., 2012). When causes , the mechanism tends to be stable, so covariate shift is natural. When causes , as a disease causes symptoms, tends to be stable, so label shift is natural. The two are generally incompatible: if changes while stays put, Bayes' rule changes wherever the classes overlap.
Covariate shift: importance weighting
Assume and that wherever . With the importance weight , a change of measure turns target risk into a source expectation:
So the weighted average over source data is unbiased for . For a loss bounded by its variance is at most , and the second moment is a divergence:
Hence the effective sample size , estimated in practice by Kish's . For a mean shift of standard deviations in one Gaussian feature, the cost is exponential in :
A shift moves one Gaussian input feature's mean by 2 standard deviations. Out of 10,000 labeled source examples, what effective sample size remains after importance weighting?
Show the answer
180
. A one-standard-deviation shift keeps about 37%, two keep under 2%. Cortes, Mansour and Mohri (2010) prove learning bounds for importance weighting in terms of exactly this second moment, and show that guarantees fail when it is unbounded.
To estimate without estimating two densities, pool the source examples (label ) with unlabeled target inputs (label ) and train a probabilistic domain classifier. Bayes' rule converts its odds into the density ratio: . Kernel mean matching (Huang et al., 2007) and KLIEP (Sugiyama et al., 2008) estimate directly. Clipping the weights, or flattening them to with , trades bias for variance.
Label shift: estimating and correcting the prior
Under label shift, is shared. Bayes' rule in each domain gives the target posterior from the source posterior and the prior ratio :
So a calibrated source model is corrected by reweighting and renormalizing its outputs, a shift of the logit bias by . To estimate , note that is also shared. With the source joint confusion matrix and the target's predicted-label frequencies (Lipton, Wang & Smola, 2018):
This needs an invertible confusion matrix, not a calibrated model. In the binary case it is Forman's (2008) "adjusted count": a target flag rate inverts to
On source data a classifier has TPR 0.575 and FPR 0.095. It flags 12% of a large unlabeled target batch. Estimate the target prevalence, in percent.
Show the answer
5.2 %
. Taking the flag rate at face value, 12%, more than doubles the answer, because at low prevalence most flags are false positives.
The alternative is maximum likelihood by EM (Saerens, Latinne & Decaestecker, 2002): alternate between correcting the posteriors with the current prior estimate and resetting the prior to their mean. With well-calibrated models it is hard to beat (Alexandari, Kundaje & Shrikumar, 2020).
Bounds, invariance and robustness in practice
Let attain . Disagreement obeys the triangle inequality, since :
Move the disagreement to the source. Because , the change is controlled by the divergence:
Apply the triangle inequality on the source and collect terms:
The bound says that adaptation works when the source error is small, the domains look alike to the class's disagreement tests, and one hypothesis does well on both. It motivated domain-adversarial training (Ganin et al., 2016), which learns features on which a domain classifier fails. But cannot be estimated without target labels, and making feature distributions match can raise it: Zhao et al. (2019) proved that when label distributions differ, invariant representations force a lower bound on the joint error.
Empirically, ImageNet-C (Hendrycks & Dietterich, 2019) measures robustness to synthetic corruptions and WILDS (Koh et al., 2021) to real shifts across hospitals, locations and time. Across many models and natural shifts, out-of-distribution accuracy is close to a linear function of in-distribution accuracy after a probit transform, "accuracy on the line" (Miller et al., 2021); gains above that line, effective robustness (Taori et al., 2020), are rare, and the clearest came from training on much more diverse data, as in CLIP (Radford et al., 2021). On domain-generalization benchmarks, carefully tuned empirical risk minimization matched specialized algorithms (Gulrajani & Lopez-Paz, 2021). For subpopulation shift, group distributionally robust optimization minimizes the worst group's loss (Sagawa et al., 2020).
Key takeaways
- Covariate shift fixes , label shift fixes , concept shift changes ; causal direction suggests which assumption fits, and only concept shift is invisible without target labels.
- Importance weighting makes source risk unbiased for the target, but its variance scales with : a 2-SD Gaussian mean shift leaves about of the sample.
- Domain classifiers estimate density ratios; weighting matters most for misspecified models and cannot fix missing support.
- Under label shift, rescale calibrated posteriors by , and estimate from unlabeled data by inverting the confusion matrix (BBSE, adjusted count) or by EM.
- Ben-David et al.: . The unobservable limits invariant-representation methods, and in practice out-of-distribution accuracy tracks in-distribution accuracy ("accuracy on the line").
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
A diagnostic model maps symptoms to diseases. During an epidemic one disease becomes three times as common, but each disease's symptom profile is unchanged. What kind of shift is this, and what correction fits?
- Covariate shift; reweight training examples by
- Concept shift; nothing can be done without new labels
- Label shift; rescale calibrated posteriors by , estimating from unlabeled data if needed
- No shift, since the symptom profiles are unchanged
Show the answer
C. Label shift; rescale calibrated posteriors by , estimating from unlabeled data if needed
is fixed and changed: label shift, the natural case for an anticausal problem. Reweight and renormalize the posteriors, with from BBSE or EM.
Importance weighting gives an unbiased estimate of target risk. Why can it still fail badly?
- Its variance grows with , which can be enormous or infinite, leaving a tiny effective sample size
- It is biased whenever the model is misspecified
- It only works for classification losses
- It requires labels from the target domain
Show the answer
A. Its variance grows with , which can be enormous or infinite, leaving a tiny effective sample size
Unbiasedness says nothing about variance. A few examples with huge weights dominate the estimate; for a 2-standard-deviation Gaussian shift, . Without source support where the target has mass, the weights are infinite.
Which term of the bound cannot be estimated without labeled target data?
- Every term needs target labels
- , the best joint error over both domains
Show the answer
D. , the best joint error over both domains
uses source labels, and the divergence only asks whether classifiers disagree, which needs inputs alone. involves , which needs target labels, and aligning representations can silently increase it.
You have plenty of unlabeled data from the target domain. Which change could it never reveal?
- A change in the distribution of inputs
- A change in with unchanged
- A change in class prevalence under label shift
- A new cluster of inputs absent from the source
Show the answer
B. A change in with unchanged
Unlabeled data reveal only. If the inputs look identical while their labels change, every unlabeled test passes; labeled target data are the only way to see it.
Further reading
- S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira & J. W. Vaughan, "A theory of learning from different domains," Machine Learning 79, 151–175 (2010).
- H. Shimodaira, "Improving predictive inference under covariate shift by weighting the log-likelihood function," Journal of Statistical Planning and Inference 90, 227–244 (2000).
- Z. C. Lipton, Y.-X. Wang & A. Smola, "Detecting and correcting for label shift with black box predictors," ICML (2018).
- J. Quiñonero-Candela, M. Sugiyama, A. Schwaighofer & N. D. Lawrence (eds.), Dataset Shift in Machine Learning (MIT Press, 2009).
- P. W. Koh et al., "WILDS: A benchmark of in-the-wild distribution shifts," ICML (2021).