Calibration & Uncertainty Quantification
When a model’s 80% should mean 80%.
A classifier that says 95% should be wrong about one time in twenty. Guo et al. (2017) found that the deep networks of their day were systematically more confident than accurate, and that one scalar, a softmax temperature, repaired most of it. This lesson makes calibration precise, shows why its standard estimator is biased in both directions, derives the main recalibration maps, and ends with conformal prediction: uncertainty sets with a finite-sample guarantee and no distributional assumptions.
What you’ll learn
- Define canonical, classwise and top-label calibration, and prove the calibration–refinement decomposition of the Brier score.
- Derive the finite-sample bias of binned ECE, and explain why binning also hides miscalibration.
- Derive Platt scaling from a Gaussian score model, and compare it with isotonic regression and temperature scaling.
- State and prove the coverage guarantee of split conformal prediction, and explain why conditional coverage needs assumptions.
Calibration, precisely
Given , the label is , so the conditional Brier score splits into a squared bias and a variance:
Take expectations, and use (tower rule), so that :
Replacing by gives a calibrated forecaster () with the same , so recalibration removes the reliability term and leaves the rest: it never hurts. The constant forecast has zero reliability and zero resolution: calibrated and useless. Hence Gneiting, Balabdaoui and Raftery's maxim: maximize sharpness subject to calibration.
The same split holds for any proper scoring rule, with the rule's divergence in place of the squared gap. For log loss, : the expected KL divergence from the calibration function, plus the expected entropy that no recalibration can remove.
Measuring it: reliability diagrams and the biases of ECE
The population calibration error is . It cannot be computed directly, because is unknown, so practice bins the predictions into intervals and uses , the weighted gaps of the reliability diagram. Two biases pull in opposite directions.
Let be perfectly calibrated. In bin , averages independent mean-zero terms with variances :
Since , the estimator is positive on average although the true error is zero:
For confidences spread uniformly over and equal-width bins, and :
A perfectly calibrated model, with confidences spread uniformly, is evaluated on 1,000 examples with 15 equal-width bins. What binned ECE do you expect to see, in percent?
Show the answer
3.9 %
, and a simulation gives 3.9%. That is the size of ECEs often reported for "well-calibrated" models. At it is still 1.2%. Small ECE differences on small test sets can be pure noise.
The downward bias comes from binning itself: the estimator measures the calibration of the binned forecaster, and inside a bin, over- and under-confidence cancel. Kumar, Liang and Ma (2019) showed that binned estimates can substantially underestimate the calibration error of continuous recalibrators such as Platt and temperature scaling, and proposed debiased estimators. The number also moves with and with equal-width versus equal-mass bins (Nixon et al., 2019; Roelofs et al., 2022). Report , and a bootstrap interval, prefer equal-mass bins, and always pair ECE with a proper score, because the constant forecaster wins on ECE alone.
Recalibration: Platt, isotonic and temperature scaling
Recalibration fits a map from scores to probabilities on a held-out calibration set, then reports . Fitting on the training set would reuse data the model has already overfit, typically into overconfidence.
Suppose class-conditional scores are Gaussian with a common variance: . Bayes' rule gives log-odds that are linear in , because the quadratic terms cancel:
So exactly, and Platt (1999) fits by maximum likelihood: logistic regression on the single feature . In the metrics widget's model (, , , ), the calibrated probability at the starting threshold is exactly the prior, 0.30:
A new prevalence moves only the intercept, which is why label-shift correction is a bias shift (Distribution Shift & Robustness). With unequal variances the log-odds are quadratic in and the logistic form is misspecified; more flexible parametric maps such as beta calibration (Kull et al., 2017) address this.
ISOTONIC-CALIBRATE(scores s_1..s_n, labels y_1..y_n) // pool adjacent violators
sort the pairs by score
blocks ← n blocks, block i holding value v_i = y_i and weight w_i = 1
for each block, left to right:
while its left neighbour has a larger value:
merge the two: v ← (w_L v_L + w_R v_R) / (w_L + w_R), w ← w_L + w_R
return the step function mapping each score to the value of its blockLinear time after sorting. The output is the non-decreasing step function closest to the labels in squared error (Zadrozny & Elkan, 2002). It assumes only monotonicity, so it needs more data than Platt scaling and overfits small calibration sets (Niculescu-Mizil & Caruana, 2005).
Temperature scaling (Guo et al., 2017) rescales a -class network's logits: , with one scalar fit by minimizing the negative log-likelihood on the calibration set. Dividing by never changes the argmax, so accuracy is untouched, and the objective is convex in :
Validation log-likelihood is maximized at . What does temperature scaling with change?
- It raises top-1 accuracy by correcting overconfident mistakes
- It changes the predicted class of the least confident inputs
- Only the probabilities: every predicted class stays the same while confidences move toward uniform
- Nothing, since temperature cancels in the softmax
Show the answer
C. Only the probabilities: every predicted class stays the same while confidences move toward uniform
for any , so predictions and accuracy are unchanged. With the logits are compressed, so each maximum probability shrinks toward , repairing overconfidence. (In the binary case the map is monotone in the score, so AUC is unchanged too.)
Conformal prediction: coverage without assumptions
Calibration is an average property of probabilities, and nothing guarantees it for a given model, least of all after a shift. Conformal prediction instead wraps any model in a set-valued predictor with guaranteed coverage , for every model and every distribution, assuming only exchangeability.
SPLIT-CONFORMAL(model p̂, calibration pairs (x_i, y_i) for i = 1..n, level α)
for i = 1..n: s_i ← 1 − p̂(y_i | x_i) // nonconformity: high = surprising
k ← ⌈(n + 1)(1 − α)⌉ // if k > n, return every label
q̂ ← the k-th smallest of s_1..s_n
return C(x) = { y : 1 − p̂(y | x) ≤ q̂ }The model is trained on separate data; only the calibration points are used here.
Let . The scores are exchangeable, so with no ties the rank of among all of them is uniform on :
iff , the -th smallest of the first scores, iff is among the smallest of all . With :
Ties only add cases with , so the lower bound survives. Nothing was assumed about : a poor model still gives valid coverage, paid for with larger sets.
The guarantee is marginal, averaged over . Exact conditional coverage, for every , cannot be achieved distribution-free in finite samples except with trivially large sets (Vovk, 2012; Barber, Candès, Ramdas & Tibshirani, 2021). Coverage within known groups (Mondrian conformal), adaptive scores (Romano, Sesia & Candès, 2020) and weighting for known covariate shift (Tibshirani et al., 2019) narrow the gap.
Key takeaways
- Calibration means . Canonical calibration implies classwise and top-label calibration, and most reported ECEs are top-label only.
- Brier = reliability − resolution + uncertainty. Recalibration removes the reliability term and never hurts a proper score; the constant forecast is calibrated and useless.
- Binned ECE is biased upward by noise (about for a perfect model) and downward by within-bin cancellation: report , , intervals and a proper score.
- Platt scaling is exact for equal-variance Gaussian scores; isotonic regression (PAV) is nonparametric and data-hungry; temperature scaling fits one scalar, keeps the argmax and is convex in .
- Split conformal sets cover with probability in under exchangeability alone; conditional coverage needs assumptions.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
A forecaster's calibration function is . What happens to its Brier score if you replace by ?
- It can go up or down, depending on the base rate
- It falls by exactly ; the resolution and uncertainty terms are unchanged
- It falls to zero
- It is unchanged, because the Brier score only measures discrimination
Show the answer
B. It falls by exactly ; the resolution and uncertainty terms are unchanged
is calibrated and has the same calibration function, so the decomposition loses its reliability term and keeps .
Model A has ECE 2.1% and model B 3.4%, each measured with 15 bins on the same 1,000 test examples. What can you conclude?
- A is better calibrated, since its ECE is lower
- B is miscalibrated, since 3.4% exceeds the usual 3% tolerance
- Little: a perfectly calibrated model would show about 3.9% from noise alone, and binning can also hide miscalibration
- Both are perfectly calibrated, since both are below 5%
Show the answer
C. Little: a perfectly calibrated model would show about 3.9% from noise alone, and binning can also hide miscalibration
The noise floor is larger than the difference. Compare debiased estimates with bootstrap intervals, and a proper score such as log loss.
For which class-conditional score distributions is Platt scaling, , exactly correct?
- Any distributions, as long as the classifier is accurate
- Gaussians with different variances
- Uniform distributions on overlapping intervals
- Gaussians with a common variance (more generally, whenever the log-likelihood ratio is linear in )
Show the answer
D. Gaussians with a common variance (more generally, whenever the log-likelihood ratio is linear in )
With equal variances the quadratic terms cancel in the log-likelihood ratio, leaving . Unequal variances leave a quadratic, and the logistic map is misspecified.
Split conformal prediction with calibration points and , scores almost surely distinct. Which order statistic is , and what is the coverage?
- The 89th smallest score; coverage 0.89
- The 90th smallest score; coverage exactly 0.90
- The 99th smallest score; coverage 0.99
- The 10th smallest score; coverage 0.90
Show the answer
B. The 90th smallest score; coverage exactly 0.90
, and coverage is : exactly here, inside .
Further reading
- C. Guo, G. Pleiss, Y. Sun & K. Q. Weinberger, "On calibration of modern neural networks," ICML (2017).
- J. Platt, "Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods," in Advances in Large Margin Classifiers (MIT Press, 1999).
- B. Zadrozny & C. Elkan, "Transforming classifier scores into accurate multiclass probability estimates," KDD (2002).
- A. Kumar, P. Liang & T. Ma, "Verified uncertainty calibration," NeurIPS (2019).
- A. N. Angelopoulos & S. Bates, "Conformal prediction: A gentle introduction," Foundations and Trends in Machine Learning (2023).