Simple Algorithm
Artificial Intelligence · Deep Layer · 26 min

Calibration & Uncertainty Quantification

When a model’s 80% should mean 80%.

A classifier that says 95% should be wrong about one time in twenty. Guo et al. (2017) found that the deep networks of their day were systematically more confident than accurate, and that one scalar, a softmax temperature, repaired most of it. This lesson makes calibration precise, shows why its standard estimator is biased in both directions, derives the main recalibration maps, and ends with conformal prediction: uncertainty sets with a finite-sample guarantee and no distributional assumptions.

What you’ll learn

  • Define canonical, classwise and top-label calibration, and prove the calibration–refinement decomposition of the Brier score.
  • Derive the finite-sample bias of binned ECE, and explain why binning also hides miscalibration.
  • Derive Platt scaling from a Gaussian score model, and compare it with isotonic regression and temperature scaling.
  • State and prove the coverage guarantee of split conformal prediction, and explain why conditional coverage needs assumptions.

Calibration, precisely

DerivationProof, and two consequences
  1. E[(p^−Y)2∣p^]=(p^−c)2+c(1−c)\E\big[(\hat p-Y)^2\mid\hat p\big]=(\hat p-c)^2+c(1-c)

    Given p^\hat p, the label is Bernoulli(c)\mathrm{Bernoulli}(c), so the conditional Brier score splits into a squared bias and a variance:

  2. E[(p^−Y)2]=E(p^−c)2+yˉ(1−yˉ)−Var⁡(c)\E\big[(\hat p-Y)^2\big]=\E(\hat p-c)^2+\bar y(1-\bar y)-\Var(c)

    Take expectations, and use E[c]=yˉ\E[c]=\bar y (tower rule), so that E[c(1−c)]=yˉ−E[c2]=yˉ(1−yˉ)−Var⁡(c)\E[c(1-c)]=\bar y-\E[c^2]=\bar y(1-\bar y)-\Var(c):

  3. Brier(c(p^))=Brier(p^)−E[(p^−c)2]\mathrm{Brier}\big(c(\hat p)\big)=\mathrm{Brier}(\hat p)-\E\big[(\hat p-c)^2\big]

    Replacing p^\hat p by c(p^)c(\hat p) gives a calibrated forecaster (E[Y∣c]=E[c∣c]=c\E[Y\mid c]=\E[c\mid c]=c) with the same cc, so recalibration removes the reliability term and leaves the rest: it never hurts. The constant forecast yˉ\bar y has zero reliability and zero resolution: calibrated and useless. Hence Gneiting, Balabdaoui and Raftery's maxim: maximize sharpness subject to calibration.

The same split holds for any proper scoring rule, with the rule's divergence in place of the squared gap. For log loss, E[ℓlog⁡]=E[DKL(c ∥ p^)]+E[H(c)]\E[\ell_{\log}]=\E\big[\KL(c\,\|\,\hat p)\big]+\E\big[H(c)\big]: the expected KL divergence from the calibration function, plus the expected entropy that no recalibration can remove.

Measuring it: reliability diagrams and the biases of ECE

The population calibration error is CE1=E∣c(X)−p^(X)∣\mathrm{CE}_1=\E\abs{c(X)-\hat p(X)}. It cannot be computed directly, because cc is unknown, so practice bins the predictions into BB intervals and uses ECE^=∑bnbn∣yˉb−pˉb∣\widehat{\mathrm{ECE}}=\sum_b\frac{n_b}{n}\abs{\bar y_b-\bar p_b}, the weighted gaps of the reliability diagram. Two biases pull in opposite directions.

DerivationUpward bias: noise looks like miscalibration
  1. yˉb−pˉb ≈ N(0, vbnb),vb=the mean of p^i(1−p^i) in bin b\bar y_b-\bar p_b\ \approx\ \mathcal N\Big(0,\ \frac{v_b}{n_b}\Big),\qquad v_b=\text{the mean of }\hat p_i(1-\hat p_i)\text{ in bin }b

    Let p^\hat p be perfectly calibrated. In bin bb, yˉb−pˉb\bar y_b-\bar p_b averages nbn_b independent mean-zero terms Yi−p^iY_i-\hat p_i with variances p^i(1−p^i)\hat p_i(1-\hat p_i):

  2. E[ECE^] ≈ 2π 1n∑bnb vb\E\big[\widehat{\mathrm{ECE}}\big]\ \approx\ \sqrt{\tfrac2\pi}\ \frac1n\sum_b\sqrt{n_b\,v_b}

    Since E∣N(0,σ2)∣=σ2/π\E\abs{\mathcal N(0,\sigma^2)}=\sigma\sqrt{2/\pi}, the estimator is positive on average although the true error is zero:

  3. E[ECE^] ≈ 2π8Bn ≈ 0.31Bn\E\big[\widehat{\mathrm{ECE}}\big]\ \approx\ \frac{\sqrt{2\pi}}{8}\sqrt{\frac Bn}\ \approx\ 0.31\sqrt{\frac Bn}

    For confidences spread uniformly over [0,1][0,1] and equal-width bins, nb≈n/Bn_b\approx n/B and ∑bvb≈B∫01p(1−p) dp=Bπ/8\sum_b\sqrt{v_b}\approx B\int_0^1\sqrt{p(1-p)}\,dp=B\pi/8:

Predict before you look

A perfectly calibrated model, with confidences spread uniformly, is evaluated on 1,000 examples with 15 equal-width bins. What binned ECE do you expect to see, in percent?

Show the answer

3.9 %

0.3115/1000≈0.31×0.122≈0.0380.31\sqrt{15/1000}\approx0.31\times0.122\approx0.038, and a simulation gives 3.9%. That is the size of ECEs often reported for "well-calibrated" models. At n=10,000n=10{,}000 it is still 1.2%. Small ECE differences on small test sets can be pure noise.

The downward bias comes from binning itself: the estimator measures the calibration of the binned forecaster, and inside a bin, over- and under-confidence cancel. Kumar, Liang and Ma (2019) showed that binned estimates can substantially underestimate the calibration error of continuous recalibrators such as Platt and temperature scaling, and proposed debiased estimators. The number also moves with BB and with equal-width versus equal-mass bins (Nixon et al., 2019; Roelofs et al., 2022). Report BB, nn and a bootstrap interval, prefer equal-mass bins, and always pair ECE with a proper score, because the constant forecaster wins on ECE alone.

Recalibration: Platt, isotonic and temperature scaling

Recalibration fits a map gg from scores to probabilities on a held-out calibration set, then reports g(s(x))g(s(\mathbf x)). Fitting on the training set would reuse data the model has already overfit, typically into overconfidence.

DerivationWhy Platt scaling is logistic
  1. log⁡Pr⁡(Y=1∣s)Pr⁡(Y=0∣s)=log⁡π1−π+(μ1−μ0) sσ2−μ12−μ022σ2=As+B\log\frac{\Pr(Y=1\mid s)}{\Pr(Y=0\mid s)}=\log\frac{\pi}{1-\pi}+\frac{(\mu_1-\mu_0)\,s}{\sigma^2}-\frac{\mu_1^2-\mu_0^2}{2\sigma^2}=As+B

    Suppose class-conditional scores are Gaussian with a common variance: s∣Y=y∼N(μy,σ2)s\mid Y=y\sim\mathcal N(\mu_y,\sigma^2). Bayes' rule gives log-odds that are linear in ss, because the quadratic terms cancel:

  2. Pr⁡(Y=1∣s)=σ(1.5 s−1.125+ln⁡37)=σ(1.5 s−1.97)\Pr(Y=1\mid s)=\sigma\big(1.5\,s-1.125+\ln\tfrac37\big)=\sigma(1.5\,s-1.97)

    So Pr⁡(Y=1∣s)=σ(As+B)\Pr(Y=1\mid s)=\sigma(As+B) exactly, and Platt (1999) fits A,BA,B by maximum likelihood: logistic regression on the single feature ss. In the metrics widget's model (μ0=0\mu_0=0, μ1=1.5\mu_1=1.5, σ=1\sigma=1, π=0.3\pi=0.3), the calibrated probability at the starting threshold s=0.75s=0.75 is exactly the prior, 0.30:

  3. B ⟼ B+log⁡π′1−π′−log⁡π1−πB\ \longmapsto\ B+\log\frac{\pi'}{1-\pi'}-\log\frac{\pi}{1-\pi}

    A new prevalence π′\pi' moves only the intercept, which is why label-shift correction is a bias shift (Distribution Shift & Robustness). With unequal variances the log-odds are quadratic in ss and the logistic form is misspecified; more flexible parametric maps such as beta calibration (Kull et al., 2017) address this.

text
ISOTONIC-CALIBRATE(scores s_1..s_n, labels y_1..y_n)      // pool adjacent violators
  sort the pairs by score
  blocks ← n blocks, block i holding value v_i = y_i and weight w_i = 1
  for each block, left to right:
      while its left neighbour has a larger value:
          merge the two: v ← (w_L v_L + w_R v_R) / (w_L + w_R),  w ← w_L + w_R
  return the step function mapping each score to the value of its block

Linear time after sorting. The output is the non-decreasing step function closest to the labels in squared error (Zadrozny & Elkan, 2002). It assumes only monotonicity, so it needs more data than Platt scaling and overfits small calibration sets (Niculescu-Mizil & Caruana, 2005).

Temperature scaling (Guo et al., 2017) rescales a KK-class network's logits: p^=softmax⁡(z/T)\hat{\mathbf p}=\softmax(\mathbf z/T), with one scalar T>0T>0 fit by minimizing the negative log-likelihood on the calibration set. Dividing by TT never changes the argmax, so accuracy is untouched, and the objective is convex in β=1/T\beta=1/T:

d2dβ2[−βzy+log⁡∑keβzk]=Var⁡k∼softmax⁡(βz)[zk] ≥ 0\frac{d^2}{d\beta^2}\Big[-\beta z_y+\log\sum_{k}e^{\beta z_k}\Big]=\Var_{k\sim\softmax(\beta\mathbf z)}\big[z_k\big]\ \ge\ 0
Guo et al. found ResNets and DenseNets overconfident (T>1T>1 fixes it), worse for deeper and wider models, with batch normalization and less weight decay; temperature scaling matched or beat more complex recalibration methods on most of their benchmarks. Minderer et al. (2021) found newer architectures such as vision transformers better calibrated. Calibration still degrades under distribution shift (Ovadia et al., 2019).
Check your understanding

Validation log-likelihood is maximized at T=1.8T=1.8. What does temperature scaling with T=1.8T=1.8 change?

  1. It raises top-1 accuracy by correcting overconfident mistakes
  2. It changes the predicted class of the least confident inputs
  3. Only the probabilities: every predicted class stays the same while confidences move toward uniform
  4. Nothing, since temperature cancels in the softmax
Show the answer

C. Only the probabilities: every predicted class stays the same while confidences move toward uniform

arg max⁡kzk/T=arg max⁡kzk\argmax_k z_k/T=\argmax_k z_k for any T>0T>0, so predictions and accuracy are unchanged. With T>1T>1 the logits are compressed, so each maximum probability shrinks toward 1/K1/K, repairing overconfidence. (In the binary case the map is monotone in the score, so AUC is unchanged too.)

Conformal prediction: coverage without assumptions

Calibration is an average property of probabilities, and nothing guarantees it for a given model, least of all after a shift. Conformal prediction instead wraps any model in a set-valued predictor C(x)C(\mathbf x) with guaranteed coverage Pr⁡(Y∈C(X))≥1−α\Pr\big(Y\in C(X)\big)\ge1-\alpha, for every model and every distribution, assuming only exchangeability.

text
SPLIT-CONFORMAL(model p̂, calibration pairs (x_i, y_i) for i = 1..n, level α)
  for i = 1..n:  s_i ← 1 − p̂(y_i | x_i)          // nonconformity: high = surprising
  k ← ⌈(n + 1)(1 − α)⌉                           // if k > n, return every label
  q̂ ← the k-th smallest of s_1..s_n
  return C(x) = { y : 1 − p̂(y | x) ≤ q̂ }

The model is trained on separate data; only the nn calibration points are used here.

DerivationProof
  1. Pr⁡(rank⁡(sn+1)≤k)=kn+1\Pr\big(\operatorname{rank}(s_{n+1})\le k\big)=\frac{k}{n+1}

    Let sn+1=s(Xn+1,Yn+1)s_{n+1}=s(X_{n+1},Y_{n+1}). The scores s1,…,sn+1s_1,\dots,s_{n+1} are exchangeable, so with no ties the rank of sn+1s_{n+1} among all n+1n+1 of them is uniform on {1,…,n+1}\{1,\dots,n+1\}:

  2. Pr⁡(Yn+1∈C(Xn+1))=⌈(n+1)(1−α)⌉n+1 ∈ [1−α, 1−α+1n+1)\Pr\big(Y_{n+1}\in C(X_{n+1})\big)=\frac{\lceil(n+1)(1-\alpha)\rceil}{n+1}\ \in\ \Big[1-\alpha,\ 1-\alpha+\frac1{n+1}\Big)

    Yn+1∈C(Xn+1)Y_{n+1}\in C(X_{n+1}) iff sn+1≤q^s_{n+1}\le\hat q, the kk-th smallest of the first nn scores, iff sn+1s_{n+1} is among the kk smallest of all n+1n+1. With k=⌈(n+1)(1−α)⌉k=\lceil(n+1)(1-\alpha)\rceil:

  3. validity: any modelefficiency (set size): a good model\text{validity: any model}\qquad\text{efficiency (set size): a good model}

    Ties only add cases with sn+1≤q^s_{n+1}\le\hat q, so the lower bound survives. Nothing was assumed about p^\hat p: a poor model still gives valid coverage, paid for with larger sets.

The guarantee is marginal, averaged over XX. Exact conditional coverage, Pr⁡(Y∈C(X)∣X=x)≥1−α\Pr(Y\in C(X)\mid X=\mathbf x)\ge1-\alpha for every x\mathbf x, cannot be achieved distribution-free in finite samples except with trivially large sets (Vovk, 2012; Barber, Candès, Ramdas & Tibshirani, 2021). Coverage within known groups (Mondrian conformal), adaptive scores (Romano, Sesia & Candès, 2020) and weighting for known covariate shift (Tibshirani et al., 2019) narrow the gap.

Key takeaways

  • Calibration means E[Y∣p^]=p^\E[Y\mid\hat p]=\hat p. Canonical calibration implies classwise and top-label calibration, and most reported ECEs are top-label only.
  • Brier = reliability − resolution + uncertainty. Recalibration removes the reliability term and never hurts a proper score; the constant forecast is calibrated and useless.
  • Binned ECE is biased upward by noise (about 0.31B/n0.31\sqrt{B/n} for a perfect model) and downward by within-bin cancellation: report BB, nn, intervals and a proper score.
  • Platt scaling is exact for equal-variance Gaussian scores; isotonic regression (PAV) is nonparametric and data-hungry; temperature scaling fits one scalar, keeps the argmax and is convex in 1/T1/T.
  • Split conformal sets cover with probability in [1−α, 1−α+1n+1)[1-\alpha,\,1-\alpha+\tfrac1{n+1}) under exchangeability alone; conditional coverage needs assumptions.

Check yourself

In the interactive lesson these are scored with your confidence, and what you miss comes back for review.

Question 1

A forecaster's calibration function is c(p^)=E[Y∣p^]c(\hat p)=\E[Y\mid\hat p]. What happens to its Brier score if you replace p^\hat p by c(p^)c(\hat p)?

  1. It can go up or down, depending on the base rate
  2. It falls by exactly E[(p^−c)2]\E\big[(\hat p-c)^2\big]; the resolution and uncertainty terms are unchanged
  3. It falls to zero
  4. It is unchanged, because the Brier score only measures discrimination
Show the answer

B. It falls by exactly E[(p^−c)2]\E\big[(\hat p-c)^2\big]; the resolution and uncertainty terms are unchanged

c(p^)c(\hat p) is calibrated and has the same calibration function, so the decomposition loses its reliability term and keeps −Var⁡(c)+yˉ(1−yˉ)-\Var(c)+\bar y(1-\bar y).

Question 2

Model A has ECE 2.1% and model B 3.4%, each measured with 15 bins on the same 1,000 test examples. What can you conclude?

  1. A is better calibrated, since its ECE is lower
  2. B is miscalibrated, since 3.4% exceeds the usual 3% tolerance
  3. Little: a perfectly calibrated model would show about 3.9% from noise alone, and binning can also hide miscalibration
  4. Both are perfectly calibrated, since both are below 5%
Show the answer

C. Little: a perfectly calibrated model would show about 3.9% from noise alone, and binning can also hide miscalibration

The noise floor 0.31B/n≈3.8%0.31\sqrt{B/n}\approx3.8\% is larger than the difference. Compare debiased estimates with bootstrap intervals, and a proper score such as log loss.

Question 3

For which class-conditional score distributions is Platt scaling, Pr⁡(Y=1∣s)=σ(As+B)\Pr(Y=1\mid s)=\sigma(As+B), exactly correct?

  1. Any distributions, as long as the classifier is accurate
  2. Gaussians with different variances
  3. Uniform distributions on overlapping intervals
  4. Gaussians with a common variance (more generally, whenever the log-likelihood ratio is linear in ss)
Show the answer

D. Gaussians with a common variance (more generally, whenever the log-likelihood ratio is linear in ss)

With equal variances the quadratic terms cancel in the log-likelihood ratio, leaving As+BAs+B. Unequal variances leave a quadratic, and the logistic map is misspecified.

Question 4

Split conformal prediction with n=99n=99 calibration points and α=0.1\alpha=0.1, scores almost surely distinct. Which order statistic is q^\hat q, and what is the coverage?

  1. The 89th smallest score; coverage 0.89
  2. The 90th smallest score; coverage exactly 0.90
  3. The 99th smallest score; coverage 0.99
  4. The 10th smallest score; coverage 0.90
Show the answer

B. The 90th smallest score; coverage exactly 0.90

k=⌈100×0.9⌉=90k=\lceil100\times0.9\rceil=90, and coverage is k/(n+1)=90/100k/(n+1)=90/100: exactly 1−α1-\alpha here, inside [0.9, 0.9+1100)[0.9,\,0.9+\tfrac1{100}).

Further reading

  • C. Guo, G. Pleiss, Y. Sun & K. Q. Weinberger, "On calibration of modern neural networks," ICML (2017).
  • J. Platt, "Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods," in Advances in Large Margin Classifiers (MIT Press, 1999).
  • B. Zadrozny & C. Elkan, "Transforming classifier scores into accurate multiclass probability estimates," KDD (2002).
  • A. Kumar, P. Liang & T. Ma, "Verified uncertainty calibration," NeurIPS (2019).
  • A. N. Angelopoulos & S. Bates, "Conformal prediction: A gentle introduction," Foundations and Trends in Machine Learning (2023).