Simple Algorithm
Artificial Intelligence · Hidden Layer · 20 min

Evaluating Models

Metrics, ROC curves, calibration and leakage.

A fraud detector that flags nothing scores 99.8% accuracy when 0.2% of payments are fraud. A model with an excellent ROC curve can still bury every real alarm under nine false ones. Evaluation is a discipline of its own: pick the number that matches the decision, read the curves behind it, check that probabilities mean what they say, and make sure nothing leaked from the test set.

What you’ll learn

  • Compute confusion-matrix metrics and derive how precision depends on prevalence.
  • Prove that AUC is the probability that a random positive outranks a random negative, and compute it from ranks.
  • Choose a decision threshold from costs and prevalence, and decide when precision–recall curves beat ROC curves.
  • Prove that log loss and the Brier score are strictly proper, and read a reliability diagram.
  • Put an interval on a test score and recognize the common forms of data leakage.

Confusion matrices and the base rate

A binary classifier computes a score s(x)s(\mathbf x) and flags x\mathbf x when s(x)>ts(\mathbf x)>t. On a labeled test set it produces four counts, true and false positives and negatives: the confusion matrix. Every common metric is a ratio of these counts, and the ratios fall into two families: rates conditioned on the true class, and rates conditioned on the prediction.

MetricFormulaConditions onChanges with prevalence?
Recall (TPR, sensitivity)TP/(TP+FN)\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})actual positivesno
False positive rate (FPR)FP/(FP+TN)\mathrm{FP}/(\mathrm{FP}+\mathrm{TN})actual negativesno
Specificity (TNR)1−FPR1-\mathrm{FPR}actual negativesno
Precision (PPV)TP/(TP+FP)\mathrm{TP}/(\mathrm{TP}+\mathrm{FP})predicted positivesyes
Accuracy(TP+TN)/n(\mathrm{TP}+\mathrm{TN})/neverythingyes
Balanced accuracy(TPR+TNR)/2(\mathrm{TPR}+\mathrm{TNR})/2each class, averagedno
F12TP/(2TP+FP+FN)2\mathrm{TP}/(2\mathrm{TP}+\mathrm{FP}+\mathrm{FN})ignores TNyes
Class-conditional rates describe the classifier. Prediction-conditional rates describe the classifier in a particular population.
DerivationPrecision is Bayes’ rule
  1. precision=Pr⁡(Y=1∣Y^=1)=π TPRπ TPR+(1−π) FPR\text{precision}=\Pr(Y=1\mid\hat Y=1)=\frac{\pi\,\mathrm{TPR}}{\pi\,\mathrm{TPR}+(1-\pi)\,\mathrm{FPR}}

    Write π=Pr⁡(Y=1)\pi=\Pr(Y=1) for the prevalence. A flagged case is a real positive with probability

  2. precision1−precision=π1−π⋅TPRFPR\frac{\text{precision}}{1-\text{precision}}=\frac{\pi}{1-\pi}\cdot\frac{\mathrm{TPR}}{\mathrm{FPR}}

    In odds form, the classifier enters only through its likelihood ratio TPR/FPR\mathrm{TPR}/\mathrm{FPR} and the population only through the prior odds:

  3. precision≈π⋅TPRFPR  (π small),acc(flag nothing)=1−π\text{precision}\approx\pi\cdot\frac{\mathrm{TPR}}{\mathrm{FPR}}\ \ (\pi\ \text{small}),\qquad\mathrm{acc}(\text{flag nothing})=1-\pi

    As π→0\pi\to0 with the classifier fixed, precision falls in proportion to π\pi: rare positives drown in false alarms, the base-rate fallacy in metric form. Meanwhile the classifier that flags nothing scores accuracy 1−π1-\pi.

Predict before you look

A classifier has TPR 0.575 and FPR 0.095, a likelihood ratio of about 6. It is deployed where 2% of cases are positive. What fraction of its flags are real positives?

Show the answer

11 %

0.02×0.5750.02×0.575+0.98×0.095=0.01150.1046≈11%\dfrac{0.02\times0.575}{0.02\times0.575+0.98\times0.095}=\dfrac{0.0115}{0.1046}\approx11\%. Nine in ten alarms are false. At 30% prevalence the same classifier has precision 72%: nothing about the model changed, only the population.

ROC curves and what AUC measures

Sweep the threshold tt from +∞+\infty down to −∞-\infty and trace (FPR(t),TPR(t))\big(\mathrm{FPR}(t),\mathrm{TPR}(t)\big) from (0,0)(0,0) to (1,1)(1,1): the ROC curve. Both coordinates condition on the true class, so the curve does not depend on prevalence. It uses only comparisons s>ts>t, so it is unchanged by any strictly increasing transformation of the scores. The area under it, the AUC, has a clean probabilistic meaning.

DerivationProof, and how to compute it
  1. AUC=∫01TPR  d FPR=∫−∞∞Pr⁡(S+>t) f−(t) dt=Pr⁡(S+>S−)\mathrm{AUC}=\int_0^1\mathrm{TPR}\;d\,\mathrm{FPR}=\int_{-\infty}^{\infty}\Pr(S^+>t)\,f_-(t)\,dt=\Pr(S^+>S^-)

    For continuous scores with densities f±f_\pm, TPR(t)=Pr⁡(S+>t)\mathrm{TPR}(t)=\Pr(S^+>t) and FPR(t)=Pr⁡(S−>t)\mathrm{FPR}(t)=\Pr(S^->t), so d FPR=−f−(t) dtd\,\mathrm{FPR}=-f_-(t)\,dt. As tt falls from +∞+\infty to −∞-\infty, FPR rises from 0 to 1:

  2. AUC^=1n+n−∑i: yi=1 ∑j: yj=0(1[si>sj]+121[si=sj])=R+−n+(n++1)/2n+n−\widehat{\mathrm{AUC}}=\frac{1}{n_+n_-}\sum_{i:\,y_i=1}\ \sum_{j:\,y_j=0}\Big(\mathbf 1[s_i>s_j]+\tfrac12\mathbf 1[s_i=s_j]\Big)=\frac{R_+-n_+(n_++1)/2}{n_+n_-}

    On data, sort all n=n++n−n=n_++n_- scores and let R+R_+ be the sum of the positives' ranks (average ranks for ties). The positives' ranks count every score below them, including the n+(n+−1)/2n_+(n_+-1)/2 positive–positive comparisons plus one for themselves; subtract those:

  3. AUC=Pr⁡(S+−S−>0)=Φ(d/2),d=1.5 ⇒ Φ(1.061)≈0.856\mathrm{AUC}=\Pr\big(S^+-S^->0\big)=\Phi\big(d/\sqrt2\big),\qquad d=1.5\ \Rightarrow\ \Phi(1.061)\approx0.856

    Sorting makes this O(nlog⁡n)\order{n\log n} instead of O(n+n−)\order{n_+n_-}. In the binormal model, negatives score N(0,1)\mathcal N(0,1) and positives N(d,1)\mathcal N(d,1), so S+−S−∼N(d,2)S^+-S^-\sim\mathcal N(d,2):

python
import numpy as np
from scipy.stats import rankdata

def auc(scores, labels):
    """Mann–Whitney AUC in O(n log n); tied scores get average ranks."""
    labels = np.asarray(labels, dtype=bool)
    ranks = rankdata(scores)                    # 1 = lowest score
    n_pos, n_neg = labels.sum(), (~labels).sum()
    return (ranks[labels].sum() - n_pos * (n_pos + 1) / 2) / (n_pos * n_neg)

The same number scikit-learn's roc_auc_score returns for binary labels.

Check your understanding

Positives score 0.9,0.7,0.40.9, 0.7, 0.4; negatives score 0.8,0.3,0.2,0.10.8, 0.3, 0.2, 0.1. What is the AUC?

  1. 5/7≈0.715/7\approx0.71
  2. 11
  3. 10/12≈0.8310/12\approx0.83
  4. 2/3≈0.672/3\approx0.67
Show the answer

C. 10/12≈0.8310/12\approx0.83

There are 3×4=123\times4=12 positive–negative pairs. 0.90.9 beats all four negatives, 0.70.7 and 0.40.4 beat three each (not 0.80.8): 1010 correct pairs, so AUC=10/12\mathrm{AUC}=10/12. By ranks: the positives sit at ranks 4,5,74,5,7, so R+=16R_+=16 and (16−6)/12=10/12(16-6)/12=10/12.

Precision–recall curves and choosing a threshold

Prevalence-invariance makes ROC curves good for comparing rankers and misleading for deployment. The precision–recall (PR) curve plots precision against recall over all thresholds. Random scoring gives a flat line at precision π\pi, and the area, average precision (AP), falls with prevalence even when the ranking is unchanged. Davis and Goadrich (2006) proved that one classifier's curve dominates another's in ROC space if and only if it does in PR space, so the two agree on which classifier is uniformly better; they differ in what they magnify. With rare positives, the informative part of the ROC curve is squashed against FPR≈0\mathrm{FPR}\approx0, and the PR curve spreads it out.

DerivationThe cost-optimal threshold
  1. p cFN>(1−p) cFP  ⟺  p>cFPcFP+cFNp\,c_{FN}>(1-p)\,c_{FP}\iff p>\frac{c_{FP}}{c_{FP}+c_{FN}}

    Let a false positive cost cFPc_{FP} and a false negative cFNc_{FN}, and suppose the model reports a calibrated p=Pr⁡(Y=1∣x)p=\Pr(Y=1\mid\mathbf x). Flagging costs (1−p) cFP(1-p)\,c_{FP} in expectation and not flagging costs p cFNp\,c_{FN}. Flag when

  2. s>t∗=d2+1dln⁡(1−π) cFPπ cFNs>t^*=\frac d2+\frac1d\ln\frac{(1-\pi)\,c_{FP}}{\pi\,c_{FN}}

    Equal costs give 12\tfrac12; if a miss is ten times worse than a false alarm, flag above 111≈0.09\tfrac1{11}\approx0.09. In the binormal model the log-likelihood ratio is linear in the score, ln⁡φ(s−d)φ(s)=ds−d22\ln\frac{\varphi(s-d)}{\varphi(s)}=ds-\tfrac{d^2}2, so the rule becomes a score threshold that moves with prevalence:

  3. tπ=0.3∗=0.75+ln⁡(7/3)1.5≈1.31,tπ=0.02∗=0.75+ln⁡491.5≈3.34t^*_{\pi=0.3}=0.75+\frac{\ln(7/3)}{1.5}\approx1.31,\qquad t^*_{\pi=0.02}=0.75+\frac{\ln49}{1.5}\approx3.34

    With d=1.5d=1.5 and equal costs, t∗=1.31t^*=1.31 at 30% prevalence (accuracy 0.806, the widget's maximum) but 3.343.34 at 2%. A threshold tuned on balanced validation data is wrong for a rare-positive deployment.

Think first When should you report ROC AUC, and when average precision?

ROC AUC when ranking quality should not depend on the population: comparing models across sites with different prevalence, or when both classes matter symmetrically. AP, or precision at the top kk, when positives are rare and what matters is the head of the ranking: retrieval, fraud queues, screening with limited follow-up capacity. When unsure, report both, together with the operating point you will actually use.

Proper scoring rules and calibration

A model that outputs probabilities should be judged on the probabilities, not just their order. A scoring rule ℓ(p,y)\ell(p,y) charges a forecast p∈[0,1]p\in[0,1] when the outcome y∈{0,1}y\in\{0,1\} occurs. It is proper if a forecaster who believes Y∼Bernoulli(q)Y\sim\mathrm{Bernoulli}(q) minimizes the expected charge by reporting p=qp=q, and strictly proper if qq is the only minimizer. Proper rules make honesty the best policy. Training with one is what makes predicted probabilities meaningful: log loss is the cross-entropy minimized by logistic regression and softmax networks.

DerivationProof
  1. Eq[(p−Y)2]=p2−2pq+q=(p−q)2+q(1−q)\E_q\big[(p-Y)^2\big]=p^2-2pq+q=(p-q)^2+q(1-q)

    Brier: use E[Y]=E[Y2]=q\E[Y]=\E[Y^2]=q and complete the square. The last term does not depend on pp.

  2. Eq[ℓlog⁡(p,Y)]=−qln⁡p−(1−q)ln⁡(1−p)=H(q)+DKL(q ∥ p)\E_q\big[\ell_{\log}(p,Y)\big]=-q\ln p-(1-q)\ln(1-p)=H(q)+\KL(q\,\|\,p)

    Log loss: add and subtract the binary entropy H(q)H(q):

  3. DKL(q ∥ p)=−qln⁡pq−(1−q)ln⁡1−p1−q ≥ q(1−pq)+(1−q)(1−1−p1−q)=0\KL(q\,\|\,p)=-q\ln\tfrac pq-(1-q)\ln\tfrac{1-p}{1-q}\ \ge\ q\big(1-\tfrac pq\big)+(1-q)\big(1-\tfrac{1-p}{1-q}\big)=0

    Gibbs' inequality, from ln⁡x≤x−1\ln x\le x-1, shows DKL≥0\KL\ge0, with equality only when p=qp=q:

Check your understanding

Which scoring rule rewards a forecaster for exaggerating, reporting 0 or 1 instead of an honest probability?

  1. The Brier score (p−y)2(p-y)^2
  2. The absolute error ∣p−y∣\abs{p-y}
  3. The log loss
  4. None: any loss that is zero for a perfect forecast is proper
Show the answer

B. The absolute error ∣p−y∣\abs{p-y}

Eq∣p−Y∣=q(1−p)+(1−q)p=q+p(1−2q)\E_q\abs{p-Y}=q(1-p)+(1-q)p=q+p(1-2q) is linear in pp: it is minimized at p=1p=1 when q>12q>\tfrac12 and at p=0p=0 when q<12q<\tfrac12. Scored by absolute error, a forecaster should round every belief to 0 or 1. (It is the right loss for predicting a median, not a probability.)

Calibration asks that probabilities match frequencies: among the cases where the model says 0.8, about 80% should be positive. Formally, Pr⁡(Y=1∣p^(X)=p)=p\Pr\big(Y=1\mid\hat p(X)=p\big)=p for every pp. A reliability diagram bins the predictions (say ten bins of width 0.1) and plots each bin's observed positive rate against its mean prediction; perfect calibration is the diagonal. The expected calibration error summarizes the gaps: ECE=∑bnbn∣yˉb−pˉb∣\mathrm{ECE}=\sum_b\frac{n_b}{n}\abs{\bar y_b-\bar p_b}.

Honest numbers: intervals and leakage

A test score is an estimate. If the nn test examples are drawn i.i.d. and the model's true accuracy is aa, test accuracy is a binomial proportion with standard error a(1−a)/n\sqrt{a(1-a)/n}: 3 points at n=100n=100 and a=0.9a=0.9, 0.3 points at n=10,000n=10{,}000. The 95% Wilson interval behaves well even near 0 and 1:

a^+z22n ± za^(1−a^)n+z24n21+z2n,z=1.96\frac{\hat a+\frac{z^2}{2n}\ \pm\ z\sqrt{\frac{\hat a(1-\hat a)}{n}+\frac{z^2}{4n^2}}}{1+\frac{z^2}{n}},\qquad z=1.96
90 correct out of 100 gives [82.6%, 94.5%][82.6\%,\,94.5\%]. Distribution-free (Hoeffding) intervals, and what reusing a test set does to them, are derived in Measuring Intelligence & AGI.

Intervals cover sampling noise only. The more dangerous failure is leakage: information about the test labels reaching the model or the modeling decisions. Kapoor and Narayanan (2023) traced leakage through hundreds of published machine-learning studies in more than a dozen fields.

LeakExampleFix
Preprocessing or selection on all datascaling, imputation or feature screening fitted before the splitrefit every data-dependent step inside each fold (see Generalization & Overfitting)
Duplicates and non-independencethe same patient, user or near-duplicate image on both sides of the splitsplit by group: patient, user, document
Temporal leakagerandom splits of time-ordered data; features computed with future informationtrain on the past, test on the future
Target leakagea feature recorded after, or because of, the outcome, such as a treatment code when predicting the diagnosisaudit when and how each feature is generated
Test-set reusechoosing models or hyperparameters by test scorea validation set; one final test

Key takeaways

  • TPR and FPR describe the classifier; precision and accuracy also describe the population. Precision odds equal prior odds times TPR/FPR\mathrm{TPR}/\mathrm{FPR}, so precision collapses as positives become rare.
  • ROC curves ignore prevalence and monotone rescaling, and AUC=Pr⁡(S+>S−)\mathrm{AUC}=\Pr(S^+>S^-) is computable from ranks in O(nlog⁡n)\order{n\log n}. PR curves and average precision show performance on rare positives.
  • The cost-optimal decision flags when p>cFP/(cFP+cFN)p>c_{FP}/(c_{FP}+c_{FN}); the matching score threshold moves with prevalence.
  • Log loss and the Brier score are strictly proper (excess DKL(q ∥ p)\KL(q\,\|\,p) and (p−q)2(p-q)^2). Calibration, Pr⁡(Y=1∣p^=p)=p\Pr(Y=1\mid\hat p=p)=p, is a separate axis from discrimination.
  • A test score needs an interval (about ±6\pm6 points at n=100n=100), and leakage through preprocessing, duplicates, time, target proxies or test reuse invalidates any interval.

Check yourself

In the interactive lesson these are scored with your confidence, and what you miss comes back for review.

Question 1

A screening model is validated in a clinic where 20% of patients have the disease, then deployed in a population where 1% do. Which of these stays the same?

  1. Precision
  2. Accuracy
  3. The ROC curve
  4. F1
Show the answer

C. The ROC curve

The ROC curve plots TPR against FPR, and both condition on the true class, so they do not depend on how common the classes are (as long as the class-conditional score distributions are unchanged). Precision, and with it F1, collapses at 1% prevalence; accuracy drifts toward the share of negatives.

Question 2

A model has AUC 0.9. Which statement is correct?

  1. 90% of its predictions are correct
  2. At its best threshold its recall is 0.9
  3. A randomly chosen positive gets a higher score than a randomly chosen negative 90% of the time
  4. Its predicted probabilities are within 0.1 of the true probabilities
Show the answer

C. A randomly chosen positive gets a higher score than a randomly chosen negative 90% of the time

AUC =Pr⁡(S+>S−)=\Pr(S^+>S^-), ties counting half: a statement about ranking, averaged over all thresholds. It says nothing about accuracy at any particular threshold, and nothing about calibration.

Question 3

Missing a fraudulent payment costs 20 times as much as investigating a legitimate one. Given calibrated fraud probabilities pp, above what value should a payment be flagged?

  1. 1/21/2
  2. 20/2120/21
  3. 1/201/20
  4. 1/211/21
Show the answer

D. 1/211/21

Flag when p cFN>(1−p) cFPp\,c_{FN}>(1-p)\,c_{FP}, that is p>cFP/(cFP+cFN)=1/(1+20)p>c_{FP}/(c_{FP}+c_{FN})=1/(1+20). Expensive misses push the threshold down.

Question 4

Why are probabilistic classifiers trained and compared with log loss or the Brier score rather than accuracy?

  1. Log loss and Brier are strictly proper: only honest probabilities minimize their expectation, whereas 0–1 loss scores every pp on the right side of 12\tfrac12 the same
  2. Accuracy cannot be computed on a test set
  3. Log loss and Brier ignore calibration and focus on ranking
  4. Accuracy is strictly proper, but it is too slow to compute
Show the answer

A. Log loss and Brier are strictly proper: only honest probabilities minimize their expectation, whereas 0–1 loss scores every pp on the right side of 12\tfrac12 the same

Their expected excess over the honest forecast is DKL(q ∥ p)\KL(q\,\|\,p) or (p−q)2(p-q)^2, so they reward probabilities that match frequencies. 0–1 loss at threshold 12\tfrac12 is proper but not strictly: it cannot tell 0.51 from 0.99.

Question 5

A dataset has several chest X-rays per patient. Splitting images at random gives 97% test accuracy; splitting by patient gives 85%. What is the most likely explanation?

  1. The patient-level split is unlucky; average the two numbers
  2. With random splits the same patients appear in training and test, so the model can partly recognize patients instead of disease
  3. The random split has more test images, so its estimate is more accurate
  4. Patient-level splits always underestimate accuracy
Show the answer

B. With random splits the same patients appear in training and test, so the model can partly recognize patients instead of disease

Images of one patient share anatomy, devices and labels. When they straddle the split, the test set is not independent of training, and the score rewards memorization. The patient-level split estimates performance on new patients, which is the deployment question.

Go deeper

Further reading

  • T. Fawcett, "An introduction to ROC analysis," Pattern Recognition Letters 27, 861–874 (2006).
  • J. Davis & M. Goadrich, "The relationship between precision-recall and ROC curves," ICML (2006).
  • T. Gneiting & A. E. Raftery, "Strictly proper scoring rules, prediction, and estimation," Journal of the American Statistical Association 102, 359–378 (2007).
  • S. Kapoor & A. Narayanan, "Leakage and the reproducibility crisis in machine-learning-based science," Patterns (2023).