Evaluating Models
Metrics, ROC curves, calibration and leakage.
A fraud detector that flags nothing scores 99.8% accuracy when 0.2% of payments are fraud. A model with an excellent ROC curve can still bury every real alarm under nine false ones. Evaluation is a discipline of its own: pick the number that matches the decision, read the curves behind it, check that probabilities mean what they say, and make sure nothing leaked from the test set.
What you’ll learn
- Compute confusion-matrix metrics and derive how precision depends on prevalence.
- Prove that AUC is the probability that a random positive outranks a random negative, and compute it from ranks.
- Choose a decision threshold from costs and prevalence, and decide when precision–recall curves beat ROC curves.
- Prove that log loss and the Brier score are strictly proper, and read a reliability diagram.
- Put an interval on a test score and recognize the common forms of data leakage.
Confusion matrices and the base rate
A binary classifier computes a score and flags when . On a labeled test set it produces four counts, true and false positives and negatives: the confusion matrix. Every common metric is a ratio of these counts, and the ratios fall into two families: rates conditioned on the true class, and rates conditioned on the prediction.
| Metric | Formula | Conditions on | Changes with prevalence? |
|---|---|---|---|
| Recall (TPR, sensitivity) | actual positives | no | |
| False positive rate (FPR) | actual negatives | no | |
| Specificity (TNR) | actual negatives | no | |
| Precision (PPV) | predicted positives | yes | |
| Accuracy | everything | yes | |
| Balanced accuracy | each class, averaged | no | |
| F1 | ignores TN | yes |
Write for the prevalence. A flagged case is a real positive with probability
In odds form, the classifier enters only through its likelihood ratio and the population only through the prior odds:
As with the classifier fixed, precision falls in proportion to : rare positives drown in false alarms, the base-rate fallacy in metric form. Meanwhile the classifier that flags nothing scores accuracy .
A classifier has TPR 0.575 and FPR 0.095, a likelihood ratio of about 6. It is deployed where 2% of cases are positive. What fraction of its flags are real positives?
Show the answer
11 %
. Nine in ten alarms are false. At 30% prevalence the same classifier has precision 72%: nothing about the model changed, only the population.
ROC curves and what AUC measures
Sweep the threshold from down to and trace from to : the ROC curve. Both coordinates condition on the true class, so the curve does not depend on prevalence. It uses only comparisons , so it is unchanged by any strictly increasing transformation of the scores. The area under it, the AUC, has a clean probabilistic meaning.
For continuous scores with densities , and , so . As falls from to , FPR rises from 0 to 1:
On data, sort all scores and let be the sum of the positives' ranks (average ranks for ties). The positives' ranks count every score below them, including the positive–positive comparisons plus one for themselves; subtract those:
Sorting makes this instead of . In the binormal model, negatives score and positives , so :
import numpy as np
from scipy.stats import rankdata
def auc(scores, labels):
"""Mann–Whitney AUC in O(n log n); tied scores get average ranks."""
labels = np.asarray(labels, dtype=bool)
ranks = rankdata(scores) # 1 = lowest score
n_pos, n_neg = labels.sum(), (~labels).sum()
return (ranks[labels].sum() - n_pos * (n_pos + 1) / 2) / (n_pos * n_neg)The same number scikit-learn's roc_auc_score returns for binary labels.
Positives score ; negatives score . What is the AUC?
Show the answer
C.
There are positive–negative pairs. beats all four negatives, and beat three each (not ): correct pairs, so . By ranks: the positives sit at ranks , so and .
Precision–recall curves and choosing a threshold
Prevalence-invariance makes ROC curves good for comparing rankers and misleading for deployment. The precision–recall (PR) curve plots precision against recall over all thresholds. Random scoring gives a flat line at precision , and the area, average precision (AP), falls with prevalence even when the ranking is unchanged. Davis and Goadrich (2006) proved that one classifier's curve dominates another's in ROC space if and only if it does in PR space, so the two agree on which classifier is uniformly better; they differ in what they magnify. With rare positives, the informative part of the ROC curve is squashed against , and the PR curve spreads it out.
Let a false positive cost and a false negative , and suppose the model reports a calibrated . Flagging costs in expectation and not flagging costs . Flag when
Equal costs give ; if a miss is ten times worse than a false alarm, flag above . In the binormal model the log-likelihood ratio is linear in the score, , so the rule becomes a score threshold that moves with prevalence:
With and equal costs, at 30% prevalence (accuracy 0.806, the widget's maximum) but at 2%. A threshold tuned on balanced validation data is wrong for a rare-positive deployment.
Think first When should you report ROC AUC, and when average precision?
ROC AUC when ranking quality should not depend on the population: comparing models across sites with different prevalence, or when both classes matter symmetrically. AP, or precision at the top , when positives are rare and what matters is the head of the ranking: retrieval, fraud queues, screening with limited follow-up capacity. When unsure, report both, together with the operating point you will actually use.
Proper scoring rules and calibration
A model that outputs probabilities should be judged on the probabilities, not just their order. A scoring rule charges a forecast when the outcome occurs. It is proper if a forecaster who believes minimizes the expected charge by reporting , and strictly proper if is the only minimizer. Proper rules make honesty the best policy. Training with one is what makes predicted probabilities meaningful: log loss is the cross-entropy minimized by logistic regression and softmax networks.
Brier: use and complete the square. The last term does not depend on .
Log loss: add and subtract the binary entropy :
Gibbs' inequality, from , shows , with equality only when :
Which scoring rule rewards a forecaster for exaggerating, reporting 0 or 1 instead of an honest probability?
- The Brier score
- The absolute error
- The log loss
- None: any loss that is zero for a perfect forecast is proper
Show the answer
B. The absolute error
is linear in : it is minimized at when and at when . Scored by absolute error, a forecaster should round every belief to 0 or 1. (It is the right loss for predicting a median, not a probability.)
Calibration asks that probabilities match frequencies: among the cases where the model says 0.8, about 80% should be positive. Formally, for every . A reliability diagram bins the predictions (say ten bins of width 0.1) and plots each bin's observed positive rate against its mean prediction; perfect calibration is the diagonal. The expected calibration error summarizes the gaps: .
Honest numbers: intervals and leakage
A test score is an estimate. If the test examples are drawn i.i.d. and the model's true accuracy is , test accuracy is a binomial proportion with standard error : 3 points at and , 0.3 points at . The 95% Wilson interval behaves well even near 0 and 1:
Intervals cover sampling noise only. The more dangerous failure is leakage: information about the test labels reaching the model or the modeling decisions. Kapoor and Narayanan (2023) traced leakage through hundreds of published machine-learning studies in more than a dozen fields.
| Leak | Example | Fix |
|---|---|---|
| Preprocessing or selection on all data | scaling, imputation or feature screening fitted before the split | refit every data-dependent step inside each fold (see Generalization & Overfitting) |
| Duplicates and non-independence | the same patient, user or near-duplicate image on both sides of the split | split by group: patient, user, document |
| Temporal leakage | random splits of time-ordered data; features computed with future information | train on the past, test on the future |
| Target leakage | a feature recorded after, or because of, the outcome, such as a treatment code when predicting the diagnosis | audit when and how each feature is generated |
| Test-set reuse | choosing models or hyperparameters by test score | a validation set; one final test |
Key takeaways
- TPR and FPR describe the classifier; precision and accuracy also describe the population. Precision odds equal prior odds times , so precision collapses as positives become rare.
- ROC curves ignore prevalence and monotone rescaling, and is computable from ranks in . PR curves and average precision show performance on rare positives.
- The cost-optimal decision flags when ; the matching score threshold moves with prevalence.
- Log loss and the Brier score are strictly proper (excess and ). Calibration, , is a separate axis from discrimination.
- A test score needs an interval (about points at ), and leakage through preprocessing, duplicates, time, target proxies or test reuse invalidates any interval.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
A screening model is validated in a clinic where 20% of patients have the disease, then deployed in a population where 1% do. Which of these stays the same?
- Precision
- Accuracy
- The ROC curve
- F1
Show the answer
C. The ROC curve
The ROC curve plots TPR against FPR, and both condition on the true class, so they do not depend on how common the classes are (as long as the class-conditional score distributions are unchanged). Precision, and with it F1, collapses at 1% prevalence; accuracy drifts toward the share of negatives.
A model has AUC 0.9. Which statement is correct?
- 90% of its predictions are correct
- At its best threshold its recall is 0.9
- A randomly chosen positive gets a higher score than a randomly chosen negative 90% of the time
- Its predicted probabilities are within 0.1 of the true probabilities
Show the answer
C. A randomly chosen positive gets a higher score than a randomly chosen negative 90% of the time
AUC , ties counting half: a statement about ranking, averaged over all thresholds. It says nothing about accuracy at any particular threshold, and nothing about calibration.
Missing a fraudulent payment costs 20 times as much as investigating a legitimate one. Given calibrated fraud probabilities , above what value should a payment be flagged?
Show the answer
D.
Flag when , that is . Expensive misses push the threshold down.
Why are probabilistic classifiers trained and compared with log loss or the Brier score rather than accuracy?
- Log loss and Brier are strictly proper: only honest probabilities minimize their expectation, whereas 0–1 loss scores every on the right side of the same
- Accuracy cannot be computed on a test set
- Log loss and Brier ignore calibration and focus on ranking
- Accuracy is strictly proper, but it is too slow to compute
Show the answer
A. Log loss and Brier are strictly proper: only honest probabilities minimize their expectation, whereas 0–1 loss scores every on the right side of the same
Their expected excess over the honest forecast is or , so they reward probabilities that match frequencies. 0–1 loss at threshold is proper but not strictly: it cannot tell 0.51 from 0.99.
A dataset has several chest X-rays per patient. Splitting images at random gives 97% test accuracy; splitting by patient gives 85%. What is the most likely explanation?
- The patient-level split is unlucky; average the two numbers
- With random splits the same patients appear in training and test, so the model can partly recognize patients instead of disease
- The random split has more test images, so its estimate is more accurate
- Patient-level splits always underestimate accuracy
Show the answer
B. With random splits the same patients appear in training and test, so the model can partly recognize patients instead of disease
Images of one patient share anatomy, devices and labels. When they straddle the split, the test set is not independent of training, and the score rewards memorization. The patient-level split estimates performance on new patients, which is the deployment question.
Go deeper
- Calibration & Uncertainty Quantification · When a model’s 80% should mean 80%.
- Experimental Design & Statistical Testing · Significance, multiple comparisons and honest benchmarks.
- Distribution Shift & Robustness · When the test world differs from the training world.
Further reading
- T. Fawcett, "An introduction to ROC analysis," Pattern Recognition Letters 27, 861–874 (2006).
- J. Davis & M. Goadrich, "The relationship between precision-recall and ROC curves," ICML (2006).
- T. Gneiting & A. E. Raftery, "Strictly proper scoring rules, prediction, and estimation," Journal of the American Statistical Association 102, 359–378 (2007).
- S. Kapoor & A. Narayanan, "Leakage and the reproducibility crisis in machine-learning-based science," Patterns (2023).