Bayesian Reasoning
Priors, likelihoods and updating beliefs with evidence.
A spam filter weighing a suspicious word, a doctor reading a test, an engineer estimating a failure rate and a scientist comparing two theories all do the same computation: multiply a prior by a likelihood and renormalize. This lesson turns that one line into working machinery: odds and likelihood ratios, posteriors over parameters, point estimates as decisions, model comparison, and the algorithms used when the integral has no closed form.
What you’ll learn
- Derive Bayes' rule and its odds form, and prove that sequential updating equals batch updating for conditionally independent evidence.
- Compute the Beta posterior of a Bernoulli parameter, its mean and variance, and the posterior predictive (Laplace's rule of succession).
- Derive the posterior mean, median and MAP estimate as Bayes decisions under different losses, and show why MAP depends on the parameterization.
- Compute marginal likelihoods and Bayes factors, explain the automatic Occam's razor, and sample a posterior with Metropolis–Hastings when no closed form exists.
Bayes' rule, and its odds form
The product rule factors a joint probability two ways, . Dividing by gives Bayes' rule. For mutually exclusive, exhaustive hypotheses , or a continuous parameter ,
The prior is reweighted by the likelihood and renormalized into the posterior. The denominator, the evidence or marginal likelihood , does not depend on ; it returns in step 4 to score whole models.
For two hypotheses, divide Bayes' rule for by Bayes' rule for . The evidence cancels, and what is left is the cleanest form of the rule:
Let be conditionally independent given (and given ): . Apply the product rule to both pieces of evidence at once:
The bracket is proportional to the posterior after , so yesterday's posterior serves as today's prior:
In odds form the likelihood ratios simply multiply. This is the naive Bayes classifier: assume the words of an email are conditionally independent given its class, and multiply one ratio per word.
If the pieces are not conditionally independent, as with two tests that share a failure mode, multiplying their ratios counts the same evidence twice and makes the posterior overconfident.
Test A has sensitivity 99% and false-positive rate 5%. Test B has sensitivity 50% and false-positive rate 2.5%. For the same patient and prior, which positive result is stronger evidence of disease?
- Test A, because its sensitivity is much higher
- Test B, because its false-positive rate is lower
- About equal: both multiply the odds by about 20
- They cannot be compared without knowing the prior
Show the answer
C. About equal: both multiply the odds by about 20
A positive result acts only through : for A and for B. Their negative results differ enormously, though: for A versus for B, so only A can rule the disease out.
Learning a rate: the Beta–Bernoulli model
Now let the hypothesis be a number: the unknown probability that a coin lands heads, a user clicks or a part fails. Observe independent trials with successes. The likelihood of the sequence is . Take as prior a Beta distribution, , with density proportional to and mean ; is uniform.
Multiply likelihood and prior; constants that do not involve drop out:
That is again a Beta density. The prior behaves like earlier successes and earlier failures, and updating is counting:
Its mean is a weighted average of the prior mean and the maximum-likelihood estimate , and the prior's weight shrinks as data accumulate:
Its variance falls like , so the posterior standard deviation shrinks like , matching the frequentist standard error for large :
With a uniform prior , a coin lands heads in all 3 of 3 flips. What is the posterior probability that the next flip lands heads?
Show the answer
80 %
. This is Laplace's rule of succession, . The maximum-likelihood plug-in would say 100%, certainty from three flips.
In general the posterior predictive averages the model's predictions over the remaining uncertainty instead of plugging in one estimate:
A prior whose posterior stays in the same family, as the Beta does for Bernoulli data, is called conjugate. Bandit algorithms such as Thompson sampling keep exactly these Beta posteriors, one per arm. The Deep Layer lesson Conjugate Priors & Bayesian Updating extends the construction to every exponential family.
From posterior to decision
A posterior is a whole distribution, but a decision often needs a single number. Bayesian decision theory picks the action that minimizes the posterior expected loss, , and different losses give different answers.
The MAP estimate maximizes , so it is penalized maximum likelihood: . A Gaussian prior on weights gives the ridge () penalty and a Laplace prior the lasso () penalty, which is how regularization and priors turn out to be the same idea.
With a prior and 7 successes in 10 trials, the three estimates differ: the MLE is , the MAP , and the posterior mean . The difference shrinks as grows.
Think first Your posterior for a click-through rate is . Which single number should you report to minimize expected squared error, which to minimize expected absolute error, and what does MAP give?
The mean, ; the median, about ; the mode, . For a skewed posterior the three can differ by a factor of 1.5, so the loss you care about matters. A 95% credible interval answers yet another question: given these data, the parameter lies in it with probability 0.95. A 95% confidence interval instead comes from a procedure that covers the true value in 95% of repeated samples. The two often nearly coincide for large samples, but they answer different questions.
Comparing models: the evidence
The denominator of Bayes' rule scores whole models. For a model with parameters , the marginal likelihood is the probability the model assigned to the data before seeing them, averaged over its prior. Two models are compared by the odds form again, with the Bayes factor as the likelihood ratio.
: the coin is fair, . : the bias is unknown, with a uniform prior. For a particular sequence with heads in flips,
With 9 heads in 10 flips, the flexible model predicted the data about 9 times better:
With 5 heads in 10, the same computation favors the fair coin, by about 2.7 to 1:
| Heads in 10 flips | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|
| Bayes factor | 0.37 | 0.44 | 0.78 | 2.1 | 9.3 | 93 |
When the integral has no closed form
Conjugate pairs are the exception. For logistic regression, hierarchical models or neural networks, neither the evidence nor posterior expectations have closed forms. Three families of approximation cover most practice:
- Laplace's method fits a Gaussian at the MAP estimate, with covariance equal to the inverse Hessian of .
- Markov chain Monte Carlo (MCMC) runs a Markov chain whose stationary distribution is the posterior and averages over its states. It needs the posterior only up to a constant.
- Variational inference picks the member of a tractable family that maximizes the evidence lower bound, equivalently minimizes (see Information Theory for AI).
import numpy as np
def log_post(theta, k=7, n=10, a=2.0, b=2.0):
"""Log of the unnormalized posterior: Bernoulli likelihood times a Beta(a, b) prior."""
if not 0.0 < theta < 1.0:
return -np.inf # zero density outside (0, 1)
return (k + a - 1) * np.log(theta) + (n - k + b - 1) * np.log(1 - theta)
rng = np.random.default_rng(0)
theta, samples = 0.5, []
for t in range(20_000):
proposal = theta + 0.1 * rng.standard_normal() # symmetric random-walk proposal
if np.log(rng.uniform()) < log_post(proposal) - log_post(theta):
theta = proposal # accept; otherwise stay put
samples.append(theta)
draws = np.array(samples[2_000:]) # discard burn-in
print(draws.mean(), 9 / 14) # estimate vs exact posterior meanRandom-walk Metropolis on the Beta–Bernoulli posterior of step 3, where the exact answer is known. The normalizing constant never appears, only the difference of log-densities. For a symmetric proposal the Hastings correction equals 1.
Let be the posterior, known only as . From , propose and accept with probability
For , the probability flow from to is symmetric in the two points, which is detailed balance:
Integrating detailed balance over shows that one step of the chain, with kernel , leaves unchanged. If the chain is also irreducible and aperiodic, averages along it converge to posterior expectations.
Successive states are correlated, so the Monte Carlo error scales with an effective sample size . Gradient-based samplers such as Hamiltonian Monte Carlo, used by Stan, keep high in many dimensions.
Seeing is not doing, and where to go deeper
Conditioning answers "what should I expect, given that I observe ?" It does not answer "what would happen if I set ?" The second question is about the interventional distribution , the subject of causal inference, and it differs from whenever a confounder influences both.
In the classic sprinkler network, sprinklers run mostly on sunny days, so observing the sprinkler on lowers the probability of rain from 50% to 30%. Switching the sprinkler on by hand cannot change the weather. A treatment can even look worse in the pooled data and better in every subgroup, which is Simpson's paradox.
Think first In a hospital's records, patients who received a new drug died more often than those who did not. Does that show the drug is harmful?
Not by itself. If doctors gave the drug mostly to the sickest patients, severity confounds the comparison: mixes the drug's effect with the patients' prognosis. A randomized trial estimates directly, and so does adjusting for severity, provided it is the only confounder. Causal Inference makes precise when such adjustments are valid.
Key takeaways
- Posterior likelihood prior. In odds form, posterior odds = likelihood ratio × prior odds, and conditionally independent pieces of evidence multiply their ratios.
- Beta–Bernoulli updating is counting: . The posterior mean shrinks toward the prior mean, and the predictive is .
- Point estimates are decisions: the posterior mean minimizes squared loss, the median absolute loss, and the MAP 0–1 loss. MAP is penalized maximum likelihood and depends on the parameterization.
- The marginal likelihood scores models and builds in Occam's razor, but Bayes factors depend on the parameter priors.
- Without closed forms, use Laplace approximations, MCMC (detailed balance; the normalizer cancels) or variational inference. And conditioning on is not intervening on .
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
A hypothesis has prior odds , and new evidence has likelihood ratio 6. What is its posterior probability?
- 20%
- 60%
- 86%
- 40%
Show the answer
B. 60%
Posterior odds , that is , so the probability is .
With a prior on a success rate, you observe 3 successes in 10 trials. What is the posterior?
Show the answer
D.
Add successes to and failures to : , with mean , between the prior mean and the data's .
You reparameterize a success probability by its log-odds and recompute each estimate. Which one corresponds to the same value of as before?
- The maximum-likelihood estimate
- The MAP estimate
- The posterior mean
- All of them, since the beliefs have not changed
Show the answer
A. The maximum-likelihood estimate
The likelihood is a function of the parameter's value, so its maximizer maps across. Densities pick up a Jacobian, which moves the posterior mode, and expectations do not commute with non-linear maps: in general. (The posterior median is also invariant.)
A coin lands heads 5 times in 10. Comparing "fair" with "unknown bias, uniform prior", what does the Bayes factor say?
- Nothing, because both models fit 5 heads perfectly
- It favors the unknown-bias model, which contains the fair coin as a special case
- It favors the fair coin, by about 2.7 to 1
- It favors the unknown-bias model, by about 2.7 to 1
Show the answer
C. It favors the fair coin, by about 2.7 to 1
and , a ratio of about for . The flexible model spread its predictions over all outcomes; the fair coin concentrated on outcomes like this one.
Why can Metropolis–Hastings sample from without ever computing the evidence ?
- Because it approximates by a Gaussian first
- Because it uses only the likelihood and ignores the prior
- Because the samples are independent
- Because the acceptance probability uses a ratio of posterior densities, in which cancels
Show the answer
D. Because the acceptance probability uses a ratio of posterior densities, in which cancels
depends on (times the proposal ratio), and any constant factor of the posterior cancels. That is what makes MCMC the workhorse for models whose normalizers are intractable.
Go deeper
- Conjugate Priors & Bayesian Updating · Closed-form posteriors and what they teach.
- Causal Inference · Correlation, intervention and the do-calculus.
- Gaussian Processes & Bayesian ML · Distributions over functions and calibrated uncertainty.
Further reading
- A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari & D. B. Rubin, Bayesian Data Analysis, 3rd ed. (CRC Press, 2013).
- D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms (Cambridge University Press, 2003).
- R. E. Kass & A. E. Raftery, "Bayes factors," Journal of the American Statistical Association 90(430), 773–795 (1995).
- C. M. Bishop, Pattern Recognition and Machine Learning (Springer, 2006), chapters 2, 3 and 11.