Conjugate priors are the rare cases where Bayes' rule needs no integral: the posterior has the same form as the prior, and updating means adding the data's sufficient statistics to the prior's pseudo-counts. They repay study far beyond convenience, because they make visible what every Bayesian analysis does: shrink toward the prior, concentrate at rate 1/n, and eventually forget where it started, except when it does not.
What you’ll learn
Define exponential families and their conjugate priors, and prove the conjugate update and the Diaconis–Ylvisaker linearity of the posterior mean.
Derive Normal–Normal and Dirichlet–multinomial updating, with their posterior predictives and marginal likelihoods.
State the Bernstein–von Mises theorem, derive the Jeffreys prior, and identify where Bayesian asymptotics fail.
Prove that the James–Stein estimator dominates the maximum-likelihood estimate in three or more dimensions, and interpret it as empirical Bayes.
(i) Multiply prior and likelihood; h(xi) does not involve η:
0=∫∇ηeη⊤χ′−ν′A(η)dη=∫(χ′−ν′∇A(η))eη⊤χ′−ν′A(η)dη
(ii) Write (χ′,ν′) for the updated hyperparameters. The posterior density vanishes at the boundary of the parameter space under the regularity conditions, so the integral of its gradient is zero:
E[∇A(η)∣x1:n]=ν′χ′=ν+nχ+∑iT(xi)
Divide by the normalizing constant: E[χ′−ν′∇A(η)∣x1:n]=0. For Bernoulli data, η=log1−θθ, the prior is Beta(χ,ν−χ) on θ, and this is the familiar (a+k)/(a+b+n). ∎
Likelihood
Conjugate prior
Posterior
Posterior predictive
Bernoulli, rate θ
Beta(a,b)
Beta(a+k,b+n−k)
beta-binomial
Categorical, π
Dir(α)
Dir(α+n)
Dirichlet-multinomial
Poisson, rate λ
Gamma(a,b) (rate b)
Gamma(a+∑ixi,b+n)
negative binomial
Exponential, rate λ
Gamma(a,b)
Gamma(a+n,b+∑ixi)
Lomax (Pareto II)
Normal, mean μ (σ2 known)
N(μ0,τ02)
N(μn,τn2), next step
N(μn,σ2+τn2)
Normal, mean and variance
Normal–inverse-gamma
Normal–inverse-gamma
Student t
Conjugacy is essentially an exponential-family phenomenon: by the Pitman–Koopman–Darmois theorem, among regular families whose support does not depend on the parameter, only exponential families have sufficient statistics of fixed dimension for every sample size.
Prior μ∼N(μ0,τ02), data xi∣μ∼N(μ,σ2). The log-posterior is quadratic in μ:
τn21=τ021+σ2n,μn=τn2(τ02μ0+σ2nxˉ)
Collect the μ2 and μ terms and complete the square. Precisions add, and the mean is a precision-weighted average of the prior mean and the sample mean xˉ:
xnew∣D∼N(μn,σ2+τn2)
A new observation is xnew=μ+ε with independent noise, so two variances add: noise that never goes away, and parameter uncertainty that shrinks like σ2/n:
For K categories with a Dir(α1,…,αK) prior and counts nk (total n, α0=∑kαk), the posterior is Dir(α+n), and the predictive is additive smoothing, the remedy for zero counts in n-gram models and naive Bayes:
The marginal likelihood (right) is the Pólya urn: draw a ball, return it with one more of the same color. The resulting sequence is exchangeable, and by de Finetti's theorem every exchangeable sequence is a mixture of i.i.d. sequences of this kind (see Probability Essentials).
Predict before you look
A six-sided die gets a uniform Dir(1,…,1) prior on its face probabilities. In 10 rolls, a six comes up 4 times. What is the posterior predictive probability that the next roll is a six?
Show the answer
31 %
(α6+n6)/(α0+n)=(1+4)/(6+10)=5/16≈31%, between the prior's 1/6≈17% and the empirical 40%. Laplace smoothing is the special case αk=1.
Asymptotics: when the prior washes out
DerivationChecking it, and an invariant prior
I(θ)=E[(∂θ∂logp(X∣θ))2]=θ(1−θ)1
For Bernoulli data, logp(x∣θ)=xlogθ+(1−x)log(1−θ), and the Fisher information is
So Bernstein–von Mises predicts posterior variance θ0(1−θ0)/n, which matches the exact Beta posterior variance ≈θ^(1−θ^)/n for any fixed a,b.
pJ(θ)∝θ−1/2(1−θ)−1/2,Iφ(φ)=Iθ(θ)dφdθ
Jeffreys (1946) proposed the prior pJ(θ)∝I(θ). Under φ=g(θ), information transforms as Iφ=Iθ(dθ/dφ)2, so I transforms exactly like a density: the same prior in every parameterization. For Bernoulli it is Beta(21,21).
Shrinkage: James–Stein and empirical Bayes
Estimate many means at once: the click rates of d ads, the skill of d players. Observe X∼Nd(θ,I) and use squared-error risk R(θ,θ^)=E∥θ^−θ∥2. The maximum-likelihood estimate θ^=X has risk d everywhere, and Stein (1956) proved that it is inadmissible once d≥3.
DerivationProof via Stein's lemma
E[(X−θ)⊤g(X)]=E[∇⋅g(X)]
Stein's lemma. For X∼Nd(θ,I) and weakly differentiable g with finite expectations, integration by parts against the Gaussian density gives
E∥θ^−θ∥2=d−2E[∇⋅g(X)]+E∥g(X)∥2
Write θ^=X−g(X) with g(x)=(d−2)x/∥x∥2, and expand the risk:
Since ∇⋅(x/∥x∥2)=(d−2)/∥x∥2, both remaining terms are multiples of E[1/∥X∥2], which is finite exactly when d≥3:
R(0,θ^JS)=d−(d−2)=2
At θ=0, ∥X∥2∼χd2 and E[1/χd2]=1/(d−2), so the risk is 2 in every dimension, against d for the MLE. ∎
Efron and Morris (1973) explained the paradox as empirical Bayes. If θi∼N(0,A) independently, the Bayes estimate is (1−1+A1)Xi; marginally ∥X∥2∼(1+A)χd2, so (d−2)/∥X∥2 is an unbiased estimate of the shrinkage factor 1/(1+A). James–Stein is the Bayes rule with the prior's variance estimated from the data themselves. Hierarchical models do the same with a prior on A, which is why "partial pooling" beats both treating every unit separately and pooling them all.
Check your understanding
In d=10 dimensions with true mean θ=0, what is the risk of the James–Stein estimator, compared with 10 for the MLE?
0
8
2
10
Show the answer
C. 2
R=d−(d−2)2E[1/χ102]=10−64/8=2. The gain is largest near the shrinkage target and shrinks as ∥θ∥ grows, but the risk stays below d everywhere.
Key takeaways
For an exponential family, the conjugate prior updates by (χ,ν)↦(χ+∑iT(xi),ν+n): pseudo-observations plus real ones. The posterior mean of the mean parameter is linear in the data, and only conjugate priors make it so (Diaconis–Ylvisaker).
Normal–Normal: precisions add, means average by precision, and the predictive variance is σ2+τn2. Dirichlet–multinomial: additive smoothing (αk+nk)/(α0+n), the Pólya urn.
Bernstein–von Mises: in regular parametric models the posterior approaches N(θ^,(nI)−1). The Jeffreys prior I(θ) is parameterization-invariant. Infinite dimensions, misspecification and zero prior mass break the asymptotics.
James–Stein: for d≥3, shrinking toward a common point beats the MLE everywhere (risk 2 instead of d at the target). It is empirical Bayes, the logic behind partial pooling.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
Question 1
Counts 3,5,4 are modeled as Poisson(λ) with a Gamma(2,1) prior (shape 2, rate 1). What is the posterior?
Gamma(12,3)
Gamma(14,1)
Gamma(5,4)
Gamma(14,4)
Show the answer
D.Gamma(14,4)
Add the total count to the shape and the number of observations to the rate: Gamma(2+12,1+3), with mean 3.5, between the prior mean 2 and the sample mean 4.
Question 2
Prior μ∼N(0,1); one observation x=2 with noise variance σ2=1. What is the variance of the posterior predictive distribution of the next observation?
0.5
1.5
1
2
Show the answer
B.1.5
Precisions add, 1/τ12=1+1, so τ12=0.5 (and μ1=1). The predictive adds the noise variance: 1+0.5=1.5.
Question 3
In which situation does the Bernstein–von Mises theorem not guarantee that 95% credible intervals have about 95% frequentist coverage for large n?
A regular two-parameter model, with a smooth prior positive at the truth
A Beta–Bernoulli model with a Beta(5,5) prior and true rate 0.3
A misspecified model, where the posterior concentrates at the KL-closest parameter
A Normal mean with a Normal prior centered far from the truth
Show the answer
C. A misspecified model, where the posterior concentrates at the KL-closest parameter
Under misspecification the posterior's spread no longer matches the sampling variability of the estimator, so credible sets can miscover however large n is. The other three are regular cases, where the prior's influence fades.
Question 4
Under the conjugate prior with hyperparameters (χ,ν), what is the posterior mean of E[T(X)] after n observations?
ν+nχ+∑iT(xi)
n1∑iT(xi)
χ/ν
χ+∑iT(xi)
Show the answer
A.ν+nχ+∑iT(xi)
Diaconis–Ylvisaker: the posterior mean of the mean parameter is the pooled average of ν pseudo-observations and n real ones, linear in the sufficient statistic.
Further reading
P. Diaconis & D. Ylvisaker, "Conjugate priors for exponential families," Annals of Statistics 7(2), 269–281 (1979).
W. James & C. Stein, "Estimation with quadratic loss," Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability 1, 361–379 (1961).
B. Efron & C. Morris, "Stein's estimation rule and its competitors—an empirical Bayes approach," Journal of the American Statistical Association 68(341), 117–130 (1973).
A. W. van der Vaart, Asymptotic Statistics (Cambridge University Press, 1998), chapter 10.
A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari & D. B. Rubin, Bayesian Data Analysis, 3rd ed. (CRC Press, 2013).