Simple Algorithm
Artificial Intelligence · Deep Layer · 26 min

Conjugate Priors & Bayesian Updating

Closed-form posteriors and what they teach.

Conjugate priors are the rare cases where Bayes' rule needs no integral: the posterior has the same form as the prior, and updating means adding the data's sufficient statistics to the prior's pseudo-counts. They repay study far beyond convenience, because they make visible what every Bayesian analysis does: shrink toward the prior, concentrate at rate 1/n1/\sqrt n, and eventually forget where it started, except when it does not.

What you’ll learn

  • Define exponential families and their conjugate priors, and prove the conjugate update and the Diaconis–Ylvisaker linearity of the posterior mean.
  • Derive Normal–Normal and Dirichlet–multinomial updating, with their posterior predictives and marginal likelihoods.
  • State the Bernstein–von Mises theorem, derive the Jeffreys prior, and identify where Bayesian asymptotics fail.
  • Prove that the James–Stein estimator dominates the maximum-likelihood estimate in three or more dimensions, and interpret it as empirical Bayes.

Exponential families and their conjugates

DerivationProof
  1. p(η∣x1:n) ∝ exp⁡(η⊤χ−νA(η))∏i=1nexp⁡(η⊤T(xi)−A(η))=exp⁡(η⊤(χ+∑iT(xi))−(ν+n)A(η))p(\eta\mid x_{1:n})\ \propto\ \exp\big(\eta^\top\chi-\nu A(\eta)\big)\prod_{i=1}^n\exp\big(\eta^\top T(x_i)-A(\eta)\big)=\exp\Big(\eta^\top\big(\chi+\textstyle\sum_iT(x_i)\big)-(\nu+n)A(\eta)\Big)

    (i) Multiply prior and likelihood; h(xi)h(x_i) does not involve η\eta:

  2. 0=∫∇η eη⊤χ′−ν′A(η) dη=∫(χ′−ν′∇A(η)) eη⊤χ′−ν′A(η) dη0=\int\nabla_\eta\,e^{\eta^\top\chi'-\nu'A(\eta)}\,d\eta=\int\big(\chi'-\nu'\nabla A(\eta)\big)\,e^{\eta^\top\chi'-\nu'A(\eta)}\,d\eta

    (ii) Write (χ′,ν′)(\chi',\nu') for the updated hyperparameters. The posterior density vanishes at the boundary of the parameter space under the regularity conditions, so the integral of its gradient is zero:

  3. E[∇A(η)∣x1:n]=χ′ν′=χ+∑iT(xi)ν+n\E\big[\nabla A(\eta)\mid x_{1:n}\big]=\frac{\chi'}{\nu'}=\frac{\chi+\sum_iT(x_i)}{\nu+n}

    Divide by the normalizing constant: E[χ′−ν′∇A(η)∣x1:n]=0\E[\chi'-\nu'\nabla A(\eta)\mid x_{1:n}]=0. For Bernoulli data, η=log⁡θ1−θ\eta=\log\frac\theta{1-\theta}, the prior is Beta(χ,ν−χ)\mathrm{Beta}(\chi,\nu-\chi) on θ\theta, and this is the familiar (a+k)/(a+b+n)(a+k)/(a+b+n). ∎

LikelihoodConjugate priorPosteriorPosterior predictive
Bernoulli, rate θ\thetaBeta(a,b)\mathrm{Beta}(a,b)Beta(a+k, b+n−k)\mathrm{Beta}(a+k,\,b+n-k)beta-binomial
Categorical, π\boldsymbol\piDir(α)\mathrm{Dir}(\boldsymbol\alpha)Dir(α+n)\mathrm{Dir}(\boldsymbol\alpha+\mathbf n)Dirichlet-multinomial
Poisson, rate λ\lambdaGamma(a,b)\mathrm{Gamma}(a,b) (rate bb)Gamma(a+∑ixi, b+n)\mathrm{Gamma}\big(a+\sum_ix_i,\,b+n\big)negative binomial
Exponential, rate λ\lambdaGamma(a,b)\mathrm{Gamma}(a,b)Gamma(a+n, b+∑ixi)\mathrm{Gamma}\big(a+n,\,b+\sum_ix_i\big)Lomax (Pareto II)
Normal, mean μ\mu (σ2\sigma^2 known)N(μ0,τ02)\mathcal N(\mu_0,\tau_0^2)N(μn,τn2)\mathcal N(\mu_n,\tau_n^2), next stepN(μn, σ2+τn2)\mathcal N(\mu_n,\,\sigma^2+\tau_n^2)
Normal, mean and varianceNormal–inverse-gammaNormal–inverse-gammaStudent tt
Conjugacy is essentially an exponential-family phenomenon: by the Pitman–Koopman–Darmois theorem, among regular families whose support does not depend on the parameter, only exponential families have sufficient statistics of fixed dimension for every sample size.

Normal–Normal and Dirichlet–multinomial

DerivationNormal mean with known variance
  1. log⁡p(μ∣D)=−(μ−μ0)22τ02−∑i=1n(xi−μ)22σ2+const\log p(\mu\mid\data)=-\frac{(\mu-\mu_0)^2}{2\tau_0^2}-\sum_{i=1}^n\frac{(x_i-\mu)^2}{2\sigma^2}+\text{const}

    Prior μ∼N(μ0,τ02)\mu\sim\mathcal N(\mu_0,\tau_0^2), data xi∣μ∼N(μ,σ2)x_i\mid\mu\sim\mathcal N(\mu,\sigma^2). The log-posterior is quadratic in μ\mu:

  2. 1τn2=1τ02+nσ2,μn=τn2(μ0τ02+nxˉσ2)\frac1{\tau_n^2}=\frac1{\tau_0^2}+\frac n{\sigma^2},\qquad \mu_n=\tau_n^2\Big(\frac{\mu_0}{\tau_0^2}+\frac{n\bar x}{\sigma^2}\Big)

    Collect the μ2\mu^2 and μ\mu terms and complete the square. Precisions add, and the mean is a precision-weighted average of the prior mean and the sample mean xˉ\bar x:

  3. xnew∣D ∼ N(μn, σ2+τn2)x_{\mathrm{new}}\mid\data\ \sim\ \mathcal N\big(\mu_n,\ \sigma^2+\tau_n^2\big)

    A new observation is xnew=μ+εx_{\mathrm{new}}=\mu+\varepsilon with independent noise, so two variances add: noise that never goes away, and parameter uncertainty that shrinks like σ2/n\sigma^2/n:

For KK categories with a Dir(α1,…,αK)\mathrm{Dir}(\alpha_1,\dots,\alpha_K) prior and counts nkn_k (total nn, α0=∑kαk\alpha_0=\sum_k\alpha_k), the posterior is Dir(α+n)\mathrm{Dir}(\boldsymbol\alpha+\mathbf n), and the predictive is additive smoothing, the remedy for zero counts in n-gram models and naive Bayes:

P(xn+1=k∣D)=αk+nkα0+n,p(x1:n)=Γ(α0)Γ(α0+n)∏k=1KΓ(αk+nk)Γ(αk)P(x_{n+1}=k\mid\data)=\frac{\alpha_k+n_k}{\alpha_0+n},\qquad p(x_{1:n})=\frac{\Gamma(\alpha_0)}{\Gamma(\alpha_0+n)}\prod_{k=1}^K\frac{\Gamma(\alpha_k+n_k)}{\Gamma(\alpha_k)}
The marginal likelihood (right) is the Pólya urn: draw a ball, return it with one more of the same color. The resulting sequence is exchangeable, and by de Finetti's theorem every exchangeable sequence is a mixture of i.i.d. sequences of this kind (see Probability Essentials).
Predict before you look

A six-sided die gets a uniform Dir(1,…,1)\mathrm{Dir}(1,\dots,1) prior on its face probabilities. In 10 rolls, a six comes up 4 times. What is the posterior predictive probability that the next roll is a six?

Show the answer

31 %

(α6+n6)/(α0+n)=(1+4)/(6+10)=5/16≈31%(\alpha_6+n_6)/(\alpha_0+n)=(1+4)/(6+10)=5/16\approx31\%, between the prior's 1/6≈17%1/6\approx17\% and the empirical 40%40\%. Laplace smoothing is the special case αk=1\alpha_k=1.

Asymptotics: when the prior washes out

DerivationChecking it, and an invariant prior
  1. I(θ)=E[(∂∂θlog⁡p(X∣θ))2]=1θ(1−θ)I(\theta)=\E\Big[\Big(\frac{\partial}{\partial\theta}\log p(X\mid\theta)\Big)^2\Big]=\frac1{\theta(1-\theta)}

    For Bernoulli data, log⁡p(x∣θ)=xlog⁡θ+(1−x)log⁡(1−θ)\log p(x\mid\theta)=x\log\theta+(1-x)\log(1-\theta), and the Fisher information is

  2. (a+k)(b+n−k)(a+b+n)2(a+b+n+1)=θ^(1−θ^)n(1+O(1/n))\frac{(a+k)(b+n-k)}{(a+b+n)^2(a+b+n+1)}=\frac{\hat\theta(1-\hat\theta)}n\big(1+\order{1/n}\big)

    So Bernstein–von Mises predicts posterior variance θ0(1−θ0)/n\theta_0(1-\theta_0)/n, which matches the exact Beta posterior variance ≈θ^(1−θ^)/n\approx\hat\theta(1-\hat\theta)/n for any fixed a,ba,b.

  3. pJ(θ)∝θ−1/2(1−θ)−1/2,Iφ(φ)=Iθ(θ) ∣dθdφ∣p_J(\theta)\propto\theta^{-1/2}(1-\theta)^{-1/2},\qquad \sqrt{I_\varphi(\varphi)}=\sqrt{I_\theta(\theta)}\,\Big|\frac{d\theta}{d\varphi}\Big|

    Jeffreys (1946) proposed the prior pJ(θ)∝I(θ)p_J(\theta)\propto\sqrt{I(\theta)}. Under φ=g(θ)\varphi=g(\theta), information transforms as Iφ=Iθ (dθ/dφ)2I_\varphi=I_\theta\,(d\theta/d\varphi)^2, so I\sqrt{I} transforms exactly like a density: the same prior in every parameterization. For Bernoulli it is Beta(12,12)\mathrm{Beta}(\tfrac12,\tfrac12).

Shrinkage: James–Stein and empirical Bayes

Estimate many means at once: the click rates of dd ads, the skill of dd players. Observe X∼Nd(θ,I)X\sim\mathcal N_d(\theta,I) and use squared-error risk R(θ,θ^)=E∥θ^−θ∥2R(\theta,\hat\theta)=\E\|\hat\theta-\theta\|^2. The maximum-likelihood estimate θ^=X\hat\theta=X has risk dd everywhere, and Stein (1956) proved that it is inadmissible once d≥3d\ge3.

DerivationProof via Stein's lemma
  1. E[(X−θ)⊤g(X)]=E[∇⋅g(X)]\E\big[(X-\theta)^\top g(X)\big]=\E\big[\nabla\cdot g(X)\big]

    Stein's lemma. For X∼Nd(θ,I)X\sim\mathcal N_d(\theta,I) and weakly differentiable gg with finite expectations, integration by parts against the Gaussian density gives

  2. E∥θ^−θ∥2=d−2 E[∇⋅g(X)]+E∥g(X)∥2\E\|\hat\theta-\theta\|^2=d-2\,\E\big[\nabla\cdot g(X)\big]+\E\|g(X)\|^2

    Write θ^=X−g(X)\hat\theta=X-g(X) with g(x)=(d−2) x/∥x∥2g(x)=(d-2)\,x/\|x\|^2, and expand the risk:

  3. E∥θ^−θ∥2=d−2(d−2)2 E[1∥X∥2]+(d−2)2 E[1∥X∥2]=d−(d−2)2 E[1∥X∥2]\E\|\hat\theta-\theta\|^2=d-2(d-2)^2\,\E\Big[\frac1{\|X\|^2}\Big]+(d-2)^2\,\E\Big[\frac1{\|X\|^2}\Big]=d-(d-2)^2\,\E\Big[\frac1{\|X\|^2}\Big]

    Since ∇⋅(x/∥x∥2)=(d−2)/∥x∥2\nabla\cdot\big(x/\|x\|^2\big)=(d-2)/\|x\|^2, both remaining terms are multiples of E[1/∥X∥2]\E[1/\|X\|^2], which is finite exactly when d≥3d\ge3:

  4. R(0,θ^JS)=d−(d−2)=2R(0,\hat\theta_{\mathrm{JS}})=d-(d-2)=2

    At θ=0\theta=0, ∥X∥2∼χd2\|X\|^2\sim\chi^2_d and E[1/χd2]=1/(d−2)\E[1/\chi^2_d]=1/(d-2), so the risk is 22 in every dimension, against dd for the MLE. ∎

Efron and Morris (1973) explained the paradox as empirical Bayes. If θi∼N(0,A)\theta_i\sim\mathcal N(0,A) independently, the Bayes estimate is (1−11+A)Xi\big(1-\frac1{1+A}\big)X_i; marginally ∥X∥2∼(1+A) χd2\|X\|^2\sim(1+A)\,\chi^2_d, so (d−2)/∥X∥2(d-2)/\|X\|^2 is an unbiased estimate of the shrinkage factor 1/(1+A)1/(1+A). James–Stein is the Bayes rule with the prior's variance estimated from the data themselves. Hierarchical models do the same with a prior on AA, which is why "partial pooling" beats both treating every unit separately and pooling them all.

Check your understanding

In d=10d=10 dimensions with true mean θ=0\theta=0, what is the risk of the James–Stein estimator, compared with 10 for the MLE?

  1. 0
  2. 8
  3. 2
  4. 10
Show the answer

C. 2

R=d−(d−2)2 E[1/χ102]=10−64/8=2R=d-(d-2)^2\,\E[1/\chi^2_{10}]=10-64/8=2. The gain is largest near the shrinkage target and shrinks as ∥θ∥\|\theta\| grows, but the risk stays below dd everywhere.

Key takeaways

  • For an exponential family, the conjugate prior updates by (χ,ν)↦(χ+∑iT(xi), ν+n)(\chi,\nu)\mapsto(\chi+\sum_iT(x_i),\,\nu+n): pseudo-observations plus real ones. The posterior mean of the mean parameter is linear in the data, and only conjugate priors make it so (Diaconis–Ylvisaker).
  • Normal–Normal: precisions add, means average by precision, and the predictive variance is σ2+τn2\sigma^2+\tau_n^2. Dirichlet–multinomial: additive smoothing (αk+nk)/(α0+n)(\alpha_k+n_k)/(\alpha_0+n), the Pólya urn.
  • Bernstein–von Mises: in regular parametric models the posterior approaches N(θ^,(nI)−1)\mathcal N(\hat\theta,(nI)^{-1}). The Jeffreys prior I(θ)\sqrt{I(\theta)} is parameterization-invariant. Infinite dimensions, misspecification and zero prior mass break the asymptotics.
  • James–Stein: for d≥3d\ge3, shrinking toward a common point beats the MLE everywhere (risk 22 instead of dd at the target). It is empirical Bayes, the logic behind partial pooling.

Check yourself

In the interactive lesson these are scored with your confidence, and what you miss comes back for review.

Question 1

Counts 3,5,43,5,4 are modeled as Poisson(λ\lambda) with a Gamma(2,1)\mathrm{Gamma}(2,1) prior (shape 2, rate 1). What is the posterior?

  1. Gamma(12,3)\mathrm{Gamma}(12,3)
  2. Gamma(14,1)\mathrm{Gamma}(14,1)
  3. Gamma(5,4)\mathrm{Gamma}(5,4)
  4. Gamma(14,4)\mathrm{Gamma}(14,4)
Show the answer

D. Gamma(14,4)\mathrm{Gamma}(14,4)

Add the total count to the shape and the number of observations to the rate: Gamma(2+12, 1+3)\mathrm{Gamma}(2+12,\,1+3), with mean 3.53.5, between the prior mean 22 and the sample mean 44.

Question 2

Prior μ∼N(0,1)\mu\sim\mathcal N(0,1); one observation x=2x=2 with noise variance σ2=1\sigma^2=1. What is the variance of the posterior predictive distribution of the next observation?

  1. 0.50.5
  2. 1.51.5
  3. 11
  4. 22
Show the answer

B. 1.51.5

Precisions add, 1/τ12=1+11/\tau_1^2=1+1, so τ12=0.5\tau_1^2=0.5 (and μ1=1\mu_1=1). The predictive adds the noise variance: 1+0.5=1.51+0.5=1.5.

Question 3

In which situation does the Bernstein–von Mises theorem not guarantee that 95% credible intervals have about 95% frequentist coverage for large nn?

  1. A regular two-parameter model, with a smooth prior positive at the truth
  2. A Beta–Bernoulli model with a Beta(5,5)\mathrm{Beta}(5,5) prior and true rate 0.30.3
  3. A misspecified model, where the posterior concentrates at the KL-closest parameter
  4. A Normal mean with a Normal prior centered far from the truth
Show the answer

C. A misspecified model, where the posterior concentrates at the KL-closest parameter

Under misspecification the posterior's spread no longer matches the sampling variability of the estimator, so credible sets can miscover however large nn is. The other three are regular cases, where the prior's influence fades.

Question 4

Under the conjugate prior with hyperparameters (χ,ν)(\chi,\nu), what is the posterior mean of E[T(X)]\E[T(X)] after nn observations?

  1. χ+∑iT(xi)ν+n\frac{\chi+\sum_iT(x_i)}{\nu+n}
  2. 1n∑iT(xi)\frac1n\sum_iT(x_i)
  3. χ/ν\chi/\nu
  4. χ+∑iT(xi)\chi+\sum_iT(x_i)
Show the answer

A. χ+∑iT(xi)ν+n\frac{\chi+\sum_iT(x_i)}{\nu+n}

Diaconis–Ylvisaker: the posterior mean of the mean parameter is the pooled average of ν\nu pseudo-observations and nn real ones, linear in the sufficient statistic.

Further reading

  • P. Diaconis & D. Ylvisaker, "Conjugate priors for exponential families," Annals of Statistics 7(2), 269–281 (1979).
  • W. James & C. Stein, "Estimation with quadratic loss," Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability 1, 361–379 (1961).
  • B. Efron & C. Morris, "Stein's estimation rule and its competitors—an empirical Bayes approach," Journal of the American Statistical Association 68(341), 117–130 (1973).
  • A. W. van der Vaart, Asymptotic Statistics (Cambridge University Press, 1998), chapter 10.
  • A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari & D. B. Rubin, Bayesian Data Analysis, 3rd ed. (CRC Press, 2013).