Gaussian Processes & Bayesian ML
Distributions over functions and calibrated uncertainty.
Put a Gaussian prior on the weights of a linear model and something remarkable happens: predictions never need the weights, only similarities between inputs. Replace those by a kernel and you have a Gaussian process, a prior over functions with error bars in closed form. The same object describes infinitely wide neural networks, drives Bayesian optimization and sets the standard for calibrated uncertainty, at a price of .
What you’ll learn
- Derive Bayesian linear regression, show that its predictions depend on the inputs only through a kernel, and recover ridge regression as its posterior mean.
- Define Gaussian processes and derive the posterior mean and variance by Gaussian conditioning, with the Cholesky algorithm and its cost.
- Choose kernels, learn hyperparameters by maximizing the marginal likelihood, and state Bochner's theorem and posterior contraction rates.
- Explain sparse and random-feature approximations, the regret bound of GP-UCB, and the infinite-width correspondence between neural networks and Gaussian processes.
From weights to functions
Model with prior and observations , , where row of is . Completing the square in gives a Gaussian posterior:
With the posterior mean is the ridge solution, with penalty on the sum of squared errors: a prior's strength is a regularization constant.
The prediction at is . A matrix identity, , removes the weights entirely:
Only the kernel remains, so the features may be infinite-dimensional as long as is computable: the kernel trick, applied to a Bayesian model.
The posterior in closed form
With prior mean and noisy observations , the training outputs and the test value are jointly Gaussian:
The conditional of a jointly Gaussian vector is Gaussian, with the Schur complement as its covariance:
The mean is a weighted sum of kernels centered on the data, with . The variance never involves : it shrinks near the inputs and returns to the prior's far from them.
function GP-PREDICT(X, y, kernel k, noise σ², test point x*)
L ← CHOLESKY(K + σ²I) // K[i][j] = k(xi, xj); about n³/3 flops
α ← Lᵀ \ (L \ y) // two triangular solves: O(n²)
mean ← k*ᵀ α // k*[i] = k(xi, x*): O(n)
v ← L \ k* // O(n²)
var ← k(x*, x*) − vᵀ v
logZ ← −½ yᵀα − Σi log L[i][i] − (n/2) log 2π // log marginal likelihood
return mean, var, logZRasmussen and Williams's Algorithm 2.1. Training costs time and memory, then each prediction for the mean and for the variance. A few tens of thousands of points are feasible exactly; millions are not.
One observation, at . The kernel is and the noise variance is . What is the posterior mean at ?
Show the answer
0.6
. The variance there is , while at it is , about the noise level. Two units away the mean has decayed to .
Kernels and their hyperparameters
| Kernel | , with | Sample functions |
|---|---|---|
| Squared exponential (RBF) | infinitely differentiable | |
| Matérn | continuous, nowhere differentiable (Ornstein–Uhlenbeck) | |
| Matérn | once differentiable | |
| Matérn | twice differentiable | |
| Periodic (one dimension) | exactly periodic with period | |
| Linear | straight lines: Bayesian linear regression |
Hyperparameters such as , and are usually fit by maximizing the log marginal likelihood, which balances fit against complexity, the same Occam's razor as Bayes factors:
Its gradient is , also . The objective is non-convex, so restarts or priors on hyperparameters help.
Scaling up, deciding, and deep networks
Three families of methods attack the cost:
- Inducing points. Summarize the function by its values at pseudo-inputs. Titsias's (2009) variational bound trains in , and stochastic variational GPs (Hensman, Fusi & Lawrence, 2013) use minibatches.
- Random features. By Bochner's theorem, random Fourier features turn a GP into Bayesian linear regression with cost.
- Structure and iteration. Kernel interpolation on grids (KISS-GP; Wilson & Nickisch, 2015) and conjugate-gradient solvers on GPUs (GPyTorch; Gardner et al., 2018) compute the same quantities with matrix–vector products.
What can go wrong if GP-UCB is run with , that is, always picking the maximizer of the posterior mean?
- Nothing: the posterior mean is the best estimate, so following it is optimal
- It stops exploring, can keep sampling a suboptimal point forever, and its regret can grow linearly
- Its computation becomes cubic in
- It samples uniformly at random
Show the answer
B. It stops exploring, can keep sampling a suboptimal point forever, and its regret can grow linearly
Without the bonus, regions the model is unsure about are never tried, so an early, misleading posterior can lock the search onto a local optimum: regret in the worst case. The bonus is optimism in the face of uncertainty, as in UCB1 for bandits.
Neural networks connect back to GPs. Neal (1996) showed that a one-hidden-layer network with independent priors on its weights, scaled by the inverse width, converges to a Gaussian process as the width grows, by the central limit theorem; the result extends to deep networks, with kernels computed layer by layer (Lee et al., 2018; Matthews et al., 2018). Under gradient-descent training, infinitely wide networks evolve as kernel regression with the neural tangent kernel (Jacot, Gabriel & Hongler, 2018). Finite networks, however, learn their features, which no fixed kernel can do, so these limits explain some behaviors of deep learning but not its representation learning.
In practice, uncertainty for deep networks comes from approximations: Laplace approximations around a trained network, Monte Carlo dropout (Gal & Ghahramani, 2016) and deep ensembles (Lakshminarayanan, Pritzel & Blundell, 2017), which are often the best calibrated though not Bayesian in the strict sense. All of them degrade under distribution shift (Ovadia et al., 2019).
Key takeaways
- Bayesian linear regression predicts through the kernel alone, and its posterior mean is ridge regression with ; letting be any valid kernel gives a Gaussian process.
- The GP posterior has mean and variance , which does not depend on . Exact inference costs time and memory.
- Kernels encode assumptions such as smoothness, periodicity and additivity. The marginal likelihood trades fit against complexity, Bochner's theorem yields random features, and contraction is fastest, at the minimax rate, when the prior's smoothness matches the truth's.
- Inducing points, random features and iterative solvers scale GPs. GP-UCB achieves sublinear regret through information gain, and infinitely wide networks are GPs (NNGP, NTK), though finite networks learn features that fixed kernels cannot.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
In Bayesian linear regression with prior and noise variance , the posterior mean equals ridge regression minimizing with…
Show the answer
C.
The negative log-posterior is ; multiply by . A tighter prior (small ) or noisier data (large ) means more shrinkage.
Why does the GP posterior variance at a test point not depend on the observed outputs ?
- Because the outputs are assumed to be noise-free
- Because the variance is computed from the prior only
- Because the kernel is stationary
- Because a Gaussian's conditional covariance depends only on the joint covariance, not on the observed values
Show the answer
D. Because a Gaussian's conditional covariance depends only on the joint covariance, not on the observed values
involves only inputs and hyperparameters. The data's values enter through the hyperparameters if those are fit by marginal likelihood, but not directly.
Exact GP training uses a Cholesky factorization of the matrix . If the training set doubles, the training time grows by a factor of about…
- 2
- 4
- 8
- 16
Show the answer
C. 8
Cholesky takes about floating-point operations, so doubling multiplies the time by , and memory by 4.
The true regression function on has Sobolev smoothness . Which Matérn prior smoothness gives the fastest posterior contraction, and at what rate?
- , rate
- (squared exponential with a fixed length scale), rate
- , rate
- Any , rate
Show the answer
A. , rate
The rate is maximized at : , the minimax rate. A rougher prior () gives ; smoother ones are slower too.
Further reading
- C. E. Rasmussen & C. K. I. Williams, Gaussian Processes for Machine Learning (MIT Press, 2006).
- R. M. Neal, Bayesian Learning for Neural Networks (Springer, 1996).
- M. Titsias, "Variational learning of inducing variables in sparse Gaussian processes," AISTATS (2009).
- N. Srinivas, A. Krause, S. M. Kakade & M. Seeger, "Gaussian process optimization in the bandit setting: No regret and experimental design," ICML (2010).
- A. Rahimi & B. Recht, "Random features for large-scale kernel machines," NeurIPS (2007).