Autoencoders & Latent Spaces
Compression as a route to representation.
Squeeze data through a bottleneck and reconstruct it: what does the code learn? For linear maps the answer is exactly the principal subspace, with no bad local minima on the way there. Remove the bottleneck, corrupt the input with noise, and the reconstruction learns something stranger: the gradient of the data's log-density. This lesson proves both theorems, connects autoencoders to rate–distortion theory, and ends with a result that ruled out disentangling factors of variation without assumptions.
What you’ll learn
- State the Baldi–Hornik theorem and sketch why every critical point other than the PCA solution is a saddle.
- Prove Tweedie's formula and deduce that an optimal denoising autoencoder estimates .
- Relate denoising to contractive penalties and sparse autoencoders to dictionary learning, and read autoencoders through rate–distortion theory.
- State the Locatello et al. impossibility theorem for unsupervised disentanglement and the assumptions that restore identifiability.
The landscape of linear autoencoders
Expand . For a fixed decoder, is convex in , and the gradient vanishes at the pseudoinverse: the best encoder projects orthogonally onto the decoder's column space.
Stationarity in gives . Substitute , multiply on the right by and use ; then transpose:
So is an invariant subspace of ; with distinct eigenvalues it is spanned by eigenvectors , and the loss is the variance outside it:
If , some and have . Rotating toward by a small angle keeps a -dimensional subspace and lowers the loss, so the point is not a local minimum:
Kawaguchi (2016) extended the conclusion to deep linear networks of any depth: every local minimum is global, although for depth above three some saddles have no direction of negative curvature, which slows gradient descent. The landscape result says nothing about the basis of the code, which stays arbitrary up to an invertible map. Adding weight decay breaks that symmetry: the principal directions can then be read off the decoder's singular vectors (Kunin et al., 2019).
With and , gradient descent reaches a critical point whose code spans . What is it?
- A saddle: rotating toward lowers the loss by
- A spurious local minimum, the reason linear autoencoders need restarts
- The global minimum, since carries the most variance
- A local maximum
Show the answer
A. A saddle: rotating toward lowers the loss by
Its loss, , exceeds the optimum , and an arbitrarily small rotation toward decreases it. Generic initializations escape such points: gradient descent converges to the PCA subspace.
Denoising autoencoders learn the score
A denoising autoencoder (Vincent et al., 2008) corrupts the input, with , and trains to minimize . Over all functions, squared error is minimized by the conditional mean, . A classical identity says what that mean is.
Write the noisy density as a convolution with the Gaussian kernel , whose gradient is explicit:
Differentiate under the integral:
By Bayes' rule , so dividing by turns the integral into a conditional expectation:
Consequently minimizing the denoising loss over is least-squares regression of onto the score: denoising score matching, the training objective of score-based diffusion models at each noise level (Diffusion Models).
Data are 1-D Gaussian, , and the noise has variance . The optimal denoiser sees . What is ?
Show the answer
1.5
, so , and Tweedie gives . Equivalently , the familiar Gaussian shrinkage: the denoiser pulls toward high density.
Geometrically, near data concentrated on a low-dimensional manifold, noise pushes points off it and pulls them back along the normal directions. For small , a Taylor expansion of the loss gives : denoising is approximately reconstruction plus a contractive penalty on the reconstruction map, the explicit regularizer of contractive autoencoders (Rifai et al., 2011).
Sparse codes, latent spaces and rate–distortion
Sparse coding (Olshausen & Field, 1996) learns an overcomplete dictionary , , by ; on natural image patches its atoms are localized, oriented, bandpass filters resembling simple cells in visual cortex. A sparse autoencoder amortizes the inner optimization with an encoder , or keeps only the largest activations (Makhzani & Frey, 2013). Under incoherence and sparsity assumptions, overcomplete dictionaries are provably recoverable in polynomial time (Arora, Ge & Moitra, 2014). With dictionaries tens of times larger than the activation dimension, sparse autoencoders have become a main tool for decomposing language-model activations into interpretable features (Bricken et al., 2023; Cunningham et al., 2023; see Mechanistic Interpretability).
Why not sample new data by decoding random codes? The encoder maps the data onto an irregular, low-volume region of , and decoding points outside it gives garbage. Variational autoencoders fix this by pulling the code distribution toward a prior (Variational Autoencoders & the ELBO). Masked autoencoders (He et al., 2022) took the opposite route, using reconstruction of 75% hidden image patches purely as a pretraining task.
Disentanglement and identifiability
A recurring hope is a code whose coordinates are the true factors of variation: with independent factors (pose, lighting, identity), recovered one per latent unit. Methods such as the β-VAE (Higgins et al., 2017) push toward factorized codes. Can any unsupervised method guarantee success?
For , any rotation preserves the distribution, and a rotation not aligned with the axes mixes every coordinate into every other. For a general factorized density, map each coordinate to a standard normal with its CDF, rotate, and map back:
Now the data cannot tell the two models apart: the disentangled pair and the entangled pair generate exactly the same distribution of . Any method that sees only must treat them alike.
The Gaussian's rotation invariance is the crux. Linear ICA escapes it: independent non-Gaussian sources mixed linearly are identifiable up to permutation and scaling (Comon, 1994). Nonlinear ICA is not identifiable in general (Hyvärinen & Pajunen, 1999), but becomes identifiable with an auxiliary variable that modulates the factors, such as time or a class label (Hyvärinen, Sasaki & Turner, 2019; Khemakhem et al., 2020).
Locatello et al. also trained thousands of models across methods and datasets and found that differences between disentanglement methods were dominated by random seeds and hyperparameters, and that, without supervision, there was no reliable way to pick a well-disentangled model. Inductive biases and some supervision are not optional extras; they are what makes the problem well posed.
Key takeaways
- Baldi–Hornik: for linear autoencoders the full-rank critical points are projections onto eigenvectors; only the top- subspace is a minimum, all others are saddles, and the code's basis stays arbitrary.
- Tweedie: , so optimal denoisers estimate the score; this is denoising score matching, the basis of diffusion models.
- Small-noise denoising approximates a contractive penalty; sparse autoencoders amortize sparse coding and now decompose language-model activations into features.
- Rate–distortion theory bounds every autoencoder-as-compressor; for Gaussian data, reverse water-filling drops low-variance components, like PCA.
- Locatello et al.: distribution-preserving entangling bijections make unsupervised disentanglement non-identifiable; non-Gaussianity, auxiliary variables or supervision restore identifiability.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
In a linear autoencoder, the decoder (full rank) is fixed. Which encoder minimizes the squared reconstruction error?
- , so that projects orthogonally onto the column space of
- Any with
Show the answer
B. , so that projects orthogonally onto the column space of
The gradient vanishes when . Then , the orthogonal projector onto : each input is reconstructed by its closest point in the decoder's range.
A denoising autoencoder is trained to optimality with small Gaussian noise of variance . What does approximate?
- The noise vector that was added
- Zero, since the optimal reconstruction is the identity
- , a vector pointing toward higher data density
- The first principal component of the data
Show the answer
C. , a vector pointing toward higher data density
Tweedie's formula gives , and as . This is why denoisers can serve as score estimators for diffusion models.
A 3-D Gaussian source has covariance eigenvalues , and . Reverse water-filling with level gives which rate, in bits per vector?
- bit
- bits
- bits
- bits
Show the answer
D. bits
Components with get : bits. The third component () gets no bits and contributes distortion , for total distortion .
Why can no purely unsupervised method guarantee disentangled codes when the true factors are independent?
- Current networks are too small to represent the true generator
- Entangling bijections preserve the factor distribution, so an entangled model generates exactly the same data distribution as the true one
- The factors are never truly independent in real data
- Disentanglement metrics are noisy
Show the answer
B. Entangling bijections preserve the factor distribution, so an entangled model generates exactly the same data distribution as the true one
With , the models and are observationally equivalent, so the data alone cannot favor the disentangled one. Assumptions or supervision must break the tie.
Further reading
- P. Baldi & K. Hornik, "Neural networks and principal component analysis: Learning from examples without local minima," Neural Networks 2, 53–58 (1989).
- P. Vincent, H. Larochelle, Y. Bengio & P.-A. Manzagol, "Extracting and composing robust features with denoising autoencoders," ICML (2008).
- G. Alain & Y. Bengio, "What regularized auto-encoders learn from the data-generating distribution," Journal of Machine Learning Research 15 (2014).
- F. Locatello, S. Bauer, M. Lucic, G. Rätsch, S. Gelly, B. Schölkopf & O. Bachem, "Challenging common assumptions in the unsupervised learning of disentangled representations," ICML (2019).
- I. Goodfellow, Y. Bengio & A. Courville, Deep Learning, ch. 14 "Autoencoders" (MIT Press, 2016).