Representation Learning
Embeddings, autoencoders and self-supervision.
Train a network to sort a million photos into a thousand classes, cut off its last layer, and what remains is a feature extractor that helps on tasks it never saw. Representation learning makes that side effect the goal: map raw data to vectors in which the structure you care about becomes simple, ideally linear. This lesson derives the three main recipes: embeddings, autoencoders and contrastive self-supervision.
What you’ll learn
- Describe what makes a representation useful (distributed, linearly accessible, invariant, sparse) and read a network's last hidden layer as a learned feature map.
- Derive the skip-gram negative-sampling objective, its gradient and its cost, and state its link to pointwise mutual information.
- Prove that an optimal linear autoencoder spans the principal subspace, and explain how denoising, sparse and masked autoencoders avoid the identity map.
- Write the InfoNCE loss as a softmax cross-entropy, explain collapse, compare SimCLR, MoCo and BYOL, and evaluate representations with linear probes.
Features you do not have to design
For decades, machine learning meant feature engineering: experts turned pixels into edge histograms and text into word counts, and a linear model did the rest. A deep network learns the features as well. Write a classifier as , where is everything up to the last hidden layer. Training all of it end to end shapes until a linear readout suffices. That is representation learning, and is a point in a learned latent space.
The network's output is , computed from the last hidden layer , and it classifies every training point correctly. What follows about the training points?
- The two spirals are linearly separable in the input plane
- Their representations are linearly separable: a line in the plane splits the classes
- The map is invertible, so no information about is lost
- and are statistically independent features
Show the answer
B. Their representations are linearly separable: a line in the plane splits the classes
The output thresholds at zero, a line in the plane, and 100% accuracy says that line separates the classes there. The earlier layers did the hard work of bending the input space until a line suffices.
| Property | What it means | What encourages it |
|---|---|---|
| Distributed | a concept is a pattern over many units; units can express up to patterns, and similar inputs share units | any dense hidden layer or embedding |
| Linearly accessible | the factors you care about are close to linear functions of the code | training with a linear head; measured by linear probes |
| Invariant | nuisance changes (crop, lighting, paraphrase) move the code little; task-relevant changes move it a lot | data augmentation, contrastive objectives |
| Sparse | each input activates few units, so units can specialize | penalties, ReLUs, -sparse codes |
| Disentangled | separate factors of variation (pose, identity, lighting) occupy separate directions | inductive biases in architecture and data; provably impossible to guarantee without them (Deep Layer) |
Contrast the simplest code for symbols, the one-hot vector . Distinct one-hot vectors are orthogonal, , so everything learned about "cat" says nothing about "kitten". A dense code in with lets similar symbols share directions. (The debate over distributed versus local codes is told in Symbols vs. Connections.) The next step learns such codes from raw text, with no labels at all.
Embeddings and skip-gram with negative sampling
An embedding layer stores one vector per symbol in a matrix . Feeding it a one-hot vector computes , which just selects row , so a gradient step touches only the rows of the symbols in the batch.
To learn word vectors without labels, use the distributional hypothesis, "you shall know a word by the company it keeps" (Firth, 1957): train each word to predict its neighbors. The skip-gram model of word2vec (Mikolov et al., 2013) gives each word a center vector and a context vector , and for every word within a few positions of it models
The normalizer sums over the whole vocabulary: work per training pair, prohibitive for .
Write and , and use and :
Every candidate gets "prediction minus label", with label 1 for the observed context and 0 for the negatives. The chain rule through the dot products gives
A step costs instead of . Descent pulls toward its real contexts and pushes it away from random words, so words that share contexts end up with similar vectors. With and a million-word vocabulary the saving is
What does the optimum encode? If the negatives come from the unigram distribution and is large enough for every product to be set freely, the optimum is the pointwise mutual information shifted by (Levy & Goldberg, 2014; proved in the Deep Layer). Skip-gram is implicitly factorizing a word–context matrix.
import numpy as np
def sgns_step(V, U, w, c, noise, k=5, eta=0.025, rng=None):
"""One SGD step of skip-gram with negative sampling on the pair (w, c).
V, U: (|V|, d) center and context vectors; noise: P_n, proportional to count**0.75."""
rng = rng or np.random.default_rng()
rows = np.concatenate(([c], rng.choice(len(noise), size=k, p=noise)))
labels = np.r_[1.0, np.zeros(k)] # observed context, then k negatives
s = U[rows] @ V[w] # (k+1,) scores u_j . v_w
g = 1.0 / (1.0 + np.exp(-s)) - labels # dl/ds: prediction minus label
grad_v = g @ U[rows] # dl/dv_w, before U changes
np.add.at(U, rows, -eta * np.outer(g, V[w])) # dl/du_j = g_j v_w; repeats add up
V[w] -= eta * grad_vOne SGD step on one (word, context) pair. Production code also subsamples frequent words, decays linearly and runs many such steps in parallel. np.add.at matters because the same negative can be drawn twice.
Autoencoders: compression as a teacher
An autoencoder has an encoder and a decoder , trained to reconstruct its input: . With (undercomplete) the code must compress. Without a bottleneck or some other constraint, the identity map reconstructs perfectly and nothing is learned. So what does the simplest bottleneck learn?
Stack the centered data as the columns of . The reconstruction passes through a -dimensional code, so its rank is at most :
By the Eckart–Young theorem (proved in PCA, t-SNE & UMAP), the best rank- approximation keeps the top singular directions, , where holds the top- eigenvectors of the covariance. Its loss is the variance left out:
A linear autoencoder attains the bound. Since makes unique, every minimizer reconstructs the data exactly as does:
The code itself is fixed only up to an invertible matrix : has the same product and the same loss. A linear autoencoder finds the principal subspace, not the individual principal components; recovering those takes extra structure (Deep Layer).
Centered data in have covariance eigenvalues . A linear autoencoder with a 2-dimensional code is trained to its global minimum. What percentage of the total variance is lost in reconstruction?
Show the answer
20 %
The loss is out of a total variance of , so 20%. The 80% that survives is the explained-variance ratio of a 2-component PCA, because the two are the same model.
Nonlinear encoders and decoders can do far better: Hinton and Salakhutdinov (2006) trained deep autoencoders whose low-dimensional codes beat PCA's on images and documents. And a bottleneck is not the only way to rule out the identity map:
| Variant | Objective | What it learns |
|---|---|---|
| Sparse | reconstruct, plus a penalty (or keep only the largest units) | an overcomplete dictionary of parts, each input built from a few of them; now used to extract interpretable features from language models |
| Denoising (Vincent et al., 2008) | reconstruct the clean from a corrupted | a vector field pointing back toward the data, proportional to for small Gaussian noise: the idea behind diffusion models |
| Masked (BERT, 2019; MAE, 2022) | reconstruct masked-out parts: BERT hides 15% of tokens, MAE 75% of image patches | features good enough to infer missing parts from context |
| Variational | reconstruct, plus a KL term pulling codes toward a prior | a smooth latent space you can sample from, a generative model (Variational Autoencoders & the ELBO) |
Self-supervision and the contrastive loss
Self-supervised learning manufactures labels from the data itself: predict a word's neighbors (word2vec), a masked token (BERT), the next token (GPT), or, in contrastive learning, which of many candidates is another view of the same example. For images, two random augmentations of one photo (crops, color jitter, blur) form a positive pair, and views of other photos are negatives. An encoder maps each view to a unit vector , and the InfoNCE loss (van den Oord, Li & Vinyals, 2018) asks the positive to win a softmax over candidates:
Index the candidates with the positive at , and let . The loss is the cross-entropy of a softmax classifier whose correct class is :
Its gradient is prediction minus target, as for any softmax cross-entropy:
Through , descent moves the anchor toward its positive and away from the -weighted average of all candidates. Negatives that already look similar get large and the strongest push, and a smaller concentrates the push on the hardest ones:
Why should this produce good features? Minimizing maximizes a lower bound on the mutual information between the two views, a bound that can never exceed (proved in the Deep Layer). More negatives raise the ceiling, one reason contrastive methods like large batches.
| Method | Negatives | Key ingredients |
|---|---|---|
| SimCLR (Chen et al., 2020) | the other views in a batch of images | strong augmentation; an MLP projection head that is discarded after training; batches up to 8,192 |
| MoCo (He et al., 2020) | a queue of 65,536 keys from earlier batches | a momentum encoder whose weights track the query encoder's, with |
| BYOL (Grill et al., 2020) | none | an online network with an extra predictor regresses the output of a slowly moving average of itself, with a stop-gradient on that target |
| CLIP (Radford et al., 2021) | the other images and captions in the batch | positives are image–caption pairs; see Contrastive Vision–Language Models |
Measuring a representation
How do you compare two representations? The standard protocol is the linear probe (Alain & Bengio, 2016): freeze the encoder, fit logistic regression on its features for a labeled task, and report test accuracy. It measures what is linearly decodable, exactly what a downstream linear head can use. Other protocols are -nearest-neighbor accuracy in the embedding space and full fine-tuning. By linear-probe accuracy on ImageNet, SimCLR matched a supervised ResNet-50 in 2020, using a ResNet-50 four times as wide.
Think first SimCLR trains through a projection head but probes the layer before it, and that layer gives markedly better probes than . Why might the head's output be worse?
The contrastive loss rewards invariance to the augmentations, so learns to discard whatever the augmentations change, such as color and exact position. Some downstream tasks need that information. The head absorbs the invariance, and keeps more of the input.
Key takeaways
- A deep network is a learned feature map plus a simple readout, and training makes the task linear in . Good representations are distributed, linearly accessible, invariant to nuisances and often sparse.
- Embeddings are lookup tables trained by gradient descent. Skip-gram with negative sampling replaces a -way softmax by logistic regressions, costs per pair and at its optimum encodes .
- An optimal linear autoencoder spans the top- principal subspace, in an arbitrary basis, with loss . Sparse, denoising and masked autoencoders change the task so that copying fails.
- InfoNCE is softmax cross-entropy over one positive and negatives; it lower-bounds mutual information and is capped at . Negatives, stop-gradients or variance penalties prevent collapse.
- Linear probes measure what is linearly decodable from a representation, not what the model uses.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
In word2vec's negative sampling, negatives are drawn with probability proportional to rather than to the count . What does the exponent do?
- It samples frequent words even more often than their frequency
- It flattens the distribution: rare words are sampled more often than their frequency, frequent words less
- It makes negative sampling compute the exact softmax
- It removes stop words from the training text
Show the answer
B. It flattens the distribution: rare words are sampled more often than their frequency, frequent words less
A power below 1 compresses ratios: for counts and , the sampling ratio drops from to . Mikolov et al. found that worked better than both the unigram and the uniform distribution.
A linear autoencoder with a 3-dimensional code is trained to its global minimum on centered data whose covariance has distinct eigenvalues. Which statement is true?
- Its encoder's rows are the top 3 principal components, in order
- Its reconstruction error is lower than that of 3-component PCA
- It captures curved structure in the data that PCA misses
- Its codes span the same subspace as the top 3 PCA scores, but in an arbitrary basis
Show the answer
D. Its codes span the same subspace as the top 3 PCA scores, but in an arbitrary basis
is the projection onto the top-3 principal subspace, but is equally optimal for every invertible , so each code coordinate is an arbitrary mixture of the principal components.
A contrastive model is trained with InfoNCE using candidates per anchor (one positive, 255 negatives), and its loss reaches nats. What lower bound on the mutual information between the two views does this certify?
- nats
- nats
- nats
- nats
Show the answer
C. nats
nats. Even a perfect loss of certifies only nats, whatever the true mutual information.
You train a Siamese encoder only to maximize the cosine similarity between two augmented views of the same image, with no negatives and no other safeguard. What is the likely outcome?
- Collapse: every image maps to nearly the same vector, which makes every pair perfectly similar
- Excellent features, since the two views share the object's identity
- Divergence, because cosine similarity is unbounded
- An encoder that exactly undoes the augmentations
Show the answer
A. Collapse: every image maps to nearly the same vector, which makes every pair perfectly similar
A constant output maximizes the similarity of every pair, so nothing forces the encoder to tell images apart. Negatives (InfoNCE), an asymmetric predictor with a stop-gradient (BYOL, SimSiam) or variance penalties (VICReg) rule this solution out or make it unreachable in practice.
A linear probe on layer 6 of a language model predicts part-of-speech tags with 95% accuracy. What can you conclude?
- The model relies on part of speech to make its predictions
- Layer 6 encodes nothing except part of speech
- Part-of-speech information is linearly decodable from layer 6
- A nonlinear probe would necessarily do worse
Show the answer
C. Part-of-speech information is linearly decodable from layer 6
A probe shows that information is present in a linearly readable form. Whether the model uses it needs interventions, such as removing that direction and measuring the effect on the model's outputs.
Go deeper
- Word & Item Embeddings · word2vec, GloVe and the geometry of meaning.
- Autoencoders & Latent Spaces · Compression as a route to representation.
- Self-Supervised & Contrastive Learning · Learning from the data’s own structure.
Further reading
- Y. Bengio, A. Courville & P. Vincent, "Representation learning: A review and new perspectives," IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1798–1828 (2013).
- T. Mikolov, I. Sutskever, K. Chen, G. Corrado & J. Dean, "Distributed representations of words and phrases and their compositionality," NeurIPS (2013).
- T. Chen, S. Kornblith, M. Norouzi & G. Hinton, "A simple framework for contrastive learning of visual representations," ICML (2020).
- I. Goodfellow, Y. Bengio & A. Courville, Deep Learning, ch. 14 "Autoencoders" and ch. 15 "Representation Learning" (MIT Press, 2016).