Self-Supervised & Contrastive Learning
Learning from the data’s own structure.
A contrastive model never sees a label, yet a linear classifier on its features can rival a fully supervised network. Its loss is a softmax over a batch, and that softmax is secretly a bound on mutual information, one that can never certify more than the logarithm of the batch size. This lesson proves the bound and its ceiling, then turns to the sharper question the bound cannot answer: why do these features transfer?
What you’ll learn
- Define the InfoNCE objective precisely, prove , and identify the optimal critic.
- Explain the ceiling and the formal limits on estimating mutual information from samples.
- State the alignment–uniformity limit of the contrastive loss and the role of the temperature.
- Derive the spectral contrastive loss as a low-rank factorization of the augmentation graph, and compare non-contrastive methods.
InfoNCE bounds mutual information
The negatives are independent of , so by the chain rule they add no information:
For any conditional density , the Barber–Agakov bound holds, because the gap is an expected KL divergence:
Choose by reweighting the product of marginals with the critic's softmax. It is normalized: under the softmax weights are exchangeable and sum to , so each has mean :
Substituting, the product of marginals cancels and what remains is minus the loss. The softmax weight is at most , so the right side is at most :
For the optimal critic: the posterior probability that candidate is the positive is with , and a cross-entropy is minimized by the true posterior. With , the law of large numbers sends , leaving .
A contrastive run uses candidates per anchor. Whatever the encoder, what is the largest mutual information, in nats, that its InfoNCE bound can certify?
Show the answer
8.3
nats, exactly 12 bits. Two views of a photograph may share far more information than that, so the bound saturates long before it measures it.
The ceiling is not a quirk of InfoNCE. McAllester and Stratos (2020) proved that no distribution-free, high-confidence lower bound on mutual information estimated from samples can exceed . Variational bounds without the ceiling, such as the Donsker–Varadhan bound used by MINE or the Nguyen–Wainwright–Jordan bound, pay with estimator variance that grows exponentially in the true information (Poole et al., 2019; Song & Ermon, 2020).
Alignment, uniformity and temperature
The proof is one line: with bounded by , converges to by the law of large numbers, since the single positive term vanishes after dividing by . The decomposition explains the two forces: positives pulled together, everything else spread out as evenly as the sphere allows, so that no information-carrying directions collapse. Wang and Isola's direct estimates of the two terms track linear-probe accuracy across many trained encoders.
The temperature sets how the push is distributed. The gradient weight on negative is its softmax probability, , so a small concentrates almost all of it on the hardest negatives. That sharpens local separation, but other images of the anchor's own class are also negatives, and pushing hard on them can split semantically similar examples apart.
With unit vectors and similarities , consider as . What does the scaled loss become?
- The average similarity to all negatives
- : a hinge on the single hardest negative
- Zero for every encoder
- The similarity to the easiest negative
Show the answer
B. : a hinge on the single hardest negative
(log-sum-exp tends to the maximum), so , a triplet-style hinge on the hardest negative. At large the weights flatten toward all negatives equally.
Why do contrastive features transfer?
Mutual information alone cannot be the explanation. It is invariant under invertible maps, so an encoder that scrambles its input bijectively keeps all the information while destroying linear separability; Tschannen et al. (2020) also found that tighter MI estimators can yield worse representations. What matters is which features the encoder class and the augmentations make easy. Two lines of theory make this precise.
Build the augmentation graph on augmented inputs: is the probability that and are two views of the same natural example, and . For an encoder into , define the spectral contrastive loss:
Collect the rows into a matrix and normalize the adjacency, . Both sums become matrix expressions:
So minimizing the loss over all encoders is the best rank- approximation of : by Eckart–Young, spans its top- eigenvectors, the same spectral embedding as in spectral clustering. If augmentations rarely cross class boundaries, these eigenvectors are nearly constant on each class, and HaoChen et al. bound the linear-probe error by the cross-class augmentation probability divided by a spectral gap of the graph.
The picture these results share: augmentations define which inputs count as "the same", the loss learns a spectral embedding of that similarity graph, and a linear probe succeeds when classes are nearly disconnected in it. The graph is never written down, and the encoder's inductive biases decide how it is extrapolated to unseen inputs, which current theory captures only partially.
Beyond negatives, and the state of the art
| Method | Objective | What prevents collapse |
|---|---|---|
| BYOL (2020), SimSiam (2021) | predict the other view's embedding: | the predictor and the stop-gradient (plus a moving-average target in BYOL); linear analyses show collapse is avoided by the learning dynamics, not by the loss (Tian, Chen & Ganguli, 2021) |
| Barlow Twins (2021) | push the cross-correlation matrix of the two views' embeddings toward : | decorrelated dimensions cannot all carry the same signal |
| VICReg (2022) | invariance (squared distance) + a hinge keeping each dimension's standard deviation above 1 + off-diagonal covariance penalty | the variance term forbids constant outputs |
| DINO (2021) | a student matches a momentum teacher's sharpened, centered softmax outputs on different crops | centering and sharpening balance each other |
| Masked modeling (BERT 2019, MAE 2022) | reconstruct hidden tokens or patches | the target is the input itself, so a constant code cannot succeed |
Key takeaways
- InfoNCE with candidates satisfies ; the optimal critic is the log density ratio, and the bound is tight only as .
- The bound is capped at , and no high-confidence distribution-free estimator does better than ; uncapped variational bounds pay in variance.
- As the number of negatives grows, the loss splits into alignment and uniformity on the sphere; small temperatures focus on the hardest negatives.
- Mutual information cannot explain transfer by itself. Latent-class bounds and the spectral view, a low-rank factorization of the augmentation graph, tie the loss to linear-probe error.
- Non-contrastive methods avoid collapse through predictors and stop-gradients, decorrelation or variance penalties; masked modeling reconstructs the input instead.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
InfoNCE is run with a single negative per anchor (). Whatever the critic, what is the most mutual information it can certify?
- nats, i.e. one bit
- nats
- Unlimited, given a perfect critic
- Nothing: one negative gives no bound
Show the answer
A. nats, i.e. one bit
: the task is a binary choice, and a perfect score proves only that the views share at least one bit.
Which critic minimizes the InfoNCE loss?
- Any critic that is symmetric in and
- for any function
Show the answer
D. for any function
The loss is the cross-entropy of identifying the positive; the true posterior is with , which the softmax reproduces exactly when up to a term depending only on , since that term cancels.
In the large- limit of the contrastive loss, what does the uniformity term reward?
- Mapping positives to the same point
- Embeddings spread as evenly as possible over the unit sphere
- Embeddings concentrated near one pole
- Small embedding norms
Show the answer
B. Embeddings spread as evenly as possible over the unit sphere
is small when typical pairs of embeddings have low similarity, which on the sphere means a near-uniform distribution; it is what prevents collapse.
Minimizing the spectral contrastive loss over all encoders into is equivalent to which problem?
- Maximizing the mutual information between views
- -means clustering of the augmented inputs
- The best rank- approximation of the normalized augmentation-graph adjacency
- PCA of the raw input pixels
Show the answer
C. The best rank- approximation of the normalized augmentation-graph adjacency
, so the optimum spans the top- eigenvectors of , a spectral embedding of the augmentation graph.
Further reading
- A. van den Oord, Y. Li & O. Vinyals, "Representation learning with contrastive predictive coding," arXiv:1807.03748 (2018).
- B. Poole, S. Ozair, A. van den Oord, A. A. Alemi & G. Tucker, "On variational bounds of mutual information," ICML (2019).
- T. Wang & P. Isola, "Understanding contrastive representation learning through alignment and uniformity on the hypersphere," ICML (2020).
- J. Z. HaoChen, C. Wei, A. Gaidon & T. Ma, "Provable guarantees for self-supervised deep learning with spectral contrastive loss," NeurIPS (2021).
- R. Balestriero et al., "A cookbook of self-supervised learning," arXiv:2304.12210 (2023).