Loss Landscapes & Initialization
Saddle points, flat minima and signal propagation.
A network's loss lives in millions of dimensions, and for decades the worry was that gradient descent would get trapped in bad local minima. The modern picture is stranger: saddles rather than minima slow training, minima are joined by low-loss paths, training hovers at the edge of instability, and very wide networks barely move from where they start. Before any of that, the initialization decides whether a signal survives the trip through the layers at all.
What you’ll learn
- Derive the mean-field length and correlation maps of a random network, locate the edge of chaos for ReLU and tanh, and explain dynamical isometry.
- Explain why saddle points rather than poor local minima dominate high-dimensional landscapes, and state what is proven about local minima.
- Derive the stability threshold , describe the edge of stability, and prove that sharpness is not invariant under reparameterization (Dinh et al., 2017).
- Describe mode connectivity and loss-surface visualization, and state the neural tangent kernel limit with its linear convergence guarantee.
Signals at initialization: order, chaos and isometry
Before training starts, a random network must pass signals forward and gradients backward without collapsing or exploding. Mean-field theory makes this exact as the width tends to infinity (Poole et al., 2016; Schoenholz et al., 2017). Take weights and biases , and track two numbers through the layers: the mean squared pre-activation of one input, , and the correlation between the pre-activations of two inputs.
At large width each is Gaussian given the previous layer, a sum of many independent terms, with variance set by the previous layer's mean squared activation. With :
For two inputs the pair is jointly Gaussian. At the fixed point of the length map, the correlation follows a map with , and its slope at 1 decides the fate of nearby inputs:
The same multiplies the squared norm of the backpropagated gradient at each layer. If (ordered phase), correlations converge to 1 exponentially fast, so deep layers see all inputs alike, and gradients vanish. If (chaotic phase), nearby inputs decorrelate exponentially and gradients explode. At , the edge of chaos, both depth scales diverge:
For tanh with , the transition is at , moving to larger when . For ReLU, and , so criticality is exactly He initialization:
Think first Derive from the expansion of near 1.
Write , so with . For , , so and . Iterating the exact map from agrees: at against the asymptotic , and the ratio tends to 1.
Criticality controls averages. Dynamical isometry asks for more: that all singular values of the input–output Jacobian concentrate near 1, so that every direction propagates. Deep linear networks with orthogonal weights achieve it exactly, and their learning time can be independent of depth, unlike with Gaussian weights (Saxe, McClelland & Ganguli, 2014). Random-matrix calculations show that ReLU networks cannot achieve it, even with orthogonal weights, while tanh networks with orthogonal weights near the edge of chaos can (Pennington, Schoenholz & Ganguli, 2017). With a matching orthogonal initialization for convolutions, Xiao et al. (2018) trained plain CNNs 10,000 layers deep.
Critical points: saddles, minima and what is proven
A critical point's index is its fraction of negative Hessian eigenvalues. For random Gaussian functions in high dimension, critical points at high loss have large index and only those near the bottom have index near 0 (Bray & Dean, 2007): a minimum needs all eigenvalues positive at once, so a random critical point far above the minimum is almost surely a saddle. Dauphin et al. (2014) found the same index–loss relation in neural networks, argued that saddles and the plateaus around them, not poor local minima, are what slows training, and proposed a saddle-free Newton method that rescales the gradient by .
Saddles slow gradient descent but rarely stop it: it escapes strict saddles from almost every start, and perturbations make the escape provably fast (Optimization & Convexity). Near a saddle the unstable coordinate grows by a factor of about per step, so the time spent there scales like , where is the starting distance from the saddle's stable manifold. Adaptive methods change that dependence, at least when the escape direction lines up with the coordinates.
Gradient descent with starts at distance instead of from the stable manifold of a strict saddle whose negative curvature is . About how much longer does it stay near the saddle?
- About 10 times longer
- About times longer
- About more steps
- No longer: gradient descent never gets stuck at saddles
Show the answer
C. About more steps
The unstable coordinate is multiplied by per step, so leaving a fixed neighborhood from distance takes about steps, and a tenfold smaller adds about 47. In the widget, from , SGD arrives at step 109 from , 151 from and 226 from .
Sharpness: the edge of stability and flat minima
Near a minimum, , and gradient descent multiplies the component of along an eigenvector of eigenvalue by . It is stable iff every , that is, iff the sharpness . With heavy-ball momentum the condition on a quadratic becomes . Classical analyses keep the step below the threshold everywhere. What actually happens in full-batch training of networks is different (Cohen, Kaur, Li, Kolter & Talwalkar, 2021): the sharpness rises (progressive sharpening) until it reaches , then hovers just above it while the loss keeps decreasing, non-monotonically, over long time scales. This edge of stability lies outside the descent lemma; one explanation is self-stabilization, in which the growing oscillation along the top eigenvector itself pushes the sharpness back down (Damian, Nichani & Lee, 2023).
The optimizers widget uses heavy-ball momentum with . On its bowl, with Hessian , what is the largest stable learning rate for momentum?
Show the answer
1.9
Stability needs with , so , nearly twice plain gradient descent's limit of 1. Switch the widget above to the bowl to check: momentum still reaches the minimum at and blows up at .
Sharp and flat minima. Hochreiter & Schmidhuber (1997) argued that flat minima generalize better, since a flat region can be specified with fewer bits of weight precision. Keskar et al. (2017) found that large-batch training reaches sharper minima and generalizes worse than small-batch training, and the idea that SGD's noise steers training toward flat regions became popular. But sharpness measured by Hessian eigenvalues is not a property of the function a network computes:
ReLU is positively homogeneous, for , so the rescaling cancels:
is the linear map . Differentiating twice relates the Hessians at the two equivalent points:
The block is multiplied by . For the squared loss it equals the Gauss–Newton term exactly, because is linear in , and it is nonzero as soon as some hidden unit is active on the data. Testing with unit vectors in that block:
So a flatness measure must respect such symmetries before it can say anything about generalization. Sharpness-aware minimization (SAM; Foret et al., 2021) minimizes the worst loss in a small ball, , and often improves test accuracy. A large study found sharpness-based measures among the better predictors of generalization (Jiang et al., 2020), yet a later one found the correlation inconsistent across modern settings, tracking hyperparameters such as the learning rate rather than generalization itself (Andriushchenko et al., 2023). The link between flatness and generalization remains contested.
Connected minima, and how to look at a landscape
Train the same architecture twice from different seeds and you get two minimizers and . On the straight segment between them the loss usually rises over a high barrier. Yet Garipov et al. (2018) and Draxler et al. (2018) found simple curves, such as a quadratic Bézier curve or a path with one bend, along which training and test loss stay nearly constant: the minima SGD finds lie in one connected low-loss valley (mode connectivity). Garipov et al. turned the observation into a fast ensembling method.
Part of the straight-line barrier is an artifact of symmetry. Permuting the hidden units of a layer, together with the matching rows and columns of the adjacent weight matrices, gives a different parameter vector computing the same function, so every minimum has at least equivalent copies. Entezari et al. (2022) conjectured that once permutations are accounted for, SGD solutions are linearly connected with essentially no barrier, and aligning the units of two trained networks before interpolating (Git Re-Basin; Ainsworth, Hayase & Srinivasa, 2023) removes most of the barrier for wide networks. Frankle et al. (2020) found that networks sharing their early training trajectory end up linearly connected without any alignment.
Seeing the landscape. Plots of the loss landscape along one or two directions are the main visual tool. Goodfellow, Vinyals & Saxe (2015) found the loss along the straight line from initialization to the trained solution typically decreasing monotonically, with no barrier. Li, Xu, Taylor, Studer & Goldstein (2018) showed that random-direction plots mislead unless each direction is rescaled filter by filter to the norms of the network's own filters (filter normalization), because the scale invariance of ReLU networks otherwise distorts apparent sharpness; with it, they saw skip connections turn a chaotic surface into a smooth, nearly convex basin. Two-dimensional slices of a million-dimensional function can hide as much as they show, so such pictures are evidence, not proof.
The lazy regime: the neural tangent kernel
How far do the weights move during training? For a very wide network, hardly at all, and then the network behaves like a linear model in its parameters. Expand the output around the initialization :
Take with fixed signs , Gaussian , and unit-norm inputs, no two parallel; train the by gradient descent on the squared loss. The limiting Gram matrix then has smallest eigenvalue , and the predictions evolve as
At initialization , an average over neurons, concentrates around , so for large . While stays that well conditioned, the residual contracts geometrically:
Geometric decay keeps every weight vector close to its start, so only a vanishing fraction of ReLUs change their activation pattern, stays within of , and the induction closes. Width and step suffice with probability :
The kernel picture explains how gradient descent can drive a huge nonconvex model to zero training loss, and links networks to kernel methods (its consequences for generalization are in Double Descent & Modern Generalization). But it is a regime, not a law. Chizat, Oyallon & Bach (2019) showed that lazy training comes from the scaling of the output, not from width as such, and that lazy models typically generalize worse. Real networks learn features: their NTK changes substantially during training, and kernel regression with the NTK measured at the end of training performs far better than with the one at initialization (Fort et al., 2020). Under the mean-field scaling (output divided by rather than ), wide two-layer networks follow a nonlinear evolution of the distribution of neurons and do learn features (Mei, Montanari & Nguyen, 2018; Chizat & Bach, 2018), and the μP parameterization extends this to deep networks (Yang & Hu, 2021).
Key takeaways
- At initialization, the mean-field quantity separates order (vanishing gradients, collapsing correlations) from chaos (exploding gradients). For ReLU the edge is He's ; dynamical isometry asks for all Jacobian singular values near 1.
- High-dimensional critical points at high loss are mostly saddles. Deep linear networks have no spurious local minima, nonlinear ones can, and gradient descent lingers near a saddle for a time growing like .
- Gradient descent is stable near a minimum only if ; in practice training sharpens to that edge and stays there. Sharpness is not reparameterization-invariant (Dinh et al., 2017), so its link to generalization is contested.
- SGD's minima are joined by low-loss curves, and largely by straight lines once permutation symmetry is factored out; landscape plots need filter normalization and remain low-dimensional evidence.
- In the NTK limit, wide networks train like kernel machines with a fixed kernel and converge linearly; real networks leave this lazy regime and learn features.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
A deep network at initialization has . In mean-field theory, what happens over 50 layers to backpropagated gradient norms, and to the correlation between two nearby inputs?
- Squared gradient norms shrink by about , and nearby inputs become perfectly correlated: the ordered phase
- Gradients explode, and nearby inputs decorrelate: the chaotic phase
- Nothing changes: only matters for the forward pass
- Gradients are preserved, because is close to 1
Show the answer
A. Squared gradient norms shrink by about , and nearby inputs become perfectly correlated: the ordered phase
is both the per-layer factor for squared gradient norms and the slope of the correlation map at . Below 1, gradients vanish geometrically, , and correlations converge to 1, so deep layers cannot tell inputs apart. Initializing at avoids both.
Dinh et al. (2017) show that for a ReLU network the minimizers and compute the same function, yet the top eigenvalues of their Hessians can differ arbitrarily. What follows?
- Sharp minima always generalize better than flat ones
- SGD cannot find flat minima in ReLU networks
- Flatness measured by Hessian eigenvalues cannot by itself determine generalization, since it changes under reparameterizations that leave the function fixed
- ReLU networks have no flat minima
Show the answer
C. Flatness measured by Hessian eigenvalues cannot by itself determine generalization, since it changes under reparameterizations that leave the function fixed
Generalization depends only on the function computed, so a measure that varies across functionally identical parameters cannot determine it. This does not refute every flatness hypothesis; it means a useful notion of flatness must be invariant under such symmetries, or tied to a fixed parameterization.
Full-batch gradient descent with step trains a network. According to Cohen et al. (2021), what typically happens to the sharpness ?
- It stays below its initial value
- It decreases steadily toward zero
- It grows without bound, and training diverges
- It rises until it reaches about , then hovers near that value while the loss keeps decreasing non-monotonically
Show the answer
D. It rises until it reaches about , then hovers near that value while the loss keeps decreasing non-monotonically
Progressive sharpening drives up to the stability threshold . Training then enters the edge of stability: the sharpness hovers just above , and the loss falls over long time scales despite short-term oscillations. Classical analyses, which keep throughout, do not cover this regime.
In the infinite-width NTK limit, gradient flow on the squared loss drives the training outputs to the labels along each eigenvector of at a rate given by its eigenvalue. What does this imply?
- The network learns new features during training
- Training is kernel regression with a fixed kernel: it converges linearly if , and the features stay those of the initialization
- Training converges only if the loss is convex in the weights
- Directions with small eigenvalues are fitted first
Show the answer
B. Training is kernel regression with a fixed kernel: it converges linearly if , and the features stay those of the initialization
With a constant kernel the dynamics are linear, : linear convergence at rate , and the learned function is a kernel predictor built from the initial tangent features. That is why the lazy regime explains optimization but not feature learning.
Further reading
- D. A. Roberts, S. Yaida & B. Hanin, The Principles of Deep Learning Theory (Cambridge University Press, 2022).
- S. S. Schoenholz, J. Gilmer, S. Ganguli & J. Sohl-Dickstein, "Deep information propagation," ICLR (2017).
- Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli & Y. Bengio, "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization," NeurIPS (2014).
- J. M. Cohen, S. Kaur, Y. Li, J. Z. Kolter & A. Talwalkar, "Gradient descent on neural networks typically occurs at the edge of stability," ICLR (2021).
- A. Jacot, F. Gabriel & C. Hongler, "Neural tangent kernel: Convergence and generalization in neural networks," NeurIPS (2018).