Simple Algorithm
Artificial Intelligence · Deep Layer · 30 min

Loss Landscapes & Initialization

Saddle points, flat minima and signal propagation.

A network's loss lives in millions of dimensions, and for decades the worry was that gradient descent would get trapped in bad local minima. The modern picture is stranger: saddles rather than minima slow training, minima are joined by low-loss paths, training hovers at the edge of instability, and very wide networks barely move from where they start. Before any of that, the initialization decides whether a signal survives the trip through the layers at all.

What you’ll learn

  • Derive the mean-field length and correlation maps of a random network, locate the edge of chaos for ReLU and tanh, and explain dynamical isometry.
  • Explain why saddle points rather than poor local minima dominate high-dimensional landscapes, and state what is proven about local minima.
  • Derive the stability threshold λmax⁡<2/η\lambda_{\max}<2/\eta, describe the edge of stability, and prove that sharpness is not invariant under reparameterization (Dinh et al., 2017).
  • Describe mode connectivity and loss-surface visualization, and state the neural tangent kernel limit with its linear convergence guarantee.

Signals at initialization: order, chaos and isometry

Before training starts, a random network must pass signals forward and gradients backward without collapsing or exploding. Mean-field theory makes this exact as the width tends to infinity (Poole et al., 2016; Schoenholz et al., 2017). Take weights Wij(l)∼N(0,σw2/nl−1)W^{(l)}_{ij}\sim\mathcal N(0,\sigma_w^2/n_{l-1}) and biases bi(l)∼N(0,σb2)b^{(l)}_i\sim\mathcal N(0,\sigma_b^2), and track two numbers through the layers: the mean squared pre-activation of one input, q(l)=1nl∥z(l)∥2q^{(l)}=\frac1{n_l}\norm{\mathbf z^{(l)}}^2, and the correlation c(l)c^{(l)} between the pre-activations of two inputs.

DerivationThe length and correlation maps
  1. q(l)=σw2 Ez[φ(q(l−1) z)2]+σb2q^{(l)}=\sigma_w^2\,\E_z\Big[\varphi\big(\sqrt{q^{(l-1)}}\,z\big)^2\Big]+\sigma_b^2

    At large width each zi(l)z^{(l)}_i is Gaussian given the previous layer, a sum of many independent terms, with variance set by the previous layer's mean squared activation. With z∼N(0,1)z\sim\mathcal N(0,1):

  2. χ1=C′(1)=σw2 Ez[φ′(q⋆ z)2]\chi_1=\mathcal C'(1)=\sigma_w^2\,\E_z\Big[\varphi'\big(\sqrt{q^\star}\,z\big)^2\Big]

    For two inputs the pair (zi(l)(x),zi(l)(x′))\big(z^{(l)}_i(\mathbf x),z^{(l)}_i(\mathbf x')\big) is jointly Gaussian. At the fixed point q⋆q^\star of the length map, the correlation follows a map c(l)=C(c(l−1))c^{(l)}=\mathcal C(c^{(l-1)}) with C(1)=1\mathcal C(1)=1, and its slope at 1 decides the fate of nearby inputs:

  3. E∥δ(l)∥2≈χ1 L−l E∥δ(L)∥2\E\norm{\boldsymbol\delta^{(l)}}^2\approx\chi_1^{\,L-l}\,\E\norm{\boldsymbol\delta^{(L)}}^2

    The same χ1\chi_1 multiplies the squared norm of the backpropagated gradient at each layer. If χ1<1\chi_1<1 (ordered phase), correlations converge to 1 exponentially fast, so deep layers see all inputs alike, and gradients vanish. If χ1>1\chi_1>1 (chaotic phase), nearby inputs decorrelate exponentially and gradients explode. At χ1=1\chi_1=1, the edge of chaos, both depth scales diverge:

  4. ReLU:q(l)=σw22 q(l−1)+σb2,χ1=σw22=1  ⟺  σw2=2  (σb=0)\text{ReLU:}\quad q^{(l)}=\frac{\sigma_w^2}2\,q^{(l-1)}+\sigma_b^2,\qquad\chi_1=\frac{\sigma_w^2}2=1\iff\sigma_w^2=2\ \ (\sigma_b=0)

    For tanh with σb=0\sigma_b=0, the transition is at σw2=1\sigma_w^2=1, moving to larger σw\sigma_w when σb>0\sigma_b>0. For ReLU, E[ReLU(qz)2]=q/2\E[\mathrm{ReLU}(\sqrt qz)^2]=q/2 and E[ReLU′(z)2]=12\E[\mathrm{ReLU}'(z)^2]=\tfrac12, so criticality is exactly He initialization:

Think first Derive 1−c(l)∼9π2/(2l2)1-c^{(l)}\sim9\pi^2/(2l^2) from the expansion of C\mathcal C near 1.

Write ϵl=1−c(l)\epsilon_l=1-c^{(l)}, so ϵl+1=ϵl−a ϵl3/2+O(ϵl2)\epsilon_{l+1}=\epsilon_l-a\,\epsilon_l^{3/2}+\order{\epsilon_l^2} with a=223πa=\frac{2\sqrt2}{3\pi}. For ul=ϵl−1/2u_l=\epsilon_l^{-1/2}, ul+1=ul(1−aϵl)−1/2≈ul+a2u_{l+1}=u_l(1-a\sqrt{\epsilon_l})^{-1/2}\approx u_l+\tfrac a2, so ul≈a2lu_l\approx\tfrac a2l and ϵl≈4a2l2=9π22l2\epsilon_l\approx\frac4{a^2l^2}=\frac{9\pi^2}{2l^2}. Iterating the exact map from c=0c=0 agrees: 1−c(l)≈3.6×10−31-c^{(l)}\approx3.6\times10^{-3} at l=100l=100 against the asymptotic 4.4×10−34.4\times10^{-3}, and the ratio tends to 1.

Criticality controls averages. Dynamical isometry asks for more: that all singular values of the input–output Jacobian ∏lD(l)W(l)\prod_lD^{(l)}\mathbf W^{(l)} concentrate near 1, so that every direction propagates. Deep linear networks with orthogonal weights achieve it exactly, and their learning time can be independent of depth, unlike with Gaussian weights (Saxe, McClelland & Ganguli, 2014). Random-matrix calculations show that ReLU networks cannot achieve it, even with orthogonal weights, while tanh networks with orthogonal weights near the edge of chaos can (Pennington, Schoenholz & Ganguli, 2017). With a matching orthogonal initialization for convolutions, Xiao et al. (2018) trained plain CNNs 10,000 layers deep.

Critical points: saddles, minima and what is proven

A critical point's index is its fraction of negative Hessian eigenvalues. For random Gaussian functions in high dimension, critical points at high loss have large index and only those near the bottom have index near 0 (Bray & Dean, 2007): a minimum needs all PP eigenvalues positive at once, so a random critical point far above the minimum is almost surely a saddle. Dauphin et al. (2014) found the same index–loss relation in neural networks, argued that saddles and the plateaus around them, not poor local minima, are what slows training, and proposed a saddle-free Newton method that rescales the gradient by ∣H∣−1\abs{H}^{-1}.

Saddles slow gradient descent but rarely stop it: it escapes strict saddles from almost every start, and perturbations make the escape provably fast (Optimization & Convexity). Near a saddle the unstable coordinate grows by a factor of about 1+η∣λmin⁡∣1+\eta\abs{\lambda_{\min}} per step, so the time spent there scales like log⁡(1/d)/η\log(1/d)/\eta, where dd is the starting distance from the saddle's stable manifold. Adaptive methods change that dependence, at least when the escape direction lines up with the coordinates.

Check your understanding

Gradient descent with η=0.05\eta=0.05 starts at distance 10−310^{-3} instead of 10−210^{-2} from the stable manifold of a strict saddle whose negative curvature is −1-1. About how much longer does it stay near the saddle?

  1. About 10 times longer
  2. About 10\sqrt{10} times longer
  3. About ln⁡10/ln⁡1.05≈47\ln10/\ln1.05\approx47 more steps
  4. No longer: gradient descent never gets stuck at saddles
Show the answer

C. About ln⁡10/ln⁡1.05≈47\ln10/\ln1.05\approx47 more steps

The unstable coordinate is multiplied by 1+η∣λ∣=1.051+\eta\abs{\lambda}=1.05 per step, so leaving a fixed neighborhood from distance dd takes about ln⁡(r/d)/ln⁡1.05\ln(r/d)/\ln1.05 steps, and a tenfold smaller dd adds about 47. In the widget, from x0=−2.1x_0=-2.1, SGD arrives at step 109 from y0=0.03y_0=0.03, 151 from 0.0040.004 and 226 from 10−410^{-4}.

Sharpness: the edge of stability and flat minima

Near a minimum, f(θ⋆+u)≈f⋆+12u⊤Huf(\boldsymbol\theta^\star+\mathbf u)\approx f^\star+\tfrac12\mathbf u^\top H\mathbf u, and gradient descent multiplies the component of u\mathbf u along an eigenvector of eigenvalue λ\lambda by 1−ηλ1-\eta\lambda. It is stable iff every ∣1−ηλ∣<1\abs{1-\eta\lambda}<1, that is, iff the sharpness λmax⁡(H)<2/η\lambda_{\max}(H)<2/\eta. With heavy-ball momentum β\beta the condition on a quadratic becomes ηλ<2+2β\eta\lambda<2+2\beta. Classical analyses keep the step below the threshold everywhere. What actually happens in full-batch training of networks is different (Cohen, Kaur, Li, Kolter & Talwalkar, 2021): the sharpness rises (progressive sharpening) until it reaches 2/η2/\eta, then hovers just above it while the loss keeps decreasing, non-monotonically, over long time scales. This edge of stability lies outside the descent lemma; one explanation is self-stabilization, in which the growing oscillation along the top eigenvector itself pushes the sharpness back down (Damian, Nichani & Lee, 2023).

Predict before you look

The optimizers widget uses heavy-ball momentum with β=0.9\beta=0.9. On its bowl, f=x2+y2f=x^2+y^2 with Hessian 2I2I, what is the largest stable learning rate for momentum?

Show the answer

1.9

Stability needs ηλ<2+2β=3.8\eta\lambda<2+2\beta=3.8 with λ=2\lambda=2, so η<1.9\eta<1.9, nearly twice plain gradient descent's limit of 1. Switch the widget above to the bowl to check: momentum still reaches the minimum at η=1.5\eta=1.5 and blows up at η=2\eta=2.

Sharp and flat minima. Hochreiter & Schmidhuber (1997) argued that flat minima generalize better, since a flat region can be specified with fewer bits of weight precision. Keskar et al. (2017) found that large-batch training reaches sharper minima and generalizes worse than small-batch training, and the idea that SGD's noise steers training toward flat regions became popular. But sharpness measured by Hessian eigenvalues is not a property of the function a network computes:

DerivationProof
  1. α−1W2 ReLU(αW1x)=W2 ReLU(W1x)\alpha^{-1}\mathbf W_2\,\mathrm{ReLU}(\alpha\mathbf W_1\mathbf x)=\mathbf W_2\,\mathrm{ReLU}(\mathbf W_1\mathbf x)

    ReLU is positively homogeneous, ReLU(αu)=α ReLU(u)\mathrm{ReLU}(\alpha u)=\alpha\,\mathrm{ReLU}(u) for α>0\alpha>0, so the rescaling cancels:

  2. ∇2L(Dαθ)=Dα−1 ∇2L(θ) Dα−1\nabla^2L(D_\alpha\boldsymbol\theta)=D_\alpha^{-1}\,\nabla^2L(\boldsymbol\theta)\,D_\alpha^{-1}

    TαT_\alpha is the linear map Dα=diag⁡(αI,α−1I)D_\alpha=\operatorname{diag}(\alpha I,\alpha^{-1}I). Differentiating L(Dαθ)=L(θ)L(D_\alpha\boldsymbol\theta)=L(\boldsymbol\theta) twice relates the Hessians at the two equivalent points:

  3. λmax⁡(∇2L(Dαθ⋆)) ≥ α2 λmax⁡([∇2L(θ⋆)]22) → α→∞  ∞\lambda_{\max}\big(\nabla^2L(D_\alpha\boldsymbol\theta^\star)\big)\ \ge\ \alpha^2\,\lambda_{\max}\big([\nabla^2L(\boldsymbol\theta^\star)]_{22}\big)\ \xrightarrow{\ \alpha\to\infty\ }\ \infty

    The W2\mathbf W_2 block is multiplied by α2\alpha^2. For the squared loss it equals the Gauss–Newton term J2⊤J2J_2^\top J_2 exactly, because ff is linear in W2\mathbf W_2, and it is nonzero as soon as some hidden unit is active on the data. Testing with unit vectors in that block:

So a flatness measure must respect such symmetries before it can say anything about generalization. Sharpness-aware minimization (SAM; Foret et al., 2021) minimizes the worst loss in a small ball, max⁡∥ϵ∥≤ρL(θ+ϵ)\max_{\norm{\boldsymbol\epsilon}\le\rho}L(\boldsymbol\theta+\boldsymbol\epsilon), and often improves test accuracy. A large study found sharpness-based measures among the better predictors of generalization (Jiang et al., 2020), yet a later one found the correlation inconsistent across modern settings, tracking hyperparameters such as the learning rate rather than generalization itself (Andriushchenko et al., 2023). The link between flatness and generalization remains contested.

Connected minima, and how to look at a landscape

Train the same architecture twice from different seeds and you get two minimizers θA\boldsymbol\theta_A and θB\boldsymbol\theta_B. On the straight segment between them the loss usually rises over a high barrier. Yet Garipov et al. (2018) and Draxler et al. (2018) found simple curves, such as a quadratic Bézier curve or a path with one bend, along which training and test loss stay nearly constant: the minima SGD finds lie in one connected low-loss valley (mode connectivity). Garipov et al. turned the observation into a fast ensembling method.

Part of the straight-line barrier is an artifact of symmetry. Permuting the hidden units of a layer, together with the matching rows and columns of the adjacent weight matrices, gives a different parameter vector computing the same function, so every minimum has at least ∏lnl!\prod_ln_l! equivalent copies. Entezari et al. (2022) conjectured that once permutations are accounted for, SGD solutions are linearly connected with essentially no barrier, and aligning the units of two trained networks before interpolating (Git Re-Basin; Ainsworth, Hayase & Srinivasa, 2023) removes most of the barrier for wide networks. Frankle et al. (2020) found that networks sharing their early training trajectory end up linearly connected without any alignment.

Seeing the landscape. Plots of the loss landscape along one or two directions are the main visual tool. Goodfellow, Vinyals & Saxe (2015) found the loss along the straight line from initialization to the trained solution typically decreasing monotonically, with no barrier. Li, Xu, Taylor, Studer & Goldstein (2018) showed that random-direction plots mislead unless each direction is rescaled filter by filter to the norms of the network's own filters (filter normalization), because the scale invariance of ReLU networks otherwise distorts apparent sharpness; with it, they saw skip connections turn a chaotic surface into a smooth, nearly convex basin. Two-dimensional slices of a million-dimensional function can hide as much as they show, so such pictures are evidence, not proof.

The lazy regime: the neural tangent kernel

How far do the weights move during training? For a very wide network, hardly at all, and then the network behaves like a linear model in its parameters. Expand the output around the initialization θ0\boldsymbol\theta_0:

fθ(x)≈fθ0(x)+∇θfθ0(x)⊤(θ−θ0),Θ(x,x′)=∇θfθ(x)⊤∇θfθ(x′)f_{\boldsymbol\theta}(\mathbf x)\approx f_{\boldsymbol\theta_0}(\mathbf x)+\nabla_{\boldsymbol\theta}f_{\boldsymbol\theta_0}(\mathbf x)^\top(\boldsymbol\theta-\boldsymbol\theta_0),\qquad\Theta(\mathbf x,\mathbf x')=\nabla_{\boldsymbol\theta}f_{\boldsymbol\theta}(\mathbf x)^\top\nabla_{\boldsymbol\theta}f_{\boldsymbol\theta}(\mathbf x')
Θ\Theta is the neural tangent kernel (NTK). Under gradient flow on L=12∑i(f(xi)−yi)2L=\tfrac12\sum_i\big(f(\mathbf x_i)-y_i\big)^2, the outputs on the training inputs XX obey ddtft(X)=−Θt(X,X)(ft(X)−y)\frac{d}{dt}f_t(X)=-\Theta_t(X,X)\big(f_t(X)-\mathbf y\big) exactly.
DerivationWhy training converges: a sketch for two layers (Du, Zhai, Póczos & Singh, 2019)
  1. u(k+1)−u(k)≈−η H(k)(u(k)−y),H(k) the empirical Gram matrix of the active units\mathbf u(k+1)-\mathbf u(k)\approx-\eta\,H(k)\big(\mathbf u(k)-\mathbf y\big),\qquad H(k)\ \text{the empirical Gram matrix of the active units}

    Take f(x)=1m∑r=1mar ReLU(wr⊤x)f(\mathbf x)=\frac1{\sqrt m}\sum_{r=1}^ma_r\,\mathrm{ReLU}(\mathbf w_r^\top\mathbf x) with fixed signs ar∈{±1}a_r\in\{\pm1\}, Gaussian wr(0)\mathbf w_r(0), and nn unit-norm inputs, no two parallel; train the wr\mathbf w_r by gradient descent on the squared loss. The limiting Gram matrix Hij∞=Ew[xi⊤xj 1{w⊤xi≥0, w⊤xj≥0}]H^\infty_{ij}=\E_{\mathbf w}\big[\mathbf x_i^\top\mathbf x_j\,\mathbf 1\{\mathbf w^\top\mathbf x_i\ge0,\ \mathbf w^\top\mathbf x_j\ge0\}\big] then has smallest eigenvalue λ0>0\lambda_0>0, and the predictions u(k)=fW(k)(X)\mathbf u(k)=f_{W(k)}(X) evolve as

  2. ∥u(k)−y∥2≤(1−ηλ02)k∥u(0)−y∥2\norm{\mathbf u(k)-\mathbf y}^2\le\Big(1-\frac{\eta\lambda_0}2\Big)^k\norm{\mathbf u(0)-\mathbf y}^2

    At initialization H(0)H(0), an average over mm neurons, concentrates around H∞H^\infty, so λmin⁡(H(0))≥34λ0\lambda_{\min}(H(0))\ge\tfrac34\lambda_0 for large mm. While H(k)H(k) stays that well conditioned, the residual contracts geometrically:

  3. ∥wr(k)−wr(0)∥=O(n ∥y−u(0)∥m λ0) → m→∞  0\norm{\mathbf w_r(k)-\mathbf w_r(0)}=\order{\frac{\sqrt n\,\norm{\mathbf y-\mathbf u(0)}}{\sqrt m\,\lambda_0}}\ \xrightarrow{\ m\to\infty\ }\ 0

    Geometric decay keeps every weight vector close to its start, so only a vanishing fraction of ReLUs change their activation pattern, H(k)H(k) stays within λ0/4\lambda_0/4 of H(0)H(0), and the induction closes. Width m=Ω(n6/(λ04δ3))m=\Omega\big(n^6/(\lambda_0^4\delta^3)\big) and step η=O(λ0/n2)\eta=\order{\lambda_0/n^2} suffice with probability 1−δ1-\delta:

The kernel picture explains how gradient descent can drive a huge nonconvex model to zero training loss, and links networks to kernel methods (its consequences for generalization are in Double Descent & Modern Generalization). But it is a regime, not a law. Chizat, Oyallon & Bach (2019) showed that lazy training comes from the scaling of the output, not from width as such, and that lazy models typically generalize worse. Real networks learn features: their NTK changes substantially during training, and kernel regression with the NTK measured at the end of training performs far better than with the one at initialization (Fort et al., 2020). Under the mean-field scaling (output divided by mm rather than m\sqrt m), wide two-layer networks follow a nonlinear evolution of the distribution of neurons and do learn features (Mei, Montanari & Nguyen, 2018; Chizat & Bach, 2018), and the μP parameterization extends this to deep networks (Yang & Hu, 2021).

Key takeaways

  • At initialization, the mean-field quantity χ1=σw2E[φ′(q⋆z)2]\chi_1=\sigma_w^2\E[\varphi'(\sqrt{q^\star}z)^2] separates order (vanishing gradients, collapsing correlations) from chaos (exploding gradients). For ReLU the edge χ1=1\chi_1=1 is He's σw2=2\sigma_w^2=2; dynamical isometry asks for all Jacobian singular values near 1.
  • High-dimensional critical points at high loss are mostly saddles. Deep linear networks have no spurious local minima, nonlinear ones can, and gradient descent lingers near a saddle for a time growing like log⁡(1/d)/η\log(1/d)/\eta.
  • Gradient descent is stable near a minimum only if λmax⁡<2/η\lambda_{\max}<2/\eta; in practice training sharpens to that edge and stays there. Sharpness is not reparameterization-invariant (Dinh et al., 2017), so its link to generalization is contested.
  • SGD's minima are joined by low-loss curves, and largely by straight lines once permutation symmetry is factored out; landscape plots need filter normalization and remain low-dimensional evidence.
  • In the NTK limit, wide networks train like kernel machines with a fixed kernel and converge linearly; real networks leave this lazy regime and learn features.

Check yourself

In the interactive lesson these are scored with your confidence, and what you miss comes back for review.

Question 1

A deep network at initialization has χ1=σw2 E[φ′(q⋆z)2]=0.9\chi_1=\sigma_w^2\,\E[\varphi'(\sqrt{q^\star}z)^2]=0.9. In mean-field theory, what happens over 50 layers to backpropagated gradient norms, and to the correlation between two nearby inputs?

  1. Squared gradient norms shrink by about 0.950≈0.0050.9^{50}\approx0.005, and nearby inputs become perfectly correlated: the ordered phase
  2. Gradients explode, and nearby inputs decorrelate: the chaotic phase
  3. Nothing changes: χ1\chi_1 only matters for the forward pass
  4. Gradients are preserved, because χ1\chi_1 is close to 1
Show the answer

A. Squared gradient norms shrink by about 0.950≈0.0050.9^{50}\approx0.005, and nearby inputs become perfectly correlated: the ordered phase

χ1\chi_1 is both the per-layer factor for squared gradient norms and the slope of the correlation map at c=1c=1. Below 1, gradients vanish geometrically, 0.950≈5×10−30.9^{50}\approx5\times10^{-3}, and correlations converge to 1, so deep layers cannot tell inputs apart. Initializing at χ1=1\chi_1=1 avoids both.

Question 2

Dinh et al. (2017) show that for a ReLU network the minimizers (W1,W2)(\mathbf W_1,\mathbf W_2) and (αW1,W2/α)(\alpha\mathbf W_1,\mathbf W_2/\alpha) compute the same function, yet the top eigenvalues of their Hessians can differ arbitrarily. What follows?

  1. Sharp minima always generalize better than flat ones
  2. SGD cannot find flat minima in ReLU networks
  3. Flatness measured by Hessian eigenvalues cannot by itself determine generalization, since it changes under reparameterizations that leave the function fixed
  4. ReLU networks have no flat minima
Show the answer

C. Flatness measured by Hessian eigenvalues cannot by itself determine generalization, since it changes under reparameterizations that leave the function fixed

Generalization depends only on the function computed, so a measure that varies across functionally identical parameters cannot determine it. This does not refute every flatness hypothesis; it means a useful notion of flatness must be invariant under such symmetries, or tied to a fixed parameterization.

Question 3

Full-batch gradient descent with step η=0.01\eta=0.01 trains a network. According to Cohen et al. (2021), what typically happens to the sharpness λmax⁡(∇2L)\lambda_{\max}(\nabla^2L)?

  1. It stays below its initial value
  2. It decreases steadily toward zero
  3. It grows without bound, and training diverges
  4. It rises until it reaches about 2/η=2002/\eta=200, then hovers near that value while the loss keeps decreasing non-monotonically
Show the answer

D. It rises until it reaches about 2/η=2002/\eta=200, then hovers near that value while the loss keeps decreasing non-monotonically

Progressive sharpening drives λmax⁡\lambda_{\max} up to the stability threshold 2/η2/\eta. Training then enters the edge of stability: the sharpness hovers just above 2/η2/\eta, and the loss falls over long time scales despite short-term oscillations. Classical analyses, which keep η<2/λmax⁡\eta<2/\lambda_{\max} throughout, do not cover this regime.

Question 4

In the infinite-width NTK limit, gradient flow on the squared loss drives the training outputs to the labels along each eigenvector of Θ∞(X,X)\Theta_\infty(X,X) at a rate given by its eigenvalue. What does this imply?

  1. The network learns new features during training
  2. Training is kernel regression with a fixed kernel: it converges linearly if λmin⁡(Θ∞(X,X))>0\lambda_{\min}(\Theta_\infty(X,X))>0, and the features stay those of the initialization
  3. Training converges only if the loss is convex in the weights
  4. Directions with small eigenvalues are fitted first
Show the answer

B. Training is kernel regression with a fixed kernel: it converges linearly if λmin⁡(Θ∞(X,X))>0\lambda_{\min}(\Theta_\infty(X,X))>0, and the features stay those of the initialization

With a constant kernel the dynamics are linear, ft(X)−y=e−Θ∞t(f0(X)−y)f_t(X)-\mathbf y=e^{-\Theta_\infty t}(f_0(X)-\mathbf y): linear convergence at rate λmin⁡\lambda_{\min}, and the learned function is a kernel predictor built from the initial tangent features. That is why the lazy regime explains optimization but not feature learning.

Further reading

  • D. A. Roberts, S. Yaida & B. Hanin, The Principles of Deep Learning Theory (Cambridge University Press, 2022).
  • S. S. Schoenholz, J. Gilmer, S. Ganguli & J. Sohl-Dickstein, "Deep information propagation," ICLR (2017).
  • Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli & Y. Bengio, "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization," NeurIPS (2014).
  • J. M. Cohen, S. Kaur, Y. Li, J. Z. Kolter & A. Talwalkar, "Gradient descent on neural networks typically occurs at the edge of stability," ICLR (2021).
  • A. Jacot, F. Gabriel & C. Hongler, "Neural tangent kernel: Convergence and generalization in neural networks," NeurIPS (2018).