Optimizers in Depth
Adam, schedules, warmup and why they work.
Adam's default settings train most networks reasonably well out of the box, yet Adam provably fails to converge on a one-dimensional convex problem, and for years it was suspected of generalizing worse than plain SGD. This lesson proves what SGD guarantees, shows why learning rates must decay, takes Adam apart, and separates what is established from what is still argued.
What you’ll learn
- State and prove the rate of SGD on smooth nonconvex objectives (Ghadimi & Lan, 2013) and the noise-floor bound under strong convexity, and explain the role of the batch size.
- Justify decaying schedules and warmup from these bounds and from what is known about curvature early in training.
- Analyze adaptive methods: bias correction, bounded and scale-invariant steps, sign-descent behavior, and RMSProp's period-2 orbit at a constant step.
- Reconstruct Reddi, Kale & Kumar's counterexample to Adam's convergence, and assess AMSGrad, decoupled weight decay and the evidence on how adaptive methods generalize.
SGD, precisely
The descent lemma (from The Math Toolkit of AI) applied to the step :
Take the expectation over given . Unbiasedness gives , and the variance bound gives :
For the bracket is at least . Take full expectations, sum over and telescope, using :
Divide by . The first term is optimization, which shrinks with more steps; the second is noise, which shrinks only with a smaller step or a larger batch:
Batch size. With batches of the bound reads . While the noise term dominates, doubling and doubling (as long as ) halves the number of steps for the same number of examples : on parallel hardware, large batches are a free speed-up, the linear scaling rule (Goyal et al., 2017). Once the optimization term dominates, larger batches only waste computation. McCandlish et al. (2018) estimated this critical batch size by the gradient noise scale , which grows during training as the gradient shrinks.
Noise floors and learning-rate schedules
With strong convexity the picture sharpens: a constant step converges fast, but only to a noise floor proportional to the step.
Let be -strongly convex and -smooth with minimizer , and expand one step:
In expectation, use and co-coercivity, . For :
Strong convexity gives . Unroll the resulting contraction and sum the geometric series of noise terms:
A linear rate down to a floor proportional to . Halving halves the floor but also halves the rate, which is what step decay does in stages. Steps remove the floor and give the optimal ; in general, with (Robbins & Monro, 1951) ensures convergence.
Deep-learning schedules follow this logic loosely: hold a large step for fast progress, then decay it to settle into a low-loss region, by steps, by a cosine, or with the warmup–stable–decay shape used for some language models, which can extend a run without restarting it. The theory is only a guide, since the losses are nonconvex and the noise is far from isotropic, and a large step does more than add noise: it caps the sharpness of the regions where training can stay (Loss Landscapes & Initialization).
Why warmup? Three mechanisms are documented. (1) Early on, Adam's averages only squared gradients, so its per-coordinate step sizes are high-variance; RAdam corrects for this analytically (Liu et al., 2020). (2) In Post-LN transformers, with layer norm after each residual addition, the gradients near the output are large at initialization, and training without warmup diverges (Xiong et al., 2020); the Pre-LN placement largely removes the need. (3) The loss can be sharper early in training than the peak step tolerates, and warmup lets the network move to flatter regions first (Gilmer et al., 2022).
Adaptive methods, analyzed
AdaGrad (Duchi, Hazan & Singer, 2011) scales coordinate by . Its effective steps only shrink, and for online convex optimization it guarantees regret with constants that adapt to the gradients' geometry, far better than SGD's when gradients are sparse. In deep learning its steps shrink too fast; RMSProp replaces the sum by an exponential average, and Adam adds momentum and bias correction.
Write Adam's step as . Kingma and Ba note that in typical cases and at most for the defaults: sets the step length per coordinate, much like a trust region. When a coordinate's gradients keep their sign, and Adam becomes sign descent, . When they are mostly noise, and the steps shrink automatically, like a signal-to-noise ratio.
On the orbit , so at every step, and the average stays at that value:
The update divides the gradient by its own magnitude, so each step has length exactly and maps to and back:
Gradient descent converges on this function for any . RMSProp instead equalizes the coordinates and keeps moving by . In simulations the orbit attracts the iterates from typical starts: as the short-memory average shrinks, the effective step grows past the stability limit, and the iterate is pushed back out. Only a decaying , or a long memory, lets an adaptive method settle.
Back on the bowl, set . By the proposition, how far from the minimum (Euclidean distance) does RMSProp end up?
Show the answer
1.06
Each coordinate oscillates between , so the distance is , which the widget reports when the race ends. At it is . The orbit scales with , which is why adaptive methods are paired with decaying schedules.
When Adam fails, and what fixes it
With the step is . The large gradient arrives once per period and inflates at that very step: since , the update divides by roughly :
The gradients arrive twice per period. The short memory forgets within a step (), so is about 2 at the first and about 1 at the second, and each pushes up by a constant fraction of :
Per period the net drift is positive, although the average gradient is also positive and calls for moving down:
The normalization has erased the information in the gradients' magnitudes: the rare large gradient is cut to the size of the frequent small ones, and the method follows the majority sign instead of the mean. Reddi et al. trace the failure to the quantity below, positive semidefinite for SGD and AdaGrad (effective steps never grow) but indefinite for Adam and RMSProp:
AMSGrad (Reddi et al., 2018) divides by the running maximum , so effective steps never grow; with it has regret for online convex problems. On the example above it converges to , and so does Adam with , whose long memory averages over hundreds of periods. Later analyses showed that the counterexamples choose after the problem: for a fixed problem and close enough to 1, unmodified Adam converges to a neighborhood of stationary points (Zhang et al., 2022; Défossez et al., 2022). AMSGrad has seen little use in practice.
Decoupled weight decay. Adding to the loss makes Adam rescale the decay term by like everything else, so weights with large gradients are regularized less. AdamW applies the decay outside the adaptive step, ; it generalizes better and is the default for transformers (Loshchilov & Hutter, 2019). For SGD the two forms coincide up to a rescaling of .
On Reddi et al.'s example, which change makes the method converge to the minimizer ?
- A smaller , so that tracks the latest gradient more closely
- Dividing by the running maximum of (AMSGrad), or using close to 1
- Clipping every gradient to norm at most 1
- A constant step instead of
Show the answer
B. Dividing by the running maximum of (AMSGrad), or using close to 1
The failure comes from a short memory that forgets the rare large gradient. AMSGrad's running maximum never forgets it, and averages over hundreds of periods. Either way the steps become proportional to the gradients again and , as a direct simulation confirms.
Do adaptive methods generalize worse?
Wilson, Roelofs, Stern, Srebro & Recht (2017) constructed a linearly separable classification problem, with more features than examples, on which SGD started at zero classifies new points perfectly while AdaGrad, RMSProp and Adam converge to a different interpolating solution with test error close to one half. They also reported adaptive methods generalizing worse on several image and language benchmarks. The mechanism is implicit bias: when many solutions fit the data, each optimizer selects a different one, and for gradient descent on least squares it is the minimum--norm solution (Regularization Theory).
The debate has since shifted. With hyperparameters tuned equally carefully, Adam's included, more general optimizers matched or beat their special cases, so many reported gaps reflected tuning protocols (Choi et al., 2019). On transformers the gap runs the other way: SGD trains them markedly worse than Adam. Proposed explanations include heavy-tailed gradient noise (Zhang et al., 2020), Adam's resemblance to sign descent (Kunstner et al., 2023), and the heavy-tailed class imbalance of language data, where adaptive steps keep learning rare tokens that gradient descent neglects (Kunstner et al., 2024).
| Method | Update (per coordinate) | State per parameter | What is guaranteed |
|---|---|---|---|
| SGD | none | the rates of step 1, worst-case optimal | |
| Heavy ball | ; | 1 | acceleration on quadratics only |
| AdaGrad | 1 | adaptive regret (online convex) | |
| Adam | 2 | can fail for fixed ; converges near stationary points as | |
| AMSGrad | Adam with the running max of | 3 | regret (online convex) |
| AdamW | Adam step, plus | 2 | as Adam; the default for transformers |
Key takeaways
- SGD with step on a smooth nonconvex satisfies , an rate that is worst-case optimal. Batches divide and allow proportionally larger steps, up to a critical batch size.
- Under strong convexity a constant step converges linearly to a noise floor . Decaying the step trades speed for a lower floor, the logic behind step and cosine schedules; warmup protects early training from noisy second moments, large gradients and sharp regions.
- Adam's steps are scale-invariant and roughly bounded by , close to sign descent when gradients are consistent; at a constant step, RMSProp has an orbit of size per coordinate whatever the curvature.
- A short-memory normalization can drive Adam to the worst point of a convex problem (Reddi et al., 2018); AMSGrad's running maximum, or , restores convergence, and AdamW decouples weight decay from the adaptive scaling.
- Adaptive methods can pick worse-generalizing solutions on constructed problems (Wilson et al., 2017), but tuning protocols explain many reported gaps, and on transformers Adam clearly beats SGD, for reasons still debated.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
For SGD on an -smooth nonconvex function with gradient-noise variance and a tuned constant step, how does the guaranteed average of decay with the number of steps ?
Show the answer
B.
The bound is minimized by , giving once noise dominates. Without noise () it improves to , the rate of gradient descent.
RMSProp with a constant step (and ) runs on . On its period-2 orbit, what is the amplitude of the oscillation in each coordinate?
- in both coordinates
- in , but almost none in , where the gradient is negligible
- , because RMSProp converges on quadratics
- , inversely proportional to each curvature
Show the answer
A. in both coordinates
On the orbit and , so every step has length exactly and each coordinate oscillates by , whatever its curvature. Per-coordinate normalization removes curvature from the step size: that is its strength on ill-conditioned problems, and the reason must decay.
What is the core reason Adam fails on Reddi, Kale & Kumar's sequence ?
- The objective is nonconvex
- The step size decays too fast
- Momentum overshoots the minimum
- Normalizing by a short-memory average of squared gradients cuts the rare large gradient down to the size of the frequent small ones, so the method follows the majority sign instead of the mean
Show the answer
D. Normalizing by a short-memory average of squared gradients cuts the rare large gradient down to the size of the frequent small ones, so the method follows the majority sign instead of the mean
With there is no momentum, and every is linear, hence convex. The large gradient inflates exactly when it arrives, so its step is about , while each gradient also gets a step of order : two up-steps beat one down-step. AMSGrad's maximum, or near 1, keeps large and restores proportional steps.
Why does AdamW decouple weight decay from the gradient instead of adding to the loss?
- Because inside Adam the penalty's gradient is divided by , so weights with large gradients are decayed less; decoupling shrinks all weights at the same relative rate
- Because the term makes the loss nonconvex
- Because Adam ignores terms that do not depend on the data
- Because weight decay and regularization differ for SGD as well
Show the answer
A. Because inside Adam the penalty's gradient is divided by , so weights with large gradients are decayed less; decoupling shrinks all weights at the same relative rate
Inside Adam, is rescaled coordinate by coordinate like the data gradient, which turns a uniform shrinkage into a gradient-dependent one. AdamW applies directly. For SGD the two forms coincide up to a rescaling of (Loshchilov & Hutter, 2019).
Further reading
- S. Ghadimi & G. Lan, "Stochastic first- and zeroth-order methods for nonconvex stochastic programming," SIAM Journal on Optimization 23(4) (2013).
- S. J. Reddi, S. Kale & S. Kumar, "On the convergence of Adam and beyond," ICLR (2018).
- A. C. Wilson, R. Roelofs, M. Stern, N. Srebro & B. Recht, "The marginal value of adaptive gradient methods in machine learning," NeurIPS (2017).
- I. Loshchilov & F. Hutter, "Decoupled weight decay regularization," ICLR (2019).
- L. Bottou, F. E. Curtis & J. Nocedal, "Optimization methods for large-scale machine learning," SIAM Review 60(2) (2018).