Learning by Rolling Downhill

The entire training run of a billion-parameter model, the thing that costs a data centre weeks and a small fortune in electricity, is built on a trick Augustin-Louis Cauchy wrote down in 1847. To make a function smaller, find the direction in which it falls most steeply, take a step that way, and repeat. That's it. Hadamard arrived at the same idea independently around 1907; Haskell Curry first studied when it actually converges in 1944. None of them had a neural network in mind. They just wanted to minimise things, and downhill is where the minimum is.

What's genuinely surprising is not that this works. It's how badly it works in its raw form, and how much of modern machine learning is a stack of patches for the specific ways naive downhill walking falls over. So let's start with the trick, then break it on purpose, because the failure modes are where the understanding lives.

The rule, and the one number that breaks it

Gradient descent is a single line. Let $\theta$ be the parameters, $f$ the loss, $\eta$ the learning rate.

$$\theta_{t+1} = \theta_t - \eta\,\nabla f(\theta_t)$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

The learning rate $\eta$ is deceptively dangerous. Take a one-dimensional quadratic with curvature $k$ (the second derivative). One step multiplies your distance from the minimum by $(1 - \eta k)$. So you converge only when $|1 - \eta k| < 1$, that is for $0 < \eta < 2/k$. Below $1/k$ you glide in monotonically; between $1/k$ and $2/k$ you converge but oscillate, ping-ponging across the minimum with shrinking amplitude; at $2/k$ exactly you orbit forever; above it you diverge. The same lever that sets your speed sets your stability, and the boundary is the curvature you usually don't know.

Failure mode one: the ravine

Switch the lab to the Ravine preset, a scaled Rosenbrock function with a long, curved, narrow valley. (The textbook Rosenbrock uses a steepness constant of 100; the lab dials it back to 20 so the traces stay on screen, but the pathology is the same.) This is the case that humbles plain gradient descent.

Watch the orange trace, plain gradient descent. The valley walls are steep and its floor is nearly flat, so the gradient points mostly across the valley, not along it. Each step throws the dot at the opposite wall; it zig-zags wall to wall and inches forward almost not at all. All the energy goes sideways, where you don't want to move, and almost none goes down the channel, where you do.

The fix, the yellow trace, is momentum. Instead of stepping on the current gradient, accumulate a running, discounted sum of past gradients, a heavy ball that keeps its heading.

$$v_{t+1} = \beta\, v_t + \nabla f(\theta_t), \qquad \theta_{t+1} = \theta_t - \eta\, v_{t+1}$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

The across-valley components flip sign every step, so they cancel in the running sum. The along-valley component points the same way every step, so it builds. A heavy ball ignores the rattling walls and rolls down the floor. The decay factor $\beta$ (typically $0.9$) sets how much past the ball remembers.

Failure mode two: the saddle

The folk story of optimisation says the danger is getting stuck in a bad local minimum. In high dimensions that story is mostly wrong, and the Saddle preset shows why.

A saddle is a point that's a minimum along some directions and a maximum along others. The gradient there is nearly zero, so equation (1) takes vanishing steps and the optimiser can sit on the ridge as if it had arrived, when really it just needs to find the one direction that goes down.

Adapting the step per direction

Momentum fixes direction; the next idea fixes scale. Instead of one global learning rate, give every parameter its own, shrinking the step for directions whose gradients have been large and growing it for directions that have been quiet. RMSProp keeps a running average of each squared gradient and divides by its root.

$$s_{t+1} = \rho\, s_t + (1-\rho)\,\nabla f(\theta_t)^{\odot 2}, \qquad \theta_{t+1} = \theta_t - \frac{\eta\,\nabla f(\theta_t)}{\sqrt{s_{t+1}} + \varepsilon}$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

The $\odot 2$ is an elementwise square; the division is elementwise too. A direction that keeps producing big gradients (the steep valley wall) gets a big $s$ and a damped step. A direction with small gradients (the valley floor) gets a small $s$ and a relatively larger step. It directly attacks the condition-number problem from the first callout.

Adam is the one everybody reaches for, and it is just momentum and RMSProp bolted together, with one extra correction.

$$\hat{m}_t = \frac{m_t}{1-\beta_1^{\,t}}, \qquad \hat{v}_t = \frac{v_t}{1-\beta_2^{\,t}}, \qquad \theta_{t+1} = \theta_t - \frac{\eta\,\hat{m}_t}{\sqrt{\hat{v}_t} + \varepsilon}$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

On the ravine above, that's why the purple Adam trace slides almost straight down the channel while orange thrashes: per-direction scaling plus momentum is exactly the combination a narrow valley punishes you for lacking.

Why the noise is a feature

There's a piece of sleight of hand in the name. We say "gradient descent" but in practice train with stochastic gradient descent, estimating the gradient from a small random batch rather than the whole dataset. The estimate is noisy, and for years that noise was treated as a regrettable cost of not affording the full gradient. It turns out to be doing real work.

A full-batch optimiser computes the exact gradient, and at a saddle the exact gradient is genuinely tiny, so it can sit on the ridge indefinitely, fooled into thinking a mountain pass is a summit. The noisy estimate from a mini-batch is almost never exactly zero, so it jostles the optimiser off the ridge and back onto a downhill direction. The randomness we apologise for is what rescues us from the plateaus of the previous section. For this to converge you want the step sizes to satisfy the Robbins-Monro conditions, $\sum_t \eta_t = \infty$ (you can still travel any distance) and $\sum_t \eta_t^2 < \infty$ (the noise eventually gets damped), which is the formal reason learning-rate schedules decay over training.

So is Adam simply the answer? Not quite, and this is the kind of cost-aware caveat worth keeping. Wilson and colleagues showed in 2017 that adaptive methods like Adam can converge to solutions that generalise slightly worse than well-tuned plain SGD with momentum, which is why a fair amount of computer-vision work still finishes training on plain SGD. It's a tendency, not a law: most large language models train end to end on Adam and its weight-decay variant AdamW perfectly happily. The honest summary is that the per-direction rescaling that makes Adam so robust to tune is the same thing that occasionally costs it a little at the very end.

Why not just use the curvature?

A reasonable objection: if the trouble is curvature, why not measure it? Newton's method does exactly that, multiplying the gradient by the inverse Hessian (the matrix of second derivatives).

$$\theta_{t+1} = \theta_t - \big[\nabla^2 f(\theta_t)\big]^{-1}\,\nabla f(\theta_t)$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

On a convex quadratic this converges in one step; it sees the bowl's exact shape and jumps to the bottom. The reason deep learning doesn't use it is brutal arithmetic. For $n$ parameters the Hessian has $n^2$ entries to store and costs on the order of $n^3$ to solve. At $n$ in the billions, $n^2$ is already impossible. So we settle for first-order methods that only ever touch the gradient, and we spend our cleverness, momentum, per-direction scaling, bias correction, approximating what Newton would have told us for free if only we could afford it.

There's a nice unification hiding here, and it's the kind of cross-field rhyme this site keeps running into. Minimising $f$ means finding where $\nabla f = 0$, which is a root-finding problem on the gradient. Newton's method for optimisation is just Newton's method for root-finding applied to $\nabla f$, and stochastic gradient descent is a stochastic root-finder, the original setting of that 1951 Robbins-Monro paper. Seen from the right angle, SGD and Newton's method are the same algorithm wearing different budgets. Some food for thought next time someone calls one "first-order" and the other "second-order" as if they were unrelated.

The loss surface, by the way, is a dynamical system: run the update and the parameters trace a flow, and the fixed points, minima and saddles, are exactly the equilibria we classify in the phase portraits post. Everything that learns on this site, the attention layers, the zoo of architectures, the KV-cache tricks, got that way by rolling downhill on a surface like the ones in the lab above. Set the preset to Bumpy, drop the learning rate, and watch the optimisers pick their way past the little traps. It's the same 1847 idea, still doing the work.

Reading further

• Cauchy, Méthode générale pour la résolution des systèmes d'équations simultanées (1847). Comptes Rendus 25, 536-538. Two and a bit pages; the origin of steepest descent.
• Robbins & Monro, A Stochastic Approximation Method (1951). Annals of Mathematical Statistics 22, 400-407. The ancestor of SGD, and the source of the step-size conditions above.
• Sutskever et al., On the importance of initialization and momentum in deep learning (2013). ICML. The paper that showed momentum, done right, matters for deep nets.
• Kingma & Ba, Adam: A Method for Stochastic Optimization (2015). ICLR. The optimiser you actually use, equation (4) and all.
• Dauphin et al., Identifying and attacking the saddle point problem (2014). NeurIPS. The argument that saddles, not local minima, dominate high-dimensional loss surfaces.
• Wilson et al., The Marginal Value of Adaptive Gradient Methods (2017). NeurIPS. The case that well-tuned SGD can still beat Adam at generalisation.