Gradient Descent
The iterative optimization algorithm at the heart of training machine learning models: how it works, its variants, and its pitfalls.
Gradient descent is the iterative optimization algorithm used to train most machine learning models. A model has parameters θ and a loss function L(θ) that measures how badly the model fits the training data; training is the search for parameters that minimize the loss. Gradient descent performs that search by repeatedly moving the parameters in the direction of steepest decrease of the loss: θ ← θ − η∇L(θ), where ∇L is the gradient vector and η is the learning rate.
The geometric intuition is descending a hill by always stepping downhill: the gradient points uphill, so subtracting it moves downhill, and each step lands at a lower point on the loss surface. The learning rate controls step size — too large, and the steps overshoot and oscillate or diverge; too small, and training crawls. In practice the learning rate is the single most important hyperparameter to tune, and schedules that decay it during training are standard.
For the linear models of classical statistics the loss surface is convex — a bowl with one minimum — and gradient descent provably converges to the global optimum. Deep neural networks are not convex: their loss surfaces contain many local minima and, far more importantly, vast flat regions dominated by saddle points where the gradient is zero. Remarkably, in practice gradient descent on large networks still finds good solutions, and the theory of why this works — the loss landscape of overparameterized networks — remains an active research area.
Three variants differ in how much data each step uses. Batch gradient descent computes the gradient over the entire dataset — accurate but expensive. Stochastic gradient descent (SGD) uses one example per step — cheap and noisy, and the noise can help escape shallow minima. Mini-batch gradient descent uses a small random subset, typically 32–256 examples, balancing accuracy, speed, and hardware efficiency; this is the standard choice for deep learning. On modern GPUs, the matrix operations of a mini-batch are what make training large models feasible at all.
Refinements address the weaknesses of plain descent. Momentum accumulates a running average of past gradients, smoothing the path and speeding progress through ravines. Adaptive methods — AdaGrad, RMSProp, Adam — scale the learning rate per parameter using gradient statistics; Adam has become the default optimizer for many deep learning applications. Feature scaling also matters: when input features have very different magnitudes, the loss surface becomes elongated and gradient descent zigzags; normalizing features to comparable scales makes the geometry rounder and the steps efficient. Early stopping — ending training when validation error stops improving — uses the same machinery as regularization to prevent overfitting. In every modern training stack, from simple regressions to transformers, gradient descent is the engine; backpropagation is the algorithm that computes the gradients it needs.
Tags
gradient descent machine learning optimization