Backpropagation
The algorithm that computes gradients through deep networks — the workhorse of training neural networks.
Backpropagation is the algorithm that computes the gradient of the loss with respect to every weight in a neural network, making gradient descent possible for networks with millions or billions of parameters. Training requires knowing how much each weight contributed to the error; backpropagation provides exactly that, efficiently, in one backward sweep through the network.
The algorithm is a direct application of the chain rule. During the forward pass, activations propagate from input to output, and the loss is computed at the end. During the backward pass, the gradient of the loss with respect to each layer's activations is propagated from output to input: each layer receives the gradient of the loss with respect to its output, multiplies by the local derivatives of its own transformation, and passes the result to the layer below. The gradient with respect to a weight is then the product of the incoming activation and the backpropagated error — for a layer with weight matrix W_l, activation a_{l−1}, and pre-activation z_l:
δ_L = ∇a_L L ⊙ σ'(z_L) (output layer)
δ_l = (W_{l+1}ᵀ δ_{l+1}) ⊙ σ'(z_l) (hidden layers)
∂L/∂W_l = δ_l · a_{l−1}ᵀ (weight gradients)
Here σ' is the derivative of the activation function and ⊙ is elementwise multiplication. The crucial property is efficiency: the backward pass costs about the same as the forward pass, because each intermediate activation is computed once and reused — backpropagation is reverse-mode automatic differentiation, a general technique for computing gradients of composed functions that predates neural networks (Seppo Linnainmaa's 1970 work, applied to networks by Paul Werbos in 1974).
The algorithm became the foundation of deep learning through the 1986 paper of Rumelhart, Hinton, and Williams, which showed that multilayer perceptrons could learn nontrivial functions and popularized the method. Its notorious difficulty was the vanishing gradient problem: with saturating activation functions such as the logistic sigmoid, the derivative is below 1 and the chain rule multiplies it layer after layer, so gradients shrink exponentially with depth and early layers learn almost nothing. Modern deep learning is largely a catalogue of fixes: the ReLU activation (derivative 0 or 1), careful initialization, batch normalization, and residual (skip) connections, as in ResNet, which give the gradient a direct path through the network.
Today backpropagation is invisible to most practitioners because automatic differentiation frameworks — TensorFlow, PyTorch, JAX — implement it behind a single call to .backward(). It is the mechanism behind every trained deep network, from convolutional neural networks in vision to the transformers behind large language models. Research on alternatives — approximations that avoid full backpropagation, biologically plausible learning rules — continues, but no alternative has come close to its combination of generality and efficiency.
Tags
backpropagation deep learning neural networks