backpropagation: a standalone construct

Seppo Linnainmaa's 1970 master's thesis introduced the reverse mode of what is now called automatic differentiation, a procedure that computes the exact derivatives of a numerical calculation by working backward through its steps, and Paul Werbos's 1974 doctoral thesis applied that idea to training layered networks. The method was named and popularised by David Rumelhart, Geoffrey Hinton, and Ronald Williams in a 1986 Nature paper, which showed that a multi-layer network could learn useful internal representations. Backpropagation computes, for every adjustable weight in the network, the gradient of the loss, the single number that scores how far the network's output is from the desired output, by applying the chain rule of calculus one layer at a time from the output back toward the input. Because each layer reuses the derivative values already computed for the layer above it, the full set of gradients is obtained in roughly the same amount of work as one forward pass through the network, which is what makes training networks with millions of weights practical. It supplies the gradients that an optimisation method such as gradient descent then uses to adjust the weights, and this pairing underlies most modern deep learning.[1][2][3]

Unit

A gradient vector: one partial derivative of the loss with respect to each weight and bias in the network

Recovery means

Not applicable

Measured by

Verified by comparing its analytic gradients against numerical finite-difference estimates (gradient checking), and implemented as reverse-mode automatic differentiation in software such as autograd libraries