gradient descent: a standalone construct

Augustin-Louis Cauchy set out the method in an 1847 note to the French Academy of Sciences, Méthode générale pour la résolution des systèmes d'équations simultanées, proposing that to find where a function is smallest one repeatedly takes a small step in the direction in which the function falls most steeply. The gradient of a function is the vector of its partial derivatives, which points toward the steepest increase, so stepping in the opposite direction moves toward lower values; each iteration subtracts a small multiple of that gradient, the multiple being the step size or learning rate, until the steps stop lowering the value. Haskell Curry's 1944 analysis gave steepest descent for nonlinear minimization a rigorous footing, and Robbins and Monro's 1951 stochastic approximation method extended it to noisy estimates of the slope, the form now called stochastic gradient descent. Rumelhart, Hinton, and Williams showed in 1986 that backpropagation computes this gradient efficiently through the layers of a neural network, which is why the method now trains models with millions of parameters. It is distinctive as a general first-order optimization procedure: it needs only the objective and its slope, not its curvature, and it applies wherever a differentiable objective is to be minimized, from fitting a regression line to training a deep network.[1][2][3][4]

Unit

A first-order optimization procedure defined by repeated steps proportional to the negative gradient of an objective

Recovery means

Not applicable

Measured by

Identified by its update rule: parameters are repeatedly adjusted by a step proportional to the negative gradient of the objective