Where does regularization collide?

The practice of adding a penalty term to the quantity a model is fit to minimise, so that overly large or complex parameter values are discouraged and the model generalises better to new data. Hoerl and Kennard's 1970 Ridge regression: Biased estimation for nonorthogonal problems introduced adding a squared-size penalty on the coefficients, trading a little bias for a large drop in variance, and later work added the absolute-size (lasso) penalty. Here the referent is a deliberate tug against overfitting a finite, noisy dataset, chosen to improve prediction, and it is not a way of taming infinite quantities in physics or of stabilising the solution of an inverse problem.[1]

Unit

A penalty term weighted by a tuning parameter, added to the fitting objective

Recovery means

Not applicable

Measured by

Set by the penalty strength, chosen by cross-validation, and read off as shrinkage of the fitted coefficients and improved out-of-sample error

In quantum field theory, a family of techniques for temporarily rewriting a divergent integral or sum so that it becomes finite and can be manipulated, as a preparatory step before the infinities are absorbed by renormalization. 't Hooft and Veltman's 1972 Regularization and renormalization of gauge fields established dimensional regularization, which continues the number of spacetime dimensions away from four so that otherwise infinite loop integrals return finite values controlled by a small parameter. Here the referent is a bookkeeping device for controlling infinities in a physical calculation, unrelated to penalising a statistical model or to repairing an ill-posed mathematical problem.[2]

Unit

A regulator (a cutoff, or a shift in dimension) that renders a divergent expression finite

Recovery means

The regulator is removed at the end, after renormalization has absorbed the divergences into physical constants

Measured by

Tracked by how computed quantities depend on the regulator, and validated when regulator-independent predictions match experiment

For an ill-posed inverse problem, where the solution does not depend continuously on the data so that tiny measurement errors blow up into huge errors in the answer, regularization means replacing the problem with a nearby, well-posed one whose solution is stable and approaches the true answer as the data improve. Engl, Hanke, and Neubauer's 1996 Regularization of inverse problems develops this theory, including Tikhonov's method of adding a stabilising term governed by a regularization parameter. Here the referent is a repair of an unstable mathematical problem itself, which is why it shares the Tikhonov root with the statistical meaning yet is aimed at recovering a hidden cause from data, not at guarding a predictive model against overfitting.[3]

Unit

A regularization parameter balancing fidelity to the data against stability of the solution

Recovery means

Not applicable

Measured by

Assessed by convergence of the stabilised solution to the true one as noise falls, with the parameter chosen by rules such as the discrepancy principle