cross-validation: a standalone construct

Cross-validation was set out as a formal procedure by Stone (1974) in the Journal of the Royal Statistical Society and, in the same period, by Geisser (1975) as the predictive sample reuse method in the Journal of the American Statistical Association. The idea is to estimate how well a fitted model will predict new data by repeatedly splitting the available sample into a part used to fit the model and a held-out part used only to test it, then averaging the prediction error across the splits. The most common forms are k-fold cross-validation, which divides the data into k equal groups and rotates which group is held out, and leave-one-out cross-validation, in which each single observation is held out in turn; Arlot and Celisse (2010) survey the many variants and the settings in which each is appropriate. What makes cross-validation a landmark is that it estimates out-of-sample prediction error directly from the data at hand, without assuming a particular error model, which is why it is used across statistics and machine learning to compare models, tune model settings, and guard against fitting a model too closely to its own training sample.[1][2][3]

Unit

A model-assessment procedure defined by its data-splitting scheme, for example the number of folds

Recovery means

Not applicable

Measured by

Computed as the average prediction error over the held-out folds or observations, for a chosen splitting scheme such as k-fold or leave-one-out