Where does generalization collide?

A model's performance on data drawn from the same distribution as its training set but not seen during training, usually reported as the gap between training and test error. Vapnik's statistical learning theory fixed the classical account: the gap is bounded by a capacity measure of the hypothesis class, so a model that can fit anything should generalise badly. Zhang and colleagues showed in 2017 (restated in their 2021 article) that deep networks fit random labels perfectly and still generalise on real ones, which broke the bound as an explanation. Belkin and colleagues' 2019 double-descent curve showed test error falling again past the interpolation threshold. The construct is a number on a held-out set, and the open question is why it is small.[1][2][3]

Unit

A trained model, on held-out samples from the training distribution

Recovery means

Not applicable, a gap is closed by regularisation or data, not recovered

Measured by

Test minus training loss; held-out accuracy; capacity and margin bounds

The move from observed instances to a universal claim, and the question of what, if anything, entitles it. Hume posed the problem in 1748: no experience of past regularity can justify the expectation that it continues, without assuming the very uniformity in question. Hempel's 1945 studies in confirmation tried to say formally when an instance supports a generalisation and produced the raven paradox. Goodman's 1955 grue predicate showed that the problem is worse than Hume said, every body of evidence confirms indefinitely many incompatible generalisations, and only entrenched predicates get projected. Here generalisation is a logical relation between statements, and the field's finding is that no purely formal rule licenses it.[4][5][6]

Unit

A universal statement, relative to its observed instances

Recovery means

Not applicable

Measured by

Not measured; assessed by confirmation logic and by the projectibility of predicates

Whether a causal finding holds over variations in persons, settings, treatments, and outcomes (Shadish, Cook, and Campbell's external validity, one of four validity types and the one they say is settled by a programme of studies rather than by any one). Cronbach's generalizability theory, set out with Gleser, Nanda, and Rajaratnam in 1972 and restated by Brennan in 2001, is the measurement version: a score is a sample from a universe of admissible observations, and analysis of variance partitions the error into facets (raters, occasions, items), so that a generalizability coefficient replaces the single reliability number. In both, generalising is something a researcher does with a claim, and the unit of analysis is the study or the score, not the learner.[7][8][9]

Unit

A finding or a score, across a defined universe of conditions

Recovery means

Not applicable, a finding that fails to generalise is bounded, not repaired

Measured by

Replication across persons, settings, and times; G-study variance components and the coefficient built from them

The effect of learning in one context on performance in another. Thorndike and Woodworth's 1901 experiments found that practice on one task improved others only where the two shared identical elements, which demolished the doctrine that studying Latin trains the mind in general. Barnett and Ceci's 2002 taxonomy sorted a century of contested results by what transfers and how far (across knowledge domain, physical context, temporal context, functional context, social context, and modality) and concluded that far transfer is rare and often unmeasured rather than absent. The learner is a person, the criterion is a second task, and the recurring finding is that generalisation is narrower than teachers hope.[10][11]

Unit

A learner, from a trained task to a target task

Recovery means

Not applicable, transfer either occurs at the target task or does not

Measured by

Performance on a transfer task relative to an untrained control; distance along the Barnett–Ceci dimensions

The tendency of a response conditioned to one stimulus to be elicited by other stimuli in proportion to their similarity. Pavlov described it in 1927 as irradiation of excitation across the cortex. Guttman and Kalish measured it in 1956 by training pigeons to peck at one wavelength and testing them at others, producing the generalisation gradient, a peaked curve that has been the field's dependent variable since. Shepard's 1987 universal law fixed the shape: when stimuli are placed in a psychological space recovered by multidimensional scaling, the probability of generalisation falls off as a concave exponential of distance, for pigeons, people, and every modality tested. The construct is an organism's response curve, not a claim it makes.[12][13][14]

Unit

An organism's response across a stimulus dimension

Recovery means

Not applicable, a gradient is sharpened by discrimination training, not restored

Measured by

Response rate at test stimuli spaced along a dimension; slope and peak of the gradient

A child's extension of a productive grammatical pattern to forms that are exceptions, as in goed and foots. Marcus and colleagues' 1992 monograph analysed thousands of past-tense utterances and found that overregularisation is rare (a few percent of irregular verbs) begins after the irregulars are already produced correctly, and fades as memory for the exceptions strengthens. Pinker's 1999 Words and Rules made it the test case for a dual mechanism: a rule adds -ed by default, and stored irregulars block it when retrieved. Generalisation here is evidence of a rule in the child's head, and the errors are its signature, not a failure of learning.[15][16]

Unit

A child's productions of inflected forms over development

Recovery means

Retreat from overgeneralisation, irregular forms winning back as retrieval strengthens

Measured by

Rate of overregularised forms in transcripts; elicited production of novel and irregular forms