Where does calibration collide?

A forecasting system is calibrated when the events it assigns a probability p to actually occur about a fraction p of the time, so that a stated probability can be read at face value. Brier's 1950 Verification of forecasts expressed in terms of probability introduced a score for grading probability forecasts against what happened, and Dawid's 1982 The well-calibrated Bayesian gave the property a formal definition: a probability assessor is well calibrated if, over the long run, its forecasts agree with observed frequencies. The referent here is a mathematical property of a sequence of probability statements, judged against recorded outcomes, and it is not a physical adjustment of a measuring device or the choosing of an economic model's parameters.[1][2]

Unit

A property of a sequence of probability forecasts, graded against observed outcomes

Recovery means

Not applicable

Measured by

Assessed with reliability diagrams and with scores such as the Brier score, which splits into a calibration part and a refinement part

The comparison of a measuring instrument's readings against a recognised reference standard, together with the adjustment or documented correction that brings the instrument into agreement with it. Eisenhart's 1963 Realistic evaluation of the precision and accuracy of instrument calibration systems described how a calibration chain links a working instrument through reference standards up to a national standard, and how the resulting uncertainty should be reported. The practice of checking instruments against a fixed standard is centuries older than this paper, so Eisenhart 1963 is cited as a clear modern statement of the meaning rather than as the origin of the idea. The referent is a physical procedure carried out on real hardware and reported as a correction with an uncertainty, unlike a property of probability forecasts or the fitting of an economic model.[3]

Unit

A single instrument's correction and stated uncertainty relative to a reference standard

Recovery means

Instruments drift over time, so calibration is repeated at intervals to restore agreement with the standard

Measured by

Determined by comparing readings to a traceable standard and computing the correction and its uncertainty, often summarised as a calibration curve

In economic modelling, calibration is the practice of assigning a model's parameters values drawn from prior evidence, such as long-run averages and separate micro studies, so that the model reproduces selected features of observed data rather than estimating those parameters by fitting the model statistically. Kydland and Prescott's 1982 Time to build and aggregate fluctuations built a business-cycle model whose parameters were set this way, and their 1996 The computational experiment: An econometric tool laid out the approach explicitly as a method. The referent is a modelling methodology, a disciplined way of choosing numbers for a theoretical model, and it has nothing to do with adjusting a physical instrument or with checking whether stated probabilities match how often events occur.[4][5]

Unit

A set of parameter values assigned to a theoretical model

Recovery means

Not applicable

Measured by

Chosen so that model-generated quantities match selected empirical targets, then checked by how well the model reproduces the features it was aimed at