Where does validity collide?

Psychometrics

the degree to which evidence and theory support interpretations of test scores for their proposed uses.

The degree to which evidence and theory support the interpretations of test scores for their proposed uses. Cronbach and Meehl introduced construct validity in 1955 as a property of the inference from score to construct, located in a nomological network; Messick unified the earlier types (content, criterion, construct) into one evidential argument; the current Standards make it explicit that validity belongs to the interpretation, not the test. Borsboom and colleagues dissent: a test is valid if the attribute exists and causally produces variation in the scores, a property of the instrument after all.[1][2][3][4]

Unit: An interpretation of a score for a use
Recovery means: Not applicable, validity is a judgment, accumulated rather than restored
Measured by: Convergent and discriminant correlations, factor structure, criterion prediction, response processes, consequences

Experimental design

the approximate truth of a causal inference, whether the treatment produced the difference and whether it holds across settings.

The approximate truth of an inference about a cause, whether the treatment, and not history, maturation, selection, or regression, produced the observed difference (internal validity), and whether the relationship holds across persons, settings, and times (external validity). Campbell's list of threats is a checklist for a study, not a scale. Shadish, Cook, and Campbell added statistical-conclusion validity and reclaimed construct validity for the labelling of treatments and outcomes.[5][6]

Unit: A causal claim from a study
Recovery means: Not applicable
Measured by: Design features (randomization, control, blinding) that rule out named threats

Ecological psychology

how well a proximal cue predicts a distal criterion in an organism's natural environment, a property of the environment.

Brunswik's ecological validity is a correlation: how well a proximal cue predicts a distal criterion in the organism's natural environment. It is a property of the environment, and his call for representative design (sampling situations as carefully as subjects) followed from it. The phrase has since drifted to mean "how much the lab resembles real life," which is a different claim, made about a study rather than a cue.[7]

Unit: A cue in an environment
Recovery means: Not applicable
Measured by: Cue–criterion correlation across a representative sample of situations

Qualitative research

a replacement for validity in naturalistic inquiry, built from credibility, transferability, dependability, and confirmability.

Lincoln and Guba argued that validity presupposes a single reality against which findings can be checked, and replaced it for naturalistic inquiry with trustworthiness (credibility, transferability, dependability, and confirmability) each paired with a procedure such as member checking, thick description, or an audit trail. Many qualitative researchers still say "validity" and mean this; some reject the word altogether.[8]

Unit: A study and its account
Recovery means: Not applicable
Measured by: Procedures: prolonged engagement, triangulation, member checks, audit trails

Logic & philosophy

the property of an argument whose form makes it impossible for the premises to be true and the conclusion false.

An argument is valid when it is impossible for the premises to be true and the conclusion false, a property of the argument's form, entirely independent of whether any premise is actually true. A valid argument with true premises is sound. Nothing is measured, nothing is a matter of degree, and evidence is irrelevant: validity here is decided, not supported.[9]

Unit: An argument
Recovery means: Not applicable
Measured by: Not measured, demonstrated by form, or refuted by counterexample

Statistics & machine learning

estimating how a fitted model will perform on unseen data by holding data out or rotating through folds.

Not a property but a procedure: estimating how a fitted model will perform on data it has not seen, by holding some data out (a validation set) or rotating through folds (cross-validation). Stone formalized the choice of a model by its cross-validatory score. "Validation" here says nothing about whether the model's inputs mean what their names suggest (the psychometric question) only whether the predictions generalize.[10]

Unit: A fitted model
Recovery means: Not applicable
Measured by: Held-out loss, accuracy, AUC; cross-validated error

References

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061
Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54(4), 297–312. https://doi.org/10.1037/h0040950
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Brunswik, E. (1956). Perception and the representative design of psychological experiments (2nd ed.). University of California Press. https://doi.org/10.1525/9780520350519
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Sage. ISBN 9780803924314
Copi, I. M., Cohen, C., & McMahon, K. (2011). Introduction to logic (14th ed.). Pearson.
Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society: Series B, 36(2), 111–133. https://doi.org/10.1111/j.2517-6161.1974.tb00994.x