Where does recall collide?

The proportion of all relevant documents in a collection that a search or review actually retrieved. Kent, Berry, Luehrs, and Perry proposed the measure in 1955, alongside what became precision, as the operational criteria for judging a retrieval system. The pair became the yardstick of document review in litigation when Grossman and Cormack showed in 2011 that technology-assisted review reached recall at least as high as exhaustive manual review at a fraction of the effort, fixed the vocabulary in their 2013 glossary, and Da Silva Moore accepted the method in 2012 with recall as the number the parties argued about. Rio Tinto confirmed in 2015 that the method no longer needed defending, only its validation. Cormack and Grossman's 2014 SIGIR study compared training protocols on that criterion. Recall here is a completeness rate estimated by sampling against a relevance standard, and it is the measure a court is asked to accept as proof that a production is reasonable.[1][2][3]

Unit

A set of retrieved documents against the set of relevant ones

Recovery means

Not applicable, the criterion is the estimated fraction retrieved, with a confidence interval

Measured by

Sampling of the unreviewed remainder against assessor relevance judgments; recall at a stopping point

The proportion of actual positives a classifier labels positive: true positives over true positives plus false negatives. It is the same number epidemiology calls sensitivity and signal detection calls the hit rate, and Powers's 2011 survey traced the several names and showed how recall, precision, F-measure, and ROC analysis relate and where each misleads. Saito and Rehmsmeier showed in 2015 that with imbalanced classes the ROC curve, which plots recall against the false positive rate, can look excellent while the precision–recall curve reveals that most positive predictions are wrong. Recall here is one cell-ratio of a confusion matrix, defined for any binary decision, and it is traded against precision by moving a threshold.[4][5]

Unit

A classifier's decisions over a labelled test set

Recovery means

Not applicable

Measured by

True positive rate at a threshold; area under the precision–recall curve

The production from memory of material learned earlier, in contrast to recognition, where the material is presented and the person judges whether it was seen. Ebbinghaus began the experimental study in 1885 with nonsense syllables and savings scores. Murdock reported in 1962 the serial position curve of free recall: the first and last items of a list are recalled best, and the recency part vanishes with a delay. Tulving and Pearlstone showed in 1966 that giving category cues at test recovered words that free recall had missed, and drew the distinction between what is available in memory and what is accessible at the moment of retrieval. Recall here is a performance of an individual on a test, and its interest is in what the failures reveal about how memory is stored.[6][7][8]

Unit

An individual's output on a memory test

Recovery means

Retrieval of an item that was available but inaccessible, typically through a cue

Measured by

Proportion of studied items produced in free, cued, or serial recall

A copy-testing measure of whether people who were exposed to an advertisement can report it later, unaided if they volunteer the brand when asked about the product category, aided if the brand is named. Day-after recall, the surveying of viewers the day after a broadcast, was the industry standard for television for decades. Singh, Rothschild, and Churchill compared recall and recognition as measures of forgetting in 1988 and found they decayed at different rates and answered different questions. Wells's 2000 review set recall beside recognition and rating scales, and argued that each captured a different aspect of an advertisement's effect and that none was the effect itself. Recall here is a survey statistic about an audience, used to decide whether a campaign runs, and it borrows the psychology laboratory's word for a commercial purpose.[9][10]

Unit

An advertisement's audience, surveyed after exposure

Recovery means

Not applicable, the criterion is the share of the audience that remembers

Measured by

Percentage aided and unaided recall in day-after or in-market surveys