cloze: one construct, many systems

In reading research, the cloze procedure deletes words from a passage at regular intervals and asks the reader to supply what is missing, on the premise that the better a reader understands the text and its language, the more accurately they can fill the gaps. Taylor coined the term in 1953, from the idea of closure, and proposed it as a measure of how readable a passage is for a given reader. The referent is a fill-the-gap task used to gauge comprehension and text difficulty, scored by the proportion of correct restorations.[1]

Unit

The proportion of deleted words a reader correctly restores

Recovery means

Not applicable

Measured by

Scored by how many deleted words the reader supplies correctly, taken as an index of comprehension and readability

In natural language processing, the same fill-the-gap construct trains and tests language models: tokens in a text are hidden and the model must predict them from the surrounding context. Devlin and colleagues' 2019 BERT model used this masked-language-modelling objective, explicitly described as a cloze task, to learn deep bidirectional representations of language. It is the identical construct as the reading-education cloze procedure, here applied to a machine rather than a human reader, measured by how well the model recovers the hidden tokens rather than by a student's comprehension.[2]

Unit

How accurately a model predicts hidden tokens from context

Recovery means

Not applicable

Measured by

Evaluated by the model's accuracy or loss in predicting masked tokens and by downstream task performance