Where does token collide?

The distinction between an abstract kind and its particular instances. In the type-token distinction, set out by Linda Wetzel in the Stanford Encyclopedia of Philosophy and traced to Charles Sanders Peirce, a type is a repeatable, abstract kind and a token is one concrete occurrence of it. The stock example: the printed phrase "the the" has one word-type but two tokens, because the same type occurs twice as two separate ink marks. The referent is a metaphysical relation between a universal and its instances, and a token is whatever counts as a single occurrence of the type. This is the original philosophical meaning, and it differs from every other on this page because a token is defined only against its type, not against a group, a stream of text, or a reward.[1]

Unit

One concrete instance of an abstract type

Recovery means

Not applicable

Measured by

Not measured; a conceptual distinction used in logic, metaphysics, and linguistics

A discrete unit of text produced by tokenization and consumed downstream by a parser or a language model. In compiler design and natural language processing, described in Jurafsky and Martin's Speech and Language Processing, tokenization is the step that splits a stream of characters into tokens: words, punctuation, or the sub-word pieces a modern model operates on. The referent is an operational unit in a processing pipeline, individuated by an algorithm rather than by meaning, and its boundaries depend on which tokenizer is chosen. It borrows the type-token vocabulary loosely, since many identical tokens can share one vocabulary entry, but here a token is a countable slice of input, not a philosophical instance and not a person.[2]

Unit

One unit of text emitted by a tokenizer

Recovery means

Not applicable

Measured by

Counted directly; token counts index sequence length, cost, and context limits

A member of an underrepresented group who is present in very small numbers, made highly visible, and treated as a symbol of their whole category. Rosabeth Moss Kanter, in Men and Women of the Corporation, described how a lone or near-lone woman among men becomes a token: watched more closely, read as a representative of all women rather than as an individual, and pushed into stereotyped roles. The referent is a person in a skewed group, and the unit is a solo or near-solo member standing in for a category. This meaning carries a moral and political charge the others lack, and the token here is a human being whose scarcity, not any abstract type or slice of text, defines the position.[3]

Unit

A solo or near-solo group member treated as a symbol

Recovery means

Not applicable

Measured by

Studied through the proportion of the minority in a setting and its effects on visibility and role

A physical or symbolic object handed out as a reinforcer that can be exchanged later for a chosen reward. In the token economy set out by Teodoro Ayllon and Nathan Azrin in The Token Economy, tokens are generalized conditioned reinforcers: earned for target behaviours and traded for backup rewards such as goods, privileges, or time. The referent is a unit of exchangeable reinforcement inside a behavioural program, and its value comes entirely from what it can be swapped for. It shares nothing with the other meanings but the word: a token here is a lever on behaviour, not an instance of a type, a slice of text, or a person in a group.[4]

Unit

One exchangeable conditioned reinforcer

Recovery means

Not applicable

Measured by

Tracked by tokens earned and exchanged against target behaviours in an applied program