Where does tokenization collide?

In natural language processing, tokenization is the first step that breaks a stream of raw text into tokens, the small units a model actually reads, such as words, subwords, or characters. Webster and Kit's 1992 Tokenization as the initial phase in NLP set out this segmentation as the foundation on which later analysis depends, and later subword methods, such as the byte-pair encoding of Sennrich, Haddow, and Birch's 2016 Neural machine translation of rare words with subword units, learn a vocabulary of frequent character sequences so that rare and unseen words can still be represented. The referent is an operation that cuts text into processable pieces, and it has nothing to do with hiding sensitive data behind a stand-in or recording ownership of an asset on a ledger.[1][2]

Unit

One token: a word, subword, or character unit of text

Recovery means

The original text can be rejoined by concatenating the tokens in order, sometimes with recorded spacing

Measured by

Evaluated by vocabulary size, the number of tokens a text produces, and how well the segmentation supports later tasks

In data security, tokenization replaces a sensitive data element, such as a payment card number, with a non-sensitive substitute called a token that carries no exploitable value on its own, so the real data can be pulled out of a working system and held separately. Díaz-Santiago, Rodríguez-Henríquez, and Chakraborty's 2016 A cryptographic study of tokenization systems gave a formal treatment of these systems and contrasted them with encryption, where the protected value is a reversible transformation of the original rather than an unrelated stand-in mapped through a secured table. The referent is a protective substitution that keeps secrets out of harm's way, not a way of cutting text into units or of registering an asset's ownership.[3]

Unit

One token standing in for one sensitive value

Recovery means

The original value is retrieved only through a secured token vault or mapping held by a trusted service

Measured by

Assessed by whether the token is irreversible without the vault and by how many systems it removes from contact with real data

In finance and blockchain, asset tokenization represents ownership of a real-world or financial asset, such as property, a bond, or a company share, as digital tokens recorded on a distributed ledger, so the asset can be divided into small fractions and traded as those tokens. Lambert, Liebau, and Roosenboom's 2021 Security token offerings studied how firms raise capital by issuing such tokens as regulated claims on assets or cash flows, and how these offerings differ from earlier unregulated coin sales. The referent is a claim on value written onto a ledger and moved between owners, not a substitute that hides sensitive data or a unit that a language model reads.[4]

Unit

One token representing a fraction of an asset's ownership

Recovery means

Not applicable

Measured by

Tracked by ledger records of token holdings and transfers and by the value of the underlying asset