Where does embedding collide?

A dense, low-dimensional vector learned for a word or item so that its position in a continuous space reflects the contexts in which it appears. The name comes from work on distributed representations: Bengio, Ducharme, Vincent, and Jauvin's A Neural Probabilistic Language Model (2003) learned word vectors inside a language model, and Mikolov, Chen, Corrado, and Dean's Efficient Estimation of Word Representations in Vector Space (2013) made them cheap to train at scale with word2vec. The referent is a point in R^n assigned to a token, and what marks it is that the vector is fit to data so that nearby points share usage. It differs from every other meaning here because nothing is physically placed inside anything; the embedding is a coordinate, and embedded means mapped into a vector space, not lodged in a device or a unit.[1][2]

Unit

A learned dense vector for a word or item

Recovery means

Not applicable

Measured by

Cosine similarity between vectors and performance on word-analogy benchmarks

An injective, structure-preserving map that places one space inside another without self-intersection or loss of structure, so the image is a faithful copy of the original. Whitney's Differentiable Manifolds (1936) proved that every smooth manifold embeds in a Euclidean space of finite dimension, giving the meaning its canonical result: an abstract manifold can be realised as a subset of Euclidean space. The referent is a map between spaces, not an object and not a vector of learned numbers; the same word names the arrow, and the requirement is exactness, one-to-one and preserving the relevant structure, not similarity fit to data. A learned word embedding is called an embedding by analogy to this map, but it is optimised rather than proven, and it may collapse distinct items onto nearby points.[3]

Unit

An injective structure-preserving map between spaces

Recovery means

Not applicable

Measured by

Existence is proven, not measured; one exhibits an injective immersion into R^n