Word Embeddings

Dense vector representations of words learned from text, in which semantically similar words have similar vectors.

Category: Natural Language Processing · Created: 2026-08-17 · Updated: 2026-08-17

Word embeddings are dense vector representations of words learned from large text corpora, with the property that words with similar meanings end up with similar vectors. Unlike one-hot encodings, which treat every word as an unrelated atomic symbol, embeddings capture semantic and syntactic similarity: the classic example king − man + woman ≈ queen holds in the learned vector space.

The founding algorithms are word2vec (Mikolov et al., 2013), with its CBOW and skip-gram architectures, GloVe (Pennington et al., 2014), built from global co-occurrence statistics, and fastText, which adds subword information so that rare and misspelled words still get reasonable vectors. All rest on the distributional hypothesis: words that appear in similar contexts tend to have similar meanings, so the models learn representations by predicting a word from its neighbors, or vice versa, and taking the hidden-layer weights as the embeddings.

Embeddings encode more than similarity: they cluster by topic, support analogical reasoning, and — as a well-documented concern — inherit the biases of their training text, reproducing associations such as gender-stereotyped analogies. Because they are learned from data, they also reflect historical usage rather than dictionary definitions.

Modern natural language processing has largely moved to contextual representations from transformer models such as BERT, which produce different vectors for the same word in different contexts and thereby solve polysemy. Static embeddings remain useful building blocks, however, and the token embeddings of today's large language models are the direct descendants of this line of work.

Word Embeddings

Tags

Related articles