The Transformer Architecture
The attention-based architecture behind modern language models: self-attention, positional encoding, and why it replaced recurrence.
The transformer is the neural network architecture introduced in the 2017 paper "Attention Is All You Need" by Vaswani and colleagues at Google, and it has become the standard architecture for natural language processing and much of modern machine learning. Its decisive innovation was to replace the recurrence of earlier sequence models (RNNs, LSTMs) with a mechanism called self-attention, which lets every position in a sequence attend directly to every other position.
Recurrent networks processed sequences one token at a time, carrying a hidden state forward; this made them inherently sequential and slow to train, and long-range information faded as it was passed step by step. Self-attention fixes both problems. Each token is embedded and projected into three vectors — a query, a key, and a value. Attention weights are computed by comparing each query with all keys, scaled and normalized with a softmax, and the output for each position is a weighted sum of all values:
Attention(Q, K, V) = softmax(QKᵀ / √d) · V
A token thus gathers information from the whole sequence in one operation, with the weights learned from data — which is how a pronoun can look back to its antecedent or a translation can align source and target words. Multi-head attention runs several of these computations in parallel, letting the model attend over different subspaces at once, and each block adds a per-token feed-forward network, residual connections, and layer normalization. Encoder stacks process the input; decoder stacks add masked attention so generation cannot peek at future tokens.
Because attention is order-agnostic — it treats the sequence as a set — transformers add positional information explicitly, through sinusoidal or learned positional encodings. The architecture is also extraordinarily parallel: the heavy operations are matrix multiplications over whole sequences, exactly what GPUs do best, which is why transformers could scale to far larger models and datasets than recurrent networks ever could.
The consequences transformed the field. BERT (2018) showed that a pretrained encoder could be fine-tuned to state-of-the-art results across NLP tasks; GPT models showed that an autoregressive decoder trained on enough text becomes a general language model; and scaling — more parameters, more data, more compute — proved empirically to produce steady capability gains, the phenomenon behind large language models. Transformers now also process images (vision transformers), audio, code, and multimodal inputs, and they power the encoder–decoder models of modern machine translation and speech recognition.
The architecture has known costs: self-attention scales quadratically with sequence length, which limits context windows and has driven research into sparse attention and efficient implementations such as FlashAttention; and the fixed-size context means long documents must be truncated or summarized. Nevertheless, the transformer's combination of parallelism, long-range attention, and scalability made it the foundation of the current generation of artificial intelligence systems.
Tags
deep learning natural language processing transformers