Transformer
The neural network architecture that underpins virtually all modern LLMs, introduced in 2017, built around self-attention mechanisms that process entire sequences in parallel.
The Transformer architecture was introduced in the 2017 paper 'Attention Is All You Need' by Vaswani et al. at Google Brain. It replaced sequential recurrent networks (RNNs and LSTMs) with self-attention, which processes all tokens in a sequence simultaneously and can directly model long-range dependencies. This parallelism made transformers dramatically more efficient to train on modern hardware.
A standard transformer consists of stacked encoder and/or decoder blocks. Each block contains a multi-head self-attention layer, a feed-forward network, residual connections, and layer normalization. GPT-style models use only the decoder component (they predict the next token); BERT-style models use only the encoder. T5 and other models use both.
The transformer's impact cannot be overstated. It is the architecture behind GPT, Claude, Gemini, Llama, DALL-E, Stable Diffusion, AlphaFold, and most other frontier AI systems. Its combination of scalability, expressiveness, and compatibility with GPU parallelism made it the dominant deep learning building block of the 2020s.
Read next
The Transformer Architecture Explained
A deep dive into the transformer architecture — the neural network design that powers virtually every major LLM, from its attention mechanism to positional encodings.
The Attention Mechanism: How LLMs Understand Context
A clear explanation of self-attention — the mathematical operation at the heart of every transformer that allows language models to understand relationships between words.
How LLMs Work: A Technical Overview
A clear technical explanation of how large language models actually process text, generate responses, and represent knowledge — from tokenization to sampling.