The Transformer Architecture Explained

Key Takeaways

Attention Is All You Need

In June 2017, eight Google Brain researchers published a paper titled 'Attention Is All You Need.' It introduced the Transformer, an architecture that replaced recurrent connections entirely with self-attention. The claim seemed bold: could a model with no sequential processing truly understand language? The results were decisive. The transformer outperformed every existing architecture on translation benchmarks and trained in a fraction of the time.

The impact of this paper is difficult to overstate. GPT, BERT, T5, Claude, Gemini, Llama, every major AI system of the past seven years is built on the foundations laid in those pages. Understanding the transformer is understanding the engine of the AI revolution.

Encoder vs Decoder Transformers

The original transformer had two components: an encoder (which reads input text and creates contextual representations) and a decoder (which generates output tokens, attending to both the encoder's output and previously generated tokens). This architecture is natural for translation: the encoder reads French, the decoder generates English.

Modern LLMs predominantly use decoder-only architectures (GPT, Llama, Claude, Gemini). Decoder-only models are trained to predict the next token in a sequence, which makes them natural generative models. BERT and its variants use encoder-only architectures, which excel at understanding and classification tasks. T5 uses the full encoder-decoder setup, performing well on tasks that can be framed as sequence-to-sequence.

Self-Attention: The Core Mechanism

Self-attention works by computing three vectors for each token: Query (Q), Key (K), and Value (V). These are linear projections of the token's embedding. Attention scores are computed as the dot product of each Query with every Key, scaled by the square root of the key dimension, then passed through softmax to create probability weights. The output for each token is the weighted sum of all Values.

Intuitively: the Query asks 'what am I looking for?', the Key says 'what do I represent?', and the Value says 'what information do I provide?' The dot product of Q and K determines relevance; high relevance → high weight on that K's corresponding V. This is why the attention mechanism can model arbitrary dependencies between tokens regardless of their distance.

Multi-Head Attention

Modern transformers use multi-head attention: the Q, K, V projections are split into multiple 'heads' that run attention in parallel, each in a lower-dimensional subspace. Different heads learn to attend to different types of relationships, one head might track grammatical subject-verb agreement, another coreference, another semantic similarity.

The outputs of all heads are concatenated and projected back to the model dimension. Modern frontier models use 32-128 attention heads. Grouped-Query Attention (GQA), used by Llama 3 and others, reduces the number of K/V heads while keeping Q heads at full count, cutting the KV cache memory requirements without significantly impacting quality.

Positional Encoding

Attention is order-agnostic, without positional information, the model can't distinguish 'dog bites man' from 'man bites dog.' Positional encodings inject position information into the embeddings. The original transformer used sinusoidal encodings; modern LLMs use RoPE (Rotary Position Embedding), which encodes position through rotations in embedding space and generalizes well to sequences longer than those seen during training.

Positional encoding is what allows transformers to have configurable context windows. Models can be extended to handle longer sequences by adjusting their positional encoding scheme, though performance typically degrades beyond the training length. Research into better positional encodings is an active area, with YaRN and ALiBi showing promise for long-context extrapolation.

Why Transformers Displaced Everything Else

Before transformers, LSTMs and RNNs were the dominant sequence models. They processed text token by token, maintaining a hidden state that compressed previous context. This sequential processing couldn't be parallelized, making training slow. Long-range dependencies were difficult to maintain as information had to travel through many steps.

Transformers process all tokens simultaneously, directly attending between any two positions in O(n²) operations. This is both their strength (direct long-range modeling) and weakness (quadratic memory in sequence length). The parallelism maps perfectly to GPU matrix operations, enabling the massive scale-up that produced today's frontier models. Alternative architectures like Mamba and RWKV attempt to recover the efficiency of sequential models while matching transformer quality, an active research competition.