KV Cache
A memory optimization that stores previously computed attention keys and values so the model doesn't recompute them when generating each new token.
During inference, a transformer model computes attention across all tokens in the context for every new token it generates. Without a KV (key-value) cache, this requires re-processing the entire input sequence repeatedly — an O(n²) operation that becomes prohibitively slow for long contexts.
The KV cache stores the key and value matrices computed during the initial processing of the prompt. When generating the next token, the model only needs to compute attention for the new token against the cached keys and values, reducing redundant computation dramatically. This is why context length affects memory requirements more than compute for generation.
KV cache memory is a significant constraint in serving large models with long contexts. A 1M-token context with a large model can require hundreds of gigabytes of KV cache memory. Research into efficient KV cache management — including quantization, eviction policies, and offloading — is an active area of systems work.
Read next
Context Windows Explained: The Working Memory of LLMs
What context windows are, why they matter for building AI applications, how they've grown from 4K to 10M tokens, and how to manage them effectively.
The Transformer Architecture Explained
A deep dive into the transformer architecture — the neural network design that powers virtually every major LLM, from its attention mechanism to positional encodings.
LLM Inference: How Models Generate Text
Inference is what happens when an LLM produces a response. Understanding it helps you optimize for speed, cost, and quality.