Context Length as a KPI: Why Window Size Matters
How context window size affects what AI models can do, the tradeoffs of longer contexts, and how to choose the right context length for your application.
Why Context Length Matters as a KPI
Context window size determines what workflows are possible, not just what's convenient. Below 8K tokens: only small documents or simple Q&A. 32K: medium document analysis. 128K: long document analysis, multi-file codebases, extended conversations. 200K: comprehensive book analysis, large codebases. 1M: entire software projects, years of email, comprehensive research literature. 10M: unrestricted whole-organization knowledge.
Context window is often the architectural decision that determines whether an application is viable at all. A legal AI that needs to analyze 300-page contracts requires at minimum 200K context. A code assistant that needs to understand entire repositories needs 500K+. Choosing a model without checking context window against your documents is a common early mistake.
Long Context vs RAG: When to Use Each
Two architectural approaches handle large knowledge bases: put everything in context (long context) or retrieve only relevant pieces (RAG). Long context is simpler to implement, always has all information available, and handles questions that require synthesis across many sources. RAG is cheaper (fewer tokens = lower cost), faster (smaller context = lower TTFT), and works for knowledge bases too large for any context window.
Current best practice: use RAG for knowledge bases larger than ~500K tokens or when cost is critical, use long context for precision tasks where every detail matters and the document set is bounded. Hybrid approaches, retrieve to narrow the field, then use long context for the relevant documents, often provide the best of both.
Does Model Quality Hold at Long Context?
Not all models maintain quality across their full claimed context window. The 'needle in a haystack' (NIAH) test measures whether a model can reliably retrieve a specific fact placed at different positions throughout a long document. Frontier models generally pass NIAH at full context length; smaller or older models often show performance degradation beyond 50-60% of their nominal context.
Lost-in-the-middle performance is a subtler issue: models are more reliable at retrieving information at the start or end of context than in the middle. For multi-document retrieval tasks, this means document ordering in the prompt affects reliability. Frontier models have improved significantly on this, but position effects remain for all current models.
Context Length and Cost
Longer contexts cost more: every token in context is charged as an input token. Sending a 100K-token document to Claude Sonnet 4.6 ($3/M input) costs $0.30 per query. Send 1,000 such queries per day and you're spending $300/day purely on document context, before any output costs. Context efficiency is a genuine cost driver.
Prompt caching (from Anthropic and OpenAI) dramatically reduces the effective cost of repeated long contexts: pay the full price once, then 10-25% for subsequent requests that reuse the same prefix. For applications with stable system prompts or large fixed documents, caching can reduce input costs by 75-90%.
Read next
Context Windows Explained: The Working Memory of LLMs
What context windows are, why they matter for building AI applications, how they've grown from 4K to 10M tokens, and how to manage them effectively.
Tokens and Tokenization: The Building Blocks of LLMs
Everything you need to know about tokens, how LLMs split text into pieces, why tokenization matters for cost and performance, and how different languages tokenize.
Cost Per Million Tokens: The AI Economics Guide
How LLM API pricing works, why output tokens cost more than input, how to calculate actual task costs, and how prices have changed over time.