Prompt caching lets providers reuse computed KV cache from previous requests when the same prefix is repeated, dramatically reducing cost and latency. This guide covers Anthropic and OpenAI caching, and how to structure prompts for maximum cache hits.
Processing a prompt from scratch requires computing a key-value (KV) representation for every token. Prompt caching stores this KV cache on the server after the first request. When a subsequent request starts with the same tokens, the provider reuses the cached computation — skipping the expensive prefill step.
Caching is most effective when you have a large, repeated prefix: a lengthy system prompt, a long document being queried multiple times, or a large codebase being analysed. Short or highly variable prompts see little benefit.
Anthropic's API exposes caching via the `cache_control` field on message content blocks. Mark the part of your prompt you want cached: add `{"cache_control": {"type": "ephemeral"}}` to the content block. Anthropic caches the KV state up to and including that block for 5 minutes.
Example — cache a long system document: `messages=[{"role": "user", "content": [{"type": "text", "text": LONG_DOCUMENT, "cache_control": {"type": "ephemeral"}}, {"type": "text", "text": "Summarise the key points."}]}]`. The cached prefix write costs 25% more but reads cost 90% less. Break-even is at 2+ reads of the same prefix.
OpenAI automatically caches prompts longer than 1024 tokens for up to 1 hour. No code changes are required — just ensure your repeating prefix is at least 1024 tokens. Cache hits reduce your input token cost by 50%. Check `response.usage.prompt_tokens_details.cached_tokens` to verify caching is working.
Structure your prompts to put the stable content first: system prompt → static documents → dynamic user input. The cache applies to the longest matching prefix, so the more stable content at the start, the higher the cache hit rate.
The golden rule: put stable content at the top, variable content at the bottom. If you have a 5000-token system prompt followed by a 100-token user query, the 5000-token prefix will be cached across all requests regardless of the query.
For multi-turn conversations, cache the system prompt plus the entire conversation history up to the previous turn. Only the latest user message is dynamic. With a 20-turn conversation, caching the history saves approximately 90% of the input token cost from turn 2 onward.
A practical introduction to the OpenAI API covering authentication, the chat completions endpoint, streaming, error handling, and cost management — with working code in Python and JavaScript.
Read guideThe Anthropic API gives access to the Claude model family. This guide covers authentication, the Messages API, vision inputs, tool use, and the key differences from the OpenAI API format.
Read guide