LLM API costs can grow unexpectedly as usage scales. This guide covers cost attribution, per-request tracking, anomaly detection, and practical techniques that reduce costs by 50–80% without sacrificing quality.
Every LLM API response includes token usage. Log these alongside the model, feature, and user ID on every call: `{model, input_tokens, output_tokens, cost_usd, feature, user_id, timestamp}`. This gives you cost attribution at any level of granularity — by user, by feature, by model.
Calculate cost from token counts: maintain a pricing table keyed by model. Example: `PRICING = {'gpt-4o': {'input': 2.50, 'output': 10.00}}` (per million tokens). `cost = (input_tokens * PRICING[model]['input'] + output_tokens * PRICING[model]['output']) / 1e6`.
Configure spending alerts at the provider dashboard — this is your last-resort safety net. But also implement application-level alerts: if daily spend exceeds 1.5× the 7-day average, page the on-call engineer. Sudden spikes usually indicate a bug (infinite loop, runaway agent) or unexpectedly viral traffic.
Implement per-user spending limits for consumer applications. Track cumulative daily cost per user. When a user exceeds their limit (e.g. $1/day on a free tier), return an error and prompt upgrade. This prevents any single user from consuming disproportionate resources.
Repeated prompts (same user asking follow-up questions, same system prompt used across requests) are the biggest cost optimisation opportunity. Implement response caching for identical prompts: hash the full prompt and store the response in Redis with a 24-hour TTL.
Enable provider-side prompt caching (Anthropic cache_control, OpenAI automatic caching) to reduce the cost of repeated prefixes. For a system with a 2000-token system prompt sent with every request, prompt caching can reduce input token costs by 80%.
Not every request needs GPT-4o. Classify requests by complexity and route to the appropriate model: use a fast, cheap model (GPT-4o-mini, Claude Haiku, Gemini 2.0 Flash) for simple tasks like extraction, classification, and reformatting; use a capable model only for tasks that genuinely require it (complex reasoning, code generation, nuanced writing).
Implement a model cascade: try the cheap model first, check if the output meets your quality bar (via a fast assertion), and only escalate to the expensive model if needed. For many applications, 70–80% of requests are satisfied by the cheap model, reducing average cost per request by 60–70%.
Audit your system prompts for redundancy. Teams often accumulate instructions over time — 'be helpful, be concise, never make up information, always cite sources, use markdown, use bullet points...' Many can be removed or condensed. Every 100 tokens removed from the system prompt saves cost on every request.
Compress retrieved context in RAG pipelines. Instead of including full document chunks, summarise them first with a cheap model and inject the summaries. A 500-token chunk summarised to 100 tokens is 80% cheaper as context while retaining the key information.
LangSmith is LangChain's observability platform for logging, tracing, and evaluating LLM applications. This guide covers setup, automatic tracing, custom traces, and using the dashboard to debug production issues.
Read guidePromptfoo is an open-source CLI for testing, evaluating, and comparing LLM prompts. This guide covers writing test cases in YAML, running evaluations, comparing models, and catching prompt regressions in CI.
Read guideOpenTelemetry (OTel) is the vendor-neutral standard for distributed tracing. The GenAI semantic conventions extend it to LLM calls. This guide covers setting up OTel tracing for LLM applications, exporting to Jaeger or Grafana, and the GenAI conventions.
Read guide