How to handle rate limits and retries in LLM applications

LLM APIs rate limit on two dimensions: requests per minute (RPM) and tokens per minute (TPM). RPM limits cap the number of API calls; TPM limits cap the total input + output tokens. Exceeding either returns a 429 Too Many Requests error. TPM limits are more commonly hit than RPM limits in production.

Rate limits vary enormously by provider tier. OpenAI's tier 1 (newly funded accounts) has very low limits; tier 5 (high-spend accounts) has limits orders of magnitude higher. Check your current limits at platform.openai.com/account/limits. Plan your architecture around your current tier, not your expected final tier.

Never retry immediately after a 429. Use exponential backoff with jitter: wait 1 second after the first failure, 2 after the second, 4 after the third, up to a maximum of 60 seconds. Add random jitter (±20%) to prevent all clients from retrying simultaneously after an outage.

Python implementation: `import time, random; def call_with_backoff(fn, max_retries=5): for i in range(max_retries): try: return fn(); except RateLimitError: if i == max_retries - 1: raise; wait = (2**i) + random.uniform(0, 1); time.sleep(wait)`. The `tenacity` library provides a production-quality decorator for this.

Track token usage per request and maintain a rolling counter against your TPM limit. The OpenAI response includes `usage.total_tokens` — accumulate this and throttle requests when approaching the limit.

Set `max_tokens` on every request. Without it, a model might generate 4096 tokens when 200 would suffice, burning through your token budget. For most user-facing applications, 500–1000 max output tokens is appropriate. For coding tasks, 2000–4000 is reasonable.

In applications with many concurrent users, implement a request queue with rate limiting. Libraries like `asyncio-throttle` (Python) or `p-limit` (Node.js) limit concurrent API calls. Process the queue at your safe request rate, rejecting or queuing excess requests with appropriate user feedback.

For background processing jobs (batch summarisation, nightly analysis), use the OpenAI Batch API which processes requests asynchronously at half the price with a 24-hour completion window. Submit a JSONL file of requests and poll for completion.

The most robust production setup routes requests across multiple providers. If OpenAI returns a 429, retry with Anthropic; if Anthropic is slow, fall back to Gemini. OpenRouter handles this automatically — set multiple fallback models in your request: `{"models": ["openai/gpt-4o", "anthropic/claude-sonnet-4-5"], "route": "fallback"}`.

Cache responses for identical prompts. Many production applications repeat the same or similar prompts — caching saves both cost and latency. Use a semantic cache (hash the embedding of the prompt) to catch near-duplicate requests. Redis or PostgreSQL with pgvector work well for this.