Fastest LLMs: Latency Benchmarks Compared

Key Takeaways

Why Speed Matters

For conversational AI, voice interfaces, and real-time coding assistants, latency is not a nice-to-have. It is a core product requirement. Users start feeling friction when responses take more than one to two seconds to begin, and abandon workflows when generation is slow enough to disrupt their thinking.

Two metrics matter: Time to First Token (TTFT), which measures how long until text starts appearing, and tokens per second (TPS), which measures how fast text streams after it begins. TTFT dominates perceived responsiveness. TPS determines how long users wait for long responses.

Speed Rankings

Groq leads on raw speed for smaller models, achieving 500 to 800 tokens per second on Llama 3.1 70B using custom LPU hardware. This is five to ten times faster than GPU inference. For small models where speed is everything, Groq's API is unmatched. Cerebras offers similar performance on Llama variants.

Among frontier model providers, Gemini 2.5 Flash is the fastest large capable model, typically delivering under 500ms TTFT and 100-plus TPS. Claude 3.5 Haiku and GPT-4o Mini are close competitors. Claude Opus 4 and GPT-5 are the slowest frontier models due to their size.

Streaming vs. Non-Streaming

All major providers support streaming responses, which dramatically improves perceived responsiveness. With streaming, users see tokens as they are generated rather than waiting for the complete response. For conversational UIs, always enable streaming. It reduces perceived latency by three to five times even when total generation time is identical.

Reasoning models have inherently higher TTFT because they think before responding. Users see no output during the thinking phase, making them feel slower than standard models. Some providers expose thinking progress indicators to mitigate this.

Speed Recommendations

Voice AI: Use Groq with Llama 3.3 70B or Gemini 2.5 Flash for the lowest latency. Real-time coding autocomplete: GPT-4o Mini or Gemini 2.5 Flash Lite. Customer support chatbot: Claude 3.5 Haiku or GPT-4o Mini, both offering sub-second TTFT. Long-form generation where speed matters less: Claude Sonnet or GPT-4o are fine.

Test latency from your deployment region, as infrastructure geography significantly affects TTFT. A model that is fastest in the US may not be fastest for European or Asian users. Most providers offer regional endpoints or have global infrastructure.