Benchmark
A standardized test or set of tasks used to evaluate and compare the capabilities of different AI models on a common scale.
AI benchmarks are curated datasets and evaluation protocols designed to measure specific capabilities of language models. They allow researchers, developers, and the public to make meaningful comparisons between models from different companies and generations. Common benchmarks include MMLU (broad knowledge), HumanEval (coding), and GPQA (PhD-level reasoning).
A good benchmark should be challenging enough to differentiate strong from weak models, diverse enough to cover a range of real-world tasks, and resistant to 'contamination' — the risk that test questions appear in training data. Benchmark contamination is a real concern, as models trained on web-scale data may have seen benchmark questions during training.
No single benchmark tells the full story. Real-world performance often diverges from benchmark scores, which is why the community relies on a diverse ecosystem of evaluations. Human preference rankings on platforms like LMSYS Chatbot Arena complement automated benchmarks by capturing qualities like helpfulness and naturalness that are hard to quantify.
Read next
MMLU: The Massive Multitask Language Understanding Benchmark
What MMLU measures, how it's constructed, why it became the standard LLM benchmark, what top model scores reveal, and when to use MMLU-Pro instead.
HumanEval: OpenAI's Python Coding Benchmark Explained
How HumanEval measures LLM coding ability, what pass@k means, which models top the leaderboard, why it's now saturated, and what to use instead for real-world coding evaluation.
GPQA: The Graduate-Level Benchmark That Still Challenges AI
What GPQA Diamond measures, how PhD-level questions are constructed to be Google-proof, why reasoning models dominate the leaderboard, and what scores above the human expert baseline really mean.