GPQA: The Graduate-Level Benchmark That Still Challenges AI

What Makes GPQA Hard

GPQA (Graduate-Level Google-Proof Q&A) consists of multiple-choice questions written by domain experts, graduate students and PhD holders, in biology, chemistry, and physics. The defining design principle is that questions must be 'Google-proof': answering correctly requires deep conceptual understanding of the domain, not the ability to find and quote a relevant webpage. Non-expert humans with unrestricted internet access score only 34% on GPQA Diamond, barely above chance on a 4-choice format. This makes GPQA one of the few benchmarks where internet access provides almost no advantage, the bottleneck is genuine comprehension.

Unlike MMLU, which a well-trained Foundation Model can attack through pattern matching to textbook content, GPQA Diamond requires multi-step reasoning, synthesizing concepts across sub-disciplines, and catching subtle errors embedded in plausible-looking wrong answers. The questions are deliberately designed so that plausible-sounding distractors can fool domain-adjacent experts. Getting to the right answer consistently requires working through the problem from first principles.

How GPQA Was Built

GPQA was introduced by David Rein, Asa Cooper Stickland, and colleagues from various research institutions, published in 2023. Question authors were required to be PhD-level experts in the specific sub-discipline they wrote about, not just domain-adjacent. Each question went through a rigorous validation process: the author confirmed the correct answer, then separate PhD reviewers in the same field independently verified correctness and confirmed that non-experts consistently got the question wrong.

This validation pipeline is what justifies the benchmark's claim to difficulty. A question that an author found hard but that a colleague immediately solved would be filtered out. Questions that were ambiguous, that had multiple defensible answers, or that could be answered by textbook recall alone were eliminated. The result is a dataset that is small (roughly 450 questions in the full set, 198 in the Diamond subset) but extremely high-signal.

The three domains, biology, chemistry, and physics, were chosen because they support complex, multi-step reasoning problems that resist surface-level pattern matching. Questions span quantum mechanics, organic reaction mechanisms, protein folding, molecular biology, thermodynamics, electromagnetism, and related frontier sub-disciplines. The coverage is intentionally deep rather than broad.

The Diamond Subset: The Hardest Tier

GPQA Diamond is the 198-question subset of the full benchmark that passed the strictest validation criteria. For a question to qualify for Diamond, it must be: confirmed correct by the author, independently verified by PhD reviewers, and consistently incorrect for non-domain-expert participants even with internet access. Human experts (PhD holders in the exact relevant sub-discipline) score approximately 65-80% on Diamond, there's genuine difficulty even for specialists.

There is also a 'Main' set (448 questions) and an 'Extended' set. Diamond is the standard for frontier model evaluation because the full set includes some questions that turned out to be more accessible than intended. When researchers and labs report 'GPQA scores,' they almost always mean GPQA Diamond. If a report says GPQA without specifying Diamond, treat the score with additional skepticism, the difficulty tiers are meaningfully different.

Why GPQA Matters for AI Evaluation

GPQA represents a qualitative shift in what we demand of AI benchmarks. MMLU questions are often answerable by pattern-matching to common textbook content that appears verbatim in pretraining data. GPQA questions require multi-step causal reasoning, applying principles across contexts, and identifying why plausible wrong answers fail, skills that require something beyond memorization.

Performance on GPQA Diamond correlates meaningfully with performance on genuine scientific tasks: understanding complex research papers, catching methodological errors, reasoning through novel experimental designs, and generating testable hypotheses. As AI systems are increasingly considered for research acceleration roles, literature synthesis, experimental design, data analysis, GPQA provides one of the few available early signals of readiness for such work. A model that can't reliably solve GPQA Diamond problems is unlikely to be a useful research collaborator on frontier science.

Why Reasoning Models Dominate GPQA

The performance gap between reasoning models and standard instruction-tuned models on GPQA Diamond is the largest of any major benchmark. Standard models like GPT-4 and Claude 3 Opus cluster around 50-65%, meaningfully below the human expert baseline. Reasoning models like o3 and o4-mini reach 80-90%, meaningfully above it. This gap is entirely attributable to extended chain-of-thought reasoning during inference.

GPQA problems often require 5-15 reasoning steps, with each step contingent on the previous. Standard models fail because they don't have enough inference-time compute to work through this reasoning chain reliably. Reasoning models are trained to produce extended internal reasoning before answering, effectively allocating more computation to hard problems. This is why GPQA Diamond has become one of the most important benchmarks for evaluating reasoning model capability specifically.

The practical implication: for applications requiring scientific reasoning, medical diagnosis support, research literature synthesis, chemical reaction prediction, reasoning models outperform standard models by a margin much larger than MMLU scores would suggest. GPQA Diamond is the right benchmark to consult for this class of application.

Current Scores and What They Mean

As of mid-2025 on GPQA Diamond: OpenAI's o3 achieves 87.7%, meaningfully above the human expert baseline of 65-80%. This is a striking result: o3 can reason through graduate-level scientific problems more reliably than domain PhD holders. o4-mini scores 79.6%, placing it within the human expert range. GPT-5 approximately 74%, Claude Opus 4 around 69%, Claude Sonnet 4.6 around 61%.

The historical progression makes the reasoning model inflection point visible: GPT-4 (53.6%) and GPT-4o (53.6%) were both below the human expert floor. o1-preview jumped to 73.3%. o3 reached 87.7%. The jump from non-reasoning to reasoning models is larger on GPQA Diamond than on almost any other benchmark, confirming that GPQA is specifically testing the capabilities that reasoning model training addresses.

What Comes After GPQA

As o3 approaches or exceeds expert performance on GPQA Diamond, the AI benchmark community is developing harder evaluations. FrontierMath contains research-level mathematics problems that even expert mathematicians find challenging, frontier models currently score under 10%. Domain-specific evaluations in synthetic biology, materials science, and drug discovery are being developed to probe whether AI can assist at the frontier of discovery rather than just at graduate curriculum level.

The broader question GPQA raises is practical rather than philosophical: if AI models can reliably answer graduate-level scientific questions better than domain experts, how should scientific workflows change? GPQA is transitioning from being a test of AI limitations to being an indicator of AI research potential. Organizations building AI tools for scientific applications should treat GPQA Diamond performance as the minimum necessary (not sufficient) qualification for their use case.