SWE-Bench
A benchmark that measures an AI agent's ability to resolve real GitHub issues from open-source repositories — a rigorous test of practical software engineering capability.
SWE-Bench (Software Engineering Benchmark) was introduced by Princeton NLP in 2024. It consists of 2,294 real GitHub issues from popular Python open-source repositories like Django, scikit-learn, and Flask. Each issue comes with a codebase snapshot, the issue description, and a test suite that validates the correct fix. An AI agent must understand the issue, locate the relevant code, implement a fix, and pass the tests.
SWE-Bench is widely considered the most practically meaningful benchmark for coding AI because it tests the full software engineering workflow — not just generating code from scratch, but understanding existing codebases, reproducing bugs, and writing targeted fixes. Performance on SWE-Bench correlates more closely with real-world engineering utility than benchmarks like HumanEval.
The SWE-Bench Verified subset (500 human-validated problems) is used most often for direct model comparisons. Scores have improved dramatically: early agents scored under 5%, while frontier agents in 2025 exceed 50% on the verified set. Aider, Devin, and Claude Code are among the tools evaluated on SWE-Bench.
Read next
Devin: The Autonomous AI Software Engineer
Devin by Cognition is the most ambitious AI coding agent — an autonomous software engineer that can browse the web, use tools, and complete multi-hour engineering tasks independently.
Aider: AI Pair Programming in Your Terminal
Aider is an open-source AI coding assistant that runs in your terminal, edits your local files, and commits changes to git. The power tool for developers who live in the command line.
Cursor: The AI-First Code Editor Review
A deep-dive into Cursor — the VS Code fork that put AI at the center of the editing experience. Who it's for, what makes it different, and whether it's worth the switch.