SWE-bench Verified: How AI Coding Agents Are Graded

When a lab claims its new model "resolves 70% of real software issues," the number almost always comes from one place: SWE-bench Verified. It has become the default yardstick for whether an AI system can act like a working software engineer rather than just autocomplete code. Understanding how SWE-bench Verified is built, what a single task actually demands, and where its scores quietly mislead is the difference between reading a benchmark headline and understanding it.

The benchmark traces back to SWE-bench, introduced by Princeton researchers in a paper first posted in late 2023 and published at ICLR 2024, then refined into the Verified subset by OpenAI in August 2024. This explainer walks through the full picture using the primary sources: OpenAI's announcement, the evaluation notes published by Epoch AI, and the benchmark's own project site.

What SWE-bench Verified measures

The benchmark asks a narrow but demanding question: given a genuine bug report or feature request from a real open-source project, can an AI system change the codebase so the problem is actually fixed? This is not a multiple-choice quiz or a toy coding puzzle. The tasks are mined from the issue trackers of established Python libraries, and the "answer key" is the set of tests the project's own maintainers wrote to verify a fix.

That framing is what makes the benchmark respected. A model cannot pattern-match its way to a high score the way it might on a trivia set, because the only thing that counts is whether the edited repository passes its tests. As the project site describes it, a system is handed a code repository and an issue description, then must investigate the code and produce a patch that resolves the issue. The grading is mechanical and unforgiving: run the tests, see if they pass.

How a single SWE-bench Verified task works

The mechanics reward a close look, because they explain both the benchmark's strength and its blind spots. Each task instance bundles a real issue from one of the 12 Python repositories together with the state of that repository at the moment the issue was open. The model receives the issue text and access to the full codebase, and its job is to output a diff, a code patch, that it believes resolves the problem.

Evaluation then applies that generated patch to the repository and runs two sets of unit tests written by the project's actual developers. The first set, often called the fail-to-pass tests, are the ones that failed before the fix and must pass after it; they prove the issue was genuinely resolved. The second set, the pass-to-pass tests, were already passing and must keep passing; they prove the patch did not break existing behavior. A task counts as resolved only when both conditions hold. That dual check is deliberate. It stops a model from "fixing" the reported bug while silently regressing something else, which is exactly the failure mode that makes real maintainers nervous.

This design is why a SWE-bench result feels closer to real engineering than older code benchmarks. The model has to read unfamiliar code, localize the relevant section across many files, reason about the intended behavior, then write a change that survives a real test suite. Those are the same steps a human contributor takes when opening a pull request against a project they did not write.

Why the original SWE-bench needed fixing

SWE-bench arrived as a landmark, but practitioners quickly noticed that its raw scores understated model ability for reasons that had nothing to do with the models. The original set contained 2,294 tasks, and a meaningful fraction of them were flawed in ways that made success impossible or unfair.

OpenAI's announcement laid out the two biggest problems plainly. Some issue descriptions were underspecified, meaning the text did not contain enough information for anyone, human or machine, to know what the correct fix should be. Others shipped with unit tests that were too strict or too narrow, rejecting valid solutions because they checked for one exact implementation rather than the behavior the issue asked for. On top of that, some task environments simply did not run reliably, so a perfectly good patch could be marked wrong because the harness failed to execute. The net effect was a benchmark that punished models for the dataset's defects and produced scores nobody fully trusted.

The 93-developer annotation that produced 500 tasks

To clean this up, OpenAI ran a large human-validation campaign rather than trusting automation. The company worked with 93 software developers experienced in Python and had them manually screen samples from the SWE-bench test set for quality. They annotated 1,699 random samples, examining whether each issue was well specified and whether its tests were a fair check on a correct solution.

The output of that screening was SWE-bench Verified, a curated set of 500 samples that passed muster. Each retained task had a problem statement clear enough to solve and a test patch that actually validated the fix rather than over-fitting to one implementation. Epoch AI's writeup adds useful texture here: in the broader SWE-bench annotation effort, samples were reviewed by multiple separate annotators to validate the issue specification and the unit tests, and to confirm the task was free of other defects that could make grading unreliable.

The payoff was immediate and measurable. On SWE-bench Verified, GPT-4o (2024) resolved 33.2% of tasks using the best open-source scaffold of the time, Agentless, roughly double the 16% it had scored on the original SWE-bench under the same approach. The models had not changed. The measurement had simply stopped penalizing them for broken tasks, and the more honest number was about twice as high.

The repositories behind the tasks

The tasks are not drawn from toy projects. SWE-bench pulls its issues from 12 mature, heavily used Python libraries: Django, scikit-learn, SymPy, Matplotlib, Astropy, pytest, Sphinx, Flask, Requests, Seaborn, Xarray and Pylint. These are the kind of packages that sit underneath production systems and scientific work worldwide, which is part of why a fix that survives their test suites carries weight.

The distribution across those repositories is lopsided, and that matters for interpreting results. In the original 2,294-task SWE-bench, Django alone contributed roughly 850 tasks and SymPy another 386, while projects like Flask (11) and Seaborn (22) contributed only a handful, according to the per-repository breakdown documented alongside the SWE-bench paper. A model that happens to be strong on Django's web-framework idioms or SymPy's symbolic-math internals can therefore look better overall than one with more balanced skills. The Verified subset inherits a version of this skew, so two systems with the same headline score may have different strengths hiding underneath it.

The practical takeaway is that the benchmark rewards genuine code comprehension across large, real codebases. Many of these projects span thousands of files, and the relevant fix for an issue can sit far from where the symptom appears. Localizing the right lines before writing a patch is often the hardest part of a task, and it is a skill that does not show up in benchmarks built from self-contained snippets.

What the scores actually say, and what they hide

Here is the part most benchmark headlines skip. A SWE-bench Verified percentage does not measure a model in isolation. It measures a model wrapped in an agent scaffold, the surrounding loop that lets the system read files, run shell commands, edit code, then decide when it is done. Epoch AI is explicit that its evaluation uses a scaffold giving the model bash access, a text editor tool, and patch-application tooling, with the model taking actions and receiving output across a loop.

That dependence is enormous in practice. The same underlying model can post very different SWE-bench scores depending on how clever its scaffold is at navigating the repository, retrying after failed edits, and managing its context. When you read that a system "scores 70% on SWE-bench Verified," the right mental translation is that a particular model-plus-harness combination scores 70%, and swapping either piece can move the number substantially. Epoch even notes that its scaffold underwent a major upgrade to version 2.0.0 in February 2026, the kind of change that shifts scores without any new model at all.

How far the scores have climbed

The trajectory since 2024 shows both real progress and the scaffold effect at work. GPT-4o's 33.2% in August 2024 was a strong result at the time. By February 2026, an independent leaderboard maintained outside the labs had top systems clustered far higher. As Simon Willison reported, Claude Opus 4.5 led at 76.8% resolved, with Gemini 3 Flash and MiniMax M2.5 close behind at 75.8% and Claude Opus 4.6 at 75.6%. In roughly eighteen months the frontier of resolved tasks had more than doubled.

Two details from that snapshot are worth holding onto. First, the evaluation Willison cites uses the same system prompt for every model, which makes it a cleaner comparison than launch-day numbers that labs produce with their own tuned harnesses. He stresses the results carry weight precisely because they were not self-reported. Second, the ranking was crowded with models from several labs, with Chinese systems such as GLM-5 and Kimi K2.5 placing in the top ten, a sign that high SWE-bench performance is no longer the property of a single company. The gaps at the top were small enough that Opus 4.5 edged out the newer Opus 4.6 by about a percentage point, which is well inside the noise the benchmark carries.

That last point loops back to the central caution. When several systems sit within a point or two of each other near 76%, the ordering says more about scaffolds, prompts, and run-to-run variance than about any decisive capability gap. The honest reading of the modern leaderboard is that a cluster of frontier systems can now resolve roughly three-quarters of carefully validated real-world issues under a fair harness, which is genuinely impressive, while the exact ranking within that cluster is far less meaningful than the headline order suggests.

The limits of a single leaderboard number

Even with human curation, SWE-bench Verified is not a perfect oracle, and treating it as one leads people astray. Epoch AI runs its evaluation on 484 of the 500 samples, excluding 16 that do not run reliably in its infrastructure, and estimates a residual error rate of roughly 5 to 10% even after the validation pass. A handful of tasks remain ambiguous enough that a reasonable fix can still be graded wrong.

There is also the test-modification loophole. Because grading runs unit tests, a model that edits the tests themselves, or refactors code in a way that renames methods the tests depend on, can fail even when its logic is sound, or pass for the wrong reasons. Serious evaluations counter this by instructing the system not to touch the tests, but it remains a reminder that a mechanical grader rewards what it can check, not what is genuinely correct.

The deeper worry is contamination. Because the benchmark is built from public GitHub issues in popular libraries, the fixes for many of those issues exist somewhere in the training data of any large model trained on the open web. A model that has effectively seen the answer is not solving the task so much as recalling it, and there is no clean way to fully separate the two from the outside. This does not make the benchmark worthless, but it is why careful readers treat a high score as evidence of capability rather than proof of it, and why fresh, held-out variants keep getting proposed.

A family of benchmarks, not one

SWE-bench Verified is the most cited member of what has become a small family of related benchmarks, and confusing them is a common mistake. The original SWE-bench is the full 2,294-task set. SWE-bench Lite is a smaller, cheaper subset designed so researchers can iterate without paying to run the entire benchmark on every experiment. Beyond those, the maintainers and others have published multimodal and multilingual variants that push past the Python-only, text-only scope of the original, as the variant list on Epoch AI's leaderboard reflects.

Each variant exists to probe a different weakness. Lite trades coverage for speed and budget. The multilingual editions test whether a system that shines on Python can transfer to other languages, where idioms and tooling differ sharply. The multimodal versions fold in tasks where understanding a screenshot or a rendered output matters, closer to front-end work than backend library fixes. When a result cites SWE-bench without saying which member of the family it used, that omission alone should make you read the claim more skeptically, because the difficulty and the achievable scores vary widely between them.

Reading a SWE-bench result like an engineer

SWE-bench Verified earned its status because it grades the right thing: real fixes to real code, checked by real tests, on a dataset that humans cleaned up so the scores reflect models instead of broken tasks. That is a genuine improvement over the trivia-style coding benchmarks that came before, and it is why nearly every frontier lab now reports it. The 500-task set is the most credible single number we have for autonomous software engineering, and its influence on how labs frame progress is hard to overstate. Almost every model launch since 2024 has reached for it as proof that a system can do real engineering work rather than merely talk about code.

The mature way to use it is with the asterisks attached. A score is a property of a model and its scaffold together. It carries a few points of irreducible noise, and it cannot fully rule out that the model studied for the test. When you next see a launch post leading with a SWE-bench Verified figure, the useful questions are which scaffold produced it, whether the comparison models ran under the same harness, plus how much of the gap is real capability versus a better agent loop. Asked that way, the benchmark becomes what it was meant to be: a sharper lens on progress rather than a marketing trophy. For more on how these evaluations shape the models you actually use, our blog keeps tracking the field.