OpenAI Says 30% of SWE-Bench Pro Coding Tasks Are Broken

OpenAI spent the week defending a number it did not want to explain. On July 8, 2026, the company published an analysis arguing that SWE-Bench Pro, one of the most cited benchmarks for AI coding ability, is far less trustworthy than its leaderboard suggests. After reviewing the benchmark's public tasks, OpenAI concluded that roughly 30 percent of them are faulty, and it withdrew its own earlier endorsement of the test. For a field that ranks models, raises money, and makes purchasing decisions on benchmark scores, that is a big claim to put in writing.

The timing made it bigger. OpenAI released the audit one day before launching GPT-5.6, the same model family that trails Anthropic's best on the exact benchmark it was questioning. That coincidence invites cynicism, and it deserves scrutiny. It also does not, by itself, make the findings wrong. The more useful question is what OpenAI actually found inside SWE-Bench Pro, how it looked, and what a broken benchmark does to the numbers the whole industry leans on.

What SWE-Bench Pro is and why it carried weight

To understand the stakes, start with what the benchmark was built to do. SWE-Bench Pro arrived in September 2025 from Scale AI as a deliberately harder successor to the original SWE-Bench. Where earlier tests had started to saturate, this one aimed at realistic, enterprise-grade work. Its accompanying paper describes 1,865 problems drawn from 41 actively maintained repositories that span business applications, B2B services, and developer tooling.

The design was ambitious. According to Scale AI's leaderboard documentation, the benchmark splits into a public set sourced from 11 repositories, a held-out set of 12, and a commercial set of 18 proprietary repositories backed by partnership agreements with early-stage startups. The tasks are long-horizon by intent, the kind a professional engineer might need hours or days to finish, often touching many files at once. That difficulty is the point. Top models scored around 23 percent on the public set, against 70 percent or more on the easier SWE-Bench Verified, so a high SWE-Bench Pro number carried real signal about a model's engineering ability.

The three-way split was not decorative. One persistent worry with any coding benchmark is contamination, the risk that a model has already seen a repository's issues and fixes in its training data and is recalling answers rather than reasoning them out. By holding back twelve repositories and licensing eighteen private ones from startups, Scale AI tried to keep a portion of the benchmark out of the crawlable web entirely, so a model could not have memorized it. That effort to build a clean, hard test is what earned SWE-Bench Pro its standing, and it is why a finding that a third of the public tasks are miscalibrated stings more than it would for a throwaway leaderboard.

That reputation is exactly why OpenAI's audit lands hard. A benchmark nobody trusts can be ignored. A benchmark everybody cites, one that shapes how labs describe their coding models, cannot. If a meaningful share of its tasks are miscounting right answers as wrong, then every ranking built on it inherits that error.

How OpenAI audited SWE-Bench Pro

The strength of OpenAI's claim rests on how it was reached, so the methodology matters more than the headline figure. The Decoder laid out the process, and it was not a single automated pass.

OpenAI began with an automated screening tool that flagged 286 tasks as suspicious. Each of those cases then went through a layered review. AI agents built on Codex examined the flagged tasks in detail, human researchers made the final determinations, and a separate group of five experienced developers independently evaluated the same cases as a check on the machine-assisted verdicts. The point of that structure was to avoid trusting any one reviewer, human or model, on a judgment this consequential.

The screening step deserves a second look, because it shapes how the final number should be read. An automated tool flagging 286 tasks does not mean 286 are broken. It means those cases showed a signature worth a closer look, such as a test that failed for reasons unrelated to the code or a description that did not match the checks. The human and machine reviewers then confirmed a subset of those as genuinely defective. That funnel, from a wide automated net down to a confirmed count, is the difference between a scan that cries wolf and one that produces a defensible figure. It also means the true number could be higher, since the audit only examined tasks the screen surfaced, not every task in the set.

The two review tracks did not agree perfectly, which is itself informative. The Codex-based agents identified 200 tasks as genuinely flawed, about 27.4 percent of the set. The human developers were harsher, flagging 249 tasks, or 34.1 percent. The two groups reached the same conclusion on 74 percent of cases. That spread, from 27 percent to 34 percent, is where the "roughly 30 percent" figure comes from, and the gap between machine and human judgment is a reminder that even the audit of a benchmark involves subjective calls.

The four ways SWE-Bench Pro tasks fail

OpenAI sorted the defective tasks into four recurring failure modes. Each describes a different way a benchmark can punish a correct answer or reward an incomplete one.

1. Overly strict tests that reject a working solution because it does not match an arbitrary expected form.
2. Vague requirements where the real expectations are hidden inside the test cases rather than stated in the task.
3. Shallow tests that let an incomplete or wrong solution pass because the checks do not probe deeply enough.
4. Misleading descriptions that point a model toward the wrong approach entirely.

The clearest illustration is a task drawn from the OpenLibrary project. The written instruction asked for a single space, while the hidden test quietly expected two. As The Decoder put it, an AI that correctly followed the instructions would fail. The model did the right thing and was marked wrong, which is the precise inversion a benchmark is supposed to prevent. Multiply that across hundreds of tasks and the leaderboard stops measuring coding skill and starts measuring how well a model guesses at undocumented test quirks.

The misleading-description category is the most damning of the four, because it inverts what a benchmark is for. A good test rewards a model for reading a specification carefully and doing what it says. A task whose written description points one way while its hidden test expects another rewards the opposite behavior, favoring a model that ignores the instructions and pattern-matches to what the checker probably wants. On a long-horizon agentic benchmark, where the model is meant to plan and follow through over many steps, that inversion is especially corrosive. It quietly trains the leaderboard to prefer models that second-guess the prompt over models that trust it, which is the wrong lesson to encode into the numbers everyone is optimizing against.

Two of these categories, the shallow tests and the overly strict ones, pull scores in opposite directions. Shallow tests inflate results by passing bad solutions. Strict tests deflate them by failing good ones. That matters because it means the errors do not simply cancel into a harmless wash. A model that happens to write in the style a strict test expects gains an edge that has nothing to do with being a better engineer, and a model penalized by a mislabeled task loses ground it earned.

Why a faulty benchmark distorts the leaderboard

Benchmarks in this field have a short shelf life, and that context explains why noise is so damaging. The Decoder notes that on one public SWE-Bench set of 731 tasks, top models climbed from 23.3 percent to 80.3 percent accuracy in eight months. When a test saturates that fast, the interesting differences between models compress into a narrow band near the top, and small amounts of measurement error can reorder the ranking. Eight months is roughly the lifespan of a single model generation, which means a benchmark can go from genuinely hard to nearly solved inside one product cycle, and the window in which its scores carry clean signal is shorter than the marketing that quotes them.

SWE-Bench Pro was built precisely to reopen that gap, to be hard enough that scores in the low tens of percent again meant something. If a third of its tasks are broken, that carefully engineered difficulty is contaminated. A few points of separation between two frontier models could reflect real capability, or it could reflect which model's habits happened to align with a set of mislabeled tests. Buyers reading a leaderboard have no way to tell the difference, and neither do the labs writing the marketing copy around those scores.

This is where the GPT-5.6 timing becomes relevant rather than merely suspicious. As covered in our breakdown of the GPT-5.6 launch, OpenAI's flagship Sol model scored 64.6 percent on SWE-Bench Pro against Claude Fable 5's 80 percent, a wide gap on the one benchmark OpenAI was busy discrediting the day before. A skeptic reads that as motivated reasoning. A charitable reader notes that a benchmark can be genuinely flawed and also inconvenient for the party pointing out the flaws. Both can be true at once, and the audit's methodology, with human developers finding more problems than the models did, is the reason the finding survives the obvious conflict of interest.

Pulling an endorsement and asking for better

OpenAI did not stop at criticism. It withdrew its prior endorsement of SWE-Bench Pro and urged the industry to build new benchmarks with the help of experienced developers, tests that are hard to game and that measure something genuinely meaningful. That is a notable stance from a company whose own models are evaluated on these very tests, because it argues against the convenience of a stable, well-known scoreboard in favor of the harder work of building better ones.

There is a tension worth naming here. A lab calling for new benchmarks right after posting a weak score on an old one has an obvious incentive to reset the board. But the underlying request is sound regardless of who makes it. If a widely trusted benchmark mislabels a third of its tasks, the responsible move is to say so and to invest in replacements, not to keep quoting a number everyone knows is noisy. The credibility of the call depends less on OpenAI's motives than on whether independent parties can reproduce the audit, and the layered methodology at least makes that possible.

The deeper problem the audit exposes

Zoom out and this is not really a story about one test. It is a story about how the AI field measures progress at all. Coding benchmarks have become the currency of model marketing, quoted in launch posts and investor decks as if they were physical constants. The SWE-Bench Pro audit is a reminder that they are human-built artifacts, assembled quickly, and as fallible as any large dataset scraped from real repositories.

The failure modes OpenAI found are not exotic. Hidden expectations, brittle assertions, and under-specified requirements are the everyday hazards of writing test suites, and a benchmark with nearly two thousand tasks was always going to accumulate them. The mistake was treating the resulting scores as precise to the decimal point. A benchmark can be a useful compass while still being wrong often enough that a two-point difference means nothing. The industry has been reading these compasses as if they were calibrated instruments.

Faulty tasks are also only one of two ways a benchmark can lie. The other is contamination, the memorization risk that Scale AI designed its private repositories to blunt. A model can post a strong score not because it reasoned well but because it had seen the answer during training, and that failure is invisible on a public leaderboard. The two problems compound. A benchmark can simultaneously mislabel correct answers as wrong and reward memorized answers as right, and a single top-line percentage hides both. This is why researchers who take evaluation seriously keep pushing toward held-out and freshly sourced tasks, the harder and less glamorous half of the work.

What good would look like is not a mystery. Benchmarks reviewed by working engineers, with tests that check behavior rather than surface form, refreshed before they saturate, and reported with honest error bars instead of false precision. That is slower and more expensive than scraping issues from GitHub and calling it a leaderboard, which is why it has not happened at scale. OpenAI's audit, whatever its motives, at least puts a real number on the cost of not doing it.

What the SWE-Bench Pro audit changes going forward

The practical takeaway is modest and important at the same time. SWE-Bench Pro is not worthless, and nobody serious is claiming it is. Roughly 70 percent of its tasks still work, and a model that does well across that portion is genuinely capable. What changes is how much weight a single SWE-Bench Pro number can bear. A ten-point lead probably reflects a real difference in skill. A two-point lead now has to be treated as possible noise until the benchmark is cleaned up.

For anyone choosing a coding model, the lesson is to stop treating any one leaderboard as the answer. Aggregate several benchmarks, weight them against the tasks you actually run, and discount narrow margins. For the labs, the audit raises the bar on what counts as a credible claim, because a rival can now respond to any benchmark boast by asking how many of those tasks were even valid. And for Scale AI, the maker of SWE-Bench Pro, the report is both a critique and an opportunity to harden a test that clearly filled a real need. A benchmark that gets audited, corrected, and republished is more trustworthy afterward than one that never faced scrutiny, so the healthiest outcome is not that SWE-Bench Pro disappears but that its next version ships with the flagged tasks fixed and a clearer account of how they were validated. The uncomfortable truth underneath all of it is that the field has been keeping score with rulers it never fully checked, and it took a launch-week conflict of interest to force the measurement of the measurements.