A Peer-Reviewed Paper Says SWE-Bench Passed Patches That Did Not Resolve the Issue

SWE-Bench is a widely used benchmark for testing how well coding agents generate code. Each task in it is a real issue from a Python project on GitHub, with test cases written by hand. An agent's patch counts as resolving the issue when it passes all the relevant tests.

A paper in the proceedings of ACL 2025, the annual meeting of the Association for Computational Linguistics, says those tests are often not enough. It is UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench. The paper says the hand-written tests are often insufficient, so patches can pass them without resolving the underlying issue. Its authors built a tool to find such cases. Parts of the paper follow, not the whole.

Tests That Were Too Narrow

The paper gives an example. One task asked that a function that computes polynomial fits for data handle missing data in its two inputs. The original test checked only the case where both inputs had missing data. It did not check the case where only one input did.

The authors added a test where only one input had missing data. Unlike the gold patch that resolves the issue, the paper says, an agent's patch failed to handle that case. It threw an error. So the agent's patch passed the original test, the paper says, but did not resolve the issue.

Hand-written tests in SWE-Bench can be "too narrow" to comprehensively evaluate whether a patch is correct, the paper says. As a result, wrong patches from agents may be counted as resolving the issue.

A Parser That Missed Results

New tests are not enough on their own, the paper says, if the benchmark does not count them properly. SWE-Bench uses a parser built on regular expressions to pull the test cases out of each test log. The paper says the original parser fails to parse many test cases because of defects. One example: it cannot handle a test case whose log runs over more than one line.

The authors fixed those defects in their own version of the parser. With it, they say, they found wrong patches from coding agents that had been labeled as passed.

Two checks stood in a row here: the tests, and the parser that read their results. A patch that did not work could get past either one and still show as passed.

The Tool, Its Findings and Its Limits

Their tool, UTBoost, writes extra test cases with a large language model. It then runs both the gold patch and an agent's patch on the new tests. If the two behave differently, it flags the task as suspicious. Two of the authors, with four years of software testing experience, then reviewed the tests and patches by hand.

The paper reports 36 tasks with insufficient tests. Using the added tests and the fixed parser, it counts 176 wrong patches in SWE-Bench Lite and 169 in SWE-Bench Verified that the original benchmark had evaluated as passing. These corrections changed rankings for 40.9% of entries on the SWE-Bench Lite leaderboard and 24.4% on SWE-Bench Verified's. In SWE-Bench Verified, the paper says, UTBoost found 26 tasks with insufficient tests that a manual review by 93 engineers had missed.

The paper also names its limits. UTBoost can only write tests for tasks that at least one coding agent has resolved, because it compares the gold patch with agents' patches that passed the original tests. And its experiments used only one language model to write the tests.

For anyone who reads a coding agent's benchmark score, the question is what the benchmark's tests could not see.