A Paper in Findings of ACL 2026 Says Test Suites Automatically Generated by LLMs Tend to Be Superficial and Lack Sufficient Discriminative Power

SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering? is a peer-reviewed paper in Findings of the Association for Computational Linguistics: ACL 2026, published in July 2026. Its authors build a benchmark for test suites written by large language models. High-quality test suites have long been hard to obtain, their abstract says, while those "automatically generated by LLMs tend to be superficial and lack sufficient discriminative power." Parts of the paper follow, not the whole.

How the Benchmark Works

The benchmark works by introducing mutated solutions. For each software issue it starts from the repository's correct fix, the paper's "golden solution," and generates mutants: altered versions of that fix, each of which, in the paper's words, "represents an erroneous mutation of the golden solution in the repository and resembles realistic errors." The authors generated them with what they call an agentic, language-agnostic framework.

The paper sets the stakes in the caption to its first figure. "The test suite serves as the verification standard for issue resolution." While trivial mutants are easily detected, the caption says, complex mutants can get past the test, and it says this highlights that high-quality mutants are "indispensable for evaluating whether a test suite is robust enough to prevent incorrect code acceptance."

The benchmark consists of 2,636 mutated variants derived from 800 original instances, and includes a multilingual subset spanning nine programming languages.

Tests That Run, and Tests That Check

The paper scores each generated test suite three ways. The first, Pass@1, asks whether the suite, in a single attempt, is a correctly formatted git diff that can be applied to the repository and executes without triggering compilation errors. The second, the Verified Reproduction Rate, credits a suite only if it fails on the original bug and passes on the fixed golden code. The third, the Relative Detection Rate, is the share of the mutants the original test suite failed to kill that the generated suite kills; where there is no original suite, it is the share of all the mutants killed.

Seven models were tested, under two agent frameworks. "Notably, while most models achieve high Pass@1 scores, their VRR scores remain significantly lower," the paper says. "This discrepancy indicates that while models can easily generate syntactically correct code, constructing logically correct tests that reproduce bugs remains a challenge."

The best-performing model, on the task of repairing an existing test suite under one framework, attains a Verified Reproduction Rate of 59.20% and a Relative Detection Rate of 81.15%. Test generation, the paper says, proves to be significantly more challenging than test repair. Under the other framework, the same model's Verified Reproduction Rate drops from 42.60% on repair to 29.80% on generation.

By the paper's own account, a test suite that applies and runs is the easy part. Whether it catches the bug is where the scores fall.

Earlier Methods, Failures and Limits

The paper also compares its mutants with those of earlier methods. Models scored highest on rule-based mutants, which the paper says "indicates that random syntax errors are trivial to detect." Its own agentic mutation strategy, the abstract says, reduces average detection rates from 71.04% to 39.81% compared to conventional methods. Standard approaches often use simple rule-based operators or few-shot prompting to generate mutants, the introduction says, and earlier research has questioned whether such mutants are truly representative of real-world faults, as they are often easily killed by model-generated test suites. "Thus, existing benchmarks risk overestimating the quality of test suites."

When the authors checked the failed instances by hand, they found that models "tend to focus on local file content and overlook global structures," and that they "frequently confuse relative and absolute import paths or hallucinate non-existent libraries."

The paper names its own limits. Its evaluation assumes that the golden solutions and golden test suites are the absolute ground truth, though it says that even in the highly maintained repositories it draws on, the code may not be exhaustive or entirely free of errors. It also says that using one model family to generate the mutants could introduce a "same-family" bias, and reports a control experiment after which, it says, "the relative rankings are robust" and no additional advantage for that family is statistically detectable.