A Paper Says Repairing AI-Written Tests Raised Their Compile Rate but Left Their Quality Untouched

A test suite can run much of a program and still miss what goes wrong in it. A peer-reviewed paper published in October 2026 measured that gap for unit tests written by large language models.

The paper is Evaluating LLM-Based Test Generation for a Large Industrial C++ Database System, in the proceedings of ESEM 2026, a conference on empirical software engineering. Its authors had four models write tests for one component of a large closed-source database system written in C++. That code is not part of the models' training data, the paper says. They compared it with an open-source key-value store. Parts of the paper follow, not the whole.

How the Tests Scored

The paper scores the tests in several ways. Two of them matter here. Line coverage is the share of source code lines the tests execute. Mutation score is the share of injected faults, called mutants, that the tests detect. Coverage, the paper notes, is known to correlate weakly with how effective a test suite is.

In one approach, each model wrote a whole new test suite for the industrial component. All four fell short of the developer-written baseline, the paper says. Giving the models header files as well as source code improved the results of all four. The setup the paper calls best-performing reached 60.9% line coverage. Its mutation score was 25.2%. A reduced set of the developers' own tests had a mutation score of 30.4%, higher than any whole new suite.

One model, given only source code, reached 47.7% line coverage. Its mutation score was 6.4%. The generated tests ran considerable portions of the code, the paper says, without assertions strong enough to detect injected faults.

Repair Rounds and Placeholder Assertions

When a test file did not compile, a repair loop sent the compiler's errors back to the model, up to ten times. That raised the share of files that compiled.

Looking at the tests that needed the most repair, the researchers found that compiling did not guarantee meaningful test content. In one suite, the model produced an "empty assertion." Every other test in that suite followed the same pattern. The model's own comments said the functions it needed were not available.

In the source-only suites the researchers inspected, three of the four models frequently produced placeholder assertions, the paper says. The fourth took the same shortcut another way. It emitted calls to fabricated APIs, which the compiler rejected.

Repair rounds raised the compile rate several times over and left test quality untouched, the paper concludes. A model that cannot reach the API can still satisfy the compiler with a trivial assertion.

A test like that compiles, runs and passes. Its passing says nothing about the code it was meant to test.

Other Results, Limits and Advice

In the other approach, the models added tests to the reduced developer-written test files. There, three of the four models met or exceeded the reduced baseline on coverage and mutation score. On the open-source store, the paper's abstract says, the generated tests matched the quality of the existing tests.

Results on open-source code, often part of the models' training data, may not carry over to unseen industrial code, it says. But the industrial results cover a single component, four models and one prompt design. The two systems also differ in several ways, including size and code complexity. The authors say they cannot confidently attribute the gap between the two systems to any single factor.

The paper recommends mutation testing, or another measure that is sensitive to behavior, as an extra quality signal for model-generated tests in industrial settings.

If a team judges AI-written tests by coverage alone, what would show that the tests check anything?