A Paper in the ESEM 2026 Proceedings Finds No Difference Between Tests Generated by the Same LLM as the Code and Tests Generated by a Different One

A paper titled Don't Bother to Use a Second LLM and Write Tests Yourself! A Study on Elixir was published on October 5, 2026, in the proceedings of the 20th International Symposium on Empirical Software Engineering and Measurement, ESEM 2026, by Schloss Dagstuhl. Its abstract says that using the same LLM to generate both the code and its tests "might make both suffer from the same biases, potentially decreasing the tests' ability to detect faults compared to tests generated by a different LLM." Its research question is whether tests are "better at detecting faults when code and tests are not generated by the same LLM." Parts of the paper follow, not the whole.

The Experiment

The authors generated Elixir code and tests with 5 LLMs and compared them with a baseline of code and tests written by hand. The tasks were 9 specifications "derived from a start-up's software system," in Elixir and its Phoenix web framework, which the paper counts among the "low-resource languages" for which LLMs "tend to perform worse." Including the code and tests written by hand, they worked with 459 distinct implementations and 2709 test suites, and analyzed 9417 failing test cases by hand to tell whether each failure came from a fault in the implementation or a fault in the test.

The generated tests were written white-box: the models had access to the implementations they were testing. The tests written by hand were black-box.

What the Tests Found

The paper's own answer is "No, tests are not better at detecting faults when tests and code are not generated by the same LLM." It found "no significant difference between the implementation errors found by test suites from the same LLM that generated the implementations vs. other LLMs." Changing the model that writes the tests, it says, "might not increase the chances of finding errors." Among its implications, it says that "for now, there is no reason to expect big benefits of cross-model testing strategies," "at least in the context of Elixir."

The manually written tests found many more implementation errors on average than the generated ones. In all, 247 implementations were flagged as faulty, 5 of the 9 written by hand and 242 of the 450 generated. For 99 of those, "only the manual tests revealed implementation errors." Each model's tests alone flagged between 2 and 20 that no other source of tests caught. The paper says the manual tests still identified more faulty implementations than all the generated tests, "despite being written as black-box tests, while the LLMs had access to the implementations." It adds that its results "confirm that code coverage is not the best predictor for a test suite's ability to find errors."

Of the 9417 failing test cases, 5817 failed because of errors in the tests and 3600 because of errors in the implementations, the ones that "successfully detected errors." A single fault could make several test cases fail.

Implications and Limits

Among the paper's implications is "Do not skip writing tests manually, yet." Writing the tests by hand took longer, "(1-2 days)," than generating a single test suite, and the paper says "it might still be worth doing."

On how far its findings reach, it says Elixir and Phoenix likely have "a relatively low representation in current LLM training data," and the 9 specifications "do not represent a comprehensive sample of all possible Elixir programming tasks." It says "the observed gap of human-written tests versus LLM-generated tests is likely larger for Elixir than would be the case for" widely used programming languages like Java or Python. And it says, "Also note that our results are merely a snapshot of LLMs' capabilities at the time of this study (April/May 2025). Newer models might be better suited for low-resource languages such as Elixir and therefore could exhibit higher rates of correct implementations and tests." The prompts were calibrated in a pre-study and not for each specification, which "may lead to an underestimation of the achievable performance of the LLMs." The paper also says its human-written baseline implementations and tests "were written by the first author without independent review," and that a more realistic baseline "should comprise implementations and tests written by multiple humans to provide a more reliable comparison to LLM-generated artifacts." White-box generation "likely allowed the LLMs to generate better tests," it says, but it "also increased the possibility for false passes if the test copied a mistake made by the implementation, thus not finding it." A test that copies the code's mistake passes, and its pass looks like any other.

Among its threats to internal validity, the paper says, "Our analysis method did not allow us to exclude the possibility that implementation errors are present that were found by none of the tests. However, all of these internal issues affected all LLMs similarly. Thus, we still believe that our results allow a fair comparison of the scenarios using the same vs. different LLMs for implementation and test generation." The count of faulty implementations is the faulty code some suite caught. Whatever none of the suites caught is not in it.