Faults That No Existing Test Can Catch: Meta's Paper on Mutation-Guided Test Generation, January 2025

On January 22, 2025, Meta posted to arXiv a paper on ACH, a tool it built and deployed to generate unit tests. The paper carries the reference of the companion proceedings of the ACM conference on the Foundations of Software Engineering of 2025, and reports on the deployment of ACH at Meta between October 28 and December 31, 2024. Its focus is privacy, but it says the approach "can be applied to any issue and is not confined solely to tackling privacy issues." Parts of the paper follow, not the whole.

Faults the Existing Tests Did Not Catch

The paper builds on mutation testing, and calls the mutants it makes simulated faults. Compared with traditional mutation testing, ACH generates relatively few of them. It focuses instead on faults that are currently undetected and specific to an issue of concern, and from those it generates tests that can catch them.

The paper calls this its key insight: to construct mutants that are relevant to the issue and "also currently not caught (unkilled) by any existing test case", and to use them as prompts for test generation with a large language model.

What ACH Guarantees, and What It Aims For

The paper lists five assurances for the tests ACH proposes. They build. They pass, consistently. They catch faults that no existing test can catch. Many are closely coupled to the issue of concern. Most respect the coding style of the existing tests for the class under test.

Only the first three are guarantees. The paper calls them "unequivocal guarantees made by ACH about the tests it proposes to the engineer." The fourth and fifth are "more aspirational and probabilistic in nature", and are best assessed, it says, through the ordinary code review that engineers already do.

What Line Coverage Would Have Thrown Away

The paper notes that a new test need not add coverage: it "may find faults simply by covering existing lines of code in some new way; the same path with different data, for example." ACH was applied to 10,795 Android Kotlin classes in 7 software platforms deployed by Meta, and generated 571 privacy-hardening test cases. Of those 571 tests, the paper says, 277 "would have been discarded had we chosen to focus solely on the line coverage test adequacy criterion".

How the Engineers Received Them

ACH was used in test-a-thons for Messenger and WhatsApp in December 2024, where engineers reviewed its tests. The overall acceptance rate, the paper says, was 73%, and only approximately 36% of the tests were deemed either possibly or definitely related to privacy.

From that, the paper observes that "the engineers are very willing to accept tests which may not be relevant to their current concern, when they are found useful for other reasons."