TIGTA Said in May 2025 That the IRS Had Not Established Processes to Evaluate Whether Its AI Models for Selecting Returns Perform Better Than Prior Methods

The Treasury Inspector General for Tax Administration, TIGTA, issued report 2025-308-022 on May 19, 2025. Its objective was to determine the effectiveness of the AI models that two divisions of the Internal Revenue Service, Large Business and International and Small Business/Self-Employed, use in selecting tax returns and issues for examination. The report says information "determined to be restricted from public release has been redacted," and some of its passages are blacked out. Parts of it follow, not the whole.

Models That Choose What to Examine

TIGTA says the IRS was "proactively using AI prior to the December 2020 Presidential Executive Order, Promoting the Use of Trustworthy AI in the Federal Government," and that it "integrated statistical and machine-learning techniques into return selection processes." Its actions included "exploring unsupervised models, which are trained on current return data and do not rely on historical examination results."

One model the report describes is the classification model for individual returns on Form 1040, which the report says learns relationships between individual tax return line-items to detect potential noncompliant returns. It replaced the prior manual classification process. It identifies the top three issues for examination. "However, the issues identified by the model are considered recommendations only. The examiner and their manager can add or remove issues based on their independent risk analysis."

Results That Could Flow Back

On learning from results, the report says, "However, historical examination results are informative and should be used by the IRS to monitor and improve AI models when available." For the Form 1040 model, the timing of its run "generally does not allow immediate feedback on the productivity of the recommended issues," because "most of the examinations are not completed by the time the model finishes its cycle run." Still, TIGTA says the data needed to analyze examined issues and their results "are systemically available."

Among the things the IRS could do, it says, it could review whether the issues the model suggested were productive, and refine the model to stop recommending an issue that is consistently unproductive. It could review the issues the model did not suggest but examiners and their managers added. And it could "Compare identified issues to the examiners' pursued issues," which "would indicate differences between machine and human risk assessments and allow the IRS to explore the cause for differences and make timely model adjustments." An issue an examiner adds or drops can mark a place where the person and the model disagreed.

Evaluating Performance, and the Recommendations

"The IRS does not have sufficient processes for evaluating whether implemented AI models perform better than prior processes or are achieving their intended objectives," the report says. "For example, the IRS cannot readily demonstrate in real-world context if AI models are improving the IRS's examination compliance efforts overall."

IRS management told TIGTA that its evaluation included pilot programs using random sampling and controls, for the individual return classification and selection models, and back-testing of models against historical audit results. "However, these processes were conducted pre-implementation." TIGTA points to a practice from the Government Accountability Office: as part of a monitoring plan, entities "should decide and document the range of model drift that is acceptable." The report explains that "Model drift refers to the changes in relationship between the data inputs and the prediction outputs," and that it "could cause performance degradation." IRS management, it says, "agreed that it could do more to monitor performance." Testing before a model goes live shows how it did then. Whether it keeps choosing well is a question monitoring after deployment answers, and that is where TIGTA found the IRS short.

TIGTA made three recommendations, and the IRS agreed with all three. It agreed with the first and the third, on using examination results and on a measurement plan to monitor the models and correct any model drift, "subject to staffing constraints" and anticipated new guidance from the Department of the Treasury. On the second, ensemble machine learning, the IRS said it "has already tested and implemented ensemble methods in these models, where appropriate."