No Substitute for Experienced Developers: France's and Germany's Cyber Security Agencies on AI Coding Assistants, September 2024

France's national cybersecurity agency, ANSSI, and Germany's Federal Office for Information Security, BSI, wrote a joint report on AI coding assistants, last updated in September 2024. Its recommendations are for "managers and developers," and it says it is meant to "contribute to responsible and safe use of AI coding assistants." It covers their use "for professional software development." No-code and low-code tools, general chatbots and coding assistants not built on large language models fall outside it. Parts of it follow, not the whole.

Well-Worded and Flawed

The report names a risk it calls automation bias. Large language models, it says, "typically generate linguistically flawless and convincing text, which can foster an impression of human-like abilities and overconfidence in their statements," leading users "to draw false conclusions or accept unverified statements." For beginners, it says, the tools can be a valuable learning resource, but they also pose "the risk of excessive dependence and uncritical acceptance of generated code, as even flawed solutions are well-worded."

Studies the report cites show "a cognitive bias when using AI coding assistants, as many developers perceive them as secure, although security vulnerabilities are regularly identified." It adds that programmers who rely too much on such tools might lose, over time, the necessary skills to thoroughly evaluate and review source code, and that programmers "should be able to continue business operations without AI assistance (e.g. in case of incident)."

To meet the risk, the report suggests making employees aware of it and training them in prompting and in challenging answers. It also says it "might be beneficial" to take AI-generated code and the prompts used apart "in public code reviews within the company."

Checking the Output

AI coding assistants can suggest code based on best practices, the report says, "but there is a significant risk of generating incorrect, insecure, or inefficient code." It adds that explanations, comments and documentation that an assistant writes can be incorrect or completely hallucinated as well.

Among its measures, generated content, "in particular source code, should generally be reviewed and understood by the developers." It lists automatic function tests, more code reviews by security teams, automated vulnerability scanners, and chatbots that "critically question the generated source code." And it notes a matter of scale: "AI-assisted developers might produce more code, so security teams may need to scale accordingly."

It also suggests flagging AI-generated code blocks and documenting the AI tools used. That "might help security testing," it says, and could be useful to "third-party auditors, e.g. in the context of a security certification process."

What It Asks of Managers

At the management level, the report's first point is the one in its summary: "AI coding assistants are no substitute for experienced developers. An unrestrained use of the tools can have severe security implications." It asks for "a systematic risk analysis" before AI tools are introduced. And it says it is important that other teams, such as quality assurance, "also benefit from similar productivity gains through AI or are scaled up personnel-wise."

It recommends, too, an evaluation phase "with monitoring of relevant key figures (e.g., number of new checked-in code or security team workload), measurable goals, and corresponding success controls." The lists of recommendations in its conclusion, it says, "are not exhaustive but provide baseline suggestions that should be supplemented by company- and situation-specific measures."