The AI coding revolution has a security problem nobody is talking about

Steve R. Smith CISSP at Black Duck argues that AI-assisted development doesn’t reduce the need for application security tooling; it amplifies it, because risk compounds when velocity, volume and autonomous action all increase simultaneously

A particular narrative has taken hold in boardrooms and budget meetings, and it is worth examining carefully. The argument runs something like this: if AI models can generate highly functional application code at scale, they can also secure it. The premise is that security tooling belongs exclusively to the era when developers wrote every line by hand. That leads to the conclusion that in a world where AI handles the boilerplate, scaffolds the endpoints, and suggests the libraries, and writes code, the conventional AppSec stack looks like overhead rather than necessity.

Hard evidence points in a different direction. AI-assisted development does not reduce the need for application security tooling. It amplifies that need, for reasons that have nothing to do with AI being poor at generating code. Instead, it has everything to do with how risk compounds when velocity, volume, and autonomy increase simultaneously.

The productivity case for AI-assisted development is well-founded and worth acknowledging. Studies consistently show AI’s progress. Developers complete tasks significantly faster, merge cycles are shortening, and output volumes are increasing by multiples rather than percentages. These are genuine gains, and the organisations adopting AI tooling at pace are right to value them.

Each of those gains is also a risk multiplier. Faster merge cycles can mean that insecure code may reach production before a human reviewer can intervene. Higher output volumes mean the surface area requiring scanning, triaging, and remediating is growing faster than most security teams can handle. Productivity gains must be weighed against risk. Due to the potential autonomous behaviour of modern coding agents, users of GenAI should not trust everything it can do. Committing changes, calling APIs, querying databases, and invoking external tools, in an unsupervised manner assumes a level of trust in model judgement that the available evidence does not yet support in corporate settings.

These forces are not additive; instead, they compound. Merge velocity is up fourfold. Developer output is up fivefold. Combined with agents capable of taking consequential actions without human confirmation, the result is something far harder to quantify and manage than any single factor would suggest. That is because every legacy security control in a typical stack was designed with a human in the loop.

Independent research on AI-generated code quality has been pointing in one direction for long enough that it should inform how organisations approach these decisions. A Stanford study found that developers using AI assistance produced significantly less secure code than a control group writing without it.

The secondary finding was even more significant: those same developers were more confident they had written secure code. That confident but false sense of security can lead to more issues than a developer might otherwise include in a given project. The potential magnitude of this finding is more consequential than the vulnerabilities themselves. Meaning that unless they choose to include the use of independent security solutions, these developers are highly likely to put their business at risk of experiencing a security incident.

Veracode’s 2025 GenAI Code Security Report , testing over a hundred large language models across four languages, found a 45% failure rate on secure coding benchmarks. Apiiro’s analysis of real enterprise codebases found a 322% increase in privilege escalation paths in AI-generated code, a 153% increase in design flaws, and 40% more exposed secrets. These figures come from production environments where teams were already trying to use AI tooling thoughtfully.

Among the attack classes that AI-assisted development has introduced, package hallucination and the supply chain threat it enables deserve particular attention. The attack exploits a behaviour intrinsic to how language models work, rather than a flaw that can simply be patched out.

Large language models occasionally recommend importing libraries that do not exist. Pattern-matching against training data produces plausible-sounding but non-existent package names often enough to constitute a reliable attack surface. Researchers have found that approximately 20% of AI-generated code samples reference at least one package with no real counterpart on PyPI or npm. Around 43% of those hallucinated names recur consistently when the same model is re-prompted, and 58% appear across multiple separate runs.

The attack pattern that has emerged from this behaviour, now referred to as slop squatting, is straightforward. Attackers monitor which package names popular models commonly hallucinate, then register those names as real packages on public registries. They embed malicious payloads and wait for developers to follow AI suggestions without verifying that the dependency exists. Researchers have identified over 200,000 unique hallucinated package names across LLM outputs. The attack maps directly to the MITRE supply chain compromise category and has already been exploited in the wild on both PyPI and npm.

Standard Software Composition Analysis tooling already catches known-malicious packages and flags vulnerable dependencies. Slop squatting requires something more deliberate. Development teams need to verify that AI-suggested libraries are real and legitimate before installation. Automated pipelines need to confirm what gets pulled in matches what was intended. Neither of these controls is exotic, but both require awareness of the threat category, and most development teams are not yet treating AI hallucination as a supply chain risk vector.

Beneath both the code quality findings and supply chain risk sits a structural limitation that model improvement alone is unlikely to resolve. AI coding assistants optimise for code that runs. They have no reliable view of the business context surrounding an application. They don’t understand the compliance obligations it needs to satisfy or the threat actors most likely to target it. Lastly, it won’t know the institutional history that explains why a given library fell out of use. Adversarial thinking is not a default LLM behaviour, and there is no architectural reason to expect it to become one.

An AI-based coding assistant is an extraordinarily powerful tool for generating functional software at speed. Treating it as a security program creates precisely the confidence gap the Stanford research identified: teams shipping more code, faster, with a higher density of exploitable flaws with less awareness that the flaws are present.

When a CISSP with thirty years of security experience, using AI-generated tests and AI code reviews throughout a build, still returns 204 findings on a scan of an actively developed application, you should reconsider the question: can AI secure AI-written code? It’s worth considering what those numbers could look like for teams without a security background. When that team moves at the speed the productivity research describes into a threat landscape that now includes attack classes specifically designed to exploit how AI models behave, it should make the case for not exclusively using AI to secure AI-written code.

AI is the most powerful productivity tool the software industry has ever adopted. The attack surface does not shrink when developers ship faster. It grows, and the security investment required to manage it needs to grow accordingly.

Steve R. Smith CISSP is Sr. Technical Account Manager at Black Duck

Main image courtesy of iStockPhoto.com and ismagilov

Winston House, 3rd Floor, Units 306-309, 2-4 Dollis park, London, N3 1HF

[email protected]