AWS puts AI vulnerability detection to the test, and false positives pile up

AWS’ Deception Benchmark measures how well AI models distinguish genuine security vulnerabilities from code that looks risky but is safe. AWS is making it publicly available so researchers can use the dataset and evaluation process without repeating the cost of generating and refining the samples.

Security teams use AI for vulnerability triage, penetration testing, threat modeling , incident response, and code review. High false-positive rates can create more work, increase alert fatigue, and reduce confidence in legitimate findings.

The benchmark contains 14,822 samples across 16 programming languages and more than 70 Common Weakness Enumeration (CWE) categories. AWS evaluated 12 models from five providers.

Why another security benchmark?

Existing benchmarks test AI on a range of cybersecurity tasks. CyberGym includes more than 1,500 realistic tasks, Meta’s CyberSecEval covers capabilities such as exploit generation, and CYBENCH focuses on capture-the-flag challenges. ExploitGym tests whether models can go from finding a vulnerability to producing a working exploit.

AWS’ benchmark focuses on a different problem: When a model says code is vulnerable, can it tell a real vulnerability from a false positive?

That can be difficult because code may contain a dangerous pattern while another control prevents it from being exploited. The model has to understand both the code and the protections around it.

How the benchmark works

AWS calls it the Deception Benchmark because its safe samples are designed to mislead models. They contain real vulnerability patterns alongside protections that prevent exploitation. Models must decide whether each sample is vulnerable or safe without being given hints.

“Production tools rely on multi-step loops and agentic workflows to compensate, but that scaffolding masks whether the model itself understands the code. This benchmark strips the scaffolding away and asks the model to make the call in a single pass, so what it measures is understanding, not how many tries a harness takes to get there,” Anshumali Shrivastava , Amazon Scholar, and Neha Rungta , Applied Science Director at AWS Identity, explained .

AWS generated examples, tested them against frontier models , and made them harder when models classified them correctly. Samples that models could easily classify were excluded. According to AWS, generating and refining the samples consumed tens of billions of tokens.

Of the 14,822 samples, 9,695 are scored. These include 6,988 code-level and 2,707 environment-gated challenges.

Code-level challenges have vulnerable and safe versions separated by a small change. Both may look unsafe, but only one can be exploited.

Environment-gated challenges test the same kind of code under different deployment conditions. For example, a Kubernetes Network Policy may block an SSRF attack that appears possible from the code alone. The model must take those protections into account.

AWS publicly releases the samples but withholds their labels. The dataset also includes 5,127 unscored samples mixed with the scored ones. Users submit their predictions to AWS for verified scoring. This setup is intended to make it harder to optimize specifically for the benchmark.

Multiple independent reviewers check each label without seeing other reviewers’ decisions or the reasoning behind the original label. Disagreements receive further review, and unresolved cases go to human reviewers.

AWS repeats this process until fewer than 3% of scored samples remain contested by independent reviewers, with a target of fewer than 1% surviving human review. Samples that remain disputed are moved to the unscored set instead of being relabeled.

The company also reports that a human review of 100 randomly selected scored samples found no labeling errors.

Models struggle with false positives

The benchmark is roughly balanced between vulnerable and safe samples, so random guessing would score 50%. AWS considers false-positive and false-negative rates below 10% a minimum bar for production use. None of the tested configurations met both thresholds.

FPR compared to FNR for 12 models across two prompting strategies. No model reaches the generous bar. (Source: AWS)

With direct prompting, models generally found nearly all real vulnerabilities, but incorrectly flagged 41% to 99% of safe code. Precision ranged from 52% to 71%. Asking models to prove that a vulnerability could actually be exploited reduced false positives by 17 to 74 percentage points. The downside was more missed vulnerabilities, with false-negative rates ranging from 7% to 44%. Models had the most difficulty when external security controls made suspicious-looking code impossible to exploit.

AWS tested general-purpose models with single-turn prompts, not purpose-built security systems that use tools and multiple validation steps. The results therefore should not be treated as a direct measure of how complete security products perform.

IDScan confirms breach after 153 million driver’s licenses leak on dark web

Getting a stranger’s phone kicked off the cellular network costs a few dollars

Building a ransomware decision tree before the call comes in

Download: The High-Performance Team Playbook

Simplify security management with CIS SecureSuite Platform

Download: The IT and security field guide to AI adoption

IDScan confirms breach after 153 million driver’s licenses leak on dark web

Getting a stranger’s phone kicked off the cellular network costs a few dollars

Building a ransomware decision tree before the call comes in

Cisco FMC bugs exploited by nation-state and ransomware actors (CVE-2026-20079, CVE-2026-20316)

Fake GTA 6 download delivers malware-packed bundle to impatient gamers