AWS Launches Deception Benchmark to Combat AI False Positives in Security

• •AWS's Deception Benchmark tests AI models on distinguishing real vulnerabilities from false positives.
• •The benchmark includes 14,822 samples across 16 programming languages and over 70 CWE categories.
• •High false-positive rates in AI tools can lead to alert fatigue and reduced trust in security findings.

AWS has introduced the Deception Benchmark, a new tool designed to evaluate AI models' ability to distinguish real vulnerabilities from false positives in security code. The benchmark includes 14,822 samples across 16 programming languages and over 70 Common Weakness Enumeration (CWE) categories. High false-positive rates in AI security tools can lead to alert fatigue and decreased trust in legitimate findings. The benchmark aims to address this issue by testing models without hints, focusing on their understanding of both vulnerable and safe code. AWS evaluated 12 models from five providers, revealing that existing benchmarks do not adequately measure defensive precision. The dataset and evaluation process are publicly available for researchers to utilize. This initiative comes as AI is increasingly integrated into security tasks such as vulnerability triage and incident response.