Humans Miss One-Third of Malicious AI Command Requests in Testing Game

A browser-based game tested human ability to approve AI coding agent requests, revealing that players missed approximately one in three malicious commands. The game simulated 40,000 runs with 409,000 decisions, showing that 34% of the commands were threats. Players often approved commands that could exfiltrate sensitive data, such as AWS credentials, due to time pressure and permission fatigue. The most commonly missed commands were those that appeared benign but could execute harmful scripts. Developers reported that constant approvals led to fatigue and decreased attention, increasing the risk of dangerous actions slipping through. The results highlight the challenges of relying on humans as the last line of defense against AI-driven threats.

Key Points: • Players missed about 34% of malicious AI command requests in a testing game. • Commands that appeared benign were often approved, leading to potential data exfiltration. • Permission fatigue significantly impacted decision-making accuracy among players.