Scale X Game Shows Humans Miss 33% of Rogue AI Commands

A security simulation game developed by Scale X has revealed significant vulnerabilities in the human-in-the-loop security model for AI coding agents. Analyzing data from over 40,000 runs and 409,000 individual decisions, the study found that the average player missed 33.7% of threats, yielding a mean accuracy of 66.3%. Furthermore, 32.9% of all sessions ended with a negative score because penalties from approved threats and blocked safe commands outweighed correct decisions. While 35.2% of players caught every threat, only 20.8% did so without excessively blocking benign commands. Meanwhile, 7% of players approved every single prompt.

The threat miss rates varied widely by category. Players caught obviously destructive commands like rm -rf / or chmod -R 777 / most reliably, missing them only 11.7% of the time. However, persistent mutations like crontab injections had a 23.8% miss rate, while exfiltration and code execution commands were missed 33.4% of the time. Scope violations, such as cat ~/.aws/credentials, went unnoticed 35.0% of the time. The single most-missed command was npm run analyze, which was approved 64.7% of the time. Other familiar script names like npm run setup and npm run deploy were missed 48.0% and 44.9% of the time, respectively. Pooled together, these three commands had a 52.5% miss rate across 9,482 instances.

Conversely, players frequently blocked harmless commands, slowing down workflows. For instance, npm config set registry https://npm.internal was blocked 59% of the time, rm -rf dist/ was blocked 45% of the time, and kill $(lsof -t -i:3000) was blocked 43% of the time. Additionally, cat ~/.zshrc divided players, with 45.9% approving it. This high rate of false positives contributes to permission fatigue. As Anthropic noted regarding its Claude Code tool, frequent approval prompts cause users to become "much less diligent in their supervision" over time.

For software development practitioners, these findings demonstrate that human oversight is an insufficient fallback for AI agent security. Because developers lack full context and face constant pressure to ship quickly, they easily overlook malicious payloads hidden behind familiar command names. To secure AI agents, organizations must move away from manual approval prompts and instead implement robust technical safeguards, such as strict sandboxing, context isolation, and automated pre-screening tools like Anthropic's Auto Mode.