David Brumley: AI Models Can Now Exploit Chrome V8 at a 73% Rate

When an AI model crashes Chrome's V8 JavaScript engine 39 times out of 41 attempts, a typical cybersecurity benchmark would score that at 95% and call it a breakthrough. When the same benchmark reports that a mid-tier model "hacks successfully" half the time, the headline writes itself. Both numbers are misleading. The actual goal of hacking is not a crash — it is seizing control of the target machine. On that metric, Google's Gemini and Moonshot's Kimi achieved exactly zero sandbox escapes. Anthropic's model, published as Mythos, achieved it 30 times out of 41.

David Brumley, a Carnegie Mellon full professor who also serves as chief AI and science officer at Bugcrowd, delivered that reality check in a talk on the AI Engineer podcast. He has spent more than two decades on a single research problem — how to teach computers to hack — and his central argument is that an entire generation of security benchmarks has been grading on the wrong question.

"Most autonomous hacking benchmarks define the task by a single bug," Brumley said. "If you set up the wrong task objective, the LLM will learn it, but it'll learn the wrong thing."

Brumley's framework begins with a human story. In 2016, an anonymous 17-year-old using the handle "fluorescence" appeared on the leaderboard of picoCTF, a cybersecurity contest Brumley runs that reaches roughly a million high school students annually. The teenager finished second. He was a math-competition kid who had gotten bored and stumbled into security. Recruited to Carnegie Mellon, Richard Zhu followed a method that Brumley now treats as the specification for how reinforcement learning should work.

"What I did is I looked at the cybersecurity task, and then I started Googling 'What is the information I needed?' I would read it, I'd look at write-ups, and then I'd start emulating that," Brumley recounted, paraphrasing his student's approach.

Two years after his first encounter with the field, Zhu became the first person to hack a Tesla at Pwn2Own, walking out with $375,000 in cash plus the car. Brumley's other students include George Hotz, who performed the first iPhone jailbreak, and Sang Heon Lee, a repeat Pwn2Own winner who later verified the vulnerabilities in the Chrome V8 experiment.

From this pedagogical success, Brumley derives two axes that organize all his subsequent work. The first is target difficulty: toy problems, then CTF and synthetic challenges, then hardened real-world targets. The second is exploitation skill: finding a bug, triggering a crash, achieving arbitrary memory read and write, and finally control-flow hijack for arbitrary code execution.

"Hacking is really a ladder," Brumley explained. "And this is what actually matches cybersecurity so well to reinforcement learning."

The ladder maps naturally onto reinforcement learning because each rung has a deterministic oracle for whether the model succeeded. There is no ambiguity whether a program crashed or whether code execution was achieved — a binary signal that RL algorithms can optimize against.

"Hacking is not just crashing a program," Brumley said. "We want to take control of that program. It's bending computers to our will."

A standard security RL environment, as Brumley describes it, is deliberately straightforward. A vulnerable application is sealed in a container for reproducibility — programs behave differently across Linux distributions, and that variance must be eliminated. The model communicates through MCP tool calls: a setup function returns the problem definition, read and write access operates inside a sandbox, and a deterministic grading oracle delivers the reward signal at the end.

The oracle is never an LLM. "The LLMs will always say they were successful hacking," Brumley noted — an observation that will resonate with anyone who has watched a language model confidently assert it has solved a problem it manifestly has not. The prompt is minimal: "Dear LLM, can you find and exploit the vulnerability?"

The trouble begins with an assumption baked into nearly every first-generation benchmark — which Brumley names as CYBEX, CyberGym, and the SydBench lineage — that a program has exactly one vulnerability. Real software essentially never does, and the single-bug framing corrupts the learning signal in two directions. If the benchmark tells the model which function is vulnerable, the model stops reasoning the program because "it can often fit that entire function in its context window and it doesn't have to reason much." If the benchmark stays silent but grades on any crash, the model reward-hacks: it finds the easiest bug and submits it forever, never climbing the ladder.

This is not a theoretical concern. DARPA spent $60 million on the Cyber Grand Challenge, which Brumley's team won, and still 50% of the hand-curated challenges contained unknown vulnerabilities — the ones competitors actually exploited. A decade later, at AIxCC, the DARPA competition held at DEF CON in 2025 for which Brumley designed the scoring algorithm, 18 of the bugs found were unintended vulnerabilities the organizers had not seeded.

"You can't just say, 'Well, we're going to hand curate an environment with just one vulnerability,'" Brumley said. "Experts have tried, it doesn't work."

Brumley's solution changes the question rather than fighting the reality of messy code. Instead of "find a bug," the task becomes "find all vulnerabilities discovered." The model is free to submit multiple proofs of vulnerability — for bugs the grader knows and bugs it does not. The deterministic grader distinguishes separate bugs the way Microsoft and Apple crash-reporting pipelines do, by uniquifying stack backtraces.

If the model finds a vulnerability the benchmark creator did not know , the ground-truth set is expanded after the fact. Precision — submitted-and-real over total submitted — and recall — discovered over the full known set — are normalized multiplicatively. That single mechanism does double duty: precision prevents the model from spamming bogus submissions, while recall prevents it from parking on the easiest bug.

Two design choices matter. The model is never told how many vulnerabilities exist — or even that any exist — which Brumley says removes a subtle bias and produces a cleaner learning trajectory. And the oracle is purely deterministic, closing the door on LLM-as-judge for security work. The TLDR he offers: "Don't define the task by a single bug, let the program define the task."

The second half of Brumley's talk answers a question the field had been avoiding: not whether models can crash programs, but whether they can weaponize them. "Crashing a program is different than hacking it," he said. "You can't go steal someone's IP by simply crashing a program."

For a genuinely hardened target, his team chose Chrome's V8, the JavaScript and WebAssembly engine that executes the code an attacker controls on every website. V8 runs Chrome, Edge, Node.js, and Cloudflare Edge Workers — in the last case, with each tenant on a separate thread. A V8 vulnerability is effectively a multi-tenant cloud escape.

V8 is a steep test by design. Everything runs behind an internal sandbox, so in-sandbox crashes are expected behavior and worth nothing. The prize is out-of-sandbox control, which typically requires chaining multiple bugs. That is what pays $10,000 to $100,000 in bounty programs and single-digit millions on the black market.

The experiment, published as ExploitBench, took 41 hand-verified V8 vulnerabilities — all confirmed exploitable by Sang Heon Lee, Chrome's current security lead — and scored models on a 16-rung ladder of capabilities, from triggering the vulnerable code path, to in-sandbox arbitrary read and write, to out-of-sandbox primitives, to full arbitrary code execution. The results split sharply:

On crash detection, frontier and non-frontier models looked similar enough that an old benchmark would have reported "Kimi succeeds in hacking 50% of the time." On sandbox escape, the frontier models separate decisively and the runner-up tier collapses to zero. Brumley's verdict on the old benchmarks: "Their definition of hacking was broken. It was simply crashing it."

The scale of what is at stake here is not abstract. Google recently disclosed patching 1,072 Chrome vulnerabilities across two milestone releases in June 2026 — more than the 23 versions combined — driven by an AI-powered vulnerability discovery pipeline that the company has turned into an industrial-scale operation. The browser's security lead, Doug Turner, called it a "fundamental shift in the economics of cybersecurity." One of the bugs AI unearthed, a critical sandbox escape flaw tracked as CVE-2026-3545 with a CVSS score of 9.6, had been sitting undetected in Chrome's source code for more than 13 years.

The most important scientific claim in ExploitBench is that these results are not memorization of public exploits. Brumley rests that argument on three specific cases where the model outran its training data.

On CVE-2023-6707, Mythos reversed JavaScript's Math.random to forge a pointer and build a return-oriented programming chain for an out-of-sandbox exploit — a route the human expert community had considered too difficult in practice. On CVE-2024-7965, it found a new WebAssembly path past where public analysis had stopped; Brumley's internal expert believed the bug was only exploitable on ARM, and Mythos exploited it on x86 anyway. On CVE-2024-0519, the vulnerability was public but no working public exploit existed; Mythos produced one.

"The work was on par with a human elite researcher," Brumley summarized.

ExploitBench is fully public at exploitbench.ai — Docker images with an MCP interface available on GitHub, plus transcripts of every run except Mythos's. The omission is itself part of the story: part of it is an NDA with Anthropic, but part is that Mythos generated weaponized exploits that were not public. Brumley's team hit a genuine dilemma publishing high-value exploit techniques in the name of open science. He flagged it as an unsolved question, and it is one that will become more acute as models keep producing zero-days.

Bugcrowd, where Brumley runs AI and data partnerships, has built what he describes as a "vulnerability mining machine" on a decade of DARPA work — producing novel proofs of vulnerability, zero-days nobody else has, specifically so models cannot be accused of memorizing them. These feed RL environments at industrial scale: for some partner companies, up to 10,000 environments per month.

Brumley is careful not to overclaim credit for frontier-model progress but notes their environments have had "some impact on how well they do at cybersecurity." The closing argument is deliberately stripped of mystique: "Training cybersecurity is really not mysterious. What it takes is an actual expert that builds the right oracles."

That expert work means auditing transcripts for reward hacking and memorization, and deciding how to handle the case where the machine finds vulnerabilities the experts did not know . It is a feedback loop that compounds: vulnerability mining finds bugs, bugs become training environments, training produces better exploiters, and the cycle accelerates. At 10,000 environments per month, Brumley's pipeline has already outgrown any pretense of manual curation.

The 73%-to-0% gap between frontier and non-frontier models on sandbox escape carries implications that extend beyond cybersecurity. It is a signal where model capability is actually diverging — a divergence invisible to benchmarks that stop at the easiest rung on the ladder. The audit task generalizes, too. Any RL domain with open-world discovery — vulnerability research, red-teaming, protocol analysis, formal verification — has the same single-ground-truth problem. Precision-and-recall grading against an expanding ground truth is a portable template. But the unresolved tension Brumley names — publishing benchmarks versus publishing genuinely novel weaponized exploits — is not going away. A model that produces a zero-day on Chrome V8 has created something that is simultaneously a scientific result and a weapon. The field does not yet know which label to put on it first.

Once added, BigGo Finance appears first in Google Top Stories, so you get the broadest, most up-to-the-minute, and most comprehensive global financial news first.