Triage Is The Product

Notes from the Ethereum Foundation's Protocol Security team on running coordinated AI agents against real protocol code, including how we organize the work, what holds up under scrutiny, and what client teams and security researchers can take from it. This post stands on its own; later posts will go deeper on individual clients.

On the Ethereum Foundation's Protocol Security team, we've been running coordinated AI agents against the kinds of systems the network depends on, like systems software, cryptographic code, and contracts that have to be right. The agents found real bugs. One is now public: a remotely-triggerable panic in libp2p's gossipsub, a core part of the peer-to-peer layer Ethereum consensus clients run on, fixed and disclosed as CVE-2026-34219 with credit to the team.

Agents finding bugs wasn't the surprise. The surprise was how little of the work went into finding them, and how much went into telling the real bugs from the ones that just looked real.

This post is for client teams and security researchers who want to do the same thing. It covers how we organize the agents, the bar a candidate has to clear before it counts as a finding, and the habits that keep the results trustworthy.

Teams elsewhere are converging on the same recipe. Anthropic's Frontier Red Team built an agent that writes property-based tests and found real bugs across the Python ecosystem . Cloudflare ran a frontier model through a security-research harness against their own systems. Everyone lands on the same loop: point a capable model at a codebase, let it , and triage what comes back. So the real question is how to do this without drowning in confident-sounding noise.

One caveat up front: tooling for agent-driven audits moves fast, and any specific setup is out of date in a few weeks. So this post is deliberately the methods, which are persistent, rather than the tooling. Disclosure is its own topic and will probably be its own post.

An agent pointed at a codebase is a tool, a lot like a fuzzer. The difference is what comes back. A fuzzer hands you a crash and a stack trace. An agent hands you a lot more, including a write-up (call chain, impact claim, suggested severity) and the artifacts to back it, like a proof-of-concept you can run against the real code.

All of that makes the result easy to read and easy to trust, the running proof-of-concept most of all. So don't count how many candidates an agent produces. Count how many turn out to be real.

We run many agents in parallel against one target. They coordinate through the repository itself, with shared state in version control and no central process handing out work. An agent writes down a claim where the others can see it, does the work, and commits.

We got this approach from Anthropic's writeup on building a C compiler with a fleet of agents , which coordinates the same way. There's no central coordinator to build or maintain, and less that can go wrong.

The roles are generated by the work that's discovered:

We didn't invent this pipeline. Cloudflare describes the same stages, recon, parallel hunting, independent validation, deduplication, reporting, and their writeup helped shape ours.

Here's what a candidate looks like before it counts as a finding:

The schema is there for a reason. It forces a specific, testable claim and a clear definition of done. An agent that has to write down an observable proof can't fall back on "this looks risky."

One rule matters more than any other. A candidate isn't a finding until there's a self-contained artifact that reproduces the failure against the real code, and that runs for someone who didn't write it.

The reproducer doesn't read the write-up, and it doesn't care how confident the model sounded. It either runs or it doesn't.

Most of its value is in the false positives it catches. Three of them come up over and over, and each one is the agent getting a pass for the wrong reason:

None of this is new. It's the same thing as a test that passes because it doesn't actually check anything. What's new is the volume. An agent writes the useless version as fast as the real one, and just as confidently. So the check has to be automatic. You can't count on the agent to catch itself.

Most candidates are wrong, duplicate, or out of scope. That's not a problem with the method; that's how it works. The goal is to reject the wrong ones fast and back the real ones with proof that's hard to argue with.

Every candidate that survives gets two independent checks. Can a real attacker actually reach it in a normal configuration? And what does it cost the attacker to pull off, compared to what it costs the network if it works? A bug that any single peer can trigger is very different from one that needs special access or a huge amount of resources.

Everything gets checked against a running list of what's already known, fixed, or rejected. Without that, the agents keep rediscovering the same closed issue and reporting it again and again.

Acceptance rates vary a lot from target to target, and that variation is useful on its own. Run this against mature, heavily audited code and almost nothing survives, which is still worth knowing. "We looked hard and found nothing" is a real result. Run it against less-explored code, or against formally verified code, where a machine-checked proof covers a model and the deployed bytecode is only assumed to match it, and more gets through.

We're not the only ones who found that the triage is the hard part. Cloudflare's main takeaway was that a narrow scope beats broad scanning. Anthropic's property-based-testing agent generated something like a thousand candidate reports, then used ranking and expert review to get down to a top tier that held up 86 percent of the time. The generation was the easy part. I'm not going to publish our own numbers here; tied to a specific target, they'd say more the target than the method.

There's hype in both directions, so here's a plain list of what the agents do well and where they mislead.

The split isn't even steady from one task to the . Stanislav Fort, testing a range of models on real vulnerabilities, calls this a jagged frontier , or a model that recovers a full exploit chain on one codebase can fail basic data-flow tracing on another. You can't assume one good result means the will hold up, which is another reason every candidate gets checked on its own.

The last row is the important one. A single agent session is good at one-shot reasoning and bad at bugs that span a sequence of steps, where each step is valid and only the order is wrong. For those, the agent isn't the tool. Its job is to suggest which sequences are worth running through a stateful test harness . Used that way, it works well. Used as a replacement for the harness, it misses the most expensive bugs there are, the ones that only show up across a sequence.

A few habits do most of the work of making agent findings trustworthy, and none of them are complicated.

AI didn't replace the security researcher. It moved the work. The time that used to go into coming up with and chasing down hypotheses now goes into judging them at scale, including building the oracle, running the triage, keeping the list of known issues, and handling disclosure.

The bottleneck didn't go away. It moved from finding bugs to trusting the results, which is a better place for it, because that's where human judgment actually matters. But it's still a bottleneck, and ignoring that is how you end up shipping a wrong "it's fine."

The practices that make this work aren't new. Reproducible failures, real oracles, and careful triage are the same practices that turned fuzzing from a research topic into standard practice over the last fifteen years. The tools are new. The practices aren't.

How fast the tools keep changing is an open question. Nicholas Carlini, careful and once a skeptic himself, argues the exponential case is worth taking seriously , even while he keeps wide error bars on it. If the generation side climbs that fast, the judgment side has to climb with it, or the gap between what gets produced and what actually gets verified only widens.

For the systems Ethereum depends on, that's the part that matters. Agents let us cover far more ground than we could by hand. In exchange, they ask for more careful judgment, across a much bigger pile of confident-sounding claims. That's a trade worth making, as long as you remember that the judgment is the real product.