Red Agents vs. Blue Agents: How to Make AI Better At Defense

The agentic AI playing field was heavily tilted toward offense, so researchers began using red team agents to help teach their blue counterparts.

Testing AI-based security systems can be tough amid growing fears agents cheating, hallucinating, and escaping containment , but a group of researchers believe they've found a way to better measure the effectiveness of agentic defenders.

Earlier this year, Dreadnode, an AI offensive security startup, released two open source tools designed to help users evaluate the security agents deployed in their networks. The first is DreadGOAD, a reproducible Active Directory training environment that's designed to replicate "the messy deployments still common in large organizations," according to the company.

The second is Ares, an agentic red team-blue team system designed to test and study offensive and defensive effectiveness. When Ares is deployed in DreadGOAD, the red team agents discover hosts, identify weaknesses, escalate privileges, and ultimately compromise the environment. The blue team agents, meanwhile, are tasked with analyzing telemetry, triaging alerts, investigating the activity, and hopefully containing it.

The open source tools were born from a project within Dreadnode to improve benchmark testing for security agents. The problem, according to Dreadnode AI research scientist Martin Wendiggensen, was that agents seem to be very good at playing the red team role and not nearly as good on the blue team side.

"We were noticing that the models coming out of the frontier labs right now are a lot better at offense than they are at defense — right to the point where they're escaping sandboxes and hacking other people's stuff rather than just finishing the task and staying within scope," he says.

The Dreadnode team this year started looking at turning its offensive security technology into better defense. The company, which will present its findings at a Black Hat USA session week, began running automated tests between the red team and blue team agents to more accurately score the two teams. The goal is to not only measure agent performance more effectively but to also set the stage for blue team agent self-improvement in what is inherently an unfair game.

As Wendiggensen explains, offensive and defensive tasks for agents are apples and oranges; red team agents have a wide range of options to, for example, compromise an AD environment , and it's easy for the engineers to determine whether or not they succeeded in their tasks. That's not the case for the blue team, he says.

"Defense is a lot more nuanced. It's a lot less binary," Wendiggensen says. "The tasks are so different that it is difficult to generate high quality training data for defensive tasks."

The Dreadnode team found that benchmark tests "give enough data to test and to see how good the agents are at a given thing, but not really enough data to train them," he says. And generating the volume of security data to effectively train the blue team agents required significant time and effort, which creates "human bottlenecks" for the process.

"And that's why offensive models are better than defensive models, in our opinion," Wendiggensen says. "It's easier for the labs to generate data for offense."

Therefore, Dreadnode began using the red team agents to generate the data through attack simulations and then applied the data to the blue team agents to see if they could improve their performance.

The red team agents had a decided advantage in the tests, and it showed in the initial results. They typically achieved full domain control of the AD environment in less than six minutes, and because large language models (LLMs) are non-deterministic, the agents used a variety of attack techniques across the tests. "Our red teams are fantastic," Wendiggensen says.

The blue team agents, however, were another story. "One of our first findings was that they were really bad," he says. "The blue team agent I wrote was sh*t."

One of the first issues Dreadnode found was that the blue agent was struggling to manage the amount of data needed to run an investigation. Specifically, the agents "would just fill up the context window and then just die," Wendiggensen says.

The development team added a context manager for the blue agents to help them decide what information to keep, what information to store as the investigation progressed, how to store and compress that information, and how to recall it when needed.

"The other thing we found was that it didn't stay engaged during an attack, and that it would basically quit," he says.

The blue agents would handle the first few alerts that came in and stop investigating for additional activity or lateral movement . Additionally, the blue agent's reasoning degraded under pressure during a live attack.

But over time, the Dreadnode team began to see self-improvement with the blue team. The agents started reasoning backward after an attack to better understand the nature of the threat. And through prompt changes in how the agents were tasking themselves, the engineers began to see blue team scores increase.

Additionally, Dreadnode found that performance improvement paralleled cost improvement for the blue team agents. AI token costs can be significant, and Wendiggensen points out that a couple of Windows event queries can add up to around 200,000 tokens.

But Dreadnode discovered that costs went down by 25% as the blue agents got better at querying and deciding which types of data it needed to extract. "It's making much more nuanced decisions what information needs to go to the API and be reasoned and be presented to the LLM," he says.

Wendiggensen will present additional results and observations from the tests during week's session with Jayson Grace, former principal software architect at Dreadnode (now at Google), and Shane Caldwell, principal research engineer at Dreadnode.

Aug 1, 2026 TO Aug 6, 2026

Mandalay Bay Convention Center, Las Vegas, USA

The premier cybersecurity event of the year returns to Mandalay Bay with a re‑engineered, six‑day program built to ignite innovation, push boundaries, and bring the global security community together like never before. This year’s event features four days of immersive, expert‑led Trainings (August 1–4), followed by Summit Day on Tuesday, August 4, and a two‑day main conference packed with groundbreaking Briefings, open‑source tool demos in Arsenal, a dynamic Business Hall, and unlimited learning & networking opportunities. Use code: DARKREADING to save $200 on a Briefings pass or $100 on a Business pass.

Senior News Director, Dark Reading

Rob Wright is a longtime reporter with more than 25 years of experience as a technology journalist. Prior to joining Dark Reading as senior news director, he spent more than a decade at TechTarget's SearchSecurity in various roles, including senior news director, executive editor and editorial director. Before that, he worked for several years at CRN, Tom's Hardware Guide, and VARBusiness Magazine covering a variety of technology beats and trends.

Prior to becoming a technology journalist in 2000, he worked as a weekly and daily newspaper reporter in Virginia, where he won three Virginia Press Association awards in 1998 and 1999. At TechTarget and Dark Reading, he has won several Azbee awards, including the 2026 National Silver Award for a series on vibe coding.

At Dark Reading, Rob currently covers security operations, cloud security, and Internet infrastructure. He has a keen interest in malvertising activity and the certificate authority industry, and has written extensively on both topics. He graduated from the University of Richmond in 1997 with a degree in journalism and English. A native of Massachusetts, he lives in the Boston area.

The State of Cloud Security: The Latest Challenges

How Organizations Are Managing Incident Response

How Enterprises Are Developing Secure Applications

Inside RSAC 2026: security leaders reveal the risks redefining your defense strategy

Essential News & Insights from Black Hat USA 2025

Experts Explain How to Develop a Framework for Cyber-Fraud Fusion

Prevention at Machine Speed: Hunting Beyond Known Detections

0-Day to 10x Discovery: Security at the Speed of Mythos

When AI Becomes an Insider: Rethinking Risk in Critical Infrastructure

Governing the Agent; Identity Security in the Age of Autonomous AI