Stronger AI Safety Requires Peeking Inside the 'Black Box'

Researchers propose focusing on identification of certain cognitive elements in LLMs that indicate when AI systems may take an unwanted action.

Adding security to AI often treats the large language model (LLM) or AI system as a black box, focusing on analyzing the tokenized inputs and outputs but not what goes on inside the model. The resulting techniques are often complex and individualized to specific models.

A group of offensive-security researchers hopes to change that, presenting at the Black Hat USA 2026 in August a model-agnostic approach to activation analysis with standardized rules for processing activation events. Rather than labeling certain activation distributions as "cybercrime" or "hate speech," the approach uses a more granular scheme of cognitive elements (CEs) that can be combined in rules. Combining cognitive elements such as "create content," "provide/give," "click/enter," and "personal information" in a logical statement, for example, results in a rule for detecting potential phishing attacks.

The goal is to create an open system of identified cognitive elements and rules that detect specific types of safety events, similar to Snort or YARA rulesets, says Yisroel Mirsky, an assistant professor and head of the Offensive AI Research Lab at Ben-Gurion University of The Negev.

"What we really tried to do here is make that separation — all the annoying math and processes and everything that's done at the low level on the neural network — separate from what the practitioner has to do," he says. "The practitioner wants [just to deal with] high-level stuff — what the model should and shouldn't do."

Unsafe and malicious prompts, including attempts to jailbreak AI, are a major problem for AI systems, allowing bad actors to utilize the power of LLMs and other AI services for problematic or outright malicious ends. While analyzing the tokenized content of prompts and outputs allows companies to erect firewalls and guardrails, attackers and persistent users have found ways to evade these defenses. Just using a different language for the prompt can often avoid content-based guardrails. Other methods just completely bypass defenses by finding vulnerabilities or other methods to sneak content to an LLM.

Analyzing the tokenized input to or output of a model is similar to antivirus scanning and heuristics — it's fast and catches a wide variety of attacks, raising the bar for attackers, but defense in depth needs more layers, says Mirsky.

"By looking at the input or the output, just the surface level of what's going on, we can detect a lot of different attacks," he says. "The problem comes when the adversary who already knows this tries to perform some sort of representation attack, tries to make it harder for these moderators to pick up on what the attacker's trying to do."

A series of cognitive elements and the neuron layers they typically trigger in AI systems. Source: Ben-Gurion University

Instead of focusing on analyzing the tokenized content of prompts and outputs, activation analysis takes a page from the playbook of neurobiologists who illuminate the mechanistic working of the brain by recording activations across humans' neural networks while subjects view different images or consider different actions. By instrumenting the model — essentially placing the digital version of probes in the brain — researchers can detect activation patterns, analyze them, and then map those patterns to various categories.

Past efforts, however, have mapped misuse through broad datasets, labeling a broad range of activity as "cybercrime" or "misinformation," which results in a large number of false positives and reduces how nuanced the detection rules can be. Finally, with current activation analysis, the results are often hard to interpret, the BGU team stated in an April research paper .

The researchers' system, dubbed "Governance via Activation-based Verification and Extensible Logic" or GAVEL, aims to associate specific activation patterns with more granular objects and predicates in a dictionary of cognitive elements. A combination of cognitive elements such as "conspiratorial" and one or more of "engage emotionally," "masquerade as human," and/or "sycophancy" would signal a delusional danger, the researchers stated in their paper.

"It doesn't matter how I represent the request, if the model's thinking doing that dangerous task — or it doesn't have to be dangerous, let's say it's just against policy — if it's thinking of doing that task, I should be able to pick up on that inside the neural activations themselves," Mirsky says.

Peeling away the black box of AI systems has many benefits, including universality and language-independence. An attacker who tries to get around token-based security using a different language, will still be detected by activation analysis, because what matters is that, after the model translates the instructions, the same groups of neurons are activated, he says. In addition, the approach avoids focusing on the content of inputs, which can often be hard to decipher or, when malicious, be structured in a way that avoids triggering defenses.

However, it remains a research project, Mirsky says — researchers "are still very much poking in the dark and making guesses and assumptions."

At the Black Hat USA 2026 conference, the researchers plan to introduce not only the technique but also a set of tools and rules to make creating defenses and policy that much easier. The project, which is funded by the European Union, introduces a set of cognitive elements, a rule-based detection framework (GAVEL), and code and tools for the AI security community.

"With any kind of rule-based system, the only way get traction is with people adopting it, contributing to the GitHub, contributing rules, contributing issues, whatever it might be, but really start using it and playing around with it," Mirsky says. "We hope to make it grow and then make it more useful to the person down the line that needs some sort of tool like this."

Activation analysis does not replace the analysis of tokens and prompt content, Mirsky stresses. The goal is to make it easy to add activation analysis as another layer of defense, he says.

"Every technique is another layer of security, and they each solve a different aspect and it all kind of raises the difficulty bar for an adversary to get around it," he says. "You wouldn't throw away token level moderation. You still need that. It's very cheap, efficient, and effective for most cases. It's just that for reducing false positives and making sure that those really sophisticated attacks aren't making it through, you need to dig deeper."

Aug 1, 2026 TO Aug 6, 2026

Mandalay Bay Convention Center, Las Vegas, USA

The premier cybersecurity event of the year returns to Mandalay Bay with a re‑engineered, six‑day program built to ignite innovation, push boundaries, and bring the global security community together like never before. This year’s event features four days of immersive, expert‑led Trainings (August 1–4), followed by Summit Day on Tuesday, August 4, and a two‑day main conference packed with groundbreaking Briefings, open‑source tool demos in Arsenal, a dynamic Business Hall, and unlimited learning & networking opportunities. Use code: DARKREADING to save $200 on a Briefings pass or $100 on a Business pass.

Rob is an award-winning, veteran technology journalist of more than 30 years, reporting on global cybersecurity issues, the latest offensive and defensive technologies, malware incidents, cyber conflict, and AI's impact on software and cybersecurity.

A former research engineer, Rob has written for more than two dozen publications, including CNET News.com, Dark Reading, MIT's Technology Review, Popular Science, and Wired News. He has received five awards for journalism, including Best Deadline Journalism (Online) in 2003 for his coverage of the Blaster worm. Rob also analyzes data on various trends using Python and R for both his reporting and his clients. Recent reports include analyses of the shortage in cybersecurity workers, annual vulnerability trends, and annual threat reports.

Rob holds degrees from Cornell University in Electrical Engineering and Computer Science (double major).

The State of Cloud Security: The Latest Challenges

How Organizations Are Managing Incident Response

How Enterprises Are Developing Secure Applications

Inside RSAC 2026: security leaders reveal the risks redefining your defense strategy

Essential News & Insights from Black Hat USA 2025

Experts Explain How to Develop a Framework for Cyber-Fraud Fusion

Prevention at Machine Speed: Hunting Beyond Known Detections

0-Day to 10x Discovery: Security at the Speed of Mythos

When AI Becomes an Insider: Rethinking Risk in Critical Infrastructure

Governing the Agent; Identity Security in the Age of Autonomous AI