Sakana AI Fugu-Cyber Claims 86.9% Vulnerability Score; Benchmark Methodology Not Disclosed

Sakana AI released Fugu-Cyber on July 21, 2026 — a dedicated cybersecurity endpoint grafted onto its Fugu multi-agent orchestration platform — and published benchmark scores that, if they hold under independent scrutiny, would put a Tokyo lab's orchestration system above OpenAI's GPT-5.5-Cyber and Anthropic's Mythos-Preview on the two hardest public security evaluations available. The claim is specific: 86.9% on CyberGym, a UC Berkeley benchmark that tests proof-of-concept generation for 1,507 real-world vulnerabilities across 188 software projects, and 72.1% on CTI-REALM, a Microsoft open-source benchmark for end-to-end cyber threat intelligence detection rule generation. Every figure is vendor-reported. CyberGym's own creators found that top model combinations cleared roughly 20% at ICLR 2026 — making Sakana's claimed score one of the largest unexplained gaps between vendor and independent benchmark results in recent AI history. Security engineers who want to evaluate Fugu-Cyber for their pipeline need to understand what those benchmarks measure, what they cannot tell you, and what Sakana itself says is required before production deployment. Exchange rate as of July 22, 2026; conversions are approximate.

The most important architectural fact Fugu-Cyber is what it is not. It is not a new frontier model trained from scratch on security data. It is an orchestration system that presents a single, unified API endpoint to the caller while dynamically routing tasks across a pool of specialized agents underneath — the same architecture as the base Fugu system, now configured and tuned for multi-step cybersecurity work .

A security engineer sends one request to the Sakana API. The orchestrator decomposes the multi-step security task, selects and coordinates specialized agents from a model pool, agents analyze, challenge, or verify intermediate work, and the system synthesizes one response through the original endpoint. The multi-agent machinery never surfaces to the caller .

That architecture has a specific logic in the security domain. Security work is naturally multi-step and adversarial: finding a candidate weakness is only the beginning; verifying reachability, reproducing behavior, understanding production context, proposing a safe fix, and retesting require different kinds of reasoning and evidence. A single-pass model answers these serially. An orchestration system can assign each step to a specialist and verify the output before synthesis — which is the core argument Sakana is making for why orchestration should outperform any single model on domain tasks.

Fugu-Cyber is grounded in two papers Sakana presented at ICLR 2026. The first, TRINITY (arXiv:2512.04695) , is a lightweight coordinator — approximately 0.6 billion parameters — optimized through CMA-ES, an evolutionary algorithm rather than gradient descent, an unusual design choice for a production-scale system. TRINITY assigns Thinker, Worker, or Verifier roles to models in the pool across multiple turns, adaptively delegating based on task type and difficulty. The coordinator's own text output is discarded — only the pool models write the final answer.

The second paper, the Conductor (arXiv:2512.04388) , is a roughly 7-billion-parameter model trained via reinforcement learning to discover natural-language coordination strategies rather than following hand-coded rules. The Conductor designs agent-to-agent communication topologies and writes targeted prompts for each worker, learning patterns the engineers did not prescribe. Together, the two systems answer a core question: can a small, specialized routing model extract aggregate performance from a diverse pool of frontier models that exceeds what any individual model in the pool achieves alone? Sakana's published benchmark numbers claim yes. Whether that holds across all security workloads, not just the benchmark subset, is what independent evaluation will need to answer.

One additional mechanism is worth flagging for technically-minded evaluators: Fugu can call instances of itself recursively — reading its own prior output, identifying gaps, and launching corrective workflows before returning a final answer. This enables a form of test-time compute scaling without any change to the caller's integration. The depth of recursion is a tunable dial at inference time , not a training choice.

The agent pool underlying Fugu-Cyber includes Claude Opus 4.8, Gemini 3.1 Pro, GPT-5.5, undisclosed open models, and the orchestrator itself as a possible recursive worker, per the Sakana technical report . The specific pool configuration used for Fugu-Cyber's security tasks has not been publicly disclosed.

CyberGym is a serious benchmark . Created by a team at UC Berkeley and presented at ICLR 2026, it covers 1,507 real-world vulnerabilities drawn from 188 software projects, spanning 28 distinct vulnerability classes — heap buffer overflows, use-after-free bugs, null pointer dereferences, and others that constitute the actual CVE inventory that ships, gets exploited, and gets patched in production C/C++ codebases. The task is not theoretical: given a text description of a vulnerability and the codebase, generate a proof-of-concept test that reproduces the flaw. That is exactly what a skilled human security researcher does during triage.

When CyberGym's authors evaluated top model combinations at the time of the paper's ICLR 2026 submission, the best performers cleared roughly 20%. Sakana claims 86.9% for Fugu-Cyber . The gap between those two figures is not a discrepancy to be explained away — it is either a genuine capability breakthrough, a consequence of a more favorable task variant or agent scaffold, or an artifact of the evaluation setup. Sakana's published benchmark graphic omits benchmark variants , trial counts, run dates, agent scaffolds, and uncertainty bounds, making it impossible to determine which explanation is correct from publicly available information.

CTI-REALM, the second benchmark, is a Microsoft open-source evaluation released in March 2026 . It tests the end-to-end detection engineering workflow — reading a cyber threat intelligence report, mapping MITRE ATT&CK techniques, writing and refining KQL queries against telemetry, and producing validated Sigma rules — across 37 real CTI reports from public sources including Microsoft Security, Datadog Security Labs, Palo Alto Networks, and Splunk. Testing spans Linux systems, Azure Kubernetes Service, and Azure cloud environments , scored by both final detection accuracy and trajectory-based rewards. Fugu-Cyber's 72.1% on CTI-REALM means nearly three in ten detection rule generation attempts do not succeed — a rate that one independent reviewer noted "needs consistent human review before rules reach production."

On the specific comparison figures that Sakana published, Claypier reported Fugu-Cyber scored 86.9% on CyberGym versus GPT-5.5-Cyber at 85.6% and Mythos-Preview at 83.1%; on CTI-REALM, Fugu-Cyber's 72.1% compares against GPT-5.5-Cyber at 67.3% and Mythos-Preview at 68.5%. All comparison figures are vendor-reported. No third-party laboratory has independently reproduced any of these scores at the time of publication.

Fugu-Cyber is not available to any developer who creates an account and enters a credit card number. Unlike the base Fugu and Fugu Ultra tiers — available via subscription to anyone who signs up — Fugu-Cyber requires applicants to submit an intended use case and verified information for manual review by Sakana's team before access is granted . Pricing, once approved, runs exclusively through the Token Plan at $6 per million input tokens and $36 per million output tokens, with cached input tokens at $0.60 per million. For requests requiring more than 272,000 tokens of context, the rates rise to $12 per million input, $54 per million output, and $1.20 per million cached input.

This context matters because AI tools repurposed for offensive workflows are not hypothetical. The OWASP GenAI Exploit Round-up for Q1 2026 documented multiple cases of legitimate AI tooling being redirected for offensive purposes. Google Threat Intelligence reported in May 2026 that threat actors now pursue anonymized premium-tier access to models through professionalized middleware and automated registration pipelines. Sakana's manual review gate is designed specifically to prevent this.

Sakana's launch announcement includes an unusual amount of self-imposed limitation for a product launch. The company explicitly pushes back on what it calls "fearmongering" frontier cyber model capabilities — specifically the narrative that gaining API access to a state-of-the-art security model resolves an organization's vulnerability problems.

The pushback is grounded in Sakana's own enterprise experience. Working alongside major Japanese financial institutions, the company says it found a consistent pattern: even organizations with access to frontier models with strong cyber capabilities struggled to operationalize them without specialized internal security talent and deep integration into proprietary source code. A raw model run in isolation produces false positives — it flags potential vulnerabilities without understanding the specifics of a live production environment. Sakana's Applied Enterprise team exists to build the verification harnesses, proprietary-code integrations, and human review loops that turn candidate findings into defensible production decisions.

Sakana also cited a Nikkei Digital Governance report on the gap between frontier AI capability and enterprise security operations at large Japanese institutions, though that source is in Japanese and was not independently fetched for this article.

One customer testimonial published on the Sakana product page described an end-to-end security assessment produced from a single scoped instruction — from reconnaissance through XSS and SQL injection checks to authentication review and a structured report with evidence and retest steps — with the system staying within the defined scope and avoiding destructive actions. That testimonial is company-published and should be treated accordingly by any security team making an adoption decision.

The Fugu-Cyber launch carries a structural argument that goes beyond the product itself: that the competitive axis in applied AI security is shifting from which frontier model a team has access to, toward who has built the orchestration, verification, and domain-expertise layer on top. As FourWeekMBA's analysis of the launch frames it, Sakana's argument is that the model is not the product — the orchestration workflow is.

That argument has a specific economic implication. The TRINITY coordinator is approximately 0.6 billion parameters — smaller than most open-source models available today. The Conductor is 7 billion parameters. Together, they add far less compute than training any frontier model, yet Sakana's claimed scores position the system above models from OpenAI, Anthropic, and Google on the security-specific tasks they were measured against. If that holds under independent evaluation, it suggests that coordination architecture may offer a leverage point that does not require the capital base of a frontier lab.

There is also the sovereignty dimension that motivated Fugu's original June 22 launch, ten days after the US Department of Commerce suspended international access to Anthropic's Fable 5 and Mythos Preview under an export control directive. Because Fugu-Cyber coordinates underlying frontier models rather than being one itself, organizations with specific vendor or compliance constraints can configure which models participate in the agent pool — opting out of providers that do not meet their data residency or supply-chain requirements, or substituting open-weight models like Nemotron 3 Ultra that can be self-hosted .

The structural limit of this argument has not changed since the base Fugu launch: Fugu does not own the frontier model weights it orchestrates. It rents access via the same US-based APIs that are subject to the export controls it was designed to route around. A policy action broad enough to restrict multiple providers simultaneously would substantially degrade the pool's capability. Resilience comes from pool diversity — including open-weight members — not from independence from the US API ecosystem as a whole.

Fugu-Cyber is a multi-agent orchestration system — not a new language model. It presents as a single API endpoint while routing security tasks across a pool of specialized frontier models underneath, with a coordinator trained to assign Thinker, Worker, and Verifier roles dynamically. The pool includes Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5, among others. What distinguishes it from single-model security tools is the verification step: an intermediate finding is challenged by a second agent before synthesis, which Sakana argues reduces false positives for multi-step vulnerability analysis. It is also notable for what it lacks: a training corpus of proprietary security data. The security capability comes from orchestrating existing frontier models, not from specialized pre-training.

No, not at publication. Every figure in Sakana's benchmark table — 86.9% on CyberGym, 72.1% on CTI-REALM, and the comparison scores for GPT-5.5-Cyber and Mythos-Preview — is vendor-reported. CyberGym's original paper, published by UC Berkeley at ICLR 2026, found top model combinations clearing roughly 20%. The gap between that baseline and Sakana's claimed 86.9% requires an explanation that the company has not provided: which task variant was used, how many trials were run, what agent scaffold was applied, and what the confidence interval is. Those omissions are not unique to Sakana — vendor benchmark tables routinely omit this information — but they mean that any security team treating the score as evidence of production capability before independent reproduction is making a larger inferential leap than the data supports.

Not under the terms of access. Sakana's updated acceptable use policy explicitly prohibits unauthorized access to or alteration and destruction of third-party systems, networks, data, or assets, and access requires a manual review process in which applicants submit their intended use case. The policy is a contractual control, not a technical one. A system capable of 86.9% proof-of-concept generation on a vulnerability benchmark has obvious offensive applications, and the gating structure exists precisely because Sakana recognizes that. Security teams using Fugu-Cyber in authorized red team or penetration testing work remain within the policy's scope; using it against systems they do not own does not.

Fugu's routing architecture is opaque: when a request arrives, the orchestrator decides internally which models from its pool to involve, and the caller is not told which providers processed the data. Under GDPR Article 28, operators processing personal data through sub-processors must document those relationships and ensure adequate contractual safeguards. Fugu's dynamic, undisclosed routing makes that documentation requirement structurally difficult to satisfy. Sakana has confirmed it is working toward GDPR compliance to unlock EU, EEA, UK, and Switzerland access, but has not announced a timeline. For European security teams, this is a hard blocker regardless of benchmark performance.