Anthropic Warns Small AI Models Could Reach Self

Anthropic's security research team, the Frontier Red Team, published findings on its official blog on October 12 warning that small AI models could reach the capability to self-replicate on inexpensive IoT devices within months, without relying on data centers. Commercial devices such as smart ovens and budget Android TV boxes could serve as footholds for autonomous AI-driven replication and propagation.

According to the research team, a 9-billion-parameter (9B) model was able to run inference on eight of the nine hardware types tested. Models with 30 billion parameters (30B) or fewer can already execute many of the individual steps required for self-replication, though errors emerge when chaining the full process end-to-end.

In a simplified evaluation environment, models with performance at or above GLM-5.2 completed the entire process from initial infection to replication autonomously. On an eight-node test network built on Google Cloud Platform (GCP), GLM-5.2 successfully spread to all nodes in four out of five attempts, reaching seven nodes in the remaining attempt. However, the research team positioned these results as "an upper bound of what could occur in real-world conditions," noting that the cyberattack conditions were simplified—for example, no authentication was required on the communication ports serving as entry points.

If small models continue improving at their current pace, they are expected to reach this threshold by the end of 2026.

The Realism of Distributed Botnets

In a demonstration, the team connected 15 budget Android TV boxes—each costing approximately $55 ( ¥8,700)—to run Gemma4-26B-A4B in a distributed configuration. Processing speed was limited to 0.39 tokens per second. If a network comparable in scale to the Kimwolf botnet's 2 million-plus infected devices were built, it could theoretically operate approximately 133,000 replicas, with combined processing power equivalent to roughly 40 NVIDIA H100 GPUs.

Propagation speed is slow, requiring several days for the number of replicas to double. However, the research team notes that because there is no central command point to target, completely shutting down a replicated network once it has spread is extremely difficult.

The research focuses on a concept called "Autonomous Replication and Adaptation (ARA)"—the ability of models to deploy themselves beyond human control, adapting to their environment as they propagate. Unlike traditional malware, AI-embedded replication programs can independently figure out how to operate on unfamiliar hardware.

For manufacturers, the team urges a transition to secure default configurations and regulatory measures to raise baseline security standards.

Anthropic states that no fully uncontrolled autonomously replicating program has been confirmed to date. However, the company points out that AI agents are steadily mastering each step in the self-replication chain and are increasingly capable of passing acquired adaptive capabilities to subsequent replicas.

Computing resources available for model inference—both inside and outside data centers—continue to grow. While the exact form autonomous AI replication will take remains uncertain, Anthropic envisions multiple scenarios: from a slow, distributed "digital slime mold" that parasitizes poorly secured devices, to powerful attack models that rapidly breach data centers, or models that deceive humans into surrendering computing resource allocations.

Lightweight agents are likely to devote more effort than human attackers to compromising the vast numbers of IoT devices scattered worldwide. An increase in slow self-replicating programs could be a harbinger of more aggressive, faster-propagating replication programs approaching—programs potentially capable of breaching well-defended machines or efficiently stealing API keys.

Anthropic calls on the industry to strengthen systems to block the propagation of such self-replicating programs as much as possible, while also recommending preparation for a world in which they persist over the long term.