Cloudflare CEO Matthew Prince has explained the causes of the outage of the CDN giant, which brought down large chunks of the web on Tuesday.
In the opening paragraph of a blog post, he gives an assurance that the outage was not the result of a cyberattack, as had initially been suspected. Instead, it was triggered by a “change to one of our database systems' permissions”.
This change caused the ClickHouse OLAP database to output multiple entries into a ‘feature file’ used for bot detection by the Bot Management system. The engorged feature file was then propagated across the network, its unexpected size causing machines to fall over, leading to HTTP 5xx errors and service failures.
Effectively, then, the company had DDoS’d itself. The oversized feature file was being generated every five minutes by machines in a ClickHouse database cluster, which was in the process of being upgraded. Machines that had not been upgraded did not generate the file, and were mostly able to recover, while updated machines fell over, giving the appearance of a malicious DDoS attack, Prince said. This delayed the discovery of the real source of the problem.
Overall, Cloudflare was down from11:20 UTC to 17:06 UTC.
The error resulted in a prolonged outage for numerous websites, including major sites such as Spotify, ChatGPT and X.
Affected Cloudflare services included the Core CDN, Turnstile (A CAPTCHA alternative), the caching system Workers KV, customer dashboards, email security systems and access management.
Workers KV and access were partially mitigated by bypassing the core proxy, while the Bot Management feature file was rolled back to a version to restore full functionality.
Prince apologised for the outage, Cloudflare's most widespread and prolonged since 2019, promising to harden the configuration file ingestion process, enable global kill switches and review failure modes across core systems.
Following serious system failures in AWS, CrowdStrike and Microsoft Azure, the latest outage at Cloudflare - which in 2022 was used by an estimated 20% of global websites and a third of top sites - have focused attention yet again on the global economy’s dependence on a few US-based tech companies.
Tim Wright, technology partner at law firm Fladgate, said: “The recent Cloudflare outage is a stark reminder of the systemic risks underpinning today’s digital economy. When a limited number of hyperscalers underpin critical services, a single failure can ripple across industries, exposing businesses to financial and reputational damage. The parallels with last month’s AWS disruption are clear - interconnectivity is a double-edged sword. Companies must urgently reassess their contracts with a focus on cloud exit rights, redundancy and outage notification obligations to manage these risks effectively.”
It remains to be seen whether the UK government’s Cyber Security and Resilience Bill , introduced to Parliament last week, will bring major providers like Cloudflare into regulatory scope for data protection and operational resilience for the first time.
The full story
This article is one source in a clustered incident — the cluster page carries the summary, timeline and every other outlet covering it.
