Cloudflare Outage 2025: How One Database Permission Change Broke the Internet

On 18 November 2025 at 11:20 UTC, Cloudflare started returning error pages for a large slice of the internet. ChatGPT, X and Spotify were among the sites people noticed. It was not an attack. It was a configuration file that grew bigger than the software expected.

The setup: a file the proxy trusts

Cloudflare's Bot Management system scores every request. The scoring model reads a feature file, a list of signals it should look at. A query on a ClickHouse database cluster regenerates that file every five minutes, and the new copy is pushed to every machine in the network within moments.

Two details matter:

• The proxy software has a hard limit on how many features it will load. It preallocates memory for them, and the limit sits comfortably above the normal count.
• The file is treated as trusted internal data, not as input to be validated.

The trigger: a permissions change

Cloudflare was rolling out a change to ClickHouse permissions so that queries could see more of the underlying tables. The feature-file query did not filter by database name. After the change it started returning each column twice, once per database it could now see.

The file doubled in size. It now went past the proxy's limit.

Why it was so confusing

The proxy handled the oversized file by failing the request, and the bot module crashed. That would be bad but easy to spot. What made it hard was this: the permissions change was rolled out gradually. Every five minutes the query ran on a random node. If the node was already updated, the file came out bad. If not, it came out good.

So the whole network flipped between healthy and broken every few minutes. That looks exactly like a flaky, large-scale DDoS, and that is what the team suspected first. Eventually every node was updated and the failure stopped flickering and stayed on.

The fix

Around 14:30 UTC the team stopped the bad file from being generated and pushed, put a known-good older file into the distribution queue, and restarted the core proxy. Traffic was largely back by 14:30 and everything was normal by 17:06. Cloudflare's own post-mortem has the exact breakdown of which products were hit.

The code pattern behind it

The failure mode is common to many languages: a limit that is checked, with an error that nobody planned for.

// Illustrative, not Cloudflare's code: the shape of the problem.
const MAX_FEATURES: usize = 200;

fn load(features: Vec<String>) -> Result<Vec<String>, String> {
    if features.len() > MAX_FEATURES {
        return Err(format!("{} features, limit is {}", features.len(), MAX_FEATURES));
    }
    Ok(features)
}

// Crash the whole proxy on a bad config...
let cfg = load(file).unwrap();

// ...or keep serving with the last config that worked.
let cfg = load(file).unwrap_or_else(|_| last_good_config());

What to take away

1. Treat internal config like user input. Validate size, shape and sanity before it goes live, even when your own system produced it.
2. Fail safe, not fail closed. If a new config is bad, keep running on the last good one and raise an alert.
3. Roll out config the way you roll out code. A global push every five minutes with no canary is a global blast radius.
4. Have a kill switch. The recovery here was "stop the file, put the old one back". The faster you can do that, the shorter the outage.
5. Be careful with the first theory. An on-off pattern looked like an attack. It was a half-finished rollout.