04 August 2025

Cloudflare is building an App Store for the web. That’s a bit scary, isn’t it?

In summary

• Cloudflare now allows publishers to block or charge AI bots crawling their content, a technical fix with strategic consequences.
• It helps publishers regain control and value, but also signals Cloudflare’s shift from helper to potential gatekeeper of the open web.
• The open internet is quietly becoming pay-to-play. Understanding this shift is key to protecting your content, visibility, and long-term digital strategy.

A win for publishers, a warning for the web

Cloudflare has long been the internet’s quiet helper, speeding up websites, defending against bots, and keeping publishers online.

But its latest move is something else entirely. It now allows publishers to block or charge AI bots that crawl their content, a technical fix with strategic consequences.

On the surface, it’s a win for independent publishers who’ve been powerless against content-scraping AI models. But zoom out, and a bigger shift comes into focus. Cloudflare isn’t just optimising access anymore, it’s starting to mediate it.

Think of it like this: Apple and Google built App Stores that control what apps make it to your device, and who profits along the way. Cloudflare is setting up to do the same, not for apps, but for the open web.

That’s great news for publishers today. But tomorrow? It might mean every API call, AI crawler, and page view comes with a tollbooth.

So, what is Cloudflare?

While it began as a content delivery network (CDN) designed to optimise website performance, its role has expanded significantly. Today, Cloudflare provides a full-stack layer of internet infrastructure: bot management, web security, firewall services, anti-Distributed Denial-of-Service (DDoS) protection, and most recently, AI access control.

It currently sits in front of 19.6% of all websites globally and commands over 60% of the CDN market share, a level of reach that gives it outsized influence over how online content is delivered, protected, and increasingly, accessed.

In other words, Cloudflare is no longer just making the web faster, it’s deciding who gets through.

And that shift, from acceleration to mediation, is what makes its latest AI crawler feature so significant. It’s not just about blocking bots. It’s about shaping the terms of access. Who pays. Who gets priority. Who gets blocked. If that sounds like an App Store for the open web, that’s because it’s starting to function like one.

Why this is (still) a net positive for publishers

At face value, this feature is incredibly useful, particularly for publishers. AI crawlers like those from OpenAI, Anthropic, and Google have been voracious in their appetite for content. Not only do they bypass standard robot.txt protocols and ignore consent mechanisms, but they extract immense volumes of data to train their models… with little reciprocity.

To illustrate the imbalance:

• OpenAI bots make roughly 1,500 crawl requests for every visitor they send back.
• Anthropic is even more extreme, 60,000 requests for every referral.
• Even Googlebot, once the friendliest of robots, now has a crawl-to-click ratio of 18:1, up from 6:1 just six months ago, and a far cry from 2:1 a decade ago.

This isn’t just inefficient, it’s actively undermining the economics of digital publishing. Ad revenue dries up if no one visits your site.

So when Cloudflare says, “Would you like to block that traffic, or charge for it?” It’s a welcome change. For once, publishers can decide who sees their content and under what terms.

Better still, there’s an environmental upside. Less unnecessary AI crawling means fewer server requests and reduced energy use, helpful in an industry not exactly known for its carbon restraint.

But let’s talk about trajectory

Zoom out, and Cloudflare’s direction of travel starts to resemble a familiar blueprint, one perfected by Big Tech.

Right now, Cloudflare acts as a high-performance intermediary: fast, secure, and mostly invisible. But as it layers in monetisation, identity enforcement, and content gating, it’s evolving into something else entirely, an App Store for the open web.

The analogy isn’t just rhetorical. Apple and Google sit between developers and users, clipping the ticket on every app and in-app purchase. Cloudflare is setting itself up to do the same, not for apps, but for everything. Every page view, every API call, every AI crawler could soon be mediated, metered, and monetised.

The strategy is familiar: offer critical infrastructure for free (or very competitively priced), build trust and dependence at scale, and then introduce the tollbooth. It’s how Apple’s App Store became less about access and more about rent-seeking. Cloudflare’s AI bot charging tool? That’s the first visible tollgate. But it likely won’t be the last.

If Cloudflare becomes the entity that decides which bots are “good,” which ones pay, and which get blocked, the open web starts to look less like a free exchange of information, and more like a walled garden with a checkout counter.

Should we be worried?

In the short term, no. Cloudflare is solving a real, immediate problem. And they’re doing it with technically elegant solutions that benefits content creators, particularly the independent ones who’ve had little recourse against free-riding AI models.

But in the longer term? We should watch this closely.

The internet was designed to be open, interoperable, decentralised, and, dare I say it, a little idealistic. Every time a single entity begins to insert itself as the arbiter of access, we need to ask, when does protection become control? When does convenience become rent-seeking?

Cloudflare isn’t evil. They’re clever. They’ve made genuinely good infrastructure. But history tells us that when platforms gain scale, monetisation follows, and so does lock-in.

Just ask anyone trying to launch an app without paying Apple’s 30%.

Get in touch

Get in touch with Louder to discuss how we can support your business objectives and sign up to our newsletter to receive the latest industry updates straight in your inbox.