Rob's Notes 24: Closing the Castle Gates

After sharing crawl-to-refer data, Cloudflare presents its new "pay per crawl" scheme

On July 1st 2025, Cloudflare (a content-delivery network sitting in front of roughly a fifth of the web) published some fascinating data showing the ratio of HTML page crawl requests to HTML page referrals by platform. Google, for example, crawls 9.4 pages for every referral it sends to a publisher, whereas OpenAI is consuming but not sending much traffic back - it crawls 1,600 pages for every referral it sends.

Then today I saw Bill Ready’s post (he’s the CEO of Pinterest).

For years the internet’s grandees have presided over what Yanis Varoufakis brands a realm of “cloud fiefdoms”, where platform owners exact rents and users labour for likes and pennies. Media scholar Nathan Schneider goes further, noting that most online forums whether a Discord server or a subreddit, default to “implicit feudalism”, (h/t to Jonathan Bellack) in which unelected moderators rule by fiat and disgruntled peasants can only decamp elsewhere.

Enter the data-harvesters

Generative-AI firms have reinforced this order. Their giant models were trained on billions of web pages, scraped with scant regard for copyright or recompense. Bill Ready likens the practice to the file-sharing free-for-all of the Napster era. “We have not yet seen the iTunes or Spotify model for AI,” he laments; creators receive neither permission request nor payment.

A modest charter

Cloudflare quietly shifted the balance. Its new policy blocks known AI crawlers by default and, in a pilot “Pay Per Crawl” scheme, invites publishers to post tariff schedules for would-be scrapers. The drawbridge is now raised unless the bot bearing down on it comes bearing gold - or at least a signed license.

The serfs’ first leverage

The measure is hardly a revolution, but it arms smaller sites with an enforcement tool they could never afford to build themselves. If replicated by rival CDNs and hosting firms, the default of “access first, apologize later” could be replaced by contractual formality. That, in turn, might nudge AI firms towards a Spotify-style marketplace in which every crawl carries a micro-royalty.

Barons still abound

Do not pop the mead just yet. Cloudflare protects only those sites within its walls; rogue scrapers can spoof their identity; and, unless creators band together, asymmetrical bargaining will keep per-URL fees paltry. Yet even a symbolic toll matters. In medieval Europe, the first cash rents loosened the manorial grip and seeded more plural governance. By forcing AI barons to knock (and occasionally to pay) Cloudflare has chipped one more stone from the digital castle’s gatehouse.

Should the practice spread, today’s online share-croppers may find themselves negotiating, for once, on something closer to equal terms.