Gatekeepers and AI: How Paywalls Poison the Open Web

Note from the Author:

By the time I landed with BBN out of the 10 Moulton St office in Cambridge (yes, I’m dating myself here, kids), the ARPANET era was already history-acronyms flipping between ARPA, DARPA, and back again depending on who was writing the checks. The spirit of the open web was alive and well, but even then, you could already see the corporate types with dollar signs in their eyes circling, eager to carve up the net for profit.

I saw firsthand what happens when gatekeepers start appearing and try to wall off progress, making decisions for everyone else. Today, as Cloudflare and AI collide with the rotting husk of the Internets advertising model, it’s déjà vu, except the stakes are far higher.

The Latest Battle for the Open Web

Cloudflare, now managing around 20% of the world’s websites, has put up a new wall: blocking AI crawlers by default unless websites opt in or bots pay via “Pay‑Per‑Crawl.” Officially, it’s about protecting publishers’ copyright and revenue. But beneath the spin lies something deeper-and more strategic.

Two Real Reasons Behind the Move

Cloudflare’s Pay-Per-Crawl and the New AI Gatekeepers

Cloudflare’s Gatekeeper Ambitions

OpenAI bots crawl content at about 1,700:1 compared to human traffic; Anthropic hits a staggering 73,000:1. That data doesn’t come cheap. Cloudflare wants to monetize it-setting themselves up like gatekeepers for their piece of the web. But unlike electric companies, they don’t maintain all the infrastructure. They’re putting up tollbooths only where their customers want them.

The Broken Ad Model Exposed

AI bots don’t click ads-so every scrape drains those last ad pennies. But let’s face it: the current ad model is trash. Pop-ups, auto-play, endless recommendation lists-it’s driving people away. And now AI is exposing the deep underbelly of the system, laying bare how little value users actually get.

What’s at Stake?

This is about more than news. TV is fading. Newspapers are dying. The internet is quickly becoming the central exchange for news, public service bulletins, and even state/local data.
So are we locking the public out of taxpayer-funded information? It’s a real risk. There’s no proof Cloudflare is doing this yet, but with Pay‑Per‑Crawl rolling out, someone needs to start watching.
Public data paywall: need to monitor and trace.

Historic Echoes: Internet Taxes & Tollbooth Gatekeepers

In the early ’90s, only governments or universities could register domains. I grabbed mine once “regular” people were allowed-after convincing my wife it was ‘critical’ and I had the money.
Then came gatekeepers: each attempting their own toll. Outrage stopped some, but not all. The script hasn’t changed: the guise of “protection” always gives way to monetization.

Why the Ad Model Must Die (Or Evolve)

The web is a UX wasteland-all so a broken yet lucrative ad machine can keep going. AI is now pulling back the curtain, showing users the sausage-making behind the scenes. Mega-corporations are serving ad sludge, and people are done with it.
It’s time to rethink: pay-to-use, membership models, open licensing-or something we haven’t even invented.

The Risks: Fragmentation & “AI Content Ghettos”

Paywalls haven’t saved the news. Audiences won’t pay for 90% of formerly free content. The outcome? Spy-gated premium content and a mass of AI-generated junk. If paywalls become the norm, we’ll be splitting the web-not saving it.

What Comes Next?

• Micro-payments? Membership-based publishing? Fair AI licensing?
• Is there a middle path that’s better than free-for-all or a total lock-down?
• Who gets to decide: Big Tech, publishers, or users and creators?

Conclusion: What Legacy Will We Leave?

When AI first hit, many people shrugged-including me. I left tech before AI became mainstream-burned out by corporate control and the grind of life on the road. I even poo-pooed AI. But stepping away-and eventually stepping back-I now see AI for what it really is: a reveal tool, one that’s pulling back the shades and showing how mega-corporations treat regular users.
AI can be a force for good-or for the worst. It all rides on who’s holding the lever.

Call to Action: This Debate Needs You

This isn’t just a tussle between Cloudflare, AI startups, and publishers-it’s about who controls future access to knowledge in this modern public square.
Engineers, creators, educators, or anyone who remembers a freer web-your voices matter here.
Comment, share, and tag people who need to weigh in. Let’s make sure the next web is shaped by us, not just the deep pockets.

Additional Sources

theverge.com

computerweekly.com

wired.com

reuters.com

businessinsider.com

securitysenses.com

cloudflare.com

AI was used for assistance in this article and accompanying image