The end of ‘AI scraping free-for-all’? Cloudflare to block AI crawlers by default, offer pay-per-crawl service
The digital infrastructure company Cloudflare will now block AI crawlers by default. The company is also testing a pay-per-crawl service for website hosts. (Illustration via Cloudflare)
July 2, 2025 – Cloudflare, the global internet infrastructure company, yesterday announced that it will default to blocking AI crawlers from accessing content on its client websites.
The company said it will also give clients the ability to manually allow or ban these AI bots on a case-by-case basis, and it will introduce a “pay-per-crawl” service clients can use to receive compensation every time an AI bot wants to scoop up their website’s contents.
Website owners working with Cloudflare can choose if they want AI crawlers to access their content, and decide how AI companies can use it. AI companies can also now clearly state their purpose—if their crawlers are used for training, inference, or search—to help website owners decide which crawlers to allow. Cloudflare is the first major internet infrastructure company to offer the default setting.
end of ‘ai scraping free-for-all’
Wired took Cloudflare’s announcement as a significant move in the conflict between data scrapers, AI developers, and content creators: “The age of the AI scraping free-for-all may be coming to an end,” they wrote.
ZDNet headlined its story “Cloudflare just changed the internet, and it’s bad news for the AI giants.”
One day after Cloudflare’s announcements, tech consultants were posting on LinkedIn about pay-per-crawl’s potential to turn “invisible bot traffic into a visible revenue stream.”
Cloudflare’s protection-by-default and the promise of pay-per-crawl is a big deal in part because Cloudflare powers one of the world’s largest networks, helping to manage and protect traffic for 20% of the web.
MIT Technology Review offered this explainer:
The bots in question are a type of web crawler, an algorithm that walks across the internet to digest and catalogue online information on each website. In the past, web crawlers were most commonly associated with gathering data for search engines, but developers now use them to gather data they need to build and use AI systems.
However, such systems don’t provide the same opportunities for monetization and credit as search engines historically have. AI models draw from a great deal of data on the web to generate their outputs, but these data sources are often not credited, limiting the creators’ ability to make money from their work. Search engines that feature AI-generated answers may include links to original sources, but they may also reduce people’s interest in clicking through to other sites and could even usher in a “zero-click” future.
protecting content creators
In the company’s announcement, Cloudflare officials wrote:
For decades, the Internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators that produce quality content with money and a following, while helping users discover new and relevant information.
That model is now broken. AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source—depriving content creators of revenue, and the satisfaction of knowing someone is viewing their content. If the incentive to create original, quality content disappears, society ends up losing, and the future of the Internet is at risk.
“If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone—creators, consumers, tomorrow’s AI founders, and the future of the web itself,” said Matthew Prince, co-founder and CEO of Cloudflare. “Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.”
a real defense against ai crawlers
ZDNet contributor Steven Vaughan-Nichols offered his own experience with AI crawlers and the problem Cloudflare is working to address:
The change addresses a real problem. My own small site, where I track all my stories, Practical Technology, has been slowed dramatically at times by AI crawlers. It's not just me. Numerous website owners have reported that AI crawlers, such as OpenAI's GPTBot and Anthropic's ClaudeBot, generate massive volumes of automated requests that clog up websites so they're as slow as sludge. GoogleBot alone reports that the cloud-hosting service Vercel bombards the sites it hosts with over 4.5 billion requests a month.
These AI bots often crawl sites far more aggressively than traditional search engine crawlers. They sometimes revisit the same pages every few hours or even hit sites with hundreds of requests per second. While the AI companies deny that their bots are to blame, the evidence tells a different story.
Neil Vogel, CEO of Dotdash Meredith, offered this quote as part of Cloudflare’s announcement: “We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements. We're proud to support Cloudflare and look forward to using their tools to protect our content and the open web.”
default blocking now, pay-per-crawl coming soon
Cloudflare officials said the default blocking of AI crawlers would be effective immediately. The company’s pay-per-crawl service is currently in private beta.
How it works: Pay-per-crawl integrates with existing web infrastructure, leveraging HTTP status codes and established authentication mechanisms to create a framework for paid content access. Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing. Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure.
Pay per crawl grants domain owners full control over their monetization strategy. They can define a flat, per-request price across their entire site. Publishers will then have three distinct options for a crawler:
Allow: Grant the crawler free access to content.
Charge: Require payment at the configured, domain-wide price.
Block: Deny access entirely, with no option to pay.
Interested website hosts may sign up for the beta version of Cloudflare’s pay-per-crawl service here.
