Share with others:
Artificial intelligence will no longer browse websites for free
Cloudflare, a key internet infrastructure provider, will now block known AI crawlers by default to prevent them from “accessing content without permission or compensation.” Cloudflare will also ask new domain owners when they set up their domain if they want to allow access to AI, and is even introducing the option for some publishers to charge for access via a “Pay Per Crawl” model.
This model will allow publishers and creators to set the price that AI companies will have to pay to access their content. AI companies will then be able to review the price list and decide whether to pay and gain access or opt out. For now, the program is only available to a select group of the largest publishers and content creators, but Cloudflar says it will ensure that AI companies are using quality content “in the right way, with permission and appropriate compensation.”
Cloudflare has been helping domain owners fight against UI spiders for some time. Back in 2023, they allowed websites to block UI spiders, but only those that comply with robots.txt, a document that is not legally binding, but only indicates what a bot can visit. Later, they expanded the blocking option to all UI bots, regardless of whether they comply with robots.txt or not. Now, this setting is enabled by default for all new Cloudflare users. Cloudflare does not identify bots from robots.txt, but identifies them with its own list of known UI bots.
In March, they additionally introduced a feature that directs unwanted web crawlers into the so-called "UI Labyrinth", where they become trapped and cannot effectively collect content without permission.
Major websites and publishers already supporting Cloudflare’s new UI crawler limits include The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora. Websites are facing a future where more and more users are searching for information through UI chatbots rather than traditional search engines.
“Over the last six months, people have become more and more trusting of artificial intelligence, which means they are no longer reading original content,” Cloudflare CEO Matthew Prince said at the Axios Live event.
Cloudflare is also working with AI companies to verify and certify their crawlers and enable them to clearly state their purpose, such as whether they are collecting content for learning, search, or inference. All this with the aim that website owners can then verify and decide for themselves which bots to allow access to their content.
“Original content is what makes the internet one of the greatest inventions of the last century, and we must protect it together,” Prince said in a press release. “Until now, UI crawlers have been collecting content without limits. Our goal is to put the power back in the hands of creators, while still allowing UI companies to innovate.”
