Cloudflare, a key internet infrastructure provider, will now block known AI crawlers by default to prevent them from \u201caccessing content without permission or compensation.\u201d Cloudflare will also ask new domain owners when they set up their domain if they want to allow access to AI, and is even introducing the option for some publishers to charge for access via a \u201cPay Per Crawl\u201d model.<\/p>\n\n\n\n
This model will allow publishers and creators to set the price that AI companies will have to pay to access their content. AI companies will then be able to review the price list and decide whether to pay and gain access or opt out. For now, the program is only available to a select group of the largest publishers and content creators, but Cloudflar says it will ensure that AI companies are using quality content \u201cin the right way, with permission and appropriate compensation.\u201d<\/p>\n\n\n\n
Cloudflare has been helping domain owners fight against UI spiders for some time. Back in 2023, they allowed websites to block UI spiders, but only those that comply with robots.txt, a document that is not legally binding, but only indicates what a bot can visit. Later, they expanded the blocking option to all UI bots, regardless of whether they comply with robots.txt or not. Now, this setting is enabled by default for all new Cloudflare users. Cloudflare does not identify bots from robots.txt, but identifies them with its own list of known UI bots.<\/p>\n\n\n\n
In March, they additionally introduced a feature that directs unwanted web crawlers into the so-called "UI Labyrinth", where they become trapped and cannot effectively collect content without permission.<\/p>\n\n\n\n
Major websites and publishers already supporting Cloudflare\u2019s new UI crawler limits include The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora. Websites are facing a future where more and more users are searching for information through UI chatbots rather than traditional search engines.<\/p>\n\n\n\n
\u201cOver the last six months, people have become more and more trusting of artificial intelligence, which means they are no longer reading original content,\u201d Cloudflare CEO Matthew Prince said at the Axios Live event.<\/p>\n\n\n\n
Cloudflare is also working with AI companies to verify and certify their crawlers and enable them to clearly state their purpose, such as whether they are collecting content for learning, search, or inference. All this with the aim that website owners can then verify and decide for themselves which bots to allow access to their content.<\/p>\n\n\n\n
\u201cOriginal content is what makes the internet one of the greatest inventions of the last century, and we must protect it together,\u201d Prince said in a press release. \u201cUntil now, UI crawlers have been collecting content without limits. Our goal is to put the power back in the hands of creators, while still allowing UI companies to innovate.\u201d<\/p>","protected":false},"excerpt":{"rendered":"
Cloudflare, eden klju\u010dnih ponudnikov internetne infrastrukture, bo odslej privzeto blokiral znane UI pajke, da bi prepre\u010dil, da bi ti \u00bbdostopali do vsebin brez dovoljenja ali nadomestila\u00ab. Ob tem bo Cloudflare nove lastnike domen \u017ee ob vzpostavitvi vpra\u0161al, ali \u017eelijo dovoliti dostop umetni inteligenci, za nekatere zalo\u017enike pa uvaja celo mo\u017enost zara\u010dunavanja dostopa prek modela \u00bbPay […]<\/p>","protected":false},"author":2,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[66,4],"tags":[192],"class_list":["post-6387","post","type-post","status-publish","format-standard","hentry","category-programi","category-racunalnistvo-telefonija","tag-umetna-inteligenca"],"acf":{"subtitle":"","heading":"","summary":"","thumbnail_small":"https:\/\/racunalniske-novice.com\/wp-content\/uploads\/2025\/07\/cloudflare-naslovna-560x315.png","thumbnail_large":"https:\/\/racunalniske-novice.com\/wp-content\/uploads\/2025\/07\/cloudflare-naslovna-1024x706.png","thumbnail_caption":"","gallery":"","video_gallery":null,"author":"","links":null,"sources":null,"skip_language":[]},"yoast_head":"\n