Should You Allow AI Bots to Crawl Your Site?
Companies with enough leverage can negotiate substantial fees to grant AI bots access to their content.
Part 2 of Bill Sparks' series on how AI is becoming the new middleman between brands and customers. This installment looks at how a company’s business model and content strategy determines how its relationship with AI crawlers should be managed.
Should You Allow AI Bots to Crawl Your Site?
On September 15, 2026, Cloudflare started blocking mixed-use AI crawlers by default on every ad-supported page behind its network. There was no vote and no opt-in. The setting flipped for millions of websites at once. Cloudflare handles roughly 20 percent of all web traffic, which means one infrastructure company quietly changed the terms of the internet for a significant share of publishers overnight. For many, it was the first time they realized their AI crawling policy had been set not by their own decision, but by a platform default.
That was not arbitrary. It was a response to an exchange that had become very one-sided.
The lopsided exchange
Cloudflare published its crawl-to-referral ratios earlier this year, and the numbers tell the story. For every visitor Google sends back to a website, it crawls about 14 pages. That is roughly the arrangement publishers have lived with for two decades: you let the crawler in, it sends you traffic. OpenAI crawls about 1,700 pages for every visitor it returns. Anthropic crawls about 73,000. The AI companies are reading everything and sending back almost nothing.
Publishers noticed. A BuzzStream analysis found that 79 percent of top news sites now block at least one AI training crawler, with OpenAI's GPTBot the most frequently targeted. But the blocking wave has not been clean. More than half of AI crawl traffic is wasted re-fetching pages that have not changed, according to Cloudflare's own data. And the distinction between crawlers that train models and those that retrieve content for search results has become a policy headache, because the same bot often does both.
The financial response has been equally tangled. Press Gazette's tracker counts 47 disclosed licensing agreements between publishers and AI platforms alongside 29 active lawsuits, with about $439 million in disclosed deal value. News Corp signed a deal with OpenAI reported at $250 million over five years while simultaneously suing Perplexity and Brave for using its content without permission. The New York Times is suing OpenAI while licensing content to Amazon. The same publishers are building toll booths with one hand and filing complaints with the other.
That contradiction makes strategic sense. Publishers that blocked AI crawlers first created the scarcity that made licensing deals valuable. Cloudflare is now pushing further with a program called Pay Per Use. Instead of charging AI companies per page crawled, it aims to pay publishers when their content actually appears in an AI answer or creates measurable value. The model is still in early stages, but the direction is unmistakable: from free access, to paid crawling, to payment tied to use.
The same fight, with shopping carts
The content-crawling battle is also underway in commerce. Amazon sued Perplexity, alleging that its Comet browser sent an AI agent to shop on Amazon using real customer accounts. The agent logged in with the user's credentials, browsed products, and moved toward checkout, all without Amazon's authorization. Amazon's complaint said Comet disguised itself as a regular Chrome browser and refused to stop when asked.
For some companies, the best strategy may be the fortress approach. No bots allowed under any circumstances.
In March 2026, a federal judge granted Amazon an injunction blocking Comet from its platform. The court found that the agent entered accounts with the user's permission but without Amazon's, and that was enough. Five months later, the Ninth Circuit vacated the injunction, ruling that the user, not Perplexity, was the one accessing Amazon. The case continues at the district level. It has a personal wrinkle: Jeff Bezos is an investor in Perplexity.
The legal question remains unsettled. The commercial stakes are already clear. When an AI agent goes straight to a product and completes a purchase, every sponsored listing between the search and the checkout disappears. Amazon has built one of the largest advertising businesses in the world on the premise that shoppers browse before they buy. An agent that skips the browsing skips the ads. Amazon now requires all AI agents to identify themselves on its services, effective March 4, and separately blocked ChatGPT from shopping on its platform.
This is the content-crawling fight translated into commerce, with advertising revenue at stake instead of intellectual property. The question in both cases is the same: do the AI companies get to act inside your space on their terms, or on yours?
The real dilemma
Here is the tension at the center of this series. Part 1 showed that being cited by AI engines drives smaller but meaningfully higher-converting traffic. Being cited requires that the AI can read your content. Block the crawlers and you protect your content's value. You may also vanish from the answers your customers see.
If your content exists solely to drive sales, it’s best to take an open house approach to AI bots.
The data on that tradeoff is still early, but it points in a consistent direction. A study by researchers at Wharton and Rutgers found that websites blocking AI crawlers lost about 7 percent of their weekly traffic within six weeks. The same researchers found that 39.5 percent of GPTBot blocking directives were simply not honored, meaning the robots.txt file that was supposed to keep the bots out failed nearly four times out of ten. And a BuzzStream analysis of major news sites found that 70.6 percent of publishers who blocked AI crawlers were still cited by AI engines anyway, because the engines found the same information through other sources. Blocking is leaky in both directions. It does not fully keep the bots out, and it does not fully keep you out of the answers.
Those numbers frame three postures a company can take.
The first is the fortress: block the crawlers entirely and license your content selectively. This suits publishers and research firms whose content is the product, not a means to sell something else. The Wall Street Journal does not exist to promote Dow Jones's other services. Its journalism has standalone value worth protecting and licensing on its own terms.
The second is the toll booth: allow crawling but charge for it, through Cloudflare's marketplace or through direct licensing deals. This requires enough content volume and enough leverage to negotiate. A niche industry publication with proprietary data might have that leverage. A corporate blog probably does not.
The third is the open door: let the bots in and treat AI visibility as free marketing. For most brands whose content exists to drive sales rather than to be sold directly, this is the rational choice. You published that blog post to attract customers, not to license it to a language model. Blocking the crawler that might cite it in an AI answer works against the very reason you wrote it.
The right answer depends on what your content is for, and the answer may differ across your own site. Your product pages, your thought leadership, and your proprietary research may each deserve different rules. A company that treats this as one blanket decision is probably making the wrong simplification.
What you can do now
Start by finding out what your website currently allows. Most companies have an AI crawling policy that was set by a vendor default or an IT configuration rather than by a deliberate business decision. Check your robots.txt file. Ask whether your CDN or hosting provider has made a choice on your behalf, as Cloudflare now has for millions of sites. If your policy was set by someone else's default, it is not your policy.
Then make this a business decision, not an IT setting. Whether AI engines can read your content affects your marketing visibility, your intellectual property, and potentially your revenue. That question belongs in the room where you make brand and distribution decisions, not buried in a server configuration.
Finally, separate the bots that read your content from the agents that act for your customers. These are different relationships with different risk profiles. A search crawler that reads your blog and cites it in an AI answer is a distribution channel. An AI agent that logs into your customer's account and completes a purchase is something else entirely. Each deserves its own policy, and getting the distinction right matters more than getting either one perfect.
Next: when the customer is a machine
The Amazon case points to the bigger shift. The machines are not just reading your content anymore. They are shopping on your customers' behalf, logging in, comparing prices, and moving toward checkout. In Part 3, we look at what happens to brand loyalty, advertising, and the customer relationship when an AI agent starts making purchase decisions.
Pfanner Advantage works with clients to turn change into advantage at the intersection of mobility, motorsport, media, technology, and marketing. Learn more or start a conversation: contact us today.
More Cold Read Insights from Bill Sparks:
How AI Is Getting Between Brands And Customers
What 1979 Telss Detroit About 2026
What Aren’t Americans Buying EVs?
Part 8: State of the Art Branding Case Studies
Part 7: How the Ad Industry is Reinventing Itself
Part 6: AI and the Personalization of Brand Narrative
Part 5: Building Brand Loyalty Through Community
Part 4: Independent Creators as Brand Storytellers
Part 3: Brands as Publshers: The Rise of Owned Media
The Advantage Journal arrives every week. What matters in sport, mobility, media, and technology — curated and contextualized by Bill Sparks, Bill Long, and Paul Pfanner. No hedging. No filler. Subscribe — It’s Free
