Allow, Block or Charge the AI Crawlers on Your Shop

Somewhere in your analytics there is traffic that never buys anything and never bounces, because it is not a person. For most shops it is now a substantial share of requests, and the default setting for all of it was chosen by whoever set up the site, which is to say nobody.

This is a decision worth making deliberately once, because the cost of the wrong default is asymmetric. Blocking too much removes you from surfaces where customers are actively looking. Blocking too little gives away the one asset a small shop has, which is the specific product knowledge in its own pages.

Which bots are actually visiting your shop?

Six kinds, doing six different jobs. Lumping them together is what produces bad rules.

Google's documentation on the crawlers it operates lists a long roster with distinct tokens: Googlebot for search, Googlebot-Image, Google StoreBot for shopping product indexing, GoogleOther as a generic fetcher for internal product teams, and Google-CloudVertexBot for Vertex AI agent data collection. The one that matters most for this decision is Google-Extended, which Google describes as a standalone product token publishers can use to control whether crawled content is used to train future Gemini models.

The crucial sentence is the next one. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. That means the training question and the search question are genuinely separable, and you can answer them differently without paying a search penalty for it.

OpenAI does the same separation with different names. Its overview of its crawlers documents four: OAI-SearchBot, which surfaces sites in ChatGPT's search features; GPTBot, which crawls for training and whose disallowal signals that content should not train foundation models; OAI-AdsBot, which checks landing pages for ad policy compliance and whose data is not used for training; and ChatGPT-User, which fetches a page because a person asked ChatGPT a question. That last one carries a caveat worth reading twice: robots.txt rules may not apply, because the request originates from a user action rather than from an automated crawl.

What does blocking each one actually cost you?

Different things, and the table below is the assessment we would make for a shop rather than for a publisher. The difference matters: a news site loses traffic when an answer engine quotes it, while a shop with three hundred products generally wants to be the thing an answer engine names.

Read the fourth column and a policy falls out on its own. Allow everything that puts you in front of a buyer. Decide the training row on your own terms, since neither allowing nor blocking it moves your revenue this quarter. And stop expecting robots.txt to handle the last row at all.

Is robots.txt actually enforceable?

No, and the standard says so itself. RFC 9309, which formalised the Robots Exclusion Protocol, states directly that its rules are not a form of access authorisation and that the protocol is not a substitute for valid content security measures. It is a published request that well behaved crawlers honour and everyone else ignores.

Two mechanical details in the standard are worth knowing because they cause real incidents. Matching on the user agent token is case insensitive, and when several groups match a crawler their rules merge, with a wildcard group applying only when no specific match exists. And if your robots.txt becomes unreachable because the server returns an error, a compliant crawler must assume complete disallow. A shop whose robots.txt goes missing during a site migration can therefore fall out of an index quietly, which is a far more common failure than deliberate over blocking.

The third option: charging for the crawl

Since July 2025 the choice has not been binary. Cloudflare's announcement of pay per crawl introduced a mechanism in which a site can answer a crawler with an HTTP 402 Payment Required response carrying a price, and return the content with a 200 once payment is agreed. Crawlers can signal a maximum price up front or react to the quoted one, publishers get a confirmation header when a request is charged, and authentication runs on signed requests rather than on trust in a user agent string.

The interface is the interesting part for a small merchant, even before the money is. A domain owner sets a per request price and chooses allow, charge or block for each crawler independently, with the ability to exempt particular ones. That is the correct shape for this decision, and it is the shape you should hold in your head even if you never enable the payment side.

Two caveats. It was announced as a private beta, so treat availability as something to check rather than assume. And realistically, the revenue a three hundred product shop would earn from crawl payments is not a business line. The value is in the control and in the record of who took what, not in the cheque.

What should a small shop actually put in its robots.txt?

Start from what you want to be findable in, then subtract. A workable default for a shop that sells things looks like this.

1. Allow every search and shopping crawler. There is no version of this decision where hiding your products from search helps you.
2. Allow the answer crawlers. Buyers increasingly ask an assistant before they open a search results page, and being absent from that answer is the same kind of loss that being absent from page one used to be. Our look at what actually moves citations in AI answers covers how to earn a mention once you are crawlable.
3. Decide the training crawlers deliberately. Blocking them protects nothing measurable and costs nothing measurable. If your product photography and copy are the business, blocking is defensible. If you would rather your brand appear in as much training data as possible, allowing is equally defensible. What is not defensible is not knowing which you chose.
4. Keep the crawl surface clean. Disallow internal search results, faceted filter URLs and cart pages, which waste crawl budget and produce near duplicate pages for every crawler in the list.
5. Check it after every migration. Because of the unreachable file rule above, and because migrations are where robots.txt files go to die.

The related question people ask next is whether a file like llms.txt helps. We tested that argument in the piece on whether llms.txt does anything, and the honest answer is that a proposal only works when the consumers of it have agreed to read it. The published crawler tokens above are documented and honoured today, which is a different level of evidence.

Does blocking training crawlers protect your content?

Partly, and less than the framing suggests. It expresses a preference to the operators that publish tokens and honour them, a preference whose legal weight is being argued in court now that the Justice Department has backed fair use for AI training. It does nothing about anyone who does not, it does nothing about content already collected, and it does nothing about your pages appearing in a dataset someone else assembled.

There is also a second hand route that catches people out. Product data syndicated to marketplaces, comparison sites and feed partners is republished on domains you do not control, with their robots policies rather than yours. A shop that blocks training crawlers on its own site while feeding twelve aggregators has closed one door in a building with many. We went through the wider version of this in the piece on who is buying business data in bulk to train on.

The realistic goal is not preventing all use. It is making a considered choice, keeping a record of it, and not accidentally paying a visibility cost you never intended.

How do you tell what is actually hitting your site?

Server logs, and there is no substitute. Analytics tools that run JavaScript miss most crawlers entirely, which is why shop owners are often surprised by log data. Filter by user agent, group by token, and look at requests per day and which paths are being hit.

Three things usually turn up. A crawler you assumed you had blocked, still arriving, because the rule targeted a token that does not exist. A crawler hammering a faceted URL pattern thousands of times a day, which is a crawl budget problem rather than an AI problem. And a large volume of requests from something with no published token at all, which is the scraper category, and the one your robots file was never going to help with.

Verification matters here too, because user agent strings are free to invent. The major operators publish IP ranges for each of their crawlers so that a claimed identity can be checked, which is the only reliable way to tell a real answer crawler from something wearing its name.

Being crawlable is not the same as being useful

Allowing an answer crawler is the price of entry, not the outcome. What decides whether a shop gets named in an answer is whether its pages state facts an assistant can lift with confidence: a size, a material, a delivery window, a returns rule, a price with its currency. Pages built around atmosphere give a model nothing to quote, so it quotes a competitor who wrote the number down.

The practical consequence is that crawler policy and page quality are one project rather than two. A shop that opens the doors and then serves pages where the specification lives inside an image has done the hard part of the work and skipped the easy part. We went through what those pages need in the piece on what AI shoppers actually read on a product page.

Two structural details help more than they should. Keep the important facts in text rather than in images or scripts that require rendering, because not every crawler executes JavaScript and the ones that do may not wait. And keep them consistent between the page, the structured data and any feed, since a contradiction between the three is a reason for a system to trust none of them.

What about the traffic cost of all this crawling?

Real, and usually smaller than it feels for a shop of modest size, with one exception. The exception is faceted navigation. A crawler that discovers a URL pattern combining colour, size, price band and sort order will generate an effectively unbounded set of pages, and it will keep requesting them. A shop with two hundred products can end up serving tens of thousands of distinct URLs, none of which anyone should see.

That problem long predates AI crawlers and is solved the same way it always was: disallow the parameter patterns, use canonical tags honestly, and do not link to filter combinations from anywhere crawlable. Doing it well benefits your search ranking and your hosting bill at once, and it removes the most common reason a shop concludes that bots are the problem.

If your hosting bills by request or by bandwidth, check the shape of the traffic before you reach for a blanket block. In most cases the fix is a handful of disallow rules for parameter patterns, not a policy decision about artificial intelligence.

How often should this be revisited?

Twice a year is enough, plus any time a major operator publishes a new token. The pace of change is real: the four OpenAI crawlers documented today did not all exist two years ago, and the ad checking one arrived with a commercial product attached. A rule written against a token that has since been split into two will silently stop doing what you intended.

Build the review into something you already do. When you check your search performance, spend ten minutes on the crawler list, compare it against your current robots file, and confirm nothing you rely on has been quietly excluded. That review is also the natural moment to check that your robots file is actually reachable, which is the failure with the largest consequence and the smallest warning.

Keep the reasoning next to the rules. A robots.txt full of tokens with no explanation is unmaintainable by the person who inherits it, including you in a year. One comment line per decision, naming the date and the reason, turns a mystery into a document.

The part that is actually within your control

Whether you can change any of this in an afternoon depends on how your shop is built. Editing robots.txt, adding headers, setting per crawler rules and reading raw logs are all trivial when you own the deployment, and range from awkward to impossible when your storefront is a page on somebody else's platform. That constraint tends to be invisible until the day it matters.

It is one of the reasons our ecommerce website builder writes the storefront into a repository you control and deploys to infrastructure in your own account. Crawler policy is a small example of a general rule: the decisions you can make about your own shop are limited to the ones your platform exposes, and this year keeps adding new ones.

Whatever you decide, write it down in one place with the date. In twelve months a new token will exist, someone will ask why a particular bot is allowed, and a two line note will be worth an afternoon of archaeology.