An AI crawler policy for retailers: GPTBot, search bots and scrapers

Blocking every AI crawler in robots.txt is the single most common way a retailer removes itself from AI shopping answers without noticing. The bots that train language models, the bots that power AI search, and the bots that fetch a page on a shopper’s behalf are different programs with different user agents, and a one-line gptbot robots txt ecommerce rule copied from a forum treats them as one. This guide separates them, explains what each decision costs and protects, and shows how to verify which bots actually hit your store rather than which ones you think you blocked.

In short

• Three kinds of AI traffic reach a store: model-training crawlers (GPTBot, ClaudeBot, CCBot), AI search indexers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and user-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User). Each has a distinct purpose and a distinct robots.txt token.
• Blocking training bots does not remove you from AI answers in most systems, but blocking the search indexers and user fetchers does. Google documents that its Google-Extended token affects Gemini training and grounding, not Search or AI Overviews.
• robots.txt is a request, not a wall. RFC 9309 defines it as a voluntary protocol. Scrapers that ignore it need WAF rules, rate limits and bot management, and those tools misfire on legitimate agents when tuned too aggressively.
• Verify with server logs and published IP ranges, not with user-agent strings alone. OpenAI, Google and others publish the IP blocks their crawlers use, so a request claiming to be GPTBot from an unlisted range is a spoof.
• Review the policy quarterly. New tokens appear every few months (Claude-SearchBot, Applebot-Extended, meta-externalagent), and some vendors change what a token controls. A policy written in 2024 is wrong today.

What kinds of AI traffic actually reach a retail site?

AI traffic to a store splits into three operational categories, and the entire policy question depends on telling them apart. Training crawlers copy pages into datasets that teach a future model. Search indexers build a retrieval index that an AI assistant consults when answering a live question. User fetchers pull a single page in real time because a person asked an assistant about it.

The categories differ in volume, timing and business value. A training crawler may pull tens of thousands of pages once and return months later, influencing a model’s general knowledge but rarely producing a citation or a visit. A search indexer crawls continuously, much like Googlebot, and directly determines whether your product page can appear in an answer. A user fetcher produces a handful of requests exactly when a shopper is comparing options, the highest-intent moment in the funnel.

The commercial stakes are laid out in the broader guide to preparing your store for AI shopping agents: agents can only recommend, compare or buy what they can read. A crawler policy is the gate that decides whether they get to read anything at all. That is why this decision sits with marketing and e-commerce leadership, not only with the security team that owns the WAF.

Why the distinction is new

Until 2023 the only crawlers a retailer cared about were search engines (allowed, because they drove revenue) and price-comparison scrapers (blocked, because they drove only load). AI crawlers broke that binary because the same vendor now operates a training bot you might reasonably block and a search bot you almost certainly want to allow, each identified by a separate token.

The vendors themselves pushed this separation. OpenAI split GPTBot (training) from OAI-SearchBot (search) and ChatGPT-User (user-initiated fetches) in its published bot documentation, and Anthropic did the same with ClaudeBot, Claude-SearchBot and Claude-User. Understanding why a vendor offers three tokens instead of one is the fastest route to writing a sane policy.

What is each major AI crawler used for?

The table below summarizes the major tokens as described in each vendor’s own documentation. Vendors change these definitions, so the descriptions here reflect published documentation as of this writing and every entry should be re-checked at the source before it goes into a production robots.txt.

User-agent token	Operator	Documented purpose	Typical retailer decision
GPTBot	OpenAI	Collecting content that may be used to train generative models	Policy choice; blocking does not affect ChatGPT search citations per OpenAI docs
OAI-SearchBot	OpenAI	Indexing for ChatGPT search features and surfacing links in answers	Allow if you want products cited in ChatGPT
ChatGPT-User	OpenAI	Fetches a page when a user asks ChatGPT about it; not a bulk crawler	Allow; blocking hides your store at the moment of comparison
ClaudeBot	Anthropic	Training data collection	Policy choice
Claude-SearchBot	Anthropic	Search indexing to improve search result quality	Allow for AI search visibility
Claude-User	Anthropic	User-initiated page fetches	Allow
Googlebot	Google	Search indexing; also feeds AI Overviews and AI Mode	Allow; blocking removes you from Google Search entirely
Google-Extended	Google	Control token for Gemini training and grounding; does not affect Search inclusion	Policy choice with no Search ranking impact per Google docs
PerplexityBot	Perplexity	Indexing for Perplexity answers	Allow for citation visibility
Perplexity-User	Perplexity	User-initiated fetches; Perplexity states these generally do not honor robots.txt	Cannot be reliably blocked via robots.txt alone
CCBot	Common Crawl	Open web archive used by many training pipelines	Block if training opt-out is the goal; note downstream reuse
Applebot-Extended	Apple	Opt-out token for Apple foundation model training; Applebot itself feeds Siri and Spotlight	Policy choice
meta-externalagent	Meta	Training and product improvement crawling	Policy choice
Bytespider	ByteDance	Crawler associated with ByteDance products; historically high request volume	Often rate-limited or blocked on load grounds
Amazonbot	Amazon	Indexing for Alexa and other Amazon services	Policy choice

Two patterns in that table matter more than any single row: every major vendor now offers a training token that can be blocked without touching its search or user-fetch tokens, and the tokens that produce actual shopper visits are precisely the ones a blanket “block all AI” rule removes. OpenAI publishes the definitions and IP ranges for its three bots on its bot documentation page, and Google maintains an equivalent list of its common crawlers and user-triggered fetchers.

Training crawlers: what allowing them buys you

Allowing GPTBot or ClaudeBot means your catalog copy, brand story and policies may end up in the general knowledge of a future model. For a large brand this has a plausible upside: models that “know” your product line describe it more accurately in unsourced answers. For a small retailer, the direct benefit is smaller because product-level detail rarely survives training; models learn categories and brands, not SKUs and prices.

The downside is philosophical rather than operational. The real objection is that copy written at your expense improves a commercial product you do not control, and the vendor tokens exist so a retailer can act on that position without paying a visibility price.

Search indexers: the ones that decide citations

Search indexers are the closest AI analogue to Googlebot. When ChatGPT, Perplexity or Claude answers a product question with a citation and a link, the answer was assembled from pages the indexer previously fetched and stored. If OAI-SearchBot has never seen your product page, ChatGPT cannot cite it no matter how well the page is written. The mechanics of how ChatGPT selects and presents those citations are covered in detail in the breakdown of how ChatGPT cites retail content.

For retailers, these indexers are the AI equivalent of being in the Google index. Blocking them is a decision to be absent from a channel that a growing share of shoppers use for product research before opening a search engine; the exact share varies by category and survey method, so treat any single percentage as directional.

User fetchers: high intent, low volume

User fetchers are triggered when a person asks an assistant about a specific page or product. OpenAI describes ChatGPT-User as acting on behalf of a user rather than crawling the web automatically, and Anthropic describes Claude-User the same way. The request pattern is a few pages fetched within seconds, then nothing.

This is the traffic type that most resembles a customer walking into the store with a specific question. Blocking it does not reduce load in any meaningful way, but it does mean the assistant tells the shopper it cannot access your site, and the shopper moves on to a competitor whose page loads. Of all the rows in the table above, this is the category where a block is hardest to justify commercially.

How do robots.txt decisions map to visibility in AI answers?

The mapping is not one-to-one, and that is where most policies go wrong. A Disallow line for a training token removes future training use; it does not remove existing knowledge from a model already trained. A Disallow line for a search indexer removes you from that assistant’s retrievable index within weeks to months, depending on recrawl cadence. A Disallow line for a user fetcher removes you instantly from the live comparison moment.

Google’s own documentation is the clearest public statement of how one vendor handles this. Google says that Google-Extended controls whether content is used to improve Gemini models and grounding, and explicitly states that it does not affect whether a page appears in Google Search or in AI Overviews. In other words, a retailer can opt out of Gemini training while keeping full presence in AI Overviews, because AI Overviews draw on the standard Googlebot index. Any policy that blocks Googlebot to “stop AI” removes the store from Search itself.

The broader question of how visibility in generative answers differs from classic search ranking is the subject of the primer on what AIO for retailers is and why it matters alongside SEO. The crawler policy is the technical precondition for everything that primer describes.

Four policy postures and what each one costs

Most retailers land in one of four postures. The second table compares them on the two axes that matter: what the store gives up in AI visibility and what it protects against.

Posture	robots.txt shape	AI visibility impact	What it protects	Who it suits
Open	No AI-specific rules; standard Disallow for cart, account, search results	Full presence in AI search and user fetches; content used for training	Nothing beyond standard crawl hygiene	Small brands prioritizing reach; retailers with no proprietary copy concerns
Training opt-out	Disallow GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent; allow search and user tokens	Full presence in AI search and fetches; no future training use	Editorial and catalog copy from training reuse	The default recommendation for most retailers
Selective by path	Training opt-out plus Disallow for specific paths (editorial, guides) while leaving product pages open	Product pages fully visible; long-form content withheld from some bots	High-investment editorial content	Retailers with a content business alongside the catalog
Full block	Disallow every AI token including search and user fetchers	Absent from ChatGPT search, Perplexity, Claude and any agent that honors robots.txt; still in Google unless Googlebot is blocked	Everything, including revenue from AI-referred shoppers	Rarely justified for a retailer; sometimes chosen under legal advice for specific content types

The training opt-out posture is the pragmatic center for most stores. It respects the business concern about training reuse while preserving every channel that produces a shopper. The full block posture is the one most often adopted by accident, usually by copying a publisher’s robots.txt that was written for a news site whose economics differ entirely from a retailer’s.

What a training opt-out looks like in practice

A robots.txt implementing the training opt-out posture lists each training token with a Disallow for the whole site, then either omits the search and user tokens (which means they inherit the default allow) or lists them explicitly with an empty Disallow to make the intent obvious to whoever reads the file next year. Explicit is better; a robots.txt that is self-documenting survives staff turnover.

Path-level rules still apply beneath the AI rules. Cart, checkout, account, internal search result pages and faceted filter combinations should remain disallowed for every crawler, AI or otherwise, for the same crawl-budget reasons covered in the guide to handling faceted navigation without damaging SEO. An AI indexer wandering through ten thousand filter permutations is no more useful than Googlebot doing it.

Newer signals: Content-Signal and licensing headers

Because robots.txt only offers allow or disallow, several proposals in 2025 added a way to express purpose. Cloudflare introduced a Content-Signal directive for robots.txt that lets a site state separately whether content may be used for search, for AI input at answer time, and for AI training. The RSL (Really Simple Licensing) standard, backed by a group of publishers, proposes machine-readable licensing terms.

Adoption by crawlers is voluntary and uneven, so these complement token-level rules rather than replace them.

How do you block scrapers without blocking buyers?

Scrapers that steal pricing or harvest inventory data do not read robots.txt, so the file is irrelevant to them. The tools that stop them (rate limiting, bot management, WAF rules) each carry a risk of blocking a legitimate AI agent that was about to recommend your product. The question shifts from “which tokens do we allow” to “how do we tell an abusive fetch from a valuable one when both look automated.”

The key is that legitimate AI vendors identify themselves and publish verification methods, while scrapers hide. OpenAI publishes the IP ranges its bots use as machine-readable JSON files. Google offers reverse DNS verification and publishes its crawler IP ranges. A request that claims a known user agent but arrives from an unpublished range should be treated as a spoof.

A request from a residential proxy rotating through thousands of IPs with a generic browser user agent is a scraper, regardless of what it claims.

Rules that separate the two

1. Verify before you trust a user agent. Build allow lists from published IP ranges, not from user-agent strings. Refresh the ranges on a schedule because vendors add and rotate them.
2. Rate-limit by behavior, not by identity. A verified OAI-SearchBot request rate that matches its documented cadence should pass; an unverified client pulling 50 product pages per second should not, whatever it calls itself.
3. Protect pricing endpoints separately. Scrapers target price, stock and variant JSON endpoints. Those can be rate-limited far more aggressively than HTML product pages without touching AI visibility, because the indexers read HTML.
4. Do not use JavaScript challenges on product pages. Many AI fetchers do not execute JavaScript. A challenge page that a human passes in a second is a permanent wall for an agent, and the assistant reports your store as inaccessible.
5. Serve structured data in the initial HTML. Product schema, price and availability that render only after client-side JavaScript are invisible to fetchers that read the raw response. The product feed fields that decide inclusion in AI shopping agents should also appear in the page HTML, not only in a separate feed.

The Cloudflare default and what it changed

In July 2025 Cloudflare announced that new customers would have AI crawlers blocked by default and introduced a pay-per-crawl experiment that lets sites charge crawlers for access. Because a large share of retail sites sit behind Cloudflare, many stores found themselves blocking AI indexers without any deliberate decision. Any retailer on Cloudflare or a similar CDN should check the bot management dashboard, not just robots.txt, because a store can have a perfectly open robots.txt and still return a 403 to every AI fetcher.

How do rate limits, WAF rules and false positives interact?

False positives are the hidden cost of aggressive bot defense, and they are hard to see because a blocked agent does not complain; it tells the shopper the page could not be loaded and cites a competitor. The only way to measure this is to look at 403 and 429 responses in server logs, grouped by user agent and verified IP range, and count how many went to identifiable AI vendors.

The most common sources of false positives are rules written for a different threat. A rule that blocks all datacenter IP ranges blocks every AI vendor, since none of them crawl from residential connections. A rule that requires a browser fingerprint blocks every fetcher that does not run a full browser.

A rule that challenges any client requesting more than a few pages per minute blocks search indexers doing normal recrawls. Each rule was reasonable against scrapers and each one silently deletes the store from AI answers.

Tuning for the agent era

A practical tuning approach starts by creating a verified-vendor allow group in the WAF that bypasses challenges and gets a generous rate limit, populated from published IP ranges. Everything outside that group faces the existing scraper defenses. The allow group needs a refresh cadence, because a stale IP list turns into a block within months.

Rate limits for the verified group should be set from observed behavior: search indexers typically run at a fraction of Googlebot’s rate on a mid-size store, and user fetchers produce single-digit bursts. If a verified vendor exceeds what its documentation describes, that is a support ticket to the vendor, not a reason to block the token.

User fetchers that do not read robots.txt

Perplexity states in its documentation that Perplexity-User requests, because they are triggered by a person, generally do not honor robots.txt. OpenAI’s documentation makes a similar distinction for ChatGPT-User. This is contested territory.

In August 2025 Cloudflare published research alleging that Perplexity used undeclared crawlers to evade blocks; Perplexity disputed the characterization. Retailers should treat both accounts as attributed positions rather than established fact, and should recognize that the only reliable control over user-triggered fetches is at the network layer, where blocking them also blocks the shopper who triggered them.

How do you verify which bots actually reach your store?

A policy is only as good as the evidence that it works, and most retailers have never checked. The gap between the robots.txt a team wrote and the traffic the server receives is often large: the CDN applies its own rules, spoofed user agents inflate apparent AI traffic, and vendors add tokens the policy never mentioned.

Verification starts with raw access logs, not analytics, because crawlers rarely execute a JavaScript tag. Server or CDN logs record every request with its user agent, IP, path and response code, and a week of logs from a mid-size store is enough to build a baseline.

A verification routine

1. Extract all requests whose user agent matches a known AI token. Group by token, count requests, count distinct paths and tabulate response codes.
2. Check each group against the vendor’s published IP ranges. Requests outside the range are spoofs; move them to the scraper bucket and consider blocking their source.
3. Look at what the verified bots fetched. If OAI-SearchBot spent its budget on filter pages and never reached best-selling products, the crawl-budget problem is more urgent than the policy question.
4. Look at what they were denied. A verified vendor receiving 403 or 429 on product pages is a false positive to fix that week.
5. Compare the token list against your robots.txt. Any token in the logs that the file does not mention is inheriting the default and should get an explicit decision.
6. Fetch your own robots.txt from outside the network and confirm the CDN serves the version you edited. Cached or overridden files are common.

Testing from the assistant side

The complement to log analysis is asking the assistants directly: prompt ChatGPT, Perplexity and Claude with product questions your store should win and check whether they can fetch a specific product URL. An assistant that reports it cannot access the page is documenting a block the logs will confirm. The guide to what retailers should optimize for in Perplexity and Google AI Overviews covers the visibility side of this test; the crawler policy covers whether the test can succeed at all.

How often should the policy be reviewed as crawlers change?

Quarterly is the minimum for a retailer that takes AI channels seriously, with an out-of-cycle review whenever a major vendor announces a new token or changes what an existing one controls. The token landscape has changed materially every few months since 2023: OpenAI split its crawler into three, Anthropic added Claude-SearchBot, Apple added Applebot-Extended, Meta introduced meta-externalagent, and Google clarified the Google-Extended scope. A policy that was correct at any single point is incomplete within two quarters.

Ownership matters as much as cadence. The policy sits at the intersection of security (WAF and bot management), engineering (robots.txt deployment and CDN settings) and marketing (channel visibility). When no one owns the whole picture, security tightens rules after a scraping incident and marketing discovers three months later that AI referrals disappeared.

A review checklist

• Pull the current bot documentation from each major vendor and diff the token list against the last review.
• Refresh IP allow lists from published ranges and confirm the WAF group uses the new ranges.
• Re-run the log verification routine and compare verified request counts against the previous quarter.
• Check the CDN bot management settings for defaults that changed in a vendor update.
• Re-test product URL fetches from each assistant and record pass or fail.
• Review any new Content-Signal, RSL or similar directives and decide whether to adopt them.
• Confirm the robots.txt served publicly matches the version in source control.

This checklist pairs naturally with the broader retailer AIO checklist, which covers the content and structured data side. The crawler policy is one line item there, but it is the line item that makes every other item possible.

The agent-platform dimension

A newer reason for frequent review is that AI vendors are building direct commerce integrations alongside crawling. OpenAI announced Instant Checkout and an Agentic Commerce Protocol with Stripe in 2025, and Google announced its own agent-oriented commerce protocol with major retail partners at the start of 2026. These programs rely on merchant-supplied feeds, but the crawlers still verify what the feed says, so a store enrolled in an agent program with a blocked fetcher is sending contradictory signals. The full picture is in the guide to agentic commerce and preparing your store for AI shopping agents.

Common mistakes retailers make with AI crawler policy

• Copying a publisher’s robots.txt. News publishers have different economics; their full-block posture removes a retailer from the exact channels that drive product discovery.
• Blocking Googlebot to stop AI Overviews. AI Overviews use the standard Google index. Blocking Googlebot removes the store from Search. Google-Extended is the training control, and Google states it does not affect Search inclusion.
• Trusting user-agent strings. Spoofed AI user agents are common among scrapers. Unverified traffic inflates apparent AI load and leads to blocks that hit the real vendors.
• Forgetting the CDN layer. A default-on AI block in bot management overrides an open robots.txt. Check both.
• Using JavaScript challenges site-wide. Fetchers that do not execute JavaScript never see the product page. Scope challenges to endpoints scrapers actually target.

A note on legal and contractual questions

This article is general information about how crawler controls work and is not legal advice. Questions about whether a particular scraper is violating your terms of service, whether a crawler’s data use is permitted under copyright or database law in your jurisdiction, and how to draft enforceable terms are matters for a licensed attorney with experience in technology and intellectual property law. Regulatory positions on scraping and training data are evolving in the United States, the European Union and the United Kingdom. The vendor practices described here come from the vendors’ own documentation and public statements and should be re-verified at the source; allegations by one company about another’s crawling are reported as attributed claims, not established findings.

FAQ: AI crawler policy for retailers

Does blocking GPTBot remove my store from ChatGPT search results?

According to OpenAI’s bot documentation, no. GPTBot collects content that may be used for model training, while OAI-SearchBot handles indexing for ChatGPT search features and ChatGPT-User handles fetches triggered by a person. A retailer that wants to opt out of training while staying visible in ChatGPT answers disallows GPTBot and leaves OAI-SearchBot and ChatGPT-User allowed. Verify the current definitions at OpenAI’s documentation before relying on this, because the vendor controls what each token means.

Will blocking Google-Extended hurt my Google rankings or remove me from AI Overviews?

Google’s documentation states that Google-Extended controls whether content is used to improve Gemini models and for grounding, and that it does not affect a page’s inclusion or ranking in Google Search, including AI Overviews, which are built from the standard Googlebot index. A retailer can therefore opt out of Gemini training via Google-Extended with no documented Search consequence. Blocking Googlebot itself is a different matter entirely and removes the store from Search.

What is the difference between a training crawler and a user fetcher?

A training crawler visits many pages on its own schedule to build a dataset for a future model; its visits do not correspond to any shopper’s activity. A user fetcher retrieves a specific page because a person asked an AI assistant about it in real time, so each request maps to a live shopper with a question. Training crawlers can be blocked with a modest visibility cost. Blocking user fetchers means the assistant tells the shopper it could not access your store at the exact moment of comparison.

Can robots.txt stop scrapers from copying my prices?

No. robots.txt is defined by RFC 9309 as a voluntary protocol that well-behaved crawlers choose to honor. Scrapers that copy prices, descriptions or stock levels ignore it, so stopping them requires rate limiting, bot management and WAF rules, ideally targeted at the specific endpoints scrapers hit (price and stock JSON, variant APIs) rather than at all automated traffic. Rules aimed at all automation tend to block legitimate AI vendors as collateral damage.

How can I tell whether a request claiming to be GPTBot is real?

Check the source IP against the ranges OpenAI publishes for each of its bots. OpenAI provides machine-readable IP lists for GPTBot, OAI-SearchBot and ChatGPT-User. A request with a GPTBot user agent from an IP outside those ranges is a spoof and should be treated as scraper traffic, and Google offers reverse DNS verification and published IP ranges for its crawlers. Never build an allow list from user-agent strings alone; build it from verified ranges and refresh them regularly.

My site is behind Cloudflare. Is my robots.txt still what controls AI access?

Not necessarily. In 2025 Cloudflare began blocking AI crawlers by default for new customers and added tools that let sites charge for crawler access. Those settings are applied at the network edge before the request reaches your server, so a robots.txt that allows OAI-SearchBot can coexist with a CDN rule that returns 403 to it. Check the bot management dashboard as well as the file, and confirm with server or CDN logs which AI vendors are actually being served content.

How often do the crawler tokens change?

Materially every few months since 2023: OpenAI split one crawler into three, Anthropic added Claude-SearchBot, Apple introduced Applebot-Extended, Meta added meta-externalagent, and Google clarified the scope of Google-Extended. Directives like Cloudflare’s Content-Signal and the RSL licensing standard appeared in 2025. A quarterly review that diffs each vendor’s published token list against your robots.txt, refreshes IP allow lists and re-tests fetches from each assistant keeps the policy from drifting into an unintended block.

What to read next

The crawler policy decides whether AI systems can read your store at all; the product feed decides what they find when they do. The companion guide to product feeds for AI shopping agents covers the fields that determine inclusion, and the breakdown of how ChatGPT cites retail content shows what a well-crawled page needs to actually earn the citation.