The GEO Directory · 18 November 2026 · Published
Blocking AI training? Check what you are blocking from search.
A policy decision, a working connection and an AI citation are three different things. Audit them in that order.
THREE LAYERS / THREE DIFFERENT QUESTIONS
One green check
does not clear the whole path.
A founder tells a developer: "Keep our content out of AI training, but let buyers find us in AI search." The developer enables a broad bot rule. A crawler checker turns green. Weeks later, nobody can explain whether the right requests get through, which content they receive, or what the rule actually changed.
The problem starts with the phrase "AI bot." A training crawler, a search crawler and a fetcher acting for a user can belong to the same company but do different jobs. Blocking one does not automatically block the others. Allowing one does not mean your website will be recommended.
01 / Name the use
Training is not a synonym for search
OpenAI documents separate settings for GPTBot, its training crawler, and OAI-SearchBot, its search crawler. It explicitly describes allowing search while disallowing training. ChatGPT-User is a different route: certain actions initiated by a person. OpenAI says robots.txt may not apply to those requests and that this user agent does not determine search inclusion. [1]
That split is useful, but it is not a universal template. Anthropic says its bots honor robots.txt, including Claude-User. Perplexity describes its user-requested fetcher as generally ignoring robots.txt. A rule copied from one vendor can therefore mean something different at another. [2] [3]
OpenAI
- Training
- GPTBot
- Search crawler
- OAI-SearchBot
- User-requested retrieval
- ChatGPT-User
OpenAI says search and training settings are independent. ChatGPT-User is user-initiated, may not follow robots.txt, and is not the control for search inclusion. [1]
Anthropic
- Training
- ClaudeBot
- Search crawler
- Claude-SearchBot
- User-requested retrieval
- Claude-User
Anthropic says its bots honor robots.txt, including its user-directed fetcher. Use the current documented names rather than assuming old bot lists still cover the product. [2]
Perplexity
- Training
- Not the purpose of these two agents
- Search crawler
- PerplexityBot
- User-requested retrieval
- Perplexity-User
Perplexity says these agents are not used to train foundation models. Its user-requested fetcher generally ignores robots.txt. Treat that differently from the stated Anthropic behavior. [3]
Roles and stated behavior from operator documentation checked on 18 November 2026. This is not a complete inventory of every browser, agent or acquisition route, nor an independent compliance test.
02 / Separate Google's controls
Google-Extended is not an AI Overviews switch
Google-Extended is a product-control token in robots.txt, not a separate HTTP user-agent string you should expect to find in logs. Google documents its scope as training future Gemini models and grounding in Gemini Apps and specified Vertex AI uses. It says this token does not control inclusion or ranking in Google Search. Calling it only a training switch leaves out a real grounding trade-off. [4]
Google Search also now has a separate Search generative AI control in Search Console. The current help page says it rolled out worldwide on 31 August 2026 and covers AI Overviews, AI Mode and generative AI features in Discover. Its include/exclude setting is distinct from AI training. [5]
Check the exact property. A child property can inherit a parent's setting or override it. Seeing "include" on one property is not enough to conclude that every subdomain and URL-prefix property has the same effective choice. Record the property and the effective setting for the URL you are investigating.
Gemini training + specified grounding
Google-Extendedrobots.txt product tokenSearch generative AI inclusion
Search Console controlproperty setting + inheritanceDo not solve a narrow Search-AI exclusion request by casually blocking Googlebot. Googlebot's crawl preferences affect Google Search more broadly. Likewise, noindex is not a selective "no AI" label. Google's general technical guidance still matters: a supporting link needs a page that meets Search eligibility requirements, and eligibility never guarantees that a page will be indexed or served. [4] [5] [6]
03 / Trace the request
A 200 response can still be the wrong evidence
Start with the exact public URL: scheme, hostname, path and final redirect destination. Check robots.txt on that host and inspect the group that matches the intended crawler. Finding no line with a vendor's name does not mean there is no applicable rule; a wildcard group may still govern it. Matching rules and more-specific groups matter. [7]
Next, inspect delivery. Your browser may have cookies, JavaScript and a completed challenge that an automated request does not. A CDN or web application firewall can stop a crawler even when robots.txt permits access. Google's guidance explicitly includes infrastructure access; Perplexity publishes separate IP ranges for its agents and explains matching each user agent to the corresponding ranges. [6] [3]
A request you send with a copied bot user-agent string is a synthetic test, not proof that the real operator can fetch the page. User-agent strings can be spoofed. Verify request identity using the operator's published method or your provider's verification signals, keep its limitations visible, and preserve the timestamp and requested path. Do not grant a blanket firewall bypass to anyone who writes a familiar name in a header. [4] [3] [14]
Then inspect what was returned. A status code alone does not tell you whether the response contains your product facts, an empty application shell, an error message or a challenge. Check the actual body for the information the buyer needs. Do not claim an engine cannot render a page just because one generic fetch failed; record which client failed and what evidence you still need.
04 / Build the evidence packet
One row per URL, surface and check
Give the developer a record that a marketing owner can also read. Keep the three layers separate instead of compressing them into a "bot-ready" score.
Intent and effective control
Exact page and host; intended engine/product surface; training choice; applicable robots group and file snapshot; any relevant product setting, property and inheritance; time checked.
Supports: what you intend and what the visible controls say. Does not establish a successful fetch.
Identity and content
Verified or explicitly unverified requester; request time and path; edge action; response status; final URL; retained evidence of the content actually delivered. Keep synthetic requests labelled synthetic.
Supports: what happened for this request. Does not establish indexing, future access or a citation.
A dated observation
Exact question, product surface, market and date; returned answer and links; whether your URL appears; any difference between an owned link, a brand mention and a recommendation.
Supports: an observed answer under recorded conditions. Does not prove the configuration caused the result.
Copy the blank evidence-ledger CSV header
checked_at_utc,host,page_url,engine_surface,policy_goal,matched_robots_group,robots_evidence_ref,product_control_evidence_ref,verified_request_identity,request_time_utc,request_path,http_status,delivered_content_evidence_ref,observed_answer_ref,conclusion,owner,next_check
No measurements or example successes are filled in. Use explicit unknown values when evidence is unavailable. Store evidence references, not credentials or private log contents, in a shared sheet.
robots.txt is not access control. The Robots Exclusion Protocol expressly says its rules are not a form of authorization. Put genuinely private material behind real access restrictions, and do not publish private paths as though an opt-out were a security boundary. [7]
05 / Diagnose without overclaiming
Three incidents, three different next steps
Hypothetical incident cards, not tests of any named tool, website or engine.
"We blocked every AI bot."
The business wanted a training opt-out but also wanted search discovery. The effective controls block the intended search crawler too.
Next: confirm the owner's intended uses, review the vendor-specific controls and change only the unintended restriction. Preserve other path restrictions and verify the result.
Do not report: "Allowing training will fix our citations."
"robots.txt allows it, but the edge challenges it."
A verified request for the public product page receives a challenge instead of its content.
Next: inspect the specific edge rule with the technical owner. Test the narrow fix against the intended verified requester, then confirm the actual response body.
Do not report: "We need to disable bot protection."
"The right request got the right content. We still are not cited."
The retained request evidence shows successful delivery, but the recorded answer does not link the page.
Next: investigate discovery, relevance and evidence quality separately. Keep the access conclusion bounded to that request.
Do not report: "The crawler check guarantees visibility."
After a change, record the deployment time, recheck the effective policy and delivery, and assign an owner to review later observations. Account for the operator's documented propagation and recrawl behavior; do not infer failure from an immediate answer or promise a universal recovery time.
An llms.txt file is not a substitute for any of these checks. Google explicitly says no new AI text file or special markup is required for its Search AI features. That does not mean such files have no use anywhere. It means publishing one does not demonstrate that the intended engine can access, index or cite your page. [6]
The handoff
Ask for evidence, not a green badge
For this public URL, our intended policy is search access with the specified training restrictions preserved. Please identify the effective controls, verify an actual requester where possible, and retain what the request received. Mark anything we could not verify as unknown. We will assess answer appearance separately, without calling access a citation guarantee.
That request is small enough to act on and precise enough to audit. It lets a founder protect a deliberate content-use choice without treating every missing recommendation as a firewall problem.
Method and sources
What is established, and what is not
This is a documentation-led troubleshooting framework, not a product benchmark or live crawler experiment. No production controls were changed, no operator compliance was independently tested, and the incident cards are hypothetical. Product descriptions below are vendor claims, not tested advantages.
The basic separation of training, search and user retrieval is already documented by the operators. Existing guides such as InvisIT's Anthropic comparison and Citeflow's audit checklist cover parts of this workflow. The contribution here is the combined, URL-level evidence packet and its explicit decision boundaries, including newer Google controls. It is an editorial synthesis, not a newly invented protocol. [12] [13]
Sources were read on 18 November 2026. Controls, names and availability can change. The operator pages establish what their publishers state, not what every request will do in practice.
- OpenAI: overview of crawlers
Search, training and user-initiated retrieval have different controls.
- Anthropic: robots and site-owner controls
ClaudeBot, Claude-SearchBot and Claude-User; stated robots behavior.
- Perplexity: crawlers
Search and user fetchers, published IP ranges and WAF configuration.
- Google: common crawlers
Googlebot and the scope of Google-Extended. Updated 30 October 2026.
- Google: Search generative AI control
Worldwide rollout, property inheritance and effect on Search AI features.
- Google: AI features and your website
Technical eligibility, content access and limits of special AI files. Older control guidance is supplemented by source 5.
- IETF: RFC 9309
Robots Exclusion Protocol, matching rules and security limitations.
- Google: shared AI training control plans
30 October 2026 announcement: Google plans support in 2027.
- Obi: product overview
Vendor description, not a hands-on product test.
- AirOps: content engineering teams
Vendor framework for context, workflows and content-team roles.
- Profound: enterprise capabilities
Vendor description of SSO and role-based access.
- InvisIT: Anthropic bot comparison
Prior specialist explanation of distinct roles and fetcher differences. Its scan statistics were not reproduced here.
- Citeflow: robots.txt AI crawler audit
Prior checklist covering policy inventory, server logs and review cadence.
- Cloudflare: AI bot reference
Infrastructure-side reference for AI crawler identities.
The GEO Directory. Access is a condition to investigate, not a promise of inclusion, recommendation or revenue.