The GEO Directory · 18 November 2026 · Published

Blocking AI training? Check what you are blocking from search.

A policy decision, a working connection and an AI citation are three different things. Audit them in that order.

THREE LAYERS / THREE DIFFERENT QUESTIONS

One green check
does not clear the whole path.

01PolicyDo we want this use?robots + product settings
02DeliveryDid this request get the page?verified request + response
03AppearanceWas our page actually used?dated answer + source URL
Schematic audit sequence, not live crawler data. Permission is not proof of access. Access is not a promise of a citation.

A founder tells a developer: "Keep our content out of AI training, but let buyers find us in AI search." The developer enables a broad bot rule. A crawler checker turns green. Weeks later, nobody can explain whether the right requests get through, which content they receive, or what the rule actually changed.

The problem starts with the phrase "AI bot." A training crawler, a search crawler and a fetcher acting for a user can belong to the same company but do different jobs. Blocking one does not automatically block the others. Allowing one does not mean your website will be recommended.

01 / Name the use

Training is not a synonym for search

OpenAI documents separate settings for GPTBot, its training crawler, and OAI-SearchBot, its search crawler. It explicitly describes allowing search while disallowing training. ChatGPT-User is a different route: certain actions initiated by a person. OpenAI says robots.txt may not apply to those requests and that this user agent does not determine search inclusion. [1]

That split is useful, but it is not a universal template. Anthropic says its bots honor robots.txt, including Claude-User. Perplexity describes its user-requested fetcher as generally ignoring robots.txt. A rule copied from one vendor can therefore mean something different at another. [2] [3]

OpenAI

Training
GPTBot
Search crawler
OAI-SearchBot
User-requested retrieval
ChatGPT-User

OpenAI says search and training settings are independent. ChatGPT-User is user-initiated, may not follow robots.txt, and is not the control for search inclusion. [1]

Anthropic

Training
ClaudeBot
Search crawler
Claude-SearchBot
User-requested retrieval
Claude-User

Anthropic says its bots honor robots.txt, including its user-directed fetcher. Use the current documented names rather than assuming old bot lists still cover the product. [2]

Perplexity

Training
Not the purpose of these two agents
Search crawler
PerplexityBot
User-requested retrieval
Perplexity-User

Perplexity says these agents are not used to train foundation models. Its user-requested fetcher generally ignores robots.txt. Treat that differently from the stated Anthropic behavior. [3]

Roles and stated behavior from operator documentation checked on 18 November 2026. This is not a complete inventory of every browser, agent or acquisition route, nor an independent compliance test.

02 / Separate Google's controls

Google-Extended is not an AI Overviews switch

Google-Extended is a product-control token in robots.txt, not a separate HTTP user-agent string you should expect to find in logs. Google documents its scope as training future Gemini models and grounding in Gemini Apps and specified Vertex AI uses. It says this token does not control inclusion or ranking in Google Search. Calling it only a training switch leaves out a real grounding trade-off. [4]

Google Search also now has a separate Search generative AI control in Search Console. The current help page says it rolled out worldwide on 31 August 2026 and covers AI Overviews, AI Mode and generative AI features in Discover. Its include/exclude setting is distinct from AI training. [5]

Check the exact property. A child property can inherit a parent's setting or override it. Seeing "include" on one property is not enough to conclude that every subdomain and URL-prefix property has the same effective choice. Record the property and the effective setting for the URL you are investigating.

Gemini training + specified grounding

Google-Extendedrobots.txt product token

Search generative AI inclusion

Search Console controlproperty setting + inheritance

Do not solve a narrow Search-AI exclusion request by casually blocking Googlebot. Googlebot's crawl preferences affect Google Search more broadly. Likewise, noindex is not a selective "no AI" label. Google's general technical guidance still matters: a supporting link needs a page that meets Search eligibility requirements, and eligibility never guarantees that a page will be indexed or served. [4] [5] [6]

03 / Trace the request

A 200 response can still be the wrong evidence

Start with the exact public URL: scheme, hostname, path and final redirect destination. Check robots.txt on that host and inspect the group that matches the intended crawler. Finding no line with a vendor's name does not mean there is no applicable rule; a wildcard group may still govern it. Matching rules and more-specific groups matter. [7]

Next, inspect delivery. Your browser may have cookies, JavaScript and a completed challenge that an automated request does not. A CDN or web application firewall can stop a crawler even when robots.txt permits access. Google's guidance explicitly includes infrastructure access; Perplexity publishes separate IP ranges for its agents and explains matching each user agent to the corresponding ranges. [6] [3]

A request you send with a copied bot user-agent string is a synthetic test, not proof that the real operator can fetch the page. User-agent strings can be spoofed. Verify request identity using the operator's published method or your provider's verification signals, keep its limitations visible, and preserve the timestamp and requested path. Do not grant a blanket firewall bypass to anyone who writes a familiar name in a header. [4] [3] [14]

Then inspect what was returned. A status code alone does not tell you whether the response contains your product facts, an empty application shell, an error message or a challenge. Check the actual body for the information the buyer needs. Do not claim an engine cannot render a page just because one generic fetch failed; record which client failed and what evidence you still need.

04 / Build the evidence packet

One row per URL, surface and check

Give the developer a record that a marketing owner can also read. Keep the three layers separate instead of compressing them into a "bot-ready" score.

01 / POLICY

Intent and effective control

Exact page and host; intended engine/product surface; training choice; applicable robots group and file snapshot; any relevant product setting, property and inheritance; time checked.

Supports: what you intend and what the visible controls say. Does not establish a successful fetch.

02 / DELIVERY

Identity and content

Verified or explicitly unverified requester; request time and path; edge action; response status; final URL; retained evidence of the content actually delivered. Keep synthetic requests labelled synthetic.

Supports: what happened for this request. Does not establish indexing, future access or a citation.

03 / APPEARANCE

A dated observation

Exact question, product surface, market and date; returned answer and links; whether your URL appears; any difference between an owned link, a brand mention and a recommendation.

Supports: an observed answer under recorded conditions. Does not prove the configuration caused the result.

Copy the blank evidence-ledger CSV header
checked_at_utc,host,page_url,engine_surface,policy_goal,matched_robots_group,robots_evidence_ref,product_control_evidence_ref,verified_request_identity,request_time_utc,request_path,http_status,delivered_content_evidence_ref,observed_answer_ref,conclusion,owner,next_check

No measurements or example successes are filled in. Use explicit unknown values when evidence is unavailable. Store evidence references, not credentials or private log contents, in a shared sheet.

robots.txt is not access control. The Robots Exclusion Protocol expressly says its rules are not a form of authorization. Put genuinely private material behind real access restrictions, and do not publish private paths as though an opt-out were a security boundary. [7]

05 / Diagnose without overclaiming

Three incidents, three different next steps

Hypothetical incident cards, not tests of any named tool, website or engine.

Policy mismatch

"We blocked every AI bot."

The business wanted a training opt-out but also wanted search discovery. The effective controls block the intended search crawler too.

Next: confirm the owner's intended uses, review the vendor-specific controls and change only the unintended restriction. Preserve other path restrictions and verify the result.

Do not report: "Allowing training will fix our citations."

Delivery mismatch

"robots.txt allows it, but the edge challenges it."

A verified request for the public product page receives a challenge instead of its content.

Next: inspect the specific edge rule with the technical owner. Test the narrow fix against the intended verified requester, then confirm the actual response body.

Do not report: "We need to disable bot protection."

No demonstrated access fault

"The right request got the right content. We still are not cited."

The retained request evidence shows successful delivery, but the recorded answer does not link the page.

Next: investigate discovery, relevance and evidence quality separately. Keep the access conclusion bounded to that request.

Do not report: "The crawler check guarantees visibility."

After a change, record the deployment time, recheck the effective policy and delivery, and assign an owner to review later observations. Account for the operator's documented propagation and recrawl behavior; do not infer failure from an immediate answer or promise a universal recovery time.

An llms.txt file is not a substitute for any of these checks. Google explicitly says no new AI text file or special markup is required for its Search AI features. That does not mean such files have no use anywhere. It means publishing one does not demonstrate that the intended engine can access, index or cite your page. [6]

The handoff

Ask for evidence, not a green badge

For this public URL, our intended policy is search access with the specified training restrictions preserved. Please identify the effective controls, verify an actual requester where possible, and retain what the request received. Mark anything we could not verify as unknown. We will assess answer appearance separately, without calling access a citation guarantee.

That request is small enough to act on and precise enough to audit. It lets a founder protect a deliberate content-use choice without treating every missing recommendation as a firewall problem.

Method and sources

What is established, and what is not

This is a documentation-led troubleshooting framework, not a product benchmark or live crawler experiment. No production controls were changed, no operator compliance was independently tested, and the incident cards are hypothetical. Product descriptions below are vendor claims, not tested advantages.

The basic separation of training, search and user retrieval is already documented by the operators. Existing guides such as InvisIT's Anthropic comparison and Citeflow's audit checklist cover parts of this workflow. The contribution here is the combined, URL-level evidence packet and its explicit decision boundaries, including newer Google controls. It is an editorial synthesis, not a newly invented protocol. [12] [13]

Sources were read on 18 November 2026. Controls, names and availability can change. The operator pages establish what their publishers state, not what every request will do in practice.

  1. OpenAI: overview of crawlers

    Search, training and user-initiated retrieval have different controls.

  2. Anthropic: robots and site-owner controls

    ClaudeBot, Claude-SearchBot and Claude-User; stated robots behavior.

  3. Perplexity: crawlers

    Search and user fetchers, published IP ranges and WAF configuration.

  4. Google: common crawlers

    Googlebot and the scope of Google-Extended. Updated 30 October 2026.

  5. Google: Search generative AI control

    Worldwide rollout, property inheritance and effect on Search AI features.

  6. Google: AI features and your website

    Technical eligibility, content access and limits of special AI files. Older control guidance is supplemented by source 5.

  7. IETF: RFC 9309

    Robots Exclusion Protocol, matching rules and security limitations.

  8. Google: shared AI training control plans

    30 October 2026 announcement: Google plans support in 2027.

  9. Obi: product overview

    Vendor description, not a hands-on product test.

  10. AirOps: content engineering teams

    Vendor framework for context, workflows and content-team roles.

  11. Profound: enterprise capabilities

    Vendor description of SSO and role-based access.

  12. InvisIT: Anthropic bot comparison

    Prior specialist explanation of distinct roles and fetcher differences. Its scan statistics were not reproduced here.

  13. Citeflow: robots.txt AI crawler audit

    Prior checklist covering policy inventory, server logs and review cadence.

  14. Cloudflare: AI bot reference

    Infrastructure-side reference for AI crawler identities.

The GEO Directory. Access is a condition to investigate, not a promise of inclusion, recommendation or revenue.