Start by separating three different uses
AI crawler policy is no longer a single allow-or-block decision. Search discovery, model training and answer grounding are different uses of web content, and major platforms now expose separate controls for some of them. A useful policy begins by deciding what the organisation wants for each use rather than copying a generic robots.txt blocklist.
Search discovery is the retrieval and indexing work that can make a page eligible to appear as a result or citation. Training means using content to change the learned parameters of a generative model. Grounding means retrieving content when an answer is created so the response can use current information and cite its sources. Blocking one use does not necessarily block the others.
- Search discovery: can the page be found and surfaced as a result or citation?
- AI training: may the content be used to improve future generative models?
- Answer grounding: may the content inform a generated answer at response time?
- User-requested access: can an assistant fetch a page when a person asks it to?
Keep search discovery controls aligned with the visibility goal
OpenAI documents OAI-SearchBot as the crawler used to surface websites in ChatGPT search. It is independent from GPTBot, which relates to potential training. A publisher can therefore allow OAI-SearchBot while disallowing GPTBot if the goal is search visibility without training use. OpenAI notes that a robots.txt change can take about 24 hours to propagate through its systems.
Anthropic also documents separate agents for different uses: Claude-SearchBot supports search discovery, while ClaudeBot collects public web content that may contribute to model training. Keeping these agents distinct in a policy makes the organisation's intent clearer and avoids sacrificing search eligibility merely because its training preference changes.
The same principle applies more broadly: normal search crawlers still need access to important pages, scripts and styles, and indexable pages must not carry a conflicting noindex directive. An AI-specific allow rule cannot repair a page that is inaccessible, canonicalised elsewhere or missing from the site's information architecture.
Understand the new shared AI-training preference
On 30 October 2026, Google announced plans to support a shared robots.txt control developed through the IETF AI Preferences work. Microsoft announced support for the same approach and published the current syntax using the __usage_ai-training policy token. The shared control is intended to express a training preference across participating providers while company-specific controls continue to work.
The timing matters. Google describes its support as planned for 2027. Microsoft says it will begin honouring the new preferences as pages are crawled from 1 February 2027. The proposal is still developing, so organisations should record the policy decision, follow implementation updates and test the served robots.txt rather than treating an early example as a permanent standard.
- Allow is the default when no applicable shared training preference is published
- The shared robots.txt token can apply to the whole site or selected paths
- Microsoft also supports a page-level Bingbot training directive
- Provider-specific controls such as GPTBot and Google-Extended continue to matter
- A training preference does not by itself determine normal search eligibility
Treat grounding as a separate publishing decision
Microsoft's new AI Grounding preference is page-level rather than a site-wide robots.txt rule. Allow is the default. Disallow prevents the page from being used to inform generated responses, while DisallowExtended limits grounding to Microsoft-operated experiences and excludes the grounding services Microsoft supplies to third parties.
Microsoft explicitly warns that restricting grounding will significantly reduce content visibility in AI-generated experiences, even though normal Bing search eligibility is unaffected. For a business seeking discovery and citations, that trade-off is materially different from declining model training. The decision should be made deliberately, documented and applied only after checking which pages and platforms it affects.
Build a policy before editing robots.txt
A useful decision record names the content owner, the intended audience, the desired visibility and any material licensing or confidentiality constraints. Public service pages and original research may justify different choices, and private or customer-only information should not rely on crawler directives as access control in the first place.
For each provider and use, record Allow, Disallow or Not yet decided, then map that choice to the current official control. Check for conflicts between wildcard rules, named user-agent groups, page-level meta directives and HTTP headers. Preserve access for the search crawlers that support the organisation's discovery goals, and keep API or administrative paths out of public crawl space.
Validate the result as a release, not a text edit
Fetch the production robots.txt after deployment and confirm that every group, path and sitemap reference is exactly what was intended. Test representative pages for status, canonical, robots meta and X-Robots-Tag headers. If page-level training or grounding directives are used, verify the final rendered HTML or response header rather than the source template alone.
Submit genuinely changed URLs through the discovery tools available to the relevant platform, while recognising that a submission does not guarantee a recrawl or citation. Monitor Search Console, Bing Webmaster Tools, server logs and identifiable AI referrals for unintended visibility changes. Revisit the policy when providers, standards or the organisation's objectives change.