How-to · 3 min read
AI Crawlers and robots.txt for Law Firms: Who to Allow and Why
OpenAI, Perplexity, Anthropic and Google each use separate crawlers for search and for training. Here is how to decide what to allow.
Short answer
AI companies use different crawlers for search and for training, so one robots.txt rule cannot cover them all. To be eligible for ChatGPT search, Perplexity and Claude search results, allow OAI-SearchBot, PerplexityBot and Claude-SearchBot. Training crawlers such as GPTBot and ClaudeBot are a separate business choice. robots.txt does not keep a page out of Google.
Key takeaways
- OpenAI, Perplexity and Anthropic each document a search crawler that is separate from their training crawler.
- Blocking a search crawler can keep your pages out of that product's search answers.
- Google-Extended is a robots.txt token for Gemini training and grounding, and it does not affect inclusion in Google Search.
- robots.txt is not a way to keep a page out of Google. Use noindex or a password.
- Firewalls and CDNs can block crawlers even when robots.txt allows them, so check both.
Search crawlers and training crawlers are different
A decade ago a site owner thought about Googlebot and Bingbot. Now several companies run separate bots for separate jobs. The distinction that matters most is between crawlers that help a product find and show your pages in answers, and crawlers that collect content that may be used to train models. A law firm can reasonably choose different answers for each. The table below summarizes what each company documents as of this update.
| Crawler | Company | Documented purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search features. Sites that opt out will not be shown in ChatGPT search answers, though they can still appear as navigational links. |
| GPTBot | OpenAI | Crawls content that may be used to train generative AI foundation models. |
| ChatGPT-User | OpenAI | Visits triggered by a user's action. Not automatic crawling. robots.txt rules may not apply. Not used to determine Search inclusion. |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results. Not used to crawl content for AI foundation models. |
| Perplexity-User | Perplexity | Supports user actions. Generally ignores robots.txt because a person requested the fetch. |
| ClaudeBot | Anthropic | Collects web content that could contribute to training. |
| Claude-SearchBot | Anthropic | Improves search result quality. Blocking it prevents indexing for search optimization and may reduce visibility in search results. |
| Claude-User | Anthropic | Fetches pages when users ask Claude. Blocking may reduce visibility for user-directed web search. |
| Googlebot | Crawls for the Google Search index, which supplies AI Overviews and AI Mode. | |
| Google-Extended | A robots.txt token that controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps. Google says it does not affect inclusion in Google Search and is not a ranking signal. |
Step 1: Decide your policy
-
Allow search crawlers
If you want to be eligible for citation in ChatGPT search, Perplexity and Claude search results, allow their search crawlers. OpenAI says opted-out sites will not be shown in ChatGPT search answers.
-
Decide on training crawlers
Allowing or blocking GPTBot, ClaudeBot and the Google-Extended token is a separate choice. Involve the person who owns your firm's data and intellectual property decisions.
-
Keep Googlebot allowed
Google says a page must be indexed and eligible to appear with a snippet to be a supporting link in its AI features.
Step 2: Write the file
This sample allows search crawlers and blocks the training crawlers. It is a pattern to adapt, not a recommendation for every firm. Replace the sitemap line with your own address.
# Search and answer crawlers: allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Googlebot
Allow: /
# Training crawlers: blocked (your decision)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Everyone else
User-agent: *
Allow: /
Sitemap: https://www.yourfirm.example/sitemap.xmlStep 3: Check your firewall and CDN
Allowing a bot in robots.txt does nothing if your host, CDN or security plugin blocks it before it reads the file. OpenAI and Perplexity both recommend allowing their published IP ranges as well as the user agent, and Google advises checking that your CDN or hosting does not block Googlebot. Ask your web host or IT provider to review the rules, and keep a note of when you made changes.
Step 4: Do not use robots.txt to hide pages
Google says robots.txt is not a mechanism for keeping a web page out of Google, and that a blocked URL can still be listed without a description if it is linked elsewhere. To keep a page out of search results use noindex or password protection. Confidential client material should never sit on a public URL.
Step 5: Test and wait
- Open yourfirm.example/robots.txt in a browser and confirm the file shows the rules you intended.
- Check your server logs for visits from each documented user agent.
- Wait about 24 hours after a change, since OpenAI and Perplexity both say updates can take that long.
- Re-run your AI answer audit after a few weeks.
The rest of the technical setup is in the technical readiness checklist. For how these crawlers fit into the bigger picture, see how AI answers choose sources.
Common questions
Should a law firm block AI training crawlers?
That is a business and risk decision, not a technical one. Per OpenAI and Anthropic, disallowing GPTBot or ClaudeBot signals that your content should be excluded from training. That is separate from the search crawlers, so you can allow search and block training. Our sample file shows the pattern.
How long do robots.txt changes take to apply?
OpenAI and Perplexity both say changes can take about 24 hours to be reflected. Plan your testing around that.
Do user-initiated fetches follow robots.txt?
It depends on the product. OpenAI says ChatGPT-User visits are initiated by users and robots.txt rules may not apply. Perplexity says Perplexity-User generally ignores robots.txt because a person requested the fetch. Anthropic says blocking Claude-User may reduce visibility for user-directed web search.
Sources
- OpenAI: Overview of OpenAI crawlers
- OpenAI Help Center: Searching the web with ChatGPT
- Perplexity: Perplexity crawlers
- Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google Search Central: Google's common crawlers
- Google Search Central: Introduction to robots.txt
- Google Search Central: AI features and your website
Informational only. This article is general education about search and AI visibility. It is not legal advice, it does not create an attorney-client relationship, and we do not guarantee that any statement in it is accurate, complete or current. No one can guarantee that an AI tool will name your firm. Lawyer advertising rules differ by state, so check your state bar's rules and talk to a qualified attorney before you publish claims about your firm. The update date above shows when we last checked.