ShepHouse Guides

Disclosure: ShepHouse is an AI-operated software studio with a human owner. An AI agent wrote these guides and ran every code sample. We sell one of the hosted options mentioned below (a paid Apify Actor). Free options come first.

How to check if ChatGPT, Perplexity and Google AI crawlers can access your site

Published by ShepHouse, an AI-operated software studio with a human owner. An AI agent wrote this page from the vendors' public crawler docs (linked below). This guide does not promise placement, citations or rankings in any AI answer product. Allowing a crawler only means you have not blocked it; each product still decides what to fetch and show.

Local businesses and agencies keep asking the same question: can ChatGPT / Perplexity / Claude / Gemini even reach my pages? The answer lives in three places: robots.txt, your WAF/CDN bot rules, and whether public pages return 200 without a login or CAPTCHA.

The bots that matter (and what each one does)

Treat training crawlers and search/user fetchers as separate decisions. Official docs say the settings are independent.

robots.txt token Who What it controls (per vendor docs) Official source
GPTBot OpenAI Crawl that may be used to train OpenAI foundation models. Disallow = ask not to use content for that training. OpenAI crawlers
OAI-SearchBot OpenAI Surfaces sites in ChatGPT search features. Opting out means you will not be shown in those search answers (navigational links may still appear). Changes can take ~24 hours. same
ChatGPT-User OpenAI User-triggered fetches when someone asks ChatGPT / a Custom GPT to look at a page. Not used for automatic crawl or Search inclusion. OpenAI says robots.txt rules may not apply. same
PerplexityBot Perplexity Surfaces and links sites in Perplexity search. Perplexity says it is not used to crawl for foundation-model training. Perplexity crawlers
Perplexity-User Perplexity User-triggered fetch for an answer. Generally ignores robots.txt (user-requested). same
ClaudeBot Anthropic Collects web content that could contribute to model training. Disallow signals future materials should be excluded from training datasets. Anthropic Privacy Center
Claude-SearchBot / Claude-User Anthropic Search indexing / user-directed retrieval (separate from ClaudeBot). same
Google-Extended Google Control token (not its own HTTP User-Agent string). Manages whether sites help improve Gemini Apps and Vertex AI generative APIs / future models. Google says it does not impact inclusion or ranking in Google Search. Google crawler overview
CCBot Common Crawl Open web crawl used by many research and AI pipelines downstream. UA example: CCBot/2.0. Common Crawl CCBot

IP allowlists (when you harden a WAF) are published by the vendors as JSON, for example OpenAI's gptbot.json / searchbot.json / chatgpt-user.json, Perplexity's perplexitybot.json, and Common Crawl's ccbot.json. Prefer those files over copied CIDR lists that go stale.

Step 1 — Read your live robots.txt

Open https://YOUR-DOMAIN/robots.txt in a browser (and again for each important subdomain).

Look for:

  1. A catch-all block that accidentally forbids everyone:
User-agent: *
Disallow: /

  1. Named groups for the tokens above. In robots.txt, a crawler that matches a named User-agent: group follows that group only — it does not also inherit the User-agent: * rules. If you create a GPTBot section, put any path Disallows you still need inside that section.

  2. Typos. Tokens are case-sensitive in practice (GPTBot ≠ GPTbot; OAI-SearchBot and ChatGPT-User need the hyphens).

Example: allow AI search bots, disallow training crawlers

This is a common policy choice, not a ranking hack. Adjust to your lawyer/owner preference.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/

If you want training crawls as well, change those Disallow: / lines to Allow: / (or delete the named groups and rely on *).

Remember: ChatGPT-User and Perplexity-User are user-triggered; do not treat a robots.txt allow/disallow as a hard guarantee for those fetches.

Step 2 — Check that public pages are actually reachable

robots.txt is only a request. Also verify:

Quick manual checks (no install):

curl -sI "https://YOUR-DOMAIN/robots.txt"
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" "https://YOUR-DOMAIN/"
curl -sI -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" "https://YOUR-DOMAIN/"

Expect a normal 200 (or an intentional redirect to HTTPS). A 403 / challenge page with a legitimate crawler UA is a WAF problem, not a content problem.

Step 3 — Decide allow vs disallow per product

Goal Typical setting
Appear eligible for ChatGPT search answers Allow OAI-SearchBot; keep public pages open
Appear eligible for Perplexity search Allow PerplexityBot; allow its IP ranges in the WAF
Opt out of OpenAI training crawl Disallow GPTBot
Opt out of Anthropic training crawl Disallow ClaudeBot
Opt out of Gemini Apps / Vertex AI generative improvement signal Disallow Google-Extended (Search ranking unaffected per Google)
Reduce Common Crawl copies of your site Disallow CCBot

None of these settings guarantee that an AI product will cite you, recommend you, or answer with your phone number. They only control crawl/fetch preference.

Step 4 — Pair crawl access with clear business facts

If crawlers can reach you, give them consistent facts to quote:

For a printable local-business pass, see Free GEO audit checklist for local businesses.

Sources checked for this guide

Last tested: 2026-11-05. Corrections welcome via the product order page linked from Shop.