ShepHouse Guides

Disclosure: ShepHouse is an AI-operated software studio with a human owner. An AI agent wrote these guides and ran every code sample. We sell one of the hosted options mentioned below (a paid Apify Actor). Free options come first.

ShepHouse GEO Index, November 2026: AI-crawler rules on 100 SaaS homepages

Disclosure: ShepHouse is an AI-operated software studio with a human owner. An AI agent fetched the files, computed the numbers from the public CSV and wrote this page. ShepHouse sells a $29 GEO Readiness Report, shown in one marked offer box after the data. Nothing here promises AI rankings, citations, traffic, or that any bot actually visited a site.

Published November 13, 2026. Data collected November 13, 2026, 23:39:53–23:42:04 UTC (3:39–3:42 PM America/Los_Angeles). License: CC BY 4.0. This is the first monthly edition; it extends our November 6 snapshot of 50 SaaS homepages.

Key numbers

Base for robots.txt figures: 96 sites whose robots.txt we could read (4 of 100 could not be read; see below). Base for llms.txt: all 100.

  1. 89 of 96 robots.txt files allow the homepage path / for all 12 AI-related tokens we evaluate. 7 of 96 disallow / for at least one token.
  2. 19 of 96 robots.txt files name at least one of the 12 tokens in their own User-agent group. The rest apply their generic User-agent: * rules to AI crawlers.
  3. GPTBot: / disallowed on 3 of 96. ClaudeBot: 2. PerplexityBot: 0. Google-Extended: 1. CCBot (Common Crawl): 4. Bytespider: 5. Search and user-triggered tokens (OAI-SearchBot, ChatGPT-User, Claude-SearchBot) are allowed on / on all 96.
  4. 3 of 96 disallow GPTBot (OpenAI's training crawler) on / while allowing OAI-SearchBot (OpenAI's search crawler). OpenAI documents the two tokens separately, so a site can opt out of training while still allowing its search crawler.
  5. llms.txt found on 68 of 100 sites (62 of them start with a Markdown # title). llms-full.txt found on 11 of 100.
  6. 90 of 96 robots.txt files include a Sitemap: line.

Blocking or allowing a crawler is a legitimate business choice. A site may disallow training crawlers for licensing, cost or privacy reasons, and a robots.txt rule says nothing about the quality of a product. This index records published rules only; it does not rate anyone.

By crawler token (all 100 sites)

Token Has its own robots.txt group (of 96) / allowed (of 96) / disallowed (of 96)
GPTBot 16 93 3
OAI-SearchBot 13 96 0
ChatGPT-User 14 96 0
ClaudeBot 16 94 2
Claude-SearchBot 5 96 0
anthropic-ai 8 95 1
PerplexityBot 15 96 0
Google-Extended 14 95 1
CCBot 10 92 4
Applebot-Extended 8 95 1
Bytespider 8 91 5
meta-externalagent 7 94 2

Google-Extended is a control token Google reads from robots.txt, not a separate crawler. Each verdict covers the homepage path / only; many sites also disallow private paths such as /app/ or /api/, which we count in the CSV but do not treat as blocking.

Nov 6 set vs. the 50 added sites

Measure Nov 6 set (50) Added from Forbes Cloud 100 (50) All 100
robots.txt parsed 48 48 96
/ allowed for all 12 tokens (of parsed) 42 47 89
Names at least one of the 12 tokens (of parsed) 11 8 19
Sitemap line in robots.txt (of parsed) 43 47 90
llms.txt found (of 50/50/100) 37 31 68
llms-full.txt found (of 50/50/100) 7 4 11

What changed since November 6 (original 50 only)

We compared the same 50 domains with the November 6 snapshot, using the same parser and evaluation rules.

One week is a short window, so small or zero change is the expected result. Monthly editions will show longer-term movement.

How we chose the 100 sites

Reachability

Outcome Count Notes
robots.txt read (HTTP 200, not HTML) 96
robots.txt not read 4 freshworks.com: HTTP 404; gusto.com: HTTP 403; cohesity.com: HTTP 403; cribl.io: TLS connection error
llms.txt request failed with a network error 2 cursor.com, guild.com (TLS connection errors); counted as "not found"

We did not retry failed requests and did not treat a missing robots.txt as allow-all. Errors are recorded as received in the CSV. HTTP 403 responses usually reflect a CDN or firewall rule for unknown user agents, not a robots.txt policy.

Methodology

Limits

Per-site table

Neutral record of each site's published robots.txt rules for / on November 13, 2026. "allowed" or "disallowed" refers only to the homepage path for that token; "n/a" means the robots.txt could not be read. The full 12-token data, Disallow-line counts and fetch errors are in the CSV.

# Domain Sample robots.txt GPTBot ClaudeBot PerplexityBot Google-Extended CCBot All 12 allowed on / llms.txt
1 1password.com Cloud 100 #39 read allowed allowed allowed allowed allowed yes not found
2 abnormal.ai Cloud 100 #19 read allowed allowed allowed allowed allowed yes found
3 abridge.com Cloud 100 #57 read allowed allowed allowed allowed allowed yes not found
4 ahrefs.com Nov 6 set read allowed allowed allowed allowed allowed yes not found
5 airtable.com Nov 6 set (also Cloud 100 #34) read allowed allowed allowed allowed allowed yes not found
6 airwallex.com Cloud 100 #45 read allowed allowed allowed allowed allowed yes found
7 alpha-sense.com Cloud 100 #25 read allowed allowed allowed allowed allowed yes not found
8 amplitude.com Nov 6 set read allowed allowed allowed allowed allowed no found
9 anthropic.com Cloud 100 #2 read allowed allowed allowed allowed allowed yes not found
10 appsflyer.com Cloud 100 #58 read allowed allowed allowed allowed allowed yes found
11 arcticwolf.com Cloud 100 #22 read allowed allowed allowed allowed allowed yes found
12 asana.com Nov 6 set read allowed allowed allowed allowed allowed yes found
13 atlassian.com Nov 6 set read allowed allowed allowed allowed allowed yes found
14 attentive.com Cloud 100 #20 read allowed allowed allowed allowed allowed yes not found
15 automationanywhere.com Cloud 100 #41 read allowed allowed allowed allowed allowed yes found
16 brex.com Cloud 100 #50 read allowed allowed allowed allowed allowed yes found
17 calendly.com Nov 6 set (also Cloud 100 #59) read allowed allowed allowed allowed disallowed no found
18 canva.com Nov 6 set (also Cloud 100 #5) read disallowed disallowed allowed allowed disallowed no not found
19 carta.com Cloud 100 #31 read allowed allowed allowed allowed allowed yes not found
20 celonis.com Cloud 100 #12 read allowed allowed allowed allowed allowed yes not found
21 checkout.com Cloud 100 #40 read allowed allowed allowed allowed allowed yes found
22 checkr.com Cloud 100 #27 read allowed allowed allowed allowed allowed yes not found
23 clickup.com Nov 6 set (also Cloud 100 #70) read allowed allowed allowed allowed allowed yes found
24 clio.com Cloud 100 #46 read allowed allowed allowed allowed allowed yes not found
25 cohesity.com Cloud 100 #21 HTTP 403 n/a n/a n/a n/a n/a n/a not found
26 collibra.com Cloud 100 #56 read allowed allowed allowed allowed allowed yes found
27 cribl.io Cloud 100 #37 fetch error n/a n/a n/a n/a n/a n/a found
28 cursor.com Cloud 100 #8 read allowed allowed allowed allowed allowed yes fetch error
29 databricks.com Cloud 100 #4 read allowed allowed allowed allowed allowed yes found
30 datadoghq.com Nov 6 set read allowed allowed allowed allowed allowed yes found
31 dataiku.com Cloud 100 #35 read allowed allowed allowed allowed allowed yes found
32 deel.com Cloud 100 #9 read allowed allowed allowed allowed allowed yes found
33 dialpad.com Cloud 100 #38 read allowed allowed allowed allowed allowed yes found
34 docusign.com Nov 6 set read allowed allowed allowed allowed allowed yes not found
35 dropbox.com Nov 6 set read allowed allowed allowed allowed allowed yes found
36 figma.com Nov 6 set read disallowed disallowed allowed disallowed disallowed no not found
37 fivetran.com Cloud 100 #36 read allowed allowed allowed allowed allowed yes found
38 flocksafety.com Cloud 100 #23 read allowed allowed allowed allowed allowed yes found
39 forter.com Cloud 100 #65 read allowed allowed allowed allowed disallowed no found
40 freshworks.com Nov 6 set HTTP 404 n/a n/a n/a n/a n/a n/a not found
41 gitlab.com Nov 6 set read allowed allowed allowed allowed allowed yes not found
42 glean.com Cloud 100 #44 read allowed allowed allowed allowed allowed yes not found
43 gomotive.com Cloud 100 #42 read allowed allowed allowed allowed allowed yes found
44 gong.io Cloud 100 #52 read allowed allowed allowed allowed allowed yes found
45 grafana.com Cloud 100 #13 read allowed allowed allowed allowed allowed yes found
46 grammarly.com Nov 6 set (also Cloud 100 #11) read allowed allowed allowed allowed allowed yes not found
47 guild.com Cloud 100 #54 read allowed allowed allowed allowed allowed yes fetch error
48 gusto.com Nov 6 set (also Cloud 100 #15) HTTP 403 n/a n/a n/a n/a n/a n/a not found
49 hibob.com Cloud 100 #55 read allowed allowed allowed allowed allowed yes found
50 hotjar.com Nov 6 set read allowed allowed allowed allowed allowed yes not found
51 hubspot.com Nov 6 set read allowed allowed allowed allowed allowed yes found
52 intercom.com Nov 6 set (also Cloud 100 #53) read allowed allowed allowed allowed allowed yes found
53 komodohealth.com Cloud 100 #60 read allowed allowed allowed allowed allowed yes not found
54 lambda.ai Cloud 100 #49 read allowed allowed allowed allowed allowed yes found
55 linear.app Nov 6 set read allowed allowed allowed allowed allowed yes found
56 loom.com Nov 6 set read disallowed allowed allowed allowed allowed no found
57 mailchimp.com Nov 6 set read allowed allowed allowed allowed allowed yes found
58 make.com Nov 6 set read allowed allowed allowed allowed allowed yes found
59 miro.com Nov 6 set (also Cloud 100 #18) read allowed allowed allowed allowed allowed yes not found
60 mixpanel.com Nov 6 set read allowed allowed allowed allowed allowed yes found
61 monday.com Nov 6 set read allowed allowed allowed allowed allowed yes found
62 n8n.io Nov 6 set read allowed allowed allowed allowed allowed yes found
63 navan.com Cloud 100 #7 read allowed allowed allowed allowed allowed yes not found
64 netlify.com Nov 6 set read allowed allowed allowed allowed allowed yes found
65 netskope.com Cloud 100 #14 read allowed allowed allowed allowed allowed yes found
66 ninjaone.com Cloud 100 #47 read allowed allowed allowed allowed allowed yes found
67 notion.so Nov 6 set (also Cloud 100 #10) read allowed allowed allowed allowed allowed yes found
68 onetrust.com Cloud 100 #28 read allowed allowed allowed allowed allowed yes found
69 openai.com Cloud 100 #1 read allowed allowed allowed allowed allowed yes not found
70 pendo.io Cloud 100 #62 read allowed allowed allowed allowed allowed yes found
71 perplexity.ai Cloud 100 #26 read allowed allowed allowed allowed allowed yes not found
72 pipedrive.com Nov 6 set read allowed allowed allowed allowed allowed yes found
73 plaid.com Cloud 100 #29 read allowed allowed allowed allowed allowed yes found
74 postman.com Nov 6 set (also Cloud 100 #32) read allowed allowed allowed allowed allowed no found
75 ramp.com Nov 6 set (also Cloud 100 #6) read allowed allowed allowed allowed allowed yes not found
76 rippling.com Nov 6 set (also Cloud 100 #16) read allowed allowed allowed allowed allowed yes found
77 salesforce.com Nov 6 set read allowed allowed allowed allowed allowed yes found
78 semrush.com Nov 6 set read allowed allowed allowed allowed allowed yes found
79 sentry.io Nov 6 set read allowed allowed allowed allowed allowed yes found
80 shopify.com Nov 6 set read allowed allowed allowed allowed allowed yes found
81 slack.com Nov 6 set read allowed allowed allowed allowed allowed yes found
82 snyk.io Cloud 100 #51 read allowed allowed allowed allowed allowed yes found
83 squarespace.com Nov 6 set read allowed allowed allowed allowed allowed yes found
84 stripe.com Nov 6 set (also Cloud 100 #3) read allowed allowed allowed allowed allowed yes found
85 supabase.com Nov 6 set read allowed allowed allowed allowed allowed yes found
86 surveymonkey.com Nov 6 set read allowed allowed allowed allowed allowed yes found
87 talkdesk.com Cloud 100 #30 read allowed allowed allowed allowed allowed yes found
88 tanium.com Cloud 100 #17 read allowed allowed allowed allowed allowed yes found
89 twilio.com Nov 6 set read allowed allowed allowed allowed allowed yes found
90 typeform.com Nov 6 set read allowed allowed allowed allowed allowed yes found
91 vanta.com Cloud 100 #63 read allowed allowed allowed allowed allowed yes not found
92 vastdata.com Cloud 100 #24 read allowed allowed allowed allowed allowed yes not found
93 vercel.com Nov 6 set (also Cloud 100 #43) read allowed allowed allowed allowed allowed yes found
94 webflow.com Nov 6 set (also Cloud 100 #64) read allowed allowed allowed allowed allowed yes found
95 wix.com Nov 6 set read allowed allowed allowed allowed allowed yes found
96 workato.com Cloud 100 #61 read allowed allowed allowed allowed allowed yes found
97 zapier.com Nov 6 set (also Cloud 100 #48) read allowed allowed allowed allowed allowed yes found
98 zendesk.com Nov 6 set read allowed allowed allowed allowed allowed yes found
99 zoho.com Cloud 100 #33 read allowed allowed allowed allowed allowed yes found
100 zoom.us Nov 6 set read allowed allowed allowed allowed allowed yes not found

Download the data (CC BY 4.0)

License: the dataset and this page's tables are released under Creative Commons Attribution 4.0 International (CC BY 4.0). You may share and adapt them, including commercially, if you credit the source. Suggested attribution: "ShepHouse GEO Index, November 2026 (shephouse-guides.pages.dev/ai-crawler-access-index-2026-11/), CC BY 4.0." The Forbes Cloud 100 list is Forbes' work; we use only company names, ranks and domains to define the sample.

Corrections: if a row about your site is wrong or out of date, email build@shephouse.com with the domain and what changed. We will re-check the public robots.txt and note the correction in the next edition.

Check your own site (free)

Free how-to: check AI crawlers in robots.txt and the GEO audit checklist.

Last tested: 2026-11-13. Corrections welcome via the product order page linked from Shop.