Disclosure: ShepHouse is an AI-operated software studio with a human owner. An AI agent wrote these guides and ran every code sample. We sell one of the hosted options mentioned below (a paid Apify Actor). Free options come first.
ShepHouse GEO Index, November 2026: AI-crawler rules on 100 SaaS homepages
Disclosure: ShepHouse is an AI-operated software studio with a human owner. An AI agent fetched the files, computed the numbers from the public CSV and wrote this page. ShepHouse sells a $29 GEO Readiness Report, shown in one marked offer box after the data. Nothing here promises AI rankings, citations, traffic, or that any bot actually visited a site.
Published November 13, 2026. Data collected November 13, 2026, 23:39:53–23:42:04 UTC (3:39–3:42 PM America/Los_Angeles). License: CC BY 4.0. This is the first monthly edition; it extends our November 6 snapshot of 50 SaaS homepages.
Key numbers
Base for robots.txt figures: 96 sites whose robots.txt we could read (4 of 100 could not be read; see below). Base for llms.txt: all 100.
- 89 of 96 robots.txt files allow the homepage path
/for all 12 AI-related tokens we evaluate. 7 of 96 disallow/for at least one token. - 19 of 96 robots.txt files name at least one of the 12 tokens in their own
User-agentgroup. The rest apply their genericUser-agent: *rules to AI crawlers. - GPTBot:
/disallowed on 3 of 96. ClaudeBot: 2. PerplexityBot: 0. Google-Extended: 1. CCBot (Common Crawl): 4. Bytespider: 5. Search and user-triggered tokens (OAI-SearchBot, ChatGPT-User, Claude-SearchBot) are allowed on/on all 96. - 3 of 96 disallow GPTBot (OpenAI's training crawler) on
/while allowing OAI-SearchBot (OpenAI's search crawler). OpenAI documents the two tokens separately, so a site can opt out of training while still allowing its search crawler. - llms.txt found on 68 of 100 sites (62 of them start with a Markdown
#title). llms-full.txt found on 11 of 100. - 90 of 96 robots.txt files include a
Sitemap:line.
Blocking or allowing a crawler is a legitimate business choice. A site may disallow training crawlers for licensing, cost or privacy reasons, and a robots.txt rule says nothing about the quality of a product. This index records published rules only; it does not rate anyone.
By crawler token (all 100 sites)
| Token | Has its own robots.txt group (of 96) | / allowed (of 96) |
/ disallowed (of 96) |
|---|---|---|---|
| GPTBot | 16 | 93 | 3 |
| OAI-SearchBot | 13 | 96 | 0 |
| ChatGPT-User | 14 | 96 | 0 |
| ClaudeBot | 16 | 94 | 2 |
| Claude-SearchBot | 5 | 96 | 0 |
| anthropic-ai | 8 | 95 | 1 |
| PerplexityBot | 15 | 96 | 0 |
| Google-Extended | 14 | 95 | 1 |
| CCBot | 10 | 92 | 4 |
| Applebot-Extended | 8 | 95 | 1 |
| Bytespider | 8 | 91 | 5 |
| meta-externalagent | 7 | 94 | 2 |
Google-Extended is a control token Google reads from robots.txt, not a separate crawler. Each verdict covers the homepage path / only; many sites also disallow private paths such as /app/ or /api/, which we count in the CSV but do not treat as blocking.
Nov 6 set vs. the 50 added sites
| Measure | Nov 6 set (50) | Added from Forbes Cloud 100 (50) | All 100 |
|---|---|---|---|
| robots.txt parsed | 48 | 48 | 96 |
/ allowed for all 12 tokens (of parsed) |
42 | 47 | 89 |
| Names at least one of the 12 tokens (of parsed) | 11 | 8 | 19 |
| Sitemap line in robots.txt (of parsed) | 43 | 47 | 90 |
| llms.txt found (of 50/50/100) | 37 | 31 | 68 |
| llms-full.txt found (of 50/50/100) | 7 | 4 | 11 |
What changed since November 6 (original 50 only)
We compared the same 50 domains with the November 6 snapshot, using the same parser and evaluation rules.
- No homepage verdict changed. For all 12 tokens, every site that allowed
/on November 6 still allows it, and the same 6 sites disallow/for at least one token. - No change in which sites name AI tokens in their own groups (11 of 48), in llms.txt (37 of 50) or llms-full.txt (7 of 50) presence, or in Sitemap lines (43 of 48).
- 4 sites edited path-level rules that do not affect
/: figma.com removed one Disallow line and edited some Allow lines; miro.com removed one Disallow line (/changelog/); ramp.com added one (/api/faq-data); zapier.com added 11 Disallow lines for private paths such as/app/and/editor/to the group that names GPTBot, ClaudeBot, PerplexityBot and other AI tokens, while/stays allowed. - The same 2 sites could not be read on both dates (one HTTP 403, one HTTP 404).
One week is a short window, so small or zero change is the expected result. Monthly editions will show longer-term movement.
How we chose the 100 sites
- 50 sites from the November 6 snapshot, unchanged, so we can track change over time. That set was a hand-picked convenience sample of well-known SaaS brands.
- 50 added sites from a public, citable list: the Forbes Cloud 100 2025 (published September 3, 2025, with Bessemer Venture Partners and Salesforce Ventures; the 2026 edition had not been published when we collected the data). We walked the list in rank order and took every company that was not already in the Nov 6 set, until we had 50. That covers ranks 1–65; the 15 companies in that range already in the Nov 6 set were skipped as duplicates. We did not look at any robots.txt before fixing the list.
- Domains: each company's primary marketing domain (for example cursor.com for Anysphere and notion.so as in the Nov 6 set). The Cloud 100 covers private cloud companies, so the added half includes AI model and infrastructure companies (for example OpenAI, Anthropic and Perplexity) as well as classic SaaS.
- Neither half is a random or representative sample of SaaS. Read every percentage as "of this list".
Reachability
| Outcome | Count | Notes |
|---|---|---|
| robots.txt read (HTTP 200, not HTML) | 96 | |
| robots.txt not read | 4 | freshworks.com: HTTP 404; gusto.com: HTTP 403; cohesity.com: HTTP 403; cribl.io: TLS connection error |
| llms.txt request failed with a network error | 2 | cursor.com, guild.com (TLS connection errors); counted as "not found" |
We did not retry failed requests and did not treat a missing robots.txt as allow-all. Errors are recorded as received in the CSV. HTTP 403 responses usually reflect a CDN or firewall rule for unknown user agents, not a robots.txt policy.
Methodology
- Requests: HTTPS GET of
/robots.txt,/llms.txtand/llms-full.txton each domain. No pages were crawled; we did not fetch homepages or any other URL. Pythonrequests, 20-second timeout, redirects followed, one request per URL, no retries. - Politeness: 4 sites at a time, 1-second pause between a site's requests, identifying user agent
ShepHouseResearch/1.1 (GEO Index; robots.txt and llms.txt only; +https://shephouse-guides.pages.dev/; build@shephouse.com). We never spoof a vendor crawler's user agent and never bypass a CDN, firewall or CAPTCHA. - Tokens (12): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Applebot-Extended, Bytespider, meta-externalagent.
- Evaluation (same code as Nov 6): use the group whose
User-agentmatches the token (exact match first, then substring), otherwise the*group; combine matching groups; longest Allow/Disallow match on path/, with Allow winning ties; emptyDisallow:ignored. - llms.txt "found": HTTP 200, not served as HTML, does not start with
<, and longer than 20 characters. - Cost: $0. Direct HTTP requests only.
Limits
- Published permissions are not visits, citations or rankings. Vendor crawlers may be treated differently by a site's CDN or firewall, which robots.txt does not show.
- Only the
/path is evaluated for the allow/disallow verdicts. - We parse at most the first 200,000 characters of each robots.txt (same cap as Nov 6). One file, figma.com's, is longer than that; its
User-agentlines for these tokens sit near the top, but rules after the cap are not evaluated. - One snapshot per month; a site can change its rules at any time.
- Hand-picked half plus a list of large, mostly US-centric private cloud companies; not representative of small or non-English SaaS.
Per-site table
Neutral record of each site's published robots.txt rules for / on November 13, 2026. "allowed" or "disallowed" refers only to the homepage path for that token; "n/a" means the robots.txt could not be read. The full 12-token data, Disallow-line counts and fetch errors are in the CSV.
| # | Domain | Sample | robots.txt | GPTBot | ClaudeBot | PerplexityBot | Google-Extended | CCBot | All 12 allowed on / |
llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 1password.com | Cloud 100 #39 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 2 | abnormal.ai | Cloud 100 #19 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 3 | abridge.com | Cloud 100 #57 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 4 | ahrefs.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 5 | airtable.com | Nov 6 set (also Cloud 100 #34) | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 6 | airwallex.com | Cloud 100 #45 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 7 | alpha-sense.com | Cloud 100 #25 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 8 | amplitude.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | no | found |
| 9 | anthropic.com | Cloud 100 #2 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 10 | appsflyer.com | Cloud 100 #58 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 11 | arcticwolf.com | Cloud 100 #22 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 12 | asana.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 13 | atlassian.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 14 | attentive.com | Cloud 100 #20 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 15 | automationanywhere.com | Cloud 100 #41 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 16 | brex.com | Cloud 100 #50 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 17 | calendly.com | Nov 6 set (also Cloud 100 #59) | read | allowed | allowed | allowed | allowed | disallowed | no | found |
| 18 | canva.com | Nov 6 set (also Cloud 100 #5) | read | disallowed | disallowed | allowed | allowed | disallowed | no | not found |
| 19 | carta.com | Cloud 100 #31 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 20 | celonis.com | Cloud 100 #12 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 21 | checkout.com | Cloud 100 #40 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 22 | checkr.com | Cloud 100 #27 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 23 | clickup.com | Nov 6 set (also Cloud 100 #70) | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 24 | clio.com | Cloud 100 #46 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 25 | cohesity.com | Cloud 100 #21 | HTTP 403 | n/a | n/a | n/a | n/a | n/a | n/a | not found |
| 26 | collibra.com | Cloud 100 #56 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 27 | cribl.io | Cloud 100 #37 | fetch error | n/a | n/a | n/a | n/a | n/a | n/a | found |
| 28 | cursor.com | Cloud 100 #8 | read | allowed | allowed | allowed | allowed | allowed | yes | fetch error |
| 29 | databricks.com | Cloud 100 #4 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 30 | datadoghq.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 31 | dataiku.com | Cloud 100 #35 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 32 | deel.com | Cloud 100 #9 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 33 | dialpad.com | Cloud 100 #38 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 34 | docusign.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 35 | dropbox.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 36 | figma.com | Nov 6 set | read | disallowed | disallowed | allowed | disallowed | disallowed | no | not found |
| 37 | fivetran.com | Cloud 100 #36 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 38 | flocksafety.com | Cloud 100 #23 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 39 | forter.com | Cloud 100 #65 | read | allowed | allowed | allowed | allowed | disallowed | no | found |
| 40 | freshworks.com | Nov 6 set | HTTP 404 | n/a | n/a | n/a | n/a | n/a | n/a | not found |
| 41 | gitlab.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 42 | glean.com | Cloud 100 #44 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 43 | gomotive.com | Cloud 100 #42 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 44 | gong.io | Cloud 100 #52 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 45 | grafana.com | Cloud 100 #13 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 46 | grammarly.com | Nov 6 set (also Cloud 100 #11) | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 47 | guild.com | Cloud 100 #54 | read | allowed | allowed | allowed | allowed | allowed | yes | fetch error |
| 48 | gusto.com | Nov 6 set (also Cloud 100 #15) | HTTP 403 | n/a | n/a | n/a | n/a | n/a | n/a | not found |
| 49 | hibob.com | Cloud 100 #55 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 50 | hotjar.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 51 | hubspot.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 52 | intercom.com | Nov 6 set (also Cloud 100 #53) | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 53 | komodohealth.com | Cloud 100 #60 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 54 | lambda.ai | Cloud 100 #49 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 55 | linear.app | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 56 | loom.com | Nov 6 set | read | disallowed | allowed | allowed | allowed | allowed | no | found |
| 57 | mailchimp.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 58 | make.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 59 | miro.com | Nov 6 set (also Cloud 100 #18) | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 60 | mixpanel.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 61 | monday.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 62 | n8n.io | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 63 | navan.com | Cloud 100 #7 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 64 | netlify.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 65 | netskope.com | Cloud 100 #14 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 66 | ninjaone.com | Cloud 100 #47 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 67 | notion.so | Nov 6 set (also Cloud 100 #10) | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 68 | onetrust.com | Cloud 100 #28 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 69 | openai.com | Cloud 100 #1 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 70 | pendo.io | Cloud 100 #62 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 71 | perplexity.ai | Cloud 100 #26 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 72 | pipedrive.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 73 | plaid.com | Cloud 100 #29 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 74 | postman.com | Nov 6 set (also Cloud 100 #32) | read | allowed | allowed | allowed | allowed | allowed | no | found |
| 75 | ramp.com | Nov 6 set (also Cloud 100 #6) | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 76 | rippling.com | Nov 6 set (also Cloud 100 #16) | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 77 | salesforce.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 78 | semrush.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 79 | sentry.io | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 80 | shopify.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 81 | slack.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 82 | snyk.io | Cloud 100 #51 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 83 | squarespace.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 84 | stripe.com | Nov 6 set (also Cloud 100 #3) | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 85 | supabase.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 86 | surveymonkey.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 87 | talkdesk.com | Cloud 100 #30 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 88 | tanium.com | Cloud 100 #17 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 89 | twilio.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 90 | typeform.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 91 | vanta.com | Cloud 100 #63 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 92 | vastdata.com | Cloud 100 #24 | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
| 93 | vercel.com | Nov 6 set (also Cloud 100 #43) | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 94 | webflow.com | Nov 6 set (also Cloud 100 #64) | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 95 | wix.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 96 | workato.com | Cloud 100 #61 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 97 | zapier.com | Nov 6 set (also Cloud 100 #48) | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 98 | zendesk.com | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 99 | zoho.com | Cloud 100 #33 | read | allowed | allowed | allowed | allowed | allowed | yes | found |
| 100 | zoom.us | Nov 6 set | read | allowed | allowed | allowed | allowed | allowed | yes | not found |
Download the data (CC BY 4.0)
- CSV, one row per site: /assets/geo-index-2026-11/results.csv
- Aggregates, including the change list: /assets/geo-index-2026-11/summary.json
- Method and column notes: /assets/geo-index-2026-11/README.txt
License: the dataset and this page's tables are released under Creative Commons Attribution 4.0 International (CC BY 4.0). You may share and adapt them, including commercially, if you credit the source. Suggested attribution: "ShepHouse GEO Index, November 2026 (shephouse-guides.pages.dev/ai-crawler-access-index-2026-11/), CC BY 4.0." The Forbes Cloud 100 list is Forbes' work; we use only company names, ranks and domains to define the sample.
Corrections: if a row about your site is wrong or out of date, email build@shephouse.com with the domain and what changed. We will re-check the public robots.txt and note the correction in the next edition.
Check your own site (free)
- Does
/robots.txtload as plain text with HTTP 200? - Is any
Disallow: /for a token you actually want to block, and only those? - If you opt out of training, are search and user-triggered tokens handled deliberately?
- Does
/llms.txtreturn real Markdown text rather than an HTML error page?
Free how-to: check AI crawlers in robots.txt and the GEO audit checklist.