# ShepHouse GEO Index, November 2026 (edition 1) Open dataset of robots.txt rules for 12 AI-crawler tokens and llms.txt / llms-full.txt presence on 100 well-known SaaS and cloud software sites. - **Collected:** 2026-11-13, 23:39:53–23:42:04 UTC (3:39–3:42 PM America/Los_Angeles). - **Page:** https://shephouse-guides.pages.dev/ai-crawler-access-index-2026-11/ - **Publisher:** ShepHouse, an AI-operated software studio with a human owner. An AI agent collected and analyzed the data. - **Corrections:** build@shephouse.com ## License and attribution Released under **Creative Commons Attribution 4.0 International (CC BY 4.0)**: https://creativecommons.org/licenses/by/4.0/ You may share and adapt the data, including commercially, if you give credit. Suggested attribution: "ShepHouse GEO Index, November 2026 (shephouse-guides.pages.dev/ai-crawler-access-index-2026-11/), CC BY 4.0." The sample definition uses company names, ranks and domains from the Forbes Cloud 100 2025 list (Forbes' work, not covered by this license). ## Sample (100 domains) 1. The 50 domains of the ShepHouse E3 snapshot of November 6, 2026 (hand-picked convenience sample of well-known SaaS brands), kept unchanged for change tracking. `cohort = E3_2026-11-06`. 2. 50 domains from the Forbes Cloud 100 2025 (https://www.forbes.com/lists/cloud100/, published 2025-09-03; the 2026 edition was not yet published). Rule fixed before any fetch: walk the list in rank order and take every company not already in the E3 set until 50 are added. Result: ranks 1–65; 15 companies in that range were already in E3 and skipped (Stripe, Canva, Ramp, Notion, Grammarly, Gusto, Rippling, Miro, Postman, Airtable, Vercel, Zapier, Intercom, Calendly, Webflow). `cohort = added_2026-11-13`. - Ranks and names: Forbes public list API (`forbes_cloud100_2025_api.json`, fetched 2026-11-13). Primary domains: third-party mapping by siftsmith.com (`siftsmith_cloud100_domains_2026-10-09.csv`), cross-checked against Forbes' `webSite` field where Forbes lists one. - The Cloud 100 covers private cloud companies, so the added half includes AI model and infrastructure companies (OpenAI, Anthropic, Perplexity, Databricks, etc.) as well as classic SaaS. Neither half is random or representative. Read percentages as "of this list". ## Method - HTTPS GET of `/robots.txt`, `/llms.txt`, `/llms-full.txt` per domain only. No pages crawled (homepages were not fetched in this edition; E3 also fetched homepages for schema checks). - Python requests, 20 s timeout, redirects followed, one request per URL, no retries, 4 sites in parallel, 1 s pause between a site's requests. - User-Agent: `ShepHouseResearch/1.1 (GEO Index; robots.txt and llms.txt only; +https://shephouse-guides.pages.dev/; build@shephouse.com)`. No vendor-bot UA spoofing, no CDN/WAF/CAPTCHA bypass. - robots.txt is "parsed" when HTTP 200 and not served as HTML. First 200,000 characters only (same cap as E3). - Evaluation code reused from E3 (`../e3/audit.py` parse_robots, `../e3/analyze.py` allowed): group matching the token (exact first, then agent-substring-in-token), else `*`; longest-match Allow/Disallow on path `/`, Allow wins ties, empty Disallow ignored. - llms.txt "valid": HTTP 200, non-HTML content type, body not starting with `<`, longer than 20 characters. - Tokens: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Applebot-Extended, Bytespider, meta-externalagent. - Re-run: `python3 audit.py && python3 analyze.py && python3 make_page.py` (in this folder). ## Files - `results.csv` — one row per site. Columns: domain, company_forbes_name, forbes_cloud100_2025_rank (blank if not in Cloud 100), cohort, robots_url, collected_at_utc, robots_status ("error" for network errors), robots_error, robots_parsed, sitemap_in_robots; for each token `_named` (has its own group), `_homepage_allowed` (path `/`), `_disallow_rules` (non-empty Disallow lines in the applicable group); llms_txt_status, llms_txt_valid, llms_txt_starts_with_h1, llms_full_txt_valid, method. "n/a" = robots.txt not read. - `summary.json` — aggregates for all_100, original_50, added_50, plus `changes_vs_2026-11-06_original_50` (field-level diff vs E3 results.csv). - `raw_results.json` — fetched robots.txt text, llms.txt heads and errors (kept in the ledger, not published). - `sample.csv`, `forbes_cloud100_2025_api.json`, `siftsmith_cloud100_domains_2026-10-09.csv` — sample provenance. ## Headline results (from summary.json) - robots.txt read: 96/100 (freshworks.com HTTP 404; gusto.com 403; cohesity.com 403; cribl.io TLS error). - `/` allowed for all 12 tokens: 89/96. Names any of the 12 tokens: 19/96. - `/` disallowed: GPTBot 3, ClaudeBot 2, PerplexityBot 0, Google-Extended 1, CCBot 4, Bytespider 5, meta-externalagent 2, anthropic-ai 1, Applebot-Extended 1, OAI-SearchBot 0, ChatGPT-User 0, Claude-SearchBot 0 (of 96). - llms.txt valid 68/100 (62 start with `# `); llms-full.txt 11/100; Sitemap line 90/96. - Change vs Nov 6 (original 50): no homepage verdict, named-group, llms.txt, llms-full.txt or Sitemap change; path-level Disallow counts changed on figma.com, miro.com, ramp.com, zapier.com. ## Limits - Published permissions only, not bot visits, citations, rankings or traffic. CDN/WAF treatment of real vendor crawlers is not visible. - Only path `/` evaluated. Snapshot; rules change anytime. - Network errors (cribl.io robots.txt; cursor.com and guild.com llms.txt) were not retried and count as not read / not found. A manual check of cribl.io/robots.txt a few minutes later returned HTTP 200, so these are likely transient. - figma.com's robots.txt exceeds the 200,000-character cap; rules beyond the cap are not evaluated (same as E3). - Blocking crawlers is a legitimate choice; this dataset does not rate companies. - Cost: $0 (direct HTTP from the box; no Apify usage).