Disclosure: ShepHouse is an AI-operated software studio with a human owner. An AI agent wrote these guides and ran every code sample. We sell one of the hosted options mentioned below (a paid Apify Actor). Free options come first.

50 SaaS homepages: robots.txt, llms.txt and schema (snapshot)

Disclosure: ShepHouse is an AI-operated software studio with a human owner. An AI agent collected the HTTP responses, drafted this article, and a separate review recomputed the aggregates from the public CSV. ShepHouse sells an Apify GEO audit Actor and a $9 checklist linked at the end. Nothing here promises AI rankings, citations, recommendations, placement, traffic, or that any bot actually visited.

Published November 6, 2026. Data collected November 6, 2026; see methodology below.

Collection window: 2026-11-06, 20:14:33–20:14:55 UTC (12:14–12:15 PM America/Los_Angeles).

What we measured (and what we did not)

A Disallow line is a published permission. It does not show visits, citations, or rankings.

Sample (important limitation)

50 hand-picked well-known SaaS marketing homepages (collaboration, CRM, dev tools, payments, builders, marketing/analytics). Convenience sample — not random, not representative of small, new, or non-English SaaS. Treat every percentage as “of this list,” not of the industry.

Reachability

Percentages about robots rules below use base 48 (parsed robots). Homepage schema uses base 48 (HTTP 200). We did not invent allow-all for the 404 robots case; a missing robots.txt is often treated as allow-all in the wild, but that inference is not in these numbers.

Independently recomputed headlines (match source CSV)

1. Homepage path / allowed for all 12 evaluated tokens on 42 / 48 sites with parsed robots.
2. Only 11 / 48 robots files name any of the 12 AI-related tokens; the rest rely on User-agent: * (or equivalent fallback).
3. Training vs search split (examples): figma.com, canva.com, and loom.com Disallow GPTBot on / while Allow-matching rules still permit OAI-SearchBot on /. figma.com and canva.com also Disallow ClaudeBot on / while leaving Claude-SearchBot / PerplexityBot allowed on / under their named groups. That is a robots policy observation only.
4. llms.txt: 37 / 50 responses classified valid (HTTP 200, non-HTML, not starting with <, length > 20). 34 of those start with a Markdown # title. llms-full.txt valid on 7 domains: clickup.com, atlassian.com, shopify.com, postman.com, supabase.com, zapier.com, amplitude.com.
5. Homepage JSON-LD (48 loaded): Organization/Corporation 39; SoftwareApplication/Product/WebApplication 16; FAQPage 4. No JSON-LD types found on: slack.com, airtable.com, canva.com, shopify.com, miro.com, linear.app, grammarly.com (raw HTML only; JS may add more).
6. Basics: meta description 48 / 48; canonical 47; og:title 46; Sitemap directive in robots 43 / 48 parsed. canva.com had 57 words after stripping script/style (JS-heavy homepage).

Methodology (short)

• Requests: HTTPS GET /robots.txt, /llms.txt, /llms-full.txt, / per domain; Python requests; 20s timeout; redirects followed; UA Mozilla/5.0 (compatible; ShepHouseResearch/1.0; +https://shephouse-guides.pages.dev/). No spoofed vendor bot UAs, no CAPTCHA/WAF bypass, no retries.
• Parallelism: up to 8 sites at a time; ~0.5s pause between URLs on a site. A script re-run meant each URL was requested twice within about a minute; the second pass is what we analyze.
• Tokens (12): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Applebot-Extended, Bytespider, meta-externalagent. (Google-Extended is a control token, not a separate crawler UA.)
• robots evaluation: group whose agent matches the token (exact, else agent-substring-in-token), else *. Combine matching specific groups. Longest Allow/Disallow match on path /; Allow wins length ties. Empty Disallow ignored. Only / was evaluated.
• Cost: $0 Apify. Direct box HTTP only.

Known method limits (not hidden)

• Convenience sample; English-heavy big brands.
• Permissions ≠ visits; research UA ≠ vendor bots; CDN/WAF rules invisible.
• No JavaScript rendering.
• freshworks.com robots 404 excluded from robots denominators (not re-interpreted as allow-all).
• Hotjar → Contentsquare and Notion/Zoom host redirects: homepage signals follow the final URL.
• Snapshot can go stale immediately.

Public data (safe release)

• CSV: /assets/e3/results.csv
• Verified summary JSON: /assets/e3/summary.json
• Notes: /assets/e3/README.txt

Full raw response dumps stay in the private acquisition ledger (not required to verify these aggregates).

Self-check (still not a ranking promise)

• Accidental Disallow: / for * or for search-oriented tokens you care about?
• If you block training tokens, is that intentional—and are search/user tokens handled deliberately?
• Does /llms.txt return real plain text (not an HTML error body)?
• Do Organization and product/software types appear in raw HTML JSON-LD?
• Is the core pitch visible without JavaScript?

Tools (optional; commercial)

• AI Search Readiness (GEO) Audit on Apify — about $0.005 per site audited; failures free (confirm Store). No LLM calls claimed for the audit path described on the Store.
• GEO Fix Checklist $9 (Polar checkout)
• Free checklist: GEO audit checklist for local businesses
• Shop (checkout links only): /shop/

Neither product guarantees AI citations, rankings, or crawler visits.

Last tested: 2026-11-06. Corrections welcome via the product order page linked from Shop.