--- name: "SEO-AI-crawlers (m14)" repo: https://github.com/hainrixz/claude-seo-ai source: https://github.com/hainrixz/claude-seo-ai/blob/HEAD/skills/seo-ai-crawlers/SKILL.md license: "MIT" install: "npx skills add hainrixz/claude-seo-ai --skill seo-ai-crawlers" page: https://100xlab.com/resources/agent-skills/hainrixz-claude-seo-ai--seo-ai-crawlers--1lm4c8q --- # seo-ai-crawlers (M14) Controls whether AI search engines can crawl and cite the page, and whether they can read it without JS. The training-vs-search-vs-fetch distinction is everything. Reference: `references/ai-crawlers.md`. ## Audits Working from the PageSnapshot (`rendered_dom` if present, else `raw_html`) plus the site `robots.txt`: 1. **Citation access**: are retrieval/citation bots — `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Bingbot` — actually allowed (not caught by a broad `Disallow: /` or a wildcard block)? Confirm `Googlebot` is not blocked and the `Googlebot` (search) vs `Google-Extended` (Gemini training control) split is correct. 2. **User-agent classification**: bucket every AI agent in `robots.txt` into **training** (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), **search/retrieval** (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and **user-triggered fetch** (ChatGPT-User, Claude-User, Perplexity-User). Match user-agents case-insensitively; treat the table in `references/ai-crawlers.md` as a starting set, not exhaustive. 3. **Renderability for non-JS crawlers**: pull the M4 (seo-crawl-render) render result — most AI crawlers do not execute JS. If primary content only appears in `rendered_dom` and is absent from `raw_html`, flag it as invisible to AI retrieval. 4. **llms.txt / llms-full.txt** (also covers M21): presence at the site root, valid Markdown structure (H1 title, summary blockquote, sectioned link lists), and that linked URLs resolve. Follow `references/ai-crawlers.md`. ## Fixes - **AUTO** (`fixable: auto`): a citation-friendly `robots.txt` preset, **choice-gated** — the user picks `allow-citations` (default: allow search/retrieval, opt out of training), `allow-all`, or `block-all`. Deterministic, additive, verifiable; emitted as a diff for `fix`. - **AUTO** (`fixable: auto`), **disclosure-gated** and **scored 0**: `llms.txt` / `llms-full.txt`, generated from the site's own structure only on explicit request (`fix --category llms`), shown as a diff before writing. Additive and deterministic, but never sold as proven ranking value — the disclosure that it is low/uncertain impact is shown every time. - **ADVISORY** (`fixable: advisory`): edge/WAF block for bots that ignore `robots.txt` (e.g. Bytespider) — the tool never writes infra config. **Never fabricate** sitemap URLs, contact emails, or link targets — ask the user or leave a clearly-marked `TODO` placeholder. ## Verification - `node ${CLAUDE_SKILL_DIR}/../../scripts/parse-robots-sitemap.mjs --url ` (`robots_parse`) — parses `robots.txt`, resolves the effective directive for each AI user-agent, and confirms it. - `dom_assert` against the M4 render result for the renderability check (content present in `raw_html`). - When the required data tier is unavailable (e.g. `robots.txt` unfetchable, no M4 render result), status is `needs_api` — **never** a false `pass`. ## Findings Findings conform to `schema/finding.schema.json`. `evidence.observed` quotes the page/robots line; `verification.reproduce` is the runnable command above; `expected_impact` is banded + confidence-tagged (no naked %). Examples: - `M14.citation_bots.blocked` — `Disallow: /` reaches `OAI-SearchBot`/`Claude-SearchBot`/`PerplexityBot` (status `fail`, severity 4, `fixable: auto`, axis `ai`, confidence `established`). - `M14.render.content_js_only` — primary content in `rendered_dom` but absent from `raw_html`, invisible to non-JS AI crawlers (status `warn`, severity 4, `fixable: advisory`, axis `ai`, confidence `directional`). - `M14.llmstxt.missing` — no `/llms.txt` at site root (status `warn`, severity 1, `fixable: auto`, axis `ai`, confidence `speculative`, scored 0). ## Honesty - Blocking a **training** bot does NOT block the matching **search** bot — they are separate user-agents (`GPTBot` ≠ `OAI-SearchBot`, `ClaudeBot` ≠ `Claude-SearchBot`). Many "block AI" guides get this wrong. - `Bytespider` and some agents frequently ignore `robots.txt`; a robots rule is best-effort. Real enforcement needs an edge rule / WAF (advisory). - `robots.txt` controls **crawling**, not **indexing** — to keep a page out, use a `noindex` meta tag and don't also `Disallow` it. - `llms.txt` has only **partial** vendor support (Anthropic and Perplexity honor it in retrieval; Google does not use it for AI Overviews/AI Mode; OpenAI uncommitted) and is also useful as IDE/coding-agent context. Still low/uncertain — scored 0, never presented as proven ranking value. --- ## Supporting file: references/ai-crawlers.md # AI crawlers & robots.txt control (2026) The key distinction: **training bots** (used to train models) vs **search/retrieval bots** (fetch pages to answer live queries and may cite you) vs **user-triggered fetchers** (fetch a page because a user pasted/asked about it). You can allow citations while controlling training — but only if you target the right user-agents. ## User-agent reference | User-agent | Operator | Purpose | Controllable via robots.txt | |---|---|---|---| | `Googlebot` | Google | Classic search index (also feeds AI Overviews) | Don't block | | `Google-Extended` | Google | Gemini / Vertex training control | Yes | | `Bingbot` | Microsoft | Bing search (feeds Copilot) | Don't block | | `GPTBot` | OpenAI | Model **training** | Yes | | `OAI-SearchBot` | OpenAI | ChatGPT **search/retrieval** (citations) | Yes — allow for citations | | `ChatGPT-User` | OpenAI | **User-triggered** fetch | Yes (limited effect) | | `ClaudeBot` | Anthropic | Model **training** | Yes | | `Claude-SearchBot` | Anthropic | Claude **search/retrieval** (citations) | Yes — allow for citations | | `Claude-User` / `Claude-Web` | Anthropic | User-triggered fetch | Yes | | `PerplexityBot` | Perplexity | Crawl for retrieval (citations) | Yes — allow for citations | | `Perplexity-User` | Perplexity | User-triggered fetch | Yes | | `Applebot-Extended` | Apple | Apple Intelligence training control | Yes | | `Meta-ExternalAgent` | Meta | Training | Yes | | `Amazonbot` | Amazon | Indexing | Yes | | `CCBot` | Common Crawl | Public dataset (feeds many models) | Yes | | `Bytespider` | ByteDance | Training + search; **often ignores robots.txt** | Unreliable — recommend edge/WAF block | User-agent strings drift; match case-insensitively and treat the table as a starting set. Verify a bot's current docs before making strong claims to a user. ## Presets the fixer can generate (choice-gated, opt-in) **1. Allow citations, control training (recommended default for most sites):** ``` # Search engines — required for ranking User-agent: Googlebot User-agent: Bingbot Disallow: # AI search/retrieval — allow so engines can cite you User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Disallow: # AI training — opt out User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Disallow: / Sitemap: https://example.com/sitemap.xml ``` **2. Allow all** (maximum visibility, includes training): `Disallow:` for everything. **3. Block all AI** (search + training; reduces AI citations to ~zero): `Disallow: /` for every AI agent above. Tell the user this also kills AI-search visibility. ## Honesty notes - Blocking a **training** bot does NOT block the matching **search** bot — they are separate user-agents. Many "block AI" guides get this wrong. - `Bytespider` frequently ignores `robots.txt`; a robots rule is best-effort. Real enforcement needs an edge rule / WAF (advisory). - `robots.txt` controls **crawling**, not **indexing**. To keep a page out of an index use a `noindex` robots meta tag (and don't also `Disallow` it, or the crawler can't see the `noindex`). --- ## Supporting file: schema/finding.schema.json ```json { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://github.com/Hainrixz/claude-seo-ai/schema/finding.schema.json", "title": "claude-seo-ai Finding", "description": "A single audit finding. Falsifiability-first: every finding must be independently observable (evidence) and re-checkable (verification.reproduce). Impact is banded + confidence-tagged, never a fabricated percentage.", "type": "object", "required": ["id", "module", "title", "status", "severity", "scope", "evidence", "expected", "recommendation", "fixable", "verification", "expected_impact"], "additionalProperties": false, "properties": { "id": { "type": "string", "description": "Stable, module-prefixed rule id, e.g. 'M5.article.missing_datemodified'.", "pattern": "^M[0-9]{1,2}[a-z]?\\.[a-z0-9_.-]+$" }, "module": { "type": "string", "description": "Owning module id (M1..M21, optionally with a sub-module letter, e.g. M7b for mobile, M7c for headings; the scorer maps it to its parent category).", "pattern": "^M[0-9]{1,2}[a-z]?$" }, "title": { "type": "string", "minLength": 3, "maxLength": 160 }, "status": { "type": "string", "enum": ["pass", "warn", "fail", "not_applicable", "needs_api"], "description": "needs_api means the check could not be verified without an API key/MCP — it is NEVER a silent pass." }, "severity": { "type": "integer", "minimum": 0, "maximum": 5, "description": "1 = cosmetic, 5 = can tank the score (e.g. site-wide noindex). 0 is reserved for not_applicable findings (no impact; excluded from scoring)." }, "scope": { "type": "string", "enum": ["page", "template", "site"] }, "location": { "type": "object", "additionalProperties": false, "properties": { "url": { "type": "string" }, "file": { "type": "string" }, "selector": { "type": "string" }, "line": { "type": "integer", "minimum": 1 } } }, "evidence": { "type": "object", "description": "The OBSERVED fact — verbatim, reproducible.", "required": ["observed"], "additionalProperties": false, "properties": { "observed": { "type": "string", "minLength": 1 }, "snippet": { "type": "string" } } }, "expected": { "type": "string", "minLength": 1 }, "recommendation": { "type": "string", "minLength": 1 }, "fixable": { "type": "string", "enum": ["auto", "proposed", "advisory"], "description": "auto = deterministic/additive/verifiable write; proposed = draft requiring per-item accept; advisory = never written by the tool." }, "fix_preview": { "type": "string", "description": "Unified diff or new-file content. Present when fixable is auto/proposed." }, "verification": { "type": "object", "description": "How to confirm this finding independently. reproduce MUST be runnable.", "required": ["method", "assertion", "reproduce"], "additionalProperties": false, "properties": { "method": { "type": "string", "enum": ["dom_assert", "header_check", "schema_validator", "rich_results_api", "psi_api", "crux_api", "gsc_api", "robots_parse", "xml_parse", "link_graph", "render_diff", "manual_review"] }, "assertion": { "type": "string", "minLength": 1 }, "reproduce": { "type": "string", "minLength": 1, "description": "A concrete command or assertion a human/CI can run, e.g. 'node scripts/validate-jsonld.mjs --url ...'." } } }, "expected_impact": { "type": "object", "description": "Honest, banded impact. NO naked percentages — published numbers may appear only inside rationale with a citation.", "required": ["axis", "confidence", "magnitude", "rationale"], "additionalProperties": false, "properties": { "axis": { "type": "string", "enum": ["search", "ai", "both"] }, "confidence": { "type": "string", "enum": ["established", "directional", "speculative"], "description": "established = documented by the engine; directional = strong correlational/industry evidence; speculative = plausible, unproven. Speculative findings never cap a score below a band." }, "magnitude": { "type": "string", "enum": ["high", "medium", "low"] }, "rationale": { "type": "string", "minLength": 1 } } }, "doc_ref": { "type": "array", "items": { "type": "string", "format": "uri" } } } } ```