SEO-AI-crawlers (m14)

seo-ai-crawlers (M14)#

Controls whether AI search engines can crawl and cite the page, and whether they can read it without JS. The training-vs-search-vs-fetch distinction is everything. Reference: references/ai-crawlers.md.

Audits#

Working from the PageSnapshot (rendered_dom if present, else raw_html) plus the site robots.txt:

1. Citation access: are retrieval/citation bots — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot — actually allowed (not caught by a broad Disallow: / or a wildcard block)? Confirm Googlebot is not blocked and the Googlebot (search) vs Google-Extended (Gemini training control) split is correct.
2. User-agent classification: bucket every AI agent in robots.txt into training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), search/retrieval (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered fetch (ChatGPT-User, Claude-User, Perplexity-User). Match user-agents case-insensitively; treat the table in references/ai-crawlers.md as a starting set, not exhaustive.
3. Renderability for non-JS crawlers: pull the M4 (seo-crawl-render) render result — most AI crawlers do not execute JS. If primary content only appears in rendered_dom and is absent from raw_html, flag it as invisible to AI retrieval.
4. llms.txt / llms-full.txt (also covers M21): presence at the site root, valid Markdown structure (H1 title, summary blockquote, sectioned link lists), and that linked URLs resolve. Follow references/ai-crawlers.md.

Fixes#

• AUTO (fixable: auto): a citation-friendly robots.txt preset, choice-gated — the user picks allow-citations (default: allow search/retrieval, opt out of training), allow-all, or block-all. Deterministic, additive, verifiable; emitted as a diff for fix.
• AUTO (fixable: auto), disclosure-gated and scored 0: llms.txt / llms-full.txt, generated from the site's own structure only on explicit request (fix --category llms), shown as a diff before writing. Additive and deterministic, but never sold as proven ranking value — the disclosure that it is low/uncertain impact is shown every time.
• ADVISORY (fixable: advisory): edge/WAF block for bots that ignore robots.txt (e.g. Bytespider) — the tool never writes infra config. Never fabricate sitemap URLs, contact emails, or link targets — ask the user or leave a clearly-marked TODO placeholder.

Verification#

• node ${CLAUDE_SKILL_DIR}/../../scripts/parse-robots-sitemap.mjs --url <u> (robots_parse) — parses robots.txt, resolves the effective directive for each AI user-agent, and confirms it.
• dom_assert against the M4 render result for the renderability check (content present in raw_html).
• When the required data tier is unavailable (e.g. robots.txt unfetchable, no M4 render result), status is needs_api — never a false pass.

Findings#

Findings conform to schema/finding.schema.json. evidence.observed quotes the page/robots line; verification.reproduce is the runnable command above; expected_impact is banded + confidence-tagged (no naked %). Examples:

• M14.citation_bots.blocked — Disallow: / reaches OAI-SearchBot/Claude-SearchBot/PerplexityBot (status fail, severity 4, fixable: auto, axis ai, confidence established).
• M14.render.content_js_only — primary content in rendered_dom but absent from raw_html, invisible to non-JS AI crawlers (status warn, severity 4, fixable: advisory, axis ai, confidence directional).
• M14.llmstxt.missing — no /llms.txt at site root (status warn, severity 1, fixable: auto, axis ai, confidence speculative, scored 0).

Honesty#

• Blocking a training bot does NOT block the matching search bot — they are separate user-agents (GPTBot ≠ OAI-SearchBot, ClaudeBot ≠ Claude-SearchBot). Many "block AI" guides get this wrong.
• Bytespider and some agents frequently ignore robots.txt; a robots rule is best-effort. Real enforcement needs an edge rule / WAF (advisory).
• robots.txt controls crawling, not indexing — to keep a page out, use a noindex meta tag and don't also Disallow it.
• llms.txt has only partial vendor support (Anthropic and Perplexity honor it in retrieval; Google does not use it for AI Overviews/AI Mode; OpenAI uncommitted) and is also useful as IDE/coding-agent context. Still low/uncertain — scored 0, never presented as proven ranking value.

Supporting file: references/ai-crawlers.md#

AI crawlers & robots.txt control (2026)#

The key distinction: training bots (used to train models) vs search/retrieval bots (fetch pages to answer live queries and may cite you) vs user-triggered fetchers (fetch a page because a user pasted/asked about it). You can allow citations while controlling training — but only if you target the right user-agents.

User-agent reference#

User-agent strings drift; match case-insensitively and treat the table as a starting set. Verify a bot's current docs before making strong claims to a user.

Presets the fixer can generate (choice-gated, opt-in)#

1. Allow citations, control training (recommended default for most sites):

# Search engines — required for ranking
User-agent: Googlebot
User-agent: Bingbot
Disallow:

# AI search/retrieval — allow so engines can cite you
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow:

# AI training — opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

2. Allow all (maximum visibility, includes training): Disallow: for everything.

3. Block all AI (search + training; reduces AI citations to ~zero): Disallow: / for every AI agent above. Tell the user this also kills AI-search visibility.

Honesty notes#

• Blocking a training bot does NOT block the matching search bot — they are separate user-agents. Many "block AI" guides get this wrong.
• Bytespider frequently ignores robots.txt; a robots rule is best-effort. Real enforcement needs an edge rule / WAF (advisory).
• robots.txt controls crawling, not indexing. To keep a page out of an index use a noindex robots meta tag (and don't also Disallow it, or the crawler can't see the noindex).

Supporting file: schema/finding.schema.json#

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://github.com/Hainrixz/claude-seo-ai/schema/finding.schema.json",
  "title": "claude-seo-ai Finding",
  "description": "A single audit finding. Falsifiability-first: every finding must be independently observable (evidence) and re-checkable (verification.reproduce). Impact is banded + confidence-tagged, never a fabricated percentage.",
  "type": "object",
  "required": ["id", "module", "title", "status", "severity", "scope", "evidence", "expected", "recommendation", "fixable", "verification", "expected_impact"],
  "additionalProperties": false,
  "properties": {
    "id": {
      "type": "string",
      "description": "Stable, module-prefixed rule id, e.g. 'M5.article.missing_datemodified'.",
      "pattern": "^M[0-9]{1,2}[a-z]?\\.[a-z0-9_.-]+$"
    },
    "module": {
      "type": "string",
      "description": "Owning module id (M1..M21, optionally with a sub-module letter, e.g. M7b for mobile, M7c for headings; the scorer maps it to its parent category).",
      "pattern": "^M[0-9]{1,2}[a-z]?$"
    },
    "title": { "type": "string", "minLength": 3, "maxLength": 160 },
    "status": {
      "type": "string",
      "enum": ["pass", "warn", "fail", "not_applicable", "needs_api"],
      "description": "needs_api means the check could not be verified without an API key/MCP — it is NEVER a silent pass."
    },
    "severity": {
      "type": "integer",
      "minimum": 0,
      "maximum": 5,
      "description": "1 = cosmetic, 5 = can tank the score (e.g. site-wide noindex). 0 is reserved for not_applicable findings (no impact; excluded from scoring)."
    },
    "scope": { "type": "string", "enum": ["page", "template", "site"] },
    "location": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "url": { "type": "string" },
        "file": { "type": "string" },
        "selector": { "type": "string" },
        "line": { "type": "integer", "minimum": 1 }
      }
    },
    "evidence": {
      "type": "object",
      "description": "The OBSERVED fact — verbatim, reproducible.",
      "required": ["observed"],
      "additionalProperties": false,
      "properties": {
        "observed": { "type": "string", "minLength": 1 },
        "snippet": { "type": "string" }
      }
    },
    "expected": { "type": "string", "minLength": 1 },
    "recommendation": { "type": "string", "minLength": 1 },
    "fixable": {
      "type": "string",
      "enum": ["auto", "proposed", "advisory"],
      "description": "auto = deterministic/additive/verifiable write; proposed = draft requiring per-item accept; advisory = never written by the tool."
    },
    "fix_preview": {
      "type": "string",
      "description": "Unified diff or new-file content. Present when fixable is auto/proposed."
    },
    "verification": {
      "type": "object",
      "description": "How to confirm this finding independently. reproduce MUST be runnable.",
      "required": ["method", "assertion", "reproduce"],
      "additionalProperties": false,
      "properties": {
        "method": {
          "type": "string",
          "enum": ["dom_assert", "header_check", "schema_validator", "rich_results_api", "psi_api", "crux_api", "gsc_api", "robots_parse", "xml_parse", "link_graph", "render_diff", "manual_review"]
        },
        "assertion": { "type": "string", "minLength": 1 },
        "reproduce": {
          "type": "string",
          "minLength": 1,
          "description": "A concrete command or assertion a human/CI can run, e.g. 'node scripts/validate-jsonld.mjs --url ...'."
        }
      }
    },
    "expected_impact": {
      "type": "object",
      "description": "Honest, banded impact. NO naked percentages — published numbers may appear only inside rationale with a citation.",
      "required": ["axis", "confidence", "magnitude", "rationale"],
      "additionalProperties": false,
      "properties": {
        "axis": { "type": "string", "enum": ["search", "ai", "both"] },
        "confidence": {
          "type": "string",
          "enum": ["established", "directional", "speculative"],
          "description": "established = documented by the engine; directional = strong correlational/industry evidence; speculative = plausible, unproven. Speculative findings never cap a score below a band."
        },
        "magnitude": { "type": "string", "enum": ["high", "medium", "low"] },
        "rationale": { "type": "string", "minLength": 1 }
      }
    },
    "doc_ref": {
      "type": "array",
      "items": { "type": "string", "format": "uri" }
    }
  }
}