Technical SEO audit

Technical SEO Audit#

Categories#

1. Crawlability#

• robots.txt: exists, valid, not blocking important resources
• XML sitemap: run claude-seo run sitemap_discovery.py <url> --json; require a valid entry in found, and report stale or unsafe robots.txt declarations separately from working fallback locations
• Noindex tags: intentional vs accidental
• Crawl depth: important pages within 3 clicks of homepage
• JavaScript rendering: check if critical content requires JS execution
• Crawl budget: for large sites (>10k pages), efficiency matters
• Googlebot fetch limits: Googlebot fetches the first 2MB of HTML and first 64MB of a PDF (uncompressed; 15MB is the broader crawler-infra default). Long-standing, not a 2026 change, but inline base64 images, oversized inline CSS/JS, or bloated nav can push critical content/JSON-LD past the cap and out of the index. Keep key content + structured data within the first 2MB.
• Crawl rate auto-adjusts (backs off on 5xx/slow responses); there is no manual crawl-rate control (the legacy Search Console setting was removed Jan 2024). Influence crawling via sitemaps, server responsiveness, and robots controls.
• Google's canonical crawling/robots reference moved to developers.google.com/crawling (migrated 2025-11-20); IP-range files relocated to /crawling/ipranges/ and googlebot.json was renamed common-crawlers.json.

AI Crawler Management#

As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.

Known AI crawlers:

Key distinctions:

• Blocking Google-Extended prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use Googlebot)
• Blocking GPTBot prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (ChatGPT-User)
• ~3-5% of websites now use AI-specific robots.txt rules

Example, selective AI crawler blocking:

# Allow search indexing, block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow all other crawlers (including Googlebot for search)
User-agent: *
Allow: /

Recommendation: Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the seo-geo skill for the full AI crawler/fetcher taxonomy.

User-triggered fetchers ignore robots.txt by design. Google now documents Google-Agent (Project Mariner, agentic browsing) plus Google-NotebookLM and Google Messages as user-triggered fetchers that cannot be blocked via robots.txt. Use server-side access controls instead. By contrast, Google-Extended and Google-CloudVertexBot obey robots.txt. Emerging: Web Bot Auth (RFC 9421) lets bots authenticate cryptographically via a Signature-Agent header + key directory at agent.bot.goog (used by Google-Agent); reverse-DNS verification remains the fallback.

2. Indexability#

• Canonical tags: self-referencing, no conflicts with noindex
• Duplicate content: near-duplicates, parameter URLs, www vs non-www
• Thin content: pages below minimum word counts per type
• Pagination: rel=next/prev or load-more pattern
• Hreflang: correct for multi-language/multi-region sites
• Index bloat: unnecessary pages consuming crawl budget

3. Security#

• HTTPS: enforced, valid SSL certificate, no mixed content
• Security headers:
  • Content-Security-Policy (CSP)
  • Strict-Transport-Security (HSTS)
  • X-Frame-Options
  • X-Content-Type-Options
  • Referrer-Policy
• HSTS preload: check preload list inclusion for high-security sites
• Back-button hijacking (spam-policy violation, malicious practices): flag pages that defeat the Back button via history.pushState/replaceState (including scripts injected by third-party ad/library platforms). Added to Google's spam policies 2026-04-13; enforcement live since 2026-06-15 (manual actions + automated demotions): treat as Critical.

4. URL Structure#

• Clean URLs: descriptive, hyphenated, no query parameters for content
• Hierarchy: logical folder structure reflecting site architecture
• Redirects: no chains (max 1 hop), 301 for permanent moves
• URL length: flag >100 characters
• Trailing slashes: consistent usage

5. Mobile Optimization & Page Experience#

• Responsive design: viewport meta tag, responsive CSS
• Touch targets: minimum 48x48px with 8px spacing
• Font size: minimum 16px base
• No horizontal scroll
• Mobile-first indexing: Googlebot Smartphone is the primary crawler (rollout completed 2024). A mobile version is not strictly required (Google says "very strongly recommended"), sites that don't work on mobile can still be indexed, but the real risk is content/parity loss, not hard exclusion.
• Mobile/desktop content parity (highest-value mobile check): equivalent primary content, matching robots meta tags, matching titles/descriptions, equivalent structured data, crawlable resources; avoid lazy-loading primary content that requires user interaction.
• Intrusive interstitials / ad density: flag full-page interstitials, standalone consent-redirect pages, persistent blocking dialogs, and excessive/distracting ad density (a named page-experience aspect). Acceptable: small banners, standard CMS/legal dialogs.
• "Read more" deep links: keep key content immediately visible on load (not behind tabs/accordions), don't hijack scroll on load, and preserve URL hash fragments, content hidden behind expandable sections is less likely to qualify.

Page experience is guidance, not a single ranking system. Only Core Web Vitals feeds ranking directly; HTTPS is a confirmed but lightweight signal (affects <~1% of queries). Relevance can still win even when page experience is sub-par, so don't over-weight security headers. Note: the standalone Page Experience report was removed from Search Console (monitor via the Core Web Vitals + HTTPS reports).

6. Core Web Vitals#

• LCP (Largest Contentful Paint): target <=2.5s
• INP (Interaction to Next Paint): target <=200ms
  • INP replaced FID on March 12, 2024. FID was removed from Chrome's field-data tools (CrUX API, PageSpeed Insights) on September 9, 2024 (Lighthouse is a lab tool that never reported FID). Do NOT reference FID anywhere.
• CLS (Cumulative Layout Shift): target <=0.1
• Evaluation uses 75th percentile of real user data
• Use PageSpeed Insights API or CrUX data if MCP available

7. Structured Data#

• Detection: JSON-LD (preferred), Microdata, RDFa
• Validation against Google's supported types
• See seo-schema skill for full analysis

8. JavaScript Rendering#

• Check if content visible in initial HTML vs requires JS
• Identify client-side rendered (CSR) vs server-side rendered (SSR)
• Flag SPA frameworks (React, Vue, Angular) that may cause indexing issues
• Verify dynamic rendering setup if applicable

JavaScript SEO: Canonical & Indexing Guidance (December 2025)#

Google updated its JavaScript SEO documentation in December 2025 with critical clarifications:

1. Canonical conflicts: If a canonical tag in raw HTML differs from one injected by JavaScript, Google may use EITHER one. Ensure canonical tags are identical between server-rendered HTML and JS-rendered output.
2. noindex with JavaScript: If raw HTML contains <meta name="robots" content="noindex"> but JavaScript removes it, Google MAY still honor the noindex from raw HTML. Serve correct robots directives in the initial HTML response.
3. Non-200 status codes: Google does NOT render JavaScript on pages returning non-200 HTTP status codes. Any content or meta tags injected via JS on error pages will be invisible to Googlebot.
4. Structured data in JavaScript: Product, Article, and other structured data injected via JS may face delayed processing. For time-sensitive structured data (especially e-commerce Product markup), include it in the initial server-rendered HTML.

Best practice: Serve critical SEO elements (canonical, meta robots, structured data, title, meta description) in the initial server-rendered HTML rather than relying on JavaScript injection.

9. IndexNow Protocol#

• Check if site supports IndexNow for Bing, Yandex, Naver
• Supported by search engines other than Google
• Recommend implementation for faster indexing on non-Google engines

Agent-Friendly Pages & Agentic Browsing#

AI agents (not just AI summarizers) increasingly read sites through three channels: vision models on screenshots, raw HTML/DOM, and the accessibility tree (the cleanest signal). Audit criteria: semantic HTML (real <button> and <a>, not <div onclick>), label associations, interactive target sizing, layout stability across templates, cursor: pointer correctness, live in references/agent-friendly-pages.md.

Google now ships a Lighthouse Agentic Browsing category (default-on since Lighthouse 13.3.0, Chrome 150+; buckets: agent-centric accessibility, CLS + llms.txt, three WebMCP audits). It reports a fractional pass-ratio (X of N), not a 0-100 score, keep that distinct from this skill's own Agent-UX 0-100 heuristic below. The PSI REST API does not expose it; run via Lighthouse CLI --only-categories=agentic-browsing, DevTools, or the PSI web UI. See references/agent-friendly-pages.md.

Audit command#

# Render with Playwright + capture accessibility tree, then score
claude-seo run agent_ux_check.py https://example.com --json

The scanner outputs an Agent-UX score (0-100) plus itemized issues:

• HTML findings: real buttons / anchors, <div onclick> widgets, semantic landmarks, inputs without <label for>, inputs without ARIA labels
• Accessibility tree findings: total nodes, interactive nodes, unnamed interactive elements, role="generic" ratio

The accessibility-tree snapshot uses Playwright's page.accessibility.snapshot(interesting_only=False). To capture the tree without scoring, use claude-seo run render_page.py <url> --a11y-tree --json.

Surface findings as opportunities, not failures; don't gate audits on a sub-100 Agent-UX score. WebMCP origin-trial/sign-up status needs verification, and absence of WebMCP support is still an opportunity, not a defect.

Output#

Technical Score: XX/100#

Category Breakdown#

Critical Issues (fix immediately)#

High Priority (fix within 1 week)#

Medium Priority (fix within 1 month)#

Low Priority (backlog)#

DataForSEO Integration (Optional)#

If DataForSEO MCP tools are available, use on_page_instant_pages for real page analysis (status codes, page timing, broken links, on-page checks), on_page_lighthouse for Lighthouse audits (performance, accessibility, SEO scores), and domain_analytics_technologies_domain_technologies for technology stack detection.

Google API Integration (Optional)#

If Google API credentials are configured, use claude-seo run pagespeed_check.py <url> --json for real PSI + CrUX field data (replaces lab-only CWV estimates), claude-seo run crux_history.py <url> --json for 25-week CWV trends, and claude-seo run gsc_inspect.py <url> --json for real indexation status per URL.

Error Handling#

Supporting file: LICENSE.txt#

MIT License - see repository root LICENSE file for complete terms.

Copyright (c) 2026 AgriciDaniel
https://github.com/AgriciDaniel/claude-seo

Supporting file: references/agent-friendly-pages.md#

Agent-friendly pages — audit reference (June 2026)#

The next wave of AI search is not summarization — it's agents acting on the user's behalf (search, compare, buy, book). Google's AI optimization guide and the linked web.dev article describe three channels through which agents interpret your site:

1. Screenshots + vision model — interprets visual hierarchy, button prominence, layout. Slow and token-expensive.
2. Raw HTML / DOM — nesting, IDs, classes, data attributes.
3. The accessibility tree — the browser-native semantic distillation (roles, names, states). The cleanest signal of the three.

Modern agents combine all three. Optimizing for the accessibility tree is the single highest-leverage move; if your accessibility tree is broken, no amount of visual polish saves you.

Primary sources:

• Google AI optimization guide: https://developers.google.com/search/docs/fundamentals/ai-optimization-guide (opens in a new tab)
• web.dev (referenced from the guide above): article on building agent-friendly websites

Audit checklist#

1. Use real interactive elements#

If you cannot use a real interactive tag, supply ARIA: role="button", role="link", tabindex="0", plus key handlers for Enter and Space.

Why it matters: the accessibility tree exposes real interactive elements with their roles. Custom div widgets often appear in the tree with no role at all — agents skip them.

2. Label associations#

Every form input must have an associated label:

<label for="email">Email</label>
<input id="email" type="email" name="email">

Or use aria-label / aria-labelledby where a visible label isn't possible. Agents that read the accessibility tree get the field purpose directly from the associated label — without it, the input is a void.

3. Interactive target size#

Visual-analysis pipelines filter out interactive elements smaller than ~8 square pixels of unobscured area. Tap-target accessibility minimums (24×24 WCAG AA, 44×44 Apple HIG) are stricter and pass the agent gate by default.

Audit: any clickable element below 24×24px is a candidate for agent invisibility, in addition to the WCAG failure.

4. Don't cover interactive nodes with transparent overlays#

Vision models discard covered nodes when computing what's "interactive at this position". Common offenders:

• Full-card click handlers that overlay every child link.
• Transparent cookie-consent layers persisting beyond consent.
• Modal portals with pointer-events: auto left on after dismiss.
• "Ghost" tracking pixels with position: absolute; inset: 0.

5. Layout stability#

If "Add to cart" lives in different positions on /category/shoes vs /category/bags, screenshot-based agents have to relearn the page each visit. Keep functionally identical actions in the same screen quadrant across templates.

Cross-reference: this overlaps with CLS (Cumulative Layout Shift) in Core Web Vitals, but the agent-UX concern is broader — it covers page-to-page stability, not just within-page shift.

6. cursor: pointer as a legitimate signal#

Vision models read cursor: pointer (set by default on <a> / <button>) as a hint that an element is actionable. Do not override it to cursor: default on truly interactive elements just for visual minimalism.

Inverse: do not apply cursor: pointer to non-interactive elements — that makes agents click things that do nothing.

7. Stable, meaningful selectors#

Agents that fall back to DOM parsing rely on:

• Real semantic tags (<nav>, <main>, <article>, <section>, <aside>)
• Stable id attributes on top-level layout containers
• data-* attributes that describe purpose, not implementation

Avoid auto-generated class names like __sc_a4b7d9e2 as the only handle on a critical interactive element — agents can target them but cannot tell what they mean.

Lighthouse Agentic Browsing category (shipped)#

Google now ships a dedicated Agentic Browsing Lighthouse category — created in Lighthouse 13.2.0 (2026-05-01) and on by default since 13.3.0 (2026-05-07) in DevTools and the CLI (Chrome 150+). Category id: agentic-browsing. Unlike other categories it reports a fractional pass-ratio (X of N checks passed), not a 0–100 weighted score — so do not compute or report a weighted agentic "score". It groups deterministic audits into three buckets:

• Agent-centric accessibility — reuses three a11y audits: Names and labels (every interactive element has a programmatic name), Tree integrity (valid roles + parent-child relationships), Visibility (interactive content not hidden from the accessibility tree). These map directly onto checklist items 1–2 above and the agent_ux_check.py heuristics.
• Stability & discoverability — CLS (visual stability for element positioning; see checklist item 5) + an llms.txt presence-at-domain-root check (13.4.0 relaxed it to allow leading whitespace).
• WebMCP integration — three audits, see below.

Run paths: DevTools Lighthouse panel (Chrome 150+, default-on, no toggle); CLI npx lighthouse@latest <url> --only-categories=agentic-browsing (Node.js 22.19+, per-audit JSON for CI); PageSpeed Insights web UI and API. Lighthouse 13.4.1 enabled the category through the PSI API after 13.4.0 had disabled it. CrUX never provides these lab audits.

WebMCP (proposed standard; status needs recheck)#

WebMCP lets a site declare structured tools/actions for agents rather than agents inferring intent from the DOM. It is no longer "years away": a flag-gated Early Preview (Chrome 146 Canary) opened 2026-02-10, and Chrome 149 origin-trial/sign-up status needs verification from a Google-owned source (local dev flag: chrome://flags/#enable-webmcp-testing). It remains a proposed standard from the Web Machine Learning community group (github.com/webmachinelearning/webmcp), not W3C-finalized.

Lighthouse 13.2+ ships three WebMCP audits under the Agentic Browsing category (require Chrome 150+ and origin-trial registration). Shipped audit ids in the 13.2.0 release notes:

• webmcp-form-coverage: informational; flags <form> elements lacking toolname / tooldescription.
• webmcp-registered-tools: informational; lists tools registered via the declarative (HTML) and imperative (JS) APIs.
• webmcp-schema-validity: validates WebMCP form schemas.

(Code parsing Lighthouse JSON should expect the release-note ids above.)

Audit posture: WebMCP is an active opportunity, not a finding to hard-fail — absence is not a defect. Surface it as an opportunity for sites that want first-class agent actions, and note origin-trial enrollment is required for the audits to fire.

Quick-audit one-liner#

For a fast smoke check, capture the accessibility tree via Lighthouse or Chrome DevTools and look for:

• Any interactive element with role="generic" → broken semantics.
• Any input without an accessible name → missing label.
• Any <div> with onclick and no role / tabindex → custom widget that agents won't see.

claude-seo run render_page.py <URL> --mode auto --a11y-tree --json loads the page headlessly and captures Chromium's full accessibility tree through CDP. Use claude-seo run agent_ux_check.py <URL> --json for the bounded Agent-UX heuristic and its explicit complete, partial, or unavailable score status.

Last verified#

2026-08-25. Google has published the Lighthouse Agentic Browsing framework, Lighthouse 13.4.1 exposes it through the PSI API, and the local renderer now captures the Chromium accessibility tree through CDP. Update when:

• WebMCP graduates from origin trial to a stable/shipped API (or W3C status changes).
• Google adds or renames audits in the agentic-browsing category.
• web.dev publishes a follow-up article with revised criteria.