---
name: "Technical SEO audit"
repo: https://github.com/agricidaniel/claude-seo
source: https://github.com/agricidaniel/claude-seo/blob/HEAD/skills/seo-technical/SKILL.md
license: "MIT"
install: "npx skills add agricidaniel/claude-seo --skill seo-technical"
page: https://100xlab.com/resources/agent-skills/agricidaniel-claude-seo--seo-technical--79u74h
---

# Technical SEO Audit

## Categories

### 1. Crawlability
- robots.txt: exists, valid, not blocking important resources
- XML sitemap: run `claude-seo run sitemap_discovery.py <url> --json`; require a
  valid entry in `found`, and report stale or unsafe robots.txt declarations
  separately from working fallback locations
- Noindex tags: intentional vs accidental
- Crawl depth: important pages within 3 clicks of homepage
- JavaScript rendering: check if critical content requires JS execution
- Crawl budget: for large sites (>10k pages), efficiency matters
- Googlebot **fetch limits**: Googlebot fetches the first **2MB of HTML** and first **64MB of a PDF** (uncompressed; 15MB is the broader crawler-infra default). Long-standing, not a 2026 change, but inline base64 images, oversized inline CSS/JS, or bloated nav can push critical content/JSON-LD past the cap and out of the index. Keep key content + structured data within the first 2MB.
- Crawl rate **auto-adjusts** (backs off on 5xx/slow responses); there is **no manual crawl-rate control** (the legacy Search Console setting was removed Jan 2024). Influence crawling via sitemaps, server responsiveness, and robots controls.
- Google's canonical crawling/robots reference moved to **developers.google.com/crawling** (migrated 2025-11-20); IP-range files relocated to `/crawling/ipranges/` and `googlebot.json` was renamed `common-crawlers.json`.

#### AI Crawler Management

As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.

**Known AI crawlers:**

| Crawler | Company | robots.txt token | Purpose |
|---------|---------|-----------------|---------|
| GPTBot | OpenAI | `GPTBot` | Model training |
| ChatGPT-User | OpenAI | `ChatGPT-User` | Real-time browsing |
| ClaudeBot | Anthropic | `ClaudeBot` | Model training |
| PerplexityBot | Perplexity | `PerplexityBot` | Search index + training |
| Bytespider | ByteDance | `Bytespider` | Model training |
| Google-Extended | Google | `Google-Extended` | Gemini training (NOT search) |
| CCBot | Common Crawl | `CCBot` | Open dataset |

**Key distinctions:**
- Blocking `Google-Extended` prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use `Googlebot`)
- Blocking `GPTBot` prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (`ChatGPT-User`)
- ~3-5% of websites now use AI-specific robots.txt rules

**Example, selective AI crawler blocking:**
```
# Allow search indexing, block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow all other crawlers (including Googlebot for search)
User-agent: *
Allow: /
```

**Recommendation:** Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the `seo-geo` skill for the full AI crawler/fetcher taxonomy.

> **User-triggered fetchers ignore robots.txt by design.** Google now documents **Google-Agent** (Project Mariner, agentic browsing) plus **Google-NotebookLM** and **Google Messages** as *user-triggered* fetchers that **cannot be blocked via robots.txt**. Use server-side access controls instead. By contrast, `Google-Extended` and `Google-CloudVertexBot` obey robots.txt. Emerging: **Web Bot Auth** (RFC 9421) lets bots authenticate cryptographically via a `Signature-Agent` header + key directory at `agent.bot.goog` (used by Google-Agent); reverse-DNS verification remains the fallback.

### 2. Indexability
- Canonical tags: self-referencing, no conflicts with noindex
- Duplicate content: near-duplicates, parameter URLs, www vs non-www
- Thin content: pages below minimum word counts per type
- Pagination: rel=next/prev or load-more pattern
- Hreflang: correct for multi-language/multi-region sites
- Index bloat: unnecessary pages consuming crawl budget

### 3. Security
- HTTPS: enforced, valid SSL certificate, no mixed content
- Security headers:
  - Content-Security-Policy (CSP)
  - Strict-Transport-Security (HSTS)
  - X-Frame-Options
  - X-Content-Type-Options
  - Referrer-Policy
- HSTS preload: check preload list inclusion for high-security sites
- **Back-button hijacking** (spam-policy violation, malicious practices): flag pages that defeat the Back button via `history.pushState`/`replaceState` (including scripts injected by third-party ad/library platforms). Added to Google's spam policies 2026-04-13; **enforcement live since 2026-06-15** (manual actions + automated demotions): treat as Critical.

### 4. URL Structure
- Clean URLs: descriptive, hyphenated, no query parameters for content
- Hierarchy: logical folder structure reflecting site architecture
- Redirects: no chains (max 1 hop), 301 for permanent moves
- URL length: flag >100 characters
- Trailing slashes: consistent usage

### 5. Mobile Optimization & Page Experience
- Responsive design: viewport meta tag, responsive CSS
- Touch targets: minimum 48x48px with 8px spacing
- Font size: minimum 16px base
- No horizontal scroll
- Mobile-first indexing: Googlebot Smartphone is the primary crawler (rollout completed 2024). A mobile version is **not strictly required** (Google says "very strongly recommended"), sites that don't work on mobile can still be indexed, but the real risk is **content/parity loss**, not hard exclusion.
- **Mobile/desktop content parity** (highest-value mobile check): equivalent primary content, matching robots meta tags, matching titles/descriptions, equivalent structured data, crawlable resources; avoid lazy-loading primary content that requires user interaction.
- **Intrusive interstitials / ad density**: flag full-page interstitials, standalone consent-redirect pages, persistent blocking dialogs, and excessive/distracting ad density (a named page-experience aspect). Acceptable: small banners, standard CMS/legal dialogs.
- **"Read more" deep links**: keep key content **immediately visible on load** (not behind tabs/accordions), don't hijack scroll on load, and preserve URL hash fragments, content hidden behind expandable sections is less likely to qualify.

> **Page experience is guidance, not a single ranking system.** Only **Core Web Vitals** feeds ranking directly; **HTTPS** is a confirmed but lightweight signal (affects <~1% of queries). Relevance can still win even when page experience is sub-par, so don't over-weight security headers. Note: the standalone **Page Experience report was removed** from Search Console (monitor via the Core Web Vitals + HTTPS reports).

### 6. Core Web Vitals
- **LCP** (Largest Contentful Paint): target <=2.5s
- **INP** (Interaction to Next Paint): target <=200ms
  - INP replaced FID on March 12, 2024. FID was removed from Chrome's field-data tools (CrUX API, PageSpeed Insights) on September 9, 2024 (Lighthouse is a lab tool that never reported FID). Do NOT reference FID anywhere.
- **CLS** (Cumulative Layout Shift): target <=0.1
- Evaluation uses 75th percentile of real user data
- Use PageSpeed Insights API or CrUX data if MCP available

### 7. Structured Data
- Detection: JSON-LD (preferred), Microdata, RDFa
- Validation against Google's supported types
- See seo-schema skill for full analysis

### 8. JavaScript Rendering
- Check if content visible in initial HTML vs requires JS
- Identify client-side rendered (CSR) vs server-side rendered (SSR)
- Flag SPA frameworks (React, Vue, Angular) that may cause indexing issues
- Verify dynamic rendering setup if applicable

#### JavaScript SEO: Canonical & Indexing Guidance (December 2025)

Google updated its JavaScript SEO documentation in December 2025 with critical clarifications:

1. **Canonical conflicts:** If a canonical tag in raw HTML differs from one injected by JavaScript, Google may use EITHER one. Ensure canonical tags are identical between server-rendered HTML and JS-rendered output.
2. **noindex with JavaScript:** If raw HTML contains `<meta name="robots" content="noindex">` but JavaScript removes it, Google MAY still honor the noindex from raw HTML. Serve correct robots directives in the initial HTML response.
3. **Non-200 status codes:** Google does NOT render JavaScript on pages returning non-200 HTTP status codes. Any content or meta tags injected via JS on error pages will be invisible to Googlebot.
4. **Structured data in JavaScript:** Product, Article, and other structured data injected via JS may face delayed processing. For time-sensitive structured data (especially e-commerce Product markup), include it in the initial server-rendered HTML.

**Best practice:** Serve critical SEO elements (canonical, meta robots, structured data, title, meta description) in the initial server-rendered HTML rather than relying on JavaScript injection.

### 9. IndexNow Protocol
- Check if site supports IndexNow for Bing, Yandex, Naver
- Supported by search engines other than Google
- Recommend implementation for faster indexing on non-Google engines

## Agent-Friendly Pages & Agentic Browsing

AI agents (not just AI summarizers) increasingly read sites through three
channels: vision models on screenshots, raw HTML/DOM, and the **accessibility
tree** (the cleanest signal). Audit criteria: semantic HTML (real `<button>`
and `<a>`, not `<div onclick>`), label associations, interactive target sizing,
layout stability across templates, `cursor: pointer` correctness, live in
`references/agent-friendly-pages.md`.

Google now ships a Lighthouse **Agentic Browsing** category (default-on since
Lighthouse 13.3.0, Chrome 150+; buckets: agent-centric accessibility, CLS +
llms.txt, three WebMCP audits). It reports a **fractional pass-ratio (X of N),
not a 0-100 score**, keep that distinct from this skill's own Agent-UX 0-100
heuristic below. The PSI REST API does not expose it; run via Lighthouse CLI
`--only-categories=agentic-browsing`, DevTools, or the PSI web UI. See
`references/agent-friendly-pages.md`.

### Audit command

```bash
# Render with Playwright + capture accessibility tree, then score
claude-seo run agent_ux_check.py https://example.com --json
```

The scanner outputs an Agent-UX score (0-100) plus itemized issues:
- HTML findings: real buttons / anchors, `<div onclick>` widgets, semantic
  landmarks, inputs without `<label for>`, inputs without ARIA labels
- Accessibility tree findings: total nodes, interactive nodes, unnamed
  interactive elements, `role="generic"` ratio

The accessibility-tree snapshot uses Playwright's
`page.accessibility.snapshot(interesting_only=False)`. To capture the tree
without scoring, use `claude-seo run render_page.py <url> --a11y-tree --json`.

Surface findings as **opportunities**, not failures; don't gate audits on a
sub-100 Agent-UX score. WebMCP origin-trial/sign-up status needs verification,
and absence of WebMCP support is still an opportunity, not a defect.

## Output

### Technical Score: XX/100

### Category Breakdown
| Category | Status | Score |
|----------|--------|-------|
| Crawlability | pass/warn/fail | XX/100 |
| Indexability | pass/warn/fail | XX/100 |
| Security | pass/warn/fail | XX/100 |
| URL Structure | pass/warn/fail | XX/100 |
| Mobile | pass/warn/fail | XX/100 |
| Core Web Vitals | pass/warn/fail | XX/100 |
| Structured Data | pass/warn/fail | XX/100 |
| JS Rendering | pass/warn/fail | XX/100 |
| IndexNow | pass/warn/fail | XX/100 |

### Critical Issues (fix immediately)
### High Priority (fix within 1 week)
### Medium Priority (fix within 1 month)
### Low Priority (backlog)

## DataForSEO Integration (Optional)

If DataForSEO MCP tools are available, use `on_page_instant_pages` for real page analysis (status codes, page timing, broken links, on-page checks), `on_page_lighthouse` for Lighthouse audits (performance, accessibility, SEO scores), and `domain_analytics_technologies_domain_technologies` for technology stack detection.

## Google API Integration (Optional)

If Google API credentials are configured, use `claude-seo run pagespeed_check.py <url> --json` for real PSI + CrUX field data (replaces lab-only CWV estimates), `claude-seo run crux_history.py <url> --json` for 25-week CWV trends, and `claude-seo run gsc_inspect.py <url> --json` for real indexation status per URL.

## Error Handling

| Scenario | Action |
|----------|--------|
| URL unreachable | Report connection error with status code. Suggest verifying URL, checking DNS resolution, and confirming the site is publicly accessible. |
| robots.txt not found | Note that no robots.txt was detected at the root domain. Recommend creating one with appropriate directives. Continue audit on remaining categories. |
| HTTPS not configured | Flag as a critical issue. Report whether HTTP is served without redirect, mixed content exists, or SSL certificate is missing/expired. |
| Core Web Vitals data unavailable | Note that CrUX data is not available (common for low-traffic sites). Suggest using Lighthouse lab data as a proxy and recommend increasing traffic before re-testing. |

---

## Supporting file: LICENSE.txt

```text
MIT License - see repository root LICENSE file for complete terms.

Copyright (c) 2026 AgriciDaniel
https://github.com/AgriciDaniel/claude-seo
```

---

## Supporting file: references/agent-friendly-pages.md

# Agent-friendly pages — audit reference (June 2026)

The next wave of AI search is not summarization — it's **agents** acting on the
user's behalf (search, compare, buy, book). Google's AI optimization guide and
the linked web.dev article describe three channels through which agents
interpret your site:

1. **Screenshots + vision model** — interprets visual hierarchy, button
   prominence, layout. Slow and token-expensive.
2. **Raw HTML / DOM** — nesting, IDs, classes, data attributes.
3. **The accessibility tree** — the browser-native semantic distillation
   (roles, names, states). The cleanest signal of the three.

Modern agents combine all three. Optimizing for the **accessibility tree** is
the single highest-leverage move; if your accessibility tree is broken, no
amount of visual polish saves you.

**Primary sources:**
- Google AI optimization guide:
  https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
- web.dev (referenced from the guide above): article on building
  agent-friendly websites

## Audit checklist

### 1. Use real interactive elements

| Pass | Fail |
|---|---|
| `<button>` for actions | `<div onclick="...">` |
| `<a href="...">` for navigation | `<div onclick="window.location...">` |
| `<input>` / `<select>` / `<textarea>` | Custom `contenteditable` widgets |

If you cannot use a real interactive tag, supply ARIA: `role="button"`,
`role="link"`, `tabindex="0"`, plus key handlers for `Enter` and `Space`.

**Why it matters:** the accessibility tree exposes real interactive elements
with their roles. Custom div widgets often appear in the tree with no role at
all — agents skip them.

### 2. Label associations

Every form input must have an associated label:

```html
<label for="email">Email</label>
<input id="email" type="email" name="email">
```

Or use `aria-label` / `aria-labelledby` where a visible label isn't possible.
Agents that read the accessibility tree get the field purpose directly from
the associated label — without it, the input is a void.

### 3. Interactive target size

Visual-analysis pipelines filter out interactive elements smaller than **~8
square pixels** of unobscured area. Tap-target accessibility minimums (24×24
WCAG AA, 44×44 Apple HIG) are stricter and pass the agent gate by default.

Audit: any clickable element below 24×24px is a candidate for agent
invisibility, in addition to the WCAG failure.

### 4. Don't cover interactive nodes with transparent overlays

Vision models discard covered nodes when computing what's "interactive at this
position". Common offenders:

- Full-card click handlers that overlay every child link.
- Transparent cookie-consent layers persisting beyond consent.
- Modal portals with `pointer-events: auto` left on after dismiss.
- "Ghost" tracking pixels with `position: absolute; inset: 0`.

### 5. Layout stability

If "Add to cart" lives in different positions on `/category/shoes` vs
`/category/bags`, screenshot-based agents have to relearn the page each
visit. Keep functionally identical actions in the same screen quadrant across
templates.

Cross-reference: this overlaps with **CLS** (Cumulative Layout Shift) in Core
Web Vitals, but the agent-UX concern is broader — it covers page-to-page
stability, not just within-page shift.

### 6. `cursor: pointer` as a legitimate signal

Vision models read `cursor: pointer` (set by default on `<a>` / `<button>`) as
a hint that an element is actionable. Do not override it to `cursor: default`
on truly interactive elements just for visual minimalism.

Inverse: do not apply `cursor: pointer` to non-interactive elements — that
makes agents click things that do nothing.

### 7. Stable, meaningful selectors

Agents that fall back to DOM parsing rely on:

- Real semantic tags (`<nav>`, `<main>`, `<article>`, `<section>`, `<aside>`)
- Stable `id` attributes on top-level layout containers
- `data-*` attributes that describe purpose, not implementation

Avoid auto-generated class names like `__sc_a4b7d9e2` as the only handle on a
critical interactive element — agents can target them but cannot tell what
they mean.

## Lighthouse Agentic Browsing category (shipped)

Google now ships a dedicated **Agentic Browsing** Lighthouse category — created
in Lighthouse **13.2.0 (2026-05-01)** and **on by default since 13.3.0
(2026-05-07)** in DevTools and the CLI (Chrome 150+). Category id:
`agentic-browsing`. Unlike other categories it reports a **fractional
pass-ratio** (X of N checks passed), **not** a 0–100 weighted score — so do not
compute or report a weighted agentic "score". It groups deterministic audits
into three buckets:

- **Agent-centric accessibility** — reuses three a11y audits: *Names and labels*
  (every interactive element has a programmatic name), *Tree integrity* (valid
  roles + parent-child relationships), *Visibility* (interactive content not
  hidden from the accessibility tree). These map directly onto checklist items
  1–2 above and the `agent_ux_check.py` heuristics.
- **Stability & discoverability** — *CLS* (visual stability for element
  positioning; see checklist item 5) + an *llms.txt* presence-at-domain-root
  check (13.4.0 relaxed it to allow leading whitespace).
- **WebMCP integration** — three audits, see below.

**Run paths:** DevTools Lighthouse panel (Chrome 150+, default-on, no toggle);
CLI `npx lighthouse@latest <url> --only-categories=agentic-browsing` (Node.js
22.19+, per-audit JSON for CI); PageSpeed Insights web UI and API. Lighthouse
13.4.1 enabled the category through the PSI API after 13.4.0 had disabled it.
CrUX never provides these lab audits.

## WebMCP (proposed standard; status needs recheck)

**WebMCP** lets a site declare structured tools/actions for agents rather than
agents inferring intent from the DOM. It is **no longer "years away"**: a
flag-gated Early Preview (Chrome 146 Canary) opened 2026-02-10, and Chrome 149
origin-trial/sign-up status needs verification from a Google-owned source
(local dev flag: `chrome://flags/#enable-webmcp-testing`). It remains a
*proposed* standard from the Web Machine Learning community group
(github.com/webmachinelearning/webmcp), not W3C-finalized.

Lighthouse 13.2+ ships **three WebMCP audits** under the Agentic Browsing
category (require Chrome 150+ and origin-trial registration). Shipped audit ids
in the 13.2.0 release notes:

- `webmcp-form-coverage`: informational; flags `<form>` elements lacking
  `toolname` / `tooldescription`.
- `webmcp-registered-tools`: informational; lists tools registered via the
  declarative (HTML) and imperative (JS) APIs.
- `webmcp-schema-validity`: validates WebMCP form schemas.

(Code parsing Lighthouse JSON should expect the release-note ids above.)

**Audit posture:** WebMCP is an active opportunity, not a finding to hard-fail —
absence is not a defect. Surface it as an opportunity for sites that want
first-class agent actions, and note origin-trial enrollment is required for the
audits to fire.

## Quick-audit one-liner

For a fast smoke check, capture the accessibility tree via Lighthouse or
Chrome DevTools and look for:

- Any interactive element with `role="generic"` → broken semantics.
- Any input without an `accessible name` → missing label.
- Any `<div>` with `onclick` and no `role` / `tabindex` → custom widget that
  agents won't see.

`claude-seo run render_page.py <URL> --mode auto --a11y-tree --json` loads the
page headlessly and captures Chromium's full accessibility tree through CDP.
Use `claude-seo run agent_ux_check.py <URL> --json` for the bounded Agent-UX
heuristic and its explicit complete, partial, or unavailable score status.

## Last verified

2026-08-25. Google has published the Lighthouse Agentic Browsing framework,
Lighthouse 13.4.1 exposes it through the PSI API, and the local renderer now
captures the Chromium accessibility tree through CDP. Update when:

- WebMCP graduates from origin trial to a stable/shipped API (or W3C status changes).
- Google adds or renames audits in the `agentic-browsing` category.
- web.dev publishes a follow-up article with revised criteria.