AI answer accuracy auditor

AI Answer Accuracy Auditor#

Research basis#

This skill operationalizes Profound's research, The Parrot Problem: Why AI Search has a second dimension marketers can't ignore (opens in a new tab), published June 25, 2026.

Key takeaways from the study:

• Across 50,000 prompts, about 47% of response content was unsolicited editorial material such as comparisons, rankings, caveats, and recommendations.
• A brand can be visible while still being described inaccurately or framed poorly.
• Specific, current, verifiable first-party facts give answer engines stronger material than generic marketing claims.
• Profound recommends recurring claim audits, source-of-truth updates, and human-reviewed correction workflows.

Treat these findings as observational evidence, not universal model behavior. Record engine, model or surface, locale, prompt, and capture date for every answer.

Goal#

Find material claims about a brand, verify them against current evidence, diagnose where errors originate, and recommend the smallest legitimate correction.

Intake#

Ask only for missing inputs that block the audit:

• brand and canonical domain
• products or services in scope
• engines, markets, and languages
• captured AI answers or permission to research them
• current product documentation, pricing, policies, and approved claims
• competitors and high-risk topics

Never treat the brand's preferred wording as proof. Use official product material for first-party facts and reliable independent sources for external claims.

Workflow#

1. Define the audit cohort by engine, market, language, prompt family, and date.
2. Extract every decision-relevant claim, including unsolicited editorial claims.
3. Label each claim: requested fact, comparison, recommendation, ranking, caveat, price, availability, compatibility, limitation, or sentiment.
4. Verify the claim against the strongest current source available.
5. Classify the result:
   • accurate and current
   • accurate but incomplete
   • ambiguous
   • stale
   • false
   • subjective framing
   • unverifiable
6. Trace the likely source: owned page, documentation, feed, profile, third-party publisher, community page, or unknown.
7. Score severity using decision impact, exposure, confidence, and correction difficulty.
8. Fix the source of truth before proposing cosmetic rewrites.
9. Define a retest cohort and observation window.

Do not fabricate corrections, manipulate Wikipedia, astroturf communities, or pressure publishers to remove legitimate criticism.

Severity#

• Critical: safety, legal, eligibility, price, availability, or compatibility errors that can materially harm a decision.
• High: false comparisons, wrong limitations, or stale product facts in important prompt clusters.
• Medium: incomplete or misleading framing with plausible decision impact.
• Low: stylistic sentiment or immaterial wording.

Output#

Return:

1. Executive finding: accuracy rate, highest-risk narrative, and primary source-of-truth problem.
2. Claim ledger with engine, prompt, claim, type, verdict, evidence, likely source, severity, and owner.
3. Source correction plan with exact facts and canonical destinations.
4. Third-party correction plan limited to factual, evidence-backed outreach.
5. Retest plan using the same prompt cohort plus controlled variants.
6. Unknowns and limitations.

Never promise that correcting a source will change an AI answer. Report observed movement over time.