AI Search Readiness report · 2026-06-01
67/ 100
AI Crawler Access6.5 / 25
✕Blanket crawl block. robots.txt disallows all user-agents (User-agent: * / Disallow: /). AI assistants cannot read your site. Fix: Remove the site-wide Disallow: / or scope it to specific paths.
✕Blocked AI engines. Blocked from major AI engines: OpenAI GPTBot, Perplexity, Claude (Anthropic), Google Gemini / AI, Common Crawl (feeds many LLMs). Fix: Add explicit Allow rules (or remove the Disallow) for these crawlers.
▲Blocked secondary AI bots. Also blocked: Perplexity live fetch, Anthropic (legacy UA), Claude Web, Amazon AI, ByteDance / TikTok AI, Meta AI, Cohere. Fix: Decide intentionally whether to allow these secondary AI crawlers.
Content Extractability20 / 20
●Server-rendered content. 1416 words of readable text are present in the initial HTML.
Structured Data12 / 15
●Structured data. Found 1 JSON-LD block(s): faqpage, question, answer, website, searchaction, entrypoint.
Metadata & Semantics12 / 15
●Core metadata. Title, description and a single H1 are present.
Answer-Ready Content7 / 12
▲llms.txt. No /llms.txt found. Fix: Publish an /llms.txt summarizing your site for AI assistants.
Crawl Infrastructure4 / 8
▲Sitemap. No sitemap.xml found or referenced. Fix: Publish sitemap.xml and reference it in robots.txt.
Technical Hygiene5 / 5
●Technical hygiene. Responds 200 OK in 1047ms, indexable.
Top fixes, in priority order
Stop blocking every crawler
Your robots.txt blocks all user-agents from the entire site, so no AI engine can index you. Scope the Disallow rules to private paths only.
Unblock major AI engines
robots.txt blocks OpenAI GPTBot, Perplexity, Claude (Anthropic), Google Gemini / AI, Common Crawl (feeds many LLMs). These are primary sources for AI answers — allow them to crawl.
Publish /llms.txt
The emerging llms.txt standard gives AI assistants a clean, curated summary of your site. Add one at /llms.txt.
Publish a sitemap
A sitemap.xml helps every crawler discover all your pages. Generate one and reference it in robots.txt.
Close these gaps for instacart.com
Get the exact JSON-LD, llms.txt and AI-crawler robots config this report calls for — generated for instacart.com and emailed in minutes, one time. Or put it on autopilot with weekly re-scans and an email alert the moment the score drops.
Get the Fix Pack → $49
Monitor weekly → $29/mo
Is this your site? Keep it from slipping.
Get instacart.com re-scanned automatically and emailed the moment its AI readiness drops.
Show off your AI readiness
Embed this badge on your site or README — it links back to this live report and helps AI engines discover it.
