What does this skill do, and when should you use it?
This is one of 16 skills bundled in the MIT-licensed geo-seo-claude repository, located at skills/geo-crawlers/SKILL.md. It analyzes a website's accessibility to AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) — the foundational technical requirement for Generative Engine Optimization (GEO), since blocked crawlers mean the content cannot appear in AI-generated answers at all. It parses robots.txt, samples meta robots tags and X-Robots-Tag headers, checks emerging standards like llms.txt, and produces a GEO-CRAWLER-ACCESS.md report with a 0-100 score and concrete robots.txt recommendations. The parent collection installs into Claude Code via a one-command script and is invoked with /geo crawlers <url>.
1) Fetches the target domain's robots.txt via WebFetch and parses Allow/Disallow rules for each AI crawler (including wildcard inheritance and Crawl-delay); 2) Samples 5-10 key pages for meta robots tags (noindex/noai/noimageai plus bot-specific tags) and X-Robots-Tag headers; 3) Checks for /llms.txt, /ai.txt, and /.well-known/ai-plugin.; 4) Assesses JavaScript rendering dependence (GPTBot/ClaudeBot/PerplexityBot have limited JS rendering); 5) Parses Content-Signal: directives in robots.txt (IETF draft draft-romm-aipref-contentsignals); 6) Computes a weighted AI Visibility Score (Tier 1 50%, Tier 2 25%, no blanket blocks 15%, AI-specific files 10%) and generates a GEO-CRAWLER-ACCESS.md report with a 14-crawler access matrix and a complete recommended robots.txt configuration.
- An SEO/GEO consultant needs a per-crawler access report and a deployable robots.txt change list for a client's AI visibility audit
- A content site owner notices their pages never appear in ChatGPT Search or Perplexity and wants to check whether GPTBot or PerplexityBot is accidentally blocked
- A technical team inheriting a site with legacy SEO robots.txt rules needs to verify whether historical rules also block AI crawlers
- A publisher wants to allow AI search crawlers while retaining control over training data (distinguishing Tier 1 search crawlers from Tier 3 training-only crawlers like CCBot)
- A business targeting Chinese markets needs a contextual recommendation on whether to allow or block Bytespider
How do you install this skill?
- Static review only, nothing executed: all scores are low-confidence and output quality is unverified.
- The skill requests Bash and Write without scoped constraints; run it in a controlled environment and inspect generated files.
- Cited market statistics (35% block rate, etc.) carry no source links; do not reuse them in client-facing reports.
- Core value targets overseas AI platforms; Chinese AI search coverage is limited.
- robots.txt recommendations are generic; verify against your site's actual strategy (e.g., blocking CCBot is a policy choice) before applying.
- Shell / CLI
- Network access
- Local filesystem
Claude Code CLI (for collection install)Python 3.8+ (collection runtime)
The collection installs with one command (all 16 skills go into ~/.claude/skills/geo/): curl -fsSL https://raw.githubusercontent.com/zubair-trabzada/geo-seo-claude/main/install.sh | bash (Windows users run install-win.sh from Git Bash). Requires Python 3.8+ (plus python3-venv on Debian/Ubuntu), Claude Code CLI, and Git; dependencies install into an isolated venv at ~/.claude/skills/geo/.venv/ without touching system Python. The source does not document installing the geo-crawlers sub-skill on its own. Uninstall with ./uninstall.sh.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/zubair-trabzada/geo-seo-claude.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/geo-crawlers" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
After installing the collection, run in Claude Code: /geo crawlers https://example.com. The skill fetches and analyzes the domain's robots.txt, meta tags, HTTP headers, and llms.txt, then writes a GEO-CRAWLER-ACCESS.md report in the working directory. The report includes each crawler's status (Allowed/Blocked/Not Mentioned), a 0-100 AI Visibility Score, critical issues, and full recommended robots.txt content.
What are this skill's strengths and limitations?
- Covers 14 AI crawlers with a Tier 1/2/3 classification that distinguishes search crawlers (blocking directly removes AI search visibility) from training-only crawlers (blocking has no search impact), so recommendations are strategically grounded
- Broad checks: beyond robots.txt it inspects meta tags, X-Robots-Tag headers, llms.txt/ai.txt, JS rendering dependence, and the IETF-draft Content-Signal standard
- Outputs a Markdown report with a weighted 0-100 score and a deployable robots.txt configuration, ready for client delivery
- MIT licensed, free, auditable
- Some scoring inputs (e.g., '300M+ ChatGPT weekly users', the Originality.ai 35% blocking figure) cite external studies without verification links and depend on the author keeping them current
- Crawler detection relies entirely on robots.txt string matching; it cannot detect server-level blocks (firewalls, CDN rules)
- The JS rendering assessment is a static, qualitative judgment; the source offers no measured validation or test suite
- The number of sampled pages (5-10) in the 6-step analysis is left to the model, so reproducibility is not guaranteed
- How to install this single sub-skill standalone is not documented in the source
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| GEO Crawlers Access Analysis Skill this page | 51 · Use with care | ★ 11k | 20d ago | MIT |
| GEO Full-Site Audit | 48 · Use with care | ★ 11k | 20d ago | MIT |
| llms.txt Analysis & Generation Skill | 46 · Use with care | ★ 11k | 20d ago | MIT |
| GEO-SEO AI Search Optimization Audit Skill | 43 · Not recommended | ★ 11k | 20d ago | MIT |
| AI Search & GEO Optimizer | 52 · Use with care | ★ 19k | 21d ago | MIT |
How did FollowSkills review this skill?
Tool surface is restrained: Read/Grep/Glob/Bash/WebFetch/Write; the skill performs read-only analysis and writes one report file (GEO-CRAWLER-ACCESS.md); it recommends robots.txt snippets instead of editing them. Deducted for: unscoped Bash, no explicit user-confirmation step, undisclosed WebFetch data flow, no rollback notes, and no explicit Write path constraint.
The six-step procedure (robots.txt, meta tags, headers, AI-specific files, JS rendering, Content-Signal) is clear and self-consistent with a concrete output template. Deducted for: static review cannot confirm key paths run; no tests cover this skill itself (repo tests only target fetch_page.py SSR detection); thin error handling and failure feedback for robots.txt parse anomalies, WebFetch failures, or non-standard syntax.
Clear scenario (AI-crawler accessibility within GEO audits), good semantic trigger alignment with /geo crawlers, and a practical 14-crawler tier matrix. Deducted for: undeclared non-fit boundaries (heavy SPA, login-walled sites), no Chinese-language support or China-market framing, and core value depending on overseas AI platforms (ChatGPT, Claude, Perplexity) with little coverage of Chinese AI search surfaces.
Well-structured doc: crawler reference, matrix, procedure, output format, scoring; repo provides MIT LICENSE and uninstaller. Deducted for: no version or changelog in the SKILL.md, no known-limitations section (e.g., data freshness of cited stats), unverifiable Originality.ai citation, and maintenance responsibility visible only at repo level.
Deliverable (access map + recommendations + score) is complete in format and directly usable; marginal value comes from systematic 14-crawler coverage plus the emerging Content-Signal standard. Deducted for: unverified actual output correctness under static review; author-defined scoring weights not justified; UA strings risk staleness requiring manual upkeep.
The analysis steps are reproducible by users and crawler UA/purpose descriptions are broadly cross-checkable. Deducted for: key statistics (35% block rate, 300M users, 2B devices) lack source links and are author assertions; no committed tests or CI covering this skill's key paths, capping the score under static review.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →