Tiny Web Crawler Skill
Crawl from one or more starting pages, fetch readable content, search within pages, follow relevant links, and stop when the requested information is found or a bounded limit is reached.
The skill accesses external websites without explicit user confirmation, but clearly limits crawling pages and depth, and does not handle sensitive data. Scripts only fetch page content; no other external side effects beyond network requests. Data flow is transparent: skill description and script logs clearly show crawling process. However, lack of additional permission control or user confirmation mechanism for accessing external sites deducts points.
Script logic is self-consistent, handles missing inputs and network errors, but static review cannot verify actual execution and depends on external site availability, introducing uncertainty. Hence scored at static review cap.
Skill scenarios are clear, scope is defined, but it does not mention dependency on overseas sites and their reachability in mainland China, posing potential inaccessibility for Chinese users.
Documentation is well-structured, includes usage examples, parameter descriptions, and limits, but lacks changelog, version history, and clear maintenance responsibility; skill version also inconsistent with repository version.
Skill design can accomplish crawling tasks, has clear goals and expected outputs, but static review cannot verify output quality and no real usage examples provided, hence scored at static review cap.
Repository has CI workflows, but they do not cover this skill; no test suite for the skill independently; primary evidence is code and documentation, static review cannot execute, hence scored at static review cap.
- The skill accesses external websites, which may be affected by network restrictions or anti-bot measures.
- Does not explicitly mention dependency on overseas services; suggest clarifying availability in mainland China.
- No built-in user confirmation or permission control; users should stay vigilant when using.
What does this skill do, and when should you use it?
This skill provides a bounded web crawler that can start from one or more URLs, extract compact readable text, links, and query snippets. It searches within fetched content for exact names, phrases, dates, and more, and follows relevant links while prioritizing canonical internal pages. Two Node.js scripts are bundled: fetch-page.mjs for one-off page inspection and crawl-web.mjs for multi-page bounded crawling. The crawler respects limits on maximum pages, link depth, and same-domain pages, and stops early when the target information is found. It is an agent skill within the Leon personal assistant repository, following the standard SKILL.md format.
Runs two Node.js scripts: crawl-web.mjs starts from one or more URLs, follows relevant links, and stops at limits or strong matches; fetch-page.mjs fetches a single page and outputs a compact text preview, query snippets, links, and chunked text information. The scripts use Leon's managed Node.js binary or the system PATH node. The skill tracks visited URLs to avoid revisiting, and reports what was checked and what remains unresolved when limits are reached.
- A developer needs to find specific information across multiple related pages, such as API function usage in documentation or version numbers in changelogs.
- A researcher needs to extract specific data like product pricing or release dates from official websites and document sources with clear citations.
- A QA tester needs to verify that a web page contains certain text or links as part of a regression test.
- A content creator wants to gather information from multiple sources while avoiding irrelevant pages, focusing only on relevant links.
- A privacy-conscious user needs to search the web locally without relying on third-party services, as this skill can run locally with local models.
What are this skill's strengths and limitations?
- Bounded crawling with default limits on max pages, depth, and same-domain pages, preventing infinite fetches.
- Search within page content for exact names, phrases, dates, etc., with context snippets.
- Progressive fetching: fetches compact data first, then more text only when needed.
- Link selection strategy prioritizes official documentation, internal pages, and relevant keywords, avoiding navigation noise.
- Evidence rules encourage citing sources and distinguishing facts from inference.
- MIT-licensed, open source, and extensible.
- Requires Node.js environment and the Leon repository setup; not a standalone skill.
- No built-in test suite or evidence of testing; robustness not fully verified.
- May not work well on dynamic or JavaScript-rendered pages that require a browser.
- Only crawls from given start URLs; cannot handle pages requiring authentication.
- Relies on network access; offline usage is not supported.
How do you install this skill?
The skill is located in the skills/agent/tiny-web-crawler/ directory of the Leon repository. To use it, clone the repository and run pnpm install to set up dependencies. There is no separate installation step; the skill is used within the Leon environment.
How do you use this skill?
When triggering this skill in Leon, provide the target information or starting URL. The skill uses the crawl-web.mjs script, for example: node scripts/crawl-web.mjs --url "https://example.com" --query "target phrase" --max-pages 8 --max-depth 2. For single-page inspection, use fetch-page.mjs. The skill follows its workflow: clarify the target, start from the given URL, search content, follow links, avoid revisiting, and stop early when found.