What does this skill do, and when should you use it?
PinchBench is a benchmarking system that evaluates how well LLMs perform as the brain of an OpenClaw agent. Rather than synthetic tests, agents tackle real tasks — scheduling, email triage, research, coding, and multi-step workflows — graded by automated checks and/or an LLM judge. The skill ships 23 tasks, saves results as JSON, and can upload them to the public leaderboard at pinchbench.com. It is MIT-licensed and maintained by the humans at kilo.ai.
Runs benchmark.py (via uv) that feeds an OpenClaw agent 23 categorized tasks (basic, productivity, research, writing, coding, analysis, email, memory, and more), captures the agent's tool calls and outputs, grades them with Python checks or an LLM judge, saves per-run JSON results plus session transcripts as JSONL files, and auto-uploads submissions to the public leaderboard using a registered API token. It supports suite or per-task selection, multi-run averaging, timeout scaling, and custom task creation.
- Model evaluators: compare candidate models on real agent tasks before choosing one for OpenClaw
- Model vendors or researchers: benchmark a new model on OpenClaw and submit results to the public leaderboard
- OpenClaw users: verify their agent setup with task_sanity and a subset of automated tasks
- Benchmark contributors: author new tasks following tasks/TASK_TEMPLATE.md to expand coverage
- Debuggers: replay agent behavior on failed tasks using the saved JSONL session transcripts
How do you install this skill?
- Running the benchmark auto-uploads results to an external leaderboard by default and requires API token registration; privacy-conscious users should use --no-upload and inspect what is sent.
- pyproject depends on fabric/paramiko (SSH libraries) with undisclosed purpose in docs; review the actual source before running.
- SKILL.md claims 23 tasks while README claims 53 — documentation is inconsistent; core benchmark.py is not included in evidence, so runnability cannot be statically verified.
- Depends on OpenClaw and overseas model APIs (OpenRouter etc.); mainland-China reachability and Chinese-language support are not addressed.
- Publisher identity is unverified; leaderboard results cannot be independently validated.
- Shell / CLI
- Network access
- Local filesystem
Python 3.10+uv package managera running OpenClaw instance
Clone the collection: git clone https://github.com/pinchbench/skill.git && cd skill. Prerequisites: Python 3.10+, the uv package manager, and a running OpenClaw instance (the agent itself). The skill itself is at SKILL.md; the repository is a monorepo, but the pinchbench skill defined in SKILL.md works directly. No other install locations or marketplace flows are documented.
git clone --depth 1 https://github.com/pinchbench/skill.git ~/.claude/skills/skillGenerated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Once installed, send your agent any of these to trigger it:
- {average: ([.tasks[].grading.mean] | add / length)}
From the skill directory, run: uv run benchmark.py --model anthropic/claude-sonnet-4. Useful variants: --suite automated-only for faster runs; --suite task_calendar,task_stock for specific tasks; --no-upload to keep results local. To submit to the leaderboard, first register a token with uv run benchmark.py --register. Results default to results/ and can be inspected with jq, e.g. jq '{average: ([.tasks[].grading.mean] | add / length)}' results/*.. Model IDs require a provider prefix; OpenRouter is the default router.
What are this skill's strengths and limitations?
- Tasks reflect real-world scenarios (scheduling, email, coding, research), not isolated synthetic tests
- Combines automated grading with LLM judging for both objective and nuanced evaluation
- Public leaderboard makes cross-model comparison immediately visible
- Results JSON and full session transcripts are saved for post-run analysis
- Open task template lets the community contribute new tasks
- Requires a running OpenClaw instance — not usable without one
- SKILL.md version is 2.0.0-rc1, i.e., pre-release and potentially unstable
- A full run over all tasks takes time and incurs API costs for the tested model
- LLM-judge scores may vary with the judge model chosen
- The README mentions 53 tasks for the repo while this skill's SKILL.md defines 23 — the two counts differ
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| PinchBench this page | 44 · Not recommended | ★ 1.4k | 5mo ago | MIT |
| Model Catalog Refresh Skill (agent-service-toolkit) | 60 · Recommended | ★ 4.5k | 21d ago | MIT |
| Advanced LLM Evaluation | 56 · Use with care | ★ 18k | 13d ago | MIT |
| MCP Server Builder Guide | 50 · Use with care | ★ 3.1k | 8mo ago | MIT |
| Earth2Studio Diagnostic Builder ✓ NVIDIA · Official | 56 · Use with care | ★ 3.6k | 11d ago | Apache-2.0 |
The README names no specific competitor, but contrasts itself with LLM benchmarks that 'test isolated capabilities', positioning PinchBench around tool usage, multi-step reasoning, and practical outcomes.
How did FollowSkills review this skill?
Documentation transparently lists dependencies and upload behavior, with a --no-upload escape; however, results auto-upload to an external leaderboard by default and require API token registration without explicit confirmation of data egress; benchmark.py source is not shown, so least privilege cannot be verified; pyproject includes fabric/paramiko (SSH tooling) with undisclosed purpose — deducted for opaque data flow.
SKILL.md commands, flags and jq examples are internally consistent; but the core benchmark.py, tasks/ and scripts/run.sh are absent from evidence, so key paths cannot be statically verified; repo CI runs only lint, no tests for this skill; SKILL.md claims 23 tasks while README says 53 — deducted for documentation contradiction and thin failure-mode evidence.
Scenarios (benchmarking, comparing models, leaderboard submission) and triggers are clear in the description and task table; but capability boundaries and non-fit (non-OpenClaw environments) are undeclared; core function depends on OpenClaw and overseas model APIs (OpenRouter/Anthropic), with no mainland-China reachability or Chinese-language support noted — deducted for environment-fit gaps.
Well-structured docs: prerequisites, quick start, flag tables, custom-task template, MIT license, version (2.0.0-rc1), lint CI and release workflow; but no changelog, no committed tests for this skill, task-count inconsistency (23 vs 53), and maintenance ownership rests only on a metadata author field — deducted for governance gaps.
Goal is clear (real-world benchmarking of OpenClaw agents), output is directly analyzable JSON plus transcript archives; but correctness and usability cannot be statically verified, leaderboard value depends on an unverified external service, and representative outputs lack corroboration — deducted.
Auditable engineering signals exist (lint CI, release versioning, pre-commit) and README/SKILL.md corroborate the CLI interface; but no committed tests cover benchmark.py key paths (tests under .agents/skills belong to a different skill), no third-party execution evidence, and leaderboard claims are not independently verifiable — deducted.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →