PinchBench
Benchmarks LLMs as OpenClaw coding agents on real-world tasks, with a public leaderboard for side-by-side model comparison.
Documentation transparently lists dependencies and upload behavior, with a --no-upload escape; however, results auto-upload to an external leaderboard by default and require API token registration without explicit confirmation of data egress; benchmark.py source is not shown, so least privilege cannot be verified; pyproject includes fabric/paramiko (SSH tooling) with undisclosed purpose — deducted for opaque data flow.
SKILL.md commands, flags and jq examples are internally consistent; but the core benchmark.py, tasks/ and scripts/run.sh are absent from evidence, so key paths cannot be statically verified; repo CI runs only lint, no tests for this skill; SKILL.md claims 23 tasks while README says 53 — deducted for documentation contradiction and thin failure-mode evidence.
Scenarios (benchmarking, comparing models, leaderboard submission) and triggers are clear in the description and task table; but capability boundaries and non-fit (non-OpenClaw environments) are undeclared; core function depends on OpenClaw and overseas model APIs (OpenRouter/Anthropic), with no mainland-China reachability or Chinese-language support noted — deducted for environment-fit gaps.
Well-structured docs: prerequisites, quick start, flag tables, custom-task template, MIT license, version (2.0.0-rc1), lint CI and release workflow; but no changelog, no committed tests for this skill, task-count inconsistency (23 vs 53), and maintenance ownership rests only on a metadata author field — deducted for governance gaps.
Goal is clear (real-world benchmarking of OpenClaw agents), output is directly analyzable JSON plus transcript archives; but correctness and usability cannot be statically verified, leaderboard value depends on an unverified external service, and representative outputs lack corroboration — deducted.
Auditable engineering signals exist (lint CI, release versioning, pre-commit) and README/SKILL.md corroborate the CLI interface; but no committed tests cover benchmark.py key paths (tests under .agents/skills belong to a different skill), no third-party execution evidence, and leaderboard claims are not independently verifiable — deducted.
- Running the benchmark auto-uploads results to an external leaderboard by default and requires API token registration; privacy-conscious users should use --no-upload and inspect what is sent.
- pyproject depends on fabric/paramiko (SSH libraries) with undisclosed purpose in docs; review the actual source before running.
- SKILL.md claims 23 tasks while README claims 53 — documentation is inconsistent; core benchmark.py is not included in evidence, so runnability cannot be statically verified.
- Depends on OpenClaw and overseas model APIs (OpenRouter etc.); mainland-China reachability and Chinese-language support are not addressed.
- Publisher identity is unverified; leaderboard results cannot be independently validated.
What does this skill do, and when should you use it?
PinchBench is a benchmarking system that evaluates how well LLMs perform as the brain of an OpenClaw agent. Rather than synthetic tests, agents tackle real tasks — scheduling, email triage, research, coding, and multi-step workflows — graded by automated checks and/or an LLM judge. The skill ships 23 tasks, saves results as JSON, and can upload them to the public leaderboard at pinchbench.com. It is MIT-licensed and maintained by the humans at kilo.ai.
Runs benchmark.py (via uv) that feeds an OpenClaw agent 23 categorized tasks (basic, productivity, research, writing, coding, analysis, email, memory, and more), captures the agent's tool calls and outputs, grades them with Python checks or an LLM judge, saves per-run JSON results plus session transcripts as JSONL files, and auto-uploads submissions to the public leaderboard using a registered API token. It supports suite or per-task selection, multi-run averaging, timeout scaling, and custom task creation.
- Model evaluators: compare candidate models on real agent tasks before choosing one for OpenClaw
- Model vendors or researchers: benchmark a new model on OpenClaw and submit results to the public leaderboard
- OpenClaw users: verify their agent setup with task_sanity and a subset of automated tasks
- Benchmark contributors: author new tasks following tasks/TASK_TEMPLATE.md to expand coverage
- Debuggers: replay agent behavior on failed tasks using the saved JSONL session transcripts
What are this skill's strengths and limitations?
- Tasks reflect real-world scenarios (scheduling, email, coding, research), not isolated synthetic tests
- Combines automated grading with LLM judging for both objective and nuanced evaluation
- Public leaderboard makes cross-model comparison immediately visible
- Results JSON and full session transcripts are saved for post-run analysis
- Open task template lets the community contribute new tasks
- Requires a running OpenClaw instance — not usable without one
- SKILL.md version is 2.0.0-rc1, i.e., pre-release and potentially unstable
- A full run over all tasks takes time and incurs API costs for the tested model
- LLM-judge scores may vary with the judge model chosen
- The README mentions 53 tasks for the repo while this skill's SKILL.md defines 23 — the two counts differ
How do you install this skill?
Clone the collection: git clone https://github.com/pinchbench/skill.git && cd skill. Prerequisites: Python 3.10+, the uv package manager, and a running OpenClaw instance (the agent itself). The skill itself is at SKILL.md; the repository is a monorepo, but the pinchbench skill defined in SKILL.md works directly. No other install locations or marketplace flows are documented.
How do you use this skill?
From the skill directory, run: uv run benchmark.py --model anthropic/claude-sonnet-4. Useful variants: --suite automated-only for faster runs; --suite task_calendar,task_stock for specific tasks; --no-upload to keep results local. To submit to the leaderboard, first register a token with uv run benchmark.py --register. Results default to results/ and can be inspected with jq, e.g. jq '{average: ([.tasks[].grading.mean] | add / length)}' results/*.. Model IDs require a provider prefix; OpenRouter is the default router.
How does this skill compare with similar options?
The README names no specific competitor, but contrasts itself with LLM benchmarks that 'test isolated capabilities', positioning PinchBench around tool usage, multi-step reasoning, and practical outcomes.