skill
PinchBench is a benchmarking system for evaluating LLM models as OpenClaw coding agents. Made with 🦀 by the humans at https://kilo.ai
Choose how to use this skill
You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.
1. Native installer
This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.
Do not guess an installer command or replace an existing version without reviewing the diff.
2. Complete package recommended
Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.
No complete ProSkills package is published for this listing yet.3. Prompt-only
Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.
Need only the instruction file?
Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.
No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.
// RATINGS
Not yet listed on ClawHub or SkillsMP
// README
🦀 PinchBench
Real-world benchmarks for AI coding agents
Note: This repository contains the benchmark skill/tasks. It is NOT the source of official leaderboard results. To add models to the official results, modify pinchbench/scripts/default-models.yml.
PinchBench measures how well LLM models perform as the brain of an OpenClaw agent. Instead of synthetic tests, we throw real tasks at agents: scheduling meetings, writing code, triaging email, researching topics, and managing files.
Results are collected on a public leaderboard at pinchbench.com.
Why PinchBench?
Most LLM benchmarks test isolated capabilities. PinchBench tests what actually matters for coding agents:
- Tool usage — Can the model call the right tools with the right parameters?
- Multi-step reasoning — Can it chain together actions to complete complex tasks?
- Real-world messiness — Can it handle ambiguous instructions and incomplete information?
- Practical outcomes — Did it actually create the file, send the email, or schedule the meeting?
Quick Start
# Clone the skill
git clone https://github.com/pinchbench/skill.git
cd skill
# Run benchmarks with your model of choice
./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4
# Or run specific tasks
./scripts/run.sh --model openrouter/openai/gpt-4o --suite task_calendar,task_stock
Note: Model IDs must include their provider prefix (e.g.
openrouter/,anthropic/). OpenRouter is the default provider used for routing.
Requirements:
- Python 3.10+
- uv package manager
- A running OpenClaw instance
What Gets Tested
| Category | Tasks | What's tested |
|---|---|---|
| Productivity | Calendar, daily summaries | Event creation, time parsing, scheduling |
| Research | Stock prices, conferences, markets | Web search, data extraction, synthesis |
| Writing | Blog posts, emails, humanization | Content generation, tone, formatting |
| Coding | Weather scripts, file structures | Code generation, file operations |
| Analysis | Spreadsheets, PDFs, documents | Data processing, summarization |
| Triage, search | Inbox management, filtering | |
| Memory | Context retrieval, knowledge management | Long-term memory, recall |
| Skills | ClawHub, skill discovery | OpenClaw ecosystem integration |
Each task is graded automatically, by an LLM judge, or both — ensuring both objective and nuanced evaluation.
Submitting Results
To get your results on the leaderboard:
# Register for an API token (one-time)
./scripts/run.sh --register
# Run benchmark — results auto-upload with your token
./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4
Skip uploading with --no-upload if you just want local results.
Official Results
To submit an official run (marked on the leaderboard):
# Using environment variable
export PINCHBENCH_OFFICIAL_KEY=your_official_key
./scripts/run.sh --model anthropic/claude-sonnet-4
# Using command line flag
./scripts/run.sh --model anthropic/claude-sonnet-4 --official-key your_official_key
Command Reference
| Flag | Description |
|---|---|
--model MODEL | Model to test (e.g., openrouter/anthropic/claude-sonnet-4) |
--judge MODEL | Judge model for LLM grading; uses direct API when set (see below) |
--suite SUITE | all, automated-only, or comma-separated task IDs |
--runs N | Number of runs per task for averaging |
--timeout-multiplier N | Scale timeouts for slower models |
--thinking LEVEL | Reasoning depth: off, minimal, low, medium, high, xhigh, adaptive |
--output-dir DIR | Where to save results (default: results/) |
--no-upload | Skip uploading to leaderboard |
--register | Request an API token for submissions |
--upload FILE | Upload a previous results JSON |
--official-key KEY | Mark submission as official (or use PINCHBENCH_OFFICIAL_KEY env var) |
Judge
By default (no --judge flag), the LLM judge runs as an OpenClaw agent session. When --judge is specified, it calls the model API directly instead, bypassing OpenClaw personality injection.
# Default: OpenClaw agent session (no --judge needed)
./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4
# Direct API via OpenRouter
./scripts/run.sh --model openai/gpt-4o --judge openrouter/anthropic/claude-sonnet-4-5
# Direct API via Kilo Gateway
./scripts/run.sh --model openai/gpt-4o --judge kilo/anthropic/claude-sonnet-4-5
# Direct API via Anthropic
./scripts/run.sh --model openai/gpt-4o --judge anthropic/claude-sonnet-4-5-20250514
# Direct API via OpenAI
./scripts/run.sh --model openai/gpt-4o --judge openai/gpt-4o
# Headless Claude CLI
./scripts/run.sh --model openai/gpt-4o --judge claude
Required env vars: OPENROUTER_API_KEY, KILO_API_KEY, ANTHROPIC_API_KEY, or OPENAI_API_KEY depending on the judge model prefix.
Contributing Tasks
We welcome new tasks! Check out tasks/TASK_TEMPLATE.md for the format. Good tasks are:
- Real-world — Something an actual user would ask an agent to do
- Measurable — Clear success criteria that can be graded
- Reproducible — Same task should produce consistent grading
- Challenging — Tests agent capabilities, not just LLM knowledge
Transcript Archive
Session transcripts are automatically saved to results/{run_id}_transcripts/ alongside the results JSON. Each task's full agent conversation is preserved as a JSONL file (e.g. task_calendar.jsonl) for post-run analysis.
Links
- Leaderboard: pinchbench.com
- OpenClaw: github.com/openclaw/openclaw
- Issues: github.com/pinchbench/skill/issues
Star History
License
MIT — see [LICENSE](LICEN