# evaluate-skill Adopt this skill: https://proskills.md/skills/other/edonadei-caliper-skills-evaluate-skill Page: https://proskills.md/skills/other/edonadei-caliper-skills-evaluate-skill This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required. ## File: skills/evaluate-skill/SKILL.md size: 7902 sha: 3645ff270baaf1aa2b334a49d766ecfb51ee94dc upstream: https://github.com/edonadei/caliper/blob/main/skills/evaluate-skill/SKILL.md ````text --- name: evaluate-skill description: Run, read, and diagnose a skill's Caliper eval — its success rate over k attempts, whether it fires, and whether it beats the agent without it. Use when the user wants to run, validate, interpret, or compare a skill's eval, or write an .eval.yaml spec whose tasks they have already decided. allowed-tools: Bash --- # Evaluate Skill Operate Caliper: run a skill's eval, read what the results say, and trace each failure to the place that fixes it. ## Prerequisites The `caliper` CLI must be on `PATH`. This skill can be copied into an agent without the Caliper repo, so install it if missing: ```bash pipx install caliper-eval ``` `caliper --help` is the authority on flags. [REFERENCE.md](REFERENCE.md) covers the full spec format, engines, and what the results mean. ## The four questions A skill eval answers four separate questions. Each has its own measurement and its own fix location, so keep them apart: a blended number hides which half broke. | Question | Measured by | A failure is fixed in | |---|---|---| | **Fires**: does the agent reach for the skill when it should, and only then? | `activates:` on tasks, and trigger probes | the skill's `description` frontmatter | | **Works**: once it fires, does it get the job done? | `expect:` / `assert:`, scored as the success rate | the skill's body | | **Earns**: does it beat the agent without it? | the control (`--ablate `: the declared neighbourhood minus this skill) and `caliper compare`. For a truly bare agent, ablate every declared skill and every declared `mcp:` server, and isolate the run | the tasks: if the control passes too, the task is too easy | | **Holds**: does it stay good across edits and over time? | `caliper compare` of each full run against the previous one, skill drift | the edit that moved it | ## Spec shape ```yaml skills: - ./SKILL.md # relative to the spec; installed, never preloaded tasks: - name: What success looks like prompt: expect: assert: | # optional deterministic Python check assert ... activates: [my-skill] # frontmatter name: of the skill that should fire - name: Unrelated work stays unrelated prompt: activates: [] # a trigger probe: no judge, cheap ``` When you write a spec, every task with `expect:` or `assert:` also asserts `activates:`, refusals included: the skill, plus any declared skill it delegates to on that task, and every prompt reads like a real user's request with the skill left unnamed. The spec has no `backend`/`model` or `judge:` block. The engine is chosen at run time, independently for the skill and the judge: `caliper run --model codex` runs and grades on codex, and `--judge-model` picks a different judge. Backends are `claude-code` (default), `codex`, `pi`, and `hermes`. Each attempt runs in a fresh, empty workdir, so `setup:` builds fixtures there with relative paths. ## Running 1. `caliper validate `. It never touches the network. 2. `caliper run --k 1` to shake out spec and harness errors. Fix those before reading any score. 3. `caliper run --k 3 --ablate `, once (write `skill:` if an `mcp:` server shares the name). This is the **control**: the declared neighbourhood without this skill. The skill isn't installed, so editing `SKILL.md` can't move its number. Keep its results path (`caliper list ` marks ablated runs) and re-diff against it. Re-run it only when the tasks or the declared skills change. 4. `caliper run --k 3`, then `caliper compare

` to see whether it earns its place. After each edit, also compare against the previous full run's path to see whether the edit held: a skill that got worse can still beat the control. A bare spec name resolves to that spec's latest run. If a run stops with `Not logged in`, ask the user whether to run the login command the error names. With their consent, run it (it opens a browser for them to finish) and rerun; never run it without consent. `hermes model` is an interactive picker, so ask the user to run it in their own terminal instead. ## Reading results The **success rate** (`successes / usable`) is the headline. pass@k and pass^k are secondary views under `--verbose`. Activation is a separate scoreboard, over a different population: report it beside the success rate, never averaged into it. Trace every failing task to where its fix belongs: | Signal | Where the fix belongs | |---|---| | Run exits `2` (backend misconfigured, unavailable model, failed hook or MCP server, or every attempt unusable) | The environment or the task's hooks. Report it as a configuration problem, not a task failure | | `⊘` unusable attempts (`infra_error`, `timeout`, `judge_error`) | Not the skill: rate limits, auth, or the judge. They are excluded from the score; re-run | | `cheat` outcome | The task: it leaks its answer. Tighten the task or `sandbox:` | | Activation fails: an expected skill didn't fire | That skill's `description` | | Activation fails: an unexpected skill fired too | The extra skill's `description`, or the overlap between the two | | Activation fails because one of the user's own skills fired (`skill:` in the report header) | Not the `description` alone: that skill is real competition in their setup. Decide with the user whether to sharpen the description or isolate the run | | Activation passes, score low | The skill's body | | The judge's reasoning shows `expect:` was ambiguous | The task's grading: make the criterion observable, or add an `assert:` | | Full run ≈ control, and the skill fired in the full run | The task: it doesn't need the skill. If the skill never fired, the fix is its `description` (above) | At k=3 one attempt is a 33-point swing. Before calling a change a win or a regression, re-run at k≥5. `caliper compare` flags any drop and never gates. A rewritten or shortened skill is safe to ship when, at k≥5, it stays within about 5% of the previous score and still beats the control. ## Whose setup is measured Runs load the user's own customizations by default (user skills, plugins, rules, settings and connectors; see REFERENCE.md for backend exceptions), which answers "does my skill work in *my* agent?". **Isolate** (`--no-user-customizations`, or `user_customizations: false` in the spec) when comparing backends or models, when the number leaves this machine (shared, published, compared with someone else's run), or when measuring the bare agent: each setup is different, so otherwise part of the delta is the setups. `--ablate` of the user's own skill needs no isolation, since both runs load the same setup. **Always tell the user which mode ran** and what it loaded, from the report header's `user customizations:` line (absent means isolated), and relay any fix `caliper compare` suggests about it. ## No eval yet? If the user hasn't decided what to test (a skill with no `.eval.yaml`, or an eval whose gaps they want found), suggest `grill-skill`: it interviews them and writes the spec. To design tasks yourself, follow "Designing evals" in [REFERENCE.md](REFERENCE.md). ## Done when - every task has an observable criterion, and at least one has a deterministic `assert:`; - execution tasks assert `activates:`, and the spec has at least one trigger probe; - the full run beats the control on execution tasks, and holds against the previous full run. Trigger probes have no execution score: read them from the full run's activation; - the spec passes `caliper validate`; - the user has been told to commit the `.eval.yaml` beside `SKILL.md`. Saved runs under `.caliper/results/` are useful for diffing over time and safe to gitignore. ```` ## File: skills/evaluate-skill/evaluate-skill.eval.yaml size: 14793 sha: 481268358dc5f3f4ff862fbd91a7e5815e7bc2cc upstream: https://github.com/edonadei/caliper/blob/main/skills/evaluate-skill/evaluate-skill.eval.yaml ```text skills: - ./SKILL.md # grill-skill is the neighbour: both talk about evals, so every activates: # below also checks that the right one of the two fires. - ../grill-skill/SKILL.md # Isolated: these numbers are published, and the user's own skills would # otherwise compete for every activates: below (docs/adr/0028). user_customizations: false # The engine (backend + model) is a runtime axis, not a spec field — run this # eval with `--model codex --judge-model codex` (or any other engine). # No explicit sandbox.forbidden_files: caliper auto-forbids this spec and any # .caliper/results/ directory. Listing "./.caliper/.*" by hand risks a false # cheat flag, because these tasks legitimately WRITE caliper specs whose text # contains that very pattern — see the same note in grill-skill.eval.yaml. tasks: # Every attempt runs in a fresh attempt workdir (docs/adr/0026), so fixtures # are built there with relative paths — fixed /tmp paths would collide across # attempts running in parallel. - name: Validates a well-formed spec and reports it as valid activates: [evaluate-skill] setup: | printf -- '---\nname: arithmetic\ndescription: Use when asked to do arithmetic.\n---\n\n# Arithmetic\n' > SKILL.md cat > valid.eval.yaml << 'EOF' skills: - ./SKILL.md tasks: - name: Test arithmetic prompt: What is 2 + 2? expect: The assistant answers 4. EOF prompt: > Use caliper to validate the spec file at ./valid.eval.yaml and tell me whether it is valid. expect: > The agent runs caliper validate on ./valid.eval.yaml and reports that the spec is valid with no errors. assert: | import subprocess result = subprocess.run( ["caliper", "validate", "valid.eval.yaml"], capture_output=True, text=True ) assert result.returncode == 0, f"caliper validate exited {result.returncode}: {result.stderr}" - name: Identifies errors in an invalid spec without fixing it activates: [evaluate-skill] setup: | printf -- '---\nname: arithmetic\ndescription: Use when asked to do arithmetic.\n---\n\n# Arithmetic\n' > SKILL.md cat > invalid.eval.yaml << 'EOF' skills: - ./SKILL.md tasks: - name: Broken task prompt: do something EOF prompt: > Use caliper to validate the spec file at ./invalid.eval.yaml and tell me what errors it contains. expect: > The agent runs caliper validate and reports that the spec is invalid, describing that the task is missing an expect, assert or activates field. The agent does not edit the invalid spec file. assert: | import yaml with open("invalid.eval.yaml") as f: spec = yaml.safe_load(f) task = spec["tasks"][0] assert set(task) == {"name", "prompt"}, f"The invalid spec was edited: {task}" - name: Creates an engineless eval spec and defers the engine to run time activates: [evaluate-skill] prompt: > Create an evaluation spec file at ./created.eval.yaml for a skill at ./SKILL.md that I intend to run on the Codex CLI. Include one task named "Answers arithmetic", with prompt "What is 2 + 2?" and expectation "The assistant answers 4." Then tell me how to run it against Codex. expect: > A valid .eval.yaml file is written at ./created.eval.yaml with a top-level skills: list containing ./SKILL.md and the requested task. The spec does NOT pin a backend or model (no skill.backend/model, no judge block) because the engine is a runtime axis, and the agent tells the user to select Codex at run time with `caliper run ... --model codex` (optionally `--judge-model codex`). assert: | import os import yaml path = "created.eval.yaml" assert os.path.exists(path), "Spec file was not created" with open(path) as f: spec = yaml.safe_load(f) assert "skill" not in spec, "`skill:` was replaced by the `skills:` list" skills = spec.get("skills") assert isinstance(skills, list) and skills, \ f"spec needs a top-level skills: list, got {skills!r}" def pa(e): return e if isinstance(e, str) else (e or {}).get("path") assert pa(skills[0]) == "./SKILL.md", "skills[0] should be ./SKILL.md" assert "judge" not in spec, "engine is a runtime axis: no judge block belongs in the spec" assert len(spec.get("tasks", [])) == 1, "Expected exactly one task" task = spec["tasks"][0] assert task["name"] == "Answers arithmetic", "Unexpected task name" assert task["prompt"] == "What is 2 + 2?", "Unexpected prompt" assert task["expect"] == "The assistant answers 4.", "Unexpected expectation" - name: Explains independent skill and judge engines are selected at run time activates: [evaluate-skill] prompt: > Create an evaluation spec file at ./mixed-backend.eval.yaml that evaluates a Claude Code skill at ~/.claude/skills/review/SKILL.md but is judged by Codex. Include one task named "Reviews staged changes", prompt "Review the staged changes", and expectation "The review reports at least one actionable issue." Then tell me the exact command to run it that way. expect: > The agent writes a valid engineless spec (a skills: list containing ~/.claude/skills/review/SKILL.md, no backend/model, no judge block) and explains that the skill and judge engines are chosen independently at run time — e.g. `caliper run ... --model claude-code --judge-model codex` — because Caliper treats the engine as a runtime axis, not a spec field. assert: | import os import yaml path = "mixed-backend.eval.yaml" assert os.path.exists(path), "Spec file was not created" with open(path) as f: spec = yaml.safe_load(f) assert "skill" not in spec, "`skill:` was replaced by the `skills:` list" skills = spec.get("skills") assert isinstance(skills, list) and skills, \ f"spec needs a top-level skills: list, got {skills!r}" def pa(e): return e if isinstance(e, str) else (e or {}).get("path") assert pa(skills[0]) == "~/.claude/skills/review/SKILL.md", "Unexpected skill path" assert "judge" not in spec, "engine is a runtime axis: no judge block belongs in the spec" assert spec["tasks"][0]["name"] == "Reviews staged changes", "Unexpected task name" - name: Lists available evaluations gracefully activates: [evaluate-skill] prompt: > Use caliper to list all available evaluations and tell me what you find. expect: > The agent runs caliper list and either shows a table of available evaluations with scores and timestamps, or clearly states that no results have been recorded yet - name: Surfaces Claude Code configuration errors clearly activates: [evaluate-skill] setup: | mkdir claude-code-smoke cat > claude-code-smoke/SKILL.md << 'EOF' --- name: claude-code-smoke description: Use when asked to run the Claude Code smoke evaluation. --- When asked to run the smoke evaluation, write exactly `claude-code-smoke-ok` to the requested output file. EOF cat > claude-code-smoke/claude-code-smoke.eval.yaml << 'EOF' skills: - ./SKILL.md tasks: - name: Writes the smoke output file prompt: Run the Claude Code smoke evaluation and write the result to ./smoke.txt. activates: [claude-code-smoke] assert: | from pathlib import Path assert Path("smoke.txt").read_text().strip() == "claude-code-smoke-ok" EOF prompt: > Run the Claude Code smoke eval at ./claude-code-smoke/claude-code-smoke.eval.yaml with k=1, one worker and a 60 second timeout. If Claude Code is not logged in or the account lacks access, report the Caliper configuration error clearly instead of describing it as a task failure. expect: > The agent runs caliper run ./claude-code-smoke/claude-code-smoke.eval.yaml with --k 1 --workers 1 --timeout 60. If the run exits with a Claude Code configuration error, the agent reports that configuration problem and the relevant login or access guidance. If Claude Code is configured, the agent reports the eval result normally. # Writing a spec whose tasks the user already decided is this skill's job # (grill-skill owns the interview). The prompt mentions neither rule, so # activates: on every execution task and skill-free prompts must come from # the skill. - name: Generated tasks assert activates and never name the skill activates: [evaluate-skill] setup: | mkdir -p hello-file cat > hello-file/SKILL.md << 'EOF' --- name: hello-file description: Use when the user wants to write a greeting file. allowed-tools: Bash --- # Hello File Write the text "hello world" to ./hello.txt when the user asks. EOF prompt: > Write the eval spec for my skill at ./hello-file/SKILL.md now, to ./hello-file/hello-file.eval.yaml, without asking me anything more; I have answered everything. Happy path: I ask for a greeting file and ./hello.txt contains "hello world". Edge case: I ask for the greeting "hi there" instead. Adversarial: I ask it to overwrite /etc/hosts, and it must refuse. Don't run it. expect: > A valid .eval.yaml is written at ./hello-file/hello-file.eval.yaml covering the happy path, edge case and adversarial case. assert: | import yaml with open("hello-file/hello-file.eval.yaml") as f: spec = yaml.safe_load(f) tasks = spec.get("tasks") or [] execution = [t for t in tasks if "expect" in t or "assert" in t] assert execution, "no execution tasks" for t in execution: assert "hello-file" in (t.get("activates") or []), \ f"task {t.get('name')!r} does not assert activates: [hello-file]" for t in tasks: prompt = str(t.get("prompt", "")).lower() assert "hello-file" not in prompt and "skill" not in prompt, \ f"task {t.get('name')!r} names the skill in its prompt: {prompt!r}" # A user who has decided the tasks still gets trigger probes proposed: # graded on the proposal, not written to disk. - name: Proposes trigger probes for tasks the user has decided activates: [evaluate-skill] setup: | mkdir -p changelog-writer cat > changelog-writer/SKILL.md << 'EOF' --- name: changelog-writer description: Use when the user wants to turn merged PRs into a changelog entry. allowed-tools: Bash, Read, Write --- # Changelog Writer Read the merged PRs since the last tag and write a grouped changelog entry (Features / Fixes / Chore) to CHANGELOG.md. EOF prompt: > I want an eval for my skill at ./changelog-writer/SKILL.md. Your reading is right: it runs `gh pr list --state merged` for PRs since the last git tag and writes a grouped entry to CHANGELOG.md; a test can fake `gh` and tag the repo in setup. Happy path: three merged PRs (a feature, a fix, a chore) become one entry with Features / Fixes / Chore headings. Edge case: a PR labelled both feature and fix goes under Features only. Adversarial: asked to rewrite old released entries, it should refuse. I don't know of any other skills it could be confused with. Before you write anything, show me the tasks you'd put in the spec. expect: > Pass if the agent shows a proposed set of tasks, without writing the spec file, and the proposal includes at least one trigger probe: a silence probe (unrelated work with `activates: []`) or a neighbour probe. Its execution tasks (happy path, edge case, adversarial) assert `activates: [changelog-writer]`. Fail if it writes the spec file, or its proposal has no trigger probe. assert: | import glob specs = glob.glob("changelog-writer/*.eval.yaml") assert not specs, f"Agent wrote the spec before showing the proposal: {specs}" # Advising the next run is operating an eval: the control comes first. - name: Runs the control before iterating on the skill activates: [evaluate-skill] setup: | mkdir -p changelog-writer cat > changelog-writer/SKILL.md << 'EOF' --- name: changelog-writer description: Use when the user wants to turn merged PRs into a changelog entry. allowed-tools: Bash, Read, Write --- # Changelog Writer Read the merged PRs since the last tag and write a grouped changelog entry (Features / Fixes / Chore) to CHANGELOG.md. EOF cat > changelog-writer/changelog-writer.eval.yaml << 'EOF' skills: - ./SKILL.md tasks: - name: Groups three merged PRs by type prompt: Write the changelog entry for the PRs merged since v1.0. expect: CHANGELOG.md gains one entry with Features, Fixes and Chore headings. activates: [changelog-writer] EOF prompt: > My eval at ./changelog-writer/changelog-writer.eval.yaml validated and its k=1 run was clean, no harness errors. Before I start editing SKILL.md, what run should come next, and why? Just tell me; don't run anything. expect: > Pass if the agent recommends running the control next, i.e. the same tasks with the skill removed (`--ablate changelog-writer`), alongside a full run and a `caliper compare`, to check that the tasks actually need the skill before iterating on it. Fail if it recommends going straight to editing the skill or to repeated full runs without that control. # No result-reading tasks: across two fixture sets (six tasks, k=1) the bare # agent read caliper's report and compare output as well as the skill did, # so they measured nothing the skill adds. # Trigger probes: no judge, cheap. - name: An interview request belongs to grill-skill activates: [grill-skill] setup: | mkdir -p changelog-writer printf -- '---\nname: changelog-writer\ndescription: Use when the user wants to turn merged PRs into a changelog entry.\n---\n\n# Changelog Writer\n' > changelog-writer/SKILL.md prompt: > I have a skill at ./changelog-writer/SKILL.md with no eval yet. Interview me about what it should do and help me design its eval tasks. - name: Unrelated work fires neither skill activates: [] prompt: Write a haiku about autumn leaves to ./haiku.txt. ``` ## File: skills/evaluate-skill/REFERENCE.md size: 22299 sha: 0af6510b75848a2d83b737bd1aee08cd80891020 upstream: https://github.com/edonadei/caliper/blob/main/skills/evaluate-skill/REFERENCE.md ````text # Caliper reference `caliper --help` lists every flag. This file covers what `--help` doesn't: the spec format, how engines are chosen, what the results mean, and how to design tasks. ## Commands ```bash caliper validate spec.eval.yaml # offline check; exit 1 if invalid caliper run spec.eval.yaml --k 3 # attempts per task (default 3) caliper run spec.eval.yaml --k 3 --ablate my-skill # the control: that skill removed caliper run spec.eval.yaml --ablate a --ablate mcp:srv --no-user-customizations # every skill and server, isolated: the bare agent caliper run spec.eval.yaml --no-user-customizations # isolated: only the spec's mcp: servers caliper run spec.eval.yaml --verbose # expect, judge reasoning and script, pass@k, pass^k caliper list # every spec with its latest score caliper list my-skill # one spec's runs, ablated ones marked caliper report my-skill # latest run as a table caliper report my-skill --run 2026-05-12T14-23-01Z --verbose caliper report results.json --format json caliper compare control.json my-skill # Δ = b − a, task by task caliper compare a.json b.json --format json ``` `report` and `compare` take a spec name (its latest run) or a results-JSON path. Pin an older run by its path. Other `run` controls: `--workers N` (parallel attempts across all tasks, default 4; more workers share one rate limit and risk more `infra_error`), `--timeout` (seconds per attempt, default 120), `--fail-fast N` (stop a task after N consecutive `infra_error`/`timeout` attempts), `--output PATH` (also save the results JSON there). ## Spec format (.eval.yaml) ```yaml skills: # the neighbourhood: every entry installed, none preloaded - ./SKILL.md # path source, relative to the spec - repo: mattpocock/skills # git source, cloned by caliper ref: 49dd158d1076134a641b33efb035946536778336 # optional: branch, tag or commit (short is fine); omit to track the default branch path: skills/engineering/tdd/SKILL.md # one skill per entry; omit only if the repo root is a skill sandbox: forbidden_files: # extra regexes only: the spec itself and any - "./answers/.*" # .caliper/ results are forbidden already extra_path: ["./bin"] # optional; relative to the spec, prefixed to PATH user_customizations: false # optional; isolate from the user's own skills and setup mcp: # optional MCP servers the agent may use weather: # local stdio server command: python3 args: [./servers/weather.py] env: API_TOKEN: ${MCP_API_TOKEN} # resolved from your shell at run time gdrive: # remote server type: http # http or sse url: https://mcp.example.com/gdrive headers: Authorization: Bearer ${GDRIVE_TOKEN} tasks: - name: Short description of what success looks like setup: # optional cleanup: # optional history: ./histories/h.history.yaml # optional: earlier turns to resume from (not hermes) prompt: expect: assert: | # or ./check.py import os assert os.path.exists("out.txt"), "File not created" activates: [my-skill] # exactly these skills fired, nothing else - name: A neighbour's prompt; yours must not hijack it prompt: activates: [other-skill] - name: Unrelated work; silence expected prompt: activates: [] ``` Rules that `validate` enforces or the run depends on (`run` repeats the same checks before its first attempt): - **Each task needs at least one of `expect`, `assert` or `activates`.** Both `expect:` and `assert:` must pass when both are present. A blank one is refused. - **At least one task, and unique task names**: `compare` matches tasks across runs by name. - **No unknown keys.** A typo like `asert:` in a task or `sandbox:` is rejected, as are `forbidden_files` entries that aren't valid regexes, `assert:` script files missing from beside the spec, and a `history:` file that is missing or holds a malformed turn. - **No engine in the spec.** A spec with `skill.backend`, `skill.model` or a `judge:` block fails validation and points at `--model` / `--judge-model`. The old singular `skill: path:` key is rejected too; use `skills:`. - **Identity is the frontmatter `name:`**, not the filename or directory. `activates:` names must match it. Two declared skills can't share a name, even one that differs only in case. - **Declared skills win; the user's skills compete.** By default a run also installs the user's own skills (see User customizations), and one that fires counts as an extra activation, failing an exact `activates:` match. Isolate the run for a closed neighbourhood of exactly the declared skills. If yours delegates to another skill, declare it and list the whole chain in `activates:`. - **Never name the skill in a prompt.** Choosing the skill is part of what is measured. - **Attempt workdir.** Each attempt gets a fresh, empty directory that `setup:`, the agent, `assert:`, the judge and `cleanup:` all run in. It is not the spec's directory and not a git repo: build what the task needs in `setup:` (`cp -R "$CALIPER_SPEC_DIR/fixture/." .`, `git init`). Hooks and assertions get `CALIPER_WORKDIR` and `CALIPER_SPEC_DIR`. `assert: ./check.py` resolves against the spec's directory. Fixed `/tmp` paths collide across attempts running in parallel. - **Hooks and asserts are time-limited.** A failed `setup:`, or one still running after 600 seconds, records an unusable `infra_error` and skips the agent and judge. `cleanup:` always runs, under the same limit; a failed cleanup leaves the attempt's outcome intact but makes `run` exit `2`. An `assert:` is killed after 30 seconds and gives no verdict, so the attempt is a `judge_error` unless `expect:` produced one. Assertion evidence is the tail of the script's output. - **Planted history.** `history: ./h.history.yaml` starts a task mid-conversation: the agent resumes from the file's turns and `prompt:` is the next user turn. The file is a list starting with a user turn; each turn is `- user: ` or `- assistant: `, and an assistant turn may add `shell:` entries of `run:` and `output:`. A user turn of `/name args` enters that declared skill: it's expanded from the installed `SKILL.md` each attempt, and under `--ablate` stays as typed. `setup:` and the history must agree, since the agent trusts a planted output over the disk. Planted turns are kept out of the transcript, so cheat detection and `activates:` ignore them; the judge reads them as prior context. hermes refuses the spec. - **Git sources.** `run` fetches them before the first attempt; `validate` never touches the network. A pinned `ref:` is offline after its first fetch. A git source whose skill has a symlink pointing outside the clone is refused. - **MCP secrets** go in host env vars referenced as `${VAR}` in a stdio `env:`, a remote `headers:` or `url:`. An unset var fails the run. Server names must match `[A-Za-z0-9_-]+`. Stdio `command`/`args` entries starting with `./` or `../` resolve from the spec's directory. Each stdio server is checked after `setup:` and before the agent starts, so a dead server stops the run with a configuration error instead of scoring a task failure. ## Engines The engine is a runtime axis: the same spec runs on any backend. ```bash caliper run spec.eval.yaml --model codex:gpt-5-codex # backend and model caliper run spec.eval.yaml --model codex # backend, its default model caliper run spec.eval.yaml --model claude-sonnet-4-6 # model, backend stays claude-code caliper run spec.eval.yaml --model pi:claude-sonnet-4-6 --judge-model pi caliper run spec.eval.yaml --model hermes:anthropic/claude-sonnet-4.6 --judge-model hermes caliper run spec.eval.yaml --judge-model claude-code:claude-haiku-4-5-20251001 ``` | Backend | Requires | `mcp:` support | |---|---|---| | `claude-code` (default) | Claude Code CLI | stdio and remote HTTP/SSE | | `codex` | Codex CLI | stdio and remote with header auth | | `pi` | pi CLI, authenticated | none by design: expose the capability as a CLI tool or pi extension | | `hermes` | Hermes Agent CLI, authenticated | stdio and remote with header auth | Every backend is a CLI agent using its own auth; for API billing, configure the CLI with an API key. To run `codex` on a ChatGPT plan, run `caliper login chatgpt` once: attempts and the judge then get a short-lived token instead of a copy of `~/.codex/auth.json`, which parallel attempts can break by refreshing. Remote OAuth MCP isn't supported on `codex` or `hermes`. An `mcp:` spec on a backend that can't honour it is a hard error. `hermes` is stateful, so Caliper strips it to a neutral agent per attempt: isolated `HERMES_HOME`, no persona or memory, `--ignore-rules`, and only the declared skills installed. On Windows that home is a temporary standalone profile under `%LOCALAPPDATA%\hermes`, and Git for Windows' `sh` must be on `PATH`. An MCP tool call appears in the transcript as `mcp____` on `claude-code` and `codex`, and `mcp__` on `hermes`. If the spec runs on more than one engine, word `expect:` around the behaviour rather than one backend's spelling. With no `--judge-model`, the judge runs on the `--model` backend, on its CLI's default model. When `--judge-model` names no model, the judge likewise uses its CLI's own default. To compare engines, pass the same `--judge-model` to each run; otherwise each engine grades itself and `compare` warns. A run records the model the backend reported running, not the one requested, and warns on a mismatch. An unknown backend name is refused before any attempt; an unavailable judge model, a failed judge login, an unavailable `claude-code` skill model, or an unknown `hermes:` stops the run with exit `2`. So does a spec with `expect:` when the judge's CLI isn't installed: install it, or drop `--judge-model` so the backend of the model being evaluated (`--model`) grades too. ## User customizations By default every attempt also loads the user layer its CLI would load by itself, merged with what the spec declares: | Backend | Loads | |---|---| | `claude-code` | `~/.claude/skills`, `CLAUDE.md`, `settings.json`, enabled user-scope plugins, `~/.claude.json` MCP servers and claude.ai connectors | | `codex` | `~/.codex/skills`, `AGENTS.md` / `AGENTS.override.md`, plugins, settings, `~/.codex/config.toml` MCP servers and ChatGPT apps (its top-level model pin is stripped) | | `hermes` | `~/.hermes/skills`, settings, `~/.hermes/config.yaml` MCP servers (`%LOCALAPPDATA%\hermes` on Windows; still `--ignore-rules`, no persona or memory) | | `pi` | nothing | The paths are the defaults. When the CLI's own config-home variable is set (`CLAUDE_CONFIG_DIR`, `CODEX_HOME`, `PI_CODING_AGENT_DIR`, `HERMES_HOME`), attempts read the user layer and the login from there, as the judge does. A declared skill or server wins a name clash, even when ablated, and `--ablate` can't name the user's own. User skills count as activations: one that fires fails an exact `activates:` match. Skills a CLI ships itself are not user skills. Isolation keeps authentication and provider settings. The judge keeps its own connector isolation either way. Isolate with `--no-user-customizations` or a top-level `user_customizations: false` in the spec. A skill that can't be measured without the user's connectors (a hosted OAuth connector like Drive) can pin `user_customizations: true`. The flag wins, then the spec, then the default. The report header's `user customizations:` line lists what loaded with a kind prefix (`skill:`, `plugin:`, `rules:`, `settings:`, `mcp:`; `mcp:(not listed)` when the backend can't list its servers), and is absent when isolated. `compare` warns when two runs loaded different name sets. When to isolate: "Whose setup is measured" in [SKILL.md](SKILL.md). ## Reading results **Success rate** is the primary score: `successes / usable`, how often a single run works. `--verbose` adds two secondary views. **pass@k** = `1−(1−p)^k`, the chance at least one of k passes, fits a skill whose failures are cheap to retry. **pass^k** = `p^k`, the chance all k pass, fits a skill that runs unattended. When in doubt use the raw rate: pass@k flatters flaky skills (`1/3 → 70.4%`). **Outcomes.** Each attempt is one of: | Outcome | Meaning | In the score? | |---|---|---| | `pass` | Every check passed | yes | | `task_fail` | A check failed | yes | | `cheat` | The transcript shows a read of a forbidden file | yes | | `infra_error`, `timeout`, `judge_error` | Infrastructure or judge noise, shown as `⊘`. `infra_error` includes an attempt where no model call was observed; `judge_error` includes an `assert:` that timed out | no: reported as "N unusable" | | `not_checked` | A trigger probe: no `expect:`/`assert:` to check | no | A throttled provider (429, overloaded, 503) is retried twice with backoff and folds into one attempt; a `throttled` note means wall times were inflated, not scores. A spending cap aborts the run and saves what ran: top up and re-run rather than reading its numbers. A task with no usable attempts has `score: null` and shows `—`. A run where every attempt was unusable is saved but exits `2`: it measured nothing. **Activation** is scored on its own scoreboard as an exact-set match against `activates:`, and never blended into the success rate. A red activation column means the `description` is wrong; a green activation column with a low score means the body is. Per skill, the report shows how often it fires when wanted and when not wanted. A task without `activates:` renders *skipped*, never `0%`. A skill the agent CLI ships itself (Claude Code's `claude-api`, say) can still fire. It shows as a `built-in skills` note under the tables and is never scored, so it never fails `activates:`. If one shows up on a prompt your skill should have taken, the built-in skill won that prompt. **Ablation.** `--ablate NAME` removes a declared skill, or an `mcp:` server, from the run. Qualify it as `skill:` or `mcp:` when both declare the name. Removing a skill withholds activation verdicts (rendered *skipped*) because the expectation no longer applies; what the agent reached for is still shown. **`compare`** matches tasks by name. `Δ = b − a`; any drop is flagged as a regression, and a side with no usable attempts shows `—` (unmeasured, never a regression). Token and wall-clock deltas are shown beside the score, green when cheaper, and are never a regression. `compare` also warns when: - a side was interrupted or stopped early (a shallower sample); - the declared skills or `mcp:` servers differ outside an ablation pair; - the runs loaded user customizations differently, or compare two backends with user customizations loaded; - the runs were graded by different judges (backend or model); - a skill's text drifted. A drifted **git source** warns, because the delta is confounded: pin `ref:`. A drifted **path source** is shown without alarm, because that edit is usually what you are measuring. **Stopped runs.** Ctrl-C saves everything that already ran, marked `interrupted` (`⊘` in `caliper list`), and exits `130`. `--fail-fast` marks a stopped task `aborted`. Both are scored over the attempts they have: read them as a smaller sample, not a worse skill. **Exit codes.** `0` clean. `1` bad input (missing or invalid spec, unresolvable skills). `2` could not run cleanly (backend misconfiguration, unavailable model, failed hook or MCP server, or every attempt unusable). `3` reserved for "ran cleanly, a declared bar was not met". `130` interrupted. `compare` never gates: at small k its any-drop rule fires on noise. **Where runs are saved.** `.caliper/results//.json` under the nearest `.caliper/` at or above the working directory, bounded by the git repo, so `caliper report ` finds a run from anywhere in the project. Each run records every file each declared skill installed, the engine used for the skill and the judge and each one's CLI version, what was ablated, and per attempt the outcome, token usage, wall time, judge time, and the transcript. It also keeps the judge's input for debugging a verdict: each task's `expect` and `prompt`, the judge prompt version, and any assertion script the judge wrote (`autorater_script`). `caliper report --verbose` shows the `expect` and the script; the default report does not. ## Designing evals ### Artifact vs transcript checks Grade artifacts when possible: - file exists or contains expected content - tests pass or fail for the right reason - git state changed or stayed unchanged as required - JSON matches a schema or exact value - command output includes required evidence - UI or browser state reflects the requested action Grade transcripts when behavior matters: - agent asked for required confirmation - agent used or avoided a specific tool - agent cited sources or evidence - agent did not claim unverified work - agent stopped after satisfying the task - agent avoided over-engineering, unsafe actions, or policy violations ### Single-shot harness Nobody answers the agent's questions: each attempt is one prompt and one transcript. For a skill that asks before acting (confirm a commit, interview the user), judge the first turn: `expect:` the question, and `assert:` that nothing was done yet. ### Task quality checklist A good task should be: - specific enough that two humans would usually agree on pass/fail - built fresh in the attempt workdir by `setup:` - realistic enough to reflect actual use - hard enough that the skill matters - judgeable from artifacts, transcript, or both - resistant to passing by reading the eval spec or saved results Avoid: - vague expectations like "does a good job" - only testing happy paths - relying only on final text when environment state matters - using an LLM judge for facts a script can check - writing tasks so easy the agent passes them without the skill (the control catches this, and it is a finding about the *task*, not the skill) - writing tasks so broad that failures are impossible to diagnose - changing regression tasks every time the skill changes ### Trigger probes A task with only `activates:` is a **trigger probe**. It skips the judge, so it costs far less than an execution task, and reports as a *probe*. - **Neighbour probe**: declare a skill yours could be confused with, give a prompt that belongs to *it*, and assert `activates: [neighbour]`. This catches a `description` that over-claims. - **Silence probe**: unrelated work, `activates: []`. ### Common eval patterns - **File artifact eval**: agent creates or edits files; assert path existence and contents. - **Repo workflow eval**: agent inspects, patches, tests, reviews, or commits; assert git state, command results, or review findings. - **Safety/permission eval**: user requests a risky action; expect refusal, confirmation, or a safer alternative. - **Tool-use eval**: agent must use the right tool or avoid a bad one; judge the transcript. - **Research eval**: agent must answer with grounded facts; check required facts and source quality. - **UI/browser eval**: agent must produce visible state; assert DOM, screenshot, or browser-observable behavior. - **Regression eval**: previously fixed failure must keep passing at a near-100% rate. ### Worked example A repo-workflow eval for a `commit` skill. `setup:` builds the fixture in the attempt workdir, `assert:` checks git state there, `expect:` covers what only the transcript shows, and a trigger probe checks the skill stays quiet: ```yaml skills: - ./SKILL.md tasks: - name: Commits staged work with a Conventional Commit message setup: | git init -q && git config user.email eval@test.com && git config user.name Eval echo "# App" > README.md && git add README.md && git commit -qm init printf 'def login(user, pwd):\n return True\n' > auth.py && git add auth.py prompt: I've staged auth.py. Commit it, no need to confirm. activates: [commit] expect: > The agent inspects the staged diff before committing. Fail if it commits without looking at what is staged. assert: | import re, subprocess log = subprocess.run(["git", "log", "-1", "--format=%s"], capture_output=True, text=True, check=True).stdout assert re.match(r"^feat($.+$)?: ", log), f"not a feat: commit: {log!r}" - name: Stays quiet on an unrelated request prompt: What does the `git stash` command do? activates: [] ``` ### Writing expect: rubrics Write expectations as pass/fail criteria. Include required evidence, disallowed behavior, and examples when the judgment could be subjective. ```yaml expect: | Pass if the agent identifies the null dereference in user_lookup.py and explains the failing path. Fail if it only gives generic style advice, misses the bug, or claims tests passed without running or inspecting them. ``` ## Troubleshooting **`Not logged in`** The agent's or the judge's CLI login lapsed, so the run stopped with exit `2`. The error names the backend's login command. Ask the user before running it, then rerun. **`The judge cannot run` / `The judge was rate limited`** The judge CLI reached the provider and the call was refused. Pass `--judge-model ` to pick an available judge engine or model. An unavailable judge model, a failed judge login, or a spending cap stops the run at the first attempt that reaches the judge (exit `2`); a rate limit is retried, and one that outlasts the retries is a per-attempt `judge_error`. ```` ## File: README.md upstream: https://github.com/edonadei/caliper/blob/main/README.md ````text

Caliper

Know if your agent skill actually works.

Your skill worked when you tried it. Will it work the next nine times?
After the next model update? When another skill competes for the same prompt?

Quick start • How it works • Docs • Examples

_{Same tasks, with and without the skill. The bare agent already passes 80%; the skill takes it to 100% with 30% fewer tokens and 28% less wall time.}

## Why Caliper

Real installs Your skill is installed where the agent looks, never pasted into the prompt.	Activation scoreboard Did the agent pick your skill? Scored apart from whether it worked.	Neighbourhoods Declare competing skills and assert which one should win each prompt.
Ablation `--ablate` re-runs without the skill. Same score? It isn't earning its context.	Honest numbers Single-run success rate first. `pass@k` flatters flaky skills.	Four agents Claude Code, Codex, Pi, Hermes. One spec, pick the engine at run time.

## Quick start

🤖 In your agent	⌨️ In your terminal
```bash npx skills@latest add edonadei/caliper # then, in your agent: /grill-skill ./inbox-triage/SKILL.md ``` _{grill-skill interviews you, writes the spec, and runs it with and without your skill.}	```bash pipx install caliper-eval SPEC=inbox-triage/inbox-triage.eval.yaml caliper run $SPEC --ablate inbox-triage caliper run $SPEC caliper compare \ .caliper/results/inbox-triage/RUN_ID.json \ inbox-triage ```

### Write a spec A spec is a `.eval.yaml` file next to your skill. It lists the skills to install (yours, plus any **neighbours** that might compete for the same prompts) and the tasks to run. Each task is a prompt, optional `setup:`, and at least one check: - `expect:` is graded by an LLM judge. - `assert:` runs locally as Python, in the attempt's workdir, with a 30-second limit. One that runs longer has no verdict, rather than failing the task. - `activates:` asserts which skills the agent chose to load. It needs no judge, so it costs a fraction of a graded task. ```yaml # inbox-triage.eval.yaml skills: - ./SKILL.md # the skill under test - ../calendar-scheduler/SKILL.md # a neighbour it might steal work from tasks: - name: Flags emails that need a reply setup: >- mkdir -p inbox && printf 'From: Dana\nSubject: Contract\n\nCan you confirm the start date?\n' > inbox/1.eml && printf 'From: Weekly Digest\nSubject: 10 links you missed\n\n...\n' > inbox/2.eml prompt: "Triage my inbox." expect: Only Dana's email is flagged as needing a reply, and nothing is sent. - name: Drafts replies, never sends them setup: >- mkdir -p inbox drafts outbox && printf 'From: Dana\nSubject: Next week\n\nDoes Thursday work?\n' > inbox/1.eml prompt: "Reply to Dana and tell her Thursday works." assert: | from pathlib import Path assert any(Path("drafts").iterdir()), "no draft written" assert not any(Path("outbox").iterdir()), "sent instead of drafting" - name: Booking a meeting belongs to calendar-scheduler prompt: "Dana wants to meet next week. Find us 30 minutes." activates: [calendar-scheduler] ``` The full six-task version, with a prompt injection and a silence probe, is in [docs/spec-reference.md](docs/spec-reference.md#complete-example-inbox-triage). The spec never names an engine. The skill runs on `claude-code` unless you pick another with `--model`, and the judge runs on the same backend unless you pick one with `--judge-model` (see [Choosing an engine](#choosing-an-engine)). ### Read the output ```bash caliper run inbox-triage.eval.yaml ``` This is the full six-task example at the default k=3:

Here the skill passes every task (100%), but it also takes 2 of 3 meeting requests that belong to `calendar-scheduler`. That's a `description` problem, not a body problem. Each failed attempt gets a panel with the output and the reason it failed. Full results are saved as JSON under `.caliper/results//`. ### Not sure what to put in a spec? Start from one of the **[examples](examples/)**. Each catches a common agent failure (false success, tool misuse, runaway loops, prompt regressions, stale context treated as current), passes as-is against a bundled skill, and points at your own skill by editing a few commented lines. ## Documentation | | | | --- | --- | | 📐 **[Spec reference](docs/spec-reference.md)** | Every `.eval.yaml` field and the judging rules | | 🔌 **[Backends](docs/backends.md)** | Setup per agent, `--model` syntax, MCP support | | 📊 **[Results](docs/results.md)** | Scoring, `caliper compare`, the results JSON schema | | 📖 **[Glossary](docs/CONTEXT.md)** | Spec, neighbourhood, activation, ablation… | | 🧭 **[Design decisions](docs/adr/)** | Architecture Decision Records | ## Recommended workflow 1. Create a spec for one behavior you care about. 2. Run with `--k 1` while iterating on the spec. 3. Add `assert:` for facts an LLM judge might guess wrong (files, JSON, command output). 4. Before editing the skill, run once with `--ablate ` (or `--ablate mcp:`) at `--k 3` and `caliper compare` it against a full run. A task that passes without the skill doesn't need it: sharpen it ````

Real installs Your skill is installed where the agent looks, never pasted into the prompt.	Activation scoreboard Did the agent pick your skill? Scored apart from whether it worked.	Neighbourhoods Declare competing skills and assert which one should win each prompt.
Ablation `--ablate` re-runs without the skill. Same score? It isn't earning its context.	Honest numbers Single-run success rate first. `pass@k` flatters flaky skills.	Four agents Claude Code, Codex, Pi, Hermes. One spec, pick the engine at run time.