evaluate-skill

4 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

• skills/evaluate-skill/SKILL.md7.7 KB3645ff270baaupstream ↗

  ---
  name: evaluate-skill
  description: Run, read, and diagnose a skill's Caliper eval — its success rate over k attempts, whether it fires, and whether it beats the agent without it. Use when the user wants to run, validate, interpret, or compare a skill's eval, or write an .eval.yaml spec whose tasks they have already decided.
  allowed-tools: Bash
  ---
  
  # Evaluate Skill
  
  Operate Caliper: run a skill's eval, read what the results say, and trace each failure to the place that fixes it.
  
  ## Prerequisites
  
  The `caliper` CLI must be on `PATH`. This skill can be copied into an agent without the Caliper repo, so install it if missing:
  
  ```bash
  pipx install caliper-eval
  ```
  
  `caliper <command> --help` is the authority on flags. [REFERENCE.md](REFERENCE.md) covers the full spec format, engines, and what the results mean.
  
  ## The four questions
  
  A skill eval answers four separate questions. Each has its own measurement and its own fix location, so keep them apart: a blended number hides which half broke.
  
  | Question | Measured by | A failure is fixed in |
  |---|---|---|
  | **Fires**: does the agent reach for the skill when it should, and only then? | `activates:` on tasks, and trigger probes | the skill's `description` frontmatter |
  | **Works**: once it fires, does it get the job done? | `expect:` / `assert:`, scored as the success rate | the skill's body |
  | **Earns**: does it beat the agent without it? | the control (`--ablate <skill-name>`: the declared neighbourhood minus this skill) and `caliper compare`. For a truly bare agent, ablate every declared skill and every declared `mcp:` server, and isolate the run | the tasks: if the control passes too, the task is too easy |
  | **Holds**: does it stay good across edits and over time? | `caliper compare` of each full run against the previous one, skill drift | the edit that moved it |
  
  ## Spec shape
  
  ```yaml
  skills:
    - ./SKILL.md              # relative to the spec; installed, never preloaded
  tasks:
    - name: What success looks like
      prompt: <what a real user would type; never names the skill>
      expect: <natural-language pass/fail criterion>
      assert: |               # optional deterministic Python check
        assert ...
      activates: [my-skill]   # frontmatter name: of the skill that should fire
    - name: Unrelated work stays unrelated
      prompt: <work no declared skill should answer>
      activates: []           # a trigger probe: no judge, cheap
  ```
  
  When you write a spec, every task with `expect:` or `assert:` also asserts `activates:`, refusals included: the skill, plus any declared skill it delegates to on that task, and every prompt reads like a real user's request with the skill left unnamed.
  
  The spec has no `backend`/`model` or `judge:` block. The engine is chosen at run time, independently for the skill and the judge: `caliper run <spec> --model codex` runs and grades on codex, and `--judge-model` picks a different judge. Backends are `claude-code` (default), `codex`, `pi`, and `hermes`. Each attempt runs in a fresh, empty workdir, so `setup:` builds fixtures there with relative paths.
  
  ## Running
  
  1. `caliper validate <spec>`. It never touches the network.
  2. `caliper run <spec> --k 1` to shake out spec and harness errors. Fix those before reading any score.
  3. `caliper run <spec> --k 3 --ablate <skill-name>`, once (write `skill:<skill-name>` if an `mcp:` server shares the name). This is the **control**: the declared neighbourhood without this skill. The skill isn't installed, so editing `SKILL.md` can't move its number. Keep its results path (`caliper list <spec-name>` marks ablated runs) and re-diff against it. Re-run it only when the tasks or the declared skills change.
  4. `caliper run <spec> --k 3`, then `caliper compare <control.json> <spec-name>` to see whether it earns its place. After each edit, also compare against the previous full run's path to see whether the edit held: a skill that got worse can still beat the control. A bare spec name resolves to that spec's latest run.
  
  If a run stops with `Not logged in`, ask the user whether to run the login command the error names. With their consent, run it (it opens a browser for them to finish) and rerun; never run it without consent. `hermes model` is an interactive picker, so ask the user to run it in their own terminal instead.
  
  ## Reading results
  
  The **success rate** (`successes / usable`) is the headline. pass@k and pass^k are secondary views under `--verbose`. Activation is a separate scoreboard, over a different population: report it beside the success rate, never averaged into it.
  
  Trace every failing task to where its fix belongs:
  
  | Signal | Where the fix belongs |
  |---|---|
  | Run exits `2` (backend misconfigured, unavailable model, failed hook or MCP server, or every attempt unusable) | The environment or the task's hooks. Report it as a configuration problem, not a task failure |
  | `⊘` unusable attempts (`infra_error`, `timeout`, `judge_error`) | Not the skill: rate limits, auth, or the judge. They are excluded from the score; re-run |
  | `cheat` outcome | The task: it leaks its answer. Tighten the task or `sandbox:` |
  | Activation fails: an expected skill didn't fire | That skill's `description` |
  | Activation fails: an unexpected skill fired too | The extra skill's `description`, or the overlap between the two |
  | Activation fails because one of the user's own skills fired (`skill:` in the report header) | Not the `description` alone: that skill is real competition in their setup. Decide with the user whether to sharpen the description or isolate the run |
  | Activation passes, score low | The skill's body |
  | The judge's reasoning shows `expect:` was ambiguous | The task's grading: make the criterion observable, or add an `assert:` |
  | Full run ≈ control, and the skill fired in the full run | The task: it doesn't need the skill. If the skill never fired, the fix is its `description` (above) |
  
  At k=3 one attempt is a 33-point swing. Before calling a change a win or a regression, re-run at k≥5. `caliper compare` flags any drop and never gates. A rewritten or shortened skill is safe to ship when, at k≥5, it stays within about 5% of the previous score and still beats the control.
  
  ## Whose setup is measured
  
  Runs load the user's own customizations by default (user skills, plugins, rules, settings and connectors; see REFERENCE.md for backend exceptions), which answers "does my skill work in *my* agent?". **Isolate** (`--no-user-customizations`, or `user_customizations: false` in the spec) when comparing backends or models, when the number leaves this machine (shared, published, compared with someone else's run), or when measuring the bare agent: each setup is different, so otherwise part of the delta is the setups. `--ablate` of the user's own skill needs no isolation, since both runs load the same setup.
  
  **Always tell the user which mode ran** and what it loaded, from the report header's `user customizations:` line (absent means isolated), and relay any fix `caliper compare` suggests about it.
  
  ## No eval yet?
  
  If the user hasn't decided what to test (a skill with no `.eval.yaml`, or an eval whose gaps they want found), suggest `grill-skill`: it interviews them and writes the spec. To design tasks yourself, follow "Designing evals" in [REFERENCE.md](REFERENCE.md).
  
  ## Done when
  
  - every task has an observable criterion, and at least one has a deterministic `assert:`;
  - execution tasks assert `activates:`, and the spec has at least one trigger probe;
  - the full run beats the control on execution tasks, and holds against the previous full run. Trigger probes have no execution score: read them from the full run's activation;
  - the spec passes `caliper validate`;
  - the user has been told to commit the `.eval.yaml` beside `SKILL.md`. Saved runs under `.caliper/results/` are useful for diffing over time and safe to gitignore.
  

• skills/evaluate-skill/evaluate-skill.eval.yaml14 KB481268358dc5upstream ↗

  skills:
    - ./SKILL.md
    # grill-skill is the neighbour: both talk about evals, so every activates:
    # below also checks that the right one of the two fires.
    - ../grill-skill/SKILL.md
  
  # Isolated: these numbers are published, and the user's own skills would
  # otherwise compete for every activates: below (docs/adr/0028).
  user_customizations: false
  
  # The engine (backend + model) is a runtime axis, not a spec field — run this
  # eval with `--model codex --judge-model codex` (or any other engine).
  
  # No explicit sandbox.forbidden_files: caliper auto-forbids this spec and any
  # .caliper/results/ directory. Listing "./.caliper/.*" by hand risks a false
  # cheat flag, because these tasks legitimately WRITE caliper specs whose text
  # contains that very pattern — see the same note in grill-skill.eval.yaml.
  
  tasks:
    # Every attempt runs in a fresh attempt workdir (docs/adr/0026), so fixtures
    # are built there with relative paths — fixed /tmp paths would collide across
    # attempts running in parallel.
    - name: Validates a well-formed spec and reports it as valid
      activates: [evaluate-skill]
      setup: |
        printf -- '---\nname: arithmetic\ndescription: Use when asked to do arithmetic.\n---\n\n# Arithmetic\n' > SKILL.md
        cat > valid.eval.yaml << 'EOF'
        skills:
          - ./SKILL.md
        tasks:
          - name: Test arithmetic
            prompt: What is 2 + 2?
            expect: The assistant answers 4.
        EOF
      prompt: >
        Use caliper to validate the spec file at ./valid.eval.yaml and tell me
        whether it is valid.
      expect: >
        The agent runs caliper validate on ./valid.eval.yaml and reports that the
        spec is valid with no errors.
      assert: |
        import subprocess
  
        result = subprocess.run(
            ["caliper", "validate", "valid.eval.yaml"],
            capture_output=True, text=True
        )
        assert result.returncode == 0, f"caliper validate exited {result.returncode}: {result.stderr}"
  
    - name: Identifies errors in an invalid spec without fixing it
      activates: [evaluate-skill]
      setup: |
        printf -- '---\nname: arithmetic\ndescription: Use when asked to do arithmetic.\n---\n\n# Arithmetic\n' > SKILL.md
        cat > invalid.eval.yaml << 'EOF'
        skills:
          - ./SKILL.md
        tasks:
          - name: Broken task
            prompt: do something
        EOF
      prompt: >
        Use caliper to validate the spec file at ./invalid.eval.yaml and tell me
        what errors it contains.
      expect: >
        The agent runs caliper validate and reports that the spec is invalid,
        describing that the task is missing an expect, assert or activates field.
        The agent does not edit the invalid spec file.
      assert: |
        import yaml
  
        with open("invalid.eval.yaml") as f:
            spec = yaml.safe_load(f)
        task = spec["tasks"][0]
        assert set(task) == {"name", "prompt"}, f"The invalid spec was edited: {task}"
  
    - name: Creates an engineless eval spec and defers the engine to run time
      activates: [evaluate-skill]
      prompt: >
        Create an evaluation spec file at ./created.eval.yaml for a skill at
        ./SKILL.md that I intend to run on the Codex CLI. Include one task named
        "Answers arithmetic", with prompt "What is 2 + 2?" and expectation "The
        assistant answers 4." Then tell me how to run it against Codex.
      expect: >
        A valid .eval.yaml file is written at ./created.eval.yaml with a
        top-level skills: list containing ./SKILL.md and the requested task. The
        spec does NOT pin a backend or model (no skill.backend/model, no judge
        block) because the engine is a runtime axis, and the agent tells the user
        to select Codex at run time with `caliper run ... --model codex`
        (optionally `--judge-model codex`).
      assert: |
        import os
        import yaml
  
        path = "created.eval.yaml"
        assert os.path.exists(path), "Spec file was not created"
        with open(path) as f:
            spec = yaml.safe_load(f)
        assert "skill" not in spec, "`skill:` was replaced by the `skills:` list"
        skills = spec.get("skills")
        assert isinstance(skills, list) and skills, \
            f"spec needs a top-level skills: list, got {skills!r}"
        def pa(e): return e if isinstance(e, str) else (e or {}).get("path")
        assert pa(skills[0]) == "./SKILL.md", "skills[0] should be ./SKILL.md"
        assert "judge" not in spec, "engine is a runtime axis: no judge block belongs in the spec"
        assert len(spec.get("tasks", [])) == 1, "Expected exactly one task"
        task = spec["tasks"][0]
        assert task["name"] == "Answers arithmetic", "Unexpected task name"
        assert task["prompt"] == "What is 2 + 2?", "Unexpected prompt"
        assert task["expect"] == "The assistant answers 4.", "Unexpected expectation"
  
    - name: Explains independent skill and judge engines are selected at run time
      activates: [evaluate-skill]
      prompt: >
        Create an evaluation spec file at ./mixed-backend.eval.yaml that evaluates
        a Claude Code skill at ~/.claude/skills/review/SKILL.md but is judged by
        Codex. Include one task named "Reviews staged changes", prompt "Review the
        staged changes", and expectation "The review reports at least one
        actionable issue." Then tell me the exact command to run it that way.
      expect: >
        The agent writes a valid engineless spec (a skills: list containing
        ~/.claude/skills/review/SKILL.md, no backend/model, no judge block) and
        explains that the skill and judge engines are chosen independently at run
        time — e.g. `caliper run ... --model claude-code --judge-model codex` —
        because Caliper treats the engine as a runtime axis, not a spec field.
      assert: |
        import os
        import yaml
  
        path = "mixed-backend.eval.yaml"
        assert os.path.exists(path), "Spec file was not created"
        with open(path) as f:
            spec = yaml.safe_load(f)
        assert "skill" not in spec, "`skill:` was replaced by the `skills:` list"
        skills = spec.get("skills")
        assert isinstance(skills, list) and skills, \
            f"spec needs a top-level skills: list, got {skills!r}"
        def pa(e): return e if isinstance(e, str) else (e or {}).get("path")
        assert pa(skills[0]) == "~/.claude/skills/review/SKILL.md", "Unexpected skill path"
        assert "judge" not in spec, "engine is a runtime axis: no judge block belongs in the spec"
        assert spec["tasks"][0]["name"] == "Reviews staged changes", "Unexpected task name"
  
    - name: Lists available evaluations gracefully
      activates: [evaluate-skill]
      prompt: >
        Use caliper to list all available evaluations and tell me what you find.
      expect: >
        The agent runs caliper list and either shows a table of available evaluations
        with scores and timestamps, or clearly states that no results have been recorded yet
  
    - name: Surfaces Claude Code configuration errors clearly
      activates: [evaluate-skill]
      setup: |
        mkdir claude-code-smoke
        cat > claude-code-smoke/SKILL.md << 'EOF'
        ---
        name: claude-code-smoke
        description: Use when asked to run the Claude Code smoke evaluation.
        ---
  
        When asked to run the smoke evaluation, write exactly
        `claude-code-smoke-ok` to the requested output file.
        EOF
        cat > claude-code-smoke/claude-code-smoke.eval.yaml << 'EOF'
        skills:
          - ./SKILL.md
        tasks:
          - name: Writes the smoke output file
            prompt: Run the Claude Code smoke evaluation and write the result to ./smoke.txt.
            activates: [claude-code-smoke]
            assert: |
              from pathlib import Path
              assert Path("smoke.txt").read_text().strip() == "claude-code-smoke-ok"
        EOF
      prompt: >
        Run the Claude Code smoke eval at
        ./claude-code-smoke/claude-code-smoke.eval.yaml with k=1, one worker and
        a 60 second timeout. If Claude Code is not logged in or the account lacks
        access, report the Caliper configuration error clearly instead of
        describing it as a task failure.
      expect: >
        The agent runs caliper run ./claude-code-smoke/claude-code-smoke.eval.yaml
        with --k 1 --workers 1 --timeout 60. If the run exits with a Claude Code
        configuration error, the agent reports that configuration problem and the
        relevant login or access guidance. If Claude Code is configured, the agent
        reports the eval result normally.
  
    # Writing a spec whose tasks the user already decided is this skill's job
    # (grill-skill owns the interview). The prompt mentions neither rule, so
    # activates: on every execution task and skill-free prompts must come from
    # the skill.
    - name: Generated tasks assert activates and never name the skill
      activates: [evaluate-skill]
      setup: |
        mkdir -p hello-file
        cat > hello-file/SKILL.md << 'EOF'
        ---
        name: hello-file
        description: Use when the user wants to write a greeting file.
        allowed-tools: Bash
        ---
  
        # Hello File
  
        Write the text "hello world" to ./hello.txt when the user asks.
        EOF
      prompt: >
        Write the eval spec for my skill at ./hello-file/SKILL.md now, to
        ./hello-file/hello-file.eval.yaml, without asking me anything more; I have
        answered everything. Happy path: I ask for a greeting file and
        ./hello.txt contains "hello world". Edge case: I ask for the greeting
        "hi there" instead. Adversarial: I ask it to overwrite /etc/hosts, and it
        must refuse. Don't run it.
      expect: >
        A valid .eval.yaml is written at ./hello-file/hello-file.eval.yaml
        covering the happy path, edge case and adversarial case.
      assert: |
        import yaml
  
        with open("hello-file/hello-file.eval.yaml") as f:
            spec = yaml.safe_load(f)
        tasks = spec.get("tasks") or []
        execution = [t for t in tasks if "expect" in t or "assert" in t]
        assert execution, "no execution tasks"
        for t in execution:
            assert "hello-file" in (t.get("activates") or []), \
                f"task {t.get('name')!r} does not assert activates: [hello-file]"
        for t in tasks:
            prompt = str(t.get("prompt", "")).lower()
            assert "hello-file" not in prompt and "skill" not in prompt, \
                f"task {t.get('name')!r} names the skill in its prompt: {prompt!r}"
  
    # A user who has decided the tasks still gets trigger probes proposed:
    # graded on the proposal, not written to disk.
    - name: Proposes trigger probes for tasks the user has decided
      activates: [evaluate-skill]
      setup: |
        mkdir -p changelog-writer
        cat > changelog-writer/SKILL.md << 'EOF'
        ---
        name: changelog-writer
        description: Use when the user wants to turn merged PRs into a changelog entry.
        allowed-tools: Bash, Read, Write
        ---
  
        # Changelog Writer
  
        Read the merged PRs since the last tag and write a grouped changelog
        entry (Features / Fixes / Chore) to CHANGELOG.md.
        EOF
      prompt: >
        I want an eval for my skill at ./changelog-writer/SKILL.md. Your reading
        is right: it runs `gh pr list --state merged` for PRs since the last git
        tag and writes a grouped entry to CHANGELOG.md; a test can fake `gh` and
        tag the repo in setup. Happy path: three merged PRs (a feature, a fix, a
        chore) become one entry with Features / Fixes / Chore headings. Edge
        case: a PR labelled both feature and fix goes under Features only.
        Adversarial: asked to rewrite old released entries, it should refuse. I
        don't know of any other skills it could be confused with. Before you
        write anything, show me the tasks you'd put in the spec.
      expect: >
        Pass if the agent shows a proposed set of tasks, without writing the spec
        file, and the proposal includes at least one trigger probe: a silence
        probe (unrelated work with `activates: []`) or a neighbour probe. Its
        execution tasks (happy path, edge case, adversarial) assert
        `activates: [changelog-writer]`. Fail if it writes the spec file, or its
        proposal has no trigger probe.
      assert: |
        import glob
        specs = glob.glob("changelog-writer/*.eval.yaml")
        assert not specs, f"Agent wrote the spec before showing the proposal: {specs}"
  
    # Advising the next run is operating an eval: the control comes first.
    - name: Runs the control before iterating on the skill
      activates: [evaluate-skill]
      setup: |
        mkdir -p changelog-writer
        cat > changelog-writer/SKILL.md << 'EOF'
        ---
        name: changelog-writer
        description: Use when the user wants to turn merged PRs into a changelog entry.
        allowed-tools: Bash, Read, Write
        ---
  
        # Changelog Writer
  
        Read the merged PRs since the last tag and write a grouped changelog
        entry (Features / Fixes / Chore) to CHANGELOG.md.
        EOF
        cat > changelog-writer/changelog-writer.eval.yaml << 'EOF'
        skills:
          - ./SKILL.md
        tasks:
          - name: Groups three merged PRs by type
            prompt: Write the changelog entry for the PRs merged since v1.0.
            expect: CHANGELOG.md gains one entry with Features, Fixes and Chore headings.
            activates: [changelog-writer]
        EOF
      prompt: >
        My eval at ./changelog-writer/changelog-writer.eval.yaml validated and its
        k=1 run was clean, no harness errors. Before I start editing SKILL.md,
        what run should come next, and why? Just tell me; don't run anything.
      expect: >
        Pass if the agent recommends running the control next, i.e. the same
        tasks with the skill removed (`--ablate changelog-writer`), alongside a
        full run and a `caliper compare`, to check that the tasks actually need
        the skill before iterating on it. Fail if it recommends going straight to
        editing the skill or to repeated full runs without that control.
  
    # No result-reading tasks: across two fixture sets (six tasks, k=1) the bare
    # agent read caliper's report and compare output as well as the skill did,
    # so they measured nothing the skill adds.
  
    # Trigger probes: no judge, cheap.
    - name: An interview request belongs to grill-skill
      activates: [grill-skill]
      setup: |
        mkdir -p changelog-writer
        printf -- '---\nname: changelog-writer\ndescription: Use when the user wants to turn merged PRs into a changelog entry.\n---\n\n# Changelog Writer\n' > changelog-writer/SKILL.md
      prompt: >
        I have a skill at ./changelog-writer/SKILL.md with no eval yet. Interview
        me about what it should do and help me design its eval tasks.
  
    - name: Unrelated work fires neither skill
      activates: []
      prompt: Write a haiku about autumn leaves to ./haiku.txt.
  

• skills/evaluate-skill/REFERENCE.md22 KB0af6510b7584upstream ↗

  # Caliper reference
  
  `caliper <command> --help` lists every flag. This file covers what `--help`
  doesn't: the spec format, how engines are chosen, what the results mean, and how
  to design tasks.
  
  ## Commands
  
  ```bash
  caliper validate spec.eval.yaml                   # offline check; exit 1 if invalid
  caliper run spec.eval.yaml --k 3                  # attempts per task (default 3)
  caliper run spec.eval.yaml --k 3 --ablate my-skill   # the control: that skill removed
  caliper run spec.eval.yaml --ablate a --ablate mcp:srv --no-user-customizations  # every skill and server, isolated: the bare agent
  caliper run spec.eval.yaml --no-user-customizations  # isolated: only the spec's mcp: servers
  caliper run spec.eval.yaml --verbose              # expect, judge reasoning and script, pass@k, pass^k
  
  caliper list                                      # every spec with its latest score
  caliper list my-skill                             # one spec's runs, ablated ones marked
  caliper report my-skill                           # latest run as a table
  caliper report my-skill --run 2026-05-12T14-23-01Z --verbose
  caliper report results.json --format json
  
  caliper compare control.json my-skill             # Δ = b − a, task by task
  caliper compare a.json b.json --format json
  ```
  
  `report` and `compare` take a spec name (its latest run) or a results-JSON path.
  Pin an older run by its path.
  
  Other `run` controls: `--workers N` (parallel attempts across all tasks, default
  4; more workers share one rate limit and risk more `infra_error`), `--timeout`
  (seconds per attempt, default 120), `--fail-fast N` (stop a task after N
  consecutive `infra_error`/`timeout` attempts), `--output PATH` (also save the
  results JSON there).
  
  ## Spec format (.eval.yaml)
  
  ```yaml
  skills:                     # the neighbourhood: every entry installed, none preloaded
    - ./SKILL.md              # path source, relative to the spec
    - repo: mattpocock/skills          # git source, cloned by caliper
      ref: 49dd158d1076134a641b33efb035946536778336  # optional: branch, tag or commit (short is fine); omit to track the default branch
      path: skills/engineering/tdd/SKILL.md  # one skill per entry; omit only if the repo root is a skill
  
  sandbox:
    forbidden_files:          # extra regexes only: the spec itself and any
      - "./answers/.*"        #   .caliper/ results are forbidden already
    extra_path: ["./bin"]     # optional; relative to the spec, prefixed to PATH
  
  user_customizations: false  # optional; isolate from the user's own skills and setup
  
  mcp:                        # optional MCP servers the agent may use
    weather:                  # local stdio server
      command: python3
      args: [./servers/weather.py]
      env:
        API_TOKEN: ${MCP_API_TOKEN}      # resolved from your shell at run time
    gdrive:                   # remote server
      type: http              # http or sse
      url: https://mcp.example.com/gdrive
      headers:
        Authorization: Bearer ${GDRIVE_TOKEN}
  
  tasks:
    - name: Short description of what success looks like
      setup: <shell command run in the attempt workdir>     # optional
      cleanup: <shell command>                              # optional
      history: ./histories/h.history.yaml   # optional: earlier turns to resume from (not hermes)
      prompt: <what a real user would type>
      expect: <natural-language pass/fail criterion>
      assert: |                                             # or ./check.py
        import os
        assert os.path.exists("out.txt"), "File not created"
      activates: [my-skill]   # exactly these skills fired, nothing else
  
    - name: A neighbour's prompt; yours must not hijack it
      prompt: <a prompt that belongs to another declared skill>
      activates: [other-skill]
  
    - name: Unrelated work; silence expected
      prompt: <a prompt no declared skill should answer>
      activates: []
  ```
  
  Rules that `validate` enforces or the run depends on (`run` repeats the same
  checks before its first attempt):
  
  - **Each task needs at least one of `expect`, `assert` or `activates`.** Both
    `expect:` and `assert:` must pass when both are present. A blank one is
    refused.
  - **At least one task, and unique task names**: `compare` matches tasks across
    runs by name.
  - **No unknown keys.** A typo like `asert:` in a task or `sandbox:` is rejected,
    as are `forbidden_files` entries that aren't valid regexes, `assert:`
    script files missing from beside the spec, and a `history:` file that is
    missing or holds a malformed turn.
  - **No engine in the spec.** A spec with `skill.backend`, `skill.model` or a
    `judge:` block fails validation and points at `--model` / `--judge-model`.
    The old singular `skill: path:` key is rejected too; use `skills:`.
  - **Identity is the frontmatter `name:`**, not the filename or directory.
    `activates:` names must match it. Two declared skills can't share a name,
    even one that differs only in case.
  - **Declared skills win; the user's skills compete.** By default a run also
    installs the user's own skills (see User customizations), and one that fires
    counts as an extra activation, failing an exact `activates:` match. Isolate
    the run for a closed neighbourhood of exactly the declared skills. If yours
    delegates to another skill, declare it and list the whole chain in
    `activates:`.
  - **Never name the skill in a prompt.** Choosing the skill is part of what is
    measured.
  - **Attempt workdir.** Each attempt gets a fresh, empty directory that `setup:`,
    the agent, `assert:`, the judge and `cleanup:` all run in. It is not the
    spec's directory and not a git repo: build what the task needs in `setup:`
    (`cp -R "$CALIPER_SPEC_DIR/fixture/." .`, `git init`). Hooks and assertions
    get `CALIPER_WORKDIR` and `CALIPER_SPEC_DIR`. `assert: ./check.py` resolves
    against the spec's directory. Fixed `/tmp` paths collide across attempts
    running in parallel.
  - **Hooks and asserts are time-limited.** A failed `setup:`, or one still
    running after 600 seconds, records an unusable `infra_error` and skips the
    agent and judge. `cleanup:` always runs, under the same limit; a failed
    cleanup leaves the attempt's outcome intact but makes `run` exit `2`. An
    `assert:` is killed after 30 seconds and gives no verdict, so the attempt is a
    `judge_error` unless `expect:` produced one. Assertion evidence is the tail of
    the script's output.
  - **Planted history.** `history: ./h.history.yaml` starts a task mid-conversation: the
    agent resumes from the file's turns and `prompt:` is the next user turn. The
    file is a list starting with a user turn; each turn is `- user: <text>` or
    `- assistant: <text>`, and an assistant turn may add `shell:` entries of
    `run:` and `output:`. A user turn of `/name args` enters that declared skill:
    it's expanded from the installed `SKILL.md` each attempt, and under
    `--ablate` stays as typed. `setup:` and the history must agree, since the
    agent trusts a planted output over the disk. Planted turns are kept out of the
    transcript, so cheat detection and `activates:` ignore them; the judge reads
    them as prior context. hermes refuses the spec.
  - **Git sources.** `run` fetches them before the first attempt; `validate` never
    touches the network. A pinned `ref:` is offline after its first fetch. A git
    source whose skill has a symlink pointing outside the clone is refused.
  - **MCP secrets** go in host env vars referenced as `${VAR}` in a stdio `env:`,
    a remote `headers:` or `url:`. An unset var fails the run. Server names must
    match `[A-Za-z0-9_-]+`. Stdio `command`/`args` entries starting with `./` or
    `../` resolve from the spec's directory. Each stdio server is checked after
    `setup:` and before the agent starts, so a dead server stops the run with a
    configuration error instead of scoring a task failure.
  
  ## Engines
  
  The engine is a runtime axis: the same spec runs on any backend.
  
  ```bash
  caliper run spec.eval.yaml --model codex:gpt-5-codex       # backend and model
  caliper run spec.eval.yaml --model codex                   # backend, its default model
  caliper run spec.eval.yaml --model claude-sonnet-4-6       # model, backend stays claude-code
  caliper run spec.eval.yaml --model pi:claude-sonnet-4-6 --judge-model pi
  caliper run spec.eval.yaml --model hermes:anthropic/claude-sonnet-4.6 --judge-model hermes
  caliper run spec.eval.yaml --judge-model claude-code:claude-haiku-4-5-20251001
  ```
  
  | Backend | Requires | `mcp:` support |
  |---|---|---|
  | `claude-code` (default) | Claude Code CLI | stdio and remote HTTP/SSE |
  | `codex` | Codex CLI | stdio and remote with header auth |
  | `pi` | pi CLI, authenticated | none by design: expose the capability as a CLI tool or pi extension |
  | `hermes` | Hermes Agent CLI, authenticated | stdio and remote with header auth |
  
  Every backend is a CLI agent using its own auth; for API billing, configure the
  CLI with an API key. To run `codex` on a ChatGPT plan, run `caliper login
  chatgpt` once: attempts and the judge then get a short-lived token instead of a
  copy of `~/.codex/auth.json`, which parallel attempts can break by refreshing. Remote OAuth MCP isn't supported on `codex` or `hermes`. An
  `mcp:` spec on a backend that can't honour it is a hard error.
  
  `hermes` is stateful, so Caliper strips it to a neutral agent per attempt:
  isolated `HERMES_HOME`, no persona or memory, `--ignore-rules`, and only the
  declared skills installed. On Windows that home is a temporary standalone
  profile under `%LOCALAPPDATA%\hermes`, and Git for Windows' `sh` must be on
  `PATH`.
  
  An MCP tool call appears in the transcript as `mcp__<server>__<tool>` on
  `claude-code` and `codex`, and `mcp_<server>_<tool>` on `hermes`. If the spec
  runs on more than one engine, word `expect:` around the behaviour rather than
  one backend's spelling.
  
  With no `--judge-model`, the judge runs on the `--model` backend, on its CLI's
  default model. When `--judge-model` names no model, the judge likewise uses its
  CLI's own default. To compare engines, pass the same `--judge-model` to each
  run; otherwise each engine grades itself and `compare` warns.
  
  A run records the model the backend reported running, not the one requested,
  and warns on a mismatch. An unknown backend name is refused before any attempt;
  an unavailable judge model, a failed judge login, an unavailable `claude-code`
  skill model, or an unknown `hermes:<model>` stops the run with exit `2`. So does a spec with
  `expect:` when the judge's CLI isn't installed: install it, or drop
  `--judge-model` so the backend of the model being evaluated (`--model`)
  grades too.
  
  ## User customizations
  
  By default every attempt also loads the user layer its CLI would load by
  itself, merged with what the spec declares:
  
  | Backend | Loads |
  |---|---|
  | `claude-code` | `~/.claude/skills`, `CLAUDE.md`, `settings.json`, enabled user-scope plugins, `~/.claude.json` MCP servers and claude.ai connectors |
  | `codex` | `~/.codex/skills`, `AGENTS.md` / `AGENTS.override.md`, plugins, settings, `~/.codex/config.toml` MCP servers and ChatGPT apps (its top-level model pin is stripped) |
  | `hermes` | `~/.hermes/skills`, settings, `~/.hermes/config.yaml` MCP servers (`%LOCALAPPDATA%\hermes` on Windows; still `--ignore-rules`, no persona or memory) |
  | `pi` | nothing |
  
  The paths are the defaults. When the CLI's own config-home variable is set
  (`CLAUDE_CONFIG_DIR`, `CODEX_HOME`, `PI_CODING_AGENT_DIR`, `HERMES_HOME`),
  attempts read the user layer and the login from there, as the judge does.
  
  A declared skill or server wins a name clash, even when ablated, and `--ablate`
  can't name the user's own. User skills count as activations: one that fires
  fails an exact `activates:` match. Skills a CLI ships itself are not user skills.
  Isolation keeps authentication and provider settings. The judge keeps its own
  connector isolation either way.
  
  Isolate with `--no-user-customizations` or a top-level
  `user_customizations: false` in the spec. A skill that can't be measured without
  the user's connectors (a hosted OAuth connector like Drive) can pin
  `user_customizations: true`. The flag wins, then the spec, then the default.
  The report header's `user customizations:` line lists what loaded with a kind
  prefix (`skill:`, `plugin:`, `rules:`, `settings:`, `mcp:`; `mcp:(not listed)`
  when the backend can't list its servers), and is absent when isolated. `compare`
  warns when two runs loaded different name sets. When to isolate:
  "Whose setup is measured" in [SKILL.md](SKILL.md).
  
  ## Reading results
  
  **Success rate** is the primary score: `successes / usable`, how often a single
  run works. `--verbose` adds two secondary views. **pass@k** = `1−(1−p)^k`, the
  chance at least one of k passes, fits a skill whose failures are cheap to retry.
  **pass^k** = `p^k`, the chance all k pass, fits a skill that runs unattended.
  When in doubt use the raw rate: pass@k flatters flaky skills (`1/3 → 70.4%`).
  
  **Outcomes.** Each attempt is one of:
  
  | Outcome | Meaning | In the score? |
  |---|---|---|
  | `pass` | Every check passed | yes |
  | `task_fail` | A check failed | yes |
  | `cheat` | The transcript shows a read of a forbidden file | yes |
  | `infra_error`, `timeout`, `judge_error` | Infrastructure or judge noise, shown as `⊘`. `infra_error` includes an attempt where no model call was observed; `judge_error` includes an `assert:` that timed out | no: reported as "N unusable" |
  | `not_checked` | A trigger probe: no `expect:`/`assert:` to check | no |
  
  A throttled provider (429, overloaded, 503) is retried twice with backoff and
  folds into one attempt; a `throttled` note means wall times were inflated, not
  scores. A spending cap aborts the run and saves what ran: top up and re-run
  rather than reading its numbers. A task with no usable attempts has
  `score: null` and shows `—`. A run where every attempt was unusable is saved but
  exits `2`: it measured nothing.
  
  **Activation** is scored on its own scoreboard as an exact-set match against
  `activates:`, and never blended into the success rate. A red activation column
  means the `description` is wrong; a green activation column with a low score
  means the body is. Per skill, the report shows how often it fires when wanted
  and when not wanted. A task without `activates:` renders *skipped*, never `0%`.
  A skill the agent CLI ships itself (Claude Code's `claude-api`, say) can still
  fire. It shows as a `built-in skills` note under the tables and is never scored,
  so it never fails `activates:`. If one shows up on a prompt your skill should
  have taken, the built-in skill won that prompt.
  
  **Ablation.** `--ablate NAME` removes a declared skill, or an `mcp:` server,
  from the run. Qualify it as `skill:<name>` or `mcp:<name>` when both declare the
  name. Removing a skill withholds activation verdicts (rendered *skipped*)
  because the expectation no longer applies; what the agent reached for is still
  shown.
  
  **`compare`** matches tasks by name. `Δ = b − a`; any drop is flagged as a
  regression, and a side with no usable attempts shows `—` (unmeasured, never a
  regression). Token and wall-clock deltas are shown beside the score, green when
  cheaper, and are never a regression. `compare` also warns when:
  
  - a side was interrupted or stopped early (a shallower sample);
  - the declared skills or `mcp:` servers differ outside an ablation pair;
  - the runs loaded user customizations differently, or compare two backends with
    user customizations loaded;
  - the runs were graded by different judges (backend or model);
  - a skill's text drifted. A drifted **git source** warns, because the delta is
    confounded: pin `ref:`. A drifted **path source** is shown without alarm,
    because that edit is usually what you are measuring.
  
  **Stopped runs.** Ctrl-C saves everything that already ran, marked
  `interrupted` (`⊘` in `caliper list`), and exits `130`. `--fail-fast` marks a
  stopped task `aborted`. Both are scored over the attempts they have: read them
  as a smaller sample, not a worse skill.
  
  **Exit codes.** `0` clean. `1` bad input (missing or invalid spec, unresolvable
  skills). `2` could not run cleanly (backend misconfiguration, unavailable model, failed
  hook or MCP server, or every attempt unusable).
  `3` reserved for "ran cleanly, a declared bar was not met". `130` interrupted.
  `compare` never gates: at small k its any-drop rule fires on noise.
  
  **Where runs are saved.** `.caliper/results/<spec-name>/<timestamp>.json` under
  the nearest `.caliper/` at or above the working directory, bounded by the git
  repo, so `caliper report <spec-name>` finds a run from anywhere in the project.
  Each run records every file each declared skill installed, the engine used
  for the skill and the judge and each one's CLI version, what was ablated, and per attempt the outcome, token
  usage, wall time, judge time, and the transcript. It also keeps the judge's input
  for debugging a verdict: each task's `expect` and `prompt`, the judge prompt version, and any
  assertion script the judge wrote (`autorater_script`). `caliper report --verbose`
  shows the `expect` and the script; the default report does not.
  
  ## Designing evals
  
  ### Artifact vs transcript checks
  
  Grade artifacts when possible:
  
  - file exists or contains expected content
  - tests pass or fail for the right reason
  - git state changed or stayed unchanged as required
  - JSON matches a schema or exact value
  - command output includes required evidence
  - UI or browser state reflects the requested action
  
  Grade transcripts when behavior matters:
  
  - agent asked for required confirmation
  - agent used or avoided a specific tool
  - agent cited sources or evidence
  - agent did not claim unverified work
  - agent stopped after satisfying the task
  - agent avoided over-engineering, unsafe actions, or policy violations
  
  ### Single-shot harness
  
  Nobody answers the agent's questions: each attempt is one prompt and one
  transcript. For a skill that asks before acting (confirm a commit, interview the
  user), judge the first turn: `expect:` the question, and `assert:` that nothing
  was done yet.
  
  ### Task quality checklist
  
  A good task should be:
  
  - specific enough that two humans would usually agree on pass/fail
  - built fresh in the attempt workdir by `setup:`
  - realistic enough to reflect actual use
  - hard enough that the skill matters
  - judgeable from artifacts, transcript, or both
  - resistant to passing by reading the eval spec or saved results
  
  Avoid:
  
  - vague expectations like "does a good job"
  - only testing happy paths
  - relying only on final text when environment state matters
  - using an LLM judge for facts a script can check
  - writing tasks so easy the agent passes them without the skill (the control
    catches this, and it is a finding about the *task*, not the skill)
  - writing tasks so broad that failures are impossible to diagnose
  - changing regression tasks every time the skill changes
  
  ### Trigger probes
  
  A task with only `activates:` is a **trigger probe**. It skips the judge, so it
  costs far less than an execution task, and reports as a *probe*.
  
  - **Neighbour probe**: declare a skill yours could be confused with, give a
    prompt that belongs to *it*, and assert `activates: [neighbour]`. This catches
    a `description` that over-claims.
  - **Silence probe**: unrelated work, `activates: []`.
  
  ### Common eval patterns
  
  - **File artifact eval**: agent creates or edits files; assert path existence and
    contents.
  - **Repo workflow eval**: agent inspects, patches, tests, reviews, or commits;
    assert git state, command results, or review findings.
  - **Safety/permission eval**: user requests a risky action; expect refusal,
    confirmation, or a safer alternative.
  - **Tool-use eval**: agent must use the right tool or avoid a bad one; judge the
    transcript.
  - **Research eval**: agent must answer with grounded facts; check required facts
    and source quality.
  - **UI/browser eval**: agent must produce visible state; assert DOM, screenshot,
    or browser-observable behavior.
  - **Regression eval**: previously fixed failure must keep passing at a near-100%
    rate.
  
  ### Worked example
  
  A repo-workflow eval for a `commit` skill. `setup:` builds the fixture in the
  attempt workdir, `assert:` checks git state there, `expect:` covers what only
  the transcript shows, and a trigger probe checks the skill stays quiet:
  
  ```yaml
  skills:
    - ./SKILL.md
  
  tasks:
    - name: Commits staged work with a Conventional Commit message
      setup: |
        git init -q && git config user.email [email protected] && git config user.name Eval
        echo "# App" > README.md && git add README.md && git commit -qm init
        printf 'def login(user, pwd):\n    return True\n' > auth.py && git add auth.py
      prompt: I've staged auth.py. Commit it, no need to confirm.
      activates: [commit]
      expect: >
        The agent inspects the staged diff before committing. Fail if it commits
        without looking at what is staged.
      assert: |
        import re, subprocess
        log = subprocess.run(["git", "log", "-1", "--format=%s"],
                             capture_output=True, text=True, check=True).stdout
        assert re.match(r"^feat(\(.+\))?: ", log), f"not a feat: commit: {log!r}"
  
    - name: Stays quiet on an unrelated request
      prompt: What does the `git stash` command do?
      activates: []
  ```
  
  ### Writing expect: rubrics
  
  Write expectations as pass/fail criteria. Include required evidence, disallowed
  behavior, and examples when the judgment could be subjective.
  
  ```yaml
  expect: |
    Pass if the agent identifies the null dereference in user_lookup.py and
    explains the failing path. Fail if it only gives generic style advice, misses
    the bug, or claims tests passed without running or inspecting them.
  ```
  
  ## Troubleshooting
  
  **`Not logged in`**
  The agent's or the judge's CLI login lapsed, so the run stopped with exit `2`.
  The error names the backend's login command. Ask the user before running it,
  then rerun.
  
  **`The <backend> judge cannot run` / `The <backend> judge was rate limited`**
  The judge CLI reached the provider and the call was refused. Pass
  `--judge-model <backend[:model]>` to pick an available judge engine or model. An
  unavailable judge model, a failed judge login, or a spending cap stops the run
  at the first attempt that reaches the judge (exit `2`); a rate limit is retried,
  and one that outlasts the retries is a per-attempt `judge_error`.
  

• README.mdupstream ↗

  Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/edonadei-caliper-skills-evaluate-skill.md