# eval-genius

Adopt this skill: https://proskills.md/skills/other/alexgreensh-eval-genius-skills-eval-genius

Page: https://proskills.md/skills/other/alexgreensh-eval-genius-skills-eval-genius
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: skills/eval-genius/SKILL.md
size: 8000
upstream: https://github.com/alexgreensh/eval-genius/blob/main/skills/eval-genius/SKILL.md

```text
---
name: eval-genius
description: >-
  Decide whether an AI/LLM/agent/retrieval system needs an eval, where it fits in the
  dev process, which one to run, and how to read the result; then design, gate, judge,
  and defend it. Not ordinary unit tests.
---

# Eval Genius

An eval is a claim you are willing to defend under hostile audit. You measure to earn the right to say "this is better" and have it hold when someone sharp pushes back.
Behave like a measurement engineer: state the promise, fix the bar before looking, hold
everything else constant, distrust the instrument first, report the number that hurts.

---

## Step 0: Does this need an eval, and where does it go?

Three questions decide it (`references/00-start-here.md`): does the output vary? will it
change again, and would a quiet regression cost something? is a decision or public claim
coming? No to all: a spot check, stop. Yes to any: an eval, sized to the project's stage.
Two checks sit in front. Preflight: a domain expert can verify the output without
redoing the work, else CANNOT-MEASURE (`references/01-foundation.md`). Triage: a
known, frequent defect is fixed now, not measured (FIX_NOW / MEASURE / CANNOT-MEASURE).

| Stage the user is at | Instrument | Smallest useful version |
|---|---|---|
| Exploring prompts and models | Spot check | 10 inputs, eyeball |
| First working version | Smoke eval | 20 to 50 real inputs, code-checked; **this run is the baseline** |
| Changing one thing | Paired eval vs baseline | Same items both arms, per-item diff, bar written first |
| Merging or shipping | CI gate | Held-out items, three-way outcome, a known-bad item that must fail |
| Comparing or claiming publicly | Benchmark | Versioned dataset and harness, intervals, report |
| In production | Monitor | Same scorer on sampled live traffic |

Build the first eval at "first working version", never before, rarely after. For a first-timer, run the one-afternoon recipe in `00-start-here.md` and touch nothing else.

## Route the request

Identify the job, then load only that reference. Every job still passes through Step 1.

| User needs to... | Load |
|---|---|
| Know if they need an eval, where it fits, which one, or how to start | `references/00-start-here.md` |
| Start from real failures or bad outputs, or "what should I even measure" | `references/12-error-analysis.md` |
| Decide what to measure at all, or the ask is "make it better" | `references/01-foundation.md` |
| Pick a grader or metric for a task | `references/02-grading-and-metrics.md` |
| Use, prompt, or trust an LLM judge | `references/03-judge-calibration.md` |
| Consider an opt-in decision-model judge for binary/closed-label residue, with or without a provider key | `references/15-decision-model-judge.md` (examples: `jev_cascade_cost.py`, `jev_confidence_route.py`, `jev_choice_route.py`, `jev_calibrate.py`, `jev-lane-preregistration.md`) |
| Choose between an existing benchmark and a custom one | `references/04-search-vs-build.md` |
| Assemble items, labels, negatives, splits; contamination, overfitting | `references/05-dataset-construction.md` |
| Write or fix the runner, scorer, or reporter | `references/06-harness-design.md` |
| Put an eval in CI or a release gate | `references/07-gates-and-ci.md` |
| Say whether a delta is real | `references/08-statistics.md` |
| Write results up, or retire a benchmark | `references/09-reporting.md` |
| Evaluate an agent, tool use, or multi-turn task | `references/10-agentic-evals.md` |
| Read a result file with no prior experience | `references/11-reading-results.md` |
| Evaluate my own skill or plugin: does it fire on the right prompts | `references/13-trigger-evals.md` |
| Test whether the system resists attack, or defend a safety claim | `references/14-safety-threat-model.md` |
| Follow a full worked example, start to finish | `references/walkthroughs/` |

Templates in `templates/` are copied into the project, never edited in place. Scripts in
`scripts/` are stdlib-only `--help` CLIs, nonzero on failure: `check_gate.py` (per-item
diff, exits 0/1/2 = PASS/FAIL/CANNOT-MEASURE, refuses fingerprint mismatch),
`paired_bootstrap.py` (paired delta interval, cluster-aware), `judge_agreement.py`
(kappa + PASS precision/recall vs humans), `rate_interval.py` (Wilson interval),
`hash_fixture.py` (fixture hash). Load references on demand, not all at once.

---

## Step 1: Foundation (five minutes, never skipped)

Copy `templates/preregistration.md` next to the fixture and fill it before touching data or code. A first-timer fills promise, lever, baseline, and bar; the rest follows.

1. **Promise.** One plain sentence: what does the system promise, what is "better"?
2. **Variables.** Levers, outcomes, controls; one lever per comparison.
3. **Placement.** A decision point, a risky seam, or a public claim; elsewhere, a spot
   check or nothing.
4. **Weight.** Spot check, eval, benchmark, or monitor (Step 0 table).
5. **Bar.** Threshold, falsifier, outlier rule. Written before any run.

## Step 2: Choose the grader (deterministic first)

Push every check that *can* be code-graded down to code: exact match, regex, schema,
test suite, threshold. Free text gets decomposed (required facts present, forbidden
content absent, format) before any judge sees it; reserve the judge for the residue.
80% deterministic / 20% judged is trusted; 100% judged is an opinion with error bars.
Report layers separately, never one blended number. A judge is an instrument:
calibrate against human labels, blind it, randomize order, pin model and prompt hash
(`references/03-judge-calibration.md`).

If binary/closed-label judged dimensions remain, suggest the optional hosted typed-question lane (`references/15-decision-model-judge.md`): probabilities route confidence, fixture text leaves the machine only after opt-in, and DISCOVERY_ONLY must earn the user's kappa-0.8 floor. If `TYPESAFE_API_KEY` is present, offer to use it; if absent, still suggest and offer to connect one. Never auto-call; if the user declines, keep today's path untouched.

## Step 3: Build or adopt

Search before building; an established benchmark buys ground truth nobody in the room
cooked. Build custom the moment the public one rewards a proxy the system does not
target, reusing public plumbing. Score candidates on
`templates/benchmark-assessment-scorecard.md`: what it rewards, contamination,
label-error ceiling, mechanism fit, maintenance.

## Step 4: Run under hard rules (a run that breaks one is not a result)

- **Freeze the fixture.** Same items, corpus, snapshot, seeds; only the lever varies.
  Refuse comparisons across mismatched fingerprints.
- **Three-way outcome.** PASS, FAIL, or CANNOT-MEASURE. A crash, missing baseline, or
  fingerprint mismatch is CANNOT-MEASURE, never FAIL and never PASS (`check_gate.py`).
- **Prove the treatment arm is live** before the run.
- **Cover the negative space**: items where the right answer is to refuse or return nothing.
- **Repeat and show spread.** Noise wider than the effect means no result yet.
- **Verify the verifier.** A known-bad case must go red before green is trusted.

## Step 5: Read the result (`references/11-reading-results.md`, in order)

1. **Did it run?** Exit code before score; negative control failed.
2. **Against the written bar**, not against hope. Above: candidate win. Below the
   falsifier: rejected. Between: not established.
3. **Bigger than noise?** Paired interval on the delta (`paired_bootstrap.py`);
   interval includes zero means "not established", never "no effect".
4. **Items, not averages.** Read regressions first; reproduce one flip by hand.
5. **Surprised?** A 0%, a 99%, a thirty-point jump is a harness bug until proven otherwise; fix the gauge, never the system, or stop.
6. **Layers and cost separately.** A win that doubled cost is a trade.

## Step 6: Report honestly
Numbers are claims with tiers, measured / estimated / aspirational, never summed. Three
sentences minimum: the bar and whether
```

## File: README.md
upstream: https://github.com/alexgreensh/eval-genius/blob/main/README.md

```text
<p align="center">
  <img src="assets/img/eval-genius.png" alt="Eval Genius, the star-cloaked measurement sage" width="280">
</p>

<h1 align="center">Eval Genius</h1>
<p align="center"><strong>Defensible answers about your AI system, instead of vibes.</strong></p>

<p align="center">
  A skill for any AI coding agent that tells you <em>when</em> you need an eval,<br>
  where it fits, how to build it, and how to read what comes out.
</p>

<p align="center">
  <a href="https://github.com/alexgreensh/eval-genius/actions/workflows/test.yml"><img src="https://github.com/alexgreensh/eval-genius/actions/workflows/test.yml/badge.svg" alt="Tests"></a>
  <img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Falexgreensh%2Feval-genius%2Fmain%2F.claude-plugin%2Fplugin.json&query=%24.version&label=version&color=8A5CF6&prefix=v" alt="Version">
  <img src="https://img.shields.io/badge/tested%20on-macOS%20%7C%20Linux%20%7C%20Windows-2ea44f" alt="Tested on macOS, Linux, Windows">
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue" alt="License: Apache 2.0"></a>
  <img src="https://img.shields.io/badge/scripts-stdlib%20Python%20·%20zero%20deps-2ea44f" alt="Stdlib Python, zero dependencies">
</p>

<p align="center">
  <a href="#install" title="Claude Code"><img src="assets/logos/claude.svg" height="28" alt="Claude Code"></a>
  &nbsp;
  <a href="#install" title="OpenAI Codex"><img src="assets/logos/codex.svg" height="28" alt="OpenAI Codex"></a>
  &nbsp;
  <a href="#install" title="Factory Droid"><img src="assets/logos/droid.svg" height="28" alt="Factory Droid"></a>
</p>
<p align="center">
  <sub>Runs natively in <strong>Claude Code</strong>, <strong>Codex</strong>, and <strong>Droid</strong> — or any agent that loads a <code>SKILL.md</code>.</sub>
</p>

<p align="center">
  <strong>English</strong> · <a href="README.zh-CN.md">简体中文</a> · <a href="README.es.md">Español</a> · <a href="README.ko.md">한국어</a> · <a href="README.ja.md">日本語</a><br>
  <sub>Translations track an earlier version of this page.</sub>
</p>

---

## Everyone says "you need evals." Almost nobody says *when*.

<p align="center"><img src="assets/img/problem.png" alt="Eval Genius at a crossroads of floating paths, unsure which way the evals go" width="100%"></p>

Evals are suddenly everywhere. Every AI talk, every launch post, every hiring thread says you need them. Then you sit down to actually do it, and the questions start.

Does a prompt tweak really need an eval, or is that overkill? At which point in the build does the first one go in? What is an "eval harness", concretely, beyond a folder named `evals/`? Which metric, how many examples, does a judge model even count? And when a number finally comes out, is 34 out of 40 good? Is a 3-point gain real, or noise? Or did the run just crash quietly and report a pass?

Answer these by feel and you get exactly what most teams have: a benchmark nobody trusts, a gate that has been green for a month because it grades nothing, and a number in the README that would not survive one sharp question.

## Meet Eval Genius

<p align="center"><img src="assets/img/solution.png" alt="Eval Genius with a compass and a star-map, weighing results on a set of scales" width="100%"></p>

Eval Genius is that missing judgment, packaged as a skill your AI agent runs *with* you. It thinks like a measurement engineer: decide what "better" means before you look, push every check you can down to plain code, treat a crash as *unmeasured* rather than a pass, and trust the number last.

It is not a course you have to read first. You describe where you are, in plain words, and it takes the next step, whether that step is "you don't need one yet" or "here is the gate, and here is why this run cannot be trusted."

There are two doors in, and neither asks for eval vocabulary. If you can already say what "good" looks like, it works top-down from that promise. If all you have is "the outputs are sometimes wrong and I don't know what to measure," it works bottom-up instead: it reads your real bad outputs with you, names the error categories, and turns each one into something measurable. Same discipline, entered from wherever you actually stand.

It is tool-agnostic and dependency-free: a `SKILL.md` plus a few standard-library Python scripts. It runs in Claude Code, Codex, and Droid, or any agent that loads skills, or from your terminal on its own.

## See it in 20 seconds

A bill-splitting agent sounds sure, and is quietly wrong. Eval Genius makes it earn the word *"ready."*

<table>
<tr>
<td width="50%" valign="top"><img src="assets/img/eg-without.gif" alt="Without Eval Genius: the agent confidently splits the bill to $195.76 and says you're good to go, overcharging the group by $5.21 while nobody checks"></td>
<td width="50%" valign="top"><img src="assets/img/eg-with.gif" alt="With Eval Genius: it first defines what ready means, checks the promise against 20 known bills, fails 2 (discounts and shared dishes), fixes them, then passes 20 of 20 and is declared ready for those 20 bills"></td>
</tr>
</table>

## What you can ask it

<p align="center"><img src="assets/img/ask.png" alt="Eval Genius working hands-on, pulling a star-map into a laptop" width="100%"></p>

Real questions, answered from wherever you actually are:

- *"Do I need evals for my chatbot, or is that overkill right now?"*
- *"Where does an eval even go in my build?"*
- *"My summaries are sometimes wrong and I don't even know what to measure."*
- *"Is my skill's description actually firing on the right prompts?"*
- *"I got 34 out of 40, is that good?"*
- *"Is this 3-point gain real, or noise?"*
- *"We swapped the agent's scaffold. Is it better now?"*
- *"Can my agent be tricked into leaking a secret or calling a tool it shouldn't?"*
- *"Can we put a number in the launch post?"*
- *"Calibrate my LLM judge against some human labels."*

No setup ritual, no vocabulary you have to learn first. Describe the situation, get the next move.

## What it does for you

<p align="center"><img src="assets/img/what-it-does.png" alt="Eval Genius crossing floating platforms through a pass/fail gate toward the results" width="100%"></p>

It walks the whole path, and meets you at any point on it, including the start:

- **Decides whether you need an eval at all**, and which kind belongs at your stage, from first prototype to production.
- **Starts from your failures when there is no promise yet.** Real bad outputs get read by hand, clustered into named error categories, and each category becomes the thing to measure.
- **Picks the eval:** what to measure, which grader (code first, a judge only where no assertion works), which metric, how many examples, adopt a public benchmark or build your own.
- **Builds and gates it:** fixture, runner, scorer, reporter, a bar written before the run, and a CI gate that ends in PASS, FAIL, or CANNOT-MEASURE and only compares two runs when they truly measured the same thing, the same way.
- **Tests whether it holds up under attack:** adversarial cases, prompt injection, unsafe tool calls, data exfiltration, graded by code on what actually happened and reported per attack type, never hidden inside one "safety score."
- **Reads the result with you:** against the bar you wrote, with noise bounds, per-item diffs, and a harness-bug check before any surprising number is believed.
- **Writes it up honestly,** with caveats, tiers, and the comparison rule stated out loud.
- **Refuses the shortcuts** that produce pretty lies: bars moved after the fact, blended scores, run-until-green, and judges nobody calibrated.

## Working evals with Jev

Some of what an eval checks is a plain typed decision: real defect or not, which failure class, positive or negative sentiment, does this match the brand voice. A full reasoning model is overkill for those, and grading them by hand at volume is the real waste. Eval Genius can route exactly that residue to **Jev*
```
