# basic-skill

Adopt this skill: https://proskills.md/skills/other/darkrishabh-agent-skills-eval-examples-basic-skill

Page: https://proskills.md/skills/other/darkrishabh-agent-skills-eval-examples-basic-skill
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: examples/basic-skill/SKILL.md
size: 436
sha: 57ab7a6f81e3845db5cc24bec3b1f0c2e39c8754
upstream: https://github.com/darkrishabh/agent-skills-eval/blob/main/examples/basic-skill/SKILL.md

```text
---
name: basic-skill
description: Summarize small CSV files and identify the highest revenue month.
license: MIT
compatibility: Works with text-capable chat models.
metadata:
  owner: example
  domain: analytics
---

When given a CSV file with monthly revenue, identify the month with the highest revenue and explain the result concisely.

Prefer a short answer with the winning month, the revenue value, and a one sentence rationale.
```

## File: examples/basic-skill/evals/evals.json
size: 515
sha: bbe53cbafb6b8fba5a7ac01cdd0141844649ff48
upstream: https://github.com/darkrishabh/agent-skills-eval/blob/main/examples/basic-skill/evals/evals.json

```text
{
  "skill_name": "basic-skill",
  "evals": [
    {
      "id": "top-revenue-month",
      "name": "top revenue month",
      "prompt": "Use the attached CSV to identify the month with the highest revenue.",
      "files": ["evals/files/revenue.csv"],
      "expected_output": "The response identifies February as the highest revenue month with revenue of 18.",
      "assertions": [
        "The output names February as the highest revenue month.",
        "The output includes the value 18."
      ]
    }
  ]
}
```

## File: examples/basic-skill/evals/files/revenue.csv
size: 46
sha: fc19d82f08351dd7097f0d8cbecb525aebff2c49
upstream: https://github.com/darkrishabh/agent-skills-eval/blob/main/examples/basic-skill/evals/files/revenue.csv

```text
month,revenue
January,12
February,18
March,15
```

## File: README.md
upstream: https://github.com/darkrishabh/agent-skills-eval/blob/main/README.md

````text
<div align="center">

<img src="https://github.com/user-attachments/assets/094b8e11-e19e-4c96-ae82-ba701cfcf7e3" alt="agent-skills-eval — a test runner for Agent Skills" width="100%" />

<br />

# agent-skills-eval

**A test runner for [Agent Skills](https://agentskills.io).**

Write a `SKILL.md`, drop in some evals, and find out — empirically — whether your skill actually makes the model better at the task.

[![npm version](https://img.shields.io/npm/v/agent-skills-eval.svg?style=flat-square&logo=npm&label=npm)](https://www.npmjs.com/package/agent-skills-eval)
[![CI](https://img.shields.io/github/actions/workflow/status/darkrishabh/agent-skills-eval/ci.yml?style=flat-square&logo=github&label=ci)](https://github.com/darkrishabh/agent-skills-eval/actions/workflows/ci.yml)
[![license: MIT](https://img.shields.io/badge/license-MIT-green?style=flat-square)](LICENSE)
[![node](https://img.shields.io/node/v/agent-skills-eval.svg?style=flat-square&logo=nodedotjs&logoColor=white)](package.json)
[![docs](https://img.shields.io/badge/docs-GitHub%20Pages-0f766e?style=flat-square)](https://darkrishabh.github.io/agent-skills-eval/)
[![TypeScript](https://img.shields.io/badge/TypeScript-3178C6?style=flat-square&logo=typescript&logoColor=white)](https://www.typescriptlang.org/)

[Documentation](https://darkrishabh.github.io/agent-skills-eval/) · [Quickstart](#quickstart) · [SDK](#sdk) · [agentskills.io](https://agentskills.io)

</div>

---

## Why this exists

[Agent Skills](https://agentskills.io) — the open standard from Anthropic for giving agents domain knowledge — make it easy to ship a `SKILL.md` and assume your agent is now better at the task. The hard part is *proving* it.

`agent-skills-eval` is the missing piece. It runs your skill against the same prompts twice — once `with_skill` loaded into context, once `without_skill` (baseline) — has a judge model grade both outputs, and gives you a side-by-side report. If the skill doesn't make a measurable difference, you'll see it. If it does, you have receipts.

It's the test framework for the Agent Skills ecosystem, separated from any specific agent runtime so it works wherever your skills do.

## Quickstart

```bash
npx agent-skills-eval ./skills \
  --target gpt-4o-mini \
  --judge gpt-4o-mini \
  --baseline \
  --strict
```

That's it. Point it at a folder of skills, give it a target model and a judge model, and it produces a workspace with full artifacts and a static HTML report.

```text
agent-skills-workspace/
└── iteration-1/
    ├── meta.json            # run metadata
    ├── benchmark.json       # rolled-up pass/fail per skill
    ├── eval-basic/
    │   ├── with_skill/      # output, timing, judge grading
    │   └── without_skill/   # ↑ same, with the skill stripped
    └── report/
        └── index.html       # the visual report
```

Open `iteration-1/report/index.html` and you have a real, evidence-backed answer to "is my skill working?"

## What you get

|  |  |
|---|---|
| **`with_skill` vs `without_skill`** | Every eval runs both ways so you can see the actual lift from the skill — or its absence. |
| **Judge-graded outputs** | Use any chat model as a judge. Pass/fail with cited assertions, not vibes. |
| **TypeScript SDK + CLI** | One-liner CLI for CI, full SDK for custom pipelines, custom providers, and dashboards. |
| **OpenAI-compatible by default** | Works out of the box with OpenAI, Together, Groq, Anthropic via OpenAI-compat layers, local Llama servers — anything that speaks the OpenAI chat API. |
| **Tool-call assertions** | Deterministic checks for agents that call tools, not just generate text. |
| **Portable artifacts** | JSON + JSONL all the way down. Run today, diff tomorrow. Plug into your own dashboard. |
| **Static HTML reports** | A drop-in report site you can publish anywhere — no infrastructure. |
| **Fully spec-compliant** | Implements the full [agentskills.io specification](https://agentskills.io/specification): `SKILL.md` validation, `evals/evals.json`, official `iteration-N` artifact layout, frontmatter rules. |

## Install

```bash
npm install agent-skills-eval
```

Or run directly without installing:

```bash
npx agent-skills-eval --help
```

## How it works

The mental model is straightforward. For every eval defined in your skill:

```
                ┌─────────────────────────────┐
                │       same prompt           │
                └───────────────┬─────────────┘
                                │
                ┌───────────────┴─────────────┐
                ▼                             ▼
        ┌──────────────┐              ┌──────────────┐
        │ with_skill   │              │without_skill │
        │ SKILL.md in  │              │ baseline,    │
        │ context      │              │ no skill     │
        └──────┬───────┘              └──────┬───────┘
               │                             │
               ▼                             ▼
          target model                  target model
               │                             │
               ▼                             ▼
            output                        output
               │                             │
               └──────────┬──────────────────┘
                          ▼
                   ┌─────────────┐
                   │  judge      │  scores both against
                   │  model      │  the same assertions
                   └──────┬──────┘
                          ▼
                  pass / fail per side
```

The judge sees the eval's `expected_output` and `assertions` and grades each side independently. The `--baseline` flag is what enables the comparison; without it you only get the `with_skill` run.

## YAML config

For anything beyond a quick command, drop a config file at the root of your project:

```yaml
# agent-skills-eval.yaml
root: ./skills
workspace: ./agent-skills-workspace
baseline: true
target: gpt-4o-mini
judge: gpt-4o-mini
baseUrl: https://api.openai.com/v1
apiKeyEnv: OPENAI_API_KEY
include:
  - "skills/**"
exclude:
  - "**/draft-*"
evalIds:
  - "basic"
concurrency: 4
layout: iteration
strict: true
report:
  enabled: true
  title: Agent Skills Report
logging:
  format: pretty   # pretty | jsonl | silent
  verbose: false
  color: auto
targetParams:
  temperature: 0
judgeParams:
  temperature: 0
```

```bash
OPENAI_API_KEY=... npx agent-skills-eval --config agent-skills-eval.yaml
```

CLI flags always override config values.

### Options reference

These are configuration-file options and CLI defaults. See `--help` for the
available CLI flags; `targetParams`, `judgeParams`, and `logging.snippetLength`
are config-only. Logging flags use `--log-format`, `--log-file`, `--verbose`,
and `--no-color`; report flags use `--report`, `--no-report`, `--report-title`,
and `--report-output`. Supplied CLI flags override matching config values.

| Option | Type | Default | What it does |
|---|---|---|---|
| `root` | string | `.` | Directory scanned recursively for `SKILL.md` files. Positional CLI arg. |
| `workspace` | string | `./agent-skills-workspace` | Output directory for run artifacts (per-eval outputs, `grading.json`, `benchmark.json`, `meta.json`). |
| `baseline` | boolean | `false` | When `true`, runs each eval both `with_skill` and `without_skill` so the report shows the lift the skill provides. When `false`, only `with_skill` runs. |
| `target` | string | `gpt-4o-mini` | Model under evaluation — the one the skill is meant to help. |
| `judge` | string | value of `target` | Model that grades the rubric assertions. Set it to a stronger model than `target` for more reliable grading. |
| `baseUrl` | string | `OPENAI_BASE_URL` env, else **required** | Base URL of the OpenAI-compatible API for both target and judge. |
| `apiKeyEnv` | string | `OPENAI_API_KEY` | Name of the environment variable holding the API key (the key itself is never written to config). |
| `include` | string[] | all discovered skills | Glob(s) matched against each skill's pa
````
