# skillevaluator

Adopt this skill: https://proskills.md/skills/other/nvidia-skillevaluator-skills-skillevaluator

Page: https://proskills.md/skills/other/nvidia-skillevaluator-skills-skillevaluator
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: skills/skillevaluator/SKILL.md
size: 11581
sha: 5d6c5412b3fdbe3e3d9651481da5066ee72cfd8a
upstream: https://github.com/nvidia/skillevaluator/blob/main/skills/skillevaluator/SKILL.md

````text
---
name: skillevaluator
description: Run and interpret SkillEvaluator for agent skills. Use for any skill benchmark or eval - validating or scoring a SKILL.md, writing evals.json test cases or evals config, explaining benchmark reports, verdicts and exit codes, or gating skills in CI.
license: Apache-2.0
compatibility: Requires Python 3.12-3.14 and the skillevaluator CLI; live evaluation also needs Docker (or another backend) and an LLM provider key.
metadata:
  author: Christopher Kevin <christopherk@nvidia.com>
  version: "1.0.0"
  domain: [evaluation, ai-ml]
  tags: [skills, evaluation, benchmarking, ci, security]
---

# SkillEvaluator

Run the SkillEvaluator CLI on agent skills and report what the evidence shows:
Tier 1 validation (static, security, quality), Tier 2 deduplication, and Tier 3
live agent evaluation that compares runs with and without the skill.

## Purpose

Give an agent one reliable procedure for evaluating skills with
SkillEvaluator: pick the right tier, check readiness, run it, and report
scores, findings and exit codes truthfully.

## When To Use

Use this skill when the user wants to:

- check whether a skill is well-formed, safe, and good enough to publish
- get a quality score, rubric score, security or PII scan for a skill
- find repeated guidance inside a skill or overlap across a skill collection
- measure whether a skill actually helps an agent (Skill Lift, pass@k)
- write or fix `<skill-dir>/evals/evals.json`, `<skill-dir>/evals/config.yml`, custom graders, or Harbor tasks
- read a SkillEvaluator report, verdict, or exit code
- add SkillEvaluator checks to GitHub Actions or GitLab CI

Do not use it to write the target skill's domain content, to review ordinary
application code, or to evaluate an LLM without a skill. For converting an
existing benchmark into a custom grader, read
[references/custom-graders.md](references/custom-graders.md).

## Prerequisites

- A target directory containing `SKILL.md` (or a folder of skills, `.mdc`
  rules, or a plugin manifest).
- The CLI: `pip install --upgrade "skillevaluator[all]"`, or from a checkout
  `uv run --python 3.13 --extra all skillevaluator ...`. Check with
  `skillevaluator --version`. Extras: `llm` (rubric), `security` (scanners),
  `tier2`, `tier3`, `all`, `dashboard`.
- No key for keyless commands (`quality-check`, `security-scan`, `pii-scan`,
  `lint-scripts`, `tier1 PATH --no-llm`, `validate PATH --tiers 1`).
- A provider for LLM stages, Tier 2 and Tier 3. Simplest:
  `export SKILL_EVAL_LLM_PROVIDER=nv_build` and `export NVIDIA_API_KEY=...`.
  Other providers: [references/providers.md](references/providers.md).
- For Tier 3: an agent (`opencode`, `codex`, or `claude-code`) and a sandbox.
  Harbor is installed with the `tier3` extra; you do not run it yourself.
  The default `--env-mode docker` needs a Docker daemon. Without Docker, use
  `--env-mode local` (experimental): it runs the agent CLI already installed
  on the host under Bubblewrap (Linux) or Seatbelt (macOS); trusted skills only.

Never print, echo, or commit API keys; keep them in the host environment.

## Choose The Command

| User goal | Command | Needs |
| --- | --- | --- |
| Fast keyless quality score | `skillevaluator quality-check <skill-dir>` | nothing |
| Full static gate for one skill | `skillevaluator tier1 <skill-dir> --no-llm` | scanners (auto-installed) |
| Static gate plus LLM rubric and security review | `skillevaluator tier1 <skill-dir>` | provider |
| LLM rubric only | `skillevaluator rubric-eval <skill-dir>` | provider, `llm` extra |
| Security / PII scan only | `skillevaluator security-scan <skill-dir>`, `skillevaluator pii-scan <skill-dir>` | nothing |
| Repeated content inside one skill | `skillevaluator tier2 <skill-dir>` | provider with embeddings |
| Overlap across a collection | `skillevaluator similarity-check <skills-dir>` | embeddings |
| Does the skill help an agent? | `skillevaluator tier3 <skill-dir>` | provider, agent, Docker |
| Same, without Docker | `skillevaluator tier3 evaluate <skill-dir> -a opencode --env-mode local` | provider, agent CLI on host |
| Everything, with one exit code | `skillevaluator validate <skill-dir>` | all of the above |
| CI-safe keyless gate | `skillevaluator validate <skills-dir> --profile default --tiers 1 -r json,markdown,sarif -o reports` | scanners |
| Set up repo CI | `skillevaluator ci init` | GitHub checkout |

Full flag tables: [Tier 1](references/commands-tier1.md),
[validate](references/commands-validate.md),
[Tier 2, Tier 3 and setup](references/commands-tier2-tier3.md).

## Workflow

1. **Locate the target.** Confirm the path has `SKILL.md` and that its
   frontmatter `name` matches the directory name. For a folder of skills, the
   commands evaluate each skill inside it.
2. **Check readiness before paid runs.** Run `skillevaluator health-check --tier 1`
   for scanners, `skillevaluator doctor --tier 2` for providers, and
   `skillevaluator doctor --env-mode docker` (add `--verify-models`) for Tier 3.
   Fix every failing row first.
3. **Start keyless.** Run `skillevaluator tier1 <skill-dir> --no-llm`. Fix
   CRITICAL and HIGH findings before anything else; `lint` findings are advisory.
4. **Add LLM checks** when a provider is set: `skillevaluator tier1 <skill-dir>`
   (rubric, LLM security, verification) and `skillevaluator tier2 <skill-dir>`.
5. **Prepare Tier 3 cases.** If `<skill-dir>/evals/evals.json` is missing, first
   capture task requests and observable outcomes in `<skill-dir>/evals/EVAL.md`;
   headings alone do not qualify. Generate with
   `skillevaluator create-eval-dataset <skill-dir> --full --prompt <skill-dir>/evals/EVAL.md`,
   then review it. The default prefers author guidance, then matching retained
   traces; existing datasets need `--force` or `--refine`. Skill-only
   generation requires explicit opt-in with `--grounding skill`. Review:
   positive cases (explicit, implicit, contextual) and at least one negative
   case with `"expected_skill": null`. Required shape: top-level `skill_name`
   and an `evals` array; each case has `id`, `prompt`, `expected_output`, and
   `expected_skill` (optional `assertions`, `files`). Validate with
   `skillevaluator tier3 validate <skill-dir> --strict`. Format:
   [references/tier3-datasets.md](references/tier3-datasets.md); run policy and
   scoring: [references/tier3-running.md](references/tier3-running.md).
6. **Run live evaluation.** `skillevaluator tier3 <skill-dir>` (or
   `skillevaluator tier3 evaluate <skill-dir> -a opencode --env-mode docker --n-attempts 3`).
   Use `--max-cases 1` for a cheap smoke run first; such runs are never
   publication evidence.
7. **Read the results** (see below), open `skillevaluator view <skill-dir>`, and
   for follow-up questions run `skillevaluator tier3 feedback <run-dir>`.
8. **Iterate.** Change the skill, rerun the same command with the same cases,
   agent and model, and compare. Treat a single run's lift as directional.

## Read The Results

Exit codes: `0` success, `1` a gate failed, evidence is `INCOMPLETE`, or the
run could not complete, `2` usage error (bad flag or missing path; fix the
command, not the skill).

- **Quality** (0-100): Correctness 35%, Discoverability 25%, Reliability 25%,
  Efficiency 15%. Grades A>=90, B>=80, C>=70. Passing needs `--min-score`
  (default 70) and no HIGH/CRITICAL quality finding.
- **Rubric** (0-100): nine criteria scored 0-10; passing needs the weighted
  score >= `--min-score` and every criterion >= 7.
- **Tier 2**: HIGH or CRITICAL duplicate findings exit `1` in `tier2 PATH`.
- **Tier 3**: five dimensions (Security, Correctness, Discoverability,
  Effectiveness, Efficiency), each PASS >= 0.50, NEUTRAL 0.40-0.50, FAIL < 0.40.
  Verdict PASS needs every dimension >= 0.50 for at least one agent. Skill
  Lift = with-skill overall - baseline overall; PASS >= +0.05. `INCONCLUSIVE:
  skill never activated` means no positive trial loaded the skill; fix the
  description or prompts, not the scores. `tier3 PATH` exits `0` on a `FAIL`
  verdict; only an incomplete run exits `1`.
- **Missing scores are `NO SCORE` or `None`, never 0.** Do not average them in.
- **Publication**: require `publication_status == "pass"` in the JSON report,
  not just `overall_passed`.

Report anatomy and JSON fields: [references/reports-and-ci.md](references/reports-and-ci.md).

## Examples

```bash
# Keyless first look
skillevaluator quality-check <skill-dir> -r cli -r json -o <report-dir>

# Static gate for a whole collection, complete report even on failures
skillevaluator validate <skills-dir> --profile default --tiers 1 -c -r json,markdown -o reports

# LLM-backed Tier 1 and Tier 2, dedup advisory
skillevaluator validate <skill-dir> --tiers 1,2 --llm --no-block-on-dedup

# Live A/B evaluation with two agents, retaining trajectories for debugging
skillevaluator tier3 evaluate <skill-dir> -a opencode,codex --n-attempts 3 --harbor-keep-jobs

# Baseline only, to calibrate cases before the skill exists
skillevaluator tier3 evaluate <skill-dir> --skip-with-skill
```

## Limitations

- Tier 1 and the quality score are static: they check structure and wording,
  not whether the skill works. Only Tier 3 observes agent behavior.
- LLM judges (rubric, Tier 3 correctness/effectiveness) vary between runs and
  models; compare runs only with the same model, cases, and attempts.
- Tier 3 costs model tokens and sandbox time per case x attempt x agent x 2
  arms; a generated starter case gives limited coverage.
- `validate --tiers` always includes Tier 1; `--tiers 2` or `--tiers 3` alone is
  rejected. Use `tier2 PATH` or `tier3 PATH` to run a tier on its own.
- Agent credentials come only from the host environment; `<skill-dir>/evals/config.yml`
  `runtime_env` rejects provider keys.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| Exit `2` | Unknown flag, bad value, or missing path | Check `skillevaluator <command> --help`; prefix paths named like commands with `./` |
| Check shows `INCOMPLETE` | A scanner is missing or failed | Run `skillevaluator health-check --tier 1`; install the `security` extra or the missing scanner |
| "provider is required" / exit `1` before reports | No `SKILL_EVAL_LLM_PROVIDER` or key, or several keys exported | Set exactly one provider and its key; see providers reference |
| Quality below 70 | Missing `## Prerequisites`, `## Limitations`, `## Troubleshooting`, vague description, no examples | Read each finding's suggestion; do not stuff keywords |
| `INCONCLUSIVE: skill never activated` | Description does not match the prompts | Make the description name the tasks; add explicit-bucket cases |
| Docker trials fail to start | Daemon down or outdated engine | `docker info`; rerun `skillevaluator doctor --env-mode docker` |
| Agent model rejected | Default model not valid for that agent | `--agent-model AGENT=MODEL`, verify with `doctor --verify-models` |
| Run incomplete (`Scored attempt coverage is n/m`) | A trial errored or timed out | Rerun; raise `--timeout-multiplier`; inspect with `--harbor-keep-jobs` |

More cases: [references/troubleshooting.md](references/troubleshooting.md).

## Final Response

Report to the user:

- the exact commands run and their exit codes
- scores and verdicts per tier, quoting the report values (not estimates)
- the top findings to fix, each with its file and suggested change
- report paths (`<report-dir>/...`, `<skill-dir>/evals/results/<run-id>/report.html`)
- what was not run (skipped tiers, missing keys, `--max-cases` smoke runs) and
  therefore not proven
````

## File: skills/skillevaluator/references/commands-tier1.md
size: 2757
sha: 5ba21f55659ff6688c6102523bea2ceb02aa729b
upstream: https://github.com/nvidia/skillevaluator/blob/main/skills/skillevaluator/references/commands-tier1.md

```text
# Tier 1 command reference

Report options and Tier 1 commands. `skillevaluator <command> --help` is authoritative for the installed version.

Global options go before the command: `skillevaluator -v validate <skill-dir>`.

## Reports (Tier 1 and Tier 2 commands)

| Flag | Default | Effect |
| --- | --- | --- |
| `-r, --report` | `cli,json,html` for `tier1 PATH`, `tier2 PATH`, `validate`; `cli` for focused commands | `cli`, `json`, `html`, `markdown`, `sarif`; comma-separated or repeated |
| `-o, --output-dir` | `reports` | Report directory |

Passing `-r` is honored exactly: `-r cli` writes no files. Basenames:
`validate` writes `skillevaluator-output-<timestamp>.*`; `tier1`/`tier2` write
`skillevaluator-tier1.*`/`skillevaluator-tier2.*`; focused commands write
`skillevaluator-<kind>.*` (for example `skillevaluator-quality.json`) and
overwrite earlier runs.

## Direct tier workflows

A bare `tier1`, `tier2`, or `tier3` prints help. Prefix a path with `./` when
its name matches a subcommand (`tier1 <path>/validate`).

`skillevaluator tier1 PATH`: all static checks including `dependency`, plus
rubric, LLM security and finding verification when a provider is configured.

| Flag | Default | Effect |
| --- | --- | --- |
| `--checks TEXT` | all applicable | Subset: `schema`, `version`, `security`, `pii`, `license`, `code-integrity`, `unicode`, `quality`, `lint`, `dependency` |
| `--skip CHECKS` | none | Leave checks out (also `secrets` for Gitleaks); the run cannot certify Tier 1 |
| `--llm / --no-llm` | on when a provider is configured | Explicit `--llm` requires a valid provider |
| `--min-score INTEGER` | `70` | Quality and rubric minimum |
| `--quality-config PATH` | packaged | Replace expected-heading rules |
| `--pii-waivers PATH` | none | Reviewed, expiring PII waivers |
| `--profile TEXT` | `default` | Validation policy |

`skillevaluator tier2 PATH`: intra-skill duplicate analysis of one skill.
`--catalog FILE` adds inter-skill comparison (build it with
`skillevaluator similarity-check <skills-dir> --save-catalog <catalog.json>`);
`--threshold` (0.80), `--similarity-threshold` (0.75), `--full-body`.

`skillevaluator tier3 PATH`: generates one starter case when no source
exists, then runs the provider-native agent in Docker with and without the
skill. Takes every `tier3 evaluate` option. Does not accept `-r`/`-o`.

## Focused Tier 1 commands

| Command | Key flags |
| --- | --- |
| `quality-check PATH` | `--min-score` (70), `--quality-config` |
| `rubric-eval PATH` | `--min-score` (70); every criterion must reach 7/10 |
| `security-scan PATH` | `--llm`, `--llm-verify`, `--skillspector-baseline` |
| `pii-scan PATH` | `--llm-verify`, `--pii-waivers` |
| `lint-scripts PATH` | report options only; advisory |
```

## File: skills/skillevaluator/references/commands-tier2-tier3.md
size: 2530
sha: afd216c063694d3184103d4404a68437cf90425f
upstream: https://github.com/nvidia/skillevaluator/blob/main/skills/skillevaluator/references/commands-tier2-tier3.md

```text
# Tier 2, Tier 3 and setup command reference

`skillevaluator <command> --help` is authoritative.

## Tier 2 commands

| Command | Key flags |
| --- | --- |
| `context-optimization-check PATH` (alias `dedup-scan`) | `--threshold` (0.8), `--model`, `--llm-model` |
| `similarity-check PATH` | `--threshold` (0.75), `--full-body`, `--catalog`, `--save-catalog`, `--max-entries` |

## Tier 3 commands

| Command | Purpose |
| --- | --- |
| `tier3 evaluate PATH` | Live A/B run; add `--autopilot` to generate a missing case |
| `tier3 validate PATH` | Check `<skill-dir>/evals/` (`--strict`, `--json`, `--harbor-contract`) |
| `create-eval-dataset PATH` | Write `<skill-dir>/evals/evals.json` (`--grounding auto` requires substantive author guidance or usable matching retained traces; existing datasets need `--force` or `--refine`; opt in to `--grounding skill` for skill-only generation; `--full`, `--no-llm`, `--dry-run`, `--force`, `--prompt`, `--refine`) |
| `init-custom-grader PATH` | BYOG starter (`--mode`, `--language`) |
| `init-harbor-task PATH` | BYOT starter (`--case-id`, `--with-config`) |
| `tier3 select-decoys PATH` | Save neighbor skills for group workspaces |
| `view PATH`, `compare PATH` | Open latest report; compare agents |
| `tier3 feedback RUN_DIR` | Chat about one completed run (`--skill`) |
| `harbor-view JOBS_DIR` | Browse jobs kept with `--harbor-keep-jobs` |
| `dashboard [PATHS]` | Tables over saved reports (`dashboard` extra) |

Key `tier3 evaluate` flags: `-a, --agents` (`opencode`, `codex`,
`claude-code`), `--env-mode` (`docker` default, `local`, `kata`, `gke`,
`daytona`, `e2b`, `modal`, ...), `--n-attempts`, `--pass-threshold`,
`--stop-on-pass`, `--n-concurrent`, `--max-agents`, `--max-cases`, `--model`,
`--agent-model AGENT=MODEL`, `--skip-baseline`, `--skip-with-skill`,
`--grading-mode default|default_plus_custom|custom_only`, `--results-dir`,
`--include-skills`, `--skill-workspace-mode isolated|group`, `--copy-repo`,
`--timeout-multiplier`, `--harbor-keep-jobs`, `--agent-runtime-preflight`,
`--override-cpus`, `--override-memory-mb`, `--progress`, `--verbose`.

## Readiness and setup

| Command | Purpose |
| --- | --- |
| `health-check [--tier 1\|2\|3\|all]` | Offline readiness; `--checks` for Tier 1 scanners |
| `doctor [--tier ...] [--env-mode M] [-a AGENTS] [--verify-models]` | Deeper readiness; `--verify-models` makes live catalog probes |
| `models [--limit N] [--json]` | List provider catalog models |
| `ci init`, `ci status`, `ci demo` | Guided GitHub CI setup |
```

## File: skills/skillevaluator/references/commands-validate.md
size: 1564
sha: cc1dd57690ef7b01abdcee1a5665a1b715606df9
upstream: https://github.com/nvidia/skillevaluator/blob/main/skills/skillevaluator/references/commands-validate.md

```text
# validate command reference

The combined pipeline. `skillevaluator validate --help` is authoritative.

## validate

`skillevaluator validate PATH` runs Tier 1, Tier 2 and Tier 3 (with
autopilot) for skills. Tier 1 always gates; Tier 2 gates unless
`--no-block-on-dedup`; Tier 3 gates only with `--block-on-agent-eval`. A
folder without a root `SKILL.md` validates each skill inside it.

| Flag | Effect |
| --- | --- |
| `--tiers 1`, `--tiers 1,2`, `--tiers 1,3` | Scope; Tier 1 always included |
| `--type skill\|rules\|workflows\|plugin\|auto` | Force the content type |
| `--checks`, `--skip` | Tier 1 selection (`dependency` is opt-in here; `--checks` replaces the list) |
| `--llm`, `--llm-verify` | LLM security analysis and false-positive suppression |
| `--min-score`, `--quality-config`, `--pii-waivers`, `--skillspector-baseline FILE` | Tier 1 tuning |
| `--previous-version VERSION` | Require a greater `metadata.version` (single skill only) |
| `--fail-fast`, `-c, --continue-on-failure` | Stop early, or record every issue (recommended in CI) |
| `--no-dedup`, `--no-block-on-dedup` | Skip Tier 2, or make it advisory |
| `--no-tier3`, `--no-autopilot`, `--block-on-agent-eval` | Tier 3 scope and gating |
| `-a`, `--env-mode`, `--n-attempts`, `--max-cases`, `--agent-model`, `--harbor-keep-jobs`, ... | Forwarded to Tier 3 |
| `--workers N` | Validate N skills of a folder in parallel |
| `--profile`, `--policy FILE` | Policy profile and overlay |
| `--evaluated-source-repository`, `--evaluated-source-revision` | Provenance for `BENCHMARK.md` |
```

## File: skills/skillevaluator/references/custom-graders.md
size: 906
sha: cc4d8f1d873f2074621abf27c05d4c0d3afd4b8f
upstream: https://github.com/nvidia/skillevaluator/blob/main/skills/skillevaluator/references/custom-graders.md

```text
# Custom graders and Harbor tasks

Bring your own grader (BYOG) or Harbor task (BYOT).

- `default_plus_custom`: standard dimensions stay authoritative and your
  `custom_metrics` are added. `custom_only`: your grader's `overall` decides.
- Scaffold with `skillevaluator init-custom-grader <skill-dir> --language python`
  or `skillevaluator init-harbor-task <skill-dir> --case-id case-001 --with-config`,
  replace the placeholder logic, then run
  `skillevaluator tier3 validate <skill-dir> --harbor-contract`.
- A grader reads `/logs/agent/trajectory.json`, `/tests/entry.json`, and
  `/workspace/input/`, and writes `/logs/verifier/reward.json` plus
  `/logs/verifier/reward.txt` with values from 0.0 to 1.0.
- Do not name custom metrics `security`, `skill_execution`,
  `skill_efficiency`, `accuracy`, `goal_accuracy`, `behavior_check`,
  `overall`, `details`, `metrics`, `metric_set`, or `entry_id`.
```

## File: skills/skillevaluator/references/providers.md
size: 1624
sha: cd071bdd2b3f808ffbbd7514ee83c71a210fc1d4
upstream: https://github.com/nvidia/skillevaluator/blob/main/skills/skillevaluator/references/providers.md

```text
# Providers and credentials

Set exactly one provider. If `SKILL_EVAL_LLM_PROVIDER` is unset and exactly
one of `NVIDIA_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` is exported,
it is auto-selected; with more than one, LLM commands fail until the
provider is set.

| Provider | Variables | Notes |
| --- | --- | --- |
| `nv_build` | `NVIDIA_API_KEY` | One key covers Tier 1 LLM, Tier 2 embeddings, and Tier 3 agents in `docker`, `kata`, `local`; default agent `opencode` |
| `openai` | `OPENAI_API_KEY` | Default agent `codex` |
| `anthropic` | `ANTHROPIC_API_KEY` | Default agent `claude-code`; Tier 2 needs a separate embedding provider |
| `bedrock` | AWS credential chain, `AWS_REGION` | Tier 2 needs a separate embedding provider |
| `openai-compatible` | `SKILL_EVAL_LLM_BASE_URL`, `SKILL_EVAL_LLM_API_KEY` | Any gateway; override models whose catalog IDs differ |

Overrides: `SKILL_EVAL_LLM_MODEL` (evaluator and judge),
`LLM_JUDGE_MODEL` (judge only), `SKILL_EVAL_EMBEDDING_MODEL`,
`--agent-model AGENT=MODEL` (Tier 3 agent). OpenCode on a gateway takes
`--agent-model opencode=openai/MODEL`. Codex needs a Responses-API model,
OpenCode Chat Completions, Claude Code Anthropic Messages, all with tool
calls.

Rules:

- Credentials come only from the host environment (or masked CI secrets).
  `harbor.runtime_env` in `<skill-dir>/evals/config.yml` rejects provider keys and base
  URLs with a hard error.
- `skillevaluator models` and `doctor --verify-models` check catalog
  reachability; a public catalog does not prove the inference key works.
- Judge failures make a result unscored (`NO SCORE`), not `0.0`.
```

## File: skills/skillevaluator/references/reports-and-ci.md
size: 1814
sha: d46dfe05b72b382948f5ff0358081831b8482552
upstream: https://github.com/nvidia/skillevaluator/blob/main/skills/skillevaluator/references/reports-and-ci.md

````text
# Reports and CI

## JSON fields to read

From `skillevaluator-output-<timestamp>.json` (or `skillevaluator-tier1.json`):

| Field | Meaning |
| --- | --- |
| `overall_passed`, `overall_status` | Command gate: `passed`, `failed`, `incomplete` |
| `publication_status` | `pass`, `neutral`, `fail`, `incomplete`; use for publishing |
| `quality_summary[].overall_score`, `.grade` | Quality 0-100 and A-F |
| `severity_counts` | `critical`, `high`, `medium`, `low` totals |
| `finding_groups[]` | One entry per unique finding (`finding_id` `SEF-...`) |
| `tier3.run_id` | Run identity for the Tier 3 section |

`overall_passed` can be true while an advisory tier did not run; require
`publication_status == "pass"` before claiming a skill is publishable.

Example gate on a stricter score:

```bash
report=$(ls -t <report-dir>/skillevaluator-output-*.json | head -1)
jq -e '([.quality_summary[]?.overall_score] | min // 100) >= 80' "$report"
```

`validate` on a skill also writes `BENCHMARK.md`: verdict first, the three
tier states, and `Baseline -> Skill Uplift` columns.

## CI

Guided GitHub setup from a checkout: `skillevaluator ci init` (then
`ci status`). Manual: copy the GitHub or GitLab template from the
repository's `ci/bundles/<version>/` folder.

Recommended keyless gate:

```bash
skillevaluator validate <skills-dir> --profile default --tiers 1 -c \
  -r json,markdown,sarif -o reports --min-score 70
```

Adopt progressively: run with `|| true` (advisory), then gate on Tier 1,
raise `--min-score`, add `--llm --tiers 1,2` with a masked provider secret,
and finally an advisory Tier 3 job (`--block-on-agent-eval` when ready).
Keep keys in CI secrets exposed through the job environment, never in the
repository. Pin policy with `--policy <policy.yaml>` so severity changes are
reviewed like code.
````

## File: skills/skillevaluator/references/tier3-datasets.md
size: 4302
sha: 03c2e882591c57892cd143335ea5cd76d30a5bec
upstream: https://github.com/nvidia/skillevaluator/blob/main/skills/skillevaluator/references/tier3-datasets.md

````text
# Tier 3 evaluation datasets

Tier 3 runs a real agent on each case with the skill installed and without it (baseline). This file covers the `<skill-dir>/evals/` folder and dataset format.

## The <skill-dir>/evals/ directory

| Path | Purpose |
| --- | --- |
| `<skill-dir>/evals/evals.json` | Required dataset (`.jsonl`, `.yaml`, `.yml` also accepted) |
| `<skill-dir>/evals/config.yml` | Run policy |
| `<skill-dir>/evals/EVAL.md` | Guidance for `create-eval-dataset` (`## Questions`, `## Behaviors`, `## Negative Cases`, `## Notes`) |
| `<skill-dir>/evals/files/` | Fixtures staged into `/workspace/input/` |
| `<skill-dir>/evals/environment/` | `Dockerfile`, `docker-compose.yaml` sidecars, `mcp_servers.toml` |
| `<skill-dir>/evals/harbor/` | Native Harbor tasks (BYOT) |
| `<skill-dir>/evals/grader.py` or `grader.sh` | Custom grader (BYOG) |
| `<skill-dir>/evals/results/` | Output only; never edit or commit |

## Dataset format

```json
{
  "skill_name": "my-skill",
  "evals": [
    {
      "id": "my-skill-001",
      "prompt": "Convert report.docx in /workspace/input to PDF",
      "expected_output": "A PDF of the document saved next to the input",
      "assertions": ["Saves report.pdf next to report.docx", "Keeps the document's headings in the PDF"],
      "expected_skill": "my-skill",
      "files": ["files/case-001"]
    },
    {
      "id": "my-skill-002-negative",
      "prompt": "What is the capital of France?",
      "expected_output": "Answers directly without loading my-skill",
      "expected_skill": null
    }
  ]
}
```

Required: `skill_name`, `evals`, and per case `id` (letters, digits, `.`,
`_`, `-`; at most 128 chars, unique), `prompt`, `expected_output`. Optional:
`assertions`, `expected_skill` (`null` for negatives), `expected_script`,
`acceptable_skills`, `files` (paths relative to `<skill-dir>/evals/`; omit it and the
whole `<skill-dir>/evals/files/` is staged). The legacy flat array with `question`,
`ground_truth`, `expected_behavior` still runs with a deprecation warning.

Good datasets cover four buckets: explicit (names the skill), implicit
(describes the task), contextual (task inside a project), negative (should
not trigger). Write `expected_output` as an observable outcome and one
behavior per assertion. The baseline is graded on the same prompt,
`expected_output` and assertions without the skill, so keep them about the task
outcome: never point the prompt at the skill's files or assert that the agent
read `SKILL.md`, ran the skill's scripts or used the skill (`skill_execution`
scores that). `tier3 validate` warns about common wordings of these; it is a
heuristic, so review the cases too.

## Guided generation

Standalone `create-eval-dataset` defaults to `--grounding auto`: prefer author
guidance in `--prompt` / `<skill-dir>/evals/EVAL.md` with real content beyond
headings, then usable matching retained trajectories from this skill's completed
runs. Write task requests and observable outcomes in the guidance file first:

```bash
skillevaluator create-eval-dataset <skill-dir> --full --prompt <skill-dir>/evals/EVAL.md
```

If neither guidance nor matching retained traces exists, generation fails before
provider access or writes. `--refine` and `--no-llm` do not bypass that preflight.
For the prior skill-only starter workflow, explicitly opt in:

```bash
skillevaluator create-eval-dataset <skill-dir> --full --no-llm --grounding skill
```

Auto leaves an existing dataset unchanged unless you pass `--force` or
`--refine`, even when matching traces exist. To refine from completed results:

```bash
skillevaluator create-eval-dataset <skill-dir> --refine --from-results <run-dir>
```

Auto trace refinement preserves existing authored case IDs and expectations;
unmatched cases and observed answer proposals need review. When auto selects
traces, proposals that describe skill use leave `expected_output` empty for an
independent authored outcome. Existing authored expectations are preserved. Explicit
`--grounding trace` still opts into in-place refinement without `--refine`.
Generation provenance records the effective `intent` or `trace` grounding and input hashes. Explicit
`intent`, `trace`, and `skill` modes remain available. Unattended autopilot retains
its existing skill-grounded one-case starter contract.
````

## File: skills/skillevaluator/references/tier3-running.md
size: 2405
sha: 3c6801b642577984b83351d41585ecfa7ec8fd90
upstream: https://github.com/nvidia/skillevaluator/blob/main/skills/skillevaluator/references/tier3-running.md

````text
# Running and scoring Tier 3

Run policy, run sequence, scoring and cost control.

## config.yml

```yaml
schema_version: 1
harbor:
  n_attempts: 3
  pass_threshold: 0.5
  n_concurrent: 2
  timeout_multiplier: 1.5
  agents:
    opencode:
      model: your-model-id
skill_workspace:
  mode: isolated   # or group, with include: [...]
grading:
  mode: default    # default | default_plus_custom | custom_only
```

Unknown keys fail the load. CLI flags override config values.
`pre_agent_setup` runs only with `--skip-baseline`. Never put credentials in
`runtime_env`.

## Run sequence

Write task requests and observable outcomes in `<skill-dir>/evals/EVAL.md`
before generation; a file containing only headings does not qualify. For an
existing dataset, use `--refine` with matching retained results to refine it,
or `--force` to replace it after reviewing a `--dry-run` preview.

```bash
skillevaluator doctor --env-mode docker --verify-models
skillevaluator create-eval-dataset <skill-dir> --full --prompt <skill-dir>/evals/EVAL.md
skillevaluator tier3 validate <skill-dir> --strict
skillevaluator tier3 evaluate <skill-dir> -a opencode --max-cases 1   # smoke
skillevaluator tier3 evaluate <skill-dir> -a opencode --n-attempts 3
skillevaluator view <skill-dir>
```

Results land in `<skill-dir>/evals/results/<timestamp>_<pid>_<id>/` with `result.json`,
`run_config.json`, `report.html` and per-agent `lift.json`,
`with-skill/`, `without-skill/`. Use `--results-dir` to write elsewhere.

## Scoring

Standard grading scores five dimensions on 0.0-1.0 and averages them:

| Dimension | Source |
| --- | --- |
| Security | Deterministic trace checks (destructive commands, uploads) |
| Correctness | LLM judge of the answer (`accuracy`) |
| Discoverability | Whether the skill was loaded when expected (`skill_execution`) |
| Effectiveness | LLM judges `goal_accuracy` and `behavior_check` |
| Efficiency | Wasted tool or skill calls (`skill_efficiency`) |

Bands: PASS >= 0.50, NEUTRAL 0.40-0.50, FAIL < 0.40. Verdict PASS needs one
agent with every dimension passing. Lift band: PASS >= +0.05, FAIL <= -0.10.
pass@k counts a case passed when any attempt reaches `--pass-threshold`.

## Cost control

Trials = cases x attempts x agents x 2 arms. Use `--max-cases`,
`--skip-baseline` (no lift), `--stop-on-pass`, and `--n-concurrent` to
control spend. Runs that skip cases are never publication evidence.
````

## File: skills/skillevaluator/references/troubleshooting.md
size: 2577
sha: 829959a34cf853b60be0411a8ff2eed73567fea7
upstream: https://github.com/nvidia/skillevaluator/blob/main/skills/skillevaluator/references/troubleshooting.md

```text
# Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| `Error: No such command` or exit `2` | Wrong command or flag for this version | `skillevaluator --version`, then `skillevaluator <command> --help` |
| `tier1 validate` evaluated the wrong thing | Directory named like a subcommand | Use `<path>/validate` |
| `--tiers 2` rejected | `validate` always includes Tier 1 | Use `skillevaluator tier2 PATH` |
| Scanner `INCOMPLETE` | SkillSpector, Gitleaks or Bandit missing or failed | `skillevaluator health-check --tier 1`; install the `security` extra; an old scanner earlier on `PATH` wins over the pinned one |
| `--skip` used and publication fails | Skipped checks cannot certify Tier 1 | Rerun without `--skip` |
| Schema: directory does not match name | Folder name differs from frontmatter `name` | Rename one so both match (kebab-case) |
| Quality below minimum | Missing expected headings, vague description, no examples, `SKILL.md` over 500 lines or 5,000 tokens | Follow each finding's suggestion; move detail into `references/` |
| Rubric fails although the score is high | One criterion below 7/10 | Read that criterion's feedback in the JSON report |
| Tier 2 needs embeddings | Anthropic and Bedrock have no embeddings | Set `SKILL_EVAL_EMBEDDING_*` for another provider |
| `<skill-dir>/evals/config.yml` rejected | Unknown key or credential in `runtime_env` | Fix the key; move secrets to the host environment |
| `Missing required fields: question` | Legacy entry without `question`, or new entry without `prompt` | Use the agentskills.io shape |
| Skill never activated | Description and prompts do not match | Name the tasks in the description; add explicit cases |
| Agent auth fails | Agent credential not exported where `skillevaluator` runs | Export it on the host; `doctor -a AGENT` |
| `AgentTimeoutError` or rate-limit errors | Slow model or too much concurrency | `--timeout-multiplier 2`, lower `--n-concurrent` |
| Custom Dockerfile build fails | Base image lacks tools the agent or grader needs, or `python3 -I` cannot import the verifier packages (`ENV PIP_USER=1`, `PIP_TARGET` or `PYTHONPATH` installs) | Fix `<skill-dir>/evals/environment/Dockerfile`; install verifier packages into `python3`'s site-packages; rerun with `--harbor-keep-jobs` to read the build log |
| Results need debugging | Harbor jobs deleted by default | Rerun with `--harbor-keep-jobs`, then `skillevaluator harbor-view JOBS_DIR` |
| Lift differs between runs | LLM judges and agents are stochastic | Increase `--n-attempts`; compare only identical setups |
```

## File: skills/skillevaluator/evals/config.yml
size: 108
sha: 4e5c19a6d60046e3bef00bb684812ef5f7153779
upstream: https://github.com/nvidia/skillevaluator/blob/main/skills/skillevaluator/evals/config.yml

```text
schema_version: 1
harbor:
  n_attempts: 2
  pass_threshold: 0.5
  n_concurrent: 4
  timeout_multiplier: 1.5
```

## File: skills/skillevaluator/evals/evals.json
size: 6708
sha: 2b41b7a6facde8a7043de8cebbb2758f4cb5ebe7
upstream: https://github.com/nvidia/skillevaluator/blob/main/skills/skillevaluator/evals/evals.json

```text
{
  "skill_name": "skillevaluator",
  "evals": [
    {
      "id": "skillevaluator-001-explicit-keyless",
      "prompt": "Use SkillEvaluator: I have a skill at ./skills/pdf-tools and no LLM API key. Give me the exact commands to check that it is well-formed, safe, and see its quality score, and tell me what exit code 1 versus 2 would mean.",
      "expected_output": "Recommends keyless commands such as `skillevaluator tier1 ./skills/pdf-tools --no-llm` and/or `skillevaluator quality-check ./skills/pdf-tools` (or `validate --tiers 1`), and explains exit 1 = a gate failed or evidence incomplete, exit 2 = usage error such as a bad flag or missing path.",
      "assertions": [
        "Recommends a keyless Tier 1 command (tier1 --no-llm, quality-check, or validate --tiers 1) on ./skills/pdf-tools",
        "Explains exit code 1 as a failed check or incomplete evidence and exit code 2 as a usage error",
        "Does not ask for or invent an API key"
      ],
      "expected_skill": "skillevaluator"
    },
    {
      "id": "skillevaluator-002-explicit-tier3",
      "prompt": "With SkillEvaluator, how do I measure whether my skill at ./my-skill actually makes an agent better? Write the full command sequence from readiness check to report, including a cheap smoke run first, and say which number tells me the improvement.",
      "expected_output": "A sequence using doctor (with --env-mode docker), creating/validating evals (create-eval-dataset, tier3 validate), a smoke `tier3 evaluate ... --max-cases 1`, a full `tier3 evaluate` or `tier3 ./my-skill` run with and without the skill, and `view`; identifies Skill Lift (with-skill minus baseline) as the improvement measure.",
      "assertions": [
        "Includes a readiness check with skillevaluator doctor",
        "Includes dataset generation or validation (create-eval-dataset or tier3 validate)",
        "Includes a smoke run limited with --max-cases",
        "Runs tier3 live evaluation comparing with-skill and baseline",
        "Names Skill Lift as with-skill minus baseline"
      ],
      "expected_skill": "skillevaluator"
    },
    {
      "id": "skillevaluator-003-implicit-dataset",
      "prompt": "I need test cases so an agent benchmark can tell whether my 'csv-cleaner' agent skill gets used correctly. Write the evals/evals.json file for it with three cases: one that names the skill, one that only describes the task, and one prompt where the skill should NOT be used.",
      "expected_output": "A valid evals/evals.json with top-level skill_name 'csv-cleaner' and an evals array of three cases each having id, prompt, expected_output (optionally assertions), where the negative case sets expected_skill to null.",
      "assertions": [
        "Output has top-level skill_name and evals keys",
        "Every case has id, prompt, and expected_output",
        "Includes one explicit and one implicit positive case",
        "The negative case sets expected_skill to null"
      ],
      "expected_skill": "skillevaluator"
    },
    {
      "id": "skillevaluator-004-implicit-report",
      "prompt": "My skill benchmark report says 'INCONCLUSIVE: skill never activated' and the Tier 3 command exited 0. Another run says verdict FAIL but also exited 0. What do these mean and what should I change?",
      "expected_output": "Explains the INCONCLUSIVE verdict as missing positive-trial activation, so the description or prompts should be fixed rather than the scores; explains that tier3 exits 0 on a FAIL verdict and only an incomplete run exits 1, with --block-on-agent-eval in validate to gate on Tier 3.",
      "assertions": [
        "Explains never activated means the skill was not loaded in positive trials",
        "Recommends improving the skill description or eval prompts",
        "Explains that a FAIL verdict alone exits 0 and only an incomplete run exits 1",
        "Mentions --block-on-agent-eval or another way to make Tier 3 gate"
      ],
      "expected_skill": "skillevaluator"
    },
    {
      "id": "skillevaluator-005-contextual-ci",
      "prompt": "Our repo keeps agent skills under ./skills and we use GitHub Actions. Add a workflow step that fails the PR when any skill fails static validation, uploads SARIF, and does not need any API key. Show the command the step should run.",
      "expected_output": "A GitHub Actions step running `skillevaluator validate ./skills --tiers 1` with SARIF/JSON reports to an output dir (e.g. `-r json,markdown,sarif -o reports`), relying on the exit code to fail the job, with no secrets required; optionally mentions `skillevaluator ci init`.",
      "assertions": [
        "Uses skillevaluator validate on ./skills with --tiers 1",
        "Requests a sarif report and an output directory",
        "Relies on the command exit code to fail the job",
        "Does not require an API key for this gate"
      ],
      "expected_skill": "skillevaluator"
    },
    {
      "id": "skillevaluator-006-contextual-config",
      "prompt": "I'm setting up live agent evaluation for ./skills/sql-helper. I want 3 attempts per case, the opencode agent, and the agent needs my OPENAI_API_KEY. Write evals/config.yml and tell me where the API key should go.",
      "expected_output": "An evals/config.yml with schema_version: 1 and harbor.n_attempts: 3 (and opencode agent settings), explicitly NOT placing the API key in config.yml or harbor.runtime_env; the key is exported in the host environment where skillevaluator runs.",
      "assertions": [
        "config.yml sets schema_version 1 and harbor n_attempts 3",
        "Does not put OPENAI_API_KEY in config.yml or runtime_env",
        "Says to export the key in the host environment or CI secrets"
      ],
      "expected_skill": "skillevaluator"
    },
    {
      "id": "skillevaluator-007-negative-write-skill",
      "prompt": "Write a Python function that removes duplicate rows from a list of dictionaries, keeping the first occurrence.",
      "expected_output": "A direct Python implementation; no SkillEvaluator commands or skill evaluation workflow.",
      "assertions": [
        "Provides a working Python function",
        "Does not run or recommend SkillEvaluator"
      ],
      "expected_skill": null
    },
    {
      "id": "skillevaluator-008-negative-general",
      "prompt": "Explain the difference between precision and recall in classification models in two sentences.",
      "expected_output": "A short correct explanation of precision and recall without invoking skill evaluation.",
      "assertions": [
        "Correctly defines precision and recall",
        "Does not run or recommend SkillEvaluator"
      ],
      "expected_skill": null
    }
  ]
}
```

## File: README.md
upstream: https://github.com/nvidia/skillevaluator/blob/main/README.md

````text
# SkillEvaluator

![SkillEvaluator wordmark](https://raw.githubusercontent.com/NVIDIA/SkillEvaluator/ad2815fe78c045db91d230681d32863698da08cf/docs/assets/skillevaluator-wordmark.svg)

[![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](LICENSE)
[![Python](https://img.shields.io/badge/Python-3.12%20%7C%203.13%20%7C%203.14-blue.svg)](https://www.python.org/)
[![PyPI](https://img.shields.io/pypi/v/skillevaluator.svg?label=PyPI)](https://pypi.org/project/skillevaluator/)
[![Documentation](https://img.shields.io/badge/Documentation-docs.nvidia.com-blue.svg)](https://docs.nvidia.com/skills/skillevaluator/)
[![Paper](https://img.shields.io/badge/arXiv-2608.20614-b31b1b.svg)](https://arxiv.org/abs/2608.20614)
[![NVIDIA Developer Blog](https://img.shields.io/badge/Blog-NVIDIA%20Developer-76B900.svg)](https://developer.nvidia.com/blog/evaluating-ai-agent-skill-performance-with-nvidia-skillevaluator/)

> 📺 **Livestream:** [Ask the Experts: Evaluating Agent Skills](https://www.youtube.com/watch?v=4rI0zATFxYI)

SkillEvaluator checks AI agent skills and measures their effect on task results:
start with quality checks, then compare runs with and without a skill.

Agent skills are folders of instructions and supporting files defined by the
[Agent Skills specification](https://agentskills.io/). SkillEvaluator is part of
the [NVIDIA Verified Skills pipeline](https://github.com/NVIDIA/skills).

## Get started

Three ways in, one CLI, same results.

<table>
<tr>
<th width="33%">1. Run the CLI directly</th>
<th width="33%">2. Add it to your repo via CI</th>
<th width="33%">3. Wire it to your agent</th>
</tr>
<tr valign="top">
<td>

Install (or `uv tool install`):

```bash
pip install skillevaluator
```

Run all Tier 1 checks, no API key:

```bash
skillevaluator validate \
  ./my-skill --tiers 1
```

Or run from a clone: [Quickstart](#quickstart).

</td>
<td>

Check changed skills on every pull request. On GitHub, run:

```bash
skillevaluator ci init
```

It opens a setup PR. On GitLab, copy the [CI template](https://github.com/NVIDIA/SkillEvaluator/tree/main/ci/bundles/0.9.0).

</td>
<td>

Add the [SkillEvaluator skill](skills/skillevaluator/) to Claude Code, Codex or any skill-aware agent:

```bash
npx skills add \
  NVIDIA/SkillEvaluator
```

Then ask: *"Is my skill any good?"* No Node.js? [Copy the folder](docs/agent-skill.mdx#install).

</td>
</tr>
</table>

Guides: [installation](https://docs.nvidia.com/skills/skillevaluator/installation),
[CI setup](docs/cli-onboarding.mdx) ([GitHub](docs/github-actions.mdx),
[GitLab](docs/gitlab-ci.mdx), [report examples](ci/examples/README.md)) and
[agent skill](docs/agent-skill.mdx).

## Quickstart

Both paths use bundled `text-analyzer`.

| Start here | Requires | First result |
| --- | --- | --- |
| [Run without an LLM](#run-without-an-llm) | Git and uv | A deterministic quality score and findings |
| [Use NVIDIA Build](#use-nvidia-build) | Git, uv, and a Build API key | Model-backed rubric scores and feedback |

Install [uv](https://docs.astral.sh/uv/getting-started/installation/) and Git,
then get the example:

```bash
git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
```

uv provisions Python 3.13 and dependencies. For standalone installation, see
[Installation](https://docs.nvidia.com/skills/skillevaluator/installation).

### Run without an LLM

```bash
uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality
```

No API key, scanner, GPU or Docker is needed. The example scores **86.2/100 (B)**,
exits `0` and saves `reports/first-quality/skillevaluator-quality.json`. The
passing threshold defaults to 70.

This checks static quality; publication evidence stays `INCOMPLETE`. Security
needs scanners; agent impact needs live evaluation. Replace the example path
with your skill directory. See the
[keyless walkthrough](docs/quickstart.mdx#run-without-an-llm).

### Use NVIDIA Build

Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for
**free prototyping**, subject to [access terms and rate limits](https://docs.api.nvidia.com/nim/docs/product).

1. Sign in at [build.nvidia.com](https://build.nvidia.com/) and complete enrollment.
2. Open a model page, select **Get API Key**, and generate your key.
3. Set the provider and key in the same terminal, then run the rubric check:

```bash
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
uv run --python 3.13 --extra llm skillevaluator rubric-eval \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-build
```

The command sends skill documentation to the model and saves feedback in
`reports/first-build/skillevaluator-rubric.json`. No agent runtime or Docker is
needed. Completed evaluations can still fail rubric thresholds.

The same key supports Tier 2 embeddings and supported Tier 3 agents. See the
[Build walkthrough](docs/quickstart.mdx#use-nvidia-build) or [provider configuration](docs/configuration.mdx).

## Three-tier overview

![SkillEvaluator three-tier pipeline: Skill → Tier 1 Validation → Tier 2 Deduplication → Tier 3 Live Evaluation → Reports](https://raw.githubusercontent.com/NVIDIA/SkillEvaluator/ad2815fe78c045db91d230681d32863698da08cf/docs/assets/three-tier-overview.svg)

Each tier is independent. Add its extra to your command:

| Tier | Question | Example after setup | Additional setup |
| --- | --- | --- | --- |
| Tier 1: Validation | Is the skill safe, well-formed and [harness-compatible](docs/tier1-validation.mdx#the-checks)? | `uv run skillevaluator tier1 ./my-skill --no-llm` | External scanners for full static coverage; a provider and model clients enable LLM checks |
| Tier 2: Deduplication | Does guidance repeat or overlap? | `uv run --extra tier2 skillevaluator tier2 ./my-skill` | Embeddings and chat; add `--catalog ./skill-catalog.json` for inter-skill comparison |
| Tier 3: Live Evaluation | Does the skill help the agent? | `uv run --extra tier3 skillevaluator tier3 ./my-skill` | Provider access, a supported agent, and a running sandbox (Docker by default) |

Tier 1 fetches pinned, verified SkillSpector and Gitleaks releases when they
are missing; Semgrep is optional. See [scanners](docs/installation.mdx#system-tools).
`--no-llm` does not remove scanner requirements; a scanner that cannot run is
`INCOMPLETE` and exits nonzero.
[SkillSpector](https://github.com/NVIDIA/SkillSpector) provides specialized skill
security scanning; [Harbor](https://github.com/harbor-framework/harbor), the
open-source agent evaluation framework, powers sandboxed agent runs.

Tier 2 reports when inter-skill comparison did not run. Tier 3 autopilot reuses
sources or creates one skill-grounded case with limited coverage. See
[tier guides](https://docs.nvidia.com/skills/skillevaluator/).

## Run the combined pipeline

After setting up the provider, scanners, and agent sandbox, run all three tiers:

```bash
uv run --python 3.13 --extra all skillevaluator doctor --env-mode docker
uv run --python 3.13 --extra all skillevaluator validate ./my-skill
```

`doctor` checks readiness, not inference success. `validate` runs all tiers with
autopilot for skills. Use `--tiers 1,2` to skip live evaluation or `--tiers 1` for
static checks. `--no-autopilot` requires an existing source. Other artifact types
retain their applicable stages.

Tier 1 gates `validate`. Tier 2 gates unless `--no-block-on-dedup` is set;
Tier 3 gates only with `--block-on-agent-eval`. Model and managed-sandbox usage
can incur charges; see [runtime and cost](docs/tier3-live-evaluation.mdx#plan-for-cost).

Standalone generation needs substantive `evals/EVAL.md` guidance, a `--prompt`
file with task requests and outcomes, or matching retained traces. `--force`
below replaces the autopilot starter; `--refine` refines it. Explicit
`--grounding skill` permits skill-only starters. Review cases; temp
````
