⏳ This skill is pending AI review.

Scores will appear once the review pipeline completes.

v1.0.0

skillevaluator

@nvidia⭐ 1.3k stars

Run and interpret SkillEvaluator for agent skills. Use for any skill benchmark or eval - validating or scoring a SKILL.md, writing evals.json test cases or evals config, explaining benchmark reports, verdicts and exit codes, or gating skills in CI.

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

—/10

// RATINGS

⭐GitHub Stars
⭐⭐⭐⭐⭐ 1.3k on GitHubGitHub ↗

Very popular

🟢ProSkills Score
—
📍

Not yet listed on ClawHub or SkillsMP

// README

SkillEvaluator

SkillEvaluator wordmark

License Python PyPI Documentation Paper NVIDIA Developer Blog

📺 Livestream: Ask the Experts: Evaluating Agent Skills

SkillEvaluator checks AI agent skills and measures their effect on task results: start with quality checks, then compare runs with and without a skill.

Agent skills are folders of instructions and supporting files defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline.

Get started

Three ways in, one CLI, same results.

Install (or uv tool install):

pip install skillevaluator

Run all Tier 1 checks, no API key:

skillevaluator validate \
  ./my-skill --tiers 1

Or run from a clone: Quickstart.

Check changed skills on every pull request. On GitHub, run:

skillevaluator ci init

It opens a setup PR. On GitLab, copy the CI template.

Add the SkillEvaluator skill to Claude Code, Codex or any skill-aware agent:

npx skills add \
  NVIDIA/SkillEvaluator

Then ask: "Is my skill any good?" No Node.js? Copy the folder.

Guides: installation, CI setup (GitHub, GitLab, report examples) and agent skill.

Quickstart

Both paths use bundled text-analyzer.

Start hereRequiresFirst result
Run without an LLMGit and uvA deterministic quality score and findings
Use NVIDIA BuildGit, uv, and a Build API keyModel-backed rubric scores and feedback

Install uv and Git, then get the example:

git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator

uv provisions Python 3.13 and dependencies. For standalone installation, see Installation.

Run without an LLM

uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality

No API key, scanner, GPU or Docker is needed. The example scores 86.2/100 (B), exits 0 and saves reports/first-quality/skillevaluator-quality.json. The passing threshold defaults to 70.

This checks static quality; publication evidence stays INCOMPLETE. Security needs scanners; agent impact needs live evaluation. Replace the example path with your skill directory. See the keyless walkthrough.

Use NVIDIA Build

Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for free prototyping, subject to access terms and rate limits.

  1. Sign in at build.nvidia.com and complete enrollment.
  2. Open a model page, select Get API Key, and generate your key.
  3. Set the provider and key in the same terminal, then run the rubric check:
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
uv run --python 3.13 --extra llm skillevaluator rubric-eval \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-build

The command sends skill documentation to the model and saves feedback in reports/first-build/skillevaluator-rubric.json. No agent runtime or Docker is needed. Completed evaluations can still fail rubric thresholds.

The same key supports Tier 2 embeddings and supported Tier 3 agents. See the Build walkthrough or provider configuration.

Three-tier overview

SkillEvaluator three-tier pipeline: Skill → Tier 1 Validation → Tier 2 Deduplication → Tier 3 Live Evaluation → Reports

Each tier is independent. Add its extra to your command:

TierQuestionExample after setupAdditional setup
Tier 1: ValidationIs the skill safe, well-formed and harness-compatible?uv run skillevaluator tier1 ./my-skill --no-llmExternal scanners for full static coverage; a provider and model clients enable LLM checks
Tier 2: DeduplicationDoes guidance repeat or overlap?uv run --extra tier2 skillevaluator tier2 ./my-skillEmbeddings and chat; add --catalog ./skill-catalog.json for inter-skill comparison
Tier 3: Live EvaluationDoes the skill help the agent?uv run --extra tier3 skillevaluator tier3 ./my-skillProvider access, a supported agent, and a running sandbox (Docker by default)

Tier 1 fetches pinned, verified SkillSpector and Gitleaks releases when they are missing; Semgrep is optional. See scanners. --no-llm does not remove scanner requirements; a scanner that cannot run is INCOMPLETE and exits nonzero. SkillSpector provides specialized skill security scanning; Harbor, the open-source agent evaluation framework, powers sandboxed agent runs.

Tier 2 reports when inter-skill comparison did not run. Tier 3 autopilot reuses sources or creates one skill-grounded case with limited coverage. See tier guides.

Run the combined pipeline

After setting up the provider, scanners, and agent sandbox, run all three tiers:

uv run --python 3.13 --extra all skillevaluator doctor --env-mode docker
uv run --python 3.13 --extra all skillevaluator validate ./my-skill

doctor checks readiness, not inference success. validate runs all tiers with autopilot for skills. Use --tiers 1,2 to skip live evaluation or --tiers 1 for static checks. --no-autopilot requires an existing source. Other artifact types retain their applicable stages.

Tier 1 gates validate. Tier 2 gates unless --no-block-on-dedup is set; Tier 3 gates only with --block-on-agent-eval. Model and managed-sandbox usage can incur charges; see runtime and cost.

Standalone generation needs substantive evals/EVAL.md guidance, a --prompt file with task requests and outcomes, or matching retained traces. --force below replaces the autopilot starter; --refine refines it. Explicit --grounding skill permits skill-only starters. Review cases; temp

// HOW IT'S BUILT

// FILES

13 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • skills/skillevaluator/SKILL.md11 KB5d6c5412b3fdupstream ↗
    ---
    name: skillevaluator
    description: Run and interpret SkillEvaluator for agent skills. Use for any skill benchmark or eval - validating or scoring a SKILL.md, writing evals.json test cases or evals config, explaining benchmark reports, verdicts and exit codes, or gating skills in CI.
    license: Apache-2.0
    compatibility: Requires Python 3.12-3.14 and the skillevaluator CLI; live evaluation also needs Docker (or another backend) and an LLM provider key.
    metadata:
      author: Christopher Kevin <[email protected]>
      version: "1.0.0"
      domain: [evaluation, ai-ml]
      tags: [skills, evaluation, benchmarking, ci, security]
    ---
    
    # SkillEvaluator
    
    Run the SkillEvaluator CLI on agent skills and report what the evidence shows:
    Tier 1 validation (static, security, quality), Tier 2 deduplication, and Tier 3
    live agent evaluation that compares runs with and without the skill.
    
    ## Purpose
    
    Give an agent one reliable procedure for evaluating skills with
    SkillEvaluator: pick the right tier, check readiness, run it, and report
    scores, findings and exit codes truthfully.
    
    ## When To Use
    
    Use this skill when the user wants to:
    
    - check whether a skill is well-formed, safe, and good enough to publish
    - get a quality score, rubric score, security or PII scan for a skill
    - find repeated guidance inside a skill or overlap across a skill collection
    - measure whether a skill actually helps an agent (Skill Lift, pass@k)
    - write or fix `<skill-dir>/evals/evals.json`, `<skill-dir>/evals/config.yml`, custom graders, or Harbor tasks
    - read a SkillEvaluator report, verdict, or exit code
    - add SkillEvaluator checks to GitHub Actions or GitLab CI
    
    Do not use it to write the target skill's domain content, to review ordinary
    application code, or to evaluate an LLM without a skill. For converting an
    existing benchmark into a custom grader, read
    [references/custom-graders.md](references/custom-graders.md).
    
    ## Prerequisites
    
    - A target directory containing `SKILL.md` (or a folder of skills, `.mdc`
      rules, or a plugin manifest).
    - The CLI: `pip install --upgrade "skillevaluator[all]"`, or from a checkout
      `uv run --python 3.13 --extra all skillevaluator ...`. Check with
      `skillevaluator --version`. Extras: `llm` (rubric), `security` (scanners),
      `tier2`, `tier3`, `all`, `dashboard`.
    - No key for keyless commands (`quality-check`, `security-scan`, `pii-scan`,
      `lint-scripts`, `tier1 PATH --no-llm`, `validate PATH --tiers 1`).
    - A provider for LLM stages, Tier 2 and Tier 3. Simplest:
      `export SKILL_EVAL_LLM_PROVIDER=nv_build` and `export NVIDIA_API_KEY=...`.
      Other providers: [references/providers.md](references/providers.md).
    - For Tier 3: an agent (`opencode`, `codex`, or `claude-code`) and a sandbox.
      Harbor is installed with the `tier3` extra; you do not run it yourself.
      The default `--env-mode docker` needs a Docker daemon. Without Docker, use
      `--env-mode local` (experimental): it runs the agent CLI already installed
      on the host under Bubblewrap (Linux) or Seatbelt (macOS); trusted skills only.
    
    Never print, echo, or commit API keys; keep them in the host environment.
    
    ## Choose The Command
    
    | User goal | Command | Needs |
    | --- | --- | --- |
    | Fast keyless quality score | `skillevaluator quality-check <skill-dir>` | nothing |
    | Full static gate for one skill | `skillevaluator tier1 <skill-dir> --no-llm` | scanners (auto-installed) |
    | Static gate plus LLM rubric and security review | `skillevaluator tier1 <skill-dir>` | provider |
    | LLM rubric only | `skillevaluator rubric-eval <skill-dir>` | provider, `llm` extra |
    | Security / PII scan only | `skillevaluator security-scan <skill-dir>`, `skillevaluator pii-scan <skill-dir>` | nothing |
    | Repeated content inside one skill | `skillevaluator tier2 <skill-dir>` | provider with embeddings |
    | Overlap across a collection | `skillevaluator similarity-check <skills-dir>` | embeddings |
    | Does the skill help an agent? | `skillevaluator tier3 <skill-dir>` | provider, agent, Docker |
    | Same, without Docker | `skillevaluator tier3 evaluate <skill-dir> -a opencode --env-mode local` | provider, agent CLI on host |
    | Everything, with one exit code | `skillevaluator validate <skill-dir>` | all of the above |
    | CI-safe keyless gate | `skillevaluator validate <skills-dir> --profile default --tiers 1 -r json,markdown,sarif -o reports` | scanners |
    | Set up repo CI | `skillevaluator ci init` | GitHub checkout |
    
    Full flag tables: [Tier 1](references/commands-tier1.md),
    [validate](references/commands-validate.md),
    [Tier 2, Tier 3 and setup](references/commands-tier2-tier3.md).
    
    ## Workflow
    
    1. **Locate the target.** Confirm the path has `SKILL.md` and that its
       frontmatter `name` matches the directory name. For a folder of skills, the
       commands evaluate each skill inside it.
    2. **Check readiness before paid runs.** Run `skillevaluator health-check --tier 1`
       for scanners, `skillevaluator doctor --tier 2` for providers, and
       `skillevaluator doctor --env-mode docker` (add `--verify-models`) for Tier 3.
       Fix every failing row first.
    3. **Start keyless.** Run `skillevaluator tier1 <skill-dir> --no-llm`. Fix
       CRITICAL and HIGH findings before anything else; `lint` findings are advisory.
    4. **Add LLM checks** when a provider is set: `skillevaluator tier1 <skill-dir>`
       (rubric, LLM security, verification) and `skillevaluator tier2 <skill-dir>`.
    5. **Prepare Tier 3 cases.** If `<skill-dir>/evals/evals.json` is missing, first
       capture task requests and observable outcomes in `<skill-dir>/evals/EVAL.md`;
       headings alone do not qualify. Generate with
       `skillevaluator create-eval-dataset <skill-dir> --full --prompt <skill-dir>/evals/EVAL.md`,
       then review it. The default prefers author guidance, then matching retained
       traces; existing datasets need `--force` or `--refine`. Skill-only
       generation requires explicit opt-in with `--grounding skill`. Review:
       positive cases (explicit, implicit, contextual) and at least one negative
       case with `"expected_skill": null`. Required shape: top-level `skill_name`
       and an `evals` array; each case has `id`, `prompt`, `expected_output`, and
       `expected_skill` (optional `assertions`, `files`). Validate with
       `skillevaluator tier3 validate <skill-dir> --strict`. Format:
       [references/tier3-datasets.md](references/tier3-datasets.md); run policy and
       scoring: [references/tier3-running.md](references/tier3-running.md).
    6. **Run live evaluation.** `skillevaluator tier3 <skill-dir>` (or
       `skillevaluator tier3 evaluate <skill-dir> -a opencode --env-mode docker --n-attempts 3`).
       Use `--max-cases 1` for a cheap smoke run first; such runs are never
       publication evidence.
    7. **Read the results** (see below), open `skillevaluator view <skill-dir>`, and
       for follow-up questions run `skillevaluator tier3 feedback <run-dir>`.
    8. **Iterate.** Change the skill, rerun the same command with the same cases,
       agent and model, and compare. Treat a single run's lift as directional.
    
    ## Read The Results
    
    Exit codes: `0` success, `1` a gate failed, evidence is `INCOMPLETE`, or the
    run could not complete, `2` usage error (bad flag or missing path; fix the
    command, not the skill).
    
    - **Quality** (0-100): Correctness 35%, Discoverability 25%, Reliability 25%,
      Efficiency 15%. Grades A>=90, B>=80, C>=70. Passing needs `--min-score`
      (default 70) and no HIGH/CRITICAL quality finding.
    - **Rubric** (0-100): nine criteria scored 0-10; passing needs the weighted
      score >= `--min-score` and every criterion >= 7.
    - **Tier 2**: HIGH or CRITICAL duplicate findings exit `1` in `tier2 PATH`.
    - **Tier 3**: five dimensions (Security, Correctness, Discoverability,
      Effectiveness, Efficiency), each PASS >= 0.50, NEUTRAL 0.40-0.50, FAIL < 0.40.
      Verdict PASS needs every dimension >= 0.50 for at least one agent. Skill
      Lift = with-skill overall - baseline overall; PASS >= +0.05. `INCONCLUSIVE:
      skill never activated` means no positive trial loaded the skill; fix the
      description or prompts, not the scores. `tier3 PATH` exits `0` on a `FAIL`
      verdict; only an incomplete run exits `1`.
    - **Missing scores are `NO SCORE` or `None`, never 0.** Do not average them in.
    - **Publication**: require `publication_status == "pass"` in the JSON report,
      not just `overall_passed`.
    
    Report anatomy and JSON fields: [references/reports-and-ci.md](references/reports-and-ci.md).
    
    ## Examples
    
    ```bash
    # Keyless first look
    skillevaluator quality-check <skill-dir> -r cli -r json -o <report-dir>
    
    # Static gate for a whole collection, complete report even on failures
    skillevaluator validate <skills-dir> --profile default --tiers 1 -c -r json,markdown -o reports
    
    # LLM-backed Tier 1 and Tier 2, dedup advisory
    skillevaluator validate <skill-dir> --tiers 1,2 --llm --no-block-on-dedup
    
    # Live A/B evaluation with two agents, retaining trajectories for debugging
    skillevaluator tier3 evaluate <skill-dir> -a opencode,codex --n-attempts 3 --harbor-keep-jobs
    
    # Baseline only, to calibrate cases before the skill exists
    skillevaluator tier3 evaluate <skill-dir> --skip-with-skill
    ```
    
    ## Limitations
    
    - Tier 1 and the quality score are static: they check structure and wording,
      not whether the skill works. Only Tier 3 observes agent behavior.
    - LLM judges (rubric, Tier 3 correctness/effectiveness) vary between runs and
      models; compare runs only with the same model, cases, and attempts.
    - Tier 3 costs model tokens and sandbox time per case x attempt x agent x 2
      arms; a generated starter case gives limited coverage.
    - `validate --tiers` always includes Tier 1; `--tiers 2` or `--tiers 3` alone is
      rejected. Use `tier2 PATH` or `tier3 PATH` to run a tier on its own.
    - Agent credentials come only from the host environment; `<skill-dir>/evals/config.yml`
      `runtime_env` rejects provider keys.
    
    ## Troubleshooting
    
    | Symptom | Cause | Fix |
    | --- | --- | --- |
    | Exit `2` | Unknown flag, bad value, or missing path | Check `skillevaluator <command> --help`; prefix paths named like commands with `./` |
    | Check shows `INCOMPLETE` | A scanner is missing or failed | Run `skillevaluator health-check --tier 1`; install the `security` extra or the missing scanner |
    | "provider is required" / exit `1` before reports | No `SKILL_EVAL_LLM_PROVIDER` or key, or several keys exported | Set exactly one provider and its key; see providers reference |
    | Quality below 70 | Missing `## Prerequisites`, `## Limitations`, `## Troubleshooting`, vague description, no examples | Read each finding's suggestion; do not stuff keywords |
    | `INCONCLUSIVE: skill never activated` | Description does not match the prompts | Make the description name the tasks; add explicit-bucket cases |
    | Docker trials fail to start | Daemon down or outdated engine | `docker info`; rerun `skillevaluator doctor --env-mode docker` |
    | Agent model rejected | Default model not valid for that agent | `--agent-model AGENT=MODEL`, verify with `doctor --verify-models` |
    | Run incomplete (`Scored attempt coverage is n/m`) | A trial errored or timed out | Rerun; raise `--timeout-multiplier`; inspect with `--harbor-keep-jobs` |
    
    More cases: [references/troubleshooting.md](references/troubleshooting.md).
    
    ## Final Response
    
    Report to the user:
    
    - the exact commands run and their exit codes
    - scores and verdicts per tier, quoting the report values (not estimates)
    - the top findings to fix, each with its file and suggested change
    - report paths (`<report-dir>/...`, `<skill-dir>/evals/results/<run-id>/report.html`)
    - what was not run (skipped tiers, missing keys, `--max-cases` smoke runs) and
      therefore not proven
    
  • skills/skillevaluator/references/commands-tier1.md2.7 KB5ba21f55659fupstream ↗
    # Tier 1 command reference
    
    Report options and Tier 1 commands. `skillevaluator <command> --help` is authoritative for the installed version.
    
    Global options go before the command: `skillevaluator -v validate <skill-dir>`.
    
    ## Reports (Tier 1 and Tier 2 commands)
    
    | Flag | Default | Effect |
    | --- | --- | --- |
    | `-r, --report` | `cli,json,html` for `tier1 PATH`, `tier2 PATH`, `validate`; `cli` for focused commands | `cli`, `json`, `html`, `markdown`, `sarif`; comma-separated or repeated |
    | `-o, --output-dir` | `reports` | Report directory |
    
    Passing `-r` is honored exactly: `-r cli` writes no files. Basenames:
    `validate` writes `skillevaluator-output-<timestamp>.*`; `tier1`/`tier2` write
    `skillevaluator-tier1.*`/`skillevaluator-tier2.*`; focused commands write
    `skillevaluator-<kind>.*` (for example `skillevaluator-quality.json`) and
    overwrite earlier runs.
    
    ## Direct tier workflows
    
    A bare `tier1`, `tier2`, or `tier3` prints help. Prefix a path with `./` when
    its name matches a subcommand (`tier1 <path>/validate`).
    
    `skillevaluator tier1 PATH`: all static checks including `dependency`, plus
    rubric, LLM security and finding verification when a provider is configured.
    
    | Flag | Default | Effect |
    | --- | --- | --- |
    | `--checks TEXT` | all applicable | Subset: `schema`, `version`, `security`, `pii`, `license`, `code-integrity`, `unicode`, `quality`, `lint`, `dependency` |
    | `--skip CHECKS` | none | Leave checks out (also `secrets` for Gitleaks); the run cannot certify Tier 1 |
    | `--llm / --no-llm` | on when a provider is configured | Explicit `--llm` requires a valid provider |
    | `--min-score INTEGER` | `70` | Quality and rubric minimum |
    | `--quality-config PATH` | packaged | Replace expected-heading rules |
    | `--pii-waivers PATH` | none | Reviewed, expiring PII waivers |
    | `--profile TEXT` | `default` | Validation policy |
    
    `skillevaluator tier2 PATH`: intra-skill duplicate analysis of one skill.
    `--catalog FILE` adds inter-skill comparison (build it with
    `skillevaluator similarity-check <skills-dir> --save-catalog <catalog.json>`);
    `--threshold` (0.80), `--similarity-threshold` (0.75), `--full-body`.
    
    `skillevaluator tier3 PATH`: generates one starter case when no source
    exists, then runs the provider-native agent in Docker with and without the
    skill. Takes every `tier3 evaluate` option. Does not accept `-r`/`-o`.
    
    ## Focused Tier 1 commands
    
    | Command | Key flags |
    | --- | --- |
    | `quality-check PATH` | `--min-score` (70), `--quality-config` |
    | `rubric-eval PATH` | `--min-score` (70); every criterion must reach 7/10 |
    | `security-scan PATH` | `--llm`, `--llm-verify`, `--skillspector-baseline` |
    | `pii-scan PATH` | `--llm-verify`, `--pii-waivers` |
    | `lint-scripts PATH` | report options only; advisory |
    
  • skills/skillevaluator/references/commands-tier2-tier3.md2.5 KBafd216c06369upstream ↗
    # Tier 2, Tier 3 and setup command reference
    
    `skillevaluator <command> --help` is authoritative.
    
    ## Tier 2 commands
    
    | Command | Key flags |
    | --- | --- |
    | `context-optimization-check PATH` (alias `dedup-scan`) | `--threshold` (0.8), `--model`, `--llm-model` |
    | `similarity-check PATH` | `--threshold` (0.75), `--full-body`, `--catalog`, `--save-catalog`, `--max-entries` |
    
    ## Tier 3 commands
    
    | Command | Purpose |
    | --- | --- |
    | `tier3 evaluate PATH` | Live A/B run; add `--autopilot` to generate a missing case |
    | `tier3 validate PATH` | Check `<skill-dir>/evals/` (`--strict`, `--json`, `--harbor-contract`) |
    | `create-eval-dataset PATH` | Write `<skill-dir>/evals/evals.json` (`--grounding auto` requires substantive author guidance or usable matching retained traces; existing datasets need `--force` or `--refine`; opt in to `--grounding skill` for skill-only generation; `--full`, `--no-llm`, `--dry-run`, `--force`, `--prompt`, `--refine`) |
    | `init-custom-grader PATH` | BYOG starter (`--mode`, `--language`) |
    | `init-harbor-task PATH` | BYOT starter (`--case-id`, `--with-config`) |
    | `tier3 select-decoys PATH` | Save neighbor skills for group workspaces |
    | `view PATH`, `compare PATH` | Open latest report; compare agents |
    | `tier3 feedback RUN_DIR` | Chat about one completed run (`--skill`) |
    | `harbor-view JOBS_DIR` | Browse jobs kept with `--harbor-keep-jobs` |
    | `dashboard [PATHS]` | Tables over saved reports (`dashboard` extra) |
    
    Key `tier3 evaluate` flags: `-a, --agents` (`opencode`, `codex`,
    `claude-code`), `--env-mode` (`docker` default, `local`, `kata`, `gke`,
    `daytona`, `e2b`, `modal`, ...), `--n-attempts`, `--pass-threshold`,
    `--stop-on-pass`, `--n-concurrent`, `--max-agents`, `--max-cases`, `--model`,
    `--agent-model AGENT=MODEL`, `--skip-baseline`, `--skip-with-skill`,
    `--grading-mode default|default_plus_custom|custom_only`, `--results-dir`,
    `--include-skills`, `--skill-workspace-mode isolated|group`, `--copy-repo`,
    `--timeout-multiplier`, `--harbor-keep-jobs`, `--agent-runtime-preflight`,
    `--override-cpus`, `--override-memory-mb`, `--progress`, `--verbose`.
    
    ## Readiness and setup
    
    | Command | Purpose |
    | --- | --- |
    | `health-check [--tier 1\|2\|3\|all]` | Offline readiness; `--checks` for Tier 1 scanners |
    | `doctor [--tier ...] [--env-mode M] [-a AGENTS] [--verify-models]` | Deeper readiness; `--verify-models` makes live catalog probes |
    | `models [--limit N] [--json]` | List provider catalog models |
    | `ci init`, `ci status`, `ci demo` | Guided GitHub CI setup |
    
  • skills/skillevaluator/references/commands-validate.md1.5 KBcc1dd57690efupstream ↗
    # validate command reference
    
    The combined pipeline. `skillevaluator validate --help` is authoritative.
    
    ## validate
    
    `skillevaluator validate PATH` runs Tier 1, Tier 2 and Tier 3 (with
    autopilot) for skills. Tier 1 always gates; Tier 2 gates unless
    `--no-block-on-dedup`; Tier 3 gates only with `--block-on-agent-eval`. A
    folder without a root `SKILL.md` validates each skill inside it.
    
    | Flag | Effect |
    | --- | --- |
    | `--tiers 1`, `--tiers 1,2`, `--tiers 1,3` | Scope; Tier 1 always included |
    | `--type skill\|rules\|workflows\|plugin\|auto` | Force the content type |
    | `--checks`, `--skip` | Tier 1 selection (`dependency` is opt-in here; `--checks` replaces the list) |
    | `--llm`, `--llm-verify` | LLM security analysis and false-positive suppression |
    | `--min-score`, `--quality-config`, `--pii-waivers`, `--skillspector-baseline FILE` | Tier 1 tuning |
    | `--previous-version VERSION` | Require a greater `metadata.version` (single skill only) |
    | `--fail-fast`, `-c, --continue-on-failure` | Stop early, or record every issue (recommended in CI) |
    | `--no-dedup`, `--no-block-on-dedup` | Skip Tier 2, or make it advisory |
    | `--no-tier3`, `--no-autopilot`, `--block-on-agent-eval` | Tier 3 scope and gating |
    | `-a`, `--env-mode`, `--n-attempts`, `--max-cases`, `--agent-model`, `--harbor-keep-jobs`, ... | Forwarded to Tier 3 |
    | `--workers N` | Validate N skills of a folder in parallel |
    | `--profile`, `--policy FILE` | Policy profile and overlay |
    | `--evaluated-source-repository`, `--evaluated-source-revision` | Provenance for `BENCHMARK.md` |
    
  • skills/skillevaluator/references/custom-graders.md906 Bcc4d8f1d873fupstream ↗
    # Custom graders and Harbor tasks
    
    Bring your own grader (BYOG) or Harbor task (BYOT).
    
    - `default_plus_custom`: standard dimensions stay authoritative and your
      `custom_metrics` are added. `custom_only`: your grader's `overall` decides.
    - Scaffold with `skillevaluator init-custom-grader <skill-dir> --language python`
      or `skillevaluator init-harbor-task <skill-dir> --case-id case-001 --with-config`,
      replace the placeholder logic, then run
      `skillevaluator tier3 validate <skill-dir> --harbor-contract`.
    - A grader reads `/logs/agent/trajectory.json`, `/tests/entry.json`, and
      `/workspace/input/`, and writes `/logs/verifier/reward.json` plus
      `/logs/verifier/reward.txt` with values from 0.0 to 1.0.
    - Do not name custom metrics `security`, `skill_execution`,
      `skill_efficiency`, `accuracy`, `goal_accuracy`, `behavior_check`,
      `overall`, `details`, `metrics`, `metric_set`, or `entry_id`.
    
  • skills/skillevaluator/references/providers.md1.6 KBcd071bdd2b3fupstream ↗
    # Providers and credentials
    
    Set exactly one provider. If `SKILL_EVAL_LLM_PROVIDER` is unset and exactly
    one of `NVIDIA_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` is exported,
    it is auto-selected; with more than one, LLM commands fail until the
    provider is set.
    
    | Provider | Variables | Notes |
    | --- | --- | --- |
    | `nv_build` | `NVIDIA_API_KEY` | One key covers Tier 1 LLM, Tier 2 embeddings, and Tier 3 agents in `docker`, `kata`, `local`; default agent `opencode` |
    | `openai` | `OPENAI_API_KEY` | Default agent `codex` |
    | `anthropic` | `ANTHROPIC_API_KEY` | Default agent `claude-code`; Tier 2 needs a separate embedding provider |
    | `bedrock` | AWS credential chain, `AWS_REGION` | Tier 2 needs a separate embedding provider |
    | `openai-compatible` | `SKILL_EVAL_LLM_BASE_URL`, `SKILL_EVAL_LLM_API_KEY` | Any gateway; override models whose catalog IDs differ |
    
    Overrides: `SKILL_EVAL_LLM_MODEL` (evaluator and judge),
    `LLM_JUDGE_MODEL` (judge only), `SKILL_EVAL_EMBEDDING_MODEL`,
    `--agent-model AGENT=MODEL` (Tier 3 agent). OpenCode on a gateway takes
    `--agent-model opencode=openai/MODEL`. Codex needs a Responses-API model,
    OpenCode Chat Completions, Claude Code Anthropic Messages, all with tool
    calls.
    
    Rules:
    
    - Credentials come only from the host environment (or masked CI secrets).
      `harbor.runtime_env` in `<skill-dir>/evals/config.yml` rejects provider keys and base
      URLs with a hard error.
    - `skillevaluator models` and `doctor --verify-models` check catalog
      reachability; a public catalog does not prove the inference key works.
    - Judge failures make a result unscored (`NO SCORE`), not `0.0`.
    
  • skills/skillevaluator/references/reports-and-ci.md1.8 KBd46dfe05b72bupstream ↗
    # Reports and CI
    
    ## JSON fields to read
    
    From `skillevaluator-output-<timestamp>.json` (or `skillevaluator-tier1.json`):
    
    | Field | Meaning |
    | --- | --- |
    | `overall_passed`, `overall_status` | Command gate: `passed`, `failed`, `incomplete` |
    | `publication_status` | `pass`, `neutral`, `fail`, `incomplete`; use for publishing |
    | `quality_summary[].overall_score`, `.grade` | Quality 0-100 and A-F |
    | `severity_counts` | `critical`, `high`, `medium`, `low` totals |
    | `finding_groups[]` | One entry per unique finding (`finding_id` `SEF-...`) |
    | `tier3.run_id` | Run identity for the Tier 3 section |
    
    `overall_passed` can be true while an advisory tier did not run; require
    `publication_status == "pass"` before claiming a skill is publishable.
    
    Example gate on a stricter score:
    
    ```bash
    report=$(ls -t <report-dir>/skillevaluator-output-*.json | head -1)
    jq -e '([.quality_summary[]?.overall_score] | min // 100) >= 80' "$report"
    ```
    
    `validate` on a skill also writes `BENCHMARK.md`: verdict first, the three
    tier states, and `Baseline -> Skill Uplift` columns.
    
    ## CI
    
    Guided GitHub setup from a checkout: `skillevaluator ci init` (then
    `ci status`). Manual: copy the GitHub or GitLab template from the
    repository's `ci/bundles/<version>/` folder.
    
    Recommended keyless gate:
    
    ```bash
    skillevaluator validate <skills-dir> --profile default --tiers 1 -c \
      -r json,markdown,sarif -o reports --min-score 70
    ```
    
    Adopt progressively: run with `|| true` (advisory), then gate on Tier 1,
    raise `--min-score`, add `--llm --tiers 1,2` with a masked provider secret,
    and finally an advisory Tier 3 job (`--block-on-agent-eval` when ready).
    Keep keys in CI secrets exposed through the job environment, never in the
    repository. Pin policy with `--policy <policy.yaml>` so severity changes are
    reviewed like code.
    
  • skills/skillevaluator/references/tier3-datasets.md4.2 KB03c2e882591cupstream ↗
    # Tier 3 evaluation datasets
    
    Tier 3 runs a real agent on each case with the skill installed and without it (baseline). This file covers the `<skill-dir>/evals/` folder and dataset format.
    
    ## The <skill-dir>/evals/ directory
    
    | Path | Purpose |
    | --- | --- |
    | `<skill-dir>/evals/evals.json` | Required dataset (`.jsonl`, `.yaml`, `.yml` also accepted) |
    | `<skill-dir>/evals/config.yml` | Run policy |
    | `<skill-dir>/evals/EVAL.md` | Guidance for `create-eval-dataset` (`## Questions`, `## Behaviors`, `## Negative Cases`, `## Notes`) |
    | `<skill-dir>/evals/files/` | Fixtures staged into `/workspace/input/` |
    | `<skill-dir>/evals/environment/` | `Dockerfile`, `docker-compose.yaml` sidecars, `mcp_servers.toml` |
    | `<skill-dir>/evals/harbor/` | Native Harbor tasks (BYOT) |
    | `<skill-dir>/evals/grader.py` or `grader.sh` | Custom grader (BYOG) |
    | `<skill-dir>/evals/results/` | Output only; never edit or commit |
    
    ## Dataset format
    
    ```json
    {
      "skill_name": "my-skill",
      "evals": [
        {
          "id": "my-skill-001",
          "prompt": "Convert report.docx in /workspace/input to PDF",
          "expected_output": "A PDF of the document saved next to the input",
          "assertions": ["Saves report.pdf next to report.docx", "Keeps the document's headings in the PDF"],
          "expected_skill": "my-skill",
          "files": ["files/case-001"]
        },
        {
          "id": "my-skill-002-negative",
          "prompt": "What is the capital of France?",
          "expected_output": "Answers directly without loading my-skill",
          "expected_skill": null
        }
      ]
    }
    ```
    
    Required: `skill_name`, `evals`, and per case `id` (letters, digits, `.`,
    `_`, `-`; at most 128 chars, unique), `prompt`, `expected_output`. Optional:
    `assertions`, `expected_skill` (`null` for negatives), `expected_script`,
    `acceptable_skills`, `files` (paths relative to `<skill-dir>/evals/`; omit it and the
    whole `<skill-dir>/evals/files/` is staged). The legacy flat array with `question`,
    `ground_truth`, `expected_behavior` still runs with a deprecation warning.
    
    Good datasets cover four buckets: explicit (names the skill), implicit
    (describes the task), contextual (task inside a project), negative (should
    not trigger). Write `expected_output` as an observable outcome and one
    behavior per assertion. The baseline is graded on the same prompt,
    `expected_output` and assertions without the skill, so keep them about the task
    outcome: never point the prompt at the skill's files or assert that the agent
    read `SKILL.md`, ran the skill's scripts or used the skill (`skill_execution`
    scores that). `tier3 validate` warns about common wordings of these; it is a
    heuristic, so review the cases too.
    
    ## Guided generation
    
    Standalone `create-eval-dataset` defaults to `--grounding auto`: prefer author
    guidance in `--prompt` / `<skill-dir>/evals/EVAL.md` with real content beyond
    headings, then usable matching retained trajectories from this skill's completed
    runs. Write task requests and observable outcomes in the guidance file first:
    
    ```bash
    skillevaluator create-eval-dataset <skill-dir> --full --prompt <skill-dir>/evals/EVAL.md
    ```
    
    If neither guidance nor matching retained traces exists, generation fails before
    provider access or writes. `--refine` and `--no-llm` do not bypass that preflight.
    For the prior skill-only starter workflow, explicitly opt in:
    
    ```bash
    skillevaluator create-eval-dataset <skill-dir> --full --no-llm --grounding skill
    ```
    
    Auto leaves an existing dataset unchanged unless you pass `--force` or
    `--refine`, even when matching traces exist. To refine from completed results:
    
    ```bash
    skillevaluator create-eval-dataset <skill-dir> --refine --from-results <run-dir>
    ```
    
    Auto trace refinement preserves existing authored case IDs and expectations;
    unmatched cases and observed answer proposals need review. When auto selects
    traces, proposals that describe skill use leave `expected_output` empty for an
    independent authored outcome. Existing authored expectations are preserved. Explicit
    `--grounding trace` still opts into in-place refinement without `--refine`.
    Generation provenance records the effective `intent` or `trace` grounding and input hashes. Explicit
    `intent`, `trace`, and `skill` modes remain available. Unattended autopilot retains
    its existing skill-grounded one-case starter contract.
    
  • skills/skillevaluator/references/tier3-running.md2.3 KB3c6801b64257upstream ↗
    # Running and scoring Tier 3
    
    Run policy, run sequence, scoring and cost control.
    
    ## config.yml
    
    ```yaml
    schema_version: 1
    harbor:
      n_attempts: 3
      pass_threshold: 0.5
      n_concurrent: 2
      timeout_multiplier: 1.5
      agents:
        opencode:
          model: your-model-id
    skill_workspace:
      mode: isolated   # or group, with include: [...]
    grading:
      mode: default    # default | default_plus_custom | custom_only
    ```
    
    Unknown keys fail the load. CLI flags override config values.
    `pre_agent_setup` runs only with `--skip-baseline`. Never put credentials in
    `runtime_env`.
    
    ## Run sequence
    
    Write task requests and observable outcomes in `<skill-dir>/evals/EVAL.md`
    before generation; a file containing only headings does not qualify. For an
    existing dataset, use `--refine` with matching retained results to refine it,
    or `--force` to replace it after reviewing a `--dry-run` preview.
    
    ```bash
    skillevaluator doctor --env-mode docker --verify-models
    skillevaluator create-eval-dataset <skill-dir> --full --prompt <skill-dir>/evals/EVAL.md
    skillevaluator tier3 validate <skill-dir> --strict
    skillevaluator tier3 evaluate <skill-dir> -a opencode --max-cases 1   # smoke
    skillevaluator tier3 evaluate <skill-dir> -a opencode --n-attempts 3
    skillevaluator view <skill-dir>
    ```
    
    Results land in `<skill-dir>/evals/results/<timestamp>_<pid>_<id>/` with `result.json`,
    `run_config.json`, `report.html` and per-agent `lift.json`,
    `with-skill/`, `without-skill/`. Use `--results-dir` to write elsewhere.
    
    ## Scoring
    
    Standard grading scores five dimensions on 0.0-1.0 and averages them:
    
    | Dimension | Source |
    | --- | --- |
    | Security | Deterministic trace checks (destructive commands, uploads) |
    | Correctness | LLM judge of the answer (`accuracy`) |
    | Discoverability | Whether the skill was loaded when expected (`skill_execution`) |
    | Effectiveness | LLM judges `goal_accuracy` and `behavior_check` |
    | Efficiency | Wasted tool or skill calls (`skill_efficiency`) |
    
    Bands: PASS >= 0.50, NEUTRAL 0.40-0.50, FAIL < 0.40. Verdict PASS needs one
    agent with every dimension passing. Lift band: PASS >= +0.05, FAIL <= -0.10.
    pass@k counts a case passed when any attempt reaches `--pass-threshold`.
    
    ## Cost control
    
    Trials = cases x attempts x agents x 2 arms. Use `--max-cases`,
    `--skip-baseline` (no lift), `--stop-on-pass`, and `--n-concurrent` to
    control spend. Runs that skip cases are never publication evidence.
    
  • skills/skillevaluator/references/troubleshooting.md2.5 KB829959a34cf8upstream ↗
    # Troubleshooting
    
    | Symptom | Cause | Fix |
    | --- | --- | --- |
    | `Error: No such command` or exit `2` | Wrong command or flag for this version | `skillevaluator --version`, then `skillevaluator <command> --help` |
    | `tier1 validate` evaluated the wrong thing | Directory named like a subcommand | Use `<path>/validate` |
    | `--tiers 2` rejected | `validate` always includes Tier 1 | Use `skillevaluator tier2 PATH` |
    | Scanner `INCOMPLETE` | SkillSpector, Gitleaks or Bandit missing or failed | `skillevaluator health-check --tier 1`; install the `security` extra; an old scanner earlier on `PATH` wins over the pinned one |
    | `--skip` used and publication fails | Skipped checks cannot certify Tier 1 | Rerun without `--skip` |
    | Schema: directory does not match name | Folder name differs from frontmatter `name` | Rename one so both match (kebab-case) |
    | Quality below minimum | Missing expected headings, vague description, no examples, `SKILL.md` over 500 lines or 5,000 tokens | Follow each finding's suggestion; move detail into `references/` |
    | Rubric fails although the score is high | One criterion below 7/10 | Read that criterion's feedback in the JSON report |
    | Tier 2 needs embeddings | Anthropic and Bedrock have no embeddings | Set `SKILL_EVAL_EMBEDDING_*` for another provider |
    | `<skill-dir>/evals/config.yml` rejected | Unknown key or credential in `runtime_env` | Fix the key; move secrets to the host environment |
    | `Missing required fields: question` | Legacy entry without `question`, or new entry without `prompt` | Use the agentskills.io shape |
    | Skill never activated | Description and prompts do not match | Name the tasks in the description; add explicit cases |
    | Agent auth fails | Agent credential not exported where `skillevaluator` runs | Export it on the host; `doctor -a AGENT` |
    | `AgentTimeoutError` or rate-limit errors | Slow model or too much concurrency | `--timeout-multiplier 2`, lower `--n-concurrent` |
    | Custom Dockerfile build fails | Base image lacks tools the agent or grader needs, or `python3 -I` cannot import the verifier packages (`ENV PIP_USER=1`, `PIP_TARGET` or `PYTHONPATH` installs) | Fix `<skill-dir>/evals/environment/Dockerfile`; install verifier packages into `python3`'s site-packages; rerun with `--harbor-keep-jobs` to read the build log |
    | Results need debugging | Harbor jobs deleted by default | Rerun with `--harbor-keep-jobs`, then `skillevaluator harbor-view JOBS_DIR` |
    | Lift differs between runs | LLM judges and agents are stochastic | Increase `--n-attempts`; compare only identical setups |
    
  • skills/skillevaluator/evals/config.yml108 B4e5c19a6d600upstream ↗
    schema_version: 1
    harbor:
      n_attempts: 2
      pass_threshold: 0.5
      n_concurrent: 4
      timeout_multiplier: 1.5
    
  • skills/skillevaluator/evals/evals.json6.6 KB2b41b7a6facdupstream ↗
    {
      "skill_name": "skillevaluator",
      "evals": [
        {
          "id": "skillevaluator-001-explicit-keyless",
          "prompt": "Use SkillEvaluator: I have a skill at ./skills/pdf-tools and no LLM API key. Give me the exact commands to check that it is well-formed, safe, and see its quality score, and tell me what exit code 1 versus 2 would mean.",
          "expected_output": "Recommends keyless commands such as `skillevaluator tier1 ./skills/pdf-tools --no-llm` and/or `skillevaluator quality-check ./skills/pdf-tools` (or `validate --tiers 1`), and explains exit 1 = a gate failed or evidence incomplete, exit 2 = usage error such as a bad flag or missing path.",
          "assertions": [
            "Recommends a keyless Tier 1 command (tier1 --no-llm, quality-check, or validate --tiers 1) on ./skills/pdf-tools",
            "Explains exit code 1 as a failed check or incomplete evidence and exit code 2 as a usage error",
            "Does not ask for or invent an API key"
          ],
          "expected_skill": "skillevaluator"
        },
        {
          "id": "skillevaluator-002-explicit-tier3",
          "prompt": "With SkillEvaluator, how do I measure whether my skill at ./my-skill actually makes an agent better? Write the full command sequence from readiness check to report, including a cheap smoke run first, and say which number tells me the improvement.",
          "expected_output": "A sequence using doctor (with --env-mode docker), creating/validating evals (create-eval-dataset, tier3 validate), a smoke `tier3 evaluate ... --max-cases 1`, a full `tier3 evaluate` or `tier3 ./my-skill` run with and without the skill, and `view`; identifies Skill Lift (with-skill minus baseline) as the improvement measure.",
          "assertions": [
            "Includes a readiness check with skillevaluator doctor",
            "Includes dataset generation or validation (create-eval-dataset or tier3 validate)",
            "Includes a smoke run limited with --max-cases",
            "Runs tier3 live evaluation comparing with-skill and baseline",
            "Names Skill Lift as with-skill minus baseline"
          ],
          "expected_skill": "skillevaluator"
        },
        {
          "id": "skillevaluator-003-implicit-dataset",
          "prompt": "I need test cases so an agent benchmark can tell whether my 'csv-cleaner' agent skill gets used correctly. Write the evals/evals.json file for it with three cases: one that names the skill, one that only describes the task, and one prompt where the skill should NOT be used.",
          "expected_output": "A valid evals/evals.json with top-level skill_name 'csv-cleaner' and an evals array of three cases each having id, prompt, expected_output (optionally assertions), where the negative case sets expected_skill to null.",
          "assertions": [
            "Output has top-level skill_name and evals keys",
            "Every case has id, prompt, and expected_output",
            "Includes one explicit and one implicit positive case",
            "The negative case sets expected_skill to null"
          ],
          "expected_skill": "skillevaluator"
        },
        {
          "id": "skillevaluator-004-implicit-report",
          "prompt": "My skill benchmark report says 'INCONCLUSIVE: skill never activated' and the Tier 3 command exited 0. Another run says verdict FAIL but also exited 0. What do these mean and what should I change?",
          "expected_output": "Explains the INCONCLUSIVE verdict as missing positive-trial activation, so the description or prompts should be fixed rather than the scores; explains that tier3 exits 0 on a FAIL verdict and only an incomplete run exits 1, with --block-on-agent-eval in validate to gate on Tier 3.",
          "assertions": [
            "Explains never activated means the skill was not loaded in positive trials",
            "Recommends improving the skill description or eval prompts",
            "Explains that a FAIL verdict alone exits 0 and only an incomplete run exits 1",
            "Mentions --block-on-agent-eval or another way to make Tier 3 gate"
          ],
          "expected_skill": "skillevaluator"
        },
        {
          "id": "skillevaluator-005-contextual-ci",
          "prompt": "Our repo keeps agent skills under ./skills and we use GitHub Actions. Add a workflow step that fails the PR when any skill fails static validation, uploads SARIF, and does not need any API key. Show the command the step should run.",
          "expected_output": "A GitHub Actions step running `skillevaluator validate ./skills --tiers 1` with SARIF/JSON reports to an output dir (e.g. `-r json,markdown,sarif -o reports`), relying on the exit code to fail the job, with no secrets required; optionally mentions `skillevaluator ci init`.",
          "assertions": [
            "Uses skillevaluator validate on ./skills with --tiers 1",
            "Requests a sarif report and an output directory",
            "Relies on the command exit code to fail the job",
            "Does not require an API key for this gate"
          ],
          "expected_skill": "skillevaluator"
        },
        {
          "id": "skillevaluator-006-contextual-config",
          "prompt": "I'm setting up live agent evaluation for ./skills/sql-helper. I want 3 attempts per case, the opencode agent, and the agent needs my OPENAI_API_KEY. Write evals/config.yml and tell me where the API key should go.",
          "expected_output": "An evals/config.yml with schema_version: 1 and harbor.n_attempts: 3 (and opencode agent settings), explicitly NOT placing the API key in config.yml or harbor.runtime_env; the key is exported in the host environment where skillevaluator runs.",
          "assertions": [
            "config.yml sets schema_version 1 and harbor n_attempts 3",
            "Does not put OPENAI_API_KEY in config.yml or runtime_env",
            "Says to export the key in the host environment or CI secrets"
          ],
          "expected_skill": "skillevaluator"
        },
        {
          "id": "skillevaluator-007-negative-write-skill",
          "prompt": "Write a Python function that removes duplicate rows from a list of dictionaries, keeping the first occurrence.",
          "expected_output": "A direct Python implementation; no SkillEvaluator commands or skill evaluation workflow.",
          "assertions": [
            "Provides a working Python function",
            "Does not run or recommend SkillEvaluator"
          ],
          "expected_skill": null
        },
        {
          "id": "skillevaluator-008-negative-general",
          "prompt": "Explain the difference between precision and recall in classification models in two sentences.",
          "expected_output": "A short correct explanation of precision and recall without invoking skill evaluation.",
          "assertions": [
            "Correctly defines precision and recall",
            "Does not run or recommend SkillEvaluator"
          ],
          "expected_skill": null
        }
      ]
    }
    
  • README.mdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/nvidia-skillevaluator-skills-skillevaluator.md

// REPO STATS

1.3k stars