agent-collaboration

Waza

A Go CLI for evaluating AI agent skills — scaffold eval suites, run benchmarks, and compare results across models.

📖 Getting Started / Docs

Installation

Binary Install (recommended)

Download and install the latest pre-built binary with the Bash install script on macOS, Linux, or Windows Bash environments such as Git Bash, MSYS2, or Cygwin:

curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash

The Bash script auto-detects the OS and architecture of the environment where Bash is running (linux/darwin/windows, amd64/arm64), downloads the latest standalone waza CLI release, verifies the checksum, and installs to /usr/local/bin (or ~/bin if not writable).

For native Windows PowerShell:

irm https://raw.githubusercontent.com/microsoft/waza/main/install.ps1 | iex

The PowerShell script downloads the latest standalone native Windows waza binary, verifies the checksum, and installs to an existing waza.exe location or %LOCALAPPDATA%\Microsoft\Waza. On Windows, piping the Bash command from PowerShell may invoke WSL and install the Linux binary inside WSL.

Or browse the GitHub Releases page and choose the standalone waza binary assets for the version you want.

Install from Source

Requires Go 1.26+:

Note: due to the use of LFS artifacts you cannot install waza using go install. To install waza outside of a normal release, clone the repository:

git clone https://github.com/microsoft/waza.git
cd waza

# ensure git LFS-based artifacts are available (for embedded copilot binaries)
git lfs install
git lfs pull

go build -o waza ./cmd/waza
./waza <waza command line>

Waza bundles the GitHub Copilot CLI used by the copilot-sdk executor and extracts it and its runtime assets to a versioned directory in the local user cache on first use (or under COPILOT_HOME/cache when set). Set COPILOT_CLI_PATH only when you need to force a specific Copilot CLI binary. Installation failures are reported without falling back to a CLI on PATH; set COPILOT_CLI_INSTALL_VERBOSE=1 for installation diagnostics.

Azure Developer CLI (azd) Extension

Waza is also available as an azd extension:

# Add the waza extension registry
azd ext source add -n waza -t url -l https://raw.githubusercontent.com/microsoft/waza/main/registry.json

# Install the extension
azd ext install microsoft.azd.waza

# Verify it's working
azd waza --help

Once installed, all waza commands are available under azd waza. For example:

azd waza init my-eval --interactive
azd waza run examples/code-explainer/eval.yaml -v

Update Notifications

Waza automatically checks for new versions in the background. If an update is available, a notice appears after command output:

A newer version of waza is available: v0.24.0 → v0.28.0. Run: waza update

Run waza update to download and execute the official OS-specific installer after an explicit confirmation prompt. It uses the Bash installer on macOS/Linux and the PowerShell installer on native Windows. Use waza update --yes to skip the prompt in scripted environments. The check is non-blocking (never slows commands), cached for 24 hours, and can be disabled with --no-update-check or WAZA_NO_UPDATE_CHECK=1.

Quick Start

For New Users: Get Started in 5 Minutes

See Getting Started Guide for a complete walkthrough:

# Initialize a new project
waza init my-project && cd my-project

# Create a new skill
waza new skill my-skill

# Define the skill in skills/my-skill/SKILL.md
# Write evaluation tasks in evals/my-skill/tasks/
# Add test fixtures in evals/my-skill/fixtures/

# Run evaluations
waza run my-skill

# Check skill readiness
waza check my-skill

All Commands

# Build
make build

# Initialize a project workspace
waza init [directory]

# Update waza to the latest release
waza update

# Create a new skill
waza new skill skill-name

# Create a new eval scaffold from an existing SKILL.md
waza new eval skill-name

# Generate a task YAML by recording a prompt run
waza new task from-prompt "Explain this code and suggest fixes" evals/code-explainer/tasks/recorded-task.yaml

# Check if a skill is ready for submission
waza check skills/my-skill

# Suggest an eval suite from SKILL.md
waza suggest skills/my-skill --dry-run
waza suggest skills/my-skill --apply

# Discover shared registry graders and add one to an eval
waza registry search factual --kind grader
waza registry add github.com/waza-evals/fact#[email protected] --eval eval.yaml --name factuality

# Verify eval coverage against SKILL.md requirements
waza spec verify skills/my-skill evals/my-skill/eval.yaml
waza spec verify skills/my-skill evals/my-skill/eval.yaml --fail --format github-actions

# Resolve remote grader refs and write waza.lock
waza get evals/my-skill/eval.yaml

# Note: 'generate' is available as an alias for 'new' (see below for new command)
# Note: Custom agents (.agent.md) are supported — see https://microsoft.github.io/waza/guides/custom-agents/

# Run evaluations (works with both skills and custom agents)
waza run examples/code-explainer/eval.yaml --context-dir examples/code-explainer/fixtures -v

# Grade output from a previous `waza run --output results.json ...`
waza grade eval.yaml --results results.json

# Compare results across models
waza compare results-gpt4.json results-sonnet.json

# Capture snapshots during a run and replay them later for determinism checks
waza run eval.yaml --snapshot ./snapshots/
waza replay ./snapshots/my-task-run1.json

# Run offline adversarial / fault-injection packs against a skill
waza adversarial --list-packs
waza adversarial --skill ./skills/my-skill --model gpt-4o

# Check whether a schema artifact needs migration
waza migrate eval.yaml

# Generate eval coverage grid
waza coverage --format markdown

# Count tokens in skill files
waza tokens count skills/

# Compare skill token budgets vs main
waza tokens compare main --skills --threshold 10

# Suggest token optimizations
waza tokens suggest skills/

Agent-assisted eval authoring

Point your coding agent at the Writing Eval Specs guide to create or update evals and guide eval-driven implementation. It provides the canonical workflow, target-specific validation steps, and a reusable delegation prompt.

Commands

waza update

Update waza to the latest release by running the official OS-specific installer after confirmation.

Example:

waza update
waza update --yes

waza init [directory]

Initialize a waza project workspace with separated skills/ and evals/ directories. Idempotent — creates only missing files.

Creates:

• skills/ — Skill definitions directory
• evals/ — Evaluation suites directory
• .github/workflows/eval.yml — CI/CD pipeline for running evals on PR
• .gitignore — Waza-specific exclusions
• README.md — Getting started guide for your project

Example:

waza init my-project
# Optionally creates first skill interactively

waza init my-project --no-skill
# Skip skill creation prompt

waza new skill <skill-name>

Create a new skill with scaffolded structure and evaluation suite. Detects workspace context and adapts output. In interactive mode, the wizard collects spec-aligned metadata: name, description, trigger phrases, and anti-trigger phrases.

Modes:

Project mode (detects