script-exec-blocked

🧠 WikiSkill

Compile agent experience into a persistent wiki — and let skills evolve themselves.

📚 Docs site: ashutoshsinghpr7.github.io/wikiskill · arXiv: 2608.27454

A faithful, production-minded implementation of WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution (arXiv:2608.27454, Google Research). The loop is agent-agnostic — Hermes Agent is the reference backend (built natively), Claude Code, Codex and GitHub Copilot CLI ship in the box, and OpenCode is on the roadmap (issue #13). Your agent becomes both the student and the teacher.

---

What this is

Agents fail. They also learn — but the lessons usually die with the session. WikiSkill fixes that by keeping a persistent knowledge wiki alongside the skill set, and running a closed evolution loop:

1. The agent runs training tasks with its current skills → raw execution traces
2. A Wiki Maintainer agent distills the traces into pattern pages (root causes, fixes)
3. A Skill Proposer agent reads the wiki + traces and proposes one skill change (create or patch)
4. Gating: the change is validated on held-out tasks — strictly better than the best score so far → kept; otherwise rolled back. The wiki is never rolled back.

Over iterations, knowledge compounds in the wiki while only proven improvements touch the skills.

                ┌──────────────────────────────────────────────────────┐
                │              EVOLUTION LOOP (Algorithm 1)            │
                │                                                      │
   tasks ─────► │  Inference Agent ──► raw/traces/ (immutable)         │
                │        │                                            │
                │        ▼                                            │
                │  Wiki Maintainer ──► wiki/patterns/, index, log     │
                │        │                                            │
                │        ▼                                            │
                │  Skill Proposer ──► proposal (create/patch skill)   │
                │        │                                            │
                │        ▼                                            │
                │  GATE: val score > R_best? ──yes──► keep, R_best=R  │
                │        │ no                                         │
                │        ▼                                            │
                │  rollback skills; wiki retained forever             │
                └──────────────────────────────────────────────────────┘

Why Hermes?

This is not a toy simulator. Every component is a real Hermes agent turn:

Why the isolated profile matters: gating is only meaningful if the agent sees exactly the candidate skill set. Each evolution workspace gets its own HERMES_HOME (bundled skills opted out, empty memory, skills symlinked per stage) — your real profile is never touched.

Quickstart (60 seconds)

pip install wikiskill        # from PyPI (wheel + sdist, Python ≥3.10)
wikiskill init demo           # workspace + 22-task auto-graded bench (13 train / 9 val)
wikiskill status
wikiskill evolve --iters 3    # full Algorithm 1 loop with your default model

Or from source: pip install -e . (installs the same wikiskill CLI).

Workspaces are created relative to your current directory (./workspaces/<domain>/) — and because skills/ lives outside the package, the wheel carries a copy of the framework skills that init stages into each workspace. If you installed from PyPI before 0.1.5, upgrade and re-init any existing workspace: those wheels shipped no skills/, so the maintainer and proposer turns ran with no skill loaded (silently — the scores still looked well-formed). See issue #29.

Looking a workspace up (rather than creating one) also prefers ./workspaces/ and then falls back to the old <repo>/workspaces/ location with a one-line notice, so existing workspaces keep working from any directory in a checkout — or pass --ws <path> to be explicit.

That's it. Each evolution workspace lives at workspaces/<domain>/:

workspaces/demo/
├── raw/traces/iter-01/{train,val}/<task>.jsonl   # immutable execution traces
├── wiki/                                          # persistent knowledge (never rolled back)
│   ├── index.md  ·  log.md  ·  skill-impact.md  ·  patterns/*.md
├── skills/active/                                 # git-managed evolving skill set (S₀ = ∅)
├── skills/framework/                              # maintainer + proposer agent skills
├── bench/tasks/<id>/                              # task sandboxes (inputs + grader)
└── runs/                                          # per-run stdout, proposals, state

CLI

Bring your own tasks

Tasks are plain JSON (tasks.json); anything auto-gradable works:

{
  "id": "spec-format1-1", "split": "train",
  "title": "Format products according to spec",
  "prompt": "Read spec.md and products.json...",
  "sandbox": {"spec.md": "...", "products.json": "..."},
  "grader": {"type": "exact", "file": "output.txt", "expected": "alpha|35|active\n..."}
}

Graders: exact, contains, json_field, code_stdout (runs the produced script). Missing deliverables score 0, never crash.

Live results so far

Honest numbers from real agent runs on the bundled bench: