sensei

Sensei

"A true master teaches not by telling, but by refining." - The Skill Sensei

Sensei automates the improvement of Agent Skills frontmatter compliance using the Ralph loop pattern - iteratively improving skills until they reach Medium-High compliance with all tests passing.

Table of Contents

• Overview
• Quick Start
• Prerequisites
• How It Works
• Configuration
• Scoring Criteria
• Examples
• Troubleshooting
• Contributing

---

Overview

The Problem

Skills without proper frontmatter lead to skill collision - agents invoking the wrong skill for a given prompt. Common issues include:

• No triggers - Agent doesn't know when to activate the skill
• No anti-triggers - Agent doesn't know when NOT to use the skill
• Brief descriptions - Not enough context for accurate matching
• Token bloat - Oversized skills waste context window

The Solution

Sensei implements the "Ralph Wiggum" technique:

1. Read - Load the skill's current state and token count
2. Score - Evaluate frontmatter compliance
3. Improve - Add triggers, anti-triggers, compatibility
4. Verify - Run tests to ensure changes work
5. Check Tokens - Analyze token usage, gather suggestions
6. Summary - Display before/after with suggestions
7. Prompt - Ask user: Commit, Create Issue, or Skip?
8. Repeat - Until target score reached

---

Quick Start

Using with Copilot CLI

Single Skill

Run sensei on my-skill-name

Single Skill (Fast Mode)

Run sensei on my-skill-name --fast

Multiple Skills

Run sensei on skill-a, skill-b, skill-c

All Low-Adherence Skills

Run sensei on all Low-adherence skills

All Skills

Run sensei on all skills

GEPA Mode (Deep Optimization)

Run sensei on my-skill-name --gepa
Run sensei score my-skill-name

Score-only mode runs without LLM calls.

Using Scripts Directly

# Count tokens in all markdown files
npm run tokens -- count

# Count tokens in specific files
npm run tokens -- count SKILL.md references/*.md

# Check files against token limits
npm run tokens -- check

# Check with strict mode (exits 1 if limits exceeded)
npm run tokens -- check --strict

# Get optimization suggestions
npm run tokens -- suggest

# Score a skill directory (advisory checks)
npm run tokens -- score .

# Compare with previous commit
npm run tokens -- compare HEAD~1

GEPA Commands

python scripts/src/gepa/auto_evaluator.py score --skill my-skill
python scripts/src/gepa/auto_evaluator.py optimize --skill my-skill

Flags

⚠️ Note: Using --fast speeds up the loop significantly but may miss issues. Consider running full tests before final commit.

---

Prerequisites

Required

1. Node.js 18+ - For running token management scripts

   node --version
   

2. Git - For commits and comparisons

   git --version
   

Optional

1. Test Framework - Jest, pytest, or similar for trigger tests

2. Python 3.10+ and GEPA - For evolutionary optimization (pip install gepa)

Installation

Option 1: Install as Copilot CLI Skill (Recommended)

# Create the skills folder
mkdir -p "$HOME/.copilot/skills"

# Clone to your skills directory
git clone https://github.com/spboyer/sensei.git "$HOME/.copilot/skills/sensei"

# Install token CLI dependencies
cd ~/.copilot/skills/sensei/scripts && npm install

The skill is now available in Copilot CLI. Invoke with:

Run sensei on my-skill-name

Option 2: Install in Project Skills Folder

For project-specific installation:

# From your project root
mkdir -p .github/skills
git clone https://github.com/spboyer/sensei.git .github/skills/sensei

# Install dependencies
cd .github/skills/sensei/scripts && npm install

Verify Installation

# Test the token CLI
cd ~/.copilot/skills/sensei  # or your install path
npm run tokens -- check

# Should output token counts for all markdown files

---

How It Works

The Ralph Loop

┌─────────────────────────────────────────────────────────┐
│  START: User invokes "Run sensei on {skill-name}"       │
└─────────────────────┬───────────────────────────────────┘
                      ▼
┌─────────────────────────────────────────────────────────┐
│  1. READ: Load skills/{skill-name}/SKILL.md             │
│           Load tests/{skill-name}/ (if exists)          │
│           Count tokens (baseline for comparison)        │
└─────────────────────┬───────────────────────────────────┘
                      ▼
┌─────────────────────────────────────────────────────────┐
│  2. SCORE: Run rule-based compliance check              │
│     • Check description length (> 150 chars?)           │
│     • Check for trigger phrases ("USE FOR:")            │
│     • Check for anti-triggers ("DO NOT USE FOR:")       │
│     • Check for compatibility field                     │
└─────────────────────┬───────────────────────────────────┘
                      ▼
              ┌───────────────┐
              │ Score >= M-H  │──YES──▶ COMPLETE ✓
              │ AND tests pass│        (next skill)
              └───────┬───────┘
                      │ NO
                      ▼
┌─────────────────────────────────────────────────────────┐
│  3. SCAFFOLD: If tests/{skill-name}/ missing:           │
│     Create tests from references/test-templates/        │
│     Creates prompts.md and framework-specific tests     │
└─────────────────────┬───────────────────────────────────┘
                      ▼
┌─────────────────────────────────────────────────────────┐
│  4. IMPROVE FRONTMATTER:                                │
│     • Add "USE FOR:" with trigger phrases               │
│     • Add "DO NOT USE FOR:" with anti-triggers          │
│     • Add compatibility if applicable                   │
│     • Keep description under 1024 chars                 │
│     • OR with --gepa: GEPA evolutionary optimization    │
└─────────────────────┬───────────────────────────────────┘
                      ▼
┌─────────────────────────────────────────────────────────┐
│  5. IMPROVE TESTS:                                      │
│     • Update shouldTriggerPrompts (5+ prompts)          │
│     • Update shouldNotTriggerPrompts (5+ prompts)       │
│     • Match prompts to new frontmatter triggers         │
└─────────────────────┬───────────────────────────────────┘
                      ▼
┌─────────────────────────────────────────────────────────┐
│  6. VERIFY: Run tests for the skill                     │
│     • If tests fail → fix and retry                     │
│     • If tests pass → continue                          │
└─────────────────────┬───────────────────────────────────┘
                      ▼
┌─────────────────────────────────────────────────────────┐
│  7. CHECK TOKENS:                                       │
│     npm run tokens count {skill}/SKILL.md               │
│     Verify under 500 token soft limit                   │
└─────────────────────┬───────────────────────────────────┘
                      ▼
┌─────────────────────────────────────────────────────────┐
│  8. SUMMARY: Display before/after comparison            │
│     • Score change (Low → Medium-High)                  │
│     • Token delta (+/- tokens)                          │
│     • Unimplemented suggestions                         │
└─────────────────────┬───