⏳ This skill is pending AI review.
Scores will appear once the review pipeline completes.
autoresearch
Auto-improve any OpenClaw skill using iterative prompt optimization. Inspired by Karpathy's autoresearch method. Run when asked to improve, optimize, or auto-tune any skill. Runs a loop — score → change → test → keep/revert — until target quality is reached. Fully autonomous once started.
Choose how to use this skill
You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.
1. Native installer
This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.
Do not guess an installer command or replace an existing version without reviewing the diff.
2. Complete package recommended
Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.
No complete ProSkills package is published for this listing yet.3. Prompt-only
Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.
Need only the instruction file?
Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.
No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.
// RATINGS
Not yet listed on ClawHub or SkillsMP
// README
autoresearch — OpenClaw Skill
Auto-improve any OpenClaw skill using iterative, scored prompt optimization.
Inspired by Karpathy's autoresearch method. Zero manual work required.
What it does
Your Claude skills fail silently. You don't know which ones, or why.
This skill fixes that. It:
- Runs your skill on test inputs
- Scores each output against a binary yes/no checklist
- Proposes one targeted change to the skill prompt
- Keeps the change if the score improves — reverts if it doesn't
- Repeats until 95%+ or max rounds hit
Fully autonomous once started. You define the checklist. The agent does the rest.
Real results
| Skill | Before | After | Rounds |
|---|---|---|---|
ad-machine | 77.8% | 100% | 9 |
typefully | 27.8% | 100% | 12 |
Install
npx clawhub@latest install bertuccio-admachine/autoresearch
Or clone manually:
git clone https://github.com/bertuccio-admachine/openclaw-skill-autoresearch.git
# Drop the folder into your OpenClaw skills directory
Requirements
- Python 3.9+
pip3 install anthropicANTHROPIC_API_KEYset in environment (or OpenClaw auth profile configured)
Quick Start
1. Write a checklist (my-checklist.json):
[
{"question": "Does the output include a specific number or result?"},
{"question": "Is it free of buzzwords like 'revolutionary' or 'synergy'?"},
{"question": "Does it include a clear, specific call to action?"},
{"question": "Is the output under 150 words?"}
]
3–6 questions is the sweet spot. Each must be answerable yes or no.
2. Write test inputs (my-inputs.json):
[
"Write landing page copy for an AI productivity tool",
"Write landing page copy for a DTC skincare brand"
]
3. Run it:
python3 scripts/autoresearch.py \
--skill /path/to/your/SKILL.md \
--checklist my-checklist.json \
--inputs my-inputs.json \
--max-rounds 20 \
--target 0.95 \
--dashboard
Walk away. Check back when it's done.
Or just say it
With the skill installed in OpenClaw, you can just tell your agent:
"Run autoresearch on my landing-page skill"
The agent will ask for your checklist criteria, run the loop, and report back.
Live Dashboard
Pass --dashboard to get a live HTML dashboard that auto-refreshes every 8 seconds:
- Score chart over time
- Per-check pass/fail breakdown
- Full change log with keep/revert status
Outputs
| File | What it is |
|---|---|
improved-SKILL.md | Best version found (saved as you go) |
backup-SKILL.md | Original, untouched |
autoresearch-log.json | Full history: every round, score, change, decision |
autoresearch-dashboard/ | Live HTML dashboard |
The log file is the most valuable artifact. Every failure pattern becomes a permanent rule. Hand it to a future model and it picks up exactly where this one left off.
How to write a good checklist
Good questions:
- Binary — pass or fail, no "sort of"
- Specific — "mentions a number" not "is concrete"
- Independent — each checks exactly one thing
Bad questions:
- "Is the output high quality?" (too vague)
- "Is the tone right?" (too subjective)
- "Is it good?" (useless)
The tighter the checklist, the cleaner the output. Constraints do the work.
How it works
loop:
run skill on all test inputs
score each output against checklist ← Claude as judge
average scores across inputs
propose ONE targeted change ← Claude as improver
run again with new skill
if score improved → keep change
else → revert
repeat until target or max rounds
The scoring uses Claude itself as an evaluator — fast, consistent, and binary. The improvement step is also Claude, with full context of what's been tried and what's worked.
Example output
⚙️ Autoresearch: landing-page
Checklist: 5 checks | Inputs: 3 | Target: 95%
📊 Baseline: 56.0%
✗ Does the headline include a specific number: 0%
✗ Is it free of buzzwords: 33%
✓ Does it include a CTA: 100%
✗ Does the first line address a pain point: 67%
✓ Is it under 150 words: 100%
🔄 Round 1: Added banned buzzwords list → 72.0% ✓ KEPT
🔄 Round 2: Added headline rule with number requirement → 88.0% ✓ KEPT
🔄 Round 3: Added worked example → 92.0% ✓ KEPT
🔄 Round 4: Tighter word count → 84.0% ✗ reverted
✅ Done — 92.0% (was 56.0%) in 4 rounds
License
MIT — use it, fork it, improve it.
Built by Bertuccio for the Ad Machine team.
Inspired by Andrej Karpathy's autoresearch method.
// HOW IT'S BUILT
// FILES
9 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.
SKILL.md3.2 KB63a6e27a72e3upstream ↗
--- name: autoresearch description: Auto-improve any OpenClaw skill using iterative prompt optimization. Inspired by Karpathy's autoresearch method. Run when asked to improve, optimize, or auto-tune any skill. Runs a loop — score → change → test → keep/revert — until target quality is reached. Fully autonomous once started. --- # Autoresearch Skill Auto-improve any skill file in this workspace using iterative, scored prompt optimization. ## The Method (Karpathy's Autoresearch) 1. Define what "good" looks like as a binary checklist (yes/no questions) 2. Run the skill on test inputs 3. Score each output against the checklist 4. Propose ONE targeted change to the skill prompt 5. Test again — keep if score improved, revert if not 6. Repeat until 95%+ or max rounds hit ## Quick Start ```bash cd /Users/jacopo/.openclaw/workspace-bertuccio/skills/autoresearch python3 scripts/autoresearch.py \ --skill /path/to/SKILL.md \ --checklist inputs/my-checklist.json \ --inputs inputs/my-inputs.json \ --max-rounds 30 \ --target 0.95 \ --dashboard \ --out /path/to/improved-SKILL.md ``` ## Input Formats ### checklist.json ```json [ {"question": "Does the output include a specific number or result? (not vague promises)"}, {"question": "Is the output free of buzzwords like 'revolutionary', 'cutting-edge', 'synergy'?"}, {"question": "Does the output include a clear, specific call to action?"}, {"question": "Is the output under 200 words?"}, {"question": "Does the first line address a specific pain point?"} ] ``` 3-6 questions is the sweet spot. Too many and the skill overfits. ### inputs.json ```json [ "Write landing page copy for an AI productivity tool for freelancers", "Write landing page copy for a B2B SaaS that automates invoicing", "Write landing page copy for a fitness app targeting busy parents" ] ``` ## Dashboard Pass `--dashboard` to get a live HTML dashboard at: `skills/autoresearch/autoresearch-dashboard/index.html` Auto-refreshes every 8 seconds. Shows: - Current score + history chart - Per-check pass/fail breakdown - Change log with keep/revert status ## Outputs | File | Description | |------|-------------| | `improved-SKILL.md` | Best version found (saves as you go) | | `backup-SKILL.md` | Original skill, untouched | | `autoresearch-log.json` | Full history: every round, score, change, keep/revert | | `autoresearch-dashboard/` | Live HTML dashboard | ## Recommended Checklist Questions Good checklist questions are: - **Binary** — pass or fail, not "sort of" - **Specific** — "mentions a number" not "is concrete" - **Independent** — each checks one thing Bad questions: - "Is the output high quality?" (too vague) - "Is the tone right?" (too subjective) - "Is it good?" (useless) ## Requirements - `ANTHROPIC_API_KEY` set in environment - Python 3.9+ - `pip3 install anthropic` ## Notes - The improved skill is saved separately — original is always backed up - The agent uses Claude itself to both evaluate outputs and propose changes - Changes that hurt the score are automatically reverted - The changelog in `autoresearch-log.json` is the most durable artifact — hand it to a future model to continue where this one left offREADME.md4.8 KBf96c01652b6aupstream ↗
# autoresearch — OpenClaw Skill > Auto-improve any OpenClaw skill using iterative, scored prompt optimization. > Inspired by Karpathy's autoresearch method. Zero manual work required. [](https://clawhub.com) [](LICENSE) ## What it does Your Claude skills fail silently. You don't know which ones, or why. This skill fixes that. It: 1. Runs your skill on test inputs 2. Scores each output against a binary yes/no checklist 3. Proposes one targeted change to the skill prompt 4. Keeps the change if the score improves — reverts if it doesn't 5. Repeats until 95%+ or max rounds hit **Fully autonomous once started.** You define the checklist. The agent does the rest. ### Real results | Skill | Before | After | Rounds | |-------|--------|-------|--------| | `ad-machine` | 77.8% | 100% | 9 | | `typefully` | 27.8% | 100% | 12 | ## Install ```bash npx clawhub@latest install bertuccio-admachine/autoresearch ``` Or clone manually: ```bash git clone https://github.com/bertuccio-admachine/openclaw-skill-autoresearch.git # Drop the folder into your OpenClaw skills directory ``` ## Requirements - Python 3.9+ - `pip3 install anthropic` - `ANTHROPIC_API_KEY` set in environment (or OpenClaw auth profile configured) ## Quick Start **1. Write a checklist** (`my-checklist.json`): ```json [ {"question": "Does the output include a specific number or result?"}, {"question": "Is it free of buzzwords like 'revolutionary' or 'synergy'?"}, {"question": "Does it include a clear, specific call to action?"}, {"question": "Is the output under 150 words?"} ] ``` 3–6 questions is the sweet spot. Each must be answerable yes or no. **2. Write test inputs** (`my-inputs.json`): ```json [ "Write landing page copy for an AI productivity tool", "Write landing page copy for a DTC skincare brand" ] ``` **3. Run it:** ```bash python3 scripts/autoresearch.py \ --skill /path/to/your/SKILL.md \ --checklist my-checklist.json \ --inputs my-inputs.json \ --max-rounds 20 \ --target 0.95 \ --dashboard ``` Walk away. Check back when it's done. ## Or just say it With the skill installed in OpenClaw, you can just tell your agent: > "Run autoresearch on my landing-page skill" The agent will ask for your checklist criteria, run the loop, and report back. ## Live Dashboard Pass `--dashboard` to get a live HTML dashboard that auto-refreshes every 8 seconds: - Score chart over time - Per-check pass/fail breakdown - Full change log with keep/revert status ## Outputs | File | What it is | |------|-----------| | `improved-SKILL.md` | Best version found (saved as you go) | | `backup-SKILL.md` | Original, untouched | | `autoresearch-log.json` | Full history: every round, score, change, decision | | `autoresearch-dashboard/` | Live HTML dashboard | The **log file** is the most valuable artifact. Every failure pattern becomes a permanent rule. Hand it to a future model and it picks up exactly where this one left off. ## How to write a good checklist **Good questions:** - Binary — pass or fail, no "sort of" - Specific — "mentions a number" not "is concrete" - Independent — each checks exactly one thing **Bad questions:** - "Is the output high quality?" (too vague) - "Is the tone right?" (too subjective) - "Is it good?" (useless) **The tighter the checklist, the cleaner the output.** Constraints do the work. ## How it works ``` loop: run skill on all test inputs score each output against checklist ← Claude as judge average scores across inputs propose ONE targeted change ← Claude as improver run again with new skill if score improved → keep change else → revert repeat until target or max rounds ``` The scoring uses Claude itself as an evaluator — fast, consistent, and binary. The improvement step is also Claude, with full context of what's been tried and what's worked. ## Example output ``` ⚙️ Autoresearch: landing-page Checklist: 5 checks | Inputs: 3 | Target: 95% 📊 Baseline: 56.0% ✗ Does the headline include a specific number: 0% ✗ Is it free of buzzwords: 33% ✓ Does it include a CTA: 100% ✗ Does the first line address a pain point: 67% ✓ Is it under 150 words: 100% 🔄 Round 1: Added banned buzzwords list → 72.0% ✓ KEPT 🔄 Round 2: Added headline rule with number requirement → 88.0% ✓ KEPT 🔄 Round 3: Added worked example → 92.0% ✓ KEPT 🔄 Round 4: Tighter word count → 84.0% ✗ reverted ✅ Done — 92.0% (was 56.0%) in 4 rounds ``` ## License MIT — use it, fork it, improve it. --- Built by [Bertuccio](https://github.com/bertuccio-admachine) for the [Ad Machine](https://admachine.xyz) team. Inspired by [Andrej Karpathy's autoresearch method](https://x.com/karpathy).scripts/autoresearch.py18 KB2948a7b5db65upstream ↗
#!/usr/bin/env python3 """ autoresearch.py — Auto-improve any OpenClaw skill via iterative prompt optimization. Inspired by Karpathy's autoresearch method: - Run a skill with test inputs - Score each output against a binary checklist - Make one small prompt change - Keep if score improves, revert if not - Repeat until 95%+ or max_rounds reached Usage: python3 autoresearch.py \ --skill /path/to/SKILL.md \ --checklist checklist.json \ --inputs inputs.json \ [--max-rounds 30] \ [--target 0.95] \ [--dashboard] \ [--out improved-SKILL.md] """ import argparse import json import os import copy import shutil import time import datetime import re from pathlib import Path try: import anthropic except ImportError: print("ERROR: anthropic package not installed. Run: pip3 install anthropic") exit(1) def _get_api_key(): """Get API key: env var first, then OpenClaw auth profile.""" key = os.environ.get("ANTHROPIC_API_KEY") if key: return key profile_path = os.path.expanduser( "~/.openclaw/agents/bertuccio/agent/auth-profiles.json" ) if os.path.exists(profile_path): with open(profile_path) as f: d = json.load(f) key = d.get("profiles", {}).get("anthropic:default", {}).get("key", "") if key: return key raise RuntimeError( "No Anthropic API key found. Set ANTHROPIC_API_KEY or configure OpenClaw auth." ) client = anthropic.Anthropic(api_key=_get_api_key()) MODEL = "claude-haiku-4-5" # Use haiku for speed/cost in scoring loops DASHBOARD_HTML = """<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Autoresearch Dashboard</title> <style> body { font-family: system-ui, sans-serif; background: #0f0f0f; color: #e0e0e0; margin: 0; padding: 20px; } h1 { color: #fff; font-size: 1.4rem; margin-bottom: 4px; } .subtitle { color: #888; font-size: 0.85rem; margin-bottom: 24px; } .grid { display: grid; grid-template-columns: 1fr 1fr; gap: 20px; margin-bottom: 24px; } .card { background: #1a1a1a; border: 1px solid #2a2a2a; border-radius: 12px; padding: 20px; } .card h2 { font-size: 0.75rem; color: #888; text-transform: uppercase; letter-spacing: 0.1em; margin: 0 0 12px; } .big-number { font-size: 3rem; font-weight: 700; color: #fff; } .big-number.good { color: #22c55e; } .big-number.bad { color: #ef4444; } .big-number.neutral { color: #f59e0b; } .bar-container { background: #2a2a2a; border-radius: 8px; height: 8px; margin: 8px 0; overflow: hidden; } .bar { height: 100%; border-radius: 8px; background: #22c55e; transition: width 0.5s ease; } .bar.bad { background: #ef4444; } .check-row { display: flex; justify-content: space-between; align-items: center; padding: 8px 0; border-bottom: 1px solid #2a2a2a; font-size: 0.85rem; } .check-row:last-child { border-bottom: none; } .badge { padding: 2px 8px; border-radius: 20px; font-size: 0.75rem; font-weight: 600; } .badge.pass { background: #14532d; color: #86efac; } .badge.fail { background: #450a0a; color: #fca5a5; } .log-entry { font-size: 0.8rem; color: #aaa; padding: 6px 0; border-bottom: 1px solid #1f1f1f; } .log-entry .round { color: #6366f1; font-weight: 600; margin-right: 8px; } .log-entry .kept { color: #22c55e; } .log-entry .reverted { color: #ef4444; } .chart { width: 100%; height: 120px; } .full-width { grid-column: 1 / -1; } #status { color: #f59e0b; font-size: 0.85rem; margin-bottom: 12px; } .change-text { color: #c084fc; font-size: 0.8rem; margin-top: 4px; } </style> </head> <body> <h1>⚙️ Autoresearch Dashboard</h1> <div class="subtitle" id="skill-name">Skill: loading...</div> <div id="status">⏳ Initializing...</div> <div class="grid"> <div class="card"> <h2>Current Score</h2> <div class="big-number" id="current-score">—</div> <div class="bar-container"><div class="bar" id="score-bar" style="width:0%"></div></div> <div style="font-size:0.8rem; color:#666; margin-top:4px;">target: 95%</div> </div> <div class="card"> <h2>Progress</h2> <div style="display:flex; gap:20px; margin-top:8px;"> <div> <div style="font-size:0.75rem;color:#888;">Round</div> <div class="big-number neutral" id="round-num" style="font-size:2rem;">0</div> </div> <div> <div style="font-size:0.75rem;color:#888;">Best</div> <div class="big-number good" id="best-score" style="font-size:2rem;">—</div> </div> <div> <div style="font-size:0.75rem;color:#888;">Baseline</div> <div class="big-number" id="baseline-score" style="font-size:2rem;color:#888;">—</div> </div> </div> </div> <div class="card"> <h2>Checklist Breakdown</h2> <div id="checklist-breakdown">Loading...</div> </div> <div class="card"> <h2>Score History</h2> <svg class="chart" id="score-chart" viewBox="0 0 400 120" preserveAspectRatio="none"> <rect width="400" height="120" fill="transparent"/> <line x1="0" y1="100" x2="400" y2="100" stroke="#2a2a2a" stroke-width="1"/> <line x1="0" y1="24" x2="400" y2="24" stroke="#22c55e" stroke-dasharray="4" stroke-width="1" opacity="0.4"/> <text x="395" y="20" fill="#22c55e" font-size="8" text-anchor="end" opacity="0.6">95%</text> <polyline id="score-line" fill="none" stroke="#6366f1" stroke-width="2" points=""/> </svg> </div> <div class="card full-width"> <h2>Change Log</h2> <div id="change-log">No changes yet.</div> </div> </div> <script> let scores = []; async function refresh() { try { const r = await fetch('state.json?' + Date.now()); const d = await r.json(); document.getElementById('skill-name').textContent = 'Skill: ' + (d.skill_name || '?'); document.getElementById('status').textContent = d.status || 'Running...'; document.getElementById('round-num').textContent = d.round || 0; const score = Math.round((d.score || 0) * 100); const best = Math.round((d.best_score || 0) * 100); const baseline = Math.round((d.baseline || 0) * 100); document.getElementById('current-score').textContent = score + '%'; document.getElementById('current-score').className = 'big-number ' + (score >= 90 ? 'good' : score >= 70 ? 'neutral' : 'bad'); document.getElementById('score-bar').style.width = score + '%'; document.getElementById('score-bar').className = 'bar ' + (score < 50 ? 'bad' : ''); document.getElementById('best-score').textContent = best + '%'; document.getElementById('baseline-score').textContent = baseline + '%'; // Checklist const checks = d.check_scores || {}; const checkHtml = Object.entries(checks).map(([q, s]) => `<div class="check-row"><span>${q}</span><span class="badge ${s >= 0.7 ? 'pass' : 'fail'}">${Math.round(s*100)}%</span></div>` ).join(''); document.getElementById('checklist-breakdown').innerHTML = checkHtml || 'No data.'; // Chart if (d.history) { scores = d.history.map(h => h.score); const pts = scores.map((s, i) => { const x = scores.length > 1 ? (i / (scores.length - 1)) * 400 : 200; const y = 100 - s * 96; return `${x},${y}`; }).join(' '); document.getElementById('score-line').setAttribute('points', pts); } // Log const log = (d.history || []).slice().reverse().map(h => `<div class="log-entry"><span class="round">R${h.round}</span>${h.score_pct}% — <span class="${h.kept ? 'kept' : 'reverted'}">${h.kept ? '✓ kept' : '✗ reverted'}</span><div class="change-text">${h.change_summary || ''}</div></div>` ).join(''); document.getElementById('change-log').innerHTML = log || 'No changes yet.'; } catch(e) { document.getElementById('status').textContent = '⚠️ Waiting for data...'; } setTimeout(refresh, 8000); } refresh(); </script> </body> </html> """ def load_skill(path: str) -> str: return Path(path).read_text() def save_skill(path: str, content: str): Path(path).write_text(content) def score_output(output: str, checklist: list[dict]) -> tuple[float, dict]: """Ask Claude to score an output against a binary checklist. Returns (0-1 score, per-check dict).""" checks_str = "\n".join(f"{i+1}. {c['question']}" for i, c in enumerate(checklist)) prompt = f"""You are a strict quality evaluator. Score this output against each checklist item. Reply with ONLY valid JSON: a list of booleans in the same order as the checklist. Checklist: {checks_str} Output to evaluate: --- {output} --- Reply with ONLY a JSON array of booleans, e.g.: [true, false, true]""" resp = client.messages.create( model=MODEL, max_tokens=256, messages=[{"role": "user", "content": prompt}] ) text = resp.content[0].text.strip() # Extract JSON array match = re.search(r'\[.*?\]', text, re.DOTALL) if not match: return 0.0, {} results = json.loads(match.group()) per_check = {c['question'][:50]: (1.0 if r else 0.0) for c, r in zip(checklist, results)} score = sum(1 for r in results if r) / len(results) if results else 0.0 return score, per_check def run_skill(skill_content: str, test_input: str) -> str: """Run a skill against a test input and get output.""" prompt = f"""You are an AI assistant. Follow the instructions in the skill below exactly. --- SKILL START --- {skill_content} --- SKILL END --- User request: {test_input} Produce your best output now.""" resp = client.messages.create( model=MODEL, max_tokens=1024, messages=[{"role": "user", "content": prompt}] ) return resp.content[0].text.strip() def propose_change(skill_content: str, checklist: list[dict], check_scores: dict, history: list) -> tuple[str, str]: """Ask Claude to propose ONE targeted improvement to the skill.""" failing = [q for q, s in check_scores.items() if s < 0.8] history_summary = "\n".join( f"Round {h['round']}: {h['change_summary']} → {h['score_pct']}% ({'kept' if h['kept'] else 'reverted'})" for h in history[-8:] ) if history else "No history yet." checks_str = "\n".join(f"- {c['question']}" for c in checklist) failing_str = "\n".join(f"- {f}" for f in failing) if failing else "None (all passing!)" prompt = f"""You are a prompt engineering expert. Improve this skill file. FULL CHECKLIST: {checks_str} CURRENTLY FAILING CHECKS: {failing_str} RECENT CHANGE HISTORY: {history_summary} CURRENT SKILL: --- {skill_content} --- Make ONE targeted, surgical improvement to fix the most impactful failing check. - Add a specific rule, example, constraint, or banned phrase list - Do NOT rewrite the whole skill — make one focused addition or edit - Build on what's been tried before (don't repeat reverted changes) Reply with: CHANGE_SUMMARY: <one sentence describing what you changed and why> IMPROVED_SKILL: <full updated skill content>""" resp = client.messages.create( model=MODEL, max_tokens=4096, messages=[{"role": "user", "content": prompt}] ) text = resp.content[0].text summary_match = re.search(r'CHANGE_SUMMARY:\s*(.+)', text) skill_match = re.search(r'IMPROVED_SKILL:\s*\n(.*)', text, re.DOTALL) summary = summary_match.group(1).strip() if summary_match else "Unspecified change" improved = skill_match.group(1).strip() if skill_match else skill_content return improved, summary def evaluate_skill(skill_content: str, inputs: list[str], checklist: list[dict]) -> tuple[float, dict]: """Run skill on all inputs, average the scores.""" all_per_check = {} total = 0.0 for inp in inputs: output = run_skill(skill_content, inp) score, per_check = score_output(output, checklist) total += score for q, s in per_check.items(): all_per_check[q] = all_per_check.get(q, 0.0) + s n = len(inputs) avg_score = total / n avg_per_check = {q: s / n for q, s in all_per_check.items()} return avg_score, avg_per_check def write_state(state_path: str, data: dict): tmp = state_path + ".tmp" with open(tmp, "w") as f: json.dump(data, f, indent=2) os.replace(tmp, state_path) def main(): parser = argparse.ArgumentParser(description="Autoresearch: auto-improve any skill file") parser.add_argument("--skill", required=True, help="Path to SKILL.md to improve") parser.add_argument("--checklist", required=True, help="JSON file with checklist: [{question: ...}, ...]") parser.add_argument("--inputs", required=True, help="JSON file with test inputs: [\"input1\", ...]") parser.add_argument("--max-rounds", type=int, default=30, help="Maximum optimization rounds") parser.add_argument("--target", type=float, default=0.95, help="Stop at this score (0-1)") parser.add_argument("--dashboard", action="store_true", help="Write dashboard HTML + state.json") parser.add_argument("--out", help="Output path for improved skill (default: improved-SKILL.md)") args = parser.parse_args() skill_path = Path(args.skill) out_path = Path(args.out) if args.out else skill_path.parent / f"improved-{skill_path.name}" backup_path = skill_path.parent / f"backup-{skill_path.name}" dashboard_dir = skill_path.parent / "autoresearch-dashboard" state_path = str(dashboard_dir / "state.json") log_path = skill_path.parent / "autoresearch-log.json" checklist = json.loads(Path(args.checklist).read_text()) inputs = json.loads(Path(args.inputs).read_text()) skill_name = skill_path.stem print(f"\n⚙️ Autoresearch: {skill_name}") print(f" Checklist: {len(checklist)} checks | Inputs: {len(inputs)} | Max rounds: {args.max_rounds} | Target: {args.target*100:.0f}%\n") # Backup original shutil.copy(skill_path, backup_path) print(f"📦 Backup saved: {backup_path}") if args.dashboard: dashboard_dir.mkdir(exist_ok=True) (dashboard_dir / "index.html").write_text(DASHBOARD_HTML) print(f"🌐 Dashboard: open {dashboard_dir}/index.html in your browser") current_skill = load_skill(skill_path) history = [] best_skill = current_skill best_score = 0.0 consecutive_hits = 0 # Baseline print("📊 Measuring baseline...") baseline_score, baseline_checks = evaluate_skill(current_skill, inputs, checklist) best_score = baseline_score print(f" Baseline: {baseline_score*100:.1f}%") for q, s in baseline_checks.items(): print(f" {'✓' if s >= 0.8 else '✗'} {q}: {s*100:.0f}%") state = { "skill_name": skill_name, "status": f"Baseline: {baseline_score*100:.1f}%", "round": 0, "score": baseline_score, "best_score": best_score, "baseline": baseline_score, "check_scores": baseline_checks, "history": [], } if args.dashboard: write_state(state_path, state) if baseline_score >= args.target: print(f"\n✅ Already at target ({baseline_score*100:.1f}%). Nothing to do.") return # Optimization loop for round_num in range(1, args.max_rounds + 1): print(f"\n🔄 Round {round_num}/{args.max_rounds}") # Propose a change print(" → Proposing change...") try: new_skill, change_summary = propose_change(current_skill, checklist, state["check_scores"], history) except Exception as e: print(f" ⚠️ Error proposing change: {e}") continue print(f" → Change: {change_summary}") print(" → Evaluating...") try: new_score, new_checks = evaluate_skill(new_skill, inputs, checklist) except Exception as e: print(f" ⚠️ Error evaluating: {e}") continue kept = new_score >= state["score"] print(f" → Score: {state['score']*100:.1f}% → {new_score*100:.1f}% ({'✓ KEPT' if kept else '✗ reverted'})") history_entry = { "round": round_num, "score": new_score, "score_pct": f"{new_score*100:.1f}", "prev_score": state["score"], "change_summary": change_summary, "kept": kept, "timestamp": datetime.datetime.now().isoformat(), } history.append(history_entry) if kept: current_skill = new_skill state["score"] = new_score state["check_scores"] = new_checks if new_score > best_score: best_score = new_score best_skill = new_skill save_skill(out_path, best_skill) print(f" 🏆 New best! Saved to {out_path}") else: # Revert — score stays the same pass state["round"] = round_num state["best_score"] = best_score state["status"] = f"Round {round_num} — {'↑ kept' if kept else '↓ reverted'} ({new_score*100:.1f}%)" state["history"] = history if args.dashboard: write_state(state_path, state) if new_score >= args.target: consecutive_hits += 1 if consecutive_hits >= 3: print(f"\n🎉 Target reached 3x in a row! Stopping at round {round_num}.") break else: consecutive_hits = 0 # Final results print(f"\n{'='*50}") print(f"✅ Autoresearch complete") print(f" Baseline: {baseline_score*100:.1f}%") print(f" Best: {best_score*100:.1f}%") print(f" Rounds: {len(history)}") print(f" Output: {out_path}") print(f" Backup: {backup_path}") kept_changes = [h for h in history if h["kept"]] print(f"\n📋 Changes kept ({len(kept_changes)}/{len(history)}):") for h in kept_changes: print(f" R{h['round']}: {h['change_summary']}") # Save full log log_data = { "skill": str(skill_path), "baseline": baseline_score, "best_score": best_score, "rounds": len(history), "history": history, "checklist": checklist, "inputs": inputs, "completed_at": datetime.datetime.now().isoformat(), } Path(log_path).write_text(json.dumps(log_data, indent=2)) print(f" Log: {log_path}") state["status"] = f"✅ Done — {best_score*100:.1f}% (was {baseline_score*100:.1f}%)" if args.dashboard: write_state(state_path, state) if __name__ == "__main__": main().gitignore124 B973c99c164b1upstream ↗
__pycache__/ *.pyc *.pyo improved-*.md backup-*.md autoresearch-dashboard/ autoresearch-log.json autoresearch-reports/ .envinputs/ad-machine-checklist.json740 B6bfefd78755eupstream ↗
[ {"question": "Does the response include all required steps in the correct order (upload assets BEFORE creating campaign)?"}, {"question": "Does the response specify which campaign type to use (Full/Bespoke/Director/Single Asset) with clear reasoning?"}, {"question": "Does the response include the correct token cost for the chosen campaign type?"}, {"question": "Does the response include at least one concrete error-handling instruction (what to do if a step fails)?"}, {"question": "Is the response free of vague instructions like 'then proceed' or 'follow the steps' without specifics?"}, {"question": "Does the response reference the correct script or method (am.py, browser automation, or REST API) for each action?"} ]inputs/ad-machine-inputs.json550 B6c8ea3d57911upstream ↗
[ "Create a Full campaign for a skincare brand. Product image is at /tmp/moisturizer.jpg. Brief: 'Hydrating daily moisturizer for dry skin, targeting women 25-45.'", "Create a Bespoke campaign for a DTC sneaker brand. I have two style reference images and a product shot. Brief: 'Minimalist white sneakers, clean aesthetic, targeting streetwear-conscious Gen Z.'", "Create a Director campaign for a coffee subscription service. Brief: 'Premium single-origin coffee delivered monthly. Warm, artisanal feel. Target: coffee enthusiasts 30-50.'" ]inputs/typefully-checklist.json860 Bc43dd60b2472upstream ↗
[ {"question": "Does the response include the exact CLI command or API call needed (not just a description of what to do)?"}, {"question": "Does the response correctly identify which platform (Twitter/X, LinkedIn, Threads, Bluesky, Mastodon) the content is for?"}, {"question": "Does the response handle the scheduling step — either scheduling with a specific time or posting immediately — with an explicit command?"}, {"question": "Is the response free of vague instructions like 'use the skill', 'call the API', or 'follow the docs' without specifying the exact call?"}, {"question": "Does the response include the correct content format for the platform (thread array for Twitter, single string for LinkedIn, etc.)?"}, {"question": "Does the response confirm success or include instructions for verifying the post was created/scheduled?"} ]inputs/typefully-inputs.json517 B002572810fb3upstream ↗
[ "Schedule this tweet for tomorrow 9am: 'The $3,000 product photoshoot is dead. We generate 20 pro ads in 2 minutes for $59. Thread 🧵'", "Post this as a LinkedIn post immediately: 'We just shipped a feature that auto-improves AI skills using iterative scoring. 77% → 100% quality. Zero manual work.'", "Create a 3-tweet thread about Ad Machine and schedule it for Monday 8am CT: Tweet 1 about the problem (expensive photoshoots), Tweet 2 about the solution (AI ads), Tweet 3 with a CTA to try it free." ]LICENSE1.0 KBddc57de0da12upstream ↗
MIT License Copyright (c) 2026 Ad Machine Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Plain-text copy, no login: /skill-bundles/bertuccio-admachine-openclaw-skill-autoresearch.md