llm-knowledge-base
Karpathy-style LLM Knowledge Base — Ingest raw files, compile into structured wiki with backlinks, lint for consistency. No RAG, no vector DB. Just markdown + LLM librarian.
Choose how to use this skill
You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.
1. Native installer
This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.
Do not guess an installer command or replace an existing version without reviewing the diff.
2. Complete package recommended
Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.
No complete ProSkills package is published for this listing yet.3. Prompt-only
Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.
Need only the instruction file?
Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.
No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.
// RATINGS
Not yet listed on ClawHub or SkillsMP
// README
Knowledge Base Agent Skill
Karpathy's LLM Knowledge Base as an OpenClaw agent skill
An OpenClaw agent skill implementing Andrej Karpathy's LLM Knowledge Base methodology. The agent acts as an active librarian — ingesting raw materials, compiling structured wiki articles with backlinks, and self-healing inconsistencies through linting passes. No RAG, no vector DB. Just Markdown + LLM.
Attribution:
- Original concept: Andrej Karpathy on X
- Coverage: VentureBeat — Karpathy's LLM Knowledge Base
- License: Apache 2.0
OpenClaw Skill
This repo is a drop-in OpenClaw agent skill. Install it and any OpenClaw agent can run the Ingest → Compile → Lint loop as part of its workflow.
Install as Skill
# Clone into your OpenClaw skills directory
git clone https://github.com/AlexChen31337/llm-knowledge-base \
~/.openclaw/workspace/skills/knowledge-base
# Or into any OpenClaw-compatible skills dir
git clone https://github.com/AlexChen31337/llm-knowledge-base \
/path/to/skills/knowledge-base
Once installed, the SKILL.md in the repo root is automatically loaded by OpenClaw agents, giving them the full knowledge-base skill: ingest, compile, lint, search, serve, and domain forking.
What the Agent Does
When an OpenClaw agent invokes this skill, it:
- Ingests — drops URLs, PDFs, or files into
raw/viascripts/ingest.py - Compiles — synthesizes
raw/into structured wiki articles inwiki/with[[backlinks]], summaries, and anINDEX.mdnavigation hub - Lints — scans the wiki for broken links, gaps, contradictions, and duplicate articles — auto-fixing what it can
- Searches — full-text grep/FTS search over the wiki, no vector DB needed
- Serves — local web UI at
http://localhost:8000for browsing
Three-stage loop: Ingest → Compile → Lint
Philosophy
"File-over-app — you own the data. The AI is a sophisticated editor."
Traditional RAG: embed → retrieve → generate. Passive. Dumb. Lossy.
Karpathy's approach: the LLM reads everything, writes structured articles, builds backlinks, and self-heals inconsistencies. Think Wikipedia, but the editor is an LLM and the source of truth is plain Markdown.
Why this beats RAG at human scale (~100 articles, ~400K words):
| RAG / Vector DB | LLM Knowledge Base | |
|---|---|---|
| Storage | Vector embeddings | Plain Markdown files |
| Search | Approximate nearest neighbor | Full-text search (grep/FTS5) |
| Synthesis | At query time | Pre-compiled, rich articles |
| Backlinks | None | [[concept]] Obsidian-style |
| Ownership | Vendor lock-in | Files you own forever |
| Self-healing | No | Yes — linting pass |
| Fine-tuning data | Hard to use | Ready-made training data |
| Complexity | High (infra, embeddings) | Low (LLM + filesystem) |
Architecture
┌─────────────────────────────────────────────────────────────────┐
│ LLM KNOWLEDGE BASE │
│ │
│ STAGE 1: INGEST │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ URLs / PDFs / Markdown / HTML │ │
│ │ ↓ scripts/ingest.py │ │
│ │ raw/ │ │
│ │ ├── article_001.md (+ frontmatter metadata) │ │
│ │ ├── paper_002.md │ │
│ │ └── repo_003.md │ │
│ └──────────────────────────────────────────────────────────┘ │
│ ↓ │
│ STAGE 2: COMPILE │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ LLM reads raw/ → synthesizes wiki/ │ │
│ │ ↓ scripts/compile.py │ │
│ │ wiki/ │ │
│ │ ├── INDEX.md (navigation hub) │ │
│ │ ├── concepts/ │ │
│ │ │ ├── transformer.md │ │
│ │ │ └── attention.md │ │
│ │ ├── summaries/ │ │
│ │ │ └── article_001_summary.md │ │
│ │ └── topics/ │ │
│ │ └── llm_training.md │ │
│ └──────────────────────────────────────────────────────────┘ │
│ ↓ │
│ STAGE 3: LINT / HEALTH CHECK │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ LLM scans wiki for gaps, inconsistencies, new links │ │
│ │ ↓ scripts/lint.py │ │
│ │ - Fixes broken [[backlinks]] │ │
│ │ - Fills missing sections │ │
│ │ - Connects related concepts │ │
│ │ - Flags outdated info │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │
│ UTILITIES │
│ ├── scripts/search.py (FTS grep-based, no vector DB) │
│ └── scripts/serve.py (local web UI) │
└─────────────────────────────────────────────────────────────────┘
Quick Start
Standalone Use
git clone https://github.com/AlexChen31337/llm-knowledge-base
cd llm-knowledge-base
uv sync
cp config.yaml.example config.yaml
# Edit config.yaml — set your LLM provider and model
As OpenClaw Skill
git clone https://github.com/AlexChen31337/llm-knowledge-base \
~/.openclaw/workspace/skills/knowledge-base
# OpenClaw agents can now use this skill automatically
Ingest raw materials
# Ingest a URL
uv run python scripts/ingest.py url https://arxiv.org/abs/1706.03762
# Ingest a local file
uv run python scripts/ingest.py file my_paper.pdf
# Ingest a directory of files
uv run python scripts/ingest.py dir ./my_docs/
Compile the wiki
uv run python scripts/compile.py
Lint / health check
uv run python scripts/lint.py
Search
uv run python scripts/search.py "transformer attention mechanism"
Serve locally (optional)
uv run python scripts/serve.py
# Open http://localhost:8000
Project Layout
llm-knowledge-base/
├── SKILL.md # OpenClaw agent skill definition ← start here
├── README.md
├── METHODOLOGY.md # Detailed step-by-step guide
├── config.yaml.example # Copy to config.yaml and edit
├── pyproject.toml
│
├── scripts/
│ ├── ingest.py # Stage 1: raw file ingestion
│ ├── compile.py # Stage 2: LLM wiki compilation
│ ├── lint.py # Stage 3: health check & self-healing
│ ├── search.py # FTS search (no vector DB)
│ └── serve.py # Local web UI
│
├── templ
// HOW IT'S BUILT
// FILES
29 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.
SKILL.md4.1 KB032bef6557b0upstream ↗
--- name: knowledge-base description: Karpathy-style LLM Knowledge Base — Ingest raw files, compile into structured wiki with backlinks, lint for consistency. No RAG, no vector DB. Just markdown + LLM librarian. homepage: https://github.com/AlexChen31337/llm-knowledge-base metadata: {"clawdbot":{"emoji":"📚","requires":{"bins":["uv"]}}} --- # Knowledge Base Skill Karpathy's "LLM Knowledge Base" methodology as an OpenClaw skill. The LLM acts as an active librarian — ingesting raw materials, compiling structured wiki articles with backlinks, and linting for consistency. **Three-stage loop:** Ingest → Compile → Lint ## Core Philosophy - **File-over-app:** Markdown files are source of truth. No vendor lock-in. - **LLM as librarian:** Not passive search — active synthesis, writing, and self-healing. - **No RAG needed:** At human scale (~100 articles, ~400K words), structured indices + summaries are sufficient. - **Obsidian-compatible:** `[[backlinks]]` for knowledge graph navigation. ## Commands All scripts run from the knowledge-base repo. Use `uv run python` for all execution. ### Ingest Raw Materials ```bash cd /tmp/llm-knowledge-base uv run python scripts/ingest.py url "https://arxiv.org/abs/1706.03762" uv run python scripts/ingest.py file "my_paper.pdf" uv run python scripts/ingest.py dir "./my_docs/" ``` ### Compile Wiki ```bash uv run python scripts/compile.py ``` This reads all files in `raw/`, synthesizes structured articles into `wiki/` with: - Encyclopedia-style articles per concept - `[[backlinks]]` between related ideas - Summary files for each source - `INDEX.md` navigation hub ### Lint / Health Check ```bash uv run python scripts/lint.py ``` Scans wiki for: - Broken `[[backlinks]]` - Missing sections or gaps - Inconsistencies between articles - New connections to add - Outdated information Auto-fixes what it can, flags what needs human review. ### Search ```bash uv run python scripts/search.py "transformer attention mechanism" ``` Full-text search over wiki. No vector DB — just grep/FTS. ### Serve Locally ```bash uv run python scripts/serve.py # Opens http://localhost:8000 ``` ## Domain Forking To create a domain-specific knowledge base: 1. Copy this skill directory: `cp -r skills/knowledge-base skills/knowledge-base-<domain>` 2. Edit `config.yaml` — set domain-specific templates and schema 3. Edit `templates/` — customize compilation and linting prompts for the domain 4. Run the ingest→compile→lint loop with domain-specific raw materials **Example domains:** - `knowledge-base-crypto` — whitepapers, tokenomics, audit reports - `knowledge-base-codebase` — source code → architecture docs, API reference - `knowledge-base-research` — papers, courses → study wiki with spaced repetition - `knowledge-base-competitive` — competitor tracking, market intelligence ## Agent Integration When running as part of an agent workflow: 1. **During normal work:** Agent drops URLs/files into `raw/` via `ingest.py` 2. **On schedule (cron):** Run `compile.py` to synthesize new raw materials 3. **Nightly:** Run `lint.py` for health checks and self-healing 4. **On demand:** Agent queries wiki via `search.py` instead of re-researching ## Multi-Agent Pattern For compound loops with multiple agents: 1. Sub-agents write findings to `raw/` 2. Compiler agent runs `compile.py` to synthesize 3. Quality gate agent validates with `lint.py` before promotion 4. Verified wiki articles fed back as context for next agent session See `docs/multi-agent.md` in the repo for full pattern. ## Config ```yaml # config.yaml llm: provider: openai # openai | anthropic | ollama model: gpt-4o max_tokens: 4096 paths: raw_dir: "./raw" wiki_dir: "./wiki" templates_dir: "./templates" wiki: create_backlinks: true create_summaries: true create_index: true ``` ## Key Insight > *"At ~100 articles / ~400K words, LLM navigation via summaries and index files is more than sufficient. No vector DB needed."* The bottleneck is synthesis quality, not retrieval speed. Pre-compiled wiki articles are richer and more useful than on-the-fly RAG chunks.README.md11 KB0f1453bb3284upstream ↗
# Knowledge Base Agent Skill > *Karpathy's LLM Knowledge Base as an OpenClaw agent skill* [](https://github.com/clawinfra) [](LICENSE) [](https://github.com/astral-sh/uv) An [OpenClaw](https://github.com/clawinfra) agent skill implementing Andrej Karpathy's **LLM Knowledge Base** methodology. The agent acts as an **active librarian** — ingesting raw materials, compiling structured wiki articles with backlinks, and self-healing inconsistencies through linting passes. No RAG, no vector DB. Just Markdown + LLM. **Attribution:** - Original concept: [Andrej Karpathy on X](https://x.com/karpathy) - Coverage: [VentureBeat — Karpathy's LLM Knowledge Base](https://venturebeat.com) - License: Apache 2.0 --- ## OpenClaw Skill This repo is a **drop-in OpenClaw agent skill**. Install it and any OpenClaw agent can run the Ingest → Compile → Lint loop as part of its workflow. ### Install as Skill ```bash # Clone into your OpenClaw skills directory git clone https://github.com/AlexChen31337/llm-knowledge-base \ ~/.openclaw/workspace/skills/knowledge-base # Or into any OpenClaw-compatible skills dir git clone https://github.com/AlexChen31337/llm-knowledge-base \ /path/to/skills/knowledge-base ``` Once installed, the `SKILL.md` in the repo root is automatically loaded by OpenClaw agents, giving them the full knowledge-base skill: ingest, compile, lint, search, serve, and domain forking. ### What the Agent Does When an OpenClaw agent invokes this skill, it: 1. **Ingests** — drops URLs, PDFs, or files into `raw/` via `scripts/ingest.py` 2. **Compiles** — synthesizes `raw/` into structured wiki articles in `wiki/` with `[[backlinks]]`, summaries, and an `INDEX.md` navigation hub 3. **Lints** — scans the wiki for broken links, gaps, contradictions, and duplicate articles — auto-fixing what it can 4. **Searches** — full-text grep/FTS search over the wiki, no vector DB needed 5. **Serves** — local web UI at `http://localhost:8000` for browsing **Three-stage loop:** `Ingest → Compile → Lint` --- ## Philosophy > *"File-over-app — you own the data. The AI is a sophisticated editor."* Traditional RAG: embed → retrieve → generate. Passive. Dumb. Lossy. Karpathy's approach: the LLM **reads everything**, **writes structured articles**, **builds backlinks**, and **self-heals** inconsistencies. Think Wikipedia, but the editor is an LLM and the source of truth is plain Markdown. **Why this beats RAG at human scale (~100 articles, ~400K words):** | | RAG / Vector DB | LLM Knowledge Base | |---|---|---| | Storage | Vector embeddings | Plain Markdown files | | Search | Approximate nearest neighbor | Full-text search (grep/FTS5) | | Synthesis | At query time | Pre-compiled, rich articles | | Backlinks | None | `[[concept]]` Obsidian-style | | Ownership | Vendor lock-in | Files you own forever | | Self-healing | No | Yes — linting pass | | Fine-tuning data | Hard to use | Ready-made training data | | Complexity | High (infra, embeddings) | Low (LLM + filesystem) | --- ## Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ LLM KNOWLEDGE BASE │ │ │ │ STAGE 1: INGEST │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ URLs / PDFs / Markdown / HTML │ │ │ │ ↓ scripts/ingest.py │ │ │ │ raw/ │ │ │ │ ├── article_001.md (+ frontmatter metadata) │ │ │ │ ├── paper_002.md │ │ │ │ └── repo_003.md │ │ │ └──────────────────────────────────────────────────────────┘ │ │ ↓ │ │ STAGE 2: COMPILE │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ LLM reads raw/ → synthesizes wiki/ │ │ │ │ ↓ scripts/compile.py │ │ │ │ wiki/ │ │ │ │ ├── INDEX.md (navigation hub) │ │ │ │ ├── concepts/ │ │ │ │ │ ├── transformer.md │ │ │ │ │ └── attention.md │ │ │ │ ├── summaries/ │ │ │ │ │ └── article_001_summary.md │ │ │ │ └── topics/ │ │ │ │ └── llm_training.md │ │ │ └──────────────────────────────────────────────────────────┘ │ │ ↓ │ │ STAGE 3: LINT / HEALTH CHECK │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ LLM scans wiki for gaps, inconsistencies, new links │ │ │ │ ↓ scripts/lint.py │ │ │ │ - Fixes broken [[backlinks]] │ │ │ │ - Fills missing sections │ │ │ │ - Connects related concepts │ │ │ │ - Flags outdated info │ │ │ └──────────────────────────────────────────────────────────┘ │ │ │ │ UTILITIES │ │ ├── scripts/search.py (FTS grep-based, no vector DB) │ │ └── scripts/serve.py (local web UI) │ └─────────────────────────────────────────────────────────────────┘ ``` --- ## Quick Start ### Standalone Use ```bash git clone https://github.com/AlexChen31337/llm-knowledge-base cd llm-knowledge-base uv sync cp config.yaml.example config.yaml # Edit config.yaml — set your LLM provider and model ``` ### As OpenClaw Skill ```bash git clone https://github.com/AlexChen31337/llm-knowledge-base \ ~/.openclaw/workspace/skills/knowledge-base # OpenClaw agents can now use this skill automatically ``` ### Ingest raw materials ```bash # Ingest a URL uv run python scripts/ingest.py url https://arxiv.org/abs/1706.03762 # Ingest a local file uv run python scripts/ingest.py file my_paper.pdf # Ingest a directory of files uv run python scripts/ingest.py dir ./my_docs/ ``` ### Compile the wiki ```bash uv run python scripts/compile.py ``` ### Lint / health check ```bash uv run python scripts/lint.py ``` ### Search ```bash uv run python scripts/search.py "transformer attention mechanism" ``` ### Serve locally (optional) ```bash uv run python scripts/serve.py # Open http://localhost:8000 ``` --- ## Project Layout ``` llm-knowledge-base/ ├── SKILL.md # OpenClaw agent skill definition ← start here ├── README.md ├── METHODOLOGY.md # Detailed step-by-step guide ├── config.yaml.example # Copy to config.yaml and edit ├── pyproject.toml │ ├── scripts/ │ ├── ingest.py # Stage 1: raw file ingestion │ ├── compile.py # Stage 2: LLM wiki compilation │ ├── lint.py # Stage 3: health check & self-healing │ ├── search.py # FTS search (no vector DB) │ └── serve.py # Local web UI │ ├── templates/ │ ├── compile_article.md # Prompt: write encyclopedia article │ ├── compile_index.md # Prompt: build index │ ├── compile_summary.md # Prompt: summarize raw material │ ├── lint_check.md # Prompt: health check pass │ └── lint_fix.md # Prompt: self-heal issues │ ├── docs/ │ ├── architecture.md # Deep architecture docs │ └── multi-agent.md # Multi-agent swarm pattern │ ├── examples/ │ ├── raw/ # Example raw ingested files │ └── wiki/ # Example compiled wiki │ ├── raw/ # Your raw ingested materials (gitignored) └── wiki/ # Your compiled wiki (gitignored by default) ``` --- ## LLM Backends Supports OpenAI, Anthropic, and Ollama (local): ```yaml # config.yaml llm: provider: openai # openai | anthropic | ollama model: gpt-4o # model: claude-opus-4-5 # model: llama3.2 ``` --- ## Domain Forking Fork the skill for a specific domain: ```bash cp -r ~/.openclaw/workspace/skills/knowledge-base \ ~/.openclaw/workspace/skills/knowledge-base-crypto # Edit config.yaml and templates/ for your domain ``` **Example domains:** - `knowledge-base-crypto` — whitepapers, tokenomics, audit reports - `knowledge-base-codebase` — source code → architecture docs, API reference - `knowledge-base-research` — papers, courses → study wiki with spaced repetition - `knowledge-base-competitive` — competitor tracking, market intelligence --- ## Multi-Agent Pattern See [`docs/multi-agent.md`](docs/multi-agent.md) for the compound loop pattern where multiple agents dump raw outputs, a compiler organizes them, and a quality gate validates before promotion. --- ## The Scale Insight > *At ~100 articles / ~400K words, LLM navigation via summaries and index files is more than sufficient. No vector DB needed.* This is the key insight: **human-scale knowledge doesn't need vector search**. A well-structured index + summaries lets the LLM navigate efficiently. The bottleneck is synthesis quality, not retrieval speed. --- ## License Apache 2.0 — see [LICENSE](LICENSE)scripts/__init__.py0 Be69de29bb2d1upstream ↗
scripts/compile.py15 KBac27f82783b8upstream ↗
#!/usr/bin/env python3 """ compile.py — Stage 2: Compile raw/ into wiki/ using an LLM. Usage: uv run python scripts/compile.py uv run python scripts/compile.py --force # recompile everything uv run python scripts/compile.py --scope "transformers" # filter by topic """ import json import os import re import sys from datetime import datetime, timezone from pathlib import Path from typing import Optional import typer import yaml app = typer.Typer(help="Compile raw materials into a structured wiki.") # ───────────────────────────────────────────────────────────────────────────── # Config # ───────────────────────────────────────────────────────────────────────────── def load_config(config_path: str = "config.yaml") -> dict: p = Path(config_path) if p.exists(): with open(p) as f: return yaml.safe_load(f) or {} return {} def get_paths(cfg: dict): raw = Path(cfg.get("paths", {}).get("raw_dir", "raw")) wiki = Path(cfg.get("paths", {}).get("wiki_dir", "wiki")) templates = Path(cfg.get("paths", {}).get("templates_dir", "templates")) for d in [raw, wiki, wiki / "summaries", wiki / "concepts", wiki / "topics"]: d.mkdir(parents=True, exist_ok=True) return raw, wiki, templates # ───────────────────────────────────────────────────────────────────────────── # LLM Client # ───────────────────────────────────────────────────────────────────────────── class LLMClient: def __init__(self, cfg: dict): self.provider = cfg.get("llm", {}).get("provider", "openai") self.model = cfg.get("llm", {}).get("model", "gpt-4o") self.temperature = cfg.get("llm", {}).get("temperature", 0.3) self.max_tokens = cfg.get("llm", {}).get("max_tokens", 4096) self.api_base = cfg.get("llm", {}).get("api_base", "") or None def complete(self, system: str, user: str) -> str: if self.provider == "openai": return self._openai(system, user) elif self.provider == "anthropic": return self._anthropic(system, user) elif self.provider == "ollama": return self._ollama(system, user) else: raise ValueError(f"Unknown provider: {self.provider}") def _openai(self, system: str, user: str) -> str: from openai import OpenAI kwargs = {} if self.api_base: kwargs["base_url"] = self.api_base client = OpenAI(**kwargs) resp = client.chat.completions.create( model=self.model, temperature=self.temperature, max_tokens=self.max_tokens, messages=[ {"role": "system", "content": system}, {"role": "user", "content": user}, ], ) return resp.choices[0].message.content or "" def _anthropic(self, system: str, user: str) -> str: import anthropic client = anthropic.Anthropic() resp = client.messages.create( model=self.model, max_tokens=self.max_tokens, system=system, messages=[{"role": "user", "content": user}], ) return resp.content[0].text if resp.content else "" def _ollama(self, system: str, user: str) -> str: import httpx base = self.api_base or "http://localhost:11434" payload = { "model": self.model, "messages": [ {"role": "system", "content": system}, {"role": "user", "content": user}, ], "stream": False, "options": {"temperature": self.temperature}, } r = httpx.post(f"{base}/api/chat", json=payload, timeout=120) r.raise_for_status() return r.json()["message"]["content"] # ───────────────────────────────────────────────────────────────────────────── # Template loader # ───────────────────────────────────────────────────────────────────────────── def load_template(templates_dir: Path, name: str) -> str: p = templates_dir / name if p.exists(): return p.read_text(encoding="utf-8") # Fallback: inline defaults return TEMPLATE_DEFAULTS.get(name, "") TEMPLATE_DEFAULTS = { "compile_article.md": """\ You are a research librarian compiling an encyclopedia-style wiki. Given raw research material below, write a well-structured Markdown article about the KEY CONCEPT identified. Rules: - Use ATX headings (##, ###) - Use [[WikiLink]] backlinks for related concepts (Obsidian style) - Include a ## Sources section citing the source files - Include a ## Open Questions section if gaps exist - Be factual and concise — this is a reference article, not a blog post - 300-800 words unless the concept demands more Article structure: # Concept Name ## Summary One paragraph TL;DR. ## Core Idea Detailed explanation. ## Key Properties / Characteristics Bullet list. ## Related Concepts [[Concept A]] | [[Concept B]] | [[Concept C]] ## Sources - Source title — key excerpt or insight ## Open Questions - Unanswered question (if any) """, "compile_summary.md": """\ You are a research librarian. Summarize the following raw document into a concise wiki summary. Write a Markdown summary with: - Title as # heading - ## TL;DR (2-3 sentences) - ## Key Points (bullet list, max 8 points) - ## Key Concepts (comma-separated [[WikiLinks]] to concepts mentioned) - ## Source metadata (if present in frontmatter) Keep it under 400 words. Be factual and dense. """, "compile_index.md": """\ You are a research librarian. Given a list of wiki articles and their summaries, write a navigation INDEX.md. The index should have: - # Knowledge Base Index - ## Topics (group concepts by theme) - ## All Articles (A-Z list with one-line descriptions) - ## Recent Additions (last 10 by ingested_at) - ## Coverage Gaps (concepts referenced but lacking full articles) Use [[WikiLink]] style for all article references. Keep descriptions to one line each. """, } # ───────────────────────────────────────────────────────────────────────────── # Core compilation logic # ───────────────────────────────────────────────────────────────────────────── def parse_frontmatter(text: str) -> tuple[dict, str]: """Extract YAML frontmatter and body from a Markdown file.""" m = re.match(r"^---\n(.*?)\n---\n?(.*)", text, re.DOTALL) if m: try: meta = yaml.safe_load(m.group(1)) or {} except yaml.YAMLError: meta = {} return meta, m.group(2).lstrip() return {}, text def needs_recompile(source: Path, output: Path) -> bool: if not output.exists(): return True return source.stat().st_mtime > output.stat().st_mtime def summarize_raw_file(raw_file: Path, llm: LLMClient, templates_dir: Path) -> str: """Generate a wiki summary for a raw file.""" content = raw_file.read_text(encoding="utf-8", errors="replace") system = load_template(templates_dir, "compile_summary.md") user = f"Raw document to summarize:\n\n{content[:12000]}" # cap at ~12k chars return llm.complete(system, user) def extract_concepts_from_summaries(summaries: list[str]) -> list[str]: """Extract unique [[WikiLink]] concepts from summaries.""" concepts: set[str] = set() for s in summaries: for m in re.finditer(r"\[\[([^\]]+)\]\]", s): concepts.add(m.group(1).strip()) return sorted(concepts) def write_article(wiki_dir: Path, slug: str, content: str, subdir: str = "concepts") -> Path: d = wiki_dir / subdir d.mkdir(parents=True, exist_ok=True) path = d / f"{slug}.md" path.write_text(content, encoding="utf-8") return path def slugify(text: str) -> str: text = text.lower() text = re.sub(r"[^a-z0-9\s_-]", "", text) text = re.sub(r"[\s_-]+", "_", text).strip("_") return text[:60] or "untitled" # ───────────────────────────────────────────────────────────────────────────── # Main compile command # ───────────────────────────────────────────────────────────────────────────── @app.command() def main( force: bool = typer.Option(False, "--force", "-f", help="Recompile all files"), scope: Optional[str] = typer.Option(None, "--scope", help="Filter raw files by keyword"), config: str = typer.Option("config.yaml", "--config", "-c"), dry_run: bool = typer.Option(False, "--dry-run", help="Show what would be compiled without writing"), ): """Compile raw/ → wiki/ using LLM synthesis.""" cfg = load_config(config) raw_dir, wiki_dir, templates_dir = get_paths(cfg) compile_cfg = cfg.get("compile", {}) llm = LLMClient(cfg) typer.echo(f"🤖 LLM: {llm.provider}/{llm.model}") # Find raw files raw_files = sorted(raw_dir.glob("*.md")) if scope: raw_files = [f for f in raw_files if scope.lower() in f.read_text().lower()] typer.echo(f"📂 Found {len(raw_files)} raw files") if not raw_files: typer.echo("⚠️ No raw files found. Run: uv run python scripts/ingest.py url <URL>") raise typer.Exit(0) # ── Step 1: Summarize raw files ─────────────────────────────────────────── summaries: dict[str, str] = {} typer.echo("\n📝 Step 1/3: Generating summaries...") for raw_file in raw_files: summary_path = wiki_dir / "summaries" / f"{raw_file.stem}_summary.md" if not force and not needs_recompile(raw_file, summary_path): typer.echo(f" ↷ Skipping (up to date): {raw_file.name}") summaries[raw_file.stem] = summary_path.read_text(encoding="utf-8") continue typer.echo(f" ✍️ Summarizing: {raw_file.name}") if dry_run: summaries[raw_file.stem] = f"# {raw_file.stem}\n\n[dry run — not generated]" continue try: summary = summarize_raw_file(raw_file, llm, templates_dir) summary_path.write_text(summary, encoding="utf-8") summaries[raw_file.stem] = summary typer.echo(f" ✅ → {summary_path}") except Exception as e: typer.echo(f" ❌ Failed: {e}", err=True) # ── Step 2: Extract concepts and write articles ─────────────────────────── if compile_cfg.get("generate_concepts", True): typer.echo("\n🧩 Step 2/3: Building concept articles...") concepts = extract_concepts_from_summaries(list(summaries.values())) min_refs = compile_cfg.get("concept_min_references", 2) # Count concept references concept_counts: dict[str, int] = {} for summary in summaries.values(): for m in re.finditer(r"\[\[([^\]]+)\]\]", summary): c = m.group(1).strip() concept_counts[c] = concept_counts.get(c, 0) + 1 typer.echo(f" Found {len(concepts)} concepts ({len([c for c, n in concept_counts.items() if n >= min_refs])} meet min_references={min_refs})") for concept in concepts: if concept_counts.get(concept, 0) < min_refs: continue slug = slugify(concept) article_path = wiki_dir / "concepts" / f"{slug}.md" if not force and article_path.exists(): typer.echo(f" ↷ Skipping (exists): {concept}") continue typer.echo(f" ✍️ Writing article: {concept}") if dry_run: continue # Build context: summaries that reference this concept relevant_summaries = [ s for s in summaries.values() if f"[[{concept}]]" in s ] context = "\n\n---\n\n".join(relevant_summaries[:5]) # max 5 summaries as context system = load_template(templates_dir, "compile_article.md") user = f"Write an encyclopedia article for the concept: **{concept}**\n\nRelevant source summaries:\n\n{context[:10000]}" try: article = llm.complete(system, user) article_path.write_text(article, encoding="utf-8") typer.echo(f" ✅ → {article_path}") except Exception as e: typer.echo(f" ❌ Failed for {concept}: {e}", err=True) # ── Step 3: Build INDEX.md ──────────────────────────────────────────────── if compile_cfg.get("rebuild_index", True): typer.echo("\n📑 Step 3/3: Building INDEX.md...") if dry_run: typer.echo(" [dry run — INDEX.md not written]") else: # Collect all wiki articles for index all_articles = list((wiki_dir / "concepts").glob("*.md")) + \ list((wiki_dir / "topics").glob("*.md")) article_listing = "\n".join( f"- {a.stem}: {a.read_text()[:200].split(chr(10))[0].lstrip('#').strip()}" for a in all_articles[:80] ) system = load_template(templates_dir, "compile_index.md") user = f"Articles in wiki:\n\n{article_listing}\n\nRecently ingested files:\n{chr(10).join(f.stem for f in raw_files[-10:])}" try: index = llm.complete(system, user) index_path = wiki_dir / "INDEX.md" index_path.write_text(index, encoding="utf-8") typer.echo(f" ✅ → {index_path}") except Exception as e: typer.echo(f" ❌ Failed to build index: {e}", err=True) typer.echo(f"\n🎉 Compilation complete → {wiki_dir}/") typer.echo(f" Summaries: {len(list((wiki_dir / 'summaries').glob('*.md')))}") typer.echo(f" Concepts: {len(list((wiki_dir / 'concepts').glob('*.md')))}") typer.echo(f" Topics: {len(list((wiki_dir / 'topics').glob('*.md')))}") if __name__ == "__main__": app()scripts/compile_facts.py18 KBa6cfac986685upstream ↗
#!/usr/bin/env python3 """ compile_facts.py — Export ClawMemory facts into structured raw markdown files for the knowledge-base wiki pipeline. Facts are grouped into logical categories based on content analysis: - cron_jobs : Recurring agent tasks (cron entries) - conversations : Direct messages from Bowen - system_events : Gateway/system state events - projects : Project-related activity (detected by keywords) - agent_actions : Agent execution logs and outcomes Usage: uv run python scripts/compile_facts.py uv run python scripts/compile_facts.py --db ~/.clawmemory/memory.db uv run python scripts/compile_facts.py --out-dir domains/clawmemory/raw uv run python scripts/compile_facts.py --limit 500 # process only top N facts uv run python scripts/compile_facts.py --verbose """ import re import sqlite3 import sys from collections import defaultdict from datetime import datetime, timezone from pathlib import Path from typing import Optional import typer app = typer.Typer(help="Compile ClawMemory facts into wiki raw files.") # ───────────────────────────────────────────────────────────────────────────── # Project keyword classifier # ───────────────────────────────────────────────────────────────────────────── PROJECT_KEYWORDS = { "clawchain": ["clawchain", "claw-chain", "clawinfra", "chain"], "evoclaw": ["evoclaw", "evo-claw", "evoclaw", "self-evolving"], "alphastrike": ["alphastrike", "alpha-strike", "paper trad"], "smartshift": ["smartshift", "smart-shift", "battery", "inverter", "solar", "amber"], "autoresearch": ["autoresearch", "auto-research", "arxiv", "nemotron", "qwen"], "payhip_books": ["payhip", "gumroad", "mbd", "book draft", "wellness coach", "healthcare revolution"], "rsi_loop": ["rsi loop", "rsi-loop", "rsi shim", "health score"], "infrastructure": ["openclaw", "gateway", "clawmemory", "intelligent router", "wal replay"], "atlas_bounty": ["atlas", "bounty"], } # Cron name → project mapping CRON_TO_PROJECT = { "AlphaStrike V3": "alphastrike", "SmartShift": "smartshift", "ArxivToCode": "autoresearch", "Nightly Autoresearch": "autoresearch", "Gumroad Daily Book": "payhip_books", "MbD": "payhip_books", "RSI Loop": "rsi_loop", "RSI Shim": "rsi_loop", "atlas-bounty": "atlas_bounty", "Sentinel": "infrastructure", "Isolated Heartbeat": "infrastructure", "Daily Cron Health Check": "infrastructure", "Nightly Health Check": "infrastructure", "Midnight Agent Health Check": "infrastructure", "Intelligent Router": "infrastructure", "WAL Replay": "infrastructure", "Daily workspace backup": "infrastructure", "Midday workspace backup": "infrastructure", "Foundry": "clawchain", "Alex Daily Briefing": "conversations", "Alex Morning Mood Image": "conversations", "Comm Watchdog": "conversations", "Daily Ideas Email": "conversations", "AISWEI": "smartshift", "SmartShift Post-Sunset": "smartshift", "Midnight Alpha Research": "alphastrike", "MbD书稿": "payhip_books", "Alex — Agent Supervision": "infrastructure", } def classify_fact(content: str) -> tuple[str, str]: """ Returns (category, subcategory) for a fact. Categories: cron_jobs, conversations, system_events, agent_actions, misc Subcategories: project name or cron name """ # 1. Cron jobs — extract name cron_match = re.search(r'\[cron:[a-f0-9-]+ ([^\]]+)\]', content) if cron_match: cron_name = cron_match.group(1).strip() # Map to project project = "misc" for prefix, proj in CRON_TO_PROJECT.items(): if cron_name.startswith(prefix) or prefix.lower() in cron_name.lower(): project = proj break return "cron_jobs", project # 2. System events if re.search(r'System: \[\d{4}-\d{2}-\d{2}', content): if "gateway connected" in content.lower() or "gateway disconnect" in content.lower(): return "system_events", "gateway" if "exec completed" in content.lower() or "exec failed" in content.lower(): return "system_events", "exec_results" return "system_events", "general" # 3. Conversation messages from Bowen if "Conversation info" in content or ('"sender": "Bowen Li"' in content and "sender_id" in content): return "conversations", "bowen_messages" # 4. Queued messages if "Queued messages while agent was busy" in content: return "conversations", "queued_messages" # 5. ClawMemory context injections if "[ClawMemory — relevant context]" in content: return "agent_actions", "clawmemory_context" # 6. Task completion / continuation if "Continue where you left off" in content or "The previous model attempt failed" in content: return "agent_actions", "task_continuations" # 7. OpenClaw runtime context if "OpenClaw runtime context" in content: return "system_events", "runtime_context" # 8. Project keyword matching content_lower = content.lower() for project, keywords in PROJECT_KEYWORDS.items(): if any(kw in content_lower for kw in keywords): return "agent_actions", project return "misc", "general" def extract_cron_summary(content: str) -> str: """Extract the meaningful part of a cron conversation.""" # Try to get just the assistant response portion if "Assistant:" in content: parts = content.split("Assistant:", 1) if len(parts) > 1: return parts[1].strip()[:800] # Try to get the task description from the user part if "User:" in content: user_part = content.split("User:", 1)[-1] # Remove cron header, get first meaningful lines lines = [l for l in user_part.split("\n") if l.strip() and not l.startswith("[cron:")] return " ".join(lines[:5])[:400] return content[:400] def format_timestamp(ts: Optional[int]) -> str: if ts is None: return "unknown" try: # ClawMemory stores timestamps in milliseconds if ts > 1e12: ts = ts / 1000 return datetime.fromtimestamp(ts, tz=timezone.utc).strftime("%Y-%m-%d %H:%M UTC") except Exception: return str(ts) # ───────────────────────────────────────────────────────────────────────────── # Main # ───────────────────────────────────────────────────────────────────────────── @app.command() def main( db: str = typer.Option( "~/.clawmemory/memory.db", "--db", "-d", help="Path to ClawMemory SQLite database", ), out_dir: str = typer.Option( "domains/clawmemory/raw", "--out-dir", "-o", help="Output directory for raw markdown files", ), limit: Optional[int] = typer.Option( None, "--limit", "-n", help="Max facts to process (default: all)", ), min_importance: float = typer.Option( 0.0, "--min-importance", help="Minimum importance threshold", ), verbose: bool = typer.Option(False, "--verbose", "-v"), ): """Export ClawMemory facts to structured raw markdown files.""" db_path = Path(db).expanduser() out_path = Path(out_dir) out_path.mkdir(parents=True, exist_ok=True) if not db_path.exists(): typer.echo(f"❌ Database not found: {db_path}", err=True) raise typer.Exit(1) typer.echo(f"📂 Reading facts from: {db_path}") # ── Load facts ──────────────────────────────────────────────────────────── conn = sqlite3.connect(str(db_path)) conn.row_factory = sqlite3.Row cur = conn.cursor() query = """ SELECT id, content, category, container, importance, confidence, source, created_at, updated_at FROM facts WHERE deleted = 0 AND importance >= ? ORDER BY importance DESC, created_at DESC """ params = [min_importance] if limit: query += f" LIMIT {limit}" cur.execute(query, params) rows = cur.fetchall() conn.close() typer.echo(f"✅ Loaded {len(rows)} active facts") # ── Classify facts ──────────────────────────────────────────────────────── groups: dict[str, dict[str, list]] = defaultdict(lambda: defaultdict(list)) for row in rows: cat, subcat = classify_fact(row["content"]) groups[cat][subcat].append(dict(row)) typer.echo(f"\n📊 Classification summary:") total_written = 0 for cat in sorted(groups): for subcat in sorted(groups[cat]): count = len(groups[cat][subcat]) typer.echo(f" {cat}/{subcat}: {count} facts") total_written += count # ── Write markdown files ────────────────────────────────────────────────── typer.echo(f"\n✍️ Writing raw markdown files to: {out_path}") written_files = [] for cat, subcats in groups.items(): for subcat, facts in subcats.items(): filename = f"{cat}__{subcat}.md".replace(" ", "_").replace("/", "-") filepath = out_path / filename lines = [ f"# ClawMemory: {cat.replace('_', ' ').title()} — {subcat.replace('_', ' ').title()}", "", f"**Category:** {cat} ", f"**Subcategory:** {subcat} ", f"**Facts count:** {len(facts)} ", f"**Generated:** {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}", "", "---", "", ] if cat == "cron_jobs": lines += _format_cron_section(facts, subcat, verbose) elif cat == "conversations": lines += _format_conversation_section(facts, verbose) elif cat == "system_events": lines += _format_system_events_section(facts, verbose) elif cat == "agent_actions": lines += _format_agent_actions_section(facts, subcat, verbose) else: lines += _format_generic_section(facts, verbose) filepath.write_text("\n".join(lines), encoding="utf-8") written_files.append(filepath) if verbose: typer.echo(f" ✅ {filepath.name} ({len(facts)} facts)") typer.echo(f"\n🎉 Done! Wrote {len(written_files)} raw files to {out_path}/") typer.echo(f" Total facts exported: {total_written}") typer.echo(f"\nNext step:") typer.echo(f" uv run python scripts/compile.py --config domains/clawmemory/config.yaml") return written_files # ───────────────────────────────────────────────────────────────────────────── # Formatters per category # ───────────────────────────────────────────────────────────────────────────── def _format_cron_section(facts: list, project: str, verbose: bool) -> list[str]: """Format cron job facts — deduplicate by cron name, show outcomes.""" lines = [ f"## Overview", "", f"Recurring automated tasks in the **{project.replace('_', ' ').title()}** project area.", "Each entry shows the most recent execution and its outcome.", "", ] # Group by cron name by_cron: dict[str, list] = defaultdict(list) for f in facts: m = re.search(r'\[cron:[a-f0-9-]+ ([^\]]+)\]', f["content"]) cron_name = m.group(1).strip() if m else "unknown" by_cron[cron_name].append(f) lines.append(f"## Cron Jobs ({len(by_cron)} unique)") lines.append("") for cron_name, job_facts in sorted(by_cron.items()): # Sort by updated_at descending, take most recent recent = sorted(job_facts, key=lambda x: x.get("updated_at") or 0, reverse=True)[:3] latest = recent[0] ts = format_timestamp(latest.get("updated_at") or latest.get("created_at")) lines.append(f"### {cron_name}") lines.append("") lines.append(f"- **Executions recorded:** {len(job_facts)}") lines.append(f"- **Last seen:** {ts}") # Extract assistant response if available content = latest["content"] assistant_part = "" if "Assistant:" in content: assistant_part = content.split("Assistant:", 1)[-1].strip()[:600] # Check for NO_REPLY / HEARTBEAT_OK if assistant_part.startswith("NO_REPLY"): lines.append(f"- **Last outcome:** Silent (NO_REPLY)") elif assistant_part.startswith("HEARTBEAT_OK"): lines.append(f"- **Last outcome:** Heartbeat OK") else: # Truncate and clean summary = re.sub(r'\n+', ' ', assistant_part)[:300] lines.append(f"- **Last outcome:** {summary}") lines.append("") return lines def _format_conversation_section(facts: list, verbose: bool) -> list[str]: """Format direct Bowen messages and queued conversations.""" lines = [ "## Overview", "", "Direct messages and conversations with Bowen Li, captured in ClawMemory.", "", f"## Messages ({len(facts)} entries)", "", ] # Deduplicate similar content (same message_id pattern) seen_ids = set() unique_facts = [] for f in facts: m = re.search(r'"message_id":\s*"(\d+)"', f["content"]) msg_id = m.group(1) if m else None if msg_id and msg_id in seen_ids: continue if msg_id: seen_ids.add(msg_id) unique_facts.append(f) for f in unique_facts[:50]: # Cap at 50 for readability content = f["content"] ts = format_timestamp(f.get("updated_at") or f.get("created_at")) # Extract timestamp from content ts_match = re.search(r'"timestamp":\s*"([^"]+)"', content) display_ts = ts_match.group(1) if ts_match else ts # Extract text after conversation metadata text = content # Remove JSON blocks text = re.sub(r'```json.*?```', '', text, flags=re.DOTALL) # Get meaningful lines text_lines = [l.strip() for l in text.split('\n') if l.strip() and not l.strip().startswith('{') and not l.strip().startswith('"')] clean_text = ' '.join(text_lines[:3])[:250] if clean_text: lines.append(f"- [{display_ts}] {clean_text}") return lines def _format_system_events_section(facts: list, verbose: bool) -> list[str]: """Format system/gateway events.""" lines = [ "## Overview", "", "System-level events recorded by ClawMemory: gateway connections, exec results, runtime context.", "", f"## Events ({len(facts)} entries)", "", ] # Deduplicate gateway events (only show unique state changes) seen_events: set[str] = set() for f in facts[:80]: content = f["content"] ts = format_timestamp(f.get("updated_at") or f.get("created_at")) # Extract event type and brief description event_match = re.search(r'System: \[([^\]]+)\] (.+?)(?:\n|$)', content) if event_match: event_ts = event_match.group(1) event_desc = event_match.group(2)[:150] key = event_desc[:60] if key in seen_events: continue seen_events.add(key) lines.append(f"- [{event_ts}] {event_desc}") else: # Fallback brief = content[:150].replace('\n', ' ') if brief not in seen_events: seen_events.add(brief) lines.append(f"- [{ts}] {brief}") return lines def _format_agent_actions_section(facts: list, project: str, verbose: bool) -> list[str]: """Format agent action/outcome facts.""" lines = [ "## Overview", "", f"Agent actions and outcomes related to **{project.replace('_', ' ').title()}**.", "", f"## Actions ({len(facts)} entries)", "", ] for f in facts[:60]: content = f["content"] ts = format_timestamp(f.get("updated_at") or f.get("created_at")) # Extract assistant response if available if "Assistant:" in content: assistant = content.split("Assistant:", 1)[-1].strip() summary = re.sub(r'\n+', ' ', assistant)[:300] lines.append(f"- [{ts}] **Outcome:** {summary}") else: brief = re.sub(r'\n+', ' ', content)[:200] lines.append(f"- [{ts}] {brief}") return lines def _format_generic_section(facts: list, verbose: bool) -> list[str]: lines = [ "## Facts", "", ] for f in facts: ts = format_timestamp(f.get("updated_at") or f.get("created_at")) brief = re.sub(r'\n+', ' ', f["content"])[:250] lines.append(f"- [{ts}] {brief}") return lines if __name__ == "__main__": app()scripts/deep_recall.py13 KBbbe79c95de5dupstream ↗
#!/usr/bin/env python3 """ deep_recall.py — Two-pass LLM recall over the ClawMemory wiki. Pass 1: Send the wiki INDEX.md + query to the LLM, asking it to identify the most relevant articles. Pass 2: Load those articles + summaries and send them with the query for a detailed, synthesized answer. Usage: uv run python scripts/deep_recall.py "What projects is Bowen working on?" uv run python scripts/deep_recall.py "Explain the SmartShift system" --limit 3 uv run python scripts/deep_recall.py --serve # HTTP server on port 7438 The --serve flag starts a minimal HTTP server with: POST /recall → {"query": "...", "limit": 5} → {"answer": "...", "articles_used": [...]} """ import json import os import re import sys from http.server import BaseHTTPRequestHandler, HTTPServer from pathlib import Path from typing import Optional import typer app = typer.Typer(help="Deep recall over ClawMemory wiki articles.") # ───────────────────────────────────────────────────────────────────────────── # Config # ───────────────────────────────────────────────────────────────────────────── DEFAULT_WIKI_DIR = "domains/clawmemory/wiki" DEFAULT_API_BASE = "http://localhost:8081/v1" DEFAULT_MODEL = "qwen9b-opus" DEFAULT_PORT = 7438 def llm_complete( system: str, user: str, api_base: str = DEFAULT_API_BASE, model: str = DEFAULT_MODEL, temperature: float = 0.2, max_tokens: int = 2048, ) -> str: """Call the local LLM via OpenAI-compatible API.""" from openai import OpenAI client = OpenAI( base_url=api_base, api_key=os.environ.get("OPENAI_API_KEY", "dummy-local-llm"), ) resp = client.chat.completions.create( model=model, temperature=temperature, max_tokens=max_tokens, messages=[ {"role": "system", "content": system}, {"role": "user", "content": user}, ], ) return resp.choices[0].message.content or "" # ───────────────────────────────────────────────────────────────────────────── # Wiki reader # ───────────────────────────────────────────────────────────────────────────── def load_index(wiki_dir: Path) -> str: """Load the wiki INDEX.md content.""" index_path = wiki_dir / "INDEX.md" if not index_path.exists(): raise FileNotFoundError( f"INDEX.md not found at {index_path}. " f"Run: uv run python scripts/compile.py --config domains/clawmemory/config.yaml" ) return index_path.read_text(encoding="utf-8") def list_articles(wiki_dir: Path) -> dict[str, Path]: """Return {slug: filepath} for all wiki articles (concepts + summaries + topics).""" articles: dict[str, Path] = {} for subdir in ["concepts", "summaries", "topics"]: d = wiki_dir / subdir if d.exists(): for f in d.glob("*.md"): articles[f.stem] = f return articles def load_article(path: Path, max_chars: int = 4000) -> str: """Load article content, truncated for context window.""" text = path.read_text(encoding="utf-8", errors="replace") if len(text) > max_chars: text = text[:max_chars] + "\n\n[... truncated ...]" return text # ───────────────────────────────────────────────────────────────────────────── # Two-pass recall # ───────────────────────────────────────────────────────────────────────────── def deep_recall( query: str, wiki_dir: Path, limit: int = 5, api_base: str = DEFAULT_API_BASE, model: str = DEFAULT_MODEL, verbose: bool = False, ) -> dict: """ Two-pass recall: 1. Ask LLM to pick relevant articles from INDEX.md 2. Load those articles and ask LLM for a synthesized answer """ index_content = load_index(wiki_dir) all_articles = list_articles(wiki_dir) # ── Pass 1: Article selection ───────────────────────────────────────────── select_system = f"""You are a knowledge retrieval agent. Given a user query and a wiki index, identify the {limit} most relevant article slugs that would help answer the query. Return ONLY a JSON array of article slugs, e.g.: ["smartshift", "ai_advisor", "cron_jobs__smartshift_summary"] Available article slugs: {json.dumps(sorted(all_articles.keys()))} Rules: - Return {limit} or fewer articles - Include both concept articles and relevant summaries - Prefer specific articles over generic ones - Return valid JSON only, no explanation""" select_user = f"Query: {query}\n\nWiki Index:\n\n{index_content[:3000]}" if verbose: print(f"🔍 Pass 1: Identifying relevant articles for: {query}", file=sys.stderr) raw_selection = llm_complete( select_system, select_user, api_base=api_base, model=model, temperature=0.1, max_tokens=512, ) # Parse selection — extract JSON array from response selected_slugs = _parse_slug_list(raw_selection, all_articles) if verbose: print(f"📚 Selected {len(selected_slugs)} articles: {selected_slugs}", file=sys.stderr) if not selected_slugs: # Fallback: just use all summaries selected_slugs = [s for s in all_articles if "summary" in s][:limit] if verbose: print(f"⚠️ No articles matched, falling back to summaries", file=sys.stderr) # ── Pass 2: Synthesize answer ───────────────────────────────────────────── context_parts = [] for slug in selected_slugs: if slug in all_articles: content = load_article(all_articles[slug]) context_parts.append(f"### {slug}\n\n{content}") context = "\n\n---\n\n".join(context_parts) answer_system = """You are a knowledgeable assistant with deep access to a personal knowledge base. Answer the user's question using ONLY the provided wiki articles as context. Be specific, cite facts from the articles, and be comprehensive. If the articles don't contain enough information, say what you know and what's missing. Use markdown formatting for clarity.""" answer_user = f"""Question: {query} Knowledge Base Articles: {context[:8000]}""" if verbose: print(f"🤖 Pass 2: Synthesizing answer...", file=sys.stderr) answer = llm_complete( answer_system, answer_user, api_base=api_base, model=model, temperature=0.3, max_tokens=2048, ) return { "query": query, "answer": answer, "articles_used": selected_slugs, "total_articles_available": len(all_articles), } def _parse_slug_list(raw: str, available: dict[str, Path]) -> list[str]: """Extract article slugs from LLM response (robust JSON parsing).""" # Try direct JSON parse raw = raw.strip() # Extract JSON array from response (might have markdown code fences) json_match = re.search(r'\[.*?\]', raw, re.DOTALL) if json_match: try: slugs = json.loads(json_match.group(0)) if isinstance(slugs, list): # Filter to only existing articles return [s for s in slugs if s in available] except json.JSONDecodeError: pass # Fallback: extract anything that looks like a slug found = [] for slug in available: if slug in raw: found.append(slug) return found[:10] # ───────────────────────────────────────────────────────────────────────────── # HTTP Server # ───────────────────────────────────────────────────────────────────────────── class RecallHandler(BaseHTTPRequestHandler): """Simple HTTP handler for /recall endpoint.""" wiki_dir: Path = Path(DEFAULT_WIKI_DIR) api_base: str = DEFAULT_API_BASE model: str = DEFAULT_MODEL def do_POST(self): if self.path != "/recall": self.send_error(404, "Not Found - use POST /recall") return try: length = int(self.headers.get("Content-Length", 0)) body = json.loads(self.rfile.read(length)) if length else {} except (json.JSONDecodeError, ValueError): self.send_error(400, "Invalid JSON body") return query = body.get("query", "").strip() if not query: self.send_error(400, "Missing 'query' field") return limit = body.get("limit", 5) try: result = deep_recall( query=query, wiki_dir=self.wiki_dir, limit=limit, api_base=self.api_base, model=self.model, verbose=True, ) self._json_response(200, result) except Exception as e: self._json_response(500, {"error": str(e)}) def do_GET(self): """Health check.""" if self.path == "/health": self._json_response(200, {"status": "ok", "service": "deep-recall"}) else: self.send_error(404, "Use POST /recall or GET /health") def _json_response(self, status: int, data: dict): payload = json.dumps(data, indent=2, ensure_ascii=False).encode("utf-8") self.send_response(status) self.send_header("Content-Type", "application/json; charset=utf-8") self.send_header("Content-Length", str(len(payload))) self.end_headers() self.wfile.write(payload) def log_message(self, format, *args): print(f"[deep-recall] {args[0]}", file=sys.stderr) # ───────────────────────────────────────────────────────────────────────────── # CLI # ───────────────────────────────────────────────────────────────────────────── @app.command() def main( query: Optional[str] = typer.Argument(None, help="Question to answer"), wiki_dir: str = typer.Option(DEFAULT_WIKI_DIR, "--wiki-dir", "-w"), limit: int = typer.Option(5, "--limit", "-n", help="Max articles to retrieve"), api_base: str = typer.Option(DEFAULT_API_BASE, "--api-base"), model: str = typer.Option(DEFAULT_MODEL, "--model", "-m"), serve: bool = typer.Option(False, "--serve", help="Run as HTTP server on port 7438"), port: int = typer.Option(DEFAULT_PORT, "--port", "-p"), verbose: bool = typer.Option(False, "--verbose", "-v"), ): """Two-pass deep recall over the ClawMemory wiki.""" wiki_path = Path(wiki_dir) if serve: # Verify wiki exists if not (wiki_path / "INDEX.md").exists(): typer.echo(f"❌ INDEX.md not found at {wiki_path}", err=True) raise typer.Exit(1) RecallHandler.wiki_dir = wiki_path RecallHandler.api_base = api_base RecallHandler.model = model server = HTTPServer(("0.0.0.0", port), RecallHandler) typer.echo(f"🚀 Deep Recall server running on http://0.0.0.0:{port}") typer.echo(f" POST /recall {{\"query\": \"...\", \"limit\": 5}}") typer.echo(f" GET /health") typer.echo(f" Wiki: {wiki_path} ({len(list_articles(wiki_path))} articles)") try: server.serve_forever() except KeyboardInterrupt: typer.echo("\n🛑 Server stopped.") server.server_close() return if not query: typer.echo("Usage: uv run python scripts/deep_recall.py 'Your question here'", err=True) typer.echo(" uv run python scripts/deep_recall.py --serve", err=True) raise typer.Exit(1) result = deep_recall( query=query, wiki_dir=wiki_path, limit=limit, api_base=api_base, model=model, verbose=verbose, ) # Print answer print(result["answer"]) if verbose: print(f"\n---\n📊 Articles used: {result['articles_used']}", file=sys.stderr) print(f"📚 Total available: {result['total_articles_available']}", file=sys.stderr) if __name__ == "__main__": app()scripts/ingest.py11 KBc3d29e4f830eupstream ↗
#!/usr/bin/env python3 """ ingest.py — Stage 1: Ingest raw materials into raw/ Usage: uv run python scripts/ingest.py url https://arxiv.org/abs/1706.03762 uv run python scripts/ingest.py file my_paper.pdf uv run python scripts/ingest.py dir ./my_docs/ """ import hashlib import re import sys from datetime import datetime, timezone from pathlib import Path from typing import Optional from urllib.parse import urlparse import typer import yaml app = typer.Typer(help="Ingest raw materials into the knowledge base.") # ───────────────────────────────────────────────────────────────────────────── # Config helpers # ───────────────────────────────────────────────────────────────────────────── def load_config(config_path: str = "config.yaml") -> dict: p = Path(config_path) if p.exists(): with open(p) as f: return yaml.safe_load(f) or {} return {} def get_raw_dir(cfg: dict) -> Path: raw = cfg.get("paths", {}).get("raw_dir", "raw") d = Path(raw) d.mkdir(parents=True, exist_ok=True) return d # ───────────────────────────────────────────────────────────────────────────── # Slug / filename helpers # ───────────────────────────────────────────────────────────────────────────── def slugify(text: str) -> str: """Convert a title or URL into a filesystem-safe slug.""" text = text.lower() text = re.sub(r"https?://[^\s]*", lambda m: urlparse(m.group()).path, text) text = re.sub(r"[^a-z0-9\s_-]", "", text) text = re.sub(r"[\s_-]+", "_", text).strip("_") return text[:80] or "untitled" def unique_slug(raw_dir: Path, slug: str) -> str: """Ensure slug is unique in raw_dir.""" candidate = slug i = 2 while (raw_dir / f"{candidate}.md").exists(): candidate = f"{slug}_{i}" i += 1 return candidate # ───────────────────────────────────────────────────────────────────────────── # Frontmatter builder # ───────────────────────────────────────────────────────────────────────────── def build_frontmatter( source: str, title: str, content_type: str, tags: list[str] | None = None, extra: dict | None = None, ) -> str: meta = { "source": source, "title": title, "ingested_at": datetime.now(timezone.utc).isoformat(), "type": content_type, "tags": tags or [], } if extra: meta.update(extra) lines = ["---"] for k, v in meta.items(): if isinstance(v, list): lines.append(f"{k}: {v}") else: lines.append(f"{k}: {yaml.dump(v, default_flow_style=True).strip()}") lines.append("---") return "\n".join(lines) # ───────────────────────────────────────────────────────────────────────────── # Converters # ───────────────────────────────────────────────────────────────────────────── def fetch_url(url: str, timeout: int = 30) -> tuple[str, str]: """Fetch URL and return (title, markdown_content).""" try: import httpx from markdownify import markdownify as md headers = { "User-Agent": "Mozilla/5.0 (compatible; LLMKnowledgeBase/1.0)" } r = httpx.get(url, timeout=timeout, follow_redirects=True, headers=headers) r.raise_for_status() html = r.text # Extract title title_match = re.search(r"<title[^>]*>([^<]+)</title>", html, re.IGNORECASE) title = title_match.group(1).strip() if title_match else urlparse(url).netloc # Convert HTML → Markdown content = md(html, heading_style="ATX", strip=["script", "style", "nav", "footer"]) # Clean up excessive whitespace content = re.sub(r"\n{4,}", "\n\n\n", content) return title, content except ImportError: typer.echo("⚠️ httpx/markdownify not installed. Run: uv sync", err=True) raise def convert_pdf(path: Path) -> tuple[str, str]: """Convert PDF to (title, markdown_content).""" try: import pymupdf4llm # type: ignore md_text = pymupdf4llm.to_markdown(str(path)) title = path.stem.replace("_", " ").replace("-", " ").title() return title, md_text except ImportError: typer.echo("⚠️ pymupdf4llm not installed. Run: uv sync", err=True) raise def read_markdown(path: Path) -> tuple[str, str]: """Read a Markdown/text file.""" content = path.read_text(encoding="utf-8", errors="replace") # Check for existing frontmatter title m = re.match(r"^---\n.*?^title:\s*(.+?)\s*$.*?^---", content, re.MULTILINE | re.DOTALL) if m: title = m.group(1).strip("\"'") else: # Try first heading hm = re.search(r"^#\s+(.+)", content, re.MULTILINE) title = hm.group(1).strip() if hm else path.stem.replace("_", " ").title() return title, content # ───────────────────────────────────────────────────────────────────────────── # Writer # ───────────────────────────────────────────────────────────────────────────── def write_raw( raw_dir: Path, slug: str, frontmatter: str, content: str, source_label: str, ) -> Path: out = raw_dir / f"{slug}.md" # Strip existing frontmatter from content to avoid double frontmatter content = re.sub(r"^---\n.*?^---\n?", "", content, flags=re.MULTILINE | re.DOTALL).lstrip() out.write_text(f"{frontmatter}\n\n{content}\n", encoding="utf-8") typer.echo(f"✅ Ingested → {out} (source: {source_label})") return out # ───────────────────────────────────────────────────────────────────────────── # CLI commands # ───────────────────────────────────────────────────────────────────────────── @app.command() def url( target_url: str = typer.Argument(..., help="URL to fetch and ingest"), tags: Optional[str] = typer.Option(None, "--tags", "-t", help="Comma-separated tags"), title_override: Optional[str] = typer.Option(None, "--title", help="Override auto-detected title"), config: str = typer.Option("config.yaml", "--config", "-c"), ): """Ingest a URL into raw/.""" cfg = load_config(config) raw_dir = get_raw_dir(cfg) timeout = cfg.get("ingest", {}).get("fetch_timeout", 30) tag_list = [t.strip() for t in tags.split(",")] if tags else [] typer.echo(f"🌐 Fetching {target_url} ...") title, content = fetch_url(target_url, timeout=timeout) if title_override: title = title_override slug = unique_slug(raw_dir, slugify(title)) fm = build_frontmatter(target_url, title, "web", tags=tag_list) write_raw(raw_dir, slug, fm, content, target_url) @app.command() def file( path: Path = typer.Argument(..., help="File to ingest (.md, .txt, .pdf, .html)"), tags: Optional[str] = typer.Option(None, "--tags", "-t", help="Comma-separated tags"), title_override: Optional[str] = typer.Option(None, "--title", help="Override title"), config: str = typer.Option("config.yaml", "--config", "-c"), ): """Ingest a local file into raw/.""" cfg = load_config(config) raw_dir = get_raw_dir(cfg) tag_list = [t.strip() for t in tags.split(",")] if tags else [] if not path.exists(): typer.echo(f"❌ File not found: {path}", err=True) raise typer.Exit(1) ext = path.suffix.lower() typer.echo(f"📄 Ingesting {path} ...") if ext == ".pdf": title, content = convert_pdf(path) content_type = "pdf" elif ext in (".html", ".htm"): try: from markdownify import markdownify as md content = md(path.read_text(encoding="utf-8", errors="replace"), heading_style="ATX") except ImportError: content = path.read_text(encoding="utf-8", errors="replace") title = title_override or path.stem.replace("_", " ").title() content_type = "html" elif ext in (".md", ".txt", ""): title, content = read_markdown(path) content_type = "markdown" else: # Generic: read as text content = path.read_text(encoding="utf-8", errors="replace") title = path.stem.replace("_", " ").title() content_type = "file" if title_override: title = title_override slug = unique_slug(raw_dir, slugify(title)) fm = build_frontmatter(str(path.resolve()), title, content_type, tags=tag_list) write_raw(raw_dir, slug, fm, content, str(path)) @app.command() def dir( directory: Path = typer.Argument(..., help="Directory of files to ingest"), tags: Optional[str] = typer.Option(None, "--tags", "-t", help="Comma-separated tags"), recursive: bool = typer.Option(True, "--recursive/--no-recursive"), config: str = typer.Option("config.yaml", "--config", "-c"), ): """Ingest all files in a directory.""" if not directory.is_dir(): typer.echo(f"❌ Not a directory: {directory}", err=True) raise typer.Exit(1) pattern = "**/*" if recursive else "*" supported = {".md", ".txt", ".pdf", ".html", ".htm"} files = [f for f in directory.glob(pattern) if f.is_file() and f.suffix.lower() in supported] typer.echo(f"📁 Found {len(files)} files in {directory}") for f in files: try: # Invoke file command for each ctx = typer.Context(file) file.callback(path=f, tags=tags, title_override=None, config=config) except Exception as e: typer.echo(f"⚠️ Skipped {f}: {e}", err=True) if __name__ == "__main__": app()scripts/lint.py19 KB020262a424f5upstream ↗
#!/usr/bin/env python3 """ lint.py — Stage 3: Health check and self-heal the wiki. Usage: uv run python scripts/lint.py # full lint + auto-fix uv run python scripts/lint.py --report-only # report without fixing uv run python scripts/lint.py --check backlinks # specific check """ import re import sys from datetime import datetime, timezone from pathlib import Path from typing import Optional import typer import yaml app = typer.Typer(help="Health check and self-heal the wiki.") # ───────────────────────────────────────────────────────────────────────────── # Config # ───────────────────────────────────────────────────────────────────────────── def load_config(config_path: str = "config.yaml") -> dict: p = Path(config_path) if p.exists(): with open(p) as f: return yaml.safe_load(f) or {} return {} def load_template(templates_dir: Path, name: str) -> str: p = templates_dir / name if p.exists(): return p.read_text(encoding="utf-8") return LINT_TEMPLATE_DEFAULTS.get(name, "") LINT_TEMPLATE_DEFAULTS = { "lint_check.md": """\ You are a wiki quality auditor. Analyze the following wiki articles for issues. For each issue found, output a JSON line: {"type": "broken_backlink"|"stub"|"inconsistency"|"gap"|"duplicate"|"missing_connection", "file": "filename.md", "description": "what's wrong", "severity": "high"|"medium"|"low", "auto_fixable": true|false} Types: - broken_backlink: [[Link]] references article that doesn't exist - stub: Article is too short (<100 words) and needs expansion - inconsistency: Two articles contradict each other (flag both files) - gap: Important concept is missing an article - duplicate: Two articles cover the same concept - missing_connection: Two related articles don't link to each other Only output JSON lines, one per issue. No prose. """, "lint_fix.md": """\ You are a wiki editor. Fix the following issue in a wiki article. Return the COMPLETE corrected Markdown content for the file. Make minimal changes — only fix the specific issue. Preserve all existing [[WikiLinks]], headings, and content. """, "lint_expand_stub.md": """\ You are a research librarian. The following wiki article is a stub that needs expansion. Expand it into a full encyclopedia article following this structure: # Concept Name ## Summary ## Core Idea ## Key Properties ## Related Concepts ## Sources ## Open Questions Draw on your knowledge. Keep factual and concise. """, } # ───────────────────────────────────────────────────────────────────────────── # LLM client (same as compile.py) # ───────────────────────────────────────────────────────────────────────────── class LLMClient: def __init__(self, cfg: dict): self.provider = cfg.get("llm", {}).get("provider", "openai") self.model = cfg.get("llm", {}).get("model", "gpt-4o") self.temperature = cfg.get("llm", {}).get("temperature", 0.3) self.max_tokens = cfg.get("llm", {}).get("max_tokens", 4096) self.api_base = cfg.get("llm", {}).get("api_base", "") or None def complete(self, system: str, user: str) -> str: if self.provider == "openai": from openai import OpenAI kwargs = {} if self.api_base: kwargs["base_url"] = self.api_base client = OpenAI(**kwargs) resp = client.chat.completions.create( model=self.model, temperature=self.temperature, max_tokens=self.max_tokens, messages=[ {"role": "system", "content": system}, {"role": "user", "content": user}, ], ) return resp.choices[0].message.content or "" elif self.provider == "anthropic": import anthropic client = anthropic.Anthropic() resp = client.messages.create( model=self.model, max_tokens=self.max_tokens, system=system, messages=[{"role": "user", "content": user}], ) return resp.content[0].text if resp.content else "" elif self.provider == "ollama": import httpx base = self.api_base or "http://localhost:11434" payload = { "model": self.model, "messages": [ {"role": "system", "content": system}, {"role": "user", "content": user}, ], "stream": False, } r = httpx.post(f"{base}/api/chat", json=payload, timeout=120) r.raise_for_status() return r.json()["message"]["content"] else: raise ValueError(f"Unknown provider: {self.provider}") # ───────────────────────────────────────────────────────────────────────────── # Static checks (no LLM needed) # ───────────────────────────────────────────────────────────────────────────── def get_all_wiki_slugs(wiki_dir: Path) -> set[str]: """Get all article slugs (filename without .md) in wiki.""" slugs = set() for md_file in wiki_dir.rglob("*.md"): if md_file.name != "INDEX.md": slugs.add(md_file.stem) return slugs def slugify(text: str) -> str: text = text.lower() text = re.sub(r"[^a-z0-9\s_-]", "", text) text = re.sub(r"[\s_-]+", "_", text).strip("_") return text[:60] or "untitled" def check_broken_backlinks(wiki_dir: Path) -> list[dict]: """Find [[WikiLinks]] that reference non-existent articles.""" issues = [] all_slugs = get_all_wiki_slugs(wiki_dir) # Build a map of display name → slug slug_map = {s.replace("_", " ").lower(): s for s in all_slugs} slug_map.update({s: s for s in all_slugs}) for md_file in wiki_dir.rglob("*.md"): content = md_file.read_text(encoding="utf-8", errors="replace") for m in re.finditer(r"\[\[([^\]|]+)(?:\|[^\]]*)?\]\]", content): link_text = m.group(1).strip() link_slug = slugify(link_text) link_lower = link_text.lower() if link_slug not in all_slugs and link_lower not in slug_map: issues.append({ "type": "broken_backlink", "file": str(md_file.relative_to(wiki_dir)), "description": f"[[{link_text}]] references non-existent article", "severity": "medium", "auto_fixable": True, "link_text": link_text, "link_slug": link_slug, }) return issues def check_stubs(wiki_dir: Path, stub_threshold: int = 100) -> list[dict]: """Find articles that are too short.""" issues = [] for md_file in (wiki_dir / "concepts").glob("*.md"): content = md_file.read_text(encoding="utf-8", errors="replace") word_count = len(content.split()) if word_count < stub_threshold: issues.append({ "type": "stub", "file": str(md_file.relative_to(wiki_dir)), "description": f"Article is only {word_count} words (threshold: {stub_threshold})", "severity": "low", "auto_fixable": True, "word_count": word_count, "path": md_file, }) return issues def check_missing_connections(wiki_dir: Path) -> list[dict]: """Find articles that discuss the same topic but don't link each other.""" issues = [] articles: dict[str, str] = {} for md_file in (wiki_dir / "concepts").glob("*.md"): articles[md_file.stem] = md_file.read_text(encoding="utf-8", errors="replace") # Simple heuristic: articles that mention each other's name but don't backlink names = {slug: slug.replace("_", " ") for slug in articles} for slug_a, content_a in articles.items(): for slug_b, name_b in names.items(): if slug_a == slug_b: continue # Does A mention B's name but not [[link]] it? if name_b.lower() in content_a.lower() and f"[[{name_b}]]" not in content_a and f"[[{slug_b}]]" not in content_a: issues.append({ "type": "missing_connection", "file": f"concepts/{slug_a}.md", "description": f"Mentions '{name_b}' but lacks [[{name_b}]] backlink", "severity": "low", "auto_fixable": True, }) return issues # ───────────────────────────────────────────────────────────────────────────── # LLM-powered checks # ───────────────────────────────────────────────────────────────────────────── def llm_health_check(wiki_dir: Path, llm: LLMClient, templates_dir: Path) -> list[dict]: """Use LLM to find inconsistencies, gaps, and duplicates.""" import json articles = list((wiki_dir / "concepts").glob("*.md")) if not articles: return [] # Build article digest for LLM digest_parts = [] for a in articles[:30]: # limit to 30 articles per batch content = a.read_text(encoding="utf-8", errors="replace") first_200 = " ".join(content.split()[:200]) digest_parts.append(f"FILE: {a.name}\n{first_200}") digest = "\n\n---\n\n".join(digest_parts) system = load_template(templates_dir, "lint_check.md") user = f"Wiki articles to audit:\n\n{digest}" try: response = llm.complete(system, user) issues = [] for line in response.strip().splitlines(): line = line.strip() if line.startswith("{"): try: issue = json.loads(line) issues.append(issue) except json.JSONDecodeError: pass return issues except Exception as e: typer.echo(f"⚠️ LLM health check failed: {e}", err=True) return [] # ───────────────────────────────────────────────────────────────────────────── # Fixes # ───────────────────────────────────────────────────────────────────────────── def fix_stub(issue: dict, wiki_dir: Path, llm: LLMClient, templates_dir: Path) -> bool: """Expand a stub article using LLM.""" article_path = wiki_dir / issue["file"] if not article_path.exists(): return False content = article_path.read_text(encoding="utf-8", errors="replace") system = load_template(templates_dir, "lint_expand_stub.md") user = f"Expand this stub article:\n\n{content}" try: expanded = llm.complete(system, user) article_path.write_text(expanded, encoding="utf-8") return True except Exception as e: typer.echo(f" ❌ Failed to expand stub: {e}", err=True) return False def fix_broken_backlink_create_stub(issue: dict, wiki_dir: Path) -> bool: """Create a stub article for a broken [[backlink]].""" slug = issue.get("link_slug", slugify(issue.get("link_text", ""))) if not slug: return False stub_path = wiki_dir / "concepts" / f"{slug}.md" if stub_path.exists(): return False # Already created by another fix link_text = issue.get("link_text", slug.replace("_", " ").title()) stub_content = f"# {link_text}\n\n> *Stub article — needs expansion.*\n\n## Summary\n\nArticle for **{link_text}** is pending.\n\n## See Also\n\n*No links yet.*\n" stub_path.write_text(stub_content, encoding="utf-8") return True # ───────────────────────────────────────────────────────────────────────────── # Report writer # ───────────────────────────────────────────────────────────────────────────── def write_lint_report(wiki_dir: Path, issues: list[dict], fixed: list[str], flagged: list[str]): report_path = wiki_dir / "LINT_REPORT.md" lines = [ "# Lint Report", f"\n*Generated: {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}*\n", f"**Total issues found:** {len(issues)}", f"**Auto-fixed:** {len(fixed)}", f"**Flagged for review:** {len(flagged)}\n", ] if fixed: lines.append("## Auto-Fixed\n") for f in fixed: lines.append(f"- [FIXED] {f}") lines.append("") if flagged: lines.append("## Needs Human Review\n") for f in flagged: lines.append(f"- [FLAGGED] {f}") lines.append("") if not issues: lines.append("## Result\n\n✅ No issues found. Wiki is healthy.") report_path.write_text("\n".join(lines), encoding="utf-8") return report_path # ───────────────────────────────────────────────────────────────────────────── # Main # ───────────────────────────────────────────────────────────────────────────── @app.command() def main( report_only: bool = typer.Option(False, "--report-only", help="Report without fixing"), check: Optional[str] = typer.Option(None, "--check", help="Run specific check: backlinks|stubs|connections|llm"), config: str = typer.Option("config.yaml", "--config", "-c"), ): """Health check and self-heal the wiki.""" cfg = load_config(config) wiki_dir = Path(cfg.get("paths", {}).get("wiki_dir", "wiki")) templates_dir = Path(cfg.get("paths", {}).get("templates_dir", "templates")) lint_cfg = cfg.get("lint", {}) auto_fix = lint_cfg.get("auto_fix", True) and not report_only stub_threshold = lint_cfg.get("stub_threshold_words", 100) human_review_types = set(lint_cfg.get("human_review_flags", ["contradictions", "outdated_info"])) if not wiki_dir.exists(): typer.echo("⚠️ Wiki directory not found. Run compile first.", err=True) raise typer.Exit(1) llm = LLMClient(cfg) typer.echo(f"🔍 Linting wiki at {wiki_dir}/ [auto_fix={auto_fix}]") all_issues: list[dict] = [] # ── Static checks ──────────────────────────────────────────────────────── if check in (None, "backlinks"): typer.echo(" Checking backlinks...") backlink_issues = check_broken_backlinks(wiki_dir) typer.echo(f" → {len(backlink_issues)} broken backlinks") all_issues.extend(backlink_issues) if check in (None, "stubs"): typer.echo(" Checking stubs...") stub_issues = check_stubs(wiki_dir, stub_threshold=stub_threshold) typer.echo(f" → {len(stub_issues)} stub articles") all_issues.extend(stub_issues) if check in (None, "connections"): typer.echo(" Checking missing connections...") conn_issues = check_missing_connections(wiki_dir) typer.echo(f" → {len(conn_issues)} missing connections") all_issues.extend(conn_issues) # ── LLM checks ─────────────────────────────────────────────────────────── if check in (None, "llm"): typer.echo(" Running LLM health check (inconsistencies, gaps, duplicates)...") llm_issues = llm_health_check(wiki_dir, llm, templates_dir) typer.echo(f" → {len(llm_issues)} issues from LLM check") all_issues.extend(llm_issues) if not all_issues: typer.echo("\n✅ Wiki is healthy — no issues found!") write_lint_report(wiki_dir, [], [], []) raise typer.Exit(0) typer.echo(f"\n📋 Total issues: {len(all_issues)}") # ── Apply fixes ────────────────────────────────────────────────────────── fixed: list[str] = [] flagged: list[str] = [] for issue in all_issues: issue_type = issue.get("type", "unknown") desc = issue.get("description", "") file_ref = issue.get("file", "?") needs_human = issue_type in human_review_types or not issue.get("auto_fixable", False) if needs_human or not auto_fix: flagged.append(f"{file_ref}: {desc}") typer.echo(f" 🚩 [FLAGGED] {file_ref}: {desc}") continue # Auto-fix success = False if issue_type == "broken_backlink": success = fix_broken_backlink_create_stub(issue, wiki_dir) msg = f"Created stub for [[{issue.get('link_text', '?')}]]" elif issue_type == "stub": success = fix_stub(issue, wiki_dir, llm, templates_dir) msg = f"Expanded stub: {file_ref}" else: # Generic: flag for human review flagged.append(f"{file_ref}: {desc}") typer.echo(f" 🚩 [FLAGGED] {file_ref}: {desc}") continue if success: fixed.append(msg) typer.echo(f" ✅ [FIXED] {msg}") else: flagged.append(f"{file_ref}: {desc} (fix failed)") typer.echo(f" ⚠️ [FAILED] {file_ref}: {desc}") # ── Report ─────────────────────────────────────────────────────────────── report_path = write_lint_report(wiki_dir, all_issues, fixed, flagged) typer.echo(f"\n📄 Lint report → {report_path}") typer.echo(f" Fixed: {len(fixed)} | Flagged: {len(flagged)}") if __name__ == "__main__": app()scripts/search.py4.8 KB4d501e1f942eupstream ↗
#!/usr/bin/env python3 """ search.py — Full-text search over the wiki. No vector DB. Usage: uv run python scripts/search.py "transformer attention" uv run python scripts/search.py "RLHF" --include-raw uv run python scripts/search.py "scaling laws" --max-results 5 """ import re import sys from pathlib import Path from typing import Optional import typer import yaml app = typer.Typer(help="Search the knowledge base (full-text, no vector DB).") def load_config(config_path: str = "config.yaml") -> dict: p = Path(config_path) if p.exists(): with open(p) as f: return yaml.safe_load(f) or {} return {} def highlight(text: str, query_terms: list[str]) -> str: """Bold the matched terms in output.""" for term in query_terms: pattern = re.compile(re.escape(term), re.IGNORECASE) text = pattern.sub(lambda m: f"**{m.group()}**", text) return text def search_files( directory: Path, query_terms: list[str], context_lines: int = 2, max_results: int = 20, ) -> list[dict]: """Grep-style search across Markdown files.""" results = [] for md_file in sorted(directory.rglob("*.md")): content = md_file.read_text(encoding="utf-8", errors="replace") lines = content.splitlines() matches_in_file = [] for i, line in enumerate(lines): if any(t.lower() in line.lower() for t in query_terms): # Gather context start = max(0, i - context_lines) end = min(len(lines), i + context_lines + 1) snippet = "\n".join(lines[start:end]) matches_in_file.append({ "line_num": i + 1, "snippet": snippet, }) if matches_in_file: # Score: number of term occurrences score = sum( content.lower().count(t.lower()) for t in query_terms ) # Title: first heading or filename title_match = re.search(r"^#\s+(.+)", content, re.MULTILINE) title = title_match.group(1).strip() if title_match else md_file.stem.replace("_", " ").title() results.append({ "file": md_file, "title": title, "score": score, "match_count": len(matches_in_file), "matches": matches_in_file[:3], # top 3 matches per file }) # Sort by score descending results.sort(key=lambda r: r["score"], reverse=True) return results[:max_results] @app.command() def main( query: str = typer.Argument(..., help="Search query (space-separated terms)"), include_raw: bool = typer.Option(False, "--include-raw", "-r", help="Also search raw/ files"), max_results: int = typer.Option(20, "--max-results", "-n"), context_lines: int = typer.Option(2, "--context", help="Context lines around match"), config: str = typer.Option("config.yaml", "--config", "-c"), ): """Search the knowledge base using full-text grep.""" cfg = load_config(config) search_cfg = cfg.get("search", {}) wiki_dir = Path(cfg.get("paths", {}).get("wiki_dir", "wiki")) raw_dir = Path(cfg.get("paths", {}).get("raw_dir", "raw")) if search_cfg: include_raw = include_raw or search_cfg.get("include_raw", False) max_results = search_cfg.get("max_results", max_results) context_lines = search_cfg.get("context_lines", context_lines) query_terms = [t.strip() for t in query.split() if t.strip()] if not query_terms: typer.echo("❌ Empty query", err=True) raise typer.Exit(1) typer.echo(f"🔎 Searching for: {' '.join(query_terms)}") # Search wiki dirs_to_search = [wiki_dir] if include_raw: dirs_to_search.append(raw_dir) all_results = [] for d in dirs_to_search: if d.exists(): all_results.extend(search_files(d, query_terms, context_lines, max_results)) # Deduplicate and re-sort all_results.sort(key=lambda r: r["score"], reverse=True) all_results = all_results[:max_results] if not all_results: typer.echo("No results found.") raise typer.Exit(0) typer.echo(f"\n📖 {len(all_results)} results:\n") for i, result in enumerate(all_results, 1): rel_path = result["file"].relative_to(Path.cwd()) if result["file"].is_relative_to(Path.cwd()) else result["file"] typer.echo(f"{'─' * 60}") typer.echo(f" {i}. {result['title']}") typer.echo(f" 📄 {rel_path} ({result['match_count']} matches, score: {result['score']})") for match in result["matches"]: snippet = highlight(match["snippet"], query_terms) typer.echo(f" L{match['line_num']}: {snippet[:200]}") typer.echo() if __name__ == "__main__": app()scripts/serve.py7.4 KB5c01237cc132upstream ↗
#!/usr/bin/env python3 """ serve.py — Serve the wiki as a local web UI. Usage: uv run python scripts/serve.py uv run python scripts/serve.py --port 8080 """ import re from pathlib import Path from typing import Optional import typer import yaml app = typer.Typer(help="Serve the wiki as a local web UI.") def load_config(config_path: str = "config.yaml") -> dict: p = Path(config_path) if p.exists(): with open(p) as f: return yaml.safe_load(f) or {} return {} def slugify(text: str) -> str: text = text.lower() text = re.sub(r"[^a-z0-9\s_-]", "", text) text = re.sub(r"[\s_-]+", "_", text).strip("_") return text[:60] def render_wikilinks(md_text: str) -> str: """Convert [[WikiLink]] to HTML links.""" def replace_link(m): link_text = m.group(1) slug = slugify(link_text) return f'<a href="/article/{slug}" class="wikilink">{link_text}</a>' return re.sub(r"\[\[([^\]|]+)(?:\|[^\]]*)?\]\]", replace_link, md_text) def create_app(wiki_dir: Path): """Create the FastAPI app.""" from fastapi import FastAPI, Request from fastapi.responses import HTMLResponse import markdown as md_lib fapi = FastAPI(title="LLM Knowledge Base", docs_url=None, redoc_url=None) STYLE = """ <style> body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; max-width: 900px; margin: 0 auto; padding: 20px; background: #1a1a2e; color: #e0e0e0; } a { color: #64b5f6; text-decoration: none; } a:hover { text-decoration: underline; } .wikilink { color: #81c784; font-weight: 500; } .wikilink:hover { color: #a5d6a7; } h1 { color: #fff; border-bottom: 2px solid #333; padding-bottom: 8px; } h2 { color: #ccc; } pre { background: #0d1117; padding: 12px; border-radius: 6px; overflow-x: auto; } code { background: #0d1117; padding: 2px 6px; border-radius: 3px; font-size: 0.9em; } .nav { background: #16213e; padding: 12px 20px; border-radius: 8px; margin-bottom: 20px; } .nav a { margin-right: 20px; } .article-list { list-style: none; padding: 0; } .article-list li { padding: 6px 0; border-bottom: 1px solid #222; } .search-box { width: 100%; padding: 8px 12px; background: #0d1117; border: 1px solid #333; color: #e0e0e0; border-radius: 4px; font-size: 1em; margin-bottom: 16px; } .badge { background: #333; padding: 2px 8px; border-radius: 10px; font-size: 0.75em; margin-left: 8px; } </style> """ NAV = """ <div class="nav"> <a href="/">📚 Home</a> <a href="/concepts">🧩 Concepts</a> <a href="/summaries">📝 Summaries</a> <a href="/topics">📂 Topics</a> </div> """ def html_page(title: str, body: str) -> str: return f"<html><head><title>{title} — LLM KB</title>{STYLE}</head><body>{NAV}{body}</body></html>" @fapi.get("/", response_class=HTMLResponse) async def index(): # Show INDEX.md if it exists, else list all articles index_path = wiki_dir / "INDEX.md" if index_path.exists(): content = index_path.read_text(encoding="utf-8") html = render_wikilinks(md_lib.markdown(content, extensions=["tables", "fenced_code"])) return html_page("Index", html) # Fallback: list sections sections = [] for subdir in ["concepts", "summaries", "topics"]: d = wiki_dir / subdir if d.exists(): files = sorted(d.glob("*.md")) items = "".join( f'<li><a href="/article/{f.stem}">{f.stem.replace("_", " ").title()}</a></li>' for f in files ) sections.append(f"<h2>{subdir.title()}</h2><ul class='article-list'>{items}</ul>") return html_page("Knowledge Base", "\n".join(sections) or "<p>Empty wiki. Run compile first.</p>") @fapi.get("/concepts", response_class=HTMLResponse) @fapi.get("/summaries", response_class=HTMLResponse) @fapi.get("/topics", response_class=HTMLResponse) async def section(request: Request): section_name = request.url.path.strip("/") d = wiki_dir / section_name if not d.exists(): return html_page(section_name.title(), f"<p>No {section_name} yet.</p>") files = sorted(d.glob("*.md")) items = "".join( f'<li><a href="/article/{f.stem}">{f.stem.replace("_", " ").title()}</a>' f'<span class="badge">{len(f.read_text().split())} words</span></li>' for f in files ) return html_page(section_name.title(), f"<h1>{section_name.title()}</h1><ul class='article-list'>{items}</ul>") @fapi.get("/article/{slug}", response_class=HTMLResponse) async def article(slug: str): # Search for article in all subdirectories for subdir in ["concepts", "summaries", "topics", ""]: d = wiki_dir / subdir if subdir else wiki_dir path = d / f"{slug}.md" if path.exists(): content = path.read_text(encoding="utf-8") html = render_wikilinks(md_lib.markdown(content, extensions=["tables", "fenced_code"])) title = slug.replace("_", " ").title() return html_page(title, html) return html_page("Not Found", f"<h1>404</h1><p>Article '{slug}' not found.</p>") @fapi.get("/search", response_class=HTMLResponse) async def search(q: str = ""): if not q: return html_page("Search", '<h1>Search</h1><form><input class="search-box" name="q" placeholder="Search..."></form>') query_terms = q.lower().split() results = [] for md_file in wiki_dir.rglob("*.md"): content = md_file.read_text(encoding="utf-8", errors="replace").lower() score = sum(content.count(t) for t in query_terms) if score > 0: results.append((score, md_file)) results.sort(key=lambda r: r[0], reverse=True) items = "".join( f'<li><a href="/article/{f.stem}">{f.stem.replace("_", " ").title()}</a>' f'<span class="badge">score: {s}</span></li>' for s, f in results[:20] ) body = f'<h1>Search: {q}</h1><form><input class="search-box" name="q" value="{q}" placeholder="Search..."></form>' body += f'<p>{len(results)} results</p><ul class="article-list">{items}</ul>' if items else '<p>No results.</p>' return html_page(f"Search: {q}", body) return fapi @app.command() def main( host: str = typer.Option("127.0.0.1", "--host"), port: int = typer.Option(8000, "--port", "-p"), config: str = typer.Option("config.yaml", "--config", "-c"), ): """Serve the wiki as a local web UI.""" cfg = load_config(config) wiki_dir = Path(cfg.get("paths", {}).get("wiki_dir", "wiki")) server_cfg = cfg.get("server", {}) host = server_cfg.get("host", host) port = server_cfg.get("port", port) reload = server_cfg.get("reload", True) if not wiki_dir.exists(): typer.echo("⚠️ Wiki directory not found. Run compile first.", err=True) raise typer.Exit(1) typer.echo(f"🌐 Serving wiki at http://{host}:{port}") typer.echo(f" Wiki: {wiki_dir.resolve()}") typer.echo(" Press Ctrl+C to stop.\n") import uvicorn fapi = create_app(wiki_dir) uvicorn.run(fapi, host=host, port=port) if __name__ == "__main__": app().gitignore525 B2e7cd96bfa30upstream ↗
# Knowledge base data (user-generated, not committed) # Only ignore top-level directories, not examples/ /raw/ /wiki/ /wiki-draft/ # Domain-specific raw and wiki data domains/*/raw/ domains/*/wiki/ # Python __pycache__/ *.pyc *.pyo *.egg-info/ dist/ build/ .venv/ venv/ # IDE .idea/ .vscode/ *.swp *.swo *~ # OS .DS_Store Thumbs.db # Config with secrets (use config.yaml.example as template) # config.yaml is safe to commit (no secrets by design) # Lint reports (generated) LINT_REPORT.md # uv .python-version uv.lockconfig.yaml5.3 KB6359486e2ed8upstream ↗
# LLM Knowledge Base Configuration # Copy to config.yaml and edit. Never commit real API keys. # ───────────────────────────────────────────── # LLM Provider # ───────────────────────────────────────────── llm: # Provider: openai | anthropic | ollama provider: openai # Model name for your chosen provider # OpenAI: gpt-4o | gpt-4o-mini | gpt-4-turbo # Anthropic: claude-opus-4-5 | claude-sonnet-4-5 | claude-haiku-3-5 # Ollama: llama3.2 | mistral | qwen2.5 model: gpt-4o # Temperature (0.0 = deterministic, 1.0 = creative) temperature: 0.3 # Max tokens per LLM response max_tokens: 4096 # API base URL (leave empty for defaults, or set for Ollama: http://localhost:11434) api_base: "" # API key — set via environment variable instead: # export OPENAI_API_KEY=sk-... # export ANTHROPIC_API_KEY=sk-ant-... # Never hardcode keys here. # ───────────────────────────────────────────── # Paths # ───────────────────────────────────────────── paths: # Where raw ingested files are stored raw_dir: raw # Where compiled wiki lives wiki_dir: wiki # Where draft articles go before quality gate draft_dir: wiki-draft # Template directory templates_dir: templates # ───────────────────────────────────────────── # Ingest Settings # ───────────────────────────────────────────── ingest: # Timeout in seconds for URL fetching fetch_timeout: 30 # Max file size to ingest (MB) max_file_size_mb: 50 # Strip HTML tags (True = plain text, False = keep some structure) strip_html: true # Default tags to add to all ingested files default_tags: [] # ───────────────────────────────────────────── # Compilation Settings # ───────────────────────────────────────────── compile: # Force recompile all files (ignore timestamps) force: false # Generate summaries for each raw file generate_summaries: true # Generate concept articles generate_concepts: true # Generate topic overview articles generate_topics: true # Rebuild INDEX.md rebuild_index: true # Minimum number of raw files referencing a concept before creating an article concept_min_references: 2 # Max concurrent LLM calls max_concurrency: 3 # ───────────────────────────────────────────── # Lint Settings # ───────────────────────────────────────────── lint: # Auto-fix issues (False = report only) auto_fix: true # Issues requiring human review (never auto-fixed) human_review_flags: - contradictions - outdated_info - major_restructuring # Minimum article length (words) before flagging as stub stub_threshold_words: 100 # ───────────────────────────────────────────── # Quality Gate # ───────────────────────────────────────────── quality_gate: # Enable quality gate before promoting drafts to wiki enabled: false # Model to use for validation (can be cheaper than main model) model: gpt-4o-mini # Minimum score (1-10) to pass quality gate min_score: 7.0 # Criteria to evaluate criteria: - factual_accuracy - source_coverage - backlink_completeness - clarity # ───────────────────────────────────────────── # Search Settings # ───────────────────────────────────────────── search: # Include raw/ files in search results include_raw: false # Maximum results to return max_results: 20 # Show context lines around match context_lines: 2 # ───────────────────────────────────────────── # Server Settings # ───────────────────────────────────────────── server: host: 127.0.0.1 port: 8000 # Reload on file changes (dev mode) reload: true # ───────────────────────────────────────────── # Domains (multi-domain support) # ───────────────────────────────────────────── domains: clawmemory: config: domains/clawmemory/config.yaml description: "ClawMemory facts compiled from the agent SQLite store"config.yaml.example953 B88bd5677de40upstream ↗
# LLM Knowledge Base Configuration # Copy to config.yaml and edit. Never commit real API keys. # Set API keys via environment variables: # export OPENAI_API_KEY=sk-... # export ANTHROPIC_API_KEY=sk-ant-... llm: provider: openai # openai | anthropic | ollama model: gpt-4o temperature: 0.3 max_tokens: 4096 api_base: "" # For Ollama: http://localhost:11434 paths: raw_dir: raw wiki_dir: wiki draft_dir: wiki-draft templates_dir: templates ingest: fetch_timeout: 30 max_file_size_mb: 50 compile: force: false generate_summaries: true generate_concepts: true generate_topics: true rebuild_index: true concept_min_references: 2 max_concurrency: 3 lint: auto_fix: true stub_threshold_words: 100 quality_gate: enabled: false model: gpt-4o-mini min_score: 7.0 search: include_raw: false max_results: 20 context_lines: 2 server: host: 127.0.0.1 port: 8000 reload: truedocs/architecture.md6.7 KBa21087da6fc5upstream ↗
# Architecture: LLM Knowledge Base ## System Overview The LLM Knowledge Base is a three-stage pipeline that transforms raw research materials into a structured, interlinked wiki maintained by an LLM. ``` ┌──────────┐ ┌───────────┐ ┌──────────┐ │ INGEST │────▶│ COMPILE │────▶│ LINT │ │ raw data │ │ LLM wiki │ │ self-heal│ └──────────┘ └───────────┘ └──────────┘ │ │ │ ▼ ▼ ▼ raw/ wiki/ LINT_REPORT.md *.md ├── INDEX.md ├── summaries/ ├── concepts/ └── topics/ ``` ## RAG vs. LLM Knowledge Base | Dimension | RAG / Vector DB | LLM Knowledge Base | |-----------|----------------|-------------------| | **Data format** | Embedding vectors | Plain Markdown files | | **Storage** | Pinecone, Weaviate, ChromaDB | Local filesystem | | **Search** | Approximate nearest neighbor | grep, FTS5, keyword | | **When synthesis happens** | At query time (every query) | At compile time (once) | | **Result quality** | Depends on retrieval quality | Pre-synthesized, high quality | | **Cross-references** | None | `[[WikiLinks]]` backlinks | | **Data ownership** | Vendor-dependent | You own the files | | **Self-healing** | No | Yes — lint pass | | **Inspection** | Opaque embeddings | Human-readable Markdown | | **Fine-tuning data** | Hard to extract | Ready-made training pairs | | **Infrastructure** | Vector DB + embedding model | LLM + filesystem | | **Incremental updates** | Re-embed changed docs | Re-compile changed files | | **Scale sweet spot** | 10K-1M+ documents | 50-500 articles | | **Latency** | Retrieval at each query | Pre-computed articles | | **Hallucination risk** | High (retrieval can miss) | Lower (pre-vetted synthesis) | ### When to Use Which **Use RAG when:** - You have 10K+ documents - Documents change frequently (daily) - You need semantic search across massive corpora - You don't need cross-referencing between docs **Use LLM Knowledge Base when:** - You have 50-500 focused documents - You want deep synthesis and cross-references - You want to own your data as plain files - You want the wiki to self-improve over time - You want Obsidian-compatible navigation ## Data Flow Detail ### Stage 1: Ingest (`scripts/ingest.py`) ``` Input: URL → fetch HTML → markdownify → raw/{slug}.md PDF → pymupdf4llm → raw/{slug}.md .md → copy with frontmatter → raw/{slug}.md Output: raw/ ├── attention_is_all_you_need.md (with YAML frontmatter) ├── scaling_laws_2020.md └── flash_attention_blog.md Metadata (frontmatter): --- source: https://... title: "Paper Title" ingested_at: 2025-01-15T10:30:00Z type: paper|web|pdf|markdown tags: [tag1, tag2] --- ``` ### Stage 2: Compile (`scripts/compile.py`) ``` Step 2a: Summarize For each raw/*.md → LLM generates wiki/summaries/{stem}_summary.md - TL;DR - Key Points - [[WikiLink]] concept extraction Step 2b: Concept Articles Extract unique [[concepts]] from summaries Filter by min_references threshold For each concept → LLM writes wiki/concepts/{slug}.md - Encyclopedia-style article - Backlinks to related concepts - Citations to source summaries Step 2c: Index LLM reads all articles → generates wiki/INDEX.md - Topic clusters - A-Z listing - Coverage gaps ``` ### Stage 3: Lint (`scripts/lint.py`) ``` Static Checks (no LLM): - Broken backlinks: [[Link]] → no matching article - Stubs: articles < 100 words - Missing connections: mentions concept name without [[linking]] it LLM Checks: - Inconsistencies between articles - Coverage gaps (important concepts missing) - Duplicates (two articles on same topic) Auto-Fix: - Create stub articles for broken backlinks - Expand stubs using LLM - Add missing [[backlinks]] Human Review: - Contradictions (flagged, not auto-fixed) - Outdated info (flagged) - Major restructuring (flagged) Output: wiki/LINT_REPORT.md ``` ## LLM Provider Abstraction ``` ┌─────────────┐ │ LLMClient │ ├─────────────┤ │ .complete() │──┬──▶ OpenAI API │ │ ├──▶ Anthropic API │ │ └──▶ Ollama (local) └─────────────┘ ``` The `LLMClient` class provides a unified interface. Configure in `config.yaml`: ```yaml llm: provider: openai # openai | anthropic | ollama model: gpt-4o temperature: 0.3 max_tokens: 4096 api_base: "" # Override for Ollama: http://localhost:11434 ``` ## Directory Structure ``` llm-knowledge-base/ ├── config.yaml # Configuration ├── scripts/ │ ├── ingest.py # Stage 1 │ ├── compile.py # Stage 2 │ ├── lint.py # Stage 3 │ ├── search.py # FTS search │ └── serve.py # Web UI ├── templates/ # LLM prompt templates │ ├── compile_summary.md │ ├── compile_article.md │ ├── compile_index.md │ ├── lint_check.md │ └── lint_fix.md ├── raw/ # Ingested raw materials │ └── *.md ├── wiki/ # Compiled wiki │ ├── INDEX.md │ ├── summaries/ │ ├── concepts/ │ └── topics/ └── wiki-draft/ # (optional) Pre-quality-gate ``` ## Quality Gate (Optional) ``` ┌──────────┐ wiki-draft/*.md ──▶│ Quality │──▶ wiki/*.md (promoted) │ Gate │ │ score≥7 │──▶ rejected (stays in draft) └──────────┘ ``` An independent (typically cheaper) model scores each draft on: - Factual accuracy - Source coverage - Backlink completeness - Clarity Only drafts scoring ≥ threshold are promoted to the live wiki. ## Scaling Considerations | Metric | Comfortable | Pushing It | Consider Alternatives | |--------|------------|------------|----------------------| | Articles | 50 | 200 | 500+ (split into sub-wikis) | | Total words | 100K | 400K | 1M+ (add 2-level index) | | Raw sources | 100 | 500 | 1000+ (add FTS5 index) | | Compile time | 5 min | 30 min | 2hr+ (add caching) | | LLM calls/compile | 50 | 200 | 500+ (add batching) | At ~100 articles / ~400K words, the LLM navigates via INDEX.md and summaries efficiently. No vector DB needed.docs/multi-agent.md10 KB27f73079848cupstream ↗
# Multi-Agent Knowledge Base Pattern > *The compound loop: multiple agents feeding a shared knowledge base* ## Overview The single-user knowledge base pattern scales into a powerful multi-agent system when multiple AI agents contribute raw materials and consume compiled knowledge. ``` ┌──────────────────────────────────────────────────────────────┐ │ COMPOUND KNOWLEDGE LOOP │ │ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ Agent A │ │ Agent B │ │ Agent C │ (Specialist Agents)│ │ │ Research │ │ Code │ │ Monitor │ │ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌────────────────────────────────────────┐ │ │ │ raw/ (drop zone) │ │ │ │ Agent outputs, discoveries, reports │ │ │ └───────────────────┬────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────┐ │ │ │ COMPILER (LLM Librarian) │ │ │ │ - Reads new raw materials │ │ │ │ - Updates wiki articles │ │ │ │ - Builds cross-references │ │ │ │ - Maintains INDEX.md │ │ │ └───────────────────┬────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────┐ │ │ │ QUALITY GATE (Validator) │ │ │ │ - Independent model scores drafts │ │ │ │ - Rejects low-quality content │ │ │ │ - Promotes verified articles │ │ │ └───────────────────┬────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────┐ │ │ │ wiki/ (verified) │ │ │ │ Trusted, interlinked knowledge base │ │ │ └───────────────────┬────────────────────┘ │ │ │ │ │ ┌───────────┼───────────┐ │ │ ▼ ▼ ▼ │ │ Agent A Agent B Agent C │ │ (reads wiki for context and decisions) │ └──────────────────────────────────────────────────────────────┘ ``` ## Agent Roles ### Specialist Agents (Producers) These agents do domain-specific work and dump their findings into `raw/`: | Agent | Role | Output to raw/ | |-------|------|----------------| | Research Agent | Reads papers, blogs, docs | Summaries, key findings | | Code Agent | Analyzes repos, writes code | API docs, architecture notes | | Monitor Agent | Watches feeds, news, markets | Event reports, trend data | | Interview Agent | Processes transcripts | Key quotes, insights | ### Compiler (Organizer) A dedicated LLM that: 1. Watches `raw/` for new files 2. Runs `compile.py` to update the wiki 3. Builds cross-references between new and existing articles 4. Updates INDEX.md ### Quality Gate (Validator) An independent LLM (can be a different model) that: 1. Reads each draft article 2. Scores on accuracy, coverage, clarity 3. Promotes to `wiki/` only if score ≥ threshold 4. Rejects back to `wiki-draft/` with feedback ### Consumer Agents All agents can read `wiki/` for context: - Research Agent checks existing coverage before researching - Code Agent reads architecture articles for context - Monitor Agent consults wiki to contextualize new events ## Contamination Mitigation The multi-agent pattern introduces a risk: agent-generated hallucinations can contaminate the knowledge base. ### Defense Layers ``` Layer 1: Source Tracking Every raw file has frontmatter with source and agent attribution wiki articles cite their sources via [[source: ...]] Layer 2: Quality Gate Independent model validates before promotion Catches factual errors, unsupported claims Layer 3: Vault Separation wiki/ → Verified, promoted articles (clean vault) wiki-draft/ → Agent-generated, unverified (messy vault) raw/ → Unprocessed agent outputs Layer 4: Lint Health Checks Regular passes catch inconsistencies introduced by agents Contradictions flagged for human review Layer 5: Human Review Queue High-severity issues require human sign-off Periodic human audit of random articles ``` ### The "Messy Vault" Pattern Karpathy's insight: let agents play in a **messy vault** — a workspace where hallucinations and rough drafts are fine. Only bring useful artifacts into the **clean vault** after quality checks. ``` messy vault (wiki-draft/) clean vault (wiki/) ├── rough_ideas.md ──▶ promoted_article.md ├── hallucinated_claim.md ✘ (rejected by quality gate) ├── duplicate_concept.md ✘ (merged by linter) └── good_synthesis.md ──▶ concept_article.md ``` ## Ephemeral Wiki Pattern For focused tasks, spawn a temporary mini knowledge base: ```bash # Create ephemeral wiki for a specific research sprint mkdir -p /tmp/sprint-wiki/{raw,wiki} cp config.yaml /tmp/sprint-wiki/ # Ingest sprint-specific materials uv run python scripts/ingest.py url https://paper1.com --config /tmp/sprint-wiki/config.yaml uv run python scripts/ingest.py url https://paper2.com --config /tmp/sprint-wiki/config.yaml # Compile uv run python scripts/compile.py --config /tmp/sprint-wiki/config.yaml # Use wiki for the task... # ...write your report... # Extract useful insights back to main wiki cp /tmp/sprint-wiki/wiki/concepts/useful_concept.md raw/from_sprint_useful_concept.md uv run python scripts/compile.py # recompile main wiki # Dissolve rm -rf /tmp/sprint-wiki ``` ## Implementation Pattern ### Cron-Based (Simple) ```bash # crontab # Compile every hour 0 * * * * cd /path/to/kb && uv run python scripts/compile.py 2>&1 >> /var/log/kb-compile.log # Lint daily 0 3 * * * cd /path/to/kb && uv run python scripts/lint.py 2>&1 >> /var/log/kb-lint.log ``` ### Event-Driven (Advanced) ```python # Watch raw/ for new files and auto-compile import watchdog # or inotify # On new file in raw/: # 1. Run compile.py --incremental # 2. Run quality gate on new articles # 3. Run lint.py --check backlinks ``` ### Agent Integration Example ```python # Agent code: dump findings to knowledge base from pathlib import Path from datetime import datetime def save_to_kb(title: str, content: str, agent_name: str, tags: list[str]): """Agent saves raw output to knowledge base.""" slug = title.lower().replace(" ", "_")[:60] raw_dir = Path("raw") raw_dir.mkdir(exist_ok=True) frontmatter = f"""--- source: agent:{agent_name} title: "{title}" ingested_at: {datetime.utcnow().isoformat()}Z type: agent_output tags: {tags} agent: {agent_name} ---""" (raw_dir / f"{slug}.md").write_text(f"{frontmatter}\n\n{content}") ``` ## Fine-Tuning Endpoint As the wiki matures through repeated compile + lint cycles, it becomes increasingly high-quality training data: ``` Compile Pass 1: Raw → rough summaries, basic articles Lint Pass 1: Fix stubs, broken links, gaps Compile Pass 2: Incorporate lint fixes, deepen articles Lint Pass 2: Check consistency, merge duplicates ... Pass N: High-quality, consistent, well-linked wiki ↓ Export as fine-tuning data ``` Each article becomes a training pair: - **Input**: "Write an article about {concept} given these sources: {summaries}" - **Output**: The refined wiki article This creates a domain-specific model that can answer questions without the full wiki context. ## Scaling the Multi-Agent Pattern | Scale | Pattern | Notes | |-------|---------|-------| | 1 agent | Single loop | One agent ingests, one LLM compiles | | 2-5 agents | Shared raw/ | Agents write to same raw/, one compiler | | 5-20 agents | Partitioned | Sub-wikis per domain, meta-index across | | 20+ agents | Federated | Multiple knowledge bases with cross-references | At any scale, the core principle holds: **agents produce raw materials, the compiler synthesizes, the quality gate validates, and the wiki self-improves.**domains/clawmemory/config.yaml393 Bc7d3b8ae7890upstream ↗
llm: provider: openai model: qwen9b-opus temperature: 0.3 max_tokens: 4096 api_base: "http://localhost:8081/v1" paths: raw_dir: domains/clawmemory/raw wiki_dir: domains/clawmemory/wiki compilation: batch_size: 3 force: false generate_concepts: true concept_min_references: 1 rebuild_index: true lint: auto_fix: true check_backlinks: false check_sections: trueexamples/raw/attention_is_all_you_need.md2.0 KB61c48db4d8a0upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
examples/raw/scaling_laws_for_neural_lms.md1.8 KB629e1625a0d8upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
examples/wiki/INDEX.md1.5 KBb99a6cbf47a4upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
examples/wiki/self_attention.md2.0 KB88bb8c9edc4cupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
examples/wiki/transformer_architecture.md2.2 KB8c54f8ce2767upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
LICENSE10 KBc8002a2eeddfupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
METHODOLOGY.md8.7 KB7e01a8b8e002upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
pyproject.toml922 B4c63c93c8660upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
templates/compile_article.md968 Bc7bad67c2a05upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
templates/compile_index.md655 B5965eb0a7126upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
templates/compile_summary.md612 Ba393ada30eeeupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
templates/lint_check.md972 Bfd002efa0740upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
templates/lint_fix.md400 Bbbf0a57b7afaupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
Plain-text copy, no login: /skill-bundles/alexchen31337-llm-knowledge-base.md