--- name: layered-memstack description: > 3-layer persistent memory system for OpenClaw agents. Provides structured memory with L1 (core facts in MEMORY.md), L2 (topic files + daily notes), L3 (deep references + knowledge graph). Includes BOOTSTRAP.md compilation (single compiled snapshot replacing multiple per-session file reads), automated maintenance via OpenClaw native Dreaming (nightly consolidation, 2026.4.8+), weekly audit cron for TTL cleanup and archiving, deduplication engine, and knowledge graph. Optionally extends with memory-wiki for structured claims, contradiction detection, and staleness dashboards. Use when: setting up agent memory from scratch, organizing existing workspace notes into layers, automating memory maintenance, preventing duplicate entries, managing a knowledge graph of entities, archiving old daily notes, reducing per-session token cost via BOOTSTRAP.md, or configuring memorySearch for multi-layer retrieval with temporal decay and MMR. --- # layered-memstack 3-layer memory system: L1 core → L2 topics/dailies → L3 deep references. ## Architecture ``` MEMORY.md ← L1: always loaded, ~80-110 lines max (breadcrumbs + pointers) memory/ ├── {topic}.md ← L2: topic breadcrumbs (viajes, salud, tecnico...) ├── YYYY-MM-DD.md ← L2: daily notes (auto-generated at 3 AM) ├── archive/ ← dailies older than 14 days reference/ ├── entities.md ← knowledge graph (people, places, projects, relations) ├── *.md ← L3: deep dives (loaded on demand via memory_search) scripts/ ├── memory-dedup.js ← dedup engine ├── memory-check-pointers.js ← broken-pointer detector (live memory only) └── cron-health.js ← pre-digests cron state for the audit pass BOOTSTRAP.md ← compiled snapshot (generated nightly, single read at session start) ``` ## Setup ### 1. Create directory structure ```bash mkdir -p memory/archive reference scripts ``` ### 2. Copy the scripts Copy `scripts/memory-dedup.js`, `scripts/memory-check-pointers.js` and (optionally) `scripts/cron-health.js` from this skill to the workspace `scripts/` directory. They are zero-dependency Node and resolve the workspace as the parent of `scripts/` (override with `MEMSTACK_WORKSPACE`). If your memory files and your `scripts/`/`data/` live in different directories, pass the extra roots to the pointer checker with `MEMSTACK_EXTRA_ROOTS=/path/a:/path/b` so those pointers resolve. ### 3. Configure memorySearch Add to `openclaw.json` under `agents.defaults.memorySearch`: ```json5 { "extraPaths": ["MEMORY.md", "USER.md", "IDENTITY.md", "memory/", "reference/", "projects/"], "sources": ["memory", "sessions"], "query": { "hybrid": { "mmr": { "enabled": true, "lambda": 0.7 }, "temporalDecay": { "enabled": true, "halfLifeDays": 30 } } } } ``` ### 3b. Embedding provider — local (recommended) vs cloud `memorySearch` needs an embedding provider. **We recommend local embeddings** over a cloud provider (Gemini/OpenAI/Voyage), for two reasons learned in production: 1. **Robustness** — cloud providers suffer quota cutoffs (HTTP 429) and config *drift* on updates, both of which pause semantic search until the index is rebuilt. Local depends on no one. 2. **Privacy** — with cloud, every memory chunk (health, finances, family) is sent to the provider's API to be embedded. With local, **memory never leaves the server**. This is the correct architecture for a personal 24/7 assistant holding sensitive data. The only trade-off is retrieval quality (local EmbeddingGemma = 768 dims vs Gemini's 3072), which is marginal for a personal corpus of a few thousand notes, plus one always-on service (~600 MB–1 GB RAM). **Option A — Local (Ollama + EmbeddingGemma 300M) — recommended** Install Ollama in user space (no sudo needed): download the release tarball from the [Ollama releases page](https://github.com/ollama/ollama/releases), unpack it under `~/.local`, and run `ollama serve` as a `systemd --user` unit so it survives reboots (with linger enabled). Then pull the embedding model: ```bash loginctl enable-linger "$USER" # user services keep running after logout/reboot systemctl --user enable --now ollama # unit ExecStart=%h/.local/ollama/bin/ollama serve ollama pull embeddinggemma # 621 MB · 768 dims · multilingual (good in Spanish) ``` Declare the provider in `openclaw.json` under `models.providers`, then point `memorySearch` at it: ```json5 { "models": { "providers": { "ollama-local": { "api": "ollama", "baseUrl": "http://127.0.0.1:11434", "apiKey": "ollama-local", "models": [{ "id": "embeddinggemma", "name": "EmbeddingGemma 300M (local)" }] } } }, "agents": { "defaults": { "memorySearch": { "provider": "ollama-local", "model": "embeddinggemma" } } } } ``` **Option B — Cloud (Gemini)** Zero setup and slightly higher retrieval quality (3072 dims), but subject to quotas (429) and sends every chunk to Google. Fine when privacy is not critical. Set `memorySearch.provider: "gemini"` and provide a `GEMINI_API_KEY`. > **On any provider change**, the stored vectors no longer match the new provider/dimensions > and OpenClaw **pauses semantic search until the index is rebuilt**. Always run > `openclaw memory index --force` after switching, then verify with > `openclaw memory status --deep` (look for `Vector store: ready`, the correct `Vector dims`, > and a real `memory_search` returning hits). ### 4. Create starter files - **MEMORY.md** — see `references/memory-template.md` - **reference/entities.md** — see `references/entities-template.md` - **INDEX.md** — catalog of all files with tags and line counts ### 5. Enable Dreaming (OpenClaw 2026.4.8+) Dreaming replaces the manual 3 AM auto-summary cron. Enable it in OpenClaw config: ```bash openclaw config patch '{"plugins":{"entries":{"memory-core":{"config":{"dreaming":{"enabled":true,"frequency":"0 3 * * *","timezone":"Europe/Madrid"}}}}}}' ``` Then restart the gateway. OpenClaw will automatically create and manage the nightly consolidation sweep. ### 6. Set up remaining crons Run the setup script to create the weekly audit (and optional MCP audit) cron: ```bash bash scripts/setup-crons.sh --tz Europe/Madrid --channel telegram --to "CHAT_ID" ``` Options: - `--tz IANA` — timezone (default: Europe/Madrid) - `--channel telegram|whatsapp|discord` — delivery channel for summaries - `--to CHAT_ID` — delivery target (Telegram chat ID, phone number, etc.) - `--model alias` — model override (default: uses your configured default) - `--dry-run` — show what would be created without creating - `--mcp-audit` — also create the nightly MCP memory security audit cron Or create them manually (see Cron Setup below). ## Layer Rules | Layer | When to load | What goes here | Size target | |-------|-------------|----------------|-------------| | L1 | Every session start | Breadcrumbs + pointers to L2/L3. Core facts, active project names, pending items. **No detail here.** | ~80-110 lines | | L2 | Today + yesterday auto-loaded; older via search | Topic summaries, daily notes with decisions/actions/facts | No limit | | L3 | Only via memory_search | Deep dives, travel details, health data, technical docs | No limit | ### L1 Writing Rules - MEMORY.md is **breadcrumbs + pointers only**. Detailed info goes in reference/ or memory/ files. - If something already has a pointer in L1, update the reference file — NOT MEMORY.md. - Before writing to MEMORY.md, always check for duplicates first: ```bash node scripts/memory-dedup.js --query "text to check" # Exit 0 = duplicate (skip), Exit 1 = new (safe to add) ``` - After any write, run dedup fix: ```bash node scripts/memory-dedup.js --fix ``` - Use TTL comments for time-bound items: `` - Keep entries atomic — one fact per line - Use L2 breadcrumbs to point to topic files: `Viajes → memory/viajes.md` ### L2 Writing Rules - Daily notes: `## Checkpoint [HH:MM]` sections with decisions, actions, atomic facts - Topic files: organized by subject, mid-level detail - Archive dailies older than 14 days to `memory/archive/` ### L3 Writing Rules - Detailed docs only — don't duplicate L1/L2 content - Update `reference/entities.md` when new entities appear - Structure: sections by entity type (people, places, projects, etc.) ### Compress-on-completion pattern (L3) When a long-running L3 doc closes (a trip ends, a project ships, an investigation wraps), the operational detail stops being read but the references (booking codes, IDs, lessons learned, pending refunds) still matter. **Pattern:** split the doc in two files instead of deleting either: - `reference/archive/.md` — compressed summary (~2–4 KB): key refs, totals, lessons, outstanding items. This is what `memory_search` surfaces and what humans skim. - `reference/archive/_full.md` — original verbatim (preserved, untouched). Linked from the summary's header (`Detalle completo → ...`). Why keep both: - The summary is what you actually need 99% of the time after completion. - The full doc is irreplaceable when a refund argument, a flight rebook, or a forensic question pops up months later. Compressing in place loses it. - Net effect: searches/loads hit ~4 KB instead of ~20 KB, with zero data loss. Rule of thumb: apply when the doc is ≥ 10 KB, has been read-only for ≥ 2 weeks, and represents a closed event. The optional analytical audit (see Cron Setup → Cron 1b) is a natural place to flag candidates. ## Dedup Engine `scripts/memory-dedup.js` — token-based similarity without embeddings. ### Modes ```bash # Check for duplicates (read-only) node scripts/memory-dedup.js --check # Fix: remove exact dupes, mark semantic dupes with node scripts/memory-dedup.js --fix # Query single text against MEMORY.md node scripts/memory-dedup.js --query "GitHub configured" # Exit 0 = duplicate, Exit 1 = new # Batch query: filter lines from file node scripts/memory-dedup.js --query-batch /tmp/candidates.txt ``` ### Options - `--file path` — target file (default: MEMORY.md) - `--threshold 0.65` — similarity threshold (default: 0.65) - `--verbose` — show comparison details ### Algorithm Jaccard similarity + containment ratio + entity overlap + segment-level comparison. Entities extracted: dates, version numbers, hex IDs, phone numbers, amounts, chat IDs. Threshold 0.65 balances false positives vs missed duplicates. > **Note (OpenClaw 2026.4.8+):** Dreaming injects `` provenance markers into MEMORY.md. The dedup engine automatically skips these lines to avoid false positives (fixed in `fd389b9`). ## Cron Setup ### Dreaming — nightly consolidation (replaces Cron 1) As of OpenClaw 2026.4.8, the daily 3 AM auto-summary is handled natively by **Dreaming** in `memory-core`. No manual cron needed. Dreaming runs a multi-phase sweep (light → REM → deep) that: - Consolidates session transcripts into structured memory - Extracts atomic facts and decisions - Updates MEMORY.md with genuinely new content (dedup built-in) - Updates the knowledge graph - Archives old daily notes Enable in config: ```json { "plugins": { "entries": { "memory-core": { "config": { "dreaming": { "enabled": true, "frequency": "0 3 * * *", "timezone": "Europe/Madrid" } } } } } } ``` ### Cron 1: Weekly mechanical audit (Monday 3:40 AM recommended) ``` Schedule: 40 3 * * 1 (user timezone) Session: isolated agentTurn timeoutSeconds: 300 ← always set this (see "Cron hygiene" below) ``` Prompt should instruct the agent to: 1. **Run `node scripts/memory-check-pointers.js`** and fix every broken pointer it reports, re-running until it exits 0. This is step one on purpose: a dead pointer is the one failure mode that makes recall actively lie. 2. Move daily notes older than 14 days to `memory/archive/` 3. Clean expired TTL entries from MEMORY.md (lines with `` past today) 4. Run `--fix` on MEMORY.md 5. If MEMORY.md exceeds ~70 lines → warn to prune (move detail to reference/) 6. Report summary of changes (or stay silent if nothing to do) This pass is mechanical: no judgment calls, just rules. Even so, prefer a mid-tier model over the cheapest one — the file-moving steps involve `find`/`mv` against your memory directory, and a model that misreads an exclusion list will archive something it shouldn't. **Schedule it away from Dreaming.** If Dreaming runs at 3:00, don't run this at 3:00 too: the audit will be reading and rewriting MEMORY.md while the deep phase is writing to it. 3:40 leaves room. **Guard the file-moving step.** Any `find memory/ -name '*.md' -mtime +14 | xargs mv` style step must exclude your topic files by name, not just by pattern — topic files (`viajes.md`, `salud.md`, `trading.md`, …) live in the same directory as the dailies and are often older than 14 days. The safe form is an explicit date-shaped match: ```bash find memory -maxdepth 1 -name '20[0-9][0-9]-[0-1][0-9]-[0-3][0-9].md' -mtime +14 ``` ### Pointer integrity — why a script and not a prompt `scripts/memory-check-pointers.js` checks that every workspace-relative path cited in **live** memory actually exists on disk. ```bash node scripts/memory-check-pointers.js # human report, exit 1 if broken node scripts/memory-check-pointers.js --quiet # one line per break (for crons) node scripts/memory-check-pointers.js --json # machine-readable ``` It only scans live memory (root files, `memory/*.md`, `reference/*.md` — no recursion), because archived notes legitimately point at files that were later moved. Mark example paths in prose with ``. The direction matters. An LLM audit that asks "are there files with no pointer?" finds orphans; it does **not** find pointers with no file, and that is the harmful direction — the agent reads a breadcrumb saying "detail in `reference/foo.md`" and reports on a file that no longer exists. Real case: a weekly audit reported OK for weeks while two dead pointers sat in a topic file. Deterministic check, 50 ms, exit code a cron can branch on. Don't ask a model to eyeball this. ### Cron hygiene — the watchdog paradox If you run an "audit my crons" cron, do **not** let it list the crons itself. Handing the model raw `openclaw cron list --all --json` means feeding it every job object you own: measured at 51 jobs that's ~141 KB, and runs were burning 250–630k tokens and failing ~40% of the time. That failure is self-concealing — the watchdog is what reports other crons failing, so when it dies of indigestion, nothing reports anything, including this memory audit. Use `scripts/cron-health.js`, which reads live gateway state and emits only the actionable part (141 KB → 1.7 KB, −98.8%; the same run went from error/390k tokens to ok/174k tokens in 44 s): ```bash node scripts/cron-health.js # compact text report node scripts/cron-health.js --json # compact JSON # exit 0 = clean · 1 = jobs in error · 2 = gateway unreadable ``` The cron prompt should run the script and explicitly forbid listing jobs by hand, *with the reason* — otherwise a model that finds the digest terse will "helpfully" fall back to the raw dump and reintroduce the bug. Two related traps this surfaces: - **Always set `timeoutSeconds` on `agentTurn` crons.** A job with no limit doesn't fail fast, it fails silently and expensively. `cron-health.js` counts these for you. - **Never call the in-agent cron tool from inside a cron** to enumerate jobs: on isolated cron sessions the narrow self-cleanup grant filters the response to the current job, so the audit sees only itself and cheerfully reports all-clear. ### Cron 1b (optional): Analytical audit (Sunday 22:00 recommended) The mechanical audit above ships fixes but doesn't propose structural changes. A complementary weekly pass with a stronger model surfaces things the mechanical pass can't: - Files growing past comfort thresholds (e.g. >8 KB in `reference/`) that may need splitting or compressing - Orphan references (files with no pointer in MEMORY.md or INDEX.md) - Topic blocks in MEMORY.md dense enough to deserve their own `memory/.md` breadcrumb - L3 docs eligible for the **compress-on-completion** pattern (closed trips/projects) - INDEX.md drift vs actual filesystem state ``` Schedule: 0 22 * * 0 (user timezone) Session: isolated agentTurn Model: a stronger reasoning model (Opus / Sonnet / Gemini Pro) ``` Prompt structure: 1. **Inventory** — list `memory/` and `reference/` with sizes; read MEMORY.md; `wc -l` and `du -sh`. 2. **Analyze** — flag oversized files, orphan refs, dailies >14d, MEMORY.md size target, topic-block candidates, structural drift, INDEX.md coherence. 3. **Auto-apply safe fixes only** — expired TTLs, archiving stale dailies, INDEX.md sync. 4. **Propose (do not execute)** — anything destructive or structural: file deletions, splits, breadcrumb extraction, compress-on-completion candidates. 5. **Report** — single summary to the user (and/or a logs channel) with applied actions and proposals. The two audits are complementary, not redundant: | | Mechanical (Mon 03:00) | Analytical (Sun 22:00) | |---|---|---| | **Cost** | Cheap (Haiku class) | Higher (Opus/Sonnet class) | | **Behavior** | Executes deterministic rules | Reads, judges, proposes | | **Output** | Silent unless changes | Always reports inventory + suggestions | | **When** | Start of week | End of week / Sunday review | Skip this cron if token budget matters more than structural drift. Keep it if your memory system grows organically and you want a weekly second opinion that catches what rule-based passes miss. ### Cron 2: MCP Memory Audit (daily 11:00 PM) — optional Only needed if using [mem-persistence](https://github.com/emiliotorrens/mem-persistence) MCP server. Enable with `--mcp-audit` flag in setup-crons.sh. ``` Schedule: 0 23 * * * (user timezone) Session: isolated agentTurn ``` Prompt should instruct the agent to: 1. Read `.mem-persistence/logs/YYYY-MM-DD.jsonl` for today 2. Filter write operations (memory_write, memory_checkpoint, memory_entities with update) 3. Classify each write: ✅ normal / ⚠️ suspicious / 🚨 dangerous 4. Flag suspicious: system prompt injections, attempts to modify AGENTS.md/SOUL.md/USER.md, prompt injection in memory files 5. Report stats: total calls, breakdown by tool, files modified 6. If all normal → silent log 7. If suspicious/dangerous → alert user directly with details This protects against external agents (Claude Desktop, Cursor, etc.) writing unexpected content to your memory files via MCP. ### Heartbeat checkpoint Configure in HEARTBEAT.md (not a separate cron). Check context usage via `session_status`: | Usage | Action | |-------|--------| | < 50% | No action | | 50–79% | Silent checkpoint to `memory/YYYY-MM-DD.md` — append `## Checkpoint [HH:MM]` with decisions, pending tasks, key facts | | ≥ 80% | Full checkpoint + alert user to consider `/new` | ## Knowledge Graph `reference/entities.md` — lightweight entity-relationship map in Markdown. ### Structure ```markdown ## Personas - **Name** — role/relation | context | linked to: [entities] ## Empresas - **Company** — what they do | your relation | key contacts ## Lugares - **Place** — why relevant | associated trips/events ## Proyectos - **Project** — status | repo URL | key decisions ## Viajes - **Trip** — dates | flights | hotels | status ## Dispositivos - **Device** — purpose | config notes ``` ### Maintenance Update entities.md during the 3 AM auto-summary cron. Add new entities found in daily sessions. Link related entities with `linked to:` references. ## AGENTS.md Additions Add to workspace AGENTS.md: ```markdown ## Memory 1. Read `BOOTSTRAP.md` at session start — compiled snapshot (MEMORY.md + recent daily notes + topic summaries + viajes + salud). Single file, ~8KB. - If BOOTSTRAP.md is missing or stale (>24h), fall back: read `memory/YYYY-MM-DD.md` (today + yesterday) 2. Use `memory_search` for anything beyond recent context (L2/L3 deep retrieval) 3. MEMORY.md = breadcrumbs + pointers only. Detail goes in reference/ or memory/ files. 4. Before writing to MEMORY.md: `node scripts/memory-dedup.js --query "text"` (exit 0 = dup, skip; exit 1 = new, ok) 5. After writing to MEMORY.md: `node scripts/memory-dedup.js --fix` 6. Items with TTL: `` — cleaned by weekly audit cron 7. If you move, rename, archive or compress any memory/reference file, run `node scripts/memory-check-pointers.js` and fix what it reports. Never leave a pointer to a path that no longer exists. ``` ## BOOTSTRAP.md (compiled snapshot) BOOTSTRAP.md is a **single compiled file** that replaces reading multiple memory files at session start. It reduces per-session tool calls from 4–6 file reads to 1. ### Generate ```bash node scripts/build-bootstrap.js ``` ### Contents - `MEMORY.md` (curated facts, truncated to 2000 chars) - Daily notes: today + yesterday (1500 chars each) - Topic files: `memory/viajes.md`, `memory/salud.md`, `memory/tecnico.md` (800 chars each) - `reference/viajes-kayak.md` first 60 lines (upcoming trips) - `reference/salud-datos.md` last 15 lines (recent health data) ### Maintenance cron (2:50 AM daily) ```json { "name": "Build BOOTSTRAP.md", "schedule": { "kind": "cron", "expr": "50 2 * * *", "tz": "Europe/Madrid" }, "payload": { "kind": "agentTurn", "model": "google/gemini-2.5-flash", "message": "Ejecuta: node /path/to/scripts/build-bootstrap.js\nResponde siempre: OK" } } ``` ## Credits - **Local embeddings setup** (Ollama + EmbeddingGemma for private, quota-free memory search) contributed and validated by **Mario Andújar** — https://github.com/marioandujar