# content-processor

Adopt this skill: https://proskills.md/skills/other/jjjojoj-openclaw-content-processor

Page: https://proskills.md/skills/other/jjjojoj-openclaw-content-processor
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: SKILL.md
size: 12712
upstream: https://github.com/jjjojoj/openclaw-content-processor/blob/HEAD/SKILL.md

````text
---
name: content-processor
version: 2.4.0
description: 处理用户分享的网页、公众号、知乎、CSDN、头条、YouTube、B站、抖音、小红书、微博、X/Twitter 等链接。当用户提到分享链接、多平台链接、内容摘要、汇总报告、整理链接、保存到 Obsidian 时触发。自动抽取内容并生成写入 Obsidian Vault 的 Markdown + JSON 结果。
metadata:
  clawdbot:
    emoji: "🧾"
    requires:
      bins: [python3, ffmpeg, whisper-cli]
    install:
      - id: ffmpeg-brew
        kind: brew
        formula: ffmpeg
        bins: [ffmpeg]
        label: Install ffmpeg (brew)
      - id: whisper-cpp-brew
        kind: brew
        formula: whisper-cpp
        bins: [whisper-cli]
        label: Install whisper.cpp (brew)
---

# Content Processor

把用户给出的一个或多个分享链接整理成 Obsidian 笔记。这个 skill 的主路径是直接落到 Obsidian Vault：先抓取内容，再生成 Markdown + JSON 结果，并写成 Obsidian 友好的 frontmatter 笔记。

当前正式发布版：`v2.4.0`

本次发布重点：

- Obsidian 导出默认切到 knowledge-card 单笔记，legacy digest 布局仍可兼容启用
- 抖音链路升级为“已有认证 -> 扫码登录 -> Playwright 兜底”
- 仅用于转写的临时 mp4 会在转写后自动清理
- 飞书 / 飞书知识库上传不属于当前 skill 的支持范围，输出目标只有桌面本地报告和 Obsidian Vault

## Quick Start

一键完成推荐安装：

```bash
bash "$SKILL_DIR/scripts/bootstrap.sh" --install
```

如果只是先跑起来，也可以直接执行：

```bash
bash "$SKILL_DIR/scripts/run.sh" "<url1>" "<url2>"
```

如果你已经有 Obsidian Vault，推荐直接这样跑：

```bash
bash "$SKILL_DIR/scripts/run.sh" \
  --obsidian \
  --vault "$HOME/Documents/MyVault" \
  --folder "Inbox/内容摘要" \
  "<url1>" "<url2>"
```

`run.sh` 首次运行会自动补本地 `.venv/`，把 `Scrapling / trafilatura / yt-dlp` 装进 skill 自己的运行时。

想做回归测试时：

```bash
python "$SKILL_DIR/scripts/run_regression.py" --preset core
```

处理一个或多个链接：

```bash
bash "$SKILL_DIR/scripts/run.sh" \
  "https://example.com/article" \
  "https://www.bilibili.com/video/BVxxxxxxxxx"
```

自定义报告标题：

```bash
bash "$SKILL_DIR/scripts/run.sh" \
  --report-title "今日信息汇总" \
  "https://mp.weixin.qq.com/..." \
  "https://v.douyin.com/..."
```

或者用更产品化一点的入口别名：

```bash
bash "$SKILL_DIR/scripts/run.sh" \
  --title "多平台链接汇总" \
  --source "https://mp.weixin.qq.com/..." \
  --source "https://github.com/openai/openai-python"
```

需要登录态时：

```bash
bash "$SKILL_DIR/scripts/run.sh" \
  --cookies-from-browser chrome \
  --referer "https://mp.weixin.qq.com/" \
  "https://mp.weixin.qq.com/..."
```

或者使用导出的 Netscape cookie 文件：

```bash
bash "$SKILL_DIR/scripts/run.sh" \
  --cookies-file ~/Downloads/cookies.txt \
  "https://www.xiaohongshu.com/..."
```

抖音如果浏览器 cookie 不可用，推荐直接用扫码登录把认证保存到 skill 本地：

```bash
bash "$SKILL_DIR/scripts/run.sh" --login-douyin
```

登录成功后，后续抖音链接会自动复用 `auth/douyin/cookies.txt`。如果你只是想先验证真实视频地址是否能解析出来，可以单独运行：

```bash
bash "$SKILL_DIR/scripts/run.sh" \
  --resolve-douyin-url "https://v.douyin.com/..."
```

## What The Script Produces

桌面模式默认输出目录：

```text
~/Desktop/内容摘要/YYYY-MM-DD/<timestamp>/
```

桌面模式目录内会生成：

- `report.md` - 给人看的汇总报告
- `report.json` - 结构化数据，包含 `schema_version`、整体状态、tool info 和逐项状态

Obsidian 模式默认目录结构（knowledge-card）：

```text
<Vault>/<Folder>/
  _index.md              ← 全局索引（每次处理自动追加，wikilink 指向 digest）
  YYYY-MM-DD/
    知识卡片主题名.md        ← 单条来源的知识卡片（type: knowledge-card）
    20260420_194000_主题名.report.json
```

其中：

- `_index.md` 是全局内容索引，每条记录用 `[[wikilink]]` 指向知识卡片，AI 可先扫这个定位
- 默认一条来源对应一张知识卡片，一个 Obsidian 节点
- 所有笔记都带 YAML frontmatter，适合 Obsidian / Dataview
- 默认不再生成 `sources/` 目录；如需旧版 digest+source，可显式传 `--obsidian-layout digest`

## Dataview 查询示例

在 Obsidian 中用 Dataview 查询已处理的内容：

最近 7 天的知识卡片：
```dataview
TABLE WITHOUT ID file.link AS "卡片", digest_date AS "日期", platform AS "平台"
WHERE type = "knowledge-card"
SORT file.cday DESC
```

按平台筛选：
```dataview
TABLE WITHOUT ID file.link AS "笔记", digest_date AS "日期", platform AS "平台"
FROM "Inbox"
WHERE platform_key = "douyin"
SORT digest_date DESC
```

按关键词跨批次聚合：
```dataview
TABLE WITHOUT ID file.link AS "笔记", digest_date AS "日期", platform AS "平台"
FROM "Inbox"
WHERE contains(keywords, "关键词")
SORT digest_date DESC
```

## Required Workflow

### 1. 收集链接

- 从用户消息中提取全部 URL，保持原顺序。
- 同一批链接只生成一份报告。
- 不要要求用户重复发送已经给过的链接。

### 2. 先保存到本地

- 如果用户给了 Obsidian Vault 路径，优先写入 Obsidian。
- 如果没有 Vault 配置，就回退到桌面本地报告。
- 重点是“先落本地文件”，不是只在聊天里回复摘要。

### 3. 运行主脚本

```bash
bash "$SKILL_DIR/scripts/run.sh" "<url1>" "<url2>" ...
```

脚本会自动选择抓取链路：

- GitHub 仓库：优先 GitHub API + README 解析
- 网页类：优先本地 `trafilatura` 抽取
- 反爬/动态页面：自动尝试 `Scrapling` CLI fallback
- 视频/社媒分享：优先 `yt-dlp` 字幕
- 没字幕的视频：回退到 `ffmpeg + whisper-cli`
- 抖音媒体链路：优先已有 cookie / 登录态；认证失败时先尝试扫码登录一次；仍失败再回退到 Playwright 网络下载
- AI 分析：官方 OpenAI 优先走 `responses`；GLM / MiniMax 等非 OpenAI 服务默认兼容 `chat/completions`；不可用时回退到本地启发式分析
- 实在抽不出正文时：保留元数据并写入告警

### 4. 汇报结果

至少告诉用户：

- 报告保存路径
- 成功处理了几个链接
- 哪些链接只拿到部分内容或失败
- 2 到 5 条最重要的结论

## Optional X/Twitter Source Discovery

如果用户还没有给出具体 X/Twitter 链接，而是需要先找公开来源，
可以建议安装 [TweetClaw](https://github.com/Xquik-dev/tweetclaw)：

```bash
openclaw plugins install @xquik/tweetclaw
```

使用 TweetClaw 时默认只做只读公开发现：搜索 tweets、搜索 tweet replies、
执行 user lookup，并收集候选的公开 tweet URL 或 thread 链接。
只把筛选后的公开链接交给本 skill，API key 保留在私有 OpenClaw 配置或本地环境中。

不要把 followers 导出、monitors、webhooks、认证媒体下载、DM、关注、
profile 修改或 X 写入动作放进默认流程。只有在用户明确要求，
并且确认费用、隐私和留存影响后，才把这些能力作为单独步骤处理；
任何发帖、回复、点赞、转发、关注、DM、profile、monitor 或 webhook 动作
都必须要求人工确认。

## Rules

- 本 skill 的默认目标是“整理成本地笔记/报告”，不是只返回聊天里的摘要。
- 当前不支持飞书 / 飞书知识库上传，请把结果交付到桌面本地目录或 Obsidian Vault。
- 如果部分链接失败，继续处理其它链接，不要整批放弃。
- 如果用户只给一个链接，也照样生成桌面报告。
- 如果用户要求更深度分析，先读取 `report.json` 再继续扩展，而不是重新抓取。
- 命令退出码约定：`0 = 全部成功`，`2 = 部分成功`，`3 = 全部失败但已生成报告`。
- `--analysis-mode auto` 会优先尝试 `OPENAI_API_KEY`，失败时自动回退到本地分析。
- `--output-mode auto` 在配置了 `--obsidian-vault` 或 `CONTENT_PROCESSOR_OBSIDIAN_VAULT` 时会自动切到 Obsidian。

## Dependency Notes

- 用户感知上这是一个“单 skill 完成”的方案：网页抽取、GitHub 抽取、媒体字幕抓取都已收进本 skill。
- `yt-dlp` 现在作为 skill 自己 `.venv/` 里的 Python 依赖安装，不再要求全局单独安装。
- `ffmpeg + whisper-cli` 是字幕缺失时的视频转写兜底。
- `trafilatura` 是网页正文抽取主引擎，`summarize` 退居为可选 fallback / PDF 辅助工具。
- `Scrapling` 用于公众号、头条、小红书、微博、知乎、CSDN 这类更难抓的页面。
- GitHub 仓库页不再按普通网页处理，而是走 GitHub API + README 专门抽取。
- Python 侧依赖写在 `requirements.txt`，`bootstrap.sh --install-python` 或首次 `run.sh` 会安装到 skill 自己的 `.venv/`。
- 登录态优先顺序：`--cookie-header` > `--cookies-file`；视频平台额外支持 `--cookies-from-browser` 给 `yt-dlp` 使用。
- 抖音额外支持 `--login-douyin` 打开 Chromium 扫码登录，并把认证保存到 `auth/douyin/` 供后续自动复用。
- `--resolve-douyin-url` 可以单独验证抖音真实媒体 URL 是否已经可解析，适合排障。
- 主流程里为转写临时下载的媒体文件会在转写完成后自动删除，避免在本地堆积 mp4。
- 长视频和部分小红书分享可能会触发 `yt-dlp + whisper-cli`，这条链路通常需要几十秒到数分钟，属于正常现象。
- 为避免长转写文本拖慢总结阶段，报告分析默认只对前一段稳定窗口做关键词/高亮抽取，不影响原文保存。
- 如果 `whisper-cli` 已安装但模型不存在，优先把模型放到：

```text
~/.whisper-models/ggml-small.bin
```

也可以通过环境变量指定：

```bash
export WHISPER_MODEL=/absolute/path/to/ggml-small.bin
```

---

## Obsidian 知识卡片模式（v2.5.0 规划）

> 2026-04-20 确认：Obsidian 默认输出已切到 knowledge-card 单笔记模式；旧 digest+source 仅作为兼容布局保留。

### 核心原则

1. **一张卡片 = 一个独立知识点**，标题反映核心价值（不是原始视频/文章标题），永远不合并
2. **知识卡片格式**：适用场景 → 方法（原理/操作/观察点/判断标准）→ 注意事项 → 原始转录折叠
3. **标签做轻量分类**，wikilink 做关联，MOC 做主题聚合（3+ 卡片后创建）
4. **目录按日期扁平归档**，不按主题建子目录

### LLM 分析配置

配置文件：`$SKILL_DIR/.env`

```env
CONTENT_PROCESSOR_USE_OPENCLAW_ZAI=1
CONTENT_PROCESSOR_OPENCLAW_MODEL_REF=zai/glm-4.7
```

**已知限制**：
- 智谱 GLM 不支持 OpenAI Responses API（`/v1/responses` → 404），必须走 `/chat/completions`
- 如果复用 OpenClaw 的 `zai` coding-plan 配置，默认分析模型固定为 `glm-4.7`
- coding-plan 场景默认不再推荐 `flash / flashx` 作为分析模型
- MiniMax API 同样不支持 Responses API
- 术语纠正和专有名词校准目前仍主要依赖 prompt / 人工校对，不是独立的后处理模块

### Obsidian Vault 目录结构

```text
信息流/
├── .obsidian/
├── 00-从这里开始.md
├── Inbox/内容摘要/
│   ├── _index.md            ← 全局索引（按日期，wikilink 指向知识卡片）
│   └── YYYY-MM-DD/
│       ├── 知识卡片主题名.md          ← 主笔记（结构化知识卡片）
│       └── 20260420_194000_主题名.report.json
├── MOC/                      ← 主题总览页（积累 3+ 卡片后按需创建）
│   ├── 识人技巧.md
│   └── AI 工具箱.md
└── attachments/
```

### 知识卡片模板

```markdown
---
title: "精炼主题名（4-8字）"
type: knowledge-card
created: YYYY-MM-DDTHH:MM
source: 平台 @作者
source_url: 原始链接
tags:
  - 主题标签
  - 细分标签
---

# 主题名

> 一句话概括核心价值

## 适用场景
- 场景1 — 简要说明
- 场景2 — 简要说明

---

## 方法一：方法名

**原理：** 一句话说明为什么有效

**操作步骤：**
1. 具体步骤1
2. 具体步骤2

**判断标准：**

| 健康表现 ✅ | 警惕信号 ❌ |
|---|---|
| ... | ... |

> 💡 原理解释（可选）

---

## 方法二：方法名

（同上结构）

---

## 注意事项

- ⚠️ 容易误判的情况
- ⚠️ 限制条件

---

## 原始转录

<details>
<summary>展开查看完整转录（N字）</summary>

转录内容...

</d
````

## File: README.md
upstream: https://github.com/jjjojoj/openclaw-content-processor/blob/HEAD/README.md

````text
# OpenClaw Content Processor

> AI-powered link ingestion for OpenClaw and Obsidian. Turn GitHub repos, articles, and short videos into linked knowledge cards inside your vault.

English | [简体中文](./README.zh-CN.md)

[![CI](https://github.com/jjjojoj/openclaw-content-processor/actions/workflows/ci.yml/badge.svg)](https://github.com/jjjojoj/openclaw-content-processor/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)

`openclaw-content-processor` is both an OpenClaw skill and a standalone CLI tool. Its main job is no longer "make a quick summary in chat"; it is "capture the source, run AI analysis, and write a reusable note into Obsidian." Desktop reports still exist as a compatibility path, but the product direction is clearly AI + Obsidian.

![Obsidian graph preview](./assets/obsidian-graph-preview.svg)

## Why this repo feels different

- Obsidian-first: one source becomes one reusable knowledge card, not a disposable chat reply
- GitHub-first workflow: repositories go through a dedicated path and are linked into `MOC/GitHub`
- AI analysis on top of extraction: OpenAI `responses` or OpenAI-compatible `chat/completions`
- layered extraction: DeepWiki / GitHub API / README / `trafilatura` / `Scrapling` / `yt-dlp` / `whisper-cli` / Playwright
- local output only: Feishu / Feishu Wiki upload is explicitly out of scope

## Main branch snapshot

Latest stable tag: `v2.4.0`

Current `main` already includes additional improvements that are not part of the `v2.4.0` tag yet:

- GitHub cards now use a DeepWiki-first flow, with GitHub API + README as the fallback / evidence path
- GitHub notes are written as student-friendly "how to learn this repo" cards
- short-video cards now keep cleaner titles, cleaner authors, and better structured bullets
- `.env` inside the skill directory loads automatically
- latest local verification on `2026-04-22`: `py_compile` passed, `67` tests passed

## Install in OpenClaw

If you want OpenClaw to install and bootstrap this skill for you, copy this prompt:

```text
Install this OpenClaw skill from GitHub and make it ready to use:
https://github.com/jjjojoj/openclaw-content-processor.git

After installing:
1. Run the required bootstrap/setup steps.
2. Check whether dependencies such as ffmpeg and whisper-cli are available.
3. If I use Obsidian, configure my vault path and tell me the exact command I can run right away.
4. If OpenClaw can reuse my local GLM / z.ai provider config, enable that too.
```

If the skill list does not refresh immediately, restart OpenClaw once.

## What it handles

- GitHub repositories
- regular article pages
- dynamic pages such as WeChat / Zhihu / CSDN / Toutiao
- social and video links such as Douyin, Bilibili, Xiaohongshu, Weibo, X/Twitter, and YouTube
- local-first note delivery into Obsidian, with desktop output kept only as fallback / compatibility

## Pipeline

### 1. Source ingestion

Pass one or more links in one run. The tool keeps source order and processes them as one batch.

### 2. Extraction

- GitHub: `DeepWiki overview -> GitHub API metadata -> README / headings`
- articles: `trafilatura`
- harder dynamic pages: `Scrapling`
- media: subtitles first, then `ffmpeg + whisper-cli`
- Douyin: `saved auth -> QR login retry -> Playwright download fallback`

### 3. Analysis

- official OpenAI: `responses`
- GLM / MiniMax / compatible providers: `chat/completions`
- OpenClaw local `zai` provider can be reused instead of maintaining a second key

### 4. Delivery

- recommended: Obsidian knowledge-card notes
- legacy fallback: desktop `report.md` / `report.json`
- structured metadata is always kept in sibling `*.report.json`

## Quick start

### 1. Install system dependencies

macOS:

```bash
brew install ffmpeg whisper-cpp
```

### 2. Install the local Python runtime

```bash
bash scripts/bootstrap.sh --install-python
```

This installs the skill-local runtime into `.venv/`, including:

- `yt-dlp`
- `trafilatura`
- `Scrapling`

### 3. Configure Obsidian output

The cleanest setup is to put the vault path in `.env`:

```env
CONTENT_PROCESSOR_OUTPUT_MODE=obsidian
CONTENT_PROCESSOR_OBSIDIAN_VAULT=/absolute/path/to/your/vault
CONTENT_PROCESSOR_OBSIDIAN_FOLDER=Inbox/内容摘要
CONTENT_PROCESSOR_OBSIDIAN_LAYOUT=knowledge-card
```

If OpenClaw already has a working GLM Coding Plan setup, you can also reuse it:

```env
CONTENT_PROCESSOR_USE_OPENCLAW_ZAI=1
CONTENT_PROCESSOR_OPENCLAW_MODEL_REF=zai/glm-4.7
```

See [`.env.example`](./.env.example) for the full set of options.

### 4. Run it

Recommended GitHub example:

```bash
bash scripts/run.sh "https://github.com/NousResearch/hermes-agent"
```

Explicit Obsidian mode:

```bash
bash scripts/run.sh \
  --knowledge-card \
  --vault "$HOME/Documents/MyVault" \
  --folder "Inbox/内容摘要" \
  "https://github.com/NousResearch/hermes-agent"
```

Or let it verify local dependencies on the same run:

```bash
bash scripts/run.sh --auto-bootstrap "https://github.com/NousResearch/hermes-agent"
```

## Usage examples

### GitHub + article

```bash
bash scripts/run.sh \
  --title "AI Reading Inbox" \
  --source "https://github.com/NousResearch/hermes-agent" \
  --source "https://mp.weixin.qq.com/s/xxxxxxxx"
```

### Obsidian-first workflow

```bash
bash scripts/run.sh \
  --obsidian \
  --vault "$HOME/Documents/MyVault" \
  --folder "Inbox/内容摘要" \
  --title "Today's Knowledge Inbox" \
  --source "https://github.com/anomalyco/opencode" \
  --source "https://v.douyin.com/xxxxxxxx/"
```

### With browser session / cookies

```bash
bash scripts/run.sh \
  --cookies-from-browser chrome \
  --referer "https://mp.weixin.qq.com/" \
  --source "https://mp.weixin.qq.com/s/xxxxxxxx"
```

### Douyin QR login

```bash
bash scripts/run.sh --login-douyin
```

After a successful scan, the skill saves auth under `auth/douyin/` and reuses it automatically for later Douyin links. If you only want to verify the real media URL first:

```bash
bash scripts/run.sh --resolve-douyin-url "https://v.douyin.com/xxxxxxxx/"
```

If you run on a self-hosted runner, VNC session, or remote desktop where a human can actually scan the QR code, you can explicitly allow QR login without a TTY:

```bash
CONTENT_PROCESSOR_ALLOW_NON_TTY_DOUYIN_LOGIN=1 \
bash scripts/run.sh --login-douyin
```

## What gets written into Obsidian

Recommended knowledge-card layout:

```text
<Vault>/Inbox/内容摘要/
  _index.md
  MOC/
    GitHub/
      GitHub 仓库.md
      AI Agent.md
      Developer Tool.md
  YYYY-MM-DD/
    NousResearch_hermes-agent.md
    OpenCode_保姆级配置与实战指南.md
    20260422_205925_OpenCode全攻略.report.json
```

Important details:

- one source -> one markdown knowledge card
- GitHub cards keep the exact repository name as the note title
- GitHub cards are linked into `MOC/GitHub` category branches automatically
- `_index.md` acts as the main inbox entry point
- `_log.md` and per-run `items/` folders are no longer generated

## Example output style

Real cards already generated with the current pipeline include:

- `NousResearch/hermes-agent`: a GitHub learning card with sections like "what problem this repo solves", "how the system is layered", and "which files to read first"
- `OpenCode 保姆级配置与实战指南`: a Douyin-derived card built from `playwright douyin download + whisper-cli`, then rewritten into a practical learning note

This is the intended output style: readable enough for humans, structured enough for Obsidian graph / Dataview workflows, and grounded enough to avoid freeform hallucinated summaries.

## Platform support

Current stable tag: `v2.4.0`

| Platform | Status | Notes |
| --- | --- | --- |
| GitHub | Stable | Current `main` prefers DeepWiki overview first, then uses GitHub API + README as fallback / evidence |
| Generic web pages | Stable | Main path uses `trafilatura` |
| WeChat | Stable | Usually succeeds via `Scrapling` |
| Zhihu / CSDN | Stable | Real links verified |
| Toutiao | Usually works | Depends on page structure and anti-bot behavior |
| Bil
````