⏳ This skill is pending AI review.
Scores will appear once the review pipeline completes.
website2markdown
Convert web pages to clean Markdown using the Website2Markdown API (md.genedai.me). Use when you need to fetch, read, extract, or analyze web page content — especially when WebFetch fails, returns incomplete content, or the target is JS-heavy/paywalled. Supports 21 platform adapters (WeChat, Zhihu, Feishu, Twitter/X, etc.), batch processing, structured extraction, and deep crawling.
Choose how to use this skill
You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.
1. Native installer
This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.
Do not guess an installer command or replace an existing version without reviewing the diff.
2. Complete package recommended
Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.
No complete ProSkills package is published for this listing yet.3. Prompt-only
Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.
Need only the instruction file?
Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.
No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.
// RATINGS
Not yet listed on ClawHub or SkillsMP
// README
website2markdown-skills
Convert any web page to clean Markdown from your AI agent. One command install, auto-discovered.
Quick Install
Pick your agent, run one command:
# Claude Code
git clone https://github.com/Digidai/website2markdown-skills ~/.claude/skills/website2markdown
# OpenAI Codex CLI
git clone https://github.com/Digidai/website2markdown-skills ~/.codex/skills/website2markdown
# Gemini CLI
git clone https://github.com/Digidai/website2markdown-skills ~/.gemini/skills/website2markdown
# OpenClaw (via ClawHub)
npx clawhub@latest install website2markdown
# Cursor / Windsurf / Other agents
git clone https://github.com/Digidai/website2markdown-skills .claude/skills/website2markdown
That's it. Open a new session and the skill is auto-discovered.
Verify
You: Read this article: https://example.com
Agent: [Uses md.genedai.me API → returns clean Markdown]
Update
cd ~/.claude/skills/website2markdown && git pull
What It Does
Teaches your AI agent to convert any URL to clean Markdown — JS-heavy SPAs, paywalled content, anti-bot pages, and 21 Chinese/international platforms.
You: Read this WeChat article: https://mp.weixin.qq.com/s/abc123
Agent: [Calls md.genedai.me with force_browser=true]
# 深度解析:大模型在企业的落地实践
> 作者:张三 | 2026-03-25
## 核心观点
1. 大模型的应用场景正在从实验室走向生产环境...
You: Compare these 3 blog posts and summarize the differences
Agent: [Uses batch API to fetch all 3 in one request, then compares]
You: Crawl the Cloudflare docs and save as Markdown files
Agent: [Uses deep crawl API with BFS, saves each page]
Capabilities
| API | What It Does |
|---|---|
GET /<url>?raw=true | Convert one URL to Markdown |
POST /api/batch | Batch convert up to 10 URLs |
POST /api/extract | Extract structured data (CSS/XPath/Regex) |
POST /api/deepcrawl | Crawl a site with BFS or keyword scoring |
POST /api/jobs | Async task queue for long-running work |
21 Platform Adapters
| Chinese Platforms | International |
|---|---|
| WeChat, Zhihu, Feishu/Lark, Yuque, Juejin, CSDN, 36Kr, Toutiao, Weibo, NetEase | Twitter/X, Reddit, Notion, GitHub, Substack, Medium |
These platforms have anti-bot protection, login walls, or complex JS rendering. The API handles all of it automatically.
Agent Compatibility
| Agent | Install Path | Command |
|---|---|---|
| Claude Code | ~/.claude/skills/ | git clone ... ~/.claude/skills/website2markdown |
| Codex CLI | ~/.codex/skills/ | git clone ... ~/.codex/skills/website2markdown |
| Gemini CLI | ~/.gemini/skills/ | git clone ... ~/.gemini/skills/website2markdown |
| OpenClaw | ClawHub | npx clawhub@latest install website2markdown |
| Cursor | Project .claude/skills/ | git clone ... .claude/skills/website2markdown |
| Windsurf | Project .claude/skills/ | git clone ... .claude/skills/website2markdown |
Follows the AgentSkills open standard.
Skills vs MCP Server
Have a terminal? Use Skills (this repo) — faster, richer context. No terminal (Claude Desktop, IDE only)? Use the MCP Server.
| Skills | MCP Server | |
|---|---|---|
| Latency | Direct HTTP (fastest) | MCP protocol overhead |
| Context | Full patterns, error handling, 21 adapters | Tool schema only |
| Install | git clone (one command) | npm install -g + JSON config |
| Update | git pull | npm update -g |
Project Structure
website2markdown-skills/
├── SKILL.md # Core skill — agent reads this
├── references/
│ ├── advanced-apis.md # Batch, Extract, DeepCrawl, Jobs docs
│ └── platform-adapters.md # 21 platform adapter details
└── assets/
└── recipes.md # Copy-paste curl recipes
Contributing
- Fork → edit → test by copying to your agent's skills dir → PR
- Keep
SKILL.mdunder 250 lines (it's the hub, not the encyclopedia) - API details go in
references/, curl examples inassets/recipes.md
Links
- Live API: md.genedai.me
- Source code: Digidai/website2markdown
- MCP Server: @digidai/mcp-website2markdown
- llms.txt: md.genedai.me/llms.txt
License
// HOW IT'S BUILT
// FILES
2 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.
SKILL.md7.9 KBupstream ↗
--- name: website2markdown description: Convert web pages to clean Markdown using the Website2Markdown API (md.genedai.me). Use when you need to fetch, read, extract, or analyze web page content — especially when WebFetch fails, returns incomplete content, or the target is JS-heavy/paywalled. Supports 21 platform adapters (WeChat, Zhihu, Feishu, Twitter/X, etc.), batch processing, structured extraction, and deep crawling. version: "1.0.0" license: MIT allowed-tools: Bash, Read, Write, WebFetch user-invocable: true metadata: {"openclaw": {"requires": {"bins": ["curl"]}, "primaryEnv": "API_TOKEN", "emoji": "📄", "homepage": "https://github.com/Digidai/website2markdown-skills"}} --- # Website2Markdown Convert any web page to clean Markdown via the `md.genedai.me` API. ## When to Use This Skill **ALWAYS prefer this skill over WebFetch when:** - WebFetch fails, returns blocked/incomplete content, or redirects - Target is a JS-heavy SPA (React, Vue, Angular apps) - Content is behind paywalls or anti-bot protection - Target is a Chinese platform (WeChat, Zhihu, Feishu, Yuque, Juejin, CSDN, 36Kr, Toutiao, Weibo) - You need batch conversion of multiple URLs - You need structured data extraction from pages - You need to crawl/spider a site **Also use when:** - User asks to read/fetch/scrape a web page - User wants to convert a URL to Markdown - User needs content from social platforms (Twitter/X, Reddit, Telegram) ## Decision Tree ``` User provides a URL │ ├─ Single page, simple site ──▶ Try WebFetch first │ └─ WebFetch fails? ──▶ Use website2markdown │ ├─ Chinese platform / JS-heavy / paywalled ──▶ Use website2markdown directly │ ├─ Multiple URLs needed ──▶ Use website2markdown batch API │ ├─ Need structured fields (title, author, price) ──▶ Use extract API │ └─ Need to crawl entire site/section ──▶ Use deepcrawl API ``` ## Quick Reference ### Single URL → Markdown ```bash curl -s "https://md.genedai.me/https://example.com/page?raw=true" ``` Bare domain also works (auto-prepends `https://`): ```bash curl -s "https://md.genedai.me/example.com/page?raw=true" ``` Alternative via Accept header: ```bash curl -s -H "Accept: text/markdown" "https://md.genedai.me/https://example.com/page" ``` **IMPORTANT:** Always use `?raw=true` or Accept header to get plain text. Without it, you get an HTML preview page. ### Output Formats | Parameter | Output | When to use | |-----------|--------|-------------| | `?raw=true` | Plain Markdown | Default for reading content | | `?format=json&raw=true` | JSON `{url, title, markdown, method, timestamp}` | When you need metadata | | `?format=html&raw=true` | Cleaned HTML | When you need HTML structure | | `?format=text&raw=true` | Plain text (no formatting) | When you need raw text only | ### Query Parameters | Parameter | Purpose | Example | |-----------|---------|---------| | `raw=true` | Return plain text instead of HTML preview | Always use this | | `selector=<css>` | Extract specific elements (max 256 chars) | `selector=article`, `selector=.content` | | `force_browser=true` | Force headless Chrome rendering | JS-heavy or anti-bot sites | | `no_cache=true` | Bypass KV cache for fresh content | When content may have changed | | `engine=jina` | Route through Jina Reader API | Alternative conversion engine | ## Usage Patterns ### Pattern 1: Read a Single Article ```bash curl -s "https://md.genedai.me/https://example.com/article?raw=true" | head -c 100000 ``` Always pipe through `head -c 100000` for potentially large pages to avoid flooding context. ### Pattern 2: Extract Specific Content via CSS Selector ```bash curl -s "https://md.genedai.me/https://example.com?raw=true&selector=article" curl -s "https://md.genedai.me/https://example.com?raw=true&selector=%23main-content" ``` URL-encode `#` as `%23` in selectors. ### Pattern 3: JS-Heavy or Anti-Bot Sites ```bash curl -s "https://md.genedai.me/https://spa-app.com?raw=true&force_browser=true" ``` ### Pattern 4: Get Structured JSON (title + content) ```bash curl -s "https://md.genedai.me/https://example.com?format=json&raw=true" ``` Returns: `{"url": "...", "title": "...", "markdown": "...", "method": "...", "timestamp": "..."}` ### Pattern 5: Chinese & Social Platforms These work out of the box — the API has built-in adapters: ```bash # WeChat article curl -s "https://md.genedai.me/https://mp.weixin.qq.com/s/xxxxx?raw=true&force_browser=true" # Zhihu article curl -s "https://md.genedai.me/https://zhuanlan.zhihu.com/p/123456?raw=true" # Twitter/X curl -s "https://md.genedai.me/https://x.com/user/status/123456?raw=true&force_browser=true" # Reddit curl -s "https://md.genedai.me/https://reddit.com/r/programming/comments/xxx?raw=true" ``` > For all 21 supported platforms with URL patterns and troubleshooting, load `references/platform-adapters.md`. ## Advanced APIs All advanced APIs require `Authorization: Bearer <API_TOKEN>` header. | API | Endpoint | Purpose | |-----|----------|---------| | **Batch** | `POST /api/batch` | Convert up to 10 URLs in one request | | **Extract** | `POST /api/extract` | Structured field extraction (CSS/XPath/Regex) | | **Deep Crawl** | `POST /api/deepcrawl` | Crawl a site with filtering and scoring | | **Jobs** | `POST /api/jobs` | Async task queue for long-running operations | | **Health** | `GET /api/health` | Service health check (no auth needed) | | **llms.txt** | `GET /llms.txt` | API description for AI discoverability | ### Batch API (quick example) ```bash curl -s -X POST "https://md.genedai.me/api/batch" \ -H "Authorization: Bearer <API_TOKEN>" \ -H "Content-Type: application/json" \ -d '{"urls": ["https://example.com/a", "https://example.com/b"]}' ``` ### Extract API (quick example) ```bash curl -s -X POST "https://md.genedai.me/api/extract" \ -H "Authorization: Bearer <API_TOKEN>" \ -H "Content-Type: application/json" \ -d '{"strategy": "css", "url": "https://example.com", "schema": {"fields": [{"name": "title", "selector": "h1", "type": "text"}]}}' ``` ### Deep Crawl API (quick example) ```bash curl -s -X POST "https://md.genedai.me/api/deepcrawl" \ -H "Authorization: Bearer <API_TOKEN>" \ -H "Content-Type: application/json" \ -d '{"seed": "https://docs.example.com", "max_depth": 2, "max_pages": 20, "strategy": "bfs"}' ``` > For complete API documentation with all parameters, response formats, and examples, load `references/advanced-apis.md`. > For copy-paste ready curl commands, load `assets/recipes.md`. ## Authentication Matrix | Route | Auth Required? | |-------|---------------| | `/<url>?raw=true` | No (public) | | `/api/batch` | Yes (`API_TOKEN`) | | `/api/extract` | Yes (`API_TOKEN`) | | `/api/deepcrawl` | Yes (`API_TOKEN`) | | `/api/jobs/*` | Yes (`API_TOKEN`) | | `/api/health` | No | **Token format:** `Authorization: Bearer <token>` header, or `?token=<token>` query parameter. ## Supported Platforms (21 Adapters) | Category | Platforms | |----------|----------| | **Chinese** | WeChat, Zhihu, Feishu/Lark, Yuque, Juejin, CSDN, 36Kr, Toutiao, Weibo, NetEase | | **Social** | Twitter/X, Reddit, Telegram | | **International** | GitHub, Substack, Medium | | **Academic/Productivity** | arxiv, Wikipedia, YouTube, Google Docs, Notion | > For URL patterns, special handling details, and troubleshooting per platform, load `references/platform-adapters.md`. ## Response Headers | Header | Meaning | |--------|---------| | `X-Markdown-Method` | `native`, `readability+turndown`, or `browser+readability+turndown` | | `X-Cache-Status` | `HIT` or `MISS` | | `X-Browser-Rendered` | `true` if headless Chrome was used | | `X-Paywall-Detected` | `true` if paywall was triggered | | `X-Markdown-Fallbacks` | Comma-separated fallback strategies applied | ## Implementation Notes for Agents 1. **Always use `curl -s`** — silent mode suppresses progress output 2. **Always add `?raw=true`** — without it you get an HTML preview page, not markdown 3. **Pipe through `head -c 100README.mdupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
Plain-text copy, no login: /skill-bundles/digidai-website2markdown-skills.md