# website2markdown

Adopt this skill: https://proskills.md/skills/other/digidai-website2markdown-skills

Page: https://proskills.md/skills/other/digidai-website2markdown-skills
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: SKILL.md
size: 8085
upstream: https://github.com/digidai/website2markdown-skills/blob/HEAD/SKILL.md

````text
---
name: website2markdown
description: Convert web pages to clean Markdown using the Website2Markdown API (md.genedai.me). Use when you need to fetch, read, extract, or analyze web page content — especially when WebFetch fails, returns incomplete content, or the target is JS-heavy/paywalled. Supports 21 platform adapters (WeChat, Zhihu, Feishu, Twitter/X, etc.), batch processing, structured extraction, and deep crawling.
version: "1.0.0"
license: MIT
allowed-tools: Bash, Read, Write, WebFetch
user-invocable: true
metadata: {"openclaw": {"requires": {"bins": ["curl"]}, "primaryEnv": "API_TOKEN", "emoji": "📄", "homepage": "https://github.com/Digidai/website2markdown-skills"}}
---

# Website2Markdown

Convert any web page to clean Markdown via the `md.genedai.me` API.

## When to Use This Skill

**ALWAYS prefer this skill over WebFetch when:**
- WebFetch fails, returns blocked/incomplete content, or redirects
- Target is a JS-heavy SPA (React, Vue, Angular apps)
- Content is behind paywalls or anti-bot protection
- Target is a Chinese platform (WeChat, Zhihu, Feishu, Yuque, Juejin, CSDN, 36Kr, Toutiao, Weibo)
- You need batch conversion of multiple URLs
- You need structured data extraction from pages
- You need to crawl/spider a site

**Also use when:**
- User asks to read/fetch/scrape a web page
- User wants to convert a URL to Markdown
- User needs content from social platforms (Twitter/X, Reddit, Telegram)

## Decision Tree

```
User provides a URL
  │
  ├─ Single page, simple site ──▶ Try WebFetch first
  │     └─ WebFetch fails? ──▶ Use website2markdown
  │
  ├─ Chinese platform / JS-heavy / paywalled ──▶ Use website2markdown directly
  │
  ├─ Multiple URLs needed ──▶ Use website2markdown batch API
  │
  ├─ Need structured fields (title, author, price) ──▶ Use extract API
  │
  └─ Need to crawl entire site/section ──▶ Use deepcrawl API
```

## Quick Reference

### Single URL → Markdown

```bash
curl -s "https://md.genedai.me/https://example.com/page?raw=true"
```

Bare domain also works (auto-prepends `https://`):

```bash
curl -s "https://md.genedai.me/example.com/page?raw=true"
```

Alternative via Accept header:

```bash
curl -s -H "Accept: text/markdown" "https://md.genedai.me/https://example.com/page"
```

**IMPORTANT:** Always use `?raw=true` or Accept header to get plain text. Without it, you get an HTML preview page.

### Output Formats

| Parameter | Output | When to use |
|-----------|--------|-------------|
| `?raw=true` | Plain Markdown | Default for reading content |
| `?format=json&raw=true` | JSON `{url, title, markdown, method, timestamp}` | When you need metadata |
| `?format=html&raw=true` | Cleaned HTML | When you need HTML structure |
| `?format=text&raw=true` | Plain text (no formatting) | When you need raw text only |

### Query Parameters

| Parameter | Purpose | Example |
|-----------|---------|---------|
| `raw=true` | Return plain text instead of HTML preview | Always use this |
| `selector=<css>` | Extract specific elements (max 256 chars) | `selector=article`, `selector=.content` |
| `force_browser=true` | Force headless Chrome rendering | JS-heavy or anti-bot sites |
| `no_cache=true` | Bypass KV cache for fresh content | When content may have changed |
| `engine=jina` | Route through Jina Reader API | Alternative conversion engine |

## Usage Patterns

### Pattern 1: Read a Single Article

```bash
curl -s "https://md.genedai.me/https://example.com/article?raw=true" | head -c 100000
```

Always pipe through `head -c 100000` for potentially large pages to avoid flooding context.

### Pattern 2: Extract Specific Content via CSS Selector

```bash
curl -s "https://md.genedai.me/https://example.com?raw=true&selector=article"
curl -s "https://md.genedai.me/https://example.com?raw=true&selector=%23main-content"
```

URL-encode `#` as `%23` in selectors.

### Pattern 3: JS-Heavy or Anti-Bot Sites

```bash
curl -s "https://md.genedai.me/https://spa-app.com?raw=true&force_browser=true"
```

### Pattern 4: Get Structured JSON (title + content)

```bash
curl -s "https://md.genedai.me/https://example.com?format=json&raw=true"
```

Returns: `{"url": "...", "title": "...", "markdown": "...", "method": "...", "timestamp": "..."}`

### Pattern 5: Chinese & Social Platforms

These work out of the box — the API has built-in adapters:

```bash
# WeChat article
curl -s "https://md.genedai.me/https://mp.weixin.qq.com/s/xxxxx?raw=true&force_browser=true"

# Zhihu article
curl -s "https://md.genedai.me/https://zhuanlan.zhihu.com/p/123456?raw=true"

# Twitter/X
curl -s "https://md.genedai.me/https://x.com/user/status/123456?raw=true&force_browser=true"

# Reddit
curl -s "https://md.genedai.me/https://reddit.com/r/programming/comments/xxx?raw=true"
```

> For all 21 supported platforms with URL patterns and troubleshooting, load `references/platform-adapters.md`.

## Advanced APIs

All advanced APIs require `Authorization: Bearer <API_TOKEN>` header.

| API | Endpoint | Purpose |
|-----|----------|---------|
| **Batch** | `POST /api/batch` | Convert up to 10 URLs in one request |
| **Extract** | `POST /api/extract` | Structured field extraction (CSS/XPath/Regex) |
| **Deep Crawl** | `POST /api/deepcrawl` | Crawl a site with filtering and scoring |
| **Jobs** | `POST /api/jobs` | Async task queue for long-running operations |
| **Health** | `GET /api/health` | Service health check (no auth needed) |
| **llms.txt** | `GET /llms.txt` | API description for AI discoverability |

### Batch API (quick example)

```bash
curl -s -X POST "https://md.genedai.me/api/batch" \
  -H "Authorization: Bearer <API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/a", "https://example.com/b"]}'
```

### Extract API (quick example)

```bash
curl -s -X POST "https://md.genedai.me/api/extract" \
  -H "Authorization: Bearer <API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"strategy": "css", "url": "https://example.com", "schema": {"fields": [{"name": "title", "selector": "h1", "type": "text"}]}}'
```

### Deep Crawl API (quick example)

```bash
curl -s -X POST "https://md.genedai.me/api/deepcrawl" \
  -H "Authorization: Bearer <API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"seed": "https://docs.example.com", "max_depth": 2, "max_pages": 20, "strategy": "bfs"}'
```

> For complete API documentation with all parameters, response formats, and examples, load `references/advanced-apis.md`.
> For copy-paste ready curl commands, load `assets/recipes.md`.

## Authentication Matrix

| Route | Auth Required? |
|-------|---------------|
| `/<url>?raw=true` | No (public) |
| `/api/batch` | Yes (`API_TOKEN`) |
| `/api/extract` | Yes (`API_TOKEN`) |
| `/api/deepcrawl` | Yes (`API_TOKEN`) |
| `/api/jobs/*` | Yes (`API_TOKEN`) |
| `/api/health` | No |

**Token format:** `Authorization: Bearer <token>` header, or `?token=<token>` query parameter.

## Supported Platforms (21 Adapters)

| Category | Platforms |
|----------|----------|
| **Chinese** | WeChat, Zhihu, Feishu/Lark, Yuque, Juejin, CSDN, 36Kr, Toutiao, Weibo, NetEase |
| **Social** | Twitter/X, Reddit, Telegram |
| **International** | GitHub, Substack, Medium |
| **Academic/Productivity** | arxiv, Wikipedia, YouTube, Google Docs, Notion |

> For URL patterns, special handling details, and troubleshooting per platform, load `references/platform-adapters.md`.

## Response Headers

| Header | Meaning |
|--------|---------|
| `X-Markdown-Method` | `native`, `readability+turndown`, or `browser+readability+turndown` |
| `X-Cache-Status` | `HIT` or `MISS` |
| `X-Browser-Rendered` | `true` if headless Chrome was used |
| `X-Paywall-Detected` | `true` if paywall was triggered |
| `X-Markdown-Fallbacks` | Comma-separated fallback strategies applied |

## Implementation Notes for Agents

1. **Always use `curl -s`** — silent mode suppresses progress output
2. **Always add `?raw=true`** — without it you get an HTML preview page, not markdown
3. **Pipe through `head -c 100
````

## File: README.md
upstream: https://github.com/digidai/website2markdown-skills/blob/HEAD/README.md

````text
# website2markdown-skills

> Convert any web page to clean Markdown from your AI agent. One command install, auto-discovered.

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![API](https://img.shields.io/badge/API-md.genedai.me-22d3ee)](https://md.genedai.me)
[![Source](https://img.shields.io/badge/Source-website2markdown-black?logo=github)](https://github.com/Digidai/website2markdown)
[![npm](https://img.shields.io/npm/v/@digidai/mcp-website2markdown?label=MCP%20Server)](https://www.npmjs.com/package/@digidai/mcp-website2markdown)

## Quick Install

**Pick your agent, run one command:**

```bash
# Claude Code
git clone https://github.com/Digidai/website2markdown-skills ~/.claude/skills/website2markdown

# OpenAI Codex CLI
git clone https://github.com/Digidai/website2markdown-skills ~/.codex/skills/website2markdown

# Gemini CLI
git clone https://github.com/Digidai/website2markdown-skills ~/.gemini/skills/website2markdown

# OpenClaw (via ClawHub)
npx clawhub@latest install website2markdown

# Cursor / Windsurf / Other agents
git clone https://github.com/Digidai/website2markdown-skills .claude/skills/website2markdown
```

That's it. Open a new session and the skill is auto-discovered.

### Verify

```
You: Read this article: https://example.com
Agent: [Uses md.genedai.me API → returns clean Markdown]
```

### Update

```bash
cd ~/.claude/skills/website2markdown && git pull
```

## What It Does

Teaches your AI agent to convert **any URL** to clean Markdown — JS-heavy SPAs, paywalled content, anti-bot pages, and 21 Chinese/international platforms.

```
You: Read this WeChat article: https://mp.weixin.qq.com/s/abc123

Agent: [Calls md.genedai.me with force_browser=true]
       # 深度解析：大模型在企业的落地实践
       > 作者：张三 | 2026-03-25
       ## 核心观点
       1. 大模型的应用场景正在从实验室走向生产环境...
```

```
You: Compare these 3 blog posts and summarize the differences

Agent: [Uses batch API to fetch all 3 in one request, then compares]
```

```
You: Crawl the Cloudflare docs and save as Markdown files

Agent: [Uses deep crawl API with BFS, saves each page]
```

### Capabilities

| API | What It Does |
|-----|-------------|
| `GET /<url>?raw=true` | Convert one URL to Markdown |
| `POST /api/batch` | Batch convert up to 10 URLs |
| `POST /api/extract` | Extract structured data (CSS/XPath/Regex) |
| `POST /api/deepcrawl` | Crawl a site with BFS or keyword scoring |
| `POST /api/jobs` | Async task queue for long-running work |

### 21 Platform Adapters

| Chinese Platforms | International |
|-------------------|---------------|
| WeChat, Zhihu, Feishu/Lark, Yuque, Juejin, CSDN, 36Kr, Toutiao, Weibo, NetEase | Twitter/X, Reddit, Notion, GitHub, Substack, Medium |

These platforms have anti-bot protection, login walls, or complex JS rendering. The API handles all of it automatically.

## Agent Compatibility

| Agent | Install Path | Command |
|-------|-------------|---------|
| **Claude Code** | `~/.claude/skills/` | `git clone ... ~/.claude/skills/website2markdown` |
| **Codex CLI** | `~/.codex/skills/` | `git clone ... ~/.codex/skills/website2markdown` |
| **Gemini CLI** | `~/.gemini/skills/` | `git clone ... ~/.gemini/skills/website2markdown` |
| **OpenClaw** | ClawHub | `npx clawhub@latest install website2markdown` |
| **Cursor** | Project `.claude/skills/` | `git clone ... .claude/skills/website2markdown` |
| **Windsurf** | Project `.claude/skills/` | `git clone ... .claude/skills/website2markdown` |

Follows the [AgentSkills](https://agentskills.io/specification) open standard.

## Skills vs MCP Server

**Have a terminal?** Use Skills (this repo) — faster, richer context.
**No terminal (Claude Desktop, IDE only)?** Use the [MCP Server](https://www.npmjs.com/package/@digidai/mcp-website2markdown).

| | Skills | MCP Server |
|---|---|---|
| **Latency** | Direct HTTP (fastest) | MCP protocol overhead |
| **Context** | Full patterns, error handling, 21 adapters | Tool schema only |
| **Install** | `git clone` (one command) | `npm install -g` + JSON config |
| **Update** | `git pull` | `npm update -g` |

## Project Structure

```
website2markdown-skills/
├── SKILL.md                 # Core skill — agent reads this
├── references/
│   ├── advanced-apis.md     # Batch, Extract, DeepCrawl, Jobs docs
│   └── platform-adapters.md # 21 platform adapter details
└── assets/
    └── recipes.md           # Copy-paste curl recipes
```

## Contributing

1. Fork → edit → test by copying to your agent's skills dir → PR
2. Keep `SKILL.md` under 250 lines (it's the hub, not the encyclopedia)
3. API details go in `references/`, curl examples in `assets/recipes.md`

## Links

- **Live API**: [md.genedai.me](https://md.genedai.me)
- **Source code**: [Digidai/website2markdown](https://github.com/Digidai/website2markdown)
- **MCP Server**: [@digidai/mcp-website2markdown](https://www.npmjs.com/package/@digidai/mcp-website2markdown)
- **llms.txt**: [md.genedai.me/llms.txt](https://md.genedai.me/llms.txt)

## License

[MIT](LICENSE)
````
