⏳ This skill is pending AI review.

Scores will appear once the review pipeline completes.

v1.0.0

website2markdown

digidai@digidai⭐ 2 stars· last commit 7mo ago· 0 open issues

Convert web pages to clean Markdown using the Website2Markdown API (md.genedai.me). Use when you need to fetch, read, extract, or analyze web page content — especially when WebFetch fails, returns incomplete content, or the target is JS-heavy/paywalled. Supports 21 platform adapters (WeChat, Zhihu, Feishu, Twitter/X, etc.), batch processing, structured extraction, and deep crawling.

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

—/10

// RATINGS

⭐GitHub Stars
⭐ 2 on GitHubGitHub ↗

New / niche

🟢ProSkills Score
—
📍

Not yet listed on ClawHub or SkillsMP

// README

website2markdown-skills

Convert any web page to clean Markdown from your AI agent. One command install, auto-discovered.

License: MIT API Source npm

Quick Install

Pick your agent, run one command:

# Claude Code
git clone https://github.com/Digidai/website2markdown-skills ~/.claude/skills/website2markdown

# OpenAI Codex CLI
git clone https://github.com/Digidai/website2markdown-skills ~/.codex/skills/website2markdown

# Gemini CLI
git clone https://github.com/Digidai/website2markdown-skills ~/.gemini/skills/website2markdown

# OpenClaw (via ClawHub)
npx clawhub@latest install website2markdown

# Cursor / Windsurf / Other agents
git clone https://github.com/Digidai/website2markdown-skills .claude/skills/website2markdown

That's it. Open a new session and the skill is auto-discovered.

Verify

You: Read this article: https://example.com
Agent: [Uses md.genedai.me API → returns clean Markdown]

Update

cd ~/.claude/skills/website2markdown && git pull

What It Does

Teaches your AI agent to convert any URL to clean Markdown — JS-heavy SPAs, paywalled content, anti-bot pages, and 21 Chinese/international platforms.

You: Read this WeChat article: https://mp.weixin.qq.com/s/abc123

Agent: [Calls md.genedai.me with force_browser=true]
       # 深度解析:大模型在企业的落地实践
       > 作者:张三 | 2026-03-25
       ## 核心观点
       1. 大模型的应用场景正在从实验室走向生产环境...
You: Compare these 3 blog posts and summarize the differences

Agent: [Uses batch API to fetch all 3 in one request, then compares]
You: Crawl the Cloudflare docs and save as Markdown files

Agent: [Uses deep crawl API with BFS, saves each page]

Capabilities

APIWhat It Does
GET /<url>?raw=trueConvert one URL to Markdown
POST /api/batchBatch convert up to 10 URLs
POST /api/extractExtract structured data (CSS/XPath/Regex)
POST /api/deepcrawlCrawl a site with BFS or keyword scoring
POST /api/jobsAsync task queue for long-running work

21 Platform Adapters

Chinese PlatformsInternational
WeChat, Zhihu, Feishu/Lark, Yuque, Juejin, CSDN, 36Kr, Toutiao, Weibo, NetEaseTwitter/X, Reddit, Notion, GitHub, Substack, Medium

These platforms have anti-bot protection, login walls, or complex JS rendering. The API handles all of it automatically.

Agent Compatibility

AgentInstall PathCommand
Claude Code~/.claude/skills/git clone ... ~/.claude/skills/website2markdown
Codex CLI~/.codex/skills/git clone ... ~/.codex/skills/website2markdown
Gemini CLI~/.gemini/skills/git clone ... ~/.gemini/skills/website2markdown
OpenClawClawHubnpx clawhub@latest install website2markdown
CursorProject .claude/skills/git clone ... .claude/skills/website2markdown
WindsurfProject .claude/skills/git clone ... .claude/skills/website2markdown

Follows the AgentSkills open standard.

Skills vs MCP Server

Have a terminal? Use Skills (this repo) — faster, richer context. No terminal (Claude Desktop, IDE only)? Use the MCP Server.

SkillsMCP Server
LatencyDirect HTTP (fastest)MCP protocol overhead
ContextFull patterns, error handling, 21 adaptersTool schema only
Installgit clone (one command)npm install -g + JSON config
Updategit pullnpm update -g

Project Structure

website2markdown-skills/
├── SKILL.md                 # Core skill — agent reads this
├── references/
│   ├── advanced-apis.md     # Batch, Extract, DeepCrawl, Jobs docs
│   └── platform-adapters.md # 21 platform adapter details
└── assets/
    └── recipes.md           # Copy-paste curl recipes

Contributing

  1. Fork → edit → test by copying to your agent's skills dir → PR
  2. Keep SKILL.md under 250 lines (it's the hub, not the encyclopedia)
  3. API details go in references/, curl examples in assets/recipes.md

Links

License

MIT

// HOW IT'S BUILT

// FILES

2 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • SKILL.md7.9 KBupstream ↗
    ---
    name: website2markdown
    description: Convert web pages to clean Markdown using the Website2Markdown API (md.genedai.me). Use when you need to fetch, read, extract, or analyze web page content — especially when WebFetch fails, returns incomplete content, or the target is JS-heavy/paywalled. Supports 21 platform adapters (WeChat, Zhihu, Feishu, Twitter/X, etc.), batch processing, structured extraction, and deep crawling.
    version: "1.0.0"
    license: MIT
    allowed-tools: Bash, Read, Write, WebFetch
    user-invocable: true
    metadata: {"openclaw": {"requires": {"bins": ["curl"]}, "primaryEnv": "API_TOKEN", "emoji": "📄", "homepage": "https://github.com/Digidai/website2markdown-skills"}}
    ---
    
    # Website2Markdown
    
    Convert any web page to clean Markdown via the `md.genedai.me` API.
    
    ## When to Use This Skill
    
    **ALWAYS prefer this skill over WebFetch when:**
    - WebFetch fails, returns blocked/incomplete content, or redirects
    - Target is a JS-heavy SPA (React, Vue, Angular apps)
    - Content is behind paywalls or anti-bot protection
    - Target is a Chinese platform (WeChat, Zhihu, Feishu, Yuque, Juejin, CSDN, 36Kr, Toutiao, Weibo)
    - You need batch conversion of multiple URLs
    - You need structured data extraction from pages
    - You need to crawl/spider a site
    
    **Also use when:**
    - User asks to read/fetch/scrape a web page
    - User wants to convert a URL to Markdown
    - User needs content from social platforms (Twitter/X, Reddit, Telegram)
    
    ## Decision Tree
    
    ```
    User provides a URL
      │
      ├─ Single page, simple site ──▶ Try WebFetch first
      │     └─ WebFetch fails? ──▶ Use website2markdown
      │
      ├─ Chinese platform / JS-heavy / paywalled ──▶ Use website2markdown directly
      │
      ├─ Multiple URLs needed ──▶ Use website2markdown batch API
      │
      ├─ Need structured fields (title, author, price) ──▶ Use extract API
      │
      └─ Need to crawl entire site/section ──▶ Use deepcrawl API
    ```
    
    ## Quick Reference
    
    ### Single URL → Markdown
    
    ```bash
    curl -s "https://md.genedai.me/https://example.com/page?raw=true"
    ```
    
    Bare domain also works (auto-prepends `https://`):
    
    ```bash
    curl -s "https://md.genedai.me/example.com/page?raw=true"
    ```
    
    Alternative via Accept header:
    
    ```bash
    curl -s -H "Accept: text/markdown" "https://md.genedai.me/https://example.com/page"
    ```
    
    **IMPORTANT:** Always use `?raw=true` or Accept header to get plain text. Without it, you get an HTML preview page.
    
    ### Output Formats
    
    | Parameter | Output | When to use |
    |-----------|--------|-------------|
    | `?raw=true` | Plain Markdown | Default for reading content |
    | `?format=json&raw=true` | JSON `{url, title, markdown, method, timestamp}` | When you need metadata |
    | `?format=html&raw=true` | Cleaned HTML | When you need HTML structure |
    | `?format=text&raw=true` | Plain text (no formatting) | When you need raw text only |
    
    ### Query Parameters
    
    | Parameter | Purpose | Example |
    |-----------|---------|---------|
    | `raw=true` | Return plain text instead of HTML preview | Always use this |
    | `selector=<css>` | Extract specific elements (max 256 chars) | `selector=article`, `selector=.content` |
    | `force_browser=true` | Force headless Chrome rendering | JS-heavy or anti-bot sites |
    | `no_cache=true` | Bypass KV cache for fresh content | When content may have changed |
    | `engine=jina` | Route through Jina Reader API | Alternative conversion engine |
    
    ## Usage Patterns
    
    ### Pattern 1: Read a Single Article
    
    ```bash
    curl -s "https://md.genedai.me/https://example.com/article?raw=true" | head -c 100000
    ```
    
    Always pipe through `head -c 100000` for potentially large pages to avoid flooding context.
    
    ### Pattern 2: Extract Specific Content via CSS Selector
    
    ```bash
    curl -s "https://md.genedai.me/https://example.com?raw=true&selector=article"
    curl -s "https://md.genedai.me/https://example.com?raw=true&selector=%23main-content"
    ```
    
    URL-encode `#` as `%23` in selectors.
    
    ### Pattern 3: JS-Heavy or Anti-Bot Sites
    
    ```bash
    curl -s "https://md.genedai.me/https://spa-app.com?raw=true&force_browser=true"
    ```
    
    ### Pattern 4: Get Structured JSON (title + content)
    
    ```bash
    curl -s "https://md.genedai.me/https://example.com?format=json&raw=true"
    ```
    
    Returns: `{"url": "...", "title": "...", "markdown": "...", "method": "...", "timestamp": "..."}`
    
    ### Pattern 5: Chinese & Social Platforms
    
    These work out of the box — the API has built-in adapters:
    
    ```bash
    # WeChat article
    curl -s "https://md.genedai.me/https://mp.weixin.qq.com/s/xxxxx?raw=true&force_browser=true"
    
    # Zhihu article
    curl -s "https://md.genedai.me/https://zhuanlan.zhihu.com/p/123456?raw=true"
    
    # Twitter/X
    curl -s "https://md.genedai.me/https://x.com/user/status/123456?raw=true&force_browser=true"
    
    # Reddit
    curl -s "https://md.genedai.me/https://reddit.com/r/programming/comments/xxx?raw=true"
    ```
    
    > For all 21 supported platforms with URL patterns and troubleshooting, load `references/platform-adapters.md`.
    
    ## Advanced APIs
    
    All advanced APIs require `Authorization: Bearer <API_TOKEN>` header.
    
    | API | Endpoint | Purpose |
    |-----|----------|---------|
    | **Batch** | `POST /api/batch` | Convert up to 10 URLs in one request |
    | **Extract** | `POST /api/extract` | Structured field extraction (CSS/XPath/Regex) |
    | **Deep Crawl** | `POST /api/deepcrawl` | Crawl a site with filtering and scoring |
    | **Jobs** | `POST /api/jobs` | Async task queue for long-running operations |
    | **Health** | `GET /api/health` | Service health check (no auth needed) |
    | **llms.txt** | `GET /llms.txt` | API description for AI discoverability |
    
    ### Batch API (quick example)
    
    ```bash
    curl -s -X POST "https://md.genedai.me/api/batch" \
      -H "Authorization: Bearer <API_TOKEN>" \
      -H "Content-Type: application/json" \
      -d '{"urls": ["https://example.com/a", "https://example.com/b"]}'
    ```
    
    ### Extract API (quick example)
    
    ```bash
    curl -s -X POST "https://md.genedai.me/api/extract" \
      -H "Authorization: Bearer <API_TOKEN>" \
      -H "Content-Type: application/json" \
      -d '{"strategy": "css", "url": "https://example.com", "schema": {"fields": [{"name": "title", "selector": "h1", "type": "text"}]}}'
    ```
    
    ### Deep Crawl API (quick example)
    
    ```bash
    curl -s -X POST "https://md.genedai.me/api/deepcrawl" \
      -H "Authorization: Bearer <API_TOKEN>" \
      -H "Content-Type: application/json" \
      -d '{"seed": "https://docs.example.com", "max_depth": 2, "max_pages": 20, "strategy": "bfs"}'
    ```
    
    > For complete API documentation with all parameters, response formats, and examples, load `references/advanced-apis.md`.
    > For copy-paste ready curl commands, load `assets/recipes.md`.
    
    ## Authentication Matrix
    
    | Route | Auth Required? |
    |-------|---------------|
    | `/<url>?raw=true` | No (public) |
    | `/api/batch` | Yes (`API_TOKEN`) |
    | `/api/extract` | Yes (`API_TOKEN`) |
    | `/api/deepcrawl` | Yes (`API_TOKEN`) |
    | `/api/jobs/*` | Yes (`API_TOKEN`) |
    | `/api/health` | No |
    
    **Token format:** `Authorization: Bearer <token>` header, or `?token=<token>` query parameter.
    
    ## Supported Platforms (21 Adapters)
    
    | Category | Platforms |
    |----------|----------|
    | **Chinese** | WeChat, Zhihu, Feishu/Lark, Yuque, Juejin, CSDN, 36Kr, Toutiao, Weibo, NetEase |
    | **Social** | Twitter/X, Reddit, Telegram |
    | **International** | GitHub, Substack, Medium |
    | **Academic/Productivity** | arxiv, Wikipedia, YouTube, Google Docs, Notion |
    
    > For URL patterns, special handling details, and troubleshooting per platform, load `references/platform-adapters.md`.
    
    ## Response Headers
    
    | Header | Meaning |
    |--------|---------|
    | `X-Markdown-Method` | `native`, `readability+turndown`, or `browser+readability+turndown` |
    | `X-Cache-Status` | `HIT` or `MISS` |
    | `X-Browser-Rendered` | `true` if headless Chrome was used |
    | `X-Paywall-Detected` | `true` if paywall was triggered |
    | `X-Markdown-Fallbacks` | Comma-separated fallback strategies applied |
    
    ## Implementation Notes for Agents
    
    1. **Always use `curl -s`** — silent mode suppresses progress output
    2. **Always add `?raw=true`** — without it you get an HTML preview page, not markdown
    3. **Pipe through `head -c 100
  • README.mdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/digidai-website2markdown-skills.md

// REPO STATS

2 stars
0 open issues
Last commit: 7mo ago