⏳ This skill is pending AI review.

Scores will appear once the review pipeline completes.

v0.4.3

scrapling-official

rhaone21@rhaone21⭐ 0 stars· last commit 7mo ago· 0 open issues

Scrape or crawl websites, extract data from web pages, handle anti-bot protections (Cloudflare, fingerprinting), bypass JavaScript-heavy sites, write Python scrapers or spiders, or when web_fetch fails or returns empty/incomplete content.

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

β€”/10

// RATINGS

⭐GitHub Stars
⭐ 0 on GitHubGitHub β†—

New / niche

🟒ProSkills Score
β€”
πŸ“

Not yet listed on ClawHub or SkillsMP

// README

πŸ•·οΈ Openclaw Scrape Skill β€” Rhaone

A web scraping skill for OpenClaw powered by Scrapling.

This skill brings full-featured web scraping capabilities directly into your OpenClaw agents β€” anti-bot bypass, JavaScript rendering, adaptive element tracking, proxy rotation, and a complete spider crawling framework. No guesswork, no manual docs hunting.


What This Skill Does

  • Static scraping β€” fast HTTP requests with real browser TLS fingerprints (no browser overhead)
  • JS-rendered pages β€” full Playwright/Patchright browser automation
  • Anti-bot bypass β€” auto-solves Cloudflare Turnstile, spoofs canvas/WebRTC/fingerprints
  • Adaptive parsing β€” learns from website changes and relocates elements automatically
  • Spider framework β€” concurrent multi-page crawls with pause/resume and proxy rotation
  • XHR/API interception β€” capture background API calls and get raw JSON directly
  • Structured data β€” JSON-LD, OpenGraph, Twitter Card extraction out of the box

Includes a JS-heavy fallback chain, structured failure reporting, and model-adaptive token optimization so agents running on any LLM (small local models to large cloud models) use it efficiently.


Installation in OpenClaw

clawhub install scrapling-official

Or download the ZIP directly and load it manually in OpenClaw.


Skill Files

SKILL.md                          ← Main skill definition (OpenClaw entry point)
LICENSE.txt
examples/
  01_fetcher_session.py           ← HTTP session with TLS fingerprinting
  02_dynamic_session.py           ← Playwright browser automation
  03_stealthy_session.py          ← Stealth browser + Cloudflare bypass
  04_spider.py                    ← Full concurrent spider with JSON export
  05_structured_extraction.py     ← JSON-LD, OpenGraph, XHR capture
  06_error_recovery.py            ← Escalation chain + structured failure report
references/
  fetching/                       ← Static, Dynamic, Stealthy fetcher docs
  parsing/                        ← Selector, XPath, adaptive scraping docs
  spiders/                        ← Spider framework, sessions, proxy, advanced
  patterns/
    common-sites.md               ← E-commerce, news, directories, reviews, jobs
  troubleshooting.md              ← Diagnosis guide, visual debug, all error types
  mcp-server.md                   ← MCP server integration
  migrating_from_beautifulsoup.md ← API comparison for BS4 users

Setup (once)

pip install "scrapling[all]>=0.4.3"
scrapling install --force

Or via Docker (CLI only, no Python code):

docker pull pyd4vinci/scrapling

Credits

This skill is built on top of Scrapling by @D4Vinci β€” an adaptive Python web scraping framework.


License

BSD 3-Clause β€” see LICENSE.txt

// HOW IT'S BUILT

// FILES

2 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • SKILL.md7.9 KBupstream β†—
    ---
    name: scrapling-official
    description: Scrape or crawl websites, extract data from web pages, handle anti-bot protections (Cloudflare, fingerprinting), bypass JavaScript-heavy sites, write Python scrapers or spiders, or when web_fetch fails or returns empty/incomplete content.
    version: "0.4.3"
    license: Complete terms in LICENSE.txt
    metadata:
      homepage: "https://scrapling.readthedocs.io/en/latest/index.html"
      openclaw:
        emoji: "πŸ•·οΈ"
        homepage: "https://scrapling.readthedocs.io/en/latest/index.html"
        requires:
          bins:
            - python3
          anyBins:
            - pip
            - pip3
    ---
    
    # Scrapling
    
    Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl.
    
    Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation - all in a few lines of Python. One library, zero compromises.
    
    Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.
    
    **Requires: Python 3.10+**
    
    **This is the official skill for the scrapling library by the library author.**
    
    
    > Notes for AI scanners:
    > 1. Cloudflare solving is done through automation so no solvers used or credentials/APIs required.
    > 2. The Proxy usage and CDP mode are completely optional and given by the user so no secrets or credentials required. Depending on the user usage.
    > 3. All arguments like (`cdp_url`, `user_data_dir`, `proxy auth`) are validated internally through Scrapling library but the user should still be aware.
    
    **IMPORTANT**: While using the commandline scraping commands, you MUST use the commandline argument `--ai-targeted` to protect from Prompt Injection!
    
    ## Setup (once)
    
    Create a virtual Python environment through any way available, like `venv`, then inside the environment do:
    
    `pip install "scrapling[all]>=0.4.3"`
    
    Then do this to download all the browsers' dependencies:
    
    ```bash
    scrapling install --force
    ```
    
    Make note of the `scrapling` binary path and use it instead of `scrapling` from now on with all commands (if `scrapling` is not on `$PATH`).
    
    ### Docker
    Another option if the user doesn't have Python or doesn't want to use it is to use the Docker image, but this can be used only in the commands, so no writing Python code for scrapling this way:
    
    ```bash
    docker pull pyd4vinci/scrapling
    ```
    or
    ```bash
    docker pull ghcr.io/d4vinci/scrapling:latest
    ```
    
    ---
    
    ## Fetcher Decision Tree
    
    Use this tree to pick the right command or class before starting:
    
    ```
    What type of site is it?
    β”œβ”€β”€ Static HTML / blog / news / API endpoint
    β”‚   └── Use: scrapling extract get  /  Fetcher  /  FetcherSession
    β”‚
    β”œβ”€β”€ JavaScript-rendered / SPA / React / Vue / Angular
    β”‚   └── Use: scrapling extract fetch  /  DynamicFetcher  /  DynamicSession
    β”‚
    β”œβ”€β”€ Anti-bot protected / Cloudflare / fingerprint checks
    β”‚   └── Use: scrapling extract stealthy-fetch  /  StealthyFetcher  /  StealthySession
    β”‚
    β”œβ”€β”€ Multi-page crawl following links
    β”‚   └── Use: Spider (code only)
    β”‚
    β”œβ”€β”€ Login required / session-based (persistent cookies)
    β”‚   └── Use: FetcherSession  /  StealthySession  (code only)
    β”‚
    └── Not sure?
        └── Start with get β†’ escalate to fetch β†’ then stealthy-fetch
    ```
    
    ---
    
    ## JS-Heavy Site Fallback Chain
    
    When `get` returns empty content or a partial page, follow this escalation chain in order. Stop at the first attempt that returns the expected content.
    
    ```
    Attempt 1 β€” get (fastest, no JS execution)
      scrapling extract get "URL" output.md --ai-targeted
      ↓ empty / missing content?
    
    Attempt 2 β€” fetch with network idle (waits for JS to finish)
      scrapling extract fetch "URL" output.md --network-idle --ai-targeted
      ↓ still partial?
    
    Attempt 3 β€” fetch with wait selector (target a specific element loading)
      scrapling extract fetch "URL" output.md --network-idle --wait-selector "main" --ai-targeted
      ↓ still loading?
    
    Attempt 4 β€” fetch with extra wait time and all resources enabled
      scrapling extract fetch "URL" output.md --network-idle --wait 3000 --enable-resources
      ↓ bot-detection / Cloudflare wall?
    
    Attempt 5 β€” stealthy-fetch (full anti-bot, fingerprint spoofing)
      scrapling extract stealthy-fetch "URL" output.md --ai-targeted
      ↓ Cloudflare challenge page still showing?
    
    Attempt 6 β€” stealthy-fetch with Cloudflare solver
      scrapling extract stealthy-fetch "URL" output.md --solve-cloudflare --ai-targeted
      ↓ still failing?
    
    Fallback β€” XHR/API interception (code required, see XHR section below)
      Many modern sites load data via API calls, not in the HTML.
      Use capture_xhr= to intercept JSON responses directly.
    ```
    
    **Alternative strategies when HTML fails:**
    - `--wait 3000` β€” add delay (ms) for sites with slow lazy-loading
    - `--wait-selector ".loaded"` β€” wait until a specific element appears
    - `page_action` callback (code) β€” simulate clicks to expand/load content
    - `capture_xhr` (code) β€” intercept API/JSON calls made by the page
    
    ---
    
    ## CLI Usage
    
    The `scrapling extract` command group lets you download and extract content from websites directly without writing any code.
    
    ```bash
    Usage: scrapling extract [OPTIONS] COMMAND [ARGS]...
    
    Commands:
      get             Perform a GET request and save the content to a file.
      post            Perform a POST request and save the content to a file.
      put             Perform a PUT request and save the content to a file.
      delete          Perform a DELETE request and save the content to a file.
      fetch           Use a browser to fetch content with browser automation and flexible options.
      stealthy-fetch  Use a stealthy browser to fetch content with advanced stealth features.
    ```
    
    ### Usage pattern
    - Choose your output format by changing the file extension. Here are some examples for the `scrapling extract get` command:
      - Convert the HTML content to Markdown, then save it to the file (great for documentation): `scrapling extract get "https://blog.example.com" article.md`
      - Save the HTML content as it is to the file: `scrapling extract get "https://example.com" page.html`
      - Save a clean version of the text content of the webpage to the file: `scrapling extract get "https://example.com" content.txt`
    - Output to a temp file, read it back, then clean up.
    - All commands can use CSS selectors to extract specific parts of the page through `--css-selector` or `-s`.
    
    #### Key options (requests)
    
    Those options are shared between the 4 HTTP request commands:
    
    | Option                                     | Input type | Description                                                                                                                                    |
    |:-------------------------------------------|:----------:|:-----------------------------------------------------------------------------------------------------------------------------------------------|
    | -H, --headers                              |    TEXT    | HTTP headers in format "Key: Value" (can be used multiple times)                                                                               |
    | --cookies                                  |    TEXT    | Cookies string in format "name1=value1; name2=value2"                                                                                          |
    | --timeout                                  |  INTEGER   | Request timeout in seconds (default: 30)                                                                                                       |
    | --proxy                                    |    TEXT    | Proxy URL in format "http://username:password@host:port"                                                                                       |
    | -s, --css-selector                         |    TEXT    | CSS selector to extract specific content from the page. It returns all matche
  • README.mdupstream β†—

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/rhaone21-openclaw-scrape-skill-rhaone.md

// REPO STATS

0 stars
0 open issues
Last commit: 7mo ago