⏳ This skill is pending AI review.

Scores will appear once the review pipeline completes.

version unknown

x-twitter-scraper

rrrrrredy@rrrrrredy⭐ 1 stars· last commit 4mo ago· 0 open issues

Use when the user needs public X/Twitter data without login: user profile lookup, recent timeline up to about 20 tweets, full tweet content including Note Tweet long-form, pinned tweet lookup, or JSON export. Trigger for check Twitter/X user, view recent tweets, scrape public X profile, and fetch a specific tweet. Not for full history timelines, authenticated/private content, or advanced search requiring login.

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

—/10

// RATINGS

⭐GitHub Stars
⭐ 1 on GitHubGitHub ↗

New / niche

🟢ProSkills Score
—
📍

Not yet listed on ClawHub or SkillsMP

// README

x-twitter-scraper

X/Twitter public data scraper — no login required.

OpenClaw Skill — works with OpenClaw AI agents

What It Does

Scrapes public X/Twitter data using the GraphQL API with guest tokens — no account needed. Supports fetching user profiles, recent timelines (up to ~20 tweets), and full single-tweet content including Note Tweet long-form text. Outputs clean text or structured JSON.

Quick Start

openclaw skill install x-twitter-scraper
# Or:
git clone https://github.com/rrrrrredy/x-twitter-scraper.git ~/.openclaw/skills/x-twitter-scraper

Run dependency setup on first use:

bash scripts/setup.sh
pip install httpx -q

Features

  • No login required: guest token authentication for public data
  • User profiles: followers, following, bio, pinned tweets
  • Timeline scraping: up to ~20 most recent tweets per user
  • Full tweet content: includes Note Tweet long-form text
  • JSON output: add --json flag for structured data
  • Auto-updatable: GraphQL query IDs refreshable via setup script
  • Hard stop protection: auto-stops after 3 consecutive failures

Usage

Trigger with natural language:

  • "查 @elonmusk 的 X 主页"
  • "看 @某人 最新推文"
  • "获取这条推文的完整内容"
  • "查 X 动态" / "看推特"

CLI examples:

python3 scripts/x_scraper.py profile _LuoFuli
python3 scripts/x_scraper.py timeline elonmusk --count 5
python3 scripts/x_scraper.py tweet 2034379957913129140
python3 scripts/x_scraper.py profile _LuoFuli --json

Project Structure

x-twitter-scraper/
├── SKILL.md              # Skill definition and workflow
├── scripts/
│   ├── setup.sh          # Dependency setup & query ID extraction
│   └── x_scraper.py      # Main scraper CLI tool
├── references/
│   └── .gitkeep
└── .gitignore

Requirements

  • OpenClaw agent runtime
  • Python 3 with httpx (pip install httpx)
  • Upstream proxy for X API access (set X_PROXY env var)

License

MIT

// HOW IT'S BUILT

// FILES

7 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • SKILL.md4.0 KB87b18a5b6b93upstream ↗
    ---
    name: x-twitter-scraper
    description: 'Use when the user needs public X/Twitter data without login: user profile lookup, recent timeline up to about 20 tweets, full tweet content including Note Tweet long-form, pinned tweet lookup, or JSON export. Trigger for check Twitter/X user, view recent tweets, scrape public X profile, and fetch a specific tweet. Not for full history timelines, authenticated/private content, or advanced search requiring login.'
    ---
    
    # x-twitter-scraper 1.0.0
    
    ## 首次使用
    
    首次使用前请运行依赖检测脚本:
    ```bash
    bash scripts/setup.sh
    ```
    > Agent 会在首次触发时自动执行此脚本,通常无需手动操作。
    
    通过 X GraphQL API + guest token,**无需账号**抓取 X/Twitter 公开数据。
    
    ---
    
    ## ⚠️ Gotchas
    
    > 遇到问题先看这里。
    
    ⚠️ **沙箱必须走代理** → 沙箱直连 x.com 可能超时。使用前设置环境变量 `X_PROXY`(如 `export X_PROXY=http://your-proxy:3128`),未设置时脚本直连。
    
    ⚠️ **普通 UA 获取 guest token 返回 403** → 必须用 Android UA(`TwitterAndroid/10.21.0-release.0`),脚本已处理,不要修改 UA。
    
    ⚠️ **Timeline 只返回最近约 20 条** → X GraphQL API 对未登录用户限制 timeline 深度,无法翻页拉取历史。完整历史、受限内容或登录态能力不属于本 skill 范围。
    
    ⚠️ **搜索(search 命令)guest token 下几乎无效** → SearchTimeline endpoint 对 guest token 高度受限,通常返回空或 404。不要请求或使用 X 登录 cookie、auth token、ct0 或用户浏览器登录态;改用公开 web search 降级。
    
    ⚠️ **GraphQL query ID 可能过期** → X 会不定期更新 endpoint 的 query ID。失效时删除 `scripts/.x_config.json`,升级 twikit(`pip install -U twikit`),再重新运行 `bash scripts/setup.sh` 即可自动提取最新值。
    
    ⚠️ **置顶推文不在 timeline 里** → 需用 profile 接口取 `pinned_tweet_ids_str`,再用 tweet 命令逐条获取。
    
    ---
    
    ## 🛑 Hard Stop
    
    同一工具调用失败超过 3 次,立即停止,列出失败原因,标记 **"需要人工介入"**,等待人工确认。
    
    ---
    
    ## 场景映射表
    
    | 用户说 | 执行命令 |
    |--------|---------|
    | 查 @_LuoFuli 的 X 主页 | `python3 x_scraper.py profile _LuoFuli` |
    | 看 @某人 最新推文 | `python3 x_scraper.py timeline 某人 --count 10` |
    | 获取这条推文的完整内容 | `python3 x_scraper.py tweet <tweet_id>` |
    | 查置顶推文 | 先 `profile` 取 pinned_tweet_ids,再 `tweet <id>` |
    | 搜推特关键词 | guest search 通常受限;改用公开 web search,不请求登录态 |
    | 拿 JSON 数据做后续处理 | 任意命令加 `--json` 参数 |
    
    ---
    
    ## 快速开始
    
    ```bash
    SKILL_DIR=$(find ~/.openclaw/skills ~/.openclaw/workspace/skills -name "x_scraper.py" 2>/dev/null | head -1 | xargs dirname)
    pip install httpx -q
    
    # 查用户
    python3 $SKILL_DIR/x_scraper.py profile _LuoFuli
    
    # 查最新推文
    python3 $SKILL_DIR/x_scraper.py timeline elonmusk --count 5
    
    # 查单条完整推文(含长文)
    python3 $SKILL_DIR/x_scraper.py tweet 2034379957913129140
    
    # JSON 输出
    python3 $SKILL_DIR/x_scraper.py profile _LuoFuli --json
    ```
    
    ---
    
    ## GraphQL Endpoints 参考
    
    Query IDs 由 `setup.sh` 运行时生成到 `scripts/.x_config.json`。支持的 endpoints:
    - 用户信息(by handle / by ID)
    - 用户 Timeline / 推文+回复
    - 单条推文(含 Note Tweet 长文)
    - 搜索 Timeline
    - 关注者 / 正在关注
    
    更新方式(query ID 过期时):
    ```bash
    rm scripts/.x_config.json
    pip install -U twikit
    bash scripts/setup.sh
    ```
    
    ---
    
    ## 技术原理
    
    1. POST `api.twitter.com/1.1/guest/activate.json`(Android UA)→ 获取 guest token
    2. GET `x.com/i/api/graphql/<query_id>/<endpoint>`(带 guest token + Bearer)→ 拿数据
    3. 必须走上游代理(沙箱直连超时)
    
    依赖:`httpx`(`pip install httpx`)
    
    ---
    
    ## Changelog
    
    ### 1.0.0
    - Initial release. Guest token approach for scraping public X/Twitter data without login.
    
  • README.md2.0 KBa62ddfbc9372upstream ↗
    # x-twitter-scraper
    
    X/Twitter public data scraper — no login required.
    
    > OpenClaw Skill — works with [OpenClaw](https://github.com/openclaw/openclaw) AI agents
    
    ## What It Does
    
    Scrapes public X/Twitter data using the GraphQL API with guest tokens — no account needed. Supports fetching user profiles, recent timelines (up to ~20 tweets), and full single-tweet content including Note Tweet long-form text. Outputs clean text or structured JSON.
    
    ## Quick Start
    
    ```bash
    openclaw skill install x-twitter-scraper
    # Or:
    git clone https://github.com/rrrrrredy/x-twitter-scraper.git ~/.openclaw/skills/x-twitter-scraper
    ```
    
    Run dependency setup on first use:
    ```bash
    bash scripts/setup.sh
    pip install httpx -q
    ```
    
    ## Features
    
    - **No login required**: guest token authentication for public data
    - **User profiles**: followers, following, bio, pinned tweets
    - **Timeline scraping**: up to ~20 most recent tweets per user
    - **Full tweet content**: includes Note Tweet long-form text
    - **JSON output**: add `--json` flag for structured data
    - **Auto-updatable**: GraphQL query IDs refreshable via setup script
    - **Hard stop protection**: auto-stops after 3 consecutive failures
    
    ## Usage
    
    Trigger with natural language:
    
    - "查 @elonmusk 的 X 主页"
    - "看 @某人 最新推文"
    - "获取这条推文的完整内容"
    - "查 X 动态" / "看推特"
    
    CLI examples:
    ```bash
    python3 scripts/x_scraper.py profile _LuoFuli
    python3 scripts/x_scraper.py timeline elonmusk --count 5
    python3 scripts/x_scraper.py tweet 2034379957913129140
    python3 scripts/x_scraper.py profile _LuoFuli --json
    ```
    
    ## Project Structure
    
    ```
    x-twitter-scraper/
    ├── SKILL.md              # Skill definition and workflow
    ├── scripts/
    │   ├── setup.sh          # Dependency setup & query ID extraction
    │   └── x_scraper.py      # Main scraper CLI tool
    ├── references/
    │   └── .gitkeep
    └── .gitignore
    ```
    
    ## Requirements
    
    - OpenClaw agent runtime
    - Python 3 with `httpx` (`pip install httpx`)
    - Upstream proxy for X API access (set `X_PROXY` env var)
    
    ## License
    
    [MIT](LICENSE)
    
  • scripts/setup.sh2.0 KB7a739ca1dc6cupstream ↗
    #!/usr/bin/env bash
    # setup.sh - 首次使用依赖检测、安装、运行时配置生成
    # Usage: bash scripts/setup.sh
    
    set -e
    
    SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
    CONFIG_FILE="$SCRIPT_DIR/.x_config.json"
    
    echo "🔍 检测依赖..."
    
    MISSING=0
    
    if ! python3 -c "import httpx" 2>/dev/null; then
      echo "📦 安装 httpx..."
      pip install -q httpx
      MISSING=1
    fi
    
    if ! python3 -c "import twikit" 2>/dev/null; then
      echo "📦 安装 twikit(用于提取运行时配置)..."
      pip install -q twikit
      MISSING=1
    fi
    
    if [ "$MISSING" -eq 0 ]; then
      echo "✅ 所有依赖已就绪"
    else
      echo "✅ 依赖安装完成"
    fi
    
    # 从 twikit 包动态提取 Bearer Token + GraphQL endpoints 生成运行时配置
    if [ ! -f "$CONFIG_FILE" ]; then
      echo "📝 从 twikit 提取运行时配置..."
      _X_CONFIG_PATH="$CONFIG_FILE" python3 << 'PYEOF'
    import json, os
    from twikit.constants import TOKEN
    from twikit.client.gql import Endpoint
    
    config = {
        "bearer": TOKEN,
        "proxy": os.environ.get("X_PROXY", ""),
        "gql": {
            "UserByScreenName": Endpoint.USER_BY_SCREEN_NAME.rsplit("/i/api/graphql/", 1)[-1],
            "UserTweets": Endpoint.USER_TWEETS.rsplit("/i/api/graphql/", 1)[-1],
            "UserTweetsAndReplies": Endpoint.USER_TWEETS_AND_REPLIES.rsplit("/i/api/graphql/", 1)[-1],
            "TweetResultByRestId": Endpoint.TWEET_RESULT_BY_REST_ID.rsplit("/i/api/graphql/", 1)[-1],
            "SearchTimeline": Endpoint.SEARCH_TIMELINE.rsplit("/i/api/graphql/", 1)[-1],
        }
    }
    
    with open(os.environ["_X_CONFIG_PATH"], "w") as f:
        json.dump(config, f, indent=2)
    print("✅ 配置已写入 .x_config.json")
    PYEOF
    else
      echo "✅ 运行时配置已存在"
    fi
    
    # 最终验证
    python3 -c "import httpx; print('验证通过: httpx')"
    python3 -c "
    import json
    with open('$CONFIG_FILE') as f:
        cfg = json.load(f)
    assert cfg.get('bearer'), 'Bearer token missing'
    assert cfg.get('gql', {}).get('UserTweets'), 'GQL endpoints missing'
    print('验证通过: 运行时配置')
    "
    echo "🎉 setup 完成,可以正常使用"
    
  • scripts/x_scraper.py16 KB62cb676473b5upstream ↗
    #!/usr/bin/env python3
    """
    X/Twitter 无登录公开数据抓取脚本
    使用 Guest Token + GraphQL API,无需账号
    
    用法:
      python3 x_scraper.py profile <handle>
      python3 x_scraper.py timeline <handle> [--count 20]
      python3 x_scraper.py tweet <tweet_id>
      python3 x_scraper.py search <query> [--count 10]
    
    全局参数:
      --json      以 JSON 格式输出
      --proxy URL 覆盖默认代理(或设置 X_PROXY 环境变量)
    """
    
    import argparse
    import json
    import os
    import sys
    import httpx
    
    # ── 常量 ──────────────────────────────────────────────────────────────────────
    # 运行时配置从外部文件加载(setup.sh 生成),避免高熵字符串硬编码
    _SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
    _CONFIG_FILE = os.path.join(_SCRIPT_DIR, ".x_config.json")
    
    def _load_config():
        """从 .x_config.json 加载运行时配置(Bearer Token + GQL endpoints)"""
        if os.path.exists(_CONFIG_FILE):
            with open(_CONFIG_FILE) as f:
                return json.load(f)
        # 回退:从环境变量读取
        return {}
    
    _CFG = _load_config()
    BEARER = os.environ.get("X_BEARER_TOKEN", _CFG.get("bearer", ""))
    ANDROID_UA = "TwitterAndroid/10.21.0-release.0 (310210000-r-0) ONEPLUS+A3010/9 (OnePlus;ONEPLUS+A3010;OnePlus;OnePlus3;0;;1;2016)"
    # 代理地址:从环境变量读取,未设置时留空(直连)
    DEFAULT_PROXY = os.environ.get("X_PROXY", _CFG.get("proxy", ""))
    
    # GraphQL endpoint query IDs(运行时从配置加载)
    GQL = _CFG.get("gql", {
        "UserByScreenName":    "",
        "UserTweets":          "",
        "UserTweetsAndReplies":"",
        "TweetResultByRestId": "",
        "SearchTimeline":      "",
    })
    
    USER_FEATURES = {
        "hidden_profile_likes_enabled": True,
        "hidden_profile_subscriptions_enabled": True,
        "responsive_web_graphql_exclude_directive_enabled": True,
        "verified_phone_label_enabled": False,
        "subscriptions_verification_info_is_identity_verified_enabled": True,
        "subscriptions_verification_info_verified_since_enabled": True,
        "highlights_tweets_tab_ui_enabled": True,
        "creator_subscriptions_tweet_preview_api_enabled": True,
        "responsive_web_graphql_skip_user_profile_image_extensions_enabled": False,
        "responsive_web_graphql_timeline_navigation_enabled": True,
    }
    
    TWEET_FEATURES = {
        "rweb_tipjar_consumption_enabled": True,
        "responsive_web_graphql_exclude_directive_enabled": True,
        "verified_phone_label_enabled": False,
        "creator_subscriptions_tweet_preview_api_enabled": True,
        "responsive_web_graphql_timeline_navigation_enabled": True,
        "responsive_web_graphql_skip_user_profile_image_extensions_enabled": False,
        "communities_web_enable_tweet_community_results_fetch": True,
        "c9s_tweet_anatomy_moderator_badge_enabled": True,
        "tweetypie_unmention_optimization_enabled": True,
        "responsive_web_edit_tweet_api_enabled": True,
        "graphql_is_translatable_rweb_tweet_is_translatable_enabled": True,
        "view_counts_everywhere_api_enabled": True,
        "longform_notetweets_consumption_enabled": True,
        "responsive_web_twitter_article_tweet_consumption_enabled": True,
        "tweet_awards_web_tipping_enabled": False,
        "freedom_of_speech_not_reach_fetch_enabled": True,
        "standardized_nudges_misinfo": True,
        "tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled": True,
        "rweb_video_timestamps_enabled": True,
        "longform_notetweets_rich_text_read_enabled": True,
        "longform_notetweets_inline_media_enabled": True,
        "responsive_web_enhance_cards_enabled": False,
    }
    
    
    # ── HTTP 客户端 ────────────────────────────────────────────────────────────────
    def make_client(proxy: str) -> httpx.Client:
        if proxy:
            transport = httpx.HTTPTransport(proxy=proxy)
            return httpx.Client(transport=transport, timeout=15, follow_redirects=True)
        return httpx.Client(timeout=15, follow_redirects=True)
    
    
    def get_guest_token(client: httpx.Client) -> str:
        """获取 guest token(必须用 Android UA,否则 403)"""
        r = client.post(
            "https://api.twitter.com/1.1/guest/activate.json",
            headers={"Authorization": f"Bearer {BEARER}", "User-Agent": ANDROID_UA},
        )
        r.raise_for_status()
        return r.json()["guest_token"]
    
    
    def gql_headers(guest_token: str) -> dict:
        return {
            "Authorization": f"Bearer {BEARER}",
            "x-guest-token": guest_token,
            "User-Agent": ANDROID_UA,
            "x-twitter-client-language": "en",
            "x-twitter-active-user": "yes",
            "Content-Type": "application/json",
        }
    
    
    def gql_get(client, endpoint_key, variables, features, guest_token):
        path = GQL[endpoint_key]
        r = client.get(
            f"https://x.com/i/api/graphql/{path}",
            params={
                "variables": json.dumps(variables),
                "features": json.dumps(features),
            },
            headers=gql_headers(guest_token),
        )
        r.raise_for_status()
        return r.json()
    
    
    # ── 解析工具 ───────────────────────────────────────────────────────────────────
    def extract_tweet_text(tweet_result: dict) -> str:
        """从 tweet result 中提取正文,优先取 note_tweet 长文"""
        legacy = tweet_result.get("legacy", {})
        note = tweet_result.get("note_tweet", {}).get("note_tweet_results", {}).get("result", {})
        if note:
            return note.get("text", legacy.get("full_text", ""))
        return legacy.get("full_text", "")
    
    
    def parse_tweet(entry) -> dict | None:
        """从 timeline entry 解析推文"""
        try:
            content = entry.get("content", {})
            item_content = content.get("itemContent", content)
            tweet_result = item_content.get("tweet_results", {}).get("result", {})
            if not tweet_result or tweet_result.get("__typename") == "TweetUnavailable":
                return None
            legacy = tweet_result.get("legacy", {})
            if not legacy:
                return None
            return {
                "id": legacy.get("id_str"),
                "created_at": legacy.get("created_at"),
                "text": extract_tweet_text(tweet_result),
                "likes": legacy.get("favorite_count", 0),
                "retweets": legacy.get("retweet_count", 0),
                "replies": legacy.get("reply_count", 0),
                "views": tweet_result.get("views", {}).get("count", "?"),
                "is_retweet": "retweeted_status_result" in legacy,
            }
        except Exception:
            return None
    
    
    def flatten_timeline(data: dict) -> list[dict]:
        """从 UserTweets 响应中提取推文列表"""
        tweets = []
        try:
            instructions = (
                data["data"]["user"]["result"]["timeline_v2"]["timeline"]["instructions"]
            )
            for inst in instructions:
                for entry in inst.get("entries", []):
                    t = parse_tweet(entry)
                    if t:
                        tweets.append(t)
                    # TimelineAddToModule(推文+回复)
                    for item in entry.get("content", {}).get("items", []):
                        t = parse_tweet(item)
                        if t:
                            tweets.append(t)
        except (KeyError, TypeError):
            pass
        return tweets
    
    
    # ── 子命令实现 ─────────────────────────────────────────────────────────────────
    def cmd_profile(args):
        with make_client(args.proxy) as client:
            gt = get_guest_token(client)
            data = gql_get(client, "UserByScreenName",
                {"screen_name": args.handle, "withSafetyModeUserFields": True},
                USER_FEATURES, gt)
    
        user = data["data"]["user"]["result"]
        legacy = user["legacy"]
        result = {
            "id": user["rest_id"],
            "handle": legacy["screen_name"],
            "name": legacy["name"],
            "bio": legacy.get("description", ""),
            "followers": legacy.get("followers_count", 0),
            "following": legacy.get("friends_count", 0),
            "tweets_count": legacy.get("statuses_count", 0),
            "verified": legacy.get("verified", False),
            "blue_verified": user.get("is_blue_verified", False),
            "location": legacy.get("location", ""),
            "created_at": legacy.get("created_at", ""),
            "profile_url": f"https://x.com/{legacy['screen_name']}",
        }
    
        if args.json:
            print(json.dumps(result, ensure_ascii=False, indent=2))
        else:
            print(f"@{result['handle']} ({result['name']})")
            print(f"ID: {result['id']}")
            print(f"Bio: {result['bio']}")
            print(f"Followers: {result['followers']:,} | Following: {result['following']:,} | Tweets: {result['tweets_count']:,}")
            print(f"Blue Verified: {result['blue_verified']} | Location: {result['location']}")
            print(f"Joined: {result['created_at']}")
    
    
    def cmd_timeline(args):
        with make_client(args.proxy) as client:
            gt = get_guest_token(client)
            # 先获取 user ID
            user_data = gql_get(client, "UserByScreenName",
                {"screen_name": args.handle, "withSafetyModeUserFields": True},
                USER_FEATURES, gt)
            user_id = user_data["data"]["user"]["result"]["rest_id"]
    
            data = gql_get(client, "UserTweets", {
                "userId": user_id,
                "count": args.count,
                "includePromotedContent": False,
                "withQuickPromoteEligibilityTweetFields": True,
                "withVoice": True,
                "withV2Timeline": True,
            }, TWEET_FEATURES, gt)
    
        tweets = flatten_timeline(data)
    
        if args.json:
            print(json.dumps(tweets, ensure_ascii=False, indent=2))
        else:
            print(f"@{args.handle} 的最近推文({len(tweets)} 条)\n")
            for t in tweets:
                print(f"[{t['created_at']}] ID:{t['id']}")
                print(f"  {t['text'][:200]}{'...' if len(t['text']) > 200 else ''}")
                print(f"  ❤️ {t['likes']}  🔁 {t['retweets']}  💬 {t['replies']}  👁 {t['views']}")
                print()
    
    
    def cmd_tweet(args):
        with make_client(args.proxy) as client:
            gt = get_guest_token(client)
            data = gql_get(client, "TweetResultByRestId", {
                "tweetId": args.tweet_id,
                "withCommunity": False,
                "includePromotedContent": False,
                "withVoice": False,
            }, TWEET_FEATURES, gt)
    
        # X API 有时返回 tweetResult,有时返回 tweet_result
        raw = data.get("data", {})
        tweet_result = (raw.get("tweetResult") or raw.get("tweet_result") or {}).get("result", {})
        legacy = tweet_result.get("legacy", {})
        result = {
            "id": legacy.get("id_str"),
            "created_at": legacy.get("created_at"),
            "text": extract_tweet_text(tweet_result),
            "likes": legacy.get("favorite_count", 0),
            "retweets": legacy.get("retweet_count", 0),
            "replies": legacy.get("reply_count", 0),
            "views": tweet_result.get("views", {}).get("count", "?"),
            "author": tweet_result.get("core", {}).get("user_results", {}).get("result", {}).get("legacy", {}).get("screen_name", ""),
        }
    
        if args.json:
            print(json.dumps(result, ensure_ascii=False, indent=2))
        else:
            print(f"Tweet by @{result['author']} [{result['created_at']}]")
            print(f"\n{result['text']}\n")
            print(f"❤️ {result['likes']}  🔁 {result['retweets']}  💬 {result['replies']}  👁 {result['views']}")
    
    
    def print_transport_error(args, exc: Exception):
        detail = str(exc) or exc.__class__.__name__
        if getattr(args, "cmd", "") == "search":
            msg = (
                f"网络请求失败: {detail}. X guest search 很容易超时或受限;不要提供登录态、cookie 或 auth token,"
                "请改用公开 web search 降级。"
            )
        else:
            msg = (
                f"网络请求失败: {detail}. 仅支持公开 X/Twitter 内容;不要提供登录态、cookie 或 auth token,"
                "可稍后重试、设置 --proxy,或改用公开 web search 验证。"
            )
    
        if getattr(args, "json", False):
            print(json.dumps({"error": msg}, ensure_ascii=False))
        else:
            print(f"错误: {msg}", file=sys.stderr)
    
    
    def cmd_search(args):
        """搜索(guest token 下功能受限,可能返回空或被 429)"""
        with make_client(args.proxy) as client:
            gt = get_guest_token(client)
            data = gql_get(client, "SearchTimeline", {
                "rawQuery": args.query,
                "count": args.count,
                "querySource": "typed_query",
                "product": "Latest",
            }, TWEET_FEATURES, gt)
    
        # 解析搜索结果(结构与 UserTweets 类似)
        tweets = []
        try:
            instructions = data["data"]["search_by_raw_query"]["search_timeline"]["timeline"]["instructions"]
            for inst in instructions:
                for entry in inst.get("entries", []):
                    t = parse_tweet(entry)
                    if t:
                        tweets.append(t)
        except (KeyError, TypeError):
            pass
    
        if not tweets:
            msg = "搜索结果为空(guest token 下搜索功能受限;不要提供登录态,改用公开 web search 降级)"
            if args.json:
                print(json.dumps({"error": msg, "tweets": []}, ensure_ascii=False))
            else:
                print(f"⚠️ {msg}")
            return
    
        if args.json:
            print(json.dumps(tweets, ensure_ascii=False, indent=2))
        else:
            print(f"搜索「{args.query}」结果({len(tweets)} 条)\n")
            for t in tweets:
                print(f"[{t['created_at']}] ID:{t['id']}")
                print(f"  {t['text'][:200]}{'...' if len(t['text']) > 200 else ''}")
                print(f"  ❤️ {t['likes']}  🔁 {t['retweets']}  💬 {t['replies']}")
                print()
    
    
    # ── CLI ───────────────────────────────────────────────────────────────────────
    def main():
        parser = argparse.ArgumentParser(description="X/Twitter 无登录数据抓取")
        parser.add_argument("--json", action="store_true", help="JSON 格式输出")
        parser.add_argument("--proxy", default=DEFAULT_PROXY, help="HTTP 代理地址")
    
        subs = parser.add_subparsers(dest="cmd", required=True)
    
        p_profile = subs.add_parser("profile", help="获取用户 profile")
        p_profile.add_argument("handle", help="Twitter handle(不含@)")
    
        p_timeline = subs.add_parser("timeline", help="获取用户推文列表")
        p_timeline.add_argument("handle", help="Twitter handle(不含@)")
        p_timeline.add_argument("--count", type=int, default=20, help="获取推文数(最多约20条)")
    
        p_tweet = subs.add_parser("tweet", help="获取单条推文完整内容")
        p_tweet.add_argument("tweet_id", help="推文 ID")
    
        p_search = subs.add_parser("search", help="关键词搜索(guest token 下受限)")
        p_search.add_argument("query", help="搜索关键词")
        p_search.add_argument("--count", type=int, default=10, help="结果数量")
    
        args = parser.parse_args()
    
        dispatch = {
            "profile": cmd_profile,
            "timeline": cmd_timeline,
            "tweet": cmd_tweet,
            "search": cmd_search,
        }
        try:
            dispatch[args.cmd](args)
        except httpx.HTTPStatusError as e:
            print(f"HTTP 错误 {e.response.status_code}: {e.response.text[:300]}", file=sys.stderr)
            sys.exit(1)
        except httpx.TransportError as e:
            print_transport_error(args, e)
            sys.exit(1)
        except Exception as e:
            print(f"错误: {e}", file=sys.stderr)
            sys.exit(1)
    
    
    if __name__ == "__main__":
        main()
    
  • references/.gitkeep0 Be69de29bb2d1upstream ↗
  • .gitignore289 B691f28939531upstream ↗
    # Python
    __pycache__/
    *.py[cod]
    *$py.class
    *.so
    *.egg-info/
    dist/
    build/
    *.egg
    .eggs/
    venv/
    .venv/
    
    # Node
    node_modules/
    npm-debug.log*
    
    # OS
    .DS_Store
    Thumbs.db
    *.swp
    *.swo
    *~
    
    # IDE
    .vscode/
    .idea/
    *.sublime-*
    
    # Environment
    .env
    .env.local
    
    # Testing
    .pytest_cache/
    htmlcov/
    .coverage
    
    
  • LICENSE1.0 KBe936eacb4056upstream ↗
    MIT License
    
    Copyright (c) 2026 rrrrrredy
    
    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:
    
    The above copyright notice and this permission notice shall be included in all
    copies or substantial portions of the Software.
    
    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    SOFTWARE.
    

Plain-text copy, no login: /skill-bundles/rrrrrredy-x-twitter-scraper.md

// REPO STATS

1 stars
0 open issues
Last commit: 4mo ago