version unknown

vision-skills

anionex@anionex⭐ 1.3k stars· last commit 1mo ago· 4 open issues

>-

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

7.5/10
Verified
Oct 12, 2026

// RATINGS

⭐GitHub Stars
⭐⭐⭐⭐⭐ 1.3k on GitHubGitHub ↗

Very popular

🟢ProSkills ScoreAI Verified
7.5/10
📍

Not yet listed on ClawHub or SkillsMP

// README

agent-vision-toolkit

X (Twitter) GitHub stars GitHub forks License: MIT

Agent Skills Extensions Shell

What it thinks is what it sees — give any text-only coding agent eyes: image Q&A, long-screenshot OCR, frontend UI restoration, and GUI automation, as a vision toolkit plus a skill, with optional drop-in integration for Codex, Claude Code, Pi, Oh My Pi, and OpenCode.

🎯 An agent's vision capability doesn't have to live in the model — it can live in the harness.

🌐 中文 | English

If your agent already runs on a text-only model such as DeepSeek but is held back by the lack of multimodality — unable to see images, with every attempt to use an image tool blocked by the system — this repository provides tools, skills, and proxy integrations that let text-only models handle visual tasks on equal or even better footing. The goal is to make the experience of using a text-model agent as seamless as using a multimodal one, and ultimately let a tool-equipped text-model agent outperform a native multimodal agent that does not use this toolkit and its methods.

This repository provides two kinds of components:

  1. Vision tool CLIs — multiple CLIs, plus a skill that teaches the agent when to use each one. Any agent that can invoke a shell can use them.
  2. Seamless integration (optional upgrade) — a transparent local proxy and single-file native plugins, so images we paste and the agent's built-in image tools both work seamlessly, with no extra tool installation or additional prompting.

All code has been verified in real Codex + DeepSeek sessions, and the same pipeline has been live-verified end-to-end in Claude Code, Pi, Oh My Pi, and OpenCode.

If this project helps you or gives you some inspiration, feel free to star🌟 & fork.

❤️ Sponsor

Want to sponsor this project? See FUNDING.md or email [email protected].

Latest Update

2026-08-18 — Skill renamed: the included agent skill is now vision-skills (formerly vision-tools), so the name describes the capability rather than the underlying tools.

2026-08-13 — Native DeepSeek Harness support is now available. The new dsh-vision-toolkit linked package brings this toolkit into DSH Web and Headless profiles as a native Profile Bundle. It provides 10 structured visual tools for intent-aware image Q&A, grounding, detection, tracing, cropping, pixel diff, long-screenshot OCR, foreground extraction, dominant-color analysis, and HTML screenshots, while adding DSH Credentials, a managed isolated runtime, previewable Artifacts, Web Settings, and Agent-scoped progressive tool exposure.

The package is tracked here as a Git submodule and maintained independently at Anionex/dsh-vision-toolkit. Clone this repository with --recurse-submodules, or run git submodule update --init --recursive in an existing checkout.

Highlights

  • More than image descriptions — it captures what the LLM actually cares about: when viewing an image, it passes along the user's or model's latest intent, producing the details needed for the current turn instead of a broad, unfocused description.
  • Both pasted images and built-in image tools work: the agent can understand images pasted directly as well as images opened through its built-in tools.
  • A battle-tested methodology for visual tasks: the included skill teaches the agent what to inspect, which tool to choose, what sequence to follow, and how to verify the final result.
  • One-sentence install: ask your agent to install it — it follows the verified flow end to end, toolkit, skill, and seamless integration included.

Use-case Playbooks

The included vision-skills skill contains complete examples that an agent can follow directly. When to use them, the order in which to call tools, and how to verify the result are all documented in the corresponding skill guides:

Use caseWhat the agent learns to do
Extract long screenshots, chat histories, and scrolling pagesFind low-content cut bands, OCR each chunk in order, preserve chat speakers/timestamps/quotes, merge only duplicated overlap, and surface risky boundaries for verification. See the Telegram reference run →
Rebuild a UI from a screenshot or designReuse project components and assets first, then combine code-native UI, extracted visuals, rendered screenshots, and visual comparison to align a page or component.

// HOW IT'S BUILT

// FILES

14 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • skills/vision-skills/SKILL.md16 KB830ac12be1a3upstream ↗
    ---
    name: vision-skills
    description: >-
      Local vision CLIs: glance (describe/ask/OCR an image), ground (locate a
      target, pixel box), detect (element inventory), trace (image to SVG
      geometry), crop (cut a pixel box to a file), and scripts/html_shot.py (HTML
      file to image). Use for any task involving an image — questions, text,
      splitting and transcribing long screenshots or chat histories, locating elements,
      comparing, rebuilding as HTML/SVG, digitizing a sketch or diagram, reading
      values off a chart, operating a GUI from screenshots — and to re-check an
      image yourself when a description you were given lacks a detail.
    ---
    
    # vision-skills
    
    Five local CLIs that give a text-only agent eyes. They read one shared
    vision config (`VISION_API_KEY` / `VISION_BASE_URL` / `VISION_MODEL` /
    `LANG`), plus the optional Python-client settings `VISION_API_PROTOCOL`,
    `VISION_REASONING_EFFORT`, and `VISION_USER_AGENT` — no extra credentials.
    
    Pick the tool by the question you are answering:
    
    | Question | Tool |
    |---|---|
    | "What does this image show / say?" | `glance` |
    | "Where is X?" — a thing you can name | `ground` |
    | "Where are all the Xs?" — every instance of a kind | `detect` |
    | "What is its exact shape, size, offset?" | `trace` |
    | "Cut this box out as its own image file" | `crop` |
    | "OCR this long screenshot / scrolling page / chat history" | `scripts/long_screenshot_ocr.py` |
    | "Extract the icon/logo foreground as transparent PNG — manual region or auto (cropped+scaled screenshots)" | `scripts/extract_fg.py` |
    | "Turn this HTML file into a viewport or full-page screenshot" | `scripts/html_shot.py` |
    | "Which colours dominate a region, and which palette value fits it?" | `scripts/dominant_colors.py` |
    | A relation none of them return — a gap, a distance between two located things | code over the pixels (Pillow) |
    
    `glance` answers what something is; `ground` and `detect` answer where.
    You give `ground` a description of a particular thing; you give `detect` a
    kind and it enumerates the instances.
    
    Both give real coordinates, but they are not pixel-exact: the box arrives
    on a 0-1000 grid and is scaled to your image, so the last pixel or few are
    not reliable. That is accurate enough to crop with, to click, to compare
    positions against. When a number has to be exact, `trace` derives it from
    the actual pixels — offsets, sizes, shapes.
    
    ## Use the provided tools before hand-rolled pixels
    
    Everything this toolkit ships a tool for, call the tool — do not rewrite
    it with Pillow in the middle of a task. The CLIs exist so the same pixel
    work is not hand-coded differently every time:
    
    - cut a box out of an image → `crop`, not `Image.open(...).crop(...)`
    - sample a region's palette → `scripts/dominant_colors.py`
    - compare two images → `scripts/pixel_diff.py`
    - vectorize to SVG → `trace`
    - locate / inventory elements → `ground` / `detect`
    - describe / OCR an image → `glance`
    - safely split, OCR, and merge a long screenshot → `scripts/long_screenshot_ocr.py`
    - HTML file to a viewport or full-page screenshot → `scripts/html_shot.py`
    
    Hand-written Pillow is only for what none of them return: a relation
    between two things you already located (a gap, a distance), a resize or
    overlay, drawing. If you catch yourself writing `.crop()`, `.convert()`,
    or histogram code where one of the tools above fits, replace it with the
    tool call — same coordinates, same box format, and the output feeds the
    next tool directly.
    
    ## glance — ask about an image
    
    ```bash
    glance <image>                                 # detailed description
    glance <image> -q "<question>"                 # targeted question (qualitative only)
    glance <image> --ocr                           # verbatim OCR
    glance <image> --region X1,Y1,X2,Y2 -q "..."   # zoom into a crop
    glance <img1> <img2> -q "..."                  # compare in ONE call
    ```
    
    When you do compare with `glance`, pass all paths to one call — separate
    calls cannot see both images, so two descriptions compared afterwards are
    two hallucination surfaces, not a comparison. `--region` uploads only the
    crop, so small text and icons become readable.
    
    But "what changed between these two?" is not a glance question. A one-word
    badge or a small shift is a rounding error to a vision model and exact to
    `scripts/pixel_diff.py`. Diff first to get the box, then `glance --region`
    that box to read what the change actually is.
    
    For a tall scrolling screenshot, do not send the whole image through one OCR
    call and accept the model's downscaling loss. Run the long-screenshot workflow,
    which finds low-content cut bands, invokes `glance` on each chunk, uses
    structured extraction for chat histories, merges only duplicated overlap, and
    writes a boundary audit:
    
    ```bash
    python3 scripts/long_screenshot_ocr.py work/page.png -o work/page.ocr.md
    python3 scripts/long_screenshot_ocr.py work/chat.png --mode chat --resume -o work/chat.ocr.md
    ```
    
    Read `references/long-screenshot-ocr.md` before using it. It defines the
    verification pass for unsafe cuts and chat-message boundaries.
    
    ## ground — locate a named target
    
    ```bash
    ground <image> "<target description>"
    ground <image> "<target>" --region X1,Y1,X2,Y2
    ```
    
    Output: `x1: .., y1: .., x2: .., y2: ..` in original-image pixels — with
    `--region` too (crop hits are mapped back).
    
    Provider-native 0-1000 boxes do not all use the same array order: Gemini uses
    `[y0, x0, y1, x1]`, while Qwen3-VL, Qwen3.5, and Qwen3.6 use
    `[x0, y0, x1, y1]`. Grounding code must select the order by model family (or
    an explicit override) before scaling to pixels; never parse every provider as
    Gemini-style `yxyx`.
    
    If several boxes come back numbered, your description matched more than
    one element rather than picking out a single thing. Narrow it with what
    distinguishes the one you mean — its text, its position, the block it sits
    in — and ask again.
    
    The box is a handle, not just an answer — it feeds the next call:
    
    ```bash
    $ ground screenshot.png "the send button"
    x1: 1067, y1: 841, x2: 1108, y2: 881
    $ glance screenshot.png --region 1067,841,1108,881 -q "is it enabled or greyed out?"
    ```
    
    That two-step is how you inspect anything too small to survive a
    full-image pass.
    
    ## detect — find every instance of a kind
    
    ```bash
    detect <image>                        # every UI element
    detect <image> "buttons"              # one kind only
    detect <image> --region X1,Y1,X2,Y2   # inside one box
    detect <image> "text" --fail-on-empty # fail if no text is detected
    ```
    
    The default category targets UI screenshots. For photographs or video frames,
    pass an explicit category such as `"objects"` or `"text"`. An empty inventory
    normally prints `no elements detected` and exits 0; `--fail-on-empty` instead
    exits 1 with a stderr diagnostic and no stdout. This reports an empty model
    result, not proof that the image has no matching elements.
    
    You name a particular thing for `ground`; you name a kind for `detect` and
    it enumerates the instances. Output is a numbered list with each item's
    visible text and box. A full-screen
    pass is a fast first draft — counts vary run to run on dense screens. For
    completeness, detect the layout blocks first, then `detect --region` each
    block.
    
    ## trace — exact shape geometry (local, no vision API)
    
    ```bash
    trace <image>                                  # b/w spline SVG to stdout
    trace <image> --polygon                        # boxy diagrams/wireframes
    trace <image> --region X1,Y1,X2,Y2 -o out.svg  # crop first
    ```
    
    Coordinates come from the actual pixels, not a model's estimate. Flat,
    high-contrast graphics only; text becomes curves (pair with `--ocr` when
    the text matters). Small images are upscaled automatically before tracing,
    so a 30px icon traces as readily as a screenshot — size is not a reason to
    skip the tool. Before shipping or reusing a traced SVG, read
    `references/restore-graphic.md` — it holds the reuse traps and the
    ship-vs-hand-write call.
    
    ## crop — cut a pixel box out of an image (local, no vision API)
    
    ```bash
    crop <image> --region X1,Y1,X2,Y2             # writes <image-stem>.crop.png next to the input
    crop <image> --region X1,Y1,X2,Y2 -o out.png
    crop <image> --region X1,Y1,X2,Y2 --scale 4   # upscale the cut-out 4x (LANCZOS) first
    ```
    
    The same X1,Y1,X2,Y2 pixel boxes `ground`/`detect` print, clamped to the
    image bounds. Once a box is worth keeping — the same crop is about to feed
    `pixel_diff`, `dominant_colors`, and `trace` in turn — cut it to a file
    once and reuse it, instead of re-cropping in memory on every call.
    `--scale N` upscales the cut-out before writing (default output name becomes
    `<image-stem>[email protected]`): for icons too small for `ground`/`trace` to see
    clearly, crop with `--scale 4`, then run `ground`/`trace` on the upscaled
    file — coordinates it returns are in the upscaled grid, divide by `N` to map
    back to the original image. Requires the optional `pillow`.
    
    ## extract_fg — icon foreground as transparent PNG: manual region or auto (local, no vision API)
    
    ```bash
    # manual: you know the region (and optionally the background colour)
    python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 -o icon.png
    python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 --mode dark          # grey/black line logos
    python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 --exclude-color '#E6E6E6'
    # auto: `crop --scale` cut-outs with the icon centred — no region needed
    crop shot.png --region X1,Y1,X2,Y2 --scale 4 -o d/icon1.png
    python3 scripts/extract_fg.py d/icon1.png d/icon2.png       # writes <stem>.clean.png next to each input
    python3 scripts/extract_fg.py d/icon1.png --disc-radius 60
    python3 scripts/extract_fg.py d/icon1.png --boxes "101,84,184,171"
    ```
    
    Manual mode keeps every sufficiently large connected component of the
    region (separate logo sub-shapes stay together; specks drop out). Auto mode
    takes a `crop --scale` cut-out with the icon centred (disc + glyph): the
    disc centre is the image centre, the disc radius defaults to
    `min(w,h)/2 * 0.6`, and the disc colour is sampled from a ring around the
    centre; that colour is excluded and the glyph is picked as the most
    saturated among the three largest coloured components (white rings,
    ripples, and text fall away), output as a 1:1 transparent PNG. When auto
    inference fails, override the radius with `--disc-radius`, or pass a
    `ground` box (in the upscaled grid) as `--boxes` to recentre and re-filter
    by overlap. Multiple images may be passed at once (auto mode).
    Requires the optional `pillow` (and `numpy` for auto mode).
    
    ## html_shot — render an HTML file to an image (local, needs a Chrome-family browser)
    
    ```bash
    python3 scripts/html_shot.py page.html                      # writes page.png, 1280x800
    python3 scripts/html_shot.py page.html --width 1440 --height 900 -o page.png
    python3 scripts/html_shot.py page.html --scale 2            # 2x pixels: small text stays readable
    python3 scripts/html_shot.py page.html --full-page           # complete scroll height, same layout viewport
    python3 scripts/html_shot.py page.html --full-page --max-pixels 40000000
    ```
    
    The visual-alignment loop: write HTML, screenshot it at the reference
    viewport, then compare it with the design. Use `pixel_diff` to locate
    material differences, not to chase a zero-difference score. Rendering
    happens in headless Chrome/Chromium/Edge — no Python dependencies. The default
    captures only the viewport. Use `--full-page` for the complete document while
    keeping `--width` and `--height` as the layout viewport, so `vh`/`svh` and
    responsive breakpoints do not change. Add `--max-pixels N` when the page height
    is untrusted. `--wait-ms N` pauses for fonts, images, or animation before
    capturing. Paths are relative to this skill's own directory.
    
    ## pixel_diff — where two images differ (local, no vision API)
    
    ```bash
    python3 scripts/pixel_diff.py <a> <b>      # path is relative to this skill dir
    ```
    
    Prints an overall difference percentage plus the worst regions as `x1: ..`
    boxes you can feed straight into `glance --region`. Exact where a vision
    model rounds off.
    
    ## dominant_colors — a region's palette, and the exact value among candidates (local, no vision API)
    
    ```bash
    python3 scripts/dominant_colors.py <image> --region X1,Y1,X2,Y2          # top colour clusters + shares
    python3 scripts/dominant_colors.py <image> --region X1,Y1,X2,Y2 \
      --candidates '#F9FAFA,#F5F5F5,#F3F3F3,#EDEDED'                        # pick the best candidate
    ```
    
    A vision model names a colour ("light gray") but not its value. The first
    mode downsamples, quantizes, and merges near-duplicates to list the region's
    significant colours with the share each owns — the histogram shows which
    colour is the background and which is the accent. Given the candidate palette
    your label implies, the second mode scores each candidate by how close the
    region's pixels are to it and prints the winner. Take the value from here,
    never from `glance`'s prose. Paths are relative to this skill's own
    directory.
    
    ## Work from a copy, not a temp path
    
    If the image lives in a temp directory, before your first tool call on one, copy it somewhere durable and run everything against the copy — that is what keeps the image reachable later:
    
    ```bash
    cp "<the temp path>" work/shot.png
    glance work/shot.png -q "..."
    ```
    
    Exception: the user asked for the image to stay in a temp folder.
    
    ## When you have a description instead of the image
    
    If an image reached you only as text — a description written by a person,
    a tool, or another model — and the image's file path is visible in the
    conversation, do not reason past a missing detail. Look again yourself:
    
    1. `glance <path> -q "<the specific detail>"` — one qualitative follow-up.
    2. `ground <path> "<target>"` then `glance <path> --region <that box> -q "..."` —
       locate, then zoom. The reliable way to inspect one element closely.
    
    If the file no longer exists, say so instead of guessing.
    
    ## Coarse to fine — the method behind every task above
    
    For a single question about an image, `glance` is the whole answer. For
    anything multi-step, work outside-in:
    
    1. One full-image pass (`glance`, or a description you already have) for
       the layout and an inventory of what is where.
    2. For any element that matters, `ground` it, then zoom with
       `glance --region <box> -q "..."`. Full-image passes routinely miss small
       text and icons; a crop puts all the pixels on one detail, so the model
       sees it at effectively higher resolution. When the same box will be
       checked more than once, cut it to a file first with `crop`.
    3. Never take a *prose* answer for a pixel-level fact — exact colors, small
       offsets, sizes. Vision models confidently report styling that is not
       there: coloured syntax highlighting in a monochrome code block, a border
       that does not exist. Get the number from `trace`, from a `ground` box, or
       from `pixel_diff`; sample the pixels yourself only for what those cannot
       return.
    
    ## Use cases
    
    Each file below is one job, start to finish: when it applies, the call
    sequence, and how to tell you got it right.
    
    | The job | Read |
    |---|---|
    | OCR a long screenshot, scrolling page, or chat history without losing text at chunk boundaries | `references/long-screenshot-ocr.md` |
    | Rebuild a page or component as HTML/CSS, including a roughly three-minute fast approximation mode, or align an existing UI with its reference image | `references/restore-ui.md` |
    | Extract or rebuild an icon, logo, illustration, or other isolated graphic as transparent PNG/SVG | `references/restore-graphic.md` |
    | Turn a sketch, diagram, or whiteboard into Mermaid, Graphviz, or another structured representation | `references/restore-structure.md` |
    | Operate a GUI from screenshots — locate, act, verify each step | `references/gui.md` |
    
    ## Notes
    
    - Only PNG / JPEG / GIF / WebP images are supported.
    - If a command is not found, the optional tools were not installed — report
      this to the user instead of improvising a replacement.
    - If the vision API fails, relay the error faithfully; never fabricate
      image content.
    
    Source repository: https://github.com/Anionex/agent-vision-toolkit
    
    Installation guide: https://github.com/Anionex/agent-vision-toolkit/blob/main/AGENT_INSTALL.md
    
  • skills/vision-skills/scripts/dominant_colors.py10.0 KB3718618007deupstream ↗
    #!/usr/bin/env python3
    """dominant_colors: a region's significant colours, and the exact value among a candidate palette.
    
    Two halves of one job. First you extract the region's palette -- downsample,
    quantize, merge near-duplicates -- so you can see which colours are actually
    there and how much of the region each one owns. That share histogram is the
    role map: a background takes a big share, an accent a small one.
    
    A vision model (glance) names those clusters in prose ("white", "light gray",
    "orange") but not their values. So for the value itself you supply the
    candidate palette your label implies ("light gray" -> #F9FAFA / #F5F5F5 /
    #F3F3F3 / #EDEDED) and this script decides from the pixels: each candidate is
    scored by a distance filter over the region's pixels, and the best one wins.
    The label comes from the model; the number always comes from here.
    """
    
    from __future__ import annotations
    
    import argparse
    from pathlib import Path
    
    try:
        from PIL import Image, ImageChops
    except ImportError:
        Image = None
        ImageChops = None
    
    
    def load_rgb(path: Path) -> "Image.Image":
        # Composite transparency on white the way a viewer would; unflattened
        # alpha reads as black and skews every colour statistic.
        image = Image.open(path)
        if image.mode in ("RGBA", "LA", "P"):
            image = image.convert("RGBA")
            canvas = Image.new("RGB", image.size, "white")
            canvas.paste(image, mask=image.split()[-1])
            image = canvas
        else:
            image = image.convert("RGB")
        return image
    
    
    def parse_region(region: str, width: int, height: int) -> tuple[int, int, int, int]:
        try:
            x1, y1, x2, y2 = (int(v) for v in region.split(","))
        except ValueError:
            raise ValueError("--region expects four integers: X1,Y1,X2,Y2 (pixels)") from None
        box = (max(0, min(x1, x2)), max(0, min(y1, y2)),
               min(width, max(x1, x2)), min(height, max(y1, y2)))
        if box[2] <= box[0] or box[3] <= box[1]:
            raise ValueError(f"--region {region} is empty after clamping to {width}x{height}")
        return box
    
    
    def hex_of(rgb: tuple[int, int, int]) -> str:
        return "#{:02X}{:02X}{:02X}".format(*rgb)
    
    
    def parse_hex(text: str) -> tuple[int, int, int]:
        value = text.strip().lstrip("#")
        if len(value) != 6:
            raise ValueError(f"invalid colour {text!r}: expected #RRGGBB")
        return (int(value[0:2], 16), int(value[2:4], 16), int(value[4:6], 16))
    
    
    def chebyshev(a: tuple[int, int, int], b: tuple[int, int, int]) -> int:
        return max(abs(x - y) for x, y in zip(a, b))
    
    
    def bar(share: float, max_share: float, width: int = 20) -> str:
        fill = round(share / max_share * width) if max_share else 0
        return "#" * fill
    
    
    class _Cluster:
        __slots__ = ("rgb", "count")
    
        def __init__(self, rgb: tuple[int, int, int], count: int) -> None:
            self.rgb = rgb
            self.count = count
    
    
    def extract(image: "Image.Image", box: tuple[int, int, int, int], top: int,
                quantize_k: int, max_pixels: int, merge_tol: int) -> list[_Cluster]:
        """Downsample, quantize, merge near-duplicates; return clusters sorted by count."""
        crop = image.crop(box)
        width, height = crop.size
        scale = min(1.0, max_pixels / max(width, height))
        if scale < 1.0:
            crop = crop.resize((max(1, round(width * scale)), max(1, round(height * scale))), Image.LANCZOS)
        quantized = crop.quantize(colors=quantize_k, method=Image.MEDIANCUT)
        palette = quantized.getpalette()
        clusters: list[_Cluster] = []
        for count, index in sorted(quantized.getcolors(maxcolors=quantize_k), reverse=True):
            rgb = (palette[index * 3], palette[index * 3 + 1], palette[index * 3 + 2])
            for existing in clusters:
                if chebyshev(rgb, existing.rgb) <= merge_tol:
                    total = existing.count + count
                    existing.rgb = tuple(
                        round((existing.rgb[i] * existing.count + rgb[i] * count) / total)
                        for i in range(3))
                    existing.count = total
                    break
            else:
                clusters.append(_Cluster(rgb, count))
        clusters.sort(key=lambda c: c.count, reverse=True)
        return clusters
    
    
    def pick(image: "Image.Image", box: tuple[int, int, int, int], candidates: list[str],
             tol: int) -> tuple[list[dict], dict, dict]:
        """Score each candidate by how close the region's pixels are to it."""
        crop = image.crop(box)
        width, height = crop.size
        total = width * height
        bands = crop.split()
        rows = []
        for text in candidates:
            rgb = parse_hex(text)
            farthest = None
            for band, value in zip(bands, rgb):
                diff = ImageChops.difference(band, Image.new("L", (width, height), value))
                farthest = diff if farthest is None else ImageChops.lighter(farthest, diff)
            histogram = farthest.histogram()
            mean_distance = sum(i * count for i, count in enumerate(histogram)) / total
            hard = sum(count for i, count in enumerate(histogram) if i <= tol)
            # Soft support: pixels closer to the candidate weigh more, so a match
            # is not a binary in/out -- a candidate exactly on the colour wins over
            # a near neighbour that is also inside the tolerance.
            weighted = sum(max(0, tol - i) * count for i, count in enumerate(histogram))
            rows.append({"text": text, "rgb": rgb, "mean_distance": mean_distance,
                         "hard": hard, "weighted": weighted,
                         "share": hard / total * 100})
        winner = max(rows, key=lambda r: (r["weighted"], r["hard"]))
        closest = min(rows, key=lambda r: r["mean_distance"])
        return rows, winner, closest
    
    
    def format_extract(clusters: list[_Cluster], top: int, box: tuple[int, int, int, int],
                       merge_tol: int) -> list[str]:
        total = sum(c.count for c in clusters)
        if not total:
            return ["(region has no pixels)"]
        width = box[2] - box[0]
        height = box[3] - box[1]
        max_share = max(c.count / total for c in clusters) * 100
        lines = [f"region {box[0]},{box[1]},{box[2]},{box[3]} - {width}x{height} px",
                 f"top {top} of {len(clusters)} clusters (merged at distance <= {merge_tol}):"]
        for cluster in clusters[:top]:
            share = cluster.count / total * 100
            lines.append(f"{hex_of(cluster.rgb)}  {share:5.1f}%  {bar(share, max_share)}")
        return lines
    
    
    def format_pick(rows: list[dict], winner: dict, closest: dict,
                    box: tuple[int, int, int, int], tol: int) -> list[str]:
        width = box[2] - box[0]
        height = box[3] - box[1]
        total = width * height
        max_share = max(r["share"] for r in rows) or 1.0
        max_wt = max(r["weighted"] for r in rows)
        lines = [f"region {box[0]},{box[1]},{box[2]},{box[3]} - {width}x{height} px ({total} px sampled)",
                 "candidate   share   mean_d  wt    bar"]
        for row in rows:
            mark = "*" if row is winner else " "
            wt = (row["weighted"] / max_wt * 100) if max_wt else 0.0
            lines.append(f"{mark}{row['text']:<9} {row['share']:5.1f}%  {row['mean_distance']:4.1f}  "
                         f"{wt:4.0f}%  "
                         f"{bar(row['share'], max_share)}")
        if winner["hard"] == 0:
            lines.append(f"note: no candidate is within distance <= {tol} of the region; "
                         f"closest by mean distance is {closest['text']}")
        else:
            lines.append(f"winner: {winner['text']} (* in table) - wt is soft-match closeness, "
                         f"so the winner need not have the highest share; "
                         f"{winner['share']:.1f}% of region pixels within distance <= {tol}")
        return lines
    
    
    def main() -> None:
        parser = argparse.ArgumentParser(
            prog="dominant_colors",
            description="Extract a region's significant colours, or pick the concrete value among a candidate palette",
        )
        parser.add_argument("image", type=Path, help="the image")
        parser.add_argument("--region", metavar="X1,Y1,X2,Y2",
                            help="restrict to this pixel box (e.g. from ground); default: whole image")
        parser.add_argument("--candidates", metavar="LIST",
                            help="pick mode: comma-separated candidate palette, e.g. #F9FAFA,#F5F5F5,#F3F3F3,#EDEDED")
        parser.add_argument("--top", type=int, default=5, help="how many significant clusters to show (default: 5)")
        parser.add_argument("--quantize", type=int, default=16,
                            help="palette size before merging near-duplicates (default: 16)")
        parser.add_argument("--max-pixels", type=int, default=96,
                            help="downsample the crop to at most this on the long side (default: 96)")
        parser.add_argument("--merge-tol", type=int, default=8,
                            help="merge clusters whose centres are within this per-channel distance (default: 8)")
        parser.add_argument("--tol", type=int, default=16,
                            help="pick mode: per-channel distance for 'matches this candidate' (default: 16)")
        args = parser.parse_args()
    
        if Image is None:
            parser.exit(1, "dominant_colors: requires Pillow; install the optional dependency pillow first\n")
        path = args.image.expanduser()
        if not path.is_file():
            parser.exit(1, f"dominant_colors: image not found: {path}\n")
        try:
            image = load_rgb(path)
            width, height = image.size
            box = parse_region(args.region, width, height) if args.region else (0, 0, width, height)
            if args.candidates:
                candidates = [candidate.strip() for candidate in args.candidates.split(",") if candidate.strip()]
                if not candidates:
                    parser.exit(1, "dominant_colors: --candidates needs at least one #RRGGBB\n")
                rows, winner, closest = pick(image, box, candidates, args.tol)
                print("\n".join(format_pick(rows, winner, closest, box, args.tol)))
            else:
                clusters = extract(image, box, args.top, args.quantize, args.max_pixels, args.merge_tol)
                print("\n".join(format_extract(clusters, args.top, box, args.merge_tol)))
        except ValueError as exc:
            parser.exit(1, f"dominant_colors: {exc}\n")
    
    
    if __name__ == "__main__":
        main()
    
  • skills/vision-skills/scripts/extract_fg.py12 KB833f7a95d3cbupstream ↗
    #!/usr/bin/env python3
    """extract_fg.py — 从截图提取图标/logo 前景(透明 PNG)。
    
    手动模式(2026-08 实测):搜索区域内取满足判定的像素,整体做 8 邻域连通分量
    分析,保留最大连通分量即为前景。背景噪点是散点、图标线条是连续线,连通性
    自动分离,无需预先知道主色,抗锯齿全部保留。
    
    用法:
      python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 -o icon.png
      python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 --mode dark   # 灰色/黑色线条(logo)
      python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 --exclude-color '#E6E6E6'  # 彩色背景干扰
      自动模式(crop --scale 输出、图标居中、浅色圆底;多张可一次传):
      python3 scripts/extract_fg.py d/icon1.png d/icon2.png       # 圆心=图中心,圆底半径/颜色自动推断
      python3 scripts/extract_fg.py d/icon1.png --disc-radius 60  # 圆底半径兜底
      python3 scripts/extract_fg.py d/icon1.png --boxes "101,84,184,171"  # ground 框校正
    
    输出: 透明背景 PNG(仅前景像素保留原色,其余 alpha=0),打印 bbox / 像素数 / 分量数。
    自动模式输出单一图形分量(前 3 大彩色分量中饱和度最高者,白环/文字自动出局)。
    """
    from __future__ import annotations
    
    import argparse
    import sys
    from pathlib import Path
    
    from PIL import Image
    
    
    def connected_components(ink: set, w: int, h: int) -> list[list[tuple[int, int]]]:
        """8 邻域连通分量,按大小降序返回。"""
        seen: set = set()
        comps: list[list[tuple[int, int]]] = []
        for p in ink:
            if p in seen:
                continue
            stack = [p]
            seen.add(p)
            comp = []
            while stack:
                cx, cy = stack.pop()
                comp.append((cx, cy))
                for dx in (-1, 0, 1):
                    for dy in (-1, 0, 1):
                        if dx == 0 and dy == 0:
                            continue
                        q = (cx + dx, cy + dy)
                        if q in seen or q not in ink:
                            continue
                        seen.add(q)
                        stack.append(q)
            comps.append(comp)
        comps.sort(key=len, reverse=True)
        return comps
    
    
    def process_one(image: str, args: argparse.Namespace) -> int:
    
        try:
            im = Image.open(image).convert("RGB")
        except Exception as e:
            print(f"error: cannot open {image}: {e}", file=sys.stderr)
            return 1
        w, h = im.size
    
        auto_mode = args.region is None
        if auto_mode:
            try:
                import numpy as np
            except ImportError:
                print("error: auto mode requires numpy; pass --region instead", file=sys.stderr)
                return 1
            arr = np.asarray(im).astype(int)
            box = None
            if args.boxes:
                try:
                    box = tuple(int(v) for v in args.boxes.split(","))
                except ValueError:
                    print("error: --boxes must be X1,Y1,X2,Y2", file=sys.stderr)
                    return 1
                cx, cy = (box[0] + box[2]) / 2, (box[1] + box[3]) / 2
                radius = args.disc_radius or max(box[2] - box[0], box[3] - box[1]) * 0.8
            else:
                cx, cy = w / 2, h / 2
                radius = args.disc_radius or min(w, h) / 2 * 0.6
            yy, xx = np.mgrid[0:h, 0:w]
            dist = np.sqrt((xx - cx) ** 2 + (yy - cy) ** 2)
            ring = (dist > radius * 0.75) & (dist < radius * 0.95)
            if ring.sum() == 0:
                print("error: disc ring empty (--disc-radius too small?)", file=sys.stderr)
                return 1
            mean = arr[ring].mean(0).round(0).astype(int)
            auto_excl = tuple(int(v) for v in mean)
            print(f"auto: center=({cx:.0f},{cy:.0f}) disc radius≈{radius:.0f} "
                  f"exclude-color=#{auto_excl[0]:02X}{auto_excl[1]:02X}{auto_excl[2]:02X}")
            x1, y1, x2, y2 = 0, 0, w, h
        else:
            auto_excl = None
            box = None
            try:
                x1, y1, x2, y2 = (int(v) for v in args.region.split(","))
            except ValueError:
                print("error: --region must be X1,Y1,X2,Y2", file=sys.stderr)
                return 1
        x1, y1 = max(0, x1), max(0, y1)
        x2, y2 = min(w, x2), min(h, y2)
        if x2 <= x1 or y2 <= y1:
            print("error: empty region", file=sys.stderr)
            return 1
    
        excl = None
        if auto_excl:
            excl = auto_excl
            exclude_tol = 35  # 自动模式:圆底渐变排除容差(默认 24 排不干净渐变)
        elif args.exclude_color:
            v = args.exclude_color.lstrip("#")
            try:
                excl = tuple(int(v[i:i + 2], 16) for i in (0, 2, 4))
            except ValueError:
                print("error: --exclude-color must be #RRGGBB", file=sys.stderr)
                return 1
            exclude_tol = args.exclude_tol
        else:
            exclude_tol = args.exclude_tol
    
        ink: set = set()
        px = im.load()
        for y in range(y1, y2):
            for x in range(x1, x2):
                r, g, b = px[x, y]
                mx, mn = max(r, g, b), min(r, g, b)
                if args.mode == "color":
                    if mx - mn <= args.sat:
                        continue
                else:
                    if mx >= args.dark:
                        continue
                if excl is not None:
                    d = ((r - excl[0]) ** 2 + (g - excl[1]) ** 2 + (b - excl[2]) ** 2) ** 0.5
                    if d <= exclude_tol:
                        continue
                ink.add((x - x1, y - y1))
    
        if not ink:
            print("error: no foreground pixels found in region (raise --sat or lower --dark?)",
                  file=sys.stderr)
            return 1
    
        comps = connected_components(ink, x2 - x1, y2 - y1)
        # 图标可能由多个分离子形状组成(云朵 logo 的 ">_" 与轮廓不相连),
        # 保留所有足够大的分量;噪点散点(远小于最大分量)自动排除。
        min_size = max(len(comps[0]) * 0.02, 8)
        mx0 = [p[0] for p in comps[0]]
        my0 = [p[1] for p in comps[0]]
        main_box = (min(mx0), min(my0), max(mx0), max(my0))
    
        def overlaps_main(c) -> bool:
            cx = [p[0] for p in c]
            cy = [p[1] for p in c]
            return not (max(cx) < main_box[0] or min(cx) > main_box[2]
                        or max(cy) < main_box[1] or min(cy) > main_box[3])
    
        kept = [c for c in comps if len(c) >= min_size or overlaps_main(c)]
        if auto_mode:
            # 自动模式:选单一图形分量——前 3 大彩色分量中饱和度最高者(白环/波纹 sat 低出局)。
            colored = []
            for c in kept:
                rs = [px[x1 + p[0], y1 + p[1]][0] for p in c]
                gs = [px[x1 + p[0], y1 + p[1]][1] for p in c]
                bs = [px[x1 + p[0], y1 + p[1]][2] for p in c]
                n = len(c)
                sat = max(sum(rs) / n, sum(gs) / n, sum(bs) / n) - min(sum(rs) / n, sum(gs) / n, sum(bs) / n)
                if sat > 25:
                    colored.append((len(c), sat, c))
            if colored:
                colored.sort(key=lambda t: t[0], reverse=True)
                chosen = max(colored[:3], key=lambda t: t[1])[2]
            else:
                chosen = kept[0]
            if box:
                bx0, by0, bx1, by1 = box
                scored = []
                for c in kept:
                    overlap = sum(1 for p in c if bx0 <= p[0] < bx1 and by0 <= p[1] < by1)
                    scored.append((overlap, len(c), c))
                scored.sort(key=lambda t: (t[0], t[1]), reverse=True)
                chosen = scored[0][2]
            best = chosen
            pad = 0
        else:
            best = [p for c in kept for p in c]
            pad = args.pad
        bx1 = x1 + min(p[0] for p in best) - pad
        by1 = y1 + min(p[1] for p in best) - pad
        bx2 = x1 + max(p[0] for p in best) + 1 + pad
        by2 = y1 + max(p[1] for p in best) + 1 + pad
        bx1, by1 = max(0, bx1), max(0, by1)
        bx2, by2 = min(w, bx2), min(h, by2)
    
        out = Image.new("RGBA", (bx2 - bx1, by2 - by1), (0, 0, 0, 0))
        o = out.load()
        for lx, ly in best:
            o[x1 + lx - bx1, y1 + ly - by1] = px[x1 + lx, y1 + ly] + (255,)
    
        if not args.no_keep_whites:
            # 背景填充:近白像素从 bbox 边缘 flood fill -> 透明(外部背景);
            # 被彩色前景包围的近白像素(图标内部白色镂空细节)-> 保留纯白。
            w, h = out.width, out.height
            near: set = set()
            for y in range(h):
                for x in range(w):
                    r, g, b, a = o[x, y]
                    if a == 0:
                        continue
                    mx, mn = max(r, g, b), min(r, g, b)
                    if mx >= 240 and mx - mn <= 25:
                        near.add((x, y))
            bg: set = set()
            stack = [p for p in near if p[0] == 0 or p[1] == 0 or p[0] == w - 1 or p[1] == h - 1]
            while stack:
                cx, cy = stack.pop()
                if (cx, cy) in bg:
                    continue
                bg.add((cx, cy))
                for dx in (-1, 0, 1):
                    for dy in (-1, 0, 1):
                        q = (cx + dx, cy + dy)
                        if q in near and q not in bg:
                            stack.append(q)
            for x, y in bg:
                o[x, y] = (255, 255, 255, 0)
            for x, y in near - bg:
                o[x, y] = (255, 255, 255, 255)
    
        if args.output:
            dest = Path(args.output)
        elif auto_mode:
            dest = Path(image).with_suffix(".clean.png")
        else:
            dest = Path(image).with_suffix(".fg.png")
        out.save(dest)
    
        print(f"bbox (原图像素): x1: {bx1}, y1: {by1}, x2: {bx2}, y2: {by2}")
        print(f"前景像素: {len(best)}  保留分量: {len(kept)}/{len(comps)}  最大分量占比: {len(comps[0]) / len(ink) * 100:.0f}%")
        print(f"wrote {dest} ({out.width}x{out.height})")
        return 0
    def main() -> int:
        ap = argparse.ArgumentParser(description="从截图区域提取图标/logo 前景(透明 PNG)")
        ap.add_argument("image", nargs="+", help="截图路径(PNG/JPEG/WebP)")
        ap.add_argument("--region", metavar="X1,Y1,X2,Y2",
                        help="手动模式:搜索区域(原图像素,宽松即可);不传则自动推断(图标居中)")
        ap.add_argument("-o", "--output", default=None,
                        help="输出 PNG 路径(默认 <image-stem>.fg.png;自动模式 <image-stem>.clean.png)")
        ap.add_argument("--disc-radius", type=float, default=None,
                        help="自动模式:圆底半径(像素),缺省 = min(宽,高)/2 × 0.6")
        ap.add_argument("--boxes", default=None, metavar="X1,Y1,X2,Y2",
                        help="自动模式:ground 在放大图上的框,用于圆心/半径校正与分量重叠筛选")
        ap.add_argument("--mode", choices=("color", "dark"), default="color",
                        help="color=彩色线条(默认);dark=灰色/黑色线条(logo、图标)")
        ap.add_argument("--sat", type=int, default=12,
                        help="color 模式饱和度阈值(RGB max-min,默认 12)")
        ap.add_argument("--dark", type=int, default=215,
                        help="dark 模式亮度阈值(RGB max,默认 215,即 #BABBBC 级灰线可收)")
        ap.add_argument("--exclude-color", default=None, metavar="#RRGGBB",
                        help="排除与该颜色接近的像素(彩色背景干扰时用)")
        ap.add_argument("--exclude-tol", type=float, default=24,
                        help="--exclude-color 的距离容差(默认 24)")
        ap.add_argument("--pad", type=int, default=3, help="输出 bbox 每边外扩像素(默认 3)")
        ap.add_argument("--no-keep-whites", action="store_true",
                        help="不保留被前景包围的内部白色细节(默认保留)")
        args = ap.parse_args()
        code = 0
        for image in args.image:
            code |= process_one(image, args)
        return code
    
    
    if __name__ == "__main__":
        sys.exit(main())
    
  • skills/vision-skills/scripts/html_shot.py23 KBe378e0a5404aupstream ↗
    #!/usr/bin/env python3
    """Screenshot an HTML file with headless Chrome/Chromium/Edge.
    
    The default path intentionally remains the small Chrome CLI wrapper used by
    the original upstream tool. ``--full-page`` uses Chrome DevTools Protocol
    instead: the requested viewport remains the layout viewport while the page's
    CSS scroll height is measured and captured in one pass.
    """
    
    import argparse
    import base64
    import hashlib
    import json
    import os
    import shutil
    import socket
    import struct
    import subprocess
    import tempfile
    import time
    from pathlib import Path
    from urllib.error import URLError
    from urllib.parse import urlparse
    from urllib.request import urlopen
    
    
    CHROME_CANDIDATES = (
        # macOS
        "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
        "/Applications/Chromium.app/Contents/MacOS/Chromium",
        "/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",
        # Linux
        "google-chrome", "google-chrome-stable", "chromium", "chromium-browser",
        "microsoft-edge", "brave-browser",
    )
    
    FULL_PAGE_TIMEOUT_SECONDS = 30.0
    FULL_PAGE_MAX_GROWTH_ROUNDS = 25
    FULL_PAGE_MAX_SCROLL_STEPS = 2000
    FULL_PAGE_QUIET_MILLISECONDS = 1250
    
    
    def find_chrome() -> str | None:
        for candidate in CHROME_CANDIDATES:
            path = shutil.which(candidate)
            if path:
                return path
            if candidate.startswith("/") and Path(candidate).is_file():
                return candidate
        for env in ("PROGRAMFILES", "PROGRAMFILES(X86)", "LOCALAPPDATA"):
            base = os.environ.get(env)
            if not base:
                continue
            for sub in ("Google/Chrome/Application/chrome.exe",
                        "Microsoft/Edge/Application/msedge.exe"):
                path = Path(base) / sub
                if path.is_file():
                    return str(path)
        return None
    
    
    def default_output(source: str) -> str:
        if source.startswith(("http://", "https://", "file://", "data:")):
            stem = Path(urlparse(source).path).stem
            return f"{stem or 'page'}.png"
        return f"{Path(source).stem}.png"
    
    
    def read_http_json(port: int, path: str) -> object:
        with urlopen(f"http://127.0.0.1:{port}{path}", timeout=1.0) as response:
            return json.loads(response.read().decode("utf-8"))
    
    
    class DevToolsSocket:
        """Minimal RFC 6455 client for the Chrome DevTools Protocol."""
    
        def __init__(self, websocket_url: str):
            parsed = urlparse(websocket_url)
            if parsed.scheme != "ws" or parsed.hostname is None or parsed.port is None:
                raise RuntimeError("invalid DevTools websocket URL")
            self.socket = socket.create_connection((parsed.hostname, parsed.port), timeout=5.0)
            self.socket.settimeout(5.0)
            self.buffer = bytearray()
            key = base64.b64encode(os.urandom(16)).decode("ascii")
            request = (
                f"GET {parsed.path or '/'} HTTP/1.1\r\n"
                f"Host: {parsed.hostname}:{parsed.port}\r\n"
                "Upgrade: websocket\r\n"
                "Connection: Upgrade\r\n"
                f"Sec-WebSocket-Key: {key}\r\n"
                "Sec-WebSocket-Version: 13\r\n\r\n"
            ).encode("ascii")
            self.socket.sendall(request)
            response = self._read_until(b"\r\n\r\n")
            if not response.startswith(b"HTTP/1.1 101"):
                raise RuntimeError("Chrome did not accept the DevTools websocket")
            expected_accept = base64.b64encode(hashlib.sha1(
                f"{key}258EAFA5-E914-47DA-95CA-C5AB0DC85B11".encode("ascii")
            ).digest()).decode("ascii")
            headers = response.decode("latin-1").split("\r\n")
            accept = next((line.split(":", 1)[1].strip() for line in headers
                           if line.lower().startswith("sec-websocket-accept:")), None)
            if accept != expected_accept:
                raise RuntimeError("Chrome returned an invalid DevTools websocket handshake")
            self.next_id = 0
    
        def _read_until(self, marker: bytes) -> bytes:
            data = bytearray()
            while marker not in data:
                chunk = self.socket.recv(4096)
                if not chunk:
                    raise RuntimeError("DevTools websocket closed during handshake")
                data.extend(chunk)
            boundary = data.index(marker) + len(marker)
            self.buffer.extend(data[boundary:])
            return bytes(data[:boundary])
    
        def _send_frame(self, opcode: int, payload: bytes) -> None:
            mask = os.urandom(4)
            masked = bytes(value ^ mask[index % 4] for index, value in enumerate(payload))
            length = len(masked)
            if length < 126:
                header = struct.pack("!BB", 0x80 | opcode, 0x80 | length)
            elif length < 65536:
                header = struct.pack("!BBH", 0x80 | opcode, 0x80 | 126, length)
            else:
                header = struct.pack("!BBQ", 0x80 | opcode, 0x80 | 127, length)
            self.socket.sendall(header + mask + masked)
    
        def _read_frame(self) -> tuple[bool, int, bytes]:
            header = self._read_exact(2)
            first, second = header
            final = (first & 0x80) != 0
            opcode = first & 0x0F
            length = second & 0x7F
            masked = (second & 0x80) != 0
            if length == 126:
                length = struct.unpack("!H", self._read_exact(2))[0]
            elif length == 127:
                length = struct.unpack("!Q", self._read_exact(8))[0]
            mask = self._read_exact(4) if masked else None
            payload = self._read_exact(length)
            if mask is not None:
                payload = bytes(value ^ mask[index % 4] for index, value in enumerate(payload))
            return final, opcode, payload
    
        def _read_message(self) -> tuple[int, bytes]:
            fragments: list[bytes] = []
            message_opcode: int | None = None
            while True:
                final, opcode, payload = self._read_frame()
                if opcode == 0x9:
                    self._send_frame(0xA, payload)
                    continue
                if opcode == 0x8:
                    raise RuntimeError("DevTools websocket closed")
                if opcode in (0x1, 0x2):
                    fragments = [payload]
                    message_opcode = opcode
                elif opcode == 0x0 and message_opcode is not None:
                    fragments.append(payload)
                else:
                    continue
                if final and message_opcode is not None:
                    return message_opcode, b"".join(fragments)
    
        def _read_exact(self, length: int) -> bytes:
            data = bytearray()
            if self.buffer:
                take = min(length, len(self.buffer))
                data.extend(self.buffer[:take])
                del self.buffer[:take]
            while len(data) < length:
                chunk = self.socket.recv(length - len(data))
                if not chunk:
                    raise RuntimeError("DevTools websocket closed unexpectedly")
                data.extend(chunk)
            return bytes(data)
    
        def command(self, method: str, params: dict | None = None) -> dict:
            self.next_id += 1
            request_id = self.next_id
            request = {"id": request_id, "method": method}
            if params is not None:
                request["params"] = params
            self._send_frame(0x1, json.dumps(request, separators=(",", ":")).encode("utf-8"))
            while True:
                opcode, payload = self._read_message()
                if opcode != 0x1:
                    continue
                message = json.loads(payload.decode("utf-8"))
                if message.get("id") != request_id:
                    continue
                if "error" in message:
                    raise RuntimeError(f"{method}: {message['error'].get('message', 'CDP command failed')}")
                return message.get("result", {})
    
        def close(self) -> None:
            try:
                self._send_frame(0x8, b"")
            except OSError:
                pass
            self.socket.close()
    
    
    def evaluate(client: DevToolsSocket, expression: str) -> object:
        result = client.command("Runtime.evaluate", {
            "expression": expression,
            "returnByValue": True,
            "awaitPromise": True,
        })
        if "exceptionDetails" in result:
            raise RuntimeError(result["exceptionDetails"].get("text", "Runtime.evaluate failed"))
        remote = result.get("result", {})
        if remote.get("subtype") == "error":
            raise RuntimeError(remote.get("description", "Runtime.evaluate failed"))
        return remote.get("value")
    
    
    def wait_for_target(port: int, deadline: float) -> dict:
        last_error: Exception | None = None
        while time.monotonic() < deadline:
            try:
                targets = read_http_json(port, "/json/list")
                if isinstance(targets, list):
                    for target in targets:
                        if (isinstance(target, dict) and target.get("type") == "page"
                                and target.get("webSocketDebuggerUrl")):
                            return target
            except (OSError, URLError, ValueError) as error:
                last_error = error
            time.sleep(0.05)
        detail = f": {last_error}" if last_error else ""
        raise RuntimeError(f"timed out waiting for Chrome DevTools{detail}")
    
    
    def wait_for_devtools_port(profile: Path, process: subprocess.Popen,
                               deadline: float) -> int:
        active_port = profile / "DevToolsActivePort"
        last_error: Exception | None = None
        while time.monotonic() < deadline:
            if process.poll() is not None:
                raise RuntimeError(f"Chrome exited before DevTools was ready (code {process.returncode})")
            try:
                lines = active_port.read_text().splitlines()
                port = int(lines[0])
                if 0 < port <= 65535:
                    return port
            except (OSError, ValueError, IndexError) as error:
                last_error = error
            time.sleep(0.05)
        detail = f": {last_error}" if last_error else ""
        raise RuntimeError(f"timed out waiting for Chrome DevTools port{detail}")
    
    
    def wait_for_ready(client: DevToolsSocket, deadline: float) -> None:
        while time.monotonic() < deadline:
            if evaluate(client, "document.readyState") == "complete":
                return
            time.sleep(0.05)
        raise RuntimeError("timed out waiting for the HTML document")
    
    
    def pause_for_frame(seconds: float) -> None:
        time.sleep(max(0.02, seconds))
    
    
    def build_full_page_command(chrome: str, profile: Path,
                                width: int, height: int, scale: int) -> list[str]:
        command = [
            chrome, "--headless=new", "--disable-gpu", "--hide-scrollbars",
            "--no-first-run", "--no-default-browser-check", "--use-mock-keychain",
            "--blink-settings=imagesLazyLoadingEnabled=false",
            f"--window-size={width},{height}",
            "--remote-debugging-port=0",
            "--remote-debugging-address=127.0.0.1",
            f"--user-data-dir={profile}",
        ]
        if scale != 1:
            command.append(f"--force-device-scale-factor={scale}")
        command.append("about:blank")
        return command
    
    
    def measure_page_height(client: DevToolsSocket) -> int:
        return int(evaluate(client, """
            Math.max(document.documentElement.scrollHeight,
                     document.body ? document.body.scrollHeight : 0)
        """) or 0)
    
    
    def enforce_pixel_limit(width: int, page_height: int, scale: int,
                            max_pixels: int | None) -> None:
        if page_height <= 0:
            raise RuntimeError("document has no measurable scroll height")
        output_pixels = width * page_height * scale * scale
        if max_pixels is not None and output_pixels > max_pixels:
            raise RuntimeError(
                f"full-page screenshot would exceed --max-pixels "
                f"({output_pixels} > {max_pixels})"
            )
    
    
    def ensure_before_deadline(deadline: float, message: str) -> None:
        if time.monotonic() >= deadline:
            raise RuntimeError(message)
    
    
    def wait_for_dom_quiet(client: DevToolsSocket, deadline: float) -> None:
        remaining_ms = int((deadline - time.monotonic()) * 1000)
        if remaining_ms <= FULL_PAGE_QUIET_MILLISECONDS:
            raise RuntimeError("timed out before the page reached a quiet state")
        max_wait_ms = min(4500, remaining_ms)
        result = evaluate(client, f"""
            (() => new Promise(resolve => {{
              const quietMs = {FULL_PAGE_QUIET_MILLISECONDS};
              const maxWaitMs = {max_wait_ms};
              const height = () => Math.max(
                document.documentElement.scrollHeight,
                document.body ? document.body.scrollHeight : 0
              );
              let lastHeight = height();
              let quietTimer;
              let pollTimer;
              let maxTimer;
              let finished = false;
              const finish = quiet => {{
                if (finished) return;
                finished = true;
                clearTimeout(quietTimer);
                clearInterval(pollTimer);
                clearTimeout(maxTimer);
                resolve({{quiet, height: height()}});
              }};
              const restartQuietTimer = () => {{
                clearTimeout(quietTimer);
                quietTimer = setTimeout(() => finish(true), quietMs);
              }};
              pollTimer = setInterval(() => {{
                const currentHeight = height();
                if (currentHeight !== lastHeight) {{
                  lastHeight = currentHeight;
                  restartQuietTimer();
                }}
              }}, 100);
              maxTimer = setTimeout(() => finish(false), maxWaitMs);
              restartQuietTimer();
            }}))()
        """)
        if not isinstance(result, dict) or result.get("quiet") is not True:
            raise RuntimeError("page did not become quiet before the full-page deadline")
    
    
    def settle_full_page(client: DevToolsSocket, width: int, height: int, scale: int,
                         max_pixels: int | None, deadline: float) -> int:
        page_height = measure_page_height(client)
        enforce_pixel_limit(width, page_height, scale, max_pixels)
        scan_start = 0
        growth_rounds = 0
        scroll_steps = 0
        waited_for_images = False
    
        while True:
            ensure_before_deadline(deadline, "timed out while stabilizing the full page")
            measured_before_sweep = page_height
            for position in range(scan_start, measured_before_sweep, max(1, height)):
                scroll_steps += 1
                if scroll_steps > FULL_PAGE_MAX_SCROLL_STEPS:
                    raise RuntimeError("full page requires too many scroll steps to capture safely")
                ensure_before_deadline(deadline, "timed out while scrolling the full page")
                evaluate(client, f"window.scrollTo(0, {position}); true")
                pause_for_frame(0.12)
            evaluate(client, f"window.scrollTo(0, {measured_before_sweep}); true")
            pause_for_frame(0.12)
            wait_for_dom_quiet(client, deadline)
    
            measured_after_sweep = measure_page_height(client)
            enforce_pixel_limit(width, measured_after_sweep, scale, max_pixels)
            if measured_after_sweep != measured_before_sweep:
                growth_rounds += 1
                if growth_rounds > FULL_PAGE_MAX_GROWTH_ROUNDS:
                    raise RuntimeError("page is still changing after repeated full-page sweeps")
                scan_start = (max(0, measured_before_sweep - height)
                              if measured_after_sweep > measured_before_sweep else 0)
                page_height = measured_after_sweep
                waited_for_images = False
                continue
    
            if not waited_for_images:
                evaluate(client, """
                    Promise.race([
                      Promise.all(Array.from(document.images)
                        .filter(image => !image.complete)
                        .map(image => new Promise(resolve => {
                          image.addEventListener('load', resolve, {once: true});
                          image.addEventListener('error', resolve, {once: true});
                        }))),
                      new Promise(resolve => setTimeout(resolve, 3000))
                    ]).then(() => true)
                """)
                waited_for_images = True
                ensure_before_deadline(deadline, "timed out while waiting for full-page images")
                measured_after_images = measure_page_height(client)
                enforce_pixel_limit(width, measured_after_images, scale, max_pixels)
                if measured_after_images != page_height:
                    growth_rounds += 1
                    if growth_rounds > FULL_PAGE_MAX_GROWTH_ROUNDS:
                        raise RuntimeError("page is still changing after image loading")
                    scan_start = (max(0, page_height - height)
                                  if measured_after_images > page_height else 0)
                    page_height = measured_after_images
                    waited_for_images = False
                    continue
    
            evaluate(client, "window.scrollTo(0, 0); true")
            wait_for_dom_quiet(client, deadline)
            final_height = measure_page_height(client)
            enforce_pixel_limit(width, final_height, scale, max_pixels)
            if final_height == page_height:
                return final_height
            growth_rounds += 1
            if growth_rounds > FULL_PAGE_MAX_GROWTH_ROUNDS:
                raise RuntimeError("page is still changing before full-page capture")
            scan_start = max(0, page_height - height) if final_height > page_height else 0
            page_height = final_height
            waited_for_images = False
    
    
    def capture_full_page(chrome: str, source: str, output: Path, width: int, height: int,
                          scale: int, wait_ms: int, max_pixels: int | None) -> int:
        profile = Path(tempfile.mkdtemp(prefix="agent-vision-html-shot-"))
        process: subprocess.Popen | None = None
        client: DevToolsSocket | None = None
        try:
            command = build_full_page_command(chrome, profile, width, height, scale)
            process = subprocess.Popen(command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
            startup_deadline = time.monotonic() + 15.0
            port = wait_for_devtools_port(profile, process, startup_deadline)
            target = wait_for_target(port, startup_deadline)
            client = DevToolsSocket(str(target["webSocketDebuggerUrl"]))
            client.command("Page.enable")
            client.command("Runtime.enable")
            client.command("Emulation.setDeviceMetricsOverride", {
                "width": width,
                "height": height,
                "deviceScaleFactor": scale,
                "mobile": False,
                "screenWidth": width,
                "screenHeight": height,
            })
            client.command("Page.navigate", {"url": source})
            wait_for_ready(client, time.monotonic() + 15.0)
            if wait_ms > 0:
                pause_for_frame(wait_ms / 1000.0)
    
            # Keep the requested viewport fixed. Scrolling, rather than growing
            # the viewport, wakes IO/reveal handlers without changing vh/svh.
            evaluate(client, """
                (() => {
                  document.documentElement.style.scrollBehavior = 'auto';
                  if (document.body) document.body.style.scrollBehavior = 'auto';
                  window.scrollTo(0, 0);
                  return true;
                })()
            """)
            page_height = settle_full_page(
                client, width, height, scale, max_pixels,
                time.monotonic() + FULL_PAGE_TIMEOUT_SECONDS,
            )
            result = client.command("Page.captureScreenshot", {
                "format": "png",
                "fromSurface": True,
                "captureBeyondViewport": True,
                "clip": {"x": 0, "y": 0, "width": width, "height": page_height, "scale": 1},
            })
            data = result.get("data")
            if not isinstance(data, str):
                raise RuntimeError("Chrome did not return screenshot data")
            output.write_bytes(base64.b64decode(data))
            return page_height
        finally:
            if client is not None:
                client.close()
            if process is not None and process.poll() is None:
                process.terminate()
                try:
                    process.wait(timeout=2)
                except subprocess.TimeoutExpired:
                    process.kill()
                    process.wait(timeout=2)
            shutil.rmtree(profile, ignore_errors=True)
    
    
    def main() -> None:
        parser = argparse.ArgumentParser(
            prog="html_shot",
            description="Screenshot an HTML file (or URL) to PNG with headless Chrome/Chromium/Edge",
        )
        parser.add_argument("source", help="path to a local .html file, or an http(s):// URL")
        parser.add_argument("-o", "--output", help="output PNG path (default: <source-stem>.png in the current directory)")
        parser.add_argument("--width", type=int, default=1280, help="viewport width in CSS pixels (default: 1280)")
        parser.add_argument("--height", type=int, default=800, help="viewport height in CSS pixels (default: 800)")
        parser.add_argument("--scale", type=int, default=1,
                            help="device scale factor: 2 makes the PNG 2x for small text (default: 1)")
        parser.add_argument("--wait-ms", type=int, default=0,
                            help="wait time in ms before capturing (default: 0, capture immediately)")
        parser.add_argument("--full-page", action="store_true",
                            help="capture the full CSS scroll height while keeping the requested viewport")
        parser.add_argument("--max-pixels", type=int,
                            help="reject full-page captures larger than this many output pixels")
        args = parser.parse_args()
    
        chrome = find_chrome()
        if not chrome:
            parser.exit(1, "html_shot: no Chrome/Chromium/Edge found; install one and retry\n")
    
        source = args.source
        if not source.startswith(("http://", "https://", "file://", "data:")):
            path = Path(source).expanduser()
            if not path.is_file():
                parser.exit(1, f"html_shot: file not found: {path}\n")
            source = path.resolve().as_uri()
    
        output = Path(args.output).expanduser().resolve() if args.output else Path(default_output(source)).resolve()
        if args.full_page:
            try:
                page_height = capture_full_page(
                    chrome, source, output, args.width, args.height,
                    args.scale, args.wait_ms, args.max_pixels,
                )
            except Exception as error:
                parser.exit(1, f"html_shot: full-page capture failed: {error}\n")
            print(f"wrote {output} ({args.width * args.scale}x{page_height * args.scale}; pageHeight={page_height})")
            return
    
        command = [
            chrome, "--headless=new", "--disable-gpu", "--hide-scrollbars",
            "--no-first-run", "--no-default-browser-check",
            f"--window-size={args.width},{args.height}",
            f"--screenshot={output}",
        ]
        if args.scale != 1:
            command.append(f"--force-device-scale-factor={args.scale}")
        if args.wait_ms > 0:
            command.append(f"--virtual-time-budget={args.wait_ms}")
        command.append(source)
    
        result = subprocess.run(command, text=True, capture_output=True)
        if result.returncode != 0 or not output.is_file():
            message = result.stderr.strip() or result.stdout.strip() or f"chrome exited with code {result.returncode}"
            parser.exit(1, f"html_shot: capture failed: {message}\n")
        print(f"wrote {output} ({args.width * args.scale}x{args.height * args.scale})")
    
    
    if __name__ == "__main__":
        main()
    
  • skills/vision-skills/scripts/long_screenshot_ocr.py44 KB9546bc430131upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/vision-skills/scripts/pixel_diff.py3.9 KB36689712fc23upstream ↗
    #!/usr/bin/env python3
    """pixel_diff: compare a rebuilt image against the original and rank the worst regions.
    
    Verification is the step agents skip or fake, because comparing two
    descriptions feels like comparing two images. It is not. This runs the real
    comparison and answers the only question that matters next: which region is
    most wrong, so you know where to look and what to fix first.
    
    Boxes are printed in the same `x1: .., y1: ..` form as ground/detect, so a
    bad region can be pasted straight into `glance --region` or `detect --region`.
    """
    
    from __future__ import annotations
    
    import argparse
    from pathlib import Path
    
    try:
        from PIL import Image, ImageChops, ImageStat
    except ImportError:
        Image = None
    
    
    def load(path: Path, size: tuple[int, int] | None = None) -> "Image.Image":
        # Traced SVGs and screenshots of transparent UI render with an alpha
        # channel; unflattened transparency reads as black and shows up as a huge
        # phantom diff. Compositing on white is what a viewer would show.
        image = Image.open(path)
        if image.mode in ("RGBA", "LA", "P"):
            image = image.convert("RGBA")
            canvas = Image.new("RGB", image.size, "white")
            canvas.paste(image, mask=image.split()[-1])
            image = canvas
        else:
            image = image.convert("RGB")
        return image.resize(size, Image.LANCZOS) if size and image.size != size else image
    
    
    def cell_scores(diff: "Image.Image", grid: int) -> list[tuple[float, tuple[int, int, int, int]]]:
        width, height = diff.size
        grey = diff.convert("L")
        scores = []
        for row in range(grid):
            for column in range(grid):
                box = (round(column * width / grid), round(row * height / grid),
                       round((column + 1) * width / grid), round((row + 1) * height / grid))
                if box[2] > box[0] and box[3] > box[1]:
                    mean = ImageStat.Stat(grey.crop(box)).mean[0]
                    scores.append((mean / 255 * 100, box))
        return sorted(scores, reverse=True)
    
    
    def main() -> None:
        parser = argparse.ArgumentParser(
            prog="pixel_diff",
            description="Pixel-diff a rebuilt image against the original and rank the worst regions",
        )
        parser.add_argument("original", type=Path, help="the reference image")
        parser.add_argument("rebuilt", type=Path, help="your rendered reproduction")
        parser.add_argument("--grid", type=int, default=6, help="split into GRID x GRID cells (default: 6)")
        parser.add_argument("--top", type=int, default=5, help="how many worst regions to print (default: 5)")
        parser.add_argument("-o", "--output", type=Path, help="write a diff heatmap image here")
        args = parser.parse_args()
        if Image is None:
            parser.exit(1, "pixel_diff: requires Pillow; install the optional dependency pillow first\n")
        for path in (args.original, args.rebuilt):
            if not path.expanduser().is_file():
                parser.exit(1, f"pixel_diff: image not found: {path}\n")
        original = load(args.original.expanduser())
        with Image.open(args.rebuilt.expanduser()) as probe:
            raw_size = probe.size
        rebuilt = load(args.rebuilt.expanduser(), size=original.size)
        if raw_size != original.size:
            # Worth saying out loud: a size mismatch is itself a finding, and every
            # box printed below is in the original's coordinates, not the rebuild's.
            print(f"note: rebuilt was {raw_size[0]}x{raw_size[1]}, scaled to {original.size[0]}x{original.size[1]}")
        diff = ImageChops.difference(original, rebuilt)
        overall = ImageStat.Stat(diff.convert("L")).mean[0] / 255 * 100
        print(f"overall difference: {overall:.2f}%")
        if args.output:
            diff.save(args.output.expanduser())
            print(f"heatmap: {args.output}")
        for index, (score, box) in enumerate(cell_scores(diff, args.grid)[:args.top], 1):
            print(f"{index}. {score:.2f}% x1: {box[0]}, y1: {box[1]}, x2: {box[2]}, y2: {box[3]}")
    
    
    if __name__ == "__main__":
        main()
    
  • skills/vision-skills/references/gui.md3.9 KB39968a17ed9dupstream ↗
    # Operating a GUI from screenshots
    
    **When to use**: the task is to act on a live screen — click, type,
    scroll — guided by screenshots: driving an app, automating a workflow,
    walking a bug reproduction. The vision layer answers "where is it" and
    "what state is it in"; the acting itself goes through whatever automation
    channel the task already has.
    
    Tool syntax lives in `SKILL.md`. This file is the sequence and the
    pass/fail test.
    
    ## Steps
    
    **1. Calibrate coordinates once, before the first click.**
    
    Screenshot pixels and pointer coordinates are often different spaces: on
    HiDPI displays a screenshot is 2-3× the logical points the click API
    takes. Compare the screenshot's pixel width to the screen's logical width
    and divide every box by that ratio. Skipping this puts every click at half
    or double distance from the origin — consistently wrong in a way that
    looks like bad grounding.
    
    **2. Prefer the UI tree; fall back to vision.**
    
    If the environment exposes a UI tree (Android `uiautomator dump`, desktop
    accessibility tree, browser DOM), read coordinates and state from there —
    it is exact, fast, and carries semantic attributes (enabled, checked,
    focusable) that pixels cannot. Use `ground` only when:
    
    - No tree is available (games, custom-rendered canvases, remote desktops),
    - The target element is absent from the tree (canvas-drawn overlays,
      non-standard widgets), or
    - You need to verify a visual state the tree doesn't expose (color, icon
      appearance).
    
    When using `ground`: centers, not corners — ground boxes are not
    pixel-exact at the edges. For tiny targets (checkboxes, close buttons), go
    coarse-to-fine: ground the containing block, then `ground --region` inside
    it.
    
    For elements that stay put across interactions (toolbar buttons, sidebar
    links, fixed panels), ground them once and record a coordinate table — an
    id/label plus center coordinates. Subsequent clicks reference the table
    entry directly instead of re-running ground each time. Invalidate the
    table when the layout changes (window resize, navigation to a different
    page, scroll).
    
    **3. One screenshot per action — act, re-shoot, verify, then continue.**
    
    Every action invalidates the current screenshot: menus open, focus moves,
    content loads; scrolling invalidates every box at once. Chaining several
    clicks on one stale screenshot is how you click through where a dialog
    used to be.
    
    **4. Verify as a question about the expected state.**
    
    `glance <after.png> -q "is the settings dialog open?"` — ask for the state
    the action should have produced, not a general description. If the answer
    is no or unexpected, stop and re-inventory (`detect`) instead of pressing
    on against an assumed screen. When the change is small (a toggle, one
    badge), diff first: `pixel_diff before.png after.png` finds the changed
    region, `glance --region` reads it.
    
    **5. After typing, read the field back.**
    
    Focus loss eats keystrokes silently. `glance --region <field box> --ocr`
    confirms the text actually landed before you submit.
    
    ## Verify
    
    The loop in steps 3-4 *is* the verification: no action counts as done
    until a fresh screenshot answers the expected-state question. For a
    multi-step flow, the final screenshot must show the end state the task
    defines — the confirmation page, the saved indicator. A completed action
    list is not evidence; the pixels are.
    
    ## Boundaries
    
    - Loading is asynchronous: after an action that triggers it, poll —
      re-screenshot until two consecutive shots stop differing (`pixel_diff`
      near 0) — rather than trusting one fixed sleep.
    - Irreversible actions (send, delete, pay) follow the calling agent's own
      confirmation policy. The vision layer reports what is on screen; it
      never makes that call.
    - A screen the task didn't predict — an error dialog, a permission
      prompt, a login wall — is a stop-and-surface, not an obstacle to click
      through. Clicking through unknown dialogs is how automations do damage.
    
  • skills/vision-skills/references/long-screenshot-ocr.md3.3 KBaeefaf4d85daupstream ↗
    # OCR a long screenshot without losing boundary text
    
    **Use for**: a vertically scrolling screenshot, chat history, long web page,
    requirements conversation, log view, or other image that would be downscaled too
    aggressively in one vision call. For a PDF or an exported document, prefer a
    format-aware document parser instead of screenshot OCR.
    
    ## Run the workflow
    
    Work from a durable copy, then choose the content mode:
    
    ```bash
    # Web pages, documents, logs, tables, and other general content
    python3 scripts/long_screenshot_ocr.py work/page.png -o work/page.ocr.md
    
    # Chat histories: preserve message grouping, senders, timestamps, and quotes
    python3 scripts/long_screenshot_ocr.py work/chat.png --mode chat -o work/chat.ocr.md
    ```
    
    Without `-o`, write `<input-stem>.ocr.md`; store chunks, sidecars, the manifest,
    and the audit in `<input-stem>_chunks/` unless `--chunks-dir` overrides it.
    
    The script performs four operations:
    
    1. Measure per-row content density and find low-content cut bands near the
       target height.
    2. Add pixel overlap only when no safe band exists, so text crossing a risky
       cut appears in both adjacent chunks.
    3. Run `glance` on the chunks using the existing `VISION_*` configuration;
       chat mode requests structured messages while general mode uses verbatim OCR.
    4. Merge only confident repeated lines or messages and write `manifest.json` plus
       `ocr_audit.md` beside the chunks.
    
    Use `--resume` after an interrupted run. A chunk is reused only when its image,
    mode, and custom prompt fingerprint still match:
    
    ```bash
    python3 scripts/long_screenshot_ocr.py work/chat.png --mode chat --resume -o work/chat.ocr.md
    ```
    
    Use `--split-only` when you need to inspect or tune the chunks before spending
    vision calls:
    
    ```bash
    python3 scripts/long_screenshot_ocr.py work/page.png --split-only
    ```
    
    If the defaults produce awkward chunks, rerun with `--target-height`,
    `--min-height`, `--max-height`, or `--overlap`. Keep enough height for local
    context; do not make tiny OCR tiles unless the source text is unusually small.
    
    ## Verify before delivering
    
    1. Read the merged Markdown from top to bottom and compare its opening and
       ending lines with the source.
    2. Open `ocr_audit.md`. Review every boundary marked `yes` against the two
       adjacent `chunk_*.png` files. A marked boundary used pixel overlap or fuzzy
       text matching and is not safe to accept blindly.
    3. Check sender changes, timestamps, quoted messages, table row breaks, code
       indentation, and paragraphs that cross chunk boundaries.
    4. For any doubtful text, run targeted OCR on the relevant chunk or crop:
    
       ```bash
       glance work/page_chunks/chunk_002.png --ocr "Re-check the final five lines carefully."
       glance work/page_chunks/chunk_002.png --region X1,Y1,X2,Y2 --ocr
       ```
    
    5. Keep visible spelling and punctuation verbatim. Write `[unreadable]` for
       text that remains illegible; do not silently guess or editorially repair it.
    
    ## Output contract
    
    - Return the merged `.ocr.md` file as the primary result.
    - Keep the chunk directory until verification is complete; it is the evidence
      for ordering and boundary decisions.
    - Report unresolved `[unreadable]` text and every boundary that still needs
      human review.
    - Do not claim a complete transcription when the first or last screenshot edge
      visibly clips content.
    
  • skills/vision-skills/references/restore-graphic.md3.7 KBb14219ab6bdfupstream ↗
    # Extracting or rebuilding a graphic asset
    
    **When to use**: the deliverable is an isolated icon, logo, illustration,
    decorative mark, or other visual asset as a transparent PNG or SVG. For a full
    page or component, read `restore-ui.md` and use this workflow only for the
    individual visual assets it identifies.
    
    ## Choose the least destructive representation
    
    Use this order:
    
    1. Reuse the original source asset if it already exists in the project.
    2. Extract a transparent PNG from the reference when the visual only needs to
       be displayed at the captured size. This preserves the original pixels and
       is the default for screenshot-only brand artwork.
    3. Trace an irregular, flat, high-contrast shape when SVG is required.
    4. Hand-write simple editable geometry or stroke icons, using the trace or
       pixel measurements as evidence rather than visual estimates.
    
    ## Extract a transparent PNG
    
    The final extraction region must be tight around the target ink. A loose crop
    that touches neighboring text, borders, or decoration may preserve those
    pixels as foreground.
    
    ```bash
    # Known tight region
    python3 scripts/extract_fg.py shot.png \
      --region X1,Y1,X2,Y2 -o asset.png
    
    # Dark or gray line art
    python3 scripts/extract_fg.py shot.png \
      --region X1,Y1,X2,Y2 --mode dark -o asset.png
    
    # Center and enlarge a small visual before automatic extraction
    crop shot.png --region X1,Y1,X2,Y2 --scale 4 -o work/asset4x.png
    python3 scripts/extract_fg.py work/asset4x.png
    ```
    
    Use `--exclude-color '#RRGGBB'` when a connected background color would
    otherwise be retained. If automatic centering is wrong, provide a tighter
    region or a target box rather than repeatedly widening the crop.
    
    Inspect the result on both light and dark backgrounds. Confirm complete ink,
    clean alpha edges, and no neighboring fragments before reuse.
    
    ## Rebuild an SVG only when the deliverable needs one
    
    ```bash
    trace asset.png -o asset.svg
    trace asset.png --polygon -o asset.svg
    trace shot.png --region X1,Y1,X2,Y2 -o asset.svg
    ```
    
    Ship traced paths directly for organic or irregular shapes. For rectangles,
    circles, pills, and other simple editable geometry, use the trace as a
    measurement and write the simpler SVG yourself.
    
    A small stroke icon is usually a hand-written SVG case, not a no-trace case:
    the trace outlines both sides of the raster stroke, while the desired asset is
    normally a centerline path with `stroke` and `fill="none"`. Use
    `trace --polygon` to recover endpoints, corners, and stroke width, then write
    the clean centerline path.
    
    When reusing traced paths:
    
    - Composite transparent SVG output on the intended background before judging
      it; transparent regions may appear black in some viewers.
    - Copy each path's `transform` together with its `d` data. Dropping the
      transform displaces otherwise correct geometry.
    - Keep opposite-winding subpaths together so holes remain holes.
    
    ## Verify
    
    Render the asset at its intended size and compare it with the same tight box
    from the reference. Use an overlay or `scripts/pixel_diff.py` to locate missing
    parts, contamination, wrong scale, or displaced paths. Judge the asset at its
    actual delivery size as well as zoomed in; a technically detailed SVG that
    looks worse at 16px is not an improvement.
    
    ## Boundaries
    
    - Whole screenshots and photos do not trace usefully.
    - Low-contrast art may disappear during binarization. Tighten the region,
      increase scale, or invert light-on-dark input before reaching for color
      tracing.
    - Use color tracing only for genuinely multicolor artwork; antialiased single-
      color input can otherwise fragment into many gray-level paths.
    - If the visual contains live text, controls, or data-driven content, rebuild
      those parts natively and use the extracted asset only for the static artwork.
    
  • skills/vision-skills/references/restore-structure.md1.9 KB3261fcff7033upstream ↗
    # Turning a visual diagram into structured code
    
    **When to use**: the task is converting a sketch, flowchart, architecture
    diagram, or whiteboard into Mermaid, Graphviz, JSON, or another structured
    representation. The goal is semantic and topological fidelity, not matching
    the source renderer pixel for pixel.
    
    ## Steps
    
    1. Run one full-image `glance` pass for the diagram type, reading direction,
       major groups, and visual conventions.
    2. Use `detect` for the initial node and label inventory. Refine dense groups
       with `detect --region`.
    3. Use OCR for labels. Preserve visible spelling, punctuation, and
       abbreviations verbatim; write `[unreadable]` instead of guessing.
    4. Locate ambiguous arrows, connectors, legends, and group boundaries with
       `ground` and targeted `glance --region` calls.
    5. Build an explicit intermediate inventory before writing the output:
       nodes, labels, groups, edges, directions, and edge labels.
    6. Generate the Mermaid, Graphviz, or requested structure from that inventory.
    
    ## Verify structure, not pixels
    
    Render the result, inventory it again, and compare these facts with the
    reference:
    
    - node count and label set
    - group membership and nesting
    - edge list and direction
    - edge labels and branch conditions
    - start/end or input/output roles
    
    Pixel differences are expected because the target renderer chooses its own
    fonts, spacing, and routing. A missing node or reversed arrow is a failure even
    when the two images look broadly similar; a different curve or gap is not a
    failure when the structure is correct.
    
    ## Boundaries
    
    - Do not tidy labels, expand abbreviations, or silently repair wording unless
      the user asks for editorial cleanup.
    - Do not infer an edge from proximity alone. If its endpoint or direction is
      unclear, zoom into the connector and mark unresolved facts explicitly.
    - If the user wants a visual SVG copy rather than structured code, follow
      `restore-graphic.md` instead.
    
  • skills/vision-skills/references/restore-ui.md10.0 KBf49f796dbe9bupstream ↗
    # Rebuilding a UI from a reference image
    
    **When to use**: the task is turning a screenshot or design image into a
    page, component, or application UI in HTML/CSS or the project's existing
    frontend stack. Checking an existing implementation against its reference is
    the same workflow entered at Verify. For an isolated icon, logo, or
    illustration, read `restore-graphic.md`; for Mermaid, Graphviz, or another
    structured diagram, read `restore-structure.md`.
    
    ## Choose the restore mode
    
    - Use **fast restore mode** when the user asks for a quick, rough,
      approximate, prototype, or first-pass reconstruction, or explicitly values
      speed over fidelity. Its target is a recognizable screenshot in about three
      minutes when the project already runs.
    - Use the **standard restore workflow** below when the user asks for close,
      precise, pixel-level, or production-ready alignment, or does not opt into a
      faster approximation.
    
    ## Fast restore mode: first screenshot in about three minutes
    
    Fast mode preserves the page's hierarchy, major regions, visible text, and
    primary state. It deliberately approximates fine spacing, exact colors,
    typography, shadows, decorative details, and icon geometry.
    
    ### Hard limits
    
    1. Inspect the existing frontend stack, component library, icon set, source
       assets, and design tokens before writing code, but stop searching as soon as
       a usable local primitive is found.
    2. Run one full-image `detect` pass. Treat its boxes as layout estimates and do
       not start a region-by-region inventory.
    3. After that `detect` pass, use at most **six sequential image-inspection
       rounds** through `view_image` (or the host's equivalent built-in viewer) and
       `glance`. Each round may launch up to three independent calls concurrently,
       so the hard ceiling is 18 calls across six rounds. Batch unrelated regions
       or questions into the same round instead of waiting for each result before
       starting the next call. Normally one or two rounds are enough.
    4. Do not use `trace`, foreground extraction, repeated color sampling, or
       iterative `pixel_diff` work in fast mode. Those are fidelity tools and will
       consume the delivery window.
    
    ### Build the approximation
    
    1. Implement the largest layout regions first, then visible text, primary
       controls, and the most important state. Ignore details that are only visible
       when zoomed in.
    2. Reuse the project's existing components and CSS tokens. If its frontend or
       icon library contains a reasonably similar component or icon, use it
       directly instead of recreating the reference. Prefer an approximate library
       icon over cropping, tracing, or hand-drawing a new one.
    3. Use nearby existing palette tokens or visually similar CSS values. Exact
       sampled hex values, gradients, subtle borders, and shadow opacity are out of
       scope unless one of them defines the whole composition.
    4. Keep text and controls native, selectable, and interactive. Fast mode relaxes
       visual fidelity, not basic UI behavior.
    
    ### Render once, fix once, deliver
    
    1. Render the target viewport with `scripts/html_shot.py` or the project's
       existing browser setup. Use `--full-page` only when the reference covers a
       complete scrolling document; keep the same logical viewport either way.
    2. Inspect the screenshot once. If there is an obvious structural failure such
       as a missing major region, broken wrapping, or a wildly wrong scale, make
       one focused correction and render once more.
    3. Deliver the screenshot. Stop instead of spending the remaining time on
       small color, icon, font, shadow, radius, or spacing differences.
    
    A practical time box is roughly 30 seconds for project inspection plus
    `detect`, 90 seconds for implementation, and the remaining minute for startup,
    rendering, one correction, and screenshot delivery. Dependency installation or
    a project that does not already run may extend that target; do not compensate
    by silently switching back to a long precision loop.
    
    ## Standard restore workflow
    
    ### Core strategy: code-native UI plus screenshot-backed visuals
    
    Do not choose one reconstruction mode for the whole page. Classify each
    element separately. Most finished pages should combine both kinds:
    
    | Kind | Build it this way | Typical examples |
    |---|---|---|
    | **Code-native component** | Reuse the project's component library or build it with semantic HTML, CSS, and simple SVG primitives | layout, text, buttons, inputs, cards, tabs, dividers, backgrounds, simple geometric badges |
    | **Screenshot-backed visual** | Extract the original pixels, remove the background when needed, and place the result as an image asset | logos, proprietary icons, illustrations, decorative artwork, textures, complex static marks |
    
    Use this decision order for every element:
    
    1. **Reuse an existing component or source asset from the project.** Search
       before recreating anything.
    2. **Use a code-native component** when the element contains text, accepts
       input, changes state, must respond to layout, or is simple to express with
       the project's normal primitives.
    3. **Use a screenshot-backed visual** when it is static, visually distinctive,
       expensive to redraw, and no source asset exists.
    4. Rebuild a vector only when the user requires SVG/editability, the visual
       must scale beyond the screenshot resolution, or extraction cannot isolate
       it cleanly. Follow `restore-graphic.md` for that branch.
    
    Never flatten text, controls, or a large layout block into a screenshot. Never
    redraw extractable brand artwork merely to make the implementation look more
    "native". For a complex widget, keep the shell, text, and interaction native;
    use an extracted image only for its static decorative layer.
    
    ## Steps
    
    ### 1. Preserve the reference and inspect the target project
    
    If the reference is in a temporary path, copy it to durable work storage
    before the first tool call. Inspect the existing stack, component library,
    fonts, icons, images, and design tokens before writing replacements.
    
    ### 2. Establish the coordinate system
    
    Record the reference image dimensions and the target viewport. Screenshots may
    be HiDPI: `detect`, `ground`, and `crop` report image pixels, while CSS uses
    logical pixels. Derive the scale from the actual image and viewport dimensions;
    do not assume a fixed 2x ratio.
    
    ### 3. Inventory the UI outside-in
    
    1. Run one full-image `detect` pass for the initial element list and boxes.
    2. Use `detect --region` on dense layout blocks; a full-screen pass is only a
       scaffold.
    3. Use `glance` for hierarchy, component type, visual state, and styling.
    4. Use OCR for visible text and `ground` for a specific element that remains
       ambiguous.
    
    Treat model boxes as approximate handles. Use them to organize the page, but
    do not mistake their final few pixels for measured boundaries.
    
    ### 4. Classify before implementing
    
    Make a short element list with one of these decisions: reuse existing source,
    code-native, screenshot-backed, or hybrid. This prevents two costly mistakes:
    rebuilding complex artwork from scratch and turning functional UI into static
    image patches.
    
    ### 5. Extract screenshot-backed visuals
    
    Use a padded crop only to inspect and center the target. Before foreground
    extraction, tighten the final region to the visual's own ink; adjacent text or
    rules inside the region can become foreground components too.
    
    ```bash
    crop shot.png --region X1,Y1,X2,Y2 --scale 4 -o work/icon4x.png
    python3 scripts/extract_fg.py work/icon4x.png
    
    # Or extract directly from a known tight region:
    python3 scripts/extract_fg.py shot.png \
      --region X1,Y1,X2,Y2 -o work/icon.png
    python3 scripts/extract_fg.py shot.png \
      --region X1,Y1,X2,Y2 --mode dark -o work/logo.png
    ```
    
    Inspect the transparent result before use. Confirm that the whole visual is
    present, no neighboring text or border leaked in, and the alpha edge remains
    clean. When extracting several assets, inspect them together in a contact
    sheet instead of checking them from filenames alone.
    
    ### 6. Implement the native frame
    
    - Follow the project's existing framework and component patterns.
    - Build content, interaction, responsive layout, borders, shadows, and simple
      shapes natively.
    - Use `scripts/dominant_colors.py` for important background, text, and accent
      values; use `glance` to name a color, not to invent its numeric value.
    - Place extracted assets with explicit logical dimensions. Preserve their
      aspect ratio and avoid baking surrounding whitespace into the asset.
    - Match structure and proportions before tuning small spacing.
    
    ## Verify and converge by impact
    
    For an existing implementation, start here.
    
    1. Render at the same logical viewport with `scripts/html_shot.py` or the
       project's browser test setup. For a long-page reference, add `--full-page`
       rather than increasing `--height`, which would change `vh`-based layout.
    2. Compare the render and reference at the same dimensions. Inspect them side
       by side; use `scripts/pixel_diff.py` to locate differences that are hard to
       spot or explain.
    3. Fix material discrepancies: missing or wrong content, incorrect hierarchy,
       broken wrapping, visibly wrong alignment or scale, wrong component state,
       distorted assets, and clearly different primary colors.
    4. Re-render after a meaningful fix. Continue while the comparison exposes a
       user-visible defect; stop when the remaining difference is font
       rasterization, antialiasing, subpixel placement, or another imperceptible
       rendering detail.
    
    `pixel_diff` is a locator, not the acceptance target. Do not keep iterating only
    to lower its aggregate percentage, and do not promise zero-difference output
    from a screenshot without a separately scoped browser, font, viewport, and
    rendering environment.
    
    ## Boundaries
    
    - Do not ship a screenshot of the whole page as the implementation.
    - Do not use screenshot-backed text or controls that must remain selectable,
      accessible, interactive, localized, or responsive.
    - Do not stretch an extracted bitmap beyond the resolution supported by the
      reference. Obtain a source asset or rebuild a vector when it must scale.
    - Always compare against the original reference, never only against an earlier
      version of your own render.
    
  • skills/vision-skills/agents/openai.yaml242 Bab4885f8ecb2upstream ↗
    interface:
      display_name: "Agent Vision Toolkit"
      short_description: "Ask, OCR long screenshots, locate, inventory, and vectorize images"
      default_prompt: "Use $vision-skills to inspect this image or run the long-screenshot OCR workflow."
    
  • skills/vision-skills/work/shot.png694 KB95a6ac2109cdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • README.mdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/anionex-agent-vision-toolkit-skills-vision-skills.md

// REPO STATS

1.3k stars
4 open issues
Last commit: 1mo ago