vision-skills
>-
Choose how to use this skill
You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.
1. Native installer
This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.
Do not guess an installer command or replace an existing version without reviewing the diff.
2. Complete package recommended
Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.
No complete ProSkills package is published for this listing yet.3. Prompt-only
Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.
Need only the instruction file?
Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.
No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.
// RATINGS
Not yet listed on ClawHub or SkillsMP
// README
agent-vision-toolkit
What it thinks is what it sees — give any text-only coding agent eyes: image Q&A, long-screenshot OCR, frontend UI restoration, and GUI automation, as a vision toolkit plus a skill, with optional drop-in integration for Codex, Claude Code, Pi, Oh My Pi, and OpenCode.
🎯 An agent's vision capability doesn't have to live in the model — it can live in the harness.
🌐 中文 | English
If your agent already runs on a text-only model such as DeepSeek but is held back by the lack of multimodality — unable to see images, with every attempt to use an image tool blocked by the system — this repository provides tools, skills, and proxy integrations that let text-only models handle visual tasks on equal or even better footing. The goal is to make the experience of using a text-model agent as seamless as using a multimodal one, and ultimately let a tool-equipped text-model agent outperform a native multimodal agent that does not use this toolkit and its methods.
This repository provides two kinds of components:
- Vision tool CLIs — multiple CLIs, plus a skill that teaches the agent when to use each one. Any agent that can invoke a shell can use them.
- Seamless integration (optional upgrade) — a transparent local proxy and single-file native plugins, so images we paste and the agent's built-in image tools both work seamlessly, with no extra tool installation or additional prompting.
All code has been verified in real Codex + DeepSeek sessions, and the same pipeline has been live-verified end-to-end in Claude Code, Pi, Oh My Pi, and OpenCode.
If this project helps you or gives you some inspiration, feel free to star🌟 & fork.
❤️ Sponsor
Want to sponsor this project? See FUNDING.md or email [email protected].
Latest Update
2026-08-18 — Skill renamed: the included agent skill is now vision-skills (formerly vision-tools), so the name describes the capability rather than the underlying tools.
2026-08-13 — Native DeepSeek Harness support is now available. The new dsh-vision-toolkit linked package brings this toolkit into DSH Web and Headless profiles as a native Profile Bundle. It provides 10 structured visual tools for intent-aware image Q&A, grounding, detection, tracing, cropping, pixel diff, long-screenshot OCR, foreground extraction, dominant-color analysis, and HTML screenshots, while adding DSH Credentials, a managed isolated runtime, previewable Artifacts, Web Settings, and Agent-scoped progressive tool exposure.
The package is tracked here as a Git submodule and maintained independently at Anionex/dsh-vision-toolkit. Clone this repository with --recurse-submodules, or run git submodule update --init --recursive in an existing checkout.
- Latest Update
- Highlights
- Use-case Playbooks
- Real-world Effects
- Quick Start
- The Tools
- Upgrade: Seamless Integration
- How It Works
- Configuration
- FAQ
- Donation
- Community
- About
Highlights
- More than image descriptions — it captures what the LLM actually cares about: when viewing an image, it passes along the user's or model's latest intent, producing the details needed for the current turn instead of a broad, unfocused description.
- Both pasted images and built-in image tools work: the agent can understand images pasted directly as well as images opened through its built-in tools.
- A battle-tested methodology for visual tasks: the included skill teaches the agent what to inspect, which tool to choose, what sequence to follow, and how to verify the final result.
- One-sentence install: ask your agent to install it — it follows the verified flow end to end, toolkit, skill, and seamless integration included.
Use-case Playbooks
The included vision-skills skill contains complete examples that an agent can follow directly.
When to use them, the order in which to call tools, and how to verify the result are all documented in the corresponding skill guides:
| Use case | What the agent learns to do |
|---|---|
| Extract long screenshots, chat histories, and scrolling pages | Find low-content cut bands, OCR each chunk in order, preserve chat speakers/timestamps/quotes, merge only duplicated overlap, and surface risky boundaries for verification. See the Telegram reference run → |
| Rebuild a UI from a screenshot or design | Reuse project components and assets first, then combine code-native UI, extracted visuals, rendered screenshots, and visual comparison to align a page or component. |
// HOW IT'S BUILT
// FILES
14 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.
skills/vision-skills/SKILL.md16 KB830ac12be1a3upstream ↗
--- name: vision-skills description: >- Local vision CLIs: glance (describe/ask/OCR an image), ground (locate a target, pixel box), detect (element inventory), trace (image to SVG geometry), crop (cut a pixel box to a file), and scripts/html_shot.py (HTML file to image). Use for any task involving an image — questions, text, splitting and transcribing long screenshots or chat histories, locating elements, comparing, rebuilding as HTML/SVG, digitizing a sketch or diagram, reading values off a chart, operating a GUI from screenshots — and to re-check an image yourself when a description you were given lacks a detail. --- # vision-skills Five local CLIs that give a text-only agent eyes. They read one shared vision config (`VISION_API_KEY` / `VISION_BASE_URL` / `VISION_MODEL` / `LANG`), plus the optional Python-client settings `VISION_API_PROTOCOL`, `VISION_REASONING_EFFORT`, and `VISION_USER_AGENT` — no extra credentials. Pick the tool by the question you are answering: | Question | Tool | |---|---| | "What does this image show / say?" | `glance` | | "Where is X?" — a thing you can name | `ground` | | "Where are all the Xs?" — every instance of a kind | `detect` | | "What is its exact shape, size, offset?" | `trace` | | "Cut this box out as its own image file" | `crop` | | "OCR this long screenshot / scrolling page / chat history" | `scripts/long_screenshot_ocr.py` | | "Extract the icon/logo foreground as transparent PNG — manual region or auto (cropped+scaled screenshots)" | `scripts/extract_fg.py` | | "Turn this HTML file into a viewport or full-page screenshot" | `scripts/html_shot.py` | | "Which colours dominate a region, and which palette value fits it?" | `scripts/dominant_colors.py` | | A relation none of them return — a gap, a distance between two located things | code over the pixels (Pillow) | `glance` answers what something is; `ground` and `detect` answer where. You give `ground` a description of a particular thing; you give `detect` a kind and it enumerates the instances. Both give real coordinates, but they are not pixel-exact: the box arrives on a 0-1000 grid and is scaled to your image, so the last pixel or few are not reliable. That is accurate enough to crop with, to click, to compare positions against. When a number has to be exact, `trace` derives it from the actual pixels — offsets, sizes, shapes. ## Use the provided tools before hand-rolled pixels Everything this toolkit ships a tool for, call the tool — do not rewrite it with Pillow in the middle of a task. The CLIs exist so the same pixel work is not hand-coded differently every time: - cut a box out of an image → `crop`, not `Image.open(...).crop(...)` - sample a region's palette → `scripts/dominant_colors.py` - compare two images → `scripts/pixel_diff.py` - vectorize to SVG → `trace` - locate / inventory elements → `ground` / `detect` - describe / OCR an image → `glance` - safely split, OCR, and merge a long screenshot → `scripts/long_screenshot_ocr.py` - HTML file to a viewport or full-page screenshot → `scripts/html_shot.py` Hand-written Pillow is only for what none of them return: a relation between two things you already located (a gap, a distance), a resize or overlay, drawing. If you catch yourself writing `.crop()`, `.convert()`, or histogram code where one of the tools above fits, replace it with the tool call — same coordinates, same box format, and the output feeds the next tool directly. ## glance — ask about an image ```bash glance <image> # detailed description glance <image> -q "<question>" # targeted question (qualitative only) glance <image> --ocr # verbatim OCR glance <image> --region X1,Y1,X2,Y2 -q "..." # zoom into a crop glance <img1> <img2> -q "..." # compare in ONE call ``` When you do compare with `glance`, pass all paths to one call — separate calls cannot see both images, so two descriptions compared afterwards are two hallucination surfaces, not a comparison. `--region` uploads only the crop, so small text and icons become readable. But "what changed between these two?" is not a glance question. A one-word badge or a small shift is a rounding error to a vision model and exact to `scripts/pixel_diff.py`. Diff first to get the box, then `glance --region` that box to read what the change actually is. For a tall scrolling screenshot, do not send the whole image through one OCR call and accept the model's downscaling loss. Run the long-screenshot workflow, which finds low-content cut bands, invokes `glance` on each chunk, uses structured extraction for chat histories, merges only duplicated overlap, and writes a boundary audit: ```bash python3 scripts/long_screenshot_ocr.py work/page.png -o work/page.ocr.md python3 scripts/long_screenshot_ocr.py work/chat.png --mode chat --resume -o work/chat.ocr.md ``` Read `references/long-screenshot-ocr.md` before using it. It defines the verification pass for unsafe cuts and chat-message boundaries. ## ground — locate a named target ```bash ground <image> "<target description>" ground <image> "<target>" --region X1,Y1,X2,Y2 ``` Output: `x1: .., y1: .., x2: .., y2: ..` in original-image pixels — with `--region` too (crop hits are mapped back). Provider-native 0-1000 boxes do not all use the same array order: Gemini uses `[y0, x0, y1, x1]`, while Qwen3-VL, Qwen3.5, and Qwen3.6 use `[x0, y0, x1, y1]`. Grounding code must select the order by model family (or an explicit override) before scaling to pixels; never parse every provider as Gemini-style `yxyx`. If several boxes come back numbered, your description matched more than one element rather than picking out a single thing. Narrow it with what distinguishes the one you mean — its text, its position, the block it sits in — and ask again. The box is a handle, not just an answer — it feeds the next call: ```bash $ ground screenshot.png "the send button" x1: 1067, y1: 841, x2: 1108, y2: 881 $ glance screenshot.png --region 1067,841,1108,881 -q "is it enabled or greyed out?" ``` That two-step is how you inspect anything too small to survive a full-image pass. ## detect — find every instance of a kind ```bash detect <image> # every UI element detect <image> "buttons" # one kind only detect <image> --region X1,Y1,X2,Y2 # inside one box detect <image> "text" --fail-on-empty # fail if no text is detected ``` The default category targets UI screenshots. For photographs or video frames, pass an explicit category such as `"objects"` or `"text"`. An empty inventory normally prints `no elements detected` and exits 0; `--fail-on-empty` instead exits 1 with a stderr diagnostic and no stdout. This reports an empty model result, not proof that the image has no matching elements. You name a particular thing for `ground`; you name a kind for `detect` and it enumerates the instances. Output is a numbered list with each item's visible text and box. A full-screen pass is a fast first draft — counts vary run to run on dense screens. For completeness, detect the layout blocks first, then `detect --region` each block. ## trace — exact shape geometry (local, no vision API) ```bash trace <image> # b/w spline SVG to stdout trace <image> --polygon # boxy diagrams/wireframes trace <image> --region X1,Y1,X2,Y2 -o out.svg # crop first ``` Coordinates come from the actual pixels, not a model's estimate. Flat, high-contrast graphics only; text becomes curves (pair with `--ocr` when the text matters). Small images are upscaled automatically before tracing, so a 30px icon traces as readily as a screenshot — size is not a reason to skip the tool. Before shipping or reusing a traced SVG, read `references/restore-graphic.md` — it holds the reuse traps and the ship-vs-hand-write call. ## crop — cut a pixel box out of an image (local, no vision API) ```bash crop <image> --region X1,Y1,X2,Y2 # writes <image-stem>.crop.png next to the input crop <image> --region X1,Y1,X2,Y2 -o out.png crop <image> --region X1,Y1,X2,Y2 --scale 4 # upscale the cut-out 4x (LANCZOS) first ``` The same X1,Y1,X2,Y2 pixel boxes `ground`/`detect` print, clamped to the image bounds. Once a box is worth keeping — the same crop is about to feed `pixel_diff`, `dominant_colors`, and `trace` in turn — cut it to a file once and reuse it, instead of re-cropping in memory on every call. `--scale N` upscales the cut-out before writing (default output name becomes `<image-stem>[email protected]`): for icons too small for `ground`/`trace` to see clearly, crop with `--scale 4`, then run `ground`/`trace` on the upscaled file — coordinates it returns are in the upscaled grid, divide by `N` to map back to the original image. Requires the optional `pillow`. ## extract_fg — icon foreground as transparent PNG: manual region or auto (local, no vision API) ```bash # manual: you know the region (and optionally the background colour) python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 -o icon.png python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 --mode dark # grey/black line logos python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 --exclude-color '#E6E6E6' # auto: `crop --scale` cut-outs with the icon centred — no region needed crop shot.png --region X1,Y1,X2,Y2 --scale 4 -o d/icon1.png python3 scripts/extract_fg.py d/icon1.png d/icon2.png # writes <stem>.clean.png next to each input python3 scripts/extract_fg.py d/icon1.png --disc-radius 60 python3 scripts/extract_fg.py d/icon1.png --boxes "101,84,184,171" ``` Manual mode keeps every sufficiently large connected component of the region (separate logo sub-shapes stay together; specks drop out). Auto mode takes a `crop --scale` cut-out with the icon centred (disc + glyph): the disc centre is the image centre, the disc radius defaults to `min(w,h)/2 * 0.6`, and the disc colour is sampled from a ring around the centre; that colour is excluded and the glyph is picked as the most saturated among the three largest coloured components (white rings, ripples, and text fall away), output as a 1:1 transparent PNG. When auto inference fails, override the radius with `--disc-radius`, or pass a `ground` box (in the upscaled grid) as `--boxes` to recentre and re-filter by overlap. Multiple images may be passed at once (auto mode). Requires the optional `pillow` (and `numpy` for auto mode). ## html_shot — render an HTML file to an image (local, needs a Chrome-family browser) ```bash python3 scripts/html_shot.py page.html # writes page.png, 1280x800 python3 scripts/html_shot.py page.html --width 1440 --height 900 -o page.png python3 scripts/html_shot.py page.html --scale 2 # 2x pixels: small text stays readable python3 scripts/html_shot.py page.html --full-page # complete scroll height, same layout viewport python3 scripts/html_shot.py page.html --full-page --max-pixels 40000000 ``` The visual-alignment loop: write HTML, screenshot it at the reference viewport, then compare it with the design. Use `pixel_diff` to locate material differences, not to chase a zero-difference score. Rendering happens in headless Chrome/Chromium/Edge — no Python dependencies. The default captures only the viewport. Use `--full-page` for the complete document while keeping `--width` and `--height` as the layout viewport, so `vh`/`svh` and responsive breakpoints do not change. Add `--max-pixels N` when the page height is untrusted. `--wait-ms N` pauses for fonts, images, or animation before capturing. Paths are relative to this skill's own directory. ## pixel_diff — where two images differ (local, no vision API) ```bash python3 scripts/pixel_diff.py <a> <b> # path is relative to this skill dir ``` Prints an overall difference percentage plus the worst regions as `x1: ..` boxes you can feed straight into `glance --region`. Exact where a vision model rounds off. ## dominant_colors — a region's palette, and the exact value among candidates (local, no vision API) ```bash python3 scripts/dominant_colors.py <image> --region X1,Y1,X2,Y2 # top colour clusters + shares python3 scripts/dominant_colors.py <image> --region X1,Y1,X2,Y2 \ --candidates '#F9FAFA,#F5F5F5,#F3F3F3,#EDEDED' # pick the best candidate ``` A vision model names a colour ("light gray") but not its value. The first mode downsamples, quantizes, and merges near-duplicates to list the region's significant colours with the share each owns — the histogram shows which colour is the background and which is the accent. Given the candidate palette your label implies, the second mode scores each candidate by how close the region's pixels are to it and prints the winner. Take the value from here, never from `glance`'s prose. Paths are relative to this skill's own directory. ## Work from a copy, not a temp path If the image lives in a temp directory, before your first tool call on one, copy it somewhere durable and run everything against the copy — that is what keeps the image reachable later: ```bash cp "<the temp path>" work/shot.png glance work/shot.png -q "..." ``` Exception: the user asked for the image to stay in a temp folder. ## When you have a description instead of the image If an image reached you only as text — a description written by a person, a tool, or another model — and the image's file path is visible in the conversation, do not reason past a missing detail. Look again yourself: 1. `glance <path> -q "<the specific detail>"` — one qualitative follow-up. 2. `ground <path> "<target>"` then `glance <path> --region <that box> -q "..."` — locate, then zoom. The reliable way to inspect one element closely. If the file no longer exists, say so instead of guessing. ## Coarse to fine — the method behind every task above For a single question about an image, `glance` is the whole answer. For anything multi-step, work outside-in: 1. One full-image pass (`glance`, or a description you already have) for the layout and an inventory of what is where. 2. For any element that matters, `ground` it, then zoom with `glance --region <box> -q "..."`. Full-image passes routinely miss small text and icons; a crop puts all the pixels on one detail, so the model sees it at effectively higher resolution. When the same box will be checked more than once, cut it to a file first with `crop`. 3. Never take a *prose* answer for a pixel-level fact — exact colors, small offsets, sizes. Vision models confidently report styling that is not there: coloured syntax highlighting in a monochrome code block, a border that does not exist. Get the number from `trace`, from a `ground` box, or from `pixel_diff`; sample the pixels yourself only for what those cannot return. ## Use cases Each file below is one job, start to finish: when it applies, the call sequence, and how to tell you got it right. | The job | Read | |---|---| | OCR a long screenshot, scrolling page, or chat history without losing text at chunk boundaries | `references/long-screenshot-ocr.md` | | Rebuild a page or component as HTML/CSS, including a roughly three-minute fast approximation mode, or align an existing UI with its reference image | `references/restore-ui.md` | | Extract or rebuild an icon, logo, illustration, or other isolated graphic as transparent PNG/SVG | `references/restore-graphic.md` | | Turn a sketch, diagram, or whiteboard into Mermaid, Graphviz, or another structured representation | `references/restore-structure.md` | | Operate a GUI from screenshots — locate, act, verify each step | `references/gui.md` | ## Notes - Only PNG / JPEG / GIF / WebP images are supported. - If a command is not found, the optional tools were not installed — report this to the user instead of improvising a replacement. - If the vision API fails, relay the error faithfully; never fabricate image content. Source repository: https://github.com/Anionex/agent-vision-toolkit Installation guide: https://github.com/Anionex/agent-vision-toolkit/blob/main/AGENT_INSTALL.mdskills/vision-skills/scripts/dominant_colors.py10.0 KB3718618007deupstream ↗
#!/usr/bin/env python3 """dominant_colors: a region's significant colours, and the exact value among a candidate palette. Two halves of one job. First you extract the region's palette -- downsample, quantize, merge near-duplicates -- so you can see which colours are actually there and how much of the region each one owns. That share histogram is the role map: a background takes a big share, an accent a small one. A vision model (glance) names those clusters in prose ("white", "light gray", "orange") but not their values. So for the value itself you supply the candidate palette your label implies ("light gray" -> #F9FAFA / #F5F5F5 / #F3F3F3 / #EDEDED) and this script decides from the pixels: each candidate is scored by a distance filter over the region's pixels, and the best one wins. The label comes from the model; the number always comes from here. """ from __future__ import annotations import argparse from pathlib import Path try: from PIL import Image, ImageChops except ImportError: Image = None ImageChops = None def load_rgb(path: Path) -> "Image.Image": # Composite transparency on white the way a viewer would; unflattened # alpha reads as black and skews every colour statistic. image = Image.open(path) if image.mode in ("RGBA", "LA", "P"): image = image.convert("RGBA") canvas = Image.new("RGB", image.size, "white") canvas.paste(image, mask=image.split()[-1]) image = canvas else: image = image.convert("RGB") return image def parse_region(region: str, width: int, height: int) -> tuple[int, int, int, int]: try: x1, y1, x2, y2 = (int(v) for v in region.split(",")) except ValueError: raise ValueError("--region expects four integers: X1,Y1,X2,Y2 (pixels)") from None box = (max(0, min(x1, x2)), max(0, min(y1, y2)), min(width, max(x1, x2)), min(height, max(y1, y2))) if box[2] <= box[0] or box[3] <= box[1]: raise ValueError(f"--region {region} is empty after clamping to {width}x{height}") return box def hex_of(rgb: tuple[int, int, int]) -> str: return "#{:02X}{:02X}{:02X}".format(*rgb) def parse_hex(text: str) -> tuple[int, int, int]: value = text.strip().lstrip("#") if len(value) != 6: raise ValueError(f"invalid colour {text!r}: expected #RRGGBB") return (int(value[0:2], 16), int(value[2:4], 16), int(value[4:6], 16)) def chebyshev(a: tuple[int, int, int], b: tuple[int, int, int]) -> int: return max(abs(x - y) for x, y in zip(a, b)) def bar(share: float, max_share: float, width: int = 20) -> str: fill = round(share / max_share * width) if max_share else 0 return "#" * fill class _Cluster: __slots__ = ("rgb", "count") def __init__(self, rgb: tuple[int, int, int], count: int) -> None: self.rgb = rgb self.count = count def extract(image: "Image.Image", box: tuple[int, int, int, int], top: int, quantize_k: int, max_pixels: int, merge_tol: int) -> list[_Cluster]: """Downsample, quantize, merge near-duplicates; return clusters sorted by count.""" crop = image.crop(box) width, height = crop.size scale = min(1.0, max_pixels / max(width, height)) if scale < 1.0: crop = crop.resize((max(1, round(width * scale)), max(1, round(height * scale))), Image.LANCZOS) quantized = crop.quantize(colors=quantize_k, method=Image.MEDIANCUT) palette = quantized.getpalette() clusters: list[_Cluster] = [] for count, index in sorted(quantized.getcolors(maxcolors=quantize_k), reverse=True): rgb = (palette[index * 3], palette[index * 3 + 1], palette[index * 3 + 2]) for existing in clusters: if chebyshev(rgb, existing.rgb) <= merge_tol: total = existing.count + count existing.rgb = tuple( round((existing.rgb[i] * existing.count + rgb[i] * count) / total) for i in range(3)) existing.count = total break else: clusters.append(_Cluster(rgb, count)) clusters.sort(key=lambda c: c.count, reverse=True) return clusters def pick(image: "Image.Image", box: tuple[int, int, int, int], candidates: list[str], tol: int) -> tuple[list[dict], dict, dict]: """Score each candidate by how close the region's pixels are to it.""" crop = image.crop(box) width, height = crop.size total = width * height bands = crop.split() rows = [] for text in candidates: rgb = parse_hex(text) farthest = None for band, value in zip(bands, rgb): diff = ImageChops.difference(band, Image.new("L", (width, height), value)) farthest = diff if farthest is None else ImageChops.lighter(farthest, diff) histogram = farthest.histogram() mean_distance = sum(i * count for i, count in enumerate(histogram)) / total hard = sum(count for i, count in enumerate(histogram) if i <= tol) # Soft support: pixels closer to the candidate weigh more, so a match # is not a binary in/out -- a candidate exactly on the colour wins over # a near neighbour that is also inside the tolerance. weighted = sum(max(0, tol - i) * count for i, count in enumerate(histogram)) rows.append({"text": text, "rgb": rgb, "mean_distance": mean_distance, "hard": hard, "weighted": weighted, "share": hard / total * 100}) winner = max(rows, key=lambda r: (r["weighted"], r["hard"])) closest = min(rows, key=lambda r: r["mean_distance"]) return rows, winner, closest def format_extract(clusters: list[_Cluster], top: int, box: tuple[int, int, int, int], merge_tol: int) -> list[str]: total = sum(c.count for c in clusters) if not total: return ["(region has no pixels)"] width = box[2] - box[0] height = box[3] - box[1] max_share = max(c.count / total for c in clusters) * 100 lines = [f"region {box[0]},{box[1]},{box[2]},{box[3]} - {width}x{height} px", f"top {top} of {len(clusters)} clusters (merged at distance <= {merge_tol}):"] for cluster in clusters[:top]: share = cluster.count / total * 100 lines.append(f"{hex_of(cluster.rgb)} {share:5.1f}% {bar(share, max_share)}") return lines def format_pick(rows: list[dict], winner: dict, closest: dict, box: tuple[int, int, int, int], tol: int) -> list[str]: width = box[2] - box[0] height = box[3] - box[1] total = width * height max_share = max(r["share"] for r in rows) or 1.0 max_wt = max(r["weighted"] for r in rows) lines = [f"region {box[0]},{box[1]},{box[2]},{box[3]} - {width}x{height} px ({total} px sampled)", "candidate share mean_d wt bar"] for row in rows: mark = "*" if row is winner else " " wt = (row["weighted"] / max_wt * 100) if max_wt else 0.0 lines.append(f"{mark}{row['text']:<9} {row['share']:5.1f}% {row['mean_distance']:4.1f} " f"{wt:4.0f}% " f"{bar(row['share'], max_share)}") if winner["hard"] == 0: lines.append(f"note: no candidate is within distance <= {tol} of the region; " f"closest by mean distance is {closest['text']}") else: lines.append(f"winner: {winner['text']} (* in table) - wt is soft-match closeness, " f"so the winner need not have the highest share; " f"{winner['share']:.1f}% of region pixels within distance <= {tol}") return lines def main() -> None: parser = argparse.ArgumentParser( prog="dominant_colors", description="Extract a region's significant colours, or pick the concrete value among a candidate palette", ) parser.add_argument("image", type=Path, help="the image") parser.add_argument("--region", metavar="X1,Y1,X2,Y2", help="restrict to this pixel box (e.g. from ground); default: whole image") parser.add_argument("--candidates", metavar="LIST", help="pick mode: comma-separated candidate palette, e.g. #F9FAFA,#F5F5F5,#F3F3F3,#EDEDED") parser.add_argument("--top", type=int, default=5, help="how many significant clusters to show (default: 5)") parser.add_argument("--quantize", type=int, default=16, help="palette size before merging near-duplicates (default: 16)") parser.add_argument("--max-pixels", type=int, default=96, help="downsample the crop to at most this on the long side (default: 96)") parser.add_argument("--merge-tol", type=int, default=8, help="merge clusters whose centres are within this per-channel distance (default: 8)") parser.add_argument("--tol", type=int, default=16, help="pick mode: per-channel distance for 'matches this candidate' (default: 16)") args = parser.parse_args() if Image is None: parser.exit(1, "dominant_colors: requires Pillow; install the optional dependency pillow first\n") path = args.image.expanduser() if not path.is_file(): parser.exit(1, f"dominant_colors: image not found: {path}\n") try: image = load_rgb(path) width, height = image.size box = parse_region(args.region, width, height) if args.region else (0, 0, width, height) if args.candidates: candidates = [candidate.strip() for candidate in args.candidates.split(",") if candidate.strip()] if not candidates: parser.exit(1, "dominant_colors: --candidates needs at least one #RRGGBB\n") rows, winner, closest = pick(image, box, candidates, args.tol) print("\n".join(format_pick(rows, winner, closest, box, args.tol))) else: clusters = extract(image, box, args.top, args.quantize, args.max_pixels, args.merge_tol) print("\n".join(format_extract(clusters, args.top, box, args.merge_tol))) except ValueError as exc: parser.exit(1, f"dominant_colors: {exc}\n") if __name__ == "__main__": main()skills/vision-skills/scripts/extract_fg.py12 KB833f7a95d3cbupstream ↗
#!/usr/bin/env python3 """extract_fg.py — 从截图提取图标/logo 前景(透明 PNG)。 手动模式(2026-08 实测):搜索区域内取满足判定的像素,整体做 8 邻域连通分量 分析,保留最大连通分量即为前景。背景噪点是散点、图标线条是连续线,连通性 自动分离,无需预先知道主色,抗锯齿全部保留。 用法: python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 -o icon.png python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 --mode dark # 灰色/黑色线条(logo) python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 --exclude-color '#E6E6E6' # 彩色背景干扰 自动模式(crop --scale 输出、图标居中、浅色圆底;多张可一次传): python3 scripts/extract_fg.py d/icon1.png d/icon2.png # 圆心=图中心,圆底半径/颜色自动推断 python3 scripts/extract_fg.py d/icon1.png --disc-radius 60 # 圆底半径兜底 python3 scripts/extract_fg.py d/icon1.png --boxes "101,84,184,171" # ground 框校正 输出: 透明背景 PNG(仅前景像素保留原色,其余 alpha=0),打印 bbox / 像素数 / 分量数。 自动模式输出单一图形分量(前 3 大彩色分量中饱和度最高者,白环/文字自动出局)。 """ from __future__ import annotations import argparse import sys from pathlib import Path from PIL import Image def connected_components(ink: set, w: int, h: int) -> list[list[tuple[int, int]]]: """8 邻域连通分量,按大小降序返回。""" seen: set = set() comps: list[list[tuple[int, int]]] = [] for p in ink: if p in seen: continue stack = [p] seen.add(p) comp = [] while stack: cx, cy = stack.pop() comp.append((cx, cy)) for dx in (-1, 0, 1): for dy in (-1, 0, 1): if dx == 0 and dy == 0: continue q = (cx + dx, cy + dy) if q in seen or q not in ink: continue seen.add(q) stack.append(q) comps.append(comp) comps.sort(key=len, reverse=True) return comps def process_one(image: str, args: argparse.Namespace) -> int: try: im = Image.open(image).convert("RGB") except Exception as e: print(f"error: cannot open {image}: {e}", file=sys.stderr) return 1 w, h = im.size auto_mode = args.region is None if auto_mode: try: import numpy as np except ImportError: print("error: auto mode requires numpy; pass --region instead", file=sys.stderr) return 1 arr = np.asarray(im).astype(int) box = None if args.boxes: try: box = tuple(int(v) for v in args.boxes.split(",")) except ValueError: print("error: --boxes must be X1,Y1,X2,Y2", file=sys.stderr) return 1 cx, cy = (box[0] + box[2]) / 2, (box[1] + box[3]) / 2 radius = args.disc_radius or max(box[2] - box[0], box[3] - box[1]) * 0.8 else: cx, cy = w / 2, h / 2 radius = args.disc_radius or min(w, h) / 2 * 0.6 yy, xx = np.mgrid[0:h, 0:w] dist = np.sqrt((xx - cx) ** 2 + (yy - cy) ** 2) ring = (dist > radius * 0.75) & (dist < radius * 0.95) if ring.sum() == 0: print("error: disc ring empty (--disc-radius too small?)", file=sys.stderr) return 1 mean = arr[ring].mean(0).round(0).astype(int) auto_excl = tuple(int(v) for v in mean) print(f"auto: center=({cx:.0f},{cy:.0f}) disc radius≈{radius:.0f} " f"exclude-color=#{auto_excl[0]:02X}{auto_excl[1]:02X}{auto_excl[2]:02X}") x1, y1, x2, y2 = 0, 0, w, h else: auto_excl = None box = None try: x1, y1, x2, y2 = (int(v) for v in args.region.split(",")) except ValueError: print("error: --region must be X1,Y1,X2,Y2", file=sys.stderr) return 1 x1, y1 = max(0, x1), max(0, y1) x2, y2 = min(w, x2), min(h, y2) if x2 <= x1 or y2 <= y1: print("error: empty region", file=sys.stderr) return 1 excl = None if auto_excl: excl = auto_excl exclude_tol = 35 # 自动模式:圆底渐变排除容差(默认 24 排不干净渐变) elif args.exclude_color: v = args.exclude_color.lstrip("#") try: excl = tuple(int(v[i:i + 2], 16) for i in (0, 2, 4)) except ValueError: print("error: --exclude-color must be #RRGGBB", file=sys.stderr) return 1 exclude_tol = args.exclude_tol else: exclude_tol = args.exclude_tol ink: set = set() px = im.load() for y in range(y1, y2): for x in range(x1, x2): r, g, b = px[x, y] mx, mn = max(r, g, b), min(r, g, b) if args.mode == "color": if mx - mn <= args.sat: continue else: if mx >= args.dark: continue if excl is not None: d = ((r - excl[0]) ** 2 + (g - excl[1]) ** 2 + (b - excl[2]) ** 2) ** 0.5 if d <= exclude_tol: continue ink.add((x - x1, y - y1)) if not ink: print("error: no foreground pixels found in region (raise --sat or lower --dark?)", file=sys.stderr) return 1 comps = connected_components(ink, x2 - x1, y2 - y1) # 图标可能由多个分离子形状组成(云朵 logo 的 ">_" 与轮廓不相连), # 保留所有足够大的分量;噪点散点(远小于最大分量)自动排除。 min_size = max(len(comps[0]) * 0.02, 8) mx0 = [p[0] for p in comps[0]] my0 = [p[1] for p in comps[0]] main_box = (min(mx0), min(my0), max(mx0), max(my0)) def overlaps_main(c) -> bool: cx = [p[0] for p in c] cy = [p[1] for p in c] return not (max(cx) < main_box[0] or min(cx) > main_box[2] or max(cy) < main_box[1] or min(cy) > main_box[3]) kept = [c for c in comps if len(c) >= min_size or overlaps_main(c)] if auto_mode: # 自动模式:选单一图形分量——前 3 大彩色分量中饱和度最高者(白环/波纹 sat 低出局)。 colored = [] for c in kept: rs = [px[x1 + p[0], y1 + p[1]][0] for p in c] gs = [px[x1 + p[0], y1 + p[1]][1] for p in c] bs = [px[x1 + p[0], y1 + p[1]][2] for p in c] n = len(c) sat = max(sum(rs) / n, sum(gs) / n, sum(bs) / n) - min(sum(rs) / n, sum(gs) / n, sum(bs) / n) if sat > 25: colored.append((len(c), sat, c)) if colored: colored.sort(key=lambda t: t[0], reverse=True) chosen = max(colored[:3], key=lambda t: t[1])[2] else: chosen = kept[0] if box: bx0, by0, bx1, by1 = box scored = [] for c in kept: overlap = sum(1 for p in c if bx0 <= p[0] < bx1 and by0 <= p[1] < by1) scored.append((overlap, len(c), c)) scored.sort(key=lambda t: (t[0], t[1]), reverse=True) chosen = scored[0][2] best = chosen pad = 0 else: best = [p for c in kept for p in c] pad = args.pad bx1 = x1 + min(p[0] for p in best) - pad by1 = y1 + min(p[1] for p in best) - pad bx2 = x1 + max(p[0] for p in best) + 1 + pad by2 = y1 + max(p[1] for p in best) + 1 + pad bx1, by1 = max(0, bx1), max(0, by1) bx2, by2 = min(w, bx2), min(h, by2) out = Image.new("RGBA", (bx2 - bx1, by2 - by1), (0, 0, 0, 0)) o = out.load() for lx, ly in best: o[x1 + lx - bx1, y1 + ly - by1] = px[x1 + lx, y1 + ly] + (255,) if not args.no_keep_whites: # 背景填充:近白像素从 bbox 边缘 flood fill -> 透明(外部背景); # 被彩色前景包围的近白像素(图标内部白色镂空细节)-> 保留纯白。 w, h = out.width, out.height near: set = set() for y in range(h): for x in range(w): r, g, b, a = o[x, y] if a == 0: continue mx, mn = max(r, g, b), min(r, g, b) if mx >= 240 and mx - mn <= 25: near.add((x, y)) bg: set = set() stack = [p for p in near if p[0] == 0 or p[1] == 0 or p[0] == w - 1 or p[1] == h - 1] while stack: cx, cy = stack.pop() if (cx, cy) in bg: continue bg.add((cx, cy)) for dx in (-1, 0, 1): for dy in (-1, 0, 1): q = (cx + dx, cy + dy) if q in near and q not in bg: stack.append(q) for x, y in bg: o[x, y] = (255, 255, 255, 0) for x, y in near - bg: o[x, y] = (255, 255, 255, 255) if args.output: dest = Path(args.output) elif auto_mode: dest = Path(image).with_suffix(".clean.png") else: dest = Path(image).with_suffix(".fg.png") out.save(dest) print(f"bbox (原图像素): x1: {bx1}, y1: {by1}, x2: {bx2}, y2: {by2}") print(f"前景像素: {len(best)} 保留分量: {len(kept)}/{len(comps)} 最大分量占比: {len(comps[0]) / len(ink) * 100:.0f}%") print(f"wrote {dest} ({out.width}x{out.height})") return 0 def main() -> int: ap = argparse.ArgumentParser(description="从截图区域提取图标/logo 前景(透明 PNG)") ap.add_argument("image", nargs="+", help="截图路径(PNG/JPEG/WebP)") ap.add_argument("--region", metavar="X1,Y1,X2,Y2", help="手动模式:搜索区域(原图像素,宽松即可);不传则自动推断(图标居中)") ap.add_argument("-o", "--output", default=None, help="输出 PNG 路径(默认 <image-stem>.fg.png;自动模式 <image-stem>.clean.png)") ap.add_argument("--disc-radius", type=float, default=None, help="自动模式:圆底半径(像素),缺省 = min(宽,高)/2 × 0.6") ap.add_argument("--boxes", default=None, metavar="X1,Y1,X2,Y2", help="自动模式:ground 在放大图上的框,用于圆心/半径校正与分量重叠筛选") ap.add_argument("--mode", choices=("color", "dark"), default="color", help="color=彩色线条(默认);dark=灰色/黑色线条(logo、图标)") ap.add_argument("--sat", type=int, default=12, help="color 模式饱和度阈值(RGB max-min,默认 12)") ap.add_argument("--dark", type=int, default=215, help="dark 模式亮度阈值(RGB max,默认 215,即 #BABBBC 级灰线可收)") ap.add_argument("--exclude-color", default=None, metavar="#RRGGBB", help="排除与该颜色接近的像素(彩色背景干扰时用)") ap.add_argument("--exclude-tol", type=float, default=24, help="--exclude-color 的距离容差(默认 24)") ap.add_argument("--pad", type=int, default=3, help="输出 bbox 每边外扩像素(默认 3)") ap.add_argument("--no-keep-whites", action="store_true", help="不保留被前景包围的内部白色细节(默认保留)") args = ap.parse_args() code = 0 for image in args.image: code |= process_one(image, args) return code if __name__ == "__main__": sys.exit(main())skills/vision-skills/scripts/html_shot.py23 KBe378e0a5404aupstream ↗
#!/usr/bin/env python3 """Screenshot an HTML file with headless Chrome/Chromium/Edge. The default path intentionally remains the small Chrome CLI wrapper used by the original upstream tool. ``--full-page`` uses Chrome DevTools Protocol instead: the requested viewport remains the layout viewport while the page's CSS scroll height is measured and captured in one pass. """ import argparse import base64 import hashlib import json import os import shutil import socket import struct import subprocess import tempfile import time from pathlib import Path from urllib.error import URLError from urllib.parse import urlparse from urllib.request import urlopen CHROME_CANDIDATES = ( # macOS "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "/Applications/Chromium.app/Contents/MacOS/Chromium", "/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge", # Linux "google-chrome", "google-chrome-stable", "chromium", "chromium-browser", "microsoft-edge", "brave-browser", ) FULL_PAGE_TIMEOUT_SECONDS = 30.0 FULL_PAGE_MAX_GROWTH_ROUNDS = 25 FULL_PAGE_MAX_SCROLL_STEPS = 2000 FULL_PAGE_QUIET_MILLISECONDS = 1250 def find_chrome() -> str | None: for candidate in CHROME_CANDIDATES: path = shutil.which(candidate) if path: return path if candidate.startswith("/") and Path(candidate).is_file(): return candidate for env in ("PROGRAMFILES", "PROGRAMFILES(X86)", "LOCALAPPDATA"): base = os.environ.get(env) if not base: continue for sub in ("Google/Chrome/Application/chrome.exe", "Microsoft/Edge/Application/msedge.exe"): path = Path(base) / sub if path.is_file(): return str(path) return None def default_output(source: str) -> str: if source.startswith(("http://", "https://", "file://", "data:")): stem = Path(urlparse(source).path).stem return f"{stem or 'page'}.png" return f"{Path(source).stem}.png" def read_http_json(port: int, path: str) -> object: with urlopen(f"http://127.0.0.1:{port}{path}", timeout=1.0) as response: return json.loads(response.read().decode("utf-8")) class DevToolsSocket: """Minimal RFC 6455 client for the Chrome DevTools Protocol.""" def __init__(self, websocket_url: str): parsed = urlparse(websocket_url) if parsed.scheme != "ws" or parsed.hostname is None or parsed.port is None: raise RuntimeError("invalid DevTools websocket URL") self.socket = socket.create_connection((parsed.hostname, parsed.port), timeout=5.0) self.socket.settimeout(5.0) self.buffer = bytearray() key = base64.b64encode(os.urandom(16)).decode("ascii") request = ( f"GET {parsed.path or '/'} HTTP/1.1\r\n" f"Host: {parsed.hostname}:{parsed.port}\r\n" "Upgrade: websocket\r\n" "Connection: Upgrade\r\n" f"Sec-WebSocket-Key: {key}\r\n" "Sec-WebSocket-Version: 13\r\n\r\n" ).encode("ascii") self.socket.sendall(request) response = self._read_until(b"\r\n\r\n") if not response.startswith(b"HTTP/1.1 101"): raise RuntimeError("Chrome did not accept the DevTools websocket") expected_accept = base64.b64encode(hashlib.sha1( f"{key}258EAFA5-E914-47DA-95CA-C5AB0DC85B11".encode("ascii") ).digest()).decode("ascii") headers = response.decode("latin-1").split("\r\n") accept = next((line.split(":", 1)[1].strip() for line in headers if line.lower().startswith("sec-websocket-accept:")), None) if accept != expected_accept: raise RuntimeError("Chrome returned an invalid DevTools websocket handshake") self.next_id = 0 def _read_until(self, marker: bytes) -> bytes: data = bytearray() while marker not in data: chunk = self.socket.recv(4096) if not chunk: raise RuntimeError("DevTools websocket closed during handshake") data.extend(chunk) boundary = data.index(marker) + len(marker) self.buffer.extend(data[boundary:]) return bytes(data[:boundary]) def _send_frame(self, opcode: int, payload: bytes) -> None: mask = os.urandom(4) masked = bytes(value ^ mask[index % 4] for index, value in enumerate(payload)) length = len(masked) if length < 126: header = struct.pack("!BB", 0x80 | opcode, 0x80 | length) elif length < 65536: header = struct.pack("!BBH", 0x80 | opcode, 0x80 | 126, length) else: header = struct.pack("!BBQ", 0x80 | opcode, 0x80 | 127, length) self.socket.sendall(header + mask + masked) def _read_frame(self) -> tuple[bool, int, bytes]: header = self._read_exact(2) first, second = header final = (first & 0x80) != 0 opcode = first & 0x0F length = second & 0x7F masked = (second & 0x80) != 0 if length == 126: length = struct.unpack("!H", self._read_exact(2))[0] elif length == 127: length = struct.unpack("!Q", self._read_exact(8))[0] mask = self._read_exact(4) if masked else None payload = self._read_exact(length) if mask is not None: payload = bytes(value ^ mask[index % 4] for index, value in enumerate(payload)) return final, opcode, payload def _read_message(self) -> tuple[int, bytes]: fragments: list[bytes] = [] message_opcode: int | None = None while True: final, opcode, payload = self._read_frame() if opcode == 0x9: self._send_frame(0xA, payload) continue if opcode == 0x8: raise RuntimeError("DevTools websocket closed") if opcode in (0x1, 0x2): fragments = [payload] message_opcode = opcode elif opcode == 0x0 and message_opcode is not None: fragments.append(payload) else: continue if final and message_opcode is not None: return message_opcode, b"".join(fragments) def _read_exact(self, length: int) -> bytes: data = bytearray() if self.buffer: take = min(length, len(self.buffer)) data.extend(self.buffer[:take]) del self.buffer[:take] while len(data) < length: chunk = self.socket.recv(length - len(data)) if not chunk: raise RuntimeError("DevTools websocket closed unexpectedly") data.extend(chunk) return bytes(data) def command(self, method: str, params: dict | None = None) -> dict: self.next_id += 1 request_id = self.next_id request = {"id": request_id, "method": method} if params is not None: request["params"] = params self._send_frame(0x1, json.dumps(request, separators=(",", ":")).encode("utf-8")) while True: opcode, payload = self._read_message() if opcode != 0x1: continue message = json.loads(payload.decode("utf-8")) if message.get("id") != request_id: continue if "error" in message: raise RuntimeError(f"{method}: {message['error'].get('message', 'CDP command failed')}") return message.get("result", {}) def close(self) -> None: try: self._send_frame(0x8, b"") except OSError: pass self.socket.close() def evaluate(client: DevToolsSocket, expression: str) -> object: result = client.command("Runtime.evaluate", { "expression": expression, "returnByValue": True, "awaitPromise": True, }) if "exceptionDetails" in result: raise RuntimeError(result["exceptionDetails"].get("text", "Runtime.evaluate failed")) remote = result.get("result", {}) if remote.get("subtype") == "error": raise RuntimeError(remote.get("description", "Runtime.evaluate failed")) return remote.get("value") def wait_for_target(port: int, deadline: float) -> dict: last_error: Exception | None = None while time.monotonic() < deadline: try: targets = read_http_json(port, "/json/list") if isinstance(targets, list): for target in targets: if (isinstance(target, dict) and target.get("type") == "page" and target.get("webSocketDebuggerUrl")): return target except (OSError, URLError, ValueError) as error: last_error = error time.sleep(0.05) detail = f": {last_error}" if last_error else "" raise RuntimeError(f"timed out waiting for Chrome DevTools{detail}") def wait_for_devtools_port(profile: Path, process: subprocess.Popen, deadline: float) -> int: active_port = profile / "DevToolsActivePort" last_error: Exception | None = None while time.monotonic() < deadline: if process.poll() is not None: raise RuntimeError(f"Chrome exited before DevTools was ready (code {process.returncode})") try: lines = active_port.read_text().splitlines() port = int(lines[0]) if 0 < port <= 65535: return port except (OSError, ValueError, IndexError) as error: last_error = error time.sleep(0.05) detail = f": {last_error}" if last_error else "" raise RuntimeError(f"timed out waiting for Chrome DevTools port{detail}") def wait_for_ready(client: DevToolsSocket, deadline: float) -> None: while time.monotonic() < deadline: if evaluate(client, "document.readyState") == "complete": return time.sleep(0.05) raise RuntimeError("timed out waiting for the HTML document") def pause_for_frame(seconds: float) -> None: time.sleep(max(0.02, seconds)) def build_full_page_command(chrome: str, profile: Path, width: int, height: int, scale: int) -> list[str]: command = [ chrome, "--headless=new", "--disable-gpu", "--hide-scrollbars", "--no-first-run", "--no-default-browser-check", "--use-mock-keychain", "--blink-settings=imagesLazyLoadingEnabled=false", f"--window-size={width},{height}", "--remote-debugging-port=0", "--remote-debugging-address=127.0.0.1", f"--user-data-dir={profile}", ] if scale != 1: command.append(f"--force-device-scale-factor={scale}") command.append("about:blank") return command def measure_page_height(client: DevToolsSocket) -> int: return int(evaluate(client, """ Math.max(document.documentElement.scrollHeight, document.body ? document.body.scrollHeight : 0) """) or 0) def enforce_pixel_limit(width: int, page_height: int, scale: int, max_pixels: int | None) -> None: if page_height <= 0: raise RuntimeError("document has no measurable scroll height") output_pixels = width * page_height * scale * scale if max_pixels is not None and output_pixels > max_pixels: raise RuntimeError( f"full-page screenshot would exceed --max-pixels " f"({output_pixels} > {max_pixels})" ) def ensure_before_deadline(deadline: float, message: str) -> None: if time.monotonic() >= deadline: raise RuntimeError(message) def wait_for_dom_quiet(client: DevToolsSocket, deadline: float) -> None: remaining_ms = int((deadline - time.monotonic()) * 1000) if remaining_ms <= FULL_PAGE_QUIET_MILLISECONDS: raise RuntimeError("timed out before the page reached a quiet state") max_wait_ms = min(4500, remaining_ms) result = evaluate(client, f""" (() => new Promise(resolve => {{ const quietMs = {FULL_PAGE_QUIET_MILLISECONDS}; const maxWaitMs = {max_wait_ms}; const height = () => Math.max( document.documentElement.scrollHeight, document.body ? document.body.scrollHeight : 0 ); let lastHeight = height(); let quietTimer; let pollTimer; let maxTimer; let finished = false; const finish = quiet => {{ if (finished) return; finished = true; clearTimeout(quietTimer); clearInterval(pollTimer); clearTimeout(maxTimer); resolve({{quiet, height: height()}}); }}; const restartQuietTimer = () => {{ clearTimeout(quietTimer); quietTimer = setTimeout(() => finish(true), quietMs); }}; pollTimer = setInterval(() => {{ const currentHeight = height(); if (currentHeight !== lastHeight) {{ lastHeight = currentHeight; restartQuietTimer(); }} }}, 100); maxTimer = setTimeout(() => finish(false), maxWaitMs); restartQuietTimer(); }}))() """) if not isinstance(result, dict) or result.get("quiet") is not True: raise RuntimeError("page did not become quiet before the full-page deadline") def settle_full_page(client: DevToolsSocket, width: int, height: int, scale: int, max_pixels: int | None, deadline: float) -> int: page_height = measure_page_height(client) enforce_pixel_limit(width, page_height, scale, max_pixels) scan_start = 0 growth_rounds = 0 scroll_steps = 0 waited_for_images = False while True: ensure_before_deadline(deadline, "timed out while stabilizing the full page") measured_before_sweep = page_height for position in range(scan_start, measured_before_sweep, max(1, height)): scroll_steps += 1 if scroll_steps > FULL_PAGE_MAX_SCROLL_STEPS: raise RuntimeError("full page requires too many scroll steps to capture safely") ensure_before_deadline(deadline, "timed out while scrolling the full page") evaluate(client, f"window.scrollTo(0, {position}); true") pause_for_frame(0.12) evaluate(client, f"window.scrollTo(0, {measured_before_sweep}); true") pause_for_frame(0.12) wait_for_dom_quiet(client, deadline) measured_after_sweep = measure_page_height(client) enforce_pixel_limit(width, measured_after_sweep, scale, max_pixels) if measured_after_sweep != measured_before_sweep: growth_rounds += 1 if growth_rounds > FULL_PAGE_MAX_GROWTH_ROUNDS: raise RuntimeError("page is still changing after repeated full-page sweeps") scan_start = (max(0, measured_before_sweep - height) if measured_after_sweep > measured_before_sweep else 0) page_height = measured_after_sweep waited_for_images = False continue if not waited_for_images: evaluate(client, """ Promise.race([ Promise.all(Array.from(document.images) .filter(image => !image.complete) .map(image => new Promise(resolve => { image.addEventListener('load', resolve, {once: true}); image.addEventListener('error', resolve, {once: true}); }))), new Promise(resolve => setTimeout(resolve, 3000)) ]).then(() => true) """) waited_for_images = True ensure_before_deadline(deadline, "timed out while waiting for full-page images") measured_after_images = measure_page_height(client) enforce_pixel_limit(width, measured_after_images, scale, max_pixels) if measured_after_images != page_height: growth_rounds += 1 if growth_rounds > FULL_PAGE_MAX_GROWTH_ROUNDS: raise RuntimeError("page is still changing after image loading") scan_start = (max(0, page_height - height) if measured_after_images > page_height else 0) page_height = measured_after_images waited_for_images = False continue evaluate(client, "window.scrollTo(0, 0); true") wait_for_dom_quiet(client, deadline) final_height = measure_page_height(client) enforce_pixel_limit(width, final_height, scale, max_pixels) if final_height == page_height: return final_height growth_rounds += 1 if growth_rounds > FULL_PAGE_MAX_GROWTH_ROUNDS: raise RuntimeError("page is still changing before full-page capture") scan_start = max(0, page_height - height) if final_height > page_height else 0 page_height = final_height waited_for_images = False def capture_full_page(chrome: str, source: str, output: Path, width: int, height: int, scale: int, wait_ms: int, max_pixels: int | None) -> int: profile = Path(tempfile.mkdtemp(prefix="agent-vision-html-shot-")) process: subprocess.Popen | None = None client: DevToolsSocket | None = None try: command = build_full_page_command(chrome, profile, width, height, scale) process = subprocess.Popen(command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) startup_deadline = time.monotonic() + 15.0 port = wait_for_devtools_port(profile, process, startup_deadline) target = wait_for_target(port, startup_deadline) client = DevToolsSocket(str(target["webSocketDebuggerUrl"])) client.command("Page.enable") client.command("Runtime.enable") client.command("Emulation.setDeviceMetricsOverride", { "width": width, "height": height, "deviceScaleFactor": scale, "mobile": False, "screenWidth": width, "screenHeight": height, }) client.command("Page.navigate", {"url": source}) wait_for_ready(client, time.monotonic() + 15.0) if wait_ms > 0: pause_for_frame(wait_ms / 1000.0) # Keep the requested viewport fixed. Scrolling, rather than growing # the viewport, wakes IO/reveal handlers without changing vh/svh. evaluate(client, """ (() => { document.documentElement.style.scrollBehavior = 'auto'; if (document.body) document.body.style.scrollBehavior = 'auto'; window.scrollTo(0, 0); return true; })() """) page_height = settle_full_page( client, width, height, scale, max_pixels, time.monotonic() + FULL_PAGE_TIMEOUT_SECONDS, ) result = client.command("Page.captureScreenshot", { "format": "png", "fromSurface": True, "captureBeyondViewport": True, "clip": {"x": 0, "y": 0, "width": width, "height": page_height, "scale": 1}, }) data = result.get("data") if not isinstance(data, str): raise RuntimeError("Chrome did not return screenshot data") output.write_bytes(base64.b64decode(data)) return page_height finally: if client is not None: client.close() if process is not None and process.poll() is None: process.terminate() try: process.wait(timeout=2) except subprocess.TimeoutExpired: process.kill() process.wait(timeout=2) shutil.rmtree(profile, ignore_errors=True) def main() -> None: parser = argparse.ArgumentParser( prog="html_shot", description="Screenshot an HTML file (or URL) to PNG with headless Chrome/Chromium/Edge", ) parser.add_argument("source", help="path to a local .html file, or an http(s):// URL") parser.add_argument("-o", "--output", help="output PNG path (default: <source-stem>.png in the current directory)") parser.add_argument("--width", type=int, default=1280, help="viewport width in CSS pixels (default: 1280)") parser.add_argument("--height", type=int, default=800, help="viewport height in CSS pixels (default: 800)") parser.add_argument("--scale", type=int, default=1, help="device scale factor: 2 makes the PNG 2x for small text (default: 1)") parser.add_argument("--wait-ms", type=int, default=0, help="wait time in ms before capturing (default: 0, capture immediately)") parser.add_argument("--full-page", action="store_true", help="capture the full CSS scroll height while keeping the requested viewport") parser.add_argument("--max-pixels", type=int, help="reject full-page captures larger than this many output pixels") args = parser.parse_args() chrome = find_chrome() if not chrome: parser.exit(1, "html_shot: no Chrome/Chromium/Edge found; install one and retry\n") source = args.source if not source.startswith(("http://", "https://", "file://", "data:")): path = Path(source).expanduser() if not path.is_file(): parser.exit(1, f"html_shot: file not found: {path}\n") source = path.resolve().as_uri() output = Path(args.output).expanduser().resolve() if args.output else Path(default_output(source)).resolve() if args.full_page: try: page_height = capture_full_page( chrome, source, output, args.width, args.height, args.scale, args.wait_ms, args.max_pixels, ) except Exception as error: parser.exit(1, f"html_shot: full-page capture failed: {error}\n") print(f"wrote {output} ({args.width * args.scale}x{page_height * args.scale}; pageHeight={page_height})") return command = [ chrome, "--headless=new", "--disable-gpu", "--hide-scrollbars", "--no-first-run", "--no-default-browser-check", f"--window-size={args.width},{args.height}", f"--screenshot={output}", ] if args.scale != 1: command.append(f"--force-device-scale-factor={args.scale}") if args.wait_ms > 0: command.append(f"--virtual-time-budget={args.wait_ms}") command.append(source) result = subprocess.run(command, text=True, capture_output=True) if result.returncode != 0 or not output.is_file(): message = result.stderr.strip() or result.stdout.strip() or f"chrome exited with code {result.returncode}" parser.exit(1, f"html_shot: capture failed: {message}\n") print(f"wrote {output} ({args.width * args.scale}x{args.height * args.scale})") if __name__ == "__main__": main()skills/vision-skills/scripts/long_screenshot_ocr.py44 KB9546bc430131upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/vision-skills/scripts/pixel_diff.py3.9 KB36689712fc23upstream ↗
#!/usr/bin/env python3 """pixel_diff: compare a rebuilt image against the original and rank the worst regions. Verification is the step agents skip or fake, because comparing two descriptions feels like comparing two images. It is not. This runs the real comparison and answers the only question that matters next: which region is most wrong, so you know where to look and what to fix first. Boxes are printed in the same `x1: .., y1: ..` form as ground/detect, so a bad region can be pasted straight into `glance --region` or `detect --region`. """ from __future__ import annotations import argparse from pathlib import Path try: from PIL import Image, ImageChops, ImageStat except ImportError: Image = None def load(path: Path, size: tuple[int, int] | None = None) -> "Image.Image": # Traced SVGs and screenshots of transparent UI render with an alpha # channel; unflattened transparency reads as black and shows up as a huge # phantom diff. Compositing on white is what a viewer would show. image = Image.open(path) if image.mode in ("RGBA", "LA", "P"): image = image.convert("RGBA") canvas = Image.new("RGB", image.size, "white") canvas.paste(image, mask=image.split()[-1]) image = canvas else: image = image.convert("RGB") return image.resize(size, Image.LANCZOS) if size and image.size != size else image def cell_scores(diff: "Image.Image", grid: int) -> list[tuple[float, tuple[int, int, int, int]]]: width, height = diff.size grey = diff.convert("L") scores = [] for row in range(grid): for column in range(grid): box = (round(column * width / grid), round(row * height / grid), round((column + 1) * width / grid), round((row + 1) * height / grid)) if box[2] > box[0] and box[3] > box[1]: mean = ImageStat.Stat(grey.crop(box)).mean[0] scores.append((mean / 255 * 100, box)) return sorted(scores, reverse=True) def main() -> None: parser = argparse.ArgumentParser( prog="pixel_diff", description="Pixel-diff a rebuilt image against the original and rank the worst regions", ) parser.add_argument("original", type=Path, help="the reference image") parser.add_argument("rebuilt", type=Path, help="your rendered reproduction") parser.add_argument("--grid", type=int, default=6, help="split into GRID x GRID cells (default: 6)") parser.add_argument("--top", type=int, default=5, help="how many worst regions to print (default: 5)") parser.add_argument("-o", "--output", type=Path, help="write a diff heatmap image here") args = parser.parse_args() if Image is None: parser.exit(1, "pixel_diff: requires Pillow; install the optional dependency pillow first\n") for path in (args.original, args.rebuilt): if not path.expanduser().is_file(): parser.exit(1, f"pixel_diff: image not found: {path}\n") original = load(args.original.expanduser()) with Image.open(args.rebuilt.expanduser()) as probe: raw_size = probe.size rebuilt = load(args.rebuilt.expanduser(), size=original.size) if raw_size != original.size: # Worth saying out loud: a size mismatch is itself a finding, and every # box printed below is in the original's coordinates, not the rebuild's. print(f"note: rebuilt was {raw_size[0]}x{raw_size[1]}, scaled to {original.size[0]}x{original.size[1]}") diff = ImageChops.difference(original, rebuilt) overall = ImageStat.Stat(diff.convert("L")).mean[0] / 255 * 100 print(f"overall difference: {overall:.2f}%") if args.output: diff.save(args.output.expanduser()) print(f"heatmap: {args.output}") for index, (score, box) in enumerate(cell_scores(diff, args.grid)[:args.top], 1): print(f"{index}. {score:.2f}% x1: {box[0]}, y1: {box[1]}, x2: {box[2]}, y2: {box[3]}") if __name__ == "__main__": main()skills/vision-skills/references/gui.md3.9 KB39968a17ed9dupstream ↗
# Operating a GUI from screenshots **When to use**: the task is to act on a live screen — click, type, scroll — guided by screenshots: driving an app, automating a workflow, walking a bug reproduction. The vision layer answers "where is it" and "what state is it in"; the acting itself goes through whatever automation channel the task already has. Tool syntax lives in `SKILL.md`. This file is the sequence and the pass/fail test. ## Steps **1. Calibrate coordinates once, before the first click.** Screenshot pixels and pointer coordinates are often different spaces: on HiDPI displays a screenshot is 2-3× the logical points the click API takes. Compare the screenshot's pixel width to the screen's logical width and divide every box by that ratio. Skipping this puts every click at half or double distance from the origin — consistently wrong in a way that looks like bad grounding. **2. Prefer the UI tree; fall back to vision.** If the environment exposes a UI tree (Android `uiautomator dump`, desktop accessibility tree, browser DOM), read coordinates and state from there — it is exact, fast, and carries semantic attributes (enabled, checked, focusable) that pixels cannot. Use `ground` only when: - No tree is available (games, custom-rendered canvases, remote desktops), - The target element is absent from the tree (canvas-drawn overlays, non-standard widgets), or - You need to verify a visual state the tree doesn't expose (color, icon appearance). When using `ground`: centers, not corners — ground boxes are not pixel-exact at the edges. For tiny targets (checkboxes, close buttons), go coarse-to-fine: ground the containing block, then `ground --region` inside it. For elements that stay put across interactions (toolbar buttons, sidebar links, fixed panels), ground them once and record a coordinate table — an id/label plus center coordinates. Subsequent clicks reference the table entry directly instead of re-running ground each time. Invalidate the table when the layout changes (window resize, navigation to a different page, scroll). **3. One screenshot per action — act, re-shoot, verify, then continue.** Every action invalidates the current screenshot: menus open, focus moves, content loads; scrolling invalidates every box at once. Chaining several clicks on one stale screenshot is how you click through where a dialog used to be. **4. Verify as a question about the expected state.** `glance <after.png> -q "is the settings dialog open?"` — ask for the state the action should have produced, not a general description. If the answer is no or unexpected, stop and re-inventory (`detect`) instead of pressing on against an assumed screen. When the change is small (a toggle, one badge), diff first: `pixel_diff before.png after.png` finds the changed region, `glance --region` reads it. **5. After typing, read the field back.** Focus loss eats keystrokes silently. `glance --region <field box> --ocr` confirms the text actually landed before you submit. ## Verify The loop in steps 3-4 *is* the verification: no action counts as done until a fresh screenshot answers the expected-state question. For a multi-step flow, the final screenshot must show the end state the task defines — the confirmation page, the saved indicator. A completed action list is not evidence; the pixels are. ## Boundaries - Loading is asynchronous: after an action that triggers it, poll — re-screenshot until two consecutive shots stop differing (`pixel_diff` near 0) — rather than trusting one fixed sleep. - Irreversible actions (send, delete, pay) follow the calling agent's own confirmation policy. The vision layer reports what is on screen; it never makes that call. - A screen the task didn't predict — an error dialog, a permission prompt, a login wall — is a stop-and-surface, not an obstacle to click through. Clicking through unknown dialogs is how automations do damage.skills/vision-skills/references/long-screenshot-ocr.md3.3 KBaeefaf4d85daupstream ↗
# OCR a long screenshot without losing boundary text **Use for**: a vertically scrolling screenshot, chat history, long web page, requirements conversation, log view, or other image that would be downscaled too aggressively in one vision call. For a PDF or an exported document, prefer a format-aware document parser instead of screenshot OCR. ## Run the workflow Work from a durable copy, then choose the content mode: ```bash # Web pages, documents, logs, tables, and other general content python3 scripts/long_screenshot_ocr.py work/page.png -o work/page.ocr.md # Chat histories: preserve message grouping, senders, timestamps, and quotes python3 scripts/long_screenshot_ocr.py work/chat.png --mode chat -o work/chat.ocr.md ``` Without `-o`, write `<input-stem>.ocr.md`; store chunks, sidecars, the manifest, and the audit in `<input-stem>_chunks/` unless `--chunks-dir` overrides it. The script performs four operations: 1. Measure per-row content density and find low-content cut bands near the target height. 2. Add pixel overlap only when no safe band exists, so text crossing a risky cut appears in both adjacent chunks. 3. Run `glance` on the chunks using the existing `VISION_*` configuration; chat mode requests structured messages while general mode uses verbatim OCR. 4. Merge only confident repeated lines or messages and write `manifest.json` plus `ocr_audit.md` beside the chunks. Use `--resume` after an interrupted run. A chunk is reused only when its image, mode, and custom prompt fingerprint still match: ```bash python3 scripts/long_screenshot_ocr.py work/chat.png --mode chat --resume -o work/chat.ocr.md ``` Use `--split-only` when you need to inspect or tune the chunks before spending vision calls: ```bash python3 scripts/long_screenshot_ocr.py work/page.png --split-only ``` If the defaults produce awkward chunks, rerun with `--target-height`, `--min-height`, `--max-height`, or `--overlap`. Keep enough height for local context; do not make tiny OCR tiles unless the source text is unusually small. ## Verify before delivering 1. Read the merged Markdown from top to bottom and compare its opening and ending lines with the source. 2. Open `ocr_audit.md`. Review every boundary marked `yes` against the two adjacent `chunk_*.png` files. A marked boundary used pixel overlap or fuzzy text matching and is not safe to accept blindly. 3. Check sender changes, timestamps, quoted messages, table row breaks, code indentation, and paragraphs that cross chunk boundaries. 4. For any doubtful text, run targeted OCR on the relevant chunk or crop: ```bash glance work/page_chunks/chunk_002.png --ocr "Re-check the final five lines carefully." glance work/page_chunks/chunk_002.png --region X1,Y1,X2,Y2 --ocr ``` 5. Keep visible spelling and punctuation verbatim. Write `[unreadable]` for text that remains illegible; do not silently guess or editorially repair it. ## Output contract - Return the merged `.ocr.md` file as the primary result. - Keep the chunk directory until verification is complete; it is the evidence for ordering and boundary decisions. - Report unresolved `[unreadable]` text and every boundary that still needs human review. - Do not claim a complete transcription when the first or last screenshot edge visibly clips content.skills/vision-skills/references/restore-graphic.md3.7 KBb14219ab6bdfupstream ↗
# Extracting or rebuilding a graphic asset **When to use**: the deliverable is an isolated icon, logo, illustration, decorative mark, or other visual asset as a transparent PNG or SVG. For a full page or component, read `restore-ui.md` and use this workflow only for the individual visual assets it identifies. ## Choose the least destructive representation Use this order: 1. Reuse the original source asset if it already exists in the project. 2. Extract a transparent PNG from the reference when the visual only needs to be displayed at the captured size. This preserves the original pixels and is the default for screenshot-only brand artwork. 3. Trace an irregular, flat, high-contrast shape when SVG is required. 4. Hand-write simple editable geometry or stroke icons, using the trace or pixel measurements as evidence rather than visual estimates. ## Extract a transparent PNG The final extraction region must be tight around the target ink. A loose crop that touches neighboring text, borders, or decoration may preserve those pixels as foreground. ```bash # Known tight region python3 scripts/extract_fg.py shot.png \ --region X1,Y1,X2,Y2 -o asset.png # Dark or gray line art python3 scripts/extract_fg.py shot.png \ --region X1,Y1,X2,Y2 --mode dark -o asset.png # Center and enlarge a small visual before automatic extraction crop shot.png --region X1,Y1,X2,Y2 --scale 4 -o work/asset4x.png python3 scripts/extract_fg.py work/asset4x.png ``` Use `--exclude-color '#RRGGBB'` when a connected background color would otherwise be retained. If automatic centering is wrong, provide a tighter region or a target box rather than repeatedly widening the crop. Inspect the result on both light and dark backgrounds. Confirm complete ink, clean alpha edges, and no neighboring fragments before reuse. ## Rebuild an SVG only when the deliverable needs one ```bash trace asset.png -o asset.svg trace asset.png --polygon -o asset.svg trace shot.png --region X1,Y1,X2,Y2 -o asset.svg ``` Ship traced paths directly for organic or irregular shapes. For rectangles, circles, pills, and other simple editable geometry, use the trace as a measurement and write the simpler SVG yourself. A small stroke icon is usually a hand-written SVG case, not a no-trace case: the trace outlines both sides of the raster stroke, while the desired asset is normally a centerline path with `stroke` and `fill="none"`. Use `trace --polygon` to recover endpoints, corners, and stroke width, then write the clean centerline path. When reusing traced paths: - Composite transparent SVG output on the intended background before judging it; transparent regions may appear black in some viewers. - Copy each path's `transform` together with its `d` data. Dropping the transform displaces otherwise correct geometry. - Keep opposite-winding subpaths together so holes remain holes. ## Verify Render the asset at its intended size and compare it with the same tight box from the reference. Use an overlay or `scripts/pixel_diff.py` to locate missing parts, contamination, wrong scale, or displaced paths. Judge the asset at its actual delivery size as well as zoomed in; a technically detailed SVG that looks worse at 16px is not an improvement. ## Boundaries - Whole screenshots and photos do not trace usefully. - Low-contrast art may disappear during binarization. Tighten the region, increase scale, or invert light-on-dark input before reaching for color tracing. - Use color tracing only for genuinely multicolor artwork; antialiased single- color input can otherwise fragment into many gray-level paths. - If the visual contains live text, controls, or data-driven content, rebuild those parts natively and use the extracted asset only for the static artwork.skills/vision-skills/references/restore-structure.md1.9 KB3261fcff7033upstream ↗
# Turning a visual diagram into structured code **When to use**: the task is converting a sketch, flowchart, architecture diagram, or whiteboard into Mermaid, Graphviz, JSON, or another structured representation. The goal is semantic and topological fidelity, not matching the source renderer pixel for pixel. ## Steps 1. Run one full-image `glance` pass for the diagram type, reading direction, major groups, and visual conventions. 2. Use `detect` for the initial node and label inventory. Refine dense groups with `detect --region`. 3. Use OCR for labels. Preserve visible spelling, punctuation, and abbreviations verbatim; write `[unreadable]` instead of guessing. 4. Locate ambiguous arrows, connectors, legends, and group boundaries with `ground` and targeted `glance --region` calls. 5. Build an explicit intermediate inventory before writing the output: nodes, labels, groups, edges, directions, and edge labels. 6. Generate the Mermaid, Graphviz, or requested structure from that inventory. ## Verify structure, not pixels Render the result, inventory it again, and compare these facts with the reference: - node count and label set - group membership and nesting - edge list and direction - edge labels and branch conditions - start/end or input/output roles Pixel differences are expected because the target renderer chooses its own fonts, spacing, and routing. A missing node or reversed arrow is a failure even when the two images look broadly similar; a different curve or gap is not a failure when the structure is correct. ## Boundaries - Do not tidy labels, expand abbreviations, or silently repair wording unless the user asks for editorial cleanup. - Do not infer an edge from proximity alone. If its endpoint or direction is unclear, zoom into the connector and mark unresolved facts explicitly. - If the user wants a visual SVG copy rather than structured code, follow `restore-graphic.md` instead.skills/vision-skills/references/restore-ui.md10.0 KBf49f796dbe9bupstream ↗
# Rebuilding a UI from a reference image **When to use**: the task is turning a screenshot or design image into a page, component, or application UI in HTML/CSS or the project's existing frontend stack. Checking an existing implementation against its reference is the same workflow entered at Verify. For an isolated icon, logo, or illustration, read `restore-graphic.md`; for Mermaid, Graphviz, or another structured diagram, read `restore-structure.md`. ## Choose the restore mode - Use **fast restore mode** when the user asks for a quick, rough, approximate, prototype, or first-pass reconstruction, or explicitly values speed over fidelity. Its target is a recognizable screenshot in about three minutes when the project already runs. - Use the **standard restore workflow** below when the user asks for close, precise, pixel-level, or production-ready alignment, or does not opt into a faster approximation. ## Fast restore mode: first screenshot in about three minutes Fast mode preserves the page's hierarchy, major regions, visible text, and primary state. It deliberately approximates fine spacing, exact colors, typography, shadows, decorative details, and icon geometry. ### Hard limits 1. Inspect the existing frontend stack, component library, icon set, source assets, and design tokens before writing code, but stop searching as soon as a usable local primitive is found. 2. Run one full-image `detect` pass. Treat its boxes as layout estimates and do not start a region-by-region inventory. 3. After that `detect` pass, use at most **six sequential image-inspection rounds** through `view_image` (or the host's equivalent built-in viewer) and `glance`. Each round may launch up to three independent calls concurrently, so the hard ceiling is 18 calls across six rounds. Batch unrelated regions or questions into the same round instead of waiting for each result before starting the next call. Normally one or two rounds are enough. 4. Do not use `trace`, foreground extraction, repeated color sampling, or iterative `pixel_diff` work in fast mode. Those are fidelity tools and will consume the delivery window. ### Build the approximation 1. Implement the largest layout regions first, then visible text, primary controls, and the most important state. Ignore details that are only visible when zoomed in. 2. Reuse the project's existing components and CSS tokens. If its frontend or icon library contains a reasonably similar component or icon, use it directly instead of recreating the reference. Prefer an approximate library icon over cropping, tracing, or hand-drawing a new one. 3. Use nearby existing palette tokens or visually similar CSS values. Exact sampled hex values, gradients, subtle borders, and shadow opacity are out of scope unless one of them defines the whole composition. 4. Keep text and controls native, selectable, and interactive. Fast mode relaxes visual fidelity, not basic UI behavior. ### Render once, fix once, deliver 1. Render the target viewport with `scripts/html_shot.py` or the project's existing browser setup. Use `--full-page` only when the reference covers a complete scrolling document; keep the same logical viewport either way. 2. Inspect the screenshot once. If there is an obvious structural failure such as a missing major region, broken wrapping, or a wildly wrong scale, make one focused correction and render once more. 3. Deliver the screenshot. Stop instead of spending the remaining time on small color, icon, font, shadow, radius, or spacing differences. A practical time box is roughly 30 seconds for project inspection plus `detect`, 90 seconds for implementation, and the remaining minute for startup, rendering, one correction, and screenshot delivery. Dependency installation or a project that does not already run may extend that target; do not compensate by silently switching back to a long precision loop. ## Standard restore workflow ### Core strategy: code-native UI plus screenshot-backed visuals Do not choose one reconstruction mode for the whole page. Classify each element separately. Most finished pages should combine both kinds: | Kind | Build it this way | Typical examples | |---|---|---| | **Code-native component** | Reuse the project's component library or build it with semantic HTML, CSS, and simple SVG primitives | layout, text, buttons, inputs, cards, tabs, dividers, backgrounds, simple geometric badges | | **Screenshot-backed visual** | Extract the original pixels, remove the background when needed, and place the result as an image asset | logos, proprietary icons, illustrations, decorative artwork, textures, complex static marks | Use this decision order for every element: 1. **Reuse an existing component or source asset from the project.** Search before recreating anything. 2. **Use a code-native component** when the element contains text, accepts input, changes state, must respond to layout, or is simple to express with the project's normal primitives. 3. **Use a screenshot-backed visual** when it is static, visually distinctive, expensive to redraw, and no source asset exists. 4. Rebuild a vector only when the user requires SVG/editability, the visual must scale beyond the screenshot resolution, or extraction cannot isolate it cleanly. Follow `restore-graphic.md` for that branch. Never flatten text, controls, or a large layout block into a screenshot. Never redraw extractable brand artwork merely to make the implementation look more "native". For a complex widget, keep the shell, text, and interaction native; use an extracted image only for its static decorative layer. ## Steps ### 1. Preserve the reference and inspect the target project If the reference is in a temporary path, copy it to durable work storage before the first tool call. Inspect the existing stack, component library, fonts, icons, images, and design tokens before writing replacements. ### 2. Establish the coordinate system Record the reference image dimensions and the target viewport. Screenshots may be HiDPI: `detect`, `ground`, and `crop` report image pixels, while CSS uses logical pixels. Derive the scale from the actual image and viewport dimensions; do not assume a fixed 2x ratio. ### 3. Inventory the UI outside-in 1. Run one full-image `detect` pass for the initial element list and boxes. 2. Use `detect --region` on dense layout blocks; a full-screen pass is only a scaffold. 3. Use `glance` for hierarchy, component type, visual state, and styling. 4. Use OCR for visible text and `ground` for a specific element that remains ambiguous. Treat model boxes as approximate handles. Use them to organize the page, but do not mistake their final few pixels for measured boundaries. ### 4. Classify before implementing Make a short element list with one of these decisions: reuse existing source, code-native, screenshot-backed, or hybrid. This prevents two costly mistakes: rebuilding complex artwork from scratch and turning functional UI into static image patches. ### 5. Extract screenshot-backed visuals Use a padded crop only to inspect and center the target. Before foreground extraction, tighten the final region to the visual's own ink; adjacent text or rules inside the region can become foreground components too. ```bash crop shot.png --region X1,Y1,X2,Y2 --scale 4 -o work/icon4x.png python3 scripts/extract_fg.py work/icon4x.png # Or extract directly from a known tight region: python3 scripts/extract_fg.py shot.png \ --region X1,Y1,X2,Y2 -o work/icon.png python3 scripts/extract_fg.py shot.png \ --region X1,Y1,X2,Y2 --mode dark -o work/logo.png ``` Inspect the transparent result before use. Confirm that the whole visual is present, no neighboring text or border leaked in, and the alpha edge remains clean. When extracting several assets, inspect them together in a contact sheet instead of checking them from filenames alone. ### 6. Implement the native frame - Follow the project's existing framework and component patterns. - Build content, interaction, responsive layout, borders, shadows, and simple shapes natively. - Use `scripts/dominant_colors.py` for important background, text, and accent values; use `glance` to name a color, not to invent its numeric value. - Place extracted assets with explicit logical dimensions. Preserve their aspect ratio and avoid baking surrounding whitespace into the asset. - Match structure and proportions before tuning small spacing. ## Verify and converge by impact For an existing implementation, start here. 1. Render at the same logical viewport with `scripts/html_shot.py` or the project's browser test setup. For a long-page reference, add `--full-page` rather than increasing `--height`, which would change `vh`-based layout. 2. Compare the render and reference at the same dimensions. Inspect them side by side; use `scripts/pixel_diff.py` to locate differences that are hard to spot or explain. 3. Fix material discrepancies: missing or wrong content, incorrect hierarchy, broken wrapping, visibly wrong alignment or scale, wrong component state, distorted assets, and clearly different primary colors. 4. Re-render after a meaningful fix. Continue while the comparison exposes a user-visible defect; stop when the remaining difference is font rasterization, antialiasing, subpixel placement, or another imperceptible rendering detail. `pixel_diff` is a locator, not the acceptance target. Do not keep iterating only to lower its aggregate percentage, and do not promise zero-difference output from a screenshot without a separately scoped browser, font, viewport, and rendering environment. ## Boundaries - Do not ship a screenshot of the whole page as the implementation. - Do not use screenshot-backed text or controls that must remain selectable, accessible, interactive, localized, or responsive. - Do not stretch an extracted bitmap beyond the resolution supported by the reference. Obtain a source asset or rebuild a vector when it must scale. - Always compare against the original reference, never only against an earlier version of your own render.skills/vision-skills/agents/openai.yaml242 Bab4885f8ecb2upstream ↗
interface: display_name: "Agent Vision Toolkit" short_description: "Ask, OCR long screenshots, locate, inventory, and vectorize images" default_prompt: "Use $vision-skills to inspect this image or run the long-screenshot OCR workflow."skills/vision-skills/work/shot.png694 KB95a6ac2109cdupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
README.mdupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
Plain-text copy, no login: /skill-bundles/anionex-agent-vision-toolkit-skills-vision-skills.md