sentry-visual-qa

sentry-visual-qa

An agent skill — a SKILL.md with its scripts and references — that makes a coding agent verify user-visible Sentry UI changes in a real browser, and report only what the captured screenshots, videos, and DOM checks actually support.

The problem it solves: agents are happy to call a CSS change "correct" from the diff alone. This skill makes the agent open the rendered surface, gate on the page having really loaded, capture the smallest evidence set that answers the question, and say plainly which surfaces it did not check.

What it does

• Runs a dedicated UI-only devserver (pnpm run dev-ui) on its own port, so a QA run never collides with the devserver you are working in.
• Drives a dedicated headless browser session with its own Sentry login — never your personal browser profile.
• Gates every capture behind a readiness check (READY / LOGIN_REQUIRED / UNREACHABLE / REFUSED / BUILD_BROKEN), so a login page or a loading spinner can't be reported as evidence.
• Picks evidence by request type: screenshots for stable states, short video for timing and motion, DOM checks as support.
• Captures before/after pairs by switching the checkout to the merge-base on the same server.
• Covers the component's scraps story (Sentry's component-story system) alongside the in-app surface when one exists.

Requirements

• A local getsentry/sentry checkout (the dev-ui server is started from it)
• A Sentry account to run QA as — a dedicated one, see Authentication
• The agent-browser CLI on PATH — brew install agent-browser (developed against 0.37.1); on a remote workspace install from npm instead, see Signing in from a remote workspace
• python3 — used only to mirror auth cookies onto the dev origin
• curl, bash
• ffmpeg, for video capture only

Install

The repository root is the skill, so clone it straight into your agent's skills directory and git pull updates the skill in place:

git clone https://github.com/getsentry/sentry-visual-qa.git ~/.agents/skills/sentry-visual-qa

If your agent reads skills from somewhere else, or you keep your checkouts elsewhere, clone once and link it in under the same directory name:

ln -s ~/.agents/skills/sentry-visual-qa <your-agent-skills-dir>/sentry-visual-qa

Authentication

QA runs as a real Sentry account in a dedicated browser session, never your everyday browser profile. Session cookies live in ~/.agent-browser/sessions/, and scripts/sqa passes the session flags on every call — invoking agent-browser directly drops them and silently captures a logged-out page.

Use a dedicated account, not your own

Whatever the account can see, the agent can capture. The dev-ui server proxies every API call to production sentry.io, so a run renders live production data — issues, events, member names — for every organization that account belongs to.

Do not sign in with an account that has production or customer access. Create a separate Sentry user whose only membership is an organization with nothing sensitive in it, and use it for QA and nothing else. sentry-sdks is the default target here for exactly that reason: it holds SDK projects rather than customer data.

scripts/sqa does refuse URLs pointing at Sentry's own sentry org (exit 4, configurable via SENTRY_QA_FORBIDDEN_ORGS) — but treat that as a backstop, not the boundary. A blocklist only stops the targets someone thought to name, while an account with no access to sensitive data cannot render it whatever URL it is handed.

Confirm which account is live before trusting any capture:

scripts/sqa whoami
# {"authed":true,"user":"[email protected]","org":"my-org","devUi":true,...}

Why signing in takes two steps

The dev-ui server proxies /api to sentry.io server-side, so the browser needs Sentry session cookies on the dev origin — and cookies set on .sentry.io are never sent to *.dev.getsentry.net. SSO cannot complete through the dev origin either, so signing in at the dev-ui /auth/login/ does not work. Sign in upstream, then mirror the cookies across:

scripts/sqa login   # headed sentry.io sign-in; a human completes SSO
scripts/sqa sync    # mirror the session cookies onto .dev.getsentry.net
scripts/sqa ready   # expect READY

sync copies session, sentry-sc, and sentry_react_auth. sentry-sudo is skipped unless you pass --with-sudo — rendering does not need elevated access. Cookie values are never printed, logged, or written to disk by these scripts.

Re-run sqa sync whenever the sentry.io session refreshes; the mirrored copies do not update themselves. references/session-and-login.md covers account switching and auth troubleshooting.

Signing in from a remote workspace (coder.dev)

sqa login opens a headed browser, and a remote workspace has no display to open it on. On a coder.dev workspace it fails with Missing X server or $DISPLAY; agent-browser's automatic Xvfb fallback only helps if Xvfb is installed, and a framebuffer nobody can see does not get you through SSO anyway. Drive the sign-in through the agent-browser dashboard instead — it streams the headless browser and forwards clicks and keystrokes.

Homebrew's agent-browser has no dashboard. The homebrew-core formula builds without the dashboard frontend, so dashboard start serves a Dashboard not built stub (vercel-labs/agent-browser#1412). Install from npm for this flow.

1. Let Chrome start. Ubuntu 24.04 ships kernel.apparmor_restrict_unprivileged_userns=1, which blocks Chrome's namespace sandbox. Every agent-browser command then dies before a browser exists, the daemon never creates its socket, and the CLI reports Failed to connect: No such file or directory (os error 2) — or, from agent-browser doctor, No usable sandbox!. Export the launch flag for the whole session:

export AGENT_BROWSER_ARGS="--no-sandbox"

2. Open the sign-in page headless, bypassing the --headed that sqa login forces:

agent-browser --session sentry-qa --restore sentry-qa open https://sentry.io/auth/login/

3. Start the dashboard for your proxied origin. Coder serves each port at the origin in $VSCODE_PROXY_URI. The dashboard refuses any non-loopback origin it was not told about, and it wants an access token on top of that — an allowed origin alone still returns Origin, Referer, or dashboard access token is invalid.

agent-browser dashboard start \
  --allowed-origins "https://4848--main--<workspace>--<user>.coder.sentry.dev"

Open the URL it prints including the #dashboard-access-token=... fragment. That fragment is the only thing that sets the auth cookie, and the page strips it from the address bar once read, so a bookmarked copy will not authenticate a fresh browser. The token is regenerated every time the dashboard restarts.

Port-forwarding sidesteps the token entirely, because loopback is trusted by default:

coder port-forward <workspace> --tcp 4848:4848   # from your laptop, then http://localhost:4848

4. Sign in through the Viewport tab, then mirror the cookies as usual:

scripts/sqa sync
scripts/sqa ready    # expect READY

Typing in the dashboard viewport

Clicking and navigation work. Text entry has two gaps, so fill fields from the CLI instead:

• . arrives as Delete. The viewport sends key.charCodeAt(0) as the virtual key code, and ".".charCodeAt(0) is 46 — VK_DELETE. A period is not merely dropped: it deletes whatever is to the right of the cursor, so typing one mid-string silently eats a character.
• There is no clipboard bridge. The viewport forwards keyboard and mouse events only, so Ctrl+V pastes