fp-check

VulnVerify Skills

VulnVerify Skills helps AI agents verify findings in website and web application vulnerability reports from penetration testing and bug hunting. Reports may come from agentic AI workflows, manual testing, or a combination of both. VulnVerify provides a verification pass to determine whether each reported finding is supported by evidence, is a false positive, or needs more information, and to prepare minimal proofs of concept in authorized systems.

It supports OpenCode, OpenAI Codex, and Claude Code through their native skill mechanisms, including their existing VS Code integrations. The project version is recorded in VERSION.

The project adapts selected Trail of Bits skills with source-optional evidence handling and three explicit verdicts. It is a set of agent instructions, not a scanner or an automatic guarantee of correctness. Review tested compatibility and limitations before relying on an assessment.

What VulnVerify adds

VulnVerify combines the adapted Trail of Bits methodology with a portable report-to-verification workflow:

• Direct browser verification through Playwright MCP. When browser testing is relevant and authorized, the agent can navigate the permitted application, interact with its actual forms and pages, deliver the reported trigger, and compare the observed result with suitable controls. Screenshots, page snapshots, and relevant console/network output can support the verdict.
• A connected evidence workflow. Report intake, seven-brocard triage, standard/deep technical analysis, PoC reproduction, and six-gate review carry the same claim and evidence through to the final decision.
• Report-driven verification. Start with existing pentest or bug-hunting findings in PDF, JSON, or Markdown, subject to the host's reading capabilities. Correlate supplied requests, responses, screenshots, source excerpts, configuration, and runtime evidence to assess the exact claimed behavior.
• Three explicit outcomes. Confirmed, False Positive, and Inconclusive keep missing evidence or an unavailable environment from becoming an unsupported dismissal. The assessment separates reported behavior, actual observations, and unverified internal causes.
• One methodology across three clients. The same canonical skill files serve OpenCode, Codex, and Claude Code, with client-specific setup kept in the integration layer.

For example, a report may claim that an input executes JavaScript in a web page. With an available Playwright MCP connection and explicit target/action scope, the agent can submit the reported input through the real form, observe its effect, and compare benign input or another appropriate control. The verdict must explain whether that exact delivery path and claimed security impact were established. Browser execution remains conditional on access, setup, and authorization; an unavailable or unsuccessful attempt alone does not prove False Positive. Findings that do not need a browser use suitable non-browser evidence or reproduction instead.

Three skills, one workflow

The three skills share one evidence record. Each has a specific responsibility; reproduction results return to technical review before a final verdict is issued.

1. Intake and triage: vulnerability-triage-brocards

The agent extracts the affected website/component, reported version, attacker capabilities, account roles, resource, trigger, expected security rule, and claimed impact. It keeps source locators for each assertion and reconciles conflicting or duplicate report exports without discarding their original IDs.

Seven brocards examine the threat model, gained capability, actual usage, standards, documented behavior, remediation proportionality, and the technical merits independent of a reporter's label or severity score. The result is ACCEPT, DISMISS, or NEEDS-MORE-INFO, with reasons and missing evidence. A triage dismissal does not automatically establish a technical False Positive.

2. Technical analysis: fp-check

The agent checks whether attacker-controlled input can reach the reported operation and violate the stated security boundary. Available source can establish callers, validation, transformations, and the sink. Requests, responses, account roles, session state, and policy help establish the observable application path. The agent correlates this evidence while keeping uninspected implementation details unknown.

Standard verification handles clear, straightforward claims without an async or concurrency trigger. Deep verification handles ambiguity, cross-component paths, browser callbacks, concurrency, unresolved historical versions, or an escalation from standard review. Both retain the required adversarial questions, false-positive checklist, controls, and evidence review.

The six gates are Process, Reachability, Real Impact, PoC Validation, Math Bounds, and Environment. Each receives PASS, REFUTED, UNKNOWN, or a justified N/A where allowed. Gate rules and applicability are defined in gate-reviews.md.

3. Controlled reproduction: poc-builder

When the evidence warrants reproduction, fp-check passes the exact claim, version, roles/resource, authorization, prerequisites, expected observable effect, and planned controls to poc-builder. The agent records a pseudocode/data-flow sketch, then chooses an appropriate executable or unit-test PoC if feasible. Python, HTML/JavaScript, or the target's language may be suitable; every finding does not require every artifact type.

Browser findings use Playwright MCP to exercise the real permitted page/form and capture relevant observations. Non-browser findings use an appropriate request, code, or existing test interface. Positive setup and negative/benign controls establish whether the test exercised the intended resource and whether the claimed effect differs from ordinary behavior. Any version, role, resource, or handler mismatch remains explicit.

The return record distinguishes planned, created, executed, and observed results. If access, authorization, or prerequisites are missing, it records a blocked/skipped reproduction and the reason. A created file or unsuccessful test alone is not proof of a vulnerability or a false positive.

4. Evidence review and final verdict: back to fp-check

fp-check compares the PoC return with the original claim, reviews the controls and all six gates, and checks that required work was actually completed. It issues one technical verdict per exact finding: