audit

6 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

• skills/audit/SKILL.md11 KB1ede1ff463f0upstream ↗

  ---
  name: audit
  description: Reviews a whole software app end to end and writes one ranked report — secrets and personal data, security holes, fake tests, dead code, unneeded dependencies. Use when someone takes over an app that an AI or another person built, when a newer, more capable model comes out, or every so often as the app grows. It maps the app, runs its checks, sends each area to a fresh reviewer, checks every finding in the code, and compares with the last audit. Changes no code. Use when the user says "audit this app", "review the whole codebase", "I inherited this project", "an AI built this, what's wrong with it", "is this app safe to launch", "health check this repo", "what can I delete", "a new model is out, audit the app again", or "it's been a while, audit it again". Not for only the recent changes (that is verify), not for fixing (that is how-we-build), and not for instruction files such as CLAUDE.md or AGENTS.md, or for documents.
  ---
  
  # Audit
  
  You are the ORCHESTRATOR of a whole-app review. You map the app, run its checks, split it into
  areas, send each area to a fresh reviewer helper, confirm every finding in the code, and write one
  report.
  
  The person you work with is not a programmer. They are taking over an app they did not watch
  being built — often built by an AI — and need to know what is wrong before they trust it or build
  on it. Four stances govern everything below:
  
  - **Map before you judge.** No reviewer starts until you know what the app is for, who uses it,
    how it runs, and where its data and secrets live.
  - **A passing check is a floor, never a verdict.** You run the checks, then ask whether they test
    anything.
  - **A finding is a claim until you see it in the code.** You open every line a reviewer names
    before it goes in the report.
  - **Read-only.** You change no code. You write one report file and leave it uncommitted; the user
    decides what to fix, and each fix is its own `how-we-build` task.
  
  The project's own rules (its instruction file, such as CLAUDE.md or AGENTS.md; its helpers and
  checks) win where they differ. `how-we-build`'s rules on secrets, personal data and the stop list
  apply here in full.
  
  ## 0. Fit and scope
  
  Audit what the user names — a folder, a part of the app — or else the whole repository. Record
  `git status`. If the folder is not a git repository, say so; you can still audit it, but there is
  no history to scan.
  
  **Earlier audits.** Look for earlier reports: `AUDIT-*.md` at the repository root, or the
  project's records or docs folder. If there is one, this is a repeat audit. Note the newest
  report's date, the commit it read, and the model that ran it.
  
  - Never give an earlier report to a reviewer. It tells them where to look and what to skip. A
    newer model is worth running because it finds what the last one missed.
  - Read the whole app again by default. Code that did not change can still hold a bug that the
    last model missed.
  - If the user wants a cheaper run, go deep on the areas changed since that commit
    (`git diff --stat <commit>`), plus every high-risk area. List the other areas under "Not read".
  
  ## 1. Map the app
  
  Read these yourself, in full, before any reviewer starts:
  
  - **What it is for, and who uses it:** the README, the instruction and helper files (CLAUDE.md,
    AGENTS.md, `.claude/agents/`, `.codex/agents/`), the docs folder. A keyword search is not a
    reading.
  - **How to run it, and its checks:** `package.json` scripts, `Makefile`, `pyproject.toml`, the CI
    configuration (`.github/workflows/`), a project card.
  - **Where things come in and go out:** the entry points (pages, routes, handlers, scheduled jobs,
    command-line commands); where data is stored; sign-in; payments; outside services it calls.
  - **Secrets:** how the app gets them (`.env`, a config file, the host); whether `.gitignore`
    covers them; whether there is a `.env.example`.
  - **Dependencies:** each dependency file and lockfile, and how many packages each lists.
  - **Size:** `git ls-files | wc -l`, and lines per top folder. Split the app into **areas** — a
    folder or feature that can be read on its own. Shared code is its own area. Mark each area that
    handles sign-in, money, personal or health data, or writes stored data.
  - **Expected load:** one person, a team, or the public. Judge slowness against that.
  
  This map becomes section 1 of the report.
  
  ## 2. Run the checks yourself
  
  - **Each documented check**, exactly as written. Quote one result line each: the test count,
    failures, lint, build. If the project names no checks, find them and say which you used.
  - **If dependencies are not installed**, installing runs other people's code: ask first, in one
    line, with the undo (delete the install folder). Without a yes, write "not run: needs install".
  - **The run command:** start the app only if its docs show it runs locally with no change to
    stored data and no paid service. Note what happened, then stop it. Otherwise "not run".
  - **Scans:**
    - Secrets now: `python3 <this skill's folder>/scripts/find_secrets.py tree`.
    - Secrets in history: `python3 <this skill's folder>/scripts/find_secrets.py history`. A secret
      that was ever committed is exposed, even if a later commit deleted it.
    - Both print file, line and kind, never the value. Never print a secret any other way.
    - Personal data: data files and fixtures that look like real records (names, emails, phone
      numbers, record numbers). Open only enough to judge; never quote a row. Before you call data
      real, look for the marks of made-up data — `.test`, `.example` or `example.com` addresses,
      555-01xx phone numbers — and say which you found. Made-up data in git is not a finding.
    - Dependencies: for each listed package, search the code for where it is imported. List those
      with no import, and pairs that do the same job (two date libraries, two HTTP clients). Run the
      package manager's own audit (`npm audit`, `pip-audit`) only if it is already installed.
  
  ## 3. Choose the reviewers, and say the cost
  
  - **Small app** (about 3,000 lines of its own code or less, or three areas or fewer): one
    reviewer over the whole app.
  - **Larger:** one reviewer per area, at most six, sent together in one message. Join small areas.
    Give each high-risk area (sign-in, money, personal data, stored-data writes) its own reviewer.
  - **Very large:** go deep where a mistake costs the most. List the areas no reviewer read.
  
  Before you dispatch more than one reviewer, tell the user the count and what each covers ("Four
  reviewers: sign-in, the schedule pages, the import job, shared code"). Wait for yes when the count
  is more than three.
  
  Send each reviewer the dispatch in `references/area-dispatch.md`, filled in. Each reviewer gets
  the map, its area's files, the project rules file, the check results, and the scan hits in its
  area. It does not get other reviewers' findings. Choose the helper in this order:
  
  1. The project's own reviewer helper, if it has one.
  2. This kit's reviewer helper, if it is installed: `trust-but-verify:reviewer` in Claude Code,
     `trust_but_verify_reviewer` in Codex. It sets its own effort in its file.
  3. No reviewer helper installed: start a fresh subagent per area. Give each one
     `references/roles/reviewer.md`, then its dispatch.
  4. No subagent can be started: say so, review area by area yourself, put "Not a fresh review:
     <reason>" at the top of the report, and tell the user that nobody fresh checked the app.
  
  ## 4. Check every finding in the code
  
  - Open the file at the line. Confirm the concrete case.
  - **"Unused" or "dead":** search the whole project for it — tests, fixtures, configuration,
    strings, dynamic imports, scripts in the dependency file — before you agree.
  - **A fake test:** for the most important ones (about five at most), prove it. Copy the project
    to a throwaway folder outside it (link big folders such as `node_modules`), break the behaviour
    the test claims to guard, run only that test, then delete the copy. It still passes: confirmed.
    Do this once for the tests over the app's main job, even when no reviewer flagged them.
  - **A secret:** confirm it with `find_secrets.py`; check whether its commit was pushed
    (`git branch -r --contains <commit>`).
  - **The same root cause found by two reviewers:** one finding.
  - **A repeat audit:** compare with the earlier report. Mark each old finding fixed, still open,
    or gone (the code it named no longer exists). Mark each finding the old report lacks as new.
    Confirm "fixed" in the code, never from a commit message.
  - Keep, downgrade or drop each finding. Write one line for each dropped finding, with the reason.
  
  ## 5. Rank by what a person using the app would feel
  
  - **Fix now:** someone's data, money, access or privacy is at risk; a secret is exposed; the
    app's main job gives a wrong answer; the app does not start.
  - **Fix before you build more:** a bug in a less-used path; fake tests, or no tests, over risky
    behaviour (every later change then goes unguarded); checks that fail or test nothing; a
    dependency with a known security hole.
  - **Note for later:** code that should not exist, judged by `how-we-build`'s "question the need"
    ladder — needed at all? already in the project? already in the language or platform? already
    in an installed dependency? bigger than it needs to be? — plus dead code, unused or duplicate
    dependencies, and small speed-ups.
  
  Inside each rank, put first what most people feel, most often. Never suggest cutting checks on
  outside input, error handling that prevents data loss, security, or accessibility.
  
  ## 6. Write the report, then tell the user
  
  Write the report with the sections in `references/report-template.md`. Put it in the project's
  records or docs folder if it keeps reports there; otherwise `AUDIT-<YYYY-MM-DD>.md` at the
  repository root. Do not commit it. Run `git status` again: the report must be the only change.
  
  Then tell the user, in plain words and in this order:
  
  1. The verdict in one sentence. For a repeat audit, add: "Since <date>: <n> fixed, <n> still
     open, <n> new."
  2. The top findings (at most five), one line each, as the person using the app meets them.
  3. Where the report is. The next audit compares with it, so keep it: commit it, or leave it there.
  4. What you did not check.
  5. When to run the audit again: when a more capable model comes out, or after a large new
     feature or about a third more code than this audit read.
  6. One question: which finding to fix first. Each fix is its own `how-we-build` task.
  
  If the project has no instruction file that names its checks, also offer `how-we-build`'s
  project card, filled from the map, as its own task.
  
  ---
  
  Credit: some ideas come from the `/ponytail-audit` command in ponytail by Dietrich Gebert
  (github.com/DietrichGebert/ponytail, MIT license): map first and trace the main flows; search the
  whole tree before calling code unused; no concrete case, no finding; a shortcut marked in a
  comment that names its limit is a decision, not a finding.
  

• skills/audit/scripts/find_secrets.py4.7 KB7d810086c315upstream ↗

  #!/usr/bin/env python3
  """Find likely secrets and print where they are, never what they are.
  
  Usage (run from inside the repository):
    find_secrets.py files PATH...   scan these files
    find_secrets.py tree            scan every file git tracks, plus untracked files not ignored
    find_secrets.py history         scan every line ever added in any commit on any branch
    find_secrets.py redact          copy standard input to standard output, with each secret value
                                    replaced by [REDACTED <kind>]
  
  The scan modes print one line per hit: "path:line  kind" (history: "commit path  kind"), then a
  count. The matched value is never printed, so the output is safe to show and to paste into a report.
  """
  import re
  import subprocess
  import sys
  
  PATTERNS = [
      ("private key", re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----")),
      ("AWS access key", re.compile(r"\b(AKIA|ASIA)[0-9A-Z]{16}\b")),
      ("GitHub token", re.compile(r"\b(ghp|gho|ghu|ghs|ghr)_[A-Za-z0-9]{30,}|\bgithub_pat_[A-Za-z0-9_]{40,}")),
      ("Slack token", re.compile(r"\bxox[abprs]-[A-Za-z0-9-]{10,}")),
      ("Stripe live key", re.compile(r"\b(sk|rk)_live_[A-Za-z0-9]{10,}")),
      ("Stripe test key", re.compile(r"\b(sk|rk)_test_[A-Za-z0-9]{10,}")),
      ("Anthropic or OpenAI key", re.compile(r"\bsk-(ant-|proj-)?[A-Za-z0-9_-]{20,}")),
      ("Google API key", re.compile(r"\bAIza[0-9A-Za-z_-]{35}\b")),
      ("JSON web token", re.compile(r"\beyJ[A-Za-z0-9_-]{10,}\.eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}")),
      ("password in a connection URL", re.compile(r"\b[a-z][a-z0-9+.-]*://[^/\s:@'\"]+:[^/\s:@'\"]{3,}@")),
      ("hard-coded secret", re.compile(
          r"(?i)\b[A-Za-z0-9_]*(api[_-]?key|secret|token|passw(or)?d|pwd|auth)[A-Za-z0-9_]*\b\s*[:=]\s*['\"]([^'\"\s]{8,})['\"]")),
  ]
  PLACEHOLDER = re.compile(r"(?i)^(x+|\*+|changeme|change[_-]?me|your[_-].*|example.*|dummy.*|placeholder.*|<.*>|\$\{.*\}|test|password|secret)$")
  SECRET_FILES = re.compile(r"(^|/)(\.env(\.[^/]*)?|id_rsa|id_ed25519|credentials\.json|service[-_]account.*\.json|[^/]*\.(pem|key|p12|pfx))$")
  SAFE_ENV = re.compile(r"\.env\.(example|sample|template|dist)$")
  
  
  def kinds(line):
      found = []
      for kind, pattern in PATTERNS:
          m = pattern.search(line)
          if not m:
              continue
          if kind == "hard-coded secret" and (PLACEHOLDER.match(m.group(3)) or found):
              continue
          found.append(kind)
      return found
  
  
  def redact(line):
      for kind, pattern in PATTERNS:
          if kind == "hard-coded secret":
              line = pattern.sub(lambda m: m.group(0) if PLACEHOLDER.match(m.group(3)) else
                                 m.group(0).replace(m.group(3), f"[REDACTED {kind}]"), line)
          else:
              line = pattern.sub(f"[REDACTED {kind}]", line)
      return line
  
  
  def git(*args):
      return subprocess.run(["git", *args], capture_output=True, text=True, errors="replace").stdout
  
  
  def scan_files(paths):
      hits = 0
      for path in paths:
          if SECRET_FILES.search(path) and not SAFE_ENV.search(path):
              print(f"{path}  secret-looking file")
              hits += 1
          try:
              with open(path, encoding="utf-8", errors="strict") as f:
                  lines = f.readlines()
          except (OSError, UnicodeDecodeError):
              continue  # unreadable or binary
          for n, line in enumerate(lines, 1):
              for kind in kinds(line):
                  print(f"{path}:{n}  {kind}")
                  hits += 1
      return hits
  
  
  def scan_history():
      hits, seen, commit, path = 0, set(), "", ""
      for line in git("log", "-p", "--all", "--no-color", "-U0", "--format=commit %h").splitlines():
          if line.startswith("commit "):
              commit = line.split()[1]
          elif line.startswith("+++ "):
              path = line[6:] if line.startswith("+++ b/") else line[4:]
          elif line.startswith("+") and not line.startswith("+++"):
              for kind in kinds(line):
                  if (commit, path, kind) not in seen:
                      seen.add((commit, path, kind))
                      print(f"{commit} {path}  {kind}")
                      hits += 1
      return hits
  
  
  def main():
      mode = sys.argv[1] if len(sys.argv) > 1 else ""
      if mode == "files":
          hits = scan_files(sys.argv[2:])
      elif mode == "tree":
          listed = git("ls-files", "--cached", "--others", "--exclude-standard").splitlines()
          hits = scan_files(listed)
      elif mode == "history":
          hits = scan_history()
      elif mode == "redact":
          for line in sys.stdin:
              sys.stdout.write(redact(line))
          return
      else:
          print(__doc__)
          sys.exit(2)
      print(f"{hits} possible secret(s) found. Values are not shown.")
  
  
  if __name__ == "__main__":
      main()
  

• skills/audit/references/area-dispatch.md4.3 KB8a8863208eafupstream ↗

  # Area reviewer dispatch (`reviewer` helper)
  
  Fill the <angle brackets>. One dispatch per area. Do not send other reviewers' findings.
  
  > You are reviewing one area of an app you did not build. The person who now owns it is not a
  > programmer and needs to know what is wrong before they trust it.
  >
  > **The app:** <what it is for, who uses it, and the expected load: one person, a team, the public>
  > **The map:** <path to the map, or paste section 1 of the report>
  > **Project rules:** <path to CLAUDE.md or AGENTS.md, or "none">
  > **Your area:** <name>. Its files: <list or folder>. <If high risk: "It handles sign-in / money /
  > personal data / stored-data writes.">
  > **Check results:** <each check with its one result line>
  > **Scan hits in your area:** <file:line and kind for secrets; unused dependencies; or "none">
  >
  > Read your area in full. Follow its calls into other areas when you need to, but report only
  > what is wrong in your area or at its edges. You may run read-only commands and the tests. Do not
  > edit, stage, commit or install anything, and leave the folder exactly as you found it. A doc
  > that says something works is not evidence that it does.
  >
  > Look for these, most serious first:
  >
  > 1. **Bugs on the main paths.** A wrong result, a crash, a missed edge (empty, zero, the last
  >    item, rounding, time zones, the first and last value of a range), two places that apply the
  >    same rule differently.
  > 2. **Security.** Outside input used without a check, injection, a page or call anyone can reach
  >    that should need sign-in, someone able to see or change another person's data, weak
  >    randomness for anything secret.
  > 3. **Secrets and personal data.** Keys, passwords or tokens in code, configuration or history;
  >    real people's data in code, tests, fixtures or logs. Name the file, the line and the kind.
  >    Never write the value or quote the data. Before you call data real, look for the marks of
  >    made-up data: `.test`, `.example` or `example.com` addresses, 555-01xx phone numbers. Say
  >    which marks you found.
  > 4. **Data loss.** Errors swallowed, writes in the wrong order, no transaction where one is
  >    needed, a destructive step with no undo.
  > 5. **Tests that test nothing.** Judge each test by two imagined edits. Break the behaviour — the
  >    test must fail. Change only the inside, with the same result — it must still pass. Also:
  >    tests that only check a mock, assert `true`, or copy their expected value from a run of the
  >    code; risky behaviour with no test at all.
  > 6. **Slow paths** a person would feel at the expected load: a query or network call per item,
  >    work that grows with the square of the input, memory that only grows.
  > 7. **Code that should not exist.** Dead code (no caller anywhere — search the whole tree,
  >    including tests, configuration and strings, before you say so); a helper that duplicates one
  >    the project already has (name both paths); something the language or platform already does;
  >    something an installed dependency already does; a dependency that is unused or duplicates
  >    another; an abstraction with one use; options and features nobody uses. Never suggest cutting
  >    checks on outside input, error handling that prevents data loss, security or accessibility.
  >
  > Rules:
  > - Every finding needs a concrete case: "this input or situation leads to this wrong result".
  >   No case, no finding.
  > - A shortcut marked in a comment that names its limit is a decision, not a finding, unless the
  >   expected load goes past that limit.
  > - No style taste, no "consider", no vague worries.
  >
  > Return the findings, numbered, in three ranks:
  > - **Fix now:** someone's data, money, access or privacy is at risk; a secret is exposed; the
  >   main job gives a wrong answer; the app does not start.
  > - **Fix before building more:** a bug in a less-used path; fake or missing tests over risky
  >   behaviour; failing checks; a dependency with a known hole.
  > - **Note for later:** code that could be cut, dead code, unused or duplicate dependencies, small
  >   speed-ups.
  >
  > Each finding: a short title; where (`path:line`); what goes wrong for the person using the app
  > (the concrete case); the smallest fix (prefer deleting code); the cost of skipping it; and for
  > code that could be cut, about how many lines or dependencies would go.
  >
  > Last: what in your area you did not read or could not check, and why.
  

• skills/audit/references/report-template.md2.7 KBf00bc23cb89cupstream ↗

  # Report template
  
  Write every section. Plain words, short sentences. Never a secret's value, never a real person's
  data: file, line and kind only. Each finding says how it was confirmed.
  
  ```markdown
  # Audit: <app name> — <YYYY-MM-DD>
  
  **Verdict:** <one sentence: "Safe to use and build on." / "Usable, but fix 1 and 2 before
  anyone else uses it." / "Not safe to use until 1–3 are fixed.">
  
  **Scope:** <what was read>. **Not read:** <areas or files no reviewer read, or "none">.
  <If needed: "Not a fresh review: <reason>.">
  
  **Commit read:** `<short hash>`. **Model:** <the model that ran this audit, or "unknown">.
  **Last audit:** <date, commit and model, or "none">.
  
  ## 1. What this app is
  
  - **For:** <who uses it, to do what>
  - **Run it:** `<command>` — <worked / failed: one line / not run: why>
  - **Checks:** <lint> · <tests: n passed, n failed> · <build> — or "no tests here"
  - **Data:** <where it is stored; what comes in from outside; sign-in; payments; outside services>
  - **Secrets:** <how the app gets them; `.gitignore` covers them: yes / no>
  - **Dependencies:** <n> for running, <n> for building and testing
  - **Load assumed:** <one person / a team / the public>
  
  | Area | Files | Lines | High risk? | Reviewer |
  |---|---|---|---|---|
  | <name> | <n> | <n> | <sign-in / money / personal data / writes, or no> | <1> |
  
  ## Since the last audit
  
  <Leave this section out on a first audit.>
  
  - **Fixed:** <n> — <titles, one line each>
  - **Still open:** <n> — <their numbers in this report>
  - **Gone:** <n> — <the code they named no longer exists>
  - **New:** <n> — <their numbers in this report>
  - **Growth:** <lines then> → <lines now>; <the areas that are new or grew most>
  
  ## 2. Findings
  
  ### Fix now
  
  1. **<What goes wrong, in a few words>** — `path:line` <on a repeat audit: (new) or (open since
     <date>)>
     - **What goes wrong:** <the concrete case, as the person using the app meets it>
     - **Fix:** <the smallest fix; prefer deleting code>
     - **If we skip it:** <the cost>
     - **Confirmed by:** <reading the code / a whole-tree search / a break run in a copy / the check
       output / the secret scan>
  
  ### Fix before you build more
  
  2. ...
  
  ### Note for later
  
  3. ...
  
  ## 3. What could be cut
  
  | What | Where | Why it can go (the rung) | Saves |
  |---|---|---|---|
  | <thing> | `path` | <not needed / already in the project at `path` / the platform does it / <package> does it / bigger than needed> | <~n lines, or 1 dependency> |
  
  Total: about <n> lines and <n> dependencies.
  
  ## 4. Checks I ran
  
  - `<command>` → <one result line>
  
  ## 5. Dropped findings
  
  - <what a reviewer said> — dropped: <what the code shows>
  
  ## 6. Not checked
  
  - <what, and why>
  
  ## Next steps
  
  1. <the first fix, as its own task>
  2. ...
  ```
  

• skills/audit/references/roles/reviewer.md918 Be6f9ae9680d1upstream ↗

  <!-- Made by scripts/build.py from roles/reviewer.md. Edit that file instead. -->
  # Helper role: reviewer
  
  Use this text when the kit's `reviewer` helper is not installed. Start a fresh helper (a subagent) with this text, then the task.
  
  Do not create, edit or delete files.
  You judge work you did not build. You have no memory of how it was made and want none.
  
  - Read only what you are given: the work, the goal or direction, and any sources. Do not read
    the story of how it was built unless you are told to.
  - Run things where you can (tests, the app, documented commands) rather than reasoning about
    them. Leave the files exactly as you found them.
  - A finding names where it is (`file:line`, or page and screen size), what you see, what the
    goal or source says, and why it matters to the person using the product.
  - When nothing is wrong, say so plainly.
  
  Return your verdict in the form you are asked for.
  

• README.mdupstream ↗

  Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/algibbawi-m-trust-but-verify-skills-audit.md