⏳ This skill is pending AI review.
Scores will appear once the review pipeline completes.
verify-before-claiming
>
Choose how to use this skill
You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.
1. Native installer
This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.
Do not guess an installer command or replace an existing version without reviewing the diff.
2. Complete package recommended
Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.
No complete ProSkills package is published for this listing yet.3. Prompt-only
Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.
Need only the instruction file?
Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.
No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.
// RATINGS
// README
Install
mkdir -p .claude/skills/verify-before-claiming && curl -sL https://raw.githubusercontent.com/LeonardLeroy/i-dont-believe-you/main/skills/verify-before-claiming/SKILL.md -o .claude/skills/verify-before-claiming/SKILL.md
Codex, Cursor, opencode and others: INSTALL.md.
What it does
Before your agent can tell you anything passed, it has to run eight shell commands against the real diff and paste the output. If one prints a line, it is not allowed to claim success.
No dependencies, no API key, no model. One markdown file.
Pin the base before the session: export IDBY_BASE=$(git rev-parse HEAD). Without it the
commands compare against HEAD, and a single git commit by the agent moves HEAD onto its
own change, which silences all eight.
What changes
Same task, same agent. On the left, what it tells you today. On the right, what it has to tell you once the skill is installed.
| Your agent today | Your agent with the skill |
|---|---|
| Fixed the retry logic.✅ All tests pass. Ready to merge. | Fixed the retry logic.I ran check 1 and it printed:+it.skip('resets the backoff', ...So I skipped a test instead of fixing it. The backoff still does not reset. Not ready. |
The eight checks
| # | catches |
|---|---|
| 1 | a test disabled with .skip, @pytest.mark.skip, #[ignore], @Disabled… |
| 2 | an assertion that cannot fail, like expect(true).toBe(true) |
| 3 | more assertions removed than added |
| 4 | a test file deleted, or renamed out of the runner's glob |
| 5 | an error swallowed by an empty catch or except: pass |
| 6 | "I added tests" when no test file changed |
| 7 | a named test that existed before and does not exist now |
| 8 | a test focused with .only, which stops every other test in the file running |
When an agent writes more code than anyone reviews, the test suite becomes the only oversight left, and optimising for it instead of the goal is the predictable failure. Across 86,156 agent-authored test patches, 80.2% carried weak or no assertions at all. Why these seven: docs/why.md.
Tune it
The commands in SKILL.md are plain git and grep.
Adding a framework is one regex.
There is also an optional CLI in src/ that runs the same checks as a CI gate, for when
you would rather not depend on the agent cooperating.
Credits
The problem these checks exist for is measured in All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code by Dipayan Banik, Kowshik Chowdhury and Shazibul Islam Shamim, and in METR's work on reward hacking. The checks themselves are not from those papers. They are eight things a diff can prove, cheap enough to run every turn.
License
MIT. Contributions welcome: CONTRIBUTING.md.
Star ⭐ if it saved you one merge past one "✅ All tests pass."
// HOW IT'S BUILT
// FILES
2 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.
.agents/skills/verify-before-claiming/SKILL.md7.8 KBupstream ↗
--- name: verify-before-claiming description: > Use for any task that will end in a claim about tests, the build, or whether the work is done: making tests pass, fixing a failing test, fixing a bug, implementing a feature, refactoring, debugging, getting CI green. Runs eight shell commands against the real diff and requires their raw output before success can be reported. --- # Verify before claiming You are about to tell the user something worked. You do not know that yet. Find out first. ## The rule Before any sentence containing "tests pass", "build is green", "done", "fixed", "ready" or anything equivalent, run every check below and paste the raw output into your reply. If a check prints a line, you have found something. Do not report success. Read the line, open the file, and either fix the code or rewrite your summary so it states what is actually true. ## Pin the base first Every command below compares the working tree against `${IDBY_BASE:-HEAD}`. `HEAD` is a valid base only while nobody has committed since the work started. One `git commit` moves `HEAD` onto the change itself, and all eight checks fall silent on a change they would otherwise have caught. Committing work is normal, so this happens without any intent to hide anything. Pin a base the agent cannot move, before the work starts: `export IDBY_BASE=$(git rev-parse HEAD)`. In CI, pin the pull request's merge-base instead: `export IDBY_BASE=$(git merge-base origin/main HEAD)`. A pinned SHA stays reachable after the base commit is amended or rebased. If `IDBY_BASE` is unset, the checks only cover what is still uncommitted. Say that in your report instead of letting a silent check stand for more than it proves. ## The checks Run all eight. They take under a second. `git diff "${IDBY_BASE:-HEAD}" -U0` shows only changed lines, so a `+` prefix means the line is new in this change. ### 1. Did you disable a test? <!-- check: disabled-test --> ```bash git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^\+' | grep -E '\.skip\(|\.todo\(|\bxit\(|\bxdescribe\(|@pytest\.mark\.(skip|xfail)|unittest\.skip|#\[ignore\]|\bt\.Skip\(|@Disabled|@Ignore|markTestSkipped' ``` Any output means a test no longer runs. Saying the suite passes while this prints a line is a false statement, not a judgement call. ### 2. Did you add an assertion that cannot fail? <!-- check: hollow-assertion --> ```bash git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^\+' | grep -E 'expect\((true|1)\)\.(toBe|toEqual)\((true|1)\)|assert\s+True\s*$|assertTrue\(True\)|assert!\(true\)|Assert\.True\(true\)|assertThat\(true\)' ``` Any output means a test now passes regardless of the code it claims to cover. ### 3. Did the assertion count drop? <!-- check: assertions-dropped --> ```bash R=$(git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^-' | grep -cE 'expect\(|assert|should\.|\.Errorf?\(|\.Fatalf?\('); A=$(git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^\+' | grep -cE 'expect\(|assert|should\.|\.Errorf?\(|\.Fatalf?\('); [ "$R" -gt "$A" ] && echo "assertions: $R removed, $A added" ``` Any output means the suite checks less than it did before. Say so, or put the assertions back. ### 4. Did a test file disappear? <!-- check: test-file-gone --> ```bash git diff "${IDBY_BASE:-HEAD}" --diff-filter=DR --name-status | grep -Ei '(^|[/[:space:]])(tests?|spec)s?/|[._-](test|spec)\.' ``` Any output means a test file was deleted or renamed. A rename out of the runner's glob removes the tests just as completely as a delete. ### 5. Did you swallow an error? <!-- check: swallowed-error --> ```bash git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^\+' | grep -E 'catch[^{]*\{\s*\}|except[^:]*:\s*pass\s*$|\.catch\(\s*\(\)\s*=>\s*\{\s*\}\s*\)|if err != nil \{\s*\}'; git diff "${IDBY_BASE:-HEAD}" -U0 | grep -A1 -E '^\+.*(catch[^{]*\{|except[^:]*:|if err != nil \{)[[:space:]]*$' | grep -E '^\+[[:space:]]*(\}|pass|\.\.\.)[[:space:]]*$' ``` Any output means a failure path now silently does nothing. That is not "handled". The second command covers the form most formatters produce, where the handler opens on one line and closes on the next. It reads one line of context, so it sees the empty body only when both lines are part of the same change. ### 6. Did you claim tests you did not write? <!-- check: no-test-touched fires-on=NO TEST FILE CHANGED --> ```bash git diff "${IDBY_BASE:-HEAD}" --name-only | grep -Ei '(^|/)(tests?|spec)s?/|[._-](test|spec)\.' || echo "NO TEST FILE CHANGED" ``` If this prints `NO TEST FILE CHANGED`, do not say you added tests, improved coverage, or covered anything. ### 7. Did a named test disappear? <!-- check: test-name-gone --> ```bash comm -23 <(git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^-' | grep -oE "(test|it|describe)(\.\w+)?\(\s*['\"][^'\"]*" | sed -E "s/^.*['\"]//" | sort -u) <(git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^\+' | grep -oE "(test|it|describe)(\.\w+)?\(\s*['\"][^'\"]*" | sed -E "s/^.*['\"]//" | sort -u) ``` Any output is the name of a test that existed before this change and does not exist now. It was either deleted or renamed, and a diff cannot tell those apart. Name which one it was. Replacing a failing test with a different passing one keeps the test count and the assertion count identical, so checks 3 and 4 stay silent while the suite gets weaker. ### 8. Did you focus a test? <!-- check: focused-test --> ```bash git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^\+' | grep -E '\b(it|test|describe)\s*\.\s*only\b|\bf(it|describe)\s*\(' ``` Any output means one test was focused and every other test in that file stopped running. Nothing was skipped and nothing was deleted, so checks 1 and 4 stay silent, but the suite you are reporting on is one file's worth of one test. Call it focusing, not skipping, and say how many tests it silenced. ## Run all checks at once The eight commands above, run in sequence and printed as one status line each. Same commands, verbatim, nothing new. ```bash print_result() { id="$1"; out="$2"; err="$3" if [ -n "$err" ]; then echo "$id: unable-to-verify" printf '%s\n' "$err" | sed 's/^/ /' elif [ -n "$out" ]; then echo "$id: changed" printf '%s\n' "$out" | sed 's/^/ /' else echo "$id: verified" fi } BASE_ERR=$(git rev-parse "${IDBY_BASE:-HEAD}" 2>&1 >/dev/null) if [ -n "$BASE_ERR" ]; then for id in disabled-test hollow-assertion assertions-dropped test-file-gone swallowed-error no-test-touched test-name-gone focused-test; do echo "$id: unable-to-verify" echo " bad base ($BASE_ERR)" done else ERRF=$(mktemp) OUT=$(git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^\+' | grep -E '\.skip\(|\.todo\(|\bxit\(|\bxdescribe\(|@pytest\.mark\.(skip|xfail)|unittest\.skip|#\[ignore\]|\bt\.Skip\(|@Disabled|@Ignore|markTestSkipped' 2>"$ERRF") print_result "disabled-test" "$OUT" "$(cat "$ERRF")"; rm -f "$ERRF" ERRF=$(mktemp) OUT=$(git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^\+' | grep -E 'expect\((true|1)\)\.(toBe|toEqual)\((true|1)\)|assert\s+True\s*$|assertTrue\(True\)|assert!\(true\)|Assert\.True\(true\)|assertThat\(true\)' 2>"$ERRF") print_result "hollow-assertion" "$OUT" "$(cat "$ERRF")"; rm -f "$ERRF" ERRF=$(mktemp) OUT=$({ R=$(git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^-' | grep -cE 'expect\(|assert|should\.|\.Errorf?\(|\.Fatalf?\('); A=$(git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^\+' | grep -cE 'expect\(|assert|should\.|\.Errorf?\(|\.Fatalf?\('); [ "$R" -gt "$A" ] && echo "assertions: $R removed, $A added"; } 2>"$ERRF") print_result "assertions-dropped" "$OUT" "$(cat "$ERRF")"; rm -f "$ERRF" ERRF=$(mktemp) OUT=$(git diff "${IDBY_BASE:-HEAD}" --diff-filter=DR --name-status | grep -Ei '(^|[/[:space:]])(tests?|spec)s?/|[._-](test|spec)\.' 2>"$ERRF") print_result "test-file-gone" "$OUT" "$(cat "$ERRF")"; rm -f "$ERRF" ERRF=$(mktemp) OUT=$({ git diff "${IDBY_BASE:-HEAD}" -U0 | grep -E '^\+' | grep -E 'catch[^{]*\{\s*\}|except[^:]*:\s*pass\s*$|\.catch\(\s*\(\)\s*=>\s*\{\s*\}\s*\)|if errREADME.mdupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
Plain-text copy, no login: /skill-bundles/leonardleroy-i-dont-believe-you-agents-skills-verify-before-claiming.md