⏳ This skill is pending AI review.

Scores will appear once the review pipeline completes.

v0.3.9

qikly

gal-a@gal-a⭐ 19 stars· last commit 8h ago· 0 open issues

Write tests that can actually fail, by withholding the acceptance criteria from the agent that writes the code. Use when someone does not trust a suite that passes. Use when they want tests written from a specification rather than from the code. Use when they ask whether a specification is testable, or want an existing suite scored by planting faults in the code. Python modules, through the qikly tool. Also use whenever qikly or spec-driven testing is mentioned.

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

—/10

// RATINGS

⭐GitHub Stars
⭐⭐ 19 on GitHubGitHub ↗

Growing

🟢ProSkills Score
—
📍

Not yet listed on ClawHub or SkillsMP

// README

qikly

Agent Skill 2,100+ tests pypi python license marketplace Claude Code VS Code

qikly: one spec in, code and tests out, written by a coding agent and a test agent that are kept apart

The problem: Your AI writes both the code and its tests. How do you know the tests are really valid?

One logged run. The test agent builds the test suite: it was told, from the spec's acceptance criteria, that a rate may not exceed 100. The coding agent never saw the acceptance criteria and discovered that rate limit from a failing test: it then added or rate > 100 by itself, so the 9 of 9 is code eventually passing the test suite after applying fixes in an automated repair loop.

qikly --score on the bundled pricing example: 10 of 13 planted faults missed, and three of the questions they leave open

Try it now, free and with no API key: pip install --upgrade qikly, then qikly --score my_module.py from your project's root folder. It plants faults in a copy of your code, one at a time, and turns each one your tests miss into a question about behaviour nothing checks. Above, a real run on the bundled pricing example.

Run on 50 young Python libraries, it surfaced 4 real problems as well as the test gaps, each confirmed by hand and reported upstream. 2 are fixed so far: cyclopts #1044, fixed in #1047, and prompt-template #48, fixed in v1.2.1. Fixes for markdown-code-runner and xmltodict are in review.

Why use qikly?

A student who writes the exam paper, writes the answer key and then sits the exam will pass. That is what happens when one model is given the acceptance criteria and asked to produce both the implementation and the suite that checks it. Everything goes green, and the green can mean nothing. qikly takes the answer key away from the student: the coding agent never sees the acceptance criteria.

Start here

Before you start, three things:

  1. Install qikly: pip install --upgrade qikly.

  2. Set one API key, for steps 2 and 3: Google Gemini (free tier, no card), Anthropic or OpenAI. In the terminal you will run qikly from:

    export GEMINI_API_KEY="..."          # macOS and Linux
    $env:GEMINI_API_KEY = "..."          # Windows PowerShell
    

    Getting a key, and the other two providers. Step 1 needs no key; without one, step 2 leaves every answer for you to write.

  3. Check your tests pass on their own, from your project's root folder: python -m pytest. qikly runs them the same way, so a missing package or the wrong Python version stops step 1 too.

Then run these from that root folder. Each takes one module or a whole folder, and the same path in every step:

qikly --score my_module.py       # 1. what your tests miss         free, no key
qikly --scaffold my_module.py    # 2. the drafted spec, to check   a fraction of a cent
qikly --tasks my_module.py       # 3. the run                      a cent or so
  1. --score scores the tests you have. It plants faults in a copy of your code, one at a time, and names the ones your tests did not notice, each as a question about behaviour nothing checks. It finds your tests itself: the test_*.py and *_test.py files that import your code, searched from the project root. To name them instead, add --score-tests tests/. No model is called and nothing of yours is changed.

  2. --scaffold drafts the spec, inputs_private/config/tasks/MY_MODULE_VERIFY.yaml, with everything it can read from your code filled in, and a data file of the values your tests already call it with. With a key set (Google Gemini, Anthropic or OpenAI), a spec agent then drafts the parts only an owner could write: a line on what the module is for, and an answer to each of step 1's questions as an acceptance criterion. It reads your README and docstrings, never your code, so it says what the code should do rather than what it does, and a test written from it can disagree with the code:

    #? What should apply_discount(100, 100) do?
    #? Answer below drafted by qikly from your README or docstrings: "...so a rate
    #? of 100 or more is rejected with ValueError". Keep it, or change it.
    - "apply_discount(100, 100) raises ValueError"
    

    Each answer says where it came from: a quote from your documentation, checked word for word, or CHECK where the agent judged it from the

// HOW IT'S BUILT

// FILES

2 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • .claude/skills/qikly/SKILL.md29 KB14a3d8f115e8upstream ↗
    ---
    name: qikly
    description: Write tests that can actually fail, by withholding the acceptance criteria from the agent that writes the code. Use when someone does not trust a suite that passes. Use when they want tests written from a specification rather than from the code. Use when they ask whether a specification is testable, or want an existing suite scored by planting faults in the code. Python modules, through the qikly tool. Also use whenever qikly or spec-driven testing is mentioned.
    license: Apache-2.0
    metadata:
      author: Gal Arav
      homepage: https://github.com/gal-a/qikly
      version: 0.3.9
      requires: qikly >= 0.6.1
    ---
    
    # qikly: tests written from a spec the coder never read
    
    > **If you read one thing here, read this.** Asked for tests for a module with
    > no specification: do not write tests, and do not run the module to find out
    > what it should do. A value obtained by executing an implementation is not a
    > contract. A suite built from those values passes by construction and cannot
    > disagree with a bug, because the bug is what it was written from. Say that in
    > one sentence, then offer the two routes in *The request this arrives as*.
    >
    > **Four things look settled and are not:** whether a threshold is inclusive,
    > whether a discount applies before or after tax, where a floor or cap lands,
    > which way a half-cent rounds. An implementation answers all four. None of
    > those answers is a specification. Ask the user. Do not read them off the code.
    >
    > **If they insist on tests anyway, write them.** Refusing a second time is
    > not your call. Say which lines of the implementation you had to treat as
    > correct, name every boundary and tie-break you copied rather than were
    > told, and the suite is at least honest about its foundation.
    
    ## What this is for
    
    If you write a module and then write its tests, both come from one reading of
    the same ambiguous sentences. The
    suite goes green and the green can mean nothing.
    
    **Python modules only.** qikly scaffolds from Python signatures and generates
    pytest suites; this Skill says nothing about any other language, and you should
    say so rather than guess if asked.
    
    ## The request this arrives as, and what to do with it
    
    **"Write tests for `my_module.py`", with no specification anywhere.** That is
    how this almost always begins, and answering it literally is the mistake this
    whole tool exists to prevent. Tests read off an implementation can only
    describe what it already does: they pass by construction, they encode every
    choice the code happened to make as though it were intended, and they cannot
    disagree with a bug because the bug is what they were written from.
    
    So do not open by writing tests. Say that in one sentence, then offer the
    things that do work, and let the user choose:
    
    The path is three commands, each taking the same module or folder:
    
    - **`qikly --score my_module.py`**, the best first move when any suite
      exists (before 0.6.0 the flag is `--score-code`). Free, no model call,
      nothing of theirs modified. It finds the tests that import the module, or
      take `--score-tests their_tests.py`, and answers "is the suite we have
      worth anything" with named faults it missed, each as a question.
    - **`qikly --scaffold my_module.py`** then writes the task file,
      `inputs_private/config/tasks/<NAME>_VERIFY.yaml`, with the interface, a
      data file of the values their tests already use, and those questions under
      `acceptance_criteria`. With their key set, qikly's own spec agent drafts
      the `requirements` line and an answer to each question from their README
      and docstrings, never their code; each drafted answer carries its source,
      a quote from their documentation or `CHECK` where it was judged. **Show
      them the draft, the `CHECK` answers first, and change only what they tell
      you to.** Never redraft an answer yourself: you have read their code, and
      an answer read off the code agrees with it by construction. What the draft
      leaves as a TODO is theirs: put the question and any numbered options
      under it to them **word for word**, and write what they answer, the
      option's number, `1: <value>`, `keep`, or their own sentence naming a
      value. A TODO left in place is skipped. Without a key every answer is a
      TODO and `requirements` is marked `TODO: MODIFY THIS:`; replace it only
      with their words. `qikly --validate my_module.py` checks it, free.
    - **`qikly --tasks my_module.py`** runs it, which needs their key. It ends by
      pointing at `qikly --take-tests my_module.py`, which copies the new tests
      home and runs them on their code; a copied test that fails there is for
      them to compare with its criterion, not proof the code is wrong.
    
    ## Ask economically, or you will lose them
    
    Every decision you surface costs the user attention, and there is no version of
    this where nobody has to decide anything. So the cost has to come out of the
    **shape** of the asking, not out of the number of decisions.
    
    **Propose a complete draft and invite corrections. Never interrogate.** One
    decision, wait, next decision, wait, is how a ten-line module produces four
    modal questions and a user who closes the window. Instead:
    
    > Here is what I am going to assume, all of it going into `requirements` where
    > the coding agent will read it:
    > - percent is between 0 and 100 inclusive
    > - quantity, unit price and subtotal are zero or more
    > - money is not rounded to the cent
    >
    > Tell me which of those is wrong, or say go ahead.
    
    The discipline is identical: the decisions are still stated, still in the half
    the coding agent reads, still the user's. What changes is that agreeing costs
    one keystroke, and disagreeing is easier too, because somebody reading a list
    spots the wrong line faster than somebody three dialogs deep.
    
    **And only raise a decision the tests actually turn on.** If no criterion you
    are going to write would differ between the two answers, you are spending their
    attention for nothing. The question earns its place when you can say which test
    changes.
    
    **Two you should always raise, because they are silent and expensive:** the
    rounding rule wherever money appears, since "nearest cent" settles nothing at a
    half cent, and the inclusive or exclusive end of any boundary. Those two account
    for most of the decisions that get hidden in the wrong half.
    
    **If they insist on tests now**, write them, and then say plainly which lines
    of the implementation you had to treat as correct: every boundary, every
    tie-break, every validation rule you copied rather than were told. Those are
    the decisions nothing has settled, and naming them is the difference between a
    suite that is honest about its foundation and one that looks authoritative.
    
    A real session got this half right: it wrote seventeen tests from the code,
    checked they could fail by planting faults, and only then said "I wrote these
    by reading your code, so where the code made a choice, the tests assume that
    choice was right". The saying so was correct. The order was backwards.
    
    qikly splits one specification in two. Test generation reads the whole thing.
    The coding agent receives the same file with the `acceptance_criteria` section
    cut out, and when a test fails it sees the failure, never the criterion it
    broke. That separation is enforced in qikly's own code, not by instructions in
    this file, which matters: **this skill cannot keep anything hidden. The tool
    does that. This skill only helps you use the tool correctly.**
    
    **And be precise about what the withholding covers**, because a user will
    eventually stretch it. It is one thing: inside a qikly run, the coding agent's
    prompt is assembled without the `acceptance_criteria` section. It says nothing
    about this conversation. If someone pastes their criteria to you, you have
    read them, and no part of qikly prevented that or knows it happened. Say so
    plainly if you are asked whether talking to you is covered.
    
    ## Install
    
    ```bash
    pip install qikly
    qikly --version          # version, package directory and interpreter
    ```
    
    **This Skill needs qikly 0.5.4 or later**, which is where `--score-code`
    arrives (called `--score` from 0.6.0, and `--score-code` still works), alongside `--score-suite` and the reachability warning in `--validate`
    from 0.5.3. Against an older install an
    agent following this page will recommend a flag that does not exist, so check
    `qikly --version` before trusting the command table below. The Skill's own
    version is separate from the tool's: it changes when these instructions
    change, not when qikly releases.
    
    A run needs one provider key, `GEMINI_API_KEY`, `OPENAI_API_KEY` or
    `ANTHROPIC_API_KEY`. Several commands need no key and cost nothing; the table
    further down says which, and when to reach for each.
    
    **Which key is present decides how long a run takes.** On
    `ANTHROPIC_API_KEY` qikly uses `claude-sonnet-5`, which thinks before every
    answer, so a ten-call loop becomes minutes plus thinking tokens you are billed
    for. The published figures come from `gemini-3.5-flash-lite`, fast and cheap
    enough to repeat. Say which key a run will use, and what it means for the wait,
    before starting it. That goes for you too: a reasoning model driving this tool
    charges the user the same wait at every step, so keep the mechanical steps
    mechanical.
    
    **So set the model rather than only warning about it.** Before the first paid
    command, if `ANTHROPIC_API_KEY` is the key in play, set `LLM_MODEL` to
    `claude-haiku-4-5` for the run and say you have done it and why. That is the
    only provider whose default thinks: Gemini's is already
    `gemini-3.5-flash-lite` and OpenAI's is `gpt-4o`, neither of which has a
    reasoning step, so there is nothing to change on either. If the user has
    chosen a thinking model deliberately, leave it alone and say what the wait
    will be: the stricter suite it writes is a real reason to want one.
    
    ## The one question that decides everything
    
    Every line of a specification goes in one of two halves, and this settles it:
    
    > **Given only the requirements, could two competent developers legitimately
    > disagree about this line?**
    
    **Yes, it is a decision.** It belongs in `requirements`, where the coding agent
    reads it. Nobody can guess a choice somebody made: a threshold, a unit, a
    measurement convention, an exemption.
    
    **No, it follows.** It belongs in `acceptance_criteria`, which are withheld.
    The exact boundary, the identity that must hold, the case a careless reading
    gets wrong.
    
    | In `requirements`, a decision | In `acceptance_criteria`, a consequence |
    |---|---|
    | Keep at least 2.5 m from the vehicle ahead, centre to centre | At exactly 2.5 m, no violation is raised |
    | Amounts are currency, rounded to the nearest cent | For every accepted row, total equals subtotal plus tax, exactly |
    | Dates are written YYYY-MM-DD | 2026-02-30 is rejected, because it is not a real date |
    
    **A worked case, because this one is easy to get backwards.** A spec says
    *"warn when following distance breaks the two-second rule"*, and the
    acceptance criteria say *"a headway of exactly 2.00 s does not raise a
    warning"*. That split is **wrong**, and the reason is the question above:
    "breaks the two-second rule" reads as "below two seconds" just as naturally as
    "at or below two seconds", so two competent developers can disagree about
    2.00 s itself. It is a decision, and it has to move into `requirements`.
    Measured on this exact task, leaving it in the criteria converged 3 runs in 10;
    moving that one sentence converged 10 in 10. Read that as what it is: evidence
    that the agent was guessing, not evidence that the suite got better. A run that
    cannot converge is a run that proves nothing at all, which is a different
    problem from a suite that converges and proves little.
    
    Note what moving it does **not** mean. Moving a decision is a correction the
    spec always needed, and it is justified by the question alone, without looking
    at any code. Copying a criterion's boundary value into the requirements to get
    a run green is the opposite: it tells both agents the answer, and the test that
    checks it then passes first try and proves nothing.
    
    **Both mistakes have a signature.**
    
    A *decision* hidden in the criteria shows up as repetition: near-identical FIX
    and PATCH each round, or two tests disagreeing where every patch fixes one and
    breaks the other. Sometimes it guesses right and the run goes green, which is
    worse, because nothing then tells you the line was in the wrong half.
    
    A *consequence* left in the requirements is quieter: the test for it passes
    first try, nothing was learned, and the run looks entirely normal.
    
    **Spotting one is your job; settling it is not.** You can tell a decision from
    a consequence by applying the question above to the words on the page, and you
    should: say which line you think is in the wrong half and why. What you cannot
    do is choose the answer. Whether a headway of exactly 2.00 s warns, whether an
    empty string counts as missing, whether currency rounds half up or half even:
    nothing in the specification settles those, which is what makes them decisions,
    and guessing on the user's behalf puts an invented choice into their
    requirements where it will look decided. Name the ambiguity, propose the
    wording, ask which way they want it.
    
    qikly's own refinement loop cannot do even the spotting: it can tighten a bar
    the code already attempts, and it cannot tell you a line is in the wrong
    place.
    
    ## Writing criteria that can be tested
    
    **Name values, not adjectives.** "Reject large amounts" produces a test at some
    arbitrary large number. "100 is accepted and 101 is rejected" forces the
    boundary.
    
    **Name both sides of a boundary in one criterion.** "The 250 limit is
    inclusive: exactly 250 is accepted and 250.01 is rejected" is one criterion
    closing one ambiguity, and it tells you exactly which two rows the data needs.
    
    **But check first which half the boundary belongs to, because this technique
    and the worked case above look identical on the page.** One rule separates
    them, and it is about the requirement's own wording:
    
    > Does the requirement already settle which side the edge falls on?
    
    "Keep **at least** 2.5 m" settles it: at 2.5 m you comply, so a criterion
    saying no violation is raised at exactly 2.5 m only spells out what was
    already decided. Write it as a criterion.
    
    "Warn when the headway **breaks** the two-second rule" does not settle it, and
    neither does "rounded to the **nearest** cent" when a value lands exactly
    halfway, or "reject rows where quantity is **missing**" when nobody said
    whether an empty string counts. In each case the criterion would be making the
    choice rather than recording it. Move the choice into the requirement, then
    write the criterion for its consequence.
    
    Words that usually settle it: at least, at most, above, below, strictly, on or
    after. Words that usually do not: nearest, breaks, exceeds a limit, missing,
    invalid, malformed.
    
    **Every value you name has to exist in the data.** A criterion no input row can
    trigger produces a test that passes whatever the code does. This is the most
    common reason a suite measures less than it appears to.
    
    **Then go back through the requirements and pair them.** Every decision in the
    requirements should have a criterion that would catch it being implemented
    wrong, and this is the step people skip: they write the decisions carefully,
    write criteria for the two or three boundaries that worry them, and leave the
    rest of the specification unchecked. "Line totals are quantity times unit
    price" is a decision; the criterion that pairs with it is an identity, "for
    every accepted row, line_total equals quantity times unit price, to the penny".
    Without the pair, the agent can get the arithmetic wrong and nothing fails.
    
    Ask it as a sweep: for each requirement, what would a wrong implementation of
    this look like, and which criterion catches it? A requirement with no answer is
    a requirement nothing is testing.
    
    **And read your own requirements back against the word list above.** This
    applies to your own wording too, which is where it gets missed: writing
    "rounded to the nearest cent" creates the same ambiguity you would flag in
    somebody else's spec. If a requirement you drafted uses
    one of those words, say so and ask which way the boundary falls, rather than
    writing criteria that quietly avoid the case. A tie nobody decided is not a
    withheld consequence, it is a decision nobody made, and it will surface as a
    stalled loop later.
    
    ## The five steps, on the user's own module
    
    ```bash
    qikly --scaffold my_metrics.py          # reads real signatures, writes a task file
    # add example rows to the data file it started: the first line
    #   already names the columns; for code given a file, copy a real one in
    # fill in `requirements` and `acceptance_criteria`, using the question above
    qikly --validate --tasks MY_METRICS_VERIFY    # free, no model call
    qikly --tasks MY_METRICS_VERIFY               # the paid one
    ```
    
    **Before that last line, check which provider key is set.** It decides the
    model, and therefore the wait and the bill; see Install above. On
    `ANTHROPIC_API_KEY` every call thinks before it answers, and a run of ten calls
    is minutes rather than seconds. Tell the user which one they are about to spend
    on before they spend it.
    
    **A task file has a third section.** `interface` names the module and the
    function signatures, and the coding agent reads it: it is how both agents agree
    what to call things. Scaffolding fills it in from the real signatures, so it
    rarely needs editing, but a line about the shape of the output belongs there
    rather than in either half above.
    
    `--scaffold my_metrics.py` writes a task that tests code that already exists.
    Add `--fresh` for one that writes a new implementation of the same interface
    and tests that. Both land in `inputs_private/config/tasks/`.
    
    **The task id comes from the module's filename, upper-cased**, so
    `pricing.py` gives `PRICING`, and the plain scaffold adds `_VERIFY` because it
    tests code you already have: `PRICING_VERIFY`. `--fresh` gives `PRICING`. The
    command prints the name and the path it wrote, so read that rather than
    guessing.
    
    **Never edit the user's module to make a test pass.** qikly does not, and
    neither should you.
    
    **If the project tells you to write code and tests together, say so rather
    than choosing silently.** A house rule like "always write the implementation
    and its tests in the same session so they stay consistent" is reasonable on its
    own terms and is the exact thing this tool exists to prevent: consistency by
    construction is what makes a suite unable to disagree. You cannot follow both.
    Tell the user the two conflict, in one sentence, and let them decide which
    applies here.
    
    To see the whole shape first, `qikly --example` lays down a finished worked
    task, module and sample data included, so you can read a filled-in pair before
    writing one, in the directory you are standing in.
    
    **`qikly --demo` is a different command and the difference matters.** It runs a
    bundled task end to end in a throwaway `demo/throwaway_<timestamp>/` folder that exists to
    be deleted, and it is the one most people try first. A user who has just watched
    it work is standing in something that looks exactly like a working project, and
    the obvious next move is to start theirs there. **Never set up someone's real
    project inside a demo folder.** If the user says they ran "the demo" and wants
    to continue where they are, establish which of the two commands they ran before
    writing anything. This has already cost a first-time user an afternoon.
    
    ## The free checks, and when to reach for each
    
    **If you arrived straight here, this is the rule you skipped, stated in full
    so you do not have to go back for it.** Asked for tests for a module with no
    specification: do not write tests, and do not run the module to find out what
    it should do. A value obtained by executing an implementation is not a
    contract, and a suite built from those values passes by construction and
    cannot disagree with a bug. Whether a threshold is inclusive, whether the
    discount applies before or after tax, where a floor lands, which way a
    half-cent rounds: the code answers all four and none of those answers is a
    specification. Deriving the expected values yourself is the failure this
    Skill exists to prevent, and it is the one an agent commits while believing it
    is being careful.
    
    **Which command fits depends on what they have.** With a suite already
    written, `--score` scores it and needs nothing else. With no suite and no
    spec, which is the case this paragraph is usually about, the route is
    `qikly --scaffold their_module.py` and then filling in the two sections with
    them. Everything else in the table below needs a task file that does not exist
    yet.
    
    | Command | Cost | Use it when |
    |---|---|---|
    | `qikly --validate --tasks X` | free | always, before any run. Catches a missing input file, a leftover TODO, a criterion made of adjectives, a requirement restating a criterion, and criteria the data cannot reach |
    | `qikly --explain X` | free | to show the user exactly what each agent receives, criteria present on one side and absent on the other |
    | `qikly --score-suite --tasks X` | free, and slow | after a run converges, to find what the suite would not have noticed. It plants one fault at a time in the code and reports which ones the tests missed. Free because every fault is an edit to the code's syntax tree and no model is asked anything; slow because each fault means running your whole suite again |
    | `qikly --score PATH --score-tests PATH` | free, and slow | **for a suite qikly did not write**, which is what somebody already has before they have anything else. Point it at a module or package and the tests for it, and it reports which planted faults the tests did not notice, with a question for the user about each, which `--scaffold` then writes into the task file. No task file, no run, no model call. The report lands in `qikly-suite-score/` in the directory they ran it from |
    | `qikly --check-criteria --tasks X` | one model call | when a spec may contradict itself, before spending a run on it |
    | `qikly --propose-fixtures --tasks X` | one model call | when `--validate` says a criterion's values are missing from the data, to get the rows it would take |
    
    **Read `--score-suite` beside the reachability warning it prints above the
    number.** A suite cannot catch a fault in behaviour no input row exercises, so
    unreachable criteria lower the score for a reason that is about the fixtures
    and not about the tests. Fix the data first, then score.
    
    **A high score is not a clean bill of health, and say so before they read it
    as one.** Mutation scoring asks whether their tests notice changes to the code
    that exists. It cannot ask about a rule nobody implemented, because there is
    nothing there to break. In a real session a seventeen-test suite written by
    reading the code scored at the top of the scale while a suite written from the
    specification found four genuine bugs in the same file. The number is a floor,
    not a verdict.
    
    **`--score` prints no such warning, and you must not imply it does.**
    There is no task file and therefore no criteria to be unreachable, so the
    report carries the score and nothing above it. The underlying problem has not
    gone away: a fault that survives may be on a line no test ever executes, which
    is a gap in what the tests reach rather than in what they assert. Say that to
    the user rather than handing them a percentage as a verdict on their suite.
    
    **When `--score` reports misses, its report ends with a question about
    each one** (qikly 0.6.0 or later), such as whether `rate > 100` or
    `rate >= 100` is what they meant, and what should happen at exactly 100.
    `qikly --scaffold` on the same module then writes those questions into the
    task file's `acceptance_criteria`, each as a comment above a `TODO`.
    **Those questions are the user's to answer, never yours.** Put them to the
    user in plain words and write down what they say in place of each `TODO`,
    deleting any they skip. Do not answer them from the code: what the code does
    now is exactly what is in question, and an answer read off it is a criterion
    that agrees with the code by construction. `qikly --validate` says how many
    answers are still `TODO`. Leave the `#?` question lines where they are or
    delete them: qikly removes them from what either agent receives, so moving
    them elsewhere in the file exposes nothing.
    
    ## When a run does not converge
    
    It exits non-zero, names the tests that blocked it, and ships nothing. Read
    which stage stopped first: the unit stage is last and strictest, and most
    failures are there.
    
    Then, in order:
    
    1. **Is the loop repeating itself?** Near-identical FIX and PATCH each
       iteration means a decision is in the wrong half. Move it into
       `requirements`.
    2. **Do two tests disagree,** each patch fixing one and breaking the other? The
       specification contradicts itself. `--check-criteria` finds that before a run.
    3. **Can the data reach every criterion?** `--validate` now says, and
       `--propose-fixtures` drafts the rows.
    
    Then the cheap levers: a larger model, and more attempts. And never loosen a
    criterion to get green, for the reason given above.
    
    If no patch ever applies at all, and you are on macOS, that is section 12 of
    `references/TROUBLESHOOTING.md`.
    
    **What the agent sees when a test fails, exactly.** pytest's output for the
    failing test: its name, its own source and docstring, and the assertion error.
    Not the acceptance criterion. Because a generated test's docstring usually
    restates the rule it came from, a failing test does tend to give away its own
    case, and the project says so rather than pretending otherwise. It does not
    undo the split: the suite was written first, from criteria the coder never
    read, and nothing learned afterwards changes a test already on disk. There is
    a setting that narrows this, `diagnostic_feedback: staged` under `agent:` in
    settings.yaml, which starts the agent at a one-line error and climbs
    towards the full traceback, a rung for every patch that changes nothing and a
    rung every third attempt whatever happens. **It is off by default**, because every
    published convergence figure was measured with the full traceback and nobody
    has measured what starting narrow costs.
    
    **A run prints nothing while a model call is in flight**, which on a reasoning
    model can be minutes. After ten seconds it starts saying so, one line every
    fifteen: `[patch] still waiting on the model, 45s`. Those lines are the
    difference between slow and stalled, so pass them on rather than swallowing
    them, and do not conclude a run has hung while they are still arriving.
    `QIKLY_NO_PROGRESS=1` turns them off.
    
    **To stop a run spending more than you meant**, three environment variables
    bound it: `QIKLY_MAX_CALLS` and `QIKLY_MAX_TOKENS` bound one task's process,
    and `QIKLY_MAX_SWEEP_TOKENS` bounds a whole sweep. Set them before a first run
    on somebody's real code rather than after.
    
    **In CI**, qikly ships a GitHub Action. A run costs a model call per attempt,
    so per pull request is a budget decision rather than a technical one, and the
    free checks are the ones that belong on every commit: `--validate`,
    `--score-suite` for a suite qikly generated, and `--score` for one that
    predates it, which on most real repositories is the relevant half.
    
    ## What to tell the user honestly
    
    Roughly 8 runs in 10 produce code passing every integration and system test,
    and roughly 6 in 10 pass everything including unit tests, measured over 967
    runs and reproduced over 390 more. **Always say what those runs were:** a
    small, inexpensive model (`gemini-3.5-flash-lite`), chosen so the sweeps could
    be repeated affordably, so the figures are a floor rather than a ceiling. And
    they measure convergence, whether generated code passes the generated tests,
    not whether those tests catch real defects.
    
    **Two things are settled and one is not, and they are easy to confuse.**
    
    *Settled:* the coding agent never receives the acceptance criteria. That is a
    property of the code, checkable with `qikly --explain` and held by a test that
    fails the build if any call site ever leaks one. It is not a benchmark result
    and cannot go stale.
    
    *Settled:* the convergence figures above, measured and re-measured.
    
    *Open:* whether a suite written from withheld criteria catches **more real
    defects** than one written with sight of the code. That comparison has not been
    run, here or anywhere. A Google team has measured the step before it, that
    generating tests from a written contract rather than from the code raises bug
    detection by 9.8 points (arXiv 2608.17177), which is adjacent and not the same
    claim. Separately again, six experiments asked whether automatically *refining*
    the criteria produces a sharper bar and none detected an effect, which is an
    absence of evidence rather than evidence of absence. Three different questions.
    Do not offer any of them as evidence for another.
    
    ## References, and when to open them
    
    Do not read these by default. Each one is a full page, and everything above is
    enough for a first run.
    
    **Read `references/TASK_FILE_REFERENCE.md`** when you need a field this page
    does not name, when the user already has criteria written somewhere else and
    wants them imported, when they want to seed their own implementation or suite
    rather than have one generated, or when you need to know where a run writes
    each artefact.
    
    **Read `references/TROUBLESHOOTING.md`** when a run has already failed and the
    three questions above did not explain it. It has eleven numbered causes, each
    with its own fix, and the triage table at the top maps a symptom to a number.
    
    The repository, for anything neither covers:
    <https://github.com/gal-a/qikly>
    
  • README.mdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/gal-a-qikly-claude-skills-qikly.md

// REPO STATS

19 stars
0 open issues
Last commit: 8h ago