How quell verify proves a test catches bugs

quell verify answers one question: would this test fail if the code were wrong? It runs your test on the original source, then on small deliberate breakages (mutants) of that source inside a temporary copy of the project, and reports whether the test noticed. Use it on a test that an AI agent or a teammate just wrote, before you trust it, or in CI on the test files a pull request touches.

npx quell verify tests/pay.test.js --source src/pay.js --runner jest

Verified: test killed 4/11 mutants (36%) [copy]
  survived: || -> ?? (0 no longer falls back) (line 8)
  survived: guard removed (line 3)
  ...

The exit code is 0 when the test is verified and 4 when it is not.

What are the rules for "verified"?

A test is verified only if all three hold:

1. The baseline passes. The test must pass on the original, unmodified source. A failing baseline is reported as test fails on the original code with the command and the first lines of the error.
2. An assertion observes the code. The test must contain an assert, expect, t. or should call with an argument that depends on something other than literals (a call, an await, or an identifier). expect(true).toBe(true) and a test that only calls the function without asserting do not count: test has no assertion that observes the code under test.
3. At least one mutant fails the test. Optionally --min-score N also requires that N% of decided mutants are killed.

If every mutant survives, the result is test passes on every mutant of the code under test. If there is nothing to mutate: no applicable mutation in the code under test.

What is a mutant?

Mutants are deterministic text edits chosen to hit the places where bugs hide. They are ordered so that mutants aimed at risky code come before generic operator flips when the cap cuts the list:

At most 30 mutants run per verification and at most 8 per kind, so a route file is not 30 dropped awaits. A mutant that does not parse is dropped instead of being counted as a kill. Restrict mutation with --function NAME (repeatable) or --findings (only lines the rule pack flags).

What are kills, survivors and inconclusive runs?

Only genuine test failures are kills. The mutation score is killed / (killed + survived); inconclusive runs are excluded and printed as N inconclusive. If nothing was killed and some runs were inconclusive, the reason says so (no mutant was killed (N timed out or could not run)).

Where does it run my code?

By default in a temporary copy of the project, so your real source and test files are never written. The copy skips node_modules, .git, coverage, dist, build, .next, .cache and .turbo (except where they hold the source or test), links node_modules, and dereferences symlinks so a mutant can never be written through a link. The output ends with [copy].

• The copy is the nearest folder with a package.json that contains both files, otherwise their common folder (never a filesystem root or your home).
• The test runs in a subprocess, never in-process, so module caches cannot hide a mutation.
• The test command's working directory is the source file's folder inside the copy.
• Projects over 20,000 files or 200 MB cannot be mirrored. Verification then mutates the source in place, keeps a backup file next to it, and always restores it in a finally block; a backup left by a killed run is repaired by the next find, generate or verify. The output says [in-place]. --in-place forces this mode.

Which runners are supported?

--runner node (the default, node --test), jest, vitest and mocha. quelltest locates the runner's entry point in node_modules and runs it with Node, with no shell involved. verify does not detect your runner: for a Jest test pass --runner jest. Without it, the Jest test above fails on the baseline with ReferenceError: test is not defined and the command exits 4.

Vitest and project root: the runner starts in the source file's folder. Vitest treats that folder as its root, so a test in tests/ for a source in src/ is not found (No test files found). Keep the test next to the source, or use --command (below) with --root ...

How do I use my own test command?

--command replaces the runner with a command of yours, run for the baseline and for every mutant. {test} is replaced by the path of the test in the copy. The command runs through your shell, in the source file's folder inside the copy.

npx quell verify tests/orders.int.test.js --source src/orders.js \
  --command "npm run test:integration -- {test}" --timeout 300

npx quell verify tests/pay.test.js --source src/pay.js \
  --command "node ../node_modules/vitest/vitest.mjs run --root .. {test}"

Both forms were run against a sample project: npm finds the nearest package.json upwards, while a direct path to a binary has to account for the working directory (../node_modules/...). The test path reaches your command through an environment variable, so a file name with $(...), backticks or % is never interpreted by the shell.

Reserve an exit code for "the environment failed" (a container did not start) and pass it as --infra-exit CODE (repeatable). Such runs are inconclusive, never kills. Use --command with Testcontainers or an in-memory server to measure a test against a real dependency instead of a mock; see code that calls APIs.

What is the handoff report?

--handoff FILE writes a Markdown file listing every surviving mutant with the changed line, surrounding source and a specific instruction, so an agent can strengthen the test and run verify again:

## 1. || -> ?? (0 no longer falls back) (line 8) in `applyDiscount`

```diff
- const rate = pct || 10;
+ const rate = pct ?? 10;
```

**What to do:** Falling back with `||` instead of `??` (or the reverse) goes unnoticed. Add a case
with `0`, `""` or `false` and assert the value is preserved or replaced as intended.

The report also shows counts of inconclusive and ignored mutants and labels survivors on code your test replaces with a double (jest.mock, vi.mock, sinon.stub, jest.spyOn, t.mock.method) as mockBoundary: a survivor there may only be observable against the real dependency.

If a survivor is an equivalent mutant (changing the line cannot alter behaviour), mark the line with // quell-ignore-mutant <reason> (or put the comment on the line above) and it is never mutated again. Ignored mutants are counted in the output.

How does the benchmark measure verify itself?

npx quell eval runs the verifier on three source units, each with one strong test and several weak variants (tautology, literal-only, no assertion, wrong target, happy path only). At a 60% mutation floor it accepted 3 of 3 strong tests and rejected 7 of 7 weak ones. These fixtures are written by the authors, so treat the result as a regression guard, not as an independent benchmark. See limits.