Skip to content
All posts
·7 min read·Shashank Bindal

Does Your Test Fail When the Code Is Wrong? How quell verify Works

quell verify runs a test on the original code, then on mutants in a throwaway copy, and rejects tests nothing can break. Real output, with a rejection.

TL;DR. quell verify answers that question by experiment. It checks that the test passes on the real code, then changes the code in small ways in a throwaway copy of your project and runs the test again. If the test still passes on every change, or if it contains no assertion that looks at the code, it is not verified and the command exits with code 4.

Captured with quelltest@0.3.1 from npm, Node.js 24.11.0, Windows. The test runner is node:test.

What does quell verify check, in order?

  1. Baseline. The test must pass on the original source. A test that fails on correct code proves nothing about bugs.
  2. The assertion observes the code. The test file must contain an assertion whose arguments depend on something other than literals. assert.ok(true) and assert.equal(1 + 1, 2) do not count.
  3. Mutants. The source is mutated one change at a time, and the test runs against each mutant in a subprocess. Never in-process, because module caches would hide the change.
  4. Kill. At least one mutant must make the test fail. With --min-score N, N% of decided mutants must be killed.

The mutants are aimed at places where bugs tend to hide: guards, sanitisers, await, ?? and || fallbacks, thrown errors, comparison operators, arithmetic. Risk-aimed mutants are ordered ahead of generic operator flips.

What does a run look like?

The fixture is one function:

export function applyDiscount(price, percent) {
  if (typeof price !== "number" || price < 0) {
    throw new RangeError("price must be a non-negative number");
  }
  const capped = Math.min(percent ?? 0, 50);
  return price - price * (capped / 100);
}

and a test file with four tests: a 10% discount gives 180 on a price of 200, a 90% discount is capped at 50, a missing percent means no discount, and a negative price throws RangeError.

npm install --save-dev quelltest
npx quell verify tests/discount.strong.test.js --source src/discount.js

Verified: test killed 6/8 mutants (75%) [copy]
  survived: ?? -> || (line 5)
  survived: < -> <= (line 2)

[copy] means the mutants ran in a temporary copy of the project. We checked the file hash of src/discount.js before and after a run: identical. If a project is too large to mirror, quelltest falls back to in-place mutation with a backup file that is restored in a finally block, and it warns you. --in-place forces that mode.

Two mutants survived. That is useful information, not a failure, because the default only requires one kill.

How does a weak test get rejected?

The weak test checks only the type of the result:

test("returns a number", () => {
  assert.equal(typeof applyDiscount(200, 10), "number");
});

npx quell verify tests/discount.weak.test.js --source src/discount.js --min-score 60

Not verified: mutation score 13% is below the required 60% [copy]
  survived: ?? -> || (line 5)
  survived: guard removed (line 2)
  survived: throw removed (line 3)
  survived: || -> && (line 2)
  survived: < -> <= (line 2)
  survived: - -> + (line 6)
  survived: * -> / (line 6)

Be aware of the default. Without --min-score, this same test prints Verified: test killed 1/8 mutants (13%) and exits 0, because one mutant was enough: flipping !== to === in the typeof check makes the function throw, and that failed the test. We recommend a floor of 60 in CI. The bundled benchmark uses that value.

The other two rejections need no mutants at all:

Not verified: test has no assertion that observes the code under test [copy]

That is what a test with assert.equal(1 + 1, 2) gets, and what a test with no assertion at all gets. And a test with a wrong expectation:

Not verified: test fails on the original code (node --test <temp copy>\tests\discount.wrong.test.js): ✖ wrong expectation (1.0177ms) | ℹ tests 1 | ℹ suites 0 | ℹ pass 0 [copy]

Both exit with code 4.

What counts as a kill, and what is inconclusive?

Only a genuine test failure counts as a kill. Each mutant run has one of four outcomes in the JSON report: killed, survived, timeout or error. A timeout, a runner crash, a signal, or an exit code you reserved with --infra-exit is inconclusive. It is never counted as a kill, because a crashed runner says nothing about your test.

The mutation score is kills divided by decided mutants, where decided means killed or survived. If a run has inconclusive mutants, the summary line adds , N inconclusive. If nothing was killed and some runs were inconclusive, the reason says so instead of claiming the test is weak.

Mutants that do not parse are dropped before any run instead of being counted as kills. Candidate mutants are capped at 30 per run by default and at 8 per kind, so a run costs at most about 30 executions of your test file.

What are equivalent mutants, and how do I handle them?

Some mutants cannot change behavior. In our function, percent ?? 0 and percent || 0 agree for undefined and for 0, which are the only values anyone realistically passes. No test can tell them apart, so the survivor is noise.

Mark the line instead of contorting the test:

  // quell-ignore-mutant ?? and || agree here: both give 0 for 0 and undefined
  const capped = Math.min(percent ?? 0, 50);

The comment applies to its own line and the line after it. The report then shows , 1 ignored.

The other survivor, < -> <= on price < 0, is real: nothing tested a price of exactly zero. We added one test, applyDiscount(0, 10) equals 0, and ran with a floor of 90:

npx quell verify tests/discount.strong.test.js --source src/discount.js --min-score 90

Verified: test killed 7/7 mutants (100%) [copy], 1 ignored

Before the extra test, the same floor of 90 produced Not verified: mutation score 75% is below the required 90%, exit 4, and with --handoff handoff.md wrote a Markdown file for each survivor with the diff, context and a specific instruction, such as "add an assertion on both sides of this operator". That file is meant for a coding agent to act on.

What are the limits?

  • It checks whether tests notice changes in your code. If your tests mock a dependency, the mutants that survive on mocked code are labelled mockBoundary in the handoff, but quelltest cannot tell you whether the mock matches the real service.
  • It is slower than a plain test run because the test executes once per mutant.
  • It does not give a suite-wide mutation score. Stryker does.
  • It works on one test file and one source file at a time. Pass --function NAME to focus on a function.
  • The verify command uses node:test unless you pass --runner jest, vitest or mocha. It does not read your package.json to choose. See the runner post.
  • Passing verify shows that the test is capable of failing. It does not show that the test checks the right thing.

Read the verify reference and the limits. Release details are in the changelog.

FAQ

How is this different from coverage?

Coverage records that a line ran. verify records that the test failed when the line changed. A test with no assertions can reach 100% coverage and still be rejected here.

Will verify modify my source files?

By default it works in a temporary copy, and the real source is not written. Only the in-place fallback touches the file, and it restores it afterwards.

How many mutants does it try?

At most 30 per run by default, at most 8 of each kind. The --findings option aims mutants at lines the scanner flagged.

Can I run my own integration test as the verifier?

Yes. --command "npm run test:integration -- {test}" runs your command for the baseline and every mutant. Use --infra-exit for an exit code that means the environment failed.

Sources