Skip to content
All posts
·8 min read·Shashank Bindal

The Gate for AI-Written Tests: Introducing quelltest 0.3

quelltest 0.3 proves a test fails when the code is wrong, flags weakened tests in a diff, and finds untested risky edges. Free, offline, Apache-2.0.

TL;DR. A coding agent that writes both the code and the tests can make the suite pass without making it meaningful. quelltest 0.3 for npm adds two checks that do not depend on the agent's honesty: quell verify runs a test against mutants of your source and fails it if nothing notices, and quell gate fails a change that deleted, skipped or loosened tests. Both are free and offline.

Everything below was run on Windows with Node.js 24.11.0 and the published quelltest@0.3.1 package from npm.

What problem does quelltest 0.3 solve?

When an agent finishes a task, it reports that the tests pass. That sentence has two weak points. The tests may never have been able to fail, and the agent may have edited the tests to get to green.

Coverage does not help with either. A line that ran is not a line that was checked. We wrote about that in 100% code coverage is a lie.

quelltest 0.3 is our answer for JavaScript and TypeScript. It is three commands you can run locally, in CI, or from inside an agent session.

What do verify, gate and find do?

CommandQuestion it answers
quell verify TEST --source FILEWould this test fail if the code were wrong?
quell gate --diff REFDid this change weaken the tests?
quell find --diff REF --coverage FILEWhich risky edges in my change have no real test?

verify copies your project to a temporary directory, checks that the test passes on the original source, then runs it again on small mutations of the source: a removed guard, a flipped <, a ?? swapped for ||, a removed await. A test that passes on every mutant is not protecting that code. Your real files are never written. The details are in How quell verify works.

gate compares the test files in your branch with their version at the merge-base. It reports deleted tests, skipped tests, a new .only, removed assertions and equality loosened to a truthiness check. See Did the AI weaken your tests?.

find parses your source with a real parser and lists risk-ranked candidates per function: thrown errors, boundaries, nullish paths, and a rule pack for defects that lack a guard, such as a regular expression built from request data. It follows request data one hop from req.body, req.query and req.params into dangerous sinks.

Why don't prompts asking agents to be rigorous hold up?

This is our position, not a measurement. An instruction in a prompt is a request. The agent can forget it, misread it, or satisfy its wording while missing its intent. When the same agent writes the code and the tests, an assertion that is inconvenient is also easy to loosen.

A program with an exit code does not negotiate. quell gate exits with code 5 when it finds weakened tests. quell verify exits with code 4 when it cannot verify a test. A CI job or an agent hook can act on those codes without trusting anyone's summary. We cover the hook setup in Giving Claude Code, Cursor and other agents a verifier.

What does it look like on a real test?

Take a small function and two tests for it.

export function applyDiscount(price, percent) {
  if (typeof price !== "number" || price < 0) {
    throw new RangeError("price must be a non-negative number");
  }
  const capped = Math.min(percent ?? 0, 50);
  return price - price * (capped / 100);
}

The first test checks only the type of the result:

test("returns a number", () => {
  assert.equal(typeof applyDiscount(200, 10), "number");
});

It passes, and it has an assertion, so a naive check would accept it. This is what verify says with a 60% floor:

npm install --save-dev quelltest
npx quell verify tests/discount.weak.test.js --source src/discount.js --min-score 60

Not verified: mutation score 13% is below the required 60% [copy]
  survived: ?? -> || (line 5)
  survived: guard removed (line 2)
  survived: throw removed (line 3)
  survived: || -> && (line 2)
  survived: < -> <= (line 2)
  survived: - -> + (line 6)
  survived: * -> / (line 6)

One mutant out of eight was caught. A fuller test file that checks the discount, the cap, the missing percent, the negative price and the zero boundary gets this (one mutant that cannot change behavior is marked with quell-ignore-mutant, which is explained in the post on verify):

Verified: test killed 7/7 mutants (100%) [copy], 1 ignored

Without --min-score, a test is verified as soon as it kills at least one mutant. That is why the weak test above reports Verified: test killed 1/8 mutants (13%) and exits 0 by default. We recommend setting a floor; the benchmark below uses 60%.

Now the gate. We made a branch in a throwaway git repository where an agent "fixed" the suite by weakening it:

npx quell gate --diff main

Gate failed: 4 issue(s) in 1 changed test file(s)
  ✖ tests/discount.test.js :: rejects negative prices — test deleted
  ✖ tests/discount.test.js :: accepts a price of exactly zero — test deleted
  ✖ tests/discount.test.js :: applies a percentage discount — assertions weakened (0 -> 1 weak, e.g. equality replaced by ok/truthy/not-null)
  ✖ tests/discount.test.js :: caps the discount at 50 percent — test is now skipped (skip/todo/x-prefixed)

npm test on that branch printed pass 2, fail 0, skipped 1. The suite was green.

What does the benchmark show?

quelltest ships a benchmark, and npx quell eval prints it. This is the actual output from the published 0.3.1 package:

{
  "rules": {
    "total": 18,
    "detected": 11,
    "recall": 0.6111111111111112,
    "detectableRecall": 1,
    "missedDetectable": [],
    "knownMisses": ["B3", "B8", "B10", "B12", "F3", "F5", "F6"],
    "cleanFiles": 7,
    "noise": []
  },
  "verifier": {
    "minScore": 60,
    "strong": 3,
    "strongAccepted": 3,
    "weak": 7,
    "weakRejected": 7,
    "accuracy": 1,
    "wrong": []
  }
}

(We reformatted the knownMisses array onto one line. The values are unchanged.)

Read it carefully. The rule pack detected 11 of 18 defects that were reported against a real code base. The other seven are documented known misses, mostly wrong business rules that a line-level rule cannot see. On seven files of guarded code it raised no findings. The verifier accepted all three strong tests and rejected all seven weak variants (tautology, literal-only, no assertion, wrong target, happy path only) at the 60% floor.

These fixtures were written by the authors. Treat the numbers as regression guards for the tool, not as an independent benchmark of how it performs on your code.

How do I try it?

npm install --save-dev quelltest
npx quell verify tests/pay.test.js --source src/pay.js --min-score 60
npx quell gate --diff origin/main
npx quell find src --diff origin/main --coverage coverage/lcov.info

find never invents a score. With no coverage evidence it prints this instead of a number:

Score: not scored (no coverage evidence) | 13 candidates

Quickstart: /docs/npm/quickstart. Installation: /docs/npm/installation. The full command list is in the CLI reference.

What are the limits?

  • verify measures whether a test notices changes in your code. It does not know whether your mocks resemble the real services. See Testing code that calls APIs and services.
  • verify runs your tests once per mutant, so it is slower than one test run. The default cap is 30 mutants.
  • gate does not catch a test rewritten from scratch under a new name, weakening done by changing the source, or logic hidden in helper functions that the test calls.
  • Findings from find are review hints. Line-level rules cannot see wrong business rules.
  • Generation handles guards the rules can read statically. Everything else becomes a TODO note, never a placeholder test.
  • It is a young tool. Read the changelog and limits page.

FAQ

Does quelltest send my code anywhere?

No. It runs locally and sends no code to any service. An optional --llm hook runs a module you supply, and that module is your code.

Which runners and Node versions work?

Jest, Vitest, Mocha and node:test. The published 0.3.1 declares Node.js 20 or newer; the next release will require Node.js 22.18+ or 24.11+. See one verification layer for four runners for the runner details, including one --runner caveat.

Is this a replacement for Stryker?

No. Stryker produces a full-suite mutation score. quelltest checks one test file or one change at a time and aims mutants at risky code. If you need a suite-wide score, use Stryker.

What licence is it under?

The npm package is Apache-2.0. The Python package on PyPI is a separate product under the MIT licence. Picking between them has a decision table.

Sources