quelltest npm limits and known issues
quelltest for npm is young (0.3.x) and deliberately narrow. This page lists what it does not do and the rough edges found while writing these docs against version 0.3.1, each with a workaround where there is one. Read it before you put a check in CI.
What does it deliberately not do?
- No full-suite mutation score.
verifymutates one source file against one test file andgatelooks at one change. If you need a complete mutation score for a whole suite, use StrykerJS. Putverifyandgatein front of tests an agent writes, not in place of a suite-wide mutation run. - It does not know whether your stubs resemble real services. It checks that tests notice changes in your code.
quell mocksandverify --commandnarrow the gap; see code that calls APIs. - It generates tests for few real functions. Generation handles named, exported functions with statically readable guards (a parameter or property compared with literals,
typeof, truthiness, what an injected client, model or globalfetchreturns, JSDoc@example). Express handlers, React components and everything else become TODO notes. Route handlers also get a list of request cases derived from their guards. - Findings are review hints, not proof of defects. Rules are line-level. They cannot see wrong business rules.
- A high mutation score is not proof of a good test. A test that kills 4 of 11 mutants is verified; the survivors show what it misses.
How accurate is the rule pack?
npx quell eval measures it on fixtures written by the authors: treat these numbers as regression guards, not as an independent benchmark.
| Measure | Result |
|---|---|
| Rule recall on 18 defects reported against a real Express and React code base | 11 of 18 detected (61%); 11 of 11 for defects a line-level rule can see |
| Known misses | 7 (B3, B8, B10, B12, F3, F5, F6): wrong normalisation, business rules, UI behaviour. eval/defects.json lists them. |
| False positives on guarded code | 0 risk findings on 7 clean files |
| Verifier accuracy on strong versus weak tests (60% mutation floor) | 3 of 3 strong tests accepted, 7 of 7 weak variants rejected |
What are the known issues in 0.3.1?
| Issue | What happens | Workaround |
|---|---|---|
Relative SF: paths in lcov | --coverage coverage/lcov.info resolves relative SF: lines against the folder of the lcov file, so no file matches: score prints 0/100 with coverage evidence: lcov. Jest and Node's built-in lcov reporter both write relative paths. Seen with Jest 30 and Node 24.11. | Use coverage/coverage-final.json (Istanbul), or rewrite SF: to absolute paths (an absolute-path lcov scored correctly, line-granular). |
verify does not detect the runner | It defaults to node --test. A Jest, Vitest or Mocha test fails the baseline (for example ReferenceError: test is not defined) and exits 4. | Pass --runner jest, vitest or mocha. (generate does detect the runner.) |
| Working directory is the source file's folder | verify starts the runner in the folder of --source, inside the temporary copy. Vitest takes that folder as its project root and does not find a test in a sibling tests/ folder (No test files found). A --command that uses relative paths is relative to that folder. | Keep the test next to the source, or use --command with explicit paths, for example node ../node_modules/vitest/vitest.mjs run --root .. {test}. Jest and Mocha were found correctly from tests/. |
| Added test files are not gated | gate skips test files that are new in the change. | Run verify on new test files. |
What is the verification cost?
verify runs your test once for the baseline and once per mutant (30 mutants at most, 8 per kind by default), so it is slower than one test run: about 1.4 seconds per mutant with mocks in one report, much more with containers. It mirrors the project into a temporary folder; very large projects (over 20,000 files or 200 MB) fall back to in-place mutation with a backup file. Statement coverage is matched to candidates by position, and lcov.info is line-granular, so lcov scores are rougher than Istanbul JSON (the same sample scored 50 with absolute-path lcov and 33 with Istanbul JSON).
What has not been tested here?
- The Claude Code Stop hook and MCP recipes were not run inside live Claude Code or Cursor sessions; the MCP server itself was exercised over stdio. See MCP server and hooks.
- These docs were run on Node.js 24.11 on Windows. According to its README, the package CI runs on Node.js versions 20, 22 and 24, plus Windows and macOS on Node.js 22. Version 0.3.1 is the one these docs describe.
- Vitest and Mocha verification were run on a CommonJS sample; Jest and the default runner on the same project.
What changes in the next release?
The next release requires Node.js 22.18+ or 24.11+, because its parser dependency moved to @babel/parser 8. Version 0.3.1, the one you can install today, supports Node.js 20 and newer. Check the changelog for what ships.
Where do I report problems?
Open an issue on the quelltest-npm repository with the version (npx quell --version), your Node version and OS, the command and the smallest input that shows the problem. Security problems go to the private channels in privacy.