quell gate: detect weakened tests in a change

quell gate --diff REF compares every test file changed since the merge-base with REF against its previous version and fails (exit code 5) when a change makes the suite weaker: tests deleted or skipped, .only added, assertions removed, or equality loosened to a truthy check. Use it in pull requests and as an agent hook, because an agent that writes both the code and the tests can turn a red check green by weakening a test.

npx quell gate --diff origin/main

Gate failed: 3 issue(s) in 1 changed test file(s)
  ✖ tests/pay.test.js :: charge returns something — test deleted
  ✖ tests/pay.test.js :: charge subtracts the amount — assertions weakened (0 -> 1 weak, e.g. equality replaced by ok/truthy/not-null)
  ✖ tests/pay.test.js :: charge rejects zero — test is now skipped (skip/todo/x-prefixed)

What does it compare?

The gate runs git merge-base REF HEAD and compares the working tree (committed, uncommitted and untracked changes) with the files at that merge-base. A test file is any .js, .jsx, .mjs, .cjs, .ts or .tsx file named *.test.* or *.spec.*, or inside a test, tests, __tests__, __mocks__, e2e or cypress directory. It parses both versions with Babel and matches tests by their full name (describe > test). Repeated names stay distinct (name #2). Newly added test files are not checked, since there is nothing to weaken.

You need the history: in CI use fetch-depth: 0 and a base ref such as origin/main that exists locally. An invalid ref or a failing git command exits 2.

What does it catch?

Errors make the gate fail. Review items are printed with a ? and pass unless you use --strict:

npx quell gate --diff main --strict

Gate failed: 1 issue(s) in 1 changed test file(s)
  ? tests/pay.test.js :: charge subtracts the amount — expected value changed in assertion 1 ([30,100,70] -> [30,100,71])

Renamed tests are paired first by their assertions, so renaming a test or moving a test file (git reports a rename) is not a deletion. expect.assertions(n), expect.any(...), asymmetric matchers and t.mock/t.plan are not counted as assertions.

How do I allow an intentional change?

Put // quell-allow-weaken <reason> anywhere inside the test body (a comment on any line of the test). The test is exempt from the weakening checks (skip, .only, removed and loosened assertions, changed values). Use it when you deliberately replace an assertion, and write the reason for the reviewer.

test('charge rejects zero', () => {
  // quell-allow-weaken: covered by the property test in charge.prop.test.js
  expect(() => charge(0, 100)).toThrow();
});

What output formats exist?

Upload SARIF with github/codeql-action/upload-sarif for inline annotations; see the CI guide.

What does it not catch?

• A test rewritten from scratch under a new name with different, weaker assertions, unless it shares assertions or a subject with the deleted one.
• Weakening done by changing the source instead of the test.
• Logic hidden in helper functions that the test calls.
• Weakness in newly added test files: use quell verify for those.

The gate checks that a change did not make existing tests weaker. It does not say whether the remaining tests are good: that is what verify is for.

How do I run it in a hook or CI?

npx quell gate --diff origin/main || exit 2

Exit code 5 means weakened tests; exit 2 means a usage or git error. For a Claude Code Stop hook, see MCP server and hooks.