# Did the AI Weaken Your Tests? Catching Deleted, Skipped and Loosened Tests With quell gate

> quell gate compares test files with their merge-base version and fails on deleted, skipped, .only and loosened tests. A real diff and its real output.

Published: 2026-10-20 | Author: Shashank Bindal | Tags: npm, ai agents, code review, ci, javascript testing

Canonical page: https://quelltest.com/blog/did-the-ai-weaken-your-tests-quell-gate

**TL;DR.** `quell gate --diff main` compares every changed test file with the version at the merge-base and fails with exit code 5 if tests were deleted, skipped, narrowed with `.only`, or had their assertions loosened. We built a throwaway repository where the suite stayed green after an "agent" removed two tests, skipped one and softened one. The gate flagged all four.

Captured with `quelltest@0.3.1` from npm on Windows, Node.js 24.11.0, `node:test`.

## Why would a change weaken tests?

An agent with a failing test and a task to finish has two ways out: fix the code, or change the test. Changing the test is often a legitimate edit. It is also the cheapest way to turn a red build green, and a diff of a few hundred lines hides it well.

`gate` does not judge intent. It reports test-file changes that reduce what the suite can catch, so a human or a CI rule decides whether each one is justified.

## What does a weakened change look like?

The baseline test file on `main` had five tests for a `applyDiscount` function. On a branch, the file became this:

```diff
 test("applies a percentage discount", () => {
-  assert.equal(applyDiscount(200, 10), 180);
+  assert.ok(applyDiscount(200, 10));
 });

-test("caps the discount at 50 percent", () => {
+test.skip("caps the discount at 50 percent", () => {
   assert.equal(applyDiscount(100, 90), 50);
 });

-test("rejects negative prices", () => {
-  assert.throws(() => applyDiscount(-1, 10), RangeError);
-});
-
 test("treats a missing percent as no discount", () => {
   assert.equal(applyDiscount(100), 100);
 });
-
-test("accepts a price of exactly zero", () => {
-  assert.equal(applyDiscount(0, 10), 0);
-});
```

`npm test` on that branch:

```text
ℹ tests 3
ℹ pass 2
ℹ fail 0
ℹ skipped 1
```

Green. Now the gate:

```bash
npm install --save-dev quelltest
npx quell gate --diff main
```

```text
Gate failed: 4 issue(s) in 1 changed test file(s)
  ✖ tests/discount.test.js :: rejects negative prices — test deleted
  ✖ tests/discount.test.js :: accepts a price of exactly zero — test deleted
  ✖ tests/discount.test.js :: applies a percentage discount — assertions weakened (0 -> 1 weak, e.g. equality replaced by ok/truthy/not-null)
  ✖ tests/discount.test.js :: caps the discount at 50 percent — test is now skipped (skip/todo/x-prefixed)
exit=5
```

For machines, `--json` gives each issue a `kind` (`test-deleted`, `assertions-weakened`, `test-skipped`), a `severity`, a test name and a line. `--sarif` produces SARIF 2.1.0 for GitHub code scanning.

## What does the gate catch?

- Deleted tests and deleted test files, and a test deleted and then rewritten (shown as a replacement).
- Skipped tests: `skip`, `todo`, `x`-prefixed names and `{ skip: true }`.
- A newly added `.only` or `describe.only`, which silently skips every other test.
- Removed or reduced assertions.
- Equality loosened to `ok`, truthy, defined or not-null.
- Changed expected values, as a review item.

Git renames are handled, and an unparsable test file is an error, not a pass.

## What happens when the expected value changes, or `.only` appears?

On another branch we changed `50` to `10` in one expectation. That is sometimes a legitimate fix and sometimes not, so by default it is a review item and the gate passes:

```text
Gate passed: 1 changed test file(s) checked, no weakened tests
  ? tests/discount.test.js :: caps the discount at 50 percent — expected value changed in assertion 1 ([100,90,50] -> [100,90,10])
exit=0
```

With `--strict`, the same change fails:

```text
Gate failed: 1 issue(s) in 1 changed test file(s)
  ? tests/discount.test.js :: caps the discount at 50 percent — expected value changed in assertion 1 ([100,90,50] -> [100,90,10])
exit=5
```

Adding `.only` to a test is always an error:

```text
  ✖ tests/discount.test.js :: accepts a price of exactly zero — `.only` silently skips every other test
```

## How do I allow a deliberate change?

Sometimes a test should get weaker: a requirement was dropped. Say so in the test, with a reason, so the exception is visible in review:

```js
test("accepts a price of exactly zero", () => {
  // quell-allow-weaken zero price now handled by checkout, see ADR 41
  assert.ok(applyDiscount(0, 10));
});
```

Without the comment the gate fails on `assert.ok` replacing `assert.equal`. With it:

```text
Gate passed: 1 changed test file(s) checked, no weakened tests
```

## Can I also check that the remaining tests still catch faults?

Yes, `gate` and `verify` answer different questions. After the weakening above, we ran the changed test file through `verify` with a floor of 60:

```bash
npx quell verify tests/discount.test.js --source src/discount.js --min-score 60
```

```text
Not verified: mutation score 29% is below the required 60% [copy], 1 ignored
  survived: guard removed (line 2)
  survived: throw removed (line 3)
  survived: || -> && (line 2)
  survived: < -> <= (line 2)
  survived: - -> + (line 7)
```

On `main`, the same command printed `Verified: test killed 7/7 mutants (100%) [copy], 1 ignored`. The "1 ignored" is an equivalent mutant marked in the source, which [the verify post](https://quelltest.com/blog/does-your-test-fail-when-the-code-is-wrong.md) explains. The score fell from 100% to 29%. The gate told us what changed; `verify` told us what the suite can no longer see.

## How do I run it in CI or in an agent loop?

```yaml
- uses: actions/checkout@v4
  with: { fetch-depth: 0 }
- uses: actions/setup-node@v4
  with: { node-version: 22 }
- run: npm ci
- run: npx quell gate --diff "origin/${{ github.base_ref }}"
```

`fetch-depth: 0` matters because the gate needs the merge-base. For agents, the same command can run in a Claude Code `Stop` hook; see [the MCP and hook post](https://quelltest.com/blog/quelltest-mcp-server-and-stop-hook-for-coding-agents.md). More recipes are in [CI integration](https://quelltest.com/docs/npm/ci.md) and the [gate reference](https://quelltest.com/docs/npm/gate.md).

## What are the limits?

- A test rewritten from scratch under a new name with different assertions is not caught.
- Weakening done by changing the source instead of the tests is not caught. That is `verify`'s job.
- Logic inside helper functions that the test calls is not analysed.
- The gate looks at the shape of assertions. In our run, replacing `assert.equal(x, 0)` with `assert.ok(x === 0)` raised no issue, which is reasonable because the strength is the same, but it shows the check is syntactic.
- Changed expected values are review items by default, because the tool cannot know whether the new value is the correct one.
- Exit code 5 is the gate's. Your CI must treat any non-zero exit as failure, which most do.

Read the [limits page](https://quelltest.com/docs/npm/limits.md) and the [changelog](https://quelltest.com/changelog).

## FAQ

### What does `--diff main` compare against?

The merge-base between your working tree and the ref you pass. Use `origin/main` in CI.

### Does the gate run my tests?

No. It reads files and their git history. `verify` runs tests.

### Does it work with Jest and Vitest?

The gate does not need a runner: it reads test files and git history. The demo here used `node:test`. We did not run the same diff through Jest, Vitest or Mocha, so check your own suite's shape on a scratch branch first.

### Can an agent bypass it by adding `quell-allow-weaken`?

Yes, and it will show up in the diff for a reviewer, with a reason. That is the design: the exception is explicit, not silent.

## Sources

- quelltest README, gate behavior and exit codes: [github.com/quelltest/quelltest-npm](https://github.com/quelltest/quelltest-npm)
- Agents and CI recipes, including what the gate does and does not catch: [docs/agents.md](https://github.com/quelltest/quelltest-npm/blob/main/docs/agents.md)
- GitHub Actions `checkout` and `fetch-depth`: [github.com/actions/checkout](https://github.com/actions/checkout)
- SARIF upload for code scanning: [GitHub documentation](https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/uploading-a-sarif-file-to-github)