Skip to content

The five-gate verification pipeline

Every test the Python package generates must pass five gates before it is written to disk. Gates 4 and 5 are the core idea: the test must pass on your real code and must fail when the requirement it tests is deliberately violated. A test that cannot fail is never written. This page describes each gate and what happens when one fails.

What are the gates?

GateNameWhat it checksIf it fails
1AST validThe test parses, its top-level imports resolve in the current environment, and there are no obviously undefined names.Retried once, then flagged.
2OriginalThe test name does not already exist, its AST fingerprint is not a duplicate, its only assertion is not the boilerplate assert result is not None, and its 5-gram token overlap with existing tests is at most 80%.The test is silently not written.
3SecureNo hardcoded credentials, no eval/exec/os.system/os.popen/subprocess.call, no unmocked network calls, no file writes outside tmp_path, no os.environ mutation.Flagged: generated test failed security review.
4Passes correctThe test runs against the original source and passes.Rejected as a likely false positive (rejected_fails_on_correct).
5Fails violatedThe test runs against a violated source and fails.Rejected: it would not catch the bug (rejected_no_catch).

How does the verifier work?

The verifier follows a fixed algorithm:

  1. Write the candidate test to a temporary file, not to your real test file.
  2. Run it on the original source with pytest in a subprocess (never in-process, because module caches would hide changes). It must pass.
  3. Inject a violation into the source for the requirement's constraint kind.
  4. Run it again. It must fail.
  5. Always restore the source in a finally block, even if the run crashes or times out.

There is deliberately no flag to skip verification.

How is the violation injected?

Constraint kindViolation, applied inside the target function
must_raise, not_null, enum_valid, type_check, auth_check, magic_value, customThe raise statement is replaced with pass # QUELL_VIOLATION, so the guard no longer fires.
boundaryA numeric literal after a comparison operator is replaced with -9999, weakening the threshold (> 0 becomes > -9999).
must_returnEvery non-None return is replaced with return None.
silent_failThe first silent return None (or bare return) becomes raise ValueError("quell_violation").
mutationmutmut apply is run (mutation integration is off by default).
bug_reproNothing: the source is already considered broken.
# Gate 4: original code           # Gate 5: violation injected (must_raise)
def charge(amount, balance):      def charge(amount, balance):
    if amount <= 0:                   if amount <= 0:
        raise ValueError(...)             pass  # QUELL_VIOLATION
    return balance - amount           return balance - amount
# the test passes                 # the test must fail

What happens to a test that fails a gate?

It is not written. The gap becomes SCAFFOLDED or FLAGGED with a reason, recorded in quell-report.json. See three-bucket output.

How is this different from the npm package?

The npm package does not inject a violation of one requirement. It applies a small set of generic and risk-aimed source mutants and requires a test to fail on at least one, in a throwaway copy of your project. See how verify works.