The five-gate verification pipeline
Every test the Python package generates must pass five gates before it is written to disk. Gates 4 and 5 are the core idea: the test must pass on your real code and must fail when the requirement it tests is deliberately violated. A test that cannot fail is never written. This page describes each gate and what happens when one fails.
What are the gates?
| Gate | Name | What it checks | If it fails |
|---|---|---|---|
| 1 | AST valid | The test parses, its top-level imports resolve in the current environment, and there are no obviously undefined names. | Retried once, then flagged. |
| 2 | Original | The test name does not already exist, its AST fingerprint is not a duplicate, its only assertion is not the boilerplate assert result is not None, and its 5-gram token overlap with existing tests is at most 80%. | The test is silently not written. |
| 3 | Secure | No hardcoded credentials, no eval/exec/os.system/os.popen/subprocess.call, no unmocked network calls, no file writes outside tmp_path, no os.environ mutation. | Flagged: generated test failed security review. |
| 4 | Passes correct | The test runs against the original source and passes. | Rejected as a likely false positive (rejected_fails_on_correct). |
| 5 | Fails violated | The test runs against a violated source and fails. | Rejected: it would not catch the bug (rejected_no_catch). |
How does the verifier work?
The verifier follows a fixed algorithm:
- Write the candidate test to a temporary file, not to your real test file.
- Run it on the original source with pytest in a subprocess (never in-process, because module caches would hide changes). It must pass.
- Inject a violation into the source for the requirement's constraint kind.
- Run it again. It must fail.
- Always restore the source in a
finallyblock, even if the run crashes or times out.
There is deliberately no flag to skip verification.
How is the violation injected?
| Constraint kind | Violation, applied inside the target function |
|---|---|
must_raise, not_null, enum_valid, type_check, auth_check, magic_value, custom | The raise statement is replaced with pass # QUELL_VIOLATION, so the guard no longer fires. |
boundary | A numeric literal after a comparison operator is replaced with -9999, weakening the threshold (> 0 becomes > -9999). |
must_return | Every non-None return is replaced with return None. |
silent_fail | The first silent return None (or bare return) becomes raise ValueError("quell_violation"). |
mutation | mutmut apply is run (mutation integration is off by default). |
bug_repro | Nothing: the source is already considered broken. |
# Gate 4: original code # Gate 5: violation injected (must_raise)
def charge(amount, balance): def charge(amount, balance):
if amount <= 0: if amount <= 0:
raise ValueError(...) pass # QUELL_VIOLATION
return balance - amount return balance - amount
# the test passes # the test must fail
What happens to a test that fails a gate?
It is not written. The gap becomes SCAFFOLDED or FLAGGED with a reason, recorded in quell-report.json. See three-bucket output.
How is this different from the npm package?
The npm package does not inject a violation of one requirement. It applies a small set of generic and risk-aimed source mutants and requires a test to fail on at least one, in a throwaway copy of your project. See how verify works.