Skip to content

Three-bucket output: written, scaffolded, flagged

Every edge case the Python package finds ends in exactly one of three buckets: WRITTEN (a verified test was written), SCAFFOLDED (a stub was written for you to finish) or FLAGGED (it could not be auto-tested, with a reason). Nothing is dropped silently. Read this page to interpret the summary at the end of quell find and the quell-report.json file.

What does each bucket mean?

BucketMeaning
WRITTENA test was generated and passed the gates: it parses, is original, is secure, passes on the original code and fails when the requirement is violated. It was written to your test file with libcst. Each written test carries a confidence score from 0 to 100 (see the score).
SCAFFOLDEDNo deterministic, verifiable test could be built, so a stub with correct imports and a # quell: complete assertion placeholder was appended to tests/scaffold/test_<module>.py (the scaffold_dir setting). The directory is added to .gitignore by quell init.
FLAGGEDDocumented gap with a reason and the source location. The summary prints file:line and the reason.

Gaps that need a human assertion are stubs; gaps that cannot be tested at all are flags. The reasons defined in the model for flagged items are: depends on external API, depends on environment variable, depends on object state, local variable guard, async function, generator produced invalid syntax, test failed on correct code (likely false positive), test passed even with bug injected (would not catch it), generated test failed security review, duplicate of existing test, assertion too weak, too similar to existing test.

How does quell find assign outcomes?

For each gap, quell find --fix records an outcome in quell-report.json (under results) and the summary groups them:

Outcome in the reportBucket in the summary
verified (test written)WRITTEN
skipped_no_rule, skipped_local_var, skipped_async, skipped_no_genSCAFFOLDED
rejected_no_catch, rejected_fails_on_correct, detected_not_fixed, framework skipsFLAGGED

detected_not_fixed is what a plain quell find (without --fix) records for every gap it only detected.

Known quirk in the version tested (repository 2.0.1): the summary printed at the end of quell find --fix listed a test that had just been written under FLAGGED with the reason verified, and WRITTEN (0). The test was written to disk and recorded as verified in quell-report.json; only the summary grouping was off. Check quell-report.json and your test file when in doubt.

What does a scaffold look like?

A scaffold file starts with a header and one stub per requirement, with a docstring that says how far the test got through the gates:

# quell: scaffold — complete the assertions below and move to your test suite
from __future__ import annotations

import pytest

Move finished tests into your normal test suite.

Why not just report coverage?

Coverage tools report which lines ran. A test that calls a function and asserts result is not None can reach 100% line coverage while catching no bug. The buckets tell you which requirements are proven by a test that fails when the code is wrong, which need your help, and which are documented gaps.