# I Let Claude Code Write All My Tests for a Week. Here's What It Got Wrong.

> Field notes from a week of fully agentic test writing — what AI test generation nails, where it quietly fails, and why 'the tests pass' is the least interesting thing a test can tell you.

Published: 2026-07-15 | Author: Shashank Bindal | Tags: ai coding, claude code, test generation, agentic development, python testing

Canonical page: https://quelltest.com/blog/i-let-claude-code-write-my-tests-for-a-week

Last week I ran an experiment on myself. Every test in my payments service —
a real codebase, real Stripe integration, real consequences — would be
written by Claude Code. I wouldn't write a single `assert` by hand. I build
a testing tool for a living, so this was either going to be humbling or
validating. It was both.

## What the agent nailed

Let me be fair first, because the good parts are genuinely good.

Claude Code is *fast* at scaffolding. Point it at a module and say "write
tests for this" and thirty seconds later you have a tidy file: fixtures,
parametrized cases, sensible names, docstrings on the test functions. It
reads type hints properly. It notices `Optional` parameters and writes a
`None` case. It saw my `Decimal` amounts and didn't once compare floats.

If your bar is "a test file exists and pytest is green," the experiment was
over on day one. Total success.

## Where it quietly failed

Day three is where it got interesting. I started injecting bugs on purpose —
the kind of small, dumb regressions that actually ship. Flip a `<=` to `<`.
Delete a guard clause. Swap a default.

Then I ran the AI-written suite against the broken code.

**Roughly one in five of those broken versions passed the tests.** Green
checkmarks, broken code. And the failures clustered in a specific way:

1. **Mirror tests.** The agent read `if amount <= 0: raise ValueError` and
   wrote a test asserting exactly that — by re-deriving the assertion from
   the same code it was testing. When I flipped the boundary, it happily
   re-derived the flipped logic in its head and the old test still passed at
   `-1` but never checked `0`. The one input that mattered.

2. **Over-mocked tests.** Several tests mocked the exact function under
   test's collaborators so aggressively that the test asserted the mock,
   not the code. Deleting the entire function body and returning `None`
   passed two of them.

3. **Vibes-based asserts.** `assert result is not None`. `assert
   isinstance(response, dict)`. These execute the code — coverage tools
   love them — and verify almost nothing.

None of this is a Claude problem specifically. I've seen humans write all
three patterns. The difference is throughput: an agent writes them at 50x
the speed, with confident naming, and they *look* excellent in review.

## The question nobody asks in code review

Here's the thing that changed how I think about this. In review, we ask
"do the tests pass?" We almost never ask the inverse: **"if the code were
broken, would this test fail?"**

That inverse question is checkable. Mechanically. You inject the violation
the test claims to catch, run the test, and demand a red result. The
research community has called versions of this mutation testing for
decades; I think of it as making the test *earn* its green.

When I ran that check on the week's output, the suite shrank. A bunch of
confident-looking tests turned out to be decorative. The ones that survived
were genuinely load-bearing — and interestingly, they were mostly the tests
tied to *documented* behavior: docstring contracts, Pydantic constraints,
explicit guard clauses. The specs that already existed in the code.

## What I actually changed

I didn't stop letting Claude write tests. That would be the wrong lesson —
the speed is real. What changed is that nothing gets committed on green
alone anymore. Every generated test has to fail against a violated version
of the code before it earns a place in the suite.

This is, transparently, the thesis of [Quelltest](https://quelltest.com) —
it's Gate 5 of the [five-gate pipeline](https://quelltest.com/docs/reference/gates.md), and yes, I
built the tool because of weeks exactly like this one. But you don't need
my tool to apply the idea. Break your code on purpose. Run your tests.
If they stay green, you don't have tests — you have documentation that
lies.

The agents are going to keep writing more of our code. Fine. Then the one
thing we should hold sacred is the machinery that proves the code does what
it claims. That part, I'm not willing to vibe.
