I Let Claude Code Write All My Tests for a Week. Here's What It Got Wrong.

Last week I ran an experiment on myself. Every test in my payments service — a real codebase, real Stripe integration, real consequences — would be written by Claude Code. I wouldn't write a single assert by hand. I build a testing tool for a living, so this was either going to be humbling or validating. It was both.

What the agent nailed

Let me be fair first, because the good parts are genuinely good.

Claude Code is fast at scaffolding. Point it at a module and say "write tests for this" and thirty seconds later you have a tidy file: fixtures, parametrized cases, sensible names, docstrings on the test functions. It reads type hints properly. It notices Optional parameters and writes a None case. It saw my Decimal amounts and didn't once compare floats.

If your bar is "a test file exists and pytest is green," the experiment was over on day one. Total success.

Where it quietly failed

Day three is where it got interesting. I started injecting bugs on purpose — the kind of small, dumb regressions that actually ship. Flip a <= to <. Delete a guard clause. Swap a default.

Then I ran the AI-written suite against the broken code.

Roughly one in five of those broken versions passed the tests. Green checkmarks, broken code. And the failures clustered in a specific way:

1. Mirror tests. The agent read if amount <= 0: raise ValueError and wrote a test asserting exactly that — by re-deriving the assertion from the same code it was testing. When I flipped the boundary, it happily re-derived the flipped logic in its head and the old test still passed at -1 but never checked 0. The one input that mattered.

2. Over-mocked tests. Several tests mocked the exact function under test's collaborators so aggressively that the test asserted the mock, not the code. Deleting the entire function body and returning None passed two of them.

3. Vibes-based asserts. assert result is not None. assert isinstance(response, dict). These execute the code — coverage tools love them — and verify almost nothing.

None of this is a Claude problem specifically. I've seen humans write all three patterns. The difference is throughput: an agent writes them at 50x the speed, with confident naming, and they look excellent in review.

The question nobody asks in code review

Here's the thing that changed how I think about this. In review, we ask "do the tests pass?" We almost never ask the inverse: "if the code were broken, would this test fail?"

That inverse question is checkable. Mechanically. You inject the violation the test claims to catch, run the test, and demand a red result. The research community has called versions of this mutation testing for decades; I think of it as making the test earn its green.

When I ran that check on the week's output, the suite shrank. A bunch of confident-looking tests turned out to be decorative. The ones that survived were genuinely load-bearing — and interestingly, they were mostly the tests tied to documented behavior: docstring contracts, Pydantic constraints, explicit guard clauses. The specs that already existed in the code.

What I actually changed

I didn't stop letting Claude write tests. That would be the wrong lesson — the speed is real. What changed is that nothing gets committed on green alone anymore. Every generated test has to fail against a violated version of the code before it earns a place in the suite.

This is, transparently, the thesis of Quelltest — it's Gate 5 of the five-gate pipeline, and yes, I built the tool because of weeks exactly like this one. But you don't need my tool to apply the idea. Break your code on purpose. Run your tests. If they stay green, you don't have tests — you have documentation that lies.

The agents are going to keep writing more of our code. Fine. Then the one thing we should hold sacred is the machinery that proves the code does what it claims. That part, I'm not willing to vibe.