Since late 2025 I have been running coding agents against a fleet of my own applications. I spent a lot of time fixing problems that turned out to be in their reports. An agent would file a bug that didn’t exist, or a feature board would show a pass from before the code changed. It took me a while to start checking those reports as carefully as the code.

Five false reports

On April 14, 2026 I ran the second eleven-phase test-find-fix pipeline across seven repositories: Spernakit v3.1.24 and six applications derived from it. Seven parallel Claude Code agents used Claude Opus 4.6 with its 1M-context configuration, the /spernakit-tester instructions from that day, and a Puppeteer-based browser CLI. The testers filed twelve bug reports across six of the seven repositories; the seventh produced nothing to ingest.

Five of the twelve described behavior that did not happen.

They were caught because an analysis step sat between filing and fixing. One analysis agent logged into the running server, captured the session’s security token, and used curl to hit three of the endpoints in question directly. The server was returning correct responses in every case. The testers had misread what they were looking at in a browser and filed against their misreading.

The most instructive one was a chore-management app. A tester reported that clicking Done on a chore assignment deleted the chore item, high severity. A triage agent accepted it and filed a remediation pointing at a suspected miswired button. During review, a different agent opened the file, read the handler, and established that the Done button could not reach the delete path at all. The likeliest explanation is that the tester clicked the trash icon sitting next to it, which is, correctly, a delete button. Without that middle step I would have committed a “fix” that broke working code to resolve a bug that never existed.

Twelve bug reports filed by testers; verification against the running server confirmed seven and rejected five filed by testers 12 each claim re-checked against the running server held up 7 false positives 5
Twelve reports, one run. The five on the bottom right are the ones that would have become “fixes” to code that was working.

In that April 14 configuration, five of twelve reports, 42%, were false positives. That is not a general false-positive rate for agent testing. Change the model, tester instructions, scenarios, or reproduction rule and the number can change. Seven of those twelve reports were real, and I would not have found all seven by hand. I kept using the testers with that verification step in place.

Re-checking reports

I use a separate agent to re-check reports, with a tool that can give it independent evidence.

For a report filed from the browser, that might mean hitting the API directly, reading the handler, or checking the database. Repeating the same browser actions can reproduce the tester’s mistake as easily as the bug, so I want another way to establish what happened.

The verifier needs three things:

  • Access to the running system, not just the source. Three of the five false positives above were only detectable by asking the server what it actually returned.
  • Permission to reject or leave unverified. If the verifier’s incentive is to confirm, it will confirm. Contradictory evidence earns rejection; ambiguous or missing evidence leaves the claim unverified rather than turning absence of proof into proof of absence.
  • A saved result. A rejected or unverified report should stay visible with its evidence and reason. You want to be able to notice that one tester profile produces four times the false positives of another.
A stored audit finding marked for removal, showing the original claim, verifier evidence, and the UNNECESSARY reason together.
The saved claim, evidence, and rejection reason show why I removed this finding.

I later added a live-reproduction rule to the testers themselves. Before filing, the agent has to produce the failure again against the running system and attach the steps and results. A request, response, and the requirement it contradicts give the reviewer something to work with. This helped more than the separate verification step did. The tester already has the browser open and the server running; asking it to check again is much cheaper than reconstructing the session later.

Stale results and invalid statuses

In February 2026 I noticed the feature board was flattering me. Twenty-five features were marked passing on the strength of verification runs that predated the code they were sitting on. They had genuinely passed once, against implementations that had since been audited, refactored, or replaced.

So I flipped all twenty-five back to failing and made the loop re-earn every one against the current build. Most came back quickly. A handful did not, because their passing status had been describing behavior that no longer existed. I could not have told you in advance which was which, and neither could the board.

Since then I expire a passing result when the underlying code changes. I need to know which revision was tested before I can rely on it.

The related failure is vocabulary drift. An audit prompt of mine once taught agents to write the status done, where the canonical status in my system is completed. Feature badges started rendering gray, so I shipped a quick fix aliasing done to green, and reverted it the next release, because the badge was right and the data was wrong. The real fix went in three places: the prompt that taught the bad word, the feature store, which now refuses non-canonical statuses outright, and the UI, which now renders invalid data red instead of quietly passing it as green.

Before widening a validator, check whether the agent’s output is actually valid. I had made the badge look right without fixing the status data, a fairly good example of Goodhart’s law.

Run the suite until it stops finding things

The same April pipeline included a heavy test suite run against my orchestrator. The first attempt died on a Docker registry timeout, an environmental failure. The first retry exposed a stale remote image overwriting my local build. Later runs found three application bugs: a feature toggle’s backend wrote to the database without updating the in-memory cache; a crawler exemption logged every request until SQLite crashed with a disk I/O error; and three global coordinator endpoints incorrectly required a workspace header.

Once those four problems were fixed, the suite passed clean: 38 routes crawled, zero failures. If I had stopped after the first retry, which felt entirely reasonable at the time, the three application bugs would have shipped.

In the April 17-19 fleet sweep, the first of five upgrade cycles produced more than fifty findings; the fifth found none. I kept testing after fixes because each batch changed what the next run could exercise. A clean result still needs to be read alongside what the suite actually ran.

This is also why I keep separate light and heavy suites and stopped treating them as interchangeable. A latent typo in one backend, a file calling a function it never imported, survived the light suite for weeks because smoke-test caching masked it, then died instantly under the full suite. “Ready to commit” now means the full suite. That takes longer, but the light suite had missed a startup failure.

If you want the general version of the light/heavy tradeoff, Martin Fowler’s test pyramid is still the clearest statement of it. The AI-specific wrinkle is only that agents are very good at producing a broad, shallow layer quickly, which makes the pyramid easy to invert without noticing.

Prove it on code the tool has never seen

Everything above still leaves you inside your own habits. Every project my orchestrator had ever touched was a project I created, with a contract I wrote, and that bothered me more the longer I looked at it. So before making the tool public I ran four adversarial demonstrations, one per creation lane, each on a different stack, with one hard rule: every failure becomes a filed backlog item, never a silent manual fix.

The one I cared about most was flaskr, the canonical Flask tutorial app, in Python, a stack the tool shares nothing with. It inferred Python, Flask, and pytest with zero Bun or Node leakage, produced four security findings grounded to file and line, and held the guarantee I was actually testing for: every intake commit touched only its own metadata directory, with zero writes to the app’s own code. Then, unplanned, it fixed all four findings in idiomatic Flask, gated by the app’s own pytest suite.

The hardest by a distance was an SMB infrastructure dashboard built from my own template. Every gate step went green eventually, but it took roughly 14 hours, with the creation step alone needing seven attempts. The other two, a habit tracker scaffolded fresh and a kanban board built from a third-party Vite starter, needed less work. They exposed a Windows path-mangling bug in template argument substitution, a conceptual gap where the create-then-ingest lane orphaned the original spec, and a product gap where a backlog outliving one run’s time budget needs a human to press launch again.

Several failures came from the tool assuming it had scaffolded every project itself. It dumped Bun-flavored scaffold files into the Flask app’s root. Its default directives told an ingested Python codebase to be “like my template,” forcing a keep-or-port decision onto an app that had not asked. My favorite: it tripped its own dirty-tree safety gate on its own bookkeeping files.

Eleven backlog items came out of three days. Running on a clean virtual machine against code I hadn’t written exposed assumptions I had missed in my own projects.

I required every failure during a proof to be filed against the release and worked like any other item. Fixing one by hand during the demonstration would have hidden how much help the tool needed. The filed items let me count the failures and check that each was dealt with.

The same principle scaled down works well for ordinary audits. I run audit passes with distinct lenses, security and performance and dead code and accessibility and so on, and every finding is emitted as a backlog item in the same format as a feature, then worked, verified, and closed like anything else. One security pass over a database admin panel caught a genuine SQL injection through unvalidated column names, which is exactly the kind of thing eyeballing a diff does not catch and a dedicated lens does. If you need a starting list of lenses, the OWASP Top 10:2025 is a reasonable spine for the security one. Its expanded Software Supply Chain Failures category is also a reminder that the code under review includes dependencies, build systems, and distribution machinery, not just application handlers.

Cost

The verification layer roughly doubles the token cost of a testing pass, and adds wall-clock time to every bug that turns out to be real. The live-reproduction rule makes testers slower. Expiring green marks means re-running verifications you already ran. The build proofs cost three days and produced no features at all.

I’d pay that cost again for these projects. I had already spent hours chasing false reports, and I wanted to find the setup assumptions before someone else tried the tool.

I added these checks as I ran into the failures. For a smaller project, I’d start with reproducing reports and recording which revision passed, then add more if the results called for it.

Changelog

  • August 27, 2026: Corrected the repository count in the April 14 run and the number of false positives that needed the running server to detect.
  • August 15, 2026: Distinguished contradictory evidence, which can reject a claim, from missing or ambiguous evidence, which leaves it unverified.
  • August 5, 2026: Corrected the timeline and retry accounting, tied the 42% result to its tester configuration, and updated the security guidance for OWASP Top 10:2025.
  • August 3, 2026: First published.

Where this came from

The original posts have more detail on these runs: