Since late 2025 I have been running coding agents against a fleet of my own applications. I spent a lot of time fixing problems that turned out to be in their reports. An agent would file a bug that didn’t exist, or a feature board would show a pass from before the code changed. It took me a while to start checking those reports as carefully as the code.

Five false reports

On April 14, 2026 I ran the second eleven-phase test-find-fix pipeline across seven repositories: Spernakit v3.1.24 and six applications derived from it. Seven parallel Claude Code agents used Claude Opus 4.6 with its 1M-context configuration, the /spernakit-tester instructions from that day, and a Puppeteer-based browser CLI. The testers filed twelve bug reports across six of the seven repositories; the seventh produced nothing to ingest.

Five of the twelve described behavior that did not happen.

They were caught because an analysis step sat between filing and fixing. One analysis agent logged into the running server, captured the session’s security token, and used curl to hit three of the endpoints in question directly. The server was returning correct responses in every case. The testers had misread what they were looking at in a browser and filed against their misreading.

A separate case in a second pipeline that evening involved a chore-management app. A tester reported that clicking Done on a chore assignment deleted the chore item, high severity. A triage agent accepted it and filed a remediation pointing at a suspected miswired button. During review, a different agent opened the file and found that the Done button’s handler did not call the delete logic. The likeliest explanation is that the tester clicked the trash icon sitting next to it, which is, correctly, a delete button. Without that middle step I would have committed a “fix” that broke working code to resolve a bug that never existed.

Twelve bug reports filed by testers; verification against the running server confirmed seven and rejected five filed by testers 12 each claim re-checked against the running server held up 7 false positives 5
Twelve reports, one run. The five on the bottom right are the ones that would have become “fixes” to code that was working.

In that April 14 configuration, five of twelve reports, 42%, were false positives. That is not a general false-positive rate for agent testing. Change the model, tester instructions, scenarios, or reproduction rule and the number can change. Seven of those twelve reports were real, and I would not have found all seven by hand. I kept using the testers with that verification step in place.

Re-checking reports

I use a separate agent to re-check reports, with a tool that can give it independent evidence.

For a report filed from the browser, that might mean hitting the API directly, reading the handler, or checking the database. Repeating the same browser actions can reproduce the tester’s mistake as easily as the bug, so I want another way to establish what happened.

The verifier needs three things:

  • Access to the running system, not just the source. Three of the five false positives above were only detectable by asking the server what it actually returned.
  • Permission to reject or leave unverified. Ask the verifier to check the claim, not confirm it. Reject reports contradicted by the evidence. Leave a claim unverified when the evidence is missing or ambiguous.
  • A saved result. A rejected or unverified report should stay visible with its evidence and reason. You want to be able to notice that one tester profile produces four times the false positives of another.
A stored audit finding marked for removal, showing the original claim, verifier evidence, and the UNNECESSARY reason together.
The saved claim, evidence, and rejection reason show why I removed this finding.

I later added a live-reproduction rule to the testers themselves. Before filing, the agent has to produce the failure again against the running system and attach the steps and results. A request, response, and the requirement it contradicts give the reviewer something to work with. This helped more than the separate verification step did. The tester already has the browser open and the server running; asking it to check again is much cheaper than reconstructing the session later.

Stale results and invalid statuses

On February 20, 2026, removing module-level seed data from 20 backend services in Deeper, my SQL Server estate application, caused 25 features to regress. They had been returning hardcoded arrays instead of exercising the integration work those features needed. Remediation and re-verification brought the project back to 306 of 318 features passing.

A passing result needs context: which revision was tested, what data it used, and what the check actually exercised. My coding-agent orchestrator, aidd, does not automatically expire a stored passing status when the code changes.

The related failure is vocabulary drift. An audit prompt of mine once taught agents to write the status done, where the canonical status in my system is completed. Feature badges started rendering gray, so I shipped a quick fix aliasing done to green, and reverted it the next release, because the badge was right and the data was wrong. The real fix went in three places: the prompt that taught the bad word, the feature store, which now refuses non-canonical statuses outright, and the UI, which now renders invalid data red instead of quietly passing it as green.

Before widening a validator, check whether the agent’s output is actually valid. I had made the badge look right without fixing the status data, a fairly good example of Goodhart’s law.

Keep running the suite after fixes

The second pipeline that evening included a heavy test suite run against my orchestrator. A Docker registry timeout interrupted it, an environmental failure. A retry exposed a stale remote image overwriting my local build. Later runs found three application bugs: a feature toggle’s backend wrote to the database without updating the in-memory cache; a crawler exemption logged every request until SQLite crashed with a disk I/O error; and three global coordinator endpoints incorrectly required a workspace header.

Once those four problems were fixed, the suite passed clean: 38 routes crawled, zero failures. If I had stopped after the first retry, which felt entirely reasonable at the time, the three application bugs would have shipped.

In the April 17-19 fleet sweep, the first four upgrade cycles produced more than fifty findings between them; the fifth found none. I kept testing after fixes because each batch changed what the next run could exercise. A clean result still needs to be read alongside what the suite actually ran.

This is also why I keep separate light and heavy suites and stopped treating them as interchangeable. A latent typo in one backend, a file calling a function it never imported, survived the light suite for weeks because smoke-test caching masked it, then died instantly under the full suite. I need the full suite before calling the work verified. aidd’s current coding prompt commits after fast checks and runs full smoke:qc afterward, so a commit alone does not establish that result.

If you want the general version of the light/heavy tradeoff, Martin Fowler’s test pyramid is still the clearest statement of it. Agents produce a broad, shallow layer quickly, so it is easy to invert that pyramid without noticing.

Proofs on code I did not write

Every project my orchestrator had ever touched was a project I created, with a contract I wrote, and that bothered me more the longer I looked at it. So before making the tool public I ran four adversarial demonstrations, one per creation lane, across Bun/React and Python/Flask projects, with one hard rule: every failure becomes a filed backlog item, never a silent manual fix.

The one I cared about most was flaskr, the canonical Flask tutorial app, in Python, a stack the tool shares nothing with. It correctly identified Python, Flask, and pytest and produced security findings with file and line references. Onboarding also updated the app’s README.rst, so its writes were not confined to the metadata directory. After intake, it fixed security findings in idiomatic Flask, gated by the app’s own pytest suite. I hadn’t planned that extra work.

The hardest by a distance was an SMB infrastructure dashboard built from my own template. Every gate step went green eventually, but it took roughly 14 hours, with the creation step alone needing seven attempts. The other two, a habit tracker scaffolded fresh and a kanban board built from a third-party Vite starter, needed less work. They exposed a Windows path-mangling bug in template argument substitution, a conceptual gap where the create-then-ingest lane orphaned the original spec, and a product gap where a backlog outliving one run’s time budget needs a human to press launch again.

Several failures came from the tool assuming it had scaffolded every project itself. It dumped Bun-flavored scaffold files into the Flask app’s root. Its default directives told an ingested Python codebase to be “like my template,” forcing a keep-or-port decision onto an app that had not asked. My favorite: it tripped its own dirty-tree safety gate on its own bookkeeping files.

Eleven backlog items came out of three days. Running on a clean virtual machine against code I hadn’t written exposed assumptions I had missed in my own projects.

I required every failure during a proof to be filed against the release and worked like any other item. Fixing one by hand during the demonstration would have hidden how much help the tool needed. The filed items let me count the failures and check that each was dealt with.

I use the same process for ordinary audits. Each pass has a focus: security, performance, dead code, accessibility, and so on. Every finding becomes a backlog item in the same format as a feature, then gets worked, verified, and closed like anything else. One security pass over a database admin panel caught SQL injection through unvalidated column names that I had missed in review. The OWASP Top 10:2025 is a useful starting list for a security pass. Its expanded Software Supply Chain Failures category includes dependencies, build systems, and distribution machinery, so the review needs to cover more than application handlers.

Cost

The verification layer roughly doubles the token cost of a testing pass, and adds wall-clock time to every bug that turns out to be real. The live-reproduction rule makes testers slower. Checking changed code means repeating some verifications. The build proofs cost three days and produced no features at all.

I’d pay that cost again for these projects. I had already spent hours chasing false reports, and I wanted to find the setup assumptions before someone else tried the tool.

I added these checks as I ran into the failures. For a smaller project, I’d start with reproducing reports and recording which revision passed, then add more if the results called for it.

Where this came from

The original posts have more detail on these runs: