The Test Passed, Then Deleted Its Evidence

I had one check pass after examining zero files, and another test keep running after its log disappeared. Both on the same day.

The first was the circular-dependency section of an aidd hygiene audit. It called Madge, Madge exited zero, and the audit reported no cycles. Except the command was not configured to follow the repository’s TypeScript files, so “no cycles” actually meant “no input.” It had processed zero files and treated the empty result as proof. Once the command used the repository’s TypeScript configuration and extensions, it processed 1,029 files and found 12 dependency cycles immediately.

I had already written Green Is Not Evidence after finding checks that measured the wrong thing. Here was another one.

The audit now requires every Madge run to report how many files it processed, and zero files fails the check even when the command exits successfully. After splitting the cycles through smaller leaf contracts, the same check examined about a thousand files and reported zero cycles.

The missing transcript

The second one appeared during a Spernakit upgrade pass across ten of the derived apps. I was redirecting long test runs into logs/supertest.log so I could keep the complete transcript while watching only the short result in the terminal.

Spernakit also had a cleanup script which removed every .log and .pid file from that directory, and reset and screenshot runs call it near the beginning. The script knew the directory but did not know which files belonged to it. On Windows it successfully unlinked the open transcript while the test process still held its handle. The writer kept writing and reported no problem. The filename was gone, so nothing else could open what it was writing. Three applications lost their test transcript that way in one afternoon before I worked out what was happening, and what made it hard to see is that the clean directory looked like success and so did the running process.

The replacement script removes only the runtime files the repository creates: backend and frontend logs, error logs, process IDs, their rotated forms, and files explicitly named by the smoke runbook. Everything else stays. The script also names the files it refused to delete, so an operator can see that cleanup ran into something it did not own. If the runbook cannot be parsed, the cleanup does not broaden its guess - it keeps undeclared files and lets the smoke runner report the bad configuration. There is a fixture-tree regression proving the old version would delete the foreign transcript and the new one will not.