Covers August 24 - 25, 2026.
Back in April, five of twelve bug reports turned out to be wrong. I got more careful about reviewing findings after that. I still didn’t have a repeatable way to tell whether changing an audit had improved it.
On August 24 I added a small test project with two deliberate security defects and one harmless lookalike. The security audit runs against it and files findings the usual way. A scorer then checks what it found and where it pointed.
The location is part of the test. A finding about a problem somewhere else in the file shouldn’t get credit just because it uses the right words. And the harmless code needs to stay harmless in the report. An audit that flags everything would find the planted bugs too.
The first run, using gpt-5.5, found both defects and left the decoy alone. That gave it perfect
precision and recall on this particular fixture. It was a very small fixture, so I wasn’t about
to treat the result as a rating for the whole audit catalog.
Another model attempt returned a provider error because the account didn’t support it. I kept that attempt in the evidence and left it out of the measured set. It hadn’t reviewed the code.
I also tied each passing result to the inputs that produced it: the audit text, the fixture and model configuration, plus the scoring rules. Changing those makes the old measurement stale. Otherwise I could edit the audit until it was doing something quite different and still be pointing at the earlier result.
The following work was about the findings from actual projects. Until then, removing a dismissed finding also removed useful information about the audit that filed it. aidd gained a local history for findings, identified consistently across runs, so it can record a duplicate or a problem that comes back after being fixed.
Dismissing one now requires a reason. The history entry is written before the finding is removed; if that write fails, the finding stays. That ordering is a small detail I wanted to keep, because a failed attempt to improve the records shouldn’t leave me with fewer records.
By the next day I could start looking at acceptance and false-positive rates, and the cost per accepted finding. Runs also carry the revision of the instructions that drove them, so changing a skill doesn’t mix its new results indistinguishably with the old ones.
I didn’t have enough history yet to choose audits from those numbers. The first useful result was the small test I could rerun. The project history would take longer to become useful.