Back in April, five of twelve bug reports turned out to be wrong. I got more careful about reviewing findings after that. I still didn’t have a repeatable way to tell whether changing an audit had improved it.
On August 24 I added a small test project with two deliberate security defects and one harmless lookalike. The security audit runs against it and files findings the usual way. A scorer then checks what it found and where it pointed.
The location is part of the test. At first, a finding needed to name the planted file and match the defect wording. That still allowed a finding elsewhere in the file to get credit. Matching needed to be more precise. The harmless code needs to stay harmless in the report too. An audit that flags everything would find the planted bugs.
The first run, using gpt-5.5, found both defects and left the decoy alone. That gave it perfect
precision and recall on this particular fixture. It was a very small fixture, so I wasn’t about
to treat the result as a rating for the whole audit catalog.
Another model attempt returned a provider error because the account didn’t support it. I kept that attempt in the evidence and left it out of the measured set. It hadn’t reviewed the code.
I also tied each passing result to the audit text and the eval manifest, including the model configuration, scoring rules, and defect catalog. Changing the recorded inputs makes the old measurement stale. Otherwise I could edit the audit until it was doing something quite different and still be pointing at the earlier result.
The following work was about the findings from actual projects. Until then, removing a dismissed finding also removed useful information about the audit that filed it. aidd gained a local history for findings, identified consistently across runs, so it can record a duplicate or a problem that comes back after being fixed.
Dismissing one now requires a reason. The history entry is written before the finding is removed; if that write fails, the finding stays. That ordering is a small detail I wanted to keep, because a failed attempt to improve the records shouldn’t leave me with fewer records.
By the next day I could start looking at acceptance and false-positive rates, and the cost per accepted finding. Runs also carry the revision of the instructions that drove them, so changing a skill doesn’t mix its new results indistinguishably with the old ones.
I didn’t have enough history yet to choose audits from those numbers. The first useful result was the small test I could rerun. The project history would take longer to become useful.
Later update, August 26, 2026: I tightened the scorer to require symbol-or-line matching and added the fixture code itself to the recorded hashes. A finding elsewhere in the same file no longer earns credit for the planted defect, and changing the fixture makes the old result stale.