The Run Failed. The Result Didn’t.

An aidd batch audit produced 43 complete reports and emitted them in its final result. Then the command-line process died in the few seconds before it wrote any of them to the project.

The web panel did roughly what it was supposed to do with a dead process. The heartbeat went stale, the run moved to failed, the stored exit code became -1. It even found the final result sitting in the run log. Then it kept a note saying the result existed and threw the result itself away.

This was the tenth time since August 12 that a run had reported its result and died before the outer process finished the handoff, so I finally went and looked at why.

I wanted to keep the failed status and recover the reports. The process had died, but its output was still there and could be validated.

The normal audit command already knew how to validate reports, write them, and create finding records. So I moved that work into one shared persistence path and made stale-run recovery use the same path. That keeps recovery subject to the same validation.

It also cannot write immediately. The reaper reads and validates the possible result first, then has to win the atomic transition that marks the run stale, and only that winner may replay the audit artifacts. A second sweep sees a terminal run and leaves the recovered files alone. If the stale transition itself fails, nothing gets written and the run remains retryable.

The log reader needed work too. It used to inspect a fixed 256 KB tail, which is fine until you remember that a batch audit carries its entire final result on one line, and that line can be wider than the window. The old reader could land in the middle of it, discard the fragment as incomplete, and miss the only copy of the work. It now widens its bounded read as needed, up to 16 MB, while still failing closed on missing, truncated, or malformed output.

One smaller thing while I was in there: recovered audit names now have to come from the run’s launch context or the project’s own audit catalog, rather than from anything the payload in the log claims about itself.

After recovery the run still says failed, with heartbeat_stale as its stop reason. The reports and findings now exist in the project, and the summary lists what was recovered.

I’ve spent a lot of time checking whether agents really finished their work. In this case, the finished reports were sitting in the log while I was looking at a failed run.