beazley.org

Building with AI · Part 8

Build Proofs: making the tool prove itself

Posted July 5, 2026

Every AI dev tool demo you've seen was stage magic: a happy path, rehearsed, on the author's machine. The last thing aidd needed before going public was the opposite - an instrumented, adversarial, public demonstration that the released build can do what the README claims, with every failure filed as a bug against the release instead of quietly hand-fixed. I called them build proofs in the plan, and the rationale is the whole idea in one sentence:

"The release claim is 'aidd can take a spec to a working app and can adopt code it didn't write.' The only honest evidence is doing exactly that, on camera, with the public build - and filing everything it breaks as bugs against the release."

Four proofs, one per creation lane, each on a different stack, so together they cover the surfaces the product claims. Every run recorded manual interventions, failed assumptions, elapsed time, and final gate results. The hard rule: failures become backlog items. Never a silent manual fix.

They ran over the last three days. Here's how it went.

The four lanes

(a) Fresh scaffold: habit/streak tracker (Bun + TypeScript + React). From a spec to 25/25 features passing, 27 commits, a clean quality gate, about 2.5 hours across three runs. Its top finding was an honest product gap, not a bug: aidd doesn't auto-chain runs, so a backlog that outlives one run's time budget needs a human to press launch again.

(b) Third-party template: kanban board (a Vite + React starter aidd has no special knowledge of). 23/23 features plus 10/10 headless acceptance scenarios, 26 commits, and a real exercise of the approval-gate flow. It surfaced a Windows path-mangling bug in template argument substitution, and a sharper conceptual gap: the create-then-ingest lane orphaned the spec. The scaffold output got ingested, but the original spec never became a feature backlog. Filed, fixed.

(c) Spernakit init: SMB infrastructure dashboard (Bun + Elysia + React + Drizzle). The golden-path proof, and the hardest one. Final score: 26/26 features, all 30 gate steps green including strict template-drift checking, roughly 14 hours total. The create step alone took seven attempts and produced six real product findings before a single feature was built. My verdict in the plan doc stands: the create saga was the proof working as intended. This one proof drove five Spernakit releases.

(d) Existing codebase ingest: flaskr, the canonical Flask tutorial app, in Python. A stack aidd shares literally nothing with. The profile inferred Python/Flask/pytest with zero Bun or Node leakage; a security audit produced four findings grounded to file and line; and the headline guarantee held: every intake commit touched only .aidd/. Zero writes to the app's own code. Then, as an unplanned fifth proof, aidd fixed the four findings - in idiomatic Flask, gated by the app's own pytest suite, 24/24 green.

What they broke (the point)

The proofs failed in embarrassing, specific, fixable ways, and nearly all the failures shared one root cause: aidd grew up assuming projects it scaffolded itself.

  • aidd dumped nine Bun-flavored scaffold files into the Flask app's root. The root contract now installs only in the initializer phase, for projects aidd actually created.
  • The default project directives told every ingested codebase to be "spernakit-like," forcing a keep-Python-or-port decision onto a Flask app that never asked. Directives are now stack-neutral for ingested projects.
  • Best of all: aidd tripped its own dirty-tree safety gate on its own metadata. 93 files of its own .aidd/ bookkeeping counted as "your tree is dirty, refusing to run." aidd was blocking itself on its own dirt. That's the kind of bug you only find by actually being a stranger to your own tool.

On July 3, four aidd minor releases shipped in one day, most of it proof fallout. The Spernakit findings were their own genre: fresh apps generated by the template's own init failed the template's own first quality gate, because init's transforms could never pass strict drift comparison against the pristine template. And the deepest lesson of the campaign came from proof (c): gates must be un-maskable. The quality gate died on its first failing step, so a known-red drift check at position one masked every gate behind it. Agents normalized "the gate fails on the known thing" while 24 files quietly grew past the 300-line cap, the largest reaching 1,647 lines. The fix - run all order-independent checks and report the aggregate - shipped on launch day itself, because a public template cannot afford exactly that species of quality theater.

Campaign totals: four lanes, four stacks, eleven aidd backlog items and five Spernakit releases out of the wreckage, and no silent manual procedure anywhere. One detail I love: the aidd fixes landed on main same-day via aidd's own self-runs.

The tool repaired itself, using itself, from bugs it filed against itself.

July 4

Both repos went public yesterday, on Independence Day. Unplanned symbolism, gratefully accepted. Each shipped as a single squashed "Initial public release" commit under fresh history, with a front-door site whose receipts section is just the build-proof table: the lanes, the stacks, the pass counts. The launch feature payload (multi-session embedded terminals, approval cards that carry the failing gate's actual output so decisions are never blind) was tagged v2.99.0 in the morning.

The released build is v2.100.0. The gap between those numbers is the last lesson of the whole run, and it's three patches wide:

  • v2.99.1 - the example config's keys weren't sorted; the CI format gate failed on the tag.
  • v2.99.2 - a package-manager-enforcement hook couldn't resolve its own dependency on the fresh CI runner; replaced with a dependency-free inline check.
  • v2.100.0 - a loose CI cache restore-key let a run that died mid-install poison the next run's dependency tree; dependencies now install fresh, and only the content-addressed cache persists.

No functional changes in any of them. Twenty-two months of private iteration - three template generations, a supervisor that went from a forked shell script to an operating surface - and the moment of going public was config key order and a CI cache. The last mile of shipping is always someone else's machine.

What the whole arc taught

Compressing twenty-two months and eight posts into what I'd actually tell someone building an agent orchestrator:

  1. The right abstraction comes from concrete experience. Three wrappers before one orchestrator; three template generations before the freeze. Rewrites are tuition, not waste.
  2. Findings become features. The audit loop - build, audit, remediate, re-verify - is the difference between output and verified output. High numbers or real numbers. Pick real.
  3. Put a mechanical check where a hope used to be. Write allowlists, worktree isolation, canonical statuses, typed event contracts, un-maskable aggregate gates. Prompts request, agents hallucinate, guards enforce.
  4. Verify before you file, and distrust "it compiles." Curl-verified bugs had a 0% false-positive rate; UI-observed ones ran 42%. The end-to-end test is the gate. The compile check is a courtesy.
  5. Automation you can't review is worse than none. The auto-upgrade tool died for making decisions I couldn't inspect. Parallel manual cherry-picking was slower in theory and faster in every way that mattered.
  6. Dogfooding has a structural blind spot: your own machine, your own projects, your own habits. Clean VMs and foreign codebases find in an afternoon what two years at home cannot.
  7. Ship the receipts. Not testimonials, not a demo reel. Instrumented proofs with the failures left in. The failures are the credibility.

aidd is public now with a spec anyone may implement and a run ledger for the operator who asks "show me." The loop closed. Time to see what other people's machines make of it.

Let me know what you think and what you build.