I spent ten months building an orchestrator because I did not trust what AI agents said about their own work. Then I spent two weeks finding out I had gotten completely relaxed about a green line printed by my own code.

Checks across my template and tooling had been passing for months while measuring the wrong thing. I had stopped looking closely at what they actually ran.

Several of the original nine incidents had the same cause. I’ve grouped those together here, with a few related problems I found later.

I now deliberately break the thing a new check protects and confirm that the build stops. These are the failures that got me into that habit, and the things I look for when reviewing a pipeline.

Measuring the wrong thing

My template’s crawl-test harness has a preview mode that is supposed to measure a production build. Its argument parser split on = only, and the package scripts pass the space form, so the mode silently fell back to dev. Every “production” crawl in the project’s history had been profiling the development server.

The output looked normal. It had crawled routes and produced measurements, just against the wrong server.

Check how your parser handles unknown flag values, especially modes. An unrecognized mode should fail instead of quietly selecting a default.

A cousin of this bit me the same month. I filed two alarming performance findings against an application, and both turned out to be dev-only instrumentation reporting intermediate values. The app was fine. I measured it badly and then acted on the measurement, which cost more than the performance problem I did not have.

Meanwhile the bundle-size gate was passing at 197 KB, wrong for a different reason. React had quietly migrated into a lazily-loaded chunk. Total bytes stayed inside budget, so the gate was satisfied, but the app now needed a full extra network round trip before it could boot. The budget counted bytes and had no idea where they sat. Fixing the chunking took first load from 218 KB down to 184 KB and, more to the point, from two waterfall hops to one. Those measurements came from Spernakit 3.26.0 on July 20, 2026, using React 19 and Vite 8 with Rolldown.

First load fell from 218 KB over two sequential network hops to 184 KB over one before 218 KB two sequential requests before the app can start after 184 KB −34 KB one request; the framework is back in the entry chunk
Spernakit 3.26.0 on July 20, 2026: the 197 KB the gate had been reporting is on neither bar. It was measuring the dev server, and counting bytes without asking how many round trips they arrived in.

Byte count only tells you part of what affects initial load. If you care about when the main content appears, Largest Contentful Paint and the request waterfall behind it are closer to what you mean, and a gate that never looks at request count cannot see half of what moves it. Responsiveness is another shape: Interaction to Next Paint replaced FID as the interaction Core Web Vital in 2024, and a byte budget cannot see it either. Keep those checks alongside the byte budget.

Nothing was examined, so nothing failed

I found two checks that passed when they had examined nothing.

My PowerShell tooling’s test wrapper checked individual Pester results. But Pester can fail during discovery, before producing any individual results at all. The wrapper received an empty set, found no failures in it, and reported success. Zero tests ran and the build went green. It checks Pester’s overall result now. Elsewhere, a checker scanned built HTML for critical-path scripts with a case-sensitive tag match. Feed it <SCRIPT> and it counted zero scripts, then passed.

Both wrappers needed to confirm that they had examined something before reporting success. Look for that in your own harnesses. A route crawler finding zero routes should fail, as should a script check finding no scripts when the build is supposed to contain them.

I found the case-sensitivity one through a CodeQL alert I had nearly dismissed a second time after republishing the repository. Which is to say I ignored that alert once already.

The same rule applies one level up, and that version is much harder to spot. During a fleet upgrade I found an entire admin API router that had never been mounted. Routes existed, handlers existed, the feature was marked complete, and no request had ever reached any of it. There was a check for exactly this, an integration checker whose job is confirming that declared features are wired up, and it could not see the problem because the router lived in a flat module rather than the directory structure the checker walked. The checker only knew about one directory layout, so it missed the flat module entirely.

Two defenses there. Have the gate report what it examined rather than just its verdict, because “verified 14 of 14 declared features” is auditable in a way that “PASS” is not. And treat a mismatch between the checker’s inventory and the project’s inventory as its own failure.

The gate you turned off yourself

I had caused some of the gaps myself.

While upgrading the fleet I found that five applications allowed sensitive database tables to be edited through the generic admin screen. My template already had an assertion for exactly that problem. Each affected app had frozen or disabled it during earlier template work.

I had treated those overrides as a legitimate way to preserve app-owned behavior, and each one sounded reasonable at the moment I made it. Meanwhile the interfaces around the frozen files kept moving and the assertions stayed off. I was the reason the gate had stopped running in the place it mattered most.

If your system supports per-project overrides of shared checks, the overrides need an expiry, a visible inventory, and a reason recorded at the point of suppression. Otherwise it is easy to forget that a check has been turned off.

Caching

Several failures traced back to my smoke-step cache. One key ignored hidden .github inputs; another step had no cache classification at all. A step that is cached and skipped reports the cached result, which is correct behavior right up until the key does not capture something that matters.

The version that bit me hardest was a latent typo in one backend, a file calling a function it never imported, which survived my light test suite for weeks because smoke-test caching masked it and then died instantly under the full suite. Two changes fixed the category: a startup assertion that catches any step with no cache classification, so a new step cannot quietly inherit a default, and redefining “ready to commit” to mean the full suite rather than the light one. This was my runner’s cache, not Docker’s build cache. The surprises were in the inputs its own keys left out.

One runner, called by everything

My aggregate quality gate used to die on the first failing step, which meant everything behind that step went unmeasured. You fix one thing, re-run, and discover a second failure that was there the whole time. Fix that and a third appears. That meant several runs just to discover failures the first run could have reported.

[RUN] check:max-lines
[RUN] typecheck
[RUN] format:check
[PASS] format:check 3.4s
[RUN] lint
[PASS] lint 10.6s
[FAIL] check:max-lines exited with code 1
[FAIL] typecheck exited with code 2

2 step(s) failed:
  [FAIL] check:max-lines
  [FAIL] typecheck

Two failures, one run. The old gate would have shown the first one and stopped.

Steps that genuinely depend on earlier ones still have to skip, since you cannot crawl a build that failed to compile. But most steps in a typical gate are merely sequential rather than dependent, and those should all run.

The other half of this was that I had hand-maintained command lists in more than one place, CI configuration and local hooks, each listing the steps to run. Those lists drifted. A step added locally never reaches CI, or a step removed in CI keeps running locally and produces failures nobody else sees. Both callers invoke the same runner now.

Licensing

Once people can download releases, third-party licensing obligations follow what those releases actually redistribute. My license lists were hand-maintained and had already drifted, packages missing and versions stale, and I had not noticed because nothing was checking. They are generated from the installed dependency graph now, and CI fails when the generated output and the lockfile disagree.

What took longer to get right was where the checks look. The gate has to follow the distribution form. A source-only release checks material vendored in the repository without claiming that its npm dependencies travel with it. A compiled executable checks its embedded runtime closure, a container checks its contents, and required notices have to survive into browser assets. Even a static site with no runtime JavaScript embeds MIT-licensed CSS and fonts, and MIT requires the notice to travel with the code.

I checked this by removing a required notice and confirming that the build stopped.

What I do now

I also tested the artifact-parity check by deleting the rule it protects. For each check, I want a specific change I can make that should cause it to fail. If I can’t identify one, I need to look more closely at what it checks.

Formatting is another check I’ve handed over completely. I argued about it for decades and eventually let the formatter have the final say, including on layouts I would not have chosen. I run it before review and spend the review time on what the code does.

I still have a long list of gates I have not tried to break yet.

Changelog

  • August 31, 2026: Reframed license gates around the actual distribution form, including source-only releases that do not redistribute their npm dependency graph.
  • August 5, 2026: Reconciled the incident accounting, corrected the cache mechanism, dated the bundle measurements, and added INP to the performance discussion.
  • July 26, 2026: First published.

Where this came from