I now deliberately break the thing a new check protects and confirm that the build stops. I spent two weeks finding checks that had been passing while measuring the wrong thing. These are the failures that got me into that habit, and what I look for when reviewing a pipeline.

Try the failure before trusting the pass

For a new check, I use a disposable copy or fixture and work through this sequence:

  1. Name the behavior the check is meant to protect and run the unchanged case successfully.
  2. Make one deliberate change that violates that behavior, leaving the real project untouched.
  3. Confirm the change actually applied to the copy the check will inspect.
  4. Run the check and require failure for the expected reason. A setup error, missing import, or unrelated assertion is a failed experiment, not proof that the check detected the defect.
  5. Restore only the fixture change, confirm the check passes again, and retain the revision, mutation, output, and exit result together.

In October, a regression test for this site’s search exposed a defect in its own negative control: the intended retry mutation had never applied. The corrected harness requires exactly one mutation target before running the case, then checks the identity and declared message of the assertion it catches. Separate controls prove that import and cleanup failures cannot pass as a caught search defect. The expected red result has to come from the behavior being tested.

Measuring the wrong thing

My template’s crawl-test harness has a preview mode that is supposed to measure a production build. Its argument parser split on = only, and the package scripts pass the space form, so the mode silently fell back to dev. Every “production” crawl in the project’s history had been profiling the development server.

The output looked normal. It had crawled routes and produced measurements, just against the wrong server.

Check how your parser handles unknown flag values, especially modes. An unrecognized mode should fail instead of quietly selecting a default.

A cousin of this bit me the same month. I filed two alarming performance findings against an application, and both turned out to be dev-only instrumentation reporting intermediate values. The app was fine. I measured it badly and then acted on the measurement, which cost more than the performance problem I did not have.

React had also quietly migrated into a lazily-loaded chunk, adding a network round trip before the app could boot. Fixing the chunking took first load from 218 KB down to 184 KB and from two waterfall hops to one. I added a critical-path gate that checked chunk placement as well as size, then rebuilt the old configuration to test it. That negative test passed the size budget at 197 KB but failed both structural checks. Counting bytes alone would have missed the problem. Those measurements came from Spernakit 3.26.0 on July 20, 2026, using React 19 and Vite 8 with Rolldown.

First load fell from 218 KB over two sequential network hops to 184 KB over one before 218 KB two sequential requests before the app can start after 184 KB −34 KB one request; the framework is back in the entry chunk
Spernakit 3.26.0 on July 20, 2026: blocking first-load bytes before and after the chunking fix. The separate 197 KB negative-test rebuild passed the size budget but failed the new chunk-placement checks.

Byte count alone doesn’t tell you when the main content appears. Check Largest Contentful Paint and the request waterfall too, including which requests have to wait for others. For responsiveness, Interaction to Next Paint replaced FID as the interaction Core Web Vital in 2024. A byte budget cannot measure that either. Keep those checks alongside it.

Nothing was examined, so nothing failed

I found two checks that passed when they had examined nothing.

My PowerShell tooling’s test wrapper checked individual Pester results. But Pester can fail during discovery, before producing any individual results at all. The wrapper received an empty set, found no failures in it, and reported success. Zero tests ran and the build went green. It checks Pester’s overall result now. Elsewhere, a checker scanned built HTML for critical-path scripts with a case-sensitive tag match. Feed it <SCRIPT> and it counted zero scripts, then passed.

Both wrappers needed to confirm that they had examined something before reporting success. Look for that in your own harnesses. A route crawler finding zero routes should fail, as should a script check finding no scripts when the build is supposed to contain them.

I found the case-sensitivity one through a CodeQL alert I had nearly dismissed a second time after republishing the repository. Which is to say I ignored that alert once already.

During a fleet upgrade I found an entire admin API router that an earlier template upgrade had unmounted. Routes existed, handlers existed, and the feature was marked complete, but requests could no longer reach it. There was a check for exactly this: an integration checker that confirms declared features are wired up. It walked route directories, but this router lived in a flat module and never entered its inventory.

Two defenses there. Have the gate report what it examined rather than just its verdict, because “verified 14 of 14 declared features” is auditable in a way that “PASS” is not. And treat a mismatch between the checker’s inventory and the project’s inventory as its own failure.

The gate you turned off yourself

I had caused some of the gaps myself.

While upgrading the fleet I found that five applications allowed sensitive database tables to be edited through the generic admin screen. My template already had an assertion for exactly that problem. Each affected app had frozen or disabled it during earlier template work.

I had treated those overrides as a legitimate way to preserve app-owned behavior, and each one sounded reasonable at the moment I made it. Meanwhile the interfaces around the frozen files kept moving and the assertions stayed off. I was the reason the gate had stopped running in the place it mattered most.

If your system supports per-project overrides of shared checks, the overrides need an expiry, a visible inventory, and a reason recorded at the point of suppression. Otherwise it is easy to forget that a check has been turned off.

Caching

Several failures traced back to my smoke-step cache. One key ignored hidden .github inputs; another step had no cache classification at all. A step that is cached and skipped reports the cached result, which is correct behavior right up until the key does not capture something that matters.

The version that bit me hardest was a latent typo in one backend: a file calling a function it never imported. Smoke-test caching masked it for weeks; the full suite hit it immediately. I added a startup assertion that catches any step with no cache classification, so a new step cannot quietly inherit a default. I stopped treating a light-suite pass as finished verification. In aidd’s current coding workflow, fast checks come before the commit and full smoke:qc comes afterward; the commit itself is not the final verdict. This was my runner’s cache, not Docker’s build cache. The surprises were in the inputs its own keys left out.

One runner, called by everything

My aggregate quality gate used to die on the first failing step, which meant everything behind that step went unmeasured. You fix one thing, re-run, and discover a second failure that was there the whole time. Fix that and a third appears. That meant several runs just to discover failures the first run could have reported.

[RUN] check:max-lines
[FAIL] check:max-lines exited with code 1
[RUN] typecheck
[FAIL] typecheck exited with code 2
[RUN] format:check
[PASS] format:check 3.4s
[RUN] lint
[PASS] lint 10.6s

2 step(s) failed:
  [FAIL] check:max-lines
  [FAIL] typecheck

Illustrative output: two failures, one run. The old gate would have shown the first one and stopped.

Steps that genuinely depend on earlier ones still have to skip, since you cannot crawl a build that failed to compile. But most steps in a typical gate are merely sequential rather than dependent, and those should all run.

The other half of this was that I had hand-maintained command lists in more than one place, CI configuration and local hooks, each listing the steps to run. Those lists drifted. A step added locally never reaches CI, or a step removed in CI keeps running locally and produces failures nobody else sees. Both callers invoke the same runner now.

Licensing

Once people can download releases, third-party licensing obligations follow what those releases actually redistribute. My license lists were hand-maintained and had already drifted, packages missing and versions stale, and I had not noticed because nothing was checking. They are generated from the installed dependency graph now, and CI fails when the generated output and the lockfile disagree.

What took longer to get right was where the checks look. The gate has to follow the distribution form. A source-only release checks material vendored in the repository without claiming that its npm dependencies travel with it. A compiled executable checks its embedded runtime closure, a container checks its contents, and required notices have to survive into browser assets. Even a static site with no runtime JavaScript embeds MIT-licensed CSS and fonts, and MIT requires the notice to travel with the code.

I checked this by removing a required notice and confirming that the build stopped.

Where this came from