Quality Gates That Pass Without Running: four failure modes to hunt in your CI

There’s a CI failure mode that shows up on no dashboard: the quality gate that stamps ✅ without ever having done the work. No stack trace, no red job, no alert — the step runs, returns zero, and the automatic PR comment says everything’s fine. What it doesn’t say is that it looked at nothing.

It’s a treacherous pattern because it disguises itself as success, which is why it tends to last months before anyone notices. What follows is a teardown of four real variations, all in the same pipeline of one static site (this blog’s CI is the specimen), with the numbers for each and the defense against each. The interest isn’t in any one of them alone — it’s the pattern, and the five-minute test at the end that catches all of them at once.

---

Gate 1: the link check looked at 21 of 74 pages

quality-gate.yml runs lychee against the built dist/, in two steps: internal links and external links. Both received the same file list:

./dist/**.html ./dist/**/index.html

Looks reasonable. It isn’t. Bash’s ** only recurses into subdirectories with shopt -s globstar enabled, and the GitHub Actions run: shell does not enable it by default. Without globstar, ** is an ordinary *. What actually reached lychee was:

./dist/*.html        # the root
./dist/*/index.html  # exactly one level down

Everything two levels down was excluded: /pt/blog/<post>/, /zh/blog/<post>/, /produtos/<slug>/. In other words, the entire site outside the root — which is where essentially all the content lives.

Measured with the same lychee and the same flags, run under /bin/bash to reproduce the run: step (and not under zsh, which expands ** on its own and therefore hides the bug when you test it on your laptop):

	files	links	unique	errors
before (./dist/**.html …)	21	705	150	0
after (list from find)	74	2917	212	0

Zero errors in both cases. The gain here is coverage, not a broken link found — and that distinction is what this whole post is about.

The fix was not to turn on globstar. It was to take the list out of the shell’s hands: one step builds it with find, and both lychee steps read the same file. That way it depends on neither a shell option nor the runner’s bash version, and two hand-written lists cannot drift apart. The step also prints N HTML pages built and runs test -s on the list — because the difference between “found no broken links” and “looked at nothing” has to be in the log.

A small detail with a specific reason: the read uses while IFS= read -r rather than mapfile, so it doesn’t require bash 4+. The new shell snippet was also run under bash 3.2 — older than the runner’s — to confirm.

---

Gate 2: the binary died in the linker, inside the container

With the glob fixed, the branch’s first run showed the real log:

./.bin/lychee: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by ./.bin/lychee)
./.bin/lychee: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by ./.bin/lychee)

lychee is downloaded on the runner host and executed inside the node:22 container, which is Debian bookworm — glibc 2.36. The -unknown-linux-gnu build of lychee 0.24.2 requires 2.38/2.39. The binary never opened a single file. Right glob or wrong glob, it made no difference.

Which means the 21-file coverage measured above is the coverage the step would have had if it had run. It hadn’t run since the day it moved into a container.

The fix is the musl build, which is statically linked and does not depend on the image’s libc. And the install step now runs lychee --version immediately after downloading — so it fails where the problem is, not two steps later, with a message about something else.

---

Gate 3: and the ”✅” was false

Here is why the first two lasted so long.

lychee ... | tee link-check.log || echo "ISSUES=true" >> $GITHUB_OUTPUT

|| evaluates the status of the last command in the pipe, which is tee. And tee always exits 0. The ISSUES flag never went up. The automatic PR comment announced ”✅ No broken links” — with the GLIBC log right below it, in the same comment, full of errors.

The lie was automated and pasted onto the PR every time.

Fix: set -o pipefail on both steps. Now both a broken link and a tool that fails to run raise the flag. Distinguishing between the two is what the log is for, and the comment already embeds it.

It’s worth separating the two things that || true was doing:

• Not blocking a merge over an external link outside your control — a legitimate product decision, and it still stands (continue-on-error: true on the link check steps).
• Not being able to tell “green” from “did not execute” — not a decision. A bug.

You want the first. The second kills the first.

---

Gate 4 (bonus): the lint that was never installed

Same pattern, different step. Both workflows ran astro check as their lint step. It was a double no-op:

1. @astrojs/check and typescript were not in the dependencies. The command just printed the install prompt and exited without checking anything.
2. 2>&1 || true plus continue-on-error: true guaranteed it could not fail even if it had run.

Two independent layers of “this must not fail.” Either one alone was already enough.

Actually installed, the check reported 62 errors. But 54 of them were an artifact of the repo having no tsconfig.json: without it, the types generated in .astro/types.d.ts never enter the program and every getCollection() comes back as never. A five-line tsconfig.json extending astro/tsconfigs/base clears all 54 at once.

The remaining 12 were real:

• functions/_middleware.ts — Request is the Workers class at runtime, but the DOM lib’s type shadows it. Two casts re-label the value with the shape env.ASSETS.fetch expects; nothing changes at runtime.
• AntesDepois.astro, PreviewGallery.astro — querySelector returned Element, and the script read .style, .value, .dataset, .src and .alt. It now asks for the right type at the query.
• RelatedPosts.astro — allPosts only accepted CollectionEntry<'blog'>, but the /en/ and /zh/ pages pass blogEn and blogZh. Widened to all three collections.
• blog/author/[slug].astro — the og:image was author.avatar || '/og-image.png', and AuthorData has no avatar field: no author ever defined one. Dead branch, removed.

None of them took the site down. All of them were latent bugs waiting on a specific code path.

The detail that closes this fix: the dist built before and after is byte-for-byte identical across every page — only the worker bundle hashes and the ordering of the _routes.json exclusion list change. Twelve type errors fixed, zero behaviour change. Which is exactly what you expect from a lint that spent two years switched off: it wasn’t hiding a fire, it was hiding twelve small debts.

---

The first honest signal

After all four fixes, the link check ran for real for the first time: 74 files, 2917 links, 212 unique — and 10 errors. All pre-existing, none caused by the changes.

They were recorded in the PR rather than silenced, because it’s the first honest signal that step has ever produced. Among them: a GitHub repository that turned 404, a subdomain whose DNS no longer resolves, and an external server that rejects the TLS SNI. Nothing catastrophic — but 10 broken links the gate swore did not exist.

And the comparison is worth stating: 21 → 74 files and 705 → 2917 links checked, with 0 → 10 errors found. The gate did not get stricter. It started existing.

---

The defense, which is one sentence

A gate that cannot fail is not a gate. It’s decoration with a maintenance cost.

The operational test is simple and takes five minutes: break the tool on purpose and see whether CI complains. Point the linter at a file that doesn’t exist. Swap the binary for one that won’t run. Make the test fail.

If the job stays green, you don’t have a check — you have a badge.

Three requirements worth demanding of any CI step, each of which would have caught one of the four cases above on the day it landed:

1. The step prints its denominator. “0 broken links” says nothing; “0 broken links across 2917 links in 74 files” says everything. A number without a denominator is indistinguishable from a number that was never measured.
2. set -o pipefail wherever there is a |. The tee-to-a-log case is the most common and the most treacherous, because the log sits right there next to the summary, full of errors, saying the opposite.
3. The tool introduces itself before it works. A --version after the download costs 200 ms and turns “0 errors found” into “0 errors found by a tool that exists”.

The detail that closes the story: the run that surfaced the GLIBC problem was the run of the very PR fixing the glob. The half-fixed gate is what exposed the other half. That’s usually how it goes — you don’t discover that CI lies by auditing the CI. You discover it when it finally speaks.