The Failures That Only Happen in CI 🏗️

Our release gate went red with sixty-one failures. Every one passed locally, on more than one machine. The tests had not changed.

The cause was a class of file that macOS writes when files pass through certain filesystems and archives: a small metadata sidecar named after the original with a ._ prefix. Our test suite walked directories looking for content files, matched those sidecars, tried to parse them as content, and failed sixty-one times. On a developer machine they are hidden from listings, so nobody had ever seen one. In the build container they were just files.

One .gitignore line fixed it. The interesting part is not the fix.

Your environment is hiding things from you

The reason this bug is worth writing about is that it is a member of a large family, and the family is defined by a single property: your local environment silently normalizes something that CI does not.

The usual members:

Case-insensitive filesystems. macOS and Windows default to case-insensitive. Linux does not. import Button from "./button" works locally and fails in the build container, and the error — module not found, on a file that plainly exists — is genuinely bewildering the first time.

Hidden and metadata files. The ._ sidecars, .DS_Store, editor swap files. Invisible in your file browser, present to any code that reads a directory.

Locale and collation. Sorting depends on the locale. A container defaulting to C sorts differently than a workstation on a regional locale, so a test asserting order passes in one and fails in the other.

Timezone. Containers usually run UTC; laptops rarely do. Any test involving date boundaries has a decent chance of being timezone-dependent without its author realizing.

Line endings. A checkout with different line-ending normalization changes file hashes and breaks exact-match assertions.

Available cores. Concurrency-dependent tests behave differently on eight cores than on two, and CI runners are frequently smaller than you think.

In each case, local is the lenient environment and CI is the strict one. That asymmetry is why "works on my machine" remains true and useless.

The second finding: a baseline that measured the wrong thing

The same gate surfaced a subtler version of the same problem. We keep a lint baseline — a recorded count of existing warnings, so new ones fail the build without requiring a big-bang cleanup. It had been recorded from a developer checkout.

The container produced a different count. Not because the code differed, but because the environment did: a different set of files present, slightly different resolution, one additional warning. The baseline was correct for a machine that does not gate anything and wrong for the machine that does.

The fix generalizes: a threshold must be measured in the environment that enforces it. A baseline, a performance budget, a coverage floor — any number compared against in CI has to be produced in CI, or it encodes one environment's quirks as a rule applied to another's.

Making the class of bug cheap

Three practices, in order of payoff:

Make the difference visible early. Print the environment at the top of every CI run — locale, timezone, core count, filesystem case sensitivity, tool versions. Ten lines of output that turn a bewildering failure into an obvious one.

locale; date; nproc; node --version
# Is this filesystem case-sensitive?
touch /tmp/Case && [ -e /tmp/case ] && echo "case-INsensitive" || echo "case-sensitive"

Be explicit about anything environmental. Set the timezone and locale in the test setup rather than inheriting them. Sort with an explicit comparator rather than the default. Assert on parsed values, not formatted strings.

Make file discovery precise. Our specific bug was a glob that matched more than intended. Directory walks that feed a test suite should match an explicit pattern and ignore dotfiles by default — a permissive pattern will eventually match something the filesystem created on its own.

The framing worth keeping

When something fails only in CI, the instinct is that CI is misconfigured. It is more useful to invert it: CI is showing you a truth your machine has been hiding. The code was always fragile to case, or locale, or unexpected files — you just had an environment generous enough to cover for it.

Which means the fix is rarely "make CI more like my laptop." It is to remove the dependence on the leniency, because production containers are strict too, and they do not report failures as a red build.