The Pipeline Was Green for Three Weeks. It Had Been Shipping a Build That Never Compiled.

The Pipeline Was Green for Three Weeks. It Had Been Shipping a Build That Never Compiled.

Leader 1 5 34
calendar_today agoschedule4 min read

For three weeks a deployment pipeline reported every step green and shipped a build that had failed to compile on every single run. The build step ended in npm run build | tee build.log so the output could be archived. That pipe is the whole story: bash returns the exit status of the last command in a pipeline, which was tee, and tee always succeeds at copying text. The compiler's non-zero exit got thrown away the instant the pipe handed off. The error was sitting right there in build.log. GitHub Actions saw exit code 0, painted the step green, and deployed the broken artifact. Nobody read the log, because the checkmark said there was nothing to read.

That's the defining property of bash in CI, and it's why I treat pipeline scripts differently from anything I run in a terminal: a silent failure can present as success. On a server you watch a command fail in front of you. In a pipeline, a swallowed exit code produces a green checkmark over broken code, and the gap between "the logs show an error" and "the pipeline reports failure" is exactly where outages are born. I wrote the full guide because I've now been burned by every variation of this, and there's a consistent set of habits that close the gap.

There are four failure modes that are specific to CI and barely ever bite you at an interactive prompt. Exit codes swallowed by a pipe — the story above, any command | tee, command | grep, command | sort. Shell provisioning differencesubuntu-latest gives you bash 5.x, macos-latest gives you bash 3.2 from 2007, and a script using associative arrays or ${var,,} passes on one runner and throws a syntax error on the other in the same workflow. Environment variable gaps — CI sets variables you don't control and omits ones you assume exist, and without set -u a missing $DEPLOY_TARGET becomes an empty string and does something quietly wrong. Interactive-shell assumptions — CI runs a non-interactive, non-login shell that never sources your .bashrc, so a command that works when you type it dies with command not found because the thing that defined it was never loaded.

The header that closes most of these is short, and every line earns its place:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

set -e exits the moment any command fails, so the step exits non-zero and the workflow actually registers a failure. set -u treats an unset variable as an error, so a typo'd $DPLOY_TARGET dies immediately instead of expanding to nothing and corrupting a path. set -o pipefail makes a pipeline return the first non-zero exit among its commands rather than only the last — that one flag is the direct fix for the | tee bug that ran green for three weeks.

Secrets deserve their own paragraph because CI hands you env: values and secrets: values identically — the shell can't tell them apart, the only difference is that Actions masks the secret's literal string in the log. The trap is that the moment you transform a secret (base64-decode it, slice it, interpolate it), the transformed value no longer matches the mask and prints in clear text. Validate required values up front with ${VAR:?} so a missing secret fails at startup with a clear message instead of on line 47 with a cryptic permission denied, and be very careful with set -x in any step that touches a secret.

The pipe-exit-code problem is worth one concrete tool beyond pipefail: PIPESTATUS is an array holding the exit code of every command in the last pipeline, read immediately after:

npm run build | tee build.log
build_rc="${PIPESTATUS[0]}"   # npm's code, not tee's
[[ "$build_rc" -eq 0 ]] || { echo "build failed: $build_rc" >&2; exit "$build_rc"; }

pipefail has one well-known false positive — grep returns 1 when it finds no matches, which is often fine, and under pipefail plus set -e that aborts the script. Absorb it deliberately with || true only where a non-match is genuinely acceptable, and nowhere else, because blanketing every command in || true just reinvents the silent-success problem you're trying to kill.

Docker has its own landmine: every entrypoint script must end with exec "$@". Without it, your script stays PID 1 and your app runs as a child, so when the orchestrator sends SIGTERM on docker stop or a rolling deploy, the signal hits the script, which doesn't forward it, and after the grace period the orchestrator escalates to SIGKILL — abrupt termination, dropped connections, lost in-flight work. exec "$@" replaces the shell with your app so it becomes PID 1 and receives signals directly. The guide pairs this with a wait_for dependency-check pattern and a trap, and the Bash trap & Signal Handler Builder generates the exact signal block an entrypoint needs.

Deploys get the same treatment: deploy into a fresh timestamped directory, flip a current symlink atomically with ln -sfn so traffic never sees a half-written release, keep the last several releases so rollback is just re-pointing the symlink, run a health check after the swap and fail the deploy if it doesn't pass, and stamp the git SHA into the release so "what's running right now" always has an answer. And when a step fails and the logs won't say why, set -x around just the suspect section shows you each command with its variables expanded — a doubled slash or an empty segment in the trace is usually your bug standing in plain sight.

The full guide is the field manual version of all of this — the four failure modes, the safe header, secret validation with a validate_env function, PIPESTATUS and pipefail, Docker entrypoints, the atomic-symlink deploy script with rollback and health check, debugging with set -x, and a production-ready checklist at the end: https://bashsnippets.xyz/guides/bash-scripting-for-ci-cd-pipelines

If your pipeline scripts are dying on the small stuff first — unquoted loops, bad argument parsing, missing traps — the snippet library that feeds into this guide is at https://bashsnippets.xyz

Part 8 of 8 in Bash Snippets Pages
🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

The Audit Trail of Things: Using Hashgraph as a Digital Caliper for Provenance

Ken W. Algerverified - Apr 28

My Nginx Died at 2 AM and Nobody Noticed for 6 Hours. Now I Have a Watchdog Script

BashSnippets - May 21

How I Built a React Portfolio in 7 Days That Landed ₹1.2L in Freelance Work

Dharanidharan - Feb 9

Optimizing the Clinical Interface: Data Management for Efficient Medical Outcomes

Huifer - Jan 26

The Validation Bottleneck: Why Testing Is the New Speed Limit

Tom Smithverified - Apr 13
chevron_left
2.2k Points40 Badges
North Americabashsnippets.xyz
24Posts
24Comments
3Connections
Linux user who got tired of Googling the same bash commands every time I sat down at a terminal. Sta... Show more

Related Jobs

View all jobs →

Commenters (This Week)

7 comments
2 comments
1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!