Part 4 of a series on ten years of deployment evolution for one legacy app on AWS — no Kubernetes required. Series index.
Before you replace a production deploy system, read the one you already have. All of it. Line by line, as if a stranger wrote it and you've been handed it for review.
I say this because I did the opposite for a long time, and then I finally did it right, and the difference was five bugs I had personally shipped and personally forgotten. The system in question was the blue/green deploy pipeline I built about four years ago for a workforce-management product — a ~10-year-old PHP/Node monolith on EC2, three environments, a DR region, and a small platform team. It had run thousands of deploys. It was, for its era, good work, and I was proud of it. I was about to throw most of it away for a fully-baked-AMI design (that's Part 5), and before I did, I sat down and audited it.
This is not a post about bad code. It's about the specific way good code goes bad without changing a single character: the assumptions underneath it quietly expire. The five defects I found weren't sloppy. Four years ago every one of them was a reasonable decision resting on something that was true at the time. Then the environment moved, the truth stopped being true, and the code kept executing the old belief. None of them had fired, because our deploys mostly succeeded. I want to be precise about that, because "mostly succeeded" is not the same sentence as "was safe," and the gap between those two sentences is exactly where these lived.
So this is cognitive archaeology, not a takedown. I'm digging through my own reasoning from four years ago and asking, for each thing I find: what did I know then that made this correct, and when did it stop being true? Every section below has a subsection for that question. The person I'm being hard on is a version of me who no longer exists, which is the only kind of code review that's genuinely comfortable.
One more piece of context. I didn't go looking on a whim. Part 2 of this series was a multi-day cascading incident where a latent race — one that had been survivable for years — finally met an emergency AMI and an incident-era workaround and stopped being survivable. The lesson I took from that wasn't "fix that one race." It was: don't wait for the next incident to audit the deploy path. Go find the bombs on a quiet afternoon. This audit was me doing that.
How the old system worked, briefly
Each environment had three tiers behind their own internet-facing ALBs: a web tier (HTTP and a queue worker), a socket tier (the singleton that owns websockets, database migrations, cache clears, and cron), and a command box (a manual-only tier, normally at zero capacity). Every tier had a blue ASG and a green ASG. At steady state exactly one color had capacity; the other sat at
min=max=desired=0.A deploy scaled up the idle color, waited for it to go healthy, flipped the ALB listener weights from
old=0 / new=100, then scaled the old color to zero. The health signal was end-of-deploy: /validate returned 200 only once the app finished booting. Terraform never did the cutover — it provisioned both colors at zero and seeded initial weights, and every capacity and traffic change at deploy time was an imperative AWS CLI call from two scripts. There was an orchestrator (I'll call it the driver) that made global decisions, and three per-tier workers it launched in parallel, one per tier, each of which owned the scale-up for its ASG.Hold onto that split — a driver making global decisions and workers making per-tier ones. Two of the five bugs live in the seam between them.
Defect 1: two scripts deciding the color from two different signals
The driver picked the cutover color one way: it read the socket ALB's
:443 listener weights and asked "which color currently has 100%?" The next color was the inverse. Meanwhile each per-tier worker picked its scale-up color a different way: it looked at its own ASG pair and scaled up whichever color was empty.At steady state these two methods agree perfectly, because they're reading the same underlying fact from two angles. One color has weight 100 and has capacity; the other has weight 0 and is empty. Ask "who has the weight" or ask "who is empty," you get complementary answers that point at the same next color.
Now desync them. Suppose a previous deploy failed partway, or someone scaled an ASG by hand during an incident, and one tier's live color no longer matches what the listener weights imply. The driver reads the weights and decides "next color is green." A worker reads its ASG and finds green already has capacity (it's actually the live color), so it scales up... the color that's already serving. Health checks pass immediately, because that instance is up and running the old code. The driver flips weights to green, sees green healthy, declares victory, and scales down the other color. You have just run a completely successful deploy that shipped nothing. The new build is sitting in S3. Production is running last week's code and the pipeline is green.
That's the worst kind of bug: not an outage, not an error, but a silent lie. A stale deploy that reports success.
Why this made sense four years ago
When I wrote it, the two derivations were deliberate redundancy — belt and suspenders, two independent reads of the same invariant. The invariant was ironclad: exactly one color ever has capacity, and it's the one with weight 100. Every deploy either completed fully or aborted the whole run; there was no code path that left capacity and weights disagreeing. Given that invariant, having the driver and the workers each derive the color independently felt safer than threading one decision between them. Two readers of one truth can't diverge if the truth is single-valued.
The assumption that expired was the invariant itself. The moment the system grew partial-failure states — or the moment a human touched an ASG mid-incident — capacity and weights could disagree, and then two independent readers of a now-inconsistent world produce two different answers with total confidence. My redundancy turned into two subsystems arguing.
The fix: the driver decides the color exactly once, up front, before any scale-up, and passes it explicitly to all three workers. One decision, one source. Then a pre-flight assertion I didn't have before: the next color's socket and web ASGs must be at desired 0, or the run aborts. That single check does double duty — it catches the desync (if the "next" color isn't actually empty, something is wrong, stop), and it replaces the workers' old per-component concurrency guard, because a second concurrent deploy trips the same assertion. Decide once, then refuse to proceed if reality already disagrees with the plan.
Defect 2: treating a failed verification as success and tearing down the old side
After flipping the listener weights, the driver re-read them to confirm the flip took — six polls, five seconds apart. Reasonable. Verify the thing you just did.
Here's what the old code did when it couldn't confirm the flip. It logged the failure, set the internal
new_color variable to the intended next color anyway — there was a comment, # Assume it worked — and proceeded to teardown, scaling the old color's ASGs to zero.Walk the lethal case slowly. The
modify-listener call genuinely failed. Traffic is still on the old color; the old color is the only side carrying production. The verification did its job perfectly: it detected "I cannot confirm the new color is live." And the code's response to that correct detection was to destroy the old color — the only in-service capacity in the entire environment. Full outage. The exact moment the check exists to protect you is the exact moment the old behavior guarantees the worst possible outcome.Why this made sense four years ago
modify-listener almost never fails. It's one API call against an AWS control plane that is up essentially all the time. In four years I never once saw the flip fail. So I wrote the verification as a formality — a confirmation log line for the happy path, not a branch I expected anyone to take. # Assume it worked encoded a real and mostly-correct belief: the flip always works, so if the re-read looks off, it's the re-read that's flaky, not the flip. That was true to something like four nines.The assumption that expired is subtle, because it was almost right: I assumed a failed verification meant a flaky check rather than a failed switch. When your only safety check is written on the premise that it will never legitimately trip, you've built something that's correct precisely when you don't need it and catastrophic precisely when you do. The check wasn't decoration. I'd written it as decoration.
The fix: a verification failure now aborts, leaving both colors running, exits non-zero, and hands it to a human to confirm which side is actually live before anything is torn down. The asymmetry makes this easy. Conservative failure costs a few dollars of doubled capacity for an hour and one engineer's attention. The old behavior's downside was the whole site. When one branch costs pocket change and the other costs an outage, you don't need to be clever — you need to stop assuming.
Defect 3: a half-finished cutover with no way back
The cutover walked the tiers in order — socket, then web, then command box — each a separate set of
modify-listener / modify-rule calls. If the socket flip succeeded and the web flip failed halfway through, you were left with sockets pointing at new code and the web tier pointing at old code. Split routing. The old code exited on the failure and did nothing about the tiers it had already switched. The half-flipped state just sat there, live, until a human noticed the symptoms.Why this made sense four years ago
Same root as Defect 2: the flip doesn't fail, so "a failure partway through a three-tier switch" was a state I never modeled because I never observed it. And the three tiers are logically independent — separate ALBs, separate listeners. Switching them in sequence felt atomic because in practice all three flipped in well under a second with no failures in between. The window for a partial failure was real on paper and invisible in operation.
What expired: the belief that three independent operations would always succeed or fail as a group. Nothing made them atomic. They flipped together for four years because they were fast and the API was reliable, and I let "fast and reliable" stand in for "atomic," which it is not.
The fix: on any mid-sequence failure, roll the already-switched tiers back to the old color and scale the new color down. Rolling back to the old weights is idempotent and safe by construction — the old color still exists and is still healthy, so re-pointing traffic at it cannot make anything worse. The tier that failed mid-flip gets registered into the rollback set before the attempt, so it's included even if it left partial state. You land back exactly where you started: all traffic on the old color, nothing shipped, but the routing is coherent and the site is up.
Defect 4: a health gate hard-coded to one instance
Before cutover, the driver waited for the new color to become healthy. The old gate was a flat constant,
MIN_EC2_HEALTHY_COUNT=1. One healthy instance in the new color's target group and the driver would flip.In the test environment every tier runs
desired=1, so "one healthy" and "the whole fleet healthy" are the same statement. In production the web tier ran min=2, max=6, desired=2. There, "one healthy" meant the driver would route 100% of production traffic onto a single fresh instance while its sibling was still booting — or, if the sibling was never going to come up, permanently. Half the intended capacity absorbing all the load, with the old fully-scaled color already being torn down behind it. If that one instance was marginal, the deploy had just concentrated every request onto the weakest new node in the fleet.Why this made sense four years ago
I built and tested the entire pipeline in the test environment, where everything is
desired=1. "Wait for 1 healthy" and "wait for the fleet" were literally identical in the world I developed in. The constant was a hard-coded literal that happened to be correct there. Production's desired=2 was configured later, in Terraform, in a different repository, by a version of me who wasn't looking at that shell script. The capacity config and the deploy logic drifted apart, and nothing connected them because nothing had to — they lived in different repos and agreed by coincidence.The expired assumption is a one-liner: desired capacity is 1. True in test forever. False in production from the day prod scaled past a single instance, which was roughly day one of prod.
The fix: the gate reads the tier's live desired count at deploy time and requires exactly that many healthy before it flips. No magic number — it asks the ASG how many instances it's supposed to have right now and waits for all of them. Correct in test (still 1), correct in prod (2, or whatever autoscaling has set it to at this moment). The bug wasn't the number 1. The bug was hard-coding a fact that belonged to the infrastructure into a script that couldn't see the infrastructure.
Defect 5: reconstructing the version from git tags
Deploys are triggered by pushing a tag. The pipeline needs the version string for that build, and the old tag pipelines reconstructed it from the commit with
git tag --contains <sha> | sort -V | tail -1 — "of all tags that contain this commit, take the highest-sorting one."That reconstruction fails three ways, and all three showed up eventually:
The first is multiple tags on one commit. When you promote an identical build from stage to prod, the same commit carries both tags.
--contains returns both, and sort -V | tail -1 picks one by version-string ordering, which is not necessarily the tag that actually triggered this run. You can build with the wrong environment's version string and never know until something downstream looks wrong.The second is shallow clones. CI checkouts are frequently shallow to save time, and a shallow clone doesn't have the tags outside its fetched history.
--contains then silently sees fewer tags than exist and returns the wrong answer, or nothing.The third is garbage that fails far from its cause. A malformed tag — say a hand-typed hotfix tag that split on the wrong delimiter and reduced to something like
hotfix — flowed through the script unquoted and unvalidated, then blew up deep inside an AWS lookup with an error message that pointed nowhere near a tag-parsing problem. You'd debug the AWS call for an hour before realizing the input was rotten three hundred lines earlier.Why this made sense four years ago
At the start, one commit had one tag, clones were full, and tag names followed one shape.
git tag --contains was a perfectly sensible way to answer "what version is this checkout?" from inside the checkout. The pipeline reconstructed the tag from the commit because it could, and doing so avoided threading the triggering tag through as an explicit input — one less thing to pass around.The expired assumptions here are plural, which is why this one is my favorite piece of archaeology. Single-tag-per-commit broke when we began promoting byte-identical builds across environments. Full clones broke when CI optimized checkout depth. Clean tag formats broke the first time a human typed a hotfix tag by hand under pressure. Not one of these was wrong in year one. Every one of them was wrong by year four, and the code went on faithfully reconstructing a value it should simply have been handed.
The fix: the CI runner already knows exactly which tag triggered the run — it exports it as an environment variable. Derive the version from that, fail fast if it's empty, quote every argument on the way through, and regex-validate the format in the driver before it's used for anything real. Stop recomputing a fact the runner already has, and reject a bad value at the top of the run rather than in the basement.
The methodology: audit the incumbent as step zero
The findings above weren't the goal of a project. They were the first task of one. I was heading for a new architecture, and I made myself read the old one first, cover to cover, before designing anything.
I now think this should be step zero of any migration, non-negotiable. There are two payoffs and only the first is obvious.
The obvious one: you find the bombs before they go off, and some you can defuse in place. All five of these got patched in the old system the same week I found them, because the replacement was months away and production still had to deploy in the meantime anyway. An audit that only produces a design doc for the future leaves the present undefended. Fix what you can where it lives.
The non-obvious payoff is political, and it mattered more. "We should move to immutable AMIs and instance refresh" is an opinion, and opinions get argued with. "Here are five specific ways the current pipeline can silently ship nothing or take the whole site down, with the exact failure path for each" is evidence, and evidence ends the argument. When I proposed the replacement, I didn't open with the new design. I opened with the incumbent's failure modes. Nobody debates a list of concrete ways production can break; they ask when it'll be fixed. The audit was the business case, and it was more persuasive than any slide I could have drawn of the target architecture.
I logged every finding one line at a time in a plain-text
DECISIONS.md in the repo — greppable, diffable, no ceremony. The format is one row per decision:For example, in the spirit of what I actually wrote:
One line per finding: the decision point, what changed, the verdict (adopted / recommended / accepted-as-is), and a single sentence of reasoning. The point of the log isn't documentation for its own sake. It's so that a class of problem costs you tuition exactly once. When the next audit surfaces something in the same family, you cite the precedent instead of re-deriving the argument from scratch. Cheap to write, and it compounds.
The open question: four years of silence — luck, or just rare?
Every one of these had been in production for about four years, and not one had ever fired. I want to sit with that honestly, because the comfortable answer ("I got lucky") and the smug answer ("they were edge cases, no big deal") are both wrong, and the true answer is the interesting part.
It's mostly arithmetic. Each bomb triggers only on a specific, rare failure path: a failed listener flip, a mid-sequence cutover failure, a partial-failure state that desyncs capacity from weights, a production instance that won't come healthy, a malformed tag typed by hand. And deploys themselves aren't constant. Multiply a rare failure path by an infrequent trigger and you get a joint probability low enough to stay dark for years without any luck required. Rareness compounds. Two independently uncommon things almost never coincide, so a bug that needs both can sleep through a lot of calendar.
The trap is what a long run of green pipelines does to your confidence. A high deploy success rate reads, emotionally, as "the system is safe." It is not the same claim. A 99.9% success rate over thousands of deploys tells you the happy path is solid. It tells you almost nothing about the 0.1%, because you have so few samples of failure to reason from — and every one of these bugs lives entirely inside that 0.1%. Four years of successful deploys was four years of evidence about the path that works and near-zero evidence about the paths that don't. I had mistaken the absence of observed failures for the absence of failure modes. Those are different things, and the difference is precisely the set of bugs this audit found.
Which is the whole argument for going looking rather than waiting. The failures hadn't happened yet; that's not the same as safe, it's just untriggered. The incident in Part 2 was what made this concrete for me — a latent race that stayed survivable right up until the day two rare conditions lined up and it wasn't, at which point it cost days. The takeaway wasn't "patch that race." It was: the deploy path is full of untriggered failure modes, and the cheapest time to find them is a quiet afternoon, not the middle of the outage where the path you never modeled finally meets the trigger you didn't expect.
So I went looking, on my own four-year-old code, on the way to replacing it. And the replacement — Gen 4, the subject of the rest of this series — is designed to delete most of these failure modes structurally rather than patch them one at a time. No two-color cutover means no color-decision desync and no partial-switch split-brain. A pointer in SSM Parameter Store replaces "reconstruct the version from git tags." Native ASG instance refresh replaces the hand-rolled scale-up / health-gate / flip / teardown choreography where four of these five bugs were hiding. I'm not sure I would have designed it as confidently, or defended it as easily, if I hadn't first spent a week proving to myself exactly what it needed to replace and why.
Part 5 takes apart that pointer model — how a single SSM parameter per environment became the release pointer, what "deploy version X" means when there's no git-tag reconstruction anywhere in the boot path, and why moving the version handoff out of the deploy scripts closed the last of the silent-stale-deploy class for good. It's the piece I most wish I'd had four years ago.