Part 3 of a series on ten years of deployment evolution for one legacy app on AWS — no Kubernetes required. Series index.
In Part 2 I walked through the week everything broke at once: the ondrej PPA vanished, the Ubuntu archive got DDoS'd, Launchpad went dark, Ansible Galaxy started throwing 502s, and — because all of that blocked our Packer pipeline — someone snapshotted a running Stage instance into an emergency AMI. That AMI carried live services and stale opcache into production, tripped a latent blue/green race, and handed traffic to half-deployed instances while the pipeline reported success.
The fixes I described there were the right fixes. Retry wrappers around apt. Mirror fallbacks inside the VPC. A five-layer defense that made end-of-deploy the only path to a serving instance. A connection-pool bump on the managed MongoDB from a 500-connection tier to a 1,500-connection tier. All of it real, all of it shipped.
And all of it mitigation.
That distinction is the whole point of this article. Retries and mirrors make a fragile dependency less likely to hurt you. They don't change the fact that the dependency is there. Every single release still reached out to upstream package repositories at deploy time, because every release ran the full Ansible tree — apt, PHP, nginx, node, system users, and app code — against fresh instances. The blast radius of "nginx.org is having a bad day" was "nobody can deploy anything." We had made that blast radius less likely to detonate. We had not made it smaller.
The structural fix is to change the shape of the dependency, not its reliability. Move the part that touches upstream repos out of the release path entirely, so it happens once a week instead of on every deploy. That's what Gen 3 did, and the mental model I reached for turned out to be one every reader already knows from Docker.
The layer model
Here's the legacy shape. One base AMI, built by Packer plus Ansible. Then every release ran the entire Ansible role tree against fresh ASG instances at boot: OS packages, language runtimes, nginx and PHP config, and application code, all in one combined pass. The base AMI was barely a cache. The real work happened on every deploy.
Split it into layers instead:
If you have ever written a Dockerfile, you already know this trade-off in your bones. You put
apt-get install near the top so it lands in a cached layer, and you COPY your application code near the bottom so a code change only rebuilds the cheap part. The whole discipline of ordering a Dockerfile is about deciding what changes rarely and what changes constantly, and making sure the constant stuff never invalidates the rare stuff.That's exactly the split. The app-base AMI is the cached apt layer. It gets baked once a week (plus on demand), it's the only thing that talks to Ubuntu's archive and PHP's PPA and Ansible Galaxy, and it changes rarely. The release AMI is the
COPY at the bottom: it's derived from the current app-base, it adds only what's fast and app-specific, and it gets rebuilt for every git tag.Two things fall out of this immediately, and both matter more than the speed win.
Cadences decouple. OS security patches now land by rebuilding app-base — no release required, no developer involved. Application releases land by cutting a git tag — no OS churn, no apt at all. Before, those two clocks were welded together: you couldn't patch the OS without running a release, and you couldn't run a release without re-pulling the OS. Now they tick independently.
And deploys got faster, because the slow layer stopped running on the hot path. A release deploy dropped from around 10 minutes per ASG instance to 3–5 minutes. That number isn't the headline — the headline is failure isolation, which I'll get to — but it's the part your developers feel every day, so it's the part that buys you goodwill for the rest of the change.
Failure isolation, for free
Here's the part I didn't fully appreciate until I saw it work in production.
Terraform's
aws_ami data source, with most_recent = true, resolves to the newest AMI matching a name filter. The launch templates point at whatever that resolves to. So the release AMIs and the app-base AMIs each form a pool, and the infrastructure always picks the latest good member of the pool.Now play the May incident forward against this structure. nginx.org's package repo goes down on a Saturday. The weekly app-base bake fails — it can't pull nginx, so Packer's cleanup provisioner runs and the AMI never gets a publishable name, which means it never matches the
data.aws_ami filter, which means it never enters the pool. A Slack alert fires. I look at it Monday morning over coffee.And releases keep shipping. Because
most_recent is still resolving to last week's app-base AMI — the last known good one — every release that week builds on top of a base that was assembled before the outage. Developers cut tags. Deploys go green. Nobody outside the platform team knows nginx.org was down, because the only thing that cared about nginx.org was a build that runs once a week and is allowed to fail quietly.That is the entire incident, structurally prevented. Not mitigated with a retry — prevented, because the failing operation is no longer anywhere near the release path. Had this layering existed in May, the emergency AMI would never have been created, because there would have been nothing to emergency-fix: releases would have carried on using the prior week's base while I sorted out the upstream mess on my own schedule. The whole cascade started with "the pipeline is blocked, we have to do something drastic right now." Structure removes the "right now."
The two weekly builds isolate each other too, and that came out of a limitation I initially cursed at. I wanted one scheduled pipeline that built app-base and then built a release AMI on top of it. Bitbucket Pipelines schedules apply their variables at the level of the whole run, so two steps in one chained pipeline reading the same schedule variable can't be told apart — I couldn't disambiguate "which step am I configuring." So I gave up on the chain and made two sibling schedules instead: app-base build Saturday 14:00 UTC, release build Saturday 15:00 UTC, the one-hour offset acting as a buffer.
The workaround turned out better than the design I wanted. Two independent schedules fail independently. If app-base build fails, the release build still fires at 15:00 and uses the prior week's app-base via
most_recent. If the release build fails, last week's release AMI stays in the pool. The chained pipeline I originally wanted would have coupled the two — a broken app-base step would have taken the release step down with it. The platform limitation pushed me toward mutual isolation I wouldn't have thought to build on purpose.The health-signal inversion
This is the technical core, and it's a direct answer to the headline lesson from Part 2: an ALB health check at the infrastructure level tells you a process is up, not that your deploy finished. A
200 from nginx means nginx is running. It says nothing about whether the code behind it is the code you meant to ship.In the legacy combined role, the readiness model was "keep nginx down through the whole deploy, then start nginx and php-fpm together at the very end." nginx-down-means-not-ready. It sort of worked, right up until an emergency AMI booted with nginx already running and the ALB called it healthy 16 seconds after boot, before Ansible had even started. The gate service and the readiness signal were the same thing, and that thing could be true for the wrong reasons.
Gen 3 inverts it. nginx starts early — deliberately, in the vhost task, well before the deploy is done. And php-fpm becomes the one and only readiness gate.
That sounds backwards until you look at what the health check actually probes. We replaced the hardcoded
location = /validate { return 200; } workaround — the one from Part 2 that lived weeks instead of the two days it was scoped for — with a real PHP-rendered endpoint, /validate.php. Now the health check can only pass if PHP is actually executing:State | Health check | Verdict |
nginx up + php-fpm up + /validate.php present | 200 | healthy |
nginx up + php-fpm stopped | 502 | unhealthy |
nginx stopped | connection refused | unhealthy |
Starting nginx early buys you something real: the ALB health-check window opens while the slower deploy tasks — cron, the S3 code download, config cache rebuild, PM2 — are still running, so nginx warmup overlaps with work instead of happening after it. But nothing gets declared healthy, because
/validate.php returns 502 the entire time php-fpm is down. The single task that starts php-fpm is the last task in the deploy. From that moment, and not one second before, the ALB can see a 200.That's the inversion that matters. The health signal got upgraded — from "a process is up" to "the deploy actually completed." php-fpm being alive isn't an infrastructure fact anymore; it's the deploy telling the truth about itself.
Here's the comparison in full:
Aspect | Legacy combined role | Gen 3 release role |
Gate service | nginx | php-fpm |
Started early | no (nginx down throughout) | nginx (opens health window) |
Started end-of-deploy | nginx + php-fpm | php-fpm only |
ALB healthy when | end-of-deploy nginx start | end-of-deploy php-fpm start |
Making php-fpm the gate means protecting an invariant: php-fpm is stopped and disabled in the app-base AMI, and the only thing allowed to start it is that last task. Any other php-fpm state during a release is a regression, and regressions get in silently unless you build tripwires. So there are four defense layers, each guarding a different vector.
Layer 1, at bake time. Flush the Ansible handlers, then stop and disable php-fpm before the snapshot. The AMI ships with php-fpm both intent-disabled and process-stopped — belt and braces, so neither a stray
enable nor a lingering process survives into the image.Layer 2, the canary, at the top of the release role before any imports run. Stop php-fpm and assert it did not change. If php-fpm was already stopped — the expected state — the stop task reports
ok, not changed, and the assert passes. If something started php-fpm early, the stop task reports changed, the assert fires loud, a Slack alert goes out, and the ASG marks the instance unhealthy. This is the cheapest, meanest test in the whole system: it costs one task and it catches any future regression that tries to bring php-fpm up before its time, for free, forever.Layer 3, the last task. Restart and enable php-fpm. This is the single source of truth for "ready to serve." The ALB goes healthy from here.
Layer 4, failure propagation. If Ansible fails anywhere, the entrypoint marks the instance unhealthy and the ASG terminates it. The previous color keeps serving. This is the same lesson Part 2 paid for in production: a pipeline that reports success on a failed deploy is worse than one that crashes, because a crash is visible and a silent success routes users to broken code.
Put those together and the failure math is boring, which is the goal. If a release task fails mid-flow, php-fpm never starts,
/validate.php returns 502 the whole time, the ALB never goes healthy, the pipeline eventually times out at the 90-minute health-check ceiling, the entrypoint marks the instance unhealthy, the ASG terminates it, and the blue/green switch simply never happens. Zero customer impact, every path.Three gates as an ownership model
The layering also cut the release into three gates, and the useful thing about them is that different people own each one and they're decoupled in time.
Gate 1, PUBLISH — owned by the Packer pipeline. A build succeeds with a publishable AMI name, which is what makes it match the
data.aws_ami filter. If the build fails, or the cleanup provisioner runs, the AMI never gets that name and never enters the pool. This gate is machinery; no human touches it.Gate 2, PROPAGATE — owned by whoever holds the infra repo. An operator reviews the terraform plan and applies it, which updates the launch templates across environments to point at the new AMI. Crucially, this does not recycle running instances. Propagating an AMI just means "the next instance to boot will use this." Nothing serves it yet.
Gate 3, DEPLOY — owned by developers. A developer cuts a git tag —
test-X.Y, stage-X.Y, release-X.Y — which triggers the blue/green deploy in that environment. This is the moment the new AMI actually serves traffic.The gates run on their own clocks. An AMI gets built Saturday, propagated Monday, deployed to test Monday afternoon, to stage Friday, to prod the following Wednesday. And here's the part I like most: promotion is implicit in the order you cut the tags. There is no "promote to prod" command anywhere in the system.
stage-4.2 promotes 4.2 to stage because that's what the tag means and the pipeline is wired to that trigger. Nobody can fat-finger a promotion command because the command doesn't exist. The tag is the promotion.The bake-phase sentinel
One role, two very different contexts. The release role runs at Packer bake time — where it builds the release AMI — and it runs again at ASG boot, where a real instance comes up. Some tasks only make sense in one of those contexts. The clearest example: at deploy time, an instance looks up its sibling "socket" instance by querying
describe-instances, because the app needs to know where its socket peer lives. At bake time there is no sibling. There's one throwaway build instance and nothing to look up. Running that lookup at bake is meaningless and would hang.So the role needs to know which phase it's in, and the mechanism is a sentinel file. On the Packer side, a shell provisioner touches
/tmp/packer_ami_bake_phase right before the Ansible provisioner runs. On the Ansible side, an early task stats that file and sets a fact, ami_bake_phase. The socket-peer lookup is gated when: not ami_bake_phase. The vhost template checks the same fact: at bake it renders a 127.0.0.1:3000 proxy_pass fallback, and at deploy it renders the real dynamic socket host.The sentinel is a one-way signal, and it's cleanup-controlled so no stale state ships. The
/tmp cleanup wipes the file before the snapshot, which means a real deploy never sees it — a booted instance stats /tmp/packer_ami_bake_phase, finds nothing, and correctly concludes it's at deploy time.There's a security payoff bundled in here that's easy to miss. Bake always uses the test environment config. A build instance never holds prod credentials, because at bake there's no environment specialization at all — the sentinel makes the AMI env-agnostic. The specialization happens at boot, when deploy-time Ansible pulls the right
.env for whichever environment the instance actually belongs to. The AMI is generic; the instance is specific. No production secret ever touches a machine whose only job is to produce an image.The multi-week mystery
Now the war story, because every layering system has one failure mode that will cost you a week of your life the first time, and this was mine.
I'd fixed a set of latent bugs in the release role. Real fixes, verified correct — I'd read the code, reasoned through it, watched it do the right thing in isolation. Then I'd bake, deploy, and watch the old broken behavior come right back. I'd re-verify the fix. Still correct. Deploy again. Still broken. This went on, on and off, for weeks. The code was right and the code was not running, and both of those were true at the same time, which is the kind of sentence that makes you question your career.
The cause was filesystem state, and it was almost elegant in how quietly it ruined everything.
The app-base bake left the app's code directory — call it
/opt/app-cm — baked into the image. The release entrypoint then did mv /tmp/app-cm /opt/app-cm to drop in fresh code. But under POSIX, when you mv a directory into a path that's already an existing directory, it doesn't replace it. It nests inside it. So mv /tmp/app-cm /opt/app-cm produced /opt/app-cm/app-cm/ — the fresh code tucked one level down — and the very next line, cd /opt/app-cm, walked straight into the stale baked-in code and ran that. Every verified-correct fix I'd shipped landed in a nested directory that nothing ever executed.The fix is one line,
rm -rf /opt/app-cm before the mv, in both entrypoints, plus a matching rm in the app-base final cleanup so the baked copy doesn't linger. Trivial once you see it. Invisible until you do.The lesson is worth more than the fix. When code you've verified as correct appears not to run, stop re-reading the source. The source is not where the bug is. Suspect the state of the thing consuming the code — the filesystem the deploy lands on, the cache in front of the runtime, the directory that was supposed to be empty and wasn't. This is the exact same class of bug as the stale opcache from Part 2: correct code deployed on top of a consumer that was quietly ignoring it. Twice now that pattern has cost me days. I've learned to check the consumer's state first.
While I'm handing out debugging scars: there's a Mitogen gotcha in the same neighborhood. Under
strategy = mitogen_linear, Ansible-level error suppression — block/rescue, failed_when, ignore_errors — is silently bypassed for systemd module failures. You write what looks like a guarded, tolerant task, and Mitogen routes around your guard for that specific module. If you run Mitogen for the speed, know that your error-handling around systemd isn't doing what the playbook says it's doing.Testing failure, not just success
Most deploy test suites prove the happy path works. That's necessary and it's the easy half. The half that actually caught the May incident's ghosts was testing what happens when things go wrong, on purpose.
There's a synthetic regression canary. On a throwaway branch, I add a task that deliberately starts php-fpm mid-deploy — exactly the regression Layer 2 exists to catch. The next deploy's canary has to fail loud, fire the Slack alert, mark the ASG instance unhealthy, and leave the previous color serving. If it doesn't, the tripwire is decorative and I want to know before I'm trusting it in prod, not after.
There's a fail-injection test. Drop a bare
fail: task before the blessed php-fpm start. php-fpm never comes up, /validate.php stays 502 throughout, the ALB stays unhealthy, the pipeline reports failure, and no traffic switch happens. This proves the failure-propagation path — Layer 4 — actually propagates, rather than doing the Part 2 thing where a broken deploy reports success.And then two items on the Definition of Done that I want to single out, because they're about psychology as much as engineering.
The first: at least one upstream package outage handled gracefully in the wild — or deliberately injected. Not "we believe failure isolation works." Not "the design implies it works." One real, observed instance of nginx.org or Galaxy or the Ubuntu archive falling over while releases kept shipping, witnessed and logged. A design isn't proven by its author's confidence in it. It's proven by watching it eat a real failure and not flinch. If the wild won't provide one on schedule, inject it — the point is the observation, not the source of the outage.
The second is the one I think about most. Delete the emergency AMI snapshot.
The emergency AMI from May still existed. It was tempting to keep it around "just in case" — a warm fallback, a security blanket. But the whole reason Gen 3 exists is that the emergency AMI was a mistake we should never be able to make again. As long as that snapshot sits in the account, there's a path back to the old failure mode, and under enough 3am pressure someone will take it. Deletion isn't cleanup. Deletion forces commitment. It removes the escape hatch so that when the next bad night comes, the only way out is forward through the system we actually built. I put it in the Definition of Done as an explicit line item, because "we'll delete it later" is how a two-day workaround lives for weeks — which is precisely the mistake that started the whole incident in Part 2. The commitment isn't real until the fallback is gone.
What it didn't fix
I want to close honestly, because Gen 3 was a large improvement and also not the end of the road, and the next two parts of this series exist because of what it left on the table.
Blue/green is still here, and blue/green still means a six-ASG state space per environment: three instance types times two colors. Every deploy still shuffles that state, and reasoning about it is still more work than it should be.
There are still minutes of Ansible running at instance boot. The release AMI is faster because the slow layer moved to the weekly bake, but the fast layer is still Ansible pulling code from S3, rendering vhosts, rebuilding config cache, and starting services at boot. 3–5 minutes is better than 10, but it isn't zero, and every one of those minutes is a place a boot-time deploy can still fail.
And rollback is still pipeline-dependent. To back out a bad release you deregister the AMI, let
most_recent resolve to the previous one, review the terraform plan, apply, and re-trigger. That works, and Part 6 is a whole article on why I'm oddly proud of it, but it's a procedure with steps, not a pointer you flip.Every one of those is a thread Gen 4 pulls on. If the slow layer being baked once a week is good, what happens when you bake everything — one fully-baked AMI per commit, promoted across environments, with zero configuration at boot? What happens to the six-ASG state space when you replace blue/green with a single ASG and instance refresh? And what does rollback look like when the release pointer is just a value in SSM Parameter Store that you flip?
That's Part 4, where I audit my own blue/green driver script line by line before daring to replace it — and it's the setup for the pointer model in Part 5. The layering was the moment I stopped making the dependency more reliable and started making it disappear. Turns out that instinct had further to run.