Rollback is a feature: drills, alarm gates, and an incident mode

Part 6 of a series on ten years of deployment evolution for one legacy app on AWS — no Kubernetes required. Series index.

Every deploy system I have ever inherited had a rollback plan written down somewhere. Almost none of them had a rollback plan that had been run since the last time the deploy system changed. Those are two different things, and the gap between them is where a lot of bad afternoons come from.

The system I want to talk about here is the deployment stack for a workforce-management product — a ~10-year-old PHP/Node monolith on EC2, three environments, a DR region, and a small platform team. Over ten years its rollback story went from "re-run the tool and pray" to something I am comfortable calling a feature: designed on purpose, gated by alarms, drilled on a cadence, and given a dedicated incident mode that exists precisely because the naive version breaks at the worst possible moment.

The thesis is simple. An untested rollback is not a rollback. It's a hope. And hope does not have a measured recovery time.

Rollback across four generations

Here is how "get me back to the last good release" changed across the four deployment generations I covered in Part 1.

Gen	Rollback mechanism	Recovery cost	External dependency
1 (host + scripts + CodeDeploy)	re-run the CodeDeploy deployment against the previous revision	minutes, manual, per-host uncertainty	CodeDeploy agent + revision still present
2 (Packer + Ansible + blue/green)	re-tag the previous commit, re-run the pipeline	~10 min, non-deterministic (rebuild from git)	CI must be online and healthy
3 (split-AMI blue/green)	same as Gen 2, faster boot	~3–5 min	CI must be online and healthy
4 (baked AMI + instance refresh)	flip the SSM release pointer, refresh onto the previous AMI	first good instance ~2 min, full fleet ~5–7 min	none (AMI already exists; incident mode bypasses gates)

The interesting movement is in the last two columns. Recovery cost dropped, which everyone celebrates. But the column that actually changed the on-call experience is the dependency column. In Gen 2 and Gen 3, rollback was a rebuild: to get back to a known-good release you re-tagged the old commit and asked the pipeline to reconstruct it through composer, npm, and gulp. That works right up until the reason you are rolling back is also the reason CI is unhappy, or the upstream package mirror that broke your deploy is still broken (see Part 2 — that exact week, four upstream repositories went down together). A rollback that depends on rebuilding from source is a rollback with a hidden clause: assuming the build still works.

Gen 4 removed the clause. The previous release is already a baked AMI sitting in the account. Rolling back is relaunching instances onto an AMI that already exists — no composer, no npm, no gulp, no git. And because I stopped trusting that "just relaunch the old thing" is automatically safe, Gen 4 also grew an explicit incident mode, which is most of this article.

I built Gens 2, 3, and 4, and later audited my own work on each (Part 4 is the audit war story). So when I say the earlier rollback stories were fragile, that is not a vendor slide. That is me reading my own old scripts and wincing.

The deploy-gate alarms: a short list on purpose

Before rollback, talk about what stops a bad deploy in the first place, because the two are the same machinery pointed in opposite directions.

Three CloudWatch alarms have the authority to block or reverse a deploy:

a target-group 5xx alarm on the web tier,

a target-group latency alarm on the web tier,

a 5xx alarm on the socket tier's target group.

That's the whole list. Three alarms, per environment, sized from that environment's real traffic. During an instance refresh they are attached to the refresh as its AlarmSpecification: if any of them goes to ALARM while the refresh is in progress, AWS rolls the refresh back to the previous launch-template version automatically. Between deploys the same three alarms keep watching the release target groups, fed by a standing trickle of synthetic canary traffic, so an ALARM outside a deploy window tells me the running release is misbehaving, not the deploy machinery.

The list is short and I intend to keep it short. Every alarm with deploy-blocking authority is a potential spurious block: a flappy alarm doesn't just page you, it can refuse your deploys and reverse your refreshes. The bar for putting an alarm on the gate is therefore high. It has to be a signal that (a) means "users are being hurt right now" and (b) shows up fast enough to catch a bad release inside the refresh window. 5xx rate and latency clear that bar. CPU, memory, queue depth, disk — all useful, none of them get a vote on whether a deploy proceeds. They are operational signals, not deploy gates, and conflating those two categories is how you end up with a deploy pipeline that blocks itself every time a batch job runs hot.

One more property matters for what follows: the driver refuses to start a normal deploy while a gate alarm is already firing. Perfectly sensible for a routine deploy. It becomes a problem during an incident, and I'll get there.

Auto-rollback vs manual rollback: draw the line and keep it sharp

There are two rollback mechanisms in Gen 4, and the most important design decision was to keep them in separate lanes.

	Alarm auto-rollback	Manual rollback
Active window	only while a refresh is in progress (including the post-refresh soak) — roughly 13–14 minutes for a prod deploy	any time
Trigger	gate alarm goes to ALARM; AWS restores replaced instances to the previous launch-template version; the driver rewinds the pointer and re-pins	a human decides
Covers	bad releases that show up immediately: a 5xx burst, a latency cliff	problems that surface after the deploy window: data-dependent bugs, slow leaks, tenant-specific code paths, business-logic errors that no 5xx alarm will ever see
Human involvement	zero	one pipeline run, or one command

The division of labor is one sentence: inside the deploy window it's automatic; after the window, it's yours. Once a refresh finishes successfully the alarms keep watching the target groups, but nothing rolls back on its own anymore. That is deliberate. Automatic reversal is only safe when the system knows what "before" was and can restore it atomically — which is exactly the situation during a refresh, and exactly not the situation an hour later when the fleet has scaled, instances have cycled, and "the previous version" is a question rather than a fact.

Blurring this line is where I have seen teams get hurt. Push auto-rollback past the deploy window — "just keep the alarms armed and let them roll back whenever" — and you get a fleet that flips itself between two releases every time an alarm blips, with no human deciding which release is actually correct. Pull manual rollback into the deploy window — "the operator will watch and pull the cord" — and you have signed a human up to out-react a 17-second automated response, at 3am, under stress. Each mechanism is good at exactly one job. The machine is good at the fast, mechanical, known-state reversal inside a bounded window. The human is good at the judgment call about a bug the machine cannot see. Neither is good at the other's job, and pretending otherwise is how rollback tooling earns its bad reputation.

There is a small interaction surface between the two, and I drilled it rather than reasoned about it. A manual rollback fired while an auto-rollback is already in flight cannot collide: the driver waits for any active refresh to reach a terminal state before it does anything, and it queues on a deploy lock if the failing deploy's driver is still running. Worst case, you wait, and then your manual rollback hits a same-version no-op guard — because the auto-rollback already restored that version — and exits harmlessly in a few seconds. That "harmlessly" is a claim I get to make because I ran it, not because the state machine diagram looked right.

Incident mode and the alarm-bypass paradox

Here is the trap that gives this whole article its reason to exist.

During a real incident, your gate alarms are firing. That is what an incident is: the release you shipped is throwing 5xx or blowing latency, and the alarms you carefully built to catch bad releases are doing their job. Now you go to roll back.

A naive rollback — "just run a deploy of the previous version" — walks straight into two of its own safety gates:

The driver refuses to start a deploy while a gate alarm is firing. That refusal is correct for routine deploys and completely wrong right now: the alarm is firing because of the release you are trying to escape.

If the rollback did start and it attached the gate alarms as its AlarmSpecification like a normal refresh, the alarms — still firing, because the bad release is still serving during the swap — would trip AWS's auto-rollback and revert your rollback back onto the bad release. Your safety system would "roll back the rollback."

So the safety machinery that makes normal deploys safe actively obstructs recovery during the one event it was all built for. This is not a bug you patch. It's a category difference between "deploying a new, unproven release" and "returning to a known-good one," and it needs its own code path.

That path is the incident rollback mode. In the tooling it's a --rollback flag; operationally the default entry point is a dedicated one-purpose pipeline (Target + the known-good version, both prefilled into the red failure alert in Slack), with a CLI fallback for when the CI provider itself is down — because a rollback tool that depends on the thing you might be rolling back is not much of a rollback tool. The mode differs from a normal deploy in three specific ways, each with a reason:

Single batch, no canary checkpoint. The target is a release that already ran in production. Canary-observing it just prolongs the bad release's time on the wire. Get the whole fleet back, fast.

Tiers in parallel. The target's migrations ran long ago, and the schema policy is expand/contract, so the old code is safe against the current schema. There is no ordering constraint to respect.

Gate alarms neither pre-checked nor attached. This is the crux. Not pre-checked, so the in-alarm start refusal cannot block recovery. Not attached, so AWS cannot revert the rollback onto the bad release. The health-check auto-rollback stays armed — if the known-good AMI somehow fails its health checks, you want to know — but the 5xx/latency gates are deliberately out of the loop for the duration of the rollback.

That last point is the one that makes people nervous, and it should, which is why I did not ship it on the strength of an argument. I drilled it. I forced the target-5xx alarm to ALARM with set-alarm-state and ran two things back to back. First the negative control: a normal deploy under that forced alarm, which the driver correctly refused with an in-alarm error. Then the real thing: --rollback under the same forced alarm, which started and completed normally because it attaches no alarm spec. First known-good instance serving at +2m14s on a four-instance fleet; full fleet converged in 5m30s. A separate run on a smaller fleet, with the alarm forced to ALARM for the entire duration, converged in 6m49s. The bypass is not a paragraph in a design doc. It's a measured, negative-controlled drill result.

Which brings me to the reason I trust drills over diagrams at all, and it comes straight from Part 2.

During that cascading incident, a broken instance was serving traffic and the deploy machinery tried to do the right thing: the failure-notify path called aws autoscaling set-instance-health --health-status Unhealthy to pull the bad instance out. It got rejected. The instance was still inside the ASG's health-check grace period, and AWS will not let you mark an instance unhealthy during grace. The failure-propagation path — the code whose entire job was to take a broken instance out of rotation — silently failed at the exact moment it was needed, because nobody had ever exercised it against a fresh instance inside its grace window.

That is the lesson that built the incident mode. A failure-handling path you have not run under failure conditions is not a safety feature. It is an assumption wearing a safety feature's clothes. The --rollback alarm bypass is a failure-propagation path just like set-instance-health was, and I refused to ship it on reasoning for the same reason I no longer trust set-instance-health on reasoning. I forced the alarm and watched the bypass work, with a negative control proving the gate still blocks the thing it should block.

Drills: what they look like, what they actually caught

A rollback drill in test or stage is not glamorous. You take a fleet, you put it on release A, you deploy release B, and then you provoke a failure and watch the system come home. The provocation is usually set-alarm-state forcing a gate alarm to ALARM mid-refresh; other runs use a mid-refresh cancel-instance-refresh, a deliberately corrupt payload, or yanking an instance's health to watch the ASG replace it. The point is not to confirm the happy path. The point is to make the failure paths run, because those are the ones that never run on their own.

Here is what my drills actually caught, none of which I would have found by reading code:

Auto-rollback restores instances but not the launch template. The single most valuable thing a drill ever told me. When AWS auto-rolls-back a refresh, it restores the instances to the previous launch-template version — but it leaves the ASG's pinned LT version and the launch template's $Default pointing at the version you just rolled back from. Everything looks fine: traffic is healthy, instances are running the old good code. Then the ASG scales out at 3am, launches a fresh instance on the still-forward $Default, and silently serves the exact release you thought you had rolled back. The drill surfaced this by forcing an alarm rollback and then checking the pin against the running instances. I would never have caught it in a code review because the reverting is AWS's job, and AWS does exactly what it documents — just not the part I assumed. The fix was to make the driver re-pin the ASG and the launch-template default to whatever the instances actually run, at the end of every failed run. More on that state in the next section.

The re-pin picked the wrong version. Once re-baking a version could produce multiple launch-template versions for the same release, a cancel drill caught the cleanup code re-pinning to the oldest matching version instead of the newest, because the AWS API returns versions newest-first and the code took the last element of the list. Latent until re-bakes made duplicate versions possible; found by a cancel-mid-refresh drill; fixed by taking the max instead of the last.

A refresh can wedge in Pending forever. If a gate alarm happens to be in ALARM at the exact instant a refresh initializes, the refresh never self-starts — and it does not start later when the alarm clears. It just sits in Pending with no StartTime. The response (cancel, let the driver's cancelled-state repair run, redeploy) only exists in the runbook because a drill hit it. The window is tiny and only an injection-style drill will land on it, which is precisely why only a drill found it.

The cadence argument follows directly from that list. Every one of those bugs was introduced or exposed by a change to the system — a new re-bake capability, a new capacity-adaptive batching mode, a new alarm wiring. A rollback you last drilled two architecture revisions ago is not a validated rollback. It's a validated rollback of a system that no longer exists. My standing rule is a quarterly --rollback game-day, plus a mandatory re-drill any time the deploy driver, the alarm set, or the ASG topology changes. It sounds like a lot until you price it against discovering the LT-pin bug during an actual incident, at the next scale-out, when you are already down.

Post-rollback state: the unglamorous part

Most rollback documentation ends at "traffic restored." Mine doesn't, and this section is why.

When AWS auto-rollback finishes, the world looks almost correct and is quietly wrong in one place. The instances are back on the previous release. The load balancer is happy. The alarms have cleared. And the ASG's pinned launch-template version, plus the launch template's $Default, are still sitting on the version that was just rolled back from. Nothing serving traffic reflects that. It's a landmine, not a fire: it does nothing at all until the next scale-out or instance replacement, at which point a brand-new instance boots the release you thought was gone, and now half the fleet disagrees with the other half about what's deployed.

The driver repairs this automatically at the end of a failed run: it reads what the instances actually run and re-pins the ASG and the launch-template default to match. But if the driver was killed before it got to the repair — someone Ctrl-C'd it, the runner died — you fix it by hand, per tier: read the launch-template version off a running instance, set the ASG to it, set the launch template's $Default to it. Three commands. Skipping them means the next replacement "silently launches the rolled-back-from release," which is a sentence I put in the runbook in bold because it is the single most surprising thing about the whole system.

The other half of post-rollback state is the release pointer and the AMIs, which I covered in depth in Part 5. Briefly: the SSM pointer is the source of truth for "which version is this environment," and a rollback rewinds it. Any previously baked version redeploys with zero rebuild because its AMI is still in the account — retention keeps roughly the last ten per environment, so "roll forward again" after a rollback is the same fast operation as the rollback itself, found by an AMI tag rather than reconstructed from source.

And migrations are never rolled back. The schema policy is expand/contract: old code runs fine against newer schema, so there is no such thing as "rolling back a migration." That is not a convenience, it is the load-bearing assumption that makes one-command version rollback safe at all. If a developer ever ships a migration that drops a column the previous release still reads, the entire rollback story collapses — the old code you roll back to will fault against the schema. So the migration contract is written down as team policy, front and center, and it is the first thing I tell anyone joining the deploy rotation. Rollback safety is not a property of the deploy tool. It is a property of the deploy tool plus the discipline that every schema change is backward-compatible with the release before it.

That is the unglamorous part. Traffic being restored is the visible 90%. The invisible 10% — the drifted launch-template pin, the pointer, the expand/contract contract — is the part that bites you a week later if the docs stopped at "restored."

Guardrails beyond deploys: know your ceilings

The same philosophy that makes rollback a feature shows up in a place that has nothing to do with deploys, and it's worth a short detour because it's the same idea.

During the cascading incident in Part 2, one of the compounding failures was connection-limit exhaustion on the managed MongoDB. Elevated traffic plus concurrent blue/green deploys — which briefly double the fleet, and therefore double the PHP-FPM workers, and therefore double the open connections — pushed past the 500-connection hard cap of the tier we were on. Connections are not a graceful resource. You don't degrade as you approach the cap; you hit it and things start refusing.

The fix had two parts. The obvious part: upgrade to the 1,500-connection tier, buying headroom. The part that actually mattered: a Slack alert at 70% of the cap, around 1,050 connections. The tier upgrade bought room; the alert bought warning. The whole failure mode of a hard cap is that it's invisible until you're on top of it, so the engineering move is to make the ceiling visible well before you reach it — 70% is far enough out to react, close enough that it isn't crying wolf.

That is the same instinct as rollback drills, pointed at a different resource. A hard limit you don't monitor is a hard limit you will discover by hitting it, in production, at peak. Whether the limit is database connections, ASG max size, account vCPU quota, or subnet IP space (the last two became real constraints once fast boots let the fleet temporarily exceed its own max size during a refresh — a topic for Part 7), the rule is the same: know your ceilings before you hit them, and put an alarm at 70% of every hard cap you can name. Capacity planning and rollback drills are the same discipline. Both are about refusing to let the first time you meet a failure mode be the time it's also an outage.

A rollback-runbook skeleton you can steal

Here is the shape I've settled on, generic enough to port. Fill in your own commands; keep the structure.

Steps 5 and 6 are the ones that separate a runbook someone wrote from a runbook someone ran. "Traffic restored" is step 5's first line, not its last. And step 6's re-drill note is the whole cadence argument compressed into one line: the runbook decays every time the system underneath it changes.

Rollback earned its way into being a feature the same way anything does. It got designed on purpose, gated by a deliberately short list of alarms, given a separate incident mode for the case where its own safety gates turn against it, and drilled — with negative controls — until the measured recovery time was a number I'd stake an on-call shift on. An untested rollback is a hope. A drilled one is a feature. The difference is entirely in whether you've run it since the last time you changed the thing it rolls back.

In Part 7 I'll get into the part I've been waving at all article: the capacity-adaptive instance refresh that makes all of this work across a one-instance test pilot and a six-instance prod fleet without a single hardcoded batch size — including the drill where launch-before-terminate ran six instances against a max size of four, on purpose, and why that's a feature too. It's the most number-dense piece in the series, which is exactly how I like them.