Part 5 of a series on ten years of deployment evolution for one legacy app on AWS — no Kubernetes required. Series index.
Here is the whole argument, up front, so you can decide whether the rest is worth your time. A deployment system is a pointer plus a convergence mechanism. Everything else is decoration. Store the release SHA somewhere durable; that is the pointer. Let a control loop drag reality toward it; that is convergence. Once you have those two things, "deploy" and "rollback" are the same verb: write a different value to the pointer and let convergence run.
I did not start out believing this. I got there by deleting a working system that had accreted too much state, and being surprised by how little was left.
The old world was not bad. It was full of state.
The system I replaced was a blue/green deployment built on ALB weighted target groups. I built it myself about four years ago, and for its era it was the right design. I am not going to trash it here — I already told that story in Part 3 (the split-AMI layering that made it faster) and Part 4 (the audit where I read my own driver script line by line and found the bodies). If you want the full anatomy, read those. This section is only about one property of that system: how many distinct states it could be in.
Count them with me. Each environment had two colors — blue and green. Each color had three ASG types: a normal web tier, a socket tier, and a command/cron tier. That is six ASGs per environment. In front of them sat an ALB with weighted target groups, so at any instant traffic could be 100/0, 0/100, or anything between while a switch was in flight. Which color was "live" was not stored in one place — it was implied by two sources at once: which ASG had running instances, and which target group carried the ALB weight. When those two sources disagreed, you had a silent stale deploy, and figuring out which was authoritative was a manual investigation.
On top of that state sat the logic to keep it consistent: color decision (which side do we deploy to next?), a "next color must be empty" pre-assertion, health gates that had to wait for the full desired count on the new side before flipping weight, partial-switch rollback for when the socket tier switched but the normal tier did not. By the time I finished auditing it, the two driver scripts came to roughly 1,400 lines of bash whose entire job was to keep this state machine from lying to itself.
None of those 1,400 lines were stupid. Every one existed because some real failure mode had bitten once and someone wrote a guard. That is exactly the problem. The state space was large enough that it kept generating new failure modes, and each new guard enlarged the surface that could go wrong. The system worked. It also could not stop growing.
The pitch of the pointer model is embarrassingly small: shrink the state space to one value. Not "manage the state better." Delete the state.
The prerequisite: an artifact that carries zero configuration
You cannot collapse a deploy to a pointer flip until the thing the pointer points at is genuinely interchangeable across environments. That is the piece Part 1 set up, so I will only recap it in a paragraph.
For years our release AMI pipeline already ran the full application build at bake time — the app code was physically inside the image. Then at every single instance boot, the same automation wiped the web root and redeployed the app from scratch: resolve the latest git tag, pull a zip from S3, unzip, render config, start services. We baked a perfectly good artifact and then threw it away, every boot, using the baked copy only to warm the page cache. The move in Part 1 was to stop throwing the bake away: pin the bake to an explicit commit SHA, so the artifact is the AMI, and shrink boot to a sub-60-second "activation" step that fetches secrets and starts services. Boot dropped from 5-plus minutes to about 90 seconds.
The property that makes the pointer model possible is that the AMI is env-agnostic by construction. Everything environment-specific — the
.env, the vhost URL, the socket peer address — was already resolved at boot, not at build. So one bake serves test, stage, and prod: what soaked in test is byte-identical to what runs in prod. Build once, promote many. The first environment to see a SHA pays for the bake; every later environment looks the AMI up by its GitCommit tag and skips straight to convergence. That single fact — one artifact, promoted, not rebuilt per environment — is what lets the pointer be a single value instead of one-per-environment-with-a-rebuild-in-between.The pointer mechanics
Three moving parts, and only three.
The pointer is an SSM Parameter Store parameter holding the release SHA for an environment:
Parameter Store keeps history for free, so the parameter's version history is the release history for that environment — who deployed what, when, in order, queryable with one API call. No separate deploy-ledger to maintain.
The launch template is where convergence reads the pointer. In the baked model the launch template's default version carries the concrete AMI ID. I want to be honest about the shape here, because it matters for a senior audience: there are really two coupled pointers. The SSM parameter is the human-and-audit pointer and the rollback source of truth; the launch template default version is the machine pointer that a newly launched instance actually boots from. A deploy advances both. I deliberately did not wire the launch template to read the SSM parameter live (via
resolve:ssm:), and section 6 explains why that would quietly break rollback.Instance refresh is the convergence mechanism.
start-instance-refresh is an AWS-native rolling replacement: it cycles the ASG's instances in batches, launching new ones on the current launch template and terminating old ones, respecting health checks and an optional alarm specification, and — the part that earns its keep — automatically rolling back to the previous launch template version if a CloudWatch alarm fires mid-refresh. I did not have to write the rolling logic, the batching, the health gating, or the auto-rollback. Those are the platform's job now.Here is the whole driver, generalized and cleaned up from what I actually run. The design sketch I started from was about 60 lines; I will be honest below about what the real thing grew into, but the spine is still this short:
Read what the platform hands you for free in that
refresh function. MaxHealthyPercentage=200 gives you launch-before-terminate, so capacity never dips during a deploy, including on the single-instance socket ASG. The AlarmSpecification wires a 5xx/latency breach directly to an automatic revert to the previous launch template version — auto-rollback with no code of mine involved. And a second start-instance-refresh against an ASG that is already refreshing is rejected by the API, which means the entire "is a deployment already running?" problem class — the thing those 1,400 legacy lines spent so much effort guarding — simply does not exist. The concurrency lock is the platform's, not mine.The
CheckpointPercentages=[50,100] on the normal tier is the production canary: replace half the fleet, pause 180 seconds while the gate alarms watch real traffic on the new release, then finish. If the release is bad, the refresh rolls back with half the old fleet still standing.Deploy and rollback are the same verb
Deploy is: write the pointer, run the refresh. That is what the script above does.
Rollback is: write the old SHA, run the refresh. Same script, different argument:
There is no rollback code path. Rollback is a deploy that happens to target a SHA you ran before. Because every SHA you ever deployed still has its baked AMI sitting in the account (retention keeps the last ten per environment, and snapshots are incremental, so this costs pennies), rolling back is not a rebuild — it is relaunching instances from an image that already exists. Measured on a four-instance fleet: first known-good instance serving in about two minutes, full fleet converged in five and a half.
Now the sentence that is the reason this article exists, the one I would put on a slide if I gave this talk:
Your rollback path should not depend on your CI vendor's uptime.
In the blue/green world, rollback meant re-tagging the previous commit and re-running the pipeline. That pipeline rebuilt the application from git (composer, npm, gulp, the whole toolchain), which took upwards of twenty minutes and was non-deterministic, because "rebuild from source" resolves dependencies at build time and dependency resolution is never truly pinned. Worse, it required Bitbucket Pipelines to be up. If your CI is degraded, and CI is degraded far more often than AWS's EC2 control plane is, your rollback is degraded with it at exactly the moment you most need it. You have coupled your emergency brake to a third party's status page.
The pointer model breaks that coupling. Rollback is three AWS API calls: bump the launch template to an AMI that already exists, start an instance refresh, write the SSM parameter. It runs from a laptop with an admin role and the AWS CLI. It does not build anything. It does not clone the repo. It does not ask CI for permission. The default entry point is still a browser button for convenience — a custom pipeline that any developer can run, prefilled from the failure alert — but the button is a convenience, not a dependency. When the button's host is down, the CLI path is right there, and it is the same three calls.
That is the headline. Everything below is me being honest about what the one-liner hides.
What the one-liner hides
A deploy that collapses to "change one value" is a lovely story, and if I stopped there I would be selling you the same over-simplification that every deployment-tool landing page sells. Three things do not fit inside the pointer flip, and pretending otherwise gets people paged.
Migrations move forward; the pointer moves both ways. During every deploy, and after every rollback, the other release's code runs against your current schema. When you deploy, old instances serve traffic against the new schema until they cycle out. When you roll back, the old code you are rolling back to runs against a schema the newer release already migrated forward — because rollback does not reverse migrations, and it should not. The only thing that makes bidirectional pointer movement safe is that schema changes are expand/contract: additive, backward-compatible, never dropping or renaming a column the currently-serving release still reads in the same release that stops using it. This was always the policy under blue/green too. The difference is that under blue/green, a botched migration mostly hurt during the deploy window and you could often muddle through. Under the pointer model, one-command rollback is a headline feature, and expand/contract is the load-bearing assumption that makes that feature safe. The discipline did not get harder. Its consequences got sharper. I now treat "is this migration backward-compatible?" as a release blocker, not a code-review nicety.
In the current implementation the socket tier runs the migration once per deploy as part of its activation, and it refreshes first for exactly that reason: the new schema must be in place before the new web code takes traffic. A cleaner follow-up runs migrations as an explicit pipeline step before any instance is touched, so a migration failure stops the deploy before a single server is replaced. Either way, the ordering constraint — migrations before new web code — is real work the pointer flip does not show you.
Post-rollback, the fleet is in a subtly wrong state. This one is in my runbook under a heading that just says "important," because it bit me in a drill. When AWS's auto-rollback fires mid-refresh, it restores the instances to the previous launch template version — but it leaves the ASG's pinned launch template version and the launch template's
$Default pointing at the version it just rolled away from. So the running instances are correct, and the next scale-out or instance replacement silently launches the bad version again. The instances lie about being fixed. My driver repairs this at the end of a failed run by re-pinning the ASG and the default to whatever the instances are actually running, judged by reading an instance's ImageId rather than trusting the launch template metadata. But if the driver is killed before it repairs, you have to do it by hand, per tier, and a tier with zero running instances (typically the cmd tier) has no instance truth to read from and must be re-pinned manually. "Auto-rollback" restored service; it did not restore the pointer. Those are two different claims and the gap between them is an incident waiting for a 3am scale-out.Replacing a defective AMI for a SHA is not a redeploy. Now and then the artifact itself is wrong — the AMI baked for a SHA is defective, not the code. (This happened to me: a bake-purity bug shipped a set of stray service files into every release AMI for a stretch, poisoning early boot until I caught it.) You cannot fix that by flipping the pointer, because the pointer already points at that SHA and the AMI it resolves to is the broken one. The fix is to re-bake the same SHA — which is safe, because the driver always resolves the newest available AMI for a SHA — and then let the convergence callback adopt it by flipping the launch templates onto the new image with no refresh. Running instances keep serving; the next launch boots the clean image. That is a real procedure with its own runbook section, and it is invisible from the "just change the pointer" framing because the pointer never changed. The thing under the pointer changed.
Dirty edge cases worth stealing
These are the sharp corners that do not make it into the architecture diagram. I am including them because they are the parts I would actually want from someone else's writeup.
Deriving the SHA from a tag is a trap. The deploy is triggered by a git tag, and early on I resolved the version by asking git which tag contained the commit.
git tag --contains can return the wrong tag when one commit carries multiple tags, and a shallow clone — which CI runners love, for speed — can miss the tag entirely and hand you a garbage value that only explodes three steps later, deep inside an AWS call, with an error that names none of this. I moved to deriving the version directly from the triggering tag ref and failing immediately on an empty value. There is a related, quieter trap: our CI build once ran rm -rf .git mid-pipeline for cleanup, which destroyed the repo before a later step needed git history to detect whether the release contained migrations. It worked locally (where the .git survived) and silently mis-fired in CI for weeks, forcing every CI deploy into the slower serial path. The lesson I keep relearning: when verified-correct logic behaves differently in CI than on your laptop, suspect the filesystem state the consumer sees before you suspect the logic.Fail-fast format validation earns its keep. The pointer's value is a short SHA, and a short SHA has a shape. Validating that shape at the top of the driver — reject anything that is not seven hex characters — turns a class of latent disasters into an immediate, loud, harmless exit. The value I most wanted to catch was the literal string
"None", which is what a failed AWS CLI query returns as text; without a guard, "None" sails into a put-parameter call and becomes your release pointer, and now every instance that launches tries to boot a release called None. Validate the pointer before you write it. It is three lines and it has saved me twice.The pointer write and the refresh are deliberately separate steps. It would be tidy to fuse them — one command that writes the pointer and converges the fleet atomically. I kept them apart on purpose, because a pointer write with no refresh is a useful operation in its own right: it stages a deploy. Write the pointer, do not converge, and the currently-running fleet keeps serving the old release while the next instance to launch — a scale-out, an auto-replacement — comes up on the new one. I use exactly this to repair a broken environment without a disruptive rollout, and to fix a legacy scale-out path that was resolving a release whose artifact had gone missing: write the pointer to the last known-good SHA, touch nothing running, and the next launch heals itself. The two steps answer two different questions. The pointer answers "what is the desired release?" The refresh answers "make it true now." Now and then you want the first without the second, and separating them is the only way to have that.
The honest before/after
This is the comparison that matters, blue/green versus the pointer model, with the numbers measured after full cutover in the test environment and a couple dozen live drills — not design-time predictions:
Blue/green (Gen 2) | Pointer + refresh (Gen 4) | |
Deploy, tag → serving | ~10 min target | 6m59s (fresh build), 6m10s (pre-baked) |
Rollback path | re-tag + re-run pipeline; rebuild from git ~20 min, non-deterministic | write old SHA + refresh; ~2 min to first good instance, 5m30s full fleet |
Rollback depends on CI? | yes — needs Pipelines up | no — three AWS API calls |
Auto-rollback on bad release | none | native, alarm-gated, ~17s to start |
ASGs per env | 6 (2 colors × 3 tiers) | 3 |
"Is a deploy running?" guard | hand-rolled ASG-emptiness checks | AWS rejects overlapping refresh |
Version on scale-out | newest matching tag at that moment (a real race) | pinned AMI — deterministic |
Self-healing dead instance | none | ELB health, replaced in ~2.5 min |
Driver size | ~1,400 lines of bash | ~60-line spine (~1,200 hardened) |
Two things in that table need honesty, not cheerleading.
The driver did not actually shrink to 60 lines and stay there. The spine is 60 lines; the real thing is around 1,200, because I added capacity-adaptive batching, a deploy lock, per-tier failure repair, eager-deploy handling, and a dedicated incident-rollback mode. So did I really delete complexity, or move it? I moved some and deleted most. The 1,400 legacy lines existed to manage state I no longer have — colors, weights, dual-source truth. The 1,200 new lines exist to drive AWS primitives that now carry the batching, health-gating, concurrency, and rollback that the legacy script implemented by hand. The complexity that remains is orchestration of platform features. The complexity that is gone is a bespoke state machine. That is a real trade even though the line count is not the win it looks like.
And what got riskier, because everything real has a cost ledger. The mixed-version window is longer than blue/green's instant weight flip: a rolling refresh runs both releases side by side for several minutes instead of switching in one atomic step. I mitigated it with one-hour session stickiness, so no individual user ever bounces between releases — the residual "mixing" is user A on the new release while user B is still on the old, which is the literal definition of a canary and exists in every rolling or weighted deploy ever built. The other honest cost: AMI sprawl (bounded by retention), and the fact that the very first environment to see a SHA now waits for a bake (offset by boots that dropped from five minutes to ninety seconds). I traded blue/green's theoretical warm-fleet instant flip — which in practice was already near zero, because the legacy pipeline tore the old fleet down with
sleep 0 — for a rollback that beats the legacy real fallback by fifteen minutes and does not need CI to exist.Count your states
Here is the generalization, and then I will let you go.
Sit down and count the distinct states your deployment system can be in. Not the happy path — all of it. Blue live at 100. Green live at 100. Weight mid-flip. Blue's ASG full but green holding the weight. A partial switch where the socket tier moved and the web tier did not. A scale-out that raced a teardown. Every one of those is a state, and every state is a configuration your automation has to recognize, reason about, and correctly leave. Most of your incidents do not live on the happy path. They live in the states you forgot were reachable.
The pointer model is not magic and it is not new — it is what every well-built control loop already does. Its whole value is that it makes the reachable-state count small enough to hold in your head. The pointer has one value. Convergence is either running or it is not. That is close to the entire state space, and I can enumerate it out loud. Every state you delete is not just less code. It is an entire class of incident that can no longer happen, because the configuration that produced it is no longer expressible. The version race that launched the wrong release at 3am did not get fixed. It got made impossible, because there is no longer a moment where "which version?" is resolved fresh at launch. The dual-source-of-truth stale deploy did not get a better guard. It got deleted, because there is only one source now.
Deleting state is the single highest-return thing I have done to this system in ten years, and it is almost never framed that way, because "we removed the ability to be in a bad state" does not demo as well as a new dashboard.
If you want to try the model without reassembling it from prose, I published a clean-room implementation as a Terraform module:
davidlu1001/instance-refresh-deploy/aws (source on GitHub). It ships the pointer, the launch template wiring, and a capacity-adaptive deploy driver — and if you already run an ASG, there is a zero-migration mode where the module only manages the pointer and your launch template adopts it with a one-line resolve:ssm change. The design decisions, including the ones I rejected, are written up in the repo's docs/DESIGN.md.Which brings me to the next piece. I have mentioned rollback about a dozen times in this article as if it were a happy side effect of the pointer model. It is not a side effect. In Part 6 I want to argue the opposite: that rollback is a feature you design for deliberately, across every generation of a deployment system, and that the quality of your rollback story is the single best predictor of how calm your incidents are. The pointer model made rollback cheap. Part 6 is about what you build once it is.