Capacity-adaptive instance refresh: measured numbers from a real fleet

Part 7 of a series on ten years of deployment evolution for one legacy app on AWS — no Kubernetes required. Series index.

Most of what you can find online about ASG instance refresh is the API reference and a handful of "here's how to start one" tutorials. What I could not find, when I was building this, was numbers. How long does a refresh actually take on a four-instance fleet? What does the 50% checkpoint pause measure out to in wall-clock seconds? Does launch-before-terminate really exceed your ASG max size, or is that a documentation footnote nobody hits? How fast is an alarm-triggered auto-rollback, in seconds, from the moment the alarm fires?

I have those numbers now, because I ran the drills. This is the data piece. Every figure below came off a real fleet running a ~10-year-old PHP/Node monolith — three environments, a DR region, a small platform team — during the July 2026 rebuild I've been describing across this series. If you are about to replace blue/green with instance refresh and you want to know what you're signing up for, this is the article I wish I'd had.

The one thing I'll ask you to carry in from earlier parts: by this point each release is baked into a single AMI, pinned to a commit SHA, promoted byte-identical across environments, with an SSM Parameter Store pointer holding "the SHA that is the current release." Part 5 covers the pointer, Part 3 covers the baked-AMI layering. Deploying is now "tell the ASG to roll onto the new launch-template version." Everything here is about how that roll is shaped.

The problem: one set of refresh parameters can't fit every fleet

Instance refresh is configured by a preferences block. The two knobs that matter most are MinHealthyPercentage (how much of the fleet must stay in service) and MaxHealthyPercentage (how far above 100% the ASG may temporarily surge, which is what enables launch-before-terminate). You can also attach CheckpointPercentages to pause the refresh partway and let alarms watch the new release before it finishes.

The trap is that these are usually written as static values. You pick numbers once, put them in Terraform or a script, and every deploy in every environment uses them. That works right up until your fleets are different sizes — and mine are. The test pilot runs at a desired capacity of 1. Stage sits at 1–2. Production runs 2–6 today and will grow. The socket tier is always exactly 1 instance, everywhere. No single set of preferences is correct across that range, and the failure isn't cosmetic.

Take the design sketch I started from, which hardcoded MinHealthyPercentage=100, MaxHealthyPercentage=200 — the shape that was validated on the one-instance pilot. Now apply it to a production fleet of six.

The min-healthy math. With MaxHealthyPercentage=200, the refresh is allowed to double the fleet. At desired=6 that means it can launch all six replacements at once, alongside the six old instances, wait for the new ones to go healthy, then terminate the old six. The whole fleet flips to the new release in a single batch. There is no point at which half the fleet is old and half is new, which means there is no window in which an alarm can catch a bad release before it's serving 100% of traffic. That is precisely the blue/green failure mode I spent this whole project escaping: a bad release reaches everyone, and only then does anything notice. The pilot shape is safe at one instance because one instance is the whole fleet either way. At six it quietly deletes your canary.

Now run it backwards. Take the parameters you'd want for a six-instance canary — MaxHealthyPercentage=150 with a checkpoint at 50% — and apply them to the single-instance socket ASG. Fifty percent of one instance is zero. The checkpoint has nothing to pause between. Worse, at Max150 the ASG is allowed to surge by only half, and half of one rounds to nothing, so there is no headroom to launch a replacement before terminating the original: you either dip to zero capacity or the refresh can't proceed. A single-instance ASG must surge to 200% to replace its instance without a gap. The canary-shaped parameters are actively wrong here.

And refresh duration scales with the wrong thing. Warmup and checkpoint delays that feel fine on a one-instance pilot become the dominant cost on a large fleet, because they're paid per batch. A refresh of six instances in three batches of two, with a 180-second checkpoint delay and a 120-second instance warmup, is a very different wall-clock event from a one-shot pilot refresh. If your delays are tuned for the pilot, prod under-observes each batch; if they're tuned for prod, the pilot wastes minutes waiting on alarm windows it doesn't need.

So the parameters have to come from somewhere other than a constant. The obvious "somewhere" is the fleet itself.

The strategy: derive the shape from live capacity, per tier

The refresh driver computes its preferences per tier, at refresh start, from that ASG's current desired capacity. Same script, no per-environment edits, no flags to remember. The rules are small enough to state in full:

desired <= 1 → MaxHealthyPercentage=200, single batch. Launch the full replacement set at once, wait healthy, terminate the old. This is the validated pilot shape, and it's the only shape that works for a one-instance ASG. The socket tier always takes this path, because the socket tier is always exactly one instance.

desired >= 2 → MaxHealthyPercentage=150, CheckpointPercentages=[50,100]. Replace half the fleet per batch. Pause at 50% while the gate alarms watch real traffic on the new release, then finish. If an alarm breaches during the pause, the refresh rolls back with half the old fleet still standing, instead of after a 100% flip.

That [50,100] is worth pausing on, because the sketch I inherited wrote it as [50] and that is a latent bug: the last value in CheckpointPercentages must be 100, or the refresh does exactly what you told it and stops at half, permanently, waiting for a continuation that never comes. It's the kind of thing you only find by running it. I found it by running it.

The tiers get different treatment beyond capacity, too:

The command tier carries no target group and takes no traffic, so it's always a single-batch launch-template bump — there's nothing to canary.

The socket tier is single-instance by design and always takes the Max200 path above. During its replacement there's a brief dual-socket window, which is the same window blue/green had at cutover; websocket clients reconnect.

The web tier is the one that actually gets the capacity-adaptive batching, because it's the one that scales.

Two AWS behaviors make this safe to run without babysitting. First, SkipMatching=true: if the ASG's dynamic scaling pulls up a new instance mid-refresh, that instance already launches on the target launch-template version, so the refresh doesn't pointlessly replace it a second time. Second, and this is the one I most wanted a real measurement of: launch-before-terminate can temporarily push the running count above the ASG's MaxSize. This is documented AWS behavior, but "documented" and "I watched it happen" are different confidence levels. I watched it happen (numbers below). The practical consequence is that MaxSize needs no headroom for refresh — but your account's On-Demand vCPU quota and your private-subnet free IPs must cover roughly 2× your largest fleet, because for a few minutes you really do run 2× the instances.

The policy knobs that aren't derived from capacity — how long to warm an instance, how long to hold the checkpoint, how long to keep alarms armed after the last batch — live in a per-environment profile:

profile	InstanceWarmup	CheckpointDelay	Soak, alarms armed
test	90s	60s	0
stage	120s	120s	60s
prod	120s	180s	120s

That last column is the soak after the final batch, kept short because the gate alarms are armed for the entire refresh, not just the tail. The checkpoint delay has a hard floor: the 5xx gate alarms need two 60-second datapoints to evaluate, so anything under 120s can't see a full alarm window. Prod keeps one extra period of margin on top. A --prefs <env> flag lets the test environment borrow another environment's profile, which is how I rehearse prod behavior without a prod fleet — more on that in a second.

The measured latency model

Here is the part that barely exists online. All of these are measured, not estimated, on the test environment after full cutover, unless noted.

Start with the number that actually matters — tag to serving traffic — and its two siblings, because a fresh tag drives three pipelines and it's easy to quote the wrong clock:

Clock	Measured	What it is
Tag → serving traffic	6m59s	new web instance target-group-healthy; the deploy-latency number
Tag → tag pipeline green	10m5s	• the refresh tail: drain/terminate, pointer write, cleanup
Tag → fully converged on baked AMI	~16 min	• background bake (~10 min) + converge LT flip (~1.5 min, no user-visible effect)

Only the first is on the serving path. The bake and the converge callback are background convergence; the tag pipeline itself stays green-pending about four minutes after traffic is already on the new release, on purpose, because it carries the drain/terminate tail so failure handling and the pointer write happen inside the pipeline. The "serving" moment is the one I put in a Slack message and the one you should benchmark against.

Where does the 6m59s go? I broke it down:

runner pickup: ~1 min

build: 2m45s (the gulp asset build dominates at ~2 min; dependency installs are warm-cached and run in parallel)

bake trigger: 5s (fire-and-forget; the bake runs in the background)

driver pre-flight: ~50s

refresh to healthy: ~2.5 min (AWS refresh scheduling + 94s boot + two 10-second target-group health checks)

The single biggest lever is not in the deploy machinery at all — it's the two-minute front-end asset build. The refresh itself, the part everyone worries about, is ~2.5 minutes and most of that is AWS's own state machine.

The per-operation numbers, which are the ones you'd actually reuse for capacity planning:

Operation	Measured
Instance launch → serving, baked AMI	~90s
Instance launch → serving, eager (base AMI + boot-time payload download)	~94s
Legacy boot (git resolve + S3 zip + full Ansible)	~5 min
Refresh, no migrations, tiers in parallel, 2 instances	4m56s
Refresh, with migrations, socket-first serial	~9 min
Bake	~8–10 min (±2 min EBS snapshot variance)
Boot-time activation play (Ansible portion)	28.4s
Bake-time payload play (Ansible portion)	39.7s
Pre-baked fast path, tag → serving	6m10s
Unattended replacement of a dead instance	~2.5 min
Alarm auto-rollback: ALARM → rollback starts	~17s
Alarm auto-rollback: fully restored	~2.5 min

The boot number is worth staring at. I SSH'd into a booting instance and read the cloud-init timing profile: 70.13 seconds end to end, split as ~10s from EC2 to cloud-init, ~35s of userdata (asset download, unzip, user sync), and 28.4s for the activation play itself, of which the two slowest tasks were Swagger generation at 5.96s and fact-gathering at 4.8s, and everything else was under two seconds. Target-group-healthy lands at roughly launch + 90s. That measurement killed four separate optimization ideas I'd been carrying (moving Swagger out of boot, a systemd-oneshot activation, fast snapshot restore, asset slimming) because when the whole play is 28 seconds, there's nothing left to shave. The old boot was ~5 minutes of git and Ansible; the new one is 90 seconds, and the 90 seconds is already mostly AWS scheduling. This is why I stopped optimizing boot and started optimizing the asset build.

Now the capacity-adaptive drill, which is the whole reason this article exists. I scaled the test web tier to four instances and ran a real batched refresh with prod-shaped preferences:

4-instance fleet, original 300s/300s checkpoint-and-bake delays: 18m26s, green.

Max150 produced 2-instance batches (50% of 4) — the derivation working as designed.

The 50% checkpoint pause measured 300s between the two batches' launch timestamps — exactly the configured CheckpointDelay, confirmed against wall-clock, not inferred.

During batch 1, the group ran 6 instances against a max_size of 4. The MaxSize breach, confirmed in practice. No headroom needed; the launches all succeeded.

Baking was correctly treated as in-progress by the driver's wait loop.

After that drill I re-tuned prod from 300s/300s down to 180s/120s. The budget lands at ~13–14 min per prod-prefs tier, with a mixed-version window of ~7–8 min — and the first new instance serves at ~+2 min regardless, because that's just boot time. The mixed window is the honest cost of a canary: for those 7–8 minutes, two releases coexist on the fleet. I make that invisible per-user with a one-hour lb_cookie session stickiness on the web target group, so no individual user ever alternates between releases; what remains is fleet-level coexistence, user A on the new release and user B on the old, which is the literal definition of a canary and exists in every rolling or weighted deploy ever built. Coexistence time equals alarm observation time. It's the same knob. If I ever need it shorter, dropping the checkpoint to 120s, warmup to 90s, and deregistration delay to 30s gets it to ~5–5.5 min while still keeping one full alarm window — I've measured that path and deliberately left it off by default, because it trades away alarm margin for a window users already can't perceive.

For contrast, the incident rollback path — single batch, tiers parallel, alarms bypassed because during an incident they're firing because of the release you're escaping — measured first known-good instance serving at +2m14s and full fleet converged at 5m30s on the four-instance fleet, against the ~13–14 min a standard prod deploy takes. Part 6 is the whole rollback story; I mention it here only because it's the same fleet and the same clocks, measured the same day.

The pre-bake fast path: taking the bake off the critical path

The bake is ~10 minutes. For a long time it sat on the deploy's critical path, which is why the very first deploy of any commit was slow. But the bake doesn't have to be there. The AMI is env-agnostic and keyed by SHA, so a commit whose by-sha zip and AMI already exist can deploy refresh-only, straight from the tag runner — skip the build, skip the bake:

Pre-baked commit, tag → serving: 6m10s, measured (refresh Successful at 8m39s).

A commit gets pre-baked three ways: it was deployed to some environment before (re-tags and cross-env promotion are free), someone ran the explicit pre-bake pipeline on it, or — if you adopt it — a branch trigger pre-bakes every push to your main branch. The last one sounds appealing until you look at your own git history. I did, across 393 deploy tags: 95% of tags land on a commit created less than two minutes earlier, which means a push-triggered pre-bake would have a near-zero head start on the tag that follows it. So I didn't wire it up by default; the win doesn't materialize for how this team actually tags.

And here's the honest part about the fast path: after I fixed the fresh path (see the TIL below), the non-pre-baked deploy dropped to 6m59s. The pre-baked path is 6m10s. The fast path's entire remaining advantage is about 50 seconds plus the skipped bake compute. Worth having for promotions and re-tags, where it's free anyway. Not worth restructuring your CI around. The lesson I took: the eager deploy path — boot from a base AMI, download the payload at boot, adopt the baked AMI later in the background — collapsed the gap so far that the "fast path" stopped being a meaningfully different thing.

Runtime caches are part of the artifact

There's a lesson baked into all of this that I paid for once, during the May 2026 incident that kicked off this whole rebuild (Part 2). During that incident an emergency AMI was snapshotted from a running instance, and it carried a live PHP opcache. New code, deployed correctly over the top by Ansible and the S3 zip, was ignored at runtime — PHP-FPM served bytecode compiled from the previous instance's state. The deploy succeeded by every check it had, and the running code was stale. The immediate fix was an explicit php-fpm restart plus opcache flush in the deploy sequence; the long-term option is opcache.validate_timestamps=1 with revalidate_freq=2, which auto-invalidates at a small CPU cost that had been traded away for performance.

That was a mutable-infrastructure bug, and immutable AMIs mostly dissolve it: every instance now boots fresh, php-fpm starts cold at the end of the activation play, and a cold process compiles the code that's actually on disk. But the principle survives the fix, and it's the one thing I'd underline for anyone baking artifacts: a baked artifact's contract includes its runtime caches. Opcache, realpath cache, a warmed framework cache, a preloaded class map — if any of that is captured in the image, it is part of what you shipped, and it must correspond to the code you think you shipped. Bake it deliberately and warm it against the baked code, or don't bake it and build it fresh at activation. What you cannot do is let a cache from one release ride into the image of another and assume the filesystem wins. It doesn't. The bytecode wins. I have the 502s to prove it.

Two small sharp edges (TIL)

Two things I learned the way you actually learn things, which is by breaking them.

Launch-template versions come back newest-first, and I assumed the opposite. When a refresh fails or is cancelled, the driver's cleanup re-pins each tier back to the launch-template version matching the pre-deploy SHA. The code selected that version with a JMESPath | [-1], taking the last element, on the assumption that the API returns versions oldest-first. It returns them newest-first. For a long time this was latent, because there was only ever one launch-template version per SHA and last-of-one equals first-of-one. Then I added re-baking, which can produce multiple versions for the same SHA, and the bug woke up: a cancel after a re-bake re-pinned the tier to the oldest matching version (the superseded, in this case contaminated, bake) instead of the newest. A mid-refresh cancel drill caught it live. The fix is one line: max_by on the version number instead of [-1]. The lesson is older than the fix. AWS list APIs have an ordering, that ordering is part of the contract, and "it worked in the test that only ever had one item" is not evidence about the contract. Read the ordering; don't infer it from a fleet of size one.

"Rebuild the same version" needs to be a first-class operation. I hit this when a batch of release AMIs turned out to be subtly contaminated: a role default made a payload-only bake run some service tasks it shouldn't have, so those AMIs shipped a service and a process dump that had no business being in an env-agnostic image. The right response is to re-bake the same SHA: same code, new, clean image. That sounds trivial and it exposed a design gap. My whole pointer-and-converge model was built around "one SHA, one AMI," and a duplicate bake of a SHA that was already running was a same-SHA no-op forever: the convergence step looked at the running SHA, saw it already matched, and did nothing, so the fresh clean AMI was never adopted. I had to teach convergence to notice that a release <sha> pin whose image is no longer the newest available AMI for that SHA should be flipped onto the newer bake (running instances keep serving; the next launch boots the clean image). Once that existed, re-baking a defective AMI became routine: trigger the bake with the SHA pinned, ~10 minutes later the callback flips the launch templates, done. If you're building anything that treats "the artifact for commit X" as a singleton, ask early what happens when you need to rebuild X. You will need to. A bad bake, a compromised base image, a re-bake to pick up a security patch in a lower layer: "produce a fresh artifact for an unchanged version" is a real operation, and if your identity model can't express it, you'll find out at the worst time.

Evidence, not projection

If there's a throughline to this series, it's this last point. Every decision in this rebuild was made against a measured number. The boot optimizations I didn't do, I dropped because I SSH'd in and read the 28-second play timing and there was nothing there to win. The prod checkpoint delays I settled on, I set after watching a 300-second pause measure out to 300 seconds and deciding I could afford 180. The MaxSize headroom I didn't provision, I skipped because I watched six instances run against a max of four and every launch succeed. The launch-template ordering bug, the same-SHA re-bake gap, the [50]-that-should-be-[50,100] checkpoint — none of those were caught by reasoning about the design. They were caught by running it and looking at what came out. A design review tells you what should happen. A drill tells you what does. When those two disagree, the drill is right, and the entire value of this work is that I let the drills correct the design instead of the other way around.

That's also why I wrote this part with the numbers in it. The instance-refresh docs will tell you the API exists. They won't tell you a four-instance prod-shaped refresh takes 18 minutes at 300/300 and 13–14 at 180/120, or that the first new instance always serves at +2 minutes because that's just boot, or that launch-before-terminate really will blow past your max size and you should size your vCPU quota for it. Now something online does.

This is the last part. Across the arc I've traced one workforce-management monolith through four deployment generations — inherited EC2-and-scripts, a blue/green rebuild I was proud of for years, an incident-driven split-AMI redesign, and this final collapse to fully-baked AMIs and single-ASG instance refresh, all of it without a Kubernetes cluster in sight. If you're arriving here first, Part 1 is the retrospective that frames the whole thing, and it reads better forward. The consistent lesson, generation over generation, was that the boring native primitive plus a measured understanding of your own system beats the impressive platform you don't have the team to run. You don't need to be Google to deploy safely. You need to know your own numbers. I hope some of mine are useful to you.