Infrastructure failures happen. Your architecture should act before an operator does.

Background

An AWS Region is a physical area — Northern Virginia, Frankfurt, Sydney — and inside each one sits a small number of Availability Zones: at least three, each its own cluster of data centers with independent power, cooling, and networking, set miles apart but stitched together by fast, redundant fiber. The separation is the point. A zone can lose power or fall off the network without dragging its neighbours down with it, which is why the standard advice is to spread a service across multiple Availability Zones and let the survivors carry the load if one fails.

These events are uncommon, but they happen — a power failure, a fiber cut, a network fault that isolates a single zone. Architecting so events pass without a human in the loop buys back real operator time.

Across every Availability Zone impairment we saw during our time at Amazon, our services failed away before neighbouring teams had alarmed on impact. The notification we received was not “you are down.” It was “you are operating without an AZ” — something to be aware of, not something to act on. No operator intervened. No customer was affected. The people on call kept doing what they were already doing.

Other teams in the same region, on the same event, spent the next several hours mitigating, and the next several days writing it up.

The difference was not heroics or better monitoring. It was one architectural decision made months earlier: we did not trust the load balancer to route around a broken Availability Zone.

The architecture everyone draws

A customer-facing endpoint on AWS almost always looks like this:

One load balancer, enabled across three Availability Zones
A fleet of hosts behind it, spread across the same three AZs
A regional datastore — in our case DynamoDB

The load balancer here stands in for whichever managed option you reach for — today that is a Network Load Balancer, an Application Load Balancer, or API Gateway. All three hand you cross-AZ load balancing out of the box.

Customers reach one Network Load Balancer spanning three Availability Zones, each holding app hosts, all sharing one regional DynamoDB. — The reference architecture. Every failover decision in this picture belongs to AWS.

It is a good architecture. It is the one in the reference diagrams, and for most failure modes it works exactly as advertised. A host dies, health checks catch it, traffic moves. A deployment goes bad in one zone, the same thing happens.

But there is an assumption buried in that picture, and it is rarely stated out loud:

When an Availability Zone has a problem, the load balancer will notice, and route around it.

That assumption holds for some AZ events. It does not hold for the ones that hurt.

What we changed

The change is small enough to describe in a sentence:

One Network Load Balancer per Availability Zone, each with its own DNS name, with Route 53 health checks deciding which zones receive traffic.

Three NLBs, each pinned to a single AZ, each fronting only that zone’s hosts. A Route 53 record set across all three, with a health check attached to each.

Same components, one topology change. Each zone is independently addressable, and the decision to remove one is now ours to define, observe and trigger.

Route 53’s health checkers are a distributed global fleet. That lets you run health checks at two levels, each answering a different question.

The outer check answers the question a customer actually cares about — can I reach this endpoint? — by probing the zone from outside the region, along the same public path a customer would travel. The inner check keeps running where it always did, inside the zone, answering a narrower one: are the hosts behind the load balancer up and running? A zone only takes traffic when both pass: a customer can reach it, and the hosts at the end of the path are healthy.

Once the decision moves to Route 53, the visibility and control become yours: you can see the health check state, alarm on it, and force a zone out manually if you need to. Nothing exotic is happening here — it is DNS-based failover, which has been in use for decades.

What we observed

Over every AZ impairment we saw:

Traffic shifted away from the affected zone before neighbouring teams’ impact alarms fired
Failover was automatic — no operator action, at any point
Operators had a fast, safe way to pull a zone out by hand — whether for a partial failure like packet loss, or to weight traffic away after a bad deployment
None of them produced a mitigation call, correction-of-error, or postmortem

That last category deserves more attention than it usually gets. Avoiding an outage avoids more than downtime. It avoids the mitigation call, the monitoring for hours afterward, the correction-of-error document, the review, and the days of senior engineering attention that never return to roadmap work. Teams that sat inside those events paid that tax. We did not.

What it actually costs

NLB pricing has two parts, and the split moves neither. The consumption-based capacity charge tracks total traffic, which is unchanged — three balancers each carry a third of what one carried. The hourly charge is billed per Availability Zone the balancer is enabled in, so one balancer across three zones and three balancers each in one zone come to the same three zone-hours. The bill is flat.

The only genuinely new line item is DNS: one Route 53 health check per zone, plus query volume. A handful of dollars a month, and bounded — it does not grow with the fleet behind it. For customer-facing zonal resilience, it is worth it every time.

The takeaway

The default architecture is not wrong. For a single topology change, you gain back the control and insight to decide for yourself when a zone is out, without giving up any of the resilience that running across multiple AZs already buys you.