Three Availability Zones Are Not Three Independent Failures

“We run across three availability zones, so a regional problem cannot take us down.”

Three zones.

Good.

Now tell me what the three zones have in common.

Three availability zones are not three independent failures.

AWS describes a Region as a minimum of three physically separate Availability Zones, each with independent power, cooling and networking.

That is a real engineering commitment.

It means a fire, a power loss or a network cut in one facility leaves your copies in the other two running.

It is redundancy.

More copies of the same thing, placed so that one copy’s loss does not remove the capability.

Redundancy defends against independent failure.

It does nothing against a cause that reaches every copy at once.

And every zone in a region still calls through the same regional layer: storage metadata, internal networking, the control plane that launches, scales and reconfigures what you run.

Nobody would say “we have three copies of the database on three disks, so a bad schema migration cannot hurt us.”

Say the same thing about zones, draw it with three coloured boxes, and suddenly it is a resilience claim.

You have counted facilities.

You have not counted causes.

1. “But Each Zone Has Independent Power, Cooling and Networking.”

Yes.

That is exactly what each zone is independent in.

Power.

Cooling.

Networking.

The physical layer of one facility.

It is not a statement that every service your workload touches is replicated independently per zone, with nothing regional underneath.

The resilience vocabulary has a word for the thing that is left: a common-mode dependency.

A shared element beneath a set of redundant components that defeats the independence redundancy assumes.

David Woods draws the line that makes this a coherent question rather than a paradox.

Robustness is the ability to deal with challenges anticipated in advance.

Resilience is the ability to adapt when a challenge arrives that nobody anticipated.

Multi-zone deployment is robustness against an anticipated challenge: losing a facility.

It is well designed for that challenge.

It was never designed for the other one.

So the useful diagnostic is not “how many zones?”

It is this.

If the thing that broke were a dependency none of your copies were diverse against, would this architecture survive?

For US-EAST-1 customers, the answer has already been measured.

Twice.

2. “But Our Instances Kept Running Through the Last Outage.”

Probably.

Let us look at which outage.

28 February 2017.

An authorized S3 engineer ran a routine command to remove a small number of servers from S3’s billing subsystem.

The command was entered incorrectly.

A larger set of servers was removed than intended.

Among them were servers supporting two other subsystems: index and placement.

Those two hold the metadata and location information for every object stored in the region.

They are needed to service every GET, LIST, PUT and DELETE request in US-EAST-1.

S3 request handling went down region-wide, from 9:37 AM PST to full recovery at 1:54 PM PST.

About four hours and seventeen minutes.

Then it spread to services that had no reason to think of themselves as S3 customers.

EC2 instance launches.

EBS snapshot retrieval.

AWS Lambda.

Each depended, at some remove, on the one region-wide subsystem that had just gone away.

Your instances in three zones may well have kept running.

If they kept state in S3, they were running against a data layer that could not answer a request.

Redundant against instance failure. Zero redundancy against the subsystem every instance’s data depended on.

7 December 2021.

An automated scaling activity in AWS’s internal network triggered unexpected behaviour in internal network clients.

The surge of connections overwhelmed the networking devices linking that internal network to the main AWS network in US-EAST-1.

EC2, Route 53, RDS, EMR, Workspaces, Elastic Load Balancer and API Gateway APIs were disrupted.

So were EventBridge event delivery and the AWS Console itself.

The window ran from roughly 7:30 AM to 4:37 PM PST, depending on the service.

That is a little over nine hours at its widest, if you do the subtraction.

And here is the sentence that answers your objection.

Already-running EC2 instances stayed up, and existing load balancers remained healthy.

Your instances kept running.

What did not keep running was the regional control plane: the layer every zone-redundant deployment calls through to launch, scale or reconfigure anything.

It was impaired across the region.

However many zones you spanned.

So if your plan for a bad day involved autoscaling, replacing an instance, or changing a route, your plan depended on the precise thing that was down.

“Our instances stayed up” is a true statement about the data plane.

It is not a statement that you could have done anything about what happened next.

3. “But We Cannot See or Change the Provider’s Control Plane Anyway.”

Correct.

You cannot fix it.

You can still decide how much of your service depends on it being available.

Those are different questions, and only the second one is yours.

Look at what the December 2021 account separates.

Serving traffic that is already configured: survived.

Changing the configuration: impaired.

So split your own requirements the same way.

What must keep serving during a regional incident?

What must remain changeable during one?

If the first depends on the second, you have wired your availability to the layer you just admitted you cannot see.

Examples, stated plainly.

A service that holds its peak capacity already running does not need a launch call during the incident.

A service that scales out on demand does.

A failover that is already provisioned and switched by traffic already flowing does not need a reconfiguration.

A failover that begins by creating resources does.

You do not need visibility into AWS’s internal network to write that down.

You need an honest list of which of your operations are launches, scales and reconfigurations.

The visibility problem is real, and it is not unique to you.

The Cloud Security Alliance’s 2024 Top Threats survey, of more than 500 practitioners, lists “Limited cloud visibility and observability” among its eleven top concerns.

That is an argument for writing the dependency down.

It is not an argument for pretending it isn’t there.

4. “But a Second Region Doubles the Cost.”

It costs money.

That is not a rebuttal.

It is the price of diversity.

Where redundancy answers “what if this instance fails,” diversity answers “what if the reason it failed is present in every instance I have.”

Diversity is materially more expensive than redundancy.

It means more than one design to maintain, more than one set of expertise, more than one thing to test.

Which is why most systems apply it selectively, not everywhere.

That is fine.

What is not fine is buying the cheap thing and describing it as the expensive thing.

A fourth zone would not have helped in February 2017.

It shares the same index and placement subsystems.

A fourth zone would not have helped in December 2021.

It shares the same regional control plane.

More identical copies do not buy independence from a cause every copy shares.

So the honest options are three.

Pay for a second region where the consequence of losing the first justifies it.

Design the critical path so it does not need the shared layer during an outage.

Or accept the dependency, in writing, with a named owner and a stated consequence.

What the evidence shows is that the second region only counts if it has been exercised.

20 September 2015.

A DynamoDB outage in US-EAST-1 cascaded into more than 20 dependent AWS services.

Many sites on the internet were intermittently unavailable for six to eight hours.

Netflix saw what it called “a brief availability blip.”

Not zero impact.

A brief, contained disruption.

Because Netflix had been running regular Chaos Kong exercises, which simulate losing an entire AWS Region, specifically to rehearse evacuating traffic out of an impaired one.

The failover had already worked before it was needed.

That is what the second region was for.

Not the diagram.

The rehearsal.

5. “But Going Multi-Cloud Would Solve It.”

It would move it.

A second provider removes the regional layer of the first provider from your shared dependencies.

It also hands you a second, structurally different platform to configure correctly.

Start with identity.

AWS IAM roles issue temporary STS credentials with no persistent long-term key, assumed by an EC2 instance profile, a Lambda execution context, or another AWS service acting as a service principal.

Azure managed identities come in two kinds.

System-assigned: bound one-to-one to a single resource and deleted automatically with it.

User-assigned: a standalone identity assignable to multiple resources.

GCP Workload Identity Federation is the outlier.

It replaces static service account keys with short-lived tokens from an OAuth 2.0 exchange against Google’s Security Token Service, and it federates from external identity providers: AWS, Azure, Active Directory, GitHub, GitLab, Kubernetes, generic OIDC and SAML.

Three mechanisms.

Not one mechanism with three names.

Your workload in two providers does not have one identity model replicated.

It has two, with different lifecycles and different failure modes, and your team has to get both right.

Defaults diverge too.

The same risk, a storage container reachable by anyone, takes different amounts of administrator action to close.

AWS has closed it by default for new S3 buckets since April 2023.

Azure keeps new containers private, but the account-level AllowBlobPublicAccess setting must be explicitly set to False before public containers are foreclosed across the account.

GCP’s Public Access Prevention must be enforced deliberately, per bucket or through an organization policy.

Same intent.

Three configurations.

NIST saw the shape of this in 2011.

SP 800-145 defines hybrid cloud as “two or more distinct cloud infrastructures… that remain unique entities.”

Unique entities.

That is the point, and the cost.

Diversity beyond the provider is paid for in configuration divergence.

Sometimes that is worth it.

It is never free, and it is never a copy.

6. “But Deliberately Failing a Region in Production Is Reckless.”

Pulling a power cord with no monitoring, no rollback and no idea who is affected is reckless.

It is also not chaos engineering.

It is an outage with extra steps.

The published method has safeguards in its definition.

Form a hypothesis about steady-state behaviour before the experiment.

Minimize the blast radius.

Automate experiments so they run continuously, under monitoring, rather than as one-off events.

Netflix built up to regional exercises.

The Simian Army, made public in 2011, began with Chaos Monkey terminating production instances during business hours.

Chaos Gorilla simulates the loss of a whole Availability Zone.

Chaos Kong simulates the loss of a whole Region.

Notice what that ladder means for your three zones.

Instance loss, zone loss and region loss are three different experiments.

Passing the first two says nothing about the third.

The practice is not rare, either.

Gremlin’s 2021 State of Chaos Engineering report, from a survey of more than 400 practitioners plus platform data, found 60% of organizations had run at least one chaos experiment, and 34% ran them against production.

Gremlin’s platform recorded 459,548 individual attacks in the report period.

That volume would not be sustainable if each experiment carried a real risk of an uncontained outage.

The same report found the teams experimenting most often reported the highest availability: 99.99% and above for top performers, above 99.9% for regular experimenters.

That is a correlation, not a controlled result.

Teams disciplined enough to run experiments are probably disciplined about other things too.

Do not sell chaos engineering to your board on that number.

Sell it on what Netflix saw in September 2015.

A real incident, a rehearsed evacuation, a brief blip.

The reckless option is the one where the first regional evacuation you ever attempt is during the regional outage.

7. No, This Does Not Mean Multi-Zone Is Wasted.

It isn’t.

Zone redundancy handles the failures it was built for, and facilities do fail.

Keep it.

It also does not mean every workload needs a second region or a second provider.

Diversity is expensive enough to apply selectively, and most systems should.

Nor were these outages failures of redundancy.

The redundancy that existed worked exactly as designed.

Instances in separate zones did not fail because of each other.

The failures were in a layer the redundancy never covered.

What this does mean is narrower, and harder to dodge.

A resilience claim has to name the shared layer that remains.

Regional storage metadata.

The regional control plane.

Identity.

DNS.

And it has to say whether the business can accept losing it.

“We are multi-AZ” is an honest answer to a question about facilities.

It is not an answer to a question about US-EAST-1.

8. What You Should Actually Do

List the shared layers for each critical service. For every redundant set, write down the provider layers every copy still calls through: the regional control plane, storage metadata, identity, DNS. Mark each one accepted or not accepted, with an owner.

Use the 2017 and 2021 US-EAST-1 events as a test of the list. If either would have reached the service, the list should say so.

Separate what must keep serving from what must remain changeable. Identify which operations in your incident path are launches, scales and reconfigurations. Keep the serving path from depending on them. Pre-provision what the critical path needs instead of creating it during the event.

Rehearse regional evacuation. Zone-loss testing does not cover region loss. Run scoped, monitored experiments with a steady-state hypothesis and a limited blast radius, and work up to region-level exercises the way Chaos Kong did.

Budget for diversity where you buy it. If a second region or provider is the answer for a service, budget for its identity model, its storage defaults and the expertise to keep both correct. A second provider configured by habit from the first is a new exposure, not a failover.

Date the evidence. Outage timelines, survey figures and provider defaults change. The shared-dependency question does not. Re-check the defaults and identity mechanisms you rely on before each design review.

For the wider split between recovery and resilience, see Disaster Recovery Is Not Resilience.

9. The Model Is Not Complicated.

Redundancy asks: if one copy fails on its own, do the others carry on?

Diversity asks: if the reason it failed is present in every copy, what is different about the next one?

Three zones answer the first question well.

They do not answer the second at all, because every zone in the region shares the layer beneath it.

Redundant copies fail together when they share a layer beneath them. In cloud platforms, that layer is regional and provider-owned.

The S3 subsystems in 2017, the control plane in 2021, and DynamoDB in 2015 are dated evidence.

The design question outlives them.

What do your three zones have in common?

If the answer is “nothing, they are independent,” go and read those three outage summaries again.

Three availability zones are not three independent failures.