Most resilience conversations about LLM providers imagine the wrong incident. The imagined one is a total, unambiguous outage lasting a few minutes, during which requests fail cleanly with a connection error and everybody understands what is happening.
The real one is slower and much harder to handle. On 10 June 2025, from 06:36 to 22:00, OpenAI's status page recorded elevated error rates across ten API components — roughly fifteen and a half hours. Not down. Elevated. Some requests succeeded, some failed, latency wandered, and any system whose failure handling assumed a binary spent the day flapping between states.
This is a teardown of what has to exist for that day to be uneventful, walked through in the order the minutes actually arrive, and — because it is the part that gets skipped — what the machinery costs you on the days it works.
Minute 0 to 2: the failures are indistinguishable from load
The first symptom is not an error. It is a latency distribution that got wider. P50 barely moves, P95 doubles, and a few requests start returning 500s at a rate that looks like noise if your baseline error rate was never measured.
Almost every naive handler makes the same call here and retries. Which is correct — for a transient fault. It is exactly wrong for a degraded provider, because retrying a struggling upstream adds load to the thing that is struggling, and if every client does it simultaneously the provider's partial degradation becomes a total one. Retry storms are not a hypothetical; they are the standard second act.
The mechanism that distinguishes the two cases is a circuit breaker attached to each provider connection, tracking three signals rather than one:
- Consecutive failures past a threshold — catches hard, fast breakage.
- Error rate over a rolling window — catches the 10 June shape, where most requests still succeed and no consecutive-failure counter ever trips.
- Sustained latency past a bound — catches the case where nothing errors at all and every request takes eleven seconds, which for a conversational surface is an outage with extra steps.
The second signal is the one people leave out, and it is the one that mattered on 10 June.
Minute 2: the circuit opens, and the state machine has three states
Two states are not enough. Closed passes traffic. Open blocks it and routes elsewhere immediately, without paying the timeout first. The third state is the one that makes the system recoverable: half-open, where after a cooldown a single probe request is allowed through. If it succeeds, the circuit closes and normal traffic resumes; if it fails, the circuit re-opens and the cooldown restarts.
Without half-open you have built a switch, not a breaker, and someone has to flip it back manually — which on a fifteen-hour incident means someone watching a status page all day. With it, recovery is a property of the system. The tuning knob is the probe interval, and it is a genuine trade-off: probe too often and you are part of the retry storm, probe too rarely and you serve from your fallback for an hour after the provider recovered.
Minute 2, continued: failover needs somewhere to fail over to
Here is where most architectures discover they cannot do this at all, for a reason that has nothing to do with resilience engineering.
If the model is named inside the workflow definition — an OpenAI node, with gpt-4.1 in its configuration — then "use a different provider" is not a routing decision. It is an edit to every workflow, deployed under incident conditions, by whoever is awake. The failover you want is not expressible, and no amount of circuit breaking helps, because there is nowhere to route to.
Failover requires that the workflow declares a kind of thinking rather than a model, and that something resolves that declaration at call time. Nodes ask for a level — low, normal, high, plus specialised ones for voice and embeddings — and a per-tenant control plane maps each level to a concrete connection and model. That indirection is worth its own argument, and resilience is the clearest reason to pay for it: once the mapping is data rather than code, it can hold an ordered list instead of a single pair.
Level "normal" resolves to:
1. connection=openai-prod model=gpt-4.1 ← circuit OPEN
2. connection=anthropic-prod model=claude-sonnet ← circuit CLOSED, serving
3. connection=google-prod model=gemini-pro
Three fallbacks is a reasonable cap, and the cap is not arbitrary: each additional hop is latency the user waits through while the runtime discovers that the next option is also unhealthy. A chain of eight is a slow error.
Minute 3: the fallback is not as independent as the diagram says
This is the part that makes chains look better on paper than in production.
On 19–20 October 2025, AWS published a post-event summary for an event in the US-EAST-1 region that ran from 11:48 PM PDT on 19 October to 2:20 PM PDT on 20 October — about fourteen and a half hours. The root cause was a latent race condition in DynamoDB's DNS management system that produced an incorrect empty DNS record for the regional endpoint, which the automation could not repair on its own. The blast radius reached EC2, Lambda, ECS, STS, IAM console authentication and more.
A fallback chain listing three different model vendors is three vendors. Whether it is three failure domains depends on where each one actually runs, and on whether your own runtime, your secret store, your session database and your queue are in the same region as any of them. Depth is not independence. The question to answer before an incident, on paper, is which single event takes out two rows of your chain at once — and the honest answer is often that one does.
Minute 3, still: rate limits are a different failure wearing the same clothes
A 429 looks like an outage to code that only checks whether the call succeeded, and it needs the opposite response. An outage means route away. A rate limit means wait — the provider is telling you, in a Retry-After header, precisely how long.
Failing over on a 429 is a mistake with a delayed bill: you push traffic onto a fallback that may be priced differently, and because it works, nobody notices until the invoice. The distinction is worth encoding, and it is worth knowing which of your limits you are hitting. Limits stack at four levels, and they fail differently:
| Level | What it protects | What hitting it means |
|---|---|---|
| Provider connection | The upstream's own quota | Back off and retry; failover only if it persists |
| Per API key | One integration misbehaving | Something on your side is looping — do not route around it |
| Per tenant | One client's fair share | Working as designed; the client needs a higher ceiling, not a fallback |
| Global | The platform's total capacity | You are the outage now |
Routing around your own limits defeats the point of having them.
Hour 4: what the layer actually buys
Three things, and they are worth naming separately because teams often build the machinery for the first and never collect the other two.
The user does not see the incident. Traffic shifts inside a single request. The conversation continues on a different model and nobody is told, which is the entire objective.
The incident becomes reportable. Circuit state per connection, time in each state, failover counts, latency percentiles and error rate per window are all things you now measure because the breaker needed them. That is the difference between telling a client "there was an issue with our AI provider" and telling them "between 06:40 and 22:00 we served 4% of your conversations from a secondary model, with no failed requests."
The dependency becomes negotiable. A platform that can move providers in a config change during an incident can also move them during a contract negotiation. That is not a resilience property, but it is bought with the same code.
When not to build this
A single provider with retry-and-backoff is the right answer more often than resilience writing admits, and there are three distinct cases.
When your volume is low and your tolerance is real. If you serve a few hundred conversations a day to internal users, an outage is an apology, not a business event. Retry with exponential backoff and jitter, surface an honest error, and spend the engineering time on something a customer can see. Multi-provider failover has a running cost — a second vendor relationship, a second key to rotate, a second set of quotas, a second model's behaviour to keep track of.
When you cannot switch anyway. If a client's contract or a regulator pins processing to one provider or one jurisdiction, failover across vendors is not available to you. The resilience story there is graceful degradation: shed the expensive levels first, queue what can be answered late, and tell the user honestly, which is a design problem rather than a routing one.
When the fallback would be worse than the outage. This is the one that gets underestimated. Failing over changes the model, and the model is not a commodity part.
That last point deserves its own paragraph, because it is the real cost of the mechanism working. Prompts tuned on one model are not equally good on another. Structured-output schemas that a primary honours reliably may come back malformed from a fallback, and the failure will be a parse error in your own code rather than an error from the provider. Tone shifts, so a support conversation can change voice mid-thread. Your evaluations were run against the primary; during failover you are serving a configuration you have never measured. And the fallback's price per token is probably not the primary's, so unit economics move silently in the direction nobody is watching — which is a large enough problem to be its own arithmetic.
The mitigation is unglamorous: run your evaluation suite against every model in every chain, not just the primaries, and treat a fallback you have not evaluated as untested code that only executes during incidents. Which is the worst possible time to find out.
Questions to answer before the next incident
Portable, vendor-neutral, and answerable in an afternoon with a whiteboard.
- What is your baseline error rate and P95? If you cannot state both, you cannot detect degradation — only total failure.
- Does your breaker trip on error rate over a window, or only on consecutive failures? The 10 June shape defeats the second.
- Is there a half-open state, and what is the probe interval?
- Can you change which model serves a level without editing a workflow? If not, everything below this line is unavailable to you.
- Draw the dependency graph of your fallback chain down to region and cloud. Which single event removes two rows?
- Does a 429 route away or back off? Check the code, not the intent.
- Have you run your evaluations against the fallback models? If not, name the date you will.
- After a failover, who finds out, and how? A transparent failover that nobody can see afterwards is an undetected change in what you are shipping.
Fifteen and a half hours on 10 June. Fourteen and a half on 19 October. These are not rare events at the tail of a distribution — they are the observable behaviour of the infrastructure everyone is building on, published on the operators' own status pages. The design question is not whether the provider will have a bad day. It is what your system is doing at hour nine of it.
