Reliability5 min read
Multi-provider LLM failover
If your product calls one provider directly, that provider is a single point of failure. Model APIs have outages, rate limits, and latency spikes like any other service. Multi-provider failover routes around those problems so your users never notice.
The failure modes you need to cover
- Hard outages — the provider returns 5xx or is unreachable
- Rate limits — you're throttled at peak
- Latency spikes — the request eventually succeeds, but too slowly
- Capacity flapping — intermittent errors that a naive retry makes worse
Failover done right
- Health-aware candidate selection.Temporarily exclude a provider that's throwing a recent burst of errors instead of hammering it.
- Automatic retry to the next best option, not a fixed second-choice — the fallback should still respect your cost and quality goals.
- Hedging for latency. For latency-critical requests, race a deferred second attempt and take whichever responds first.
- Idempotency so retries never double-charge or double-execute.
Failover is one case of routing: the router already knows the ranked candidates, so recovering from a failure is just continuing down the list — with the same cost and quality constraints applied.