LLM rate limiting and quota management

Rate limits are the most common way an AI feature fails in production, and usually the least gracefully. Traffic spikes, the provider returns 429, naive retry logic amplifies the spike, and the outage lasts longer than the burst that caused it.

Know which limit you hit

• RPM — requests per minute. Many small calls trip this first.
• TPM — tokens per minute. Long-context calls trip this while RPM stays low.
• Concurrency — simultaneous in-flight requests.
• Daily or spend caps — the ones that fail at the worst time because nobody was watching the counter.

These are enforced independently. Halving your request count does nothing if you were TPM-bound.

Retry correctly

1. Respect Retry-After when the provider sends it. Ignoring it and backing off on your own schedule extends the problem.
2. Exponential backoff with full jitter. Without jitter, every client retries in lockstep and you rebuild the spike precisely.
3. Cap total retry budget per request so a queue cannot grow without bound.
4. Never retry non-retryable errors. A 400 will be a 400 again.

Spill over instead of backing off

With several providers serving equivalent models, a 429 becomes a routing event: mark the provider saturated, shed traffic to the next eligible candidate, and probe for recovery. Users never see the limit. See provider routing and multi-provider failover.

Then enforce your own quotas

Provider limits are not the only ones you need. Per-key and per-team budgets stop one batch job from consuming the capacity your production traffic depends on.