LLM gateway vs API gateway
A reasonable objection to buying an AI gateway: we already run Kong / NGINX / an API gateway — why do we need another one? It is a fair question, and the answer is specific rather than hand-wavy. Traditional API gateways make four assumptions that LLM traffic violates.
1. Requests are cheap and uniform
API gateways rate-limit on request count because HTTP requests cost roughly the same. LLM requests do not — one call can cost a thousand times another. Counting requests controls nothing that matters. You need token-based and cost-basedlimits, which requires parsing the request and the response.
2. Responses are short and complete
Traditional gateways buffer responses. LLM responses stream for tens of seconds, and buffering destroys time-to-first-token — the single metric users perceive as speed. AI traffic needs true streaming passthrough with per-chunk handling.
3. Routing is by path, not content
API gateways route on URL, header, and method. Useful AI routing depends on the content of the request — how hard is this prompt, how long is the context, does it need tools, is a cheaper model sufficient. That is a different class of decision.
4. Caching is exact-match
HTTP caching keys on the URL. Two prompts that differ by one word are different cache keys and identical in meaning. Semantic caching requires embedding the request and matching on similarity — not something an HTTP cache can do.
| Concern | API gateway | LLM gateway |
|---|---|---|
| Rate limiting | Requests per second | Tokens and cost per minute |
| Response handling | Buffered | Streamed, chunk-aware |
| Routing basis | Path and headers | Request content and model fit |
| Caching | Exact-match on URL | Semantic similarity |
| Failover | Health-check based | Quality- and capacity-aware |
| Cost visibility | None | Per request, per token |
Related: LLM gateway vs LLM router and the AI gateway comparison.