Self-hosted LLM vs managed API
Self-hosting an open model looks dramatically cheaper on a spreadsheet. Per-token, it often is. The spreadsheet usually omits the parts that dominate the real total.
What self-hosting actually costs
- GPUs you pay for idle. Managed APIs bill per token; a reserved GPU bills per hour whether or not traffic arrives. Utilization is the whole game — an H100 at 15% utilization is far more expensive per token than any API.
- Engineering time. Serving engine tuning, autoscaling, batching configuration, model upgrades, and being on call for it.
- Capacity headroom. You must provision for peak, not average.
- Model lag. Frontier models are not available to self-host. You are choosing an open model, which for some workloads is a quality decision, not just a cost one.
Where the break-even really sits
As a rule of thumb: self-hosting starts winning when you have sustained, predictable, high-volume traffic on a model you control — steady utilization above roughly 60–70% on the hardware you have reserved. Spiky or growing-but-unpredictable traffic favors managed APIs, because you are not paying for the trough.
Most teams end up hybrid
The stable end state is usually both: self-hosted open models absorbing high-volume, well-specified work at low marginal cost, with managed frontier APIs handling the hard minority of requests and the overflow when your fleet saturates.
That only works if something routes between them on live signals — utilization, queue depth, latency, and cost. Neural Router treats a self-hosted endpoint as just another provider, so a single OpenAI-compatible endpoint spans both. For engine selection, see the best LLM inference engine.