Prompt caching

Read this first, because the two mechanisms share a screen and a word and are otherwise unrelated.

• The semantic cache is ours. It stores answers we have already served and returns one without calling a provider at all, when a new prompt is close enough to an old one. It saves the whole call. You turn it on, give it a TTL, and can bypass it per request.
• The prompt cache is the provider’s. Every request still reaches a provider and is still answered fresh; the provider simply charges less for the part of the context it has already processed for you. It saves a fraction of the input tokens on a call that happens anyway.

They can both apply to the same workspace and they never apply to the same request: a semantic-cache hit never reaches a provider, so it has no cached tokens to discount. Their savings are reported separately on the console’s Caching screen for exactly that reason, and the two figures must not be added together as if they were the same number measured twice.

An agent loop re-sends its accumulated context on every turn: the system prompt, the file it is editing, the tool output from three steps ago, and only then the new instruction. By turn twenty the new part is a rounding error and the repeated part is the bill.

Providers price that asymmetry. Once an endpoint has processed a prefix, it will bill the same prefix again at a cache read rate well below its list input rate. That is what makes a long agent session affordable.

The catch is that the cache belongs to one model on one provider. Route turn twenty-one somewhere else and the new endpoint has never seen any of it: the entire context is re-billed at full input rate, on the largest prompt of the session. A router that chased the cheapest list price every turn would generate exactly that bill, and would look like it was saving you money the whole time.

So Neural Router does not rank on list price. It ranks each candidate on an effective price for this request: the endpoint’s list input rate blended toward its cached rate in proportion to how much of your session is warm there. Every objective sorts on that number, including quality-per-dollar. A warm endpoint is genuinely cheaper for the request in hand, and the arithmetic says so — but it says so in dollars, not as a rule that pins you in place, so a colder endpoint that is cheaper even after the re-billing still wins.

Warmth is not a guess about what the provider probably cached. It is the provider’s own reported cached_tokens over prompt_tokens from your session’s previous turn on that endpoint. Nothing is inferred from prompt length, message count, or how a cache is supposed to behave.

Two consequences worth planning around:

• On the first turn nothing is warm anywhere, so there is no discount to weigh and the router chooses freely on quality, price and latency as it always would. Cache-aware routing costs you no flexibility at the moment you have the most of it.
• A session can be warm in more than one place at once. Cascade, hedging and failover all move a session between endpoints, and each endpoint’s warmth is tracked separately, from its own reported usage. Nothing is ever summed across endpoints and credited to one.

The correction is as literal as the measurement. If a provider reports no cache hit where the router had priced the endpoint as warm, that record is dropped immediately rather than decayed — one request is the most a wrong belief can cost.

Whether an endpoint caches at all, and what it charges to read that cache, is the provider’s behaviour and its price list. Neural Router does not create the cache; it reads what the provider reports about it and prices accordingly.

The depth differs enough to change decisions. At the time of writing, Anthropic’s cache read is roughly a tenth of its input rate — around 90% off — while OpenAI’s cached input discount is shallower. Those are the vendors’ published rates and they move; the point is the shape, not the digits. A deep discount can make an expensive warm model cheaper for a turn than a mid-priced cold one, while a shallow one may not survive contact with a genuinely cheaper rival.

That is precisely why this belongs in the router rather than in your client. A client-side rule — “stick to whatever answered last” — cannot know either rate, so it cannot tell a 90% discount from a 25% one and holds on to both with equal conviction. Neural Router holds the whole catalog: it compares the warm endpoint’s discounted price against every cold rival’s list price, and moves only when the arithmetic says to.

One honest limit: the router will not apply a saving it cannot quantify. If an endpoint has no cached rate on file, it is priced at its full input rate even when it is warm — which is exactly what your invoice will say, since billing falls back to the same rate.

Provider prompt caches are short-lived, so the router’s record of them is too. Both sides of that expire on their own clock, and both expire in the same direction:

• The provider’s cache goes cold on its own schedule. Pause a session over lunch and the context you paid to warm is gone at the provider’s end; the next turn is a full-price turn wherever it lands.
• The router’s record expires on a hard TTL — five minutes by default, matching the shortest common provider cache lifetime — and an entry that has not been refreshed within it is simply not consulted. An entry is refreshed only by a real measurement, never by a request that merely looked similar.

An expired record produces list-price ranking, which is the behaviour the router had before this feature existed. The same is true if the store backing it is unavailable: no warmth is reported, everything is priced at list, and a routing decision is never made on stale belief. The failure mode is a missed discount, not a wrong bill.

Warmth is tracked per session, so the router has to know which requests are the same conversation. There are two ways it finds out and you do not have to do anything for either:

• Send X-NR-Session with a stable id of your own if your client has one.
• Send nothing and the router derives a key from the stable prefix of the conversation — your system message and the first user message. That prefix does not change as an agent appends turns, so every turn of one loop groups together. This is the normal case for coding tools, which speak plain OpenAI and have no session header to send.

Session ids are scoped to your workspace before anything is stored, so a session id is never a boundary between tenants and two workspaces that both pick session-1 share nothing.

There is no request field to turn this on or off, and no header that forces a warm route. The behaviour is a pricing input to the ordinary routing decision, not a mode.

The Prompt cache section of the console’s Caching screen — below, and deliberately separate from, the semantic-cache figures — reports the last 30 days:

• Prompt cache savings. Realized dollars, summed across the window.
• Requests served warm. The share of requests where the provider reported cached tokens.
• Switched anyway. How many routes left a warm endpoint because a colder one was genuinely cheaper for that request. This is the feature working, not a fault — a router that never switched would be the sticky rule this one replaces. A count of zero over a busy window is the number worth asking about.

Per request, Logs carries a Cache saved column, so a session’s savings can be read turn by turn rather than only in aggregate.

The savings figure is measured, not projected. Each request’s saving is its provider-reported cached tokens priced at the difference between that endpoint’s input rate and its cached rate. It is not hits multiplied by an average, and it is not a modelled “what you would have paid”. Where a cached rate is not below the input rate, the saving recorded is zero.

Note the contrast on the same screen: the semantic cache’s savings line is explicitly an estimate (hits times the average cost of a non-cached request), because a call that never happened has no invoice to measure. The prompt-cache line has one.

Response headers name the endpoint that served the request and, when session grouping is surfaced for your deployment, the session it was grouped under:

X-NR-Model: claude-sonnet
X-NR-Route-Reason: best quality-per-dollar within latency budget
X-NR-Session: auto:9c1f4ab27e30d5b8   # the session this request was grouped under

The warmth itself is on the routing trace, under keys that are deliberately prefixed so they can never be mistaken for the semantic cache’s own cache field:

{
  "prompt_cache_warmth": 0.87,
  "prompt_cache_decision": "stayed_warm"
}

• prompt_cache_warmth is the measured warm fraction on the endpoint that was chosen, between 0 and 1.
• prompt_cache_decision is what that meant: cold (no warmth anywhere for this session), stayed_warm, or switched_anyway — the router held warmth somewhere and the arithmetic still sent the request elsewhere.

X-NR-Cache is unrelated to any of this: it reports the semantic cache only.

Cache-aware pricing is enabled per deployment. While it is off, ranking is arithmetically identical to list-price ranking, the trace carries no prompt_cache_* fields, and the console’s prompt-cache figures stay at zero. Nothing about your requests changes either way — there is no field to add and none to remove.

Compare it against the mechanism it is most often confused with on Semantic caching, see how effective price feeds selection on Model routing, and read how per-hop usage is billed on Pricing & service types.