Cache reuse is the performance story for locally hosted AI, not tokens per second

Almost every benchmark you can find for a local model measures tokens per second. For a chat window that is the right number: a person is reading the output as it arrives, and all that matters is whether the text keeps up with their eyes.

For an agent working through a long task it is close to irrelevant. The number that decides whether a local model is usable there is what the eleventh tool call costs, and that one is almost never measured.

Depth

Each tool call is a fresh request. The agent sends the whole conversation, the server prefills it into the KV cache, and only then does the first token of the reply appear. Prefill is the wait. Generation is the part you can watch.

If the server can reuse the prefix it already has in its slot, prefill covers only the new bytes on the end and the wait is basically nothing. If it cannot, prefill covers everything, again, on every single call.

Industrial work goes deep in tool calls, deeper than people expect, because the code is the short part. Before a line gets written the agent has to find its bearings: walk an EtherCAT segment and see what is actually on it, browse an OPC UA server nobody documented, read holding registers to work out what the previous integrator meant. After the code is written it has to prove the thing works, which is another long run of reading the machine back. Driving devices, discovering topology, mapping address spaces. Hundreds of tool calls on either side of the writing. One conversion job we finished recently took 762 model tool calls and 3.59 million input tokens to get through, and that is an ordinary shape for this work rather than an outlier.

Measured on our own harness against an OpenAI-compatible server on our own hardware, on a 13.8k-token conversation, appending one turn:

Two orders of magnitude, from the same model on the same box.

Now put that inside a tool loop. At 0% reuse the cost of call n is the size of the conversation at call n, and the conversation grows with every result you feed back into it. The per-call cost is linear, so the cost across a session is quadratic. Ten calls deep is not ten times the first call, it is worse, and it keeps getting worse.

Arithmetic on a measured prefill rate of roughly 800 tokens per second, a 22k-token system and tool prefix, and about 600 tokens added per call:

That is arithmetic, not a benchmark, and it ignores compaction. We have all experienced this before: fine for three calls, sluggish by ten, unusable by forty. People conclude their GPU is too small. Usually their GPU is fine and their prefix is being thrown away.

None of this is much of a problem if you are calling one of the big hosted providers. Their compute is enormous and their caching is far more forgiving than anything you will run in a cabinet. I want to be straight about that: on a frontier API you can be careless with your prefix and mostly get away with it. On local inference with a good mid-tier model you cannot, and the use case simply dies.

The law

We went looking for partial credit and did not find any. Warm a slot with a long prompt, then send a variant:

It is not about growing or shrinking, and it is not about how much moved. What matters is how much of the slot's own cache the new prompt still covers. At a ratio of 0.90 the reuse is zero. At 0.97 and 0.999 it is near total. In practice only an append clears the bar.

Appending is free. Editing is all or nothing.

The consequence is the uncomfortable bit. There is no graceful degradation. A prompt that diverges 95% of the way through costs exactly what a prompt that diverges at token zero costs. We measured one of those: 1,984 cached tokens went to 1, and 120 ms went to 2,074 ms, because the request touched something near the end of a history the server was already holding verbatim.

Small changes

This is the part worth passing on, because every instance we have found has been small enough to review without noticing.

Case one: two lines of the system prompt. An earlier version of our harness emitted a model identity line and a "today" line into the system prompt, once per turn. Two lines. The system prompt therefore changed on every turn, the cache invalidated at token 0, and the entire prompt was re-prefilled. Three separate runs, including ones where the server had already seen an identical prefix, reported 0% reuse. Nobody had done anything reckless. Somebody had put a value that changes into a position that must not.

Case two: throwing away the model's own working. Our harness was dropping the model's reasoning blocks out of the history before sending the next request. The intent was reasonable, since reasoning is often the largest thing in the conversation and dropping it buys context headroom. The effect was that the history we sent back was not byte-identical to what the model had produced, so the next request was an edit of the slot rather than an append, and the law above prices an edit at zero.

Measured on a conversation of only ~2,900 tokens, one loop iteration:

Four hundred and forty-one milliseconds per iteration is nothing on a three-call task. On 762 calls it is the difference between a working tool and a demo.

The tell, once we knew to look for it, was almost funny: run the same message array through the template and compare the bytes. One render carried the model's actual working. The other carried an empty <think></think> where six thousand characters of reasoning had been. The model was being handed a transcript in which it had thought about nothing, and then asked to continue.

The bill

Back to the conversion project. 3.59M input tokens were presented to the model across those calls. What the harness actually had to prefill is the part that was not already in the slot.

Seventy minutes of an engineer's day, spent watching a cursor, on one job. Not spent generating anything. Spent re-reading a preamble the server had already read.

The invariant

Once we accepted the measurements, this stopped being an optimisation and became a constraint on the design. Our constraints for the harness were fixed:

• it must perform well with smaller models
• it must perform well on a local computer
• it must perform well on long running tasks

The third one is the one industrial work forces on you, and it is the one that makes the cache non-negotiable. So we wrote the rule down and made it outrank features. The short version:

1. Append, never insert or rewrite. Nudges, warnings, system events: all appended at the tail, where they cost only their own tokens.
2. Nothing in the prefix may vary per turn. No timestamps, no counters, no "turn N of M", no elapsed time. If it differs between turn 1 and turn 2 it does not belong in the prefix.
3. Nothing in the prefix may vary by machine. Directory listings get sorted before they enter a prompt, because a prefix that depends on filesystem ordering cannot be cached across a fleet.
4. Optional things register last, and their absence must be byte-identical. Turning on the Modbus pack must not shift a single byte in front of it. Adding one MCP server must not move the built-in tool schemas.
5. Compaction is the one sanctioned invalidation. It rewrites history, so it costs a cold prefill. It is paid once and bought deliberately, never per turn.
6. Session-constant is fine, turn-variant is not. A value fixed at process start costs one miss on the first turn. That is a different class of problem entirely.

The enforcement matters more than the list. Our assembled system prompt is asserted byte for byte by a fixture in the test suite. A failure there is not a broken test, it is the rule catching you, and the fixture only changes as a deliberate act. Any change to the system prompt, the tool registry, a tool schema, registration order or history rendering needs explicit sign-off with the measured cost stated. "It is only a few tokens" is precisely the reasoning that produced the 0% column above.

What that buys, measured across live sessions: 94 to 100% prefix reuse in steady state with turn boundaries included, a turn boundary at 134 ms, and a persistent turn landing in about 0.35 s.

Prefix economics

The rule also changed how we add capability, which was not something we anticipated.

Every tool you register is permanent prefix. Five protocol tool schemas cost roughly 1,300 tokens of prefix on every request for the life of the session. Five skill descriptions, one name and one line each, cost roughly 100, with the body loading on demand and arriving as a tool result at the tail where it is cache-neutral.

Thirteen times the permanent cost for the same domain knowledge. So when we can express something as a skill rather than another tool, we do, and the descriptions are kept short and prescriptive because the description is the part you pay for forever.

Limitations

Compaction still costs a full cold prefill. We spent a while believing that pinning the head of the conversation byte-identically would let the front of the prompt survive a compaction. It does not, and the measurement is unambiguous: a 31,268-token prompt compacted to 15,018 behind a provably identical 9,940-token head reused zero tokens and took 18.5 seconds. The head was genuinely reusable, and the next request proved it by reusing 9,927 of it. Compaction is simply an edit, and edits are all or nothing. What follows from that is a policy question about when to compact, not a trick that makes it free.

Some of this is server-dependent. The all-or-nothing threshold is a property of the inference server's slot matching, and there are server flags that should widen it. Ours is remote and its launch flags are not ours to set, so that remains a hypothesis rather than a result and we have not claimed it as one.

One claim of ours is still an observation, not a measurement. Carrying the model's reasoning forward appears to reduce redundant tool calls, because without its own working in front of it the model re-derives the same conclusion and restates it as new. One session spent 15 of 18 tool calls re-verifying two facts it had already established. That is what prompted the work, and the cache numbers above are measured, but the behavioural half needs a real session counted properly before we will put a number on it.

If you are running a coding agent against a local model and it feels fine for three tool calls and unusable by twenty, look at your prefix before you look at your GPU. The failure will not be big. It will be a line someone added without thinking about where it sat.

Oasis CLI is free to download and free to use, on as many machines as you like. Everything above is what it does by default.

curl -fsSL https://get.mutexer.com/oasis/install.sh | bash

More on what it is and why it goes on the machine rather than beside it is in the launch post, and every published build is on the releases page.

If you have run a long agent session on local inference and measured something that disagrees with any of this, we would genuinely like to see it.