Prompt Caching: Measure the Prefix Before Flipping the Flag

This note is about an LLM cost optimization, LLM being a large language model. But the real subject is the order of operations: the measurement came before the optimization, and it overturned the hypothesis that had motivated the measurement. What remained was a small change, a number that justifies every line of it, and an honest list of what still cannot be claimed.

The context, in generic terms: an agent service in production, coordinator plus subagents, handling operational messages via webhook, the call another system sends into ours. Models served through OpenRouter, with the OpenAI protocol in front and Anthropic behind. That detail will matter.

---

The wrong hypothesis, and the number that overturned it

The original suspicion was plausible. A good share of messages is trivial (“ok”, “thanks”, “good morning”), and trivial messages traversing an agent graph are expensive. The plan was to classify and divert the trivial ones.

The first step, though, wasn’t building the classifier. It was instrumenting: tagging trivial messages in analytics and measuring what they cost. Result of the production measurement (one day in 2026-08):

metric	value
p50, whole message	9.7s
p95	21.9s
input tokens per call	~18,000
calls per message	2
cost per message	$0.07

The conclusion the measurement imposed: it’s not the trivial share that hurts. It’s the price of any message. Every call, trivial or not, carried ~18,000 input tokens, the chunks of text that go into the price. The remedy wasn’t a classifier. It was looking at what those 18,000 tokens are made of.

The decomposition: what an agent payload is made of

The next step was assembling the real payload offline, the package that goes out to the model. The HTTP client was the same one production uses, to capture the exact serialization. And then, the decomposition:

part	size (chars)
tool definitions (24, as JSON)	26,087
system prompt (coordinator + areas + modules + routing + memory + middlewares)	34,767
the user’s message	~105
stable prefix	60,854 = 99.8% of the payload

Then came the verification that turns the number into a decision. The prefix is the opening stretch of the payload that doesn’t change from one message to the next: two different messages produce it byte-for-byte identical. Not “similar”: identical, verified. That’s what prompt caching demands. The provider caches by exact prefix, and a single differing byte in the middle invalidates everything after it.

This is the typical profile of any agent service with tools: the conversation is a drop, the scaffolding is the ocean. But “typical” is not a measurement. What authorized the change was the 99.8% of this system, measured on this payload. If your stable prefix is 60%, the whole calculation changes.

The implementation: one breakpoint, in the right place, for the right reason

With the number in hand, the change is small. And every decision in it has a why:

Where to intercept. A subclass of the chat client overrides the last point before HTTP. That is after all the agent framework’s middlewares have appended their blocks to the system prompt, the fixed instructions that ride along on every call. Marking earlier is marking a payload that will still change. In the tests, which inject fake models, the subclass isn’t involved.

Where the mark goes. A single cache_control: {"type": "ephemeral"} on the last block of the last system message. The insight that avoids a hack is Anthropic’s rendering order: tools → system → messages. A breakpoint at the end of the system, that is, the point where the cache is cut, also caches the tool definitions, which come before it. There is no per-tool mark in the OpenAI format OpenRouter accepts, and there doesn’t need to be: one well-placed breakpoint covers the payload’s two biggest blocks.

Why the mark is explicit. OpenRouter doesn’t enable caching on its own for Anthropic models (checked in the docs on the date of the change): either the payload carries cache_control, or there is no cache. The “let the provider decide” flag doesn’t exist on this path.

The guards. The marking is idempotent: applied again, it does not create a second breakpoint. A payload without a system message passes through untouched. And there’s a size floor (~4,000 chars ≈ 1,024 tokens) below which the mark isn’t applied. The provider has cache minimums (1,024 tokens on the larger model, 4,096 on the smaller), and marking a payload below the minimum only spends bytes.

One env var turns it all off (PROMPT_CACHE=0). The extended 1-hour TTL, the time the cache survives, was documented but left off by default. Writes at 2× instead of 1.25× only pay off with three or more reads in the window: it’s arithmetic, not preference.

What can be claimed — and what can’t yet

Here the note gets boring on purpose, because this is where most caching posts lie out of enthusiasm.

The cost effect is, for now, an arithmetic model, not a production measurement. Cache reads come in at ~0.1× and writes at 1.25× of the input price. With that, an isolated message drops to ~0.67× of the previous cost. A second message within the 5-minute window drops to ~0.10×. The ~18,000 tokens are still being sent: what changes is their price.

On latency, the provider stops reprocessing the prefix. That should cut most of the time-to-first-token, the wait until the answer’s first word, on calls 2..N. “Should”: still no number measured after activation.

The house rule is not to publish the number before two weeks of real use. The instrumentation for it shipped with the feature, not after. There are cache token columns, counted as a subset of input and not additive to it, a distinction that prevents double-counting in reports. There is also a hit_pct in the internal analytics and an exported metric.

When the number exists, it will come out of telemetry, not estimation. If you copy one thing from this note, copy that: a cost feature is born with its own meter.

The signal’s runbook was born alongside it too. A runbook here is the operator’s procedure: hit_pct near zero for two days without a deploy means investigation. Something is varying in a prefix that should be stable.

And the same ruler explains the cache’s hidden cost: editing any file that composes the system prompt (coordinator, areas, skills, memory) changes the prefix and invalidates the cache. At deploy time, expected; outside it, a symptom. Caching turns “I tweaked a prompt” from a free change into a priced one. For a system that treats prompts as versioned configuration, that is almost healthy: the price was already paid in review, and now it shows up on the meter.

A second-order effect on design. Before the cache, enabling a new module for a client made every message more expensive (more tools, more system prompt, more tokens on every call). With the prefix cached, the bigger scaffolding is paid roughly once per window, not per message. The marginal cost of installed capability dropped. What remains expensive per use is delegation: each subagent invoked is a fresh model round. The system’s cost accounting shifts from “how many modules” to “how many delegations”, and that reorders what’s worth optimizing.

The trap this note wants you to avoid

The symmetric error exists on both sides:

• Enabling the cache without measuring the prefix. A timestamp, a session id, or a block of dynamic context in the middle of the system prompt limits the cache to a fraction. You find out late, or worse, you don’t find out: the provider dashboard shows “caching active” and everyone stays happy.
• Optimizing the hypothesis instead of the measurement. The trivial-message classifier would have been built and would have worked. And it would not have changed the bill, because the bill was never about the trivial messages.

The boring path that worked: instrument → measure → decompose → verify byte-for-byte stability → change little code → instrument the effect → only then hold an opinion. The 99.8% wasn’t the argument for enabling the cache; it was the fact that made argument unnecessary.

What to do on Monday

1. Instrument before you optimize. Tag the hypothesis you want to test in your analytics and measure the real cost per message. Only then decide whether the classifier (or the bypass, or the smaller model) needs to exist at all.
2. Assemble the real payload offline, using the same HTTP client production uses, and decompose it part by part: tool definitions, system prompt, user message. The size of each one decides everything else.
3. Prove byte-for-byte stability. Generate the payload for two different messages and compare the prefix. “Similar” does not cache: a single differing byte invalidates everything from that point on.
4. Set exactly one breakpoint, on the last block of the last system message. The rendering order (tools → system → messages) makes it cover the tool definitions too. Add a size floor so you never mark a payload below the provider’s minimum, and an env var that turns it off.
5. Ship the instrumentation with the feature, never after: cache tokens as a subset of input (not additive to it), hit_pct in analytics, and the runbook written down. hit_pct near zero for two days without a deploy is an investigation, not patience.