Governance · Cost management · Deep dive
Prompt caching: the cheapest tokens are the ones you already sent
Every call an agent makes re-sends its whole prompt: the instructions, the tools, the conversation so far. Prompt caching lets the model provider keep the part it has already processed and charge a fraction for it. It is the largest single lever on agent token spend, and it works only when someone manages it, per model family.
What it is
A language model has no memory between calls. Each request carries the full prompt, and the provider charges for every token in it, every time. For a chat message that is a few hundred tokens. For an agent it is different: a working agent carries a long system prompt, a list of tool definitions and a growing transcript, and it makes a new request for every tool call. Forty requests per task is ordinary. Most of those forty prompts are identical up to the last few lines.
Prompt caching exploits that. The provider stores the processed form of a prompt prefix, and when the next request starts with exactly the same bytes, it reuses the stored work instead of redoing it. The reused part is billed at a cached rate, typically a tenth of the normal input price and on some models less, and it also comes back faster.
The word that matters is prefix. The cache matches from the start of the prompt forward, in the order the provider renders it: tool definitions, then the system prompt, then the conversation. A change anywhere invalidates everything after it. A stable prompt caches well. A prompt with a clock in its header caches nothing.
Why it matters for agents in particular
Take an agent whose fixed context (instructions plus tools) is 60,000 tokens, and a task that takes 40 requests. Priced at a current frontier model's public rates (4 USD per million input tokens, 5 USD to write to the cache, 0.20 USD to read from it):
| Fixed context, 40 requests | Input tokens billed | Cost |
|---|---|---|
| No caching | 2.4 million at the normal rate | about 9.60 USD |
| Cached after the first request | 60,000 written once, 2.34 million read | about 0.77 USD |
Twelve times less, on the part of the bill that dominates agent work. The growing transcript caches the same way, so the real gap is usually wider. Output tokens are not cached and are unaffected either way.
We learned this on our own bill. In our previous environment the proxy logs showed 8,716 calls to a frontier model with zero cached tokens. The prompts were cacheable; nothing had asked the provider to cache them. Adding the cache markers at the proxy, once, changed the unit economics of every agent behind it. (Source: our own platform logs, 2026.)
Every model family does it differently
This is the part that turns a feature into a management problem. Caching is not one thing you switch on. Each provider has its own rules for whether it happens at all, how much of a prompt qualifies, what it costs to write, how long the entry lives and how you can tell it worked.
| Family | How it is switched on | Smallest cacheable prompt | Cached read price | Lifetime |
|---|---|---|---|---|
| Anthropic Claude (direct, Vertex AI, Bedrock) | Opt-in. The caller marks a breakpoint, or sets one request-level marker and lets the provider place it. | 512 to 4,096 tokens, by model | 0.1× base; 0.05× and 0.025× on the newest models. Writes cost 1.25× base (2× for the long lifetime). | 5 minutes, refreshed on every hit; 1 hour as an option |
| OpenAI | On by default. A cache key can steer requests to the same cache. | 1,024 tokens | About 0.1× base on current models; writes at base price | About 30 minutes after last use; longer retention on some models |
| Google Gemini | Implicit caching on by default on 2.5 and newer. Explicit caches can be created and held. | 2,048 to 4,096 tokens, by model | 0.1× base. Explicit caches add an hourly storage fee. | Implicit: short. Explicit: whatever lifetime you set, and pay for |
| Open-weight models on a serving platform (GLM, Kimi, DeepSeek on Fireworks) | On by default. A session-affinity header keeps one conversation on one replica. | Not published | About 0.5× base on shared capacity; near free on dedicated capacity | Not published |
| xAI Grok | Provider-managed | Not published | 0.16× to 0.25× base, by model | Not published |
Figures are from each vendor's public documentation on 30 September 2026 and change often. The point is not the numbers. It is that a rule which is true on one family is false on the next, and an estate that routes work across families needs to know which rule applies to each request.
Why it has to be managed, not assumed
Five things go wrong quietly. None of them produces an error. All of them show up only as a bill.
- Silent invalidators. A timestamp in the system prompt, a per-request ID, a tool list that renders in a different order, a memory file that is rewritten before every call. Each one breaks the prefix, and the provider processes the whole prompt at full price without saying so. The only symptom is a cached-token count of zero.
- Family drift. A harness tuned for one provider assumes its rules. Route the same agent to a second family and the markers are ignored, the minimum length is different, or the cache is keyed on something the harness never sends. Caches are also scoped to a model: a router that moves a session between models mid-task throws away the cache with every hop.
- Lifetime versus cadence. A cache that lives five minutes is worth nothing to a job that runs every eight. Worse, it costs more than no caching, because the write price is above the base price and every call is a write. Scheduled agents, sweeps and long tool calls all sit near this line.
- Misattributed spend. If the metering layer counts every input token at the normal price, budgets overstate spend, the saving is invisible, and nobody can see which seats cache well and which do not.
- Load balancing. Provider caches live on specific servers. Traffic spread evenly across replicas, or across regions, misses caches that a single sticky route would hit.
What Governance does about it
Caching is a property of the request path, so it belongs in the same place as metering, budgets and routing: the governed request path. On supported paths Brightwire treats it as a managed control, not a vendor default.
- Priced in the registry. Every model in the approved model registry carries a cached-input price alongside its input and output prices. Metering books cached tokens at the cached rate on every request, so attribution shows the saving and a cache-read count of zero is visible instead of buried.
- Stable prefixes by design. Agent harnesses are assembled with the fixed material first and the volatile material last. Clocks, request IDs and per-turn notes go after the last breakpoint. It is a convention the reviewer seat checks, not a reminder to authors.
- Markers added once, at the proxy. Where a family needs explicit cache markers, the right place to add them is the shared proxy, once, for every agent behind it, so no team has to know which provider it is talking to today. That is how we fixed our own zero-cache problem, and it is enabled family by family on the current platform (see below).
- Routing that respects the cache. Model routing is designed to decide once per session and hold the model while a tool loop is in flight, so a task does not hop between families and lose its prefix on the way.
- Hit rate as a budget signal. Cached tokens divided by input tokens, per seat and per model, is a number the cost owner can watch. A drop is usually a harness change, and it is caught the day it happens rather than on the invoice.
Five questions to ask about any agent you pay for
- What share of its input tokens are cached reads? If nobody knows, assume zero.
- Which model families does it run on, and does the harness meet each one's rules for caching?
- Does anything in its prompt change on every call?
- How often does it run, and how long does the provider's cache live?
- Does the bill you see price cached tokens at the cached rate?
What we do not claim
- We do not control the vendors' caches. Lifetimes, prices and minimums are theirs and change without notice; the table above is a snapshot.
- We do not promise a hit rate. Hit rates depend on how an agent is written and how often it runs, and some workloads are genuinely uncacheable.
- Marker injection is enabled per family and verified environment by environment. Where it is not yet on, attribution still shows the cached-token count, so the gap is visible.
- Closed vendor user interfaces that do not pass through the governed path cannot be observed, and no cache figure is claimed for them.
Soft next step: if you want to see the cached-token share for one of your own agents, start a conversation.