A stable context, for caching and for consistency

Most providers charge far less, and respond faster, for the start of a request they have already seen recently: this is prefix caching. For an agent that sends back an ever-lengthening context at every step, it is the major economic lever. But it follows one rule with no exceptions: the cache only holds up to the first character that changes.

The mistakes that kill the cache

• A timestamp at the top of the system prompt, accurate to the second: every request has a fresh prefix, nothing is ever reused.
• Unstable serialisation: a JSON object whose key order varies from one call to the next produces different text for identical content.
• Rewriting the past: modifying, summarising or reordering earlier steps in the middle of the history invalidates everything that follows.
• Changing the tool list along the way: tool definitions sit at the start of the context, so touching them invalidates almost everything.

Rules for building the context

• From most stable to most variable: instructions and tool definitions first, then reference documents, then the history, and the current request at the end.
• A history that only grows: you append to the end, you do not modify what came before.
• A date at the right granularity: the day is almost always enough, and it goes after the stable parts.
• Restrict rather than remove: if a tool should no longer be used at a given point, it is better to prevent its use (through a decoding constraint or a rule in your code) than to take it off the list.

Consistency benefits too

A context that is constantly rewritten does not only disturb the cache. If a tool disappears while the history still refers to it, the model may try to call it anyway, or get confused about what has been done. An append-only history is also a faithful record of what happened, for the model as well as for you in the event of an incident.

Measuring

Provider responses generally indicate how many tokens were read from the cache. Track this rate by task type: a sudden drop after a release almost always signals a character that has started moving at the start of the context.

Key points

• The cache holds up to the first character that changes: stabilise the start.
• Order: stable first, variable last; an append-only history.
• Restrict a tool's use rather than removing it from the list.
• Track the cache read rate: a drop reveals an unstable prefix.