Keeping the bill down.

A prototype costs a few euros. The same system at full scale can cost a thousand times more, and the surprise always arrives at the first complete bill.

The three cost drivers, in order of importance

Re-emitted context. In a conversation, the entire history goes back out on every turn. Twenty exchanges mean quadratic growth in cumulative cost. This is the biggest line item, by far, on any conversational product.

Output tokens. Three to five times the price of input. A conciseness instruction is a budget decision.

Invisible reasoning. On a reasoning model, it can account for most of the bill on a short answer.

The levers, from most to least cost-effective

Prefix caching. Put everything stable, instructions, examples, schemas, strictly at the top and identical. The reduction is substantial and costs only a bit of discipline. A single changed character cancels the benefit.

Routing. Classify the request and send only what deserves it to the expensive model. On real traffic, most requests don't deserve it.

History compression. Beyond N turns, summarise older messages rather than re-emitting them.

Batch processing. For anything non-interactive, deferred pricing is significantly lower. An overnight report doesn't need to respond in two seconds.

The essential safeguard

A spending cap per user and per period, enforced server-side. Without it, a poorly written loop or abusive usage produces a four-figure bill overnight, and you'll find out the next day.