Observing a non-deterministic system

A classic service either fails or responds. A model-based system always responds, and can degrade slowly without any alert going off. That is what makes operating it so particular.

What to track, and what is not obvious

The usual metrics, latency, HTTP error rate, availability, capture none of the real problem. You need to add, for each call: the exact model and version, the number of input and output tokens, the calculated cost, the prompt version, and whether a fallback was triggered.

Without the prompt version in your traces, you will never be able to link a degradation to a change. And prompts change more often than code.

The signals that warn you before the complaints

Response length. It drifts before quality collapses. An average that doubles in a week almost always signals a problem.

The rate of non-conforming outputs. If you expect JSON and 3% fail parsing, that is a free and very sensitive thermometer.

The rephrasing rate. A user who asks their question again in different words has just told you the answer was bad, without filling in any form.

Version prompts like code

In the repository, with a review, a version identifier, and the ability to roll back. A prompt changed in production without a trace is an incident you will not be able to diagnose.

Sample for review

Keep a sample of complete interactions, and manually review thirty of them per week. No metric replaces reading: that is where you notice the tone has shifted, the answers have become evasive, or the model has picked up a tic.