Observability for Production Integrations

Production integrations need to be observable so that regressions are caught before users notice them, and so that failures can be diagnosed rather than guessed at.

What to Log and Trace

• Request/response pairs
• Tool invocations and their arguments
• Retrieval hits
• `stop_reason`
• Per-hop token usage

Tracing Multi-Step Runs

Trace multi-step runs so a failure can be walked back to its *first* deviation rather than inspected only at the point it became visible. This connects directly to the evaluation/testing domain's diagnostic practice: the earliest deviation in a trace is usually the actual root cause, not the step where the failure surfaced.

Scaling Observability

At scale, four practices let you detect regressions — a broken re-index, model-version drift — before users do:

• Sampling — inspect a representative subset of traffic rather than everything
• Structured logs — machine-parseable records, not free text
• Latency/error dashboards — surface drift in real time
• Per-domain quality metrics — catch regressions specific to one integration surface

A Worked Diagnostic

If a RAG system suddenly returns confident but incorrect answers right after a document refresh, while latency and model version are unchanged, the retrieval/indexing step is the first place to look — not the model. A broken re-index or mismatched embeddings will produce exactly this symptom: high-confidence, wrong answers, with nothing else in the system having changed.