Production Monitoring & Observability
CoreMonitor production systems with logging, dashboards, and alerting · Difficulty 2/5
Explanation
Evaluation doesn't stop at launch. Production monitoring is what turns the offline eval into a continuous quality signal.
What to Log
Log request/response pairs, tool calls, retrieval results, `stop_reason`, and per-hop token usage for every production interaction. This is the raw data that both dashboards and later trace analysis depend on -- if it isn't logged, it can't be diagnosed after the fact.
What to Dashboard
Surface the operational signals that matter continuously:
- Latency
- Error rates
- Cost
- Per-domain quality metrics
What to Alert On
Alert on regressions, not just absolute thresholds:
- A spike in retrieval failures
- A latency SLA breach
- A cost anomaly
- A drop in eval score after a change
Sampling and Re-Scoring
Sample and re-score live traffic periodically to catch drift the offline eval didn't cover. Real-world input distributions shift over time in ways a static held-out set can't anticipate; periodic re-scoring is how you detect that shift before it becomes a visible incident.
Closing the Feedback Loop
Monitoring closes the feedback loop: it turns the offline eval into a continuous quality signal that feeds the next iteration -- design changes, A/B tests, and optimizations are all informed by what monitoring surfaces in production, and the cycle continues into the broader solution lifecycle.
Key Takeaways
- Log request/response pairs, tool calls, retrieval results, stop_reason, and token usage per hop
- Dashboard latency, error rate, cost, and per-domain quality metrics
- Alert on regressions: retrieval-failure spikes, SLA breaches, cost anomalies, and eval-score drops
- Sample and re-score live traffic periodically to catch drift the offline eval missed
- Monitoring turns the offline eval into a continuous signal feeding the next design/eval iteration
Glossary Terms
Related Concepts