4.4 Monitoring, Logging & Observability
4.4.1 Evaluation Doesn't Stop at Launch
Every lesson so far in this domain has been about OFFLINE evaluation — measuring quality against a held-out dataset before something ships. But a held-out set, no matter how representative, is a snapshot frozen at the moment you built it. Real users ask things you didn't anticipate, documents get added and removed, and the distribution of inputs quietly shifts over months in ways no offline test set can predict. If your only quality signal is the eval you ran before launch, you're flying blind the moment the system goes live.
Task Statement 4.6 is about closing that gap: PRODUCTION MONITORING turns the offline eval into a CONTINUOUS quality signal. Instead of measuring quality once before launch and hoping it holds, you log what's actually happening in production, watch dashboards built from those logs, alert when something drifts, and periodically re-score a sample of real traffic the same way you scored your held-out set. Monitoring is evaluation that never stops.
Monitoring is the link that closes the loop: production signals feed back into the next design change, A/B test, and eval, rather than the offline eval being a one-time gate.
The one idea to hold onto
Production monitoring turns the offline eval into a continuous quality signal. It closes the feedback loop — production signals inform the next design change, A/B test, and evaluation cycle, connecting Domain 4 back into the broader solution lifecycle.
4.4.2 What to Log — the Raw Material for Everything Else
Dashboards, alerts, and trace analysis are all downstream of one thing: whether you actually captured the right data in the first place. This is worth stating plainly because it's easy to treat logging as boring infrastructure work you'll get to later — but if it isn't in place BEFORE an incident, there is no trace to analyze after the fact, no matter how good your diagnostic skills are (recall trace analysis from Lesson 4.2).
For every production interaction, log: REQUEST/RESPONSE pairs (what the user asked, what the system answered), every TOOL CALL the agent made and what it returned, every RETRIEVAL RESULT (which chunks were fetched and from where), the stop_reason for each turn, and PER-HOP TOKEN USAGE (how many tokens each step of a multi-step pipeline consumed). Together, this is exactly the raw material trace analysis needs to find the earliest deviation in a multi-step failure — the logging and the diagnostic technique are two halves of the same capability.
- •Request/response pairs — what was asked, what was answered
- •Tool calls and their results — what the agent did and what came back
- •Retrieval results — which chunks were fetched, from where
- •stop_reason — why each turn ended (tool_use, end_turn, max_tokens, etc.)
- •Per-hop token usage — cost and volume at each step of a multi-step pipeline
4.4.2 — Key Concept
Log request/response pairs, tool calls, retrieval results, stop_reason, and per-hop token usage for every production interaction. This is the raw data both dashboards and later trace analysis depend on — without it in place beforehand, there is nothing to diagnose after the fact.
4.4.3 What to Dashboard and What to Alert On
Raw logs are too granular to watch in real time, so you aggregate them into dashboards that surface the signals an architect actually needs to check on continuously: LATENCY, ERROR RATES, COST, and PER-DOMAIN QUALITY METRICS. Notice that this list is essentially the five-dimension eval framework from Lesson 4.1, made continuous — the same categories you measured offline before launch are the ones you watch live after launch.
Dashboards answer "what's the current state," but you don't want to have to stare at a dashboard all day to catch a problem — that's what ALERTS are for. Alert on REGRESSIONS, not just static absolute thresholds: a spike in retrieval failures, a latency SLA breach, a cost anomaly, or a drop in eval score following a recent change. The key word is regression — you're watching for a change in a signal relative to its normal baseline, which is what actually indicates something broke, rather than a fixed number that might be perfectly normal for this particular workload.
| Surface | Purpose | Examples |
|---|---|---|
| Dashboard | Continuous visibility into current state | Latency, error rate, cost, per-domain quality |
| Alert | Proactive notice of a regression, without needing to watch constantly | Retrieval-failure spike, SLA breach, cost anomaly, eval-score drop after a change |
Dashboards make current state visible on demand; alerts proactively surface a regression against baseline so nobody has to be watching continuously to catch it.
4.4.3 — Key Concept
Dashboard latency, error rates, cost, and per-domain quality metrics — the same categories from the offline eval, made continuous. Alert on REGRESSIONS relative to baseline (retrieval-failure spikes, SLA breaches, cost anomalies, eval-score drops after a change), not on fixed absolute thresholds alone.
4.4.4 Sampling and Re-Scoring: Catching Drift the Offline Eval Missed
Dashboards and alerts catch problems that show up as an obvious spike or breach. But some quality problems are subtler than that — a slow, gradual drift in the distribution of real user inputs that never trips an alert threshold but steadily erodes quality all the same. A held-out set built six months ago simply can't represent input patterns that emerged after it was assembled.
The fix is to SAMPLE and RE-SCORE live traffic periodically, using the same scoring methods (code grading, validated LLM-as-judge, human review) you built for the offline eval. Pull a representative sample of recent real requests, score them the same way, and compare against the bar the offline eval set. This is how you catch drift the static held-out set was never positioned to see — because it's measuring the CURRENT distribution of real traffic, not a snapshot from months ago.
This closes the loop described back in 4.4.1: what monitoring surfaces — a drift detected through re-scoring, a regression flagged by an alert, a cost anomaly on a dashboard — feeds directly into the next design change, the next A/B test (Lesson 4.2), and the next run of the offline eval (Lesson 4.1). Monitoring isn't a separate, terminal phase after "the real work" of building and evaluating the system — it's the mechanism that keeps that work honest for as long as the system stays live.
4.4.4 — Key Concept
Sample and re-score live traffic periodically, using the same scoring methods as the offline eval, to catch drift a static held-out set can't anticipate. Monitoring signals — drift, regressions, anomalies — feed back into the next design/eval iteration, closing the feedback loop into the broader solution lifecycle.
4.4.4 — Exam Trap
A scenario describing quality that degrades gradually, with no single spike or breach, is testing whether you reach for periodic sampling-and-re-scoring rather than assuming the existing dashboards and alerts would have caught it. Static offline evals and threshold alerts are not designed to detect slow distributional drift — only ongoing re-scoring of live traffic is.
4.4.5 Put It Together: Close the Loop
You now have the full arc of Domain 4: define metrics across five dimensions (4.1), build a representative held-out eval with mixed scoring methods (4.1), improve deliberately via A/B testing and diagnose regressions via symptom-matching and trace analysis (4.2), optimize cost and latency without breaking quality (4.3), and monitor the live system so the whole cycle continues after launch (4.4). Each lesson depends on the one before it — you can't diagnose without an eval to compare against, you can't monitor without logging that trace analysis also depends on, and you can't optimize responsibly without a quality bar to verify against.
4.4.5 — Build Exercise (45 min)
(1) Instrument a multi-step pipeline to log request/response pairs, tool calls, retrieval results, stop_reason, and per-hop token usage for every request. (2) Build a simple dashboard summarizing latency, error rate, cost, and a quality metric over a rolling window. (3) Configure at least one regression-based alert — for example, a retrieval-failure-rate spike relative to a 7-day baseline, not a fixed absolute number. (4) Set up a periodic job that samples 1% of live traffic, scores it with the same method used in your offline eval, and compares the score against the offline bar. (5) Trace a manufactured incident end-to-end: an alert fires, you pull the logs, run trace analysis to find the earliest deviation, fix it, and confirm the fix by re-running the offline eval before shipping.
This closes Domain 4. The next domain in the CCAR-P exam guide, Governance, Safety & Risk, picks up a related thread: many of the same logs and traces you built here for quality monitoring are also the evidence trail an architect needs for safety review and incident response.
Where this shows up on the exam
4.6 questions describe a production symptom (a teammate missing a signal, a slow drift, an alert design) and ask what to log, dashboard, alert on, or how to catch drift. Anchor on: log the five categories, dashboard the eval's dimensions made continuous, alert on regressions not fixed thresholds, and re-score sampled live traffic to catch what the static held-out set can't.
Key Takeaways
- ✓Production monitoring turns the offline eval into a continuous quality signal — it doesn't stop at launch.
- ✓Log request/response pairs, tool calls, retrieval results, stop_reason, and per-hop token usage for every production interaction — this is the raw data trace analysis and dashboards both depend on.
- ✓Dashboard latency, error rates, cost, and per-domain quality metrics — the same categories as the offline eval's five dimensions, made continuous.
- ✓Alert on REGRESSIONS relative to baseline (retrieval-failure spikes, SLA breaches, cost anomalies, eval-score drops after a change), not fixed absolute thresholds alone.
- ✓Sample and re-score live traffic periodically, using the same scoring methods as the offline eval, to catch drift a static held-out set can't anticipate.
- ✓Without logging in place BEFORE an incident, there is no trace to analyze after the fact — logging is a prerequisite for diagnosis, not an optional add-on.
- ✓Monitoring closes the feedback loop: signals it surfaces feed the next design change, A/B test, and evaluation cycle.
Check Your Understanding
Test what you learned in this lesson.
Q1.Six months after launch, a system's offline eval score (last measured at launch) still looks fine on paper, but real users have started asking a new category of question the held-out set never covered, and quality on that category is quietly poor. What practice is designed to catch this?
Q2.A production incident occurs, and the team wants to run trace analysis to find the earliest deviation in a multi-step pipeline, but discovers that tool-call results and retrieval hits were never logged. What does this reveal?
Q3.A team sets a fixed alert threshold: "page on-call if latency exceeds 3 seconds." Six months later, a gradual latency creep from 1.2s to 2.8s goes completely unnoticed because it never crosses 3 seconds. What alerting principle would have caught this earlier?
Q4.Which set of items should be logged for every production request in a Claude-based agent system, to support both dashboards and later trace analysis?
Practice This Lesson