Runbooks & Escalation Paths: Making Diagnosis Reusable
CoreSupport debugging and operational issue resolution · Difficulty 2/5
Explanation
Layer isolation and trace analysis (above) describe how to diagnose a SINGLE incident well. But an architect who wants a team to be operationally self-sufficient -- not dependent on the one person who happens to remember how a similar issue was solved six months ago -- needs to turn each diagnosis into a durable, reusable artifact. Two named deliverables do this: the RUNBOOK and the ESCALATION PATH. Neither is a new diagnostic technique; both are what you build FROM every diagnosis you already know how to do.
The Runbook: Turning a From-Scratch Investigation Into a Five-Minute Lookup
A RUNBOOK is a living document mapping SYMPTOM -> LIKELY ARCHITECTURE CAUSE -> FIRST DIAGNOSTIC ACTION, built up from real incidents as they happen. The first time a given failure pattern occurs, someone has to do the full trace-analysis, layer-isolation investigation from scratch -- that's unavoidable, and it can eat hours. The point of a runbook is that the SECOND occurrence of that same pattern shouldn't cost hours again. It should be a five-minute lookup: recognize the symptom, check the runbook, take the first action it names, confirm or move on to deeper investigation only if the known cause doesn't match this time.
| Symptom | Likely cause | First action |
|---|---|---|
| Sudden latency spike with no traffic change | Provider-side model version rollout | Check stop_reason and response headers for a model-version change; compare against the pinned version |
| Retrieval results all subtly off-topic starting at a specific timestamp | A scheduled re-index job ran with a stale embedding config | Check the re-index job's logs and config version against the last known-good run |
| A specific tool call starts silently returning empty results, no error thrown | Downstream API changed its response shape without a version bump | Diff a fresh raw response against a captured known-good sample from before the symptom started |
Notice the shape of every row: the symptom is something anyone on call would actually notice, the cause names a specific architectural layer (not "something's wrong"), and the first action is a concrete, checkable step -- not "investigate further," which isn't actionable. A runbook entry that just restates the symptom back as the "cause" isn't useful; the cause has to point at a specific mechanism the first action can actually go check.
The Escalation Path: Who Gets Paged When the Runbook Doesn't Cover It
An ESCALATION PATH is a defined, documented chain -- for example, on-call engineer -> platform team -> vendor support -- for issues that exceed what the runbook covers. Its entire purpose is to remove guessing about who to page during an actual incident, when guessing costs real minutes and real stress. Without a documented path, an on-call engineer facing a genuinely novel failure has to improvise who to contact, often by asking around a Slack channel and hoping someone senior happens to be online -- exactly the kind of ad hoc, person-dependent process the rest of this domain has been arguing against since Lesson 7.1's CLAUDE.md-hierarchy discussion of tribal knowledge trapped in one person's head.
A usable escalation path names, for each step, WHO (a role, not a specific person who might be on vacation), WHEN to escalate to that step (what specifically the previous step couldn't resolve), and WHAT information travels with the escalation (so the next person isn't starting from zero). "Escalate to the platform team if the runbook's first action doesn't identify the cause within 15 minutes, and hand them the incident timeline plus whatever traces have already been pulled" is a usable escalation criterion; "ask someone if you're stuck" is not.
The Relationship Between the Two Artifacts
The runbook and the escalation path work together as a triage funnel: check the runbook first (fast, cheap, self-service); if the symptom doesn't match a known entry, or the known first action doesn't resolve it, escalate along the defined path rather than either (a) the on-call engineer spending hours re-deriving a diagnosis from scratch when someone else already knows the answer, or (b) escalating immediately for every issue regardless of whether it's actually novel, which burns the time of increasingly senior (and increasingly scarce) people on things a runbook entry would have handled in five minutes.
Common exam traps
- Treating "whoever's around" as an escalation path. A scenario describing an incident where the on-call engineer has to guess who to contact, or wait for someone to notice a Slack message, is describing the absence of an escalation path -- the fix is a documented, role-based chain with clear criteria for when to move to the next step, not "communicate better" in the abstract.
- A runbook entry that only restates the symptom. "Symptom: latency is high. Cause: latency is high. Action: investigate." isn't a runbook entry -- it doesn't point at a specific architectural mechanism or give a concrete first action, so it saves no time on the second occurrence. Look for entries that name a specific cause (a layer, a config, a version) and a specific checkable first step.
Key Takeaways
- A runbook maps symptom -> likely architecture cause -> first diagnostic action, built up from real incidents, so the second occurrence of a known issue is a five-minute lookup instead of a from-scratch investigation
- An escalation path is a defined, documented chain (e.g., on-call -> platform team -> vendor support) for issues that exceed what the runbook covers, removing guesswork about who to page
- A usable escalation path names WHO (a role), WHEN to escalate (specific unresolved criteria), and WHAT information travels with the escalation
- The runbook and escalation path work together as a triage funnel: fast self-service lookup first, defined escalation only when the runbook doesn't cover it
- Both artifacts are what you build FROM every diagnosis you do -- not a new diagnostic technique, but the deliverable that makes a diagnosis reusable by the whole team
Related Concepts
Issue Isolation: Localizing Failures Across Layers
Isolate a failure to transport, integration/parsing code, retrieval, or model output before applying any fix
Production Monitoring & Observability
Log request/response pairs, tool calls, retrieval results, stop_reason, and token usage per hop