Confidence Calibration
PatternsDefinition
The practice of having Claude estimate and report its confidence in its output, then using that estimate to determine whether to proceed autonomously or escalate. Requires explicit confidence scoring in prompts and defined thresholds for escalation vs. auto-approval.
Example Usage
Instruct Claude to rate confidence 1-5 and escalate anything below 3 to human review before taking action.
Why It Matters for the CCA-F Exam
Exam questions on confidence calibration test whether candidates understand the full loop: prompt design produces a score, a threshold maps the score to an action, and the action connects to an escalation or approval workflow. A common distractor presents a system that collects confidence scores but has no threshold-to-action mapping — the exam expects you to identify this as an incomplete implementation.
In Depth
What Calibration Means in This Context
Calibration, in statistics, means that a system's stated confidence matches its actual accuracy rate — if a calibrated system says it is 80% confident, it should be right about 80% of the time. In LLM agent design, perfect statistical calibration is not achievable, but *useful* calibration is: you can prompt Claude to produce relative confidence estimates that distinguish high-certainty outputs from uncertain ones well enough to drive escalation decisions.
The key practitioner insight is that confidence calibration is not a property of the model — it is a property of the prompt-plus-model-plus-task combination. A well-calibrated confidence estimate on one task (entity extraction from structured text) may be poorly calibrated on another (legal clause interpretation). You must test calibration empirically per task type, not assume it from general model behavior.
Designing a Confidence Signal
Three design decisions determine whether a confidence score is actionable:
1. Scale and anchoring. A 1–5 ordinal scale is easier for models to apply consistently than a 0–1 float. Anchor each level in the prompt with a concrete behavioral description:
> 5 = I can cite specific evidence from the provided text. 4 = Strong inference from context. 3 = Plausible but some ambiguity. 2 = Multiple interpretations, guessing. 1 = No usable signal.
Undefined scales produce scores that mean different things across invocations — a 3 in one run may represent a 4 in another.
2. Separation of confidence from output. Require the model to output confidence as a structured field *before* the primary output, not after. Post-hoc confidence scores are subject to anchoring bias — the model has already committed to an answer and rationalizes a high score. Pre-output confidence forces genuine uncertainty acknowledgment.
3. Threshold-to-action mapping. Confidence without a routing table is decoration. Define explicitly:
| Confidence | Action |
|---|---|
| 5 | Auto-approve, no review |
| 4 | Auto-approve, log for audit |
| 3 | Flag for spot-check (10% sampled) |
| 2 | Route to human review queue |
| 1 | Escalate immediately, block downstream |
The Overconfidence Trap
The most common calibration failure in practice is systematic overconfidence: Claude returns 4s and 5s on tasks where empirical accuracy is 60–70%. This is not dishonesty — it reflects that LLMs are trained on confident-sounding text. Counter it with adversarial prompting: "Before scoring, actively try to identify reasons your answer might be wrong. If you find any, lower your score by at least one level." This simple instruction reliably reduces overconfidence in extraction tasks.
A second trap is treating confidence calibration as a replacement for human review on high-stakes decisions. Confidence calibration determines *routing* — it tells you which outputs need human review. It does not replace the review itself. The escalation pattern is what activates when confidence falls below threshold; human-in-the-loop is the workflow that receives the escalation. The concept guide Confidence Calibration & Review Thresholds covers threshold-setting methodology and how to back-test your routing table against historical data.
Frequently Asked Questions
Can you trust Claude's confidence scores without empirical testing?
No. LLMs tend toward overconfidence, and calibration varies significantly by task type, domain, and prompt phrasing. Any confidence-scoring system deployed in production should be back-tested against a labeled evaluation set to measure whether scores actually correlate with accuracy. Until that test is done, treat confidence scores as ordinal rankings (this output is more certain than that one) rather than probability estimates.
Should confidence be a single overall score or per-field?
Per-field confidence is almost always more useful for structured extraction tasks. An overall score of 3 tells you "something is uncertain" but not which field to scrutinize. Per-field scoring like `{"customer_name": 5, "contract_value": 2, "expiry_date": 4}` lets the coordinator escalate only the uncertain field while auto-approving high-confidence fields, reducing human review volume significantly.
How does confidence calibration interact with the escalation pattern?
Confidence calibration produces the signal; the [escalation pattern](/glossary/escalation-pattern) consumes it. When a confidence score falls below the defined threshold, the escalation pattern fires: the agent halts that decision branch, formats a structured escalation message with the specific field and its competing interpretations, and routes it to the human review queue. Neither pattern is complete without the other — calibration without escalation produces scores that are never acted on; escalation without calibration produces arbitrary handoffs.