Evaluator-Optimizer Pattern

Patterns

Definition

A two-pass architecture where a generator produces output and a separate evaluator assesses it against explicit criteria. For true quality assurance, the evaluator must be a separate Claude instance with independent context — using the same instance creates confirmation bias.

Example Usage

Use one Claude call to generate a pull request review, then a second independent call to evaluate the review's completeness and accuracy.

Why It Matters for the CCA-F Exam

This pattern appears in Domain 4 (Quality Assurance and Verification) of the CCA-F exam. Key exam questions distinguish a same-instance self-critique (which the exam treats as weak) from a separate-instance evaluator (which the exam treats as the architecturally correct solution). Understand why separate context windows matter and when the pattern's latency cost is justified.

In Depth

The Core Problem: Self-Evaluation Bias

When you ask a model to review its own output, it tends to find it acceptable. The model's generation and evaluation share the same context, the same implicit assumptions, and the same blind spots. This is not a failure of intelligence — it mirrors how humans struggle to proofread their own writing.

The evaluator-optimizer pattern solves this by architectural separation: a generator call produces output, and a distinct evaluator call — running in an independent context window — assesses it against explicit criteria. Because the evaluator has no memory of the reasoning that produced the output, it can spot gaps the generator glossed over.

Anatomy of the Pattern

Generator call
  └─ produces: draft output

Evaluator call  (separate instance, no shared context)
  inputs: draft output + explicit evaluation rubric
  └─ produces: pass/fail + structured feedback

Optimizer loop
  └─ if fail: inject feedback into generator, regenerate
  └─ if pass (or max iterations reached): return output

The rubric passed to the evaluator is the most important design choice. Vague rubrics like "is this good?" reproduce the same bias. Concrete rubrics list exact criteria: required sections present, all cited sources real, no contradictions with supplied facts, tone matches persona.

When to Use It

This pattern adds latency and cost (at least two Claude calls per result). Reserve it for:

High-stakes documents: contracts, medical summaries, compliance reports.
Code that will be deployed automatically: the evaluator acts as a programmatic code review gate.
RAG outputs where hallucination risk is high: evaluator checks every factual claim against retrieved sources.

Connection to Related Patterns

The evaluator-optimizer is a specific instantiation of the broader validation loop concept. Where a validation loop typically checks structure or schema, the evaluator-optimizer checks semantic quality. It also relies on the same multi-instance principle described in the Multi-Pass Review Architecture concept page.

For the evaluator to be truly independent, it must be a separate API call — not a second message in the same conversation thread. Conversation threads share context and the model can still access its previous reasoning implicitly.

How It Compares

Pattern	Instances	Context shared?	Catches own blind spots?
Self-critique (same call)	1	Yes — full	No
Self-Critique (follow-up turn)	1	Yes — full	No
Evaluator-Optimizer	2+	No	Yes
Validation Loop	2+	No	Partial (structural only)

Example

Evaluator-optimizer with a fresh client call for the evaluator (no shared context) and a bounded retry loop.

python

import anthropic

client = anthropic.Anthropic()

def generate(prompt: str) -> str:
    resp = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )
    return resp.content[0].text

def evaluate(draft: str, rubric: str) -> dict:
    """Independent evaluator — fresh client call, no shared context."""
    resp = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=512,
        messages=[{
            "role": "user",
            "content": f"Evaluate this draft against the rubric.\n\nRUBRIC:\n{rubric}\n\nDRAFT:\n{draft}\n\nReturn JSON: {{\"pass\": bool, \"feedback\": str}}"
        }]
    )
    import json
    return json.loads(resp.content[0].text)

# Optimizer loop
rubric = "1. All claims cited. 2. No contradictions. 3. Conclusion follows from evidence."
draft = generate("Write a summary of quantum entanglement for a non-specialist.")

for _ in range(3):  # max 3 iterations
    result = evaluate(draft, rubric)
    if result["pass"]:
        break
    draft = generate(f"Revise this draft based on feedback.\n\nFEEDBACK: {result['feedback']}\n\nPREVIOUS DRAFT:\n{draft}")

print(draft)

Frequently Asked Questions

Why can't I just add a second message in the same conversation asking the model to critique its answer?

A follow-up message in the same thread still runs in the same context window — the model can see its own prior reasoning and is implicitly anchored to it. Studies on LLM self-evaluation consistently show models rate their own outputs higher than independent reviewers do. A genuinely separate API call with no conversation history is the minimum bar for unbiased evaluation.

Should the evaluator use the same model as the generator, or a more capable one?

For most use cases, use the same or a slightly more capable model. The evaluator's job is harder than it looks — it must apply a nuanced rubric rather than just generate fluent text — so under-powering it (e.g., always using Haiku) risks approving flawed output. For high-stakes pipelines, using Claude Opus 4.8 as the evaluator and Sonnet 4.6 as the generator is a common pattern that balances cost against rigour.

Referenced in These Lessons

Self-Critique Limitations & Independent Review

Prompt Engineering

Self-review in the same context suffers from confirmation bias -- the model retains generation reasoning

Multi-Pass Review Architecture