Quote-Grounding and Best-of-N Comparison

Two More Techniques Beyond the Basics

Lesson 2.2 already covered grounding in source material, allowing an "I don't know" exit, requesting traceable citations, and cross-checking against an authoritative source. Two further techniques round out the validation toolkit, and both work by making Claude's reasoning -- not just its conclusion -- visible for you to check.

Quote First, Then Analyze

On a lengthy document, have Claude pull out the specific passages that support its answer *before* it commits to any conclusion. Anchoring the analysis to pulled quotes puts both the logic and any mistakes in plain sight: rather than taking a summary paragraph on faith, you can trace each claim to an exact line in the source and check whether the quote actually supports the interpretation Claude drew from it. A conclusion with no attached quote is a conclusion you have to take on faith; a conclusion built on a quote is a conclusion you can check in ten seconds.

Worked Example: Ungrounded vs. Grounded Analysis, Side by Side

Suppose a 30-page vendor contract is uploaded and the request is "flag any clause that lets the vendor unilaterally raise prices."

Ungrounded version: "Section 7 allows the vendor to increase pricing at their discretion with limited notice to the customer, which creates meaningful cost-exposure risk." This reads like a real finding. But there's nothing to check it against -- is there actually a Section 7, does it actually say that, and what does "limited notice" actually mean in days? You'd have to reopen the whole contract and search for it yourself, which defeats the point of asking Claude to find it.

Grounded version: "Section 7.2 states: 'Vendor may adjust the Fees upon thirty (30) days' written notice to Customer, at Vendor's sole discretion.' This is a unilateral price-increase clause -- it requires only notice, not customer consent, and the window is 30 days." Now the claim is falsifiable in ten seconds: open the contract to 7.2, confirm the quoted sentence exists verbatim, and confirm the interpretation ("sole discretion," "30 days") matches the quote. If the quote were fabricated or the section number wrong, that would be immediately visible instead of buried inside a confident paraphrase.

The difference isn't that the grounded version is better written -- it's that the grounded version is *checkable* and the ungrounded version isn't, even though both could be describing the exact same real clause.

Best-of-N Comparison

Send the identical request through several separate runs and line up the results. When independent attempts land on the same answer, that convergence is a real, if imperfect, reason to trust it more. When they disagree, that disagreement has told you something a single response never could: this is a question the model doesn't reliably have a fixed answer to, and it's exactly the kind of soft spot a human should look at directly.

Concretely: ask the same contract-review question three times in three fresh conversations. If all three flag Section 7.2 with the same 30-day figure, that convergence is meaningful evidence. If one run says 30 days, another says 60, and a third doesn't mention a notice period at all, you've just learned that the notice period is exactly the detail to open the contract and verify by hand -- something a single run, however confident it sounded, would never have told you.

Why These Belong Alongside Grounding and Cross-Checking

Both techniques share the same underlying logic as the rest of this lesson: confidence and fluency are not evidence, so validation has to manufacture some external signal to check against. Quote-grounding manufactures that signal by forcing traceability into the output itself. Best-of-N manufactures it by using disagreement across repeated runs as a proxy for uncertainty the model itself won't report. Combined, they're complementary rather than redundant: quote-grounding checks whether a single answer is traceable; best-of-N checks whether that answer is stable. A claim can pass one check and fail the other -- a well-quoted answer can still turn out to be an unstable one once you run it three times, and a stable answer that converges every time can still lack a checkable quote underneath it.

Common exam traps

• Treating a single well-written analysis of a long document as sufficient because it "sounds thorough." Without pulled quotes underneath it, there's nothing to trace the claims back to.
• Assuming one run of a request is as good as three. Agreement across independent best-of-N runs is a confidence signal; a single run gives you no way to detect the soft spots that divergence would reveal.
• Assuming quote-grounding and best-of-N are interchangeable. Quote-grounding tests traceability of a single answer; best-of-N tests stability across repeated answers -- a scenario can call for one, the other, or both.