Code Execution: Computing Numbers Instead of Writing Them

Compute It, Don't Write It

When a number has to be right, have Claude compute it rather than write it. Left to prose generation, Claude produces a plausible-looking figure -- "Membership renewals brought in roughly $2.3 million last quarter" reads confidently, but it is a guess dressed up as an answer, with no way to trace where it came from. Code execution is a different mechanism entirely: Claude writes and runs code that actually performs the calculation over the real data and returns a computed, checkable result -- "$2,286,910" -- along with the option to produce supporting charts or processed files. The second number is traceable back to the rows that produced it; the first is not traceable to anything.

Worked Example 1: A Sum

Asked "what did membership renewals bring in last quarter," a prose answer might land on "around $2.3 million" -- a figure that sounds like a reasonable rounded estimate and is nearly impossible to distinguish from a genuinely calculated one just by reading it. A code-execution answer instead runs df[df.transaction_type == "renewal"]["amount"].sum() against the actual uploaded ledger and returns $2,286,910.47, plus (if asked) the row count it summed over. If that figure looks off, you can ask to see the filter, confirm it excluded the right rows, and rerun it -- none of which is possible with the prose version, because there is no computation behind it to inspect.

Worked Example 2: A Statistical Claim, Not Just a Sum

Code execution is not only for totals. Suppose the question is "is there a meaningful difference in average deal size between the East and West sales regions this quarter?" A prose answer might assert: "East deals tend to run somewhat larger, though the difference may not be significant." That sentence *sounds* appropriately hedged and statistically literate, but it is still a guess -- there is no actual mean, sample size, or test statistic behind "somewhat larger" or "may not be significant." A code-execution answer instead computes both group means, runs an actual t-test (or reports the effect size and sample sizes), and returns something like: "East mean deal size: $58,200 (n=142). West mean: $51,900 (n=118). Difference is statistically significant at p=0.03." That is a claim you can check by asking for the underlying group counts, or by asking Claude to rerun the test with a different grouping to see if the result holds up -- exactly the kind of claim a confident-sounding prose hedge cannot support, because "may not be significant" is not a number, it's a vibe.

What the Guarantee Actually Covers

The reliability guarantee covers the *act of running the computation*, not the judgment behind how it was set up. Claude is still the one writing the code, so a wrong filter, an off-by-one error, a misapplied formula, or a test run against the wrong column can slip in undetected. What you actually gain from code execution isn't an automatic guarantee of correctness -- it's a calculation exposed for you to inspect, verify, and rerun on demand, something a prose-generated figure never offers.

When Code Execution Is NOT the Right Tool

Code execution's power is narrow and specific: it verifies computations over data that already exists in a machine-readable form. It has clear boundaries, and recognizing them is as important as recognizing its use case.

Subjective judgments -- tone, persuasiveness, ethical framing -- have no numeric ground truth for code to compute against, so there's nothing for a script to verify. Qualitative synthesis -- pulling common themes out of open-ended interview notes -- involves judgment calls about what counts as "the same theme" that code execution doesn't make; that work still benefits from the quote-grounding technique in the previous lesson (trace each theme back to a quote), but it is not a computation problem, and reaching for code execution here just produces a script that mechanically pattern-matches keywords while missing the actual synthesis. The test is simple: if the claim has a single correct numeric answer given the data, code execution is the right tool; if the claim is a judgment call about meaning, quality, or appropriateness, it isn't, and a different validation technique (quote-grounding, best-of-N, human review) fits better.

A Fourth Thing, Not a Fourth Surface

This lesson already covers editing, adapting, refining, and comparing an output, and the next lesson covers choosing among inline, Artifact, and structured-data surfaces. Code execution is neither -- it doesn't govern how a number is *presented*, it governs whether the number is *genuinely verified* in the first place. A computed total can still be delivered inline, as an Artifact, or as structured data; code execution is what happened *before* that presentation choice, not an alternative to it.

Common exam traps

• Treating a confident-sounding prose figure ("about $2.3 million") as equivalent to a computed one. Prose generation produces a plausible guess; only code execution produces a number traceable to the data that produced it.
• Assuming code execution makes a result automatically correct because it's "deterministic." Determinism covers the execution, not the logic -- Claude-written code can still contain a bug, so the result should be checked, not blindly trusted.
• Confusing code execution with an output surface (inline/Artifact/structured data). It's a computation-reliability mechanism, not a presentation choice, and the two operate independently.
• Reaching for code execution on a subjective or qualitative question (tone, persuasiveness, thematic synthesis) where there's no numeric ground truth for a script to verify against.