5.2 Sampling, Non-Determinism, Thinking Modes, and Prompting Fundamentals

At each step of the autoregressive loop from Lesson 5.1, the model doesn't deterministically select one "correct" next token. It produces a probability distribution over the vocabulary and samples from it. That single design choice explains a lot of what looks surprising about LLM behavior at first: ask the same question twice and you can get two different, equally reasonable phrasings, because two different tokens were sampled at some early step and everything downstream diverged from there.

Temperature is the main lever over this sampling process. Lower temperature (toward 0) concentrates probability mass on the most likely tokens, producing more focused, more repeatable output. Higher temperature spreads probability across more of the distribution, producing more diverse and more creative — but less predictable — output. Temperature changes the shape of the sampling distribution; it is not a cost or latency knob, and it does not set the context-window size.

This is the single most commonly tested misconception in this domain: "temperature 0 makes Claude deterministic." It is false. Temperature 0 concentrates sampling on the single most likely token at each step, which does make output far more repeatable than at higher temperatures — but LLMs remain inherently non-deterministic at the implementation level (floating-point and hardware effects, among other factors), so even at temperature 0 you are not guaranteed byte-identical output across calls.

The practical consequence reaches all the way into how you test LLM-backed systems. If you can't rely on exact reproducibility, you cannot validate correctness with an equality assertion that expects one specific string. Instead, LLM-backed systems are tested with evals — criteria-based or model-graded assessments of whether an output is good enough — which tolerate legitimate variation in phrasing while still catching real failures.

Beyond a plain call, Claude offers modes that spend more compute before answering in exchange for better quality on hard problems. Extended thinking is an explicit reasoning budget the model works through before producing its final answer. Adaptive thinking, or effort levels, is a related but distinct idea: rather than a fixed budget you set explicitly, the model itself varies how much reasoning it spends depending on how difficult it judges the task to be. At the opposite end of the spectrum, fast modes are tuned for latency-sensitive work where minimal reasoning overhead is preferred.

None of this is free. Thinking tokens are billed as output tokens, and the extra reasoning time adds latency. The tradeoff genuinely pays off on hard, multi-step reasoning or ambiguous judgment calls — but applying a thinking mode to a task that doesn't need the extra depth (simple classification, straightforward extraction) pays both costs for no quality benefit. And support for a given thinking mode isn't uniform across model tiers, so choosing a tier for a workflow step has to account for whether that tier actually supports the thinking mode the step needs.

The most fundamental prompting spectrum is how many examples you give the model before asking it to perform the task. Zero-shot gives instructions only, with no worked examples. One-shot (single-shot) gives exactly one example. Multi-shot (few-shot) gives several. Moving along that spectrum from zero-shot toward few-shot steers the model's output format and behavior more precisely — examples are a powerful way to communicate a pattern that's hard to describe purely in words.

That steering isn't free, though: every example you add is more input tokens, on every single call that uses this prompt. A few-shot prompt with five detailed examples costs meaningfully more per request than the zero-shot version of the same instruction, forever, for as long as that prompt is in production. Choosing a point on this spectrum is a real tradeoff between how much steering the task needs and how much extra you're willing to pay per call to get it — not a decision to reflexively maximize examples "to be safe." (Domain 6 goes deeper on prompting technique; the fundamentals here are what this domain expects you to know.)

• •Zero-shot — instruction only, no examples; cheapest per call, least steering.
• •One-shot / single-shot — exactly one worked example; light steering at a small added token cost.
• •Multi-shot / few-shot — several worked examples; strongest steering, but the added input tokens repeat on every call.

Task Statement 5.2 clusters four ideas — sampling, thinking modes, and example-based prompting — that share one theme: each has a real cost or limitation that's easy to gloss over if you only remember the upside.

• •Treating temperature 0 as deterministic. ✗ Any answer claiming temperature 0 guarantees identical output. ✓ The answer stating lower temperature reduces randomness but never guarantees identical results.
• •Testing LLM output with exact-match assertions. ✗ An answer that validates output with equality checks against one expected string. ✓ The answer that validates with evals tolerant of legitimate variation.
• •Treating thinking modes as free or always beneficial. ✗ An answer enabling extended thinking uniformly, or describing it as costless. ✓ The answer reserving thinking modes for tasks whose difficulty justifies the added tokens and latency.
• •Maximizing examples "to be safe." ✗ An answer defaulting to few-shot regardless of task need. ✓ The answer matching the number of examples to how much steering the task actually needs, given the added token cost.