Sampling, Temperature & Non-Determinism

Generation Samples From a Distribution

At each step of next-token generation, the model produces a probability distribution over possible next tokens and samples from it rather than deterministically picking a single "correct" answer. This is why the same prompt can produce different, equally valid, phrasings across calls.

Temperature Controls Randomness, Not Determinism

**Temperature** is the primary lever over that sampling process:

• **Lower Temperature** (toward 0) concentrates sampling on the most likely tokens -- more focused, more repeatable output.
• **Higher Temperature** spreads probability mass across more tokens -- more diverse, more creative output.

The key exam point: **Temperature 0 does not guarantee identical output across calls.** LLMs are inherently non-deterministic at the implementation level (floating-point/hardware effects, among other factors), so even the most focused sampling setting does not produce byte-identical output every time. Temperature changes the *degree* of randomness; it does not eliminate it.

Testing Implication

Because exact reproducibility is never guaranteed, testing and validating LLM-backed systems is done with evals -- criteria-based or model-graded assessments of whether output is good enough -- rather than exact-match/equality assertions that assume a single correct string.

Common exam traps

• "Temperature 0 makes Claude deterministic." This is false -- it reduces randomness but does not guarantee identical output.
• Assuming higher Temperature reduces token cost or latency -- Temperature affects the *sampling distribution*, not billing or speed.
• Writing tests that assert exact string equality against LLM output instead of using evals, which is the correct response to inherent non-determinism.