Why do LLMs give different answers each time, and how do you get deterministic output?
General LLMs sample from a probability distribution, so the same input can produce different output on different runs — a problem when you need the same document to extract the same values every time. Setting temperature to 0 reduces variation but doesn't fully guarantee it, and it doesn't fix accuracy. For deterministic tasks like extraction, you want a model designed for repeatable, structured output.
Last updated 2026-07-20
Why output varies
Token sampling plus non-determinism in how models are served means run-to-run drift, and it gets worse at higher temperature. For extraction, drift shows up as the same field coming back slightly different across runs.
What helps
Temperature 0 (greedy decoding), fixed schemas via structured output, and pinning a specific model version all reduce variation. They're necessary but not sufficient — they don't guarantee identical output across infrastructure, and they don't make a wrong answer right.
For extraction specifically
A model built for deterministic developer tasks produces more stable structured output, and returning confidence and location lets you detect the runs that drift instead of trusting them blindly.
FAQ
Does temperature 0 make an LLM deterministic?
It greatly reduces variation but isn't an absolute guarantee across serving infrastructure. Treat it as necessary, not sufficient.