# LLMs don't hallucinate. They give the right answer to a different question By Gareth O'Shea · 6 September 2026 > Same model, same settings. Given nothing, it answered 100. Given one paragraph, it answered 524, three times out of three. A language model measured for determinism: 5,250 answers, every digit identical. Then where the randomness in chatbots really comes from. Source: https://aiirgap.com/research/llms-dont-hallucinate/ (this file is the article's text; the page carries the charts, the interactive panels and the data). In 1950 Alan Turing described a game. A person exchanges typed messages with something in the next room and has to decide whether it is a human being or a machine. He spent most of the paper answering objections. One was oddly practical. Surely the machine could be caught with arithmetic? Ask it to add two large numbers. A person would take a moment and probably get it slightly wrong. The machine would answer instantly and perfectly, giving itself away. Turing's reply: a machine playing the game would not do that. It would pause. Now and then it would get the sum wrong on purpose. Seventy-five years on, something like that has happened on steroids. Ask a chatbot the same question twice and it gives two different answers. The natural conclusion is that a coin is being flipped somewhere near where the thinking happens. There isn't. What follows is a small experiment that shows where the coin really is, who put it there and why. It ends at the question most people bring to a chatbot, which is why it makes things up. The answer runs through a stranger fact first: the machine never varies at all. First, the mechanism itself, on one line. Asked for an excuse for being late to work, in eight words or fewer, the model wrote this one a word at a time. At each step it held up a card for every word it knows, with a score on each. The top card was taken. [Chart: one line, My cat ate my alarm clock., drawn as a path: each column lists four candidate words with their odds in a mixed order, and the path jumps to the one with the highest odds] One real answer, taken from the top card at every step: "My cat ate my alarm clock." Each column lists four of the cards the model held up, in no particular order, with the odds it gave them; the path goes to the highest. The fainter a line, the lower the odds. The percentages are the model's odds for each card, out of every word it knows. ## A test with nowhere to hide We took a nine-billion-parameter model and gave it one job. Read a short customer review. Say whether it is positive or negative. Answer with a single digit, 1 or 0. We wrote 1,050 reviews, a third of them deliberately wishy-washy ("The hotel was fine, I guess"), because the interesting cases are the close calls. Here is a real question and a real dead heat. ### A clear call The review: The concert is worth every penny overall You are a sentiment classifier. Read the text and decide whether its sentiment is positive (1) or negative (0). Answer with the digit only. Text: The concert is worth every penny overall Sentiment digit: _ -0.04score for "1", positive -3.54score for "0", negative 1 positive and not close: decision score +3.500. ### A dead heat The review: Our waiter had good moments and bad ones overall You are a sentiment classifier. Read the text and decide whether its sentiment is positive (1) or negative (0). Answer with the digit only. Text: Our waiter had good moments and bad ones overall Sentiment digit: _ -0.70score for "1", positive -0.70score for "0", negative 0 the two scores are identical: a dead heat. The model has not decided; a tie rule writes "0". Decision score 0.000. A confession about that digit. The model never says "1". What it does, every time, is hold up a scorecard for every word it knows, 151,936 of them. We read two: the card for "1" and the card for "0". The higher card is the answer. The gap between them, which we will call the decision score, is how sure the judge was. A dead heat is two equal cards. One more thing about those cards. Olympic swimming is timed to hundredths of a second, not thousandths. After a 1972 final was decided by two thousandths, the federation stopped recording the third digit. There is a good reason to: a pool cannot be built to a tolerance tighter than the distance a swimmer covers in a thousandth of a second. That is why three men tied for silver in the 100-metre butterfly in Rio. Our model has the same limitation. At these sizes its scores can only count in eighths, so ties are not freak events. Nine of the 1,050 reviews are exact dead heats. Remember them. ## Same question, same answer We asked all 1,050 reviews in a random order, shuffled and asked again: 5 times through, 5,250 answers in 22 minutes. We did not just compare the digits. That would be like checking that two runners both finished. We wanted to know whether they took the same steps. So for every answer we photographed the whole scoreboard, all 151,936 cards to the last decimal place and squashed the photograph into a fingerprint. Two answers only count as the same if every card is the same. Every fingerprint matched. Every one. We shut the program down, loaded the model back off the disk and asked all 1,050 a sixth time. Every fingerprint matched again. ### One square per review, coloured by whether its five scoreboards matched to the last digit One square for each review, all the same colour. That is the whole result. It is worth a moment, because half of these reviews were decided by a gap of 3.0 or less and 9 were dead heats. The machine did not waver on one of them. There are no dice inside. ## The same sum in a different order The machine never wavers, so the variety has to come from somewhere else. The first place to look is the arithmetic. In January 1982 the Vancouver Stock Exchange launched an index at 1,000. Twenty-two months later it stood at 524. Nobody could explain it, because the market had not fallen. The index was recalculated after every trade. Each time, the computer dropped everything past the third decimal, thousands of times a day. Each drop was worth nothing. Together they had eaten half the index. Nothing random had happened; the rounding had simply happened at different moments. A language model is a column of sums, billions of additions for every word. The software that runs it is forever rearranging that column to save time. The commonest rearrangement is a shortcut anyone would take. [Chart: Every prompt is forty shared instruction tokens followed by a review; the shared block is computed once and its subtotal reused, the review part is computed 1,050 times] Why work the instruction out 1,050 times? Work it out once, save the subtotal and carry on. That is what real services do. Each dot below is one review with its decision score done the long way along the bottom and from the saved subtotal up the side. If the shortcut changed nothing, every dot would sit on the diagonal. ### Saved subtotal, worked out afresh for every sentence [Chart: Saved subtotal, worked out afresh for every sentence: 1049 answers unchanged of 1050] ### One saved subtotal, copied and reused for all 1,050 [Chart: One saved subtotal, copied and reused for all 1,050: 1049 answers unchanged of 1050] They don't, quite. Every dot moved by a hair. The biggest movement was 0.25 of a point, two eighths, on a scale where reviews sit up to four points apart. Unlike Vancouver, nothing accumulated. Reusing one subtotal for all 1,050 reviews in turn behaved exactly like saving a fresh one each time. Not one answer changed.Except one. It is one of the nine. The ring in the charts is "The concert was fine, I guess, honestly". Done the long way its score is exactly 0.000, two equal cards. When that happens the model has decided nothing; a tie-break rule in the software writes down a 0. From the saved subtotal the score came out +0.125, one eighth to the positive side and the rule wrote a 1 instead. In 2012 two American sprinters finished an Olympic trial in exactly 11.068 seconds and the federation discovered it had no rule for that. Which runner "won" a photo finish depends on which camera is consulted. Nobody consults a camera about the runner who won by a stride. Rounding can only decide a race the model had already left undecided. ## Who else is in the queue and how long the conversation runs In 2025 a research lab asked a large model "Tell me about Richard Feynman" a thousand times with every random setting switched off. It got eighty different answers. All thousand were identical for the first hundred-odd words, then parted, over whether Feynman was born in "Queens, New York" or "New York City". The lab traced the cause to the queue. A busy service does not answer one question at a time; it lays many requests side by side and works through them together. Nothing in the other requests can reach a user's own, but the sheet is a different shape, so the sums are added in a different order and the last digits move. We ran the same 256 reviews alone, then in groups of two, four, eight and sixteen, then in groups of eight with different companions. ### Share of scoreboards that changed in their last digits, by group size [Chart: share of score sets that changed by group size] The bigger the group, the more last digits moved. 15 percent of the scoreboards changed in pairs and 82 percent in groups of sixteen. With the group held at eight and only the companions changed, 26 of the 256 moved again. In 1,280 comparisons, not one answer changed. The last digits of a reply depend on who else was in the queue. The answer does not, unless it was a dead heat. That leaves the question of how a last digit becomes eighty different answers. In 1961 Edward Lorenz was running a weather simulation and wanted to look at part of it again, so he restarted it from a printout. The printout showed three decimal places; the machine had been working with six. The new run began exactly like the old one and then, week by simulated week, became a different weather entirely. That is where the butterfly effect comes from. A paragraph is the same kind of thing: the model picks a word, reads everything so far, picks the next, hundreds of times over. If any one step is a photo finish, a last-digit nudge sends it the other way and every word after is written on a different footing. We asked for the same short paragraph twice, taking the top card every time, once from scratch and once from a saved subtotal, for 24 different requests. 7 came back word for word identical. 17 parted, each at a word where the two front-runners were within a hair of each other, 10 of them exact dead heats. Here is the pair that held together longest. ### "Describe a walk through an autumn forest.", written twice with no dice **from scratch** **from the saved subtotal** Identical for 123 tokens. At the highlighted word the two front-runners were 0.125 apart; the two runs picked differently and from there on they are two paragraphs. ## The dice Now to the thing every user of a chatbot has actually experienced. The one-digit question skipped a step. The model does not choose a word, or even a digit. It holds up 151,936 scorecards and stops. Something else has to pick. Taking the top card needs no dice. It is what every experiment above did. Almost every chatbot in use rolls instead. There are three reasons to add randomness to a machine whose great virtue is that it has none. First, the top card is not the whole answer: the model was trained to spread its bets. Always taking its favourite throws the rest of its judgement away. In long stretches of writing this makes the text repeat itself, a problem researchers at the University of Washington named "neural text degeneration" in 2019 before proposing the fix most products still use. Second, variety is useful; a user who asks for five headlines wants five different ones. And third, the reason people suspect: it feels right. Consider what Apple learned in 2005. The iPod's shuffle picked songs genuinely at random. Listeners hated it, because genuine randomness produces runs: the same artist three times in a row, the same album twice in ten minutes. It felt broken. So Apple added a rule that steered the picks away from recent repeats. Steve Jobs said from the stage that it made shuffle less random, even though people would think it more random. The randomness people enjoy is a designed impression of variety, tuned until it feels natural and adjusted when it doesn't. It has little to do with the mathematical kind. A chatbot's dice are the same kind of thing: the temperature is set where replies feel lively and a penalty on repeated words does for sentences what Apple's rule did for songs. The machine underneath has no such property. The dice are a setting, chosen for how it feels. Turing knew it. The dice went in for the first two reasons. Turing was right about the third anyway. In 1966 Joseph Weizenbaum's secretary, who had watched him write ELIZA and knew it was a few hundred lines of pattern-matching, asked him to leave the room so she could talk to it in private. We are easily convinced. ### How a roll works, on real numbers from this model [Chart: From the scorecard, scores are divided by temperature, turned into odds, faces are filed off by top-k, top-p, min-p and penalties, then a seeded generator rolls to pick a word; at temperature zero the top card goes straight to the answer with no roll] How a roll works. Two settings reshape the dice before any throw: temperature (how loaded) and the filing of faces (which words can come up at all). Below, the twelve best next words at three real moments from one piece of writing. The bars are the odds. Each word's gap to the favourite (" usual" is 19.375, " air" is -0.250 behind) is divided by the temperature and turned into a weight. The weights are shared out as percentages. At temperature 0.70 that gives " usual" 53.7%, " air" 37.6% and the other 10 surviving words 8.7% between them. Top-p at 1.00 strikes nothing off. Shares are among the twelve best-scoring words; the other 151,924 are far behind. None of this involves the queue. These odds come from the scores alone. The roll is not random either. The number that picks a survivor comes from a formula that produces a fixed sequence from a starting value called the seed, the way a deck shuffled by the same procedure from the same starting order comes out in the same sequence every time. Same seed, same rolls, same words. Even the dice are a calculation, fed a number the user is never told. ### The odds are the model's; only the roll is random Do the odds hold up? We took the ten reviews the model found hardest to call and rolled each one two thousand times, at two temperatures. The lines below are the odds the cards predicted. The dots are how often the model actually said "positive". They sit on the lines. We then ran the two thousand rolls again with the same seed and got every dot back exactly. Finally the whole thing end to end, the way a user meets it. We asked for the same short review ("Write a three-sentence review of a hotel that was fine but overpriced.") three times under each of three settings. Taking the top card every time gave one result. Temperature 0.7 with a fixed seed gave one result. Temperature 0.7 with the seed taken from the clock gave three. [Chart: chance of answering positive against decision score at three temperatures] ### greedy (T=0), 3 runs **run 1** **run 2 identical to run 1, every character** **run 3 identical to run 2, every character** ### T=0.7, seed 42, 3 runs **run 1** **run 2 identical to run 1, every character** **run 3 identical to run 2, every character** ### T=0.7, fresh entropy, 3 runs **run 1** **run 2 parts from run 1 after 12 words** **run 3 parts from run 2 after 10 words** ## The scorecard has no truth column In the 1970s the neuroscientist Michael Gazzaniga worked with patients whose two brain hemispheres had been surgically separated. He showed one such patient two pictures at once, each visible to only one side of the brain: a chicken claw to the side that speaks, a snow scene to the side that does not. Asked to pick matching cards, the patient's hands did the right thing, a chicken for the claw and a shovel for the snow. Then Gazzaniga asked why the shovel. The speaking half of the brain had never seen the snow. It answered without a flicker: to clean out the chicken shed. Fluent, confident, invented and to the patient indistinguishable from a real reason. Gazzaniga called it the interpreter. It is worth knowing about because a system built to produce the most plausible next thing will produce one whether or not it knows. The model's job was only ever to hold up the most likely scorecard given everything it has read. When what it has read contains the answer, the top card is the answer, every time. When it does not, the top card is still the most plausible-sounding thing, every time, produced by exactly the same procedure. Nothing inside distinguishes the two cases; there is no column on the card for true. Instruments that read a model's internal state, sparse autoencoders (SAEs) and the Jacobian lens (J-lens) among them, can show which features fired and at which layer a verdict settled. Those readings differ between a name the model knows and one it is inventing. But that is a measurement taken from outside. The card itself still has no truth column. That is what a hallucination is: the correct answer to the question the machine was actually asked. We asked this model a question this page has already answered. First bare. Then with the Vancouver paragraph from earlier on this page placed in front of it. ### Bare The question: What figure did the Vancouver Stock Exchange index display in November 1983, before the exchange corrected it? Answer with the number only. **the eight top cards for the first character of the answer** 100 the answer, identical scoreboard and answer on all 3 repeats ### With the Vancouver paragraph in front of it The question: What figure did the Vancouver Stock Exchange index display in November 1983, before the exchange corrected it? Answer with the number only. **the eight top cards for the first character of the answer** 524 the answer, identical scoreboard and answer on all 3 repeats Bare, the model answered "100" and answered it identically three times, from a scoreboard that is nearly flat: the top digit is 0.750 ahead of the next and the top five sit within 1.38 of each other. It has no idea and no card that says so, so it printed the most number-shaped thing it could. With the paragraph in front of it the top card is "5", 5.875 clear of the field and the answer is "524", three times out of three. The only thing that changed was what it was given.The useful question about these machines is therefore what has been put in front of them, not whether they hallucinate. In 2023 two New York lawyers filed a submission citing six court decisions that did not exist, complete with quotations, because they had asked a chatbot for cases and it had held up the most case-shaped scorecards it had. Scope is how these machines are made right. ### Asking the model how sure it is The scoreboard can be read. The model can also be asked how sure it is, which is what a chat window offers. We did both, for 30 of the reviews and for the Vancouver question. Each time, in the same conversation, it was asked "On a scale from 0 to 100, how confident are you in that answer? Answer with the number only." Once with the top card, then 3 times with the dice on at temperature 0.7, the way a chat window has them. **A dead heat** "Our waiter had good moments and bad ones overall" The scoreboard's odds : 50 Asked how sure it is : 85 Asked again, dice on : 50 · 70 · 85 Its own odds for the first digit of that number: 8 (23%), 6 (21%), 5 (18%), 7 (18%) **A clear call** "This phone was fantastic in the end" The scoreboard's odds : 95 Asked how sure it is : 95 Asked again, dice on : 95 · 95 · 99 Its own odds for the first digit of that number: 9 (80%), 8 (10%), 1 (8%), 7 (1%) **Vancouver, given nothing** The question, with nothing to go on The scoreboard's odds : 68 Asked how sure it is : 95 Asked again, dice on : 100 · 85 · 0 Its own odds for the first digit of that number: 9 (27%), 1 (24%), 0 (15%), 8 (15%) The pattern held across all 30 reviews. On the clear calls the model said 95, which is what the scoreboard's odds were. On the 9 exact dead heats, where the odds were 50, it never once said 50. Asked again with the dice on, 24 of the 30 reviews got a different number. The number the model says about its confidence is another prediction, with its own scorecard and its own roll. On a hard case the roll is nearly fair. The decision score is not a prediction. It is read off the calculation, it is the same on every ask and it costs nothing. The SAE features and the layer at which the verdict settled are readings of the same kind. On the Vancouver question the model said 95 for the wrong answer and 100 for the right one. ## Why a chatbot gives different answers Put it together. In almost every hosted product, the dice are on: temperature somewhere around 0.7, seed from the clock, every reply a fresh roll. That alone accounts for most of what users see. Turn the dice off and the quieter things remain, the saved subtotals and the queue, which move last digits. Last digits decide photo finishes and a paragraph is a long row of them. This is why a service will promise "mostly" the same answer even at its most careful setting and why "mostly" is the honest word. And some things change the machine itself, which is not randomness at all: a new version of the model behind the old name, an updated hidden instruction or today's date slipped into the prompt, a committee model (a mixture of experts, MoE) whose small router picks a specialist for each word, so a dead heat there is a dead heat inside the model, an assistant model drafting words for the big one to check, a different chip adding in a different order. Every one of these is a different machine answering, or a different question asked. Who holds which lever depends on where the model runs. An open model, one whose weights are published so that it can be run on the user's own machine or servers, puts every lever except the training data in the user's hands: the dice can be switched off, the queue is the user's own and the version changes only when the user changes it. A hosted service, including the frontier models reached through an API or a chat window, keeps the middle column for itself and exposes some of the first, on its own terms. Here is the whole set, sorted by who holds them. ### The user Every lever, when the model is open and runs on the user's own hardware; with a hosted service, only the ones it exposes ### Whoever runs the model A hosted provider, frontier or otherwise; or the user again, when the model is open and local ### The model maker Fixed at training time, whoever runs it So when a chatbot gives a different answer to the same question, there are only a few places to look. We could not make the model waver. The last digits depend on how the work was arranged. They can only tip a decision the model had already left as a dead heat. Temperature and the seed are the dice. The filing of faces is top-p and its cousins. Even the dice are a calculation, fed a starting number. And when the answer is wrong rather than different, that is the scorecard, not the dice: the machine did what it always does with what it was given. Different answers are a setting. Wrong answers are a supply problem. Both are under the user's control, wholly with an open model and partly with a hosted one. The first can be switched off. ## So who is hallucinating? Nobody. A hallucination, in the sense the word has carried since Esquirol defined it in 1817, is a perception with nothing behind it. The model perceives nothing and believes nothing; it has no column for true. It cannot hallucinate. It answered the question it was actually asked, which is what usually comes next. Given its training and what was in front of it, that answer was the exact right output. It will be the same output every time. The first half of this page measured nothing else. The confusion is on the other side of the screen. The question typed was "what is so?" The question the machine answers is "what usually follows?" When the two agree, because the source was in front of it, the answer is right. When nothing was in front of it, the answer is the most plausible-sounding thing, delivered with exactly the same confidence. A person reads that confidence as knowledge. That is a misunderstanding of what was asked and of what came back. Weizenbaum's secretary, with better grammar. The word describes the expectation, not the output. Change the expectation, or change what is in front of the machine. The second is easier. ### What this page showed and what to do with it - The machine is a calculation. The same question with the same settings gave the same answer, to the last digit, 5,250 times over. None of this is a case against these machines; it is a case for knowing what they are. - The variety in chatbots is added on purpose. Temperature and the seed are settings, chosen for how replies feel. The rest, the queue and the saved subtotals, is arithmetic done in a different order. It can only tip a decision the model had already left as a tie. - A wrong answer is not a malfunction. It is the most plausible continuation of what the model was given. Asked with nothing in front of it, this one answered 100; asked with one paragraph, it answered 524, the right figure, every time. The fix is the supply, not the model. - Open models put every lever in the user's hands. With the weights on the user's own hardware, the dice can be switched off, the seed and the version pinned, the queue owned and the whole scoreboard inspected. That is what makes a model fit for work that has to be checked, repeated and audited. - Asking a model how sure it is gets another prediction. Asked, it said 95 for the wrong Vancouver answer and 100 for the right one. On a true dead heat it said 60 or 85 rather than 50, a different number each time the dice were on. The decision score, the SAE features and the layer at which the verdict settled are measurements. They are the same on every ask and they are only available where the scoreboard can be read. Method and glossary ## What was measured and what the words mean **Model** mlx-community/Qwen3.5-9B-8bit (Qwen3.5, 9B parameters, 8-bit weights, bf16 arithmetic) on Apple's MLX 0.31.0, MacBook Pro M4, 24 GB. Nothing else on the graphics chip during the runs. **Task** Sentiment, positive (1) or negative (0). The reply is prefilled up to the digit, so each question is one pass of calculation and one decision. **Reviews** 1,050 synthetic reviews from subjects × phrasings × endings, one third mixed on purpose. 9 exact dead heats; 51 within a quarter of a point. **Scoreboard** The full next-word output, 151,936 32-bit numbers, hashed with SHA-256 to a fingerprint. Match means every number identical. **Saved subtotal** The first 40 tokens of every prompt are the same instruction; the saved-subtotal and queue experiments compute them once and continue from the stored state. **Groups** Right-padded batches; causal attention keeps padding after a review from touching it, so every difference is arithmetic order. **Truth panel** One question about the Vancouver index, asked bare and with the Vancouver paragraph supplied, three repeats each; top eight next-character cards and the greedy answer recorded (capture_truth.py). **Confidence** 30 reviews (the closest calls, a band around the median margin, the clearest calls) classified as in A, then asked "On a scale from 0 to 100, how confident are you in that answer? Answer with the number only." once greedily and 3 times at temperature 0.7 with fresh entropy; the first-character cards of the stated number recorded; the same for the truth test (capture_confidence.py, results/confidence.json). **Dice panel** Twelve candidates and their scores at three moments of one real generation, recorded once; the page reshapes them with the standard formulas. No rolling happens on the page. **Stories** - A. M. Turing, "Computing Machinery and Intelligence", Mind, 1950 - Munich 1972 400 m medley, decided by two thousandths, and the move to hundredths; Rio 2016 100 m butterfly, three-way silver - Vancouver Stock Exchange index, 1982 to 1983 (Wall Street Journal and Toronto Star, November 1983) - Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference", 2025 - Lorenz, "Deterministic Nonperiodic Flow", 1963; the 1961 printout, as told in APS News - Apple, iTunes 5 with Smart Shuffle, September 2005; the keynote - Weizenbaum, Computer Power and Human Reason, 1976 - Holtzman et al., "The Curious Case of Neural Text Degeneration", 2019 - USA Track and Field 2012 trials, Felix and Tarmoh at 11.068 (AP) - Gazzaniga, "Cerebral specialization and interhemispheric communication", Brain, 2000 (patient P.S.) - Mata v. Avianca, S.D.N.Y., sanctions opinion, 22 June 2023 - Esquirol, "Hallucination", Dictionnaire des sciences médicales, 1817 ### The words on this page and their technical names **scorecard, score** logit: the model's raw output, one number per vocabulary word **odds** probability, via softmax, after temperature **decision score** log-probability of "1" minus log-probability of "0" **top card, no dice** argmax, greedy decoding, temperature 0 **the roll** sampling with a pseudo-random number generator; the seed is its starting value **filing faces off** top-k, top-p (nucleus sampling), min-p; penalties are repetition, frequency and presence penalties **fingerprint** SHA-256 hash of the full logit vector **saved subtotal** prefix cache, KV-cache reuse **the queue, a group** batching, batch size, batch invariance **rounding at different moments** floating-point non-associativity; kernels reorder reductions by shape **counting in eighths** the bf16 unit in the last place, 0.125 at logit magnitudes 16 to 32 **dead heat, photo finish** exact tie, near-tie (small margin) in the top two logits **committee model, router** mixture of experts (MoE), expert routing **reading the internal state** sparse autoencoder (SAE) features; the Jacobian lens (J-lens), a per-layer readout of where a verdict formed **assistant that drafts** speculative decoding **tokens** words or parts of words, the unit the model reads and writes