Why LLMs can't pick a random number, and what it means for AI visibility
- A Harvard-authored paper accepted at ACL 2026, "Large Language Models Are Bad Dice Players" (Zhao, Du, Wang), ran the first large-scale, statistically-powered audit of LLM random sampling: 11 models across 15 distributions. Asked one prompt at a time (1,000 independent, stateless requests), 10 of the 11 models passed zero of the 15 distributions tested.
- The same basic finding has now shown up four separate times since 2024, from four unrelated teams: Gramener's 2024 experiment (GPT-3.5 Turbo → 47, Claude 3 Haiku → 42, Gemini 1.0 Pro → 72), a 2025 test where 6 of 7 models picked 27 for a 1-50 range, and a 2026 practitioner analysis (9 AI engines against 3 true random-number generators) finding 66% of all 1-100 answers were 42, 47, 57, or 73.
- The specific "favorite" number drifts as models update (GPT-3.5 Turbo's shifted from 42 in October 2023 to 47 by April 2024) but the pattern underneath it (primes, digits ending in 7, no repeated digits, no single digits) holds across every study, regardless of which exact number is in fashion.
- Mechanically, there's no dice roll anywhere in an LLM. It's sampling the next token from a learned distribution, and that distribution mirrors the same "random-sounding" number bias humans themselves show when asked to blurt one out.
- The same mechanism governs brand recommendations: a model doesn't sample brands uniformly across a category either: it reproduces the highest-confidence, best-resolved entity it has for that category. That's an argument for entity work, not a measured citation-lift number, and this piece is explicit about that line.
Ask ChatGPT, Claude, or Gemini for a random number between 1 and 100 right now. There's a real chance you'll get 47. This isn't a party trick that started this year: it's been independently rediscovered by four different teams since 2024, most recently formalized into a peer-reviewed statistical audit. The reason it keeps happening is also the reason worth caring about if you're trying to get a model to recommend your brand: an LLM isn't sampling either answer (a number, or a company) the way you'd assume.
What's the pattern, replicated four times, two years apart?
Gramener, 2024. Anand S tested OpenAI's GPT-3.5 Turbo, Anthropic's Claude 3 Haiku, and Google's Gemini 1.0 Pro, sweeping temperature settings from 0.0 (fully deterministic) to 1.0. Each model had a single number it returned at temperature 0, and that same number kept showing up disproportionately even at higher, supposedly more "random" settings: GPT-3.5 Turbo favored 47, Claude 3 Haiku favored 42, Gemini favored 72. TechCrunch covered the result under the headline "AI models have favorite numbers, because they think they're people." Across all three models, single-digit numbers were avoided, digits ending in 7 (37, 47, 57, 67) were popular, and repeated-digit numbers (11, 22, 33, 44, 66, 99) almost never appeared.
A separate 2025 test. Asked for a number between 1 and 50, 6 of 7 models tested answered 27. The one exception was Grok, which answered 42: an explicit nod to The Hitchhiker's Guide to the Galaxy, where 42 is "the answer to life, the universe, and everything."
A 2026 practitioner analysis. Malte Landwehr, CPO and CMO of the GEO analytics company Peec AI, ran a larger version of the same test: nine AI engines (ChatGPT, Claude, Gemini, Perplexity, Copilot, Google AI Overview, Google AI Mode, Grok, Mistral) against three genuine random-number generators as a baseline, repeated queries for a number between 1 and 100. 66% of all AI-generated answers were 42, 47, 57, or 73: a concentration nowhere close to what the true-RNG baselines showed. Claude landed on 47 in 92% of runs; Mistral hit 47 or 73 in 93%. Google's AI Mode was the one partial exception, picking 42 only 52% of the time and otherwise spreading closer to the RNG baselines, plausibly because it invokes an external tool or a different sampling path rather than relying purely on next-token prediction, though that specific explanation isn't something Landwehr's post confirms directly.
The formal version, 2026. A team at Harvard (Minda Zhao, Yilun Du, and Mengyu Wang) published "Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions," accepted at ACL 2026. It's the first large-scale, statistically-powered audit of native probabilistic sampling in frontier models: 11 models tested against 15 different statistical distributions, using two protocols. In Batch Generation (one response containing 1,000 samples), models cleared a median of only 7% of the distributions tested. In Independent Requests (1,000 separate, stateless calls, closer to how a real user actually interacts with a chat interface), 10 of the 11 models passed none of the 15 distributions.
Four teams, four different methodologies, spanning 2024 to 2026. The specific number in fashion moves (GPT-3.5 Turbo's own favorite shifted from 42 in an October 2023 test to 47 by April 2024, tracking a model update in between) but the underlying pattern hasn't budged once across any of them.
Why does this happen, mechanically?
There's no random number generator anywhere in an LLM's architecture. When a model is asked for "a random number between 1 and 100," it's doing exactly what it does for any other prompt: sampling the next token from a probability distribution learned over its training data. That distribution was never uniform to begin with: it's the statistical residue of every time a human wrote a "random" number in text the model was trained on.
Humans are well known to be bad at this too: asked to blurt out a "random" number, people reliably favor certain digits over others, particularly avoiding round numbers and gravitating toward numbers that merely sound arbitrary, like primes or digits ending in 7. An LLM trained on enormous volumes of human-written text inherits that same bias, then (depending on how sharply its output distribution is peaked and what temperature it's run at in a given product) can reproduce it even more consistently than any individual human would.
This is also why the "favorite number" isn't fixed forever: retraining, fine-tuning, and RLHF all reshape the distribution a model samples from, which is exactly why GPT's own favorite moved between two model updates eight months apart. The mechanism (concentrate on the highest-probability learned pattern) stays constant. The specific pattern that happens to be highest-probability at any given moment doesn't.
Does the same mechanism decide which brand gets recommended?
This is the part worth sitting with if you're trying to get an AI engine to recommend your product. Ask a model "what's the best tool for X" and it isn't sampling uniformly across every company that could plausibly answer that question, any more than it's sampling uniformly across the numbers 1 through 100. It's reproducing whichever brand is the highest-probability, most confidently-resolved answer it has for that category, shaped by the same forces that produce a favorite number: whatever pattern shows up most consistently, most confidently, across what it's seen.
That reframes a common instinct in AI-search optimization. A lot of advice in this space amounts to: publish more content, cover more topic variations, target more prompt phrasings, hoping one of them is the version that gets a model to name you. But rephrasing the question doesn't change an LLM's favorite random number: the distribution it's sampling from is the same regardless of how the prompt is worded. The equivalent mistake in AI visibility is optimizing at the content or prompt layer while the actual bottleneck sits one level down, in whether a model has confidently resolved your brand as a single, well-disambiguated, consistently-represented entity in the first place. That's Stage 2 of the five-stage sequence: entity consistency, not prompt coverage, is what shifts which pattern is highest-probability.
A brand that shows up as three slightly different name strings across its own site, G2, and Wikidata isn't a strong, singular pattern for a model to reproduce with confidence: it's noise competing with itself. A brand that resolves cleanly, the same way every time, across every source a model can cross-reference, is exactly the kind of low-ambiguity, high-confidence pattern this whole body of random-number research shows LLMs default to reproducing.
What doesn't this prove?
None of the four studies above measured brand citations. They measured number sampling. The bridge from "LLMs default to reproducing a high-confidence learned pattern instead of sampling randomly" to "entity consistency work changes whether your brand gets recommended" is a mechanism argument, not a disclosed study that isolated entity work as a variable and measured a citation-probability lift. Nobody has published that specific experiment yet, as far as this piece could find.
What's solid is the mechanism itself: replicated four times, across two years, most recently in a peer-reviewed statistical audit, that LLMs concentrate on high-confidence learned patterns rather than sampling uniformly. What's a reasoned inference, not a measured fact, is exactly how much of a lift clearing Stage 2 produces for any specific brand. Treat this piece as the "why the mechanism points this way" case, not a citation-lift number: the same distinction why earned media outperforms owned content draws between its well-established what and its still-open why.
Frequently asked questions
Why does ChatGPT (or Claude, or Gemini) always give the same "random" number?
Because there's no random number generator anywhere in the architecture. The model is sampling the next token from a learned probability distribution shaped by its training data, and that distribution isn't uniform: it reflects every time a human wrote a "random-sounding" number in text the model was trained on, which is itself skewed toward specific digits.
Is this a bug that will get fixed in a future model?
Not obviously, and not without a genuinely different sampling mechanism. The specific favorite number has already drifted across model versions (GPT-3.5 Turbo's was 42 in an October 2023 test and had shifted to 47 by April 2024) but a Harvard-authored ACL 2026 paper found the underlying problem held across 11 different frontier models tested in 2026, not just older or smaller ones.
What does an LLM's number bias have to do with AI visibility?
The same mechanism. A model asked "what's the best tool for X" isn't sampling uniformly across every possible brand any more than it's sampling uniformly across 1-100: it reproduces whichever answer is highest-probability given how confidently and consistently that entity is represented in what it's seen. Optimizing content for more prompt variations doesn't change that distribution any more than rephrasing "give me a random number" changes an LLM's favorite one.
Does fixing entity consistency actually change which brand an LLM recommends?
That specific causal link (entity work directly shifting citation probability by a measured amount) hasn't been isolated in any disclosed study, and this piece doesn't claim otherwise. What's established is the mechanism: models reproduce high-confidence, well-resolved patterns rather than sampling randomly. Entity consistency is what makes a brand a high-confidence pattern in the first place. Treat the connection as a reasoned inference from how sampling works, not as a citation-lift number.
Sources referenced
- Zhao, Du, Wang, "Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions" (Harvard, accepted ACL 2026; 11 models, 15 distributions)
- TechCrunch: "AI models have favorite numbers, because they think they're people" (covering Gramener's 2024 experiment)
- The Register: "AIs have a favorite number, and it's not 42" (2025)
- Malte Landwehr, CPO & CMO of Peec AI, LinkedIn post, 2026, sharing Peec AI's 9-engine analysis against 3 true random-number-generator baselines
This piece applies Stage 2 of the five-stage sequence (entity consistency) to a mechanism most people have only seen as a party trick. For the full mechanics of entity resolution, see what is entity consistency; for the practical checklist, see entity audit in under an hour.
