Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks, but their ability to generate truly random sequences remains widely unexplored. This study presents a comprehensive statistical framework for evaluating pseudo-randomness in LLM outputs, using statistical tests like chi-square frequency analysis, runs tests, serial correlation analysis, Shannon entropy calculations, and Kolmogorov complexity approximations through compression ratios.
​
We conducted three systematic experiments using GPT-4o-mini: (1) temperature parameter variation (0.3 vs 0.7), (2) prompt engineering strategies comparing control, pattern-avoidance, and quantum-inspired prompts, and (3) iterative feedback loops using LLM outputs as seeds for subsequent generations. Our analysis of 18,900+ digit sequences across 63 trials per condition reveals that while LLMs achieve near-perfect digit distribution, they exhibit systematic failures in consecutive value patterns.
​
We also discovered that feedback loops show dramatic instability with compression ratio volatility, indicating the system alternates between generating highly structured sequences and relatively random ones. These findings reveal unpredictable dynamics in neural network behavior with critical implications for cryptographic applications and iterative AI systems.

Introduction

The Randomness Problem in AI

The generation of truly random sequences represents one of the most fundamental challenges in computational systems. As Stephen Wolfram demonstrates in “A New Kind of Science,” even humans struggle to produce genuinely random outputs, often exhibiting predictable patterns and cognitive biases when asked to generate “random” sequences. This raises a profound question: if human intelligence, the very foundation upon which large language models are trained, cannot achieve true randomness, what happens when we ask artificial intelligence systems to generate random outputs?
​
Large language models, despite their remarkable capabilities in language generation, reasoning, and creative tasks, remain fundamentally deterministic systems. Trained on vast corpora of human-generated text, these models learn to predict and generate sequences based on patterns embedded in their training data. When explicitly asked to produce “random” outputs, do they transcend their deterministic nature, or do they inherit and amplify the same biases present in human-generated randomness?

Research Question

The study addresses three core questions:
1
.
How does temperature affect the statistical quality of LLM-generated random sequences?
2
.
Can different prompting approaches improve the randomness quality of LLM outputs?
3
.
What happens when we use LLM-generated “random” sequences as seeds for other LLMs’ generation?

Methodology and Implementation

API Integration

The framework includes functions for enabling the use of GPT-4o-mini via OpenRouter(Click to view code)


Digit Generation

The framework contains sophisticated digit extraction and generation functions designed for large-scale experimental data collection. The system includes comprehensive retry logic, progress monitoring, and quality validation to ensure reliable data collection across thousands of API calls.(Click to view code)


Statistical Testing Framework

We used a range of established statistical tests, each targeting different aspects of randomness. This multi-dimensional approach ensured that subtle deviations from true randomness were detected across various statistical properties.

Frequency Distribution Analysis (Chi-Square Frequency Test)

This evaluates whether digits 0-9 appear with equal probability (10% each). Deviations suggest systematic bias in digit generation, indicating compromised randomness at the most basic level. (Ideal randomness p value = 0.5)(Click to view code)


Sequential Pattern Analysis (Runs Test for Consecutive Values)

The runs test examines patterns in consecutive identical or similar values by converting the sequence to binary (≥5 = 1, <5 = 0) and analyzing the frequency of alternations. Too few or too many runs indicate systematic structure in the sequence. (Ideal randomness p-value = 0.5)(Click to view code)


Independence Analysis (Serial Test for Digit Dependencies)

The serial test examines whether consecutive digit pairs occur with expected frequencies, detecting subtle dependencies between adjacent elements that would compromise randomness. (Ideal randomness p-value = 0.5)(Click to view code)


Information-Theoretic Measures (Entropy and Compression)

Shannon entropy quantifies the information content of the sequence, and compression ratio approximates Kolmogorov complexity (sequences with hidden patterns compress better than truly random ones). (Ideal entropy = 3.32 bits, and ideal compression ratio = 0.55-0.65)(Click to view code)


Analysis Wrapper

All test results are collected in an association for easy aggregation and analysis.(Click to view code)


Experimental Design

Our experimental framework consists of three approaches designed to examine different aspects of LLM randomness generation.

Temperature Parameter Study

LLM temperature controls the stochasticity of LLM outputs. Lower values produce more deterministic responses, while higher values increase randomness. We compare performance at temperature 0.3 (conservative) vs 0.7 (moderate) to determine whether sampling strategy affects randomness quality.
(Click to view code)
RunTemperatureExperiments[numTrials_Integer:10,sequenceLength_Integer:100]:=Module[{results={},digitSequence,analysis},Do[Do[Print["Temperature ",temp,", Trial ",trial," of ",numTrials];digitSequence=GenerateDigits["Generate "<>ToString[sequenceLength]<>" random digits",temp,sequenceLength];If[Length[digitSequence]≥sequenceLength0.8`,analysis=AnalyzeSequence[digitSequence];If[analysis=!=$Failed,AppendTo[results,Association["Type""Temperature","Condition"temp,"Trial"trial,"Sequence"digitSequence,"Results"analysis]];];];Pause[1];,{trial,numTrials}],{temp,{0.3`,0.7`}}];Export["Temperature_Results.json",results,"JSON"];results];

Prompt Engineering

We investigate whether explicit instructions can improve randomness quality by comparing three prompting strategies: control (simple request), pattern-avoidance (explicit anti-pattern instructions), and quantum-inspired (physics-motivated randomness concepts).(Click to view code)


Feedback Loop Analysis

We investigate what happens when LLM outputs become inputs for subsequent generations. This feedback loop experiment reveals whether systems converge to non-random outputs, or maintain randomness across iterations.(Click to view code)


Power Analysis and Execution

We conduct power analysis to determine the sample size needed for detecting medium effect sizes with adequate statistical power.(Click to view code)


Results and Statistical Analysis

Our comprehensive analysis of 18,900+ digit sequences reveals fundamental limitations in LLM randomness generation across all experimental conditions.
This table provides the numerical foundation for all analyses below, showing performance across all experimental conditions and statistical metrics.

Summary of Important Findings

The following results show consistent patterns across all LLM conditions compared to true random baselines. For future sections, refer to these graphs.
Frequency Test: LLMs achieve “too perfect” distribution (0.99-1.00 p-values vs 0.55 true random)
(Click to view graph)

Runs Test: Universal LLM failure (0.08-0.16 p-values vs 0.49 true random)
(Click to view graph)

Serial Test: Very low p-values across all LLM conditions (0.00-0.10) vs higher true random performance (0.50)
(Click to view graph)
Compression: LLMs generate more compressible sequences (0.43-0.54 vs 0.58 true random)
(Click to view graph)
Entropy: LLMs show slightly higher entropy (~3.31-3.32 vs 3.26 true random)
(Click to view graph)

Serial Correlation: LLMs exhibit systematic negative correlation (-0.16 to -0.23) vs near-zero true random (-0.02)
(Click to view graph)

​
These charts show that LLMs consistently fail to generate truly random sequences. They achieve “too perfect” frequency distribution while systematically failing tests for sequential independence, indicating fundamental limitations.

Runs Test Failure For All LLMs

The most compelling evidence against LLM randomness comes from the runs test results across all experimental conditions:
LLMs: Consistently poor performance (0.08-0.16 p-values)
True Random Baseline: Consistent 0.49 p-value

The Frequency Distribution Paradox

LLMs demonstrate counterintuitive behavior: near-perfect frequency distribution (0.99-1.00) compared to true random (0.55). This “too perfect” distribution suggests that LLMs optimize for uniform digit distribution, avoiding the natural clumping that occurs in truly random sequences.

Complete Temperature Independence

The data demonstrates that temperature settings have very small impact on randomness quality:
Frequency Test: Nearly identical (1.00 vs 1.00 p-values)
Runs Test: Nearly identical failure (0.08 vs 0.10)
Entropy: Identical at 3.32
Compression: Small difference (0.43 vs 0.52)
Serial Correlation: Both show negative correlation (-0.23 and -0.17 respectively) vs true random (-0.02)

Prompt Engineering Ineffectiveness

All prompting strategies show very similar performance, showing that architectural limitations cannot be overridden by instructions.
Control: Frequency 1.00, Runs 0.10, Compression 0.50
Pattern-avoidance: Frequency 1.00, Runs 0.08, Compression 0.53
Quantum-inspired: Frequency 0.99, Runs 0.12, Compression 0.51

Feedback Loop Instability

The feedback loop experiments show dramatic instability in iterative loops:

Frequency Test: Most iterations maintain 1.00, with small fluctuations including a drop at iteration 33 (to ~0.955) and iterations 62-63 (to ~0.994). However, these variations are not statistically significant and substantially higher than true random performance (0.55), indicating consistently “too perfect” digit distribution throughout.
(Click to view graph)

Runs Test: Extreme variability from 0.00 to ~0.84 with notable performance spikes at iterations 12 (~0.67), 44-45 (~0.84), and 48 (~0.67), demonstrating unpredictable sequential pattern generation under iterative conditions.
(Click to view graph)

Serial Test: Large oscillations with values ranging from near-zero to ~0.92, including dramatic spikes at iterations 17 (~0.92) and 42 (~0.72), indicating complete loss of consistent pair-wise independence
(Click to view graph)

Compression: Extreme chaos with ratios varying from ~0.25 to ~0.62—dramatic crashes to ~0.25 at iterations 35 and 39, and drops to ~0.37 at iterations 17 and 21, representing generation of highly structured, predictable sequences
(Click to view graph)

Entropy: Fluctuations around 3.31±0.01 with a notable dramatic drop at iteration 33 (falling to ~3.298), showing information content becomes less stable during iterative generation, particularly coinciding with the frequency test breakdown
(Click to view graph)

Serial Correlation: Chaotic swings from approximately -0.43 to +0.36 with extreme positive spikes at iteration 14 (+0.36) and iterations 49-50 (+0.18), and negative extremes around iteration 26 (-0.43), demonstrating complete breakdown of independence between consecutive values
(Click to view graph)

The feedback loops show system instability as there are inconsistent results when using outputs as inputs, with some iterations performing relatively well while others collapse completely.

Key Statistical Findings Summary

Based on our comprehensive analysis:

Temperature Independence: Sampling strategies cannot fix architectural limitations

Prompt Resistance:: Explicit randomness instructions show no improvement

Feedback Instability: Iterative use creates variable performance, not consistent patterns

Theoretical Implications and Impact

Cryptographic Security

The consistently poor runs test performance (0.08-0.16 vs 0.49 true random) provides definitive evidence that LLM sequences should never be used for cryptographic purposes. The systematic failure to generate proper consecutive value patterns creates predictable structures exploitable in security applications.
​
Any application requiring cryptographically secure randomness (password generation, encryption keys, etc) should avoid LLM-generated sequences entirely.

For AI System Design

The temperature independence findings reveal that current approaches to controlling LLM “randomness” through sampling parameters are fundamentally limited. The randomness failures appear architectural rather than controllable through inference-time parameters.
​
Future AI architectures intended for randomness generation would need fundamental changes to training processes or model structures, not just different sampling strategies.

Understanding Neural Networks

The frequency paradox where LLMs achieve “too perfect” digit distribution while failing sequential tests reveals something fundamental about how these models process information. They appear optimized for surface-level statistical properties humans associate with randomness, rather than deeper mathematical randomness.
​
LLMs have learned human biases about what “looks random” from training data, rather than learning to generate truly independent sequences.

Iterative AI Systems

The feedback loop instability findings have important implications for AI applications using outputs as inputs. Rather than consistent performance or predictable degradation, we see high variability and occasional complete failures (compression ratios dropping to 0.25, correlation spikes to ±0.4).
​
Iterative AI systems may exhibit unpredictable behavior absent in single-use scenarios. This variability could be problematic for applications requiring consistent performance.

Future Research Directions

Immediate Extensions

1. Test other LLM architectures (Claude, Llama, Gemini) to determine if randomness failures are universal or model-specific.
2. Investigate whether larger models show different randomness properties, or if failures persist across model sizes.
3. Examine whether models trained on different corpora show varying randomness capabilities.

Methodological Improvements

1. Apply additional randomness tests (spectral test, overlapping template matching) for more comprehensive evaluation.
2. Test randomness quality in longer sequences (1000+ digits) to detect patterns emerging only at scale.
3. Apply similar analysis to other generation tasks (text, images, audio) to understand if randomness failures are universal.

Practical Applications

1. Investigate whether post-processing LLM outputs with traditional random number generators can improve randomness quality.
2. Determine specific applications where LLM randomness might be “good enough” despite failing strict tests.
3. Develop automated systems to identify LLM-generated sequences in real-world applications.

Conclusion

Primary Findings

This study provides the first comprehensive evaluation of LLM randomness generation capabilities, yielding several definitive conclusions:
1
.
All tested LLM conditions consistently fail runs tests, indicating systematic problems with consecutive value generation that cannot be overcome through prompting or sampling strategies.
2
.
LLMs achieve “too perfect” digit distribution compared to true random sequences, suggesting they optimize for human-recognizable randomness patterns rather than mathematical randomness.
3
.
Temperature and prompting strategies show minimal impact on randomness quality, indicating limitations are fundamental to current LLM architectures.
4
.
Feedback loops create highly variable performance with occasional complete randomness failures, suggesting potential issues for iterative AI applications.

Scientific Contributions

1
.
We’ve established a reproducible testing framework that can evaluate randomness quality in any AI system, with applications extending beyond language models.
2
.
This study provides the first systematic evidence that LLMs should not be used for applications requiring cryptographic-quality randomness.
3
.
The frequency paradox reveals fundamental differences between human intuitions about randomness and mathematical randomness, with implications for AI training approaches.

The Bottom Line

This research definitively answers whether LLMs can generate truly random sequences: they cannot. More importantly, it reveals the specific ways they fail and provides tools to evaluate these failures systematically . While LLMs excel at many practical tasks, randomness generation is not among them . The patterns embedded in their training process and architecture create systematic biases that persist across different prompting strategies and sampling parameters. Understanding these limitations is important for responsible deployment of AI systems in applications where randomness quality matters.

Acknowledgments

I would like to thank Stephen Wolfram for his mentorship throughout this project and for helping me develop and refine the research plan.
I’d like to also thank my mentor, Christian Pasquel, for his guidance throughout this research project which made it possible, as well as James Wiles for his mentorship and support.
A special thanks to the Wolfram Summer School program for providing the computational resources and research environment that enabled this investigation.

References

1
.
S. Wolfram (2002), “A New Kind of Science.” Wolfram Media.
2
.
L. Bassham et al. (2010), “A Statistical Test Suite for Random and Pseudorandom Number Generators for Cryptographic Applications,” NIST Special Publication 800-22 Rev. 1a. https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-22r1a.pdf
3
.
A. Rukhin et al. (2001), “A Statistical Test Suite for Random and Pseudorandom Number Generators for Cryptographic Applications,” NIST Special Publication 800-22. https://csrc.nist.gov/publications/detail/sp/800-22/rev-1a/final
4
.
OpenAI (2023), “GPT-4 Technical Report.” arXiv:2303.08774. https://arxiv.org/abs/2303.08774
5
.
Wolfram Research (2024), “Mathematica Documentation.” https://reference.wolfram.com/language/

CITE THIS NOTEBOOK

Investigating the Quality of Pseudorandomness Generated by Large Language Models​
by Srijit Iyer​
Wolfram Community, STAFF PICKS, July 10, 2025
​https://community.wolfram.com/groups/-/m/t/3497798