How to test a random number generator yourself

A random number generator makes a checkable claim: every outcome equally likely, no memory, no pattern. This article is a working guide to checking it — from a five-minute browser test to the batteries used in formal certification — and, just as importantly, to what a passing grade does not prove.

Level 1: chi-square — the workhorse

The chi-square goodness-of-fit test, documented in the NIST/SEMATECH e-Handbook of Statistical Methods, asks: are the observed counts consistent with the expected ones?

Worked example. Generate 4,500 single picks from 1–45. A fair generator expects each number 4,500/45 = 100 times. Compute:

χ² = Σ (observed − expected)² / expected, summed over the 45 numbers.

With 45 categories there are 44 degrees of freedom, and the standard critical value at the 5% significance level is 60.48. If your χ² lands below 60.48, the counts are consistent with uniformity; far above it, something is skewed.

This test has teeth. Take the modulo-biased generator, where numbers 1–31 have probability 6/256 and 32–45 have 5/256. Its expected χ² contribution per draw is small — but it accumulates linearly with sample size, while chance fluctuation only grows with the square root. At 4,500 draws the biased generator's expected χ² is already ≈ 44 (fair baseline) + 4,500 × Σ(pᵢ − 1/45)²/(1/45) ≈ 44 + 55 ≈ 99 — comfortably past the 60.48 line, so the bias is usually caught with one afternoon's data. Our randomness tester runs exactly this test on pasted or generated numbers and shows the full working; the draw simulator will happily manufacture test batches.

Level 2: order matters — runs and Kolmogorov–Smirnov

Chi-square only counts frequencies; a generator that emitted 1, 2, 3, …, 45, 1, 2, 3, … forever would pass it perfectly. So you add tests that see sequence:

• Runs test (e-Handbook reference): count maximal stretches that stay above or below the median. Too few runs means sticky, trending output; too many means an oscillating, over-corrected pattern. For n values the run count has a known mean and variance, so the deviation converts to a standard z-score.
• Kolmogorov–Smirnov test (e-Handbook reference): compare the empirical cumulative distribution against the ideal uniform ramp and take the largest vertical gap, D. Where chi-square needs arbitrary binning, KS uses every data point's exact position — it's the standard check that supposedly-uniform fractions really do spread evenly across [0, 1).

A generator worth trusting should be passing frequency, runs and KS simultaneously, on fresh data, repeatedly.

Level 3: the professional batteries

Serious validation stacks hundreds of tests, each hunting a different pattern family:

• NIST SP 800-22 — the US standards institute's suite of 15 tests for cryptographic generators, examining bit-level properties: frequency within blocks, longest runs of ones, ranks of binary matrices, spectral (Fourier) structure, template matches, approximate entropy, random excursions. It's the baseline reference in gaming and security certification alike.
• Dieharder — Robert G. Brown's open-source successor to George Marsaglia's famous Diehard battery; birthday spacings, overlapping permutations, parking-lot and craps tests among many others, all runnable from a command line against your own generator.
• TestU01 — L'Ecuyer and Simard's C library (ACM TOMS, 2007), whose escalating batteries SmallCrush, Crush and BigCrush constitute the de facto academic gold standard; BigCrush runs on the order of a hundred tests over tens of billions of numbers. "Passes BigCrush" is the strongest statistical compliment a generator commonly receives.

These are what testing labs actually run when certifying gambling RNGs — the certification side is covered in certified RNGs and testing labs.

What passing proves — and what it can't

Here is the crucial, frequently-fumbled distinction.

Passing proves: the output is statistically indistinguishable from uniform, independent draws — no frequency skew, no serial correlation, no structure of any kind the battery knows how to look for. A generator with modulo bias or a sticky pattern will be caught fast.

Passing cannot prove:

1. Unpredictability. The definitive counterexample sits in your browser: V8's Math.random() uses xorshift128+, which passes TestU01 — yet published work recovers its full internal state from a handful of outputs and predicts everything that follows. Statistical tests interrogate the output's shape; predictability is about whether the mechanism can be reverse-engineered. No battery of shape-tests can see that. Only design analysis can — which is why the trustworthy stack is a CSPRNG by construction plus passing tests, never tests alone.
2. Future behaviour. Tests certify the sample you fed them. A generator that degrades, or is swapped out after certification, passed then, not now — the gap that made the Eddie Tipton insider fraud possible.
3. Certainty. Every test is probabilistic: a genuinely fair generator should fail a 5%-level test about 1 time in 20, and a suite of 100 tests should show a few marginal p-values. A vendor reporting hundreds of tests with zero borderline results is itself a red flag — that pattern is too clean to be honest data.

A practical recipe

1. Generate ≥ 4,500 values and run chi-square in the randomness tester; repeat with fresh batches.
2. Check ordering with runs and KS on the same data.
3. For anything that matters, demand a generator whose design is cryptographic — like ours — and treat statistical passes as a necessary check, never a sufficient one.

Testing is how you catch broken generators. Understanding why a pass isn't a security proof is how you avoid being fooled by intact-looking ones.