Exploring the central limit theorem computationally
by Matt Kafker
University of Washington, Department of Physics
Junior Research Affiliate, Wolfram Physics Project
July 14, 2021
by Matt Kafker
University of Washington, Department of Physics
Junior Research Affiliate, Wolfram Physics Project
July 14, 2021
Introduction
Introduction
Hello everyone!
Today, I would like to write a bit about the Central Limit Theorem.
I have taken several courses in the past in which I was taught the Central Limit Theorem, but it has never quite made sense to me. However, in a conversation with Stephen Wolfram at the 2021 Wolfram Summer School, Nassim Taleb provided a Mathematica demonstration which allowed me to understand the Central Limit Theorem better than I ever have before. In what follows, I have tried to reproduce what I was able to understand from Dr. Taleb’s presentation. All credit for brilliance goes to Dr. Taleb, and all errors are my own.
I hope that you will find the demonstration as valuable as I did.
Today, I would like to write a bit about the Central Limit Theorem.
I have taken several courses in the past in which I was taught the Central Limit Theorem, but it has never quite made sense to me. However, in a conversation with Stephen Wolfram at the 2021 Wolfram Summer School, Nassim Taleb provided a Mathematica demonstration which allowed me to understand the Central Limit Theorem better than I ever have before. In what follows, I have tried to reproduce what I was able to understand from Dr. Taleb’s presentation. All credit for brilliance goes to Dr. Taleb, and all errors are my own.
I hope that you will find the demonstration as valuable as I did.
What Is The Central Limit Theorem (CLT)?
What Is The Central Limit Theorem (CLT)?
First, we state the Central Limit Theorem directly (using the definition given on Wikipedia).Theorem: If ,,..., are random samples drawn from a population with overall mean μ and finite variance , and if is the sample mean, then the limiting form of the distribution is a standard normal distribution.In this post, we will attempt to render this theorem intuitive with the use of simple computational experiments.
X
1
X
2
X
n
n
2
σ
X
n
Z-μ
lim
n∞
n
X
n
σ
The Essence of The Central Limit Theorem
The Essence of The Central Limit Theorem
More colloquially, the CLT says that sums of independent, identically distributed random variables will themselves tend to be normally distributed, regardless of the underlying distribution of the random variables. When stated like this, I have to think very hard to understand what the CLT is actually saying. But Taleb’s demonstration makes things much clearer.
First, let’s get some random variables. We’ll start with a uniform distribution on the interval [0, 1].
First, let’s get some random variables. We’ll start with a uniform distribution on the interval [0, 1].
RandomVariate[UniformDistribution[],100]
Out[]=
{0.73917,0.505651,0.288759,0.929301,0.841769,0.0759922,0.925939,0.744536,0.0153736,0.177514,0.523461,0.284085,0.305589,0.0354152,0.247349,0.147824,0.128593,0.363716,0.587513,0.873385,0.179262,0.131607,0.87461,0.855447,0.336418,0.146995,0.184858,0.206149,0.990793,0.521213,0.146787,0.605557,0.901931,0.998488,0.551421,0.441776,0.0329921,0.643736,0.328289,0.424934,0.824564,0.507153,0.88973,0.0288004,0.281552,0.736313,0.186584,0.132378,0.116741,0.324821,0.894849,0.908483,0.524415,0.637309,0.823861,0.858348,0.80094,0.0309205,0.00584757,0.0346577,0.10226,0.746967,0.198067,0.265402,0.807853,0.297974,0.0951325,0.0530313,0.262561,0.672165,0.0113086,0.263758,0.784652,0.775503,0.640431,0.561314,0.211606,0.041886,0.117352,0.714457,0.452272,0.724811,0.174257,0.318111,0.385131,0.323331,0.00793365,0.363437,0.89505,0.646206,0.705054,0.710125,0.792056,0.847488,0.362604,0.0613654,0.0207438,0.60968,0.693351,0.482992}
This is what is meant by “independent, identically-distributed random variables.” We are just sampling our distribution over and over again, and we end up with a bunch of values between zero and one. Now, we sum them up.
Total[%]
Out[]=
44.9921
We do not have any reason to care about the particular value of 44.9921 by itself. However, a pattern emerges if we repeat this procedure above many times and compare the results.
Total[RandomVariate[UniformDistribution[],100]]
Out[]=
50.5945
Total[RandomVariate[UniformDistribution[],100]]
Out[]=
48.7883
Total[RandomVariate[UniformDistribution[],100]]
Out[]=
45.3809
Total[RandomVariate[UniformDistribution[],100]]
Out[]=
51.5965
Evidently, the answer tends to be around 45-50ish, but it fluctuates each time we run the computation. To understand the fluctuations, let’s repeat this procedure a large number of times, and look at the histogram of our results.
Table[Total[RandomVariate[UniformDistribution[],100]],100000]//Histogram
Out[]=
That’s a Gaussian! And this is the Central Limit Theorem: sums of random variables tend to form a normal distribution when you run the experiment many times. But if this verbiage is confusing, just look at the Mathematica code! That one line contains the heart of the CLT.
People tend to discuss the CLT in terms of averages of the random variable being sampled. But of course, if a sum of random variables is normally distributed, so too will be the average––the average is proportional to the sum!
People tend to discuss the CLT in terms of averages of the random variable being sampled. But of course, if a sum of random variables is normally distributed, so too will be the average––the average is proportional to the sum!
Table[Mean[RandomVariate[UniformDistribution[],100]],100000]//Histogram
Out[]=
The only difference is that now we have replaced “Total” with “Mean”. We see a gaussian yet again––a different mean and a different variance, but a gaussian all the same.
How Robust is the CLT?
How Robust is the CLT?
Let us now poke and prod at this result from various angles. How robust is it? We made a few choices above. Does this central-limiting behavior depend on any of those choices?
Firstly, we decided to sum up 100 random samples. What happens if we change this number? Let’s use 2, 3, 5, 10, 1000, and 10000 samples, and see what happens.
Firstly, we decided to sum up 100 random samples. What happens if we change this number? Let’s use 2, 3, 5, 10, 1000, and 10000 samples, and see what happens.
TableHistogramTable[Total[RandomVariate[UniformDistribution[],n]],100000],,{n,{2,3,5,10,1000,10000}}
Out[]=
,
,
,
,
,
Evidently, two random samples is not quite enough to get the gaussian distribution. But even with three samples, the distribution is clearly taking shape. So central-limiting behavior appears to emerge regardless of the number of random variables summed, above some small lower bound.
We also arbitrarily decided to run the experiment 100,000 times and plot a histogram of the results. How does central-limiting behavior depend on the number of trials? We expect that we need a certain minimum number of trials to even know what shape the distribution takes. Let us try powers of 10:, , , , , , and . (We now return to using 100 samples).
We also arbitrarily decided to run the experiment 100,000 times and plot a histogram of the results. How does central-limiting behavior depend on the number of trials? We expect that we need a certain minimum number of trials to even know what shape the distribution takes. Let us try powers of 10:
1
10
2
10
3
10
4
10
5
10
6
10
7
10
TableHistogramTable[Total[RandomVariate[UniformDistribution[],100]],n],,{n,{10,100,1000,10000,100000,1000000,10000000}}
Out[]=
,
,
,
,
,
,
Perhaps this is what we should have expected! Below a certain number of trials, it is not at all clear what the distribution will ultimately look like. But around 1000-10000 trials, the distribution is clearly taking shape.
We conclude that above a certain minimum number of samples and trials, we can expect central limiting behavior to emerge, at least for the uniform distribution.
We conclude that above a certain minimum number of samples and trials, we can expect central limiting behavior to emerge, at least for the uniform distribution.
Changing the Distribution of the Random Variables
Changing the Distribution of the Random Variables
So let us finally turn to the most interesting variable in our experiment: the statistical distribution being sampled. We originally chose a uniform distribution on the interval [0, 1]. What happens when we choose other distributions? As it turns out, some distributions converge much more slowly to central-limiting behavior.
Suppose, to test the dumbest case first, we sample instead from a normal distribution. We would certainly hope that the sums of these random samples are distributed normally. And indeed this is what we observe.
Suppose, to test the dumbest case first, we sample instead from a normal distribution. We would certainly hope that the sums of these random samples are distributed normally. And indeed this is what we observe.
Table[Total[RandomVariate[NormalDistribution[],100]],100000]//Histogram
Out[]=
How about discrete distributions? The Bernoulli distribution describes events with only two possible outcomes, one of which has probability p and the other (1 - p). This distribution might describe the outcomes of flipping a (potentially biased) coin. If p = 0.75 and we flip 10,000 coins, we might see a distribution like this.
RandomVariate[BernoulliDistribution[0.75],10000]//Histogram
Out[]=
Does the Bernoulli distribution exhibit central-limiting behavior? It is easy enough to test.
Table[Total[RandomVariate[BernoulliDistribution[0.75],100]],1000000]//Histogram
Out[]=
Indeed it does as well!
What happens if the distribution is very wide, so that the random numbers are sampled from a large range?
What happens if the distribution is very wide, so that the random numbers are sampled from a large range?
Table[Total[RandomVariate[UniformDistribution[{-100000,100000}],100]],1000000]//Histogram
Out[]=
Clearly, this does not affect central-limiting behavior either!
Thus, we conclude that central-limiting behavior is clearly at least a somewhat robust phenomenon to the choice of statistical distribution.
Now, let’s really try to break the CLT.
Thus, we conclude that central-limiting behavior is clearly at least a somewhat robust phenomenon to the choice of statistical distribution.
Now, let’s really try to break the CLT.
Breaking the CLT with Fat Tails
Breaking the CLT with Fat Tails
To “break” the CLT, we will use fat-tailed distributions, which Dr. Taleb has written about extensively in the Incerto. One example of a fat-tailed distribution is the Pareto distribution, which has a power-law tail.
PlotEvaluate@Table[PDF[ParetoDistribution[1,α],x],{α,1,5}],{x,0,10},
Out[]=
We can see that for , the tail of this distribution does indeed get long. What does this mean for our samples? Let , which appears to be close to the edge separating fat vs. thin-tailed. We sample this distribution a large number of times.
α=1.2
Histogram[RandomVariate[ParetoDistribution[1,1.2],1000000],100]
Out[]=
At first, this seems like a boring histogram with all data concentrated near 0, but if we look closer, we can see that 1 of our 1,000,000 random samples exceeded a value of 10,000! Perhaps you can already see why fail-tailed distributions are a problem for the central limit theorem.
Let’s run our usual experiment again on the Pareto distribution.
Let’s run our usual experiment again on the Pareto distribution.
Table[Total[RandomVariate[ParetoDistribution[1,1.2],100]],1000000]//Histogram
Out[]=
That’s not a gaussian! It appears that we get nearly 1,000,000 results clustered near 0, and yet a very small number of sums, just 70, exceed 20,000!
So we can already see that fat-tailed distributions are a different beast. With this number of trials and samples-per-trial, central-limiting behavior had already emerged in every other distribution we have look at thus far. Let us see if increasing both of these numbers allows us to get closer to the normal distribution we expect. Let’s try increasing the number of samples to 10,000 and the number of trials to 10,000,000.
So we can already see that fat-tailed distributions are a different beast. With this number of trials and samples-per-trial, central-limiting behavior had already emerged in every other distribution we have look at thus far. Let us see if increasing both of these numbers allows us to get closer to the normal distribution we expect. Let’s try increasing the number of samples to 10,000 and the number of trials to 10,000,000.
ParallelTable[Total[RandomVariate[ParetoDistribution[1,1.2],10000]],10000000]//Histogram
Out[]=
It appears that things are not getting better: we are still nowhere close to the Gaussian we were hoping for! And my computer is really starting to sweat. So, when α is below a certain value, the Pareto distribution does not exhibit central limiting behavior––at least, not without a very large amount of computation. Is this very slow convergence to central-limiting behavior a particular property of the Pareto distribution?
Another fat-tailed distribution is the Cauchy distribution. We determine whether the Cauchy distribution exhibits this same property.
Another fat-tailed distribution is the Cauchy distribution. We determine whether the Cauchy distribution exhibits this same property.
PlotPDF[CauchyDistribution[],x],{x,-7,7},
Out[]=
Although the distribution looks bell-shaped, in fact the formula for the PDF is given by
PDF[CauchyDistribution[],x]
Out[]=
1
π(1+)
2
x
and hence it decays much less quickly with x than the Gaussian does. Let us now perform our central limit test again.
ParallelTable[Total[RandomVariate[CauchyDistribution[],100]],10000000]//Histogram
Out[]=
Again, this is nowhere close to a gaussian, even having sampled 10,000,000 times! Almost every sum is approximately zero, except for the one that exceeds 60 million! (With a few in between.)
Discussion
Discussion
I could continue along these lines, but I think that the central message should by now be clear. The CLT describes a real phenomenon. Central-limiting behavior emerges robustly from a wide variety of statistical distributions––discrete, continuous, finitely supported, infinitely supported––after a non-astronomical number of trials. It is easy to observe this behavior with small computational experiments run on a laptop.
However, there exist distributions that do not practically obey the Central Limit Theorem. These distributions, fat-tailed distributions, have the property that there is a non-negligible (but still small) probability of generating a sample with a very large magnitude. In fact, these distributions do exhibit central-limiting behavior eventually, but our explorations have revealed that we would need astronomical numbers of trials before we observe this behavior.
The Central Limit Theorem is widely used for statistical inference––for example, when using a sample mean to estimate a population mean. It is thus heartening to know that we can reliably reproduce central-limiting behavior in controlled computer experiments under rather general circumstances. Nevertheless, in real-world experiments, if there is reason to believe a variable is fat-tailed, other techniques will probably have to be used for reliable inference.
However, there exist distributions that do not practically obey the Central Limit Theorem. These distributions, fat-tailed distributions, have the property that there is a non-negligible (but still small) probability of generating a sample with a very large magnitude. In fact, these distributions do exhibit central-limiting behavior eventually, but our explorations have revealed that we would need astronomical numbers of trials before we observe this behavior.
The Central Limit Theorem is widely used for statistical inference––for example, when using a sample mean to estimate a population mean. It is thus heartening to know that we can reliably reproduce central-limiting behavior in controlled computer experiments under rather general circumstances. Nevertheless, in real-world experiments, if there is reason to believe a variable is fat-tailed, other techniques will probably have to be used for reliable inference.
Concluding Remarks
Concluding Remarks
I hope that you have found this demonstration illuminating! I certainly learned a lot while exploring these ideas. I also enthusiastically recommend all of Dr. Taleb’s books.
Thank you for reading!
-Matt
Thank you for reading!
-Matt