WOLFRAM|DEMONSTRATIONS PROJECT

A Limit Theorem from Information Theory

​
show legend
sequence length 
5
How many distinct sequences can be created by rearranging

binary symbols? If the binary symbols are, say,

and
○
, and if the fraction of

's is denoted
=

×
/
, then the number of possible sequences is


. When

is zero or one there is only one possible sequence for any value of

, but when
0<<1
, the number of possible sequences increases exponentially with

. The logarithm of the number of possible sequences, expressed on a per symbol basis, is
f(,)≡
log
2



, and the limit
lim
∞
f(,)=-
log
2
()-(1-)
log
2
(1-)
can be interpreted as the average number of bits needed per symbol to describe a long binary sequence with symbol probabilities

and
1-
. This Demonstration shows
log
2



for
0.1≤≤100
with the limit
-p
log
2
()-(1-)
log
2
(1-)
.