Even the most foolish of men, by some instinct of nature, by himself and without any instruction, is convinced that the more observations have been made, the less danger there is of missing the mark.

— Jakob Bernoulli, Ars conjectandi (1713), part IV

With a die, we know the chance of each face in advance: $\frac16$. But what is the chance that a man of 40 will live another twenty years? That it will rain tomorrow? That a medicine will work? In these cases there are no equal faces to count. Jakob proposed another route: observe many times and use the frequency as an estimate of the probability. If, of many men aged 40, half reached 60, the chance must be close to one half.

Everyone already thought this way, as the quotation above says. Jakob’s question went deeper: can it be proved? Does the frequency really approach the probability as the number of observations grows, or can the error stay large forever? And how many observations are needed for practical certainty? It took him twenty years to answer, and he was so proud of the result that he called it the golden theorem.

The urn

Jakob framed the problem with a hidden urn containing 3,000 white stones and 2,000 black ones. Someone who cannot look inside wants to find the proportion of white stones, which is $\frac35$. To do so, they draw a stone, note its colour, put it back and repeat.

After $n$ draws with $S_n$ white stones, the observed frequency is $\frac{S_n}n$. The precise question is: how many draws are needed for it to be morally certain, that is, with a chance of at least $\frac{1000}{1001}$, that the frequency lies between $\frac{29}{50}$ and $\frac{31}{50}$, that is, within $\frac1{50}$ of $\frac35$?

What the theorem says

Jakob’s answer has two parts. The first is qualitative: for any margin of error $\varepsilon$, however small, the chance of the frequency straying from $p$ by more than $\varepsilon$ goes to zero as $n$ grows. In today’s language,

$$\boxed{\lim_{n\to\infty}P\left(\left|\frac{S_n}n-p\right|>\varepsilon\right)=0.}$$

It is the first version of the law of large numbers, the name Poisson would give it in 1837. The second part is numerical: Jakob calculated that, for the urn, 25,550 draws are enough for moral certainty.

Simulated tosses of a coin. At first the proportion of heads swings widely; with thousands of tosses it approaches 1/2. The horizontal axis is on a logarithmic scale.
Simulated tosses of a coin. At first the proportion of heads swings widely; with thousands of tosses it approaches 1/2. The horizontal axis is on a logarithmic scale.Illustration by Euler’s Eyes

A proof with today’s tools

Jakob’s proof takes up several chapters of inequalities between the terms of the binomial. Today the same idea fits in a few lines, using an inequality that the Russian Pafnuty Chebyshev published in 1867.

Each draw is worth $1$ if the stone is white and $0$ if it is black. The mean of each draw is $p$, and its variance, which measures how much it swings around the mean, is $p(1-p)$. Adding $n$ independent draws and dividing by $n$, the mean is still $p$, but the variance is divided by $n$:

$$E\left[\frac{S_n}n\right]=p,\qquad \operatorname{Var}\left(\frac{S_n}n\right)=\frac{p(1-p)}n.$$

Here lies the secret: the mean of many observations swings less than each observation. Chebyshev’s inequality turns this into a probability:

$$P\left(\left|\frac{S_n}n-p\right|\geq\varepsilon\right)\leq\frac{p(1-p)}{n\,\varepsilon^2}.$$

With $p$ and $\varepsilon$ fixed, the right-hand side goes to zero as $n$ grows. The golden theorem is proved.

The 25,550 draws

Jakob knew his estimate was generous. With Chebyshev’s inequality it would be even worse: with $p=0.6$, $\varepsilon=0.02$ and a chance of error of $\frac1{1001}$,

$$n\geq\frac{0.6\cdot0.4\cdot1001}{0.02^2}\approx600{,}600.$$

The exact calculation with the binomial, which a computer does in seconds, shows that about 6,500 draws are enough. Neither Jakob nor Chebyshev comes close, because their inequalities hold in every situation and therefore waste precision. The fair estimate only became possible with the normal curve, which De Moivre found in 1733 precisely while improving Jakob’s theorem.

Exercise: using Chebyshev's inequality, how many tosses of a coin guarantee, with a chance of at least 95%, that the proportion of heads is within 0.05 of 1/2?

With $p=\frac12$, $p(1-p)=\frac14$. We want $\frac{1/4}{n\cdot0.05^2}\leq0.05$, that is, $n\geq\frac{0.25}{0.0025\cdot0.05}=2{,}000$. Chebyshev guarantees 2,000 tosses. The exact calculation shows that about 400 are already enough: the inequality is safe, but pessimistic.

Exercise: an opinion poll interviews 1,000 people, and 52% vote for candidate A. Does the law of large numbers guarantee that A is ahead?

No. It says the frequency approaches the true proportion as $n$ grows, but with $n=1{,}000$ there is still fluctuation. The variance of the proportion is about $\frac{0.25}{1{,}000}$, which gives a typical deviation of $\sqrt{0.00025}\approx0.016$, or 1.6 points. A difference of 2 points above 50% is close to the typical error: the result is compatible both with A ahead and with a tie.

Sources: Jakob Bernoulli, The Art of Conjecturing, part IV, trans. Edith Dudley Sylla (Johns Hopkins, 2006) · Stephen M. Stigler, The History of Statistics (Harvard, 1986), ch. 2 · Anders Hald, A History of Probability and Statistics and Their Applications before 1750 (Wiley, 1990).

All topics on James Bernoulli · ← The art of conjecturing: counting the chances · The Bernoulli numbers: adding a thousand powers in seven minutes →

Image credits.