Law of Large Numbers

Convergence in probability

$Y_n \overset{P}\longrightarrow c$

$\epsilon - n$ language: $\{Y_n, n\ge 1\}, \exists c, \forall \epsilon >0, \lim_{n \rightarrow +\infty} P\{ |Y_n - c| \ge \epsilon \} =0$

Note: Comparison between Convergence and Convergence in Probability Convergence: as $n$ grows, $Y_n$ gets closer to $c$ ($c+1 → c+0.5 → c+0.1 → … → c+\epsilon$); Convergence in Probability: as $n$ grows, the probability that $Y_n$ is not so close to $c$ gets lower ($P\{Y_n\text{ is not so close}\} = 0.1 →0.05 →0.01 → …→\epsilon$)

$X_n \overset{P}→ a, Y_n \overset{P} → b, \Rightarrow g(X_n, Y_n) \overset{P} →g(a,b) (n \rightarrow +\infty)$

Markov Inequality

for random variant $Y$, if $\mu_k$ exists, then $\forall \epsilon >0$, $P\{|Y| ≥ \epsilon\} ≤\frac{E(|Y|^k)}{\epsilon^k} \iff P\{|Y|\le \epsilon\}\ge1-\frac{E(|Y|)^k}{\epsilon^k}$.

Proof: let $Z = \left \{ \begin{array} {lcr} \epsilon & , |Y|\ge \epsilon \\ 0 & , |Y|<\epsilon \end{array} \right.$, then obviously $0≤Z≤|Y|$, then $0≤Z^k≤|Y|^k$, $E(Z^k)≤E(|Y|^k)$ while $E(Z^k) = 0+ \epsilon^k \cdot P\{|Y|≥\epsilon\},$ therefore $P\{|Y| ≥ \epsilon\} ≤\frac{E(|Y|^k)}{\epsilon^k}$.

Chebyshev Inequality

for random variant $X$, if $E(X)\equiv\mu, D(X)\equiv\sigma^2$ exists, then $\forall \epsilon >0, P\{|X-\mu|\ge \epsilon\}\le\frac{\sigma^2}{\epsilon^2}\iff P\{ |X-\mu|\le \epsilon\}\ge 1-\frac{\sigma^2}{\epsilon^2}$.

This inequality gives an important measure of the probability of a random variant getting into or out of the area near its expectation.

(Weak) Law of large numbers

Random variant sequence $\{ Y_i, i≥1\}$ obeys law of large numbers, if

$$ \exists \text{constant sequence} \{c_n, n\ge 1\}, s.t. \forall\epsilon>0, \lim_{n\rightarrow +\infty}P\{|\frac{1}{n}\sum^n_{i=1}Y_i-c_n|\ge\epsilon\}=0 $$

that is

$$ \frac{1}{n}\sum^n_{i=1}Y_i -c_n\overset{P}\longrightarrow 0, n\rightarrow+\infty $$

specially, if $c_n = c, n=1,2,...$, then

$$ \frac{1}{n}\sum^n_{i=1}Y_i \overset{P}\longrightarrow c, n\rightarrow+\infty $$

1. Bernoulli’s law of large numbers: i.i.d. $X_i\sim B(0,p)$, then $\bar{X}\xrightarrow{P}p$

random variants $X_1, X_2, …, X_n, ...$ are independent and obey $B(1,p)$, then $\frac{1}{n}\sum^n_{i=1} X_i \overset{P}\longrightarrow p$.

in other words, in an $n$-fold Bernoulli trial, event $A$ with $P(A) = p$ happens $n_A$ times, then $\forall \epsilon >0, |\frac{n_A}{n}|\overset{P}\longrightarrow p$.