====== -#4 Total probability ======
**Bayes' law** is tightly related with **the law of total probability**.
For example, let us view the exercise form previous lesson:
We have two urns, $\text{I}$ and $\text{II}$. Urn $\text{I}$ contains $2$ black balls and $3$ white
balls. Urn $\text{II}$ contains $1$ black ball and $1$ white ball. An urn is drawn at random
and a ball is chosen at random from it. What is the probability of choosing urn $\text{I}$ if the obtained
ball is black?
It can be observed right away that the probability $\text{Pr}[B|\text{I}]=\frac{2}{5}$, where
$B:=\text{"a black ball is drawn"}$ and $\text{I}:=\text{"urn I is chosen"}$. Thus, it is reasonable
to use Bayes' law
$$\text{Pr}[\text{I}|B] = \frac{\text{Pr}[B|\text{I}]\cdot\text{Pr}[\text{I}]}{\text{Pr}[B]}$$
for answering the question. The probability $\text{Pr}[\text{I}]=\frac{1}{2}$, so only
$\text{Pr}[B]$ is unknown. On order to understand the event $B$, let us draw the following diagram:
{{:probability:total_probability_1.png|}}
i.e. the event $B$ can happen in relation with urn $\text{I}$ or in relation with urn $\text{II}$. Hence,
$$\text{Pr}[B]=\text{Pr}[(B\cap\text{I})\cup (B\cap\text{II})]=\text{Pr}[B\cap\text{I}]+
\text{Pr}[B\cap\text{II}],$$
since the events $\text{"black ball is drawn from urn I"}$ and $\text{"black ball is drawn from urn II"}$
are mutually exclusive.
This is precisely what **the law of total probability** says:
If $A_i$, where $i=1, 2, 3, \dots, n$, are pair vise mutually exclusive outcomes of an experiment/process
with random results, such that
$\text{Pr}[A_1]+\text{Pr}[A_2]+\text{Pr}[A_3]+\dots+\text{Pr}[A_n]=1$, then for any event $B$ that
happens in relation with the events
$A_i$, we have
$$\text{Pr}[B]=\text{Pr}[B\cap A_1]+\text{Pr}[B\cap A_2]+\text{Pr}[B\cap A_3]+\dots +\text{Pr}[B\cap A_n]=
\sum_{i=1}^n\text{Pr}[B\cap A_i].$$
Diagram:\\
{{:probability:total_probability_2.png|}}
In our example, $n=2$, $A_1=\text{I}$, and $A_2=\text{II}$. One of two urns is chosen, thus the events are
mutually exclusive.
Note that $\text{Pr}[\text{I}]+\text{Pr}[\text{II}]=
\frac{1}{2}+
\frac{1}{2}=1$. We already obtained, that
$$\text{Pr}[B]=\text{Pr}[B\cap\text{I}]+\text{Pr}[B\cap\text{II}].$$
Actually, in order to calculated the probability, we can give to the formula also an
alternative form. Just let us take into account that the events $B$ and $\text{I}$, and also the events
$B$ and $\text{II}$ are not independent, in which case $\text{Pr}[B\cap\text{I}]=
\text{Pr}[B|\text{I}]\cdot\text{Pr}[\text{I}]$ and $\text{Pr}[B\cap\text{II}]=
\text{Pr}[B|\text{II}]\cdot\text{Pr}[\text{II}]$. By replace these equalities into the formula:
$$\text{Pr}[B]=\text{Pr}[B\cap\text{I}]+\text{Pr}[B\cap\text{II}]=
\text{Pr}[B|\text{I}]\cdot\text{Pr}[\text{I}]+\text{Pr}[B|\text{II}]\cdot\text{Pr}[\text{II}].$$
Thus,
$\text{Pr}[B]=\frac{2}{5}\cdot\frac{1}{2}+\frac{1}{2}\cdot\frac{1}{2}=\frac{9}{20}$.
Alternatively, the law of total probability can be written
$$\text{Pr}[B]=\text{Pr}[B|A_1]\cdot\text{Pr}[A_1]+\text{Pr}[B|A_2]\cdot\text{Pr}[A_2]+\dots +\text{Pr}[B|A_n]\cdot\text{Pr}[A_n]=
\sum_{i=1}^n\text{Pr}[B|A_i]\cdot\text{Pr}[A_i].$$
The illustrative diagram:
{{:probability:total_probability_tree.png|}}
What is the probability that $y=1$, where $y$ is evaluated randomly by one element from the list $A$
following the algorithm below?
let us choose $x$ randomly from $\{0, 1, 2\}$\\
if $x=0$, let $A=(1,0,0,0)$\\
if $x=1$, let $A=(1,0,0,0,0,0,0,0,0,0)$\\
if $x=2$, let $A=(1,1,0,0,0,0,0)$\\
$y\leftarrow A$
As you all know, information is transmitted digitally as a binary sequence
(bits). However, noise on the channel corrupts the signal, in that a
digit transmitted as $0$ is received as $1$ with probability $1 −\alpha $ , with a similar
random corruption when the digit $1$ is transmitted. It has been observed that,
across a large number of transmitted signals, the $0$s and $1$s are transmitted
in the ratio $3 : 4$.
Given that the sequence $101$ is received, what is the probability distribution
over transmitted signals? Assume that the transmission and reception
processes are independent.
__Hint__: "the $0$s and $1$s are transmitted
in the ratio $3 : 4$" tells us, that in general out of seven bits four are $1$s. The task should be solved
under the assumption that all the sent bits are independent.
++Answer| Note, that the set $\{000, 001, 010, 011, 100, 101, 110, 111\}$
describes all possible signals that could be sent/received.\\
Let us denote\\
signal events correspondingly $\{S_0, S_1, S_2, S_3, S_4, S_5, S_6, S_7\}$ and\\
received events correspondingly $\{R_0, R_1, R_2, R_3, R_4, R_5, R_6, R_7\}$.\\
We received $101$ that is $R_5$. Our task is to calculate $\text{Pr}[S_i|R_5]$ for
$i=1,\dots,7$. However, on the information given, we can only compute $\text{Pr}[R_5|S_i]$. We need to
use the Bayes' law:\\
$\text{Pr}[S_i|R_5] = \frac{\text{Pr}[R_5|S_i]\cdot\text{Pr}[S_i]}{\text{Pr}[R_5]}$. According to the law of
total probability $\text{Pr}[R_5]=\sum^7_{i=0}\text{Pr}[R_5|S_i]\cdot\text{Pr}[S_i]$. Consider, for
example, $\text{Pr}[R_5|S_0]$. If $000$ is transmitted, the probability that $101$ is received
is $(1 − \alpha )\cdot\alpha\cdot (1 − \alpha ) = \alpha (1 − \alpha )^2$
(corruption, no corruption, corruption). Analogously,\\
$\text{Pr}[R_5|S_0]=\alpha (1 − \alpha )^2$\\
$\text{Pr}[R_5|S_1]=\alpha ^2(1 − \alpha )$\\
$\text{Pr}[R_5|S_2]=(1 − \alpha )^3$\\
$\text{Pr}[R_5|S_3]=\alpha (1 − \alpha )^2$\\
$\text{Pr}[R_5|S_4]=\alpha ^2 (1 − \alpha )$\\
$\text{Pr}[R_5|S_5]=\alpha ^3$\\
$\text{Pr}[R_5|S_6]=\alpha (1 − \alpha )^2$\\
$\text{Pr}[R_5|S_7]=\alpha ^2 (1 − \alpha )$.\\
The probability of transmitting a $1$ is $\frac{4}{7}$, thus,\\
$\text{Pr}[S_0]=\frac{3}{7}^3$\\
$\text{Pr}[S_1]=\frac{4}{7}\frac{3}{7}^2$\\
$\text{Pr}[S_2]=\frac{4}{7}\frac{3}{7}^2$\\
$\text{Pr}[S_3]=\frac{4}{7}^2\frac{3}{7}$\\
$\text{Pr}[S_4]=\frac{4}{7}\frac{3}{7}^2$\\
$\text{Pr}[S_5]=\frac{4}{7}^2\frac{3}{7}$\\
$\text{Pr}[S_6]=\frac{4}{7}^2\frac{3}{7}$\\
$\text{Pr}[S_7]=\frac{4}{7}^3$.\\
Hence,\\
$\text{Pr}[R_5]=\frac{48\alpha^3+136\alpha^2(1-\alpha)+123\alpha(1-\alpha)^2+36(1-\alpha)^3}{343}$ and now
the Bayes' law can be used. We find here just one probability:\\
$\text{Pr}[S_5|R_5]=\frac{48\alpha^3}{48\alpha^3+136\alpha^2(1-\alpha)+123\alpha(1-\alpha)^2+36(1-\alpha)^3}$\\
which is the probability of correct reception.
++
[[probability:05_expected_value]]\\