Independence

Statistics: Statistics *

Independence

Motivation A very important notion in probability theory is independence. Intuitively, when two events are independent, the outcome of either event does not inform us about the outcome of the other event. This will allow us to 1) simplify many of our calculations when doing machine learning, and 2) help us to parametrize out joint distributions in a more efficient way. In this section, we will look at what it means for two variables to be independent, and then expand that understanding of why studying when two variables are independent conditional on some extra information.

Basic independence

Going back to our initial example with a customer rating our movie on a scale of 1 to 5, denoted as $R$ , and them being happy, denoted as $H$ . Suppose we now introduce yet another variable, say $B$ , which denotes whether the user has blue eyes, again taking on values 1 and 0 denoting whether or not the user has blue eyes respectively. In this case, there is no reason to assume that my belief that someone is happy changes after observing whether or not someone has blue eyes, i.e.

$\mathbb{P}(H \mid B=1) = \mathbb{P}(H).$ Without defining these notions formally here, you can convince yourself that there is no information about $H$ in $B$ . However, we could imagine a situation where someone gives a higher rating for a movie because they are happy, i.e. if we know that $H=1$ , our believe in $R=5$ increases, that is:

$\mathbb{P}(H \mid B=1) = \mathbb{P}(H).$ When there is no information about $X$ in $Y$ , or when

$\mathbb{P}(X \mid Y) = \mathbb{P}(X),$ we say that $X$ and $Y$ are independent. You may also have encountered this as

$\mathbb{P}(X, Y) = \mathbb{P}(X) \cdot \mathbb{P}(Y),$ which is exactly the same thing when multiplied by $\mathbb{P}(Y)$ and making using of the fact that $\mathbb{P}(X, Y) = \mathbb{P}(Y) \cdot \mathbb{P}(X \mid Y).$
When $X$ and $Y$ are not independent, i.e. when $X$ tells me something about $Y$ , we call them dependent.

Suppose we want to parametrize a joint distribution over 10 binary random variables, $X_1, \cdots, X_{10}$ . If we do not assume any independence, we would need $2^{10} - 1 = 1023$ parameters to describe this joint distribution. Suppose now that all $X_i$ are independent, i.e. $\mathbb{P}(X_1, \cdots, X_{10}) = \prod_{i=1}^{10} \mathbb{P}(X_i)$ . Explain why now we only need $10$ parameters to describe our joint distribution.

Conditional Independence

In a real-life situation, simple independence is almost something you never encounter. A much more common scenario is that of two random variables being independent given some third variable.

Suppose we consider two students that both travel by the same train to the university, and we denote $S_1$ as the binary variable describing whether student 1 is on time for the lecture, and $S_2$ the binary variable describing whether student 2 is on time. Intuitively, these events are related, since when you tell me that $S_1$ is on time, that increased my belief that $S_2$ also is on time, for the reason that the trains were not delayed. In other words,

$\mathbb{P}(S_2 = 1 \mid S_1 = 1) > \mathbb{P}(S_2 = 1).$ Similarly, if $S_1$ is not on time, my belief increases that $S_2$ won’t be on time either. Therefore, we observe that $S_1$ and $S_2$ are not independent.

However, let us now introduce yet a third binary variable $T$ denoting whether the train was on time. How does knowing $T$
change the above situation? In this case, observing that $S_1$ is or is not on time if I already know $T$ will not tell me any more information than I could already derive from $T$ . The reason is that the only thing it could tell me is something about whether or not the train is delayed, and since we already knew that the train was not delayed we obtain no information.

In this case, we have

$\mathbb{P}(S_2 \mid T, S_1) = \mathbb{P}(S_2 \mid T).$ n that case, we say that $S_2$ is conditionally independent of $S_1$ given $T$ . In other words, we say that $X$ is conditionally independent of $Y$ given $Z$ , if my belief in is not affected by the outcome of $Y$ if I already know that $Z$ .

In the above example, we had a situation where $X$ and $Y$ were dependent, but $X$ and $Y$ were independent when informed about some $Z$ . That is., even if there is an ‘information flow’ between $X$ and $Y$ , there can be no ‘information flow’ from $X$ to $Y$ if we know that $Z$ . The other case is also possible, the situation where $X$ and $Y$ are independent variables, but they become dependent given some variable $Z$ . To illustrate this, let us consider a new - quite inspired on our breakfast - scenario.

Suppose that $S$ is the random variable that describes whether or not the first strawberry I eat on a day is sweet. Moreover, let $E$ denote whether or not the first coffee I prepare is an espresso. Needless to say, the outcomes of these events do not affect each other, and hence $S$ and $E$ are independent. The reason we picked such a weird example is that even such two (very) independent events can be made conditionally dependent if we consider a crazy enough third variable...

Assume for a moment we asked someone to follow us around during breakfast, making sure to light some fireworks exactly when either the first strawberry we eat on a given day is sweet or the first coffee we drink is an espresso (or both), denoted with $F$ . Even though this example is perhaps not the most realistic one, it does show us an interesting behavior that can occur with these random variables: even though $S$ would not tell me anything about $E$ in general, suppose now that $F=1$ , i.e. I have seen some fireworks gone off, and I also just had a very sour strawberry. Then, it must be the case that indeed my first coffee was an espresso, i.e. now $S=0$ did indeed tell me that $E=1$ through our third random variable $F$ . So here, fireworks did allow for an information stream between my morning espresso and strawberry. Of course, this example is slightly silly but it does illustrate something fundamental: one needs to be careful when considering how random variables affect one another, being imprecise about which variables we assume to be (conditionally) independent can remove and create patterns in our observations that are unexpected.

Summary In this theory page, you have learned about the independence of random variables.