Eduki

Overlapping distribution makes comparing two groups difficult

Comparing two individuals is fairly straightforward. The question "Which dog is taller?" can be answered by measuring the heights of two dogs and comparing them directly. Comparing two groups can be more challenging. What does it mean for beagles to generally be taller than pugs?

Let’s look at three cases. Each case compares two distributions. In each case, the mean of the two distributions, marked in red, is the same and the difference in the mean is x cm. Also, the variability of the two distributions in each case is similar. Distributions in Case I have the lowest variability and those in Case II have the highest variability.

[Dot plots here]

So, what can we conclude for each of these cases?

As you can see, higher variability in the distributions leads to bigger areas of overlap and therefore less separation. 

• In case I, there is no overlap between the distributions. All dogs in Group B are taller than those in Group A. 
• In case II, there is some overlap. As you can see, there are some dogs in group A that are taller than dogs in Group B. So, not all dogs in Group B are taller than those in Group A. 
• In Case III, there is a significant overlap in the distribution. In fact, there are many dogs in group A that are taller than the dogs in Group B. 

In general, the lesser the overlap (or bigger the separation) in distributions, the more meaningful the differences in the mean of the distributions. In Case I, you can see that there is no overlap in the distribution. Therefore, we can say that the difference in the mean (of x cm) is meaningful. So, we can conclude that the difference in the means is large enough to matter and that Group B is taller in general than Group A. 

In the second and third cases, there is quite some overlap. In fact, there is even more overlap and less separation in the third case. The important question is “Given the variability in the two distributions, are the differences in the mean large enough for us to conclude that Group B is taller than Group A”? How do we know that the differences in the mean are meaningful?

Let's explore this further in the next lesson.

How do we know whether the difference in the mean of the two distributions is meaningful?

In general, if the measures of center (mean or median) for two distributions (with similar amounts of variability) are at least two measures of variability apart, we say the difference in the measures of the center is meaningful. Visually, this means the range of typical values does not overlap. If they are closer, then we don't consider the difference to be meaningful.

So, when comparing means, we would then divide the difference between the means by the MAD. If the quotient (Q) is more than 2, then we can say that the difference between the means is meaningful. The larger this number is, the more meaningful the difference between the centers is. 

We can write:

Q = difference in the means / MAD

Let's compare two groups using this metric. 

[distributions here]

For Case I, Q = 

For Case II, Q = 

So, as you can see the difference in the mean can be considered meaningful for Case I and we can safely conclude that Group B is generally taller than Group A. However, for case II, the difference in the means is smaller than 2 and we can not safely conclude that Group B is taller than Group A. 

In summary, just knowing the difference between means (of medians) of two distributions is not enough, we also need to know the amount of variability within each distribution before we can draw any meaningful conclusions about the distribution.  

Larger differences in the mean can compensate for high variability

In the last lesson, we concluded that in Case II, the difference in the means of two distributions was not meaningful. This was because the variability in the distribution was too high. Now, would our conclusion change if the differences in the mean were larger?

[distribution here]

Let's look at our formula:

Q = difference in the means / MAD

In Case II, MAD was too big and therefore the value of Q was smaller than 2. This means that there was a big overlap between the two distributions.

What if the difference in the mean was bigger?

Let's see what happens when we move the distributions further apart. Now, the difference in the means is xxx? The new value of Q would be ..... > 2. Now, we can safely conclude that Group B is taller than Group A. 

[distributions here]

In the dot plots, you can see that the MAD is the same for both distributions in each case, but in Case B, the means are further apart. This leads to less overlap and more separation and we can safely conclude that Group B is taller than Group A. 

In summary, there are two main factors that affect how meaningful the difference in the means is:

1. The variability of the distributions, and
2. The magnitude of the difference in the means

Does the sample size affect how meaningful is the difference in the means?

Well, it turns out there there is a third factor that influences whether the differences in means between two distribution is meaningful. 

For the same difference in means and the same variability, a larger sample size leads to a more meaningful difference. For example, if our distributions had the same means and the same measures of variability, but twice the sample size, then we would consider the difference between the means to be more meaningful for the data. 

[distributions]

Below are the two distributions with identical differences in the mean and MADs. However, the distributions in Case II have twice the sample size. Therefore, the difference in the means is more meaningful in Case B. 

Comparing two groups with median and interquartile range (IQR)

Just like with means and MADs, we can also see whether the difference in the medians is twice the interquartile range (IQR) to decide whether the difference is meaningful.