How to Avoid Simpson’s Paradox in Data Analysis: A Guide for Data Enthusiasts

Have you ever encountered a situation where the data you are analyzing seems to contradict itself? For example, you might find that a certain investment strategy performs better than another in each of the past five years, but when you look at the overall performance over the entire period, the opposite is true. How can this be possible?

This is an example of what is known as Simpson’s paradox, a statistical phenomenon that can lead to misleading or erroneous conclusions if not properly accounted for. In this blog post, we will explain what Simpson’s paradox is, how it can occur, and how to avoid it.

What is Simpson’s paradox?

Simpson’s paradox, also known as Yule-Simpson effect, is named after Edward Simpson, who described the phenomenon in 1951¹. However, he was not the first one to notice it. Similar cases were reported by Karl Pearson et al. in 1899² and George Udny Yule in 1903³.

Simpson’s paradox occurs when a trend or a relationship between two variables appears in several groups of data but disappears or reverses when the groups are combined. This can happen when there is a third variable that influences both the independent and the dependent variable and is not evenly distributed across the groups.

For example, suppose you want to compare the success rates of two treatments for a disease. You have data from two hospitals, A and B, that used both treatments on different patients. The data looks like this:

Hospital	Treatment	Success	Failure	Success Rate
A	X	80	20	80%
A	Y	20	80	20%
B	X	10	90	10%
B	Y	90	10	90%

If you look at each hospital separately, you might conclude that treatment X is better than treatment Y in hospital A, and treatment Y is better than treatment X in hospital B. However, if you combine the data from both hospitals, you get a different picture:

Treatment	Success	Failure	Success Rate
X	90	110	45%
Y	110	90	55%

Now it seems that treatment Y is better than treatment X overall. How can this be?

The answer is that there is a hidden variable that affects both the choice of treatment and the outcome of the treatment. This variable could be the severity of the disease, for example. Suppose that hospital A treats more severe cases than hospital B, and that treatment X is more effective for severe cases than treatment Y, while treatment Y is more effective for mild cases than treatment X. Then it makes sense that hospital A would use treatment X more often than hospital B, and that treatment X would have a higher success rate in hospital A than in hospital B. Similarly, hospital B would use treatment Y more often than hospital A, and treatment Y would have a higher success rate in hospital B than in hospital A. However, when we combine the data from both hospitals, we are ignoring the severity of the disease, which confounds the comparison between treatments.

To avoid Simpson’s paradox, we need to control for the hidden variable by stratifying the data according to its levels. In this case, we need to compare the success rates of treatments X and Y within each level of severity. For example, if we divide the patients into two groups based on their severity score (low or high), we might get something like this:

Severity	Treatment	Success	Failure	Success Rate
Low	X	30	70	30%
Low	Y	70	30	70%
High	X	60	40	60%
High	Y	40	60	40%

Now we can see that treatment Y is better than treatment X for low severity patients, and treatment X is better than treatment Y for high severity patients. This is consistent with our hypothesis that treatment X works better for severe cases and treatment Y works better for mild cases.

How to detect and avoid Simpson’s paradox?

Simpson’s paradox can be tricky to detect because it can occur in any type of data analysis that involves comparing groups or aggregating data. It can also lead to serious errors or biases if not properly accounted for. For example, Simpson’s paradox has been observed in various fields such as medicine, education, sports, and social science.

To avoid Simpson’s paradox, we need to be careful about how we interpret and present our data. Here are some tips to help you prevent or correct Simpson’s paradox:

• Always check for possible confounding variables that might affect both the independent and the dependent variable and stratify your data accordingly. For example, if you are comparing the performance of two groups of students on a test, you might want to control for variables such as gender, age, socioeconomic status, prior knowledge, etc.
• Always look at the raw data and the sample sizes before drawing conclusions from aggregated data. For example, if you are comparing the average income of two regions, you might want to check how many people live in each region and how their incomes are distributed. A small number of outliers or a skewed distribution can distort the average and mask the true differences between groups.
• Always use appropriate statistical methods and tests to analyze your data and account for possible confounding variables. For example, if you are comparing the effects of two treatments on a continuous outcome variable, you might want to use a regression analysis or an analysis of covariance (ANCOVA) instead of a simple t-test or an analysis of variance (ANOVA). These methods allow you to adjust for covariates and test for interactions between variables.
• Always report your results with confidence intervals and p-values to indicate the uncertainty and significance of your findings. For example, if you are comparing the success rates of two treatments, you might want to report something like this: “Treatment X had a success rate of 45% (95% CI: 40% to 50%, p = 0.05), while treatment Y had a success rate of 55% (95% CI: 50% to 60%, p = 0.05). The difference between treatments was statistically significant (p < 0.01).” This way, you can show how confident you are about your estimates and how likely it is that the difference is due to chance or sampling error.
• Always be transparent and honest about your data sources, methods, assumptions, and limitations. For example, if you are using secondary data from different sources or surveys, you might want to acknowledge the potential issues with data quality, reliability, validity, and comparability. If you are using a convenience sample or a self-selected sample, you might want to admit the possible biases or generalizability problems. If you are making causal claims or policy recommendations based on your data analysis, you might want to justify your reasoning and provide evidence or arguments to support your claims.

Conclusion

Simpson’s paradox is a fascinating and important phenomenon that illustrates the complexity and subtlety of data analysis. It shows that we need to be careful and critical when we interpret and present our data, and that we need to consider all the relevant factors that might influence our results. By following the tips we have provided in this blog post, we hope that you can avoid Simpson’s paradox and make better decisions based on your data.

References

¹: Simpson EH (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B (Methodological) 13(2):238–241.

²: Pearson K et al. (1899). Mathematical contributions to the theory of evolution. On the law of reversal of frequency in heredity. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 192:151–179.

³: Yule GU (1903). Notes on the theory of association of attributes in statistics. Biometrika 2(2):121–134.

: Bross IDJ (1958). How to use ridit analysis. Biometrics 14(1):18–38.

: Bickel PJ et al. (1975). Sex bias in graduate admissions: Data from Berkeley. Science 187(4175):398–404.

: Wardrop RL (1995). Simpson’s paradox and the hot hand in basketball. The American Statistician 49(1):24–28.

: O’Brien RM (2019). Simpson’s paradox in psychological science: A practical guide. Frontiers in Psychology 10:2497.