Over the past few decades, the scientific community has grappled with what is now known as the replication crisis—a widespread recognition that many significant results published in academic research cannot be replicated under rigorous testing. This crisis has shown critical flaws in traditional statistical methods, particularly the over-reliance on P-values, and has profound implications for scientific research and statistical practices.
In this article, we will go through the genesis of the replication crisis, explore the specific issues with P-values and common research practices, and how these challenges manifest by the lack of reproducibility and the excessive analyst degrees of freedom. We will also look at the statistical community’s response to the crisis and how these questions relate to modern Marketing Mix Modeling (MMM).
The Genesis of the Replication Crisis
Let’s start at the top: what is a P-value?
A P-value is a measure that helps determine whether the results of an experiment are statistically significant.
Specifically, it indicates the probability of obtaining results at least as extreme as those observed, assuming the null hypothesis (which posits no effect or no difference) is true.
In simpler terms, a P-value helps us gauge whether the observed data could have happened by random chance.
Over the past 50 years, statisticians have relied on a P-value threshold of 0.05 to determine the significance of findings, essentially dictating what gets published in academic research.
This threshold has long been considered the gold standard, but it’s become increasingly clear that this reliance has contributed to a replication crisis in the scientific community.
However, generating statistically significant results using a P-value of 0.05 is easier than it seems.
Let’s touch on that:
Problems with P-values and Research Practices
The P-value threshold of 0.05 has been a cornerstone of statistical analysis for decades, but its application has led to significant issues in research practices.
- One of the primary problems is the ease with which statistically significant results can be generated. By running multiple experiments, roughly one out of every twenty will appear significant purely by chance, even if there is no real signal in the data.
- Also, researchers often analyze a single experiment’s data in numerous ways to find a statistically significant result. This practice, known as “p-hacking,” involves trying different analytical methods or data subsets until a significant result comes up. The analyst’s degrees of freedom allow for many potential pathways to significance, leading to results that are more about the selected analysis method than any underlying true effect.
If statistical significance does not necessarily equate to a meaningful or reproducible finding, that’s a huge problem.
Why does this happen? One reason behind this is incentives: the academic pressure for publication and tenure can make researchers motivated to produce publishable results to gain recognition and tenure. Consequently, they may keep searching until they find something significant, regardless of whether it represents a true effect.
This leads to the “file drawer problem,” where non-significant results are not published and instead are kept hidden away in researchers’ files. This creates a publication bias where the scientific literature is skewed towards positive findings.
As a result, the true nature of the evidence is obscured, and many published significant results fail to replicate when re-examined. Positive results are overrepresented, while null findings are underreported, distorting the true state of evidence.
This has very meaningful implications for fields like marketing, econometrics, even medicine, and healthcare where decisions based on unreliable research can lead to wasted resources. For example, a large-scale project to replicate 100 studies found that only 36% of the replications yielded significant results.
So, how does the statical community move forward when this has been a cornerstone for so long?
Statistical Community’s Response and Best Practices for Marketing Mix Modeling
A significant consensus has emerged on the inadequacy of the 5% P-value threshold as a reliable marker of significance. It’s an arbitrary threshold that has proven to be insufficient for rigorous statistical analysis.
One of the key responses is the pre-registration of studies —documenting the study design and analysis plan before data collection begins— to limit the degrees of freedom of researchers and reduce the risk of p-hacking.
Additionally, there is a growing emphasis on confidence intervals and effect sizes over only relying on P-values. Confidence intervals provide a range of values within which the true effect size is likely to fall which helps to contextualize its practical significance beyond mere statistical significance.
This is one of the reasons why Recast’s MMM is Bayesian and not frequentist – which relies on hypothetical repeated sampling. Bayesian methods incorporate prior knowledge and give marketers a probabilistic interpretation of the data and a clearer understanding of the likelihood that a particular hypothesis is true.
If you’re interested in a more detailed explanation of why Recast is Bayesian, we’ve written about it here.
TL;DR – Key Takeaways
- The replication crisis reveals that many published research findings fail to replicate due to flaws in traditional statistical methods, particularly the over-reliance on P-values.
- Generating statistically significant results is often too easy, leading to unreliable findings due to practices like running multiple experiments or analyzing data in various ways.
- Academic incentives drive researchers to prioritize publishable results, contributing to the “file drawer problem” where non-significant results remain unpublished.
- The statistical community is responding by re-evaluating best practices, emphasizing pre-registration, confidence intervals, effect sizes, and Bayesian methods for more reliable analyses.
