19:36
The Replication Crisis: Why Studies Fail and How Psychology Is Fixing Itself | PSYC1040 Week 13A
SocialNeuro
Overview
This video explains the replication crisis in psychology, where many published findings failed to be reproduced by other researchers. It details how issues like low statistical power, researcher degrees of freedom (p-hacking and harking), publication bias, and the winner's curse contribute to this problem. The lecture highlights that this is a crisis of systems, not dishonesty, and discusses the significant reforms psychology is implementing, such as preregistration, data sharing, registered reports, and larger sample sizes, to improve the reliability of research.
How was this?
Save this permanently with flashcards, quizzes, and AI chat
Chapters
- Daryl Bem published a paper in 2011 reporting statistically significant evidence of precognition, which passed peer review.
- Subsequent attempts by other labs to replicate Bem's findings using identical methods failed to produce similar results.
- This situation raised questions about how a seemingly valid finding could be false and highlighted issues within the scientific process.
This case serves as a concrete, early example that exposed potential flaws in how psychological research was conducted and validated, prompting a re-evaluation of scientific standards.
Daryl Bem's 2011 paper, 'Feeling the Future,' which claimed to show humans could predict future events, but failed replication.
- A large-scale project by the Open Science Collaboration in 2015 attempted to replicate 100 psychological findings.
- Only about 39% of these studies successfully replicated, indicating a widespread issue with the reliability of published research.
- This demonstrated that the traditional peer-review system was not sufficient to guarantee the accuracy of findings.
This large-scale replication effort provided empirical evidence of the extent of the problem, forcing the field to confront the unreliability of many previously accepted findings.
The Open Science Collaboration's 2015 study, which replicated 100 psychological findings and found that only 39% were reproducible.
- Low statistical power, often due to small sample sizes, means studies have a high chance of missing a real effect.
- Researcher degrees of freedom, or 'p-hacking,' refers to making numerous analytical choices (e.g., data exclusions, variable selections) that can nudge results toward statistical significance without outright fraud.
- Harking (Hypothesizing After Results are Known) involves presenting exploratory findings as if they were pre-planned hypotheses, masking the exploratory nature of the discovery.
Understanding these factors is crucial because they explain how researchers, without intending to deceive, can inadvertently produce and publish false positive results.
Deciding to exclude participants who seemed distracted or running an analysis with and without a covariate, which can alter the p-value.
- Publication bias, also known as the 'file drawer problem,' means that studies with significant, positive results are more likely to be published than those with null or negative results.
- This creates a skewed view of the evidence, making effects appear stronger or more consistent than they are.
- The 'winner's curse' describes how the first studies to find an effect, often with small sample sizes, tend to overestimate the true effect size due to random chance.
These systemic biases distort the scientific literature, leading to an inflated understanding of effect sizes and a false sense of certainty about findings.
A study finding a strong effect is published, while 19 other studies with similar designs find no effect and are not published, creating a misleading impression of the effect's prevalence.
- Preregistration requires researchers to specify hypotheses and analysis plans before data collection, making p-hacking and harking more difficult.
- Open science practices, such as sharing raw data and study materials, allow for verification and easier replication.
- Registered reports involve peer review of study designs and analysis plans before data collection, with a commitment to publish results regardless of their significance.
- A shift towards larger sample sizes increases statistical power and leads to more reliable estimates of effect sizes.
These reforms are actively addressing the systemic issues that led to the replication crisis, aiming to increase the trustworthiness and reproducibility of psychological research.
Specifying exactly which statistical tests will be used and registering this plan publicly before conducting the experiment.
- The replication crisis is not a sign that psychology is not a science, but rather evidence of its self-correcting nature.
- The field's ability to identify its own problems and implement reforms demonstrates a commitment to scientific rigor.
- Failed replications don't automatically invalidate original findings but signal a need for more careful investigation and interpretation.
Recognizing that science is a process of continuous improvement, rather than a collection of infallible truths, allows for a more accurate and nuanced understanding of research findings.
A study with p=0.049 failing to replicate with a larger sample (p=0.21) suggests the original finding was likely a false positive or an overestimation due to the winner's curse, prompting caution.
Key takeaways
- Many published psychological findings are not reproducible, a phenomenon known as the replication crisis.
- Low statistical power, p-hacking, harking, publication bias, and the winner's curse are key systemic factors contributing to false positives in research.
- The replication crisis is a crisis of scientific systems, not a crisis of individual dishonesty.
- Preregistration, open data, registered reports, and larger sample sizes are crucial reforms improving the reliability of psychological research.
- The ability of a scientific field to identify and correct its own errors, as psychology is doing, is a sign of its health and rigor.
- A single statistically significant finding, especially one that fails to replicate, should be interpreted with caution.
Key terms
Replication CrisisStatistical PowerResearcher Degrees of FreedomP-hackingHarkingPublication BiasFile Drawer ProblemWinner's CursePreregistrationRegistered ReportsOpen Science
Test your understanding
- What is the 'replication crisis' in psychology, and what are its main causes?
- How do researcher degrees of freedom, such as p-hacking and harking, contribute to the publication of false findings?
- Explain the concepts of publication bias and the winner's curse, and how they distort the scientific literature.
- What are the key reforms psychology is implementing to address the replication crisis, and how do they work?
- Why is the self-correction of a scientific field, like psychology's response to the replication crisis, considered a sign of scientific strength?