The Replication Crisis: Researchers Reveal a Hidden Universe of Uncertainty

Wide variation in results from 73 teams testing the same hypothesis with the same data. The distribution of estimated AMEs across all converged models (n=1,253) includes negative (yellow; in the direction predicted by the given hypothesis the teams were testing), non-zero (grey), or positive (blue) results. using a 95% CI. MEAs are xy-normalized. The y-axis contains two scale jumps at ±0.05. The numbers inside the circles represent the percentages of the distribution of each result inversely weighted by the number of models per team. Credit: Proceedings of the National Academy of Sciences (2022). DOI: 10.1073/pnas.2203150119

The Department of Social Sciences of the University of Luxembourg contributed to a large-scale replication study aimed at understanding the role of decisions made by scientists during the research process. Posted in the Proceedings of the National Academy of Sciencesthe study highlights the importance of open science and collaboration between scientists.

During the COVID 19 pandemic, a certain crisis of credibility of science has manifested itself. It has also become apparent that the public and experts are struggling to deal with scientific uncertainty.

Part of this mix is ​​the “replication crisis”: in several scientific fields, studies have repeatedly failed to replicate a significant proportion of previously published results. While this may be due to errors, publication pressures and biases, or even questionable research practices, the failure of replications undermines the role of science as a reliable producer of knowledge.

A study by Nate Breznau and his colleagues investigated the role of decisions made by scientists during the research process. Andreas Hadjar and Robin Samuel from the Department of Social Sciences of the Faculty of Humanities, Education and Social Sciences of the University of Luxembourg contributed to this large-scale replication study, which involved 161 researchers in 73 teams of the whole world.

One hypothesis, one set of data, different results

The aim of the replication was to test the hypothesis that an increase in immigration reduces public support for government provision of social measures.

Interestingly, the 1,253 statistical models provided by the research teams produced very different results, even though they used the same data. The analytical choices, the expertise of the researchers and their expectations could not explain these enormous differences in the results.

The authors concluded that a “hidden universe of uncertainty remains”. Therefore, “scientists, especially those who work with the complexities of human societies and behaviors, should exercise humility and strive to better account for the uncertainty of their work.” The authors further emphasize the potential and importance of transparent and collaborative research.