Mass pre-registered replication project: Insights, findings, and opportunities
By Gilad Feldman, Assistant Professor
Psychology Department
The University of Hong Kong
Presented at NTU on 3 April 2019.
Social sciences is in need for large scale replication studies to validate prior findings and theories. In one instance, Prof Gilad shared a case study on the failure to validate Ego Depletion Theory. This was one of the theories that I learnt in PhD OB course just in 2015, before it was disproved in 2016. Prof Gilad also shared how ManyLabs did 97 replications and found that only 36% of all replications were significant, and 83% of all effect sizes are smaller than original. This has serious implications for social science research, as many of the theories that we draw from today may not have empirical support for them.
Prof Gilad shared some ways forward that the social science research can embark on, which include (a) focusing on simplicity rather than having complex models with small sample size, (b) focus on effect size rather than p value significance, (c) always conduct power analysis, (d) adopt a phenomenon focus (evidence based theory), (e) pre-register studies before conducting data collection, (f) allow publications of non-significant results as these are valid findings, and (g) engage in full transparency by sharing all materials, data, and codes. Researchers also need to report all research decisions, including exclusions, conditions, and variables.
One valuable research program is to first conduct a pre-registered meta-analyses of the variables/hypotheses that you have. Follow up on this meta-analyses with a pre-registered replication study with extensions to show how you can validate and extend existing research. Given the slow uptake of publishing replication studies in top tier management journals at the moment, this might be a much better and substantial approach as compared to just replicating existing studies and expecting it to be published.
Prof Gilad is from the Psych/OB field, so the studies he shared were mainly individual level psychology constructs. However, this reminded me of Prof Herman Aguinis, who also presented his work on replicating strategy studies (refer to references below). This is getting me interested in looking at replicating studies in the strategic management field, especially in the area of corporate sustainability. What if some of the key theories we draw from today are not valid? I hope to be able to do a study like this some time down in the future.
Recently, there has been more discourse around the replicability of social science research and the research-practice gap, of how we are talking in ivory towers and nobody ever reading our research papers. Another stream of critique on academia is the chasing publications game in order to obtain promotion and tenure. While I am glad that senior scholars in the field are having this discussion, I am still hoping for a faster pace of change in the academic institution to address some of these long-standing criticisms that are on point. It will require universities and academic journals to have the courage and will to make radical transformations to an institution that has kept with its traditions and precedents.
Aguinis, H., Cascio, W. F., & Ramani, R. S. 2017. Science’s reproducibility and replicability crisis: International business is not immune. Journal of International Business Studies, 48: 653-663.
Bergh, D. D., Sharp, B. M., Aguinis, H., & Li, M. 2017. Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings. Strategic Organization, 15: 423-436.
