The Transparent Psi Project (TPP), A Consensus-Based Replication of Bem 2011 Experiment 1 

Introduction: 

Growing evidence for a systematic positive bias in the published research reports in various scientific fields had led to a ‘crisis of confidence’. Studies testing controversial hypotheses, such as studies of extrasensory perception (ESP), suffer even more of the burden of the confidence crisis because the lack of trust promotes risk-aversion and the maintenance of the status quo. We need to develop objective indicators of trustworthy, reliable research studies independent of the hypothesis that is tested. 

Methods: 

This project aims to develop methodological tools that facilitate highly credible and rigorous research. Furthermore, we aim to conduct a multi-site, fully transparent replication of Bem’s (2011) Experiment 1 using these credibility-enhancing methodologies. 

We have conducted a multi-site, fully transparent replication of Bem’s (2011) Experiment 1. We have developed a study protocol for this replication effort via a Consensus Design Process. During this process more than twenty experts on the field (both proponents and opponents of the original ESP hypothesis) contributed to finalizing the protocol.  

The protocol includes a comprehensive toolkit of safeguards against researcher biases and mistakes that are often thought of as the primary cause for the abundance of non-replicable findings in psychology and biomedicine. The safeguards include radical transparency about the whole research pipeline via Born Open Data, Direct Data Deposition, Real-time Research Reports, automation, trusted third party oversight, tamper evident seals on data and software, documented training, and lab logs. 

Results: 

We have conducted a pilot study involving two research sites and one hundred eighty-four participants. This pilot study demonstrated the feasibility of our approach, and the adequacy of the consensus-derived study protocol. During this pilot investigation we observed 49.49% successful guesses (99.5% CI = 47%, 51.9%; posterior mode = 50.6%, posterior 90% HDI = 49.4%, 51.7%). 

Following the acceptance of our research protocol as a Stage 1 registered report in Royal Society Open Science, we started the main study. We have collected data from 2097 participants at nine research sites worldwide. The data were more consistent with the model assuming that humans’ guesses about the future, randomly determined, position of a target do not have a higher than chance success rate, rather than the model assuming that they do.  

In the main study so far we observed a total of 49.87% successful guesses (99.5% CI = 49.1%, 50.6%; posterior mode = 50%, posterior 90% HDI = 49.5%, 50.3%). Observing this percentage of successful guesses is 76 times more likely if the guesses are successful at random than if they have a better than chance success rate.  The results proved to be robust to different statistical approaches, increasing our confidence in our inference. Taken at face value, the data provide strong evidence that the probability of successfully guessing later computer-generated random events is not higher than chance level contrary to what was previously reported by Bem (2011) and others (Bem et al., 2016). 

Discussion: 

The findings of this study are not consistent with the predictions of the ESP model in this particular paradigm. The methodology of the present study reasonably addressed all alternative explanations stemming from deficiencies in modal research practice (LeBel & Peters, 2011) that we were able to identify, with extensive input from other researchers. The failure to replicate previous positive findings with this strict methodology indicates that it is likely that the overall positive effect in the literature might be the result of recognized methodological biases rather than ESP. However, the occurrence of ESP effects could depend on some unrecognized moderating variables that were not adequately controlled in this study, or ESP could be very rare or extremely small, and thus undetectable with this study design. Nevertheless, even if ESP would exist, our findings strongly indicate that this particular paradigm, utilized in the way we did, is unlikely to yield evidence for its existence. 

At the submission of this abstract we were 15 participants short of achieving our target sample size. We will reach our sample size target by the time of the conference and report on the final findings. During the presentation we will also discuss our experiences with using the credibility enhancing toolkit in this project, and the feasibility and acceptability of these approaches to other researchers.