The Transparent Psi Project (TPP), A Consensus-Based Replication of Bem 2011 Experiment 1 

Course Information

Categories: ,

Tags:

Zoltan Kekecs1,2, Balazs Aczel1, Bence Palfi3, Barnabas Szaszi1, Peter Szecsi1, Mark Zrubka4, Marton Kovacs1, Bence E. Bakos1, Kyra Giran1, Denis Cousineau5, Gabrielle Parent5, Patrizio Tressoldi6, Massimo Grassi6, Arianna Ballati6, Dana Arnold7, Thomas Rhys Evans8, Yuki Yamada9, Jeremy K. Miller10, Huanxu Liu11, Fumiya Yonemitsu11, Dmitrii Dubrov12, Jan Röer13, Marvin Becker13,Roxane Schnepper13, Sera Wiechert4, Linda Do4, Sophie te Marvelde4, Teddy Raichert4, Sophie te Marvelde4, Nele Põldver14, Kairi Kreegipuu14, Raquel Oliveira15, Patrícia Arriaga15 

1, Institute of Psychology, ELTE, Eotvos Lorand University, Budapest 
2, Department of Psychology, Lund University, Lund 
3, School of Psychology, University of Sussex, Sackler Centre for Consciousness Science, University of Sussex, Brighton, Falmer, Brighton 
4, Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands 
5, École de psychologie, University of Ottawa, Ontario, Canada 
6, Dipartimento di Psicologia Generale, Università di Padova, Padova, Italy 
7, Department of Psychology, University of North Florida, Jacksonville, Florida, USA 
8, School of Psychological, Social and Behavioural Sciences, Coventry University, United Kingdom 
9, Faculty of Arts and Science, Kyushu University, Nishi-ku, Fukuoka, Japan 
10, Department of Psychology, Willamette University, Salem, OR, USA 
11, Graduate School of Human-Environment Studies, Kyushu University, Nishi-ku, Fukuoka, Japan   
12, National Research University Higher School of Economics, Moscow, Russian Federation. 
13, Herdecke University, Witten, Germany 
14, University of Tartu, Tartu, Estonia 
15, Instituto Universitário De Lisaboa, Lisabon, Postugal 

Introduction: 

Growing evidence for a systematic positive bias in the published research reports in various scientific fields had led to a ‘crisis of confidence’. Studies testing controversial hypotheses, such as studies of extrasensory perception (ESP), suffer even more of the burden of the confidence crisis because the lack of trust promotes risk-aversion and the maintenance of the status quo. We need to develop objective indicators of trustworthy, reliable research studies independent of the hypothesis that is tested. 

Methods: 

This project aims to develop methodological tools that facilitate highly credible and rigorous research. Furthermore, we aim to conduct a multi-site, fully transparent replication of Bem’s (2011) Experiment 1 using these credibility-enhancing methodologies. 

We have conducted a multi-site, fully transparent replication of Bem’s (2011) Experiment 1. We have developed a study protocol for this replication effort via a Consensus Design Process. During this process more than twenty experts on the field (both proponents and opponents of the original ESP hypothesis) contributed to finalizing the protocol.  

The protocol includes a comprehensive toolkit of safeguards against researcher biases and mistakes that are often thought of as the primary cause for the abundance of non-replicable findings in psychology and biomedicine. The safeguards include radical transparency about the whole research pipeline via Born Open Data, Direct Data Deposition, Real-time Research Reports, automation, trusted third party oversight, tamper evident seals on data and software, documented training, and lab logs. 

Results: 

We have conducted a pilot study involving two research sites and one hundred eighty-four participants. This pilot study demonstrated the feasibility of our approach, and the adequacy of the consensus-derived study protocol. During this pilot investigation we observed 49.49% successful guesses (99.5% CI = 47%, 51.9%; posterior mode = 50.6%, posterior 90% HDI = 49.4%, 51.7%). 

Following the acceptance of our research protocol as a Stage 1 registered report in Royal Society Open Science, we started the main study. We have collected data from 2097 participants at nine research sites worldwide. The data were more consistent with the model assuming that humans’ guesses about the future, randomly determined, position of a target do not have a higher than chance success rate, rather than the model assuming that they do.  

In the main study so far we observed a total of 49.87% successful guesses (99.5% CI = 49.1%, 50.6%; posterior mode = 50%, posterior 90% HDI = 49.5%, 50.3%). Observing this percentage of successful guesses is 76 times more likely if the guesses are successful at random than if they have a better than chance success rate.  The results proved to be robust to different statistical approaches, increasing our confidence in our inference. Taken at face value, the data provide strong evidence that the probability of successfully guessing later computer-generated random events is not higher than chance level contrary to what was previously reported by Bem (2011) and others (Bem et al., 2016). 

Discussion: 

The findings of this study are not consistent with the predictions of the ESP model in this particular paradigm. The methodology of the present study reasonably addressed all alternative explanations stemming from deficiencies in modal research practice (LeBel & Peters, 2011) that we were able to identify, with extensive input from other researchers. The failure to replicate previous positive findings with this strict methodology indicates that it is likely that the overall positive effect in the literature might be the result of recognized methodological biases rather than ESP. However, the occurrence of ESP effects could depend on some unrecognized moderating variables that were not adequately controlled in this study, or ESP could be very rare or extremely small, and thus undetectable with this study design. Nevertheless, even if ESP would exist, our findings strongly indicate that this particular paradigm, utilized in the way we did, is unlikely to yield evidence for its existence. 

At the submission of this abstract we were 15 participants short of achieving our target sample size. We will reach our sample size target by the time of the conference and report on the final findings. During the presentation we will also discuss our experiences with using the credibility enhancing toolkit in this project, and the feasibility and acceptability of these approaches to other researchers.  

Bem, D. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425.
https://doi.org/10.1037/a0021524 

Bem, D., Tressoldi, P., Rabeyron, T., & Duggan, M. (2016). Feeling the future: A meta-analysis of 90 experiments on the anomalous anticipation of random future events. F1000Research, 4, 1188.
https://doi.org/10.12688/f1000research.7177.2 

LeBel, E. P., & Peters, K. R. (2011). Fearing the future of empirical psychology: Bem’s (2011) evidence of psi as a case study of deficiencies in modal research practice. Review of General Psychology, 15(4), 371–379.
https://doi.org/10.1037/a0025172 

Course Instructor

Zoltan Kekecs
Zoltan Kekecs Author

Dr. Kekecs is currently working as assistant professor at Eötvös Loránd Tudományegyetem, Institute of Psychology. His research is related to the effectiveness and psychophysiological mechanisms underlying hypnosis used in medicine. He is also developing methods to improve the credibility of research in psychological science. Dr. Kekecs is the treasurer of the Society for Clinical and Experimental Hypnosis, and he is a member of the Data and Methods Committee of the Psychological Science Accelerator.

Please log in to watch the video.

Responses

  1. Hello Zoltan! I wish you success in trying to tackle the crisis of confidence. I was curious to know, did any participants make any claims of ESP abilities prior to the study? Or was this something even considered?

Course Information

Categories: ,

Tags: