It has been 8 years since Simmons and colleagues’ article “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant” came out. This paper opened the eyes of many researchers in psychology and adjacent fields that we are in a deep replication crisis. Since then, Open Science (OS) practices – such as sharing data, code, and preregistering studies – have become more and more popular for facilitating transparent and reproducible research.
While recent years have seen a huge spread in OS practices, they have not yet reached all institutes, universities, and researchers. To promote the use of OS practices, many institutes have started initiatives. In this article, we are outlining how we started such an OS initiative at our institute, the Max Planck Institute for Human Cognitive and Brain Sciences (MPI-CBS) in Leipzig, Germany, and provide you with some tips, tricks, and resources to start your own.
Bottom-up or top-down?
Initiatives can be organized bottom-up – by early career researchers – or top-down – by professors and directors of the institute, or even funding agencies (see e.g. a recent position paper by Dutch funding agencies). As OS initiatives go, there have been instances of both. The LMU Open Science Center in Munich and the Montreal Neurological Institute are two examples of OS initiatives that have been largely organized top-down. Our OS initiative at MPI-CBS started out more bottom-up – at coffee machines and in institute hallways: “Did I hear you say ‘preregistration’? Let’s talk!”. It soon became clear that many researchers at the PhD and postdoc level have a growing interest in implementing OS practices in their projects and that we are not alone in our struggles with it. What good practices to go for? What platforms to use? How to do it? And how to convince your collaborators and supervisors that it’s worth the time and effort? We discussed our desire to further educate ourselves and our colleagues with the directors of our institute. With their support, our institute’s OS initiative was established. Although the initiative is indeed led by early career researchers, it should not be underestimated how helpful top-down support can be for motivating ourselves as well as others to spend valuable time on improving our scientific practices. Aligning the incentives of good scientific practice involves top-down decisions and encouragement. Therefore, finding allies higher up the academic ladder is very much recommended.
What does Open Science mean to you?
OS means many things to many people, as Tal Yarkoni has pointed out in his blog post, “I hate Open Science”. We were inspired by this article and had a discussion about what aspects of OS we want to work on with our institute’s initiative. In line with what Tal Yarkoni points out, each of us had a slightly different idea of what OS meant to them. For the sake of our initiative, we decided on a fluid definition of OS that is open to the interests of new people who join the team. So far, our main focus has been on reproducibility, trying to take small steps to increase the use of preregistrations, open data, and transparent statistics at our institute. We’re not trying to push for all steps (and hurdles) to be taken at once. Instead, we’re informing researchers about the different options available to them and helping them apply those that they find relevant to their own research. We’ve also started to work on topics of accessibility, diversity, and changing incentive structures, but these are works in progress. We recommend you to define early on in your initiative what you mean with Open Science, but remain open to change. We only had this conversation a couple of weeks after our kick-off event (see below) with new people joining the team. It was incredibly helpful in shaping and communicating the directions we wanted to head towards. Remaining open to the interests of potentially new members will also help you grow. After all, it is the motivation of its members that drives the success of your bottom-up initiative.
Kicking off in style
As a kickoff event, we organized an OS day at our institute with the aim to bring together interested researchers and start a discussion at our institute. While we’re aiming to open future events to a larger community (which is, of course, in the spirit of OS), we kept the audience of this kick-off event largely internal, with most people coming from our institute and some from other local universities and institutes. The reason for this was to get people at our institute excited about OS and to join us in kick-starting the initiative.
We invited several external speakers to give talks about different OS practices, including preregistrations, registered reports, open data, and open access. Luckily, researchers interested in OS in Germany and the Netherlands have organized networks including lists of speakers on different OS topics. This list has been an invaluable resource for us to find local speakers who might be interested in giving talks at our kickoff event. Getting the ‘big names’ in the field to travel to your institution and give a talk can be tricky given the number of requests that they get. However, everyone we asked – even the really ‘big names’ – was really positive about our event, and if they declined due to time constraints, suggested other possible speakers. One of the great things about the OS movement is that a lot of the people who are big in the movement are early career researchers. In the end, we had a nice representation of junior and more senior people and hugely benefited from the mixture of expertise.
One of the highlights was the talk by Prof. Felix Schönbrodt, who heads the LMU Open Science Center in Munich. In a talk entitled “How to p-hack like the pros”, he instructed us in the art of manipulating data containing absolutely no true underlying effect until a significant result (p<.05) was obtained. His clever demonstration – during which he not once broke character – demonstrated the need to transparently report methods, publish data and analysis scripts alongside manuscripts, and preregister analyses. To see how scarily easy it is to get to that famous p<.05, try out Felix’ p-hacking shineyapp.
Keeping momentum
The OS day started a discussion that we continued over pizza and beer in the evening, and, more importantly, that we are aiming to maintain. Keeping the momentum is our main priority now. To do so, we are hosting follow-up workshops on specific OS practices in smaller teams. We meet every month to update each other about current affairs and welcome new team members anytime. The initiative is set up to accommodate an ever-changing team, so that the initiative can continue to exist in the academic climate of temporary contracts and limited resources to work on anything else but science. Since we started, our team has continuously changed, with some people being there from the beginning, some focusing on non-OS related topics after the kick-off event, and some joining after the event. We’ve had continuous support from some senior researchers at our institute, but the main drive to push things forward has come from early career researchers. We’ve also had conversations with early career and senior researchers at the institute who are more critical of adopting new OS practices. Critical comments and questions have sparked interesting discussions about the advantages and disadvantages of OS practices. So far we have found that a lot of the criticism of OS practices comes from not fully understanding them. That is why we mainly focus on organizing educational workshops, lectures, and on providing relevant information in an accessible way.
We also make sure to have an active presence within the institute. For example, by taking the opportunity to present our aims and next steps in an institute-wide meeting. In this meeting, we presented some very concrete steps we would like to take, such as signing the San Francisco Declaration on Research Assessment (SF DORA), implementing BIDS to facilitate data sharing, improving statistical skills of researchers, and including preregistrations in our standard workflow. Finally, we try to be as visible as we can and share our experiences and resources. You can follow what we do on our website and on Twitter.
We hope that our story can pass on some of the momentum and help you set up an OS initiative of your own. If it does, do have a look at the fantastic tips by the Utrecht Open Science Initiative and the FIT4RRI initiative. Together we can create the critical mass necessary for the much-needed changes that make research more open, transparent and reproducible.
About the Authors:
Mariella Paul is a doctoral researcher at the Max Planck Institute for Human Cognitive and Brain Sciences and the Berlin School of Mind and Brain. Follow her on twitter @mariella_n_paul.
Lieneke Janssen is a postdoc at Leipzig University Medical Center and Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany. Follow her on Twitter @lkjanssen.
