By Steven Burgess (@sjb015)
When will you get round to that paper that you keep meaning to read – but only after you have done one more experiment? How do you learn to review papers as an early-career researcher? and why won’t anyone give you feedback on your latest preprint?
To address some of these concerns, several eLife Ambassadors have been running preprint journal clubs throughout 2018. This was inspired in part by PREreview, a platform co-founded by eLife Ambassadors Daniela Saderi and Sam Hindle, to utilize the hours spent by journal clubs analyzing manuscripts and provide useful feedback to authors of preprints. It’s been a fantastic experience and I highly recommend giving it a go. There are a host of documents to help get you started here.
I’m a postdoc and have been running a preprint journal club focused on Plant Physiology at the University of Illinois at Urbana-Champaign, below are a few things I’ve learned along the way that might be helpful.
1. Make it interactive
If you are like me, there are few things worse than sitting through a turgid slide presentation about a paper while everyone sits in silence. Such was my first experience with the ‘traditional’ journal club and it made me hate them for a long time. I was delighted when my old lab decided to discontinue the practice – however, in hindsight, this was a shame, journal clubs are a great way to hone your analytical skills and stay on top of the literature.
To make a journal club interesting you really need to make it interactive. My advice would be to look up some in-class activities and try implementing them in your club. We have used the think-pair-share, affinity grouping, and group discussion approaches with success – you can also try enticing incentives to get people to join: eLife Ambassador Ed Emmott says “a previous institution tried straightforward bribery – there was free pizza provided if you attended”.
Needless to say, I’m now a born again journal club enthusiast and all it took was a tweak to the format.
Source: The culture of pedagogy (https://www.cultofpedagogy.com/think-pair-share/)
2. Setting a schedule
I receive email alerts of the latest papers posted to bioRxiv and keep tabs on potentially interesting papers on a spreadsheet, Twitter is also a great resource with a number of bots that highlight the latest advances in specific fields (e.g. @ribosome_papers, @biorxiv_sysbio). We run a monthly club (it seems to give people enough of a break between sessions) and a couple of weeks before I will suggest a preprint from the list. After consultation with regular attendees, our meeting is set for the first Monday of the month. Having a regular time slot helps people plan in advance and avoids confusion around scheduling.
3. Read the paper in advance (if possible)
It is really important to read the paper in advance if to want to have a vibrant discussion, otherwise, comments tend to be limited to the superficial and participants don’t really know what they are reviewing. If you are too pressed for time, an alternate suggestion is to focus on one subsection of the paper. Allot 10-15 mins at the start of your journal club for participants to read that section and make notes for the rest of the discussion.
3. Reach out to experts
One thing that I think has been particularly successful was inviting specific members of our department who are real experts in the technique used in the preprint. Generally, people are receptive to direct invites. It really helps raise the quality of critique and we have learned a lot about a wide range of topics by doing this. Experts are much more likely to be familiar with relevant literature and be able to point out technical shortcomings as well as identifying parts which are particularly well done.
4. Assign a discussion leader
It was useful to appoint someone to chair the journal club. Their task is to keep things on time, make sure everyone is involved and listened too while trying to ensure one person doesn’t dominate. The chair can also be the person to collate all people’s points and write them up to a final report (if you wish to create one).
Ed points out “this is particularly useful if trying to give feedback to the authors. Getting everyone to give feedback is tricky unless done in the session itself”.
5. Get feedback
At the end of the first 3 sessions we ran, I asked everyone to fill out a questionnaire asking things such as how useful this was, how we can improve it, what they particularly enjoyed. It really helped us refine the format of the journal club. For example, we have tried various formats and means of collecting people’s opinions including hypothes.is, etherpad and good old-fashioned paper sticky notes. Sticky notes actually worked best for us, but I know others have had a lot of success with the alternatives.
Ed comments “we tried a couple of methods including a shared google doc/etherpad to encourage people to put down points. You really want to try and do this within the session time. When we tried asking people for feedback after the journal club it didn’t work.”
The key is to find out what works best for you, it will be highly dependent on the size of your group, career stage and time constraints.
6. Set on a structure
The first journal clubs we ran had participants with a mixture of experiences, from undergraduates to senior postdocs. Based on feedback it became obvious many wanted to understand: what makes a good paper? With this in mind, we developed a rubric to help assess the quality of a paper (you can download it here). The rubric was then used in conjunction with think-pair-share and group discussions to structure the journal club. In our experience, this worked really well as a format when the aim of the sessions was educational.
The PREreview team have also developed a template with questions to consider and it proved successful in the live journal clubs we ran back in October. The requirements will be unique to your group, Ed shares his experience “The one we had [here] was basically our lab so similar mix from undergrad to postdoc/PI. As we already have a good number of meetings/seminars each week we wanted to have a fixed time – so 1h max.”
“The rubric is a good idea, and we had our own included in the etherpad we were using for shared notes which helped encourage note taking.” agrees Sarvenaz Sarabipour, who runs a journal club at John Hopkins “many trainees in the journal club I run … do not know how to assess a paper/manuscript because they are not seasoned researchers…. Having a rubric and performing a few JCs with it will train them to have an eye for good science, distinguish significance of results, questions goodness of the experiments, theoretical set up, criticize hypothesis etc.”
7. Write up a report
“The ability to give feedback on ‘live’ manuscripts is a particular strength of ppjc’s” says Ed, and writing a report has also been a key aspect of our preprint journal club at UIUC. We decided to post our reviews publically on the PREreview platform, but if you are not comfortable doing this you can also email it directly to the author.
Although writing the report takes a little extra effort it is well worth it. The reason for this is that open reviewing can help bring about a culture change in scientific publishing: many of my colleagues learned how to assess papers either through receiving reviews of their own or during combative lab meetings. The temptation all too often is to tear a paper apart or suggest experiments one would like to see rather than what is necessary. Having to write a report that will be public, and putting your name on it encourages a more constructive approach to providing positive feedback.
Secondly, by encouraging the practice of open commenting and providing feedback to authors it is hoped we can use our knowledge and time for the mutual benefit of authors, who can choose whether or not to modify their manuscripts, potentially reducing the number of revisions required after formal peer review at a journal.
Finally, in our experience, the response reviews are generally really positive, authors tend to appreciate your efforts. Further, as one of the undergraduates attending our journal club remarked it can be a great feeling for the participants too, exclaiming “wow, they are actually reading our comments!”
Many thanks to Ed Emmott, Sarvenaz Sarabipour, Kora Korzec, Zach Hensel and Daniela Saderi for there input in creating this blog. Disclaimer: I am an Ambassador for ASAPbio which promotes the use of preprints in the life sciences.