Science Club Events - 20 November 2013

Science Club Meetings - Michaelmas 2013

20 November 2013 - 1pm - Wallace 108 (note room change)

Journal-Club Discussion on 

"The Assessment of Science"

led by Dr. Luca Börger 

(Swansea University, UK)

From: http://wps.prenhall.com/wps/media/objects/3310/3390101/imag0105/AAAUATB0.JPG

It is probably safe to say that measuring/evaluating science and scientific production has never been so contentious as well as influential in determining the fate of individual scientists and research organisations. And never has been there so much money been spent on science evaluation exercises such as the current Research Evaluation Framework (REF) 2014 in the UK.

It may hence come as a surprise that quantitative assessments of the reliability of evaluating research are mostly, if not entirely, lacking! The recent paper by Eyre-Walker and Stoletzki in PLoS Biology is therefore a very timely contribution, as it provides a novel and interesting way to analyse large datasets with subjective post-publication assessments of scientific publications made by experts. Specifically, a dataset by the Wellcome Trust and the Faculty of 1000 (F1000) database. These datasets cover a large range of Impact Factor (IF) values:

Figure 1. The distribution of the impact factor in the two datasets.

doi:10.1371/journal.pbio.1001675.g001

Using these datasets the authors investigated three methods of assessing the merit of a scientific publication: subjective post-publication peer review, the number of citations a paper accrues, and the IF. Their results are quite interesting:

1. Scientists show a good level of agreement between assessors, yet appear to be poor at judging scientific merit and the likely impact of a paper. Furthermore, their judgment is strongly influenced by the journal in which the paper is published.

Figure 2. The correlation between assessor score and impact factor in the two datasets.

2. The number of citations a paper accumulates is a poor measure of merit.

3. Notwithstanding the problems associated with the number of citations, the impact factor, of the journal in which a paper is published, may be the best measure of scientific merit currently available.


This paper has attracted considerable debate, starting with an editorial commentary in PLos Biology itself, by Eisen et al. "Expert Failure: Re-evaluating Research Assessment".

Also, F1000 stepped in with an editorial "Isn’t all expert peer review subjective?".


Hence, I think there are several issues to discuss and I hope to see many of you tomorrow!