Improving peer review for JoPP
Hi all Openness undoubtedly has great virtues, but in the case of academic publishing it can also generate some bad side-effects. For this issue of JoPP five papers were sent out for review. Three of the papers will be published with reviews and signals. Two other papers were not great. Reviewers worked long and hard to address shortcomings and make suggestions. One author decided that it would not be possible to make these adjustments though much time kept being added. The other agreed to make changes but then used the time excuse as well as sickness. There is nothing preventing either author from now submitting their much-improved papers to another journal... In my view, we should try to address this obvious waste of reviewer (and editorial) work/energy. One way of addressing this would be to say something like: -the critical reviews (which would have been published alongside the articles) might explain why they did not want to be published by us; -in order to prevent people from pulling out after they have been reviewed we need to improve the quality of their reviews; -this means we need to send out better papers for review; -in turn this means we need to weed out less-than-great papers before they are sent out for review. Since we now have an editorial team of six people the danger of one editor arbitrarily and unfairly deciding that articles are not ready to be sent out for review is lessened. But at the same time this is moving away from the initial spirit of the project (publish everything with signals). (This reminds me that Nate suggested to me a while back that a summary of signals need to feature much more prominently alongside research papers to make sure readers realise what we think about the quality of published papers...) So it seems clear that something needs to be done (I have already removed the reference to "list / community" feedback on the peer review section of the site as that never happened). The question is to what extent this is formalised...? cheers Mathieu -- **
Hi Mathieu and all, On 03/26/2012 04:17 PM, Mathieu ONeil wrote:
Openness undoubtedly has great virtues, but in the case of academic publishing it can also generate some bad side-effects.
For this issue of JoPP five papers were sent out for review. Three of the papers will be published with reviews and signals. Two other papers were not great. Reviewers worked long and hard to address shortcomings and make suggestions.
One author decided that it would not be possible to make these adjustments though much time kept being added.
The other agreed to make changes but then used the time excuse as well as sickness.
There is nothing preventing either author from now submitting their much-improved papers to another journal...
In my view, we should try to address this obvious waste of reviewer (and editorial) work/energy.
hmm, isn't this a problem of being (maybe) not open enough instead of being too open? In the experience from my own academic this, this is a quite possible scenario in the traditional peer review process: reviewers send criticism and suggestions, the author might then revise the paper and send back a revised version, or submit the revised version elsewhere. Especially if a paper is re-submitted by multiple journals (after being refused -- with reviewer feedback -- by each of them), it would cause reviewers a lot of work. (Say if there are 3 reviewers per paper and you submit it sequentially to 4 journals, you would already occupy a dozen reviewers, while none of them would benefit of the work already done by others, since they don't know about it.) Also, if you re-submit a text sufficiently often, it becomes more and more likely to be accepted somewhere by pure chance, almost regardless of the quality of the paper, I would presume. The only chance to avoid that would be more openness, not less, i.e. publishing all versions of a paper from the first submitted one (or, at least, the last negotiated version of each paper), without allowing the authors to pull out. Not sure if we want to go this way, but blaming "openness" for the shortcomings of the current approach strikes my as definitively wrong. Best regards Christian -- |------- Dr. Christian Siefkes ------- [email protected] ------- | Homepage: http://www.siefkes.net/ | Blog: http://www.keimform.de/ | Peer Production Everywhere: http://peerconomy.org/wiki/ |---------------------------------- OpenPGP Key ID: 0x346452D8 -- UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things. -- Doug Gwyn
participants (2)
-
Christian Siefkes -
Mathieu ONeil