[JoPP-Public] JoPP-Public Digest, Vol 3, Issue 5

[Mathieu ONeil]

Hi Johan, all

On 04/15/12, Johan Söderberg  <johan.soderberg at sts.gu.se> wrote:
> Hi everyone,
> 
> >Not clear to me what confusion?
> 
> I had to ask you twice over email, Mathiue, before I figured out just what "3/3" as opposed to "0/3" stand for. The meaning of the numbers are not self-evident to anyone just looking at the site for the first time (and once knowing what the numbers stand for, there is the much larger didactic task of making the reader FEEL the importance of a high or a low signal). Granted, it is not rocket science, but the small amount of energy it takes, I only put into it because I have an "official" role in the journal. As a reader, I would not have bothered, and I wage on that most people would go straight to the article and read it as they are accustomed to from any other peer review process. Thus we will not save the reputation cost of the journal when posting crappy articles.
> 
 
 
Well, we could work on making the meaning of signals more clear and prominent, sure. 


Getting rid of signals would imply a radical turnaround in both the philosophy and practice of the journal which aims to widen the scope of publishable articles. It really is a core part of the process, and once again is absolutely compatible with wider community - reader appreciation.

> 
> 

> 
> 
> > in my opinion a great advantage of the signals is that they exist, whereas comments by readers are...
> 
> I would say exactly the opposite. Comments and rating-systems posted by readers are tried and tested in many places, the reader encountering our website will know exactly what she sees. With the signaling system, we face the uphill battle of trying to educate our readers to read a new rating system while announcing to them that we are a peer review, academic journal (thus readers will expect what they are accustomed to, that the rating has been done for them at a previous stage, through an accept/reject decision by reviewers and editors).
> 


What I meant was: the signals have been produced, they are there, we can use them straight away.
In contrast reader comments may take months or years to materialise in which case we have no metric at all. 
This is a serious disadvantage of relying solely on reader comments in my view.

> 
> 
> 
> >a) not yet technically implemented, and
> 
> This is an issue. But then the question becomes if we should make it a priority to implement the system. If it cannot be done, we should then declare that the feature is being worked on, and will be up soon (the latter, however, looks worse than relasing the journal with the rating system up - taking the risk that there will be no/few comments by readers, as a response to your point "b)").
> 


OK 

> 
> 
> 
> >Having myself been "signaled" in CSPP 1 I can understand the trepidation but reviewers were supportive / fair 
> 
> That can happen. But part of the rationale behind the journal was a discontent with a defunct peer review process in traditional academic journals. In fact, if we succeed with educating the readers about the signaling system, then more power has been given to the two/three reviewers than they normally enjoy. 
> 
Not really, see below. 

> Firstly, because in a normal journal, reviewers' comments and their say on accept/reject are just recomendations, the final decision lies with the editor. 
> 

Any paper submitted to the journal will be published irrespective of the reviewers or editors opinion of it : the decision lies with the author. 
This does not mean that reviewers have no impact - in fact the radical openness (publish original subs, publish reviews)  may constitute a disincentive to publish as some authors may not wish to have it known how much their original submissions needed revision let alone their low signals.


> Secondly, the familiar critique against the Audit Society applies here too, what is lost when a qualitative text is boild down to a few numbers ("is the paper well argued: yes/no"). In the case of readers' comments à la Slashdot, it is assumed that such reduction to numbers (together with the autocracy and arbitrariness it fosters) is evened out by the vast number of participants (wisdom of the crowd, etc.).   
> 


Well, OK, but to compare our readership numbers to Slashdot is, ah, optimistic to say the least. Once again : why not have both (if reader comments are possible?)

> 
> 
> A few months ago, I think it was Nath (?) who cautioned us against getting too experimental with the peer review process, as it will be a challenge anyway to win the confidence of academics and convince them to send their best texts to a tiny journal like ours. We can compensate for beling lightweight in terms of advancing academic careers by having a novel agenda, an activist outlook, a strong community, etc. But in any case, we shouldnt make things harder for ourselves. Experiments with the peer review procee
> 


I think we will attract people because of the quality of review that we can provide and also because people want to support our process.
Look, its not set in stone obviously but if we abandon signals and rely on (non existent for the moment) reader appreciation then who decides what is publsihed - do we revert to editors making the sole decision if a paper is still not great after reviews and revisions? Then if we publish it we have published a not-great article without being clear that this is understood - in the end signals do help to protect our reputation...


cheers
Matthieu 

> 
> 
>    
> 
>  however, we must weight benefits against costs. 
> 
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ourproject.org/pipermail/jopp-public/attachments/20120415/2700ff51/attachment.htm