[JoPP-Public] JoPP-Public Digest, Vol 3, Issue 5

[Johan Söderberg]

Hi everyone,

>Not clear to me what confusion?

I had to ask you twice over email, Mathiue, before I figured out just what "3/3" as opposed to "0/3" stand for. The meaning of the numbers are not self-evident to anyone just looking at the site for the first time (and once knowing what the numbers stand for, there is the much larger didactic task of making the reader FEEL the importance of a high or a low signal). Granted, it is not rocket science, but the small amount of energy it takes, I only put into it because I have an "official" role in the journal. As a reader, I would not have bothered, and I wage on that most people would go straight to the article and read it as they are accustomed to from any other peer review process. Thus we will not save the reputation cost of the journal when posting crappy articles. 

> in my opinion a great advantage of the signals is that they exist, whereas comments by readers are...

I would say exactly the opposite. Comments and rating-systems posted by readers are tried and tested in many places, the reader encountering our website will know exactly what she sees. With the signaling system, we face the uphill battle of trying to educate our readers to read a new rating system while announcing to them that we are a peer review, academic journal (thus readers will expect what they are accustomed to, that the rating has been done for them at a previous stage, through an accept/reject decision by reviewers and editors).


>a) not yet technically implemented, and

This is an issue. But then the question becomes if we should make it a priority to implement the system. If it cannot be done, we should then declare that the feature is being worked on, and will be up soon (the latter, however, looks worse than relasing the journal with the rating system up - taking the risk that there will be no/few comments by readers, as a response to your point "b)").


>Having myself been "signaled" in CSPP 1 I can understand the trepidation but reviewers were supportive / fair 

That can happen. But part of the rationale behind the journal was a discontent with a defunct peer review process in traditional academic journals. In fact, if we succeed with educating the readers about the signaling system, then more power has been given to the two/three reviewers than they normally enjoy. Firstly, because in a normal journal, reviewers' comments and their say on accept/reject are just recomendations, the final decision lies with the editor. Secondly, the familiar critique against the Audit Society applies here too, what is lost when a qualitative text is boild down to a few numbers ("is the paper well argued: yes/no"). In the case of readers' comments à la Slashdot, it is assumed that such reduction to numbers (together with the autocracy and arbitrariness it fosters) is evened out by the vast number of participants (wisdom of the crowd, etc.).   

A few months ago, I think it was Nath (?) who cautioned us against getting too experimental with the peer review process, as it will be a challenge anyway to win the confidence of academics and convince them to send their best texts to a tiny journal like ours. We can compensate for beling lightweight in terms of advancing academic careers by having a novel agenda, an activist outlook, a strong community, etc. But in any case, we shouldnt make things harder for ourselves. Experiments with the peer review procee

   

 however, we must weight benefits against costs.