Message 00237 [Homepage] [Navigation]
Thread: joxT00189 Message: 12/77 L4 [In date index] [In thread index]
[First in Thread] [Last in Thread] [Date Next] [Date Prev]
[Next in Thread] [Prev in Thread] [Next Thread] [Prev Thread]

Re: [jox] Cutting the Knot



Dear Mathieu and colleagues:

Regarding the questions that Mathieu poses in this long but very interesting e-mail
I suggest the following:

--Regarding Categories--
As a general desiderata:
     The number of categories for a multi-category rating should be small,
     very clear and universal (every paper could be assessed in relation to
     the category).

This is a call to minimise the temptation to expand categories.


Then, more specifically,
1. Objective should be objective -- what is activist, academic, or theoretical can be subjective for a particular instance. I suggest that 'prospective' (based upon developments that have not yet occurred) and 'formalised' (based upon formal logic or mathematical technique) are objective. Native English is ok but perhaps should be an author rather than a reviewer category. Empirical might be further defined as 'based upon data or experience' to avoid confusion. 2. Subjective should be meaningful for the audience -- what constitutes 'rigour' other than the 'utilisation of formal logic or mathematics' may be so subjective as to be useless to the audience while the phrase in quotes here is an objective
   criterion.
3. To my taste, congratulations is too generic and again reveals more about the reviewer (their tolerance and appreciation of the effort required to write a paper at all for example) than the paper. It is a category that suggests an additional element is needed and would be better labelled 'commendations' which would
   specifically evoke a listing of what is to be commended.
I do think that objective, subjective, additional notes is a useful thematic division
for the grouping of categories.

--Regarding Open or Limited Discussion--
I think that this is a crucial choice for the community. I believe that an open forum will discourage reviewers for several reasons, the main ones are: a) except for those individuals who routinely express their opinions to any who will listen, the public exposure of messages directed to the author that have critical content will have the effect of watering down the candour of comments or discouraging them altogether, particularly if this is done for papers for which a decision has not been reached (this is a general problem with any opening of the editorial process)
and b) reviewing will risk becoming a forum and hence potentially
indeterminate in outcome (see comments below on editorial process).
My conclusion is that papers that are rejected should not have the reviews
published (see below suggestion -- this would apply to papers with two rejects or one reject seconded by a third reviewers) although the author would of course have access to all reviews). The corollary of this line of argument is a web-based and 'closed' (to all except editorial board, authors, and reviewers) review process
prior to publication.

--Regarding Editorial Process--
I do not think that the process as outlined scales very well. If one were speaking of a limited number of submissions that were initially well-executed in most respects with a few missing or contested bits, it might be possible to employ this process to achieve a periodic 'issue' of a journal (time delimited publication). If the aim is to create an archive which grows incrementally (not time delimited publication) then the 'time in process' may be less severe. The process as outlined seems likely to sacrifice quality in favour of closure. Here is why I reach that conclusion:

In my role as a reviewer, I rarely see papers that are sufficiently polished to be published without significant revision. The process as outlined suggests that I make my comments for revision and then no longer have any responsibility for the final
decision.

Viewing this process (as outlined with a change in reviewers after a revise and
resubmit), as a reviewer, I have three choices:
1) consistently reject all papers that I think need significant revision in the
   interest
   of the quality of the journal (i.e. the preponderance of papers),
2) make my comments for revision as my contribution to the community and ignore
   the subsequent process,
3) consistently accept papers that I think have some merit (even if the execution is flawed in various ways). Executing 1) will lead to my exclusion as a reviewer in fairly short order or, if others are similarly minded (i.e. also believe that most papers are under-polished), a thin journal, 2) is problematic for the author because they will then draw additional reviewers with their own (different) choices for revisions and thus discourages author submissions.
Thus 3) is the likely outcome of the process as outlined.

Alternatively if revisions must be executed to satisfy the initial reviewers (i.e. review of revised manuscripts by initial reviewers) the objections raised concerning the 'closed' (elitist) tendencies of the peer review process come to the fore. In other words, this solves the problem identified for 2) but reproduces the current system with the only difference being that the community of reviewers becomes more
like a cadre or community capable of developing collective norms.

It may be that a cadre of reviewers who can effectively co-ordinate and reach consensus will emerge. I suggest that this cadre will consist of people who have quite a bit of time to engage in the process and thus it is an open (to be determined by experience) question whether a) there is a sufficient number of such people to create a reviewer community of adequate size and b) that achieving coherence will reproduce some of the features of 'closed' communities which are being contested, i.e. the norms will again be seen as discouraging by authors.

The only way out of this box that I can see is that revise and resubmit constitutes a commitment to publish. In this case, I think reviewers should be given one more round of comments regarding the revised papers. Adopting this proposal would make it possible to cut out the consensus formation part of the process. I would
suggest:
1.  Two rejects means reject.
2. One reject and one revise and resubmit leads to a third reviewer being appointed. If this reviewer votes reject, then the paper is rejected, otherwise the paper is published after revisions in response to those suggested (by first round reviewers -- a reviewer might vote to reject but have ability to comment on the basis of 'if this paper were published it would be desirable to...' -- and possibly the third 3. Two revise and resubmit reviews is a commitment to publish after revisions have been made with the revised submission subject to further published comments
   by the reviewers
4. This last category would also be applicable to one accept and one revise and resubmit or two accepts (one might still wish to suggest minor corrections)

This streamlines the process and makes it more determinate and less costly in terms of negotiation potentially improving the reviewer recruitment process. It may reduce the quality of the journal and discourage some reviewers but it will allow more heterodox work to appear. It is an open (to be determined by experience) question how reviewers (which are not easy to recruit) will view the prospects of their reviews
being published for papers that are eventually published.

Where I have not made comments, I agree with the elements of the proposal
Mathieu has made.

Hope this helps...

Best regards, Ed Steinmueller


Mathieu ONeil wrote:
Hi everyone

Well, it's taken a bit of a while, but I finally got around to attempt to
square the circle / find the North West Passage / cut the Gordian
knot, in short: address the issue which caused so much heat last
December, our peer review system. I have come up with a solution
which I hope will satisfy most. Let me say from the start that it
involves some kind of post-publication public assessment of
submissions – I think there is no getting around that. Anyone who
is completely against this can decide if they want to continue with
the project or not, but hear me out first, yall.

Menu:

1-summary of issue
2-proposed solution
3-decision: categories
4-decision: review discussion system
5-time-frame for decisions
6-references
7-CSPP peer review process: main stages

1-summary of issue

This is by no means everything that was said, I tried to get to the main
nut, apologies if I forgot anyone.
The central issue we wrestled with was whether to follow the
recommendation laid out in Whitworth and Friedman's paper [1] to
accept significantly more papers, and rate these according to
different categories. There was a nice summary by Andreas Wittel who
said it was a question of whether to have “pre-selection”
(through accept / reject) or “post-selection” (through rankings).

[See http://www.oekonux.org/journal/list/archive/msg00200.html]

Brian Whitworth logically argued for ratings:
"A journal that can't be bothered rating its submissions doesn't deserve
to succeed. Equally one that selects the best and leaves the rest is
elitist. There is no easy way between these options, so we suggested
both highly selective reviewing and completely open publishing. The
multi-grade system lets anyone publish but all needn't be rated equal
- though there can be multiple criteria. I guess this goes against
the politically correct idea that we are all equal, but actually some
of us run faster, others cook better and a few of us can actually do
mathematics - so really "equality" is a myth. The real
equality is of opportunity not ability, which is why this approach
lets everyone in who wants to come in."
[See http://www.oekonux.org/journal/list/archive/msg00194.html]

However a number of people (Athina, Biella, Andreas, Felix) expressed
reservations about ratings because of their perceived overly
hierarchical, discriminatory or “school-like” nature; especially,
some did not like the numerical ranking. [See http://www.oekonux.org/journal/list/archive/msg00198.html]

Many points in favour of ratings were made by StefanMn:
[See http://www.oekonux.org/journal/list/archive/msg00211.html]

StefanMn then proposed a choice where authors submitting a proposal could
indicate whether they want a binary model (publish or reject) or a
multi-dimensional rating system:
[See http://www.oekonux.org/journal/list/archive/msg00212.html]

2-proposed solution

There are two main problems with his proposal. First, I think we should be
as consistent as possible in what we present to the world. It would
be weird to have some papers with an appreciation and others without.

Second, if we want to open up the journal selection process and provide rewards to those who do
normally invisible work (i.e. reviewers), in line with Toni Prug's
proposal [2] for a community peer review system (through a list where proposals are
vetted and reviews are released), then by definition we are rejecting
the publish / don't publish model: vetting and orientation occur
upstream, even before an actual full submission. If we agree to work
with authors by reviewing and discussing their proposal it would be
hard to come to the end of that process, and reject. So we will
necessarily publish more than a traditional journal, including papers
that are not written in perfect English (for example), or that are
problematic in some other way. As was said:

Felix Stalder said that “we should publish only papers that we agree are fit for publication” and
StefanMn responded:
“But "fit for publication" is not based on a single reason.
There may be articles which we consider great in many dimensions but
they lack some certain feature. Lack of this feature normally would
make them unacceptable but if we c25an express this lack by a rating
then the credibility of our journal is maintained *and* the article
is published.”

[See http://www.oekonux.org/journal/list/archive/msg00177.html]

This is the essential point, and, upon reflection, I agree with it. And
yet: the fact remains that people are uncomfortable with ratings. I
think it all depends how we do them. In order to protect the
reputation of the journal, we need to alert readers that we are aware
of flaws, but that _we decided to publish anyway_. Hence the need to
“qualify” or “signal” (rather than “rate”) published
submissions. Another thing: I think it is important to distinguish
between appreciations of what a submission clearly “is”
(activist, academic, native-English speaking authored, etc) and
appreciations as to what a submission “is worth” (original,
rigorous, etc): the first appreciations may be described as more
“objective” than the second.
3-decision: categories

So we could have either a simplified choice of rating (outstanding,
excellent, fair) or an even simpler choice (yes, no) as in: this
paper is a grand grassroots testimony (Activist: yes) but the English
is not perfect (Native English: no) or: this paper is
academic-research oriented (Academic: yes) but it is not based on
empirical evidence (Empirical: no); or: this is a utopian fantasy
about how a peer-produced society would produce gastronomical
delights (Theoretical, Activist: yes); etc.

A-For “objective” categories we could have:
-Native English
-Activist
-Academic
-Empirical
-Theoretical
-?...

B-For “subjective” categories we could have:
-Rigour
-Originality
-?...

Or,
we could ask reviewers to indicate whether they want to express:
-Reservations (yes/no) -Congratulations (yes/no) about the submission.

[Thus indicating clearly that this is a personal opinion].

In all cases these “appreciations” (whether Activism, Originality,
Congratulations) would have to be clearly linked to the reviews which
would be released when the submission is published.

Finally the editor should have the right to intervene and add an appreciation
if he or she feels that reviewers have been unduly harsh or
sympathetic.

So, this is the first point to decide: what categories do we have?

4-decision: review discussion system

The other point to be decided concerns the process of discussion of
submissions: should these be held on a restricted mailing list (to be
clear: not the one we are using now, which is open, but one that
would be reserved to reviewers and authors) or on a protected part of
the website?

StefanMn argued for the website option:

“Well, in general I'm a big fan of mailing lists. But in this case I think a web based system would be more useful. I'd suggest to offer potential authors a place where they can propose an article in the way outlined above and reviewers can help the author to write a great article. I think a web page is more useful because it gives every stakeholder a clear structure where the subject is *one* proposed article.” [See: lost the ref, sorry]

It's true that having one (restricted) mailing list where all submissions
are discussed could be messy (though less so if people do not
interfere with titles of emails thereby breaking threads). And it
might be easier to create files that can be used later on in the
website when publishing, I don't know. At the same time I see some
problems with setting up discrete pages for articles: a) authors and
reviewers might in fact benefit from reading discussions on other
articles; b) not sure about this, but there might be complications in
access rights – who can access what article page, etc?: c) finally
the advantage of the list is that you are kept abreast of discussion
as they go along, whether you seek the information or not –
otherwise many people (myself included) might not go to the website
very often: with a list, you have no choice, you get the message.
This is a strong advantage, in my view.

So it would be great to hear people's opinion on this second issue.
5-time-frame for decisions

Ideally we should be able to formally announce the journal and call for
papers at the conference which Athina has kindly organised as our
launch party – VIRT3C in Hull, in a month – woo-hoo!
[What do you mean, it wasn't organised for that?! Oh well, my mistake. ;-)]

In any case decisions should be finalised a couple weeks before the
conference – in order to give ourselves time to set up anything
that needs to be set up (particularly if we go with the website
discussion option).  So I think a
two-week period should be enough - today is Friday 19 February March
2010 – I am invoking Maintainer Mojo to request that final
decisions be reached by Saturday 6 March 2010. No further debate
after that will be accepted as relevant. Tough love, people!

6-References
[1]
Whitworth B and R Friedman (2009a) "Reinventing academic publishing
online. Part I: Rigor, relevance and practice", First
Monday, Volume 14, Number 8 - 3 August 2009; Whitworth B and R Friedman
(2009b) "Reinventing academic publishing online. Part II: A
socio-technical vision", First Monday, Volume 14, Number 9 - 7 September 2009.

[2]
Toni Prug, “Open-process Academic Publishing”
[http://www.oekonux.org/journal/index.html]







7-CSPP
peer review process: main stages

Prospective authors submit a proposal to the list.

All list members vet this proposal during a reasonable period of time (1-2 weeks?): is it appropriate for the journal, are arguments or references missing?

Authors write their submission.

Authors submit to the journal.

The submission is posted by the editor to a password-protected part of the website [mailing list?] who also alerts the main journal list that he has done so.

The editor suggests two expert reviewers (volunteers welcome).

The two expert reviewers read and evaluate the submission during a reasonable period of time (3 weeks?). Reviewers are encouraged to coordinate their
review.

Reviewers post their reviews and recommendations to a password-protected part of site [mailing list?] and alert the list that they have done so.

The list discusses this during a reasonable period of time (1-2 weeks?).

During this time consensus emerges: publish, revise and resubmit (to two other reviewers, for example?), or

During this time consensus does not emerge: the decision then moves to a formal vote on the Governance Board: publish, revise and resubmit
(to two other reviewers, for example) or reject.

Submission and review process published.

Readers can comment and rate.

Authors can respond in comments section [and add links in the text to relevant comments and responses - no updating of text though].

=-=-=-=-

cheers,
Mathieu
______________________________
http://www.oekonux.org/journal

______________________________
http://www.oekonux.org/journal



Thread: joxT00189 Message: 12/77 L4 [In date index] [In thread index]
Message 00237 [Homepage] [Navigation]