[jox] Multi-rating mode of evaluation (was: Multi-rating mode of evaluation / Updating papers)
- From: Stefan Merten <smerten oekonux.de>
- Date: Wed, 02 Dec 2009 13:01:38 +0100
Hi Mathieu, Felix, StefanMz, Christian, all!
Last week (11 days ago) Mathieu O'Neil wrote:
Thanks to everyone who has contributed to the process this week. I
just wanted to alert everyone that there is an issue that needs to
be resolved: Felix and Athina (see below) have objected to our
stated policy of "rating" submissions according to a scale of -1 to
+5 (inspired by Brian Whitworth and Rob Friedman's papers on
academic publishing in First Monday).
Frankly I don't understand these concerns. I mean a journal to me is
most of all a filter on the stuff which is out there anyway. If I
don't want a filter I can just use Google...
As you know, this would in
theory facilitate the publication of a greater number of submissions
than is the case in other journals (though if you consider the fact
that some academic journals publish 5 or 6 issues each year, that
represents a fair number of papers).
The traditional approach to maintain credibility of a journal and thus
maintain the filter function is to cut down on the numbers of
published articles - like the FirstMonday article describes very well.
Besides the problems discussed in this article there is also the
problem that our journal should be able to publish more controversial
and/or community/activist pieces also - which may have an impact on
the credibility of our journal unless flagged appropriately.
We could perhaps simplify the rating system but this would not
change the fact that there would be a situation where some published
submissions are given a higher "expert rating" than others before
publication (as well as a "reader rating" given by registered users
after publication). The question of how people who have been given
an inferior rating by reviewers would feel about this situation has
not been addressed so far and I think it needs to be. If we do not
rate then do we go to (in theory) a lower publication rate where
only excellent contributions are published?
I definitely think we need a multi-dimensional rating system which
needs to be accessible by readers. This way we can do flagging like
mentioned above in a proper way and can maintain credibility of the
journal as well as publishing stuff which is less than optimal in a
few dimensions.
I'm not a fan of creating a single dimension rating though - i.e.
ratings like +2. But admittedly giving an average rating gives people
quick access who are not interested in all the details. I discussed_
this when we had this topic before.
.. _discussed: http://www.oekonux.org/journal/list/archive/msg00132.html
Last week (11 days ago) Felix Stalder wrote:
Why would editorial rating allow us to publish more papers? Since this is
not a paper publication, we can publish as many as we want anyway.
Editorial rating would allow us to publish some papers, but grade them
poorly, flagging them as 'barely good enough'. We shouldn't do this.
Why?
We
should publish only papers that we agree are fit for publication.
But "fit for publication" is not based on a single reason. There may
be articles which we consider great in many dimensions but they lack
some certain feature. Lack of this feature normally would make them
unacceptable but if we can express this lack by a rating then the
credibility of our journal is maintained *and* the article is
published.
Last week (10 days ago) Stefan Meretz wrote:
IIRC language was part of the rating, right? This would discriminate
against those who are not so familiar with english language.
Not if you have the choice to ignore a language rating (i.e. weight
0).
The journal might contain strong pov articles, which is good. One aim of
the Journal must be to support a "thinking against the mainstream". If
peer production is really a new thing, then theorzing around this topic
will and has to be new and unfamiliar. -- Will the expert committee work
in this fashion? Or are unfamiliar povs are rated out, because the pov
is not shared? (this does say anything against the persons listed which
I don't know).
This also could be solved by a multi-dimensional rating. If it is a
very original article (i.e. an unusual POV) then this can be reflected
in the respective rating.
Last week (10 days ago) Mathieu O'Neil wrote:
In my view, all published texts have to be be in good English.
Well, that really raises a problem. Transforming bad English into good
English is a tedious task and needs experts in the author's native
language and English. In other words: This is hard to do and given our
limited resources easily can take lots of time.
Unless we have a mechanism to improve the language in a week or so I
don't think we should delay publication because of English.
A rating of the language would solve this problem. Later on an article
where language is improved can have a new language rating.
Last week (9 days ago) Christian Siefkes wrote:
if a public rating system is used, POV and other controversial articles will
generally end up with an average rating, since some reviewers like them,
while others don't.
Only if you use an average rating with fixed weights. However, this is
the sort of abstraction which cuts off too much information.
Hence it might indeed be better to stick with a binary
"publish / don't publish" decision from the journal editors / reviewing
committee, and let the readers do the ratings. Of course, reader ratings
will have the same effect of punishing controversial items, but at least
reader ratings don't sound quite as "official" as committee ratings.
Well, if an author feels really punished by a rating s/he might as
well decide to retract the article. But if the official reviewers
think someone is crap in some dimensions this should be flagged IMHO.
Of course, internally the reviewers will probably use some kind of rating
system, since peer review usually does, and if we want transparency we can't
hide these ratings from the readers -- but at least I wouldn't show them by
default, but place them somewhere in the background information about the
article (reader has to click on "reviewing process / reviewer feedback" or
something like that).
That would be an option in any case. However to maintain credibility
of our journal it would be more useful to give a small table listing
the review ratings in the various dimensions.
6 days ago Christian Siefkes wrote:
But
what I mean with "controversial" is simply stuff like, for example, my "From
Exchange to Contributions" (peerconomy) work -- some people like the
approach very much, others think it's crap or nothing really new. Such
differing estimations will automatically lead to a mediocre rating.
Good example. For me this would an article about this should appear in
the journal but should also be clearly flagged as - say -
"controversial". Without such an option I'd try hard to prevent
publishing such material.
What we really need IMHO is a list of dimensions. I remember we had
one once. May be it could be put to the website so it is readily
available?
Grüße
Stefan
______________________________
http://www.oekonux.org/journal