Message 00108 [Homepage] [Navigation]
Thread: joxT00000 Message: 78/176 L21 [In date index] [In thread index]
[First in Thread] [Last in Thread] [Date Next] [Date Prev]
[Next in Thread] [Prev in Thread] [Next Thread] [Prev Thread]

Re: [jox] Re: Peer Review



[Converted from multipart/alternative]

[1 text/plain]
Dear all

Apologies for slackness, I've been travelling and am currently snowed under. I want to re-read those papers again before commenting on Stefans poins. But: It does not make sense to be discussing openly something which only a few have seen so I'm taking it upon myself to post the references here. I never got an answer from the authors to my question as to whether it would be OK to distribute. In French there is a saying: "Qui ne dit mot consent". To potential readers: please remember the second part of the First Monday paper is still a draft so do not quote. 
My biggest question is: they suggest publishing everything that is submitted with ratings ie 5 excellent, 4 good, etc even -1 (dont publish) if the author wants. I think this is an interesting concept but I also think this is potentially going towards the "process" option we discussed before. Perhaps  articles rated 4 or 5 could periodically be gathered into a "stable" release that could be advertised across the weboverse?
Anyway here are the refs:
 Hi Mathieu Thanks for your enquiry and interest. Your journal  looks good. The social design of a KES is quite complex, but you can see our  first attempt to lay out the details and spirit of the endeavor at http://brianwhitworth.com/BWRF-FM-Part2.pdf In  particular note the section on privacy, where we allow reviewers to reveal  themselves after the review is over if they want to. Also note it is still draft  until it comes out in FM.   Some more detail on Socio-technical design in  general is also given in our Handbook of STS Design, see http://brianwhitworth.com/sts  for which mashup is a first go at making such stuff available. In particular,  check out my chapter 1 which gives an idea of some of the complexity, see  http://brianwhitworth.com/STS/STS-chapter1.pdf    Note that the full design is not yet specified - we  intend to develop this working with collaborators in a feedback process, so if  you want to be part of that let us know. I am working with Rob Friedman and  Michael Browstein on this project, so I copy this email to them  also.   all the best   Brian Whitworth  

Cheers
Mathieu

----- Original Message -----
From: Stefan Merten <smerten oekonux.de>
Date: Monday, September 7, 2009 10:50 pm
Subject: Re: [jox] Re: Peer Review
To: journal oekonux.org
Cc: Stefan Merten <smerten oekonux.de>

Dear Mathieu, CSPP Board Members, all!

Last week (9 days ago) Mathieu O'Neil wrote:
OK, I contacted one of the authors of the FM paper and he kindly
sent me links to his second part as well as another relevant
chapter. 

Mathieu sent the links privately. Indeed a very useful paper. I'll
respond to this here because otherwise the communication becomes to
confusing for me ;-) .

In his mail he also mentioned that they are seeking feedback from
people on these issues. I responded that would be possible but that
I was acting as part of a group and queried whether it would 
be OK
to circulate the papers to this group (without really saying 
who was
in it though I had originally sent a link to this list's 
archive). 

They are welcome. But then we need the right to put the link here
(it's a public link anyway so this should not matter too much).

@Mathieu: I hope you won't mind that I quote the following questions
from your private mail here.

Last week (7 days ago) Mathieu O'Neil wrote:
I guess the main questions are: which features would we want?

In general I think those guys are very much on the same track as we
:-) . I think the key question to solve I already mentioned in 
my post
from `Mon, 24 Aug 2009 12:25:45 +0200`__. To prevent leaking too much
information from the paper here I only reply to what is new or very
different from my suggestions there or comments from others made here
already. I relate to the chapter "DEMOCRATIC ONLINE KNOWLEDGE
EXCHANGE".

__ http://www.oekonux.org/journal/list/archive/msg00091.html

* Multi-level ratings

  Similar to my previous suggestion the submissions can and 
should be
  given a number of levels reflecting how the reviewers / 
editorial  board assesses the submission. That solves the 
problem of the binary
  accept / reject scheme. On a website it is also easy to flag
  articles appropriately.

  Also using several rating dimensions is a good idea. What 
is needed,
  however, is to reduce the many dimensions to a single 
number to give
  orientation to readers. May be the scientific vs. 
activist category
  could be one such dimension.

  I think the reviewers should still have the 
responsibility to rate
  an article. May be the reader rating could be one more 
dimension -
  given that there are enough ratings from the readers.

  Readers need to have an option to tune what dimensions 
they consider
  most important. Technically this would mean to have different
  reports using a reader's weights. That is technically simple.

* Submissions

  I think every submission should be addressed to the 
editorial board.
  IMHO this also adds a level of responsibility on the part 
of the
  submitters: It's a difference whether you just drop 
something in an
  electronic system or ask real people to consider your 
submission.
  This doesn't mean that a submission is withheld from the 
website. It
  can be put there immediately if the author wishes so.

* Anonymous contributions

  In general I don't like anonymous contributions as well 
as I don't
  like near-anonymous contributions from obvious 
pseudonyms. I see
  that creation of an account actually feels like an 
obstacle - at
  least to me. But nonetheless I don't like anonymous 
contributions.  IMHO this at least reduces the spam problem 
very much.

Which of those
we want can we have without too much hassle?

I think most if not all of their suggestions and our ideas are
technically feasible. IMHO the real work (i.e. human labor) is reading
submissions, thinking about them and making decisions. This real work
needs to be done anyway. The rest can and should be left to the
machine.


                                        	Grüße

                                        	Stefan
______________________________
http://www.oekonux.org/journal

****
Dr Mathieu O'Neil
Adjunct Research Fellow
Australian Demographic and Social Research Institute
College of Arts and Social Science
The Australian National University

E-mail: mathieu.oneil anu.edu.au
Tel.: (61 02) 61 25 38 00
Web: http://adsri.anu.edu.au/people/visitors/mathieu.php
Mail: Coombs Building, 9
Canberra, ACT 0200 - AUSTRALIA





[2 text/html]
______________________________
http://www.oekonux.org/journal



Thread: joxT00000 Message: 78/176 L21 [In date index] [In thread index]
Message 00108 [Homepage] [Navigation]