Re: [jox] Cutting the Knot
- From: Toni Prug <tony irational.org>
- Date: Sat, 27 Feb 2010 17:07:06 +0000
Hi all,
given the length, here's the summary:
i support StefaMn proposal for authors choosing the qualifying model
(binary/ratings)
i extend it with the choices of _early screening_ and peer reviewing
models (both open/closed choices).
However, i'm not convinced _rating_ is the most suitable name for
signaling the attributes of an article.
i also support use of Plone over email list.
Journal Commons is a new project working on providing advanced tools and
organizational techniques for cooperation in knowledge production -
we're using Plone, working closely with two journals (in private for
now), i will inform the list of our milestones in coming months.
yours,
toni
-------
my firm belief is that academic journals should facilitate, assist and
improve the production and spread/distribution of new knowledge.
Journals do not produce new knowledge. Authors do. This, in my
experience, is lost on almost all journals i learned about so far.
Instead, there's rigidity on the side of journals, dictating to authors
a set of rules which, given our level of technological development, make
little sense today. At the same time, i believe to a large extent
because of the opaqueness of peer reviewing system, authors are not
being helpful to journal editors either: they regularly send articles
with vast deficiencies in quality of argument, novelty, or simply
required formatting, which rightly drives editorial boards mad and adds
to their workload by what it is often seen as lack of respect on the
side of authors.
I think that two moves could help to reduce tensions on both sides and
improve the production: journals should be more accommodating/open to
the needs of authors, simultaneously with being more strict in social
pressures on authors to observe these new, more flexible, and more
appropriate to our times, set of rules.
StefanMn then proposed a choice where authors submitting a proposal could
indicate whether they want a binary model (publish or reject) or a
multi-dimensional rating system:
[See http://www.oekonux.org/journal/list/archive/msg00212.html]
I like this proposal. I see it as a desirable increases in flexibility
on the side of journal, which authors, i believe, will appreciate. As to
readers, i don't see the problem with some articles being rated, and
others not. There could be an icon always displayed next to the article,
indicating it is a rated article. I expect that as the time goes, more
authors will choose ratings, since they will figure out that it gives
them more chance of being published (with problems that article might
have noted by the ratings), rather then rely on the binary
publish/reject decision.
My view is that binary model is terrible, unscientific, and antagonistic
for entirely wrong reasons (i'm a big supporter of antagonism as method,
when suitable) and should be gradually replaced. However, new models
should not be imposed, but rather offered as alternatives that are
monitored, evaluated and improved. We have had too many centuries of
binary model to displace it over night, and authors would rightly be
skeptical of such move without seeing the benefit of it first in practice.
Hence, i think that StefanMn proposal is an excellent way to win over
authors in favor of new systems, in a way which gives them the chance to
both choose and observe how the new models work. I much prefer this
approach, than having only new systems (ratings, open-process, or
otherwise), because i see how it would enable us to win authors over and
to demonstrate that we are not claiming that we definitely know what is
right and what is wrong. This, position of 'we know best for sure' is
the attitude of almost all the journals i interacted with (perhaps i was
unlucky, but it seems a pattern), and no wonder authors send all kind of
junk - they see journals as arrogant, uncooperative and self-serving
gatekeepers to their career advance (i've sent my first ever journal
article submission, one on the open process in academic publishing
recently to a journal i.e. this is not entirely my own experience, but
what i observed from years of being surrounded by academic friends and
colleagues + the interactions i had with several journals in the past
year). Pushed by the publish-or-perish evaluating model, by the academic
rewards model in general, and by the arrogance and uncooperativeness of
journals, authors do often behave badly. But it comes partially, perhaps
largely, out of desperation.
We can improve this relation a lot by these new models.The result, i
hope (convinced by the arguments and analysis), will be a far better
mutual respect and relationship authors-journal-readers and it's fitting
for a peer production journal to be self-reflective and hence the
innovator in own field of production. The general spirit of the two
papers referenced (Reinventing academic publishing online + Open
Process) is for me the spirit of peer production, of less centralized,
yet more structured and more beneficially (for all sides) organized
systems - in the context and sphere of knowledge production.
Second, if we want to open up the journal selection process and provide rewards to those who do
normally invisible work (i.e. reviewers), in line with Toni Prug's
proposal [2] for a community peer review system (through a list where proposals are
vetted and reviews are released), then by definition we are rejecting
the publish / don't publish model: vetting and orientation occur
upstream, even before an actual full submission.
Indeed. The open process model relies on discovering, and fixing when
possible, problems in the early stage, when the cost of doing so is low
on both sides (authors, editors/reviewers). However, in humanities and
social sciences, it is often the case that the quality and novelty of
the contribution cannot be seen until the whole argument is developed
into a longer, more fined grained piece i.e. the details are sometimes
all that matters in a piece, and we cannot see it in an early stage.
Situation seems even worse for natural sciences, where months, or years,
of lab experimental work might not results into a single publishable
paper nor new findings (i was told). Yet, it is funders, or heads of
research centres, who still make a judgment on the plausibility of the
project before it starts, hence acting as a form of early peer review.
We can try to act in a similar way, judging the plausibility of a short
proposal being developed in something we consider worthy of being
publishable. Also, many ideas are visibly worthy of developing in their
early stage, within the first thousand words, as a rough proposal.
In short, there are issues with the open process as well, but i also do
not see it as a binary either/or. I would like to see us offering to
authors the options to choose between open/closed twice:
a) EARLY SCREENING: authors choose whether to submit proposal (let's say
up to 1000 words) for a paper; they can do so either in an open OR
closed way i.e.their proposals, our comments, their comments back, and
our decision (either Yes, please develop and submit; OR No, we don't
think it's for this journal) are either visible, or not, publicly,
depending on what authors choose.
b) REVIEWING: use open OR closed peer reviewing for full length articles
i.e. authors do not have to go through a) at all, they simply follow the
traditional model and submit full paper, choosing for peer reviewing to
be open or closed.
(c) special cases: There are several possible complications in which the
journal will have to make decision. One example is that a series of peer
reviewers approached can refuse to peer review the article if their name
is used, since they might be intimidated by the importance of the
author, or for some other reason. In that case, the journal might
approach the author and insist that peer reviews must be done without
peer reviewers names used. There are other possible twists and
resolutions, we'll learn as we do it.
Later, we could move to a more refined set of early screening and peer
reviewing workflows (where some aspect might be open to some groups at
some stage in the process, and not in other stages) as we all learn by
practices what works well, what not, and how to develop it further.
In order to protect the
reputation of the journal, we need to alert readers that we are aware
of flaws, but that _we decided to publish anyway_. Hence the need to
“qualify” or “signal” (rather than “rate”) published submissions.
Yes, i agree, very well put. I'm not convinced _rating_ is the best term
for what we're intending to do either, although i like the procedure and
support it. Qualify or signal seem clumsy, though not entirely unusable
substitutes. When we say that a paper has been rated, it is intuitive.
Not so when we say it has been signaled, or qualified. I'm undecided.
Any other ideas how to name this?
So, this is the first point to decide: what categories do we have?
i have to think about this separately.
4-decision: review discussion system
+ email OR web platform for reviewing - the shades of openness
I love email lists, and i find it hard to accept more complex web tools
for collaboration, since, as a rule with rare exceptions, i find them
less, not more helpful as tools/environments to assist and change work
positively. However, i was recently given an introduction to Plone, and
i was won over by it. Excuse me for the technical language for a moment:
it seems to me that worklows, transitions and fine grained access for
groups, including the acquisition mechanism [1], will lend themselves
well to variety of degrees of openness and structures of workflows for
peer reviewing. Even more so, if each Folder, or Collection object
(which contain other objects i.e submitted documents) can have RSS feeds
to which we can subscribe too, including a unified feed - this is likely
to be not so diffifult to add, even if it currently does not exist in
the Plone and its plugins (call products in Plone). For example, see
this product, http://plone.org/products/collective.watcherlist which
enables any object within Plone (documents, folder, etc - if i'm not
mistaken) to be watched, so that watchers automatically receive emails
on any change in the observed object.
For the two of the projects i'm working on with two other journals, we
decided to use Plone. We're now working on building worklfows to present
it to academics involved in those projects.
I would therefore recommend using Plone (and not just any other web
tool), and not an email list, with one important note: to increases the
chances of Plone academic users being satisfied with it as a helpful
platform, we must keep things simple to start with i.e we must not
alienate early users with complex workflows and procedures.
Complexity can be added once the initial usage picks up and once the
usual resistance (i have plenty of it too) to new systems is overcome
through the satisfaction with the benefits that the platform brings to
its users.
When we're done with the projects i'm currently working on, we will
share workflows and all the information on our Plone setup, use, and
comments we got from academic communities we're working within. So far,
journals and communities expressed the wish to do this development
process in private, until they reach the decision on whether to use the
new models we are presenting - this should be within the next month or two.
Finally, Juan Grigera (who is working with me on this) and i decided
the create a project out of this, we named it Journal Commons. We will
launch it once the proposals we're working on are decided on. We aim to
support journals to implement new processes of cooperation, using
advanced web tools and organizational techniques. Given the number of
technically skilled people here, and the use of Plone, we have a basis
for a potential close cooperation on the Journal Commons as a separate
project. We'll keep the list informed with major project milestones in
the coming months.
--------
[1] 'Acquisition allows behavior to be distributed hierarchically
throughout the system [...] you can change an object’s behavior by
changing where it is located in the object hierarchy.'
http://docs.zope.org/zope2/zope2book/Acquisition.html
______________________________
http://www.oekonux.org/journal