HEP
Libraries Webzine
Issue 6 / March 2002
Peer Reviewing and Electronic Publishing
Renato Spigler(*)
21/02/2002
Abstract
Several issues about organizing worldwide open archives, refereeing system,
and quality control and certification of research publications are reviewed.
Advantages and drawbacks of using the web are discussed, and peculiarities of
different research areas pointed out.
Content
Introduction: Worldwide Open Archives and Global
Knowledge Network
It is worth debating the future of scientific
publications, especially in connection with the emergence of (purely?)
electronic scientific journals, alongside or instead of the traditional
journals.
Over the last few years, Stevan Harnad [1], [2], [3],
and Harnad et al. [4], put forth a concrete proposal about
organizing a worldwide open archive, based on local archives
but fully interoperable, in order to maximize the research
impact and reducing all inherent costs. A precise analysis has
been made by Paul Ginsparg [5] about creating a global
knowledge network. Choosing between a centralized data archive, and
a physically distributed system may still be an issue, see Ref. [5],
which may be a matter of trade-off between minimizing costs and having
a comfortable redundancy.
These proposals anyway should be promptly accepted, and what is more, strongly
supported, by the scientific community, in order to increase the impact
of their findings, reduce costs, and jump into the future of scientific
publishing systems.
Quality Control and Certification by Peer Review
An essential rather than merely important aspect of current
research activities is the quality control and certification
(QC/C) [4], usually carried out by the so-called
peer-review system, that is through review by peers, to which every
published paper should be subject.
Great concerns about guaranteeing quality, especially when scientific
articles circulate online freely and for free, have been expressed
very recently, for instance, by the Executive Director of the American
Mathematical Society [6].
In principle, through the web all scientific reports may be made
available to everybody, promptly and as they are, without waiting for
peers' scrutiny, and by omitting other
publishing delays. It is true that a QC/C
should be provided, but during recent decades, when the number of
scientific reports has increased enormously, the refereeing of papers has
become dependent on increasingly more uncertain outcomes.
The point is that the process of refereeing a paper requires
involving at least one expert (and there are few, for every given rather
narrow topic) and his or her time. Too many papers and too few good
experts make it difficult to properly judge a paper in a reasonable
amount of time. Nevertheless, something had to be done: Better little than
nothing.
Putting everything indiscriminately on the web, without (or, say,
before) refereeing, would imply already a pre-filter, since, hopefully
(but very likely) most people would be careful in committing themselves
officially before the community in formal writings.
It is the author's responsibility to choose the periodical to
which submit his or her paper. One should consider that a good paper would be accepted by a bad journal,
but, conversely, every paper accepted by a good journal should meet a
certain (sufficiently high) standard. In the former case, the author may
be damaged regarding the impact that his or her paper will have along with
its possibly associated benefits, such as promotion, tenure or research
grants, even though the journal involved
might eventually raise its level.
In the second case, however, the community will benefit from a reasonably
well assessed paper, whose results will be reliably used for further
research and reference.
Recall that a good scientific journal is made good by the quality of
its Editorial Board and by the prestige of its Editor-in-Chief, who, among
other things, sets the standard for selecting among the submitted papers
those which deserve to be accepted for publication. Nowadays, when a
commercial publisher decides to start a new journal, it is natural that
it invites a number of highly reputed and qualified scholars to join the
Editorial Board. I guess that almost nobody refuses: After all, it should
be a duty to serve the community in this way. However, given the increasing
number of periodicals (at a rate in some fields exponential), how many
papers will every member of the Editorial Board receive for refereeing?
And, if the Editor-in-Chief does not send papers to them, will
the level of the journal still be high? Of course, every member of the board
may send, in turn, some papers that he or she receives to some other expert
that he or she trusts. Some may consider
the journals published by commercial
publishers to be less reliable (concerning QC/C) than those
produced by scientific (not-for-profit) organizations, such as the American
Mathematical Society, the Society for Industrial and Applied Mathematics,
or the American Physical Society, to name
just a few.
The Peculiarities of Different Research Areas
Here I wish only to report my own experiences, as an author, a referee,
and also as a simple attentive reader of scientific publications, even those
at popular level. Being interested in a number of subjects, in the field of
(mostly) applied Mathematics and (in some fields of) Physics and
Engineering, I think that I have a broader feeling than if I
were focused only on a single narrow research area. Nevertheless, I
fully realize that different disciplines have different character,
peculiarities, and needs. Therefore, what is valid for Mathematics may not
be valid for Medicine or for the Humanities, and so on, as we shall see below.
A Few Anecdotes About Referees
Here are a few anecdotes. When scholars were few, refereeing was
perhaps an unnecessary hobby. In
Italy, around the 1950s, the mathematician Francesco G. Tricomi distrusted
the refereeing system (new for him), and so he used to
send his papers to an Italian journal (where he was well-known), and to a
journal where the refereeing system was adopted, at the same time. Such
a procedure nowadays is usually expressly forbidden.
Some years ago, the famous mathematician Gian-Carlo Rota told me that
when he received a paper for some very high standard periodicals of which
he was the Editor-in-Chief (namely, ``Advances in Mathematics", and
``Advances in Applied Mathematics"), he knew that he could decide
a priori about acceptance on the basis of the referee to whom he
would choose to send the paper for examination. He also added that when
a paper was not so good, he would write to the author that the paper
was good but it should be rewritten: Usually this would never happen,
and the problem was solved.
In the mid 1980s, Peter D. Lax, a top mathematician, told me and other
people that one of his papers (submitted under his secretary's name), was
rejected by the very good but in principle expository periodical
``The American Mathematical Monthly". The rejection was motivated by the
fact that the paper was concerned with research (while the Monthly
publishes expository papers), but what if Peter Lax had signed the paper
with his own name?
Still in the mid 1980s, I read a curious debate in the Monthly. An
author complained, but actually wanted to open a discussion about the behaviour of a certain (anonymous) referee. The point was that, at some
point of his career, this author had the opportunity to make some
(financial) advance in his academic career. This advance was based on
scientific productivity (quality and quantity of publications), so he took
out from his folders a few incompleted pieces of research, and completed
them. I do not argue about the ``ethical" issue of doing research all
of a sudden for the above purpose, but the author had a hard time with a
referee, when he submitted a paper that contained only some incomplete,
partial solution of a certain mathematical problem. The point was that the
referee rejected the paper by saying that he knew the author and his work,
and he judged that the author could have done better, that is, in that paper
he could have solved completely the problem he coped with. Now, the question
is whether the referee's behaviour was legitimate or not.
The "Weight" of the Referees: Fashion in Science
and Other Things
What is the domain of judgments by a referee? If an author
is known to the referee, how might this bias the referee's report and
decision? And how are the Editor-in-Chief's related decisions affected by
his possible personal acquaintance with or criticism against the author(s)
and the referee(s)? This may not matter when authors are top scientists
(many referees themselves may be at the same level or even at a level
below them), but what about when merely ``trendy subjects" are considered?
Fashion does exist even in Science. The point is that, whenever
``fashionable" research topics are pursued by leading scholars, there
should be a good reason for this. Some of these issues clearly belong to
the psychological or social rather than to the scientific domain, but
they affect the research world in some way, for instance in weighing
the consequent impact of the published work.
A number of problems can be easily solved, for instance by adopting
a ``blind refereeing" policy, as some journals provide:
Usually referees' names are not disclosed to the authors, but the
referee(s) can read the author(s) name on the paper itself. Unless the
author(s) ask to remain anonymous to the referee(s).
There are, however, more subtle biases in the process of refereeing a
scientific paper. For example, every active researcher knows that
acceptance or rejection of a paper strongly depends, not only on quality,
originality, and perhaps exposition, but also on the choice of the subject
itself. A contribution devoted to a very classical (old-fashioned) topic,
will be rejected, in general if not surely, by all major periodicals. The
Editorial Board makes the policy: A paper can be accepted on the basis of
the subject, and one cannot complain about
this. The
periodical's editorial policy may change with a change in the Board,
and papers rejected in
the past could be accepted, and conversely. The point is now, since there
exist several sub-areas in a given research area, can a paper in the
sub-area A be compared with one in the sub-area B? Is the best paper in
the sub-area A better or worse than the worst paper in the sub-area B?
All these questions are answered in practice, and are the responsibility
of the Editorial Board which makes the final decisions.
For these reasons, authors may choose one journal or another. If they
choose a very high standard journal, and their paper is rejected, they can
(and often do) try to improve their paper on the basis of the referee(s)
report(s), and re-submit the paper to a lower level journal, and such a
process may continue. In this sense one could argue (as pointed out in
Ref. [4]) that there exists at the end some place where every paper
(hopefully correct, though not great) might be published. The interaction
among authors, referees and Editor is useful in any case, and should be
regarded as positive, though time-consuming. By the way, as was observed
in Ref. [5], most of the editorial time (and hence cost) seems to
be devoted to papers that will be rejected. It is, however, impossible to
avoid this burden, which is not at all useless, as nobody would consider a
loss of time the time that a sculptor spends to remove inessential material
(cf. Ref. [5] for this example).
One should stress that the QC/C process is of paramaount importance also
for the following reasons. In the mathematical literature, a few years ago it
was said that every specialized article was read, on the average, by no more
than 1.5 readers (somebody maliciously claimed that this included the
referee(s)). Considering that it is very likely that authors do not read,
at least in full
or at least in detail all papers that they quote, it becomes even more
essential that somebody (it is to be hoped an expert) undertakes a
critical examination
of every paper submitted to every serious scientific periodical. I do not
believe that all authors check carefully all mathematical proofs, all
physical experiments, all materials and methods that they refer to.
Sometimes this is actually impossible: How could a given reader reproduce
experiments conducted over many months by a hundred or more scientists on
a big particle accelerator? How could you expect even a smart pure
mathematician to go through the full proof of the so-called ``Fermat's last
theorem", which took over 300 years and a very special ingenuity
in order to be established, and, at least for a number of years, will be
understandable by only a handful of keen experts?
Moreover, it would be impossible for
an author to read all papers he
quotes, and then, why not, all papers quoted in the papers he quotes, and
so on. Scientific papers are written for the experts in the field, otherwise
no progress would be possible if one were to go down to the elementary points
on which his or her paper ultimately is based.
Again on Different Research Areas
There are great differences among very distant disciplines concerning
the refereeing (peer-review) system itself, for which reason a systematic
assessment of the current system has so far been missing. ``Systematic"
with respect to all disciplines, and this despite the broad use and the
ensuing effects on accepting papers for publication, and even to allocate
research grants.
It has been reported in Ref. [5] (see Ref. [7]) that within
the Health Sciences, the peer-review system can be ``expensive, slow,
subjective and biased, open to abuse, patchy at detecting important
methodological defects, and almost useless at detecting fraud or
misconduct". The reasons for such a crude and hopeless judgement should be
better investigated. Certainly, the financial interests of pharmaceutical
firms and the kind of scientific method followed by some
researchers, play a role in all this. In Ref. [5], Ginsparg observes
that, in this respect, there is a deep difference between medical doctors
and physicists (and closely related scientists). In the former community,
the percentage of acceptance of papers is quite low, around 10%, and
there is a relatively small number of authors who write for a large body of
clinicians. On the contrary, in the latter community, the rate of
acceptance is much higher, but the populations of the writers and of the
readers almost coincide. These points make a big difference, and one
should start explaining them. For instance, are the medical doctors more
selective, or do they write (and do resesarch) ``very badly"? Are the
physicists more serious than medical doctors or is it ``easier" to write
about Physics than about Medicine?
Within the Humanities, to give another example, the refereeing system
itself may be definitely much more questionable. In fact, writing about
literature or Philosophy clearly involves a certain degree of subjectivity
and thus arbitrariness. Therefore, it may be difficult sometimes to judge
originality and ``correctness", as in the scientific domain, and thus
providing a QC/C might be very hard if not fully irrelevant. For instance,
it may be more important for Humanities scholars to publish a book
(instead of papers), because this provides a measure of the value of the
work, due to the financial support received (even by a commercial
publisher) to realize a full book.
We should not forget that the audience is specific about the way of
judging papers and books, concerning their style, quality and
quantity: If a Humanities scholar, or a medical doctor, writes a paper,
this will be read and judged by Humanities readers or by medical doctors,
respectively. After all, it is well known that a language barrier
separates the various different, even close, scientific fields, thus
preventing interdisciplinary communication.
Another reason is that, according to the statistical enquiry reported
above, about 40% of the published mathematical papers turned out to be
affected by errors. Of course, there exist several kinds of errors, even
within the mathematical literature: Misprints or, occasionally, some much
more relevant mistakes or omissions may be easily fixed by an attentive
scholarly reader; some other errors, on the other hand, might be rather
serious and in some cases it can be harmful to the subsequent research. In
other disciplines things can be different, less or more harmful than in the
case of Mathematics.
Summary: Advantages and Drawbacks of E-Publishing
Besides the considerable time likely spent by authors, referees, and
Editors, however, one should also take into account the mailing time,
whenever regular (that is traditional) mail is used, even though it
has become popular to use couriers, i.e. express mail services, which
are faster and (sometimes) more reliable. Clearly, also this process may
take advantage from using the web intensively. Mail communication is
faster, and the only time that cannot be reduced is the genuine time that
authors need to do research, referees need to read, and Editors need to
spend to make seriously weighed decisions.
Waiting times, whose length increases, as a rule, with the quality of
the journal where a given paper has been submitted, are due to careful
refereeing but also to backlog and to the fact that magazines are
published regularly (periodically) but at fixed dates.
Besides all this, the web may speed up the overall processes inherent
in doing research and publishing, in checking or finding results in the
existing literature and thus the originality of the paper under
examination.
Psychological obstacles might exist, here and there, against switching
completely to a purely electronic publishing system. This attitude belongs
to human nature, and might be difficult to remove within the present
generation. People may be afraid that a power failure could destroy all
they have, while a solid (paper) version of their scientific work is not
subject to this type of accident. However, one should consider that many
electronic copies could be generated for increasing reliability, and their
physical support located even in geographically distant areas. One should
also remember that a fire may destroy entire libraries, and this has
happened over the centuries. The point is that we should accept that there
is some statistical aspect pervading all our lives, and we can only be
advised to pay some attention in any case, both to fire and to power
failures.
- 1
- Harnad, Stevan, "The invisible hand of peer review",
Nature (online), November 5, 1998,
http://helix.nature.com/webmatters/invisible/
invisible.html
- 2
- Harnad, Stevan, "Free at Last: The Future of
Peer - Reviewed Journals", D-Lib Magazine 5(12), December 1999,
http://www.dlib.org/dlib/december99/12harnad.html
- 3
- Harnad, Stevan, "Ingelfinger Over-Ruled: The Role of
the Web in the Future of Refereed Medical Journal Publishing",
The Lancet Perspectives 256, December 2000, supplement, s16.
- 4
- Stevan Harnad, Les Carr, and Tim Brody, "How and Why To
Free All Refereed Research From Access- and Impact-Barriers Online,
Now", High Energy Physics Libraries Webzine, Issue
4, June 2001, http://webzine.web.cern.ch/webzine/4/papers/1/
- 5
- Ginsparg, Paul, "Creating a global knowledge network", Invited
Contribution for Conference held at UNESCO HQ, Paris, 19-23 February,
2001, Second Joint ICSU Press - UNESCO Expert Conference on Electronic
Publishing Science, during the Session `Responses from the scientific
community', February 20, 2001, http://arxiv.org/blurb/pg01unesco.html,
July 16, 2001.
- 6
- Ewing, John H., "No Free Lunches: We Should Resist the Push to
Rush Research Online", Chronicle of Higher Education, October 12,
2001, http://www.chronicle.com
- 7
- Godlee, Fiona, and Jefferson, Tom, Eds., ``Peer Review in Health
Sciences", BMJ Books, 1999.
Renato Spigler
Dipartimento di Matematica, Universita di "Roma Tre"
Largo S.L. Murialdo 1, 00146 Rome, Italy
email : spigler@mat.uniroma3.it
For citation purposes:
Renato Spigler, "Peer-Reviewing and Electronic Publishing",
High Energy Physics Libraries Webzine, issue 6, March 2002
URL: <http://webzine.web.cern.ch/webzine/6/papers/5/>
Reader Response
If you have any comments on this
article, please contact the Editorial Board
Maintained by: HEPLW
Team
Last modified: March 28, 2002
High Energy Physics Libraries Webzine