High Energy Physics Libraries Webzine

Editorial Board
Issue 6

 HEP Libraries Webzine
Issue 6 / March 2002

Peer Reviewing and Electronic Publishing

Renato Spigler(*)



Several issues about organizing worldwide open archives, refereeing system, and quality control and certification of research publications are reviewed. Advantages and drawbacks of using the web are discussed, and peculiarities of different research areas pointed out.



Introduction: Worldwide Open Archives and Global Knowledge Network

It is worth debating the future of scientific publications, especially in connection with the emergence of (purely?) electronic scientific journals, alongside or instead of the traditional journals. Over the last few years, Stevan Harnad [1], [2], [3], and Harnad et al. [4], put forth a concrete proposal about organizing a worldwide open archive, based on local archives but fully interoperable, in order to maximize the research impact and reducing all inherent costs. A precise analysis has been made by Paul Ginsparg [5] about creating a global knowledge network. Choosing between a centralized data archive, and a physically distributed system may still be an issue, see Ref. [5], which may be a matter of trade-off between minimizing costs and having a comfortable redundancy. These proposals anyway should be promptly accepted, and what is more, strongly supported, by the scientific community, in order to increase the impact of their findings, reduce costs, and jump into the future of scientific publishing systems.

Quality Control and Certification by Peer Review

An essential rather than merely important aspect of current research activities is the quality control and certification (QC/C) [4], usually carried out by the so-called peer-review system, that is through review by peers, to which every published paper should be subject. Great concerns about guaranteeing quality, especially when scientific articles circulate online freely and for free, have been expressed very recently, for instance, by the Executive Director of the American Mathematical Society [6]. In principle, through the web all scientific reports may be made available to everybody, promptly and as they are, without waiting for peers' scrutiny, and by omitting other publishing delays. It is true that a QC/C should be provided, but during recent decades, when the number of scientific reports has increased enormously, the refereeing of papers has become dependent on increasingly more uncertain outcomes. The point is that the process of refereeing a paper requires involving at least one expert (and there are few, for every given rather narrow topic) and his or her time. Too many papers and too few good experts make it difficult to properly judge a paper in a reasonable amount of time. Nevertheless, something had to be done: Better little than nothing. Putting everything indiscriminately on the web, without (or, say, before) refereeing, would imply already a pre-filter, since, hopefully (but very likely) most people would be careful in committing themselves officially before the community in formal writings.

The Choice of the Journal

It is the author's responsibility to choose the periodical to which submit his or her paper. One should consider that a good paper would be accepted by a bad journal, but, conversely, every paper accepted by a good journal should meet a certain (sufficiently high) standard. In the former case, the author may be damaged regarding the impact that his or her paper will have along with its possibly associated benefits, such as promotion, tenure or research grants, even though the journal involved might eventually raise its level. In the second case, however, the community will benefit from a reasonably well assessed paper, whose results will be reliably used for further research and reference.

The Quality of the Journal and the QC/C by Commercial Publishers

Recall that a good scientific journal is made good by the quality of its Editorial Board and by the prestige of its Editor-in-Chief, who, among other things, sets the standard for selecting among the submitted papers those which deserve to be accepted for publication. Nowadays, when a commercial publisher decides to start a new journal, it is natural that it invites a number of highly reputed and qualified scholars to join the Editorial Board. I guess that almost nobody refuses: After all, it should be a duty to serve the community in this way. However, given the increasing number of periodicals (at a rate in some fields exponential), how many papers will every member of the Editorial Board receive for refereeing? And, if the Editor-in-Chief does not send papers to them, will the level of the journal still be high? Of course, every member of the board may send, in turn, some papers that he or she receives to some other expert that he or she trusts. Some may consider the journals published by commercial publishers to be less reliable (concerning QC/C) than those produced by scientific (not-for-profit) organizations, such as the American Mathematical Society, the Society for Industrial and Applied Mathematics, or the American Physical Society, to name just a few.

The Peculiarities of Different Research Areas

Here I wish only to report my own experiences, as an author, a referee, and also as a simple attentive reader of scientific publications, even those at popular level. Being interested in a number of subjects, in the field of (mostly) applied Mathematics and (in some fields of) Physics and Engineering, I think that I have a broader feeling than if I were focused only on a single narrow research area. Nevertheless, I fully realize that different disciplines have different character, peculiarities, and needs. Therefore, what is valid for Mathematics may not be valid for Medicine or for the Humanities, and so on, as we shall see below.

A Few Anecdotes About Referees

Here are a few anecdotes. When scholars were few, refereeing was perhaps an unnecessary hobby. In Italy, around the 1950s, the mathematician Francesco G. Tricomi distrusted the refereeing system (new for him), and so he used to send his papers to an Italian journal (where he was well-known), and to a journal where the refereeing system was adopted, at the same time. Such a procedure nowadays is usually expressly forbidden. Some years ago, the famous mathematician Gian-Carlo Rota told me that when he received a paper for some very high standard periodicals of which he was the Editor-in-Chief (namely, ``Advances in Mathematics", and ``Advances in Applied Mathematics"), he knew that he could decide a priori about acceptance on the basis of the referee to whom he would choose to send the paper for examination. He also added that when a paper was not so good, he would write to the author that the paper was good but it should be rewritten: Usually this would never happen, and the problem was solved. In the mid 1980s, Peter D. Lax, a top mathematician, told me and other people that one of his papers (submitted under his secretary's name), was rejected by the very good but in principle expository periodical ``The American Mathematical Monthly". The rejection was motivated by the fact that the paper was concerned with research (while the Monthly publishes expository papers), but what if Peter Lax had signed the paper with his own name? Still in the mid 1980s, I read a curious debate in the Monthly. An author complained, but actually wanted to open a discussion about the behaviour of a certain (anonymous) referee. The point was that, at some point of his career, this author had the opportunity to make some (financial) advance in his academic career. This advance was based on scientific productivity (quality and quantity of publications), so he took out from his folders a few incompleted pieces of research, and completed them. I do not argue about the ``ethical" issue of doing research all of a sudden for the above purpose, but the author had a hard time with a referee, when he submitted a paper that contained only some incomplete, partial solution of a certain mathematical problem. The point was that the referee rejected the paper by saying that he knew the author and his work, and he judged that the author could have done better, that is, in that paper he could have solved completely the problem he coped with. Now, the question is whether the referee's behaviour was legitimate or not.

The "Weight" of the Referees: Fashion in Science and Other Things

What is the domain of judgments by a referee? If an author is known to the referee, how might this bias the referee's report and decision? And how are the Editor-in-Chief's related decisions affected by his possible personal acquaintance with or criticism against the author(s) and the referee(s)? This may not matter when authors are top scientists (many referees themselves may be at the same level or even at a level below them), but what about when merely ``trendy subjects" are considered? Fashion does exist even in Science. The point is that, whenever ``fashionable" research topics are pursued by leading scholars, there should be a good reason for this. Some of these issues clearly belong to the psychological or social rather than to the scientific domain, but they affect the research world in some way, for instance in weighing the consequent impact of the published work. A number of problems can be easily solved, for instance by adopting a ``blind refereeing" policy, as some journals provide: Usually referees' names are not disclosed to the authors, but the referee(s) can read the author(s) name on the paper itself. Unless the author(s) ask to remain anonymous to the referee(s). There are, however, more subtle biases in the process of refereeing a scientific paper. For example, every active researcher knows that acceptance or rejection of a paper strongly depends, not only on quality, originality, and perhaps exposition, but also on the choice of the subject itself. A contribution devoted to a very classical (old-fashioned) topic, will be rejected, in general if not surely, by all major periodicals. The Editorial Board makes the policy: A paper can be accepted on the basis of the subject, and one cannot complain about this. The periodical's editorial policy may change with a change in the Board, and papers rejected in the past could be accepted, and conversely. The point is now, since there exist several sub-areas in a given research area, can a paper in the sub-area A be compared with one in the sub-area B? Is the best paper in the sub-area A better or worse than the worst paper in the sub-area B? All these questions are answered in practice, and are the responsibility of the Editorial Board which makes the final decisions. For these reasons, authors may choose one journal or another. If they choose a very high standard journal, and their paper is rejected, they can (and often do) try to improve their paper on the basis of the referee(s) report(s), and re-submit the paper to a lower level journal, and such a process may continue. In this sense one could argue (as pointed out in Ref. [4]) that there exists at the end some place where every paper (hopefully correct, though not great) might be published. The interaction among authors, referees and Editor is useful in any case, and should be regarded as positive, though time-consuming. By the way, as was observed in Ref. [5], most of the editorial time (and hence cost) seems to be devoted to papers that will be rejected. It is, however, impossible to avoid this burden, which is not at all useless, as nobody would consider a loss of time the time that a sculptor spends to remove inessential material (cf. Ref. [5] for this example). One should stress that the QC/C process is of paramaount importance also for the following reasons. In the mathematical literature, a few years ago it was said that every specialized article was read, on the average, by no more than 1.5 readers (somebody maliciously claimed that this included the referee(s)). Considering that it is very likely that authors do not read, at least in full or at least in detail all papers that they quote, it becomes even more essential that somebody (it is to be hoped an expert) undertakes a critical examination of every paper submitted to every serious scientific periodical. I do not believe that all authors check carefully all mathematical proofs, all physical experiments, all materials and methods that they refer to. Sometimes this is actually impossible: How could a given reader reproduce experiments conducted over many months by a hundred or more scientists on a big particle accelerator? How could you expect even a smart pure mathematician to go through the full proof of the so-called ``Fermat's last theorem", which took over 300 years and a very special ingenuity in order to be established, and, at least for a number of years, will be understandable by only a handful of keen experts? Moreover, it would be impossible for an author to read all papers he quotes, and then, why not, all papers quoted in the papers he quotes, and so on. Scientific papers are written for the experts in the field, otherwise no progress would be possible if one were to go down to the elementary points on which his or her paper ultimately is based.

Again on Different Research Areas

There are great differences among very distant disciplines concerning the refereeing (peer-review) system itself, for which reason a systematic assessment of the current system has so far been missing. ``Systematic" with respect to all disciplines, and this despite the broad use and the ensuing effects on accepting papers for publication, and even to allocate research grants. It has been reported in Ref. [5] (see Ref. [7]) that within the Health Sciences, the peer-review system can be ``expensive, slow, subjective and biased, open to abuse, patchy at detecting important methodological defects, and almost useless at detecting fraud or misconduct". The reasons for such a crude and hopeless judgement should be better investigated. Certainly, the financial interests of pharmaceutical firms and the kind of scientific method followed by some researchers, play a role in all this. In Ref. [5], Ginsparg observes that, in this respect, there is a deep difference between medical doctors and physicists (and closely related scientists). In the former community, the percentage of acceptance of papers is quite low, around 10%, and there is a relatively small number of authors who write for a large body of clinicians. On the contrary, in the latter community, the rate of acceptance is much higher, but the populations of the writers and of the readers almost coincide. These points make a big difference, and one should start explaining them. For instance, are the medical doctors more selective, or do they write (and do resesarch) ``very badly"? Are the physicists more serious than medical doctors or is it ``easier" to write about Physics than about Medicine? Within the Humanities, to give another example, the refereeing system itself may be definitely much more questionable. In fact, writing about literature or Philosophy clearly involves a certain degree of subjectivity and thus arbitrariness. Therefore, it may be difficult sometimes to judge originality and ``correctness", as in the scientific domain, and thus providing a QC/C might be very hard if not fully irrelevant. For instance, it may be more important for Humanities scholars to publish a book (instead of papers), because this provides a measure of the value of the work, due to the financial support received (even by a commercial publisher) to realize a full book. We should not forget that the audience is specific about the way of judging papers and books, concerning their style, quality and quantity: If a Humanities scholar, or a medical doctor, writes a paper, this will be read and judged by Humanities readers or by medical doctors, respectively. After all, it is well known that a language barrier separates the various different, even close, scientific fields, thus preventing interdisciplinary communication. Another reason is that, according to the statistical enquiry reported above, about 40% of the published mathematical papers turned out to be affected by errors. Of course, there exist several kinds of errors, even within the mathematical literature: Misprints or, occasionally, some much more relevant mistakes or omissions may be easily fixed by an attentive scholarly reader; some other errors, on the other hand, might be rather serious and in some cases it can be harmful to the subsequent research. In other disciplines things can be different, less or more harmful than in the case of Mathematics.

Summary: Advantages and Drawbacks of E-Publishing

Besides the considerable time likely spent by authors, referees, and Editors, however, one should also take into account the mailing time, whenever regular (that is traditional) mail is used, even though it has become popular to use couriers, i.e. express mail services, which are faster and (sometimes) more reliable. Clearly, also this process may take advantage from using the web intensively. Mail communication is faster, and the only time that cannot be reduced is the genuine time that authors need to do research, referees need to read, and Editors need to spend to make seriously weighed decisions. Waiting times, whose length increases, as a rule, with the quality of the journal where a given paper has been submitted, are due to careful refereeing but also to backlog and to the fact that magazines are published regularly (periodically) but at fixed dates. Besides all this, the web may speed up the overall processes inherent in doing research and publishing, in checking or finding results in the existing literature and thus the originality of the paper under examination. Psychological obstacles might exist, here and there, against switching completely to a purely electronic publishing system. This attitude belongs to human nature, and might be difficult to remove within the present generation. People may be afraid that a power failure could destroy all they have, while a solid (paper) version of their scientific work is not subject to this type of accident. However, one should consider that many electronic copies could be generated for increasing reliability, and their physical support located even in geographically distant areas. One should also remember that a fire may destroy entire libraries, and this has happened over the centuries. The point is that we should accept that there is some statistical aspect pervading all our lives, and we can only be advised to pay some attention in any case, both to fire and to power failures.


Harnad, Stevan, "The invisible hand of peer review", Nature (online), November 5, 1998,
http://helix.nature.com/webmatters/invisible/ invisible.html

Harnad, Stevan, "Free at Last: The Future of Peer - Reviewed Journals", D-Lib Magazine 5(12), December 1999,

Harnad, Stevan, "Ingelfinger Over-Ruled: The Role of the Web in the Future of Refereed Medical Journal Publishing", The Lancet Perspectives 256, December 2000, supplement, s16.

Stevan Harnad, Les Carr, and Tim Brody, "How and Why To Free All Refereed Research From Access- and Impact-Barriers Online, Now", High Energy Physics Libraries Webzine, Issue 4, June 2001, http://webzine.web.cern.ch/webzine/4/papers/1/

Ginsparg, Paul, "Creating a global knowledge network", Invited Contribution for Conference held at UNESCO HQ, Paris, 19-23 February, 2001, Second Joint ICSU Press - UNESCO Expert Conference on Electronic Publishing Science, during the Session `Responses from the scientific community', February 20, 2001, http://arxiv.org/blurb/pg01unesco.html, July 16, 2001.

Ewing, John H., "No Free Lunches: We Should Resist the Push to Rush Research Online", Chronicle of Higher Education, October 12, 2001, http://www.chronicle.com

Godlee, Fiona, and Jefferson, Tom, Eds., ``Peer Review in Health Sciences", BMJ Books, 1999.

Author Details

Renato Spigler
Dipartimento di Matematica, Universita di "Roma Tre"
Largo S.L. Murialdo 1, 00146 Rome, Italy

email : spigler@mat.uniroma3.it

For citation purposes:
Renato Spigler, "Peer-Reviewing and Electronic Publishing", High Energy Physics Libraries Webzine, issue 6, March 2002
URL: <http://webzine.web.cern.ch/webzine/6/papers/5/>

Reader Response

If you have any comments on this article, please contact the  Editorial Board
Editorial Board
Issue 6
Maintained by: HEPLW Team

Last modified:  March 28, 2002