Peer review

A method used to improve the quality of written works is called "peer review" in some academic fields and "refereeing" in others. This method subjects the work to scrutiny and, often, annotation or editing, by a number of the author's peers. The principle employed is that different people will see different errors, so increasing the number of people reviewing the work will increase the opportunity for errors to be found and fixed.

The panel of reviewers for peer review is selected informally by the editor. Typically, the editor will have access to a large pool of reviewers and will randomly assign three to five to review a particular paper. One important part of the process is that the reviewers are considered the authors "peers", and are not considered more knowledgeable, more accomplished, or of higher social status than the author. In this respect, peer review differs sharply from forms of evaluation which occur in an educational or business context.

The process of peer review in a scientific journal involves the editor of a journal sending an article to several peer reviewers who remain anonymous. The peer reviewers evaluate the article and add comments for improvement. These comments are sent to the editor who then transmits the comments to the author.

Some of the characteristics of the systems are:

• The peer reviewers remain anonymous and all communications with the author goes through the editor.
• In journals, peer reviewers generally are not allowed to communicate with each other and in fact do not know each others identities. In other contexts (such as grant proposal review), the reviewers do communicate with each other, because often the reviewers are experts from different fields.
• All of the comments of the peer reviewers are transmitted to the author. This serves as a check because it allows the author to see the rationale behind a publication decision.
• The peer reviewers act in an advisory capacity and the final decision to publish or not to publish remains with the editor.

While this includes the power to completely ignore the reviewers' comments, this is rare. Much more common is the situation in which the reviewers end up with a lukewarm evaluation of a paper and there is reasonable disagreement whether the paper should be published or not. Still more common is the situation in which the reviewers sharply disagree among themselves on the quality of the paper.

Sociologists of science have analysed the way in which the system of peer review works in practice and have advanced the criticism that it is vulnerable to capture by an elite. Experiments have been performed which suggest that peer reviewers tend to report, or abstain from reporting, logically identical errors in papers submitted for peer review accordingly as the paper's conclusions contradict, or substantiate, the conclusions of the reviewer's own research.

This tradition has long been known to the academic community[?], especially with scientific papers, where publication of the paper allows its quality and veracity to be criticised, the purported goal being to improve the quality of work.

In the field of computer software, the principle mentioned above has been stated as Linus's law, often formulated as "given enough eyeballs, all bugs are shallow". Eric S. Raymond has also written extensively about peer review as it pertains to open source software, in a series of papers starting with The Cathedral and the Bazaar.

Having one's work criticised and improved is intended to have the beneficial effect on improving the quality of their future work.

See also: preprint