How peer review works, and what it doesn't catch

Peer review is the system in which a journal sends a submitted manuscript to outside experts in the field, who assess its methods, reasoning, and significance before publication. It improves what gets published — but it cannot verify the underlying data, replicate experiments, or reliably detect fabrication, limitations documented in the journal editors' own accounts of the process, most comprehensively by the International Committee of Medical Journal Editors' recommendations. Peer review is a filter, not proof. Engevity News covers how science knows; this primer explains the mechanism readers rely on whenever a headline says "a study found."

The system's power and its limits both come from the same fact: reviewers are unpaid outside experts reading the paper the authors wrote, usually in a few hours, without seeing the raw data.

What actually happens to a submitted paper?

The sequence at a typical journal:

1. An editor screens the submission for fit and basic soundness, rejecting most papers outright at this stage.
2. Two to four peer reviewers — scientists in the same field, usually anonymous to the authors — read the manuscript and write assessments: are the methods appropriate, do the results support the claims, what is missing.
3. The editor weighs the reviews and decides: reject, revise, or accept. Most published papers pass through at least one round of major revision.
4. The revised paper is checked again — by the same reviewers or new ones — before final acceptance.

The process takes months at most journals, longer at the most selective ones. Reviewers are not paid; the work is a professional obligation, done alongside the reviewer's own research, which is why review quality varies with the reviewer's available time as much as their expertise.

How do we know peer review works?

Partly by experiment. Controlled studies of the process exist: the most cited, a 1998 study in Annals of Emergency Medicine by researchers who inserted deliberate errors into manuscripts before review, found reviewers caught on average only a fraction of the planted errors — a result that has shaped editorial training since. Studies of reviewer agreement find two reviewers frequently disagree on the same paper more than they agree, a finding repeated across the journals that have audited their own processes. And the system's screening value shows in what it rejects: preprints that later failed review have been found to contain more errors and weaker statistics than their published counterparts, in comparisons the meta-science literature documents.

What doesn't peer review catch?

The honest list is long. Reviewers rarely see raw data, so fabricated or altered data can pass unless something looks wrong statistically — which is exactly what allowed the largest documented fraud cases, like the anesthesiologist case retracted across multiple journals in 2009 after 89 publications fell, to run for years. Peer review does not replicate experiments; a plausible-sounding result stays plausible until someone repeats it, and most results are never directly repeated. It does not guarantee importance — a methodologically sound study of something trivial sails through on soundness alone. And it carries known biases: against novelty, against null results, and toward famous authors, biases the meta-research community has quantified in study after study.

What comes after publication?

The part that matters more than most readers realize: post-publication scrutiny. Readers who reanalyze published data, replication projects that repeat findings, and comment letters that catch what review missed — the post-publication record corrects the pre-publication one, which is why retraction exists and why a retracted paper's traces persist: studies of citation find retracted papers continue to be cited, often approvingly, years later. Publication is the beginning of a finding's evaluation, not the end.

What the evidence establishes is a system that makes papers better on average while missing errors, fraud, and unreproducible results at documented rates. That is a reason to read studies with attention to their design — not a reason to dismiss them.