Learning Checklists to Reduce Lottery Effects in Scientific Peer Review

Peer review is partly a lottery. A paper gets three or four reviewers, and each one notices only a handful of the issues a larger pool would raise. Swap in a different committee and the verdict can change.

We learn a checklist from thousands of past reviews and show it to every reviewer, so each committee starts from the same baseline checks. The hard part is granularity: too specific and items never recur, too broad and they fit every paper. We pick the level at which reviewers of the same paper agree most, beyond what reviewers of different papers share by chance.

The formulation. Checklist design as choosing a coarsening of the review claim space that removes more lottery variance than the bias it adds, selected by expected correlated agreement (ECA).
The pipeline. An LLM-assisted pipeline that turns a review corpus into a tree of paper-checkable checklist items at every granularity.
The evidence. Lower committee error in simulated review on NeurIPS 2021 and ICLR 2022, and in a 20-person human ranking study.

58% less error for a three-reviewer committee on NeurIPS 2021

21% less error for the same committee size on ICLR 2022

~10 unaided reviewers needed to match the checklist on NeurIPS

0.48 → 0.22 ranking error of human participants, without and with the checklist

The problem

In the NeurIPS 2021 experiment, every paper was reviewed by two independent committees. We broke each review into atomic claims and matched them across the two committees to see how much reviewers actually overlap.

6.4% of the issues a larger reviewer pool would raise are caught by one reviewer

34.3% of claims are also raised by the other committee

81% of issues are raised by only a single reviewer

18.2% of concrete criticisms are clearly refuted by the paper itself

A shared checklist can fix part of this: every reviewer checks the same things, and checkable facts get checked correctly. But a checklist also steers attention. Anything it leaves out may get less weight from everyone at once, which shifts the committee's average away from what unaided reviewers would conclude. A good checklist is one where the variance it removes outweighs the bias it adds.

Method

The pipeline reads a historical review corpus and grows one tree of checklists. Cutting the tree near the root gives a few broad items; cutting near the leaves gives thousands of narrow ones.

Pipeline diagram in three stages, following one NeurIPS 2021 review sentence about missing baselines. Stage one, reviews to atomic claims: split, label as a fact settled by the paper, and filter to concrete claims, giving 15,118 claims from reviews of 298 papers. Stage two: strip paper-specific details, embed, and let an LLM merge claims that say exactly the same thing. Stage three: grow one checklist tree, name each cluster; at k equals 100 the item reads 'Are baselines and SOTA comparisons adequate?', at k equals 6,000 it reads 'Missing important experimental baselines'. — An LLM splits each review into atomic claims and keeps only those the **paper itself can settle**, so a model can later check them reliably. Claims are stripped of paper-specific names, embedded, merged when identical, and clustered into a tree whose nodes are named as questions to ask of a paper.

Which cut should reviewers see? Past reviews cannot tell us how reviewers would have scored with a checklist they never saw, so we cannot measure error directly.

Instead we score each cut by expected correlated agreement (ECA): how often two reviewers of the same paper raise the same item (A_same), minus how often reviewers of two different papers do (A_diff). Broad items inflate both terms, narrow items shrink both, and the difference peaks in between. We prove that, under stated conditions, the ECA-maximizing checklist approximates the one with the lowest error.

Items k	NeurIPS 2021			ICLR 2022
Items k	A_same	A_diff	ECA	A_same	A_diff	ECA
50	22.1	12.4	+9.8	38.9	27.2	+11.7
100	16.2	6.1	+10.2	28.8	15.9	+12.9
200	12.3	3.2	+9.0	21.7	9.2	+12.6
500	8.7	1.4	+7.3	15.8	4.2	+11.6
600	8.2	1.1	+7.1	14.9	3.5	+11.5
1,000	6.6	0.7	+5.9	13.1	2.3	+10.8
2,000	4.5	0.4	+4.2	10.9	1.2	+9.8
6,000	2.0	0.1	+1.8	8.1	0.4	+7.7

Agreement at eight cuts of the checklist tree (all values in %). Coarser cuts raise both A_same and A_diff; ECA, their difference, peaks at 100 items on both venues (highlighted), and that is the checklist reviewers see.

Results

Measuring review error needs a stable target, which would take dozens of independent reviewers per paper. We therefore built a simulated reviewer population of LLM agents, each covering a narrow slice of concerns like a human reviewer, on 30 held-out papers per venue.

Three plots for NeurIPS 2021 against committee size from 1 to 16. Variance: without a checklist it starts at 0.82 and falls as one over N; with the learned or the official checklist it starts near 0.15. Bias: zero without a checklist by construction, minus 0.07 on average for the learned checklist, plus 0.21 for the official NeurIPS checklist. MSE: the learned checklist is lowest at small committees; the unaided committee only catches up at about N equals 10. — Committee error on NeurIPS 2021. The checklist cuts **variance** sharply and adds only a small **bias**, so its total error stays below unaided review until committees reach about ten reviewers.

With the 100-item checklist, a single reviewer's error on NeurIPS drops from 0.82 to 0.21, and a committee of four halves its error (0.21 to 0.10). The official NeurIPS 2021 checklist lowers variance just as much but adds more bias, leaving it about 0.04 worse at every committee size. On ICLR, a single reviewer's error drops from 0.84 to 0.34.

ECA also picks the right granularity. On both venues, the checklist with the highest ECA has the lowest error for every committee of three or more, and the one with the lowest ECA has the highest.

In a human study, 20 participants ranked two sets of three NeurIPS 2021 papers, one set with the checklist already filled in by an LLM and one without. Measured against the NeurIPS order, ranking error fell from 0.48 to 0.22; for a committee of three, from 0.32 to 0.11.

Team

This is research from the Epistemic Decisions Lab at Northwestern University by:

Publication

This work is under review at ICLR 2027. The preprint is available here (PDF).

Liu, X., Guo, Z., Hartline, J., & Hullman, J. (2026). Learning Checklists to Reduce Lottery Effects in Scientific Peer Review. Manuscript under review.

@misc{liu2026checklists,
  title={Learning Checklists to Reduce Lottery Effects in Scientific Peer Review},
  author={Liu, Xuanyou and Guo, Ziyang and Hartline, Jason and Hullman, Jessica},
  year={2026},
  note={Under review}
}

Overview

The problem

Method

Results

Team

Publication