Peer Assessment Reliability in Large Cohorts: A Dataset of Sixty Thousand Paired Judgements

Abstract

Peer Assessment Reliability in Large Cohorts has moved quickly from a niche scientific question to something a wider community
can now investigate with reasonably reliable tools, but the evidence behind it has not always kept pace. We pull together the
current evidence for peer assessment reliability in large cohorts, covering the methodological, practical, regulatory and cost
angles, with an eye on where practice has genuinely settled and where researchers and reviewers are still largely working it out
for themselves. Selective, context-specific pursuit looks better supported by the evidence gathered here than either broad rollout or
blanket avoidance.