The Effects of Dispersion and Reciprocity on Assessment Fidelity in Peer-Review Systems: A Simulation Study

Over the last 20 years, online peer review and assessment have become widely used and well-researched practices in education. Their use increased, especially with the proliferation of nonconventional large-scale and online modes of teaching and learning, such as Massive Open Online Courses (MOOCs). A well-designed peer-review system is expected to produce valid and reliable assessments of the artifacts created by participants. These systems notably vary in designs, particularly in the structure of the peer-review networks, i.e., how participants are linked to each other as creators and reviewers. To date, little research has been done on how different network structures impact a system's ability to accurately assess the quality of the evaluated artifacts. We begin to address this gap, using a simulation approach to demonstrate that two network characteristics-dispersion and reciprocity-may indeed affect assessment fidelity in peer-review systems in conjunction with other design choices, such as evaluation scale and aggregation method. We also outline directions for further investigations of peer-review systems design.