The Effects of Dispersion and Reciprocity on Assessment Fidelity in Peer-Review Systems: A Simulation Study

Dmytro Babik, Scott P. Stevens, Andrew Waters, David Tinapple

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


Over the last 20 years, online peer review and assessment have become widely used and well-researched practices in education. Their use increased, especially with the proliferation of nonconventional large-scale and online modes of teaching and learning, such as Massive Open Online Courses (MOOCs). A well-designed peer-review system is expected to produce valid and reliable assessments of the artifacts created by participants. These systems notably vary in designs, particularly in the structure of the peer-review networks, i.e., how participants are linked to each other as creators and reviewers. To date, little research has been done on how different network structures impact a system's ability to accurately assess the quality of the evaluated artifacts. We begin to address this gap, using a simulation approach to demonstrate that two network characteristics-dispersion and reciprocity-may indeed affect assessment fidelity in peer-review systems in conjunction with other design choices, such as evaluation scale and aggregation method. We also outline directions for further investigations of peer-review systems design.

Original languageEnglish (US)
Article number8982044
Pages (from-to)580-592
Number of pages13
JournalIEEE Transactions on Learning Technologies
Issue number3
StatePublished - Jul 1 2020


  • Assessment
  • clustering
  • dispersion
  • evaluation systems
  • information quality
  • knowledge artifacts
  • network structures
  • peer assessment
  • peer review
  • reciprocity.

ASJC Scopus subject areas

  • Education
  • General Engineering
  • Computer Science Applications


Dive into the research topics of 'The Effects of Dispersion and Reciprocity on Assessment Fidelity in Peer-Review Systems: A Simulation Study'. Together they form a unique fingerprint.

Cite this