Challenges of data collection on mturk: A human-AI joint face-matching task

During the COVID-19 pandemic, many human-subject studies have stopped in-person data collection and shifted to virtual platforms like Amazon Mechanical Turk (MTurk). This shift involves important considerations for study design and data analysis, particularly for studies involving behavioral assessment and performance with technology. We report on lessons learned from a recent study that used MTurk for a face-matching task with an open-source AI. Participants received $5 compensation for completing a 45-minute session that included questionnaires. To help address data validity issues, Qualtrics fraud-detection features (i.e., reCAPTCHA, ID-Fraud), trap-items (e.g., Respond with Often), and a modified-batch-randomization-process were employed. Participants' accumulative accuracy and response rates were also assessed. Out of 272 participants, 121 passed the data inclusion criteria. The questionnaires' reliability was within range (average 0.78) for the healthy dataset. Accumulative accuracy in the face-matching task decreased approximately halfway through the task. Subsequent data inspection revealed that almost half of the participants spent longer than 20 seconds and up to 12 minutes on a random image pair. It is possible that participants were interrupted during the study or they elected to take unscheduled breaks. Environmental factors that were easier to control during in-person laboratory studies now require built-in controls for virtual study environments. We learned that: (1) it is imperative to monitor performance measures over time for each participant; (2) the study duration may need to be kept shorter on virtual platforms compared to in-person studies; (3) an optional, planned break during the task might help prevent other unplanned breaks.