TY - GEN
T1 - Enhancing Image Classification Capabilities of Crowdsourcing-Based Methods through Expanded Input Elicitation
AU - Yasmin, Romena
AU - Grassel, Joshua T.
AU - Hassan, Md Mahmudulla
AU - Fuentes, Olac
AU - Escobedo, Adolfo R.
N1 - Publisher Copyright:
© 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
PY - 2021
Y1 - 2021
N2 - This study investigates how different forms of input elicitation obtained from crowdsourcing can be utilized to improve the quality of inferred labels for image classification tasks, where an image must be labeled as either positive or negative depending on the presence/absence of a specified object. Three types of input elicitation methods are tested: binary classification (positive or negative); level of confidence in binary response (on a scale from 0-100%); and what participants believe the majority of the other participants’ binary classification is. We design a crowdsourcing experiment to test the performance of the proposed input elicitation methods and use data from over 200 participants. Various existing voting and machine learning (ML) methods are applied and others developed to make the best use of these inputs. In an effort to assess their performance on classification tasks of varying difficulty, a systematic synthetic image generation process is developed. Each generated image combines items from the MPEG-7 Core Experiment CE-Shape-1 Test Set into a single image using multiple parameters (e.g., density, transparency, etc.) and may or may not contain a target object. The difficulty of these images is validated by the performance of an automated image classification method. Experimental results suggest that more accurate classifications can be achieved when using the average of the self-reported confidence values as an additional attribute for ML algorithms relative to what is achieved with more traditional approaches. Additionally, they demonstrate that other performance metrics of interest, namely reduced false-negative rates, can be prioritized through special modifications of the proposed aggregation methods that leverage the variety of elicited inputs.
AB - This study investigates how different forms of input elicitation obtained from crowdsourcing can be utilized to improve the quality of inferred labels for image classification tasks, where an image must be labeled as either positive or negative depending on the presence/absence of a specified object. Three types of input elicitation methods are tested: binary classification (positive or negative); level of confidence in binary response (on a scale from 0-100%); and what participants believe the majority of the other participants’ binary classification is. We design a crowdsourcing experiment to test the performance of the proposed input elicitation methods and use data from over 200 participants. Various existing voting and machine learning (ML) methods are applied and others developed to make the best use of these inputs. In an effort to assess their performance on classification tasks of varying difficulty, a systematic synthetic image generation process is developed. Each generated image combines items from the MPEG-7 Core Experiment CE-Shape-1 Test Set into a single image using multiple parameters (e.g., density, transparency, etc.) and may or may not contain a target object. The difficulty of these images is validated by the performance of an automated image classification method. Experimental results suggest that more accurate classifications can be achieved when using the average of the self-reported confidence values as an additional attribute for ML algorithms relative to what is achieved with more traditional approaches. Additionally, they demonstrate that other performance metrics of interest, namely reduced false-negative rates, can be prioritized through special modifications of the proposed aggregation methods that leverage the variety of elicited inputs.
UR - http://www.scopus.com/inward/record.url?scp=85134175718&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85134175718&partnerID=8YFLogxK
U2 - 10.1609/hcomp.v9i1.18949
DO - 10.1609/hcomp.v9i1.18949
M3 - Conference contribution
AN - SCOPUS:85134175718
SN - 9781577358725
T3 - Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
SP - 166
EP - 178
BT - HCOMP 2021 - Proceedings of the 9th AAAI Conference on Human Computation and Crowdsourcing
A2 - Kamar, Ece
A2 - Luther, Kurt
PB - Association for the Advancement of Artificial Intelligence
T2 - 9th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2021
Y2 - 14 November 2021 through 18 November 2021
ER -