A Checklist to Combat Cognitive Biases in Crowdsourcing

A Checklist to Combat Cognitive Biases in Crowdsourcing

Draws, T.A. (TU Delft Web Information Systems)
Rieger, A. (TU Delft Web Information Systems)
Inel, O. (TU Delft Web Information Systems)
Gadiraju, Ujwal (TU Delft Web Information Systems)
Tintarev, N. (Universiteit Maastricht)

Kamar, Ece (editor)
Luther, Kurt (editor)

2021

Recent research has demonstrated that cognitive biases such as the confirmation bias or the anchoring effect can negatively affect the quality of crowdsourced data. In practice, however, such biases go unnoticed unless specifically assessed or controlled for. Task requesters need to ensure that task workflow and design choices do not trigger workers’ cognitive biases. Moreover, to facilitate the reuse of crowdsourced data collections, practitioners can benefit from understanding whether and which cognitive biases may be associated with the data. To this end, we propose a 12-item checklist adapted from business psychology to combat cognitive biases in crowdsourcing. We demonstrate the practical application of this checklist in a case study on viewpoint annotations for search results. Through a retrospective analysis of relevant crowdsourcing research that has been published at HCOMP in 2018, 2019, and 2020, we show that cognitive biases may often affect crowd workers but are typically not considered as potential sources of poor data quality. The checklist we propose is a practical tool that requesters can use to improve their task designs and appropriately describe potential limitations of collected data. It contributes to a body of efforts towards making human-labeled data more reliable and reusable.

Crowdsourcing
Human-labeled Data
Subjective Judgments
Cognitive Bias
Data Quality

http://resolver.tudelft.nl/uuid:38945fc7-2ea8-4acf-ac2f-666e5e9bc3b1

Association for the Advancement of Artificial Intelligence (AAAI)

2022-04-04

978-1-57735-872-5

Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 9

The Ninth AAAI Conference on Human Computation and Crowdsourcing, 2021-11-14 → 2021-11-18, Virtual conference

Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Institutional Repository

conference paper

© 2021 T.A. Draws, A. Rieger, O. Inel, Ujwal Gadiraju, N. Tintarev