Print Email Facebook Twitter To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systems Title To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systems Author Bharos, Abri (TU Delft Electrical Engineering, Mathematics and Computer Science) Contributor Gadiraju, Ujwal (mentor) He, G. (graduation committee) Houben, G.J.P.M. (graduation committee) Cavalcante Siebert, L. (graduation committee) Degree granting institution Delft University of Technology Programme Computer Science Date 2022-12-22 Abstract Powerful predictive AI systems have demonstrated great potential in augmenting human decision-making. Recent empirical work has argued that the vision for optimal human-AI collaboration requires ‘appropriate reliance’ of humans on AI systems. However, accurately estimating the trustworthiness of AI advice at the instance level is quite challenging, especially in the absence of performance feedback pertaining to the AI system. In practice, the performance disparity of machine learning models on out-of-distribution data makes the dataset-specific performance feedback unreliable in human-AI collaboration. Inspired by existing literature on critical thinking and explanation-based human debugging, we propose the use of debugging an AI system as an intervention to foster appropriate reliance. In this paper, we explore whether a critical evaluation of AI performance within a debugging setting can better calibrate users’ assessment of an AI system and lead to more appropriate reliance. Through a quantitative empirical study (N = 234), we found that our proposed debugging intervention does not work as expected in facilitating appropriate reliance. Instead, we observe a decrease in reliance on the AI system after the intervention — potentially resulting from early exposure to the AI system’s weakness. We explored the dynamics of user confidence to help explain how inappropriate reliance patterns occur and found that human confidence is not independent of AI advice, which is potentially dangerous when trying to achieve appropriate reliance. Our findings have important implications for designing effective interventions to facilitate appropriate reliance and better human-AI collaboration Subject explanation-based human debuggingappropriate reliancehuman-ai collaborationcrowd computing To reference this document use: http://resolver.tudelft.nl/uuid:be10b0a2-2826-4b2d-8865-6519cf628776 Part of collection Student theses Document type master thesis Rights © 2022 Abri Bharos Files PDF ARJ_Bharos_Master_Thesis_Report.pdf 1.38 MB Close viewer /islandora/object/uuid:be10b0a2-2826-4b2d-8865-6519cf628776/datastream/OBJ/view