Evaluation of feedback generated from agent-based social skills training systems

Evaluation of feedback generated from agent-based social skills training systems: A qualitative analysis on the comprehensibility, usability, and improvement points of the generated feedback

Ntasi, Nikola (TU Delft Electrical Engineering, Mathematics and Computer Science; TU Delft Interactive Intelligence)

Brinkman, W.P. (mentor)
Al Owayyed, M. (mentor)
Eisemann, E. (graduation committee)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2023-07-03

Chatbots are tools that can potentially be utilized in chat-based child helpline training. In this type of training, the quality of the feedback received is of vital importance. This paper aims to analyze the automated feedback generated by such a chatbot. The domains analyzed include user comprehension, usefulness, and potential improvement points. In a user study, a formative assessment and two interviews were conducted for each domain, respectively.
For comprehensibility, all participants could easily understand the feedback report. They found that the bot could be easier to work around after reading the feedback, with the table being of much guidance. They found the transcript to be a welcome addition, but missing constructive feedback. Regarding improvement points, two of them were tightly related to the limitations of the chatbot, rather than the report itself. Extra guidance and instructions were deemed necessary by the participants, alongside an easier-to-read transcript interface.

Chatbot
feedback evaluation
feedback report
Evaluation

http://resolver.tudelft.nl/uuid:6ffeb7c6-8b92-443d-b45c-452ce528d7ac

https://doi.org/10.4121/85110f4b-40e1-4567-9f6e-e97c6337ad92

Student theses

bachelor thesis

© 2023 Nikola Ntasi