Print Email Facebook Twitter The noticeability of behavioral changes of a conversational agent Title The noticeability of behavioral changes of a conversational agent: An evaluation of an agent-based social skills training system Author Zaheer, Ashma (TU Delft Electrical Engineering, Mathematics and Computer Science; TU Delft Interactive Intelligence) Contributor Brinkman, W.P. (mentor) Al Owayyed, M. (mentor) Eisemann, E. (graduation committee) Degree granting institution Delft University of Technology Programme Computer Science and Engineering Project CSE3000 Research Project Date 2023-07-03 Abstract Helpline counselors can be trained faster using agent-based social skills training systems. These systems utilize conversational agents that simulate interactions with users and provide feedback. This research evaluates the noticeability of behavior changes in such a conversational agent. By examining participants' ability to detect and interpret these changes, valuable insights can be gained to enhance the agent's representation of real-life scenarios and improve its effectiveness as a training tool.A mixed-methods approach was employed, combining quantitative data using descriptive statistics and qualitative data through content analysis. The quantitative analysis revealed a precision score of 93% and a recall score of 36%, indicating a higher accuracy in correctly identifying behavior changes compared to capturing all instances of change. The qualitative analysis revealed that participants' perceptions fell within four distinct categories: communication style, positive emotion, negative emotion, and tone & attitude. Participants demonstrated heightened attentiveness to changes in positive behavior, particularly instances of happiness expressed by the agent, and displayed less attention towards changes in negative behavior.In conclusion, this research demonstrates that behavior changes in the conversational agent are noticeable, albeit with varying degrees of attention across different types of behavior. Further exploration with a larger sample size is warranted to ascertain the extent of noticeability. The diverse range of behavior changes indicates the agent's capability to adapt and exhibit visible shifts in behavior. These insights can inform the refinement and development of agent-based training systems, ultimately enhancing the training experiences of chat-based helpline counselors. Subject EducationArtificial intelligencetraining systemsAgents behavior To reference this document use: http://resolver.tudelft.nl/uuid:e81a9db3-ca5a-44ec-a4bd-ed89094eef6c Part of collection Student theses Document type bachelor thesis Rights © 2023 Ashma Zaheer Files PDF Final_Paper.pdf 988.89 KB Close viewer /islandora/object/uuid:e81a9db3-ca5a-44ec-a4bd-ed89094eef6c/datastream/OBJ/view