Print Email Facebook Twitter Effective crowdsourced generation of training data for chatbots natural language understanding Title Effective crowdsourced generation of training data for chatbots natural language understanding Author Bapat, Rucha (Student TU Delft) Kucherbaev, P. (TU Delft Web Information Systems) Bozzon, A. (TU Delft Web Information Systems) Date 2018 Abstract Chatbots are text-based conversational agents. Natural Language Understanding (NLU) models are used to extract meaning and intention from user messages sent to chatbots. The user experience of chatbots largely depends on the performance of the NLU model, which itself largely depends on the initial dataset the model is trained with. The training data should cover the diversity of real user requests the chatbot will receive. Obtaining such data is a challenging task even for big corporations. We introduce a generic approach to generate training data with the help of crowd workers, we discuss the approach workflow and the design of crowdsourcing tasks assuring high quality. We evaluate the approach by running an experiment collecting data for 9 different intents. We use the collected training data to train a natural language understanding model. We analyse the performance of the model under different training set sizes for each intent. We provide recommendations on selecting an optimal confidence threshold for predicting intents, based on the cost model of incorrect and unknown predictions. Subject Conversational agentsCrowdsourcingNatural language understanding To reference this document use: http://resolver.tudelft.nl/uuid:93fdad6e-7d97-4441-b3ca-a1e9dde637d7 DOI https://doi.org/10.1007/978-3-319-91662-0_8 Publisher Springer ISBN 978-3-319-91661-3 Source Web Engineering - 18th International Conference, ICWE 2018, Proceedings Event 18th International Conference on Web Engineering, ICWE 2018, 2018-06-05 → 2018-06-08, Caceres, Spain Series Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 0302-9743, 10845 LNCS Bibliographical note Accepted Author Manuscript Part of collection Institutional Repository Document type conference paper Rights © 2018 Rucha Bapat, P. Kucherbaev, A. Bozzon Files PDF paper61_ICWE2018_NLU.pdf 445.15 KB Close viewer /islandora/object/uuid:93fdad6e-7d97-4441-b3ca-a1e9dde637d7/datastream/OBJ/view