Print Email Facebook Twitter A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms Title A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms Author Kille, Benjamin (Technical University of Berlin) Lommatzsch, Andreas (Technical University of Berlin) Hopfgartner, Frank (University of Glasgow) Larson, M.A. (TU Delft Multimedia Computing) de Vries, A.P. (Radboud Universiteit Nijmegen) Date 2017 Abstract Recommender System research has evolved to focus on developing algorithms capable of high performance in online systems. This development calls for a new evaluation infrastructure that supports multi-dimensional evaluation of recommender systems. Today’s researchers should analyze algorithms with respect to a variety of aspects including predictive performance and scalability. Researchers need to subject algorithms to realistic conditions in online A/B tests.We introduce two resources supporting such evaluation methodologies: the new data set of stream recommendation interactions released for CLEF NewsREEL 2017, and the new Open Recommendation Platform (ORP). The data set allows researchers to study a stream recommendation problem closely by “replaying” it locally, and ORP makes it possible to take this evaluation “live” in a livinglab scenario. Specifically, ORP allows researchers to deploy their algorithms in a live stream to carry out A/B tests. To our knowledge, NewsREEL is the first online news recommender system resource to be put at the disposal of the research community. In order to encourage others to develop comparable resources for a wide range of domains, we present a list of practical lessons learned in thedevelopment of the dataset and ORP. Subject streamsrecommender systemmulti--dimensional benchmarking To reference this document use: http://resolver.tudelft.nl/uuid:7430fe8b-8f63-4233-9189-e0a92d91621e DOI https://doi.org/10.1145/3077136.3080726 Publisher Association for Computing Machinery (ACM), New York, NY ISBN 978-1-4503-5022-8 Source Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017 Event SIGIR 2017, 2017-08-07 → 2017-08-11, Tokyo, Japan Part of collection Institutional Repository Document type conference paper Rights © 2017 Benjamin Kille, Andreas Lommatzsch, Frank Hopfgartner, M.A. Larson, A.P. de Vries Files PDF 35744048.pdf 967.33 KB Close viewer /islandora/object/uuid:7430fe8b-8f63-4233-9189-e0a92d91621e/datastream/OBJ/view