Print Email Facebook Twitter S-QUERY Title S-QUERY: Opening the Black Box of Internal Stream Processor State Author Verheijde, Jim (Student TU Delft) Karakoidas, Vassilios (Delivery Hero SE) Fragkoulis, M. (TU Delft Web Information Systems) Katsifodimos, A (TU Delft Web Information Systems) Contributor O'Conner, L. (editor) Date 2022 Abstract Distributed streaming dataflow systems have evolved into scalable and fault-tolerant production-grade systems. Their applicability has departed from the mere analysis of streaming windows and complex-event processing, and now includes cloud applications and machine learning inference. Although the advancements in the state management of streaming systems have contributed significantly to their maturity, the internal state of streaming operators has been so far hidden from external applications. However, that internal state can be seen as a materialized view that can be used for analytics, monitoring, and debugging. In this paper we argue that exposing the internal state of streaming systems to outside applications by making it queryable, opens the road for novel use cases. To this end, we introduce S-QUERY: an approach and reference architecture where the state of stream processors can be queried - either live or through snapshots, achieving different isolation levels. We show how this new capability can be implemented in an existing open-source stream processor, and how queryable state can affect the performance of such a system. Our experimental evaluation suggests that the snapshot configuration adds only up to 8ms latency in the 99.99thpercentile and negligible increase in 0-90thpercentiles. To reference this document use: http://resolver.tudelft.nl/uuid:464df924-f824-4c27-8607-9152df4c6d41 DOI https://doi.org/10.1109/ICDE53745.2022.00103 Publisher IEEE, Piscataway Embargo date 2023-07-01 ISBN 978-1-6654-0884-4 Source Proceedings of the 2022 IEEE 38th International Conference on Data Engineering (ICDE) Event 2022 IEEE 38th International Conference on Data Engineering (ICDE), 2022-05-09 → 2022-05-12, Kuala Lumpur, Malaysia Series Proceedings - International Conference on Data Engineering, 1084-4627, 2022-May Bibliographical note Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. Part of collection Institutional Repository Document type conference paper Rights © 2022 Jim Verheijde, Vassilios Karakoidas, M. Fragkoulis, A Katsifodimos Files PDF S_QUERY_Opening_the_Black ... _State.pdf 2.29 MB Close viewer /islandora/object/uuid:464df924-f824-4c27-8607-9152df4c6d41/datastream/OBJ/view