Print Email Facebook Twitter A Coflow-based Co-optimization Framework for High-performance Data Analytics Title A Coflow-based Co-optimization Framework for High-performance Data Analytics Author Cheng, Long (Eindhoven University of Technology) Wang, Ying (Chinese Academy of Sciences) Pei, Yulong (Eindhoven University of Technology) Epema, D.H.J. (TU Delft Dataintensive Systems) Date 2017 Abstract Efficient execution of distributed database operators such as joining and aggregating is critical for the performance of big data analytics. With the increase of the compute speedup of modern CPUs, reducing the networkcommunication time of these operators in large systems is becoming increasingly important, and also challenging current techniques. Significant performance improvements have been achieved by using state-of-the-art methods, such as reducing network traffic designed in the data management domain, and data flow scheduling in the data communications domain.However, the proposed techniques in both fields just view each other as a black box, and performance gains from a co-optimization perspective have not yet been explored. In this paper, based on current research in coflow scheduling,we propose a novel Coflow-based Co-optimization Framework(CCF), which can co-optimize application-level data movementand network-level data communications for distributed operators,and consequently contribute to their performance inlarge distributed environments. We present the detailed designand implementation of CCF, and conduct an experimentalevaluation of CCF using large-scale simulations on large datajoins. Our results demonstrate that CCF can always performfaster than current approaches on network communications inlarge-scale distributed scenarios. Subject big datacoflow schedulingdistributed joinsnetwork communicationsdata-intensive applications To reference this document use: http://resolver.tudelft.nl/uuid:4ffef8f8-5ca3-4a47-a321-933a23ee0282 DOI https://doi.org/10.1109/ICPP.2017.48 Publisher IEEE, Los Alamitos, CA ISBN 978-1-5386-1042-8 Source Proceedings - 46th International Conference on Parallel Processing, ICPP 2017 Event ICPP 2017, 2017-08-14 → 2017-08-17, Bristol, United Kingdom Part of collection Institutional Repository Document type conference paper Rights © 2017 Long Cheng, Ying Wang, Yulong Pei, D.H.J. Epema Files PDF ICPP_Cheng_Epema_2017.pdf 355.2 KB Close viewer /islandora/object/uuid:4ffef8f8-5ca3-4a47-a321-933a23ee0282/datastream/OBJ/view