View in EDS

SEIZE: Runtime Inspection for Parallel Dataflow Systems.

Saved in:
Bibliographic Details
Title: SEIZE: Runtime Inspection for Parallel Dataflow Systems.
Authors: Li, Youfu1 youfuli@cs.ucla.edu, Interlandi, Matteo2 mainterl@microsoft.com, Psallidas, Fotis2 fotis.psallidas@microsoft.com, Wang, Wei1 weiwang@cs.ucla.edu, Zaniolo, Carlo1 zaniolo@cs.ucla.edu
Source: IEEE Transactions on Parallel & Distributed Systems. Apr2021, Vol. 32 Issue 4, p842-854. 13p.
Subjects: Computer workstation clusters, Task analysis
Abstract: Many Data-Intensive Scalable Computing (DISC) Systems provide easy-to-use functional APIs, and efficient scheduling and execution strategies allowing users to build concise data-parallel programs. In these systems, data transformations are concealed by exposed APIs, and intermediate execution states are masked under dataflow transitions. Consequently, many crucial features and optimizations (e.g., debugging, data provenance, runtime skew detection), which require runtime datafow states, are not well-supported. Inspired by our experience in implementing features and optimizations over DISC systems, we present $\mathsf {SEIZE}$ SEIZE , a unified framework that enables dataflow inspection—wiretapping the data-path with listening logic—in MapReduce-style programming model. We generalize our lessons learned by providing a set of primitives defining dataflow inspection, orchestration options for different inspection granularities, and operator decomposition and dataflow punctuation strategy for dataflow intervention. We demonstrate the generality and flexibility of the approach by deploying $\mathsf {SEIZE}$ SEIZE in both Apache Spark and Apache Flink, and by implementing a prototype runtime query optimizer for Spark. Our experiments show that, the overhead introduced by the inspection logic is most of the time negligible (less than 5 percent in Spark and 10 percent in Flink). [ABSTRACT FROM AUTHOR]
Copyright of IEEE Transactions on Parallel & Distributed Systems is the property of IEEE and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Description
Abstract:Many Data-Intensive Scalable Computing (DISC) Systems provide easy-to-use functional APIs, and efficient scheduling and execution strategies allowing users to build concise data-parallel programs. In these systems, data transformations are concealed by exposed APIs, and intermediate execution states are masked under dataflow transitions. Consequently, many crucial features and optimizations (e.g., debugging, data provenance, runtime skew detection), which require runtime datafow states, are not well-supported. Inspired by our experience in implementing features and optimizations over DISC systems, we present $\mathsf {SEIZE}$ SEIZE , a unified framework that enables dataflow inspection—wiretapping the data-path with listening logic—in MapReduce-style programming model. We generalize our lessons learned by providing a set of primitives defining dataflow inspection, orchestration options for different inspection granularities, and operator decomposition and dataflow punctuation strategy for dataflow intervention. We demonstrate the generality and flexibility of the approach by deploying $\mathsf {SEIZE}$ SEIZE in both Apache Spark and Apache Flink, and by implementing a prototype runtime query optimizer for Spark. Our experiments show that, the overhead introduced by the inspection logic is most of the time negligible (less than 5 percent in Spark and 10 percent in Flink). [ABSTRACT FROM AUTHOR]
ISSN:10459219
DOI:10.1109/TPDS.2020.3035170