Viper: Communication-layer determinism and scaling in low-latency stream processing
Paper in proceedings, 2018

Stream Processing Engines (SPEs) process continuous streams of data and produce up-to-date results in a real-time fashion, typically through one-at-a-time tuple analysis. When looking into the vital SPE processing properties required from applications, determinism has a strong position besides scalability in throughput and low processing latency. SPEs scale in throughput and latency by relying on shared-nothing parallelism, deploying multiple copies of each operator to which tuples are distributed based on the semantics of the operator. The coordination of the asynchronous analysis of parallel operators required to enforce determinism is then carried out by additional dedicated sorting operators. In this work we shift such costly coordination to the communication layer of the SPE. Specifically, we extend earlier work on shared-memory implementations of deterministic operators and provide a communication module (Viper) which can be integrated in the SPE communication layer. Using Apache Storm and the Linear Road benchmark, we show the benefits that can be achieved by our approach in terms of throughput and energy efficiency of SPEs implementing one-at-a-time analysis.

Data streaming

Low-latency shared-nothing and shared-memory parallelism stream processing engines

Author

Ivan Walulya

Chalmers, Computer Science and Engineering (Chalmers), Networks and Systems (Chalmers)

Ioannis Nikolakopoulos

Chalmers, Computer Science and Engineering (Chalmers), Networks and Systems (Chalmers)

Vincenzo Massimiliano Gulisano

Chalmers, Computer Science and Engineering (Chalmers), Networks and Systems (Chalmers)

Marina Papatriantafilou

Chalmers, Computer Science and Engineering (Chalmers), Networks and Systems (Chalmers)

Philippas Tsigas

Chalmers, Computer Science and Engineering (Chalmers), Networks and Systems (Chalmers)

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

03029743 (ISSN) 16113349 (eISSN)

Vol. 10659 129-140

International Workshops on Parallel Processing, Euro-Par 2017
Santiago de Compostela, Spain,

Subject Categories

Computer Engineering

Computer Science

Computer Systems

DOI

10.1007/978-3-319-75178-8_11

More information

Latest update

4/20/2018