Viper: Communication-layer determinism and scaling in low-latency stream processing
Paper i proceeding, 2018

Stream Processing Engines (SPEs) process continuous streams of data and produce up-to-date results in a real-time fashion, typically through one-at-a-time tuple analysis. When looking into the vital SPE processing properties required from applications, determinism has a strong position besides scalability in throughput and low processing latency. SPEs scale in throughput and latency by relying on shared-nothing parallelism, deploying multiple copies of each operator to which tuples are distributed based on the semantics of the operator. The coordination of the asynchronous analysis of parallel operators required to enforce determinism is then carried out by additional dedicated sorting operators. In this work we shift such costly coordination to the communication layer of the SPE. Specifically, we extend earlier work on shared-memory implementations of deterministic operators and provide a communication module (Viper) which can be integrated in the SPE communication layer. Using Apache Storm and the Linear Road benchmark, we show the benefits that can be achieved by our approach in terms of throughput and energy efficiency of SPEs implementing one-at-a-time analysis.

Data streaming

Low-latency shared-nothing and shared-memory parallelism stream processing engines

Författare

Ivan Walulya

Chalmers, Data- och informationsteknik, Nätverk och system

Ioannis Nikolakopoulos

Chalmers, Data- och informationsteknik, Nätverk och system

Vincenzo Massimiliano Gulisano

Chalmers, Data- och informationsteknik, Nätverk och system

Marina Papatriantafilou

Chalmers, Data- och informationsteknik, Nätverk och system

Philippas Tsigas

Chalmers, Data- och informationsteknik, Nätverk och system

Lecture Notes in Computer Science

0302-9743 (ISSN)

Vol. 10659 129-140

International Workshops on Parallel Processing, Euro-Par 2017
Santiago de Compostela, Spain,

Ämneskategorier

Datorteknik

Datavetenskap (datalogi)

Datorsystem

DOI

10.1007/978-3-319-75178-8_11

Mer information

Senast uppdaterat

2018-04-20