STRETCH: Scalable and elastic deterministic streaming analysis with virtual shared-nothing parallelism
Paper i proceeding, 2019

Despite the established scientific knowledge on efficient parallel and elastic data stream processing, it is challenging to combine generality and high level of abstraction (targeting ease of use) with fine-grained processing aspects (targeting efficiency) in stream processing frameworks. Towards this goal, we propose STRETCH, a framework that aims at guaranteeing (i) high efficiency in throughput and latency of stateful analysis and (ii) fast elastic reconfigurations (without requiring state transfer) for intra-node streaming applications. To achieve these, we introduce virtual shared-nothing parallelization and propose a scheme to implement it in STRETCH, enabling users to leverage parallelization techniques while also taking advantage of shared-memory synchronization, which has been proven to boost the scaling-up of streaming applications while supporting determinism. We provide a fully-implemented prototype and, together with a thorough evaluation, correctness proofs for its underlying claims supporting determinism and a model (also validated empirically) of virtual shared-nothing and pure shared-nothing scalability behavior. As we show, STRETCH can match the throughput and latency figures of the front of state-of-the-art solutions, while also achieving fast elastic reconfigurations (taking only a few milliseconds).

Shared-nothing parallelism

Elasticity

Data streaming

Scalability

Författare

Hannaneh Najdataei

Chalmers, Data- och informationsteknik, Nätverk och system

Ioannis Nikolakopoulos

Chalmers, Data- och informationsteknik, Nätverk och system

Marina Papatriantafilou

Chalmers, Data- och informationsteknik, Nätverk och system

Philippas Tsigas

Chalmers, Data- och informationsteknik, Nätverk och system

Vincenzo Massimiliano Gulisano

Chalmers, Data- och informationsteknik, Nätverk och system

DEBS 2019 - Proceedings of the 13th ACM International Conference on Distributed and Event-Based Systems

7-18

13th ACM International Conference on Distributed and Event-Based Systems (DEBS 2019)
Darmstadt, Germany,

HAREN: Självdistribuerad och anpassningsbar dataströmningsanalys i dimman

Vetenskapsrådet (VR) (2016-03800), 2017-01-01 -- 2020-12-31.

Molnbaserade produkter och produktion (FiC)

Stiftelsen för Strategisk forskning (SSF) (GMT14-0032), 2016-01-01 -- 2020-12-31.

Ämneskategorier

Datorteknik

Inbäddad systemteknik

Datorsystem

Styrkeområden

Informations- och kommunikationsteknik

Produktion

DOI

10.1145/3328905.3329509

Mer information

Senast uppdaterat

2024-01-03