Haren: A Framework for Ad-Hoc Thread Scheduling Policies for Data Streaming Applications
Paper in proceedings, 2019

In modern Stream Processing Engines (SPEs), numerous diverse applications, which can differ in aspects such as cost, criticality or latency sensitivity, can co-exist in the same computing node. When these differences need to be considered to control the performance of each application, custom scheduling of operators to threads is of key importance (e.g., when a smart vehicle needs to ensure that safety-critical applications always have access to computational power, while other applications are given lower, variable priorities).

Many solutions have been proposed regarding schedulers that allocate threads to operators to optimize specific metrics (e.g., latency) but there is still lack of a tool that allows arbitrarily complex scheduling strategies to be seamlessly plugged on top of an SPE. We propose Haren to fill this gap. More specifically, we (1) formalize the thread scheduling problem in stream processing in a general way, allowing to define ad-hoc scheduling policies, (2) identify the bottlenecks and the opportunities of scheduling in stream processing, (3) distill a compact interface to connect Haren with SPEs, enabling rapid testing of various scheduling policies, (4) illustrate the usability of the framework by integrating it into an actual SPE and (5) provide a thorough evaluation. As we show, Haren makes it is possible to adapt the use of computational resources over time to meet the goals of a variety of scheduling policies.

Scheduling

Stream processing

Middleware

Author

Dimitrios Palyvos-Giannas

Chalmers, Computer Science and Engineering (Chalmers), Networks and Systems (Chalmers)

Vincenzo Massimiliano Gulisano

Chalmers, Computer Science and Engineering (Chalmers), Networks and Systems (Chalmers)

Marina Papatriantafilou

Chalmers, Computer Science and Engineering (Chalmers), Networks and Systems (Chalmers)

Proceedings of the 13th ACM International Conference on Distributed and Event-based Systems

19-30

13th ACM International Conference on Distributed and Event-based Systems
Darmstadt, Germany,

Future factories in the Cloud (FiC)

Swedish Foundation for Strategic Research (SSF), 2016-01-01 -- 2020-12-31.

HARE: Self-deploying and Adaptive Data Streaming Analytics in Fog Architectures

Swedish Research Council (VR), 2017-01-01 -- 2020-12-31.

INDEED

Chalmers, 2016-01-01 -- 2020-12-31.

STAMINA - GE

Göteborg Energi, Foundation for Research and Developmen, 2017-01-01 -- 2021-12-31.

Subject Categories

Computer Engineering

Computer Science

Computer Systems

Areas of Advance

Information and Communication Technology

DOI

10.1145/3328905.3329505

More information

Latest update

11/13/2019