Streaming Aggregation using Reconfigurable Hardware
Licentiatavhandling, 2019

High throughput and low latency streaming aggregation is essential for many applications that analyze massive volumes of data in real-time. In many cases, high speed stream aggregation can be achieved incrementally by computing partial results for multiple windows. However, for particular problems, temporarily storing all incoming raw data to a single window before processing is more efficient or even the only option. This thesis presents the first FPGA-based single window stream aggregation designs for tuple-based and time-based windowing policies. The proposed approach is able to support challenging queries required in realistic stream processing problems. More precisely, holistic, distributive, and algebraic aggregation functions, as well as custom ones can be supported. Our designs offer aggregation for large number of concurrently active keys and handles large window sizes and frequent aggregations. Maxeler's dataflow engines (DFEs), which suit well the stream processing characteristics, are used to implement the designs. DFEs have a direct feed of incoming data from the network as well as direct access to off-chip DRAM. The tuple-based single window DFE processes up to 8 million tuples-per-second (1.1 Gbps) offering 1-2 orders of magnitude higher throughput than a state-of-the-art stream processing software system. The processing latency is less than 4 usec, 4 orders of magnitude lower latency than software. The time-based single-window stream aggregation DFE offers high processing throughput, up to 150 Mtuples/sec, similar to related GPU systems, which however do not support both time-based and single windows. It also offers an ultra-low processing latency of 1-10 usec, at least 4 orders of magnitude lower than software-based solutions.

Streaming

Aggregation

Dataflow

FPGA

ED, lecture hall, Hörsalsvägen 11, EDIT trappa C, D och H
Opponent: Prof. Jens Teubner, TU Dortmund, Germany

Författare

Prajith Ramakrishnan Geethakumari

Chalmers, Data- och informationsteknik, Datorteknik, Computer Systems

Single Window Stream Aggregation using Reconfigurable Hardware

2017 INTERNATIONAL CONFERENCE ON FIELD PROGRAMMABLE TECHNOLOGY (ICFPT),; (2018)p. 112-119

Paper i proceeding

Prajith Ramakrishnan Geethakumari, Vincenzo Gulisano, Pedro Trancoso and Ioannis Sourdis, Time-SWAD: A Dataflow Engine for Time-based Single Window Stream Aggregation

ScalaNetS: Skalbara nätverks- och dataströmsberäkningar

Vetenskapsrådet (VR), 2017-01-01 -- 2020-12-31.

Styrkeområden

Informations- och kommunikationsteknik

Ämneskategorier

Inbäddad systemteknik

Datorsystem

Technical report L - School of Computer Science and Engineering, Chalmers University of Technology: 205L

Utgivare

Chalmers tekniska högskola

ED, lecture hall, Hörsalsvägen 11, EDIT trappa C, D och H

Opponent: Prof. Jens Teubner, TU Dortmund, Germany

Mer information

Senast uppdaterat

2019-11-14