Single Window Stream Aggregation using Reconfigurable Hardware
Paper i proceeding, 2017

High throughput and low latency stream aggregation - and stream processing in general - is critical for many emerging applications that analyze massive volumes of continuously produced data on-the-fly, to make real time decisions. In many cases, high speed stream aggregation can be achieved incrementally by computing partial results for multiple windows. However, for particular problems, storing all incoming raw data to a single window before processing is more efficient or even the only option. This paper presents the first FPGA-based single window stream aggregation design. Using Maxeler's dataflow engines (DFEs), up to 8 million tuples-per-second can be processed (1.1 Gbps) offering 1-2 orders of magnitude higher throughput than a state-of-the-art stream processing software system. DFEs have a direct feed of incoming data from the network as well as direct access to off-chip DRAM processing a tuple in less than 4 mu sec, 4 orders of magnitude lower latency than software. The proposed approach is able to support challenging queries required in realistic stream processing problems (e.g. holistic functions). Our design offers aggregation for up to 1 million concurrently active keys and handles large windows storing up to 6144 values (24 KB) per key.

Författare

Prajith Ramakrishnan Geethakumari

Chalmers, Data- och informationsteknik, Datorteknik

Vincenzo Massimiliano Gulisano

Chalmers, Data- och informationsteknik, Nätverk och system

Joel Bo Svensson

Chalmers, Data- och informationsteknik, Datorteknik

Pedro Petersen Moura Trancoso

Chalmers, Data- och informationsteknik, Datorteknik

University of Cyprus

Ioannis Sourdis

Chalmers, Data- och informationsteknik, Datorteknik

2017 INTERNATIONAL CONFERENCE ON FIELD PROGRAMMABLE TECHNOLOGY (ICFPT)

112-119

16th IEEE International Conference on Field-Programmable Technology (ICFPT)
Melbourne, Australia,

Ämneskategorier

Datorteknik

Systemvetenskap

Datavetenskap (datalogi)