Single Window Stream Aggregation using Reconfigurable Hardware
Paper in proceeding, 2017

High throughput and low latency stream aggregation - and stream processing in general - is critical for many emerging applications that analyze massive volumes of continuously produced data on-the-fly, to make real time decisions. In many cases, high speed stream aggregation can be achieved incrementally by computing partial results for multiple windows. However, for particular problems, storing all incoming raw data to a single window before processing is more efficient or even the only option. This paper presents the first FPGA-based single window stream aggregation design. Using Maxeler's dataflow engines (DFEs), up to 8 million tuples-per-second can be processed (1.1 Gbps) offering 1-2 orders of magnitude higher throughput than a state-of-the-art stream processing software system. DFEs have a direct feed of incoming data from the network as well as direct access to off-chip DRAM processing a tuple in less than 4 mu sec, 4 orders of magnitude lower latency than software. The proposed approach is able to support challenging queries required in realistic stream processing problems (e.g. holistic functions). Our design offers aggregation for up to 1 million concurrently active keys and handles large windows storing up to 6144 values (24 KB) per key.

Author

Prajith Ramakrishnan Geethakumari

Chalmers, Computer Science and Engineering (Chalmers), Computer Engineering (Chalmers)

Vincenzo Massimiliano Gulisano

Chalmers, Computer Science and Engineering (Chalmers), Networks and Systems (Chalmers)

Joel Bo Svensson

Chalmers, Computer Science and Engineering (Chalmers), Computer Engineering (Chalmers)

Pedro Petersen Moura Trancoso

Chalmers, Computer Science and Engineering (Chalmers), Computer Engineering (Chalmers)

University of Cyprus

Ioannis Sourdis

Chalmers, Computer Science and Engineering (Chalmers), Computer Engineering (Chalmers)

2017 INTERNATIONAL CONFERENCE ON FIELD PROGRAMMABLE TECHNOLOGY (ICFPT)

Vol. 2018-January 112-119
978-1-5386-2656-6 (ISBN)

16th IEEE International Conference on Field-Programmable Technology (ICFPT)
Melbourne, Australia,

Subject Categories

Computer Engineering

Information Science

Computer Science

DOI

10.1109/FPT.2017.8280128

More information

Latest update

7/11/2024