Time-SWAD: A dataflow engine for time-based single window stream aggregation

Paper in proceeding, 2019

High throughput and low latency streaming aggregation is essential for many applications that analyze massive volumes of data in real-time. Incoming data need to be stored in a single sliding window before processing, in cases where incremental aggregations are wasteful or not possible at all; this puts tremendous pressure to the memory bandwidth. In addition, particular problems call for time-based windows, defined by a time-interval, where the amount of data per window may vary and as a consequence are more challenging to handle. This paper describes Time-SWAD, the first accelerator for time-based single-window stream aggregation. Time-SWAD is a dataflow engine (DFE), implemented on a Maxeler machine, offering high processing throughput, up to 150 Mtuples/sec, similar to related GPU systems, which however do not support both time-based and single windows. It uses a direct feed of incoming data from the network and has direct access to off-chip DRAM, enabling ultra-low processing latency of 1-10 μsec, at least 4 orders of magnitude lower than software-based solutions.