Streamzip: Compressed Sliding-Windows for Stream Aggregation
Paper in proceeding, 2021

High performance stream aggregation is critical for many emerging applications that analyze massive volumes of data. Incoming data needs to be stored in a sliding-window before processing, in case the aggregation functions cannot be computed incrementally. Updating the window with new incoming values and reading it to feed the aggregation functions are the two primary steps in stream aggregation. Although window updates can be supported efficiently using multi-level queues, frequent window aggregations remain a performance bottleneck as they put tremendous pressure on the memory bandwidth and capacity. This paper addresses this problem by introducing Streamzip, a dataflow stream aggregation engine that is able to compress the sliding-windows. Streamzip deals with a number of data and control dependency challenges to integrate a compressor in the stream aggregation pipeline and alleviate the memory pressure posed by frequent aggregations. In doing so, Streamzip offers higher throughput as well as larger effective window capacity to support larger problems. Streamzip supports diverse compression algorithms offering both lossless and lossy compression to integers as well as floating point numbers. Compared to designs without compression, Streamzip lossless and lossy designs achieve up to 7.5x and 22x higher throughput, while improving the effective memory capacity by up to 5x and 23x, respectively.







Prajith Ramakrishnan Geethakumari

Chalmers, Computer Science and Engineering (Chalmers), Computer Engineering (Chalmers)

Ioannis Sourdis

Chalmers, Computer Science and Engineering (Chalmers), Computer Engineering (Chalmers)

2021 International Conference on Field-Programmable Technology (ICFPT) 203-211
978-166542010-5 (ISBN)

International Conference on Field-Programmable Technology (FPT’21)
Auckland, Virtual, New Zealand,

ScalaNetS: Skalbara nätverks- och dataströmsberäkningar

Swedish Research Council (VR) (Dnr2016-05231), 2017-01-01 -- 2020-12-31.

Subject Categories

Computer Engineering

Computer Science

Computer Systems

Areas of Advance

Information and Communication Technology





More information

Latest update

2/1/2022 2