An Algorithm for Tunable Memory Compression of Time-Based Windows for Stream Aggregates

Paper in proceeding, 2024

Cloud-to-edge device continuums transform raw data into insights through data-intensive processing paradigms such as stream processing and frameworks known as Stream Processing Engines (SPEs). The control of resources in streaming applications within and across such continuums has been a prominent topic in the literature. While several techniques have been proposed to control resources like CPU, limited control exists for other resources such as memory.

Based on this observation, this work proposes an algorithm for streaming aggregation that allows for control of memory usage through lossless compression. The algorithm provides a "knob" to control the amount of state that should be compressed, prioritizing the compression of old over fresh data when performing streaming aggregation. Together with a detailed algorithmic description, this work presents preliminary results from a fully implemented prototype on top of the Liebre SPE, showing the effectiveness of the proposed approach.