LMQ-Sketch: Lagom Multi-Query Sketch for High-Rate Online Analytics

Paper i proceeding, 2025

Data sketches balance resource efficiency with controllable approximations for extracting features in high-volume, high-rate data. Two important points of interest are highlighted separately in recent works; namely, to (1) answer multiple types of queries from a single data structure built in one pass over the data, and (2) perform both queries and updates concurrently. In this work, we now tackle the new challenges arising when combining these useful directions together.
We investigate the trade-offs around efficiency, consistency, and accuracy to be balanced and synthesize key ideas into LMQ-Sketch, a single, composite data sketch supporting concurrent updates and multiple queries (frequency point queries, frequency moments F₁, and F₂ as representative selection). Our method "Lagom" is a cornerstone of LMQ-Sketch for low-latency global querying (<100µs), combining freshness, timeliness, and accuracy with a low memory footprint and high throughput (>2B updates/s). We analyze and evaluate the accuracy of Lagom, which builds on a simple geometric argument and efficiently combines work distribution with synchronization for proper concurrency semantics - monotonicity of operations and intermediate value linearizability. Comparing with state-of-the-art methods, which, as mentioned, provide either mixed queries or concurrency separately, LMQ-Sketch shows highly competitive throughput, with additional accuracy guarantees and concurrency semantics, while also reducing the required memory budget by an order of magnitude. We expect the methodology to have broader impact on concurrent multi-query sketches.