READY: Rethinking Monitoring for Large Distributed Systems

Research Project, 2024 – 2029

The objective of the READY project is to develop data-aware theoretical tools and dedicated algorithms to build tomorrow’s continuous monitoring systems for large distributed networks. Following the edge computing’s paradigm, the project intends to insert some clever and innovative monitoring logic close to data sources in order to drastically reduce the quantity of monitoring data that needs to be transmitted over the network. The monitoring problem consists in being able to keep track at a centralized component of important information about the distribution of locally generated values at any given time (such as the maximum, top-k highest, quantiles etc). This is a very challenging problem because such distributed monitoring requires in the worst case all data to be transmitted at all times, so we propose to adopt a novel data-driven approach for modeling the monitored data as randomly evolving objects. With this approach, typical characteristics of the continuously measured data will be exploited to lower the costs associated with the monitoring process. This will allow us to continuously monitor large distributed systems with both proven performance guarantees and demonstrated efficiency in practice, while ensuring the developed solutions to be IoT and CPS-friendly and integrable in high-performance data pipelines. The proposed solutions are expected to outperform all current state-of-the-art algorithms on three distinct metrics: (1) Communication Complexity (amount of data transmitted over the network by all nodes, including the coordinator node), (2) Processing Time (time spent on the distributed devices to compute monitoring decisions), and (3) Energy Efficiency (difference in energy consumption of using active monitoring on the distributed devices).