Quality trade-offs in ML-enabled systems: a multiple-case study

Paper in proceeding, 2025

When building a machine-learning-enabled system, quality objectives are achieved through architectural and non-architectural tactics, including general ones as well as specific ones that address machine learning specifics, such as the focus on data. However, implementing these tactics typically compromises other quality attributes that are not the primary focus of the tactic at hand. Previous research has investigated quality aspects and tactics for machine-learning-enabled systems, but there is a lack of detailed insights on quality trade-offs observed in industrial practice, and how companies address them. A study in this direction could especially help start-ups and SMEs to benefit from the insights of other companies, and academics to develop improved tactics addressing these trade-offs in alternative, potentially more effective ways. In this paper, to fill this gap, we present a multiple-case study of four companies in the AI sphere. As AI solution providers, all companies are faced with a variety of quality priorities, tactics, and trade-offs in their addressed application domains. We find that our subject companies consistently address a common set of core quality priorities, encompassing reliability, functional suitability, and resource efficiency, which they address with recurring architectural tactics such as the use of cloud-based components for resource efficiency, and non-architectural ones such as Scrum practices for functionality suitability. Finally, we find a variety of trade-offs appearing in different companies with several recurring ones, two of them-efficiency vs. reliability, and system accuracy vs. explainability-manifesting themselves in three out of the four companies.