Characterizing and Mitigating Performance Variability in Parallel Applications on Modern HPC multicore Systems

Paper in proceeding, 2025

In high-performance computing (HPC), OpenMP has become the de facto programming model for shared-memory systems. However, running OpenMP-based parallel applications on multicore systems often faces the challenge of performance variability, particularly as core counts increase in modern HPC clusters. Factors spanning from Operating Systems (OS) and hardware feature to OpenMP implementation can significantly impact performance stability. This paper evaluates execution time variability across five multicore systems from multiple HPC clusters, covering two different ISAs and using five OpenMP benchmarks and a real-world mini-app compiled with both gcc and llvm/clang. We analyze the effects of various factors such as thread-pinning, OpenMP runtime implementations, OpenMP scalability, simultaneous multithreading (SMT), core resource reservation, frequency scaling, and platform-specific features such as hybrid architecture core configurations. Our findings highlight the complex interplay of these factors in performance variability and propose lightweight mitigation strategies to enhance the stability of OpenMP programs for developers and system users.