Beyond Microbenchmarks: The SPEC-RG Vision for a Comprehensive Serverless Benchmark
Paper i proceeding, 2020

Serverless computing services, such as Function-as-a-Service (FaaS), hold the attractive promise of a high level of abstraction and high performance, combined with the minimization of operational logic. Several large ecosystems of serverless platforms, both open- and closed-source, aim to realize this promise. Consequently, a lucrative market has emerged. However, the performance trade-offs of these systems are not well-understood. Moreover, it is exactly the high level of abstraction and the opaqueness of the operational-side that make performance evaluation studies of serverless platforms challenging. Learning from the history of IT platforms, we argue that a benchmark for serverless platforms could help address this challenge. We envision a comprehensive serverless benchmark, which we contrast to the narrow focus of prior work in this area. We argue that a comprehensive benchmark will need to take into account more than just runtime overhead, and include notions of cost, realistic workloads, more (open-source) platforms, and cloud integrations. Finally, we show through preliminary real-world experiments how such a benchmark can help compare the performance overhead when running a serverless workload on state-of-the-art platforms.

serverless computing




Erwin van Eyk

Vrije Universiteit Amsterdam

Joel Scheuner

Chalmers, Data- och informationsteknik, Software Engineering

Simon Eisman

Julius-Maximilians Universität Würzburg

Cristina L. Abad

Escuela Superior Politecnica del Litoral

Alexandru Iosup

Vrije Universiteit Amsterdam

Companion of the ACM/SPEC International Conference on Performance Engineering

978-1-4503-7109-4 (ISBN)

3rd Workshop on Hot Topics in Cloud Computing Performance, HotCloudPerf 2020
Edmonton, Canada,


Informations- och kommunikationsteknik



Datavetenskap (datalogi)



Mer information

Senast uppdaterat