Beyond Microbenchmarks: The SPEC-RG Vision for a Comprehensive Serverless Benchmark
Paper in proceeding, 2020

Serverless computing services, such as Function-as-a-Service (FaaS), hold the attractive promise of a high level of abstraction and high performance, combined with the minimization of operational logic. Several large ecosystems of serverless platforms, both open- and closed-source, aim to realize this promise. Consequently, a lucrative market has emerged. However, the performance trade-offs of these systems are not well-understood. Moreover, it is exactly the high level of abstraction and the opaqueness of the operational-side that make performance evaluation studies of serverless platforms challenging. Learning from the history of IT platforms, we argue that a benchmark for serverless platforms could help address this challenge. We envision a comprehensive serverless benchmark, which we contrast to the narrow focus of prior work in this area. We argue that a comprehensive benchmark will need to take into account more than just runtime overhead, and include notions of cost, realistic workloads, more (open-source) platforms, and cloud integrations. Finally, we show through preliminary real-world experiments how such a benchmark can help compare the performance overhead when running a serverless workload on state-of-the-art platforms.

serverless computing

performance

function-as-a-service

Author

Erwin van Eyk

Vrije Universiteit Amsterdam

Joel Scheuner

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers)

Simon Eisman

University of Würzburg

Cristina L. Abad

Escuela Superior Politecnica del Litoral

Alexandru Iosup

Vrije Universiteit Amsterdam

Companion of the ACM/SPEC International Conference on Performance Engineering

26-31
978-1-4503-7109-4 (ISBN)

3rd Workshop on Hot Topics in Cloud Computing Performance, HotCloudPerf 2020
Edmonton, Canada,

Areas of Advance

Information and Communication Technology

Subject Categories

Software Engineering

Computer Science

DOI

10.1145/3375555.3384381

More information

Latest update

1/22/2021