Engineering for a science-centric experimentation platform
Paper in proceeding, 2020

Netflix is an internet entertainment service that routinely employs experimentation to guide strategy around product innovations. As Netflix grew, it had the opportunity to explore increasingly specialized improvements to its service, which generated demand for deeper analyses supported by richer metrics and powered by more diverse statistical methodologies. To facilitate this, and more fully harness the skill sets of both engineering and data science, Netflix engineers created a science-centric experimentation platform that leverages the expertise of scientists from a wide range of backgrounds working on data science tasks by allowing them to make direct code contributions in the languages used by them (Python and R). Moreover, the same code that runs in production is able to be run locally, making it straightforward to explore and graduate both metrics and causal inference methodologies directly into production services. In this paper, we provide two main contributions. Firstly, we report on the architecture of this platform, with a special emphasis on its novel aspects: how it supports science-centric end-to-end workflows without compromising engineering requirements. Secondly, we describe its approach to causal inference, which leverages the potential outcomes conceptual framework to provide a unified abstarction layer for arbitrary statistical models and methodologies.

Software architecture

A/B testing

Science-centric

Experimentation

Causal inference

Author

Nikos Diamantopoulos

Netflix, Inc.

Jeffrey Wong

Netflix, Inc.

David Issa Mattos

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers)

Ilias Gerostathopoulos

Technical University of Munich

Vrije Universiteit Amsterdam

Matthew Wardrop

Netflix, Inc.

Tobias Mao

Netflix, Inc.

Colin McFarland

Netflix, Inc.

Proceedings - International Conference on Software Engineering

02705257 (ISSN)

191-200 3381349
9781450371230 (ISBN)

42nd ACM/IEEE International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP 2020
Virtual, Online, South Korea,

Subject Categories

Software Engineering

Computer Science

Computer Systems

DOI

10.1145/3377813.3381349

More information

Latest update

11/9/2020