Data Pipeline Management in Practice: Challenges and Opportunities
Paper i proceeding, 2020

Data pipelines involve a complex chain of interconnected activities that starts with a data source and ends in a data sink. Data pipelines are important for data-driven organizations since a data pipeline can process data in multiple formats from distributed data sources with minimal human intervention, accelerate data life cycle activities, and enhance productivity in data-driven enterprises. However, there are challenges and opportunities in implementing data pipelines but practical industry experiences are seldom reported. The findings of this study are derived by conducting a qualitative multiple-case study and interviews with the representatives of three companies. The challenges include data quality issues, infrastructure maintenance problems, and organizational barriers. On the other hand, data pipelines are implemented to enable traceability, fault-tolerance, and reduce human errors through maximizing automation thereby producing high-quality data. Based on multiple-case study research with five use cases from three case companies, this paper identifies the key challenges and benefits associated with the implementation and use of data pipelines.

Organizational

Opportunities

Data quality

Data pipelines

Challenges

Issues

Infrastructure

Författare

Aiswarya Raj Munappy

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Testing, Requirements, Innovation and Psychology

Jan Bosch

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Testing, Requirements, Innovation and Psychology

Helena Holmström Olsson

Malmö universitet

Lecture Notes in Computer Science

0302-9743 (ISSN) 1611-3349 (eISSN)

Vol. 12562 168-184

PROFES
Turin, Italy,

Ämneskategorier

Annan data- och informationsvetenskap

Mediateknik

Bioinformatik och systembiologi

DOI

10.1007/978-3-030-64148-1_11

Mer information

Senast uppdaterat

2021-05-05