Ambidexterity in large-scale software engineering

Doctoral thesis, 2015

Software is pervading our environment with products that become smarter and smarter every day. In order to follow this trend, software companies deliver continuously new features, in order to anticipate their competitors and to gain market share. For this reason, they need to adopt processes and organization solutions that allow them to deliver continuously. A key challenge for software organizations is to balance the resources in order to deliver enough new features in the short-term but also to support the delivery of new features in the long-term. In one word, companies need to be ambidextrous. In this thesis we investigate what ambidexterity is, what are the factors that hinder large software companies to be ambidextrous, and we provide initial solutions for the mitigation of such challenges. The research process consists of an empirical investigation based on the Grounded Theory approach, in which we conducted several case studies based on continuous interaction with 7 large software organizations developing embedded software. The results in this thesis are grounded in a large number of data collected, and corroborated by a combination of exploratory and confirmatory, as well as qualitative and quantitative data collection. The contributions of this thesis include a comprehensive understanding of the factors influencing ambidexterity, the current challenges and a proposed solution, CAFFEA. In particular, we found that three main challenges where hampering the achievement of ambidexterity for large software companies. The first one is the conflict between Agile Software Development and software reuse. The second one is the complexity of balancing short-term and long-term goals among a large number of stakeholders with different views and expertize. The third challenge is the risky tendency, in practice, of developing systems that does not sustain long-term delivery of new features: this is caused by the unbalanced focus on short-term deliveries rather than on the system architecture quality. This phenomenon is referred to as Architectural Technical Debt, which is a financial theoretical framework that relates the implementation of suboptimal architectural solutions to taking a debt. Even though such sub-optimal solutions might bring benefits in the short-term, a debt might have an interest associated with it, which consists of a negative impact on the ability of the software company to deliver new features in the long-term. If the interest becomes too costly, then the software company suffers delays and development crises. It is therefore important to avoid accumulation, in the system, of Architectural Technical Debt with a high interest associated with it. The solution proposed in this thesis is a comprehensive framework, CAFFEA, which includes the management of Architectural Technical Debt as a spanning activity (i.e., a practice shared by stakeholders belonging to different groups inside the organization). We have recognized and evaluated the strategic information required to manage Architectural Technical Debt. Then, we have developed an organizational framework, including roles, teams and practices, which are needed by the involved stakeholders. This solutions have been empirically developed and evaluated, and companies report initial benefits of applying the results in practice.