Transformative innovation policy evaluation: characteristics, challenges, and lessons from practice

Doctoral thesis, 2023

The emergence of the transformative innovation policy (TIP) framing, which builds upon previous science, technology, and innovation policy (STI) approaches to address transformative change or “grand challenges,” implies new challenges for policy evaluation. While most frameworks for TIP evaluation are quite similar to those of previous STI policy evaluation approaches, it is here argued that issues such as capturing the effect of such policies, such as related to the direction of transformative change and its additionality, remains unaddressed in the literature. This PhD thesis focuses on three research questions: (RQ1) From a theoretical perspective, what could be the distinguishing characteristics of an “ideal type” TIP evaluation compared with previous innovation policy evaluation approaches? (RQ2) How can additionality and directionality be accounted for in TIP program evaluation? (RQ3) What are the challenges related to translating an “ideal type” TIP evaluation approach into practice?

RQ1 is mainly addressed by insights provided by Papers I and II. Specifically, Paper I develops a systematic literature review of TIP and identifies challenges for TIP evaluation, while Paper II develops an analytical framework specifying how key evaluation categories differ for an “ideal type” TIP evaluation compared with previous STI approaches. RQ2 is covered mostly by Papers III and IV. Paper III illustrates the development of the framework, which combines insights from theory-based evaluation and TIP to address directionality and additionality. Paper IV complements Paper III by specifying key transformative processes evaluators should look for when evaluating the effects of TIP interventions. RQ3 in turn is covered mainly by Papers II and V. Paper II uses the aforementioned analytical framework to make a cross-case analysis of three Swedish innovation programs funded by Vinnova, the Swedish Innovation Agency: Vinnväxt, Challenge-Driven Innovation (CDI), and the Strategic Innovation Programs (SIPs). Based on this analysis, it identifies three challenges for making evaluations more aligned with an “ideal type” TIP evaluation. Paper V critically analyzes the feasibility of the integrated framework from Paper III by performing an evaluation of the BioInnovation SIP to understand the challenges that emerge for evaluation practice. Finally, the thesis discusses the key characteristics of an “ideal type” TIP evaluation, reflects on the feasibility of the integrated framework for TIP program evaluation, highlights remaining issues for TIP evaluation, and concludes by presenting the key contributions of the thesis.