Towards an integrated framework for evaluating transformative innovation policy
Towards an integrated framework for evaluating transformative innovation policy
Abstract
In recent years, a new frame for innovation policy has emerged, namely "transformative innovation policy" (T I P), which aims at addressing transformative change or "Grand Challenges". Such a shift in policy theory should ideally be reflected in policy evaluation, but the literature has so far provided little advice on how to address T I P-related evaluation challenges such as directionality and system-level behavioural additionality. This paper discusses how the evaluation of policy interventions targeting system innovation can be designed to address these challenges. Combining the literature on sustainability transitions with policy evaluation, we propose an integrated evaluation framework composed of three main components: (i) programme theory (programme goals, systems boundaries and desired (or accepted) development paths); (ii) system analysis (transformative outcomes); and (iii) synthesis and overall assessment (including revision of programme theory). By integrating the two sets of literature, we provide a bridge between academic research on transitions and current evaluation practices. We briefly illustrate the applicability of the integrated framework in the BioInnovation Strategic Innovation Programme in Sweden.
1. Introduction
1. Introduction
Since the mid-2000s, a new "frame" (or paradigm) for innovation policy has emerged, which some authors refer to as transformative innovation policy (T I P) ; . While this new frame partly builds on previous science, technology and innovation policy ideas, it implies a broader perspective on both the innovation policy agenda and the innovation process as such . Current examples include the O E C D S and T Policy 2025, E U's Fit for 55 legislative packages, the Swedish Innovation Agency's Impact Innovation Programme, and the German High-Tech Strategy 2025.
Although both policymakers and researchers are still trying to understand how to approach this new policy frame, some distinguishing characteristics compared with previous frames have been identified in the literature . Most notably, T I P comes with a shift towards "purposive and directional innovation" ; see also ; where innovation policy is mobilised to address broader societal goals, such as "Grand Challenges" or sustainability transitions in important societal sectors
; . It involves multiple levels of governance as well as interactions with other (sectoral) policy domains , resulting in an increasing need for policy coordination ; . Drawing on literature on system innovation and sustainability transitions , T I P also implies a broader view of the innovation process, in terms of both policy mixes ; and stakeholder involvement . With regard to policy mixes, the T I P literature stresses the need for innovation policy to include not only both supply- and demand-side instruments for innovation (as in the previous innovation system frame) but also measures to support niche creation and regime destabilisation . Regarding stakeholder involvement, the wider agenda implies that a more diverse set of stakeholders will be affected and, consequently, should be allowed to become engaged in policymaking processes ; . This might also require new governance modes .
Such a shift in policy approach should, ideally, be reflected in policy evaluation . So far, researchers have tended to use existing analytical frameworks to analyse the influence of traditional policies and policy mixes (and other factors) on transition processes in different sectors (cf. ; ). However, the emergence of innovation programmes that have been designed for the explicit purpose of being transformative, such as those mentioned above, stresses the need for a new, more specific evaluation approach. Moreover, evaluation practices are still very much based on earlier innovation policy paradigms, and implementing TIP thinking into these practices can therefore be rather challenging ( ). Most notably, evaluating TIP would require evaluators to handle directionality and system-level behavioural additionality ( ). With regard to directionality, there is a need to evaluate how policy mixes impact the direction of socio-technical change, for example in terms of assessing whether specific societal needs, demands, and challenges are addressed and acceptable development paths for different sub-systems identified (cf. ; ). Concerning behavioural additionality, TIP evaluation needs to go beyond traditional input-output analysis ( ; ) to assess how policy generates transformative outcomes ( ; ) and – in the longer term – contributes to achieving sustainability transition and meeting targeted societal goals ( ). Accordingly, evaluations need to explain how specific interventions cause certain system-level impacts ( ; ).
We can, thus, conclude that TIP evaluation requires a framework that can handle directionality and is focused on behavioural additionality in terms of socio-technical outcomes and impacts. Some valuable attempts have been made towards developing such a framework, including efforts to analyse directionality and the influence of policy instruments/mixes on innovation systems or socio-technical niches and regimes (cf. ; ). However, these frameworks rarely describe how to operationalise directionality for evaluation purposes, beyond studying the capacity of policy programmes to build shared visions (cf. ; ), or how to move beyond traditional system analysis and evaluate behavioural additionality at the systems level (for an exception, see ).
In this paper, we argue that closer integration between the literature on TIP and the policy evaluation literature can be a useful approach to addressing these gaps. On the one hand, some evaluation theory streams include rather well-developed ideas that relate to directionality and additionality, which could be relevant for TIP evaluation. Most notably, the theory of change, T O C, literature is goal oriented and includes a structured way to consider both outcomes and impacts in relation to what a programme is meant to achieve ( ; ), and the realist evaluation, R E, literature provides new perspectives on causality ( ; ) and presents a mechanism-based approach to addressing additionality ( ). On the other hand, the evaluation literature does not consider the specific phenomenon at hand ( ), which means that T O C and R E thinking needs to be adapted to the particular context of sustainability transitions to be useful for TIP evaluation. To achieve this, a theory-based approach which links general evaluation concepts and approaches with transition-specific conceptualisations of goals, outcomes, processes, and mechanisms – as described in current innovation and transition studies literature – is key ( ; ).
Against this background, the purpose of this paper is to develop a framework for evaluating transformative innovation programmes, which integrates insights from literature on sustainability transitions and policy evaluation to address directionality and system-level behavioural additionality.
As noted above, innovation policy evaluation practices already lag behind advances in innovation theory ( ; ), and the advent of transformative innovation policy perspectives makes this gap even more accentuated ( ). In this context, an additional benefit of integrating sustainability transitions literature and policy evaluation literature is that the latter, especially T O C, is well established among public policymakers and evaluators and, therefore, can serve as a bridge between the suggested TIP evaluation framework and current evaluation practices.

