Bulgaria Functional and Governance Analysis Bulgaria Functional and Governance Analysis March 2021 Bulgaria Functional and Governance Analysis Note This report is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this report are entirely those of the authors and they do not necessarily reflect the views of The World Bank, its Executive Directors, or the governments they represent, nor do they reflect the views of the Bulgarian government or its institutions. The World Bank does not guarantee the accuracy of the data included in this work, which is drawn from multiple external sources. Nothing herein shall constitute, or be considered to be, a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Bulgaria Functional and Governance Analysis Contents Abbreviations and Acronyms....................................................................................................................................... 5 Acknowledgements......................................................................................................................................................... 6 Executive Summary.......................................................................................................................................................... 7 Introduction........................................................................................................................................................................ 11 1. Methodology / 13 2. Findings/ 17 2.1 Cluster Analysis...................................................................................................................................................30 3. Cross-Cutting Challenges / 39 3.1 Developing Justifications for Policy Interventions.................................................................................41 3.2 Theories of Change.............................................................................................................................................44 3.3 Budgeting and Solicitations............................................................................................................................48 3.4 Monitoring and Evaluation...............................................................................................................................51 3.5 Targeting of Beneficiaries and Selection Criteria...................................................................................55 3.6 Improving Project Selection...........................................................................................................................58 3.7 Administrative Burden on Beneficiaries.................................................................................................... 61 3.8 Human Resources and Implementation Capacity................................................................................64 3.9 Governance and Coordination of STI Policies.........................................................................................66 4. Recommended Actions for Better Functionality and Governance of the STI Policy Mix / 70 4.1 Design of STI instruments............................................................................................................................... 72 4.2 Implementation of STI instruments.............................................................................................................76 4.3 Governance........................................................................................................................................................... 81 References.........................................................................................................................................................................83 Appendices / 85 / Appx. I: Individual Indicator Scores..............................................................................................................86 / Appx. II: Summary of Legal and Regulatory Environment for STI Policy.......................................98 / Appx. III: Process Flows...................................................................................................................................108 / Appx. IV: Developing Theories of Change and Indicators.................................................................. 111 / Appx. V: Innovation Agencies Case Studies...........................................................................................140 Bulgaria Functional and Governance Analysis Abbreviations and Acronyms CoC Center of Competence CoE Center of Excellence DG OPIC Directorate General of the Operational Program Innovation and Competitiveness EA OPSESG Executive Agency for Operational Programme Science and Education for Smart Growth ERC European Research Council ESIF European Structural and Investment Funds EU European Union FoF Fund of Funds FTE Full-time equivalent GBARD Government budget appropriations on research and development GERD Gross domestic expenditure on research and development HEI Higher education institution HERD Higher education expenditure on research and development IPR Intellectual property rights ISIS Innovative Strategy for Intelligent Specialization M&E Monitoring and evaluation MoE Ministry of Economy MLSP Ministry of Labor and Social Policy MoES Ministry of Education and Science NEAA National Evaluation and Accreditation Agency NIF National Innovation Fund NSDSR National Strategy for Development of Scientific Research NSF National Science Fund OP Operational Programme OPIC Operational Programme Innovation and Competitiveness OPHRD Operational Program Human Resources Development OPSESG Operational Programme Science and Education for Smart Growth PER STI Public Expenditure Review for Science, Technology, and Innovation PRO Public research organization R&D Research and development R&I Research and innovation SMEPA Small and Medium Size Enterprises Promotion Agency STI Science, technology, and innovation STEM Science, technology, engineering, and mathematics ToC Theory of Change UMIS Unified Management Information System 5 Bulgaria Functional and Governance Analysis Acknowledgements This report was prepared by a World Bank team led by Anwar Aridi (Senior Private Sector Specialist, Task Team Leader), and comprising Daniel Querejazu (Innovation Policy Specialist), Teodora Georgie- va (Research and Innovation Expert), Lyubomira Dimitrova (Research and Statistical Expert), and Juan Rogers (Public Administration Expert). Ana Goicoechea (Senior Economist) and Ventsislav Benov (Legal Expert) provided valuable expertise and analytical inputs to the report. William Shaw edited the report. The report benefited from the guidance of the World Bank management, Fabrizio Zarcone (Country Manager) and Ilias Skamnelos (Practice Manager), and from feedback and comments provided by Mar- cio Cruz (Senior Economist), Yehia Eldozar (Monitoring and Evaluation Specialist), and Lukasz Marc (Economist). The team would like to thank the Government of Bulgaria, and in particular the Ministry of Educa- tion and Science, the Executive Agency for Operational Programme Science and Education for Smart Growth, the Ministry of Economy, as well as all of the program and administrative staff that took part in the functional and governance analysis exercise. 6 Bulgaria Functional and Governance Analysis Executive Summary Overcoming the key challenges facing Bulgaria’s research and innovation system will be critical to boosting the country’s lagging economic growth; while growth has been strong since EU accession, the pace of growth has slowed greatly since the 2008 financial crisis and has not been enough to catch up to regional peers in terms of labor productivity and living standards. This slow growth will likely be exacerbated by the COVID-19 pandemic and ensuing economic crisis, which have disproportionately affected Bulgaria’s smaller and younger firms1 – a typically the firms with the highest potential for pro- ductivity growth. Moreover, a rapidly aging population and high emigration rates raise the potential for increasing labor shortages and skills mismatches over the medium term. Accelerating productivity growth will require large improvements to the country’s science, technology, and innovation (STI) per- formance, which currently ranks among the weakest in the EU across multiple indicators. Contributing factors to this poor STI performance include inadequate levels of funding for research, an uncoordinat- ed STI policy agenda, and serious capacity issues on the part of STI implementing bodies. The coming year presents an important opportunity for Bulgaria to improve its STI support policies and programs. The government of Bulgaria is preparing its National Recovery and Resilience Plan to facili- tate economic and social recovery in the aftermath of the pandemic, which will allocate approximately €1.25 billion in funding for innovation-related activities. At the same time, Bulgaria’s preeminent STI implementing bodies – the Ministry of Education and Science, National Science Fund, Ministry of Econ- omy, and Executive Agency for the Operational Programme Science and Education for Smart Growth – are defining the priorities and targets for the new programming period (Bulgaria’s third since EU ac- cession), which will include a research and innovation budget of approximately €1.17 billion. These new investments would benefit from careful review of the implementation and coordination of the current period’s policy portfolio to draw lessons and introduce reforms to ensure that this substantial funding yields productive impacts on the STI system. The Bulgarian government recognizes the governance and implementation challenges facing its re- search and innovation system and has taken concrete steps to address them. A new State Agency for Research and Innovation was established by decree of the Bulgarian Council of Ministers in 2020, which presents an excellent opportunity for addressing several of the challenges found within the STI policy mix. This opportunity is not limited to improving the implementation of policy instruments and programs that will be directly administered by the new Agency, but also includes the important task of coordinating the national STI policy agenda in collaboration with other ministries, public agencies, and even private sector representatives. 1 Bulgaria World Bank Business Pulse Survey 2020 7 Bulgaria Functional and Governance Analysis For Bulgaria to fully leverage the benefits of the European funding over the next programming period, it will need to build and upgrade the capacity of its national STI institutions, including the new State Agency for R&I. National STI institutions (those outside of operational programme administrative struc- tures) have been chronically underfunded and understaffed, which is not a sustainable arrangement for the long-term health of Bulgaria’s STI system. The coming period should bring with it a focus on upgrading the capacities of national institutions and programs through increased budget allocation, professionalization of the workforce, and technical and capacity building efforts. Critically, it will be important that the new R&I Agency be positioned as the anchor institution for building professional, analytical, and technical capacity for R&I national programs and not just serve as an implementer of ES- IF-financed programming. To this end, there are several high performing nationally-financed programs that present opportunities for scaling up, which would provide the dual benefits of expanding proven policy instruments and at the same time building up the organizational and professional capabilities of national implementing bodies through programs with established good practices. This report provides an assessment of the functionality and quality of governance of key policy instru- ments that support research and innovation in Bulgaria. This analysis represents the second phase of the World Bank’s Public Expenditure Review for Science, Technology, and Innovation (PER STI) project in Bulgaria. The first phase of the PER STI project, the Country Needs and Policy Mix Assessment, provid- ed a comprehensive assessment of the country’s research and innovation needs, an overview of the na- tional policies devoted to supporting STI in Bulgaria, and an analysis of the alignment or gaps between policy support and the research and innovation needs of the nation’s public and private sectors. This report builds on the first phase of the project by exploring the functionality of a representative set of STI policy instruments through a review of their design, implementation, and governance. The findings from this report inform a set of recommendations to improve the functionality and governance of the portfolio of STI support programs. The functional analysis identified areas of strength, as well as many areas for improvement across the policy portfolio: Instrument design. There is a general disconnect between program objectives (what instruments are trying to accomplish) and program activities (what instruments actually do), which is largely due to the lack of use of theories of change in instruments. Theories of change can help depict the linkages between program inputs, activities, outputs, and outcomes, and how each of these elements contribute to the ultimate objectives of a program. The lack of adoption of theories of change negatively impacts other areas of instrument design and implementation, such as the definition of program indicators, monitoring and evaluation (M&E) frameworks, and learning and knowledge management systems. However, instrument design generally includes strong formal mechanisms for interacting with stakeholders, and engagement with stakeholders is an area of relative strength among Bulgarian STI policy instruments. Program implementation. All institutions engaged in implementing STI policies, to varying de- grees, suffer from a lack of capacity and resources to fully implement their portfolios, although this challenge is most severe for nationally-funded instruments due to insufficient and unpre- dictable budgets. This lack of resources has had negative impacts on instrument functionality, limiting implementing bodies’ ability to administer calls for proposals, evaluate projects, and has even resulted in program cancellation. Capacity issues are particularly severe in M&E, and almost no evaluations of instruments have been done in the current programming period due to a lack of resources and staff. 8 Bulgaria Functional and Governance Analysis Governance and coordination of the instrument portfolio. STI institutions are disconnected from one another, resulting in fragmented policies and programs and an uncoordinated national STI agenda. While formal coordination mechanisms between STI institutions are largely in place, very little coordination or collaboration occurs that is relevant to individual programs. Importantly, a cluster analysis of the functional analysis scoring found that many of the differences in instrument functionality across the portfolio can be attributed to the organization that designs and administers the individual instruments. This means that reforms aimed at improving the functionality of instruments should be addressed at the level of the implementing bodies, rather than through port- folio-wide reform efforts. These findings inform a set of recommendations to improve the current and future portfolio of STI sup- port programs, in the areas of design, implementation, and governance: Design evidence-based policy instruments, which are grounded in analytics to identify, define, Design and quantify the failure and the affected actors. Once identified, the full range of alternative in- strument designs should be considered to address the identified failure. Articulate a theory of change (and related indicators) for each instrument to show the connec- tions between instrument inputs, activities, outputs, and desired outcomes. Each instrument should include a results framework (tied to the theory of change) with a full catalog of input, ac- tivity, output, and outcome indicators. Design instruments to allow for more continuity in policy support and accumulation of organi- zational capabilities. This entails instruments issuing regular calls for proposals, where possible, rather than designing instruments that expend their entire budgets in a single solicitation. This will allow for more efficiencies in implementation and for learning and improvement of the instru- ments and the organizations administering them. Improve M&E practices and capacity of STI implementing bodies by developing impact evaluation strategies for instruments that includes clear objectives, theories of change and results frame- works, evaluation plans, supporting systems, and protocols. This will require committing to regular performance and impact evaluations of instruments; and providing resources and incentives for STI implementing bodies to carry out these M&E activities and train their staff and administrators. Develop eligibility and selection criteria that maximize the impact of policy interventions. Eligibili- Implementation ty criteria should be reviewed to ensure alignment with policy objectives and selection criteria be revised to target projects with the highest potential impacts from public investments. Improve the project proposal evaluation quality by developing pools of external and international experts for evaluation panels and increasing renumeration for expert reviews. Reduce administrative burdens on beneficiaries through a number of targeted reforms, including the development of an online portal for nationally-funded instruments where beneficiaries can submit and receive information throughout the application and reporting processes; acquiring requested documentation from beneficiaries ex officio, where possible; harmonizing public pro- curement processes across programs; and digitizing procurement procedures. Increase budget support to functioning and scalable national programs and organizations to 9 Bulgaria Functional and Governance Analysis bridge the capacity divide with OP-funded programs. Improve human resource management and capacity of STI implementors by increasing staffing across the STI system, both for full-time and part-time positions, especially in nationally-funded programs; providing discretionary budgets for consultants, external experts, and other part-time positions; and improving HR management practices related to STI staff incentives and training opportunities. Governance Improve coordination of the STI agenda by activating existing coordination channels, such as the Council for Smart Growth, Inter-Institutional Working Group, and Regional Partnership Network, to set a commonly agreed upon R&I vision and strategic objectives among national and regional STI actors. Enable the new R&I Agency to take on the mission of monitoring and coordinating the implemen- tation of the national R&I agenda through; (i) mandating the agency to collect and manage data on the performance of the national STI system and the implementation of STI programs; and (ii) accumulating the analytical and professional expertise needed to fulfill this mission. Establish the representation mechanisms that ensure representation of the R&I Agency in the governance of key implementing bodies and vice versa (governing boards, steering committees). Provide technical assistance to the founding team of the new agency through knowledge sharing, training, and partnerships to ensure that the design, governance, and operations of the organiza- tion build on international good practice. 10 Bulgaria Functional and Governance Analysis Introduction The goal of this report is to provide in-depth assessments of the functionality and governance of key Bulgarian STI support instruments and to provide recommendations for improvement of the govern- ment’s current and future portfolio of STI support programs. The work represents the second phase of the World Bank’s Bulgaria Public Expenditure Review for Science, Technology, and Innovation (PER STI) project. The first phase of the PER STI, the Country Needs and Policy Mix Assessment (Aridi et al, 2020), high- lighted several key issues of relevance to this analysis: Bulgaria exhibits one of the lowest innovation performances in Europe, driven by very low levels of research investment compared to peers. There is a clear need to modernize the national research system – particularly in the public sector – to improve the performance of research institutions, with a focus on research excellence, market-oriented research agendas, and technology transfer. Governance, coordination, and implementation are all key challenges to implementing the STI policy portfolio. Government STI institutions are disconnected from one another and suffer from weak governance structures, which has resulted in the lack of a coordinated national R&I agenda and fragmented policies and programs. In particular, there is a disconnect between the research and innovation agendas. Further, severe lags in allocation and disbursement of government funds for STI indicates serious challenges in Bulgaria’s ability to implement the STI policy mix, which has likely hindered the effectiveness of its policy instruments. Analysis of the alignment of the policy mix with identified country needs shows gaps in support for technology transfer, Industry 4.0 technology adoption, early-stage company support, improve- ments to the business environment, and development of digital skills. Technology adoption in firms has improved over the last ten years, helping to drive productivity growth, but Bulgaria still lags behind most peers in technology adoption rates among firms. Bul- garian companies have among of lowest levels of firm-level digitization in Europe, for both basic and advanced digital technologies. Bulgaria also lags behind its peers in the development of a conducive business environment and a competitive market. In particular, regulations related to starting a business appear to represent a large constraint on market access and consequently on competition in Bulgaria. Previous assessments and analyses highlight additional challenges in the STI system. These include a 2015 analysis (European Commission, 2015), which found a need for major structural reforms to ad- 11 Bulgaria Functional and Governance Analysis dress the poor performance of the research and innovation system. Recommended reforms included concentrating funding on the better performing institutions to address the lack of capacity on the part of research performers, as well as improvements required to bring the processes for the evaluation and funding of project proposals up to international standards. An assessment carried out under the Policy Support Facility (European Commission, 2017) found a need for system-wide institutional reform to address disjointed approaches to higher education, R&I policy, and institutions; an over-reliance on international funding; poor public-private research ties; as well as an erosion of trust due to misuse of public funding and low performance. An interim evaluation of the implementation of the Innovation Strategy for Smart Specialization 2014 – 2020 found that a lack of capacity on the part of implementing bodies has contributed to delays in implementation of the strategy. This Functional and Governance Analysis expands on phase one of the PER STI project and previous assessments by conducting in-depth assessments of the functionality of 28 key instruments of the STI policy mix, focusing on elements of the design, implementation, and governance of Bulgarian STI sup- port instruments. While this report covers the functionality of individual instruments, it does not assess their efficiency (value of outputs per unit input). The last deliverable of the PER STI will constitute an efficiency analysis of a select set of instruments covering both sides of the research and the innovation agenda. This report is structured in four sections: 1) Methodology; 2) Findings, which provide a descriptive anal- ysis of the overall design, implementation, and governance of the analyzed instruments and a cluster analysis to uncover relationships between different programs and functional variables; 3) Cross-cutting challenges that pervade the Bulgarian STI portfolio across programs and institutions; and 4) Recom- mendations for improving the functionality of the STI policy mix. 12 Bulgaria Functional and Governance Analysis 1 Methodology This report aims to assess the functionality and governance of key Bulgarian STI support programs and to provide evidence-based recommendations for improving the design, implementation, and governance of the country’s STI portfolio. The analysis assesses the functionality, rather than the impact, of STI support instruments. Functionality, here, refers to the quality of processes involved in creating and implementing each instrument, including design, implementation, monitoring and evaluation systems, human resources, and governance (the instrument’s integration and interactions with other programs, institutions, and regulations). 13 Bulgaria Functional and Governance Analysis The methodology for the analysis is based on a comprehensive analytical framework benchmarked to international best practices. The analytical framework, developed by the World Bank, scores support programs along a total of 31 variables: 14 related to program design, 13 related to implementation, and four related to program governance (Figure 1). A short description of each variable can be found in Appendix I. The functionality of each program is scored on a scale of 1 to 5 for each variable, where 5 denotes international best practice and 1 denotes the greatest distance from, or absence of, best prac- tices. Figure 1: Functional and governance analysis indicators Origin Justification Objectives Logical framework Relationship with policy mix Design Inputs Activities Products Beneficiaries Audiences Results and impact Eligibility and selection Choice of instrument M&E Learning Implementation Calls Eligibility and selection Application process Information management Closing and follow-up Budget adequacy Organization / Autonomy / Incentives / Monitoring Staff adequacy M&E and IE Relationship between Governance instruments Relationship between institutions Relationship with other policy frameworks – limitations Relationship with other policy frameworks – responses Source: Correa, 2014. 14 Bulgaria Functional and Governance Analysis Data was collected through semi-structured interviews with government program managers and ad- ministrators and reviews of publicly available program documentation. These interviews were supple- mented with semi-structured interviews with several program beneficiaries from the public and private sectors, as well as a review of relevant laws, rules, and regulations that impact the design and imple- mentation of the policy mix (see Appendix II for a summary of the legal review). The analysis aims to reveal whether key elements of effective innovation policies have been met in three key areas: Design. Public interventions must be designed through a process covered by the rule of law and consistent with general national or regional goals for research and innovation. The interventions must be properly justified and address real problems, avoiding the trap of addressing false failures. This justification can also help avoid capture of public resources by certain vested beneficiaries. Once a system failure is identified, policy makers should consider the full range of alternative in- tervention designs rather than simply copying existing programs. By design, interventions should have a clear, well-articulated theory of change, which depicts the shared relationships and causal linkages between program inputs, activities, outputs, and outcomes and logically connects them to higher level strategic objectives. Theories of change should define indicators for inputs, activ- ities, outputs, and outcomes that allow for monitoring and evaluation of program performance. Implementation. Processes for administering the program, including application, selection, and reporting, should be clear and transparent, and knowledge management systems should be in place to allow for systematic learning and improvement of the instrument during implementation. Implementing agencies must have adequate manpower and organizational structures to admin- ister the program, and staff should have training opportunities and incentives that are relevant to program performance (rather than generic to public administration staff). Internal and external monitoring and evaluation of the instrument should take place, and, critically, evaluation results should be used to improve and adapt the program. Governance. Coordination mechanisms should be in place to minimize overlap and enhance complementarities between the instrument and other programs and agencies. Implementing staff should also be aware of external laws and regulations that can inhibit the implementation of the instrument and should be proactive in taking steps to adapt to ensure the optimal operation of the instrument. This analysis covers 28 instruments supporting STI from 2014-2019, which represent 90 percent of the total allocated STI funding in Bulgaria over that period (Table 1). This includes many instruments directly support R&I and others that target boosting the economic growth and competitiveness in Bulgaria (e.g. the instruments from the Operational Program Human Resources Development [OPHRD] and several programs from the portfolio of the Operational Programme Innovation and Competitiveness [OPIC]). This includes instruments from three operational programs (OPSESG, OPIC, and OPHRD) and all of the key agencies, directorates, and ministries engaged in implementing STI policies in the country, in- cluding the Ministry of Education and Science (MoES), the Managing Authority for the Operational Programme Science and Education for Smart Growth (OP SESG), National Science Fund (NSF), the Ministry of Economy (MoE), Directorate General for Operational Programme Innovation and Competi- tiveness (DG OPIC), Bulgarian SME Promotion Agency (SMEPA), and the Fund of Funds (FoF). 15 Bulgaria Functional and Governance Analysis Table 1: Instruments included in the functional and governance analysis Instrument Name Implementing OP/National Target Beneficiaries Allocated Body Instrument Budget 2014-2021 Creation and Development of Centres of Competence EA OPSESG OPSESG PROs, HEIs, Researchers €108,878,569 Creation and Development of Centres of Excellence EA OPSESG OPSESG PROs, HEIs, Researchers €80,839,246 Improving the production capacity of SMEs DG OPIC OPIC Firms €338,661,880 Energy Efficiency for Small and Medium-sized Enterprises DG OPIC OPIC Firms €266,088,576 Increasing energy efficiency in large enterprises DG OPIC OPIC Firms €129,717,388 Support for the introduction of innovation in enterprises DG OPIC OPIC Firms €95,073,668 Support for pilot and demonstration initiatives DG OPIC OPIC Firms €30,765,429 for effective use of resources Development of product and process innovations DG OPIC OPIC Firms €51,513,826 Enhancing the entrepreneurship DG OPIC OPIC Firms €42,716,611 Development of management capacity and growth of SMEs DG OPIC OPIC Firms €42,673,816 Support for development of innovations DG OPIC OPIC Firms €17,543,339 by start-up companies Development of clusters in Bulgaria DG OPIC OPIC Firms €11,883,189 Phase 2 of the project “Establishing a science DG OPIC OPIC Firms €6,297,914 and technology park” Sofia Tech Park Enhancing the growth of SMEs through pilot DG OPIC OPIC Firms €5,847,504 application of a voucher scheme by BSMEPA Provision of institutional support to The Executive DG OPIC OPIC Firms €1,938,699 Agency “Bulgarian Accreditation Service” for improving the quality infrastructure Support for entrepreneurship MLSP OPHRD Firms €8,487,944 Risk-sharing Micro-Finance facility FoF OPHRD Firms, entrepreneurs €5,310,734 Technostart MoE National Entrepreneurs €444,721 National Innovation Fund SMEPA National Firms €12,767,156. Research Infrastructure MoES National PROs, HEIs NA* National Science Programs 2018-2022 MoES National PROs, HEIs, Researchers €26,448,000 Financing of scientific or artistic activity inherent MoES National PROs, HEIs €24,599,883 in public higher education institutions National program Young scientists and postdoctors MoES National PROs, HEIs €10,000,000 Doctoral fellowships MoES National PROs, HEIs €4,800,000 VIHREN MoES National PROs, HEIs, Researchers €1,114,616 Fundamental Research NSF National PROs, HEIs, Researchers €25,447,981 Fundamental Research on Societal Challenges NSF National PROs, HEIs, Researchers €2,914,000 Bilateral cooperation programmes Bulgaria-Russia NSF National PROs, HEIs, Researchers €740,849 Note: *The Research Infrastructure instrument does not have an allocated budget, but rather operates as an open call through which public research institutions can apply for funding for new or improve research infrastructure projects. Determination of funding is made on a case- by-case basis by MoES. 16 Bulgaria Functional and Governance Analysis 2 Findings This section presents the key findings from the functional and governance analysis of key Bulgarian STI support instruments. Overall, this analysis uncovered many areas for improvement in the functionality of STI instruments across the policy portfolio related to instrument design, implementation, and governance. 17 Bulgaria Functional and Governance Analysis Many instruments lack a clear justification for intervention – in other words, they lack an explicit Design description of the market or system failure being addressed by the instrument. This lack of clear identification of the failure being addressed is a large contributing factor to an observed discon- nect between program activities and objectives and negatively impacts other areas of instrument design, such as setting program objectives and developing eligibility and selection criteria. Almost no instruments have an explicit theory of change or logic model, which further adds to the disconnect between activities and objectives. The lack of theories of change also contrib- utes to poorly defined and disconnected indicators for inputs, activities, outputs, and outcomes. Outcome indicators, in particular, tend to be poorly defined and weakly connected to program activities. M&E frameworks tend to be pro forma, and monitoring processes are largely focused on whether beneficiaries are in compliance with administrative regulations, rather than assessing program performance. Stakeholder engagement is an area of strength, with almost all instruments utilizing strong formal mechanisms for engagement with non-beneficiary stakeholders Organizational capacity and human resource management are pervasive challenges across the Implementation STI system, especially among nationally funded instruments. Nearly all national implementing bodies have suffered from insufficient staff at some stage of implementation, which has impacted other areas of implementation – particularly M&E. Few program staff have access to training that is relevant to their specific roles. Similarly, few STI staff are given incentives that are specific to their programs; instead, most staff are evaluated using a generic public administration framework. A majority of instruments in the STI portfolio only issued a single call for solicitations over the life of the program, rather than issuing regular annual or semi-annual calls. This “one-time” instru- ment design severely limits opportunities for learning, adaptation, and improvement of instru- ments and has negative impacts on other areas of functionality, such as learning and knowledge management and the development of selection criteria. Almost no impact evaluations of instruments have been done in the current programming peri- od, and few are planned for the future. There are few formal knowledge management systems in place; adjustments to instruments are generally ad hoc and not well documented. Governance Formal coordination mechanisms between STI institutions are largely in place, although very little coordination or collaboration occurs that is relevant to individual instruments’ implementation. STI programs can be constrained by external rules and regulations, severely in some cases, and implementors are mostly reactive in adjusting programs to deal with these constraints. The average scores of the functional analysis across the policy mix show many areas for improvement in the design, implementation, and governance of STI instruments. When comparing the functional 18 Bulgaria Functional and Governance Analysis analysis scores of Bulgarian STI instruments to instruments of CEE peers, Bulgarian STI instruments had an overall average score of 2.7, below average scores for Croatian (3.2), and Polish (3.3) support instruments, as shown in Figure 2.2 Bulgarian instruments score particularly low relative to peers in vari- ables related to implementation, largely due to the lack of capacity of Bulgarian implementing agencies. Figure 2: Average scores 5,0 for instrument design, implementation, and governance in Bulgaria, Croatia, and Poland 4,0 3,5 3,3 3,2 3,3 3,3 BULGARIA 3,2 3,2 3,1 2,9 CROATIA 2,8 3,0 2,7 2,5 POLAND Source: Authors’ 2,0 calculations based on data collected from WB Functional analysis reviews 1,0 Overall Design Implementation Governance As shown in Figure 3, there are some areas of common strength in the functionality of Bulgaria’s STI instruments: program origin (the process through which instruments are designed), audiences (en- gagement with stakeholders and beneficiaries), and project closures (processes for the termination or discontinuation of projects) all had average scores above 3.8. However, the scores of 21 of the 31 variables fell below 3.0, and the scores for four variables (logical framework3, expected outcomes and impact, program database, and staff and training) fell below 2.0, indicating poor practices in these areas across the policy portfolio. These scores highlight the positive and negative role that EU regulations can play in the functionality of Bulgarian support instruments. Interviews with program staff of OP-financed instrument revealed a strong emphasis on compliance with EU regulations, with the assumption that compliance indicates good instrument functionality. However, it is important to highlight that such compliance does not guarantee optimum functionality; Bulgarian instruments received higher scores in areas with clear EU regulations to guide the implementers of operational programme instruments and low scores in areas where EU regulations do not provide clear guidance or, in fact, inhibit the optimum functioning of in- struments. Program origin, which received the highest score of any variable in this analysis, is governed by EU regu- lation No. 1303/2013, which specifies the common rules for the preparation, adoption, and amending of OP-financed programs and grants. Similarly, stakeholder engagement, which also received high scores, is also impacted by EU regulation No. 1303/2013, which lays out composition and functions of the op- erational programme monitoring committees – the formal mechanisms through which OP instruments engage with external stakeholders. Project closures is guided by EU regulations Nos. 1299/2013 and 1303/2013, which establish the processes reporting and reviewing OP programs and projects. In each 2 The World Bank conducted similar functional and governance analyses as part of PER STI projects in Poland (Haven et al, 2020), and Croatia (Milchevski et al, 2020). 3 The logical framework indicators covers the use of logic models, theories of change, or similar methodologies. 19 Bulgaria Functional and Governance Analysis of these areas of strength, OP-financed instruments received higher scores on average than national- ly-financed instruments. Figure 3: Average scores Rules and Regulations (External) Program Origin for all instruments Rules and Regulations (Internal) Program Justification design Institutions Relationship Portfolio Relationship implementation Programs Relationship Program Objectives governance Alternative Instrument Source: Authors’ M&E Implementation Logical Framework calculation Process monitoring Inputs Staff Incentives Activities Staff & training Products/Outputs Roles & Autonomy Main beneficiaries Program management Selection criteria Budget Audiences Project closures Expected Outcomes & Impact Program database Monitoring & Evaluation Application & Selection Processes Eligibility Criteria Learning Solicitations Conversely, the absence of clear regulations has negatively affected the functionality of instruments in some areas. Considering the scores related to instrument design (Figure 4), many instruments scored poorly for the variables program justification and program objectives, stemming from a lack of explicit identification of the system failure being addressed by STI instruments. EU regulations specify the oper- ational programs develop justifications through ex ante evaluations, but in practice ex ante evaluations are largely performed as straightforward feasibility studies, rather than proper statements of the prob- lems instruments are intended to address and the antecedents that would provide lessons for new ver- sions of past policies; operational programme instruments scored lower relative to nationally-financed instruments in program justification (see an extended discussion of this issue in Section 3.1). The indi- cator logical framework also received very low scores across the portfolio due to the almost complete lack of instruments that utilize formal logic models or theories of change. While EU regulations require the overall OP to utilize a logical framework, they do not require theories of change/logic models for individual instruments. The lack of use of logic models contributed to lower scores for inputs, activities, products/outputs, and expected outcomes and impacts, as these program elements are often poorly defined and lack logical connections to each other. Expected outcomes and impacts were scored par- ticularly poorly, as many programs had incoherent outcome indicators or lacked them completely (see Section 3.2). 20 Bulgaria Functional and Governance Analysis 1 2 3 4 5 Figure 4. Average Program Origin 4,5 Scores for Design Program Justification 2,6 Indicators Portfolio Relationship 3,1 Source: Authors’ calculation Program Objectives 2,1 Alternative Instrument 2,3 Logical Framework 1,7 Inputs 2,8 Activities 2,8 Products/Outputs 3,1 Main beneficiaries 3,0 Selection criteria 2,6 Audiences 4,1 Expected Outcomes & Impact 1,6 Monitoring & Evaluation 2,7 average score Considering the scores for implementation (shown in Figure 5), the variable Solicitations received very low scores because most STI instruments in Bulgaria only issued a single call for solicitations (as opposed to a series of calls issued annually or semi-annually)4, which severely limits opportunities to make adjustments and improve the functionality of the program over time (see Section 3.3). Many pro- grams received low scores in variables related to program administration and human resources due to insufficient staff and training opportunities and a lack of autonomy (see Section 3.8). Monitoring and evaluation also received below average scores, because most M&E processes are focused on whether beneficiaries comply with rules and regulations, rather than the performance of the project being fund- ed, and almost no instruments have undergone evaluations during the current programming period. Again, the lack of clear guidance from EU regulations on the evaluation of individual instruments is a contributing factor to the low scores in this area (see Section 3.4). 1 2 3 4 5 Figure 5. Average Scores Learning 2,4 for Implementation Solicitations 2,0 Indicators Eligibility Criteria 2,6 Source: Authors’ calculation Application & Selection Processes 2,7 Program database 1,8 Project closures 3,9 Budget 2,7 Program management 2,1 Roles & Autonomy 3,0 Staff & training 1,9 Staff Incentives 3,3 Process monitoring 2,3 M&E Implementation 2,3 average score 4 Regular calls for proposals are important for financial support and technical assistances schemes, and less relevant to infrastructure projects. 21 Bulgaria Functional and Governance Analysis Most instruments received average or slightly below average scores in indicators related to governance, as seen in Figure 6. While formal coordination mechanisms between STI institutions are largely in place, coordination primarily takes place at the strategic, rather than the tactical level, so instruments are largely implemented in isolation from the activities of other implementors (see Section 3.9). STI pro- grams are also, to varying degrees, constrained by external rules and regulations, and implementors are mostly reactive in adjusting programs to deal with these constraints. 1 2 3 4 5 Figure 6. Average Programs Relationship 2,6 Scores for Institutions Relationship 3,0 Governance Rules and Regulations (Internal) 3,0 Indicators Source: Authors’ Rules and Regulations (External) 2,8 calculation average score Bulgaria has a few areas of strength and many areas of relative weakness in instrument functionality when compared to its CEE peers that have undergone similar exercise. Table 2 compares functional analysis scores for each variable for Bulgarian, Croatian, and Polish instruments, with the highest av- erage scores observed for each variable across countries shaded in green and the lowest observed average scores shaded in red. Bulgarian instruments scored lower than Polish and Croatian instru- ments in many functional areas, performing particularly poorly compared to its peers in areas related to instrument design (in the use of logical frameworks and developing indicators for activities, outputs, and outcomes) and implementation (in learning, solicitations, eligibility and selection criteria, and proj- ect databases. However, Bulgaria does have a few areas of relative strength, outperforming its peers in defining program origin and staff incentives. Table 2: Average scores for STI instruments in Bulgaria, Croatia, and Poland Blue shading denotes the highest score observed for each variable among peers. Red shading denotes the lowest score observed for each variable. Indicator Bulgaria Croatia Poland Program Origin 4.5 3.5 3.6 Design Program Justification 2.6 2.7 2.7 Portfolio Relationship 3.1 3.3 3.4 Program Objectives 2.1 2.7 3.0 Alternative Instrument 2.3 2.6 2.4 Logical Framework 1.7 2.7 3.3 Inputs 2.8 3.2 2.8 Activities 2.8 3.4 3.3 Products/Outputs 3.1 3.8 4.2 Main beneficiaries 3.0 3.3 2.9 Selection criteria 2.6 3.2 2.8 Audiences 4.1 3.5 4.5 22 Bulgaria Functional and Governance Analysis Indicator Bulgaria Croatia Poland Expected Outcomes & Impact 1.6 2.4 2.8 Monitoring & Evaluation 2.7 3.0 3.8 Learning 2.4 3.4 3.5 Implementation Solicitations 2.0 3.3 3.6 Eligibility Criteria 2.6 3.8 3.3 Application & Selection Processes 2.7 3.6 4.3 Program database 1.8 3.4 3.1 Project closures 3.9 4.4 5.0 Budget 2.7 3.1 3.6 Program management 2.1 3.0 3.3 Roles & Autonomy 3.0 3.0 3.2 Staff & training 1.9 3.0 3.3 Staff Incentives 3.3 2.4 2.5 Process monitoring 2.3 2.6 3.2 M&E Implementation 2.3 2.9 3.2 Programs Relationship 2.6 3.4 3.6 Governance Institutions Relationship 3.0 3.0 3.3 Rules and Regulations (Internal) 3.0 3.9 4.4 Rules and Regulations (External) 2.8 3.0 1.8 Source: Authors’ calculations based on data collected from WB Functional analysis reviews. There is also significant variability in scores across programs. Figure 7 shows the variation in scores across variables, displaying the average scores for each variable (the blue line), as well as the highest score recorded across instruments (orange) and lowest score (grey). Notably, there are examples of very low scores for almost every indicator, including indicators with high average scores. This indicates considerable potential for internal learning of good practices. 23 Bulgaria Functional and Governance Analysis Figure 7: Minimum and maximum scores across all average score programs Source: Authors’ calculation | Notes: Maximum score maximum scores denotes the highest score received by any program for each variable. Minimum score denotes the lowest minimum scores score received by any program for each variable. Interaction of Jurisdiction Rules and Regulations Program Origin (Seriousness of external constraint) Program Justification Interaction of Jurisdiction Rules and Regulations (Internal response) Portfolio Relationship Institutions Relationship Program Objectives Programs Relationship Alternative Instrument M&E Logical Framework (or Logic Model) Process monitoring Inputs Incentives (related to performance with the instrument) Activities Staff and training Products/Outputs Roles and Autonomy Main beneficiaries Program management and Selection criteria organization quality Audiences Budget and financial resources Expected Outcomes and Project closures Impact Program database and information on Monitoring and Evaluation participants and applications Learning evidence during Application and Selection Processes implementation Eligibility Criteria and Application Information Solicitations (Calls for proposals and their Products) A closer inspection of scores across programs reveals areas requiring systemwide improvements and areas with high potential for learning opportunities between programs. Figure 8 plots all variables by their mean score (x-axis) and standard deviation (y-axis), with the axes intersecting at the median point of each metric. Variables that fall in the upper two quadrants (those with high standard deviations) are areas within the policy mix with a high potential for internal learning, as there are programs in these ar- eas with both high scores (good practices) and low scores (poor practices). In particular, variables that fall in the upper left quadrant show great variation in scores but lower averages, so learning opportuni- ties may be the greatest in these areas. Variables in the bottom left quadrant are those with both low scores and low standard deviation, indi- cating the need for systemwide improvement with few, if any, examples of good practices in the current policy mix. There are a number of variables related to design and implementation that fall in this catego- ry, including program objectives, logic models, expected outcomes and impacts, program information systems, process monitoring, and staff and training. Indicators in the bottom right quadrant are those with high scores and low standard deviation, indicat- ing areas of systemwide strength. 24 Bulgaria Functional and Governance Analysis Figure 8: Average score and standard high standard deviation deviation across all instruments Source: Authors’ calculation 1,6 Project design closures implementation governance Solicitations Application Selection Budget Audiences Products Outputs Incentives Logic Alternative Selection Model Instrument criteria Jurisdiction (External) M&E Implementation Institutions Relationship Learning Programs Relationship Justification low 1 2 4 5 high mean Staff Eligibility Information mean score training score Beneficiaries Management quality M&E Design Jurisdiction (Internal) Origin Inputs Activities Portfolio Relationship Process Program monitoring information Objectives Outcomes, Impacts Roles Autonomy 0,2 low standard deviation 25 Bulgaria Functional and Governance Analysis There is not a great deal of variation in scores for the policy objectives of the instrument (see Figure 9 below). Instruments focused on non-R&D business support (business infrastructure, access to finance, market access, etc.) scored slightly above instruments targeting other objectives on average, while instruments supporting research (research excellence, technology transfer, R&D infrastructure, etc.) scored slightly lower on average. Figure 9: Scores by 5,0 Policy Objectives of Instruments Source: Authors’ calculation 4,0 design 2,9 2,9 2,9 2,9 implementation 3,0 2,8 2,8 2,6 2,7 2,6 2,5 2,4 2,4 governance 2,0 1,0 innovation in firms other business support skills research Instruments directly implemented by the Ministry of Economy (not including instruments financed by OPIC) stand out as having high scores, particularly in the areas of design and governance, as shown in Figure 10. Instruments from the two key STI-related operational programs (administered by the Execu- tive Agency for OP Science and Education for Smart Growth [EA OPSESG] and Directorate General OP Innovation and Competitiveness [DG OPIC]) had similar scores overall, with EA OPSESG scoring higher in design and implementation and lower in governance. Instruments administered by the National Science Fund (NSF) and Ministry of Education and Science (MoES) tend to have the lowest scores observed in the policy mix, particularly in the areas of design and implementation. This is important, as these organizations are the primary funders of basic research to the Bulgarian public sector, and Phase I of this project identified the poor research performance of the public sector as a large contributing factor to the country’s overall poor innovation performance (Additional details on resources and functionality of basic research instruments are provided in Box 1). Figure 10: Scores 5,0 by implementing organization Source: Authors’ calculation 4,0 3,3 3,4 design 3,2 3 2,8 2,8 2,8 2,9 implementation 3,0 2,7 2,6 2,5 2,2 2,3 governance 2,1 2,1 2,0 1,0 nsf moes ma sesg dg opic moe 26 Bulgaria Functional and Governance Analysis Box 1 Support for Basic Research in Bulgaria Instruments that support basic research in Bulgaria are implemented by the NSF and MoES (shown in table B1), although there are additional instruments within the STI portfolio that provide funding for both basic and applied research proj- ects. Budgets for basic research support instruments tend to be small, with no in- strument having a budget larger than €26 million over the 2014-2021 programming period. By contrast, the budgets for the OPSESG instruments Centres of Com- petence (€108.8 million) and Centres of Excellence (€80.8 million), which support R&D infrastructure, are much higher and the budget for the applied research-fo- cused National Science Programmes is €26.4. Box Table B1: Basic Instrument Implementor Beneficiaries Allocated Budget Instrument Type Research Instruments 2014-2021 Fundamental Research NSF PROs, HEIs, €25,447,981 R&D Grants Researchers Financing of scientific MoES PROs, HEIs €24,599,883 R&D Grants or artistic activity inherent in public higher education institutions Fundamental Research NSF PROs, HEIs, €2,914,000 R&D Grants on Societal Challenges Researchers Bilateral cooperation NSF PROs, HEIs, €740,849 R&D Grants programmes Bulgaria-Russia Researchers When looking at the functionality of basic research support instruments, a few ar- eas of strength and many areas of weakness emerge. In design, instruments have strong origination processes, including engagement with external stakeholders, and well cataloged program inputs, but none of the instruments use logic models or theories of change or well-defined objectives, resulting in poorly defined and disconnected indicators for program outcomes. In implementation, these instru- ments received relatively high scores in solicitations, eligibility criteria and selec- 27 Bulgaria Functional and Governance Analysis Box 1 Support for Basic Research in Bulgaria tion processes, but suffer from a lack of implementing capacity due to lack of staff and funding. This lack of capacity severely impacts M&E, as well. In governance, the instruments can be impacted by external rules and regulations, such as State Aid and procurement laws, but tend to have average scores related to inter-orga- nizational coordination. Box Figure B1: design Functional and Governance Analysis implementation Scores for Basic Research Instruments governance Source: Authors’ calculation Interaction of Jurisdiction Rules and Regulations Program Origin (Seriousness of external constraint) Program Justification Interaction of Jurisdiction Rules and Regulations (Internal response) Portfolio Relationship Institutions Relationship Program Objectives Programs Relationship Alternative Instrument M&E Implementation Logical Framework Process monitoring Inputs Incentives (related to Activities performance with the instrument) Products/Outputs Staff & training Main beneficiaries Roles & Autonomy Selection criteria Program management and organization quality Audiences Budget and financial resources Expected Outcomes & Impact Project closures Monitoring & Evaluation Program database and information Learning evidence during on participants and applications implementation Application & Selection Processes Solicitations (calls for proposals and their products) Eligibility Criteria and Application Information 28 Bulgaria Functional and Governance Analysis Instruments targeting researchers, universities, and other researcher entities received the lowest scores on average, while those that target formal firms, clusters and consortia, and business support institutions received the highest scores, as shown in Figure 11. This continues the trend of lower scores for instruments that target research activities. Figure 11: Scores by targeted beneficiaries Source: Authors’ calculation 5,0 design implementation 4,0 governance 3,1 3 3 2,9 2,9 2,9 2,9 3,0 2,8 2,7 2,8 2,8 2,8 2,7 2,7 2,4 2,4 2,5 2,5 2,5 2,5 2,5 2,4 2,3 2,3 2,3 2,3 1,9 2,0 1,0 other business consortia, formal research private other universities researchers government support associations, firms institutes research research agencies institutions clusters entities entities Instruments with large budgets (€100 million or more) received slightly higher scores than instruments with smaller budgets, as shown in Figure 12, although there was not a large amount of variance between instruments based on budget size. Figure 12: Scores by size 5,0 of instrument budget Source: Authors’ calculation 4,0 design implementation 3 2,9 2,9 2,7 2,8 2,8 3,0 governance 2,6 2,5 2,6 2,0 1,0 small medium large 29 Bulgaria Functional and Governance Analysis 2.1 Cluster Analysis A cluster analysis was conducted to identify connections and patterns of functionality. The analysis found that many of the differences in functionality across programs can be attributed to the implement- ing body of the instruments. A cluster analysis is an unsupervised machine learning technique that helps guide the detection of patterns in complex data. The analysis uses measures of distance between items to group those that seem close to each other. The clustering of STI instruments produced seven clusters of similar instrument groups, which are largely divided by implementing STI bodies (see Table 3). This indicates that many of the differences in instrument functionality across the portfolio are likely due (at least in part) to the organization that designs and administers the individual instruments. This means that reforms aimed at improving the functionality of instruments should be addressed at the level of the implementing bodies, rather than through portfolio-wide reforms. It also means there may be opportunities for cross-organizational learn- ing of good practices. Table 3: Instrument Clusters Cluster Instrument STI Body 1 National Science Programs 2018-2022 MoES Doctoral fellowships MoES Young Scientists and Postdoctorates MoES 2 VIHREN NSF Bilateral cooperation programs Bulgaria-Russia NSF Fundamental Research NSF Fundamental Research on Societal Challenges NSF 3 Phase 2 Sofia Tech Park DG OPIC Support for EA Accreditation Service DG OPIC 4 Centres of Competence EA OPSESG Centres of Excellence EA OPSESG Enhancing the growth of SMEs through pilot DG OPIC application of a voucher scheme by BSMEPA Support for pilot and demonstration DG OPIC initiatives for effective use of resources Enhancing entrepreneurship DG OPIC Improving the production capacity of SMEs DG OPIC 30 Bulgaria Functional and Governance Analysis Cluster Instrument STI Body 4 Development of management capacity and growth in SMEs DG OPIC Development of Product and Process Innovations DG OPIC Support for development of innovations DG OPIC by start-up companies Development of clusters in BG DG OPIC Support for the Introduction of DG OPIC Innovation in Enterprises Energy Efficiency for SMEs DG OPIC Increasing Energy Efficiency in Large Enterprises DG OPIC 5 Technostart MoE National Innovation Fund SMEPA 6 Support for entrepreneurship MLSP Risk sharing micro finance facility FoF 7 Financing of scientific or artistic activity inherent in public higher education institutions MoES Research Infrastructure MoES 31 Bulgaria Functional and Governance Analysis Cluster 1. Cluster 1 comprises three instruments administered directly by MoES: two programs providing fund- ing for researchers and one for funding research. As shown in Figure 13, these programs suffer from low scores related to the lack of use of logic models and poorly defined indicators for program inputs, activities, outputs, and outcomes. Scores for implementation are also low in general, particularly vari- ables related to M&E, data management, and human resources, all stemming from a lack of capacity and resources needed to implement the MoES portfolio. Governance scores are generally average or slightly below average, except for low scores for institutional relationships, as programs in this cluster lack coordination with institutions outside of MoES. 1,0 2,0 3,0 4,0 5,0 Figure 13: Cluster Program Origin 3,3 1 scores (MoES Program Justification 3,3 instruments) Portfolio Relationship 3,7 Source: Authors’ calculation Program Objectives 2,3 Alternative Instrument 2,3 design Logical Framework 1,0 Inputs 2,0 implementation Activities 2,0 governance Products/Outputs 2,7 Main beneficiaries 3,0 Selection criteria 2,0 Audiences 4,0 Expected Outcomes and Impact 1,0 M&E (design) 1,3 Learning 2,0 Solicitations 1,0 Eligibility Criteria & Application Information 2,0 Application & Selection Processes 2,3 Program database 1,0 Project closures 1,0 Budget & financial resources 2,0 Program management 2,0 Roles & Autonomy 3,0 Staff & training 1,0 Incentives 3,0 Process monitoring 2,0 M&E (implementation) 1,0 Programs Relationship 2,7 Institutions Relationship 1,3 Interaction of Rules and Regulations (Internal) 3,3 Interaction of Rules and Regulations (External) 3,0 32 Bulgaria Functional and Governance Analysis Cluster 2. Cluster 2 comprises four programs administered by NSF: three research funding programs and one aimed at attracting researchers from abroad. As shown in Figure 14, instruments in this cluster also suf- fer from very low design scores related to the lack of use of logic models and poorly defined indicators for outputs and expected outcomes. While NSF instruments received the highest average scores relat- ed to evaluations of applications, its programs suffer from low scores in other implementation variables due a severe lack of capacity and resources needed to implement its program portfolio. 1,0 2,0 3,0 4,0 5,0 Figure 14: Cluster Program Origin 3,8 2 scores (NSF Program Justification 2,5 instruments) Portfolio Relationship 2,8 Source: Authors’ calculation Program Objectives 1,8 Alternative Instrument 1,5 design Logical Framework 1,0 Inputs 4,0 implementation Activities 2,0 governance Products/Outputs 1,0 Main beneficiaries 3,0 Selection criteria 2,3 Audiences 2,0 Expected Outcomes and Impact 1,3 M&E (design) 2,0 Learning 2,3 Solicitations 2,5 Eligibility Criteria & Application Information 3,3 Application & Selection Processes 3,8 Program database 1,0 Project closures 2,0 Budget & financial resources 2,5 Program management 1,5 Roles & Autonomy 2,8 Staff & training 1,0 Incentives 1,0 Process monitoring 3,0 M&E (implementation) 1,3 Programs Relationship 2,8 Institutions Relationship 3,5 Interaction of Rules and Regulations (Internal) 2,5 Interaction of Rules and Regulations (External) 3,0 33 Bulgaria Functional and Governance Analysis Cluster 3. Cluster 3 comprises two instruments financed by OPIC: funding for the Sofia Tech Park (a research and business infrastructure project) and a direct institutional support program. These instruments score highly in program origination and program closures (as do nearly all OP-financed instruments), but again suffer from low scores related to the lack of use of logic models and poorly defined indicators. They also have low scores related to management and human resources due to a lack of manpower and staff autonomy. The implementation of these instruments is also impeded by external rules and regulations, resulting in lower than average scores in governance. 1,0 2,0 3,0 4,0 5,0 Figure 15: Cluster Program Origin 5,0 3 scores (DG OPIC Program Justification 2,0 instruments) Portfolio Relationship 2,5 Source: Authors’ calculation Program Objectives 2,0 Alternative Instrument 1,0 design Logical Framework 1,5 Inputs 2,0 implementation Activities 2,5 governance Products/Outputs 3,0 Main beneficiaries 3,0 Selection criteria 3,5 Audiences 3,0 Expected Outcomes and Impact 1,0 M&E (design) 2,0 Learning 2,0 Solicitations 2,0 Eligibility Criteria & Application Information 2,0 Application & Selection Processes 2,5 Program database 2,0 Project closures 5,0 Budget & financial resources 2,5 Program management 2,0 Roles & Autonomy 3,0 Staff & training 2,0 Incentives 4,0 Process monitoring 2,0 M&E (implementation) 2,5 Programs Relationship 1,5 Institutions Relationship 3,0 Interaction of Rules and Regulations (Internal) 1,5 Interaction of Rules and Regulations (External) 2,0 34 Bulgaria Functional and Governance Analysis Cluster 4. Cluster 4 is a large cluster comprising the majority of the OPIC portfolio of instruments (primarily fo- cused on supporting innovation and technology adoption in firms) and two instruments from OPSESG (focused on research infrastructure). These instruments score very highly in program origination, au- diences, and program closures – there is a clear opportunity for good practices from the operational programs in these areas to be translated to the national programs, which tend to have lower scores for these variables. However, like the rest of the STI portfolio, these instruments have low average scores for the lack of use of logic models and poorly defined indicators (particularly for expected outcomes and impacts). Justifications are a consistent problem area in this cluster, as these instruments did not clearly identify the system failure the instrument was intended to address, which contributed to low scores in other variables related to instrument design. These instruments also suffer from low scores in solicitations, due to the fact that most of these programs only issued a single call for solicitations (as opposed to a series of calls), which severely limits opportunities for iteration and improvement of the instrument over time. In variables related to governance, OP instruments tend to have higher than average scores for institutional coordination. 1,0 2,0 3,0 4,0 5,0 Figure 16: Cluster 4 Program Origin 5,0 scores (EA OPSESG and Program Justification 2,2 DG OPIC instruments) Portfolio Relationship 3,0 Source: Authors’ calculation Program Objectives 1,9 Alternative Instrument 2,1 design Logical Framework 2,0 Inputs 3,0 implementation Activities 3,1 governance Products/Outputs 4,0 Main beneficiaries 2,6 Selection criteria 2,2 Audiences 4,8 Expected Outcomes and Impact 1,9 M&E (design) 3,0 Learning 1,9 Solicitations 1,2 Eligibility Criteria & Application Information 2,1 Application & Selection Processes 1,8 Program database 2,0 Project closures 5,0 Budget & financial resources 3,2 Program management 2,0 Roles & Autonomy 3,0 Staff & training 2,2 Incentives 4,0 Process monitoring 2,3 M&E (implementation) 2,7 Programs Relationship 2,3 Institutions Relationship 3,2 Interaction of Rules and Regulations (Internal) 3,2 Interaction of Rules and Regulations (External) 2,6 35 Bulgaria Functional and Governance Analysis Cluster 5. Cluster five comprises two instruments: Technostart, an entrepreneurship support program adminis- tered directly by the Ministry of Economy (MoE), and the National Innovation Fund (NIF), administered by the Small and Medium Enterprise Promotion Agency (SMEPA). This cluster had some of the high- est average scores of all instruments analyzed, but still received low scores in a number of areas. Low scoring areas include program justifications and indicators for inputs and expected outcomes. Budget, organizational management, and human resources are also areas of concern. One of these instruments (Technostart) was cancelled due to its budget being reallocated, and both instruments suffer from a lack of capacity and resources to fully implement the programs. 1,0 2,0 3,0 4,0 5,0 Figure 17: Cluster 5 Program Origin 4,0 scores (MoE and SMEPA Program Justification 2,0 instruments) Portfolio Relationship 3,0 Source: Authors’ calculation Program Objectives 3,0 Alternative Instrument 3,0 design Logical Framework 2,0 Inputs 1,5 implementation Activities 4,0 governance Products/Outputs 3,0 Main beneficiaries 5,0 Selection criteria 4,5 Audiences 4,5 Expected Outcomes and Impact 2,0 M&E (design) 4,0 Learning 4,5 Solicitations 2,5 Eligibility Criteria & Application Information 4,0 Application & Selection Processes 4,0 Program database 2,5 Project closures 3,0 Budget & financial resources 1,0 Program management 1,5 Roles & Autonomy 3,0 Staff & training 2,5 Incentives 3,5 Process monitoring 2,0 M&E (implementation) 4,0 Programs Relationship 3,0 Institutions Relationship 3,0 Interaction of Rules and Regulations (Internal) 2,5 Interaction of Rules and Regulations (External) 5,0 36 Bulgaria Functional and Governance Analysis Cluster 6. Cluster six comprises two programs financed by the OP Human Resources Development (OPHRD) – one implemented by the Fund of Funds (FoF) and one by the Ministry of Labor and Social Policy (MLSP). These instruments scored above average in many of the variables related to instrument design, with the exception of program objectives, which were poorly defined, and expected outcomes and impacts. Scores were also above average for most implementation variables, as were those for governance. Overall, these instruments received some of the highest scores of the instruments analyzed. 1,0 2,0 3,0 4,0 5,0 Figure 18: Cluster 6 Program Origin 4,5 scores (FoF and MLSP Program Justification 3,5 instruments) Portfolio Relationship 3,5 Source: Authors’ calculation Program Objectives 2,0 Alternative Instrument 4,0 design Logical Framework 2,5 Inputs 2,5 implementation Activities 2,5 governance Products/Outputs 4,0 Main beneficiaries 4,0 Selection criteria 4,0 Audiences 4,0 Expected Outcomes and Impact 2,0 M&E (design) 3,5 Learning 3,5 Solicitations 5,0 Eligibility Criteria & Application Information 3,5 Application & Selection Processes 5,0 Program database 2,5 Project closures 5,0 Budget & financial resources 4,0 Program management 4,0 Roles & Autonomy 3,0 Staff & training 4,0 Incentives 3,0 Process monitoring 2,5 M&E (implementation) 3,0 Programs Relationship 5,0 Institutions Relationship 3,5 Interaction of Rules and Regulations (Internal) 3,5 Interaction of Rules and Regulations (External) 2,0 37 Bulgaria Functional and Governance Analysis Cluster 7. Cluster seven comprises two instruments directly implemented by MoES: one providing direct insti- tutional support to higher education institutions and one supporting research infrastructure. These programs are notable for high scores for program origin and audiences, which could provide examples of good practices for the rest of the MoES and NSF program portfolios. However, these instruments suf- fer from low scores in the use of logic models and indicators related to inputs, activities, outputs, and outcomes, much like other instruments implemented by MoES. These instruments also received low scores in governance related to interaction with external rules and regulations, as the implementation of these instruments has been impeded by rules – particularly the Public Procurement Act (see Section 3.9). 1,0 2,0 3,0 4,0 5,0 Figure 19: Cluster Program Origin 5,0 7 scores (MoES Program Justification 4,0 instruments) Portfolio Relationship 3,5 Source: Authors’ calculation Program Objectives 2,5 Alternative Instrument 3,5 design Logical Framework 1,0 Inputs 2,0 implementation Activities 3,0 governance Products/Outputs 2,0 Main beneficiaries 3,0 Selection criteria 3,5 Audiences 4,5 Expected Outcomes and Impact 1,5 M&E (design) 2,5 Learning 3,5 Solicitations 4,0 Eligibility Criteria & Application Information 3,5 Application & Selection Processes 3,5 Program database 1,5 Project closures 3,0 Budget & financial resources 1,5 Program management 3,0 Roles & Autonomy 3,0 Staff & training 1,0 Incentives 3,0 Process monitoring 2,0 M&E (implementation) 1,5 Programs Relationship 3,0 Institutions Relationship 3,0 Interaction of Rules and Regulations (Internal) 4,0 Interaction of Rules and Regulations (External) 2,0 38 Bulgaria Functional and Governance Analysis 3 Cross-Cutting Challenges The functional analysis identified a number of cross-cutting challenges that pervade the STI portfolio. What follows are a set of areas where active improvements could yield better functionality and governance of the STI policy mix: Develop justifications for policy interventions. Instruments generally do not define or quantify the specific market or system failure being ad- dressed by the intervention. The lack of explicit identification of the failure being addressed leads to a disconnect between program activities and high-level objectives and contributes to other problems with instrument design. Articulate theories of change for the individual instruments. Instruments generally have no or poorly articulated theories of change, which also contributes to the disconnect between activities, results, and objectives and to challenges related to the rele- vance of indicators used. Rationalize budgeting and solicitations. Overall, there appears to be little connection between the size of a program’s budget and its policy objectives and targeted population, which is tied to the lack of identification and analysis of the specific failure being addressed by instruments. There are huge disparities between the budgets 39 Bulgaria Functional and Governance Analysis of OP-funded and nationally-funded instruments, and insufficient and unpredictable budgets im- pair the functionality of almost all nationally-funded instruments. Further, many instruments by design only issue a single call for proposals, which limits opportunities for learning, adaptation, and improvement of the instruments over time. Improve resources for monitoring and evaluation. Much of the monitoring done for STI instruments is focused on whether beneficiaries are in com- pliance with rules and regulations, rather than on tracking program performance, and very few evaluations of any kind have been done on STI instruments in the current programming period. Lack of resources is cited as the primary reason for these challenges by both national and OP funded programs. Improve targeting of beneficiaries and selection criteria. Many instruments suffer from overly generic selection criteria that do not target projects with specific desirable characteristics, and selection criteria tend to favor projects that are closer to market over those that face more risk. Improve quality outcomes of project selection. The evaluation of project proposals has been highlighted as an area of concern in past assess- ments of the STI system, and selection processes are not up to international standards. While most STI bodies use external reviewers to review project applications, the DG OPIC relies heavily on internal panels to evaluate proposals. STI programs also rely almost exclusively on domestic experts for evaluation panels. Ease administrative burdens on beneficiaries. While beneficiaries of programs financed by the OPs use the online UMIS portal for everything from applications to contracting to reporting, the nationally-financed programs use a mix of pa- per-based and online systems that are generally more burdensome than those for the OPs. Public procurement processes are particularly burdensome on beneficiaries and have caused notable delays in the implementation of several programs. Invest in professional staff to improve implementation capacity. Organizational capacity and human resource management is a pervasive challenge across the STI system. Nationally- financed programs suffer from severe shortages of staff and expertise, which negatively impact program implementation. Staff, particularly for nationally-funded instruments, are rarely provided with relevant training opportunities, and few STI personnel have performance incentives that are tied to the programs they work on. Activate governance and coordination of STI policies. STI institutions are disconnected from one another, resulting in fragmented policies and pro- grams. In general, there are minimal overlaps between instruments across the STI portfolio, but there are also very few synergies between instruments. 40 Bulgaria Functional and Governance Analysis 3.1 Developing Justifications for Policy Interventions Every government policy or program contains an implicit or explicit argument that links policy goals to the intervention selected to achieve those goals. This argument is the justification for the intervention. A program’s justification should explicitly identify the system failure that the instrument is intended to address (system failure here is defined as the necessary social good that cannot be accessed by ben- eficiaries without government action or without a change in its current form of action [Edquist 2011]). This argument is important because it focuses the instrument design on the intended problem and affected population, and it helps to avoid situations where programs are developed solely for political purposes or as “pet projects” of powerful stakeholders. It also protects the instrument design process from external interference from actors who may seek to influence the design to their own benefit. Diagnosis and justification are critical for designing effective policies that address failures with the innovation system. Camagni and Capello (2013) studied the different patterns of innovation in Euro- pean regions and the constraints they imply for innovation policies and demonstrated that standard innovation policies, such as R&D funding, facilitating foreign direct investment, and encouraging pri- vate-public R&D collaboration, did not produce equal results across regions. Similarly, Haapanen et al. (2014) showed that the choice of instruments cannot be done based only on generic criteria and that common instruments, such as grants, tax credits, or allocation rules based on past performance, are not universally efficient. Their effectiveness depends on many specific factors of the context, such as the severity of financial constraints and the capabilities of the firms themselves. The system failure and its causes must be identified and an analysis or sensible estimation of its size or intensity must be provided to gauge the scale of the intervention that is necessary. Justifications should also consider how this intervention would fit in the larger government portfolio of currently im- plemented programs and strategies. Diagnostics of failures are particularly important in the coming period of post-pandemic recovery. Bul- garian policy makers will need to understand the extent to which the firms’ current liquidity problems are turning into insolvency and leading to bankruptcy, which could destroy productive capacity. They will also need to understand the uptake and effectiveness of relief policies to avoid the introduction of distortions and subsidies to limit the dangers of misallocation and a further slowdown in productivity. 3.1.1 Current Practices in Bulgaria Bulgarian STI instruments are generally designed to achieve one or more objectives or priorities con- tained within national STI strategies. These strategies develop priorities using national-level diagnoses to identify gaps in the STI system, but at the level of the individual instruments, few instruments re- ceived further analysis to identify, describe, or quantify the specific system failure the instruments are intended to address. The only instruments to receive above average scores for program justification were those implemented by MoES and those financed by OPHRD (clusters 1, 6, and 7 of the cluster 41 Bulgaria Functional and Governance Analysis analysis in Section 2.1 of this report), while the rest of the STI portfolio received low scores due to a lack of specific diagnosis of the failure being addressed. Operational programs develop justifications through ex ante evaluations, but in practice ex ante evalu- ations are largely performed as straightforward feasibility studies, rather than proper statements of the problems instruments are intended to address and the antecedents that would provide lessons for new versions of past policies. Experiences from previous programming periods in the EU have shown that the ex-ante evaluations are carried out at the OP level, covering many instruments, so the specification of targets is often not properly connected to individual instruments or projects. Further, the justification of instruments may not be articulated fully – often the only argument offered is that there is a deficit of government funding in the area, rather than an analysis as to why the intervention is necessary in the first place (Blažek and Vozáb, 2007; Smismans, 2015; de Jong and Muhonen, 2020). World Bank func- tional analyses in Poland, Croatia, and now in Bulgaria have revealed these same deficiencies in OP instruments. For example, the OPIC instrument “Support for cluster development in Bulgaria” supports the innovation activities of industry clusters in Bulgaria in select sectors. OPIC’s ex ante evaluation (Ministry of Economy, 2014) provides a thorough description of Bulgarian firm innovativeness over the previous programming period and the role of clusters in supporting innovation but does not explain the need or justification for government intervention in supporting private sector cluster activities. An example of a program designed around an identified failure is provided by the MoES Doctoral Fel- lowships program (Box 2). Box 2 Addressing the gap The MoES Doctoral Fellowships program was created based on a recommenda- in human tion from a Horizon 2020 panel report (European Commission, 2015), which found capital for that Bulgaria has fewer researchers per capita in all disciplines compared with the EU average, and nearly half of its professors are over 65 years of age. Young research researchers generally emigrate to other EU countries or find jobs outside R&D in search of better pay and better opportunities. In addition, Bulgarian PhD candi- with post- dates are provided with a very traditional education and trained exclusively in the doctoral Bulgarian language, reducing the “transferability” of their skills and expertise to international and private sector positions. fellowships To address these challenges, the report recommended that Bulgaria introduce Source: Doctoral dedicated fellowships to target the retention of excellent graduates and attract Fellowships more international doctoral candidates. Program; European Commission, 2015 Based on this analysis and recommendation, the Doctoral Fellowship program was created, using the Marie Sklodowska Curie Individual Fellowship Scheme (part of Horizon 2020) as a model. The program aims to accelerate the reintegration and career development of scientists in Bulgarian universities and research organiza- tions by attracting and supporting scientists in their first career steps in the coun- try through funding for postdoctoral projects at Bulgarian research institutions. 42 Bulgaria Functional and Governance Analysis The lack of explicit justifications for individual instruments contributes to the observed disconnect between program designs and high-level objectives, which is evident across the portfolio. For example, there are several programs that assume that research infrastructure is the key barrier to research excel- lence and improved technology transfer outcomes in the public research sector, although the sector is faced with a number of other important challenges, including lack of research funding and resources, lack of resources for technology transfer activities, weak attraction and retention of researchers, and organizational dysfunction. Moreover, the consideration of alternative instrument designs is often cursory, and instrument designs have been copied from other EU countries without strong consideration of the Bulgarian context or capacity of the implementing body. 3.1.2 Areas for improvement Strengthen analytic capabilities of implementing agencies to define and quantify market failure to be addressed by instruments. Consider full range of alternative instrument designs, taking into account beneficiaries, resourc- es, and capacity. Cirera et al. (2020) can serve as a useful reference for instrument selection and design5. 5 WB, 2020. A Practitioner’s Guide to Innovation Policy: Instruments to Build Firm Capabilities and Accelerate Technological Catch-Up in Developing Countries. Available here: https://openknowledge.worldbank.org/handle/10986/33269 43 Bulgaria Functional and Governance Analysis 3.2 Theories of Change Theories of change6 show how an instrument is supposed to work by depicting the linkages and as- sumptions that explain how the inputs, activities, outputs, and outcomes can connect to the ultimate objectives of the program (Hatry 2006). The formulation of a theory of change requires selecting and defining the intermediate steps that connect inputs and activities with the desired objectives, as well as explicit statements of the underlying assumptions, allowing for assessments of how and how much the instrument contributes to the ultimate objectives of the program. In the absence of a valid theory of change, a functioning instrument may have implicit relationships that connect program elements, but the relationships may not be operating as assumed and not everybody associated with the instrument may have the same understanding of what those relationships are and how they are supposed to work. Each theory of change also needs a results framework, which identifies and defines the indicators un- derlying the theory of change for the program. As such, the results framework identifies indicators for all the theory of change elements (inputs, activities, outputs, and outcomes) so that program “success”, in terms of whether the intended change(s) occurred, can be measured and verified. The results frame- work should specify targets for each indicator, as well as the processes of collecting and verifying the data (including indicator verification period, process, timeline for achievement, and frequency). Importantly, the use of theories of change and results frameworks are essential to embed M&E (another cross-cutting challenge in the STI portfolio, described in Section 3.4) into the design of the instrument, making performance traceable and measurable throughout implementation. Theories of change pro- vide a common language about the expectations of the program and help benchmark progress with the desired targets or results. They are also important for knowledge management and learning, providing a structure for documenting adjustments to instruments, supporting continuity with subsequent inter- ventions, informing resource allocation, and increasing the likelihood that lessons learned will be used in the future. 3.2.1 Current Practices in Bulgaria While EU and national regulations specify procedures for instrument design and implementation, and sometimes require collecting and reporting on program indicators, these regulations do not mandate or encourage the adoption of an M&E strategy, including the use of theories of change. As a result, Bulgarian STI instruments generally have no or poorly articulated theories of change, which leads to disconnects between program activities and objectives. In fact, none of the analyzed instruments used theories of change at the level of the instrument – the operational programs are required to use a sim- plified logical framework at the level of the overall OP, but not at the level of the individual instruments. 6 Theories of change and logic models are interchangeable terms describing models that depict how and why a desired change is expected to occur. This report uses the term theory of change to describe such tools. 44 Bulgaria Functional and Governance Analysis Most national instruments have no logical framework at all. Further, the OPs are constrained in the setting of output and outcome indicators by EU regulations7, which require the use of the output indicators listed in the regulations of the European Regional Devel- opment Fund and that indicators be statistically verifiable. The lack of incorporation of M&E strategies, including theories of change, and constrains in the setting of program-level indicators has contributed to a number of challenges related to the indicators used by the analyzed instruments: Most program documents provide a list of indicators without specifying if they are inputs, ac- tivities, outputs, or outcomes, and there is generally a lack of clear connections among these indicators. While instruments generally have a good catalog of indicators for activities and outputs, they are not logically linked to outcomes in any clear way. Most instruments have very poorly defined outcome indicators, and some have no outcome in- dicators at all. There are few indicators related to process or portfolio management. 3.2.2 Areas for improvement Articulate theory of change for each instrument (see an example in Box 3) through collaborative processes with key stakeholders. Develop results frameworks, defining indicators for inputs, activities, outputs, and outcomes, along with targets and data collection protocols for each indicator. 7 Regulation (EU) No. 1303/2013, Common Provisions Regulation. 45 Bulgaria Functional and Governance Analysis Box 3 Creating Theories of Change for STI Programs As part of the Bulgaria PER STI project, theories of change and results frameworks were developed for four key programs in the Bulgarian STI portfolio as part of a capacity building exercise aimed at improving M&E practices of STI instruments in Bulgaria (see a full description of this exercise in Appendix IV). This box details the creation of a basic theory of change for the Support for the Development of the Centres of Excellence program, financed by OPSESG. According to program documentation8, the program’s primary objective is to “sup- port the enhancement of the level and the market orientation of the research ac- tivities of leading research organizations in Bulgaria and to improve the capacity for generation of research excellence”, while the main activities supported by the program are the construction of new research facilities, support for R&D in public research organizations, and dissemination of supported research. These activities are clearly connected to achieving research excellence, but without an explicit theory of change, it is unclear how these activities are connected to increasing the market orientation of public research. The development of a theory of change begins with identification of the “focal problem” – that is, the key challenge(s) the program is trying to solve – and then identifying the “root causes” of this challenge. The root causes are important be- cause this is the level at which activities can be designed, while the focal problem is merely a symptom of the root causes. Root causes may also have one or more factors that contribute to them. In our view, the key challenge being addressed by the Centres of Excellence pro- gram is that the Bulgarian public research sector currently does not produce high-quality research that is impactful on the international level (research ex- cellence) or relevant to the needs of the private sector (industry relevance). We identify three root causes: 1) poor research capacity (due to lack of human capital and poor research infrastructure); 2) a lack of linkages to international and private sector researchers (due to lack of researcher mobility and lack of research collab- oration activity); and 3) a lack of technology transfer activity (due to a lack of pub- lic-private research collaboration and lack of knowledge of technology transfer opportunities, adequate legislation and respective procedures). 8 http://sf.mon.bg/?h=newsfile&newsfileId=605 46 Bulgaria Functional and Governance Analysis Box 3 Once the key challenges and root causes are identified, they can be “flipped” (or written as objectives to be achieved) to create a basic theory of change frame- work, as illustrated by Figure B1. Once the theory of change diagram is developed, a results framework is devel- oped to define indicators for each theory of change element (input, activity, out- put, outcome, and objective), allowing for tracking and measuring of the intended change of the intervention. Figure B2: Simplified ToC diagram for the The detailed theory of change and results framework for the Centres of Excellence Centres of Excellence and three other key STI programs can be found in Appendix IV. program input activity output outcome Improved human capital Improved capacity of program strategic in the public researchers goal goal research to produce sector excellent research Improved R&D Increased capacity and capacity of research in- the Bulgarian Improved frastructure public research availability of Bulgarian sector to pro- of qualoty public re- duce research research search organ- outputs that facilities and izations to are impactful equipment enable them on the economy to perform in- and society ternationally Increased competitive collaboration research with activities economic and Improved societal value collaboration and linkages with the international Improved scientific international community mobility of researchers Increased outreach and awareness of TT activities assumption Improved technology Improved transfer public-private activity from research the public to collaboration the private sector 47 Bulgaria Functional and Governance Analysis 3.3 Budgeting and Solicitations The availability of adequate financial resources is a straightforward requirement for policy instruments, but it is a pervasive challenge in Bulgaria. Without sufficient budgets, programs may not be able to fund necessary staff and expertise or be able to commit adequate resources to achieve program objectives. Programs can and often do operate even with inadequate resources due to their political or symbolic value or simply because of institutional inertia. Further, institutional rules or budget management may require disbursements that are unrelated to policy objectives, distorting or undermining program effi- ciency and efficacy. Of equal importance to the availability of sufficient financial resources is sustained and consistent funding for programs. Programs that operate for multiple years and issue multiple solicitations have the opportunity to engage with beneficiaries, evaluate performance, benefit from economies of scale, and improve aspects of the program through learning. Programs that only issue a single call or that issue irregular and sporadic solicitations create uncertainty among the beneficiary population, have more limited opportunities for knowledge management and improvement, and may have more difficulties in achieving policy objectives. 3.3.1 Current Practices in Bulgaria Overall, there appears to be little connection between the size of a program’s budget and its policy objectives and targeted population. This is tied to the lack of identification and analysis of the specif- ic failure being addressed by instruments (see Section 3.1 on justifications); these analyses can help quantify the size/scale of the failure, the size of target beneficiary population, and thus help to quantify the resources needed to address it. Instead, program budgets are more a function of whether they are part of an operational program (OP instrument budgets can be orders of magnitude larger than national instrument budgets). For nationally-funded instruments, allocations are determined by the Ministry of Finance, with little or no input from administering bodies. National instruments, in particular, suffer from inadequate and unpredictable budgets (see Box 4), re- sulting in a lack of staff and expertise among implementing bodies, insufficient funding for program activities (i.e., solicitations receiving many more high-quality applications than the program can fund), and in some cases program cancellation. Notably, MoE’s well-regarded Technostart program, which re- ceived some of the highest scores among the instruments analyzed in this report, was cancelled when its funding was withdrawn in 2018 by the Ministry of Finance. The global COVID pandemic has exacerbated existing budgetary challenges. Several programs from the OPIC portfolio were delayed indefinitely, including the development of a series of regional innovation centers – a key strategic initiative of the current programming period. Funds were reallocated to the Bulgarian Development Bank for COVID relief measures, and it is unknown when and if the affected programs will be restarted. 48 Bulgaria Functional and Governance Analysis Box 4 Bridging budget disparities between nationally- The Phase I of this project found that Bulgaria is heavily reliant on European fund- funded and ing for its STI agenda. Bulgaria has the lowest level of government budget appro- priations on R&D (GBARD) per capita in the EU and less than ten percent of the OP instru- EU 28 average. This underscores that EU structural funds are a critical source of ments STI financing in a fiscally constrained environment. One of the negative impacts of this overreliance on operational program funding is that national STI instruments and organizations outside of the operational programs suffer from a severe lack of resources, weakening national institutional capacity outside of the administrative apparatus of the OPs. However, two of the highest scoring instruments from this functional analysis ex- ercise are nationally financed: NIF, focused on supporting innovation in firms; and Technostart, which supports entrepreneurship and startup creation among uni- versity students. As can be seen in Table B1, these programs have relatively small budgets compared to OP-funded instruments that also support entrepreneurship and innovation in firms (the Technostart program was actually cancelled when the Ministry of Finance reallocated its budget in 2018). Box Table B2: Instrument Funding source Allocated budget 2014-2021 Budgets of selected instruments supporting Support for the introduction of innovation in enterprises OP 95,073,668.88 innovation in firms and entrepreneurship Development of product and process innovations OP 51,513,826.79 Promoting entrepreneurship OP 42,716,611.44 Support for development of innovations OP 17,543,339.86 by start-up companies NIF National 12,767,156.04 Support for entrepreneurship OP 9,985,816.98 Technostart National 444,721.68 The new programming period presents an opportunity to scale up the budgets and activities of high performing national programs, such as NIF and Technostart. This would provide the dual benefits of not only increasing the scale of proven policy instruments, but also building up the organizational and professional capa- bilities of national (non-OP) implementing bodies through programs with estab- lished good practices. In the coming period, NIF is likely going to be administered by the new State Agency for R&I (see Box 8), which would make this an opportune moment to revisit the size and scope of the program. 49 Bulgaria Functional and Governance Analysis Budgetary shortages are cited as one of the primary reasons for the lack of evaluations of STI programs (see the discussion in Section 3.4 on M&E). Yet, M&E generally constitutes a small portion of overall pro- gram budgets and can yield large financial and nonfinancial benefits; evaluations provide evidence that can be used to improve program impact and efficiency, lead to the cancellation of ineffective programs (thus saving budgets that can be reallocated to more effective interventions), and provide information on the effectiveness of public programs that can be communicated to stakeholders and the general public. Importantly, many of Bulgaria’s STI support instruments have only issued a single call for proposals over the lifetime of the program. This is particularly true for the instruments of the two primary STI operation- al programs, OPSESG and OPIC. Many of the OPIC-financed instruments were designed to issue two calls for proposals over the course of the programming period, but the COVID-19 pandemic and sub- sequent economic crisis led to shifts in funding and priorities, resulting in the cancellation the second wave of calls for proposals for many OPIC instruments. The primary OPSESG instruments, the Centres of Excellence and Competence, are long-term infrastructural projects, and thus not suitable for annual calls for proposals. Nonetheless, these programs might have benefited from contracting half of the planned centres in a single call, learning from this experience and making any necessary adjustments, and then contracting the rest of the planned centres in a second wave of solicitations. As mentioned above, the lack of regular solicitations limits opportunities for program improvement through evaluation and learning; can create uncertainty among beneficiary populations, as each call for proposals is for a new program with no continuity; and such programs do not benefit from economies of scale and accumulated institutional knowledge, reducing returns on public investment. 3.3.2 Areas for Improvement Allocate budgets based on justification for intervention (see Section 3.1), with flexibility for chang- es in funding if necessary Issue multiple solicitations at annual or semi-annual intervals 50 Bulgaria Functional and Governance Analysis 3.4 Monitoring and Evaluation Monitoring and evaluation is, at its core, tied to the public’s demand for government accountability. M&E demonstrates the performance and impact of publicly-funded programs and shows whether pro- gram targets have been met – in other words, M&E establishes whether a program does what it says it does. Information for M&E must be gathered in a timely and systematic manner, with data collection methods that match the intent and nature of the instrument. M&E is closely tied to knowledge manage- ment and should be used for the continuous improvement of policy interventions. Further, M&E data should be made available to other policy makers to aid in future policy design and, ideally, to the public for improved transparency and accountability of government programming. M&E tools and processes should be embedded into the instrument design process. This includes cre- ating explicit theories of change and results frameworks for each program (see Section 3.2), which include definitions for indicators related to program activities, outputs, and outcomes, as well as the assumptions about expected causal linkages that connect all these program elements. Also, it requires defining the evaluation strategy of the program and setting up strong information systems to make the program’s performance traceable and measurable throughout implementation. In general, programs should undergo periodic evaluations and contemplate various types of evalua- tions. This includes more frequent performance evaluations, which inform whether a program has been implemented as intended and whether the expected results are being achieved, as well as impact eval- uations, which assess the impacts the instrument has had on its beneficiaries and require a counter- factual analysis. The data and results collected from M&E should be used to adapt and improve programs through for- mal knowledge management processes. These feedback loops ensure that programs remain relevant to their objectives through continuous adjustments or are sunset when these objectives have been achieved or are no longer a priority. 3.4.1 Current Practices in Bulgaria Most instruments included in this analysis received below average scores for M&E design and imple- mentation; only the instruments financed by OPHRD and those implemented by SMEPA and DG OPIC received above average scores in these areas (clusters 4 and 5 of the cluster analysis in Section 2.1 of this report). At present, much of the monitoring done for STI instruments is focused on whether beneficiaries are in compliance with rules and regulations, rather than tracking and aggregating the performance of their projects. Further, only two evaluations have been done of STI instruments in the current programming period, both by the European Commission Joint Research Centre: program eval- uations the Sofia Tech Park in 2018 (European Commission, 2018) and the Centres of Excellence and Competence in 2020 (European Commission, forthcoming). The vast majority of instruments in the Bul- garian STI portfolio have not undergone performance or impact evaluations, and very few evaluations are planned for the future. 51 Bulgaria Functional and Governance Analysis Operational programs in Bulgaria follow a common M&E process. Program-level indicators are set as part of the OP design process, while project-level indicators may be established by the managing au- thority in the solicitation documents or by the applicant in their application or business plan. Benefi- ciaries must report on indicators in technical reports filed through UMIS when submitting a request for payment (see the process flow for project implementation in Appendix III). All requests for payment contain technical and financial reports by the beneficiaries. For some grants, such as those that in- volve the purchases of equipment, beneficiaries are obliged to report on project indicators for three years after project completion. Managing Authorities also carry out periodic on-site inspections of the projects. The UMIS system tracks all program data at both the program and project level, although the system’s functionality is very limited; generating reports on a program or project requires downloading, merging, and analyzing several different spreadsheets. For national programs, M&E processes depend on the individual institutions administering them. For NSF, beneficiaries report on a common set of indicators (publications, researchers involved in the proj- ect, presentations, and conferences, etc.) and only need to submit an interim and final technical and financial report during project implementation. For the NIF, there are no common indicators across projects – all indicators are set by the applicant in their application and business plan. Reporting re- quirements also depend on the specifics of the project. For some NIF grants, beneficiaries are obliged to report on project indicators for three years after project completion. Similar to the OPs, NIF staff also conduct periodic site audits of projects. For nationally-funded instruments, program databases are ru- dimentary – program data is generally stored in excel files, which are often not cloud based but instead stored on an individual’s workstation. There are no requirements for performance or impact evaluations to be undertaken at the instrument level for instruments financed by the OPs or from national funding agencies. OPIC underwent a mid- term evaluation in 2018 and all OPs will undergo evaluations at the end of the programming period, but these evaluations take place at the level of the overall operational program. There are no plans for the individual instruments financed by the OPs to undergo impact evaluations. The almost complete lack of performance or impact evaluations, both for the OPs and for nationally-financed instruments is due, in part, to the lack of evaluation requirement, as well as a lack of resources for such activities. Implement- ing bodies reported that they have neither the staff to conduct internal evaluations nor the funding to commission external ones on their instrument portfolios. Further, there are few external evaluation providers in Bulgaria with the capacity and expertise to provide such independent evaluation services. There are few formal knowledge management systems in place for documenting events and systemiz- ing adjustments to instruments or processes. Those adjustments to instruments are generally ad hoc and not well documented. In several instances, program staff did not know about key aspects of their programs because they had not been part of the program since its inception. M&E data from STI programs is also generally not available to the public or to outside policymakers. The National Statistical Institute’s Monitorstat portal (https://monitorstat.nsi.bg) was launched in 2020 with the aim of publishing data from the monitoring of government strategies and programs, but at present the functionality of the portal is limited. It only provides data on indicators at the strategic or OP (rather than for the individual instruments) and does not include any information on performance or impact evaluations. 52 Bulgaria Functional and Governance Analysis 3.4.2 Areas for Improvement Design an impact evaluation strategy for policy instruments that include clear objectives, theo- ries of change and results frameworks, evaluation plan, supporting systems, and protocols. The development of these strategies could be guided by an evaluation framework, such as that used in the Republic of Korea, as described in Box 5. Gorgens and Zall Kusek (2009) and Menon et al. (2009) can serve as useful references for developing M&E strategies and frameworks. Box 5 Perfor- mance evaluations The Korean government utilizes a five-year M&E plan for performance evaluation and of R&D programs, first developed in 2006, which provides protocols and sets the reporting standards for results frameworks and project and program evaluations. Data and results from these evaluations are then reported in National Science &Technol- on STI in ogy Information Service (NTIS). NTIS is an inter-ministerial knowledge portal for national R&D information operated by the Korea Institute of Science and Tech- the nology Information under the Ministry of Science and ICT. It includes information Republic of about R&D programs, projects, human resources, research equipment/facilities, and evaluation results. Korea Source: Frias et al., 2020 NTIS data are provided by government agencies, national statistical offices, pub- lic funders, patent offices, universities, and public research organizations, among others. By law, ministries are required to provide their M&E results for centralized management and dissemination of information through NTIS periodically, which helps ensure the sustainability of the system. Users include researchers from uni- versities, research institutes and enterprises, policy makers including ministries and management institutes, and the general public. NTIS usability has improved through the development of user-centric services, including business support services. In terms of features, NTIS offers charts and tables with time series data for a set of standard indicators such as R&D budget and expenditure, number of researchers, and technology transfer index. The system also offers features such as information on similar projects, performance evaluation reports, and collaboration maps. NTIS has brought several benefits to the R&D ecosystem. For example, it helped improve efficiency by helping identify and avoid redundant projects or programs. Also, it contributed to cost savings by avoiding purchase of redundant research equipment and promoting the utilization of idle or unused equipment. 53 Bulgaria Functional and Governance Analysis Conduct performance evaluations for all instruments at least once per programming period Conduct external impact evaluations for strategically important programs (e.g., with large bud- gets, numerous beneficiaries, or large expected impact), ideally by third parties, at least once per programming Define methodology for each evaluation and use transparent protocols for data collection and validation Share M&E data through the Monitorstat portal or elsewhere to ensure that M&E can be used across government agencies for improving policy design Strengthen M&E capabilities of implementing bodies, including additional specialized staff and training of existing staff 54 Bulgaria Functional and Governance Analysis 3.5 Targeting of Beneficiaries and Selection Criteria For a policy intervention to be effective, it must support the right activities among with right beneficiary population that will overcome the system failure the intervention seeks to address (see Section 3.1 on justifications). This requires analysis in the instrument design phase to explicitly identify the system failure, its size/scale, as well as the size of target population of beneficiaries. While beneficiary eligibility criteria and project selection criteria are related and intertwined, the dis- tinction between them is important. Eligibility criteria define who can apply for funding (for example, startups or university researchers), while selection criteria define the activities supported by the pro- gram and provide the framework for judging the quality and appropriateness of potential beneficia- ries’ project proposals. Critically, selection criteria are the means of targeting the optimal activities for achieving policy goals and maximizing the impact of public investments. Beneficiary and selection criteria are not only important elements to the functionality of instruments but are also critical for the legitimacy of public policies. Beneficiary criteria must be coherent with a rational basis in policy objectives, and not exclude certain populations for non-transparent reasons. All potential participants must have a fair chance of entering the selection process and receive clear feedback on the viability of their candidacy. Almost all public programs will have administrative selection criteria based on rules and regulations (related to tax liabilities, financing, etc.); selection criteria should primarily focus on the technical con- tent of the projects the instrument seeks to enable, under the assumption that technical content is the vehicle for achieving the policy’s goals. Good selection criteria are those that identify the projects most likely to meet program goals and maximize the impact for public funding. 3.5.1 Current Practices in Bulgaria While beneficiary criteria are generally consistent with policy objectives, some STI programs exclude certain private sector actors (firms, private universities and research centers, and other nonprofit or- ganizations) as eligible direct beneficiaries when there does not appear to be a clear rationale for do- ing so. Examples of this include National Science Foundation grants, National Science Programs 2018- 2022, Roadmap for Research Infrastructure, and the Centres of Excellence and Competence9. Further, in many of the instruments financed by OPIC, eligibility criteria exclude certain economic activities and 9 Eligible applicants under the National Science Fund and National Science Programmes are higher schools and scientific organizations accredited by the National Evaluation and Accreditation Agency (NEAA) to provide education in doctoral programmes. Eligible applicants under the procedures for CoCs and CoEs are public research organizations, including institutes of the Bulgarian Academy of Sciences and Agricultural Academy and higher education institutions. Other public and private research organizations, NGOs, and/or innovation clusters can form partnerships with beneficiary public research organizations but cannot be a sole beneficiary of support. Eligibility criteria for the National Roadmap for Research Infrastructure are unclear as relates to private sector beneficiaries. 55 Bulgaria Functional and Governance Analysis sectors (such as agriculture) due to division of labor between ministries, rather than for reasons related to the functionality of the instruments. Many instruments across the portfolio suffer from overly generic selection criteria that do not target projects with specific desirable characteristics, which can make it difficult for evaluation panels to make consistent project award decisions; this can, in turn, lead to the perception of an unfair or untranspar- ent awards process. These generic selection criteria are, in part, linked to the lack of explicit definition of the system failure the instruments are intended to address (see Section 3.1 on justifications) – be- cause failures are not well defined, it is difficult to develop selection criteria targeting projects that face these failures (and thus, face a high level of risk) but otherwise offer a high potential benefits. Beyond this, selection criteria in the majority of the programs do not consider target projects with the highest expected return on investment. In many cases, selection criteria overemphasize economic viability by favoring projects that are closer to the market, which penalizes higher-risk projects. This focus on already economically viable projects raises questions about whether government intervention is needed at all for these activities (and thus, whether the program should exist). Finally, selection criteria tend to prioritize Bulgaria’s Smart Specialization (S3) priority areas, even for programs that do not seem to have the benefit of the dynamic entrepreneurial discovery that defines S3 and without which it reverts to a classic static sector priority approach. Selection criteria for instruments financed by the OPs are discussed with a monitoring committee, which is comprised of representatives from across the government, as well as stakeholders from the private sector, before they are finalized and published (see the process flow for drafting a grant proce- dure in Appendix III). These issues with beneficiary and selection criteria are magnified by the fact that many programs in the STI portfolio only issued a single call for proposals (see Section 3.3). Providing for multiple solicitations would help fine tune selection criteria to improve their precision in selecting the right sort of projects (see Box 6). 3.5.2 Areas for Improvement Review the rationale for excluding private research organizations as potential beneficiaries for some research infrastructure and grant programs Selection criteria should focus on the projects that will maximize the impacts of public invest- ments. Improved program justifications (Section 3.1) and solicitation practices (Section 3.3) would help strengthen targeting. 56 Bulgaria Functional and Governance Analysis Box 6 Refining criteria over successive calls for proposals: NIF case The National Innovation Fund (NIF) was created in 2004 and is managed by the Bulgarian SME Promotion Agency. The main goal of the NIF is to promote research and development in the business sector and thus to increase the competitive- ness of enterprises through encouraging the development of new or improved products, processes, or services. To target projects for maximum impact, NIF selection criteria focus on three key areas: Innovativeness of the project: the extent to which the results of the project can lead to the achievement of new scientific or technological knowledge; the degree of novelty of the product or process; and additional benefits for the society, such as contribution to environmental protection, increase of safety of the working environment and quality of workplaces, occupational safety and health improvement of the population. Efficiency of the approach: adequacy of the proposed methodology; scien- tific and technical risks; quality of the work plan; and quality of any part- nerships involved in the project, including complementarity of partners and opportunities for future cooperation. Economic impacts: the extent to which the project is in line with the com- pany’s strategy; competitive advantages of the product or process; future market potential; feasibility of the commercialization plan; expected return on investment; contribution to enhancing the export potential of the com- pany; and contribution to the development of skills and competences of the applicant’s staff Since 2005, SMEPA has implemented eleven calls for proposals for NIF, provid- ing approximately €50 million in grants to over 500 enterprises. Selection criteria have been refined over these successive calls to better target younger firms, focus more on the implementing capacity of firms, and provide more weight to criteria related to innovativeness and economic impacts of the projects. This experience exemplifies the benefits of learning, continuity, and cumulative improvements to the targeting and selection criteria of STI policy instruments. 57 Bulgaria Functional and Governance Analysis 3.6 Improving Project Selection Selecting the right projects ensures that public program maximize the impact of public investments. Most grant schemes in developed countries rely on panels of experts (or peer reviewers) to evaluate project or research proposals. The composition and management of these expert panels is critical for consistent and credible evaluations. The use of qualified reviewers allows for accurate, informed as- sessments of whether project plans are viable, whether beneficiary teams have the skills and capacity to carry out the project, and to determine which projects have the highest potential impacts or returns on investment. Therefore, developing pools of high-quality experts that work on such evaluation panels is an essential step in creating effective programs. With the exception of a few large countries at the technological frontier (such as the United States or Germany), most research and innovation agencies will need to leverage international experts to sup- plement the available pool of domestic reviews. This is because the expertise to cover the breadth of research areas, technologies, and activities covered by their STI portfolio may not be available domes- tically. Further, the risk of biased project evaluation is higher in small countries that have smaller, more closed scientific or practitioner communities, which can lead to conflicts of interest (Khan 2015) and consequently lead to low quality evaluations. As such, effective selection panels require a well-devel- oped pool that contains a mix of experienced local and international experts. 3.6.1 Current Practices in Bulgaria A 2015 peer review of the Bulgarian STI system (European Commission, 2015) found a strong need to improve processes for the evaluation and funding of project proposals, due to the lack of confidence of the research community in the fairness of funding allocations and in the established peer review system for the evaluation of projects. There are no specific obligations for STI implementing bodies to employ external experts, and while most implementing bodies make use of external reviewers to review project applications, the DG OPIC (which has the largest STI portfolio in Bulgaria, in terms of funding) relies heavily on internal panels to evaluate proposals (Box 7 describes the use of foreign experts for evaluation in Poland). DG OPIC ad- ministers a large and diverse portfolio, covering projects for the construction of business infrastructure, resources efficiency upgrades, entrepreneurship support, and technology adoption across sectors and technologies, raising doubts as to whether DG OPIC has the internal expertise to judge the huge range of project applications they receive. Internal reviews also lack transparency and can be biased towards an institution’s cultural preferences, and for these reasons are generally not in line with good practices. 58 Bulgaria Functional and Governance Analysis Box 7 Poland NCBR’s Use of External Experts Source: NCBR, 2017 Poland National Centre for Research and Development (NCBR) maintains a pool of over 3,700 experts that its programs can use to form proposal evaluation panels. NCBR supports applied R&D projects by providing grants and equity instruments to businesses, universities, state-owned research institutes, consortia of these entities, and to investment funds (seed and venture capital). Its large expert pool was developed through an open solicitation mechanism. All interested experts (foreign and domestic) who apply to the open solicitation and meet predefined quality criteria are automatically added to the pool, which is up- dated on a rolling basis. This ensures that NCBR’s program portfolio has a large, diverse, and growing pool of experts to draw on for its evaluation panels. In a 2017 evaluation of the project selection system of the OP for Smart Growth, NCBR’s expert panels were singled out as praiseworthy instruments used by many of the OPs programs. Experts are assembled in panels for in person meetings with applicants for OP SG programs, and the applicants provide brief presentations to the experts, followed by 30-minute QA sessions. Surveyed experts and beneficia- ries both found this mechanism to be very useful and informative. 59 Bulgaria Functional and Governance Analysis The use of foreign experts is not common, as most implementing bodies that use external experts rely exclusively on domestic expert evaluators; NSF and EA OPSESG are the only implementors that use foreign experts to evaluate applications. This is due to two factors: first, project proposals are written in Bulgarian for almost all programs (OP and nationally financed), meaning that the proposals would need to be translated into English to allow for evaluation by foreign experts. Second, institutions generally have very low levels of funding to pay evaluators, and foreign experts generally demand higher compen- sation for their time than Bulgarian experts. NSF pays experts €70-120 per evaluation, and some instru- ments administered by MoES only pay €40 per evaluation, which is generally not enough to attract the services of foreign expert evaluators. As discussed above, relying exclusively on domestic experts is not in line with best practices, particularly in a small country like Bulgaria. Scientific and technical commu- nities are small, which raises the risk of conflicts of interest and biased evaluations, and the expertise available for specific research areas and technologies may not be available domestically. STI institutions have the ability to set up of pools of external experts that their programs can draw on for evaluation panels, but few implementors do so. EA SESG created such a pool of over 450 external experts, including foreign experts, with a wide range of international scientific expertise at the begin- ning of the programming period, although the procurement process was subjected to a legal challenge that went to the Bulgarian Supreme Court. NIF issues calls for expert evaluators along with every call for proposals. NSF has established ten science councils covering different scientific fields which nominate and appoint external reviewers. NSF currently has a pool of over 400 external experts, only 12 of whom are from abroad. There have also been challenges in using public procurement procedures to develop pools of external experts who can take part in evaluation panels. Instead, many experts are appointed directly by the im- plementing body and contracted through civil contracts. 3.6.2 Areas for Improvement Develop pools of external experts to rely on for evaluations Consider writing project proposals in English to facilitate review by foreign experts where applica- ble (for example, projects related to advanced technologies) Increase remuneration for reviewers to attract and retain higher-quality experts 60 Bulgaria Functional and Governance Analysis 3.7 Administrative Burden on Beneficiaries Public programs are designed and funded to address important system failures and generally target a distinct set of actors (i.e.., the policy instrument beneficiaries, such as researchers, entrepreneurs, private companies, etc.). For an intervention to be effective, the support provided needs to be accessi- ble and non-burdensome on beneficiaries. Programs that have complex and/or lengthy processes and reporting requirements can deter beneficiaries from applying, reducing the overall impact of public investments. Key areas where administrative burden can be problematic are the application process, contracting after a project has been awarded, and reporting during program implementation. Some bureaucratic procedures in public programs are put in place to reduce collusion and illicit be- havior and ensure accountability, especially since public moneys are being disbursed. Nevertheless, excessive use of administrative requirements that are outdated, unnecessary, or irrelevant could also constitute artificial barriers for accessing support programs. Specifically, public support programs should not require greater compliance than existing commercial regulations. Programs targeting pri- vate companies should not ask companies to provide more detailed reports than already required for their commercial operations. Program targeting researchers – who have less experience, time, and re- sources for administrative matters – should require an even lower bureaucratic burden than that asked of companies. 3.7.1 Current Practices in Bulgaria Applications. Application processes are generally considered burdensome by beneficiaries (both for programs targeting researchers and companies), although processes vary in format, depend- ing on the implementing body and specific program. Beneficiaries have raised issues with the complexity and length of applications, as well as the amount of supporting documentation re- quested, which could include financial statements, declarations of clean criminal records, tax lia- bilities, and offers from third parties for assets. The complexity of applications leads some benefi- ciaries to use consultants, as they either do not understand or do not have the time and capacity to complete the applications on their own. For Bulgarian operational programs, applications are done entirely electronically through the UMIS system, and templates and guidelines are provided to applicants through the UMIS portal (see the process flows for opening a grant procedure in Appendix III). While all OP applications are done electronically, there have been challenges. For some program financed by OPIC, companies have complained about the number of supporting documents and the amount of financial infor- mation requested, and many companies use consultants to help them through the application process. For OP SESG, about one fourth of the applications for the Centres of Excellence program were submitted after the deadline for submission, which indicates that targeted beneficiaries had difficulties with the application process. Nationally-funded instruments do not use the online UMIS portal, and their application processes 61 Bulgaria Functional and Governance Analysis tend to be more burdensome. For example, NIF requires applicants to provide a detailed break- down of every participating researchers’ hours of work for the duration of the project. Beneficia- ries of NSF must provide financial justification of requested funds by types of expenses for each stage of the project as part of their application. The burden of applying for public programs extends to COVID relief programs targeting Bulgarian firms. The World Bank’s 2020 Business PULSE survey showed that 16 percent of surveyed busi- nesses that did not receive public COVID relief found it too difficult to apply for funding. There is also no centralized digital hub for all COVID response policies; instead, individuals and companies must consult individual programs offered by different ministries to find the relevant information. An example of good practice in public support portals for COVID relief is New Zealand’s COVID 19 information portal, which integrated both public health alerts and financial support programs targeting firms and employees and makes all information about recipients of financial support public10. The process of contracting after a project has been selected and awarded is generally simple, although it can take time – weeks to months depending on the project. For the operational pro- grams, contracting is largely done through the online UMIS portal and the process is not consid- ered lengthy by beneficiaries. For nationally-financed programs (such as grant schemes of the NSF or NIF), contracting is also generally considered a simple process. Reporting during implementation. Reporting during project implementation is not considered overly burdensome. For many programs, beneficiaries only report at the mid-term and end of their projects. As discussed in the M&E section, beneficiary reporting is largely focused on compliance with administrative regulations rather than the performance of the beneficiaries’ projects. For programs funded by the OPs, beneficiaries provide reports in standard, preloaded forms through the UMIS system. Mid-term and final technical and financial reports are filed when bene- ficiaries submit payment requests. NIF projects are divided into stages, with beneficiaries provid- ing technical and financial reports within one month after each stage is completed. Financial re- ports use standardized forms, but there is no template for the technical reports, which can make it difficult for NIF staff to find the information they need. NSF project beneficiaries submit interim and final technical and financial reports, including certified copies of the supporting documents for the incurred direct eligible costs. Public procurement processes. Public procurement processes were brought up by implemen- tors as a major challenge to beneficiaries for several programs across the portfolio. In Bulgaria, there are two main types of public procurement: Processes covered by the Public Procurement Act, which covers public entities (such as universities and PROs): The Act includes 13 different types of public procurement proce- dures, plus two simplified procedures for small procurements. Application and evaluation processes under the Act are still being digitalized, and the online portal is currently not fully functional. Timelines for each procurement, depending on the type of procedure, may last between two and four months from opening a procurement to concluding a contract, with- 10 See more information of the COVID financial support to business portal see here: https://covid19.govt.nz/business-work-and- money/financial-support/financial-support-for-businesses/ 62 Bulgaria Functional and Governance Analysis out including the evaluation process. If there is a litigation in court regarding the granting of the contract (which is relatively common in Bulgaria), the process may take up to a whole year. Several programs have experienced large delays related to the Public Procurement Act. Notably, the NIF uses a procedure from an outdated version of the Public Procurement Act that is entirely paper based and less transparent than processes covered by the current Act. Private entities (such as companies) use a simplified procurement procedure11 called “pub- lic invitation” and the whole process of applying and evaluation is performed online through UMIS. Timelines are far shorter in comparison with the procedures under Public Procure- ment Act, which leads to faster contracting of the tenders – usually around 10-12 days. National legislation directs that mistakes in procurement procedures may lead to severe sanc- tions on beneficiaries of up to 100 percent of the value of the procurement. This can represent a large financial risk for enterprises, particularly those performing multiple projects. Financial corrections. In addition to the financial corrections related to public procurement, ben- eficiaries may also be subject to financial corrections related to unfulfilled milestones, conflicts of interest, violations of public procurement norms, or other contractual irregularities under the Act on management of funding from the European structural and investment funds. Depending on the severity of the irregularity, financial corrections of up to five percent of the contract can be imposed upon beneficiaries. These corrections can be challenged and lead to litigation be- tween the beneficiaries and implementing bodies. These appeals can be costly for beneficiaries because the attorney fees are based on the financial interest at stake. The risk of financial correc- tions and corresponding high cost of appeals can act as deterrents for small companies to apply for government grants. 3.7.2 Areas for improvement Digitize all administrative processes, from application to contracting to monitoring and reporting National programs would greatly benefit from the development of an online portal similar to the UMIS system where beneficiaries can submit and receive information throughout the application and reporting processes Ease administrative burden on beneficiaries by acquiring documentation ex officio where pos- sible, verify R&D expenditures ex post rather than ex ante, and defer some documents from ap- plication to contracting phase. This approach has been adopted by NSF and some OP funded instruments and should be adopted more widely across the STI portfolio. Harmonize procurement processes, where possible 11 Defined in the Act on management of funding from the European structural and investment funds and Decree № 160 of 1 July 2016 63 Bulgaria Functional and Governance Analysis 3.8 Human Resources and Implementation Capacity Managing innovation programs requires significant managerial and analytical capabilities on the part of implementing staff, along with the flexibility and autonomy necessary to fine-tune policies to meet changing demand and conditions. An implementing body’s performance depends on whether it has capable staff with the experience and skills to work across disciplines and sectors and with a variety of external stakeholders, including entrepreneurs, researchers, investors, industry representatives, and technologists. Human resource management capabilities are critical for recruiting these professionals and for mentoring and continuously investing in their training once hired (Aridi and Kapil 2019). Human resource management is therefore a critical component to STI program success. Implementing bodies must build the HR capacity to recruit, hire, and cultivate capable staff. HR management should address management structure, role definition, autonomy, training, and incentives at both the institu- tion level and the program level. Importantly, the primary goal management structures should be to support staff to make decisions based on their professional judgment. Training and incentives for staff should be relevant to the specific program(s) they work on. The performance management literature also emphasizes the importance of process monitoring with quality indicators, information, and reporting for decision-making and improvements. 3.8.1 Current Practices in Bulgaria STI bodies across the government lack adequate staff to properly implement their program portfolios, but staff shortages are a much larger challenge for nationally-funded instruments, as shown in Table 4. MoES, NSF, SMEPA, and ME all reported challenges in implementing their portfolios due to lack of staff. Table 4: Staff sizes of key STI implementing Institution Funding source Number of personnel bodies in Bulgaria EA OPSESG OP 109 DG OPIC, ME OP 222 NSF National 12 Science Directorate, MoES National 15 SMEPA National 52 Economic Policy Directorate, ME National 19 While they have more staff at their disposal, the OPs have also faced challenges related to capacity. Lack of staff is likely the reason why the majority of OPIC instruments were designed to only issue a sin- gle call for proposals, as DG OPIC does not have the capacity to manage multiple calls at the same time. The hiring of full-time staff is governed by the Civil Service Act. Institutions are budgeted for a certain number of full-time positions, and adjustments to the agency budget must be made in order to add 64 Bulgaria Functional and Governance Analysis additional FTE positions, which is typically done at the beginning of the fiscal year. Institutions can also supplement their full-time staff with consultants and external experts (part-time positions) through the use of civil contracts, but most STI bodies lack funding for such positions. Performance incentives are rarely tied to program performance; rather, most STI program staff operate under a standard civil service performance evaluation scheme that is not tailored to the performance of the instrument or their role in implementation. Training opportunities vary depending on the institution and the specific program. Staff from national- ly-funded programs nominally have access to generic public administration training, but in reality have neither the time nor the budget to participate in such training. Staff from the OPs, on the other hand, generally do have access to public administration training, and some OP programs have budget to send program staff to more program specific training offered in the EU12. All STI programs undergo process monitoring in the form of audits. Programs financed by the OPs are subject to both internal and external audits, which can be time consuming for program staff and at times have contradictory findings. 3.8.2 Areas for Improvement Increase program staffing across the STI system, both for full-time and part-time positions, focus- ing on professionals with private sector experience. Reassess staffing needs of key programs transferred to State Agency for R&I. Provide training opportunities relevant to programs STI staff are working on. Establish incentives and evaluation frameworks tied to STI staff’s program duties. 12 For example, staff from EA OPSESG, which administers the Centres of Excellence and Competence programs, participated in 618 training sessions from 2018-2020, including 53 trainings outside of Bulgaria. 65 Bulgaria Functional and Governance Analysis 3.9 Governance and Coordination of STI Policies The governance and coordination of research and innovation policies is important because it can en- sure that gaps in the NIS are addressed and create useful synergies between programs and institutions, to the benefit of both beneficiaries and implementors. However, poor coordination can lead to over- lapping and duplicative policies that waste public resources. Coordination can take the form of formal or informal arrangements between institutions and programs and may involve direct or indirect joint efforts. Governance and coordination extends to the ways in which the STI policy mix addresses the stated overall country research and innovation vision. A coherent STI policy mix is characterized by dynamic and complementary policy portfolios. For example, a policy portfolio that supports innovation in compa- nies may have instruments that target companies at different stages of development, with complemen- tary instruments as companies evolve in capabilities, size, market presence, and so on. Governance and coordination also must account for the regulatory or legal framework in which policies operate. Policy makers must have a complete understanding of the jurisdiction context from the early stages of design to adapt to or leverage existing institutional arrangements for optimal operation of their policies. 3.9.1 Current Practices in Bulgaria The Bulgaria Country Needs and Policy Mix Assessment report (Aridi et al, 2020) found that STI insti- tutions are disconnected from one another and suffer from weak governance structures, resulting in fragmented policies and programs and an uncoordinated national STI agenda. This finding is reinforced by the functional analysis, which finds that while there are there are clear demarcations between the key STI implementing institutions and formal mechanisms in place for inter-institutional coordination, these formal coordination mechanisms are rarely used and have no impact on the functioning of indi- vidual policy instruments. Instead, each institution’s policy portfolio operates independently, without a clear national STI vision to unify efforts across the government. Similarly, there are clear demarcations between the activities of individual instruments, but there are very few examples of instruments with designed synergies to take advantage of other parts of the STI portfolio. In particular, there is a large disconnect between instruments that support research and those that support innovation, with no designed “handoffs” where research projects could graduate to downstream programs to receive support for commercialization activities. Further, there are few linkag- es between the activities of instruments financed by the OPs and those financed by the national gov- ernment; nationally-funded instruments have budgets that are orders of magnitude smaller than those financed by OPs and are largely disregarded by OP staff in their programming. Looking at the legal environment, STI programs have been constrained by external rules and regula- tions, severely in some cases, and implementing staff were not always aware of the various legal and administrative challenges that could hinder instrument effectiveness. Public procurement rules, in par- 66 Bulgaria Functional and Governance Analysis ticular, seem to represent a large challenge on the part of beneficiaries, and procurement processes delayed the implementation of several prominent programs in the current period. STI implementors are largely reactive in adjusting programs to deal with these jurisdictional constraints. The establishment of a new State Agency for Research and Innovation provides an opportunity to ad- dress many of these issues (Box 8). Box 8 Opportu- nities Presented by the New State Agency for R&I In a welcome effort to address many of the governance, implementation, and co- ordination challenges facing its research and innovation system, the Bulgarian Council of Ministers established a new State Agency for Research and Innovation by decree on September 9, 2020. This new R&I Agency represents an excellent opportunity for improving the functionality of the STI policy mix – not only for in- struments directly administered by the new Agency, but also for the national STI programs across the government. The new Agency has the potential to enhance coordination across public and private institutions, improve the professionaliza- tion of the policy workforce, and to build analytical and M&E capabilities that oth- er implementing agencies can use as a resource. It will be important that the new R&I Agency use the significant resources that have been allocated to it to serve as an anchor institution for building profession- al, analytical, and technical capacity for R&I national programs and not just act as an implementer of ESIF-financed programming. Bulgaria’s national institutions outside of OP administrative structures are underfunded and lack capacity to im- plement their current program portfolios. This is not a sustainable arrangement for the long-term health of Bulgaria’s STI system. While there is no single ideal approach for setting up a new innovation agency, for the new State Agency to fully realize its potential, it will need to leverage good practices from global experiences and build on a foundation that provides the au- thority, flexibility, and capacity to carry out its mission. A review of 13 different case studies of innovation agencies from around the world reveals that effective inno- vation agencies are built on seven building blocks (Aridi and Kapil, 2019) shown in Table B2. 67 Bulgaria Functional and Governance Analysis Key element Checklist Clear but adaptable Does the agency’s mission address market, coordination, and institutional mission failures in the NIS? Is the agency able to adapt its mission to address changes to the NIS over time? Effective Does the agency operate with the guidelines and oversight needed to meet governance and public accountability requirements and the autonomy/flexibility needed to work management with industry and adapt to a rapidly changing economy? Does the agency coordinate, rather than compete, with other STI programs? Capable staff Does the agency have the resources to recruit, train, and retain capable staff and supplement internal capabilities with external experts where needed? Can it recruit professionals with private-sector backgrounds and connections? Robust monitoring Does the agency have M&E built into its institutional design, and is M&E used to and evaluation continuously improve program design and performance? capacity Analytical Does the agency have the diagnostic capabilities to understand NIS gaps and capabilities design interventions to address those gaps? Sustainable funding Does the agency have the sustainable and diverse sources of funding needed to maintain its programs and operations over the long term? Strategic Does the agency have partnerships with key stakeholders domestically and partnerships internationally? With knowledge diaspora? With regional networks? and networks Table B2: Seven building While all of these building blocks are critical for the organization’s success, there blocks for an effective innovation agency: a are four that are particularly relevant to the functionality of the Bulgarian STI pol- preliminary check list icies and programs: Effective governance and management. The Bulgarian STI system is current- ly quite fragmented, without a unifying national vision for STI to bring together and coordinate the efforts of the various ministries and agencies engaged in im- plementing research and innovation programs across the government. The new State Agency, with its close connection to the Council of Ministers, appears ide- ally positioned to take a “wide view” of the national STI system, and help devel- op, coordinate, and implement R&I policy instruments in collaboration with other implementing bodies. This will require continuous monitoring of the national STI system to adapt the national policy portfolio to changing conditions. To ensure that the R&I Agency’s “wide view” has an impact on the practical im- plementation of the STI portfolio, the Agency should be represented in the gover- nance bodies of other key implementing ministries and agencies (boards, moni- toring committees, etc.) and vice versa to establish formal channels for high-level coordination and collaboration. Capable Staff. While professional qualifications and experience are needed in a variety of areas, individuals with private sector backgrounds can be especially 68 Bulgaria Functional and Governance Analysis important for innovation agencies because they can facilitate trust building with beneficiaries, through existing knowledge of firms and entrepreneurship dynam- ics, and bring private sector knowhow to the design and execution of STI support programs. This also applies to staff in management positions. As an example, both of Innovate UK’s Chief Executives to date having been recruited from industry rather than the civil service (see Appendix V on innovation agencies). Human resource management should be a major focus for the new State Agency as it begins to build its staff. HR management will be key to its ability to recruit skilled individuals and, once hired, develop them through mentoring and training. The new Agency should look beyond the generic civil service requirement of eval- uation and training of its staff and develop career development plans tailored to fit the responsibilities and programs of its staff. Monitoring & Evaluation. Given the lack of resources and capacity for M&E across the STI system, the new State Agency has an opportunity to build a unit special- izing in M&E for research and innovation programs that can act as a resource for other implementors around the government. There are a number of areas high- lighted in this report where an M&E unit could play a role: knowledge manage- ment services; providing resources and capacity building support for developing theories of change; developing program and project indicators; performing perfor- mance and impact evaluations; and streamlining reporting procedures. This M&E unit can also play a role in ensuring that M&E data is made accessible to other policy makers and to the general public. Analytical Capabilities. Analytical capabilities are another key gap in the Bulgaria STI system. Similar to the M&E gap, the new State Agency has an opportunity to develop and accumulate the analytical capabilities to monitor the national STI system and inform the national STI vision and policy mix to address gaps and sys- tem failures that emerge. This analytical unit can also provide services to other STI agencies. These analytical services could help address several of the challenges highlighted in this report, include the problems with program justifications and mismatches between program budgets and size of target populations. 3.9.2 Areas for Improvement Develop and propagate a clear national vision for research and Innovation that incorporates the efforts of all public and private stakeholders Enable beneficiaries to graduate from one program to another as they develop and make avail- able a menu of instruments (e.g., grants, technical assistance, equity investments, etc.) depend- ing on their needs. 69 Bulgaria Functional and Governance Analysis 4 Recommended Actions for Better Functionality and Governance of the STI Policy Mix This section describes recommendations for improving the functionality and governance of Bulgaria’s STI instrument portfolio. The recommendations are divided into three sections related to different areas for improvement: 1) instrument design; 2) implementation; and 3) governance. These recommendations are based on the finding from this functional and governance analysis and build and expand upon the findings and recommendations from Phase I of this PER STI project, detailed in the Bulgaria Country Needs and STI Policy Mix Assessment (Aridi et al, 2020). The recommendations also build on previous assessments of the Bulgarian research innovation system, such as the EC support facility report in 2015 and subsequent semester country report in 2018, as well as the EC JRC’s recent evaluations of key STI support instruments, the Sofia Tech Park and Centres of Excellence and Competence. 70 Bulgaria Functional and Governance Analysis The coming months present a particularly opportune moment for Bulgaria to improve the functionality of its STI policy programming. The government of Bulgaria is preparing its Recovery and Sustainability Plan to facilitate economic and social recovery in the aftermath of the COVID 19 pandemic, which will allocate approximately €1.25 billion in funding for innovation-related supports. Key STI implementing bodies are preparing for the new programming period, presenting an opportunity to revise and improve upon existing policy supports and scale up functioning instruments. At the same time, a new State Agency for R&I has been launched, which is tasked with the coordination of the implementation of R&I agenda through enhanced institutional, analytical, and M&E support. Any serious effort for improving the functionality of the Bulgarian STI policy mix should be focused first and foremost on upgrading the capacities of nationally-funded institutions outside of the OP admin- istrative structures, be it through increased budget allocation, professionalization of staff, or technical and capacity building. While the OP managing authorities are an integral part of Bulgaria’s support for STI, the country should not neglect the opportunity to leverage European funding and expertise to build the capabilities of its own national institutions and programs. Despite some pervasive challenges, the Bulgaria STI policy mix includes well-performing instruments that offer opportunities for scalability, learning, and upgrading. This section offers practical suggestions for improvements in the functionality and governance of the Bulgarian STI policy mix organized around three main categories of design, implementation, and gover- nance. As detailed in Section 2.1, the cluster analysis found that many of the differences in instrument functionality can be attributed to the implementing body of the instruments. This means that reforms aimed at improving the functionality of instruments should be addressed at the level of the implement- ing bodies, rather than through portfolio-wide reforms. Table 5: Summary of recommendations for better functionality and governance of the Bulgarian STI policy mix Recommendation Type of Reform Stakeholders Design Develop instruments through evidence-based policy design Programmatic State R&I Agency, MoES, MoE Articulate a theory of change (and related Programmatic State R&I Agency, MoES, MoE indicators) for each instrument Design instruments to allow for better Programmatic State R&I Agency, MoES, MoE continuity in policy support Improve M&E practices and build M&E Programmatic State R&I Agency capacity of STI implementing bodies Implementation Develop eligibility and selection Programmatic State R&I Agency, MoES, MoE criteria that maximize impact Improve the quality of project selection Programmatic State R&I Agency, MoES, MoE Reduce administrative burdens on beneficiaries of STI Programmatic State R&I Agency, MoES, MoE instruments to improve accessibility and uptake and Legal Improve the budgets for national institutions and programs Administrative Council of Ministers Improve human resource management Administrative State R&I Agency, MoES, MoE and capacity of STI implementors Governance Improve coordination of the STI agenda Coordination Council of Ministers and Governance Empower the new R&I Agency to monitor Coordination Council of Ministers and coordinate the STI agenda and Governance 71 Bulgaria Functional and Governance Analysis 4.1 Design of STI instruments Develop instruments through evidence-based policy design Priority timeline Short-term 13 Problem Many instruments across the policy portfolio lack a clear justification for intervention. This lack of clear identification of the failure being addressed is a contributing factor to the disconnect between activi- ties and objectives and negatively impacts other areas of instrument design, such as setting program objectives, and developing beneficiary and selection criteria. Many implementing bodies lack the analytical capabilities and resources required to identify, define, and quantify the failure being targeted by each instrument in their portfolio. Approach Regular assessments of the Bulgaria STI system (similar to the national-level diagnosis included in Phase I of this project – the Country Needs and Policy Mix Assessment) should be conducted to identify failures and policy gaps. The new State Agency for R&I appears to be well-positioned to carry out such assessments in its future role. Policy instruments should be evidence-based and designed around an explicitly articulated failure to ensure their relevance and minimize the risk of pet projects. Implementing bodies should be provided with capacity building support and/or resources to build or source the analytical capabilities needed to carry out this identification and quantification of failures. The new State Agency for R&I could develop a centralized analytical unit that can perform such analy- ses on behalf of other implementing bodies. A full range of alternative instrument mechanisms (e.g., grants, investments, non-financial supports, etc.) and designs should be considered to address the identified failure before the design process be- gins. Any instrument designs being copied from other countries should be adapted to accommodate for the country context and the capacity of the implementing body. This design phase should also con- sider the context of already existing instruments and those about to be implemented. Type of reform Programmatic Key Implementing Stakeholder State R&I Agency, MoES, MoE Other Stakeholders EA OPSESG, DG OPIC, other STI implementors, EC DGs14 13 Short-term recommendations relate to national and OP-supported instruments that are currently being designed for the upcoming programming period. 14 Given that EU regulations play a key role in affecting the functionality of instruments on the country level, the European Commission Directorate General (particularly DG REGIO) is considered to be an important stakeholder for defining and implementing the reform agenda 72 Bulgaria Functional and Governance Analysis Articulate a theory of change (and related indicators) for each instrument Priority timeline Short-term Problem Bulgarian STI instruments generally have no or poorly articulated theories of change, which leads to disconnects between program activities and objectives. The lack of theories of change and logic models has contributed poorly defined and connected indica- tors for inputs, activities, outputs, and outcomes, making it difficult to tell if instruments are performing well or contributing to their objectives. Approach A theory of change needs to be fully articulated for each instrument to show the connections between instrument inputs, activities, outputs, and desired outcomes. Each instrument should include a results framework (tied to the theory of change) with a full catalog of input, activity, output, and outcome indicators. Resources and capacity building support should be provided on the design and use of theories of change to STI staff at all key STI stakeholder institutions. The new State Agency for R&I could develop a unit focused on M&E that can provide such capacity building support to other implementing bodies. Type of reform Programmatic Key Implementing Stakeholder State R&I Agency, MoES, MoE Other Stakeholders EA OPSESG, DG OPIC, other STI implementors, EC DGs Design instruments to allow for better continuity in policy support Priority timeline Short-term Problem Many of Bulgaria’s STI support instruments by design only issue a single call for proposals, expending their entire budget for beneficiary support in a single call. Programs that only issue a single call or that issue irregular and sporadic solicitations create uncertain- ty among the beneficiary population, have more limited opportunities for knowledge management and improvement, and may have more difficulties in achieving policy objectives. Approach Unless there is functional reason for an instrument to only issue a single call for proposals (such as for pilot approaches), instruments should issue regular solicitations at annual or semi-annual intervals, rather than expend their entire budgets in a single solicitation. This will allow for more continuity in 73 Bulgaria Functional and Governance Analysis support for STI, more certainty among targeted beneficiaries, and for learning and improvement of the instruments themselves. The shift to supporting instruments with annual or semi-annual solicitations will require building addi- tional organizational capacity on the part of implementing bodies but will allow for continuous improve- ment and efficiency gains of instruments throughout their implementation. Type of reform Programmatic Key Implementing Stakeholder State R&I Agency, MoES, MoE Other Stakeholders EA OPSESG, DG OPIC, other STI implementors, EC DGs Improve M&E practices and build M&E capacity of STI implementing bodies Priority timeline Mid-term Problem At present, much of the monitoring done for STI instruments is focused on whether beneficiaries are in compliance with rules and regulations, rather than for tracking program performance. Only two evaluations have been done of STI instruments in the current programming period, both by the European Commission Joint Research Centre, rather than Bulgarian government bodies. The almost complete lack of program or impact evaluations is due, in part, to a lack of resources for such activities. Implementing bodies reported that they have neither the staff to conduct internal pro- gram evaluations nor the funding to commission external impact or program evaluations on their in- strument portfolios. Approach Programs should design an impact evaluation strategy that includes clear objectives, theories of change and results frameworks, evaluation plan, supporting systems, and protocols. Widespread usage of theories of change and logic models (discussed in Section 3.2) would greatly enhance current M&E frameworks. Performance and impact evaluations should be planned and conducted regularly on key programs, and evaluation results should be used for ongoing learning and improvement, as well as for communicating the results and impacts of public spending to stakeholders and the public. Before starting any evalua- tion, a methodology should be defined including the specific questions to be answered by the evalua- tion, the data that will be used, the models for analysis, and the specific ways in which the findings may be used. Further, evaluations should incorporate clear and transparent protocols for data collection. All instruments should plan for periodic performance evaluations Instruments that are strategically important (e.g., with large budgets, supporting a large number of beneficiaries, with large expected impacts, etc.) should undergo at least one performance eval- uation and one external impact evaluation per programming period. Ideally, these evaluations 74 Bulgaria Functional and Governance Analysis should be conducted by a third parties (i.e., by evaluators outside of the implementing body) that do not have conflicts of interest with the instrument being evaluated. The development the UMIS system should be continued to improve its functionality in generating data and reports on programs and projects. National programs would greatly benefit from the development of an online portal similar to the UMIS system for reporting and tracking of beneficiaries’ activities. Develop protocols that mandate the sharing or publishing of M&E data through the Monitorstat portal or elsewhere to ensure that M&E can be used across government agencies for improving policy design Improvements to M&E practices will require a dramatic ramping up of the M&E capabilities of imple- menting bodies, which currently lack resources and incentives for carrying out M&E. The new State Agency for R&I could develop a unit focused on M&E that can provide capacity building support to other implementing bodies. Type of reform Programmatic Key Implementing Stakeholder State R&I Agency Other Stakeholders MoES, EA OPSESG, MoE, DG OPIC, other STI implementors, EC DGs 75 Bulgaria Functional and Governance Analysis 4.2 Implementation of STI instruments Develop eligibility and selection criteria that maximize impact Priority timeline Short-term Problem While beneficiary criteria are generally consistent with policy objectives, some STI programs exclude private sector actors as eligible direct beneficiaries when there does not appear to be a clear rationale for doing so. Many instruments across the portfolio suffer from overly generic selection criteria that do not target projects with specific desirable characteristics, which can make it difficult for evaluation panels to make consistent project award decisions; this can, in turn, lead to the perception of an unfair or un- transparent awards process. In many cases, selection criteria overemphasize economic viability by favoring projects that are closer to the market, which penalizes higher-risk projects. Approach The rationale for excluding private research organizations as potential beneficiaries for research infra- structure and grant programs (specifically, the NSF portfolio, National Science Programs 2018-2022, Roadmap for Research Infrastructure, and the Centres of Excellence and Competence) should be re- viewed. Selection criteria should maximize the capacity of the instrument to reach the target group for optimal results. For programs that issue annual or semi-annual calls for proposals, selection criteria should be fine- tuned with experience built over successive funding cycles to improve their precision in selecting the right projects. Potential beneficiaries and other stakeholders should be consulted in the development of selection criteria. Selection criteria for instruments focused on innovation in firms should include the return on the in- vestment where applicable, as is currently done for OPIC instruments related to energy and resource efficiency. Type of reform Programmatic Key Implementing Stakeholder State R&I Agency, MoES, MoE Other Stakeholders EA OPSESG, DG OPIC, other STI implementors, EC DGs 76 Bulgaria Functional and Governance Analysis Improve the quality of project selection Priority timeline Medium-term Problem The evaluation of project proposals has been highlighted as an area of concern in past assessments of the STI system, and selection processes are not up to international standards. While most STI institutions use external reviewers to review project applications, the DG OPIC relies heavily on internal panels to evaluate proposals. Internal reviews lack transparency and can be biased towards an institution’s cultural preferences, and for these reasons are not in line with general good practices. STI programs also rely almost exclusively on domestic experts for evaluation panels. This is because project proposals are largely written in Bulgarian and renumeration for evaluators is low. Approach Implementing bodies should be encouraged to develop pools of external experts that can be leveraged to form evaluation panels for project proposals. This is particularly important for OPIC and its successor OP, given the huge range of activities and technologies covered by those programs. Renumeration for expert reviewers could be increased to improve the quality of project awards. For large budget, complex projects and projects in specialized fields, project proposals could be trans- lated to English to allow for technical review by international experts. Type of reform Programmatic Key Implementing Stakeholder State R&I Agency, MoES, MoE Other Stakeholders EA OPSESG, DG OPIC, other STI implementors, EC DGs Reduce administrative burdens on beneficiaries of STI instruments to improve accessibility and uptake Priority timeline Medium-term Problem While beneficiaries of programs financed by the OPs use the online UMIS system for everything from applications to contracting to reporting, the nationally-financed programs use a mix of paper-based and online systems that are generally more burdensome than those for the OPs. Public procurement processes are particularly burdensome on beneficiaries and have caused notable delays in the implementation of several programs. Approach National programs would greatly benefit from the development of an online portal similar to the UMIS system where beneficiaries can submit and receive information throughout the application and report- ing processes. 77 Bulgaria Functional and Governance Analysis When possible, implementing bodies could acquire needed documentation (tax documents, criminal records, etc.) ex officio. This is the current practice for instruments financed by OPSESG and was imple- mented by MoE for recent COVID relief grants15, so the EA OPSESG and MoE could share their experi- ences and learnings on this process other key implementors. R&D project expenditure plans could be approved in advance, reversing the current approach in which each expense item must be reported when incurred. For programs supporting R&D activities, R&D ac- tivities can be presumed to be eligible ex ante and then verified ex post. In such a system, applicants self-report whether they are eligible, and implementors conduct ex post audits to verify eligible costs. The submission requirements of any documents that are not essential to the application process should be deferred until the contracting phase of the project. This will result in a lower administrative burden on beneficiaries and implementors during the application stage. The remaining phases of the digitization of public procurement procedures, such as evaluation and contracting, should be completed. Where possible, public procurement processes should be harmonized across programs and institu- tions. As an example, NIF should update its public procurement processes, as it currently uses process- es from an outdated version of the Public Procurement Act (see the regulatory review in Appendix II). Some consideration should be given to lowering litigation costs related to financial corrections to bet- ter protect beneficiaries’ rights. The state tax for challenging financial corrections in civil court is 0.8 percent of the valuable interest, which can amount to far more than the tax on administrative litigation (€25 for enterprises). The tax applied for financial corrections could be brought in line with the tax on other administrative litigation. Type of reform Programmatic and Legal Key Implementing Stakeholder State R&I Agency, MoES, MoE Other Stakeholders EA OPSESG, DG OPIC, other STI implementors, EC DGs Improve the budgets for national institutions and programs Priority timeline Medium-term Problem Bulgaria’s overreliance on ESIF funding means that national STI instruments and organizations outside of the operational program administrative structures suffer from a severe lack of resources, weakening national institutional capacity outside of the administrative apparatus of the OPs. For nationally funded instruments, budget allocations are determined by the Ministry of Finance, with little or no input from administering bodies. 15 Grant BG16RFOP002-2.073, Support for Micro and Small Enterprises for countering the economic consequence of COVID-19 pandemic 78 Bulgaria Functional and Governance Analysis Nationally financed instruments suffer from inadequate and unpredictable budgets, resulting in a lack of staff and expertise among implementing bodies, insufficient funding for program activities (i.e., solic- itations receiving many more high-quality applications than the program can fund), and in some cases program cancellation. Approach Increase budgetary support to functioning and scalable national programs and organizations to accu- mulate institutional capacity and bridge the divide with OP-funded programs. Two high performing programs identified by this functional analysis – NIF and Technostart – represent opportunities for scaling up of nationally financed instruments. Type of reform Administrative Key Implementing Stakeholder Council of Ministers Other Stakeholders Ministry of Finance; MoES, MoE, R&I Agency, EC DGs Improve human resource management and capacity of STI implementors Priority timeline Medium-term Problem Organizational capacity and human resource management is a pervasive challenge across the STI sys- tem. Nationally-financed programs suffer from severe shortages of staff and expertise, which negatively impacts program implementation Most STI institutions lack funding for part-time support (consultants, experts, etc.) that could supple- ment their full-time staff. Performance incentives are rarely tied to program performance; rather, most STI program staff operate under a standard civil service performance evaluation scheme that is not tailored to the performance of the instrument or their role in implementation. Staff, particularly for nationally-funded instruments, are rarely provided with training opportunities that are relevant to their programs. Approach Staff allocations and needs across the STI system should be reviewed, both for full-time and part-time positions. When hiring/contracting additional FTE staff, implementing bodies focused on innovation and entre- preneurship (firms and startups facing) could leverage professionals with private sector backgrounds and experience. Implementing institutions could be provided with discretionary budgets for consultants, external ex- perts, and other part-time positions to bring in extra manpower and expertise when needed. 79 Bulgaria Functional and Governance Analysis Institutions should consider expanding the opportunities for paid internships by exploiting grant fi- nancing or co-financing available under the OPs or the European Commission. As part of improving HR management, STI staff should be provided with training opportunities that are relevant to the specific programs that they work on. The Institute of Public Administration offers a number of courses relevant to program management and policy analysis, though for more specialized training (e.g., innovation finance, promotion of digital adoption, technology transfer, risk capital invest- ments, etc.) implementors could leverage training opportunities provided by other public and private partners or providers16. STI staff should be provided with incentives and evaluation frameworks that are tied their duties in the programs they work on, rather than using a generic civil service performance evaluation scheme. Type of reform Administrative Key Implementing Stakeholder State R&I Agency, MoES, MoE Other Stakeholders EA OPSESG, DG OPIC, other STI implementors, EC DGs 16 Examples include trainings provided by the European Commission DGs, European Investment Fund, Joint Research Center, OECD, and other private providers 80 Bulgaria Functional and Governance Analysis 4.3 Governance Improve coordination of the STI agenda Priority timeline Medium-term Problem STI institutions are disconnected from one another and suffer from weak governance structures, result- ing in fragmented policies and programs and an uncoordinated national R&I agenda. Inter-institutional coordination mechanisms are in place but rarely used and have no impact on the functioning of individual policy instruments. Instead, each institution’s policy portfolio operates inde- pendently, without a clear national R&I vision to unify efforts across the government. Approach Activate existing coordination channels, including the Council for Smart Growth, Inter-Institutional Working Group, and Regional Partnership Network, to set a commonly agreed upon R&I vision and stra- tegic objectives among national and regional STI actors and coordinate efforts to achieve this vision. Consult with and include innovation system actors (beyond public sector institutions) in the STI articu- lation of the vision, as well as the design of the implementation plans. Representatives of private sector associations, businesses and startups, the research and education community, investor, and ecosys- tem builders should be included in the consultations. Type of reform Coordination and Governance Key Implementing Stakeholder Council of Ministers Other Stakeholders State R&I Agency, MoES, EA OPSESG, MoE, DG OPIC, other STI implementors Empower the new R&I Agency to monitor and coordinate the STI agenda Priority timeline Medium-term Problem STI institutions currently implement programs with a narrow focus on particular areas, often discon- nected from the overall STI policy objectives of improving productivity and competitiveness of the pri- vate sector. Currently there is no single institution that takes a system-wide view of STI needs and policies to ensure that gaps are being addressed and national objectives are targeted by the STI policy mix. 81 Bulgaria Functional and Governance Analysis Approach The new R&I Agency should take on the mission of monitoring and coordinating the implementation of the national R&I agenda through (i) mandating the agency to collect and manage data on the progress of the implementation of STI programs; and (ii) accumulating the analytical and professional expertise needed to fulfill this mission. Establish the administrative mechanisms that ensure representation of the R&I Agency in the gover- nance of key implementing bodies and vice versa (governing boards, steering committees). Technical assistance should be provided to the founding team of the new agency through knowledge sharing, training, and partnerships to ensure that the design, governance, and operations of the organi- zation build on international good practice. Type of reform Coordination and Governance Key Implementing Stakeholder Council of Ministers Other Stakeholders State R&I Agency, EC DGs 82 Bulgaria Functional and Governance Analysis References Aridi, Anwar, and Cirera, Xavier; Frias, Edquist, Charles. 2011. Frias, Jaime; Jin Lee; Haven, Thomas Edward; Natasha Kapil. 2019. Jaime; Hill, Justin; “Design of Innovation and Kyeyoung Shin Marc, Lukasz Marek; Innovation Agencies: Li, Yanchao. 2020. A Policy through (2020) “Innovation Policy Grover, Arti Goswami; Cases from Developing Practitioner’s Guide Diagnostic Analysis: Learning from Korea: Rogers, Juan Douglas; Economies. Washington, to Innovation Policy: Identification of Lessons for Design and Iwanowski, Damian; DC: World Bank Group. Instruments to Build Systemic Problems (or Execution of Innovation Toborowicz, Jerzy Firm Capabilities Failures).” Industrial and Policies in Emerging Jakub; Tan, Shawn Aridi, Anwar; Querejazu, and Accelerate Corporate Change 20: Economies” Washington, Weiming; Tran, Trang Daniel Enrique; Kilinc, Technological Catch-Up 1–29. D.C.: World Bank Group. Thu; Koniewski, Umut; Georgieva, in Developing Countries. Maciej Tomasz. 2020. Teodora Marinova; World Bank, Washington, European Commission Glennie, A. and K. Return on Investment Dimitrova, Lyubomira DC. © World Bank. (2015). “Peer Review of Bound. 2016. Innovation of Public Support to Stoilova. 2020. Bulgaria the Bulgarian Research agencies: International SMEs and Innovation - Country Needs and STI Correa, Paulo. 2014. and Innovation system”, lessons to inspire in Poland - Functional Policy Mix Assessment. Public Expenditure Horizon 2020 Policy and inform national Analysis Technical Note. Washington, D.C.: World Reviews in Science, Support Facility. DG strategies, London: Washington, D.C.: World Bank Group. Technology, and RTD, H2020 Policy NESTA. Bank Group. Innovation: A Guidance Support Facility Aridi, Anwar; Note. World Bank Gorgens, Marelize; Khan, B. Zorina. 2015. Querejazu, Daniel Group, Washington, European Commission Zall Kusek, Jody. 2009. “Inventing Prizes: A Enrique; Zuñiga, Pluvia; DC. World Bank. https:// (2017a). “Specific Making Monitoring and Historical Perspective Georgieva, Teodora openknowledge. support to Bulgaria Evaluation Systems on Innovation Awards Marinova; Dimitrova, worldbank.org/ - Background report Work: A Capacity and Technology Policy.” Lyubomira Stoilova. handle/10986/21064 Horizon 2020 Policy Development Toolkit. Business History Review 2020. Enhancing Support Facility. Note by World Bank. © World 89 (4): 631–60. the Contribution de Jong, Stefan PL, Ruslan Zhechkov and Bank. of Bulgaria’s Public and Reetta Muhonen. Bea Mahieu Technopolis Menon, Saraswathi, Research to Innovation: “Who benefits from ex Group, DG Research and Haapanen, M., Lenihan, Judith Karl, and Kanni A Survey-based ante societal impact Innovation, H2020 Policy H. & Mariani, M. (2014). Wignaraja. “Handbook Diagnostic. Washington, evaluation in the Support Facility. “Government policy on planning, monitoring D.C.: World Bank Group. European funding failure in public support and evaluating for arena? A cross-country European Commission for research and development results.” Camagni, R; Capello, comparison of societal (2018). “Reshaping development.” Policy UNDP Evaluation Office, R. (2013). “Regional impact capacity in the the functional and Studies, vol. 35, 557 – New York, NY (2009). Innovation Patterns and social sciences and operational capacity 575. the EU Regional Policy humanities.” Research of Sofia Tech Park,” Milchevski, Todor; Reform: Toward Smart European Commission Hatry, HP (2006). Mrkonja, Jasmina; Evaluation 29.1 (2020): Innovation Policies,” Directorate-General Performance Marc, Lukasz M; Rogers, 22-33. Growth and Change, vol. for Research and Measurement: Getting Juan; Mestric, Hrvoje; 44, 355 – 389. Blažek, Jirí, and Innovation. An expert Results. Urban Inst Pr; Iwanowski, Damian; Jan Vozáb. “Ex-ante study led by Paris 2nd Edition. Tarade, Ljiljana; Mileska, evaluation in the new Kokorotsikos, Ebba Ivana; Tolic, Martina. member states: The Lund, Liza Peretti, and 2020. Croatia Functional case of the Czech Stephen Taylor. and Governance Republic.” Regional Analysis Technical Note. Studies 40.02 (2006): Washington, D.C.: World 237-248. Bank Group. 83 Bulgaria Functional and Governance Analysis Ministry of Economy. Smismans, Stijn. 2014. Ex-ante “Policy evaluation in Assessment and the EU: The challenges Strategy for the Effective of linking ex ante and Implementation of the ex post appraisal.” Financial Instruments European Journal of Risk from the Operational Regulation 6.1 (2015): Program “Innovation 6-26. and Competitiveness” 2014-2020. World Bank (2013) - BULGARIA Poland National Centre Reimbursable Advisory for Research and Services Program on Development (NCBR). Innovation Input for 2017. Evaluation of Bulgaria’s Research and the SG OP Project Innovation Strategies Selection System - Final for Smart Specialization Report. Prepared by February 2013. IMAPP sp.z o.o.; Idea Rozwoju Foundation; Policy & Action Group; Uniconsult Sp. z o.o. (PAG Uniconsult); and Jagiellonian University Center for Evaluation and Analysis of Public Policies. Rekas Metz, Melissa; Hill, Justin Piers William. 2019. Public Expenditure Review of Small and Medium Enterprise and Competitiveness Programs in Serbia. Washington, D.C.: World Bank Group. 84 Bulgaria Functional and Governance Analysis Appendices 85 Bulgaria Functional and Governance Analysis Appendix I Individual Indicator Scores Program Origin The program origin variable refers to the formal process(es) frequency 20 through which an instrument was created. The origin of a policy instrument must be embedded in processes covered 15 by the rule of law and consistent with the general high-level goals in which it is intended as an intervention. The identi- fication of the problem being addressed and the means to 10 address it must be grounded in actual evidence through a systematic and rigorous appraisal of the issue and the op- tions to reach the goals set to address it. 5 Program origin scores are close to best practices (with a score of 5 representing best practices) for much of the Bul- 0 1 2 3 4 5 garian STI portfolio – particularly for those instruments that score are funded by the OPs. Instruments were either developed Figure 20: Distribution as part of the formal operational programme process or de- of Program Origin Scores veloped in line with one or more key national STI strategies, such as the National Strategy for Development of Research 2020 or the Innovation Strategy for Intelligent Specialization of the Republic of Bulgaria 2014-2020. Program Justification The program justification variable refers to the presence of frequency 20 specific diagnostics of the particular market or system fail- ure that an instrument is intended to address. The justifica- 15 tion should be detailed in program documentation and re- ports and provide an analysis or sensible estimation of the size or intensity of the failure and the impacted population. 10 Justification scores are generally low (with most instruments receiving a score of 2) due to many instruments lacking a di- 5 agnosis or evidence for the need of intervention. In a majority of cases, no specific market or system failure was identified. However, instruments related to research funding tended to 0 1 2 3 4 5 receive higher scores; there, need for public intervention is score clearly stated, but not described in depth. Figure 21: Distribution of Program Justification Scores 86 Bulgaria Functional and Governance Analysis Portfolio Relationship The portfolio relationship variable analyzes how a program frequency 20 operates in relation to all other related and relevant pro- grams – both at the national and European level. The scoring 15 on this indicator includes potential conflicts, complemen- tarities, or overlaps with the rest of the policy mix. A coher- ent STI policy mix is characterized by dynamic and comple- 10 mentary policy portfolios. For example, a policy portfolio that support innovation in companies may have instruments that target companies at different stages of development, with 5 complementary instruments as companies evolve in capa- bilities, size, market presence, and so on. 0 1 2 3 4 5 Scores for portfolio relationship tended to be average, with score a majority of instruments receiving a score of 3. In the case Figure 22: Distribution of OP instruments, the biggest weakness comes from a lack of Portfolio Relationship of consideration of instruments outside the OP portfolio; na- Scores tionally financed instruments have much smaller budgets that instruments funded by the OPs, and for this reason, are largely disregarded by implementors of the OP portfo- lios. Relationships between instruments within the same implementing body usually have well-defined boundaries, al- though there a few synergies or complementarities between instruments. Program Objectives The program objectives variable focuses on the objectives frequency 20 and goals that connect the instrument to desired system-lev- el changes, such as productivity growth, market competi- 15 tiveness, knowledge creation, etc. These objectives should be measurable and achievable, with concrete targets. The overall scoring on program objectives is below average 10 practices, with a majority of instruments receiving a score of 2. This is largely due to the lack of measurability of the stat- 5 ed program objectives for many programs. Another common problem is that many instruments list objectives that are in reality activities. For example, several program list invest- 0 1 2 3 4 5 ments in research infrastructure as an objective, when this score is actually an activity that is a means to achieve a real objec- Figure 23: Distribution tive, such as research excellence. of Program Objective Scores 87 Bulgaria Functional and Governance Analysis Alternative Instrument The alternative instrument variable is closely tied to program frequency 10 justification, in the sense that it evaluates whether differ- 9 ent approaches were considered that can tackle the iden- 8 tified failure that an instrument is intended to address. Cost 7 tradeoffs between alternative instrument designs should be considered. Where approaches based on international 6 examples are being copied, the designs should be adapted 5 with consideration of the local context and capacity of na- 4 tional implementing bodies. 3 The scores for alternative instruments are quite heteroge- 2 neous, though the average score is below 3. Eight of the 28 instruments included did not consider alternative instru- 1 ments at all in their design process. In the case of the OPs, 0 1 2 3 4 5 alternative designs found in other OPs from around Europe score were considered, but other possible models of intervention Figure 24: Distribution of Alternative were not. The instruments with higher scores on this indi- Instruments Scores cator considered alternative approaches going beyond the pre-defined agenda on higher institutional level. Logical Framework (or theory of change) The logical framework variable measures whether programs frequency 18 have an explicit logic model or theory of change that con- 16 nects the inputs, outputs and expected outcomes of the pro- 14 gram with clear and measurable indicators. This framework 12 should be well articulated, and the defined targets should be feasible. Logic models/ToCs should be updated regularly as 10 programs evolve over time. 8 The Logical Framework is one of lowest scoring variables for 6 this analysis, with 36 percent of programs having no logical 4 frameworks of any kind. More than a half of the instruments 2 (60 percent) have some elements of a logic model, but do not have an explicit and fully articulated logic model. Low 0 1 2 3 4 5 score scores are pervasive across the policy portfolio. Figure 25: Distribution of Logical Framework Scores 88 Bulgaria Functional and Governance Analysis Inputs The Inputs variable measures whether program inputs are frequency 16 explicitly defined and consistent with a logical framework. 14 They should cover all resources needed to implement the 12 program, including administrative and operational costs. 10 Costs should be monitored throughout the implementation phase. 8 6 The overall inputs scores low, with an overall score of 2.7. The score varies across instruments as in some cases they were 4 not considered at all and in others (most commonly NSF 2 instruments) they have been cataloged for their most part, 0 1 2 3 4 5 however they have not been systematized. The lack of logi- score cal framework is another obstacle for higher scoring on this Figure 26: Distribution of Inputs Scores indicator. Activities The activities variable measures whether all activities need- frequency 16 ed to achieve the project objectives have been cataloged and 14 are consistent with the logical framework. Activities should 12 be consistent with inputs and outputs (that is, all activities 10 have a purpose and help to reach the desired outputs). 8 The instruments included in the Functional Analysis have 6 average scores for activities. Program activities are general- ly well cataloged, but sometimes lack clear connections to 4 project inputs, outputs, and objectives due to the lack of use 2 of logic models. 0 1 2 3 4 5 score Figure 27: Distribution of Activities Scores Products/Outputs The Products/Outputs variable measures if all program frequency 16 outputs and products have been explicitly identified. They 14 should be consistent with all activities and outcomes in or- 12 der for the desired results to be achieved. All outputs should 10 be operationalized and measurable. 8 The overall outputs scores are around the average. The 6 biggest weakness the instruments face is the lack of con- nection between the project outputs and the impact they 4 should achieve. Higher scores are commonly achieved by OP 2 programs (both OPIC and OPSESG). 0 1 2 3 4 5 score Figure 28: Distribution of Products/Outputs Scores 89 Bulgaria Functional and Governance Analysis Main beneficiaries The main beneficiary’s variable focuses on whether the ben- frequency 16 eficiaries targeted in the project design are consistent with 14 the overall logic of the instrument. Beyond that, they should 12 also be specified in such a way that can maximize the pro- 10 gram efficiency. Criteria for targeting and quantitative mea- sures should be provided in a transparent manner. 8 The majority of selected instruments score around the mean 6 on this indicator. In all cases, beneficiaries well defined, but 4 maybe missing either clarity of their justification or optimal 2 targeting. In the case of the instruments under OPIC, what is generally missing is specific tailoring of the beneficiaries 0 1 2 3 4 5 score needed that goes beyond the generic sector or size criteria. Figure 29: Distribution The best performing instruments on this indicator have ex- of Main beneficiaries plicitly tailored their beneficiaries in order to achieve maxi- Scores mum results. Selection criteria The Selection criteria variable refers to whether selection frequency 16 criteria are consistent with both the objectives and the logi- 14 cal framework of an instrument. The design of the selection 12 criteria should be such that it will lead to the maximum po- 10 tential impact on the targeted population. They should be 8 transparent, simple, and easy to understand. The selected instruments have an overall score gravitating 6 towards the middle of the scale and the results are rather 4 heterogeneous with the majority of instruments having a 2 score of 2. Lower scoring instruments are characterized by 1 2 3 4 5 criteria that lack of connection with the expected outcomes. 0 score Figure 30: Distribution of Selection criteria Scores Audiences The Audiences variable refers to whether all important pro- frequency 16 gram stakeholders can provide input into the instrument 14 design and instrumentation process. There should be formal 12 mechanisms in place for stakeholders to provide input into 10 instrument design and implementation. Instruments should account for non-beneficiary stakeholders and their role for 8 the success of the instrument. 6 Audiences is one of the indicators on which the selected in- 4 struments score the highest with an average score of 4.07. 2 The non-beneficiaries are present in the design of all instru- ments, their role is formalized, and actions for their targeting 0 1 2 3 4 5 score are being taken. Figure 31: Distribution of Audiences Scores 90 Bulgaria Functional and Governance Analysis Expected Outcomes and Impacts The expected outcomes and impact variable refers to wheth- frequency 20 er outcomes and impacts at the program level are well de- fined and connected to desired changes at the system level. 15 They should also be connected to measurable results and assessment indicators. Impact indicators should be inte- grated in the broader policy context on a country or regional 10 level. Criteria should be included for tracking the evolution of outcomes that allow for ending program participation if it becomes clear that program objectives will not be met (as 5 opposed to ad hoc closure at the end of a contract or other extrinsic reason not related to results). 0 1 2 3 4 5 The overall score of the selected instruments for expected score outcomes and impact among the lowest for any variable Figure 32: Distribution observed in the functional analysis, with an average of 1.64. of Expected Outcomes The main problem arises from the lack of measurable and Scores coherent impact indicators. The absence of logical frame- work leads to a lack of connection between outputs and outcomes. Monitoring and Evaluation (Design) Monitoring and evaluation design indicator checks if there is frequency 16 an integrated M&E system embedded in the design phase of 14 the instrument. The indicators chosen should be high quality 12 operationalization of outcomes and impacts. Data collection 10 methods must be realistic and in cases when external data is used, adjustments should be made in order to guarantee 8 that the system would meet the specific needs of the in- 6 strument. If M&E for the instrument relies on external M&E 4 systems, specific adjustments or accommodation of the sys- tem must be made to fit the specific needs and aims of the 2 instrument to avoiding distortion of the instrument logic to 0 1 2 3 4 5 accommodate external bureaucratic requirements that do score not favor its functionality. Figure 33: Distribution of Monitoring and The overall score of the selected instruments is average. Evaluation Scores There are no cases with maximum score and the reason for that is that the best performing instruments on this indicator are missing state of the art key performance indicators that are tightly related to the outcomes and impact. Even in the cases where M&E system exists the weak operationalization of outcomes and impact seems to be the main challenge. 91 Bulgaria Functional and Governance Analysis Learning The learning variable refers to formal learning processes frequency 20 used for systematic improvements of instruments. Learn- 18 ing processes should be used to improve instrument design 16 and implementation procedures, with systematic and formal 14 documentation of the changes being made. 12 The overall scoring on this indicator is average and the ma- 10 jority of instruments have a score of 2. The main reason be- hind it is the lack of systematic approach towards learning. 8 In the majority of cases, there have been some at hoc adjust- 6 ments to programs, but the mechanism for their implemen- 4 tation was not part of the program design. Instruments with higher scores have formalized that process. 2 0 1 2 3 4 5 score Figure 34: Distribution of Learning evidence during implementation Scores Solicitations (Calls for proposals and their Products) The solicitations variable refers to the processes for launch- frequency 20 ing calls for proposals and whether they are reasonable and consistent with the logical framework and objectives. 15 Adjustments on subsequent calls for proposals should be well-documented and made with full justification aligned with the program objectives. Where appropriate, calls should 10 be published regularly and have consistent, predictable cal- endars. 5 The overall scoring for this indicator is average (2) but the re- sults are heterogeneous. The most common problem is that many instruments were only designed to have one call for 0 1 2 3 4 5 the program where there was no functional reason for do- score ing so. This makes adjustments to the instrument over suc- Figure 35: Distribution cessive calls impossible. The best performing instruments of Solicitations Scores have well-documented and systematized approach towards launching calls for proposal. Eligibility Criteria and Application Information The eligibility criteria and application Information variable focuses on the eligibility criteria set to reach the target pop- ulation. They should be clear and transparent and all needed information should be publicly available. Selection informa- tion should be collected and analyzed, including lists of ap- 92 Bulgaria Functional and Governance Analysis plicants, scores awarded to submitted proposals, and other frequency 14 pertinent information related to the submission and selec- 12 tion process. This information should be made available to 10 applicants, as much as general privacy regulations allow. The 8 dissemination of information on eligibility and selection and is consistent with the target population should be appropri- 6 ate. 4 The scoring on eligibility criteria and application Information 2 is average with an overall score of 2.6. One of the reasons 0 1 2 3 4 5 for the lower scores are the burdensome procedures that of- score ten require external consultants in the application process. Figure 36: Distribution of Eligibility Criteria and There are no concerns regarding the availability of informa- Application Information Scores tion to applicants. Application and Selection Processes The application and selection process variable refers to the frequency 16 mechanisms for project selection used by the implementing 14 body, which should be agile, transparent, and responsive. 12 The committees responsible for award decisions should be 10 composed of relevant, independent experts appointed in a justified and transparent manner. The mechanism for ap- 8 pealing award decisions should be accessible and clear. 6 The overall score on this indicator is average and the results 4 are heterogeneous. One of the common problems is related 2 to the lack of use of external experts, as OPIC instruments 0 1 2 3 4 5 rely heavily on internal evaluation panels. A mechanism for score appeals is in place for most of the instruments. Figure 37: Distribution of Application and Selection Process Scores Program database and information on participants and applications The program database and information variable refers to frequency 18 16 the presence of database systems that track participants, projects, follow-ups, outputs, and other data relevant to the 14 program. The system should be used to make adjustments 12 of the solicitations themselves, increase responsiveness to 10 participants’ concerns, and contribute to the general im- 8 provement of management and design of the program, and 6 should be usable by other programs across the portfolio. 4 The overall score on these indicators for the selected instru- 2 ments is low. The majority of instruments lack organized da- tabase systems, and information from these systems is rarely 0 1 2 3 4 5 score stored on the cloud. OPs use the UMIS portal for tracking Figure 38: Distribution of Program database program data, so these programs received slightly higher and information on participants and applications Scores scores than nationally financed instruments. 93 Bulgaria Functional and Governance Analysis Project closures The project closures variable focuses on the presence of frequency 18 beneficiary completion/closing report. The information ob- 16 tained from it can be used for learning and improving the im- 14 pact of the program. 12 The overall scoring on this indicator is good with an average 10 of 3.8. The majority of the programs selected follow the best 8 EU practices on the topic, although low scoring instruments have no completion or closing report in their requirements. 6 4 2 0 1 2 3 4 5 score Figure 39: Distribution of Project closures Scores Budget and financial resources The budget and financial resources variable refers to wheth- frequency 12 er resources are adequate for the implementation of the pro- 10 gram. There should be accountability in the executing of the 8 program and subcontracting entities. 6 The overall results are average, and the distribution is het- erogeneous. One of the reasons for that is the wide variety of 4 budgetary problems that arose throughout the STI portfolio 2 – from instruments with too much allocated budget to those with insufficient financial resources due to lack of connec- 0 1 2 3 4 5 tion between budgets and objectives. Some of the selected score Figure 40: Distribution of Budget and financial instruments were terminated due to insufficient budget. resources Scores Program management and organization quality The program management and organization quality indicator frequency 25 looks at whether the organizational structures in place for 20 appropriate for administering the instruments. The organiza- tional structure should ensure minimization of external and 15 internal pressures in the implementation of the program and should be reviewed for functional adequacy given changing 10 requirements with new policies and instruments. 5 Programs generally received poor scores for program man- agement and organization quality. Many of the programs are 1 2 3 4 5 0 vulnerable to external pressure and lack needed expertise. score Figure 41: Distribution of Program The better performing instruments have clearer definition of management Scores roles and relevant organizational structures. 94 Bulgaria Functional and Governance Analysis Roles and Autonomy The roles and autonomy variable evaluated the level of au- frequency 30 tonomy of implementors to introduce changes and whether 25 clear roles were in place regarding program implementation. 20 The implementing body should have the capacity to intro- duce changes and be capable of resolving conflicts and re- 15 sponding to significant changes in the political or economic 10 environment. 5 The overall results on this indicator is average and the distri- bution is homogeneous with 86 percent of the instruments 0 1 2 3 4 5 having a score of 3. For the majority of cases, there is a lack score of autonomy among implementor, making the introduction Figure 42: Distribution of Roles and Autonomy of changes difficult. Scores Staff and training The staff and training variable evaluates the level of training frequency 16 and experience of the implementing staff and whether the 14 number of staff is adequate to implement the program. 12 The overall scoring on this indicator can be considered as 10 poor with an average score of 1.9. The reason behind it is 8 that often the staff are either overwhelmed and/or training 6 opportunities are scarce and do not go beyond the standard public administration trainings. 4 2 0 1 2 3 4 5 score Figure 43: Distribution of Staff and training Scores Incentives (related to performance with the instrument) The incentives variable refers to the presence of clear and frequency 18 16 explicit criteria for assessing staff performance. Awards and punishments should be linked to clear established criteria 14 and related to the individual’s duties and the performance of 12 the instrument. 10 The overall results on this indicator are average with an av- 8 erage score of 3.3. Many instruments under Operational Pro- 6 grams are performed better than rest as the assessment cri- 4 teria is more relevant to the program performance. 2 0 1 2 3 4 5 score Figure 44: Distribution of Incentives 95 (related to performance with the instrument) Scores Bulgaria Functional and Governance Analysis Process monitoring The process monitoring variable checks for the presence frequency 25 of an audit and monitoring system with clear indicators for 20 monitoring program implementation. This system should be applied periodically, and reports should be submitted and 15 presented to higher authorities. 10 The overall scoring on this indicator is average with the ma- jority of instruments having a score of 2. All of the instru- 5 ments are subject to the standardized audit procedure on national level, although many instruments undergo simulta- 0 1 2 3 4 5 neous audits from different audit authorities, which can con- score tradict in their findings. Figure 45: Distribution of Process monitoring Scores Monitoring & Evaluation (Implementation) M&E in implementation refers to the presence of a formal frequency 16 M&E system that operates continuously and is used to mod- 14 ify programs and generated impact evaluations. The informa- 12 tion should be collected at all indicators levels and the latter 10 should be improved with time. There has to be an impact as- sessment, as well as mechanisms for learning and adapting 8 the program. Programs should be revised based on imple- 6 mentation lessons. 4 The overall scoring is average with heterogeneous distribu- 2 tion. Impact evaluation is missing for the majority of cases. The M&E system looks more like compliance than perfor- 0 1 2 3 4 5 score mance for many of the instruments, and few evaluations of Figure 46: Distribution of M&E Scores any kind are performed. The low-scoring instruments provide nothing beyond beneficiary reports. Programs Relationship The program relationship variable refers to whether a pro- frequency 14 gram acknowledges the existing portfolio of STI programs 12 and has established good coordination and communication 10 with other programs. In cases when overlaps occur, they 8 should be resolved as programs are combined or coordi- nated. Explicit complementarity criteria should exist for the 6 overall effectiveness of related programs. 4 The overall result on this criterion is average with a hetero- 2 geneous distribution. The performance vary as some instru- 0 1 2 3 4 5 ments were terminated after their first year as they dupli- score Figure 47: Distribution of Program cated an already existing program and others acknowledge Relationship Scores 96 Bulgaria Functional and Governance Analysis explicitly existing programs. The lack of systematic coordina- tion is a re-occurring problem. Institutions Relationship The institutional relationship variable refers to the coordi- frequency 14 nation and participative mechanisms with other public and 12 private institutions. Evidence of joined work with other insti- 10 tutions should be in place. 8 The overall result is average but tilted towards the higher 6 side of the scale. In most of the cases, institutional relation- 4 ships have been established, but they are either sporadic or not strategic. 2 1 2 3 4 5 0 Figure 48: Distribution of Institutional score Interaction of Jurisdiction Rules and Relationship Scores Regulations (Internal response) Internal interaction of Jurisdiction Rules and Regulations frequency 18 variable refers to the implementing staff’s knowledge of laws 16 and regulatory constraints. Programs should be adapted 14 based on jurisdictional limitations and program staff should 12 be capable of taking action to leverage positive or mitigate negative factors. 10 8 The overall result on this indicator is average, with lower scores stemming from a lack of proactive action on the part 6 of the implementing staff to make changes. Many of the in- 4 struments intervening in the Research and Development 2 area are having problems with the IP and Technology Trans- fer regulations. 0 1 2 3 4 5 score Figure 49: Distribution of Interaction of Jurisdiction Rules and Regulations (Internal response) Scores Interaction of Jurisdiction Rules and Regulations (Seriousness of external constraint) Interaction of Jurisdiction Rules and Regulation (Serious- frequency 14 ness of external constraint) variable refers to the degree in 12 which the general legislative and regulatory environment 10 inhibits program implementation and effectiveness and the 8 degree in which any regulatory obstacles are modifiable. 6 The overall scoring on this indicator is average. As in the pre- 4 vious case, many of the problems arise from the IP low and the Technology Transfer regulations. In the instruments with 2 staff exchange activities there are issues in terms of the cur- 0 1 2 3 4 5 rent labor regulations. score Figure 50: Distribution of Interaction of 97 Jurisdiction Rules and Regulation (Seriousness of external constraint) Scores Bulgaria Functional and Governance Analysis Appendix II Summary of Legal and Regulatory Environment for STI Policy I. Management and control systems OP funded projects. Every operational program has a monitoring directorate, consisting of finan- cial and technical experts. For every project are assigned at least two experts from the Manag- ing authority – one technical expert and one financial expert. They have dual role – on one side they support the development of the project by answering beneficiaries’ questions and providing guidance in regard to rules compliance. On the other side they perform audits – financial and technical – when a request for payment is submitted by the beneficiary. The experts are obliged to perform at least one audit on site to check if the activities are performed as described and if the assets are bought and delivered (not only on paper). Executive Agency Audit of European Union performs audits on random basis in all operational programs. For every project are assigned at least two experts one technical expert and one financial expert. The experts are auditing not only the beneficiaries’ execution of the project, also the Managing authority actions and conclusions. Public entities performance of procurements are also audited by Public Financial Inspection Agency, which is supervised by the Ministry of Finance. The Agency is not auditing the whole proj- ect, executed by the public entity – only its public procurements under the project. The audits are usually performed by one expert on random basis or after submitting a signal to the institution. Public Financial Inspection Agency is entitled to audit only public entities. National Science Fund. The National Science Fund is ruled by an Executive board which: adopts rules for the admissible expenditures for different grants and the rules for spending and report of the resources. adopts decisions for opening call for proposals, acknowledges the call documentation (Guidelines), including the methodology for evaluation and ranking of projects. approves a list of ranked projects for funding. approves criteria for evaluation and rules for ongoing control. The Fund Manager exercises control over the procedures for monitoring and evaluation of the implementation of funded projects. There are Permanent and Temporary Science-Expert Commissions formed by the Fund. The Tem- porary Scientific-Expert Commissions organize the evaluation and perform the ranking of the research projects. The Permanent Scientific-Expert commissions carry out current monitoring, analysis, and evaluation of the implementation of the financed projects. The financial statements of the projects are checked for eligibility for the funds spent by an auditor – a certified public accountant. As described for the EU funding, public entities performance of procurements are also audited by Public Financial Inspection Agency, which is supervised by the Ministry of Finance. 98 Bulgaria Functional and Governance Analysis National Innovation Fund. For every project are assigned at least two internal experts from NIF – one technical expert (project coordinator) and one financial expert, and also one external expert. The internal experts are verifying the reports (technical and financial) when submitted by the beneficiary and if they comply with the internal rules of NIF (Rules for managing the funds of the National Innovation Fund and Procedure for administrative and financial reporting of projects under the National Innovation Fund). The external expert is usually specialized in the field of the developed innovation and is verifying the progress of the project regarding the planned specific research and development. The project activities are not subject to other type of control, different than the above described. II. Public procurement procedures OP funded projects. For public procurements the beneficiaries should apply whether the Public procurement act if they are public entities/state authorities or the Act on management of funding from the European structural and investment funds and Decree № 160 of 1 July 2016 for defin- ing guidelines for review and evaluation of tenders and contracts in the procedure of selection through public invitation by the beneficiary of grants from the EU structural and investment funds, if they are private entities. Art. 18 of the Public procurement act defines 13 different types of public procurement procedures plus there are two simplified procedures for low threshold procurements. The threshold in the act are aligned with Directive 2014/24/EU of the European Parliament and of the Council of 26 Febru- ary 2014 on public procurement and repealing Directive 2004/18/EC Text with EEA relevance. The timeframes under the Public procurement act depend on the different type of procedure and can be up to 30 days, without including the evaluation process. When considering the evaluation process the time between opening a procurement and concluding a contract may vary between 2 and 4 months. If there is a litigation in court regarding the granting of the contract, this can delay the whole process to almost a whole year. The above described leads to conclusion that the public procurement process is cumbersome and bad planning can lead to project delay or even non-verification of funds. Currently the application process and the evaluation process under the Public procurement act is being digitalized by introducing an online system, however there are already several missed dead- lines for launching the system and currently it is not 100% functional and operational. Other plus of the digital system is that unscrupulous member of the evaluation committee cannot secretly change documents in order to favor current tenderer. Private entities apply Act on management of funding from the European structural and invest- ment funds and Decree № 160 of 1 July 2016 for defining guidelines for review and evaluation of tenders and contracts in the procedure of selection through public invitation by the beneficiary of grants from the EU structural and investment funds where the amount of the awarded grant is greater than 50 % of the total amount of the approved project and the estimated value for: construction, incl. the co-financing by the beneficiary is equal to or higher than BGN 50,000. deliveries or services, incl. the co-financing by the beneficiary is equal to or higher than BGN 30,000 99 Bulgaria Functional and Governance Analysis Private entities use simplified procurement procedure called “public invitation” and the whole process of applying and evaluation is performed online through UMIS. They are obliged to publish the tender online in UMIS and can receive proposals only online through UMIS. Electronically signed documents as well as signed and scanned documents are allowed. The deadlines are far shorter in comparison with the procedures under Public procurement act, which leads to faster awarding of the tenders – usually around 10-12 days. The protocols of the evaluation commission are also uploaded in UMIS for the purpose of monitoring by the MA, however they are not sent to the applicants. The beneficiary uploads the final decision awarding the tender in UMIS and sends it to all applicants via e-mail. There is no appeal procedure established, so the candidates who weren’t awarded can’t challenge the final decision even if they believe there were irregularities in the evaluation process. National Innovation Fund. Private entities, recipient of grants under NIF have to perform a sim- plified procedure (under Rules for managing the funds of the National Innovation Fund and Pro- cedure for administrative and financial reporting of projects under the National Innovation Fund) to select contractors for delivery of assets or services (above certain threshold) by choosing from at least 3 independent offers. These procedures cannot be considered as public procurements. The sanctions for performing such procedure with flaws is not fixed and it is upon discretion of the NIF. The procedure is similar to an outdated procurement procedure under the Public pro- curement act. The procedure is paper-based and not transparent. There are no rules for publicity and the way of acquiring offers is outdated and presupposes possibility of manipulations: written proposals. official catalogs. proposals published on the Internet. proposals published in printed publications. Public or state-owned entities are obliged to apply the Public procurement act for delivery of as- sets or services (above certain threshold). III. Administrative burden on beneficiaries OP funded projects. The application process is performed online. The most common documents that are required are financial statements, declaration for clean criminal record, and declaration for lack of tax liabilities, offers from third parties for assets. The administrative burden can be re- duced by obliging the Managing authority to acquire the information/documents ex officio, which is the current practice of instruments financed by OPSESG but not OPIC or other operational programmes. The Managing authority has the required power to request ex officio clean criminal certificate or certificate of absence of tax liabilities. The financial statements can be acquired also ex officio from the National Revenue Agency (NRA) or the Commercial Register. In contracting procedures, contract recipient has to provide clean criminal certificate and cer- tificate of absence of tax liabilities, which as mentioned before can be acquired ex officio by the contract giver (instruments financed by OPSESG currently acquire such documents ex officio, but this is not the case for other operational programmes). In some cases, the contract recipient (for procurements only) has to provide proof of his experience in similar activities or regarding his 100 Bulgaria Functional and Governance Analysis staff qualification/ experience, if those requirements were evaluated in the tender procedure only through declaration of the tenderer (without providing evidence in the evaluation process). The applicants are reporting during the implementation when they submit payment request. Ex- cluding the advance payment (it does not contain reports) the mid-term payment request and the final payment-request contain technical and financial reports. Depending on the respective indicator they are reported through the technical or the financial report. The reports are filled in standard, preloaded forms through UMIS system. The fields for the re- spective indicators depend on the current call for proposal. There are no specific requirements for reporting. However, it can be made only through UMIS. National Science Fund. The project proposal consist of two parts – administrative description (name of the competition; main scientific field; title and summary; type of the planned research- es, etc.) and scientific description. Every project has also a financial plan and a financial justifica- tion of the requested funds by types of expenses for each of the project stages. After the process of assessment of the project proposals, for those of them having been approved the Manager of the Fund sings a financing contract with a legally prescribed content. The project proposal is an inseparable part of the project. The contract shall be concluded not later than two weeks after the invitation to the beneficiary thereto. The signing may be postponed for no more than one month upon a reasoned request by the head of the research team. The Fund Manager organizes an information meeting with the heads of research teams and accountants responsible for financial services of projects, where financial specialists from the Fund present and give management and reporting instructions. The heads of scientific teams shall submit interim scientific and financial reports on the imple- mentation of the project and a final scientific and financial report on the implementation of the project. The reports are submitted in electronic format and on paper. The scientific report is presented in Bulgarian, and in cases specified in the Guidelines for Applicants or in the contract for project financing - in English. The financial report is presented in Bulgarian and includes the necessary accounting information for assessment of the eligibility of the incurred expenses. Certified copies of the supporting doc- uments for the incurred direct eligible costs shall be attached to the financial report. National Innovation Fund. The application process is different than the one for European funding or National Science Fund. One the website of NIF are published the documents that have to filled in. They consist of several declaration and an application form, which is rather complex because of the detailed distribution of work by hours for the duration of the project for every expert). The most common documents that are required are financial statements and declaration for lack of tax liabilities, declaration for code of economic activity. The administrative burden can be re- duced by obliging NIF to acquire the information ex officio. In contracting procedures, the contract recipient has to provide certificate of absence of tax lia- bilities, which as mentioned before can be acquired ex officio by the contract giver. 101 Bulgaria Functional and Governance Analysis Every project is separated in stages. After completion of each stage the applicants are required within a month to report on the progress by delivering technical and financial reports. Depending on the respective indicator they are reported through the technical or the financial report. The financial reports are filled in standard, paper forms, however the technical report is in free form. After completion of every stage the external expert is performing on site audit. The audit contains report on the progress on the research and development side of the project. Approval from the external experts is mandatory for receiving approval for the stage. IV. Evaluation of applications from solicitations OP funded projects. The Managing authority is empowered to develop the methodology for eval- uation of the project proposals. There two types of criteria – administrative and technical/finan- cial. Some of the administrative criteria are established in Act on management of funding from the European structural and investment funds and are applicable for all grants, but some admin- istrative criteria are developed by the managing authority for the specific grant. All technical/ financial criteria are tailor made by the MA for every specific grant. All criteria are discussed with the monitoring committee before publishing the Guidelines for Applicants. There are no specific obligations for the MA to employ external experts. Some MAs have per- formed procurement procedures for employing external experts with specific expertise. The goal of those tenders is to create a pool of external experts with different profiles, which can be used multiple times in the Programming Period for different calls under one OP. Through such tender, if made at the beginning of the Programming Period, the MA can create a pool of experts, which can be used to supplement the evaluation commissions and the monitoring process later (for Calls, requiring specific expertise). Even though most of the grants are evaluated by evaluation commissions, composed by internal experts, when specific expertise is required, for example for the innovation calls where R&D grant proposals are submitted, external experts are assigned to evaluate strictly specific technical parts of the proposals. There is also an option under Decree № 162 of 5 July 2016 for centralized call for selection of external experts, organized by the Central Coordination Unit in the administration of the Council of Ministers. Such centralized call is still not performed by the Central Coordination Unit in the administration of the Council of Ministers. Such calls have been performed for OPSESG through a Centralized Competition, creating two pools of experts - List of persons approved for the conducted Centralized competitions for selec- tion of external evaluators in the field of research17 (amended with additional experts18) and List of persons approved in the conducted Centralized competition for selection of external evalua- 17 List of persons approved in the conducted Centralized competitions for selection of external Appraisers 18 Candidates approved in the Centralized Competition for selection of external evaluators in the field of research 102 Bulgaria Functional and Governance Analysis tors in the grant award procedures. This List of persons approved for the conducted centralized competitions for selection of external evaluators in the field of research contains Bulgarian and international experts, who can be contracted for evaluation of research procedures. External reviewers are appointed directly by the MA and are contracted though civil contracts. Their payment is based on fixed fee per project. The fee is different if they take part in the admin- istrative evaluation or also in the technical/financial evaluation. National Science Fund. Since the Fund is financing one type of projects (unlike the Operational programs) the criteria by which the applicants are being evaluated are one and the same for years. Some of the criteria are rather abstract and prone to speculations. The Fund is required to contract external experts to evaluation proposals. The experts receive remuneration determined by the Executive Board. The Executive Board approves rules for deter- mining the composition of the scientific-expert commissions, elects their members and accepts the reports for their work. National Innovation Fund. The Fund is empowered to develop the methodology for evaluation of the project proposals. There two types of criteria – administrative and technical. The Fund can change the criteria; however, they are almost the same every year (one session per year). The Fund is obliged according to Rules for managing the funds of the National Innovation Fund to hire external experts for technical and financial evaluation of the received project proposals. The Director of the Fund makes a proposal to the Minister of Economy for approval of a list of independent experts who will perform the technical, economic evaluation of the project propos- als and the financial evaluation of the budget of the project proposals and will participate in the monitoring of the projects. The independent experts are selected by the Secretariat from the list of independent experts approved by the Minister of Economy, according to their professional qualification and in accordance with the subject of the project proposals and within the respec- tive thematic area. The external experts take part in the monitoring of the project, as already described above. The Director determines the procedure for selection of independent experts. The Secretariat has the right to propose to the Director to hire independent experts who have proven to be specialists in scientific fields or centers of knowledge (BAS, universities, institutes, etc.). The experts are contracted through civil contracts. V. Eligible beneficiaries of STI support instruments OP funded projects. There are several regulations regarding the received funds for companies, however two of them are the most important - COMMISSION REGULATION (EU) No 1407/2013 of 18 December 2013 on the application of Articles 107 and 108 of the Treaty on the Functioning of the European Union to de minimis aid and COMMISSION REGULATION (EU) No 651/2014 of 17 103 Bulgaria Functional and Governance Analysis June 2014 declaring certain categories of aid compatible with the internal market in application of Articles 107 and 108 of the Treaty. Another important applicable act is the Framework for State aid for research and development and innovation19. The most common category is de minimis aid, which is applicable for the majority of calls for pro- posals regarding private entities – the calls for start-ups and development and/or implementation of innovations under OPIC. The total amount of de minimis aid granted per Member State to a single undertaking shall not exceed EUR 200 000 over any period of three fiscal years, however for entities performing road freight transport for hire or reward shall not exceed EUR 100 000 and shall not be used for the acquisition of road freight transport vehicles. The initial date of the 3-year period is not the date of receipt of the funds, but the date, when the grant contract has been signed. COMMISSION REGULATION (EU) No 651/2014 of 17 June 2014 specifies 12 categories of aid. From those 12 categories the following are applicable for R&D projects: regional aid. aid for research and development and innovation. Regional aid is applicable for some of the grants under OPIC for development of innovations, im- plementation of innovations, investment in assets. Depending on the current investment Aid for research and development and innovation also can be applicable (also for the above described grants). The described types of aids are applicable for all EU member states and are limiting companies and research centers abilities to receive multiple grants in EU in general, so it cannot be consid- ered as a specific setback for Bulgaria’s system. Under OPSESG there are grant procedures where the beneficiaries are eligible for financing in case they are: “constituting a “research and knowledge dissemination organization” within the meaning of item 1.3. (ee) of the Framework for State aid for research and development and in- novation”. Private entities may apply as long as they are “research and knowledge dissemination organization”. National Science Fund. The Fund finances projects through calls for proposals: A/ Which go beyond the scope of Art. 107 (1) of the Treaty on the Functioning of the European Union under the State Aid Framework for Research, Development and Innovation, and B/ including measures constituting State aid under the terms of the State Aid Framework for Research, Development and Innovation, the terms of Chapter III, Section 4 of Commission Regulation (EU) No 651/2014 of 17 June 2014 on the notification of certain categories of aid compatible with the internal market pursuant to Articles 107 and 108 of the Treaty and Com- mission Regulation (EU) No 1407/2013 of 18 December 2013. on the application of Articles 107 and 108 of the Treaty on the Functioning of the European Union to de minimis aid. 19 https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:52014XC0627(01)&from=GA 104 Bulgaria Functional and Governance Analysis National Innovation Fund. Only companies can apply for grants under NIF. Project proposals are limited to 4 fields, according to ISIS: Informatics and ICT. Mechatronics and clean technologies. Healthy Living Industry and Biotechnology. New technologies in the creative and recreational industries. The received aid is considered “state aid” under COMMISSION REGULATION (EU) No 651/2014 of 17 June 2014 and is subject to its regulations and limitations. VI. Monitoring and evaluation OP Funded Projects. Reporting is based on indicators, which are tailored for every call under the respective operational program. Indicators are described in the call for proposals and the appli- cant knows the target in advance. However, there are some indicators (in most cases economic indicators) which are set by the applicant in the business plan document, which is reviewed and evaluated by the evaluation commission. In those cases, the indicators are being evaluated and points are awarded on their basis. The beneficiary is reporting the results on the indicators when submitting a request for pay- ment, excluding the request for advanced payment. All requests for payment contain technical and financial report. Every technical report should contain information on the indicators and the achieved targets. Some economic indicators cannot be reported during the project because they are targeting fu- ture financial performance of the enterprise. Those indicators track the performance of the en- terprise up to three fiscal years after completion of the project. For example, the indicator “export earnings”, which is a common one under OPIC. The beneficiary will submit one report after com- pleting the project to report the performance of the indicator. Reporting is performed through UMIS by filling out preloaded forms in the reports, which are sim- ilar for all projects. When required documents can be attached in order to provide proof for the performance of the indicator. The MA can also check the performance of the indicator with on-site inspection, if the indicator is in connection with assets. There are several issues with performance of indicators and reporting, some of which will appear in the near future. One of them is that in older grants (2013-2016) the failure to meet the target of the respective indicator was not sanctioned, however the current Ordinance to indicate irregular- ities which constitute reasons for making financial corrections and the percentage of indicators for determining financial corrections under the Act on management of funding from the Europe- an structural and investment funds the failure on indicators is being sanctioned. The issue is that some MAs try to impose financial corrections (under the new Methodology) on older projects, where failure on indicators was not a reason for financial correction. According to the law the sanction rules do not apply retroactively, that is why most of these issues end up in court. 105 Bulgaria Functional and Governance Analysis Other issue is identified for grants for start-ups and their economic indicators. The start-ups pre- dict future economic results in their business plan, which is being evaluated by the commission. They have to report on those indicators 3 fiscal years after project completion. If they fail on the indicators with more than 25% the company will be sanctioned and will have to reimburse the received aid. However, if the start-up is not super successful or if it has set the indicator too high, the financial sanction can bankrupt the company. Every grant procedure has indicators based on procedures level – these indicators has to be ac- complished as a result of all funded projects under the respective grant. The MA is collecting information on the performance of the of the grant indicators ex officio and through analysis of the performance of individual projects. Information for the indicators on Program level is acquired by the MA by analyzing every ongoing or completed grant, as well as obtaining ex officio information from National Statistical Institute for macroeconomic impact. Based on the acquired information the MA is compiling an Annual Implementation Report. The MA is reporting to the European Commission on annual basis the progress of the respective Operational Programme and the performance on the indicators. The official annual reporting is the Annual Implementation Report, submitted through the SFC system of the EU Commission. The results are also discussed unofficially on annual review meetings between the MA and the EU Commission. National Science Fund. Reporting requirements are the same for all projects. The heads of scien- tific teams shall submit within the terms stipulated in the contract: Interim scientific and financial reports on the implementation of the project. Final scientific and financial report on the implementation of the project. National Innovation Fund. Reporting requirement depend on the specific project. Since the proj- ect is separated in stages, every stage shall end by achieving specific result, described in the proj- ect proposal. The completion of the last stage shall lead to completion of the project – the result, aimed by the project is the goal of the last stage. The results of the stages are verified by the independent expert, who performs on site audits. He submits a report to NIF, and it is mandatory for the completion of the stage to be verified by the expert, that the results are achieved. When drafting the project proposal, the applicant it obliged to include indicators for the planned results that are (i) specific, (ii) measurable, (iii) relevant and (iv) traceable. Since the indicators are measurements for the results of the respective project, they are tailor made for every project. VII. Financial corrections EU funded projects. In general the financial corrections are described in Act on management of funding from the European structural and investment funds and in details in Ordinance to indicate irregularities which constitute reasons for making financial corrections and the percentage of indicators for determining financial corrections20 under the Act on management of funding from 20 http://sf.mon.bg/?go=news&p=detail&newsId=833 106 Bulgaria Functional and Governance Analysis the European structural and investment funds. The ordinance is in compliance with EC Decision21 laying down the guidelines for determining financial corrections. The above described acts are public, and all beneficiaries can get acquainted with them. MAs are not empowered to adopt different rules for financial corrections, however some of them, for example the MAs of OPIC and OPHRD, use to include in contracts with beneficiaries penalty clauses for default on beneficiaries obligations, which leads to refund of the grant. This is sort of hidden financial correction, which can be challenged in court and should be declared invalid. For unfulfillment of indicators, conflict of interest, violation of public procurement norms or other severe irregularities the Managing authority is entitled, depending on the severity of the irregu- larity, to impose a financial correction on the beneficiary which can be from 5% up to the whole amount of the tender. The imposed financial correction can be first challenged by the beneficiary in front of the MA and if confirmed by the MA can be challenged in the administrative court. The process of imposing financial corrections is streamlined, however some MAs (like the State Fund Agriculture) are mistakes the process for imposing financial corrections with the process for imposing Act for establishing a state receivable. This confusion leads to a lot of litigation. In case of appeal the litigation is rather expensive for the applicant because the attorney fees are based on the financial interest – usually 3-5% and the state tax according to Art. 27 is 0.8% of the financial interest (but no more than 1 700/4 500 BGN). The high expenditure for a litigation can be an obstruction for small companies to challenge financial corrections in court. Some MAs (like the State Fund Agriculture) are imposing financial correction (in violation of the prohibition on retroactive effect of sanction norms) for non fulfilment of indicators, even when by the time of conclusion of the grant contract non fulfilment of indicators was not sanctioned with financial correction. 21 EC Decision of May 14, 2019 laying down the guidelines for determining financial corrections to be made to expenditure financed by the Union for non - compliance with the applicable rules on public procurement 107 Bulgaria Functional and Governance Analysis Appendix III Process Flows Process flow for drafting and opening an operational programme grant procedure Managing Authority Drafting Consultations Publishing the Evaluation of Opening the call Submission the grant with the draft version the received for proposals of project procedure and Monitoring online for public proposals and proposals by documentation Committee discussion finalization applicants in MAs in of the grant “Programming documentation and negotiation” Directorate Grant procedure should available also in English should be strengthened their proposals also in member from NGOs and be made available also Draft should be made be allowed to submit Applicants should The role of the MC by including more private sector in English English Receipt of At least one week At least 60 days online proposals online for submission of for changes/ proposals amendments by the member of the Monitoring Committee Published Published in Published Published in the in UMIS the respective in UMIS respective OP OP website website Receipt of proposals for Weekly changes/amendments by publishing of NGOs and enterprises Q&A 60 days 108 Bulgaria Functional and Governance Analysis Process flow for project evaluation and contracting of operational programme instruments Managing Authority evaluation committee Assignation Evaluation Evaluation Publish list Contracts Project start of evaluation according to the according to with approved granting of committee administrative the technical proposals, the approved criteria criteria reserves and proposals rejected proposals. Approval by the Director applicant along with the experts in the evaluation their proposals also in foreign high qualified The MA should assign be allowed to submit evaluation should Applicants should administrative act be provided to the All details of the List with rejected proposals committee English Published Published in Published Published in the in UMIS the respective in UMIS respective OP OP website website Can be challenged in Can be challenged in the front of the MA’s Director administrative court by applicants with rejected proposals or included in the reserves The Director returns The Director terminates The court is obliged to solve the proposal for the proceedings for the the case within 2 months, technical evaluation applicant which rarely happens in practice. Can be challenged in the administrative court The ti mef ra me fo re va lu up to 4 atio n sh ould b months e sh or te n ed 109 Bulgaria Functional and Governance Analysis Process flow for project implementation and reporting of operational programme instruments Managing Authority Executive Agency Audit of EU funds Project start Mid-term Final payment Audit by Executive Agency Audit of payment request. request. European Union Funds (optional) Submission of Submission of technical and final technical financial report. and financial reports Audit by the MA on fulfilment of long- term indicators (if applicable) advance payment for the duration guarantee for the amount of the regular audits (at least once per required partial bank guarantee (not for the whole amount). The In case of long-term indicators For advance payment should be Beneficiary must provide bank year) should be conducted Mid-term reporting of indicators (if applicable) of the project. Advance payment Audit of the Final Audit of the Financial correction request (optional) reports by MAs reporting of reports by MAs in case of failure on experts. indicators experts. the indicators. Payment request On-site audit by verification or MAs experts. correction Payment request verification or correction Challenging the financial corrections in between the administrative court 6 and 18 months Lowering the state tax for litigation to a fixed amount of 50,00 BGN shall be considered 110 Bulgaria Functional and Governance Analysis Appendix IV Developing Theories of Change and Indicators The work detailed in this Appendix is a part of a capacity building exercise aimed at improving the M&E framework of STI instruments in Bulgaria. Specifically, this exercise aims to promote the use of Theo- ries of Change at the instrument level in Bulgaria. A ToC is a detailed description of the mechanisms through which a change is expected to occur in an intervention to achieve its objectives. As such, a ToC explicitly depicts the “pathways of change,” connecting inputs (e.g., funding, human resources, and time) to conduct various activities (or interventions), which produce outputs, and together these are expected to generate short-term and longer-term outcomes. The ToC establishes the preconditions, requirements, and assumptions needed for the inputs and activities to be logically linked to the goals. The ToC is not only a useful program design and planning tool, but it is also the essential blueprint for building a rigorous framework for results measurement, particularly by helping develop indicators for identified ToC elements. Four instruments from across the STI portfolio were selected to undergo this exercise: Support for Development of the Centres of Excellence, a program implemented by the EA OPSESG focused on developing research infrastructure and supporting research excellence; Fundamental Research, a program implemented by NSF that provides grants for basic research; Young Scientists and Postdoctorates, a program implemented by MoES that supplements the salaries of young researchers in the public sector; and The National Innovation Fund, a program implemented by SMEPA that provides research and innovation grants to firms. For each program, a ToC diagram was developed by first mapping the information available through various program documents. This mapping exercise enables a systematic assessment of the key causal pathways envisaged by the program in achieving the stated objective(s) with the activities the program finances. Once these pathways are identified, they are then used to create a ToC diagram to depict the linkages between inputs, activities, outputs, and outcomes, add missing elements and assumptions, and harmonizing the indicators across similar programs. After a ToC is developed, a results framework was then created. A results framework aims to identify and define the indicators underlying the theory of change for the program. As such, the results framework identifies indicators for all the ToC elements so that program “success”, in terms of whether the intended change(s) occurred, can be measured and verified. It should be noted that these theories of change represent the authors’ perspective on these four pro- grams, based on interviews with program staff and program documentation. However, implementing staff may have different views on the specifics of the theories of change and indicators detailed here based on their own understanding of the program. The development of a theory of change is a collabo- rative and iterative process, and these reports have been developed to “kick start” this process. 111 Bulgaria Functional and Governance Analysis Support for Development of the Centres of Excellence Program Description22 The main purpose of the operation is to increase the level of market orientation of research activities of the leading scientific organizations in Bulgaria. To overcome the lack of competitive and internationally recognized research complexes meeting the requirements of modern infrastructure and a high level of research in the areas of interest for the Bulgarian economy, OP NSSI will provide support for construc- tion and/or modernization and development of Centres of Excellence (CoEs). These centers will create the best possible conditions for attracting highly qualified researchers to conduct high-level research on a European scale in the priority areas of the Innovation Strategy for Smart Specialization and will sig- nificantly improve the potential for applied research, experimental development and innovation. CoEs will meet the need to build modern research complexes that are targeted at the areas with the greatest potential for increasing the competitiveness of the Bulgarian economy. Scientific capacity develop- ment research and innovation will open up opportunities for new partnerships with business and the creation of new enterprises. Program Objective(s) Support for market-oriented research for increased innovation capacity; increase investment in R&D and innovation and enhance research excellence; and increase investment in advanced research infra- structure and equipment. Activities ○ Construction and equipment activities of the CoEs, which will be used almost exclusively for con- ducting independent R&D T  Build new or significant modernization of existing specialized research infrastructure T  Purchase and modernization of equipment necessary for the implementation of research programs ○ Activities within the CoEs related to implementation of independent R&D for more knowledge and better understanding, including joint R&D, where the research organization or infrastructure participates in effective cooperation subject to the conditions of the Framework T  Conducting research on the most high international level in the priority areas of ISIS T  Introduction of new scientific methods research, training and education methods in the practice of the centers T  Attract leading researchers and top specialists to conduct scientific high-level research in the priority areas of ISIS T  Providing specializations in the leading research centers abroad on highly qualified researchers T  Building strategic partnerships and joint research programs with leading European research centers T  Participation in international and transnational research networks and programs / partnerships that guarantee high level of international visibility and scientific connectivity T  Conducting specializations for the participants researchers, including a high level of exchange and mobility 22 Program descriptions, objectives, and activities were taken from publicly available program documentation and lightly edited for content and readability. 112 Bulgaria Functional and Governance Analysis T Development of sustainable sources of financing and working conditions, including the development of sustainable partnerships with business and conducting joint projects with private investors ○ Activities related to the wide distribution of the results of scientific research, at non-exclusive and non-discriminatory conditions, including through teaching, databases with free access, open publications or open source software, as well as knowledge transfer activities when carried out by the research organization or infrastructure (including its departments or subsidiaries), or jointly with research infrastructure, or on behalf of other such entities, with the express observance of the terms of the Framework T  Dissemination and transfer of scientific results obtained by the participants in the CVP among the academic community and business internationally and national level Theory of Change A theory of change (ToC) is a detailed description of the mechanisms through which a change is expect- ed to occur in a program to achieve the long-term goals. As such, a ToC explicitly depicts the “pathways of change,” connecting inputs (e.g., funding, human resources, and time) to conduct various activities (or interventions), which produce outputs, and together these are expected to generate short-term and longer-term outcomes. The ToC establishes the preconditions, requirements, and assumptions needed for the inputs and activities to be logically linked to the goals. The ToC is not only a useful program de- sign and planning tool, but it is also the essential blueprint for building a rigorous framework for results measurement, particularly by helping develop indicators for identified ToC elements. In this report, the ToC diagram is developed by first mapping the information available through vari- ous program documents. This mapping exercise enables a systematic assessment of the key causal pathways envisaged by the program in achieving the stated objective(s) with the activities the program finances. Once these pathways are identified, they are then used to create a ToC diagram to depict the linkages between inputs, activities, outputs, and outcomes, add missing elements and assumptions, and harmonizing the indicators across similar programs. As illustrated in the ToC diagram (Figure 51), the program is designed with the goals of improving the market-orientation of public research; enhancing research excellence; and increasing investments in research infrastructure. To achieve these goals, the ToC relies on the following pathways of change: Increasing the capacity of public sector researchers to produce excellent research: the pro- gram seeks to a) improve the stock of human capital in the public research sector through hiring and training of researchers at beneficiary organizations, and b) improve the availability of quality research facilities and equipment in the public sector. Together, these outputs will allow public sector researchers will produce higher quality research. Improved collaboration and linkages with the international scientific community: the program seeks to a) increase international collaboration activities, and b) improve the international mo- bility of researchers. These outputs are expected to allow beneficiary organizations to develop stronger linkages to the international scientific community. Improved technology transfer activity from the public to the private sector: the program seeks to a) increase outreach and awareness of knowledge and technology transfer activities among researchers and the private sector, and b) improve public-private research collaboration. The out- puts will contribute to improve technology transfer activity from the public to the private sector. 113 Bulgaria Functional and Governance Analysis Figure 51: Centres of Excellence Theory of Change Diagram input activity output assumption outcome ○ ○Number of grants Hiring new researchers, PhDs, Improved human capital in the public research sector IF the human capital of the public research sector is Improved capacity of researchers to program goal strategic goal awarded and postdoctorates ○○ Number of full-time equivalent (FTE) improved and researchers produce excellent Improved R&D capacity and Increased capacity of the ○ ○Value of Training for researchers involved in project have access to high research implementation research infrastructure Bulgarian public research grants researchers, PhDs, quality facilities, public ○○ Number of scientific of Bulgarian public sector to produce awarded and postdoctorates ○○ Number of researchers receiving training researchers will produce publications published in scientific journals and research organizations research outputs that are ○ ○Number higher quality research indexed in ‘Web of Science’ to enable them to impactful on the economy of grants ○○ Number of researchers perform internationally and society disbursed working in improved competitive research with ○ ○Value of Purchase of new Improved availability of quality research infrastructure economic and societal grants research equipment research facilities and equipment facilities value disbursed Funding the ○○ Value of new or improved research facilities ○○ Number of competitive construction and/or equipment research projects contracted of new research after project completion facilities ○○ Share of scientific publications in the top 10% most cited in ISSS priority areas by beneficiaries Funding Increased collaboration activities IF opportunities Improved participation in ○○ Number of collaborative research projects for international collaboration international supported collaboration and mobility and linkages with projects and are improved, beneficiary the international networks organizations will develop scientific community stronger linkages to the ○○ Number of collaborative international scientific projects contracted after community project completion ○○ Value of collaborative Funding training Improved international mobility of projects contracted after project completion and support for researchers researchers in ○○ Number of full-time equivalent (FTE) international researchers hired from abroad by program mobility beneficiaries opportunities ○○ Number of researchers benefiting from mobility support Organizing and Increased outreach and awareness of IF support is provided Improved technology funding events to knowledge and technology transfer for knowledge and transfer activity from raise awareness activities among researchers and the tech transfer outreach the public to the private of technology private sector and public-private sector transfer ○○ Number of events organized or attended to collaborations, ○○ Number of IPR applications promote knowledge and technology transfer technology transfer filed following project to business sector outcomes from the public implementation to the private sectors will ○○ Number of technology transfers realized following improve project implementation Funding for Improved public-private research ○○ Value of technology transfers participation in collaboration realized following project public-private joint implementation ○○ Number of enterprises collaborating with research projects research organizations 114 Bulgaria Functional and Governance Analysis Results Framework A results framework aims to identify and define the indicators underlying the theory of change for the program. As such, the results framework identifies indicators for all the ToC elements so that program “success”, in terms of whether the intended change(s) occurred, can be measured and verified. In this section, the proposed results framework is built from, and indexed to, the ToC elements, and revised (where necessary) according to the CART principles23. As the theory of change and results framework is developer further, will need to be defined for each indicator, as well as the processes of collecting and verifying the data (including indicator verification period, process, and frequency). Level Indicator Definition Number of Definition Competitive research projects awarded to project beneficiaries as Objective competitive the main applicant (coordinator, manager), or as a partner (associate). Funding projects sources can be national, EU (Horizon 2020, ERC, FP8, ESI Funds), or other (public contracted or private). after project Disaggregation Funding source (national, Horizon 2020, ERC, FP8, ESI Funds, completion other); project role (lead beneficiary, partner) Share of Definition Share of scientific publications in the top ten percent most cited Objective scientific globally in the Web of Science in ISSS priority areas by project beneficiaries. publications Disaggregation Rank of journal where publication was published (first and in the top second-quartile journals, other); by scientific areas; by scientific fields; by S3 10% most thematic priority areas cited in  ut of which: Number of joint publications with industry partners (Publications /O ISSS priority with at least one author from a research organization and one author from the areas by industry) beneficiaries  ut of which: Number of joint publications with international coauthors /O ((Publications with at least one author from a Bulgarian organization and one author from an international organization [public or private]) Number of Definition: Scientific papers and reports published, describing original results Outcome scientific and research conducted by beneficiary organizations with the aim of informing publications the scientific community and society as a whole. Scientific papers need to be published indexed in Web of Science platform (core collection), and may include articles, in scientific reviews, proceedings papers, letters and book chapters. journals Disaggregation: Rank of journal where publication was published (first and and indexed second-quartile journals, other); by scientific areas; by scientific fields; by S3 in ‘Web of thematic priority areas Science’ /O ut of which: Number of joint publications with industry partners (Publications with at least one author from a research organization and one author from the industry) /O ut of which: Number of joint publications with international coauthors ((Publications with at least one author from a Bulgarian organization and one author from an international organization [public or private]) 23 See more on the CART principles here: https://www.poverty-action.org/right-fit-evidence/principles 115 Bulgaria Functional and Governance Analysis Level Indicator Definition Number of Definition: The number of researchers that are utilizing the new or upgraded Outcome researchers research equipment for conducting research activities in the scope of the working in supported project. improved Disaggregation: Employment duration (existing researchers, newly employed research researchers); employment origin (researchers employed at the beneficiary/partner infrastructure institutions, external/contracted, and visiting researchers); sector (public, private); facilities seniority (PhD students, post-doctoral researchers, senior researchers, other) /O ut of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Number of Definition: Number of researchers that directly carry out research and Output full-time development activities related to the implementation of the project. Workforce equivalent may be existing or new, employed at the beneficiary and partners, or contracted (FTE) from third parties. Auxiliary staff for R&D activities (jobs that are not directly researchers involved in R&D activities) are not included in this indicator and should not involved in be counted. The measurement unit is “Full-time equivalent”. Engagement of project researchers employed on the research activities supported by the project less implement- than full-time should be converted to the number of FTE employees by dividing the ation researchers’ scheduled hours with hours of the full-time workweek. Disaggregation: Employment duration (existing researchers, newly employed researchers); employment origin (researchers employed at the beneficiary/ partner institutions, external/contracted and visiting researchers); field/academic discipline; gender; source of financing of researchers’ salaries (fully or partially supported through the grant, institutional funds, other funds); seniority (PhD students, post-doctoral researchers, senior researchers, other) /O ut of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Number of Definition: The number of researchers participating in the implementation who Output researchers were provided with training (lectures, workshops, training sessions, etc.), organized receiving or attended during the project implementation period and financed by the training program. “Training” sessions must be a minimum half-day duration (four hours) to be counted. Disaggregation: Field/academic discipline; gender; seniority (PhD students, post- doctoral researchers, senior researchers, other)  ut of which: Number of young researchers (research students, PhD students, /O early stage researchers) receiving capacity-building support  ut of which: Vulnerable groups (minorities, migrants, disabled, other /O disadvantaged persons) Value of new Definition: Cost of purchase or construction of new or improved research and Output or improved development infrastructure (RDI) facilities. Indicator refers to (and is disaggregated RDI facilities to count) the number of (i) newly constructed RDI infrastructure; (ii) improved and/or existing RDI infrastructure within their field of work or to open new research equipment directions; (iii) equipped facilities for RDI (procurement and installation of equipment including laboratory and office furniture and software and IT equipment required for use of research and development equipment). Disaggregation: Type (as listed in the definition) 116 Bulgaria Functional and Governance Analysis Level Indicator Definition Number of Definition: The number of collaborative research projects, involving a Outcome collaborative beneficiary of the supported project and at least one another entity, and projects contracted after the end of implementation of the supported project. The contracted beneficiary organization must be involved as either the main beneficiary or after project partner in the implementation of the new projects contracted. completion Disaggregation: Collaboration novelty (new, existing); type of partner (research organization, enterprise, other); partner origin (domestic, foreign, diaspora); type of research (basic, applied, experimental development); science area; science field; funding source (national, EU, private, other); S3 thematic priority area Value of Definition: The value of collaborative research projects, involving the Outcome collaborative beneficiary of the supported project and at least one another entity, and projects contracted after the end of implementation of the supported project. The contracted beneficiary organization must be involved as either the main beneficiary or after project partner in the implementation of the new projects contracted. completion Disaggregation: Collaboration novelty (new, existing); type of partner (research organization, enterprise, other); partner location (domestic, foreign, diaspora); type of research (basic, applied, experimental development); science area; science field; funding source (national, EU, private, other); S3 thematic priority area Number of Definition: The number of collaborative R&D projects conducted by grantees Output collaborative with the grant awarded under the program. Projects that are counted are research implemented by research organizations, in partnership with other research projects organizations. This indicator will capture only completed projects, which is supported defined as approval of the grantee’s final project implementation report and grantee receiving the final payment. For monitoring purposes, the indicator should track the projects that are contracted, ongoing and that have been completed. Disaggregation: Project status (awarded/ongoing, completed); project results (projects which achieved their objectives, projects with objectives partially achieved, projects which were discontinued due to irregularities and/or other reasons); research field; by S3 thematic priority areas; region; novelty of collaboration (new, existing); ; type of partner (research organization, enterprise, other); partner origin (Bulgarian, foreign) 117 Bulgaria Functional and Governance Analysis Level Indicator Definition Number of Definition: Number of full-time equivalent (FTE) researchers from other Output full-time countries and/or Bulgarian researchers having worked in foreign research equivalent organizations employed at the beneficiary and partners. Auxiliary staff for (FTE) R&D activities (jobs that are not directly involved in R&D activities) are not researchers included in this indicator and should not be counted. The measurement hired from unit is “Full-time equivalent”. Engagement of researchers employed on the abroad by research activities supported by the project less than full-time should be program converted to the number of FTE employees by dividing the researchers’ beneficiaries scheduled hours with hours of the full-time workweek. Disaggregation: Employment duration (existing researchers, newly employed researchers); employment origin (researchers employed at the beneficiary/ partner institutions, external/contracted and visiting researchers); location before hiring; institution before hiring (research organization, enterprise, other); field/academic discipline; gender; source of financing of researchers’ salaries (fully or partially supported through the grant, institutional funds, other funds); seniority (PhD students, post-doctoral researchers, senior researchers, other) /  Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Number of Definition: The number of researchers supported through the project in Output researchers mobility activities. A mobility activity is defined as researchers visiting a benefiting research organization in another country, for duration of at least two weeks, from mobility to conduct research activities as a visiting researcher. The indicator does support not take into count attendance of events such as meetings, workshops and conferences. Disaggregation: Seniority (PhD students, post-doctoral researchers, senior researchers); origin (domestic, foreign); perception of quality of the mobility activity (satisfaction); \ gender /O ut of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Number of IPR Definition: The number of IPR applications (including patents, trademarks, Outcome applications industrial design, etc.) by the project beneficiary and partners involved in filed related to project implementation, related to the research activities conducted in the project imple- scope of the financed project. Applicant should state the expected number mentation of IPR applications in the project application. In the post-implementation reporting phase, applicant should provide a description of a clear link between the conducted research activities and the IPR protection filed. Disaggregation: Type of IPR (patents, industrial design, trademarks); application status (filed, registered/approved); filing location (national, international) 118 Bulgaria Functional and Governance Analysis Level Indicator Definition Number of Definition: Transfers of research results (knowledge and technology) Outcome technology realized due to project implementation, with purpose of their further transfers realized development and/or their use in development and commercialization due to project of new products (goods or services). Project results can be transferred implementation from project beneficiary and/or partners to third parties in the form of signed R&D agreements or contracts and intellectual property (IP) licensing agreements or the transfer can be realized through establish- ment of new enterprises. More specifically, the following are the types of technology transfer models captured (and disaggregated) by the indicator: /N umber of new start-ups/spin-offs/spinouts originating out of sup- ported projects: Number of new enterprises (start-up/spin-off/spin- out) established by project beneficiary and/or project partners as a result of funded project. Start-up is an enterprise less than 3 years old. Spin-off is an enterprise that has been started by a University group, but which has never left the university environment and perhaps ex- ists to offer specialist consultancy services without the intension for any further expansion or full technology transfer. Spin-out is an enter- prise in which the university or Institute has an equity stake. /N umber of licensing agreements signed with the industry: The licens- ing agreement is a contract to be used by enterprises for technology transfer through granting rights of industrial ownership (license of patents and trademarks). /N umber of other technology transfer agreements signed with the industry: Any other forms of technology transfer/commercialization agreements signed with the enterprises. /N umber of contract research agreements with the industry. Disaggregation: Type of technology transfer activity (as listed in the definition) Number of Definition: The number of collaborative R&D projects conducted by Output collaborative grantees with the grant awarded under the program. Projects that are research projects counted are implemented by research organizations, in partnership supported with other research organizations. This indicator will capture only completed projects, which is defined as approval of the grantee’s final project implementation report and grantee receiving the final payment. For monitoring purposes, the indicator should track the projects that are contracted, ongoing and that have been completed. Disaggregation: Project status (awarded/ongoing, completed); project results (projects which achieved their objectives, projects with objectives partially achieved, projects which were discontinued due to irregularities and/or other reasons); research field; by S3 thematic priority areas; region; novelty of collaboration (new, existing); ; type of partner (research organization, enterprise, other); partner origin (Bulgarian, foreign) 119 Bulgaria Functional and Governance Analysis Level Indicator Definition Value of technology Definition: The value of contractual research conducted (research Outcome transfers realized services acquired by enterprises from research organizations), or due to project knowledge and patents bought or licensed by supported entities from implementation outside sources, under market conditions and for the purposes of implementation of the project supported. Disaggregation: Type of transfer supported (IPR acquisition or licensing, contractual research); for contractual research: type service provided (product and process testing, demonstration activities, professional and technical knowledge for the purpose of product and process development etc.) Number of Definition: The number of events, workshops and conferences Output events organized organized or attended during the project period to enable or attended to dissemination of knowledge or results generated by the research promote knowledge project or transfer of technology to the business sector. The events and technology include those that intentionally support technology and knowledge transfer to business transfer to business sector for potential commercialization of the sector results. Disaggregation: Role (organizer, attendee); transfer type (knowledge transfer, technology transfer). For knowledge transfer by type: symposia, professional fairs, etc.; for technology transfer by type: direct results marketing to businesses for potential acquisition of IPR, or other focused events to link research organizations with business sector, etc. Number of Definition: The number of enterprises collaborating with beneficiary Output enterprises organizations in R&D projects. At least one enterprise and one research collaborating with organization must collaborate in the project. The collaboration may be beneficiaries new or a continuation of existing collaboration and must last at least as long as the project. All enterprises participating in the project as partners are counted as contributing to the indicator. Double counting is avoided, meaning a single enterprise is counted only once regardless of the number of projects it is participating in. Disaggregation: Research field; S3 thematic priority areas; region; novelty of collaboration (new, existing); perception of quality of collaboration (satisfaction) 120 Bulgaria Functional and Governance Analysis Key Takeaways This section provides a summary of findings stemming from our review of program documentation and the ToC mapping exercise: Theory of Change: The programs primary activities – the construction of new research facilities, support for R&D in public research organizations, and dissemination of research – appears to first glance to be very disconnected from one the program’s stated primary objectives, “to increase the market orien- tation of research activities of the leading scientific organizations in Bulgaria”. Only by explicitly defining the key causal pathways of the ToC does it become clear how the program is supposed to contribute to this objective. Two of the stated objectives of the program, “increase investment in advanced research infra- structure and equipment” and “increase investment in R&D and innovation”, are in reality activi- ties that contribute to an actual program objective, research excellence. This mixing of objectives and activities adds to the confusion around how the program is intended to work. Indicators: The current indicators utilized by the program are largely useful and have been used and/or slight- ly modified for the results framework described above. However, there are a number of supported activities that are not covered by defined indicators, such as knowledge and technology transfer activities, researcher training and mobility, and research excellence. It is unclear how the results indicator “Public expenditure of 0.03% of GDP for research and de- velopment (GOVERD plus HERD), funded by enterprises” is connected to the program. None of the described activities appear to be linked with increased R&D expenditures by enterprises. This is another example of how the ToC can help clarify the linkages between activities and intended outcomes. Fundamental Research Program Description “Funding under this procedure is only in support of the implementation of non-profit research activities for basic research to acquire new knowledge. The non-profit scientific activity is in compliance with the National Strategy for Development of Scientific Research in the Republic of Bulgaria 2017-2030 and contributes to: Sustainable restoration of the international positions of the country in terms of quantity and qual- ity of internationally visible scientific products. Increasing the quantity and quality of basic research related to issues of regional and national importance. Significant intensification of the connections of science with education, business, state bodies and society as a whole. 121 Bulgaria Functional and Governance Analysis Expanding the participation of the Bulgarian scientific community in the European Research Area and expanding international scientific cooperation.” Program Objective(s) Increasing the quantity and quality of basic research related to issues of regional and national impor- tance; Sustainable restoration of the international positions of the country in terms of quantity and quality of internationally visible scientific products; Expanding the participation of the Bulgarian sci- entific community in the European Research Area and expanding international scientific cooperation; and Significant intensification of the connections of science with education, business, state bodies and society as a whole. Activities ○ Funding of basic research projects in one of ten scientific areas  Theory of Change A theory of change (ToC) is a detailed description of the mechanisms through which a change is expect- ed to occur in a program to achieve the long-term goals. As such, a ToC explicitly depicts the “pathways of change,” connecting inputs (e.g., funding, human resources, and time) to conduct various activities (or interventions), which produce outputs, and together these are expected to generate short-term and longer-term outcomes. The ToC establishes the preconditions, requirements, and assumptions needed for the inputs and activities to be logically linked to the goals. The ToC is not only a useful program de- sign and planning tool, but it is also the essential blueprint for building a rigorous framework for results measurement, particularly by helping develop indicators for identified ToC elements. In this report, the ToC diagram is developed by first mapping the information available through vari- ous program documents. This mapping exercise enables a systematic assessment of the key causal pathways envisaged by the program in achieving the stated objective(s) with the activities the program finances. Once these pathways are identified, they are then used to create a ToC diagram to depict the linkages between inputs, activities, outputs, and outcomes, add missing elements and assumptions, and harmonizing the indicators across similar programs. As illustrated in the ToC diagram (Figure 52), the program is designed with the program-level goal of increased the capacity and reputation of the Bulgarian public research sector to perform high quality basic research. To achieve this goal, the ToC relies on the following pathways of change: Increasing the capacity of beneficiary organizations to produce impactful publications in basic research fields: the program seeks to support beneficiaries in completing fundamental research projects in key scientific areas. As beneficiaries and their research staff complete basic research projects in key areas, they will gain competencies in these areas and be able to produce higher quality basic research outputs. Increased connections to the international scientific community and the private sector in basic research: the program supports research collaborations with the international scientific com- munity and the private sector in basic research. As beneficiaries perform collaborative research projects, they should form lasting connections to their partnering organizations 122 Bulgaria Functional and Governance Analysis input activity output assumption outcome ○Grants ○ for basic research Funding of basic research projects Completed fundamental research projects in key IF beneficiaries and their research staff complete Increased capacity of beneficiary program goal strategic goal projects Dissemination of scientific areas basic research projects in organizations to Increased capacity and Increased capacity of the research results ○○ Number of full-time equivalent key areas, they will gain produce impactful reputation of public Bulgarian public research (FTE) researchers involved in competencies in these areas publications in basic project implementation research sector to sector to produce and be able to produce research fields perform high quality basic research outputs that are ○○ Number of completed higher quality basic ○○ Number of scientific research projects research impactful on the economy research outputs publications published in scientific journals and ○○ Number of project receiving and society indexed in ‘Web of Science’ additional funding after project completion ○○ Number of competitive projects contracted after project completion Funding of research Increased collaboration IF beneficiaries perform Increased connections collaborations with the international collaborative research to the international scientific community projects, they will form scientific community and the private sector in lasting connections and the private sector basic research to their partnering in basic research ○○ Number of collaborative research projects supported organizations ○○ Number of collaborative projects contracted after project completion ○○ Value of collaborative projects contracted after project completion Figure 52: Fundamental Research Theory of Change Diagram 123 Bulgaria Functional and Governance Analysis Results Framework A results framework aims to identify and define the indicators underlying the theory of change for the program. As such, the results framework identifies indicators for all the ToC elements so that program “success”, in terms of whether the intended change(s) occurred, can be measured and verified. In this section, the proposed results framework is built from, and indexed to, the ToC elements, and revised (where necessary) according to the CART principles. As the theory of change and results framework is developer further, will need to be defined for each indicator, as well as the processes of collecting and verifying the data (including indicator verification period, process, and frequency). Level Indicator Definition Number of Definition: Additional funding (grants) for continuation of research Objective project proposals activities received by project beneficiaries, as the main applicant receiving (coordinator, manager), or as a partner (associate). Funding sources can additional funding be national, EU (Horizon 2020, ERC, FP8, ESI Funds), or other (public or after project private). completion Disaggregation: By success (approved, rejected); By funding sources (national, Horizon 2020, ERC, FP8, ESI Funds, other); by project role (lead beneficiary, partner) Number of Definition: Competitive research projects awarded to project beneficiaries Objective competitive as the main applicant (coordinator, manager), or as a partner (associate). projects Funding sources can be national, EU (Horizon 2020, ERC, FP8, ESI Funds), contracted or other (public or private). after project Disaggregation: Funding source (national, Horizon 2020, ERC, FP8, ESI completion Funds, other); project role (lead beneficiary, partner) 124 Bulgaria Functional and Governance Analysis Level Indicator Definition Number of Definition: Scientific papers and reports published, describing original Outcome scientific results and research conducted within this project with the aim of publications informing the scientific community and society as a whole. Scientific published in papers need to be indexed in Web of Science platform (core collection), scientific journals and may include articles, reviews, proceedings papers, letters and book and indexed in chapters. ‘Web of Science’ Disaggregation: Rank of journal where publication was published (first and second-quartile journals, other); by scientific areas; by scientific fields; by S3 thematic priority areas /O ut of which: Number of joint publications with industry partners (Publications with at least one author from a research organization and one author from the industry) /O ut of which: Number of joint publications with international coauthors ((Publications with at least one author from a Bulgarian organization and one author from an international organization [public or private]) Number of full- Definition: Number of researchers that directly carry out research and Output time equivalent development activities related to the implementation of the project. (FTE) researchers Workforce may be existing or new, employed at the beneficiary and involved in project partners, or contracted from third parties. Auxiliary staff for R&D activities implementation (jobs that are not directly involved in R&D activities) are not included in this indicator and should not be counted. The measurement unit is “Full- time equivalent”. Engagement of researchers employed on the research activities supported by the project less than full-time should be converted to the number of FTE employees by dividing the researchers’ scheduled hours with hours of the full-time workweek. Disaggregation: Employment duration (existing researchers, newly employed researchers); employment origin (researchers employed at the beneficiary/partner institutions, external/contracted and visiting researchers); field/academic discipline; gender; seniority (PhD students, post-doctoral researchers, senior researchers, other) /O ut of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Number of Definition: Number of R&D projects conducted by grantees with the grant Output completed awarded under the program. This indicator will capture only completed research projects projects, which is defined as approval of the grantee’s final project implementation report and grantee receiving the final payment. Disaggregation: Research field; S3 thematic priority areas 125 Bulgaria Functional and Governance Analysis Level Indicator Definition Number of Definition: The number of new collaborative research projects, involving Outcome collaborative the beneficiary of the supported project and at least one another entity, projects and contracted after the end of implementation of the supported project. contracted Beneficiary organization is involved as either the main beneficiary or after project partner in the implementation of the new projects contracted. completion Disaggregation: Collaboration novelty (new, existing); type of partner (research organization, enterprise, other); partner location (domestic, foreign, diaspora); type of research (basic, applied, experimental development); science area; science field; funding source (national, EU, private, other); S3 thematic priority area Value of Definition: The value of new collaborative research projects, involving Output collaborative the beneficiary of the supported project and at least one another entity, projects and contracted after the end of implementation of the supported project. contracted Beneficiary organization is involved as either the main beneficiary or after project partner in the implementation of the new projects contracted. completion Disaggregation: Collaboration novelty (new, existing); type of partner (research organization, enterprise, other); partner location (domestic, foreign, diaspora); type of research (basic, applied, experimental development); science area; science field; funding source (national, EU, private, other); S3 thematic priority area. Number of Definition: The number of collaborative R&D projects conducted by Output collaborative grantees with the grant awarded under the program. Projects that are research projects counted are implemented by research organizations, in partnership with supported other research organizations. This indicator will capture only completed projects, which is defined as approval of the grantee’s final project implementation report and grantee receiving the final payment. For monitoring purposes, the indicator should track the projects that are contracted, ongoing and that have been completed. Disaggregation: Project status (awarded/ongoing, completed); project results (projects which achieved their objectives, projects with objectives partially achieved, projects which were discontinued due to irregularities and/or other reasons); research field; by S3 thematic priority areas; region; novelty of collaboration (new, existing); ; type of partner (research organization, enterprise, other); partner origin (Bulgarian, foreign). 126 Bulgaria Functional and Governance Analysis Key Takeaways This section provides a summary of findings stemming from our review of program documentation and the ToC mapping exercise: Theory of Change: The program documents list a large and diverse array of program objectives, including “Increas- ing the quantity and quality of basic research”, “restoration of the international positions of the country in terms of quantity and quality of internationally visible scientific products”, “Expanding the participation of the Bulgarian scientific community in the European Research Area”, and “Sig- nificant intensification of the connections of science with education, business, state bodies and society.” Given that the program activities are solely confined to funding grants for basic research projects, it is difficult to understand how this one activity contributes to all of the listed objec- tives. This requires the explicit definition of the causal linkages shown in the ToC diagram. Indicators The program does not include any indicators related to the capacity of beneficiary organizations or staff (public researchers). Given that one of the primary objectives of the program is to improve the international reputation of Bulgaria’s science sector, the program should emphasize improve- ments in the capacity of beneficiaries to perform excellent research (beyond the outputs of the funded projects). For this reason, the results framework includes indicators related to additional funding received and competitive projects awarded after project completion to show that benefi- ciaries have improve their capacity to attract competitive funding for research. Young Scientists and Postdoctorates Program Description “In fulfillment of one of the objectives of the National Strategy for Development of Scientific Research, namely the attraction and retention of talented young people for research work and the introduction of support measures to make Bulgaria an active participant in the European Research Area, developing capacity not only to produce but also to attract young scientists to work in the country has developed the current national program. The program is also in line with the recommendation of the International Evaluation Panel implemented under the Horizon 2020 Policy Support Instrument (PSF), which refers to the promotion of young sci- entists (persons engaged in research and scientific-educational activities in higher education). schools and / or scientific organizations after acquiring the first educational qualification degree “”master””, but not more than 10 years after its acquisition) and postdoctoral students (scientists who have acquired educational and scientific degree “”doctor””, but not more than 5 years after the acquisition) j) and sup- porting their career development.” Program Objective(s) To create of a new generation of highly qualified specialists engaged in quality research, responsible to society, and with results that support the implementation of ISIS. 127 Bulgaria Functional and Governance Analysis Activities ○  Funding for young scientists T Funding for the salaries of newly appointed young scientists (no lower than BGN 900) T Funding to supplement the salaries of already employed young scientists (between BGN 200 and 500) ○  Funding for postdoctorates T Funding for newly appointed postdoctorates (between BGN 1,500 and 3,000) T Funding for already employed postdoctorates (between BGN 1,500 and 3,000) Theory of Change A theory of change (ToC) is a detailed description of the mechanisms through which a change is expect- ed to occur in a program to achieve the long-term goals. As such, a ToC explicitly depicts the “pathways of change,” connecting inputs (e.g., funding, human resources, and time) to conduct various activities (or interventions), which produce outputs, and together these are expected to generate short-term and longer-term outcomes. The ToC establishes the preconditions, requirements, and assumptions needed for the inputs and activities to be logically linked to the goals. The ToC is not only a useful program de- sign and planning tool, but it is also the essential blueprint for building a rigorous framework for results measurement, particularly by helping develop indicators for identified ToC elements. In this report, the ToC diagram is developed by first mapping the information available through vari- ous program documents. This mapping exercise enables a systematic assessment of the key causal pathways envisaged by the program in achieving the stated objective(s) with the activities the program finances. Once these pathways are identified, they are then used to create a ToC diagram to depict the linkages between inputs, activities, outputs, and outcomes, add missing elements and assumptions, and harmonizing the indicators across similar programs. As illustrated in the ToC diagram (Figure 53), the program is designed with the program-level goal of increasing the pipeline of young Bulgarian public sector researchers capable of performing high qual- ity research, particularly in the priority areas of the Innovation Strategy for Smart Specialization of the Republic of Bulgaria 2014-2020. To achieve this goal, the ToC relies on the following pathway of change: Increasing the number of young scientists and postdoctorates capable of performing high qual- ity research working at beneficiary PROs and HEIs: the program seeks to increase opportunities for public research careers among young Bulgarians, thereby attracting more young people into the public research sector. With experience and training, many of them should be able to contrib- ute high quality research to the research sector. 128 Bulgaria Functional and Governance Analysis Figure 53: Young Scientists and Postdoctorates Theory of Change Diagram input activity output assumption outcome ○ ○Funding salaries for Provision of salaries for young Increased opportunities for public research IF young researchers receive research experience Increased number of young scientists program strategic goal for young researchers and careers among young and training, they will and postdoctorates objective(s) Increased capacity of researchers postdocs Bulgarians become capable of capable of performing public research sector and postdocs ○○ Number of full-time equivalent preforming high-quality high quality research Increased pipeline of to produce impactful ○ ○Funding for (FTE) researchers involved in research for the public working at beneficiary young Bulgarian public project implementation research training research sector PROs and HEIs sector researchers for young ○○ Number of projects with capable of performing researchers participation from funded high quality research and postdocs researchers ○○ No of IPR applications from projects with participation from funded researchers ○○ No of publications from projects with participation from funded researchers 129 Bulgaria Functional and Governance Analysis Results Framework A results framework aims to identify and define the indicators underlying the theory of change for the program. As such, the results framework identifies indicators for all the ToC elements so that program “success”, in terms of whether the intended change(s) occurred, can be measured and verified. In this section, the proposed results framework is built from, and indexed to, the ToC elements, and revised (where necessary) according to the CART principles. As the theory of change and results framework is developer further, will need to be defined for each indicator, as well as the processes of collecting and verifying the data (including indicator verification period, process, and frequency). Level Indicator Definition Number of Definition: Scientific papers and reports published, describing original Objective scientific results and research conducted within this project (and with involvement publications of funded researchers) with the aim of informing the scientific community published in and society as a whole. Scientific papers need to be indexed in Web of scientific journals Science platform (core collection), and may include articles, reviews, and indexed in proceedings papers, letters and book chapters. ‘Web of Science’ Disaggregation: Rank of journal where publication was published (first and from projects with second-quartile journals, other); by scientific areas; by scientific fields; by participation from S3 thematic priority areas funded researchers /O ut of which: Number of joint publications with industry partners (Publications with at least one author from a research organization and one author from the industry) /O ut of which: Number of joint publications with international coauthors ((Publications with at least one author from a Bulgarian organization and one author from an international organization [public or private]) Number of IPR Definition: The number of IPR applications (including patents, trademarks, Objective applications filed industrial design, etc.) by the project beneficiary and partners involved from projects with in project implementation, as part of a research project conducted in participation from the scope of the financed project (in this case, with the involvement of funded researchers funding young researchers and postdoctorates). Applicant should state the expected number of IPR applications in the project application. In the post-implementation reporting phase, applicant should provide a description of a clear link between the conducted research activities and the IPR protection filed. Disaggregation: Type of IPR (patents, industrial design, trademarks); application status (filed, registered/approved); filing location (national, international) Number of projects Definition: Number of research projects with participation from funded Outcome with participation researchers. Funding sources can be national, EU (Horizon 2020, ERC, FP8, from funded ESI Funds), or other (public or private). Funded researchers must perform researchers at least ten percent of the labor for the project to be counted. Disaggregation: Funding sources of projects (national, Horizon 2020, ERC, FP8, ESI Funds, private, other) 130 Bulgaria Functional and Governance Analysis Level Indicator Definition Number of full- Definition: Number of researchers that directly carry out research and Output time equivalent development activities related to the implementation of the project. (FTE) researchers Workforce may be existing or new, employed at the beneficiary and involved in project partners, or contracted from third parties. Auxiliary staff for R&D activities implementation (jobs that are not directly involved in R&D activities) are not included in this indicator and should not be counted. The measurement unit is “Full- time equivalent”. Engagement of researchers employed on the research activities supported by the project less than full-time should be converted to the number of FTE employees by dividing the researchers’ scheduled hours with hours of the full-time workweek. Disaggregation: Employment duration (existing researchers, newly employed researchers); employment origin (researchers employed at the beneficiary/partner institutions, external/contracted and visiting researchers); field/academic discipline; gender; source of financing of researchers’ salaries (fully or partially supported through the grant, institutional funds, other funds); seniority (PhD students, post-doctoral researchers, senior researchers, other) /O ut of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Key Findings This section provides a summary of findings stemming from our review of program documentation and the ToC mapping exercise: Theory of Change: At present, the program’s stated objective, “create of a new generation of highly qualified special- ists engaged in quality research …” is not explicitly linked to any higher-level strategic goals, which makes it difficult to understand how the program contributes achieving national priorities and how it fits into the larger STI policy framework. Training stands out as a missing element from the program’s logic. Training is not listed as an ac- tivity supported by the program (and thus, is not included in the ToC diagram), yet it is generally a key element in global programs that seek to build the STEM workforce. Indicators The current set of program indicators described in the program documentation are useful – most of them have been adopted and modified for the results framework described above. However, they are a mix of activity, output, and outcome indicators, with no framework to link them in log- ical sequence. There is likely an implicit logic behind their use, but without an explicit ToC and logic model to define the causal linkages between them, there may be no common understanding of the program logic among implementing staff and/or beneficiaries. 131 Bulgaria Functional and Governance Analysis National Innovation Fund Program Description The Fund shall encourage the development of innovations in research and development projects by financing innovations that do not reach the market. Program Objective(s) The main goal of the Fund is to promote the research and development activity for increasing the com- petitiveness of the enterprises. Activities Grants (up to €255,623) to support scientific and research and development projects for a period of implementation from 12 to 36 months. Grants should not exceed 50 percent of project costs. Grants for technical feasibility projects (up to €255,623) for a period of implementation up to 1 year. Grants should not exceed 25 percent of project costs. Theory of Change A theory of change (ToC) is a detailed description of the mechanisms through which a change is expect- ed to occur in a program to achieve the long-term goals. As such, a ToC explicitly depicts the “pathways of change,” connecting inputs (e.g., funding, human resources, and time) to conduct various activities (or interventions), which produce outputs, and together these are expected to generate short-term and longer-term outcomes. The ToC establishes the preconditions, requirements, and assumptions needed for the inputs and activities to be logically linked to the goals. The ToC is not only a useful program de- sign and planning tool, but it is also the essential blueprint for building a rigorous framework for results measurement, particularly by helping develop indicators for identified ToC elements. In this report, the ToC diagram is developed by first mapping the information available through vari- ous program documents. This mapping exercise enables a systematic assessment of the key causal pathways envisaged by the program in achieving the stated objective(s) with the activities the program finances. Once these pathways are identified, they are then used to create a ToC diagram to depict the linkages between inputs, activities, outputs, and outcomes, add missing elements and assumptions, and harmonizing the indicators across similar programs. As illustrated in the ToC diagram (Figure 54), the program is designed with the program-level goal of promoting the research and development activity in enterprises for increased competitiveness of the economy. To achieve this goal, the ToC relies on the following pathways of change: Increase in product development: The program aims to support firms to develop new products and services by providing grants that support R&D projects, fund research staff, and provide ac- cess to improved research equipment and facilities. Increased collaboration with public research organizations and participation in technology transfer: the program supports firms to engage in knowledge and technology transfer with public sector research organizations by providing grant funding for collaborative public-private research projects and innovation advisory support services (from public research organizations). 132 Bulgaria Functional and Governance Analysis Figure 54: National Innovation Fund Theory of Change Diagram input activity output assumption outcome ○ ○Number of grants Funding of research projects Increased financial resources for R&D in supported firms IF firms receive financial support for R&D activities Increased product development program goal strategic goal awarded ○○ Value of private investment matching public and equipment, they will ○○ Number of product Bulgarian enterprises Increased international ○ ○Value of support in RDI projects increase innovation innovations introduced introduce new or competitiveness grants ○○ Number of enterprises supported activities and develop new ○○ Number of process significantly improved of Bulgarian awarded products and services to innovations introduced products and services enterprises ○ ○Number introduce to the market ○○ Value of private investments in R&D project with growth and export of grants after project completion potential in the market disbursed ○○ Number of IPR applications ○○ Change in sales revenue after ○ ○Value of Funding personnel Increased personnel working on project completion filed due to project grants costs innovation activities implementation ○○ Change in employment after disbursed ○○ Number of full-time equivalent (FTE) staff project completion involved in project implementation Funding access to Improved availability of quality public research research facilities and equipment infrastructure ○○ Number of contracts to access public Funding tangible research infrastructure assets ○○ Value of new equipment purchased for innovation projects Funding costs of Increased knowledge and IF support is provided Increased collaboration innovation advisory technology transfer activities for knowledge transfer with public research and support services ○○ Number of enterprises that received activities and public- organizations and innovation advisory support private collaborations, participation in technology transfer technology transfer outcomes from the public ○○ Number of technology to the private sectors will transfers realized after project improve implementation ○○ Value of technology transfers realized after project implementation Funding intangible Improved public-private research assets collaboration Funding of ○○ Number of collaborative research projects collaborative supported research projects ○○ Number of enterprises collaborating with public research organizations 133 Bulgaria Functional and Governance Analysis Results Framework A results framework aims to identify and define the indicators underlying the theory of change for the program. As such, the results framework identifies indicators for all the ToC elements so that program “success”, in terms of whether the intended change(s) occurred, can be measured and verified. In this section, the proposed results framework is built from, and indexed to, the ToC elements, and revised (where necessary) according to the CART principles. As the theory of change and results framework is developer further, will need to be defined for each indicator, as well as the processes of collecting and verifying the data (including indicator verification period, process, and frequency). Level Indicator Definition Change in sales Definition: The change in sales revenue of the supported enterprises Objective revenue after after project completion. The indicator is calculated as the difference project completion between the value of sales revenue of the supported enterprises in the year preceding the submission of the project application (baseline value) and the annual value of sales revenue up to five years after project completion (target value), expressed in absolute numbers. Sales revenue is taken as annual gross sales revenue, which should not include any grant support received by the enterprise. Disaggregation: Change in revenue from sales abroad (export) Change in Definition: The change in the gross number of full-time equivalent (FTE) Objective employment after employees of the supported enterprises after project completion. The project completion indicator is calculated as the difference between the number of employees (FTE) in the year preceding the submission of the project application (baseline value) and the number of employees (FTE) up to five years after project completion (target value). Disaggregation: S3 thematic priority area; gender /O ut of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) 134 Bulgaria Functional and Governance Analysis Level Indicator Definition Number of product Definition: The number of new product innovations introduced by Outcome innovations supported entities, during and after project implementation period. A introduced product innovation is the introduction of a good or service that is new or significantly improved with respect to its characteristics or intended uses. This includes significant improvements in technical specifications, components and materials, incorporated software, user friendliness or other functional characteristics. Product innovations can utilize new knowledge or technologies or can be based on new uses or combinations of existing knowledge or technologies. The term ‘product’ is used to cover both goods and services. Product innovations include both the introduction of new goods and services and significant improvements in the functional or user characteristics of existing goods and services (Source: OECD/Eurostat). The number of product innovations is reported cumulatively, up to a data collection point set in the post-implementation period. Disaggregation: S3 thematic priority area; industry; market (domestic, international); type (goods, services); novelty (new, improved) Number of process Definition: The number of new process innovations introduced by Outcome innovations supported entities, during and after project implementation period. introduced A process innovation is the implementation of a new or significantly improved production or delivery method. This includes significant changes in techniques, equipment and/or software. Process innovations can be intended to decrease unit costs of production or delivery, to increase quality, or to produce or deliver new or significantly improved products. Process innovations include new or significantly improved methods for the creation and provision of services. They can involve significant changes in the equipment and software used in services-oriented firms or in the procedures or techniques that are employed to deliver services. Process innovations also cover new or significantly improved techniques, equipment and software in ancillary support activities, such as purchasing, accounting, computing and maintenance (Source: OECD/Eurostat). The number of process innovations is reported cumulatively, up to a data collection point set in the post-implementation period. Disaggregation: S3 thematic priority area; industry; type of process Value of private Definition: the total value of private investment by supported enterprises Outcome investment in to undertake R&D activities after the end of the supported project. This R&D projects after amount reflects the increase in private sector expenditure in R&D after project completion project completion, measured on annual basis up to 5 years after the grant period, excluding future grants awarded to the enterprise. Disaggregation: Investment purpose (cost category); S3 thematic priority area; industry 135 Bulgaria Functional and Governance Analysis Level Indicator Definition Number of IPR Definition: The number of IPR applications (including patents, trademarks, Outcome applications filed industrial design, etc.) by the project beneficiary and partners involved in due to project project implementation, related to the research activities conducted in the implementation scope of the financed project. Applicant should state the expected number of IPR applications in the project application. In the post-implementation reporting phase, applicant should provide a description of a clear link between the conducted research activities and the IPR protection filed. Disaggregation: Type of IPR (patents, industrial design, trademarks); application status (filed, registered/approved); filing location (national, international) Value of private Definition: the total value of private contribution in supported RDI projects, Output investment including non-eligible parts of the project. The amount is an addition to matching public the public funds received from the program. The amount is calculated support in RDI by subtracting the public funds (including the grant value and other projects contributions from public sources, if applicable) from the total project value (including eligible and non-eligible project costs). Disaggregation: S3 thematic area; industry Number of Definition: The number of enterprises supported by grants awarded through Output enterprises the program, in order to develop and launch new or improved products supported (including goods and services) on the market. Double counting is avoided, meaning a single enterprise is counted only once, regardless of the number of projects it is supported through. If an enterprise is supported to develop more than one product, either through a single or more than one projects, it is counted as one. Indicator achievement is subject to project completion. Disaggregation: S3 thematic priority areas; industry; type of product (goods, services); product novelty (new, improved) Number of full- Definition: The number of staff that directly carry out activities related to Output time equivalent the implementation of the project. Workforce may be existing or new. The (FTE) staff measurement unit is “Full-time equivalent”. Engagement of staff employed involved in project on the activities supported by the project less than full-time should be implementation converted to the number of FTE employees by dividing the employees’ scheduled hours with hours of the full-time workweek. Disaggregation: Employment duration (existing, newly employed); gender  ut of which: Number of full-time equivalent (FTE) researchers involved in /O the project (employed in enterprises)  ut of which: Vulnerable groups (minorities, migrants, disabled, other /O disadvantaged persons) 136 Bulgaria Functional and Governance Analysis Level Indicator Definition Number of Definition: The number of contracts between supported enterprises Outputs contracts to and public research organizations to provide access to public research access public infrastructure research Disaggregation: S3 thematic area; industry infrastructure Value of new Definition: The value of innovation equipment or machinery purchased with Outputs equipment the support of the program. The machinery or equipment must be used for purchased for product development or customization of innovative products. innovation Disaggregation: S3 thematic priority area; industry projects Number of Definition: Transfers of research results (knowledge and technology) Outcome technology realized due to project implementation, with purpose of their further transfers realized development and/or their use in development and commercialization of due to project new products (goods or services). Project results can be transferred from implementation project beneficiary and/or partners to third parties in the form of signed R&D agreements or contracts and intellectual property (IP) licensing agreements or the transfer can be realized through establishment of new enterprises. More specifically, the following are the types of technology transfer models captured (and disaggregated) by the indicator: /N umber of new start-ups/spin-offs/spinouts originating out of supported projects: Number of new enterprises (start-up/spin-off/spin-out) established by project beneficiary and/or project partners as a result of funded project. Start-up is an enterprise less than 3 years old. Spin-off is an enterprise that has been started by a University group, but which has never left the university environment and perhaps exists to offer specialist consultancy services without the intension for any further expansion or full technology transfer. Spin-out is an enterprise in which the university or Institute has an equity stake. /N umber of licensing agreements signed with the industry: The licensing agreement is a contract to be used by enterprises for technology transfer through granting rights of industrial ownership (license of patents and trademarks). /N umber of other technology transfer agreements signed with the industry: Any other forms of technology transfer/commercialization agreements signed with the enterprises. /N umber of contract research agreements with the industry. Disaggregation: Type of technology transfer activity (as listed in the definition) 137 Bulgaria Functional and Governance Analysis Level Indicator Definition Value of Definition: The value of contractual research conducted (research services Outcome technology acquired by enterprises from research organizations), or knowledge and transfers realized patents bought or licensed by supported entities from outside sources, due to project under market conditions and for the purposes of implementation of the implementation project supported. Disaggregation: Type of transfer supported (IPR acquisition or licensing, contractual research); for contractual research: type service provided (product and process testing, demonstration activities, professional and technical knowledge for the purpose of product and process development etc.) Number of Definition: the number of enterprises that received innovation advisory Output enterprises support, such as advisory support in acquisition, protection and exploitation that received of intangible assets, application of norms and regulations that cover them, innovation product development, design and testing, market research and analysis, advisory support development of marketing plans, preparation of feasibility studies or similar activities related to product innovation specifically related to the activities conducted through the supported project. Disaggregation: Innovation advisory support purpose Number of Definition: The number of collaborative R&D projects conducted by Output collaborative grantees with the grant awarded under the program. Projects that are research projects counted are implemented by research organizations, in partnership with supported other research organizations. This indicator will capture only completed projects, which is defined as approval of the grantee’s final project implementation report and grantee receiving the final payment. For monitoring purposes, the indicator should track the projects that are contracted, ongoing and that have been completed. Disaggregation: Project status (awarded/ongoing, completed); project results (projects which achieved their objectives, projects with objectives partially achieved, projects which were discontinued due to irregularities and/or other reasons); research field; by S3 thematic priority areas; region; novelty of collaboration (new, existing); ; type of partner (research organization, enterprise, other); partner origin (Bulgarian, foreign) Number of Definition: The number of supported enterprises collaborating with public Output enterprises research organizations in R&D projects. At least one enterprise and one collaborating with research organization must collaborate in the project. The collaboration public research may be new or a continuation of existing collaboration and must last at least organizations as long as the project. All enterprises participating in the project as partners are counted as contributing to the indicator. Double counting is avoided, meaning a single enterprise is counted only once regardless of the number of projects it is participating in. Disaggregation: S3 thematic priority areas; region; novelty of collaboration (new, existing) 138 Bulgaria Functional and Governance Analysis Key Findings This section provides a summary of findings stemming from our review of program documentation and the ToC mapping exercise: Theory of Change: The activities of the program are quite clearly linked to the objectives of the program, indicating a strong implicit logic behind the development of the program. However, the lack of defined indi- cators makes it extremely difficult to consistently judge whether projects are contributing to the program’s objectives Indicators According to program documents, all project indicators are set by applicants in their project pro- posals, and there do not appear to be any other program indicators. Again, this makes it very diffi- cult to understand how well the program contributed to its objectives. Because each project may have entirely different indicators, understanding the performance of NIF’s portfolio of grants does not appear to be possible. While some indicators may need to be project specific, it is important to use at least some indicators that are consistent across all grant projects. 139 Bulgaria Functional and Governance Analysis Appendix V Innovation Agencies Case Studies Recent work done by the World Bank (Aridi and Kapil, 2020) and Nesta (Glennie and Bound, 2016) distil lessons from a diverse collection of innovation agencies in both developed and developing countries, and shows that such agencies can be crucial factors for developing STI support policies and instru- ments in an adequate and effective manner. In some countries, a line ministry fills this role, but the emerging trend is the creation of a dedicated structure (state agency, council, fund, foundation, nation- al centre, etc.) with the mission to conduct and coordinate national STI policy, to enhance the inter- connectivity between different actors of the national innovation system, and to coordinate the efforts in fostering science and innovation as a primary driver of the economic growth, national and company competitiveness. Lessons learned from these reports identify seven building blocks for establishing and operating inno- vation agencies that will likely improve their innovation-related impact: Clear and adaptable mission which to explain the role of the new player and its relationships with other stakeholders, and to help the agency to set clear priorities and objectives and to mobilise the adequate instruments for their implementation and for addressing the market and system failure; Capable staff acquiring a balanced portfolio of knowledge, skills and experience in public administration but also business background; Effective governance and management structure which to ensure adherence of the principles of transparency, impartiality and professionalism through a high level of autonomy; Diagnostic-based interventions as a mean for implementing tailored mechanisms regarding the stakeholder needs and changing business environment; Robust monitoring and evaluation to guarantee the ability of the innovation agency to improve its work following the lessons learnt and good practices, to develop evidence-based policies and instruments; and to capitalise on results achieved and non-successful stories; long-term oriented, predictable and diversified sustainable funding is an important factor for the agency to stand behind its priorities and objectives, and for the stakeholders to trust their support; Strategic partnerships and networks are the way to effectively complement the efforts of different actors of the national innovation system and to leverage the collective pool of knowledge and expertise. In 2020, the Bulgarian government established a new State Agency to complement the Ministry of Sci- ence and Education, Ministry of Economy and respective executive agencies, funds and programmes, and to take over the governance of interrelations between science and business and coordination of the national innovation system. The idea is not new for Bulgaria (the official debated started in 2015) and the recommendations in this direction dated even earlier. The mission of the new State Agency for 140 Bulgaria Functional and Governance Analysis Research and Innovation is ambitious. It will operate with national and European funds and will manage both entirely national and European initiatives including the new Operational Programme Research, Innovation and Digitalisation for Smart Transformation as a main instrument between them. To support the efforts of the new Agency’s management team in developing the mission and objectives, designing the governance structures, and recruiting experts and professionals, this appendix presents two additional case studies of innovation agencies relevant to the new State Agency’s planned role and mission: Innovate UK, the UK’s innovation agency, and Innosuisse, the Swiss Innovation Promotion Agency. Both cases represent European innovation leaders with long-term experience in supporting re- searchers, research organisations and innovative businesses and their projects for transforming newly created scientific and technological knowledge into innovative products and services. What follows are a set of findings from these two cases and how they could inform or relate to the planned activities of the new Bulgarian agency. Key lessons for the new R&I Agency: Mission and objectives: For the R&I Agency, one of the key elements in constituting the new administrative structure is to define its place and role within the national innovation system and to understand which com- position of priorities and objectives would have the highest impact on the innovation potential of the national economy. It is crucial to define a mission which to allow the agency to address the main market and system failures (one of them is the missing link between the science and business), and as a second phase, to ensure administrative capacity (both capable staff and sus- tainable funding) for deliberately pursuing this mission in the long-term. Frequent changes in the mission and objectives would create uncertainty and lack of trust on behalf of the stakeholders and beneficiaries. Accumulation of administrative capacity: The ambitious plans and expectations of the R&I Agency require significant administrative capac- ity in terms of human resources, financing and internal procedures. Talented personnel in ade- quate quantity and quality is a key factor for the successful implementation of the Agency’s future projects. People with expertise in public administration, policy development, business practices and data analyses will be capable to ensure the smooth implementation of the procedures and targeted support to the business. No less important factor is the financing in terms of scope of the budget, diversified sources and financial instruments tailored to the specific needs of the potential beneficiaries. Effective governance and management structure: It is expected the R&I Agency will have relatively high autonomy due to the decision for its status (state agency) and oversight (part of the Council of Ministers’ administration). This is a good po- sition for the Agency to be involved in both policy development and in its further implementation based on a professional expertise. To exploit the full potential of this opportunity, vital manage- ment structure have to established which to be sustainable and flexible at the same time and to guarantee the application of transparent procedures, mixed usage of financial resources with different source, and responsiveness to the changing business needs. 141 Bulgaria Functional and Governance Analysis Strong traditions in knowledge management, learning, and result and efficiency orientation: The changing business environment and transforming economy create challenges and uncer- tainties for both business and government. To continue functioning in good shape and to match the stakeholder’s expectations it is important to apply internal procedures for monitoring and evaluation for ensuring, on the one hand, the Agency improves its performance based on lessons learnt, and, on the other hand, succeed in achieving the desired impact on the innovation system and beneficiaries, in particular. Linkages within the innovation eco-system and with potential beneficiaries: The newly established agency cannot acquire all needed competencies and expertise for imple- menting its mission. Due to that, it is important to establish consulting bodies and to keep a pool of external experts in different fields. Participation in partnerships and networks at national and European level will complement the efforts of different actors and will allow the Agency to capi- talise on the synergy between them. Innovate UK Agency Profile Formation July 2007 as the Technology Strategy Board, April 2018 as Innovate UK Type The UK’s innovation agency Legal status Non-departmental public body Main organ Governing Board Parent organisation UKRI Affiliations European Network of Innovation Agencies Budget c.£400m Website https://www.gov.uk/government/organisations/innovate-uk Mission and Objectives Innovate UK was established with the mission to drive productivity and economic growth by sup- porting businesses to develop and realise the potential of new ideas, including those from the domestic research base. The focus of Innovate UK is on supporting business-led innovation that has a positive, measurable impact on the economy and society. Innovate UK provides a system of financial and non-financial support that stimulates successful innovation, boosts competitive- ness and delivers economic growth. The agency has five main strategic goals24: 1/ Deliver measurable economic and societal impact across the UK. 2/ Support and invest in innovative businesses and entrepreneurs with the potential and ambition  to grow. 24 Innovate UK, UK Research and Innovation, Delivery Plan 2019. 142 Bulgaria Functional and Governance Analysis 3/  aximise the commercial impact of world-class knowledge developed in UK industries and its M research base. 4/ Identify, support and grow transforming and emerging industries through innovation. 5/ uild a coherent, supportive environment incentivising R&D investment and enabling people and  businesses to innovate. Governance The main decision-making, executive and managerial bodies at Innovate UK, part of UK Research and Innovation, consist of: Innovate UK Council The Innovate UK’ work is overseen by a Council, which sets overall objectives and direction. The Council meets 6 times a year. Members have a range of expertise in research and innovation that is drawn across business, entrepreneurship, investment, technology, economics and business impact evaluation, and have different characteristics and professional backgrounds. Innovate UK Executive Management Team The executive management team lead a team of around 300 technologists, business specialists and support staff to make sure the organisation meets its objectives. Governance of the parent organisation includes: UKRI Board The UKRI Board is UKRI’s primary governing body. It oversees and directs the activities of UKRI, including its strategy for research and innovation. It is supported by an Audit, Risk, Assurance and Performance Committee, and a Nominations and Remuneration Committee. The Board consists of the Chair, the CEO, the CFO and twelve non-executive board members. Executive Committee The Executive Committee provide strategic advice to the Board and is responsible for delivering the Board’s vision through UKRI by overseeing the organisation’s overall performance and deliv- ery. It is supported by a People, Finance and Operations Committee, a Health and Safety Commit- tee, an Investment Committee, and a Strategy Committee. The Executive Committee consists of the CEO, CFO and the Executive Chairs of the nine UKRI Councils. UKRI Councils The nine Councils are responsible for making decisions on scientific, research and innovation matters within their disciplines, as well as holding their Executive Chairs to account for perfor- mance, quality of portfolio, delivery against the council’s Delivery Plan, and reporting of outputs, outcomes and impacts. Funding Priorities Innovate UK funds innovation projects in different ways depending on an organisation’s situation and needs – from issuing competitions on specific themes seeking collaborative proposals from businesses and academia, to offering proof of concept grants or innovation vouchers to individual SMEs. 143 Bulgaria Functional and Governance Analysis It provides non-financial support for innovative businesses in many ways - through the Enter- prise Europe Network which helps companies seek global opportunities; through the Knowledge Transfer Network; by connecting companies to other resources such as the Business Growth Ser- vice and through the new Catapults – physical centres where the best scientists and engineers work together to accelerate the journey of concepts towards commercialisation.25 Impact on the Innovation Ecosystem Innovate UK is a key part of the infrastructure that makes the UK a fertile environment for innova- tive companies seeking to bring new ideas and technologies to market. Since 2007, Innovate UK has invested over £2.2 billion in innovation. This has spanned more than 11,000 projects that have generated up to £16 billion in Gross Value Added for the UK economy and 70,000 jobs. Innosuisse Agency Profile Formation 1943 as Commission to Promote Scientific Research 1996 as Commission for Technology and Innovation 1 January 2018 as Innosuisse Type Swiss Innovation Promotion Agency Legal status Federal entity under public law Main organ The Innosuisse Board Budget CHF 275 million Website www.innosuisse.ch Mission and Objectives CTI/KTI`s original mission statement hasn’t changed as the agency has evolved, and the goal of Innosuisse remains to promote science-based innovation in all disciplines represented at university-level research institutions in the interests of the economy and society. The focus is on research that has the potential to produce concrete and marketable products. By helping to transfer research results into concrete marketable products, Innosuisse helps to improve the competitiveness of Swiss businesses, particularly SMEs, thereby contributing to a strong, inno- vative economy in Switzerland. Governance The Swiss Innovation Agency is an entity under public law with a separate legal identity, which is composed of four expert bodies: Innosuisse Board The Innosuisse Board comprises seven specialist representatives from industry and academia. It is the strategic body of Innosuisse, which it manages in line with the Federal Council’s strategic 25 Science and innovation in the UK, August 2015. 144 Bulgaria Functional and Governance Analysis objectives and with an eye on the future. The composition of the Board ensures a close link be- tween academia and industry. The Board members are also familiar with innovation management and know the mechanisms and challenges surrounding funding innovation as well as the political frameworks. The Board also has proven skills in strategic management, business administration and law and compliance. The Federal Council elects the Board and the committee for a service period of four years. Innovation Council The Innovation Council is the specialist body of Innosuisse. It makes decisions on funding ap- plications and supports the execution of the funding activities in an academic and innovative sense. It also develops suggestions for the funding strategy and instruments to be approved by the Board. To carry out its work, the Innovation Council makes use of a pool of experts. In its se- lection of innovation councillors, the Innovation Council focuses on personalities that have an excellent track record in innovation, who are professionally active and can put their experience to the best use for its tasks for Innosuisse. The Innovation Council also has a diverse composition in a cultural sense: Nine members work in German-speaking Switzerland, seven in Romandy, two in Ticino and one member works abroad. The members of the Innovation Council are elected for a service period of four years. Experts Experts assist the Innovation Council with examining applications for funding and during accom- panying projects. It is their job to endorse, or oppose, applications for innovation projects. The experts come from various backgrounds and must have considerable expertise in the area of scientific innovation. They must demonstrate skills in scientific research and the ability to eval- uate innovative projects, especially in start-ups. In addition, the experts’ main professional activ- ity must pertain to scientific innovation. The experts are selected by the Innosuisse Innovation Council and proposed to the Board of Directors before they are elected. When selecting candi- dates, the Innovation Council ensures equitable representation with respect to: Executive Committee and Secretariat The management team is Innosuisse’s operational body. It directs the Secretariat, manages the activities and monitors the budget of the innovation funding agency. The Secretariat is Innosu- isse’s control centre. Around 70 people implement the funding activities in accordance with the requirements of the Board and the decisions of the Innovation Council. The executive committee holds all threads together; it directs the operational funding activities. Auditing body The Swiss Federal Audit Office SFAO is the auditing body of Innosuisse. It assesses the annual fi- nancial statements and the execution of an appropriate risk management and reports its findings to the Board and the Federal Council in the form of a report. Funding Priorities Innosuisse funds science-based innovation in the interests of industry and society with the aim of increasing the competitiveness of small and medium-sized enterprises in Switzerland. Innosuisse is representing Switzerland in EUREKA and several partnership programs under Hori- zon 2020. In this context and through selected bi-national collaborations, Innosuisse finances 145 Bulgaria Functional and Governance Analysis small and medium-sized enterprises (SMEs) to further develop their products or services togeth- er with partner firms or research teams at the international level. Innosuisse also gives companies easy access to international research networks and manages the Enterprise Europe Network (EEN) in Switzerland. In Switzerland the following network part- ners are financed by Innosuisse to provide EEN services: /Innosuisse (Innovation & technology support & advice on European research programmes. /Euresearch (H2020 support) In addition, Switzerland Global Enterprise (Business support) is financed by the State Secretariat for Economic Affairs (SECO) The support provided by Innosuisse involves: / Helping startups realise business ideas; / Assisting with innovation projects in businesses (in particular SMEs) and at public research institutions; / Helping firms do business internationally (particularly the manufacture and launch of market-ready, innovative product ideas), and supporting start-ups who want to branch out internationally; / Running networks and events in key innovation fields. Innosuisse provides support in accordance with the subsidiarity principle: it only supports proj- ects if the innovation could not be implemented and market potential would not be tapped into without funding. Innosuisse follows a bottom-up approach: Although applicants will select one of five primary funding areas (ICT, Life Sciences, Engineering, Energy & Environment and Social Sciences & Busi- ness Management), there are no predefined topics. Innosuisse identifies obstacles to innovation and its instruments help to overcome barriers ex- isting between public research institutions and the private sector. The three Innosuisse funding areas are closely linked: R&D project funding Innosuisse’s task is to help innovative products and services onto the market to the benefit of the Swiss economy, by encouraging higher education institutions and businesses to carry out joint research and development projects. Companies, especially SMEs, can thereby benefit from the infrastructure in the research institutions and bring their innovative products and services to market. Start-up & Entrepreneurship Innosuisse is tasked with promoting entrepreneurship and supporting innovative business ideas. Innosuisse programmes and networks provide the skills and methods necessary to successfully turn a business idea into a new company. The instruments used include training courses for po- tential and active young entrepreneurs and coaching and support in searching for investors. KTT Support KTT stands for knowledge and technology transfer between businesses and research circles. This funding area brings together businesses, especially SMEs, and public research institutions in cas- 146 Bulgaria Functional and Governance Analysis es where contact has not otherwise been established. The aim is to create partnerships which will then act as drivers of innovation in Switzerland. Impact on the Innovation Ecosystem In 2016, the Commission for Technology and Innovation (CTI), Innosuisse’s predecessor organisa- tion, commissioned a first phase of comprehensive impact analyses to examine both the perfor- mance and effects of promotion and also the design and implementation of the funding instru- ments in greater depth. In the second phase in 2018, the focus was on the impact aspects and benefits of the funding instruments. The analyses conducted to date have also provided Innosu- isse with valuable information for developing an overarching concept for assessing the effective- ness of its funding instruments. In addition, the results emerged from three external impact studies conducted on behalf of Inno- suisse in 2018 show that the companies and start-ups consider the support provided by the fed- eral government’s innovation promotion agency Innosuisse to be very useful overall. In addition to the purely economic benefits, the strengthening of knowledge and technology transfer and the increase in research expertise are particularly appreciated. 147 Bulgaria Country Office Advance Business Center, 2 Samara Street, 1715, fl.4 Sofia, Bulgaria Tel: (359-2) 969-72-29 Fax: (359-2) 971-20-45 www.woldbank.org/bulgaria info@worldbank.bg