Pillar 1 Bulgaria Public Expenditure Review in Science, Technology and Innovation (STI) Bulgaria: Efficiency Analysis of STI Programs Reimbursable Advisory Services Agreement on Public Expenditure Review in Science, Technology and Innovation and Support for Building Evidence-based Approach for the National Strategic Framework in Education 2030 Pillar 1 Bulgaria Public Expenditure Review in Science, Technology and Innovation (STI) Bulgaria: Efficiency Analysis of STI Programs Reimbursable Advisory Services Agreement on Public Expenditure Review in Science, Technology and Innovation and Support for Building Evidence-based Approach for the National Strategic Framework in Education 2030 Final Draft – November 2021 Bulgaria: Efficiency Analysis of STI Programs This report is a product of the staff of The World Bank. The findings, interpretations, and conclu- sions expressed in this report are entirely those of the authors and they do not necessarily reflect the views of The World Bank, its Executive Directors, or the governments they represent, nor do they reflect the views of the Bulgarian government or its institutions. The World Bank does not guarantee the accuracy of the data included in this work, which is drawn from multiple external sources. Nothing herein shall constitute, or be considered to be, a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Bulgaria: Efficiency Analysis of STI Programs Contents Acknowledgements 6 Abbreviations and Acronyms 7 Executive Summary 8 Introduction 14 1. Methodology 16 1.1 Programs Selected 17 1.2 Indicators 20 1.3 Data 21 1.4 Limitations 23 2. Results Across Programs 24 2.1 Efficiency in the Use of Inputs 26 2.2 Efficiency in Generation of Outputs 32 2.3 Efficiency in Achievement of Outcomes 41 2.4 Perceived Program Quality 50 3. Fundamental Research 58 3.1 Efficiency in Use of Inputs 60 3.2 Efficiency in Generation of Outputs 69 3.3 Efficiency in Achievement of Outcomes 73 3.4 Perceived Quality 76 4. Vihren 83 4.1 Efficiency in Use of Inputs 85 4.2 Efficiency in Generation of Outputs 93 4.3 Efficiency in Achievement of Outcomes 95 4.4 Perceived Quality 5. ICT for a Single Digital Market in Science, Education, and Security 101 5.1 Efficiency in Use of Inputs 104 5.2 Efficiency in Generation of Outputs 108 5.3 Efficiency in Achievement of Outcomes 112 5.4 Perceived Quality Bulgaria: Efficiency Analysis of STI Programs 5 6. Electronic Health in Bulgaria 118 6.1 Efficiency in Use of Inputs 121 6.2 Efficiency in Generation of Outputs 126 6.3 Efficiency in Achievement of Outcomes 130 6.4 Perceived Quality 131 7. National Innovation Fund 133 7.1 Efficiency in Use of Inputs 136 7.2 Efficiency in Generation of Outputs 143 7.3 Efficiency in Achievement of Outcomes 146 7.4 Perceived Quality 148 8. Development of Product and Process Innovations 155 8.1 Efficiency in Use of Inputs 158 8.2 Efficiency in Generation of Outputs 164 8.3 Efficiency in Achievement of Outcomes 167 8.4 Perceived Quality 169 9. Recommendations 176 9.1 Recommendations for Improving the Efficiency and Effectiveness of STI Support 177 9.2 Recommendations for Strengthening M&E Practices for New and Existing STI Programs 182 References 185 Appendices 187 Appendix A. Theories of Change for Analyzed Programs 188 Appendix B. Researcher Survey Questionnaire 222 Appendix C. Firm Survey Questionnaire 238 Appendix D. Administrative Costs 255 Bulgaria: Efficiency Analysis of STI Programs 6 Acknowledgements This report was prepared by a World Bank team led by Łukasz Marć (Economist, Task Team Leader), and comprising Daniel Querejazu (Innovation Policy Specialist), Teodora Georgieva (Research and Innovation Expert, Applied Research and Communication Fund), Lyubomira Dimitrova (Research and Statistical Expert), Ana Goicoechea (Senior Economist), and Umut Kilinc (Economist). The report was edited by Aarre Laakso. The report benefited from the guidance of the World Bank management, Fabrizio Zarcone (Country Manager), Ilias Skamnelos (Practice Manager), and Reena Badiani-Magnusson (Program Leader) and from feedback and comments provided by Justin Hill (Senior Private Sector Specialist) and Todor Milchevski (Senior Private Sector Specialist). The report also benefited greatly from the leadership of the previous Task Team Leader, Anwar Aridi. The team would like to thank the Government of Bulgaria—particularly the Ministry of Education and Science, National Science Fund, Directorate General of the Operational Programme Innovation and Competitiveness, and SME Promotion Agency—for their support in carrying out this analysis. Bulgaria: Efficiency Analysis of STI Programs 7 Abbreviations and Acronyms CART Credible, Actionable, Responsible, Transportable DG OPIC Directorate General of the Operational Program Innovation and Competitiveness DPPI Development of Product and Process Innovations eHealth Electronic Health in Bulgaria EU European Union FR Fundamental Research FTE Full-Time Employees HEI Higher Education Institution ICT ICT for a Single Digital Market in Science, Education, and Security ICT-R Improving Competitiveness and Efficiency of SMEs in ICT IP Intellectual Property IPR Intellectual Property Rights M&E Monitoring and Evaluation MoE Ministry of Economy MoES Ministry of Education and Science NIF National Innovation Fund NSF National Science Fund NSME Innovations in Newly Established SMEs OP Operational Programme PER STI Public Expenditure Review for Science, Technology, and Innovation PRO Public Research Organization SARI State Agency for Research and Innovation SMEPA Small and Medium Enterprise Promotion Agency STI Science, Technology, and Innovation ToC Theory of Change UMIS Unified Management Information System Bulgaria: Efficiency Analysis of STI Programs 8 Executive Summary Transitioning to a more knowledge- and technology-based economy will be critical if Bulgaria is to achieve convergence with its European peers in terms of economic productivity and living standards. Despite dramatic productivity growth over the last decade, Bulgaria’s productivity performance is still among the weakest in Europe. The country also faces medium- to long-term challenges in its workforce due to an aging population and high emigration. This transition to a more knowledge- and technology-based economy will require large improvements to the coun- try’s science, technology, and innovation (STI) performance, which ranks among the weakest in the EU across multiple indicators. Contributing factors to this poor STI performance include in- adequate funding for research, a lack of coordination of STI policies and programs, and serious capacity issues on the part of STI implementing bodies. The coming year presents an important opportunity for Bulgaria to improve its support for STI. The new National Recovery and Resilience Plan will allocate approximately €1.25 billion in funding for innovation-related activities. Moreover, new priorities and targets are being defined for the up- coming programming period, which will allocate approximately €1.17 billion toward research and innovation. These new investments would greatly benefit from a careful review of the strategies, policies, and programs that support STI nationally, drawing lessons and recommending reforms to ensure that the new funding is allocated efficiently and produces real impacts on the STI system. This report assesses the efficiency of selected Bulgarian STI support programs, tracks progress on outcomes, and reports on beneficiaries’ perceptions of program quality. This analysis is not intended to judge the viability of the programs included in this report, but rather to review and analyze the programs according to the needs of the Bulgarian STI system and make recommen- dations for their improvement. This report is the third and final major component of the World Bank’s Bulgaria Public Expenditure Review for Science, Technology, and Innovation (PER STI) proj- ect. The first component of the PER STI project, the Country Needs and Policy Mix Assessment, provided a comprehensive assessment of the country’s STI needs, an overview of the national policies devoted to supporting STI in Bulgaria, and an analysis of the alignment or gaps between policy support and the STI needs of the nation’s public and private sectors. The second compo- nent, the Functional and Governance Analysis, analyzed the functionality of a representative set of STI policy instruments through a review of their design, implementation, and governance and provided a set of recommendations to improve the functionality and governance of the portfolio of STI support programs. Bulgaria: Efficiency Analysis of STI Programs 9 This efficiency analysis addresses an identified gap in Bulgaria’s STI system: the monitoring and evaluation (M&E) of STI support programs. Previous components of this project found that STI implementors lack capacity and resources for M&E, and few evaluations of STI programs have been done during the current programming period. This report aims to address those gaps by identifying benchmarks for assessing the results of STI support programs in Bulgaria and provid- ing a methodology and tools for carrying out such evaluations in the future. Key Findings from Analysis of Research Support Programs Research support programs had very different levels and compositions of administrative costs of implementation, largely due to the different management structures of the programs. Programs implemented by the National Science Fund (NSF) had a relatively high share of costs for external services and low share of costs for personnel, indicating that much of the program implementation is done by external experts rather than full-time NSF staff. By contrast, the sectoral programs of Bulgaria’s National Science Programs 2018–22 portfolio have very low administrative costs over- all because part of these costs are borne by beneficiary consortiums, which are responsible for part of the program administration and monitoring. Research programs generally produced outputs and had outcomes that were in line with the program objectives, but a citation analysis raises concerns about the quality of research be- ing produced. Publications reported by research respondents generally had lower numbers of year-normalized citations than the average Bulgarian publication indexed in the Web of Science from 2016 to 2020. This low level of citations indicates that the research being funded is not gen- erating significant impact on the scientific community. Publications in peer reviewed journals were the most common outcome for all of the research programs, although publications per project and per unit of cost varied between programs. Capacity building outputs (in the form of seminars, conferences, and workshops; training activities; and hiring new researchers) were the most common outputs of almost every research program. NSF programs generated relatively few collaborations during project implementation, despite the fact that increasing collaboration activity among beneficiaries is a goal of these programs. This may be because program regulations provide few incentives that encourage beneficiaries to collaborate. Additionally, some respondents from these programs noted that they would like eligibility criteria expanded to allow for more types of collaboration partners. There were few collaborations with researchers from the Bulgarian diaspora, which could be an opportunity for additional collaboration. Bulgaria: Efficiency Analysis of STI Programs 10 A small share of respondents from NSF programs reported outcomes related to prototyping, new software development, and new technology development, which are not intended results of those programs. Such projects could represent an opportunity for follow-up applied research grants that allow beneficiaries to further develop their project results. The Country Needs and Policy Mix Assessment showed a gap in applied research programs providing funding to public research institutions: at present, NSF regulations do not allow for funding to be used for com- mercialization, intellectual property (IP) development, or technology transfer, and there are few applied research programs that public researchers can access outside of NSF that would sup- port such follow-on projects. Respondents from researchers’ programs were mostly satisfied with program processes, but some had issues with the transparency of the selection process and with monitoring requirements. Although researchers were generally satisfied with project application and selection, some were dissatisfied with the unavailability of feedback on which projects were selected, which can make selection processes appear opaque. Some respondents were also dissatisfied with program re- porting requirements, particularly financial reporting rules, which require beneficiaries to provide certified and translated copies of all eligible expenses, including things like plane tickets. Key Findings from Analysis of Firm Support Programs Firms reported higher costs than researchers in preparing their applications, likely driven by the large share of firms that used external support to prepare their applications. Average applica- tion costs for the two firm support programs range from about 2.3 percent to 3.7 percent of the contracted grant value. Roughly 90 percent of firms used some form of external support (such as legal, accounting, or other services) to prepare their applications. Application costs can be a barrier to participation in a program, particularly for SMEs and young firms, which may not have the resources to apply. Respondents from firm support programs generally achieved the outputs and outcomes intended by the programs, including developing and upgrading products, services, and processes; and in- creasing sales and employment. More than 70 percent of firm projects resulted in firms adopting a product or service for the first time. Roughly 65 percent of projects resulted in the introduction of a product or service that was new to the Bulgarian market. Firm beneficiaries reported dissatisfaction with key areas of program implementation, including the overall application process, the transparency of project selection, and financial monitoring requirements. The Functional and Governance Analysis found that application processes for firms Bulgaria: Efficiency Analysis of STI Programs 11 are burdensome, particularly the supporting documentation firms must provide to support their applications, which can include financial statements, declarations of clean criminal records, tax liabilities, and offers from third parties for assets. The Functional and Governance Analysis also found that reporting processes were easier for beneficiaries of programs funded under EU opera- tional programmes (OPs) than for nationally funded programs because beneficiaries of OP-fund- ed programs report using an online portal with preloaded templates for technical and financial reports. Despite these challenges, more than 70 percent of respondents felt their project results matched their expectations and were generally satisfied with other aspects of their programs, in- cluding program rules and regulations and the accessibility of financial and administrative support. Recommendations The key findings from this efficiency analysis suggest several changes that could improve the efficiency and effectiveness of STI support: 1. Consider the costs and benefits of the management and cost structures used for program implementation. NSF and, to a lesser extent, the SME Promotion Agency (SMEPA) rely heav- ily on external experts in program implementation to make up for the lack of full-time staff at these institutions. This arrangement is not inherently inefficient, but increasing full-time staff and reducing the use of external services would likely increase the internal capacity of these institutions (identified as a major challenge in previous components of this PER STI project) and could lead to more effective program delivery. 2. Analyze the quality of program outcomes, were possible, for evidence on the scientific im- pacts of research programs and projects. Programs currently track numbers of publications generated by beneficiaries, but additional citation analyses on publication outputs would help implementors understand whether programs are achieving their research excellence objec- tives, show which publications are most influential on the scientific community, and help im- plementors make more informed decisions about which programs and projects to fund. 3. Provide incentives for collaboration in programs with the objective of improving collabora- tion and the connections of beneficiaries. This can be done by building collaborations into the design of programs (as a precondition for a grant award, for example), accounting for the transaction costs of collaboration in program funding, providing additional points on appli- cations that include for research partners, and allowing beneficiaries to collaborate with a broader range of organizations during project implementation. 4. Explore developing programs to further support beneficiaries of NSF programs. The small share of respondents from NSF programs that reported outcomes related to prototyping, new soft- ware development, and new technology development suggests that some NSF project results Bulgaria: Efficiency Analysis of STI Programs 12 could be further developed and potentially commercialized if there were one or more applied research grant schemes to which their projects could “graduate.” This would help address the gap in applied research funding programs to public research institutions. 5. Reduce application costs for firm support programs. Nationally funded programs would great- ly benefit from an online portal similar to that used by the OPs, which allows beneficiaries to submit and receive information throughout the application and contracting process. Addition- ally, implementing bodies should acquire documentation (tax documents, criminal records, and so on) ex officio, where possible, and submission requirements for any documents that are not essential to the application process should be deferred until the contracting phase of the project. These changes would likely lower the need for firms to hire external consultants and services to prepare their applications and reduce the overall application costs for firms. 6. Reduce the administrative burdens on beneficiaries during implementation. The creation of an online portal with preloaded technical and financial reporting templates would simplify reporting processes for programs outside of the OPs. Financial report requirements could be simplified by approving R&D project expenditure plans in advance, reversing the current ap- proach in which each expense item must be reported when incurred. For programs support- ing R&D activities, expenses for R&D activities can be presumed to be eligible when reported and then verified later. In such a system, applicants would self-report whether they are eligible, and implementors would conduct audits later to verify eligible costs. STI institutions should build on this analysis by committing to systematic monitoring and eval- uation of their portfolios. As detailed in the Functional and Governance Analysis, more regular evaluations of Bulgaria’s STI programs will require increasing the M&E capacity of STI implemen- tors, improving institutional incentives for M&E, and committing much more resources to M&E than are currently available. 1. Track program-level cost data. Currently, no STI implementors systematically track all the costs related to the implementation of the programs, making it extremely difficult to understand the true costs of program implementation. Implementors can use the cost data templates that were used for this analysis to begin to more systemically collect and analyze this information. 2. Conduct regular surveys of the beneficiaries of STI programs. Regular surveys of beneficiaries would allow implementors to collect information on outputs and outcomes that is comparable across projects and programs and better understand the performance of their policy portfolios. Surveys can also provide information on beneficiaries’ perceptions of their programs, including which aspects of programs beneficiaries believe are working well or need improvement. The survey questionnaires in the appendices to this report can be adapted for beneficiaries of a range of STI support programs Bulgaria: Efficiency Analysis of STI Programs 13 3. Conduct regular efficiency analyses of STI programs. Through systematic tracking of program cost data and regular beneficiary surveys, STI implementors will be able to assess programs’ efficiency in the use of inputs, generation of outputs, and achievement of outcomes on a reg- ular basis. 4. With more complete data on costs and results, STI policymakers should use findings to im- prove the efficiency of STI programs in generating desired results with public funds. This re- port, along with future efficiency analyses, can be used to define benchmarks and targets for STI programs, understand which programs are achieving their objectives, and make informed decisions to improve the efficiency of public funding for STI. 5. Conduct impact evaluations of strategically important programs. Efficiency analyses should be complemented by impact analyses, at least for strategically important programs. Guidelines for when to evaluate individual programs can be determined by a national-level evaluation framework, such as that used in the Republic of Korea, or by frameworks at the institutional or programmatic level. Impact analyses supplement data on program efficiency by establish- ing which changes or impacts can be attributed to a given intervention, allowing for a more complete picture of program impact and effectiveness. Instruments that are strategically im- portant (for example, those with large budgets, supporting a large number of beneficiaries, and with large expected impacts) should undergo at least one performance evaluation and one external impact evaluation per programming period. Ideally, these evaluations should be conducted by third parties (that is, by evaluators outside of the implementing body) that do not have conflicts of interest with the instrument being evaluated. Bulgaria: Efficiency Analysis of STI Programs 14 Introduction This document and the analysis it reports are the third major component of the Bulgarian Public Expenditure Review in Science, Technology, and Innovation (PER STI). PER STI is a methodolog- ical approach developed by the World Bank that aims to examine public spending for STI and provide actionable recommendations to increase its effectiveness. The analysis was conducted by the World Bank at the request of the Ministry of Education and Science (MoES) and in close cooperation with the National Science Fund (NSF), the Ministry of Economy (MoE), and the SME Promotion Agency (SMEPA). This analysis builds on findings from the earlier components of the PER STI project, as well as other recent analyses of the Bulgarian STI system. Component I of the PER STI project, the Country Needs and Policy Mix Assessment (World Bank 2020), found that STI institutions often lack measurable objectives and targets. Component II, the Functional and Governance Analysis (World Bank 2021a), found three key issues that could impact the efficiency and effectiveness of STI instruments. First, STI implementors outside of the operational programs, such as the NSF and SMEPA, suffer from a lack of staff and resources to fully implement their portfolios. Second, program inputs, with the exceptions of direct financial transfers to beneficiaries, are generally not well tracked or catalogued. Third, STI implementors lack monitoring and evaluation (M&E) capacity and resources, and M&E frameworks tend to focus on monitoring and compliance with administrative regulations. Almost no evaluations take place. Moreover, only a few instruments have an explicit theory of change as part of their design, which has contributed to poorly defined and disconnected indicators for program inputs, activities, outputs, and outcomes. Several other recent reviews of the Bulgarian STI system have also identified M&E frameworks an area of con- cern (European Commission 2015; World Bank 2013). This report builds on and complements this past PER work with an assessment of the efficiency of select STI programs in converting inputs into outputs and outcomes. Analyzing program ef- ficiency allows policymakers to understand the level of investment needed to generate desired outputs and outcomes, identify inefficiencies and challenges, and make informed decisions about program design and resource allocation. This effort makes two key contributions toward the objectives of the PER STI project. First, it pro- vides a set of benchmarks for assessing the results of STI support programs in Bulgaria. Second, it demonstrates the efficiency analysis approach and methodology to evaluating STI instruments and provides templates and tools for STI implementors to carry out such analyses in the future. Bulgaria: Efficiency Analysis of STI Programs 15 The analysis was conducted on a subset of STI support programs, using a variety of data sources. Of the 120 STI instruments mapped in the first component of this project (the Country Needs and Policy Mix Assessment), 28 instruments were chosen for inclusion in component 2 (the Function- al and Governance Analysis). In consultation with MoES and following the criteria described in Section 1 of this report, six instruments were selected from the 28 for efficiency analysis. Data on outputs and outcomes were collected through two surveys, one targeting beneficiaries from programs targeting researchers and the other targeting beneficiaries from programs targeting firms. Administrative data on the costs of program implementation were collected by program staff. These data were supplemented by analysis and findings from the first two components of the PER STI project. Along with the analysis and recommendations included in this report, several tools were devel- oped to support future efficiency analyses of Bulgarian STI programs. The tools include two sur- vey questionnaires tailored to public sector researchers and to firms, respectively; a template for program cost data collection; and Microsoft Excel worksheets for analyzing and visualizing the data collected. This analysis and the accompanying tools build upon earlier efficiency analyses of STI programs in Colombia (World Bank 2018) and Croatia (World Bank 2021b). The report is structured in nine sections. Section 1 presents the methodology, Section 2 provides a comparison of key results across the six programs included in the analysis, Sections 3 through 8 provide program-specific analysis and findings, and Section 9 provides recommendations. Bulgaria: Efficiency Analysis of STI Programs 16 1. Methodology Bulgaria: Efficiency Analysis of STI Programs 17 1. Methodology This report aims to assess select STI programs in Bulgaria in terms of their efficiency in using in- puts and generating results and their perceived quality. The analysis is guided by the following research questions: • Program inputs: What are the costs of implementing the programs? Of these costs, what costs are borne by implementing bodies, and what costs are borne by beneficiaries? • Program results: Are the programs generating the intended outputs? Are the programs lead- ing to the intended outcomes? Are the programs leading to unintended outcomes? What are the outputs and outcomes per unit of cost? • Program quality: How do beneficiaries perceive the quality of program processes related to application, selection, implementation, and M&E? How do beneficiaries perceive the results of their projects? 1.1 Programs Selected Six programs were selected for analysis based on multiple criteria. The criteria includes (1) the number of projects that had been funded; (2) the prospects for the continuation of the program; (3) the availability of information and data on inputs, outputs, and outcomes; and (4) the impor- tance of the program in the portfolio of the implementing institution. Four programs were selected focused on supporting researchers and research institutions. These four programs supported a total of 690 research projects from 2016 to 2020. Two programs were selected focused on sup- porting firms. They supported a total of 235 private sector projects from 2016 to 2020. Table 1.1 lists the selected programs. Within the current STI policy mix in Bulgaria, the four programs tar- geting researchers account for 11 percent of the overall project financing allocated for research, while the two programs targeting firms account for 3 percent of the project financing allocated for firms. All programs provided beneficiaries with grants, and those targeting firms required firms to provide matching funds. Bulgaria: Efficiency Analysis of STI Programs 18 Table 1.1. Programs Included in the Efficiency Analysis Program Implementor # of Projects Calls included 2016 2017 Fundamental Research (FR) NSF 664 2018 Research 2019 Support 2020 Programs Vihren NSF 10 2019 ICT for a Single Digital Market in Science, MoES 12 2019 Education, and Security (ICT) Electronic Health in Bulgaria (eHealth) MoES 4 2019 2016 2018 Firm National Innovation Fund (NIF)* SMEPA 82 2019 Support Programs 2020 Development of Product and Process Innovations DG OPIC 154 2018 (DPPI) Source: World Bank. Note: DG OPIC = Directorate General of the Operational Program Innovation and Competitiveness; MoES = Ministry of Education and Science; NSF = National Science Fund; SMEPA = Small and Medium Enterprise Promotion Agency. a. For NIF, efficiency analysis indicators (outputs and outcomes per unit of program cost) only include beneficiaries from 2018 to 2020 because administrative cost data were only available for those years. The analysis for each program covers four areas: (1) efficiency in the use of inputs, (2) efficiency in the generation of outputs, (3) efficiency in the achievement of outcomes, and (4) perceived quality of program processes and results. These areas were defined based on the World Bank guidance note Public Expenditure Reviews in Science, Technology, and Innovation (Correa 2014) and build upon experiences and lessons learned from previous efficiency analysis conducted by the World Bank in Colombia and Croatia. Figure 1.1 presents the conceptual framework that guides this report. Bulgaria: Efficiency Analysis of STI Programs 19 Figure 1.1. Conceptual Framework inputs outputs outcomes investment in beneficiaries costs outcomes covered by the achieved program outputs achieved costs covered by the outcomes beneficiaries per cost outputs per cost beneficiary perceptions satisfaction with satisfaction with satisfaction with application and selection implementation and project results processes monitoring processes DATA SOURCE ADMINISTRATIVE COSTS PROVIDED BY PROGRAMS BENEFICIARY SURVEYS AUTHORS CALCULATIONS Source: Adapted from World Bank 2021b. Bulgaria: Efficiency Analysis of STI Programs 20 1.2 Indicators Indicators for program inputs, outputs, and outcomes were drawn from theories of change (ToCs) that were developed for each program. (See Appendix A for the theories of change for each pro- gram in this analysis.) These ToCs were based on available program documentation and interac- tions with program staff. Efficiency in the Use of Inputs The analysis of efficiency in the use of inputs includes both program inputs covered by the pro- gram and those covered by the beneficiaries. 1. Costs covered by the program include the amount and composition of program costs to the implementing body. Costs include transfers to beneficiaries, in the form of direct financial trans- fers (such as grants and matching grants), indirect financial transfers (such as tax deductions or reductions in fees), and non-financial transfers (such as technical assistance and access to equipment). They also include the administrative costs of implementing the program in the form of personnel costs (for program design, implementation, and monitoring), fixed costs (such as office equipment, office space, and IT), and external services (such as contracts for external technical experts, legal services, or other outside services). 2. Costs covered by beneficiaries include application costs and contributions (cash and in kind) from beneficiaries to their projects. 3. Efficiency in the use of inputs includes the average administrative and operating cost per proj- ect, the average cost per project, the average cost covered by beneficiaries, and the ratio of costs covered by the program to costs covered by beneficiaries. Efficiency in the Generation of Outputs The analysis of efficiency in the generation of outputs looks at two key areas. 1. Investments in beneficiaries include total funds disbursed to program beneficiaries, aver- age transfers per project, transfers per unit of administrative cost, and uses of funding by beneficiaries. 2. Intended outputs achieved includes an analysis of the outputs achieved relevant to each pro- gram (based on the program’s ToC), as well as efficiency measures such as average number of outputs achieved per project and the number of outputs achieved per unit of program cost. Bulgaria: Efficiency Analysis of STI Programs 21 Efficiency in the Achievement of Outcomes The analysis of efficiency in the achievement of outcomes relates to the average number of out- comes achieved per project relevant to each program (based on the program’s ToC). Perceived Quality of the Programs The analysis of the perceived quality of programs relies on the beneficiaries’ opinions of the quality of various program components. This includes beneficiaries’ perceptions of application, selection, and contracting processes, support provided by the implementors during project im- plementation, sufficiency of the time and funding provided for project implementation, key suc- cess factors, and perceptions of the overall quality of the program. 1.3 Data Data for this analysis were drawn from two surveys (one targeting researchers and one targeting firms) and administrative cost data provided by the staff of the selected programs. Surveys were administered to the entire population of research and firm beneficiaries of the selected programs from 2016 to 2020; no sampling was done of the beneficiary pool. The survey of public sector re- searchers was delivered to the principal investigators of 690 supported projects, while the survey of firms was delivered to a designated contact person for 236 projects. For the implementation of the beneficiary survey, the World Bank research team worked in collaboration with Alpha Re- search, a Bulgarian survey company. The World Bank team designed the survey questionnaires (with inputs and feedback from Alpha Research), and the survey was carried out by Alpha Re- search from April to July 2021. The World Bank team and Alpha Research independently verified the quality of the data collected, and Alpha Research contacted respondents when data quality flags were raised for additional verification. The survey questionnaires used for this effort were adapted to the local context and selected programs from questionnaires initially developed by the World Bank (2021b) (See Appendixes B and C for the survey questionnaires used in this anal- ysis.). Both questionnaires include questions on the characteristics of beneficiaries, beneficiary costs and contributions (cash and in kind), their experiences and perceptions with application and implementation processes, and the results achieved to date. Questionnaires were first de- veloped in English and then translated into Bulgarian by Alpha Research. Bulgaria: Efficiency Analysis of STI Programs 22 Overall response rates were relatively high for a survey of this kind, with just under 60 percent of all beneficiaries responding to the survey.1 Response rates were higher for programs targeting researchers, with 63 percent of all researcher beneficiaries responding to the survey. They were lower for firms, with 51 percent of firm beneficiaries responding to the survey (Table 1.2). Table 1.2. Program Response Rates Researcher Programs Firm Programs Fundamental Research Vihren ICT eHealth NIF DPPI Total Total beneficiaries (no.) 664 10 12 8 82 154 930 Responses (no.) 414 8 8 4 44 80 558 Response rate (percent) 62 80 67 50 54 52 60 Source: World Bank. Note: DPPI = Development of Product and Process Innovations; ICT = Information and Communication Technologies for a Single Digital Market in Science, Education, and Security; NIF = National Innovation Fund. Administrative data on program expenditures were gathered by implementing institutions using a template provided by the World Bank team. The template included instructions on how to cat- egorize and calculate program costs. Expenditures were classified into direct financial transfers to beneficiaries (grants provided), non-financial support to beneficiaries (such as technical as- sistance, facilities, or equipment lent), and administrative and operating costs (including person- nel, fixed costs, and external services). Personnel costs were calculated based on the salaries of program staff and the number of days per year they estimated working on the program. For fixed costs and external services, if the actual annual costs attributable to the programs were not avail- able, they were estimated using a weight based on the annual personnel costs of the program as a share of total annual institutional personnel costs (for the Fundamental Research, Vihren, ICT, eHealth, and NIF programs) or a weight based on the number of projects funded by the program as a share of the total projects funded by the implementor (for the DPPI program). Throughout the report, funding and cost data are reported in constant 2020 terms in Bulgarian lev (BGN), using the Harmonized Index of Consumer Prices. 1 Beneficiaries were considered as responding to the survey if they completed 50 percent or more of the questionnaire. Bulgaria: Efficiency Analysis of STI Programs 23 1.4 Limitations It is important to acknowledge certain limitations of this methodological approach. Firstly, this is an efficiency, not an effectiveness analysis. Specifically, it does not include an analysis of coun- terfactual outcomes. Therefore, it cannot assess the changes or impacts that can be attributed to the programs. Evaluating the effects attributable to the programs would require comparing data from a carefully selected group of non-beneficiaries with data from beneficiaries. Further, the methodology does not differentiate between “good” and “bad” costs. Low administrative costs are not necessarily good or efficient, but may indicate a lack of effort or attention to an in- strument’s design, monitoring, and so on. Pilots and experimental interventions generally require more effort (and therefore more administrative costs) to implement than established programs, so implementation costs should be viewed within the context of individual programs. Addition- ally, this evaluation approach could be gamed by beneficiaries familiar with the methodology by generating a large number of low-effort outputs, making their projects look more efficient at gen- erating project results per unit of cost. Limited data availability means there are some limitations to the administrative cost data pre- sented in this report. This includes some missing data on fixed costs for the ICT, eHealth, and DPPI programs and missing data on external services for Fundamental Research, meaning that those costs reported here are likely lower than the actual costs of implementing those programs. (See Appendix D for full details on the collection, analysis, and limitation of the administrative costs included in this report.) When interpreting the results, the reader should understand that not all project results are equal. Some results require more resources and a higher level of effort. (A working prototype requires more effort than a training session, for example.) Moreover, some results are more valuable than others. (A patent is worth more than a conference presentation, for example.) Where possible, in- formation on the quantity of outputs is complemented by an analysis of their quality. For example, a bibliometric analysis was conducted on the scientific publications reported by beneficiaries. However, such an analysis is not possible for many of the outputs and outcomes in this report. For example, there is no quality measure for many firm outcomes, such as new and upgraded products, services, and processes. The results should be also viewed within the context of the specific programs. In the case of the programs with a small number of beneficiaries (such as the National Innovation Fund, ICT, and eHealth), the findings have more descriptive than statistical value. Also, for some programs (such as Vihren, ICT, and eHealth), all of the beneficiary projects are still active, and respondents may not have had sufficient time as of the writing of this report to achieve certain results. Bulgaria: Efficiency Analysis of STI Programs 24 2. Results Across Programs Bulgaria: Efficiency Analysis of STI Programs 25 2 Results across Programs This section presents key results (related to inputs, outputs, outcomes, and perceived quality) that can be compared across the six programs included in this analysis. Sections 3 to 8 provide more detailed findings on each of the individual programs. When considering differences in results across programs, it is important to consider program context. The selected programs differ in their targeted beneficiaries, objectives, average contract- ed grant size, timelines, and other important dimensions. Some programs issued multiple calls for proposals, while other programs only issued a single call from 2016 to 2020 (Table 2.1). These differences impact the types of outputs and outcomes expected from funded projects, as well as the efficiency of the programs in producing results (See the ToCs for each program in Appendix A for a complete description of expected outputs and outcomes). Table 2.1. Selected Programs with Target Beneficiaries, Grant Sizes, and Objectives Target Number of Avg. Disbursed Total grants # of calls, Program Beneficiaries projects Grant Size disbursed 2016-2020 Objective(s) Fundamental Researchers 664 96,626 BGN 64,159,408 BGN 5 Research excellence Research Vihren Researchers 10 459,148 BGN 4,591,480 BGN 1 Research excellence, skills ICT PROs, HEIs 12 262,500 BGN 3,150,000 BGN 1 R&D-based innovation eHealth PROs, HEIs 4 500,000 BGN 2,000,000 BGN 1 R&D-based innovation NIF Firms 80 203,763 BGN 13,652,088 BGN 4 R&D-based innovation R&D-based innovation; DPPI Firms 154 458,579 BGN 70,621,104 BGN 1 non-R&D innovation Source: World Bank. Note: All amounts in constant 2020 BGN. Bulgaria: Efficiency Analysis of STI Programs 26 2.1 Efficiency in the Use of Inputs Program costs were primarily direct financial transfers to beneficiaries in the form of grants or matching grants. Vihren and DPPI had the largest cost per project due to larger average disbursed grant sizes than the other programs (Figure 2.1). Administrative costs represented between 0.2 percent and eight percent of total costs, depending on the program (Figure 2.2). The programs with multiple calls for proposals (Fundamental Research and NIF) had the highest ratio of admin- istrative costs to overall program costs. None of the programs reported costs related to non-fi- nancial transfers to beneficiaries (such as advisory services) or indirect financial transfers to beneficiaries (such as tax deductions). Figure 2.1. Average Cost per Project 500000 450000 400000 Constant 2020 BGN 350000 300000 250000 200000 150000 100000 50000 0 fr vihren ict ehealth nif dppi Source: World Bank. Disbursed grants Administrative costs Figure 2.2. Ratio of Disbursed Grants to Administrative Costs 100% 90% Share of program costs 80& 70% 60% 50% 40% 30% 20% 10% 0% fr vihren ict ehealth nif dppi Source: World Bank. Disbursed grants Administrative costs Bulgaria: Efficiency Analysis of STI Programs 27 For some programs, particularly those administered by NSF, personnel costs represented a very small share (less than 10 percent) of total administrative costs, while external services repre- sented a relatively high share (more than 50 percent) (Figure 2.3). Component 2 of this PER STI project, the Functional and Governance Analysis, found that STI implementing bodies outside of the operational programmes suffer from a lack of full-time staff to fully implement their program portfolios. This balance of personnel costs and external services is likely due, at least in part, to the lack of staff at NSF (which implements the Fundamental Research and Vihren programs) and, to a lesser extent, SMEPA (which implements NIF). This arrangement is not necessarily inefficient or ineffective for these programs, but it does mean that the internal capacity of NSF and SMEPA may be lower than it would be if more of the program implementation was carried out internally. Figure 2.3. Administrative Costs by Category 100% Share of administrative costs 80% 60% 40% 20% 0% fr vihren ict ehealth nif dppi Source: World Bank. Personnel costs Fixed costs External services Bulgaria: Efficiency Analysis of STI Programs 28 Costs Covered by Beneficiaries Programs targeting firms tended to have higher application costs. Beneficiaries of programs tar- geting firms reported higher costs than beneficiaries of programs targeting researchers in terms of total application costs as well as in terms of application costs as a share of contracted grant size (Figure 2.4). Application costs for programs targeting researchers ranged from about 2,200 BGN to 2,900 BGN and from 0.3 percent to 1.9 percent of the contracted grant value, while appli- cations costs for programs targeting firm range from about 7,700 BGN to 10,500 BGN and from 2.3 percent to 3.7 percent of the contracted grant value. Given that researcher and firm respon- dents took roughly similar numbers of working days to prepare their applications (see Figure 2.6 below), some of the discrepancy in costs between researchers and firms may be due to the fact that some researchers view their time as “free”, while firms tend to place a monetary value on the time of their employees. While the application costs for firms’ programs were higher than the re- searcher support programs, they were lower than the application costs of firm support programs found in a similar analysis in Croatia, where application costs for firm support programs ranged from 4 and 7 percent of the average grant amount (see Box 2.1). Firms tended to use additional support to prepare their applications to these programs. Appli- cants’ use of external support (such as administrative, legal, or accounting assistance) to prepare their applications likely contributed to the higher application costs of firm support programs. Firm respondents generally used some form of additional support, with over 70 percent of firms applying for NIF and DPPI using some form of outside assistance to prepare their application. For programs targeting researchers, respondents were largely able to prepare applications with- out additional support, with under 25 percent of respondents from the Fundamental Research and Vihren programs using such support (Figure 2.5). The Functional and Governance Analysis found that application processes are burdensome on beneficiaries (both for programs targeting researchers and companies), although processes vary in format and complexity depending on the implementing body and specific program. Lengthy and complex application forms are com- mon across programs, while collecting and providing supporting documentation is a larger chal- lenge for firm support programs. Supporting documentations can include financial statements, declarations of clean criminal records, tax liabilities, and offers from third parties for assets. High application costs can represent a barrier to participation in a program – particularly for SMEs and startups, which may not have the resources available to apply. There was no application process for the ICT and eHealth programs., so beneficiaries from those programs were not asked about application costs. Bulgaria: Efficiency Analysis of STI Programs 29 Figure 2.4. Application Costs by Program 12.000 4,0% 3,5% Average application costs Average Application Costs 10.000 (share of grant received) (constant 2020 BGN) 3,0% 8.000 2,5% 6.000 2,0% 1,5% 4.000 1,0% 2.000 0,5% 0 0% fr vihren nif dppi Average application costs Average Application cost (share of grant received) Figure 2.5. Share of Beneficiaries that Required Additional Support to Prepare their Applications by Program 100% Additional Support to Prepare Application 92% 90% Share of Beneficiaries that Required 84% 80& 70% 60% 50% 40% 25% 30% 16% 20% 10% 0% fr vihren nif dppi Source: World Bank. Note: Bars in purple are programs target researchers, and bars in blue target firms. There was no application process for the ICT and eHealth programs. The time needed to prepare applications was similar across programs. The number of working days needed to prepare applications was roughly the same across programs, ranging from 29 to 32 (Figure 2.6). When comparing the labor needed to prepare applications with the findings of the recent efficiency analysis done in Croatia in 2020, Bulgaria applicants generally used more work- ing days to prepare their applications – particularly firms (see Box 2.1). This may be indicative of more time-intensive application processes for Bulgarian firm support programs. Bulgaria: Efficiency Analysis of STI Programs 30 Figure 2.6. Time Needed to Prepare Applications by Program 35 Average Days Needed to Prepare Application 34 33 32 32 31 31 31 30 29 29 28 27 26 25 fr vihren nif dppi Source: World Bank. Note: Bars in purple are programs target researchers, bars in blue target firms. There was no application process for the ICT and eHealth programs. Box 2.1. Application costs of Croatian STI programs In an analysis of the efficiency of seven STI support programs in Croatia in 2020, the World Bank (2021b) found that application costs per project were generally higher for programs targeting firms, but that, on average, applica- tions to programs targeting researchers took more labor to prepare (Table 2.2). Average application costs represented between 1 and 7 percent of the average grant value, and applications took an average of between 17 and 30 working days to prepare. Table 2.2. Application costs and working days needed to prepare application for Croatian STI programs Research support programs Firm support programs Program IRP SIIF STRIP RP RS NSME-1 ICT-R Application costs as share of 2.0% 0.5% 0.6% 1.5% 0.9% 4.2% 6.6% total grant received Average working days to 30 29 25 24 17 24 27 prepare applications Source: World Bank 2021b. Note: ICT-R = Improving competitiveness and efficiency of through ICT, IRP = Installation Research Projects, NSME-1 = Innovations in newly-established SMEs – Phase 1, RP = Research Projects, RS = Research Scholarships, SIIF = Science and Innovation Investment Fund, STRIP = Strengthening Capacities for Research, Development and Innovation. Bulgaria: Efficiency Analysis of STI Programs 31 Due to the matching grant requirements, the average costs covered by beneficiaries were highest for firm respondents. Beneficiaries of firm-support programs invested more in their projects than those of research-support programs, largely due to the matching requirements of the DPPI and NIF programs2 (Figure 2.7). As discussed above, beneficiaries of firm-support programs reported higher application costs than researcher beneficiaries. The ICT and eHealth, programs are both sectoral programs of the National Science Programs 2018–22 portfolio, have a different design than traditional research grants: each program provides funding to a consortium of preselected research institutions in a specific scientific field. MoES delegates a portion of the administration and auditing of projects to the consortium, so some of the costs of implementing the overall pro- gram are borne by the beneficiaries; ICT and eHealth respondents also made higher contributions relative to respondents from other research support programs. Figure 2.7. Costs Covered by Beneficiaries by Program 300.000 Average beneficiary costs, 250.000 constant 2020 BGN 200.000 150.000 100.000 50.000 0 fr vihren ict ehealth nif dppi Source: World Bank. Application costs Contributions (cash and in kind) Management costs Note: Beneficiary costs include application costs and contributions (cash and in kind) from participating firms and institutions. Costs for ICT and eHealth also include costs associated with management and auditing of the beneficiary consortiums. 2 The NIF program funds between 25 and 80 percent of total project costs, depending on the type of project, size of firm, and other factors, while the DPPI program funds between 50 and 70 percent of total project costs, depending on the size of firm. Firms are responsible for funding the remainder of project costs. Bulgaria: Efficiency Analysis of STI Programs 32 Researcher respondents received the most grant funding per unit of beneficiary cost. Specifi- cally, beneficiaries from Vihren and Fundamental Research received the most funding relative to their costs. Vihren respondents received an average of 37 BGN in funding per BGN spent in ap- plication and contributions and Fundamental Research respondents received an average of 11 BGN for each BGN of cost. NIF and DPPI respondents received lowest levels of funding per unit of cost due to the matching funding requirement (Figure 2.8); the high matching grant require- ments are in line with the objectives of these programs, which aim to spur private sector invest- ment in industrial R&D. Figure 2.8. Ratio of Grant Funding to Beneficiary Cost by Program 40 Grant funding received to beneficiary 37 35 costs, constand 2020 BGN 30 25 20 13 15 11 10 5 5 1 2 0 fr vihren ict ehealth nif dppi Note: Beneficiary costs include application costs and contributions (cash and in kind) from participating firms and institutions. Costs for ICT and eHealth also include costs associated with management and auditing of the beneficiary consortium. Gray bars represent programs targeting firms. 2.2 Efficiency in Generation of Outputs There was large variation in the size of the selected programs, although the programs that dis- bursed the most funding also reached the largest numbers of beneficiaries. Fundamental Re- search and DPPI were the largest programs in terms of total funding contracted and disbursed (Figure 2.9), while Vihren and DPPI were the largest in terms of average grants disbursed (Figure 2.10). Fundamental Research and NIF were the smallest programs in terms of average grant dis- bursed, with an average disbursed grant amount of 96,626 BGN for FR respondents and 203,763 BGN for NIF respondents. All programs disbursed at least 75 percent of the funding contracted from 2016 to 2020, with the exception of Vihren (which only disbursed 40 percent of the funding contracted). Vihren projects have five-year implementation timelines and only began in 2019, so all Vihren projects will be active until 2023. Bulgaria: Efficiency Analysis of STI Programs 33 Figure 2.9. Total Disbursed Funding Size by Program 90 Total grants contracted and disbursed 80 (millions of BGN, 2020 constant) 70 60 50 40 30 20 10 0 fr vihren ict ehealth nif dppi Total Grants Contracted Grants Disbursed Figure 2.10. Average Disbursed Grant Size by Program 1.400 Average funding contracted and disbursed per projectThousand BGN, 2020 constant 1.200 1.000 800 600 400 200 0 fr vihren ict ehealth nif dppi Average Funding Contracted Average Funding Disbursed Source: World Bank. Note: Bars in purple are programs target researchers, bars in blue target firms. Bulgaria: Efficiency Analysis of STI Programs 34 The ICT and eHealth programs provided the most disbursed grants relative to the administrative costs of implementing the programs, although some of the costs of implementing and monitor- ing these programs are borne by the beneficiaries. For example, the ICT program transferred 544 BGN per every BGN spent in administrative costs, however, ICT respondents reported expenses of 4.8 percent of grant funding received for management and auditing of their projects as part of maintaining the beneficiary consortium. Therefore, the differences in the ratios are likely driven by the differences in program requirements and do not necessarily reflect differences in their ef- ficiency. NIF, Fundamental Research, and Vihren had the lowest ratio of disbursed grants to ad- ministrative costs (Figure 2.11). Figure 2.11. Ratio of Grants Disbursed to Administrative Costs 600 administrative cost, constand 2020 BGN 544 Grant funding disbursed per 500 400 345 300 199 200 100 40 15 11 0 fr vihren ict ehealth nif dppi Bulgaria: Efficiency Analysis of STI Programs 35 Outputs Achieved The ICT program achieved the highest numbers of outputs per project, but when considering outputs per unit of cost, Fundamental Research respondents were the most efficient at pro- ducing outputs (Figure 2.12 and Figure 2.13). The outputs tracked for each program depended on the ToC for each program (see Appendix A). In general, for research programs, outputs include capacity building (training activities and PhDs and postdoctoral researchers hired); project dis- semination (seminars, workshops, and conferences); and collaborations during implementation. For firm support programs, outputs generally include capacity building (improved capabilities of employees) and collaborations pursued during implementation. Figure 2.12. Outputs per Project by Program 40 35 30 Outputs per project 25 20 15 10 5 0 fr vihren ict ehealth nif dppi Source: World Bank. Capacity building Dissemination Collaborations Figure 2.13. Efficiency at Producing Outputs by Program 12 Outputs per 100,000 BGN in project cost 10 8 6 4 2 0 fr vihren ict ehealth nif dppi Source: World Bank. Capacity building Dissemination Collaborations Bulgaria: Efficiency Analysis of STI Programs 36 Capacity-Building Outputs Among research programs, ICT respondents produced the most capacity building outputs per project, while NIF respondents produced the most among firm support programs (Figure 2.14 and Figure 2.15). The relatively low numbers of capacity building outputs reported by Vihren re- spondents (particularly training outputs) are concerning given that the program focuses on de- veloping high quality research teams and that hiring and training young researchers is a core objective of the program. Figure 2.14. Capacity-Building Outputs per Project among Research Support Programs 12 Capacity-building outputs per project 10 8 6 4 2 0 fr vihren ict ehealth Source: World Bank. Training activities PhD students hired Postdoctoral researchers hired Figure 2.15. Capacity-Building Outputs per Project among Firm Support Programs 3,5 Capacity-building outputs per project 3,0 2,5 2,0 1,5 1,0 0,5 0 nif dppi Source: World Bank. Improved capabilities of employees Bulgaria: Efficiency Analysis of STI Programs 37 Dissemination Outputs eHealth and ICT respondents reported the most dissemination outputs per project, in the form of participation in the most seminars, workshops, and conferences. Seminars, workshops, and conferences attended abroad were more common across programs than those attended domes- tically (Figure 2.16). Figure 2.16. Outputs Related to Dissemination per Project by Program 16 Dissemination outputs per project 14 12 10 8 6 4 2 0 fr vihren ict ehealth Source: World Bank. Seminars, workshops and Seminars, workshops and conferences attended domestically conferences attended abroad Collaborations During Project Implementation The share of respondents that collaborated during implementation was generally low when com- pared STI programs from an efficiency analysis of STI program done in Croatia in 2020. (See Box 2.2.) The exception to this is respondents from the ICT and eHealth programs, which are programs where beneficiaries are members of consortiums that engage in collaborative project with each other, as well as with external partners. A lower share of respondents from the Fundamental Re- search and Vihren programs reported collaborations during implementation than the other pro- grams included in this analysis (Figure 2.17), which is concerning given that both programs have objectives related to increasing connections and collaboration activities of beneficiary research- ers and organizations. Respondents across programs tended to engage with research partners more often than industry partners. Low levels of collaboration may be due to several factors. Collaboration can carry a significant transaction cost to beneficiaries, and these costs may not be accounted for in the current pro- gram designs. Fundamental Research, in particular, has an average contracted grant size of only Bulgaria: Efficiency Analysis of STI Programs 38 118,657 BGN (or about €61,000), so beneficiaries may see little value in sharing their small grant award with one or more research partners. Additionally, respondents from multiple programs said they would like to see the types of partners allowed under the programs expanded (for example, several Fundamental Research respondents said they would like to be able to collaborate with non-profit organizations during implementation). Such changes may help boost the number of collaborations and share of beneficiaries that collaborate across programs. As discussed in Box 2.2, collaborations among several Croatian STI programs were boosted by the fact that partner- ships were mandatory for participating in the program; for Bulgarian research programs where increased connections and collaborations are an objective, including such a requirement may be useful in boosting collaborative activities. Figure 2.17. Collaboration by Type of Partner 100% share of corecpondents that participated in collaboration 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% fr vihren ict ehealth nif dppi Source: World Bank. research partners industry partnes total Bulgaria: Efficiency Analysis of STI Programs 39 Box 2.2. Collaboration During Implementation of Croatian Research Programs In their analysis of the efficiency of seven STI support programs in Croatia in 2020, the World Bank (2021b) found that, among researchers, most respon- dents pursued collaborative projects during implementation, averaging be- tween 1 and 3 collaborative projects per beneficiary (Table 2.3). It should be noted that partnerships were mandatory for two of these programs: the SIIF and STRIP programs required at least one collaborative partners, so a mini- mum of one collaborative project per beneficiary was expected. Table 2.3. Collaborations During Implementation of Croatian Research Programs Share of respondents that Share of respondents that Share of respondents collaborated with research collaborated with industry that engaged in any partners partners collaboration SIIF 75% 42% 83% STRIP 45% 45% 45% RP 69% 19% 73% IRP 66% 19% 69% Source: World Bank 2021b. Bulgaria: Efficiency Analysis of STI Programs 40 Firms that collaborated during project implementation reported more outcomes. Firms that col- laborated during implementation generally reported more outcomes than firms that did not col- laborate, including more prototypes, new-to-market and new-to-firm products and services, and more IP activities (Figure 2.18). However, these findings are not statistically significant due to the small number of firms that did not collaborate during implementation. Researcher respondents that collaborated during implementation produced more results (outputs plus outcomes) per project than those that did not but produced slightly fewer scientific publications per project than those respondents that did not collaborate during implementation. Figure 2.18. Outcomes Achieved by Collaboration outcomes per project 0,0 0,5 1,0 1,5 2,0 2,53,0 Prototype New to market prducts/services New to firm products/services Upgraded products/services Upgraded processes Patents New processes Patents Granted Other IP activities Source: World Bank. Collaborated during implementation No collaboration during implementation Note: N=91 for firms that collaborated during implementation; N=29 for firms that did not collaborate during implementation Bulgaria: Efficiency Analysis of STI Programs 41 2.3 Efficiency in Achievement of Outcomes Intended outcomes differ from program to program, depending on their targeted beneficiaries and objectives. The Fundamental Research and Vihren programs are scientific research grant programs and therefore aim at achieving outcomes focused on publications and increased col- laborations after project implementation. In contrast, ICT and eHealth are applied research pro- grams and their intended outcomes focused on new technology and software development. NIF and DPPI are firm support programs and focus on the development and/or adoption of new prod- ucts and processes, increased revenues, and increased employment among beneficiaries. Table 2.4 shows the intended outcomes tracked for each program. Table 2.4. Intended Outcomes across Programs Outcome FR Vihren ICT eHealth NIF DPPI Collaborative projects after implementation Technology transfer activities (transfer agreements, new enterprises, or spin-offs) Intellectual Property (patents, industrial designs, copyrights) New prototypes *Scientific publications in peer-reviewed journals *New software development *New technology development **Upgraded products, processes, or services **New products, processes, or services **Companies that adopted a new technology **Companies that increased sales **Companies that expanded to new markets **Companies that improved their export performance **Additional workers hired Source: Author’s elaboration * Outcomes tracked in beneficiary survey for researchers only ** Outcomes tracked in beneficiary survey for firms only Bulgaria: Efficiency Analysis of STI Programs 42 Publications Bulgarian researchers produced similar numbers of publications per project as Croatian re- searchers. Bulgarian research programs reported an average of between 2 and 25 publications per project (Figure 2.19), while the Croatia research programs analyzed in World Bank (2021b) found that research programs reported an average of between eight and 23 publications produced per project (Figure 2.20). Figure 2.19. Publications per Project by Program, Bulgaria 30 24,9 Publications in peer-reviewed 25 journals per project 20 15 11,8 10 6,4 5 2,3 0 fr vihren ict ehealth Source: World Bank. program Figure 2.20. Publications per Project by Program, Croatia 30 Publications in peer-reviewed 25 23 journals per project 19 20 15 11 11 10 8 5 0 IRP RP RS SIIF STRIP Source: World Bank 2021b. program Bulgaria: Efficiency Analysis of STI Programs 43 There was substantial variation across programs in the number of publications produced per project. ICT and eHealth respondents produced the most publications in peer reviewed journals per project, while Vihren respondents produced the least (Figure 2.19). The very high number of publications per project for ICT is notable, given that the primary focus of this programs is applied research and producing new software and technologies for public use. The low numbers of publi- cations produced by Vihren respondents is likely due to the fact that all Vihren projects began in 2019 and were only two years into a five-year project cycle at the time the program beneficiaries were surveyed – it would be expected that Vihren beneficiaries will produce more publications per project by the time of project completion. However, publications reported by respondents generally had fewer year-normalized citations3 than the average Bulgarian publication indexed in the Web of Science. (See Figure 2.21.) This dif- ference raises concerns about the quality of research being conducted by beneficiaries of the four research programs included in this analysis. All four programs have research excellence as an objective, which is primarily measured through the citations received by publications resulting from funded research. Research beneficiaries were asked to list the five most important publica- tions related to their projects for a bibliometric analysis. Of the programs targeting researchers, respondents provided detailed information on a total of 666 publications, of which the World Bank team were able to find 52 percent that were indexed in Web of Science and 43 percent that were in journals with impact factor4. Publications in Web of Science for all Bulgarian authors from 2016 to 2020 averaged 2.1 year-normalized citations per publication, while indexed publications produced by Fundamental Research respondents had 1.7 year-normalized citations, publications reported by ICT respondents had 1.3 per year, Vihren publications had 0.6 per year, and indexed eHealth publications had not yet received any citations. Figure 2.21. Year-Normalized Citations per Publication Year-normalized citations 0,0 0,5 1,0 1,5 2,02,5 National Average (2016-2021) FR ICT VIHREN EHEALTH Source: Elaboration based on data from Web of Science. 3 Year-normalized citations are calculated as the number of citations of a given publication divided by the publication’s age (in years). 4 Impact factor is a metric used by Web of Science based on ratio between citations and recent citable items published in a given publication. Bulgaria: Efficiency Analysis of STI Programs 44 Publications in the fields of physics, chemistry, and technical sciences (i.e., engineering) were generally the most impactful, in terms of average year-normalized citations per publication and share of publications in journals with impact factor (Table 2.5). Publications in the fields of social sciences, medicine, and agricultural science had the lowest average year-normalized citations per publication. However, it should be noted that publications in some scientific disciplines (partic- ularly biological sciences) tend to attract more citations when compared to social sciences and humanities (Harzing, 2010), which could explain some of the differences observed across fields. Table 2.5. Publication Impact by Scientific Field Total Share of Share of publications in Average publications publications journals with year-normalized Scientific Field listed indexed in WoS impact factor citations Agricultural studies 51 35% 24% 0.85 Biological Sciences 66 47% 44% 0.90 Chemistry 51 71% 65% 2.12 Earth science 31 55% 52% 1.09 Humanities 40 25% 18% 1.05 Mathematics Sciences and Informatics 126 55% 43% 0.93 Medicine 41 59% 44% 0.81 Physics 92 83% 75% 2.96 Social Sciences 69 32% 10% 0.59 Technical Sciences 99 51% 40% 2.02 Source: World Bank., Web of Science Collaborations after Project Implementation Fundamental Research and NIF respondents reported the most collaborations formed after project implementation. Collaborations after project completion (a proxy for improved connec- tions between beneficiaries and other domestic and international STI actors) were an intended outcome of the Fundamental Research and NIF programs, and respondents from both programs reported an average of over two new collaborations after implementation (Figure 2.22). Funda- mental Research respondents tended to form collaborations with research partners, which is un- derstandable given that the program focuses on basic research and increasing connections be- tween researchers and research institutions, while NIF respondents formed more collaborations with industry partners (Figure 2.23). Collaborations after implementation were not an intended outcome of the DPPI program, but DPPI respondents reported an average of 1.5 new collabora- tions after implementation, suggesting that the program encourages collaboration activity among Bulgaria: Efficiency Analysis of STI Programs 45 beneficiaries without intending to. All projects for the Vihren, ICT, and eHealth programs are still active, so the collaborations after implementation metric was not tracked for those programs. Figure 2.22. Collaborations after Implementation 3,5 Average number of collaborations 3,0 2,5 2,0 1,5 1,0 0,5 0,0 fr nif dppi Source: World Bank. With Research Partners With Firm Partners Note: The figure denotes the number of research collaborations per respondents and share of respondents that engaged in collaborations after project completion (FR n=177, NIF n=18, DPPI n=65). All projects for the Vihren, ICT, and eHealth programs are still active. Figure 2.23. Collaborations after Implementation among Respondents with Completed Projects 100% Share of respondents with completed projects 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% fr nif dppi Source: World Bank. With Research Partners With Firm Partners total Note: The figure denotes the number of research collaborations per respondents and share of respondents that engaged in collaborations after project completion (FR n=177, NIF n=18, DPPI n=65). All projects for the Vihren, ICT, and eHealth programs are still active. Bulgaria: Efficiency Analysis of STI Programs 46 Domestic partners were the most common type of research partner, while domestic and foreign partnerships were most common among private sector collaborations. Among collaborations with researcher partners formed after project implementation, collaborations with domestic research partners were the most common for Fundamental Research and DPPI respondents, followed by foreign research partners. Equal shares of NIF respondents engaged with domestic, foreign, and diaspora researchers (Figure 2.24). A slightly higher share of NIF respondents engaged with for- eign industry partners than domestic industry partners, while the reverse was true for DPPI and Fundamental Research respondents (Figure 2.25). Figure 2.24. Types of Research Partners by Program 100% Share of respondents with completed projects 80% 60% 40% 20% 0% fr nif dppi Source: World Bank. Domestic research partners Foreign research partners Diaspora research partners Note: These figures show the share of respondents with a completed project that engaged in collaborations with research partners and industry partners after project completion (FR n=177, NIF n=18, DPPI n=65). All projects for the Vihren, ICT, and eHealth programs are still active. Figure 2.25. Types of Industry Partners by Program 100% Share of respondents with a completed project 80% 60% 40% 20% 0% fr nif dppi Source: World Bank. Domestic research partners Foreign research partners Diaspora research partners Note: These figures show the share of respondents with a completed project that engaged in collaborations with research partners and industry partners after project completion (FR n=177, NIF n=18, DPPI n=65). All projects for the Vihren, ICT, and eHealth programs are still active. Bulgaria: Efficiency Analysis of STI Programs 47 Intellectual Property Firm support program respondents reported more IP activity per project than research support program respondents. Firm respondents generally reported more patents granted and patent applications than researchers, which is expected given the different objectives of research and firm support programs (Figure 2.26). Fundamental Research and Vihren respondents reported moderate levels of IP generation relative to the other programs, even though IP generation is not an intended outcome of those programs. The ICT and eHealth programs reported no IP activity; while they are applied research programs focused on developing new technologies and tools for public use, respondents’ projects only began in 2019 and it would be expected that the project generate patents and patent applications later in their implementation. Figure 2.26. IP Activity per Project by Program 1,0 0,9 0,8 Patent generation per project 0,7 0,6 0,5 0,4 0,3 0,2 0,1 0,0 fr vihren ict ehealth nif dppi Source: World Bank. Patents granted Patent applications Bulgaria: Efficiency Analysis of STI Programs 48 New Software and Technologies ICT and eHealth respondents produced the most new software development and new technology development per project, which is to be expected given that they are applied research programs that focus on the development of new digital tools and platforms for public use. Fundamental Research and Vihren programs also produced some new software and new technology outcomes, although these were not intended outcomes of those programs (Figure 2.27). These metrics were not tracked for firm support programs. Figure 2.27. New Technology and New Software Development per Project by Program 2,5 2,4 2,3 Outcomes per 100,000 BGN 2,0 1,9 in program cost 1,5 1,0 0,3 0,3 0,5 0,5 0,1 0,1 0,0 fr vihren ict ehealth Source: World Bank. New software development New technology development Note: New software development and new technology development was only tracked on researcher surveys New or Upgraded Products, Services, or Processes DPPI respondents adopted more technologies per project than NIF respondents. Specifically, DPPI respondents reported more outcomes related to technology adoption (new-to-firm prod- ucts and services) and upgraded products, services, and processes per project than NIF respon- dents (Figure 2.28). This understandable given that NIF focuses exclusively on technology creation, while DPPI’s objectives include technology creation, adoption, and upgrading. When compared to the results of an analysis of the efficiency of Croatian innovation programs, DPPI respondents adopted roughly similar numbers of new-to-firm products or services as the Croatian Innovations in Newly Established SMEs Program (NSME-1) program and fewer than the Improving Competi- tiveness and Efficiency of SMEs in ICT (ICT-R) program (see Box 2.3). Bulgaria: Efficiency Analysis of STI Programs 49 Figure 2.28. Technology Adoption per Project by Program 2,0 1,9 1,8 1,6 Average outcomes per project 1,4 1,2 1,2 1,2 1,0 1,0 0,8 0,5 0,5 0,6 0,4 0,2 0,0 nif dppi Source: World Bank. New to firm products or services Upgraded products or services Upgraded processes NIF respondents reported more new processes per project than DPPI respondents, while DPPI respondents reported more new-to-market products or services. Both programs have objec- tives related to the creation of new products, services, and processes – the major difference be- tween the two is that NIF focuses on early-stage development projects (industrial research and experimental development), while DPPI funds projects that are closer to the market (Figure 2.29). Compared to the results of the Croatia efficiency analysis of innovation programs, DPPI and NIF respondents produced similar numbers of new-to-market products or services as the Croatian innovation support programs but fewer new processes (Box 2.3). Figure 2.29. New Products, Services, and Processes per Project by Program Average outcomes per project 2,5 2,0 1,5 1,0 0,5 0 New to market products or services New processes Source: World Bank. nif dppi Bulgaria: Efficiency Analysis of STI Programs 50 Box 2.3. New Products, Services, and Processes of Croatian Firm-Support Programs In their analysis of the efficiency of two innovation support programs in Cro- atia in 2020, the World Bank (2021b) found that the analyzed programs devel- oped an average of 1 to 4 new products, services, or processes per project. On average, respondents from the ICT-R program developed more products or services per project that were new to the firm than respondents of the NSME-1 program, but respondents from the NSME-1 program developed more new-to-market products and services. These results are due to the differences in the design of the two programs, with NSME-1 being more fo- cused on commercializing new products or services, while ICT-R supported technology upgrades. Table 2.6. New product, service, and process outcomes of Croatian innovation support programs ICT-R NSME-1 New-to-firm products or services 4 2 New-to-market products or services 1 2 New processes 3 1 Source: World Bank 2021b. 2.4 Perceived Program Quality Respondents’ perceptions of their programs’ application and selection processes revealed areas of potential improvement. Respondents were generally satisfied with the ease of obtaining program information, the clarity of program objectives, and the flexibility of rules for non-compliance, with over 70 percent of respondents in the Fundamental Research, Vihren, NIF, and DPPI programs5 saying they were satisfied with these aspects of their projects. However, respondents were less satisfied with the ease of the application process, the transparency of the selection process, and the availability of feedback on project selection – NIF and DPPI beneficiaries in particular were largely dissatisfied with these aspects of the program, with under 65 percent of respondents re- 5 The ICT and eHealth programs have no application process, so beneficiaries of those programs were not asked about their percep- tions of program application and selection. See Sections 5 and 6 for more information about how beneficiaries were selected for those programs. Bulgaria: Efficiency Analysis of STI Programs 51 porting satisfaction with these aspects of their projects (Figure 2.30). Vihren respondents noted that the program uses a common selection panel for selecting projects for all the physical scienc- es (including astronomy, chemistry, materials science, and physics), yet project proposals may be hard to compare across all of these disciplines. The Functional and Governance Analysis report included several findings that supported the respondents’ perceptions of program application and selection processes (see Box 2.4). Figure 2.30. Satisfaction with Application Process by Program Share of respondents satisfied with areas of application process 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Ease of obtaining information about the program Clarity of program objectives Ease of application procedure Transparency of selection process Availability of feedback on project selection Flexibility of rules for non-compliance Source: World Bank. FR VIHREN NIIF DPPI Note: Figure shows percentage of respondents who agreed or strongly agreed with the statements asking about their satisfaction with selected stages of the application process. There was no application process for the ICT and eHealth programs. Bulgaria: Efficiency Analysis of STI Programs 52 Box 2.4. Application and Selection Processes of Bulgarian STI Support Programs Several key findings from phase 2 of this project, the Functional and Gover- nance Analysis, provide evidence that support beneficiaries’ perceptions of program application and selection processes: Application Application processes are burdensome on beneficiaries (both for programs targeting researchers and companies), although processes vary in format and complexity depending on the implementing body and specific program. Lengthy and complex application forms are common across programs, while collecting and providing supporting documentation is a larger challenge for firm support programs. Supporting documentations can include financial statements, declarations of clean criminal records, tax liabilities, and offers from third parties for assets. Because of the complexity and time needed for applications, many applications to firm support programs use external con- sultants to prepare their applications, as they either do not understand or do not have the time and capacity to complete the applications on their own. For programs under Bulgarian operational programmes (including DPPI, a procedure under OP Innovation and Competitiveness), applications are done entirely electronically through the UMIS system, and templates and guide- lines are provided to applicants through the Unified Management Informa- tion System (UMIS) portal. While all OP applications are done electronical- ly, OPIC beneficiary companies have still complained about the number of supporting documents and the amount of financial information requested, and many companies use consultants to help them through the application process. Nationally-funded instruments (i.e., Fundamental Research, Vihren, and NIF) do not use the online UMIS portal, and their application processes tend to be more burdensome. For example, NIF requires applicants to provide a detailed breakdown of every participating researchers’ hours of work for the duration of the project and beneficiaries of NSF must provide financial justification of requested funds by expense category for each stage of the project as part of their application. Bulgaria: Efficiency Analysis of STI Programs 53 Selection A 2015 peer review of the Bulgarian STI system (European Commission, 2015) found a strong need to improve processes for the evaluation and funding of project proposals, due to the lack of confidence of the research community in the fairness of funding allocations and in the established peer review sys- tem for the evaluation of projects. While most implementing bodies make use of external reviewers to review project applications, the DG OPIC (which implements DPPI) relies heavily on internal panels to evaluate proposals. Internal reviews lack transparency and can be biased towards an institution’s cultural preferences, and for these reasons are generally not in line with good practices. The use of foreign ex- pert reviewers is also not common, as most implementing bodies that use external experts rely exclusively on domestic expert evaluators; NSF (which implements Fundamental Research and Vihren) is one of the only imple- mentors that uses foreign experts to evaluate applications. Relying exclu- sively on domestic experts is not in line with best practices, particularly in a small country like Bulgaria. Scientific and technical communities are small, which raises the risk of conflicts of interest and biased evaluations, and the expertise available for specific research areas and technologies may not be available domestically. Further, some instruments (particularly those targeting firms) suffer from overly generic selection criteria that do not target projects with specific de- sirable characteristics, which can make it difficult for evaluation panels to make consistent project award decisions; this can, in turn, lead to the per- ception of an unfair or untransparent awards process. Source: World Bank 2021a. Respondents were generally satisfied with the support provided by the programs during imple- mentation. Respondents were particularly satisfied with the administrative support provided by the programs and the timeliness of financial support, where more than 70 percent of respondents in all programs agreed or strongly agreed that they were satisfied with this category of support (Figure 2.31). Bulgaria: Efficiency Analysis of STI Programs 54 Figure 2.31. Satisfaction with Implementation by Program Share of respondents satisfied with implementation support 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Accessibility of administrative support Sufficiency of administrative support provided Timeliness of financial support FR VIHREN ict ehealth NIIF DPPI Note: Figure shows percentage of respondents who agreed or strongly agreed with the statements asking about their satisfaction with administrative and financial support provided. Beneficiaries from some programs had issues with monitoring requirements. Respondents from several programs were dissatisfied with financial and technical reporting requirements, as well as data protection practices. Expert feedback from monitoring was an area of strength, with over 85 percent of respondents across programs saying that they were satisfied with the feedback they re- ceived. However, respondents from Fundamental Research, NIF, and DPPI were largely dissatisfied with financial reporting requirements, where less than 60 percent of respondents were satisfied with these requirements. Fundamental Research and Vihren respondents were also dissatisfied with technical monitoring requirements, with less than 65 percent reporting that they were sat- isfied with these requirements. Data protection practices was an area of concern for Vihren and ICT respondents, with only half of respondents saying they were satisfied with program practic- es in this area (Figure 2.32). Fundamental Research respondents noted that even small changes to project plans require NSF approval, and that NSF requires certified and translated copies of all eligible expenses, including things like plane tickets. Some NIF respondents felt that the pro- gram’s financial reporting requirements were not clear enough, yet their companies were at risk of financial corrections is they violated the requirements. The Functional and Governance Anal- ysis component of this PER STI project found that reporting requirements for projects funded through operational programmes (such as DPPI) where generally in line with good practice, while nationally funded programs had areas for improvement (see Box 2.5). Bulgaria: Efficiency Analysis of STI Programs 55 Figure 2.32. Satisfaction with Monitoring by Program Share of respondents satisfied with monitoring processes 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Appropriate financial reporting requirements Appropriate monitoring requirements (e.g. narrative, indicators, etc.) Appropriate expert feedback from monitoring (visits, reports, discussions) Satisfactory data protection practices Source: World Bank. FR VIHREN ict ehealth NIIF DPPI Note: Figure shows percentage of respondents who agreed or strongly agreed with the statements asking about their satisfaction with moni- toring processes. Box 2.5. Reporting Requirements of Bulgarian STI Support Programs The Functional and Governance Analysis report found that reporting pro- cesses for programs outside of the operational programmes had several key areas of that could be improved: For nationally funded programs (i.e., Fundamental Research, Vihren, ICT, eHealth, and NIF), there is no online portal for submitting reports and submission is done via email or by mail- ing in paper forms. Fundamental Research and Vihren beneficiaries must submit interim and final technical and financial reports to NSF, and the financial reports must include cer- tified copies of the supporting documents for the incurred direct eligible costs. NIF bene- ficiaries submit technical and financial reports after each stage of their project (projects can have up to three stages), and while beneficiaries are provided with standardized forms for financial reports, there is no template for the technical reports. Reporting is much easier for projects funded under operational programmes (such as DPPI), where beneficiaries submit reports in standard, preloaded forms through the UMIS system. Source: World Bank 2021a. Bulgaria: Efficiency Analysis of STI Programs 56 The availability of financial and human resources were the most commonly cited factors for proj- ect success across all programs. The availability of research infrastructure was reported as an important factor by 50 percent of respondents from the four research support programs, indicat- ing that the provision of access to research infrastructure may be more important to research programs than firm support programs (Figure 2.33). Figure 2.33. Project Success Factors by Program 100% Share of respondents reporting key success factor 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% fr vihren ict ehealth nif dppi Source: World Bank. Availability of Availability of Support of the Availability of research financial resources human resources research institution infrastructure Bulgaria: Efficiency Analysis of STI Programs 57 Project results largely matched respondents’ expectations. Between 60 and 80 percent of re- spondents with completed projects across programs indicated that project results matched their expectations. Fundamental Research had the most respondents with results exceeding expec- tations (21 percent), while NIF had the most with results below their expectations (28 percent) (Figure 2.34). The higher share of NIF respondents whose projects failed to match expectations is likely due to the early-stage nature of the projects supported by the program, which are inher- ently riskier than academic research projects or later-stage commercialization projects. Figure 2.34. Share of Respondents Who Indicated that Project Results Matched Expectations by Program 100% 6% 11% 90% 21% 80% Share of respondents 70% 60% 61% 80% 50% 40% 75% 30% 20% 28% 10% 14% 5% 0% FR nif dppi Source: World Bank. Below expectations Matched expectations Above expectations Note: All projects for Vihren, ICT, and eHealth are still active. Whereas this section presented key results across programs, the following six sections of this report provide a detailed review of the findings for each of the six programs included in this anal- ysis. This includes a description of each program and its beneficiaries, and findings on the pro- grams’ inputs, outputs, outcomes, and beneficiary perceptions. Bulgaria: Efficiency Analysis of STI Programs 58 3. Fundamental Research Bulgaria: Efficiency Analysis of STI Programs 59 3 Fundamental Research Program Summary The Fundamental Research program is the primary mechanism for funding basic research in Bulgaria, making it an important source of funding for public sector researchers and research institutions. The Fundamental program is designed and implemented by the NSF and provides grant funding for basic research projects in one of ten scientific areas. The program focuses on im- proving the quantity and quality of basic research produced by the Bulgaria research sector, and thus has intended outcomes related to scientific pub- lications in peer reviewed journals and collaborative projects formed after project completion. External services made up over 90 percent of the administrative costs of implementing the program from 2016 to 2020, while personnel costs only represented two percent of total administrative costs, an arrangement in costs that is likely due to the to the low number of full-time staff at the NSF. This cost structure is not necessarily inefficient but increasing number of full-time program staff would increase the internal capacity of NSF (identified as a major challenge in previous components of this PER STI project) and could lead to more effective program delivery. Scientific publications were the most common outcome reported by respondents, and Fundamental Research publications performed the best among the programs included in this study along several bibliometric indicators – however, they still had fewer year-nor- malized citations than the average Bulgarian publication indexed in the Web of Science from 2016 to 2020. Seminars, workshops, and conferences were the most common output of Fundamental Research respondents, while collaborative activities were somewhat low compared the results reported by Croatian research programs from a similar analysis in 2020. This is an area of concern because the program has objectives related to intensifying connections between science, education, business, and other innovation actors. Respondents largely felt that the outcomes of their projects matched their expectations and were generally satisfied with program processes, although some had issues with the transparency of the selection process and technical and financial reporting requirements of the program. Bulgaria: Efficiency Analysis of STI Programs 60 The Fundamental Research program began providing grants in 2016 and has issued annual calls for proposals every year since then, funding a total of 664 projects for 79 million BGN grant fund- ing (64 million of which had been disbursed by the end of 2020) between 2016 and 2020. The principal investigators (or alternative points of contact) for all 664 projects funded by the program from 2016 to 2020 were surveyed for this analysis, with 414 beneficiaries responding to the survey for a response rate of 62 percent. Respondents represent 62 percent of the con- tracted funding for the program over the 2016-2020 period, with an average grant size of 119,106 BGN. Respondents’ projects were spread across ten scientific fields, led by technical sciences (i.e., engineering disciples) (representing 17 percent of respondents’ projects), biological scienc- es (12 percent) and physics (11 percent), while agricultural sciences, medicine, chemistry, social sciences, humanities, mathematics, and earth sciences each made up 10 percent or less of re- spondents’ projects (Figure 3.1). Figure 3.1. Fundamental Research Respondents’ Projects by Scientific Field Share of respondents 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% Technical Sciences 17% Biological Sciences 12% Physics 11% Agricultural studies 10% Medicine 10% Chemistry 10% Social Sciences 9% Humanities 9% Mathematics Sciences and Informatics 8% Earth science 4% Source: World Bank. 3.1 Efficiency in Use of Inputs Disbursed grants made up the largest share of FR program costs, which peaked in 2019. Grants distributed to beneficiaries represented 94 percent of the costs of the program from 2016 to 2020, while administrative costs made up the remaining six percent (Figure 3.2). Administrative costs have remained relatively constant from 2016–20, which grant disbursements were highest in 2018 and 2019 (Figure 3.3). Bulgaria: Efficiency Analysis of STI Programs 61 Figure 3.2. Fundamental Research Program Costs by Category, 2016–20 6% 94% Administrative Costs Source: World Bank. Grants Distributed Figure 3.3. Fundamental Research Program Costs by Year, 2016–20 20 Million BGN, 2020 constant 15 10 5 0 2016 2017 2018 2019 2020 Source: World Bank. Administrative Costs Grants Distributed Administrative Costs External services, in the form of contracts to technical experts to evaluate proposals, monitor projects, and provide advisory support, made up over 90 percent of program costs, while per- sonnel costs only represented two percent of total administrative costs (Figure 3.4). Costs for ex- ternal services and fixed costs were lower in 2018 and 2019 – years where the program awarded lower numbers of projects – before climbing again in 2020, while personnel costs were relatively constant from 2016 to 2020 (Figure 3.5). The program’s administrative costs were highest in 2020. The large share of costs for external services and very small share for personnel is likely due to the low numbers of full-time staff at the NSF (which has a total of 10 full-time staff members for implementing its entire portfolio of research programs). This arrangement in costs is likely done to compensate for the lack of staff at NSF. From the perspective of this efficiency analysis, this is Bulgaria: Efficiency Analysis of STI Programs 62 not necessarily an inefficient or ineffective arrangement; however, component 2 of the PER STI project, the Functional and Governance Analysis, found that the lack of staff has negatively im- pacted the implementation of programs at NSF in several areas, including knowledge manage- ment and M&E. Figure 3.4. Fundamental Research Program Administrative Costs by Category, 2016–20 2% 7% 91% Personnel Fixed Costs Source: World Bank. External Services Figure 3.5. Fundamental Research Program Administrative Costs by Year, 2016–20 1.200.000 1.000.000 Constant 2020 BGN 800.000 600.000 400.000 200.000 0 2016 2017 2018 2019 2020 Source: World Bank. Personnel Fixed Costs External Services Bulgaria: Efficiency Analysis of STI Programs 63 Over half of personnel costs went toward implementation, while 41 percent went to program design, and seven percent went to M&E (Figure 3.6). Most of the monitoring of Fundamental Re- search projects is done by external experts, which accounts for the low share of personnel costs in this area. Similarly, the program had no personnel costs for the evaluations of project proposals – this is also done by external experts contracted by the program. Personnel costs have increased yearly since 2016 (Figure 3.7). Implementation costs were highest in 2019 and 2020, which con- tributed to the increase in overall personnel costs in those years. Figure 3.6. Fundamental Research Program Personnel Costs by Category, 2016–20 7% 52% 41% Design Implementation Source: World Bank. M&E Figure 3.7. Fundamental Research Program Personnel Costs by Year, 2016–20 25.000 20.000 Constant 2020 BGN 15.000 10.000 5.000 0 2016 2017 2018 2019 2020 Source: World Bank. Design Implementation M&E Bulgaria: Efficiency Analysis of STI Programs 64 Office equipment and ICT costs made up half of the fixed costs for the program. (See Figure 3.8.) Overall fixed costs were highest in 2017 and 2020. Costs related to office equipment and ICT were highest in 2020, while costs related to goods and services were highest in 2016 and 2017 (Figure 3.9). Figure 3.8. Fundamental Research Program Fixed Costs by Category, 2016–20 0,34 0,50 Office equipment and ICT Office space Source: World Bank. 0,17 Goods and Services Figure 3.9. Fundamental Research Program Fixed Costs by Year, 2016–20 100.000 80.000 Constant 2020 BGN 60.000 40.000 20.000 0 2016 2017 2018 2019 2020 Source: World Bank. Office equipment and ICT Office space Goods and Services Bulgaria: Efficiency Analysis of STI Programs 65 Almost 70 percent of costs for external services were related to contracting experts for the eval- uation of project proposals, while 27 percent were related to contracting experts for the moni- toring and evaluation of projects. NSF also utilizes external experts on several advisory commis- sions that provide guidance on its programming, which represented a small share of the costs for external services (Figure 3.10). Costs for experts for the evaluation of project proposals scale directly with the number of applications received in a given year, which were highest in 2016 and 2017. Costs for contracting experts for the monitoring and evaluation have risen sharply since 2018, making up over half of external services costs for the program in 2020, the year that costs for external services were highest overall (Figure 3.11). It should be noted that data were not avail- able for costs related to experts for M&E in 2017, so external services costs were likely higher in that year and overall than what is reported here. Figure 3.10. Fundamental Research Program Costs for External Services by Category, 2016–20 5% 0% 27% 68% Experts for evaluation of proposals Experts for M&E Source: World Bank. Advisory Note: Data were not available for costs related to experts for M&E in 2017. Other Figure 3.11. Fundamental Research Program Costs for External Services by Year, 2016–20 100.000 80.000 Constant 2020 BGN 60.000 40.000 20.000 0 2016 2017 2018 2019 2020 Source: World Bank. Experts for evaluation of proposals Experts for M&E Advisory Other Bulgaria: Efficiency Analysis of STI Programs 66 The total administrative costs of implementing the program from 2016 to 2020 was 4.3 million BGN, with an average cost per project of 6,481 BGN (Table 3.1). The average personnel cost per project of 119 BGN and average external services costs per project were 5,877 BGN show that much of the program implementation is done by external experts and consultants. An organization like NSF would naturally rely to some extent on external evaluators to select projects and technical experts to monitor progress, but the extremely low personnel cost per project may indicate the outsourcing of project implementation to an unusual degree. NSF could consider whether it may ultimately be more effective to allocate more resources to full-time staff and fewer to external services. This may not necessarily lead to cost savings for the program, but it would increase the capacity of NSF as an implementor and allow the program to retain more institutional knowledge. Table 3.1. Fundamental Research Administrative Costs Cost category Total costs Cost per project Personnel 78,826 BGN 119 BGN Fixed 321,933 BGN 489 BGN External services 3,902,639 BGN 5,877 BGN Source: World Bank. Note: All amounts in constant 2020 BGN. Costs Covered by Beneficiaries The application process for Fundamental Research is the quickest and cheapest among programs included in this analysis. (It should be noted that the program is also the smallest by average con- tracted grant size.) Respondents reported an average application cost of 2,258 BGN and spent an average of 29 working days preparing their application. Over 60 percent of respondents spent less than 1,000 BGN preparing their application, while less than 10 percent spent more 3,000 BGN (Figure 3.12). Two thirds of respondents spent less than 30 working days preparing their applica- tions, while over 20 percent spent more than 40 working days (Figure 3.13). Figure 3.12. Cost to Prepare Applications for the Fundamental Research Program Cost to prepare application (Share of respondents) 0% 20% 40% 60%80% 1 – 1000 BGN 61% 1001 – 3000 BGN 30% More than 3000 BGN 9% Source: World Bank. Note: Data were not available for costs related to experts for M&E in 2017. Bulgaria: Efficiency Analysis of STI Programs 67 Figure 3.13. Time to Prepare Applications for the Fundamental Research Program Working days to prepare application (Share of respondents) 0% 10% 20% 30% 40%50% Less than 20 days 40% 21-30 days 27% 31-40 days 13% More than 40 days 21% Source: World Bank. Note: Data were not available for costs related to experts for M&E in 2017. A large majority (84 percent) of respondents did not use external support (such as legal, ac- counting, or other services) to prepare their applications. Respondents’ ability to complete their applications without external support likely contributed to the relatively low cost of application preparation when compared to other programs included in this analysis. The most common ex- ternal supports used were accountants (needed by 11 percent of respondents), administrative assistants (9 percent), and expert consultants (7 percent) (Figure 3.14). Figure 3.14. Beneficiaries Using External Support in Preparing Applications for the Fundamental Research Program Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80%90% Accountant 11% Administrative assistant 9% Experts / consultants 7% Lawyer or legal consultant 2% No additional support required 84% Source: World Bank. Although the Fundamental Research program has no requirements for matching contributions from beneficiaries, respondents reported an average of 5,600 BGN in-kind and 600 BGN cash con- tributions to the implementation of their projects. (See Figure 3.15.) In-kind contributions were the most common contribution by share of respondents, with 55 percent of respondents making in kind contributions, 39 percent of respondents making no contributions, and 6 percent mak- ing both cash and in-kind contributions (Figure 3.16). In-kind contributions for research projects could take the form of researcher salaries and the usage of facilities, laboratories, and equipment. Bulgaria: Efficiency Analysis of STI Programs 68 Figure 3.15. Value of Contributions to Fundamental Research Program Projects Average contributions per project 0 2.000 4.0006.000 Average cash 600 contributions Average in kind 5.686 contributions Source: World Bank. Figure 3.16. Proportions of Contributions to Fundamental Research Program Projects 6% 55% 39% No contributions Cash 0% Source: World Bank. In kind Note: Data were not available for costs related to experts for M&E in 2017. Both cash and in kind On average, the costs covered by the program were 12 times higher than the costs covered by respondents. The average cost covered by the program, including disbursed grants and admin- istrative costs, was 103,107 BGN, while the average cost covered by beneficiaries was 8,544 BGN. Bulgaria: Efficiency Analysis of STI Programs 69 3.2 Efficiency in Generation of Outputs For every BGN of administrative costs, beneficiaries received an average of 15 BGN in disbursed grants. The total program cost from 2016 to 2020 was 68.5 million BGN, or 103,107 BGN per proj- ect. This includes 64.2 million in disbursed grants to beneficiaries and 4.3 million in administra- tive costs (Table 3.2). The program did not include any nonfinancial or indirect financial support to beneficiaries. Table 3.2. Fundamental Research Program Costs, 2016–20 Total Average per project Grant funding disbursed 64,159,408 BGN 96,626 BGN Administrative and operating costs 4,303,400 BGN 6,481 BGN Total program cost 68,462,808 BGN 103,107 BGN Source: World Bank. Note: All amounts in constant 2020 BGN. Respondents reported researchers’ salaries and travel as the largest expenditures of their project by cost category (Figure 3.17). These expenditure patterns are likely influenced by the program’s regulations, which state that a maximum of 30 percent of the project budget can be spent on researcher salaries. In the first cohorts of projects funded by the program, NSF observed that al- most all project funds were being spent on researcher salaries, so a cap on salary expenditures was introduced. Additionally, researchers in Bulgaria are typically not provided funding for trav- el (for conference, meetings, etc.) by their institutions, so Fundamental Research beneficiaries likely try to use as much funding as allowed by the program (up to 30 percent of project budget) to participate in such activities that they otherwise would not have the resources for. NSF does not allow for grant funding to be used for IP-generation or commercialization activities, which can explain why there were no expenses associated with such activities. Bulgaria: Efficiency Analysis of STI Programs 70 Figure 3.17. Fundamental Research Project Expenditures by Cost Category Share of project expenditure 0% 5% 10% 15% 20% 25% 30%35% Researchers’ salaries 29% Travel (fairs, exhibitions, conferences, etc.) 27% Consulting services (e.g. feasibility studies, survey companies) 13% Machinery, equipment, instrumentation 13% Materials, supplies, inventory 12% Other salaries 3% Marketing campaigns or PR 1% activities for project visibility IT systems, specialized software, 0% IT licenses, websites Training and events 0% Space, rent incl. labs, research 0% infrastructure Testing and certifications 0% Intellectual property 0% (patents, trademarks, copyrights) Source: World Bank. Outputs Achieved Four types of outputs were tracked for the Fundamental Research project, based on the program’s ToC: (1) project dissemination (in the form of seminars, workshops, and conferences), (2) training activities, (3) research collaborations during implementation, and (4) researchers involved in im- plementation (in the form of PhDs and postdoctoral researchers hired). Program respondents generated a total of 5,010 outputs from 2016 to 2020, or an average of 12 outputs per project. Over that period, the program invested an average of 103,107 BGN in benefi- ciary projects, so respondents generated 11.6 outputs per 100,000 BGN invested. Seminars, workshops, and conferences were the most common output of Fundamental Research respondents (an average of 6.7 per project), followed by training activities (1.8 per project), PhDs and postdoctoral researchers hired (1.9), and collaborations (1.6) (see Table 3.3). Bulgaria: Efficiency Analysis of STI Programs 71 Table 3.3. Fundamental Research Outputs Total number Average Outputs per Share of of outputs outputs per 100,000 BGN of respondents Output achieved respondent program cost reporting output Seminars, workshops, and conferences 2,810 6.7 6.5 83% Training activities 751 1.8 1.7 53% Collaborations 662 1.6 1.6 50% PhDs and postdocs hired 787 1.9 1.8 57% Source: World Bank. Roughly half of Fundamental Research respondents engaged in at least one collaboration during their project implementation, for an average of 1.6 collaborations per project. Research collabo- rations were much more common than collaborations with industry, which is to be expected giv- en that the program focuses on basic research. Collaborations with domestic research partners were the most common, with 43 percent of respondents reporting engaging in such collaborations, while collaborations with industry partners were the least common (Figure 3.18). Collaborations were somewhat low compared the results reported by Croatian research programs from a similar analysis in 2020, which is an area of concern because the program has objectives related to in- tensifying connections between science, education, business, and other innovation actors. Sev- eral Fundamental Research respondents commented that they would like the program eligibility criteria changed to allow more types of partners during implementation – particularly non-profit organizations. Such changes to program regulations may help boost the share of beneficiaries that collaborate and the number of collaborations generated by the project. Figure 3.18. Fundamental Research Program Collaborations by Type of Partner Average number of collaborations 1,0 100% Share of respondents 0,8 80% 0,6 60% 0,4 40% 0,2 20% 0,0 0% With domestic With foreign With diaspora With domestic With foreign With diaspora research research research industry industry industry partners partners partners partners partners partners Source: World Bank. Number of collaborations Share of respondents Bulgaria: Efficiency Analysis of STI Programs 72 Fundamental Research program respondents rated their collaborations highly. They were gen- erally satisfied with the quality of their collaborations during implementation, with more than 75 percent of respondents that engaged in collaborations with domestic and foreign research part- ners and domestic and foreign industry partners saying these collaborations were above average or excellent (Figure 3.19). Most collaborations reported by Fundamental Research respondents took the form of joint R&D projects or co-authoring research publications (Figure 3.20). Figure 3.19. Fundamental Research Program Quality of Collaborations by Type of Partner 100% 91% 89% 90% 80% Quality of collaborations 80% 75% (Share of respondents) 70% 63% 60% 50% 40% 30% 20% 10% 0% 0% domestic research foreign research diaspora research domestic industry foreign industry diaspora industry partners partners partners partners partners partners Source: World Bank. Note: Left figure shows the share of respondents who rated their collaborations as above average or excellent. Right figure is the share of respon- dents that engaged in a given type of collaboration during implementation (n=207). Figure 3.20. Fundamental Research Program Collaborations by Type of Collaboration Share of respondents that collaborated during implementation 0% 20% 40% 60% 80%100% Joint R&D project 79% Co-author research publication 73% Technological consultancy 16% Purchase of R&D services 8% Test of a new prototype 3% Preparation of technical documentation 1% Licensing/patent registration 1% Source: World Bank. Note: Left figure shows the share of respondents who rated their collaborations as above average or excellent. Right figure is the share of respon- dents that engaged in a given type of collaboration during implementation (n=207). Bulgaria: Efficiency Analysis of STI Programs 73 Seminars, workshops, and conferences were the most common capacity building activities. The majority of respondents attended seminars, workshops, and conferences, either domestically or internationally, while roughly half of respondents engaged in training activities or hired PhD stu- dents to work on their projects. About 40 percent of respondents hired one or more postdoctoral researchers to work on their projects (Figure 3.21). Figure 3.21. Fundamental Research Program Capacity-Building Activities by Type Share of respondents 0% 20% 40% 60% 80%100% Seminars, workshops and conferences attended domestically 79% Seminars, workshops and conferences attended abroad 65% Training activities 53% PhD students hired 51% Postdoctoral researchers hired 40% Source: World Bank. 3.3 Efficiency in Achievement of Outcomes Respondents reported a total of 3,835 outcomes (intended outcomes and other), for an average of 11.1 outcomes per project. Over that period, the program invested an average of 103,107 BGN in beneficiary projects, so respondents generated 10.8 outcomes per 100,000 BGN invested. Two outcomes were tracked for the Fundamental Research program, based on the program’s ToC: 1.) scientific publications in peer-reviewed journals and 2.) research collaborations after project completion. Scientific publications were the most common outcomes reported by respondents, with an average of 6.4 per project, while the respondents with completed projects reported an average of 3.2 collaborative project after their project was finished (Table 3.4). Bulgaria: Efficiency Analysis of STI Programs 74 Table 3.4. Intended Outcomes Achieved by Fundamental Research Respondents Total number Average Outcome per Share of of outcomes outcomes 100,000 BGN of respondents Intended outcome achieved per project program cost reporting outcome Scientific publications in peer-reviewed journals 2,650 6.4 6.2 66% Collaborative projects with domestic research 409 2.3* 2.2 56%* partners after project completion Collaborative projects with foreign research 116 0.7* 0.7 42%* partners after project completion Collaborative projects with diaspora research 20 0.1* 0.1 6%* partners after project completion Collaborative projects with industry partners 22 0.1* 0.1 8%* after project completion Source: World Bank. Note: * Reported per completed project (n=177). Most respondents (60 percent) produced five or fewer publications as part of their projects, while 32 percent produced between six and 20 publications, and six percent produced more than 20 publications (Figure 3.22). Figure 3.22. Publications Produced by Fundamental Research Projects Publications produced (share of respondents) 0% 10% 20% 30% 40% 50% 60%70% 5 or fewer 62% 6 – 10 18% 11 – 20 14% 21 – 30 4% more than 30 2% Source: World Bank. Publications reported by Fundamental Research respondents performed the best among the programs included in this study along several bibliometric indicators. Fundamental Research publications had the highest average year-normalized citations and highest share of publications with international coauthors. More than half of the publications listed by respondents were in- dexed in Web of Science and 44 percent were in journals with impact factor (Table 3.5). However, Fundamental Research publications still had fewer year-normalized citations than the average Bulgarian publication indexed in the Web of Science between 2016 and 2020. Bulgaria: Efficiency Analysis of STI Programs 75 Table 3.5. Impact Measures of Fundamental Research Publications Total publications listed in survey responses 609 Share indexed in Web of Science 54% Share in journals with impact factor 44% Average year normalized citations 1.67 Share with international coauthor(s) 56% Source: World Bank. Other Results Respondents reported some activities related to IP generation and technology development, which is unexpected given the program’s focus on basic research (Table 3.6). Respondents re- ported small numbers of patent applications and patents granted, IP activities (such as industrial designs, copyrights, transfer agreements, etc.)6, new software development, and new technolo- gy development. This may suggest that at least a small share of Fundamental Research projects generate outputs that could be further developed into new technologies, products, or services. Table 3.6. Other Outcomes Generated by Fundamental Research Projects Total number Average Outcomes per Share of of outcomes outcomes per 100,000 BGN of respondents Other outcomes achieved respondent program cost reporting outcome Patent applications 46 0.1 0.1 4% Patents granted 13 0.0 0.0 2% Other intellectual property activities (industrial designs, copyrights, 265 0.6 0.6 3% transfer agreements, etc.) Prototype 45 0.1 0.1 6% New software development 140 0.3 0.3 17% New technology development 116 0.3 0.3 20% Source: World Bank. 6 One respondent, who reported 200 copyrights resulting from their project, was responsible for 75 percent of the “other IP activities” generated by the program. Bulgaria: Efficiency Analysis of STI Programs 76 3.4 Perceived Quality Respondents were generally satisfied with most aspects of the application process, but some expressed concerns about the transparency of the selection process. Less than 70 percent of re- spondents were satisfied with the transparency of the selection process and feedback on project selection, indicating that a portion of program beneficiaries perceive the selection process to be nontransparent. Fundamental Research respondents also commented that eligibility criteria are focused on institutions, rather than on the principal investigator or research team that will car- ry out the project, which can make it more difficult for researchers outside of leading Bulgarian institutions to win grants from the program. Respondents generally felt satisfied with program regulations, contract negotiations, and funding disbursement (Figure 3.23). When asked whether they think any program eligibility criteria should be added, changed, or re- moved, 19 percent of respondents said they thought changes should be made to the current cri- teria. Suggested changes included allowing more types of research partners (such as non-profit organizations), allowing more eligible expenditures, removing maximum age requirements, re- moving criteria focused on academic titles, and allowing technical personnel as part of the proj- ect team (currently, technical staff salaries must be covered by the overhead, which is capped at seven percent of the project budget). When asked about program selection criteria, eight percent respondents said they thought changes should be made. Several respondents said that limitations on the number of programs a researcher can participate in should be removed and that technology transfer impacts should be added to selection criteria. Many respondents were dissatisfied with monitoring requirements. Fewer than 60 percent of respondents were satisfied with the program’s monitoring and financial reporting requirements, indicating that at least a portion of respondents felt these requirements were overly burdensome. Fundamental Research respondents noted that NSF requires certified and translated copies of all eligible expenses, including things like plane tickets. Unlike beneficiaries of EU operational programmes, there is no online portal with preloaded report templates for NSF beneficiaries to report on project progress. However, respondents were generally satisfied other implementation and monitoring processes of the program, including the timeliness of financial support, feedback from monitoring, and the administrative support provided (Figure 3.25). Bulgaria: Efficiency Analysis of STI Programs 77 Figure 3.23. Fundamental Research Program Beneficiaries’ Satisfaction with the Program Application and Selection Processes Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Adequate time for contract negotiation Ease in accessing the program regulations Well explained application procedure Ease of obtaining information about the program Clarity of selection process Clarity of eligibility criteria Flexibility of rules for non-compliance Clarity of program objectives Clarity of program regulations Fairness of eligibility criteria Adequate application to final results of selection time Adequate selection to funding disbursement time Appropriate information requirements in proposals Appropriate list of eligible costs Ease of contract negotiation Ease of application procedure Sufficiency of application period Timely and relevant application support Availability of feedback on project selection approval Transparency of selection process Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 78 Figure 3.24. Fundamental Research Program Beneficiaries’ Satisfaction with Implementation and Monitoring Processes Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Timeliness of financial support Appropriate expert feedback from monitoring (visits, reports, discussions) Sufficiency of administrative support provided Accessibility of administrative support Satisfactory data protection practices Appropriate monitoring requirements (e.g. narrative, indicators, etc.) Appropriate financial reporting requirements Source: World Bank. Note: Figure shows percentage of respondents who agreed or strongly agreed with the statements asking about their satisfaction with aspects of implementation and monitoring. Most respondents felt that the financial resources provided by NSF were sufficient, with 78 per- cent reporting that they had sufficient funding to successfully complete their project. Those re- spondents who said that financial support was insufficient reported that financial support would need to increase by an average of 74 percent in order to successfully complete their projects. In- creased costs of inputs (such as materials, staff, and equipment), unexpected costs, and issues with procurement were cited as the most common reasons why financial support was insufficient (Figure 3.25). This may indicate that the program does not have the flexibility or adaptability to adjust funding to changing external conditions, such as inflation. Similarly, most respondents felt that the amount of time allowed by the program to complete their project was sufficient, with 85 percent reporting that they had sufficient time to success- fully complete their projects. Of those respondents who said they did not have enough time to complete their projects, delays due to COVID work and travel restrictions7, inability to reduce workload from other projects, and inability to reduce institutional obligations were cited as the most common reasons (Figure 3.26). 7 Respondents were provided with the option to select “other” among reasons why the time allotted for their projects was not suffi- cient to successfully complete their projects. Of those respondents who selected “other”, delays due to the COVID pandemic and subsequent economic shutdown were the most common reason described. Bulgaria: Efficiency Analysis of STI Programs 79 Figure 3.25. Reasons Why Fundamental Research Financial Support was Insufficient Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Increased costs of inputs (e.g. survey, materials, lab tests, staff, etc.) Unexpected costs occured Issues with procurement Project’s scope increased beyond the original plan Inadequate budget planning Budget clearing by responsible authority was inadequate Unable to get financial support from my institution Note: N=79 Source: World Bank. Figure 3.26. Reasons Why Time Allotted for Fundamental Research Projects was Insufficient Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% I could not reduce other project workload I could not reduce institutional responsibilities I could not reduce teaching workload Other Note: N=55 Source: World Bank. Financial and human resources were the most common key success factors. The availability of financial resources and human resources were cited as key success factors by 90 percent and 83 percent of respondents respectively. The availability of research infrastructure was also re- ported as an important factor for success by more than half of respondents (Figure 3.27). When asked about any additional support needed during implementation that would have improved the outcomes of their projects, respondents indicated that they needed a range of additional sup- port from the program. They cited assistance with procurement8 (by 22 percent of respondents), 8 In the Functional and Governance Analysis, public procurement processes were identified as a major challenge to public sector ben- eficiaries, which are covered by the Public Procurement Act. Timelines for each procurement may last between two and four months (depending on the type of procedure) from opening a procurement to concluding a contract, without including the evaluation pro- cess. If there is a litigation in court regarding the granting of the contract (which is relatively common in Bulgaria), the process may take up to a whole year. Bulgaria: Efficiency Analysis of STI Programs 80 better administrative support from NSF (by 20 percent), assistance with budget preparate (by 15 percent), and assistance with finding additional funding sources (by 13 percent) as the most needed additional supports (Figure 3.28). Figure 3.27. Key Success Factors for Fundamental Research Projects Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Availability of financial resources 90% Availability of human resources such as researchers or mentors 83% Availability of research infrastructure 63% Support of the research institution 29% The way support program is designed 13% and implemented Source: World Bank. Figure 3.28. Additional Support Needed for Fundamental Research Projects Share of respondents 0% 5% 10% 15% 20%25% Assistance with procurement 22% Better administrative support by nsf 20% Assistance in the preparation of project budgets 15% Assistance with finding additional 13% funding sources Access to research infrastructure 9% and equipment Assistance in preparation of 8% monitoring reports Assistance to hire foreign researchers 7% Assistance to establish collaborations 7% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 81 Overall Project Quality Respondents’ project objectives were largely in line with the objectives of the Fundamental Re- search program. The program focuses on improving the quality and quality of basic research pre- formed in Bulgaria, increasing the capacity of beneficiary organizations, and increasing connec- tions with scientific community in basic research areas. Over 75 percent of respondents reported publishing scientific papers in peer reviewed journals was a top objective of their project, while 69 percent said developing a cadre of young researchers and 45 percent said presenting papers in seminars and conferences were top objectives – all of which relate to increasing the capacity of beneficiary organizations and improving the quality and quality of basic research. Similarly, 45 percent of respondents reported collaborating with other researchers or institutions as an ob- jective, which relates to the program goal of increasing connections with scientific community in basic research. However, a small share of researchers had objectives outside of the program’s core objectives, such as developing a new product or services or producing market-oriented re- search (Figure 3.29). Figure 3.29. Fundamental Research Program Project Objectives Share of respondents 0% 20% 40% 60% 80%100% Publish scientific papers in peer-reviewed journals 78% Develop a cadre of young researchers 69% Collaborate with other researchers or research institutions 45% Present scientific papers in seminars 33% and conferences Improve chances to get EU funding 29% Develop a new product, service, 28% or process Upgrade a product, service, 6% or process Produce market-oriented research 4% Pursue intellectual property 2% Collaborate with the private sector 1% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 82 Most respondents felt that the outcomes of their project matched their expectations. Seven- ty-five percent said so. Over 20 percent felt their project exceeded expectations, while 5 percent said their project outcome was below expectations (Figure 3.30). Figure 3.30. Match Between Fundamental Research Projects and Beneficiaries’ Expectations Share of respondents 0% 20% 40% 60% 80%100% Above expectations 21% Matched expectations 75% Below expectations 5% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 83 4. Vihren Bulgaria: Efficiency Analysis of STI Programs 84 4. Vihren Program Summary The Vihren program was created to address some of the human resource challenges in the public research sector by supporting the development of new research teams formed around leading Bulgarian scientists, with the aim of developing a new generation of public researchers. The program is designed and implemented by the NSF and MoES and funds basic or applied research projects carried out by leading and established scientists and their research teams, with the objective of increasing the capacity of the Bulgar- ian public research sector to conduct high quality research. It has intended outcomes related to scientific publications in peer reviewed journals and collaborative projects formed after project completion. Like the Fundamental Research program, external services made up a large share (55 per- cent) of the administrative costs of implementing the program, while personnel made up only six percent of administrative costs. This cost structure is likely due to the small number of full-time staff at NSF; the small staff at NSF appear to rely heavily on external experts in the implementation of the program. Most respondents hired postdoctoral researchers and PhD students, which is in line with the core objectives of the Vihren program – to support established and leading scientists to develop new research teams and new researchers capable of producing high quality re- search. However, it is notable that only two of the eight respondents engaged in training activities as part of their projects, and the Vihren program’s per project training output (0.4 per project) was considerably lower than for Fundamental Research. Respondents did not report any research collaborations thus far as part of their projects, despite the fact that the intended results of the program include expanding the scientific contacts and networks of funded researchers and strengthening the scientific networks of participating scientific organizations. Respondents were largely satisfied with the processes for program application and imple- mentation, with the exceptions of technical monitoring requirements and data protection practices. Bulgaria: Efficiency Analysis of STI Programs 85 The Vihren program has thus far only issued a single call for proposals, funding 10 projects in 2019 for a total of 11.5 million BGN in contracted funding (4.6 million BGN of which has been dis- bursed). All of the projects funded by the program were still active as of the writing of this report. The principal investigators (or alternative points of contact) for all 10 projects funded by the pro- gram in 2019 were surveyed for this analysis, with 8 beneficiaries responding to the survey for a response rate of 80 percent. Respondents represent 79 percent of the contracted funding for the program with an average grant size of 1.1 million BGN. Respondents’ projects were concentrated in one of four scientific fields, led by chemistry and physics (38 percent of projects each), and fol- lowed by health sciences and social sciences (13 percent of projects each) (Figure 4.1). Figure 4.1. Vihren Projects by Scientific Field Share of respondents 0% 5% 10% 15% 20% 25% 30% 35%40% Chemistry 38% Physics 38% Health sciences 13% Social Sciences 13% Source: World Bank. 4.1 Efficiency in Use of Inputs Grants provided to beneficiaries accounted for 98 percent of the costs of the program from 2019 to 2020, while administrative costs made up the remaining 2 percent. (See Figure 4.2.) The bulk of administrative costs of the program occurred in 2019, when the program’s single call for propos- als was issued and the 10 projects were selected for award, while grants disbursement increased from 2019 to 2020, leading to higher overall program costs in 2020 (Figure 4.3). Bulgaria: Efficiency Analysis of STI Programs 86 Figure 4.2. Vihren Program Costs by Category, 2019–20 2% 98% Administrative Costs Source: World Bank. Grants Distributed Figure 4.3. Vihren Program Costs by Year, 2019–20 30 Million BGN, 2020 constant 25 20 15 10 5 0 2019 2020 Source: World Bank. Administrative Costs Grants Distributed Administrative Costs External services made up more than half of administrative costs. External services, in the form of contracts to technical experts to evaluate proposals and provide advisory support, made up a relatively large share of program costs (55 percent), while personnel costs only accounted for 6 percent of total administrative costs (Figure 4.4). Most of the costs for external services came in 2019 for contracting technical experts to evaluate proposals. After all 10 projects were awarded in 2019, external services dropped dramatically in 2020. Personnel and fixed costs declined slightly from 2019 to 2020 (Figure 4.5). Like the Fundamental Research program, the large share of costs for external services and small share for personnel is likely due to the low numbers of full-time staff at the NSF. This arrange- ment is likely done to compensate for the lack of staff at NSF, whereby the small staff at NSF rely Bulgaria: Efficiency Analysis of STI Programs 87 heavily on external experts in the implementation of the program. From the perspective of this efficiency analysis, this is not necessarily an inefficient or ineffective arrangement; however, com- ponent 2 of the PER STI project, the Functional and Governance Analysis, found that the lack of staff has negatively impacted the implementation of programs at NSF in several areas, including knowledge management and M&E. Figure 4.4. Vihren Program Administrative Costs by Category, 2019–20 5% 55% 40% Personnel Fixed Costs Source: World Bank. External Services Figure 4.5. Vihren Program Administrative Costs by Year, 2019–20 100.000 90.000 80.000 Constant 2020 BGN 70.000 60.000 50.000 40.000 30.000 20.000 10.000 0 2019 2020 Source: World Bank. Personnel Fixed Costs External Services Bulgaria: Efficiency Analysis of STI Programs 88 Implementation made up the largest share of personnel costs. Over 60 percent of personnel costs went toward implementation, while 28 percent went to program design, and 11 percent went to M&E (Figure 4.6). Most of the monitoring of Vihren projects is done by external experts, which accounts for the relatively low share of personnel costs in this area. Similarly, the program had no personnel costs for the evaluations of project proposals, which is also done by external experts contracted by the program. Personnel costs remained fairly constant from 2019 to 2020 (Figure 4.7). Figure 4.6. Vihren Program Personnel Costs by Category, 2019–20 11% 61% 28% Design Implementation Source: World Bank. M&E Figure 4.7. Vihren Program Personnel Costs by Year, 2019–20 3.500 3.000 Constant 2020 BGN 2.500 2.000 1.500 1.000 500 0 2019 2020 Source: World Bank. Design Implementation M&E Bulgaria: Efficiency Analysis of STI Programs 89 Office equipment and ICT costs made up more than half of the fixed costs for the program. (See Figure 4.8.) While costs related to office equipment, ICT, and office space remained relatively con- stant from 2019 to 2020, costs related to goods and services were much higher in 2019 (the year all of the Vihren contracts were awarded) (Figure 4.9). Figure 4.8. Vihren Program Fixed Costs by Category, 2019–20 35% 56% Office equipment and ICT Office space Source: World Bank. 9% Goods and Services Figure 4.9. Vihren Program Fixed Costs by Year, 2019–20 30.000 Constant 2020 BGN 25.000 20.000 15.000 10.000 5.000 0 2019 2020 Source: World Bank. Office equipment and ICT Office space Goods and Services Bulgaria: Efficiency Analysis of STI Programs 90 A large majority of costs for external services were related to contracting experts for the evalua- tion of project proposals. NSF also utilizes external experts on several advisory commissions that provide guidance on its programming, which represented a small share of the costs for external services (Figure 4.10). External services were highest in 2019, when the call for proposals was is- sued; after all Vihren projects were awarded in 2019, there were no additional costs related to ex- pert evaluators, leading to a dramatic drop in costs related to external services in 2020 (Figure 4.11). Figure 4.10. Vihren Program External Services Costs by Category, 2019–20 3% 7% 90% Experts for evaluation of proposals Advisory Source: World Bank. other Figure 4.11. Vihren Program External Services Costs by Year, 2019–20 70.000 60.000 Constant 2020 BGN 50.000 40.000 30.000 20.000 10.000 0 2019 2020 Source: World Bank. Experts for evaluation of proposals Advisory other Bulgaria: Efficiency Analysis of STI Programs 91 The total administrative costs of implementing the program from 2019 to 2020 was 114,107 BGN, with an average cost per project of 11,411 BGN. In the future, NSF could consider the balance be- tween costs for full-time personnel and external experts in the implementation of its programs – it may ultimately be more effective to allocate more resources to staff and fewer to external services. This may not necessarily lead to cost savings for the program, but it would increase the capacity of NSF as an implementor and allow the program to retain more institutional knowledge. Table 4.1. Vihren Administrative Costs Cost category Total costs Cost per project Personnel 6,232 6,232 Fixed 45,273 45,273 External services 62,601 62,601 Source: World Bank. Note: All amounts in constant 2020 BGN. Costs Covered by Beneficiaries The application process for the Vihren program is slightly costlier and more labor intensive than that of the Fundamental Research program. Respondents reported an average application cost of 2,925 BGN and spent an average of 31 working days preparing their applications. However, the average grant size for Vihren is almost ten times larger than that of Fundamental Research, so application costs as a share of grants contracted is only 0.3 percent for Vihren, compared to 1.9 percent for Fundamental Research. Only two of the eight respondents reported spending more than 3,000 BGN to prepare their applications, while three respondents spent 1,000 BGN or less (Figure 4.12). Only one respondent reported spending more than 30 working days preparing their applications (Figure 4.13). Figure 4.12. Vihren Program Application Costs Application costs (Share of respondents) 0% 10% 20% 30%40% 1,000 or less 38% 1,001 – 3,000 38% More than 3,001 25% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 92 Figure 4.13. Vihren Program Time to Prepare Applications Time to prepare application (Share of respondents) 0% 20% 40%60% 20 days or less 29% 21-30 days 57% More than 30 days 14% Source: World Bank. N=7 Most respondents used no external support in preparing applications. Only two of the eight re- spondents used external support (such as legal, accounting, or other services) to prepare their applications, which likely contributed to the relatively low cost of application preparation when compared to other programs in this analysis. The only type of additional support used by respon- dents was expert consultants (used by two respondents) (Figure 4.14). Figure 4.14. Beneficiaries Using External Support in Preparing Applications for the Vihren Program Share of respondents 0% 10% 20% 30% 40% 50% 60% 70%80% Experts/consultants 29% No additional 71% support required Source: World Bank. Although the Vihren program has no requirements for matching contributions from beneficiaries, respondents reported an average of 9,500 BGN in-kind contributions to the implementation of their projects. No applicants made cash contributions. Six respondents made in-kind contribu- tions, while two respondents made no contributions (in kind or cash) to their projects. On average, the costs covered by the program were 38 times higher than the costs covered by respondents. The average cost covered by the program, including disbursed grants and adminis- trative costs, was 470,559 BGN, while the average cost covered by beneficiaries was 12,425 BGN. Bulgaria: Efficiency Analysis of STI Programs 93 4.2 Efficiency in Generation of Outputs For every BGN of administrative costs, beneficiaries received an average of 40 BGN. The total program costs from 2019 to 2020 was 4.7 million BGN or 470,559 BGN per project. This includes 4.6 million in disbursed grants to beneficiaries and 114,107 BGN in administrative costs (Table 4.2). The program did not include any nonfinancial or indirect financial support to beneficiaries. Researcher salaries made up 60 percent of the expenditures of respondents. Other salaries, ma- chinery and equipment, and materials and inventory were the next largest expenses (Figure 4.15). NSF does not allow for grant funding to be used for IP-generation or commercialization activities, which can explain why there were no expenses associated with such activities. It may be worth reconsidering this restriction for the Vihren program, given that the program can support both basic and applied research projects. Table 4.2. Vihren Program Costs, 2019–20 Total Average per project Grant funding disbursed 4,591,480 BGN 459,148 BGN Administrative and operating costs 114,107 BGN 11,411 BGN Total program cost 4,705,587 BGN 470,559 BGN Source: World Bank. Note: All amounts in constant 2020 BGN. Figure 4.15. Vihren Project Expenditures by Cost Category Share of project expenditure 0% 10% 20% 30% 40% 50% 60%70% Researchers’ salaries 60% Other salaries 13% Machinery, equipment, instrumentation 8% Materials, supplies, inventory 6% Travel (fairs, exhibitions, conferences, etc.) 4% IT systems, specialized software, IT licenses, websites 1% Training and events 1% Consulting services (e.g. feasibility studies, survey companies) 1% Marketing campaigns or PR activities for project visibility 1% Testing and certifications 0% Intellectual property (patents, trademarks, copyrights) 0% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 94 Outputs Achieved Four outputs were tracked for the Vihren project, based on the program ToC: 1.) project dissem- ination (in the form of seminars, workshops, and conferences), 2.) training activities, 3.) research collaborations during implementation, and 4.) researchers involved in implementation. Program respondents generated a total of 47 outputs from 2019 to 2020, or an average of 6.8 out- puts per project. Over that period, the program invested an average of 470,559 BGN in beneficiary projects, so respondents generated 1.4 outputs per 100,000 BGN invested. Seminars, workshops, and conferences were the most common output of Vihren respondents (an average of 3.5 per project), followed by PhDs and postdoctoral researchers hired (2.9 per proj- ect), and training activities (0.4) (see Table 4.3). Table 4.3. Outputs of Vihren Projects Total number Average Outputs per Share of of outputs outputs per 100,000 BGN of respondents Output achieved respondent program cost reporting output Seminars, workshops, and conferences 24 3.5 0.8 50% Training activities 3 0.4 0.1 25% Collaborations 0 0.0 0.0 0% PhDs and postdocs hired 20 2.9 0.6 57% Source: World Bank. Respondents did not report any research collaborations thus far as part of their projects, despite the fact that the intended results of the program include expanding the scientific contacts and networks of funded researchers and strengthening the scientific networks of participating scien- tific organizations. It should be noted that the program does not incentivize collaborations (such as awarding additional points to proposals that include research partners) or mandate that bene- ficiaries collaborate, which may contribute to the lack of collaboration activities of respondents. Most respondents hired postdoctoral researchers and PhD students. These expenses are in line with one of the core objectives of the Vihren program—to support established and leading sci- entists to develop research teams capable of producing high quality research. However, given this objective, it is notable that only two of the eight respondents engaged in training activities as part of their projects, and the Vihren programs per project training output (0.4 per project) is considerably lower than for Fundamental Research (1.7 per project) (Figure 4.16). Bulgaria: Efficiency Analysis of STI Programs 95 Figure 4.16. Vihren Program Capacity-Building Activities by Type Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Postdoctoral researchers hired 75% PhD students hired 63% Seminars, workshops and conferences attended abroad 50% Training activities 25% Seminars, workshops and conferences attended domestically 25% Source: World Bank. 4.3 Efficiency in Achievement of Outcomes Respondents reported a total of 24 outcomes (intended outcomes and other), for an average of three outcomes per project. Over that period, the program invested an average of 470,559 BGN in beneficiary projects, so respondents generated 0.6 outcomes per 100,000 BGN invested. Two outcomes were tracked for the Vihren program. Based on the program’s ToC, the program tracked two outcomes: (1) scientific publications in peer-reviewed journals and (2) research col- laborations after project completion. (Because all Vihren projects are ongoing, research collab- orations after project completion were not included in this analysis.) Respondents reported an average of 2.3 scientific publications per project (Table 4.4). Table 4.4. Intended Outcomes Achieved by Vihren Respondents Total number of Average outcomes Outcome per 100,000 Share of respondents Intended outcome outcomes achieved per respondent BGN of program cost reporting outcome Scientific publications in 18 2.3 0.5 50% peer-reviewed journals Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 96 Half of respondents had not yet produced a publication in a peer reviewed journal as of the time they were surveyed. This low rate of publications is understandable given that all Vihren projects only began in 2019 and the surveys took place in June–July 2021. Almost 40 percent of respondents produced between one and five publications, and 13 percent produced more than five (Figure 4.17). Figure 4.17. Peer-Reviewed Publications Produced by Vihren Program Beneficiaries Applications produced (share of respondents) 0% 10% 20% 30% 40% 50%60% none 50% 1 – 5 38% more than 5 13% Source: World Bank. Publications reported by Vihren respondents received few citations. They had lower numbers of year-normalized citations than the Fundamental Research and ICT programs or the average Bulgarian publication indexed in Web of Science between 2016 and 2020 (Table 4.5) However, given the small number of respondents and small number of publications detailed in the survey responses, these findings may not be significant. Table 4.5. Impact Measures of Vihren Publications Total publications listed in survey responses 13 Share indexed in Web of Science 38% Share in journals with impact factor 31% Average year normalized citations 0.6 Share with international coauthor(s) 40% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 97 Other Results Vihren projects produced a small number of other outcomes. These other outcomes tracked by the researcher surveys were related to new technology development (Table 4.6), indicating that at least a small share of respondent projects were engaging in innovation activities. At least one respondent reported one or more patent applications, other IP activities (such as industrial de- signs, copyrights, or transfer agreements), prototype, new software development, or new technol- ogy development as part of their project. Like the Fundamental Research results, this suggests that at least a share of Vihren projects generate outputs that could be further developed into new technologies, products, or services. Table 4.6. Other Outcomes Reported by Vihren Respondents Total number Average Outcomes per Share of of outcomes outcomes per 100,000 BGN of respondents Other outcomes achieved respondent program cost reporting outcome Patent applications 1 0.1 0.0 14% Patents granted 0 0.0 0.0 0% Other intellectual property activities (industrial designs, copyrights, 2 0.2 0.0 14% transfer agreements, etc.) Prototype 1 0.1 0.0 14% New software development 1 0.1 0.0 14% New technology development 1 0.1 0.0 14% Source: World Bank. 4.4 Perceived Quality Respondents were largely satisfied with application processes, with 75 percent or more of re- spondents reporting that they were satisfied with most aspects of project application and se- lection, with the exceptions of timeliness of application support and the time between selection and funding disbursement. When asked if they think any program selection or eligibility criteria should be added, changed, or removed; one respondent said that eligibility criteria should be al- tered to broaden the definition of young scientists to include researchers that had recently ob- tained a PhD. Vihren respondents also noted that the program uses a common selection panel for selecting projects for all the physical sciences (including astronomy, chemistry, materials sci- ence, and physics), yet project proposals may be hard to compare across all of these disciplines. Bulgaria: Efficiency Analysis of STI Programs 98 Respondents were also largely satisfied with implementation and monitoring processes. Seven- ty-five percent or more of respondents reporting that they were satisfied with most aspects of implementation, with the exceptions of technical monitoring requirements (where only 63 per- cent of respondents were satisfied) and data protection practices (where only 50 percent were satisfied (Figure 4.18). Along with Fundamental Research, Vihren respondents were least satisfied with technical monitoring requirements among the programs included in this analysis, which may indicate that NSF’s technical monitoring processes are more burdensome than those or other STI implementors. Unlike beneficiaries of EU operational programmes, NSF beneficiaries do not have access to an online portal with preloaded templates for reporting on project progress. Figure 4.18. Vihren Program Beneficiaries’ Satisfaction with Implementation and Monitoring Processes Satisfaction with implementation processes (Share of responents) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Timeliness of financial support Appropriate expert feedback from monitoring (visits, reports, discussions) Appropriate financial reporting requirements Accessibility of administrative support Sufficiency of administrative support provided Appropriate monitoring requirements (e.g. narrative, indicators, etc.) Satisfactory data protection practices Source: World Bank. Most respondents felt that the financial resources provided by NSF were sufficient, with 86 per- cent reporting that they had sufficient funding to successfully complete their projects. The one respondent who said that financial support was insufficient cited inadequate budget cleared by NSF and issues with procurement as reasons why they needed more financial support. Similarly, most respondents felt that the amount of time allowed by the program to complete their project was sufficient, with 86 percent reporting that they had sufficient time to success- fully complete their projects. The one respondent who said the amount of time allowed by the program was insufficient cited difficulties in hiring PhDs students and postdoctoral researchers and lack of support staff as reasons why they required more time. The availability of financial and human resources were mentioned as the main factors for project success. Each was cited by 86 percent of respondents as one of the top three success factors for Bulgaria: Efficiency Analysis of STI Programs 99 their project. The availability of research infrastructure was also mentioned by more than half of respondents as an important success factor (Figure 4.19). When asked about any additional sup- port needed during implementation that would have improved the outcomes of their projects, re- spondents cited assistance with the hiring of foreign researchers (by 38 percent of respondents), better administrative support from NSF (by 38 percent), assistance with procurement (by 25 percent), and assistance with the preparation of monitoring reports (by 25 percent) as the most needed additional supports (Figure 4.20). One quarter of respondents felt they did not require any additional support that was not already provided by the program. Given that the program’s objec- tives focus on the development of high-quality research teams led by leading researchers, more assistance for beneficiaries in hiring foreign researchers could help improve project outcomes. Figure 4.19. Key Success Factors for Vihren Projects Key success factors (Share of respondents 0% 20% 40% 60% 80%100% Availability of financial resources 86% Availability of human resources such as researchers or mentors 86% Availability of research infrastructure 57% The way support program is designed 43% and implemented Source: World Bank. Figure 4.20. Additional Support Needed for Vihren Projects Additional support needed (Share of respondents 0% 10% 20% 40%60% Assistance to hire foreign researchers 38% Better administrative support 38% Assistance with procurement 25% Assistance in preparation of 25% monitoring reports Assistance in the preparation of 13% project budgets Access to research infrastructure 13% and equipment Didn’t need additional 25% support or services Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 100 Overall Project Quality Respondent’s project objectives were largely in line with the objectives of the Vihren program. The program focuses on the formation of new research teams around leading and established researchers, and 88 percent of respondents said that one of the key goals of their projects was to develop a cadre of young researchers. A large share of respondents also had objectives relat- ed to publishing papers in peer-reviewed journals and improving their chances of obtaining EU funding for their research (Figure 4.21). Figure 4.21. Vihren Program Project Objectives Project objectives (Share of respondents) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Develop a cadre of young researchers 88% Improve chances to get EU funding 75% Publish scientific papers in peer-reviewed journals 75% Collaborate with other researchers or 25% research institutions Develop a new product, service, 13% or process Present scientific papers in seminars 13% and conferences Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 101 5. ICT for a Single Digital Market in Science, Education, and Security Bulgaria: Efficiency Analysis of STI Programs 102 5. ICT for a Single Digital Market in Science, Education, and Security Program Summary ICT for a Single Digital Market in Science, Education, and Security (ICT) is one of the sectoral programs of the National Science Programs 2018–22 portfolio. These programs, implemented by the MoES, are applied research grants focused on addressing identified public challenges in Bulgaria – in the case of the ICT program, it is aimed at building infrastructure, tools, and security for open science. The program funds applied research projects in two core research areas: 1) open science infrastructure and security and 2) digital educational tools. As an applied research program, ICT has intend- ed outcomes related to the development of prototypes, new software, and new technologies. The sectoral programs of the National Science Programs 2018-2022 portfo- lio have a different design than traditional research grants: each program provides funding to a consortium of preselected research institutions in a specific scientific field. In the case of the ICT program, the beneficiary con- sortium is made up of 10 HEIs and PROs, which were preselected based on indicators related to published scientific results in the fields of informatics and data science, so there was no application process or application costs for beneficiaries of the program. MoES also delegates a portion of the ad- ministration and monitoring of projects to the consortium, so some of the costs of implementing the overall program are borne by the beneficiaries. In the case of the ICT program, the leading beneficiary of the consortium (the University of Sofia) is responsible for distributing funding to the other members of the consortium, collecting indicators on project progress, and compiling semi-annual technical and financial reports to the ministry. Administrative costs only made up only 0.2 percent of total program costs; however, the very low share of administrative costs do not represent the full cost of program implementation –a portion of the implementation costs of the program are borne by the beneficiary consortiums. While the beneficiaries Bulgaria: Efficiency Analysis of STI Programs 103 had no costs related to the application processes, respondents reported ex- pending an average of 4.8 percent of grant funding received for management and auditing of their projects. Scientific publications in peer-reviewed journals were the most common outcome reported by ICT respondents, even though the program’s objec- tives are focused on applied research. Respondents also reported a relatively high number of outcomes related to new software and new technology de- velopment, which are the primary intended outcomes of the program. The program generated a relatively high number of collaborations during imple- mentation, which is likely due in part to the design of the program – bene- ficiaries are members of consortiums that collaborate with each other on individual projects. Respondent’s project objectives differed significantly from the objectives of the ICT program. The program primarily aims to support the development of new digital tools and software in the areas of open science and education, with objectives related to the development of new software, digital tools, and open science repositories. However, the most common project objectives reported by respondents were developing a cadre of young researchers, col- laborating with other researchers or research institutions, and publishing scientific papers in peer-reviewed journals. This may explain why respon- dents produced a higher number of publications per project than might be expected for an applied research program. This misalignment may stem from the fact that there was no application process for the program, and therefore beneficiary projects were not evaluated by the MoES. The program was started in 2018 and issued 10 grants to consortium members in 2019, totaling 4.2 million BGN. Disbursements to date total 3.2 million BGN. The principal investigators (or alternative points of contact) for all 12 lead beneficiaries of the ICT consortium were surveyed for this analysis, with 8 beneficiaries responding to the survey for a response rate of 67 percent. Respondents represent 85 percent of the contracted funding for the program with an average grant size of 444,336 BGN. All respondents had projects in the scientific field of mathematical sciences and informatics. Bulgaria: Efficiency Analysis of STI Programs 104 5.1 Efficiency in Use of Inputs Grants are the lion’s share of ICT program costs, which peaked in 2018. Grants distributed to beneficiaries made up 99.8 percent of the costs of the program from 2018 to 2020, while admin- istrative costs made up only 0.2 percent of program costs (Figure 5.1). Grant disbursement was highest in 2018, before dropping in 2019 and rising again slightly in 2020 (Figure 5.2). Figure 5.1. ICT Program Costs by Category, 2018–20 0,2% 99,8% Administrative Costs Source: World Bank. Grants Distributed Figure 5.2. ICT Program Costs by Year, 2018–20 1.400.000 1.200.000 Constant 2020 BGN 1.000.000 800.000 600.000 400.000 200.000 0 2018 2019 2020 Source: World Bank. Administrative Costs Grants Distributed Bulgaria: Efficiency Analysis of STI Programs 105 Administrative Costs Personnel costs made up 95 percent of total administrative costs, with fixed costs making up the remaining 5 percent of administrative costs. There were no costs related to external services for the program (Figure 5.3). Personnel costs were highest in 2019—the first full year after the 10 projects were active—before declining slightly in 2020 (Figure 5.4). Figure 5.3. ICT Program Administrative Costs by Category, 2018–20 5% 95% Personnel Source: World Bank. Fixed Costs Figure 5.4. ICT Program Administrative Costs by Year, 2018–20 2.500 Constant 2020 BGN 2.000 1.500 1.000 500 0 2018 2019 2020 Source: World Bank. Personnel Fixed Costs Bulgaria: Efficiency Analysis of STI Programs 106 Costs related to implementation made up over 75 percent of personnel costs for the program, with the remainder coming from costs for M&E. (See Figure 5.5.) All of the costs for M&E came after 2018 (the year the projects were awarded and launched), while costs related to implemen- tation have remained relatively constant from 2018 to 2020 (Figure 5.6). Figure 5.5. ICT Program Personnel Costs by Category, 2018–20 24% 76% Implementation M&E Source: World Bank. Figure 5.6. ICT Program Personnel Costs by Year, 2018–20 2.500 Constant 2020 BGN 2.000 1.500 1.000 500 0 2018 2019 2020 Source: World Bank. Implementation M&E Bulgaria: Efficiency Analysis of STI Programs 107 Fixed costs have risen steadily since 2018, although they still represented a very small portion (less than five percent) of total administrative costs (Figure 5.7). Due to data availability, it was not possible to separate out fixed costs by category (such as costs for goods and services, and so on). Figure 5.7. ICT Program Fixed Costs by Year, 2018–20 200 180 160 140 Constant 2020 BGN 120 100 80 60 40 20 0 2018 2019 2020 Source: World Bank. The total administrative costs of implementing the program from 2018 to 2020 was 5,790 BGN, with an average cost per project of 483 BGN (Table 5.1). However, the extremely low average ad- ministrative cost per project is misleading – as discussed in the next section, a portion of the ad- ministrative and implementation costs of the program are borne by the beneficiary consortiums, so the full cost of program implementation is not reflected here. By comparison, the administra- tive average cost per project was 483 BGN, but respondents spent an average of 21,328 BGN per project on management and auditing costs. Table 5.1. ICT Administrative Costs Cost category Total costs Cost per project Personnel 5,487 BGN 457 BGN Fixed 302 BGN 25 BGN External services 0 BGN 0 BGN Source: World Bank. Note: All amounts in constant 2020 BGN. Bulgaria: Efficiency Analysis of STI Programs 108 Costs Covered by Beneficiaries As discussed before, the National Science Programs sectoral programs have a different design than traditional grant programs – each provides funding to a preselected consortium of bene- ficiary HEIs and PROs. Much of the administration, monitoring, and auditing of the program is actually done by the consortium, rather than the MoES. Administration costs to the Ministry are low, but the beneficiaries bear additional costs because of this design. There were no application costs associated with the program. The ICT program, like the other programs in the National Science Programs 2018-2022 portfolio, has no application process; in- stead, beneficiaries are preselected to participate based on indicators related to published sci- entific results in the fields of informatics and data science. Therefore, respondents reported no costs related to preparing their applications. The ICT program has no matching requirement for beneficiaries, but 63 percent of respondents reported making in-kind contributions to their projects for an average of 34,125 BGN in in kind contributions to the implementation of their projects. In-kind contributions for a project of this kind could take the form of researcher salaries and the usage of facilities, laboratories, and equip- ment. No respondents made cash contributions to their projects. Respondents also reported expending an average of 4.8 percent of grant funding received for ad- ministration and auditing of their projects, for an average of 21,328 BGN per beneficiary. These additional costs are due to the design of the sectoral programs of the National Science Pro- grams portfolio, which delegate a portion of the administration of the program to the beneficiary consortiums. On average, the costs covered by the program was 4.7 times higher than the costs covered by respondents. The average cost covered by the program, including disbursed grants and adminis- trative costs, was 262,982 BGN, while the average cost covered by beneficiaries was 55,453 BGN. 5.2 Efficiency in Generation of Outputs For every BGN of administrative costs, beneficiaries received an average of 544 BGN. The total program costs from 2018 to 2020 was 3.2 million BGN, or 262,982 BGN per project. This includes 3.15 million in disbursed grants to beneficiaries and 5,790 BGN in administrative costs (Table 5.2). The program did not include any nonfinancial or indirect financial support to beneficiaries. Bulgaria: Efficiency Analysis of STI Programs 109 Table 5.2. ICT Program Costs, 2018–20 Total Average per project Grant funding disbursed 3,150,000 BGN 262,500 BGN Administrative and operating costs 5,790 BGN 482 BGN Total program cost 3,155,790 BGN 262,982 BGN Source: World Bank. Note: All amounts in constant 2020 BGN. Researcher salaries were the largest expenditure. Respondents reported that researcher sala- ries made up 38 percent of total project costs. Machinery and equipment (18 percent), materials and consumables (14 percent) and travel (nine percent) were the next largest expenses by size (Figure 5.8). Figure 5.8. ICT Project Expenditures by Cost Category Share of project expenditure 0% 5% 10% 15% 20% 25% 30% 35%40% Researchers’ salaries 38% Machinery, equipment, tools 18% Materials, consumables, inventory 14% Travel (fairs, exhibitions, conferences, etc.) 9% Consulting services 5% Other 5% Training and events 4% IT systems, specialized software, 3% IT licenses, websites other salaries 3% Marketing campaigns or PR 2% activities to promote the project Space, rent incl. labs, research 1% infrastructure Testing and certifications 0% Intellectual property 0% (patents, trademarks, copyrights) Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 110 Outputs Achieved Three outputs were tracked for the ICT program, based on the program ToC: 1.) project dissem- ination (in the form of seminars, workshops, and conferences), 2.) training activities, and 3.) re- search collaborations during implementation. Program respondents generated a total of 207 outputs from 2018 to 2020, or an average of 25.9 outputs per project. Over that period, the program invested an average of 262,982 BGN in bene- ficiary projects, so respondents generated 9.8 outputs per 100,000 BGN invested. Seminars, workshops, and conferences were the most common output of ICT projects. ICT proj- ects produced an average of 14.4 seminars, workshops, and conferences. Other common outputs were collaborations during implementation (average 6.8 per project), and training activities (4.8 per project) (Table 5.3). Table 5.3. Outputs Reported by ICT Beneficiaries Total number of Average outputs Outputs per 100,000 Share of respondents Intended output outputs achieved per respondent BGN of program cost reporting output Seminars, workshops, 115 14.4 5.5 88% and conferences Training activities 38 4.8 2.2 88% Collaborations 54 6.8 2.6 88% Source: World Bank. Collaborations with other members of the ICT consortium were most common. The most com- mon type of collaboration during implementation, both by average number of collaborations and share of respondents that engaged in such a collaboration, was with other members of the ICT consortium—understandable, given the design of the program. Other common collaborations were with domestic research partners (outside of the consortium) and domestic industry part- ners (Figure 5.9). Bulgaria: Efficiency Analysis of STI Programs 111 Figure 5.9. ICT Project Collaborations by Type of Partner Average number of collaborations 3,5 100% 3 Share of respondents 80% 2,5 2 60% 1,5 40% 1 20% 0,5 0 0% Other ICT domestic foreign diaspora domestic foreign diaspora consortium research research research industry industry industry members partners partners partners partners partners partners Source: World Bank. Number of collaborations Share of respondents Respondents were generally satisfied with the quality of their collaborations with other members of the consortium and other research partners, but less satisfied with their collaborations with domestic industry partners. All respondents were satisfied with their collaborations with other members of the ICT consortium and foreign research partners, while 67 percent were satisfied with their collaborations with domestic research partners, and only 40 percent were satisfied with collaborations with domestic industry (Figure 5.10). The most common types of collabora- tions reported were joint R&D projects and co-authoring research publications, while technology consultancies and testing of prototypes were rarer (Figure 5.11). Figure 5.10. Satisfaction with the Quality of ICT Collaborations 100% 90% Quality of collaobarions 80% (share of respondents) 70% 60% 50% 40% 30% 20% 10% 0% Other ICTconsortium domestic research foreign research domestic industry members partners partners partners Source: World Bank. Note: Left figure shows the share of respondents who rated their collaborations as above average or excellent Bulgaria: Efficiency Analysis of STI Programs 112 Figure 5.11. ICT Collaborations by Type Share of respondents that collaborated during implementation 0% 20% 40% 60% 80%100% Joint R&D project 88% Co-author research publication 88% Technological consultancy 25% Test of a new prototype 25% Source: World Bank. Capacity-building activities were common project outputs. The majority of respondents engaged in all three types of capacity building activities tracked by the researcher survey. Three quarters or more of respondents reported that they had completed training activities and attended domes- tic and international conferences, seminars, and workshops as part of their projects (Figure 5.12). Figure 5.12. Engagement in Capacity-Building Activities by Type Capacity building activities (Share of respondents) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Training activities 88% Seminars, workshops and conferences attended domestically 88% Seminars, workshops and conferences attended abroad 75% Source: World Bank. 5.3 Efficiency in Achievement of Outcomes Respondents reported a total of 254 outcomes (intended outcomes and other), for an average of 31.8 outcomes per project. Over that period, the program invested an average of 262,982 BGN in beneficiary projects, so respondents generated 12.1 outcomes per 100,000 BGN invested. Six outcomes were included for the ICT program, based on the program’s ToC: 1.) Scientific publi- cations in peer-reviewed journals, 2.) prototypes, 3.) new software development, 4.) new technol- ogy development, 5.) new open science repositories, and 6.) research collaborations after project completion (not included in this analysis given that all ICT projects are still ongoing). Bulgaria: Efficiency Analysis of STI Programs 113 Scientific publications in peer-reviewed journals was the most common outcome reported at 24.9 per project, followed by new software development (2.4 per project), prototypes (2), new open science repositories (1.6), and new technology development (0.5) (Table 5.4). Table 5.4. ICT Intended Outcomes Total number Average Outcome per Share of of outcomes outcomes per 100,000 BGN of respondents Intended outcome achieved respondent program cost reporting outcome Scientific publications in 199 24.9 9.5 88% peer-reviewed journals Prototype 16 2 0.8 25% New software development 19 2.4 0.9 63% New technology 4 0.5 0.2 25% development Open science repositories 13 1.6 0.6 50% Source: World Bank. Note: Open science repositories were only tracked for the ICT program An equal share of respondents produced fewer than 5 or more than 20 publications. Almost 40 per- cent of respondents produced more than 20 publications as part of their project, while 25 percent produced between five and 20 publications, and 38 percent produced less than five (Figure 5.13). Figure 5.13. Peer-Reviewed Publications Produced by ICT Beneficiaries Publications produced (Share of respondents) 0% 5% 10% 15% 20% 25% 30% 35%40% 5 or less 38% 5 – 20 25% more than 20 38% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 114 Publications reported by ICT respondents had the highest average year-normalized citations and highest share of publications with international coauthors of programs after those reported by Fundamental Research respondents. Almost half of the publications listed by respondents were indexed in Web of Science and 36 percent were in journals with impact factor (Table 5.5). How- ever, ICT publication generally had fewer year-normalized citations than the average Bulgarian publication indexed in Web of Science from 2016 to 2020. Table 5.5. Impact Measures of ICT Publications Total publications listed in survey responses 28 Share indexed in Web of Science 46% Share in journals with impact factor 36% Average year normalized citations 1.27 Share with international coauthor(s) 54% Source: World Bank. Other Results ICT respondents reported very few outcomes related to IP creation. (See Table 5.6.) The program has no objectives related to patents or technology transfer. Nevertheless, its goals of develop- ing new software and technologies for open science and digital education for public use seem like they would eventually lead to the creation and transfer of IP at some point during project implementation. Table 5.6. Other Outcomes Reported by ICT Respondents Total number of Average outcomes per Outcome per 100,000 Share of respondents Other outcomes outcomes achieved respondent BGN of program cost reporting outcome Patents grants 0 0 0.0 0% Patent applications 0 0 0.0 0% Other IP activities 3 0.4 0.2 13% Source: World Bank. Note: a. Other IP activities includes industrial designs, copyrights, transfer agreements, and so on. Bulgaria: Efficiency Analysis of STI Programs 115 5.4 Perceived Quality As mentioned earlier, beneficiaries of the program were preselected and there was no application process; therefore, beneficiaries of this program were not questioned about their perceptions of application and selection processes. Respondents were highly satisfied with the ICT program’s implementation and monitoring process- es, with more than 80 percent of respondents reporting satisfaction with all implementation pro- cesses except for data protection practices, where only 50 percent of respondents were satisfied. Most respondents felt that the financial resources provided by MoES were sufficient, with 88 percent reporting that they had sufficient funding to successfully complete their project. The one respondent who said that financial support was insufficient cited increased costs of inputs, issues with procurement, and lack of financial support from their institution as the reasons why the financial support provided by the program was not sufficient. Similarly, most respondents felt that the amount of time allowed by the program to complete their projects was sufficient, with 88 percent reporting that they had sufficient time to successfully complete their projects. Financial and human resources were commonly cited key success factors, but assistance with additional funding sources was the largest gap in program support. The availability of financial and human resources were cited as key success factors by 88 percent and 75 percent of respon- dents, respectively. At least half of respondents felt that support from their research institution and the availability of research infrastructure were also key success factors (Figure 5.14). When asked about any additional support needed during implementation that would have improved the outcomes of their projects, respondents cited assistance with finding additional funding sources (by 63 percent of respondents), assistance with the preparation of project budgets (by 38 percent), and assistance to hire foreign researchers (by 38 percent) as the most needed ad- ditional support (Figure 5.15). Bulgaria: Efficiency Analysis of STI Programs 116 Figure 5.14. Key Success Factors for ICT Projects Share of respondents 0% 20% 40% 60% 80%100% Availability of financial resources 88% Availability of human resources such as researchers or mentors 75% Support of the research institution 63% Availability of research infrastructure 50% The way support program is designed 25% and implemented Source: World Bank. Figure 5.15. Additional Support Needed for ICT Projects Share of respondents 0% 20% 40% 60% 80%100% Assistance with finding additional funding sources 63% Assistance in the preparation of project budgets 38% Assistance to hire foreign researchers 38% Assistance with procurement 25% Access to research infrastructure 25% and equipment Assistance to establish collaborations 25% Better administrative support 13% by the program Assistance in preparation of 13% monitoring reports Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 117 Overall Project Quality Respondent’s project objectives differed significantly from the objectives of the ICT program. The program primarily aims to support the development of new digital tools and software in the areas of open science and education, with objectives related to applied research and the development of new software, digital tools, and open science repositories. However, the most common proj- ect objectives reported by beneficiaries were developing a cadre of young researchers (reported by 75 percent of respondents), collaborating with other researchers or research institutions (75 percent), and publishing scientific papers in peer-reviewed journals (63 percent) (Figure 5.16). No respondents said that developing a new product, service, or process was an objective of their project, while only 38 percent said that upgrading a product, service, or process was an objective. This misalignment may stem from the fact that there was no application process for the program, and therefore beneficiary projects were not evaluated by MoES, but rather selected by members of the beneficiary consortium. This may be a weakness of this program management structure. Figure 5.16. ICT Program Project Objectives Project objectives (Share of respondents) 0% 20% 40% 60% 80%100% Develop a cadre of young researchers 75% Collaborate with other researchers or research institutions 75% Publish scientific papers in peer-reviewed journals 63% Upgrade a product, service, 38% or process Produce market-oriented research 25% Improve chances to get EU funding 13% Collaborate with the private sector 13% Present scientific papers in seminars 0% and conferences Develop a new product, service, or process 0% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 118 6. Electronic Health in Bulgaria Bulgaria: Efficiency Analysis of STI Programs 119 6. Electronic Health in Bulgaria Program Summary Electronic Health in Bulgaria (eHealth) is one of the sectoral programs of the National Science Programs 2018–22 portfolio. These programs, implemented by MoES, are focused on addressing identified public challenges in Bulgaria – in the case of the eHealth program, it is aimed at building infrastructure and tools collecting, storing, and analyzing health data. The eHealth program is also jointly implemented by the Ministry of Health The program funds ap- plied research projects in two core research areas: 1) the development of a software platform for collection, processing, and analysis of health data and 2) the development of an electronic platform for the management of large health data sets. The program has intended outputs related to the develop- ment of prototypes, new software, and new technologies. Like the other sectoral programs in the National Science Programs portfolio, eHealth provides funding to a consortium of preselected research institu- tions in specific scientific fields (medical and health sciences and computer and information sciences in the case of eHealth). The beneficiary consor- tium is made up of four lead research institutions and four secondary part- ner institutions. There was no application process or application costs for beneficiaries of the program. MoES also delegates a portion of the adminis- tration and monitoring of projects to the consortium, so some of the costs of implementing the overall program are borne by the beneficiaries. In the case of the eHealth program, the four leading beneficiaries of the consortium (Medical University – Sofia, University of Sofia, Medical University – Plovdiv, and the Bulgarian Academy of Science) are jointly responsible for distribut- ing funding to the other members of the consortium, collecting indicators on project progress, and compiling semi-annual technical and financial reports to the MoES and the Ministry of Health. Like the ICT program, administrative costs only made up only 0.2 percent of total program costs; however, but this extremely low share of administrative costs does not include the implementation costs of the program that are borne by the beneficiary consortiums. While the beneficiaries had no costs Bulgaria: Efficiency Analysis of STI Programs 120 related to the application processes, respondents reported expending an average of seven percent of grant funding received for administration and coordination of the consortium and an average of one percent of funding received for auditing (a requirement of the program). Scientific publications in peer-reviewed journals were the most common out- come reported by eHealth respondents, but respondents also reported a rela- tively high number of outcomes related to new software and new technology development, which are the primary intended outcomes of the program given its focus on applied research. Like ICT, the eHealth program also generated a relatively high number of collaborations during implementation, which is likely due in part to the design of the program – beneficiaries are members of con- sortiums that collaborate with each other on individual projects. The program was started in 2018 and issued a grant to the consortium in 2019 of 2 million BGN (all of which million BGN of which has been disbursed). The principal investigators (or alternative points of contact) for the eight of the members of the eHealth consortium were surveyed for this analysis, with four beneficiaries responding to the survey for a response rate of 50 percent. All of the funded projects are in the scientific research areas of mathematical science and informatics and healthcare. Bulgaria: Efficiency Analysis of STI Programs 121 6.1 Efficiency in Use of Inputs Administrative costs were a very small portion of total program costs, which were highest in the year the program launched. The sectoral programs under the National Science Programs 2018–22 portfolio have similar implementation processes at the ministry level, so the administrative costs for the eHealth program are nearly identical to those of the ICT program. Grants distributed to beneficiaries represented 99.8 percent of the costs of the ICT program from 2018 to 2020, while administrative costs made up only 0.2 percent of program costs (Figure 6.1). Grant disbursement was highest in 2018, before dropping in 2019 and in 2020 (Figure 6.2). Figure 6.1. eHealth Program Costs by Category, 2018–20 0,2% 99,8% Administrative Costs Source: World Bank. Grants Distributed Figure 6.2. eHealth Program Costs by Year, 2018–20 900.000 800.000 700.000 Constant 2020 BGN 600.000 500.000 400.000 300.000 200.000 100.000 0 2018 2019 2020 Source: World Bank. Administrative Costs Grants Distributed Bulgaria: Efficiency Analysis of STI Programs 122 Administrative Costs Personnel costs made up 95 percent of total administrative costs, with fixed costs representing the remaining five percent of administrative costs services. There were no costs related to exter- nal services for the program (Figure 6.3). Personnel costs were highest in 2019 – the first full year after the 10 projects were active – before declining slightly in 2020 (Figure 6.4). Figure 6.3. eHealth Program Administrative Costs by Category, 2018–20 5% 95% Personnel Source: World Bank. Fixed Costs Figure 6.4. eHealth Program Administrative Costs by Year, 2018–20 2.500 Constant 2020 BGN 2.000 1.500 1.000 500 0 2018 2019 2020 Source: World Bank. Personnel Fixed Costs Bulgaria: Efficiency Analysis of STI Programs 123 Costs related to implementation made up over 75 percent of personnel costs for the program, with the remainder coming from costs for M&E. (See Figure 6.5.) All of the costs for M&E came after 2018 (the year the projects were awarded and launched), while costs related to implemen- tation have remained relatively constant from 2018 to 2020 (Figure 6.6). Figure 6.5. eHealth Program Personnel Costs by Category, 2018–20 24% 76% Implementation M&E Source: World Bank. Figure 6.6. eHealth Program Personnel Costs by Year, 2018–20 2.500 Constant 2020 BGN 2.000 1.500 1.000 500 0 2018 2019 2020 Source: World Bank. Implementation M&E Bulgaria: Efficiency Analysis of STI Programs 124 Fixed costs have risen steadily since 2018, although they are still less than 200 BGN (approxi- mately €102) per year (Figure 6.7). Due to data availability, it was not possible to separate out fixed costs by category (such as costs for office equipment, goods and services, ICT, and so on). Figure 6.7. eHealth Program Fixed Costs by Year, 2018–20 200 180 160 140 Constant 2020 BGN 120 100 80 60 40 20 0 2018 2019 2020 Source: World Bank. The total administrative costs of implementing the program from 2018 to 2020 were 5,790 BGN, with an average cost per project of 724 BGN. (See Table 6.1.) However, the extremely low average administrative cost per project is misleading – as discussed in the next section, a portion of the administrative and implementation costs of the program are borne by the beneficiary consortiums, so the full cost of program implementation is not reflected here. By comparison, the administra- tive average cost per project was 724 BGN, but beneficiaries spent an average of 15,000 BGN per project on management and auditing costs. Table 6.1. eHealth Administrative Costs Cost category Total costs Cost per project Personnel 5,487 BGN 686 BGN Fixed 302 BGN 38 BGN External services 0 BGN 0 BGN Source: World Bank. Note: All amounts in constant 2020 BGN. Bulgaria: Efficiency Analysis of STI Programs 125 Costs Covered by Beneficiaries As discussed previously, the National Science Programs have a different design than tradition- al grant programs—each provides funding to a preselected consortium of beneficiary HEIs and PROs. Much of the administration, monitoring, and auditing of the program is actually done by the consortium, rather than the MoES. Administration costs to the Ministry are low, but the ben- eficiaries bear additional costs for program management. There were no application costs associated with the program. The eHealth program, like the other programs in the National Science Programs 2018–22 portfolio, has no application process. Instead, beneficiaries are preselected to participate based on indicators related to published scientific results in the fields of informatics, data science, and healthcare. Respondents reported no costs related to preparing their applications. Although the eHealth program has no matching requirement, respondents reported an average in kind contribution of 6,000 BGN and an average cash contribution of 5,000 BGN to the imple- mentation of their projects. (See Figure 6.8.) Half of respondents made in kind contributions to their projects, 25 percent made cash contributions, and the remaining 25 percent made no con- tributions to their projects. Figure 6.8. Beneficiary Contributions to eHealth Projects Average contribution size, BGN 0 1000 2000 3000 4000 5000 60007000 in kind cash Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 126 Beneficiaries also used grant funding for administration, coordination, and auditing. Beneficia- ries reported expending an average of 7 percent of grant funding received for administration and coordination of the consortium, for an average of 13,125 BGN per beneficiary, and an average of 1 percent of funding received for auditing (a requirement of the program), for an average cost of 1,875 BGN per beneficiary. These additional costs are due to the design of the National Sci- ence Programs, which delegate a portion of the management of the program to the beneficiary consortiums. On average, the costs covered by the program were 9.7 times higher than the costs covered by respondents. The average cost covered by the program, including disbursed grants and adminis- trative costs, was 250,724 BGN, while the average cost covered by beneficiaries was 26,000 BGN. 6.2 Efficiency in Generation of Outputs For every BGN of administrative costs, beneficiaries received an average of 346 BGN. The total program costs from 2018 to 2020 was 2 million BGN or 501,448 BGN per project. This includes 2 million in disbursed grants to beneficiaries and 5,790 BGN in administrative costs (Table 6.2). The program did not include any nonfinancial or indirect financial support to beneficiaries. Table 6.2. eHealth Program Costs, 2018–20 Total Average per project Grant funding disbursed 2,000,000 BGN 250,000 BGN Administrative and operating costs 5,790 BGN 724 BGN Total program costs 2,005,790 BGN 250,724 BGN Source: World Bank. Note: All amounts in constant 2020 BGN. Bulgaria: Efficiency Analysis of STI Programs 127 Researcher salaries were the largest expenditure reported by respondents at 40 percent of to- tal project costs. Machinery and equipment (11 percent), IT systems (11 percent), and travel (10 percent) were the next largest expenses by size (Figure 6.9). Figure 6.9. eHealth Project Expenditures by Cost Category Share of expenditure 0% 5% 10% 15% 20% 25% 30% 35% 40%45% Researchers’ salaries 40% Machinery, equipment, tools 11% IT systems, specialized software, IT licenses, websites 11% Travel (fairs, exhibitions, conferences, etc.) 10% Materials, consumables, inventory 9% Other (Administration and Auditing) 8% Training and events 5% Marketing campaigns or PR 4% activities to promote the project Testing and certifications 1% Intellectual property 1% (patents, trademarks, copyrights) Source: World Bank. Outputs Achieved Three outputs were tracked for the eHealth program based on the program ToC: 1.) project dis- semination (in the form of seminars, workshops, and conferences), 2.) training activities, 3.) and research collaborations during implementation. Program respondents generated a total of 188 outputs from 2018 to 2020, or an average of 23.6 outputs per project. Over that period, the program invested an average of 250,724 BGN in bene- ficiary projects, so respondents generated 9.4 outputs per 100,000 BGN invested. Seminars, workshops, and conferences were the most common output of eHealth respondents (an average of 12.6 per project), followed by collaborations during implementation (10.2), and training activities (0.8) (see Table 6.3). Bulgaria: Efficiency Analysis of STI Programs 128 Table 6.3. eHealth Outputs Total number of Average outputs per Outputs per 100,000 Intended outcome outputs achieved respondent BGN of program cost Seminars, workshops, and conferences 101 12.6 5.0 Training activities 6 0.8 0.3 Collaborations 81 10.2 4.1 Source: World Bank. Note: Outputs were reported at the level of the entire consortium, not for individual respondents, so it was not possible to calculate the share of respondents that reported a specific output. Every eHealth respondent engaged in collaborations with other members of the eHealth consor- tium and with other domestic research partners. Collaborations with domestic industry partners, foreign industry partners, and foreign research partners were also relatively common (Figure 6.10). Figure 6.10. eHealth Project Collaborations by Type of Partner Average number of collaborations 7 100% 6 Share of respondents 80% 5 4 60% 3 40% 2 20% 1 0 0% Other ICT domestic foreign diaspora domestic foreign industry industry consortium research research research industry partners partners members partners partners partners partners diaspora Source: World Bank. Number of collaborations Share of respondents Bulgaria: Efficiency Analysis of STI Programs 129 Collaborations were rated highly, except for those with domestic researchers. All respondents expressed satisfaction with collaborations with other members of the eHealth consortium, for- eign research partners, and domestic industry partners. However, only 25 percent of respondents were satisfied with their collaborations with domestic research partners (Figure 6.11). Prototypes and co-authorships were the most common type of collaborations. Half of respondents engaged in collaborations related to testing a new prototype and co-authoring a research publication, while 25 percent collaborated in a joint R&D project and technological consultancy (Figure 6.12). Figure 6.11. Satisfaction with the Quality of eHealth Collaborations 100% 90% Quality of collaborations 80% (share of respondents) 70% 60% 50% 40% 30% 20% 10% 0% Other ICTconsortium domestic research foreign research domestic industry members partners partners partners Source: World Bank. Note: Left figure shows the share of respondents who rated their collaborations as above average or excellent Figure 6.12. eHealth Collaborations by Type Share of respondents that collaborated during implementation 0% 20% 40% 60% 80%100% Test of a new prototype 50% Co-author research publication 50% Joint R&D project 25% Technological consultancy 25% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 130 6.3 Efficiency in Achievement of Outcomes Respondents reported a total of 135 outcomes (intended outcomes and other), for an average of 16.9 outcomes per project. Over that period, the program invested an average of 501,448 BGN in beneficiary projects, so respondents generated 6.7 outcomes per 100,000 BGN invested. Five outcomes were included for the eHealth program, based on the program’s ToC: 1.) publications in peer-reviewed journals, 2.) prototypes, 3.) new software development, 4.) new technology de- velopment, and 5.) research collaborations after project completion (not included in this analysis given that all eHealth projects are still ongoing). Publications in peer-reviewed journals were the most common outcome reported at 11.8 per project, followed by new technology development (2.3 per project), new software development (1.9 per project), and prototypes (1 per project) (Table 6.4). Table 6.4. Intended Outcomes Reported by eHealth Respondents Total number of Average outcomes Outcome per 100,000 Intended outcome outcomes achieved per respondent BGN of program cost Scientific publications in peer-reviewed journals 94 11.8 4.7 Prototype 8 1 0.4 New software development 15 1.9 0.8 New technology development 18 2.3 0.9 Source: World Bank. Note: Outcomes were reported at the level of the entire consortium, not for individual respondents, so it was not possible to calculate the share of respondents that reported a specific outcome. A lower share of the publications reported by eHealth respondents were in journals with impact factor than for the publications reported by respondents of the other programs included in this analysis – however, several of the publication listed in survey responses were still under review and may yet be published in high impact journals. None of the indexed publications reported by eHealth respondents have been cited as of the writing of this report (Table 6.5). Table 6.5. Impact Measures of eHealth Publications Total publications listed in survey responses 16 Share indexed in Web of Science 38% Share in journals with impact factor 13% Average year normalized citations 0 Share with international coauthor(s) 33% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 131 6.4 Perceived Quality Respondents were highly satisfied with the eHealth program’s implementation and monitoring processes. All respondents expressed satisfaction about all processes in the beneficiary survey. All respondents felt that the that the financial resources provided by MoES were sufficient to suc- cessfully complete their projects. Similarly, all respondents felt that the allowed by the program to complete their projects was sufficient. All respondents cited financial and human resources as key success factors for their projects, but assistance with procurement and establishing connections were the most common gaps in support. The availability of research infrastructure was cited by three quarters of respondents as a success factor (Figure 6.13). When asked about any additional support needed during imple- mentation that would have improved the outcomes of their projects, respondents cited assis- tance in establishing collaborations and assistance with procurement (both cited by 75 percent of respondents) as the most needed additional supports (Figure 6.14). Half of respondents also said they needed support in finding additional funding sources for their projects. Figure 6.13. Key Success Factors for eHealth Projects Share of respondents 0% 20% 40% 60% 80%100% Availability of financial resources Availability of human resources such as researchers or mentors Availability of research infrastructure The way support program is designed and implemented Support of the research institution Source: World Bank. Figure 6.14. Additional Support Needed for eHealth Projects Share of respondents 0% 20% 40% 60% 80%100% Assistance with procurement Assistance to establish collaborations Assistance with finding additional funding sources Assistance to hire foreign researchers Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 132 Overall Project Quality Respondents’ project objectives were largely in line with the objectives of the eHealth program. The program supports the development of software, tools, and platforms for the collection, anal- ysis, and management of health data, and 75 percent of respondents said one of the primary objectives of their projects was to develop a new product, service, or process. Other objectives reported by respondents include publishing scientific papers in peer-reviewed journals (reported by 100 percent of respondents), developing a cadre of young researchers (75 percent), collaborat- ing with other researchers or research institutions (25 percent), and establishing links to medical practitioners (25 percent) (Figure 6.15). Figure 6.15. eHealth Program Project Objectives Share of respondents 0% 20% 40% 60% 80%100% Publish scientific papers in peer-reviewed journals Develop a cadre of young researchers Upgrade a product, service, or process Collaborate with other researchers or research institutions other Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 133 7. National Innovation Fund Bulgaria: Efficiency Analysis of STI Programs 134 7. National Innovation Fund Program Summary The National Innovation Fund (NIF) is a long-running program focused on supporting industrial research in Bulgaria and is also one of the primary in- struments of the country’s Smart Specialization strategy. NIF is implemented by the Bulgarian SME Promotion Agency and provides grants for re- search and development projects and technology feasibility projects. The program pro- vides matching grants to firms for industrial research and early-stage innovation projects and not fund projects beyond the stage of experimental development9. It thus has intend- ed outcomes related to IP generation; prototypes; new products, services, and processes; and increased revenues, exports, and employment for beneficiary companies. Applica- tion costs were higher than those reported by research respondents, but lower than the DPPI program. Higher application costs were likely due to the large share (85 percent) of respondents that used some form external support (such as legal or accounting services) to prepare their applications. When looking at the costs to implement the program, exter- nal services made up a large share of total administrative costs (44 percent), compared to personnel costs and fixed costs. This relatively large share of costs for external services is likely due to the low number of full-time staff at SMEPA. More than half of respondents’ projects had results in the early stages of commercializa- tion (such as a prototype or proof of concept), which is likely due to the fact that the pro- gram does not fund projects beyond the experimental development stage. Collaborative projects after implementation were the most common outcome reported; followed by employees hired; new products, services, and processes; and prototypes. Respondents were dissatisfied with several key processes related to project application and selection, including ease of the overall application process, feedback on project selection, and the transparency of the selection process. However, among respondents that had completed their projects, more than 60 percent of respondents felt that the outcomes of their project matched their expectations 9 According to program documentation, experimental development may include prototyping, demonstration, development of pilot projects, testing and validation of new or improved products, processes, and services in an environment that is representative of real-life operating conditions, where the main goal is further technical improvement. This may include the development of commer- cially usable prototypes or pilot projects that are necessary for the final commercial product and whose production is too expensive to be used only for demonstrations and validation. Bulgaria: Efficiency Analysis of STI Programs 135 The NIF program started in 2005 and has implemented 10 calls for proposals since its inception. Between 2016 and 2020, NIF funded 81 projects over three calls for proposals for a total 17.1 mil- lion BGN in grants contracted (13.9 million BGN has been disbursed). The designated points of contact for all 82 projects funded by the program from 2016 to 2020 were surveyed for this analysis, with 44 beneficiaries responding to the survey for a response rate of 52 percent. Respondents represent 65 percent of the contracted funding for the program over that period with an average grant size of 259,710 BGN. Administrative cost data were only pro- vided for the years 2018 to 2020, so indicators related to efficiency (outputs/outcomes per cost) only include project results data for those years. The main economic activity areas of responding firms were manufacturing (40 percent of respondents), professional services (19 percent), and healthcare (16 percent) (Figure 7.1), while the main Smart Specialization priority areas of respon- dents’ projects were informatics and ICT (41 percent of respondents) mechatronics (33 percent), and biotechnologies (21 percent) (Figure 7.2). Figure 7.1. NIF Respondents by Main Economic Activity Share of respondents 0% 10% 20% 30% 40%50% Manufacturing 42% Professional, Scientific and Technical Activities 19% Healthcare and Social Work Activities 16% Information and Communication 6% Real Estate 6% Utilities 3% Wholesale and Retail Trade; Vehicles Repair 3% Financial and Insurance Activities 3% Figure 7.2. NIF Respondents by Smart Specialization Priority Area Share of respondents 0% 10% 20% 30% 40%50% Informatics and ICT 41% Mechatronics and clean technologies 33% Healthy life and biotechnology industries 21% New technologies in creative and re-creative industries 5% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 136 7.1 Efficiency in Use of Inputs Grants distributed to beneficiaries made up 92 percent of the costs of the program from 2018 to 2020, while administrative costs made up the remaining eight percent (Figure 7.3). Administra- tive costs have remained relatively constant from 2018 to 2020, while grant disbursements grew steadily from 2018 to 2020, increasing the overall costs of the program (Figure 7.4). Figure 7.3. NIF Program Costs by Category, 2018–20 8% 92% Administrative Costs Source: World Bank. Grants disbursed Figure 7.4. NIF Program Costs by Year, 2018–20 7.000.000 6.000.000 5.000.000 4.000.000 3.000.000 2.000.000 1.000.000 0 2018 2019 2020 Source: World Bank. Administrative Costs Grants disbursed Bulgaria: Efficiency Analysis of STI Programs 137 Administrative Costs External services made up almost half of administrative costs, which were highest in 2020. Ex- ternal services, in the form of contracts to technical experts to evaluate proposals and monitor project progress, made up 44 percent of program costs, while personnel costs were 31 percent and fixed costs were 24 percent of total administrative costs from 2018 to 2020 (Figure 7.5). Personnel costs increased from 2018 to 2020, while fixed costs were highest in 2018 and 2020 (Figure 7.6). Figure 7.5. NIF Administrative Costs by Category, 2018–20 44% 31% Personnel Fixed Costs Source: World Bank. 24% External Services Figure 7.6. NIF Administrative Costs by Year, 2018–20 500 Costs, thousnad BGN 2020 constant 450 400 350 300 250 200 150 100 50 0 2018 2019 2020 Source: World Bank. Personnel Fixed Costs External Services Bulgaria: Efficiency Analysis of STI Programs 138 Implementation made up more than half of personnel costs from 2018 to 2020, while M&E made up 35 percent, design made up one percent, and the remaining ten percent went to other person- nel costs (such as legal and administrative staff) (Figure 7.7). Personnel costs rose each year from 2018 to 2020, driven by increases in implementation and M&E costs over those years (Figure 7.8). Figure 7.7. NIF Personnel Costs by Category, 2018–20 1% 10% 35% 54% Design Implementation M&E Source: World Bank. other Figure 7.8. NIF Personnel Costs by Year, 2018–20 160.000 140.000 120.000 100.000 80.000 60.000 40.000 20.000 0 2018 2019 2020 Source: World Bank. Design Implementation M&E other Bulgaria: Efficiency Analysis of STI Programs 139 Goods and services was the largest cost category of fixed costs, representing 44 percent of to- tal fixed costs, while ICT software and licenses made up 31 percent and office equipment made up 24 percent of total fixed costs (Figure 7.9). Fixed costs declined every year from 2018 to 2020, driven by lower costs for office equipment in 2019 and 2020 (Figure 7.10.). Figure 7.9. NIF Fixed Costs by Category, 2018–20 0% 24% 45% 31% Office equipment ict Goods and Services Source: World Bank. Office space Figure 7.10. NIF Fixed Costs by Year, 2018–20 140.000 120.000 100.000 80.000 60.000 40.000 20.000 0 2018 2019 2020 Source: World Bank. Office equipment ict Goods and Services Office space Bulgaria: Efficiency Analysis of STI Programs 140 “Other” external services, such as legal and administrative services, made up the largest share of external services costs from 2018 to 2020, followed by costs related to experts for the evaluation of project proposals and experts for M&E of projects (Figure 7.11). Costs for external services were lowest in 2019 due to lower “other” services costs that year, while costs for experts for evaluating proposals and for M&E were relatively constant from 2018 to 2020 (Figure 7.12). Figure 7.11. NIF External Services Costs by Category, 2018–20 42% 34% Experts for evaluation of proposals Experts for M&E Source: World Bank. 23% Other Figure 7.12. NIF External Services Costs by Year, 2018–20 250.000 200.000 150.000 100.000 50.000 0 2018 2019 2020 Source: World Bank. Experts for evaluation of proposals Experts for M&E Other Bulgaria: Efficiency Analysis of STI Programs 141 The total administrative cost of implementing the program from 2018 to 2020 was 1.2 million BGN, with an average cost per project of 18,330 BGN (Table 7.1). The average administrative cost per project was significantly higher for NIF than the DPPI program (18,330 BGN per project for NIF vs. 2,308 BGN for DPPI), despite the fact that DPPI grants are considerably larger than NIF grants. This larger administrative cost per project may be due to the fact that NIF issued multiple calls for proposals from 2018 to 2020, while DPPI only issued a single call from proposals in 2019. Table 7.1. NIF Administrative Costs Cost category Total costs Cost per project Personnel 384,346 5,737 Fixed 299,789 4,474 External services 543,956 8,119 Source: World Bank. Note: All amounts in constant 2020 BGN. Costs Covered by Beneficiaries The application process for NIF is slightly less costly and less time intensive than that for the other firm support program included in this analysis (DPPI). Respondents reported an average application cost of 7,195 BGN and spent an average of 31 working days preparing their applications to NIF. However, the average grant size of the DPPI program is significantly larger than NIF, so NIF application costs are actually larger as a share of average contracted grant than DPPI. Over 60 percent of respondents spent less than 5,000 BGN in preparing their application, while 17 percent spent more than 10,000 BGN (Figure 7.13). More than 60 percent of respondents reported spend- ing 30 working days or fewer in preparing their application, while eight percent spent more than 60 working days (Figure 7.14). Figure 7.13. NIF Program Application Costs Share of respondents 0% 20% 40% 60%80% 5,000 or less 64% 5,001 – 10,000 19% 10,001 – 20,000 6% More than 20,000 11% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 142 Figure 7.14. NIF Program Application Time Share of respondents 0% 20% 40%60% 10 days or fewer 8% 11 – 30 days 57% 30 – 60 days 27% More than 60 days 8% Source: World Bank. Most respondents used some type of external support to prepare their applications. About 85 percent of respondents reported that they used some type of external support (such as legal, ac- counting, or other services) to prepare their application, which likely contributed to the relatively higher costs of applying to the program (Figure 7.15). The most common types of support required were experts or consultants (needed by 31 percent of respondents), accountants (28 percent), and administrative assistants (16 percent). Figure 7.15. External Support Used to Prepare NIF Applications Share of respondents 0% 5% 10% 15% 20% 25% 30%35% Experts / consultants 31% Accountant 28% Administrative assistant 16% Lawyer or legal consultant 7% The project team prepared 16% the application without any additional help Source: World Bank. Most beneficiaries made substantial cash and in-kind contributions to their projects. Respon- dents reported an average of 130,000 BGN cash and 50,000 BGN in kind contributions to the im- plementation of their projects (Figure 7.16). The NIF program funds between 25 and 80 percent of total project costs, depending on the type of project, size of firm, and other factors, while the firm is responsible for funding the remainder of project costs. More than 80 percent of respon- dents made both cash and in-kind contributions to the project, while 14 percent made only cash contributions (Figure 7.17). Bulgaria: Efficiency Analysis of STI Programs 143 Figure 7.16. Beneficiary Contributions to NIF Projects by Type and Amount 0 50.000 100.000150.000 Contributions in cash Contributions in kind Source: World Bank. Figure 7.17. Beneficiary Contributions to NIF Projects by Type and Frequency 5% 14% 81% No contributions reported Cash In kind Source: World Bank. Both cash and in kind On average, the costs covered by the program were slightly higher than the costs covered by re- spondents. The average cost covered by the program, including disbursed grants and adminis- trative costs, was 222,092 BGN, while the average cost covered by beneficiaries was 201,044 BGN. Therefore, for every BGN that beneficiaries put into the project, the program invested 1.01 BGN. This distribution in project costs is in line with one of the key objectives of the NIF program – to spur private investment in industrial R&D. 7.2 Efficiency in Generation of Outputs For every BGN the program invested in administrative costs, beneficiaries received an average of 11.1 BGN. The total program costs from 2018 to 2020 was 14.8 million BGN, or 222,092 BGN per project. This includes 13.7 million in disbursed grants to beneficiaries and 1.2 million BGN in ad- ministrative costs (Table 7.2). The program did not include any nonfinancial or indirect financial support to beneficiaries. Salaries made up more than 50 percent of expenditures reported by respondents. Materials, sup- plies, and inventory (19 percent) and machinery, equipment, and instrumentation (13 percent) were also significant expenses reported by respondents (Figure 7.18). Bulgaria: Efficiency Analysis of STI Programs 144 Table 7.2. NIF Program Costs, 2018–20 Total Average per project Grant funding disbursed 13,652,088 BGN 203,763 BGN Administrative and operating costs 1,228,091 BGN 18,329 BGN Total program cost 14,880,180 BGN 222,092 BGN Source: World Bank. Note: All amounts in constant 2020 BGN. Figure 7.18. NIF Project Expenditures by Cost Category Share of expenditures 0% 10% 20% 30% 40% 50%60% salaries 57% Materials, supplies, inventory 19% Machinery, equipment, instrumentation 13% Consulting services (e.g. feasibility studies, survey companies) 3% SPACE, RENT 3% IT systems, specialized software, 2% IT licenses, websites Testing and certifications 1% Access to specialized equipment 1% or laboratories Other 1% (please specify in space below) Intellectual property 0% (patents, trademarks, copyrights) Marketing campaigns or PR 0% activities for project visibility Training 0% Travel (fairs, exhibitions, 0% conferences, etc.) Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 145 Outputs Achieved Two outputs were tracked for the NIF program, based on the program’s ToC: 1.) collaborations during project implementation and 2.) improved capabilities of employees. Program respondents generated a total of 332 outputs from 2018 to 2020, or an average of 5 out- puts per project. Over that period, the program invested an average of 222,092 BGN in beneficiary projects, so respondents generated 1.6 outputs per 100,000 BGN invested. Improved capabilities of employees were the most common type of output reported (an aver- age of 3 per project), followed by collaborations during implementation (2 per project) (Table 7.3) Almost 70 percent of respondents engaged in some type of collaboration during their projects. The most common collaborations, by both average number of collaborations per project and share of respondents who engaged in such a collaboration, were collaborations with domestic researchers. Collaborations with diaspora industry partners and with foreign research partners were also fairly common among respondents (Figure 7.19). Table 7.3. Outputs of NIF Programs Outputs per Share of Total number of Average outputs 100,000 BGN of respondents Output outputs achieved per respondent program cost* reporting outcome Collaborations during implementation 73 2 0.7 68% Improved capabilities of employees 109 3 0.9 27% Source: World Bank. *The outputs per 100,000 of program cost indicator only includes data from survey respondents from 2018-2020 because administrative cost data were only available for those years. Figure 7.19. NIF Project Collaborations by Type of Partner 1,0 100% 0,9 90% 0,8 80% 0,7 70% 0,6 60% 0,5 50% 0,4 40% 0,3 30% 0,2 20% 0,1 10% 0 0% With domestic With foreign With diaspora With domestic With foreign With diaspora research research research industry industry industry partners partners partners partners partners partners Source: World Bank. Number of collaborations Share of respondents Bulgaria: Efficiency Analysis of STI Programs 146 Over 25 percent of respondents reported improving the capabilities of their employees during project implementation, increasing to 61 percent of respondents improving the capabilities of their employees after project completion. 7.3 Efficiency in Achievement of Outcomes Eight outcomes were tracked for the NIF program, based on the program’s ToC: 1.) Prototypes; 2.) patent applications; 3.) patents granted; 4.) collaborative projects after implementation; 5.) new products, processes, and services; 6.) increased company sales; 7.) improved company export performance; and 8.) new employees hired. While intended outcomes were generally lower on a per project basis than the DPPI program, NIF respondents were more efficient in producing outcomes per cost due to the higher costs of the DPPI program. Respondents reported a total of 626 outcomes (intended outcomes and other), for an average of 15 outcomes per project. From 2018 to 2020, the program invested an average of 222,092 BGN in beneficiary projects, so respondents generated 5.1 outcomes per 100,000 BGN invested over that time period. Collaborative projects after implementation were the most common outcome reported, with an average of 2.4 per project; followed by employees hired (1.8 per project); new products, services, and processes (1.3 per project); and prototypes (1.2 per project) (Table 7.4). Table 7.4. Intended Outcomes of NIF Respondents Total number Average Outcomes per Share of of outcomes outcomes per 100,000 BGN of respondents Intended outcome achieved respondent program cost reporting outcome Prototypes 51 1.2 0.4 68% Patent applications 26 0.6 0.1 30% Patents granted 23 0.5 0.1 30% Other IP activitiesa 29 0.7 0.1 30% Collaborative projects after implementation 105 2.4 1.1 50% New products, processes, or services 86 1.3 0.7 Company increased sales 14 4.0% increase 0.1 32% Company improved export performance 11 4.2% increase 0.1 25% Company hired additional employees 78 1.8 0.8 32% Source: World Bank. Note: a. Other IP activities includes industrial designs, transfer agreements, and so on. Bulgaria: Efficiency Analysis of STI Programs 147 Most respondents had project results in the early stages of commercialization. More than half of respondents had project results in the early stages of commercialization (such as a prototype or proof of concept), which is likely due to the fact that the program does not fund projects beyond the experimental development stage. Thirty-three percent of respondents reported a working prototype, and 28 percent reported a proof of concept as results of their projects. Only 13 per- cent of respondents said their project resulted in a product or service that is currently being sold, while another 13 percent said they had a product or service that is that is ready to be sold. Ten percent of respondents said their project results were not commercialized nor were likely to be commercialized (Figure 7.20). However, a significant number of respondents had not taken steps to selling their project results When asked about steps taken toward selling a product or services, 35 percent of respondents had not taken any steps as of the time they were surveyed. Again, this is likely due to the ear- ly-stage nature of the projects funded by the NIF program. Over one quarter of respondents had discussions with a vendor or firm to sell their products or services, while 14 percent had present- ed their product or service on the domestic market (Figure 7.21). Figure 7.20. NIF Project Results by Stage of Commercialization Share of respondents 0% 10% 20% 30%40% Have a working prototype that is almost ready to be sold 33% Have a proof of concept for a product or process that can be sold in the FUTURE 28% Have a product, service or process that is being sold 13% Have a product, service or process 13% that is ready to be sold The results that came out of this project will NOT be 10% COMMERCIALIZED OR ARE NOT LIKELY TO BE COMMERCIALIZED Figure 7.21. Steps Taken to Sell NIF Project Results Share of respondents 0% 10% 20% 30%40% Have not taken any steps 35% Discussions/negotiations with a vendor or firm that will sell my product, service or process 27% Presented my product or process in domestic market 14% Participated in trade fairs to showcase my product, 11% service, or process Participated as advertiser in scientific/technical 0% conferences Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 148 Other Results NIF respondents reported a number of other results not tracked by the program’s ToC, including adoption of new technologies, improved productivity, and technology upgrades. The adoption of new technologies was the most common other result reported (1.2 per project), followed by up- graded products, services, and processes (0.9 per project). Respondents also reported an average increase in productivity of 6.5 percent and a reduction in production costs of 1.8 percent (Table 7.5). Table 7.5. Other Results Achieved by NIF Respondents Total number Average Outcomes per Share of of outcomes outcomes per 100,000 BGN respondents Other outcome achieved respondent of cost reporting outcome Upgraded products, processes, or services 40 0.9 0.6 Company adopted a new technology 54 1.2 0.5 68% New business model 13 0.3 0.1 26% Expansion to new markets 29 0.7 0.3 59% Improved Productivity 20 6.5 % improvement 0.2 45% Reduced production costs 10 1.8% reduction 0.1 23% New enterprise or business spinoff 15 0.3 0.1 18% Reorganized the firm or part of the firm 7 0.2 0.1 11% Develop a new innovation unit in the firm 15 0.3 0.1 61% Source: World Bank. 7.4 Perceived Quality Respondents were dissatisfied with several key processes related to project application and selection. Areas of dissatisfaction included ease of the overall application process, feedback on project selection, and the transparency of the selection process. While respondents expressed satisfaction with many of the program’s application and selection processes, fewer than 60 per- cent of respondents were satisfied with selection transparency, contract negotiations, feedback on selection, and the overall application process (Figure 7.22). NIF requires applicants to provide a detailed breakdown of every participating researchers’ hours of work for the duration of the project, which can add to the overall time and effort required to apply to the program. The most recent call for proposals for the program did allow applicants to submit their applications elec- tronically (where all previous calls for proposals required paper submissions), which is a positive step in reducing the administrative burden on applicants. Respondents were largely satisfied with program objectives, rules, eligible costs, and other key application and selection processes. Bulgaria: Efficiency Analysis of STI Programs 149 Figure 7.22. Satisfaction with NIF Application and Selection Procedures Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Ease in accessing the program regulations Clarity of selection process 98% Adequate time for contract negotiation 95% Clarity of eligibility criteria 93% Well explained application procedure 91% Appropriate list of eligible costs 84% Ease of obtaining information about the program 82% Clarity of program regulations 82% Clarity of program objectives 80% Flexibility of rules for non-compliance 80% Sufficiency of application period 73% Appropriate information requirements in proposals 73% Adequate application to final results of selection time 73% Adequate selection to funding disbursement time 73% Timely and relevant application support 68% Fairness of eligibility criteria 68% Transparency of selection process 59% Ease of contract negotiation 52% Availability of feedback on project selection approval 48% Ease of application procedure 43% Source: World Bank. Note: Figure shows percentage of respondents who agreed or strongly agreed with the statements asking about their satisfaction with aspects of the application and selection process. Bulgaria: Efficiency Analysis of STI Programs 150 When asked if they think any program selection or eligibility criteria should be added, changed, or removed; five percent of respondents said that program eligibility criteria should be modified, with respondents saying that the criteria be modified to allow more types of research partner- ships and that the criteria that provide additional points to firms that had participated in previ- ous calls should be removed. Respondents were largely satisfied with the program’s implementation and monitoring process- es, although less than 60 percent of respondents expressed satisfaction with financial reporting requirements and data protection practices. Unlike beneficiaries of EU operational programmes, there is no online portal with preloaded report templates for NIF beneficiaries to report on project progress. Respondents were very satisfied with the expert feedback provided during monitoring visits and the administrative support provided by the program, with more than 90 percent of re- spondents saying they were satisfied with these aspects of the program (Figure 7.23). Figure 7.23. Satisfaction with NIF Implementation Processes Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Appropriate expert feedback from monitoring (visits, reports, discussions) 97% Sufficiency of administrative support provided 95% Timeliness of financial support 77% Appropriate monitoring requirements (e.g. narrative, indicators, etc.) 72% Accessibility of administrative support 66% Satisfactory data protection practices 54% Appropriate financial reporting requirements 51% Source: World Bank. Note: Figure shows percentage of respondents who agreed or strongly agreed with the statements asking about their satisfaction with program implementation and monitoring processes. Bulgaria: Efficiency Analysis of STI Programs 151 Most respondents felt that the financial resources provided by the program were sufficient, with 70 percent reporting that they had sufficient funding to successfully complete their project. Re- spondents who said that the financial support provided was not sufficient said that financial sup- port would need to increase by an average of 52 percent in order to successfully complete their project. The occurrence of unexpected costs, increased costs of inputs, and inability to secure co-financing were cited by respondents the most common reasons why the amount of financial support provided by the program was not sufficient (Figure 7.24). Figure 7.24. Reasons Why NIF Respondents Found That Financial Support Provided was Insufficient Share of respondents 0% 10% 20% 30% 40% 50% 60%70% Unexpected costs occured 60% Increased costs of inputs 40% Unable to get financial support from my institution 40% Inadequate budget planning 30% Budget clearing by NIIF was inadequate 30% Unable to ensure cofinancing 20% Issues with procurement 20% Note: N=10 Source: World Bank. Similarly, most respondents felt that the amount of time allowed by the program to complete their project was sufficient, with 80 percent reporting that they had sufficient time to success- fully complete their projects. Of those respondents who said they did not have enough time to complete their projects, inability to find qualified staff and delays in the supply of raw materials were cited as the most common reasons. Human and financial resources were commonly cited key success factors, but help finding addi- tional funding was a common gap in support. The availability of human resources was reported as a key factor for project success by more than three quarters of respondents, followed by the availability of financial resources (cited by more than half of respondents). Access to research infrastructure was reported as a key success factor by 43 percent of respondents, while program design and implementation was reported as a success factor by 27 percent (Figure 7.25). When asked about any additional support needed during implementation that would have improved the outcomes of their projects, assistance with finding additional funding sources (19 percent Bulgaria: Efficiency Analysis of STI Programs 152 of respondents), assistance with the preparation of project budgets (15 percent), better admin- istrative support from NIF (15 percent), and access to research infrastructure (15 percent) were cited as the most common types of additional support required (Figure 7.26). Figure 7.25 Key Success Factors for NIF Projects Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Availability of human resources such as researchers or mentors 78% Availability of financial resources 54% Availability of research infrastructure 43% The way support program is designed and implemented 27% Source: World Bank. Figure 7.26. Gaps in Support for NIF Projects Share of respondents 0% 5% 10% 15%20% Assistance with finding additional funding sources 19% Assistance in the preparation of project budgets 15% Better administrative support by NIIF 15% Access to research infrastructure and equipment 15% Assistance in preparation of monitoring reports 10% Establishing connections with international experts 6% Guidance for intellectual property rights management 4% Guidance for certification and standard norms 4% Guidance for market penetration and/ or COMMERCIALIZATION 4% Assistance to establish collaborations 4% Assistance with procurement 2% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 153 Overall Project Quality Project objectives—which focused on developing new products, services, and processes—were well aligned with the objectives of the NIF program. The program aims to promote research and development activity in enterprises by funding industrial research and experimental development. In line with this, 97 percent of respondents said one of the top objectives for their project was developing a new product, service, or process. Other project objectives reported by respondents include developing or starting a new enterprise (65 percent of respondents), collaborating with researchers or research institutions (61 percent), and developing the innovation capacities of the firm (58 percent) (Figure 7.27). Figure 7.27. NIF Project Objectives Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Develop a new product, service, or process 97% Develop or start a new enterprise, business, or SPIN-OFF 65% Collaborate with other researchers or research institutions 61% Develop innovation capacities and management skills 58% Pursue intellectual property (patents, industrial DESIGN RIGHT, COPYRIGHTS, ETC.) 13% Putting a new product, service, or process on the MARKET 6% Collaborate with other private sector entities 6% Training of employees 3% Source: World Bank. NIF projects had long expected ROI timelines. When asked when respondents expected to recov- er the investment made in the project, almost 70 percent expected to recover their costs three years or more after the completion of their projects (Figure 7.28). Over 20 percent of respondents did not expect to recover their costs until at least five years after project completion. NIF respon- dents generally expected longer return on investment timelines than DPPI respondents. This re- turn-on-investment timeline reflects of the nature of the NIF program, which supports industrial research and early-stage development project, rather than projects that are closer to the market. Bulgaria: Efficiency Analysis of STI Programs 154 Figure 7.28. Distribution of Expected ROI Timelines for NIF Projects 30% 27% 25% 18% 21% 18% 20% 15% 12% 10% 3% 5% 0% 1 YEAR 2 YEARS 3 YEARS 4 YEARS 5 YEARS 6 YEARS Source: World Bank. Most project results met expectations. Among respondents that had completed their projects, more than 60 percent felt that the outcomes of their projects matched their expectations (Figure 7.29). Projects exceeded expectations for 11 percent of respondents, while 28 percent said their project outcome was below expectations – the largest share of projects that fell below expecta- tions for any program included in this analysis. This again is likely due to the early-stage nature of the projects supported by the NIF program, which are inherently riskier than academic research projects or later-stage commercialization projects. Figure 7.29. Match between NIF Project Results and Beneficiary Expectations Share of respondents 0% 10% 20% 30% 40% 50% 60%70% Above expectations 11% Matched expectations 61% Below expectations 28% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 155 8. Development of Product and Process Innovations Bulgaria: Efficiency Analysis of STI Programs 156 8. Development of Product and Process Innovations Program Summary Development of Product and Process Innovations is a matching grant scheme under the OP Innovation and Competitiveness that provides financing to firms for supporting technology adoption, technology upgrading, and the development of new products, services, and processes. The program is de- signed and implemented by the Directorate General for OPIC. It has intended outcomes related to IP generation; new products, services, and processes; upgraded products, services, and processes; and increased revenues and employment for beneficiary companies. DPPI respondents reported the highest application costs of any program (although lower as a share of disbursed grants received than NIF respondents), likely due to the fact that over 90 percent of respondents used some form of external support to prepare their applications. DPPI respondents also made the largest average contributions (cash and in kind) to their projects of any of the programs included in this analysis. About 35 percent of respondents said that their project has resulted in a product or service that is currently being sold, while another 45 percent said they have a product or service that is that is ready to be sold. New products, services, and processes were the most common outcome reported by respondents; followed by upgraded products, services, or processes; new technologies adopted; and employees hired. Respondents’ satisfaction with program application and selection processes was mixed; few respondents were satisfied with the overall application process, fairness of eligibility criteria, feedback on project selection, and the time between application, selection, and disburse- ment of funds. Applications for the DPPI program are done entirely electronically through the online UMIS platform, and templates and guidelines are provided to applicants through the platform, which means that applying to these programs should be generally easier than for the other programs in this report. The fact that over 90 percent of applicants used exter- nal consultants to prepare their applications (more than any other program included in this analysis) indicates that the application process for the program remains challenging despite the adoption of the UMIS system. However, among respondents that had completed their projects, 80 percent felt that the outcomes of their project matched their expectations. Bulgaria: Efficiency Analysis of STI Programs 157 The program has thus far implemented a single call for proposals, funding a total of 154 projects in 2019 for a total of 70.8 million BGN in contracted funding (69.1 million BGN has been disbursed). The designated points of contact for all 154 projects funded by the program from 2019 to 2020 were surveyed for this analysis, with 80 beneficiaries responding to the survey for a response rate of 52 percent. Respondents represent 52 percent of the contracted funding for the program over that period, with an average grant size of 460,945 BGN. The main economic activity area of respondent firms were ICT (39 percent of respondents), professional services (38 percent), and manufacturing (23 percent) (Figure 8.1), while the main Smart Specialization priority areas of re- spondents’ projects were informatics and ICT (62 percent of respondents) and mechatronics (41 percent) (Figure 8.2). Figure 8.1. DPPI Respondents by Main Economic Activity Share of respondents 0% 10% 20% 30% 40%50% Information and Communication 39% Professional, Scientific and Technical Activities 38% Manufacturing 23% RETAIL 1% Source: World Bank. Figure 8.2. DPPI Respondents by Smart Specialization Priority Area Share of respondents 0% 20% 40% 60%80% Informatics and ICT 62% Mechatronics and clean technologies 41% Healthy life and biotechnology industries 14% New technologies in creative and 10% re-creative industries Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 158 8.1 Efficiency in Use of Inputs Administrative costs made up less than one percent of total DPPI program costs, which peaked in 2019. Grants distributed to beneficiaries made up 99 percent of the costs of the program from 2017 to 2020 (Figure 8.3). Total program costs were highest in 2019, coinciding with the highest levels of grants disbursed to beneficiaries, before dropping dramatically in 2020. Overall admin- istrative costs were highest in 2018, when the majority of projects were awarded (Figure 8.4). It should be noted, however, that cost data for office equipment and ICT were not available for the program, so administrative costs are likely higher than what is reported in this analysis. Figure 8.3. DPPI Program Costs by Category, 2017–20 1% 99% Administrative Costs Source: World Bank. Grants disbursed Figure 8.4. DPPI Program Costs by Year, 2017–20 40.000.000 35.000.000 BGN, 2020 constant 30.000.000 25.000.000 20.000.000 15.000.000 10.000.000 5.000.000 0 2017 2018 2019 2020 Source: World Bank. Administrative Costs Grants disbursed Bulgaria: Efficiency Analysis of STI Programs 159 Administrative Costs Personnel costs made up the largest share of administrative costs for the program. Personnel costs make up 40 percent of total administrative costs, followed by fixed costs and external ser- vices (Figure 8.5). The program’s single call for proposals was issued in 2018, and there was a cor- responding increase in personnel costs that year, which then decreased in the following years. Fixed costs remained relatively constant from 2017 to 2020, while external services costs were highest in 2017 (Figure 8.6). Figure 8.5. DPPI Administrative Costs by Category, 2017–20 25% 40% Personnel Fixed Costs Source: World Bank. 34% External Services Figure 8.6. DPPI Administrative Costs by Year, 2017–20 140.000 120.000 BGN, 2020 constant 100.000 80.000 60.000 40.000 20.000 0 2017 2018 2019 2020 Source: World Bank. Personnel Fixed Costs External Services Bulgaria: Efficiency Analysis of STI Programs 160 Almost 50 percent of personnel costs went toward the evaluation of project proposals, while monitoring and evaluation made up 25 percent of personnel costs and implementation costs were 17 percent (Figure 8.7). All costs for design and the evaluation of project proposals came in 2017 and 2018 (the bulk of DPPI project awards were made in 2018), while all implementation and M&E costs came in 2019 and 2020 (Figure 8.8). Figure 8.7. DPPI Personnel Costs by Category, 2017–20 6% 3% 17% 26% Design Implementation Evaluation of applications or proposals M&E Source: World Bank. 48% Other personnel costs Figure 8.8. DPPI Personnel Costs by Year, 2017–20 70.000,00 60.000,00 BGN, 2020 constant 50.000,00 40.000,00 30.000,00 20.000,00 10.000,00 0 2017 2018 2019 2020 Source: World Bank. Design Implementation Evaluation of applications M&E Other personnel costs or proposals Bulgaria: Efficiency Analysis of STI Programs 161 As mentioned previously, cost data for office equipment and ICT were not available for the pro- gram, so the fixed cost data presented here are likely lower than the actual fixed costs of the program. The fixed costs that were available (goods and services and office space) remained constant from 2017 to 2020 at 32,756 BGN per year. The DDPI program also had 70,393 BGN in external services costs for the contracting of external experts for the evaluation of proposals. This includes 38,913 BGN in 2017 and 31,480 BGN in 2018 (in constant 2020 BGN). There were no costs related to external services in 2020. The total administrative cost of implementing the program from 2017 to 2020 was 355,568 BGN, with an average cost per project of 2,308 BGN (Table 8.1). The average administrative cost per proj- ect for DPPI was significantly lower than for the NIF program, likely due to the fact that DPPI only issued a single call for proposals from 2018 to 2020, while NIF issued three over the same period. Table 8.1. DPPI Administrative Costs Cost category Total costs Cost per project Personnel 154,150 1,001 Fixed 131,024 851 External services 70,393 457 Source: World Bank. Note: All amounts in constant 2020 BGN. Bulgaria: Efficiency Analysis of STI Programs 162 Costs Covered by Beneficiaries The application process for DPPI is the most costly and time intensive among the programs in this analysis. Respondents reported an average application cost of 10,415 BGN and spent an av- erage of 32 working days preparing their application. More than half respondents spent less than 5,000 BGN in preparing their application, while 9 percent of respondents between 10,001 and 20,000 BGN and 15 percent spent more than 20,000 BGN (Figure 8.9). A majority of respondents reported spending 30 working days or fewer in preparing their application, although 11 percent of respondents reported spending more than 60 days in preparing the application (Figure 8.10). Figure 8.9. DPPI Application Costs Application costs (Share of respondents) 0% 20% 40%60% 5,000 BGN or less 51% 5,001 – 10,000 BGN 25% 10,001 – 20,000 9% More than 20,000 15% Source: World Bank. Figure 8.10. Time to Prepare DPPI Applications Time to prepare application (Share of respondents) 0% 20% 40%60% 10 days or fewer 11% 11 – 30 days 51% 31 – 60 days 26% More than 60 days 11% Source: World Bank. DPPI respondents used a range of external support to prepare their applications. About 92 per- cent of respondents reported that they used some type of external support (such as legal, ac- counting, or other services) to prepare their application, which likely contributed to the relatively high costs of applying to the DPPI program compared to other programs included in this analysis. The most common types of support used were experts or consultants (needed by 34 percent of respondents), administrative assistants (31 percent), and accountants (22 percent). Eight percent of respondents reported that they did not use any external support in preparing their application (Figure 8.11). Bulgaria: Efficiency Analysis of STI Programs 163 Figure 8.11. External Support Used to Prepare DPPI Applications Share of respondents 0% 5% 10% 15% 20% 25% 30% 35%40% Experts / consultants 34% Administrative assistant 31% Accountant 22% Lawyer or legal consultant 5% The project team prepared the 8% application without any additional help Source: World Bank. Respondents reported an average of 190,000 BGN cash and 55,000 BGN in kind contributions to the implementation of their projects (Figure 8.12). The DPPI program will fund between 50 and 70 percent of total project costs, depending on the size of the firm, while the firm is responsible for funding the remainder of project costs. Cash contributions were the most common contribution by share of respondents, with 55 percent of respondents making in kind contributions. About three quarters of respondents made both cash and in-kind contributions to the project, while 20 percent made only cash contributions (Figure 8.13). Figure 8.12. Beneficiary Contributions to DPPI Projects by Type and Amount 0 100.000200.000 Average cash Contributions Average in kind Contributions Source: World Bank. Figure 8.13. Beneficiary Contributions to DPPI Projects by Type and Frequency 4% 20% 76% None Cash In kind Source: World Bank. Both Bulgaria: Efficiency Analysis of STI Programs 164 On average, the costs covered by the program were slightly less than double the costs covered by respondents. The average cost covered by the program, including disbursed grants and adminis- trative costs, was 460,887 BGN, while the average cost covered by beneficiaries was 252,088 BGN. Therefore, for every BGN that beneficiaries put into the project, the program invested 1.83 BGN. 8.2 Efficiency in Generation of Outputs For every BGN the program invested in administrative costs, beneficiaries received an average of 199 BGN. The total program costs from 2017 to 2020 was 71 million BGN, or 460,887 BGN per project. This includes 69.1 million in disbursed grants to beneficiaries and 1.2 million BGN in ad- ministrative costs (Table 8.2). The program did not include any nonfinancial or indirect financial support to beneficiaries. Table 8.2. DPPI Program Costs, 2018–20 Total Average per project Grant funding disbursed 69,088,015 BGN 448,623 BGN Administrative and operating costs 355,568 BGN 2,308 BGN Total program cost 70,976,672 BGN 460,887 BGN Source: World Bank. Note: All amounts in constant 2020 BGN. Salaries made up the largest category of project expenditure; followed by machinery and equip- ment and IT systems. Consulting services, materials and supplies, and access to specialized equipment also represented major costs (Figure 8.14). Bulgaria: Efficiency Analysis of STI Programs 165 Figure 8.14. DPPI Project Expenditures by Cost Category Share of expenditures 0% 5% 10% 15% 20% 25% 30% 35% 40% 45%50% salaries 45% Machinery, equipment, instrumentation 17% IT systems, specialized software, IT licenses, websites 11% Consulting services (e.g. feasibility studies, survey companies) 8% Materials, supplies, inventory 8% Access to specialized equipment 5% or laboratories Intellectual property 2% (patents, trademarks, copyrights) Testing and certifications 2% SPACE, RENT 2% Other 1% Marketing campaigns or PR 0% activities for project visibility Training 0% Travel (fairs, exhibitions, 0% conferences, etc.) Source: World Bank. Outputs Achieved Two outputs were tracked for the DPPI program, based on the program’s ToC: 1.) collaborations during project implementation and 2.) improved capabilities of employees. Program respondents generated a total of 289 outputs from 2017 to 2020, or an average of 3.6 outputs per project. Over that period, the program invested an average of 460,887 BGN in bene- ficiary projects, so respondents generated 0.8 outputs per 100,000 BGN invested. Improved capabilities of employees were the most common type of output reported (an average of 2.1 per project), followed by collaborations during implementation (1.5 per project) (Table 8.3). Bulgaria: Efficiency Analysis of STI Programs 166 Table 8.3. Outputs Reported by DPPI Respondents Total number Average Outputs per Share of of outputs outputs per 100,000 BGN of respondents Output achieved respondent program cost reporting outcome Collaboration 121 1.5 0.3 76% Improved capabilities of employees 168 2.1 0.5 48% Source: World Bank. Over 76 percent of respondents engaged in some types of collaboration during their projects. The most common collaborations, by both average number of collaborations per project and share of respondents who engaged in such a collaboration, were partnerships with domestic research partners, followed by partnerships with foreign research partners (Figure 8.15). Collaborations with industry were less common that collaborations with research entities, with collaborations with industry partners from the Bulgarian diaspora being the most common type of industry partnerships. Figure 8.15. DPPI Project Collaborations by Type of Partner 0,9 80% Average number of collaborations 0,8 70% 0,7 60% Share of respondents 0,6 50% 0,5 40% 0,4 30% 0,3 0,2 20% 0,1 10% 0,0 0% With domestic With foreign With diaspora With domestic With foreign With diaspora research research research industry industry industry partners partners partners partners partners partners Source: World Bank. Number of collaborations Share of respondents Roughly half (48 percent) of respondents reported improving the capabilities of their employees during project implementation, increasing to 60 percent of respondents improving the capabili- ties of their employees after project completion. Bulgaria: Efficiency Analysis of STI Programs 167 8.3 Efficiency in Achievement of Outcomes Seven outcomes were tracked for the DPPI program, based on the program’s ToC: 1.) Patent ap- plications; 2.) patents granted; 3.) upgraded products, services, and processes; 4.) new products, processes, and services; 5.) new technologies adopted; 6.) increased company sales; and 7.) new employees hired. DPPI respondents generated more outcomes on a per project basis than NIF respondents, but due to the higher costs of the DPPI program, NIF respondents were generally more efficient in producing outcomes on a per cost basis. Respondents reported a total of 1,243 outcomes (in- tended outcomes and other), for an average of 15 outcomes per project. From 2018 to 2020, the program invested an average of 460,887 BGN in beneficiary projects, so respondents generated 3.3 outcomes per 100,000 BGN invested over that time period. New products, services, and processes were the most common outcome reported, with an av- erage of 2.6 per project; followed by upgraded products, services, or processes (2.2 per project); new technologies adopted (1.9 per project); and employees hired (1.8 per project) (Table 8.4). Table 8.4. Intended Outcomes Reported by DPPI Respondents Total number Average Outcomes per Share of of outcomes outcomes per 100,000 BGN of respondents Intended outcome achieved respondent program cost reporting outcome Patent applications 70 0.9 0.2 88% Patents granted 57 0.7 0.2 71% Upgraded products, processes, or services 178 2.2 0.5 45% New products, processes, or services 211 2.6 0.6 85% Company adopted a new technology 152 1.9 0.4 78% Company increased sales 34 6.7% increase 0.1 43% Company hired additional employees 144 1.8 0.4 46% Source: World Bank. Nearly half of respondents had a product, services, or process that is ready to be sold. About 35 percent of respondents said that their project has resulted in a product or service that is current- ly being sold, while another 45 percent said they have a product or service that is that is ready to be sold. A sizeable share of respondents also have project results that are still in earlier stages of commercialization, with 35 percent saying they have a working prototype and 28 percent re- porting they have a proof of concept. Only three percent of respondents said their project results Bulgaria: Efficiency Analysis of STI Programs 168 were not commercialized nor were likely to be commercialized (Figure 8.16). The higher share of DPPI projects that result in market ready products and services when compared to NIF is likely due to the fact that NIF only funds project up to the experimental stage of development, while DPPI will fund projects in later stages of commercialization. One quarter of respondents had introduced a product or service to the domestic market. Almost half of respondents have engaged in discussions with vendors about selling their product or ser- vice, while 24 percent had presented their product or service in the domestic market. Only nine percent of respondents had not yet taken steps toward selling their product or service (Figure 8.17. Steps Taken to Sell DPPI Project Results). Figure 8.16. DPPI Project Results by Stage of Commercialization Share of respondents 0% 10% 20% 30% 40%50% Have a product, service or process that is being sold 35% Have a product, service or process that is ready to be sold 45% Have a working prototype that is almost ready to be sold 35% Have a proof of concept for a product or process that 28% can be sold in the FUTURE The results that came out of this project will NOT be 3% COMMERCIALIZED OR ARE NOT LIKELY TO BE COMMERCIALIZED Source: World Bank. Figure 8.17. Steps Taken to Sell DPPI Project Results Share of respondents 0% 10% 20% 30% 40%50% Discussions/negotiations with a vendor or firm that will sell my product, service or process 47% Presented my product or process in domestic market 24% Participated in trade fairs to showcase my product, service, or process 13% Have not taken any steps 9% Participated as advertiser in scientific/technical 7% conferences Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 169 Other Results DPPI respondents reported several results not tracked by the program’s ToC, including prototypes, collaborative projects after implementation, and expansion to new markets. Prototypes were the most common other result reported (2.8 per project), followed by collaborative projects after implementation (2.6 per project), and expansion to new markets (1.5 per project). Respondents also reported an average increase in productivity of 9.6 percent and a reduction in production costs of 3.6 percent (Table 8.5). Table 8.5. Other Results Achieved by DPPI Respondents Total number Average Outcomes per Share of of outcomes outcomes per 100,000 BGN of respondents Other outcome achieved respondent cost reporting outcome Other IP activitiesa 29 0.5 0.1 45% Prototype 221 2.8 0.6 71% New business model 50 0.6 0.1 26% Expansion to new markets 116 1.5 0.3 59% Improved their export performance (%) 32 7.0% 0.1 40% Improved Productivity 45 9.6% increase 0.1 56% Reduced production costs 27 3.9% reduction 0.1 34% New enterprise or business spinoff 4 0.1 0.0 5% Reorganized the firm or part of the firm 16 0.2 0.0 19% Develop a new innovation unit in the firm 39 0.5 0.1 69% Collaborative projects after implementation 208 2.6 0.6 55% Source: World Bank. Note: a. Other IP activities includes industrial designs, transfer agreements, and so on. 8.4 Perceived Quality Respondents’ satisfaction with program application and selection processes was mixed. Less than 60 of respondents reported that they were satisfied with the overall application process, fairness of eligibility criteria, feedback on project selection, and the time between application, selection, and disbursement of funds. On the other hand, respondents were largely satisfied with program objectives, regulations, availability of program information, and other program elements (Figure 8.18). Applications for the DPPI program are done entirely electronically through the online UMIS platform, and templates and guidelines are provided to applicants through the platform, which means that applying to these programs should be generally easier than for nationally funded Bulgaria: Efficiency Analysis of STI Programs 170 programs. The fact that over 90 percent of applicants used external consultants to prepare their applications (more than any other program included in this analysis) indicates that the applica- tion process for the program remains challenging despite the adoption of the UMIS system. Figure 8.18. DPPI Respondents’ Satisfaction with Application Processes Satisfaction with application processes (Share of respondents) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Clarity of selection process Clarity of eligibility criteria Ease in accessing the program regulations Well explained application procedure Sufficiency of application period Adequate time for contract negotiation Clarity of program objectives Ease of obtaining information about the program Clarity of program regulations Appropriate list of eligible costs Flexibility of rules for non-compliance Timely and relevant application support Transparency of selection process Appropriate information requirements in proposals Ease of contract negotiation Ease of application procedure Fairness of eligibility criteria Adequate application to final results of selection time Availability of feedback on project selection approval Adequate selection to funding disbursement time Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 171 When asked if they think any program selection or eligibility criteria should be added, changed, or removed; four percent of respondents said they thought that program eligibility criteria should be modified. Suggested changes included allowing more types of research partnerships and us- ing a different definition of innovation that what is currently used by the program. Similarly, five percent of respondents said they thought that program eligibility criteria should be modified, with suggested changes including removing the requirement to provide evidence for the type of economic activity of the project and removing the criteria that provides extra points based on a firm’s location. Respondents were largely satisfied with the DPPI program’s implementation and monitoring processes, with the exception of financial reporting requirements and data protection practices. In the latter two areas, less than 65 percent of respondents reported they were satisfied (Figure 8.19). Financial reporting requirements, in particular, appear to be a source of dissatisfaction for respondents, and this may be an area to look for opportunities to lessen the administrative bur- den on firms. Figure 8.19. DPPI Respondents’ Satisfaction with Program Implementation and Monitoring Satisfaction with implementation processes (Share of respondents) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Appropriate expert feedback from monitoring (visits, reports, discussions) Sufficiency of administrative support provided Accessibility of administrative support Appropriate monitoring requirements (e.g. narrative, indicators, etc.) Timeliness of financial support Satisfactory data protection practices Appropriate financial reporting requirements Source: World Bank. Note: Figure shows percentage of respondents who agreed or strongly agreed with the statements asking about their satisfaction with program implementation and monitoring processes. Most respondents felt that the financial resources provided by the program were sufficient. Eighty percent reported that they had sufficient funding to successfully complete their project. Respondents who said that the financial support provided was not sufficient said that financial support would need to increase by an average of 46 percent in order to successfully complete their project. Inability to ensure co-financing and the occurrence of unexpected costs were cited Bulgaria: Efficiency Analysis of STI Programs 172 by the most common reasons why the amount of financial support provided by the program was not sufficient (Figure 8.20). Similarly, most respondents felt that the amount of time allowed by the program to complete their project was sufficient. Ninety percent reported that they had sufficient time to success- fully complete their projects. Of those respondents who said they did not have enough time to complete their projects, inability to reduce workload from other projects and inability to reduce institutional obligations were cited as the most common reasons (Figure 8.21). Figure 8.20. Reasons Why DPPI Respondents Found That Financial Support was Insufficient Share of respondents 0% 20% 40%60% Unable to ensure cofinancing 53% Unexpected costs occured 53% Unable to get financial support from 41% my institution Inadequate budget planning 41% Increased costs of inputs 41% Project’s scope increased beyond 12% the original plan Budget approved by DPPI was inadequate 12% Note: N=17 Source: World Bank. Figure 8.21. Reasons Why DPPI Respondents Found That the Time Allotted for Implementation was Insufficient Share of respondents 0% 20% 40% 60%80% Could not find qualified STAFF 60% Delays in supply of raw MATERIALS 60% Delays in production process 60% Did not have enough financial resources 40% Note: N=5 Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 173 Human and financial resources were commonly cited success factors. The availability of financial resources and human resources were cited as the main factors for project success by 75 percent and 71 percent of respondents respectively. The availability of research infrastructure and pro- gram design were also cited as important success factors by more than 40 percent of respon- dents (Figure 8.22). When asked about any additional support needed during implementation that would have improved the outcomes of their projects, assistance with finding additional funding sources (by 11 percent of respondents), guidance for IP management (10 percent), and access to research infrastructure (9 percent) were cited as the most common types of additional support required (Figure 8.23). One quarter of respondents felt they did not need any additional support not already provided by the program. Figure 8.22. Key Success Factors for DPPI Projects Share of respondents 0% 20% 40% 60%80% Availability of financial resources 75% Availability of human resources such as 71% researchers or mentors Availability of research infrastructure 47% The way support program is designed and implemented 45% Figure 8.23. Gaps in Support for DPPI Projects Share of respondents 0% 10% 20% 30% 40%50% Assistance with finding additional funding sources 11% Guidance for intellectual property rights management 10% Access to research infrastructure and equipment 9% Guidance for market penetration and/ or COMMERCIALIZATION 8% Better administrative support by OPIC 7% Establishing connections with international experts 7% Guidance for certification and standard norms 5% Assistance in preparation of monitoring reports 5% Assistance in the preparation of project budgets 3% Assistance with procurement 3% Assistance to establish collaborations 3% No additional support was needed 25% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 174 Overall Project Quality Respondents’ objectives for their projects were largely in line with the objectives of the DPPI program. The program aims to support firms in the production and marketing of innovative prod- ucts, processes and services, and 94 percent of respondents said one of the top objectives for their project and 27 percent said upgrading a product, service, or process was a top goal. Other top objectives of respondent projects included developing innovation capacities (69 percent of respondents) and engaging in research collaborations (45 percent), which are activities associ- ated with technology development and upgrading (Figure 8.24). Figure 8.24. DPPI Project Objectives Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Develop a new product, service, or process 94% Develop innovation capacities and management skills 69% Collaborate with other researchers or research institutions 45% Develop or start a new enterprise, business, or SPIN-OFF 31% UPGRADE A product, service, or process 27% Pursue intellectual property (patents, 19% industrial DESIGN RIGHT, COPYRIGHTS, ETC.) Collaborate with other private 11% sector entities Training of employees 4% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 175 DPPI respondents expected shorter ROI timelines than NIF respondents. When asked when re- spondents expected to recover the investment made in the project, 46 percent of respondents said 3 years or fewer and 56 percent said 4 years or more after the completion of their projects (Figure 8.25). On average, expected return on investment timelines were shorter for DPPI respon- dents than for NIF respondents, which is likely due to the fact that DPPI funds projects that are closer to the market than the NIF program. Figure 8.25. Distribution of Expected ROI Timelines for DPPI Projects 35% 29% 29% 30% 25% 13% 20% 15% 12% 12% 10% 6% 5% 0% 1 YEAR 2 YEARS 3 YEARS 4 YEARS 5 YEARS 6 YEARS Source: World Bank. A large majority of respondents felt their project results matched expectations. Among respon- dents that had completed their projects, 80 percent felt that the outcomes of their project matched their expectations. Six percent felt their project exceeded expectations, while 14 percent said their project outcome was below expectations (Figure 8.26). Figure 8.26. Match between DPPI Project Results and Beneficiary Expectations Share of respondents 0% 10% 20% 30% 40% 50% 60% 70% 80%90% Above expectations 6% Matched expectations 80% Below expectations 14% Source: World Bank. Bulgaria: Efficiency Analysis of STI Programs 176 9. Recommendations Bulgaria: Efficiency Analysis of STI Programs 177 9. Recommendations This section describes recommendations in two key areas: 1. Improving the efficiency of STI support in Bulgaria 2. Strengthening M&E practices for new and existing STI programs These recommendations are based on the finding from this efficiency analysis and also build and ex- pand upon the findings and recommendations from the previous components of this PER STI project. 9.1 Recommendations for Improving the Efficiency and Effectiveness of STI Support STI implementors should consider the costs and benefits of the management and cost structures used for program implementation Priority timeline: Mid-term Findings: The programs included in this analysis have differing arrangements and responsibilities in program implementation, which has resulted in radically different administrative costs, in terms of scale and composition. The Funda- mental Research and Vihren programs (implemented by NSF), and to a lesser extent NIF (implemented by SMEPA), rely heavily on external experts in program implementation to make up for the lack of full-time staff at the implementing institutions. ICT and eHealth (both sectoral programs of the National Science Programs 2018–22 portfolio) are partially implemented by the beneficiary con- sortiums formed for these programs, an arrangement that comes with mini- mal administrative costs on the part of the Ministry but likely does not result in overall lower costs in implementing the programs due to the costs borne by the beneficiary consortiums. Approach: Increasing full-time program staff and making more of the responsi- bilities related to program implementation internal would increase the capacity of implementing institutions (identified as a major challenge in previous compo- nents of this PER STI project) and could lead to more effective program delivery. Key Implementing Stakeholder(s): MoES (Fundamental Research, Vihren, ICT, and eHealth programs), MoE (NIF and DPPI programs) Bulgaria: Efficiency Analysis of STI Programs 178 Analyze the quality of program outcomes, were possible, to understand which programs and projects are most impactful Priority timeline: Short-term Findings: Publications reported by research respondents generally had lower numbers of year-normalized citations than the average Bulgarian publication indexed in the Web of Science from 2016 to 2020, indicating that the research being funded is generating low impacts on the scientific community. All research programs included in this report track the number of publications generated by beneficiaries but do not track citations or other impact measures of the in- dividual publications produced by beneficiaries. Approach: Additional bibliometric analyses can reveal which of these publica- tions are most impactful and influential on the scientific community. This data can be used not just to understand the impact of individual projects but also to identify which programs and which scientific fields produce the most impactful research with public support. Key Implementing Stakeholder(s): NSF (Fundamental Research, Vihren), MoES (ICT and eHealth) Provide incentives for collaboration in programs with the objective of improving collaboration and the connections of beneficiaries Priority timeline: Short-term Findings: Fundamental Research and Vihren respondents generated relative- ly few collaborations during project implementation, despite the fact that in- creasing collaboration activity among beneficiaries is a goal of these programs. Approach: Increasing collaboration activities during implementation could be mple, mak- done by building collaborations into the design of programs (for exa­ ing collaborative agreements preconditions for grant awards), accounting for the transaction costs of collaboration in program funding, providing additional Bulgaria: Efficiency Analysis of STI Programs 179 points on applications that include research partners, and broadening eligibility criteria to allow beneficiaries to collaborate with a broader range of organiza- tions (such as non-profit organizations) during project implementation. Key Implementing Stakeholder(s): NSF Explore developing follow-up grant schemes for beneficiaries of NSF programs Priority timeline: Mid-term Findings: A small share of Fundamental Research and Vihren respondents re- ported outcomes related to prototyping, new software development, and new technology development. The Country Needs and Policy Mix Assessment showed a gap in applied research funding programs to public research institutions: at present, NSF regulations do not allow for funding to be used for commercial- ization, IP development, or technology transfer activities, and there are few applied research programs that public researchers can access outside of NSF that could support such follow-on projects. Approach: These findings suggest that some NSF project results could be fur- ther developed and potentially commercialized if there were applied research grant schemes that their projects could “graduate” to for additional funding and support. The new State Agency for Research & Innovation (SARI) would be a logical implementor to develop such grant schemes. Key Implementing Stakeholder(s): SARI Bulgaria: Efficiency Analysis of STI Programs 180 Reduce application costs for firm support programs Priority timeline: Short-term Findings: Application costs were higher for firm respondents, likely due to the fact that roughly 90 percent of firms used some form of external support (such as consultants, legal services, and accountants) to prepare their applications. The Functional and Governance Analysis found that application processes for firms are indeed burdensome, particularly the supporting documentation firms must provide to support their applications. The Functional and Governance Analysis also found that reporting processes were much easier for beneficia- ries of programs funded under EU operational programmes than for nationally funded programs. Approach: Programs outside of the EU operational programmes (OPs) would greatly benefit from an online portal similar to that used by Bulgarian OPs where beneficiaries can submit and receive information throughout the application and contracting process. Additionally, implementing bodies should acquire documentation (tax documents, criminal records, etc.) ex officio, where possi- ble, and submission requirements of any documents that are not essential to the application process should be deferred until the contracting phase of the project. These changes would likely lower the need for firms to hire external consultants and services to prepare their applications and reduce the overall application costs to firms. Key Implementing Stakeholder(s): SMEPA (NIF), MoE (DPPI) Bulgaria: Efficiency Analysis of STI Programs 181 Reduce the administrative burdens on beneficiaries during implementation Priority timeline: Mid-term Findings: Financial monitoring requirements were an issue for respondents across research and firm support programs, with fewer than 60 percent of re- spondents from Fundamental Research, NIF, and DPPI saying they were sat- isfied with financial reporting requirements. Financial reporting rules require beneficiaries to provide certified and translated copies of all eligible expenses, including things like plane tickets. The Functional and Governance Analysis found that reporting processes were much easier for beneficiaries of programs funded under EU operational programmes than for nationally funded programs because beneficiaries of OP-funded programs report using an online portal with preloaded templates for reports. Approach: The creation of an online portal and technical and financial reporting templates would reduce the burden of reporting on beneficiaries of programs outside of the OPs. Financial reporting requirements could be simplified by approving R&D project expenditure plans in advance, reversing the current ap- proach in which each expense item must be reported when incurred. For pro- grams supporting R&D activities, R&D activities can be presumed to be eligible when reported and then verified later. In such a system, applicants self-report whether they are eligible, and implementors conduct audits later to verify eli- gible costs. Key Implementing Stakeholder(s): NSF (Fundamental Research and Vihren), SMEPA (NIF), MoE (DPPI) Bulgaria: Efficiency Analysis of STI Programs 182 9.2 Recommendations for Strengthening M&E Practices for New and Existing STI Programs Track program-level cost data Priority timeline: Short-term Findings: Currently, no STI implementors systematically track all costs related to the implementation of the programs (such as personnel costs, fixed costs, and the costs of external services), making it extremely difficult to understand the true costs of program implementation. The programs implemented by NSF (Fundamental Research and Vihren) could only provide an estimation of admin- istrative costs based on both programs’ share of the NSF budget. For the ICT, eHealth, and DPPI programs, the implementing institutions could not provide detailed data on fixed costs. Approach: Implementors can adopt the administrative cost data templates used for this analysis to begin more systemically collecting and analyzing this information. Key Implementing Stakeholder(s): All STI implementors – MoES, NSF, SARI, MoE, DG OPIC, SMEPA Conduct regular surveys of beneficiaries of STI programs Priority timeline: Short-term Findings: Beneficiaries from STI programs generally submit mid-term and end- of-project monitoring reports and may receive site visits from program moni- toring staff, but no STI programs regularly survey their beneficiaries. Approach: Beneficiary surveys will allow implementors to collect information on outputs and outcomes that is comparable across projects and programs to better understand the performance of their policy portfolios (rather than Bulgaria: Efficiency Analysis of STI Programs 183 just the performance of individual projects), as well as provide information on beneficiaries’ perceptions of their programs. The survey questionnaires in the appendices of this report can be adapted for beneficiaries of a range of STI sup- port programs. Key Implementing Stakeholder(s): MoES, SARI, MoE Conduct regular efficiency analyses of STI programs Priority timeline: Short-term Findings: No analyses of the efficiency of STI programs were performed in the most recent programming period. Approach: Through systematic tracking of program cost data and regular ben- eficiary surveys, STI implementors will be able to regularly assess programs’ ef- ficiency in the use of inputs and generation of outputs and outcomes. Key Implementing Stakeholder(s): MoES, SARI, MoE With more complete data on costs and results, policymakers should use findings to improve the efficiency of STI programs in generating desired results with public funds Priority timeline: Short-term Findings: Due to the lack of data on the costs of program implementation and comparable data on project results across programs, STI implementors are generally unable to make decisions based on the efficiency of programs in pro- ducing desired results. Approach: This report, along with future efficiency analyses, can be used to define benchmarks and targets for STI programs, understand which programs Bulgaria: Efficiency Analysis of STI Programs 184 are achieving their objectives, and make informed decisions to improve the ef- ficiency of public funding for STI. Key Implementing Stakeholder(s): MoES, SARI, MoE Conduct impact evaluations of strategically important programs Priority timeline: Short-term Findings: Few evaluations of any kind were done of STI programs in the most recent programming period. Approach: Efficiency analyses should be complemented by impact analyses, at least for strategically important programs. Guidelines for when to evaluate in- dividual programs can be determined by a national-level evaluation framework, such as that used in the Republic of Korea (detailed in World Bank 2021a), or by frameworks at the institutional or programmatic level. World Bank 2009 and Me- non et al. (2009) can also serve as useful references for developing M&E frame- works. Impact analyses supplement data on program efficiency by establishing which changes or impacts can be attributed to a given intervention, allowing for a more complete picture of program impact and effectiveness. Instruments that are strategically important (for example, those with large budgets, supporting a large number of beneficiaries, with large expected impacts, and so on) should undergo at least one performance evaluation and one external impact evalua- tion per programming period. Ideally, these evaluations should be conducted by a third parties (that is, by evaluators outside of the implementing body) that do not have conflicts of interest with the instrument being evaluated. Key Implementing Stakeholder(s): MoES, SARI, MoE Bulgaria: Efficiency Analysis of STI Programs 185 References Bulgaria: Efficiency Analysis of STI Programs 186 References Correa, Paulo. 2014. Public Expenditure Reviews in Science, Technology, and Innovation: A Guid- ance Note. World Bank Group, Washington, DC. World Bank. European Commission. 2015. “Peer Review of the Bulgarian Research and Innovation system.” Horizon 2020 Policy Support Facility. DG RTD, H2020 Policy Support Facility. Harzing, Anne-Wil. 2010. “Citation Analysis across Disciplines: The Impact of Different Data Sourc- es and Citation Metrics.” Scientometrics 106.2: 787-804. Menon, Saraswathi, Judith Karl, and Kanni Wignaraja. “Handbook on planning, monitoring and evaluating for development results.” UNDP Evaluation Office, New York, NY (2009). World Bank. 2009. Making Monitoring and Evaluation Systems Work: A Capacity Development Toolkit. Washington, DC: World Bank. World Bank. 2013. “Bulgaria Reimbursable Advisory Services Program on Innovation: Input for Bul- garia’s Research and Innovation Strategies for Smart Specialization.” Washington, DC: World Bank. World Bank. 2018. Análisis de Eficiencia del Gasto Público en Ciencia, Tecnología, e Innovación en Colombia. Washington, DC: World Bank. World Bank. 2020. “Bulgaria: Country Needs and STI Policy Mix Assessment.” Washington, DC: World Bank. World Bank. 2021a. “Bulgaria Functional and Governance Analysis.” Washington, DC: World Bank. World Bank. 2021b. “Croatia Public Expenditure Review in Science, Technology and Innovation: Analysis of Outputs and Outcomes.” Washington, DC: World Bank. Bulgaria: Efficiency Analysis of STI Programs 187 Appendices Bulgaria: Efficiency Analysis of STI Programs 188 Appendix A Theories of Change for Analyzed Programs A theory of change (ToC) is a detailed description of the mechanisms through which a change is expected to occur in an intervention to achieve its objectives. As such, a ToC explicitly depicts the “pathways of change,” connecting inputs (for example, funding, human resources, and time) to conduct various activities (or interventions), which produce outputs, and together these are expected to lead to short-term and longer-term outcomes. The ToC establishes the preconditions, requirements, and assumptions needed for the inputs and activities to be logically linked to the goals. The ToC is not only a useful program design and planning tool, but it is also the essential blueprint for building a rigorous framework for results measurement, particularly by helping de- velop indicators for identified ToC elements. The development of a ToC begins with identification of the “focal problem” – that is, the key chal- lenge(s) the program is trying to solve – and then identifying the “root causes” of this challenge. The root causes are important because this is the level at which activities can be designed, while the focal problem is merely a symptom of the root causes. Root causes may also have one or more factors that contribute to them. For each program included in this analysis, a ToC diagram was developed by first mapping the in- formation available in various program documents, which was supplemented by interviews with program staff conducted during component 2 of this project, the Functional and Governance Analysis. This mapping exercise enables a systematic assessment of the key causal pathways envisaged by the program in achieving the stated objective(s) with the activities the program finances. Once these pathways are identified, they are then used to create a ToC diagram to de- pict the linkages between inputs, activities, outputs, and outcomes, add missing elements and assumptions, and harmonizing the indicators across similar programs. After a ToC is developed, a results framework was then created. A results framework aims to identify and define the indi- cators underlying the theory of change for the program. As such, the results framework identifies indicators for all the ToC elements so that program “success”, in terms of whether the intended change(s) occurred, can be measured and verified. Bulgaria: Efficiency Analysis of STI Programs 189 A.1 Fundamental Research Program Objective(s) Increasing the quantity and quality of basic research related to issues of regional and national importance; improving the quantity and quality of Bulgaria’s internationally visible scientific prod- ucts; expanding the Bulgarian scientific community’s participation in the European Research Area and expanding international scientific cooperation; and significant intensification of the connec- tions of science with education, business, state bodies, and society as a whole. Activities Funding of basic research projects in one of ten scientific areas Theory of Change As illustrated in the ToC diagram (Figure A.1), the program is designed with the program-level goal of increased the capacity and reputation of the Bulgarian public research sector to perform high quality basic research. It is important to note that grant funding from NSF cannot be used for any direct commercial application or use, so the ToC does not include any activities or outputs relat- ed to IP generation, technology transfer, or commercialization. To achieve the program goal, the ToC relies on the following pathways of change: • Increasing the capacity of beneficiary organizations to produce impactful publications in basic research fields: the program seeks to support beneficiaries in completing fundamental research projects in key scientific areas. As beneficiaries and their research staff complete basic research projects in key areas, they will gain competencies in these areas and be able to produce higher quality basic research outputs. • Increased connections to the international scientific community and the private sector in ba- sic research: the program supports research collaborations with the international scientific community and the private sector in basic research. As beneficiaries perform collaborative research projects, they should form lasting connections to their partnering organizations Bulgaria: Efficiency Analysis of STI Programs 190 Figure A.1. Fundamental Research Theory of Change Diagram strategic increased capacity of the bulgarian public research sector to produce research goal outputs that are impactful on the economy and society increased capacity and reputation of public research sector program to perform high quality basic research goal • number of projects receiving additional funding after project completion • number of competitive projects contracted after project completion increased capacity of beneficiary increased connections to the organizations to produce impactful international scientific community and publications in basic research fields the private sector in basic research outcome • number of scientific publications • number of collaborative projects published in scientific journals and contracted after project completion indexed in “Web of Science” • value of collaborative projects contracted after project completions completed fundamental research increased collaboration with the projects in key scientific areas international scientific community and the private sector in basic research output • number of full-time aquivalent (FTE) researchers involved in project implementation • number of collaborative research projects • number of training activities attended supported • number of completed research projects activity • funding of basic research projects • funding of research collaborations • dissemination of research results input • grants for basic research projects Bulgaria: Efficiency Analysis of STI Programs 191 Results Framework A results framework aims to identify and define the indicators underlying the theory of change for the program. As such, the results framework identifies indicators for all the ToC elements so that program “success”, in terms of whether the intended change(s) occurred, can be measured and verified. In this section, the proposed results framework is built from, and indexed to, the ToC elements, and revised (where necessary) according to the CART principles10. As the theory of change and results framework is developer further, will need to be defined for each indicator, as well as the processes of collecting and verifying the data (including indicator verification pe- riod, process, and frequency). Level Indicator Definition Objective Number Definition: Additional funding (grants) for continuation of research activities received of project by project beneficiaries, as the main applicant (coordinator, manager), or as a partner proposals (associate). Funding sources can be national, EU (Horizon 2020, ERC, FP8, ESI Funds), or receiving other (public or private). additional funding Disaggregation: By success (approved, rejected); By funding sources (national, Horizon after project 2020, ERC, FP8, ESI Funds, other); by project role (lead beneficiary, partner) completion Objective Number of Definition: Competitive research projects awarded to project beneficiaries as the main competitive applicant (coordinator, manager), or as a partner (associate). Funding sources can be projects national, EU (Horizon 2020, ERC, FP8, ESI Funds), or other (public or private). contracted after project Disaggregation: Funding source (national, Horizon 2020, ERC, FP8, ESI Funds, other); completion project role (lead beneficiary, partner) Outcome Number of Definition: Scientific papers and reports published, describing original results and scientific research conducted within this project with the aim of informing the scientific publications community and society as a whole. Scientific papers need to be indexed in Web of published Science platform (core collection), and may include articles, reviews, proceedings papers, in scientific letters, and book chapters. journals and indexed in ‘Web Disaggregation: Rank of journal where publication was published (first and second- of Science’ quartile journals, other); by scientific areas; by scientific fields; by S3 thematic priority areas • Out of which: Number of publications with impact factor11 • Out of which: Number of joint publications with international coauthors ((Publications with at least one author from a Bulgarian organization and one author from an international organization [public or private]) 10 See more on the CART principles here: https://www.poverty-action.org/right-fit-evidence/principles 11 Impact factor is a Web of Science metric that measures the ratio between citations and recent citable items published in a given journal or publication. Bulgaria: Efficiency Analysis of STI Programs 192 Output Number of Definition: Number of researchers that directly carry out research and development full-time activities related to the implementation of the project. Workforce may be existing or new, equivalent (FTE) employed at the beneficiary and partners, or contracted from third parties. Auxiliary staff researchers for R&D activities (jobs that are not directly involved in R&D activities) are not included in involved this indicator and should not be counted. The measurement unit is “Full-time equivalent”. in project Engagement of researchers employed on the research activities supported by the project implementation less than full-time should be converted to the number of FTE employees by dividing the researchers’ scheduled hours with hours of the full-time workweek. Disaggregation: Employment duration (existing researchers, newly employed researchers); employment origin (researchers employed at the beneficiary/partner institutions, external/ contracted, and visiting researchers); field/academic discipline; gender; seniority (PhD students, post-doctoral researchers, senior researchers, other) • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Output Number of Definition: The number of training activities (lectures, workshops, training sessions, training etc.) attended by researchers participating in the implementation during the project activities implementation period and financed by the program. In order to be taken into account as attended ‘training’, minimum half-day duration (four hours) of a capacity-building activity is required. Disaggregation: By field/academic discipline; by gender • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Output Number of Definition: Number of R&D projects conducted by grantees with the grant awarded under completed the program. This indicator will capture only completed projects, which is defined as research approval of the grantee’s final project implementation report and grantee receiving the projects final payment. Disaggregation: Research field; S3 thematic priority areas Outcome Number of Definition: The number of new collaborative research projects, involving the beneficiary collaborative of the supported project and at least one another entity, and contracted after the end of projects implementation of the supported project. Beneficiary organization is involved as either contracted the main beneficiary or partner in the implementation of the new projects contracted. after project completion Disaggregation: Collaboration novelty (new, existing); type of partner (research organization, enterprise, other); partner location (domestic, foreign, diaspora); type of research (basic, applied, experimental development); science area; science field; funding source (national, EU, private, other); S3 thematic priority area Bulgaria: Efficiency Analysis of STI Programs 193 Output Value of Definition: The value of new collaborative research projects, involving the beneficiary of collaborative the supported project and at least one another entity, and contracted after the end of projects implementation of the supported project. Beneficiary organization is involved as either contracted the main beneficiary or partner in the implementation of the new projects contracted. after project completion Disaggregation: Collaboration novelty (new, existing); type of partner (research organization, enterprise, other); partner location (domestic, foreign, diaspora); type of research (basic, applied, experimental development); science area; science field; funding source (national, EU, private, other); S3 thematic priority area. Output Number of Definition: The number of collaborative R&D projects conducted by grantees with collaborative the grant awarded under the program. Projects that are counted are implemented by research research organizations, in partnership with other research organizations. This indicator projects will capture only completed projects, which is defined as approval of the grantee’s final supported project implementation report and grantee receiving the final payment. For monitoring purposes, the indicator should track the projects that are contracted, ongoing and that have been completed. Disaggregation: Project status (awarded/ongoing, completed); project results (projects which achieved their objectives, projects with objectives partially achieved, projects which were discontinued due to irregularities and/or other reasons); research field; by S3 thematic priority areas; region; novelty of collaboration (new, existing); type of partner (research organization, enterprise, other); partner origin (Bulgarian, foreign). A.2 Vihren Program Objective(s) The long-term goal of the Vihren program is to significantly increase the quality of human resourc- es for conducting research in Bulgaria at the highest level, along with the implementation of a lasting positive change in the institutional culture to support research. The program has the following specific objectives: • Support the individual potential for high-level research and to attract additional research funding; to achieve this goal and in accordance with the principles of the ERC program of ex- cellence, the Vihren program does not pre-set the topics and scientific areas of the proposed projects (bottom-up approach); • Create a supportive institutional environment for the work of the research team, formed and led by the principal investigator of the project; • Improve the administrative and technical capacity of the host organization to support the preparation and implementation of national, European, and international projects. Bulgaria: Efficiency Analysis of STI Programs 194 Activities Funding of research projects carried out by leading and established scientists and the teams formed by them. Theory of Change As illustrated in the ToC diagram (Figure A.2), the program is designed with the program-level goal of increasing the capacity of the public research sector for conducting high quality research. It is important to note that grant funding from NSF cannot be used for any direct commercial ap- plication or use, so the ToC does not include any activities or outputs related to IP generation, technology transfer, or commercialization. To achieve this goal, the ToC relies on the following pathways of change: • Increased generation of impactful research from new research teams: the program supports established scientists in forming high quality research team around a funded research project. These high-quality research teams will, in turn, generate impactful research while working in Bulgarian public research institutions. • Increased connections to the international scientific community and the private sector in re- search: the program supports the recruitment of established researchers from abroad, who will engage in research collaborations with the international scientific community. As these established researchers are recruited, they will bring with them international connections and form new connections through new collaborative research projects. Bulgaria: Efficiency Analysis of STI Programs 195 Figure A.2. Vihren Theory of Change Diagram strategic increased capacity of the bulgarian public research sector to produce research goal outputs that are impactful on the economy and society increased capacity and reputation of public research sector program to perform high quality research goal • number of projects receiving additional funding after project completion • number of competitive projects contracted after project completion increased generation of impactiful increased connections to the research from new research teams international scientific community • number of completed research projects • number of collaborative projects outcome • number of scientific publications contracted after project completion published in scientific journals and • value of collaborative projects indexed in “Web of Science” contracted after project completions • number of IPR applications filed following project implementation high quality research teams formed increased collaboration with the at public research institutions international scientific community • number of funded Established Researchers • number of collaborative research projects output • number of funded Lead Researchers supported • Full-time aquivalent (FTE) researchers involved in project implementation • number of training activities attended activity • funding of research projects • funding of research collaborations • dissemination of research results input • grants for basic research projects Bulgaria: Efficiency Analysis of STI Programs 196 Results Framework A results framework aims to identify and define the indicators underlying the theory of change for the program. As such, the results framework identifies indicators for all the ToC elements so that program “success”, in terms of whether the intended change(s) occurred, can be measured, and verified. In this section, the proposed results framework is built from, and indexed to, the ToC elements, and revised (where necessary) according to the CART principles. As the theory of change and results framework is developer further, will need to be defined for each indicator, as well as the processes of collecting and verifying the data (including indicator verification period, process, and frequency). Level Indicator Definition Objective Number of Definition: Additional funding (grants) for continuation of research activities received project proposals by project beneficiaries, as the main applicant (coordinator, manager), or as a partner receiving (associate). Funding sources can be national, EU (Horizon 2020, ERC, FP8, ESI Funds), or additional other (public or private). funding after project Disaggregation: By success (approved, rejected); By funding sources (national, Horizon completion 2020, ERC, FP8, ESI Funds, other); by project role (lead beneficiary, partner) Objective Number of Definition: Competitive research projects awarded to project beneficiaries as the main competitive applicant (coordinator, manager), or as a partner (associate). Funding sources can be projects national, EU (Horizon 2020, ERC, FP8, ESI Funds), or other (public or private). contracted after project Disaggregation: Funding source (national, Horizon 2020, ERC, FP8, ESI Funds, other); completion project role (lead beneficiary, partner) Outcome Number of Definition: Scientific papers and reports published, describing original results and scientific research conducted within this project with the aim of informing the scientific publications community and society as a whole. Scientific papers need to be indexed in Web of published in Science platform (core collection), and may include articles, reviews, proceedings scientific journals papers, letters, and book chapters. and indexed in ‘Web of Science’ Disaggregation: Rank of journal where publication was published (first and second- quartile journals, other); by scientific areas; by scientific fields; by S3 thematic priority areas • Out of which: Number of joint publications with industry partners (Publications with at least one author from a research organization and one author from the industry) • Out of which: Number of joint publications with international coauthors ((Publications with at least one author from a Bulgarian organization and one author from an international organization [public or private]) Bulgaria: Efficiency Analysis of STI Programs 197 Outcome Number of IPR Definition: The number of IPR applications (including patents, trademarks, industrial applications filed design, etc.) by the project beneficiary and partners involved in project implementation, following project related to the research activities conducted in the scope of the financed project. implementation Applicant should state the expected number of IPR applications in the project application. In the post-implementation reporting phase, applicant should provide a description of a clear link between the conducted research activities and the IPR protection filed. Disaggregation: Type of IPR (patents, industrial design, trademarks); application status (filed, registered/approved); filing location (national, international) Output Number Definition: The number of R3 Established Researchers receiving funding, as defined of funded in the European Commission’s communication “Towards a European Framework for Established Research Careers” Researchers Disaggregation: Employment duration (existing researchers, newly employed researchers); employment origin (researchers employed at the beneficiary/partner institutions, external/contracted and visiting researchers); field/academic discipline; gender • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Output Number of Definition: The number of R4 Lead Researchers receiving funding, as defined in the funded Lead European Commission’s communication “Towards a European Framework for Research Researchers Careers” Disaggregation: Employment duration (existing researchers, newly employed researchers); employment origin (researchers employed at the beneficiary/partner institutions, external/contracted and visiting researchers); field/academic discipline; gender • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Output Number of full- Definition: Number of researchers that directly carry out research and development time equivalent activities related to the implementation of the project. Workforce may be existing (FTE) researchers or new, employed at the beneficiary and partners, or contracted from third parties. involved Auxiliary staff for R&D activities (jobs that are not directly involved in R&D activities) in project are not included in this indicator and should not be counted. The measurement unit is implementation “Full-time equivalent”. Engagement of researchers employed on the research activities supported by the project less than full-time should be converted to the number of FTE employees by dividing the researchers’ scheduled hours with hours of the full-time workweek. Disaggregation: Employment duration (existing researchers, newly employed researchers); employment origin (researchers employed at the beneficiary/partner institutions, external/contracted and visiting researchers); field/academic discipline; gender; seniority (PhD students, post-doctoral researchers, senior researchers, other) • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Bulgaria: Efficiency Analysis of STI Programs 198 Output Number of Definition: The number of training activities (lectures, workshops, training sessions, training activities etc.) attended by researchers participating in the implementation during the project attended implementation period and financed by the program. In order to be taken into account as ‘training’, minimum half-day duration (four hours) of a capacity-building activity is required. Disaggregation: By field/academic discipline; by gender • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Output Number of Definition: Number of R&D projects conducted by grantees with the grant awarded completed under the program. This indicator will capture only completed projects, which is defined research projects as approval of the grantee’s final project implementation report and grantee receiving the final payment. Disaggregation: Research field; S3 thematic priority areas Outcome Number of Definition: The number of new collaborative research projects, involving the beneficiary collaborative of the supported project and at least one another entity, and contracted after the end of projects implementation of the supported project. Beneficiary organization is involved as either contracted the main beneficiary or partner in the implementation of the new projects contracted. after project completion Disaggregation: Collaboration novelty (new, existing); type of partner (research organization, enterprise, other); partner location (domestic, foreign, diaspora); type of research (basic, applied, experimental development); science area; science field; funding source (national, EU, private, other); S3 thematic priority area Output Value of Definition: The value of new collaborative research projects, involving the beneficiary of collaborative the supported project and at least one another entity, and contracted after the end of projects implementation of the supported project. Beneficiary organization is involved as either contracted the main beneficiary or partner in the implementation of the new projects contracted. after project completion Disaggregation: Collaboration novelty (new, existing); type of partner (research organization, enterprise, other); partner location (domestic, foreign, diaspora); type of research (basic, applied, experimental development); science area; science field; funding source (national, EU, private, other); S3 thematic priority area. Output Number of Definition: The number of collaborative R&D projects conducted by grantees with collaborative the grant awarded under the program. Projects that are counted are implemented by research projects research organizations, in partnership with other research organizations. This indicator supported will capture only completed projects, which is defined as approval of the grantee’s final project implementation report and grantee receiving the final payment. For monitoring purposes, the indicator should track the projects that are contracted, ongoing and that have been completed. Disaggregation: Project status (awarded/ongoing, completed); project results (projects which achieved their objectives, projects with objectives partially achieved, projects which were discontinued due to irregularities and/or other reasons); research field; by S3 thematic priority areas; region; novelty of collaboration (new, existing); type of partner (research organization, enterprise, other); partner origin (Bulgarian, foreign). Bulgaria: Efficiency Analysis of STI Programs 199 A.3 ICT for a Single Digital Market in Science, Education, and Security Program Objective(s) The program was created with the specific goals of: • Ensuring free access to electronic infrastructure for teams of students and doctoral students from various scientific disciplines; • Identifying opportunities for scientific applications in key areas such as medicine and bioin- formatics, ecology and biodiversity, new materials, transport, energy efficiency, humanities and social sciences, etc.; • Formation of interdisciplinary teams corresponding to the best achievements in information technology; • Modeling of products and phenomena with processing of large volumes of data and complex mathematical and computer models; • Expanding the participation of Bulgarian research teams in European projects and in research programs. Activities 1. Developing electronic infrastructure for open science and open access to scientific results a. Development of the Center for High Performance and Distributed Computing as national electronic infrastructure b. Computer modeling of problems and tasks in the field of natural sciences with technolog- ical applications c. Creation of a national electronic library for open scientific results d. Creating a modern infrastructure for three-dimensional digitization e. Storage and analysis of large volumes of data and technologies for the application of AI Development of the Center for Grid and Cloud Technologies built at Sofia University 2. Creating digital technologies in education a. Creating national publicly available educational resources with materials created with public funding b. Language and content-based technologies for the better education Development of lan- guage and content-based technologies modern means to improve formal and non - formal learning c. Introduction of augmented virtual reality and three - dimensional models (real and virtual) for illustrating learning material that offers a new, more attractive approach to learning of new knowledge Bulgaria: Efficiency Analysis of STI Programs 200 3. Improving information security related to open science a. Security of the electronic infrastructure for open science. The national electronic infra- structure managed by the consortium operating in this national program will implements policies and good practices of European initiatives for access to electronic infrastructures, including the certified access system. b. Establishment of a research center for monitoring and development of preventive mea- sures information security and incident response policies. Theory of Change The ICT for a Single Digital Market in Science, Education, and Security program is somewhat dif- ficult to fit into a single theory of change, given the number and diversity of activities supported by the program. The ToC described in this report puts all of these activities into a single ToC, but an alternative could be considering the program as two separate activities with separate ToCs – (i) open science infrastructure and security and (ii) digital educational tools. As illustrated in the ToC diagram below (Figure A.3), the program is designed with the program-lev- el goal of increasing the capacity of the Bulgarian public research sector to perform high quality research. To achieve this goal, the ToC relies on the following pathways of change: • Improved open science infrastructure through the development of open science infrastruc- ture and tools: the program seeks to improve research collaboration (domestically and inter- nationally) and research excellence through the creation of electronic infrastructure for open science and open access to scientific results, developing new digital tools, and improving in- formation security related to open science. • Improved use of digital technologies in the Bulgarian education system: the program seeks to improve the use of digital technologies in the Bulgarian education system (and thus improve the digital skills of the future workforce) through the development of digital educational re- sources and content for use in education, and by improving information security related to digital educational tools. Bulgaria: Efficiency Analysis of STI Programs 201 Figure A.3. ICT Theory of Change Diagram strategic goal INCREASED CAPACITY OF BULGARIAN RESEARCH SYSTEMS TO PRODUCE IMPACTFUL RESEARCH increased OPEN SCIENCE COLLABORATION USING NEWLY DEVELOPED OPEN SCIENCE program INFRASTRUCTURE AND DIGITAL TOOLS goal • number of users using new open science infrastructure and tools • number of educational institutions using newly developed educational tools new open science infrastructure creation of digital educational and tools tools and consent • number of open science tools developed • number of educational tools developed outcome • number of publications and datasets made available through new open science infrastructure • number of scientific publications published in scientific journals and indexed in “Web of Science” from supported projects developed of open development of creation of digital science infrastructure information security for educational tools and and tools open science content • number of open science • number of cyber security • number of digital educational infrastructure projects tools developed projects supported supported • number of publications output • number of trainings related published in scientific journals to using open science and indexed in “Web of Science” infrastructure and tools from supported projects • number of publications published in scientific journals and indexed in “Web of Science” from supported projects • funding of open • funding for development activity • funding for information infrastructure science of educational tools and security activities infrastructure materials input • number of grants awarded • value of grants awarded Bulgaria: Efficiency Analysis of STI Programs 202 Results Framework A results framework aims to identify and define the indicators underlying the theory of change for the program. As such, the results framework identifies indicators for all the ToC elements so that program “success”, in terms of whether the intended change(s) occurred, can be measured and verified. In this section, the proposed results framework is built from, and indexed to, the ToC elements, and revised (where necessary) according to the CART principles. As the theory of change and results framework is developer further, will need to be defined for each indicator, as well as the processes of collecting and verifying the data (including indicator verification period, process, and frequency). Level Indicator Definition Objective Number of users using new Definition: Number of users of open science infrastructure and tools whose open science infrastructure development was supported by this project. and tools Disaggregation: Type of user (public researcher, student, general public); user origin (domestic, foreign, diaspora); field/academic discipline • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Objective Number of educational Definition: Number of educational institutions using newly developed institutions using newly educational tools. developed educational tools Disaggregation: Type of institution (HEI, secondary education institution, primary education institution, other); type of educational material (tool, educational module, other) Outcome Number of open science Definition: Number of computational tools and applications developed with tools and applications the support of the program. The tools and applications must be accessible developed through open science protocols. Disaggregation: Scientific field; type of research (basic, applied, experimental development); type of tool (software, technology) Outcome Number of publications Definition: Number of scientific publications, datasets, and other documents and datasets made made available through new open science infrastructure developed with available through new open support of the project. science infrastructure Disaggregation: Type (publication, dataset, other); scientific field; S3 thematic priority area Bulgaria: Efficiency Analysis of STI Programs 203 Outcome Number of scientific Definition: Scientific papers and reports published, describing original results publications published and research conducted within this project with the aim of informing the in scientific journals and scientific community and society as a whole. Scientific papers need to be indexed in ‘Web of Science’ indexed in Web of Science platform (core collection), and may include articles, from supported projects reviews, proceedings papers, letters, and book chapters. Disaggregation: Rank of journal where publication was published (first and second-quartile journals, other); by scientific areas; by scientific fields; by S3 thematic priority areas • Out of which: Number of joint publications with industry partners (Publications with at least one author from a research organization and one author from the industry) • Out of which: Number of joint publications with international coauthors ((Publications with at least one author from a Bulgarian organization and one author from an international organization [public or private]) Outcome Number of educational Definition: The number of digital educational tools and applications tools and applications developed by the program. To be counted, the tool or application must have developed reached the pilot stage of development and been tested by at least five users. Disaggregation: Type of tool (software, technology) Output Number of open science Definition: The number of open science projects conducted by grantees with projects supported the grant awarded under the program. For monitoring purposes, the indicator should track the projects that are contracted, ongoing, and completed. Disaggregation: Project status (awarded/ongoing, completed); project results (projects which achieved their objectives, projects with objectives partially achieved, projects which were discontinued due to irregularities and/or other reasons) Output Number of users trained Definition: The number of users participating in trainings (lectures, workshops, on using open science training sessions, etc.) on the use of new open science infrastructure and infrastructure and tools tools organized by the program. “Training” sessions must be a minimum half- day duration (four hours) to be counted. Disaggregation: Field/academic discipline; gender; seniority (Non-PhD students, PhD students, post-doctoral researchers, senior researchers, other) Out of which: Number of young researchers (research students, PhD students, early-stage researchers) receiving capacity-building support Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Bulgaria: Efficiency Analysis of STI Programs 204 Output Number of cyber security Definition: The number of cyber security tools and applications developed by tools developed the program. To be counted, the tool or application must have reached the pilot stage of development and been tested by at least five users. Disaggregation: Type of tool (software, technology) Output Number of digital Definition: The number of digital projects conducted by grantees with the education projects grant awarded under the program. For monitoring purposes, the indicator supported should track the projects that are contracted, ongoing, and completed. Disaggregation: Project status (awarded/ongoing, completed); project results (projects which achieved their objectives, projects with objectives partially achieved, projects which were discontinued due to irregularities and/or other reasons) A.4 eHealth Program Objective(s) Improving the quality and timeliness of medical care while optimizing the costs of computerized activities and reducing the workload of medical staff with technical and administrative tasks. The program follows the guidelines of the European Action Plan for the introduction of e-health by 2020 and is established in implementation of the National Strategy for Research Development (NRNI) 2030 to achieve specific objectives: • Encouragement of problem-oriented scientific research in the priority areas of the Innovation Strategy for Smart Specialization (ISIS); • Significant intensification of the connections of science with education, with business, with state bodies and with the society as a whole; • Avoiding fragmentation in Bulgarian science by promoting integration and interaction between different public research institutes and universities in order to build a critical mass and avoid overlapping and duplication of resources. Activities Creating preconditions for overcoming the fragmentation of the data generated in healthcare: 1. A set of defined essential requirements for systems for processing and exchange of clinical data; 2. A formal model of systems for processing and exchange of clinical data; 3. Building primary pseudonymous anonymous databases; 4. Creating prototypes and demonstrations of semantic interoperability of information health services; Bulgaria: Efficiency Analysis of STI Programs 205 Theory of Change (ToC) As illustrated in the ToC diagram (Figure A.4), the program is designed with the program-level goal of increasing the capacity of the public research sector for conducting high quality research To achieve this goal, the ToC relies on the following pathways of change: • Improved interoperability of health information through the development of health data collection and processing tools and technologies: the program seeks to improve tools and techniques for the collection and sharing of health data though research projects aimed at developing new tools and technologies for collecting, analyzing, and processes health data and the demonstration of such technologies to public health officials. • Improved access to health data through the development of health data exchanges and plat- forms: the program seeks to improve access to health data through research projects aimed at the development of new health data exchanges and platforms and training public health officials and researchers in their use. Bulgaria: Efficiency Analysis of STI Programs 206 Figure A.4. eHealth Theory of Change Diagram strategic goal INCREASED CAPACITY OF BULGARIAN RESEARCH SYSTEMS TO PRODUCE IMPACTFUL RESEARCH increased access ti and interoperability of health data by public health officials program goal • number of new health data collection and processing technologies in use in a public health setting • number of new health data exchanges and platforms accessible to public health officials new health data collection and new health data exchange processing technologies and platforms • number of health data collection and • number of health data exchanges processing technologies developed and platforms developed outcome • number of health data collection and • number of publications published in processing technologies demonstrated scientific journals and indexed in “Web of in a public health setting Science” from supported projects • number of publications published in scientific journals and indexed in “Web of Science” from supported projects development of health data collection development of health data exchanges and processing technologies and platforms output • number of health data collection and • number of health data exchanges and processing technologies supported platforms supported • number of trainings related to using health • number of trainings related to using health data collection and processing technologies data exchanges and platforms activity • funding for health data technologies and platforms • dissemination of project results input • number of grants awarded • value of grants awarded Bulgaria: Efficiency Analysis of STI Programs 207 Results Framework A results framework aims to identify and define the indicators underlying the theory of change for the program. As such, the results framework identifies indicators for all the ToC elements so that program “success”, in terms of whether the intended change(s) occurred, can be measured and verified. In this section, the proposed results framework is built from, and indexed to, the ToC elements, and revised (where necessary) according to the CART principles. Level Indicator Definition Objective Number of new health Definition: Number of health data collection tools and technologies whose data collection and development was supported by this project that are deployed for practical use processing technologies (i.e., non-experimental use) in a public health setting (such as a hospital, clinic, in use in a public health laboratory, or other medical setting) setting Disaggregation: Type of public health setting (hospital, clinic, laboratory, or other medical setting) Objective Number of new health Definition: Number of users of new health data exchanges and platforms data exchanges and whose development was supported by this project. platforms accessible to public health officials Disaggregation: Type of user (researcher, medical practitioner, student, other); user origin (domestic, foreign, diaspora) • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Outcome Number of health data Definition: Number of health data collection and processing technologies collection and processing developed with the support of the program. technologies developed Disaggregation: Scientific field; type of research (basic, applied, experimental development); type of tool (software, technology) Outcome Number of health Definition: Number of health data collection and processing technologies data collection and developed with support of the project that have been demonstrated in a processing technologies public health setting (such as a hospital, clinic, laboratory, or other medical demonstrated in a public setting). health setting Disaggregation: Type of public health setting (hospital, clinic, laboratory, or other medical setting) Outcome Number of health data Definition: Number of health data exchanges and platforms developed with exchanges and platforms the support of the program. developed Disaggregation: Type of access (restricted, open) Bulgaria: Efficiency Analysis of STI Programs 208 Outcome Number of scientific Definition: Scientific papers and reports published, describing original results publications published and research conducted within this project with the aim of informing the in scientific journals and scientific community and society as a whole. Scientific papers need to be indexed in ‘Web of Science’ indexed in Web of Science platform (core collection), and may include articles, from supported projects reviews, proceedings papers, letters, and book chapters. Disaggregation: Rank of journal where publication was published (first and second-quartile journals, other); by scientific areas; by scientific fields; by S3 thematic priority areas • Out of which: Number of joint publications with industry partners (Publications with at least one author from a research organization and one author from the industry) • Out of which: Number of joint publications with international coauthors ((Publications with at least one author from a Bulgarian organization and one author from an international organization [public or private]) Output Number of eHealth Definition: The number of eHealth projects conducted by grantees with the projects supported grant awarded under the program. For monitoring purposes, the indicator should track the projects that are contracted, ongoing, and completed. Disaggregation: Project status (awarded/ongoing, completed); project results (projects which achieved their objectives, projects with objectives partially achieved, projects which were discontinued due to irregularities and/or other reasons) Output Number of training Definition: The number of training activities (lectures, workshops, training activities related to sessions, etc.) on the use of eHealth technologies and platforms organized eHealth technologies and or attended by the program. “Training” sessions must be a minimum half-day platforms duration (four hours) to be counted. Disaggregation: Field/academic discipline; gender • Out of which: Number of young researchers (research students, PhD students, early-stage researchers) receiving capacity-building support • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Bulgaria: Efficiency Analysis of STI Programs 209 A.5 National Innovation Fund Program Objective(s) The main goal of NIF is to promote the research and development activity for increasing the com- petitiveness of the enterprises. Activities Grants to support scientific and research and development projects for a period of implementa- tion from 12 to 36 months. Grants should not exceed 50 percent of project costs. Grants for technical feasibility projects for a period of implementation up to 1 year. Grants should not exceed 25 percent of project costs. Theory of Change As illustrated in the ToC diagram (Figure A.5), the program is designed with the program-level goal of promoting the research and development activity in enterprises for increased competi- tiveness of the economy. To achieve this goal, the ToC relies on the following pathways of change: • Increase in product development: The program aims to support firms to develop new products and services by providing grants that support R&D projects, fund research staff, and provide access to improved research equipment and facilities. • Increased collaboration with public research organizations and participation in technology transfer: the program supports firms to engage in knowledge and technology transfer with public sector research organizations by providing grant funding for collaborative public-private research projects and innovation advisory support services (from public research organizations). Bulgaria: Efficiency Analysis of STI Programs 210 Figure A.5. National Innovation Fund Theory of Change Diagram strategic goal INCREASED international competitiveness of bulgarian enterprises program bulgarian enterprises introduce new or significantly improved products and services with growth and export potential in the market goal • change in sales revenue after project completion     • change in employment after project completion increased product development increased collaboration with public research organizations and participation in technology transfer • number of product innovations introduced • number of process innovations introduced • number of technology transfers realized after • value of private investments in R&D project after project completion project implementation outcome • number of IPR applications filed due to project implementation • value of technology transfers realized after project implementation • number of collaborations after project implementation increased financial increased personnel improved availability increased knowledge improved public-private resources for r&d in working on innovation of quality research and technology research collaboration supported firms activities facilities and equipment transfer activities • number of collaborative • value of private • number of full-time • number of contracts to • number of enterprises research projects output investment matching equivalent (FTE) staff access public research that received innovation supported public support in RDI involved in project infrastructure advisory support • number of enterprises projects implementation • value of new equipment collaborating with public • number of enterprises purchased for innovation research organizations supported projects • funding • funding access to public • funding costs of activity • funding of research intangible assets • funding personnel costs research infrastructure innovation advisory and projects • funding of collaborative • funding tangible assets support services research projects input • number of grants awarded • number of grants disbursed • value of grants awarded • value of grants disbursed Bulgaria: Efficiency Analysis of STI Programs 211 Results Framework A results framework aims to identify and define the indicators underlying the theory of change for the program. As such, the results framework identifies indicators for all the ToC elements so that program “success”, in terms of whether the intended change(s) occurred, can be measured and ver- ified. In this section, the proposed results framework is built from, and indexed to, the ToC elements, and revised (where necessary) according to the CART principles. As the theory of change and results framework is developer further, will need to be defined for each indicator, as well as the processes of collecting and verifying the data (including indicator verification period, process, and frequency). Level Indicator Definition Objective Change in Definition: The change in sales revenue of the supported enterprises after project sales revenue completion. The indicator is calculated as the difference between the value of sales after project revenue of the supported enterprises in the year preceding the submission of the project completion application (baseline value) and the annual value of sales revenue up to five years after project completion (target value), expressed in absolute numbers. Sales revenue is taken as annual gross sales revenue, which should not include any grant support received by the enterprise. Disaggregation: Change in revenue from sales abroad (export) Objective Change in Definition: The change in the gross number of full-time equivalent (FTE) employees of employment the supported enterprises after project completion. The indicator is calculated as the after project difference between the number of employees (FTE) in the year preceding the submission completion of the project application (baseline value) and the number of employees (FTE) up to five years after project completion (target value). Disaggregation: S3 thematic priority area; gender • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Outcome Number of Definition: The number of new product innovations introduced by supported product entities, during and after project implementation period. A product innovation is the innovations introduction of a good or service that is new or significantly improved with respect to introduced its characteristics or intended uses. This includes significant improvements in technical specifications, components and materials, incorporated software, user friendliness or other functional characteristics. Product innovations can utilize new knowledge or technologies or can be based on new uses or combinations of existing knowledge or technologies. The term ‘product’ is used to cover both goods and services. Product innovations include both the introduction of new goods and services and significant improvements in the functional or user characteristics of existing goods and services (Source: OECD/Eurostat). The number of product innovations is reported cumulatively, up to a data collection point set in the post-implementation period. Disaggregation: S3 thematic priority area; industry; market (domestic, international); type (goods, services); novelty (new, improved) Bulgaria: Efficiency Analysis of STI Programs 212 Outcome Number of Definition: The number of new process innovations introduced by supported process entities, during and after project implementation period. A process innovation is the innovations implementation of a new or significantly improved production or delivery method. introduced This includes significant changes in techniques, equipment and/or software. Process innovations can be intended to decrease unit costs of production or delivery, to increase quality, or to produce or deliver new or significantly improved products. Process innovations include new or significantly improved methods for the creation and provision of services. They can involve significant changes in the equipment and software used in services-oriented firms or in the procedures or techniques that are employed to deliver services. Process innovations also cover new or significantly improved techniques, equipment, and software in ancillary support activities, such as purchasing, accounting, computing, and maintenance (Source: OECD/Eurostat). The number of process innovations is reported cumulatively, up to a data collection point set in the post- implementation period. Disaggregation: S3 thematic priority area; industry; type of process Outcome Value of private Definition: the total value of private investment by supported enterprises to undertake investment in R&D activities after the end of the supported project. This amount reflects the increase in R&D projects private sector expenditure in R&D after project completion, measured on annual basis up after project to 5 years after the grant period, excluding future grants awarded to the enterprise. completion Disaggregation: Investment purpose (cost category); S3 thematic priority area; industry Outcome Number of IPR Definition: The number of IPR applications (including patents, trademarks, industrial applications design, etc.) by the project beneficiary and partners involved in project implementation, filed following related to the research activities conducted in the scope of the financed project. project Applicant should state the expected number of IPR applications in the project application. implementation In the post-implementation reporting phase, applicant should provide a description of a clear link between the conducted research activities and the IPR protection filed. Disaggregation: Type of IPR (patents, industrial design, trademarks); application status (filed, registered/approved); filing location (national, international) Outcome Number of Definition: The number of new collaborative research projects, involving the beneficiary collaborative of the supported project and at least one another entity, and contracted after the end of projects implementation of the supported project. Beneficiary organization is involved as either contracted the main beneficiary or partner in the implementation of the new projects contracted. after project completion Disaggregation: Collaboration novelty (new, existing); type of partner (research organization, enterprise, other); partner location (domestic, foreign, diaspora); type of research (basic, applied, experimental development); science area; science field; funding source (national, EU, private, other); S3 thematic priority area Bulgaria: Efficiency Analysis of STI Programs 213 Output Value of private Definition: the total value of private contribution in supported RDI projects, including investment non-eligible parts of the project. The amount is an addition to the public funds received matching public from the program. The amount is calculated by subtracting the public funds (including support in RDI the grant value and other contributions from public sources, if applicable) from the total projects project value (including eligible and non-eligible project costs). Disaggregation: S3 thematic area; industry Output Number of Definition: The number of enterprises supported by grants awarded through the program, enterprises in order to develop and launch new or improved products (including goods and services) supported on the market. Double counting is avoided, meaning a single enterprise is counted only once, regardless of the number of projects it is supported through. If an enterprise is supported to develop more than one product, either through a single or more than one projects, it is counted as one. Indicator achievement is subject to project completion. Disaggregation: S3 thematic priority areas; industry; type of product (goods, services); product novelty (new, improved) Output Number of Definition: The number of staff that directly carry out activities related to the full-time implementation of the project. Workforce may be existing or new. The measurement unit equivalent (FTE) is “Full-time equivalent”. Engagement of staff employed on the activities supported by staff involved the project less than full-time should be converted to the number of FTE employees by in project dividing the employees’ scheduled hours with hours of the full-time workweek. implementation Disaggregation: Employment duration (existing, newly employed); gender • Out of which: Number of full-time equivalent (FTE) researchers involved in the project (employed in enterprises) • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Outputs Number of Definition: The number of contracts between supported enterprises and public research contracts to organizations to provide access to public research infrastructure access public research Disaggregation: S3 thematic area; industry infrastructure Outputs Value of new Definition: The value of innovation equipment or machinery purchased with the support equipment of the program. The machinery or equipment must be used for product development or purchased customization of innovative products. for innovation projects Disaggregation: S3 thematic priority area; industry Bulgaria: Efficiency Analysis of STI Programs 214 Outcome Number of Definition: Transfers of research results (knowledge and technology) realized due to technology project implementation, with purpose of their further development and/or their use transfers in development and commercialization of new products (goods or services). Project realized results can be transferred from project beneficiary and/or partners to third parties in after project the form of signed R&D agreements or contracts and intellectual property (IP) licensing implementation agreements or the transfer can be realized through establishment of new enterprises. More specifically, the following are the types of technology transfer models captured (and disaggregated) by the indicator: • Number of new start-ups/spin-offs/spinouts originating out of supported projects: Number of new enterprises (start-up/spin-off/spin-out) established by project beneficiary and/or project partners as a result of funded project. Start-up is an enterprise less than 3 years old. Spin-off is an enterprise that has been started by a University group, but which has never left the university environment and perhaps exists to offer specialist consultancy services without the intension for any further expansion or full technology transfer. Spin-out is an enterprise in which the university or Institute has an equity stake. • Number of licensing agreements signed with the industry: The licensing agreement is a contract to be used by enterprises for technology transfer through granting rights of industrial ownership (license of patents and trademarks). • Number of other technology transfer agreements signed with the industry: Any other forms of technology transfer/commercialization agreements signed with the enterprises. • Number of contract research agreements with the industry. Disaggregation: Type of technology transfer activity (as listed in the definition) Outcome Value of Definition: The value of contractual research conducted (research services acquired by technology enterprises from research organizations), or knowledge and patents bought or licensed by transfers supported entities from outside sources, under market conditions and for the purposes of realized due implementation of the project supported. to project implementation Disaggregation: Type of transfer supported (IPR acquisition or licensing, contractual research); for contractual research: type service provided (product and process testing, demonstration activities, professional and technical knowledge for the purpose of product and process development etc.) Output Number of Definition: the number of enterprises that received innovation advisory support, such enterprises as advisory support in acquisition, protection and exploitation of intangible assets, that received application of norms and regulations that cover them, product development, design and innovation testing, market research and analysis, development of marketing plans, preparation of advisory feasibility studies or similar activities related to product innovation specifically related to support the activities conducted through the supported project. Disaggregation: Innovation advisory support purpose Bulgaria: Efficiency Analysis of STI Programs 215 Output Number of Definition: The number of collaborative R&D projects conducted by grantees with the collaborative grant awarded under the program. Projects that are counted are implemented by research research organizations, in partnership with other research organizations. This indicator will capture projects only completed projects, which is defined as approval of the grantee’s final project supported implementation report and grantee receiving the final payment. For monitoring purposes, the indicator should track the projects that are contracted, ongoing and that have been completed. Disaggregation: Project status (awarded/ongoing, completed); project results (projects which achieved their objectives, projects with objectives partially achieved, projects which were discontinued due to irregularities and/or other reasons); research field; by S3 thematic priority areas; region; novelty of collaboration (new, existing); type of partner (research organization, enterprise, other); partner origin (Bulgarian, foreign) Output Number of Definition: The number of supported enterprises collaborating with public research enterprises organizations in R&D projects. At least one enterprise and one research organization must collaborating collaborate in the project. The collaboration may be new or a continuation of existing with public collaboration and must last at least as long as the project. All enterprises participating research in the project as partners are counted as contributing to the indicator. Double counting organizations is avoided, meaning a single enterprise is counted only once regardless of the number of projects it is participating in. Disaggregation: S3 thematic priority areas; region; novelty of collaboration (new, existing) A.6 Development of Product and Process Innovations Program Objective(s) The main goal of the procedure is to provide integrated consulting and investment support to Bulgarian enterprises in the implementation of successful projects for implementation in the production and marketing of innovative products, processes, and services (the services should be the result of implemented innovative process or product). Activities 1. Investments a. Purchase of new equipment necessary for implementation in the production of innovative products, processes or services created by the applicant or contributed to the applicant’s capital by the patent holder / holder of the utility model certificate or created on the basis of a contract under which the applicant is a contracting authority; b. Purchase of complete units, aggregates and details, which are assigned when creating parts of or the entire production line for the production of the innovative product / realiza- tion of the innovative process; c. Improving the functional characteristics of existing buildings and / or production facilities through construction and installation works (construction and installation works), leading Bulgaria: Efficiency Analysis of STI Programs 216 to an increase in their value, when this is directly related to the investments in the element and is necessary for their commissioning. Construction and installation costs must not exceed 30% of the total eligible costs of the project; d. Acquisition of intangible fixed assets directly related to the implementation of innovative products, processes, or services - purchase of specialized software related to the imple- mentation of the innovative product, process, or service. 2. Services a. Services related to the optimization of production technologies, respectively for the cre- ation or implementation of the innovative product, process or service; b. Development of specialized software related to the implementation of the innovative prod- uct, process or service; c. Development of business and marketing strategies for the implementation of innovative products, processes, services implemented in production - conducting marketing research and analysis, planning channels for communication with the market and customers, de- velopment of distribution networks; d. Consulting and legal services related to the protection of intellectual property rights intel- lectual property on the innovative products / processes / services implemented under the project. These services can be financed under the project only if they concern the granting of the rights of the applicant for intellectual property over the innovation implemented by the project to other economic entities (for example, licensing). Consulting and legal ser- vices do not concern initial protection related to filing applications before a patent office or legal protection in case of contesting intellectual property rights; e. Consulting services of engineering and technical nature related to the process of imple- mentation of the innovative product, process or service. Theory of Change As illustrated in the ToC diagram (Figure A.6), the program is designed with the program-level goal of supporting firms in introducing new or significantly improved products and services with growth and export potential in the market. To achieve this goal, the ToC relies on the following pathways of change: • Increased product development: The program aims to support firms to develop new products and services by providing grants that support innovation projects, funding research staff, in- vestments in equipment and facilities, and support for consulting and advisory services. Bulgaria: Efficiency Analysis of STI Programs 217 Figure A.6. DPPI Theory of Change Diagram strategic goal INCREASED international competitiveness of bulgarian enterprises bulgarian enterprises introduce new or significantly improved products and services with program growth and export potential in the market goal • change in sales revenue after project completion • change in employment after project completion increased project development outcome • number of product innovations introduced • number of process innovations introduced • value of private investments in R&D project after project completion • number of IPR applications filed due to project implementation increased financial increased improved increased access resources for personnel working availability of to innovation innovation in on innovation quality innovation advisory and supported firms activities facilities and support services equipment (e.g. product • value of private • number of full-time development, investment matching equivalent (FTE) staff • value of new marketing, output public support in involved in project equipment purchased strategic, innovation projects implementation for innovation legal, and other consulting services • number of projects enterprises • value of infrastruc­ • number of supported ture development innovation projects or upgrades for receiving innovation innovation projects advisory support • product development activity • funding of research • funding of personnel • funding of tangible • marketing strategy projects costs assets development • IP protection input • number of grants awarded • number of grants disbursed • value of grants awarded • value of grants disbursed Bulgaria: Efficiency Analysis of STI Programs 218 Results Framework A results framework aims to identify and define the indicators underlying the theory of change for the program. As such, the results framework identifies indicators for all the ToC elements so that program “success”, in terms of whether the intended change(s) occurred, can be measured and verified. In this section, the proposed results framework is built from, and indexed to, the ToC elements, and revised (where necessary) according to the CART principles. As the theory of change and results framework is developer further, will need to be defined for each indicator, as well as the processes of collecting and verifying the data (including indicator verification period, process, and frequency). Level Indicator Definition Objective Change in sales Definition: The change in sales revenue of the supported enterprises after project revenue after project completion. The indicator is calculated as the difference between the value of sales completion revenue of the supported enterprises in the year preceding the submission of the project application (baseline value) and the annual value of sales revenue up to five years after project completion (target value), expressed in absolute numbers. Sales revenue is taken as annual gross sales revenue, which should not include any grant support received by the enterprise. Disaggregation: Change in revenue from sales abroad (export) Objective Change in Definition: The change in the gross number of full-time equivalent (FTE) employees employment after of the supported enterprises after project completion. The indicator is calculated project completion as the difference between the number of employees (FTE) in the year preceding the submission of the project application (baseline value) and the number of employees (FTE) up to five years after project completion (target value). Disaggregation: S3 thematic priority area; gender • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Bulgaria: Efficiency Analysis of STI Programs 219 Outcome Number of product Definition: The number of new product innovations introduced by supported innovations entities, during and after project implementation period. A product innovation is the introduced introduction of a good or service that is new or significantly improved with respect to its characteristics or intended uses. This includes significant improvements in technical specifications, components and materials, incorporated software, user friendliness or other functional characteristics. Product innovations can utilize new knowledge or technologies or can be based on new uses or combinations of existing knowledge or technologies. The term ‘product’ is used to cover both goods and services. Product innovations include both the introduction of new goods and services and significant improvements in the functional or user characteristics of existing goods and services (Source: OECD/Eurostat). The number of product innovations is reported cumulatively, up to a data collection point set in the post- implementation period. Disaggregation: S3 thematic priority area; industry; market (domestic, international); type (goods, services); novelty (new, improved) Outcome Number of process Definition: The number of new process innovations introduced by supported innovations entities, during and after project implementation period. A process innovation is the introduced implementation of a new or significantly improved production or delivery method. This includes significant changes in techniques, equipment and/or software. Process innovations can be intended to decrease unit costs of production or delivery, to increase quality, or to produce or deliver new or significantly improved products. Process innovations include new or significantly improved methods for the creation and provision of services. They can involve significant changes in the equipment and software used in services-oriented firms or in the procedures or techniques that are employed to deliver services. Process innovations also cover new or significantly improved techniques, equipment, and software in ancillary support activities, such as purchasing, accounting, computing, and maintenance (Source: OECD/Eurostat). The number of process innovations is reported cumulatively, up to a data collection point set in the post-implementation period. Disaggregation: S3 thematic priority area; industry; type of process Outcome Value of private Definition: the total value of private investment by supported enterprises to investment in undertake R&D activities after the end of the supported project. This amount reflects R&D projects after the increase in private sector expenditure in R&D after project completion, measured project completion on annual basis up to 5 years after the grant period, excluding future grants awarded to the enterprise. Disaggregation: Investment purpose (cost category); S3 thematic priority area; industry Bulgaria: Efficiency Analysis of STI Programs 220 Outcome Number of IPR Definition: The number of IPR applications (including patents, trademarks, industrial applications filed design, etc.) by the project beneficiary following project implementation, related to following project the innovation activities conducted in the scope of the financed project. Applicant implementation should state the expected number of IPR applications in the project application. In the post-implementation reporting phase, applicant should provide a description of a clear link between the conducted research activities and the IPR protection filed. Disaggregation: Type of IPR (patents, industrial design, trademarks); application status (filed, registered/approved); filing location (national, international) Output Value of private Definition: the total value of private contribution in supported RDI projects, including investment non-eligible parts of the project. The amount is an addition to the public funds matching public received from the program. The amount is calculated by subtracting the public funds support in RDI (including the grant value and other contributions from public sources, if applicable) projects from the total project value (including eligible and non-eligible project costs). Disaggregation: S3 thematic area; industry Output Number of Definition: The number of enterprises supported by grants awarded through the enterprises program, in order to develop and launch new or improved products (including goods supported and services) on the market. Double counting is avoided, meaning a single enterprise is counted only once, regardless of the number of projects it is supported through. If an enterprise is supported to develop more than one product, either through a single or more than one projects, it is counted as one. Indicator achievement is subject to project completion. Disaggregation: S3 thematic priority areas; industry; type of product (goods, services); product novelty (new, improved) Output Number of full- Definition: The number of staff that directly carry out activities related to the time equivalent implementation of the project. Workforce may be existing or new. The measurement (FTE) staff unit is “Full-time equivalent”. Engagement of staff employed on the activities involved in project supported by the project less than full-time should be converted to the number of implementation FTE employees by dividing the employees’ scheduled hours with hours of the full- time workweek. Disaggregation: Employment duration (existing, newly employed); gender • Out of which: Number of full-time equivalent (FTE) researchers involved in the project (employed in enterprises) • Out of which: Vulnerable groups (minorities, migrants, disabled, other disadvantaged persons) Bulgaria: Efficiency Analysis of STI Programs 221 Outputs Value of new Definition: The value of innovation equipment or machinery purchased with the equipment support of the program. The machinery or equipment must be used for product purchased for development or customization of innovative products. innovation projects Disaggregation: S3 thematic priority area; industry Outputs Value of Definition: The value of infrastructure or facility development or upgrades made with infrastructure the support of the program. The facilities must be used for product development or development customization of innovative products. or upgrades for innovation projects Disaggregation: S3 thematic priority area; industry Output Number of Definition: the number of enterprises that received innovation advisory support, enterprises that such as advisory support in acquisition, protection and exploitation of intangible received innovation assets, application of norms and regulations that cover them, product development, advisory support design and testing, market research and analysis, development of marketing plans, preparation of feasibility studies or similar activities related to product innovation specifically related to the activities conducted through the supported project. Disaggregation: Innovation advisory support purpose Bulgaria: Efficiency Analysis of STI Programs 222 Appendix B Researcher Survey Questionnaire A. General Information About the Beneficiary 1. Please mark the option that applies to your case: I am the project leader for the institution 1 I am not the project leader, but can provide detailed information 2 2. What was your role in this project? 3. What is your age? Under 31 years 1 31-40 years 2 41-50 years 3 51-65 years 4 Over 65 years 5 4. Position in your institution at end of the project Professor 1 Associate Professor 2 Chief Assistant Professor 3 Assistant Professor 4 Postdoctoral researcher 5 Doctoral student 6 Other (specify in space below) 7 Bulgaria: Efficiency Analysis of STI Programs 223 5. Have you received funding from public programs for the continuation of this project after its end? (e.g., funds for research, scholarships, etc.) Yes 1 No 2 6. From which program did you get funding for the continuation of this project the project ended? B. About the Application Process 1. Please indicate the extent to which you disagree or agree with the following statements. Neither Strongly agree nor Strongly disagree Disagree disagree Agree agree It was easy to identify and obtain information about the program Program’s objectives were clear Support for helping with the application and clarifying application procedures was timely and relevant Application procedure was well explained Application procedure was easy to follow and fill Application period was sufficient The quantity and type of information required in proposals were adequate Eligibility criteria were clear Eligibility criteria were fair Selection process was fair Selection process was transparent It was easy to access the regulations of the program The regulations of the program were clear Time between application and final results of selection was adequate Time between communication of results of selection and receiving funding was adequate The rules in cases of non-compliance with the call for proposals were flexible Bulgaria: Efficiency Analysis of STI Programs 224 Neither Strongly agree nor Strongly disagree Disagree disagree Agree agree There was feedback on the reasons why the project was approved Contract negotiation procedure took adequate time Contract negotiation was easy The list of eligible costs was appropriate for the development of the project 2. Think about all the expenses incurred in preparing your application, such as the time to pre- pare paperwork, consultants, assistants, materials, etc. Please provide your best estimate of the total cost, in BGN, of preparing your application for this project. For example, if the total expense was one thousand BGN, write 1000. 3. Approximately, how many full person days (8 hours each) did it take you to prepare the application? Number Full person days 4. What kind of human resources did you use to prepare your application? Select all that apply. Administrative assistant 1 Lawyer or legal consultant 2 Accountant 3 Experts/ consultants 4 The project team prepared the application without any additional help 5 Other (please specify in the space below) 6 Bulgaria: Efficiency Analysis of STI Programs 225 5. Were the costs for applying to the program (monetary and non-monetary) adequate when compared with the benefits? Yes 1 No 2 6. Now you will be asked about the eligibility criteria for participating in the program. If you do not remember it, you can check it again. Do you think some eligibility criteria should be add- ed, deleted, or changed? YES NO If yes Changed What eligibility criteria would you change, and why? Added What eligibility criteria would you add, and why? Deleted What eligibility criteria would you delete, and why? 7. Now you will be asked about the selection criteria for participating the program. If you do not remember it, you can check it again. Do you think some selection criteria should be added, deleted, or changed? YES NO If yes Changed What eligibility criteria would you change, and why? Added What eligibility criteria would you add, and why? Deleted What eligibility criteria would you delete, and why? Bulgaria: Efficiency Analysis of STI Programs 226 C. Project Execution and Resources Allocated to the Project Intro: Now you will be asked about project execution and resources allocated to the project. 1. What is the main scientific field of this project? Natural sciences Mathematics 01 Computer and information sciences 02 Physical sciences 03 Chemical sciences 04 Earth and related environmental sciences 05 Biological sciences 06 Other natural sciences 07 Engineering and technology Civil engineering 08 Electrical engineering, electronic engineering, information engineering 09 Mechanical engineering 10 Chemical engineering 11 Materials engineering 12 Medical engineering 13 Environmental engineering 14 Environmental biotechnology 15 Industrial Biotechnology 16 Nanotechnology 17 Other engineering and technologies 18 Medical and health sciences Basic medicine 19 Clinical medicine 20 Health sciences 21 Health biotechnology 22 Other medical sciences 23 Agricultural sciences Agriculture, forestry, and fisheries 24 Animal and dairy science 25 Veterinary science 26 Bulgaria: Efficiency Analysis of STI Programs 227 Agricultural biotechnology 27 Other agricultural sciences 28 Social sciences Psychology 29 Economics and business 30 Educational sciences 31 Sociology 32 Law 33 Political Science 34 Social and economic geography 35 Media and communications 36 Other social sciences 37 Humanities History and archaeology 38 Languages and literature 39 Philosophy, ethics, and religion 40 Art (arts, history of arts, performing arts, music) 41 Other humanities 42 2. Are there other scientific fields related to this project? Yes 1 No 2 3. What are other scientific field related to this project? Natural sciences Mathematics 01 Computer and information sciences 02 Physical sciences 03 Chemical sciences 04 Earth and related environmental sciences 05 Biological sciences 06 Other natural sciences 07 Bulgaria: Efficiency Analysis of STI Programs 228 Engineering and technology Civil engineering 08 Electrical engineering, electronic engineering, information engineering 09 Mechanical engineering 10 Chemical engineering 11 Materials engineering 12 Medical engineering 13 Environmental engineering 14 Environmental biotechnology 15 Industrial Biotechnology 16 Nanotechnology 17 Other engineering and technologies 18 Medical and health sciences Basic medicine 19 Clinical medicine 20 Health sciences 21 Health biotechnology 22 Other medical sciences 23 Agricultural sciences Agriculture, forestry, and fisheries 24 Animal and dairy science 25 Veterinary science 26 Agricultural biotechnology 27 Other agricultural sciences 28 Social sciences Psychology 29 Economics and business 30 Educational sciences 31 Sociology 32 Law 33 Political Science 34 Social and economic geography 35 Media and communications 36 Other social sciences 37 Bulgaria: Efficiency Analysis of STI Programs 229 Humanities History and archaeology 38 Languages and literature 39 Philosophy, ethics, and religion 40 Art (arts, history of arts, performing arts, music) 41 Other humanities 42 4. How did you distribute the funding received? Leave 0 (zero) if not applicable. (Percentages must add up to 100) Percent Machinery, equipment, instrumentation Space, rent incl. labs, research infrastructure Materials, supplies, inventory IT systems, specialized software, IT licenses, websites Researchers’ salaries Other salaries Consulting services (e.g., feasibility studies, survey companies) Training and events Testing and certifications Intellectual property (patents, trademarks, copyrights) Marketing campaigns or public relations (PR) activities for project visibility Travel (fairs, exhibitions, conferences, etc.) Other (please specify in space below) 5. Was the funding you received for this project delivered according to the terms of the contract signed with the program? Yes 1 No (please specify in the space below) 2 Bulgaria: Efficiency Analysis of STI Programs 230 6. Which of the following are the most important objectives of your project? (Select and rank up to 3 objectives. Write 1 for the most important, 2 for the second most important, 3 for the third most important.) Rank Improve chances to get EU funding Publish scientific papers in peer-reviewed journals Present scientific papers in seminars and conferences Produce market-oriented research Develop a cadre of young researchers Collaborate with the private sector Collaborate with other researchers or research institutions Develop a new product, service, or process Upgrade a product, service, or process Develop or start a new enterprise, business, or spin-off Pursue intellectual property (patents, industrial design right, copyrights, etc.) Other (please specify in the space below) 7. Did your institution make in kind or in cash contributions to this project? In cash Yes 1 No 2 I don’t know 3 Does not apply 4 In kind (e.g., researchers, admin support, offices, etc.) Yes 1 No 2 I don’t know 3 Does not apply 4 Bulgaria: Efficiency Analysis of STI Programs 231 8. Please estimate the resources your institution contributed to this project (in BGN). For exam- ple, if your institution contributed with ten thousand BGN, enter 10000. Number a. Cash b. In kind (e.g., researchers, admin support, offices, etc.) 9. Was the amount of financial support provided by the program sufficient to successfully com- plete your project objectives? Yes 1 No 2 10. By what percentage should the financial support have been increased to successfully com- plete your project objectives? 11. What were the most important reasons why the amount of financial support provided by the program was not sufficient? Select and rank up to 3 reasons. Write 1 for the most important, 2 for the second most important, 3 for the third most important. Importance Inadequate budget planning Increased costs of inputs (e.g., survey, materials, lab tests, staff, etc.) Unable to get financial support from my institution Unexpected costs occurred Budget clearing by responsible authority was inadequate Project’s scope increased beyond the original plan Issues with procurement Other (please specify in space below) Bulgaria: Efficiency Analysis of STI Programs 232 Please explain in more detail why the amount of financial support provided by the program was not sufficient 12. Was the amount of time allowed by the program for project implementation, including any extensions, sufficient to successfully complete your project objectives? Yes 1 No 2 13. What were the most important reasons why the amount of time allowed by the program was not sufficient? Select and rank up to 3 reasons. Write 1 for the most important, 2 for the sec- ond most important, 3 for the third most important. Importance I could not reduce my teaching workload I could not reduce other projects’ workload I was engaged in I could not reduce other activities within my Institution (e.g., participation in boards) Other (please specify in space below) 14. Please indicate the extent to which you disagree or agree with the following statements. Neither Strongly agree nor Strongly disagree Disagree disagree Agree agree Administrative support provided by [A1] was easily accessible Administrative support provided by the program was sufficient to help the project advance smoothly Financial support was provided on time Financial reporting requirements were acceptable Monitoring requirements (e.g., narrative, indicators, etc.) were acceptable Expert feedback from monitoring (visits, reports, discussions) under [program name] was appropriate Data protection practices were satisfactory Bulgaria: Efficiency Analysis of STI Programs 233 15. What program support or services did you need during implementation that would have im- proved the outcomes of your project, and were not present? Mark all that apply. Assistance in the preparation of project budgets Assistance to hire foreign researchers Better administrative support Assistance with procurement Access to research infrastructure and equipment Assistance in preparation of monitoring reports Assistance with finding additional funding sources Assistance to establish collaborations Other (please specify in space below) 16. In the future, how would you prefer program support or services to be provided? The institution should provide technical experts 1 Eligible activities of the project should include hiring experts for program support 2 Both of the above 3 Bulgaria: Efficiency Analysis of STI Programs 234 D. Results 1. Indicate the project results that you achieved in the following periods: a. During the project b. After project until today If your project is still active, please enter any results that have been achieved thus far. All spac- es need to be filled with a number. If you did NOT achieve a result, leave 0. During the After project project until today Training activities (courses, workshops, etc.) Number of seminars, workshops and conferences attended domestically Number of seminars, workshops and conferences attended abroad PhD students hired Postdoctoral researchers hired Scientific publications in peer-reviewed journals Patent applications Patents granted Industrial designs Copyrights Transfer agreements Prototype New software development New technology development Other results (please specify in space below) Bulgaria: Efficiency Analysis of STI Programs 235 2. Choose the most important factors that contributed to the achievement of the results. Select and rank up to 3 factors. Write 1 for the most important, 2 for the second most important, 3 for the third most important. Importance Availability of financial resources Availability of human resources such as researchers or mentors Support of the research institution The way support program is designed and implemented Availability of research infrastructure Other (please specify in the space below) 3. How many scientific research papers related to this project have you (or other team mem- bers) published in peer-reviewed journals so far? Select 0 if you have not published any paper related to this project. 4. How many, if any, collaborating partners did/do you have in the context of this project? Select 0 (zero) if you do not have any in a category. Partners are defined as parties with which you have a formal or informal agreement related to the project and who contribute to the project either in cash or in kind. Domestic research partners 1 Foreign research partners 2 Diaspora research partners 3 Domestic industry partners 4 Foreign industry partners 5 Diaspora industry partners 6 There were no collaborating partners 7 Bulgaria: Efficiency Analysis of STI Programs 236 5. Evaluate the overall quality of collaborations related to this project. Extremely poor Below average Average Above average Excellent Domestic research partners Foreign research partners Diaspora research partners Domestic industry partners Foreign industry partners Diaspora industry partners 6. What was the nature of the collaborations related to this project? Mark all that apply. Joint R&D project 1 Purchase of R&D services 2 Technological consultancy 3 Licensing/patent registration 4 Test of a new prototype 5 Preparation of technical documentation 6 Co-author research publication 7 Other (please specify in space below) 8 7. How many, if any, collaborative research projects have you engaged in after the completion of the project until today? Select 0 (zero) if you do not have any in a category. Number Collaborative projects with domestic researchers or research institutions Collaborative projects with foreign researchers or research institutions Collaborative projects with diaspora researchers or research institutions Collaborative projects with domestic enterprises Collaborative projects with foreign enterprises Bulgaria: Efficiency Analysis of STI Programs 237 8. How do you evaluate the outcome of this project based on your expectations? Above my expectations 1 It matched my expectations 2 Below my expectations 3 What is the main reason? 9. List any other public support programs from which you received funding related to science, technology, or innovation after the start of this project. No. Project Name Funding start year 1 2 3 4 10. This is the last question of the survey. You previously indicated that scientific papers related to this project were published in peer-reviewed journals. Please list the publication(s) related to this project. No. Title Name of the journal Year of publication 1 2 3 11. This is the last question of the survey. You previously indicated that scientific papers related to this project were published in peer-reviewed journals. Please list the 5 most important pub- lications related to this project. No. Title Name of the journal Year of publication 1 2 3 Bulgaria: Efficiency Analysis of STI Programs 238 Appendix C Firm Survey Questionnaire A. General Information About the Beneficiary Intro: This section of the survey will ask you general information about you. 1. Please mark the option that applies to your case: I am the project leader 1 I am not the project leader, but can provide detailed information 2 2. What was your role in this project? 3. Have you received funding from public programs for the continuation of this? (e.g., funds for research, scholarships, etc.) Yes 1 No 2 4. From which program did you get funding for the continuation of this project? 5. List any other public support programs (aside from this one) from which you received funding for any project related to science, technology, or innovation between 2015 and 2020 Bulgaria: Efficiency Analysis of STI Programs 239 6. At the end the project, how many full and part time employees did the company have in the company, including you? Write 0 (zero) if there were no full-time or part-time employees Number Full time employees Part time employees 7. Please indicate the highest level of education obtained by the chief executive officer of your company Primary, elementary education or lower 1 Secondary education 2 Industrial/crafts vocational (1-3 years) 3 Technical/vocational (4+ years) 4 Grammar school 5 Bachelor’s or equivalent level 6 Master’s or equivalent level 7 Doctoral or equivalent level 8 I don’t know 9 Bulgaria: Efficiency Analysis of STI Programs 240 B. About the Application and Selection Process 1. Please indicate the extent to which you disagree or agree with the following statements. Neither Strongly agree nor Strongly disagree Disagree disagree Agree agree It was easy to identify and obtain information about the program Program’s objectives were clear Support for helping with the application and clarifying application procedures was timely and relevant Application procedure was well explained Application documentation was easy to follow and fill Application period was sufficient The quantity and type of information required in proposals were adequate Eligibility criteria were clear Eligibility criteria were fair Selection process was fair Selection process was transparent It was easy to access the regulations of the program The regulations of the program were clear Time between application and final results of selection was adequate Time between communication of results of selection and receiving funding was adequate The rules in cases of non-compliance with the call for proposals were flexible There was feedback on the reasons why the project was approved Contract negotiation procedure took adequate time Contract negotiation was easy The list of eligible costs was appropriate for the development of the project Bulgaria: Efficiency Analysis of STI Programs 241 2. Think about all the expenses incurred in preparing your application, such as the time to pre- pare paperwork, consultants, assistants, materials, etc. Please provide your best estimate of the total cost, in BGN, of preparing your application for this project. For example, if the total expense was one thousand BGN, write 1000. 3. Approximately, how many full person days (8 hours each) did it take to prepare the application? Number Full person days 4. What kind of human resources did you use to prepare your application? Select all that apply. Administrative assistant 1 Lawyer or legal consultant 2 Accountant 3 Experts/ consultants 4 The project team prepared the application without any additional help 5 Other (please specify in the space below) 6 5. Were the costs for applying to the program (monetary and non-monetary) adequate when compared with the benefits? Yes 1 No 2 Bulgaria: Efficiency Analysis of STI Programs 242 6. Now you will be asked about the eligibility criteria for participating in the program. Do you think some eligibility criteria should be added, deleted, or changed? YES NO If yes Changed What eligibility criteria would you change, and why? Added What eligibility criteria would you add, and why? Deleted What eligibility criteria would you delete, and why? 7. Now you will be asked about the selection criteria for participating the program. Do you think some selection criteria should be added, deleted, or changed? YES NO If yes Changed What eligibility criteria would you change, and why? Added What eligibility criteria would you add, and why? Deleted What eligibility criteria would you delete, and why? 8. Why did you request financial support from this program? Mark all that apply. Note that fi- nancial institutions include private investors, venture capitalists, banks, and other financial agencies that provide funding. Financial institutions did not give me credit 1 Financial institutions gave me credit, but it is very expensive or takes too long to get the funding (skip to E1) 2 Obtaining a grant from public sources was the only funding option (skip to E1) 3 We needed mentoring, information, or technical advice for upgrading or learning new technologies (skip to E1) 4 Other (specify in the space below) (skip to E1) 5 Bulgaria: Efficiency Analysis of STI Programs 243 9. Please mark the reason(s) why financial institutions did not give you credit. Please mark all that apply. I do not have a collateral 1 I do not have a long credit history 2 The project is risky and returns are uncertain 3 Other (please specify below) 4 C. Project Execution and Resources Allocated to the Project Intro: Now you will be asked about project execution and resources allocated to the project. 1. What is the main area of economic activity related to this project? Economic area NACE code Agriculture, Forestry and Fishing A Mining and Quarrying B Manufacturing C Electricity, Gas, Steam and Air Conditioning Supply D Water Supply; Sewerage, Waste Management and Remediation Activities E Construction F Wholesale and Retail Trade; Repair of Motor Vehicles and Motorcycles G Transportation and Storage H Accommodation and Food Service Activities I Information and Communication J Financial and Insurance Activities K Real Estate Activities L Professional, Scientific and Technical Activities M Administrative and Support Service Activities N Public Administration and Defense; Compulsory Social Security O Education P Human Health and Social Work Activities Q Other (Please specify in the next page) Bulgaria: Efficiency Analysis of STI Programs 244 2. What is the main Smart Specialization (S3) priority area for this project? Informatics and ICT 1 Healthy life and biotechnology industries 2 Mechatronics and clean technologies 3 New technologies in creative and re-creative industries 4 None of these 5 3. How did you distribute the funding received? Leave 0 (zero) if not applicable. Percent Machinery, equipment, instrumentation Space, rent Access to specialized equipment or laboratories Materials, supplies, inventory IT systems, specialized software, IT licenses, websites Salaries Consulting services (e.g., feasibility studies, survey companies, market research) Training Testing and certifications Intellectual property (patents, trademarks, copyrights) Marketing campaigns or public relations (PR) activities for project visibility Travel and events (fairs, exhibitions, conferences, etc.) Other (please specify in space below) 4. Was the funding you received for this project delivered according to the terms of the contract signed with the program? For example, were the amounts received, disbursement times and conditions, etc., provided according to the terms of contract? Yes 1 No (please specify in the space below) 2 Bulgaria: Efficiency Analysis of STI Programs 245 5. Which of the following are the most important objectives of your project? (Select and rank up to 3 objectives. Write 1 for the most important, 2 for the second most important, 3 for the third most important.) Objective Number Related to collaboration Collaborate with other private sector entities Collaborate with researchers or research institutions Related to capacity building Participate in international fairs Training of employees Develop innovation capacities and management skills Related to new products and spinoffs Pursue intellectual property (patents, industrial design right, copyrights, etc.) Develop a new product, service, or process Putting a new product, service, or process on the market Upgrade a product, service, or process Develop or start a new enterprise, business, or spin-off Other (please specify in the space below) 6. Did your company make in kind or in cash contributions to this project? In cash Yes 1 No 2 I don’t know 3 Does not apply 4 In kind (e.g., researchers, admin support, offices, etc.) Yes 1 No 2 I don’t know 3 Does not apply 4 Bulgaria: Efficiency Analysis of STI Programs 246 7. Please estimate the resources your institution contributed to this project (in BGN). For exam- ple, if your institution contributed with ten thousand BGN, enter 10000. Number a. Cash b. In kind (e.g., researchers, admin support, offices, etc.) 8. Was the amount of financial support provided by the program sufficient to successfully com- plete your project objectives? Yes 1 No 2 9. By what percentage should the financial support have been increased to successfully com- plete your project objectives? 10. What were the most important reasons why the amount of financial support provided by the program was not sufficient? Select and rank up to 3 reasons. Write 1 for the most important, 2 for the second most important, 3 for the third most important. Importance Inadequate budget planning Increased costs of inputs Unable to get financial support from my institution Unable to ensure cofinancing Unexpected costs occurred Budget approved by the public entity was inadequate Project’s scope increased beyond the original plan Issues with procurement Other (please specify in space below) Bulgaria: Efficiency Analysis of STI Programs 247 Please explain in more detail why the amount of financial support provided by the program was not sufficient 11. Was the amount of time allowed by the program for project implementation, including any extensions, sufficient to successfully complete your project objectives? Yes 1 No 2 12. What were the most important reasons why the amount of time allowed by the program was not sufficient? Select and rank up to 3 reasons. Write 1 for the most important, 2 for the sec- ond most important, 3 for the third most important. Reason Importance Did not have enough financial resources Could not find qualified staff Delays in supply of raw materials Delays in production process Inadequate planning Other (please specify in space below) Bulgaria: Efficiency Analysis of STI Programs 248 13. Please indicate the extent to which you disagree or agree with the following statements. Strongly Neither agree Strongly disagree Disagree nor disagree Agree agree Administrative support provided by the public entity was easily accessible Administrative support provided by the program was sufficient to help the project advance smoothly Financial support was provided on time Financial reporting requirements were acceptable Monitoring requirements (e.g., narrative, indicators, etc.) were acceptable Expert feedback from monitoring (visits, reports, discussions) under the program was appropriate Data protection practices were satisfactory 14. What program support or services did you need during implementation that would have im- proved the outcomes of your project, and were not present? Mark all that apply. Guidance for intellectual property rights management 1 Guidance for certification and standard norms 2 Guidance for market penetration and/or commercialization 3 Establishing connections with international experts 4 Assistance in the preparation of project budgets 5 Better administrative support by the program 6 Assistance with procurement 7 Access to research infrastructure and equipment 8 Assistance in preparation of monitoring reports 9 Assistance with finding additional funding sources 10 Assistance to establish collaborations 11 Other (please specify in space below) 12 Bulgaria: Efficiency Analysis of STI Programs 249 D. Results 1. Indicate the project results that you achieved in the following periods: a. During the project b. After project until today All spaces need to be filled with a number. If you did NOT achieve a result, leave 0. During the After project project until today Market-oriented research projects Defined an intellectual property right strategy Patent /Utility models applications Patents /Utility models granted Other intellectual property activities (e.g., industrial designs, copyrights, transfer agreements, etc.) New enterprise, business, or spin-off Prototype Products or services that are new to the firm Products or services that are new to the market New processes Upgraded products or services Upgraded processes New design for a product, process, or service New business model Expanded to new markets Develop a new innovation unit in the firm Improved the capabilities of employees Reorganized the firm or part of the firm Other results (please specify in space below) Bulgaria: Efficiency Analysis of STI Programs 250 2. If the project led to the adoption of a new process or a new technology, were you be able to implement this new process or technology in your firm? Yes 1 No 2 3. Which of the following stages of commercialization have you attained for results related to this project? Mark all that apply. Have a product, service or process that is being sold 1 Have a product, service or process that is ready to be sold 2 Have a working prototype that is almost ready to be sold 3 Have a proof of concept for a product or process that can be sold in the future 4 Other (please specify in the space below) 5 The results that came out of this project will NOT be commercialized or are NOT likely to be commercialized 6 4. Select up to three of the most difficult challenges you company faced in commercialization of the project. Write 1 for the most important, 2 for the second most important, 3 for the third most important. The project did not provide enough support for commercialization Lack of time Lack of financial resources Lack of human resources Lack of information about markets Lack of companies interested Legal complexity/ambiguity concerning commercialization Competition The project is not ready to be commercialized Other (please specify in space below) Bulgaria: Efficiency Analysis of STI Programs 251 5. Choose the most important factors that contributed to the achievement of the results. Se- lect and rank up to 3 factors. Write 1 for the most important, 2 for the second most important, 3 for the third most important Availability of financial resources Availability of human resources such as researchers or mentors Support of the research institution The way support program is designed and implemented Availability of research infrastructure Other (please specify in the space below) 6. Have you taken any of the following steps towards selling the product or process coming out of the project? Mark all that apply Discussions/negotiations with a vendor or firm that will sell my product, service, or process Participated in trade fairs to showcase my product, service, or process Participated as advertiser in scientific/technical conferences Presented my product or process in domestic market Other (please specify below) 7. What percentage of the company’s sales do you expect from the commercialization of the main result of the project? Expected percentage of sales from commercializing the main results of the project I don’t know 8. Was this project effective in… Yes No Don’t Know Improving the sales of the company? 1 2 3 Improving the productivity of the company? 1 2 3 Reducing the production costs of the company? 1 2 3 Improving the export performance of the company? 1 2 3 Allowing the company to access new markets? 1 2 3 Bulgaria: Efficiency Analysis of STI Programs 252 9. Please indicate the following effects of the project, approximately: If E9a=YES, by how much (in percentage) did the sales increase? If E9b=YES, by how much (in percentage) did productivity improve? If E9c= YES, by how much (in percentage) did production costs decrease? If E9d= YES, by how much (in percentage) did export performance improve? If E9e= YES, by how much (in percentage) did export performance improve? 10. Approximately when do you expect to recover the investment your company made for this project? 11. Has this project contributed to increase the numbers of employees in your company? Yes 1 No 2 I don’t know 3 12. How many additional workers were hired due to the project? Mark 0 (zero) if no additional full or part-time workers were hired. Full time workers Part time workers Bulgaria: Efficiency Analysis of STI Programs 253 13. How many, if any, collaborating partners did/do you have in the context of this project? Select 0 (zero) if you do not have any in a category. Partners are defined as parties with which you have a formal or informal agreement related to the project and who contribute to the project either in cash or in kind. Number Domestic research partners Domestic industry partners Diaspora research partners Diaspora industry partners Foreign research partners Foreign industry partners 14. Evaluate the overall quality of collaborations related to this project. Extremely poor Below average Average Above average Excellent Domestic research partners Foreign research partners Diaspora research partners Domestic industry partners Foreign industry partners Diaspora industry partners 15. What was the nature of the collaborations related to this project? Mark all that apply. Joint R&D project 1 Purchase of R&D services 2 Technological consultancy 3 Licensing/patent registration 4 Test of a new prototype 5 Preparation of technical documentation 6 Co-author research publication 7 Selling a product 8 Other (please specify in space below) 9 Bulgaria: Efficiency Analysis of STI Programs 254 16. How many, if any, collaborative research projects have you engaged in after the completion of [A5] until today? Select 0 (zero) if you do not have any in a category. Number Collaborative projects with domestic researchers or research institutions Collaborative projects with foreign researchers or research institutions Collaborative projects with diaspora researchers or research institutions Collaborative projects with domestic enterprises Collaborative projects with foreign enterprises 17. How do you evaluate the outcome of this project based on your expectations? Above my expectations It matched my expectations Below my expectations What is the main reason? Bulgaria: Efficiency Analysis of STI Programs 255 Appendix D Administrative Costs This appendix describes the administrative cost data provided by the program staff of the select- ed programs, any estimations or calculations made by program staff, and any adjustments made to the administrative cost data by the World Bank team. D.1 Fundamental Research Personnel Costs Personnel costs were calculated based on the salaries of program staff and the number of days per year they estimated working on the program. Fixed Costs The NSF does not have detail information on the fixed costs that can be attributed to individual programs. Therefore, fixed costs were estimated based on the personnel costs of the Fundamen- tal Research program as a share of total NSF personnel costs, using the following formula: Fixed costs FR = Fixed costs NSF × ( Personnel costs FR Personnel costs NSF ) However, data for the total personnel costs for NSF were only available for 2019 and 2020, so an additional estimation was done to estimate total personnel costs from 2016-2018. The World Bank team assumed that personnel costs at NSF grew at the same rate as personnel costs at MoES (NSF’s parent ministry), resulting in an annual growth rate of personnel costs of approximately 10 percent from 2016 to 2018. External Services Costs for external evaluators of project proposals were estimated based on an estimated cost of 1,440 BGN per application multiplied by the number of applications received by the program per year. Costs for external services for M&E, advisory (costs related to members of NSF’s temporary and permanent advisory committees), and other (such as legal services) were estimated based on the total cost to NSF for each of these services multiplied by the personnel costs of the Fun- damental Research program as a share of total NSF costs. Bulgaria: Efficiency Analysis of STI Programs 256 Cost data for external experts for M&E were not available for 2017, so the overall costs for exter- nal services presented in this report are likely lower than the actual costs of external services for the program. D.2 Vihren Personnel Costs Personnel costs were calculated based on the salaries of program staff and the number of days per year they estimated working on the program. Fixed Costs The NSF does not have detail information on the fixed costs that can be attributed to individual programs. Therefore, fixed costs were estimated based on the personnel costs of the Vihren pro- gram as a share of total NSF personnel costs, using the following formula: Fixed costs SV = Fixed costs NSF × ( Personnel costs SV Personnel costs NSF ) External Services Costs for external evaluators of project proposals were estimated based on an estimated cost of 1,440 BGN per application multiplied by the number of applications received by the program per year. Costs for external services for M&E, advisory (costs related to members of NSF’s temporary and permanent advisory committees), and other (such as legal services) were estimated based on the total cost to NSF for each of these services multiplied by the personnel costs of the Fun- damental Research program as a share of total NSF costs. D.3 ICT for a Single Digital Market in Science, Education, and Security and Electronic Health in Bulgaria Personnel Costs Personnel costs were calculated based on the salaries of program staff and the number of days per year they estimated working on the program. Fixed Costs Fixed costs were estimated based on the personnel costs of the NIF program as a share of total SMEPA personnel costs, using the following formula: Bulgaria: Efficiency Analysis of STI Programs 257 Fixed costs ICT = Fixed costs MoES × ( Personnel costs ICT Personnel costs MoES ) Cost data was not available to office equipment and ICT, so these costs were not included in this analysis; therefore, the actual fixed costs attributable to the program are likely higher than what is presented in this report External Services There were no external services used by the program D.4 Electronic Health in Bulgaria Personnel Costs Personnel costs were calculated based on the salaries of program staff and the number of days per year they estimated working on the program. Fixed Costs Fixed costs were estimated based on the personnel costs of the NIF program as a share of total SMEPA personnel costs, using the following formula: Fixed costs eHealth = Fixed costs MoES × ( Personnel costs eHealth Personnel costs MoES ) Cost data was not available to office equipment and ICT, so these costs were not included in this analysis; therefore, the actual fixed costs attributable to the program are likely higher than what is presented in this report. External Services There were no external services used by the program Bulgaria: Efficiency Analysis of STI Programs 258 D.5 Development of Product and Process Innovations Personnel Costs Personnel costs were calculated based on the salaries of program staff and the number of days per year they estimated working on the program. Fixed Costs Costs for office space, utilities, and maintenance were estimated using a weight based on num- ber of supported contracts under the DPPI program as a share of all contracts supported under DG OPIC. Cost data was not available to office equipment and ICT, so these costs were not included in this analysis; therefore, the actual fixed costs attributable to the program are likely higher than what is presented in this report. External Services External services costs were calculated from the external contracts funded by the program D.6 National Innovation Fund Personnel Costs Personnel costs were calculated based on the salaries of program staff and the number of days per year they estimated working on the program. Fixed Costs Fixed costs were estimated based on the personnel costs of the NIF program as a share of total SMEPA personnel costs, using the following formula: Fixed costs NIF = Fixed costs SMEPA × ( Personnel costs NIF Personnel costs SMEPA ) External Services External services costs were calculated from the external contracts funded by the program