WORKING PAPER #12 RESULTS-BASED FINANCING IN EDUCATION FOR SUB-NATIONAL GOVERNMENT AND SCHOOL ADMINISTRATORS A Conceptual Framework and Practical Recommendations Arushi Terway Nicholas Burnett Marina Dreux Frotté NORRAG is an Associate Programme of the Graduate Institute of International and Development Studies (IHEID) ABOUT NORRAG NORRAG is a global membership-based network of international policies and cooperation in education and training. In 1977, the Research Review and Advisory Group (RRAG) was established, which then founded several regional RRAGs, one of which became NORRAG in 1986. NORRAG’s core mandate and strength is to produce, disseminate and broker critical 20, Rue Rothschild l P.O. Box 1672 knowledge and to build capacity for and with academia, governments, NGOs, international organizations, foundations and the private sector 1211 Geneva 1, Switzerland who inform and shape education policies and practice, at national and +41 (0) 22 908 45 47 international levels. By doing so, NORRAG contributes to creating the norrag@graduateinstitute.ch conditions for more participatory, evidence-informed decisions that improve equal access to and quality of education and training. NORRAG is an associate programme of the Graduate Institute of International and Development Studies, Geneva. More information NORRAG is supported by about NORRAG, including its scope of work and thematic areas, is available at www.norrag.org. @twitter.com/norrag @facebook.com/norrag.network Federal Department of Foreign Affairs FDFA Swiss Agency for Development and Cooperation SDC @vimeo.com/norrag Education Unit ABOUT REACH Results in Education for All Children (REACH) seeks to help countries strengthen their education services by focusing programmes and initiatives on results, with the ultimate goal of boosting learning outcomes, especially among the most vulnerable populations. Established in 2015, the World Bank REACH programme funds results- based financing projects and provides technical support and advice on results-based financing in education to other World Bank teams and development partners. One of its main goals is to contribute to the NORRAG Working Paper #12, December 2021 evidence base around results-based financing in education, gathering Published under the terms and conditions of the Creative data and knowledge that can be used to develop better education Commons licence: Attribution-NonCommercial 4.0 programmes and policies, with the goal of boosting education outcomes. International (CC BY-NC 4.0) REACH is funded by the Government of Norway through NORAD, the Government of the United States of America through USAID and the Government of Germany through the Federal Ministry for Economic Cooperation and Development. More information: www.worldbank.org/en/programs/reach ACKNOWLEDGEMENTS ABOUT THE PROJECT This report has been commissioned to NORRAG by the World Bank’s NORRAG’s work in the area of Innovative Financing for Education (IFE) Results in Education for All Children (REACH) programme. It is authored aims to develop capacity and deepen collective knowledge about the by Arushi Terway, Nicholas Burnett and Marina Dreux Frotté. Tomas mounting numbers and modalities of innovative financing mechanisms Esper, Anaka Harish Ganesh and Georgia Ellen Thorne provided valuable in the education sector. By conducting research on diverse IF assistance in the data collection and analysis. experiences in education, NORRAG is building empirical evidence on the The team would like to thank all interviewees for generously sharing with additionality and efficiency of IFE and its impacts on provision, quality us their experiences in designing and implementing RBF initiatives. The and equity in education. team would also like to thank Saamira Halabi, Jessica Lee, Gita Steiner- More information: www.norrag.org/ife Khamsi and Moira Faul for their insightful feedback and comments on the research design and drafts of the report. The team is also grateful for the feedback on the report draft from Andre Loureiro, Renata Freitas ABOUT THE AUTHORS Lemos, Kai Kaiser, Shahram Paksima and Shinsaku Nomura. The report Dr Arushi Terway is the Senior Lead Research Associate at NORRAG layout was designed by Aspire Design. Financial support for this work directing the portfolio on innovative finance in education and private was provided by the REACH trust fund. sector engagement. Dr Nicholas Burnett is a Senior Fellow at Results for Development (R4D), a Senior Research Fellow at 3ie, and a Visiting Professor of International Education Policy at the Universities of Nottingham (UK) and Kobe (Japan). Ms Marina Dreux Frotté is Programme Coordinator for NORRAG’s portfolio on innovative financing in education and private sector engagement, as well as Monitoring, Evaluation and Learning (MEL) Lead for the Global Partnership for Education (GPE) Knowledge, Innovation and Exchange (KIX) Europe, Asia and Pacific (EAP) Hub. RESULTS-BASED FINANCING IN EDUCATION FOR SUB-NATIONAL GOVERNMENT AND SCHOOL ADMINISTRATORS A Conceptual Framework and Practical Recommendations Arushi Terway Nicholas Burnett Marina Dreux Frotté III IV Contents Executive Summary 1 1 Introduction 5 2 Principal-Agent Theory in Results-Based Financing at the Meso-Level of 9 Education Systems 3 Research Methodology 17 4 Overall Trends in the Literature and Initiatives 19 5 Evaluation Results of Initiatives 23 6 Observed Divergence from Theory in Practice 27 7 A Differentiated Approach Linking Financing to Results 39 8 Recommendations for Designing RBF at the Meso-Level of Education Systems 49 9 Looking Ahead 59 References 60 Appendices 63 Appendix 1: Description of RBF at the Meso-Level in Literature 63 Appendix 2: Profile of RBF Initiatives at the Meso-Level of Education System Initiatives 65 Appendix 3: Meso-Level RBF Interventions in Education Systems 68 Appendix 4: Findings on Impacts of RBF on Education at the Meso-Level 87 Appendix 5: Keyword Search Terms 92 Appendix 6: List of Interviewees 93 V List of Tables, Figures and Boxes Table 1. Types of RBF in education 10 Table 2. Types of result indicators linked to RBF payments used in RBF at the meso-level of education 21 Table 3. Types of RBF arrangements 41 Table 4. Number of RBF interventions at the meso-level of the education system per funder in low- and middle-income 66 countries Table 5. Key features of meso-level RBF interventions in education systems 68 Figure 1. Distribution of RBF at the meso-level of education publications by school level (left) and by countries’ income 19 levels (right) Figure 2. Distribution of RBF initiatives at the meso-level of the education sector in low- and middle-income 21 countries by type Figure 3. Principal-Agent relationships within an education system 28 Figure 4. Number of publications with a focus on meso-level RBF per sector across time (2010–2021) 63 Figure 5. Number of publications focusing on RBF at the meso-level by countries’ income levels (2010–2021) 64 Figure 6. Geographical distribution of RBF initiatives at the meso-level of the education system in low- and middle-income 65 countries Figure 7. Distribution of RBF initiatives at the meso-level of the education sector in low- and middle-income countries by 66 educational level Figure 8. Distribution of principals in RBF initiatives at the meso-level of the education sector in low- 67 and middle-income countries Figure 9. Distribution of agents in RBF initiatives at the meso-level of the education sector in low- and middle-income 67 countries Box 1: Cameroon Education Reform Support Project 30 Box 2: Proyectá Tu Futuro: A Social Impact Bond in Buenos Aires, Argentina 34 Box 3: Benin Global Partnership for Education Project Phase 3 35 Box 4: Odhisa Higher Education Programme for Excellence and Equity 43 Box 5: RBF in Ceará, Brazil 44 Box 6: Sindh Secondary Education Improvement Project 46 Box 7: India Education Outcomes Fund: Rate Card for Education Outcomes 47 Box 8: Creation of competition between Agents may exacerbate equity challenges 52 VI Abbreviations ADB Asian Development Bank CIDA Canadian International Development Agency DFAT Department of Foreign Affairs and Trade (Australia) ETEP Enhancing Teacher Education Program FCDO Foreign, Commonwealth & Development Office (United Kingdom) GPE Global Partnership for Education GPRBA Global Partnership for Results-Based Approaches IEOF India Education Outcomes Fund NGO Non-Governmental Organisation OECD Organisation for Economic Co-Operation and Development PBB Performance-Based Budgeting PISA Programme for International Student Assessment PSM Public Service Motivation PTA Parent-Teacher Association RBA Results-Based Aid RBF Results-Based Financing RCT Randomised Control Trial REACH Results in Education for All Children (World Bank) SDG Sustainable Development Goal SMC School Management Committee SUBEBs States Universal Basic Education Boards (Nigeria) TEQIP Technical Education Quality Improvement Project (India) TVET Technical and Vocational Education and Training USAID United States Agency for International Development VII RBF IS A FINANCING MODALITY IN WHICH PAYMENTS ARE CONTINGENT ON THE ACHIEVEMENT OF PREDEFINED AND SUBSEQUENTLY VERIFIED RESULTS. VIII Executive Summary R esults-based financing (RBF) arrangements have gained popularity due to their expected potential in positively impacting education service delivery by improving the effectiveness, efficiency and accountability of funding. RBF is a financing modality in which payments are contingent on the achievement of predefined and subsequently verified results. This represents a fundamental change in development financing, comprising a move away from providing funds for inputs and activities and towards tying payments to the achievement of measurable results. Previous reviews and research have focused on the use of RBF to incentivise actors at three different levels: 1) national governments to design and implement certain policy choices; 2) frontline workers, such as teachers, to improve teaching practices in the classroom; and 3) beneficiaries to avail education services. There is a noticeable gap in education management research, specifically in examining how RBF could motivate middle-management or meso-level actors to improve the education service delivery process. This study explains the Principal-Agent theoretical framework in RBF interventions for meso-level actors and applies it to a sample of interventions in low- and middle-income countries. We define meso-level as actors and/or institutions in charge of education system management and administration at the local government (e.g. province, region, district) or school level, such as district education officers, school management committees and school directors. Interventions that engage non-government service providers as meso-level actors are also included since they play a similar role to local government actors in operationalising the education policy for service delivery to beneficiaries. Situated in the middle of the This study used qualitative data education value chain (the range of activities from start to finish to analysis to draw both theoretical deliver services), meso-level actors play a crucial role in translating, transferring, brokering – and, therefore, in potentially disrupting and practical lessons from and hindering – the gains sought by actors at other levels to ensure diverse experiences with RBF at quality education service delivery. Thus, RBF directed at the meso- level has the potential for cascading (rather than disconnecting) the the meso-level of the education effects of incentives at other levels of the education system value sector, particularly in low- and chain. middle-income countries. The Principal-Agent theory explains the relationship between two actors: the Principal, who wants a task completed, and the Agent, who is delegated to complete the task on behalf of the Principal. However, Principals and Agents often do not share the same priorities. The Principal-Agent theory attempts to overcome this disparity by designing an effective contract with an incentive structure that motivates the Agent to achieve the optimum output or 1 outcome that the Principal desires. In an RBF arrangement, Principal and Agent priorities are aligned through payments being made to the Agent only when the desired results are achieved. Since the Agent does not get paid if the results are not achieved, the assumption is that RBF provides greater motivation for the Agent to perform according to the Principal’s objectives. In this case, the Agent bears the risk of failure of the programme but has greater autonomy over how to deliver the results. The Principal, in this case, forgoes some level of control over how the results are achieved but gains value for money by only paying for results achieved. To date, however, theoretical RBF literature has not conceptualised this relationship between national policy makers and agencies at lower levels and its potential outcomes (Rendmans, Paul, and Dujardin 2016). This study used qualitative data analysis to draw both theoretical and practical lessons from diverse experiences with RBF at the meso-level of the education sector, particularly in low- and middle-income countries. We drew upon the experiences of those who have designed, implemented and/or researched such RBF initiatives. The review included four different sources of data: 1) academic literature; 2) project documents; 3) a global survey; and 4) key stakeholder interviews. An analysis of published literature revealed that research in the education sector is growing but the field is still dominated by research in the health sector. Majority of the published research in education have been on higher education in the United States. Meso-level RBF initiatives in low- and middle-income countries have been implemented in all regions worldwide, with most of them directed at primary school education. A large proportion of these initiatives were designed to motivate school leaders to improve management processes, school access, student participation and learning outcomes. Moreover, the limited number of published evaluations on RBF at the meso-level in low- and middle-income countries reveal mixed results. Thus, it is difficult to generalise the learnings from these studies to other contexts. The present study establishes four significant differences between the theoretical model of RBF and the actual experience at the meso-level. These are the following: 1 Multiple Principal-Agent relationships Even when the RBF contract was between one Principal and one Agent, as theorised in the Principal-Agency framework, the examined initiatives accommodated the presence of multiple Principals for the meso-level Agent. Moreover, while the theoretical framework of RBF primarily explains the Principal-Agent relationships in terms of firms and employees, at the meso- level of education systems, the relationship manifests in a layered relationship that involves the Principal funder-Agent organisation-employee. In addition, the initial design of most of the RBF experiences studied did not sufficiently consider applying RBF in a cascaded way, down from the Principal to the Agent organisation and then, in turn, down to the Agent’s employees. 2 Need for operational funds to achieve results The disbursement of funds after achieving results is a key element of RBF arrangements. However, in a large majority of the initiatives examined, at least a portion, if not the entire amount, was disbursed to the Agent upon signing the agreement or upon the implementation of specific management processes in an effort to achieve results. This reflects the budgeting reality of meso-level agencies that are highly reliant on regular budget appropriations for operational funds. 2 3 Difficulty in defining results In general, defining results in education is a complex process. The demarcations between outputs and outcomes are complex, in part because ultimate outcomes take a very long time to be achieved in education and because not all countries have robust systems for measuring these eventual outcomes, such as learning. Attributing results to the actions of a single meso-level Agent is even more complicated. Most of the meso-level RBF initiatives analysed did not use either outputs or outcomes to define results. Instead, they tended to focus on processes (the overall management process, financial management, data management, teaching process, teacher support, etc.) and defined the indicators for these processes. Thus, in most cases, the Principals incentivised effort and specific ways to achieve results rather than defining the results and then paying for their achievement. Most of the ‘results’ adopted were themselves the product of negotiations between the Principal and the Agent and may well indicate the challenge faced by meso-level Agents in terms of their capacity to influence some educational results, such as learning outcomes. On occasion, they may also reflect the difficulty (and expense) of measuring ultimate outcomes. 4 Lack of Agent autonomy A basic mechanism of RBF is to move the financial risk of achieving results onto the Agents while providing them with greater autonomy over tasks to achieve the results. However, little evidence of increased Agent autonomy is present in the RBF initiatives. There was, however, some indication that when meso-level Agents have little appetite or ability to take on additional financial risks or additional workloads, they may be willing to forgo a greater level of autonomy in exchange for funding reliability. The divergence from the RBF theoretical framework at the meso-level of education draws attention to the need to broaden the definition of RBF so that it responds to contextual realities while preserving the purpose of results-based management in education service delivery. We propose drawing lessons from the experience of Performance-Based Budgeting reforms that seek to link funding to results, based on performance information, for the purpose of increasing the efficiency and effectiveness of government spending. Specifically, we propose differentiating various types of RBF designs that meet the pre-existing conditions in a given context without compromising the potential benefits of RBF. The following three differentiated types of Performance-Based Budgeting can be adapted to RBF for the meso-level of education service delivery: 1 2 3 Funding-linked Agency-level budget Purchaser-provider model performance targets performance The Agent is treated like a incentives Links the level of funding ‘standalone business’ that is to a specific, measurable Introduces paid a price per result achieved, results target (output or performance incentives with the Agent making a profit outcome) that the Agent (rewards or sanctions) or loss reflecting the difference is expected to deliver to motivate the between what it cost the Agent without conditioning Agent to improve to achieve the results and disbursement upon the performance or achieve what the Principal paid for the achievement of results. results. results. 3 These three differentiated types of RBF implementation cover the spectrum of most initiatives analysed in this study, where early-stage interventions start with linking funding to results and more advanced-stage RBFs make the entire intervention funding contingent on the verification of achieved results. The type of RBF that can be designed and implemented is contextual and should be based on: • Whether the identified problem in the given context can be solved with modifications in financing arrangements; • The level of data availability and utilisation in education planning and implementation decision making; and • The level of control and autonomy meso-level service delivery Agents have over decisions on activities that yield results. This study identified a need for robust evaluations of the implementation of RBF arrangements at the meso-level of education with a focus on examining the causal mechanism by which the financial arrangement leads to better results. However, our examination of the design and implementation of interventions at the meso- level in low- and middle-income countries and some early learnings about their implementation lead to some lessons on what could work under which conditions. Based on these learnings, we postulate six key recommendations to capture the elements for consideration when designing RBF interventions that motivate meso- level actors to improve education service delivery. 1 Conduct a situation analysis to understand any existing experience with RBF and determine whether there is a culture conducive to results- Recommendations based management and if there are educational challenges that RBF can help solve. 2 Agree on a shared theory of change with all stakeholders, especially Principals and Agents, that will also ensure sufficient autonomy for Agents to achieve results. 3 Define results targets that can be transparently and objectively measured. 4 Design an incentive structure that cascades financial and non-financial incentives at both organisational and individual levels. 5 Provide capacity building support and technical assistance to foster a culture of results orientation. 6 Ensure sustainability with a plan for moving through the differentiated levels and phasing out external funding over time. 4 1 Introduction R esults-based financing (RBF) has gained prominence within the domain of international development funding and, more recently, within funding for education, as an approach to improve the effectiveness, efficiency and accountability of development spending. A recent review of development projects identified that between 1993 and 2017, at least USD25 billion in development spending was contingent on results in more than 300 RBF programmes. Notably, the health sector has the highest number of projects utilising RBF, with education and public administration coming in at second and third (Instiglio 2018). RBF approaches are expected to have the potential to engender positive effects on education systems by incentivising specific stakeholders to achieve intended results (Lee and Medina 2019). RBF is a financing arrangement wherein payments are contingent on the achievement of predefined and subsequently verified results (Instiglio 2018). This represents a fundamental change in development financing, constituting a move away from providing funds for inputs and activities and towards tying payments to the achievement of measurable results – which could be outputs or programme outcomes. Experimentation with RBF within education has occurred in response to: 1) the recognition of a shortfall in funding from both international aid and national budget allocation; 2) limited progress towards SDG4 (UNESCO 2015, 3) the perception that traditional financing is inefficient, ineffective and prone to waste and corruption (UNESCO 2018; Birdsall and Savedoff 2010; R4D 2016); and 4) a growing focus on increasing accountability towards achieving measurable results in terms of educational outcomes (R4D 2016; Education Commission 2016). RBF interventions and RBF research within education in low- and The roles of sub-national or middle-income countries have been primarily focused on incentivising meso-level actors in education the following actors: were perceived to be significant • Country governments (finance ministry or education line ministry) • Frontline workers (teachers and school directors) by the survey respondents of • Beneficiaries (students and households) the REACH research, with 84% Typically, national governments are incentivised by international considering them the most aid agencies through Results-Based Aid (RBA), frontline workers important to incentivise. are incentivised through some types of performance bonuses and beneficiaries are given conditional cash transfers. Empirical research on RBF interventions for these actors has demonstrated mixed results, especially in terms of the impact on learning outcomes (Lee and Medina 2019; R4D 2016). The majority of available research focuses on the impact of incentives on frontline workers or on beneficiaries because of the complexity of isolating the 5 impact of aid to national governments on the entire education system. Previous reviews have acknowledged that RBF arrangements in any form are made within complex systems where the context of the entire education system influences the impacts of the tasks undertaken by the incentivised actor. Guidance from Global Partnership for Results-Based Approaches (GPRBA) (Instiglio 2018) and REACH (Lee and Medina 2019) further elaborates that the introduction of RBF interventions at one level could initiate a chain of RBF interventions (a ‘cascade’) at other levels to align actors at the different levels towards the same defined results. However, to date, there is no comprehensive review designed to understand the impact of the chain of incentivisation of actors at different levels on education outcomes. Previous research on education systems in low- and middle-income countries has specifically identified that, in many instances, national governments have initiated RBF to incentivise sub-national actors (like district/provincial offices) (R4D 2016; Lee and Medina 2019; GPE 2020). The roles of sub-national or meso-level actors in education were perceived to be significant by the survey respondents of the REACH research, with 84% considering them the most important to incentivise. Situated in the middle of the education value chain (the range of activities from start to finish to deliver services), meso-level actors play a crucial role in translating, transferring, brokering – and, therefore, in potentially disrupting and hindering – the gains sought by actors at other levels to ensure quality education service delivery. In most countries, sub-national governments are responsible for education service provision and account for over 50% of total public education spending (Al-Samarrai and Lewis 2021). Thus, RBF directed at the meso-level has the potential for cascading (rather than disconnecting) the effects of incentives at other levels of the education system value chain (Lee and Medina 2019). National policymakers have been utilising agencies at lower levels of governments We define mesolevel to drive progress; however, theoretical RBF literature to date does not conceptualise this relationship and its potential outcomes (Rendmans, Paul, and Dujardin 2016). as actors and/or Current theoretical and empirical research leaves a gap in explaining the unique institutions who are characteristics – linkage between levels – of the roles that sub-national actors play in education service delivery. This report aims to fill this research gap through an in charge of education examination of the motivation and incentives for sub-national government and system management other actors, referred to as meso-level actors, as potential brokers or disruptors of the causal chain underpinning the logic of RBF in improving the effectiveness, and administration efficiency and accountability in education financing. in sub-national There is considerable heterogeneity in the types of actors and their specific roles at governments (e.g. the sub-national level, ranging from state education ministers to school directors. For the purpose of this study, we examine some common attributes for the purpose provincial, regional, of RBF management and group them as meso-level actors. We define meso-level as district) or at the school actors and/or institutions who are in charge of education system management and administration in sub-national governments (e.g. provincial, regional, district) or at level, such as district the school level, such as district education officers, school management committees education officers, and school directors. Interventions that engage non-government service providers as meso-level actors are also included, as they play a similar role to sub-national school management government actors in operationalising education policy for service delivery to committees and school beneficiaries. directors. This review applies the Principal-Agent theory to explain the causal mechanisms at work in the use of RBF to improve education service delivery. It uses this theoretical 6 framework to examine RBF interventions to incentivise meso-level actors. In doing so, it finds a divergence from the universal theory needed to explain the unique nature of meso-level actors’ work. We, therefore, propose broadening the definition of RBF and viewing it as a set of reforms1 that align the objectives of multiple stakeholders by focusing on a common results framework, linking financing to these results, strengthening results measurement and giving autonomy to actors to achieve the results. We further propose to apply learnings from the Performance- Based Budgeting literature so that RBF for the middle management of education systems can be implemented in a differentiated way depending on the contextual reality of the existing system. This review then ends with some recommendations to consider when designing RBF interventions for meso-level actors. Research for this review was based on published literature and project documents on meso-level RBF interventions in education systems of low- and middle-income countries. Published academic and grey literature were reviewed to understand the impacts of RBF interventions, while the project documents provided us with an overview of the intervention design considerations that arose. Given the very limited empirical research available on the impacts of RBF in this domain, this study relied most heavily on design and operational data from project documents and from interviews of those involved with donor-funded projects. 1 This broader definition for RBF as a ‘set of reforms’ has previously been supported by Rendmans, Paul and Dujardin (2016). 7 RBF IS GROUNDED IN THE PRINCIPAL-AGENT THEORY. THE PRINCIPAL- AGENT THEORY EXAMINES THE RELATIONSHIP BETWEEN TWO ACTORS, NAMELY, THE PRINCIPAL, WHO WANTS A TASK COMPLETED, AND THE AGENT, WHO IS DELEGATED TO COMPLETE THE TASK ON BEHALF OF THE PRINCIPAL. 8 2 Principal-Agent Theory in Results- Based Financing at the Meso-Level of Education Systems T he primary theoretical framework of RBF is grounded in the Principal-Agent theory. The Principal-Agent theory examines the relationship between two actors, namely, the Principal, who wants a task completed, and the Agent, who is delegated to complete the task on behalf of the Principal. However, Principals and Agents often do not share the same priorities. The Principal-Agent theory attempts to overcome this disparity by designing an effective contract with an incentive structure that motivates the Agent to achieve the optimum output or outcome that the Principal desires. In an RBF arrangement, the Principal pays the Agent based on the delivery of In a Principal-Agent verifiable results. In a Principal-Agent relationship, where an Agent’s effort yields the relationship, where an results desired by the Principal, the Principal would want to manage and motivate the Agent to extract the best results possible. The compensation provided to the Agent’s effort yields Agent is a function of effort and result. If the causal link between effort and result of the results desired the Agent is weak, incentivising Agent effort will not maximise the desired results. Furthermore, Agent effort is often difficult to measure. Proxy measures (e.g. hours by the Principal, worked) are frequently used, which do not reflect the quality of effort. One solution the Principal would to the ambiguous relationship between both effort and measurement of effort with the actual result is to make the payment contingent on actual outputs or outcomes. want to manage and The measurement of the outputs and outcomes should be independently verifiable, motivate the Agent to so that both the Principal and the Agent agree that it is indeed delivered. extract the best results RBF interventions work towards aligning the interests of the two actors, that is, possible. the Principal only disburses payment when the Agent achieves the desired results. Since the Agent does not get paid if results are not achieved, the assumption is that RBF provides greater motivation for Agents to perform according to the Principal’s objectives. The Agent bears the risk of failure of the programmes but has greater 9 autonomy over how to deliver the results. The Principal forgoes some control over how the results are achieved but gains value for money by only paying for verifiable results actually achieved (Savedoff 2010; Clist and Verschoor 2014). Thus, RBF is designed to overcome the problem of an Agent acting in a manner contrary to the best interest of a Principal. Through this mechanism, the financing provided by the Principal will be used: • More effectively since the Agent wants to get paid for the work done; • More efficiently because the Agent will want to avoid any wastage, as there is only payment for results delivered; and • For the intended purpose as the Agent is held accountable for delivering verifiable results with the sanction of no payment. RBF arrangements in the education sector can be classified based on the Agent that is incentivised, as presented in Table 1. In many cases, RBF interventions for multiple Agents can be combined. Table 1. Types of RBF in education2 Level Who incentivises? (Principal) Who is incentivised? (Agent) RBF instruments 1 Donor Organisation National Government Results/performance-based aid 2 National Government (sometimes donor Local government, school manage- Performance-based transfers, organisations) ment (state, district, municipalities, output-based disbursement school) 3 Donor/National Government/Local Govern- Service providers (public or private Performance-based contracts, ment education service providers) or school grants, school vouchers, investors impact bonds3 4 Government/School Teachers or school directors Performance pay or bonus pay 5 Government/School Beneficiaries (students and house- Conditional cash transfers holds) Majority of RBF interventions in education and the subsequent lessons learned (R4D 2016; Lee and Medina 2019; GPE 2020) from these initiatives have concentrated on incentivising actors at two ends of the education service delivery system: • National-level policy planners • Frontline service delivery Agents like teachers and school directors In addition, many RBF interventions are also directed to the beneficiaries of the education system to incentivise them to access and maximise the benefits of education. A gap remains in understanding the motivation mechanism and impact of RBF when it is applied to Agents who work at the meso-level of education systems as administrators or managers (levels 2 and 3 in Table 1). While the Principal-Agent theory provides a basic framework to examine RBF contractual frameworks for meso- level actors, a deeper ‘attention to the choices available to Agents, their motivations, and appropriate framing of opportunities are all factors that will influence the success of an RBF initiative’ (Savedoff 2010, 11). 2.1 Principal-Agent Organisation-Employee Relationship The management structure in education systems places certain agencies, such as sub-national departments, service providers, school leadership and school management committees, in middle management, for whom we use the term 2 Adapted from Instiglio (2018); Gustafsson-Wright & Boggid-Jones (2020); Lee & Medina (2019) 3 Impact bonds have a more complex Principal-Agent structure, wherein the RBF contract is typically between the donor/government agency and the investor, whereas the service delivery is the responsibility of a service provider who may or may not be incentivised with an RBF contract. 10 meso-level actors. However, it is important to consider individuals working in these agencies as critical actors for making the RBF initiative work. Much of the literature applying the Principal-Agent theory as the basis of RBF uses analytical frameworks with a labour contract (i.e. a firm-employee relationship) in mind (Clist and Verschoor 2014; Rendmans, Paul, and Dujardin 2016). The incentive and motivation theories in the literature often apply to individuals rather than the firm. Overall, there is very little conceptual consideration of the cascading funder-agency-employee relationships. At the meso-level (and, arguably, also at the national level), the RBF contract is between a funding Principal organisation (like national line ministry) and an Agent organisation (sub-national office, school or a service provider). The actions of employees within the Agent organisation becomes a Blackbox – with no explicit stipulation of incentives for the employees to align their objectives with those of the Agent organisation (their employer) and, hence, also not with those of the Principal organisation. Overall, there is very There are two levels of Principal-Agent relationships present within an Agent organisation: little conceptual 1. Funder-Agent organisation relationship where the funder, like a national consideration of the government, donor or line ministry, is the Principal and the service delivery department, like a regional or district education office, school or university, or cascading funder- non-government service provider is the Agent. agency-employee 2. Agent organisation-employee relationship where the service delivery agency relationships. is the Principal and the employees of the agency are the Agents. Agent organisation performance is dependent on the performance of the individuals who work within the agency. ‘To ask about the motivational impact on agencies is really to ask about the motivational impact upon individuals of objectives, measures and targets specified for the agency as a whole’ (Paul and Robinson 2007, 331). The incentive structures at play in the organisation as a whole would not necessarily motivate individual employees to achieve results. A simple aggregation of individual employee efforts or contributions will also not guarantee the achievement of results by the organisation as a whole. 11 Organisations, by their very nature, are a manifestation of interdependence between units and individuals, and internal organisational management exists to orchestrate these interdependencies (Knott 1993). Depending on the context, individual employees need to be incentivised to maximise their individual output in isolation or to collaborate with other employees to achieve the results as a team. Therefore, the causal mechanism for incentivising the Agent organisation should include two critical elements. The first is a measurement system that aligns the Agent organisation with the goals and objectives of the Principal, and then further aligns these to work teams and individual employees’ performance goals. This should be made explicit within the performance contract between the Principal and the Agent organisation, and then aligned within the employment contract between the Agent organisation and the individual employee. The second element is the cascading of incentives from the Agent organisation to individual employees to motivate them to improve their performances towards the attainment of the Principal’s objectives. Incentives at the individual level are both financial and non-financial. The appropriate mix of incentives depends on the characteristics of the culture, organisation and individual, as well as on the objectives to be achieved. While generalisation is impossible, the incentive theory, especially incentives for the delivery of public services like education, can provide some options to consider when establishing incentives (see Section 2.3). 2.2 Defining and Verifying Results Making payment contingent on results requires setting measurable result targets that Making payment the Principal desires, followed by the alignment of these results with the objectives for the Agent organisation and its employees. Therefore, the definition of results is contingent on results crucial in terms of both indicators and indicator values, as it is a consideration of the requires setting political economy of who determines them. In certain situations where indicators for the results are difficult to define, indicators for effort may be used as proxy, in which measurable result case an agreement on the causal link between effort and result would be required. targets that the The Principal and the Agent should agree that the chosen indicators represent the results to be achieved and are observable by all actors involved. Once the Principal Principal desires, has an agreement with the Agent organisation, this agreement needs to flow down followed by the to the work teams and individual employees as well. alignment of these The engagement of an independent verifier who is responsible for auditing the results indicators provides transparency and accountability to the observation of results with the results. The Principal-Agent theory also implies negotiations between Principals objectives for the Agent and Agents, wherein the Agent (especially the individual employee) will tend to have more expert knowledge about what can be achieved. Thus, the Agent will have an organisation and its advantage in negotiations, likely resulting in a lesser result than what is potentially employees. achievable (Gailmard 2012, 14). It is in the interest of the Agent to inflate the results to maximise payment, while the Principal’s interest is to ensure that maximum results are delivered for the minimum payment. Once the results are achieved, the Agent will again have first-hand knowledge of the results achieved, which the Principal will need to measure and observe. An independent verifier for the results can serve as an arbitrator to foster transparency and accountability in both the Principal and the Agent. 12 2.3 Incentivising Agents The traditional incentive theory in economics refers to incentives exclusively in monetary terms or financial incentives for workers. Workers are assumed to be motivated solely by materialistic self-interest, that is, they want to accumulate material wealth for their own benefit. Workers also prefer leisure and want to minimise their amount of work while maximising payment. However, research on organisational psychology and behavioural economics, specifically on incentive motivation and accountability in public administration (in high-, middle- and low- income countries), show that public servants or individuals working for organisations with a social mission are often motivated by non-financial goals, as well as purely financial ones. An OECD publication on public sector modernisation asserts that in any agency, financial incentives may motivate a minority of staff, while ‘job content and career Research on development have been found to be the strongest incentives for public employees’ organisational (OECD 2005, 176–77). This is not to argue that financial incentives have no merit in designing RBF at the meso-level. Instead, we propose that the inclusion of non- psychology and financial incentives could complement financial incentives by considering additional behavioural motivating factors for individuals working in the education sector. economics, specifically In the subsections that follow, we first examine the merit of performance pay or on incentive bonuses as the financial incentives in motivating employees to deliver results desired by the employer and, in turn, the Principal in the education sector. We follow this up motivation and with a description of potential non-financial incentives and potential unintended accountability in consequences of incentives. public administration 2.3.1 Financial Incentives (in high-, middle-and In education, performance pay or bonuses for teachers to improve student learning outcomes have been tested in many low-, middle- and high-income countries; low-income however, system-wide reforms have been highly contested in the context of a teaching countries), show that force that is characterised by a considerable level of unionisation. Performance pay has been introduced in some interventions to motivate school directors to modify public servants or school management practices to improve education outcomes, though only a few individuals working have improved student learning. These incentives are designed to work by rewarding 4 employees with additional pay for delivering results. for organisations with a social mission are In a typical labour contract, employees receive payment or remuneration as compensation for work undertaken. Any payment to the workers is considered often motivated by ‘high powered’ when it is strongly linked to the output of their work, while it is ‘low non-financial goals, as powered’ if it has a weak or indirect link to the output, such as linked to time spent in doing the work (i.e. effort) (Paul and Robinson 2007). Performance pay or bonuses, well as purely financial which are ‘high powered’, are usually used as financial incentives to motivate ones. individuals to achieve the results desired by the employer. Employees are rewarded with additional payments on top of their regular remuneration (which is often ‘low powered’) if they deliver the results. The extent to which a bonus pay can motivate employees depends on the level of their regular remuneration. Theoretically, the higher the bonus compared to the regular pay, the more motivated the employee will be. 4 See Goff, Goldring and Canney (2016) for a review of interventions in the US; see Ganimian and Murnane (2016), Lee and Medina (2019) and Breeding, Beteille and Evans (2021) for reviews of interventions in low- and middle-income countries. 13 However, in many situations, especially in low- and middle-income countries, if the PSM is defined as regular pay itself is extremely low, then increasing the regular pay without any link to the performance itself can improve employee performance (Paul and Robinson an individual’s 2007). predisposition to This type of financial incentive can work in contexts in which the simple sum of act on motivations results achieved by individual employees meets the Principal’s objectives (Dixit 2002). However, in contexts wherein achieving the results require team cooperation, that exist uniquely incentivising individuals may lead to unnecessary competition and ‘can often in public institutions reduce job satisfaction, disrupt social relationships in the workplace, decrease performance...decrease quality...and increase [staff] turnover’ (Ferraro, Pfeffer, and and comprises four Sutton 2005, 20). dimensions: Other financial incentives could include career progression with salary increases or 1) attraction to public employment benefits like provision of housing, transportation or healthcare. policymaking; 2.3.2 Non-financial Incentives 2) commitment to public Non-financial incentives can involve motivational elements that counter the view of interest and civic duty; an individual motivated by materialistic self-interest alone. Some non-financial 3) compassion; and incentives can represent benefits to others or society at large more than the individual gaining materialistic self-interest. While there is no universal theory on non-financial 4) self-sacrifice. incentives, Paul and Robinson (2007) identify three sources of non-financial motivational sources: 1 2 3 Social motivation – Employees Moral motivation – Employees Intrinsic motivation – Employees pursue social relationships in the have a desire to act altruistically also draw motivation from the workplace like ‘acceptance and with an internalised sense of enjoyment of work, for instance, ‘the approval of others, adherence to commitment to work or to their pleasure of undertaking activities norms, and search for status and own work ethic. This is driven by one likes (input orientation) or from power’ (344). This motivation comes the ‘desire to behave in accordance internal satisfaction derived from from the human desire to belong with one’s moral beliefs and values, work achievements (independent or identify with a group and could for reasons separate from any of any social or material rewards translate into adhering to group personal advantages (material which those achievements might norms, responding to peer pressure, or social) which may accrue from generate)’ (344). avoiding social disapproval or gaining such behaviour’ (344). The worker social recognition. Additionally, a is motivated by a sense of duty sense of fairness, as in the treatment towards the beneficiaries whom of self, relative to others in the they have the responsibility to reference group, is an important serve. element of social motivation. These three types of motivations work together to incentivise employees to maximise their performance; however, the relationship between the three is highly dependent on the circumstances and the individual. For example, in a context wherein a large proportion of employees have moral or intrinsic motivation to accomplish the goals of the Principal, the employees that do not have this kind of motivation could feel the social pressure to perform. Research evidence suggests that public sector and social service organisations (even when they are for-profit) have a higher proportion of employees that have high levels of Public Service Motivation (PSM). PSM is defined as an individual’s predisposition to act on motivations that exist uniquely in public institutions and comprises four dimensions: 1) attraction to public policymaking; 2) commitment to public interest and civic duty; 3) compassion; and 4) self-sacrifice (Gailmard 2010). While the bulk of the research on PSM has been conducted in high- 14 income countries, there is some evidence that high levels of PSM also exist in public service in low- and middle-income countries.5 It has been argued that lower wages and limited use of ‘high-powered’ financial incentives in the public sector play a role in individuals with a high level of PSM self-selecting into public service employment. 2.3.3 Unintended Consequences of Incentives A combination of financial and non-financial incentives are at play in motivating individual employees to deliver results for the employer. Previous literature on the use of RBF for individuals has identified several behavioural distortions and undesired effects of financial incentives.6 However, in some cases, non-financial incentives may further exacerbate the potential negative effects of financial incentives or may help enhance them. Depending on the context, non-financing motivations may even crowd out the motivation from materialistic self-interest (Clist and Verschoor 2014). The following specific challenges in the use of incentives may arise and affect the overall performance of the Agent’s organisation: • Reallocation of effort to incentivised results – A sub-national government agency, such as the meso-level actor, is responsible for multiple activities. For example, a district education office may be responsible for providing pedagogical support to teachers, managing teaching and learning material procurement, education data collection, and managing employee payroll, just to name a few. When employees are incentivised for delivering a particular service, they may reduce their effort on other tasks, and the overall service delivery of the Agent organisation may suffer (R4D 2016; Lee and Medina 2019). This can be true for financial incentives like bonus pay or even for non-financial incentives like gaining recognition for oneself or the team. • Deliver what is measured – In most circumstances, measurements of results are proxy measures, i.e. they do not accurately capture the full intent of the desired outcome. This is especially true when the result is service provision rather than delivery of goods, as with most meso-level education agencies (Paul and Robinson 2007). ‘High-powered’ financial incentives or even certain non-financial incentives like recognition awards can motivate the employee to work towards only delivering the results that are measured, which may not necessarily achieve the full extent of the Principal’s desired result, and the Agent organisation may not succeed in achieving the results. For example, if improvements in student learning are incentivised, the Agent may only test the ‘easy-to-teach’ students and exclude those with special learning needs. • Gaming and cheating – High-value incentives can motivate individuals to manipulate the information on results to maximise their reward. This could include distorting the metrics by engaging in activities that require less effort but produce higher values on results measurement (Lee and Medina 2019). For example, if school inspectors are incentivised to increase the number of schools they visit to provide pedagogical support to teachers, they could increase the number of schools they visit without providing meaningful support or they 5 For empirical research on PSM in different regions of the world, see Vandenabeele and Van de Walle (2008), Van der Wal (2015), Gould-Williams, Mostafa and Bottomley 2015), Kim and Kim (2016), Ritz, Brewer and Neumann (2016) and Meyer-Sahling, Mikkelsen and Schuster (2020). 6 See Dixit (2002), Paul and Robinson (2007), Clist and Verschoor (2014), R4D (2016), Ganimian and Murnane (2016) and Lee and Medina (2019). 15 could report inflated data on school visits, which may not be easily verifiable. They may also under-report or create fake baselines to make the result targets appear higher. • Competition reduces overall productivity – When performance pay is based on individual employee output, employees may compete against each other, with an increase in the output produced by one employee resulting in lower levels of outputs produced by others. In public sector agencies, cooperation between employees is necessary, and ‘excessive wage differential between workers of the same level would be very damaging to team cooperation’ (Paul and Robinson 2007, 341). There may be no overall improvement in the results for the entire organisation. Workers who believe that they are unfairly treated in comparison to others may even resign or work counter to the employer’s interest (Paul and Robinson 2007). • Free-rider problem – When group bonuses or recognition awards are implemented to encourage collaboration, a free-rider problem could emerge wherein one or more team members might reduce their individual efforts, expecting to receive the incentive on the basis of others in the group delivering the result. If too many team members do this, the entire team will fail in delivering results (Paul and Robinson 2007; Dixit 2002). It is possible that internal group social relationships and peer pressure could discourage individual team members from shirking responsibilities. Additional funding to the team budget could also serve as a motivation by improving the resources for the employees to deliver results and reduce the opportunity cost of worker effort (Paul and Robinson 2007). • Individual performance may not lead to organisational performance – Results of managerial tasks are often an intermediate step towards the overall result desired by the Principal. Depending on the strength of the causal link between the managerial task and the ultimate result, ‘high-powered’ incentives for Agent organisation managers may not lead to overall results. Unless managers within the Agent organisation are actively working towards achieving the results desired by the Principal (for which they may not be directly financially incentivised), overall results may not be achieved (Paul and Robinson 2007). Intrinsic motivation from the enjoyment of work, respect from peers or subordinates or department reputation may serve as immediate motivation for managers to find the best avenues for reaching the goals. • Alter the pool of employees – It is possible that in the long-term, only individuals who are motivated by financial incentives would enter the profession, while others (with higher PSM) would exit the system. Eventually, this could increase the cost of public service delivery, as the number of individuals willing to work for a lower pay would decrease (Gailmard 2010; Lee and Medina 2019). The theoretical framework helps us hypothesise the ideal functioning of an RBF mechanism wherein an education sector Principal contracts an Agent to deliver results and receive payment contingent on achieving the results. An examination of the implementation of RBF targeting meso-level education Agents would help us further understand the level of alignment with theoretical predictions and how the mechanism functions in practice. 16 3 Research Methodology T his study used qualitative data analysis to draw lessons from diverse experiences with RBF at the meso-level of the education sector, particularly in low- and middle- income countries. We drew upon the experience of those who have designed, implemented and/or researched such RBF initiatives. The review included four different data sources: 1) academic literature; 2) project documents; 3) a global survey; and 4) key stakeholder interviews. First, a systematic literature review of theoretical and empirical research captured the RBF research findings to date at the meso-level. Although priority was given to literature focusing on the education sector, research articles on RBF at the meso-level in general, as well as those focusing on other sectors such as health and public administration, were also included to enrich the analysis. This intent was reflected in the literature collection parameters. Searches were conducted in the following electronic databases: Education Resource Information Center [ERIC], Jstor, SAGE, EBSCO, Taylor & Francis Online, WebofScience and NBER. The search was guided by 64 keyword combinations (see Appendix 5 for the full list), including variants of terms related to RBF and its specific modalities. The timeframe was unrestricted for education-related searches and limited to 2010 to June 2021 for research outside education, thereby allowing the study to capture the most recent debates in the general A total of 699 articles field. Additional literature shared by experts were added to the systematic were analysed, search. A total of 699 articles were analysed, among which 408 included theoretical and/or practical considerations of RBF at the meso-level and 128 among which 408 focused on the education sector. A review of the literature helped analyse the included theoretical key theoretical discussions in RBF at the meso-level in education and across sectors, along with the impacts of initiatives on education systems in low- and and/or practical middle-income countries. A pattern analysis of the literature and the initiatives considerations of RBF identified in this review can be found in Section 4, with more details provided in Appendices 1 and 2. at the meso-level and 128 focused on the Second, given that this literature was limited in capturing the diversity of the practical experiences of RBF at the meso-level of the education sector, this study education sector. also elicited important findings from design and operational data in education project documents, as well as from interviews with the key stakeholders involved in such projects. A review of over 300 education projects implemented in low- and middle-income countries was conducted to identify projects that explicitly outlined a component wherein RBF strategies were targeted at the meso-level of education. Relevant project design and monitoring documents were collected from the following project databases: 1) World Bank and its REACH programme; 2) 17 Global Partnership for Education (GPE); 3) Brookings (for impact bonds); 4) Global Partnership for Results-Based Approaches (GPRBA); 5) Asian Development Bank (ADB); 6) UK Foreign, Commonwealth and Development Office (FCDO); and 7) United States Agency for International Development (USAID).7 Third, to complement the sample of RBF initiatives, a project identification survey was distributed to 2000 education sector professionals via email. The survey was also distributed further through NORRAG’s social media channels (Facebook, LinkedIn and Twitter). After a thorough review of the project design from all the sources, 51 projects were identified as having RBF interventions at the meso-level of education. The key features of the interventions are summarised in Appendix 3. Nearly half (24) of these projects were analysed in depth, allowing additional learnings on various elements of their design and implementation. Finally, based on the criteria of maximum variation in the type of Agents, education level and geographic region, seven projects were selected for interviews with their key stakeholders (see Appendix 6 for the list of interviewees). These interviews gave rise to a richer understanding of how RBF plays out at the meso-level of the education sector, including its practical opportunities and challenges. Due to resource limitations, a smaller sample of projects were included for in- depth analysis and interviews. It is possible that additional learnings could have emerged through a deeper look at all 51 interventions. However, the research team is reasonably confident that the major findings are captured with this sampling technique since we a found high level of similarity across all projects included in the in-depth analysis. 7 Ideally, the researchers would have opted searching for projects that are not funded by external donors. However, this proved to be an extremely time-consuming exercise as no consolidated database of such projects is currently available. 18 4 Overall Trends in the Literature and Initiatives B ased on the 408 identified research publications on RBF at the meso-level, it is evident that this topic has been increasing in prominence over the last decade in all sectors. Looking at the entire 2010–2021 period, the focus of the majority of the publications was on health (38%). Research articles focusing on education were the second most frequent, accounting for 29% of the publications. In addition, the research literature is predominantly focused on high-income countries (52%), with only 35% of the publications focused on low- and middle-income countries.8 The number of research publications focusing on RBF at the meso-level of Out of the 113 research education has also been increasing since 2014, mainly leveraged by those articles analysed, 79% focusing on high-income contexts. As shown on the right side of Figure 1, out of the 1139 research articles analysed, 79% are focused on high-income are focused on countries. Additionally, the higher education level represents more than half of high-income countries. all publications on RBF in education at the meso-level (as shown on the left side of Figure 1), with the majority examining higher education in the United States. Figure 1. Distribution of RBF at the meso-level of education publications by school level (left) and by countries’ income levels (right) 6% 25 Early Childhood Development 20 4% 15 Not Applicable 19% 10 5 Primary 0 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 (until June) 14% Secondary No particular focus Focus on high income countries 55% 14% Focus on low-and-middle income countries Higher Education Secondary 8 Please refer to Appendix 1 for additional details on the RBF at meso-level literature analysed. 9 To allow a comparison with the trends in RBF at meso-level publications in all sectors, only publications from 2010 to 2021 focusing on education were included in this part of the analysis. Among the 128 research articles focusing on education collected from 1973 to 2021,113 were published after 2010. 19 The trend in the RBF initiatives implemented in projects in low- and middle-income countries is considerably different from the published research on RBF in the meso-level of education. By combining the initiatives in low- and middle-income countries explored in the research publications and those identified through project documents, 51 meso-level education RBF initiatives were identified. Majority of the initiatives are implemented at the primary school level, followed by the secondary school level.10 These RBF initiatives have been implemented in a greater variety of countries in all regions of the world, with a majority of them in Sub-Saharan Africa (19 initiatives) and South Asia (16). India (8) represents the country with the highest number of initiatives.11 The majority of initiatives were implemented under World Bank-funded projects. It is possible that other initiatives are being implemented within government Even with the structures that are not funded by donors; however, capturing those initiatives was predominance of beyond the scope of this study. Even with the predominance of external donors as funders of projects, in the majority (48%) of the meso-level education RBF initiatives, external donors as national government agencies were the ones who acted as the Principal, followed funders of projects, by regional/ state/district/departments (24%).12 When it comes to Agents, the most frequently incentivised Agent were in schools (29%), such as school directors or in the majority school management committees, followed by service providers (20%) and tertiary (48%) of the meso-level education institutional management (20%).13 education RBF initiatives, national government agencies were the ones who acted as the Principal, followed by regional/ state/district/ departments (24%). 10 More details on the distribution by school level can be seen in Figure 7, Appendix 2. 11 The distribution by countries is presented in Figure 6, Appendix 2. 12 The distribution of Principals in the RBF at the meso-level of education initiatives in low- and middle- income countries analysed is presented in Figure 8, Appendix 2. 13 The distribution of Agents in RBF at the meso-level of education initiatives in low- and middle-income countries analysed is presented in Figure 9, Appendix 2. 20 As shown in Figure 2, conditional school grants (29%) were the most common RBF modality, while impact bonds, performance-based contracts and performance- based intergovernmental transfers each represented 13%. Figure 2. Distribution of RBF initiatives at the meso-level of the education sector in low- and middle-income countries by type 2% 4% 2% Performance Based Bonus Payment By Result Reward Payment 2% 4% Results-Linked Budget Transfer Performance-Based Grant 29% 8% Conditional School Grant Public Private Partnership 10% Performance Based Institutional Grant 13% Impact Bond 13% 13% Performance Based Performance Based Contract Intergovernmental Transfer Most RBF initiatives at the meso-level of education used multiple results indicators to trigger financing to the Agent. When categorising the different types of education system indicators, we observed a predominance of results that represent outputs or processes (Table 2). The most common type of education indicator used in the RBF initiatives relates to student participation (e.g. attendance, retention, transition, access/enrolment). These are captured in the Access and Participation categories in Table 2. The second most common indicator type relates to management processes for the Agent to adopt or improve on to achieve results (e.g. preparing school development plans, timely hiring of teachers). Learning outcomes feature as the third most frequently defined results indicator. Other indicators used were a combination of input and process-type indicators like infrastructure development, provision of learning materials (teaching input) or support to teachers. Table 2. Types of results indicators linked to RBF payments used in RBF at the meso-level of education Frequency in the Analysed Educational Indicator Type Sample Indicators14 Initiatives Improved survival rates to grade 5 (Ethiopia General Participation 20% Education Quality Improvement Program for Equity) Creation of a school development plan (Project d’ Management process 18% Amélioration de la Qualité d’Éducation, Democratic Republic of the Congo) Student performance on curriculum-based mathematics Learning outcomes 17% tests administered to grades 10–12 (Aligning Learning Incentives, Mexico) Increased enrolment of women and other currently under- Access 13% represented groups, such as Schedule Tribes (SkillsStrength- ening for Industrial Value Enhancement Operation, India) 14 For more details on each project, please refer to the table on Appendix 3. 21 Frequency in the Analysed Educational Indicator Type Sample Indicators14 Initiatives Compliance with teaching hours (The Performance Teaching process 8% Commitments Program, Peru) Improved quality of Regional Flagship TVET Institutes Quality: Institutional 5% (Eastern Africa Skills for Transformation and Regional Integration Project) Number of teachers and principals satisfied with professional Teacher support 5% development (Enhancing Teacher Education Program, Vietnam) Sustained placement in formal employment (Proyectá Tu Employment 3% Futuro, Argentina) Construction of solid waste disposal (RBF mechanism in Infrastructure 3% Ceará, Brazil) Developing student assessment system (Sindh Secondary Data management: Learning 2% Education Improvement Project, Pakistan) Improved health and hygiene practices (Sindh Basic Health 2% Education Program, Pakistan) Textbook availability (School Sector Development Program, Others15 4% Nepal) 15 ‘Other’ includes the education indicator types ‘Quality of teaching and learning’, ‘Sports’, ‘Teaching input’ and ‘Financial management’, each of which represented 1% of the indicators identified in the analysed initiatives. 22 5 Evaluation Results of Initiatives W hile most of the initiatives identified through the project documents do not have published final evaluation or mid-term evaluation reports, especially in light of the challenging COVID-19 pandemic context, the initiatives identified in the research publications frequently did. The table in Appendix 4 summarises the findings from these published evaluation studies of RBF interventions. In general, the evaluations show mixed impacts of RBF for meso-level actors on educational outcomes, with some showing positive association with student learning outcome and managerial behaviour change. However, in many studies, the outcomes could not be attributed to the RBF arrangement. Several programmes that used RBF arrangements at the school or higher In Colombia, secondary education level showed positive effects on education indicators. Two schools that received initiatives in two provinces of Pakistan showed that students in schools with RBF arrangements to incentivise school leaders exhibit increased levels of payments contingent participation and test results (only in Sindh) (Barrera-Osorio and Raju 2017; IEG on performance 2019). Incentive pay to school administrators in Mexico (along with teachers and students) was also associated with improved student test scores (Behrman standards (on quality et al. 2015). Furthermore, performance-based subsidies to higher education of education and institutions in Mexico were associated with improvements in the quality of educational programmes, capacities of the teaching staff and student enrolment quantity of students) (Mungaray et al. 2016). In Colombia, secondary schools that received payments also resulted in contingent on performance standards (on quality of education and quantity of students) also resulted in students achieving higher test scores compared to students achieving other schools (Barrera-Osorio 2007). higher test scores Other programmes that incentivised actors to support school-level actors did compared to other not show any improvements in education. The Generasi programme in Indonesia schools. showed positive effects on health outcomes in the villages that received incentive payments. However, no improvements in education outcomes were observed. The incentives were associated with an increase in community and parental participation in school management, but that did not translate into improvements in school resources, teaching environment or classroom learning (Olken, Onishi, and Wong 2014; Aizawa 2019). 23 A pilot project in India that used non-financial incentives (certificate of excellence) also did not demonstrate any educational improvements (Mani, Vivek, and Bhattacharjee , n.d.). One system reform initiative in Peru showed that additional budget support to local government departments had a potential indirect positive impact on the learning outcomes of students. Incentive payments to local government departments were associated with a higher rate of achievement of the targeted results but had a weak association with learning outcomes in a regression analysis. Further difference-in- difference analysis revealed that the implementation of the incentive programme itself was associated with the improvement of test scores in Peru as compared to previous years regardless of the level of incentives received by the local government (Correa Miranda, García Medina, and Ugarte Vera 2017). A few other partnerships with non-governmental actors showed improvements in educational outcomes; however, these could not be directly attributed to the RBF arrangements either. The Educate Girls Development Impact Bond programme in India achieved remarkable improvements in both enrolment and learning outcomes. However, these improvements could not be directly attributed to the RBF arrangement, as the evaluation was only conducted to assess the impact of the programmatic interventions by the NGO Educate Girls (Loraque 2018; Gustafsson- Wright and Boggild-Jones 2019). Similarly, in Liberia, schools managed by private sector providers experienced an increase in enrolment and learning outcomes, but there was no evidence that this happened due to the RBF arrangement (Romero and Sandefur 2019). Improvements in the number of students sitting and passing the exams also increased during the RBF programme period in Ethiopia. The evaluators, however, could not attribute the improvements to the RBF arrangement without a counterfactual; RBF payments were made to all regions of the country (Cambridge Education 2015). 24 25 RBF INITIATIVES DESIGNED FOR MESO-LEVEL AGENTS DIVERGED SIGNIFICANTLY FROM THE PRINCIPAL- AGENT THEORY. 26 6 Observed Divergence from Theory in Practice W hile our review of the existing literature, including evaluation studies, on Principal-Agent relationships has been somewhat helpful, it does not provide much insight into these relationships at the meso- level. Therefore, we conducted a deeper examination of the design and implementation of RBF initiatives at the meso-level through programme documents and interviews. This analysis provides a better understanding of the unique context of middle- management Agents in which they translate, transfer and broker the results sought by the Principal. While the examination of intervention design and implementation in the absence of programme results evaluation does not give definitive answers on the causal mechanism yielding results, we are able to The incorporation of observe the fidelity to the Principal-Agent theory in practice. In fact, we observed design modifications to that RBF initiatives designed for meso-level Agents diverged significantly from the Principal-Agent theory, as discussed in Section 2. The incorporation of design the ‘ideal’ theoretical modifications to the ‘ideal’ theoretical model of RBF is essential in responding model of RBF is to the specific contexts of different education systems. However, the level of success of RBF in achieving the desired results will also depend on the level of essential in responding dilution of the envisioned causal mechanism of RBF. to the specific contexts In this section, we specifically discuss the divergence in RBF operationalisation of different education due to the presence of multiple Principals, lack of operational funds, challenges systems. in defining results and lack of Agent autonomy. The key features of all intervention examples in the specific programmes and countries used in this section can be found in the table provided in Appendix 3 of this report. 6.1 Multiple Principal-Agent Relationships A core element of the Principal-Agent theory is to establish ‘who is (or is supposed to be) accountable to whom’ (Gailmard 2012, 1). For simplicity’s sake, in RBF contracts, this relationship is theorised between one Principal and one Agent. At the meso-level, this could be the national line ministry and its sub-national offices, or a national or sub-national ministry office and school management or a non-government service provider. Appendix 3 lists the Principals and Agents for all the interventions examined in this review. For example, in Ceará, Brazil, 27 the municipal government (Agent) enters into a contract with the state government In a meso-level RBF (Principal), whereas in the national Upper Secondary Education reform programmes, also in Brazil, each of the State Secretariats of Education (Agent) has a performance arrangement, the Agent agreement with the national Secretariat of Basic Education under the Ministry of may have one Principal Education (Principal). In Nepal, the school (Agent) has a contract with the District Education Office (Principal) to receive conditional school grants. In the All Children defined in the contract, Learning Grand Challenge, USAID, World Vision and the Australian Government but in reality, it may (Principals) enter into a contract with private service providers (Agents) to develop books for multiple regions. have multiple reporting In a meso-level RBF arrangement, the Agent may have one Principal defined in the relationships with contract, but in reality, it may have multiple reporting relationships with actors at actors at various levels various levels of the education system who can all be the Principal. Public sector management is made up of multiple tiers of Principal-Agent relationships (Dixit of the education system 2002). Meso-level actors represent only one layer of the entire education system; the who can all be the effect of an RBF contract for these actors is influenced by several other Principal- Agent relationships. Figure 3 illustrates the network of multiple relationships that Principal. exist around the meso-level actors creating a relationship network. Agents at the meso-level end up needing to respond to multiple Principals who may or may not have aligned objectives. This can lead to the weakening of the overall incentives for the Agent to deliver on results for any of the Principals, including the one explicitly stated in the RBF contract (Rendmans, Paul, and Dujardin 2016). Figure 3. Principal-Agent relationships within an education system Employee Organisation RBF Contract Payment Results Principal Agent Students (Beneficiaries) National Authority Independent Verifier Intermediary Parents Donor Principal-Agent Relationship RBF Action RBF Contract Fundamentally, all Agents in an education system are meant to respond to the needs of students who are both the beneficiaries and the ultimate Principal for education Agents. However, in public education systems, students have neither the financial nor the authoritative power (Bolman and Deal 2011) over the Agents to align their objectives. Frontline workers, like teachers, may be able to observe or elicit information on student performance and needs due to their proximity. However, this 28 information is not easily accessible to meso-level Agents like provincial or district offices without an explicit information system management created for this purpose. In some higher-education interventions like the ones in Chile, East Africa, and Mexico, where payment is linked to student enrolment, it can be argued that adult students use their power of choice to enrol in a given higher education institution as an avenue to hold Higher Education Administration accountable. In the Vietnam Enhancing Teacher Education Program (ETEP) programme, satisfaction reporting on the delivery of continuous professional development programmes by teachers was used as a results indicator. Another mechanism to align student objectives with the meso-level Agent is to introduce management authority of parents over the Agents. Some form of parental involvement in school management committees (SMCs) has been functioning in most countries for decades. However, in most cases, parents or the SMC do not have any authority over payment disbursement. Several RBF interventions that utilise conditional school grants show evidence of active engagement of the SMCs. In Senegal, the SMC is a co-recipient of the school grant and is engaged in managing the activities undertaken by the school director and teachers. Similarly, in Cameroon and Indonesia Generasi, the larger school community is responsible for holding the school leadership accountable for the results, but the Principal making the payment is a national-level actor. Beyond SMCs (or other community-level actors), many projects had additional intermediary entities tasked to manage the Agent on behalf of the Principal. This was especially visible in higher education initiatives. In the Eastern and Southern Africa regional programme, the participating Agent institutions were supervised and managed by the Regional Facilitation Unit, Regional Steering Committee and a National Steering Committee. The programme in East Africa had supervision oversight from multiple entities, namely, National Project Coordination Unit, Regional Facilitation Unit at Inter-University Council for East Africa, Regional Steering Committee, Regional Technical Advisory group and regional Flagship TVET Institutes. In the India Technical Education Quality Improvement Project (TEQIP), the national Ministry of Human Resource Development delegated the day-to-day implementation of RBF for the training institutes to the National Project Implementation Unit. In nearly all donor-funded programmes, an independent non-governmental organisation was engaged in the independent verification of the results. In Cameroon, the independent verifier ended up also serving as an intermediary manager for the participating schools, with additional responsibilities such as training and oversight over the implementation of national policies for free primary education. Similarly, in Zanzibar, Tanzania, an international firm was hired to provide end-to-end support for school grant management, which included not only the monitoring of activities but also the training of SMCs and implementation of outreach and communication with other actors. 29 Box 1: Cameroon Education Reform Support Project The Education Reform Support Project is a nationwide project with a special focus on disadvantaged areas in Cameroon. At the meso-level, the Ministry of Basic Education (MINEDUB) enters into a performance-based contract with primary schools. If the schools meet the targets of improved teacher attendance, lesson preparation, student retention, budget transparency, textbook utilisation, community satisfaction and implementation of the fee-free education policy, they receive grants (performance indicators related to a school’s inclusion of refugees are also considered). The grants range from USD 500–1500. The disbursement of these school grants is contingent on the independent verification of the schools’ performances. It began with a pilot project reaching 400 schools and is planned to be scaled up to reach 5,000 schools (including the additional finance scale-up and timeline extension). The selected primary schools must also have school management councils to oversee the development and implementation of the school improvement plans. The independent verifying agency also had oversight responsibility, as they are meant to not only verify results but also provide coaching and training to school leadership in meeting the results targets. Within a decentralised education system, local government Agents would find multiple layers of Principals within the education administrative system. For example, a district office may be answerable to a state or provincial office and to the national government. In the Senegal Quality Improvement and Equity of Basic Education programme, it was difficult to discern the true Principal for the school level Agent; funds were disbursed from the national ministry but the regional and district authorities maintained oversight within the RBF arrangement. By contrast, in Ceará, Brazil, given that the state government had financial autonomy, it was able to enter into the RBF contract with the municipal authorities without any direct oversight by the national government. In donor-funded programmes, external donors, who are not the Principal in the RBF contract, bring in their own objectives that could be highly influential for the actors. This was previously identified as a complicating factor in applying the Principal- Agent theory in the health sector and is also evident to some extent in education (Rendmans, Paul, and Dujardin 2016). In almost all of the initiatives where we conducted interviews, the concept of RBF was introduced by the donor agency. Since, in most interventions, at least part of the RBF payment was from donor funds, the donors were heavily engaged in negotiating the results indicators that were linked to payments. Donor engagement further complicates the identification of the Principal because the funds often flow through ministries of finance. The line ministry of education, and its sub-national office down the chain, becomes the Agent of the Ministry of Finance ‘because they are required to produce a certain level of public output – including the quality of this output – exchange for their budget appropriation’ (Leruth and Paul 2006). Within the Vietnam ETEP project, the Principal-Agent relationship between the Ministry of Finance and the Lead Teacher Training Institutes is more explicitly demonstrated than in other projects. Funding for the Lead Teacher Training Institutes is appropriated by the Ministry of Finance (within the education ministry budget) as part of the national annual budgeting process. The pressure to disburse all committed funds from the donors could also dilute the incentive structure (Rendmans, Paul and Dujardin 2016). For any Principal (including donor agencies), non-performance by the Agent is an undesirable consequence, and 30 the Principal carries a reputational risk in the case of failure to achieve results. Since disbursement of funds is a proxy measure of performance, the donors are keen to ensure that the funds are spent. This makes the donor more open to negotiating easier-to-achieve results to avoid non-performance (Clist and Verschoor 2014). Consistent reasoning behind the changes is difficult to ascertain within this study as many renegotiations took place during the COVID-19 pandemic. In most of the interviews, the COVID-19 pandemic was mentioned as the most challenging hurdle for RBF success. In many projects, the pre-agreed targets were either no longer relevant in responding to the education crisis during the pandemic or were unrealistic given the scope of the challenge. However, it is possible that renegotiation could have happened even outside a pandemic context because of the pressure on donors to disburse committed funds regardless of the results achieved. If the Agents are aware of this pressure on the Principal to make payments, they are able to negotiate easier targets, which can dilute the Principal’s objectives. During the second year of the programme in Bangladesh, the RBF component was restructured to reallocate funds from the indicator Teachers’ Time Spent on Teaching to other indicators to enhance the likelihood of the achievement of results within the implementation timeline (World Bank 2020). In the Odhisa Higher Education Program in India, the World Bank, the state government and the higher education institutions negotiated achievable targets each year. As identified in Section 2.1, when the Agent is an organisation, the role of the Many RBF employees as the Agents of the meso-level Agent itself becomes critical. However, in the initiatives examined, consideration of this relationship was largely absent. interventions have Few programmes included individual-level incentives. In Mexico’s Programa Integral been designed to de Fortalecimiento Integral and in Pakistan’s Sindh Education Sector Project and Improvers Bonus Program for Government School Teachers, performance bonuses provide a portion were given to teachers and school directors. In India’s Strengthening Teacher of the payment to Mentoring and Monitoring System project, Resource Persons were given a certificate of excellence for attending the training and capacity-building activities. It is possible the Agent prior to that in other initiatives, the Agent passes on some of the performance financing that the achievement or the organisation receives to its employees to incentivise them. However, this design feature is not mentioned in project documents and was not evident in the projects verification of the for which we conducted interviews. results. 6.2 Need for Operational Funds to Achieve Results Typically, RBF in development has been defined by the disbursement of funds after the achievement of results, i.e. provision of ‘rewards [by the Principal] after the credible verification of an achieved result [by the Agent]’ (Lee and Medina 2019, 10). This shift in paying for results rather than paying for inputs and activities is not only needed to align the Principal’s and Agent’s objectives but is also intended to result in efficiency gains for the Principal. Since the Agent does not get paid if the results are not achieved, the assumption is that RBF provides greater motivation for Agents to perform according to the Principal’s objectives. The Agent bears the risk of failure, and the Principal only pays for achieved results rather than wasting funds and management time on unsuccessful projects. However, many RBF interventions have been designed to provide a portion of the payment to the Agent prior to the achievement or verification of the results. 31 It appears that many (at least 36 out of the 51 presented in Appendix 3) RBF interventions have implemented this model, wherein at least a portion of the payment is made after the verification of achieved results. In Ceará, Brazil, the state government gave financial rewards to municipalities with the highest levels and highest gains in the education quality index. In Benin, the regional and sub-regional inspectorates received incentive payments if they demonstrate improvements on the agreed indicators compared to the previous year. The FCDO-funded programme in Ethiopia made payments to regional education bureaus based on increases in the number of students taking and passing secondary school exams. In Indonesia, the Government of Jakarta awarded additional school grants to the best-performing schools in each district. In the World Bank-funded Sri Lanka General Education Modernization Project, the Ministry of Education paid schools based on improvements in learning outcomes and improving teacher professional development. In at least 10 out of the 51 initiatives16, it appears that the full payment to the Agent was triggered based on the commitment to achieve results – either or both with a Agent required upfront signed agreement and an approved implementation plan to achieve results in the funding to undertake future. However, they did not require any actual education results or even processes. The actual disbursement of the payment in these instances may have taken place the planned activities, at different intervals of the project cycle, but they were not conditional on the and it was not feasible verification of achieved results. to pay them after the The majority of meso-level actors in education systems are within the public verification of results. administrative system with large recurrent expenses and very low levels of unallocated funds. If payments are made after the achievement of results, the Agent is obliged to arrange the capital to fund the input and process costs from other sources. The interviewees of the India Odisha Higher Education Program and Brazil Support to Upper Secondary Reform specifically noted that the Agent required upfront funding to undertake the planned activities, and it was not feasible to pay them after the verification of results. Therefore, in an RBF arrangement, the Agent frequently needs an alternate revenue source to undertake the required activities. Several meso-level interventions examined in-depth through the project document review showed an alternate RBF design feature wherein the Agent received partial funding prior to achieving verified results. For example, in the East Africa programme, the first tranche of the grant was transferred to the Technical and Vocational Education and Training (TVET) institutes on the approval of a plan or upon signing of the Memorandum of Understanding. In East and Southern Africa, close to 18% of the funds were disbursed to the universities with the signing of the agreement and approval of the detailed implementation plan. In Nigeria, an advance payment of 15% was given, and additional advance payments were made with the approval of the state’s literacy programmes, selection of participating schools and estimates of the textbooks and teacher guides needed. In India’s Strengthening Teaching-Learning and Results for States, the Industry Cluster received a portion of the payment with the adoption of guidelines for the Industry Apprentice Initiative Grant Mechanism by the Ministry of Skills Development and Entrepreneurship. 16 See Appendix 3 - Brazil Support to Upper Secondary Reform, Cameroon Education Reform Support Project, Chile Programa de Convenios para el Desempeño, Democratic Republic of Congo Project d’Amélioration de la Qualité d’Éducation, India Odisha Higher Education Program, Madagascar Basic Education Support Project, Mexico Programa Integral de Fortalecimiento Integral, Senegal Quality Improvement and Equity of Basic Education, Tanzania Education and Skills for Productive Jobs and Vietnam Enhancing Teacher Education Program 32 In other interventions, payments were made based on agreed process management milestones. In Bangladesh, a portion of the conditional school grants was disbursed to the school by demonstrating minimum accountability requirements, including a functioning SMC, school data submission and annual social audits. In the Sindh Pakistan Secondary Education Improvement Project and the Basic Education Program, improvements in community engagement triggered payments, while in the Sindh Education Sector Project, the establishment of employment conditions itself was used to release payments. When the RBF arrangement is made for additional activities or pilot interventions that are not part of the regular operations of the Agent, it may be possible to pay after the achievement of results. However, for regular operations, government offices may not be able to take on the higher financial risk because they do not have the liquidity to spend on inputs. Non-governmental or private sector organisations that are dependent on outside funding may thus be more open to RBF agreements. In Ethiopia and Nepal, Helvetas paid the private training providers after verification that the target beneficiaries completed the training to improve their skills and sustain employment. For some of the awards in the Grand Challenges (FCDO, USAID, Australian Government, World Vision), the prize-winning agency receives payments based on actual deliverables rather than reimbursement for activities. However, private organisations’ risk appetites also depend on financial liquidity. They have to invest the capital needed to deliver results. Smaller organisations may be more risk averse, as the RBF transfer may represent a large portion of their budget, and this could make service delivery more unstable. Audits of RBF in the UK showed that small service providers could not ‘carry the financial risk of failure if the amount of money at risk is very large and so may not bid to provide services’ (Clist and Verschoor 2014, 8). This could reduce the pool of Agents needed to carry out educational services. To an extent, impact bonds (see Argentina Proyectá Tu Futuro, India Educate Girls, India Education Outcomes Fund, Quality Education India, South Africa Impact Bond Innovation Fund, Uzbekistan Promoting Early Childhood Development Project) have structured a financing arrangement that solves the upfront capital challenge for the Agent. With an impact bond, there are essentially two Principal-Agent relationships. First, the Outcome Payer (government or donor or philanthropy) contracts with an Investor to repay the investor’s investment plus a financial return if a certain result is achieved. Second, the Investor gets into a contract with a Service Provider to produce the result; this second contract may or may not be results-based.17 However, the existence of the Investor means that operating capital is available to the Service Provider from the outset. This sub-section has shown that in initiatives wherein at least a portion, if not all of the payment, is made before the achievement of results, the ideal theoretical structuring of RBF leaves the meso-level Agent with the problem of financing its operations to deliver the results. Agents need operational capital; traditionally, this is provided by the Principal. Indeed, there is usually no other option if both the Principal and the Agent are within the public sector. However, this is inconsistent with the Principal paying for achieved results. 17 This is a theoretical financial arrangement in impact bonds. In practice, other stakeholders like intermediaries and guarantors are often engaged to mitigate risks to investors. 33 Box 2: Proyectá Tu Futuro: A Social Impact Bond in Buenos Aires, Argentina Proyectá Tu Futuro (PTF), the first SIB in the country, was launched in Buenos Aires, Argentina, aiming to simultaneously tackle the low graduation rates in secondary education and unemployment among young adults. PTF was commissioned by the Government of the City of Buenos Aires to help 1,000 participants receive training in soft skills from one of the four service providers (AMIA Foundation, Forge Foundation, Pescar Foundation, Reciduca Foundation) in the programme, which would also support and monitor the participants in finding job placements. The service providers received nearly USD 1,000,000 upfront investment for managing the programme operations from the following private investors: Bank of the City of Buenos Aires, Bank of Galicia, IRSA Local Commercial, Exotrade S.A. (Organización Román), Jorge Luis García Belmonte. By October 2020, a total of 751 young adults had finished their training, 304 were in training and 46 had dropped out. Furthermore, of those who had finished their training, 96 (12.8%) had finished secondary school (Outcome 1) and 254 (33.8%) had successful placements (Outcome 2). Of this number, 141 (18.8%) had sustained their job for four months (Outcome 3) and 22 (2.9%) had already maintained it for a year thus far (Outcome 4). The Government of the City of Buenos Aires made payments only for verified programme outcomes. 6.3 Difficulty in Defining Results Making payments contingent on results requires the Principal and the Agent to clearly define the results and indicators to measure those results. The Principal and the Agent need to have an agreement regarding the defined results, especially since the achievement of such results needs to be attributed to the Agent’s action. Defining the results of education service delivery by meso-level actors is in itself a complex task. In some sectors, defining the results indicators is relatively straightforward where the supply of the service often results in the expected outcomes. For example, in the health sector, injecting a vaccine, in most cases, would result in immunisation against the disease. In infrastructure, the number of houses connected to electricity, telecommunications and water services, or kilometres of road constructed, to a certain standard, often results in the use of these services for economic and social activities. However, educating an individual, as in achieving learning, cannot be attributed to the actions of any one actor. Changes in learning outcomes take time to achieve, and a series of actors and external factors influence it. A longer ongoing process is involved rather than a specific discrete action like a vaccination in health or an electrical hook-up in the power sector. Except for inputs (e.g. number of students or schools), education process and learning outcome indicators are not as simple (or as cheap) to measure as in some other sectors. These complications mean that education projects using RBF rarely use outcome measures. In theory, in education RBF projects, four types of results indicators could be used: ‘input (e.g. set number of textbooks delivered to the school), process (e.g. an effective curriculum put into place), intermediate (e.g. number of schools inspected with reports published on the government website) and outcome (e.g. demonstrated improvement in student test scores)’.18 As shown in Table 2 in Section 4, process indicators are featured frequently as a results indicator – management process, financial management, data management, teaching process and teacher support. Some form of learning outcome indicator was used only in 25 initiatives. Instead, many initiatives set targets for results with access (19) and participation (29) indicators. 18 Interview with Jaime Saavedra, https://www.worldbank.org/en/news/feature/2021/02/01/results- based-financing-in-education-peru-incentive-program-and-performance-commitments 34 Box 3: Benin Global Partnership for Education Project Phase 3 Performance-Based Incentives (PBI) were given to Regional and Sub-Regional Inspectorates by the DDEMP (Direction départementale des enseignement maternel et primaire) to improve teachers’ pedagogical support. The DDEMP pays the Inspectorates a maximum of USD15,000 in a given school year it they achieve at least 75% of the agreed results. There are three incentivised results: (i) inspectors’ impact on the overall quality improvement at the school level, including teachers’ pedagogical support, following inspectors’ and educational advisors’ class visits; (ii) improvement in school enrolment, especially for girls in communes where the Gross Enrolment Ratios deviate too much from the national average; and (iii) a reduction in repetition and dropout rate in both Primary and Lower Secondary Education. The dominance of the results targets defined with process indicators (at least 49 indicators in various initiatives in Table 2) shows that even when RBF is used at the meso-level, the Principal is often incentivising effort and the process to achieve the results rather than paying for the result itself as defined by the outcome indicators. This means that RBF is being used to incentivise the Agents for their action, but not necessarily for an action that leads to the desired results. This feature of RBF in education could be an indication of the challenge faced by meso-level Agents who may or may not actually be able to influence the results, such as learning outcomes. For example, a school or school manager Agent may or may not be able to acquire the needed textbooks, recruit teachers of adequate quality, etc. This is not specific to the education sector, but it is much more pronounced given that a wide variety of inputs are needed for learning to be achieved, including 35 competent teachers, class size, hours of instruction, books and materials, children who are sufficiently well-nourished to learn and parental involvement. The Principal- Agent theory assumes that the Agent is able to deliver the education but, in practice, the Agents at the meso-level of the education sector may not have the autonomy to do so (see subsequent discussion on autonomy in Section 6.4). The Generasi health and education intervention in Indonesia provides an interesting example of the difference between education and other sectors. Village communities received block grants for health and education. An RCT was established in which a control group received no performance-based payment, but a treatment group received 20% of the grant in the second and subsequent years according to their results in terms of eight health and four education indicators (primary and lower secondary enrolment and attendance). This incentive led to better, although somewhat modest, performance in health and no improvement in education, including in the classroom environment (though there was some increase in community and parental involvement in school management). A possible explanation relates to the existence Evaluators of the of an RBF cascade in the health sector (midwives were partially paid according to Peru’s Performance performance and also worked longer hours as a result) but not in the education sector (teachers were not paid according to performance) (Olken, Onishi, and Wong 2014; Commitment reform Aizawa 2019). However, this result has to be interpreted with caution as teachers are programme suggested not necessarily responsible for the enrolment and attendance indicators used that are affected by many factors, including household attitudes and risks. that having a well developed There is merit in including some process and output indicators as milestone targets to trigger payment instalments to the Agent. Having these milestone targets can support theory of incremental changes in the Agents mode of operation for them to eventually achieve change which explains the overall results. For example, in the School Sector Development programme in Nepal, some payments were made based on the improvement of teaching and the ‘transmission school management process (like availability of textbooks, improved time spent on channels from teaching, use of EMIS); others on improved student participation (attendance and retention) and additional payments on improvements in learning outcomes. programme activities to educational outcomes’ Having a results chain linking the process and outputs to eventual outcomes could reward the Agents for behavioural change in focusing on results. Evaluators of the could have helped the Peru’s Performance Commitment reform programme suggested that having a Principal maintain well-developed theory of change which explains the ‘transmission channels from programme activities to educational outcomes (Correa Miranda, García Medina and some predictability that Ugarte Vera 2017, 30)’ could have helped the Principal maintain some predictability incentivising the Agent’s that incentivising the Agent’s action would lead to results. action would lead to 6.4 Lack of Agent Autonomy results. Another basic mechanism of RBF is moving the financial risk of achieving results onto the Agent while providing it with greater autonomy over tasks to achieve the intended results.19 The assumption here is that the Agent has better information than the Principal on the level of effort and strategies that are needed to achieve results. This is especially relevant for meso-level Agents who are closer to the local context of education service delivery. This is also aligned with the rationale for decentralisation in education, where local actors have more contextual knowledge (McGinn & Welsh 1999; Dixit 2002; Al-Samarrai and Lewis 2021). While decentralisation reforms (administrative, fiscal or political) are not a necessary precondition for the use of 19 For more discussions on the theoretical underpinning for shift of risk, see Savedoff (2010), Birdsall and Savedoff (2011), Rendmans, Paul, and Dujardin (2016), Instiglio (2018) and GEMR (2018). 36 RBF, our analysis revealed that the level and breadth of autonomy of meso-level actors, such as district education officers or school directors, in supervising teachers may impact the success of the RBF intervention. Interviewees in the evaluation of Peru’s Performance Commitment programme attributed the success of the incentive payment partially to the flexibility to spend the incentive funds on activities that the local government deemed necessary rather than on activities mandated by the national Ministry of Education (Correa Miranda, García Medina, and Ugarte Vera 2017). Past empirical research has not demonstrated the presence of management autonomy of Agents in RBA projects (Clist 2019; Honig 2015). Even in systems that are decentralised, the devolution of management decision-making autonomy to meso-level actors is not a given (Al-Samarrai and Lewis 2021). In this review, the true level of management autonomy of Agents in the interventions is difficult to discern as we relied primarily on programme documents and limited interview responses. However, several examples point towards a lack of autonomy in RBF initiatives. In the Vietnam ETEP and India Odhisa Higher Education Program, the performance agreements signed between the department of education (Principal) and the tertiary education institutions (Agent) had a detailed list of activities that the tertiary education institutions needed to undertake in order to receive the funds. This suggests that the Agents did not have much freedom to change the activities once the agreement was in place. In most conditional school grants, the school director, who is the school level The level and breadth manager, does not have the decision-making power over the recruitment or compensation of teachers. In Chile, the 2011 Ley de Calidad y Equidad de la Educación of autonomy of (Law for Quality and Equity in Education) provides greater but regulated autonomy to meso-level actors, school directors over staffing and budget matters. For example, a school director can fire up to 5% of teachers each year, but they have to be identified as underperforming such as district through the national teacher assessment system (Montecinos et al. 2015). Research education officers or on competitive school grants in Senegal suggests that schools that spent the grant funds on human resources for pedagogical improvements had better student test school directors, in scores than schools that spent the funds on school materials (Carneiro et al. 2020). supervising teachers It is possible that the reason why there is little to no evidence available in the may impact the education sector to support that RBF increases the autonomy of Agents is that there are no pure forms of RBF interventions that have been implemented, an explanation success of the RBF Clist (2019) gave for the challenge in examining Agent autonomy in RBA. intervention. Another explanation comes from the theoretical framework of RBF itself. Agents face a trade-off between accepting the financial risk of receiving payments for results in exchange for greater autonomy over the management of tasks to achieve results. When the Agents do not have the appetite or means to take on the financial risk (lack of liquidity as discussed earlier), they may be willing to forgo autonomy in exchange for funding reliability. In some cases, even when the problem of operating costs is resolved, the Agent may not have the motivation to take on the additional burden or workload. Two research in Chile found that under the RBF or performance-based agreements, Agents saw their workload increase significantly, giving them less time for their previous tasks and responsibilities. At the primary and secondary school levels, on the average, school directors spent 24% of their time performing marketing tasks to increase 37 enrolment rates (Montecinos et al. 2015). Similarly, in the tertiary education system, greater autonomy for the Agent (university managers) came with an increased workload for ensuring accountability within the regulatory framework defined by the Principal (the state). The Agents (state university managers in Chile) gained many new responsibilities – they not only had to deal with reporting results to the Principal but also with changing the institutional culture to adopt a monitoring culture in their administrative, financial and educational processes (Araneda-Guirriman 2016; 2018). In the India Odhisa Higher Education Program, the World Bank and state government staff introduced a competitive funding process by requiring higher education institutions to submit results-focused project plans with the vision of building the capacity of the institution for potentially adopting an RBF arrangement in the future. However, majority of the higher education institutions in the state found the process demanding and preferred to not receive additional funds at all. In Senegal, schools that applied for the competitive school grants appeared to be better resourced and better organised than the schools that did not apply for the grant (Carneiro et al. 2020). In many of the initiatives (19 out of 24) where we conducted an in-depth analysis of the entire programme design, the Agent received additional capacity building or technical assistance to deliver the results. The provision of this additional support from the funder also implies that the Agent is not perceived to have the capacity to deliver the result even though they have more contextual knowledge. These examples provide an indication of the need for some threshold capacity (skills and resources) for the Agents to be willing to take on the additional workload of greater autonomy and financial risk. 38 7 A Differentiated Approach Linking Financing to Results A n examination of the RBF interventions directed at meso-level actors in education shows that currently, very few arrangements represent an ‘ideal type’ of RBF, where payments from a Principal are made to an Agent only after the achievement of verified results. As discussed in Section 6, the design of RBF interventions to incentivise meso- level Agents in education reflects several structural challenges faced in an education system at this level: • Multiple Principal-Agent relationships • Need for operational funds to achieve results • Difficulty in defining results • Lack of Agent autonomy Regardless of the divergence from the theoretical causal logic of using RBF, all of the project interventions examined clearly present a definite focus on results with contractual arrangements between a Principal and an Agent. In all interventions, the Principals explicitly or implicitly aimed to: • Improve programme effectiveness by creating an agreement with explicit results with the Agent; and • Encourage the Agent to achieve results by linking financing to the stated results. However, if the rationale for modifications in the RBF structure is not carefully considered and the logic of focus on results for fund disbursement is not made explicit, key implementation stakeholders may not accept the model or implement critical elements to their full extent. This would, in turn, impact the level of success the RBF intervention would have on producing the desired results (with fewer negative unintended consequences). RBF can be viewed as a set of wider reforms in results-based management that focus on results planning and budgeting, alignment of stakeholder interests, 39 and systems strengthening for monitoring results (Rendmans, Paul, and Dujardin 2016). The wider reform strategies lead to an expansion of the RBF definition linking funding to results rather than only to disbursement after the achievement of results. In many interventions where the payments were made prior to the achievement of results, the interview respondents indicated that, in the future, these interventions were intended to evolve into arrangements with payments made against achieved results. This evolution of financing strategy implies the potential for a differentiated approach to Results-Based Financing, where certain RBF strategies are introduced within the project or system financing prior to the implementation of an ‘ideal type’ of RBF. The broader definition can provide a useful starting point for introducing RBF at the meso-level of education (and potentially other sectors) without always limiting the timing of fund disbursement until after the achievement of results but still maintaining a focus on financing for results. This more extensive RBF definition allows for lessons to be extracted from experiences in the field of Performance-Based Budgeting (PBB), with a parallel set of The most fundamental objectives and strategies for public service delivery. ‘Performance budgeting seeks to tool for improving link the funding provided to government agencies to the results they deliver in order to increase the efficiency and effectiveness of public expenditure. For this purpose, it Agent performance is makes systematic use of performance information in the budgeting/funding process’ to focus on results – (Robinson 2007, 15). Performance information at its core constitutes information on results achieved and on the costs of achieving those results. explicit specification of objectives, the PBB does not necessarily disburse funds after the delivery of results. Instead, PBB focuses on achieving expenditure efficiency by strengthening the link between measurement of funding and results, thereby strengthening the credibility of target setting itself. performance against The most fundamental tool for improving Agent performance is to focus on results – explicit specification of objectives, the measurement of performance against those those objectives and the objectives and the setting of performance targets – and to link those results to budget setting of performance allocation and/or disbursement (Robinson 2007). targets – and to link PBB takes on various forms that vary by the primary Results-Based Management those results to budget objective, the degree to which the funding is linked to results and the level of performance information required. In his seminal publication on Performance-Based allocation and/or Budgeting, Robinson (2007) categorised three forms of PBB reforms: disbursement. • Funding-linked performance targets • Agency-level budget performance incentives • Purchaser-Provider model These three forms of PBB can be thought of as differentiated types of RBF implementation, where early-stage interventions start with linking funding to results and more advanced-stage RBFs make entire intervention funding contingent on the verification of achieved results. The type of RBF that can be designed and implemented should be based on: • Whether the identified problem in the given context can be solved with modifications in financing arrangements; • The level of data availability and utilisation in education planning and implementation decision making; and • The level of control and autonomy that meso-level service delivery Agents have over decisions on activities that yield results. 40 In examining the funding disbursement arrangements in the 51 meso-level RBF initiatives, we were able to categorise 46 initiatives under the three PBB categories.20 Table 3 summarises the data and financial context, along with the benefits of each type. Table 3. Types of RBF arrangements Existing Form of PBB Number of Availability and Operational Fund Type Arrangement for Potential Benefits Initiative Example Initiatives Use of Results Arrangements the RBF Data 1 Funding-linked 10 Low The Agent requires Institutionalisation of Support to Upper Secondary performance targets operational funds to focus on results Reform in Brazil Operation: deliver the desired Secretariats of education at results Moderate efficiency and the state level are provided effectiveness gains additional funding by the national education ministry Better utilisation of of basic education based on data on results by the an agreement and plan to Principal achieve results. 2 Agency-level budget 21 Moderate Operational funds Increased effectiveness Transforming Secondary performance available to deliver of activities Education for Results incentives the desired results Operation in Bangladesh: Better utilisation of The national education data on results by both ministry gives schools the Principal and the a conditional grant Agents based on the following results achieved in the previous year: minimum accountability (like a functional SMC/PTA, EMIS data submission, social audit) and student body performance targets (like student attendance, retention, learning outcomes). 3 Purchaser-Provider 15 High Alternate Increased efficiency and The Employment Fund in model mechanisms effectiveness Nepal: Using donor funding, available for securing Helvetas, a development operational funds to Increased transparency organisation, pays training deliver the desired and use of performance and employment service results information providers for every beneficiary that completes skills training and is gainfully employed or establishes an enterprise. PBB reforms are part of the larger umbrella of Results-Based Management21 reforms, and countries as diverse as Australia, Belarus, Brazil, Colombia, Ethiopia, France, India, South Africa, United Kingdom, United States and Vietnam have started implementing it within their national budget systems (Robinson 2007). 20 The categorisation of individual initiatives and their key features are provided in Appendix 3. 21 ‘RBM is a broad management strategy aimed at changing the way institutions operate by improving performance, programmatic focus and delivery. It reflects the way an organisation applies processes and resources to undertake interventions to achieve desired results. RBM is a participatory and team- based management approach to programme planning that focuses on performance and achieving results and impacts. It is designed to improve programme delivery and strengthen management effectiveness, efficiency and accountability. RBM helps in moving the focus of programming, managing and decision-making from inputs and processes to the results to be achieved’ (7). UNESCO. (2015). Results-Based Programming, Management, Monitoring and Reporting (RBM) approach as applied at UNESCO. Guiding Principles. BSP/RBM/2008/1.REV.6. Paris. 41 PBB models and approaches can also provide some foundational and design lessons for implementing RBF within public education management and programming as well. In Vietnam, PBB reforms have provided foundational support for the World Bank’s funding to flow to the line ministries for health and education through the annual budget cycle with a focus on results. The rest of this section explains the operationalisation of the three forms of PBB into types of RBF for the meso-level of education. 7.1 Type 1: Funding-Linked Performance Targets In the funding-linked performance targets form of PBB, fund disbursement does not need to be made after the achievement of results. This form of PBB primarily links the level, but not the disbursement, of funding to specific quantitatively measurable result targets (output or outcome) that the Agent is expected to deliver. The target- setting exercise can aim at improving either or both the effectiveness or efficiency of the Agent’s work; effectiveness is improved where the result targets are outcome measurements and efficiency is improved where the result targets are outputs of the Agent’s work. It is often implemented in the form of ‘contracts’ between ministries and agencies, between civil service managers and their staff, or between ministries/ agencies and chief executives, where the Principal pays the Agent on the basis of the costs incurred by the Agent to deliver results. The fundamental priority here is the use of performance information for financing prioritisation (Robinson 2007). This form of performance contracts or agreements with funding linked to output The fundamental and outcomes results was evident in 10 interventions (see Table 5 and Appendix 3). In the India Odhisa Higher Education Program, the Odhisa state department of priority here is the education provided funds to higher education institutions to implement institutional use of performance improvement results with the long-term aim of improving the quality of higher education in the state. In the India Strengthening Teaching-Learning and Results for information for States, the central government line ministry provided funds to the departments of financing education at the state level to undertake activities to achieve the following outputs: provide reading and numeracy training to teachers, participate in PISA 2021, prioritisation. implement learning enhancement programmes, establish teacher performance evaluation and introduce career guidance programmes for secondary school students. Similarly, in the Brazil Support to Upper Secondary Reform, the national line ministry gave funds to the state authorities to adopt the new Upper Secondary curriculum and increase the number of schools with a full-time schedule, all with the aim of achieving better outcomes in the form of reduced dropout and repetition and improved learning outcomes. Several conditional school grants (Cameroon, Madagascar, Senegal) also made payments based on the production of school development plans to achieve results. RBF designs wherein financing is linked to performance targets can be utilised in contexts that are in the early stages of adopting a focus on results in education sector financing, i.e. there is low availability and use of performance data. This can be especially useful in contexts where the Agents require operational funds to undertake activities needed to achieve the results. The interview respondents in Brazil, Cameroon, India and Senegal indicated that while the RBF intervention did not disburse funds after the verification of results achieved, it still aligned the fund recipient agency (Agent) with the goals of the fund disbursement authority (Principal). All respondents believed that once the stakeholders become more familiar with the operations of RBF approaches, future interventions could make payments contingent on achieving results. 42 Box 4: Odhisa Higher Education Program for Excellence and Equity The project aims to improve the quality of education and students’ equitable access to Higher Education Institutions through institutional development grants for Higher Education Institutions (HEI) to focus on quality-improving initiatives. The HEI competed for the grants based on their institutional development plan, assessed against the Grant Operations Manual by an Evaluation Committee, which chose 140 HEIs. Eligible activities for the Institutional Development Grant include efforts on assisting affiliated colleges in achieving autonomous status, curriculum delivery innovations, teacher training, improving of the examination system, establishing of Resource Support Centre for Affiliated Colleges in tribal- dominated districts, modernisation of university and infrastructure support. The selected HEIs received two types of payments. Initial payment to the HEI was made based on the approval of the Institutional Development Plan. Subsequent tranches of payments were made every year based on the HEI demonstrating 60% spending of the previous year’s disbursement and with all selected HEIs showing progress on the result targets collectively. 7.2 Type 2: Agency-Level Budget Performance Incentives In this form of PBB, the Agent receives payment after the verification of past performance. The primary feature of agency-level budget performance incentives is to introduce financial rewards and sanctions to motivate Agents to improve performance or achieve results – ‘mechanisms whereby strongly performing programmes or agencies receive additional budget funding, and poor performers lose money’ (Robinson 2007, 10). This is essentially a performance bonus incentive structure used at the organisational level rather than at the individual level. Based on their project design documents, we were able to categorise 21 initiatives under this category. The aim of the financial incentive to the Agent is to motivate it to improve the effectiveness of activities to achieve results. This form of PBB is aligned with the RBF structures of payment disbursement after the achievement of results. Within the initiatives we examined in low- and middle-income countries, this approach was also used in higher education in Eastern Africa, Southern Africa and The primary feature of India. Some conditional school grants were also contingent on schools achieving agency-level budget certain results. In the Bangladesh Transforming Secondary Education for Results performance incentives Operation and Nepal School Sector Development Program, school grants were contingent on improvements in school management and student performance like is to introduce financial retention rates and learning outcomes. Several interventions further introduced rewards and sanctions competition between schools by awarding the grants only to a certain number or percentage of top-performing schools (some examples are the Jakarta School Grants to motivate Agents to Program in Indonesia, Sri Lanka General Education Modernization Project, Ethiopia improve performance or General Education Quality Improvement Program for Equity, Tanzania Education Program for Results and Zanzibar Improving Student Prospects Project). achieve results. Agency-level budget performance incentives were also provided to sub-national education departments to motivate the Agents to improve their support to schools and teachers in the region. In Benin, regional and sub-regional inspectorates were incentivised to improve pedagogical support to teachers and student enrolment in primary and secondary schools while decreasing repetition and dropout rates. In Brazil’s Ceará state, pay-for-performance tax-revenue transfers to the municipal government were used to reward or sanction Agents for improving primary school progression and learning outcomes. As a result of this incentivisation, Ceará’s municipalities have achieved some of the highest educational outcomes in the 43 country with very low per-pupil expenditures (Loureiro et al. 2020). FCDO piloted an RBA intervention in Ethiopia, where regional education bureaus received additional funds if they increased the numbers of secondary school exam sitters and passers compared to those in the previous year (Cambridge Education 2015). This model can be implemented in contexts where the Agent does not necessarily need additional operational funds to undertake the activity but can utilise its existing funds more effectively. In turn, improved results then increase funds available in the future. Box 5: RBF for Ceará, Brazil In 2007, the state of Ceará pioneered the use of RBF in Brazil as part of a comprehensive education reform programme that aimed at having all students in public schools reading by the end of grade 2. In this arrangement, the state government of Ceará, the Principal, conditioned the tax revenue quota transfers to municipalities, the Agents, to the achievement of established education goals. In its original design, the reform focused on improvements in literacy rates to bring the municipalities with poor education outcomes up to par with national standards. Municipalities with greater improvements in literacy rates, pass rates and learning scores received a higher fraction of the tax revenue transfer than municipalities with lower overall performance. However, if improvements in learning scores were accompanied by restrictions to the participation of students in learning assessments or by widening disparities between high and low performers, the municipalities were sanctioned with a lower tax revenue transfer. As a complement to the RBF arrangement, the state provided technical assistance to all municipalities for achieving the agreed education goals (e.g. training and knowledge exchange between schools). In 2012, as the quality of education improved across the state, the RBF incentive formula was partially reformulated. In its new design, the formula gave more weight to the levels of performance in literacy and attainment and increased rewards for municipalities seeing improvements in learning outcomes for students who were lagging behind. Despite having one of the lowest GDP per capita among Brazilian states, Ceará was able to substantially improve education outcomes without increasing public spending on education. Nearly all of its 184 municipalities departed from very low levels of student learning and progression to become one of the highest performers in the national quality index, including the municipality of Sobral, which had the highest score (Loureiro et al. 2020). A recent evaluation comparing test scores in schools located in municipalities at the border between Ceará and adjacent states in Brazil showed that the RBF incentive increased student performance in Portuguese and mathematics by approximately three months’ learning equivalent for 9th grade. When the municipalities received both RBF and technical assistance, increases to learning outcomes were equivalent to around three months’ learning for both grades 5 and 9 (Lautharde, De Oliveira, and Loureiro 2021). The increased funding can take the form of ‘bonus’ funding to supplement the core funding of the Agent. The application of performance incentives would also require robust systems for agency performance data, where both the Principal and the Agent can utilise the data to measure progress towards the results. 7.3 Type 3: Purchaser-Provider Model In the purchaser-provider model of PBB, the Principal pays the Agent a price per result achieved, which can either be output- or outcome-type results. This form of PBB goes beyond the incentive payments described above as it treats the Agent as a ‘standalone business’, and the Agent needs to find an alternate initial source to finance operational costs (Robinson 2007). The Agent makes a profit or loss, reflecting the difference between what it cost the Agent to achieve the results and what the Principal paid for the result. Since the Agent does not receive funds upfront to pay for the activities, the use of this PBB approach in RBF can theoretically maximise the 44 efficient use of fund goal for the Principals since they do not spend funds on activities that do not yield results. The purchaser-provider model follows a similar approach to formula funding, which has been used in school budget allocation for decades. The funding to the Agent is based on some form of unit cost formula, like average cost x quantity = total cost; however, the unit is meant to be the actual results rather than an input, such as the number of expected students served.22 This is intended to create pressure on the Agent to deliver results to get funding and improve efficiency with a strong profit/ loss incentive. ‘This applies the usual practice of the market, where businesses generally do not get paid for products they do not supply. It contrasts with the usual public sector budgeting and funding practice that agencies receive funds which are essentially an expenditure allowance—there is no requirement to return funds if output and/or outcome expectations are not met’ (Robinson 2007, 13). This model also gives the Agent autonomy over how to spend the funds. The use of the purchaser-provider model in the interventions examined appears to be in the early stages, where nearly all of the 15 initiatives contracted non-governmental organisations as Agents. The treatment of a non-governmental organisation as a ‘business’ is potentially easier than treating a public sector organisation as, in most cases, their financial management includes alternate revenue sources to fund activities. In the Employment Fund in Nepal and SKY project in Ethiopia, private and public training providers were paid on a per-trainee basis to deliver skills training and sustained employment to beneficiaries. In the Vietnam Upper Secondary Education Enhancement Project, non-governmental secondary schools received tuition subsidies as a unit cost of each student maintaining a minimum level of attendance and grade point average. The Sindh Province in Pakistan contracted non-governmental organisations to provide school management services. The Agent received payments for results in improved management practices and some student performance outcomes like attendance, retention, health and hygiene practices. The Payment-by-Results funding modality of All Children Reading Begin with Books Prize and the Girls Education Challenge can also be considered purchaser-provider models wherein various non-governmental organisations were paid on a per-outcome basis for delivering learning materials and learning outcomes. 22 In cases where increase in enrolment is the expected result, the unit cost can be established based on the student enrolment number. 45 Box 6: Sindh Secondary Education Improvement Project The School Education and Literacy Department (SELD) of Sindh Province in Pakistan contracted private Education Management Organisations (EMOs) to operate schools with the aim of improving inclusiveness in secondary education. SELD entered into a Concession Agreement with the EMOs. The EMOs invested in new and existing school infrastructure and were responsible for the operation of the schools during the period of the contract. The government retained ownership of the school lands and buildings while the EMOs were responsible for all aspects of school operations, including delivery of the curriculum, human resource management and management and maintenance of school facilities. The concession agreement lasted for 10 years. The management fee paid to the EMOs was meant to be reimbursed for budgeted costs, as stipulated and agreed upon in their financial proposals. Payments to the EMOs were made quarterly and were contingent on them meeting the short-term Key Performance Indicators (KPIs). These KPIs covered a range of areas, including school management, teacher attendance, student enrolment, curriculum delivery, staff professional development, student assessment, maintenance of school facilities, community engagement, and health and hygiene. Each KPI was accompanied by measurable progress indicators. The attainment of KPIs was assessed quarterly by experts hired by the EMOs to independently monitor and evaluate performance. The quarterly management fee payable to the EMOs was adjusted based on assessed performance. For example, EMOs whose quarterly performance was assessed at 95% were paid only 95% of the quarterly management fee. Payments to EMOs were only made once they were authorised by an independent auditor based on quarterly reports received from experts hired by the EMOs. The contracts of EMOs that do not meet performance targets were designed to be terminated. Admission to EMO schools was strictly non-discriminatory, and the EMOs were not allowed to charge any students any fee. Schools managed by the EMOs were required to deliver the Sindh curriculum, and students were assessed through the same public examination boards as students in regular government schools. The teaching staff was a mixture of government and contractual teachers hired directly by the EMOs. Government teachers were paid significantly more than those hired directly by EMOs. The EMOs could not terminate government teachers, but the SELD could transfer non-performing government teachers to other schools once their non-performance was confirmed by the EMOs’ independent experts. Impact Bonds are a special case of the purchaser-provider approach. While the service provider receives upfront capital to undertake activities, it is the investor, not the outcome payer (Principal), who experiences a financial loss if the results are not achieved. In India, two impact bonds have been used to improve learning outcomes; in Argentina, an impact bond incentivised employment for the youth. In Argentina’s Proyectá tu Futuro, estimation of costs for the outcomes was a complicated task for the government as previous data on activities did not exist (Acosta and Esper 2021). Without prior reliable data, it was difficult for the government to determine whether the costs negotiated with the service provider were the most efficient use of funds or the best value for money. 46 Box 7: India Education Outcomes Fund: Rate Card for Education Outcomes In order to decrease the costs associated with Impact Bond outcomes cost negotiations, the India Education Outcomes Fund (IEOF) aims to establish a Rate Card for educational outcomes. After the IEOF administers multiple impact bonds, it will consolidate the experience, evidence and data to develop a standardised pricing mechanism in the form of a ‘rate card’, which will specify a standard ‘price per outcome’. This will reduce the costs of determining what the outcomes would cost every time procurement is conducted. The Rate Card will create transparency for pricing outcomes across service providers, risk investors and the outcome payer (Mehendale and Singh 2020). Previous attempts to implement the purchaser-provider model for public agencies have not succeeded even in high-income countries because in government agencies, the relationship between results (outputs or outcomes) and costs is too uncertain (Robinson 2007). Additionally, alternative provision of upfront financing for activities will be required given that public sector agencies are, in most contexts, entirely dependent on budget transfers from the higher levels of the government for operational funds. In Ethiopia’s SKY project, where both public and private training institutes were engaged, some institutes were given an advance against the contract to finance their upfront costs (Zanola et al. 2021). This arrangement of ‘advancing’ funds from the purchaser-provider agreement has been utilised in the US, New Zealand and Australia for several years, where some type of operational or working capital fund is established for the Agent to use a portion of allocated funds. Once the results are achieved, the Agent reports a profit or a loss on the basis of costs of results achieved (Robinson 2007). While the differentiated-type approach to RBF presented here does not solve all of the challenges in designing and implementing RBF interventions at the meso-level of education, it provides a way to take the context into account to deliberately design modifications in the RBF arrangement while maintaining a focus on results through financing. 47 RBF ’S WIDER DEFINITION, WHICH LINKS FUNDING TO THE DELIVERY OF RESULTS BY GOVERNMENT AGENCIES OR OTHER EDUCATION SERVICE PROVIDERS, IS A USEFUL STARTING POINT FOR ASSESSING UNDER WHICH CONDITIONS CAN FINANCING ARRANGEMENTS IMPROVE EDUCATION OUTCOMES. 48 8 Recommendations for Designing RBF at the Meso-Level of Education Systems E xpanding the definition of RBF to a set of wider reforms, which link funding to the delivery of results by government agencies or other education service providers, is a useful starting point for assessing under which conditions can financing arrangements improve education outcomes. This section expands the learning a step further and makes specific recommendations to guide those seeking to design RBF interventions at the meso-level of education systems. Recommendation 1: Conduct a situation analysis to understand any existing experience with RBF and determine whether there is a culture conducive to results-based management and if there are educational challenges that RBF can help solve. An understanding of the existing management of the education system is an essential prerequisite for designing an RBF arrangement. A situation analysis will help the designer consider the level of results-based management culture that already exists within the education system and help choose what sort of RBF approach to use. It is essential to remember that complex RBF mechanisms may be unsuccessful in contexts that lack necessary pre-conditions (especially operational funds and data management systems). If the education system has an existing culture of results-based management and some previous experience with RBF, it is possible to introduce agency-level budget performance incentives (Type 2), whereas contexts where the linking financing to results is new concept may benefit from the gradual introduction of funding-linked performance targets (Type 1). 49 Some questions to consider for a situation analysis: • Is there any prior experience with RBF in education and/or in other sectors? • What are the existing reporting and management systems for the target meso-level Agent? • How are the activities of the meso-level Agent currently financed? What is its minimum level of required operational capital? • How flexible are existing staff compensation schemes? Situation • Which education results indicators in planning and implementation, if any, are currently in use and Analysis by which actors? • What data systems exist that do or could be used to measure results and how independent and objective are they? • What is the willingness of potential Principals to cede sufficient operational autonomy to potential Agents? • Within the existing system, which stakeholders hold meso-level actors to account? • If independent verification is intended, what institutions have or potentially have the capacity to do it? REACH grants in various countries were used essentially to understand – or test the potential of – the current situation. In Bangladesh and Nepal, for instance, they tested whether establishing information systems and providing data could lead meso-level decision makers (district education officers and school directors) to make decisions on the basis of results. In Colombia, the ‘School Monitor’ system established data about six dimensions of educational quality for individual schools based on inputs from various stakeholders. Conversely, where no in-depth situation analysis was carried out, as in Senegal, progress has been very slow. Even seven years after the initial RBF scheme was introduced, no finances flow on the basis of results, although a results orientation is slowly being established. Recommendation 2: Agree on a shared theory of change with all In Colombia, the stakeholders, especially Principals and Agents, that will also ensure sufficient autonomy for Agents to achieve ‘School Monitor’ system results. established data about A common vision among all stakeholders on how RBF could motivate meso-level six dimensions of Agents to achieve results should provide widespread support for the intervention. educational quality When actors are involved in co-constructing the intervention design and accepting the causal logic of the intervention, they are more likely to own the process. for individual schools based on inputs from A theory of change for an RBF intervention should establish the logical sequencing of how the performance agreement and the linking of payments to results would result various stakeholders. in the Agent achieving the specified outputs and outcomes. This theory of change Conversely, where no process is different from the typical results framework development process in most donor-funded projects, where specified activities are linked to expected outputs situation analysis and outcomes. By contrast, in an RBF arrangement, the Agent has the flexibility to was carried out, as in choose the activities that will reach the desired output. A theory of change (and the corresponding results framework) for RBF should stipulate the causal mechanism by Senegal, progress has which outputs and outcomes will be achieved through the financing arrangement been very slow. with the Agent rather than through specified activities. 50 An important and often forgotten aspect of having an agreed theory of change is to ensure that the Agent has sufficient autonomy to achieve the agreed result. This means that the Agent must be free to implement its programmes as it thinks best. Yet many Agents lack sufficient autonomy to do this. A sufficient level of decision- making autonomy is not a given in decentralised systems, and the level of actual autonomy on various aspects of education management should be examined. In many countries, for example, schools do not have the autonomy to hire, much less fire, teachers. In others, the compulsory curriculum is so overloaded that it may not be possible to devote sufficient time to achieve the agreed results. Those in regional and district offices may be used to receiving orders from Principals at the upper levels of line ministries, and these Principals may be accustomed to giving such orders. This will have to change if RBF is to work; policy exceptions may be needed to bequeath Agents with sufficient autonomy and old attitudes may need to be revised. Some questions to consider during the RBF theory of change building process: • What is the identified problem that the meso-level RBF intervention could solve? • Is the identified problem related to the efficient use of financing, ineffective functioning of service delivery or challenges with the accountability of actors? • What are the expected results or changes envisioned? If the funder and the Principal are different actors, do they both agree on the expected results? • How would the introduction of RBF impact equity? Would any special design considerations be needed to mitigate any negative impact on equity? • What modifications in Agent effort or work are needed to achieve the expected results? • Is there an agreement among all stakeholders (including the Agent) that the Agent controls the delivery of the results? Theory of Change • Would the Agent need to undertake an entirely new set of activities to achieve the results or could it modify its current activities? • What are the various Principal-Agent relationships that affect the functioning of the primary meso- level Agent in the RBF setup? • Is the Principal clearly identifiable by all, especially the Agent? Who is meant to hold the Agent accountable and how? For example, for school grants, is it clear to the school director who is the decision maker on whether the performance targets are eligible for grants? Is it the central Ministry or its sub-national offices like District Education Offices? • Do the following actors agree on the theory of change: Principal, Agent, Agent employees, intermediaries, independent verifiers and external funders (if there are any)? • Is the RBF approach more cost effective than input-based financing? If so, why? • What level of autonomy can the Agent have in planning the activities? The establishment of a clear and common understanding among all stakeholders is not very prevalent in donor-funded projects when the Agent of the RBF may not interact with the donor organisation. Majority of the negotiations take place between the donor and the national government, which may or may not be the Principal within the RBF. Since project design often takes place during donor-government negotiations, other stakeholders are not engaged. For example, in the Brazil Support to Upper Secondary Reform and India Odhisa Higher Education Program, the national governments and the state governments were much clearer on what was supposed to happen than were the meso-level Agents. Interestingly, impact bonds, which are inherently more complicated than performance-based budgeting, typically had a broader stakeholder understanding than did decentralised public education systems, possibly because their instigators were very aware of the novelty and complexity of 51 impact bonds. In South Africa, this was the result of a long process that began with all stakeholders reading the same materials about such bonds, and then was continued with the University of Cape Town training provincial government officials at the same time as two intermediaries (Mothers to Mothers and Volta Capital) trained and strengthened the Agent. Though not an impact bond, a similar initial process was followed for the USAID All Children Reading Challenge. A key vehicle for laying out the theory of change explicitly or implicitly is the performance contract between the Principal and the Agent, a process supported in Morocco by a REACH grant to pilot contracts between the Ministry of Education’s national headquarters and its regional offices. Box 8: Creation of competition between Agents may exacerbate equity challenges When actors compete for the same funds, the RBF scheme may assist in weeding out weaker actors who can be replaced by better-performing ones (Bruns, Filmer, and Patrinos 2011). However, re-allocating limited education budget funds towards Agents who are already achieving high-quality results could end up limiting funds to support Agents who may be dealing with harder-to-solve problems or are unable to control some circumstances that impact the achievement of results. A central issue in using RBF for public education is the balance between rewarding the Agent and maintaining commitments to universal education for students. If a district management performs poorly under RBF, such that its students learn less than those in another district, for example, it will not get paid or not get fully paid. Thus, in the future, there will be even less funding for the already relatively poorest-performing children. In some situations, this can be tackled by rewarding the value added (e.g. in test scores) instead of or in addition to rewarding the achievement of a particular level of performance. The case of Ceará, Brazil is a positive example of how this can be tackled at the stage of designing an incentive structure. While the state tax revenues are redistributed based on municipal competition on education outcomes, greater rewards are not only given to municipalities achieving the highest levels in the education quality index, but also to those with the highest improvements in the index. The amount received by each municipality is defined by a formula with clear indicators for learning outcomes and progression rates. The state government created an education quality index that considers the level and improvements in literacy at grade 2, learning at grade 5 (reading and mathematics), and the average progression rates in primary education (grades 1 to 5). Greater rewards are given to municipalities with the highest levels and gains. The system also penalises municipalities that increase inequality among schools or try to reduce the take-up rate of low-performing students in exams. When the incentive was created, the formula designated more weight to improvements in literacy to stimulate municipalities with poor education outcomes. As the quality of education improved in the state, the formula was changed to give more weight to the levels of literacy at grade 2 and attainment at grade 5. A further issue with equity is that to maximise the incentive payment, the Agent may prioritise results that are easy to achieve. When services to a certain number of beneficiaries are used as the measure of results achieved, the hard-to-reach populations are often left out. Some RBF projects (Ethiopia Pilot Project of Results Based Aid, Ethiopia SKY, Nepal Employment Fund and East Africa) have resolved this by including greater incentives for the Agent to improve services to underserved groups such as women and students from lower socioeconomic status. In Ceará, Brazil, the state government penalises municipalities that increase inequality among schools or try to reduce the take-up rate of low-performing students in exams. 52 Recommendation 3: Define result targets that can be transparently and objectively measured. Defining the expected result is the most essential part of any RBF intervention wherein financing needs to be linked to achieved results. At the most fundamental level, it helps make the Principal’s objectives explicit to the Agent and facilitates alignment between them. Subsequently, it also makes the RBF intervention visible to the entire education system and facilitates the alignment of goals among all stakeholders in the education value chain – from the education policy level down to the classroom level. The establishment of appropriate results must be driven as much as possible by the objective measurement of data in order to offset the information asymmetries between Principals and Agents, the latter likely having a better understanding of what is feasible on the ground and so more able to negotiate lower targets than are Principals able to negotiate higher ones. Objective data measurement does not necessarily mean quantitative data measurement. Depending on the expected results, some qualitative assessments will also need to be included. A strong result data system is not only beneficial to the Principals but also to the Agents in assessing their own progress towards achieving the results. Some questions to consider while developing results targets and measurement plans: • Will the expected results comprise processes, outputs or outcomes or a combination? What are the benefits of this design? • Are appropriate objective indicators available for all these results? Results Targets • If more than one result is to be achieved, is the weighting of the financial incentives for each result appropriate or does it excessively reward results that are the easiest to achieve? • Can the results be measured and reported through existing data management systems? Do these systems need to be strengthened. If so, at what cost? • Is external verification needed? If so, by whom could it be conducted and at what cost? The appropriate initial levels for targets may also depend on how long the RBF has been in place. In some cases, initial expectations have been set quite low in order to ensure success and build support for future more significant RBF interventions. The initial use of process indicators, for example, can support the adoption of foundational activities to establish a system focused on results and consensus building about the theory of change. This was the case in Cameroon, for example, where schools and SMCs had to take only a few simple steps to establish accountability systems and get the first grant, subsequently followed by more specific improvements in school management. Target setting also has to be done in conjunction with the establishment of incentives and the financial weight given to each target. In Cameroon, again, schools could achieve an adequate result by showing improvement on some, but not necessarily all, performance indicators. In South Africa, no payment was ever made against the learning target as it was never achieved, in part because it represented a very small portion of the total financial incentive. Getting upfront agreement on how results will be verified is also necessary. Governance concerns and fear of potential gaming have almost always led external donors to insist upon there being an independent verification of the results by an organisation that is neither the Principal nor the Agent. Such an independent 53 verification adds to the costs of an intervention. Independent verification may not be strictly necessary, however, especially in cases where national or education statistics offices are independent and competent and their work is respected and accepted by both Principals and Agents. The cost effectiveness of result measurement systems should be considered in a systematic manner. While cumbersome supervision and data verification activities may add to the cost of the initial RBF intervention, systematic integration of the process within the education system may prove cost effective. For example, in Benin, the Directorate of Pedagogical Inspection and Innovation within the Ministry of Education is responsible for collecting and verifying the data on results achieved to release payments. In such cases, however, some randomised verification is probably still useful to ensure that there is no joint gaming of the system or collusion between Agents and Principals. Recommendation 4: Design an incentive structure that cascades financial and non-financial incentives at both organisational and individual levels. ‘Fit naturally with As discussed in Section 2, incentives play a central role in an RBF arrangement in motivating the Agents to deliver results effectively and efficiently. When RBF targets this approach to meso-level actors, it needs to incorporate incentive structures that motivate not only motivating individual the Agent organisation as a whole but also its individual employees. performance – at least if Agency-level budget performance incentives (Type 2) ‘fit naturally with this approach to motivating individual performance—at least if there is a link between agency- there is a link between level funding rewards and individual incentives’ (Paul and Robinson 2007, 331). In agency-level funding purchaser-provider arrangements (Type 3), the Agent organisation theoretically has full autonomy to design and implement an incentive system for its employees. Even rewards and individual in simpler types of RBF arrangements where payments are linked to budgeting rather incentives’. than to the achievement of results (Type 1), the Agent organisation budget should incorporate incentives for work teams and individuals. Incentives that are purely financial may introduce perverse motivations and work contrary to the non-financial motivation of public employees in terms of providing a public service. Furthermore, in some contexts where results (especially outcomes) are less measurable or are not directly caused by Agents’ actions, incentives for Agent effort can and should be considered, as this could encourage changes in Agent behaviours that will lead to the desired outcomes in the long run. Some questions to consider when designing the incentive structure: • Is there an agreement by all stakeholders (both actors engaged in the design and those who are not) on why the Agent will be motivated to perform? • What are the primary incentives for the Agent to perform? Are these financial or non-financial Incentive incentives or both? Are they compatible with each other? Do the financial incentives have any Structure negative implications in terms of PSM? • What are the conceived consequences of non-performance? Where would unused funds be allocated if they are not fully disbursed assuming that not all results are achieved? • How can incentives be modified to motivate non-performing Agents to improve in the future? 54 Some experiments with group incentives have shown positive effects at the school level. In Uganda, an intervention saw improvements in teacher attendance when teacher absence was collectively reported by school directors and parents and the school directors received a bonus payment (Cilliers et al. 2014). In India, incentives at the group level for teachers led to similar gains in student learning as did incentives for individual teachers (Muralidharan and Sundararaman 2011). Several interventions combine financial and non-financial incentives for meso- level Agent organisations and their employees. In the Tanzania’s Improving Student Prospects Project, the Government of Zanzibar provided funding for in-service training to teachers along with a higher salary for finishing training and a recognition award for performance. In the Ethiopia General Education Quality Improvement Program for Equity, 10% of the schools that show the highest year-to-year gains in early grade enrolment ratios and survival-to-grade-5 rates receive incentive pay and public recognition. In the Brazil Support to Upper Secondary Reform project, the interviewees considered non-financial motivation as a critical aspect of the programme. Implementation of the Upper Secondary Education reform process by the state not only triggers payments from the national government but also the supply of textbooks for the new curriculum. Moreover, it allows for participation in the national upper secondary level assessment. There was also an expectation that parents and communities would pressure the state authorities to implement the national reform as not being part of it could leave students disadvantaged (in terms of university entrance exams, entering the job market, etc.). Recommendation 5: Provide capacity building support and technical assistance to foster a culture of results orientation. Shifting from input-based to RBF requires fostering a culture of result orientation for activity planning and implementation. The enhancement of this culture should be done by building the capacity of various actors to collect and utilise results data and the provision of finance and technical support to strengthen data management 55 systems. In the Ethiopia Pilot Project of Results Based Aid, where regional bureaus received payments for improving the numbers sitting and passing secondary school exams, there was no capacity in the regional bureaus to administer a reward- based system despite it being a prerequisite for success. Support to enhance result orientation is especially needed in contexts with little or no previous experience with RBF or other results-based management approaches. Some questions to consider in planning capacity building and technical assistance for data management: • Is there a robust Education Management Information System that collects and analyses disaggregated data on a regular basis? • What is the capacity and existing skillset of the Principal-Agency to monitor and assess results? Capacity Building • What additional capacity is needed to prepare data management systems to be used for monitoring and Technical Assistance and assessing the performance of various departments and offices of the education system? • What data management and utilisation processes does the Agent use? • Are there any discussions of other interventions/projects that would strengthen data systems? If so, how can the RBF intervention interact with these projects? • What additional investments are needed by whom to strengthen the data management system and the capacity of the various actors and at what cost? Fifteen of the 24 projects subjected to in-depth document review included capacity building or technical assistance activities explicitly related to improving in data management and utilisation for the RBF component. In the Brazil Support to Upper Secondary Reform project, the entire focus of implementing an RBF programme was to strengthen the central and local governments’ capacity to use data for decision making and to reorient planning and monitoring tools away from inputs to outputs In the ACE programmes, and outcomes. there were some The India TEQUIP and West and Central Africa ACE programmes each have the unallocated funds same unique feature of having capacity building within the RBF incentive payment within each country. structure itself. In the TEQUIP project, poorly performing engineering institutions received reduced funding but were mentored intensively to improve their future During project performance. In the ACE programmes, there were some unallocated funds within implementation, these each country. During project implementation, these unallocated funds were intended to be assigned either as an additional reward (beyond the initial incentive) to high- unallocated funds were performing institutions or to institutions that needed additional unforeseen support intended to be assigned for activities that were critical to achieving the results. either as an additional Recommendation 6: reward (beyond the Ensure sustainability with a plan for moving through the initial incentive) to differentiated RBF types and phasing out external funding over time. institutions RBF interventions can fall into one of three types, as we have seen. Those in type 1 or to institutions that (Funding-linked performance targets) and type 2 (Agency-level budget performance incentives) could, over time, progress to type 3 (Purchaser-Provider model). While needed additional experience with each type will strengthen both RBF awareness and institutions unforeseen support (Principals and Agents), it will be helpful to have a plan from the beginning about subsequent moves to the next type. This is because experience so far shows for activities that were that lack of such planning, rather than just an implicit expectation, will occur. In critical to achieving the particular, while there has been a general (if implicit) expectation that Agency-level budget performance incentives or the Purchaser-Provider model will eventually results. 56 be implemented, there has been no thought given to timing or to the need to first pass through the funding-linked performance targets. Indeed, hitherto, it seems most interventions that are type 1 (funding-linked performance targets) and type 2 (Agency-level budget performance incentives) have not been distinguished as different approaches to RBF. Some questions to consider in developing the sustainability plan: • What are the longer-term objectives of the RBF initiative? • What is the time required to observe the desired results (output or outcome)? • Is it feasible to observe results within the project funding period? Sustainability • Does the project sustainability plan consider long-term sustainability of the education outcomes or the sustainability of the results-based management and financing approach? • What activities are needed to sustain or institutionalise RBF? • If starting with funding-linked performance targets (type 1), could measures be taken to design Agency-level budget performance incentives or a purchaser-provider model in subsequent projects? • Could the RBF arrangement be financed by the government after the initial donor funding? In addition to the progression through different types of RBF, broader sustainability has also been relatively ignored. This is especially important when donor or other external time-bound funding is used for results financing and/or for capacity building of institutions and data systems. In the case of the former, government financing should progressively replace external financing. Meanwhile, for the latter, it will be important to have a plan for institutions that have reached appropriate capacity before the external funding ends. More nebulous but very important is given that most donor-funded projects also originated with donors, it is necessary to ensure that there is full ownership of RBF among all the domestic stakeholders before the donor funding ends. The sustainability of RBF within the education budget was considered in the project design of Benin, where financial simulation models in education planning included the inclusion of the RBF payments to sub-regional inspectorates within the expected future increase in the education budget. Similarly, in the Ethiopia General Education Quality Improvement Program for Equity, conditional school grants were also partially funded by the national education budget to ensure the sustainability of the RBF beyond the duration of the World Bank-funded project. 57 RBF INTERVENTION DESIGNS SHOULD RESPOND TO THE CONTEXTUAL REALITIES TO ENSURE ACCEPTANCE OF THE APPROACH BY ALL STAKEHOLDERS AND MINIMISE ANY UNINTENDED NEGATIVE CONSEQUENCES. 58 9 Looking Ahead N o theoretical framework can propose a ‘one-size- fits-all’ type of intervention; contextual realities will always need to be factored into the design. A broad framework that takes as many contextual variations as possible into consideration and outlines the specific pre- conditions needed for successful implementation could lead to an intervention design that responds to contextual realities while preserving the intent of the intervention. The differentiated types of RBF provide a starting point for designing interventions that link financing to results while considering the pre-existing conditions within the education system. Financing modalities alone, including RBF, cannot solve the challenges faced in education service delivery and will need to be considered in conjunction with other reforms. However, given the popularity of RBF in development, it is critical to design interventions that respond to the contextual realities in order to ensure acceptance of the approach by all the stakeholders and minimise the unintended negative consequences. There remains a need for additional research focusing on RBF arrangements at Additional intervention the meso-level of the education sector in low- and middle-income countries. evaluations should Additional intervention evaluations should investigate whether the causal mechanism of linking financing to results leads to improved outcomes. It is also investigate whether important to conduct in-depth comparative case studies to further improve our the causal mechanism understanding of the conditions under which RBF is likely to be a successful financing modality for the meso-level of education. of linking financing to results leads to Finally, research findings should be used to inform and engage in dialogues with key education stakeholders to develop a shared theory of change and improved outcomes. intervention design. 59 References Acosta, Felicitas, and Tomas Esper. 2021. “Proyecta Tu Futuro: A Social Impact Bond for Education and Employability Training in Buenos Aires.” NORRAG Working Paper #15, NORRAG, Geneva. Aizawa, Toshiaki. 2019. “Impacts of the Community Block Grant Programme on School Resources, Environment and Management in Indonesia.” Education Economics 27 (5): 521–45. doi:10.1080/09645292.2019.1641587. Araneda-Guirriman, Carmen A., and Liliana M. Pedraja-Rejas. 2016. “Performance-Based Funding In Chile: Theoretical Framework Analysis For Financial Instrument In Higher Education Institutions.” Formación Universitaria 9 (3): 75–86. http://dx.doi.org/10.4067/S0718-50062016000300009. Araneda-Guirriman, Carmen A., Joaquín Gairín-Sallán, and Liliana M. Pedraja-Rejas. 2018. “Autonomy In Higher Education: Reflections From The Actors In The Context Of Performance-Based Funding In Chile.” Formación Universitaria 11 (4): 65–74. http://dx.doi.org/10.4067/S0718- 50062018000400065. Barrera-Osorio, Felipe, and Dhushyanth Raju. 2017. “Teacher Performance Pay: Experimental Evidence From Pakistan.” Journal of Public Economics 148: 75–91. doi:10.1016/j.jpubeco.2017.02.001. Barrera-Osorio, Felipe. 2007. “The Impact of Private Provision of Public Education: Empirical Evidence from Bogota’s Concession Schools.” World Bank Policy Research Working Paper No. 4121, The World Bank, Washington, D.C. https://ssrn.com/abstract=960184 Behrman, Jere R., Susan W. Parker, Petra E. Todd, and Kenneth I. Wolpin. 2015. “Aligning Learning Incentives of Students And Teachers: Results From A Social Experiment In Mexican High Schools.” Journal of Political Economy 123 (2): 325–64. doi:10.1086/675910. Birdsall, Nancy, and William D Savedoff. 2010. Cash On Delivery: A New Approach To Foreign Aid. With An Application To Primary Schooling. Washington, D.C.: Center For Global Development. Blimpo, Moussa P., David K. Evans, and Nathalie Lahire. 2015. “Parental Human Capital And Effective School Management: Evidence From The Gambia.” Policy Research Working Papers. The World Bank. doi:10.1596/1813-9450-7238. Bolman, Lee G., and Terrence E. Deal. 2011. Reframing Organizations: Artistry, Choice And Leadership. 4th ed. San Francisco, CA: Jossey-Bass. Breeding, Mary, Tara Béteille, and David K. Evans. (2021). “Teacher Pay-for-Performance: What Works? Where? And How?” The World Bank, Washington, D.C. https://documents1.worldbank.org/curated/en/183331619587678000/pdf/Teacher-Pay-for-Performance-What-Works-Where- and-How.pdf Bruns, Barbara, Deon Filmer, and Harry Anthony Patrinos. 2011. Making Schools Work: New Evidence On Accountability Reforms. Washington, D.C.: World Bank. Cambridge Education. 2015. Evaluation of the Pilot Project of Results-Based Aid in the Education Sector in Ethiopia. Final Report EC 2004-2006. Cambridge Education. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/576168/Eval-pilot- project-result-based-aid-Education-Ethiopia.pdf Carneiro, Pedro, Oswald Koussihouèdé, Nathalie Lahire, Costas Meghir, and Corina Mommaerts. 2019. “School Grants And Education Quality: Experimental Evidence From Senegal.” Economica 87 (345): 28–51. Doi:10.1111/ecca.12302. Cilliers, Jacobus, Ibrahim Kasiryeb, Clare Leaverc, Pieter Serneelsd, and Andrew Zeitline. 2014. “Improving Teacher Attendance Using a Locally Managed Monitoring Scheme: Evidence from Ugandan Primary Schools.” Policy Note 14/0189. International Growth Centre. https://www.theigc. org/wp-content/uploads/2014/09/Cilliers-Et-Al-2014-Policy-Brief.pdf Clist, Paul. 2019. “Payment By Results In International Development: Evidence From The First Decade.” Development Policy Review 37 (6): 719–34. Doi:10.1111/dpr.12405. Clist, Paul, and Arjan Verschoor. 2014. “The conceptual basis of payment by results.” London: DFID. https://www.gov.uk/research-for-development- outputs/the-conceptual-basis-of-payment-by-results Correa Miranda, Adelaida, Geraldine García Medina, and Gabriel Ugarte Vera. 2017. “Evaluation of Educational Incentives Program in Peru ‘Performance Commitments’ ‘Compromisos de Desempeño’.” Dixit, Avinash. 2002. “Incentives and Organizations in the Public Sector: An Interpretative Review.” The Journal of Human Resources 37 (4): 696–727. doi:10.2307/3069614. Ferraro, Fabrizio, Jeffrey Pfeffer, and Robert I. Sutton. 2005. “Economics Language And Assumptions: How Theories Can Become Self- Fulfilling.” Academy of Management Review 30 (1): 8–24. doi:10.5465/amr.2005.15281412. 60 Gailmard, Sean. 2010. “Politics, Principal–Agent Problems, and Public Service Motivation.” International Public Management Journal 13 (1): 35–45. doi:10.1080/10967490903547225. Ganimian, Alejandro J., and Richard J. Murnane. 2016. “Improving Education in Developing Countries: Lessons From Rigorous Impact Evaluations.” Review of Educational Research 86 (3): 719–55. doi:10.3102/0034654315627499. Goff, Peter, Ellen Goldring, and Melissa Canney. “The Best Laid Plans: Pay for Performance Incentive Programs for School Leaders.” Journal of Education Finance 42 (2): 127–52. http://www.jstor.org/stable/44162579. Global Partnership for Education (GPE). 2020. An Early Stage Review of Country Programmes Designs and Implementation Experiences with GPE’s Variable Part Financing Mechanism (2015–2019). Washington, D.C.: Global Partnership for Education. https://www.globalpartnership.org/sites/ default/files/document/file/2020-05-review-of-design-and-implementation-experiences-of-gpe-variable-part-financing.pdf Gustafsson-Wright, Emily, and Izzy Boggid-Jones. 2020. “Financing for Impact: Why pay for outcomes in Education?” NORRAG Special Issue 04: New Philanthropy and the Disruption of Global Education: 130–33. Geneva: NORRAG. https://resources.norrag.org/resource/592/new-philanthropy- and-the-disruption-of-global-education Gustafsson-Wright, Emily, and Izzy Boggild-Jones. 2019. Paying for Education Outcomes at Scale in India. Center for Universal Education at the Brookings Institution. https://www.brookings.edu/wp-content/uploads/2019/11/Paying-for-education-outcomes-at-scale-in-India-FINAL-FOR- WEB.pdf Honig, Daniel. 2015. “Navigating by Judgment: Organizational Structure, Autonomy, and Country Context in Delivering Foreign Aid.” Doctoral dissertation, Harvard University, Graduate School of Arts & Sciences. https://dash.harvard.edu/handle/1/17467366 Independent Evaluation Group (IEG). 2019. Implementation Completion Report Review: Second Sindh Education Sector Project. ICRR0021769. World Bank. https://documents1.worldbank.org/curated/en/641021576777403518/text/Pakistan-PK-Second-Sindh-Education-Sector-Project.txt International Commission on Financing Global Education Opportunity. 2016. The Learning Generation: Investing in Education for a Changing World. International Commission on Financing Global Education Opportunity. https://report.educationcommission.org/wp-content/uploads/2016/09/ Learning_Generation_Full_Report.pdf Instiglio. (2018). A guide for effective results-based financing strategies. The Global Partnership on Output-Based Aid. Washington, DC: The World Bank. https://documents1.worldbank.org/curated/en/265691542095967793/pdf/A-Guide-For-Effective-Results-Based-Financing-Strategies.pdf Knott, Jack H. 1993. “Comparing Public And Private Management: Cooperative Effort And Principal-Agent Relationships.” Journal of Public Administration Research and Theory 3 (1): 93–119. doi:10.1093/oxfordjournals.jpart.a037164. Lautharte, Ildo, Victor Hugo de Oliveira, André Loureiro, 2021. “Incentives for Mayors to Improve Learning: Evidence from State Reforms in Ceará, Brazil.” Policy Research Working Paper 9509. Education Global Practice, World Bank Group. http://hdl.handle.net/10986/35024 Lee, Jessica D., and Octavio Medina. 2019. Results-Based Financing in Education: Learning from What Works. Results in Education for All Children (REACH). Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/31250 Leruth, Luc, and Elisabeth Paul. (2006). “A principal-agent theory approach to public expenditure management systems in developing countries.” IMF Working Paper WP/06/204. International Monetary Fund. https://www.imf.org/external/pubs/ft/wp/2006/wp06204.pdf Loraque, Julia. 2018. “Development Impact Bonds: Bringing Innovation to Education Development Financing and Delivery.” Childhood Education 94 (4): 64–8. doi:10.1080/00094056.2018.1494454. Loureiro, André, Louisee Cruz, Ildo Lautharte, and David K. Evans. (2020). The State of Ceará in Brazil is a Role Model for Reducing Learning Poverty. World Bank Group Education. http://hdl.handle.net/10986/34156 Mani, Subha, Kumar Vivek, and Pradyumna Bhattacharjee (n.d.). Impact Evaluation of Training and Incentives for Middle Management in Education. Mehendale, Archana, and Ajay Singh. 2020. “India Education Outcomes Fund. A Case Study.” Working Paper #12. Geneva: NORRAG. https:// resources.norrag.org/resource/view/606/362 McGinn, Noel F., and Thomas Welsh. 1999. Decentralization of education: Why, when, what and how? Fundamentals of Educational Planning, 64. Paris: UNESCO. https://unesdoc.unesco.org/ark:/48223/pf0000120275/PDF/120275eng.pdf.multi Montecinos, Carmen, Luis Ahumada, Sergio Galdames, Fabián Campos, María Verónica Leiva 2015. “Targets, threats and (dis) trust: The managerial troika for public school principals in Chile.” Education Policy Analysis Archives, 23, 87. doi.org/10.14507/epaa.v23.2083 61 Mungaray, Alejandro, Marco Tulio Ocegueda, Patricia Moctezuma, and Juan Manuel Ocegueda. 2016. “La Calidad De Las Universidades Públicas Estatales De México Después De 13 Años De Subsidios Extraordinarios.” Revista De La Educación Superior 45 (177): 67–93. doi:10.1016/j. resu.2016.01.008. Muralidharan, Karthik, and Venkatesh Sundararaman. 2011. “Teacher Performance Pay: Experimental Evidence From India.” Journal of Political Economy 119 (1): 39–77. doi:10.1086/659655. Naidoo, Jordan and Sylvia Montoya. 2015. “A Data Revolution for Education 2030.” World Education Blog (blog), November 2, 2015. GEM Report. https://es.unesco.org/gem-report/data-revolution-education-2030 Organisation for Economic Cooperation and Development (OECD). 2005. “Modernising Government: The Way Forward.” Paris: OECD. https://doi. org/10.1787/9789264010505-en Olken, Benjamin A., Junko Onishi, and Susan Wong. 2014. “Should Aid Reward Performance? Evidence From A Field Experiment On Health And Education In Indonesia.” American Economic Journal: Applied Economics 6 (4): 1–34. doi:10.1257/app.6.4.1. Renmans, Dimitri, Elisabeth Paul, and Bruno Dujardin. 2016. “Analysing Performance-based Financing through the Lenses of the Principal-agent Theory.” Working Paper / 2016.14. Institute of Development Policy and Management (IOB). Antwerp: Universiteit Antwerpen. https://ideas.repec. org/p/iob/wpaper/201614.html Results for Development Institute (2016). Literature Review: Situating Output-based Aid in the Context of Results-based Financing in Education. Global Partnership on Output-Based Aid (GPOBA). https://r4d.org/resources/literature-review-situating-output-based-aid-context-results-based- financing-education/ Robinson, Marc. 2007. Performance Budgeting: Linking Funding and Results. London: Palgrave Macmillan, London. https://doi. org/10.5089/9780230553569.071 Romero, Mauricio, and Justin Sandefur. 2019. “Beyond short-term learning gains: The impact of outsourcing schools in Liberia after three years.” Working Paper 521. Center for Global Development. https://www.cgdev.org/sites/default/files/beyond-short-term-learning-gains-impact- outsourcing-schools-liberia-after-three-years.pdf Paul, Elisabeth, and Marc Robins. 2007. “Performance Budgeting, Motivation, and Incentives.” In Performance Budgeting: Linking Funding and Results, edited by Marc Robinson, 330–75. London: Palgrave Macmillan. Al-Samarrai, Samer, and Blane Lewis. 2021. The Role of Intergovernmental Fiscal Transfers in Improving Education Outcomes. International Development in Focus. Washington, D.C.: World Bank. doi:10.1596/978-1-4648-1693-2. Savedoff, William D. 2010. Basic Economics of Results-Based Financing in Health. Results Based Financing for Health. Bath: Social Insight. United Nations Educational, Scientific and Cultural Organization (UNESCO). 2015. Pricing the right to education: The cost of reaching new targets by 2030. Policy Paper 18. Education for All Global Monitoring Report. https://unesdoc.unesco.org/ark:/48223/pf0000232197_eng UNESCO. 2018. Walk before you run: the challenges of results-based payments in aid to education. Policy Paper 33. Global Education Monitoring Report. https://unesdoc.unesco.org/ark:/48223/pf0000261149_eng World Bank. 2020. Restructuring paper on a proposed programmes restructuring of transforming secondary education for results operation. Report No: RES37791. Education Global Practice South Asia Region. World Bank. Zanola, Lea, Markus Maurer, Chalachew Gebeyehu, Sabrina Würmli, and Bettina Jenny. 2021. “Helvetas’s Skills and Knowledge for Youth Project: A Case Study of Results-Based Financing for Vocational Education and Training.” Working Paper #14. Geneva: NORRAG. https://resources.norrag. org/resource/view/636/367 62 Appendices Appendix 1: Description of RBF at the Meso-Level in Literature As shown in Figure 4, since 2010, there has been a growing number of publications focusing on RBF at the meso-level. The health sector, in particular, had led in terms of the number of publications in most of the period, except for 2015 and 2016, when education publications spiked. More recently, publications that do not focus on a particular sector but instead discuss RBF at the meso-level more generally have also grown significantly, demonstrating a peak in 2020. This reflects how RBF at the meso- level has been gaining prominence in research beyond sector-related experiences. This greater prominence of RBF at the meso-level is also evident in publications focusing on the education sector. Since 2014, an increasing number of research articles with this focus have been published, with spikes in the number of publications in 2016 and 2019. Considering the whole period for which literature in all sectors was collected (2010–2021), publications focusing on health represented 38%, followed by education at 29% and by research that does not focus on a particular sector at 16%. Figure 4. Number of publications with a focus on meso-level RBF per sector across time (2010–2021)23 80 70 60 50 40 30 20 10 0 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 (until june) Health Management Public Administration Social Services Education General Environment Others The literature on meso-level RBF is predominantly focused on high-income countries. Out of the 37624 research articles analysed that discussed RBF at the meso-level, the majority (52%) focused on high-income countries, 35% focused on low- and middle- income countries, and 13% did not have a particular focus. Figure 5 shows how this predominant focus on high-income countries was consistent across the analysed period and that publications that focused on high-income contexts, together 23 For better visualisation of the distribution of publications per sector, sectors that had only one or two publications in the period (2010–2021) were grouped into an Others category. Those were justice, employability, infrastructure, finance, mining and manufacturing. 24 This figure includes publications from 2010 to June 2021 on RBF at the meso-level in all sectors. However, when looking at the education sector in particular, 32 additional publications from 1973– 2009 were included in the analysis, for a total of 407 publications analysed. Those are represented in the descriptive patterns of RBF initiatives and in the analytical sections of this study. 63 with those that did not have a particular income-level focus, drove the overall growth of publications on RBF at the meso-level. Figure 5. Number of publications focusing on RBF at the meso-level by countries’ income levels (2010–2021) 80 70 60 50 40 30 20 10 0 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 (until june) Focus on low-and-middle income countries Focus on high income countries No particular focus 64 Appendix 2: Profile of RBF Initiatives at the Meso-Level of Education System Initiatives Empirical research on specific initiatives represented 74% of the literature collected within the 2010 to June 2021 period on RBF at the meso-level of the education system. The profile of the low- and middle-income country initiatives was more geographically diverse than that of the high-income countries, including countries in Latin America (Chile, Colombia and Mexico), South Asia (India and Pakistan), East Asia & the Pacific (Indonesia) and Sub-Saharan Africa (Tanzania). Figure 6 shows the geographical diversity of these RBF experiences. The Sub-Saharan Africa region is not only the one with the highest number of RBF initiatives (36%) but also the one with the highest number of countries that used RBF at the meso-level, followed closely by South Asia, which accounts for 31%. It is also in South Asia that the individual countries with the highest number of RBF experiences at the meso- level of education, namely, India and Pakistan, are located. Figure 6. Geographical distribution of RBF initiatives at the meso-level of the education system in low- and middle-income countries Multiple Countries Central Asia Europe & Uzbekistan Colombia South Asia East Asia & Latin America & Argentina Caribbean Peru Mexico Chile Brazil Indonesia Pacific Vietnam Sri Lanka Bangladesh Nepal Pakistan India South Africa Senegal Nigeria Malawi Madagascar Liberia Sub-Saharan Africa East and Southern Africa East Africa Democratic Republic of the Congo Cameroon Burundi Benin Tanzania Ethiopia 0 1 2 3 4 5 6 7 8 9 65 Moreover, some diversity can be observed when it comes to the funders of such initiatives. As displayed in Table 4 below, although 36 of the 51 RBF initiatives were funded by either the World Bank or governments (which entails national or sub- national government bodies), the remaining ones are largely distributed among the funders. Other funders include bilateral and multilateral organisations, non- governmental organisations and philanthropies25. Table 4. Number of RBF interventions at the meso-level of the education system per funder in low- and middle-income countries Number of Funder Initiatives World Bank 24 Government Bodies 12 GPE 5 Non-identified donors or philanthropies 3 Cordaid 2 Global Partnership for Results-Based Approaches (GPRBA) 2 Results in Education for All Children (REACH) Programmes 2 United Kingdom Foreign, Commonwealth and Development Office (FCDO) 2 ApexHi Charitable Trust 1 Asian Development Bank (ADB) 1 British Asian Trust 1 Canadian International Development Agency (CIDA) 1 Children’s Investment Fund Foundation 1 Finland Ministry for Foreign Affairs 1 Helvetas 1 Michael and Susan Dell Foundation 1 Swiss Agency for Development and Cooperation 1 UNICEF 1 United States Agency for International Development (USAID) 1 World Vision 1 While the overall literature on RBF at the meso-level of the education system focuses predominantly on higher education, this is not the case in the 51 initiatives identified in low- and middle-income countries. As shown in Figure 7, the primary and secondary education levels are the most common, jointly accounting for 73% of the initiatives. Those levels are also frequently targeted together on a single initiative. Early childhood development, higher education and vocational skills development are mainly targeted in isolation and not in combination with other educational levels. Figure 7. Distribution of RBF initiatives at the meso-level of the education sector in low- and middle-income countries by educational level 7% 11% Early Childhood Higher Development 9% Vocational 41% Primary 24% Secondary 25 The predominance of the World Bank is, in part, due to this study’s reliance on World Bank and REACH documents; the study also conducted extensive research on other databases as outlined in Section 3, but did not identify any large number of initiatives in further projects. 66 This analysis also identified which stakeholders played the role of Principals and Agents in meso-level RBF initiatives in education. As displayed in Figure 8, the majority of the Principals are actors at the national level (e.g. National/Federal Government, Ministry of Education, etc.). While not the majority, actors at the regional, state, provincial or departmental levels also frequently play the role of Principal, disbursing the payment upon the Agent’s achievement of the expected results. An ‘Other’ category groups philanthropic, national and international non- governmental organisations. Few of the Principals are at the international level (only 8%), as domestic Principals predominate. Figure 8. Distribution of Principals in RBF initiatives at the meso-level of the education sector in low- and middle-income countries 13% 8% District/ International Municipality 2% Other 48% National 24% Region/State/ Province/Departments When it comes to Agents at the meso-level, the most commonly incentivised are those that are the closest to and able to make the biggest impact on the desired results. As presented in Figure 9, nearly one-third of the initiatives have Agents at the school level, followed by other service providers (20%) and tertiary education institutions (20%). Thus, education providers are the Agents in 69% of the initiatives, with another 23% being sub-national governments at the regional, state, province, district and municipality levels. In addition, 6% of the agents belong in an ‘investors’ category, representing impact bond investors, incentivised by the outcome payer, who, at the same time, incentivise the service provider to achieve the expected results, sometimes using RBF arrangements themselves. Figure 9. Distribution of Agents in RBF initiatives at the meso-level of the education sector in low- and middle-income countries 6% Investor 2% 13% Other District/ Municipality 10% 29% Region/State/ School Province 29% 20% Service Provider Tertiary Education Institution 67 68 Appendix 3: Meso-Level RBF Interventions in Education Systems Table 5: Key features of meso-level RBF interventions in education systems Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Funding-linked performance targets Brazil Support to National Ministry of 2018–2023 Upper Performance-based The Secretariat of Basic The State Outputs: State Secretariats of Upper Secondary Re- Education secondary agreement for in- Education (SEB) under Secretaries (i) adopt the new high education in states are form in Brazil Operation tergovernmental the of Education school curriculum under provided additional funding transfers Ministry of (SEE) of each the Novo Ensino Médio by the Brazil National Ministry Education (MEL) state in Brazil reform and of Basic Education upon (ii) promote the expansion reaching an agreement and of full-time schools plan to achieve results Expected outcomes: (i) Decrease in dropout rate, (ii) Decrease in repetition rate and (iii) Increase in learning outcomes. Cameroon Cameroon Education World Bank 2018–2026 Primary schools Conditional school Ministry of Basic Educa- Schools Expected results: im- The Ministry of Basic Reform Support Project grants tion (MINEDUB) proved teacher atten- Education (MINEDUB) dance, lesson preparation, awards grants that range school retention, budget from US$500–US$1500 to transparency, textbook primary schools that develop utilisation, communities’ approved plans to meet the satisfaction and imple- targets mentation of the fee-free education policy. Chile Programa de Convenios World Bank and From Higher Education Performance-based Ministry of Education State Results are unique to each Universities sign a para el Desempeño Ministry of Education 1999 contracts Universities contract but they normally performance agreement with (Performance Agreement (Chile) involve teacher training, the national government programmes) curriculum harmonisation, that makes the fund internationalisation of transfer contingent on the doctoral programmes, achievement of agreed results, international academic as per their institutional exchange and mobility, development plan professional training and/ or innovations. Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Democratic Project d’Amélioration GPE 2017–2021 Primary Conditional school Ministry of Primary, 1,350 schools Expected results include Payments to schools are Republic of de la Qualité d’Éduca- education grants Secondary and Tertiary the creation of school linked to predefined quality the Congo tion Education development plans, and quantity performance improved primary student indicators learning, increased girl enrolment and overall school performance. India Odisha Higher Education World Bank 2018–2023 Higher education Performance-based Program Management Higher The results to be achieved The HED pays HEIs their first Program (OdhisaHEP) Institutional grants Unit (PMU) housed in Education by the HEIs to receive grant if they develop an the Higher Education Institution yearly IDG funding fall institutional development Department (HED) (HEI) under the following plan that meets certain types categories: equitable of activities, as well as a access, teaching-learning management plan to achieve environment, quality, civil results works and equipment, governance, and management. Madagascar Madagascar Basic Ministry of Education 2018–2023 Primary schools Conditional school Ministry of Education Primary Submit improvement Madagascar’s Ministry of Education Support grants Schools plans, progressively Education provides funding Project modify the academic in the form of school grants calendar to begin in March to primary schools within 10 instead of November Subdistrict Education Offices by 2022 and improve graduation rates by 2% since the 2016/17 academic year. Mexico Programa Integral Federal Government Since Higher education Performance-based Federal Government Public State Indicators associated State universities receive de Fortalecimiento of Mexico 2011 contracts Universities with enrolment, quality competitive performance- Integral (PIFI) (Integral and accreditation of based subsidies from the Programs of Institutional degree programmes, federal government (Public Improvement) quality of learning and Education Secretary) based on professionalisation of the achievements of targets teaching staff. set in their institutional improvement plans (mainly quality of education and equity in degree programmes) 69 70 Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Senegal Quality Improvement World Bank, GPE, 2014-2021 Primary and Performance based There are 3 Principals: There are 4 Incentivised results for System-wide PBCs between a and Equity of Basic CIDA Secondary contracts and (i) Regional educational Agents: (i) district level offices: higher level to a lower level of Education Conditional school authorities, (ii) Ministry Regional (i) improvement of the education management grants of Education and (iii) Authorities, district management, system. district authorities (ii) District (ii) improvement of District Authorities are given Authorities, the quality of support a portion of the payment at (iii) Schools to schools and (iii) the signing of the PBC, then and (iv) improvement of additional disbursements are Daaras attendance in schools. made based on the The incentivised results achievement of the evaluation for schools were based targets. on the signing off quality Schools are currently paid improvement agreements. regardless of performance but The incentivised results the aim for the next phase is to for Daaras were (i) allocate funds in two parts: (i) implementation of in September upon approval specific curriculum and (ii) of PBC and (ii) in February proficiency in reading and after the mid-year evaluation. maths. The most-improved schools will receive top-up rewards, while the bottom 20% of schools will receive specialised attention in the following year. Tanzania Education and Skills for World Bank 2017–2021 Higher Performance-based Tanzania Education Public and (i) Training proposals The Tanzania Education Productive Jobs education; tech- Institutional grants Authority private that focus on results and Authority provides a one-off nical and training (ii) agreed targets like payment to the training vocational institutions pass percentage rate, institutes upon achievement education; and post-placement support of the first result and a top-up alternative and feedback from grant for achieving the second training. employers. result Vietnam Enhancing Teacher The World Bank 2016–2022 Higher education Results-linked bud- Ministry of Education Teacher Improvement in Teacher The MOE pays universities if Education Program and the Ministry of get transfers Training Education Institution they achieve expected results. (ETEP) Finance Universities Development Index and number of teachers and principals satisfied with professional development. Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Agency-level budget performance incentives Bangladesh Transforming Secondary World Bank 2018–2022 Secondary Conditional school Ministry of Education School Minimum accountability The MOE pays US$5,000 in Education for Results education grant requirements (such as year 2 and US$4,000 in year 4 Operation functional SMC/PTA, of the project to the schools if annual social audit, EMIS they achieve results data submission) and performance targets (such as student attendance, retention, student learning outcomes and achievement in Secondary and Higher Secondary School Certificate (SSC/ HSC)). Benin Benin Global Partnership Global Partnership 2019–2023 Primary and Performance-based DDEMP (Direction Regional and There are 3 incentivised The DDEMP pays a maximum for Education project for Education lower secondary contract départementale des sub-regional results: (i) inspectors’ of US$15,000 to each regional phase 3 enseignement maternel inspectorate impact on the overall inspectorate in a given school et primaire) and DIPIQ quality improvement year if they achieve at least (Direction de at the school level, 75% of the results Innovation including teachers’ Pédagogique et de la pedagogical support, Qualité) following inspectors’ and educational advisors’ class visits; (ii) incentives for school enrolment, especially for girls in communes where the GERs deviate too much from the national average; and (iii) repetition and dropout rate reduction in both Primary and Lower Secondary Education. 71 72 Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Brazil RBF mechanism in Ceará The state govern- Since 2008 Primary and Pay for performance The state government The In the beginning, the The state government of ment of Ceara secondary Intergovernmental of Ceara municipality project focused on Ceara gives greater rewards Transfers government improvements in literacy to municipalities with the rates to bring the highest levels of education municipalities up to par quality index and gains. The with national standards. government also penalises As the quality of education municipalities that increase improved, the government inequality among schools or increased the weight try to reduce the take-up rate given to improvements of low-performing students in the levels of literacy in in exams. grade 2 and attainment in language and mathematics in grade 5. Child mortality and construction of solid waste disposal were also considered. Chile Performance-based Municipalities in From Primary and Performance-based Municipal governments Public school Improvements in School principals sign a contract under the 2011 Chile 2011 secondary contracts principals enrolment, attendance 5-year performance-based Ley de Calidad y Equidad education and learning assessment contract with the municipal de la Educación (Quality scores. governments (principal) and Equity in Education specifying sanctions and Act) incentives for meeting predefined results (improving results on assessments of students’ learning, enrolment and attendance) East Africa Eastern Africa Skills for World Bank (IBRD/ 2018-2024 Higher education Performance-based Respective TVET Increasing access, The respective governments Transformation and IDA) Institutional grants governments in institutes improving quality, and provide funds to TVET Regional Integration Ethiopia, Kenya, and improving regional institutes if they achieve Project (Ethiopia, Kenya, Tanzania integration. incentivised results. Some Tanzania and Inter- funds are disbursed annually University council for while others are one-time East-Africa) payments. Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure East and Eastern and Southern World Bank (IDA) 2016–2022 Higher education Performance-based Respective Ministries of African Improve research capacity The respective ministries Southern Africa Higher Education Institutional grants Education in Burundi, Centres of and development impact, provide a maximum funding Africa Centres of Excellence Ethiopia, Kenya, Excellence improve quality, improve of US$6 million to individual Project Malawi, Mozambique (ACE) financial management. ACEs if they achieve the Rwanda, Tanzania, incentivised results. Uganda, Zambia and Some funds are disbursed Zimbabwe annually, while others are one-time payments. Ethiopia Ethiopia General World Bank (IBRD/ 2017–2023 Primary Conditional school The Planning and All primary Two main results: (i) grade The PRMD of the MOE gives Education Quality IDA), DFID, Finland education grants Resource Mobilization schools 2 to grade 1 enrolment annual grants of 30,000 ETB Improvement Program Ministry for Foreign Directorate (PRMD) ratio and (ii) survival rate (=US$682.70) to the top 10 for Equity Affairs, UNICEF of the Ministry of to grade 5 percent of primary schools Education that score highest in year-to- year improvements in the two indicators. Ethiopia Pilot Project of Results FCDO 2012–2016 Lower secondary Reward payments Ministry of Education Regional Increase in the number of The Ministry of Education Based Aid in the education Education sitters and passers of the gave reward payments to the Education Sector in Bureaus Ethiopian General Second- Regional Education Bureaus, Ethiopia ary Education Certificate half of which is based on the Examination (EGSECE) number of additional sitters within each region com- and passers of the EGSECE pared to the previous year. and half on the total number of sitters and passers India Skills Strengthening World Bank 2017–2022 Higher education Performance-based State and Central Two agents: Results for ITIs: Incentive structure for ITIs: for Industrial Value grants government (i) Industrial Selected agreed targets The state governments pay Enhancement Operation Training in individual industrial the ITIs performance grants if (STRIVE) Institutes strategic plans they achieve results. (ITIs) and Results for ICs: Incentive structure for (ii) Industry Increased enrolment Industry Clusters: The Clusters (ICs) of women and government awards grants to other currently ICs when they introduce the underrepresented groups, apprenticeship programme such as Schedule Tribes. and achieve results 73 74 Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure India Technical Education IDA 2016–2020 Higher education Performance- Ministry of Human There are 3 Incentivised results (i) MHRD provides funds to Quality Improvement based grants and Resource Development agents: (i) for participating participating institutes if they Project (TEQIP) performance-based (MHRD) Participating institutes in focus perform well. Poor-performing contracts institutes in states: improvements institutes will be mentored focus states, in equity and quality intensively but will receive (ii) Affiliated Incentivised results reduced funding from the Technical for affiliated technical project if they fail to show Universities universities: improvement serious efforts to improve. (ATUs) and in equity and quality (ii) MHRD provides up to US$3 (iii) High- of all colleges affiliated million each to ATUs if their performing to them Incentivised performance targets are met. engineering results for high- For the above two agents, institutes performing engineering annual (or semi-annual) institutions: performance performance benchmarks in twinning agreements are to be met for successive with participating rounds of funding to be institutes. The focus released. of these agreements (iii) The MHRD provides an will be knowledge initial funding of US$300,000 transfer, exchange of each to high-performing experience, resource engineering institutions usage optimisation and to participate in twinning development of long-term activities, and then up to strategic partnerships. US$1.1 million based on their performance India Strengthening Teaching- World Bank 2021–2026 Preschool, Performance-based Ministry of Human State The results include, The MHRD provides funding Learning and Results for primary and intergovernmental Resource Development government among other goals: (i) to six selected states to States secondary transfer (MHRD) training in early reading undertake activities to achieve and numeracy to 75% the agreed results teachers in selected states; (ii) participation in PISA 2021; (iii) implementation of learning enhancement programmes; (iv) establishment of teacher performance evaluation; and (v) implementation of a career guidance programme reaching at least 40% of secondary education students. Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Indonesia Jakarta School Grants Government of Launched Primary and Conditional school Government of Jakarta Primary and Improved learning The Government of Jakarta Program Jakarta 2014 lower secondary grants junior schools outcomes seen through gave the best-performing education examination performance. schools (top 25%) an additional per student allocation equivalent to 20% of the basic grant allocation. The ranking exercise was conducted separately in each district to incentivise more schools and make the scheme more equitable. Indonesia Program National Government of Launched Primary and Performance-based Federal Government Villagers- Targeted indicators in Villages receive an annual Pemberdayaan Indonesia, World in 2007 secondary grants municipalities education included block grant to improve 12 Masyarakat – Generasi Bank, and donor education (i) primary education maternal and child health and Sehat dan Cerdas countries’ support enrolment, (ii) monthly education (school enrolment (National Community primary attendance equal and attendance) indicators. Empowerment Program to or above 85%, (iii) The size of the subsequent – middle school enrolment, year’s block grant depends on Healthy and Smart and (iv) monthly middle performance relative to other Generation), also known school attendance equal villages in the subdistrict. as Generasi program to or above 85%. Malawi Malawi Education Sector GPE 2017–2021 Primary schools Conditional school Ministry of Education Primary Improve promotion and The Ministry of Education Improvement Project grants school retention rates of stu- transfers funds to school (MESIP) principals dents, particularly girls. principals who achieve results in the form of performance- based grants. Nepal School Sector World Bank 2016–2021 Primary and Conditional school The District Education Schools Expected accountability The government provides Development Program secondary grants Office results include the use funding to community of EMIS and improved schools if they achieve the time spent on teaching. expected results in the Expected performance form of performance-based results, among others, grants. Further, the Ministry include improved of Education provides extra attendance and funding to community schools retention rates, textbook that offer science streams availability and results in within their curriculum in standardised exams. grades 11 and 12 in the form of pro-science enhancement grants. 75 76 Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Nigeria Better Education Service World Bank 2018–2022 Primary schools Performance-based Federal Ministry of States Reduction in the number The Federal Ministry of Delivery for All (BESDA) intergovernmental Education Universal of out-of-school children Education disburses transfers Basic in primary education, funding to SUBEBs upon the Education particularly girls and achievement of results in the Boards pastoralist students. form of federal grants. (SUBEBs) Peru The Performance Peru Ministry of 2014–2016 Primary and Performance-based Ministry of Education Executing Incentivised results related The Performance Commitments Program Education secondary intergovernmental Units (ExU) to school management, Financing Unit under the transfer attendance and Secretariat of Strategic improvement of learning Planning of the Ministry of achievements (e.g. Education of Peru provides timely hiring of teachers, additional funding to timely distribution of decentralised Educational school materials and Management Units, known improvement of learning as ExU, conditional to the achievements in Spanish achievement of 20 targets. and Math). Sri Lanka General Education World Bank 2019–2024 Primary and Conditional school Ministry of Education Primary and There are two separate Sri Lanka Ministry of Modernization Project secondary grants secondary incentivised results: Education provides funding schools (i) improved learning to up to 500 primary and outcomes in foundational secondary schools that language skills (reading, achieve the first expected vocabulary and writing) result. Furthermore, the MOE and mathematics; and (ii) provides funding to schools pre-established teacher that achieved the second professional development expected result through a goals. school-based professional teacher development grant. Tanzania Zanzibar Improving World Bank 2017–2022 Upper primary Conditional school Government of Upper Improvement in student The government of Zanzibar Student Prospects and secondary grants Zanzibar primary and performance in Maths, provides top-up grants to the Project secondary Science and English. 100 best-performing primary schools and 100 best-performing secondary schools. Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Tanzania Education Program World Bank 2018–2020 Primary and Performance-based MoEST Local Govern- Schools having a Financing is released for Results (Additional lower secondary intergovernmental ment Author- pupil–teacher ratio to LGAs in response to Financing) transfers ities within the acceptable schools moving into the Ministry of Education range (35–50:1). acceptable PTR range, with Science and incentives for improvement Technology at both understaffed and overstaffed schools. Conditional School MoEST 400 schools Improved outcomes Monetary rewards from Grants (300 in National the MoEST to the 300 primary; 100 Assessments most-improved primary secondary). (Primary School and 100 most-improved Leaving Examination lower secondary schools and Certificate of in students’ national Secondary Education examination performances. Examination). Performance-Based MoEST Local Year-on-year increase The MoEST rewards the Intergovernmental Government in aggregate primary LGAs that show the greatest Transfers Authorities and lower secondary improvements in year-on- survival rates and year average primary and achieving year-on- lower secondary survival year increase in girls’ rates. transition rate from Primary Standard 7 to Secondary Form 1. 77 78 Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure West and First Africa Higher World Bank 2019–2023 Higher education Performance-based Respective governments in (i) New ACE There are seven The new ACE impact centres Central Education Centers of institutional grants Burkina Faso, Republic of impact cen- funded results receive between US$4 Africa Excellence Project Guinea, Republic of Djibouti, tres and (ii) 6 to be achieved: million and US$8 million Republic of Ghana and ACEs already (i) institutional upon achieving the funded Republic of Senegal included in the readiness; (ii) results. The 6 ACEs already project development impact included in the project also of the ACE impact receive funding based on centre; (iii) quantity achieving the same set of of students with a funded results. However, focus on gender and the amounts given to each regionalisation; (iv) centre varies. quality of education and research through international accreditation, research publication and improved teaching and research infrastructure; (v) relevance of education and research through externally generated revenue internships and entrepreneurship timeliness and quality of fiduciary management; (vi) timeliness and quality of fiduciary management; and (vii) institutional impact to be accomplished by the ACE host institution. Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Purchaser-Provider Argentina Proyectá Tu Futuro Government of the 2018–2021 Secondary Social Outcome payer: Investors: There are four incentivised Five investors provide upfront City of Buenos Aires, schools impact Government of Bank of the results: (i) support for capital for the implementation IMF bond the City of Buenos City Buenos secondary education of interventions by four Aires Aires; Bank completion; (ii) placement service providers. The of Galicia; in formal employment; (iii) Government of the City of IRSA Local employment sustained Buenos Aires repays the Comercial; for 4 months; and (iv) investor once results have Exotrade S.A. employment sustained for been achieved. (Organización 12 months. Román); Jorge Luis García Belmonte; Inter-American Development Bank Service providers: AMIA Foundation, Forge Foundation, Pescar Foundation and Reciduca Foundation Colombia Concession school City of Bogotá, 1999–2014 Primary and Public-private City of Bogotá Private school Minimum performance The city of Bogotá provides programme Colombia secondary partnership (Secretary of providers standards relate to the private schools with a education Education) quality of learning and school infrastructure and quantity of students. pays a pre-agreed sum per Each school has to full-time student per year in surpass the mean score exchange for the provision of standardised tests in of education to a specific public schools with similar assigned population. For conditions (outside the continuity of funding, the area of influence of the concession schools must meet concession schools). the performance standards set by the Secretary of Education and surpass the mean score of standardised tests of public schools in similar conditions. 79 80 Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Ethiopia Skills and Knowledge for Helvetas 2015-2017 Vocational Performance-based Helvetas Public and (i) Beneficiaries Helvetas makes 25% and 45% Youth (SKY) Training for and 2018- training contracts private completing training; (ii) of the payment to providers Young People 2021 vocational skill improvement of upon achievement of the first training beneficiaries; and (iii) and second expected results, providers trainees have found steady respectively. The remaining and gainful employment 30% of the payment is made (earning at least the upon achievement of the third minimum wage) expected result. or have successfully established their own business. India Educate Girls Children’s Invest- 2015–2018 Primary Impact Bond Outcome payer: Investor: Increasing the enrolment UBS-OF provided US$270,000 Development Impact ment Fund Founda- education CIFF UBS Optimus of identified out-of-school of upfront capital to Educate Bond tion (CIFF) Foundation children by 79% and Girls to implement activities Service improving learning for and deliver predefined Provider: boys and girls in rural, outcomes. CIFF committed to Educate Girls remote and marginalised pay back UBS-OF the original communities. amount plus a return if targets were met. UBS-OF incentivised Educate Girls to achieve the above- expected results by providing them with an incentive payment out of the payment they received from CIFF. India India Education IEOF pooled funds Launched (i) Early Impact bonds Outcome payer: Investors: IEOF is expected to The investor will receive Outcomes Fund 2018 Childhood IEOF pooled funds providing develop into a ‘rate-card payments for the results Development upfront approach’, reducing achieved by the service (ii) Foundational capital the contracting and provider. The amount of Learning Skills transaction costs of payment is envisioned to (iii) Secondary Service impact bonds and be based on a ‘Rate Card’ Learning providers: enabling the disbursement approach using a per unit cost (iv) School contracted to of funds across multiple formula. to Workforce implement projects. It seeks to create Transition interventions a long-term stable source (v) Skills Training of funding and capacity and Specialised building for service Courses providers, aiming to create impact at scale. Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure India Quality Education India Michael and Susan 2018–2022 Primary educa- Impact Bond Outcome payers: MSDF Investor: Improvements in learning UBS-OF invests US$3million Development Impact Dell Foundation tion and British Asian Trust UBS Optimus outcomes (namely, grade- upfront capital for the Bond (MSDF) and British Foundation appropriate numeracy and implementation of Asian Trust (BAT) Service literacy skills), which are interventions by the four Provider: defined as the difference service providers. Outcome Gyan Shala; between a baseline and payments from MSDF and BAT Kaivalya an end line score on a to UBS-OF are paid once the Education standardised test at the results have been achieved. Foundation start and end of each (KEF); school year. An incentive payment is in Society for place to encourage service All Round providers to overperform. development (SARD); Educational Initiative and Pratham InfoTech Foundation (EI-PIF) Liberia Liberian Education Third party 2016–2019 Primary educa- Public-private part- Social Finance Bridge Payment is based on Private service providers Advancement philanthropies tion nership manages the funds International the number of students receive funding from external Programme (LEAP) provided by third party Academies; enrolled. Private philanthropies based on the philanthropies Omega management of schools number of students enrolled Schools; is meant to Increased in the treatment schools. Rising learning outcomes (in Academies; particular, words read BRAC; per minute) of children in Street Child; LEAP treatment schools More than in comparison to control Me; Youth schools. Movement for Collective Action; Stella Maris Nepal The Employment Fund Swiss Agency for 2008–2016 Vocational Performance-based Helvetas Training and (i) Training completion by Helvetas pays 40% of the Development and training contracts employment beneficiaries; (ii) Trainees outcome costs to the T&Es Cooperation (SDC), service are gainfully employed upon achievement of the UK Aid and the World providers or have established an first result. The remaining Bank (T&Es) enterprise. payment is made in two instalments after two rounds of verification. 81 82 Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Pakistan Sindh Secondary Asian Development 2020–2025 Secondary edu- Public-Private School Education and Education (i) School management SELD pays the EMOs Education Improvement Bank cation Partnership Literacy Department Management plan; (ii) improved management fees for Project (SELD) Organisations staff availability and operating the school. They are (EMO) attendance; (iii) improved reimbursed for budgeted costs student enrolment, as stipulated and agreed upon attendance, and in their financial proposals. retention; (iv) lesson Additional payments to EMOs planning and student are made quarterly and are assessment system; (v) contingent on them achieving continuous professional the incentivised results. development; (vi) school repair and maintenance and improved community engagement; and (vii) improved health and hygiene practices and sporting activities. Pakistan Sindh Basic Education USAID Since 2018 Primary and Public-Private School Education and Education (i) School management The SELD pays the EMOs Program (SBEP) Secondary Partnership Literacy Department Management plan; (ii) improved based on their achievement of (SELD) Organisations staff availability and incentivised results. Individual attendance; (iii) improved results have different weights student enrolment, and payment will be made attendance and retention; accordingly. If the EMOs fail to (iv) improved session achieve results by the required planning and student date, their contracts will be assessment system; (v) terminated. continuous professional development of staff; (vi) school repair and maintenance and improved community engagement; and (vii) improved health and hygiene practices and sporting activities. Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure South Impact Bond Innovation Department for 2018–2020 Early Childhood Impact Bond Outcome Payers: Investors: Outcome payments The three investors provided Africa Fund Social Development Development Department for Social The Standard were dependent on R7.5 million (US$ 0.65million) and ApexHi Development (DSD) and Bank Tutuwa recruitment, retention to finance FCW’s operations. Charitable Trust ApexHi Charitable Trust Community and attendance of These funds were recycled Foundation; beneficiaries into the each year. The DSD and Future home visiting programme, ApexHi Charitable Trust repaid growth Asset as well as the Early the investors based on the Management; Learning Outcomes results achieved. A total of and LGT Measure (ELOM). R18.7 million out of the R20.4 Venture million available was paid. Philanthropy. (Note: The total cost that the Service outcome funders paid was Provider: The lower than the maximum Western Cape available due to the non- Foundation for achievement of the ELOM Community target.) Work (FCW) Home visitors were incentivised through extra payments as well as non- financial incentives. Uzbekistan Promoting Early World Bank, GPE & 2020–2025 Early Childhood Impact Bond Outcome Payer: Investor: Potential incentivised An investor will provide Childhood Development GPRBA Development Government of Not result areas are: (i) upfront capital to private Project Uzbekistan identified preschool occupation preschools to carry out Service rate; (ii) proportion of educational interventions. The Provider: disadvantaged children Government of Uzbekistan Private attending SIB preschools; will pay back the investor Preschools and (iii) quality of SIB upon the achievement of preschools’ learning predetermined results. environments. Vietnam Upper Secondary GPRBA 2010–2013 Secondary Conditional School GPRBA Non-public There were two expected Upper secondary schools Education Enhancement Education Grants secondary results: student received US$90 tuition Project and attendance of at least subsidy per student per year, professional 80% and achievement of a while PSS received US$160 secondary minimum 5.0 grade point per student per year. GPRBA schools average at the end of the released these funds after the (PSS) in 11 term. independent verification of provinces in results. Vietnam 83 84 Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Multiple All Children Reading: USAID, DFAT and Since 2014 Primary Payment by results All Children Reading There are There are 4 incentivised Prize-winning organisations countries A Grand Challenge for World Vision Secretariat 4 different results: (i) 200 existing are paid after the delivery of Development – Begin agents: (i) books minimum are the books. with Books Prize eKitabu; (ii) adapted for language, Rochester level or context; Institute of (ii) 200 existing books Technology minimum are converted – National to digitally accessible Technical formats; (iii) 50 new books Institute for minimum are developed the Deaf; (iii) to a specific context; SIL LEAD; and (iv) 50 existing or and new books minimum are (iv) The Asia adapted or developed for Foundation communication in local sign language. Multiple Girls Education Chal- FCDO 2013–2017 Early Childhood Payment by Results FCDO 15 projects Expected results focus on The GEC projects on Payment countries lenge Development, under learning outcomes, which by Results (PbR) are those Primary the Girls are measured by literacy that have some of their Education, Education and numeracy testing. expenditures (10%–20%) Secondary Challenge linked to the achievement of Education, outcomes at the programme’s Teacher Training two post-baseline evaluation points: midline and end line. If the projects meet their learning outcome targets, they receive this final portion of funds, known as the ‘at-risk’ proportion of their budget. If they over-achieve, they receive a bonus (up to a capped amount) referred to as the PbR upside. If the projects fail to meet their targets, they lose money up to the full amount of the ‘at-risk’ proportion of the budget. This is referred to as the PbR downside. Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Others Burundi Projet d’Appui Cordaid 2020–2021 Primary Conditional School Cordaid and Ministère 74 primary Increase accessibility for Schools are paid upon the à l’Amélioration Education Grants de l’Education schools in school-aged children in achievement of results. de la Qualité de Muramvya the intervention schools, l’Enseignement Province improved quality education. Pakistan Second Sindh Education World Bank 2013–2018 Primary and Performance-Based Sindh Government There are 4 (i) Performance terms and The Sindh government renews Sector Project (ESP) Secondary Contracts, Public- agents: conditions; (ii) performance- contracts with teachers, Education Private Partnership (i) teachers; related employmentterms education managers and (ii) education and conditions; (iii) student headmasters and schools managers and test results; and (iv) under the PPRS programme headmasters; performance criteria. if they meet the performance (iii) schools conditions. The government under the disburses part of the annual Promoting grant to district-level Private authorities. Schools in Rural Sindh (PPRS) programme; and(iv) dis- trict-level authorities. Pakistan Improvers Bonus World Bank 2010 Primary Performance-Based School Education and Public Improvement in the The government gave bonuses Programme for Education Bonus Literacy Department primary composite score of school in 3 different ways: (i) only to Government School (SELD) school head performance. The headteachers; (ii) to principals Teachers teachers and composite score is made and teachers; and (iii) a higher teachers up of: bonus to principals and a (i) the gain in the school’s standard bonus to teachers if enrolment from first to they achieved the results. fifth grade; (ii) the gain in the school’s mean score on Punjab’s standardised fifth grade examination; and (iii) the participation rate of the school’s fifth- grade students in the examination. 85 86 Project RBF Funded Funding Country Project Funder School level Principal Agent period modality results to be achieved structure Mexico Aligning Learning Federal Government 2008–2011 Upper Secondary Performance-Based Federal Federal Student performance Different constituencies in Incentives of Mexico Education Bonus Government High School on curriculum-based federal high schools receive Principals mathematics tests financial incentives (cash administered to students payments) from the Federal in grades 10–12. Government for student performance on mathematics tests. Treatment 1 provides incentives to students only, treatment 2 to teachers only, and treatment 3 gives both individual and group incentives to students, teachers and school administrators. India Strengthening Teacher REACH 2017–2019 Primary School Certificate of State Government of Block and Attendance in training Resource persons received Mentoring and Excellence Jharkhand, India Cluster and capacity building tiered certificates of excel- Monitoring System Resource programme. lence based on varying levels Person of attendance in the training and capacity building pro- gramme. Appendix 4: Findings on Impacts of RBF on Education at the Meso-Level Title of Intervention Funding Structure Discussed Intervention Results Loraque, Julia. 2018. “Development Impact Bonds: Bringing Innovation to Education Development Financing and Delivery.” Childhood Education 94 (4): 64–8. doi:10.1080/00094056.2018.1494454. Educate Girls Development Impact Bond The investor UBS Optimus Foundation By July 2017, results after two years showed provided upfront capital to the service that Educate Girls had achieved 87.7% of the provider Educate Girls to bring out- 3-year enrolment target (very much on track) of-school girls into the classroom and and 50.3% of the 3-year learning target. The improve the learning outcomes for boys investor remains on track to recoup all of the and girls in Rajasthan, India. If results initial funding of $267,000.26 were achieved, the investment would be repaid by the CIFF. Gustafsson-Wright, Emily, and Izzy Boggild-Jones. 2019. Paying for Education Outcomes at Scale in India. Center for Universal Education at the Brookings Institution. https://www.brookings.edu/wp-content/uploads/2019/11/Paying-for-education-outcomes-at-scale-in-In- dia-FINAL-FOR-WEB.pdf Educate Girls Development Impact Bond The investor UBS Optimus Foundation After three years, the Development Impact provided upfront capital to the service Bond had overachieved its enrolment and provider Educate Girls to bring out- learning targets, and the investment was of-school girls into the classroom and repaid by the Children’s Investment Fund improve learning outcomes for boys Foundation (CIFF). The project had enrolled and girls in Rajasthan, India. If results 768 of the out-of-school girls identified at were achieved, the investment would be baseline, or 92% against a target of 79%. repaid by the Children’s Investment Fund The intervention achieved student learning Foundation (CIFF). outcome gains equivalent to 160% of the target. Olken, Benjamin A., Junko Onishi, and Susan Wong. 2014. “Should Aid Reward Performance? Evidence From a Field Experiment on Health and Education in Indonesia.” American Economic Journal: Applied Economics 6 (4): 1–34. doi:10.1257/app.6.4.1. National Community Empowerment Program Villages in Indonesia receive an annual Comparing incentivised and non-incentivised – Healthy and Smart Generation), also known block grant to improve 12 maternal treatment villages, the incentives led to as Generasi programme and child health and education (school greater initial performance on health but enrolment and attendance) indicators. no differential performance on education. The size of the subsequent year’s block Specifically, the average standardised grant depends on performance relative to effect across the 8 health indicators was those of other villages in the subdistrict. about 0.04 standard deviations higher in incentivised rather than non-incentivised areas. While this difference is modest, the incentives’ impact was more pronounced in less-developed areas (at baseline). When it comes to education, while the programme improved overall enrolments, there were no differences between incentivised and non-incentivised areas on the 4 education indicators examined (primary and junior secondary enrolment and attendance). 26 Please note that at the time in which this paper was published, the RBF intervention (Development Impact Bond) had not yet ended. Therefore, the results reported are not the final results of the intervention. 87 Title of Intervention Funding Structure Discussed Intervention Results Aizawa, Toshiaki. 2019. “Impacts of the Community Block Grant Programme on School Resources, Environment and Management in Indonesia.” Education Economics 27 (5): 521–45. doi:10.1080/09645292.2019.1641587. National Community Empowerment Program Villages in Indonesia receive an annual Beyond the 4 education targets of the – Healthy and Smart Generation, also known block grant to improve 12 maternal programme, this study looked into the as Generasi programme and child health and education (school results of the intermediate outcomes of the enrolment and attendance) indicators. initiative. While the incentive programme The size of the subsequent year’s block increased community and parental grant depends on performance relative to participation in school management, those of other villages in the subdistrict. it did little to improve school resources and classroom learning and teaching environments. Especially compared with the notable improvements in health indicators reported in previous studies, the impacts of the programme on school resources and classroom environments are limited. With regards to the average class size and the student/teacher ratio, some deteriorations were actually observed as unintended net effects. Mungaray, Alejandro, Marco Tulio Ocegueda, Patricia Moctezuma, and Juan Manuel Ocegueda. 2016. “La Calidad De Las Universidades Públicas Estatales De México Después De 13 Años De Subsidios Extraordinarios.” Revista De La Educación Superior 45 (177): 67–93. doi:10.1016/j.resu.2016.01.008. Programa Integral de Fortalecimiento Mexican public state universities The quality of the undergraduate and Integral (Integral Programs of Institutional receive competitive performance-based postgraduate educational programmes, the Improvement) subsidies from the federal government increased coverage of enrolment (equity) (Public Education Secretary) based on and the strengthening of the capacities of the achievement of targets set in their the teaching staff have been the institutional institutional improvement plans (mainly performance indicators awarded more quality of education and equity in degree frequently. Twelve universities presented programmes). institutional performance levels above the national average for most of the years and were awarded more frequently with the competitive funds. The method has proven effective in promoting institutional progress in terms of quality of education and academic consolidation, as well as in diminishing the performance gap among institutions. 88 Title of Intervention Funding Structure Discussed Intervention Results Behrman, Jere R., Susan W. Parker, Petra E. Todd, and Kenneth I. Wolpin. 2015. “Aligning Learning Incentives of Students and Teachers: Results From a Social Experiment in Mexican High Schools.” Journal of Political Economy 123 (2): 325–64. doi:10.1086/675910. Aligning Learning Incentives Different groups in 88 Mexican federal Results show that (i) providing incentives to high schools receive financial incentives students alone increased mathematics test (cash payments) from the Federal scores by 0.2–0.3 of a standard deviation; Government for student performance (ii) providing incentives to teachers alone on curriculum-based mathematics tests did not affect test scores; and (iii) providing administered to students in grades incentives to students and mathematics 10–12. Treatment 1 provides incentives teachers both for their own performance and to students only, treatment 2 to teachers for that of their peers and for other teachers only and treatment 3 gives both individual and school administrators led to the largest and group incentives to students, teachers treatment effect estimates, increasing test and school administrators. scores by 0.3–0.6 of a standard deviation. Barrera-Osorio, Felipe, and Dhushyanth Raju. 2017. Teacher Performance Pay: Experimental Evidence From Pakistan.” Journal of Public Economics 148: 75–91. doi:10.1016/j.jpubeco.2017.02.001. Improvers Bonus Program for Government The provincial government of Punjab, In the first three years of implementation, School Teachers Pakistan gave cash bonuses i) only the programme, on the average, increased to school principals; ii) to principals school enrolment by 4.1 percent and student and teachers; and iii) a higher bonus exam participation rates by 3.4 percentage to principals and a standard bonus to points. The positive impact on school teachers for improving student enrolment enrolment was mainly seen in urban schools, in grades 1–5, student participation in the while the positive impact on student exam fifth grade standardised exam and scores participation rates was only seen in rural in this examination. schools. The analysis did not find that the programme increased student exam scores in any year. The highest increase in school enrolment and exam participation happened when a higher bonus was given to the principal and a standard bonus to teachers. Barrera-Osorio, Felipe. 2007. “The Impact of Private Provision of Public Education: Empirical Evidence from Bogota’s Concession Schools.” World Bank Policy Research Working Paper No. 4121. The World Bank, Washington, D.C. https://ssrn.com/abstract=960184 Bogota’s Concession Schools The city of Bogotá, Colombia, provides There is strong evidence of a direct impact private schools with school infrastructure of concession schools in reducing dropout and pays a pre-agreed sum per full-time rates. There is also some evidence of an student per year in exchange for provision indirect impact of concession schools on of education to a specific assigned the dropout rates in nearby regular public population. For continuity of funding, schools. Furthermore, there is evidence of the concession schools must meet the a positive impact on test scores of students performance standards (on quality of in concession schools when compared with education and quantity of students) set students in other public schools. by the Secretary of Education. In addition, each school commits to surpass the mean score of standardised tests in similar schools. 89 Title of Intervention Funding Structure Discussed Intervention Results Cambridge Education. 2015. Evaluation of the Pilot Project of Results-Based Aid in the Education Sector in Ethiopia. Final Report EC 2004-2006. Cambridge Education. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/ file/576168/Eval-pilot-project-result-based-aid-Education-Ethiopia.pdf Pilot Project of RBA in the Education Sector in The Ethiopian Ministry of Education gave By the end of the three-year pilot, although Ethiopia reward payments to Regional Education there were changes in the numbers of Bureaus, half of which is based on the sitters and passers, none of the estimated number of additional sitters and passers impacts were either statistically significant or of the EGSECE and half on the total reasonably attributable to the RBA pilot. number of sitters and passers. Several possible explanations exist for in- creases in the number of sitters and passers independent of the RBA pilot, such as trends in the number of students enrolled in Ethiopia’s lower secondary schools. Independent Evaluation Group (IEG). 2019. Implementation Completion Report Review: Second Sindh Education Sector Project. ICRR0021769. World Bank. https://documents1.worldbank.org/curated/en/641021576777403518/text/Pakistan-PK-Second-Sindh- Education-Sector-Project.txt Second Sindh Education Sector Project The Sindh government renews contracts Quality assurance and accountability with teachers, education managers, frameworks were introduced to 804 rural headmasters and schools under the schools under public-private partnerships. Promoting Private Schools in Rural Schools with competitively selected Sindh (PPRS) programme if they meet head teachers were high performers in performance conditions. The government the last year Student Achievement Test, disburses part of an annual grant to suggesting improved management and district-level authorities. quality of learning in these schools. Quality improvements in service delivery have plausibly contributed to increased enrolment in underserved communities (rural areas), which increased fivefold from 55,256 in 2012 to 303,405 in 2017. Accountability was strengthened by incorporating participation in student testing in the performance agreements, thus contributing to efforts to improve quality. Romero, Mauricio, and Justin Sandefur. 2019. “Beyond short-term learning gains: The impact of outsourcing schools in Liberia after three years.” Working Paper 521. Center for Global Development. https://www.cgdev.org/sites/default/files/beyond-short-term-learning- gains-impact-outsourcing-schools-liberia-after-three-years.pdf Liberian Education Advancement Programme Private service providers in Liberia receive After three years, the programme had a (LEAP) funding from external philanthropies positive treatment effect on enrolment — a based on the number of students enrolled net increase of 36 students per school. This in the treatment schools. treatment effect came from enrolment shrinking less in treatment schools since overall enrolment fell in both treatment and control schools during this period. Nonetheless, there was an increased dropout rate for students originally enrolled in those schools. In addition, outsourcing led to modest learning gains. After the first year, treatment effects on learning gains plateaued. 90 Title of Intervention Funding Structure Discussed Intervention Results Mani, Subha, Kumar Vivek, and Pradyumna Bhattacharjee (n.d.). Impact Evaluation of Training and Incentives for Middle Management in Education. Strengthening Teacher Mentoring and Interventions included non-financial Teachers mentored by RPs who received Monitoring Systems incentives for resource persons (RP, in certificates were 15 percentage points more Jharkhand, India) to improve mentoring likely to be present on the day of the research and managing of teachers. RPs received visit than teachers in the control group who tiered certificates of excellence based received no mentoring. However, it is not on varying levels of attendance in possible to observe whether this difference the training and capacity building exists due to the increased mentoring programme. The impact of the incentive of teachers or due to the non-financial was measured in terms of teacher incentives given to RPs. absenteeism and performance. There was no impact on self-reported RP mentoring and monitoring capabilities. Despite improvements in teacher presence, there was no evidence of increased teaching activity in the classroom. Correa Miranda, Adelaida, Geraldine García Medina, and Gabriel Ugarte Vera. 2017. “Evaluation of Educational Incentives Program in Peru ‘Performance Commitments’ ‘Compromisos de Desempeño’.” The Performance Commitments Programme The Performance Financing Unit (UFD) The evaluation found that the higher the under the Secretariat of Strategic number of schools and students managed, Planning (SPE) of the Ministry of the worse the physical characteristics of the Education of Peru (MINEDU) provides schools; the lower the budget per student additional funding to decentralised and the less experienced the ExU director, Educational Management Units, known the worse the performance related to the as ExU, conditional on the achievement management targets. Financial incentives of 20 targets. Targets include, but are were associated with higher achievement not limited to, timely hiring of teachers, of committed targets. Incentives could not students’ attendance and timely be directly linked to educational outcomes. distribution of school materials. However, the programme as a whole increased test scores between 9%–18% in reading comprehension and between 11%– 26% in math. 91 Appendix 5: Keyword Search Terms Results-based financing education meso ‘Impact Bonds’ meso Results-based financing education ‘Impact Bonds’ Results-based financing meso ‘Special-Purpose transfers’ education meso Results-based financing ‘Special-Purpose transfers’ education Results-based funding education meso ‘Special-Purpose transfers’ transfers meso Results-based funding education ‘Special-Purpose transfers’ Results-based funding meso ‘Block grants’ education meso Results-based funding ‘Block grants’ education Outcomes-based financing education meso ‘Block grants’ meso Outcomes-based financing education ‘Block grants’ Outcomes-based financing ‘Cash on delivery’ education meso Outcomes-based financing meso ‘Cash on delivery’ education Outcomes-based funding education meso ‘Cash on delivery’ meso Outcomes-based funding education ‘Cash on delivery’ Outcomes-based funding meso ‘Per-student funding’ allocation meso Outcomes-based funding ‘Per-student funding’ allocation Outcomes-based contracting education meso ‘School grants’ meso Outcomes-based contracting education ‘School grants’ Outcomes-based contracting meso Outcomes-based contracting ‘Pay for performance’ education meso ‘Pay for performance’ education ‘Pay for performance’ meso ‘Pay for performance’ ‘Pay for success’ education meso ‘Pay for success’ education ‘Pay for success’ meso ‘Pay for success’ Performance-based financing education meso Performance-based financing education Performance-based financing meso Performance-based financing Performance-based contracts education meso Performance-based contracts education Performance-based contracts meso Performance-based contracts Performance-based transfers education meso Performance-based transfers education Performance-based transfers meso Performance-based transfers Output-based disbursement education meso Output-based disbursement education Output-based disbursement meso Output-based disbursement ‘Impact Bonds’ education meso ‘Impact Bonds’ education 92 Appendix 6: List of Interviewees Project Name Interviewee Name Organisation Education Reform Support Project, Cameroon Vincent Perrot World Bank Bertin Mbassi PAREC Quality Improvement and Equity of Basic Education, Hamoud Abdel Wedoud Kamil World Bank Senegal Impact Bond Innovation Fund, South Africa Fiona Burtt Mothers2Mothers Riedewhaan Allie FCW Nafeesa Rahbeeni FCW Support to Upper Secondary Reform in Brazil André Loureiro World Bank Operation, Brazil Marcelo Becerra World Bank ODISHA Higher Education Program for Excellence Sangeeta Dey World Bank and Equity, India Kurt Larsen World Bank Parameswaran Balakrishnan Odhisa State Government Enhancing Teacher Education Program, Vietnam Dung Kieu Vu World Bank All Children Reading: Begin with Books Sergio Ramirez-Mena World Vision Michelle Malecki World Vision 93 NORRAG Working Paper # 12, December 2021 20, Rue Rothschild l P.O. Box 1672 1211 Geneva 1, Switzerland +41 (0) 22 908 45 47 norrag@graduateinstitute.ch @norrag @norrag.network www.norrag.org/ife 94