RESULTS-BASED APPROACHES IN DEVELOPMENT: A REVIEW January 2015 This report was authored by Vivid Economics, under contract to The World Bank. It is one of several outputs commissioned by the Energy Sector Management Assistance Program (ESMAP), a multi-donor trust fund administered by The World Bank, under a multi-year work program on Results- Based Funding for Energy Sector Development [P127532] led by Oliver Knight, Senior Energy Specialist, ESMAP. This report is a review of the literature surrounding results-based funding approaches. It was originally commissioned as part of the first report in this series, Results-Based Financing in the Energy Sector: An Analytical Guide (2013), and then updated for the second, Results-Based Aid in the Energy Sector: An Analytical Guide (2015). However, it was not included in the latter publication, as the decision was taken to publish it separately owing to the strong cross-sectoral nature of the material. The version contained in this report is therefore not specific to the energy sector, and is published with a view to informing ongoing research and discussion related to results-based funding more generally. Copyright © 2015 International Bank for Reconstruction and Development / THE WORLD BANK Washington DC 20433 Telephone: +1-202-473-1000 Internet: www.worldbank.org This work is a product of the consultants listed, and not of World Bank staff. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work and accept no responsibility for any consequence of their use. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for non-commercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: +1-202-522-2625; e-mail: pubrights@worldbank.org. Furthermore, the ESMAP Program Manager would appreciate receiving a copy of the publication that uses this publication for its source sent in care of the address above, or to esmap@worldbank.org. Results-Based Approaches in Development: A Review Report prepared for ESMAP January 2015 Results-Based Approaches in Development: A Review 2 Contents 1 Introduction .......................................................................................... 4 2 Results-based instruments ................................................................... 5 3 Overview of the research landscape .................................................. 7 4 Theoretical research on results-based approaches ............................ 9 5 Empirical research on results-based instruments ........................... 14 6 Conclusion ........................................................................................... 23 Results-Based Approaches in Development: A Review 3 List of tables Table 1. Sixteen separate cautionary findings have been identified ................. 11 Results-Based Approaches in Development: A Review 4 1 Introduction After falling out of fashion somewhat (Schmitz, 2006) there has been a resurgence of interest in conditional or results-based instruments over the last few years. Faced with increasing pressure on budgets and sometimes frustrated with the perceived ineffectiveness of development spending, policy makers have started to explore new ways of structuring development support in order to do more with less. Starting in the health sector, aid arrangements where the disbursing of funds is tied to the delivery of pre-agreed results, and where funds are disbursed only after those results have been achieved, have been piloted in increasing numbers. However, as is well known, conditionality has a mixed track record. It is therefore important to understand where, when, and how these new instruments are best deployed; what their strengths are, and what their weaknesses; and what critical information we are still missing about them. Initial research on these new instruments is emerging, but so far there is no overarching structure or overall research programme that unifies these efforts. A review that provides a general overview of this burgeoning field may therefore be useful both to policy makers and to researchers: it can both summarise the current state of the art and it may help to prevent duplicate research as well as identify gaps that could usefully be filled. This paper conducts such a review and seeks to summarise the already existing research on this topic.1 The report is structured as follows. First we give an overview of the subject matter, describing the concept and the terminology of results-based approaches. Next, we survey the research landscape on this topic, pointing out which areas are well covered, and which ones less so. In sections four and five we then structure and summarise the start of the art in theoretical research (section four) and in empirical research (section five). We conclude with some overarching findings and questions for further research. 1 This review takes a general and non-sector based approach. Results-based instruments are likely to be more effective in some sectors than in others, and this is an important area for further research; however, this paper does not offer sectoral assessments. Results-Based Approaches in Development: A Review 5 2 Results-based Instruments Results-based instruments can be defined as aid interventions that defer payments (associated with the promotion of development outcomes) until after the verified delivery of pre-defined and pre-agreed development results. Within this area, a significant amount of innovation is currently taking place, with new instruments and modalities being proposed and tested. As result of this, the terminology in this area is currently unclear, with no agreed designations for either the category as a whole or for specific types of instruments. The key features of results-based instruments, as identified by the UK Department for International Development (DFID) (Department for International Development (UK), 2012) are: – (part of) payments based on (previously agreed) results; – recipient discretion as to how results are achieved; and – verification of results by an independent third party as the trigger for disbursement. Thus the concept of these instruments is: to achieve, through dialogue, clarity on what results are desired; to give the recipient maximum discretion as to how these results should be achieved; and then to reward him in proportion to independently verified results. Note that this definition already identifies a significant difference between the ‘new’ conditionality and its more traditional predecessor: a focus is placed on determining the targeted results through a co-operative dialogue with the recipient, and on giving recipients maximum discretion as to how these results are to be achieved. Apart from a definition in policy terms, given above, it is possible to define results-based approaches according to their economic characteristics. ESMAP (2013), along with other development partners such as DFID, identify risk transfer as the defining economic characteristic of results-based approaches: where a project does not succeed, conventional instruments place financial costs with the donor. Results-based approaches on the other hand transfer this risk to the recipient, thereby significantly altering the incentive structure of development interventions. However, while there is some clarity about the key concept, there is so far no agreement on the name for the category as a whole. It is variously known as ‘aid on delivery’ (KfW, 2010) ‘payment by results’ (Department for International Development (UK), 2012), ‘performance-based financing’ (Canavan, Toonen, & Elovaino, 2008), ‘performance-based funding’ (The Global Fund, 2012), ‘performance incentives’ (Eichler & Levine, 2009), ‘quantity-performance instruments’(Ghosh, Müller, Pizer, & Wagner, 2012a), ‘results-based approaches’ (Neal 2012; Birdsall and Savedoff 2011), ‘results-based financing’ (Brenzel 2009), or ‘results-based funding approaches’ (Müller, Fankhauser, & Forstater, 2013). In addition to the lack of clarity about the term for the category as a whole, there is some confusion about the terms used to denote particular instruments. A range of specific terms exist which denote particular types of instruments or particular sub-categories: for example the Global Partnership on Output-Based Aid (GPOBA) distinguishes between ‘output-based aid’ (OBA), ‘output-based disbursement’ (OBD), ‘advance market Results-Based Approaches in Development: A Review 6 commitments’ (AMCs), ‘conditional cash transfers’ (CCTs), ‘cash on delivery aid’ (COD Aid), ‘performance-based financing for health’ (PBFH), ‘payment for environmental services’ (PES), and ‘carbon finance’ (Global Partnership on Output-Based Aid, 2011). Ghosh et al. (2012), focussing on instruments used to reduce carbon dioxide emissions, distinguish between three specific types of results-based instruments, namely ‘direct purchase’ (the direct purchase of verified emission reductions), ‘top -up’ (the topping up of revenue from selling verified emission reductions to achieve a pre-agreed minimum price), and ‘tradable put option’ (a tradable option that gives the holder of the option the right to sell verified emission reductions at a specified price before a specified date). Savedoff (2010) on the other hand distinguishes between OBA, ‘performance-based incentives’, ‘performance-based contracting’, ‘performance-based financing’, ‘performance-based aid’, and COD. Thus, while there are names for specific results-based instruments, there is so far no internationally agreed taxonomy that structures the space of results-based instruments. The specific instrument names hence have a certain ad-hoc nature and there is some overlap between different instrument names. A first attempt to clarify some of this confusion has been undertaken by DFID (Department for International Development (UK), 2012). DFID defines the category as a whole as ‘payments by results’ (PBR). This category is then subdivided into ‘results-based aid’ (RBA) and ‘results-based financing’ (RBF). RBA refers to funding arrangements between donors and recipient governments, while RBF refers to funding arrangements between funders (which can be development agencies, developing country governments, or others) and service providers (such as firms or civil society organisations). ESMAP (ESMAP, 2013)and the German Development Institute (Klingebiel, 2012) have also adopted this terminology, so that it may perhaps serve as the foundation for a more rigorous taxonomy in the future.2 While DFID’s attempt is a good start, one urgent task for academics and policy analysts is therefore to define a more complete common vocabulary, in order to facilitate discussion and communication in this area. This is certainly challenging, given the variety of instruments and relevant dimensions already in play, but it might greatly aid the emergence of a connected literature.3 2 There are informal discussions as to whether and how development impact bonds (DIBs) might be included in this taxonomy. 3 In this article, we use ‘results-based approaches’ or ‘results-based instruments’ to refer to the category as a whole, while avoiding more specific definitions unless referring to a particular instrument. Results-Based Approaches in Development: A Review 7 3 Overview of the research landscape Having outlined the concept of results-based approaches and the surrounding terminology, we briefly survey the existing literature before presenting the substantive findings from the review. The existing literature largely falls into three categories: first, specific project evaluations and analysis papers (Cashin & Chi, 2011; Duflo, Greenstone, Pande, & Ryan, 2012; Duflo, Hanna, & Ryan, 2012; Gertler & Vermeersch, 2013; Morgan, 2010; World Bank, 2008; World Bank, 2011; World Bank, 2012c;World Bank, 2012d;World Bank, 2012b; World Bank, 2012a) second, policy-oriented reports considering design and implementation challenges of results-based approaches (Birdsall & Savedoff, 2011; Brenzel, 2009; ESMAP, 2013; GPOBA, 2011; Loevinsohn, 2008; Mumssen, Johannes, & Kumar, 2010; Department for International Development (UK), 2010); and lastly, large-scale or meta-evaluations (Eichler & Levine, 2009; Fiszbein, 2010; Klingebiel, 2012; Parks & Rice, 2013; Perino, 2013). Contributions in all three areas have come from both academics and from policy analysts, resulting in a diversity of perspectives and approaches. Within this literature, the healthcare sector is particularly well covered4, and there is substantial research on three specific instruments: Advance Market Commitments5, Cash on Delivery Aid6, and Output-Based Aid7. Partly due to the very recent nature of pilots of results-based approaches, the literature as a whole is slightly skewed towards theoretical assessments, with theoretical studies outnumbering empirical works. However, project implementation reports, particularly from the World Bank (see above) and the Global Partnership for Output Based Aid (GPOBA, 2011a; GPOBA, 2012b; GPOBA, 2012a; GPOBA, 2012c) are a significant source of empirical evidence, and additional empirical publications can be expected in the near future, as a number of results-based instrument interventions near completion and will be evaluated8. 4 The literature on health-related results-based projects covers economic aspects (Savedoff, 2010), practical implementation issues in general (Brenzel & Naimoli, 2009; Naimoli, 2009; Toonen et al., 2009), as well as specific issues such as monitoring (Naimoli, Brenzel, & Sturdy, 2009), contracting (Loevinsohn, 2008), and individual (Cashin & Chi, 2011; Morgan, 2010) and comparative (van den Looij, 2009) project reviews. 5 Starting with a comprehensive report commissioned by the G7 finance ministers (Tremonti 2005), three major publications have assessed the potential of AMCs to enable public-private sector co-operation (Center for Global Development 2005, Vivid Economics 2010, Cernuschi et al., 2011). In addition, the GAVI Alliance publishes annual reports detailing the progress of a large scale AMC pilot, the $1.5 billion pneumococcal vaccines programme (GAVI 2012). 6 The literature on Cash on Delivery Aid (COD Aid) centres on the work of Birdsall and Savedoff, who give a description of their proposal and make a first principles argument in its favour in Birdsall & Savedoff 2011. Additional literature on COD Aid includes Birdsall (2012), who considers an extension to energy access, Birdsall and Perakis (2012), who describe the design of a COD pilot in Ethiopia, and Savedoff and Martel (2011), who ask which indicators might be best suited for using COD instruments. 7 Research on Output-Based Aid (OBA) is supported by the Global Partnership on Output-Based Aid (GPOBA), a multi-donor trust fund administered by the World Bank, which actively funds and supports OBA approaches and disseminates and documents topics related to OBA. Besides an annual report written by GPOBA (Global Partnership on Output-Based Aid, 2011b), GPOBA published ‘Evaluation Notes’, ‘Lessons Learned’, (which are working papers), and resources for practitioners (Global Partnership on Output -Based Aid, 2012a, 2012b; Mimmi, 2012). 8 Considering World Bank projects alone, for which post-completion implementation completion and results reports (ICRs) are mandatory, there are at least three projects that may soon result in new publications: a rural electrification project in Ethiopia (Project ID P105651) and a solar home system based rural electrification project in Bolivia, both completed in June 2013 (Project ID P102479); a Results-Based Approaches in Development: A Review 8 In addition, reflecting the lack of a common vocabulary, the literature as a whole can be characterised as partly disorganised. Reflecting this somewhat disorganised nature, the substantive findings from the review have been structured as follows: first, findings are divided depending on whether they are empirical or theoretical. Second, within the theoretical category, findings are divided into positive and cautionary findings. Third, within the empirical category, results are divided into positive, cautionary, and negative. These, respectively two or three, categories represent a spectrum rather than fixed and firmly delineated categories; a finding might seem positive to one reader, but cautionary to another. Nevertheless, these categories may be helpful in structuring an otherwise largely disorganised set of empirical and theoretical findings. further rural electrification project based on solar home systems in Bangladesh (scheduled to be completed in June 2014, project ID P119549). Results-Based Approaches in Development: A Review 9 4 Theoretical research on results-based approaches 4.1 Positive theoretical findings Results-based approaches hold great promise, according to a number of theoretical contributions to the literature. The advantages identified by theoretical research include both direct benefits, such as an alignment of donor and recipient incentives, stronger incentives and more independence for the recipient, or a better risk profile for the donor; and indirect benefits, such as leveraging additional funds, increasing accountability, and encouraging capacity building. Results on direct benefits are presented first, followed by findings on indirect benefits. First, there are several directly beneficial effects that various theoretical assessments identify: The key to many of these benefits is the observation by (Savedoff, 2010) that provision of aid often has many of the hallmarks of a ‘principal-agent’ problem: there is a principal (a development partner) who wants to achieve a set of objectives, an agent (the recipient) who will have somewhat different objectives and the information that the principal has over the agent’s actions (inputs) are imperfect. In these cases, the principal is concerned that any aid provided may not be used for the purposes initially intended, but in a way which it is not possible for the principal to easily monitor. In these cases, results-based approaches are contracts that have the potential to align the motivations, incentives and objectives of donors and aid recipients. Furthermore, results-based approaches also give recipients independence with respect to the means by which they choose to deliver the goods, which in turn encourages innovation: ‘It gives recipients discretion, permitting experimentation and innovation’ (Birdsall & Savedoff, 2011). They can also be cost effective: ‘QP instruments have the potential to be cost-effective’ (Ghosh et al., 2012a). Lastly, they may also be more accurate in targeting the poor: ‘In sum, output specification and verification, payment only on delivery of outputs, and demand management incentives all allow OBA to more effectively reach the poor’ (GPOBA, 2008). Looking specifically at AMCs, these have been identified as instruments that can overcome various market failures in relation to the development of vaccines. Most notably they have been praised as providing a stable policy platform, to allow private capital investment (if sufficient private capital markets exist) while allowing latitude in the technical means of achieving the public health outcome (Center for Global Development, 2005). Further, Tremonti (2005) identified that while private investment in health research and development responds to the private returns, it failed to take into account public benefits. AMCs allow policy makers to internalise the public good benefits to firms’ research and development calculations by using subsidies to increase the size of the potential market revenue in line with these public benefits. In other words, AMCs are a specific instance and example of how results-based approaches can effectively align the incentives of different actors. Assessments also identify indirect positive effects. Results-based instruments may also be capable of leveraging and directing considerably larger funds than just the anticipated payments: ‘the RBA payment will also be an incentive to use existing donor funds and the government’s own resources more efficiently’ Results-Based Approaches in Development: A Review 10 (Birdsall & Perakis, 2012). This may go some way towards reducing the potential risk of insufficient upfront financing availability, which is required to deliver the incentivised outcomes. Due to its focus on monitoring and verification, results-based approaches may be able to improve the accountability of both providers and recipients (Birdsall & Savedoff, 2011; Eichler & Levine, 2009). An indirect effect of the increased autonomy resulting from results-based approaches may be to encourage capacity building, particularly in monitoring and evaluation. This in turn may lead to better data quality and availability (Birdsall & Perakis, 2012; Eichler & Levine, 2009). In addition, due to reduced monitoring and micro-management by donors, results-based approaches may help to change the donor-recipient relationship into a more equal one (Birdsall & Savedoff, 2011; Holmqvist, 2012). Another indirect impact on the donor-recipient relationship may be that results- based approaches can lead to greater clarity concerning the objective that both donors and recipients aim for: ‘the decision to pay for results focuses attention on what results are really desired’(Savedoff, 2010). In sum, the literature identifies considerable advantages that results-based instruments have, in theory, over conventional approaches to aid. These advantages result both directly from the changed incentive structure, and indirectly from the activities needed to respond well to this new incentive structure. 4.2 Cautionary theoretical findings Besides straightforwardly positive findings, the theoretical literature reveals a number of more cautionary results. While the positive findings often focus on generic features of results-based approaches in general, the more cautionary results point out specific circumstances that are necessary in order for results-based approaches to succeed. Reflecting their granularity, there are 16 separate cautionary findings that have been identified. In the interest of clarity and accessibility, these are provided in the form of a table below (see Table 1) and grouped into three classes: necessary preconditions for results-based approaches to be effective; design challenges that need to be overcome; and inherent tensions that, although possible to mitigate, are unlikely to be overcome completely (though they do not appear to fully undermine the approach). A cautionary finding that is worth pointing out separately, and which may be of special interest to policy makers, is related to the risk of non-performance. (Ghosh et al., 2012a) point out that, on the one hand, results-based instruments do protect the public sector from facing the financial risk for failure; if a results- based approach fails to deliver results, the public sector or donor does not have to pay for it. This is one of the most attractive features of results-based approaches for policy makers subject to tight budget constraints, and can help promote value for money. However, non-performance still has an impact on those considering the use of results-based approaches: firstly, the funds to be disbursed have to be earmarked in order to be available, and so might not be available for other purposes. If they are not disbursed, the public sector or donor can therefore incur an opportunity cost and, indeed, in some cases, unused funds may be forfeited. Second, it is of course the case that the outcome which the funder originally desired did not occur. In other words, ‘non-performance will still [even with a conditional instrument] imply that public money has been tied up for some period of time, and no [emission] reductions have occurred’ (Ghosh et al., 2012a). Results-Based Approaches in Development: A Review 11 Table 1. Sixteen separate cautionary findings have been identified Category Finding Description Source Results-based approaches, to the extent that they only pay out funding after the verification of results, require Birdsall & Savedoff, the recipient to obtain pre-financing in order to deliver the results. This limits the range of circumstances under 2011; Vivid Availability of pre- which results-based approaches can be used to those where pre-financing can be obtained at non-prohibitive Economics, 2010 finance cost. This holds true even though results-based instruments may also leverage in funds from previously untapped sources, which may attenuate this finding. Results-based approaches are only effective if the supply side of the particular market is able to respond to the Pearson, 2011; Vivid Need for supply side new incentive. If a general enabling environment is lacking, a single results-based intervention may fail to deliver Economics 2010 capacity its intended outcome. However, results-based approaches can also be targeted at the supply side to overcome Necessary pre- this challenge. conditions Results-based approaches pre-suppose the ability to effectively monitor the outcome on which payouts depend. Birdsall & Savedoff, Sufficient monitoring Where such monitoring capacity does not exist, it will either need to be developed or results-based approaches 2011 capability cannot be used. Results-based approaches are only feasible if the desired outcome can be measured with simple and verifiable Brenzel, 2009; indicators that are affordable to measure, difficult to game or manipulate, and closely track the desired outcome. Center for Global Need for simple and While these may exist in some areas, for example vaccination, they may not exist in others, for example, Development & appropriate preventive care, public sanitation, or governance. Deutsches Institut für indicators Entwicklungshilfe, 2012 Need for simple Overly complex results-based instruments may be misunderstood, or may demotivate recipients. Neal, 2012; Eichler & design Levine, 2009 Design challenges The possibility of corruption is moved rather than removed under results-based approaches. ‘Capture’ of Farlow, Light, Corruption still monitoring and verification staff would allow for corrupt practices to take place. This is a direct result of tying Wahoney, & Widdus, possible payments to results, which (unintentionally) creates an incentive to manipulate data through ‘capturing’ 2005; Klingebiel, independent verification staff. 2012; Pearson, 2011 Results-Based Approaches in Development: A Review 12 Results-based approaches can create rents, for example through excessive per-unit payments or through Vivid Economics, Possibility of rents efficiency improvements that reduce costs faster than the fall in per-unit payments. 9 2010b; Birdsall and Savedoff 2011 Results-based approaches, in targeting outcomes and results, inherently favour the most competitive and those Ghosh et al. 2012; most able to deliver the targeted good or service. Where an agency, a firm, or a region has a strong competitive Pearson, 2011 Trade-off between advantage, it may lead, at the limit, to all funds flowing towards one agency, firm, or region. This can be prevented equity and efficiency by adding further conditions; but this will likely reduce the efficiency with which the originally desired goal will be achieved. In other words, ‘any approach will necessitate decisions on balancing overall cost-effectiveness with concerns about equity’ (Ghosh et al., 2012b). In order for a results-based approach to provide an effective incentive, donors need to coordinate their efforts. Klingebiel, 2012 Need for donor For example, where a recipient knows that in case of non-performance (and thus non-disbursal) another donor coordination is ready to provide non-results-based funding as a backup option, perhaps due to strong humanitarian preferences, the original incentive is greatly weakened. Two features of results-based approaches can increase total project costs compared to conventional ESMAP, 2013, Possibility of project instruments. First, where financing costs are substantially higher for the recipient than for the donor, the shift of Mumssen et al., 2010 cost increase pre-financing onto the recipient increases total project costs of capital. Second, the need for independent verification of result may lead to increased MRV costs (though also, potentially, to higher quality data). Where a recipient fails to deliver the pre-agreed results, a donor may feel pressured to disburse funds Birdsall & Savedoff, regardless; humanitarian motivation, as well as donor budgeting processes, may make it hard to not disburse Humanitarian time- 2013; Öhler, funds. However, such a disbursal despite non-delivery would greatly weaken the credibility, and hence power inconsistency Nunnenkamp, & of incentive, of any future results-based instrument used by that donor. A, potentially painful, tolerance for failure Dreher, 2012 is therefore a necessary component of successful results-based policy. Inherent tensions within results- Finding the right baseline and incentive modality is inherently difficult: if a baseline is established through based approaches negotiations, then the recipient has a strong incentive to negotiate a low baseline and high per unit payment; if Inherent challenge of historical performance is used for baseline setting, service providers may be discouraged from making large Savedoff 2010; baseline and performance improvements as they know that it will raise future expectations (ratchet effect). Unilaterally setting Cernuschi et al. 2011 incentive setting a baseline is likely to reduce recipient buy-in. Providing incentives in a single results-based intervention where multiple recipients are desirable to facilitate competition can be difficult when recipients have different costs and strategic objectives. 9 While this is a valid point of caution, not every instance of rents is to be regarded critically. In cases where rents come from efficiency improvements, they represent a particular way of splitting up the gains from innovation in which the innovator reaps most/all of them. This is broadly similar to the way in which patents allocate the gains from innovation, and partly compensates for the risk that the aid recipient bears. Nevertheless, the possibility of ‘excessive’ rents may undermine the political acceptability of results-based approaches. At the same time, rent creation is also possible when more conventional instruments are used. Results-Based Approaches in Development: A Review 13 Vivid Economics Results-based approaches primarily shift risk from one party to another. However, if the party to which the risks 2010; Savedoff 2010; Risk shifting, not risk are shifted are better able to manage that risk then this can reduce total cost of bearing risk, while also aligning Mumssen, Johannes, reduction incentives. This suggests that RBF approaches should be limited to contexts where aid recipients are well-suited and Kumar 2010; to take on the relevant risks or where significant capacity support is provided. ESMAP, 2013 Results-based instruments often require long term commitments in order to create a stable investment environment for firms to enable investment. However, careful consideration must be given to the balance of risk between the funders of the instrument and the recipients. Problems can arise due to inherent uncertainties in Hecht, Palriwala, and Challenges of long inflation, currency fluctuations, medium and long-term demand, or the possibility of technology shocks. There is Rao 2010; Farlow et term commitments a risk that contracts that are too tightly defined may discourage innovation but that allowing too much flexibility al. 2005 may discourage investment. Further, tax, regulatory and budgeting restrictions may make the future binding commitment of funds required by some RBF instruments difficult. As funds can only flow from provider to recipient when results are delivered, neither can be certain as to what Unpredictability of resources will flow or when. This can create challenges for recipients who may have plans to use these funds Klingebiel, 2012 resource flows for a particular purpose and for donors who may have funding commitments to meet. There is a risk that, after results have been achieved, and hence funds disbursed, the recipient uses these funds Misuse of funds Klingebiel, 2012 for purposes that the donor does not approve of. In incentivising only what is measurable, results-based approaches may cause quality to diminish as recipients Birdsall & Savedoff, Reductionist side- focus on quantity. It may lead to harder-to-measure but important interventions to go unfunded. This is an 2011; Leroy, 2012; effect inherent tension, insofar as results-based approaches can indeed only be deployed in support of measurable Savedoff, 2010 goals, and as quality will always be hard to quantify. Source: the authors Results-Based Approaches in Development: A Review 14 4.3 Summary of theoretical In sum, the theoretic literature identifies a strong set of advantages that results-based instruments have, in theory, over conventional aid instruments. However, these positive findings, particularly concerning stronger incentives, a better allocation of risk, and more independence for the recipient, are tempered by a large number of detailed cautionary results. These findings show that moving from more abstract theory to practical implementation may turn out to be quite challenging. Of these, three are particularly worth highlighting: cost increases due to the need to pre-finance and to monitor and verify results carefully; the risk of perverse incentives, particularly with regards to quality erosion and diversion of effort from unmonitored to monitored results; and the possibility of increasing inequality, as winners get rewarded while weaker recipients do not. The theoretical research therefore shows that results-based instruments can be useful, but not in all circumstances, and only if designed appropriately. The key lesson is that the success of results- based approaches will depend on the interaction between an instrument’s specific design features and the particular circumstances of the relevant context in which it is deployed. Given this preliminary summary, two natural follow-on questions emerge: under which circumstances are results-based instruments appropriate, and how should they be designed in these circumstances? One contribution in this direction10, ESMAP (2013), uses economic theory, in particular principal-agent theory, to arrive at a set of answers. We conclude the theory section by briefly describing its findings, which are broadly consistent with others in the literature, for example DFID (2010) Regarding the circumstances under which results-based approaches are appropriate, ESMAP (2013) identifies a set of necessary preconditions that must be met in order for results-based approaches to be a viable option. In addition, it also identifies criteria that, where met, render results-based approaches more attractive than conventional instruments. The necessary preconditions are: the ability for potential recipients to pre-finance the investment; the possibility to clearly identify and monitor the results; and the presence of sufficient institutional capacity on both the donor and the recipient side to respectively craft and respond to the incentive of a results-based instrument. Where these conditions are met, results-based instruments are likely to be more cost-effective than conventional approaches if three further conditions obtain: the recipient can control delivery risk; results are well-defined and there is a clear line of sight directly to the results for both the donor and the recipient; and the lag between making the investment and delivering and verifying the results is short. Where these three features exist, in addition to the necessary preconditions being met, the positive incentive effect associated with the risk transfer of a results-based instrument is likely to outweigh the additional costs caused by it. Regarding the question of instrument design, ESMAP (2013) presents six detailed questions that can be used to give structure to design efforts. These six questions are: who is eligible to deliver the results and hence to receive the relevant payments? What is the precise definition of ‘results’, i.e. the trigger for paying out the results-based payment? How is the payout itself structured (for example lump-sum payment after reaching a threshold, per unit payments, or multiple payments spaced over time)? What 10 The financial assistance provided for this literature review by ESMAP stemmed from an interest in considering how results-based approaches could be used in the energy sector. Results-Based Approaches in Development: A Review 15 is the size of the payment(s)? Is the donor procuring or subsidising? What is the exit strategy, and therefore the planned duration of the results-based instrument? These questions, when fully answered, can move a long way towards specifying the instrument. ESMAP (2013) gives guidance on how each of these six questions might be answered; taken as a whole this approach therefore seems promising as a first guide for policy makers seeking to design tailor-made results-based instruments. In sum, the theoretical literature provides a nuanced picture of results-based approaches, highlighting both its benefits and its drawbacks. There are also first recommendations as to when or where they should be deployed, and as to how they can be designed in a rigorous and systematic way. However, as with any theory, there is a degree of uncertainty as to whether theoretical predictions will turn out to be accurate in practice. We therefore present a summary of the relevant empirical findings next. Results-Based Approaches in Development: A Review 16 5 Empirical research on results-based instruments The empirical research on results-based approaches is, in some ways, similar to the theoretical findings: though there are highly pertinent results, the sum of empirical findings is somewhat unstructured. The results from the empirical literature review are therefore presented in a similar framework as the theoretical findings, grouped into positive and cautionary findings. However, due to their particular force, two results are presented separately under the heading ‘negative results’: first, results-based approaches will fail where recipients have no access to pre-financing; and second, avoiding perverse incentives is difficult in practice. 5.1 Positive empirical findings There is strong evidence for the effectiveness of well-designed incentives deployed under the right circumstances. Large-scale studies, randomised control trial experiments, and individual project implementation reports all confirm that results-based instruments can be highly effective in the right circumstances. In addition, there is some evidence that results-based instruments increase transparency and spur capacity-building in measuring and monitoring; that results-based instruments are popular with recipients; and that, at least with one instrument, unintended consequences have not materialised. Regarding evidence for the effectiveness of incentives, there have been a number of positive findings from large-scale evaluations of past results-based projects. Mumssen et al., (2010), covering 197 World Bank and non-World Bank OBA projects, find that ‘the piloting phase of OBA has generally been a success’. OBA projects, for example, significantly outperformed traditional projects on cost overruns and benefit shortfalls: in a sample of 50 projects, all with quantifiable outputs as their objectives, 85% of all OBA projects met or exceeded their objectives, while only 49% of traditional projects succeeded in doing so (Mumssen et al., 2010). Eichler and Levine (2009), looking specifically at healthcare, similarly find that ‘incentives can work in a variety of settings’, and that ‘large-scale performance incentive programs in Haiti and Rwanda, for example, have improved health outputs’ (Eichler & Levine, 2009). The strength of well-designed incentives is not just confirmed by large-scale evaluations, but also by smaller studies using randomised control trial (RCT) designs. A recent study of performance-based incentives in Ugandan healthcare finds a 20% productivity increase when the payments are restructured from unconditional to performance based (Gertler & Vermeersch, 2013). Since the change from unconditional to performance-based payments was separated from changes in the overall level of payment (the average pay- outs were kept constant in both the treatment and the control group), this provides strong evidence that the change in payment structure, not an increase in total payments, drove the productivity increase. A further case in point is an RCT study of different incentives to reduce teacher absenteeism: where a results-based incentive was administered impersonally, the incentive led to a reduction of teacher absenteeism by 21% Results-Based Approaches in Development: A Review 17 (Duflo, Hanna, & Ryan 2012).11 A similar result was found by an RCT study of results-based incentives to combat absenteeism among nurses: the incentive was highly effectiveness at increasing attendance, at least in the short run.12 A specific case can illustrate the effectiveness of a well-designed incentive: environmental auditors in India were found to systematically understate pollution when hired directly by the plant they were investigating (Duflo, Greenstone, et al. 2012); two changes to these arrangements were each found to have a significant impact. The commissioning and payment structure was changed (without changing the amount paid to auditors) such that auditors are paid from a central pool, and assigned to plants on a random basis. In addition, introducing a results-based element, the pay of a subgroup of auditors was made variable, depending on their performance (as measured by comparing their findings with findings obtained by independent auditors paid by the experimenter). As a result, ‘the treatment group improved the accuracy of their reports by 0.21 standard deviations, on average, enough that by the end of the experiment they were statistically equal to the truth. … Both financial incentives and back-checks appear to have played independent roles in improving auditors’ accuracy’ (Duflo, Greenstone, et al. 2012). Specific project implementation completion reports further confirm the effectiveness of results-based instruments: evaluating a rural water project in Andhra Pradesh, India, the authors conclude that ‘The project has confirmed that paying on outputs effectively transfers procurement and FM-related risks to service providers.’ It did this ‘without encroaching on the implementers’ freedom to use its own systems to deliver outputs’(World Bank, 2011). Similar results were pointed out in an implementation completion report for a natural gas extension project in rural Colombia (World Bank, 2008). In addition, positive results were reported (from a different project13) even in circumstances of political instability and post-conflict recovery: ‘solid results [were] delivered in the post- conflict situation despite the political uncertainty and changes in Nepal’ (World Bank, 2012a). Lastly, there is evidence for the effectiveness of results-based approaches not just at the level of service providers, but also at the society-wide level. Öhler, Nunnenkamp, & Dreher (2012), investigating whether or not the Millennium Challenge Corporation’s results-based approach has been effective, ‘find that the MCC was successful in promoting better control of corruption’. This is confirmed by Parks & Rice, 2013: 62% of respondents viewed the Millennium Challenge Account (MCA) compact program, a conditional form of budget support, as more successful than other donor-funded assistance programs. In addition, more than 90% of respondents in their survey indicated that performance-based financial incentives played at least a marginal role in domestic reform (Parks & Rice, 2013). While these findings are based on a survey of the perception of the MCC’s stakeholders, and therefore require some independent corroboration, they align well with Öhler et al.’s findings. 11 ‘However, in settings when supervisors were given discretion over administering incentives, these programs became entirely ineffective’ (J-PAL, 2012). This is further explored under cautionary findings below. 12 Mirroring the cautionary finding from the RCT study of results-based incentives against teacher absenteeism (see footnote 6), this incentive rapidly became ineffective as discretion in the awarding of the incentive was widely abused (Banerjee, Glennerster and Duflo 2008). Cf. cautionary findings below. 13 Nepal Biogas Support Program IV. World Bank Project ID P103979. Results-Based Approaches in Development: A Review 18 Beyond strong evidence for the effectiveness of results-based incentives, there is some evidence for three further positive outcomes. First, regarding measurement and data outcomes, Mumssen, Johannes, and Kumar (2010) find that the process of verification associated with results-based approaches improves overall transparency. In addition, Müller et al. (2013) find that, in the case of Norway’s International Forest Climate Initiative, ‘the verification process established has provided important capacity-building functions [for Guyana], by providing concrete input for the improvement of methodology and performance indicators’. The World Bank finds that Output-Based Aid (a results-based instrument) led to ‘small but significant steps towards improving the institutional capacity of the implementing agencies’ (World Bank, 2012c). With regards to measurement and data outcomes, there is hence preliminary evidence that results-based approaches improve transparency and spur improvements in performance management. Second, concerning the perception of results-based approaches amongst stakeholders, Parks & Rice (2013) find widespread approval and endorsement. Based on a survey of policy makers and practitioners both on the donor and on the recipient side,14 Parks & Rice, 2013 report the following: ‘Eliminating all forms of conditionality’ was the second-least popular change to aid (named by 27 respondent), with only ‘Dramatically reducing funding’ less popular, while ‘Providing payments to the government based on specific, measurable improvements in development outcomes’, was the second-most popular change, superseded only by ‘Aligning all forms of aid with the government’s national development strategy’. Significantly, participants from governments of developing countries (364 out of the 640 participants) favoured conditionality-linked payments by a larger proportion than the sample as a whole (38%, as compared to 37% in the sample as a whole), and by a larger proportion than US government officials (33%) (Parks & Rice, 2013). These findings are reflected in specific project reports: concerning a sanitation project in Senegal, ‘the project’s contracting authority, provided a detailed project completion report, attesting to its positive assessment of project implementation and outputs. … [It] also notes that, since the work was completed, it has received a large number of requests from households to continue the project’(World Bank, 2012c). In the implementation completion report for the natural gas extension project in Colombia, ‘The recipient and implementing agencies have provided very positive feedback on the project. The OBA mechanism has enabled them to rely on their own systems, which significantly has contributed to avoid delays and meeting the project’s objectives.’ (World Bank, 2008) Lastly, Parks and Rice (2013) also asked about unintended consequences; in particular, whether MCC indicators exercise outsized influence; divert attention away from other important policy areas; limit domestic autonomy; lead to ‘gaming’; or push recipient countries into the arms of less-demanding donors (Russia, China). They found little evidence for any of these effects. However, it should be noted once more that Parks and Rice’s results are based on an opinion survey, and may hence require independent corroboration. 14 The survey included 640 policy makers and practitioners (both on donor and on recipient side) from 100 countries. Results-Based Approaches in Development: A Review 19 5.2 Cautionary empirical findings As shown above, empirical investigation confirms that well-designed results-based instruments can improve the effectiveness of aid. However, the empirical research has also turned out a number of cautionary results, which provide a more detailed insight into the strengths and weaknesses of results-based instruments. The overarching conclusion is that circumstances matter greatly, and that the success of an intervention depends both on choosing appropriate circumstances in which to deploy it, and on tailoring it to these circumstances. More specifically, the research can be grouped into four conclusions: - First, the effectiveness of results-based instruments can be strongly undermined by the difficulties that may be experienced in honouring commitments (time inconsistency) or badly designed incentives; - second, where there are obstacles both on the demand and the supply side, a single RBF instrument addressing either only demand or only supply is unlikely to be successful; - third, recipient buy-in at both the local and national level is vital for success; - fourth, the additional costs associated with results-based approaches may or may not be significant; this is an on-going debate which is not yet settled. First, concerning the effectiveness of incentives, Öhler, Nunnenkamp, & Dreher (2012, p. 151) find that vacillation on the part of the donor has weakened the incentive effect of the MCC. In addition, the MCC’s incentive effect was found to be weaker both for countries that had cleared the threshold (upon which eligibility for funds depended), and for countries that were a long way below the threshold. Further highlighting the danger of defective incentives, Banerjee, Duflo, & Glennerster, (2008), in their RCT study of incentives to combat absenteeism of nurses in India, found that allowing for discretion in awarding the incentive quickly reduced its effectiveness to near-zero. This was confirmed in a further RCT study of incentives to combat teacher absenteeism, where impartially administered incentives were highly effective, while the introduction of discretion reduced its effectiveness to zero (J-PAL, 2012). This confirms some of the drawbacks identified in the theoretical literature, and indicates that time-inconsistent behaviour, threshold incentives, or undue discretion in administration can all seriously weaken the effectiveness of results-based instruments. Second, evidence from project evaluations have shown that demand stimulation alone may not be successful where supply is insufficient (in quantity or quality). Thus Fiszbein (2010) finds that conditional cash transfers15 have been highly effective in encouraging demand for school attendance, and that school attendance did increase significantly. However, actual learning outcomes did not improve: this was the case both in Mexico and Cambodia; in both countries children who stayed at school for an additional year due to a CCT programme did not outperform children who left school a year earlier. Fiszbein interprets this as ‘evidence that weaknesses in the supply of services are a limiting factor to the effectiveness of CCT 15 Conditional cash transfers (CCT) programmes ‘provide cash payments to poor households that meet certain behavioural requirements, generally related to children’s health care and education’ (Global Partnership on Output-Based Aid, 2011). Results-Based Approaches in Development: A Review 20 programs’ (Fiszbein, 2010). Brenzel (2009) supports this finding, citing a voucher scheme in India that led ‘to a massive increase in utilization of institutions for delivery, but in some areas, this demand may outstrip the ability of the health system to cope’ (Brenzel, 2009). Third, there is strong evidence that recipient buy-in is highly important for the success of any project supported through a results-based intervention; where buy-in is not present, results-based instruments are unlikely to succeed. Toonen et al. (2009), in a multi-country review of result-based instruments find that the ‘buy-in from national governments is one of the major determinants that will influence the progress and scale up of performance-based financing’. This is supported both by positive results where strong buy-in was present, and by negative results where it was absent: Brenzel and Naimoli (2009) find that ‘success in Rwanda with Health Results-Based Financing was linked to the strong political commitment of the President’, but also that ‘lack of ownership by local government authorities … hampered project implementation in Indonesia’. Fourth and last, there is cautionary evidence concerning the additional costs associated with results-based instruments in general, and with M&E and verification in particular. Oxman and Fretheim (2008), discussing the magnitude of these costs, estimate that ‘the flows of money required for RBF interventions may be substantial, including … administrative costs’. Savedoff and Martel (2011) agree, explicitly asking whether the costs of measurement and verification are manageable. Toonen et al. (2009) and GPOBA supplement these critical statements with quantitative evidence. Toonen et al. find that, in healthcare projects where administrative costs could be separated out, they turned out to be considerable: between 15 and 30% of total costs of the project, with one project in the Democratic Republic of the Congo reporting that administrative costs amounted to 40% of total costs (Toonen et al., 2009). However, these figures include all administrative costs, not just the additional costs specifically associated with results-based instruments, such as improved monitoring and evaluation and independent verification. Thus they do not allow for straightforward cost comparison between results-based and other instruments, nor do they sustain the conclusion that results-based instruments are necessarily more costly than traditional input-based approaches. Nevertheless, an evaluation of a clean-water OBA scheme that does allow for a direct comparison found that the additional costs of hiring an independent verification agent amounted to 6.25% of total project costs(GPOBA, 2012a). While it may be the case that results-based instruments have slightly higher administrative costs, they may still be more cost-effective overall. In a randomised controlled trial in Cambodia, a minimum package of health care (immunisation, family planning, antenatal care, provision of micronutrients and basic curative services) was provided either through traditional government provision, or competitively tendered out to NGOs, with contracts featuring financial bonuses for exceeding baselines, and financial penalties for underperformance. Despite slightly higher administrative costs, the results-based contracted provision of health care cost $19.12 per head per annum, while non-incentivised government provided health care cost $21.69 per head per annum. In addition the uptake and quality of care was reported to have been considerably higher for results-based contracted-out health care (Loevinsohn, 2008). Results-Based Approaches in Development: A Review 21 Summing up this fourth cautionary finding, results-based instruments may or may not be more costly than traditional input-based aid. The evidence on this matter is currently too weak to reach a decisive judgement either way, and further research is required. The conclusion, ‘performance incentives are by no means a panacea’ (Eichler & Levine, 2009) is a recurring theme in the empirical literature about results-based approaches. Results from RCT trials show that they work well under certain circumstances and if well-designed. They equally show that results-based instruments can be ineffective, and that this is more likely to occur in cases – such as RCT study reported by Banerjee et al., 2008, referred to above – where discretion is introduced into the administration of incentives. While these findings allow for conclusions on the optimal design of results-based instruments, much less research is available regarding the circumstances under which they are appropriate. ESMAP, 2013 is a notable contribution in this area, using economic theory to identify the circumstances in which results-based instruments are most appropriate, and empirical research to confirm or reject ESMAP’s findings might be of particular value to policy makers and academics. However, further research in this area is a high priority for gaining a well-rounded understanding of results-based approaches. 5.3 Negative empirical findings In addition to positive and cautionary findings, the empirical literature has revealed two negative results, both of which appear to be ingrained within the structure of results-based approaches. While neither rules out the use of results-based instruments in general, they reveal their limitations: first, where pre-financing is not available, and cannot be built into the intervention, then results-based approaches are not an option. Second, even where circumstances allow for results-based approaches to aid, avoiding perverse incentives is a difficult task. First, the need for pre-financing is a serious challenge in practice. Mumssen et al., 2010 for example, point out that ‘dealers are reluctant to extend credit to rural customers with little credit history, and credit administration and collections may be costly’. While conventional up-front aid can avoid this problem by eliminating the need to extend credit, results-based approaches cannot do this, since aid is only disbursed upon verification of the result. Results-based approaches will therefore fail where recipients have no access to pre-financing. In practice, this will be reflected in partial or very slow uptake of the results-based incentive, which may only be remediable in part. Such a situation occurred in a water and sanitation extension project in Senegal, and it is worth quoting the report’s description in full: ‘One of the basic GPOBA principles is that beneficiaries pay their full contribution before the work gets under way. This slowed down project implementation, as beneficiaries often had great difficulty coming up with the full amount. To ease this constraint, AGETIP partnered with a microfinance organization, the Partnership for the Mobilization of Savings and Credit in Senegal (PAMECAS), but it did not produce the expected results, as the cost of access to credit turned out to be too high’(World Bank, 2012c). While this need not always be the case, and while it may often be possible to combine a results-based instrument with a credit-facilitation intervention, this highlights how important the issue of pre-financing is for this class of aid interventions. Second, avoiding perverse incentives may be challenging in practice. Whenever the desired outcome cannot be measured directly (or only at prohibitive costs), this may be an unavoidable problem. DFID’s Ethiopia Cash-on-Delivery pilot serves to demonstrate some of the challenges: the basic idea is that the UK provides Results-Based Approaches in Development: A Review 22 results-based funding to the government of Ethiopia for improvements in education. Initially the number of students sitting a particular exam was considered as the disbursement-driving indicator. The main alternative, the number of students passing the exam, was seen to ‘create public pressure for improved test scores over time’ (Birdsall & Perakis, 2012), which in turn might lead to an erosion of exam and learning standards. While targeting the number of students sitting the exam may reduce downwards pressure on quality, it is unclear how good a proxy sitting the exam is for ‘student learning, the real outcome of interest’ (Birdsall & Perakis, 2012). Furthermore, paying for exam attendance, thereby incentivising an increase in the quantity of students, could still lead to a decline in quality, as resources might be shifted away from quality assurance and instead put towards increasing the number of students sitting the exam. Faced with these trade-offs, DFID opted for a dual incentive structure, with payments both for sitting and passing. However, even with this structure, worries remain: independent of how DFID chooses to structure its payments, pressure to cheat or reduce test quality may result from the way in which the Ethiopian government decides to distribute the additional funds across schools. Distribution by performance ‘would increase the stakes and thus the incentive for cheating – and the more so the lower the level at which performance is rewarded’ (Birdsall & Perakis, 2012). In this particular case, an evaluation due in 2016 will help to identify the extent to which these ex ante concerns have been accommodated. The more general point is that avoiding perverse incentives is challenging whenever the desired outcome cannot be measured directly. While there are a number of interventions where this does not apply, particularly in the infrastructure sector, it may be a serious problem for a significant subset. 5.4 Summary of empirical results The empirical evidence can be summarised into three broad conclusions. First, the evidence suggests that results-based approaches can indeed lead to improved outcomes. This conclusion is supported by a wide range of evidence, from broad review studies, to RCT experiments, to individual project reviews. Second, micro-oriented RCT studies in particular have shown that the success of results-based instruments depends strongly on their specific design features, particularly on monitoring and accountability structures. Third, a range of studies produced cautionary evidence that suggests that results-based approaches are no panacea, and that their success depends on the specific interaction between instrument design and circumstances. In particular: – some of the most important elements of results-based approaches (pre-financing, monitoring and evaluation, and verification) are both difficult and potentially expensive; – results-based interventions that target only supply may fail if demand is not also addressed, or vice versa; – results-based interventions depend crucially on recipient buy-in, both at local and national level; and – it is not yet clear if the additional costs of results-based approaches are a serious concern, or whether they are outweighed by improved efficiency; – avoiding perverse incentives may be challenging in some contexts Results-Based Approaches in Development: A Review 23 6 Conclusion Although the literature on result-based approaches to aid is somewhat disorganised a number of overarching conclusions emerge. First, results-based approaches have significant potential to increase the effectiveness of development aid; this finding, prominent in the theoretical literature, has been borne out by a range of empirical research and evidence. Second, however, it is equally clear that results-based approaches are not a panacea. A number of challenges have been identified in theory and in practice, especially the avoiding of perverse incentives, adverse equity effects, and the need to pre-finance the delivery of results. Third, taken together, this implies that results-based approaches are limited in terms of the development outcomes and sectors for which they are suitable; but that where they are suitable, they seem likely to make a valuable contribution towards improving development aid. Surveying the literature as a whole, three gaps appear to exist: first, there is some remaining uncertainty as to the circumstances under which results-based approaches are best deployed. As we suggested above, the performance of results-based instruments is likely to vary substantially from one set of circumstances to another. While ESMAP, 2013 makes a first contribution in this area, further research in this direction would be especially valuable, particularly if it includes an empirical analysis of whether ESMAP’s (theoretical) findings are borne out in practice, and if yes, where the relevant circumstances actually obtain. Second, to support the substantive research programme, and to unify it to a degree, progress on terminology would be highly useful. Lastly, a third contribution that might greatly advance research on results-based approaches would be a comprehensive compilation of existing results-based approaches and pilots. Once these gaps are filled, and once currently operating pilot projects and programmes have progressed far enough to be evaluated, it should be possible to assess whether results-based approaches are here to stay. Extrapolating from our current, still somewhat limited knowledge, it looks as though results-based approaches to development support may become a valuable addition to, though certainly not a replacement of, the already existing policy tools for development. Results-Based Approaches in Development: A Review 24 References Banerjee, R., Duflo, E., & Glennerster, R. (2008). Putting a band-aid on a corpse: incentives for nurses in the public health care system. Journal of the European Economic Association, 6(2-3), 487–500. Birdsall, N. (2012). Cash on Delivery Aid: For Energy Access? What is COD Aid?. Presentation given in Washington D.C. Birdsall, N., & Perakis, R. (2012). Cash on Delivery Aid: Implementation of a Pilot in Ethiopia (pp. 1–8). Birdsall, N., & Savedoff, W. D. (2011). Cash On Delivery - A new approach to foreign aid. Washington D.C.: Center for Global Development. Birdsall, N., & Savedoff, W. D. (2013). A Critical Moment for COD Aid or “How to Be Patient When It Matters .” Global Development: Views from the Center. Brenzel, L. (2009). Taking Stock: World Bank Experience with Results-Based Financing (RBF) for Health. Washington D.C.: World Bank Group. Brenzel, L., & Naimoli, J. (2009). Health Results-Based Financing (RBF) in Practice: Ten Institutional Challenges (pp. 1–3). Washington D.C.: RBF Health. Canavan, A., Toonen, J., & Elovaino, R. (2008). Performance Based Financing: An international review of the literature. Amsterdam: Royal Dutch Tropical Institute. Cashin, C., & Chi, Y.-L. (2011). Australia: The Practice Incentives Program (PIP). Washington D.C.: RBF Health. Center for Global Development. (2005). Making Markets for Vaccines - Report of the Working Group. Washington D.C. Center for Global Development, & Deutsches Institut für Entwicklungshilfe. (2012). Summary of “Results- Based Aid: Workshop on implementing Cash on Delivery Aid and other outcome-oriented approaches.” Workshop held in Bonn. Cernuschi, B. T., Furrer, E., Mcadams, S., & Schwalbe, N. (2011). The GAVI Alliance Pneumococcal Advance Market Commitment : Lessons Learnt on Disease and Design Choices and Processes Lessons Learnt on Disease and Design Choices and Processes. Department for International Development (UK). (2010). UK International Climate Fund Tackling climate change , reducing poverty. Retrieved from http://www.dfid.gov.uk/Documents/publications1/uk-int- clim-fund-tack-clim-chge-red-pov.pdf Department for International Development (UK). (2012). DFID pilots on payment by results. London: Department for International Development. Duflo, E., Greenstone, M., Pande, R., & Ryan, N. (2012). Evaluations - Improving Third-Party Audits and Regulatory Compliance in India. Cambridge, MA: Abdul Latif Jameel Poverty Action Lab. Duflo, E., Hanna, R., & Ryan, S. P. (2012). Incentives Work: Getting Teachers to Come to School. American Economic Review, 102(4), 1241–1278. Eichler, R., & Levine, R. (2009). Performance Incentives for Global Health: Potential and Pitfalls. Washington D.C.: Center for Global Development. ESMAP. (2013). Results-based financing in the energy sector. Farlow, A., Light, D., Wahoney, R., & Widdus, R. (2005). Concerns Regarding the Center for Global Development Report “Making Markers for Vaccines.” Results-Based Approaches in Development: A Review 25 Fiszbein, A. (2010). Using financial incentives to achieve education results: A review of evidence. Washington D.C.: World Bank Group Working Paper. Gertler, P., & Vermeersch, C. (2013). Using Performance Incentives to Improve Medical Care Productivity and Health Outcomes. National Bureau of Economic Research Working Paper Series, No. 19046. doi:10.3386/w19046 Ghosh, A., Müller, B., Pizer, W., & Wagner, G. (2012a). Mobilizing the Private Sector: Quantity- Performance Instruments for Public Climate Funds. Oxford Energy and Environment Brief. Ghosh, A., Müller, B., Pizer, W., & Wagner, G. (2012b). Oxford Energy and Environment Brief Mobilizing the Private Sector. Oxford Energy and Environment Brief, (August). Global Partnership on Output-Based Aid. (2011). Output-Based Aid in the Results-Based Financing Universe. Washington D.C. GPOBA. (2008). OBApproaches - Targeting Subsidies through Output-Based Aid. Washington D.C. GPOBA. (2011a). OBApproaches - Output-Based Aid in Vietnam: Access to Piped Water Services for Rural Households. Washington D.C. GPOBA. (2011b). Output-Based Aid in the Results-Based Financing Universe. Washington D.C. GPOBA. (2012a). Lessons Learned: Improved Rural Community Water in Andhra Pradesh. Washington D.C. GPOBA. (2012b). Lessons Learned: Uganda Reproductive Health Voucher Scheme. Washington D.C. GPOBA. (2012c). OBApproaches - Output-Based Aid in Bangladesh: Solar Home Systems for Rural Households. Washington D.C. Hecht, R., Palriwala, A., & Rao, A. (2010). Innovative Financing for Global Health: A Moment for Expanded US Engagement?. Holmqvist, G. (2012). External Financing of Social Protection: Opportunities and Risks. Development Policy Review, 30(1), 5–27. J-PAL. (2012). Policy Lessons - Teacher Attendance. Cambridge, MA: Abdul Latif Jameel Poverty Action Lab. KfW. (2010). Ideenskizze zu Aid on Delivery, Vermerk für BMZ. Klingebiel, S. (2012). Results-Based Aid (RBA) - New aid approaches, limitations and the application to promote good governance. Bonn. Leroy, M. (2012). The Aid Industry is Threatening Partner Countries with its ROD… Results Obsession Disorder. NORRAG NEWS, No. 47, 55–5. Loevinsohn, B. (2008). Performance-Based Contracting for Health Services in Developing Countries. Washington D.C.: World Bank Group. Morgan, L. (2010). Some Days Are Better Than Others: Lessons Learned from Uganda’ s First Results- Based Financing Pilot. Washington D.C.: World Bank Group. Müller, B., Fankhauser, S., & Forstater, M. (2013). Quantity Performance Payment by Results - Operationalizing enhanced direct access for mitigation at the Green Climate Fund. Retrieved from http://www.oxfordclimatepolicy.org/publications/documents/QPPOIESEV59.pdf Mumssen, Y., Johannes, L., & Kumar, G. (2010). Output-Based Aid: Lessons Learned and Best Practices. Washington D.C.: World Bank Group. Naimoli, J. (2009). Contributing to Global Learning About Results-Based Financing (RBF): The Health Results Innovation Trust Fund (pp. 1–3). Washington D.C.: RBF Health. Results-Based Approaches in Development: A Review 26 Naimoli, J., Brenzel, L., & Sturdy, J. (2009). Thinking Strategically About Monitoring Health Results-Based Financing (RBF) Schemes: Core Questions and Other Practical Considerations. Washington D.C.: RBF Health. Neal, C. (2012). “Simplicity” Key to Results-Based Aid & Financing. Retrieved from http://blogs.worldbank.org/voices/simplicity-key-to-results-based-aid-financing Öhler, H., Nunnenkamp, P., & Dreher, A. (2012). Does conditionality work? A test for an innovative US aid scheme. European Economic Review, 56(1), 138–153. doi:10.1016/j.euroecorev.2011.05.003 Oxman, A., & Fretheim, A. (2008). An overview of research on the effects of results-based financing. Oslo: Nasjonalt kunnskapssenter for helsetjenesten. Parks, B. C., & Rice, Z. J. (2013). Measuring the Policy Influence of the Millennium Challenge Corporation: A Survey-Based Approach. The College of William and Mary. Pearson, M. (2011). Results based aid and results based financing: What are they? Have they delivered results?. London: HLSP Institute. Perino, G. (2013). Private provision of public goods in a second-best world : Cap-and-trade schemes limit green consumerism. ???, 1–22. Savedoff, W. D. (2010). Basic Economics of Results-Based Financing in Health (pp. 1–18). Bath, ME: RBF Health. Savedoff, W. D., & Martel, K. D. (2011). Cash On Delivery Aid for Health: What Indicators Would Work Best? CGD Working Paper 275. Schmitz, A. (2006). Conditionality in Development Aid Policy. Berlin. The Global Fund. (2012). Performance-based Funding. Retrieved from http://www.theglobalfund.org/en/performancebasedfunding/ Toonen, J., Canavan, A., Vergeer, P., & Elovaino, R. (2009). Learning Lessons on Implementing Performance Based Financing, From a Multi-County Evaluation. Amsterdam: KIT (Dutch Royal Tropical Institute). Tremonti, G. (2005). Background papers to Advanced Market Commitments for vaccines: a new tool in the fight against disease and poverty. Van den Looij, F. (2009). Performance-Based Financing (PBF) within the Catholic Organisation for Relief and Development Aid (CORDAID). Washington D.C.: RBF Health. Vivid Economics. (2010a). Advance Market Commitments for low-carbon development: an economic assessment. Vivid Economics. Vivid Economics. (2010b). Assessment of the Economic Impact of Market-Based Measures (Vol. August). World Bank. (2008). Implementation Completion and Results Report on a Grant in the Amount of US$5.1 Million Equivalent to the Fundación Promigas for a Natural Gas Distribution for Low Income Families in the Carribean Coast Project. World Bank. (2011). Implementation Completion and Results Report on a Grant in the Amount of US$0.85 Million Equivalent to the Naandi Foundation for a Andhra Pradesh Rural Water Scheme Project. World Bank Group. World Bank. (2012a). Implementation Completion and Results Report on a Grant in the Amount of US$4.3 Million to the Republic of Uganda for a Reproductive Health Vouchers in Western Uganda. Washington D.C. Results-Based Approaches in Development: A Review 27 World Bank. (2012b). Implementation Completion and Results Report on a Grant in the Amount of US$5.0 Million Equivalent to the Government of Nepal for a Nepal Biogas Support Program IV (BSP). Washington D.C. World Bank. (2012c). Implementation Completion and Results Report on a Grant in the Amount of US$5.76 Million to the Republic of Senegal for Supporting Access to On-Site Sanitation Services Through Output-Based Aid Scheme. Washington D.C. World Bank. (2012d). Implementation Completion and Results Report on a Grant in the Amount of US$7 Million to the Kingdom of Morocco, Societe des Eaux et D’Electricite du Nord, Lyonnaise des Eaux de Casablanca, Regie Autonome de Distribution D’Eau et D’Electricite de Meknes f. Washington D.C. Contact us: Vivid Economics Limited T: +44 (0)844 8000 254 Evergreen House North E: enquiries@vivideconomics.com 160 Euston Road Grafton Place London NW1 2DX United Kingdom Company Profile Vivid Economics is a leading strategic economics consultancy with global reach. We strive to create lasting value for our clients, both in government and the private sector, and for society at large. We are a premier consultant in the policy-commerce interface and resource- and environment-intensive sectors, where we advise on the most critical and complex policy and commercial questions facing clients around the world. The success we bring to our clients reflects a strong partnership culture, solid foundation of skills and analytical assets, and close cooperation with a large network of contacts across key organisations.