52607 FAST TRACKBRIEF September 26, 2008 The IEG report "Annual Review of Development Effectiveness 2008: Shared Global Challenges" was discussed by CODE on September 9, 2008 Annual Review of Development Effectiveness 2008: Shared Global Challenges Reducing poverty in any individual country is increasingly intertwined with making progress on shared global challenges-- fostering global public goods (GPGs) such as climate protection and communicable disease control. This year's ARDE tracks Bank performance in Part I and examines the Bank's work in fostering GPGs in Part II. Development outcomes from Bank lending have improved over the medium term. But in FY07 over-optimism in the Bank's ongoing assessment of project performance rose sharply, while the share of projects rated moderately satisfactory or better dropped to 76 percent from 83 percent a year earlier. Vigilance is needed to identify problem projects in real-time and ensure that the FY07 drop in performance does not foreshadow a persistent decline. Practical steps can be taken to better use M&E in projects and programs, including proper baseline information and clearer links between outputs and outcomes. The Bank's country-based model has worked relatively well in fostering global public goods when national and global interests dovetail and grants support country investments. But the greatest challenges, such as climate change, arise where local, national and global benefits--actual or perceived--diverge significantly. Here the country model comes under considerable strain. To more effectively bridge the gap between global needs and country concerns, the Bank should consider: creating dedicated budgets and better incentives for country teams to work on GPGs; better deploying its global knowledge networks; and more powerfully using its standing to give greater voice to developing countries in the governance of global programs. Part I: Tracking Bank Performance Project outcomes have improved in most sectors, but average ratings have slipped for projects in the fields of health and Development outcomes from Bank lending have public sector governance over FY03-07 compared to FY98- improved over the medium term. Over the three years to 02. Project performance among the Bank's Regions has FY07, IEG's evaluations confirm that 80 percent of projects improved most in Africa--about three-quarters of projects have been moderately satisfactory or better in meeting their weighted by disbursement over FY03-07 have been development objectives. This meets the Bank's own moderately satisfactory or better in meeting development performance target and is a significant improvement from the objectives compared to 60 percent over FY98-02. There is start of the decade. A Bank-supported water project in still a challenge for Africa projects to improve further and get Cambodia, which brought clean water to 750,000 people in closer to the performance in other Bank Regions. Phnom Penh, illustrates such development outcomes. But Bank management should avoid over-optimism in At the project level, the overall quality of M&E has been assessing ongoing project performance in order to low--rated as modest or negligible in two-thirds of improve real-time management for results. This is projects for which data are available--since FY06. Some revealed by the considerable increase in FY07 in the of the factors contributing to low M&E quality assessments difference between the Bank's self-ratings of project were poorly designed results frameworks, poorly articulated performance and IEG's final ratings of development results chains linking outputs with outcomes, and outcomes (sometimes called the "disconnect"). In FY07, over performance indicators lacking baselines and targets. two-thirds of projects rated moderately unsatisfactory or Effective results frameworks at the country level are key worse by IEG had been reported by the Bank as moderately to managing for results. While staff are gaining satisfactory or better just before they had closed. Such a wide experience with results frameworks, too often such disconnect, about twice as large as in FY05 and FY06, means frameworks have been poorly formulated and hence management is less likely to identify problem projects and their usefulness is undermined. In many cases, frameworks take timely remedial action. identify too many outcomes and monitoring indicators and Such management attention is important given that the share lack baselines and targets. Their use for monitoring and of projects with moderately satisfactory or better outcomes managing the country program, and for informing country has fallen from nearly 83 percent in FY06 to 76 percent in assistance evaluations, is very limited because of poor design FY07. A single year's data is not itself a cause for alarm, but and the absence of incentives to conduct M&E. Even so, vigilance is needed to ensure that it does not foreshadow a there are examples of emerging good practice such as the persistent decline. Excessively complex project design and "Moldova results scorecard" that links country program overly ambitious assumptions on political ownership and management and resource allocation. implementation capacity lay at the heart of many poorly The Bank has improved its approach to managing and performing projects that exited in FY07. monitoring global programs and partnerships (GPPs). Securing strong development outcomes at the country The Bank now has more robust systems to track involvement level has proved challenging. Evaluations over the past 10 in GPPs, encouraging selectivity and quality at entry. All years of 81 Bank country programs--incorporating projects, programs receiving Development Grant Facility funding of policy and technical advice, and other types of assistance-- $300,000 or more over the life of the program are also subject show that three-fifths of them were moderately satisfactory to independent program-level evaluations. But an IEG or better in meeting their development outcomes. Looking at assessment of a cross-section of such evaluations found their specific grades on IEG's ratings scale, the Bank succeeded in quality frequently compromised by weak M&E systems, supporting satisfactory outcomes in 30 percent of evaluated particularly a lack of systematic evidence on the achievement programs, including several large and important countries of programs' objectives at the outcome level. Thus, it is such as Brazil and China, that have made strides in reducing difficult to say whether the global programs reviewed-- poverty. A further 30 percent of country programs were rated together accounting for around $100 million of annual moderately satisfactory. But the remaining 40 percent of spending--ultimately had a substantial effect on the ground. programs--concentrated in countries that are smaller or have Two recent developments may hold promise for the extensive poverty, such as Malawi--were moderately Bank's results agenda, although they are in their early unsatisfactory or worse in meeting their stated development days. The first is the use of impact evaluations, and the objectives. Very few country programs are producing best- number supported by the Bank has more than doubled to 158 practice results--indeed, of 36 programs rated since FY02, over the past year. Impact evaluations are not a panacea but not one has been highly satisfactory. At the same time, no can create better understanding of the causal links and factors program has ever been rated highly unsatisfactory. contributing to the outcomes of projects, programs, and How well is the Bank using and learning from good policies. However, they are concentrated in a few areas monitoring and evaluation systems, which are key to (education, health, and conditional cash transfers) and need improving its effectiveness over the longer term? The to be managed more strategically to draw more knowledge Bank's overall approach to M&E has many strengths, and in from them. recent years there has been considerable progress in updating The second development is a new approach to its policies on lending and country strategies to emphasize measuring and reporting on development results for M&E. The introduction of results-based country assistance IDA--the Bank's main concessional finance. The results strategies has been a particularly significant step. But management system for IDA, initiated in IDA14 and with considerable room for improvement remains in putting all commitments to enhance it for IDA15, tries among other this into practice. things to spotlight changes in indicators, including access to 2 water and measures of child health. It is premature to assess Can the Bank's Country-Based Model Foster GPGs? how well this will work, but it is an important step in corporate-level monitoring and evaluation. At the same time, Relying on the country-based model as the platform for there are difficult questions as to whether and how a more the Bank's work on GPGs is a double-edged sword. The comprehensive results framework for the Bank as a whole model works well when national partners see an alignment could evolve. Currently it remains difficult to piece together between domestic and global benefits, and when the Bank the various M&E indicators to form a view of the Bank's has an attractive instrument to help implement action at the overall development results. country level. For example, the Bank's successful work in client countries to help phase out ozone-depleting substances There are two broad lessons for better tracking Bank benefited from the existence of the Montreal Protocol--a performance. First, practical steps are needed (a) at the binding agreement that committed countries to globally project level and in global and regional programs, to enhance agreed action--and the Multilateral Fund that provided the quality of the M&E systems, especially by working to put resources for investments. Global Environment Facility in place good baseline information and elucidate clearly the (GEF) grants have also been well integrated into Bank link between project outputs and targeted outcomes; (b) at country programs, such as in China where a large GEF the country level, to simplify results frameworks and so make portfolio has buttressed growing attention to environmental them more useful in guiding and evaluating programs; (c) at issues. And in Vietnam, the Bank has been able to use its the institutional level for the Bank and in partner countries, to multisectoral expertise, combined with concessional finance, manage and learn from a growing number of impact to help the authorities cope with the threat of avian flu, in evaluations, including by better integrating them into country part because there was a strong national interest in averting programs and exploiting cross-country synergies in economic fallout in the domestic food industry. conducting and sharing studies. Second, the Bank and IEG should strengthen the evaluation knowledge base for the But the country-based model comes under strain, Bank's corporate results. Progress on these two fronts will especially when global and country interests are seen to improve the prospects for greater development impact in the diverge significantly and the Bank's traditional tools, years ahead. including its lending, do not gain traction with clients. This makes it doubly difficult to secure progress with GPGs. Part II: Shared Global Challenges Tackling climate change requires huge adjustments in various economic behaviors, including reducing emissions and The Challenge of Global Public Goods improving economy-wide energy efficiency and use. For many countries the benefits of such actions seem remote Tackling global climate change and providing other while the costs accrue in the near term. To date, though, the important global public goods present some of the Bank has not been able to call on an attractive large-scale greatest challenges of our time. Indeed, many global public funding program or invoke an international framework to goods are chronically under-supplied. Why? Because it is encourage comprehensive action on climate change. It will be difficult to secure collective action among nations to provide important to see how far the recently discussed Climate a public good--such as keeping air clean--particularly when Investment Funds help improve this situation. the costs are borne locally while the benefits are largely captured nationally or globally. Yet there is a growing The Bank pays attention to fostering global public interconnection between the different types of investments goods in its high-level corporate strategies and the topic and actions needed at various levels to foster global public has been emphasized by the President as one of the goods. Bank's six strategic pillars. However, that attention wanes as one moves down from corporate strategies to The World Bank Group has emphasized the need to foster sectoral or regional strategies, and then down one level global public goods (GPGs) as one of its main priorities in further to country strategies. The Bank's GPG Framework and the future. The effective provision of such GPGs increasingly Long-Term Strategic Exercise both discussed GPGs extensively influences development results (discussed in Part I above), but lacked specifics on how to translate corporate priorities especially addressing the many dimensions of poverty, into country action. The treatment in strategies at the next including vulnerability. The Bank's framework for its role in level down--the Bank's Networks and Regions--varies GPGs strategy notes that it can connect global concerns to significantly. Attention to GPGs is more prominent in both country programs and advocate for collective international sector and regional strategies dealing with environment than action. How can the Bank enhance its effectiveness in this in those dealing with the health sector. This may be due to area? the type of intervention needed in health sector GPGs--such as communicable disease control, which requires a strong 3 national focus that might not be explicitly connected to global would be needed to avoid fragmenting IDA's overall action. framework. The systems for integrating GPGs into country When the Bank has had a clear and viable instrument to strategies are underdeveloped. Environmental commons is help its country partners take action on some GPGs, the GPG most frequently noted in country strategies (in part there has been progress--with the GEF being a good because GEF projects are mainstreamed in the Bank's example. Where the Bank has not had an obviously attractive systems), but other GPGs are less often emphasized. There is financial instrument--and/or where there has been a lack of no evidence that over time the treatment of GPGs in Bank demand from country partners--it is less easy to see country strategies has expanded, but very recent examples of progress. Measures to protect and conserve important forest good practice--such as in Brazil--may pave the way for resources around the world, for example, have produced a more thorough and consistent strategic planning. highly varied picture. In Indonesia, an evaluation of the Bank's country assistance program from 1999 to 2006 The Bank has at least three levers to draw on to move showed that it covered forestry issues with large-scale from strategy to action at the country level--budget and analytical work but little lending. Over that period the trust fund allocation, financing instruments, and global traction achieved by the Bank was very limited, and programs. Each is discussed in turn below. deforestation continued at a rapid clip. Resource Allocation There is often a mismatch between country needs (and The Bank estimates its administrative expenditure on resources) and global ambitions for GPGs. In middle- GPGs at around $110 million in FY07, nearly half of income countries, the Bank's ability to influence (or persuade) which is from sources such as trust funds that are a country to take concrete action on some GPGs is inherently outside the Bank's core budget. At about 4 percent of its limited, even though effective provision of those goods overall operating budget, this is one of the smaller requires deep participation by these middle-income countries. allocations for the Bank's six strategic priorities. These The limits of non-concessional finance are clear, for example, estimates should be treated with some caution since they may in the Bank's work on avian influenza, in which only 7 of the vary significantly depending on definitions and data 50 projects approved are IBRD-financed, and to date only classifications used. Going forward, a more precise definition $12 million of the $94 million in IBRD loans have been and tracking of spending on global public goods would be a disbursed. useful management tool. Global Programs A heavy reliance on trust funds for financing GPG work may The Bank is now a partner in some 160 global programs itself increase the difficulties of mainstreaming such activity and partnerships, and around 90 percent of the total alongside long-standing work financed by the Bank's own spending of these GPPs overseen by the Bank is budget. Spending on GPGs as a whole has risen rapidly over directed at global public goods. A few large initiatives the past five years, with the biggest increase for work on account for most of this spending: the Global Fund to Fights environmental commons. Aids, TB, and Malaria (GFATM), the GEF, and the Financing Instruments Consultative Group for International Agricultural Research (CGIAR). The Bank's administrative effort in GPPs is not Concessional finance is important to foster many GPGs, fully driven by GPG concerns, however, since more than 100 and in recent years, the Bank has committed substantial IDA of these programs are focused largely on national public funding to help countries in programs with clear GPG goods such as urban development or infrastructure market dimensions, such as HIV/AIDS and environmental regulation. commons. Often country-level implementation capacity is stretched, however, and national priorities may take Despite the Bank's direct role as a partner in global precedence over some GPG considerations. Staff report that programs, systematic linkages to country programs have there is great reluctance among national partners and Bank at times been lacking. For example, many of the programs country teams to allow IDA allocations targeted for poverty had only modest participation by middle-income countries. reduction to be diverted to fostering GPGs whose benefits Task managers for global programs have not commonly been may not be immediately felt by the poorest. A recent required to demonstrate how such programs have added innovation in IDA is a specific allocation for regional value to country programs and Bank operations, and often (multicountry) projects. Although it is too early to assess how lack the incentive or administrative budget to do so. well this is working, it should be monitored for lessons in Merely locating a global program in the Bank--as 57 mirroring this approach for some GPGs, although great care such programs are--does not guarantee effective 4 country linkages. For example, linkages were weak in the proper voice and representation of developing countries Population Reproductive Health Capacity Building Program in such programs improves their responsiveness and despite the potential synergies with Bank investment long-term sustainability. Yet developing country voices operations in various countries. IEG evaluations have also remain underrepresented--not least in the governance of found that greater legitimacy of a global program does appear many global programs--and whether the Bank could have to foster stronger linkages with country operations. pushed harder on this remains a question. It is encouraging that governance arrangements in several programs, including In the Bank's efforts to provide regional public goods-- the GEF and CGIAR, have improved over time. For large and to link regional and country concerns and new global programs aimed at climate change, it is critical to opportunities--it faces challenges similar to those for ensure sound and equitable governance arrangements that GPGs. Regional programs have risen in importance in recent balance the interests of the key parties involved. years, but their integration into country programs remains the exception rather than the rule, and they still account for a modest share of Bank lending. Improving the Bank's Support for GPGs: The Bank's Advocacy on GPGs: Lessons from Experience What Has Worked and What Has Not The Bank's country model has its place in fostering Successful advocacy goes beyond encouraging action at GPGs. It has worked well when national and global interests the country level. It also involves producing collective coincide--often with an agreed international framework for global responses and promoting the development action, such as the Montreal Protocol--and when grant interests of the poor in international agreements and finance supports country-based investments. frameworks for action. Looking ahead, some of the great shared global Promoting improvements in the global trading challenges arise where national and global benefits framework is an example of the Bank's advocacy at its diverge significantly--most notably in climate protection. best. Key ingredients included a long period of work directly In tackling those, the Bank--including through cooperation with partner countries, the assembly of first-rate intellectual with the IFC and MIGA--needs to find a way to bridge the and analytical research capacity, and proactive and highly gap more effectively between global needs and country visible dissemination and the willingness to engage in public preferences. Lessons from this review suggest some effective debate. These were combined to excellent effect, and the measures in five areas that may help the Bank upgrade its Bank's work also had an opportunity to gain traction in the ability to foster GPGs. context of "live" negotiations for the Doha round of a new trade agreement. First, the Bank can create better incentives to deliver GPGs effectively at the country level. This would include The experience with avian flu also illustrates the Bank's new approaches to setting budgets and recognizing strengths as an advocate and convener. The Bank's performance of managers and staff. On budget setting, one contributions to a global response built on robust economic option is to set aside at the corporate level significant analysis, convening power, fiduciary reputation, and administrative funding to be allocated to country teams-- multisectoral expertise. It also helped that the ground was transparently and possibly competitively--to high-priority fertile for the Bank's advocacy, given that global and national GPG work at the country level. Care would be needed to concerns aligned as country needs were urgently felt. make sure such funding was used as a genuine addition by Advocacy on the environmental commons has proved a teams and not simply displace other activity. To provide more complex challenge. The Bank has played a positive better incentives to staff, managers at all levels need to advocacy role in some very practical settings, including the consider recognizing country and global-level work on GPGs securing of resources for the GEF, the launch of the in performance management systems. Prototype Carbon Fund (and subsequent Carbon Funds), and Second, the Bank can consider clearer organizational methodologies to put the Clean Development Mechanism arrangements to best select, and indeed link together, into action. The extent to which the Bank has been a leading responses at country, regional, and global levels. Some influential advocate on climate change is more debatable, but Regions may want to have dedicated staff advancing work on there is now a platform on which to build, including the regional programs (and regional public goods), as has been Bank's new Strategic Framework for Climate Change. done in Africa, and perhaps expand their remit to cover Advocacy through global programs has become an GPGs as well. But this is not a one-size-fits-all prescription, increasingly important channel for fostering GPGs, and 5 and other Regions may have different arrangements suitable to their circumstances. Third, a more effective approach to the delivery of the Bank's global knowledge and capacity to country teams working on GPGs would be beneficial. To this end, the way the Bank can best deploy its expertise, particularly that of its specialists located at the center of the institution in the Network anchors, should be reviewed. Fourth, the Bank and its stakeholders could renew attention to ensuring that the perspective of developing countries is connected effectively with global responses. The Bank might be able to use its standing more powerfully to give greater voice to developing countries in the governance of significant global programs. It should take a more proactive stance in advocating for development interests--and developing country partners--in international forums (and agreements) dealing with GPGs. That would include the Bank continuing to secure additional development assistance and promote the design and use of market-based instruments to help developing countries provide GPGs. The Bank could also explore further ways to stimulate South- South exchange of knowledge and the development and application of new technologies designed with and for the South to contribute to GPGs such as climate-friendly energy production and use. Finally, a firmer and more precise justification is needed for the costs and benefits of actions being proposed for the Bank's work on fostering GPGs, to ensure that such work in financially and institutionally sustainable over the long-term. Particularly for global programs, the Bank must redouble its efforts to be more selective in its engagement and more forthright in exiting those programs whose benefits and cost-effectiveness are questionable. It should also be insistent on putting in place--and using--sound results frameworks underpinned by realistic and cost-effective monitoring and evaluation systems. The views expressed here are those of IEG and should not be attributed to the World Bank Group. The findings do not support any general inferences beyond the scope of the evaluation, including any references Contact IEG: about the World Bank Group's past, current or prospective overall Director-General, Evaluation: Vinod Thomas performance. Director: Cheryl Gray (IEG-WB) Senior Manager: Mark Sundberg (IEGCG) The Fast Track Brief, which summarizes major IEG evaluations, will Task Manager: Thomas O'Brien (IEGCR) be distributed to World Bank Group staff. If you would like to be added to the subscription list, please email us at ieg@worldbank.org, Copies of the report are available at: with "FTB subscription" in the subject line and your mail-stop http://www.worldbank.org/ieg/arde08 number. If you would like to stop receiving FTBs, please email us at IEG Help Desk: (202) 458-4497 ieg@worldbank.org, with "FTB unsubscribe" in the subject line. E-mail: ieg@worldbank.org