56811 FAST TRACK BRIEF July 15, 2009 The IEG report "Annual Review of Development Effectiveness 2009: Achieving Sustainable Development," was discussed by CODE on July 15, 2009 Annual Review of Development Effectiveness 2009: Achieving Sustainable Development Bank project performance continues the upward trend seen over the past 15 years, but new measures shows the increase since 1993 to be lower, around 6 percent, compared with 16 per- cent using the traditional measure. Information reporting on results achieved by Bank-supported projects on recently closed projects continues to be disappointing, although important new measures are underway to im- prove it. Bank support for the environment has recovered since 2002, due to new sources of concessional finance, but huge environmental challenges remain on the other side of the financial crisis. The outcomes of environmental projects have been improving, but mainstreaming of environ- mental concerns across others sectors needs to be deepened and accompanied by systematic monitoring and reporting of environmental outcomes. Project performance shows a clear improving the threshold into the moderately satisfactory category in the six-point scale. Changes over time in the Bank's portfolio trend toward better performing regions and sectors also explain a The 2008 World Bank project performance data shows im- small part of the improved performance. provement in achieving development outcomes, allaying con- Analysis of the dates of the major turnarounds in project per- cerns that the weakened 2007 performance could signal a new formance suggests that a combination of better Bank sector downward trend. The decline in performance in 2007 was policies and improved country circumstances outside of Bank modest, and it has rebounded in 2008. control may explain much of the turnaround, rather than in- Bank performance is rated on a six-point scale, from highly ternal administrative reforms at the Bank, although the latter satisfactory to highly unsatisfactory. The percentage of satis- may have facilitated improvement already underway. factory projects (that is, the top three ratings) increased in Evaluating the results of analytic and advisory activities is 2008, continuing a steady upward trend over the past 15 inherently difficult. IEG does not regularly rate the Bank's years. Average performance, which uses the full six-point AAA, although it underpins projects and accounts for about scale, also shows important progress, but the gains are less one-third of spending on country services. However, a recent than half as large compared with the percent satisfactory ver- IEG evaluation indicates that about two-thirds of AAA is sus unsatisfactory measure. Percent satisfactory tends to give moderately satisfactory or better, and one-third less success- high weight to borderline changes--those that just pass over ful. Performance patterns suggest that impact and results for Delays in accurate reporting lead to final supervision assess- countries depends on technical quality, concrete and country- ments that come too late to take corrective action. specific recommendations, country ownership of the findings, An outgrowth of the IDA negotiations since 2003 has been and sustained follow-up after task completion. IEG intends development of a Results Measurement System to measure to pilot an approach to systematic AAA evaluation. and aggregate development outcomes in a way that is relevant The 2009 Management Action Record (MAR) tracks Bank to stakeholders and donors. This is a commendable effort to adoption of recent IEG recommendations and shows that track and report results in a coordinated fashion. A brief as- adoption levels are declining. IEG is currently examining sessment in advance of the November 2009 IDA midterm ways to improve the MAR to create a more effective product review, based on documentary evidence, staff interviews, and for tracking implementation of recommendations and identify a survey of IDA donors, suggests that the initiative has made reasons for the trend decline. progress, yet faces significant challenges to fuller implementa- tion. Efforts to strengthen performance indicators Impact evaluation is gaining much greater traction in the need strong and continuing support Bank as a tool that holds promise for providing estimates of In response to the international drive for measurement and project efficacy and impact. Over 250 impact evaluations are management for development results, the World Bank has underway in the Bank, many in the Human Development launched major efforts to improve performance information network. A review of recently closed projects in agriculture, in Bank-supported projects. The Bank has moved to streng- environment, and water indicated that few collected even the then monitoring and evaluation, introduced a Results Mea- minimum information to assess results, and those projects surement System for IDA, and expanded rigorous evaluation with baselines amenable to analysis are concentrated in East of project interventions. These efforts are beginning to influ- Asia and Europe and Central Asia. Thus requisite informa- ence institutional priorities and business practices. However, tion for basic quantitative analysis is still rare. IEG will initiate in many projects information is not sufficiently rigorous or analysis over the coming year to understand better the quality, comprehensive to provide stakeholders a picture of what real- relevance, and costs of the current impact evaluations in the ly changed as a result of the project. Bank. Project monitoring and evaluation are building blocks for Finally, economic cost-benefit analysis is the traditional me- generating good information on development outcomes and thod, long practiced in the Bank, for organizing information have received increasing Bank attention in recent years. IEG on project benefits and costs in a manner relevant to decision began to assess M&E quality in 2006, and has rated the quali- making. The Bank's use of cost-benefit analysis peaked in the ty of M&E in projects that were reviewed in the past two early 1970s but is now applied to only around one-quarter of years as modest or negligible in three-fifths of cases. These projects. This indicates that cost-benefit criteria are used far reviews point to several shortcomings: indicators are too nu- less as a basis for project funding decisions. Although part of merous and often measure outputs rather than outcomes; this may be attributed to the shifting nature of development baseline data are infrequently collected; few projects collect assistance, not all of it is. An assessment of the use of cost- the data that would be required to assess impact; and client benefit tools in environmental projects largely confirms this feedback suggests reluctance to adopt project M&E practices conclusion. This issue will be taken up in greater detail in an and considerable fragmentation of M&E efforts. These rat- IEG special report in FY2010. ings can only be assigned to projects once they close, and are therefore lagging indicators. They provide a reference point Support for environmental sustainability is for monitoring future trends and tracking whether current expanding but needs to go much further efforts to improve M&E bear fruit. The low M&E ratings, Addressing environmental degradation is one of the central however, suggest a continuing need to revisit the incentives, challenges of our time. The Bank is uniquely positioned to use, and resourcing of M&E in Bank-supported projects with support country, regional, and global efforts to address envi- a view toward simplicity, tractability, and decision relevance. ronmental challenges, such as climate change, air and water Shortcomings in M&E design and implementation may con- pollution, deforestation and biodiversity loss, in a manner tribute to problems found in project supervision and report- compatible with economic growth and poverty reduction. ing. For an average of 12 percent of projects each year there Greening the growth path need not come at the expense of is a disconnect between the Bank's self-assessments and IEG growth or of helping the poor: for example, extensive un- project performance ratings. For these projects, the internal tapped opportunities exist to reduce greenhouse gas emis- supervision of project performance through the life of a sions through incentives for greater efficiency. Moreover, project frequently appears to be overly optimistic and some- providing 2 billion poor people with basic electricity access times lack candor regarding risks to development objectives. would add little to world greenhouse gas emissions--less 2 than 0.4 percent of current emissions even if using carbon- Analytic work has also lent support to key environmental intensive means. initiatives across a broad spectrum. Examples include work on improving policies and institutions for sustainable forest This year's ARDE spotlights the Bank's activities and per- management and on developing carbon markets and project- formance in helping countries to address these environmental based technical knowledge that has helped development of challenges, based upon the findings of recent IEG evalua- internationally approved methods for the certification of tions and supplementary analysis. These evaluations include, emissions reductions. but are not limited to the following: Environmental Sustainabili- ty: An Evaluation of World Bank Group Support; Climate Change There are areas of success, but significant internal and exter- and the World Bank Group Phase I: An Evaluation of World Bank nal forces constrain the Bank's environmental portfolio. Win-Win Energy Policy Reforms; The Development Potential of Re- Weak country demand for loans arises from the public good gional Programs: An Evaluation of World Bank Support of Multi- nature of many environmental interventions. Where the bene- country Operations; and a forthcoming review of the Bank's fits can be locally captured, the incentive to act is stronger, forestry activities. such as the case of pollution control in China. Political com- petition and corruption surrounding resource rents may also Bank management has begun to initiate new strategies for the constrain demand. Knowledge and capacity constraints, par- World Bank Group, including for climate change, energy, and ticularly where Bank support has wavered over time, inhibits the environment. This ARDE reflects on the findings of the lending. Internal knowledge gaps, inadequate technical and body of IEG's recent evaluation work in the area of the envi- operational skills to integrate environment considerations into ronment, but also goes beyond these findings by updating the investment and policy reform projects, and poor dissemina- portfolio and performance data through fiscal year 2008 and tion of evidence on effectiveness within the Bank impede supplementing materials from IEG's project, country, and effectiveness and limit growth. Finally, internal staff and global evaluation products. management incentives favor large projects, such as infra- The Bank's environment strategy, launched in 2001, furthered structure or power, which disadvantages the typically smaller the shift in the Bank's stance from do-no-harm to active environmental projects. promotion of environmental goals, a shift that began in the early 1990s. This created the potential for much greater effec- Some noteworthy results have been achieved, tiveness through mainstreaming attention to the environment but weak monitoring compromises results across the Bank's project portfolio and country assistance measurement strategies. It is too early to assess the 2007 organizational changes that aimed to facilitate mainstreaming and consoli- The overall outcomes of environment projects, like the date environment into the Sustainable Development Net- Bank's portfolio as a whole, have been improving over time. work. Preliminary indicators suggest that mainstreaming has But projects managed by sectors other than the environment, decreased in some sectors, such as agriculture, energy, and which comprise most of the Bank's support for the environ- transport, and that cross-sector collaboration has been partic- ment, generally lack systematic reporting of environmental ularly weak between water supply and health. outcomes. Since 2001, new funding sources have contributed to a recov- The Bank's direct project support for the environment has ery in Bank support for the environment, including carbon helped develop market-based instruments for environmental funds, expanded Global Environment Fund resources, and management and has extended lessons on the design of these new trust funds. This funding has also shifted the focus of systems throughout Latin America and into Africa. The Bank work from more traditional mainstreamed activities like pol- has also achieved some gains in the challenging but critically lution management toward innovative approaches relating to threatened area of biodiversity conservation. The Bank has climate change. The Bank is currently stepping up its role in been the world's largest source of support for biodiversity, helping finance solutions to tough issues such as climate miti- including blended finance from the Global Environment Fa- gation and adaptation or biodiversity losses, where the institu- cility. This has provided support for a significant expansion tion can potentially make a vital difference. of protected areas as well as for the sustainable use of biodi- versity outside these areas, but more attention needs to be Although constituting only 5 percent of total environmental paid to the effectiveness and sustainability of these efforts. commitments, regional (multicountry) environmental projects have been a promising area of growth. Regional environmen- Environmental results are not limited only to projects. At the tal commitments more than doubled over the past six years, country level, environmental concerns have received growing the largest volume being in Africa. The regional pilot initia- attention in assistance strategies and country program evalua- tive, introduced in IDA14, provided additional grant financ- tions, but performance has been found wanting. Environ- ing from IDA to help overcome incentives that weigh against mental components on average have performed less well than more complex multicountry programs. overall country strategies. The Bank's record on environmen- 3 tal stewardship has been uneven. The Bank has successfully there are gaps in our knowledge of where the Bank is making helped some countries reform energy pricing by supporting headway and achieving real results. Progress will depend on the removal of costly subsidies while protecting the poor far better mainstreaming of the environment into Bank deci- through social safety nets. But the Bank has only given mod- sions across sectors and on addressing internal skills, staff est attention to national policies to promote energy efficiency. incentives, and external demand constraints that impede Bank effectiveness. The growing body of regional projects offers the opportunity to address transnational environmental challenges surround- Looking ahead, crises offer a rare chance for transformational ing the use of shared bodies of water such as the Nile Basin, change, change that is essential to achieving the vision and Lake Victoria, and the Black and Aral Seas. The Bank has potential for Bank support for environmental sustainability. been effective in fostering multi-country cooperation to es- The challenge for the Bank is to increase attention to key tablish regional institutions to address regional environmental environmental areas and to facilitate more effective develop- challenges through analytical work, project preparation, and ment outcomes by leveraging its knowledge, financial re- resource mobilization, but it has so far been less effective in sources, and convening authority. helping countries resolve longstanding resource conflicts. At the global level, there is evaluation evidence that global programs are adding value to country programs by providing About Fast Track Briefs concessional financing for country-level investments that Fast Track Briefs help inform the World Bank Group (WBG) have global environmental benefits and by generating know- managers and staff about new evaluation findings and recom- ledge about best practices in environmental management. mendations. The views expressed here are those of IEG and New partnerships and facilities such as the Climate Invest- should not be attributed to the WBG or its affiliated organiza- ment Funds and the Forest Carbon Partnership Facility--that tions. Management's Response to IEG is included in the pub- aim to promote energy efficiency, climate change adaptation, lished IEG report. The findings here do not support any general and greenhouse gas emissions reduction from deforestation inferences beyond the scope of the evaluation, including any infe- and land degradation--are deepening Bank involvement and rences about the WBG's past, current or prospective overall potential impact on climate change issues. performance. Conclusion Project outcomes represent a key building block in achieving The Fast Track Brief, which summarizes major IEG evalua- results through World Bank support to countries. These tions, will be distributed to selected World Bank Group staff. If show significant improvement over the past 15 years. Bank- you would like to be added to the subscription list, please email wide there has been a much greater focus in recent years on us at ieg@worldbank.org, with "FTB subscription" in the results, yet there remains a long distance to go to demonstrate subject line and your mail-stop number. If you would like to a real change in measurable outcomes and impacts. As the stop receiving FTBs, please email us at ieg@worldbank.org, volume of Bank lending is expected to increase in response to with "FTB unsubscribe" in the subject line. the current financial crisis, this agenda becomes all the more vital. Environmental sustainability is one of the most compelling areas for urgent action to deliver development results at the Contact IEG: country, regional, and global levels. Bank support for the en- Director-General, Evaluation: Vinod Thomas vironment has partially recovered since hitting a low point in Director: Cheryl Gray (IEG-WB) 2002, and there are encouraging signs that project results, Manager: Mark Sundberg (IEGCG) long below Bank-wide averages, are now improving for the Task Managers: Andrew Warner (IEGCG) roughly one-quarter of environmental projects whose results Chris Gerrard (IEGCG) are systematically tracked. For the other three-quarters of projects where environmental initiatives are embedded in Copies of the report are available at: other sectors (transport, water and sanitation, etc.), the prom- http://www.worldbank.org/ieg/arde09 ise of mainstreaming that emerged from the Bank's 2001 en- IEG Help Desk: (202) 458-4497 vironment strategy has not been realized. E-mail: ieg@worldbank.org In some increasingly important and innovative areas, such as the development of carbon funds and the extension of pro- tective areas to support sustainable forestry and biodiversity, 4