52459 IEG PUBLICATIONS 2006 Annual Report on Operations Evaluation Annual Review of Development Effectiveness 2006: Getting Results Addressing the Challenges of Globalization: An Independent Evaluation of the World Bank's Approach to Global Programs Assessing World Bank Support for Trade, 1987­2004: An IEG Evaluation Books, Buildings, and Learning Outcomes: An Impact Evaluation of World Bank Support to Basic Education in Ghana Brazil: Forging a Strategic Partnership for Results--An OED Evaluation of World Bank Assistance Bridging Troubled Waters: Assessing the World Bank Water Resources Strategy Capacity Building in Africa: An OED Evaluation of World Bank Support China: An Evaluation of World Bank Assistance The CGIAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research Committing to Results: Improving the Effectiveness of HIV/AIDS Assistance--An OED Evaluation of the World Bank's Assistance for HIV/AIDS Control Country Assistance Evaluation Retrospective: OED Self-Evaluation Debt Relief for the Poorest: An Evaluation Update of the HIPC Initiative A Decade of Action in Transport: An Evaluation of World Bank Assistance to the Transport Sector, 1995­2005 The Development Potential of Regional Programs: An Evaluation of World Bank Support of Multicountry Operations Development Results in Middle-Income Countries: An Evaluation of the World Bank's Support Economies in Transition: An OED Evaluation of World Bank Assistance Engaging with Fragile States: An IEG Review of World Bank Support to Low-Income Countries Under Stress The Effectiveness of World Bank Support for Community-Based and ­Driven Development: An OED Evaluation Evaluating a Decade of World Bank Gender Policy: 1990­99 Evaluation of World Bank Assistance to Pacific Member Countries, 1992­2002 Extractive Industries and Sustainable Development: An Evaluation of World Bank Group Experience Financial Sector Assessment Program: IEG Review of the Joint World Bank and IMF Initiative From Schooling Access to Learning Outcomes: An Unfinished Agenda--An Evaluation of World Bank Support to Primary Education Hazards of Nature, Risks to Development: An IEG Evaluation of World Bank Assistance for Natural Disasters How to Build M&E Systems to Support Better Government IEG Review of World Bank Assistance for Financial Sector Reform Improving Investment Climates: An Evaluation of World Bank Group Assistance Improving the Lives of the Poor Through Investment in Cities Improving the World Bank's Development Assistance: What Does Evaluation Show? Maintaining Momentum to 2015? An Impact Evaluation of Interventions to Improve Maternal and Child Health and Nutrition Outcomes in Bangladesh New Renewable Energy: A Review of the World Bank's Assistance Pakistan: An Evaluation of the World Bank's Assistance Pension Reform and the Development of Pension Systems: An Evaluation of World Bank Assistance Poland Country Assistance Review: Partnership in a Transition Economy The Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank's Support Through 2003 The Poverty Reduction Strategy Initiative: Findings from 10 Country Case Studies of World Bank and IMF Support Power for Development: A Review of the World Bank Group's Experience with Private Participation in the Electricity Sector Putting Social Development to Work for the Poor: An OED Review of World Bank Activities Small States: Making the Most of Development Assistance--A Synthesis of World Bank Findings Social Funds: Assessing Effectiveness Sourcebook for Evaluating Global and Regional Partnership Programs Water Management in Agriculture: Ten Years of World Bank Assistance, 1994­2004 World Bank Assistance to the Financial Sector: A Synthesis of IEG Evaluations The World Bank in Turkey: 1993­2004--An IEG Country Assistance Evaluation World Bank Lending for Lines of Credit: An IEG Evaluation All IEG evaluations are available, in whole or in part, in languages other than English. For our multilingual selection, please visit http://www.worldbank.org/ieg ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 2009 Achieving Sustainable Development 2009 The World Bank http://www.worldbank.org/ieg Washington, D.C. ©2009 The International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org All rights reserved 1 2 3 4 5 12 11 10 09 This volume, except for the "Management Response" and the "Chairman's Summary," is a product of the staff of the Independent Evaluation Group of the World Bank Group. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. This volume does not support any general inferences beyond the scope of the evaluation, including any inferences about the World Bank Group's past, current, or prospective overall performance. The World Bank Group does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank Group concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development / The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-4470; Internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 , H Street NW Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@worldbank.org. Photo credits: Cover: Paul Klee, "Dream city," 1921, 106 watercolour on paper on cardboard, 47,5 x 31 cm; Die Sammlung Berggruen in den Staatlichen Museen zu Berlin; © Alinari/Art Resource, NY. Used by permission of Art Resource, NY, and the Artists Rights Society, NY. Part I: Building archway, India. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Part II: Canadian forest. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. ISBN: 978-0-8213-8135-9 e-ISBN: 978-0-8213-8136-6 DOI: 10.1596/978-0-8213-8135-9 Library of Congress Cataloging-in-Publication Data has been applied for. Environmental sustainability : an evaluation of World Bank Group support. p. cm. Includes bibliographical references. ISBN 978-0-8213-7670-6 1. Economic assistance--Developing countries--Environmental aspects. 2. Sustainable development--Developing countries. 3. World Bank--Developing countries--Evaluation. I. World Bank. HC60.E53 2008 338.9'07091724--dc22 2008028078 World Bank InfoShop Independent Evaluation Group E-mail: pic@worldbank.org Communication, Learning, and Strategy (IEGCS) Telephone: 202-458-5454 E-mail: ieg@worldbank.org Facsimile: 202-522-1500 Telephone: 202-458-4497 Facsimile: 202-522-3125 Printed on Recycled Paper Contents vii Abbreviations ix Acknowledgments xi Foreword xiii Executive Summary xix Management Comments xxv Chairperson's Comments: Committee on Development Effectiveness (CODE) 3 1 Introduction 5 PART I: TRACKING BANK PERFORMANCE 7 2 World Bank Performance in Perspective 9 Project Performance Recovered in 2008 12 Long-Term Performance Shows a Clear Positive Trend 13 Shifting Regional and Sectoral Composition Is Part of the Reason for Improved Performance 15 When Did Performance Begin to Improve and Why? 17 Performance Ratings of Country Programs: No Similar Upward Trend 19 Bank AAA Is Important but Not Regularly Evaluated 21 Level of Bank Adoption of Recent IEG Recommendations Has Been Declining 21 Conclusions 23 3 Toward Measurable Results and Improved Performance Data 25 Progress Implementing Bank-Wide M&E Reforms Is Evident but Remains Challenging 29 Bank-Wide Results Measurement System: Making Progress, but toward Uncertain Ends 30 Use of Impact Evaluation Is Growing, but Its Benefits Are Not Fully Achieved 35 Real-Time Project Information: Continued Overoptimism 38 Economic Cost-Benefit Analysis: A Dormant Subject 39 Conclusions 41 PART II: ACHIEVING SUSTAINABLE DEVELOPMENT--LESSONS FROM THE BANK'S EXPERIENCE 43 4 World Bank Support for Environmental Sustainability 45 The Bank's Approach to the Environment Has Evolved in Response to Changing Needs 50 World Bank Support for the Environment Has Recovered Since 2002 53 Mainstreaming Environmental Concerns across Sectors Is More Apparent than Real iii ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 55 Analytical Work Is Helping to Catalyze Innovative Environmental Instruments 57 Internal and External Constraints Shape and Inhibit the Bank's Effectiveness 59 Major Lessons 61 5 Achieving Environmental Results from the Project to the Global Level 63 Project-Level Results Show Areas of Success, but Monitoring Is Weak 65 Country-Level Results Involve More than Successful Projects 67 Regional Projects Offer Growing Potential to Achieve Regional Environmental Benefits 68 Global Environmental Programs Add Value to Country-Level Efforts 70 Economic Analysis of Environmental Projects Is Challenging 71 Major Lessons 73 Appendixes 75 A: Supporting Data for 2008 Project Performance Analysis 77 B: Criteria for IEG Evaluations 79 C: Summary of Management Action Record 2009 83 D: The Bank's Environment and Natural Resource Management (ENRM) Portfolio 99 E: Performance of the Bank's ENRM Portfolio 105 F: Evaluating Environmental Performance at the Project Level 112 G: Evaluating Environmental Performance at the Country Level-- An Analysis of Recent CASCR Reviews 119 H: Evaluating the Performance of Regional Environmental Projects 129 I: Evaluating the Performance of Global Environmental Programs 133 J: The Results Measurement System under IDA: A Review 141 Endnotes 145 References Boxes 9 2.1 IEG Outcome Ratings 10 2.2 What Does a Satisfactory or Unsatisfactory Project Look Like? 15 2.3 What Accounts for Substantially Worse Outcomes in Health and Public Sector Projects? 18 2.4 What Do Moderately Satisfactory and Unsatisfactory Country Programs Look Like? 27 3.1 Recent Evaluations Highlight Problems in M&E 28 3.2 The Paradox of Satisfactory Performance and Unsatisfactory M&E 32 3.3 Reasons for Impact Evaluation 38 3.4 True Pilot Projects: Why Are They Not Used More Often? 49 4.1 IFC and MIGA Have Also Strengthened Attention to the Environment in Recent Years 52 4.2 Lessons from the Midterm Review of the GEF's Resource Allocation Framework 54 4.3 Mainstreaming Environmental Themes into Bank-Supported Projects 55 4.4 Meaningful Mainstreaming: Enhancing the Quality of Life through an Integrated Health-Environment Project in Eritrea iv CONTENTS 57 4.5 Optimal Allocation of Environmental Resources Is Constrained by Political Realities at the Country Level 58 4.6 The Importance of Client Commitment and Sustained World Bank Engagement: The Case of China 65 5.1 The Indigenous and Community Biodiversity Conservation Project in Mexico (COINBIO) 66 5.2 Risks to Sustaining Environmental Outcomes: The Case of the Central America Mesoamerican Barrier Reef System 69 5.3 What Does a Highly Satisfactory Regional Project Look Like? The Case of the Pilot Integrated Silvopastoral Approaches to Ecosystem Management Project in Latin America Figures 10 2.1 World Bank Project Ratings Have Improved since 1993 11 2.2 Percent Satisfactory Outcome Ratings by Region and Sector, 2006­08 versus 2003­05 12 2.3 Improvement in World Bank Project Ratings since 1993, Alternative Measures 16 2.4 Percentage of Operations Judged Satisfactory, by Year of Entry and Exit 17 2.5 Agriculture: Percentage of Operations Judged Satisfactory, by Year of Entry and Exit 19 2.6 Mixed Performance in Country Program Ratings over Time 27 3.1 M&E Was Rated High or Substantial in Only about a Third of Projects Exiting in 2007 and 2008 35 3.2 Timeline of Key Project Ratings 36 3.3 Bad Information Tends to Be Revealed Only at the End of Implementation 39 3.4 Percentage of Projects with Economic Rate of Return Estimates, by Year of Approval 47 4.1 Absolute Changes in CO2 Emissions and Income, 1992­2004 48 4.2 The World Bank Group and the Environment: A Summary Timeline of Activities 50 4.3 Bank Commitments to ENRM Operations, Fiscal Years 1990­2008 51 4.4 Bank-Implemented ENRM Operations by Source of Financing, 1995­2001 and 2002­08 51 4.5 Bank-Implemented ENRM Operations by Environmental Theme, 1995­2001 and 2002­08 53 4.6 Bank-Implemented Global and Regional ENRM Operations by Region, 1995­2001 and 2002­08 55 4.7 Bank-Implemented ENRM Commitments by Sector Board, 1995­2001 and 2002­08 56 4.8 Environment-Related Analytical and Advisory Services, Fiscal Years 2002­08 59 4.9 Bank-Implemented ENRM Commitments by Region, 2002­08 64 5.1 Overall Project Outcomes of Environment-Related Projects in Comparison with the Bank-Wide Average, Exit Years 1995­2008 66 5.2 ENRM Projects in the Environmental Sector Have the Same Sustainability Ratings as the Bank-Wide Average v ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Tables 12 2.1 Average Annual Changes in Ratings between 1993 and 2008 14 2.2 Changes in Project Composition Account for a Small Portion of Overall Improvement 31 3.1 The Bank's Progress in Implementing IDA15 Recommendations 33 3.2 Summary of Data Used for Evaluating Project Outcomes, by Region 34 3.3 Summary of Evaluation Types 37 3.4 Low Ratings at Project Midpoint Foreshadow Worse Outcomes 37 3.5 Nearly Half of Projects Are Ultimately Rated Lower than Projected at Project Midpoint 71 5.1 Relationship between Economic Rate of Return/Financial Rate of Return and ENRM Projects by Subtheme vi Abbreviations AAA Analytic and advisory activities AIDS Acquired Immune Deficiency Syndrome ARDE Annual Review of Development Effectiveness CAE Country Assistance Evaluation CAS Country Assistance Strategy CASCR Country Assistance Strategy Completion Report CEPF Critical Ecosystem Partnership Fund DEC Development Economics Group DIME Development Impact Evaluation Initiative ENRM Environment and natural resource management ESW Economic and sector work GDP Gross domestic product GEF Global Environment Facility GPR Global Program Review GWP Global Water Partnership HIPC Heavily Indebted Poor Countries HIV Human Immunodeficiency Virus IAS Invasive alien species IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report IDA International Development Association IEG Independent Evaluation Group IFC International Finance Corporation ISR Implementation Status and Results Report IWRM Integrated water resource management MAR Management Action Record M&E Monitoring and evaluation MIGA Multilateral Investment Guarantee Agency NGO Nongovernmental organization OECS Organization of Eastern Caribbean States OPCS Operations Policy and Country Services PAD Project Appraisal Document PARIS21 Partnership in Statistics for Development in the 21st Century PROFOR Program on Forests QAG Quality Assurance Group REDD Reducing Emissions from Deforestation and Degradation RMS Results Measurement System TA Technical assistance UNFCCC United Nations Framework Convention on Climate Change WBG World Bank Group WBI World Bank Institute WDI World Development Indicators vii Cotton plant, Abu Asher, Sudan. Photo by Arne Hoel, courtesy of World Bank Photo Library. Acknowledgments This report was prepared by a core team of Chris Valuable commentary was also provided by Martha Gerrard, Andrew Warner, John Eriksson, Lauren Ainsworth, Hans-Martin Boehmer, Ken Chomitz, Kelly, Nidhi Khattri, and Brett Libresco, under the . April Connelly, Ali Khadr, Nalini Kumar, J.P Singh, leadership of Mark Sundberg. Additional support Yoshine Uchimura, and Richard Worden. was provided by Anna Aghumian, Javier Baez, Arup Banerji, Gayatri Datar, Ximena Del Carpio, The evaluation team greatly appreciates the time Elaine Ooi, Diana Salvemini, Jesse Torrence, and insights given by the many individuals inside Pamela Velez-Vega, and Andrew Waxman. It was and outside the Bank interviewed for this report. processed by Diana Hakobyan and Yvette The evaluation benefited substantially from the Jarencio-Lukban, with assistance from Yezena constructive advice and feedback from many Yimer. The report was edited by William Hurlbut staff in the World Bank Group. and Caroline McEuen. The evaluation was conducted under the guidance Johannes Linn and Susmita Dasgupta served as of Mark Sundberg, Manager, IEG Corporate and peer reviewers of the final evaluation report. Global Evaluation and Methods Unit. Director-General, Evaluation: Vinod Thomas Director, Independent Evaluation Group, World Bank: Cheryl Gray Manager, Corporate and Global Evaluation and Methods: Mark Sundberg Joint Task Managers: Andrew Warner, Part I, Chris Gerrard, Part II ix Terraced rice paddies near a Red Zao village, outside Spa, Lao Cai Province, northern Vietnam. Photo by Tran Thi Hoa, courtesy of the World Bank Photo Library. Foreword Even as the global financial crisis consumes areas, but have shown some improvement in world attention, environmental degradation recent years. The Bank's direct project support remains among the foremost challenges of our for the environment has helped develop market- time. Indeed, the dangers associated with based instruments for environmental manage- climate change have recently been recognized, ment. The Bank has also achieved some gains in but other environmental problems--air and the highly challenging but critically threatened water pollution, soil erosion and desertification, area of biodiversity conservation, although far water scarcity and biodiversity loss--continue to more remains to be done. Mainstreaming these be neglected. The World Bank is uniquely and other environmental programs into country positioned to support country, regional, and strategies and programs remains a crucial gap. global efforts to confront these challenges in a Much more needs to be done in monitoring and way that promotes long-term economic growth evaluation: three-quarters of environment- and poverty reduction. related projects--those managed by sectors other than the environment--lack systematic This year's Annual Review of Development reporting of environmental outcomes. Effectiveness (ARDE) spotlights the Bank's role in helping countries to address these environmental As is customary, the ARDE also reviews the challenges. World Bank support for environment Bank's performance in getting results from the projects has recovered since 2002 with the projects and country programs it supports. The support of new sources of concessional finance. 2008 project performance data show improve- Environmental concerns have received growing ment in achieving development outcomes, attention in Country Assistance Strategies. adding to encouraging evidence on the Bank's Growing numbers of regional projects are long-term performance trends. But the record addressing the shared use of regional bodies of on monitoring and reporting of development water such as the Nile Basin, Lake Victoria, and the results continues to pose challenges for the Black and Aral Seas. New partnerships and facili- institution. The efforts under way to deepen the ties at the global level are deepening Bank involve- results agenda are encouraging. Further progress ment and potential impact on climate change. will be crucial for the international discussions on International Development Association (IDA) Outcomes of environment projects have histori- replenishment and for the post-Kyoto agreement cally been weaker than the Bank's average for all at the Copenhagen Summit. Vinod Thomas Director-General, Evaluation xi Man in a field sifting seeds along Red River in northern Vietnam. Photo by Quy-Toan Do, courtesy of the World Bank Photo Library. Executive Summary his year's Annual Review of Development Effectiveness (ARDE) is being T written against the backdrop of a global financial crisis, declining growth, and massive fiscal stimulus efforts to revitalize markets. Demand for greater development support from the World Bank has grown, along with concerns that resources be used effectively and efficiently to achieve their de- velopment objectives. This ARDE focuses on the Bank's performance record in getting results from its projects and country programs and, as is custom- ary, examines in depth one topic relating to development effectiveness. The focus of this ARDE is on the Bank's support for environmental sustainability, in response to a Board request for a synthesis of findings from recent Inde- pendent Evaluation Group (IEG) reports on the environment. The report shows that the Bank's performance of financing, including resources for global record displays a moderately strong upward efforts, have helped lending and support for the trend over the medium term, and recent environment to recover from the lows of the performance has continued this trend. It also early 2000s; analytic work has fostered innova- shows that there has been a mixed record on tive approaches and enhanced environmental implementing the agenda to strengthen the awareness; and direct support for environmental Bank's orientation, capacity, and monitoring projects is showing improved performance, with of development results--the results agenda. tangible results. But implementation of the Institutional initiatives to strengthen project- strategy in mainstream environmental work level monitoring and evaluation (M&E), to put in across sectors has been weak and must be place a Results Measurement System for the strengthened. Project performance shows a clear International Development Association (IDA), improving trend. and to implement results-based Country Assistance Strategies are at different stages of Project Performance Shows a Clear deployment and have yet to demonstrate a clear Improving Trend impact on outcomes. The 2008 World Bank project performance data shows improvement in achieving development This year's special focus--getting results for outcomes, allaying concerns that the weakened sustainable development--reflects the vital role performance of 2007 could signal a new of sound environmental stewardship for downward trend. The decline in 2007 was development and the grave threat that inaction modest, and performance rebounded in 2008. poses of reversing gains in growth and poverty reduction. The Bank's record in implementing Bank performance is rated on a six-point scale, the 2001 Environment Strategy and advancing from highly satisfactory to highly unsatisfactory the results agenda is quite mixed. New sources (see box 2.1 in chapter 2 for an explanation of xiii ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 IEG ratings). The percentage of satisfactory Efforts to Strengthen Performance projects (that is, the top three ratings) increased Indicators Need Strong and Continuing in 2008, continuing the upward trend of the past Support 15 years. Average performance, which uses the In response to the international drive for full six-point scale, also shows important measurement and management for develop- progress, but the gains are less than half as large ment results, the World Bank has launched major compared with the percent satisfactory measure. efforts to improve performance information in Percent satisfactory tends to give high weight to Bank-supported projects. The Bank has moved borderline changes--those that just pass over to strengthen M&E, introduced a Results the threshold into the moderately satisfactory Measurement System for IDA, and expanded category on the six-point scale. Changes over rigorous evaluation of project interventions. time in the Bank's portfolio favoring better- These efforts are beginning to influence institu- performing Regions and sectors also explain a tional priorities and business practices. However, small part of the improved performance. in many projects information is not sufficiently rigorous or comprehensive to provide stakehold- Analysis of the dates of the major turnarounds in ers with a picture of what really changed as a project performance suggests that a combina- result of the project. tion of better Bank sector policies and improved country circumstances outside Bank control may Project M&E provides building blocks for explain much of the turnaround, rather than generating good information on development internal administrative reforms at the Bank, outcomes and has received increasing Bank although the latter may have facilitated attention in recent years. IEG began to assess improvement already under way. M&E quality in 2006 and has rated the quality of M&E in projects that were reviewed in the past Evaluating the results of analytic and advisory two years as modest or negligible in three-fifths activities (AAA) is inherently difficult. IEG does of cases. These reviews point to several short- not regularly rate the Bank's AAA, although it comings: indicators are too numerous and often underpins projects and accounts for about one- measure outputs rather than outcomes; baseline third of spending on country services. However, data are infrequently collected; few projects a recent IEG evaluation (IEG 2008e) indicates collect the data that would be required to assess that about two-thirds of AAA is moderately impact; and client feedback suggests reluctance satisfactory or better, and one-third is less to adopt project M&E practices and considerable successful. Performance patterns suggest that fragmentation of M&E efforts. These ratings can impact and results for countries depend on only be assigned to projects after they close, and technical quality, concrete and country-specific are therefore lagging indicators. They provide a recommendations, country ownership of the reference point for monitoring future trends and findings, and sustained follow-up after task tracking whether current efforts to improve M&E completion. IEG intends to pilot an approach to work. The low M&E ratings, however, suggest a systematic AAA evaluation. continuing need to revisit the incentives, use, and resourcing of M&E in Bank-supported The 2009 Management Action Record (MAR), projects with a view toward simplicity, tractability, which tracks Bank adoption of recent IEG and decision relevance. recommendations, indicates that adoption levels have declined in the Bank. IEG is currently Shortcomings in M&E design and implementa- examining ways to improve the MAR to create a tion may contribute to problems found in project more effective product for tracking adoption of supervision and reporting. For an average of 12 recommendations and to identify reasons for the percent of projects each year, there is a trend decline. disconnect between the Bank's self-assessments xiv E X E C UT I V E S U M M A RY and IEG project performance ratings. For these a basis for project funding decisions. Although projects, the internal supervision of project part of this change may be attributed to the performance through the life of a project shifting nature of development assistance, not all frequently appears to be overly optimistic and of it is. An assessment of the use of cost-benefit sometimes lacks candor regarding risks to tools in environmental projects largely confirms development objectives. Delays in accurate this conclusion. This issue will be taken up in reporting lead to final supervision assessments greater detail in an IEG special report in fiscal that come too late for corrective action to be year 2010. taken. Support for Environmental Sustainability An outgrowth of the IDA negotiations since 2003 Is Expanding but Needs to Go Much has been development of a Results Measurement Further System to measure and aggregate development Addressing environmental degradation is one of outcomes in a way that is relevant to stakehold- the central challenges of our time. The Bank is ers and donors. This is a commendable effort to uniquely positioned to support country, regional, track and report results in a coordinated fashion. and global efforts to address environmental A brief assessment in advance of the November challenges such as climate change, air and water 2009 IDA Midterm Review, based on documen- pollution, and deforestation and biodiversity loss tary evidence, staff interviews, and a survey of in a manner compatible with economic growth IDA donors, suggests that the initiative has made and poverty reduction. Greening the growth path progress, yet faces significant challenges to fuller need not come at the expense of growth or of implementation. helping the poor. For example, there are exten- sive untapped opportunities to reduce green- Impact evaluation is gaining much greater traction house gas emissions through incentives for in the Bank as a tool that holds promise for provid- greater efficiency. Moreover, providing 2 billion ing estimates of project efficacy and impact. Over poor people with basic electricity access would 250 impact evaluations are under way in the Bank, add little to world greenhouse gas emissions-- many in the Human Development Network. A less than 0.4 percent of current emissions, even if review of recently closed projects in the Agricul- carbon-intensive means are used. ture, Environment, and Water Sectors indicated that few collected even the minimum information This year's ARDE spotlights the Bank's activities to assess results, and projects with baselines and performance in helping countries to address amenable to analysis are concentrated in the East these environmental challenges, based on the Asia and Europe and Central Asia Regions. Thus findings of recent IEG evaluations and supple- requisite information for basic quantitative mentary analysis. These evaluations include, but analysis is still rare. IEG will initiate analysis over are not limited to, the following: Environmental the coming year to better understand the quality, Sustainability: An Evaluation of World Bank relevance, and costs of the current impact Group Support; Climate Change and the World evaluations in the Bank. Bank Group, Phase I: An Evaluation of World Bank Win-Win Energy Policy Reforms; The Finally, economic cost-benefit analysis is the Development Potential of Regional Programs: traditional method, long practiced in the Bank, An Evaluation of World Bank Support of for organizing information on project benefits Multicountry Operations; and a forthcoming and costs in a manner relevant to decision review of the Bank's forestry activities. making. The Bank's use of cost-benefit analysis peaked in the early 1970s but is now applied to Bank management has begun to initiate new only about one-quarter of projects. This indicates strategies for the World Bank Group, including that cost-benefit criteria are now used far less as for climate change, energy, and the environment. xv ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 This ARDE reflects on the findings of the body of that weigh against more complex multicountry IEG's recent evaluation work in the area of the programs. environment, but also goes beyond these findings by updating the portfolio and perform- Analytic work has also lent support to key ance data through fiscal year 2008 and supple- environmental initiatives across a broad menting materials from IEG's project, country, spectrum. Examples include work on improving and global evaluation products. policies and institutions for sustainable forest management and on developing carbon markets The Bank's environment strategy, launched in and project-based technical knowledge that has 2001, furthered the shift in the Bank's stance helped development of internationally approved from "do no harm" to active promotion of methods for the certification of emissions environmental goals, a shift that began in the reductions. early 1990s. This created the potential for much greater effectiveness through mainstreaming There are areas of success, but significant attention to the environment across the Bank's internal and external forces constrain the Bank's project portfolio and country assistance strate- environmental portfolio. Weak country demand gies. It is too early to assess the 2007 organiza- for loans arises from the public good nature of tional changes intended to facilitate main- many environmental interventions. Where the streaming and consolidate environment into the benefits can be locally captured, the incentive to Sustainable Development Network. Preliminary act is stronger, as in the case of pollution control indicators suggest that mainstreaming has in China. Political competition and corruption decreased in some sectors, such as agriculture, surrounding resource rents may also constrain energy, and transport, and that cross-sector demand. Knowledge and capacity constraints, collaboration has been particularly weak between particularly where Bank support has wavered water supply and health. over time, inhibit lending. Internal knowledge gaps, inadequate technical and operational skills Since 2001, new funding sources--including to integrate environment considerations into carbon funds, expanded Global Environment investment and policy reform projects, and poor Fund resources, and new trust funds--have dissemination of evidence on effectiveness contributed to a recovery in Bank support for the within the Bank impede effectiveness and limit environment. This funding has also shifted the growth. Finally, internal staff and management focus of work from traditional mainstreamed incentives favor large projects, such as infrastruc- activities such as pollution management toward ture or power, which disadvantages the typically innovative approaches relating to climate smaller environmental projects. change. The Bank is currently stepping up its role in helping finance solutions to tough issues Some Noteworthy Results Have Been such as climate mitigation and adaptation or Achieved, but Weak Monitoring biodiversity losses, where the institution can Compromises Results Measurement potentially make a vital difference. The overall outcomes of environment projects, like the Bank's portfolio as a whole, have been Although constituting only 5 percent of total improving over time. But projects managed by environmental commitments, regional (multi- sectors other than the environment, which country) environmental projects have been a comprise most of the Bank's support for the promising area of growth. Regional environmen- environment, generally lack systematic reporting tal commitments more than doubled over the of environmental outcomes. past six years; the largest volume of commitments is in Africa. The regional pilot initiative, The Bank's direct project support for the introduced in IDA14, provided additional grant environment has helped develop market-based financing from IDA to help overcome incentives instruments for environmental management and xvi E X E C UT I V E S U M M A RY has extended lessons on the design of these designed to promote energy efficiency, cli- systems throughout Latin America and into Africa. mate change adaptation, and greenhouse gas The Bank has also achieved some gains in the emissions reduction from deforestation and land challenging but critically threatened area of degradation--such as the Climate Investment biodiversity conservation. It has been the world's Funds and the Forest Carbon Partnership largest source of support for biodiversity, includ- Facility--are deepening Bank involvement and ing blended finance from the Global Environment potential impact on climate change issues. Facility. This has provided support for a significant expansion of protected areas as well as for the Conclusions sustainable use of biodiversity outside these areas, Project outcomes represent a key building block but more attention needs to be paid to the in achieving results through World Bank support effectiveness and sustainability of these efforts. to countries. These outcomes have shown signif- icant improvement over the past 15 years. Bank- Environmental results are not limited to projects. wide, there has been a much greater focus in At the country level, environmental concerns recent years on results, yet there remains a long have received growing attention in assistance distance to go to demonstrate a real change in strategies and country program evaluations, but measurable outcomes and impacts. Because the performance has been found wanting. Envi- volume of Bank lending is expected to increase ronmental components, on average, have in response to the current financial crisis, this performed less well than overall country strate- agenda becomes all the more vital. gies. The Bank's record on environmental stewardship has been uneven. For example, the Environmental sustainability is one of the most Bank has helped some countries reform energy compelling areas for urgent action if develop- pricing by supporting the removal of costly ment results are to be delivered at the country, subsidies while protecting the poor through regional, and global levels. Bank support for the social safety nets, but has only given modest environment has partially recovered since attention to national policies to promote energy hitting a low point in 2002, and there are encour- efficiency. aging signs that project results, long below Bank-wide averages, are now improving for the The growing body of regional projects offers the roughly one-quarter of environmental projects opportunity to address transnational environ- whose results are systematically tracked. For the mental challenges surrounding the use of shared other three-quarters of projects, where environ- bodies of water such as the Nile Basin, Lake mental initiatives are embedded in other sectors Victoria, and the Black and Aral Seas. The Bank (transport, water and sanitation, and so on), the has been effective in fostering multicountry promise of mainstreaming that emerged from cooperation to establish regional institutions to the Bank's 2001 Environment Strategy has not address regional environmental challenges been realized. through analytical work, project preparation, and resource mobilization, but it has been less In some increasingly important and innovative effective in helping countries resolve long- areas, such as the development of carbon funds standing resource conflicts. and the extension of protective areas to support sustainable forestry and biodiversity, there are At the global level, there is evaluation evidence gaps in our knowledge of where the Bank is that global programs are adding value to country making headway and achieving real results. programs by providing concessional financing Progress will depend on far better mainstreaming for country-level investments that have global of the environment into Bank decisions across environmental benefits and by generating sectors and on addressing issues of in-house knowledge about best practices in environmen- skills, staff incentives, and external demand tal management. New partnerships and facilities constraints that impede Bank effectiveness. xvii ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Looking ahead, crises offer a rare chance for key environmental areas and to facilitate more transformational change, change that is essential effective development outcomes by leveraging its to achieving the vision and potential for Bank knowledge, financial resources, and convening support for environmental sustainability. The authority. challenge for the Bank is to increase attention to xviii Management Comments anagement welcomes the findings in the 2009 Annual Review of De- M velopment Effectiveness (ARDE) by the Independent Evaluation Group (IEG). It is a great strength of the Bank that it subjects itself to independent evaluation and uses the findings to enhance its development effectiveness by learning from experience and that IEG consistently ranks as the benchmark for independent evaluation in the development field. The risk, of course, is that the Bank's effective- projects in fiscal year 2008 rated as satisfactory ness as measured by IEG in its evaluations may (moderately satisfactory, satisfactory, or highly be taken out of the context of the challenges satisfactory) in terms of meeting development inherent in development assistance or compared objectives, exceeding management's target of 80 across agencies with less strong evaluation percent. Management also notes that the ARDE functions. The ARDE differs from most IEG confirms the continuing underlying positive outputs in that it draws on a wide range of evalua- trend in the quality of the operations supported tive findings and provides a summary overview. by the Bank. The ARDE goes back to 1993 to look Because of the nature of the ARDE, it does not at the evolution of Bank performance; clearly, the make specific recommendations for manage- Bank of today is very different from the Bank in ment and does not elicit specific management 1993. The focus on results is much more central responses to issues raised in the document. now. As an example, monitoring and evaluation However, management finds it important to (M&E) is a key element for monitoring projects provide its views on ARDE findings and to give an to achieve results, and the ARDE cites a weakness overview of a set of key management actions that in Bank M&E for projects exiting the portfolio in are directly relevant to those findings. fiscal years 2007 and 2008, projects that on average entered the portfolio early in this Organization of the Comments. This note first decade. More recent experience is different: 59 provides comments on the findings and sugges- percent of new IDA projects that filed supervi- tions on the Bank's overall performance in part I sion reports for the first time in fiscal 2009 had of the ARDE. It then responds to what IEG puts baseline data for all project development forward in part II on the special topic for this outcome indicators and 85 percent had baseline year, sustainable development. The final section data for at least one. Management expects a of the note provides a brief set of conclusions. significant change in IEG findings on M&E when these operations exit the portfolio in the coming Tracking Bank Performance years. However, it is important that the Bank do Management is encouraged to note the rebound more to adapt its work to enable it to be more in the quality of Bank-supported operations agile and innovative in supporting the needs of reported by IEG, with 81 percent of exiting borrowing countries, in particular paying more xix ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 attention to implementation and to adapting implementation. Those changes, which will support quickly in response to new information focus management and staff attention more on or changing circumstances and to improve implementation, are designed to overcome the performance in lagging Regions and sectors. problem of overoptimism cited by IEG. Addressing Long-Standing Issues--the Bank's Project Restructuring. IEG cites the issue of an Results-Focused Reform Efforts apparent reluctance to restructure projects that To address the challenges of a changing Bank and are not meeting development objectives. A key to take on long-standing issues, including those part of investment lending reform is a reworking identified in the ARDE, management has an of the confusing guidelines for project restruc- extensive results-focused response effort under turing, which staff cite as a major factor in way across virtually the whole spectrum of Bank delayed restructurings. Bank executive directors operational support--from Country Assistance will be asked in the coming months to approve a Strategies (CASs) to investment lending, develop- simplified restructuring process. ment policy operations, and analytic and advisory activities (AAA). Development Policy Operations. The framework for development policy operations was modernized CASs. During fiscal 2010 the Bank will implement in fiscal 2005. To date, results from self- and a set of actions based on a recent review of independent evaluation have been robust (as experience with results-based CASs. These reconfirmed in the ARDE). Recent self-evaluation actions include guidance on how to rate Bank has identified ways to further strengthen performance under the preceding CAS and on performance. In fiscal 2010, management will preparing results frameworks for new CASs. The provide stronger guidance to staff on supporting CAS Progress Report process will be strength- borrowers on the design of development policy ened, providing an early opportunity for gauging objectives (including in the context of joint progress and the potential need for adjustments. budget support with other donors) for develop- Guidance has already been issued on assessing ment policy operations and in consolidating good country statistical capacity--an issue cited by practice on poverty and social impact analysis. IEG as important for the results agenda. Cost-Benefit Analysis and Impact Evaluation. The The Investment Lending Reform Program. Several of ARDE cites the decreased use of cost-benefit the issues raised by the ARDE are ones that analysis and the potential of impact evaluation. management is addressing through investment Management looks forward to working with IEG lending reform (World Bank 2009). Specifically, on its upcoming in-depth review, notably on these reforms are aimed at adapting and differ- when cost-benefit analysis is most appropriate entiating the investment lending model to better and how useful it is relative to or along with fit the kind of support that the Bank provides in alternative tools for appraisal, monitoring, and sectors such as health and public sector manage- evaluation across the spectrum of investment ment, where there is no "engineering blueprint" projects supported by the Bank. Management or way to calculate a credible expected economic agrees with IEG that impact evaluation can be a rate of return, but a need for a more flexible valuable tool and has recently expanded its use design and more emphasis on support to across sectors and Regions in operations where partners during implementation. The central it is most appropriate and cost-effective and tenet of the reforms is a stronger focus on a owned by the client country. Again, management clearly defined set of results the project is looks forward to working with IEG in 2010 on its designed to achieve (with well-defined monitor- impact evaluation analysis. ing indicators), combined with the identification of the key risks to achieving those objectives and AAA. Management is taking on reforms to its a regular reevaluation of these risks throughout results framework for AAA in the context of its xx M A N AG E M E N T C O M M E N T S work on a new knowledge agenda. The approach other donors, the Bank is launching a new Statis- is to look at AAA in a more programmatic fashion, tics for Results Facility to help the poorest as opposed to individual products. Management countries build capacity to increase the availabil- is actively working with IEG on the new AAA ity of development statistics, starting with those results framework, because a common approach that are most urgently needed for the implemen- and understanding is central to strong self- and tation of strategies for poverty reduction and independent evaluation. These are sequenced for national development. More broadly, country later in the reform program timetable. (In general, capacity building has always been part of Bank management would note the importance of activities, especially since Accra; for example, the sequencing these reforms to keep them within Bank-wide emphasis on greater use of country staffing and budget constraints.) systems also includes a greater reliance on and development of country M&E systems. (IEG is The Results Agenda Going Forward. To consolidate also active in this area through activities to build all of the above, management is working to evaluation capacity). However, we recognize that establish a "results monitoring platform" to this is a longer-term effort. strengthen monitoring and reporting on the outcomes of Bank-supported activities. One key Lagging Regions and Sectors step is the adoption of core sector indicators--a IEG, while commending the Bank on the few standardized indicators for each sector that increase in operational quality, notes a history of are comparable and aggregable at the institu- poor performance in certain Regions and tional, country, or sector level, which will facili- sectors. Work is ongoing to address this issue. In tate reporting on activity outputs and outcomes, particular, management has taken on two, the especially under the IDA Results Measurement Africa Region and the health sector, as a priority. System (IDA RMS). The core indicators for the road transport, water supply, education, and The Africa Region's Quality Enhancement Program. health sectors were launched in July, together To improve outcomes of the operations it with an improved data system for collecting and supports, the Africa Region has taken the tracking quantitative data. Core indicators for following actions: other sectors (urban, environment, micro, small, and medium enterprises, and so on) are also (i) Restructuring the weakest-performing port- under preparation. These indicators (mainly folios (HIV/AIDS, regional integration, fragile output and outcome indicators appropriate for states) tracking at the project level), together with the (ii) Providing just-in-time operations and M&E country- or sector-level indicators available from support to country and task teams to im- other sources (for example, Tier 1 indicators), prove the results frameworks and ensure bet- will improve our ability to focus on results. In ter compliance with IDA15 monitoring addition, the new results platform offers staff and requirements managers tools for monitoring country portfolio (iii) Training task team leaders in the development results and for making more strategic and of results-focused operations and programs effective use of impact evaluations where these and results-based M&E systems are appropriate. IDA at Work narratives are also (iv) Developing sector results chains to help im- being expanded, with an emphasis on thematic prove the underlying causal logic in designing and country level briefs. operations (v) Developing the Africa Results Monitoring Sys- Country Capacity. Management agrees with IEG tem (ARMS), which is now being rolled out on the importance of building country M&E as a Bankwide system to track results for IDA capacity, both as a goal in itself and to enable us credits. The ARMS entails the development to better understand our own development of clear and realistic outcome-based project effectiveness. For example, in partnership with development objectives as well as standard- xxi ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 ized and measurable indicators. It will har- with a recommendation, or when, as in the case monize and facilitate results reporting. of the IEG evaluation of analytic and advisory services, management noted that it would need The ARDE provides an early indicator of progress. to sequence many of the actions later in the It cites fiscal 2008 data showing 77 percent of overall reform effort. Another key issue is the projects in the Africa Region as satisfactory sheer number of recommendations and subrec- (moderately satisfactory, satisfactory, or highly ommendations. The current MAR contains 63 satisfactory) in terms of meeting their develop- separate recommendations, often with a signifi- ment objectives, up sharply from fiscal 2007. cant number of subrecommendations under each recommendation. (In addition to IEG, The Health Sector. The Bank's Health Network has management commits to take action in response taken action to improve the outcomes of country to the findings of other entities, including the health sector operations it supports. Based on Inspection Panel, the Internal Audit Department, the findings of an IEG evaluation of the health and the Integrity Vice Presidency.) Management sector, it developed a portfolio-quality improve- would like to work with IEG on a system that ment plan and launched it in fiscal 2009. It is reduces and prioritizes IEG recommendations regularly monitored by senior management to and monitors progress directly against ensure that it stays on track. management commitments. The Management Action Record (MAR) Achieving Sustainable Development As noted above, management values independ- Management welcomes IEG's attention to the ent evaluation and takes it very seriously. As a issue of sustainable development. The ARDE result, it takes seriously the recommendations draws on a range of findings, including two that come out of IEG evaluations. All Draft recent IEG evaluations. As management noted in Management Responses are discussed by its responses to those evaluations, there are executive directors (normally at the Board's several areas of agreement: the centrality of Committee on Development Effectiveness-- environmental sustainability in the Bank's work, CODE). These documents include an initial MAR the importance of country ownership, the key that sets out what management commits to do as role for analytic support to countries, and the a result of the IEG recommendations. Given the need for strong collaboration across sectors and importance of these commitments, which often across World Bank Group institutions on have resource implications, they are vetted and environmental sustainability issues. At the same cleared by senior management. Management time, management expressed concerns about then revises these commitments in light of the some of the analysis and recommendations (IEG CODE discussion. Management reports annually 2008c, 2009a). However, it is more important to to IEG on progress against its commitments. note how management is using the lessons While this system is valuable and necessary as an summarized in the ARDE going forward. accountability tool, management agrees with IEG that it is time for a reevaluation. Bank Progress on Environmental Mainstreaming. The ARDE emphasizes that environmental improve- Possible Improvements to the MAR. The biggest ment requires mainstreaming environmental issue is that in its annual monitoring of progress, awareness into investment decisions and technol- IEG starts with its recommendations, rather than ogy choices across sectors. Management believes with the part of the matrix that sets out that the criterion on which Bank performance for management's commitments--the actions environmental mainstreaming is now measured-- management agreed to undertake and for which confluence of projects with environmental it can be held accountable. That approach often themes with sector board mapping--is a second- leads to disagreements over ratings on progress, best indicator. Background work for the Environ- notably when management agreed only partially ment Strategy 2010 will address how to improve xxii M A N AG E M E N T C O M M E N T S environmental mainstreaming results and will also Bank Group should engage to support its clients assess to what extent, after only two years, the in more effectively addressing environmental merger of the two Bank networks into the Sustain- priorities and moving toward greater environ- able Development Network is leading to stronger mental sustainability and will help inform the mainstreaming outcomes. Management is contin- development of a results framework for measur- uing to strengthen the ability of staff in the areas ing progress in priority areas. In addition, the of forestry, natural resource management, and strategy will address how the World Bank Group energy to deliver integrated climate change­ can better support client countries in their based advice to clients by supporting learning efforts to strengthen the environmental institu- modules on climate change­related tools and tions and governance mechanisms that are mechanisms, such as ecosystem-based adaptation crucial for sustainable development. and Reducing Emissions from Deforestation and Degradation (REDD). Conclusions Management appreciates that submitting the Measuring and Achieving Project, Country, and Bank to the learning and accountability discipline Global Results. The ARDE emphasizes that in the of independent evaluation is a strength that sets new Environment Strategy, the Bank should it apart from many other development institu- include a focus on better monitoring and report- tions, notably given IEG's acknowledged standing ing of environmental outcomes at the project as best among its peers. It is confident that the and country levels. In the IDA15 RMS exercise, benefits are worth the costs of being held to this management is making progress in creating high standard, with the resulting misunderstand- environmental outcome indicators for each ings that can arise. Even in a balanced report like environmental subtheme, all of which will be this one, the positive outcomes risk being aggregated across sectors. Country-level environ- overlooked relative to the less positive findings. mental reporting is under development in a Management accepts the overall message from "green" data sets reporting format. Management the ARDE--a finding of generally good and agrees on the importance of expanding environ- improving quality in the operations the Bank mental partnerships, especially for strengthen- supports, but also a strong challenge to do even ing environmental governance, across all better, notably in supporting a results culture in relevant institutional actors, especially at the partner countries and in the Bank and strength- country level. To this end, the preparation of the ening Bank support for sustainable development. Environment Strategy 2010 will involve extensive These are priorities that management has taken consultations with such external stakeholders as on in its fiscal 2010 work program and beyond, governments, the private sector, civil society, and notably in a set of actions that address some long- development partners. The consultations will standing issues found by both self- and seek to identify key areas in which the World independent evaluation. xxiii People gathered under a tree, Nigeria. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Chairperson's Comments: Committee on Development Effectiveness (CODE) n July 15, 2009, the Committee on Development Effectiveness met to O consider the Annual Review of Development Effectiveness (ARDE) 2009: Achieving Sustainable Development, prepared by the Independent Evaluation Group (IEG), together with Draft Management Comments. This re- port to the Board follows the structure of the report -- part I: Tracking Bank Performance, and part II: Achieving Sustainable Development. Part I: Tracking Bank Performance addition, members believed that IEG and manage- ment should work together on ways to improve Overall Summary the current MAR structure. Overall, the Commit- The discussion followed the proposed Chairman's tee felt that the ARDE as well as other IEG reviews Issues Note on development outcomes of Bank provide a valuable contribution to CODE's support; implementation of the results agenda; oversight role. and status of the Management Action Record (MAR). First, on development outcomes, the Recommendations and Next Steps Committee took note of the Bank's upward To enhance the implementation of the results performance record. During the discussion some agenda, CODE recommended the following: questions were raised on IEG's rating system and methodologies. Second, on the results agenda, · Strengthen measurement efforts at the project, the Committee stressed the need for the Bank to sector, and country levels. have a refined and simple evaluation model; · Improve coordination between IEG and man- adequate incentives to monitor development agement on the design of an effective World results; cost-benefit analysis where appropriate as Bank Group approach. a basis for project decisions; and appropriate staffing and financial resources. Third, the need to Next steps are: improve the context and structure of the MAR resonated among many speakers given the · The ARDE will be considered by the Board on decline in the level of adoption of IEG September 10, 2009. recommendations. To enhance the follow-up of · A CODE meeting on IEG methodologies will IEG recommendations, it was suggested that be scheduled in due course. CODE should play a constructive role where ­ IEG will submit to CODE or CODE Sub- differences of opinion emerge between IEG and committee an evaluation on economic management on specific recommendations. In analysis in projects in March 2010. xxv ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 ­ For next year, CODE proposed consolidat- recommendations. In this regard, they stressed ing the discussion of IEG's ARDE together the need to examine how to improve the content with the Annual Report on Portfolio Per- and structure of the MAR, assess the relevance formance (ARPP) prepared by Quality As- and timing, prioritize recommendations, and surance Group (QAG). monitor progress, although one speaker cautioned against creating unnecessary bureau- The following main issues were raised at the cracy. To this end, they recommended that IEG meeting: and management work together to enhance the MAR's effectiveness. In this regard, some Development Outcomes. Members commended members raised questions about CODE's management on the upward trend in perform- oversight role, including monitoring of the MAR. ance in recent years but noted that there were lagging Regions and sectors and room for Other Issues. There were comments on the improvement. Questions were raised on the importance of reconsidering the role of part II of adequacy of the current rating system, particu- the ARDE, including reviewing similar evalua- larly for difficult or complex projects, and IEG tions undertaken by other multilateral agencies methodologies. There were recommendations and reconsidering the theme selection, or for more in-depth analysis of causality to dropping it altogether. understand the impact of the Bank's projects, systematic evaluation of analytic and advisory Part II: Achieving Sustainable activities (AAA), which may contribute to the Development knowledge agenda, and integrated reporting on development outcomes at the World Bank Group Overall Summary level. IEG highlighted the difficulties in having Some members wondered what the value added one report covering the three institutions--the was to discussing sustainable development as Bank, IFC, and MIGA. part II of the ARDE, given the extensive coverage of this topic by IEG reviews and management Results Agenda. Members took note of the mixed reports discussed recently by CODE and the records identified by IEG in implementing the Board. IEG replied that the choice was a results agenda, while acknowledging the response to a Board request to look at recent IEG challenges of reforming the monitoring and reports on environment and synthesize their evaluation (M&E) system. In this regard, the findings before management's new environment importance of cost-benefit analysis of projects, strategy was prepared. Members questioned the which should provide relevant information for real meaning of "mainstreaming environmental decision making, was underlined. Members work" and the application of existing tools. Some queried the adequacy of human and financial members cautioned against overemphasizing resources and the incentive systems for staff and the tension between national programs and clients to strengthen M&E of development results. environment, as the latter also features strong Questions were also raised on how to measure national and local dimensions. They stressed the the effectiveness of the Bank's assistance in importance of developing awareness and relation to other factors (such as country circum- leverage on national and local instruments and stances) at the project and country levels; how to policies. They also queried IEG's findings about be selective and consolidate M&E efforts; and how "weaknesses" in the system and "less than to strengthen accountability while addressing adequate" implementation of mainstreaming overoptimism in reporting project performance and management's comment on the methodol- during implementation. ogy used to evaluate environmental mainstreaming; that is, using a second-best MAR. Members raised concerns about the indicator that relates projects with an decline in the level of adoption of IEG environmental theme to sector board mapping. xxvi CHAIRPERSON'S COMMENTS: COMMITTEE ON DEVELOPMENT EFECTIVENESS (CODE) Recommendations and Next Steps port (for example, on water, energy, and land The Committee invited IEG to have preliminary management). consultation on the thematic topic of part II of next year's ARDE to gain a broader support for Bank's Role. Members felt that the Bank should the selected option. have an integrated approach, starting from the project concept and design, to ensure a more The following issues were raised in part II of the effective implementation of environmental proj- meeting: ects. In this regard, it was underlined that weaknesses in staff training and incentives may Environmental Sustainability. Some members inhibit the impact of Bank's work in this field. stressed the importance of a demand-driven approach and adequate measurement systems. Organizational Structure. A few members raised They raised questions about how to integrate concerns about IEG's finding of an absence national programs and global public goods and of environmental themes in health projects, short-term needs with longer-term objectives; which seems to reflect poor communication and and how to operationalize environmental policies partnership between the Sustainable Develop- more effectively. In addition, they underscored ment Network and the Human Development the cross-sectoral nature of environmental sup- Network. Giovanni Majnoni, Chairman xxvii Chapter 1 Schoolboys in India. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Introduction his year's Annual Review of Development Effectiveness (ARDE) is T about achieving results from Bank support--is it happening, and how do we know that it is? The search for development results is at the cen- ter of the debate over aid effectiveness. It is driving changes in the interna- tional aid architecture, including a shift away from multilateral agencies toward vertical funds and earmarking of assistance--through global funds, non- governmental organizations (NGOs), and donor agencies, as well as ear- marked multilateral channels that taxpayers and philanthropists think will enable their money to yield more tangible results.1 Of course, it is difficult to know if an institution is the development agenda and a central challenge achieving tangible results unless results are to getting results: environmental sustainability. measured and compared across agencies, Sound environmental stewardship is known to be sectors, or projects. This is why, although pivotal for sustainable development and is one of measurement and evaluation are dull topics for our greatest collective challenges. Neglecting it most people, they are at the heart of answering may pose the greatest threat to development by whether the finances are being transformed into unraveling gains in growth and poverty reduction. actions that make a difference in people's lives. But achieving and monitoring environmental results is exceedingly complex because of its This matter has been hotly debated during the cross-sectoral nature, involving health, infrastruc- current crisis and emergency responses. Chinese, ture, education, and energy, among other con- American, or Brazilian citizens want to know cerns. Mainstreaming environmental work and whether stimulus funds are being properly used tracking results across such diverse Bank- and are effective in getting the global economy supported interventions was an explicit objective growing again. The Bank's fast-track loans to of the Bank's 2001 Environment Strategy (World governments are similarly under scrutiny. Quick Bank 2001). However, this central strategy of answers are not available because evaluation mainstreaming has been weakly implemented, takes time and is, by its nature, backward looking. and numerous internal and external forces have But learning from evaluation can often guide constrained the Bank's ability to achieve better efforts to improve results in the future. results. A hidden risk of the short-term financial crisis is The methods and systems for measuring results that it may divert attention from a vital element of are imperfect. Probing into the reporting by aid 3 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 agencies, NGOs, and global development funds allocation remains daunting. Bank-wide results on what they have accomplished, one soon finds measurement tools are still being developed and that there is little precision about how they have not yet been tested. Other performance measure their results and whether the measure- evaluation tools (such as impact evaluation, cost- ment is accurate, or even meaningful. This report benefit analysis, and real-time monitoring) appear aims to advance debate on the World Bank's to be used sporadically, although more in-depth contribution in this area. assessment is ongoing. The 2009 ARDE has two parts. Both parts assess Part II offers an assessment of Bank performance Bank performance in advancing development in the critical area of sustainable environmental effectiveness through different performance results, particularly Bank support since 2001. Here lenses. Part I looks first at Bank performance using too the record is mixed: the portfolio has been conventional tracking measures. The story is growing, and catalytic areas of excellence for broadly positive: performance, measured as the environmental initiatives are evident, but experi- satisfactory achievement of development out- ence with mainstreaming and monitoring results is comes that are targeted in Bank-supported weak. Serious efforts to move the environmental projects, is high and has been gradually improving agenda forward and to help deliver on the global for the past 15 years. Part I then examines efforts hopes and aspirations of the 2009 Copenhagen to improve performance data. Here the progress Summit will hinge in part on renewed efforts to is mixed: the challenge of transforming project strengthen environmental mainstreaming and to monitoring and evaluation (M&E) practices into address both Bank and government disincentives meaningful tools for management and resource to encouraging greater cross-sector collaboration. 4 PART I Tracking Bank Performance Chapter 2 Evaluation Highlights · Project performance has shown steady improvement over the past 15 years. · Project performance recovered in 2008 to 81 percent satisfactory, driven by strong performance in trans- portation and water and in Africa, South Asia, and Latin America. · Social development, environment, urban development, and social pro- tection have shown strong im- provements over the past three years. · New evidence suggests that im- provements in project performance over the long term were in part the result of Bank sector policies and external conditions. · Better evaluation of the quality and impact of analytic and advisory ac- tivities is greatly needed. Men fishing in Colombia. Photo by Edwin Huffman, courtesy of the World Bank Photo Library. World Bank Performance in Perspective he Independent Evaluation Group (IEG) has traditionally tracked World T Bank performance using a simple metric of the percentage of projects rated satisfactory (moderately satisfactory, satisfactory, and highly sat- isfactory) as opposed to unsatisfactory (moderately unsatisfactory, unsatisfactory, and highly unsatisfactory) from all projects that have closed in the preceding year (box 2.1). Bank-supported country programs are rated by a similar metric. This chapter uses these measures to examine The drop in project ratings between Project performance recent performance and to assess whether the 2006 and 2007 of nearly 7 percentage recovered in 2008, downturn identified last year represented a new points (the sharpest drop since IEG allaying concerns over declining trend. It also places recent perform- began tracking performance) received the 2007 drop. ance in an historical perspective by examining attention in last year's annual report longer-term drivers of performance and events and raised concern about a turnaround trend. associated with the major turnaround in But performance recovered in 2008, as shown in performance. Finally, the chapter presents a figure 2.1.3 discussion of the performance of the Bank's analytic and advisory activities (AAA). Box 2.1: IEG Outcome Ratings Project Performance Recovered in 2008 To rate project performance, IEG uses project- IEG rates project outcomes according to the extent to which the pro- generated data to determine the extent to which a ject's major relevant objectives were achieved, or are expected to be project has met (or is likely to meet) its stated achieved, efficiently. Possible ratings are: objectives. Box 2.2 provides illustrations of what satisfactory and unsatisfactory mean in a project 1. Highly satisfactory: there were no shortcomings in the operation's context. The objectives of projects are important achievement of its objectives, in its efficiency, or in its relevance. to development outcomes such as growth, poverty 2. Satisfactory: there were only minor shortcomings. reduction, and other social and environmental 3. Moderately satisfactory: there were moderate shortcomings. goals, or indirect contributors to these outcomes.1 4. Moderately unsatisfactory: there were significant shortcomings. 5. Unsatisfactory: there were major shortcomings. Among Bank-supported projects that closed in 6. Highly unsatisfactory: there were severe shortcomings. fiscal 2008, 2 81 percent were rated satisfactory. 9 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Box 2.2: What Does a Satisfactory or Unsatisfactory Project Look Like? IEG rates project outcomes according to three criteria--their rel- project to control the spread of HIV infection within high-risk evance, efficacy, and efficiency. In general, satisfactory projects groups and to limit its spread into the general population. This fully deliver on their development objectives, appear likely to be project did not meet its development objectives and was rated un- sustained, and generate clear benefits in an efficient manner. satisfactory. Efficacy in controlling the spread of human immun- Some examples help to illustrate. odefiencey virus (HIV) among high-risk groups was modest. Coverage of brothel-based sex workers with prevention programs Burkina Faso was high, but it was lower among street-based sex workers. Con- In Burkina Faso, the Bank partially financed a $270 million water dom use among both sex worker groups exceeded the target of supply project in Ouagadougou. The project was rated highly sat- 50 percent, but it was lower among hotel-based sex workers. isfactory, fully meeting its development objectives of increasing re- While needle sharing by injection drug users declined in some cities, liability of, and access to, clean drinking water for 680,000 it rose in others, and the target of 95 percent not sharing needles low-income people in peri-urban areas, as well as strengthening was not met. Coverage of men who have sex with men only reached urban water sector management. Surveys showed happy con- 15 percent, against the 100 percent target. Efficacy in preventing sumers--85 percent were satisfied or highly satisfied with the the spread of HIV in the general population was also modest. All quality and availability of water--as well as efficiency gains--the blood was screened before transfusion in the targeted blood trans- National Office for Water and Sanitation cut in half accounts re- fusion centers, besting the 20 percent goal. However, there is no ceivable from private consumers. comprehensive vision or plan for a safe and sustainable blood trans- fusion system countrywide, raising doubt whether the nonproject Bangladesh centers will benefit and if the country is on track for a safe blood In Bangladesh, the Bank financed the majority of a $27 million supply. Performance data by Region and sector development tend to receive the lowest ratings, are reported in figure 2.2. Of the sectors, while projects in transport tend to receive the health, nutrition, and population; public sector highest ratings (see appendix table A.1 for annual governance; and financial and private sector data). Figure 2.1: World Bank Project Ratings Have Improved since 1993 Percent of projects rated satisfactory or better 85 (weighted by number of projects) 80 75 70 65 60 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 Exit year Percent satisfactory Five-year moving average Source: World Bank data. 10 WO R L D BA N K P E R F O R M A N C E I N P E R S P E C T I V E Much of the year-to-year variation in performance with 2003­05 gives a more complete picture: by sector, when viewed in isolation, is not statisti- social development, environment, urban develop- cally significant and not representative of any ment, social protection, and transport all showed particular trend. Comparing three-year averages performance improvements of 10 percentage (to reduce the noise in annual data) for 2006­08 points or more, while education and health Figure 2.2: Percent Satisfactory Outcome Ratings by Region and Sector, 2006­08 versus 2003­05 a. Outcome by Region East Asia and Pacific 13 Percentage point change in performance Latin America and the Caribbean 5 Europe and Central Asia 0 World Bank average 4 Region South Asia 1 Middle East and North Africa 3 Africa 4 0 10 20 30 40 50 60 70 80 90 100 Share of projects with outcomes of moderately satisfactory or better (weighted by number of projects) b. Outcome by Sector Board Transport 10 Poverty Reduction 7 Social Protection 12 Change in performance (percent) Water 1 Economic Policy 5 Agric. & Rural Dvlpmt Sector Board 5 Urban Development 12 Environment 22 Social Development 23 Energy and Mining 6 Education Financial & Private Sector Dvlpmt 15 1 Public Sector Gov. 5 Health, Nutrition & Population 10 0 20 40 60 80 100 120 Share of projects rated satisfactory or better (weighted by projects, percent) 2003­05 2006­08 Source: World Bank data. 11 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table 2.1: Average Annual Changes in Ratings between 1993 and 2008 15-year change in Average annual change percentage points in percentage points Standard error T-ratio (1993­2008) Highly satisfactory 0.18 0.087 2.04 2.7 Satisfactory 0.025 0.15 0.17 -- Moderately satisfactory 0.97 0.14 6.91 14.6 Moderately unsatisfactory 0.46 0.12 3.80 6.9 Unsatisfactory 0.99 0.096 10.31 14.9 Highly unsatisfactory 0.24 0.05 4.4 3.6 Source: IEG calculations using World Bank data. experienced a decline of 10 percentage points or dle from lower to higher categories, or a uniform more. The 22 percentage point improvement in improvement across the board? Table 2.1 shows environment projects was driven by an unusual that, over the period from 1993 to 2008, the dip in performance for only two years (2003­04). cumulative increase in the percent moderately More details concerning the performance of satisfactory was about 15 percentage points. Within projects managed by the Environment Sector categories, the major movement has been from the Board are in chapter 5. two lowest ratings (highly unsatisfactory and unsatisfactory) toward the middle two ratings Long-Term Performance Shows a Clear (moderately unsatisfactory and moderately satisfac- Positive Trend Large changes in headline indicators in a single year underline the need for longer-term perspec- Figure 2.3: Improvement in World Bank tive and analysis. Such an exercise can help Project Ratings since 1993, interpret the significance of year-to-year changes Alternative Measures and assist in understanding the factors that drive 120 Percent satisfactory or above, relative to base year performance more generally. Despite year-to-year Figure 2.1 shows the percentage of fluctuations, the trend in 115 satisfactory projects Bank-wide since performance is clearly 1993 and reveals the steady improve- (1993 = 100) upward. ment in ratings of Bank-supported projects for many years. Despite the 110 year-to-year fluctuations, the trend is clearly up (as measured by the five-year moving average). 105 The percentage of projects rated satisfactory reflects the achievement of the threshold for a satisfactory rating, ignoring other points on the six- 100 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 point rating scale. But within satisfactory and 19 19 19 19 19 19 19 20 20 20 20 20 20 20 20 20 unsatisfactory ratings are subratings that Exit year Much of the increase in yield further insight. Is the improve- Percent satisfactory (traditional scale) performance came from a ment in percent satisfactory driven by a Average rating (6-point scale) Source: IEG calculations using World Bank data. drop in unsatisfactory decline in the percentage of very low Note: These are five-year centered moving averages rebased to 1993 = 100. ratings. performing projects, a shift in the mid- 12 WO R L D BA N K P E R F O R M A N C E I N P E R S P E C T I V E tory). Much of the overall increase in performance scale). (Box 2.3 contains further Much of the rise in came through a drop in unsatisfactory ratings. discussion of factors underlying poor percent satisfactory came performance in the two worst- from small changes in In addition, some of the rise in the percent satisfac- performing sectors.) borderline subcategories. tory category comes from the relatively small step between the two borderline subcategories (from Operations in the Africa, Middle East moderately unsatisfactory to moderately satisfac- and North Africa, and South Asia tory). An alternative performance metric is to take Regions had, on average, significantly Although percent the average over all six categories, assigning values poorer performance than those in East satisfactory rose 20 from 1 (for highly unsatisfactory) to 6 (for highly Asia and the Pacific. The top-perform- percent, average satisfactory). This measure gives equal weight to ing sector, transport, averaged more performance rose just 8 improvements or deteriorations. Charting the than half a percentage point higher percent. change over time since 1993 (the first year the six (0.563 in table 2.2) on the six-point categories were widely used by IEG) reveals a rise scale than the lowest performer, health, nutrition, in performance of 8 percent by 2006--compared and population. The results also confirm that with a 20 percent increase using the percent development policy operations, on average, satisfactory metric (figure 2.3). Thus, the Bank's received higher ratings than invest- average performance increase since 1993 is ment operations. Controlling for sectoral smaller than that suggested by IEG's traditional and Regional changes, measure of percent satisfactory.4 The results also confirm that the Bank operations still improved performance of Bank- improved over time. Shifting Regional and Sectoral supported programs is not solely a Composition Is Part of the Reason for matter of the project mix shifting to high- Improved Performance performing Regions or sectors. Within sectors Changing Regional and sectoral composition of and Regions there has still been a statistically the project portfolio--shifting the proportion of significant improvement in performance over projects across low- or high-performing sectors time during 1993­2008, as indicated by the or Regions--can change overall Bank perform- estimated 0.021 point annual improvement in ance. This would happen if certain categories of column D. In addition, the performance projects have statistically significant differences improvement has been fairly steady over time in performance and the composition of projects since 1993, without major breaks in the time changed across those same categories. A regres- trend. sion analysis is used to measure the performance improvement in Bank-supported projects after Columns A through C of table 2.2 also describe controlling for the influence of sector and the changes in composition over the period. Region, and also to estimate sector- or Region- There were major shifts in the percentage of specific performance differences, using all operations in the energy and mining sector and projects completed between 1993 and 2008. the agriculture and rural development sector, which declined as a share of all operations. In The results confirm that transport has been the addition, the share of operations in Europe and highest-performing sector and East Asia and the Central Asia has risen strongly, while the shares Pacific the highest-performing Region since in the Middle East and North Africa, Africa, and 1993.5 Column D of table 2.2 shows mean Latin America declined.6 performance relative to these two markers, and hence negative numbers. Taking education as an Such changes contribute positively to overall example, the results show that, holding all other performance only to the extent that operations variables constant, projects in the education shift away from low-performing categories sector performed, on average, 0.22 points worse toward high-performing categories. Column E in than projects in transport (using the six-point table 2.2 shows estimates of the relative contri- 13 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table 2.2: Changes in Project Composition Account for a Small Portion of Overall Improvement E. Effect of change on A. Share of B. Share of C. Change, D. Estimated overall Bank operations operations 1993­2008 effect/ performance Variable in 1993d in 2008 (B A) impactc (C D) Year (time trend) 15 0.021 0.316 Transport (sector of reference)a 0.130 0.180 0.051 (reference)a (reference)a Agriculture and rural development 0.273 0.148 0.126 0.433 0.054 Economic policy 0.093 0.033 0.060 0.611 0.037 Education 0.083 0.066 0.018 0.220 0.004 Energy and mining 0.227 0.074 0.153 0.527 0.081 Environment 0.000 0.074 0.074 0.383 0.028 Financial and private sector development 0.000 0.074 0.074 0.604 0.045 Health, nutrition, and population 0.032 0.098 0.066 0.563 0.037 Public sector governance 0.051 0.066 0.015 0.704 0.010 Social protection 0.000 0.057 0.057 0.245 0.014 Urban development 0.069 0.033 0.037 0.335 0.012 Water 0.042 0.057 0.016 0.494 0.008 Total sector 0.046 East Asia and Pacific (Region of reference)a 0.176 0.197 0.021 (reference)a (reference)a Africa 0.296 0.246 0.050 -0.632 0.032 b b Europe and Central Asia 0.046 0.205 0.159 Latin America and Caribbean 0.190 0.139 0.050 b b Middle East and North Africa 0.148 0.074 0.074 0.418 0.031 South Asia 0.144 0.139 0.004 0.267 0.001 Total Regiona 0.064 Development Policy Loan 0.120 0.066 0.055 0.368 0.020 Total sector/Region/ Development Policy Loan 0.090 a. Average performance among sectors is estimated relative to transport; among Regions, relative to East Asia and the Pacific; and among lending instruments, relative to investment operations. b. Only Regions with significant estimated performance effects are included in column D. Furthermore, six small sectors are excluded due to small sample size. c. This column reports the estimated effects from the regression explained in more detail in appendix A. For the year variable, 0.02 is the estimated time trend. For the sector and Region variables, what is reported is the average increase (on the six-point rating scale) in project performance associated with each sector or Region. For example, projects in agriculture and rural development performed, on average, 0.43 points worse than those in transport (the reference sector). d. Share of number of projects. bution of these shifts to the overall improvement relatively low-performing sector over the in Bank performance. The decrease in agricul- period.7 Taken together, the impact of sector ture and rural development, economic policy, shifts was a net positive, albeit modest (0.046 and energy and mining all boosted overall points of the six-point index with a mean of 4). performance because these were relatively low- Regional changes also show a net positive impact performing sectors. The increase in health, of 0.064 points, as Bank operations shifted nutrition, and population detracted from overall toward Europe and Central Asia and away from performance for the same reason--it was a Africa and the Middle East and North Africa. 14 WO R L D BA N K P E R F O R M A N C E I N P E R S P E C T I V E Box 2.3: What Accounts for Substantially Worse Outcomes in Health and Public Sector Projects? IEG's 2009 evaluation of the Bank's support for health, nutrition, lems-- for example, the absence of comprehensive site visits or and population (IEG 2009e) noted that performance in the sec- physical inspections in India health projects and reliance on bor- tor remained flat as other sectors improved. It identified several rower reporting. reasons for this low performance. Projects in Africa have per- Public sector governance is consistently the second-poorest- formed particularly poorly, with only about one-quarter of health, performing sector. IEG's 2008 evaluation (IEG 2008d) addresses the nutrition, and population projects achieving satisfactory results key constraints for satisfactory results in this sector. Bank support over the past decade, compared to 70 percent among low- to public financial management, tax administration, and trans- income countries in other Regions. parency typically improved country performance in these areas. Civil Complex projects--such as multisectoral investment lending service reform, however, did not, and direct measures to reduce and sectorwide approaches--in low-capacity countries have corruption rarely succeeded. For example, Tanzania made a good been strongly associated with worse results than less complex inquiry into the forms, loci, causes, and remedies for corruption, but work. And weak Bank supervision reduces attention to solving prob- the government has yet to implement its recommendations. Sources: IEG 2009e, 2008d. Adding everything together--sector changes, to improve performance through changing the Regional changes, and changes in the propor- composition of its portfolio. tion of operations between investment and policy loans-- the full performance improve- When Did Performance Begin to ment from all the composition changes comes Improve and Why? to 0.09 of a point on the six-point scale, which The rise in performance in recent years is positive but small. Table 2.2 shows that this raises the question of when this rise Sectoral and Regional is much smaller than the 0.316 contribution started, and what caused the change. shifts had a small but net from the time variable. If improvements associ- One factor cited is the influence of a positive effect on overall ated with the time variable are attributed to special portfolio management task project performance. Bank performance, it is concluded that 78 force--which issued the Wapenhans percent of the performance improvement Report in 1992 (Wapenhans 1992). Within the since 1993 is attributable to the Bank, and the Bank it is popularly believed that performance was remaining 22 percent is attributable to compo- dropping steadily throughout the 1980s and that sition changes.8 the Wapenhans Report sounded the alarm and triggered a major turnaround in performance. An open question is the extent to which these The report cited the need to reform implementa- composition changes were part of a deliberate tion, noting, "The number of projects judged policy to shift to high-performing activities. The unsatisfactory at completion increased from 15 Regional shifts were driven mostly by a rise in the percent of the cohort reviewed in FY81 to 30.5 share of operations in China and Eastern percent of the FY89 cohort and 37.5 percent of the Europe--the result of a sudden increase in new FY91 cohort." demand from these areas. The sector shifts-- away from low-performing agriculture and Current IEG data show a similar decline in toward high-performing transport, and also performance during the 1980s, but with slightly toward low-performing health, nutrition, and different figures. Projects judged unsatisfactory population--are bidirectional: improvements at completion rose from 20 percent in 1981 to 37 from better-performing sectors are offset by percent in 1989, and 29 percent in 1991. In declines in worse-performing sectors. Overall, retrospect, it can be seen that the decision of the the composition changes do not appear to have Wapenhans authors to examine the performance been the result of a purposeful drive by the Bank of projects organized by the date of termination 15 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 rather than the date of entry (a difference of policies during the 1980s. And there were major seven years, on average) was crucial. Had they policy changes in the Bank at about the time of examined project performance by the date of the turnaround. One of these was a policy shift entry they would have seen a different picture-- in the rural portfolio. The data show that the a deterioration in performance for projects that decline in performance ratings during the 1970s entered the pipeline during the 1970s, was quite general, appearing across a variety of Analysis shows a followed by a plateau starting around sectors and Regions. However, in 1980 the turnaround in Bank 1982, and recovery by the late 1980s, decline stopped--this change is particularly performance before the rather than the continuous deteriora- clear for agriculture projects (figure 2.5). The Wapenhans Report. tion that they emphasized (figure 2.4). 1970s brought a large increase in lending, partic- ularly to agriculture, following a push by With the benefit of hindsight, the full picture President McNamara. The year 1980 was a emerges (figure 2.4). The hypothesis that watershed; growing disillusionment with the internal Bank reforms motivated by the rural portfolio (especially integrated rural Wapenhans Report were the principal driver of development programs) led to a revised sector improved performance is difficult to support, approach (and the Bank's first structural adjust- because the improvement began before the 1992 ment loan).9 Performance in the agriculture and Wapenhans Report--a few years before, judged rural development sector began to increase by the exiting cohort of projects, and many years shortly afterward. The late-1980s were an active before, judged by the entering cohort. period of policy reform, both at the Bank and in client countries, and this coincided with the Another possible reason for a turnaround in improvement in performance ratings of Bank- performance is improvements in Bank sector supported programs. Figure 2.4: Percentage of Operations Judged Satisfactory, by Year of Entry and Exit 90 Performance decline plateaus Wapenhans Report is released 85 Performance begins to improve Percent satisfactory 80 75 70 65 60 0 72 74 76 78 80 82 84 86 88 90 92 94 96 98 00 02 04 06 08 7 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 20 20 20 20 20 Year At entry At exit Sources: World Bank data, IEG calculations. Note: Five-year moving average. 16 WO R L D BA N K P E R F O R M A N C E I N P E R S P E C T I V E Changes in economic conditions outside the 1990s also contributed, but suggests Changes in Bank sector Bank (such as improved macroeconomic con- at a minimum that economic policy policies coincided with ditions or higher growth rates) should also be and conditions were also important. the turnaround in mentioned. Results from the Latin America performance. Region show that performance turned around Performance Ratings of Country sharply for projects that entered just after 1982, Programs: No Similar Upward Trend the first year in which the debt crisis broke across In addition to project ratings, IEG also provides the Region. This period saw a general retreat from two ratings of the Bank's overall program for inward-looking, state-led development through- specific countries: Country Assistance Evaluations out the Region. Results from Europe and Central (CAEs) and Country Assistance Strategy Comple- Asia show performance improvement after 1989, tion Report (CASCR) Reviews. IEG produces about a period associated with market-oriented policy five CAEs each year; each covers, on average, 12 reforms in that Region.10 Other Regions, in years. They examine whether the objectives of contrast, do not exhibit significantly different Bank assistance were relevant, the program was patterns in performance from the Bank-wide effectively designed and consistent with its average. objectives, and the program achieved its objectives and had a substantial impact on the country's Overall, although it is not possible with these development. Box 2.4 summarizes two recent data to prove causality, the timing is consistent CAEs. CASCRs are Bank documents with the idea that changes in policies and that evaluate the outcomes of the Changes in country economic conditions outside the Bank played a Bank's assistance strategies with policies also coincide role in the turnaround in project performance. partner countries upon completion; with long-term The evidence does not reject the idea that Bank- they cover, on average, 4.5 years. performance centered administrative reforms in the mid- CASCR Reviews are IEG's independent improvements. Figure 2.5: Agriculture: Percentage of Operations Judged Satisfactory, by Year of Entry and Exit 90 Performance decline stops Wapenhans Report is released 85 80 75 Percent satisfactory 70 65 60 55 50 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 00 02 04 06 08 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 20 20 20 20 20 Year At entry At exit Sources: World Bank data, IEG calculations. Note: Five-year moving average. 17 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Country ratings correlate validation of the Bank's CASCRs. Both IEG data show a mixed picture of country with project performance, the CAEs and the CASCR Reviews rate program ratings in recent years. CAEs, which growth rate, and income outcomes at the country level without reflect a longer time frame (often spanning level. isolating the separate contributions of economic and political shocks across administra- the Bank and the country. tions) do not show an upward trend, but have remained level or declined slightly. CASCR For most countries, then, three different kinds of ratings, introduced in 2002 and covering much outcome ratings could be used for the Bank's shorter Bank assistance periods, have improved work: the two sources of country ratings described as the new results focus has gained traction. In above, with the addition of a third, constructed 2007­08, however, the performance trend from the average of the ratings given to all of the flattened, which may merit further analysis in Bank-supported projects, as described earlier in next year's ARDE (figure 2.6). this chapter. However, because the expected outcomes of country programs differ from those Aside from those mentioned above, there are of projects, there is no presumption that the several reasons for ratings of country programs ratings should be the same; in fact, the to differ from the average ratings of Bank- Country ratings were country ratings tend to be lower. Across supported projects in a given country. The slightly lower than all countries the average rating for the country evaluations take into account the project ratings. country evaluations since 1993 has relevance of the overall objectives of the Bank been 3.7 on a six-point scale. This is program, the achievement of those objectives slightly lower than the average rating for all against standards set in the CAS, the quality of projects of 4.0. the interaction with the government, and the Box 2.4: What Do Moderately Satisfactory and Unsatisfactory Country Programs Look Like? Nigeria Georgia The moderately unsatisfactory outcome of the Bank's program in The Bank's program in Georgia was rated moderately satisfactory. Nigeria between 1998 and 2007 hinged on poor outcomes at the The program focused on macroeconomic stabilization, governance, state and local levels. The program focused on macroeconomic private sector development, and human development between 1993 stability, a basis for sustainable non-oil growth, and better deliv- and 2007. There were a number of severe setbacks between 1998 ery of social services; improved governance was a key factor for and 2003, including poor tax collection, a near collapse of an elec- all three. During the latter half of the review period, in particular, tricity sector hampered by corruption, and worsening access to good the government undertook important actions, including creating health care. But after the Rose Revolution in 2003, the Bank quickly an oil surplus account, improving public financial management, recognized the potential for change and extended its support to the and deepening the privatization program. The Bank provided im- new government. Tax revenues increased from 14 percent to 25 per- portant assistance, despite its relatively small financial contribu- cent of gross domestic product (GDP); an anticorruption campaign tion. But there was little progress in tackling key structural successfully led to amelioration of corruption indicators; electric- weaknesses. The Bank was successful in supporting the im- ity supply stabilized; and a well-designed social assistance target- provement of governance (though not in reducing corruption) at ing program was implemented, while poverty rates slowly declined. the federal level, but little has been accomplished at the state level. Challenges remain--including helping farmers to access credit Similarly, there was little progress in improving municipal water and building judicial independence--but the program's midcourse and urban infrastructure--the main barriers to non-oil growth. And correction led to several satisfactory outcomes. targets for improving health and education outcomes were missed due to the lack of focus on building institutions at the state level, which carries primary responsibility for social service delivery. Sources: IEG forthcoming a, 2008i. 18 WO R L D BA N K P E R F O R M A N C E I N P E R S P E C T I V E quality and relevance of technical assistance and building capacity, stimulating public There are several reasons analytic work that may not be part of any specific debate, and facilitating knowledge for country ratings to project. Whether the objectives in the CAS are exchange. It found that users of AAA differ from project achieved often depends on other donors, on rated achievement of these objectives ratings. regulatory changes, or on exogenous develop- average or higher.11 It also found that if ments such as natural disasters or financial crisis, high technical quality, country ownership, and and not solely on the success or failure of the Bank follow-up are present, AAA is likely to have projects supported by the Bank. an impact. The Quality Assurance Group's (QAG) FY07 Technical quality, the report found, was the most Annual Review of Portfolio Performance important of the three ingredients listed above. In- (ARPP), using evidence from IEG evaluations, country stakeholders' ratings of technical quality cited effective policy dialogue and weak design strongly correlate with their ratings of the extent to of country programs as major reasons for which every objective has been met. The technical divergence. In countries with poor outcomes, quality of the Bank's AAA is generally good. A problems centered on significant omissions, review by independent specialists of a weak analytical underpinning, a failure to sample of AAA produced for middle- Technical quality was the account for in-country capacity, and a lack of income countries found that all the most important of the coherence across operations. Moreover, as reports in the sample reflected cutting- three ingredients of noted in the 2008 ARDE and the 2007 ARPP, edge knowledge in their sectors, and 56 success. synergies between and among the Bank's lending, knowledge services, and policy dialogue Figure 2.6: Mixed Performance in Country Program were not always fully exploited, and some Ratings over Time projects were not relevant to the country's core development challenges. Nevertheless, project 4.4 performance is part of the overall country evalua- tion, so there is inevitably some correlation. 4.2 Project rating (on 6-point scale) Bank AAA Is Important but Not Regularly 4.0 Evaluated AAA has become a growing part of the Bank's 3.8 administrative expenditures. It accounted for al- most a third of the Bank's spending on country 3.6 services in fiscal 2008 (along with work on new lending and supervision of existing projects). It has also grown as a share of the Bank's operational 3.4 spending--by about 6 percentage points between fiscal years 2004 to 2008. 3.2 At its best, analytical work is valued and 3.0 demanded by countries as a major source of 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 19 19 19 19 19 19 19 19 19 20 20 20 20 20 20 20 20 20 knowledge about reforms and global knowledge. Year It has been most effective when high technical Mean project rating Mean CAE rating quality is combined with sustained engagement Mean CASCR Review rating with clients and stakeholders. Sources: World Bank data and IEG calculations. Note: To compute the country program rating by fiscal year, the relevant CAE and CASCR Review rat- IEG's evaluation of AAA from 2000 to 2006 (IEG ings are applied to each year covered by the evaluation. Poorer results in earlier CASCR Reviews may 2008e) assessed the extent to which AAA achieved indicate a steep learning curve for the Bank in using the CASCR as an instrument (it was introduced in fiscal year 2003). objectives of informing lending, informing policy, 19 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 percent of them were rated highly satisfactory in limited capacity--as was seen, for example, in this regard. Clients have a particular appreciation reviews of the forest sector in the Democratic for the application of the latest international Republic of Congo and the private sector experience to their policy challenges. development strategy in Lesotho. But in Indone- sia, a series of analytical products on pricing, Evaluating the effectiveness and results of AAA is marketing, and extension in the agricultural inherently difficult. Sometimes clients will only sector got little traction because there was no own analytical work that supports their preferred interested interlocutor for the policy dialogue conclusions, irrespective of quality. Sometimes (IEG 2008h). analytical work is not immediately influential but becomes so upon the removal of political A common form of follow-up is lending--by the constraints. This was the case in Peru, where the Bank or by other donors. In Lesotho, the Sierra Rural Development Strategy did not recommendations of an education study were become influential until a full four years after its implemented through grant support from the completion, when there was a change in govern- Education for All Fast Track Initiative Catalytic ment. Compromising technical quality to achieve Fund. But sustained policy dialogue for a period ownership can have reputational risks for the after the production and initial dissemination Bank in other countries. In addition, it is difficult can also be powerful--as with the workshops to measure the important outcomes associated and TA in Jordan that followed analytical work with analytical work rather than the inputs on ways to prevent money laundering and (reports completed or technical assistance [TA] eventually led to improved legislation. By delivered). contrast, a technically sound report on popula- tion issues in Nigeria had little evident effect on IEG country reviews have pointed to Bank AAA as sector policy or strategy, partly because the data better at diagnosis than at implementing country- and analysis were not used to engage the author- specific recommendations. In the agriculture ities on fertility and related reproductive health sector in Nigeria, for instance, the 2008 CAE (IEG issues (IEG 2008i). 2008i) reported that "the Bank's analytic work provides a useful diagnostic, but is at In contrast to the detailed protocols and metrics There is little systematic too aggregated a level to contribute to that follow lending products, there is currently self-evaluation of AAA the action-oriented national agricul- little systematic self-evaluation of analytical work products. tural strategy Nigeria needs. . . . At this done by the Bank. A rudimentary M&E system time, little impact can be discerned does exist, but Bank staff do not take it seriously. on the government's policies and actions." Indeed, the results of a full third of AAA products go unrecorded. Of course, quality sometimes is not enough to ensure results. AAA from the Bank is more This lack of regular information about the effective in fostering reforms when the govern- outcomes and usefulness of completed ment or other donors have ownership of the knowledge products is compounded by discon- findings and recommendations and use them to tinuation of the occasional AAA reviews by QAG, inform their policies. In its operations, the Bank which had examined selected AAA in depth since encourages partnerships with national counter- fiscal 2000. IEG also does not do systematic parts--partly to build capacity, but also to evaluations of AAA, except as part of country enhance ownership of the findings. Counter- program evaluations and selected sector and parts rate the Bank average in using partner- thematic evaluations. The major evaluation of ships to produce AAA. In addition to part- knowledge products in 2008 (IEG 2008e) was its nerships, high-level government champions can first systematic look at this issue. IEG intends to help deliver results even in countries with pilot an approach to more systematic evaluation 20 WO R L D BA N K P E R F O R M A N C E I N P E R S P E C T I V E of AAA products, but lacks resources to do more over the past 15 years. The traditional than a small sample. measure--the share of projects rated satisfac- tory for outcomes--shows a steady upward Level of Bank Adoption of Recent IEG trend. Evidence from disaggregated perform- Recommendations Has Been Declining ance ratings also shows improvement, but IEG influences the Bank's effectiveness through suggests that the ratings are overstated. After recommendations to management as part of its correcting for shifts in the Bank's portfolio over major evaluations. Bank management is account- time to better-performing sectors, Regions, and able to the Board for follow-up. The extent to loan types (factors largely outside Bank con- which management implements IEG recommen- trol), and measuring average performance dations and findings in policy advice, program rather than the percent satisfactory, the design, and project design is an intermediate performance gains since the early 1990s are measure of IEG's influence on Bank effective- reduced by more than half. ness. IEG tracks this indicator through an annual report, the Management Action Record (MAR), Project performance recovered in There has been a as summarized in appendix C. This year's MAR 2008 from 2007 levels. The downturn downward shift in the shows that in recent years, there has been a in Bank performance in 2007, re- level of Bank adoption of downward shift in the level of Bank adoption of ported in last year's ARDE, does not IEG recommendations. IEG recommendations: adoption of recommen- represent a serious deterioration or dations assessed by IEG as substantial or better change in the Bank's positive trend. Evidence accounted for 41 percent of recommendations from disaggregated data does not alter this in the 2009 MAR, as compared with 42 percent in conclusion. the 2008 MAR and 60 percent in the 2007 MAR. These trends have raised questions about the Overall outcome ratings for 2008 projects were need for more careful tracking and discussion of especially strong in the transport and water recommendations. sectors and in Africa, South Asia, and Latin America, while health and education showed IEG is undertaking an initiative to improve the much weaker performance. In addition, trends MAR, to create a more timely and effective between 2003­05 and 2006­08 show large product for assessing the implementation of improvements in social development, environ- IEG's recommendations. Key elements under ment, urban development, social protection, and consideration for the reform include measures to transport, but an overall decline in education. improve the quality of IEG recommendations and to facilitate stronger engagement of the Board in The performance improvement is a long-term the process. Further elements are aimed at trend that dates back to the late 1980s, and even simplifying the reporting and monitoring process to 1980 by some measures. The Wapenhans and moving toward a more continuous engage- Report (1992), sometimes credited with initiat- ment and support in the implementation of ing the improvements, actually came several recommendations. IEG may also combine the years after the improvements started. Data on MAR and Implementation Report into a single the timing of the performance turnaround is report that assesses management actions taken consistent with the idea that reforms to Bank over the past year. Under the forthcoming update sector policies, country-level economic reforms, to IEG's disclosure policy, the combined and economic conditions also contributed. document would be disclosed to the general public from fiscal 2010 onward. About one-third of resources spent on country services go to AAA, but its quality and impact is Conclusions not systematically assessed. A recent IEG review Project performance has shown improvement indicates that about two-thirds of AAA is 21 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 moderately satisfactory or better, and one-third is recommendations, country ownership of the less successful. Performance patterns indicate findings, and sustained follow-up after task that impact and results for clients depend on completion. Greater effort is needed to evaluate technical quality, concrete and country-specific AAA and to revive its self-evaluation. 22 Chapter 3 Evaluation Highlights · Following revision of Bank proce- dures and guidelines to improve M&E, monitoring has improved, but evaluation remains weak. · M&E ratings are associated with overall project outcomes--better M&E tends to be accompanied by better outcomes. · Information sharing is critical to en- gaging client interest in M&E. · Information on results is increas- ingly being used in planning country strategies. · Less than a quarter of projects un- dertook an evaluation or assess- ment of impact. · More than two-thirds of projects claim to be pilots in full or in part. · Self-assessments of performance continue to lack realism, but the net disconnect has declined. · Restructuring does not occur as fre- quently, as early, or as deeply as needed. Quay crane on new docks, Sri Lanka. Photo by Dominic Sansoni, courtesy of the World Bank Photo Library. Toward Measurable Results and Improved Performance Data he satisfactory/unsatisfactory ratings discussed in chapter 2 became T widely used, partly because such a rating system was flexible enough that it could be applied to a wide variety of operations, which made op- erations easily comparable. However, there are increasing calls for more meaningful, accurate, and decision-relevant information on performance, both at the World Bank and in the overall international development community, under the managing for results agenda. To clarify terms, meaningful information refers · The monitoring and evaluation initiatives to the measurement of real results. If poverty · The IDA15 Results Measurement System reduction is the goal, the measurement of impact · Impact evaluation initiatives on incomes of poor individuals is a meaningful · Real-time project information indicator. For health projects, examples of · Economic cost-benefit analysis. meaningful indicators would include morbidity and mortality data; for education projects, learning Each of these five areas bears on some aspect of attainment. Accuracy refers to quality in terms of the goals of meaningful, accurate, and decision- basic principles of measurement as well as rigor, relevant performance information, and this when required, in the assessment of impact. chapter reviews efforts to improve performance Decision-relevant refers to whether the informa- information in each. tion is sufficiently comprehensive to underpin important decisions, such as whether the particu- Progress Implementing Bank-Wide M&E lar project or activity should be done again, should Reforms Is Evident but Remains be scaled up, or should be pursued in another way. Challenging Improving M&E is a perennial concern that is There are, of course, Bank-supported projects raised in IEG reports and discussed in the Bank. that deliver excellent information on results, but The Bank has offered general encouragement, there is broad acceptance that this does not detailed guidelines, and specific requirements happen frequently enough. The Bank is moving aimed at this goal for over a decade. IEG, for its toward generating better information through part, has drawn attention to inadequate M&E in several initiatives. This chapter reviews five areas numerous sectoral evaluations and in its annual to assess the progress being made: reviews, including the 2008 ARDE. Chapter 4 25 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 returns to the issue with special reference to and the extent to which such guidelines improve environmental sustainability and Bank support. M&E practice in projects. Have they led to improved information and better use of informa- The more recent M&E initiatives have included tion for decisions? two broad goals. The first is to ensure that projects have a results orientation and that they Given the short amount of time since the changes state the meaningful development outcomes and lags in reporting on projects, it is still too they are striving for, as well as the logic linking early to tell if behavior has changed or perform- outputs to outcome (the results framework). ance information has improved. It is clear, The second goal is to measure and use informa- though, that there have been real improvements tion on achievement of output and outcome in the results orientation of Bank operations, the targets to manage implementation and to report first goal listed above. For instance, as the country results. program has become the focal point for the Bank's development support, it has fully adopted Bank procedures and Revisions to the Bank's procedures results-based CASs, which include a standard guidelines have been and guidelines have been ongoing results framework. IEG's health evaluation found revised to strengthen since at least the mid-1990s. Recent substantial improvements (comparing projects M&E. revisions include the following: entering in 2007 with those entering in 1997) in key prerequisites for sound M&E, including the · In 2004, the Bank began to require, first, that collection of baseline indicators and setting investment operations include results frame- targets. A recent Bank survey of IDA-supported works outlining intermediate and final out- projects in 2007 reported that 80 percent have comes and the process for carrying out M&E1 collected baseline information on at least one and, second, that development policy opera- important indicator (IDA 2007). And the Annual tions specify expected outcomes and measur- Review of Portfolio Performance, conducted by able indicators (World Bank 2004a). QAG, gives high marks to the results frameworks · In 2005, the Bank began to require that Coun- in recent projects. try Assistance Strategies (CASs) be results- based and include a self-assessment upon their Nonetheless, these achievements have tended to completion. be on the compliance end of the spectrum, · In 2005, the Bank replaced the Project Super- rather than in generating timely data to help vision Report with the Implementation Status guide implementation or hard data that can and Results (ISR) report. The regular ISRs now illuminate whether or not projects have met include outcome and intermediate outcome in- their targets or achieved results. This is dicators in the main report rather than in a important because the measurement of results rarely updated optional appendix, as before. provides the crucial information base for improv- · In 2006, the Bank began to require that Im- ing development effectiveness. The 2009 Annual plementation Completion and Results (ICR) re- Review of Portfolio Performance points to a ports include an assessment of project M&E continued lack of realism and lack of attention to quality along three dimensions--design, im- measurement of outcomes and impacts as key plementation, and utilization. Separately, IEG, remaining challenges. as part of its reviews of ICRs, now rates over- all M&E quality using a four-point scale.2 IEG's rating of the design, implementation, and There is limited The ICRs themselves assess--but are use of M&E systems, which only began in fiscal information on the extent not required to rate--M&E quality. 2007 for projects that closed in fiscal 2006, to which the mandates establishes a reference point for future trends in have improved The important issue is whether such M&E performance. The rating categories are information or decision mandates are honored in spirit and high, substantial, modest, or negligible. Data making. not just in the letter of the mandate from projects closing in fiscal 2007 and 2008 26 T O W A R D M E A S U R A B L E R E S U LT S A N D I M P R O V E D P E R F O R M A N C E D ATA Figure 3.1: M&E Was Rated High or design and implement good projects The Bank's monitoring Substantial in Only about a Third of also tend to design and implement has improved, but its Projects Exiting in 2007 and 2008 good M&E. Whether there is causal- evaluation remains ity--that good M&E tends to improve weak. project outcomes--is difficult to determine. Substantial 33% Second, broad indicators of country M&E ratings are Modest capacity (statistical capacity and GDP associated with overall 45% per capita) are uncorrelated with project outcomes. project M&E ratings. This suggests that, so far, country capacity has not been a strong determinant of the quality of project High M&E. This may indicate that good project M&E 4% can be conducted even in a low-capacity context. Negligible 18% Staff incentives are also cited as a Some evidence suggests constraint to improving M&E. An that project M&E quality Source: IEG ICR review database. internal Bank report on M&E in India, is unrelated to statistical for example, recommended revamp- capacity in the country. ing internal systems to write contracts show that only 37 percent of projects exiting the for outcomes rather than deliverables. portfolio received ratings of high or substantial, while the remaining 63 percent were rated A further aspect of incentives is the culture of modest or negligible, as shown in figure 3.1. approval--the drive to get projects launched. Typical problems with M&E systems are Development professionals care deeply about highlighted in box 3.1. moving the projects they believe in toward the approval stage. In addition, staff and manage- Two conclusions can be drawn from statistical ment performance evaluations depend greatly analysis of these ratings. First, M&E ratings have on project approvals. If it were known that a significant and positive association with overall approvals depended on having solid information project outcomes,3 controlling for a number of on the results of similar projects, behavior might project characteristics, including loan size, type change significantly. If M&E is seen as a second- of lending instrument, fiscal year of approval, and ary issue, simply a tax on overburdened Bank project duration. This implies that those who staff, mandates and requirements are unlikely to Box 3.1: Recent Evaluations Highlight Problems in M&E A recent review of ICRs for health, nutrition, and population proj- the views of many clients that for M&E to succeed, cities have to ects approved during fiscal 1997­2006 found that more than 80 consider it a useful tool for themselves, not just an instrument for percent of the ICRs reported a problem related to the choice of government control or academic use by others. This suggests indicators or targets, over a third reported that indicators in the that there should be more interaction between municipalities, projects were not clearly defined, and 23 percent mentioned governments, and the Bank at the design phase of M&E in order that the project established indicators that were not measurable. to build around existing systems and country priorities and to gain An IEG evaluation of M&E in municipal management reflected local ownership. Source: IEG 2009f. 27 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Box 3.2: The Paradox of Satisfactory Performance port us with advice on MIS [management in- and Unsatisfactory M&E formation system] systems and tips on how to develop a good enough MIS. They should take The collection of relevant information is an obvious requirement for a a proactive rather than reactive role." meaningful evaluation of project performance. So how does IEG know · Second, several projects have helped encour- that project outcomes have been improving if M&E has been so weak? age the use of independent evaluations as a way IEG's project evaluations--which assess projects on their relevance, to generate unbiased performance data for efficiency, and effectiveness in meeting their development objectives-- decision making. The organizations commis- use multiple sources of evidence to arrive at a nuanced answer to this sioned to conduct the evaluations tended to question. Direct measures of the achievement of objectives are some- rely on free-standing data collection and analy- times obscured by poor M&E, so evaluators use other evidence--such sis activities. as interviews with task teams, clients, and beneficiaries; survey data on · Third, clients point to several factors limiting related indicators; and information on intermediate outcomes--to assess their own government's receptivity to the M&E whether or not objectives have been (or are likely to be) achieved. initiatives. Clients emphasized the general lack The better the underlying project M&E systems, the more direct of skills, understanding, knowledge, and in- measures of outcomes are available to IEG, and the greater the confidence centives within government agencies to de- in the accuracy of ratings. velop, sustain, and use outcomes-based M&E systems, especially at the subnational levels. They emphasized the competitive advantage result in the collection and use of more meaning- the Bank has in sharing information and ex- ful information. periences across projects and countries and ex- pressed their eagerness to learn from others' Review of the Client's Perspective in experiences. Argentina and India The Bank is pursuing improved M&E in environ- Clients also emphasized barriers to the use of the ments where clients have significant say over information, even in projects with highly rated what is done and how it is done: how do they M&E systems. These include staff turnover, a react to the M&E reforms? steep curve for learning how to use performance data, and lack of understanding and capacity at The Bank tends to be in Interviews with clients in Argentina and the subnational levels. One respondent in India the lead in introducing India--focused on the most recently noted: cutting-edge M&E to closed and most recently approved clients. projects (to take account of reform First, [the] mindset of government has to be efforts)--suggest the following: changed. M&E skills are not very strong-- so far it has been only monthly progress re- · First, Bank-supported projects helped to in- porting . . . what is monitoring? It's actually troduce the concept of systematic program very scientific . . . that is why we are going monitoring, which is intended to provide a for a professional external agency. Sec- comprehensive basis for decision-relevant in- ond, the challenge is to sustain those skills formation. This has been accomplished . . . until government has a pool of persons through management information systems to draw upon, there is not going to be a way that have introduced new technologies (such to do it in-house. as Internet-based systems). It is clear that these systems have improved and generated client Clients in several instances did not see eye-to- interest in basic management processes that eye on selection of indicators, which led to the were lacking earlier. Clients generally ask for perception of a Bank-owned but client-tolerated greater Bank leadership in this area. results framework. This situation does not Clients ask the Bank to be As one client noted, "[The Bank] portend well for the continuity and use of data more proactive. should be more proactive and sup- introduced through the project. Clients may 28 T O W A R D M E A S U R A B L E R E S U LT S A N D I M P R O V E D P E R F O R M A N C E D ATA comply with the system to fulfill Bank require- Under the auspices of the triennial agreements ments, but with little ownership and scant to fund IDA credits, there have been increasing predilection to use the information produced for demands from the international development monitoring, let alone for decision making. community for the Bank to demonstrate development results in IDA countries. The 2002 The general picture is one of reluctance on the IDA13 Agreement recommended that the Bank part of the client to use M&E and considerable establish a capacity to measure results and track fragmentation of M&E efforts. A recent Bank performance. Subsequent IDA Agreements review noted that fragmentation of evaluation (IDA14 and IDA15) have become activities across the Bank's India portfolio has progressively more detailed. IDA's Results resulted in a lack of sectorwide measurement of Measurement System impacts and outcomes, which could have been The basic structure of the RMS seeks to measure IDA's used to help build client demand and capacity includes the following three elements contribution to country for M&E. The India review recommended (see details in appendix J): outcomes and to harmonizing data collection and analysis across a strengthen statistical number of projects in collaboration with key 1. The Bank is to monitor country out- capacity. government and civil society agencies (Fiszbein comes with 14 specified indicators and Shah 2007). (called Tier 1 indicators). 2. The Bank is to monitor IDA's contribution to Harmonization of M&E country outcomes with the following Tier 2 in- The Bank has been actively engaged with dicators: countries and other donors in harmonizing M&E · At the country level, through results-based processes and approaches. This has been partic- CASs ularly true in Africa. The 2005 Paris Declaration · At the project level, through four indica- on Aid Effectiveness and its five main princi- tors (measured at entry, implementation, ples--country ownership, donor alignment with and completion) country policies and procedures, harmonization · In four sectors, all projects are to adopt and among donors, management for development measure core indicators that can be com- results, and mutual accountability--have given pared across projects and countries. impetus to these efforts. The challenge is to 3. The Bank is to strengthen statistical capacity in ensure that harmonization is accomplished countries. around the generation of best-practice informa- tion, and not lowest-common-denominator While the main emphasis was on project results, practice. the agreements also proposed a country-based system to "measure, monitor, and manage for Bank-Wide Results Measurement results" and envisioned a system "capable of System: Making Progress, but toward cross-country aggregation." Under this initiative, Uncertain Ends the Bank is tackling two important issues: first, A second area of important activity is the Results developing a set of core indicators that can be Measurement System (RMS) initiated to track used for many projects of a given type and, results for International Development Associa- second, formulating a method to enable aggrega- tion (IDA) credits and grants. The RMS attempts tion of project-specific results at the country and to standardize measurement around meaning- Bank-wide levels for reporting to stakeholders. ful indicators that can be aggregated into an overall picture of development outcomes. The Since IDA14, 14 Tier 1 indicators for 4 broad areas system is about decision relevance because (growth and poverty reduction, governance and donors would like the information to be com- private sector development, infrastructure, and prehensive and relevant for strategic planning human development), and 9 Tier 2 indicators and budgeting. have been posted on the internal and external 29 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Bank Web sites. But coverage over time for a supported projects is the growing use of impact number of countries and indicators (such as evaluation analysis as a complementary tool for access to an all-season rural road) remains quite assessing the effectiveness of development spotty. interventions and advancing the results agenda. The innovative contribution of impact evalua- The essential recommendations from the IDA15 tions is explicit identification of beneficiaries, Agreement and the current status of their along with control or comparison groups, to implementation are summarized in table 3.1. The achieve greater accuracy in the measurement of assessment that follows is based on interviews, the causal impacts of interventions.5 Unlike M&E documents, and questionnaire responses from initiatives, the impact evaluation initiatives are seven IDA donors, accounting for decentralized and not required as a matter of Information on results is about half of cumulative IDA subscrip- Bank policy. increasingly used in tions and contributions.4 preparing CASs. The increasing attention to impact evaluation Information on results is increasingly within the Bank is largely led by three initiatives. being used in the Bank's main planning The Spanish Trust Fund for Impact Evaluation document, the CAS, as well as in the development supports impact evaluation of programs aimed at of a country program results-monitoring tool and improving human development outcomes; the in piloting the integration of CAS results with Development Impact Evaluation Initiative (DIME) budget allocation and work program- serves as a repository and channel to disseminate IDA donors expect a ming. Three of seven IDA donors results from completed evaluations; and the Africa deeper institutionaliza- surveyed indicated that they would like Impact Evaluation Initiative (similar to DIME) aims tion of managing for to see the Bank give more attention to to mainstream impact evaluation work in Bank- development results. using results as a management tool, supported operations in Africa. not only for strategic planning but also for setting budget priorities, determining staffing, The existence of adequate baseline data for and producing performance evaluation. projects is a useful indicator of the potential for conducting some form of M&E. The Bank Four surveyed IDA donors view the Bank as a recently reviewed the availability of such data in leader in results measurement. But three expect ISRs for IDA Operations (IDA 2007, p. 16, table deeper institutionalization in the Bank of 6). They found that the percentage of ISRs that managing for development results. They included baselines on at least one outcome envision a management tool that would feed into indicator rose from 59 percent in fiscal year 2004 the Bank's planning and evaluation processes; to 80 percent in 2007. real-time project results disclosure; and, ultimately, aggregation of project results. One For the present report, IEG further analyzed all IDA donor indicated a desire to see the Bank 44 projects in three sectors--agriculture, work with other donors in developing an environment, and water--that started after fiscal approach to results measurement for fragile 2001 (two years before the Bank issued a states. Two respondents also encourage the recommendation for baseline data collection for continued efforts of multilateral development Bank-supported projects) and closed before banks to converge on standard country and December 2008. There is no claim that these project-level performance indicators. three sectors are necessarily representative of all Bank operations.6 Table 3.2 reports the number Use of Impact Evaluation Is of projects that collected baseline data and end- Use of impact evaluation Growing, but Its Benefits Are of-project data for measuring results. For this has not been driven from Not Fully Achieved exercise, projects were considered as having pre- above as a matter of The third major area in improving project data if they developed either baseline Bank-wide policy. performance information in Bank- indicators and/or a full survey. Collecting such 30 T O W A R D M E A S U R A B L E R E S U LT S A N D I M P R O V E D P E R F O R M A N C E D ATA Table 3.1: The Bank's Progress in Implementing IDA15 Recommendations Recommendation Implementation status Tier 1 Review private sector development indicators and RMS includes two private sector development indicators explore development of a public expenditure and drawn from the Doing Business survey (cost and time required financial accountability (PEFA) aggregate indicator. for business startup). Aggregation of the six PEFA component indicators is believed to lack useful policy or comparative value. Tier 2 Monitor quality, not just number of results-based CASs. All CASs are now results-based. Their quality is monitored in CAS Completion Reports and CASCR Reviews conducted by IEG. The treatment of results is now more uniform and evidence-based, but outcomes tend to lack realism and inter- mediate milestones. Due to the difficulty of getting donor agreement on objectives, joint results-based CASs tend to be weaker. The financial crisis is resulting in two-year indicative CASs with weak results orientation. Improve availability of key outcome indicator baselines. Availability and quality of baselines is improving, but gaps re- main in coverage and quality (see table J.1 in appendix J for more detail). Improve quality of output and outcome reporting in ICRs. According to IEG assessments, the quality of M&E, including output and income reporting, improved from fiscal 2006 to 2007, but leveled off in fiscal 2008. Develop lists of indicators for 4­5 sectors by the IDA15 Based on the Africa Results Monitoring System, the RMS is Midterm Review to improve outcome indicator quality being extended to all sectors, and the system will be anchored and allow more accurate aggregation of sector outputs. in a database at the ISR level. Lists of indicators are being de- veloped for all key sectors and themes to improve outcome quality and aggregation of sector outputs. Ten IDA countries are piloting a Country Program Results Monitoring Tool. Con- sideration is also being given to incorporating AAA. Strengthening Monitor statistical capacity in line with partner-country Statistical capacity is being monitored by the Bank's Develop- statistical priorities. ment Economics Data Group in line with partner country capacity priorities. Develop a data standards system to help countries track The Bank is encouraging movement toward data standards by their progress. co-chairing the PARIS 21 (Partnership in Statistics for Develop- ment in the 21st Century ) with the Development Assistance Committee of the Organisation for Economic Co-operation and Development and is about to launch a Web-based tool to en- able countries to monitor their progress in strengthening sta- tistical capacity. Collaborate with donors to strengthen statistical With other donors, the Bank helped create a Statistics for Re- capacity, including exploring expansion of the Trust sults Facility Catalytic Fund to supervise, finance, and co- Fund for Statistical Capacity Building. finance larger investments in statistical capacity. Work toward the collection and analysis of gender- The Development Economics Vice Presidency Data Group col- disaggregated statistics at the country level. lects gender-disaggregated data whenever available and fea- sible. The Bank's Gender Group is working on an improved ap- proach to the collection and analysis of gender-disaggregated data under IDA16. Include more comprehensive treatment of statistical A guidance note and online learning tool are being developed systems in CASs. for Regional staff. Source: IEG interviews and analysis. 31 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Box 3.3: Reasons for Impact Evaluation There is ongoing discussion at the Bank and elsewhere about when Many social science examples reveal unintended or sur- to perform formal impact evaluations. Formal impact evaluation prising consequences of interventions. For example, the United means a planned effort at the beginning of a project to ensure that States Labor Department established a program to give groups there are data for a control or comparison group, so that impact with low employment rates vouchers entitling prospective em- can be measured accurately by comparing outcomes of the ben- ployers to a wage subsidy for employing them. The program was eficiaries against those of the comparison group. The more diffi- expected to boost employment among hard-to-employ groups, but cult it is to determine or measure what would have happened an impact evaluation found that the targeted groups had lower, without the project, the greater the need for an impact evaluation. not higher, employment rates than the control groups. One pos- Impact evaluations used to be controversial in other fields, such sible explanation is that the vouchers had the unintended effect as medicine, but are now widely accepted as vital sources of ac- of stigmatizing the targeted group as risky and undesirable curate information. What proved influential in medicine was evidence workers. from landmark clinical trials that showed conventional practice to A similar result occurred in Argentina's Jefes y Jefas de Hogar, be mistaken or even damaging. In one example, clinical trials showed an anti-poverty program established in the midst of the 2001 cri- that providing additional oxygen to premature infants, which many sis to provide cash transfers to unemployed household heads assumed to be beneficial, actually caused blindness as children ma- with children in school. Participation in the program was condi- tured. Expert opinion can reach a consensus around plausible in- tional on the household head not being employed. The program terpretations that later prove faulty, based on experimental evidence. reached around two million beneficiaries, representing 13 percent In one application to development, impact evaluation provided of the economically active population. An impact evaluation found valuable new knowledge. A series of interventions in western that during 2002­05, a period of rapid employment growth, program Kenya were evaluated to understand the determinants of the participants were less likely to find formal jobs than were non- adoption of fertilizer. Evaluations of various facets of the project participants with similar observable characteristics. revealed obstacles to the adoption of new fertilizer technologies, These examples demonstrate the most important reason for high-lighted aspects of the projects that worked to change behavior conducting impact evaluations: more accurate information can of farmers, and identified other aspects that failed to deliver results. overturn reasonable and widely held views that are nevertheless mis- Opportunities to address information asymmetries and financing taken, or it can enhance knowledge of development practitioners constraints for farm inputs were highlighted and further explored. to improve projects. Sources: Burtless 1985; Brown 2005; Kremer, Duflo, and Robinson 2009; Gasparini, Haimovich, and Olivieri 2006. pre- and post-project data is usually a minimum disconnect is partly the result of the sources used requirement for sound causal evaluation work.7 and the differing time periods examined. The Bank report took its information from the first Twelve of the 44 projects, the bulk of which were interim progress report (the ISR) of all IDA in East and Central Asia, collected baseline data. projects that completed an ISR in fiscal year 2007, Of these, 11 were agriculture projects, 9 in these which means that the projects had just begun (in 2 Regions. A similar picture emerges for the post- 2006 or 2007). IEG, in contrast, evaluates only project data collection. There was no baseline information from completed project reports, and data collection among the seven African agricul- thus examined a sample of projects that closed ture projects examined. before December 2008, and had thus been approved between 2001 and 2008. The discrep- There is an apparent discrepancy. The Bank's IDA ancy in results is likely the product of a combina- report found that 80 percent of projects had tion of improving baseline data collection by Bank baseline data for at least one indicator (up from 50 projects, the different time periods of the samples, percent in 2005), while the present review of ICRs and the possibility that the IEG sample covers in three sectors found that only 27 percent include sectors with lower baseline data collection than baseline data for at least one indicator. This the Bank-wide average. 32 T O W A R D M E A S U R A B L E R E S U LT S A N D I M P R O V E D P E R F O R M A N C E D ATA Table 3.2: Summary of Data Used for Evaluating Project Outcomes, by Region Baseline data Final project data Sector board/Region Total Yes No Percent Yes No Percent Agriculture Africa 7 0 7 0 2 5 29 East Asia and Pacific 5 4 1 80 4 1 80 Europe and Central Asia 7 5 2 71 5 2 71 Latin America and Caribbean 4 1 3 25 1 3 25 Middle East and North Africa 2 1 1 50 0 2 0 South Asia 1 0 1 0 0 1 0 Environment Europe and Central Asia 4 0 4 0 1 3 25 Latin America and Caribbean 5 1 4 20 0 5 0 Middle East and North Africa 1 0 1 0 0 1 0 Water Africa 2 0 2 0 0 2 0 East Asia 1 0 1 0 0 1 0 Europe and Central Asia 2 0 2 0 0 2 0 Latin America and Caribbean 1 0 1 0 0 1 0 Middle East and North Africa 1 0 1 0 0 1 0 South Asia 1 0 1 0 0 1 0 Totals 44 12 32 27 13 31 30 Sources: World Bank data and IEG calculations. Note: Based on World Bank agriculture, environment, and water projects exiting fiscal 2002­08. The existence of baseline and follow-up data is tion or use a nonstandard evaluation Less than a quarter of not the only condition for thorough evaluation method (classified under the heading projects undertook an analysis. It says nothing about the quality of the of other8). impact evaluation or data, the coverage and relevance of the indica- impact assessment. tors measured, or the quality and existence of More than 70 percent of all projects comparison or control groups. Table 3.3 records in the sample claim to be pilots or have some whether an evaluation was envisaged by the part that is being piloted. In theory, pilots project and, if so, the kind of evaluation. have to be assessed-- ideally with method- ologically valid evaluations-- before moving In table 3.3, the term impact evaluation is used into full implementation, which should occur for quantitative evaluations that involve control only if the evaluation indicates positive or comparison groups; impact assessment refers results. However, only 65 percent of projects to quantitative before/after evaluations with no designed as pilots or containing some pilot comparison group; and economic analysis components had a quantitative includes any effort to compare benefits with evaluation, and 30 percent of these costs. The other evaluation categories rely on had an impact evaluation or assess- More than two-thirds of qualitative information gathered through benefi- ment. Focusing exclusively on projects claim to be ciary surveys, stakeholder meetings, and limited quantifying project effects, one- pilots. Of these, only 30 quantitative data obtained from national statis- fourth of the projects reviewed percent had an impact tics and/or combined with qualitative informa- undertook either an impact assess- evaluation or assessment. 33 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table 3.3: Summary of Evaluation Types Evaluation typea Impact Impact Economic Beneficiary Stakeholder Sector board/Region Total evaluation assessment analysis survey information Othera Agriculture Africa 7 0 1 0 2 1 5 East Asia and Pacific 5 1 4 4 0 1 4 Europe and Central Asia 7 1 2 6 5 2 3 Latin America and Caribbean 4 0 1 2 2 1 2 Middle East and North Africa 2 0 0 1 1 0 1 South Asia 1 0 0 1 1 1 1 Environment Europe and Central Asia 4 0 1 2 1 1 0 Latin America and Caribbean 5 0 0 1 1 1 3 Middle East and North Africa 1 0 0 0 0 0 0 Water Africa 2 0 0 2 2 2 1 East Asia 1 0 0 1 0 0 0 Europe and Central Asia 2 0 0 1 0 0 2 Latin America and Caribbean 1 0 0 0 0 0 0 Middle East and North Africa 1 0 0 0 0 0 0 South Asia 1 0 0 0 0 0 0 Totals 44 2 9 21 15 10 22 Sources: World Bank data and IEG calculations. a. Some projects use more than one evaluation approach. Other refers to ad hoc information designed to fulfill ICR requirements. ment or an impact evaluation to measure the When impact evaluation and assessments are success or failure of the project, mainly in East carefully done, they can generate information Asia and Eastern Europe and Central Asia. relevant to decision making. To accurately inform policy, they must meet the minimum standards There is great variation in research design among of rigor needed to ensure credible findings with projects that conducted impact assessments or potential applicability to other contexts. IEG has impact evaluations. Of the nine impact assess- initiated some preliminary analysis of evaluating ments, most collected data at discrete points so the quality, relevance, and use of evaluations that before and after comparisons could be focused on identifying impacts. made, although two used data that were collected after the program was completed, Rating the quality of impact evaluation raises relying on retrospective questions to establish a several concerns: baseline. One of the two impact evaluations derived conclusions using a noncom- · What is the importance of the question being parable counterfactual group.9 The addressed? Is it central to the justification of the The majority of majority of the quantitative evalua- project and relevant to the benefit stream of the evaluations identified tions used qualitative methods to help project? used qualitative corroborate, illustrate, or interpret the · Is the data used of high quality and relevant, information. results. to capture the outcomes of interest? 34 T O W A R D M E A S U R A B L E R E S U LT S A N D I M P R O V E D P E R F O R M A N C E D ATA · Is there internal validity? Is the identification This short review suggests that the The lack of baseline data of the intervention's effect credible, including lack of baseline data (in three-fourths is an important factor the appropriateness of the methodology and of the sample) is an important factor limiting the ability to the validity of identifying treatment and con- limiting the ability to perform quanti- perform quantitative trol groups? tative impact analysis--at least in the impact analysis. · Is there external validity? What is the likeli- water, agriculture, and environment hood that impact results apply more broadly sectors. Development of a formal methodology (geographically, culturally, environmentally), and evaluation of the quality, relevance, and use including the representativeness of the sample? of impact evaluations in the Bank is planned for · Are cost-benefit concerns addressed? How well completion in fiscal 2011. are project benefits compared with the costs incurred to obtain them? Real-Time Project Information: Continued Overoptimism Based on the impact focus, 12 projects were Another window on the accuracy and use of considered in this analysis; only 5 of these were information reporting in Bank-financed projects available for further review.10 A rapid assessment is provided by comparing the self-ratings that of these projects yields some insights. First, with emerge over time as a project is implemented regard to internal validity, all posed relevant with independent ratings just after its closing. questions focused directly on project outcomes. Real-time project information refers to the Establishing external validity appears to be a quality of information that emerges about bigger challenge: each project extrapolated its projects as they are being implemented or findings and applicability to the broader context, immediately afterward. This is also about but few addressed features such as project size, accuracy, because the pattern of revisions over target population, variety of interventions, and time suggests a lack of realism in midcourse subnational and national context, which may information about project performance. limit the generalization of their effects. Only three of the five projects reported economic data A timeline of the evaluations conducted during that helped determine costs and benefits in a the project cycle is shown in figure 3.2. The first comparable manner. And none of them explicitly official Bank document to rate a project's separated fixed from variable costs, limiting progress toward achieving its development interpretation of the results. objectives is the ISR, completed by the project Figure 3.2: Timeline of Key Project Ratings Project approved Project implementation Project by Board and supervision completed Evaluation PCN PAD First ISR Additional Additional ICR IEG PPAR ISR ISR QAG Additional Additional Final ISR IEG ICR QAE ISR ISR Review Sources: Operations Policy and Country Services, World Bank; IEG. Note: PCN = Project Concept Note; PAD = Project Appraisal Document; ISR = Implementation Status and Results Report; QAE = Quality at Entry; ICR = Imple- mentation Completion Report; PPAR = Project Performance Assessment Report. 35 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 team. The ISR is a tool that provides information assessments of project performance by IEG for project and portfolio monitoring. A project's have tended to downgrade performance--IEG first ISR must be completed within 12 months of rates the percentage of satisfactory projects, on the project's approval by the Bank's Board, and average, 15 percentage points lower than the further ISRs are due at 12-month intervals Bank's self-assessment in its final ISR, just thereafter until the project is closed. Each ISR before implementation completion. This so- focuses on key issues and actions for manage- called net disconnect has improved substan- ment attention and rates the likelihood that the tially in the most recent fiscal year, declining project will meet its development objectives. from 17 percent to under 10 percent, indicating that these end-of-project ratings have become Within six months of project completion, the more realistic. ARDE 2009 extends this analysis Bank produces an ICR, which evaluates the to look not only at the final supervision report effectiveness of the project in meeting its stated ratings of likely outcomes, but at the previous objectives. Within three months after the ICR is ratings as well. finalized, IEG completes an independent review of the ICR using the evidence contained in the There is a low degree of variance among projects ICR. This ICR Review assigns its own rating, in these ratings. More than 88 percent of all ISR which can differ from the Bank's self-ratings, and ratings for active projects between fiscal years provides additional analysis. 2004 and 2008 have been satisfactory or highly satisfactory. Moreover, even with the smaller net Previous assessments (including ARDE 2008) disconnect, the evidence suggests that self- suggest that Bank staff assessments of ongoing assessments lack realism during project project performance lack realism. Independent implementation. Figure 3.3 highlights a pattern: Figure 3.3: Bad Information Tends to Be Revealed Only at the End of Implementation HS Projects rated MS or better by S IEG Average rating MS All projects rated by IEG MU Projects rated MU or worse by IEG U HU ISR ISR ISR ISR ISR ISR ISR ISR ISR ISR ICR IEG 0­10% 10­20% 20­30% 30­40% 40­50% 50­60% 60­70% 70­80% 80­90% 90­100% complete complete complete complete complete complete complete complete complete complete Evaluation Sources: World Bank data, author's calculations. Note: HS = highly satisfactory; S = satisfactory; MS = moderately satisfactory; MU = moderately unsatisfactory; U = unsatisfactory; HU = highly unsatisfactory. 36 T O W A R D M E A S U R A B L E R E S U LT S A N D I M P R O V E D P E R F O R M A N C E D ATA Table 3.4: Low Ratings at Project Midpoint Foreshadow Worse Outcomes IEG outcome rating Likelihood of ISR rating Unsatisfactory Satisfactory unsatisfactory rating (%) ISR rating at midpoint Unsatisfactory 51 54 49 Satisfactory 149 664 18 ratings decline right at the end of a project, downgraded than upgraded: 420 out Bad news is revealed only suggesting a lack of candor in the earlier report- of 918 projects (46 percent) were at the end of project ing. Bad information tends to be revealed only at ultimately rated lower than at implementation. the end of project implementation. Even after midterm (the area enclosed by dotted unsatisfactory projects are 90 percent complete, lines in table 3.5), and 105 projects (11 percent, a majority are still rated moderately satisfactory the area enclosed by the dark bars) were or higher (one-third are even rated satisfactory ultimately rated higher than anticipated at or highly satisfactory). midpoint. Furthermore, when restructuring does occur, it usually happens late in the Despite a pattern of downgrades right at the end, project's lifetime: on average, nearly five and a the performance ratings in midproject supervision half years into the life of a project. 12 reports foreshadow unsatisfactory performance at closing.11 This is seen by calculating the probability The Bank's own reports have Projects with low ratings of getting an unsatisfactory rating conditional on concluded that restructuring is not in midproject supervision the midterm ratings. As demonstrated in table 3.4, done frequently or early enough in the reports are more likely to projects rated unsatisfactory at midpoint had a 49 project's lifetime. QAG's Quality have low ratings at percent chance of being unsatisfactory in the final Assessment of the Lending Portfolio closing. reckoning. In contrast, projects rated satisfactory (QAG 2009) concludes that projects at midpoint had only an 18 percent chance of are not restructured as early or as deeply as being labeled unsatisfactory in the final rating. needed. It finds that its panelists--who are at arm's length from project implementation-- Table 3.5 shows more detailed information using identified twice as many problem projects as did the six-point scale; it also suggests a lack of project team leaders. And only about one-third of realism. Project ratings are more likely to be projects that panelists asserted needed restruc- Table 3.5: Nearly Half of Projects Are Ultimately Rated Lower than Projected at Project Midpoint IEG outcome rating Highly Moderately Moderately Highly Rating at midpoint unsatisfactory Unsatisfactory unsatisfactory satisfactory Satisfactory satisfactory Highly unsatisfactory 1 Unsatisfactory 4 19 16 19 12 Moderately unsatisfactory 1 5 5 11 12 Moderately satisfactory 1 5 5 2 Satisfactory 6 66 70 234 359 33 Highly satisfactory 1 5 22 4 Sources: World Bank data and author's calculations. 37 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Restructuring does not turing had actually been restructured, Bank, including better monitoring, more frequent occur as frequently, early, while the average age at restructuring baseline data collection, and more accurate and or deeply as needed. was 4.3 years. While most restructur- sharper measurement of benefits through impact ings occurred around the midterm evaluations. But positive benefits alone do not review, many of the project weaknesses were justify projects. Most development projects that evident by the second year of implementation. spend millions of dollars should be able to point to some positive benefits. The crux of the issue is Economic Cost-Benefit Analysis: whether those benefits justify the costs, or A Dormant Subject whether there are other, less costly ways of achiev- To make effective decisions about whether to ing similar benefits. fund projects, or whether to restructure or scale up pilot projects, it seems necessary to have Economic cost-benefit analysis has its critics and information on both costs and benefits, as well as supporters, but it is difficult to deny the simple a framework for organizing the information. This point that projects should be pursued only if the is what cost-benefit analysis and the economic benefits exceed the costs. For this reason it is rate of return estimates produced by cost-benefit official Bank policy that economic analysis analysis are intended to achieve. should be conducted for all investment projects to ensure that net benefits are positive.13 Cost-benefit analysis is ultimately about translating Nevertheless, there are indications of a decline otherwise fragmented performance information in economic analysis at the Bank, at least as into a form that is relevant to decision indicated, imperfectly, by the frequency of Rates of returns in some making. Previous sections of this economic rate of return estimates in project Bank-supported projects chapter have cited areas where infor- reports. To illuminate this, figure 3.4 shows the exceed 50 percent. mation on results is improving at the percentage of investment projects, by year, that Box 3.4: True Pilot Projects: Why Are They Not Used More Often? A substantial number of development projects are begun with- impact evaluation section of this report, as well as from IEG's out hard evidence that the anticipated benefits will be of the mag- health, nutrition, and population (HNP) sector evaluation (IEG nitude anticipated. Faced with this uncertainty, a prudent 2009e). While 32 recently closed projects in the HNP portfolio in- approach would be to implement some projects on a modest clude pilot components with the objective of testing the effec- scale for a few years, test the benefits rigorously, and scale up tiveness or acceptability of an activity, only 8 even mentioned a the project if the benefits prove worth the costs. This is one plan for a control group. Only four actually conducted the pro- way to reduce the risk of large expenditures on unsatisfactory posed evaluation of the pilot, and all four included a detailed eval- projects. uation design in the Project Appraisal Document. However, it is rare for Bank-financed projects to be pilots in Pilot projects are, by design, small in scale and intended to sup- this sense of the word. IEG finds that many projects claim to be port learning. Thus, the evaluative output is essential, and les- pilots, but the term is used loosely. Such projects rarely contain sons from experience in HNP suggest that for a meaningful midterm testing or enumerate objective criteria that will trigger evaluation to be completed, the evaluation must be planned in de- scaling up (or do not even have a scaling-up plan within the life tail while the project is being designed. of the project), and they often do not collect baseline data. The Why then are pilots not pursued more frequently? One possi- scaling up is evidently intended to happen in future projects. If this ble answer is that not scaling up is viewed as failure, and task team is the case, the pilot projects should have rigorous evaluations to leaders resist exposing themselves to this outcome. If this is the inform future projects, but this rarely happens. impediment, one solution is to reward managers similarly for Evidence for this comes from the review of projects for the scaling up good projects and for not scaling up poor projects. Sources: Author's estimates and IEG 2009e. 38 T O W A R D M E A S U R A B L E R E S U LT S A N D I M P R O V E D P E R F O R M A N C E D ATA contained economic rate of return estimates in M&E systems, introducing a compre- Positive benefits alone do their initiating documents. This percentage has hensive Results Monitoring System for not justify projects. declined from approximately 70 percent in the IDA, and expanding rigorous evalua- 1970s to just over 20 percent in the early 2000s. tion of project interventions. However, when judged against the criteria of meaningful, ac- Second, in contrast to the large amount of curate, and decision-relevant information, the activity surrounding M&E, impact evaluation, evidence here suggests that further progress is and the IDA15 RMS, there is little activity or needed. commentary in current Bank documents on estimating economic rate of return. The IDA15 Much attention has been given to upgrading M&E RMS could have chosen to report rates of return systems. IEG's assessments of M&E quality in in IDA projects as a way to summarize perform- closed projects reviewed in fiscal years 2007 and ance, but it was apparently decided to eschew 2008 rate the quality as modest or this approach. Reported economic rates of negligible in three-fifths of cases. Rates of return in some return estimated at closing exceed 50 percent in Indicators are too numerous and often Bank-supported projects some Bank-financed projects: if correct, this is a measure outputs rather than out- exceed 50 percent. considerable achievement. Finally, if Bank staff comes; baselines are infrequently have decided not to use cost-benefit analysis or collected; and few projects collect the data that net present value calculations as project would be required to conduct good M&E of selection criteria, it raises the question of what impact. These IEG data tend to cover projects decision criteria have or would be used instead. approved several years earlier and provide a reference to gauge improvements going forward. Conclusions The Bank is in the midst of far-reaching efforts to The Results Monitoring System for IDA is an improve performance information: strengthening ambitious effort to measure development results Figure 3.4: Percentage of Projects with Economic Rate of Return Estimates, by Year of Approval 80 Percent with economic rate of return at entry 50 20 1969 1980 1990 2001 Year of approval Sources: World Bank data, IEG calculations. 39 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 in a coordinated fashion and aggregate them in a Overall, it is still rare for the full requisite data to be way that is relevant to prospective IDA donors. collected to permit quantitative impact evaluations. The evidence suggests that the initiative has made progress but faces ongoing challenges to Assessments of project performance through the full implementation. life of a project continue to appear inaccurate. They provide overly optimistic assessments until Impact evaluations hold promise to provide it is too late to take corrective action. accurate estimates of impact, and they are increas- ingly being pursued in projects financed by the Economic cost-benefit analysis is the traditional Bank. An assessment based on a review of projects method to organize information on benefits and in three sectors that started after 2002 and closed costs of projects and render it relevant for before December 2008 reveals a low making decisions. Current Bank policy is that The use of cost-benefit rate of baseline data collection (27 economic analysis of net present value is analysis is declining at percent) and a concentration of what required for all Bank-supported investment the Bank. little data collection there is in two projects. Nevertheless, this kind of economic Regions--East Asia and Europe and analysis appears to be in decline at the Bank, at Central Asia. Bank-generated data reports a higher least as indicated by the declining proportion of rate--80 percent--for more recent (2007) projects with reported economic rate of return projects, but measures only the existence of estimates. IEG will conduct a more in-depth baseline data for one indicator, a relatively low bar. analysis of this issue in fiscal 2010. 40 PART II Achieving Sustainable Development Lessons from the Bank's Experience Chapter 4 Evaluation Highlights · Huge environmental challenges will remain on the other side of the global financial crisis. · New sources of global financing have contributed to a recovery in Bank commitments. · Mainstreaming of environmental is- sues across other sectors has been shallow. · Analytical work and advocacy have enhanced global environmental awareness. · Various external and internal con- straints hamper the Bank's effec- tiveness at the country level. Parched soil by the White Nile, Khartoum, Sudan. Photo by Arne Hoel, courtesy of World Bank Photo Library. World Bank Support for Environmental Sustainability hile the global financial crisis has attracted worldwide attention, com- W bating environmental degradation and ensuring environmental sus- tainability compatible with economic growth and poverty reduction remain among the foremost challenges of our times. The sustainability of the world's arable land, These evaluations include, but are not The Bank has a central water, forests, air, climate, and other natural limited to, the following: Environmen- role in helping countries resources is under ever-increasing pressure; tal Sustainability: An Evaluation of address environmental their careful stewardship is essential for long- World Bank Group Support; Climate challenges. term growth and well-being in both developed Change and the World Bank Group and developing countries. The risks associated Phase I: An Evaluation of World Bank Win-Win with climate change have recently taken center Energy Policy Reforms; The Development stage, but other environmental challenges--air Potential of Regional Programs: An Evaluation and water pollution, soil erosion and desertifica- of World Bank Support of Multicountry tion, water scarcity, and loss of biodiversity-- Operations; a forthcoming review on the Bank's remain enormous and increasing. These forestry activities updating the portfolio and challenges affect all countries and peoples, but performance data through fiscal year 2008; and the poor are often most strongly affected and supplementary materials from IEG's ICR Reviews, least able to mitigate environmental risks. CASCR Reviews, and Global Program Reviews (GPRs). These two chapters synthesize the major The World Bank has a central role to play in findings and lessons from these recent evalua- helping developing and transition countries tions. address environmental challenges. Its broad and long-term mandate, global reach, deep knowl- The Bank's Approach to the edge base, diverse set of public and private clients, Environment Has Evolved in Response wide range of financial and technical assistance to Changing Needs instruments, and substantial convening power position it to help mobilize the world's knowledge Environmental Sustainability Can Go Hand-in- and resources to address critical environmental Hand with Growth and Poverty Reduction, but concerns. Requires a Shift in Incentives Sustainable development is usually defined as The following two chapters spotlight the Bank's "development that meets the needs of the present activities and performance in helping countries to without compromising the ability of future address these environmental challenges, based generations to meet their own needs." For this to upon the findings of recent IEG evaluations. happen, the productivity of today's investments 45 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 in physical, human, and social capital must more given their limited economic opportunities and than compensate for any direct or indirect losses greater vulnerability. Rapid depletion of forests in society's natural capital--the environment. and fisheries poses major threats to rural liveli- Goal 7 of the U.N. Millennium Development hoods, and global warming is expected to dispro- Goals specifically phrases this as a target to portionately affect the world's poorer regions "integrate the principles of sustainable develop- through adverse weather conditions and sea-level ment into country policies and programs and rise. Better management of forests is critical both reverse the losses of environmental resources." for efforts to reduce emissions from deforesta- tion1 and to improve the livelihoods of the For example, economic growth is strongly associ- hundreds of millions of the world's poorest ated with greater greenhouse gas emissions, but people who live in and around forests. public policies greatly affect the exact nature of this relationship in any given country by altering Thus, efforts to improve environmental sustain- economic incentives and technological choices. ability and reduce poverty can go hand-in-hand. Some countries have followed a greener growth But this requires careful policy choices that facili- path, as evidenced by the wide differ- tate greener paths to economic growth and that ences in energy use and emissions effectively address the genuine trade-offs Environmental among countries, even after adjusting between the economic winners and losers within sustainability can be for levels of development (figure 4.1). countries, as well as between the present and the compatible with both Holding income constant, there is as future. Changing direction requires recognizing growth and poverty much as a sevenfold difference in the real costs that have to be borne in each reduction. emissions intensity among countries. country context and softening the impacts on Some countries that used to have a those most seriously affected. Greater awareness relatively high level of emissions per capita have of short- and long-term environmental con- succeeded in reducing emissions while also sequences has to be mainstreamed into invest- increasing their income by substantial restructur- ment decisions and choices of technology in all ing of their economies and adjustments in energy economic sectors; natural resources such as prices (most notably in transition economies in water and pollutants such as carbon dioxide have Central and Eastern Europe) (IEG 2009a, pp. to be priced to reflect their true economic costs; 13­17). and appropriate mitigation, adjustment, and compensation measures have to be put in place Moving to a cleaner growth path need not punish as economies adjust to a cleaner, environmen- the poor. Poor countries emit only a tiny fraction tally sustainable growth path. of emissions per capita compared with rich ones, and providing 2 billion poor people with basic The World Bank's Approach to the electricity access would boost world greenhouse Environment Has Become Increasingly gas emissions by less than 0.4 percent of current Proactive and Multilevel emissions, even if power was provided by the The World Bank has given increasing attention to most carbon-intensive means. The energy access environmental issues since the 1970s, as shown agenda can proceed independently of the mitiga- in the timeline in figure 4.2. The Bank began with tion agenda (IEG 2009a, pp. 19­20). a "do no harm" approach--designing and applying environmental safeguards to guard The linkages between growth, poverty, and the against negative environmental impacts in Bank- environment also run in the other direction, since supported projects. Then it took on a broader environmental degradation--soil and water loss, and more proactive "do good" approach in the declining natural resources, and pol- 1990s, as the world's attention shifted increas- Good public policies are lution--tends to reduce productivity ingly to environmental issues. The 1992 World necessary for greener and lower economic growth. The poor Development Report: Development and the economic growth. tend to be disproportionately affected Environment (World Bank 1992), prepared for 46 Figure 4.1: Absolute Changes in CO2 Emissions and Income, 1992­2004 30 ARE 20 USA AUS CAN SAU FIN CZE NLD 10 KAZ RUS DEUBEL KOR ISR SGP GBR IRL DNK JPNAUT POL GRC NZLESP ITA NOR ZAF CO2 per capita (tons per person: 1992­2004) UKRBLR SVK FRA CHE IRN BGR MYS HUN PRT SWE UZB VEN HRV ROU AZE CHN MEXCHL ARG JOR THA TUR SYR DZA EGY DOMTUN MDA BRA IDN ECU W O R L D B A N K S U P P O R T F O R E N V I R O N M E N TA L S U S TA I N A B I L I T Y MAR COL CRI KGZ BOL PER 1 VNM IND YEM TJK HND GTMPHL SLV PAK ZWE NIC GEO LKA PRY 0.5 AGO NGA SEN BEN TGOCIV KEN SDN GHA BGD ZMB ERI HTI CMR 0.1 TZA NPL MOZ ETH ZAR 10 20 30 40 GDP per capita purchasing power parity (constant 2000 intl $) ($000 per person: 1992­2004) Source: IEG 2009a, p. 14. Note: Blue = CO2 1992 > CO2 2004 and GDP 1992 < GDP 2004. Countries with population < 4 million (2004) are excluded. 47 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 the Rio Earth Summit in 1992, put forward a the goals of combating deforestation and doing fourfold agenda that called for, in addition to no harm enshrined in the previous strategy to safeguards, enhanced environmental steward- encompass the positive goals of poverty reduction ship through focused environmental (including strengthening the rights of forest- projects and programs, mainstream- dependent people), sustainable economic growth The Bank has moved ing of environmental issues in a broad (including improved forest governance), and beyond a "do no harm" range of Bank activities, and increased protecting local and global environmental services approach to the attention to global issues of environ- and values (including expanding protected areas environment. mental sustainability. and developing markets and finance for global public goods such as biodiversity and carbon The first World Bank Environment Strategy, sequestration). issued in 2001, placed environment clearly in the context of the Bank's mission to combat poverty, Both the Environment and Forest Strategies moving beyond the earlier focus on conservation underwent a thorough and expensive consulta- and doing no harm. It set three interrelated tion process over an extended period before objectives--improving the quality of life being approved by the Bank's Board. More (focusing on linkages between environment and recently, climate change has become an poverty, health, and vulnerability), enhancing the additional area of focus for both environment quality of growth (including concerns with and forestry activities, although many Bank environment governance), and protecting activities over the past two decades have had regional and global environmental clear implications for climate change. commons. The 2001 Environment As endorsed by the 2001 Environment Strategy, Strategy offered a more The Bank's 2002 Forest Strategy also mainstreaming environmental sustainability proactive "do good" had a much broader scope than the requires that "environmental improvement will approach. 1991 Forest Strategy. It went beyond come not just from interventions directly on Figure 4.2: The World Bank Group and the Environment: A Summary Timeline of Activities 1970 1980 1990 2000 World Bank project focus: "do no harm" IFC: Deepening attention to project- World Bank Group: Increasing focus level impacts from 1991 on climate change World Bank Group: 2001 Environmental Strategy World Bank: World Development Report IFC: Equator Principles 2002, Advisory Services (for Rio Summit) (1992) World Bank: 2003 World Development Report World Bank: Increasingly proactive role from 1992 Fourfold agenda: Safeguards, Stewardship, Mainstreaming, Global Sustainability MIGA: Enhanced project-level focus from 1998 Source: IEG. Note: IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency. 48 W O R L D B A N K S U P P O R T F O R E N V I R O N M E N TA L S U S TA I N A B I L I T Y environmental issues but also from the incorpo- transportation, and urban develop- The 2001 strategy ration of environmental sensitivities and ment in addition to environment, emphasized objectives into the work of other sectors, both in agriculture and rural development, and mainstreaming the Bank and in our client countries."2 This social development--that are responsi- environmental issues into moves significantly beyond the simple applica- ble for overseeing almost 95 percent of the work of many sectors. tion of proper safeguards to protect the environ- the Bank's environment-related proj- ment, although adequate attention to safeguards ects. While the merger provides the opportunity is a necessary precondition to ensure that a for stronger collaboration among these sectors, it project does no harm.3 It also includes a compre- also entails the risk that environmental concerns hensive assessment of risk under the new, may--with the exception of climate change-- harmonized Operations Policy and Country become less visible now that those responsible are Services [OPCS]/IEG criteria for implementation subsumed within a much larger network (IEG completion reporting (2006) so that the relative 2008c, p.75). importance of environmental risk is weighed against the impact on an operation's develop- Today, the Bank is working at the project, country, ment outcome as a whole. regional, and global levels to operationalize the concepts of sustainable development and In 2006­07, the World Bank merged the Environ- environmental sustainability. At the project level, it mentally Sustainable Development Network and seeks to mainstream an environmental conscious- the Infrastructure Network into the Sustainable ness into the design and implementation of Bank- Development Network, specifically to enhance supported projects in many sectors. At mainstreaming of environmental issues through- the country level, it provides AAA to out the new network and to strengthen the Bank's help countries formulate country The Bank is engaged in focus on sustainability as it increases its investment strategies and put in place economy- environmental issues in infrastructure. The new network now comprises wide policies and institutions to from the project to the the very sectors--energy and mining, water, promote environmental sustainability. global level. Box 4.1: IFC and MIGA Have Also Strengthened Attention to the Environment in Recent Years The ARDE is a World Bank, rather than a World Bank Group, laboration across and within the World Bank Group have been sub- report. However, IEG's evaluation of environmental sustain- optimal, since policy and regulatory reforms by the public sector ability (IEG 2008c) looked at the effectiveness of the entire create the framework for private sector activity, and the World Bank World Bank Group's support to the environment from 1990 to Group has the tools to support both the public and private sectors. 2007. While neither IFC nor MIGA have devoted major re- For example, the Bank can help create opportunities for private sec- sources to investments focused directly on environmental is- tor participation in environmental utilities through public-private sues, both have increasingly strengthened their attention to partnerships, IFC can help to find partners and undertake joint fi- environmental impacts over time. In 2003, IFC developed the nancing of these partnerships, and MIGA can provide insurance Equator Principles to help guide lending practices of private to lower the risks to private investors. Good examples of such co- banks; 60 banks now participate (although actual impact has ordination include policy and projects in the power and water not been evaluated). In 2006, IFC adopted new Policy and Per- sectors in Brazil, China, and India. However, there have also been formance Standards on Social and Environmental Sustain- occasions when IFC and the Bank have pursued different ap- ability, and MIGA did the same in 2007. IFC has also recently proaches or priorities--for example, differing approaches to de- started a business line for technical assistance on environ- velopment in the Brazilian Amazon. These coordination issues mental and social sustainability, which has also not been eval- are presently being addressed through World Bank Group col- uated due to its recent timing. laboration on the preparation of the new Environment Strategy, These improvements notwithstanding, coordination and col- scheduled for launch in 2010. 49 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 At the regional and global levels, it is helping to reforms.4 Excluding development policy opera- raise sources of grant finance for country-level tions, ENRM investment lending in 2006­08 was investments that have regional and global environ- still 35 percent below that of 1993­95 in nominal mental benefits, so that the costs of making these terms. investments are shared by the entire global community. New Sources of Financing Are Driving the Thematic Agenda World Bank Support for the Environment The sources of financing for Bank-supported Has Recovered Since 2002 projects have become more diversified. The Environment and natural resource management number and volume of International Bank for (ENRM) is a theme, not an economic sector. Reconstruction and Development [IBRD]/IDA Therefore, the Bank's portfolio of environmen- investment operations have declined since the tal projects consists of those projects--about 2001 Environment Strategy was issued, compared one-third of all projects--coded with at least with the previous seven years, 1995­2001 (figure one environmental theme but crossing many 4.4). But development policy operations have sectors. The share of the environment theme in increased, as have projects supported by the the Bank's total commitments has averaged Global Environment Facility (GEF) and other about 11 percent since 1990, peaking at 18 trust funds (appendix table D.3). percent in 1994 and declining sharply to a low of 5 percent in 2002, before recovering (appendix Global and regional programs--including the table D.1). GEF, the Multilateral Fund to Implement the Montreal Protocol, various carbon finance New ENRM commitments have programs, and the Climate Investment Funds Bank lending for the recovered from their low of $1.1 billion approved in 2008--have become significant environment has in 2002 (figure 4.3). But a significant sources of financing for country-level environ- recovered since 2002 and portion of the increase since 2002 has mental investments. These concessional funds sources of financing for comprised development policy opera- are helping the Bank to overcome the limitations environmental projects tions providing budgetary support in of its country-based model--discussed in the have diversified. exchange for policy and institutional Annual Review of Development Effectiveness Figure 4.3: Bank Commitments to ENRM Operations, Fiscal Years 1990­2008 4,000 3,500 All operations 3,000 2,500 US$ millions 2,000 Investment 1,500 operations 1,000 Development policy operations 500 0 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 Approval fiscal year Source: Appendix table D.2. 50 W O R L D B A N K S U P P O R T F O R E N V I R O N M E N TA L S U S TA I N A B I L I T Y Figure 4.4: Bank-Implemented ENRM Operations by Source of Financing, 1995­2001 and 2002­08 a. Number of operations b. Commitments (US$ millions) 500 16,000 14,000 400 12,000 300 10,000 8,000 200 6,000 4,000 100 2,000 0 0 IBRD/IDA IBRD/IDA GEF- Other IBRD/IDA IBRD/IDA GEF- Other investments DPLs financed trust funds investments DPLs financed trust funds 1995­2001 2002­08 1995­2001 2002­08 Source: Appendix table D.3. Note: DPL = Development Policy Loan. 2008: Shared Global Challenges (IEG 2008a)--in also--and not surprisingly--led to shifts in the terms of addressing global environmental thematic focus of Bank commitments (figure 4.5). concerns, such as climate change and deforesta- The decline in IBRD/IDA investment lending in tion, when the benefits of country-level invest- 2002­08 compared with 1995­2001 has coin- ments are not easily appropriated at the national cided with a declining incidence of projects level or where the costs of these investments fall coded to pollution management and environ- disproportionately on individual countries. mental health. The increase in GEF and carbon finance is directly related to the increased These expanding sources of financing have emphasis on climate change. Figure 4.5: Bank-Implemented ENRM Operations by Environmental Theme, 1995­2001 and 2002­08 Pollution management Water resource management Climate change Environmental policies and institutions Land administration and management Biodiversity Other environmental management 0 1,000 2,000 3,000 4,000 5,000 6,000 Millions of U.S. dollars 1995­2001 2002­08 Source: Appendix table D.5. 51 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 The thematic focus has The changing sources of concessional resent 5 percent of total environment commit- shifted toward climate finance for country-level investments ments. Some are multicountry projects, some are change. will continue to influence the the- related national projects, and others are formally matic focus of the Bank's environmen- constituted regional partnership programs (such tal projects. Biodiversity projects, for example, as the Black Sea and Danube Partnership, the Nile are somewhat more dependent on the provision Basin Initiative, Africa Stockpiles, and TerrAfrica) of concessional finance than other focal areas. that draw upon IBRD/IDA, GEF, and other trust Although biodiversity remains a highly important funds to finance their activities in individual area of Bank support, new operational and countries. The largest numbers are in Africa, procedural reforms introduced by the GEF may followed by Latin America and Europe and Central affect the ability of the Bank to develop as many Asia (figure 4.6). blended investments for biodiversity conserva- tion as it has historically (box 4.2). The increased number of regional projects in Africa partly reflects the influence of the New Changes in IDA Rules Have Led to Partnership for Africa launched in 2001, which More Regional Environmental Projects has called for greater regional cooperation and Regional environmental projects are another area integration through increased attention to where new sources of finance are helping the Bank regional partnerships. But IEG's recent review of move beyond its individual country-based model Bank support for multicountry operations (IEG of lending as well as influencing the nature of the 2007d) found that the regional pilot initiative Bank's environmental operations. Bank commit- introduced in IDA14 created an incentive for the ments to regional projects to assist shift. Under this pilot, which was continued Regional environmental neighboring countries to more under IDA15, two-thirds of a country's share of projects have grown, effectively manage their shared envi- the cost of a regional project could be covered by particularly in Africa, ronmental resources (such as lakes and resources from the overall IDA funding, so that due to changes in IDA river systems) more than doubled from only a third of the IDA resources would come rules. 1995­2001 to 2002­08, and now rep- from the country's individual IDA allocation. IDA Box 4.2: Lessons from the Midterm Review of the GEF's Resource Allocation Framework GEF's new Resource Allocation Framework (RAF), introduced in ticipation. At the time of the review, the World Bank share had 2006 and now called the System for a Transparent Allocation of dropped from more than half of GEF resources to 32 percent of Resources (STAR), represents "a system for allocating resources GEF RAF resource utilization in the two focal areas of biodiver- to countries in a transparent and consistent manner based on sity and climate change, while the United Nations Development global environmental priorities and country capacity, policies and Program (UNDP) share increased to 43 percent, up from 28 per- practices relevant to successful implementation of GEF proj- cent historically. The role of the seven executing agencies also ects." A midterm review of the RAF conducted by the GEF Eval- increased to 17 percent of RAF utilization, compared with 2 per- uation Office found that the new system was proving more cent of all historical resources. These shifts reflected the spread- successful in channeling GEF resources to countries with high ing of small RAF allocations over many countries, which made global environmental benefits as measured by the GEF Environ- it more difficult for the World Bank to blend GEF finance with Bank mental Index, but less so to countries with strong performance lending, since other funds were now easier to utilize than GEF as measured by the GEF Performance Index. RAF support. The UNDP has greater ability to provide technical The midterm review also found that the RAF, coupled with assistance and capacity building supported by local offices and other operational changes (such as a change in the rules gov- is more ready to engage in relatively small projects under the RAF erning financing of project preparation), affected agency par- (now STAR). Source: GEF Evaluation Office 2009. 52 W O R L D B A N K S U P P O R T F O R E N V I R O N M E N TA L S U S TA I N A B I L I T Y Figure 4.6: Bank-Implemented Global and Regional ENRM Operations by Region, 1995­2001 and 2002­08 a. Number of operations b. Commitments (US$ millions) 40 500 35 450 400 30 350 25 300 20 250 15 200 150 10 100 5 50 0 0 Africa Latin America Europe East Asia Middle Africa Latin America Europe East Asia Middle and the and and East and and the and and East and Caribbean Central Pacific North Caribbean Central Pacific North Asia Africa Asia Africa 1995­2001 2002­08 1995­2001 2002­08 Source: Appendix table D.4. commitments for regional projects increased Recent IEG evaluations have identified successful more than threefold, from $67 million in cases of mainstreaming--for example, in land 1995­2001 to $310 million in 2002­08--which and watershed management, urban environ- represented 40 percent of all commitments to ment, and river basin management projects in regional projects in the second period. China--but found that many Bank managers and staff continue to Mainstreaming Environmental approach their work from a sector The 2001 Environment Concerns across Sectors Is More perspective (IEG 2008c, p. 76). The Strategy recognized the Apparent than Real Bank's matrix management structure, importance of The 2001 Environment Strategy recognized the in which the Bank's task team leaders mainstreaming importance of mainstreaming environmental are primarily accountable for work in environmental issues issues across many sectors--that is, addressing their own sectors, influences the across sectors. environmental issues from a multidisciplinary incentives and opportunities for cross- perspective, with strong cross-sector collabora- sector collaboration. tion and through projects mapped to many different sector boards (box 4.3). For example, One measure of mainstreaming--the volume of effective work on land and watershed manage- ENRM commitments mapped to sector boards ment needs to engage technical expertise from other than environment--has actually declined infrastructure, agriculture, urban development, (notably in agriculture, energy, and transport) and social development. And a comprehensive since the 2001 Environment Strategy was issued approach to reducing greenhouse gas emissions compared with the previous seven years, from reduced deforestation and land degrada- 1995­2001 (figure 4.7). Most of the increase in tion requires strong collaboration between development policy operations, GEF, and other technical forestry experts, carbon finance trust fund resources during the second period has experts, economists looking at poverty and been mapped to the Environment Sector Board, growth, and public management specialists with causing its share of total commitments to increase expertise on how to build administrative and from 21 percent to 27 percent between the two judicial capacity. periods (appendix table D.6). 53 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Box 4.3: Mainstreaming Environmental Themes into Bank-Supported Projects Each Bank-supported project is supervised by a task team leader identified with at least one environmental or national resource man- who reports to a regional manager. This manager is represented agement (ENRM) theme and mapped to sector boards other than on a Bank-wide sector or thematic board, and the project is the Environment Board. Of the 871 projects approved between thereby "mapped" to--or becomes the responsibility of--that 2002 and 2008 that identified at least one environmental theme, only particular board. Each project can also identify up to five sec- 29 percent were mapped to the Environment Board--the other 71 tors of the economy supported by the project (such as agricul- percent were "mainstreamed." The extent to which the other ture, energy, transportation, and the like) and up to five 1,690 projects approved during the same period (those not identi- developmental themes (such as environment, social, or urban de- fied with any environmental theme) were also addressing envi- velopment) promoted by the project. ronmental issues is unknown, apart from information made available Operationally speaking, "mainstreaming" refers to projects through the monitoring of environment-related safeguards. Number of Bank-supported projects approved, fiscal 2002­08, by sector board mapping and thematic coding Sector board mapping Thematic coding Environment Other sector boards Total At least one ENRM theme 249 622 871 No ENRM themes 3 1,687 1,690 Total 252 2,309 2,561 Source: Appendix table D.7. A second measure of mainstreaming--the Another 18 percent of projects had environmental proportion of ENRM projects (with at least one components but lacked stated environmental environmental theme) in the total number of objectives (appendix F). projects mapped to each sector board--has also declined since 2001 compared with the The absence of environmental themes in projects But the degree of previous seven years (appendix table mapped to the Health Sector Board is striking-- mainstreaming appears D.7). To find out the nature of only 2 projects in 14 years included such themes. to have declined since mainstreaming in a typical project in There are indeed few compelling examples of 2001. each sector, IEG conducted an in- integrated health-environment projects in the depth analysis of the environmental Bank's portfolio (box 4.4), even though the 2001 objectives and components of the entire popula- Environment Strategy confirmed that unsafe tion of ENRM projects that were approved since water and air pollution are major factors in 2000, closed in 2007 and 2008, and have been generating or exacerbating diseases and their reviewed by IEG--a total of 51 projects. One- attendant economic costs, especially for the poor.5 quarter of these projects were the responsibility of the environment sector (mainly biodiversity Health themes have more commonly featured in conservation projects), and therefore had the Bank's water projects since the adoption of environmental development objectives and the Bank's water resources management strategy components. At the other extreme, about 30 in 1993. But IEG's recent review of the Bank's percent of the projects had no environmental work in the health, nutrition, and population objectives or components. They were coded with sector found that only 10 percent of water supply an environmental theme only because their and sanitation projects between 1997 and 2006 choice of technologies in the energy had objectives to improve health outcomes, and Cross-sector and transport sectors, for example, only 3 percent targeted health outcomes among collaboration between were presumed to have positive the poor.6, 7 water supply and health externalities in terms of reduced has been low. pollution or greenhouse gas emissions. It is still unclear to what extent the merger of the 54 W O R L D B A N K S U P P O R T F O R E N V I R O N M E N TA L S U S TA I N A B I L I T Y Figure 4.7: Bank-Implemented ENRM Commitments by Sector Board, 1995­2001 and 2002­08 Environment Agriculture & Rural Development Energy & Mining Water Transport Urban Development Other 0 1,000 2,000 3,000 4,000 5,000 Millions of U.S. dollars 1995­2001 2002­08 Source: Appendix table D.8. two former Bank networks into the Sustainable Reviews, Strategic Environmental Analyses, and Development Network is leading to stronger so on.8 But the studies have not always been collaboration, particularly at the project and influential: Poverty and Social Impact Analyses of country levels. energy pricing and efficiency appear to have had uneven impact.9 In addition, it is important for the Social Development Network to forge stronger ties to Box 4.4: Meaningful Mainstreaming: the Human Development and Poverty Reduction Enhancing the Quality of Life through an and Economic Management Networks, given the Integrated Health-Environment Project in Eritrea linkages between environment, health, growth, and poverty reduction. One compelling example of an integrated health-environment project Analytical Work Is Helping to Catalyze is the HIV/AIDS, Malaria, STD, and Tuberculosis Control Project (HAM- Innovative Environmental Instruments SET) in Eritrea. The project paired attention to treatment with inter- Recent IEG evaluations underscore the Bank's ventions that tackled underlying environmental threats. It implemented important role in generating and disseminating environmental control work at the community level, including larvici- knowledge. IEG found that Bank economic and dal activities (using chemicals and biological agents), source reduc- sector work (ESW) and research (including tion for thousands of mosquito-breeding sites, research on vector country-level ESW, regional and global work- control, and the planting of Mim trees in some regions. According to shops, two World Development Reports, and IEG, these environmental interventions, alongside prompt diagnosis and several Policy Research Reports) have made treatment of malaria cases, were significantly associated with lower important contributions to knowledge genera- malaria incidence. tion (IEG 2008c, pp. 20­21). The list of such Source: IEG 2009c. contributions is long: Energy-Environment 55 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Analytical and advisory Bank expenditures on environment- helped to extend the benefit of this market to work has more than related ESW and TA more than doubled beneficiaries that would otherwise be bypassed: doubled since 2002. from 2002 to 2008 (figure 4.8).10 for example, through the development of the Roughly half has been financed by the Community Development Carbon Fund that Bank's administrative budget and the other half supports small projects in low-income countries by donor trust funds. Roughly half was mapped to and projects in poor areas in other countries. In the Environment Sector Board--implying less some cases, the Bank has been the first to mainstreaming in environment-related ESW and purchase emission reductions in specific TA than in lending. countries, poor communities, or technologies or sectors. It has also been a major contributor to Almost 40 percent of Bank expenditures on the development of internationally approved environmental ESW and TA have been global or Clean Development Mechanism methodologies regional in nature, compared with only 5 percent and has contributed its experience to decisions of lending. Indeed, the environment is among by the parties to the UN Framework Convention the top sectors Bank-wide in terms of global and on Climate Change and the Clean Development regional analytic work (IEG 2008e). This global Mechanism Executive Board. analytic work has frequently been financed by global partnership programs such as the However, the state of the carbon market today Program on Forests (PROFOR), which conducts suffers from uneven access across countries. analytic work on improving policies and institu- Eighty-five percent of the Clean Development tions for sustainable forest management.11 Mechanism market share is in Brazil, China, and India. Projects in Africa are only beginning to Advocacy for global public goods such as mitigat- emerge. Many middle-income countries with high ing climate change and developing carbon emissions have relatively low presence in carbon markets has also been an important role for the markets. And, until recently, reductions from World Bank. The Bank launched the reforestation and avoided deforestation have Environmental work has Prototype Carbon Fund in 2000 and largely been absent. The agreement reached at been more global and now manages a series of carbon funds Bali in December 2008 to move forward on the regional than has work in and facilities, most notably the Forest Reducing Emissions from Deforestation and other sectors. Carbon Partnership Facility. It has Degradation (REDD) agenda can potentially help Figure 4.8: Environment-Related Analytical and Advisory Services, Fiscal Years 2002­08 a. Type of activity b. Source of financing 14,000 14,000 12,000 12,000 Technical Trust funds 10,000 assistance 10,000 US$ thousands US$ thousands 8,000 8,000 6,000 6,000 Bank budget 4,000 Economic and 4,000 sector work 2,000 2,000 0 0 2002 2003 2004 2005 2006 2007 2008 2002 2003 2004 2005 2006 2007 2008 Year Year Source: Appendix table D.12. Note: Share of total AAA expenditures assigned to the environment themes. 56 W O R L D B A N K S U P P O R T F O R E N V I R O N M E N TA L S U S TA I N A B I L I T Y countries with tropical moist forests access carbon for environment-related policy making Countries' incentives to markets. The Bank, among others, has a role to and management is uneven and tends act are stronger when play in helping these countries build capacity to to be particularly weak in smaller and environmental effects are measure and verify forest-related emissions and poorer developing countries and at felt locally. to bring these assets to market. Through the the subnational level. Capacity has BioCarbon Fund the Bank has supported a expanded most where Bank support has been prototype to demonstrate and benchmark the use strong and continuous over an extended period, of carbon finance in forestry and agriculture as in China. Capacity building has been less projects with strong co-benefits in biodiversity, effective where Bank support has wavered over land protection, and poverty reduction. time and/or used shifting instruments--as in India and Africa (IEG 2008c, pp. 72­73). Internal and External Constraints Shape Lack of country and Inhibit the Bank's Effectiveness Third, internal constraints have knowledge and capacity Recent IEG evaluations have found a number of inhibited Bank effectiveness. IEG has have inhibited lending. internal and external constraints to the Bank's found that the Bank faces significant effectiveness at the country level. First, demand knowledge gaps, including measurements of the for environmental projects has been low in many environmental outcomes arising from its ENRM countries. There is tension between the Bank's portfolio (IEG 2008c, p. 74). The evaluation also desire to implement a comprehensive agenda for found that the Bank has been only partially environment and the strategic dialogue at the effective at internal knowledge dissemination country level that leads to selectivity in Bank about what works and what does not. A number support across sectors (box 4.5). The modest of less effective approaches--such as the use of demand for environmental investments in credit lines for industrial pollution abatement-- comparison with demand in other sectors is only have been discontinued in some but not all partly offset by the availability of GEF and other Regions, and the lessons of more effective concessional resources. Where the benefits of projects--such as certain land and watershed environmental action are localized, as with management projects--have not yet been pollution control in China, a country's incentive incorporated in all Regions. to take action is stronger. A country's demand for lending may also be constrained by domestic Fourth, current Bank staff capacity in the environ- political competition and corruption surrounding mental area appears to be inadequate to meet valuable resource rents. But the China example (box 4.6) shows what can be achieved when both the country and the World Bank dedicate high Box 4.5: Optimal Allocation of Environmental levels of attention and resources to environmen- Resources Is Constrained by Political Realities at tal issues. Led by China, East Asia has received the the Country Level largest share of the Bank's ENRM commitments, both in absolute terms and relative to its share of Three years after the 2001 Environment Strategy was issued, the Bank's the Bank's total commitments (figure 4.9). China Development Economics Group conducted a gap analysis of the Bank's received 22 percent of all ENRM commitments environmental allocations (lending and AAA) to determine where ad- during 2002­08, and four countries (Brazil, China, justments in environmental support would be most effective in relation India, and Mexico) accounted for 45 percent of all to environmental burden and institutional capacity across countries. such commitments during this period. The Africa Environmental burden was measured by biodiversity loss, land degra- Region as a whole received the least amount of dation, and disability-adjusted life years (DALYs) lost due to environ- ENRM commitments relative to its share in total mental contamination--water pollution, outdoor air pollution, and Bank commitments. indoor air pollution. This proposed System for Environmental Priority Assessment was presented to the Environment Department, but its im- Second, knowledge and capacity constraints plementation was constrained by political realities at the country level. inhibit lending. Country knowledge and capacity 57 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 growing needs--not only for technical specialists, preferred large infrastructure projects that focus but also for integrating environmental considera- on electricity generation and distribution over tions into infrastructure and other types of smaller energy-efficiency projects to deal with projects (IEG 2008c, p. 76). The Bank has substan- demand-side management or appliance standards tially enhanced its skills in carbon finance and and building codes: climate change, but greater attention is needed to continue strengthening skills in forestry, natural Discussions with staff and other stake- resource management, and energy, in particular in holders are consistent with some standard working with clients on climate change adapta- diagnoses about the neglect of energy effi- tion, including the REDD agenda and ecosystem- ciency opportunities. Energy efficiency is based adaptation. This includes cutting-edge simply not as visible as energy generation. analytic and technical capabilities, operational It is hard to quickly spend large sums of skills to help borrowers prepare and implement money on energy efficiency (except with the environmentally sustainable projects, and en- mass distribution of CFLs or in some supply- hanced environmental awareness among Bank side projects), and yet energy efficiency country managers.12 projects are often complex or difficult. This makes them less attractive to managers Preference for larger Finally, internal incentives favor and agencies that use disbursements as a projects has also projects with large commitments, measure of action and large turbines as a hindered environment which can disadvantage environmental visible symbol of achievement. Energy ef- lending. initiatives. Managers and staff have ficiency is viewed by some as being less Box 4.6: The Importance of Client Commitment and Sustained World Bank Engagement: The Case of China Although recent IEG evaluations point to many examples of suc- · Support for forest plantations and marketing of commercial cessful World Bank country support, the China country program timber, helping to achieve an increase in overall forest cover stands out for the breadth, depth, and longevity of Bank support. in the country since the 1990s China faces many serious environmental challenges, including air · Strong financial and analytical support to curbing local and and water pollution, soil erosion, deforestation, greenhouse gas global greenhouse gas emissions, including through a focus emissions, and vulnerability to natural disasters. The government on energy efficiency and renewable energy in Bank-financed has recognized these enormous challenges and has shown a investment projects strong determination to address them and a willingness to engage · Successful support for phasing-out of ozone-depleting sub- the Bank in that effort. The Bank's assistance has included: stances through analytic services and lending · Assistance in designing and implementing (albeit sometimes · Consistent attention to urban air and water pollution (including, with mixed results) approaches to environmental manage- most recently, a 2007 report The Cost of Pollution in China ment, including pollution levies, public regulations, and pub- [World Bank 2007a]) and significant lending to help raise citizen lic disclosure of polluters awareness and reduce pollution in many metropolitan areas · Mainstreaming of environmental issues across the country · Support to land management in the Loess Plateau and else- program, with nearly 60 percent of all projects approved by where, resulting in environmental improvement together with the Bank from 1990 to 2007 having environmental components. sustained increases in agricultural output and rural poverty reduction Although many problems and challenges remain, the experi- · Significant lending for river basin management in the Yellow, ence over the past two decades in China shows the progress that Yangtze, and other major river basins, thereby reducing vul- can be achieved through a combination of client commitment and nerability due to flooding strong and consistent World Bank engagement. Source: IEG evaluations. 58 W O R L D B A N K S U P P O R T F O R E N V I R O N M E N TA L S U S TA I N A B I L I T Y Figure 4.9: Bank-Implemented ENRM Commitments by Region, 2002­08 Sub-Saharan Africa East Asia and Pacific Europe and Central Asia Latin America and the Caribbean Middle East and North Africa South Asia 0 5 10 15 20 25 30 35 Share of each Region in Bank commitments (percent) ENRM only World Bank total Source: World Bank data. Note: There is no appendix table for this figure. real than generation, when in fact nu- ing are helping the Bank to overcome the limita- merous analyses show that much of the tions of its country-based model--discussed in demand for energy services over the next the 2008 ARDE--by supporting country-level 30 years can be more cheaply provided investments to address global environmental through increased efficiency than through concerns such as climate change and deforesta- increased generation.[IEG 2009a, p. 83.] tion, and the volume of the Bank's environmen- tal commitments has recovered since 2002. But Yet IEG's evaluation also noted that a small group given the critical importance of environmental of dedicated enthusiasts has pursued energy- sustainability for long-term development and the efficiency projects despite this internal disincen- magnitude of the current risks facing the planet, tive, and the Bank has included energy-efficiency the transformational shift in how the Bank sees components in three recently approved Climate and addresses these issues, which is occurring at Investment Fund projects in Egypt, Mexico, and the corporate level, needs to be appropriately Turkey.13 embedded into country strategies and individual Bank operations. The China example shows what Major Lessons can be achieved when both the Bank and the client dedicate high levels of attention and Seize Opportunities to Elevate Attention resources to environmental issues. to Environmental Sustainability The World Bank has a central role in helping Mainstream Environmental Awareness into developing and transition countries address Investment Decisions and Technology Choices environmental challenges in a manner that is across Sectors compatible with economic growth and poverty Environmental improvement requires main- reduction. New sources of concessional financ- streaming environmental awareness into invest- 59 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 ment decisions and technology choices across knowledge. But the Bank's effectiveness at the sectors. The merger of the former Environmen- country level has suffered from both internal and tally Sustainable Development Network and the external knowledge and capacity constraints. Staff former Infrastructure Network into the Sustain- skills need to be strengthened in a number of able Development Network provides the opportu- technical and operational areas in order to deliver nity for stronger mainstreaming of environmental environmentally sound investment and policy issues across the expanded network. But how to reform projects. Strong technical expertise is stimulate greater attention to environmental needed in forestry, natural resource management, issues across sectors in the context of the Bank's and energy to support the efforts to mitigate and current matrix structure is an outstanding question adapt to climate change, including the REDD that management needs to address carefully. IEG agenda. Effective mechanisms for knowledge evaluations suggest area-based approaches and sharing are also needed to ensure that knowledge point to successful programs and projects that is created, lessons learned, and weak approaches have focused on the needs of a particular geo- discontinued and replaced with more successful graphic area (such as the Loess Plateau and the ones. Energy efficiency is one area where rapid Yellow River Basin in China). learning is needed. Forest policy and manage- ment practices to reduce deforestation and foster Strengthen Internal Skills and the sustainable use of forest resources are other Knowledge Sharing topics where sustained attention to knowledge The Bank's AAA has generated important creation and dissemination is needed. 60 Chapter 5 Evaluation Highlights · The Bank's direct support for the environment has yielded gains in biodiversity conservation and the development of market-based in- struments for environmental management. · However, most of the Bank's sup- port for the environment is main- streamed, and these projects lack systematic reporting of environ- mental performance. · The growing body of regional proj- ects offers an opportunity to address transnational environmental chal- lenges, but a more strategic ap- proach is needed to help resolve long-standing resource conflicts. · Global environmental programs are adding value to traditional develop- ment assistance, but greater col- laboration is needed at the country level for investments that produce regional and global environmental benefits. Glacial ice, Chile. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Achieving Environmental Results from the Project to the Global Level art I of this report addressed the challenges of implementing the Bank's P results agenda--strengthening M&E, implementing a Bank-wide re- sults measurement system, and making real-time monitoring a realistic tool to inform project implementation and restructuring. These tools can gen- erate valuable lessons and contribute to tangible results if they are based on meaningful information and are accurate and relevant to decision making. Advancing results for environmental sustain- 3)1 and that poorly designed and implemented ability imposes two additional challenges. First, monitoring systems have compromised the environmental interventions are most often quality of evaluation at all levels. mainstreamed--that is, embedded in projects managed by other sectors, whose M&E systems Project-Level Results Show Areas of typically neglect reporting on environmental Success, but Monitoring Is Weak outcomes. This lack of metrics for tracking the The main challenge to evaluation of environ- results of mainstreamed projects is not in line mental results at the project level is the absence with the spirit of the 2001 Environment Strategy of a system across sectors to take stock of, to track and evaluate the results of environmen- aggregate, and assess progress in achieving tal components across all sectors. environmental outcomes. This is less problem- atic for projects directly under the management Second, environmental results are tied not of the environment sector, but is a serious merely to individual projects but also to efforts at shortcoming for the three-quarters of projects the country, regional, and global levels. This is with environmental components that fall under because environment interventions are charac- the management of other sectors. terized by externalities (both positive and negative) that transcend the boundaries of Improvement in ENRM Project Outcomes over projects and countries. Therefore, this chapter Time Reflects General Portfolio Trends examines efforts to get results from the project, The overall outcomes of ENRM projects have country, regional, and global levels, highlighting been improving over time. There has been little the different challenges at each level. Two major difference between the overall outcomes of findings that emerge from this review are that ENRM projects (mapped to all sector boards) and good monitoring is correlated with better overall the Bank-wide average of all projects (figure 5.1). outcomes (similar to what was shown in chapter The overall outcomes of both project groups 63 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 The outcomes of ENRM have trended upward since the mid- 15 percentage point improvement in the satisfac- projects have been 1990s, from about 70 percent satisfac- tory outcomes of projects managed by the improving over time. tory in 1995 to about 80 percent Environment Sector Board between the periods satisfactory in 2008 (appendix E). 2003­05 and 2006­08. (See figure 2.2b.) IEG's evaluations shed some light on the environmen- However, a shortcoming of these overall ENRM tal performance of one subset of the portfolio-- outcome ratings is that for most projects, they do biodiversity conservation projects, implemented not reflect performance of the environmental mostly in Latin America and Africa. component. Roughly three-quarters of the portfo- lio has been mainstreamed through other sectors, The Bank has been the world's largest source of and the ratings reflect mainly the performance of support for biodiversity, including blended the managing sectors. The Bank has clearly found finance from the GEF. Biodiversity conservation it difficult to deliver on the commit- projects have achieved an increase in the number But overall outcome ment in the 2001 Environment Strategy and extent of protected areas; however, less is ratings do not reveal to implement a corporate portfolio known about the sustainability of biodiversity in results in environmental monitoring system to track the envi- the production landscapes surrounding these terms. ronmental components of sector areas, especially concerning the effectiveness of portfolios. the introduction of alternative livelihoods. Projects Managed by the Environment Sector Between 2002 and 2008, Bank assistance report- Show Strong Improvement in Results edly helped to support the creation or strength- As shown in chapter 2, there has been a sharp ening of protected areas by bringing at least 34 increase in the percentage of environment million additional hectares under protection projects rated satisfactory on outcome: there is a and an additional 16 million hectares under improved management. Most of this assistance Figure 5.1: Overall Project Outcomes of was partially funded by GEF. Further field-level Environment-Related Projects in Comparison with assessment would be required to ascertain the Bank-Wide Average, Exit Years 1995­2008 whether the protected areas supported by the Bank are being effectively and sustainably 90 managed after the project closes. 85 Biodiversity conservation projects in Latin ENRM projects America helped to collect and aggregate biologi- cal data into the larger, landscape-level environ- 80 Percent satisfactory mental management systems needed for making critical land-use decisions (box 5.1). 75 All projects The Bank has also helped to develop and promote 70 market-based instruments for environmental management, including payments for environ- mental services. The Bank assisted Costa Rica 65 in bringing some 100,000 hectares of additional land, half of which is privately owned, into 60 the national Environmental Services Payment 1995 1997 1999 2001 2003 2005 2007 Schemes, while also increasing the participation of Year women and indigenous groups. Based on this Source: IEG database. experience, the Bank has entered into a second Note: ENRM projects are those coded with at least one environment and natural resource manage- ment theme. This is a three-year moving average of both series. phase of assistance in Costa Rica and has rolled out similar schemes in Brazil, Colombia, and Mexico. 64 A C H I E V I N G E N V I R O N M E N TA L R E S U LT S F R O M T H E P R O J E C T T O T H E G L O B A L L E V E L The Sustainability of Outcomes of Environment the environmental area and to help More evidence is Projects Mirror the Bank-Wide Average countries achieve strategic objectives available about protected Sustainability is measured by assessing and relating to the environment. Achiev- areas than about the rating the risk to the development outcomes ing these results involves more than sustainability of the achieved by a project. The assessment takes into implementing successful projects. As surrounding production account both the probability and likely impact of discussed in chapter 1, IEG evalua- landscapes. various threats to outcomes, and considers how tions of country programs (CAEs) and these threats have been mitigated in the reviews of CASCRs assess both these dimensions operation's design or by actions taken during its of environmental performance. initial implementation. Overall, there has been no significant difference between the risk to Environmental Components Have Performed development outcomes of environment proj- Less Well than Overall Country Strategies ects and the risk of other Bank-supported Environmental concerns have received growing projects, either before or after the change in the attention in CASs since 1990. However, the rating system in 2006 from degrees of sustain- achievement of strategic objectives related to the ability to degrees of risk to development environment has been less than satisfactory in outcome (figure 5.2). about half of the CASCRs reviewed by IEG over the past three fiscal years, despite the moderately However, IEG has raised the risk to development satisfactory or better performance of overall outcome rating of a significant portion of environ- country programs in two-thirds of all cases. Even ment projects since 2006, particularly for biodiver- where the achievement of an environment- sity projects in Africa. This stems from a difference related pillar has been rated satisfactory, the of opinion regarding the necessary conditions for CASCR Reviews have found that overall environ- ensuring the sustainability of the benefits of mental performance has been mixed, as in Brazil, biodiversity projects. These conditions are: where land management and biodiver- sity projects have not performed as The Bank has successfully · A thorough assessment of underlying threats well as water resource management piloted and is expanding coupled with an adequately financed man- projects. And even where a CASCR its support for market- agement plan (under implementation) to han- Review has rated environmental per- based environment dle these threats over the long term formance as satisfactory, it has pointed schemes. · Well-managed and -incentivized organizational or institutional structures · Real buy-in from local communities whose Box 5.1: The Indigenous and Community Biodiversity livelihoods depend in part on the area under Conservation Project in Mexico (COINBIO) conservation · Sustainable livelihood activities that sustain or The COINBIO project put 1 percent of its project costs toward the de- raise the income of these communities velopment of an interactive geo-referenced database, the Integrated · The development and implementation of sus- Information System (SII). The biodiversity data collected for the proj- tainable financing mechanisms or long-term ect was to be fed into the SII, which would then be made publicly fund-raising schemes for site management. available through a Web site and included as part of the Clearing House Mechanism managed by the government of Mexico. The inte- The review of the Mesoamerican Barrier Reef grated information system can be viewed at: http://coinbio.iacatas System project provides a good example of this .org.mx. It includes a wide range of both alphanumeric and spatial disconnect (box 5.2). (GIS) data. However, although the SII provided information needed to determine and report on outcomes at project completion, it was less Country-Level Results Involve More than useful in providing real-time feedback for the project during Successful Projects implementation. At the country level, the Bank aims to provide Source: IEG ICR Review. strategic, analytical, and operational support in 65 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Figure 5.2: ENRM Projects in the Environmental on environmental sustainability, and the CASCR Sector Have the Same Sustainability Ratings as for Benin failed to provide any performance the Bank-Wide Average indicators related to environmental manage- ment, even though environment was included in 78 one of its major pillars. 76 74 In cases where the outcomes for the environ- 72 mental pillar were rated less than satisfactory, the most common reasons were lack of commitment 70 by the borrower (Turkey, Ukraine), lack of Percent 68 capacity in key social and environmental sectors 66 (Bosnia), lack of mainstreaming across other critical sectors (Guatemala), and the failure of 64 some environment-related projects to even 62 materialize (Maldives). Two noteworthy excep- 60 tions were Colombia and South Africa, where the 1997­2006 2006­08 environment objectives outperformed other Percent likely sustainable Percent moderate or lower complementary objectives. ENV projects All projects Source: IEG database. The Bank's Environmental Stewardship Has Note: In 2006, the new harmonized OPCS/IEG criteria for implementation completion reporting changed Been Uneven, but There Have Been Areas of the rating of sustainability to risk to development outcome (RDO). About 57 percent of projects exit- Success, Such as Energy Pricing ing in 2006 were rated under the old rating system and the remainder under the new system. Supporting environmental stewardship involves helping countries set environmental priorities, to the lack of environmental indicators. For build institutions, and implement projects for example, the CASCRs for Vietnam and Ethiopia environmental improvement and sustainability. lacked appropriate indicators to gauge progress Recent IEG evaluations have found that the Bank Box 5.2: Risks to Sustaining Environmental Outcomes: The Case of the Central America Mesoamerican Barrier Reef System The Mesoamerican Barrier Reef System (MBRS) Project aimed cant due to its assessment that, in a regional environment project to enhance protection of the ecologically unique and vulnera- financed by GEF, risk is more defined by the first criterion above-- ble marine ecosystems comprising the MBRS, the longest bar- ecological sustainability. That is, the likelihood of the sustainabil- rier reef system in the Western Hemisphere, by helping the ity of the environmental gains, based on the reduction of existing littoral states strengthen and coordinate their national policies, threats, is a more important factor to consider than the continued regulations, and institutional arrangements for the conservation support for the means to achieve these gains (such as regional col- and sustainable use of this global public good. laboration and organizational structures, monitoring and data col- Sustainability in the context of this project must be defined in lection systems, and so on). The project demonstrated the possibility terms of both ecological and program sustainability. That is, it of interregional cooperation and agreement for transfrontier nat- should maintain the biological communities and ecological ural resource management. However, the project was less suc- processes that comprise the MBRS and are responsible for the cessful in attempting to manage negative tourism impacts, promote goods and services it produces, as well as establish the institutional sustainable tourism, and create alternative livelihoods for those en- arrangements, financial commitments, and economic and social gaged in unsustainable natural resource extraction and fisheries. incentives to maintain a strategic set of well-coordinated activi- Nonetheless, the strong foundation set by this project provided a ties that will create the conditions for ecological sustainability. sound framework for future investments, and IEG recognized this IEG raised this project's level of risk from moderate to signifi- as a strong aspect of project performance in its assessment. 66 A C H I E V I N G E N V I R O N M E N TA L R E S U LT S F R O M T H E P R O J E C T T O T H E G L O B A L L E V E L has been a leader in providing analysis and financ- Rainforests. A partnership between Environmental concerns ing to governments to help address environmental the Brazilian and German govern- are receiving increasing priorities, but the Bank has not generally ments, GEF, the World Wildlife Fund, attention in country succeeded in integrating environmental steward- and the Bank, the program has programs. ship centrally or integrally into country programs, strengthened local community and based on country case studies of a representative civil society participation and contributed to the sample of countries (IEG 2008c, pp. 25, 30). establishment of extensive new indigenous ecological and extractive reserves. For example, the Bank has been a leader in helping countries reform energy pricing by Regional Projects Offer Growing removing costly subsidies while protecting the Potential to Achieve Regional poor through safety nets. Its detailed analytical Environmental Benefits work and more than 250 operations involving The purpose of regional projects has typically energy pricing issues have supported significant been to help neighboring countries manage a reforms in the transition economies of Europe common natural resource such as a body of and Central Asia (such as Georgia, Romania, water (such as the Aral Sea or Lake Russia, and Ukraine) and in Indonesia and China. Victoria), a river system (such as the But the outcomes of But a series of past CASs in India, Mexico, and Nile), or a shared ecosystem (the about half of all Pakistan repeated the same set of concerns, but Mesoamerican Biological Corridor, for environmental country with modest apparent impact. Nor has there example). Their specific evaluation objectives have been been much progress on price reform in challenges are, first, to aggregate unsatisfactory. Argentina, Kazakhstan, Nigeria, or Vietnam, or country-level results to ascertain the in the two biggest diesel subsidizers--Iran and achievements and the sustainability of regional Venezuela. environmental outcomes and, second, to assess the effectiveness and efficiency of their program- The Bank has also given modest attention to level institutional arrangements, such as gov- national policies to promote energy efficiency, ernance and management. even though its 1993 energy policy (World Bank 1993) pointed to increased energy efficiency as IEG's recent evaluation of Bank The Bank has helped an important area for attention. Potentially high- support to multicountry operations clients with energy impact approaches involving demand-side (IEG 2007d) found that regional pricing reforms and has management, as well as building and appliance projects offer substantial potential to given modest attention to standards and codes, have been underempha- achieve positive regional environmen- energy-efficiency policies. sized. As elsewhere, M&E in energy-efficiency tal outcomes, but this requires paying projects has also been weak, although there are attention to the political economy of relations some signs of improvement. One example is among countries to gain their acceptance of the projects promoting the use of compact fluores- obligations involved in acting cooperatively. The cent light bulbs, which is easy to monitor and Bank has been effective in fostering multicountry where evaluation could answer a number of cooperation to establish regional institutions critical questions for policy and program design. to address regional environmental challenges through analytical work, project preparation, and Although there is always room for more effective resource mobilization, but the 2007 IEG evaluation collaboration at the country level, the Bank has of regional programs showed the Bank to be less been fairly successful at establishing productive effective in helping countries resolve longstanding partnerships with other donors and develop- resource conflicts and establish effec- ment agencies to help countries strengthen their tive enforcement mechanisms. Regional project design is environmental management. One example of a complex, since benefits successful country-level partnership is the Pilot Regional projects have successfully and costs of participation Program for the Conservation of Brazilian built new assets or protected existing vary. 67 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 ones in ways expected to benefit participating more successful at using and applying the countries, but they have not adequately helped findings of project research than the early phase countries make the complementary policy of the Aral Sea Water and Environmental Manage- changes needed to sustain project outcomes or ment Project, Guarani Aquifer, and the Organiza- reach agreement on the use of shared resources. tion of Eastern Caribbean States Ship-Generated Regional environmental projects that are Waste Management Project. designed specifically to address issues that require trade-offs among countries (such as Global Environmental Programs Add water resource management) are by their nature Value to Country-Level Efforts more challenging than projects that support Global partnership programs have become an compatible interests (such as preventing the increasingly important modality for channeling spread of HIV/AIDS). and delivering development assistance to address global concerns that can only be Accountable governance arrangements take time addressed, or addressed more efficiently, by to establish but are essential to gaining country many partners acting together (such as conserv- ownership. Clear delineation and coordination ing global environmental commons). In addition of the roles of national and regional institutions to evaluating the outcomes of their individual are also crucial to the implementation of activities, the essential evaluation challenge is to program activities and the sustainability of assess the value added by the partnership outcomes. Countries also need to plan well in compared with more traditional modes of advance to absorb the costs of sustaining the development assistance (both multilateral and outcomes of regional projects after external bilateral) and with other existing, sometimes support ends. competing, global initiatives. To what extent do the benefits of the partnership outweigh the What has generally worked best is reliance on additional transactions and other costs associ- national institutions for execution and implemen- ated with operating in a partnership? tation of program interventions at the country level and on regional institutions for supportive Evaluations of global partnerships are the respon- services such as coordination, data gathering and sibility of the governing bodies of these programs, sharing, technical assistance, dispute resolution, which typically commission external evaluations and M&E. This delineation characterizes the every five years or so. In cases where the Bank sits satisfactory implementation of the first phase of on the governing body or provides a financial the Lake Victoria Environment Management contribution, the Bank's representative to that Program, in which national institutions conducted body and the task team leader also have oversight scientific studies and pilot activities, while a small responsibilities--analogous to Bank supervision regional secretariat coordinated information of an investment project. These responsibilities sharing and interactions with donors. include, among other things, advocating for sound monitoring and periodic independent All of the regional environmental projects evaluations. IEG began conducting Global reviewed by IEG supported some form of Program Reviews based on these evaluations in scientific research, but project mechanisms 2006, and has since conducted 14, including varied greatly in the way they attempted to reviews of three environmental programs--the connect the research with enhanced learning Critical Ecosystem Partnership Fund, Global Water and decision-making capacity. For example, the Partnership, and Global Invasive Species Program. Lake Victoria Environment Management Pro- IEG has found that: gram, the Central Asia Biodiversity Project, and the Integrated Silvopastoral Approaches to · The Critical Ecosystem Partnership Fund Ecosystem Management Project (box 5.3) were (CEPF) made strong progress in its first phase. 68 A C H I E V I N G E N V I R O N M E N TA L R E S U LT S F R O M T H E P R O J E C T T O T H E G L O B A L L E V E L As of March 2007, the CEPF had contributed to mented and difficult, and the GWP's National and regional the creation or expansion of 9.4 million support for countries and regions capacities need to be hectares of protected areas in 15 countries was judged to be at the expense of matched to effectively and had helped strengthen protected area leadership in global policy advocacy implement regional management across more than 21 million (IEG forthcoming a). project activities. hectares in 16 countries. However, as with · The Global Invasive Species Pro- Bank-financed projects, the CEPF's investments gram has been successful in collating infor- in production landscapes lack aggregate evi- mation and raising awareness about invasive dence of the effectiveness of introducing al- alien species (IAS), but has had few linkages to ternative livelihoods in protected area the World Bank's operations. Further research management (IEG 2007b). and analysis is needed to identify and prioritize · The Global Water Partnership (GWP) has a such IAS threats and related cost- niche in facilitating multistakeholder dialogue effective mitigation or eradication Global environment on integrated water resource management strategies. The Bank could consider programs need to add (IWRM). However, it has been slow to move di- supporting research on what the value to traditional or alogue beyond water-focused organizations, literature has identified as the great- alternative modes of which runs counter to the comprehensive and est needs. One of these is the need development assistance. integrated approach the GWP advocates. While to improve the basis on which IAS there have been some notable advances in control strategies are evaluated and on which moving countries toward more integrated man- the potential impacts of species introductions agement of their water resources, attribution are valued (since the valuation of costs and ben- to the GWP is difficult to verify. Also, it has efits of control options is still quite rudimen- proven difficult to influence country develop- tary) (IEG 2009b). ment dialogue, particularly for Poverty Re- duction Strategy Papers, to increase budgetary IEG has also found that Bank oversight of global allocation to GWP activities, and to facilitate programs has generally been very weak because grassroots implementation of GWP principles of inadequate budget allocations and through country partnerships. Engagement in poorly defined expectations of the Bank oversight of global transboundary water resource issues--the roles, responsibilities, and account- programs has generally GWP's comparative advantage--was frag- abilities of Bank staff overseeing been very weak. Box 5.3: What Does a Highly Satisfactory Regional Project Look Like? The Case of the Pilot Integrated Silvopastoral Approaches to Ecosystem Management Project in Latin America The Integrated Silvopastoral Approaches to Ecosystem Man- countries), improved water quality (reduction of biological oxy- agement Project aimed to demonstrate and measure the ef- gen demand), soil retention, soil productivity, reduced inor- fects of the introduction of payment for environmental services. ganic fertilizer through nitrogen-fixing legumes, diversification The project has been successful in demonstrating and measur- of systems with risk-reduction potential, and scenic beauty ing the effects of offering payment incentives to farmers to enhancement. adopt integrated silvopastoral farming systems on degraded IEG rated the project highly satisfactory because the highly fo- pasturelands in Colombia, Costa Rica, and Nicaragua. cused, very carefully monitored pilot project seems to have yielded The project met its target, with some 12,000 hectares show- a high payoff both for profitability (and thus growth) and for the en- ing improved biodiversity and carbon sequestration. Measured vironment. Moreover, there was a strong focus on measurement benefits included carbon sequestration (20,000 tons), increased so that evidence rapidly accumulated and lessons were learned biodiversity (more birds, butterflies, and mollusks in all three quickly within the project period. Source: PAD, ICR, and IEG ICR Review of the Integrated Silvopastoral Approaches to Ecosystem Management Project. 69 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 individual programs. This is both a Bank-wide incremental cost has aimed to ensure that the and a sector issue. Except for a guidance note on funds provided by the GEF are additional, that is, conflicts of interest and the reporting require- that they do not substitute for existing develop- ments associated with Development Grant ment funds. Facility application forms and progress reports, there are no Bank-wide guidelines for oversight Nine of the 11 biodiversity projects reviewed by of global programs, despite Internal Auditing IEG were fully financed by GEF and conducted Department and IEG recommendations for an incremental cost assessment. In addition, the dealing with this issue going back to 2004. And, two biodiversity projects that blended GEF and except for identifying and catalyzing new IBRD/IDA finance conducted some type of cost- environmental partnerships in the context of the benefit analysis.3 The lack of a financial or growing interest in climate change issues, the economic analysis in biodiversity projects may Environment Sector Board has not generally pose a more severe problem in the future due to followed through on its commitments in the a recent GEF decision to drop the requirement 2001 Environment Strategy to effectively monitor for incremental cost assessment altogether its engagement in global partnerships.2 starting in fiscal 2008 (the result of a recommen- dation submitted by an independent evaluation Economic Analysis of Environmental of the GEF's incremental cost assessment Projects Is Challenging requirements).4 The economic analysis of projects can Environmental projects help form the basis for resource There are clearly untapped opportunities to do not calculate allocation and the rationale for public incorporate analysis of the costs and benefits of economic rates of return, support. However, this practice has environmental outcomes in projects outside the although some cost- been in sharp decline in the World environment sector. For example, projects that benefit analysis is Bank (chapter 2). The environmental mainstream support for environmental health and available. sector has followed this trend. pollution management can yield important, quantifiable health benefits or have unintended IEG reviewed the economic analyses conducted negative impacts. In Ecuador, for example, the for the complete population of 51 ENRM projects Bank applied a willingness-to-pay approach to approved since 2000 and closed in 2007 and calculating the costs of adding a water quality 2008. More than half of the projects did not component (ex ante) and calculated the estimated conduct any economic or financial rate of return cost savings associated with a reduction in health analysis (table 5.1). Of the projects that did expenditures related to water-borne diseases. conduct economic or financial rates of return analyses, none attempted to calculate the The costs and benefits of the environmental monetary value of their environmental impacts outcomes of irrigation and drainage schemes can (positive or negative). also be measured. Bank-financed projects attempted to mainstream water resource Biodiversity conservation projects tend not to management into irrigation and drainage conduct economic rate of return or financial rate projects in Azerbaijan, Mali, Pakistan, and of return analyses because most World Bank Tunisia--activities that would have added costs biodiversity projects are partly or fully financed to irrigation and drainage rehabilitation by GEF, and such projects have required the use schemes--yet only the Tunisia project attempted of a different analytical tool--an incremental cost to qualify the potential environmental benefits of assessment. Incremental cost--the additional one intervention, treated wastewater use. In cost of transforming a project with local or Mali, where the Bank supported environmental national benefits into one that also generates and social micro-investments in irrigation, road global environmental benefits--is a founding rehabilitation, and rural water supply, no principle of the GEF. The valuation of the economic analysis was conducted of related 70 A C H I E V I N G E N V I R O N M E N TA L R E S U LT S F R O M T H E P R O J E C T T O T H E G L O B A L L E V E L Table 5.1: Relationship between Economic Rate of Return/Financial Rate of Return and ENRM Projects by Subtheme Economic rate of return/financial rate of return ENRM subtheme(s) None Both Appraisal only Completion only Total Biodiversity 1 1 Biodiversity and EPI 9 9 Climate change 1 3 4 Climate change and EPI 1 1 Climate change and EHPM 2 2 EPI 2 1 3 EPI and NRM 1 1 EPI and EHPM 3 3 6 Land administration 3 3 Land administration and NRM 2 2 NRM 2 1 1 4 EHPM 1 7 8 ERM 3 4 7 Total 27 21 2 1 51 Source: IEG Review of PADs and ICRs of 51 ENRM projects approved since 2000 and closed in 2007 and 2008. See appendix F. Note: EPI = environmental policies and institutions; EHPM = environmental health and pollution management; NRM = natural resources management; ERM = environmental resources management. investments in integrated pest management and ects, including revising its preparation Few ENRM projects assess tree planting, even though such analysis should guidelines for ICRs to require a more or qualify their potential have been both feasible and desirable. The systematic review of environmental to produce positive Bank's Development Economics Group, for dimensions and results. Those assess- environmental example, has conducted quantitative research on ments should account for externali- externalities, including the costs of supporting farmer field schools in ties, including the net impact on health benefits or other projects with integrated pest management greenhouse gas emissions, and should improved water quality. activities. seek to measure indirect and policy- related impacts, which could be much larger than Major Lessons those related to projects. Beyond project-level monitoring, the Bank should also invest in Critically Examine Success in Achieving building stronger data and indicators at the Environmental Results through Mainstreaming country, regional, and global levels, including, for as the Bank Develops Its New Environment example, comparative data on energy prices, Strategy collection rates, subsidies, policies, and perform- As the Bank develops its new environment ance data at the national and subnational levels. strategy, it needs to focus carefully on the system- atic tracking of environmental components of Foster More Effective Partnerships at All projects across sectors and on better monitoring Levels to Enhance Environmental Management and reporting of environmental outcomes at the The environmental agenda is so large that project and country levels. The old adage of concerted action among many players will be "what gets measured gets done" is all too often essential. The World Bank, bilateral donors, and true. For the Bank, this means more rigorous other development partners need to ensure economic and environmental assessment, adequate financing (loan, concessional, and grant) monitoring, evaluation, and reporting for proj- for country-level investments that have regional 71 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 and global environmental benefits. They also need Bank-supported activities at the country level. At to continue to support partnerships that are that level, donors also need to work closely with generating and disseminating knowledge about one another and with environmental NGOs good policies and practices in environmental representing civil society, and both groups need to management. However, the benefits of increased ensure that governments are fully informed and knowledge will not be fully realized unless the empowered to participate actively in global knowledge is disseminated and incorporated into partnerships. 72 Appendixes Lumber floating on water, Indonesia. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. APPENDIX A: SUPPORTING DATA FOR 2008 PROJECT PERFORMANCE ANALYSIS Table A.1: Project Performance Rebounded in 2008 Fiscal year 2006 (percent) 2007 (percent) 2008 (percent) Five-year average (percent) Overall 82.6 75.8 81.0 79.4 By sector Transport 100 95 95 94 a Poverty reduction 100 71 88 Water 75 90 100 88 Agriculture and rural development 85 83 94 87 a Urban development 87 78 86 a Economic policy 81 88 85 Social protection 84 100 71 83 Education 77 76 50 80 Environment 84 80 89 79 Energy and mining 73 79 67 77 a Social development 67 100 74 Financial/private sector development 81 63 67 70 Public sector governance 68 60 75 64 Health, nutrition, and population 80 41 55 59 By Region East Asia and Pacific 92 86 88 85 Europe and Central Asia 79 87 84 85 Latin America and the Caribbean 93 73 81 83 South Asia 86 54 88 81 b Middle East and North Africa 71 88 56 74 Africa 66 60 77 68 Source: World Bank data. a. Subcategories with fewer than five observations were excluded. b. Only nine projects exiting the portfolio in 2008 from the Middle East and North Africa have thus far been evaluated. 75 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table A.2: Regression Estimates of Average Differences in Performance by Sector, Region, and Lending Instrument, 1993­2008 Element Estimated effecta T-ratio Year (time trend) 0.02 4.45 a Transport -- Agriculture and rural development 0.43 5.75 Economic policy 0.61 5.34 Education 0.22 2.53 Energy and mining 0.53 6.28 Environment 0.38 3.46 Finance and private sector development 0.60 6.42 Health, nutrition, and population 0.56 5.93 Public sector governance 0.70 7.27 Social protection 0.24 2.31 Urban development 0.34 3.25 Water 0.49 4.64 a East Asia and the Pacific -- Sub-Saharan Africa 0.63 10.10 Europe and Central Asia 0.06 0.86 Latin America and Caribbean 0.08 1.25 Middle East and North Africa 0.42 5.05 South Asia 0.27 3.48 Development Policy Loan 0.37 5.08 Source: World Bank data and author's calculations. Note: N = 3,936; R2 = 8 percent; time period: 1993­2008 a. The dependent variable is the rating of project performance, measured on the six-point scale. The sector and Region effects are impacts relative to the highest-performing sector and Region: transport and East Asia and Pacific. Hence performance ratings for agriculture were 0.43 points lower, on average, than for transport over this period. Performance ratings for de- velopment policy operations were 0.37 points higher than for investment operations. The estimated time trend indicates that performance ratings increased, on average, 0.02 points per year. . 76 APPENDIX B: CRITERIA FOR IEG EVALUATIONS IEG's ICR Reviews assess Bank-supported projects Shortcomings in the achievement of objectives on a number of dimensions. The primary rating is may involve either the number of objectives on an operation's outcome, or the extent to which achieved or the extent to which they were its major relevant objectives were, or are expected achieved. Shortcomings in efficiency may involve to be, achieved efficiently. These outcomes are the extent to which an operation has failed to rated on a six-point scale, as shown in the table achieve a return higher than the opportunity cost below. of capital and is not the least-cost alternative. Shortcomings in relevance may involve the extent For evaluation purposes, an operation's objectives to which an operation's objectives, design, or include the official project development objectives implementation are inconsistent with a country's contained in documents approved by the Bank's current development priorities and the Bank's Board, as well as key associated outcome targets. goals. Outcome: the extent to which the operation's major relevant objectives were, or are expected to be, achieved efficiently There were ______________ in the operation's achievement of its objectives, in its efficiency, or in its relevance. Rating . . . no shortcomings . . . Highly satisfactory . . . minor shortcomings . . . Satisfactory . . . moderate shortcomings . . . Moderately satisfactory . . . significant shortcomings . . . Moderately unsatisfactory . . . major shortcomings . . . Unsatisfactory . . . severe shortcomings . . . Highly unsatisfactory 77 APPENDIX C: SUMMARY OF MANAGEMENT ACTION RECORD 2009 IEG influences the Bank's effectiveness through Level of Adoption of IEG recommendations to management as part of Recommendations sector, thematic, and corporate evaluations, as In recent years, there has been a downward shift well as through findings of CAEs. Bank manage- in the distribution of adoption ratings among the ment is accountable to the Board for follow-up. top two categories, substantial and high. In the One of the intermediate outcomes for IEG is the 2009 MAR, as shown in figure C.1, IEG rated extent to which management implements IEG adoption substantial or better for 41 percent of recommendations and findings in policy advice, recommendations, compared with 42 percent in program design, and project design. the 2008 MAR and 60 percent in the 2007 MAR. The lowest rating, negligible, has remained The Management Action Record (MAR) allows relatively stable, dipping from 15 percent in 2007 IEG to monitor the adoption1 and implementa- to 5 percent in 2008 to 13 percent in 2009. tion2 of individual recommendations from sector, thematic, and corporate evaluations. It Of the 36 recommendations that have been includes the ratings of both management and carried over from previous years, IEG rated IEG on these two indicators, updated annually. management's adoption high or substantial for 53 percent (18) of these recommendations and The MAR includes all recommendations issued medium for 44 percent (15). The adoption of in the past three calendar years (2006­08). one recommendation was rated negligible, and Recommendations from earlier years are submit- for two recommendations there was a difference ted to the Bank's Board in another document, of opinion between IEG and management. the Implementation Report, and archived if there are no objections. The MAR for 2009 tracks It is unreasonable to expect the immediate management's progress on 61 recommenda- adoption of recommendations in the first year; tions, including 25 new recommendations from however, ratings for these new recommenda- the seven IEG studies (excluding CAEs) tions are substantially lower than was reported in presented to the Board in calendar year 20083 the 2008 MAR, as shown in figure C.2. Of the 25 and 36 recommendations carried forward from new recommendations from calendar 2008, the calendar 2006 and 2007.4 adoption of 20 percent (4) was high or substan- tial, of 50 percent (10) was medium, and of 30 There has been a decline in the level of Bank percent (6) was negligible.6 agreement with IEG's recommendations. Of the 61 recommendations in the 2009 MAR, 88 Among the reasons for the increase in negligible percent (54 recommendations) have been ratings was the limited information management accepted by Bank management.5 This is lower provided that linked concrete actions to the than the rate of acceptance in earlier years (96 specific recommendations and the absence of percent in MAR 2008 and 95 percent in MAR objective indicators to verify the progress of 2007). adoption. The 2009 MAR ratings for new 79 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Figure C.1: The Level of Adoption of IEG Recommendations Is Lower than in Previous Years 100 5 15 13 Share of rating category (%) 80 25 53 46 60 40 44 31 26 20 16 11 15 0 2007 2008 2009 N = 55 N = 55 N = 54 MAR High Substantial Medium Negligible Source: World Bank data. recommendations highlight the need to sharpen on the rating on level of adoption for 67 percent the focus on effective recommendations and their of recommendations (36 out of 54).7 In particu- speedy adoption. lar, there were disagreements on high and substantial adoption ratings, which IEG assigned Disagreement between IEG and to 41 percent of recommendations, compared Management on Level of Adoption with 54 percent by management. Where there In the 2009 MAR, IEG and management agreed was disagreement on the level of adoption, IEG's ratings were lower than management's for all but Figure C.2: The Level of Adoption of one recommendation. New IEG Recommendations Dropped in 2009 Table C.1 shows the distribution of adoption ratings by management and IEG, which was 100 similar to that in previous years (67 percent in 6 30 2009 compared with 65 percent in 2008). The Share of recommendations (%) 80 disconnect between IEG and management was 57 13 percent in 2009, compared with 18 percent in 60 2008 and 28 percent in 2007. IEG and manage- ment agreed on the level of adoption for 65 50 40 percent (22) of recommendations carried over from previous years, which compares favorably 26 with that reported in the 2008 MAR (53 percent). 20 10 11 10 0 Improving the MAR New recommendations New recommendations IEG is currently undertaking an initiative to 2008 2009 improve the MAR to create a more timely and High Substantial Medium Negligible effective product for assessing the implementa- Source: World Bank data. tion of IEG's recommendations. Key elements under consideration include measures to 80 A P P E N D I X C : S U M M A RY O F M A N AG E M E N T AC T I O N R E C O R D 2 0 0 9 improve the quality of IEG recommendations and bining the MAR and Implementation Report into to facilitate stronger engagement of the Board in a single report that offers a summary assessment the process. Further elements are aimed at of management actions taken over the past year. simplifying the reporting and monitoring process Under the forthcoming update to IEG's disclo- and moving toward a more continuous engage- sure policy, the combined document would be ment and support in the implementation of disclosed to the general public from fiscal 2010 recommendations. IEG is also considering com- onward. Table C.1: The Bank Tended to Rate the Level of Adoption of Recommendations Higher than IEG IEG Ratings Sum of Difference management Management ratings High Substantial Medium Negligible of opinion ratings High 7 5 1 13 Substantial 1 9 5 1 16 Medium 19 5 24 Negligible 1 1 Difference of opinion 7 7 Sum of IEG ratings 8 14 25 7 7 61 Source: IEG data. Note: Fields in lighter blue show agreement between management and IEG. The last column shows the sum of ratings in each category by management. The last row shows the sum of ratings by IEG. 81 APPENDIX D: THE BANK'S ENVIRONMENT AND NATURAL RESOURCE MANAGEMENT (ENRM) PORTFOLIO Table D.1: Share of ENRM Operations in Total Bank Operations, Fiscal Years 1990­2008 Number of operationsa Commitments (US$ millions)b Approval ENRM All Bank Percent ENRM All Bank Percent year operations operations of total operations operations of total 1990 24 222 11 2,153 20,702 10 1991 21 230 9 2,119 22,630 9 1992 36 228 16 2,867 21,718 13 1993 43 305 14 3,405 23,600 14 1994 52 324 16 3,712 20,992 18 1995 65 351 18 3,268 23,097 14 1996 50 369 14 2,618 21,701 12 1997 47 361 13 2,575 19,603 13 1998 47 401 12 2,732 28,871 9 1999 46 405 11 1,614 29,684 5 2000 47 334 14 2,004 15,938 13 2001 52 349 15 1,527 18,159 8 2002 31 333 9 1,065 21,283 5 2003 44 349 13 1,240 18,997 7 2004 51 371 14 1,521 20,720 7 2005 76 418 18 2,840 23,136 12 2006 69 418 17 2,602 25,635 10 2007 66 467 14 2,309 25,857 9 2008 70 451 15 2,989 25,723 12 Total 937 6,686 14 45,160 428,046 11 Source: World Bank data. a. Weighted by the share of each project assigned to the environment themes. b. Share of total commitments assigned to the environment themes. 83 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table D.2: ENRM Operations by Type of Instrument, Fiscal Years 1990­2008 Number of operationsa Commitments (US$ millions)b Approval Investment Development Investment Development year operations policy operations Total operations policy operations Total 1990 58 3 61 2,128 25 2,153 1991 54 2 56 1,955 164 2,119 1992 81 3 84 2,855 12 2,867 1993 83 4 87 3,314 90 3,405 1994 95 4 99 3,657 55 3,712 1995 101 5 106 3,225 42 3,268 1996 95 3 98 2,591 28 2,618 1997 97 3 100 2,471 104 2,575 1998 110 3 113 2,561 171 2,732 1999 88 8 96 1,250 362 1,612 2000 90 4 94 1,937 67 2,004 2001 84 4 88 1,505 20 1,525 2002 77 3 80 879 186 1,065 2003 86 6 92 1,064 176 1,240 2004 113 5 118 1,463 59 1,521 2005 140 8 148 2,031 809 2,840 2006 135 9 144 2,328 274 2,602 2007 140 7 147 1,977 332 2,309 2008 133 10 143 2,265 724 2,989 Total 1,860 94 1,954 41,456 3,700 45,156 Source: World Bank data. a. Number of operations with at last one environment theme. b. Share of total commitments assigned to the environment themes. 84 A P P E N D I X D : T H E B A N K ' S E N V I R O N M E N T A N D N AT U R A L R E S O U R C E M A N A G E M E N T P O R T F O L I O Table D.3: ENRM Operations by Source of Financing, Fiscal Years 1995­2001 and 2002­08 Approval years Fiscal 1995­01 Fiscal 2002­08 Number of Commitments Number of Commitments Source of financing operations (US$ millions) operations (US$ millions) IBRD/IDA projects Investment projects 486 14,269 434 9,325 Development policy lending 30 793 46 2,550 Guarantees 4 174 3 45 Subtotal 520 15,236 483 11,920 Bank-implemented GEF projects World Bank/GEF blendsa (31) 169 (59) 312 Freestanding GEF projects 67 451 128 813 Medium-size GEF projectsb 41 23 53 34 Subtotal 139 644 240 1,159 Selected trust funds Montreal Protocol 18 400 3 48 Carbon Offsets 1 0 101 1,173 Other trust funds 17 55 45 264 Subtotal 36 455 149 1,483 Totalc 664 16,334 813 14,562 Annual average 95 2,333 116 2,080 Source: World Bank data. a. These commitment amounts include the GEF commitments to the 31 blended projects in 1995­2001 and 59 blended projects in 2002­08. All but one of these projects (Cameroon in fiscal 2006) were investment operations. b. These projects are for less than $1.0 million. c. These totals count World Bank/GEF blends as one project. 85 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table D.4: ENRM Global/Regional Operations by Source of Financing, Fiscal Years 1995­2001 and 2001­08 Approval years Fiscal 1995­01 Fiscal 2002­08 Number of Commitments Number of Commitments Source of financing operations (US$ millions) operations (US$ millions) IBRD/IDA projects IBRD investment projects 4 102.3 6 115.4 IDA investment projects 7 67.1 13 309.7 IDA guarantees 0 0.0 1 16.5 Subtotal 11 169.4 20 441.7 Bank-implemented GEF projects World Bank/GEF blendsa (4) 39.2 (3) 21.4 Freestanding GEF projects 17 94.6 40 241.2 Medium-size GEF projectsb 3 2.0 8 5.9 Subtotal 24 135.7 51 268.6 Selected trust funds Rainforest Trust Fund 8 6.6 11 16.2 Carbon Offsets 0 0.0 2 3.0 Other trust funds 1 4.3 4 48.1 Subtotal 9 10.9 17 67.4 Totalc 40 316.0 85 777.6 Annual average 6 45.1 12 111.1 Source: World Bank data. a. These commitment amounts include the GEF commitments to the four blended projects in 1995­2001 and the one blended project in 2002­08. b. These projects are for less than $1 million. c. These totals count World Bank/GEF blends as one project. 86 A P P E N D I X D : T H E B A N K ' S E N V I R O N M E N T A N D N AT U R A L R E S O U R C E M A N A G E M E N T P O R T F O L I O Table D.5: ENRM Operations by Source of Financing and Theme, Fiscal Years 1995­2001 and 2002­08 (US$ millions commitments) IBRD/IDA IBRD/IDA development investment operations policy operations GEF Other trust funds Total Theme 1995­2001 2002­08 1995­-2001 2002­-08 1995­2001 2002­08 1995­2001 2002­08 1995­2001 2002­08 Pollution management and environmental health 4,902 2,942 175 277 144 145 322 142 5,544 3,506 Water resources management 3,214 2,643 75 264 33 71 8 129 3,329 3,108 Climate change 1,431 1,150 25 383 81 312 115 1,156 1,652 3,001 Environmental policies and institutions 2,388 946 261 1,024 203 234 182 24 3,034 2,228 Land administration and management 1,606 992 217 412 24 82 1 35 1,848 1,521 Biodiversity 282 282 0 130 145 255 0 6 427 673 Other environment and natural resources management 446 362 39 59 14 68 1 41 500 530 Total 14,269 9,317 793 2,550 644 1,167 628 1,533 16,334 14,566 Source: World Bank data. 87 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table D.6: ENRM Operations by Source of Financing and Sector Board, Fiscal Years 1995­2001 and 2002­08 (US$ millions commitments) IBRD/IDA IBRD/IDA development investment operations policy operations Sector Board 1995­2001 2002­08 1995­-2001 2002­-08 Agriculture and Rural Development 3,822 2,700 358 173 Environment 1,596 744 25 1,657 Energy and Mining 3,299 1,403 219 118 Water 2,614 2,423 0 49 Transport 1,624 758 29 4 Urban Development 1,018 1,158 0 80 Other sector boards 296 138 163 468 Total 14,269 9,325 793 2,550 Percentage of total Agriculture and Rural Development 27 29 45 7 Environment 11 8 3 65 Energy and Mining 23 15 28 5 Water 18 26 0 2 Transport 11 8 4 0 Urban Development 7 12 0 3 Other sector boards 2 1 21 18 Total 100 100 100 100 Source: World Bank data. 88 A P P E N D I X D : T H E B A N K ' S E N V I R O N M E N T A N D N AT U R A L R E S O U R C E M A N A G E M E N T P O R T F O L I O GEF Other trust funds Total 1995­2001 2002­08 1995­2001 2002­08 1995­2001 2002­08 99 151 2 66 4,347 3,090 394 578 409 1,108 2,424 4,086 131 316 15 150 3,772 2,033 7 60 24 101 2,646 2,634 1 38 0 2 1,654 801 11 6 3 14 1,031 1,258 1 10 1 42 461 660 644 1,159 455 1,484 16,334 14,562 15 13 0 4 27 21 61 50 90 75 15 28 20 27 3 10 23 14 1 5 5 7 16 18 0 3 0 0 10 6 2 0 1 1 6 9 0 1 0 3 3 5 100 100 100 100 100 100 89 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table D.7: Share of ENRM Projects and Commitments in the Total Number of Projects and Volume of Commitments to Each Sector Board, 1995­2008 ENRM as percent of total Sector Board 1995­2001 2002­08 Total Number of projects Environment 95 99 97 Water 85 68 75 Energy and Mining 68 62 64 Agriculture and Rural Development 56 57 56 Urban Development 51 35 41 Transport 24 19 21 Other sector boards 3 5 4 Total 32 34 33 Commitments (US$ millions) Environment 62 85 75 Water 41 30 34 Energy and Mining 22 15 19 Agriculture and Rural Development 25 19 22 Urban Development 18 11 13 Transport 8 3 6 Other sector boards 1 1 1 Total 10 9 10 Source: World Bank data. 90 A P P E N D I X D : T H E B A N K ' S E N V I R O N M E N T A N D N AT U R A L R E S O U R C E M A N A G E M E N T P O R T F O L I O Number of ENRM projects Total number of projects 1995­2001 2002­08 Total 1995­2001 2002­08 Total 179 249 428 189 252 441 87 99 186 102 146 248 117 164 281 172 264 436 181 202 383 323 357 680 47 55 102 92 157 249 47 39 86 193 208 401 37 63 100 1,078 1,177 2,255 695 871 1,566 2,149 2,561 4,710 2,424 4,086 6,509 3,895 4,825 8,720 2,646 2,634 5,280 6,520 8,842 15,361 3,772 2,033 5,805 16,950 13,642 30,592 4,347 3,090 7,437 17,247 16,633 33,880 1,031 1,258 2,290 5,647 11,731 17,378 1,654 801 2,455 19,879 23,375 43,254 461 660 1,120 86,915 82,338 169,253 16,334 14,562 30,897 157,053 161,386 318,439 91 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table D.8: ENRM Operations by Sector Board, Fiscal Years 1995­2001 and 2002­08 Number of operations Commitments (US$ millions) Sector Board 1995­2001 2002­08 Total 1995­2001 2002­08 Total Agriculture and Rural Development 181 202 383 4,347 3,090 7,437 Environment 179 249 428 2,424 4,086 6,509 Energy and Mining 117 164 281 3,772 2,033 5,805 Water 87 99 186 2,646 2,634 5,280 Transport 47 39 86 1,654 801 2,455 Urban Development 47 55 102 1,031 1,258 2,290 Public Sector Governance 6 14 20 85 234 319 Economic Policy 13 13 26 122 155 277 Financial and Private Sector Development 6 13 19 91 83 174 Education 3 3 6 108 17 126 Poverty Reduction 0 7 7 0 85 85 Social Protection 6 4 10 43 32 76 Social Development 2 8 10 8 52 60 Health, Nutrition, and Population 1 1 2 3 1 4 Total 695 871 1,566 16,334 14,562 30,897 Percent of total Agriculture and Rural Development 26 23 24 27 21 24 Environment 26 29 27 15 28 21 Energy and Mining 17 19 18 23 14 19 Water 13 11 12 16 18 17 Transport 7 4 5 10 6 8 Urban Development 7 6 7 6 9 7 Public Sector Governance 1 2 1 1 2 1 Economic Policy 2 1 2 1 1 1 Financial and Private Sector Development 1 1 1 1 1 1 Education 0 0 0 1 0 0 Poverty Reduction 0 1 0 0 1 0 Social Protection 1 0 1 0 0 0 Social Development 0 1 1 0 0 0 Health, Nutrition, and Population 0 0 0 0 0 0 Total 100 100 100 100 100 100 Source: World Bank data. 92 A P P E N D I X D : T H E B A N K ' S E N V I R O N M E N T A N D N AT U R A L R E S O U R C E M A N A G E M E N T P O R T F O L I O Table D.9: ENRM Operations by Region, Fiscal Years 1995­2001 and 2002­08 Number of operations Commitments (US$ millions) Region 1995­2001 2002­08 Total 1995­2001 2002­08 Total Africa 141 216 357 1,893 1,688 3,582 East Asia and the Pacific 126 160 286 5,716 5,097 10,814 Europe and Central Asia 153 168 321 2,559 2,282 4,841 Latin America and the Caribbean 153 202 355 2,821 2,516 5,336 Middle East and North Africa 55 55 110 931 831 1,762 South Asia 66 68 134 2,394 2,134 4,528 Global 1 2 3 20 14 34 Total 695 871 1,566 16,314 14,548 30,862 Percent of total Africa 20 25 23 12 12 12 East Asia and the Pacific 18 18 18 35 35 35 Europe and Central Asia 22 19 20 16 16 16 Latin America and the Caribbean 22 23 23 17 17 17 Middle East and North Africa 8 6 7 6 6 6 South Asia 9 8 9 15 15 15 Global 0 0 0 0 0 0 Total 100 100 100 100 100 100 Source: World Bank data. 93 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table D.10: Incidence of ENRM Themes by Sector Board, Fiscal Years 1995­2008 Agriculture & Rural Energy & Urban Theme Environment Development Mining Water Development Transport Other Total Number of projectsa Environmental policies and institutions 266 102 62 39 14 22 28 533 Biodiversity 226 88 1 6 2 2 10 335 Pollution management and environmental health 148 42 103 132 54 57 8 544 Land administration and management 94 141 8 6 26 2 32 309 Water resources management 87 149 15 103 25 7 11 397 Climate change 93 21 191 8 10 8 9 340 Other environment and natural resources management 76 75 15 1 1 3 19 190 Total 990 618 395 295 132 101 117 2,648 Commitments (US$ millions) Environmental policies and institutions 1,956 977 845 482 155 509 339 5,263 Biodiversity 599 399 1 29 7 19 47 1,101 Pollution management and environmental health 1,339 413 1,968 2,664 1,004 1,587 75 9,050 Land administration and management 430 1,855 132 91 598 3 261 3,369 Water resources management 611 3,015 167 1,899 439 186 119 6,437 Climate change 1,439 104 2,641 115 73 135 145 4,653 Other environment and natural resources management 142 674 51 0 13 16 135 1,031 Total 6,516 7,437 5,805 5,280 2,290 2,455 1,122 30,905 Percent of total commitments Environmental policies and institutions 30 13 15 9 7 21 30 17 Biodiversity 9 5 0 1 0 1 4 4 Pollution management and environmental health 21 6 34 50 44 65 7 29 Land administration and management 7 25 2 2 26 0 23 11 Water resources management 9 41 3 36 19 8 11 21 Climate change 22 1 45 2 3 6 13 15 Other environment and natural resources management 2 9 1 0 1 1 12 3 Total 100 100 100 100 100 100 100 100 Source: World Bank data. a. Number of projects in which each theme occurs. The differences among sector boards are significant at the 1 percent level. 94 A P P E N D I X D : T H E B A N K ' S E N V I R O N M E N T A N D N AT U R A L R E S O U R C E M A N A G E M E N T P O R T F O L I O Table D.11: Incidence of ENRM Themes by Region, Fiscal Years 1995­2008 Europe Latin Middle East East Asia & Central America & North South Theme Africa & Pacific Asia & Caribbean Africa Asia Global Total Number of projectsa Environmental policies and institutions 138 82 95 146 33 38 1 533 Biodiversity 109 42 41 118 10 13 2 335 Pollution management and environmental health 77 111 164 89 59 44 544 Land administration and management 74 49 67 95 9 15 309 Water resources management 90 67 80 65 51 43 1 397 Climate change 55 91 76 75 15 28 340 Other environment and natural resources management 69 29 27 53 6 6 190 Total 612 471 550 641 183 187 4 2,648 Commitments (US$ millions) Environmental policies and institutions 713 1,433 604 1,483 196 824 10 5,263 Biodiversity 264 176 85 437 14 101 23 1,101 Pollution management and environmental health 755 3,517 1,673 1,108 714 1,283 9,050 Land administration and management 406 1,017 839 765 74 268 3,369 Water resources management 795 1,912 867 848 625 1,389 1 6,437 Climate change 350 2,440 671 537 110 545 4,653 Other environment and natural resources management 301 318 101 160 34 116 1,031 Total 3,583 10,814 4,841 5,339 1,767 4,528 34 30,905 Percent of total commitments Environmental policies and institutions 20 13 12 28 11 18 29 17 Biodiversity 7 2 2 8 1 2 69 4 Pollution management and environmental health 21 33 35 21 40 28 0 29 Land administration and management 11 9 17 14 4 6 0 11 Water resources management 22 18 18 16 35 31 2 21 Climate change 10 23 14 10 6 12 0 15 Other environment and natural resources management 8 3 2 3 2 3 0 3 Total 100 100 100 100 100 100 100 100 Source: World Bank data. a. Number of projects in which each theme occurs. The differences among sector boards are significant at the 1 percent level. 95 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table D.12: ENRM Analytical and Advisory Services by Type of Activity and Source of Funding, Fiscal Years 2002­08 `2002 2003 2004 2005 2006 2007 2008 Total a Number of activities ESW 33 56 72 66 61 64 62 414 TA 43 42 41 68 24 56 65 339 Grand total 76 98 113 134 85 120 127 753 Expenditures (US$ thousands)b ESW Bank administrative budget 1,746 2,853 3,611 4,796 7,396 5,874 5,620 31,896 Bank-administered trust fund 1,501 734 2,029 3,818 5,838 3,173 5,375 22,469 Total 3,247 3,588 5,640 8,614 13,235 9,046 10,995 54,365 TA Bank administrative budget 2,910 2,523 2,409 3,689 1,082 4,325 3,445 20,382 Bank-administered trust fund 1,424 3,345 6,398 7,142 3,751 5,370 3,884 31,314 Total 4,334 5,868 8,807 10,831 4,833 9,695 7,328 51,696 AAA total Bank administrative budget 4,656 5,376 6,020 8,485 8,478 10,199 9,065 52,278 Bank-administered trust fund 2,925 4,080 8,428 10,961 9,589 8,543 9,259 53,783 Total 7,581 9,456 14,447 19,445 18,067 18,741 18,324 106,062 Source: World Bank data. a. Activities with at least one environmental theme. b. Share of total expenditures assigned to the environment themes. Table D.13: ENRM Analytical and Advisory Services by Country Level versus Global/Regional Level, Fiscal Years 2002­08 Expenditures (US$ thousands) Percentb Bank Bank- Bank Bank- Number of administrative administered administrative administered Type of activity activitiesa budget trust fund Total budget trust fund Total Country-level activities ESW 312 25,372 14,668 40,040 24 14 38 TA 259 12,480 12,511 24,992 12 12 24 Subtotal 571 37,852 27,179 65,032 36 26 61 Global/regional activities ESW 102 6,524 7,801 14,325 6 7 14 TA 80 7,902 18,803 26,705 7 18 25 Subtotal 182 14,426 26,604 41,030 14 25 39 All activities ESW 414 31,896 22,469 54,365 30 21 51 TA 339 20,382 31,314 51,696 19 30 49 Total 753 52,278 53,783 106,061 49 51 100 Source: World Bank data. a. Activities with at least one environmental theme. b. Share of total expenditures assigned to the environment themes. 96 A P P E N D I X D : T H E B A N K ' S E N V I R O N M E N T A N D N AT U R A L R E S O U R C E M A N A G E M E N T P O R T F O L I O Table D.14: ENRM Analytical and Advisory Services by Sector Board, Fiscal Years 2002­08 Number of activities Expenditures (US$ thousands) Sector Board ESW TA Total ESW TA Total Environment 144 187 331 26,957 25,335 52,292 Agriculture and Rural Development 88 42 130 11,476 6,010 17,486 Energy and Mining 89 44 133 7,382 8,878 16,260 Water 41 30 71 5,747 6,230 11,977 Financial and Private Sector Development 9 9 3,278 3,278 Urban Development 11 7 18 952 755 1,707 Economic Policy 15 5 20 837 298 1,136 Social Development 8 6 14 477 517 994 Poverty Reduction 6 2 8 258 40 299 Transport 6 1 7 172 13 186 Public Sector Governance 2 1 3 65 11 75 Other 4 5 9 42 330 372 Total 414 339 753 54,365 51,696 106,062 Percent of total Environment 35 55 44 50 49 49 Agriculture and Rural Development 21 12 17 21 12 16 Energy and Mining 21 13 18 14 17 15 Water 10 9 9 11 12 11 Financial and Private Sector Development 0 3 1 0 6 3 Urban Development 3 2 2 2 1 2 Economic Policy 4 1 3 2 1 1 Social Development 2 2 2 1 1 1 Poverty Reduction 1 1 1 0 0 0 Transport 1 0 1 0 0 0 Public Sector Governance 0 0 0 0 0 0 Other 1 1 1 0 1 0 Total 100 100 100 100 100 100 Source: World Bank data. 97 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table D.15: ENRM Analytical and Advisory Services by Region, Fiscal Years 2002­08 Number of activities Expenditures (US$ thousands) Region ESW TA Total ESW TA Total Africa 89 76 165 9,897 15,429 25,326 East Asia and the Pacific 80 110 190 12,586 14,451 27,036 Europe and Central Asia 82 47 129 8,978 6,120 15,097 Latin America and the Caribbean 37 17 54 5,766 999 6,764 Middle East and North Africa 48 49 97 4,275 6,917 11,192 South Asia 44 27 71 9,505 2,952 12,457 Africa 34 13 47 3,359 4,829 8,188 Total 414 339 753 54,365 51,696 106,062 Percent of total Africa 21 22 22 18 30 24 East Asia and the Pacific 19 32 25 23 28 25 Europe and Central Asia 20 14 17 17 12 14 Latin America and the Caribbean 9 5 7 11 2 6 Middle East and North Africa 12 14 13 8 13 11 South Asia 11 8 9 17 6 12 Africa 8 4 6 6 9 8 Total 100 100 100 100 100 100 Source: World Bank data. 98 APPENDIX E: PERFORMANCE OF THE BANK'S ENRM PORTFOLIO Table E.1: Overall Project Outcomes of ENRM Projects in Comparison with the Bank-Wide Average, Exit Years 1995­2008 Based on number Weighted by Number of of projects net commitments ENRM ENRM Percent Percent projects commitments satisfactory Bank-wide satisfactory Bank-wide Exit year evaluateda evaluatedb outcomes average (%) outcomes average (%) 1995 10 718 80.0 67.9 96.4 77.9 1996 64 5,680 67.2 68.9 61.8 71.8 1997 75 5,358 74.7 71.8 82.5 73.5 1998 87 6,794 67.8 69.9 71.1 76.9 1999 83 5,152 65.1 67.4 75.5 71.9 2000 80 6,173 76.3 75.8 69.9 80.2 2001 78 4,999 71.8 74.5 68.5 77.3 2002 84 5,520 77.4 75.5 89.0 83.2 2003 82 5,338 70.7 73.6 69.3 72.7 2004 93 6,167 83.9 77.3 90.7 87.2 2005 91 5,066 84.6 80.9 80.9 83.9 2006 81 5,573 80.2 82.6 89.7 90.2 2007 59 3,006 78.0 75.7 72.7 82.3 2008 30 2,320 86.7 80.2 95.6 92.3 Total 997 67,865 75.4 74.3 77.8 79.3 Source: IEG project ratings database. a. Number of projects with at least one environment theme. b. Commitments to projects with at least one environment theme. 99 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table E.2: Likely Sustainability of ENRM Projects in Comparison with the Bank-Wide Average, Exit Years 1995­2008 Based on number Weighted by Number of of projects net commitments ENRM ENRM Percent Percent projects commitments likely Bank-wide likely Bank-wide Exit year evaluateda evaluatedb sustainable average (%) sustainable average (%) 1995 8 681 87.5 46.4 97.6 57.6 1996 40 3,662 67.5 47.1 67.3 53.7 1997 50 2,974 74.0 54.0 77.1 59.4 1998 59 4,617 69.5 49.5 74.7 65.4 1999 66 4,070 65.2 55.5 80.9 74.3 2000 75 5,931 74.7 70.8 66.7 75.0 2001 73 4,900 71.2 72.8 66.9 81.8 2002 75 5,063 68.0 74.3 83.4 83.7 2003 72 5,245 73.6 72.9 74.3 71.4 2004 83 5,728 78.3 78.8 81.7 88.0 2005 82 5,398 84.1 82.0 82.1 87.9 2006c 79 5,524 70.9 78.0 81.3 88.9 2007 60 3,011 61.7 69.6 69.6 75.6 2008 30 2,320 66.7 61.8 81.6 77.7 Total 852 59,125 72.1 65.2 76.3 73.5 Source: IEG project ratings database. a. Number of projects with at least one environment theme. b. Commitments to projects with at least one environmental theme. c. IEG changed its rating criterion from sustainability to risk to development outcome in 2006. 100 APPENDIX E: PERFORMANCE OF THE BANK'S ENRM PORTFOLIO Table E.3: ENRM Projects, Overall Project Outcomes by Type of Operation Number of projectsa Net commitments (percent)b Investment Development Investment Development Exit year operations policy operations Total operations policy operations Total 1995 9 1 10 78 100 80 1996 61 3 64 66 100 67 1997 71 4 75 73 100 75 1998 83 4 87 69 50 68 1999 79 4 83 65 75 65 2000 77 3 80 75 100 76 2001 76 2 78 71 100 72 2002 79 5 84 76 100 77 2003 76 6 82 71 67 71 2004 88 5 93 84 80 84 2005 86 5 91 86 60 85 2006 76 5 81 80 80 80 2007 56 3 59 80 33 78 2008 30 0 30 87 87 Total 947 50 997 75 78 75 Source: IEG project ratings database. a. Number of projects with at least one environment theme. b. Commitments to projects with at least one environment theme. Table E.4: ENRM Operations, Overall Project Outcomes by Source of Financing, Exit Years 1995­2008 Source of Number of projectsa Percent satisfactory financing 1995­2001 2002­08 1995­2008 1995­2001 2002­08 1995­2008 IBRD/IDA 432 449 893 69 80 74 GEF 37 53 90 84 81 82 Montreal Protocol 4 8 12 100 100 100 Rainforest Trust Fund 1 1 2 100 100 100 Special financing 3 9 12 100 67 75 Total 477 520 997 71 80 75 Source: IEG project ratings database. a. Number of projects with at least one environmental theme. 101 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table E.5: ENRM Operations, Overall Project Outcomes by Sector Board, Exit Years 1995­2008 Number of projectsa Percent satisfactory Sector Board 1995­2001 2002­08 1995­2008 1995­2001 2002­08 1995­2008 Environment 71 99 170 69 77 74 Agriculture and Rural Development 137 144 281 71 83 77 Energy and Mining 94 84 178 71 76 74 Water 59 71 130 56 82 70 Urban Development 44 44 88 73 77 75 Transport 41 39 80 88 97 93 Other 31 39 70 74 64 69 Total 477 520 997 71 80 75 Source: IEG project ratings database. a. Projects with at least one environmental theme. Table E.6: ENRM Operations, Overall Project Outcomes by Region, Exit Years 1995­2008 Number of projectsa Percent satisfactory Region 1995­2001 2002­08 1995­2008 1995­2001 2002­08 1995­2008 Africa 89 111 200 48 71 61 East Asia & Pacific 102 102 370 76 85 81 Europe & Central Asia 68 98 166 88 81 84 Latin America & Caribbean 101 114 427 75 86 81 Middle East & North Africa 55 43 211 73 77 74 South Asia 62 51 113 65 75 69 Total 477 519 996 71 80 75 Source: IEG project ratings database. a. Number of projects with at least one environmental theme. 102 APPENDIX E: PERFORMANCE OF THE BANK'S ENRM PORTFOLIO Table E.7: ENRM Operations, Overall Project Outcomes by Theme, Exit Years 1995­2008 Number of projectsa Percent satisfactory Theme 1995­2001 2002­08 1995­2008 1995­2001 2002­08 1995­2008 Biodiversity 60 85 145 70 76 74 Climate change 48 62 110 75 79 77 Environmental policy and institutions 188 186 374 73 77 75 Land administration 105 112 217 73 80 77 Pollution management 196 211 407 70 82 75 Water resources management 133 150 283 68 85 77 Other ENRM 34 52 86 56 79 70 Total 764 858 1,622 71 80 76 Source: IEG project ratings database. a. Number of projects in which each theme occurs. Therefore, projects with more than one environmental theme are double-counted. 103 APPENDIX F: EVALUATING ENVIRONMENTAL PERFORMANCE AT THE PROJECT LEVEL At the project level, overall outcome ratings do In order to assess the heterogeneity of the Bank's not necessarily reveal how projects have ENRM portfolio and its capacity to achieve performed in environmental terms. This is environmental results, IEG conducted an in-depth because the ENRM portfolio includes mostly review of the complete population of ENRM projects whose main objective is unrelated to the projects that were approved since 2000 and closed environment, and ICRs do not systematically in 2007 or 2008 and have been reviewed by IEG-- report on the environmental outcomes of a total of 51 projects (see table F.1). While a more projects at closing. When combined with the rigorous and exhaustive review of the entire Bank deficiencies in project coding, it is evident that portfolio would be required to determine the the Bank does not have a clear measure of how extent to which the Bank's strategy and efforts much of its project portfolio is supporting relating to environmental mainstreaming have environmental improvements or the extent to been optimal and effective in achieving environ- which this is having a positive impact on the mental outcomes, this portfolio offers a snapshot environment. of the heterogeneous nature of environmental mainstreaming and the quality of the M&E To understand how projects have performed in systems that are designed in line with the environ- environmental terms, it is necessary to disaggre- mental mainstreaming effort. gate the overall portfolio. Projects vary in their commitment to mainstream environmental Figure F.1 provides an overview of the 51 projects concerns. The ENRM portfolio is not homoge- in terms of their sector with main responsibil- neous. ENRM projects, which are mapped ity and depth of environmental emphasis directly to the Environment Sector Board, are (mainstreaming). The core environment proj- designed exclusively to achieve environmental ects--those mapped to the Environment outcomes. But the majority of projects in the Board--are mainly biodiversity conservation ENRM portfolio, which are mapped to other projects, have environmental development sectors, vary widely in their commitment to objectives and environmental components, and achieve environmental results. Some projects account for 25 percent of the ENRM portfolio. have environmental objectives and environmen- The other 75 percent of the projects are mapped tal components and have developed M&E to other sector boards and give different levels systems that adequately measure and monitor of attention to environmental development environmental results. Others have only environ- objectives and components. At the extreme, mental components and often do not include about 30 percent of the projects had no environ- indicators that are designed to monitor or mental objectives or components; they were measure environmental results. Some projects coded with an environmental theme only have neither an environmental objective nor because their choices of technologies in the component, but are coded with an ENRM energy and transport sectors, for example, were subtheme. These projects for the most part do presumed to have positive externalities in terms not include any indicators in the M&E systems reduced pollution or greenhouse gas emissions. that relate directly to the environmental theme. Another 18 percent of projects had environmen- 105 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 .1: Figure F Levels of Environmental Mainstreaming in the ENRM Portfolio Core ENV ENV DO + ENV ENV M&E component expected ENV component only ENRM projects w/o Missed ENV DO or ENV opportunities component w/o ENV M&E 0 2 4 6 8 10 12 14 16 18 Number of projects Environment Agriculture & rural Energy & mining Transport Water Urban Other Source: IEG analysis of 51 ENRM projects approved since 2000 and closed in 2007 and 2008. Note: DO = development objective of the project; ENV = environment or environmental; w/o = without. tal components but lacked stated environmental and institutions, but their indicators were objectives. restricted to outputs and processes in almost all cases, making it difficult for the Bank to assess Core environment projects generally have indica- the achievement of their presumed positive tors in their M&E systems that were constructed environmental outcomes. In Chad, for example, to monitor and report on environmental where the Bank lent support for the reform outcomes (see chapter 4). The large share of of the environmental and social regulatory biodiversity conservation projects--both terres- framework applicable to petroleum develop- trial and marine--in this subgroup were mainly ment projects, the only indicators applied were in Africa and Latin America and helped countries process indicators: the development of environ- build broader biological information systems. mental regulations for the sector and the preparation of a new hydrocarbon code. In Projects with an environmental objective and Nicaragua, the Bank supported decentralization dedicated environmental components (but not of environmental management, emphasizing the mapped to the Environment Sector Board) included application and enforcement of environmental environmental indicators in their results norms and instruments, yet the only reported frameworks. These projects tended to be gains made after implementation were the mapped to the Agricultural and Rural Develop- approval of municipal land use plans, the ment and Energy and Mining Sector Boards, but drafting and passage of municipal laws, and the unlike the core environmental projects men- signing of interagency agreements. tioned above, their indicators were not generally constructed to monitor and report on environ- There were only a few exceptions to the general mental outcomes. lack of environmental indicators at the outcome level in this subset of projects. For example, the A significant number of these projects were Azerbaijan Rehabilitation and Completion of designed to strengthen environmental policies Irrigation and Drainage Infrastructure Project 106 A P P E N D I X F : E V A L U AT I N G E N V I R O N M E N TA L P E R F O R M A N C E AT T H E P R O J E C T L E V E L included relevant environmental indicators in its have indicators related to the coded subtheme. M&E system, such as waterlogging and soil salinity This represents a missed opportunity for the arising from its irrigation and drainage schemes, Bank to fully account for its greening efforts, either of which could lead to soil degradation and particularly in relation to the positive environ- negatively affect agricultural productivity. mental externalities generated by projects that promote environmentally friendly technologies. Most of the nine projects with an environmental In China, for example, the Bank's ENRM portfo- component but no environmental objective did not lio in the transport sector has helped to reduce include indicators for their environmental reliance on fossil fuels, scale-up hydropower, and interventions in their results frameworks. These promote electric traction over the use of diesel projects included support for soil erosion fuel in the rail industry. control and reforestation (Algeria), an environ- mental cell in the power sector (Albania), Overall, IEG has rated 57 percent of the M&E watershed protection (Rwanda), and restructur- systems of these 51 projects as modest or negligi- ing of the Sub-secretariat of Environmental ble. Similar to the Bank's overall portfolio Sanitation (Ecuador). This set of projects also (discussed in part I of the main report), there is included some small grant schemes to support a weak but positive correlation between the environmentally sustainable rural development quality of project-level M&E and better project through a subproject approach (Burkina Faso outcomes (table F.1). The simple correlation and Ecuador), but the projects failed to achieve between M&E quality and project outcomes is the environmental outcomes aligned with the 0.60. ENRM projects with outcome ratings of projects' rural development objectives. IEG has highly satisfactory had M&E systems rated as also flagged this issue of the lack of reporting and high or substantial; ENRM projects with outcome dissemination of results and lessons related to ratings of unsatisfactory have only modest or the implementation of environmental invest- negligible M&E ratings. But these overall ments at the subgrant level in several of its outcome ratings still relate to the project as a evaluations, most recently in its review of the whole, not necessarily to its environmental Critical Ecosystem Partnership Fund (IEG outcomes. The analysis is not very meaningful 2007b). apart from biodiversity conservation and other projects under the Environment Sector Board. It The M&E systems of the remaining 16 projects, is clear that M&E systems are weak overall and which were designed without environmental nearly invisible for ENRM projects under other development objectives or components, did not sectors. .1: Table F Relationship between IEG Outcome and M&E Ratings for the ENRM Portfolio IEG outcome ratings Highly Moderately Moderately M&E rating satisfactory Satisfactory satisfactory unsatisfactory Unsatisfactory Total High 1 5 6 Substantial 2 10 4 16 Modest 10 10 1 2 23 Negligible 2 3 1 6 Total 3 25 16 4 3 51 Source: IEG ICR Reviews of 51 ENRM projects approved since 2000 and closed in 2007 and 2008. 107 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 .2: Table F Summary of the 51 Projects Reviewed Approval Exit fiscal fiscal Sector Project year year Financing Country Board ID Project name A. Projects with environmental development objectives and components 2000 2007 GEF Costa Rica ENV P061314 Ecomarkets 2000 2007 IDA Mozambique ENV P070305 Coastal and Marine Biodiversity Management 2001 2007 GEF Central America ENV P053349 Mesoamerican Barrier Reef System 2001 2008 GEF Colombia ENV P063317 Conservation and Sustainable Use of Biodiversity in High Andes Region 2001 2007 IBRD Latvia ENV P058476 Liepaja Region Solid Waste Management 2001 2007 GEF Peru ENV P065200 Indigenous Management of Protected Areas in the Peruvian Amazon 2002 2008 GEF Burkina Faso ENV P052400 Partnership for Natural Ecosystem Management 2002 2008 GEF Croatia ENV P042014 Karst Ecosystem Conservation 2002 2008 GEF Latin America ENV P072979 Integrated Silvopastoral Approaches to Ecosystem Management 2002 2007 GEF Romania ENV P066065 Agricultural Pollution Control 2003 2008 GEF Ecuador ENV P066752 National System of Protected Areas 2003 2007 GEF Europe & Central Asia ENV P048795 Baltic Sea Regional ­ Phase I 2004 2007 GEF Argentina ENV P078143 Enabling Activity for 2nd National Communication of Argentine Government to the Convention on Climate Change 2000 2007 IDA Azerbaijan ARD P008284 Rehabilitation and Completion of Irrigation and Drainage Infrastructure 2000 2008 IBRD Tunisia ARD P035707 Water Sector Investment Loan 2001 2007 IDA Nicaragua ARD P055823 Second Rural Municipal Development 2002 2008 IDA Albania ARD P069479 Pilot Fishery Development 2000 2007 IDA Chad EMT P048202 Petroleum Sector Management Capacity Building 2000 2007 IBRD Romania EMT P056337 Mine Closure and Social Mitigation 2001 2008 IBRD Algeria EMT P067567 Energy and Mining Technical Assistance Loan (Emtal) 2001 2007 IDA Mozambique EMT P001808 Mineral Resources Management Capacity Building 2005 2008 IBRD Poland EMT P083093 Coal Mine Closure 2001 2007 IBRD Argentina SDV P057473 Indigenous Community Development 2000 2007 IBRD Philippines TR P039019 First National Roads Improvement 2001 2008 IBRD China TR P045915 Urumqi Urban Transport Improvement 2000 2007 IBRD Uruguay WAT P063383 Ose Modernization and Systems Rehabilitation B. Projects with environmental components, but without environmental development objectives 2000 2008 IDA Mali ARD P041723 National Rural Infrastructure 2001 2007 IDA Burkina Faso ARD P035673 Community-Based Rural Development 2002 2007 IBRD Ecuador ARD P039437 Poverty Reduction and Local Rural Development (Prolocal) 108 A P P E N D I X F : E V A L U AT I N G E N V I R O N M E N TA L P E R F O R M A N C E AT T H E P R O J E C T L E V E L Risk to Commitments development M&E Themes (US$ millions) Outcome outcome system Biodiversity, climate change, env pol / inst 32.6 Moderately satisfactory Moderate Modest Biodiversity, env pol / inst, water resource mgt 6.1 Moderately unsatisfactory High Negligible Biodiversity, env pol / inst, water resource mgt 11.03 Satisfactory Significant High Biodiversity, env pol / inst 15 Moderately unsatisfactory Significant Negligible Climate change, pollution mgt 2.2 Moderately satisfactory Moderate Modest Biodiversity, other ENRM 10 Satisfactory Moderate Substantial Biodiversity, env pol / inst 7.5 Moderately satisfactory High Modest Biodiversity, env pol / inst 5.07 Highly satisfactory Negligible to low Substantial Biodiversity, climate change, land admin 4.5 Highly satisfactory Negligible to low High Env pol / inst, land admin, pollution mgt, water res mgt 5.15 Satisfactory Moderate High Biodiversity, env pol / inst 8 Satisfactory Moderate Substantial Biodiversity, env pol / inst, pollution mgt, water res mgt 5.5 Satisfactory Moderate Modest Biodiversity, climate change, env pol / inst, other ENRM 1.14 Satisfactory Negligible to low High Water res mgt 48.0 Satisfactory Moderate High Water res mgt 132.0 Moderately satisfactory Significant Negligible Env pol / inst, other ENRM 30.7 Moderately satisfactory Moderate Modest Pollution mgt, water res mgt 5.9 Moderately satisfactory High Modest Env pol / inst, pollution mgt 23.9 Moderately unsatisfactory High Negligible Env pol / inst 44.5 Satisfactory Negligible to low Modest Env pol / inst 12.9 Unsatisfactory Significant Modest Env pol / inst, other ENRM 19.5 Satisfactory Moderate Substantial Land admin, pollution mgt 55.3 Moderately unsatisfactory Significant Modest Land admin 5.0 Moderately satisfactory Moderate Negligible Pollution mgt 138.2 Moderately satisfactory High Modest Climate change, pollution mgt 97.6 Satisfactory Moderate Modest Env pol / inst, pollution mgt 27.0 Moderately satisfactory Moderate Substantial Land admin, water res mgt 127.0 Moderately satisfactory Significant Substantial Other ENRM 73.0 Satisfactory Moderate Substantial Land admin 25.2 Satisfactory Moderate Modest (Table continues on the following page.) 109 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 .2: Table F Summary of the 51 Projects Reviewed (continued) Approval Exit fiscal fiscal Sector Project year year Financing Country Board ID Project name 2003 2007 IBRD Algeria ARD P076784 Second Rural Employment 2002 2007 IDA Albania EMT P074905 Power Sector Rehabilitation and Restructuring 2002 2007 IBRD Brazil TR P055954 Goias State Highway Management 2003 2007 IBRD Algeria UD P067605 Urban Natural Hazard Vulnerability Reduction in the Wilaya of Algiers 2000 2008 IDA Rwanda WAT P045182 Rural Water Supply and Sanitation 2001 2007 IBRD Ecuador WAT P049924 Rural and Small Towns Water Supply and Sanitation (Praguas) C. Projects without environmental development objectives or components 2001 2007 IDA Honduras ARD P073035 Access to Land Pilot (Pacta) 2001 2008 IDA Pakistan ARD P071092 North West Frontier Province On-Farm Water Management 2001 2007 IDA Sri Lanka ARD P050738 Land Titling and Related Services 2005 2007 IDA Congo, Rep of EP P083627 Economic Recovery Credit 2002 2007 IBRD Lithuania ED P070112 Education Improvement 2000 2008 IBRD China EMT P056424 Tongbai Pumped Storage 2000 2007 IDA Vietnam EMT P056452 Rural Energy 2000 2007 SPF West Bank & Gaza EMT P040506 Electricity Sector Investment and Management 2001 2008 IBRD China TR P056199 Third Inland Waterways 2001 2008 IBRD India TR P010566 Gujarat State Highway 2001 2008 IDA Mongolia TR P056200 Transport Development 2004 2008 IBRD China TR P075602 Second National Railways (Zhe-Gan Line) 2000 2007 IBRD Turkey UD P068368 Marmara Earthquake Emergency Reconstruction 2002 2007 IDA Tonga UD P075171 Cyclone Emergency Recovery and Management 2000 2008 IBRD Brazil WAT P039199 Low Income Sanitation Technical Assistance - Prosanear 2001 2008 IBRD Ukraine WAT P035786 Lviv Water and Wastewater Note: ENV = environment; ARD = agriculture and rural development; EMT = energy and mining; SDV = social development; TR = transport; WAT = water; UD = urban development; EP = economic policy; ED = education. 110 A P P E N D I X F : E V A L U AT I N G E N V I R O N M E N TA L P E R F O R M A N C E AT T H E P R O J E C T L E V E L Risk to Commitments development M&E Themes (US$ millions) Outcome outcome system Land admin, water res mgt 7.7 Satisfactory Moderate Modest Pollution mgt 34.5 Moderately satisfactory Significant Substantial Other ENRM 64.4 Satisfactory Negligible to low Substantial Water res mgt 5.6 Moderately satisfactory Moderate Modest Water res mgt 22.3 Satisfactory Negligible to low Modest Pollution mgt 31.9 Satisfactory Significant Modest Land admin 8.8 Satisfactory Moderate Substantial Water res mgt 25.2 Moderately satisfactory Significant Substantial Land admin 4.6 Unsatisfactory High Negligible Other ENRM 17.6 Unsatisfactory High Modest Climate change 33.9 Satisfactory Negligible to low High Climate change 197.5 Satisfactory Negligible to low Substantial Pollution mgt 138.8 Satisfactory Negligible to low Substantial Pollution mgt 15.0 Moderately satisfactory High Modest Climate change 99.3 Satisfactory Negligible to low Substantial Pollution mgt 280 Highly satisfactory Negligible to low Substantial Pollution mgt 34.4 Moderately satisfactory Significant Modest Climate change 199.0 Satisfactory Negligible to low Modest Land admin 294.1 Satisfactory Substantial Env pol / inst 6.1 Satisfactory Moderate Modest Pollution mgt 16.7 Moderately satisfactory Moderate Modest Pollution mgt 24.0 Satisfactory Moderate Modest 111 APPENDIX G: EVALUATING ENVIRONMENTAL PERFORMANCE AT THE COUNTRY LEVEL--AN ANALYSIS OF RECENT CASCR REVIEWS CASCR's IEG Review Fiscal ratings of period year Environment IEG environment (fiscal of IEG pillar/ outcome pillar/ Country years) review objective rating objective South Africa 2000­06 2008 Yes Moderately unsatisfactory Moderately unsatisfactory Maldives 2000­07 2008 Yes Moderately satisfactory Moderately unsatisfactory Tanzania 2001­06 2007 No Satisfactory NA Belarus 2002 ­F06 2008 Yes Moderately satisfactory Satisfactory Chile 2002­06 2007 Yes Satisfactory Satisfactory Uzbekistan 2002 ­07 2008 No Unsatisfactory NA Kyrgyz Republic 2003­06 2007 Yes Moderately unsatisfactory Moderately unsatisfactory Senegal 2003­06 2007 No Moderately satisfactory NA Vietnam 2003­06 2007 Yes Satisfactory Satisfactory Sri Lanka 2003­07 2008 No Moderately unsatisfactory NA Gambia, The 2003­07 2008 No Moderately unsatisfactory NA Nicaragua 2003­07 2008 Yes Moderately satisfactory Satisfactory Mauritania 2003­07 2008 Yes Moderately unsatisfactory Moderately unsatisfactory 112 A P P E N D I X G : E V A L U AT I N G E N V I R O N M E N TA L P E R F O R M A N C E AT T H E C O U N T R Y L E V E L IEG CASCR Review comments Environment outperformed other complementary objectives. The Bank engaged successfully and contributed in this area, primarily through GEF operations. The CAS focused on protecting the invaluable environment of Maldives, particularly its coral reefs and coast. Bank support was to be provided through fisheries sector ESW and through policy dialogue. The first attempt at preparing the fisheries report had to be abandoned due to quality concerns. However, a study on fisheries has recently been completed. Although the preparatory process for the fisheries report may have influ- enced policy dialogues and some outputs, outcomes are still marginal. The provision of reliable supplies of safe drinking water remains a chal- lenge on some islands and most still lack appropriate sanitation and solid waste collection systems. NA The Bank's analytical support related to the environment helped in the design of environmental institutions, mainstream environmental concerns, and enhance international compliance. The Bank's engagement has also been important in generating knowledge about energy-environment linkages and energy efficiency. The Bank financed (under the Prototype Carbon Fund) a hydropower plant that has reduced emissions of CO2 equivalent to 80,000 tons per year. The Bank also financed the studies and some of the works for the transport system in the metropolitan region of Santiago, which is expected to reduce the number of bus kilometers by 30 percent and the particle contamination of buses by 75 percent. NA The CASCR does not address progress in several important areas such as the environment. The CASCR is silent on environmental protection, although it does note that a planned fiscal 2006 Environmental Expenditure and Policy Project was dropped. NA The second CAS objective was to help improve social and gender indicators and environmental conditions in the country, especially in poorer areas. Within an overall context of rapid poverty reduction, most indicators show good progress in relation to the objective. On this basis, even though the absence of outcome indicators to gauge progress on environmental sustainability makes it difficult to assess this area, IEG has rated the outcome associated with this pillar as satisfactory. NA NA The CAS objective of reducing environmental vulnerability was largely met. IDA supported this goal with the Sustainable Forestry, GEF-Atlantic Biological Corridor, and Disaster Prevention Projects and other donors funded supporting studies. Nicaragua is also involved in IDA's regional ini- tiative (Corazon Transboundry Biosphere Reserve Project) to improve management of natural resources. The CAS describes the current environmental situation as one of inadequate information, severe shortages of qualified labor, and an uncon- trolled exploitation of all natural resources that, in the case of the fisheries, has led to overfishing in both an economic and a biological sense. 113 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 CASCR's IEG Review Fiscal ratings of period year Environment IEG environment (fiscal of IEG pillar/ outcome pillar/ Country years) review objective rating objective Colombia 2003­07 2008 Yes Moderately unsatisfactory Moderately unsatisfactory Niger 2003­07 2008 No Moderately unsatisfactory NA Ethiopia 2003­07 2008 Yes Moderately satisfactory Moderately satisfactory Rwanda 2003­08 2009 No Moderately satisfactory NA Benin 2004­06 2009 Yes Moderately satisfactory Satisfactory Macedonia, FYR 2004­06 2007 Yes Moderately satisfactory Moderately satisfactory Turkey 2004­07 2008 Yes Satisfactory Moderately unsatisfactory Mali 2004­07 2008 No Moderately satisfactory NA Ukraine 2004­07 2008 Yes Moderately satisfactory Moderately unsatisfactory Brazil 2004­07 2008 Yes Moderately satisfactory Moderately satisfactory Zambia 2004­07 2008 No Moderately unsatisfactory Moderately unsatisfactory Ghana 2004­07 2007 No Moderately satisfactory NA Mozambique 2004­07 2007 Yes Moderately satisfactory Moderately satisfactory Madagascar 2004­07 2007 No Moderately satisfactory NA 114 A P P E N D I X G : E V A L U AT I N G E N V I R O N M E N TA L P E R F O R M A N C E AT T H E C O U N T R Y L E V E L IEG CASCR Review comments Environment outperformed other complementary objectives. IEG rated this pillar moderately unsatisfactory mainly because of the failure to cor- rect the fundamental causes of the fiscal imbalance and the limited contribution of the improvements in infrastructure, the other another major component under this pillar, to stimulating growth. NA The Bank has been criticized by other donors for neglecting to follow through in its supervision of environmental issues (see IEG ICR Review of PRSC program). In sum, the impact--in terms of reversing loss of environmental resources--is not yet known. NA No performance indicators related to environmental management were identified in the CAS but the GEF-financed National Parks Conservation and Management Program provided support for sustainable management of Benin's national parks, while the PRSC and the GEF Forests and Ad- jacent Lands Management Project supported reforms in forestry management. Some progress has also been made in improving the manage- ment of national forests and wildlife reserves, although absent continued donor financing, the sustainability of progress made in these areas is doubtful. During the fiscal 2004 CAS period, the third phase of the 15-year environmental program started to be implemented. The government, with support from donors, designated 4.5 percent of the national territory as protected areas. Deforestation appears to have de- creased in protected areas, but continues in other areas, although at a slower pace than before. The legislation for the protection of the coun- try's natural resources was enacted, but it is still to be implemented. Important environmental institutions were established but their effective- ness remains an issue, given their overlapping responsibilities and weak enforcement capacities. Community-based approaches for natural re- source management are being implemented, but progress is constrained by weak capacity of the communes, poor governance, a complicated contracting process, and unclear property rights. While there was forward progress on both environmental management and disaster prevention, the pace was slower than the Bank had pro- jected in the CAS. The Bank found it difficult to structure appropriate interventions faced with an apparent lack of commitment and interest on the part of the Turkish authorities. The Bank was less successful in getting the Turkish government to focus on broader environmental issues, despite the importance of this in the European Union accession negotiations. NA Bank CAS support targeted strengthening Ukraine's institutional capacity to implement Ukraine's National Environmental Strategy (NES). The in- stitutional strengthening project did not materialize and the Avoz Black Sea Project was cancelled due to lack of government commitment. Bank objectives in supporting implementation of Ukraine's National Environmental Strategy through strengthening environmental institutional capac- ity were not achieved. However, negotiation of an emissions-reduction purchase agreement under the Kyoto Protocol was achieved with Bank support. Brazil took important steps to improve environmental sustainability, but overall progress was mixed. The outcome of Bank support is rated mar- ginally satisfactory because of good Bank contributions and outcomes on water resource management, more modest contributions in land man- agement and biodiversity, and poor results in access to local services (and problems with municipal lending). With respect to the mining sector and environmental regulations, the original indicator was substituted with an indicator to measure non- compliance with environmental regulations. However, the indicator was not monitored. There was a major spillage of effluent into the Kafue River in 2007, undermining public confidence in environmental controls. NA IDA and GEF programs of assistance related to sustainable management of natural resources experienced substantial implementation bottle- necks (including weak government capacity and commitment) and most of the intermediate indicators established were not attained. NA 115 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 CASCR's IEG Review Fiscal ratings of period year Environment IEG environment (fiscal of IEG pillar/ outcome pillar/ Country years) review objective rating objective Malawi 2004­07 2007 Yes Moderately unsatisfactory Satisfactory Indonesia 2004­08 2009 No Moderately satisfactory NA Costa Rica 2004­08 2009 Yes Moderately satisfactory Satisfactory Serbia 2005­07 2008 No Moderately satisfactory NA Bosnia 2005­07 2008 Yes Moderately unsatisfactory Moderately unsatisfactory Montenegro 2005­07 2007 No Satisfactory NA Guatemala 2005­08 2009 Yes Moderately satisfactory Moderately unsatisfactory Moldova 2005­08 2009 Yes Moderately satisfactory Moderately satisfactory Croatia 2005­08 2009 Yes Moderately satisfactory Satisfactory India 2005­08 2009 No Moderately satisfactory NA Mexico 2005­08 2008 Yes Moderately satisfactory Moderately satisfactory Note: NA = Not applicable. 116 A P P E N D I X G : E V A L U AT I N G E N V I R O N M E N TA L P E R F O R M A N C E AT T H E C O U N T R Y L E V E L IEG CASCR Review comments Sustainable use of natural resources in the Mulanje Massif. The Bank program contributed to the preservation of biodiversity and the unique ecosystems of Mulanje Massif by incorporating biodiversity objectives into management of the reserve and focusing on community involvement in forest protection. NA The Bank has effectively contributed to enhancing Costa Rica's biodiversity and forest management program. Substantial improvements have been made through the innovative system for Payment for Environmental Services (PSA) and decentralized administration of conservation areas. NA Inefficiencies in key social and environmental sectors remain to be addressed. NA There is limited evidence of achieving outcomes in mainstreaming environmental concerns. The second pillar sought to improve access to social services, capital, and community infrastructure, while minimizing environmental risks. With regard to the strategic objectives under this pillar, strongest areas of performance were registered with regard to health, water and sanitation, and environment. The focus of the CAS under Pillar 4--sustainable natural resource management--was on environmental management capacity, wastewater and water supply management, and energy efficiency. Significant improvements in environmental management have taken place and the govern- ment has developed a strategy for meeting the Kyoto Protocol, thereby meeting the CAS objective. NA Bank support has contributed to the development of policies for improved domestic environmental resource management and Mexico's active participation in addressing global environmental problems. Country outcome targets were presumably mostly achieved either fully or substan- tially. 117 APPENDIX H: EVALUATING THE PERFORMANCE OF REGIONAL ENVIRONMENTAL PROJECTS The purpose of regional projects has typically components. Their M&E systems have therefore been to help neighboring countries manage a tended to include a large number of indicators common natural resource, such as a body of water that, without prioritization, have proven to be (such as the Aral Sea or Lake Victoria), a river somewhat unmanageable (the M&E system for system (such as the Nile), or a shared ecosystem South America's Guarani Aquifer System Project (the Mesoamerican biological corridor, for included more than 70 indicators). Some projects example). Their specific evaluation challenges are did not develop a results framework until project to aggregate country-level results to ascertain the closing (Organization of Eastern Caribbean States achievements and the sustainability of regional [OECS] Ship-generated Waste Management Proj- environmental outcomes and to assess the ect), and in other cases, such as the first phase of effectiveness and efficiency of their program-level Lake Victoria Environmental Management institutional arrangements, such as governance Program, effective feedback loops to link and management. scientific knowledge to policy and decision making were lacking. IEG's recent evaluation of Bank support to multicountry operations reviewed the M&E Regional projects have relied on country-level data systems of a representative sample of seven such for effective reporting. One of the major challenges projects (IEG 2007d). With one exception, these in designing and implementing an M&E system projects had overly ambitious and poorly defined has been the uneven capacity of country systems objectives, although for some this may reflect the across the regional partnership in developing a need to compromise between partners with standardized system for data collection, manage- disparate demands rather than poor project ment, and reporting. While some regional projects design. Nevertheless, project indicators across addressed this concern by hiring large firms to many of the programs suffered from lack of carry out highly specialized studies, as in the case specificity and immeasurable targets, and some of the Guarani Aquifer System Project, this resulted projects included objectives without any indica- in long delays due to procurement issues. Other tors to measure them. projects, such as the OECS Ship-generated Waste Management Project, relied too heavily on external Regional environmental projects have tended to consultants to design M&E systems in which the be structurally complex, with a large number of users did not ultimately own the data. 119 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table H.1: Summary of Seven Regional Projects Reviewed Approval Exit fiscal fiscal Sector year year Financing Country Board Project ID Project name Mediterranean Environment Technical Assistance Program (METAP) 1996 2005 IBRD Algeria ENV P004960 Industrial Pollution Control 1998 2005 IBRD Egypt, Arab Rep of ENV P054958 Egypt Pollution Abatement 2001 Special West Bank and Gaza ENV P054051 Solid Waste and Environmental Management Financing 2003 IBRD Iran ENV P074499 Environmental Management Support 2006 IBRD Egypt, Arab Rep of ENV P090073 Second Pollution Abatement Project 2007 IBRD Tunisia ENV P095012 Sustainable Municipal Solid Waste Management OECS Ship-Generated Waste Management 1995 2003 IBRD / OECS countries ENV P006970 / OECS Ship-Generated Waste Management GEF P006957 Lake Victoria Environmental Management (LVEMP) 1997 2003 IDA / Kenya ENV P046838 / Lake Victoria Environmental Management GEF P046871 1997 2006 IDA / Tanzania ENV P046837 / Lake Victoria Environmental Management GEF P046872 1997 2006 IDA / Uganda ENV P046836 / Lake Victoria Environmental Management GEF P046870 2005 2006 IDA Tanzania ENV P090680 Lake Victoria Environmental Management Project--Supplemental Credit 2005 GEF Africa ARD P085782 Lake Victoria Transboundary medium size 2009 IDA / Eastern Africa ENV P100406 / Lake Victoria Environmental Management Project II GEF P103298 Aral Sea Water and Environment Management 1998 2003 GEF Aral Sea WAT P008326 Aral Sea Water and Environmental Management 2001 IBRD Kazakhstan ARD P046045 Syr Darya Control North Aral Sea 2003 IBRD Uzbekistan ARD P009127 Drainage, Irrigation and Wetlands Improvement Central Asia Biodiversity 1999 2006 GEF Central Asia ENV P042573 Central Asia Biodiversity Silvopastoral Integrated Ecosystem Management 2002 2008 GEF Latin America ENV P072979 Silvopastoral Integrated Ecosystem Management Guarani Aquifer Project 2002 2009 GEF Latin America WAT P068121 Guarani Aquifer Project Source: World Bank data. Note: ENV = environment; ARD = agriculture and rural development; EMT = energy and mining; SDV = social development; TR = transport; WAT = water; UD = urban development; EP = economic policy; ED = education. 120 A P P E N D I X H : E V A L U AT I N G T H E P E R F O R M A N C E O F R E G I O N A L E N V I R O N M E N TA L P R O J E C T S Sustainability/ Commitments risk to development Themes (US$ millions) Outcome outcome Env pol / inst, pollution mgt 78.0 Moderately satisfactory Likely Env pol / inst, pollution mgt 35.0 Satisfactory Likely Env pol / inst, pollution mgt 9.5 Env pol / inst, pollution mgt, water res mgt 20.0 Pollution mgt 20.0 Climate change, env pol / inst, pollution mgt 22.0 Env pol / inst, pollution mgt, water res mgt 11.5 / 9.75 Moderately satisfactory Likely Biodiversity, Env pol / inst, Pollution mgt, Water res mgt 12.8 / 11.5 Unsatisfactory Likely Biodiversity, Env pol / inst, Pollution mgt, Water res mgt 10.1 / 10.3 Moderately satisfactory Likely Biodiversity, env pol / inst, pollution mgt, water res mgt 12.1 / 13.2 Moderately satisfactory Likely Biodiversity, env pol / inst, other ENRM 3.5 Water res mgt 1.0 Biodiversity, water res mgt, other ENRM 90.0 / 7.0 Biodiversity, env pol / inst, water res mgt 12.2 Moderately unsatisfactory Likely Biodiversity, water res mgt 64.5 Land admin, pollution mgt, water res mgt 60.0 Biodiversity, climate change 10.15 Unsatisfactory Significant Biodiversity, climate change, land admin 4.5 Highly satisfactory Negligible Climate change, env pol / inst, water res mgt 13.4 Unsatisfactory (2006) Significant 121 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table H.2: Regional Projects: Objectives and Components Project name Project dates Objectives Components Regional partnership Mediterranean Approved fiscal 1990 Supported by the World Bank, European · To generate investments in selected Environment (METAP) Union, European Investment Bank, and areas (hazardous wastes, integrated UNDP, METAP aims to improve the coastal zone management, water environmental quality around the quality, and the like) Mediterranean Sea by providing · To build national capacity for policy technical assistance to 14 countries development, project preparation, bordering the sea for the preparation training, institutional strengthening, of environmental projects. It is currently and local empowerment, and in its fifth phase of implementation. strengthen regional capacity with im- A disengagement strategy will be proved knowledge sharing and re- developed in this phase. gional networks Regional projects Eastern Caribbean Approved fiscal The project was designed to strengthen · To improve solid waste management Waste Management 1995, closed 2003 the capacity of the countries to manage including facilities for storage, collec- their solid waste and process ship- tion, and disposal generated waste generated largely by · To strengthen countries' capacity to tourism and to preserve the quality of effectively manage solid waste land and marine environments. · To facilitate compliance with the Six countries participated in this project. MARPOL treaty on ship-generated waste · To reduce terrestrial and marine pollution Lake Victoria LVEMP approved The projects seek to reverse the bio- · To maximize sustainable benefits to Environmental fiscal 1997, diversity decline in Lake Victoria, the riparian communities from use of basin Management (LVEMP) closed 2005 second-largest body of fresh water in resources to generate food, income, I and II LVEMP II approved the world and an important resource for safe water, and reduced disease in 2009 food, drinking and irrigation water, energy, · To conserve biodiversity and genetic and transport to the three riparian countries resources for the benefit of the ripar- and numerous communities around the lake. ian countries and the global community · To harmonize national management programs to reverse the increasing environmental degradation Aral Sea Water and Approved fiscal The overall goals of the project were to · To stabilize the environment of the Environmental 1998, closed assist the five riparian countries (Kyrgyz Aral Sea Basin Management 2003 Republic, Kazakhstan, Tajikistan, Turkmen- · To rehabilitate the disaster area istan, and Uzbekistan) to improve water around the sea allocations among them, and to provide · To improve the management of the technical and financial support for regional- international waters of the Aral Sea and national-level actions to deal with Basin the Aral Sea environmental crises. · To build the capacity of institutions at the regional and national levels to ad- vance the program's aims 122 A P P E N D I X H : E V A L U AT I N G T H E P E R F O R M A N C E O F R E G I O N A L E N V I R O N M E N TA L P R O J E C T S Project name Project dates Objectives Components Central Asia Approved fiscal 1999, The project sought to help Kazakhstan, · To support vulnerable biological com- Biodiversity closed 2006 the Kyrgyz Republic, and Uzbekistan to munities and ensure conservation of conserve and enhance biodiversity in the globally important biodiversity in the Western Tien Shan, which is home to West Tien Shan territory 3,000 species of flora and fauna and · To assist three countries to strengthen endangered wildlife. and coordinate national policies, leg- islative frameworks, and institutional arrangements for biodiversity protec- tion · To identify alternative income-generat- ing activities for local communities and thus reduce pressure on protected na- ture reserves · To establish a regional coordination system for biodiversity conservation to prevent fragmentation of habitat corridors Latin America Approved fiscal The project sought to demonstrate and · To determine a level of payments suffi- Land Use 2002, measure the effects of the introduction cient to cause farmers to switch to in- closed 2008 of payment incentives for environmental tegrated silvopastoral farming systems services to farmers on their adoption of · To develop methodologies for monitor- integrated silvopastoral farming systems ing and payment for environmental in degraded pasturelands in Colombia, services Costa Rica, and Nicaragua. It was a pilot · To monitor the impact of silvopastoral operation that capitalized on economy of systems on biodiversity and water scale to demonstrate that silvopastoral resources farming and ecosystem improvement · To develop and promote policies for can coexist. sustainable livestock husbandry Guarani Aquifer Approved fiscal The project objective was to study and · To support the four countries to jointly (SAG) 2002 and due to plan for the long-term management of elaborate and implement a common close in 2009 the SAG, one of the world's largest institutional and technical framework groundwater reservoirs. It is bordered for the management and preservation by Argentina, Brazil, Paraguay, of the Guarani Aquifer System, with a and Uruguay. focus on prevention of overuse and contamination 123 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table H.3: Regional Projects: Outcomes and Sustainability Program name Outcomes (achievements) Sustainability Mediterranean Program assistance was invaluable in finding Although the sustainability of policy reforms and the out- Environment appropriate technologies and cost-effective solutions comes of individual country-level technical assistance activi- (METAP) and in resolving project management issues. But it ties is high in many instances, at the overall program level, is difficult to find conclusive evidence about the the lack of country counterpart funding and the complete de- impact of the majority of these interventions. For pendence of the program on external grant funding are major the capacity-building objective, the METAP program risks to long-term sustainability. Countries are also not di- has led to enhanced national institutional capacity rectly involved in implementing program activities. METAP IV in several member countries. However, without began devolving management and implementation of certain a clear statement of expected regional outcomes activities (still financed by METAP) to regional entities at the program level and related performance (Mediterranean Regional Facility and UN Economic and measures, it is not possible to assess impact Social Commission for Western Asia, and so on), thus build- on regional capacity building. ing up lower levels of capacity. Should such external sources of financing be terminated it is unlikely that regional-level activities will be continued. Eastern Caribbean The project achieved most of its solid waste Project achievements in strengthening institutional, techni- Waste Management management objectives; there were significant cal, and financial capacity of individual countries to manage improvements in collection coverage and disposal solid waste have been strengthened and are likely to be in the five countries (except Dominica). But recycling sustained after project closure. and waste separation activities were not fully implemented, and there was little progress in management of marine pollution and ship- generated wastes. Lake Victoria Although the pace of implementation was slow, the Phase I achievements are limited given the enormous and Environmental project made significant impact in three areas: ambitious scope of work, but the sustainability of these Management fisheries research, fisheries management, and achievements is likely. (LVEMP) I hyacinth control. Limited achievements were Phase II of the program was approved in April 2009 and by realized in research on improving water quality and design has taken into account the challenges of Phase I pollution control. Pilot schemes in afforestation and (including potential intercountry conflicts over water usage watershed management were completed. An array and insufficient regional-level coordination) and the need to of scientific knowledge was amassed, mostly by follow through with interventions such as scaling up and local institutions and academia, enhancing local policy development. ownership. The strategic use of such knowledge during this first phase was lacking however. Plans to scale-up these pilots through feedback loops into practical policy and country-level action plans and investments were not developed. Regional-level coordination needs further strengthening. LVEMP II No outcomes yet as Phase II has just been approved. NA 124 A P P E N D I X H : E V A L U AT I N G T H E P E R F O R M A N C E O F R E G I O N A L E N V I R O N M E N TA L P R O J E C T S Program name Outcomes (achievements) Sustainability Aral Sea Water The main achievements of the program have been Sustainability of country-level achievements is likely, as are and Environmental in mobilizing support and defining actions needed the modest achievements at the regional level. Management to stabilize the environment around the Aral Sea. The program clarified what actions were needed for improved water management and improved dam safety, installed water monitoring stations, and restored the wetlands surrounding Lake Sudoche. But the program did not achieve much in the areas of water conservation and the translation of water management studies into action plans that would have a basin-wide impact, due in large part to weak country ownership of the problem assessments and their proposed solutions. Achievement of regional- level coordination was modest. Central Asia The project had mixed success in achieving its From the standpoint of sustainability of legislative achieve- Biodiversity objectives. It was successful in expanding the territory ments and establishment of protected area management under nature reserves and national parks and signifi- systems within countries, there is good likelihood of their cantly improving the management of protected areas. being sustained, and prospects for financial support for Numbers of some endangered species were restored national-level activities look good. Sustainability of the most to or exceeded safe levels. Good regional cooperation important biodiversity benefits that depend on integrated for joint scientific work (research, training, and regional action does not look as promising. At the time of knowledge sharing) was achieved and management IEG's ICR review in 2007, neither the tripartite agreement to capacities strengthened within individual countries. establish the transboundary Western Tien Shan biosphere An ecosystems approach to conservation was adopted nor the accompanying regional coordination of management and a bioregional plan was developed for each pro- plans had been ratified. tected area using a common methodology to collect baseline data. But the outcome of conservation efforts in areas adjacent to protected areas is yet to become fully evident, as proposed changes in legislation are yet to be adopted by the countries. Latin America Actual gains in biodiversity, carbon sequestration, The project was primarily a measurement and demonstration Land Use and other environmental benefits using the payment intervention. The demonstration effect was powerful and for environmental services (PES) scheme, even with even cattle ranchers are persuaded about the profitability of cattle-ranching farms (instead of adopting land environmental conservation activities on a farm. Even if conservation, the project proved cattle ranching could external financing incentives are limited, stakeholders are coexist with modified silvopastoral farming and likely to continue environmental activities and scale them up. environmental gains). Rigorous measurement and Colombia and Nicaragua are moving forward with follow-up documentation proved viability of the model and projects. In Costa Rica, payments for environmental services concept. Lessons were easily extracted and will be absorbed into and financed by the national forestry disseminated through regional and international financing fund. forums and research publications. (Table continues on the following page.) 125 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table H.3: Regional Projects: Outcomes and Sustainability (continued) Program name Outcomes (achievements) Sustainability Guarani Aquifer (SAG) At the time of IEG's case study in 2006, the NA, since an ICR has not yet been issued. project was delayed to the point that the overall project goal or even the four sub- objectives could not be achieved (PAD closing date was 2006). Progress on the substantial body of scientific work had been limited due to delays in negotiating the large contracts. The project finally closed in January 2009, and an ICR is expected soon. Source: IEG 2007d. NA = Not applicable. 126 A P P E N D I X H : E V A L U AT I N G T H E P E R F O R M A N C E O F R E G I O N A L E N V I R O N M E N TA L P R O J E C T S Table H.4: Regional Projects: Monitoring and Evaluation Program name Monitoring and evaluation Mediterranean · As a program with multiple phases, METAP undergoes a formal exercise to assess its efficacy before each re- Environment plenishment cycle. The evaluation covers program activities, individual technical assistance projects, and in (METAP) some instances specific pillars of support such as solid waste management and environmental impact assess- ments · Overall monitoring by program management is weak · The Secretariat's reporting to stakeholders on program activities is of variable quality and consistency · Not all completed projects are evaluated, but some projects (Regional Solid Waste Management Project) have good monitorable performance indicators and conduct external midterm and final reviews Eastern Caribbean · Weak M&E system; performance indicators developed only at ICR stage Waste Management · Project health objective was not tracked · At the country level, a monitoring system for the solid waste management component was developed (by external consultants) but it was not owned or used Lake Victoria · Complex project with large number of components and indicators and an inadequate logical framework Environmental · Inconsistent reporting format across countries despite project capacity-building efforts aimed at creating a Management (LVEMP) I harmonized M&E framework · Absence of historical data compounded project difficulty in measuring impact on communities (focus of first phase was on developing a knowledge base about the lake situation) · Good country ownership of project data sources because project used in-country resources (academia, research groups, civil society organizations) to conduct studies, and so on · Feedback loops from implementation experience not well linked to policy making LVEMP II · Major improvement in M&E design as lessons from Phase I are applied to Phase II · Project remains complex with large numbers of components, but Phase I studies and large amount of data collected have improved selectivity and identification of indicators · M&E framework is more harmonized and facilitates reporting across countries · Feedback loops are designed to be linked to policy making at country and regional levels Aral Sea Water and · Inadequate M&E framework was largely due to project complexity and the broad, poorly defined project Environmental objectives Management · Targets and monitoring indicators were not quantified, were set in general terms, and largely measured outputs · National-level activities enjoyed better-quality M&E · Lessons from implementation experience were applied in design of new country-level projects · Difficulty linking and aggregating results of national-level projects with regional benefits Central Asia · Project M&E lacked specific targets and was overly complex, leading to limited use during project implementation Biodiversity · In particular, lack of specificity in performance indicators hindered overall project monitoring and demonstration of outcomes · The M&E system made no provisions for collection of baseline data, though for some specific interventions the use of state-of­the-art scientific methods (for example, IUCN scorecard method for biodiversity), pertinent to performance measurement of the management of protected areas, enabled project demonstration of the restoration or increase in several plant and animal species · Thus, the impact on other objectives, such as improved livelihoods of communities, could not be demonstrated (Table continues on the following page.) 127 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Table H.4: Regional Projects: Monitoring and Evaluation (continued) Program name Monitoring and evaluation Latin America Land Use · M&E system was exemplary in this pilot project's design, implementation, and utilization · Manageable project size helped achieve positive results, enabled proof of concept that cattle ranching could coexist with ecomanagement · Project created a baseline documenting every element of the ecosystem that was included in the piloted areas · Good selection of indicators that were highly prioritized, measurable, and finite · Methodology of analysis and sampling frequencies was laid out clearly in the M&E framework · Project M&E demonstrated clear results in farming--productivity increases persuaded farmer associations to accept increased tree coverage as complementary to their lifestyle · Detailed documentation by M&E system of project experience has enabled widespread dissemination of project concept and experience, including international forums Guarani Aquifer (SAG) · Inadequate M&E framework was largely due to project complexity and the broad, ill-defined project objectives · A large part of the project depended on the gathering of scientific and technical knowledge and data before subsequent management and other intervention measures can be designed · Serious delays in the scientific and technical studies that had been contracted out affected the quality and implementation of the M&E system · Indicators selected were numerous (more than 65) and were mostly process indicators, while impact indicators were not quantified 128 APPENDIX I: EVALUATING THE PERFORMANCE OF GLOBAL ENVIRONMENTAL PROGRAMS Evaluations of global partnerships are primarily the historical inputs, outputs, and outcomes of the responsibility of the governing bodies of these the program. Although many of the evaluations programs, which typically commission external were able to document notable achievements of evaluations every five years or so. In cases where selected activities at the output level, none found the Bank sits on the governing body or provides a much systematic evidence of the achievements financial contribution, the Bank's representative of the programs' objectives at the outcome level. on the governing body and the task team leader also have oversight responsibilities--analogous to The Critical Ecosystem Partnership Fund Bank supervision of an investment project-- (CEPF) among other things, to advocate for sound CEPF is a global partnership that provides grants to monitoring and periodic independent evalua- NGOs and other private sector partners to protect tions. IEG began conducting Global Program critical ecosystems (ecological hotspots) in Reviews (GPRs) of programs based on these developing countries. The partnership is housed evaluations in 2006. in Conservation International and includes Conservation International, the World Bank, the IEG has conducted 14 global program reviews since Global Environment Facility, the MacArthur it initiated this new product, including reviews of Foundation, the government of Japan, and the three environmental programs--the Critical French Development Agency. During its first phase Ecosystem Partnership Fund, the Global Water (December 2000 to June 2007), CEPF awarded Partnership, and the Global Invasive Species approximately 1,000 grants to more than 600 Program. Overall, IEG has found that the independ- NGOs, community groups, and private sector ence of the programs' external evaluations has organizations in 33 countries. Averaging $16­17 been improving, but the quality of most evaluations million a year, CEPF grants have supported has been compromised by a weak M&E system: expansion and enhanced management of protected areas; promotion of alternative sustain- · Either the objectives and strategies of the pro- able livelihoods in production landscapes; and gram have not been well defined (not focused, environmental education, awareness, and capacity too process-oriented, difficult to measure, or building. open to different interpretations by different stakeholders), or An external evaluation conducted in the fall of 2005 · The M&E system has not been well designed found that CEPF had made strong progress during (focusing only on inputs and outputs, and not its first five years. However, there was some outcomes), or variation in the performance of individual · Data on the progress of activities and the hotspots, and the scope of the evaluation was achievements or outcomes have not been sys- limited by a lack of assessment of individual grant tematically collected. outcomes. As of March 2007, CEPF had con- tributed to the creation or expansion of 9.4 million As a result, most evaluations have had to spend hectares of protected areas in 15 countries, and additional resources attempting to reconstruct CEPF activities in protected area buffer zones and 129 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 production landscapes, for the most part, were ment, and broader development concerns achieving their environmental objectives. How- related to IWRM. While there were some notable ever, there is a need to better understand how advances in moving countries toward more these interventions are affecting the livelihoods of integrated management of their water re- the people living in these areas. The major sources--particularly when supported by partners have recently approved a second phase earmarked bilateral funding--attribution to the for the program (IEG 2007b). GWP is difficult to verify. It also has proved difficult to influence country development The Global Water Partnership (GWP) dialogue, particularly for PRSPs; to increase The Global Water Partnership was formed in 1996 budgetary allocation to IWRM activities; and and became an independent intergovernmental to facilitate grassroots implementation of organization, based in Stockholm, in 2002. It IWRM principles through country partnerships. supports countries in the sustainable manage- Engagement in transboundary water resource ment and development of their water resources issues--its comparative advantage--was frag- through a worldwide network of 1,800 members mentary and difficult. And GWP's support for formed into 71 Country Water Partnerships and countries and regions was judged to be at the 12 Regional Water Partnerships. The GWP expense of leading global policy advocacy and includes all aspects of water management and is a attention to redressing poor internal gover - working partnership among all those involved in nance that hindered its effectiveness. water management: government and NGOs, public institutions, private companies, profes- The Bank played a major role in convening and sional organizations, multilateral development providing legitimacy to the GWP and benefited agencies, and others. The primary focus is the from the relationship, but has lost interest in the development of country and river basin integrated last five years. GWP's advocacy prepared the water resources management. The number of ground for the Bank's high-profile Water donors steadily increased from 3 in 1996 to 12 in Resources Sector Strategy of 2003 (World Bank 2008, and over the same period GWP's annual 2004b), and its sponsorship with the World Water grant expenditure rose from $0.7 million to $16.5 Council for the World Panel on Financing Water million. Three donors--the United Kingdom, Infrastructure led to the Camdessus Report Sweden, and the Netherlands--have provided (Camdessus 2003), which provided the rationale more than two-thirds of GWP's total income. Total in 2004 for the Bank's Municipal Fund, which Bank contributions were $5.7 million, 18 percent enabled subsovereign lending. of GWP's grant income to 2002, when the Bank funding ceased. Since then, however, the Bank's participation in the GWP has declined rapidly as the Bank has An independent evaluation of the GWP's become more focused on the Millennium performance in 2008 found that the GWP Development Goals and water supply and sanita- retained its clear niche in facilitating multistake- tion, and the GWP is no longer listed on the holder dialogue on integrated water resources Bank's Web site as one of its ongoing global water management (IWRM). The major growth area of partnerships. Concern for the poor governance the GWP was at the country level and in the of the GWP evaporated with cessation of Bank maturing of country partnerships. Even so, it has financing. Almost 80 percent of Bank water staff been slow to move dialogue beyond water- responding to an IEG questionnaire saw GWP's focused organizations, which runs counter to benefit to Bank operations as modest or worse, the comprehensive and integrated approach or did not know. And only 4 of 1,864 projects advocated by the GWP . approved by the Bank between fiscal 1997 and 2007 referred to the GWP. Interest in the GWP The evaluation recommended that GWP refocus among Bank staff has been further eclipsed by on the cross-cutting themes of water, environ- the rise of the Bank-Netherlands Water Partner- 130 A P P E N D I X I : E V A L U AT I N G T H E P E R F O R M A N C E O F G L O B A L E N V I R O N M E N TA L P R O G R A M S ship Program, which provides direct support to the Bank's commitment at the World Summit on the Bank's ESW and TA. Thus, unless there are Sustainable Development in Johannesburg in incentives, as in the case of the GEF, it appears to August 2002, and the Bank's Development Grant be extremely difficult to foster linkages between Facility contribution represented a significant the GWP and the Bank's country operations (IEG portion of the program's financing during fiscal forthcoming a). years 2004­06. However, the Bank has only modestly addressed the issue of IAS in its The Global Invasive Species Program operations. With a few exceptions, links to (GISP) country operations are weak. Moreover, the IBRD GISP is an independent nonprofit association does not have an operational policy on IAS, nor whose mission is to conserve biodiversity and are IAS specifically referred to in any of the sustain human livelihoods by minimizing the existing operational policies, such as Environ- spread and impact of invasive alien species (IAS). mental Assessment (OP 4.01), Natural Habitats It is located in the Centre for Agriculture and (OP 4.04), and Forestry (OP 4.36). Biosciences International (CABI) in Nairobi, Kenya. GISP was supported for three years (fiscal IEG found that guidance and training are needed 2003­06) by the World Bank with funds made for Bank task managers working in fields where available through the Development Grant Facility development interventions may pose potential and continues to receive Bank support through IAS threats. At the same time, research and the Bank-Netherlands Partnership Program. At analysis is needed to identify and prioritize IAS the request of the World Bank, an external evalua- threats and related cost-effective mitigation tion was commissioned by the GISP Secretariat in and/or eradication strategies. The Bank could the spring of 2006, and IEG followed up with a consider supporting research on what the litera- Global Program Review in 2008. ture has identified as a clear need to improve the basis on which IAS control strategies are The World Bank leveraged both its reputation and evaluated and on which the potential impacts of financial resources to support the creation of a species introductions are valued (since the GISP Secretariat in South Africa. The director of valuation of costs and benefits of control options the Bank's Environment Department announced is still quite rudimentary) (IEG 2009d). 131 APPENDIX J: THE RESULTS MEASUREMENT SYSTEM UNDER IDA: A REVIEW This appendix offers a stocktaking of implemen- others 2002) issued by the heads of the five tation of the IDA Results Measurement System multilateral development banks, following the (RMS), with special reference to the RMS Monterrey Conference on Financing for Develop- recommendations of the IDA15 Agreement. A ment. The Second International Roundtable on formal review will take place during the IDA15 Managing for Development Results, held in Midterm Review in the autumn of 2009.1 Marrakech, Morocco, in February 2004, gave additional international emphasis to results The RMS has significant implications for strength- measurement and use. ening statistical capacity and a results orientation not only in the Bank, but also in borrowing On July 1, 2002, the Final Deputies' Report on the countries, as well as among other development IDA13 Replenishment negotiations called for a agencies and donors. This appendix outlines the "system to measure, monitor, and manage for origins of the RMS in IDA13 and its evolution development results" and cited the Millennium through IDA14 to its current state under IDA15. Development Goals as a point of reference. The The appendix assesses how the Bank is approach- focus of the deputies' recommendation was on a ing the implementation of IDA RMS recommen- country-based system "capable of aggregating dations, including early indicators of progress, across country programs in order to report and key constraints to progress, and the use of the assess IDA-wide results" (IDA 2002, pp. 7­8). The RMS system to improve development outcomes. results measurement requirements implied by the requests of the IDA deputies have become progres- Sources include documents dealing with the sively more detailed from IDA13 through IDA15. RMS produced by IDA replenishment negotia- tions, as well as by Bank operations and evalua- Based on experience under IDA13 and respond- tion management and staff. Interviews with Bank ing to IDA deputies' requests, at the beginning of staff have also provided important insights, as the IDA14 period, July 1, 2005, Bank manage- have responses to a questionnaire by seven IDA ment proposed an RMS with three components: donors. Tier 1: Monitor country outcomes Background (14 indicators in 4 categories)2 The RMS has its origins in the 1990s. During those years, there was growing demand in the interna- Growth and poverty reduction tional development community to demonstrate · GDP per capita (constant year 2000 US$) development results. In the Bank, this was · Population below $1 a day (percent). reflected in the Country Development Frame- work of early 1999; results orientation (or results Public financial management and investment focus) was one of its four pillars. This was climate reinforced by a statement in March 2002 on · Public financial management (number of heav- "Measuring, Monitoring, and Managing for ily indebted poor country [HIPC] benchmarks Development Results" (AfDB, ADB, EBRD, and set) 133 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 · Cost required for business startup (percentage · Seven in water supply and sanitation (for ex- of gross national income per capita) ample, beneficiaries from water supply or san- · Time required for business startup (days). itation interventions) · Seven in transport (for example, total kilome- Infrastructure for development ters of roadway built). · Access to improved water source (percentage of population) Strengthening Statistical Capacity · Fixed line and mobile phone subscribers (per The IDA14 Agreement recommended that "in 1,000 people) addition to continuing support for the PARIS21 · Access to an all-season road (percentage of consortium, IDA should intensify support within rural population) its CASs and projects for national statistical · Household electrification rate (percentage of capacity building, and work in partnership to households). implement the global action plan to strengthen statistical systems, including household surveys."5 Human development · Under-5 mortality rate (per 1,000) The IDA14 Midterm Review Report (IDA 2006) · Prevalence of HIV/AIDS (percent adult popu- devotes a chapter to "Progress in Statistical lation ages 15­49) Capacity Building" and mentions Bank and other · Births attended by skilled health staff (percent donor support for the Trust Fund for Statistical of total) Capacity Building (TFSCB), the STATCAP multi- · Primary completion rate, total (percent of rel- country lending program, and support for the evant age group) Marrakech Action Plan for Statistics (MAPS). The · Ratio of girls to boys in primary and secondary Bank's support for MAPS has included assistance education (percent). for the development of national strategies for the development of statistics as well as for an Tier 2: Monitor IDA's contribution to Accelerated Data Program for Africa (IDA 2004, country outcomes (nine indicators in p. 17­18). three categories) The IDA15 RMS Recommendations Country level Based on experience with the RMS under IDA14, · Number of Results-Based CASs. the final report of the IDA15 Replenishment Meeting, held in December 2007, asked Bank Project level management to: · Percentage of operations with satisfactory qual- ity at entry 1. Strengthen Tier 1 of the RMS by reviewing the · Percentage of Implementation Status and Re- private sector development indicators and ex- sults Reports (ISRs) with satisfactory outcome ploring the feasibility of developing a public ex- baseline data3 penditure and financial accountability­based · Percentage of projects with satisfactory out- aggregate indicator to measure the quality of come ratings public financial management in IDA countries. · Percentage of projects with satisfactory Im- 2. Strengthen Tier 2: plementation Completion Report (ICR) quality. · Move from monitoring the number of CASs to monitoring their quality. Project outputs (example output indicators):4 · Improve the availability of baselines for key · Five indicators in health (for example, health outcome indicators in IDA projects. professionals trained) · Improve the quality of output and outcome · Six in education (for example, classrooms built reporting in Implementation Completion or rehabilitated) Reports for IDA operations. 134 A P P E N D I X J : T H E R E S U LT S M E A S U R E M E N T S Y S T E M U N D E R I D A : A R E V I E W · Improve the quality of outcome indicators IDA15 Recommendations and Their by developing lists of indicators for four- Implementation Status to-five sectors by the IDA15 midterm re- The IDA15 Agreement made specific recommen- view to allow a more accurate indicative dations, building on the system established aggregation of sector-specific outputs. during IDA14. Table J.1 (at the end of this 3. Strengthen statistical capacity in IDA countries: appendix) juxtaposes each recommendation · Monitor statistical capacity in line with part- with an assessment of the current status of ner country priorities during IDA15. implementation. The overall picture is one of a · Develop a data standards system to help work in progress. The Bank continues to make IDA countries track their progress in build- progress at the level of Results-Based Country ing statistical capacity. Assistance Strategies (RBCASs), but delineation · Collaborate with donors to strengthen sta- of outcomes and outputs at the project level is tistical capacity. problematic. Quality rather than quantity has become a central issue for RBCASs and baseline In support of this, management was urged to: indicators. Weak statistical capacity at the · Include a more comprehensive discussion on country level limits the availability of Tier 1 country statistical systems in IDA CASs. indicators. The use of results information for · Follow up on progress made. decision making is still in a nascent, pilot stage. Overall Accomplishments RMS Constraints and Challenges To facilitate a systematic approach to the RMS, a A number of constraints and challenges face the Bank-wide Results Steering Group was es- Bank in implementing each element of the RMS. tablished under IDA14 with members from the Five sets of issues are delineated below: Operations Policy and Country Services Vice measurement, aggregation, expansion, country Presidency (OPCS), the Regions, networks, and statistical capacity, and utilization. Development Economics (DEC), which provides statistical and data support. OPCS provides the Measurement: Clarity, Availability and Quality Secretariat for the RSG. While the main emphasis of Outcomes, Outputs, and Indicators has been on project results, as noted previously, As illustrated in box J.1, the distinction between the IDA agreements also proposed a country- outcome and output levels is not always clear in based system to "measure, monitor, and manage either Tier 1 or Tier 2. Baseline indicators are for results" and envisioned a system "capable of typically shown at the ISR phase rather than at cross-country aggregation." In response to this project entry, and they do not always relate well proposal, the Bank has pursued two approaches: to project objectives or expected outcomes. developing a set of core indicators that could be Definitions are relatively clear for Tier 1 indica- used for projects of a given type and developing tors, but they still leave room for differences in a method to enable aggregation of project- interpretation that hinder aggregation and specific results to the country and Bank-wide comparability among countries. Sheer availabil- level for reporting to stakeholders. ity is a serious problem with some indicators that also constrains comparability. During IDA14, 14 Tier 1 indicators for four broad areas (growth and poverty reduction, governance Aggregation and private sector development, infrastructure, There are two dimensions to the challenge of and human development) and 9 Tier 2 indicators aggregation: within countries and among coun- were posted on the internal and external Bank tries. The intention of the Bank, through the Web sites. But timely coverage for a number of Results Secretariat, is to establish results-based countries and indicators (for example, access to indicators at the sector level in order to provide an all-season rural road) is lacking. a link between the project and country levels. 135 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Box J.1: Issues Regarding Outcomes, Outputs, and Indicators · Some indicators that could be viewed as outputs are in- · However, without a reliable census there is an element of sub- cluded in Tier 1 outcome indicators, such as births attended jective judgment regarding number of people within 2 kilo- by skilled health staff. An indicator that is arguably an meters of an all-season road. Definitions of "motorable" and outcome--jobs created--is treated as an output in Tier 2. "short duration" may also vary from country to country. · Indicators are defined relatively clearly. For example, the · A more serious problem is the sheer lack of data. For exam- percentage of the rural population with access to an all- ple, access to an all-season rural road is not reported at all season roads is defined as: for a number of IDA countries for the period from 1990 to 2008, and when it is, the indicator is typically reported only for one The number of rural people who live within 2 km (typ- year. For IDA countries as a whole, an indicator (38 percent) ically equivalent to a 20-minute walk) of an all-season is shown only for 2003. Other Tier 1 indicators showing only road as a proportion of the total rural population. An one or two data points for IDA as a whole include: poverty all-season road is a road that is motorable all year by head count, public financial management (measured by num- the prevailing means of rural transport (often a pick- ber of HIPC benchmarks met), and prevalence of HIV. There up or truck that does not have four-wheel drive). Pre- are no data points at all shown for ratio of girls to boys in pri- dictable interruptions of short duration during mary and secondary education. inclement weather (e.g., heavy rainfall) are accepted, particularly on low-volume roads. Note: Definitions of results chain terms such as outputs and outcomes vary. They are often defined by examples and by their relationship to other terms in a results chain or logical framework. IEG cites the OECD Glossary of Key Terms in Evaluation and Results Based Management (2002), which contains brief definitions of these terms. The Operations Policy and Country Services Vice Presidency of the Bank makes the following distinction in its guidance: The key distinction between an output (a specific good or service) and an outcome is that an output typically is a change in the supply of goods and services (or de- liverables), while an outcome reflects changes in the utilization of goods and services (demand side) . . . by beneficiaries. [World Bank 2007b] In the examples cited in this box, skilled health staff would arguably be seen as a supply of deliverables or outputs from the perspective of beneficiaries, whereas jobs created would be seen as a demand for services, or an outcome from the perspective of worker beneficiaries. This effort is at an early stage. However, IDA of accountability and can serve as a guide to donors feel that project indicators need to be overall resource allocation. But monitoring found that can be better aggregated to show project outputs and outcomes is essential for country performance. effective project management. While the Bank's focus has been on the country- Expanding to Other Sectors, Themes, based development approach, there is interest Situations, and Modes among IDA deputies in aggregation across The Results Secretariat is currently working with countries. Such comparisons are problematic, Bank networks to expand the RMS from four owing to differing definitions and timeframes. sectors to all sectors and themes, anchoring There is also a potential tension between the indicators in an electronic database at the ISR RMS objective of identifying a selected number stage. This is an ambitious and complex of project outcome indicators that can be undertaking. It will be particularly challenging in aggregated within a sector and the need to some thematic areas, such as environment, measure project outputs and outcomes even if where output and outcome indicators are they do not correspond to RMS indicators (for diverse and vary widely in availability. The example, for a tertiary education project). Both piloting of the Country Program Results Monitor- approaches have complementary rationales. The ing Tool, as well as the CAPScan pilot diagnostic aggregation of outcomes provides an indication reviews of country statistical capacity, are 136 A P P E N D I X J : T H E R E S U LT S M E A S U R E M E N T S Y S T E M U N D E R I D A : A R E V I E W important initiatives, but they need to be Gaps and Weaknesses in Country carefully monitored and evaluated for lessons Statistical Capacity learned. Efforts to strengthen statistical capacity are slowly gathering momentum, with the new A major challenge for the RMS is extension to Statistics for Results Facility Catalytic Fund fragile states or fragile situations. The Bank constituting a potentially significant develop- believes a results-focused approach is possible in ment. The spotty availability of most of the 14 such situations, as conveyed in the first quote in Tier 1 indicators (GDP per capita is an exception) box J.2. IDA donors are looking to the Bank to during the 1990­2008 period is one manifesta- take the lead in developing results-based tion of weakness in statistical capacity. As noted approaches for fragile situations. in box J.3, even in the area of vital statistics, capacity in IDA countries is often weak. The Bank is attempting to include AAA in the RMS. The major challenge here is to measure IDA donors see the building of appropriate statis- outcomes as well as outputs. Bank guidance calls tical and M&E capacity--along with country will for the expected results of AAA to be specified at and ownership--as main constraints to building entry, but this is done only sporadically. One an effective RMS. They consequently continue to reason is the inherent difficulty of identifying place substantial importance on the Bank's with confidence the outcomes of a research performance in this area. or analytical paper, policy note, workshop, or training course. Yet it is argued that the value How Is the RMS Currently Being Used and added from these uses of the Bank's resources What Are the Plans for Its Future Utilization? should be assessed, just as they are for invest- Measuring for results is necessary but not ment operations. Another difficulty is that the sufficient for managing for results. While the results of AAA may become apparent only in the IDA15 agreement did not specifically refer to the long run, when it may be difficult to attribute a application of the RMS, the IDA13 agreement result to specific AAA. Finally, there is the envisioned a country-based system to "measure, challenge of aggregating the results of different monitor, and manage for results."6 Information kinds of AAA. on results is increasingly being used in the Bank's Box J.2: How Applicable Is the RMS to Fragile Situations? The Bank IDA RMS Web site envisages the possibility of a well depend on the degree of fragility. One recent study suggests results-based approach in fragile states: that in situations of extreme fragility marked by: Experience in several fragile states has demonstrated that Deteriorating governance, increasing risk of conflict, pro- even where capacity is low, information is limited, and risk longed impasse between national governments and the is high, it is possible to take a results-focused approach. international community, or ongoing violent conflict, the Donor harmonization and clarity of objectives are central model of development partnership [including results ori- in such settings, and a focus on building results man- entation] that the Paris Declaration envisages is unlikely to agement capabilities can bolster early efforts to reestab- apply. lish core institutional capacities. But approaches to these challenges must be feasible and adaptive to enable re- But even in this case, "harmonization [among donors] remains sponsiveness in an uncertain institutional setting. a key priority" [OPM /the IDL Group 2008]. Some IDA donors have flagged fragile states as a major chal- The scope for a results-based approach in fragile situations may lenge for the RMS and see the Bank as leading the way in devel- 137 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Box J.3: Civil Registration: A Missing Link in IDA Country Statistical Capacity For vital statistics on births, marriages, and deaths, it is gener- Annual civil registration is a third important element in a functional ally assumed that data are relatively timely and reliable, but system of vital statistics. This is taken for granted in industrial coun- they actually are often patchy and of questionable reliability in tries but is often absent and neglected in IDA countries. The Bank is IDA countries. In the absence of more reliable and frequently re- helping the development community find solutions to strengthen and ported data, the Bank and other donors tend to rely on donor- expand civil registration efforts that are under way in several coun- financed surveys and other estimation methods. But such surveys tries. For example, it sponsored a meeting on Strengthening Civil are often ad hoc and do little to build the institutional capacity Registration: Improving Vital Statistics for Evidence-Based Decisions of partner countries. Censuses provide a snapshot of such in- in June 2009 to learn from efforts to establish civil registration systems formation at 10-year or longer intervals. in Cambodia, Ghana, Mozambique, Sri Lanka, and other countries. main planning document, the CAS. But little steeped in an approvals-based culture rather attention has been given to using RMS data than one based on performance by results. beyond the CAS for such purposes as budget allocation or reallocation, staffing decisions, and Four surveyed IDA donors view the Bank as a the like. leader in the field of results measurement. But three expect a deeper institutionalization in the Donor Perspectives7 Bank of managing for development results. They Three of seven IDA donors surveyed indicated envision a management tool that would feed into that they would like to see the Bank move the Bank's planning and evaluation processes, real- beyond measurement to using results as a time project results disclosure, and ultimately management tool, not only for strategic planning aggregation of project results. Two respondents but also for budget priorities, staffing, and also encourage the continued efforts of multilat- performance evaluation. One of these donors eral development banks to converge on standard saw the Bank and its partner countries still country- and project-level performance indicators. Table J.1: The Bank's Progress in Implementing IDA15 Recommendations Recommendation Status of implementation Tier 1 (1) Review private sector development Since IDA14, the RMS has included two PSD indicators drawn from the Doing (PSD) indicators Business survey (cost and time required for business startup). No additional review has been reported. Tier 1 (2) Explore development of a public Aggregation of the six PEFA component indicators is viewed as conceptually expenditure and financial accountability difficult and of little value to decision making or making country comparisons. (PEFA) aggregate indicator Tier 2 (1) Monitor quality, not just number of All new CASs are results-based, and their quality is monitored in CAS Comple- results-based CASs tion Reports and CASCR Reviews done by IEG. Treatment of results is now more uniform and evidence-based, but outcomes tend to lack realism and intermedi- ate milestones. Due to difficulty of getting donor agreement on objectives, joint RBCASs tend to be weaker. The financial crisis is resulting in two-year indica- tive CASs with weak results orientation. 138 A P P E N D I X J : T H E R E S U LT S M E A S U R E M E N T S Y S T E M U N D E R I D A : A R E V I E W Table J.1: The Bank's Progress in Implementing IDA15 Recommendations (continued) Recommendation Status of implementation Tier 2 (2) Improve availability of key outcome Availability and quality of baselines is improving, but gaps remain. The Bank indicator baselines reports that more than 80 percent of 131 fiscal 2006­07 ISRs reviewed had at least one outcome indicator with baseline data, and 53 percent had outcome baseline data for all indicators. But only 63 percent of ISRs included baselines that were judged to have indicators of satisfactory quality. Tier 2 (3) Improve quality of output and outcome IEG assessments of M&E report an improvement in M&E quality, which reporting in ICRs included output and outcome reporting, from fiscal 2006 to 2007, but a plateau in fiscal 2008. Tier 2 (4) Develop lists of indicators for four or five Based on the Africa Results Monitoring System, the RMS is being extended to sectors by the IDA15 Midterm Review to all sectors and themes, and the system will be anchored in a database at the improve outcome indicator quality and ISR level. Lists of indicators are being developed for all key sectors and themes allow more accurate aggregation of in order to improve outcome quality and aggregation of sector outputs. Ten IDA sector outputs countries are piloting a Country Program Results Monitoring Tool.a These are ambitious efforts. The incorporation of AAA is also being considered but raises thorny issues of outcome and attribution identification. Strength- Monitor statistical capacity in line with Statistical capacity is being monitored by the Bank's Development Economics ening partner-country priorities Data Group unit in line with partner-country priorities. statistical capacity (1) Strength- Develop a data standards system to help The Bank encourages progress toward data standards by co-chairing the ening countries track their progress PARIS21 Consortium and will launch an Internet-based tool to enable countries statistical to monitor their progress in strengthening statistical capacity. capacity (2) Strength- Collaborate with donors to strengthen With other donors, the Bank helped create a Statistics for Results Facility ening statistical capacity, including expansion Catalytic Fund to supervise, finance, and cofinance larger investments in statistical of the Trust Fund for Statistical Capacity statistical capacity. capacity (3) Building The DEC Data Group collects gender-disaggregated data whenever available Work toward the collection and analysis and feasible. The Bank's Gender Group is working on an improved approach to of gender-disaggregated statistics at the the collection and analysis of gender-disaggregated data under IDA16. country level Strength- Include more comprehensive treatment of A CAS guidance note and online learning tool are being developed for Regional ening statistical systems in CASs staff. statistical capacity (5) Source: IEG interviews and analysis. a. Burkina Faso, Burundi, Democratic Republic of Congo, Mali, Mauritania, Moldova, Mongolia, Nigeria, Rwanda, and Zambia. 139 ENDNOTES Chapter 1 is the estimated coefficient on the year variable from this 1. This shift has been widely documented. See, for regression. Average performance of projects by sector example, the World Bank Group 2008. and Region is estimated using the coefficients on the sec- tor and Region dummy variables. Chapter 2 6. After the fall of the Soviet Union, Bank support to 1. More specifically, IEG rates the outcome of a proj- transition economies grew rapidly in the 1990s. Recent ect highly satisfactory if there were no shortcomings in years have seen a decline in Bank support, as a num- the operation's achievement of its objectives, in its ef- ber of Eastern European countries have graduated from ficiency, or in its relevance. Satisfactory projects have only the Bank. Nonetheless, there is, in total, more lending minor shortcomings in relevance, efficacy, or efficiency; to countries in this region than was the case in 1993. moderately satisfactory projects have moderate short- 7. Note that whether a sector is a low or high per- comings; moderately unsatisfactory projects have sig- former is measured by its mean performance over the nificant shortcomings; unsatisfactory projects have full period 1993­2008 from the regression; no further major shortcomings; and highly unsatisfactory projects adjustment is made for annual trends within this period. have severe shortcomings. 8. This is calculated by dividing the total sector + Re- 2. Fiscal year 2008 runs from July 1, 2007, through gion + development policy lending effect (0.90) by the June 30, 2008. The Implementation Completion Re- combined composition and time effects (0.09 + 0.316). ports are issued within six months of project closing, and An alternative approach using the same calculation but are then reviewed and rated by IEG. omitting extreme outliers in project ratings yields an es- 3. There is a possible bias in the overall evaluation timate of 35 percent. Incorporating log GDP of the of project ratings because not all projects that exited in country to the regression and recalculating yields an es- fiscal 2008 were rated by the time this report was pre- timate of 32 percent. This analysis suggests that a small pared, whereas data for fiscal years 2007 and 2006 re- amount of the improvement in project performance over flect the full sample of projects. However, adjusting for time is because Bank client countries where slightly this does not significantly change the reported trends. richer in 2008 than in 1993. The percents satisfactory with the adjusted sample are 9. See chapter 8 in Kapur, Lewis, and Webb 1997. 82.6 (2006), 75.8 (2007), and 81.0 (2008). 10. Note, however, that the Bank did not work ex- 4. This calculation is based on five-year moving av- tensively in the Region before 1989, so the ability to com- erages of the performance data, which is why the last pare pre- and post-1989 results is limited. year shown is 2006. Some argue that crossing the hur- 11. IEG 2008e. Unless otherwise noted, country ex- dle into satisfactory is an important threshold, and that amples in this section of the evaluation are taken from increments above moderately satisfactory are less so, but the same report. this is not self-evident. 5. The regression uses the project ratings on a six- Chapter 3 point scale as the dependent variable. The sample in- 1. Before 2004, the Bank had required logframes in cludes all projects exiting between 1993 and 2007. their projects. The results framework places greater em- Independent variables are an annual time trend and sec- phasis on the outcomes, differentiating between the tor, Region, and development policy lending dummy final outcomes (the project development objective) and variables. The 0.021 trend improvement cited in the text intermediate outcomes that could be used to monitor 141 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 progress toward achieving the objective. The logframe cality; therefore, outcome differences are likely attrib- included inputs and outputs as well as outcomes. uted to distinct initial conditions between treatment and 2. An M&E system rated "high" is expected to strongly control localities. influence project performance, provide sufficient in- 10. A sixth project was also available, but all the doc- formation to satisfactorily assess the stated project ob- umentation was in the local language and could not be jectives, and contribute to testing the underlying included in this assessment. development model (or results chain). Conversely, a sys- 11. A rating at the project's midpoint is here de- tem rated negligible would have many weaknesses, fined as an ISR completed between the 40th and 50th have very little impact on the project or program, be in- percentiles of a project's lifespan. sufficient to satisfactorily assess the stated project ob- 12. In fiscal 2006, the Bank took several steps to en- jectives, and contribute little to testing the development courage restructuring projects when necessary, in- model. cluding allowing some restructuring (that is, major and 3. Coefficient = .324, t=8.98. significant changes that do not affect the project de- 4. See appendix J, endnote 7, for a list of the re- velopment objectives or associated outcome targets) sponding donors. These donors accounted for 48 per- with approval from a vice president rather than the cent of cumulative IDA subscriptions and contributions Board. as of June 30, 2007. 13. See Operational Policy 10.04 in the Operations 5. Impact evaluation, like all analytical techniques, Manual: "For every investment project, the Bank staff should be applied where it is best suited. Deciding conduct economic analysis to determine whether the when and where to pursue impact evaluations is an project creates more net benefits to the economy than important continuing agenda, which the Bank is pur- other mutually exclusive options for the use of the re- suing in part through the Development Impact Evalu- sources in question." The policy further states that this ation Initiative. Most observers agree that the scope should be assessed using the positive net present value for impact evaluations is much greater than currently criterion. practiced, but the technique will never be appropriate for all projects. Chapter 4 6. The total number of projects opened after fiscal 1. The Reducing Emissions from Deforestation and year 2001 exceeded 44, but the criteria for selection Degradation (REDD) Initiative was officially launched (closed by December 2008) limited the sample to 44. at the United Nations Conference on Climate Change 7. Collecting good baseline data is a fundamental hall- in Bali in December 2007. The Intergovernmental Panel mark of good practice in monitoring, evaluation, and on Climate Change has estimated that 17 percent of even project design. Nevertheless, for certain special eval- global CO2 emissions are the result of human interac- uations--randomized controlled trials--it is not strictly tions with forests, thus providing the essential ration- necessary, since the random selection ensures that av- ale for the REDD initiative. erage characteristics of treatment and control groups, 2. World Bank Environment Department Web site: including initial conditions, are similar. And, of course, http://go.worldbank.org/ATD8DARR80 collection of baseline data in itself is no guarantee of a 3. IEG is currently undertaking a major study on the good evaluation. World Bank Group's experience with safeguards from 8. Other refers to evaluation efforts using inputs 1999 to 2008 that will be completed in fiscal year 2010. from various sources. Examples are participatory M&E 4. The peak levels of commitments in 2005 and 2008 stakeholder meeting reports, assessments of commu- are explained by large environmental development pol- nity satisfaction surveys for a subset of the project, and icy operations in Brazil, Colombia, and Mexico in 2005 field visit reports from operational staff. and in Mexico in 2008. The extent to which these op- 9. In this case, the counterfactual is inadequate be- erations will translate into public and private invest- cause treatment localities were purposively located in ments in environmental improvements is uncertain, the poorest area in the region, rendering significantly and their value for actual environmental improvements different socio-demographic and geographic charac- is extremely difficult to determine. teristics from any other locality. No attempts (such as 5. Bank support was to focus on cost-effective meas- econometric tools) were made to find a comparable lo- ures to prevent and reduce environmental health risks 142 E N D N OT E S through minimizing people's exposure to indoor and evance (Salmi 2006). But there has been no systematic urban pollution, waterborne diseases, and toxic chemicals assessment of the outcomes of PROFOR's work in terms along with providing access to cleaner fuels and improved of improved forest policies, incentive frameworks, for- design of water and sanitation services. QAG findings at est practices, and investments at the country level. entry and supervision have also highlighted the absence 12. IEG would like to note that Bank management of framing environmental health concerns in the context appears aware of these needs and has recently begun of environmental planning and management. a training program to increase environmental knowledge 6. Accountability for health outcomes in water sup- and awareness of Bank managers. ply and sanitation projects also appears to have de- 13. Management notes that internal incentives are clined since 2002. Water supply and sanitation projects not only measured by project commitment values. It approved since then are less likely to have been justi- should be noted that several of the small GEF energy- fied by health benefits, to have explicit health objectives, efficiency and GEF ODS phase-out projects received or to plan to collect health indicators compared with QAG "1" highly satisfactory ratings, and task team lead- projects approved in the previous five years (IEG 2009e, ers and task teams thus received strong positive feed- pp. 73­77). back and recognition. Bulgaria Energy Efficiency GEF and 7. Management notes that attribution of health im- Jordan ODS Phase-out II are two such examples. provements in improved access to water is very difficult and expensive to monitor. It is much easier to monitor Chapter 5 time savings on water collection. 1. Based on a review of the complete population of 8. Energy-Environment Reviews have been able to ad- ENRM projects approved since 2000 and closed in 2007 dress issues of fuel subsidization (in Egypt, Iran, and Mex- and 2008 and reviewed by IEG, there is a positive cor- ico), Strategic Environmental Analyses have been used relation between better M&E and better outcomes. The to assess hydropower options (in Laos, Nepal, and the correlation coefficient is 0.60. See appendix F. Nile Equatorial Lakes Region), and Country Environ- 2. The 2001 strategy (p. 81) committed the Envi- mental Analyses have analyzed energy-environment ronment Sector Board to (a) defining more precisely the linkages and energy efficiency (in Bangladesh, Belarus, monitoring, quality enhancement, and reporting re- Colombia, Egypt, India, Pakistan, and Serbia-Montene- sponsibilities; (b) reviewing exit strategies for existing gro). (IEG 2009a, pp. 30­31.) partnerships to ensure effective transition; (c) identify- 9. The Bank has used Poverty and Social Impact ing and catalyzing new partnerships that could con- Analyses to analyze the poverty and distributional im- tribute to the strategy's implementation; (d) setting pacts of energy pricing reform and possible changes in guidelines for improving governance, management, and social safety nets to offset these impacts. In Indonesia reporting; (e) ensuring that accurate records were kept and Ghana, for example, detailed Poverty and Social Im- on partnerships, their budgetary implications, and their pact Analyses and recommendations appear to have performance; and (f) initiating periodic reviews of port- shaped successful policies of price increases with com- folio performance. pensation, while such analyses appear to have been 3. Two biodiversity projects--the Mozambique less effective or evident in Bolivia, Ghana, and Yemen. Coastal and Marine Biodiversity Management Project and (IEG 2009a, pp. 49­50). the Latin America Integrated Silvopastoral Approaches 10. The lack of available data before 2002 does not to Ecosystem Management Project--conducted cost- permit a comparison with the 1995­2001 period like benefit analysis and cost-effectiveness analysis, respec- other comparisons made in this report. tively, in parallel to their incremental cost assessments. 11. PROFOR's outputs have included publications, 4. This is not to say that incrementality will not be knowledge tools, seminars, and technical workshops. considered in approving future GEF projects. Rather, the The 2006 Midterm Review of PROFOR (Salmi 2006) GEF evaluation (GEF 2007) found that incrementality found substantial demand for specific publications, in- of the GEF portfolio is achieved through the initial ne- cluding the highly regarded Forest Institutions in Tran- gotiations between the GEF agents and the recipients, sition: Experiences and Lessons from Eastern Europe in the determination of baselines, and in expected ben- (World Bank 2005), which brought knowledge to the re- efits rather than through the cost assessments that gion from credible external experts on issues of high rel- were being administered after project design. 143 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Appendix C erations; Development Results in Middle-Income Coun- 1. Levels of adoption ratings are: high (fully adopted); tries: An Evaluation of the World Bank's Support; and substantial (largely adopted but not yet fully incorpo- Using Training to Build Capacity for Development: rated into policy, strategy, or operations); medium An Evaluation of Project-based and WBI Training (see (adopted in some operational and policy work but not references). to a significant degree in key areas); and negligible (no 5. The recommendations that have not been ac- evidence or plan for adoption, or plans and actions for cepted by management will be excluded from the fol- adoption are in a very preliminary stage). lowing data analysis. 2. The categories for the implementation status of 6. As mentioned in endnote 5, the five new recom- recommendations are: active and remain actionable by mendations that have been only partly accepted by management, active and archived, complete and management will be excluded from the following data archived, incomplete and archived, obsolete or over- analysis. taken by events and archived, and difference of opin- 7. This excludes the two recommendations on which ion between management and IEG. there was a difference of opinion between IEG and 3. The following reports were submitted to the Board management. These did not receive an adoption rating by IEG in calendar year 2008: Public Sector Reform: and are excluded from the sample when comparing What Works and Why? An Evaluation of World Bank levels of adoption. Support; Doing Business: An Independent Evaluation-- Taking the Measure of the World Bank-IFC Doing Busi- Appendix J ness Indicators; World Bank Group Guarantee 1. Further information on interviews, documents, and Instruments 1990­2007--An Independent Evaluation; questionnaires may be obtained from IEG upon re- Environmental Sustainability--An Evaluation of World quest. Bank Group Support; Decentralization in Client Coun- 2. These indicators were selected on the basis of dis- tries: An Evaluation of World Bank Support 1990­2007; cussion by IDA deputies from 18 possible country out- Climate Change and the World Bank Group, Phase I: come indicators proposed in a paper prepared by Bank An Evaluation of World Bank Win-Win Energy Policy management (IDA 2004, p. 6, table 1). Reforms; and Using Knowledge to Improve Development 3. This indicator was introduced as an additional Effectiveness--An Evaluation of World Bank Economic project-level indicator during IDA14. and Sector Work and Technical Assistance, 2000­06 4. These indicators were introduced during IDA14. (see references). The original IDA14 recommendation was that "aggre- 4. The following IEG reports were submitted to the gate progress" be monitored on "selected project out- Board during calendar year 2005 and 2006: Financial puts . . . in four sectors where IDA is active across a range Sector Assessment Program--IEG Review of the Joint of countries" (IDA 2004, p. 13). Twenty-five indicators World Bank and IMF Initiative; Hazards of Nature, were delineated in IDA 2006 (p. 14). Risks to Development: An IEG Evaluation of World 5. IDA 2004, p. 13. PARIS21 stands for "Partnership Bank Assistance for Natural Disasters; Evaluation of in Statistics for Development in the 21st Century." World Bank Support for Primary Education; Engag- 6. Management for results was also explicit in the ing with Fragile States: World Bank Support to Low-In- statements of heads of multilateral development banks come Countries Under Stress (LICUS); 2006 Annual following the Monterrey and Marrakech meetings. Report on Operations Evaluation; Country Financial 7. A questionnaire regarding perceptions of the Accountability Assessments and Country Procurement Bank's performance in implementing the IDA RMS was Assessment Reports: How Effective Are World Bank Fi- sent to eight IDA donors. The seven donors who re- duciary Diagnostics?; A Decade of Action in Trans- sponded accounted for 48 percent of cumulative IDA port: An Evaluation of World Bank Assistance to the subscriptions and contributions, as of June 30, 2007. The Transport Sector, 1995­2005; World Bank Assistance to author appreciates the time and effort taken to respond Agriculture in Sub-Saharan Africa, An IEG Review; by the following IDA donors: Australia, Canada, Den- The Development Potential of Regional Programs: An mark, Netherlands, Sweden, the United Kingdom, and Evaluation of World Bank Support of Multicountry Op- the United States. 144 REFERENCES AfDB (African Development Bank), ADB (Asian Devel- ------. 2002. "Additions to IDA Resources: Thirteenth opment Bank), EBRD (European Bank for Recon- Replenishment--Final Text." IDA, Washington, DC. struction and Development), IADB (Inter-American IEG (Independent Evaluation Group, World Bank Development Bank, and the World Bank Group. Group). Forthcoming a. "The Global Water Part- 2002. "Measuring, Monitoring, and Managing for nership (GWP)." IEG Global Program Review. IEG, Development Results." http://www.mfdr.org/docu Washington, DC. http://www.globalreviews.org mentsMDBProgressReportonMfDRAnnexA.pdf ------. Forthcoming b. The World Bank in Georgia: Brown, David. 2005. "Establishing Proof." Washington Country Assistance Evaluation. Washington, DC: Post, April 19, page HE1. World Bank. Burtless, Gary. 1985. "Are Targeted Wage Subsidies Harm- ------. 2009a. Climate Change and the World Bank ful? Evidence from a Wage Voucher Experiment." Group, Phase I: An Evaluation of World Bank Win- Industrial and Labor Relations Review 39 (1). Win Energy Policy Reforms. Washington, DC: World Camdessus, Michel. 2003. Financing Water for All: Re- Bank. port of the World Panel on Financing Water Infra- ------. 2009b. Decentralization in Client Countries: structure. http://www.financingwaterforall.org/file An Evaluation of World Bank Support 1990-2007. admin/wwc/Library/Publications_and_reports/Camde Washington, DC: World Bank. ssusReport.pdf ------. 2009c. "Eritrea HIV/AIDS, Malaria, STD and Tu- Fiszbein, A., and S. Shah. 2007. "Making Results Count: berculosis Control Project." IEG, Washington, DC. A Strategy for World Bank M&E Work on India." ------. 2009d. "The Global Invasive Species Program." World Bank, Washington, DC. Photocopy. IEG Global Program Review No. 12. IEG, Washing- Gasparini, L., F. Haimovich, and S. Olivieri. 2006. "Labor ton, DC. http://www.globalreviews.org Informality: Effects of the Programa Jefes de Hogar." ------. 2009e. Improving Effectiveness and Outcomes CEDLAS, Buenos Aires, Argentina. Photocopy. for the Poor in Health, Nutrition and Population: GEF (Global Environment Facility) Evaluation Office. An Evaluation of World Bank Group Support since 2009. Mid-Term Review of the Resource Allocation 1997. Washington, DC: World Bank. Framework. Evaluation Report No. 47. Washington, ------. 2009f. Improving Municipal Management for DC: GEF Evaluation Office. Cities to Succeed. IEG Special Study. Washington, DC: ------. 2007. Evaluation of Incremental Cost Assess- World Bank. ment. GEF Evaluation Report No. 34. Washington, ------. 2008a. Annual Review of Development Effec- DC: GEF Evaluation Office. tiveness, 2008: Shared Global Challenges. Wash- IDA (International Development Association). 2007. ington, DC: World Bank. "Focus on Results: The IDA14 Results Measurement ------. 2008b. Doing Business: An Independent System and Directions for IDA15." IDA, OPCS, Wash- Evaluation--Taking the Measure of the World Bank- ington, DC. IFC Doing Business Indicators. Washington, DC: ------. 2006. "IDA14 Results Measurement Systems: World Bank. Midterm Review Report." IDA, OPCS, Washington, DC. ------. 2008c. Environmental Sustainability--An ------. 2004. "Bank Management, IDA Results Meas- Evaluation of World Bank Group Support. Wash- urement System: Recommendations for IDA14." ington, DC: World Bank. IDA, Washington, DC. ------. 2008d. Public Sector Reform: What Works and 145 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 09 Why?--An IEG Evaluation of World Bank Support. comes: An Unfinished Agenda. An Evaluation of Washington, DC: World Bank. World Bank Support for Primary Education. Wash- ------. 2008e. Using Knowledge to Improve Develop- ington, DC: World Bank. ment Effectiveness--An Evaluation of World Bank Kapur, Devesh, John Lewis, and Richard Webb, eds. Economic and Sector Work and Technical Assis- 1997. The World Bank: Its First Half-Century. Wash- tance, 2000­06. Washington, DC: World Bank. ington, DC: The Brookings Institution. ------. 2008f. Using Training to Build Capacity for De- Kremer, Michael, Esther Duflo, and Jonathan Robin- velopment: An IEG Evaluation of Project-based son. 2009. " Nudging Farmers to Use Fertilizer: The- and WBI Training. Washington, DC: World Bank. ory and Experimental Evidence from Kenya." ------. 2008g. The World Bank Group Guarantee In- http://econ-www.mit.edu/faculty/eduflo/papers. struments 1990­2007--An Independent Evalua- OECD (Organisation for Econoomic Co-operation and tion. Washington, DC: World Bank. Development ). 2002. Glossary of Key Terms in Eval- ------. 2008h. The World Bank in Indonesia, uation and Results-Based Management. Paris: OECD. 1999­2006. Washington, DC: World Bank. OPM (Oxford Policy Management/the IDL Group). 2008. ------. 2008i. The World Bank in Nigeria, 1998­2007. Evaluation of the Implementation of the Paris Dec- Washington, DC: World Bank. laration: Thematic Study--The Applicability of the ------. 2007a. Country Financial Accountability As- Paris Declaration in Fragile and Conflict-Affected sessments and Country Procurement Assessment Situations. http://www.dfid.gov.uk/aboutdfid/evalu Reports: How Effective Are World Bank Fiduciary ation.asp and http://www.oecd.org/dac/evaluation Diagnostics? Washington, DC: World Bank. network ------. 2007b. "The Critical Ecosystem Partnership QAG (Quality Assurance Group, World Bank). 2007. Fund (CEPF)." IEG Global Program Review No. 1. "FY07 Annual Review of Portfolio Performance IEG, Washington, DC. http://www.globalreviews.org (ARPP)." QAG, Washington, DC. ------. 2007c. A Decade of Action in Transport: An ------. 2009. "QAG's Quality Assessment of the Lend- Evaluation of World Bank Assistance to the Trans- ing Portfolio." QAG, Washington, DC. port Sector, 1995­2005. Washington, DC: World Bank. Salmi, Jyrki. 2006. "PROFOR Midterm Review." World ------. 2007d. The Development Potential of Regional Bank, Washington, DC. Programs: An Evaluation of World Bank Support Wapenhans, Willi, 1992. The Report of the Task Force on of Multicountry Operations. Washington, DC: World Portfolio Management. Washington, DC: World Bank. Bank. World Bank. 2009. "Investment Lending Reform-- ------. 2007e. Development Results in Middle-Income Concept Note." Operations Policy and Country Serv- Countries: An Evaluation of the World Bank's Sup- ices Department. World Bank, Washington, DC. port. Washington, DC: World Bank. ------. 2007a. The Cost of Pollution in China. Wash- ------. 2007f. World Bank Assistance to Agriculture in ington, DC: World Bank. Sub-Saharan Africa: An IEG Review. Washington, ------. 2007b. "Results Framework Terminology." Re- DC: World Bank. sults Secretariat, Operations Policy and Country ------. 2006a. 2006 Annual Report on Operations Services Department, World Bank, Washington, DC. Evaluation. Washington, DC: World Bank. ------. 2005. Forest Institutions in Transition: Expe- ------. 2006b. Engaging with Fragile States: World riences and Lessons from Eastern Europe. PRO- Bank Support to Low-Income Countries Under FOR book number 4. Washington, DC: PROFOR at Stress. Washington, DC: World Bank. the World Bank. ------. 2006c. Financial Sector Assessment Program-- ------. 2004a. "Operational Policy/Bank Procedure IEG Review of the Joint World Bank and IMF Ini- 8.60. Development Policy Lending." Operations Pol- tiative. Washington, DC: World Bank. icy and Country Services Department, World Bank, ------. 2006d. Hazards of Nature, Risks to Develop- Washington, DC. ment: An IEG Evaluation of World Bank Assistance ------. 2004b. Water Resources Sector Strategy: Strate- for Natural Disasters. Washington, DC: World Bank. gic Directions for World Bank Engagement. Wash- ------. 2006e. From Schooling Access to Learning Out- ington, DC: World Bank. 146 REFERENCES ------. 2001. Making Sustainable Commitments: An ------. 1992. The 1992 World Development Report: Environment Strategy for the World Bank. Wash- Development and the Environment. New York: Ox- ington, DC: World Bank. ford University Press for the World Bank. ------. 1993. Energy Efficiency and Conservation in World Bank Group. 2008. Aid Architecture: An Overview the Developing World: The World Bank's Role. Wash- of the Main Trends in Official Development Assis- ington, DC: World Bank. tance Flows. Washington, DC: World Bank. 147 THE WORLD BANK GROUP WORKING FOR A WORLD FREE OF POVERTY The World Bank Group consists of five institutions--the International Bank for Reconstruction and Development (IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), the Multilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of Investment Disputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their environment by providing resources, sharing knowledge, building capacity, and forging partnerships in the public and private sectors. THE INDEPENDENT EVALUATION GROUP ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Independent Evaluation Group (IEG) is an independent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (The World Bank) and IDA, IEG-IFC focuses on assessment of IFC's work toward private sector development, and IEG-MIGA evaluates the contributions of MIGA guar- antee projects and services. IEG reports directly to the Bank's Board of Directors through the Director-General, Evaluation. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group's work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by iden- tifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. ISBN 978-0-8213-8135-9 Ë|xHSKIMBy381359zv*:+:!:+:! SKU 18135