THEWORLD BANK 46130 2008 Annual Review of Development Effectiveness Shared Global Challenges IEG PUBLICATIONS 2006 Annual Report on Operations Evaluation Annual Review of Development Effectiveness 2006: Getting Results Addressing the Challenges of Globalization: An Independent Evaluation of the World Bank's Approach to Global Programs Assessing World Bank Support for Trade, 1987­2004: An IEG Evaluation Books, Buildings, and Learning Outcomes: An Impact Evaluation of World Bank Support to Basic Education in Ghana Brazil: Forging a Strategic Partnership for Results--An OED Evaluation of World Bank Assistance Bridging Troubled Waters: Assessing the World Bank Water Resources Strategy Capacity Building in Africa: An OED Evaluation of World Bank Support China: An Evaluation of World Bank Assistance The CGIAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research Committing to Results: Improving the Effectiveness of HIV/AIDS Assistance--An OED Evaluation of the World Bank's Assistance for HIV/AIDS Control Country Assistance Evaluation Retrospective: OED Self-Evaluation Debt Relief for the Poorest: An Evaluation Update of the HIPC Initiative A Decade of Action in Transport: An Evaluation of World Bank Assistance to the Transport Sector, 1995­2005 The Development Potential of Regional Programs: An Evaluation of World Bank Support of Multicountry Operations Development Results in Middle-Income Countries: An Evaluation of the World Bank's Support Economies in Transition: An OED Evaluation of World Bank Assistance Engaging with Fragile States: An IEG Review of World Bank Support to Low-Income Countries Under Stress The Effectiveness of World Bank Support for Community-Based and ­Driven Development: An OED Evaluation Evaluating a Decade of World Bank Gender Policy: 1990­99 Evaluation of World Bank Assistance to Pacific Member Countries, 1992­2002 Extractive Industries and Sustainable Development: An Evaluation of World Bank Group Experience Financial Sector Assessment Program: IEG Review of the Joint World Bank and IMF Initiative From Schooling Access to Learning Outcomes: An Unfinished Agenda--An Evaluation of World Bank Support to Primary Education Hazards of Nature, Risks to Development: An IEG Evaluation of World Bank Assistance for Natural Disasters How to Build M&E Systems to Support Better Government IEG Review of World Bank Assistance for Financial Sector Reform Improving Investment Climates: An Evaluation of World Bank Group Assistance Improving the Lives of the Poor Through Investment in Cities Improving the World Bank's Development Assistance: What Does Evaluation Show? Maintaining Momentum to 2015? An Impact Evaluation of Interventions to Improve Maternal and Child Health and Nutrition Outcomes in Bangladesh New Renewable Energy: A Review of the World Bank's Assistance Pakistan: An Evaluation of the World Bank's Assistance Pension Reform and the Development of Pension Systems: An Evaluation of World Bank Assistance Poland Country Assistance Review: Partnership in a Transition Economy The Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank's Support Through 2003 The Poverty Reduction Strategy Initiative: Findings from 10 Country Case Studies of World Bank and IMF Support Power for Development: A Review of the World Bank Group's Experience with Private Participation in the Electricity Sector Putting Social Development to Work for the Poor: An OED Review of World Bank Activities Small States: Making the Most of Development Assistance--A Synthesis of World Bank Findings Social Funds: Assessing Effectiveness Sourcebook for Evaluating Global and Regional Partnership Programs Water Management in Agriculture: Ten Years of World Bank Assistance, 1994­2004 World Bank Assistance to the Financial Sector: A Synthesis of IEG Evaluations The World Bank in Turkey: 1993­2004--An IEG Country Assistance Evaluation World Bank Lending for Lines of Credit: An IEG Evaluation All IEG evaluations are available, in whole or in part, in languages other than English. For our multilingual selection, please visit http://www.worldbank.org/ieg Annual Review of Development Effectiveness 2008 Shared Global Challenges 2008 The World Bank http://www.worldbank.org/ieg Washington, D.C. ©2008 The International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org All rights reserved 1 2 3 4 5 11 10 09 08 This volume, except for the "Management Response" and the "Chairman's Summary," is a product of the staff of the Independent Evaluation Group of the World Bank Group. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. This volume does not support any general inferences beyond the scope of the evaluation, including any inferences about the World Bank Group's past, current, or prospective overall performance. The World Bank Group does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgement on the part of The World Bank Group concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development / The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-4470; Internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@worldbank.org. Cover: Indonesian elementary schoolchildren wear surgical masks in response to 2003 outbreak of SARS in China; photo ©Reuters/ Corbis, reproduced by permission. Satellite image of hurricane; photo ©Adastra/ Getty Images, reproduced by permission. Bengal tiger; photo ©DLILLC/Corbis, reproduced by permission. Wind turbine farm; photo ©Frank Whitney/ Getty Images, reproduced by permission. ISSN: 1520-9733 ISBN-13: 978-0-8213-7714-7 e-ISBN-13: 978-0-8213-7716-1 DOI: 10.1596/978-0-8213-7714-7 Environmental sustainability : an evaluation of World Bank Group support. p. cm. Includes bibliographical references. ISBN 978-0-8213-7670-6 1. Economic assistance--Developing countries--Environmental aspects. 2. Sustainable development--Developing countries. 3. World Bank--Developing countries--Evaluation. I. World Bank. HC60.E53 2008 338.9'07091724--dc22 2008028078 World Bank InfoShop Independent Evaluation Group E-mail: pic@worldbank.org Knowledge Programs and Evaluation Capacity Telephone: 202-458-5454 Development (IEGKE) Facsimile: 202-522-1500 E-mail: ieg@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 Printed on Recycled Paper Contents vii Abbreviations ix Acknowledgments xi Foreword xiii Executive Summary xxi Management Comments: Summary xxiii Chairman's Summary: Committee on Development Effectiveness (CODE) xxvii Evaluation Snapshot in Selected Languages 1 1 Introduction 5 PART I: TRACKING BANK PERFORMANCE 7 2 Development Outcomes: Indicators of Performance 9 Measuring Project Performance: Trends from IEG Monitoring 12 Project Outcomes in Fiscal 2007 Data 14 How It Adds Up: Outcomes of Bank Country Programs 19 3 Underpinning Impact--M&E and Results Management 21 Monitoring and Evaluation Systems at the Project Level 24 Country-Level M&E: Early Evidence from Results-Based CASs 26 Managing Global Programs and Partnerships: An Emerging Agenda 28 Improving Our Understanding of Causality: The Use of Impact Evaluations 30 Monitoring Institutional Effectiveness 33 4 Lessons and Opportunities 35 Practical Lessons for the Near Term 35 Directions for the Overall Bank Agenda 37 PART II: SHARED GLOBAL CHALLENGES--LESSONS FROM THE BANK'S EXPERIENCE 39 5 The Challenge of Global Public Goods 43 6 Using the Bank's Country-Based Model to Foster Global Public Goods: Does It Work? 45 How It Works in Theory--the Bank's Strategic Setting for Fostering Global Public Goods 49 Country Programs in Practice--from Strategy to Action 57 7 The Bank's Advocacy on Global Public Goods: What Has Worked and What Has Not? 59 The Dimensions of Advocacy 59 Advocacy at Its Best: The Bank's Experience with Trade 60 The Complex Challenge of Environmental Commons and Climate Change 63 Creating a Unified Response: Learning from Avian Flu iii ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 65 Effective Advocacy Benefits from Voice and Representation 68 New Dimensions of Advocacy: Innovation through Financial Capabilities 71 8 Improving the Bank's Support for Global Public Goods: Lessons from Experience 75 Appendixes 77 A: Project Performance Results 91 B: Monitoring and Evaluation Overview 95 C: IEG's Self-Evaluation: Improving Effectiveness 107 D: Features of Global Public Goods 111 E: Management Comments 119 Endnotes 123 Bibliography Boxes 10 2.1 What Does a Satisfactory Project Look Like? Illustrations of Development Impact 16 2.2 What Does a Satisfactory Country Program Look Like? 23 3.1 M&E Findings and Recommendations in Recent IEG Evaluations 24 3.2 Armenia Joint Country Portfolio Performance Review 26 3.3 Results Measurement and Monitoring in LICUS 27 3.4 Use of CAS Results Frameworks in Country Program Management 41 5.1 Key Characteristics of Global Public Goods 47 6.1 Brazil: A Best Practice in Integrating GPG Themes in Country Strategies 48 6.2 Emerging Good Practice from the Regions 53 6.3 Bank GPG and MIC Strategies: Fates Entwined 61 7.1 Advocacy for Carbon Finance: The Bank's Role in the Growth of a World Market 62 7.2 Clean and Dirty Energy--Can the Bank Do Both? 65 7.3 Importance of Collaboration on Advocacy: HIV/AIDS 67 7.4 Governance as Institutional Experimentation: GEF 67 7.5 From Shareholder to Stakeholder Model: CGIAR Figures 9 2.1 Project Performance Has Improved over the Medium Term 11 2.2 Trends in Sectoral Performance 12 2.3 Africa's Projects Have Improved Substantially but Still Lag Behind Other Regions 15 2.4 CAEs Show Three-Fifths with Outcomes Moderately Satisfactory or Better 17 2.5 CASCR Reviews Indicate That Bank Programs in MICs Outperform Those in LICs 22 3.1 Projects with Higher Outcome Ratings Have Better M&E Ratings 22 3.2 M&E Is Rated Modest or Lower in Two-Thirds of ICR Reviews 29 3.3 About 70 Percent of Ongoing Evaluations Are Clustered in Five Areas 29 3.4 Nearly Two-Thirds of Ongoing Evaluations Are Located in Two Regions 49 6.1 Bank Expenditures on Main GPG Themes 50 6.2 IBRD and IDA Lending for Main GPG Themes iv CONTENTS 55 6.3 Most Global Programs Do Not Focus on GPGs, but Most of the Bank's GPP Resources Are Devoted to Those That Do 66 7.1 Less Than Half of Funds Committed to Integrated Country Plans Have Been Disbursed Tables 11 2.1 Distribution of Project Ratings Moved Up the Scale in FY03­07 13 2.2 Disconnect between the Bank's Self-Ratings and IEG Ratings Increased Dramatically in FY07 15 2.3 Summary of CAE Ratings, FY98­08 17 2.4 Summary of CASCR Review Ratings, FY03­08 v Abbreviations AfricaRMS Africa Results Monitoring System ARDE Annual Review of Development Effectiveness CAE Country Assistance Evaluation CAS Country Assistance Strategy CASCR Country Assistance Strategy Completion Report CGIAR Consultative Group on International Agriculture Research CODE Committee on Development Effectiveness CPS Country Partnership Strategy DGF Development Grant Facility FY Fiscal year GEF Global Environment Facility GPG Global public good GPP Global programs and partnerships IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results report IDA International Development Association IEG Independent Evaluation Group IFFIm International Financing Facility for Vaccines and Immunization ISR Implementation Status and Results report LIC Low-income country LICUS Low-Income Countries Under Stress M&E Monitoring and evaluation MAR Management Action Record MIC Middle-income country NGO Nongovernmental organization PCF Prototype Carbon Fund PRSC Poverty Reduction Support Credit QAG Quality Assurance Group R&D Research and development RMS Results Measurement System vii Acknowledgments This report was written by Thomas O'Brien, and Alan Gelb, Steve Radelet, and Salvatore Schiavo- prepared by a team under his leadership. The Campo served as peer reviewers of the final team's core members were Brett Libresco, Yoshi evaluation report. Uchimura, Pamela Velez-Vega, Victoria Chang, and Jan Rielaender. The report draws upon The evaluation team greatly appreciates the time significant research and contributions provided and insights from many individuals inside and by Scott Barrett, Robin Broadfield, John outside the World Bank who were interviewed Eriksson, Nidhi Khattri, Shonar Lala, Domenico for this report. The evaluation benefited substan- Lombardi, and Andrew Waxman. Marinella Yadao tially from the constructive advice and feedback and Yvette Jarencio-Lukban provided support from many staff in the World Bank Group. and assistance to the team during the project. William Hurlbut edited an earlier version of the The evaluation was conducted under the guidance manuscript. Helen Chin edited the report for of Mark Sundberg, Manager, IEG Corporate and publication. Global Evaluation and Methods Unit. Director-General, Evaluation: Vinod Thomas Director, Independent Evaluation Group, World Bank: Cheryl Gray Manager, Corporate and Global Evaluation and Methods: Mark Sundberg Task Manager: Thomas O'Brien ix Indonesian elementary schoolchildren wear surgical masks in response to 2003 outbreak of SARS in China; photo ©Reuters/ Corbis, reproduced by permission. Foreword A farmer in Kenya faces a hopeful future as he commitment to delivering development results, exports to growing markets opened up by global including through the use of the new round of trading agreements. But another in Vietnam International Development Association funding stares at ruin as her poultry is culled so that the to which donors have committed a record of threat of avian flu can be curtailed. And others in nearly $42 billion. the Bay of Bengal face longer-term danger from a growing risk of floods as the earth's climate This year's Annual Review of Development warms. Living continents apart, these farmers Effectiveness focuses on assessing the World have more in common than their efforts to Bank's development effectiveness, with special escape poverty. Their futures are increasingly attention to global public goods. It notes some interlinked with the world's response to shared encouraging developments. Project global challenges. performance has improved over the medium term; country programs have worked relatively Such global public goods--a global trading well in several large nations that house a majority system, biodiversity--and the issue of combating of the world's poor; and the Bank has increased global public bads--climate change, transborder attention to collective international action on contagion--share the problem of undersupply. global public goods and advocated effectively on To increase supply, and to avoid the "tragedy of some of those important challenges. But work is the commons," collective action is required, required to remedy weaknesses. Notably there is often with coordinated responses at the local, a need to go beyond the Bank's country-based regional, and global levels. Where the benefits model when tackling issues where the perceived can be directly felt at the local level, such as local and national benefits of action do not match halting transmission of diseases, motivating global benefits from collective action. Attention action is less difficult. Where fully appropriating should be paid to improving weak performance the benefits locally is not possible or takes a long of country programs in smaller states and those time--such as climate change--collective action with extensive poverty, and redressing shortcom- is far more difficult and requires global efforts to ings in applying monitoring and evaluation in help motivate local action. projects and country programs. The World Bank has set fostering global public Over the next decade and beyond, the success of goods as one of its six strategic priorities under the international community and the World Bank its new president, and is deeply involved in Group in rising to the shared global challenges of efforts to strengthen their supply through invest- our time will be crucial to reducing poverty and, ments spanning local, regional, and global public indeed, to solving the looming challenges the goods. This goes hand-in-hand with the Bank's world collectively faces. Vinod Thomas Director-General, Evaluation xi Bengal tiger; photo ©DLILLC/Corbis, reproduced by permission. Executive Summary F or the World Bank and its partners, the ever-present test is to deliver results--to lift people out of poverty and promote socially and envi- ronmentally sustainable development. Achieving such success in any in- dividual country is increasingly intertwined with making progress on shared global challenges. A fair and efficient international trade regime, for example, is a global public good that allows developing countries to trade more and grow faster. The increasing global threat of climate change--a "public bad," by con- trast--particularly imperils the poor, who bear the brunt of more frequent nat- ural disasters and hazards to health and agriculture. This year's Annual Review of Development Development outcomes from Bank lending have Effectiveness is in a new format and presents improved over the medium term, mostly evidence on the Bank's efforts in two important through a rise in the share of projects rated as and connected areas. Part I, which is a standard moderately satisfactory in meeting their section of the new format, helps to track Bank objectives. Achievements of country programs in performance, notably trends in outcomes of meeting their objectives--typically including Bank projects and country programs, the growth, poverty reduction, and environmental evolution of monitoring and evaluation (M&E), sustainability--have been moderately satisfac- and the role of evaluation in the results agenda. tory or better in three-fifths of cases, including in Part II examines a special topic of great several large countries, home to the majority of relevance to the results described in the first the world's poor. But too many other programs, part: the Bank's work in fostering global public particularly in impoverished countries, have goods, such as protecting the earth's climate been moderately unsatisfactory or worse. The and preventing the spread of dangerous Bank's overall approach to M&E has many communicable diseases. Global public goods strengths, including recent progress in updating tend to be undersupplied, as are all public its policies on lending and country strategies to goods. Motivating local action is easier when the emphasize M&E. Yet significant overoptimism in benefits are captured locally: efforts to stop the the Bank's self-assessment of ongoing project transborder spread of pandemic disease are performance and weaknesses in the use of M&E more easily motivated when the results directly systems are of concern. The quality of project benefit local populations. By contrast, reducing M&E is often quite low, and results frameworks greenhouse gas emissions is harder to motivate in country assistance strategies need clearer and because of a lack of perceived local benefits, simpler articulation with baseline indicators if particularly in the near term. The report they are to be effective as management tools. examines both situations, but it is the latter-- where global and local benefits diverge--in The Bank has paid growing attention to global which the challenges are greatest and the role of public goods, which increasingly influence the Bank is potentially pathbreaking. development outcomes. It has helped foster xiii ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 global public goods through country activities, Project performance among the Bank's Regions and its country model has worked well when improved most in Africa--about three-quarters national and global interests dovetail--often of projects, weighted by disbursement, during with an agreed international framework for fiscal 2003­07 were moderately satisfactory or action--and when grant finance supports better in meeting development objectives, as country-based investments. The Bank has also compared with 60 percent during fiscal been a strong advocate for changes in global 1998­2002. There is still a challenge for Africa systems, such as international trade reform, projects to improve further and get closer to the where it has expertise and is willing to engage in performance in other Bank Regions. public debate. But the greatest challenges arise where local, national, and global benefits--actual Bank management should avoid overoptimism in or perceived, immediate or for the next genera- assessing ongoing project performance to improve tion--diverge significantly from each other. For real-time management for results. The considerable example, the investments needed to protect the increase, in fiscal 2007, in the difference between earth's climate and environmental commons the Bank's self-ratings of project performance vary considerably at the local, national, and and IEG's final ratings of development outcomes global levels, as do the costs and benefits of such (sometimes called the "disconnect") illustrates actions. To more effectively bridge the gap the point. In fiscal 2007, over two-thirds of between global needs and country concerns, the projects rated moderately unsatisfactory or Bank should consider: creating dedicated worse by IEG had been reported by the Bank as budgets and better incentives for country teams moderately satisfactory or better just before they to work on global public goods; deploying its closed. Such a wide disconnect--about twice as global knowledge networks more effectively; large as in fiscal 2005 and fiscal 2006--means developing new financial instruments and management is less likely to identify problem securing additional resources, including grant projects and take timely remedial action. funds, to support country-level investments; and using its standing more powerfully to give Such management attention is important given greater voice to developing countries in the that the share of projects with moderately governance of global programs. satisfactory or better outcomes has fallen from nearly 83 percent in fiscal 2006 to 76 percent in Part I: Tracking Bank Performance fiscal 2007. A single year's data is not, by itself, a cause for alarm, but vigilance is needed to ensure Development outcomes from Bank lending have that it does not foreshadow a persistent decline. improved over the medium term. Over the three Excessively complex project design and overly years to end-fiscal 2007, IEG's evaluations ambitious assumptions on political ownership confirm that 80 percent of projects were and implementation capacity lay at the heart of moderately satisfactory or better in meeting their many poorly performing projects that exited in development objectives. This meets the Bank's fiscal 2007. own performance target and is a significant improvement from the start of the decade. A Securing strong development outcomes at the Bank-supported water project in Cambodia, country level has proved challenging. Over the past which brought clean water to 750,000 people in 10 years, evaluations of 81 Bank country Phnom Penh, illustrates such development programs--incorporating projects, policy and outcomes. technical advice, and other types of assistance-- show that three-fifths of them were moderately Project outcomes improved in most sectors, but satisfactory or better in meeting their develop- average ratings slipped for projects in the fields of ment outcomes. Looking at specific grades on health and public sector governance during fiscal IEG's ratings scale, the Bank succeeded in 2003­07, as compared with fiscal 1998­2002. supporting satisfactory outcomes in 30 percent xiv EXECUTIVE SUMMARY of evaluated programs--including several large poor design and the absence of incentives to and important countries such as Brazil and conduct M&E. Even so, there are examples of China, which have made strides in reducing emerging good practice such as the "Moldova poverty. A further 30 percent of country results scorecard," which links country program programs were rated moderately satisfactory. But management and resource allocation. the remaining 40 percent of programs--concen- trated in countries that are smaller or have The Bank has improved its approach to managing extensive poverty, such as Malawi--were and monitoring global programs and partnerships. moderately unsatisfactory or worse in meeting The Bank now has more robust systems to track their stated development objectives. Very few involvement in global programs and partner- country programs are producing best-practice ships, thus encouraging selectivity and quality at results. Indeed, of 36 programs rated since fiscal entry. All programs receiving Development Grant 2002, not one has been highly satisfactory. At the Facility funding of $300,000 or more, over the life same time, no program has ever been rated of the program, are also subject to independent highly unsatisfactory. program-level evaluations. But an IEG assess- ment of a cross-section of such evaluations found How well is the Bank using and learning from good their quality frequently compromised by weak monitoring and evaluation (M&E) systems, which are M&E systems, particularly a lack of systematic key to improving its effectiveness over the longer evidence on the achievement of programs' term? The Bank's overall approach to M&E has objectives at the outcome level. Therefore, it is many strengths, and in recent years there has difficult to say whether the global programs been considerable progress in updating its reviewed--together accounting for about $100 policies on lending and country strategies to million of annual spending--ultimately had a emphasize M&E. The introduction of results- substantial effect on the ground. based country assistance strategies has been a particularly significant step. But considerable Two recent developments may hold promise for the room for improvement remains in putting all of Bank's results agenda, although they are in their this into practice. early days. The first is the use of impact evaluations: the number supported by the Bank has more At the project level, the overall quality of M&E has than doubled, to 158 over the past year. Impact been low--rated as modest or negligible in two- evaluations are not a panacea but can create thirds of projects for which data are available-- better understanding of the causal links and since fiscal 2006. Some of the factors contributing factors contributing to the outcomes of projects, to low M&E quality assessments were poorly programs, and policies. However, these evalua- designed results frameworks, poorly articulated tions are concentrated in a few areas (education, results chains linking outputs with outcomes, health, and conditional cash transfers) and need and performance indicators lacking baselines to be managed more strategically to draw more and targets. knowledge from them. Effective results frameworks at the country level are The second development is a new approach toward key to managing for results. While staff are gaining measuring and reporting on development results for experience with results frameworks, too often such the International Development Association (IDA)-- frameworks have been poorly formulated and hence the Bank's main source of concessional finance. The their usefulness is undermined. In many cases, results management system for IDA--initiated in frameworks identify too many outcomes and the 14th replenishment of IDA, with commit- monitoring indicators and lack baselines and ments to enhance it in the 15th replenishment targets. Their use for monitoring and managing (IDA15)--tries, among other things, to spotlight the country program, and for informing country changes in indicators, including access to water assistance evaluations, is very limited because of and measures of child health. It is premature to xv ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 assess how well this will work, but it is an The World Bank Group has emphasized the need important step in corporate-level monitoring to foster global public goods as one of its main and evaluation. At the same time, there are priorities in the future. The effective provision of difficult questions as to whether and how a more global public goods increasingly influences comprehensive results framework for the Bank, development results (discussed in Part I above), as a whole, could evolve. It continues to be especially addressing the many dimensions of difficult to piece together the various M&E poverty, including vulnerability. The Bank's indicators to form a view of the Bank's overall strategic framework for its role with regard to development results. global public goods notes that the Bank can connect global concerns to country programs There are two broad lessons for better tracking Bank and advocate for collective international action. performance. First, practical steps are needed (a) How can the Bank enhance its effectiveness in at the project level, and in global and regional this area? programs, to enhance the quality of the M&E systems, especially by working to put in place Can the Bank's Country-Based Model Foster good baseline information and to elucidate Global Public Goods? clearly the link between project outputs and targeted outcomes; (b) at the country level, to Relying on the country-based model as the platform simplify results frameworks and so make them for the Bank's work on global public goods is a more useful in guiding and evaluating programs; double-edged sword. The model works well when and (c) at the institutional level, for the Bank and national partners see an alignment between in partner countries, to manage and learn from a domestic and global benefits, and when the Bank growing number of impact evaluations, includ- has an attractive instrument to help implement ing by better integrating them into country action at the country level. For example, the programs and exploiting cross-country synergies Bank's successful work in client countries, to in conducting and sharing studies. Second, the help phase out ozone-depleting substances, Bank and IEG should strengthen the evaluation benefited from the existence of the Montreal knowledge base for the Bank's corporate results. Protocol--a binding agreement that committed Progress on these two fronts will improve the countries to globally agreed action--and the prospects for greater development impact in the Multilateral Fund, which provided resources for years ahead. investments. Global Environment Facility (GEF) grants have also been well integrated into Bank Part II: Shared Global Challenges country programs, such as in China, where a large GEF portfolio has buttressed growing The Challenge of Global Public Goods attention to environmental issues. And in Vietnam, the Bank has been able to use its Tackling global climate change and providing other multisectoral expertise, combined with conces- important global public goods present some of the sional finance, to help the authorities cope with greatest challenges of our time. Indeed, many the threat of avian flu, in part because there was global public goods are chronically undersup- strong national interest in averting economic plied. Why? Because it is difficult to secure collec- fallout in the domestic food industry. tive action among nations to provide a public good--such as keeping air clean--particularly The country-based model, however, comes under when the costs are borne locally while the strain, especially when global and country benefits are largely captured nationally or interests are seen to diverge significantly and the globally. Yet there is a growing interconnection Bank's traditional tools, including its lending, do between the different types of investments and not gain traction with clients. This makes it actions needed at various levels to foster global doubly difficult to secure progress with global public goods. public goods. Tackling climate change requires xvi EXECUTIVE SUMMARY huge adjustments in various economic The Bank has at least three levers for moving from behaviors, including reducing emissions and strategy to action at the country level--budget and improving economywide energy efficiency and trust fund allocation, financing instruments, and use. For many countries, the benefits of such global programs. Each is discussed in turn below. actions seem remote while the costs accrue in the near term. To date, though, the Bank has not RESOURCE ALLOCATION been able to call on an attractive large-scale The Bank estimates its administrative expenditure funding program or invoke an international on global public goods at around $110 million in framework to encourage comprehensive action fiscal 2007, nearly half of which is from sources that on climate change. It will be important to see are outside the Bank's core budget, such as trust how the recently discussed Climate Investment funds. At about 4 percent of its overall operating Funds help improve this situation. budget, this is one of the smaller allocations for the Bank's six strategic priorities. These estimates The Bank pays attention to fostering global public should be treated with some caution because goods in its high-level corporate strategies and the they may vary significantly, depending on the topic has been emphasized by the president as one definitions and data classifications used. Going of the Bank's six strategic pillars. However, forward, a more precise definition and tracking attention wanes as one moves down the levels of spending on global public goods would be a from corporate strategies to sectoral or regional useful management tool. strategies, and then down one more level to country strategies. Both the Bank's GPG A heavy reliance on trust funds for financing Framework and Long-Term Strategic Exercise global public goods work may itself increase the discussed global public goods extensively but difficulties of mainstreaming such activity lacked specifics on how to translate corporate alongside long-standing work financed by the priorities into country action. The treatment in Bank's own budget. Spending on global public strategies at the next level down--the Bank's goods, as a whole, has risen rapidly over the past networks and Regional units--varies signifi- five years, with the biggest increase for work on cantly. Attention to global public goods is more environmental commons. prominent in both sectoral and Regional strate- gies dealing with the environment than in those FINANCING INSTRUMENTS dealing with the health sector. This may be due Concessional finance is important to foster many to the type of intervention needed in health global public goods, and in recent years, the Bank sector global public goods--such as communi- has committed substantial IDA funding to help cable disease control, which requires a strong countries in programs with clear global public national focus that might not be explicitly goods dimensions, such as HIV/AIDS and connected to global action. environmental commons. Often, country-level implementation capacity is stretched, however, The systems for integrating global public goods into and national priorities may take precedence over country strategies are underdeveloped. Environ- some global public goods considerations. Staff mental commons is frequently noted in country report that there is great reluctance among strategies (in part because GEF projects are national partners and Bank country teams to mainstreamed in the Bank's systems), but other allow IDA allocations targeted for poverty global public goods are less often emphasized. reduction to be diverted to fostering global There is no evidence that over time the public goods, which may not immediately treatment of global public goods in Bank country benefit the poorest populations. A recent innova- strategies has expanded, but very recent tion in IDA is a specific allocation for regional examples of good practice--such as in Brazil-- (multicountry) projects. Although it is too early may pave the way for more thorough and consis- to assess how well this is working, it should be tent strategic planning. monitored for lessons in mirroring this approach xvii ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 for some global public goods; great care would partnerships is not fully driven by global public be needed to avoid fragmenting IDA's overall goods concerns, however, since more than 100 of framework. these programs are focused largely on national public goods, such as urban development or When the Bank has had a clear and viable instrument regulation of the markets for infrastructure. to help its country partners take action on some global public goods, there has been progress--the Despite the Bank's direct role as a partner in global GEF is a good example. Where the Bank has not programs, systematic linkages to country programs had an obviously attractive financial instru- have been lacking at times. For example, many of ment--and/or where there has been a lack of the programs had only modest participation by demand from country partners--it is less easy to middle-income countries. Task managers for see progress. Measures to protect and conserve global programs have not commonly been important forest resources around the world, for required to demonstrate how such programs example, have produced a highly varied picture. have added value to country programs and Bank In Indonesia, an evaluation of the Bank's country operations, and often lack the incentive or assistance program from 1999 to 2006 showed administrative budget to do so. that it covered forestry issues with large-scale analytical work but little lending. Over that Merely locating a global program in the Bank--there period, the traction achieved by the Bank was are 57 such programs--does not guarantee effective very limited, and deforestation continued at a country linkages. For example, linkages were rapid clip. weak in the Population Reproductive Health Capacity Building Program, despite the potential There is often a mismatch between country needs synergies with Bank investment operations in (and resources) and global ambitions for global various countries. IEG evaluations have also public goods. In middle-income countries, the found that greater legitimacy of a global program Bank's ability to influence (or persuade) a does appear to foster stronger linkages with country to take concrete action on some global country operations. public goods is inherently limited, even though effective provision of those goods requires deep In the Bank's efforts to provide regional public participation by these middle-income countries. goods--and to link regional and country concerns The limits of nonconcessional finance are clear, and opportunities--it faces challenges similar to for example, in the Bank's work on avian those for global public goods. Regional programs influenza, where only 7 of the 50 projects have risen in importance in recent years, but approved are financed by the International Bank their integration into country programs remains for Reconstruction and Development (IBRD), the exception rather than the rule, and they still and, to date, only $12 million of the $94 million account for a modest share of Bank lending. in IBRD loans have been disbursed. The Bank's Advocacy on Global Public Goods: GLOBAL PROGRAMS What Has Worked and What Has Not The Bank is now a partner in some 160 global programs and partnerships, and about 90 percent of Successful advocacy goes beyond encouraging the total spending of these global programs and action at the country level. It also involves producing partnerships, which is overseen by the Bank, is collective global responses and promoting the directed at global public goods. A few large initia- development interests of the poor in international tives account for most of this spending: the Global agreements and frameworks for action. Fund to Fight Aids, Tuberculosis, and Malaria; the GEF; and the Consultative Group for International Promoting improvements in the global trading Agricultural Research (CGIAR). The Bank's framework is an example of the Bank's advocacy at administrative effort in global programs and its best. Key ingredients included a long period of xviii EXECUTIVE SUMMARY working directly with partner countries, the ments that balance the interests of the key assembly of first-rate intellectual and analytical parties involved. research capacity, proactive and highly visible dissemination, and the willingness to engage in Improving the Bank's Support for Global Public public debate. These were combined to excellent Goods: Lessons from Experience effect, and the Bank's work also had an opportu- nity to gain traction in the context of "live" The Bank's country model has its place in fostering negotiations for the Doha round of a new global public goods. It has worked well when international trade agreement. national and global interests coincide--often with an agreed international framework for The experience with avian flu also illustrates the action, such as the Montreal Protocol--and Bank's strengths as an advocate and convener. The when grant finance supports country-based Bank's contributions to a global response was investments. built on robust economic analysis, convening power, fiduciary reputation, and multisectoral Looking ahead, some of the great shared global expertise. It also helped that the ground was challenges arise where national and global benefits fertile for the Bank's advocacy, given that global diverge significantly--most notably on climate and national concerns aligned as country needs protection. In tackling these challenges, the were urgently felt. Bank--including through cooperation with the International Finance Corporation and Multilat- Advocacy on environmental commons has proved a eral Investment Guarantee Agency--needs to more complex challenge. The Bank has played a find a way to bridge the gap more effectively positive advocacy role in some very practical between global needs and country preferences. settings, including the securing of resources for Lessons from this review suggest some measures the GEF, the launch of the Prototype Carbon in five areas that may help the Bank upgrade its Fund (and subsequent carbon funds), and ability to foster global public goods. methodologies to put the Clean Development Mechanism into action. The extent to which the First, the Bank can create better incentives to deliver Bank has been a leading influential advocate on global public goods effectively at the country level. climate change is more debatable, but there is This would include new approaches to setting now a platform on which to build future budgets and recognizing the performance of advocacy work, including the Bank's new Strate- managers and staff. On budget setting, one gic Framework for Climate Change. option is to set aside, at the corporate level, significant administrative funding to be allocated Advocacy through global programs has become an to country teams--transparently and possibly increasingly important channel for fostering global competitively--for high-priority global public public goods. Giving proper voice and representa- goods work at the country level. Care would be tion to developing countries in such programs needed to make sure such funding was used as a improves their responsiveness and long-term genuine addition by teams and not simply to sustainability. Yet, developing country voices displace other activity. To provide better remain underrepresented--not least in the incentives to staff, managers at all levels need to governance of many global programs--and consider recognizing country- and global-level whether the Bank could have pushed harder on work on global public goods in performance this issue remains a question. It is encouraging management systems. that governance arrangements in several programs, including the GEF and CGIAR, have Second, the Bank can consider clearer organiza- improved over time. For large new global tional arrangements to best select, and indeed link programs aimed at climate change, it is critical to together, responses at country, regional, and global ensure sound and equitable governance arrange- levels. Some Regions may want to have dedicated xix ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 staff advancing work on regional programs (and with global public goods. That would include the regional public goods), as has been done in Bank's continuing to secure additional develop- Africa, and perhaps expand their purview to ment assistance and promote the design and use cover global public goods as well. But this is not of market-based instruments to help developing a one-size-fits-all prescription, and other Regions countries provide global public goods. The Bank may have different arrangements suitable to their could also explore further ways to stimulate circumstances. South-South exchange of knowledge--and the development and application of new technolo- Third, a more effective approach to the delivery of gies designed with and for the South--to the Bank's global knowledge and capacity to country contribute to global public goods, such as teams working on global public goods would be climate-friendly energy production and use. beneficial. To this end, the way the Bank can best deploy its expertise, particularly that of its Finally, a firmer and more precise justification is specialists located at the center of the institution needed for the costs and benefits of actions being in the network anchors, should be reviewed. proposed for the Bank's work on fostering global public goods, to ensure that such work is Fourth, the Bank and its stakeholders could renew financially and institutionally sustainable over the attention to ensuring that the perspective of develop- long term. Particularly for global programs, the ing countries is connected effectively with global Bank must redouble its efforts to be more responses. The Bank might be able to use its selective in its engagement and more forthright standing more powerfully to give greater voice to in its exiting programs whose benefits and cost- developing countries in the governance of signif- effectiveness are questionable. The Bank should icant global programs. It should take a more also be insistent about putting in place, and proactive stance in advocating for development using, sound results frameworks, underpinned interests--and developing country partners--in by realistic and cost-effective monitoring and international forums (and agreements) dealing evaluation systems. xx Management Comments: Summary T he 2008 Annual Review of Development Effectiveness (ARDE) tracks Bank performance and examines a particular thematic topic, the Bank's work in fostering global public goods (GPGs). Management values the review of project and program outcomes and of monitoring and evaluation (M&E) practice, and we appreciate in particular the clarity of analysis, which helps the Bank learn from experience. We very much welcome also the review of Bank Regions are reviewing their portfolios and country program support for GPGs and of the working actively with staff on measures to Bank's advocacy work on GPGs, particularly the improve rating practices and strengthen ongoing recommendations on bridging the gap between operations. A more fundamental issue, however, global needs and country preferences. Manage- is that the traditional supervision model, which ment's complete comments are included as an was geared to infrastructure projects in middle- appendix; this note summarizes the main points, income countries, needs to be adapted to the with a focus on recent actions. circumstances of projects in softer sectors and in fragile states, which call for more Bank engage- Tracking Bank Performance ment in project implementation, better risk reporting, and customized implementation Project Outcomes. The ARDE confirms that support directed at capacity building. Manage- project outcomes have significantly improved ment is addressing this issue in the context of over the medium term, exceeding the Bank's investment lending reform. performance benchmarks in fiscal 2004­06. Management notes with particular satisfaction Country Program Outcomes. The ARDE reports that impressive improvements in the projects of the outcome ratings, averaged over a long period, are Africa Region and of the water supply and sanita- considerably lower for Bank-supported programs tion sector. Management shares IEG's concerns in low-income countries (LICs) than for programs about the dip in project outcomes in fiscal 2007, in middle-income countries (MICs), and are notably in noninfrastructure sectors and low- lower for programs than for projects. Manage- income countries, and appreciates the need to ment is concerned about the gap between MIC be vigilant so the dip does not become a trend. and LIC programs and would very much welcome Overoptimism in self-ratings of operations close IEG's deeper analysis of the underlying factors-- to closure parallels findings in our more detailed notably an analysis of changes in outcomes over review of International Development Association time for subgroups of countries, taking into (IDA) controls, reports by the Bank's internal account such developments as the introduction Quality Assurance Group (QAG), and the India of Country Assistance Strategy (CAS) results Detailed Implementation Review. In response, frameworks and changes in evaluation methodol- xxi ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 ogy. Management believes that the gap between Management agrees that country-based support program and project outcomes primarily reflects for GPGs will need to increase and be better a difference in performance standards, not a integrated into CASs. We also believe that the failure of programs to exploit synergies. The Bank appropriate role for the Bank is specific to the only recently introduced results-based CASs that particular GPG being supported and to the clearly distinguish between the country's country context; hence variations in the extent objectives and CAS outcomes based on those of Bank involvement among GPGs and among objectives, and include results chains setting out countries are to be expected. Since the Bank is how the Bank will contribute to those outcomes. only one player among many, our framework for Most of the programs evaluated by IEG are on support to GPGs calls for identifying where gaps based on CASs in which this distinction was much are not being met by other agencies and then less clear. The upcoming CAS Retrospective will helping to fill the gaps where the Bank has the provide a further opportunity for an informed capability and comparative advantage. Country discussion of program outcomes. ownership and response to client demand remain the primary principles of country-based Monitoring and Evaluation. Management agrees support for GPGs. with the ARDE's finding that the implementation of the Bank's strong policy framework for M&E The Country Program Model. The ARDE highlights continues to face challenges in projects and the challenges to country-based support for GPGs programs. Management appreciates and is acting in cases where global and country interests on IEG's recommendations to focus on the diverge and there is no international framework provision of good baseline information, articu- for collective action. Management believes that late more clearly the link between project such challenges can be addressed without outputs and targeted outcomes, simplify CAS earmarked funding at the corporate level for results frameworks, and use M&E more country work. We do not agree with the broad effectively for program management. To do this conclusion that relying on the country program effectively will require not only Bank action but model is a "double-edged sword." The Bank's measures to address the issue of statistical ultimate clients are poor people. By using the capacity in member countries. As announced country program model, the Bank is better able to during the High-Level Forum on Aid Effective- provide analysis that puts growth and poverty ness in Accra, our joint efforts with the Nether- reduction at the core and relates GPG challenges lands and the United Kingdom have succeeded to this goal, and to ensure that the actions it in establishing a new Statistics for Results Facility supports are owned by the country. In that will help countries implement their national Management's view, the key challenge--discussed plans for improving statistical systems. in the upcoming CAS Retrospective--is to more thoroughly integrate GPGs into the CAS diagnosis Shared Global Challenges-- of country development challenges and the Lessons from the Bank's Experience dialogue with the government, and on that basis determine the appropriate contribution of global Bank Support at the Country Level. The ARDE programs and trust funds as part of the CAS observes that GPGs other than environmental support program. We also agree that mobilizing commons are not sufficiently emphasized in concessional funding is important for engage- CASs and that the extent of Bank involvement in ment on GPGs in MICs, and we note that efforts to GPG issues varies widely among countries. that effect are under way. xxii Chairperson's Summary: Committee on Development Effectiveness (CODE) O n July 23, 2008, the Committee on Development Effectiveness met to consider the Annual Review of Development Effectiveness 2008: Shared Global Challenges (ARDE), prepared by the Independent Evaluation Group (IEG), and Draft Management Comments. 2008 ARDE the country level; (ii) consider clearer organiza- Part I of the report tracked the Bank's perform- tional arrangements to best select and link ance (including trends in outcomes of projects responses at country, regional, and global levels; and country programs), the evolution of monitor- (iii) enhance the delivery of global knowledge and ing and evaluation (M&E), and the role of evalua- capacity to country teams working on GPGs; (iv) tion in the results agenda. IEG found that overall ensure that the perspectives of developing development outcomes of Bank lending have countries are effectively connected with global improved over the medium term, albeit not over responses; and (v) improve the justifications for the past year. Noting the growing "disconnect" the costs and benefits of actions being proposed between project supervision ratings and final to foster GPGs. project outcome ratings (which could point to weak incentives for accurate project reporting), Draft Management Response IEG cautioned against overoptimism in assessing Management welcomed the insightful review ongoing project performance, and called for and agreed that vigilance is warranted with vigilance to ensure that the drop in project respect to the weakening development performance in fiscal 2007 does not foreshadow a outcomes of exiting projects and the increasing persistent decline. IEG also noted opportunities "disconnect" between the Bank's self-ratings of to strengthen M&E. project performance and IEG's final ratings of development outcomes in fiscal 2007. In this Part II focused on a special theme, which this year regard, it noted that a deeper analysis of was the Bank's work in fostering global public contributing factors, especially of differences goods (GPGs). IEG found that the country-based among regions and sectors, would have been model worked well when national and global useful. Management commented on the need to interests coincided and when grant finance address the different outcomes of middle- supports country-level investment, but the income country (MIC) and low-income country greatest challenges arise when actual or perceived (LIC) programs, and the issue of lower outcome local, national, and global benefits diverge signifi- ratings for country programs than for projects. It cantly. IEG drew lessons for the Bank's considera- had some different views about the obstacles to tion: (i) strengthen incentives to deliver GPGs at better M&E. Management also offered its xxiii ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 perspectives on efforts to integrate GPGs at the country programs and operations, the discussion sector, country, and regional levels; its role in also drew attention to several findings of mobilizing and providing innovative financing; corporate relevance, namely the advocacy role support for different types of global programs where the Bank must push harder to strengthen and partnerships; the proposal to set aside the developing countries' perspectives and administrative funds to support high-priority voices, which are presently underrepresented. GPGs at the country level; and the Bank's role in To this end, it was noted that GPGs are relatively advocating the climate change agenda. new in the development agenda, and the Bank is at the beginning of its learning curve in assisting Overall Conclusions countries to address GPG issues. Therefore, the The Committee heard that management found Committee should keep an open mind about the the structure and analytical context of the report challenges of GPGs and avoid reaching to be of high quality and was broadly comfort- premature conclusions. able with the findings, with one exception regarding the recommendation to set aside Next Steps earmarked country program budget for GPGs. The Board is scheduled to consider the report This issue and, more broadly, the matter of the on September 9, 2008. IEG will provide a appropriate business model and incentives for summary of actions to which management integrating global and more traditional, local committed, in response to IEG recommenda- development interests elicited comments from tions and where IEG has seen limited progress most speakers, who expressed a diversity of after three years. For future ARDE reports, IEG views and also made suggestions. Another area was asked to consider a closer link between Parts that resonated with most participants was the I and II. Management noted that M&E and results "disconnect" between staff self-ratings and IEG's management in country programs will be final ratings after project completion, which was discussed under the upcoming Country higher in fiscal 2007 than in recent years. Assistance Strategy (CAS) Retrospective Report. The Committee appreciated management's Main issues raised at the meeting were the focus on: (i) the differentials between country following: groupings or sectors with the aim of strengthen- ing Bank business in dealing with the challenges Project Performance. Members agreed that of low-capacity countries and softer sectors; (ii) vigilance is needed to identify problem projects the findings related to low-income country and risks in real time, an issue previously raised programs, especially since IEG did not rate the by the Quality Assurance Group (QAG). They overall development outcome of a single took note of management's recognition that the program beyond moderately satisfactory during current supervision for large infrastructure the evaluation period; (iii) the distinction projects does not differentiate between between project complexity and development countries (low-income vs. middle-income risks related to challenging country circum- countries and fragile situations) and between stances; (iv) the emphasis on M&E not only to hard (infrastructure) and softer sectors (human ensure data availability for assessing results (and development or public sector management). avoiding complacency) but also to obtain a Some members acknowledged the higher fiscal clearer perspective on the complementarities 2007 "disconnect" between the Bank's self-rating and tradeoffs between public goods and more of project performance and IEG's final ratings of traditional economic growth and development development outcomes. They sought further concerns; and (v) the new structure of the ARDE. comments on the cause of this "disconnect." One member suggested that the Bank and IEG With respect to Part II on GPGs, in addition to should find a mutually agreed scheme; IEG ways of striking the appropriate balances in noted that the rating systems were comparable. xxiv CHAIRPERSON'S SUMMARY: COMMITTEE ON DEVELOPMENT EFECTIVENESS (CODE) This member also recommended that the CAS further suggested that Implementation Comple- Progress Report should address the issue of tion Reports should include a formal require- quality of supervision. ment to rate a project's M&E. Questions were raised more generally on the quality of M&E, Issues were raised relating to project complexity simplification of country results framework, and and development risks under challenging knowledge base of corporate results. country circumstances. A point was made that project complexity should be commensurate Challenge of GPGs. Members generally agreed that with implementation capacity. Speakers GPGs were a relatively new topic in the develop- cautioned against messages or incentives that ment agenda, and the Bank needed to gain may induce staff to avoid risks necessary to experience about what may work well, including achieve development impact. One member enhancement of country ownership and the noted that (i) for measuring complex projects, demand-driven approach, as well as strength- there was a question about how to integrate a ened partnerships. They also stressed the need risk premium into the rating system, to ensure to identify the Bank's comparative advantage in quality but without undermining the risk-taking the GPG work, given that the Bank was only one incentives to staff; (ii) innovative approaches of many players in this field. One member was were needed to address development issues in disappointed about the perception that difficult countries, and (iii) the challenge was in "networks are not working." Some members designing interventions adapted to different encouraged the identification of sectoral categories of countries (for example, LICs, MICs, comparative advantages beyond health and or fragile states). climate change, such as market stabilization of key commodities like oil and grains, and the Country Programs. One member noted, with development of innovative financing. If GPGs are concern, IEG's finding that not a single program in conflict with a country's interest, there is a concentrated in smaller countries or countries question of ownership, incentives, and percep- with extensive poverty has been rated "satisfac- tion of tradeoffs between GPGs and develop- tory" in meeting their stated development ment. In this sense, one member felt the Bank objectives. Given the long engagement of the was focusing on "rounding a square." This Bank with LICs, the question arose about member also felt that it was difficult to whether there are some systemic issues and operationalize IEG's recommendations to whether these have been suitably identified. improve the Bank's support for GPGs. M&E and Results Management. Members suggested GPG Country-Based Model. Members felt that the that the use of impact evaluation, though still at country-based model was a clear comparative an early stage, could improve understanding of advantage of the Bank's work on GPGs. the outcomes of projects, programs, and policies. However, they regretted that systems for The importance of disseminating the methodolo- integrating GPGs into the country-based model gies of impact evaluation was noted. One were underdeveloped when country and global member observed that the quality of M&E was interests diverge significantly. In this context, associated with the quality of outcomes, and some members underscored the importance of expressed concern that while measuring interme- addressing the limitations of the current diate and final outputs require appropriate data business model to promote GPGs, while gathering, these activities were not built into avoiding imposing conditionalities and supply- project design. While acknowledging that results driven approaches. There was a sentiment that on the ground and success may be hard to attrib- the country-based model was not enough to ute to interventions by the Bank, he questioned address GPGs and should be complemented how it can be a knowledge bank if it cannot learn with a global-level framework, which will provide from its successes and failures. This member incentives to countries, including financial xxv ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 mechanisms for incremental costs. In addition, a Speakers welcomed the new format but would question was raised about whether the issue was have liked greater substantive integration of the the effectiveness of Bank instruments rather two parts. A more specific suggestion was to than the country model. deepen the analysis of several key points in Part I, to make the report a more robust instrument of Resource Allocation for GPGs. There were management's accountability. One member comments on the need for a new approach in found that the aggregate view of the Bank's allocating and tracking spending on GPGs (both development effectiveness did not give an indica- budget and trust funds), and in recognizing tion of how various areas of the Bank perform, performance, but there were a variety of views as what lessons could be drawn from the report, and to how this should be done. One member shared thus how resources should be allocated. A few management's reservation about IEG's suggested members sought clarification on why the report option of setting aside significant administrative was not accompanied by the Management Action funding for allocation of high-priority GPG work Record for monitoring progress of the Bank's at the country level. agreed actions. IEG provided additional informa- tion regarding the implementation report that ARDE Format. The new structure of the report sets out which older IEG recommendations will generally found favor with the Committee. be retired from formal consideration. Jiayi Zou, Chairperson xxvi Evaluation Snapshot in Selected Languages T he key findings and recommendations of the Annual Review of Development Effectiveness 2008: Shared Global Challenges are pre- sented in snapshot below. Translations of the report's full summary into each of the languages shown here are available at www.worldbank.org/ieg, and hard copies are available from IEG and World Bank Public Information Centers. Arabic Chinese xxvii ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 English French Français · Reducing poverty in any individual country is Dans chaque pays, la réduction de la pauvreté est increasingly intertwined with making progress de plus en plus étroitement liée aux progrès réa- on shared global challenges--that is, fostering lisés face aux grands problèmes communs à l'en- global public goods (GPGs) such as climate semble de la planète ­ autrement dit, à la protection and communicable disease con- promotion des biens publics mondiaux, tels que trol. This year's ARDE tracks World Bank per- la protection du climat et la lutte contre les ma- formance in Part I and examines the Bank's ladies communicables. Cette année, l'Examen an- work in fostering GPGs in Part II. nuel de l'efficacité du développement consacre · Development outcomes from Bank lending have improved over the medium term. But in sa Première partie à suivre les résultats obtenus fiscal 2007 overoptimism in the Bank's ongo- par la Banque mondiale dans ce domaine et sa ing assessment of project performance rose Deuxième partie à l'action menée par la Banque sharply, while the share of projects rated mod- pour promouvoir les biens publics mondiaux. erately satisfactory or better dropped to 76 Un bilan des résultats obtenus par les prêts de percent from 83 percent a year earlier. la Banque laisse apparaître une amélioration sur · Vigilance is needed to identify problem proj- le moyen terme. Toutefois, au cours de l'exer- ects in real time and ensure that the fiscal cice 2007, l'excès d'optimisme affiché par la 2007 drop in performance does not fore- Banque dans son évaluation de la performance shadow a persistent decline. Practical steps de ses projets a grimpé fortement alors même can be taken to better use M&E in projects and que le pourcentage de ses projets classés comme programs, including proper baseline infor- modérément satisfaisants ou mieux est tombé à mation and clearer links between outputs and 76% contre 83% l'année précédente. outcomes. Il faut faire preuve de vigilance si l'on veut iden- · The Bank's country-based model has worked tifier les projets à problème en temps réel et relatively well in fostering global public goods veiller à ce que la baisse de résultats de 2007 ne when national and global interests dovetail soit pas annonciatrice d'un déclin persistant. On and grants support country investments. But peut prendre des mesures pratiques afin de the greatest challenges, such as climate change, mieux utiliser le Suivi et l'Évaluation dans les arise where local, national, and global projets et programmes, notamment une bonne benefits--actual or perceived--diverge sig- information de base et des liens clairement éta- nificantly. Here the country model comes under considerable strain. blis entre les produits et les résultats. · To more effectively bridge the gap between Le modèle par pays de la Banque contribue assez global needs and country concerns, the Bank efficacement à promouvoir les biens publics should consider: creating dedicated budgets mondiaux lorsque les intérêts nationaux concor- and better incentives for country teams to dent avec les intérêts mondiaux et que des dons work on GPGs; deploying its global knowl- soutiennent les investissements du pays. En re- edge networks better; and using its standing vanche, les problèmes les plus graves, tels que more powerfully to give greater voice to de- le changement climatique, se posent lorsque les veloping countries in the governance of global intérêts nationaux et mondiaux ­ réels ou per- programs. çus ­ divergent sensiblement. En pareil cas, le modèle par pays est mis à rude épreuve. our mieux combler le fossé entre les besoins mondiaux et les préoccupations nationales, la Banque devrait envisager : d'établir des bud- gets spécifiques et de meilleures incitations pour les équipes-pays à travailler à la réalisation des biens publics mondiaux ; de mieux utiliser ses réseaux mondiaux de savoir ; et d'utiliser plus énergiquement sa position afin de donner plus de voix aux pays en développement dans la ges- xxviii tion des programmes mondiaux. EVALUATION SNAPSHOT IN SELECTED LANGUAGES Portuguese Português Spanish Español redução da pobreza em qualquer país está La reducción de la pobreza en un país deter- cada vez mais vinculada ao progresso em en- minado está cada vez más relacionada con los frentar desafios globais compartilhados ­ ou progresos que se hagan con respecto a los de- seja, promover os bens públicos globais (GPGs), safíos mundiales comunes, es decir, la promo- tais como proteção climática e controle de do- ción de los bienes públicos mundiales (BPM), enças transmissíveis. Revisão da Eficá- como la protección del clima y la lucha contra cia do Desenvolvimento deste ano las enfermedades transmisibles. En la Parte I de analisa o desempenho do Banco Mundial na la versión del ARDE de este año se hace un se- Parte I e o trabalho do mesmo na promoção dos guimiento del desempeño del Banco Mundial GPGs na Parte II. y en la Parte II se pasa revista a su labor de pro- Os resultados do desenvolvimento oriundos da moción de los BPM. concessão de empréstimos pelo Banco Mundial os resultados en términos de desarrollo de las melhoraram no médio prazo. Mas no exercício operaciones de financiamiento del Banco han financeiro de 2007 o superotimismo da ava- mejorado en el mediano plazo. Sin embargo, en liação contínua do Banco Mundial no tocante el ejercicio de 2007, el exceso de optimismo de ao desempenho dos projetos aumentou con- las autoevaluaciones en curso con respecto al de- sideravelmente, ao passo que a parcela de pro- sempeño de los proyectos aumentó marcada- jetos classificados como moderadamente mente, mientras que la proporción de proyectos satisfatórios ou melhores caiu para 76% com re- que recibieron una calificación de moderada- lação a 83% do ano anterior. mente satisfactorios o superior bajó al 76%, en preciso vigilância para identificar projetos comparación con el 83% en el ejercicio anterior. problemáticos em tempo real e assegurar que Se debe prestar atención para detectar de in- a queda de desempenho ocorrida no exercício mediato los proyectos que presentan problemas financeiro de 2007 não seja um prenúncio de y velar por que la baja registrada en el ejercicio um declínio persistente. Podem-se tomar me- de 2007 no sea un presagio de un deterioro per- didas para melhorar o uso de Monitoramento sistente. Se pueden tomar medidas prácticas e valiação em projetos e programas, in- para utilizar de mejor manera los sistemas de se- clusive informação básica adequada e vínculos guimiento y evaluación en los proyectos y pro- mais claros entre produtos e resultados. gramas, como recopilar información básica modelo baseado no país, utilizado pelo Banco adecuada y definir más claramente los vínculos Mundial, tem funcionado relativamente bem na entre los productos y los efectos directos. promoção dos bens públicos globais em uma El modelo del Banco centrado en los países ha época em que os interesses nacionais e globais funcionado relativamente bien para promover se concatenam e subvenções apóiam investi- los BPM cuando se han conjugado los intereses mentos dos países. No entanto, os maiores de- nacionales y de alcance mundial y las inversio- safios, tais como a mudança climática, surgem nes en los países se han respaldado con dona- quando os benefícios locais, nacionais e globais ciones. Sin embargo, los mayores desafíos se ­ reais ou percebidos ­ divergem de forma sig- plantean cuando los beneficios locales, nacio- nificativa. Neste aspecto o modelo de país sofre nales y mundiales ---ya sean reales o percibi- pressão considerável. dos-- difieren considerablemente. En esos casos, ara cobrir com mais eficácia o hiato entre as el modelo se debilita de manera significativa. necessidades globais e as preocupações dos pa- ara salvar de manera más eficaz la brecha entre íses, o Banco Mundial deve considerar mu- las necesidades de alcance mundial y los temas danças tais como: Criar orçamentos dedicados que interesan a los países, el Banco debería con- e maiores incentivos para as equipes de país tra- templar la posibilidad de establecer presupuestos balharem na promoção dos GPGs; melhorar a especiales y ofrecer mejores incentivos para que implantação de suas redes globais de conhe- los equipos a cargo de las operaciones del Banco cimentos; e utilizar sua posição de forma mais en los países se dediquen a los PBM; utilizar sus incisiva para dar maior expressão aos países em redes de conocimientos mundiales de manera desenvolvimento na governabilidade de pro- más eficaz, y aprovechar de manera más enér- gramas globais. gica su posición para dar mayor voz a los países en desarrollo en la gestión de los programas mundiales. xxix ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Russian xxx Chapter 1 Evaluation Highlights · Development success in any indi- vidual country is increasingly inter- twined with making progress on shared global challenges. · The Annual Review of Development Effectiveness is in a new, two-part format. · Part I tracks World Bank perform- ance by analyzing projects, country programs, and results from monitoring and evaluation. · Part II examines a special topic: the Bank's work in fostering global public goods. Chinese lanterns at night; photo ©Frank Krahmer/zefa/Corbis, reproduced by permission. Introduction F or the World Bank and its partners, the ever-present test is to deliver results--to lift people out of poverty and promote socially and environ- mentally sustainable development. Achieving such success in any individual country is increasingly intertwined with making progress on shared global chal- lenges. A fair and efficient international trade regime, for example, is a global public good (GPG) that allows developing countries to trade more and grow faster. The growing global threat of climate change--a "public bad" by contrast-- particularly imperils the poor who bear the brunt of more frequent natural dis- asters and hazards to health and agriculture (IEG 2006b; Stern 2006). This year's Annual Review of Development All of this is important because a successful M&E Effectiveness (ARDE) is in a new format and system that focuses on results can have three presents evidence on the Bank's efforts in two major benefits. It can provide an early warning important and connected areas. Part I, which is a when an intervention is not working well, and a standard section of the new format, analyzes the basis for improving a project during implementa- trends in outcomes of Bank projects and country tion. It can provide a broad measure of results for programs in recent years. Equally important-- accountability and demonstrate to stakeholders and building on past practice1--the report goes the degree to which the Bank has contributed to beyond performance indicators and digs deeper development results. Finally, it can contribute to into the framework employed by the Bank to global knowledge for development, providing monitor and evaluate its work. This ARDE useful lessons for other projects and programs assembles the available evidence on how within the same country, or within the same monitoring and evaluation (M&E) systems are sector in other countries. progressing, both in terms of quality and coverage. It illustrates how new techniques are Part II of the report examines a special topic. beginning to be employed, including impact There has been enormous growth in interna- evaluations and new results management systems tional attention to the great shared global for the International Development Association challenges of our time--a set of supranational (IDA). The review also refers to evidence to issues collectively known as global public goods. benchmark the Independent Evaluation Group's This year's ARDE looks at the Bank's work in (IEG's) own contribution to influencing the fostering GPGs, such as protecting the earth's Bank's performance. climate and preventing the spread of dangerous 3 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 communicable diseases. GPGs are clearly fostering GPGs as one of its six strategic themes. important in their own right and are becoming This report focuses on two roles the Bank has ever more closely associated with the Bank's identified as its comparative advantage: (i) development impact (as described in Part I). To identifying how the Bank's country-based model fight the many dimensions of poverty-- has helped or hindered its support for GPGs, and including vulnerability--the world must be able (ii) drawing lessons from the Bank's experience as not only to help antipoverty programs at the an advocate for action on GPGs. individual country level, but also to create the framework for collective action on transnational The thematic focus of ARDE 2008, taken together issues that can support or undermine the fate of with the components of Part I of the report, the poor. helps inform important choices for the Bank's strategy and its implementation. It also sets a As the Bank has emphasized inclusive and sustain- framework for work by IEG in other able globalization, the importance of coordinated evaluations--including next year's ARDE--to global solutions to cross-border problems has focus attention on development impact in the risen in tandem. Indeed, the Bank has identified future. 4 PART I Tracking Bank Performance Chapter 2 Evaluation Highlights · Project outcome ratings improved over the medium term. · Attention is needed to address the growing lack of realism in assess- ments of ongoing project perform- ance. · Unsatisfactory projects in fiscal 2007 were hampered by overly ambitious objectives, complexity, and inade- quate design. · Many country programs produced satisfactory outcomes, especially those in large countries where many of the world's poor live. · Overall, however, some 40 percent of country programs have been mod- erately unsatisfactory or worse in outcomes. Tuareg child turns water pump wheel; photo ©Yann Arthus-Bertrand/Corbis, reproduced by permission. Development Outcomes: Indicators of Performance O ver the past five years, the Bank's lending has risen in real terms, and IDA made record commitments of $12 billion in fiscal 2007, alongside some $13 billion of International Bank for Reconstruction and De- velopment (IBRD) finance provided to middle-income countries (MICs). In- deed, with the successful 15th replenishment round for IDA resources ($42 billion), the Bank plans to maintain total annual lending in the $22 to $26 bil- lion range during fiscal 2008­10. Lending volumes by themselves, of course, are at the start of the decade, as shown in Lending outcomes have not an indicator of development impact, and figure 2.1. improved over the there have been many cautions for the Bank (and medium term. other development agencies) to take care to avoid a "lending culture" which places undue Figure 2.1: Project Performance Has Improved over emphasis on volumes delivered rather than the MediumTerm outcomes secured.1 There are indicators of performance that can tell us about the Bank's 90 development impact--notably IEG evaluations of Bank-supported projects and country (%) programs. Overall trends and recent develop- ments revealed in both of these metrics are better outcomes 80 or presented below. with Measuring Project Performance: Trends from IEG Monitoring projects satisfactory 70 of How are Bank-supported projects performing in terms of achieving their development objectives? Share moderately The performance of Bank projects in delivering 60 development results has unquestionably im- 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 proved over the medium term.2 In the three years Exit fiscal year (last in a 3-year rolling average) to end-fiscal 2007, 80 percent of projects were By number of projects Weighted by value of project disbursements moderately satisfactory or better in delivering Source: World Bank database. their targeted results, up from around 70 percent 9 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Project ratings moved up A comparison of five-year subperiods The Bank's lending is spread across a spectrum of the scale in recent years. shows 78 percent of projects from sectors. It is encouraging that, as shown in figure fiscal 2003 to 2007 rated as moderately 2.2, project performance in about two-thirds of satisfactory or better in achieving their develop- sectors has improved over the medium term.5 ment results. This is a significant improvement The turnaround in the water supply and sanita- from fiscal 1998­2002, in which 73 percent of tion sector projects is especially dramatic: in fiscal projects had outcomes rated moderately satisfac- 1998­2002 only a little more than 60 percent of tory or better. Illustrations of development projects (by value of disbursements) were impact from some of these projects are in box moderately satisfactory or better, and this sector 2.1. Moreover, a growing share of projects has now comes close to leading the way in fiscal outcomes that are sustainable.3 Almost 80 2003­07, with over 90 percent of projects having percent of projects evaluated in fiscal 2003­07 moderately satisfactory or better outcomes. were rated as likely to be sustainable, up from 64 While there has been a creditable and significant percent in the previous five years.4 A more uplift in economic policy project performance, detailed assessment of project performance is the share of moderately satisfactory or better available in appendix A. project outcomes in fiscal 2003­07 still trails other sectors by quite some margin. The largest Against the backdrop of this overall improvement declines in performance were in health, nutrition, in outcomes, there are several features that and population (which, along with economic illustrate changes in the performance of Bank- policy and the environment, were significantly supported projects, as shown in table 2.1. There below the Bank-wide average for the fiscal has been a steep decline in the share of projects 2003­07 period) and in public sector governance. rated unsatisfactory while, simultaneously, the share reaching the satisfactory standard held Looking at the Regional aspects of project steady. In addition, the Bank and its clients have performance, figure 2.3 shows that the East Asia improved a number of projects, which in and Pacific and the Europe and Central Asia Outcomes of projects in the past may have been rated moderately Regions led the way for the fiscal 2003­07 cohort. two-thirds of sectors unsatisfactory, to reach the level of Their share of projects with moderately satisfac- improved, while health moderately satisfactory. As a result, a tory or better outcome ratings significantly and public sector greater proportion of projects is being exceeded the Bank average of 83 percent governance worsened. rated in the middle of the scale. (weighted by disbursement). A good illustration Box 2.1: What Does a Satisfactory Project Look Like? Illustrations of Development Impact There is no single metric that can be used across projects to People's Democratic Republic provided main-power-grid elec- assess satisfactory outcomes. Rather, development results span tricity to 51,000 households in 721 rural villages, while another a range of different social and economic indicators depending 6,000 households gained access to off-grid electricity through on the sector and type of project. A project is rated satisfactory solar and hydro systems. Having electricity greatly increased pro- when the operation's objectives have been achieved with only ductivity of new small-scale, home-based businesses, and en- minor shortcomings. This is illustrated with one type of devel- abled children to study at night. opment impact in Mozambique, where Bank-supported water and Finally, another type of development impact is demonstrated in sanitation projects built 130 water source points, bringing drink- Mali, where the Bank-supported Grassroots Initiatives to Fight ing water to 62,000 people. And in Cambodia, a Bank-financed proj- Poverty and Hunger Project helped more than 6,000 children to at- ect brought clean water to 750,000 people in Phnom Penh. tend school, as well as created small-scale health centers in 19 In the energy sector, a Bank electrification project in the Lao villages. 10 DEVELOPMENT OUTCOMES: INDICATORS OF PERFORMANCE Table 2.1: Distribution of Project Ratings Moved Up the Scale in FY03­07 IEG rating of projects meeting Percentage of Projects in Rating Category development objectives FY98­02 FY03­07 Percentage point change Highly satisfactory 6% 4% ­2 Satisfactory 45% 45% 0 Moderately satisfactory 21% 29% +8 Moderately unsatisfactory 9% 10% +1 Unsatisfactory 16% 10% ­6 Highly unsatisfactory 2% 1% ­1 Source: World Bank database. of such successful performance can be seen in new plant varieties and cultivation techniques, an agricultural development project in China's and added value to production through process- Anning Valley. The Bank's $120 million of support ing. Local farmers in the project's area enjoyed a supplied more reliable irrigation, introduced tripling in their per capita income; the propor- Figure 2.2:Trends in Sectoral Performance Share of projects with outcomes moderately satisfactory or higher (weighted by disbursement) Social protection Transport Financial and private sector development Water supply and sanitation Agriculture and rural development IMPROVERS Bank-wide Energy and mining Economic policy Environment board 40% 50% 60% 70% 80% 90% 100% Sector Health, nutrition, and population Public sector governance DECLINERS Urban development Education 40% 50% 60% 70% 80% 90% 100% FY98­02 FY03­07 Source: World Bank database. Note: The Sector Board classification applies to the whole project and enables outcomes to be matched to it. 11 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Figure 2.3: Africa's Projects Have Improved Substantially but Still Lag Behind Other Regions East Asia and Pacific Europe and Central Asia Bank-wide South Asia Region Latin America and the Caribbean Middle East and North Africa Sub-Saharan Africa 40 50 60 70 80 90 100 Share of projects with outcomes moderately satisfactory or better (weighted by disbursements, %) FY98­02 FY03­07 Source: World Bank database. tion of the area's families living in poverty fell northern Ethiopia. By removing landmines, from one-third to one-tenth; and the project building new homes, and equipping households generated a 27 percent rate of return on its with basic goods and agricultural inputs--includ- investment. ing seeds and fertilizer--some 67,000 families returned home, rebuilt their lives, and generated In these Regions, MICs make up a large share of the new economic activity, which delivered a 50 client base, and their relatively strong institutional percent return on the project's investment. capacity is conducive to stronger project implementation. Nonetheless, the Bank could well Project Outcomes in Fiscal 2007 Data be faced with increasing demands from these Project development outcome ratings fluctuate, clients to take on projects that are complex or sometimes significantly, from year to year. Differ- located within a country's lagging or remote ences are sometimes a result of the vicissitudes subregions, under difficult operating conditions. of a particular cohort of projects rather than a These are challenges for the Bank to embrace, but substantive change in underlying institutional or expectations for the success of projects must be partner performance. So while a single year's adjusted to recognize concomitant risks. data should not be taken in isolation, the most recent data are worth examining to see whether Africa's performance In the Africa Region, the share of any patterns or signals emerge. There are two improved the most of projects with moderately satisfactory movements in fiscal 2007 that warrant the any Region but still or better outcomes improved the attention of Bank management.6 lags behind other parts most of any Region over fiscal of the Bank. 2003­07. Again, it should be acknowl- First, increasingly the Bank is too optimistic in its edged that difficult operating own assessment of ongoing project performance. conditions can make it difficult to secure Of the 45 projects that exited in fiscal 2007 and success--and that may at least partly explain why were rated as moderately unsatisfactory or worse project performance, overall, in Africa continues by IEG, over two-thirds--32 projects in total--had to lag behind all other Regions. But the Bank has been reported by the Bank as moderately satisfac- to find ways to deliver outcomes in adversity, as a tory or better just before they were closed. This central part of its mission. Indeed, this can be failure to identify problem projects early impairs done, as illustrated in the Bank's $220 million real-time managing for results, since overly investment to assist war-affected households in optimistic ongoing ratings mean management is 12 DEVELOPMENT OUTCOMES: INDICATORS OF PERFORMANCE less likely to take remedial action. Most Regions portfolio from one year to the next Ongoing self-assessments were too optimistic in their ongoing reporting of can be influenced by a change in the of project performance project ratings for fiscal 2007, with almost a fifth of composition of projects being are increasingly overly all projects--and in Africa close to a third-- evaluated--for example, if there is a optimistic. downgraded during the process of self-evaluation larger share of projects in challenging and independent evaluation.7 Comparatively, sectors or countries. But the change in portfolio these "disconnect" rates in previous years were composition in fiscal 2007--related to Region, much lower--less than 1 in 10 were downgraded sector, instrument, lending arm, and other in fiscal 2005 and 2006, as shown in table 2.2. factors--does not explain the fall in ratings. Even Reducing this disconnect--by more carefully and if the fiscal 2007 cohort had maintained the same accurately identifying underperforming projects composition of lending to conflict-affected and during implementation--is needed to focus postconflict countries as the fiscal 2004­06 attention on project supervision and to improve cohorts, the result would not be materially performance. Of course, to subsequently improve affected. Appendix A shows the impact of various development outcomes will then require project- changes in the fiscal 2007 composition in more specific actions on the ground. detail. Second, in fiscal 2007 there was a significant fall Another possible explanation is that the drop in to 76 percent in the share of projects rated measured project performance is due to method- moderately satisfactory or better, from 83 ological changes in the way projects percent in fiscal 2006.8 This absolute level of were evaluated in fiscal 2007. Some Project outcomes performance still meets the 75 percent target set changes in methods were introduced deteriorated in fiscal by the Bank a decade ago, although it is lower by IEG and the Bank together in fiscal 2007. than the 80 percent benchmark noted by the 2007 to strengthen the robustness of Bank's own Quality Assurance Group (QAG) in project ratings and to cover new elements of recent annual reviews. If a 75 percent threshold project design. In the near term they may have for project portfolio performance is considered introduced some element of discontinuity in the reasonable, given the risks of the "development data series between fiscal 2007 and earlier years.9 business," then this one-year drop, by itself, is It is estimated that the influence of methodology not too worrisome. But certainly vigilance is changes has been small--accounting for around needed to ensure that this drop does not 1 percentage point of the fall. foreshadow a persistent decline. The final possibility is that in the fiscal 2007 What has caused the drop in ratings in fiscal 2007 cohort, there was simply a greater occurrence of and produced 45 projects with outcomes five key factors influencing weak outcomes. First, moderately unsatisfactory or worse? Sometimes poor or overly complex project design has been a a difference in the performance of the Bank's problem in more than half of these underper- Table 2.2: Disconnect between the Bank's Self-Ratings and IEG Ratings Increased Dramatically in FY07 Share of projects rated moderately satisfactory or better Source of rating FY05 FY06 FY07 Bank's final Implementation Status and Results report (ISR) 87.9% 91.1% 93.0% Bank's Implementation Completion and Results report (ICR) 85.7% 89.9% 83.3% IEG's ICR review 80.8% 82.6% 75.8% Difference between ISR and ICR review ratings--"disconnect" ­7.1% ­8.5% ­17.2% Source: World Bank database. 13 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Five factors have forming loans, a finding also made by projects (and many such projects were not identi- influenced weak project the World Bank's QAG (World Bank fied as problems in ongoing status reports). And outcomes. 2008a). For instance, two projects Bank overall performance--as distinct from failed to recognize the importance of borrower performance or the effects of uncontrol- an appropriate legal and regulatory framework as lable events--was ranked moderately unsatisfac- a precondition to a privatization process. Several tory or worse in two-thirds of these problem health projects failed to ensure a heightened projects, compared with only about one-fifth of focus on those interventions that would yield the the full sample. All of this points to a challenge in greatest impact, leading, for instance, to re-emphasizing a proactive quality control in inadequate targeting of the poor, the absence of a management's attention to ongoing project cost effective package of health services, or performance. inadequate funding for behavior change interven- tions to prevent HIV/AIDS transmission among How It Adds Up: Outcomes of Bank high-risk groups. Country Programs While Bank-supported projects have largely Second, overambition was a weakness. While yielded positive development outcomes, country project objectives were almost always relevant, a program outcomes--measured against their own majority were too far-reaching. Sometimes this objectives, which typically include growth, was in terms of assessing political commitment poverty reduction, and the environment--are far and the feasibility of certain reforms. Other times less satisfactory. Some previous assessments it was in assessing government effectiveness and suggest that synergies between and among the capacity, or in requiring coordination across Bank's lending, knowledge services, and dialogue several ministries or cumbersome financial are not fully exploited or that the projects are not management procedures that were not manage- always directly relevant to the country's core able by the parties involved. IEG's evaluation of development challenges (World Bank 2008a; IEG public sector reform (2008c) shows several 2006c). examples where these factors led to unsatisfac- tory outcomes. The lower country-program ratings could also be influenced by other factors. The number and Third, delays in implementation caused difficul- scale of the factors that affect broader country- ties, because circumstances changed and project level objectives is typically much greater than at design or implementation could not respond. the smaller-scale project level. Hence, outcomes About one-fifth of underperforming projects measured at this level are more likely to be suffered from this problem. affected by pressures outside the control of development partners. It is also possible that Fourth, a majority of the unsatisfactory projects objectives may be relatively more ambitious at the had a weak results framework with poor or no program rather than project level (and therefore baseline data, making it difficult to assess the less frequently attained) or indeed that it is more outcomes of the project, and outcomes were difficult to establish the connection between often not well linked to inputs and outputs. Bank inputs and outcomes at the program level (as opposed to at the project level). Finally, various gaps in the Bank's own perform- ance contributed to a lack of success. For example, Over the past 10 years, evaluations of 81 Bank despite being flagged by the QAG for poor quality country programs--incorporating projects, at the outset, three projects were not reassessed policy and technical advice, and other types of or redesigned. The quality of the Bank's supervi- assistance--show that three-fifths of them were sion was rated as moderately unsatisfactory or moderately satisfactory or better in meeting their worse in two-thirds of all underperforming development objectives. Looking at specific 14 DEVELOPMENT OUTCOMES: INDICATORS OF PERFORMANCE Figure 2.4: CAEs ShowThree-Fifths with Outcomes Moderately Satisfactory or Better Distribution of CAE ratings, FY98­08 (23) Albania (98­04) (22) (21) Bhutan (93­03) Argentina (91­00) 20 ­ Bulgaria (91­97) Bolivia (85­96) Armenia (93­02) Cameroon (82­94) Burkina Faso (89­99) Bosnia­Herz.(96­03) CAEs Costa Rica (90­00) Cambodia (92­99) Brazil (90­02) ni Croatia (94­01) Cameroon (95­00) Bulgaria (98­01) gsn Ecuador (94­98) Dom. Republic (85­02) China (93­02) atir 15 ­ Guatemala (85­89) Egypt, Arab Rep. (91­00) Croatia (02­03) Haiti (86­01) Eritrea (92­00) Ethiopia (90­00) amr Jamaica (80­98) India (90­00) Ghana (95­99) ogrp Malawi (96­05) (12) Indonesia (90­98) Guatemala (90­01) yrt Mauritania (92­03) Angola (91­06) Indonesia (99­06) Lithuania (91­02) un 10 ­ Moldova (93­03) Bolivia (98­04) Jordan (90­00) Maldives (80­98) co Nepal (90­99) Ethiopia (01­06) Kazakhstan (90­99) Mexico (89­91) ofr Pap. New Guinea (90­99) Honduras ( 95­05) Kyrgyz Rep. (93­00) Mexico (97­00) Paraguay (90­00) Lesotho (90­99) Mali (95­05) Peru (90­96) mbeu Peru (97­00) Madagascar (95­05) Mexico (95­96) Romania (00­04) N 5 ­ Romania (91­99) Mexico (92­94) Mongolia (91­01) Russian Fed. (99­01) Russian Fed. (92­98) Morocco (97­00) Rwanda (95­01) Tunisia (90­03) Rwanda (90­94) Pac. Islands (92­02) Senegal (94­04) Uganda (87­99) (3) Ukraine (93­98) Pakistan (94­03) Sri Lanka (89­98) Vietnam (88­01) Chile (85­00) Zambia (96­01) Yemen, Rep. (90­95) Turkey (93­04) W. Bank & Gaza (93­00) El Salvador (89­00) (0) Zimbabwe (90­02) Yemen, Rep. (99­05) Ukraine(99­06) Yemen, Rep. (96­98) Uruguay (87­99) Highly Unsatisfactory Moderately Moderately Satisfactory Highly unsatisfactory unsatisfactory satisfactory satisfactory Outcome rating of county programs in CAEs Source: IEG Country Assistance Evaluations. Note: CAEs assess country program performance over a long period. Years in parentheses indicate the period that Bank country programs are assessed. grades on IEG's ratings scale, as shown in figure country programs were rated moderately 2.4 and table 2.3, the Bank succeeded in support- satisfactory. But the remaining 40 percent of ing satisfactory outcomes in 30 percent of programs--concentrated in countries that are evaluated programs; a further 30 percent of smaller or have extensive poverty, such as Table 2.3: Summary of CAE Ratings, FY98­08 MICs LICs All countries Highly satisfactory 6% 0% 4% Satisfactory 33% 18% 27% Moderately satisfactory 25% 33% 28% Moderately unsatisfactory 13% 18% 15% Unsatisfactory 23% 30% 26% Highly unsatisfactory 0% 0% 0% Satisfactory and highly satisfactory 39% 18% 31% Moderately satisfactory or better 64% 51% 59% Source: IEG Country Assistance Evaluations. 15 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Box 2.2: What Does a Satisfactory Country Program Look Like? No single template exists for a satisfactory Bank-supported nearly 100 percent of gross domestic product in 1999, to under 50 country program, since such programs vary greatly depending percent by 2005. These gains largely managed to turn around the on the country's institutional capacity, stage of development, and effects of the crisis--the percentage of Indonesians living on less particular development needs. The characteristics of the Indone- than one dollar per day was 16 percent in 2005, compared with 23 sia country program illustrate some features of good outcomes percent in 1999. The fall in income poverty and improvement in found in several cases. human development indicators were assisted by the Bank's work During fiscal 1999­2006, the Indonesia country program was in community-driven development through the Kecamatan Devel- largely successful. Bank support provided help to national au- opment Program and by reconstruction following the 2004 tsunami. thorities in securing economic recovery from the late 1990s eco- These achievements outweighed some areas in which outcomes nomic crisis. By 2004, per capita income achieved precrisis levels, from the Bank's program were less satisfactory--including in de- inflation fell to under 7 percent, and public debt was cut from centralization and fighting against corruption. Outcomes of country Malawi--were moderately unsatisfac- satisfactory or better, this was true of only 18 programs are less tory or worse in meeting their stated percent in LICs. Using a lower threshold, the differ- satisfactory than project development objectives. ence is smaller but still substantial: 64 percent of outcomes. ratings in MIC programs were moderately satisfac- Very few country programs are tory or better, compared with 51 percent of ratings producing best-practice results--indeed, of 36 for LIC programs. programs rated since fiscal 2002, not one has been highly satisfactory. And among the 14 low- A less in-depth but more up-to-date snapshot of income countries (LICs) rated since fiscal 2002, country program outcomes is provided by IEG not a single program has been rated satisfactory reviews of Country Assistance Strategy Comple- or better. At the same time, no program has ever tion Reports (CASCRs). CASCR reviews differ from been rated highly unsatisfactory. Country Assistance Evaluations (CAEs) in that they are a review of the Bank's own self-evaluation of Many programs in the largest countries that house its country assistance program and cover a shorter the majority of the world's poor--including Brazil, time period, typically three to five years, as China, India, Indonesia, and Mexico--were rated compared with a decade or more in CAEs. CASCR moderately satisfactory or better in their develop- review data, available since fiscal 2004, present a ment outcomes. Box 2.2 shows such outcomes slightly more favorable assessment. Two-thirds of from the Indonesia country program. When all programs reviewed were rated moderately country programs are weighted by the number of satisfactory or better, as shown in figure 2.5 and people in poverty (living on less than $2 per day), table 2.4.10 Here too, none of the country 86 percent of country programs are given a programs were rated highly unsatisfactory. satisfactory rating. Even when exclud- Few country programs ing China and India, both above the ConsistentwiththeCAEs,higherratingsarealmost have produced best- line, the poverty-weighted satisfactory entirely due to better outcomes in MICs than in practice results. share is nearly two-thirds. LICs. More than three-fourths of the MICs had moderately satisfactory or better country out- There are also significant differences among comes, compared with half of LICs reviewed. And subgroups of countries, as shown in table 2.3. Bank in only one MIC was Bank assistance rated unsatis- programs in MICs far outperform those in LICs. factory, as compared with four LICs, even though While IEG rated 39 percent of programs in MICs there were many more MICs in the sample. 16 DEVELOPMENT OUTCOMES: INDICATORS OF PERFORMANCE Figure 2.5: CASCR Reviews IndicateThat Bank Programs in MICs OutperformThose in LICs Distribution of CASCR Review ratings, FY04­08 (26) Albania (FY02­05) 25 ­ Argentina (4/04­12/05) Azerbaijan (FY03­05) Bangladesh (FY01­04) eviewsr Belarus (FY02­06) Bhutan (FY00­05) 20 ­ Cameroon (FY96­02) CASCR Egypt, Arab Rep. (FY02­04) ni Ghana (FY04­07) gsn Honduras (FY03­06) (15) Jordan (FY03­05) atir 15 ­ Bosnia­Herz. (FY05­07) Macedonia FYR (FY04­06) amr Gabon (FY99­04) Madagascar (FY04­06) ogrp Gambia, The (FY03­07) Maldives (FY00­07) Georgia (FY98­05) Mali (FY04­07) (11) yrt Guatemala (FY99­03/4) Mauritius (FY02­04) Armenia (FY02­04) 10 ­ un Kyrgyz Republic (FY03­06) Mozambique (FY04­07) Brazil (FY00­03) co Malawi (FY04­06) Nicaragua (FY03­07) Bulgaria (FY03­05) ofr Mauritania (FY03­07) Pakistan (FY02­05) Burkina Faso (FY01­05) Morocco (FY01­04) Peru (FY03­06) Chile (FY02­06) mbeu (5) Org. E. Carib. States (FY02­05) Philippines (FY03­05) China (FY03­05) N 5 ­ Cambodia (FY00­03/05) Russian Federation (FY03­06) Romania (FY02­04) Montenegro (FY05­07) Lesotho (FY98­05) São Tomé & Princ. (FY01­05) Senegal (FY02­06) Tanzania (FY03­06) Nigeria (FY99­04) South Africa (FY00­06) Serbia (FY05­07) Turkey (FY04­07) Uzbekistan (FY02­05) Uganda (FY01­03) Tajikistan (FY03­05) Uruguay (FY01­05) (0) Zambia (FY00­03) Yemen, Rep. (FY03­05) Ukraine (FY04­07) Vietnam (FY03­06) (0) Highly Unsatisfactory Moderately Moderately Satisfactory Highly unsatisfactory unsatisfactory satisfactory satisfactory Outcome rating of county programs in CASCR reviews Source: IEG reviews of Country Assistance Strategy Completion reports. Note: CASCR Reviews cover a shorter period than CAEs. Years in parentheses indicate the period that Bank country programs are assessed. Table 2.4: Summary of CASCR Review Ratings, FY03­08 MICs LICs All countries Highly satisfactory 0% 0% 0% Satisfactory 22% 13% 18% Moderately satisfactory 54% 39% 48% Moderately unsatisfactory 22% 30% 26% Unsatisfactory 3% 17% 8% Highly unsatisfactory 0% 0% 0% Satisfactory and highly satisfactory 22% 13% 18% Moderately satisfactory or better 76% 52% 66% Source: IEG reviews of Country Assistance Strategy Completion Reports. 17 Chapter 3 Evaluation Highlights · The Bank's overall approach to M&E has many strengths. · Progress has been made in updating policies and frameworks, but there is considerable room to improve how M&E is put into practice. · At the project level, the overall qual- ity of M&E is low. · At the country level, results frame- works are increasingly produced but are often poorly formulated. · The independence of external eval- uations of global programs is im- proving, but their M&E systems are often weak. · Impact evaluations are a useful ad- dition, but topics need to be chosen strategically. Child studying in public school in the Amazon region of Brazil; photo by Julio Pantoja, courtesy of the World Bank Photo Library. Underpinning Impact-- M&E and Results Management Monitoring and Evaluation Systems at the Project Level T he Bank's overall approach to M&E has many strengths (see appen- dix B for an overview). Indeed, its policies for monitoring and evalu- ating projects have been revised, to place greater emphasis on project outcomes. Specifically, the Bank: · Required (a) investment projects to include re- wereidentifiedtomonitorprogresstowardproject sults frameworks, outlining (final) project and development objectives. The assessment of M&E intermediate outcomes as well as the process implementation is the extent to which appropriate for carrying out M&E;1 and (b) development data were actually collected, and utilization policy operations to specify expected out- reviews the extent to which appropriate data were comes and measurable indicators for M&E in evaluated and used to inform decision making and 2004 (World Bank 2004a). resource allocation. IEG, as part of its ICR reviews, · Replaced the Project Supervision Report with has started rating overall M&E quality using a four- the Implementation Status and Results report point scale: high, substantial, modest, and negligi- (ISR) in 2005. The ISR now gives more promi- ble.2 This quantitative assessment does not reflect nence to project outcome and intermediate a methodology that has been agreed to by Bank outcome indicators by including them in the management; ICRs assess, but are not required to main report, whereas in the past they were in rate, M&E quality. an optional annex (infrequently updated). · Harmonized the Bank and IEG's evaluation A review of the quality of project M&E There is a positive criteria for Implementation Completion and Re- showed a positive association association between good sults reports (ICRs) and IEG's ICR reviews in between good project M&E and project M&E and better 2006. The procedures for programmatic de- better project outcomes. It found that project outcomes, but the velopment policy loan ICRs were also simpli- projects with highly satisfactory overall quality of project fied to improve the effectiveness of the ICRs. project outcome ratings had, on M&E is low. average, higher M&E quality rating As part of the revisions, ICRs are now required to (3.0 or equivalent to a rating of "substantial") include an assessment of project M&E quality compared with projects with unsatisfactory along three dimensions--design, implementation, outcome ratings that had lower M&E quality (1.7 and utilization. The review of M&E design or equivalent to a rating of "negligible") as shown examines the extent to which adequate indicators in (figure 3.1). 21 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Figure 3.1: Projects with Higher Outcome Ratings Have Better M&E Ratings High (4) system 3.0 M&E Substantial (3) of 2.6 rating 2.0 Modest (2) IEG 1.7 1.8 verageA Negligible (1) Unsatisfactory Moderately Moderately Satisfactory Highly satisfactory unsatisfactory satisfactory IEG project outcome rating Source: IEG. But the overall quality of project M&E has been systems, where the monitoring indicators lacked low. Of the 242 ICR reviews for which M&E baselines and targets and not enough attention quality ratings are available,3 IEG rated that was given to implementation. quality as modest or negligible in two-thirds of cases, as shown in figure 3.2. Poor design was a Projects with high M&E quality ratings included key factor in low M&E quality ratings. Some M&E systems that were well designed, projects had weak results frameworks where the implemented, and used. The Second Rural project outcomes were poorly defined and/or Roads Project in Peru included performance the link among the project outcomes, that is, the monitoring indicators for outputs, intermediate project development objectives, intermediate outcomes, and outcomes that were conceptually outcomes, and the project outputs was not clear. clear, realistic, and measurable and were linked Another factor was poorly designed M&E closely to the project's four main objectives. Impact evaluations were designed into the project, and the evaluation of the previous Figure 3.2: M&E Is Rated Modest or project provided the baseline for this project. Lower inTwo-Thirds of ICR Reviews The M&E framework also constituted an active Substantial learning process for the project's executing 32% agency. The project was rated satisfactory for (78 projects) outcome. The Partnership for Polio Eradication Project in Pakistan was rated highly satisfactory for project outcome and high for M&E quality. The project was able to use existing systems, Modest combined with substantial external technical and 50% (121 projects) financial support, to develop a system and create High demand for data, which are essential for disease 2% (6 projects) eradication. Negligible The impact of the changes in Bank M&E 15% (37 projects) procedures may not yet be fully reflected in this analysis because most projects approved by the Source: IEG. Board since fiscal 2005 are still under implemen- 22 UNDERPINNING IMPACT--M&E AND RESULTS MANAGEMENT tation. Of the 242 projects with M&E quality ing got less priority during implemen- Poor M&E impairs the ratings, 23 have been approved since fiscal 2005, tation. Difficulties in M&E have also ability to manage of which 21 are development policy operations. been highlighted in several IEG sector ongoing projects and Close to half (48 percent) of these projects were and thematic evaluations, as noted in subsequently to evaluate rated substantial or high for M&E quality. box 3.1. them robustly. However, it is not clear whether this is due to better M&E quality ratings for development Many of these IEG findings are echoed in reports policy operations or because of improvements in from the Bank's Quality Assurance Group. The M&E policies. A clearer picture will emerge in QAG's most recent Quality of Supervision subsequent years as more projects from fiscal (QSA7) assessment noted satisfactory focus on 2005 and afterward exit the portfolio and have management of inputs and outputs, but less so ICRs and ICR reviews prepared. on outcomes (World Bank 2007a). The QAG's latest quality at entry assessment (World Bank It may not necessarily be policies and procedures, 2008b) found improvements in results frame- but rather a lack of incentives and priority that is work design (where more than 95 percent of the constraining the design and use of M&E systems. project were rated marginally satisfactory or As far back as 2000, a Bank working group on better), but also observed that project develop- improving M&E found that M&E tended to be ment objective clarity and realism, consistency of perceived as the least important of the project design with outcomes, and impact and dimensions of operational quality (World Bank outcome measurement could be improved. The 2000). Focus group participants for IEG consulta- Bank is proposing to strengthen monitoring tions stated that they sometimes set ambitious under IDA15 by tracking the quality of the development objectives and then faced difficulty project development objectives at design and linking them to projects, that there was excessive the adequacy of baselines during implementa- emphasis on quantitative data, and that monitor- tion (World Bank 2007b). Box 3.1: M&E Findings and Recommendations in Recent IEG Evaluations IEG's evaluation of the World Bank's project-based and World crops received support and how, what has been the comparative Bank Institute training (IEG 2008d) found that Bank projects and cost effectiveness, and to what can one attribute gains. It recom- the World Bank Institute both report on outputs (the number of mendedthattheBankimprovedatasystemstobettertrackactivities people trained, the number of training days, participant satisfaction it supports and strengthen M&E to report on project activities in with training), but seldom on outcomes (changes in workplace various agro-ecological zones for different crops and farmer cat- behavior, institutional capacity development). It recommended that egories, including women. the Bank: (a) develop guidance and quality criteria for the design IEG's recent evaluation of the Bank's Group's work in sup- and implementation of training, to enable quality assurance and portingenvironmentalsustainability(IEG2008a)recommendedthat monitoring and evaluation of all its training support; and (b) im- the Bank Group, as a whole, improve its ability to monitor and prove the quality and impact of training by making available tech- evaluate the impact of its environment-related interventions. This nical expertise on the design, implementation, and monitoring and included using current International Finance Corporation and Mul- evaluation of training to its staff and borrowers. tilateral Investment Guarantee Agency evaluation systems as a IEG'sevaluationofWorldBankassistancetoagricultureinSub- starting point for betting monitoring for the whole Bank Group. It Saharan Africa (IEG 2007d) found that data systems and support could further mobilize efforts for: (a) better baseline environmen- for M&E have been insufficient to adequately inform the Bank's ef- tal assessment studies; (b) more holistic evaluation methods; (c) forts to develop agriculture in Africa. M&E at the project level is further development of environmental performance indicators; often of limited value for answering fundamental outcome, im- and (d) improved monitoring and reporting on project implemen- pact, and efficiency questions, such as who benefited, which tation and results. 23 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Country-Level M&E: Early Evidence from corporated the CAS outcomes into their country Results-Based CASs portfolio performance review and is jointly The Bank adopted the results-based Country managing the Bank program with the borrower to Assistance Strategy (CAS) approach in 2005,4 focus on CAS outcomes (box 3.2). following a review that found that the evaluation framework remained the weakest area of the CAS, Poorly designed results frameworks limited their with the key issue being the "missing middle" usefulness for country program monitoring and between the CAS objectives and Bank operations management. It is difficult to interpret the (World Bank 2003a). A results-based CAS would information on a project's status if the expected be "a country assistance strategy that contains a outcome is vaguely defined ("banking system strong orientation toward achieving financial capacity strengthened") and/or lacks The Bank strengthened realistic outcomes and results-oriented baselines and targets to measure progress. the results focus of monitoring and evaluation system" Another issue is the volume of information to be its CASs. (World Bank 2005a). provided. Assembling the data has proven challenging: the Task Team Leaders for the The two main innovations of the results-based Progress and Completion Reports mentioned that CAS approach were the CASCR as a self-evalua- they had to devote a significant amount of time tion tool and the results framework as a monitor- and effort to collecting the data and updating the ing, management, and evaluation tool for the results framework. In some cases, the amount of Bank country program. The results framework is information provided was overwhelming and yet expected to link higher-level country outcomes failed to answer the basic question: "Is the Bank with Bank CAS outcomes, intermediate making sufficient progress toward achieving its outcomes/milestones, and operations, and CAS objectives." The Ghana Progress and Comple- would include measurable indicators of progress tion Reports addressed this issue by reporting that can be tracked through CAS implementa- actuals only for the 35 CAS monitoring indicators, tion, encouraging active management and which (except for those under the Governance allowing both self- and independent evaluation Pillar) had baselines and targets. (World Bank 2005a). IEG has also raised concerns over the poor Experience with this new approach suggests that, design of CAS results frameworks. Many CAS so far, the CAS results framework typically has been used as a reference for evaluation, but not for monitoring or program management. This Box 3.2: Armenia Joint Country emerged from a review of results-based or results- Portfolio Performance Review oriented CASs and interviews with the Task Team Leaders of the reports to examine the extent the As preparation for the 2007 CAS Progress Report, the results framework was used for monitoring, Bank conducted a joint portfolio review with the management, and evaluation and to enhance the government of Armenia, which covered not only the Bank's development effectiveness. CAS results overall status of the portfolio, implementation is- frameworks served as an inventory of commit- sues, follow-up on previous commitments, and indi- ments in the CAS Progress Reports and were used vidual problem projects, but also the progress toward to report on the status of CAS outcomes and/or CAS objectives and the contribution of Bank instru- intermediate outcomes without further ments. Information was prepared on the status of 29 CAS results frameworks analysis. The results framework was CAS outcomes and discussed with the government. are used as a basis for also used as a reference in evaluating Recommendations included specific steps, typically evaluation but less so for the country program in the CASCRs, broader and policy-oriented, which are needed to im- country program which included an annex with the prove project performance and to ensure achieve- monitoring and current status of the CAS outcomes. ment of CAS outcomes. management. One exception is Armenia, which in- 24 UNDERPINNING IMPACT--M&E AND RESULTS MANAGEMENT results frameworks had poorly articulated One of the lessons learned from the Poor design limited the results chains--including in several IDA Progress Report and CASCRs was the usefulness of the results countries where the distinction between CAS need to better align Bank activities with frameworks for Bank's outcomes and intermediate outcomes/mile- CAS outcomes. The Armenia CAS country programs. stones was unclear and the links between Bank Progress Report noted that it is better products and CAS outcomes were poor. Many to build the results framework around Poverty results frameworks had too many outcome Reduction Strategy Paper outcomes and indica- measures--more than 40 CAS outcomes indica- tors. Baselines are important and other indicators tors and over 60 intermediate indicators-- should be sought if baseline data are not available, and/or lacked indicators with baselines and and data sources should be verified to make sure targets, making them less effective as a manage- that information can be made available in a timely ment and evaluation tool. manner. The Ghana CASCR observed that one of the weaknesses was that some indicators were The lack of outcome data makes it more difficult only indirectly related to Bank interventions. The to assess the achievement of Bank CAS objectives Mozambique CASCR also commented on the in the CASCR reviews. While most of the CASs importance of the Poverty Reduction Strategy reviewed to date were prepared before the Paper, but noted that the CAS should results-based CASs were introduced, some did set more realistic targets, distinguish- Incentives favoring include expected outcomes and/or results ing them from Poverty Reduction project preparation and frameworks that were retrofitted in the CASCRs. Strategy Paper targets and choosing delivery over M&E IEG reviewers noted that the CASCRs discussed such targets only if they are related to continue to be a concern. what the Bank did (process, inputs and outputs), the Poverty Reduction Support Credit but did not necessarily provide convincing (PRSC) or where the Bank is the main source of evidence of the success of the Bank country support to achieve the Poverty Reduction Strategy program in achieving its objectives. Paper target. Whether staff have the right incentives to Another lesson was the importance of strength- promote effective results-based frameworks for ening M&E capacity at the sector or project level. CASs remains open to question. In interviews for The Mozambique CASCR noted that projects also this report, some Task Team Leaders talked of the need to strengthen their monitoring of results, difficulty of getting support from their sector and focus on helping sectors to develop monitor- colleagues when updating the information in the ing and evaluation systems, which would help CAS results frameworks. Some sector staff did project data collection and reporting. The not see value in the monitoring indicators Bosnia-Herzegovina CASCR made a similar point, because they were not involved in the formula- noting that the focus on results should go tion of the original CAS results framework. Given beyond Bank-financed operations and that the the demands on their time, monitoring of the Bank teams should help their counterparts to CAS results framework and analyzing Bank develop appropriate results frameworks for all country performance gets low priority, reflecting their operations. some skepticism about the value added by the exercise. This echoes findings from previous IEG Linking the CAS and project results It is important to link evaluations (IEG 2004a, 2006d) which concluded frameworks and strengthening the CAS and project results that "getting results" was not yet part of the country's capacity to conduct M&E frameworks, and reward system for individual staff and that the would not only reduce the cost of data strengthen country Bank's incentives rewarded the work that the collection, but also increase its rele- capacity to conduct M&E. staff do at the early stages of the project cycle-- vance and use. that is, preparing new operations--more highly than work at the later stages of supervision, such While the discussion focused on countries with as evaluation and learning lessons. CASs, M&E is also important in Low-Income 25 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 M&E is just as critical in Countries Under Stress (LICUS), Systems need to be in place to ensure that the Low-Income Countries which are covered by Interim Strategy information is collected. This may require Under Stress. Notes or Transitional Support Strate- trading off data availability with desirability. gies. An IEG evaluation (2006e) con- cluded that M&E is just as critical, maybe more Finally, integrating the CAS results framework into so, in LICUS (box 3.3). Bank operational procedures would establish the use of the performance data and could provide an Going forward, country teams should examine incentive to update and utilize the CAS results results chains and explore synergies among framework. There are some pilots under way. At various interventions in country programs the country level, the Moldova Country Manage- (projects, analytic and advisory activities, trust- ment Unit is piloting a scorecard to link CAS funded activities, and so on) to identify shared outcomes, operational performance, and Bank higher-level outcomes. This could lead to a budget allocation (box 3.4). At the regional level, reduction in the number of country outcomes to the East Asia and Pacific Region piloted and is be measured, making the frameworks more expanding an approach to link the achievement of strategically focused and usable. Teams could also CAS objectives with budget allocation (box 3.4). identify areas where the project results Linking Bank resource allocation with progress The Bank could take steps framework could be linked into the toward CAS outcomes would also provide an to strengthen the results CAS results framework. Establishing incentive to monitor and analyze Bank country frameworks and better synergies between interventions and performance indicators. The Bank could learn manage for results. identifying possible gaps in the Bank from and build on these pilots. program could go toward reducing the disconnect between country and project portfolio Managing Global Programs and outcomes. Partnerships: An Emerging Agenda The Bank has put in place new business processes The basics need to be ensured. CAS results for global programs and partnerships (GPPs), frameworks need to have monitoring indicators which recognize these as a separate product line with baselines and targets to measure progress. of the Bank, and which aim to integrate GPP business processes and information with the Bank's regular operational business systems (IEG Box 3.3: Results Measurement and Monitoring in 2004b). Throughout their life cycle, GPPs now LICUS follow procedures similar to a simplified lending process--using familiar concepts and systems and Monitoring and evaluation are at least as important in LICUS as they allowing new GPPs to be tracked from their start are in any other country. Monitoring and evaluation are crucial in through to evaluation and results assessment. LICUS for a number of reasons. First, the Bank, like other donors, is still learning what approaches work in LICUS contexts. Closely monitoring Enhanced processes have also been put in place experiences in order to draw lessons is critical, and learning and shar- to encourage greater selectivity although their ing needs to become a more prominent feature of LICUS work. Second, effect remains to be assessed fully. Proposals for given that progress is often slow in these countries, it is important to new GPPs and requests for new Development reassess continually whether the program is on course to achieve the Grant Facility (DGF) funding are now presented desired outcomes. Third, a constantly changing and volatile LICUS en- in a standard Partnership Review Note and consid- vironment where progress is often nonlinear means that program adap- ered, based on input from relevant parties in a tation is essential--closely tracking performance will help determine review meeting chaired by a director. The GPP when and what kind of adaptation is necessary. Effective learning-by- Group and the Legal Department are providing doing to improve the Bank's future effectiveness in LICUS can only hap- early advice to task teams with regard to pen with strong monitoring and evaluation. governance, which focuses on the quality of the Source: IEG 2006e. governance structure (how the partners will form the partnership and interact with each other), 26 UNDERPINNING IMPACT--M&E AND RESULTS MANAGEMENT Box 3.4: Use of CAS Results Frameworks in Country Program Management Moldova Results Scorecard CAS results frameworks to increase management ac- The Moldova Country Management Unit is developing countability for results. The new approach is grounded a Moldova Results Scorecard which will integrate, in one in CASs and, at the same time, focused across core Re- place, the different elements of country program man- gional priorities and actionable milestones. It is in- agement: progress toward country strategic outcomes, tended to be consistent across Regional and country lending and delivery costs of analytic and advisory ac- levels and enables Regional aggregation to allow dis- tivities, and quality (QAG and IEG assessments). The cussion with senior management. Country teams took scorecard will be used to link country program man- their respective CAS results frameworks, agreed on the agement and resource allocation and improve program critical CAS outcomes where they would focus their ef- and project results. forts, and narrowed the monitoring indicators to a man- ageable number in a one-page table. This information Strategically Managing Country Programs in East Asia was used in the annual and midyear program/strategy The East Asia and Pacific Region piloted the use of discussions. legitimacy, the voice of developing countries, and the Bank's existing information systems Bank management has the Bank's roles (particularly whether the Bank's include financial resources that are not put in place new business accountability and responsibility are aligned with channeled through the Bank, and the processes that aim to the Bank's formal authority and actual control). A Bank's Task Team Leaders are not integrate GPPs with the new results framework and performance indica- required to provide or update this Bank's regular business. tors were added to the Partnership Review Note information on an annual basis. and the DGF Progress Report on a voluntary basis in fiscal 2007, and made compulsory in fiscal 2008. The requirement for all programs receiving DGF funding of $300,000 or more, over the life of the Quality-at-entry reviews of DGF-supported pro- program, to undertake an independent evalua- grams by QAG also provide feedback that con- tion every three to five years has led to an increas- tributes to enhanced selectivity. QAG noted several ing number of program-level evaluations. improvements in fiscal 2007, as compared with However, the Bank has not yet extended this fiscal 2006, including stronger quality of program requirement to the many other programs not design and improved readiness for implementa- receiving DGF support, as previously tion, as well as a number of areas needing improve- recommended by IEG (2002), though Feedback from QAG ment, such as unrealistic objectives, weak results it is currently revising the relevant quality-at-entry reviews frameworks, and poor documentation. Operational Policy to achieve this. of DGF-supported Implementing such a requirement programs is contributing Taken together, these represent a substantial should be fairly straightforward for to enhanced selectivity. improvement over the previous partnership programs supported by Bank-adminis- approval and tracking system for GPPs, which was tered trust funds. For programs that are not not integrated into the Bank's regular operational supported either by the DGF or by Bank-adminis- business systems. However, to date the improve- tered trust funds, the Bank should still use its ments have mainly occurred with respect to DGF- influence, as a member of the governing body, to supported programs, since these programs have encourage periodic independent evaluations. to comply with the new business processes in order to receive grants. Except for DGF grants The independence of external evaluations is received, the financial information in the Bank's improving. Out of the first seven evaluations that databases (on the sources and uses of funds for IEG reviewed, five were executed independent of each program) is not complete or reliable. None of the management of the program, and without 27 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 The independence of apparent conflicts of interest. The about the effectiveness of development interven- external evaluations is governing bodies generally commis- tions. The challenge now is how best to use this improving. sioned the evaluation, approved the approach for increasing knowledge regarding terms of reference, managed the selec- development issues, advancing the broader tion process, and reviewed the draft report results agenda, and stimulating client engage- independent of program management. In four ment in, and capacity for, monitoring and evalua- out of these five cases, the selection of the tion. external evaluators was competitive. The results agenda can potentially benefit from However, the quality of three of the seven was impact evaluations, although they are clearly not compromised by a weak M&E system for the a panacea. Impact evaluations explicitly identify a program. Either the objectives and strategies of counterfactual through experimental or quasi- the program were not well defined (too diffuse, experimental designs to measure the true impact process-oriented, difficult to measure, or open to of development interventions. Accurate measure- different interpretations by different stakehold- ment of results can help for efficient allocation of ers), the M&E system was not well designed resources and effective project/policy selection (focusing only on inputs and outputs, and not and design. The potential benefits of an impact outcomes), or the data on the progress evaluation extend beyond any particular project The quality of some of activities and on the achievement of or program because the evaluation can provide evaluations was outcomes were not systematically information and lessons for the broader develop- compromised by a weak collected. And in three of the seven ment community. It is possible, therefore, that M&E system for the programs an inadequate budget did impact evaluations may be underfunded unless program. not permit any fieldwork to verify facts there is explicit support from management and on the ground or elicit the perceptions clients. On the other hand, it must be recognized of implementers and beneficiaries in developing that impact evaluations are highly specific to the countries. As a result, neither the external evalua- context and often quite costly to launch, and their tions nor IEG found much systematic evidence benefits are uncertain until they are completed. relating to the achievement of the seven programs' objectives at the outcome level, and it The number of impact evaluations at the Bank is difficult to say whether the $100 million spent has grown rapidly recently, with a rise from 60 on the programs annually ultimately had a under way in 2006, to 158 in mid-2008.5 Yet, the substantial effect on the ground. evaluations are concentrated in a few sectors and topics. Almost 70 percent of the ongoing evalua- External evaluations are Notwithstanding these shortcomings, tions are clustered in five areas, as shown in influencing the strategic the external evaluations have had figure 3.3, and three of these--education, health, directions and designs significant impacts on the programs and conditional cash transfers (many of which of GPPs. reviewed. The evaluation led to a also focus on education outcomes)--also revision in the strategic direction or comprise the top categories of previously design of the program in six out of seven cases. completed evaluations. There is also a concen- The evaluation led to a change in the governance tration by Region, with nearly two-thirds of of the program in four cases, and led directly to ongoing evaluations located in Sub-Saharan continued or increased funding in four cases. Africa and South Asia (see figure 3.4). Improving Our Understanding of Why evaluations are concentrated in certain areas Causality: The Use of Impact Evaluations and whether the observed distribution is efficient In response to the pressures for accountability, are questions for Bank management. Some the development community in recent years has substantive areas are more amenable to impact accorded increasing attention to impact evalua- evaluations, and Bank clients differ in their tion as another tool for generating knowledge receptivity to evaluations. But staff incentives may 28 UNDERPINNING IMPACT--M&E AND RESULTS MANAGEMENT Figure 3.3: About 70 Percent of Ongoing Evaluations Are Clustered in Five Areas Education 23 Health 17 Community-driven topics development 14 Conditional cash transfer 10 Evaluation Microfinance 7 All other areas 29 0 5 10 15 20 25 30 35 Share of ongoing impact evaluations (%) Source: Development Impact Evaluation database. also play an important role, and care must be The Bank has the capacity and re- Impact evaluations are taken not to overexamine some topics while sources to be a global leader in using not a panacea, but they others are underresearched. In an ideal setting, impact evaluations to drive effective are an important part of the decision to fund impact evaluations in a given development assistance. Required steps the results agenda. area would take into account one or more of the include identifying important gaps in following five criteria: the value of answering the our knowledge; closing these gaps with effective question in terms of benefits and costs of a specific and replicated evaluations; leveraging knowledge project, the value of answering the question for from impact evaluations through effective discus- other current or future projects, the cost of the sion, dissemination, and application across the evaluation, the innovative nature of the project, institution; and shifting expenditures from and the likely feasibility of designing a convincing unsuccessful to successful interventions. Clients impact evaluation. should be closely engaged and ideally, in many Figure 3.4: NearlyTwo-Thirds of Ongoing Evaluations Are Located inTwo Regions 50 40 39 impact (%) 30 24 20 ongoing 20 of 13 evaluations 10 Share 2 2 0 Sub-Saharan South Asia Latin America and East Asia Europe and Middle East and Africa the Caribbean and Pacific Central Asia North Africa Region Source: Development Impact Evaluation database. 29 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Impact evaluation topics cases, leading the process of and institutional development or donor coordina- should be chosen implementing evaluations and learning tion; impacts are harder to measure and attribute strategically and the from their findings. A major challenge to the Bank. In response, the report "Additions to results should influence facingtheBankistoassessandcompare IDA Resources: Fifteenth Replenishment" (World future funding decisions. the relative effectiveness of alternative Bank 2008c) served to illustrate accomplishments interventions rather than largely assess- in individual projects, sectors, and countries, thus ing the effectiveness of different ways of complementing the information provided in the implementing a given intervention. Existing initia- IDA Results Measurement System (RMS). tives, such as the Development Impact Evaluation Initiative, have made some progress toward these The IDA RMS assesses Bank-wide performance, goals. but only for IDA countries. The RMS has two tiers. Tier 1 focuses on monitoring country outcomes IEG itself is aiming to increase its own use of and includes 14 country outcome indicators that impact evaluations and capacity to review impact are consistent with the Millennium Development evaluationswithintheBank.Throughmeta-evalua- Goals, are priorities in poverty reduction strate- tions and mixed methods approaches, impact gies, and reflect IDA's activities. These are high- evaluations provide important opportunities to level country outcomes that are influenced by the enhance IEG's evaluation role. Bank, but for which the Bank is not solely respon- sible. Tier 2 monitors IDA's effectiveness. It Monitoring Institutional Effectiveness consists of indicators: (a) at the country level, Thedevelopmentcommunityisfocusingonresults where the cumulative introduction of results- on the ground, with the Millennium Development based CASs in IDA countries is monitored; (b) at Goals serving as a common framework. In 2002, the project level, where four project quality the heads of the multilateral development banks indicators are monitored; and (c) outputs from issued a joint statement on their commitment to completed projects in fours sectors (health, scale up their work on measuring, education, water supply, and rural transport). monitoring, and managing for results, The development following the Monterrey Conference on IDA prepared a progress report on the RMS in community is focused on Financing for Development. 2007 and made suggestions for improved results on the ground and monitoring under IDA15. It reported, among is interested in the impact The IDA15 replenishment negotiations other things, that IDA had built 7,500 km of road, of Bank operations. distilled this focus on results in a very trained 81,400 health professionals and 282,500 real way. During the negotiations, the teachers, and made 87,000 new water connec- Bank faced competition for donor funding from tions based on IDA project ICRs in fiscal 2006­07. different organizations and programs, many of The report concluded that the introduction of them focused on specific issues. Donor govern- the RMS "has generated a sharper focus on ments had to answer to their citizens on the results at the country level and a stronger effective use of their resources and therefore internal results culture." It also acknowledged requested that IDA demonstrate that it was achiev- that internal process weaknesses remain in the ing results on the ground and what those results quality of Tier 2 outcome indicators and the were. IDA's competition often focuses on single availability of baselines for all outcome indicators issues--such as children, health, and the environ- at entry in IDA projects. IDA recognized that ment--and are better able to present because "many ICRs still lack figures on outputs, aggregated output data (such as or report them in a variety of forms (different The IDA Results numbersofbeneficiariesassisted,goods units, no standard categories), aggregation Monitoring System and provided, and so on) and to describe remains very challenging." For IDA15, the report AfricaRMS provide a their results. The Bank, on the other proposed that IDA track the quality of the project broader view of results, hand, carries out not only projects but development objectives at design, the adequacy but not for all countries. also additional activities, such as policy of baselines during implementation, and the 30 UNDERPINNING IMPACT--M&E AND RESULTS MANAGEMENT reporting quality of outputs and outcomes in the Region and individual countries. It not only ICRs, based on the original results framework as helps teams focus on achieving results but also part of Tier 2. The report also suggested includ- provides internal and external clients access to ing ratings on CAS implementation from IEG's live information about stories from projects and CASCR reviews to measure the quality of Bank beneficiaries, impact evaluation results, and data country programs. for priority sectors. The Africa Region developed the Africa Results The system is still being refined and faces several Monitoring System (AfricaRMS) as a comprehen- challenges. The Region is working to improve sive online system in the Bank for monitoring on- the availability of country data and harmonized the-ground results of Bank activities. It consists of indicator measurement across Bank operations. a set of harmonized project and country indicators It is also making efforts to better integrate to facilitate counting and aggregating results at the financial data, and output and outcome data country, sector, Region, and Bank-wide levels. from operations at the sector and country level. AfricaRMS partnered with the education, malaria, Measurement systems across countries and HIV, water, private sector development, gov- among development partners will also need to ernance, agriculture, energy, and roads teams to be harmonized. build monitoring frameworks that are fully aligned with existing global initiatives. It also includes IEG's mandate is to evaluate the outcome of write-ups to tell the story behind the numbers, Bank-supported projects and programs, as describing how good results are changing people's measured against their objectives, and its work-- lives, and descriptions of impact evaluations including ARDE 2008--obviously represents a carried out in Africa. The AfricaRMS is aligned with part of the Bank Group's efforts to monitor the Africa Action Program. institutional effectiveness. An assessment of IEG's effectiveness, including the perspectives of The goal of the AfricaRMS is to use select its clients and the extent to which its recent information to show the results of Bank recommendations have been implemented, can operations and development progress for both be found in appendix C. 31 Chapter 4 Man sifting seeds along the Red River in northern Vietnam; photo by Quy-Toan Do, courtesy of the World Bank Photo Library. Lessons and Opportunities Practical Lessons for the Near Term S ome practical lessons emerge from this analysis. The increasing dis- connect between Bank and IEG project ratings needs to be reversed by renewing attention to project reporting and self-assessments consistent with standards set by IEG. The drop in 2007 project ratings also must be watched carefully. To improve the focus on results, steps are Despite increased international interest, the Bank needed: (a) at the project level and in global and is not yet well positioned to articulate the results of regional programs, to enhance the quality of the its interventions. The IDA RMS and the AfricaRMS M&E systems, especially by working to put in aim to report on the Bank's achievements in a place good baseline information, to elucidate systematic manner, but for a specific set of clearly the link between project outputs and countries. The Bank is considering a corporate- targeted outcomes, and to extend evaluation level results report, which is expected to synthe- requirements to global programs beyond those size the Bank's contribution to results, in place of supported by DGF; (b) at the country level, to the Sector Strategy Implementation Update. simplify results frameworks and thus make them more useful in guiding and evaluating programs; The experience of developing M&E systems at and (c) at the institutional level for the Bank and the project and country level, and for the IDA in partner countries, to manage and learn from a RMS and AfricaRMS, provides lessons and growing number of impact evaluations, includ- challenges for developing a Bank-wide results ing by better integrating them into country report, including: programs and exploiting cross-country synergies in conducting and sharing studies. · Performance measures should be limited to a manageable number that capture the critical el- Directions for the Overall Bank Agenda ements reflecting development effectiveness The development community's focus on results and can be measured and monitored rigorously. on the ground has also generated interest in the · Amonitoringsystemshouldbedesignednotonly Bank's overall development impact. The IDA15 to report on results but also to manage for re- replenishment exercise reinforced the importance sults, that is, to collect performance information of articulating the results (both outputs and to assess whether the Bank is on track to achieve outcomes) of Bank operations to donors and its corporate strategic objectives, make changes other external stakeholders. where needed, and allocate resources. Opera- 35 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 tional use of the performance measures would formation and minimize the data assembly provide an incentive for managers and staff to or- costs, for example, by standardizing project ganize, collect, and analyze the data. and country performance indicators as much · The Bank would need to ensure that its infor- as possible (a move already under way in the mation systems are aligned, to supply the in- Africa Region and in several networks). 36 PART II Shared Global Challenges Lessons from the Bank's Experience Chapter 5 Evaluation Highlights · Development issues that involve in- ternational collective action, such as climate protection, present some of the greatest challenges of our time. · Fostering global public goods is now one of the Bank's six strategic prior- ities. · The differing nature of the various GPGs influences the way in which the Bank and its development partners address them. Satellite image of hurricane; photo ©Adastra/ Getty Images, reproduced by permission. The Challenge of Global Public Goods I n an era of rapid globalization, the importance of coordinated global solutions to cross-border problems has risen in tandem. In recent years there has been enormous growth in international attention to climate change, avian flu, international trade agreements, and other supranational is- sues collectively known as global public goods. Many important GPGs--whose key characteris- The Bank, for its part, is taking steps to address tics are highlighted in box 5.1--need collective some of these issues. In previous decades, while action across countries to be delivered most of the Bank's financial and advisory support effectively. This applies, for example, to keeping was about providing public goods, it was the air clean or concerted campaigns against traditionally focused at the country level, on communicable diseases. Individual countries national public goods such as public health, may not have the incentive or wherewithal to regulatory frameworks, and the provision of take action, and thus GPGs can be chronically infrastructure. More recently, the Bank has undersupplied even when there is widespread increasingly been called upon to help in the recognition of the urgency of worldwide action. supply of public goods that transcend borders. Box 5.1: Key Characteristics of Global Public Goods Economists describe pure public goods as "nonrival" and often be coordinated by using government powers (including tax- "nonexcludable." Nonrival means the supply of the good, such ation, spending, and regulation). as clean air, to one person (or country) does not lead to there Importantly, public goods also have a spatial dimension. Their being less of it for another. Nonexcludable means that once the geographic reach runs across a continuum from local community good is provided for one person, it is available for all to bene- boundaries, to national borders, to regions of several countries, to fit from it. Typically, at the margin, the net benefits accruing to the global sphere. The usual problems in supplying public goods private individuals from such goods are less than the net ben- are exacerbated for truly global public goods. That is because there efits for society as a whole, and hence the public good is un- is a divergence between the costs and benefits captured at the na- dersupplied in private markets. Public goods require collective tional and global levels, and it is particularly difficult to secure action to be properly provided and, at the national level, this can collective action across countries. 41 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 To this end, the World Bank Group president has institution with considerable reach, is potentially promoted the fostering of GPGs as one of the well positioned to help the international com- Bank's six strategic pillars (World Bank 2007c). munity work at all levels in fostering GPGs. The And the Development Committee endorsed a Bank claims (World Bank 2007d) its comparative new framework for the role of the Bank in this advantage in fostering GPGs is that it can: arena in October 2007 (World Bank 2007d). The World Bank Global Public Goods Working Group · Work with countries to help them integrate has been appointed to take forward the GPGs into their national policies and programs president's strategic pillar. Further steps are as well as country assistance strategies, ana- needed for the Bank to achieve its objectives lytical and advisory activities, and lending. outlined in the framework. · Display constructive advocacy by using its re- search and analytical capacity to communicate It is important to recognize that the variety of the perspectives and interests of developing GPGs have very different features, as discussed countries in international arenas. in more detail in appendix D. From a demand · Be an active partner in responding to global side, when viewed from an individual country challenges and a source of innovative financing perspective, some have benefits that are closely mechanisms. aligned to national interest (for example, the control of HIV/AIDS). In others, such as protect- Putting such comparative advantage into action ing the earth's climate by reducing emissions of has posed challenges for the Bank and its greenhouse gases, there is a great divergence partners. The Bank has focused on five identified between national and global benefits. Part of the GPG clusters: preserving the environmental Bank's role--particularly in advocacy work and commons, controlling communicable diseases, in providing grant finance--can be viewed as a enhancing the participation of developing way of "internalizing the externality" and countries in the global trading system, strength- bringing national and global demand for GPGs ening the international financial architecture, closer together. and creating and sharing knowledge relevant for development (World Bank 2007d). Global public goods have From a supply side, GPGs also have different features, which different characteristics. Some require This report examines the ways in which the has implications for only one or a few countries to be Bank's country-based model helps or hinders its the way the Bank deals involved in their supply, whereas support for GPGs. Given the breadth of the with them. others need all countries to be active topic--spanning numerous programs and (for example, ensuring the participa- virtually all sectors--and the lack of comprehen- tion of a country that would, otherwise, be the sive evaluative evidence, this report is necessar- weakest link in a strategy to stop the spread of a ily a partial stocktaking of the situation. It pays communicable disease). Some GPGs require special attention to evidence on experiences many donors to be involved--opportunities for with two of the most prominent GPG themes-- the Bank to use its convening power--while the the environmental commons and communicable supply of others may rest with other major public diseases--and refers to other examples (includ- and private institutions. ing trade) where the context and available information warrants. The closing chapters ThedifferingnatureofGPGsthereforehasimplica- review the Bank's experience as an advocate for tions for the way in which the Bank (and other action on GPGs and draw overall lessons for the bodies) deals with them. The Bank, as a global Bank as it develops its approach in this area. 42 Chapter 6 Evaluation Highlights · Systems for integrating GPGs into country strategies are under- developed. · Attention to GPGs gets diluted as it moves from the Bank's corporate strategy to lower levels. · Bank administrative spending on GPGs is among the smallest for its six priorities, and the growing role of trust funds presents challenges. · Concessional finance is important to foster many GPGs. · A mismatch between country needs and global ambitions for GPGs lim- its the use of the country model, especially without an attractive financial instrument. Wind turbine farm; photo ©Frank Whitney/ Getty Images, reproduced by permission. Using the Bank's Country- Based Model to Foster Global Public Goods: Does It Work? How It Works in Theory--the Bank's Strategic Setting for Fostering Global Public Goods R elying on the country-based model as the platform for the Bank's work on GPGs is a double-edged sword. Why? Because the Bank's systems largely mirror international structure, giving primacy--for obvious rea- sons--to national decision making on policies and programs supported by the Bank. GPGs that permit significant benefits to be captured at the country level-- and so global and national interests coincide--are best suited for the Bank's country-based model. Country directors and their teams are able to directly appeal to their counterparts. Nor do they work in the traditional way with their counter- typically have the internal budget or instruments parts to develop approaches supported by the to be able to do so effectively.1 In short, it is Bank, be it through knowledge, convening difficult to bridge the gap between global needs power, or finance. and country preferences. When a GPG has benefits, however, which are Experience with Bank Country Assistance not easily or meaningfully appropriated at the Strategies national level (for example, cleaner air provided Country Assistance Strategies2 for partner as a result of more energy-efficient production nations define proposed country programs and technologies), or where the costs of providing form the bedrock of the Bank's overall engage- the GPG fall disproportionately on an individual ment with developing countries. Moreover, country (for example, income foregone by not because a great deal of the Bank's work, particu- using forest resources), then the country model larly its financing and knowledge services, is comes under strain. Staff interviews confirm that explicitly incorporated into individual CASs, such country directors and their teams do not have strategies are of pivotal importance as a starting the incentive to advance an agenda that does not point for the Bank's work in connecting global 45 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Systems for integrating and country issues. What does experi- 70 percent of CASs reviewed, the documents GPGs into CASs are ence tell us about how GPGs are mention work on either HIV/AIDS, tuberculosis, underdeveloped and playing out in country strategies? malaria, or avian flu--which might be taken as worth upgrading. the upper bound of attention on these GPG The templates for compiling a CAS do issues. About one-third of the CASs reviewed not make explicit reference to global programs mention Bank support for national disease or GPGs as issues that must be considered. surveillance systems. Likewise, templates for CAS Completion Reports and IEG's own Country Assistance Evaluations This pattern has been confirmed in interviews do not give explicit guidance on GPG issues. This with Bank country directors and their staff.3 They may well be a gap worthy of further review. The report that they are open to considering and Bank's internal CAS review process does allow including important GPG issues in their dialogue for draft material to be circulated for comment to with counterparts and, ultimately, in core various parts of the Bank beyond the specific country strategies. When the Bank is able to draw country team, including relevant groups a practical connection between a GPG and the involved with GPGs, such as the Sustainable poverty focus of the Bank, it has a base upon Development Network and the Bank's Global which to build. The key then is to exploit the Programs and Partnerships (GPP) unit. This comparative advantage of combining the Bank's process gives some opportunity for that perspec- global and local knowledge, because it is the mix tive on GPGs to be added. It is not common, of the two that allows the greatest added value to however, for experts from the centrally based be delivered to the client. But country directors anchor units (in the Bank networks) to be active report two significant drawbacks in delivering members of the CAS preparation team. these potential benefits. Which GPGs feature strongly in Bank country First, many respondents argue that, in a nutshell, strategies? A review of major recent CASs found "the networks are not working." From their that the environmental commons is the GPG perspective, the Bank's centralized anchor units most frequently noted explicitly in country have not been providing the valuable cross- strategies. Climate change, air pollution, and country research, expertise, and support needed carbon emissions are reported (in the vast to help foster GPGs at the individual country majority of CASs) as being addressed level. In part, this may reflect gaps in the Bank's through the Bank's own lending or own global knowledge but respondents also Environmental commons analytic and advisory activities, the suggested it was a problem of ineffective dissem- is the GPG noted most Global Environment Facility (GEF) or ination (internally, and to external audiences) of frequently in country the carbon funds. This observation global expertise and research. This last theme is strategies. resonates with IEG's earlier evaluation a recurring one--several evaluations from IEG of MICs (IEG 2007a), which found that and others have drawn attention to continuing mention and integration of global programs in flaws in the way the Bank transmits its analytical CASs was more likely for those close to the Bank and research work (IEG 2007a). such as the GEF and the carbon funds. Second, country directors remain concerned Attention to communicable diseases is the about what, at times, appears to be a bewildering second most frequently emphasized GPG in array of global programs with which the Bank is CASs. But in examining CASs it is often difficult associated. In some instances, the information to determine whether work on communicable made available on global programs is so volumi- diseases is primarily focused on national nous and broad that it is difficult for country teams interests (and national goods) or whether the to digest and use. In other cases, some global interests of the wider global community are also programs (and their internal sponsors) circum- driving or influencing the Bank's stance. In about vent the normal country dialogue approach, and 46 USING THE BANK'S COUNTRY-BASED MODEL TO FOSTER GLOBAL PUBLIC GOODS: DOES IT WORK? foist their agenda onto the national dialogue. pressures (often including electoral There is no evidence that Many country directors appeal for a rationalized cycles). Yet, global strategies to deal the treatment of GPGs approach that tailors information from global with GPGs have time horizons in CASs has expanded programs much more carefully to country circum- sometimes measured in decades, and across-the-board stances, which would facilitate a stronger engage- implementation measures typically over time. ment over the long term. for five years or more. This "mismatch of time horizons" poses an overarching problem With the Bank's increased attention at the for bringing public policy to bear on medium- corporate level to GPGs, has there been a similar and long-term global challenges. It is also a increase in coverage of GPGs in CASs? This source of tension reflected in the country-level report reviewed a relevant sample of countries in client dialogue and priorities across different which the Bank has significant engagement and groups (for example, country teams and anchor for which recent and historic CASs are available. specialists) within the Bank. How to make a In sum, there does not seem to be enough more effective link between available short-term evidence to say that the treatment of any of the instruments and medium-term challenges main GPGs is significantly different in the most deserves further consideration (IEG 2008a). recent CASs from their earlier equivalents. In fact, the picture is rather mixed. For example, the ThelatestBrazilCPS4 providesaverygoodexample latest Vietnam CAS and Brazil Country Partner- of clear and prominent integration of GPG themes ship Strategy (CPS) seem to address GPGs in a in country-level strategies (see box 6.1). more extended fashion than earlier strategies. In Pakistan, the treatment is similar between the The Experience with Bank Network (Sector) two CAS periods, as is also the case for the China and Regional Strategies CAS (which has had reasonably good coverage Given the sporadic attention to GPGs in country over an extended period). strategies, are strategies above the country level providing more attention to GPGs? That is the One important issue highlighted by staff in case with regard to corporate-level strategy, country and network positions is the differing where the Bank has paid significant attention to time horizons that apply in different strategies. GPGs, most notably in the GPG Framework Country strategies typically cover two to four (World Bank 2007d) and Long-Term Strategic years, which has to be aligned with national Exercise (World Bank 2007e). But these strate- Box 6.1: Brazil: A Best Practice in Integrating GPGThemes in Country Strategies The latest Brazil CPS is unique among the CASs reviewed in this Brazil is special for its central role in dealing with many global public ARDE for its treatment of GPG issues. The partnership strategy goods (including AIDS, climate change, biodiversity, and clean en- explicitly references GPGs and particularly highlights the in- ergy and for the fact that it is a demanding borrower that `pushes the herent tension in promoting GPGs at the country level: Banktothenextlevel,'byinsistingthattheBankenterareaswherethe Bank has been reluctant to participate . . . that the Bank engage with The Bank will engage with Brazil with climate change, trade, and global public goods from the perspective of the developing world. other global public goods issues. In so doing, the Bank will help to ensure that Brazil's perspectives and interests are repre- The CPS is also notable because it is a joint strategy between sented, in ensuring that the line between global goods and na- IBRD and the International Finance Corporation, which includes tional ownership is not blurred, in ensuring that development both private and public sector responses to GPGs, particularly needs are given equal billing. In short, the Bank's perspectives on environmental commons issues. The CPS is so recent that the will be to `level the playing field' on global public goods. effectiveness of the new approach remains to be evaluated. 47 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 The Bank's attention to gies could have been more specific on Sector Board has identified and catalyzed new fostering GPGs gets how to translate corporate priorities partnerships for emerging climate change issues, diluted as it moves into country-level action. Indeed, the but in other areas has not fully followed through from corporate­level Long-Term Strategic Exercise acknowl- on its stated commitments to realign global strategy down. edges the limitations of the country- partnerships with strategic objectives and to based model as a tool for fostering improve the governance and management of such GPGs, particularly regarding country incentives partnerships (IEG 2008b). Similarly, the DGF to diffuse country knowledge (World Bank Council stated during the fiscal 2007 screening 2007e). process that the environment sector "seemed to lack a unifying strategy and that proposals were What about the strategies "in the middle"--those not linked with country-based and other global of the networks5 and Regions? A review of such programs" (IEG 2008b). documents found that explicit attention to GPGs becomes diluted as it passes from network anchor The Bank's Regional strategies (which vary signif- strategy to Regional strategy. And typically, neither icantly in format and vintage across Bank Regions, network nor Regional strategies fully elucidate with Africa being particularly comprehensive) how these strategies will play out at the country typically cover GPGs somewhat superficially. level.6 Attention to GPGs is more prominent in Climate change and HIV/AIDS are mentioned in both network and Regional strategies dealing with all Regions, but scant attention is paid to looking the environment, as compared with those dealing at the promotion of GPGs in a more comprehen- with the health sector.7 The likely reason is sive or strategic manner. Nonetheless, some good because of the type of intervention needed for practice is emerging in Regional and sector- containing communicable diseases, which specific strategies, as highlighted in box 6.2. requires a strong national focus that might not be explicitly connected to global action. Box 6.2: Emerging Good Practice from the Regions A good example is the Bank's environment strategy, Making Sustainable Commitments (World Bank 2001), and the 2007 Sector The current Latin America and Caribbean Regional Implementation Strategy Update (World Bank strategy includes concrete references to proposed ac- 2007f), which provides a strong focus on institu- tion on important GPGs. Global issues is one of the four tions, governance, and outcomes. It is one of the pillars of Bank action for the Region and includes top- few network strategies that elucidate in detail how ics such as climate change, trade negotiations, each Bank Region should undertake actions to HIV/AIDS, and avian flu. The Regional vice president foster global/regional environmental commons. elevated the importance of climate change and the Translating strategy into practice, however, proved Bank's involvement in the external media by noting in to be difficult. an op-ed piece that a fair global system should make lower carbon-intense emissions possible by leveling The Sustainable Development Network reported the playing field between developing and developed in 2003 that mainstreaming environmental consid- countries. erations in sectoral projects, programs, The 2005 Health Strategy developed by the Africa and policies had been slower than Region includes a substantive discussion of how to in- Though mainstreaming expected. IEG (2008a) found that tegrate global priorities with country priorities (World environmental although mainstreaming has advanced Bank 2005c). Acknowledging the growing number of considerations have since 2001, it has remained incomplete global initiatives aimed at controlling communicable advanced since 2001, it because of limited incentives and too diseases, the document outlines the importance of the has remained incomplete few independent Bank resources to in- Bank in helping client countries manage and benefit due to incentives and tegrate environmental components from the complex array of global initiatives. resource limitations. into other projects. The Environment 48 USING THE BANK'S COUNTRY-BASED MODEL TO FOSTER GLOBAL PUBLIC GOODS: DOES IT WORK? Several factors could contribute to the dilution popular tool for the Bank to deploy. Each of of GPG focus in Bank strategic documents from these is discussed below. the corporate level to the networks and Regions and finally to the country. It may be that account- Allocating the Right Level of Corporate ability for GPG issues is lessened, or control over Resources resources diminishes, as one moves down the The Bank reports that its expenditures and management chain. Both could compromise a lending for the purposes of fostering GPGs have country team's ability and desire to advocate for risen rapidly in recent years (using a widely drawn and to deliver GPGs. Some interview respon- definition of GPGs, owing to the lack of specific dents also postulate that task managers and administrative coding).8 These estimates should country teams feel they have their hands full with be treated with some caution because the Bank's traditional mandate and are naturally they may be subject to significant Bank expenditures are reluctant to take on another. Without accompa- variations depending on the defini- rising rapidly, with the nying resources or accountability, a corporate tions and classifications used in their biggest increase for work mandate from above becomes more difficult to construction. Indeed, going forward, on environmental trace down to the country implementation level. more precise definition and tracking of commons. spending would be important as Country Programs in Practice-- management tools. Expenditures, including trust from Strategy to Action funds, have nearly doubled in five years, from In moving from strategy to action at the country $56.2 million in fiscal 2002 to $108.0 million in level, at least three factors come into play. First, fiscal 2007, reaching approximately 4 percent of the broad attention paid by the Bank to GPGs is total Bank expenditures. The bulk of this $50 partly reflected in the overall allocation of million rise is concentrated on increased spending corporate resources--the Bank's own budget in the GPG areas of environmental commons (an and trust funds--to this priority. Second, the increase of $22 million) and trade (an increase of particular financial instruments at the Bank's $16 million), as shown in figure 6.1.9 disposal--including grants, concessional IDA, IBRD loans--influence country activities. Third, These funds come from three sources, including global programs have become an increasingly the Bank's core budget, Bank-executed trust Figure 6.1: Bank Expenditures on Main GPGThemes 120 International financial trust 100 architecture and million) ($ Trade 80 budget GPGs on 60 40 Environmental commons administrative expenditures 20 Bank Communicable diseases fund 0 2002 2003 2004 2005 2006 2007 FIscal year Source: World Bank, Global Public Goods Working Group. 49 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Bank lending for GPGs funds, and other funding sources such over $1 billion of finance in fiscal 2006 alone. While has more than doubled as GEF. The core budget expenditures this amount was a peak, these funds accounted for over the past five years-- have nearly doubled over the period, an average of $270 million during fiscal 2002­07. to about 10 percent from $32.3 million in fiscal 2002 to of total lending in $59.7 million in fiscal 2007. In Are these numbers--particularly the Bank's fiscal 2007. addition, trust fund expenditures have administrative budget--adequate for the task? It risen from $13.9 million to $26.2 is impossible to say because there is no million, and the other sources have increased benchmark against which to calibrate. It is from $10.1 million to $22.1 million. possible to highlight the difficulties of budgets and incentives staff face, as reported in IBRD and IDA lending has grown even faster, interviews with key respondents.10 The budget from $968 million (and 5 percent of all lending) systems of the Bank have largely been designed to $2.5 billion (10 percent) over the same period. to fit its model of business, which has been in The bulk of this increase was in the area of trade, place for several decades--with lending as the where lending volumes increased from $76 centerpiece--and have not yet been smoothly million in fiscal 2002 to $988 million in fiscal 2007 adapted to reflect the requirements of new as shown in figure 6.2. A large increase also business. Country directors and their sector occurred in the area of environmental commons, director colleagues can struggle to meet from $467 million to $879 million. Lending for corporate pressure to conduct GPG work at the communicable diseases has fluctuated around country level when there is modest or little $400 million over the period, and that for counterpart demand and no obvious instrument international financial architecture has been very for follow-up. Some innovative responses have small. been developed, such as the Latin America and The Bank's Caribbean Region's "beam" funding, which administrative Other sources of funds have been allocates a modest share of Bank budget adminis- expenditure on GPGs is prominentinfinancingworkinenviron- trative funding on a competitive basis to Sustain- one of the smaller mental commons. The carbon funds, able Development Network sector teams to allocations for its six GEF, Montreal Protocol, and other stimulate their work on GPGs in specific strategic priorities. special funds, together, accounted for countries. Figure 6.2: IBRD and IDA Lending for Main GPGThemes 3,000 2,500 International financial million) ($ architecture 2,000 Trade lending 1,500 IDA 1,000 and Environmental commons IBRD 500 Communicable diseases 0 2002 2003 2004 2005 2006 2007 FIscal year Source: World Bank, Global Public Goods Working Group. 50 USING THE BANK'S COUNTRY-BASED MODEL TO FOSTER GLOBAL PUBLIC GOODS: DOES IT WORK? At the network level, staff also report that budget through the concessional--and hence Trust funds are a growing and incentive pressures are inhibiting, but it is more attractive--financing of IDA. force in the delivery of difficult to assess this from an external viewpoint. Indeed, in recent years, the Bank has GPGs--this poses The Bank's need or desire to be responsive to its committed substantial IDA funding to challenges with regard to shareholders and stakeholders leads it to be an help countries in programs with clear fragmentation, donor active partner in a growing number of global initia- GPG dimensions--such as HIV/AIDS, requirements, and tives. Network staff often do not have adequate avian influenza, and environmental preferences. budgets to fund their work--such as attending commons. For example, in Vietnam, planning and governance meetings of global the Bank has been able to use its multisectoral programs, or producing tailored research in expertise, combined with concessional IDA response to partner requests. And against such a finance, to help the authorities cope with the squeeze, the staff are placed in difficult positions threat of avian flu, in part, because there was a of trying to fulfill other services, including provid- strong national interest in averting economic ing operational and other specific support to fallout in the domestic food industry. country teams. Often, implementation capacity on the ground is Overall, while the increase in resources of the stretched, however, and in those circumstances, Bank to GPGs has been substantial, renewed Bank staff report that national demands, attention to those GPGs--both globally and understandably, may take precedence over some within the Bank--implies there will be pressure GPG considerations. Furthermore, for national for the Bank to devote substantially more counterparts who are dealing with wretched emphasis and, therefore, resources to GPGs. poverty or postconflict reconstruction on a day- Since the Bank's administrative budget is to-day basis, the goals of GPGs can seem rather projected to remain flat, and across-the-board distant and lofty. Again, staff report that there is increases in lending volumes will be modest at great reluctance among national partners and most, the Bank would then have to rely increas- country teams to allow IDA allocations targeted ingly on other sources. Grant resources and for poverty reduction to be diverted to fostering market-based mechanisms may be used to finance GPGs whose benefits may not be felt by the country-based projects, and trust funds to finance poorest. Bank expenditures. If not, the Bank would have to make a major reallocation of its own Bank budget A related and pressing point that has been away from its traditional work and toward GPGs. observed in several high-profile cases in Africa is the growing presence of vertical global programs This corporate decision to use external funding and funding mechanisms. Most vertical funds are streams, particularly trust funds, to address GPGs in the health sector and they focus much needed could lead to fragmentation of efforts, rigidity attention and resources on specific problems, over allocations, greater administrative costs, such as HIV/AIDS, malaria, and other communi- subjection to particular donor requirements and cable diseases. While these are vital issues that preferences, and a disconnect from the basic pose national, regional, and sometimes global resource allocation mechanism that governs the challenges, the influence of vertical funds risks country-based model. The Bank has sought to diverting domestic resources from other health mitigate such risks through several measures, sector and national priorities. Such funds may including standardizing its trust fund policies and place more pressure on national administrations engaging with donors to harmonize approaches. and can exacerbate aid fragmentation. Using Different Financing Instruments For example, in Ethiopia, more than half of the country's health budget recently came from Bank Concessional Finance--IDA HIV/AIDS global programs (IEG 2005). In cases For LICs, the Bank can support country action with large resources from global programs, the 51 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Bank must work with client countries to carefully this is the GEF. Formed in 1991, the GEF had a weigh its role and comparative advantage. In very clear mandate, with the full backing of the doing this, there may be instances in which a track international community. Its basic operational record of successful Bank support for, say, structure and financial management originated in HIV/AIDS projects in a specific country warrants the Bank itself, and the professional experts who continuing active engagement. In other circum- help promote GEF projects are Bank staff. Its grant stances it may be better to reallocate resources finance has proved equally appealing to MICs and away from "inundated" areas to where the Bank LICs alike (IEG 2007a). Indeed, in China--which can be more effective, for example, in integrated has benefited from about $510 million of GEF health care systems. The Bank should also financing in 45 projects since 1991--the presence continue its aim to support government-led aid of the GEF has been a key ingredient in helping coordination mechanisms, and at the global level, the Bank and national authorities form a strong through the Paris Declaration and follow-up and practical partnership to tackle issues that measures. Progress needs to be accelerated. would not otherwise have been addressed. Client representatives report that Bank staff have kept IDA has innovated with A recent innovation in IDA has been them adequately informed on a range of global special support for the creation of a separate funding programs and initiatives. Furthermore, Bank Regional projects. envelope to support Regional projects. involvement has been valued where it helps Established on a pilot basis under demonstrate new approaches that can be IDA13, the approach was continued under IDA14, subsequently scaled up--an important feature of and the allocation for it increased by over 70 China's 11th five-year plan (IEG 2007a). percent under the IDA15 framework (World Bank 2008c). These resources can be used to "top-up" In cases where the Bank has not had an obviously IDA resources provided to countries through the attractive financial instrument--and/or where regular country performance-based allocation there has been a lack of demand from country system, as applied to pertinent Regional projects. partners--it is less easy to see progress. For The rationale is to encourage and facilitate example, Bank effectiveness in promoting global countries' participation in Regional projects, help environmental sustainability, including tackling defray the extra costs associated with such cross- climate change, has been mixed. Considerable border cooperation, and meet what is perceived attention has been given to biodiversity conser- to be a significant demand and opportunity for vation, but, with the significant exception of more (and more ambitious) Regional projects. China, less has been given to greenhouse gas The Bank's Africa Region has helped push Bank mitigation, and, until recently, almost none to support for Regional cooperation through the help countries adapt to the likely future impacts creation of a director position and a dedicated unit of climate change. In Senegal and Uganda, for to promote Regional projects and programs. example, the link between natural resource Although it is still early, there are signs that this management and poverty has been largely IDA initiative is bearing fruit. It should be overlooked in Bank lending (IEG 2008a). Much monitored for lessons as to whether this approach of this could be the result of a wider concern should be replicated for some GPGs, though great about how developing countries are compen- care would be needed to avoid fragmenting IDA's sated for their investments in GPGs. The Bank's overall framework. approach is beginning to change, however, and much greater attention is envisaged by both the Other Concessional Finance--Grants Bank and International Finance Corporation to When the Bank has an When the Bank has had a clear and climate-related challenges, including with the attractive instrument to viable instrument to support its newly emerging Climate Investment Funds. support its country country partners in taking action on partners in taking action, some environmental concerns, there Several developing countries have expressed there are signs of success. are signs of success. A good example of apprehension about having to design national 52 USING THE BANK'S COUNTRY-BASED MODEL TO FOSTER GLOBAL PUBLIC GOODS: DOES IT WORK? projects that include global interests without Only some of the benefits of control- Although the inclusion of receiving additional funding for the benefits ling avian influenza can be captured GPGs in national accruing to the international community (ITF by individual countries, and so it tends strategies is desirable, 2006). Although the inclusion of GPG concerns in to remain a low national priority. So doing so remains national strategies is socially desirable, the issue far, the Bank has supported 50 avian contentious. still remains contentious about how developing influenza and pandemic preparedness countries will be compensated for having to incur projects under its Global Program for Avian higher costs (while benefits are distributed Influenza.12 But only seven of these projects globally) when implementing development include IBRD finance, for a total of $94 million, projects. For example, the creation of renewable and among these, only two are for sizable sums. energy electrification projects, without compen- Moreover, to date, only $12 million of IBRD loans sation for higher costs (as compared with have been disbursed. traditional electrification projects), will mean a reduction in the number of beneficiaries. In the two large IBRD projects--in Romania and Turkey--the positive externality of controlling Nonconcessional and Market-Based Finance avian influenza was largely captured within A key point reported by some operational staff national borders by factors having nothing to do and client representatives is that there is often a with the Bank. In both cases, poultry exporters mismatch between country needs (and were prevented from getting their resources) and global ambitions on many GPGs.11 products to the important European The Bank's work on avian In MICs, for example, the Bank tends not to be a Union market because of fears that the influenza illustrates the significant player in financial terms and, indeed, disease could spread to people, provid- limits of the Bank's its main instrument--IBRD lending--is noncon- ing strong national incentive to take nonconcessional finance. cessional (IEG 2007a). Hence the Bank's ability to quick action. This is a good example of influence (or persuade) a country to take how external pressure (economic or legal) can concrete action on some GPGs is inherently make the most of the Bank's country-based model limited, but the effective provision of those goods for fostering certain GPGs. requires deep participation by these MICs, as discussed in box 6.3. In Indonesia, an evaluation of the Bank's country assistance program from 1999 to 2006 showed The limits of nonconcessional finance are that it covered forestry issues with large-scale illustrated in the Bank's work on avian influenza. analytical work but little lending. Over that Box 6.3: Bank GPG and MIC Strategies: Fates Entwined The provision of many GPGs depends critically on the actions In addition to these potential synergies, however, there are also of middle-income countries. Indeed, MICs are home to many of tensions between these two agendas. MIC clients are increasingly the world's most important environmental assets. They are the calling on the Bank to become more client-focused and respon- source of more than 40 percent of the world's carbon emis- sive to their needs, given the vast expansion of choice they have sions; the strength of their financial systems directly affects the enjoyed in financial and technical support for development (IEG fortunes of other countries globally, and the prevention of com- 2007a). At the same time, almost by definition, many GPGs--par- municable diseases by MICs could be critical to avoiding wide- ticularly those with the weakest link and aggregate effort char- spread contagion. Actions taken by MICs are therefore essential acteristics--require action that countries would not otherwise to the GPG agenda. Moreover, the Bank has indicated that GPGs choose to take. The Bank must carefully navigate these inherent will be a critical focus of its engagement with MICs, providing tensions and trade-offs. Doing this well would lead to success and one of the primary justifications for a continuing relationship effectively deliver on both the MIC and GPG agendas, but failing (World Bank 2006a). on one would likely weaken the other. 53 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 period, the traction achieved by the Bank was utilize the Clean Development Mechanism of the very limited, and deforestation continued at a Kyoto Protocol through carbon funds, focusing rapid clip. particularly on the capture of the greenhouse gas, trifluromethane. In contrast, when national and global interests (and benefits) are closely aligned, nonconces- Deploying Global Programs at the Country sional finance can prove a workable instrument. Level Over the last two decades, the Bank has been able Global and regional programs and partnerships14 to secure very substantial country-based action on are another vehicle the Bank can use to bring the promotion of trade--recognized as an national and global interests closer together. The important GPG. To complement its global-level number of these global programs supported by work on trade regimes, the Bank has supported the Bank has grown rapidly in recent years, projects and programs in some 117 countries, reaching about 160 in fiscal 2008. Global and with a total of $38 billion of finance (8 percent of regional programs are very diverse, in both their total Bank commitments) since the late 1980s objectives and structure. A program can act (IEG 2006a).13 About 70 percent of those projects primarily as a cross-country network, with little have delivered satisfactory develop- or no financial resources available; it can finance In China, the Bank has ment results in the countries technical assistance; it can finance specific invest- successfully used market- concerned, although the poverty- ments; or it can act in a combination of these based finance alongside reducing aspects of trade reform have ways. It can support GPGs through investments its other, more not often been fully delivered (IEG at the country, regional, or global level; or it can established instruments. 2006a). In recent years, the Bank has have nothing to do with GPGs at all. also creatively brought together the nexus of global, regional, and country interven- Indeed, when one looks at the financing for tions on trade--for example through successful, these programs, three characteristics emerge. regionally structured, trade and transport facilita- First, only 56 global programs (35 percent of the tion projects (IEG 2007c). total) focus primarily on providing GPGs, as shown in figure 6.3a.15 Second, a majority The use of market-based finance, alongside other share--57 percent--of Bank-executed resources more-established instruments, is demonstrated that the Bank devotes to global programs are well in the Bank's work on China, which is one of allocated to the GPG-focused programs. Third, the world's largest emitters of carbon dioxide and when one looks at all funds that the Bank where 70 percent of energy is coal-based. The manages, including recipient-executed trust Chinese authorities have actively engaged with funds, the share devoted to GPGs grows to a the Bank, and priority has been given in the Bank's large majority, as shown in figure 6.3b--92 country strategies to financing clean and percent. These figures raise interesting issues renewable energy as well as projects for the clean regarding the prioritization of GPGs in the storage of carbon dioxide emissions, afforestation, Bank's support for global programs. and recycling. Indeed, to date the Bank's program has helped the Chinese to adopt a Renewable Many global programs--particularly those that Energy Law, while other analytic and advisory primarily deliver GPGs through national invest- activities and technical assistance work has ments and technical assistance--require action focused on enhancing biodiversity. Furthermore, at the country level. But despite the Bank's direct the Bank has been able to leverage funds from role as a partner in these global programs, global programs like the GEF and the Multilateral systematic linkages to country programs have Fund for the Implementation of the Montreal proved challenging. Many of the global programs Protocol for renewable energy and have garnered only modest participation by MICs Global programs have ozone-depleting substance projects. (IEG 2007a). And Bank performance in global grown rapidly in number. Finally, the Bank has helped China programs overall has been better at the global 54 USING THE BANK'S COUNTRY-BASED MODEL TO FOSTER GLOBAL PUBLIC GOODS: DOES IT WORK? Figure 6.3: Most Global Programs Do Not Focus on GPGs, but Most of the Bank's GPP Resources Are Devoted toThoseThat Do a. Bank-executed resouces b. Bank-managed resouces 250 2,500 (20 programs) (20 programs) 200 Financing investments (98 programs) 2,000 Supporting technical assistance or network 150 1,500 activities millions millions US$ 100 US$1,000 50 (36 programs) 500 (98 programs) (4 programs) (36 programs) (4 programs) - - Global public goods National public goods and Global public goods National public goods and other issues other issues Source: World Bank database. Note: Bank-executed resources cover Bank budget and Bank-executed trust funds. Bank-managed resources also include recipient-executed trust funds. than at the country level, in part because, in the program staff and Bank staff to focus It has proven challenging absence of a requirement to do so, task on their own activities, for which they to make systematic managers for global programs rarely are held accountable, while collaborat- linkages between global demonstrate how the program will help specific ing primarily with established counter- programs and country countries (IEG 2004b). parts. For example, Bank country team programs. staff often liaise with ministry officials One reason for limited participation by develop- overseeing country programs, while global ing country partners is that the Bank's decision program staff often liaise with a different set of to support a global program can be the result of officials, donors, specific interest groups, and pressures beyond those coming from its client specialists dealing with GPGs. countries, including strong donor interests, advocacy by international nongovernmental Indeed, Bank Task Team Leaders of global organizations (NGOs), and other geopolitical programs have reported that there are sharp considerations. Sometimes these programs incentives against close linkages between country attract limited interest from developing and global programs. On occasion, the strong countries. presence of a global program in a country sometimes leads a country team to take a division- How can global programs be better integrated of-labor approach, scaling back its program in the into country operations? Simply basing a global same sector. Importantly, country teams have few program within the Bank--there are 57 such resources to work on global programs unless there programs--does not guarantee effective linkages. is a demonstrated interest by the country client. For example, these linkages were weak in the This is even more acute when a global program is Population Reproductive Health Capacity housed outside the Bank because there is little or Building Program, in spite of the potential no core Bank budget earmarked for synergies with Bank investment operations program oversight. Moreover, while Merely basing a global occurring in the same countries. Even when country teams are interested in budget program in the Bank does country teams and global program teams and concessional finance that may not guarantee effective communicate, the natural tendency is for global come with participation, their interest linkages. 55 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 There are, in fact, strong depends on the flexibility of the funding $100 million of spending in 2007--ultimately had incentives against to support country priorities that may or any effect (IEG 2008b). making such linkages. may not overlap with global priorities.16 The multicountry aspect of GPGs also applies to IEG has found that stronger legitimacy of a topics and responses that are best handled at the program appears to foster stronger linkages regional level, that is, by groups of neighboring with country operations (IEG 2004b, 2008b). countries. The Bank faces a similar challenge in This legitimacy is enhanced by substantial demonstrating how it can link regional and developing-country representation in the country concerns and opportunities. One major governing body (discussed in the chapter 7). instrument for doing this is through regional Stronger linkages emerge in the programs when programs, which have increased in importance developing countries have greater voice, in part in recent years, though they still account for a because these programs are more likely to modest share of Bank lending. In reviewing reflect developing-country interests and to be CASs, it is clear that mentioning regional relevant to developing-country needs and programs is also much more the exception than circumstances. the rule. The approach toward M&E for global and regional As the cross-border dimensions of health, programs is substantially less developed than that environment, and trade facilitation are expand- which has been established over time for the ing worldwide, the contribution of regional Bank's traditional IBRD and IDA instruments. programs to address regional public goods and Global programs are relatively new and rarely self- GPGs is likely to grow in significance. Consensus sustaining financially, and hence it is among participating countries regarding the The approach to M&E for very important to establish a robust distribution of program benefits and costs, as global programs is M&E framework at the outset of a well as strong country voice in governance substantially less program. But IEG's recent review of arrangements, are key characteristics which IEG developed than seven global program evaluations identified in successful regional programs (IEG approaches used in found the quality of all seven was 2007c). An example of an effective regional Bank lending. compromised by weak M&E systems program is the Regional Hydropower Develop- for the program. Therefore, neither the ment Project, which was designed to manage the external evaluations nor IEG found much system- Senegal River Basin serving Mali, Mauritania, and atic evidence relating to the achievement of the Senegal. This project built a hydroelectric plant seven programs' objectives at the outcome level. that successfully responded to the needs of the It is impossible to say whether the global program three countries by providing a reliable, low-cost interventions--together accounting for about power supply and increased electricity access. 56 Chapter 7 Evaluation Highlights · Bank advocacy goes beyond the country level and involves producing collective global responses and pro- moting the development interests of the poor. · Promoting global trade reform rep- resents the Bank's advocacy at its best--in its analytical and practical depth and its proactive dissemination. · The Bank's advocacy work on avian influenza built on robust economic analysis, convening power, fiduci- ary reputation, and multisectoral expertise. · The Bank has had some advocacy successes related to environmen- tal commons, though its influence on advancing work on climate change is more debatable. · The voice of developing countries is still underrepresented in global programs, particularly in their gov- ernance. Indonesian elementary schoolchildren wear surgical masks in response to 2003 outbreak of SARS in China; photo ©Reuters/ Corbis, reproduced by permission. The Bank's Advocacy on Global Public Goods: What Has Worked and What Has Not? The Dimensions of Advocacy T he Bank can advocate for action on GPGs in at least three ways. First, it can encourage specific actions supported by the global community at the country level. This lies at the heart of the Bank's country-based model and was discussed in chapter 6. Second, the Bank can help inform collective of new arrangements will affect the world's global responses to providing GPGs. For example, poor. independent and robust research on the costs of communicable disease threats (such as avian flu) This chapter explains aspects of advocacy using can help inform international partners in develop- three high-profile examples--trade, environ- ing an appropriate response. Practical analysis, mental commons, and avian influenza. It draws and sometimes the Bank's convening power, can on the importance of voice for developing also be used to help develop markets--such as countries, especially related to the governance of that for carbon finance--and so deliver new tools global programs. Finally, it presents a perspective to provide GPGs. on very recent innovations of the Bank in supporting advocacy through its financial Third, the Bank's focus on poverty and develop- capabilities. ment can be used to promote the interests of developing countries in international dialogue Advocacy at Its Best: The Bank's on mechanisms to tackle GPG issues. This is Experience with Trade particularly applicable where the community of The role of the Bank as a constructive advocate nations considers developing and implement- for developing countries on GPG issues has been ing (binding) international agreements to set a shown in its work on trade (IEG 2006a). Arguably, framework for collective action. In such cases, the Bank was somewhat slow to emphasize this the Bank has a potentially powerful ability to strand of its work, but it was given prominence ensure that development interests are properly from 2001 onward, significantly later than other considered--not least, how costs and benefits country-based components of the Bank's work 59 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 which had been growing since the late 1980s. But items produced in significant quantities in the the Bank emerged as a leader in global trade developing world. The research itself was work with high-profile reports ahead of major accompanied by a proactive and highly visible international trade reform meetings in Doha dissemination effort. One aspect was a willing- (2002) and Cancun (2004).1 ness to engage in public debate--including in mainstream media channels--and take what Furthermore, the Bank's advocacy role was some might regard as fairly strong positions on useful in positioning the Bank among client issues of some contentiousness.3 All of this took countries, especially those with which it might place in a very specific setting: the live negotia- otherwise be less engaged on trade issues. For tions for the Doha round of the World Trade instance, even though some Latin America Organization rules-based regime. This last point countries had been skeptical with regard to the is also critical, because it gave the Bank's work a Bank's policy advice because of the perceived direct opportunity to gain traction. failure of the Washington Consensus (Rodrik 2006), the Bank has had an active country-level The advocacy role was not an unqualified engagement on trade. This illustrates success, and at least two drawbacks have Key ingredients included an important connection--that the appeared. Recently, the analytical basis and a long period of working Bank may indeed have to be seen as a findings of some of the Bank's earlier work has directly with partner constructive advocate in the global been challenged, and indeed some recalibrations countries, first-rate arena if it is to open doors at the have been made. This, in itself, hardly undoes research capacity, and a country level, to persuade and work the impact of the Bank's advocacy but it willingness to engage in with national partners to deliver on- illustrates the difficulty of putting in place analyt- public debate. the-ground action. It may well be that ical foundations that are deep and can stand the on other GPG priorities--for example, test of time. The second drawback relates to how climate change--the Bank is not yet seen as such advocacy is then followed through with practical a high-profile advocate for developing countries, action to build developing-country capacity. The hence constraining its delivery capacity at the most high-profile initiative in trade capacity- country level. building work was the Bank's involvement in the Integrated Framework for Trade-Related Techni- What were the ingredients that contributed to the cal Assistance--a multidonor, multiagency Bank's advocacy role in trade? Certainly a long collaboration set up in 1997 to provide trade- gestation period of working on the ground and related technical assistance to 49 less-developed directly with partner countries (especially countries. This global program instrument through the late 1980s and into the 1990s) on exhibited some weaknesses, however, including trade issues, gave the Bank a good understanding a slow pace of implementation, insufficient focus of, and motivation to promote, the impact of on improved trade outcomes, and a shortage of trade on development. This was a case in which funds. The absence of a good results-based the global-country link worked in both directions. management framework also typifies a problem common to a good number of global programs. Beyond that, the assembly in-house of first-rate intellectual and analytical research capacity on The Complex Challenge of Environmental trade was essential (Deaton and others 2006).2 Commons and Climate Change This gave the platform for the Bank to produce The Bank has long paid attention to the global detailed, innovative, and well- dimension of the environmental commons--and The Bank's work also regarded reports that informed the more recently to climate change itself. Indeed, gained traction in the ongoing debate in trade, most notably there are very specific examples in which the context of "live" the benefits to be gained from a more Bank has played a practical advocacy role to negotiations for the Doha free-market approach to the markets develop new approaches to what is a very round of trade talks. for agricultural products and other complex challenge. When the GEF was being put 60 THE BANK'S ADVOCACY ON GLOBAL PUBLIC GOODS: WHAT HAS WORKED AND WHAT HAS NOT? together, there was considerable debate among experts in the field because they assess On environmental partners as to whether resources should be new the methodologies it has developed commons, the Bank has or drawn from existing allocations to the Bank for ascertaining additionality to be played a positive (and others). For its part, the Bank did play an relatively sound, especially as com- advocacy role in some advocacy role and is reported as holding the pared with other projects approved by very practical settings. position that "it would participate only if the United Nations Clean Develop- additional funding was made available." In the ment Mechanism. Some 22 methodologies for end, an agreement was reached in London in measuring additionality developed in the PCF 1990 with "the World Bank, and specifically the and Carbon Finance Unit of the Bank have been president of the bank, clearly designated as the approved by the Clean Development Mechanism. administrator and manager of the central But there is not unanimity on these issues, as function of the fund: financing projects and shown in box 7.1. programs to meet the incremental costs of article 5 [that is, developing country] parties" Advocacy has also been very noticeable in the (Benedick 1998). Bank's contribution to setting up new Climate Investment Funds. Emerging from preliminary The Prototype Carbon Fund (PCF) emerged from high-level discussions at the Gleneagles Summit advocacy work by the Bank and others, as in 2005, the creation of these funds has been explained in box 7.1. The main, universally assisted by the Bank and other partners over the recognized contribution of the PCF is that it gave last three years. Now it is reported that a multibil- credibility to the burgeoning field of carbon lion dollar pledge has been made by the govern- finance. The PCF is also respected by several ments of Japan, the United Kingdom, and the Box 7.1: Advocacy for Carbon Finance:The Bank's Role in the Growth of a World Market Since the 1990s, the Bank has supported proposals for carbon detailed IEG examination of additionality in Bank-supported car- finance--an instrument to contain carbon dioxide and other harm- bon projects should give a more accurate assessment. ful emissions by allowing a developed country government or com- The PCF's advocacy position was initially hampered around the panytomeetsomeofitsownenvironmentalobligationsbyinvesting issue of governance. The host developing countries had no voice in projects in developing countries, to help reduce emissions there. in PCF governance. In response to this situation, the Bank, after TheBank'spresidentpromulgatedtheconceptattheUnitedNations the first year of PCF operation, created a Host Country Committee, General Assembly in 1997 (World Bank 1997), and the Bank's advo- which now covers all Bank-managed carbon funds. But the Host cacyborefruitwiththelaunchofthePrototypeCarbonFundin2000, Country Committee remains strictly advisory and some members the world's first carbon fund. Since then, the PCF's catalytic effect still complain that they have no contact with the funds' participants has been an example of advocacy through demonstration, and has committees.a Other members and Bank managers argue that some been one factor in the global carbon market's exponential growth, aspects of the operational structure cannot be fully inclusive be- from $38 million in 2002, to $64,000 million in 2007. cause of the need to discuss confidential pricing information. Carbonfinanceisnotwithoutcontroversy,andcriticshaveques- Some Host Country Committee members would like to see in- tioned the extent to which additionality occurs and if invested country capacity-building efforts on a greater scale and impact than projects are truly sustainable. For example, two studies by outside the $21 million allocated to the Carbon Finance Assist program, observers conclude that "additionality is unlikely or questionable housed in the World Bank Institute. for up to 40 percent of registered projects" and that "left to mar- The new Carbon Partnership Facility, scheduled to become ket forces, the Clean Development Mechanism does not signifi- operational later in 2008, will need to carefully address these and cantly contribute to sustainable development." A forthcoming other tensions. a. The two committees meet at the same time but in different rooms. The current Chair of the PCF Participants Committee has never met the Chair of the Host Country Committee. Under the new Bank-supported Carbon Partnership Facility, expected to become operational in late 2008, host-country members are to have equal repre- sentation and voting rights with company members. 61 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 United States to kick-start this significant initia- our time. The Bank's analytical and research tive. But as illustrated in box 7.2, there is substan- work has certainly been less prominent in this tial pressure on the Bank to mainstream sphere than in the trade arena. While major climate-friendly approaches into all its develop- publications such as the Stern report, and the ment work. advocacy of Nobel Prize winners, have been recognized worldwide, the Bank's materials have The example of the Montreal Protocol (and neither been as hard-hitting nor attracted the indeed of the carbon finance business, which same level of attention. This may be because the emerged from the incentives created by Kyoto Bank's need to balance different viewpoints Protocol commitments) illustrates an important among its shareholders has constrained its room point: the World Bank can play a strong role in to maneuver. It may reflect a less-than-strong skill the supply of GPGs, but primarily when set to produce the necessary leading research, as supported by an appropriate treaty instrument. suggested in IEG's environment evaluation The Montreal Protocol is one of the most (2008a). Or perhaps at this stage of the cycle, the successful treaties in history. The Bank has Bank is only now (appropriately) beginning to contributed to its success, but the real credit gear up on the advocacy front. belongs to the treaty itself and to the nature of this particular global challenge (Barrett, 2008). Whatever the mix of factors in the explanation, it Similarly, the Bank's ability to contribute to the is becoming ever more pressing for the Bank to much harder problem of climate change will approach the advocacy element of climate depend on the effectiveness of the treaty change (and its important dimension on arrangement for this challenge. development) more visibly and forcefully. The Bank president's public statements,4 and the The extent to which the Bank has opportunity opened by the new Strategic The extent to which the been a leading advocate to date on the Framework for Climate Change, provide a Bank has been a leading broad issue of climate change is more platform upon which to build. influential advocate on debatable, particularly now that global climate change is more climate change is considered by many One important area in which the Bank's debatable. as one of the greatest challenges of advocacy can be ramped up relates to the adapta- Box 7.2: Clean and Dirty Energy--Can the Bank Do Both? Some external commentators criticize the Bank Group, argu- overnight (House of Commons 2008). Moreover, some critics argue ing that while it is doing the right thing in expanding its climate- the Bank has little incentive to scale down a line of business that, friendly initiatives--through GEF projects, the creation of the particularly in IBRD countries, provides a steady flow of profitable Prototype Carbon Fund, and some analytical research--it is doing projects in the context of declining lending volumes (Redman 2008). the wrong thing by continuing to finance large energy and infra- Can the seeming contradiction be reconciled? There are steps structure projects that emit carbon dioxide on a massive scale the Bank Group can take. One would be to include a shadow price (Wheeler 2008). for carbon emissions in project appraisals, and to set a timetable Withdrawing from or reducing investments in traditional proj- for its introduction as a key factor in decision making by, say, 2010. ects would be a major challenge. As recently as fiscal 2006, these Another would be to start costing out and regularly reporting on energy investments accounted for 92 percent of the Bank's total en- the subsidy that would be required in any given year to help these ergyportfolio,ornearly$2billion(WorldBank2007g).Andtherecon- energy and infrastructure projects become climate-friendly. This tinues to be equally substantial pipeline demand. It is argued that would provide the international community with better ongoing these investments are meeting a real development need, and one knowledge of the price tab for collective action, and how far it is cannot expect such a transformational "sea change" to occur being met. 62 THE BANK'S ADVOCACY ON GLOBAL PUBLIC GOODS: WHAT HAS WORKED AND WHAT HAS NOT? tion to climate change. Adaptation is needed adapted to suit local circumstances, It is becoming ever more because of the failure (to date primarily by and how these technologies should pressing for the Bank to industrialized countries) to supply the GPG of be deployed and used. All of this advocate for development climate change mitigation. IEG's evaluations requires a robust technical and interests in climate have argued that the Bank could be strategic and scientific capability. This links to change more visibly and effective with regard to reducing the vulnerabil- another of the Bank's strategic priori- forcefully. ity of countries and the poor to natural disasters ties--engaging with MICs, including and climate change (IEG 2008a). They also Brazil, China, India, and Russia. This strategy is concluded that the Bank Group has not been essential if the Bank is to play a meaningful role able to provide financial resources to assist in helping the development transformation countries to address environmental concerns as needed to mitigate climate change. Technology high priorities. The emphasis in the IDA15 R&D is also needed for the world's poorest replenishment on the Bank's role in assisting countries, but for reasons of scale, a policy that eligible countries to adapt to climate change is a focuses exclusively on this latter group of step forward in this regard. countries would be unlikely to properly address the mitigation challenge. The industrialized countries have acknowledged a responsibility to assist poor countries in adapting Such an advocacy position is consistent with that to climate change. What they have not determined put forward by the International Task Force on is, first, a basis for calculating this assistance and, Global Public Goods (ITF 2006), an exercise led by second, an arrangement for determining burden- respected developing country leaders, sharing among the industrialized countries. The practitioners, and opinion makers. It Other opportunities for World Bank can help in calculating the level of recommended establishing an Interna- advocacy include financial support needed and then in implement- tional Consultative Group on Clean promoting research and ing an adaptation assistance program. Indeed, the Energy Research, "which includes both development tailored for connection between adaptation and development developed and developing countries, to the developing world. is so intimate that the Bank is arguably the only collaborate and exchange information global institution capable of playing this role. on research and development of more efficient and cleaner technologies." The recent report by the Another angle on which advocacy could be Global Leadership for Climate Action (GLCA 2007) enhanced relates to research and technology endorses this idea. These proposals draw inspira- capability to produce climate-friendly solutions tion from the Consultative Group on International in many areas--including transport and energy Agricultural Research (CGIAR)--a network to production--suited for developing countries. which the World Bank is tightly connected. Most attention has focused on the need for research and development (R&D) to be Climate change adaptation will also require undertaken by the industrialized countries, with innovation in other areas. A key area for future the technologies embodying this new innovation might be called climate-resilient knowledge being transferred to developing agriculture, a need which CGIAR has recognized. countries.5 However, technologies appropriate World Bank research has also begun to show the for industrialized countries may be less so for effects of climate change on agriculture.6 developing countries, given the different contexts. New technologies may be needed but Creating a Unified Response: currently this kind of R&D is seen to be Learning from Avian Flu somewhat neglected. The World Bank is seen by some partners to be playing an important role in the global Developing countries need to be able to response to threats emerging from The Bank has advocated determine which technologies are necessary, avian flu and a potential human for global action on whether the technologies available have to be pandemic influenza. In terms of global avian flu. 63 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 action, the Bank identifies its comparative that could operate effectively, and an agreement advantages as coordinating global responses and of this type was reached during the run-up to and mobilizing finance, and thus far it has been quite including the Geneva conference. an effective advocate along those dimensions. Close collaboration with other major international The Bank's leadership role in coordinating a organizations, with each working to its compara- global response to avian and human influenza-- tive advantage, was essential in the response to beyond what is typically expected of a financial avian flu. There have not been any major conflicts institution--has been seen as critical.7 Its early between the Bank and the relevant UN agencies analysis of the global costs to the spread of avian on avian flu. Some counterparts have expressed influenza and the threat of a global pandemic concern in IEG interviews, however, that the Bank (Brahmbhatt 2005) provided an umbrella under may have gone a bit beyond its comparative which the international technical agencies and the advantage (to arrange finance for the technical United Nations System Influenza Coordinator and strategies of the World Health Organization, Food donors could operate. This work identified the and Agricultural Organization, and World Organi- control of avian and human influenza as a GPG, zation for Animal Health) and taken a more wide- noted its urgency, and helped bring global ranging role that considered technical agencies as attention to the issue. inputs. Others have been critical that the interna- tional community's collaboration has a weak spot Given the urgency of this threat, the Bank helped in underestimating the severe global economic quickly to convene major global partners around costs--both direct and indirect--that are likely in a strategy for action. Along with the World Health the event of a pandemic (Osterholm 2007). Organization, Food and Agricultural Organization, Overall, this experience illustrates a wider point and World Organization for Animal Health, the about the importance of collaboration among Bank sponsored the first large-scale international international agencies, also pertinent to the meeting on the topic in Geneva in November Bank's advocacy position on HIV/AIDS, as 2005. The meeting was attended by more than 600 described in box 7.3. experts from over 100 countries, with substantial high-level government participation. The Bank also supported efforts to mobilize finance to address avian and human influenza, According to high-level participants, the Bank particularly with regard to the "nuts and bolts" made several contributions at these meetings. aspects of the challenge. The Bank's research First, it convinced international partners of the into the possible global economic impact of a importance of integrated country plans that take pandemic influenza that might be the result of into account animal health, human health, and uncontrolled avian influenza, noted above, pandemic preparedness. This was important found that the ultimate costs to the world because the Bank was the only sponsoring economy could be several percentage points of organization with a multisector perspective, gross domestic product (Brahmbhatt 2005). without prejudice toward intervention in one More recent research has put the price tag at $2 sector over another. Second, the Bank argued trillion, and economic (but not human) costs that a global program should benefit would be concentrated among the wealthy Important ingredients developing countries, given that the countries. This revealed to developed nations included the Bank's response would have to be delivered the potential costs of inaction. The Bank also robust economic analysis, on a country level and country-owned estimated country-by-country financing needs to convening power, initiatives have proven to be most control avian influenza and delay or reduce the fiduciary reputation, and effective.Third,ithelpedtokeepfocus probability of a pandemic (World Bank 2006b). multisectoral expertise on developing the implementation This created a baseline against which to measure and orientation. arrangements for a global program pledges, commitments, and disbursements. 64 THE BANK'S ADVOCACY ON GLOBAL PUBLIC GOODS: WHAT HAS WORKED AND WHAT HAS NOT? Box 7.3: Importance of Collaboration on Advocacy: HIV/AIDS The Bank's work on HIV/AIDS stepped up consider- with developing country partners to spearhead a global ably in the late 1990s after internal pressure from the campaign to make existing drugs for HIV/AIDS and other health sector staff in Bank Regions and the Research De- diseasesaffordabletothepopulationsofdevelopingcoun- partment, and externally from newly created interna- tries.Theylobbiedforpreferentialpricingfordrugsanddrug tional agencies such as UNAIDS. The Bank then donations to developing countries. They supported de- proactively raised awareness and demand for HIV/AIDS veloping countries in the potential to exercise their rights support among its staff and client countries. under international trade and intellectual property rights The collaboration with civil society also ultimately in- agreements. While the Bank and World Health Organiza- fluenced HIV/AIDS advocacy within the Bank on other tionwereslowtotakeapositionontheseissues,theyhave importantglobalhealthissues.InternationalNGOsworked come to support wider access to drugs (IEG 2004b). Working with the European Commission and tackle this global challenge. Perhaps this other partners, the Bank helped prompt donors difficulty could have been ameliorated either to pledge $1.8 billion (well above the $1.2 billion with better coordination among donor partners target) at an early 2006 conference in Beijing. or with a centralized funding mechanism Since that time, there has not been an outbreak dedicated to actions in developing countries. of a pandemic, media attention has receded, and the topic may be seen as less urgent. Although Effective Advocacy Benefits from Voice an additional $900 million was pledged at later and Representation conferences in Bamako and New Delhi, the To be an effective advocate, the programs the amount was below the financing needs identified Bank supports must be seen to be legitimate. But on those occasions. the voice of developing countries in GPG arenas--not least in the governance of many The Bank and United Nations System Influenza global programs--is still underrepresented, and Coordinator have since worked closely to the question remains as to whether the Bank monitor commitments and disbursements of could have pushed harder to rectify this funds pledged at the three conferences, and to imbalance. While the organizational and strategic encourage donors to follow through on their settings of such global programs vary, they share commitments. Thus far, resources actually the common feature that they need to promote committed have not kept up with existing financ- the cooperation and collective action of their ing gaps (UNSIC and World Bank 2007). And own members to be effective. This puts the while the international partners agreed that emphasis squarely on how to integrated country plans were the key platform promote the participation of develop- Giving proper voice and from which to tackle avian flu--and would ing countries, which are among the representation in global involve activities for which very significant intended beneficiaries of global programs improves their resources would be needed--only 40 percent of programs and without whose engage- responsiveness and long- committed funds have been allocated to these ment the effectiveness of many such term sustainability. plans. Furthermore, as shown in figure 7.1, of programs may not be assured (Woods, those $700 million of funds committed to forthcoming). country programs so far, less than half have been disbursed, a much lower share than for funds There is not, however, a single best-practice committed to international or regional banks.8 governance framework for global programs This has left a gap in the finances available in because they differ markedly in size and scope many poorer countries, particularly in Africa, to and employ a diverse array of governance 65 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Figure 7.1: LessThan Half of Funds Committed to Integrated Country Plans Have Been Disbursed 800 90 707 )n 700 80 millio 76% ($ 600 73% sed ur 70 dibro 500 66% (%) 433 atertn 400 60 332 committed, 317 324 seme ur 300 Disb 50 220 pledged,t 206 46% un 200 157 Amo 40 100 0 30 Regional body International Other organization Country or territory organization Channel for funding Left-hand scale: Right-hand scale: Commitments Disbursements Disbursement rate Source: UNSIC and World Bank 2007. Note: Approximately $650 million of pledged funds have yet to be committed to specific countries, projects, or programs. models associated with the history and culture of and governance structure. But that was time the each program (see box 7.4 for an example). IEG stakeholders considered a good investment, in has emphasized the need for a global program light of the inclusiveness of the governance "to establish legitimacy on a basis other than arrangements on which they were able to agree shareholder rights," pointing to the value of (IEG 2007c). It paid off because the structures adopting an inclusive and participatory were able to help resolve ongoing differences governance framework (IEG 2007b). It can take and sustain the buy-in of participants. And a far considerable time and effort by partners and the bigger global program--the GEF--took a Bank to develop good governance arrangements considerable degree of institutional experimen- at the start-up of a global or regional program-- tation to achieve a governance framework that is but this is time well spent. currently regarded by many as participatory and inclusive, as noted in box 7.4. The evidence from important programs bears this out. In the case of the regional Nonetheless, there is often a tension--actual or Governance programs for Central Asia biodiversity perceived--between organizational efficiency arrangements in several and for the West Africa HIV/AIDS (which may be fostered by streamlined programs--including the project, it took about two years to governance arrangements) and voice. While in GEF and CGIAR--have reach agreements among the partici- many programs, developing countries are improved over time. pants on the appropriate institutional represented in the governing bodies, "stake- 66 THE BANK'S ADVOCACY ON GLOBAL PUBLIC GOODS: WHAT HAS WORKED AND WHAT HAS NOT? Box 7.4: Governance as Institutional Experimentation: GEF The GEF began as a pilot program in 1991, with 16 OECD mem- cent majority of the total number of participants in the GEF, as well bers and 9 developing countries pledging resources and repre- as of a majority representing 60 percent of total contributions to senting the sole form of governance. Subsequently, the move from the facility. Another feature supporting a greater sense of owner- a pilot to a more established program resulted in a governance ship of developing countries over the GEF is that the council structure that melds those of the United Nations and the World cochairperson--in addition to the permanent cochairperson po- Bank. Governance is based on a constituency system that allows sition provided by the chief executive--rotates between developed for "a relatively small and effective Board," instead of the "one- and developing countries. country, one-seat, and one-vote system in the United Nations The GEF governance also provides an institutionalized setting General Assembly," while maintaining the potential for universal for engaging NGOs--a crucial factor in enhancing country-level representation (Woods and Lombardi 2006). coordination and country ownership. Participation by NGOs, both The more balanced representation of developing countries-- local and international, does not only take place at the project level through 18 of the 32 constituencies--is enhanced by the double- but also at the policy-making level. GEF provides observer status majority voting rule. When decisions are not supported by a for NGOs at council meetings and holds consultations with them consensus, they must garner the formal votes of at least a 60 per- in conjunction with each council meeting. holder and shareholder influence is not always its former steering committee into an executive balanced," and "donors and international body (IEG 2008b). agencies still largely govern the programs" (IEG 2004b). In some cases, the increase in represen- The emphasis in the current debate on fostering tation in the main governing body has created public goods raises the issue of the extent to problems of organizational efficiency. But which MICs are appropriately engaged in the governance arrangements need not be static, governance of relevant global programs. The and the changes in the CGIAR (highlighted in Bank itself has indicated that "a promising aspect box 7.5) illustrate how adjustments and compro- of its relationship with MICs lies in its role in mises can be made. Likewise, the Association for making connections between these partner the Development of Education in Africa has countries and the provision of global public recently decided to establish an executive goods" (World Bank 2006a). Yet insufficient voice committee and Cities Alliance has transformed in global program governance is still a concern Box 7.5: From Shareholder to Stakeholder Model: CGIAR CGIAR, founded in 1971 and jointly sponsored by the World Bank, ing number of centers and donors, led CGIAR to establish an exec- Food and Agriculture Organization, International Fund for Agri- utivecouncil,withtheaimofsimplifyingthenetwork'sdecisionmak- cultural Development, andUnited Nations Development Programme, ing. Following the recommendations of the Change Design and has grown from an informal small club of donors with a shared con- ManagementTeam,in2001,thecouncilwassetupasastakeholder cern for agricultural research into a partnership that now includes body appointed by CGIAR members to follow up on the group's de- 25 developing and 22 industrialized countries, 4 private foundations, cisions, to ensure alignment and congruence of recommendations, 13 regional and international organizations, which provide fi- and to act on decisions requiring a more urgent time frame than the nancing, technical, and strategic support to a network of 15 re- schedule of the group's Annual General Meeting would allow. search centers, mostly located in developing countries. Theestablishmentofthecouncil,withformallyelectedmembers The expansion of the membership, the widening of the research accountable to the particular groups they represent, was needed agenda, and the collective-action problems arising from the grow- to increase the legitimacy as well as the efficiency of CGIAR. 67 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 for MICs, inhibiting their enthusiasm for and countries, resulting in greater effectiveness for the engagement in such programs (IEG 2007a). Even overall program. This reinforces the Bank the larger MICs are represented in the decision- Development Committee's own underscoring of making bodies of only a small fraction of global "the importance of enhancing voice and participa- programs, while, in contrast, several high- tion [in the Bank] for all developing and transition income countries are included in the governance countries" through an "inclusive and consultative" of a much larger number. process among shareholders.10 Governance The governance arrangements for any New Dimensions of Advocacy: arrangements for global specific global program--and how its Innovation through Financial Capabilities programs can change advocacy position then emerges over The creativity and innovation of the Bank has over time. time--are the results of many been displayed in its work to leverage funds for influences. The Bank is but one of innovative approaches to development, especially those influences and hence responsibility for applied to GPG challenges. The International success or failure cannot rest solely--or perhaps Financing Facility for Vaccines and Immunization even primarily--at its door. Indeed, the Bank's (IFFIm) provides an example. IFFIm sought to position is complex, and in a program it can play find a way for the international community to up to 11 different roles including that of increase resources for life-saving vaccines and convener and trust-fund manager.9 But that health services in developing countries. Its multiplicity of roles creates tensions and, indeed, solution is to borrow from the international potential for conflicts of interest (such as that capital markets, based on donors' firm medium- between being a funder and a beneficiary), which term commitments, and then delivering that must be carefully managed. finance quickly and predictably through the Global Alliance for Vaccines and Immunization. Moving forward, the growing involvement of the Bank in the provision of GPGs is associated with The Bank can hardly be said to have played an its own governance framework. Good explicit, traditional advocacy role in proposing governance is essential to help the Bank shape and developing the concept of IFFIm. Rather, the its broader strategy for GPGs, define and be original idea was formulated in a proposal by the selective in its own role in global programs, United Kingdom's government, with the support manage potential or actual conflicts of interests of other interested parties, including the Bill and arising from the various global programs, and Melinda Gates Foundation. But the Bank has enhance its role as an advocate for developing- played an important role in using its financial country needs. Alternatively, there is a risk that capabilities and reputation in moving this "strategies will be hard to define, legitimacy will advocacy to action. Specifically, at the request of suffer, and implementation will lag" (ITF 2006). IFFIm, the Bank has a range of treasury manage- ment and related services, which has gotten IFFIm The experience of some global programs is quite off the ground. The Bank, in effect, acted as the encouraging in certain respects. Once all the start-up treasurer for this global initiative, and stakeholders acknowledged the dynamic nature deployed creative approaches with prudent of a program's governance, they fiduciary standards in the international capital The Bank's financial embarked on a reform process markets. It raised nearly US$1 billion through a robustness and through which considerable changes bond issue in November 2006, almost all the reputation can be put to have been introduced to the proceeds of which Global Alliance for Vaccines good use through governance framework. Far from and Immunization scheduled to spend on innovative financial impairing the overall effectiveness of immunization and health programs in poor instruments that help the program itself, this has prompted countries through 2007. The initiative is still young move advocacy to action. wider engagement of developing and it will be important to assess the extent to 68 THE BANK'S ADVOCACY ON GLOBAL PUBLIC GOODS: WHAT HAS WORKED AND WHAT HAS NOT? which IFFIm delivers additional financing over the technical expertise and reputation for financial long run. soundness has been called upon by partners to move this advocacy to action. The Bank has Another innovative approach is the advanced contributed to an international advisory group market commitment for vaccines, also aimed at and independent expert committee that has tackling communicable diseases on a global worked on the technical and structural options scale. Again, it would not appear that the Bank for a pilot advance market commitment, and was a prime mover in the genesis of the proposal, proposed that such a commitment target and hence the potential for it to play a first-mover pneumococcal diseased as the first problem and advocacy role was not captured. But the Bank's conduct subsequent work on malaria. 69 Chapter 8 Bengal tiger; photo ©DLILLC/Corbis, reproduced by permission. Improving the Bank's Support for Global Public Goods: Lessons from Experience T he Bank's country-based model works well in fostering GPGs when there is one (and preferably more) of the following: (a) a reasonable con- vergence at the country level of national and global interests and time horizons for taking action; (b) a dedicated cadre of staff with credible tech- nical and managerial competence to advance specific GPG concerns; and/or (c) grant finance to provide the right incentives for action. So the country model has its place, but these will include new approaches to setting budgets features are not always present. Where they and recognizing the performance of managers cannot be established, a very significant shift in and staff. On budget setting, one option is to set the country-based model is needed for the Bank aside significant administrative funding at the to find a way to bridge the gap even more corporate level to be allocated--transparently and effectively between global needs and country perhaps competitively--to high-priority GPG preferences. work at the country level. Care would be needed to make sure such funding was used as a genuine Indeed, looking ahead, the great shared global addition by country teams and not simply to challenges are increasingly those where national displace other activity. To provide better incentives and global benefits--actual or perceived, to staff, managers at all levels should consider immediate or for the next generation--diverge recognizing country- and global-level work on significantly from each other. For example, the GPGs in performance management systems. investments needed to protect the earth's climate and environmental commons vary Second, the Bank can consider clearer organiza- considerably at local, national, and global levels, tional arrangements to best select, and indeed as do the costs and benefits of such actions. link together, responses at country, regional, and There are lessons emerging from this review that global levels. Some Regions may want to have a suggest effective measures in five areas that may dedicated staff resource advancing work on help the Bank upgrade its ability to foster GPGs regional programs (and regional public goods), and to bridge the gap between global needs and as has been done in Africa, and perhaps expand country concerns more effectively. their purview to cover GPGs as well. But this is not a one-size-fits-all prescription, and other First, the Bank can create better incentives to Regions may have different arrangements deliver GPGs effectively at the country level. This suitable to their circumstances. 73 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Third, a more effective approach to the delivery instruments to help developing countries of the Bank's global knowledge and capacity to provide GPGs. The Bank could also explore country teams working on GPGs would be further ways to stimulate South-South exchange beneficial. To this end, the way the Bank can best of knowledge--and the development and deploy its expertise, particularly that of its application of new technologies designed with specialists located at the center of the institution and for the South--on contributing to GPGs, in the network anchors, should be reviewed. such as climate-friendly energy production and use. Fourth, the Bank and its stakeholders could renew attention toward ensuring that the Finally, a firmer and more precise justification is perspective of developing countries is connected needed for the costs and benefits of actions effectively with global responses. The Bank being proposed for the Bank's work on foster- might be able to use its standing more powerfully ing GPGs, to ensure that such work is financially to give greater voice to developing countries in and institutionally sustainable over the long the governance of significant global programs. It term. Particularly for global programs, the Bank should take a more proactive stance in must redouble its efforts to be more selective in advocating for development interests--and its engagement and more forthright in exiting developing country partners--in international those programs whose benefits and cost- forums (and agreements) dealing with GPGs. effectiveness are questionable. The Bank should That would include the Bank continuing to also be insistent about putting in place, and secure additional development assistance and to using, sound results frameworks, underpinned promote the design and use of market-based by realistic and cost-effective monitoring and 74 Appendixes APPENDIX A: PROJECT PERFORMANCE RESULTS This appendix presents long-term trends in percent in fiscal 2006. Project performance also project performance, based on IEG project declined when weighted by the value of evaluations using the year 1990 as a starting point disbursements, from 90 percent in fiscal 2006, to and with data up until June 16, 2008. The analysis 83 percent in fiscal 2007. of the Bank's lending effectiveness focuses on IEG's key performance criteria: the development How much do the results of fiscal 2007 affect the outcome of projects. three-year rolling average of the Bank's project performance? In the medium term, lending Performance Trends outcomes were satisfactory--in the three years up to end-fiscal 2007, 80 percent of projects Outcome satisfactorily delivered their targeted results, up Seventy-six percent of projects (by number) in from around 70 percent at the start of the exit fiscal 2007 were rated moderately satisfac- decade, as shown in figure A.2. But the three- tory or better, as shown in figure A.1, just about year average, which had increased every year meeting the Strategic Compact target of 75 since 1999, except for a small dip in fiscal 2003, percent. But this was a sharp decline of 7 was flat in 2007 because of the weak fiscal 2007 percentage points in performance from 83 cohort. Figure A.1: Project Performance Improves in FY06, But FY07's Cohort Declines 100 (%) 90 better 80 outcomes or with 70 projects satisfactory 60 of Share 50 moderately 40 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 Exit fiscal year By number of projects Weighted by value of project disbursements Source: World Bank database. 77 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Figure A.2: Project Performance Has Improved over the MediumTerm 90 (%) better 80 outcomes or with projects satisfactory 70 of Share moderately 60 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 Exit fiscal year (last in a 3-year rolling average) By number of projects Weighted by value of project disbursements Source: World Bank database. What drove the fiscal 2007 decline? Sometimes, a IDA versus IBRD lending, did not materially affect difference in the performance of the Bank's the outcomes. And the change in the sectoral portfolio from one year to the next can be composition and the share of projects utilizing influenced by a change in the composition of special lending instruments was only marginally projects being evaluated--for example, if there is unfavorable (affecting the total by less than 2 a larger share of projects in challenging sectors percent). or countries. But the change in portfolio composition in fiscal 2007 from the previous Another possible explanation is that the drop in three years--related to Region, sector, instru- measured project performance is due to ment, lending arm, proportion of loans to methodological changes in the way in which conflict-affected countries, and other factors-- projects were evaluated in fiscal 2007. Some does not explain the fall in ratings. Indeed, the changes in methods were introduced by IEG and fiscal 2007 composition of projects was actually the Bank, together in fiscal 2007, to strengthen favorable along several dimensions. the robustness of project ratings and to cover new elements of project design. In the near term For instance, even if the fiscal 2007 cohort had they may have introduced some element of maintained the same composition of adjustment discontinuity in the data series between fiscal and investment loans as in fiscal 2004­06, one 2007 and earlier years. It is estimated that the would expect the cohort to have been 75.3 influence of methodology changes has been percent satisfactory, practically the same result small--accounting for about 1 percentage point that occurred in the event. The Regional of the fall. composition in fiscal 2007 actually favored the outcome--had the Regional composition Besides these small changes in composition, remained unchanged, the overall fiscal 2007 what explains the weak performance of the fiscal outcome would have actually declined to 73.3 2007 cohort? It was driven by projects in health, percent. The change in proportion of lending to financial and private sector development, and conflict-affected and postconflict countries, or to public sector governance, which performed very 78 APPENDIX A: PROJECT PERFORMANCE RESULTS Table A.1: Project Performance in FY07 Was Not Dependent on the Composition of the Cohort Percentage of Satisfactory Projects Adjusting FY07 composition to reflect that of Would yield the following FY04­06 along the following dimensions: hypothetical outcomes: Region 73.3 FY07 composition of projects was slightly favorable Adjustment or investment 75.3 Proportion of conflict- or postconflict-affected countries 75.7 Actual FY07 Outcomes 75.8 percent FY07 composition of projects had no effect IDA, IBRD, GEF, or SPF 76.0 Sector 76.7 FY07 composition of projects was slightly unfavorable Lending instrument 77.5 Source: World Bank database. Note: Conflict- or postconflict-affected countries are those defined as such by the World Bank for any of the years fiscal 2003­06. poorly relative to the Bank average, as well as Second, overambition was a weakness. While projects in South Asia and Africa. In fiscal 2007, project objectives were almost always relevant, a health, financial and private sector development, majority was too far-reaching. Sometimes this was education and urban development showed in terms of assessing political commitment and the major declines, when measured by the number feasibility of certain reforms. Other times it was in of moderately satisfactory or better projects, as assessing government effectiveness and capacity, in South Asia, Africa, and Latin America, when or in requiring coordination across several compared with the prior 3-year period (see ministries or cumbersome financial management figure A.3). The portfolio of loans to conflict and procedures that were not manageable by the postconflict countries also showed signs of parties involved. IEG's 2008 evaluation of public worsening, with 76 percent moderately satisfac- sector reform shows several examples where these tory or better in fiscal 2007, as compared with 82 factors led to unsatisfactory outcomes. percent in the previous three years. However, they still perform better than loans to noncon- Third, delays in implementation caused difficul- flict countries, of which only 69 percent were ties, as circumstances changed and project moderately satisfactory or better. design or implementation could not respond. About one-fifth of underperforming projects In fiscal 2007, there was a greater occurrence of suffered from this problem. five key factors influencing weak outcomes. First, poor or overly complex project design was a Fourth, a majority of the unsatisfactory projects problem in more than half of underperforming had a weak results framework, with poor or no loans, a finding also made by the World Bank's baseline data, making it difficult to assess the QAG (World Bank 2008a). For instance, two outcomes of the project; and outcomes were projects failed to recognize the importance of an often not well linked to inputs and outputs. appropriate legal and regulatory framework as a precondition to a privatization process. Several Finally, various gaps in the Bank's own perform- health projects failed to ensure a heightened focus ance contributed to a lack of success. For example, on those interventions that would yield the in spite of being flagged by the Bank's QAG as greatestimpact,leadingforinstance,toinadequate having poor quality at the outset, three projects targeting of the poor, absence of a cost-effective were not reassessed or redesigned. The quality of package of health services, or inadequate funding the Bank's supervision was rated as moderately for behavior change interventions to prevent unsatisfactory or worse in two-thirds of all HIV/AIDS transmission among high-risk groups. underperforming projects (and many such 79 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Figure A.3: Projects in Financial and Private Sector Development, Health, and Public Sector Governance Decline in FY07; South Asia and Africa Lag Behind a. Outcome by sector board 100 %) 90 better outcomes or 80 with 70 disbursement, satisfactory by projects 60 of 50 Share moderately (weighted 40 mining supply policyrural reduction Educationprivate Transport and protection Environment development and and Bank-widenutrition,governance population Watersanitation development development Poverty developmentHealth, and Economic sector and Social Urban Energy Financial Social sector Agriculture Public Sector Board FY04­06 FY07 b. Outcome by Region 100 %) 90 better outcomes or 80 with 70 disbursement, satisfactory by 60 projects of 50 Share moderately (weighted 40 South Sub-Saharan Latin America Bank-wide Middle East Europe and East Asia Asia Africa and the and North Central and Caribbean Africa Asia Pacific Region FY04­06 FY07 Source: World Bank database. Note: Some changes in performance may be explained by a small sample size of projects that exited in fiscal 2007, including in the Environment (14 projects) and Poverty Reduction (4 projects) Sector Boards, and the South Asia Region (9 projects). projects were not identified as problems in to a challenge in re-emphasizing a proactive ongoing status reports). Bank overall perform- quality control in management's attention to ance--as distinct from borrower performance or ongoing project performance the effects of uncontrollable events--was ranked moderately unsatisfactory or worse in two-thirds Sustainability and Institutional of these problem projects, compared with only Development Impact about one-fifth of the full sample. All of this points Under the new harmonized evaluation criteria 80 APPENDIX A: PROJECT PERFORMANCE RESULTS for project evaluations, approved in October are the top performers for the fiscal 2003­07 2005, projects would no longer be rated for their cohort, exceeding the Bank average of 83 sustainability and institutional development percent. The Latin America and Caribbean impact (World Bank 2005b). Thus, only 122 of Region is the only one that declined in perform- the 249 exit fiscal 2006 projects that were ance for the fiscal 2003­07 cohort. In spite of evaluated received a rating for their sustainabil- improving the most in performance for fiscal ity, and 132 for institutional development. 2003­07, Africa continues to lag behind all other regions. Its performance declined in fiscal 2006 For this partial fiscal 2006 cohort, 83 percent of to 69 percent of outcomes moderately satisfac- projects are rated likely or highly likely to be tory or better, weighted by disbursement, and to resilient to future risks, maintaining the fiscal 2005 64 percent in fiscal 2007, an overall reduction of rating for sustainability. Fifty-five percent of the 10 percentage points from fiscal 2005. partial fiscal 2006 cohort is rated substantial or high on institutional development impact, a small Sectoral Performance decline from fiscal 2005.1 However, sustainability Compared with the fiscal 1998­2002 cohort, the ratings represented a 10 percentage point outcome performance weighted by disburse- increase, and institutional development impact ment for the fiscal 2003­07 cohort improved in 8 ratings a 3 percentage point increase, for the fiscal of 12 sector boards.2 Figure A.6 presents the 2006 (partial) cohort over the depressed fiscal sector boards' (and Bank-wide) outcome 2003 cohort ratings (figure A.4). performance in order of their fiscal 2003­07 performance. The biggest improvements in Regional Performance outcome ratings were in water supply and sanita- Figure A.5 presents the percentage of satisfac- tion, economic policy, energy and mining, and tory project outcomes, weighted by disburse- the social protection sector. The largest declines ment, for the fiscal 2003­07 cohort, as compared in performance were in health, nutrition, and with the fiscal 1998­2002 cohort. The East Asia population (which, along with economic policy and Pacific and Europe and Central Asia Regions and the environment, were significantly below Figure A.4: Long-termTrends in Sustainability and Institutional Development a. b. 100 (%) 100 better 80 or 80 sustainability (%) institutional with better 60 with or substantial 60 projects likely projects impact of 40 of 40 Share Share 20 1990 1993 1996 1999 2002 2005 development 20 1990 1993 1996 1999 2002 2005 Exit fiscal year Exit fiscal year By number of projects Weighted by value of project disbursements Source: World Bank database. Note: Data for 2006 are partial (shown by dashed line). 81 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Figure A.5: Africa Improves Relative to FY98­02, But Continues to Lag a. Share of projects with outcomes moderately satisfactory or better b. Distribution of disbursements, by Region, (weighted by disbursement) FY98­07 East Asia and Pacific South Asia Africa Europe and 15% 14% Middle East Central Asia and North Africa Bank-wide 5% RegionMiddle East and North Africa East Asia South Asia and Pacific Latin America and Latin America 24% the Caribbean and the Caribbean Africa 26% 40% 50% 60% 70% 80% 90% 100% Europe and Central Asia FY98­02 FY03­07 16% Source: World Bank database. the Bank-wide average for the fiscal 2003­07 PRSCs being ranked satisfactory in fiscal 2006. period) and in public sector governance. Some of the project ratings are still provisional, however, with full ratings available for only 25 Lending Instrument Performance percent of PRSCs. IEG's ongoing evaluation of The most recent data show a small dip in the PRSCs will shed more light on the performance share of development policy lending projects of this instrument. However, Learning and rated moderately satisfactory or better, both by Innovation Loans are performing below the Bank number of projects and value of disbursements average, and Adaptable Program Loan ratings (see figure A.7) as measured on a three-year have shown a decline in the two years since fiscal rolling average basis. Investment lending, too, 2005 in performance, but an increase as a has seen a decline in outcome performance. proportion of the portfolio in fiscal 2007. Twenty- one Development Policy Loans, the newest Bank New Lending Instruments instrument, have been rated so far, and have IEG has evaluated 241 operations employing the performed just above the Bank average, Bank's four new lending instruments-- weighted by disbursement, over fiscal 2003­07. Adaptable Program Loans, Development Policy Loans, Learning and Innovation Loans, and Bank-Managed Special Programs Poverty Reduction Support Credits. More than IEG has evaluated 87 operations financed under 90 percent of these operations exited the Bank's four Bank-managed special programs that have portfolio during the fiscal 2003­07 period, exited the Bank's portfolio since fiscal 2003 amounting to $10.5 billion in disbursements, and (table A.2). Seventy-seven percent of the special 17 percent of all the projects that exited during operations had satisfactory outcomes, a decline that period. PRSCs have pulled up the perform- of 8 percent as compared with operations exiting ance of the group, exceeding the Bank average between fiscal 1998 and fiscal 2002. This decline for the past five years of 77 percent and 82 in performance is mainly due to the lower percent, respectively, of moderately satisfactory satisfactory outcomes of Special Financing or better projects and disbursements (figure Grants or multidonor-sponsored trust funds that A.8). Their outcome ratings have increased have assisted five conflict-affected or postcon- steadily since fiscal 2003 with 100 percent of flict countries in the past decade.3 The resources 82 APPENDIX A: PROJECT PERFORMANCE RESULTS Figure A.6:Trends in Sectoral Performance a. Share of projects with outcomes moderately satisfactory or better (weighted by disbursement) Social protection Transport Financial and private sector development Water supply and sanitation Agriculture and rural development IMPROVERS Bank-wide Energy and mining Economic policy Environment board 40% 50% 60% 70% 80% 90% 100% Sector Health, nutrition, and population Public sector governance DECLINERS Urban development Education 40% 50% 60% 70% 80% 90% 100% FY98­02 FY03­07 b. Disbursements by Sector Board, FY98­07 Water supply and Urban sanitation Agriculture and development 4.0% rural development 4.0% 10.8% Transport Education 12.0% 8.5% Social protection 6.2% Energy and mining Social development 11.5% 0.4% Public sector governance 7.5% Environment 2.0% Poverty reduction 1.2% Health, Economic policy nutrition, and 10.9% population Financial 6.6% Global and private information sector and technology development 0.8% 13.5% Source: World Bank database. 83 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Figure A.7: Long-TermTrends in Development Policy Lending and Investment Lending a. Development policy lending b. Investment lending 100 100 (%) (%) 80 better better 80 outcomes or outcomes or with with 60 60 projects satisfactory projects satisfactory of of 40 40 Share Share moderately moderately 20 20 1992 1995 1998 2001 2004 2007 1992 1995 1998 2001 2004 2007 Exit fiscal year (last in a 3-year rolling average) Exit fiscal year (last in a 3-year rolling average) By project Weighted by disbursement Source: World Bank database. devoted to Special Financing Grants exiting between fiscal 2003 and fiscal 2007 were $493 million, or 0.5 percent of the Bank's overall project portfolio during this period.4 Figure A.8: Outcome Performance of New Lending Instruments 100 better outcomes or 80 with satisfactory 60 projects of Share moderately 40 Learning and Bank-wide Adaptable Development Poverty Reduction Innovation Loans Program Loans Policy Loans Support Credit New operations (FY03­07) By number of projects Weighted by value of project disbursements Source: World Bank database. 84 APPENDIX A: PROJECT PERFORMANCE RESULTS Table A.2: Bank-Managed Special Programs Are Performing on Par with Bank Average Exit FY98­02 Exit FY03­07 Disbursements Disbursements Number of (% of Outcome: Number of (% of Outcome: evaluated World Bank % evaluated World Bank % Special program type projects portfolio) satisfactorya projects portfolio) satisfactorya Global Environment Facility 39 0.1 79.5 39 0.46 84.6 Montreal Protocol Fund 4 0.0 100.0 5 0.05 100.0 Rainforest Initiative 1 0.0 100.0 2 0.00 100.0 Special Financing Grants 17 0.2 94.1 41 0.54 65.9 All Special Programs 61 0.3 85.2 87 1.0 77.0 Source: World Bank database. a. Projects whose outcomes are rated moderately satisfactory, satisfactory, or highly satisfactory are here referred to as "satisfactory." 85 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Table A.3: Outcome, Sustainability, and Institutional Development (ID) Impact by Various Dimensions, by Project, FY98­07 Exit FY98­02 ID impact: Number Outcome: Sustainability: % of Share % % likely substantial projects (%) satisfactorya or better or better Sector Board Agriculture and Rural Development 229 16 66.4 53.7 37.8 Economic Policy 74 5 56.8 65.2 29.7 Education 133 10 81.2 66.9 40.6 Energy and Mining 153 11 68.0 59.4 43.3 Environment 72 5 71.8 73.9 50.7 Financial and Private Sector Development 159 11 64.9 66.4 38.8 Gender and Development 0 na na na Global Information/Communications Technology 16 1 100.0 100.0 68.8 Health, Nutrition, and Population 108 8 65.7 61.0 36.9 Poverty Reduction 0 na na na Public Sector Governance 98 7 84.4 79.8 54.2 Social Development 1 0 100.0 100.0 100.0 Social Protection 84 6 83.1 58.3 45.8 Transport 137 10 88.0 78.4 64.7 Urban Development 67 5 71.2 50.0 36.4 Water Supply and Sanitation 67 5 65.2 46.9 33.3 Overall Result 1,398 100 72.5 63.8 43.4 Lending Instrument Type Dev Pol Lending 177 13 75.1 73.8 43.2 Investment 1220 87 72.1 62.3 43.4 Not Assigned 1 0 100.0 100.0 0.0 Overall Result 1,398 100 72.5 63.8 43.4 Network Financial and Private Sector Development 172 12 66.5 64.7 38.8 Human Development 325 23 76.6 62.8 40.8 Poverty Reduction & Economic Management 172 12 72.4 73.4 43.5 Sustainable Development 729 52 72.1 61.8 45.5 Overall Result 1,398 100 72.5 63.8 43.4 Region Africa 385 28 59.5 47.1 32.4 East Asia and Pacific 206 15 76.2 64.4 45.5 Europe and Central Asia 274 20 82.8 80.2 52.4 Latin America and Caribbean 277 20 76.4 69.4 50.8 Middle East and North Africa 105 8 73.8 64.9 39.8 Not Assigned 0 0 na na na South Asia 151 11 74.2 64.8 41.1 Overall Result 1,398 100 72.5 63.8 43.4 Agreement Type Not Assigned 1 0 100.0 100.0 100.0 Global Environmental Facility 39 3 79.5 63.2 51.3 IBRD 671 48 75.9 71.6 48.7 IDA 665 48 67.9 55.5 37.4 Montreal Protocol Fund 4 0 100.0 100.0 75.0 Rainforest Initiative 1 0 100.0 100.0 0.0 Special Financing Grants 17 1 94.1 71.4 41.2 Overall Result 1,398 100 72.5 63.8 43.4 Source: World Bank database. na = not applicable. a. Projects whose outcomes are rated moderately satisfactory, satisfactory, or highly satisfactory are here referred to as "satisfactory." 86 APPENDIX A: PROJECT PERFORMANCE RESULTS Exit FY03­07 Exit FY98­07 ID impact: ID impact: Number Outcome: Sustainability: % Number Outcome: Sustainability: % of Share % % likely substantial of Share % % likely substantial projects (%) satisfactorya or better or better projects (%) satisfactorya or better or better 188 14 83.1 78.0 60.1 417 15 73.9 62.5 46.5 76 6 81.1 80.0 41.0 150 5 68.9 71.4 34.8 136 10 81.3 85.4 59.8 269 10 81.3 74.6 48.7 83 6 77.5 80.0 67.9 236 9 71.3 64.8 49.8 84 6 75.3 73.1 52.5 156 6 73.7 73.6 51.5 124 9 72.1 80.7 56.1 283 10 68.1 71.9 45.6 1 0 100.0 100.0 100.0 1 0 100.0 100.0 100.0 11 1 100.0 100.0 50.0 27 1 100.0 100.0 62.5 109 8 61.5 72.6 59.2 217 8 63.6 65.9 46.4 21 2 81.0 90.9 53.8 21 1 81.0 90.9 53.8 124 9 64.5 74.4 41.9 222 8 73.3 77.1 48.1 23 2 71.4 66.7 42.9 24 1 72.7 69.2 46.7 96 7 83.0 74.5 50.0 180 7 83.1 65.0 47.7 119 9 92.4 86.5 62.7 256 9 90.0 81.4 63.9 78 6 79.2 77.0 44.8 145 5 75.5 63.2 40.6 71 5 84.3 78.8 59.6 138 5 75.0 61.2 45.5 1,344 100 78.1 78.9 54.8 2,742 100 75.2 69.9 48.1 202 15 79.8 88.1 52.6 379 14 77.6 80.3 47.6 1,140 85 77.7 77.1 55.1 2,360 86 74.8 68.2 48.2 2 0 100.0 100.0 100.0 3 0 100.0 100.0 50.0 1,344 100 78.1 78.9 54.8 2,742 100 75.2 69.9 48.1 124 9 72.1 80.7 56.1 296 11 68.9 70.5 45.2 341 25 75.4 78.4 56.8 666 24 76.0 69.4 47.7 222 17 71.9 77.7 42.9 394 14 72.1 75.5 43.2 657 49 82.7 79.1 57.6 1,386 51 77.1 68.5 50.4 1,344 100 78.1 78.9 54.8 2,742 100 75.2 69.9 48.1 316 24 67.1 66.3 47.7 701 26 62.9 54.2 38.3 202 15 83.5 80.9 64.4 408 15 79.9 71.2 53.6 282 21 83.2 89.6 62.9 556 20 83.0 84.4 57.0 318 24 82.6 81.3 56.7 595 22 79.7 74.6 53.4 111 8 73.6 70.9 33.7 216 8 73.7 67.6 37.0 1 0 100.0 na na 1 0 100.0 na na 114 8.5 78.1 83.8 54.8 265 9.7 75.8 71.3 46.0 1,344 100 78.1 78.9 54.8 2,742 100 75.2 69.9 48.1 0 0 na na na 1 0 100.0 100.0 100.0 39 3 84.6 76.2 53.8 78 3 82.1 67.8 52.3 603 45 82.3 83.9 57.6 1,274 46 78.9 76.6 52.4 654 49 74.4 75.7 53.2 1,319 48 71.1 63.6 44.0 5 0 100.0 100.0 50.0 9 0 100.0 100.0 66.7 2 0 100.0 100.0 100.0 3 0 100.0 100.0 50.0 41 3.1 65.9 48.2 37.5 58 2.12 74.1 56.1 38.8 1,344 100 78.1 78.9 54.8 2,742 100 75.2 69.9 48.1 87 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Table A.4: Outcome, Sustainability, and Institutional Development (ID) Impact by Various Dimensions, by Disbursement, FY98­07 Exit FY98­02 ID impact: Disburse- Outcome: Sustainability: % ments Share % % likely substantial ($m) (%) satisfactorya or better or better Sector Board Agriculture and Rural Development 11,669 11 77.8 67.3 45.4 Economic Policy 11,497 11 51.9 77.5 37.7 Education 8,125 8 86.6 76.4 44.7 Energy and Mining 15,234 14 68.6 62.3 44.0 Environment 1,517 1 58.8 84.9 37.4 Financial and Private Sector Development 18,019 17 84.1 85.9 51.3 Gender and Development 0 0 na na na Global Information/Communications Technology 1,297 1 100.0 100.0 61.5 Health, Nutrition, and Population 5,622 5 77.9 74.5 45.8 Poverty Reduction 0 0 na na na Public Sector Governance 7,638 7 86.8 91.1 52.0 Social Development 5 0 100.0 100.0 100.0 Social Protection 6,289 6 84.5 77.1 40.9 Transport 11,908 11 90.8 85.1 66.0 Urban Development 3,868 4 84.6 56.7 40.8 Water Supply and Sanitation 4,342 4 62.3 40.4 26.1 Overall Result 107,031 100 77.5 74.9 46.8 Lending Instrument Type Dev Pol Lending 38,247 36 77.6 84.8 46.0 Investment 68,783 64 77.5 69.7 47.3 Not Assigned 0 0 na na na Overall Result 107,031 100 77.5 74.9 46.8 Network Financial and Private Sector Development 19,658 18 84.2 82.3 48.9 Human Development 20,036 19 83.5 76.1 43.8 Poverty Reduction & Economic Management 19,135 18 65.8 83.0 43.4 Sustainable Development 48,202 45 77.0 68.7 48.8 Overall Result 107,031 100 77.5 74.9 46.8 Region Africa 13,143 12 60.0 48.5 28.7 East Asia and Pacific 28,502 27 80.6 81.4 50.3 Europe and Central Asia 19,187 18 75.3 82.6 51.4 Latin America and Caribbean 25,040 23 86.2 80.1 54.5 Middle East and North Africa 5,544 5 79.9 67.3 49.4 Not Assigned 0 0 na na na South Asia 15,616 15 74.7 70.8 37.5 Overall Result 107,031 100 77.5 74.9 46.8 Agreement Type Not Assigned 32 0 100.0 100.0 100.0 Global Environment Facility 138 0 80.0 69.6 55.6 IBRD 79,302 74 78.5 78.5 50.0 IDA 27,332 26 74.6 64.9 38.1 Montreal Protocol Fund 9 0 100.0 100.0 81.4 Rainforest Initiative 0 0 na na na Special Financing Grants 219 0 96.8 69.2 33.4 Overall Result 107,031 100 77.5 74.9 46.8 Source: World Bank database. na = not applicable. a. Projects whose outcomes are rated moderately satisfactory, satisfactory, or highly satisfactory are here referred to as "satisfactory." 88 APPENDIX A: PROJECT PERFORMANCE RESULTS Exit FY03­07 Exit FY98­07 ID impact: ID impact: Disburse- Outcome: Sustainability: % Disburse- Outcome: Sustainability: % ments Share % % likely substantial ments Share % % likely substantial ($m) (%) satisfactorya or better or better ($m) (%) satisfactorya or better or better 9,800 11 85.2 82.2 70.3 21,469 11 81.2 73.0 55.1 10,081 11 65.9 55.6 27.3 21,578 11 58.4 67.6 33.0 8,798 10 84.7 95.5 70.2 16,922 9 85.6 85.0 56.3 7,635 8 80.4 76.3 74.4 22,869 12 72.5 65.7 51.4 2,393 3 66.4 76.9 40.4 3,911 2 63.0 80.3 39.1 8,847 10 92.1 95.6 81.0 26,866 14 86.8 89.0 61.4 3 0 100.0 100.0 100.0 3 0 100.0 100.0 100.0 214 0 100.0 100.0 33.2 1,511 1 100.0 100.0 58.0 7,426 8 69.0 78.2 61.1 13,048 7 72.8 76.1 52.6 2,287 3 84.8 92.5 28.9 2,287 1 84.8 92.5 28.9 7,337 8 81.8 88.2 51.8 14,975 8 84.3 89.8 51.9 847 1 65.4 94.2 44.2 852 0 65.6 94.2 44.6 6,025 7 93.3 85.2 72.2 12,313 6 88.8 79.5 51.9 11,906 13 93.3 90.4 60.9 23,814 12 92.0 87.3 63.9 4,156 5 81.7 80.6 53.0 8,024 4 83.1 68.6 46.9 3,598 4 91.3 78.8 66.1 7,941 4 75.4 55.8 42.3 91,353 100 82.8 82.5 59.3 198,383 100 80.0 77.9 51.8 30,125 33 82.2 81.9 53.1 68,372 34 79.6 83.7 48.9 61,228 67 83.1 82.9 62.7 130,011 66 80.1 74.9 53.4 0 0 na na na 0 0 na na na 91,353 100 82.8 82.5 59.3 198,383 100 80.0 77.9 51.8 8,847 10 92.1 95.6 81.0 28,505 14 86.7 86.2 59.1 22,248 24 81.8 87.8 67.9 42,284 21 82.6 80.9 54.0 19,708 22 74.0 70.7 35.7 38,843 20 70.0 77.2 39.8 40,549 44 85.6 82.8 62.8 88,751 45 80.9 74.1 54.2 91,353 100 82.8 82.5 59.3 198,383 100 80.0 77.9 51.8 14,572 16 73.0 74.8 52.0 27,714 14 66.9 60.1 39.2 18,261 20 92.4 90.2 74.7 46,763 24 85.2 84.2 59.0 12,053 13 87.4 92.9 70.7 31,240 16 80.0 86.4 58.2 28,920 32 80.8 73.7 56.4 53,960 27 83.3 77.3 55.4 4,032 4 82.1 82.3 31.2 9,575 5 80.8 72.5 43.1 0 0 na na na 0 0 na na na 13,491 15 80.8 87.8 48.5 29,107 15 77.5 77.5 41.8 91,353 100 82.8 82.5 59.3 198,383 100 80.0 77.9 51.8 0 0 na na na 32 0 100.0 100.0 100.0 422 0 90.9 91.8 52.3 560 0 88.2 84.1 53.3 58,907 64 84.7 82.9 62.4 138,209 70 81.1 80.1 54.6 31,487 34 79.4 82.1 53.8 58,820 30 77.1 72.5 45.3 43 0 100.0 100.0 4.0 52 0 100.0 100.0 21.2 0 0 na na na 0 0 na na na 493 1 66.9 44.3 25.8 712 0 76.1 52.8 28.5 91,353 100 82.8 82.5 59.3 198,383 100 80.0 77.9 51.8 89 APPENDIX B: MONITORING AND EVALUATION OVERVIEW The Bank has in place the policies and the relevance, efficacy, and efficiency of Bank procedures to monitor, self-evaluate, and operational activities and processes. conduct independent evaluation of its opera- tions. The Regions conduct the monitoring and The Bank formally revised some of its policies to self-evaluating of their lending, analytical, and better monitor and evaluate the results of Bank advisory activities, and country assistance operations. The Bank issued a new OP/BP 8.60, programs under guidelines issued by the Development Policy Lending, in 2004, requiring Operations Policy and Country Services Vice- outcomes and measurable indicators for M&E in Presidency. Monitoring and self-evaluation of policy loans/credits as well as investment projects. operations Bank-wide is carried out by the BP 2.11, Country Assistance Strategies, was Quality Assurance Group, Operations Policy and updated in 2005, mainstreaming the results-based Country Services, and other units. The Bank is country assistance strategy approach. In 2007, the also strengthening its monitoring and self- Bank revised OP 13.60, which had focused on evaluation activities financed through trust funds dissemination and utilization of IEG findings, and its involvement in global programs and making it a policy on M&E. The new OP 13.60, partnerships. Monitoring and Evaluation, includes a section on Bank monitoring and self-evaluation and another Independent evaluation is carried out by IEG. on independent evaluation, outlining IEG's role. The Director-General of Evaluation reports OP/BP 14.40, Trust Funds, also provides guidance directly to the Executive Board, which approves on monitoring and evaluation for activities falling the Director-General's mandate and IEG's terms under that rubric, and has been updated to of reference. IEG validates the Bank's self- introduce enhancements, including greater evaluations, verifies their results, and assesses coverage of periodic independent evaluation. 91 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Table B.1: Summary of Bank Monitoring and Evaluation MONITORING Defining outcomes and Tracking and reporting Type of intervention or instrument monitoring indicators on implementation progress Loans and credits Results frameworks in Project Appraisal Documents Implementation Status Report (for investment loan/credits) and Program Documents (for development policy loans/credits) Analytical and advisory activities Concept paper specifies development objectives Activity update summary and tracking indicators Country programs Results framework in Country Assistance Strategy CAS Progress Report Trust-funded activities Initiating Brief for Trust Fund includes program Grant reporting and monitoring system objectives and accompanying performance for child trust fund/grant indicators Source: IEG. 92 APPENDIX B: MONITORING AND EVALUATION OVERVIEW EVALUATION Independent evaluation by IEG Self-evaluation Individual activity Aggregate level · Ex post: Implementation Completion ICR reviews (all); Project Performance ARDE and Results Reports Assessment Report (selected) · Ex ante: QAG Quality-at-Entry Assessments. · Bank-wide: QAG Annual Report on Portfolio Performance · Ex post: Activity completion summary Reviewed as part of sector/thematic and · Country: QAG analytic and advisory activity country assistance evaluations assessments · Bank-wide: QAG Annual Report on Portfolio Performance · Ex post: CASCR · CASCR reviews; ARDE, sector/thematic · Bank-wide: CAS retrospectives. · Country Assistance Evaluations evaluations · Less than $1 million: grant report and monitoring completion report · More than $1 million: Implementation Completion and Results Report 93 APPENDIX C: IEG'S SELF-EVALUATION: IMPROVING EFFECTIVENESS Overview tions from sector, thematic, and corporate evalua- IEG contributes to the Bank's effectiveness by tions. Bank management is accountable to the supporting the Board's oversight function and Board for follow-up. The MAR tracks the level of promoting learning within the Bank and the adoption and the status of individual recommenda- development community. One tool to gather tions. It presents management's ratings on these feedback on IEG's work is a client survey, albeit two indicators and IEG's assessment of the same. any such survey has limitations, with respect to sample size and scale of response. The most The 2008 MAR shows a continuing high level of recent survey found that among respondents, agreement by the Bank with IEG's recommenda- both inside and outside the Bank, there was tions. Some 96 percent of IEG proposals made in some increase in awareness of IEG's products. the last three years' evaluations have been Compared with earlier years, an increasing accepted by the Bank. In terms of the Bank then proportion of respondents reported general adopting those recommendations and putting satisfaction with the quality and timeliness of IEG them into practice, some 95 percent have been products. But they gave IEG lower marks for adopted with medium, substantial, or high depth of analysis and incorporation of all ratings, a level slightly above the previous year. available information. The use of evaluations for However, the share of recommendations adopted oversight among World Bank Executive Directors with substantial or high ratings was 42 percent, who responded to the survey remains high. Use which is below the level of previous years. of evaluation findings in the design of new operations by respondents who are operational Going forward, the challenge is for IEG to staff was higher in 2007 than a year earlier, but continue producing high-quality evaluations remains low. with sensible and practical recommendations, retain the high level of agreement on those The lower ratings for operational use and recommendations, and for the Bank to lift its satisfaction with depth of analysis point to the intensity of adoption and implementation. challenge of finding the right breadth and length for IEG products. IEG's work program for fiscal Improving IEG's Effectiveness 2009­11, discussed with CODE in May 2008, This appendix provides an overview of IEG's role assembles a good mix of activities, including in improving the Bank's development effective- increased emphasis on newer products such as ness. It includes a results framework that links CASCR reviews, clustered Project Performance IEG's mandate and objectives to its operations. Assessment Reports, and quicker turnaround Within this framework, the appendix updates reports on selected major issues of topical IEG's efforts to increase its evaluation focus on interest. results. The appendix includes findings from the annual client survey, an update on the communi- Beyond client feedback, the Management Action cations and outreach strategy, and the status of Record (MAR) allows IEG to track its recommenda- the MAR. 95 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 IEG's Results Framework The final outcome for IEG outputs is the use of IEG has three functions in improving develop- IEG knowledge about what works and why, ment effectiveness. First, it provides accountabil- leading to improved effectiveness of Bank ity by independently reporting on the results operations, and development assistance in achieved by Bank operations. Second, it distills general, in reducing poverty. For example, IEG's the Bank's operational experience into evaluation of regional programs has contributed knowledge of what works and why, and makes significantly to the recognition of the importance that knowledge widely available to Bank person- of regional approaches for the delivery of global nel and the global development community. public goods (GPGs) by the Bank.3 Outside the Third, it supports client governments with its Bank, the Bill and Melinda Gates Foundation technical knowledge on M&E through its evalua- used IEG's recent evaluation of World Bank tion capacity development activities. Assistance to Agriculture in Sub-Saharan Africa (2007d) as one of the main sources for the IEG has a mandate to assess "whether the World development of its agriculture strategy. The Bank Group's programs and activities are produc- Bank's operational staff, management, Board, and ing the expected results, including global, external clients use IEG's outputs to strengthen regional, and other programs in which the World actions taken in client countries. Measures that Bank Group is a participant."1 By reporting the attribute achievement of the final outcome to results of its evaluations to the Board of Directors IEG, however, are difficult to construct. and communicating the findings and lessons from its work to Bank management, operational Underscoring the importance of the above results staff, and the development community, IEG framework, the 2006 AROE recommended that expects to increase the Bank's effectiveness and "To further strengthen IEG's contribution to the influence Bank and client country decisions on workings of the results agenda in the World Bank policies, programs, and procedures. While fully Group, IEG should continue to follow its own independent,2 IEG is placed within the Bank to results framework and monitor it through the maximize its operational effectiveness and to AROE. Its focus on the usefulness of evaluation provide operational staff and strategic decision findings for its core audiences should be makers with knowledge that helps them work enhanced: for the Board through oversight, for more effectively. management through the incorporation of recommendations into Bank policies and strate- IEG's results chain is summarized in figure C.1, and gies, for Bank staff through the use of evalua- related to measures of performance that have tion findings for policy advice to country partners been collected for this report. IEG's outputs are and in project design, for external partners the findings, lessons, and recommendations from through the use of evaluation findings to improve its evaluations, and evaluation capacity support in their programs and policies, and for the client countries. Dissemination efforts are the countries more broadly. In playing this role, IEG intermediate step between outputs and outcomes. should specifically: For IEG's accountability function, the intermediate outcomes of these outputs are the use of evalua- · Improve the timeliness of its evaluations, tions by the Board to fulfill its oversight function, · Strengthen the operational relevance of the and the incorporation of IEG recommendations in findings, and Bank internal policies and procedures. For IEG's · Increase access to and exchange of the les- function as a knowledge provider, the intermedi- sons." ate outcome of IEG's outputs is the influence and use of these outputs to improve the Bank's policy This appendix describes the key links in IEG's advice and program and project design. It also results chain, giving special attention to the includes use of these outputs by external partners recommendations from last year, and highlights to improve their development work. recent developments in the results of IEG's 96 APPENDIX C: IEG'S SELF-EVALUATION: IMPROVING EFFECTIVENESS Figure C.1: IEG Results Chain Government institutions, policies, Final outcome programs made more effective Data sources Accountability through use by Learning from evaluation by Management Action Board and senior management operational staff Record Use of evaluation findings by Directors and their advisors Board members to provide staff to improve policy advice to clients and program and project Client survey on use and oversight function design Evaluation recommendations evaluations Use of evaluation findings by evaluations in Project Appraisal Intermediate operations and policy programs and policies Documents and sector strategies outcomes References to IEG in the media Dissemination of IEG products evaluations (client survey) Outputs --Findings, lessons, and recommendations Client survey on evaluation quality satisfaction Inputs IEG-WB budgetary resources (staff, consultants, and so forth) Source: IEG. approach. First, it looks at the activities IEG has At the level of country evaluation, the adoption undertaken to improve its output and the achieve- of CASCR reviews4 represents a major shift in the ment of outcomes; and second, it analyzes the Bank's evaluation system, because it ensures change in indicators that measure the quality of systematic evaluation coverage of CAS results. In outputs and achievement of IEG's outcomes. CASCR reviews, IEG reviews the relevance of the CAS to the country's development priorities, IEG's Outputs: Increasing Relevance and implementation of the country program, the Focus on Learning achievement of CAS objectives, and the quality of At the product level, IEG has continued its line of the CASCR itself. IEG then gives a rating for CAS quickly produced papers that address immediate implementation and Bank performance.5 As of needs for evaluative findings and lessons of March 2008, IEG has reviewed 59 CASCRs, 17 of experience, in the form of notes, presentations, which were completed in fiscal 2007. Going or briefing papers. IEG's short papers, in 2007, forward, IEG is planning to review up to 30 on governance received especially positive feed- CASCRs each year. Besides its rating function, the back. IEG's work plan for fiscal 2009­11 contin- CASCR review is intended as a learning tool that ues the shift to fewer and more influential distills lessons learned from implementation of evaluations, including some shorter, topical the preceding CAS for incorporation into the products, and an increasing number of evalua- design and implementation of the following CAS. tions conducted jointly across IEG (World Bank, To make its country reviews more useful and International Finance Corporation, and Multilat- relevant, IEG is also planning an enhanced eral Investment Guarantee Association). CASCR review, which would combine the timely 97 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 delivery of IEG's results, when the new CAS is importance of mutual learning, IEG added a discussed at the Board, with the insights of a learning event to the 2007 and 2008 awards country mission which would allow a much more ceremonies (see box C.1). thorough assessment of CAS achievements and challenges by IEG. Deepening Strategic Partnerships IEG's success in achieving greater focus on It has to be noted, however, that the existing results and learning requires strategic partner- process limits the effectiveness of the CASCR ships. For example, to capitalize on the impact review as a learning tool. IEG receives the final evaluation expertise available in other groups in CASCR for validation when the preparation of the Bank, IEG has been collaborating with the the new CAS is nearly complete. Therefore, it thematic group for poverty analysis, monitoring, may be too late for the country team to incorpo- impact evaluation, and the Development rate much, if any, of IEG's findings directly into Economics Department. Together, the three the CAS document. But lessons learned on the groups organized a two-day event in January implementation of the CAS should be consid- 2008, Making Smart Policy: Using Impact ered by the country team as it moves to Evaluation for Policy Making. implement the new CAS. This issue has not been addressed so far. In the case of enhanced CASCR Evaluation capacity development. IEG is strength- reviews, it would be even more important ening its support of results-oriented monitoring because these would need more preparation and evaluation systems and capacities in client time. countries. As a result of its 2004 self-evaluation of evaluation capacity development, IEG refocused To create incentives within the Bank for good its high-intensity support on a few targeted, performance in design, implementation, M&E, demonstration countries while maintaining low- and development effectiveness, IEG has been intensity support to a much broader range of giving annual Good Practice Awards to countries. In recent years, these efforts were operations that exemplify strong performance in particularly evident for Colombia, Mexico, and these areas.6 In addition to providing incentives China, and this support will now be extended to for high performance, the awards heighten the other regions. Also, in June 2008, IEG hosted the profile of operations that offer examples of good annual International Program in Development practices. To foster the exchange of lessons Evaluation Training (IPDET), in collaboration between operational staff and to highlight the with Carleton University, for the eighth time. This Box C.1: IEG Good Practice Awards: Secrets of Successful Operations-- An IEG Learning Event Since 2004, the World Bank's Independent Evaluation Practice Award winners to share their experiences and Group (IEG) has selected a small number of winners, lessons learned with other Bank staff in a learning event, from among operations evaluated in the previous fiscal Secrets of Successful Operations. In these engaging year, for its Annual Good Practice Awards. The main ob- discussions, Bank experts--including award winners jective is to highlight exemplary design, implementation, from the Europe and Central Asia and the Latin America and self-evaluation in Bank projects and country pro- and Caribbean Regions, who led the way in using eval- grams, and to create incentives among Bank staff for uation effectively--helped identify the challenges they greater learning from evaluation, to enhance development faced in their work and what they did to address them. effectiveness. The experts offered lessons for future operations and par- To facilitate replication of these good practices in ticipants were encouraged to identify specific lessons to other operations, in 2007 and 2008, IEG invited Good apply to their own work. 98 APPENDIX C: IEG'S SELF-EVALUATION: IMPROVING EFFECTIVENESS four-week course draws broad interest from Quality and Relevance of IEG Outputs: evaluation professionals and policy makers Results of the 2007 Client Survey worldwide. In addition, IEG has a formal partner- IEG measures the quality and relevance of its ship with the Chinese government to help outputs as part of its annual internal and external develop a regional center for development client surveys. The 2007 surveys asked target evaluation training in Shanghai, and a regional audiences for their views of IEG products version of the development evaluation training is prepared in 2006 and 2007, including 5 sector offered there on a twice-yearly basis. Since its and thematic studies, 5 corporate reports, 6 inception in 2001, more than two thousand CAEs, about 40 Project Performance Assessment practitioners have participated in the Interna- Reports, and an evaluation capacity development tional Program in Development Evaluation report. IEG surveyed a targeted sample of 4,218 Training. Evaluation capacity development internal clients, consisting of Bank staff and products on M&E methodology, impact evalua- Executive Directors and their advisors. The tion, and influential evaluations consistently response rate was 24 percent, as compared with draw strong interest among practitioners. 22 percent last year.7 The survey of external clients on published evaluations approached a Harmonizing development evaluation. As the largest sample of 6,238 individuals and achieved a and oldest of the evaluation units in the multilat- response rate of 19 percent. Given these eral development banks, IEG has taken a leader- response rates, it has to be noted that the survey ship role in harmonization efforts among the results are indicative for respondents, but cannot international evaluation community. IEG has be generalized to the surveyed population. actively promoted harmonization of develop- ment evaluation methods through the multilat- Readership and awareness. Sixty-one percent of eral development banks' Evaluation Cooperation Bank staff who responded to the most recent Group, the Development Assistance Committee annual client survey (2007) were aware of the Evaluation Network, and the United Nations evaluation for which they were surveyed. This is Evaluation Group. The Evaluation Cooperation above the 56 percent in 2006 and continues the Group has developed good practice standards for trend of increasing awareness of IEG products evaluation of both public and private sector among Bank staff respondents, only 39 percent development work. Member institutions have of whom indicated awareness in 2004. Among been benchmarked against these standards. respondents to the external survey, 75 percent Beyond that, IEG led the work on a joint Evalua- were aware of the evaluation for which they were tion Cooperation Group paper synthesizing surveyed, compared with 76 percent in 2006. findings and lessons on the linkages between infrastructure and environment operations. IEG The quality of IEG evaluations. As shown in figure took the lead in creating the Network of C.2, Bank staff were asked to rate their satisfaction Networks on Impact Evaluation, with members with IEG's evaluation for 10 attributes of quality such as the Development Assistance Committee on a six-point scale. Bank staff respondents were Evaluation Network, the Evaluation Cooperation more satisfied with the quality of IEG products in Group, and the United Nations Evaluation Group, 2007 than in 2006, across all attributes of quality. as well as developing countries and NGOs. IEG is Bank staff respondents continue to report highest providing the secretariat for this group, which is satisfaction with the relevance of IEG's evaluations developing good practice standards for develop- to their work, with 80 percent of respondents ment impact evaluation and promoting impact rating it 4 or higher, the highest rating in the past evaluations of development work. IEG also led four years. Bank staff respondents continue to be the Development Assistance Committee Evalua- least satisfied with the incorporation of all tion Network's work on developing good practice available information and the depth of analysis, standards and guidelines for evaluating global with 63 percent and 66 percent of respondents and regional programs. expressing their satisfaction, respectively. 99 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Figure C.2: Bank Staff Satisfaction with Quality of IEG Evaluations Improved in 2007 6 scale 5 4.34.4 4.24.4 4.3 4.1 4.2 4.0 4.1 4.0 4.0 4.0 rating 4 3.9 3.8 3.7 3.7 3.9 3.8 3.6 3.6 Survey 3 Client 2 IEG 1 Relevance to Ease of Concise Usefulness Timeliness Unbiased and Strong link Transparency Depth of Incorporation your work understandinga presentation of objective between and clarity analysisa of all of conclusionsa recommen- analysisa conclusions of the available dationsa and evidencea methodologya informationa 2006 2007 Source: IEG data. Note: Bars show the means on a six-point scale, where 1 = highly unsatisfied, and 6 = highly satisfied. The sample size for 2006 is 292, for 2007 is 417. a. Difference is significant at 90 percent confidence level. Due to changes in the survey methodology, a evaluations, the decline in ratings stopped in longer time series is only possible for a few 2007, but there is room for improvement. quality attributes. Figure C.3 shows that IEG continuously improved the timeliness and Executive Directors and their advisors who relevance of its products to operational staff. responded to the survey are very satisfied with Regarding views on the depth of analysis in IEG the quality of IEG products. As figure C.4 shows, Figure C.3: Satisfaction of Bank Staff with Quality of IEG Evaluations, 2004­07 90 80 80 80 76 73 76 71 70 67 66 67 60 56 50 40 Percent 30 20 10 0 Relevance of lessons to Bank's Presentation of conclusions Link between conclusions work and evidence 2004 (rated good or excellent) 2005 (rated good or excellent) 2006 (rated 4, 5, or 6, out of 6) 2007 (rated 4, 5, or 6, out of 6) Source: IEG data. 100 APPENDIX C: IEG'S SELF-EVALUATION: IMPROVING EFFECTIVENESS in 2007, Executive Directors and advisor respon- means of 3.85 for IEG's influence on how outputs dents were most satisfied with IEG's relevance to are linked to outcomes at the country level, and their work (with a mean of 4.95, 88 percent of 3.96 for IEG's influence on sector strategies. respondents rating it 4 or higher). External respondents' satisfaction with IEG's quality was Use of evaluations for assessing the Bank's sector very high, with 89 percent of respondents rating and country strategies is high among Executive all attributes of IEG's evaluations with a score of Directors and their advisors, with more than 90 4 or higher. percent rating it at 4 or higher, and policies and procedures, with 87 percent assigning ratings of In response to the question of whether evalua- 4 or higher. Executive Directors and their tions influenced their understanding of the advisors who responded use evaluations less for subject area, 74 percent of Bank staff respondents assessing Bank projects (73 percent rating it 4 or rated it 4 or higher, a marked increase over 58 higher). Overall, Executive Director and advisor percent in 2006.8 Ninety-four percent of respondents made more use of IEG evaluations Executive Directors and their advisors who in 2007 than in 2006. The same holds true for responded rated this aspect at 4 or higher, Bank staff respondents, whose self-reported use compared with 90 percent in 2006. Among has increased by 19 percent, on average, for all external respondents, 83 percent rated the types of usage since 2006.9 Bank staff respon- influence of evaluations on their understanding dents use evaluations mostly for commenting on of the subject area at 4 or higher, compared with the work of others, making a case for a particular 81 percent in 2006. At the more practical level of course of action, and providing advice to clients, influencing Bank strategies and the design of and less for modifying strategies or operations, results frameworks, Bank staff respondents rated or designing new projects or programs. IEG's influence a bit lower. On average, 58 However, in 2007, 45 percent of respondents percent rated these options at 4 or higher, with rated use for modifying ongoing operations at 4 Figure C.4: Satisfaction of Executive Directors with Quality of IEG Evaluations 6 Improvement over previous year Decline from previous year 5 4.9 4.9 4.8 4.8 4.8 4.8 scale 4.4 4.6 4.7 4.7 4.5 4.5 4.5 4.3 4.3 4.2 4.3 4.4 4.3 4.0 rating 4 Survey 3 Client 2 IEG 1 Relevance to Timelinessa Transparency Incorporation Ease of Usefulness Unbiased and Concise Strong link Depth of your work and clarity of all understanding of objective presentation between analysis of the available recommen- analysis of conclusions conclusions methodology information dations and evidencea 2006 2007 Source: IEG data. Note: Bars show means on a six-point scale, where 1 = highly unsatisfied, and 6 = highly satisfied. Sample size for 2006 is 50, for 2007 is 58. a. Difference is significant at 90 percent confidence level. 101 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 or higher, compared with only 27 percent in Communicating Knowledge from 2006. External clients use IEG evaluations mostly IEG Evaluations for research (76 percent), while 71 percent use it Effective communication of IEG's knowledge to to refocus ongoing strategies or programs. Bank staff, governments, other donors, and the Although use of evaluations was higher in 2007 international community is a crucial link than in 2006, respondents continued to point to between evaluations and outcomes. Over the last obstacles to usefulness, such as the lack of two years, IEG has undertaken extensive efforts specificity of recommendations to suggest how to improve its outreach to Bank staff and the to put them into practice, and lack of reflection wider development community. After a series of on the country context where the Bank operates, pilot initiatives, IEG mainstreamed several new which would make findings relevant to Bank approaches on media outreach, e-mail market- operations on the ground. ing, Web promotions and an improved Help Desk, and successfully increased awareness When asked about how to improve the evaluation among target audiences. for which they had been surveyed, 59 percent of staff respondents recommended that IEG make Awareness of evaluations, as reported by Bank its findings more operational, compared with 57 staff who responded to the 2007 client survey, percent in 2006. Broadening consultation with rose from 39 percent in 2004, to 61 percent in Bank staff was the second recommendation made 2007. The number of follow-up inquiries to IEG's by 47 percent of respondents (48 percent in Help Desk increased tenfold during 2005 and has 2006). This was followed by improving depth of stayed at the same high level since, with about analysis (2007: 41 percent; 2006: 45 percent) and 2,400 inquiries each year. Help Desk inquiries are broadening external consultation (2007: 40 concerned with evaluation methodology, advice percent; 2006: 35 percent). Executive Directors on M&E systems, and IEG's product portfolio, and their advisor respondents made similar suggesting that IEG's outreach campaigns are suggestions, but 54 percent emphasized the triggering follow-up questions among key importance of broadening external consultation audience segments. more than consultation with Bank staff, which only 27 percent of respondents chose as a Over the past two years, IEG organized 11 media recommendation to IEG. External respondents' outreach campaigns to coincide with the release main recommendation was to broaden external of its reports. These campaigns have produced consultation (59 percent), followed by more extensive media coverage and generated an operational findings (51 percent) and obtaining estimated 200 articles and reports in audiovisual more evidence (47 percent). media over the last two years.10 The growing number of references in the media to IEG evalua- In 2007, the survey also included questions tions indicates strong interest in the Bank's about the timeliness and relevance of four IEG performance on key development initiatives. quick products from 2006 and 2007. These were generally well received. Eighty-four percent of IEG's communication and outreach can be Bank staff and Executive Directors and advisor strengthened by providing more summaries of respondents rated them at 4 or higher on timeli- IEG findings. In the client surveys, about two- ness, and 80 percent rated them at 4 or higher on thirds of all client groups made this recommen- relevance. However, awareness of these notes dation in both 2006 and 2007. Bank staff (39 was low, with senior management respondents percent of respondents) and Executive Directors (42 percent indicated no awareness with any of and advisors (53 percent of respondents) the four notes) and staff, whose awareness was continue to ask for more online accessibility to below 40 percent for all but one note. Executive IEG content. Among external respondents, 60 Directors and advisors who responded were percent asked for more training/education generally well aware of these notes. material. Forty percent of Executive Directors and 102 APPENDIX C: IEG'S SELF-EVALUATION: IMPROVING EFFECTIVENESS advisor respondents made this recommendation The MAR for 2008 tracks management's progress as well. Going forward, IEG will work on its Web on 57 recommendations. These include 22 new site, focusing on the development of new search recommendationsfromthesixIEGstudies(exclud- tools that will make its content more accessible. ing CAEs) presented to the Board in calendar year 2007, and 35 recommendations carried forward IEG Intermediate Outcomes: Evaluation from calendar years 2005 and 2006. Recommendations Incorporated into Bank Operations and Policy The 2008 MAR shows a continuing high level of IEG influences the Bank's effectiveness through agreement from the Bank with IEG's recommen- recommendations to management as part of dations. Of the 57 recommendations in the 2008 sector, thematic, and corporate evaluations, as MAR, 96 percent (55 recommendations) have well as Country Assistance Evaluations. Bank been accepted by Bank management,13 similar to management is accountable to the Board for the rate of acceptance in earlier years (95 percent follow-up. One of the intermediate outcomes for in MAR 2007 and MAR 2006). IEG is the extent to which management incorpo- rates IEG recommendations and findings in In the 2008 MAR, as shown in figure C.5, IEG rated policy advice, program design, and project adoption medium or better for 95 percent of 52 design (see box C.2). recommendations. This compares with 85 per- cent (47 recommendations) in the 2007 MAR and The Management Action Record allows IEG to 94 percent (72 recommendations) in the 2006 track its recommendations and to monitor the MAR. The share of recommendations adopted degree to which they have been adopted by with substantial or high ratings was 42 percent in management. The MAR tracks two indicators: the the 2008 MAR, down from 60 percent in the 2007 level of adoption11 and the status of individual MAR. Therefore, the distribution of adoption recommendations.12 The MAR presents manage- ratings among the top three categories (medium, ment's ratings on these two indicators and IEG's substantial, and high) has shifted toward medium. assessment of the same. The MAR includes IEG recommendations from the previous three The Bank's own assessment shows this shift and calendar years. it is confirmed by IEG. In this regard, it is Box C.2: Selected Impacts of IEG Evaluations · The recently announced credit line to deal with catastrophes, grades. A major donor-financed workshop in March 2008 such as earthquakes and hurricanes in MICs, directly reflects used findings and benchmarks identified by IEG. IEG's recommendation that a new mechanism be established · IEG's review of lines of credit led to a Bank-wide effort to iden- to meet urgent needs in the early days of a disaster response. tify and review their quality, sharing of experience among · The Bank's recently promulgated strategy "Strengthening donors, and an Operations Policy and Country Services ini- the World Bank's Rapid Response and Long-Term Engagement tiative to require early identification of lines of credit. in Fragile States" responds directly to IEG's recommendation · IEG's evaluation of private sector development in the power that internal Bank support for fragile states--including staff sector contributed to a reassessment of the Bank's approach numbers, staff skills, guidance, and organizational incen- to infrastructure. tives--needs to be strengthened. · The evaluation of social development fed directly into the · Prompted by IEG's primary education study and the accom- preparation of the Bank's Social Development strategy. panying volume of science-based sectoral knowledge, Bank · The global programs review resulted in greater scrutiny and projects more often now aim to measure learning outcomes, streamlining of the Bank's approach to and governance of and, in particular, the reading fluency acquired in the early global programs. 103 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Figure C.5: Adoption of IEG Recommendations Has Declined, but Agreement with Management Is Higher in 2008 100 6% 15% 5% (%) 80 27% 25% 53% category 60 44% rating 40 44% of 31% 20 Share 22% 16% 11% 0 2006 2007 2008 Sample size = 77 Sample size = 55 Sample size = 55 Management Action Record High Substantial Medium Negligible Source: IEG data. noteworthy that agreement on adoption ratings program and project designs. The 2006 AROE between IEG and management was higher in found that IEG provides high-level knowledge 2008 than in previous years. IEG and manage- that is useful for assessing programs, giving ment agreed on the rating on level of adoption advice to clients, and making comments, but for 65 percent of recommendations (36 out of needs to focus on influencing ongoing and future 55).14 And the disconnect between high and operations. Focus groups conducted for that substantial adoption ratings by IEG and manage- report showed that Task Team Leaders are least ment was 18 percent (IEG rated 42 percent of likely to report that they incorporate evaluation recommendations high or substantial; manage- findings into planning and design. Their ment 60 percent) in 2008, compared with 28 operational context requires information on how percent in 2007 and 21 percent in 2006. to conduct monitoring, establish indicators, and prepare projects for evaluation. Respondents in Going forward, the challenge is for IEG to management positions reported greater use and continue producing high-quality evaluations usefulness of IEG products. CODE members rely with sensible and practical recommendations, heavily on IEG's reports, advice, and recommen- retain the high level of agreement on those dations. These findings were also borne out in recommendations, and for the Bank to lift up its the 2007 client survey in which 72 percent of intensity of adoption and implementation. Bank staff respondents reported using IEG evaluations for making comments and giving Since 2006, recommendations that are older advice to others. However, only 54 percent of staff than three years are being retired from the MAR, report using evaluations for designing new subject to review by CODE and the Board. operations, and 45 percent report using evalua- tions for modifying ongoing operations. IEG Intermediate Outcomes: Use of Executive Directors and their advisors make Evaluation Findings by Bank Staff at the heavy use of evaluations (90 percent indicating Operational Level substantial use) for their oversight function of IEG's performance on this intermediate outcome sector and country strategies, and Bank policies, can be measured by whether Bank staff use IEG but less so for project-level oversight (76 percent findings to improve Bank policy advice and of respondents indicating substantial use). 104 APPENDIX C: IEG'S SELF-EVALUATION: IMPROVING EFFECTIVENESS To improve IEG's impact on Bank operations, IEG evaluation will deliver operationally relevant will need to continue giving greater emphasis to results and be geared toward learning. At the same applied learning and real-time use of evaluation time, close engagement challenges the evaluator's findings, to improve Bank performance. To meet independence and might distract from the this challenge, IEG needs to define its stance on necessary accountability perspective. Finding the engagement and learning with guidelines and right balance and providing the proper incentives funding for its staff. The more that operational to IEG staff for engagement with operational staff staff are involved up front in the evaluation design during the evaluation and afterward for learning and during the evaluation, the more likely the requires clear directions from IEG leadership. 105 APPENDIX D: FEATURES OF GLOBAL PUBLIC GOODS Both the demand and supply features of GPGs work well to encourage countries to integrate the can provide an obstacle to their provision. GPG into their national programs because the Individual countries "demand" GPGs only to the country's interests are already closely aligned extent that this demand serves a national interest. with the worldwide interests. For example, a Moreover, the features of different GPGs imply country will likely be more interested in control- large variations in the manner through which a ling HIV/AIDS within its borders if it has a high GPG should be supplied. This section provides a HIV prevalence rate (and hence the health and framework for considering how the Bank and the social consequences are already felt at the international community should arrange for the national level). In such cases, the Bank should most efficient approach toward providing GPGs. have influential national counterparts who are receptive to its support. In such circumstances, From the Demand Side traditional Bank instruments such as IBRD and The extent of divergence between national and IDA lending can often be used. global benefit (or cost) of a GPG varies across a spectrum, and so the extent to which a particular At the other extreme, when the net benefits of good is truly a GPG is not always easy to assess the GPG are large at the dispersed global level precisely. Clean air provides a good example of a but only minimally appropriable at the level of an GPG. In contrast, some facets of communicable individual nation state--as is the case for climate diseases are primarily national in scope, notwith- change--the Bank's country-based model may standing the fact that there may be other reasons not be as useful because of that very divergence whyresponsesaresupportedbyglobalpartnerships. (depicted in figure D.2). In such instances, the This distinction is noteworthy. A country's Figure D.1: Country Perception of Net Benefits When perception of the divergence between national National and Global Interests Coincide and global benefits and costs of GPGs influences whether a country "demands" a GPG and, National net benefit Global net benefit therefore, whether there is action on the country level, the extent to which such engagement occurs, and the types of instruments that are used. For the Bank, the level of overlap between national and global net benefits can be one The Bank's country-based model can determinant of the type of role it can play in work well, for example, in the case of an HIV/AIDS control program. fostering GPGs, and the extent to which strategic intent is translated into action on the ground. At one extreme, when the national benefits of the GPG coincide in large part with the benefits accruing to the global community (depicted in Source: IEG. figure D.1), the Bank's country-based model can 107 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 Figure D.2: Country Perception of Net Benefits When can use to, as it were, "push the circles together" National and Global Interests Diverge and create a coincidence of national and global interest. Those levers include: (i) financing National net benefit Global net benefit instruments; (ii) global programs, and (iii) advocacy--including convening power and support for international frameworks. The Bank's convening power and global-level From the Supply Side advocacy may be more effective, for example, GPGs also have different features in the way they in addressing climate have to be supplied and this has implications for change. the way in which the Bank (and other bodies) deal with them. As shown in table D.1, some public goods are supplied through aggregate efforts, Source: IEG. others by single best efforts. Some depend on the weakest link, others require coordination. These Bank may be more successful in fostering GPGs distinctions are important because they reveal by leveraging its convening power and providing different incentive challenges. While the desire of constructive advocacy at the global level. some countries to coordinate may be strong, the supply of public goods requiring an aggregate Between the two extremes, when there is some effort is usually prone to free riding. Supply of level of convergence in national and global net public goods requiring a single best effort can be benefit, the Bank has a chance to prompt easier to achieve than supply of a weakest-link national action, but has to "push hard at the public good, which can be undermined by a single door." In fact there are several levers the Bank failed state. 108 APPENDIX D: FEATURES OF GLOBAL PUBLIC GOODS Table D.1: Bank Uses Different Vehicles to Support the Provision of DifferentTypes of GPGs Supply-type of Global Public Good Single best effort Weakest link Aggregate effort Coordination The action of a single actor Inaction or a weak effort A critical mass of countries Countries act in harmony or country (or small group of by a single country leads each contributes individually to jointly achieve an actors) provides a GPG. to undersupply of a GPG. to a GPG agreed upon GPG. For example, researchers in For example, the inability of For example, countries reduce For example, countries agree to the United States developed a country to eradicate a or eliminate their production standards of measurement two polio vaccines in the communicable disease forces of ozone-depleting that facilitate international 1950s that were available to other countries to vaccinate substances to preserve trade and cross-country many other countries. against it. the global ozone layer. comparisons Crises and systemic risk Climate change The Financial Sector The Carbon Fund mitigates Assessment Program helps against climate change through countries identify market-based mechanisms. vulnerabilities in their The Global Environment Country-based financial systems and Facility (GEF) fights climate interventions determine needed reforms in change and protects order to avoid financial crises biodiversity through grant and potential global contagion. finance. activity Vaccines Avian influenza Bank-supported The Global Alliance for The Bank conducted research Vaccines and Immunizations that outlined the costs and orld conducts research to provide benefits of avian influenza W new vaccines for communi- control, which served as an cable diseases that could then umbrella for international Advocacy and be shared with all countries. action. research Trade The Bank's research and ad- vocacy on trade provides pres- sure and evidence to encour- age a prodevelopment Doha round of WTO negotiations. Source: IEG. 109 APPENDIX E: MANAGEMENT COMMENTS Introduction performance benchmarks in each of the three This year's Annual Review of Development years to end-fiscal 2007. Improvements have Effectiveness (ARDE) tracks World Bank perform- been particularly impressive in the Africa Region ance and examines a particular thematic topic, the and in the water supply and sanitation sector. Bank's work in fostering global public goods The report goes on to highlight and discuss two (GPGs). Management welcomes the new two-part signs of weaknesses in the data for fiscal 2007, format of the ARDE and considers the thematic including a jump in the so-called disconnect-- topic timely and highly relevant. In Part I, the that is, the difference between the outcome Bank's Independent Evaluation Group (IEG) ratings provided by Bank staff in the final reports on recent trends in the outcomes of Bank Implementation Status and Results reports projects and country programs and analyzes (ISRs) for ongoing projects and IEG's ratings in progress in the quality and coverage of monitor- its reviews of the Bank's Implementation ing and evaluation (M&E) systems employed by Completion and Results reports (ICRs)--and a the Bank. In Part II, IEG reviews the Bank's decline in the share of projects with satisfactory experience with (a) country-based support for outcomes, from a high of 83 percent in fiscal client contributions to the supply of GPGs and (b) 2006 to 76 percent in fiscal 2007. its advocacy for action on GPGs. This appendix provides brief responses to IEG's findings and Actions in response to warning signals. Management suggestions in Part I and Part II, respectively. concurs that a correct rating of likely outcomes in ongoing projects is necessary for its ability to take Tracking Bank Performance timely remedial action in problem projects, and Management values the review of development that the fall in IEG exit ratings and the increased outcomes and the Bank's M&E practice. IEG's disconnect in fiscal 2007 must be recognized as a feedback fosters learning from experience and is warning sign. This parallels findings under our a factor in the medium-term improvement of more detailed review of IDA controls as well as Bank performance documented in the ARDE. QAG reports and the India detailed implementa- Management agrees that the weakening of tion review. Management is taking on these issues development outcomes in projects that exited in the context of investment lending reform, the portfolio in fiscal 2007 warrants attention and notably changes in our supervision practices. As outlines its actions below. Management also IEG notes, the more problematic areas are the agrees with most of the suggestions to further noninfrastructure sectors and low-income improve the quality and coverage of M&E in countries (LICs), especially fragile states. Manage- projects, country programs, and global programs ment is considering a more customized approach and partnerships (GPPs). for implementation to reflect the complexities of these operational situations, moving beyond the Project Outcomes traditional notion of supervision and incorporat- The ARDE confirms that the ratings for project ing the possibility of implementation support outcomes have significantly improved over the directed at capacity building. However, manage- medium term and have exceeded the Bank's ment is not waiting for the introduction of these 111 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 reforms. Each Region has conducted its own likely reflects in part the temporary impact of review of ratings in ongoing operations. In the a recent decision to report on programmatic South Asia, East Asia and the Pacific, and Latin development policy loans only at the end of the America and Caribbean Regions, Regional vice series. As a result, for a few years, one-off de- presidents have taken the lead in reviewing ISRs, velopment policy loans will make up a larger with instructions for action by task teams. The share of the sample for which outcome ratings Africa Region has adopted specific measures are available. With programmatic development (involving country management, sector manage- policy loans rated better than one-off devel- ment, and the quality and knowledge services opment policy loans, on average, this shift in units) to reduce the disconnect, ensure realism in composition has an adverse, but likely transi- the ratings of ongoing operations, and strengthen tory, effect on the average rating for the sample. accountability for project quality. In addition, the · Vigilance about "overly complex" project de- upcoming Development Policy Lending signs must not discourage staff from responding Retrospective will review the Bank's rating to client demand for sophisticated--and at practice for development policy loan operations times necessarily complex--products, and, as necessary, will recommend appropriate particularly in middle-income countries (MICs). actions to ensure that management receives The risk of not achieving satisfactory outcomes timely information on weak performance in such may be higher in such circumstances, but tak- operations. ing that risk is necessary for the Bank's con- tinued relevance to its members. The issue is Decline in fiscal 2007 project outcome ratings and not so much reducing complexity as it is hav- increase in disconnect--some observations. ing clear objectives and effective systems for Effective support for members' development tracking operational performance. efforts requires the Bank to take risks; · In addressing task team performance, it is im- consequently, not all the operations it supports portant to take into account the challenges will achieve the desired outcomes. Against this that result from the Bank's commitment to in- background, the share of Bank projects with a creased harmonization and collaboration with satisfactory development outcome in recent development partners, particularly in LICs. years has risen to a reasonable level--a level at Staff must devote more time to coordination which year-to-year variations are to be expected. and are generally more dependent on the ac- However, management agrees that vigilance is tions of others for achieving results, notably warranted to ensure that the fiscal 2007 drop in pending greater harmonization around agreed ratings signifies a variation around that trend and results frameworks. not the beginning of a decline. IEG is to be · The jump in the disconnect can in part be commended for examining the possible impact traced to the introduction of a new evaluation of sample composition bias and of changes in the methodology in fiscal 2007, specifically to dif- evaluation methodology on the fiscal 2007 ferent paces of implementing the agreed rating outcomes, and for recalling key factors about system and rating criteria between IEG and the which the Bank must always be vigilant, includ- larger and more layered Bank operational com- ing poor or overly complex project design, plex. The ARDE estimates this effect as ac- articulation of overly ambitious expected counting for one percentage point of the jump outcomes, implementation delays, and in the disconnect, but it may have been higher. weaknesses in project supervision and staff As more ISRs and ICRs apply the methodology performance. In addition, management believes already used in IEG's reviews, the divergence the following considerations to be pertinent. in ratings can be expected to decline. · Reduced outcome ratings for development Country Program Outcomes policy loans contribute to the reported over- The ARDE reports the average development all drop in fiscal 2007 ratings. That reduction outcome ratings for some two decades of 112 APPENDIX E: MANAGEMENT COMMENTS country programs on the basis of IEG's Country cally the Bank's role in supporting a country's Assistance Evaluations (CAEs), and, for more development strategy, outcome ratings tend to recent country programs (starting in fiscal 1999), improve. It is also important to recognize that on the basis of IEG's reviews of Country CAS program outcomes can change during Assistance Strategy Completion Reports the CAS period as the Bank responds flexibly (CASCRs). The ARDE observes that in both sets, to changing client demand for its services. average outcomes for MIC programs surpass those for LIC programs by a considerable margin, Monitoring and Evaluation and that higher outcome ratings for the more The ARDE recognizes that as part of the results recent country programs--those for which agenda the Bank has put in place strong policies CASCRs are available--reflect improvements in and procedures for the monitoring and evalua- MIC programs only. It also compares average tion (M&E) of project and country program country program outcomes with average project outcomes. For the projects and country outcomes and suggests that the lower rating for programs reviewed, however, it reports a low country program outcomes may indicate a failure quality of M&E and results frameworks. Manage- of such programs to exploit synergies between ment agrees with IEG's recommendations for the Bank's development services. Management improving the quality and use of M&E, including offers two comments. increased focus on providing good baseline information for project outcomes, articulating · The gap between the outcomes of MIC and LIC more clearly the link between project outputs programs is a matter of concern to manage- and targeted outcomes, simplifying the CAS ment. However, averages over long durations results frameworks, and using them not only for do not shed light on the underlying factors. evaluation but also for program management. Management encourages IEG to analyze Ongoing efforts in the Regions to improve M&E changes in outcomes over time for subgroups are indeed focusing on these aspects, as is of countries, with the analysis taking account management in its reviews of proposed develop- of major changes, such as the introduction of ment policy loan operations and CASs. Manage- CAS results frameworks and changes in evalu- ment believes that the ARDE could have given ation methodology. Moreover, management more recognition to initiatives such as Regional wants to note that it has not yet introduced a M&E support for task teams, the development of standard rating methodology for CASCRs, and scorecards in a number of countries, and greater the methodology used by IEG self-initiated use of M&E frameworks for management ratings of CASCRs has been evolving and has purposes (for example, for annual country not yet been finalized. portfolio performance reviews in the Latin · IEG's suggestion that lower outcome ratings for America and the Caribbean Region). The ARDE country programs than for projects indicate might have also given greater recognition to the that country programs fail to exploit synergies commitments and actions to date under the among Bank development services is not sub- IDA14 and IDA15 Results Measurement Systems, stantiated in the report. The report acknowl- notably in terms of baseline operational data. edges some of the factors that, in management's view, deprive the comparison of Management does not share IEG's view that staff informative value (for example, different per- incentive problems continue to be a major formance standards). Management notes that obstacle to the improvement of M&E in projects performance in country programs predating and country programs; rather, it believes that the the introduction of CAS results frameworks in new policies and procedures and the ongoing fiscal 2005 is measured against objectives that efforts for enhancing practice are improving do not clearly distinguish between country quality in the more recent projects and programs. outcomes and CAS program outcomes. As pro- It would have been appropriate in this regard to gram outcomes come to reflect more realisti- highlight the challenge of country capacity for 113 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 results management, since lack of data in poorer that date. As noted in the report, in the small countries often is the most important obstacle to sample of reviewed projects approved since improving M&E frameworks. Helping partner fiscal 2005, the share of projects with substantial countries overcome these data problems is a or high M&E quality doubled. As more IEG Bank priority, notably through the Marrakech reviews of results-based CASs from the second Action Plan for Statistics, the framework round become available, the ratings for the discussed by the Board in May 2006. While this is quality of results frameworks are likely to a long-term effort, there has been progress. A key improve as well. target of this plan is to help all low-income countries develop a national strategy for improv- Management finds the ARDE's review of experi- ing their statistical system. All but one Sub- ence with the Bank's impact evaluations helpful Saharan African country (Somalia) have a strategy and agrees that their increased use is important or are working on one. The Bank's multidonor for strengthening the knowledge base for Trust Fund for Statistical Capacity Building has reporting on the Bank's corporate results; provided 20 new grants to countries for strategy however, management is also aware of practical preparation since 2006. Following the Third barriers to impact evaluation work in the context International Roundtable on Managing for of specific operations. Management concurs Development Results, in Hanoi, staff have been with the emphasis in the report on strengthen- working closely with other development partners ing results reporting at the corporate level. It through the Partnership for Statistics for recently proposed to prepare a Bankwide results Development in the 21st Century to find the report that will provide an overview of results resources to scale up the implementation of achieved through Bank activities and report on these strategies. Part of the effort is mainstream- progress in the Bank's results focus and ing statistical capacity building in country measurement systems. operations. New statistical capacity-building (STATCAP) lending projects have been approved Shared Global Challenges for Kenya and Russia, and pipeline countries The ARDE reviews the Bank's experience with include Bolivia, Tanzania, and India. Two country program support for GPGs and offers a donors--the Netherlands and the United number of recommendations on bridging the gap Kingdom--are working with the Bank to between global needs and country preferences. It establish a new Statistics for Results Facility, which also reviews the Bank's advocacy work on GPGs. will promote better coordinated efforts by donors at the country level, and will provide grants to Country-Based Support for GPGs help finance the implementation of national Management welcomes the report's many statistical improvement plans and link them more valuable insights into the challenges of support- closely with national development strategies. ing countries' contributions in cases where Staff are working to launch this facility in Ghana at global and country interests diverge significantly the forthcoming High-Level Forum on Aid and international agreement on a course of Effectiveness. collective action has not yet been reached. Management also appreciates the careful review In discussing the M&E of projects and country of the Bank's work on the country level--in the programs reviewed, the ARDE could have paid context of country programs and through more attention to an obvious legacy effect. The country-level activities of global programs and large majority of projects reviewed were partnerships--and the proposals for strengthen- approved prior to the adoption of the new ing that work. Management agrees with many of policies and procedures; and nearly all results- the observations and assessments. Some, based CASs reviewed were from the first round, however, require comments, notably those on in which results came from an existing portfolio the extent of Bank involvement on the country of operations that had been approved prior to level, the Bank's ability to provide attractive 114 APPENDIX E: MANAGEMENT COMMENTS financial support for the GPG efforts of MICs, the Financial support for MICs' contributions to GPGs. reliance on the country program model, the The ARDE presents ample evidence of the attention paid to fostering GPGs in country and effectiveness of concessional finance in bridging regional strategies, the deployment of global gaps between global needs and country prefer- programs, and the option of setting aside ences in International Development Association- corporate administrative funds for high priority eligible countries. With respect to International GPG work at the country level. Bank for Reconstruction and Development (IBRD)-eligible countries, the report recognizes Bank involvement in GPGs at the country level. The the Global Environment Facility (GEF) as a ARDE observes that GPGs other than environ- source of concessional financing and carbon mental commons are not sufficiently finance as an innovative mechanism the Bank emphasized in CASs and that the extent of Bank can deploy also in MICs. But the report pays involvement in GPG issues varies widely among insufficient attention to the Bank's ability to countries. This observation fails to take into further mobilize and innovate finance for scaling- account the framework for such Bank involve- up action on climate change, in cooperation with ment.1 Since the Bank is only one player among development partners and the private sector, many, the framework calls for identifying where and it also neglects the possible leveraging of there is a gap not being met by other agencies IBRD and International Finance Corporation and then filling the gap in areas where the Bank financing and risk management services through has the capability and a comparative advantage. blending with concessional donor funds. At the country level, country ownership and Management considers overly pessimistic the response to client demand is the primary report's conclusion with respect to future Bank principle of Bank involvement--that is, the Bank work on GPGs in MICs that ". . . the Bank has not works with clients on issues of global or regional been able to call on an attractive large-scale concern and supports their efforts at addressing funding program . . . to encourage comprehen- such issues. In sum, the appropriate role for the sive action on climate change." Given that the Bank at the country level is situation-specific to World Bank Group only recently began full- the GPG being supported and to the country fledged efforts to step up climate action to scale, context; hence, variations in the extent of Bank the conclusion appears premature. For example, involvement at the country level among GPGs the Bank works with donor countries to mobilize and among countries are to be expected. For some $5 billion in new and additional financing example, on communicable diseases, substantial for the proposed Clean Technology Fund, with funding is flowing from large vertical funds. Here the expectation that these new resources will the challenge is to ensure that the funding is leverage another $25­30 billion in financing for balanced by Bank support for health sector low-carbon investments. systems and other government priorities for the delivery of health services. While not presented Use of the country program model. Management is in CASs as support for a global public good, such aware that country-based work on GPGs faces support by the Bank is key for bridging the gap challenges in cases where global and country between global and country interests in the area interests diverge and there is no international of communicable diseases (that is, it provides framework for collective action, but it believes indirect support to the respective GPG). Regard- that such challenges can be addressed. Manage- ing climate change, pending a future global ment does not agree with the report's broad climate agreement, country ownership and an conclusion that relying on the country program effective country support program can be built model for the Bank's work on GPGs is a "double- only on demonstrated development edged sword." The Bank's ultimate clients are opportunities and advantages of a low-carbon, poor people. Using the country program model, climate-resilient strategy tailored to specific the Bank is better able to provide analysis that country circumstances. puts growth and poverty reduction at the core 115 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 and relates GPG challenges to this goal. ment welcomes the report's proposal to Importantly, using the country program model strengthen M&E quality in GPPs, though it notes also helps ensure country ownership of the that an M&E framework for Development Grant actions supported by the Bank. These considera- Facility-supported global programs is in place and tions argue for integrating the country-level the Consultative Group on International Agricul- activities of GPPs into country programs and ture Research program offers an example of good against allowing them to bypass country practice. Management appreciates IEG's insight programs. The example of Ethiopia's health into factors influencing the integration of global budget, which due to the proliferation of vertical program activities into country operations, health funds devotes some 50 percent of notably the strong positive effect of substantial resources to HIV/AIDS, is instructive. That the developing country representation in the govern- Bank can balance resource flows by targeting its ing body of a GPP in both enhancing legitimacy resources to other health needs in the country is and fostering stronger linkages with country a considerable strength of the country program operations. But management also believes that model. The key challenge, in management's the report should more clearly distinguish among view, is to more thoroughly integrate GPGs into subsets of GPPs (instead of treating GPPs as if they the diagnosis of countries' development are all interchangeably applicable at the country challenges and the dialogue with the govern- level or all deal with GPG issues) and should also ment as part of the CAS process. With this recognize that the Bank's role at the country level analysis in place, the appropriate contribution of is limited in GPPs where the Bank is not an global programs and trust funds as part of the implementing agency. Management agrees that CAS support program can then be determined, better integration of global funds at the country consistent with the plans by vice-presidential level is important. To that end, over the last two units for implementation of the Trust Fund years, the Bank has been working closely with the Management Framework. Development Assistance Committee of the Organisation for Economic Co-operation and Country and regional strategies. The report Development and with bilateral donors and provides valuable information on the treatment partner countries on an initiative to better align of GPGs in CASs, Regional strategies, sector global programs with country operations. This strategies, and Bankwide strategy documents. In initiative has contributed to preparations for the light of recent developments, however, manage- Accra High-Level Meeting. ment finds too categorical the general statement in the report that the Bank's attention to GPGs ". Strengthening country program support for GPGs. . . wanes as one moves down from corporate Management agrees with the report's sugges- strategies to sector or regional strategies, and tions for strengthening the Bank's country-based then down one level further to country strate- support for GPGs, notably through improved gies." For example, the 2008 Latin America and organizational arrangements to coordinate Caribbean Regional strategy update emphasizes global, regional, and national activities; better support for clients' efforts to address global delivery of the Bank's global knowledge to issues, and this priority is translated into specific country teams and improved deployment of actions planned in the recent round of CASs, network anchor experts; support for clients' including those for Brazil, Colombia, and Mexico. efforts to gain greater voice in shaping responses (In March 2008, the Board approved a $500 to global issues; and the exercise of greater million loan to Mexico to support implementa- selectivity in the Bank's engagement in global tion of its National Climate Change Action Plan.) programs. Management agrees on the desirabil- ity of strengthening incentives to deliver GPGs at Global programs. The report shows that there is the country level where that is appropriate, but room for improvement in deploying global and has reservations about the suggested option of regional programs at the country level. Manage- setting aside significant administrative funding at 116 APPENDIX E: MANAGEMENT COMMENTS the corporate level for allocation to high-priority advocate on climate change is more debat- GPG work at the country level. This could distort able" does not fully reflect this reality. incentives and encourage supply-driven · Management agrees that increased Bank ef- initiatives. A more effective approach--as already forts are merited to strengthen the voice and demonstrated in the Latin American and representation of developing countries in the Caribbean Region--is Regional management governance of global programs. For example, commitment and clear guidance to staff on the the participation of developing countries in need to be attuned to client demands for the design of the Climate Investment Funds, support for their GPG priorities. There also including the design of a governance struc- needs to be an ongoing dialogue among the ture with equal representation of donor and re- networks and Regions on staffing and resource cipient countries, has been critical to providing allocation, to address key corporate priorities legitimacy for these funds. This participation through country operations where there is is essential to set the stage at the global level demand at the country level. This would be for climate change mitigation programs at the strengthened by recognition in staff perform- country level. ance evaluations, as indicated in the report. Conclusion Advocacy on GPGs This year's ARDE has again provided a valuable Management appreciates IEG's thoughtful service by tracking development outcomes and analysis of the Bank's successful advocacy work improvements in M&E frameworks, a service on GPGs and its achievements in creating that helps the Bank improve its performance. innovative financing mechanisms. Management Management commends IEG for an insightful agrees with most of the assessments, though it review of the Bank's experience with fostering believes that the report could have provided global public goods through support for more recognition to the Bank's performance in countries' efforts and through constructive preserving biodiversity, with funding from the advocacy. In its lessons of experience, the report GEF and own Bank resources. Two comments appropriately focuses on ways to strengthen the are warranted. Bank's country-level support for GPGs. As detailed above, management generally agrees · The report acknowledges the Bank's many ac- with the report's suggestions, although it has tivities in support of the international climate reservations about the option of setting aside change agenda, including securing resources administrative funding at the corporate level for the GEF (the financial mechanism of the earmarked for country-level GPG work. But United Nations Framework Convention on Cli- management also stresses the need for clarity on mate Change), developing methodologies to the Bank's role in supporting a country's efforts put the Kyoto Protocol's Clean Development at addressing issues of global or regional Mechanism into action, launching carbon concern. Such support must build on the Bank's funds, and supporting the demonstration, core mandate to foster the country's growth and deployment, and transfer of low-carbon tech- poverty reduction goals, support measures that nologies and adaptation technologies. In this are owned by the country, and take into account, light, management believes that the report's for each GPG, the involvement of other agencies summary assessment that ". . . the extent to and partners and the Bank's capability and which the Bank has been a leading influential comparative advantage. 117 ENDNOTES Chapter 1 5. The analysis excludes the sector boards on gen- 1. Until this year, IEG produced an Annual Review der, global information and technology, poverty re- of Operations Evaluation (AROE), which included analy- duction, and social development because IEG has sis of the Bank's monitoring and evaluation systems, as evaluated very few projects managed by these sector well as an IEG self-evaluation. In consultation with boards. CODE, the AROE and ARDE have been combined this 6. Data for fiscal 2007 remains partial because only year into a single document that maintains the essen- three-quarters of the projects that closed in fiscal 2007 tial features of both. have been evaluated to date. 7. During implementation, the Bank manages a proj- Chapter 2 ect by means of an ongoing Implementation Status and 1. See, for instance, Wappenhans (1992) for an early Results report (ISR). Upon completion of a project, the critique of this "lending culture." Bank conducts a self-evaluation. This self-evaluation is 2. IEG's measure of outcome considers three factors; called an Implementation Completion and Results re- relevance, efficacy, and efficiency. Relevance measures port (ICR), which is then independently evaluated by the expected development impact of a project design IEG, leading to an Implementation Completion and by weighing the continuing relevance of a project's ob- Results review. jectives. Efficacy refers to the extent to which each ob- 8. Project outcomes have also declined when jective was achieved, or expected to be achieved. weighted by disbursements, from 90 percent of projects Efficiency measures the cost-effectiveness of a project, rated moderately satisfactory or better in fiscal 2006, to based mainly on sectorwide best practices and indica- 83 percent in fiscal 2007. tors where available. Combining these three factors, 9. IEG, in agreement with the Bank, introduced an overall outcome is rated on a 6-point scale, ranging explicit requirement for evidence to substantiate as- from highly satisfactory to highly unsatisfactory. sessments of performance in projects, which fully went 3. IEG's sustainability measure assesses the re- into effect at the beginning of fiscal 2007. silience to risk of net benefit flows over time by an- 10. CASCR reviews tend to assess a shorter time-frame swering the following questions: At the time of for the period of a single CAS, generally between 3 and evaluation, what is the resilience to risks affecting future 5 years. Country Assistance Evaluations (CAEs) assess net benefit flows? How sensitive is the intervention to Bank programs over a longer period, typically between changes in the operating environment? Will the inter- 6 and 10 years, and involve an in-depth mission to the vention continue to produce net benefits as long as in- country. This analysis draws on IEG reviews of 59 CASCRs tended, or even longer? How well will the intervention completed since fiscal 2004, and one CASCR completed weather shocks and changing circumstances? near the end of fiscal 2003. 4. Under the new harmonized evaluation criteria for project evaluations, approved in October 2005, proj- Chapter 3 ects would no longer be rated for their sustainability and 1. Prior to 2004, the Bank required log-frames in institutional development impact. Therefore, the fiscal their projects. The results framework places greater 2003­07 cohort includes about 50 percent of evaluated emphasis on the outcomes, differentiating between fiscal 2006 projects, and seven projects in fiscal 2007. the final outcomes (the project development objec- 119 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 tives) and intermediate outcomes, which can be used 9. Estimates are drawn from the Global Public Goods to monitor progress toward achieving the project de- Working Group. velopment objectives. The log-frame included inputs and 10. In-person and telephone interviews of 23 World outputs as well as outcomes. Bank operational managers, including country and 2. An M&E system with a "high" rating is expected sector directors. to strongly influence project performance, provide suf- 11. In-person and telephone interviews of 23 World ficient information to satisfactorily assess the stated Bank operational managers, including country and sec- project objectives, and contribute to testing the un- tor directors. derlying development model (or results chain). Con- 12. Information from Avian Flu Resources by World versely, a system rated "negligible" would have many Bank Operations Policy and Country Services (OPCS). weaknesses, have very little impact on the project or pro- 13. It should be noted that this estimate covers all gram, be insufficient to satisfactorily assess the stated Bank-supported trade projects, a significant share of project objectives, and contribute little to testing the de- which were directed at domestic trade issues, as opposed velopment model. to the global dimension of trade. 3. As of February 2008. 14. For shorthand, this section uses the term "global 4. BP 2.11, Country Assistance Strategies, was up- programs," and the points made largely refer also to re- dated in June 2005 to reflect the results-based CAS ap- gional programs, unless otherwise stated. proach. 15. Here the GPGs included fall under four of the 5. Data on Bank-supported evaluations were drawn Bank's five GPG areas, excluding creation and sharing from the Development Impact Evaluation database, of knowledge. May 13, 2008. 16. Findings from an IEG focus group conducted with eight Task Team Leaders from Global and Regional Part- Chapter 6 nership Programs directly involved with global public 1. In-person and telephone interviews of 23 World goods, April 2, 2008. Bank operational managers, including country and sec- tor directors. Chapter 7 2. The "successor" to the CAS--the Country Part- 1. Making Trade Work for Poor People (Stern 2002) nership Strategy (CPS)--is used in more recent cases. was considered a timely input to the Doha Trade Min- All references here to CASs, therefore, also cover CPSs. isterial, and Global Economic Prospects 2004: Realizing 3. In-person and telephone interviews of 23 World the Development Promise of the Doha Agenda (World Bank operational managers, including country and sec- Bank 2004c) was published ahead of the Cancun Trade tor directors. Ministerial. World Trade Organization members cited 4. Discussed by the Executive Board in May 2008. these sources as contributing to their understanding of 5. Such as the World Bank's sustainable develop- the issues and to the debate (IEG 2006a). ment strategy: Making Sustainable Commitments: An 2. The Deaton report (2006), which evaluated the Environment Strategy for the World Bank. 2001. Bank's research program from 1998 to 2005, noted that 6. With the exception of the East Asia and Pacific and "historically, the Bank has had a very active, vibrant and the Europe and Central Asia environment strategies, influential research program on international trade and which have a strong focus on GPGs. trade policy . . . [although] suspects that it has not 7. The Health, Nutrition, and Population strategy, al- been as influential in recent years as once was the case." though it mentions the promotion of GPGs, provides 3. For instance, the World Bank's position on trade very little detail on its GPG strategy. policy--including pressuring the United States and Eu- 8. Here GPGs are defined to include all work on the rope to reduce agricultural subsidies--was widely re- following Bank-defined themes: HIV/AIDS, other com- ported in the popular press, including nearly 50 municable diseases, biodiversity, climate change, water references in the Economist and the Financial Times resources management, other environment and natu- since January 2005. World Bank (2006a) discusses the ral resources management, international financial ar- effectiveness of these public interventions more fully. chitecture, regional integration, trade facilitation and 4. For example, an op-ed piece in the Financial market access, and other trade and integration. Times at the time of the Bali conference, late 2007. 120 ENDNOTES 5. Barrett (2006) shows that the incentives for co- ruary 23, 2003; and "2004 External Review of OED" on operation in R&D depend on the prospects of the tech- IEG's Web site at www.worldbank.org/ieg/intro. nologies embodying R&D being diffused. If new 3. The Bank's new framework for global public goods technologies are diffused to developing countries, rich (World Bank 2007d) makes explicit reference to IEG's countries will have greater incentive to finance R&D. evaluation of regional programs as the foundation for 6. Refer to the following on CGIAR's Web site at expanding the Bank's work in regional programs. www.cgiar.org: "Research & Impact," "CGIAR on Global 4. Good examples for a CASCR and CASCR Review Issues," "Global Climate Change: Can Agriculture Cope?"; are the Philippines CASCR from April 2005 and IEG's re- and the World Bank's Web site on "Climate Change Re- view from May 2005, and the Yemen CASCR from May search: Sustainable Rural and Urban Development" at 2006 and IEG's review from June 2006. These CASCRs http://go.worldbank.org/7Q6I1HUPZ0. won IEG 2006 and 2007 Good Practice Awards. 7. In-person and telephone interviews of 11 senior 5. Going forward, CASCR ratings will become a experts from international institutions, international tier 2 indicator in the IDA15 Results Measurement Sys- NGOs, partner countries, and the World Bank. tem. 8. This is in part because loans tend to be slower in 6. The award categories are Projects, Implementa- disbursing, and the program is still very new. tion Completion and Results reports, Country Pro- 9. In detail, those roles are: lender, founder, mem- grams, Country Assistance Strategy Completion Reports, ber of the governing body, convener, financial contrib- Monitoring and Evaluation, and Initiatives with Demon- utor, trust-fund trustee, house secretariat, implementing strated Impact/Results. agency, chair of the governing body, trust-fund manager, 7. Although 24 percent is not a high response rate, and cosponsor (IEG 2004b). a comparison of respondent and nonrespondent char- 10. Development Committee Communiqué, April 13, acteristics (grade level, region, headquarters/country of- 2008. fice) shows that both groups are very similar. The only dimension with a difference of more than 6 percent be- Appendix A tween respondents and nonrespondents is the share of 1. IEG's institutional development impact measure headquarters and country office staff. evaluates the extent to which an intervention improves 8. It should be noted that this question included a the ability of a country or region to make more efficient, "not applicable" answer option in 2007, but not in 2006. equitable, and sustainable use of its human, financial, The percentages presented in the text include these re- and natural resources. Such improvements can derive sponses in the denominator. from changes in values, customs, laws and regulations, 9. Previous note applies. and organizational mandates. 10. This compares with about seven reports about 2. The analysis excludes the sector boards on gen- IEG in the media each year between 1996 and 2004. The der, global information and technology, poverty re- IEG reports with the highest coverage were evaluation duction, and social development. of Bank assistance to middle-income countries, from 3. The five countries/areas that received Special Fi- 2007, and the evaluation of Bank assistance to fragile nancing Grants in the past decade are Bosnia and Herze- states, from 2006. For both reports, IEG undertook govina, Kosovo, Serbia, Timor-Leste, and the West Bank multiple media launches in Washington, D.C. and in the and Gaza. field. 4. Of this amount, $351 million financed operations 11. Level of adoption ratings are high--fully adopted; in the West Bank and Gaza. substantial--largely adopted but not fully incorporated into policy, strategy, or operations as yet; medium-- Appendix C adopted in some operational and policy work but not 1. Mandate of the Director-General, Evaluation. to a significant degree in key areas; and negligible--no 2. IEG's independence has been validated according evidence or plan for adoption, or plans and actions for to the standards of the OECD's Development Assis- adoption are in a very preliminary stage. tance Committee and the U.S. Government Accounta- 12. The status of recommendations is rated as: ac- bility Office (among others), as documented in two tive and remains actionable by management; complete papers: IEG, "OED Reach: Independence of OED," Feb- and archived in the e-MAR; obsolete or overtaken by 121 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 events and archived in e-MAR; difference of opinion be- and management. These did not receive an adoption rat- tween management and IEG. ing and are excluded from the sample when compar- 13 . Recommendations that were not accepted by ing levels of adoption. management will be excluded from the following data analysis. Appendix E 14. This excludes the four recommendations on 1. See World Bank 2007m. which there was a difference of opinion between IEG 122 BIBLIOGRAPHY Baer, Paul, Tom Athanasiou, and Sivan Kartha. 2007. OECD. http://cd4cdm.org/Publications/Overcoming The Right to Development in a Climate Constrained BarriersCDMprojects.pdf. World: The Greenhouse Development Rights Frame- GLCA (Global Leadership for Climate Action). 2007. work. Heinrich Böll Stiftung Publications on Ecology. Framework for a Post-2012 Agreement on Climate Berlin: Heinrich Böll Foundation. Change. http://www.unfoundation.org/files/pdf/2007/ Barrett, Scott. 2006. "Climate Treaties and `Breakthrough' GLCA_Framework2007.pdf. Technologies." American Economic Review: Papers House of Commons--International Development Com- and Proceedings 96 (2): 22­25. mittee. 2008. "DFID and the World Bank: Sixth Re- ------. 2007. Why Cooperate? The Incentive to Supply port of Session 2007­08," Vol. 1. London, UK: House Global Public Goods. New York: Oxford University of Commons. Press. IEG (Independent Evaluation Group). 2002. The World ------. 2008. "The World Bank's Role in Fostering Bank's Approach to Global Programs: An Inde- Global Public Goods: Climate Change and Essential pendent Evaluation. Phase 1, Report No. 26284- Medicines." ARDE background paper. Available on GLB. Washington, DC: World Bank. this evaluation's Web site at www.worldbank.org/ieg. ------. 2004a. Annual Report on Operations Evalua- Benedick, R.E. 1998. Ozone Diplomacy: New Direc- tion. IEG Study Series. Washington, DC: World Bank. tions in Safeguarding the Planet. Cambridge, MA: ------. 2004b. Addressing the Challenges of Global- Harvard University Press. ization: An Independent Evaluation of the World Brahmbhatt, Milan. 2005. "Avian and Human Pandemic Bank's Approach to Global Programs. IEG Study Influenza--Economic and Social Impacts." Speech Series. Washington, DC: World Bank. delivered during the "Global Partners Meeting on ------. 2004c. The CGIAR at 31--An Independent Avian Influenza and Human Pandemic Influenza," Meta-Evaluation of the Consultative Group on In- hosted by the World Health Organization, Geneva, ternational Agricultural Research. Washington, DC: Switzerland, November 7­9. World Bank. CAN-E (Climate Action Network Europe). 2002. "CAN ------. 2005. Committing to Results: Improving the Europe Comments on PCF Plantar Project." May 7. Effectiveness of HIV/AIDS Assistance. IEG Study Brussels: CAN-E. http://www.climnet.org/pubs/CAN Series. Washington, DC: World Bank. EuropePlantar.pdf. ------. 2006a. Assessing World Bank Support for Trade Carr, Christopher, and Flavia Rosembuj. 2007 "World (1987­2004): An IEG Evaluation. IEG Study Series. Bank Experiences in Contracting for Emission Re- Washington, DC: World Bank. ductions." Environmental Liability, No. 2. ------. 2006b. Hazards of Nature, Risks to Develop- Deaton, Angus, Abhijit Banerjee, Nora Lustig, and Ken ment: An IEG Evaluation of World Bank Assistance Rogoff. 2006. An Evaluation of World Bank Re- for Natural Disasters. Washington, DC: World Bank. search, 1998­2005. Washington, DC: World Bank. ------. 2006c. Annual Review of Development Effec- Ellis, Jane, and Sami Kamel. 2007. "Overcoming Barri- tiveness: Getting Results. Washington, DC: World ers to Clean Development Mechanism Projects." Bank. Organisation for Economic Co-operation and De- ------. 2006d. Annual Report on Operations Evalua- velopment and International Energy Agency. Paris: tion. IEG Study Series. Washington, DC: World Bank. 123 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 ------. 2006e. Engaging with Fragile States: An IEG Re- http://www.ips-dc.org/reports/#292. view of World Bank Support to Low-Income Coun- Rodrik, Dani. 2006. "Goodbye Washington Consensus, tries under Stress. Washington, DC: World Bank. Hello Washington Confusion? A Review of the World ------. 2007a. Development Results in Middle-Income Bank's Economic Growth in the 1990s: Learning Countries: An Evaluation of the World Bank's from a Decade of Reform." Journal of Economic Lit- Support. Washington, DC: World Bank. erature 44 (4): 973­87. ------. 2007b. Sourcebook for Evaluating Global and Schneider, Lambert. 2007. Is the CDM Fulfilling Its En- Regional Partnership Programs: Indicative Princi- vironmental and Sustainable Development Objec- ples and Standards. Washington, DC: World Bank. tives?: An Evaluation of the CDM and Options for ------. 2007c. The Development Potential of Regional Improvement. Report prepared for the World Wildlife Programs: An Evaluation of World Bank Support of Fund. November 5. Berlin: World Wildlife Fund. Multicountry Operations. Washington, DC: World http://www.wwf.org.uk/filelibrary/pdf/cdm_fill_objecti Bank. ves.pdf ------. 2007d. World Bank Assistance to Agriculture Smith, Richard, David Woodward, Arnab Acharya, and in Sub-Saharan Africa: An IEG Review. Washington, Robert Beaglehole. 2004. "Communicable Disease DC: World Bank. Control: A `Global Public Good' Perspective." Health ------. 2008a. Supporting Environmental Sustain- Policy and Planning 19 (5): 271­78, ability: An Evaluation of World Bank Group Expe- Stern, Nicholas 2002. "Making Trade Work for Poor Peo- rience, 1990­2007. IEG Study Series. Washington, ple." Speech delivered at the National Council of Ap- DC: World Bank. plied Economic Research, New Delhi, November ------. 2008b. Progress Report on IEG's Activities Re- 28. http://siteresources.worldbank.org/INTRES/ lating to Global and Regional Partnership Pro- Resources/stern_speech_makingtrworkforpoor_nov grams. CODE Subcommittee Discussion Paper. 2002.pdf Washington, DC: World Bank. ------. 2006. The Economics of Climate Change: The ------. 2008c. Public Sector Reform: What Works and Stern Review. Cabinet Office, Her Majesty's Treasury, Why?: An IEG Evaluation of World Bank Support. United Kingdom. Washington, DC: World Bank. Sweden Ministry for Foreign Affairs. 2001. Financing ------. 2008d. Using Training to Build Capacity for and Providing Global Public Goods: Expectations Development: An Evaluation of Project-Based and and Prospects. Study 2001:2. Prepared by Francisco WBI Training. IEG Study Series. Washington, DC: Sagasti and Keith Bezanson, on behalf of the Insti- World Bank. tute of ITF (International Task Force on Global Public Goods). Development Studies, Sussex, U.K.; Stockholm, 2006. Meeting Global Challenges: International Co- Sweden. operation in the National Interest. Final report. UNSIC (United Nations System Influenza Coordinator) Stockholm: ITF. and World Bank. 2007. Responses to Avian Influenza Kaul, Inge, Isabelle Grunberg, and Marc Stern, eds. 1999. and State of Pandemic Readiness: Third Global Global Public Goods: International Cooperation in Progress Report. New York: UNSIC-World Bank. the 21st Century. New York: Oxford University Press. Wappenhans, W.A. 1992. Report of the Portfolio Man- Müller, Benito, and Harald Winkler. 2008. "One Step For- agement Task Force. World Bank, Washington, DC. ward, Two Steps Back? The Governance of the World Wheeler. David. 2008. "Crossroads at Mmamabula: Will Bank Climate Investment Funds." Energy and Envi- the World Bank Choose the Clean Energy Path?" ronment Comment. February. Oxford, UK: Oxford Working Paper No. 140. Center for Global Institute for Energy Studies. http://www.oxfordenergy Development, Washington, DC. .org/pdfs/comment_0208-1.pdf. Woods, Ngaire. Forthcoming. "Governance Matters: The Osterholm, Michael. 2007. "Unprepared for a Pandemic IMF and Sub-Saharan Africa." In Finance, Develop- (H5N1 avian influenza virus)." Foreign Affairs 86 ment, and the IMF,ed.JamesBoughtonandDomenico (2): 47­57. Lombardi. Oxford, UK: Oxford University Press. Redman, Janet. 2008. World Bank: Climate Profiteer. Woods, Ngaire, and Domenico Lombardi. 2006. "Uneven Washington, DC: Institute for Policy Studies. Patterns of Governance: How Developing Countries 124 BIBLIOGRAPHY Are Represented at the IMF." Review of Interna- gagement with IBRD Partner Countries, Develop- tional Political Economy 13 (3): 480­515. ment Committee." Report No. DC2006-0014, World World Bank. 1997. "Toward Global Sustainability," James Bank, Washington, DC. D. Wolfensohn's speech to the United Nations Gen- ------. 2006b. "Avian and Human Influenza: Financing eral Assemply Special Session on the Environment, Needs and Gaps." Technical note prepared for the New York, June 25. International Pledging Conference on "Avian and ------. 2000. Report of the Working Group on Im- Human Influenza," in Beijing, China, January 17­18. proving Quality of Monitoring and Evaluation in ------. 2006c. "Avian and Human Influenza: Multi- Bank Financed Operations. Operations Policy and Donor Financing Framework." Technical note pre- Country Services Department. Washington, DC: pared for the International Pledging Conference on World Bank. "Avian and Human Influenza," in Beijing, China, Jan- ------. 2001. Making Sustainable Commitments: An uary 17­18. Environment Strategy for the World Bank. ------. 2006d. Forest Carbon Partnership Facility: A Washington, DC: World Bank. Framework for Piloting Activities to Reduce Emis- ------. 2003a. Country Assistance Strategies: Retro- sions from Deforestation and Forest Degradation. spective and Future Directions (CAS Retro III). Op- Washington, DC: World Bank. erations Policy and Country Services Department. ------. 2006e. Protecting and Improving the Global Washington, DC: World Bank. Commons: 15 Years of the World Bank Group ------. 2003b. Putting Our Commitments to Work: Global Environmental Facility Program. Washington, An Environment Strategy Implementation Progress DC: World Bank. Report. Washington, DC: World Bank. ------. 2007a. Quality of Supervision in FY05­06 ------. 2004a. "OP/BP 8.60, Development Policy Lend- (QSA7). Quality Assurance Group. Washington, DC: ing." Operations Policy and Country Services De- World Bank. partment. World Bank, Washington, DC. ------. 2007b. Focus on Results: The IDA14 Results ------. 2004b. Responding to Climate Change: Pro- Measurement System and Direction for IDA15. In- posed Action Plan for the World Bank in Latin ternational Development Association / Operations America. Washington, DC: World Bank. Policy and Country Services Department. ------. 2004c. Global Economic Prospects 2004: Re- Washington, DC: World Bank. alizing the Development Promise of the Doha ------. 2007c. Note from the President of the World Agenda. Washington, DC: World Bank. Bank, Robert Zoellick, for the 76th Meeting of the De- ------. 2005a. Results Focus in Country Assistance velopment Committee. October 19, World Bank, Strategies: A Stock Taking of Results-Based CASs. Op- Washington, DC. erations Policy and Country Services Department. ------. 2007d. Global Public Goods: A Framework for Washington, DC: World Bank. the Role of the World Bank. Global Programs and Part- ------. 2005b. Harmonized Evaluation Criteria for ICR nerships (GPP). Washington, DC: World Bank. and OED Evaluations. OPCS Guidelines. Opera- ------. 2007e. Meeting the Challenges of Global tions Policy and Country Services Department. Development: A Long-Term Strategic Exercise for the Washington, DC: World Bank. World Bank Group. Washington, DC: World Bank. ------. 2005c. Improving Health, Nutrition, and ------. 2007f. Sector Implementation Strategy: Third Population Outcomes in Sub-Saharan Africa: The Review. Operations Policy and Country Services De- Role of the World Bank. Washington, DC: World livery Management. Washington, DC: World Bank. Bank. ------. 2007g. Focus on Sustainability 2005/2006. ------. 2005d. "Program Framework Document for Washington, DC: World Bank. Proposed Loans/Credits/Grants in the Amount of ------. 2007h. Clean Energy for Development Invest- US$500 Million Equivalent for a Global Program for ment Framework: The World Bank Group Action Avian Influenza Control and Human Pandemic Pre- Plan. Washington, DC: World Bank. paredness and Response." Report No. 34386. ------. 2007i. A Brief History of the Prototype Carbon Washington, DC: World Bank. Fund: The PCF 1997--2007, Pioneers in a Decade ------. 2006a. "Strengthening the World Bank's En- of Change. Carbon Finance Unit. Washington, DC: 125 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 20 08 World Bank. ------. 2008a. Annual Report on Portfolio Performance ------. 2007j. Carbon Finance for Sustainable De- ­ 2007. Quality Assurance Group. Washington, DC: velopment: 2007. Washington, DC: World Bank. World Bank. ------. 2007k. Healthy Development: The World Bank ------.2008b.QualityatEntryinFY06­07(QEA8).Qual- Strategy for Health, Nutrition, and Population ity Assurance Group. Washington, DC: World Bank. Results. Washington, DC: World Bank. ------. 2008c. Additions to IDA Resources: Fifteenth Re- ------. 2007l. IDA and Climate Change: Making Cli- plenishment. IDA: The Platform for Achieving Results mate Action Work for Development. Washington, at the Country Level: Report from the Executive Di- DC: World Bank. rectors of the International Development Association ------. 2007m. Global Public Goods: A Framework for to the Board of Governors. Approved by the Exec- the Role of the World Bank. Report no. DC2007-0020. utive Directors of IDA, February 28. Washington, Background report for the Development Commit- DC: World Bank. tee, World Bank, Washington, DC. ------. 2008d. "Consultation Draft on Climate Invest- 126 THE WORLD BANK GROUP WORKING FOR A WORLD FREE OF POVERTY The World Bank Group consists of five institutions--the International Bank for Reconstruction and Development (IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), the Multilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of Investment Disputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their envi- ronment by providing resources, sharing knowledge, building capacity, and forging partnerships in the public and private sectors. THE INDEPENDENT EVALUATION GROUP ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Independent Evaluation Group (IEG) is an independent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (The World Bank) and IDA, IEG-IFC focuses on assessment of IFC's work toward private sector development, and IEG-MIGA evaluates the contributions of MIGA guarantee projects and services. IEG reports directly to the Bank's Board of Directors through the Director-General, Evaluation. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group's work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. ISBN 978-0-8213-7714-7 THEWORLD BANK SKU 17714