Results and Performance of the World Bank Group 2017 AN INDEPENDENT EVALUATION © 2018 International Bank for This work is a product of the staff of The RIGHTS AND PERMISSIONS Reconstruction and Development / The World Bank with external contributions. The material in this work is subject World Bank The findings, interpretations, and to copyright. Because The World 1818 H Street NW conclusions expressed in this work Bank encourages dissemination Washington, DC 20433 do not necessarily reflect the views of of its knowledge, this work may be Telephone: 202-473-1000 The World Bank, its Board of Executive reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they noncommercial purposes as long as full represent. attribution to this work is given. Attribution—Please cite the report as: World Bank. 2018. Results and The World Bank does not guarantee Any queries on rights and licenses, Performance of the World Bank Group the accuracy of the data included in including subsidiary rights, should be 2017. Independent Evaluation Group. this work. The boundaries, colors, addressed to Washington, DC: World Bank. denominations, and other information World Bank Publications, The World shown on any map in this work do not Bank Group, 1818 H Street NW, imply any judgment on the part of The Washington, DC 20433, USA; fax: Cover Photo: Olga Kashubin / World Bank concerning the legal status 202-522-2625; e-mail: pubrights@ Shutterstock.com. of any territo y or the endorsement or worldbank.org. acceptance of such boundaries. Results and Performance of the World Bank Group 2017 AN INDEPENDENT EVALUATION Careful observation and analysis of program data and the many issues impacting program efficacy reveal what works as well as what could work better. The knowledge gleaned is valuable to all who strive to ensure that World Bank goals are met and surpassed. contents abbreviations  vi acknowledgments  vii prefaceviii overview  ix management response  xix 1. Addressing Environmental Sustainability in World Bank Group Operations  1 Environmental Sustainability in the World Bank Group  3 Environmental Sustainability in the World Bank Group Portfolio  4 Environmental Sustainability in World Bank Group Country Strategies  11 Measuring Environmental Sustainability in the World Bank Group  13 Environmental and Social Risk Management  17 Results and Performance of the 2. Performance of World Bank Group Operations  28 World Bank Group 2015 Introduction  30 World Bank Project Ratings Trends  30 35 Factors Associated with Performance of World Bank Projects  AN INDEPENDENT EVALUATION Performance of IFC Investment Projects  39 Performance of IFC Advisory Projects  45 Performance of MIGA Projects  50 Performance of World Bank Group Country Strategies  52 3. Tracking Implementation through the Management Action Record  60 Trends in Implementation of IEG Recommendations and Management Action Plans  61 Have Action Plans Made a Difference?  65 bibliography  69 Box Careful observation and analysis of program data Box 2.1. The International Finance Corporation Is Implementing Several Initiatives to Improve and the many issues Development Results 46 program impacting efficacy reveals what Figures works as well as what could work better. The World Bank Project Activities and Financing Commitments With Potential Environmental Benefits Figure 1.1.  knowledge gleaned is Increased for Clean and Resilient but Decreased for Green Activities  valuable to all 6who strive Figure 1.2. A to ensure that World  ctivities With Potential Environmental Benefits Have Spread Beyond the Sustainable Development Bank goals are met and Practice Group  7 surpassed. Independent Evaluation Group | World Bank Group iii Figure 1.3. East Asia and Pacific Saw the Largest Increase in Environmental Support  8 FC’s Share of Projects with Potential Environmental Benefits Increased Slightly, Mostly in Figure 1.4. I Infrastructure and Manufacturing, Industry, and Agribusiness  10 Figure 1.5. MIGA Increased Its Share of Projects with Potential Environmental Benefits  10  orld Bank Group Country Strategies Show Consistency and No Clear Pattern in the Level of Figure 1.6. W Attention to Environment  11 Figure 1.7. The Type of Benefits Supported by Country Strategy Objectives Varies Over Time  12  orld Bank Project Outcome Ratings Were Essentially Stable by Number of Projects; By Volume, Figure 2.1. W Some Improvement Over Time Was Associated With A Few Large Projects  31 By Volume, Bank Performance Ratings for World Bank Projects Improved, Whereas, By Number of Figure 2.2.  Projects, Ratings Remained Stable Over Time  33 Outcome Ratings and World Bank Performance Ratings across IBRD/IDA, IPF/DPF, and FCV Status: Figure 2.3.  A Few Improvements Observed  34 FC Investment Projects with Mostly Successful or Better Development Outcome Ratings Figure 2.4. I Are Shrinking  40  ong-Term Trend of Environmental and Social Effects Ratings Remained Flat While Other Indicators Figure 2.5. L Trended Downward  41 Figure 2.6. Development Outcome Ratings Are Highly Associated with IFC’s Work Quality  43 FC Advisory Projects with Mostly Successful or Better Development Effectiveness Ratings Figure 2.7. I Are Shrinking  47  atings of IFC Advisory Development Effectiveness and Each Core Component Show Weak Results in Figure 2.8. R FY14–16  48 Figure 2.9. Development Effectiveness Ratings Are Highly Associated with IFC Work Quality  49 Figure 2.10. The Long-Term Trend of MIGA Development Outcome Ratings Has Not Changed  51 Ratings of Implementation of Recommendations by the Independent Evaluation Group and Figure 3.1.  Management of the World Bank Group Institutions (Cumulative)  62 Comparison of Recommendations Rated Substantial and Above and High and Complete by the Figure 3.2.  Independent Evaluation Group, by Category  63 Tables Table 1.1. Environmental Sustainability across Clean, Green, and Resilient areas  4 Table 1.2. Environmental Sustainability Indicators in the FY14–17 Corporate Scorecard  14 Table 1.3. Most Project Completion Reports Record Safeguard Triggers and Compliance, but Some Do Not  20 Table 2.1. Factors Associated with World Bank Project Performance  36 Table 3.1. Action Plan Items For Environment Evaluations Have Largely Been Outputs  66 Appendixes Appendix A. Environmental Sustainability in the World Bank Group Appendix B. Methodology for Environmental Sustainability Analysis Appendix C. Additional Findings on Environmental Sustainability Appendix D. Background on World Bank Project Performance Ratings iv Results and Performance of the World Bank Group 2017 | Contents Appendix E. Additional Information on the World Bank Portfolio Appendix F. Analysis of Distribution of World Bank Financing Volume Across Projects and Across Countries Appendix G. Risk and IEG Outcome Ratings Appendix H. Factors of Performance of World Bank Projects: Structured Review of Research Appendix I. Gender Incorporation in Project Development Objectives, Components, and Indicators Appendix J. Additional Information on the International Finance Corporation Portfolio Appendix K. Additional Findings: Management Action Record Tracking Data and Methodology Appendix L. Regional Results and Performance Appendix M. Independent Evaluation Group Note on Global Practice Performance Cross-Cutting Evaluation Findings Appendix N.  Independent Evaluation Group | World Bank Group v abbreviations AIMM Anticipated Impact Measurement and Monitoring CAS country assistance strategy CLR Completion and Learning Reports CODE Committee on Development Effectiveness CPF Country Partnership Frameworks DeCODE Delivery Challenges in Operations for Development Effectiveness DPF development policy financing E&S environmental and social ESF Environmental and Social Framework FCV fragility, conflict, and violence FCS fragile and conflict-affected situations FI financial intermediary FIG financial institutions group IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report IDA International Development Association IEG Independent Evaluation Group IFC International Finance Corporation INR infrastructure and natural resources IPF investment project financing LIC low-income country LMIC lower-middle-income country M&E monitoring and evaluation MAR Management Action Record MAS manufacturing, agribusiness, and services MIGA Multilateral Investment Guarantee Agency PDO project development objective RAP Results and Performance of the World Bank Group SIP Small Investment Program UMIC upper-middle-income country All dollar amounts are U.S. dollars unless otherwise indicated. vi Results and Performance of the World Bank Group 2017 | Abbreviations acknowledgments This review was conducted by a team led by Soniya Carvalho, Stephen Hutton, Aurora Medina Siy (co-task team leaders), and Nidhi Khattri (former co-task team leader), and was written together with Joy Behrens. The review was conducted under the guidance of Auguste Tano Kouame (director), Emanuela Di Gropello (manager), Marie Gaarder (former manager), and Rasmus Heltberg (acting manager), as well as the overall direction of Caroline Heider (Director-General, Evaluation). The team comprised Joy Behrens, April Connelly, Ebru Karamete, Gurkan Kuntasal, Bahar Salimova, Anis Dani, Jouni Eerikainen, Judith Lewetchou Efouefack, Mitko Grigorov, William Langley, Yunsun Li, Andres Liebenthal, Franz Loyola, Lauren Kelly, Raghavan Narayanan, Elaine Wee-Ling Ooi, Daniel Palazov, Xiaoxiao Peng, John Redwood, Maria Shkaratan, and Yoshine Uchimura. Additional analytic support was provided by Ana Belen Barbeito, Leonardo Bravo, Unurjargal Demberel, Hiroyuki Hatashima, and Ichiro Toda under the guidance of Beata Lenard. Brenda Barbour and Kristin Strohecker supervised the analyses related to the 2017 Management Action Record, which was led by Bahar Salimova, with technical support from Xiaoxiao Peng and William Langley. Kristin Strohecker supervised production of online resources, with technical contributions by Kyun Hee (Ken) Lee, Maria Gabriela Padrino, and Gastón Tchicourel. Anthony Tyrrell coordinated the Regional and Global Practice updates, with inputs from staff across all Independent Evaluation Group departments and technical support from Xiaoxiao Peng and William Langley. The Independent Evaluation Group would also like to thank Judith Lewetchou Efouefack, Lauren Kelly, and Gurkan Kuntasal for timely and insightful work on assessing World Bank Group Performance Standards and safeguards in projects that closed in fiscal year 2016, and Judith Hahn Gaubatz for supporting analysis related to the reasons behind the high and low ratings of quality of supervision. This report benefited from peer review comments from Ali Khadr (consultant and former Independent Evaluation Group director), Kwame Boakye-Agyei (Director of Health, Social, Environmental and Security at Kosmos Energy), and Olav Kjørven (chief strategy officer of the EAT Foundation and former director of public partnerships at the United Nations Children’s Fund). Marie Charles, Gaby Loibl, and Yezena Yimer provided administrative support. William Hurlbut and Amanda O’Brien edited the report. Information technology support, training, and troubleshooting were provided by Yang Soo (Tom) Yoon and Mohit (Sunny) Sharma. Independent Evaluation Group | World Bank Group vii preface The Results and Performance of the World Bank Group (RAP) report is an annual review by the Independent Evaluation Group (IEG) of the development effectiveness of the World Bank Group. The RAP covers the World Bank (International Bank for Reconstruction and Development and International Development Association), the International Finance Corporation, and the Multilateral Investment Guarantee Agency (four of the five institutions of the World Bank Group). The report synthesizes existing evidence from IEG evaluations, validations, and engagements, complemented by analysis of relevant information from other sources (for example, World Bank Group documents). Results and performance are assessed separately for each of the institutions because of their differing time frames, operating models, and clients. Beginning with RAP 2013, one chapter of each year’s RAP has focused on a special theme. The theme this year is environmental sustainability, focusing on selected questions related to the contribution of the World Bank Group to environmental sustainability (chapter 1). This RAP (i) assesses the extent to which the World Bank Group portfolio includes activities that support environmental sustainability (and changes over a 10-year period), (ii) takes stock of how the World Bank Group is measuring its contribution to environmental sustainability, and (iii) presents analysis on the World Bank Group’s systems for managing environmental and social risks. The RAP does not seek to provide an overall assessment of the environmental effects of the World Bank Group’s projects because data are available only for the subset of projects where environmental goals were the primary objective. As in previous years, this RAP reports on the results and performance of the World Bank Group’s country- and project-level work as captured in recent IEG validation reports, evaluations, and other analytic work, including on projects with environment-related objectives, as well as the environmental and social effects element of International Finance Corporation and Multilateral Investment Guarantee Agency development outcome ratings (chapter 2). Reports on specific Regions and Global Practices and summaries of relevant IEG findings can be found in appendixes L, M, and N. Also continuing a past practice, this RAP reviews the degree to which recommendations based on IEG’s evaluations have been implemented (chapter 3). This RAP also presents a “deep dive” on implementation of recommendations related to the environmental sustainability theme. The RAP is a review, not a major IEG evaluation, and it does not collect new evaluative data. As such, it does not attempt to provide an all-encompassing evaluative treatment. It can, however, identify evaluative gaps for future analysis (see appendix B). viii Overview Results and Performance of the World Bank Group 2017 | Preface highlights 1 Corporate Results and Performance The World Bank exceeded its corporate target for project outcome ratings by volume but not by number of projects. For the World Bank Overview performance rating one element, quality of supervision, improved over time and exceeded the corporate target, whereas the other, quality at entry, did not meet the corporate target. Some evidence suggests that the performance of large projects masked ratings patterns that characterized the vast majority of relatively smaller projects. International Finance Corporation (IFC) Investments and Advisory Services projects’ outcome ratings declined. Multilateral Investment Guarantee Agency (MIGA) projects’ development outcome ratings remained flat. Many factors of performance for the World Bank, IFC, and MIGA are fully within each institution’s control. Therefore, there is an opportunity to address these factors and aim for greater development effectiveness. 2 Addressing Environmental Sustainability in World Bank Group Projects The share of World Bank project components and IFC projects with potential environmental benefits increased by four percentage points over the past decade. Is this sufficient, given Independent Evaluation Group | World Bank Group ix the threats to development results from climate change, pollution, and environmental degradation? Most of the increase has been for climate change mitigation and adaptation, for which there have been specific targets and mandates. However, support has declined in some areas, such as biodiversity conservation and water resource management. 3 Implementation of Evaluation Recommendations Analysis of recommendations in the 2017 Management Action Record produced by the Independent Evaluation Group (IEG) review finds that Management Action Plans are not always aligned with the intent of IEG’s recommendations. This finding also applies to this Results and Performance of the World Bank Group report’s special topic, environmental sustainability. Consequently, the risk exists that IEG recommendations remain underaddressed even when Management Action Plans are fully implemented. x Results and Performance of the World Bank Group 2017 | Overview Addressing Environmental Sustainability in World Bank Group Operations T H E N OT I O N O F S U S TA I N A B I L I T Y of development progress, particularly environmental sustainability, is a central tenet of the World Bank Group’s strategy. In its 2013 strategy, the World Bank Group committed to reaching the twin goals—to eradicate extreme poverty and promote shared prosperity—in a sustainable manner. The strategy defined sustainability as a three-pillar paradigm: fiscal, social, and environmental. The 2016 “Forward Look: A Vision for the World Bank Group in 2030” confirmed the importance of the inclusion of environmental sustainability in two of the three ways the World Bank Group seeks to achieve the twin goals: accelerating inclusive and sustainable economic growth and fostering resilience to global shocks and threats. The “Forward Look” also noted that pressures on environmental sustainability are increasing, giving this agenda a sense of urgency. Under the 2016 Climate Change Action Plan, the World Bank Group announced targets for increasing climate finance. The World Bank environmental sustainability emphasis is also aligned with the global agenda. Given its importance, environmental sustainability is the special topic of this year’s report on the results and performance of the World Bank Group. As in previous years, this report also presents an assessment of World Bank Group operational performance, drawing on Independent Evaluation Group (IEG) evaluations. Chapter 1 explores selected topics on environmental sustainability, examining how the World Bank Group portfolio incorporates potential environmental benefits, assessing the indicators that measure environmental results, and reviewing systems for addressing potential negative environmental effects. Chapter 2 assesses the results and performance of the respective portfolios of the World Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA), and identifies factors associated with performance ratings. The chapter also highlights some environmental sustainability aspects of the portfolios’ performance. Chapter 3 presents an assessment of the World Bank Group’s implementation of recommendations from recent IEG evaluations as well as providing a long-term view on progress of implementing IEG’s recommendations related to environmental sustainability over the past 10 years. The World Bank Group has sought to mainstream environmental goals across its institutions. The environment and climate change have been part of high-level strategies for the World Bank Group institutions for at least 15 years. Since the adoption of the World Bank’s first environmental strategy in 2002, efforts have sought to mainstream environmental goals in projects and country strategies. But to what extent have these strategies led to changes in the institutions’ projects? The World Bank Group increased its support for environmental sustainability in projects across the  four institutions from FY08–10 to FY15–17 (based on large samples): Independent Evaluation Group | World Bank Group xi „„ For the World Bank, the share of project components with potential environmental benefits increased from 33 percent to 37 percent (with the share of commitments for those components increasing from 31 percent to 43 percent). „„ For the IFC, the share of investment projects with potential environmental benefits from financed activities increased from 31 percent to 35 percent. „„For  the MIGA, the share of political risk guarantee projects with potential environmental benefits increased from a low base of 8 percent to 36 percent. An important question would be whether these increases are sufficient, given the scale of pollution, environmental degradation, and climate change. Because the RAP is a review and not an evaluation, it does not address this question but flags it for attention by the management of the World Bank Group institutions. This increased focus on environmental sustainability comes primarily from climate change adaptation and mitigation (where there have been corporate targets and tracking). However, support has declined in some areas, such as biodiversity conservation and water resource management. The World Bank Group has defined environmental sustainability in terms of Clean (climate change mitigation, pollution abatement), Green (natural resource management and biodiversity), and Resilient (climate change adaptation and climate disaster risk management). For the World Bank, Resilient project components saw substantial increases and Clean project components saw moderate increases, a trend driven largely by climate change mitigation. At the same time, support for Green project components decreased, including in some traditional areas of environmental sustainability, particularly biodiversity conservation, water resource management, and irrigation efficiency improvements. These declines have been larger than increases in support for other Green activities, such as forest and fisheries management. Energy efficiency support has also declined. A limited analysis suggests similar patterns for the shares of World Bank analytical and advisory activities with potential environmental benefits, though the absolute number has increased substantially, in line with the overall increase in World Bank analytical and advisory work. IFC investment project support for environmental sustainability remains focused on Clean activities and specific sectors. Key activities have been for climate change mitigation—largely renewable energy and energy efficiency—and water pollution abatement through wastewater treatment. IFC support for environmental sustainability remains concentrated in the infrastructure sector and the manufacturing, agribusiness, and industry sectors. World Bank project support for environmental sustainability has been more common in higher-income countries. The share of support for environmental sustainability is largest for upper-middle-income countries, followed by lower-middle-income countries, and finally low-income countries. This gap has widened further over time, though support for climate change mitigation has become more even across country income groups. Environmental activities in projects are concentrated in the 10 World Bank clients with the largest commitment volumes; collectively, projects in these countries include roughly one-third of all project components but comprise half of components with potential environmental benefits. The Regions with the largest shares of support for environmental sustainability xii Results and Performance of the World Bank Group 2017 | Overview have been Latin America and the Caribbean (FY08–10) and East Asia and Pacific (FY15–17), while Sub-Saharan Africa has consistently seen the lowest share. The types of environmental activities supported in country strategies have largely been in line with the priorities of those Regions. World Bank Group country strategies show consistent attention to environment, with no clear trend. World Bank objectives with potential environmental benefits have been consistently present in country strategies over FY08–17, with 74 percent of strategies including at least one environmental objective or outcome, and the median share of environmental objectives varying between 11 percent to 17 percent across the 10-year period. The World Bank Group’s corporate, country, and project results systems provide useful data on environmental sustainability for interventions with environmental goals, with some limitations. Country strategy indicators on environmental sustainability are generally of adequate quality but provide an incomplete picture of country-level results because they do not capture all major activities and are adequate to assess achievements for only 77 percent of objectives or outcomes. Project indicators on environmental sustainability are broadly adequate for Clean and Resilient projects, but because of indicator design issues, they are often not adequate to demonstrate the effectiveness of Green projects or assess capacity building across all sectors. Several aspects of the World Bank Group environmental results indicators could benefit from improvement. Pressures to prioritize indicators that can feed into corporate indicators have sometimes led teams to deprioritize other potentially useful indicators and thereby weakened measurement systems for assessing project-level results. Evidence gaps on environmental outcomes are a major cause of lower outcome ratings for World Bank projects with environmental objectives, along with a lack of economic analysis for these projects. Existing measurement systems for environmental results can be improved through greater use of intermediate outcome indicators and by building quality standards into indicators. Knowledge gaps may require results assessment outside of the project cycle. Existing systems are limited by what can be observed at the point of evaluation and are often not able to capture longer-term results. The World Bank Group has improved its systems for managing environmental risks. For example, the World Bank strengthened provisions for climate change, expanded the scope of social safeguards, increased attention to supervision, and engaged in stronger stakeholder consultation throughout the project life cycle. IFC improved policy relating to climate change, supply chains, and financial intermediaries. However, implementation improvements are needed on the measurement of greenhouse gas emissions, supervision of financial intermediary clients, and use of advisory services for environmental management. Some new disparities across institutions could be harmonized. Monitoring, reporting, and disclosure systems could be improved. For the World Bank, guidance and tools for monitoring and reporting systems are not yet in place for the Environmental and Social Framework (which becomes operational in FY19) and so have not been assessed by IEG, and reporting on compliance for existing safeguards is not always clear. For IFC, disclosure of project-level environmental and social information from monitoring and supervision reports during implementation is still inadequate, and third-party monitoring could be used more widely. Independent Evaluation Group | World Bank Group xiii Most IFC projects (81 percent) and MIGA projects (82 percent) have positive ratings for environment- related Performance Standards at the time of evaluation. Both IFC and MIGA showed improvements in performance between appraisal or underwriting and evaluation. For both institutions, compliance with environmental Performance Standards has been highest for biodiversity conservation, followed by pollution prevention and then assessment and management of risks. For IFC, patterns in these ratings depend on project characteristics, with no clear trend over time. Corporate Results and Performance World Bank Outcome ratings for World Bank projects closed in FY14–16 were above the corporate target by volume but below it by number of projects. By number of projects, ratings were 73 percent moderately satisfactory or above compared with the target of 75 percent. By volume, outcome ratings were 84 percent compared with the target of 80 percent. One of the elements of the World Bank performance rating, quality of supervision, met the corporate target of 80 percent by volume, while the other, quality at entry, did not. By number of projects, World Bank performance ratings for FY14–16 stood at 72 percent. Comparing FY11–13 with FY14–16, the performance of the World Bank’s project portfolio improved slightly, and relatively higher ratings by volume are associated with a handful of very large projects. By number of projects, outcome and World Bank performance ratings (percentage moderately satisfactory or above) remained flat for from FY11–13 to FY14–16. By volume, ratings (percentage moderately satisfactory or above) increased for both outcome (from 79 percent to 84 percent) and World Bank performance (from 80 to 84 percent). However, simply excluding a tiny subset of large projects lowered these ratings in both periods. This subset of large projects consisted of 28 projects, each with a commitment volume of $800 million or above, which accounted for only 2 percent of projects and 21 percent of project volume for FY11–16. Thus, projects with a very large volume masked rating patterns that characterized the vast majority of projects with a relatively smaller volume (these accounted for 98 percent of projects and 79 percent of volume). Outcome ratings have improved in country contexts traditionally seen as challenging, although World Bank performance ratings fell short in these settings compared with other country contexts. For example, in FY11–13 and FY14–16, outcome ratings for projects in countries affected by fragility, conflict, and violence (FCV) went from 62 percent to 66 percent by number of projects and from 65 percent to 76 percent by volume. For the World Bank performance rating, only 75 percent of volume in FCV countries received a rating of moderately satisfactory or above during FY14–16, compared with 85 percent of volume in non-FCV countries. Among International Development Association (IDA) projects in FY11–13 and FY14–16, outcome ratings went from 67 percent to 70 percent by number of projects and from 70 percent to 80 percent by volume. Some improvements in ratings for investment project financing (IPF) projects were seen, particularly by volume. xiv Results and Performance of the World Bank Group 2017 | Overview Ratings within Regions varied over time. Comparing ratings within each Region between FY11–13 and FY14–16 indicates improvements in the Africa and East Asia and Pacific Regions on ratings for outcome and World Bank performance, both by number of projects and volume of commitments. Ratings within each Global Practice also varied over time. By volume, most Global Practices improved on outcome ratings and World Bank performance ratings between the two periods. However, four Global Practices did not improve on outcome ratings and four did not improve on World Bank performance ratings. Multiple factors are associated with the performance of World Bank projects. A structured review of sources both inside and outside the World Bank found that commonly identified project-level success factors included supervision quality and the ratings record of projects managed by the task team leader. Incorporation of lessons learned and analytical work, the quality of the project’s results chain and project design also factored in. An analysis of Implementation Completion and Results Report Reviews of projects with very high or very low World Bank performance ratings found that specific factors associated with low quality of supervision ratings were poor supervision reporting, lack of candor in supervision ratings, and lack of timeliness or effectiveness in response to project restructuring opportunities. The factors negatively associated with the performance of World Bank projects are broadly consistent with those identified in previous RAPs. Over the years, the World Bank has responded to these problems, but several remain underaddressed, leading to recent initiatives, such as the Agile World Bank, Implementation Completion and Results Report Reform, the Multiphase Programmatic Approach, the Global Delivery Initiative (including DeCODE), the new restructuring policy, and knowledge package delivery to task teams. Determining the extent to which these recent initiatives succeed in addressing the problems will require close monitoring and appropriate follow-up by management. Economic analysis can help the World Bank define outcomes, incentivize achievement, and allocate scarce resources efficiently. This RAP finds that among the most important factors associated with outcome ratings for World Bank projects with environmental objectives were the measurement of outcomes and the presence of economic analysis. There has been an evolution in the World Bank’s guidance to task managers on discount rates over 1998–2016, in particular, toward greater leeway to task managers in determining discount rates. This evolution raises the question of how the World Bank is ensuring coherence and consistency in (i) the discount rates, (ii) the methodologies for determining the discount rates, and (iii) the selection of time periods used in economic analysis for similar types of projects in the same sector and country. Clearer guidance and strong oversight are needed if the management of the World Bank is to ensure that the methodology, discount rates, and time periods used for economic analysis are not arbitrarily selected by project teams. IFC Development outcome ratings for IFC investment projects continued a downward trend that started in calendar year (CY)08. Only half the projects evaluated during CY14–16 were rated mostly successful or better. This is a sharp decline from CY11–13, when 60 percent of the projects achieved mostly successful or better development outcomes. The decline is also observed for industry groups, Independent Evaluation Group | World Bank Group xv projects in IDA and non-IDA countries, and in FCV and non-FCV countries. Among the four indicators of development outcome, only the percentage of satisfactory or better ratings for environmental and social effects did not decline. Projects in the South Asia Region and those in the infrastructure and natural resources industry cluster did better than average in the recent three-year period. Projects in IDA countries had better outcome ratings by both number and volume than projects in non-IDA countries, but the difference has narrowed. By Region and by number and volume, more projects in South Asia were rated mostly successful or better for development outcome in CY14–16. By industry group and both by number and volume, infrastructure and natural resources projects (mostly in the power sector) achieved better development outcome ratings compared with projects in other industry groups. For IFC Advisory Services, the decline in development effectiveness rating was even sharper than for IFC Investment Services projects. Projects rated mostly successful or better for development effectiveness fell from 63 percent in FY11–13 to 49 percent in FY14–16. This is the lowest positive development effectiveness rating registered since IEG first started reviewing IFC Advisory Services projects in 2008. Ratings for outcome achievement, strategic relevance, and efficiency indicators of development effectiveness have declined. In FY14–16, advisory projects in Africa and in Latin America and the Caribbean had the lowest share of positive development effectiveness ratings by number of projects and volume, respectively. The period of the steep decline in ratings seems to coincide with the restructuring of IFC Advisory Services in 2014. IEG began rating IFC work quality in FY15, so historical comparisons are not possible. However, results for FY15 and FY16 show strong correlation between work quality and development effectiveness ratings. The factors affecting development outcome for IFC investment projects are known, which provides an opportunity to identify actions that can lead to performance improvement over the medium term. IEG and IFC jointly reviewed the root causes of the declining development outcome results for both IFC Investment Services projects and IFC Advisory Services projects. IFC’s work quality, particularly screening, appraisal, and structuring and IFC’s role and contribution, continued to exert significant influence on development outcome ratings. Improvement in country risk, management quality, and profit margins are the external factors that significantly influence development outcome ratings; however, their effect is smaller compared with IFC’s screening, appraisal, and supervision and its role and contribution. Greenfield projects and equity-only investments have significant but negative effects on development outcome ratings. IFC has begun implementing initiatives to address the declining development outcome in investment projects, but discussions with IEG on the root causes of development effectiveness and work quality for advisory services are ongoing. Implementation results will become evident only with the evaluation of projects approved in FY18 and onward. MIGA MIGA projects’ development outcome ratings remained constant during the FY11–16 review period. The energy and extractives sector was the best performing while the financial and capital markets sector had the lowest share of positive outcome ratings. Projects in IDA countries had a higher share of satisfactory or better outcome ratings. Although outcome ratings of projects in FCV countries were xvi Results and Performance of the World Bank Group 2017 | Overview mostly positive, results need to be interpreted carefully because of the small number of evaluated projects in FCV compared with non-FCV countries. More projects were rated satisfactory or better for their environmental and social effects (63 percent) than for the other three development outcome indicators (business performance, economic sustainability, and contribution to private sector development). Measured against MIGA’s underwriting policies and procedures, ratings for MIGA’s appraisal, underwriting, and monitoring improved slightly, but more can be done to strengthen this aspect, especially concerning assessment of risks to development outcome, better documentation, and regular data collection of project indicators after the contract of guarantee becomes effective. The lack of information on project results has been a recurrent challenge that constrains MIGA’s performance assessment. Without solid results data, the development outcome of MIGA’s new products is difficult to ascertain. World Bank Group Country Strategy Performance World Bank Group country strategy performance remains below the corporate target. Development outcome ratings (from Completion and learning reviews) remained at 68 percent moderately satisfactory or above for FY14–17, against the corporate target of 70 percent. The World Bank Group performance ratings were 59 percent good or above. Implementation of Evaluation Recommendations: Management Action Record Update Although progress on implementing recommendations in their fourth year was rated substantial or above (substantial, high, or complete) for over 90 percent of recommendations, only 52 percent were implemented to a great degree (high or complete), and only 20 percent were fully implemented (complete), based on IEG ratings. The over 90 percent rated substantial or above compares with 72 percent in RAP 2016 and 80 percent in RAP 2015. In 2017, IEG tracked implementation of actions for 146 recommendations across the World Bank Group drawn from the 23 evaluations produced between FY13 and FY16. Implementation updates have been complicated because management sometimes retires recommendations earlier than full implementation of their action plans as assessed by IEG. Implementation updates have also been complicated by the increased number of recommendations that received only partial agreement from management, which introduces ambiguity into IEG’s monitoring of the follow-up. A reform of the Management Action Record (MAR) process in 2011 resulted in the joint agreement between IEG and management to introduce Management Action Plans. This change led IEG to monitor progress against Management Action Plans rather than IEG’s recommendations. This change was meant to reduce ambiguity about what management committed to implement, but may have shifted management’s focus away from action on IEG recommendations. For the 2017 MAR update, IEG action plan ratings for recommendations on implementing projects and programs were lower than for other types of recommendations. Ratings for recommendations on knowledge work, measurement systems, or assessment of the impact of World Bank Group interventions were also low. Independent Evaluation Group | World Bank Group xvii Over the past decade, roughly half of IEG’s recommendations on environmental sustainability have been implemented to at least a high degree. The World Bank Group has made progress across a range of recommendations on environmental mainstreaming, climate change mitigation and adaptation, natural resource management, and environmental and social risk management, but some gaps remain, such as on increasing World Bank support for energy efficiency, addressing issues of long-term climate change adaptation (in addition to short- or medium-term adaptation deficits) in policy dialogue, and monitoring of forest management projects. Recommendations relating to knowledge work, measurement systems, or assessment of the impact of World Bank Group interventions have been implemented far less than other recommendations. Although the introduction of action plans has led to better tracking of implementation outputs, analysis of recommendations in IEG’s 2017 MAR review finds that not all action plans are aligned with the intent of IEG recommendations. For instance, a deep dive into the IEG evaluations focusing on environment showed that action plan items are often unambitious and insufficient to achieve the outcome envisioned in the recommendations. Action plans are not discussed by members of the Committee on Development Effectiveness or the Board of Executive Directors, nor is IEG acceptance of action plans required. Since the current process requires IEG to rate progress only against action plans, not against IEG recommendations, the risk exists that even fully implemented Management Action Plans leave IEG recommendations underaddressed. IEG recommendations with which management partially agrees have not been adequately discussed. In such cases, the monitoring of the adoption of recommendations is hampered. With partial agreement, management may decide to create action plans that do not align with the outcomes envisioned in the recommendations. To realize the full potential of the MAR as an instrument for improving the World Bank Group’s development effectiveness, two types of changes are required. The first is to improve the engagement process between World Bank Group management and IEG; the second to improve the implementation of IEG recommendations by World Bank Group management. To improve engagement around major evaluations, World Bank Group management and IEG are undertaking seven pilots. Two of the seven are for adaptable action plans, designed to allow management flexibility in adjusting action plans during the implementation cycle. The remaining five pilots are designed to improve engagement between IEG and management by soliciting early feedback on IEG’s approach papers, development of recommendations, and design of Management Action Plans. The early assessment of these pilots is currently underway. If the experience with the pilots is well assessed and the right lessons drawn, these pilots can help bring about improvements in the engagement process. To improve the implementation of IEG recommendations requires greater attention on the part of World Bank Group management to the evaluation recommendations and more action to follow up on them. The fact that only 52 percent of the actions that management identified for itself were implemented to a high or complete extent in the fourth year of implementation (with 20 percent complete) may point to the lack of an organizational environment in which knowledge from evaluation informs and underpins the World Bank’s operations as a matter of course. xviii Results and Performance of the World Bank Group 2017 | Overview management response Introduction T H E M A N AG E M E N T O F T H E WO R L D B A N K G R O U P institutions welcomes the Independent Evaluation Group (IEG) report, Results and Performance of the World Bank Group 2017 (RAP 2017), and the opportunity to respond with comments. The report presents useful analysis and insights on the Bank Group’s status and progress in three areas: (i) how environmental sustainability is addressed in Bank Group operations; (ii) the recent performance of Bank Group operations; and (iii) the implementation status of the Management Action Record (MAR). Management finds that this year’s thematic focus, environmental sustainability, is timely and relevant in the context of the Bank Group’s 2013 strategy and the 2016 “Forward Look” as the global development community works to achieve the Sustainable Development Goals. Management is pleased that the report recognizes recent positive trends in the results and performance of Bank Group operations, while noting the challenges for further improvements. World Bank Management Comments Chapter 1: Addressing Environmental Sustainability in World Bank Group Operations The analysis starts with the design of a clear and intuitive (albeit ex post) framework to identify project components (or development policy financing [DPF] policy areas) with potential environmental sustainability benefits. Applying this framework leads to the conclusions that the share of World Bank project components (or DPF policy areas) with potential environmental benefits has increased from 33 percent in fiscal year (FY)08–10 to 37 percent in FY15–17 (with the share of commitments increasing from 31 percent to 43 percent). Limitations of the methodology. Management notes that (i) IEG devised its own typology of “Clean, Green, and Resilient” operations; (ii) the mapping exercise was done manually (that is, judgment was applied to classify each operation); (iii) a project component (or a DPF policy area) could actually have “double” benefits across the Clean, Green, and Resilient categories and could be mapped to multiple categories; (iv) sample sizes are small for some thematic areas for some periods, such that trend analysis at the subtheme level is not meaningful; and (v) operations were not mapped to Global Practices (because of the reorganization between the two time periods reviewed). Although the report provides a rich analysis and some insights for Bank Group support to environmental sustainability, a number of limitations in the methodology are critical. The most important is the focus on investment project financing, whereas Bank Group environment-related engagement involves the following types of activities, which were excluded: Technical assistance activities. The report did a quick assessment of the number of total Advisory Services and Analytics with potential environmental benefits and found that they had increased by about 60 percent. However, there is no attempt to assess the area or scale of technical assistance activities, which are important to strengthening environmental management and to planning related to climate change in client countries. Independent Evaluation Group | World Bank Group xix All activities coded “carbon finance.” Although these activities will be the subject of a separate IEG report, their omission leads to an incomplete picture of environment-related activities in the report. Recipient-executed trust fund activities under $5 million—except in the case of Global Environment Fund–funded activities, for which only medium-sized projects (typically under $1 million) are excluded. The Bank Group’s convening role, which is the subject of another future IEG evaluation but deserves mentioning (for example, Carbon Pricing Leadership Coalition; promotion of regional power sharing; and nonlending transboundary water management initiatives for the Nile, Ganges, and Brahmaputra Rivers, among others). Monitoring indicators. The report makes useful observations on the fact that no single set of indicators can serve the separate needs of the Corporate Scorecard, corporate standard indicators, climate screening indicators, and operation-specific indicators. Management agrees that, while significant progress has been made on the Scorecard and climate screening, improving operation- specific indicators remains a priority to enhance the effectiveness of monitoring and assessments of operations’ outcomes, and impact evaluation. The report also rightly points out that developing outcome-level indicators for Green operations is more challenging, mainly because of the longer time frame required to achieve operations’ impacts. Management would like to clarify that the use of Corporate Results Indicators (CRIs) is required only when the task team considers them relevant for project monitoring purposes. They are meant to be used in addition to, not at the expense of, other key indicators. The guidance for the new CRIs is clear on how and when the indicators should be used. Moreover, there is no formal restriction on the number of indicators at the project development objective level, and some of the CRIs are outputs better suited to the intermediate results level. Therefore, management finds it misleading for the report to conclude, based on some anecdotal cases, that “pressure to include corporate indicators in projects can sometimes weaken project results frameworks,” especially in “demonstrating project outcomes.” Environmental and social risks in DPF. The report states that the management of environmental and social (E&S) risks DPF could be improved, drawing on IEG’s 2015 learning product, Managing Environmental and Social Risks in Development Policy Financing. Management continues to question the methodology and criteria IEG applied in the 2015 report in defining potential E&S risks because they are not consistent with corporate guidance and the findings of the 2015 Development Policy Financing Retrospective: Results and Sustainability. Management further wishes to reiterate that all DPFs go through two corporate reviews, one at the concept and one at the decision stage, and both reviews consider environmental effects, social impacts, and risks. Management would also like to mention that staff training is available on DPFs on environmental, social, and climate change issues, most notably through the DPF Academies. Chapter 2: Recent Results and Performance of World Bank Operations Management is pleased to see that satisfactory outcome ratings for World Bank operations closed in FY14–16 remain above the corporate target by volume (84 percent against 80 percent), and continue xx Results and Performance of the World Bank Group 2017 | Management Response the positive trend by number of operations (73 percent compared with 69 percent in FY12–14) even if they are still below the target (75 percent). Management is especially pleased to see that outcome ratings have improved for operations financed by the International Development Association, and in countries affected by fragility, conflict, and violence. Factors associated with the performance of World Bank operations. Management appreciates IEG’s structured review analyzing factors associated with the performance of World Bank operations. Many of the factors identified through the review resonate well with management’s own findings. As the report states, in response to factors identified as negatively associated with operations’ performance, management has taken actions such as the Implementation Completion and Results Report reform, the new restructuring policy, and the design of an Agile pilot to enhance supervision. Economic analysis for investment projects. The report raises concerns about allowing leeway in the determination of the discount rate. It is well established that discount rates for a project could vary depending on country or sector context, local or project-specific risks, project maturity, and other project characteristics. Therefore, a narrowly prescribed set of discount rates may not be appropriate. The 2016 guidance more appropriately reflects a variety of factors that should be considered for economic and financial analysis, rather than relying on a fixed range; this is an improvement on the earlier guidance.  Although it is true that clear guidance and oversight are needed for all elements of project design, management notes that discount rates are just one of many decisions in which project and client teams have discretion (others include type of intervention, scope of project, and composition of components). Chapter 3: Management Action Record Management is encouraged that the progress on over 90 percent of recommendations in their fourth year of MAR implementation was rated substantial or above by IEG. This compares with 72 percent in RAP 2016 and 80 percent in RAP 2015. Management takes the MAR process seriously, putting substantial effort into formulating action plans and annual implementation updates. Management follows a rigorous process for assessing and rating implementation progress, involving technical experts, relevant sectoral management, and corporate units. Management prepares action plans to follow up on IEG’s recommendations, while taking into account what is feasible and practical to implement and achievable within the four-year MAR implementation period. Sometimes IEG recommendations imply shifts in business priorities or resource allocations that are beyond the MAR process. It is also worth noting that the development landscape is constantly changing, and the Bank Group adjusts and adapts its business model in response. As a result, some recommendations or MAR action plans may not fit or may lose relevance under the new circumstances, leading to the early retirement of some recommendations. IEG reserves the right to disagree with such early retirement and to continue rating the implementation of the recommendations. MAR process. In implementing the current MAR process since 2011, Management has come to realize that the process could be improved to enhance its focus and effectiveness. Independent Evaluation Group | World Bank Group xxi i. First, the MAR process is static and somewhat rigid because it does not allow for adaptation and learning over the four years of implementation. It requires management to define detailed actions and four-year targets at the outset without allowing adaptability or learning- by-doing during implementation. ii. Second, many evaluations recommend that management do more and better in a particular sector or thematic area. Management is always striving to do more and better, but decisions to do more in specific areas depend on the client’s demand, prioritization among competing needs, and resource availability. This requires selectivity, which the MAR process does not encourage. Country demand and alignment with the twin goals are at the center of the Bank Group’s client engagement model, and what the World Bank will support in a country cannot be determined through the MAR. Some recommendations have budget implications, which also cannot be addressed through the MAR. iii. Finally, MAR action plans lead to a proliferation of actions and commitments. For the FY17 update, management reported progress on 146 recommendations from the 23 evaluations completed over the past four years (or close to 300 actions since each recommendation has often more than one action—roughly two per recommendation on average). A greater focus and prioritization in actions and commitments could lead to greater effectiveness. The MAR process is primarily a tool to track management actions to address the recommendations of IEG’s major evaluations. It has improved the systematic reporting and monitoring of MAR implementation. The ongoing joint IEG-Management pilots to improve the process of formulating recommendations and action plans present some lessons. Management believes that a broader review of the MAR process is warranted to take stock of experience so far and make adjustments to improve its effectiveness and impact. International Finance Corporation Management Comments The management of the International Finance Corporation (IFC) appreciates IEG’s effort to produce its annual flagship report, RAP17. With the thematic focus on environmental sustainability this year, the report provides valuable insight into IFC’s development performance. Management is especially grateful for IEG colleagues’ willing engagements, the consideration given to our comments and suggestions, and continued collaboration on further investigations into the drivers of performance. IFC’s commitment to E&S sustainability. IFC management welcomes the recognition of improvements made in the areas of environmental sustainability. As the report correctly identifies, E&S sustainability is one of IFC’s core business principles, and the E&S sustainability agenda is largely implemented through the Performance Standards framework. Although IFC strives to promote environmental sustainability in all projects it supports, it notes that not all the activities promoting environmental sustainability, including many resilience-building activities, fall in the assessed subset—directly financed by projects. Activities such as green bonds also fall outside the project xxii Results and Performance of the World Bank Group 2017 | Management Response cohort considered in this review. Nevertheless, the study documents the increase in activity on the investment side and a rising trend on the advisory side between FY08 and FY17. Environmental sustainability support. IFC management notes references and suggestions with regard to additional tools and practices to support the implementation of the 2012 policy update, including increased supply chain screening, expanded use of third-party monitoring for high-risk sectors, and E&S advisory support to lower capacity clients. The recommendation to strengthen disclosure practice on IFC project-level E&S performance during implementation needs to be balanced with IFC’s operating environment as a private sector investor and lender, including with respect to confidentiality laws, market expectations of confidentiality by IFC clients, and regulatory requirements some clients are subject to. IFC is improving the quality and consistency of its E&S disclosures through an electronic Action Plan Tracker for projects and periodic training for E&S specialists. IFC development results. IFC management notes the continuing decline in the investment development outcome rating to 50 percent on an unweighted basis and a sharp year-on-year decline in IFC Advisory Services’ development effectiveness rating to 49 percent on an unweighted basis. IFC management takes this seriously and has factored it into many of the organizational and procedural changes it has made over the past year and planned for the near future. In FY17, IFC collaborated with IEG on a joint study of the factors driving the decline in investment development outcomes, and in FY18 it has embarked on another diagnostic effort with IEG on Advisory Services’ development effectiveness. These efforts have been extremely helpful in better understanding the causes of the performance declines, and in formulating actions we have taken and plan to take to address them. Efforts to improve investment development outcome performance. Since the completion of the joint study on IFC investment work quality in early 2017, IFC management has been implementing several initiatives to address the work quality issues identified. The study found that severe external challenges IFC faced on the market front, including the global financial crisis, were compounded by diminished quality in the execution of IFC’s work, particularly up-front work (such as screening, appraisal, and structuring). As the report indicates, the Accountability Initiative launched last April deepened our understanding of issues related to roles, accountability and incentive mechanisms, the importance of proactive portfolio supervision, and the focus on the technical and developmental aspects of project assessment. These insights informed the work on accountability and decision making, careers, competencies, and incentives that is being done in support of the operations realignment. The creation of the Economics and Private Sector Development Vice Presidential Unit has helped build capacity for project and macroeconomic analyses and strengthened ties with the World Bank’s macro- and sector economics colleagues. The new Anticipated Impact Measurement and Monitoring framework, developed by the same unit, brings important enhancement to project development impact assessment with a more integral analysis during the project preparation stage. The action plan responding to IEG’s Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group, is another platform being used to address issues related to knowledge and learning found by both the self-evaluation report and the joint 2017 IEG-IFC study. IFC is moving toward a more systematic, comprehensive knowledge management system, which will ensure that evidence and lessons from past projects are fed back into new projects, along with Independent Evaluation Group | World Bank Group xxiii strengthened assessment and documentation. We should also note that other ongoing realignment and organizational changes are intended to improve overall work quality and, in turn, IFC’s financial performance and development impact. It will take time for these and other management changes to show in IEG development outcome ratings, given the lag in tracking ex post performance. IFC management believes that IFC is moving in the right direction generally, and specifically in the areas that IEG’s analysis and the joint assessment found to be problematic. Advisory business development effectiveness. IFC management thanks IEG for its advice on the widening gap between IFC’s self-ratings and IEG’s validation ratings and on the sharp decline in the validated ratings in the latest cohort. Jointly with IEG, IFC management undertook a serious analysis on the root causes of these trends, as well as a separate assessment of Advisory Services governance, focused on financial management. The studies revealed such issues as weakened accountability and decreased rigor and focus in project supervision and monitoring and evaluation. As a result, management developed and began to implement an action plan that includes several measures. Regular, detailed portfolio reviews, one of the recommendations of the joint study, are being initiated. Substantial enhancements to the advisory business model are also being introduced to better align with the IFC 3.0 strategy and investment operations and bring clearer accountability for work quality. In addition, management is discussing with IEG whether IEG could increase from 50 percent to 100 percent the number of IFC Advisory Services’ self-evaluations (Project Completion Reports) that it validates, even for a few years. Other measures include enhancing the project cycle monitoring and evaluation function and strengthening trust fund management. It is important to note that Advisory Services’ development effectiveness rating may temporarily decline further because of the termination of projects identified through the portfolio review that do not meet current strategic relevance or are performing poorly. However, effective advisory support is critical to the success of IFC 3.0 as a key tool for doing the upstream work related to market creation. IFC management is therefore committed to reversing the decline in development effectiveness ratings as soon as possible to arrive at a high-quality, strategically relevant portfolio. Management actions responding to evaluations. IFC management appreciates IEG colleagues’ collaboration and support in regard to its independent inputs to management action development and progress assessment. As World Bank Management notes, the current system can be improved to allow more robust adaptation to change. To be more efficient and effective, the system would benefit from common understandings about the roles of Management and IEG and about the handling of disagreements. Multilateral Investment Guarantee Agency Management Comments According to RAP17, the development outcome success rates of Multilateral Investment Guarantee Agency (MIGA) guarantee projects remained steady at 62 percent. MIGA also notes the report’s important finding regarding the good performance of MIGA guarantee projects in International Development Association countries and countries affected by fragility, conflict, and violence, which are key strategic priorities for MIGA. xxiv Results and Performance of the World Bank Group 2017 | Management Response E&S performance. MIGA welcomes the report’s recognition of the improvement in the environmental performance of MIGA guarantee projects between underwriting and evaluation. RAP17 shows significant progress in MIGA’s E&S supervision has resulted in E&S effects becoming the best- performing development outcome indicator. MIGA also notes that the RAP16 found that MIGA provides active guidance and monitoring to strengthen the E&S performance of its guarantee projects, citing the example of an oil and gas sector project in Uzbekistan, for which the MIGA team helped solve critical E&S issues by convening external industry experts. Assessment, underwriting, and monitoring of MIGA guarantee projects. Although RAP17 reports the improvement in MIGA’s assessment, underwriting, and monitoring of its guarantee projects, it states that more can be done to strengthen these areas, especially concerning risks to development outcome, better documentation, and regular collection of project indicators after the contract of guarantee becomes active. i. With regard to risks to development outcome, MIGA management notes that as a political risk insurance provider, MIGA undertakes extensive political risk analysis and identifies mitigants. The report acknowledges that MIGA undertakes country risk assessment work at the underwriting stage; and in addition, management continues to monitor and track country risk for the duration of its projects. MIGA also engages in extensive project risk mitigation work during project implementation and thereafter for distressed projects that have the potential to lead to a claims situation for MIGA. Indeed, the effectiveness of this work is evident from MIGA’s very low claims payout experience. MIGA also analyzes the credit risks associated with state-owned enterprises, and over the last three years, established a new team to assess these risks. In addition, MIGA mitigates E&S risks in its projects through the application of its Performance Standards and the ongoing monitoring of its projects. Finally, management notes that as a political risk insurance provider, MIGA often has limited leverage, given its indirect relationship with the project enterprise. Therefore, in management’s view, MIGA cannot be expected to mitigate risks to development outcome to the extent that a direct lender or investor, such as IFC or the World Bank, can. It is important to recognize the different, albeit important, role that MIGA plays in supporting projects compared with the roles of the World Bank and IFC. ii. Regarding documentation and data capture, MIGA management stresses the particularities of its business model, which IEG recognized in its 2013 Biennial Report on Operations Evaluation (BROE). As the BROE stated, “MIGA is constrained by the fact that its business model is based on an arms-length relationship with the project company, and access to project information is not automatic.” However, the BROE also points out MIGA’s more active role in measuring development results and underscores its significant progress in gathering development data over time. MIGA also bolstered its understanding of and reporting on the development results of its operations through the Development Effectiveness Indicators System adopted in June 2010. The program provides a system of routine data collection on development results indicators that is used for a variety of purposes (reporting, analysis, self- evaluation) and aggregated to articulate the overall development results of MIGA’s portfolio. Independent Evaluation Group | World Bank Group xxv MIGA’s implementation of 2010 IEG safeguards evaluation recommendations. The report discusses the MIGA-specific recommendations of the 2010 IEG safeguards evaluation and the IEG findings about the implementation of those recommendations. MIGA welcomes the report’s findings, noting that the three MIGA-specific recommendations were implemented successfully (IEG rated them high) and retired from the MAR system in 2014. Monitoring of subprojects of financial intermediaries (FIs). The report finds that MIGA’s monitoring of the subprojects of FIs is still inadequate. However, management points out that the findings are based on a very limited data set. During the period covered in the report, MIGA’s portfolio contained five FI-1 projects—those in which the existing or proposed portfolio of FIs is expected to include substantial business activities with potential significant adverse E&S risks or effects that are diverse, irreversible, or unprecedented—of which IEG reviewed only one for this analysis. All five FI-1 projects have been monitored according to MIGA’s E&S Review Procedures. Overall, as mentioned in the RAP17’s appendix C, MIGA has significantly strengthened the process for monitoring financial intermediary projects (for example, by reviewing Annual Monitoring Reports and making site visits to high-risk subprojects). Disclosure of project-level E&S supervision information. The report states that MIGA’s disclosure of project-level E&S supervision information is inadequate, and that E&S Action Plans (ESAPs) are not disclosed at the due diligence stage. However, MIGA currently discloses information in line with requirements of its Access to Information Policy (2013) and Performance Standards (2013), as discussed in appendix C. ESAPs have been consistently disclosed on MIGA’s website (under the “Availability of Documentation” tab) since the introduction of the 2013 Policy and Performance Standards. In addition, MIGA management recognizes that strengthening disclosure of project-level E&S supervision information would require amending the Access to Information Policy and therefore further consultation and engagement with the Board of Executive Directors. Basis for findings. Although the report presents many useful findings, MIGA management finds some conclusions unclear because they are provided without presenting evidence or the basis for the conclusions. Likewise, the report shows a tendency to generalize from a limited set of project evaluations, a fact that is not made clear to readers and that limits the usefulness and validity of the findings. A good example is the report’s statement that without solid results data, the development outcome of MIGA’s “new” products is difficult to ascertain. Management notes that the discussion of its new products is presented without any evidence or any definition of them. xxvi Results and Performance of the World Bank Group 2017 | Management Response 1 highlights 1 The share of World Bank project components with potential environmental benefits has increased by four percentage points, from 33 percent in FY08–10 to 37 percent in Addressing FY15–17. Climate change mitigation and Environmental adaptation activities increased, while Sustainability biodiversity conservation, irrigation efficiency in World Bank improvements, and water resource Group management declined. Operations 2 The share of International Finance Corporation (IFC) investment projects containing activities with potential environmental benefits has also increased by four percentage points, from 31 percent in FY08–10 to 35 percent in FY15–17, and IFC Advisory Services with potential environmental benefits have shown a small rising trend over FY08–17. 3 The share of Multilateral Investment Guarantee Agency (MIGA) projects with potential environmental benefits has grown significantly from 8 percent in FY08–10 to 36 percent in FY15–17. 4 Country strategy indicators on environmental sustainability are generally of adequate quality but provide an incomplete picture of 1 country-level results because they do not capture all major activities and could assess achievements for only 77 percent of objectives or outcomes. 5 World Bank project indicators on environmental sustainability are broadly adequate for Clean and Resilient projects, but because of indicator design issues, they are often inadequate to demonstrate the effectiveness of Green projects or show capacity building across all sectors. 6 Measurement systems for environmental results are more effective when they include intermediate outcomes or tie quality measures into indicators. Measurement systems are limited by the project cycle and leave knowledge gaps on long-term impacts. 7 Reforms to environmental and social risk management policies for World Bank investment project financing and IFC investments and MIGA guarantees resulted in some improvements, but strengthened monitoring and evaluation systems for the World Bank are not yet in place and some newly introduced differences in treatment across institutions may require further harmonization. 2 Results and Performance of the World Bank Group 2017 | Chapter 1 Environmental Sustainability in the World Bank Group S U S TA I N A B L E D E V E LO PM E N T PR O G R E S S , particularly environmental sustainability, is a central tenet of the World Bank Group’s strategy. In its 2013 strategy, the Bank Group committed to reaching twin goals—to eradicate extreme poverty and promote shared prosperity—in a sustainable manner. The strategy defined sustainability as a three-pillar paradigm: fiscal, social, and environmental. It noted that sustainability is critical to ensuring that “today’s development progress is not reversed tomorrow and that the pace of progress does not flag” (World Bank 2013b). It highlighted the urgent challenge of environmental degradation and noted that environmental constraints are already affecting progress on poverty reduction. The 2016 “Forward Look: A Vision for the World Bank Group in 2030—Progress and Challenges” confirmed the importance of environmental sustainability (World Bank Group 2017a). Environmental sustainability runs through two of the three ways that the “Forward Look” seeks to achieve the twin goals, namely, accelerating inclusive and sustainable economic growth and fostering resilience to global shocks and threats (the third way being investing more effectively in people). The “Forward Look” also noted that pressures on environmental sustainability are increasing, giving this agenda a sense of urgency. Under the 2016 Climate Change Action Plan, the Bank Group announced targets for increasing climate finance. The World Bank Group’s emphasis on environmental sustainability is also aligned with the global agenda.1 Following the definition adopted in the Bank Group’s 2012 strategy on environment, the organizing framework for assessing environmental sustainability activities in Results and Performance of the World Bank Group 2017 (RAP17) has three main components: Clean, Green, and Resilient. The Bank Group does not have a single definition of environmental sustainability, but its 2012 strategy describes three environmental areas: Green, including sustainable management of land, forests, and oceans, including ecosystems and biodiversity; Clean, including a low pollution and emission world with clean air, water, and oceans that allow for healthy lives; and Resilient, including resilience to climate shocks and adaptation to climate change (World Bank 2012a).2 In consultation with operational colleagues, the Independent Evaluation Group (IEG) developed a classification framework of environmental sustainability, which breaks down the Clean, Green, and Resilient areas into 13 subsectors and 59 activity types (table 1.1; see appendix B, table B.1). This framework is then used to identify environmental objectives in country strategies and in portfolio reviews of World Bank, International Finance Corporation (IFC), and Multilateral Investment Guarantee Agency (MIGA) projects. A separate portfolio definition based on the World Bank’s environment theme codes is used for identifying core World Bank environment projects for analysis of their results measurement and outcomes. Independent Evaluation Group | World Bank Group 3 TABLE 1.1 | Environmental Sustainability across Clean, Green, and Resilient Areas Subsectors Identified in the Independent Evaluation Group Classification Framework Clean Green Climate Resilient Climate change mitigation Biodiversity conservation Climate change adaptation Air pollution abatement Landscapes Disaster risk management Water and soil pollution abatement Oceans or fisheries Waste management Sustainable agriculture Forests Source: Independent Evaluation Group analysis. RAP17 looks broadly at the whole Bank Group portfolio to identify all project activities that could support environmental sustainability and then looks in detail at how well those effects are measured for the subset of activities where good measures exist. RAP17 does not attempt to present the overall impact of the Bank Group on environmental sustainability or to focus on performance results for environment because of data limitations.3 Most environmental effects of Bank Group interventions do not come through projects with explicit environmental goals: they often come as cobenefits or as indirect effects, and so RAP17 seeks to identify projects with these effects.4 An approach that considered only those “core” activities where this information was available would fail to capture a large portion of the Bank Group’s activities. In addition to being the theme of chapter 1, environmental sustainability permeates chapters 2 and 3. Environmental Sustainability in the World Bank Group Portfolio RAP 17 examines whether an increased focus on environmental sustainability at the strategy level has led to changes in the environmental focus of Bank Group projects. An IEG evaluation on environmental sustainability that covered up until FY07 noted that support for the environment in the World Bank had grown over the preceding 15 years (World Bank 2008). Since then, environmental objectives appear in high-level environment strategies, Bank Group–wide frameworks and action plans, in some sector strategy documents, and in corporate strategies. All three Bank Group institutions seek to support environmental sustainability through multiple instruments, both through promoting positive effects as part of the development outcomes of their activities and by managing potential negative effects through a “do no harm” approach. This includes support through projects, knowledge and advisory work, convening roles and partnerships, and application of Performance Standards. The institutions differ somewhat in that for IFC and MIGA approaches to environmental and social performance are combined. Appendix A describes these strategies and approaches and how they differ across institutions. (In particular, there were specific targets for climate change under the 2008 Strategic Framework on Development and Climate Change and the 2016 Climate Change Action Plan, though it did not have corporate targets for environmental sustainability outside of climate change.)5 A common theme of these strategies has been a desire to promote environmental sustainability in the Bank Group’s projects and to “mainstream” environmental sustainability across 4 Results and Performance of the World Bank Group 2017 | Chapter 1 its activities.6 To assess an important part of this, IEG reviewed large samples of World Bank, IFC, and MIGA projects using a detailed classification framework of activities with potential benefits for environmental sustainability (see appendix B). World Bank Project Portfolio The World Bank has increased activities with potential environmental benefits in its projects over the past decade. The share of World Bank project components with potential benefits for environmental sustainability has increased by four percentage points, from 33 percent in FY08–10 to 37 percent in FY15–17; this result is statistically significant at the 95 percent confidence level. These results are based on a comprehensive review of the components (or pillars for development policy financing) of a one-third random sample of projects approved in FY08–10 and FY15–17, covering 1,684 components from 558 projects. (Results are similar whether considered as shares or absolute numbers.) Appendix B describes the methodology. This review did not include analytical and advisory activities or nonstandard projects such as carbon finance. Most of the activities in World Bank projects with potential environmental benefits are in the Clean area, and the share of components with potential Clean benefits increased slightly over time, from 19 percent to 22 percent (figure 1.1, panel a). Within Clean, the most common activities supported climate change mitigation,7 followed in decreasing order by air pollution abatement (including cobenefits from climate change mitigation measures),8 water and soil pollution abatement, and waste management. Comparing FY08–10 to FY15–17, the latter saw an increase in some climate change mitigation activities, especially in renewable energy, green growth, and other cross-sector climate change activities, regional power pools supporting hydropower, and in activities addressing forest fires and deforestation. It also saw an increase in activities with potential air pollution abatement effects, but these were driven largely by climate change mitigation activities with air pollution cobenefits rather than activities specifically addressing air pollution. Support decreased for both supply-side and demand-side energy efficiency. A recent IEG evaluation on pollution undertook a more detailed portfolio review over a longer period (World Bank 2017c). It noted that the share of Bank Group support aimed at pollution abatement had lost relative importance, and recommended that the Bank Group better leverage its climate change mitigation portfolio to combat air pollution by explicitly including pollution concerns in these projects’ designs. Project components with potential Green benefits have seen their share in the portfolio decline from 13 percent to 10 percent between FY08–10 and FY15–17, and the structure of the Green portfolio changed. In the first period, most Green benefits were for water management, followed by biodiversity conservation and then landscapes, sustainable agriculture, and forestry. In the second period, the shares of project components supporting biodiversity conservation (including protected areas) and supporting water management (including water resource management, irrigation efficiency improvements, and other activities) have dropped substantially.9 Some Green activities, however, have seen increases in support. These include fisheries management, forest management, and sustainable agriculture.10 Starting from a low base, the share of components with potential climate resilience benefits has increased significantly, from 5 percent to 10 percent. In both periods, most resilience benefits were Independent Evaluation Group | World Bank Group 5 Figure 1.1 |W  orld Bank Project Activities and Financing Commitments With Potential Environmental Benefits Increased for Clean and Resilient but Decreased for Green Activities a. N  umber of World Bank project components  ommitments for World Bank project b. C with potential environmental benefits, by type components with potential environmental of benefit benefits, by type of benefit 60 60 43 40 37 40 Percent Percent 33 31 23 26 19 22 20 20 13 10 15 10 8 5 5 5 0 0 Clean* Green* Resilient*** Any Clean Green** Resilient*** Any environment environment*** Benefit type Benefit type 2008–10 2015–17 Sample Population Years Projects (no.) Commitment ($, millions) Projects (no.) Commitment ($, millions) 2008–10 282 31,489 848 103,780 2015–17 276 28,683 827 105,586 Note: Components including activities with potential Clean benefits are coded as Clean; likewise for Green and Resilient. “Any Environment” is not a sum of the others, because an individual component may have (for example) both Clean and Green benefits. * Indicates change between two periods with 90 percent statistical significance, ** with 95 percent, and *** with 99 percent. from disaster-related or disaster-resistant infrastructure and in institutional reform and capacity building. In the first period, the majority of disaster-related activities were for flood and storm protection, but in the second period, a broader set of activities was present, including climate resilient roads and climate resilient infrastructure across a range of sectors. The World Bank has also increased the share of its project financing going to project components with potential environmental benefits. The share of the World Bank’s financing commitments attached to components with potential environmental benefits also increased, from 31 percent to 43 percent (figure 1.1, panel b). This increase is larger than the increase in the share of components. This discrepancy is driven largely by a change in resilience activities, as the World Bank is now more often involved in financing physical infrastructure, leading to an increase in financing that was larger than the increase in the share of components. In contrast, the share of financing for Green activities decreased by more than its decline in its share of components, as large components for irrigation and water resource management in the FY08–10 sample were not matched in FY15−17. For Clean activities, the changes in share of components and share of financing were similar. The increase in overall financing occurred despite a shift away from development policy financing (DPF; which might be expected to provide larger commitments) toward investment project financing (IPF; see appendix C). 6 Results and Performance of the World Bank Group 2017 | Chapter 1 Most activities with potential environmental benefits are found in projects mapped to the Sustainable Development Practice Group, but there has been a small increase over time in other practice groups (figure 1.2).11 The Sustainable Development Practice Group has both the largest overall portfolio in the World Bank and the highest share of components with environmentally beneficial activities. In the Human Development Practice Group, activities with potential environmental benefits increased slightly. These activities included resilient schools in an education project, medical waste management and pollution management in health projects, and disaster responsive safety nets in social protection projects.12 In the Equitable Growth, Finance, and Institutions Practice Group, examples of environmental support in both IPF and DPF increased.13 These results suggest a degree of environmental mainstreaming across practice groups other than Sustainable Development. Regions differ significantly in their involvement in activities with potential environmental benefits. In East Asia and Pacific, the share of components with potential environmental benefits almost doubled from FY08–10 to FY15–17, leaving it with the highest share in the second period, driven largely by China, which had more than half of the Region’s environmental activities. In contrast, environmental activity in the Middle East and North Africa decreased over time (figure 1.3). Appendix C explores these patterns in more detail. In higher-income countries, a larger share of project components have potential environmental benefits than in lower-income countries. The share of components with potential environmental benefits has increased over time for all income groups, but the share is largest for upper-middle- income countries (UMICs), followed by lower-middle-income countries (LMICs) and finally low- Figure 1.2 |A  ctivities With Potential Environmental Benefits Spread Beyond the Sustainable Development Practice Group a.  World Bank project components with potential b. World Bank project commitments for environmental benefits, by practice group components with potential environmental benefits, by practice group 80 80 60 59 56 55 60 60 Percent Percent 40 40 14 16 20 7 20 8 5 3 7 4 0 0 Sustainable Human Economics Sustainable Human Economics Development Development and Finance Development Development and Finance (N = 484; (N=182; (N=210; (V = 15; (V=8; (V=9; N= 486) N=157) N=165) V= 19) V=5) V=5) Practice Group Practice Group 2008–10 2015–17 Source: Independent Evaluation Group analysis. Note: N = number of all components in Independent Evaluation Group sample by practice group; V = volume of all components in Independent Evaluation Group sample by practice group in $, billions. Independent Evaluation Group | World Bank Group 7 Figure 1.3 |E  ast Asia and Pacific Saw the Largest Increase in Environmental Support a. S  hare of components with environmental  hare of commitments for components with b. S activities in the World Bank portfolio, by environmental activities in the World Bank Region portfolio, by Region 100 100 80 80 73 61 Percent Percent 60 60 52 42 43 42 32 34 32 38 40 40 31 32 29 27 28 40 31 36 31 36 27 32 20 20 10 7 0 0 EAP ECA LCR MNA SAR AFR EAP ECA LCR MNA SAR AFR Region Region 2008–10 2015–17 Number of All Components in Sample by Region Year EAP ECA LCR MNA SAR SSA 2008–10 128 99 156 90 86 261 2015–17 140 83 94 28 112 294 Source: Independent Evaluation Group analysis. Note: The large increase in East Asia and Pacific is driven primarily by projects in China. AFR = Sub-Saharan Africa; ECA = Europe and Central Asia; EAP = East Asia and Pacific; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. income countries (LICs). In aggregate, the gap has widened over time, as support for environmental sustainability increased faster in higher-income countries. Moreover, activities with potential environmental benefits are concentrated in large clients. The 10 countries with the largest World Bank portfolios collectively constitute about one-third of project activities (35 percent) but roughly half of the project activities with likely environmental effects (49 percent). Over time, support for climate change mitigation has shifted from being largely the province of UMICs toward being an equal share of environmental support across country income groups. Air pollution abatement activities remain more prevalent in UMICs, but differences decreased over time because of climate change cobenefits. Resilience activities, however, are present in LICs at rates equal to or higher than other income groups, because LICs may be more vulnerable to natural disasters and climate shocks. Appendix C provides more details. The share of World Bank analytical and advisory activities in the Sustainable Development Practice Group with potential environmental benefits has remained stable, but the absolute number has increased by roughly 60 percent due to the expansion of World Bank Advisory Services and Analytics (ASA) over this period (appendix C). This calculation is based on a review of a sample of 480 ASA 8 Results and Performance of the World Bank Group 2017 | Chapter 1 activities in the Sustainable Development Practice Group (appendix B) and does not cover other practice groups. The patterns of potential environmental benefits from ASA are broadly similar to those for projects, but the changes over time in shares are smaller for ASA. IFC Project Portfolio The share of IFC investment projects with potential environmental benefits has increased by four percentage points, from 31 percent in FY08–10 to 35 percent in FY15–17. IEG reviewed a random sample of 151 of 814 long-term investment projects in FY08–10 and 151 of 712 projects in FY15–17, and the change in share is not statistically significant with this sample size. All IFC projects seek to support environmental sustainability through the application of Performance Standards, but the analysis of potential environmental benefits considered only the benefits from activities being financed by the project. For IFC investments, the majority of project activities with potential environmental benefits are in the Clean area, with a smaller share in Green, and very little in Resilient (figure 1.4, panel a). The largest shares are for climate change mitigation (mostly renewable energy) and air pollution abatement (mostly cobenefits from climate change mitigation activities), followed by water pollution abatement (largely wastewater treatment).14 Other categories of environmental activities have seen only a few projects. Environmental activities in the IFC portfolio in FY15–17 look broadly similar to those in FY08–10. The largest change has been an increase in projects with climate change mitigation benefits; other environmental benefits have not seen any significant change. By industry group, the infrastructure group has the largest share of projects with potential environmental benefits, followed by manufacturing and agribusiness (figure 1.4, panel b). In IEG’s sample, both industry groups saw sizable increases in the share of projects with potential environmental benefits, whereas sample sizes are too small for other industry groups to draw meaningful results.15 Infrastructure projects largely offer Clean benefits, with positive environmental effects from projects in electricity, power, transportation, and oil and gas.16 Manufacturing, agribusiness, and services projects included mostly Clean benefits but also have many Green benefits, including energy efficiency in manufacturing companies and from agribusiness and forestry.17 Potential benefits in financial institutions projects comprised almost entirely Clean benefits. IFC worked with financial intermediaries either to offer lines of credit or establish risk-sharing facilities dedicated to the financing of environmental equipment and activities, and often targeted at small and medium enterprises (SMEs).18 Private equity, venture capital, and telecommunications group projects have potential environmental benefits from investments in clean energy, waste recycling, and emissions control. Although there has been significant variation over time in the share of IFC Advisory Services with potential environmental benefits, the share trends slightly upward (see appendixes B and C). MIGA Project Portfolio Starting from a low base, the share of MIGA projects with potential environmental benefits has grown significantly, reaching 36 percent in FY15–17 (figure 1.5). IEG reviewed 26 of the 36 MIGA political risk guarantee projects in FY08–10 and 36 of 75 projects in FY15–17.19 The FY08–10 portfolio was Independent Evaluation Group | World Bank Group 9 Figure 1.4 |I FC’s Share of Projects with Potential Environmental Benefits Increased Slightly, Mostly in Infrastructure and Manufacturing, Industry, and Agribusiness a. I FC’S share of projects with potential FC investment projects with potential b. I environmental benefits increased slightly environmental benefits are mostly in INR and MAS 80 80 72 60 60 60 55 Percent Percent 35 38 40 30 33 31 40 20 17 18 20 8 9 11 11 0 1 0 0 Clean Green Resilient Any CTT FIG INR MAS environment Benefit Type Cluster 2008–10 2015–17 Note: For panel a, the sample consists of 151 of 814 projects approved in FY08–10 and 151 od 712 approved in FY15–17. For panel b, the FY08–10 sample consists of 18 CTT, 57 FM, 31 Infrastructure, and 45 MAS projects; the FY15–17 sample consists of 19 CTT, 64 FIG, 25 Infrastructure, and 43 MAS projects. CTT = telecom, media, technology, venture investing, and funds; FIG = Financial Institutions Group; IFC = International Finance Corporation; INR = infrastructure and natural resources; MAS = manufacturing, agribusiness, and services. Figure 1.5 |M  IGA Increased Its Share of Projects with Potential Environmental Benefits 40 36 36 Percent 20 8 8 8 4 3 0 0 Clean Green Resilient Any Environment Benefit Type 2008–10 2015–17 Note: The MIGA sample covers 26 of 36 projects approved in FY08–10 and 36 of 75 projects approved in FY15–17. MIGA = Multilateral Investment Guarantee Agency. 10 Results and Performance of the World Bank Group 2017 | Chapter 1 heavily concentrated in the financial sector; the review found two projects (8 percent of the sample) with potential positive environmental benefits. These projects were for a housing construction supply company (seeking to produce green products that would improve energy efficiency) and a wastewater treatment company. The FY15–17 portfolio is larger, with more than half of the projects in infrastructure or manufacturing or agribusiness. Of the 36 projects reviewed, 13 had potential environmental benefits. These included a metro system, roads and highways, hydro- and solar- powered plants, a large scrap metal recycling operation, and a bamboo plantation forest / processing facility. One-third of these are joint projects with the World Bank or IFC, including one to upgrade an urban transport network. The latter also provides support for developing the city’s resilience and capacity in environmental and disaster risk management. Environmental Sustainability in World Bank Group Country Strategies Country strategies over FY08–17 consistently included objectives and outcomes with potential environmental benefits (figure 1.6). The Bank Group prepared 208 strategies for 120 countries between FY08 and FY17. For RAP17, IEG analyzed the extent to which objectives or outcomes in these strategies supported environmental sustainability, using the same classification framework as for projects. Of these strategies, 154 (74 percent) included at least one environmental objective or outcome.20 The share of country strategies with environment objectives and outcomes fluctuated by fiscal year, with no discernible long-term pattern. The median share of environment to total objectives or outcomes ranged between 11 percent and 17 percent during the past 10 years. Figure 1.6 |W  orld Bank Group Country Strategies Show Consistency and No Clear Pattern in the Level of Attention to Environment 30 100 Numer of Country Strategies 80 20 Percent 60 40 10 20 0 0 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 N=21 N=22 N=17 N=17 N=23 N=22 N=24 N=15 N=21 N=26 Fiscal Year No. of Country Strategies Share of Strategies Median Share of Env. with Env. Outcomes/ Outcomes/Objectives Objectives Source: Independent Evaluation Group analysis of all country strategies approved in FY08–17. Independent Evaluation Group | World Bank Group 11 Even with annual fluctuations, the relative shares of the Clean, Green, and Resilient areas are roughly equivalent over the FY08–17 period (figure 1.7). The three areas each accounted for about one-third of the environment objectives and outcomes in country strategies: Clean at 34 percent, Green at 35 percent, and Resilient at 31 percent. The main change over time has been an increasing frequency of objectives or outcomes on climate change mitigation and adaptation. A comparison of the FY08–12 country strategies, where information was available at the start and at completion, found that the shares of the Clean, Green, and Resilient areas had not substantially altered during implementation. These results do not contradict those from the project-level analysis of environmental benefits, which found evidence of changes in support over time and of smaller support for Green objectives. Country strategies include analytical and advisory work, which is often present for Green objectives in country strategies, but this work was not captured by the analysis of projects. Appendix C describes other reasons we might expect to see different patterns in country strategy objectives versus project components; these include the fact that the analysis of country strategies counts every country equally, whereas most project components are in a few countries with the largest portfolios. Environmental sustainability received attention in country strategies in all Regions over the past 10 years but with variation in the frequency of environmental support and the type of environment objectives. The Latin American and the Caribbean Region had the largest share of country strategies with an environmental objective, while the Africa Region had the smallest share.21 All Regions had objectives or outcomes in all three environmental areas but at different rates, reflecting the different needs and challenges of each.22 Clean projects were a focus for Europe and Central Asia (60 percent of environmental objectives) and Middle East and North Africa (53 percent). Green objectives were Figure 1.7 |T  he Type of Benefits Supported by Country Strategy Objectives Varies Over Time 100 Number 0 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 Fiscal year Clean Green Resilient Source: Independent Evaluation Group analysis of all country strategies approved FY08–17. Note: Variation is due to which countries have country strategies. 12 Results and Performance of the World Bank Group 2017 | Chapter 1 most prominent in Africa, where they accounted for over half of the environmental objectives and outcomes (56 percent). Green and Resilient were priorities in Latin America and the Caribbean (81 percent combined) and East Asia and Pacific (75 percent combined), while South Asia saw the Clean and Resilient (79 percent combined) as its priorities. These priorities were consistent with the Regional challenges noted in the Bank Group’s environment strategy.23 The focus on environmental sustainability in country strategies appears to be greater in middle- income countries than in LICs; LICs do not include Clean objectives. Environment’s share is higher in middle-income country strategies: 88 percent of UMIC and 72 percent of LMIC strategies had an environment objective or outcome, whereas just under half (49 percent) of the LIC strategies did. All three areas, Clean, Green, and Resilient, are included in UMIC and LMIC strategies, but LIC strategies did not include any objectives or outcomes in the Clean area. This reflects the nature of the challenges facing countries at different levels of development and the difficulty of establishing environmental programs in LICs. IEG’s evaluation on Systematic Country Diagnostics (SCDs) and Country Partnership Frameworks (CPFs) indicates that the focus on environmental sustainability has continued under the new country engagement model (World Bank 2017d). An early-stage assessment of the first 22 SCD/CPF pairs found that environmental sustainability was addressed adequately by SCDs. Most of the SCDs contained a substantial discussion of climate change or environmental sustainability, and included these issues directly or indirectly among SCD priorities, although some SCDs in vulnerable countries (such as Mali) did not discuss climate change.24 In CPFs, 18 of the 22 strategies had objectives or subobjectives addressing climate change or environmental sustainability. Measuring Environmental Sustainability in the World Bank Group Measuring how environmental benefits are realized is an important goal. The preceding section demonstrates that the Bank Group has consistently engaged in a wide range of activities with potential environmental benefits. But what do we know about whether those benefits were actually realized? Have they been measured? This section examines the extent to which the main measurement systems of the Bank Group capture results on environmental sustainability. Measuring Environmental Sustainability at the Corporate Level The Corporate Scorecard includes indicators that track some forms of the Bank Group results on environmental sustainability. The Scorecard is designed to support monitoring of the Bank Group strategy and to provide an apex view of results and performance indicators. The FY14–17 Scorecard includes 10 indicators related to environmental sustainability (table 1.2).25 Most indicators are well chosen, and use reasonable proxies for environmental outcomes where they exist. The Scorecard can only be expected to capture a portion of results in high-priority areas, so it covers only some aspects of environmental sustainability. It is difficult to find indicators that are meaningful, available, and easy to aggregate. For FY18, management has stated that it will drop a weaker input- based indicator and add new indicators, such as measures of estimated greenhouse gas emission Independent Evaluation Group | World Bank Group 13 TABLE 1.2 | Environmental Sustainability Indicators in the FY14–17 Corporate Scorecard Tier Indicator Tier 1 (global)  Number of countries without wealth depletion  Countries mainstreaming disaster risk  Population exposed to harmful air pollution  Global CO2 emissions  Global annual average deforestation change Tier 2 (projects)  Countries institutionalizing disaster risk reduction as a national priority  Generation capacity of renewable energy  Emission reduction with support of special climate instruments  Projected lifetime energy and fuel savings Tier 3 (corporate)  Climate-related World Bank Group commitments Source: World Bank Group Corporate Scorecard. abatement from Bank Group projects and an indicator to track sustainability of groundwater use.26 These planned indicators appear to capture important environmental outcomes. The proposed FY18 Scorecard will capture some aspects of climate change mitigation, forest management, air pollution, water scarcity, and disaster risk management. It will not capture other environmental issues, such as water pollution (separately from water supply), hazardous waste, land degradation, biodiversity conservation, sustainable agriculture, or broader climate resilience.27 The indicator closest to the idea of measuring overall environmental sustainability is on “countries without wealth depletion.” This is based on substantial World Bank work to generate an estimate of adjusted net savings across financial, human, and natural capital, but it remains limited by data availability and challenges such as the difficulty of estimating human capital or many forms of natural capital (land, water, biodiversity, and so on). It may be more useful for countries where natural resource extraction (which is captured well by the methodology) is more important than in other countries. Some aspects of the Scorecard may inhibit its use for accountability purposes.28 Over the past 10 years, the Bank Group increased support for climate change mitigation and adaptation where it had corporate priorities, targets, and mandates, including in the Scorecard. But some other types of environmental activities without these declined. The inclusion of indicators in the Scorecard appears to establish institutional incentives at the operational level because operational staff see the Scorecard as a signal of corporate priorities. Yet the Scorecard cannot cover all environmental effects. Could this lead to some environmental goals being inadvertently neglected? Measuring Environmental Sustainability in Country Strategies Indicators from CPFs provide information on the Bank Group’s contribution to environmental sustainability across all major environmental subsectors. CPFs include a results framework that lists 14 Results and Performance of the World Bank Group 2017 | Chapter 1 the objectives (the development results the Bank Group aims to achieve) and indicators to provide evidence that the objectives have been achieved. An IEG review of all 52 CPFs prepared between FY15 and FY17 found that 39 (75 percent) included at least one environmental objective, based on the classification framework used in this report. A review of the 117 environment indicators in these 39 CPFs found that all aspects of the environment agenda were covered—though with emphasis on climate change (see appendix C, figure C.6). However, country strategy indicators do not and cannot fully capture the Bank Group environmental results in a country. The number of instruments (IPF, DPF, knowledge products, global programs, and IFC and MIGA projects) listed under an environment objective ranged from 1 to 43.29 Each objective averaged two indicators, intended to cover an average of nine instruments.30 So, coming up with a few specific and measurable indicators to fully reflect the results of a multidimensional environmental program is a significant challenge. There is a trade-off between simplicity (limiting the number of indicators and allowing for aggregation and measurability) and comprehensiveness (capturing the different aspects of the country program). Appendix C provides examples of cases where indicators may facilitate aggregation across different operations but do not capture important results. The impact of knowledge work is particularly hard to capture. Given these tensions, overall indicator quality was largely adequate—but with a few concerns. Practically all 117 indicators reviewed had baselines (98 percent of all indicators) and targets (97 percent), although close to two-thirds of the baselines (63 percent) were set at 0 (raising questions about whether a system exists to collect the data). Almost all indicators related logically to the objective they supported. Just over three-quarters (77 percent) were considered sufficient to demonstrate the achievement of their part of the objective. Most of the objectives were formulated in broad strategic terms, making this assessment challenging. For example, the indicator, “the adoption of a plan” was not considered adequate because a plan could be necessary but not sufficient to demonstrate whether the objective “prevent degradation of natural resources” is met. Output indicators capture achievements easily attributed to the Bank Group, but they are seldom sufficient to test whether the development results have been achieved. Measuring Environmental Sustainability in World Bank Projects Project-level indicator systems do not give an overall picture of the World Bank’s impact on the environment because of the diversity of environmental outcomes supported by projects and the wide range of individual project-specific theories of change even within a subsector. The World Bank’s project-level indicators provide information on environmental results for those projects that have environmental objectives. IEG considered 60 recently approved or evaluated projects with an environmental objective to assess the quality of those indicators and identify any information gaps (see appendix B for methodology). The information provided by project-level indicators is constrained by project scales and time frames: project-level indicators cannot generate information on long-term impacts or the cumulative effect of multiple projects or programs. Indicators for most Clean and Resilient projects are adequate to assess achievement of outcomes, but Green projects rarely generated sufficient information (see appendix C for results). Clean projects Independent Evaluation Group | World Bank Group 15 had clear links between activities and pollution outcomes, with well-established methodologies for measurement, and in some cases, payment for results approaches. Resilience projects do not measure their impact on vulnerability but can provide useful evidence on results by tying quality standards into their indicators. Green projects typically measure outputs or areas where practices are to be implemented but tend not to measure environmental results. This occurs because of their more complicated intervention logic, longer-term goals, lack of use of intermediate outcomes, and lack of measurement of economic effects. Indicators have been inadequate to capture capacity building across all project types. Pressure to include corporate indicators in projects can sometimes weaken project results frameworks if the indicators are not good measures of project objectives. Since 2009, the World Bank has gradually introduced core sector indicators across its portfolio with the aim of facilitating aggregation of results across projects. In 2016, their number was dramatically reduced to become the corporate results indicators, which improved their utility.31 Though it may not have been the intention, creation of these indicators, and in some cases their linkage to the Corporate Scorecard, established internal incentives to prioritize their use. This creates trade-offs, however, in trying to have indicators that are aggregable versus actually getting at project results based on their theories of change. Most core sector indicators are outputs. IEG’s review of results frameworks and concerns raised by project teams shows that, in some cases, the incorporation of core sector indicators as a project development objective (PDO) results indicator comes at the expense of indicators that could be of greater use in demonstrating project outcomes.32 Core sector indicators were less problematic when they measured dimensions of a project that were a fundamental part of the causal chain for reaching the project’s objectives, such as greenhouse gas emission reductions. Measuring Environmental Sustainability in IFC Projects IFC’s environmental monitoring systems address mitigation of risks and impacts but do not measure positive environmental effects, including those beyond the direct footprint of the project. IFC monitors the environmental risks and impacts of projects through application of its Policy and Performance Standards on Environmental and Social Sustainability. Direct investment clients self-report annually on environmental performance of the project with respect to greenhouse gas and air emissions, waste and hazardous materials, liquid effluent discharges, and resource and energy consumption, including energy and water management.33 The reports also cover other environmental aspects, such as environmental management system status and biodiversity conservation. Conclusions on Measuring Environmental Sustainability The Bank Group’s main results systems provide useful data on environmental sustainability and are most effective when they concentrate on the specific purpose of each measurement system. Systems are not suited to providing a comprehensive picture of the impact of the Bank Group on environmental sustainability, but it would arguably be unrealistic to expect them to be. Existing indicator systems are most effective when they prioritize the needs of each instrument: for a corporate system, to track a few high-level priorities; for a country system, to give an overall impression of progress at an aggregate level; and for a project system, to provide adequate information to assess achievement of project objectives. But adding other pressures can dilute their effectiveness at their core role—for example, 16 Results and Performance of the World Bank Group 2017 | Chapter 1 pushing project indicators to be designed in a way that is easy to aggregate rather than one that meets the specific context can reduce their ability to capture project impacts. Measurement systems are tied to the project cycle and so leave knowledge gaps on long-term effects. Indicators can seldom capture those results beyond what can be achieved and observed at the time of project closure.34 This means that a knowledge gap remains on the longer-term results of many environmental interventions, especially for those on natural resource management.35 Heeding the recommendations of IEG’s evaluation on self-evaluation systems could help to close the knowledge gap by expanding the use of voluntary self-evaluations (World Bank 2016a). This could involve use of analytical work on a selective basis (and outside its project cycle) to study medium- and longer-term environmental results in a more formal and rigorous way for key intervention types. This would also respond to the learning needs of World Bank management and teams. Environmental and Social Risk Management The Bank Group uses environmental and social (E&S) risk management policies to guard against and mitigate potential negative outcomes from its projects. For this report, IEG carried out a review of recent E&S policy updates (comparing them with previous IEG findings and recommendations), an assessment on the reporting quality of safeguards in World Bank projects, and a presentation of results and explanatory factors for IFC and MIGA Performance Standards. Environmental and Social Risk Management Policy Updates and Implementation Policy updates in 2012, 2013, and 2016 have improved the Bank Group’s E&S risk management. IEG reviewed how the findings and recommendations of IEG’s 2010 safeguards evaluation have been addressed in the Bank Group’s policy updates (World Bank 2010); the 2016 World Bank E&S Framework (ESF), which is expected to become operational in FY19; the 2012 IFC updates to the Sustainability Framework and Performance Standards; and the 2013 MIGA revisions to its Sustainability Framework.36 For IFC, IEG also reviewed implementation of the recommendations based on a sample of IFC long-term investment projects committed from FY14–17 and conducted interviews with select IFC staff (see appendix B). The World Bank’s 2016 ESF addresses E&S issues identified as priorities by IEG. IEG’s 2010 evaluation found that new issues (such as climate change and environmental health and safety) and a range of social issues (labor, social inclusion and gender, and conflict and fragility) had emerged over 1990–2010 and required greater consideration by E&S policies (World Bank 2010). The new ESF strengthens provisions for environmental sustainability, including in climate change and community health and safety.37 It significantly expands the scope of social safeguards to harmonize thematic coverage with IFC and MIGA. This provides better balance between environmental versus social issues and strengthens the synergies between them. The ESF also addresses concerns about inadequate disclosure and participation by requiring stakeholder consultation throughout the project’s life cycle. Independent Evaluation Group | World Bank Group 17 The World Bank has redressed the imbalance in safeguard management by increasing attention to supervision. The previous safeguard policies heavily emphasized E&S risks during project preparation, but the ESF has a better balance between project preparation and supervision. In anticipation of reforms to ESFs, the World Bank shifted budget responsibility for safeguard supervision to the Global Practices for Environment and Social Development. This pooled funding of E&S resources may make it easier to allocate E&S budgets across projects to match risk ratings and to respond to changes of risk ratings over time. The World Bank launched accreditation programs for safeguard specialists and pledged greater resources for training to support ESF rollout. Training programs for accreditation were introduced in FY13 and are ongoing, along with development of e-Learning modules and operational mentoring. A large-scale training program for all E&S specialists, task team leaders, and country clients is ongoing, aimed at familiarizing them with the new framework. The World Bank has pledged to strengthen monitoring, evaluation, and reporting systems for the ESF, but the guidance and tools for this are still in development. The Environmental and Social Commitment Plan requires the borrower to monitor E&S performance and provide reports on project implementation at least annually, but instruments for this have not been finalized and so cannot be assessed by IEG. The World Bank has not yet sought to integrate safeguards indicators within the results frameworks of projects. The guidelines on monitoring and evaluating safeguards performance in supervision and completion reports, which the World Bank had pledged to complete by FY11 as part of investment lending reform, have not yet materialized. The 2012 policy update by IFC successfully addressed some of the recommendations in IEG’s 2010 safeguards evaluation related to climate change, supply chains, and financial intermediary (FI) projects. First, the updated policy commits IFC to work with the private sector to reduce greenhouse gas emissions and support climate adaptation opportunities. Second, it extends protection for supply chain workers, requiring its clients to ensure that primary suppliers within the supply chain are taking necessary steps to prevent or correct life-threatening situations (appendix C). Third, it strengthens oversight of FI projects by introducing risk-based categories based on the financial exposure of expected portfolios and the potential for risk mitigation. However, implementation of these changes can be enhanced. First, assessment of climate change aspects in project appraisal and monitoring has improved,38 but measurement of greenhouse gas emissions needs strengthening. Tools created for estimating emissions have not always been applied effectively outside of the power sector.39 In projects with high greenhouse gas emissions, IFC did not always identify alternatives to carbon-intensive solutions, and technically and financially feasible options to reduce project-related greenhouse gas emissions have not been consistently applied. Second, supervision of FI clients’ and subprojects’ compliance with IFC Performance Standards needs improvement.40 High-risk FI projects have been correctly identified, but IFC’s assessments of Performance Standard compliance do not provide consistent and reliable information about compliance (appendix C).41 Supply chain issues are not adequately reported and addressed in FI subprojects.42 This observation is consistent with the Compliance Adviser / Ombudsman’s 2013 and 2017 reports on the sustainability of IFC’s financial sector projects.43 18 Results and Performance of the World Bank Group 2017 | Chapter 1 Actions taken by IFC on other IEG findings have been partial. IFC developed a new set of tools to identify and manage E&S risks in agrocommodity production but not for other sectors (appendix C). IEG’s recommendation to use third-party monitoring to assess the adequacy of E&S mitigation measures has been adopted selectively only for some high-risk projects. IFC integrated environmental advisory services for Clean technology and energy efficiency in its Sustainability Framework and investment products, and proposed these products during project scoping, but client uptake remains weak because of low demand, especially for FIs and SMEs.44 Disclosure of IFC project-level E&S performance during implementation is still inadequate. E&S risks and mitigation measures are adequately disclosed to the public during the appraisal phase, but performance and progress on action plans are not adequately disclosed during implementation. Although the policy update commits IFC to encourage governments and corporations to make extractive industry contracts public, this is not routinely done for project-level E&S reports, despite the intent to release them after redacting proprietary information. A lack of disclosure undermines policy commitments for informed stakeholder engagement. For MIGA, harmonizing E&S categorization and boosting MIGA’s E&S capacity improved, but shortcomings remain in monitoring and disclosure. MIGA increased budget, staff, and procedures for E&S due diligence, monitoring, and evaluation (appendix C). The introduction and implementation of Environmental and Social Review Policy, Environmental and Social Risk Rating, and Knowledge Gap concepts in 2013 reduced past deficiencies in E&S performance and information. The E&S staff of IFC and MIGA ramped up their coordination due to the rising number of joint, cofinanced projects (World Bank 2016g).45 However, weaknesses remain relating to (i) the lack of monitoring information on the subprojects of financial intermediaries insured by MIGA and (ii) inadequate disclosure of project-level E&S information at supervision. The IFC and World Bank policy updates aligned their policy frameworks more but created some new disparities. First, the World Bank replaced the three-tiered environmental classification system with a four-tiered system based on E&S risks, which prevents harmonization of project classification. Second, for IFC, risk classification is determined at appraisal and remains fixed, whereas the ESF adopted a dynamic risk categorization approach. IEG’s safeguards evaluation highlighted the need for harmonization of thematic coverage and guidance across the Bank Group (World Bank 2010). The new discrepancies due to the ESF may need to be reconciled to ensure harmonized treatment.46 Moreover, differences persist in the management of E&S risks across financing instruments, including policy lending. Environmental and Social Risk Management in World Bank Operations Reporting on whether safeguards were complied with in World Bank projects lacks clarity. Roughly a quarter of completion reports for World Bank investment projects do not clearly report on whether safeguards have been complied with or not (table 1.3). Harmonized guidelines for evaluation require that project completion reports comment on safeguard requirements and compliance with them.47 RAP17 assessed safeguards-related reporting in all 314 investment lending projects validated by IEG over July 2015–June 2017.48 For 41 of these projects with serious potential for adverse environmental Independent Evaluation Group | World Bank Group 19 TABLE 1.3 |M  ost Project Completion Reports Record Safeguard Triggers and Compliance, but Some Do Not (percent) All Projects Category A Projects Share of Projects for Which At Least One At Least One the Completion Report All Safeguards Safeguard All Safeguards Safeguard Accurately recorded which 77 87 71 90 safeguards had been triggered Clearly stated whether 73 82 68 80 safeguards had been complied with Note: Results from investment lending projects covered by Independent Evaluation Group Implementation Completion and Results Report Reviews in FY16–17. Category A projects are those with serious potential for adverse environmental effects. All projects, n = 314; category A projects, n = 41. effects (category A), supervision reports indicated that 35 projects complied with safeguards and 6 did not, and completion reports disclosed this noncompliance in all 6 cases. Management of E&S risks for World Bank DPF could be improved. DPF operations are covered by their own requirements under the DPF World Bank policy rather than by the safeguard policies or the new ESF.49 An IEG learning product covering 2005–14 found that most DPF policy actions did not pose E&S risks and that good-practice examples of risk management existed (World Bank 2015e). However, IEG found evidence of significant underidentification of risks. This occurred because of vagueness in World Bank policy and guidance on E&S effects in DPF, insufficient coverage of E&S effects in DPF review processes, and incentives that did not favor risk identification. The World Bank’s DPF retrospective also found that project documentation was adequate in only one-third of cases, and that in many cases documents did not provide sufficient information to allow for an assessment of likely significant effects (World Bank 2015a). The retrospective recommended that E&S management be improved. Since publication of these reports, the World Bank has made progress on improving review processes but has yet to address reporting in completion reports.50 An IEG early assessment of Program-for-Results lending recommended that the World Bank minimize the overly cautious interpretation of the high E&S risks exclusion. IEG’s early assessment on Program-for-Results found that E&S assessments were of reasonable quality but lacking in the extent to which capacity building recommendations were followed up in action plans and technical assistance,51 the adequacy of outreach and consultation with poor and vulnerable beneficiary groups, and the absence of monitoring indicators and reporting on safeguards implementation in the results frameworks of the operation (World Bank 2016c). Environmental and Social Risk Management for IFC Projects For IFC Performance Standards addressing environmental sustainability, success ratings have been highest for standards on Biodiversity Conservation (PS6), followed by Pollution Prevention and 20 Results and Performance of the World Bank Group 2017 | Chapter 1 Abatement (PS3) and Assessment and Management of E&S Risks and Impacts (PS1). IEG conducted an analysis of these environmental Performance Standard ratings and subratings for all real sector projects (that is, non-FI) evaluated from calendar year (CY)12 to CY16 (appendix B). No trend surfaced in these ratings, with variation from year to year and across projects depending on the project context, industry, and specific Performance Standard subcomponents triggered. Environmental performance of real sector projects improved after IFC’s intervention, although one- fifth remain noncompliant. The share of projects with satisfactory or excellent ratings for environment- related Performance Standards was 69 percent at appraisal and 81 percent at evaluation, indicating that performance improved with IFC’s intervention (appendix C). However, nearly a fifth of projects were not in compliance at evaluation because the relevant environmental-related Performance Standards and E&S Action Plan requirements needed to be fully implemented five years after commitment. IFC projects’ compliance varies with the three Performance Standards across environmental categories, investment sizes, and between greenfield (new) and brownfield (existing) projects. While both category A and category B projects saw improvements in every Performance Standard between appraisal and evaluation, category A projects (with “potential significant adverse impacts”) have slightly higher environmental success rates owing to their more experienced sponsors (appendix C).52 A comparison of greenfield and brownfield projects showed that, except for greenfield projects’ lower PS1 ratings at approval, all environmental-related PS3 and PS6 subratings performed better at evaluation in greenfield than in brownfield projects. In many greenfield projects, environmental considerations were already integrated in the project design stage through technology choice, and their implementation is less challenging than in brownfield projects where retrofitting existing assets can be costlier. Finally, projects involving large-size IFC investments showed greater improvement in success rates between appraisal and evaluation than small- and medium-size investment projects did. Meeting PS3 requirements proved particularly challenging for projects involving small-size IFC investments.53 Environmental and Social Risk Management for MIGA Projects Environmental performance of projects with MIGA guarantees improved between underwriting and evaluation. IEG’s review of 34 real sector guarantee projects evaluated from FY12 to FY16 found that average success rates (projects rated satisfactory and excellent) for the three environmental-related Performance Standards were 75 percent at underwriting and 82 percent at evaluation (appendix C). Success ratings were highest for Biodiversity Conservation (PS6) followed by Pollution Prevention and Abatement (PS3) and Assessment and Management of E&S Risks and Impacts (PS1). The biggest improvement between underwriting and evaluation was for PS1, where initial performance had been weakest.54 1  he Sustainable Development Goals emphasize key aspects of environmental sustainability, with specific goals on T clean water, climate action, life below water, and life on land, and important targets such as reducing deaths and illnesses from pollution-related diseases. The World Bank Group plays an important role in promoting environmental Independent Evaluation Group | World Bank Group 21 sustainability at the global level, including at international climate change meetings (primarily the United Nations Council of Parties). 2  ther World Bank Group reports use similar but slightly different definitions. In its flagship report introducing the concept O of green growth, Inclusive Green Growth: The Pathway to Sustainable Development, the World Bank defines green growth as that which “is efficient in its use of natural resources, clean in that it minimizes pollution and environmental impacts, and resilient in that it accounts for natural hazards and the role of environmental management and natural capital in preventing disasters” (World Bank 2012a, 2). The World Bank’s analytical and partnership work on natural capital accounting emphasizes that natural capital includes both easily recognizable and measurable resources such as minerals and energy, timber, agricultural land, fisheries, and water, but also “invisible” ecosystem aspects, including air and water filtration, flood protection, carbon storage, pollination for crops, and habitat for fisheries and wildlife (World Bank 2012d). 3  roject rating systems are driven largely by the primary development goal and so are often poor at capturing P environmental effects; environmental results are typically measured only in those projects that focus on environmental objectives. 4  irect effects include benefits from improvements in natural resource management, from infrastructure improvements D that help to displace polluting activities, or from improved environmental regulation, practices, and behaviors. Indirect effects may come through activities such as pricing or subsidy reform, improvements in industrial technology and efficiency, improvements in natural resource governance, and many others. 5 T  he Strategic Framework on Development and Climate Change had specific targets, including increased financing for energy efficiency and renewable energy (30 percent increases per year), the share of energy projects with low carbon (50 percent), establishment of the Climate Investment Funds ($6 billion), and incorporation of climate change in country strategies in highly vulnerable countries (at least 10; World Bank 2008). The 2016 World Bank Group Climate Change Action Plan includes targets for the share of World Bank Group financing for activities relating to climate change, from 21 percent to 28 percent by 2020 and with a target of $29 billion per year by 2020. The 2012 World Bank Group strategy proposes a results framework including actions, output and outcome results, and indicators, but has no targets for these values (World Bank 2012, table 7.1). 6 T  he World Bank Group does not have a clear definition of environmental mainstreaming, but the 2002 World Bank Environment strategy emphasizes “the importance of continuing our efforts to work with clients to integrate, or mainstream, environment into investments, programs, sector strategies, and policy dialogue mindful of the institutional requirements and capacity constraints” (World Bank 2002, 17). The 2012 strategy uses the concept of mainstreaming repeatedly to mean the integration of environment into sectoral activities but does not define it specifically. 7  itigation benefits most commonly result from energy efficiency, renewable energy, upgrades in electricity transmission M and distribution leading to reduction in technical losses, and power tariff reform. Other common types of mitigation activities are in transport and urban development and include upgrading public transportation systems, building urban rail, addressing urban mobility issues, and improving land use planning in urban areas. 8  ir pollution abatement benefits result mostly from activities in energy efficiency, public transportation, urban mobility, A and solid waste management (for example, landfill construction). 9 T  he share of components supporting biodiversity conservation dropped by 54 percent. The share supporting water management dropped by 40 percent. 10  ater management was equally divided between water resource management and irrigation in the first period, but in W the second period, irrigation benefits are less common. Increases in fisheries management have come from projects aimed at improving the sustainability of fishing practices and more effective governance of the sector. Forestry activities evolved from being often implemented in the context of protected areas and biodiversity conservation to focusing more on reforestation and deforestation prevention activities. This analysis also does not capture forest manangement activities through carbon finance instruments. 11  he Sustainable Development Practice Group includes the Agriculture; Energy and Extractives; Environment and T Natural Resources; Social, Urban, Rural, and Resilience; Transport and ICT; and Water Global Practices. The Human 22 Results and Performance of the World Bank Group 2017 | Chapter 1 Development Practice Group includes the Education; Health, Nutrition, and Population; and Social Protection Global Practices. The Economics and Finance Practice Group includes the Finance and Markets; Governance; Macroeconomics and Fiscal Management; Poverty and Equity; and Trade and Competitiveness Global Practices. 12  xamples include a mine safety project aiming at tackling occupational lung diseases (with the components aimed at E reducing industrial air pollution and environmental management of extractive industries), small-scale solid waste and wastewater treatment in community-driven development projects, a project involving citizen engagement in building environmental and social management capacity for infrastructure resiliency, and another project establishing a productive and disaster-responsive safety net. Small-scale renewable energy activities were also included in a handful of health, education, and social protection projects. 13  nvironmental activities in investment policy financing projects include a green logistics framework that involves E environmentally sustainable design of warehousing, container yards, and truck parks; technical assistance for a water resources management strategic plan; and several disaster- and climate-related risk insurance activities. Development Policy Financing operations often include environmental actions and pillars, for example, “removing constraints to expansion of renewable energy,” minimizing environmental and social costs of irrigation schemes, electricity tariff adjustments, development of environmentally sustainable tourism, and fiscal policy reforms aimed to “facilitate a transition towards a green growth strategy.” 14  limate change mitigation projects were most commonly for renewable energy and typically included construction C of solar photovoltaic units, smaller-scale run of the river hydropower facilities, and wind-powered turbines. Support for renewable energy is present across all industry groups, supporting specific power projects, captive power for industry, or targeted credit lines for financial intermediaries. Most air pollution abatement effects come as cobenefits from activities that support climate change mitigation. Benefits typically arise from the switch to cleaner fuels and more energy-efficient machinery and equipment by factories, municipalities, or transportation networks. The International Finance Corporation has worked on privatization, capital expenditure, and business expansion plans with a number of parastatals, such as utility and wastewater treatment companies. 15 In the sample there was a small decline in the share of Financial Institutions Group (FIG) projects with potential  environmental benefits, but this was likely due to sampling error. For FIG, climate business coding covering the full FIG population by the International Finance Corporation is potentially more accurate, as environmental sustainability in FIG projects comes primarily from climate change benefits. The climate business coding shows 9.5 percent of FY08–10 FIG projects had climate change benefits, and 15.9 percent of FY15–17 FIG projects did. 16  he majority of infrastructure projects are a result of supporting demand for energy in countries, retrofitting existing T facilities, or modernizing and expanding metrorail systems in major cities. Oil and gas interventions with likely environmental benefits have come from supporting cleaner production practices in parternship with new owners, or from reduction in gas flaring. 17  anufacturing projects with environmental benefits include reductions in waste from new equipment and M production methods, or from efficiency improvements in energy and water consumption. Agribusiness activities with environmental benefits included the purchase of two leading global agricultural commodity traders, with the potential to influence and facilitate sustainable sourcing and trading of important agricultural commodities in both origination and destination markets. In forestry, financing has included support for integrating plantation forestry with community agriculture and biomass energy generation, potentially reducing pressure on forest degradation. 18  here is comparatively more demand for energy efficiency and renewable energy (smaller scale hydro and solar) T financing as companies borrow to purchase new equipment or change operational processes to become more energy efficient. Savings in utility bills are more quickly realized or evident compared with benefits from green activities, which tend to be longer term, more difficult to track, and at times more subjective. 19 I dentification of guarantee projects followed Multilateral Investment Guarantee Agency’s reporting in their audited annual reports, and each project has a unique official approval document, either a Board of Executive Directors document or an Underwriting Checklist for Small Investment Program projects. The review considered all 26 projects Independent Evaluation Group | World Bank Group 23 for which Board documents or Underwriting Checklists could be located for FY08–10, and a sample of 36 projects in FY15–17. 20  nvironment outcomes and objectives were categorized using the classification framework agreed with the E management of the World Bank Group institutions, which divided environmental sustainability into three areas (Clean, Green, and Resilient) and into more specific subsectors for each area. Outcomes and objectives are used as the unit of analysis for this review, reflecting the changes in results framework format when the World Bank Group moved from Country Assistance Strategies (CASs) /Country Partnership Strategies (CPSs) to Country Partnership Frameworks (CPFs) in 2014. The CAS/CPS result frameworks gave the outcomes the World Bank Group aimed to achieve, whereas CPF result frameworks provided objectives. The methodology used is described in more detail in appendix B. 21  atin America and the Caribbean had the highest share of country strategies with at least one environmental objective L or outcome at 93 percent, while Africa had the lowest at 55 percent over FY08–17. The share of the remaining Regions fell in between: East Asia and Pacific Region at 80 percent, Europe and Central Asia Region at 75 percent, the Middle East and North Africa Region at 71 percent, and the South Asia Region at 86 percent. 22 T  he Clean area was a major focus in the Europe and Central Asia and Middle East and North Africa Regions. Climate change mitigation, water, and soil pollution abatement, waste management, and other Clean subsectors accounted for 60 percent of the environmental objectives and outcomes in Europe and Central Asia and just over half (53 percent) in Middle East and North Africa during FY08–17. The Green area was prominent in Africa, where landscapes (18 percent), forestry (11 percent), biodiversity conservation (10 percent), and other Green subsectors (15 percent) accounted for over half (56 percent) of the environmental objectives and outcomes. The Green and Resilient areas were priorities in Latin America and the Caribbean (81 percent combined) and East Asia and Pacific (75 percent combined), while the South Asia Region saw the Clean and Resilient areas (79 percent combined) as its priorities. These priorities were consistent with the Regional challenges noted in the World Bank Group’s environmental strategy. 23 A  frica gave priority to better natural resource management and expanded protected area management; Europe and Central Asia tackled the challenges of ensuring energy security while providing cleaner energy and managing legacy, current, and future pollution; Middle East and North Africa faced the threat of increasing land-, air-, and marine-based pollution; and most Regions included populations and areas vulnerable to sea level rise and more intense weather, floods, and drought (World Bank 2012). 24  he evaluation notes that there was limited attention to climate change in the Systematic Country Diagnostics in some T vulnerable countries (such as Bolivia—although it flagged climate change and natural disasters among knowledge gaps—and Mali). 25  ier 1 indicators are measured only at the global level based on global datasets, whereas tier 2 indicators are T generated from World Bank projects and tier 3 indicators are at the World Bank Group corporate level. 26 G  reenhouse gas accounting measures are being implemented for the energy sector (transmission and distribution, thermal power generation, renewable energy, energy efficiency, and energy access), forests (afforestation/ reforestation, sustainable forest management/fire control), agriculture/landscapes (land use change, crop production, grassland, livestock, land degradation, wetlands, fertilizers, irrigated crops, agribusiness value chain, fisheries and aquaculture), transport (roads, rail, waterways and urban transport), water (water treatment plants, wastewater treatment plants, desalinization plants, wastewater reuse, multipurpose water reservoirs, and irrigated rice), and waste management. An important area left uncovered is urban land use changes. 27  his gap could be meaningful because the changes in the portfolio composition are consistent with the presence of T corporate indicators. Climate change commitments have been corporate targets, and there have been increases in commitments for climate change. Water supply and sanitation has had a corporate target, which could be related to decreases in support for other water sector engagements (water resource management and irrigation). 28 T  he World Bank Group Corporate Scorecard is the “apex of the institution’s accountability framework” (World Bank Group 2017b, 5). Interviews with World Bank staff suggest that Scorecard indicators may be useful for advocacy and education purposes. However, most tier 2 environmental indicators do not have targets, which makes it difficult to meaningfully judge success for performance purposes. Tier 2 indicators are drawn from projects, so they depend on the 24 Results and Performance of the World Bank Group 2017 | Chapter 1 existence of a project in that sector and country, thus changes between years may reflect composition changes rather than performance results. Some indicators rely on data that is not collected annually, so year-on-year changes cannot be observed. No breakdown by country appears in the Scorecard or associated materials—only the global aggregate is provided. This hinders the ability to use the Scorecard to focus attention on countries where the need is highest. 29 T  he objective with the single indicator was, “Improved water resources infrastructure for climate change” (Bangladesh, FY16). The objective with 43 instruments was “Support the achievement of Brazil’s nationally determined contribution with a particular focus on land use.” 30 There was a median of two indicators per objective, intended to cover a median of six instruments per objective.  31 ndependent Evaluation Group interviews with World Bank staff suggested that this simplification greatly improved the I system because the core sector indicators were overwhelming in number, difficult to implement in practice, and tilted the balance too far toward a focus on aggregable data rather than indicators that met the need of the project results framework. 32  his has been particularly common for Green and forest-related projects. For forest projects, proxy indicators to T demonstrate interim outcomes have been developed by the World Bank but are seldom used in projects. Instead, corporate indicators emphasize the size of the area where activities were taking place—which is not a good measure of assessing whether desired changes in behavior, vegetation, or ecosystem services have occurred. Indicators on the number of beneficiaries and number of female beneficiaries have also often been emphasized. This has occurred in the presence of other institutional pressures (though not formal requirements) to limit the overall number of outcome indicators. So when all these corporate indicators are included, there is little space remaining for indicators needed to assess the specific results of the project. 33 n addition to this self-reporting through annual monitoring reports, the International Finance Corporation validates I information including through Supervision Site Visits and reviews of the monitoring reports and action plan compliance status reports and establishes the client’s degree of compliance with Environmental and Social Conditions of Disbursement prior to all disbursements. Projects are supervised at least once a year. 34  Corporate and country results tracking systems are substantially built on the foundation of project indicators, so limitations arising from the project cycle affect all three. 35 Future work could look more explicitly at the coverage of World Bank Group analytical work and research on  environmental sustainability to identify specific knowledge gaps that could be relevant for the project portfolio and to explore the institutional incentives and barriers for assessing longer-term results. 36  he World Bank began its safeguard review in July 2012. After a prolonged preparation and consultation period, a new T Environmental and Social Framework was approved on August 4, 2016. Guidance materials are still under preparation; training of environmental and social staff, task team leaders, and clients is planned for FY18; and the Environmental and Social Framework is expected to be launched in the latter half of 2018. The International Finance Corporation (IFC) updated its Sustainability Framework and Performance Standards in 2011, and these became effective on January 1, 2012. The Multilateral Investment Guarantee Agency adopted the revised Sustainability Framework in 2013. IFC’s policy update was ongoing when the Independent Evaluation Group undertook its evaluation. In accordance with Committee on Development Effectiveness guidance, IFC extended the consultation and review period to integrate the evaluation’s recommendations. Like the World Bank, IFC also undertook some actions to strengthen oversight of Performance Standards. 37 The World Bank also issued new guidance to address the impact of labor influx on host populations.  38 nternational Finance Corporation lowered the threshold above which clients must report their greenhouse gas I emissions, meaning more clients are reporting greenhouse gas emissions. 39  he International Finance Corporation (IFC) is using its Carbon Emission Estimation Tool (CEET) to calculate project- T specific emissions. Independent Evaluation Group’s desk review of a sample of IFC projects and interviews with select IFC staff noted that the CEET has not always been applied effectively outside of the power sector, and in Independent Evaluation Group | World Bank Group 25 many projects, the technical assumptions used for calculating greenhouse gas emissions were not applicable to the subindustry group, leading to potentially unreliable data about quantities of greenhouse gas emission reduction. 40  ategorization and supervision requirements for financial intermediary (FI) projects of the International Finance C Corporation (IFC) in the 2012 Environmental and Social Policy (ESP) and 2012 Environmental and Social Review Procedure (ESRP) use a risk-based approach considered more robust and detailed compared with the 2006 Sustainability Framework. IFC’s 2012 ESP classifies FIs into three environmental and social categories: (i) FI-1 with highest risks; (ii) FI-2 with limited risks; and (iii) FI-3 with minimal risks. The environmental and social requirements are described in the 2012 ESP and in the companion “Interpretation Note on Financial Intermediaries.” For project finance and corporate loans with tenor of not less than 36 months and funding-defined assets as part of a project amounting to at least $10 million of total capital cost, compliance with all PSs is required; for other corporate loans provided to a single client exceeding $5 million, on an aggregated basis over a period of 36 months, the FI should require the client’s Environmental and Social Monitoring Systems and labor practices to be consistent with Performance Standards 1 and 2. IFC’s supervision and site visit requirements are defined in a detailed manner. 41 In annual environmental performance reports.  42  or example, loans to agroprocessing companies did not always adequately address supply chain issues. The F International Finance Corporation developed a handbook for managing agrocommodity supply chains, but it seems that this was not given to FIs or the International Finance Corporation provided guidance to help its FI clients manage supply-chain issues in their portfolios. 43  he Compliance Adviser / Ombudsman 2013 and 2017 reports noted that the International Finance Corporation’s T reports on financial intermediary project compliance lack evidence on compliance with Performance Standards requirements, including supply chains in large high-risk corporate loans and project finance subprojects. 44  easons for weak client demand vary from proper sequencing of International Finance Corporation (IFC) advisory staff R involvement to client unwillingness to pay for IFC Advisory Services, or in some cases, IFC’s inability to offer clients a customized advisory package. In some of the projects reviewed, environmental Advisory Services components were not required in greenfield projects because Clean technology was applied at the outset or the management already had a strong competency and well-functioning environmental and social management system. Often, International Finance Corporation clients had already contracted environmental consultants to develop their environmental and social management system and improve energy efficiency. In one project reviewed by the Independent Evaluation Group, IFC proposed a grant for improving health and safety training and performance, but the client refused. 45 T  he 2011 expanded International Finance Corporation (IFC) and Multilateral Investment Guarantee Agency (MIGA) business development agreement formalized an arrangement wherein IFC takes the lead in processing, appraising, and supervising joint projects (including environmental and social aspects) and shares the information with MIGA. In turn, MIGA pays a IFC a fee for its support activities. 46 T  he Independent Evaluation Group has no evidence on whether or not harmonization is superior or more effective. However, confusion over risk classification could occur with different systems. In addition, harmonization across institutions has been a corporate goal of the World Bank Group in recent years. For example, the Bank Group’s approach to Country Partnership Frameworks is built around a joint strategy across the Bank Group (see World Bank Group 2014). The Environmental and Social Framework encourages a common approach in which the World Bank jointly finances projects with other multilateral and bilateral agencies. Elements of harmonization are entailed 26 Results and Performance of the World Bank Group 2017 | Chapter 1 in Operational Policy 10, under which the World Bank can apply International Finance Corporation Performance Standards when financing private sector development. 47  roject Completion Reports should “summarize key safeguard and fiduciary issues in the operation, compliance with P the World Bank policy and procedural requirements, and any problems that arose and their resolution, as applicable. Record any significant deviations or waivers from the World Bank safeguards/fiduciary policies and procedures.” 48 This analysis updates and expands the assessment carried out in Results and Performance of the World Bank Group  2016, bringing in a second year of data. 49 T  he World Bank reframed its Operational Policy 8.60 on development policy financing as “Bank Policy, Development Policy Financing” in August 2017. These policies require that the World Bank identify specific policies supported by a policy lending operation that are likely to have significant poverty and social or environmental effects, to assess the borrower’s capacity for managing any such effects, and to identify how any gaps in this capacity will be addressed. 50  he World Bank established a new environmental and social effects screening table for all program documents, which T can help to make clearer what effects have been identified. A review function for potential environmental effects has been established in Operations Policy and Country Services. Updated guidance on environmental effects is being prepared and tested but is not yet published. World Bank management reports that changes have been made to internal training in the Development Policy Financing Academy. No changes have been made yet in reporting on effects in completion reports, but an effort to reform the completion report for development policy financing has just been initiated. 51  he relevant operational policy for Program-for-Results requires the World Bank to assess the degree to which T program systems manage environmental and social impacts of the program and identify actions, if needed, to enhance these systems. The policy also requires that high-risk activities that would be assigned category A under investment lending safeguards be excluded from any Program-for-Results projects. 52 “  Success rate” is defined as the ratio of high ratings (that is, excellent and satisfactory) to total number of projects rated. Category A projects have higher share of success ratings because these projects generally have more experienced sponsors who have developed adequate environmental and social monitoring sytems at the appraisal stage. 53 Defined as those with less than $4.2 million in net commitment.  54 D  ue to the small number of evaluated Multilateral Investment Guarantee Agency projects, the Independent Evaluation Group’s analysis by environmental risk categories, subindicators, or by sectors did not yield meaningful results. Independent Evaluation Group | World Bank Group 27 2 highlights Performance 1 The World Bank exceeded the corporate target of World Bank for project outcome ratings by volume but fell Group below it by number of projects during FY14–16. Operations 2 One element of the World Bank performance rating (quality of supervision) showed improvement and exceeded the corporate target by volume, while the other (quality at entry) remained below the corporate target. 3 Outcome ratings have improved in country contexts traditionally seen as challenging, although both outcome and World Bank performance ratings were lower in these settings than in other country contexts. 4 Among the most important factors associated with outcome ratings for World Bank environment projects were the measurement of outcomes and the presence of economic analysis. 5 Development outcome ratings of International Finance Corporation (IFC) investment projects decreased from calendar year (CY)11–13 to x 28 CY14-16, as did ratings for IFC front-end work quality and IFC role and contribution. Development effectiveness ratings of IFC advisory projects also decreased. A joint Independent Evaluation Group (IEG)-IFC working group delved into the root causes of the decreases in ratings, and IFC started implementing initiatives to address the identified shortcomings, but it is too early to tell the outcomes of these initiatives. 6 Development outcome ratings of Multilateral Investment Guarantee Agency (MIGA) projects remained unchanged for FY11–16, but MIGA’s performance could be improved by addressing shortcomings relating to assessment of risk to development outcome, data capture, and documentation. 7 Many factors of performance of World Bank, IFC, and MIGA projects are fully within each institution’s control. These include quality of project design, quality of supervision, quality of the project’s results chain, analytical work, and incorporation of lessons learned. Independent Evaluation Group | World Bank Group 29 Introduction T H I S C H A P T E R discusses the performance of projects across four Bank Group institutions— the World Bank (the International Bank for Reconstruction and the Development and International Development Association), IFC, and MIGA—and presents country strategy outcomes. This report does not provide aggregate performance ratings for the Bank Group because no direct comparisons can be made across the institutions.1 World Bank Project Ratings Trends Outcome ratings for World Bank projects that closed during FY14–16 were above the corporate target by volume but below it by number of projects.2 Ratings trends reported here are based on the World Bank’s self-evaluations and IEG’s validation of them (through Implementation Completion and Results Report [ICR] Reviews). An ICR reform in July 2017 did not cover projects included in this year’s ratings (see appendix D for details). The share of projects with outcome ratings of moderately satisfactory or above was 73 percent for projects closed during FY14–16, below the corporate target of 75 percent. By volume, the percent of projects with outcome ratings of moderately satisfactory or above was 84 percent for FY14–16, above the corporate target of 80 percent (figure 2.1). For 202 projects closed in FY16 and validated by IEG, 71 percent had outcome ratings of moderately satisfactory (80 percent by volume). For 45 projects closed in FY17 and validated by IEG as of October 2, 2017, 87 percent had outcome ratings of moderately satisfactory or above (94 percent by volume); but an important caveat to the FY17 figures is that these 45 projects are not considered representative.3 See appendix D for definitions, methodology, and coverage information. From FY11–13 to FY14–16, outcome ratings went from 70 percent to 73 percent by number of projects and from 79 percent to 84 percent by volume.4 Figure 2.1 and appendix E provide details. Projects with a very large volume masked the rating patterns that characterized the vast majority of projects, which had a relatively smaller volume. If the 28 projects with volume of $800 million or more are excluded, ratings (by volume) in both periods decrease: instead of 79 percent in FY11–13 and 84 percent in FY14–16, outcome ratings were 76 percent and 80 percent, respectively— barely meeting the corporate target. These 28 projects accounted for 2 percent of projects and 21 percent of volume during FY11–16. Most of them were in countries where the International Bank for Reconstruction and Development (IBRD) has the highest credit exposure. Appendix D provides a list of the very large projects in these two periods, including their country, Global Practice, and ratings information. Appendix F explains the methodology and findings of the analysis of distribution of volume. Ex ante residual risk was associated with lower outcome ratings. A separate analysis of ex ante risk ratings data found that although there was no significant correlation between ex ante initial risk and 30 Results and Performance of the World Bank Group 2017 | Chapter 2 Figure 2.1 |W  orld Bank Project Outcome Ratings Were Essentially Stable by Number of Projects; By Volume, Some Improvement Over Time Was Associated With A Few Large Projects  orld Bank lending projects with outcome rated moderately satisfactory or above, percentage by number of a. W projects 100 80 60 Percent 40 20 0 –20 –40 2) 8) 0) 8) 0) 8) 1) 3) 9) 2) 6) 8) 7) 3) 1 4 9 8 2 5 2 5 3 4 9 4 3 1 =8 =8 =8 =8 =8 =7 =7 =7 =8 =9 =9 =9 =8 =7 (N (N (N (N (N (N (N (N (N (N (N (N (N (N 3 4 5 6 7 8 9 0 1 2 3 4 5 6 –0 –0 –0 –0 –0 –0 –0 –1 –1 –1 –1 –1 –1 –1 01 02 03 04 05 06 07 08 09 10 11 12 13 14 FY FY FY FY FY FY FY FY FY FY FY FY FY FY  orld Bank lending projects with outcome rated moderately satisfactory or above, percentage by volume b. W 100 80 60 Percent 40 20 0 –20 –40 7) 6) 4) 2) 8) 8) 3) 2) 8) 8) 4) 5) 2) 0) =6 =6 =6 =6 =5 =5 =5 =6 =7 =8 =9 =8 =8 =7 (V (V (V (V (V (V (V (V (V (V (V (V (V (V 3 4 5 6 7 8 9 0 1 2 3 4 5 6 –0 –0 –0 –0 –0 –0 –0 –1 –1 –1 –1 –1 –1 –1 01 02 03 04 05 06 07 08 09 10 11 12 13 14 FY FY FY FY FY FY FY FY FY FY FY FY FY FY Highly satisfactory Moderately unsatisfactory Satisfactory Unsatisfactory Moderately satisfactory Highly unsatisfactory Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: The Independent Evaluation Group rated 713 projects that closed in FY14–16 with a total volume (net commitment) of $70 billion, among which 635 were IPF projects with a total volume of $53 billion, and 70 were DPF projects with volume totaling $16 billion. FY16 data excludes two projects in the Republic of Yemen (see appendix D). DPF = development policy financing; IPF= investment project financing; MS+ = moderately satisfactory or above; V= volume in $, billions. project outcome rating, higher ex ante residual risk was associated with lower outcome ratings, and this correlation was significant, which supports the idea that taking steps up front to reduce and mitigate risk is important, particularly for high-risk projects. Appendix G provides details on this analysis. Independent Evaluation Group | World Bank Group 31 FIGURE 2.2 |B  y Volume, Performance Ratings for World Bank Projects Improved, Whereas by Number of Projects, They Remained Stable  orld Bank lending projects with World Bank performance rated moderately satisfactory or above, a. W percentage by number of projects 100 80 60 Percent 40 20 0 –20 –40 ) ) ) ) ) ) ) ) ) ) ) ) ) 4) 50 22 63 08 05 35 71 34 61 55 53 45 14 01 =8 =8 =8 =9 =9 =8 =7 =7 =7 =9 =9 =8 =7 =1 (N (N (N (N (N (N (N (N (N (N (N (N (N (N 1 3 4 5 6 7 8 9 0 2 4 5 6 –1 –0 –0 –0 –0 –0 –0 –0 –1 –1 –1 –1 –1 3 –1 09 01 02 03 04 05 06 07 08 10 12 13 14 11 FY FY FY FY FY FY FY FY FY FY FY FY FY FY  orld Bank lending projects with World Bank performance rated moderately satisfactory or b. W above, percentage by volume 100 80 60 Percent 40 20 0 –20 –40 9) 7) 2) 1) 7) 3) 0) 7) 6) 4) 2) 8) 8) 3) =7 =9 =9 =8 =8 =5 =5 =5 =6 =7 =6 =6 =6 =6 (V (V (V (V (V (V (V (V (V (V (V (V (V (V 1 3 2 4 5 3 4 5 6 7 8 9 0 6 –1 –1 –1 –1 –1 –0 –0 –0 –0 –0 –0 –0 –1 –1 09 11 08 10 12 13 01 02 03 04 05 06 07 14 FY FY FY FY FY FY FY FY FY FY FY FY FY FY Highly satisfactory Moderately unsatisfactory Satisfactory Unsatisfactory Moderately satisfactory Highly unsatisfactory Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: The volume-weighted percentage of projects rated moderately satisfactory or above on World Bank performance improved from FY11–13 to FY14–16, while unweighted percentage remained essentially stable. Gray box: For projects whose Implementation Completion and Result Report (ICR) Review was initiated before July 1, 2006, World Bank performance was rated on a 4-point scale (from lowest to highest: highly unsatisfactory, unsatisfactory, satisfactory, highly satisfactory). Projects whose ICR Review was initiated on or after July 1, 2006, World Bank performance was rated on the current 6-point scale (from lowest to highest: highly unsatisfactory, unsatisfactory, moderately unsatisfactory, moderately satisfactory, satisfactory, highly satisfactory); V = volume in $, billions. 32 Results and Performance of the World Bank Group 2017 | Chapter 2 Regarding World Bank performance ratings for FY14–16, the share rated moderately satisfactory or above was 72 percent by number of projects and 84 percent by volume. Figure 2.2 and appendix E provide details. From FY11–13 to FY14–16, World Bank performance ratings went from 70 percent to 72 percent by number of projects and from 80 percent to 84 percent by volume (figure 2.2). When projects of $800 million or above are excluded, however, the ratings by volume in both periods decrease, to 78 percent and 80 percent, respectively.5 An analysis of factors associated with quality of supervision ratings is also provided later in this report. For projects closed in FY16, 69 percent had World Bank performance ratings of moderately satisfactory (81 percent by volume). For the nonrepresentative group of 45 projects closed in FY17 with ratings available as of October 2, 2017 (with the same caveats mentioned above in the discussion of outcome ratings), 87 percent of projects had World Bank performance ratings of moderately satisfactory or above (94 percent by volume). By volume, one element of the World Bank performance rating, quality of supervision, met the corporate target, while the other, quality at entry, did not. The corporate target for ratings of both quality at entry and quality of supervision (which, together, feed into the World Bank performance rating) is 80 percent moderately satisfactory or above, by volume. Whereas the corporate target for the outcome rating could, arguably, create incentives that could make the Bank Group risk-averse if set too high, the World Bank performance rating comprises factors fully within the World Bank’s control, which raises the question of why the World Bank has not opted for higher targets for ratings of quality at entry and quality of supervision. By volume, the share of projects rated moderately satisfactory or above on quality at entry went from 71 percent in FY11–13 to 75 percent in FY14–16, thus not meeting the target. On ratings of quality of supervision, this share went from 81 percent to 89 percent in the same periods, meeting the target. When projects with a volume of $800 million or above are excluded, however, ratings decrease in both periods: For quality at entry ratings, the share changes to 70 percent in FY11–13 and 68 percent in FY14–16. For quality of supervision ratings, the share changes to 81 percent and 86 percent, respectively. Monitoring and evaluation (M&E) quality ratings by number of projects went from 30 percent in FY11–13 to 34 percent in FY14–16 (43 percent to 46 percent by volume). Appendix E provides details. M&E quality is an important aspect in assessment of both quality at entry and quality of supervision, which are the two elements comprised in the World Bank performance rating.6 Outcome ratings have improved in country settings traditionally seen as challenging, although both outcome and World Bank performance ratings were lower in these settings than in other country contexts.7 Among fragility, conflict, and violence (FCV) projects, outcome ratings went from 62 percent in FY11–13 to 66 percent in FY14–16 by number of projects, and from 65 percent to 76 percent by volume (figure 2.3). Regarding World Bank performance ratings, by volume, 75 percent of FCV countries received a rating of moderately satisfactory or above during FY14–16, compared with 85 percent in non-FCV countries. Given that both elements of the World Bank Independent Evaluation Group | World Bank Group 33 FIGURE 2.3 |O  utcome Ratings and World Bank Performance Ratings across IBRD/IDA, IPF/DPF, and FCV Status: A Few Improvements Observed  utcome ratings a. O By number of projects By volume FY11–13 FY14–16 FY11–13 FY14–16 Project Type IDA IBRD IPF DPF Country Type All FCV All non-FCV IDA FCV IDA non-FCV IBRD FCV IBRD non-FCV 0 20 40 60 80 100 0 20 40 60 80 100 Percentage rated MS+ Percentage rated MS+  orld Bank performance ratings b. W By number of projects By volume FY11–13 FY14–16 FY11–13 FY14–16 Project Type IDA IBRD IPF DPF Country Type All FCV All non-FCV IDA FCV IDA non-FCV IBRD FCV IBRD non-FCV 0 20 40 60 80 100 0 20 40 60 80 100 Percentage rated MS+ Percentage rated MS+ Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: Fiscal year is the project’s closing fiscal year. Caveat: IPFs and DPFs differ in structure and function, and the project rating methodology for each differs. Therefore, ratings across these two types of projects are not comparable. For country type, countries eligible for both IDA and IBRD (Blend) financing are included under the IDA category. Blend countries included were: Cabo Verde, Cameroon, Democratic Republic of Congo, Republic of Congo, Dominica, Grenada, Kenya, Moldova, Mongolia, Nigeria, Pakistan, Papua New Guinea, St. Lucia, St. Vincent and the Grenadines, Timor-Leste, Uzbekistan, and Zimbabwe. DPF = development policy financing; FCV = fragility, conflict, and violence; IBRD = International Bank for Reconstruction and Development; IDA = International Bank for Reconstruction and Development; IPF= investment project financing; MS+ = moderately satisfactory or above. performance rating are within the World Bank’s control, and that fragile contexts are particularly challenging, with little room for error, shouldn’t the World Bank aim for better quality at entry and quality of supervision? Among International Development Association (IDA) projects, outcome ratings went from 67 percent to 70 percent across the two periods by number of projects and from 70 percent to 80 percent by volume. Overall, some improvements in ratings for IPFs were seen, particularly by volume. 34 Results and Performance of the World Bank Group 2017 | Chapter 2 Regional Performance Regions varied in their performance over time. Comparing ratings within each Region between FY11– 13 and FY14–16 indicates improvements in the Africa and East Asia and Pacific Regions on ratings for outcome and World Bank performance, both by number of projects and volume of commitments (see appendix E, figure E.11).8 Appendix L provides further analysis of results by Region. Global Practice Performance Ratings within each Global Practice also varied over time. By volume, 10 of 14 Global Practices improved on outcome ratings between the two periods, and a different (and overlapping) group of 10 Global Practices improved on World Bank performance ratings. By contrast, 4 Global Practices did not improve on outcome ratings and 4 did not improve on World Bank performance ratings. Figures E.12 and E.13 in appendix E provide details. Appendix M provides further analysis of results by practice group. Appendix N provides a summary of cross-cutting evaluation findings. Factors Associated with Performance of World Bank Projects Multiple factors are associated with the performance of World Bank projects. Factors Associated with Performance of World Bank Projects Identified Through a Structured Review At the project level, supervision quality and the task team leader’s track record were commonly identified project success factors (specific to World Bank projects) according to a structured review conducted for this report of sources inside and outside the World Bank.9, 10 Other success factors included incorporation of lessons learned and analytical work, the quality of the project’s results chain, and the quality of project design. For community-driven development, the extent and quality of collaboration with local stakeholders was identified as essential to project success. By contrast, factors associated with less successful projects included projects with early warning flags (raised when a project receives an unsatisfactory outcome rating or at least three unsatisfactory ratings on implementation measures in an Implementation Status and Results Report during the first half of implementation; see Ralston 2014, 26, for additional details) or projects of longer duration. Likewise, project outcome had a negative association with the cost and time of both preparation and supervision. At the country level, the most commonly identified success factor was economic growth, while project success is less likely in fragile and conflict-affected situations (FCS) and when civil liberties protections are limited.11 These findings, summarized in table 2.1, are broadly consistent with the quantitative analysis conducted for RAP15 (World Bank 2016e).12 (See appendix H for additional details on the findings and methodology of the structured review.). Aspects of Supervision Associated with World Bank Performance Ratings Factors associated with high performance ratings for World Bank projects included quality of supervision—including safeguards, public financial management, and procurement support—as well as candor in reporting and timeliness or effectiveness of restructuring. To surface factors that could help improve World Bank performance ratings, RAP17 examined ICR Reviews of projects that closed in FY16, comparing two groups: (i) projects with both World Bank performance and Independent Evaluation Group | World Bank Group 35 TABLE 2.1 | Factors Associated with World Bank Project Performance Factors associated with World Bank project performance, by direction and consistency between sources. • = country-level factor;  = political economy factor;  = World Bank–level factor Degree of consistency Direction of the Association between the Factor and Project Performance between sources Negative Neutral Positive Near consensus • Population size • GDP growth • Human capital Alignment of donor   Initial project size & borrower interests T TL track record  Incorporations of  lessons learned & • Price distortions analytical work  Project duration  Results chain Presence of early quality warning flags  Collaboration with  Supervision cost local stakeholders  Design/analysis quality  Selectivity of projects  Supervision quality Some agreement  TTL turnover • GDP per capita • Civil liberties  Preparation cost • Ongoing conflict protections  Supervision time Natural resource  IDA funding source dependence  Provision  Aid dependence of capacity building, training, or technical assistance Little agreement Gap between  • Economic • CPIA rating approval and vulnerability (for Final project size  disbursement example, debt-to-GNI Preparation time  ratio) Source: Independent Evaluation Group structured literature review; see also appendix H. Note: Above table includes only factors cited by at least three sources in the review. CPIA = Country Policy and Institutional Assessment; GDP = gross domestic product; GNI = gross national income; IDA = International Development Association; TTL = task team leader. quality of supervision rated satisfactory or highly satisfactory and (ii) projects with both World Bank performance and quality of supervision rated unsatisfactory or highly unsatisfactory.13 (Quality at entry ratings were not used for this analysis; the intention was to provide information relevant to projects already under implementation.) Aspects of World Bank supervision most frequently highlighted as done well in the projects rated highly satisfactory or satisfactory were as follows: 36 Results and Performance of the World Bank Group 2017 | Chapter 2 „„Adequate  number and frequency of supervision missions (cited in 25 of 29 ICR Reviews)—for example, supervision missions conducted at least twice per year throughout the project period. „„ Adequate safeguards monitoring (20 of 29)—inclusion of safeguards specialists on the supervision team who reviewed and reported on the effective application of safeguards instruments. „„ financial management support (18 of 29)—for example, provision of financial Adequate management training workshops for counterpart staff. „„ Adequate procurement support (17 of 29)—intensive procurement support during lengthy and complicated bidding processes. Aspects of World Bank supervision most frequently highlighted as poorly addressed in the projects rated unsatisfactory and highly unsatisfactory were as follows: „„ Supervision reporting of low quality (cited in 9 of 12 project ICR Reviews)—for example, Implementation Status and Results Reports that did not identify ongoing challenges, did not recommend actions to address challenges, lacked focus on development outcomes, or failed to elevate issues that needed management attention. „„Lack  of candor in Implementation Status and Results Report ratings (8 of 12 projects)—for example, project ratings for development objective and implementation progress in the satisfactory range, despite evidence to the contrary. „„ Lack of timeliness or effectiveness of response to project restructuring opportunities (7 of 12 projects)—for example, extensive delays in submitting restructuring requests, failure to revise the PDO as needed, or failure to address underlying project design issues (such as implementation arrangements). The factors identified as negatively associated with the performance of World Bank projects are consistent with factors identified in past RAPs. Over the years, the World Bank has taken action to respond to these problems, but several remain underaddressed, leading to recent initiatives such as the Agile World Bank, ICR Reform, the Multiphase Programmatic Approach, the Global Delivery Initiative, DeCODE, the new restructuring policy, and knowledge package delivery to task teams. Understanding the extent to which these recent initiatives ensure, for example, shorter feedback loops, iterative and adaptive implementation, avoiding past mistakes, candid and timely reporting, and task team proactivity, will need close monitoring and appropriate follow-up by management. Factors Associated with Outcome Ratings for World Bank Environment Projects Projects with an environmental focus have received low outcome ratings. Projects mapped to the Environment and Natural Resources Global Practice have received lower outcome ratings than projects in other Global Practices, and environmental projects mapped to other Global Practices may have similar drivers of outcome ratings. IEG analyzed 50 recently evaluated environmental projects and compared high-performing projects (those with outcome ratings of satisfactory or highly satisfactory) with low-performing projects (those with outcome ratings of moderately unsatisfactory or below).14 (Appendix B describes the methodology, and appendix C describes the results in more detail.) Independent Evaluation Group | World Bank Group 37 The quality of evidence generated on environmental outcomes was a major factor associated with differences in outcome ratings, as was the presence of economic analysis to demonstrate project efficiency. Projects that did not complete their core outputs, unsurprisingly, all received low ratings. Furthermore, 80 percent of low-performing projects lacked sufficient evidence on outcomes, due to weaknesses with design of M&E systems.15 Ninety-three percent of low-performing projects had at least one of these two factors. The presence of an ex post economic analysis was also strongly related to outcome ratings. Two-thirds of projects with an economic analysis received high outcome ratings, while two-thirds of those without an economic analysis received low outcome ratings. Projects that sought to calculate a value of environmental benefits (including benefits from ecosystem services and biodiversity conservation, or reductions in pollution, greenhouse gas emissions, nutrient depletion, or natural disaster damage) were much more likely to receive high ratings than those that did not. Other hypothesized factors could not be shown to have made a difference in outcome ratings. Projects with overly ambitious objectives (in scope or timing) were more likely to receive low ratings than those that did not, but all projects with low ratings and overambitious objectives also either did not produce core outputs or did not provide evidence on outcomes, suggesting that overly ambitious objectives are not a main cause of low outcome ratings. Furthermore, projects that reported weaknesses in implementation capacity were more likely to have low ratings, but these projects also tended not to produce their core outputs. The analysis could not determine if weak implementing capacity caused weaknesses in delivery or if weaknesses in delivery caused evaluators to conclude that implementation capacity was weak. Projects implemented through environmental agencies were more likely to note weaknesses in implementation capacity. Other factors, such as project size, whether projects were financed in part by the Global Environment Facility, or whether projects were implemented across multiple sectors, were not related to differences in outcome ratings. Importance of Economic Analysis in World Bank Projects Economic analysis can help the World Bank define outcomes, incentivize achievement, and allocate scarce resources efficiently. This RAP finds measurement of outcomes and the presence of economic analysis among the most important factors associated with outcome ratings for World Bank environment projects. Historically, economic analysis has been important in World Bank projects to ensure the efficient use of project resources. The guidance to task managers provided in the World Bank’s 1998 guidance on economic analysis was, “The Bank traditionally has not calculated a discount rate but has used 10–12 percent as a notional figure for evaluating Bank–financed projects. … Task managers may use a different discount rate, as long as departures from the 10–12 percent rate have been justified in the Country Assistance Strategy” (World Bank 1998). This guidance implied that the 10–12 percent discount rate was applicable unless justification for a different discount rate appeared in the World Bank’s country strategy document. Recently, the World Bank modified its guidance on economic analysis. In 2016, the World Bank’s guidance stated, “Given reasonable parameters for the other variables in the standard Ramsey formula linking discount rates to growth rates, a 3 percent per capita growth rate translates into a 6 percent 38 Results and Performance of the World Bank Group 2017 | Chapter 2 discount rate, and per capita growth rates of 1%-5% yield discount rates of 2%-10%.... The time period used in the economic analysis of projects should reflect reasonable estimates of the full duration of costs and benefits associated with the project, rather than be capped at 20 years or some arbitrary cut-off date.”16 In 2017, Operational Policy and Country Services stated that there is no mandatory requirement for a discount rate, but the 2016 guidance identifies technical approaches task managers can use in their economic analysis and must determine the appropriate rate for the project. Thus, there has been an evolution in the World Bank’s guidance to task managers on discount rates over 1998–2016, in particular, toward greater leeway in determining discount rates. This evolution raises the question of how the World Bank is ensuring coherence and consistency in (i) the discount rates, (ii) the methodologies for determining the discount rates, and (iii) the selection of time periods used in economic analysis for similar types of projects in the same sector and country. Clearer guidance and strong oversight are needed if World Bank management is to ensure that the methodology, discount rates, and time periods used for economic analysis are not arbitrarily selected by project teams. Gender Building on the pilot analysis in RAP15 and RAP16, this RAP found that gender was an explicit part of the PDO or at least one component in 68 of 241 projects reviewed (28 percent).17 Of these, 50 (73 percent) reported either some sex-disaggregated or male- or female-specific indicators at some level in the ICR or that one of the indicators was the share or percentage of female beneficiaries. Gender was an explicit part of the PDO in 16 projects. Among these, 12 reported on gender indicators at the PDO level in the ICR (see appendix I for details). Performance of IFC Investment Projects Development outcome ratings for IFC investment projects slipped to 50 percent mostly successful or better for CY14–16, widening the gap between development outcomes results and IFC’s corporate target of 65 percent.18, 19 The proportion of projects with mostly successful or better outcome ratings is slightly higher by volume, although this too has declined since CY11–13. A detailed presentation of development outcome ratings shows the decreasing share of projects with highly successful and successful (high positive) ratings in recent years, both by number of projects and by volume (figure 2.4). After IFC’s peak of high positive ratings in CY07–09, the evaluated projects with positive outcomes increasingly received mostly successful ratings, signifying that project outcomes were still considered positive overall despite minor shortcomings in some of their important aspects.20 Among projects evaluated in this period that have poor outcome ratings (in the unsuccessful range), a higher share have extremely negative outcome ratings (highly unsuccessful and unsuccessful). Unweighted results show that among the four indicators of development outcome,21 only the ratings for E&S effects did not decline in the CY14–16, thanks to the marked increase in the share of evaluated projects (76 percent) in the CY16 Expanded Project Supervision Reports program Independent Evaluation Group | World Bank Group 39 Figure 2.4 |I FC Investment Projects with Mostly Successful or Better Development Outcome Ratings Are Shrinking  hare of ratings, three-year rolling average, by number of projects a. S 100 80 60 40 Percemt 20 0 –20 –40 –60 2) 0) 9) 9) 4) 6) 3) 0) 3) 2) 1) 8) 5) 4 1 9 7 7 7 1 2 4 3 3 3 =2 =2 =1 =1 =1 =1 =2 =2 =2 =2 =2 =2 3 =2 (N (N (N (N (N (N (N (N (N (N (N (N (N 1 4 5 6 7 8 9 0 2 4 5 6 –1 –0 –0 –0 –0 –0 –0 –1 –1 –1 –1 –1 3 –1 09 02 03 04 05 06 07 08 10 12 13 14 11 CY CY CY CY CY CY CY CY CY CY CY CY CY Evaluation Year  hare of ratings, three-year rolling average, by IFC net commitment ($, billions) b. S 100 80 60 40 Percent 20 0 –20 –40 –60 ) ) ) ) ) ) ) ) ) ) ) ) .9 .0 .8 .3 .5 .0 .2 .6 .8 .7 .4 .6 ) .1 =5 =3 =2 =2 =2 =3 =4 =4 =6 =6 =6 =6 =7 (V (V (V (V (V (V (V (V (V (V (V (V (V 1 4 5 6 7 8 9 0 2 4 5 6 –1 –0 –0 –0 –0 –0 –0 –1 –1 –1 –1 –1 3 –1 09 02 03 04 05 06 07 08 10 12 13 14 11 CY CY CY CY CY CY CY CY CY CY CY CY CY Evaluation Year Highly satisfactory Moderately unsatisfactory Satisfactory Unsatisfactory Moderately satisfactory Highly unsatisfactory Source: Independent Evaluation Group data for IFC investments projects (as of December 12, 2017). Note: IFC = International Finance Corporation; N = number of projects; V = total volume of projects ($, billions) rated satisfactory or better for this indicator (figure 2.5).22 E&S effects ratings of projects in the infrastructure and natural resources (INR) and Financial Institutions Group (FIG) contributed to the improvement.23 In addition, a confluence of other factors contributed to the improved E&S effects ratings, by number of projects, in CY14–16. At the project level, client commitment plus the client’s technical and financial capacity to manage its operations were identified as the main drivers of 40 Results and Performance of the World Bank Group 2017 | Chapter 2 Figure 2.5 |L  ong-Term Trend of Environmental and Social Effects Ratings Remained Flat While Other Indicators Trended Downward  hare of satisfactory or better ratings, three-year rolling average, by number of projects a. S 100 80 60 Percent 40 20 0 1 4 5 6 7 8 9 0 2 4 5 6 –1 –0 –0 –0 –0 –0 –0 –1 –1 –1 –1 –1 3 –1 09 02 03 04 05 06 07 08 10 12 13 14 11 FY FY FY FY FY FY FY FY FY FY FY FY FY Evaluation Year  hare of satisfactory or better ratings, three-year rolling average, by IFC net commitment b. S 100 80 60 Perecent 40 20 0 1 4 5 6 7 8 9 0 2 4 5 6 –1 –0 –0 –0 –0 –0 –0 –1 –1 –1 –1 –1 3 –1 09 02 03 04 05 06 07 08 10 12 13 14 11 FY FY FY FY FY FY FY FY FY FY FY FY FY Evaluation Year Private Sector Development Environmental and Social Effects Economic Sustainability Project Business Performance Source: Independent Evaluation Group data for International Finance Corporation investments projects (as of December 12, 2017). Note: The development outcome of an International Finance Corporation investment project consists of four indicators: project business success; economic sustainability; environmental and social effects; and private sector development. Each indicator is assessed and rated separately, and a composite rating for development outcome is then determined. positive E&S effects ratings. Projects with satisfactory or better E&S effects ratings also have positive ratings for IFC’s E&S additionality and for its E&S work quality at appraisal and at supervision. IFC staff’s frequent engagement and communication with clients throughout the project cycle enhanced client commitment and capacity—a finding consistent with IEG’s 2016 review of IFC additionality.24 Solid E&S work quality at appraisal and at supervision by IFC staff helped identify actions that clients must implement to mitigate environmental and social risks and to ensure compliance with Independent Evaluation Group | World Bank Group 41 IFC’s Performance Standards.25 Adjusted by volume, however, the percentage of positive ratings for E&S effects is down slightly in CY14–16 (64 percent) compared with CY11–13 (70 percent).26 E&S performance of evaluated manufacturing, agribusiness, and services projects were rated low and have not improved. At the corporate level, increased resources provided to E&S work, hiring more specialists, and intensified monitoring of requirements contributed to the high IFC E&S work quality ratings at appraisal and at supervision, and for the high additionality ratings (82 percent satisfactory or better ratings) in CY16. Projects with positive E&S ratings also made use of legal agreements, such as the condition for disbursement, to leverage client compliance with IFC’s E&S requirements. However, room for improving the E&S effects remains, especially in the agribusiness and forestry sector. Land acquisition and involuntary resettlement issues and implementation of Environmental, Health, and Safety Guidelines and E&S Action Plans need improvement. Satisfactory or better ratings for project business success, economic sustainability, and private sector development (three of four indicators of development outcome) continue to decline (figure 2.5). Business success, economic sustainability, and private sector development are fundamental to IFC’s mandate of promoting economic development “by encouraging the growth of productive private enterprise in member countries, particularly in the less developed areas” (IFC 2012a). Projects with poor business results experienced sustainability challenges and had difficulty achieving the projects’ development objectives. Projects experiencing poor business results also had difficulty complying with IFC’s E&S requirements and had poor E&S effects ratings too. By number of projects, nearly half (48 percent) of the evaluated projects in CY14–16 with poor business success ratings also had low development outcome ratings. Crucially, there is a strong association between IFC work quality and development outcome ratings, as illustrated in figure 1.6. Only a small percentage of projects with a low rating for IFC work quality also had positive development outcome ratings. The majority of the projects are either in the high IFC work quality–high development outcome quadrant or in the low IFC work quality–low development outcome quadrant. By Region, South Asia (including Afghanistan and Pakistan) had a higher share of projects with mostly successful or better outcome ratings in CY14–16, by number of projects and volume (appendixes J and L). By number of projects, outcome ratings also improved from 49 percent in CY11–13 to 71 percent in this review period. By IFC net commitment, positive outcome ratings improved from 67 percent in CY11–13 to 79 percent in CY14–16. High outcome ratings of projects in two (INR and manufacturing, agribusiness, and services [MAS]) of the four IFC industry groups buoyed development outcome ratings in South Asia in this review period. IFC’s front-end work quality ratings improved from 43 percent in CY11–13 to 65 percent, which helped lift the Region’s development outcome ratings. (See the appendix L section on the South Asia Region for additional analysis). By contrast, evaluated projects in the Middle East and North Africa Region (excluding Afghanistan and Pakistan) had the weakest outcome ratings in CY14–16 compared with CY11–13 results.27 Evaluated projects in IDA countries in CY14–16 had a slightly higher share of positive outcome ratings compared with projects in non-IDA countries, by number of projects and IFC net commitment 42 Results and Performance of the World Bank Group 2017 | Chapter 2 Figure 2.6 |D  evelopment Outcome Ratings Are Highly Associated with IFC’s Work Quality (percent)  y number of evaluated projects a. B  y IFC net commitment, in $, millions b. B IFC WORK QUALITY IFC WORK QUALITY LOW HIGH LOW HIGH DEVELOPMENT OUTCOME DEVELOPMENT OUTCOME 5% 45% 6% 53% HIGH HIGH n = 12 n = 106 $394 $3,513 34% 16% 26% 14% LOW LOW n = 81 n = 37 $1,743 $958 Source: Independent Evaluation Group data for IFC investments projects (as of December 12, 2017). Note: The data presented above are for CY14–16 only. Assessment of IFC’s Work Quality was revised in 2015 when the revised Instructions for Preparing Expanded Project Supervision Reports took effect. amount. By number of projects, 53 percent of evaluated projects in IDA countries in CY14–16 had mostly successful or better outcome ratings; this was 72 percent by volume. In contrast, 49 percent of evaluated projects in non-IDA countries had positive outcome ratings or 62 percent, if ratings are weighted by IFC net commitment amount. A large share of projects in IDA countries with positive development outcome ratings in CY14–16 were rated positively for economic sustainability and private sector development. Evaluated projects in IDA countries with positive outcome ratings were also rated highly for achieving IFC’s expected investment returns. However, evaluated projects in non-IDA countries had better ratings for E&S effects and for the quality of IFC’s front and back-end work.28 Outcome success ratings of projects in FCS and non-FCS countries also declined in CY14–16 compared with those in CY11–13,29 both in terms of number of projects and if weighted by amount of IFC net commitment. It should be noted, however, that only 15 projects took place in FCS countries in the CY14–16 evaluation cohort, of which 7 had positive development outcome ratings. By contrast, 190 investment projects in non-FCS countries were evaluated in CY14–16; 52 percent had mostly successful or better outcome ratings. By industry group, INR has a higher share of projects with mostly successful or better outcome ratings, both by number of projects and volume. More INR projects were also rated satisfactory or better for the quality of IFC’s work in screening, appraising, and structuring projects (see appendix J). Independent Evaluation Group | World Bank Group 43 Outcome success ratings of FIG and MAS projects declined considerably from CY11–13 to CY14–16, by both number and volume. Within INR, power sector projects delivered the largest share of mostly successful or better development outcome ratings.30 Investment outcome and IFC work quality ratings of power sector projects were also better than ratings for other projects in the INR group. Industry knowledge and thorough assessment of country, sector, market, and technical design risks were common features of successful power sector projects. Intra–Bank Group coordination to resolve or mitigate regulatory issues also contributed to better outcomes. An emerging lesson from the recently evaluated INR projects, especially in regulated sectors, is to work closely with the World Bank to support governments in creating a stable and conducive regulatory environment for investment—a goal emphasized in the Bank Group’s Maximizing Finance for Development approach and in the IFC 3.0 Creating Markets strategy. By contrast, less successful INR projects were adversely affected by external factors but also had major shortcomings in screening, appraisal, and structuring. However, for the natural resources projects (for example, oil, gas, and mining) evaluated in CY14–16, the decline in international commodity prices since 2015 adversely affected their development outcome ratings, despite the high positive ratings for IFC’s work quality. Recently evaluated investment projects in the MAS industry group experienced the biggest decline in development outcome ratings, by number of projects and volume. About half had positive outcome ratings (46 percent by number of projects; 54 percent by IFC net commitment). Compared with CY11–13 results, positive outcome ratings fell 26 and 22 percentage points, by number of projects and by IFC net commitment amount in CY14–16. MAS projects had poor outcome ratings in five of the six Regions (South Asia being the exception). MAS projects in the East Asia and Pacific Region had the weakest outcome ratings; only a third of the evaluated projects had positive outcome ratings, by number of projects. By volume, only a quarter were rated positively for their development outcomes.31 Among the MAS subgroups, tourism, retail, construction, and real estate (TRP) projects had the highest share of low outcome ratings (only 1 in 13 were rated successful). Outcome results of projects in the health and life sciences subsector, a strategic business segment for IFC, fared slightly better—especially when IFC invested in multiple projects whose sponsors consisted of a network of medical practice groups. This approach has allowed IFC to meet clients’ demand for quicker and less conditional financing. A large number of MAS projects with low outcome ratings also had low ratings for IFC work quality, especially in screening, appraisal, and structuring. Poorly performing projects were affected by cost overruns owing to change in project scope, sponsors’ lack of relevant experience in the target market, and mismatch between sponsor capacity and the required expertise. Success ratings of evaluated FIG projects in CY14–16 also declined compared with the share of positively rated projects in CY11–13, by number of projects and with an even greater percentage point decline, by IFC net commitment.32 FIG projects with low outcome ratings (mostly unsuccessful or worse) also tended to have poor ratings in the quality of IFC’s up-front work (screening, appraisal, and structuring). Projects with low outcome ratings included those intended for on-lending to micro, small, 44 Results and Performance of the World Bank Group 2017 | Chapter 2 and medium enterprises or for energy efficiency and renewable energy, which represents a new business segment for IFC’s FI clients, especially in the Africa Region. Factors Associated with Development Outcome Ratings of IFC Investment Projects IFC’s screening, appraisal, and underwriting and IFC’s role and contribution were the two most significant internal factors influencing the development outcome ratings of IFC investment projects.33 IEG’s updated statistical analysis identified that, in both real and FI projects,34 IFC’s screening, appraisal, and structuring have the most significant and positive effect on development outcome ratings, followed by IFC’s role and contribution (see appendix J). However, in FI projects, IFC’s role and contribution have a more significant positive effect on development outcome ratings than IFC’s up-front work does. The effects on the ratings of IFC’s supervision and administration work, project size (measured by IFC’s net commitment), and investments in repeat clients’ projects were still positive but less statistically significant and robust for both real and FI projects.35 Other internal risk factors negatively correlated with development outcome ratings include greenfield projects and IFC’s equity-only investments. Improvement in country risk (measured by the change in Institutional Investors Country Credit Rating at approval and at evaluation) is significantly associated with better development outcome ratings in real and FI projects. Improvements in management quality and profit margins also increase the likelihood of achieving better development outcomes (appendix J). However, the influence of these external factors on ratings is less robust than the quality of IFC’s up-front work and IFC’s role and contribution.36 As IFC’s work quality and development outcome ratings declined further in CY14–16, IEG and IFC conducted a joint diagnostic of the root causes. IFC has begun addressing these weaknesses, but it is too early to see the outcomes of these initiatives. A joint IFC-IEG working group conducted an in-depth analysis of IFC work quality in FY17 and identified major shortcomings that require attention. Among them were IFC’s inadequate assessment of macroeconomic and regulatory conditions at the project and portfolio levels and staff’s lack of awareness about project characteristics historically associated with high risk to development outcomes.37 The diagnostics also identified organizational factors that contributed to weak quality at entry and supervision work and highlighted the importance of continued portfolio monitoring to identify and address risks to development outcomes.38 IFC implemented initiatives to address the root causes of declining work quality performance (box 2.1 provides two examples) and created a new results framework to better define and measure the likely development impact of each project. However, the results of these initiatives will be evident only in the medium term. Performance of IFC Advisory Projects Development effectiveness success rating for evaluated IFC Advisory Services slipped from 63 percent in FY11–13 to 49 percent in FY14–16, the lowest since IEG started reviewing such projects in 2008.39, 40 If weighted by the amount of IFC expenditures or volume, the success ratings of projects evaluated in the current review period is slightly higher. Compared with FY11–13 results, positive Independent Evaluation Group | World Bank Group 45 Box 2.1 | T  he International Finance Corporation Is Implementing Several Initiatives to Improve Development Results The Accountability Initiative (April 2017) clarifies the roles and responsibilities in International Finance Corporation (IFC) operations, establishes clear success metrics, reflects staff performance for each transaction, and aligns incentives and business goals. The first phase of implementation in quarter four of FY17 focused on clarifying transaction roles and responsibilities and decision making in operations. In FY18, IFC introduced a new set of financial metrics and targets across business units and is piloting a new transaction- centric feedback system. Additionally, IFC’s Economics and Private Sector Development unit introduced a new measure of ex ante development impact, which will be integrated in the performance evaluation of each IFC project and business unit. Accountability for overall quality and structuring of investments at the industry level was strengthened. Staff responsibilities for portfolio and client management were clarified and reinforced. Anticipated Impact Measurement and Monitoring (2017) is IFC’s new results measurement framework, created to support the implementation of its current 3.0 strategy of creating markets. The Anticipated Impact Measurement and Monitoring (AIMM) framework will better define, measure, and articulate the development impact of each project at approval. AIMM assesses the anticipated impact at two levels: project outcome (what a specific project is expected to achieve) and project contribution to market creation (how the project is expected to contribute to market creation and development). The AIMM approach is now integrated into IFC’s existing results measurement framework and monitoring system, starting with the IFC investment projects presented for Board of Executive Directors approval in FY18. The Independent Evaluation Group is currently discussing with IFC management how the AIMM framework complements the existing self-evaluation (ex post) framework based on Expanded Project Supervision Reports. Sources: IFC Accountability Initiative (May 2017) and technical briefing on the AIMM (October 2017). ratings by volume also fell from 66 percent to 52 percent in FY14–16. The share of high positive (highly successful and successful) effectiveness ratings shrank after a peak in the FY12–14 review period. Evaluated advisory projects with positive development effectiveness ratings increasingly received mostly successful ratings, signifying that major project objectives were achieved despite some shortcomings (see figure 2.7). In the current review period, however, evaluated projects with poor development effectiveness ratings have a higher share of extremely negative outcome ratings (highly unsuccessful and unsuccessful). Despite the low development effectiveness ratings, IFC’s role and contribution achieved 75 percent and 71 percent satisfactory or better ratings for FY14–16, 46 Results and Performance of the World Bank Group 2017 | Chapter 2 Figure 2.7 |I FC Advisory Projects with Mostly Successful or Better Development Effectiveness Ratings Are Shrinking  hare of ratings, three-year rolling average, by number of projects a. S 100 80 60 40 Percent 20 0 –20 –40 –60 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 (N=214) (N=249) (N=222) (N=210) (N=189) (N=185) (N=150) Evaluation Year  hare of ratings, three-year rolling average, by IFC net commitment ($, millions) b. S 100 80 60 40 Percent 20 0 –20 –40 –60 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 (V=85) (V=161) (V=190) (V=238) (V=227) (V=237) (V=186) Evaluation Year Highly satisfactory Moderately unsatisfactory Satisfactory Unsatisfactory Moderately satisfactory Highly unsatisfactory Source: Independent Evaluation Group data for IFC advisory projects (as of December 12, 2017). Note: IFC = International Finance Corporation; N = number of projects; V = total volume of projects ($, millions) by number of projects and volume, respectively. Compared with FY11–13 performance, however, positive ratings for IFC’s role and contribution weakened in this review period (appendix J). The decline in positive ratings is evident across the subindicators of development effectiveness. Satisfactory or better ratings for outcome achievement and efficiency continue to lag behind other indicators of IFC advisory projects’ development effectiveness (figure 2.8). Simply put, while a large share of advisory projects were rated satisfactory or better for delivering the intended outputs (for example, report, training, products, and so on), expected outcomes or the short- or medium-term Independent Evaluation Group | World Bank Group 47 Figure 2.8 |R  atings of IFC Advisory Development Effectiveness and Each Core Component Show Weak Results in FY14–16  hare of mostly successful or better ratings 3-year rolling average, by number of projects a. S 100 80 60 40 20 0 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 Evaluation Year  hare of mostly successful or better ratings 3-year rolling average, by IFC expenditure b. S 100 80 60 40 20 0 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 Evaluation Year Output Achievement Strategic Relevance Efficiency Outcome Achievement Source: Independent Evaluation Group data for IFC Advisory Projects (as of December 12, 2017). Note: IFC = International Finance Corporation. changes resulting from the project were not achieved. The historic trend presented in figure 2.8 illustrates the gap between output and outcome achievement and between strategic relevance and output achievement. IEG’s review of the drivers of declining development effectiveness ratings in IFC advisory projects identified poor work quality, especially in project preparation and design, as a major factor.41 Of the 58 evaluated advisory projects with ratings for IFC work quality in FY15 and FY16, 32 projects (55 percent) were rated satisfactory or better for work quality (figure 2.9), equivalent to 51 percent by volume.42 At the portfolio level, the increase in average IFC Advisory Services expenditures 48 Results and Performance of the World Bank Group 2017 | Chapter 2 Figure 2.9 |D  evelopment Effectiveness Ratings Are Highly Associated with IFC Work Quality (percent)  y number of evaluated projects a. B  y IFC net commitment ($, millions) b. B IFC WORK QUALITY IFC WORK QUALITY LOW HIGH LOW HIGH DEVELOPMENT OUTCOME DEVELOPMENT OUTCOME 3% 34% 2% 25% HIGH HIGH n=2 n = 20 $1 $14 41% 21% 47% 28% LOW LOW n = 24 n = 12 $29 $17 Source: Independent Evaluation Group database for the Project Completion Report Program (as of December 12, 2017). Note: The International Finance Corporation (IFC) and Independent Evaluation Group added the dimension “IFC Work Quality” into the assessment of IFC Advisory Services projects in FY15 as a result of the 2015 revision of the instructions for preparing Project Completion Reports. The figures above present the development effectiveness and IFC work quality ratings for FY15 and FY16 only. Total number of evaluated advisory projects with IFC work quality ratings in FY15 and FY16 is 58, equivalent to $61 million in IFC expenditures. marks a shift toward larger-sized and longer-duration projects not matched by improved IFC work quality.43 IEG’s validations of project self-evaluations noted work quality shortcomings in project preparation and design arising from inadequate market assessment and sequencing of diagnostic work, and the tendency to include too many activities or components. The latter creates problems by spreading thinly the already insufficient resources allocated, especially in IDA and FCS countries where client capacity is low. As figure 2.9 illustrates, projects with low ratings in IFC work quality are not likely to achieve high development effectiveness ratings (upper left quadrants). M&E of IFC advisory projects had been affected by the restructuring in 2014 and, as a consequence, has weakened accountability for ensuring quality control and delivering results. “Structural” problems related to the fixed amount allocated for project preparation and design regardless of complexity also affected the evaluated projects’ effectiveness. Project preparation has suffered, and frequent changes in IFC advisory project leaders adversely affected project design and implementation.44 The reorganization of the IFC Advisory Services group in 2014 may have amplified already existing work quality weaknesses that IEG previously highlighted in its validation of IFC self-assessments. A joint IEG-IFC working group is discussing the root causes of the trend in declining advisory work quality, with the end goal of reversing the deteriorating ratings and improving the development effectiveness of IFC advisory projects. But it is too early to tell what actions and initiatives will be taken. Moreover, Independent Evaluation Group | World Bank Group 49 any improvement in results will be evident only in the medium term, when the Advisory Services projects approved from FY18 and onward are evaluated. Performance of MIGA Projects Development Outcome ratings of evaluated MIGA projects remained the same for FY11–16 at 62 percent; the long-term trend has not changed.45 The percentage of MIGA projects rated satisfactory or better for their development outcomes, based on a six-year moving average from FY11 to FY16 reached 62 percent, after re-rating seven projects with no opinion possible (NOP) ratings and adding the results from 13 evaluated projects in FY16. The result is on par with the 61 percent positive outcome ratings in the FY05–10 review period (figure 2.10). By sector group, energy and extractive industry projects had higher outcomes ratings in FY11–16 (10 of 12 projects, 83 percent, had positive outcome ratings). By contrast, finance and capital market projects showed the weakest outcome result (11 of 24 evaluated projects, 46 percent, had positive ratings). A higher percentage of projects in IDA countries had satisfactory or better outcome ratings compared with non-IDA countries. Positive outcome ratings of projects in IDA countries also showed marked improvement (15 percentage point increase) from the FY05–10 review period in contrast to the 7 percentage point slippage in the outcome ratings in non-IDA countries.46 Of the 10 projects in FCV countries evaluated in FY11–16, seven had satisfactory or better outcome ratings.47 Meanwhile, 62 percent of projects (41 of 66) in non-FCV countries were rated positively in terms of their development results. Among the four indicators of development outcome, E&S effects showed the most improvement, with the private sector development indicator receiving the highest share of satisfactory or better ratings in FY11–16. Satisfactory or better rating for project business performance in this review period was the weakest. Seventy-two percent of projects evaluated during this period were rated satisfactory or better in terms of their E&S effects (figure 2.10). The annual trend indicates marked improvement in E&S effects, with 67 percent (11 of 15 projects) and 92 percent (11 of 12 projects) rated satisfactory or better in FY15 and FY16, respectively. A marked improvement in MIGA’s E&S supervision helped improve overall E&S effects. Project business performance, one of four subindicators of development outcome, had the weakest results in FY11–16 (57 percent); positive ratings also declined by 10 percentage points from FY05–10 ratings. And among the four sector groups, financial and capital markets projects had the weakest business performance ratings in FY11–15, followed by evaluated projects in the infrastructure sector.48 Based on MIGA’s procedures and standards, ratings for assessment, underwriting, and monitoring also improved, albeit modestly, in FY11–16. MIGA’s work quality can still be enhanced. Foremost is improving the documentation of project performance throughout the project cycle, including better management of project documents. A recurring shortcoming pertains to the lack of information about a project after the contract of guarantee is signed, which makes self- and independent evaluation of MIGA projects challenging.49 While some improvements appeared in MIGA’s up-front work, several IEG reports reiterated the importance of strengthening underwriting methods, monitoring, and supervision of the project’s credit risk, obligor risk profile, and overall performance reporting in self- evaluations. Although MIGA routinely assesses country risk and corresponding risk mitigants during 50 Results and Performance of the World Bank Group 2017 | Chapter 2 Figure 2.10 |T  he Long-Term Trend of MIGA Development Outcome Ratings Has Not Changed  evelopment outcome ratings, six-year rolling averages, by number of projects a. D 100 80 60 40 Percent 20 0 –20 –40 –60 FY06–11 FY07–12 FY08–13 FY09–14 FY10–15 FY11–16 (N=50) (N=50) (N=58) (N=63) (N=78) (N=77) Evaluation Year Excellent Partly unsatisfactory Satisfactory Unsatisfactory  etailed development outcome ratings, six-year rolling average, by number of projects b. D 100 80 Perecent 60 40 20 0 FY06–11 FY07–12 FY08–13 FY09–14 FY10–15 FY11–16 Evaluation Year Private Sector Development Environmental and Social Effects Economic Sustainability Project Business Performance Source: Independent Evaluation Group data for Multilateral Investment Guarantee Agency Project Evaluation Reports (as of December 12, 2017). Note: The results are for evaluated “regular” guarantee projects only and exclude MIGA’s Small Investment Program projects, which are evaluated on a programmatic basis. MIGA development outcome results are presented based on a six-year rolling average because of the small number of evaluated MIGA projects since 2004. The results presented in the graph include the new ratings for seven projects that had a no opinion possible rating for development outcome reported in the Results and Performance of the World Bank Group 2016. MIGA = Multilateral Investment Guarantee Agency; N = number of projects. the underwriting stage, potential risks to project development outcome are not assessed during underwriting. RAP16 noted these issues in view of MIGA’s development mandate. MIGA’s results exclude the ratings for its Small Investment Program (SIP) projects, which have been evaluated on a programmatic basis since FY11. The program provides political risk insurance Independent Evaluation Group | World Bank Group 51 coverage to foreign investments in SMEs.50 Only two SIP projects were evaluated using the Project Evaluation Report methodology—one was completed in FY10 and the other in FY12. For cost- effectiveness reasons, IEG and MIGA agreed in FY11 to evaluate SIP projects on a programmatic basis. The first programmatic evaluation was conducted as part of IEG’s evaluation of the Bank Group’s targeted support to SMEs in FY14.51 Performance of World Bank Group Country Strategies Bank Group country strategy performance remains below the corporate target of 70 percent. This result is based on an analysis that considers the performance of the Bank Group’s strategy and approach in each country as defined in the CPF or country assistance strategy (CAS).52 These ratings are assigned as part of IEG’s Completion and Learning Reports (CLR) Reviews (formerly called CAS Completion Reports). The methodology for CLR Reviews changed in December 2013 to a harmonized World Bank/IEG methodology that included ratings of Bank Group performance on a 4-point scale rather than the previous 6-point scale.53 The analysis found that country development outcome ratings for the Bank Group in FY14–17 were 68 percent moderately satisfactory or above. The Bank Group performance ratings were 59 percent good or above. Ratings for IBRD country programs stood at 78 percent moderately satisfactory or above for development outcome and 56 percent good or above for Bank Group performance. Ratings for IDA countries (which include IBRD-IDA blend countries)54 were 58 percent for development outcome and 63 percent good or above for Bank Group performance (see appendix E, figure E.6). Among country programs in FCV countries, 38 percent had development outcome ratings of moderately satisfactory or above, and 54 percent had Bank Group performance ratings of good or above. Patterns of CLR Review ratings across Regions are shown in appendix E, figures E.7 and E.8.55 Performance of Country Strategy Objectives Supporting Environmental Sustainability Just over half the environment outcomes or objectives were achieved for country strategies where ratings are available, similar to other outcomes or objectives.56 IEG rated outcomes or objectives for 43 out of 74 CLR Reviews for strategies prepared between FY08 and FY12. Just over half of the environment outcomes or objectives were achieved—IEG rated 52 percent achieved and mostly achieved. This is in line with the ratings of all outcomes or objectives, 53 percent. The Green area has lower achievement rates, while the Clean area was rated higher, but the population is too small to draw general conclusions. One-quarter of the environment outcomes or objectives were not achieved, usually because planned Bank Group activities to achieve the objective were delayed rather than because these operations failed to reach the target.  ifferent time frames are generally applied to assessing the performance of projects across the institutions. D 1 International Finance Corporation (IFC) investment projects are sampled, evaluated, and reported by calendar year (CY). World Bank, IFC Advisory Services, and Multilateral Investment Guarantee Agency (MIGA) projects are 52 Results and Performance of the World Bank Group 2017 | Chapter 2 evaluated and reported on a fiscal year (FY) basis. Moreover, the World Bank, IFC, and MIGA have different operating models, different instruments, and different clients. Hence, results and performance are assessed separately for each institution, as is customarily done in the annual Results and Performance of the World Bank Group report. 2 I n the Implementation Completion and Results and the Implementation Completion and Results Review, the outcome rating derives from the ratings of relevance, efficacy, and efficiency of the project. 3  ggregate ratings for the 45 projects closed in FY17 for which ratings data are available as of October 2, 2017, should A be interpreted with great caution for several reasons: (i) 45 is a small number, estimated to be 19 percent of the full- year cohort; (ii) Implementation Completion and Results (ICRs) for many projects closed in FY17 would not even have been due yet; and (iii) quickly completed ICR Reviews tend to involve simpler projects, higher-quality reporting, less controversial findings about results, and higher overall ratings. Coverage for FY17 was calculated as follows: As of October 2, 2017, ICRs for 45 projects closed in FY17 had been received by the Independent Evaluation Group and validated. As of that time, an additional 68 ICRs for projects closed in FY17 were still under review, and additional ICRs for FY17 were expected to arrive over the course of FY18. The Independent Evaluation Group used the average number of ICRs received for projects closed in FY14, FY15, and FY16 to estimate the total expected, 237. This number is used as the denominator for the FY17 coverage percentage of 19 percent. 4  any factors could have resulted in the improvements in ratings (for example, a change in composition of the portfolio), M but it is difficult to disaggregate or identify each of them. 5 F  or Implementation Completion and Results reviewed before July 1, 2006, World Bank performance was rated on a 4-point scale. On or after that date, World Bank performance is rated on the current 6-point scale. 6  mong the 243 FY14–16 projects with monitoring and evaluation quality ratings of substantial or high, 96 percent were A rated moderately satisfactory or above on World Bank performance, compared with 60 percent moderately satisfactory or above on World Bank performance for projects with monitoring and evaluation quality ratings of modest or low. 7  Country contexts traditionally seen as challenging” here refers to fragility, conflict, and violence countries (both “ International Bank for Reconstruction and Development and International Development Association) and International Development Association countries (including blend countries). 8  ast Asia and Pacific management reports for which, following their outcome analysis in FY15, the Region implemented E a Monitoring and Evaluation Results Program comprising training, operational clinics, and capacity assessment, which covered Indonesia, Vietnam, and China (Blanc et al., 2016). 9 T  ask team leader track record is calculated as the average outcome rating for all prior projects managed by the task team leader, weighted by their time on the project. While the weighting methodology varies between studies, all reviewed studies that discuss task team leader track records find it to be positively correlated with project outcome and statistically significant. 10  his review encompassed 33 studies that met the following criteria: (i) empirical studies, (ii) with findings specific to T International Bank for Reconstruction and Development / International Development Association projects, and (iii) published by the World Bank (including Independent Evaluation Group) or in peer-reviewed academic journals. A taxonomy of factors was developed and used to code the findings. 11  even studies in the review group found a positive, statistically significant correlation between gross domestic product S (GDP) growth and project outcome as defined either by the Independent Evaluation Group outcome rating or, less frequently, by the economic rate of return. Studies that explored the relationship of other country-level conditions, including GDP per capita and human capital, either found no relationship with project outcomes, or reached different conclusions about its relevance, as in the case of the Country Policy and Institutional Assessment rating. With regard to economic policy, studies found that distortional policies such as subsidies or overvalued exchange rates are negatively associated with project outcomes. 12 Replicating the Results and Performance of the World Bank Group 2015 regression analysis of factors of project  outcome ratings with an updated data set could have been expected to produce similar insights as in previous years regarding the factors underlying the success or failure of World Bank projects, given that not much change was Independent Evaluation Group | World Bank Group 53 observed in aggregate and that only a limited set of quantitative or categorical data elements are available for such an analysis. To provide a more nuanced view of factors, and to benefit from the data collection already done by other researchers, this report undertook a structured review of research rather than a regression analysis. Accordingly, the structured review identifies areas of consensus in the academic and professional literature discussing World Bank project outcomes. Findings overlap with past years on several key factors, including the effects of quality at entry and supervision quality. Additional factors included, for example, the track record of the task team leader and the quality of the project’s results chain. 13 T  his analysis provides pointers about what World Bank staff could focus on in supervising projects in the future and what they might avoid, keeping in mind that other factors may be just as or more important to project outcomes but are not systematically captured in Implementation Completion and Results Reviews—hence the need for more thorough follow-up analysis of the drivers of the ratings for quality of supervision, World Bank performance, and outcome. The analysis categorized the quality of supervision text using the elements of the criteria for the quality of supervision ratings in the Independent Evaluation Group’s Guidelines for Implementation Completion and Results Reviews, as well as using the DeCODE delivery challenge taxonomy developed by the World Bank’s Global Delivery Initiative, available from this website: http://www.globaldeliveryinitiative.org/decode/taxonomy. The aspects of supervision highlighted above reflect mainly the criteria for ratings of quality of supervision in the Independent Evaluation Group’s guidelines, as expected, given that Implementation Completion and Results Reviews are written with reference to those guidelines. Notably, though, the World Bank’s response to delivery challenges as defined in 18 Global Delivery Initiative taxonomy categories did appear at least once among the quality of supervision sections reviewed. Further assessment of Global Delivery Initiative’s DeCODE delivery challenge taxonomy could inform a framework or checklist for projects to use in ensuring adequate supervision, as well as for improvements to guidance on completion reporting. 14 P  rojects were classified as “environment” based on several criteria, including theme codes and objectives, as explained in appendix B. 15 Independent Evaluation Group project evaluation methodology is based on available evidence and assigns lower  ratings in the absence of evidence of achievement. 16  he Operations Policy and Country Services guidance note “Discounting Costs and Benefits in Economic Analysis of T World Bank Projects” offers guidance on alternative approaches to calculating discount rates under different growth, time horizon, and other assumptions, but it gives no recommended discount rate. Earlier versions of the guidance (undated) included a recommendation of a 5 percent discount rate (OPCS 2016). 17  he percentage of projects with gender in their project development objective or components is not an accurate T indicator of the number of projects that may help reduce gender gaps. Since 2016, World Bank projects intended to reduce gender gaps are tagged at entry. This tagging system is expected to facilitate the identification of operations whose gender results and lessons should be tracked at closing. 18  ased on the Instructions for Preparing an Expanded Project Supervision Report (XPSR; updated in 2015), the B development outcome of International Finance Corporation (IFC) investment projects are rated on a 6-point scale from highest to lowest: highly successful [HU], successful [SU], mostly successful [MS], mostly unsuccessful [MU], unsuccessful [US], higly unsuccessful [HU]). Development outcome has four constitutent elements: (i) project business success, (ii) economic sustainability, (iii) environmental and social effects, (iv) contribution to private sector development. The IFC rates and the Independent Evaluation Group (IEG) validates the rating of each of these constituent elements on a 4-point scale (from highest to lowest: excellent, satisfactory, partly unsatisfactory, and unsatisfactory). Investment outcome is the second criterion. This criterion assesses the extent to which IFC has realized at the time of evaluation, and expects to realize over the remaining life of the investment, the loan income and/or equity returns that were expected at approval. Investment outcome is rated on a 4-point scale (from highest to lowest: excellent, satisfactory, partly satisfactory, and unsatisfactory). IEG also rates the work quality for IFC investment projects. As a result of the XPSR instructions revision in 2015, IFC work quality has two constituent elements: (i) at entry (screening, appraisal, and structuring of financial instruments) and (ii) supervision (supervision quality). IEG rates each of these constituent elements on a 4-point scale (from highest to lowest: excellent, satisfactory, partly unsatisfactory, and unsatisfactory). Before the XPSR instructions were revised in 2015, IFC’s work 54 Results and Performance of the World Bank Group 2017 | Chapter 2 quality rating included a third element, IFC’s role and contribution. In the 2015 XPSR instructions, IFC’s role and contribution became a fourth dimension in the measurement of IFC’s performance but amended as IFC’s additionality. In the 2015 XPSR instructions, IFC’s additionality is defined as the benefit or value addition IFC brings that a client would not otherwise have. IFC’s additionality, therefore, is subset of its role that is unique to IFC and that cannot be filled by the client or any commercial financier. 19 nternational Finance Corporation investment projects use the calendar year. International Finance Corporation I advisory projects and Multilateral Investment Guarantee Agency projects use the fiscal year. 20  rojects rated mostly successful accounted for 66 percent of the positive outcome ratings (including highly successful P and successful) in calendar years 2014–16 compared with about a third (36 percent) 10 years ago. 21  he four development outcome subindicators for International Finance Corporation investment projects are project T business performance, economic sustainability, environment and social effects, and private sector development. 22  ompared with 63 percent and 64 percent of evaluated projects rated satisfactory or better for their environmental C and social effects in the calendar years 2015 and 2014 Expanded Project Supervision Report programs, respectively. 23 E  nvironmental and social (E&S) effects comprise both the project’s E&S performance and its E&S impacts. The E&S evaluation identifies a project’s E&S effects and considers these against the criteria set by the Sustainability Framework of the International Finance Corporation (IFC) and the requirements documented in the E&S review summary and investment agreement. The E&S performance assessment covers both compliance with IFC requirements and actual effects in the field. It reflects both the performance at appraisal (ex ante) and at evaluation (ex post). The E&S evaluation also includes E&S impacts by assessing how the project contributed to any change in E&S performance from the time of appraisal to that of evaluation, and whether such change has been beneficial or adverse. E&S effects are rated on a 4-point scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory. 24  he review noted the high reliance on International Finance Corporation’s environmental and social standards to T deliver the institution’s nonfinancial additionality, especially in real sector projects. See World Bank 2015b. 25 nformation from (i) Independent Evaluation Group (IEG) Environmental and Social Highlights, October 2017; and I (ii) background paper prepared for this report on the Implementation of IEG’s Recommendations for International Finance Corporation and Multilateral Investment Guarantee Agency on Performance Standards in the 2010 evaluation of World Bank Group Safeguard Policies. November 30, 2017. 26  he lower average positive rating for environmental and social effects in calendar years (CY)14–16, by volume, was T mainly due to low share of satisfactory or better ratings (58 percent) for this subindicator in the CY14 Expanded Project Supervision Report Program cohort. In CY15 and CY16, the percentage of projects with satisfactory or better ratings for environmental and social effects were 62 percent and 74 percent, respectively. 27 n calendar years (CY)14–16, the Independent Evaluation Group (IEG) evaluated 16 International Finance Corporation I (IFC) investment projects (equivalent to $642 million in IFC net commitments or volume) in the Middle East and North Africa Region, of which only six (equivalent to $192 million by volume) had positive outcome ratings. By comparison, in CY11–13, IEG evaluated about the same number of IFC investment projects (17) in the Region and found 13 had positive outcome ratings (or 76 percent). The volume equivalent of the evaluated projects in CY11–13 was $511 million, of which 92 percent (equivalent to $472 milllion) had positive outcome ratings. Please refer to appendix L: Middle East and North Africa Region for details. 28 B  y number of projects, only 44 percent of projects in International Development Association (IDA) countries (IDA and blend countries) evaluated in calendar year (CY)14–16 had satisfactory or better ratings for the the quality of up-front work (screening, apparaisal, and structuring) by the International Finance Corporation—compared with 55 percent of projects in non-IDA countries. The share of positive ratings for the supervision and administration subindicator was lower in IDA countries (72 percent) compared with non-IDA countries (83 percent) among projects evaluated in CY14–16. By volume, the share of positive ratings for IFC’s up-front work was almost the same for projects in IDA and non-IDA countries. However, a larger share of projects in non-IDA countries had satisfactory or better ratings for supervision and administration (86 percent) compared with projects in IDA countries (68 percent). Independent Evaluation Group | World Bank Group 55 Source: Independent Evaluation Group evaluation database for International Finance Corporation Expanded Project Supervision Reports as of December 12, 2017. 29  or the current report, the Independent Evaluation Group used the 2017 Harmonized List of Fragile Situations to F classify fragility and conflict-affected situation (FCS) and Non-FCS countries. Regional and world projects were excluded from the classification. 30  f the 36 infrastructure projects (excluding oil, gas, and mining) evaluated in calendar years (CY)14–16, 16 were in the Power O subindustry group. Twelve of the 16 (75 percent) had mostly successful or better outcome ratings. By volume, the share of positively rated power sector projects in CY14–16 was 82 percent. Source: Independent Evaluation Group evaluation database for International Finance Corporation Expanded Project Supervision Reports as of December 12, 2017. 31  he share of positive outcome ratings for manufacturing, agribusiness, and services projects in the East Asia and T Pacific Region fell from 80 percent in calendar year (CY)11–13 to 30 percent in CY14–16, or a 50 percentage point decline, by number of projects and 59 percent by volume. 32  y number of projects, 49 percent of evaluated Financial Institutions Group projects in CY14–16 had positive B outcome ratings compared with 59 percent of those evaluated in CY11–13. Weighted by IFC net commitment amount, 49 percent of evaluated projects in CY14–16 were rated positively compared with 70 percent in CY11-13. Source: Independent Evaluation Group evaluation database for International Finance Corporation Expanded Project Supervision Reports as of December 12, 2017. 33 B  efore the Expanded Project Supervision Report (XPSR) instructions were revised in 2015, the work quality rating for the International Finance Corporation (IFC) included a third element: IFC’s role and contribution. In the 2015 XPSR instructions, IFC’s role and contribution became a fourth dimension in the measurement of IFC’s performance but amended as IFC’s Additionality. In the 2015 XPSR instructions, IFC’s additionality is defined as the benefit or value addition IFC brings that a client would not otherwise have. IFC’s additionality, therefore, is subset of its role that is unique to IFC and that cannot be filled by the client or any commercial financier. 34 Real and financial intermediary projects were determined using IFC’s Environmental and Social (E&S) Risk category.  Projects with E&S category FI or FI-1, FI-2 or FI-3 were considered financial sector projects. Projects with E&S category A, B, or C were considered real sector projects. 35 This finding is consistent with the results of multivariate statistical analysis performend by the Independent Evaluation  Group (IEG) in Results and Performance of the World Bank Group 2016, which found that the association between development outcome and International Finance Corporation (IFC) projects with combined advisory and investment support is a positive but not significant influencer of development outcomes in both real sector and financial sector projects. IEG’s 2017 evaluation, The International Finance Corporation’s Approach to Engaging Clients for Increased Development Impact, measured the association between development outcome and projects with IFC’s repeat client groups. Results and Performance of the World Bank Group 2017 used client-level flags as the basis for identifying projects with repeat clients. The evaluation concluded that rather than investing in a repeat client, what matters more is IFC’s ability to select sponsors with relevant market experience and the financial and technical capacity to operate and manage the project. 36 Although the association with development outcome ratings is significant, improvements in the IICCR index (at  approval and at evaluation) has a smaller likelihood (expressed in log-odds ratio) of increasing development outcome ratings. In particular, a 10-point increase in the IICCR ratings is associated with a 0.6 times increase in the likelihood of better development outcome ratings (refer to regression results in appendix J). 37 T  his finding is consistent with the survey findings in IEG’s Learning Product, “Lessons in IFC,” which concluded that IFC’s lesson learning system does not seem to be effective in improving performance of its projects, despite the high 56 Results and Performance of the World Bank Group 2017 | Chapter 2 value that staff accord to lesson learning. Source: CODE2016/0042. Lessons in IFC, Category I Learning Product. IEG. July 27, 2016. 38  mong the organizational factors identified were the high staff turnover, gaps in staff skills, inadequate industry A knowledge in the field offices, and an incentive system that rewarded volume instead of development outcomes. 39  he ratings were based on final ratings by the Independent Evaluation Group (IEG) in the Project Completion Report T (PCR) Evaluation Note (EvNote) and differs from self-ratings by the International Finance Corporation (IFC). For IFC advisory projects, the IEG validates IFC’s self-ratings of the evaluated advisory projects’ development effectiveness and IFC’s role and contribution. With the implementation of the revised guidelines for rating IFC advisory PCRs in FY15, IFC’s work quality was added as the third indicator of IFC’s performance. IEG assesses and rates IFC’s work quality; IFC does not provide self-assessement and rating for this indicator. The five subindicators of development effectiveness are (i) output achievement, (ii) outcome achievement, (iii) strategic relevance, (iv) efficiency, and (v) impact achievement. Development effectiveness is rated on a 6-point scale (from highest to lowest: highly successful, successful, mostly successful, mostly unsuccessful, unsuccessful, higly unsuccessful). IFC role and contribution and IFC work quality are rated on a 4-point scale (from highest to lowest: excellent, satisfactory, partly unsatisfactory, and unsatisfactory). 40  mong the multilateral institutions, only the International Finance Corporation (IFC) has a self-evaluation system A independently reviewed by the Independent Evaluation Group (IEG). The system was established in 2008 and, as of December 12, 2017, 520 IFC advisory projects had been self-evaluated by IFC and validated by IEG. IFC also conducts supplementary external reviews and evaluations of specific advisory projects and programs. 41 T  he Independent Evaluation Group (IEG) began rating the work quality of evaluated International Finance Corporation (IFC) Advisory Services projects in FY15. IFC project teams do not rate the work quality of their projects but are expected to provide an overview of their experience with project design and implementation and derive related lessons learned. They are also required to include an analysis of feedback received from the Advisory Services Client Survey and to provide comments (in the IFC comment section of the EvNote) on the IEG assessment. IFC’s work quality is rated by IEG along two dimensions: (i) project preparation and design and (ii) project implementation. IEG rates these dimensions on a 4-point scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory. 42  he Independent Evaluation Group began rating International Finance Corporation’s work quality in the Project T Completion Reports only in FY15. 43 T  he average project size (measured by total funding amount managed by the International Finance Corporation [IFC]) of the evaluated IFC advisory projects increased from $304,000 for the fiscal year (FY)08 evaluation program to $1.35 million and $1.38 million for the FY13 and FY16 evaluation programs, respectively. Average duration of the evaluated IFC advisory projects increased from 1.8 years for the FY08 evaluation program to 3.5 years and 3.3 years in FY13 and FY16, respectively. 44 Shortcomings are reflected in oversimplified objectives that do not reflect the market failure that the projects are  intended to address; inadequate selection of indicators or using indicators with unclear definitions and missing targets; and data inconsistencies across project documents. Information on the impacts of International Finance Corporation delivery of Advisory Services on long-term, sustainable private sector development was found to be patchy. 45  he Multilateral Investment Guarantee Agency’s development outcome is evaluated across four dimensions: T (i) business performance; (ii) economic sustainability; (iii) environmental and social effects; and (iv) contribution to private sector development. Development outcome is rated on a 4-point scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory. 46 O  nly the results of 10 or more evaluated projects were reported because of the small number of evaluated Multilateral Investment Guarantee Agency (MIGA) projects. For FY11–17, the 2 evaluated projects in South Asia, 4 of 5 in the Middle East and North Africa, and 5 of 9 in East Asia and Pacific were rated satisfactory or better for development Independent Evaluation Group | World Bank Group 57 outcome. Because of the small number of evaluated MIGA projects in these three regions, IEG did not compare the development outcome ratings by region. 47  or this report, the Independent Evaluation Group used the 2017 Harmonized List of Fragile Situations to classify F fragile and conflict-affected situation (FCS) and non-FCS countries. Regional and world projects were excluded from the classification. 48  he share of satisfactory or better ratings in project business performance of projects by the financial and capital T markets group dropped to 46 percent in FY11–16 from 71 percent in FY05–10, a 26 percentage decline. Positive business performance ratings of projects by the infrastructure group (excluding energy projects) also declined sharply, from 75 percent in FY10–15 to 53 percent in FY11–16. 49  he main issue with the seven projects that had no opinion possible (NOP) ratings in development outcome in the T Results and Performance of the World Bank Group 2016 was the inadequate or insufficient information collected by the Multilateral Investment Guarantee Agency (MIGA) after the contract of guarantees was issued to allow an objective opinion regarding their outcomes. A re-rating exercise was undertaken by the Independent Evaluation Group, in consultation with MIGA, to eliminate the NOP ratings in this report. Going forward, evaluated MIGA projects that lack information will be rated low (partly unsatisfactory or unsatisfactory), consistent with the rating criteria applied to the International Finance Corporation and World Bank project evaluations that have inadequate or insufficient information. 50  he Small Investment Program (SIP) of the Multilateral Investment Guarantee Agency (MIGA), approved by MIGA’s T Board of Directors in March 2004, aimed to fill the unmet demand for political risk coverage in small investment projects, including for small and medium enterprises (SMEs), unavailable from commercial insurers. To be eligible as an SME, the investee company must meet two of the following three size-related criteria: (i) feweer than 300 employees, (ii) annual revenues of up to $15 million, and (iii) total assets of up to $15 million. For investments in financial institutions to qualify for SIP, the investee financial institution must have a borrower base of at least 50 percent SMEs, based on the same definition. The SIP is intended for investment projects with relatively simple structures that do not raise significant environmental or social concerns. To reduce transaction costs, MIGA designed a standardized contract of guarantee that combined coverage against currency transfer restriction and inconvertibility, war and civil disturbance, and expropriation; and MIGA streamlined the underwriting and approval procedures. MIGA Board approval was waived and project approval delegated to the MIGA Director of Operations. Under the original 2004 SIP policy, MIGA guarantees of up to $5 million for cross-border investments into SMEs were eligible. The program was originally open to all sectors. Projects classified as category A under MIGA’s environmental and social policies and investments requiring coverage for breach of contract risk were ineligible under the SIP. At the end of the SIP pilot phase in 2007, MIGA increased the threshold of the guarantee amount eligible under SIP from $5 million to $10 million and restricted eligible projects to those within the financial and agribusiness, manufacturing, and services sectors only. In 2014, MIGA strengthened its policy for supporting SMEs, including the SIP. Among the changes introduced were limits to the number of projects with less than $40 million in guarantee coverage to approximately 15 projects each fiscal year and the elimination of the SIP pricing subsidy. 51  he review of the Small Investment Program undertaken by the Independent Evaluation Group involved desk-based T review of 41 operationally mature Small Investment Program projects in FY06–11, supplemented by site visits to 6 projects. 52  he Country Partnership Framework (CPF), prepared in collaboration with a member country and starting from the T country’s own vision of its development goals, “identifies the key objectives and development results through which the World Bank Group intends to support a member country in its efforts to end extreme poverty and boost shared prosperity in a sustainable manner.” (See the Country Partnership Framework Collection withn the World Bank’s Open Knowledge Repository, at https://openknowledge.worldbank.org/handle/10986/23100). A CPF draws on a Systematic Country Diagnostic and includes a proposed package of investments and knowledge services (covering the World Bank, International Finance Corporation, and Multilateral Investment Guarantee Agency, as appropriate) needed to implement the strategy. The Independent Evaluation Group reviews the performance of 58 Results and Performance of the World Bank Group 2017 | Chapter 2 the strategy and approach in each country as defined in the CPF or Country Assistance Strategy. These reviews are known as Completion and Learning Reviews. 53 n Completion and Learning Reviews (CLRs), the Independent Evaluation Group (IEG) assigns ratings of country I development outcome and of World Bank Group performance. Ratings of country development outcome are based on assessment of the extent to which objectives in the Country Partnership Framework were achieved. Ratings of World Bank Group performance are based on a structured assessment of the institution’s performance in design and implementation of the program. The methodology for IEG’s CLRs changed in December 2013 with the introduction of the “shared approach.” At that point, the rating scale for World Bank Group performance was changed from a 6-point scale (from lowest to highest: highly unsatisfactory, unsatisfactory, moderately unsatisfactory, moderately satisfactory, satisfactory, highly satisfactory) to a 4-point scale (from lowest to highest: poor, fair, good, superior). (Based on IEG’s CLR database, no CLR had ever assigned ratings of either highly unsatisfactory or highly satisfactory on World Bank Group performance when the 6-point scale was in use.) Fiscal year (FY)15 was the first year all CLRs used the 4-point scale for World Bank Group performance. To include a sufficient quantity of data points in each period to make some disaggregation possible, trend data are reported on a rolling basis using four-year periods. The most recent four-year period for which CLR data are available is FY14–17. 54 F  or this analysis, the following countries were categorized as International Development Association–International Bank for Reconstruction and Development blend during FY14–17: Cabo Verde, Cameroon, Dominica, Grenada, Kenya, Moldova, Nigeria, Pakistan, St. Lucia, St. Vincent and the Grenadines, and Uzbekistan, based on the World Bank List of Economies, June 2017 at https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world- bank-country-and-lending-groups. 55 S  ignificance of differences is not assessed because of the low number of strategies involved. It is also important to note that the number of strategies evaluated varies over time and among Regions, and performance in a few countries can affect overall performance. 56  he Independent Evaluation Group started rating achievement of World Bank Group outcomes or objectives in T mid–fiscal year (FY)14. The outcomes or objectives are rated on a 4-point scale: achieved, mostly achieved, partially achieved, and not achieved. Those rated achieved and mostly achieved are considered achieved for this review. Independent Evaluation Group | World Bank Group 59 3 highlights Tracking 1 Although the Independent Evaluation Group Implementation (IEG) rated progress on implementation of action through the plans in their fourth year of implementation Management as substantial or above (substantial, Action Record high, or complete) for over 90 percent of recommendations, only 52 percent were rated high or complete and 20 percent complete. 2 Roughly half (47 percent) of recommendations on environmental sustainability from IEG evaluations have been implemented to at least a high degree, based on the deep dive undertaken for this report. 3 Analysis of recommendations in IEG’s 2017 Management Action Record review finds that Management Action Plans are not always aligned with the intent of IEG’s recommendations. 4 Despite recent improvements in its process and its ongoing pilots, the full potential of the Management Action Record as an instrument for development effectiveness has yet to be realized. x 60 T H E M A N AG E M E N T AC T I O N R E C O R D (MAR) tracks the progress of the implementation of Bank Group Management Action Plans created in response to recommendations from IEG’s corporate, sector, and thematic evaluations. The MAR can serve as a tool for accountability and learning, through which IEG and the management of the Bank Group institutions keep the Committee on Development Effectiveness (CODE) and the public informed of implementation progress. A reform of the MAR process in 2011 resulted in the joint agreement between IEG and management to introduce Management Action Plans that had specific targets and indicators and to rate progress against those action plans (and no longer to rate progress against adoption of IEG’s recommendations). This change was intended to reduce ambiguity about what management committed to implement, but it may also have shifted management’s focus away from action on IEG recommendations. The analysis in this chapter is based on the ratings and reviews of FY17 management implementation updates, previous MAR data, focus group discussions with IEG and the management of the Bank Group institutions on evaluations exiting the MAR update process, and a deep dive related to environmental sustainability—the latter being a first in the RAP. Trends in Implementation of IEG Recommendations and Management Action Plans Although progress on recommendations in their fourth year of implementation was rated substantial or above (substantial, high, or complete) for over 90 percent of recommendations in RAP17,1 only 52 percent were rated high or complete and 20 percent complete based on IEG ratings (figure 3.1). The percentage rated substantial, high, or complete was over 90 percent in RAP17, compared with 72 percent in RAP16 and 80 percent in RAP15. In 2017, IEG tracked implementation of actions for 146 active recommendations across the Bank Group drawn from the 23 evaluations produced between FY13 and FY16. Due to the cumulative nature of the rating system, the implementation progress typically advances throughout the four-year tracking cycle.2 Recommendations focusing on implementing programs or projects are less likely to be fully implemented. A review of of all active recommendations shows that 66 percent of IEG’s recommendations in this year’s set are in the corporate and institutional category, followed by 44 percent in the knowledge, measurement, and impact category.3 Completion is lower among the recommendations in the implementing programs or projects category, followed by the knowledge, Independent Evaluation Group | World Bank Group 61 FIGURE 3.1 |R  atings of Implementation of Recommendations by the Independent Evaluation Group and Management of the World Bank Group Institutions (Cumulative) IEG 20% 19% 9% 19% 32% 15% 8% 35% 43% 41% 22% –5% –15% –2% –35% –44% –3% –5% –26% 1st Year 2nd Year 3rd Year 4th Year (N=144) (N=130) (N=99) (N=44) World Bank Group 25% 38% 4% 23% 36% 18% 24% 50% 37% 33% 35% 13% –20% –4% –32% –8% 1st Year 2nd Year 3rd Year 4th Year` (N=145) (N=119) (N=77) (N=32) Complete High Substantial Moderate Negligible Note: IEG did not rate two recommendations in the first year of implementation because the Management Action Plans were not aligned with the intent of IEG’s recommendation. Thus N = 144 for IEG instead of 146. IEG = Independent Evaluation Group. 62 Results and Performance of the World Bank Group 2017 | Chapter 3 measurement, and impact category (figure 3.2). One explanation for this could be that these types of recommendations are partially within the control of the management of the Bank Group institutions, and they may require buy-in from governments who implement the projects. Management typically rates progress in implementing action plans higher than IEG does and does so earlier in the MAR update process. As seen in figure 3.1, IEG rated only 30 percent of recommendations as substantial or above (that is, substantial, high, or complete) in their first year of implementation, whereas management rated 59 percent substantial and above. In addition, management rates itself as high or complete for 22 percent of recommendations in the first year of implementation, whereas IEG rated only 8 percent as high and none as complete. Management unilaterally retires some recommendations before full implementation of their action plans. About 18 percent of all recommendations in this year’s set were retired by management earlier than IEG’s assessment of full implementation. (IEG did not retire them because it did not agree with management that the action plans were fully implemented.) IFC used early retirement of recommendations more frequently than the World Bank. (Among recommendations retired early by management, 85 percent were retired by IFC.) Although the World Bank did not retire FIGURE 3.2 |C  omparison of Recommendations Rated Substantial or Above and High and Complete by the Independent Evaluation Group, by Category High-Level Policy or Implement programs Knowledge, Measurement, Corporate and Strategic Direction or Projects and Impact Analysis Institutional 100% 100% 86% 91% 89% 81% 86% 82% 79% 65% 58% 46% 47% 52% 39% 54% 22% 42% 33% 36% 20% 26% 29% 20% 10% 12% 24% 13% 0% 3% 1st 2nd 3rd 4th 1st 2nd 3rd 4th 1st 2nd 3rd 4th 1st 2nd 3rd 4th Year Year Year Year Year Year Year Year Year Year Year Year Year Year Year Year IEG H+ IEG S+ Source: Data from IEG’s Management Action Record database. Note: Please note that some recommendations were tagged in more than one category. IEG = Independent Evaluation Group; H+ = high and complete; S+ = substantial and above. Independent Evaluation Group | World Bank Group 63 recommendations early as often as IFC did, for some recommendations the World Bank did not provide any new updates and kept the rating at high or complete as in the previous year, which placed them in a relatively stable status. Although the number of partially agreed recommendations declined right after the MAR reform, it has been higher in the past two years. The MAR tracks only those recommendations that are agreed or partially agreed on by management.4 No instances of partial agreement occurred for IEG evaluations disclosed in 2012. In 2013–14, management gave 4 partial agreements. In this year’s set of active recommendations, management partially agreed with 15, of which 11 were from evaluations disclosed in 2015. Of the partially agreed recommendations, around 67 percent are in the corporate and institutional category and about 47 percent in the knowledge, measurement, and impact category. Ambiguity around partial agreements can mean that CODE is not clearly informed at the time of the formal discussion on how this partial disagreement will affect management’s follow-up (including how closely the intent of IEG’s recommendation will be addressed in the Management Action Plan) and IEG’s monitoring of that follow-up. Deep Dive on Recommendations on Environmental Sustainability Forty-seven percent of recommendations on environmental sustainability from IEG evaluations were implemented to a high or complete degree and 80 percent to at least a substantial degree. From evaluations completed during FY08–16, IEG identified and reviewed 59 recommendations across the Bank Group relating to environmental sustainability.5 These recommendations covered environmental mainstreaming (4), climate change mitigation (14) and adaptation (16), natural resource management (15), and E&S risk management (20). Appendix K describes the progress made and gaps that remain in implementing IEG’s recommendations. Environmental recommendations relating to knowledge work, measurement systems, or assessment of the impact of Bank Group interventions have been implemented far less by the management of the Bank Group institutions than other recommendations. On average, they received final implementation ratings a full rating level lower than other recommendations (that is, those relating to high-level policy or strategic direction, how to implement programs or projects, and corporate or institutional issues such as guidance, incentives, and structure) and were half as likely to be implemented to at least a substantial standard (see appendix K). Environmental recommendations on policy and strategy changes (9) were implemented when the recommendation aligned with a policy update or strategy process already under way. The recommendation was more likely to influence the process than to trigger it. In cases where the Bank Group did not intend to implement the recommendation, this was usually clear from the outset and saw disagreement or partial agreement with the recommendation by management.6 The only policy or strategy recommendation agreed on and not substantially implemented was for scale-up of energy efficiency by the World Bank, where management updates noted challenging market conditions following declining energy prices. Beyond environmental evaluations, focus groups with IEG and management over the past two years unanimously highlight the sentiment that management mostly adopts strategies and makes corporate commitments that have been 64 Results and Performance of the World Bank Group 2017 | Chapter 3 previously agreed on at the corporate level as opposed to adopting something new in response to IEG recommendations. For environmental recommendations on how to implement programs or projects (12), the World Bank often implemented these after carrying out deeper self-assessments triggered by the recommendation, but these types of recommendations had less influence on IFC. The World Bank often undertook analytical work aimed at informing future operational design,7 but the main challenge has been to move from production of analytical work toward widespread use of it in the institution. IFC was less likely to use these types of recommendations for strategic or programmatic decision making. IFC actions were less ambitious and more likely to already be under way at the time of the recommendation.8 IFC typically engages in projects at the stage leading up to the transaction rather than the early design stage; this means that IFC is less able than the World Bank to influence the design of projects it finances. Implementation of environmental recommendations on corporate and institutional measures (17) depended on the degree of change required by the recommendation. For recommendations requiring only outputs, such as the creation of guidance, implementation was high. But for those requiring a larger institutional change, especially in internal incentives, progress was partial. In some cases, corporate recommendations took longer to implement, and changes were completed after the MAR process had ended. Environmental recommendations to generate knowledge, enhance use of metrics, or measure the impact of Bank Group interventions (13) were the least likely to be fully implemented. Common themes were the need to provide better evidence on the impact of Bank Group interventions and to better report on environmental impacts of projects, especially outside of the core environmental interventions. Potential reasons for a lack of implementation could include (i) a lack of qualified and trained Bank Group staff available to support M&E initiatives, (ii) a lack of resources devoted to M&E initiatives, (iii) a lack of capacity for M&E in client countries and the limited size of programs aimed at building such capacity, and (iv) secondary prioritization of knowledge and learning behind operational delivery. Have Action Plans Made a Difference? Management Action Plans are not always aligned with the intent of IEG’s recommendations. Action plans are not discussed by CODE or Board members, and IEG acceptance of action plans is not required. Management updates during the implementation tracking years concentrate solely on the letter of the action plan because progress is rated against action plans and not against the underlying recommendation. Rating against action plans means that MAR ratings do not capture the extent to which IEG recommendations have been achieved. Consequently, the risk exists that IEG recommendations remain underaddressed even when action plans are fully implemented. In addition, this year IEG, for the first time, decided not to follow the implementation progress of the World Bank and IFC action plans for one of its recommendations because the action plans were not aligned Independent Evaluation Group | World Bank Group 65 with the intent of the recommendation.9 Action plans may commit to producing outputs rather than systemic change. For example, the analysis of studies focusing on environment shows that action plan items have often been unambitious and insufficient to achieve the outcome envisioned in the recommendation. The review of action plans for two of the evaluations covered in the assessment of recommendations on environmental sustainability shows that action plans made data more accessible, better structured, and easier to collate and use across evaluation periods. Yet, most action items were outputs; action plans were seldom used as results indicators to track progress (table 3.1). Focus group discussions conducted by IEG for the studies that exited the MAR update cycle in 2017 also highlighted the perceived misalignment between IEG’s recommendations and action plans.10 Management counterparts suggested in focus groups that action plans are very rigid and do not allow for changes to take place based on changing development context over the four years, and thus, management is less likely to commit to something more ambitious. IEG participants indicated that they frequently found management’s action plans unambitious in relation to IEG recommendations. Management counterparts reported in focus groups that action plan items were typically derived from activities already present in their work program. TABLE 3.1 | Action Plan Items For Environment Evaluations Have Largely Been Outputs Number Action Type of Items Results Hold workshop, event, 5 In four cases, these were not held; in one case, a workshop was or training held but the effect was unclear. Create tools/guidance 10 These were l argely achieved but saw limited use by operations. Create knowledge or 11 Products were largely produced but had limited influence on analytical product operations. Create strategic 1 These were fully achieved. approach or road map Strengthen 7 These were largely achieved, but did not require tangible collaboration or outcomes—actions were deemed achieved if information exchange partnerships and cooperation took place. Change the nature or 9 Significant changes were seen in some areas but not others. design of projects Improve monitoring 4 Indicators were developed but saw limited use in operations. and evaluation Source: Independent Evaluation Group analysis of Management Action Record updates for forestry and climate change adaptation evaluations. 66 Results and Performance of the World Bank Group 2017 | Chapter 3 Focus group discussions further revealed that the drafting of action plans and reporting on implementation progress are mechanical rather than substantive. Both IEG and management counterparts mentioned that MAR updates are more of a box-ticking exercise. Similar findings were reported in RAP15, which highlighted that IEG and management deem the MAR update process as “bureaucratic and limiting” and feel that “the move toward actions and timelines further emphasized the mechanical dimension of assessing progress made toward implementation” (World Bank 2016e, 70–72). To realize the full potential of the MAR as an instrument for improving the Bank Group’s development effectiveness, two types of changes would be required. The first is to improve the engagement process between the management of the Bank Group institutions and IEG. The second is to improve the implementation of IEG recommendations by the management of the Bank Group institutions. To improve engagement around major evaluations, management of the Bank Group institutions and IEG are undertaking seven pilots. IEG and management now share greater engagement in the process of conducting the evaluations, and the MAR is now online. Currently, Management Action Plans, once articulated, are not revised over the four-year MAR cycle. To maintain the relevance of action plans, IEG and management initiated two pilots for adaptable action plans in FY16 and FY17, which allow more flexibility in changing the action plans based on development needs while rating the progress against outcomes envisioned in IEG’s recommendations.11 In addition, management and IEG are together piloting changes to the MAR process, for example, incorporating management feedback on draft IEG recommendations. The implementation of these seven pilots is under way, and their effectiveness remains to be assessed.12 If the experience is well monitored and the right lessons are drawn, these pilots can be expected to help bring about improvements in the engagement process. Improving the implementation of IEG recommendations would require greater attention on the part of the management of the Bank Group institutions to evaluation findings and more action to follow up on them. The fact that only 52 percent of the actions that management identified for itself were implemented to a high or complete extent in the fourth year of implementation (with 20 percent complete) may point to the lack of an organizational environment in which findings from evaluation— whether from IEG or others—inform and underpin the World Bank’s operations as a matter of course. 1 D  uring the Management Action Record (MAR) updates, each recommendation may stay in the update cycle for four years to complete and exit the cycle or may be retired earlier than the fourth year. If a recommendation is retired earlier than the fourth year, it remains in the set of active recommendations with an inactive status and is labeled as retired, but it only exits the cycle at the end of the fourth year. The calculations of the fourth-year ratings in this paragraph are based only on the recommendations that stayed in the MAR update cycle until their fourth year. 2 I mplementation of each recommendation is tracked and recorded over four years, with the management of the World Bank Group institutions and the Independent Evaluation Group making their own assessments using a 5-point scale that includes the following ratings: negligible (no plan or evidence of implementation), moderate (<50 percent complete), substantial (>50 percent complete), high (≥90 percent complete), and complete. 3 T  he Independent Evaluation Group analyzed and coded all recommendations into categories based on frequently recurring themes. Some recommendations fall into more than one category, and categories are not mutually exclusive. Independent Evaluation Group | World Bank Group 67 In general, over 65 percent of recommendations in this year’s set focus on the corporate and institutional improvement category, and over 40 percent on the knowledge, measurement, and impact category. In the latter category, about 20 percent of recommendations focused on improving internal World Bank Group capacity. 4  anagement may agree, disagree, or partially agree with Independent Evaluation Group recommendations, stating its M reasoning for the degree of acceptance in the formal management response. 5 R  ecommendations are tracked and rated separately across institutions. The 59 recommendations include copies of a recommendation across institutions, so for example, one recommendation applied to both the World Bank and the International Finance Corporation would count as two recommendations. 6  or example, a recommendation to reexamine how the World Bank Group engages in industrial timber concession F reform in moist forests was rejected in part because the institution was not in a position to create the political and budgetary space for a reform that would have required analysis and time to conduct consultations, including with civil society. However, a portfolio review undertaken as part of the Management Action Record revealed that only one investment had taken place in this sensitive area. 7  or example, in response to the Independent Evaluation Group’s agriculture recommendation to take stock of its F experience in water management and crop technologies in rain-fed areas, the World Bank undertook analytical work and carried out a portfolio review of rain-fed agriculture projects, which helped inform future World Bank Group support. Likewise, in the forest sector, the World Bank developed a biodiversity road map that lays out the institutional approach for mainstreaming biodiversity into World Bank Group operations. 8  or example, an action plan item to “implement three projects with participatory forest management” covered only F projects already in progress. 9 This recommendation was from Financial Inclusion—A Foothold on the Ladder toward Prosperity? (World Bank 2015f). 10 n this year’s Management Action Record update cycle, the following studies were in their final year of implementation I tracking and will no longer be tracked in the system: Adapting to Climate Change: Assessing the World Bank Group Experience Phase III; Capturing Technology for Development—An Evaluation of the World Bank Group Activities in Information and Communication Technologies; The World Bank’s Support to Global Food Crisis: Biennial Report on Operational Effectiveness of IFC and MIGA (BROE); Knowledge-Based Country Programs: An Evaluation of the World Bank Group Experience; World Bank Group Support to Youth Employment Programs; Managing Forest Resources for Sustainable Development: An Evaluation of World Bank Group Experience; Evaluation of the International Finance Corporation’s Global Trade Finance Program; and Improving Institutional Capability and Financial Viability to Sustain Transport: An Evaluation of World Bank Group Support Since 2002. 11  ecommendations from the evaluation Behind the Mirror and Learning and Results in World Bank Operations: Toward R a New Learning Strategy (Phase II) were rated differently than the recommendations from other evaluations in a pilot of “adaptable action plans.” In the case of these pilots, the Independent Evaluation Group rated the progress against outcomes stated in recommendations rather than against management’s action plans. In addition, management was given the flexibility to adjust action plans each year in response to the strategic changes taking place across the institution. The implementation of these two pilot action plans is still under way, and the assessment of their effectiveness will come at a later stage. 12 T  he seven pilots are (i) PROACT Workshops designed for soliciting feedback on Independent Evaluation Group (IEG) approach papers; (ii) Track Your Evaluation pilot for feedback during the evaluation cycle through a dedicated web platform; (iii) Recommendations Formulation, designed to allow management to draft recommendations in one case and (iv) IEG and management draft recommendations in the other case; (v) REACT workshops, designed to solicit early and informal feedback on IEG’s recommendations; (vi) Action Plan Development Workshop, designed to solicit IEG’s in-depth feedback on the action plan; and (vii) Adaptable Action Plans designed to allow Management flexibility in adjusting action plans during the implementation cycle. 68 Results and Performance of the World Bank Group 2017 | Chapter 3 bibliography Blanc, Marc, Talib Esmail, Caroline Mascarell, and Rukshan Rodriguez. 2016. “Predicting Project Outcomes: A Simple Methodology for Predictions Based on Project Ratings.” Policy Research Working Paper No. WPS7800, World Bank, Washington, DC. Blum, Jurgen Rene. 2014. “What Factors Predict How Public Sector Projects Perform?” Policy Research Working Paper 6798, World Bank, Washington, DC. Buntaine, Mark T., and Bradley C. Parks. 2013. “When Do Environmentally Focused Assistance Projects Achieve Their Objectives? Evidence from World Bank Post-Project Evaluations.” Global Environmental Politics 13 (2): 65–88. Chauvet, Lisa, Paul Collier, and Marguerite Duponchel. 2010. “What Explains Aid Project Success in Post-Conflict Situations?” Policy Research Working Paper 5418, World Bank, Washington, DC. Coppola, Andrea, Fernando Fernholz, and Graham Glenday. 2014. “Estimating the Economic Opportunity Cost of Capital for Public Investment Projects: An Empirical Analysis of the Mexican Case.” Policy Research Working Paper 6816. World Bank, Washington, DC. https://openknowledge.worldbank.org/handle/10986/17724. Deininger, Klaus, Lyn Squire, and Swati Basu. 1998. “Does Economic Analysis Improve the Quality of Foreign Assistance?” World Bank Economic Review 12 (3): 385–418. Denizer, Cevdet, Daniel Kaufmann, and Aart Kraay. 2011. “Good Countries or Good Projects? Macro and Micro Correlates of World Bank Project Performance.” Journal of Development Economics 105: 288–302. Dollar, David, and Victoria Levin. 2005. “Sowing and Reaping: Institutional Quality and Project Outcomes in De- veloping Countries.” Policy Research Working Paper 3524, World Bank, Washington, DC. Dreher, Axel, Stephan Klasen, James Raymond Vreeland, and Eric Werker. 2013. “The Costs of Favoritism: Is Politically Driven Aid Less Effective?” Economic Development and Cultural Change 62 (1): 157–191. Geli, Patricia, Aart Kraay, and Hoveida Nobakht. 2014. “Predicting World Bank Project Outcome Ratings.” Policy Research Working Paper 7001, World Bank, Washington, DC. Girod, Desha M., and Jennifer L. Tobin. 2016. “Take the Money and Run: The Determinants of Compliance with Aid Agreements.” International Organization 70 (1): 209–239. Higgins, Julian, and Sally Green. 2008. Cochrane Handbook for Systematic Reviews of Interventions. London: Cochrane Collaboration. IFC (International Finance Corporation). 2012a. International Finance Corporation Articles of Agreement. Washing- ton, DC: World Bank Group. ———. 2012b. International Finance Corporation’s Policy on Environmental and Social Sustainability. Washington, DC: World Bank Group. Ika, Lavagnon A. 2015. “Opening the Black Box of Project Management: Does World Bank Project Supervision Influence Project Impact?” International Journal of Project Management 33 (5): 1111–1123. Isham, Jonathan, and Daniel Kaufmann. 1999. “The Forgotten Rationale for Policy Reform: The Productivity of Investment Projects.” Quarterly Journal of Economics 114 (1): 149–184. Kilby, Christopher. 2000. “Supervision and Performance: The Case of World Bank Projects.” Journal of Develop- ment Economics 62 (1): 233–259. Kilby, Christopher. 2015. “Assessing the Impact of World Bank Preparation on Project Outcomes.” Journal of Development Economics 115: 111–123. Independent Evaluation Group | World Bank Group 69 Kumar, Anjali, Sushma Narain, and Swizen Rubbani. 2015. “World Bank Lending for Financial Inclusion: Lessons from Reviews of Select Projects.” Independent Evaluation Group Working Paper Series No. 2015/1, World Bank, Washington, DC. Kuo, Chun-Yan, Glenn P. Jenkins, and M. Benjamin Mphahlele. 2003. “The Economic Opportunity Cost of Capital in South Africa.” The South African Journal of Economics 71: 3. http://onlinelibrary.wiley.com/ doi/10.1111/j.1813-6982.2003.tb00084.x/pdf. Laajaj, Rachid, and Patrick Guillaumont. 2006. “When Instability Increases the Effectiveness of Aid Projects.” Policy Research Working Paper 4034, World Bank, Washington, DC. Legovini, Arianna, Vincenzo Di Maro, and Caio Piza. 2015. “Impact Evaluation Helps Deliver Development Proj- ects.” Policy Research Working Paper 7157, World Bank, Washington, DC. Limodio, Nicola. 2011. “The Success of Infrastructure Projects in Low-Income Countries and the Role of Selectivi- ty.” Policy Research Working Paper 5694, World Bank, Washington, DC. Lopez, Humberto. 2008. “The Social Discount Rate: Estimates for Nine Latin American Countries.” Policy Research Working Paper 4639. World Bank, Washington, DC. https://openknowledge.worldbank.org/han- dle/10986/6659. MIGA (Multilateral Investment Guarantee Agency). 2013. “Policy on Environmental and Social Sustainability.” Washington, DC: World Bank Group. Moll, Peter G., Patricia Geli, and Pablo Saavedra. 2015. “Correlates of Success in World Bank Development Poli- cy Lending.” Policy Research Working Paper 7181, World Bank, Washington, DC. Pohl, Gerhard, and Dubravko Mihaljek. 1992. “Project Evaluation and Uncertainty in Practice: A Statistical Analysis of Rate-of-Return Divergences of 1,015 World Bank Projects.” World Bank Economic Review 6 (2): 255–277. Presbitero, Andrea F. 2016. “Too Much and Too Fast? Public Investment Scaling-Up and Absorptive Capacity.” Journal of Development Economics 120: 17–31. Prokopy, Linda Stalker. 2005. “The Relationship between Participation and Project Outcomes: Evidence from Rural Water Supply Projects in India.” World Development 33 (11): 1801–1819. Ramsey, Frank Plumpton. 1928. “A Mathematical Theory of Saving.” Economic Journal 38 (152): 543–559. Raimondo, Estelle. 2016. “What Difference Does Good Monitoring and Evaluation Make to World Bank Project Performance?” Policy Research Working Paper 7726, World Bank, Washington, DC. Ralston, Laura. 2014. “Success in Difficult Environments: A Portfolio Analysis of Fragile and Conflict-Affected States.” Policy Research Working Paper 7098, World Bank, Washington, DC. Smets, Lodewijk, Stephen Knack, and Nadia Molenaers. 2013. “Political Ideology, Quality at Entry and the Suc- cess of Economic Reform Programs.” Policy Research Working Paper 6130, World Bank, Washington, DC. Vawda, Ayesha Yaqub, Peter Moock, J. Price Gittinger, and Harry Anthony Patrinos. 2003. “Economic Analysis of World Bank Education Projects and Project Outcomes.” International Journal of Educational Development 23 (6): 645–660. World Bank. 1998. Handbook on Economic Analysis of Investment Operations. Washington, DC: World Bank. ———. 2001. Making Sustainable Commitments: An Environment Strategy for the World Bank. Washington, DC: World Bank. 70 Results and Performance of the World Bank Group 2017 | References ———. 2008. Environmental Sustainability: An Evaluation of World Bank Group Support. Independent Evaluation Group. Washington, DC: World Bank. ———. 2009a. Annual Review of Development Effectiveness: Achieving Sustainable Development. Independent Evaluation Group. Washington, DC: World Bank. ———. 2009b. Development and Climate Change: A Strategic Framework for the World Bank Group. Washington, DC: World Bank. ———. 2010. Safeguards and Sustainability Policies in a Changing World. Independent Evaluation Group. Wash- ington, DC: World Bank. ———. 2011. Social Safety Nets: An Evaluation of World Bank Support, 2000–2010. Independent Evaluation Group. Washington, DC: World Bank. ———. 2012a. Inclusive Green Growth: The Pathway to Sustainable Development. Washington, DC: World Bank. ———. 2012b. The Matrix System at Work: An Evaluation of the World Bank’s Organizational Effectiveness. Inde- pendent Evaluation Group. Washington, DC: World Bank. ———. 2012c. Moving Beyond GDP: How to Factor Natural Capital Accounting into Decision Making. Washington, DC: World Bank. ———. 2012d. Toward a Green, Clean, and Resilient World for All: A World Bank Group Environment Strategy 2012–2022. Washington, DC: World Bank. ———. 2013a. Toward a Sustainable Energy Future for All: Directions for the World Bank Group’s Energy Sector. Independent Evaluation Group. Washington, DC: World Bank. ———. 2013b. World Bank Group Strategy. Washington, DC: World Bank. ———. 2014. The Big Business of Small Enterprises: Evaluation of the World Bank Group Experience with Targeted Support to Small and Medium-Size Enterprises, 2006–2012. Independent Evaluation Group. Washington, DC: World Bank. ———. 2015a. Development Policy Financing Retrospective Results and Sustainability. Washington, DC: World Bank. ———. 2015b. Financial Inclusion: A Foothold on the Ladder Toward Prosperity? Washington, DC: World Bank. ———. 2015c. Learning and Results in World Bank Operations: Toward a New Learning Strategy. Independent Evaluation Group. Washington, DC: World Bank. ———. 2015d. Learning Note: Additional Financing for Transport and Information and Communication Technology (ICT). Independent Evaluation Group. Washington, DC: World Bank. ———. 2015e. Managing Environmental and Social Risks in Development Policy Financing. Independent Evaluation Group. Washington, DC: World Bank. ———. 2015f. Results and Performance of the World Bank Group 2014. Independent Evaluation Group. World Bank, Washington, DC. ———. 2016a. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Independent Evaluation Group. Washington, DC: World Bank. Independent Evaluation Group | World Bank Group 71 ———. 2016b. Lessons from Environmental Policy Lending. Independent Evaluation Group. Washington, DC: World Bank. ———. 2016c. Lessons from Land Administration Projects: A Review of Project Performance Assessments. Inde- pendent Evaluation Group. Washington, DC: World Bank. ———. 2016d. Program-for-Results: An Early-Stage Assessment of the Process and Effects of a New Lending Instrument. Independent Evaluation Group. Washington, DC: World Bank. ———. 2016e. Results and Performance of the World Bank Group 2015. Independent Evaluation Group. Washing- ton, DC: World Bank. ———. 2016f. World Bank Group Engagement in Small States: The Cases of the OECS, PICs, Cabo Verde, Djibou- ti, Mauritius, and the Seychelles. Independent Evaluation Group. Washington, DC: World Bank. ———. 2016g. World Bank Group Joint Projects: A Review of Two Decades of Experience—Lessons and Implica- tions from Evaluation. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017a. Mobile Metropolises: Urban Transport Matters. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017b. Toward a Clean World for All: An IEG Evaluation of the World Bank Group’s Support for Pollution Management. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017c. World Bank Group Country Engagement: An Early-Stage Assessment of the Systematic Country Di- agnostic and Country Partnership Framework Process and Implementation. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017d. Results and Performance of the World Bank Group 2016. Independent Evaluation Group. Washing- ton, DC: World Bank. ———. 2017e. Lessons in IFC. Independent Evaluation Group. Washington, DC: World Bank. ———. 2018a. World Bank Group Support to Health Services: Achievements and Challenges. Independent Evalu- ation Group. Washington, DC: World Bank. ———. 2018b. The International Finance Corporation’s Approach to Engaging Clients for Increased Development Impact. Independent Evaluation Group. Washington, DC: World Bank. World Bank Group. 2014. World Bank Group Directive: Country Engagement. Washington, DC: World Bank. ———. 2016a. World Bank Group Climate Change Action Plan, World Bank Group Washington, DC: World Bank Group. ———. 2016b. World Bank Group Forest Action Plan FY16–20. Washington, DC: World Bank Group. ———. 2017a. “Forward Look—A Vision for the World Bank Group in 2030.” Washington, DC: World Bank Group. ———. 2017b. World Bank Group/World Bank Corporate Scorecards, October 2017. Washington, DC: World Bank Group. Yalegama, Sugath, Nicholas Chileshe, and Tony Ma. 2016. “Critical Success Factors for Community-Driven Development Projects: A Sri Lankan Community Perspective.” International Journal of Project Management 34 (4): 643–659. 72 Results and Performance of the World Bank Group 2017 | References A PPE NDI X E S Results and Performance of the World Bank Group 2017 AN INDEPENDENT EVALUATION guide to appendixes A–K Appendix A. Environmental Sustainability in the World Bank Group 78 Appendix B. Methodology for Environmental Sustainability Analysis 82 Appendix C. Additional Findings on Environmental Sustainability 106 Appendix D. Background on World Bank Project Performance Ratings 139 Appendix E. Additional Information on the World Bank Portfolio 145 Appendix F. Analysis of Distribution of World Bank Financing Volume Across Projects and Countries 163 Appendix G. Risk and Independent Evaluation Group Outcome Ratings 168 Appendix H. Factors of Performance of World Bank Projects: Structured Review of Research 172 Appendix I. Gender Incorporation in Project Development Objectives, Components, and Indicators 186 Appendix J. Additional Information on the International Finance Corporation Portfolio 191 Appendix K. Additional Findings: Management Action Record Tracking Data and Methodology 207 Boxes Box B.1. Evaluative Questions in Assessing IFC and MIGA Implementation of the Recommendations in IEG’s 2010 Safeguards Evaluation 96 Box C.1. The World Bank Group’s Changing Approach to Environment in Selected Country Strategies 115 Box C.2. Examples Where Indicators May Not Capture Full Environmental Results 118 Figures Figure C.1. IFC Project Support For Environmental Sustainability Increased in the Middle East and North Africa and Decreased in South Asia 108 Figure C.2. Upper-Middle-Income-Countries Have the Highest Share and Larger Increase in Environmental Activities, Low- Income Countries Saw Increase over Time from a Low Base 108 Figure C.3. The Role of Development Policy Lending in Resilient and Green Decreased from FY08–10 to FY15–17 110 Figure C.4. The share of Sustainable Development Advisory Services and Analytics with Potential Environmental Benefits Is Similar to that for Projects 112 Figure C.5. Share of Advisory Services with Potential Environmental Benefits 114 Figure C.6. Achievement Rates for Environmental Outcomes or Objectives in World Bank Group Country Strategies in Aggregate Are the Same as for All Outcomes or Objectives 115 Figure C.7. Country Strategy Indicators Covered All Major Environment Subsectors 117 Figure C.8. No Projects Received Low Outcome Ratings as a Result of Overambitious Objectives 121 Figure C.9. IFC’s Satisfactory or Better Ratings, at Appraisal and Evaluation, in Three Environmental-Related Performance Standards, CY12-CY16 126 Figure C.10. IFC’s Satisfactory or Better Ratings in Three Environmental-Related Performance Standards, at Appraisal and at Evaluation, by Environmental Risk Category, CY12–16 128 Figure C.11. IFC’s Satisfactory or Better Ratings in Three Environmental-Related Performance Standards, at Appraisal and at Evaluation, by Type of Project, CY12-CY16 128 Figure C.12. IFC’s Satisfactory or Better Ratings in Three Environmental-Related Performance Standards, at Appraisal and at Evaluation, by Project Size, CY12-CY16 129 Figure D.1. Ratings Structure for IEG’s ICR Reviews Before and After ICR Reform 140 Figure D.2. Coverage of Self-Evaluation Among World Bank Financing Projects Approved FY06–08 142 Figure D.3. The Independent Evaluation Group’s ICR Review Coverage, as of October 2, 2017 143 Independent Evaluation Group | World Bank Group 75 Figure E.1. World Bank Projects: Outcome Ratings over Time, Year by Year 151 Figure E.2. World Bank Projects: Performance Ratings over Time, Year by Year 152 Figure E.3. World Bank Project Monitoring and Evaluation Ratings over Time, Year by Year 153 Figure E.4 World Bank Project Monitoring and Evaluation Ratings over Time, Rolling Three-Year Average 154 Figure E.6. Country Strategy World Bank Group Performance Ratings Vary over Time 155 Figure E.7. Country Strategy Development Outcome Ratings Vary across Regions 156 Figure E.8. Country Strategy World Bank Group Performance Ratings Vary across Regions 157 Figure E.9. World Bank Project Outcome Ratings over Time, Year by Year versus Three-Year Rolling 158 Figure E.10. World Bank Project World Bank Performance Ratings over Time, Year by Year versus 3-Year Rolling 159 Figure E.11. World Bank Outcome and World Bank Performance Ratings, by Region 160 Figure E.12. World Bank Project Outcome Ratings, by Global Practice 161 Figure E.13. World Bank Project Performance Ratings, by Global Practice 162 Figure F.1. A Few Very Large Projects Account for a Large Share of Volume 163 Figure F.2. Net commitment of World Bank Lending Projects in a Country Shows Correlation with Number of People in Poverty in the Country 165 Figure F.3. DPFs Show Some Association between World Bank Performance Rating And Total Net Commitment per Person in Poverty in the Country 166 Figure H.1. Country-Level Factors Associated with World Bank Project Performance, Based on Structured Literature Review 176 Figure H.2. Political Economy Factors Associated with World Bank Project Performance, Based on Structured Literature Review 179 Figure H.3. World Bank–Level Factors Associated with World Bank Project Performance, Based on Structured Literature Review 180 Figure I.1. Projects that Incorporated Gender in PDOs or Components, by Global Practice Cluster and Region, FY14–16187 Figure I.2. Projects Incorporating Gender in PDOs or Components, within the Human Development and Sustainable Development Global Practices, FY14–16 188 Figure I.3. Presence of Gender-Relevant Indicators among Projects Incorporating Gender in the PDO or at Least One Component, FY14–16 189 Figure J.1. Long-Term Trend of IFC Investment Projects’ Results and Performance Ratings, by XPSR Program Years (percent) 192 Figure J.2. Satisfactory or Better Ratings of the Four Development Outcome Indicators Shows that Only the E&S Effects Ratings Improved in CY14–16 (percent) 193 Figure J.3. Long-Term Trend of Mostly Successful or Better Development Outcome ratings of IFC Investment Projects in IDA-Eligible and Non-IDA Countries and in FCV and Non-FCV Countries (percent) 194 Figure J.4. Long-Term Trend of Mostly Successful or Better Development Outcome Ratings of IFC Investment Projects (percent).  195 Figure J.5. Comparison of Mostly Successful ot Better Development Outcome Ratings of IFC Investment Projects, by Region (percent) 196 Figure J.6. Comparison of Satisfactory or Better Ratings in IFC Work Quality and IFC’s Role and Contribution Indicators (percent) 199 Figure J.7. Long-Term Trend of IFC Advisory Projects with Positive Ratings for Development Effectiveness and IFC’s Role 76 Results and Performance of the World Bank Group 2017 | Guide to Appendixes A – K and Contribution (percent) 201 Figure J.8. Long-Term Ratings Trend of Development Effectiveness and Its Indicators (percent) 202 Figure J.9. Long-Term Trend of IFC Advisory Projects’ Mostly Successful or Better Development Effectiveness ratings, by IFC Region (percent) 203 Figure J.10. Long-Term Trend of Mostly Successful or Better Development Effectiveness Ratings, by Projects in IDA and Non-IDA Countries (percent) 204 Figure J.11. Long-Term Trend of Mostly Successful or Better Development Effectiveness Ratings, by Projects in FCV and Non-FCV Countries (percent) 205 Figure K.1. Ratings of Recommendations Rated Both by IEG and Management in the Third and Fourth Year of Implementation  208 Figure K.2. Recommendations Rated High and Complete, by Category (percent) 209 Tables Table B.1. Environmental Activities by Subsector 82 Table B.2. Sample versus Population for World Bank Projects, by Practice Group and Region, Approved FY08–10 86 Table B.3. Sample versus Population for World Bank Projects, by Practice Group and Region, Approved FY15–17 86 Table B.4. Sample versus Population for IFC Projects, by Industry Group and Region, Approved FY08–10 88 Table B.5. IFC Sampling Frame, by Environmental and Social Risk Category 100 Table B.6. MIGA Sampling Frame, by Environmental and Social Risk Category 100 Table C.1. Environmental Activities Are More Prevalent in the World Bank’s Largest Clients 109 Table C.2. Trends from Comparing by Project Are Similar to Those from Comparing by Component 110 Table C.3. The Share of Components with Multiple Potential Environmental Benefi s Has Increased 111 Table C.4. Carbon Finance Activities Have Changed in Composition Over Time 113 Table C.5. Environment-Related Performance Standards and Ratings of Evaluated IFC Investment Projects, CY12–16 125 Table C.6. Impact of IFC Support on Environment-Related Performance Standards 127 Table C.7. Environment-Related Performance Standards and Ratings of Evaluated MIGA Projects, CY12–16 130 Table C.8. Share of Category A Projects in IFC and MIGA Portfolio 131 Table E.1. Very Large Projects Closed in FY14–16 (Net Commitment $50 0 Million or More) 145 Table E.2. Very Large Projects Closed in FY11–13 147 Table F.1. Eight Countries Were Consistently on List of Highest Country Exposures in Credit Risk Section of IBRD Financial Statements for the Past Five Years 164 Table G.1. Pairwise Correlation Coef ficients, Investment Project Financing Projects Closed FY06–12 168 Table G.2. Risk Rating, Problem Project, Supervision Cost, and Outcome Ratings 169 Table G.3. Linear Regression Models of Project Outcome Ratings, Investment Project Financing Projects Closed FY06–12 169 Table G.4. Pairwise Correlation Coef ficients, Investment Financing Projects Approved FY09–10 171 Table H.1. Selection of Sources for Structured Literature Review 173 Table H.2. Categories of Project Outcome Factors 174 Table J.1. Regression Analysis of External and Internal Risk Factors Affecting Development Outcome Ratings in IFC Investment Projects. 198 Table K.1. IEG Recommendations on Knowledge, Measurement, and Impact Assessment Have Seen Lower Uptake 210 Independent Evaluation Group | World Bank Group 77 Appendix A. Environmental Sustainability in the World Bank Group The World Bank Group has several strategies and frameworks outlining its goals for promoting environmental sustainability. Two key strategies lay out approaches for helping client countries address their environmental challenges and ensure that World Bank Group projects integrate principles of environmental sustainability (World Bank 2001; World Bank Group 2012b). These strategies supported an approach of mainstreaming environment as a cross-cutting theme across the World Bank Group, especially for the Sustainable Development Practice Group. Aspects of environmental sustainability have been critical in other strategy frameworks and action plans, such as the Strategic Framework on Development and Climate Change, the Climate Change Action Plan, and the Forest Action Plan (World Bank 2009b; World Bank Group 2016a, 2016b). Sectoral strategy documents also integrate environmental objectives, such as in the 2013 energy directions paper (World Bank 2013a), which emphasizes the need for improvements in renewable energy and energy ef ficiency to minimize the environmental costs of expanding energy supply and reduce the environmental impact of the energy sector. The World Bank Group institutions have also incorporated environmental and social sustainability into their respective corporate strategies. The Strategy and Business Outlook FY18–20 of the International Finance Corporation (IFC) provides a recent example. A background paper developed for Results and Performance of the World Bank Group 2017 (RAP17) lays out the history of environmental sustainability in the World Bank Group since the 1970s. It describes the changing treatment of environmental sustainability in strategies, high-level analytical work, and the evolving institutional and structural context. The World Bank works to promote environmental sustainability directly through its country strategies, project operations, and Advisory Services and Analytics. It also works through global partnerships and knowledge products. Specifically, the World Bank often includes environmental pillars and objectives in its country strategies, and it has financed a substantial portfolio of work that seeks directly to support environmental objectives, including in areas such as climate change mitigation and adaptation, environmental health and pollution management, natural resource management (including forests, fisheries, biodiversity, landscapes, and watersheds), environmental policies and institutions, and water resource management—but also as a cross-cutting issue in many other sectors. It supports these through investment project financing, development policy financing, and Program- for-Results financing, as well as through technical assistance, policy dialogue, and knowledge and analytical work, including work on natural capital accounting and valuation of ecosystem services. The World Bank also has specific policies that govern risk management of the potential for adverse environmental effects, under an approach of “do no harm.” Investment lending operations have been covered by a set of operational policies (OPs) including those on environmental assessment (OP 4.01), natural habitats (OP 4.04), forests (OP 4.36), and pest management (OP 4.09). These detail the World Bank’s responsibilities for preparation and appraisal, identification and screening, and consultation and disclosure. Development policy financing considers environmental and social effects under OP 8.60, which outlines the responsibility of the World Bank for identifying and assessing potential adverse effects from supported policies, assessing the client government’s capacity for managing any such effects, and supporting the government in filling any gaps in this capacity. Program-for-results financing sets out requirements for assessing a program’s environmental systems 78 (OP 9.00), including considerations to promote environmental sustainability by avoiding, minimizing, or mitigating adverse impacts and promoting informed decision making. For investment lending, under the historic system the World Bank is required to undertake the environmental and social assessment, and activities that pose a risk of potentially significant and irreversible adverse impacts on the environment or affected people (category A) are explicitly excluded from support by the instrument. A new environmental and social framework for investment lending was approved by the Board in 2016 after a long set of consultations, and preparation work for implementation is in progress. For IFC, environmental sustainability is an important component for achieving positive development outcomes. IFC aims to support environmental sustainability through a number of avenues. For IFC investments (direct or through financial intermediaries, asset management companies or other subsidiaries) and advisory services, IFC requires that its clients have their own environmental and social management system, and that investments that have moderate to high levels of environmental risk or potential for adverse environmental impacts follow the Performance Standards and World Bank Group Environmental, Health, and Safety Guidelines, and it provides support to help clients to do so as part of its due diligence, monitoring, and supervision efforts. Throughout its project portfolio, IFC aims to support resource effi iency and environmental sustainability, such as in supply chains and fi ancial intermediary on-lending. It supports particular types of investments and advisory services that may have climate change benefi s, such as renewable energy, energy ef ficiency, and cleaner production. It has established and supported environmental standards and partnerships such as the Equator Principles, which is a risk management framework adopted by financial institutions for determining, assessing, and managing environmental and social risks in projects. More recently, it initiated the Sustainable Banking Network, a partnership for financial market regulators and central banks to encourage wider adaptation of environmental and social risk management by financial institutions. IFC’s approach to environmental sustainability is laid out in IFC’s Sustainability Framework. This framework includes a policy on environmental and social sustainability, which identifies IFC’s mission both to do no harm to the environment and to enhance the sustainability of private sector operations and the markets they work in (IFC 2012b). It calls specific attention to the serious global challenge of climate change, and IFC’s explicit support for low-carbon development through investments and advisory services, and for minimizing negative impacts on essential ecosystem services. At the core of the Sustainability Framework are the IFC Performance Standards, which define client and IFC’s responsibilities for managing their environmental and social risks. The Performance Standards present three areas for environmental risk management: (i) PS1: Assessment and Management of Environmental and Social Risks and Impacts; (ii)PS3: Resource Ef ficiency and Pollution Prevention; and (iii) PS6: Biodiversity Conservation and Sustainable Management of Living Natural Resources. The standards provide guidance on how to identify risks and impacts and are designed to help avoid, mitigate, and manage risks and impacts as a way of doing business sustainably. For the Multilateral Investment Guarantee Agency (MIGA), environmental sustainability also forms a key component for achieving positive development outcomes of its guarantee projects. It follows a similar approach to IFC, based on its Policy on Environmental and Social Sustainability and Independent Evaluation Group | World Bank Group 79 similarly adopts a do no harm approach, seeks to enhance sustainability of the private sector, and to support climate change challenges and climate finance (MIGA 2013). MIGA’s policy also adopted the Performance Standards and Environmental, Health, and Safety Guidelines to help its clients manage environmental risks but calibrated to MIGA’s business model. Through its due diligence, specific provisions in the contract of guarantee, and monitoring of projects, MIGA aims to ensure that the investments it supports are implemented in accordance with the Performance Standards and Environmental, Health, and Safety Guidelines. References IFC (International Finance Corporation). 2012b. International Finance Corporation’s Policy on Environmental and Social Sustainability. Washington, DC: World Bank Group. MIGA (Multilateral Investment Guarantee Agency). 2013. “Policy on Environmental and Social Sustainabili- ty.” Washington, DC: World Bank Group. World Bank. 2001. Making Sustainable Commitments: An Environment Strategy for the World Bank. Wash- ington, DC: World Bank. 80 Results and Performance of the World Bank Group 2017 | Appendix A Appendix B. Methodology for Environmental Sustainability Analysis As part of the Results and Performance of the World Bank Group 2017 (RAP17), the Independent Evaluation Group (IEG) conducted a range of analytical tasks on environmental sustainability. The methodology for this work is described below. Classification Framework for Environmental Sustainability To answer evaluation questions about environmental sustainability in a rigorous and consistent fashion requires a clear definition of what is meant by activities that support environmental sustainability. IEG thus developed a classification framework for identifying and categorizing activities that have the potential for positive environmental effects. The framework was based on the definition of environment in the World Bank Group’s 2012 environment strategy, and was augmented by technical consultations with 23 World Bank, International Finance Corporation (IFC), and Multilateral Investment Guarantee Agency (MIGA) staff across seven World Bank Global Practices, five IFC industry or other groups, and MIGA. A goal of the framework was to identify activities likely to have environmental effects, regardless of whether direct or indirect or whether the environmental goals were an intentional motivation of the activity. A second goal of the framework was to identify effects based on their environmental outcomes rather than on the specific approaches used to achieve them, and so knowledge work, physical investments, or policy incentives would all be categorized together if they were supporting the same outcome (and were covered by an instrument included in the IEG sample). Yet, even at the highest level of detail, many activity headings are still broad categories that capture a wide range of activities; expert reviewers also used more detailed guidance within each activity type. A single component could be assigned to multiple activity types of subsector—for example, many interventions might have both climate change mitigation and air pollution cobenefits. The framework is shown in table B.1. Table B.1. Environmental Activities by Subsector Sub- Activities with Potential Area sector Environmental Benefits Examples of Activities in Portfolio Energy efficiency: supply sid Efficiency imp ovements in power generation, power transmission, or district heating Energy efficiency: demand Lines of credit for equipment for SMEs. Climate change mitigation side Efficiency imp ovements in public buildings Extractive industries Improving environmental sustainability of oil sector environmental improvement Reduced gas flaring and fugitive emission Renewable energy Financing for hydro, wind, solar, off-grid renewables Strengthening RE regulatory and policy frameworks Feed-in tariffs or power purchase agreements for RE Clean Reduced emissions from Coal to gas conversion, Improved fuel in cookstoves fossil fuel combustion Coal mine methane capture 81 Table B.1, continued. Sub- Activities with Potential Area sector Environmental Benefits Examples of Activities in Portfolio CC mitigation: Transport Public transport expansion, sustainable transport policies, Climate change mitigation congestion-reducing measures CC mitigation: Agriculture Environmental management of livestock waste Green growth strategy in agriculture sector CC mitigation: Forestry Reforestation investments, fi e prevention CC mitigation: Urban Sustainable urban planning and land management General CC mitigation policy Electricity tariff reform, carbon tax design reform Transport air pollution Gasoline to electric or CNG vehicles, freight emissions abatement regulation Industrial air pollution Strengthening air quality regulator, financing industrial abatement equipment upgrades Other outdoor air pollution abatement Air pollution abatement Indoor air pollution abatement Abatement of ozone Improved industrial production equipment depleting substances Climate change mitigation Many activities from transport and energy activities with air pollution cobenefit Water quality management, Water quality monitoring, water resource management water pollution abatement capacity building Water and soil pollution Agriculture: soil and water Assistance to farmers on agriculture and water pollution management management, livestock waste management Marine pollution abatement Wastewater treatment and Construction or rehabilitation of wastewater plan, sewerage abatement wastewater collection expansion Water and soil pollution Slaughterhouse and market upgrades, irrigation system abatement sediment diagnostics Solid waste management New landfill investment, ehabilitation management Hazardous waste Remediation of contaminated plots management/cleanup Waste Environmental management of extractive industries General pollution institution/ capacity building General/Other Green/smart buildings Raise health and education building to Green building standards Clean technology Support to companies producing Clean products Clean Clean Other Clean 82 Results and Performance of the World Bank Group 2017 | Appendix B Table B.1, continued. Sub- Activities with Potential Area sector Environmental Benefits Examples of Activities in Portfolio Protected areas National park support and institutional strengthening management/corridors conservation Biodiversity Wildlife protection and Support for law enforcement conservation Environmental sustainability Financing tourism companies, tourism information of tourism Environmental awareness raising/education Other biodiversity Ecosystem management Sustainable land Reducing land degradation scapes management Land- Watershed management Knowledge and capacity for water management Sustainable fisherie Sustainable sourcing, fisheries gove nance, monitoring Fisheries/ oceans Coastal zone and marine Reduce coastal erosion, manage coastal biodiversity ecosystem management Sustainable water Improve water resource regulation, reduce industrial water management, water resource consumption management Irrigation efficiency Rehab of irrigation systems, improve practices improvements Loss reduction in water Reduce leakage from water supply systems supply systems Water Groundwater management Aquifer monitoring, hydrogeological studies Integrated pest management/ Monitoring chemical residues in produce Sustainable pesticide management agriculture Sustainable agriculture Environmental improvements in agribusiness, support for proenvironment agricultural practices Sustainable forest Agroforestry, reforestation, community forestry management; Forest governance/ Improved monitoring of forest cover monitoring Plantation forestry Sustainable plantation forestry investments Forests institutional Improve collection of forest data Forestry building/capacity building/ enforcement Sustainable supply chains Third-party sustainable sourcing certificatio General/Other General environmental Improve environmental standards in legislation, improve capacity building/institutional environmental impact studies support Green Green Other Green Independent Evaluation Group | World Bank Group 83 Table B.1, continued. Sub- Activities with Potential Area sector Environmental Benefits Examples of Activities in Portfolio Disaster-related/resistant Retrofit public buildings, imp ove road/bridge resilience, infrastructure drainage construction Institutional reform/capacity Emergency preparedness/hydromet/warning system Climate change adaptation/Disaster risk building Proresilience policy reform Promote sustainable cities model in urban planning Financial disaster risk Catastrophe insurance, weather index insurance management resilience from ecosystem Restore ecological functions of habitat services (Climate) Resilient Health-related climate management change adaptation Disaster-related safety nets Social safety net scale-up in drought-prone areas Climate smart agriculture Drought-resistance crops research and promotion (resilience) Note: The Resilient area was not divided into subsectors because of the dif ficulty of meaningfully separating disaster risk management and climate change adaptation. CC = climate change; CNG = compressed natural gas; FI = financial intermediary; SME = small and medium enterprises; RE = renewable energy. This framework was then used as a typology for assessing a large random sample of projects from the World Bank, IFC, and MIGA between fiscal years (FY)08–10 and FY15–17. These time periods were chosen for the following reasons: ƒƒ IEG’s evaluation on environmental sustainability had a portfolio review that covered up until FY07 (World Bank 2008). ƒƒ The 2008 period was the baseline for the Strategic Framework on Development and Cli- mate Change, which included the first major set of corporate targets on climate change. ƒƒ The goal of the assessment was to cover what the World Bank Group has been doing recently, and so it was desirable to cover the most recent project approvals (up until FY17). ƒƒ A decision was made to take a sample from three-year periods rather than take a single year for comparison, to reduce the risk of having a baseline or endline that was an un- usual year for the World Bank Group (especially during the financial crisis of 2008). World Bank Group Project Portfolio: Has Environmental Composition Changed? IEG reviewed one-third of all World Bank projects in each of the two periods under consideration. This included specific investment loans, development policy loans, emergency recovery loans, 84 Results and Performance of the World Bank Group 2017 | Appendix B adaptable program loans, financial intermediary loans, and sector investment and maintenance loans in FY08–10 and investment project financing, development policy loans, specific investment loans, adaptable program loans, and Program-for-Results in FY15–17. The definition of projects was the same as that used in the regular (nonspecial topic) parts of the RAP, which excludes recipient- executed projects under $5 million, or instruments not subject to IEG validation. Most significantly, this excluded stand-alone carbon finance projects. Projects were selected using nonstratified random sampling: the IEG sample covered 282 out of the total FY08–10 portfolio of 848 and 276 out of the total FY15–17 portfolio of 827 projects. The resulting sample was representative of the population. versus Population for World Bank Projects, by Practice Table B.2. Sample Group and Region, Approved FY08–10 (percent) Share of Projects Share of Commitments Group or Region Population Sample Population Sample Practice group Economics and Finance 20 24 30 33 Human Development 22 22 18 23 Sustainable Development 58 55 52 44 Total 100 100 100 100 Region Sub-Saharan Africa 31 33 20 15 East Asia and Pacifi 15 13 15 11 Europe and Central Asia 13 12 15 16 Latin American and the 20 21 26 29 Caribbean Middle East and North Africa 8 11 7 13 South Asia 11 10 17 17 Total 100 100 100 100 Note: Total population 846 projects, $103,780 million commitments; Sample size 282 projects, $34,184 million commitments. versus Population for World Bank Projects, by Practice Table B.3. Sample Group and Region, Approved FY15–17 (percent) Share of Projects Share of Commitments Group or Region Population Sample Population Sample Practice group Economics and Finance 24 23 26 21 Human Development 20 20 19 18 Sustainable Development 57 57 55 61 Independent Evaluation Group | World Bank Group 85 Table B.3, continued. Share of Projects Share of Commitments Group or Region Population Sample Population Sample Total 100 100 100 100 Region Sub-Saharan Africa 34 39 24 28 East Asia and Pacifi 19 17 17 19 Europe and Central Asia 14 12 14 15 Latin American and the Caribbean 14 13 16 16 Middle East and North Africa 6 4 10 6 South Asia 14 16 18 16 Total 100 100 100 100 Note: Total population 827 projects, $105,586 million commitments; Sample size 276 projects, $32,114 million commitments. IEG manually reviewed project appraisal documents or equivalent appraisal documents for all projects in the sample. Projects were then coded at the component level, as this provided a higher level of detail than the project but also had distinct preexisting boundaries. For development policy lending, pillars were considered equivalent to components. After exclusions there were 876 project components in FY08–10 and 808 components in FY15–17. Each component was assessed by an expert specialist to identify if it included activities that directly or indirectly provided potential environmental benefits and, if so, coded by what type of environmental benefits using the framework. A second expert specialist also reviewed all borderline cases and a selection of other cases. The resulting coding generated a database suitable for quantitative and qualitative analysis. Components that purely supported project management were excluded, and a small number of components that were contingent placeholders in case of future natural disasters were also excluded. Quantitative analysis focused on two main variables: the share of components that included at least some environmental activity, and the percent share of World Bank financing commitments for those components. For development policy financing (which does not allocate financing below the level of the operation) it was assumed that financing was split equally across all pillars in the operation. Results were then analyzed by (i) two levels of environmental classification reflected in the framework: Clean-Green-Resilient breakdown and the subsector level (see framework above); (ii) World Bank Region; (iii) practice group; (iv) country income group. For IFC investments, the approach was similar to that of the World Bank. For IFC, a nonstratified random sample of projects was selected for investment projects approved by the Board in FY08– 10 and FY15–17. It was challenging for IEG to locate Board documents for all projects as many documents were not filed correctly, consequently those projects where documentation could not be located were dropped from the sample. IEG reviewed 151 of 814 projects in FY08–10 and 151 of 712 projects in FY15–17. This sample was broadly representative of the population. 86 Results and Performance of the World Bank Group 2017 | Appendix B versus Population for IFC Projects, by Industry Group and Table B.4. Sample Region, Approved FY08–10 2008–10 Share of Projects 2015–17 Share of Projects (percent) (percent) Group or Region Population Sample Population Sample Industry Group CTT 11 13 13 14 FM 37 33 43 38 Infrastructure 21 20 16 16 MAS 31 34 29 32 Total 100 100 100 100 Region East Asia and Pacifi 11 13 16 19 Europe and Central Asia 28 23 19 16 Latin America and the Caribbean 22 24 24 21 Middle East and North Africa 10 9 9 9 South Asia 12 11 12 12 Sub-Saharan Africa 15 17 16 19 World 2 3 3 4 Total 100 100 100 100 Note: Population size 814 for FY08–10, 699 for FY15–17; sample size 151 in both periods. CTT = Telecom, Media, Technology, and Venture Investing Department; FM = Financial Markets; MAS = Manufacturing, Agribusiness, and Services. The review was carried out at the project level, as there is no consistently defined subunit below that of the project. An expert specialist reviewed the IFC Board document and coded it against the same framework used for the World Bank, with a second expert specialist reviewing borderline cases and a selection of other cases. Results were analyzed by two levels of environmental classification from the framework, and by industry group, and by Region. Sample sizes were not large enough for analysis of other groupings to be meaningful. For MIGA, the approach was like IFC’s. Projects where Board documents could not be located were dropped. Identification of guarantee projects followed MIGA’s reporting in their audited Annual Reports and each project has a unique of ficial approval document, either a Board document or a Small Investment Program underwriting checklist for Small Investment Program projects. The review considered all 26 projects for which Board documents or underwriting checklist could be located for FY08–10 (out of 36 projects total), and a random sample of 36 projects of 75 projects in FY15–17. The population size did not make quantitative analysis meaningful at a subgroup level.1 A number of caveats and cautions should be made about the results of this exercise: ƒƒ The exercise considers only positive environmental effects, it does not consider negative environmental effects, or trying to weigh or balance positive effects against negative Independent Evaluation Group | World Bank Group 87 effects (for example, climate change mitigation benefits from hydropower vs negative effects on in-river biodiversity). ƒƒ The exercise looks only at the potential effects of interventions based on their design, not at whether interventions were implemented successfully or actually produced environmental improvements. It does not capture for example that some early irrigation ef ficiency interventions like lining canals may have ended up having more neutral overall environmental effects because of reduction of recharge of downstream groundwater than intended. ƒƒ The shares reported are not comparable across institutions, especially between the World Bank versus IFC/MIGA. In particular, the shares for the World Bank are calculated by asking whether the component includes any environmental activity, while for IFC and MIGA the shares are calculated asking only whether the project included any environ- mental activity. All else equal, this would tend to lead to a higher percentage value for IFC and MIGA than the World Bank. ƒƒ The World Bank and IFC exercise looks only at commitments at the time of Board ap- proval.2 It does not look at whether projects became effective or actually disbursed. ƒƒ The analysis does not capture the magnitude of environmental benefits. Projects with a small environmental effect from a component are treated the same as those with a large effect. ƒƒ The exercise considers only core projects that are typically covered by IEG review. It excludes small recipient-executed activities and stand-alone carbon finance including those under the Clean Development Mechanism and reducing emissions from defor- estation and forest degradation (REDD)/REDD+. This might exclude some Clean projects in FY08–10 and some Green projects in FY15–17. ƒƒ The exercise sought to capture both direct indirect environmental effects, but in practice was likely to miss some portion of indirect effects if they were not captured in project documentation. Indirect affects likely to be captured well are those that were deliberate goals of the project, such as from fossil fuel subsidy removal. Indirect effects less likely to be captured well could be some ecosystem service benefits, indirect resilience benefits, effects from market price changes that were not a direct goal of the activity, or effects that are substantially downstream. ƒƒ The intention of the exercise is to capture the potential environmental effects of interven- tions rather than on whether or how environmental effects were identified in World Bank Group project documentation. Nonetheless, increasing high-level attention on environ- ment and climate change may have changed how authors write about environmental effects in documentation, so documentation from more recent periods may be more likely to highlight environmental benefits even if there was no change in the underlying investment. This could affect the likelihood of expert teams to identify environmental effects, and so the results may overstate the degree of change. 88 Results and Performance of the World Bank Group 2017 | Appendix B ƒƒ The analysis does not effectively capture potential changes in the quality of support for environmental benefits or improvements in “how” that proenvironment activities are done. For example, interviews with World Bank staff indicate that disaster reconstruction efforts may be done more comprehensively than in past, yet both recent and historic cases would still be identified as promoting resilience by “building back better.” This effect may understate the degree of change in terms of impact. ƒƒ For IFC and MIGA, the exercise captures only environmental benefits from financed activities, it excludes environmental benefits that come from application of Performance Standards, including through environmental action plans. This could mean, for example, that efforts to support resilience by supporting ecosystem services restoration through environmental action plans would not be captured. Analytic and advisory activities were not covered in the exercise above, but IEG carried out simple exercises to track World Bank Advisory Services and Analytics (ASA) and IFC Advisory Services. World Bank ASA were covered by a simple exercise that identified potential environmental benefits based on a review of the objective and description of the operation. IEG reviewed a sample of ASA activities approved in FY08–10 and FY15–17 and mapped to a Global Practice in the Sustainable Development Group.3 Activities where World Bank databases did not contain information on objective or description were dropped, as were product lines where the share of operations with missing data for objectives and description exceeded 10 percent: external training, impact evaluation, and donor & aid coordination; together these comprised roughly 8 percent of the population. A random sample of 240 activities was taken from the remaining activities in each period (so 480 in total): nonlending technical assistance, economic and sector work, ASA, and research services. A senior expert then coded these as to whether they were had potential Clean, Green, or Resilient environmental benefits following the classification framework described above. A simple three-stage approach was adopted to identify IFC Advisory Services supporting environmental sustainability over 2008–17. Stage 1: All Advisory Services marked as “Portfolio” and “Completed” in the Project Stage and “Advisory Services Projects with Client(s)” in the Project Type in IFC’s Advisory Services database were considered. Stage 2: Based on the new IFC Advisory Services business lines, all projects designated as “Environment, Social and Governance” were included. In addition, projects in three subbusiness lines under the cross-cutting advisory solutions were included. These are “clean energy and resource ef ficiency,” “green buildings” and “water.” Stage 3: The team then proceeded with identifying environment projects by IFC Tertiary Sector that were not captured in the above business lines. This way, projects in renewable energy, energy ef ficiency, waste management, wastewater, and environment and social sustainability were captured. To avoid double-counting, projects that fell in both sets were only counted once. Of 1,511 Advisory Services projects over FY08–17, 277 were identified as supporting environmental sustainability by this criterion. Advisory services were then counted and assessed for any trend: the individual advisory services were not examined. Independent Evaluation Group | World Bank Group 89 Country Strategy Portfolio RAP17 identified and reviewed the environment outcomes or objectives and indicators in the results frameworks of 208 country assistance strategy (CAS) / country partnership strategy (CPS) and Country Partnership Frameworks (CPFs) during FY08–17. Each outcome or objective was compared with the classification framework described above, and coded as containing Clean, Green, and/or Resilient goals. The objectives or outcomes were analyzed to observe trends by environment area and subsector: over time, across Regions, and across country income grouping. The World Bank list of economies as of June 2017 was used to categorize countries into high-income, upper-middle- income, lower-middle-income, and low-income countries. The analysis did not take into consideration changes to a country’s classification during FY08–17. The objectives or outcomes at the start of the strategy period (at entry) were compared with those at completion (at exit) for the first five years, FY08–12, for CAS/CPSs where CAS Completion Report Reviews / Completion and Learning Reviews were available, to compare planned versus actual environmental priorities. For those outcomes or objectives where IEG provided ratings of achievement, the RAP reviewed the achievement of outcomes or objectives and reasons why certain objectives or outcomes were not achieved. World Bank Group Country Strategy Indicators RAP17 reviewed the environment objectives and indicators (as identified in the preceding exercise) used to measure and assess achievement of World Bank Group objectives in all 54 CPFs produced over FY15–17 (the CPF approach was initiated in FY15). The review was limited to CPFs to keep the number of objectives and indicators manageable and to align with the project-level indicator review which looked at recently approved projects. As noted above, many of the objectives were broad, but indicators provided more specific information on what the World Bank Group expected to achieve during the CPF. Indicators were assessed to determine whether they demonstrated the achievement of the objective based on the following: ƒƒ Relevance to the objective: The broad framing of the many of the objectives made this assessment dif ficult. The review examined to what extent there was a “line of sight” (or an implicit results chain) from the indicator to the objective, whether the indicator was a process, output or outcome, and whether the indicator was suf ficient to reflect objective achievement. ƒƒ Measurability of the indicator: The indicators were reviewed to see if baseline and targets were provided. It was not possible to examine whether there were systems to provide the necessary data since this was not provided in the majority of CPFs. 90 Results and Performance of the World Bank Group 2017 | Appendix B The instruments mapped to the objectives were reviewed, as needed. The results frameworks listed the various World Bank Group interventions contributing to the objective. While they were broken down in ongoing and/planned and/or investments and knowledge products, only the titles were provided making it dif ficult to a systematic analysis of the instruments. World Bank Project Indicators IEG assessed 60 recently approved or recently evaluated projects with environmental objectives to identify what information is and is not captured by project indicators and to assess the quality of those indicators. The initial population was defined as all projects approved during FY15–17, and all projects that closed in FY16 and FY17, for which an environmental theme was recorded as the primary or secondary code (that is, the largest or second largest). Environmental themes include: disaster risk management, climate change, environmental health and pollution management, renewable natural resources asset management, environmental policies and institutions, water resource management. A manual screening was then carried out on the project development objectives of each project and only those which reflected an actual environmental objective were retained. This yielded 144 approved projects and 84 closed projects. From this list a random sample was drawn of 40 approved projects and 20 closed projects where indicators were reviewed in detail. Projects were mapped against the three pillars of the environmental strategy (Clean, Green, and Resilient) based on their objective. For each project, IEG then identified the following: ƒƒ the type of information covered by the indicators; ƒƒ the extent to which indicators capture inputs, outputs, intermediate outcomes, or more fi al outcomes; ƒƒ whether indicators were suf ficient to demonstrate achievement of the objective; ƒƒ whether indicators were suf ficient to demonstrate attribution of outcomes to the opera- tion; ƒƒ for closed projects, whether planned indicators were actually implemented; ƒƒ any other issues of indicator quality. Independent Evaluation Group | World Bank Group 91 World Bank Environmental Project Outcome Ratings The analysis took the universe of evaluated projects closed in FY15–16 that had been tagged as having a primary or secondary environment theme, there were 41 projects with primary environment theme and 25 projects with secondary environment theme (66 in total). Then the analysis conducted a secondary screening based on whether the projects had an environment-related objective. The analysis then compared high-performing projects (those with outcome ratings of satisfactory or highly satisfactory) to low-performing projects (those with outcome ratings of moderately unsatisfactory or below), excluding the midrange projects with moderately satisfactory ratings that could be borderline cases and could muddy any analysis. This yielded a population of 50 projects—of which 21 were from the Environment and Natural Resources Global Practice and the remaining 29 were from other Sustainable Development Global Practices. IEG identified a set of specific hypotheses relating to outcome ratings for environmental projects based on its own experience in evaluating and validating these projects, and on consultations with the Environment and Natural Resources Global Practice. IEG then coded the 50 projects based on characteristics relating to these hypotheses drawn from Implementation Completion and Results Report (ICR) Reviews. Analysis proceeded in two steps: when an association between project characteristics and outcome ratings seemed to emerge based on simple correlational analysis, IEG applied qualitative comparative analysis to identify patterns that differentiated low-performing projects from high-performing projects. IEG identified a set of specific hypotheses relating to outcome ratings for environmental projects based on its own experience in evaluating and validating these projects, and on consultations with the Environment and Natural Resources Global Practice. These hypotheses were based on common prior beliefs, sometimes based on historic data from older projects or experiences. The following paragraphs explain some of the theory behind these specific hypotheses. One hypothesis is that projects fail either because of lack of implementation (which is inarguably a failure), because of a lack of evidence (potentially driven by weaknesses in measurement) or because of aspects of the evaluation process (for example, potentially driven by weaknesses in objective setting combined with the objectives-based evaluation methodology). IEG experience indicated that evidence issues were especially likely to occur for environment projects because of dif ficulty in measuring outcomes (see chapter 1), and that this was common cause for differences in ratings between Global Practices and IEG. The analysis then sought to separate these factors by coding whether or not a project completed its core outputs, and whether or not projects provided suf ficient evidence on outcomes. A second hypothesis is that projects receive low outcome ratings because ratings are judged against overambitious objectives. Examples of this include objectives not being achievable by an intervention of the size of the project or representing longer-term goals that could not be observed at the time of evaluation. IEG experience suggested this was a particular problem for environmental projects because their objectives may take longer to achieve, because projects were often small, and because some had used aspirational language in objectives, especially for Global Environment Facility projects, which had 92 Results and Performance of the World Bank Group 2017 | Appendix B used global environmental objectives to quality for funding. Projects were coded based on whether ICR Reviews mentioned that objectives were overambitious and on the reason for this. A third hypothesis was that projects struggled to deliver because of weaknesses in implementing agency capacity. This was thought to be a challenge for environmental agencies, which often had less experience in managing large projects or running procurement or financial management systems than for agencies in other sectors, were smaller, and had fewer experienced project staff. Similarly, environmental projects may rely on environmental ministries, which often have little influence on decision making in cabinet as compared with more powerful ministries such as energy, transport, water, or industry. (Note that separate IEG work has found that environmental policy lending has sometimes been successful in creating a platform for multisectoral cooperation and action on environmental goals, IEG 2016b.) Projects were coded based on whether ICR Reviews noted weak implementation capacity, and on whether implementing agencies were environmental or not. A fourth hypothesis was that projects struggled to achieve good ratings on efficiency because of a lack of economic analysis. This was a problem for environmental projects, because of the difficulty of estimating the benefits or value of ecosystem services or other environmental benefits. Consequently, IEG observed that for environmental projects, efficiency was driven more by issues around implementation delays and operational and administrative inefficiencies. Projects were coded based on the presence or absence of an economic analysis and the presence of absence of an effort to value environmental benefits. Feedback from World Bank management suggested hypotheses around project size (previous IEG analysis had suggested that smaller projects receive lower ratings than others, and environment projects could be especially small), multisectorality (multisector projects might be complex and could face higher coordination challenges and so do worse), and household-level behavior change (which might be more dif ficult to affect and measure and could be particularly prevalent in natural resource management projects). Projects were coded based on World Bank commitments with different thresholds, on whether a project was implemented across multiple sectors, and on whether project intervention logic included household-level behavior change. A final hypothesis related to the presence of GEF-financed projects. These projects were often innovative and so potentially risky, were usually small in size, supported behavior change had a history of overly ambitious objectives, and sought to influence environmental outcomes that were difficult to measure. Analysis of IFC and MIGA Performance Standards In addition to the environmental sustainability analysis of IFC and MIGA projects described in the above section on World Bank Group Project Portfolio, IEG conducted two reviews as part of the special topic on Environmental Sustainability of the World Bank Group. The first review responded to the evaluation question “To what extent have environmental outcomes improved between approval and evaluation?”4 in the concept note for the RAP 2017. The desk review involved the synthesis Independent Evaluation Group | World Bank Group 93 of IEG’s environmental and social (E&S) evaluations for the Expanded Project Supervision Report and project evaluation report programs completed since the Policy on Social and Environmental Sustainability (PSES) were adapted in 2011 for IFC and 2012 for MIGA. IEG also conducted a separate review to answer evaluation question in the “Are key issues on Performance Standards noted by the IEG safeguards evaluation being implemented?”5 Concept Note of the RAP 2017. To answer the question, IEG conducted a structured review with separate evaluative questions for IFC and MIGA. Desk reviews of IFC and MIGA projects committed/issued during in FY12 to FY16 plus interviews of key IFC and MIGA staff were conducted to answer the main question in the concept note. Details of the methodology used in the two reviews are described in the next paragraphs. Review of IFC and MIGA Environmental-Related Performance Standards. IEG synthesized the evaluation results of three environmental-related Performance Standards (PSs) of IFC and MIGA that were triggered during the appraisal or due diligence process of IFC’s investment and MIGA’s guarantee projects. The desk review covered all real sector projects evaluated from FY12 to FY16 that triggered PS1, Assessment and Management of Environmental and Social Risks and Impacts; PS3, Resource Ef ficiency and Pollution Prevention; and PS6, Biodiversity Conservation and Sustainable Management of Living Natural Resources.6 For this review, evaluated financial sector projects were excluded because the PS1, PS3 and PS6 performance of the subprojects supported by IFC’s and MIGA’s financial intermediary projects during FY06-FY11 were not monitored. IEG’s methodology for evaluating the PS1, PS3 and PS6 performance of IFC and MIGA projects applied specific indicators and subindicators to each Performance Standard and are assessed both at entry (that is, “at appraisal” for IFC and “at due diligence” for MIGA) and at evaluation when the project reaches operational maturity.7 The extent of improvement (or lack thereof) between appraisal and evaluation in the project’s PS1, PS3 and PS6 indicators provided the basis for assessing effects.8 In assessing PS1, PS3 and PS6 effects in IFC investment projects, IEG reviewed 212 real sector projects evaluated and rated as part of the calendar year (CY)12 to CY16 Expanded Project Supervision Report program. All the projects covered in IEG’s review were approved before CY2012, and therefore followed the 2006 PSES and Environmental and Social Review Procedure as the applicable sustainability policy for this review. By IFC industry groups, 56 percent (118 projects) were in the manufacturing, agribusiness, and services; 34 percent (72 projects) in infrastructure and natural resources; and 10 percent (22 projects) in telecom, media, technology, and venture investing. By environmental categories, 5 percent (11 projects) were classified as category A, 89 percent (188 projects) as category B, and 6 percent (12 projects) as category C. The assessment of PS1, P3 and PS6 impacts in MIGA projects covered 34 real sector projects evaluated as part of the FY12 to FY16 project evaluation report program.9 Project composition by sector teams consisted of 17.5 percent (6 projects) in energy and extractive industries; 23.5 percent (8 projects) in infrastructure, and 59 percent (20 projects) in manufacturing, agribusiness, and services. By environmental categories, 9 percent (3 projects) were classified as category A; 82 percent (28 projects) in category B; and 9 percent (3 projects) in category C. 94 Results and Performance of the World Bank Group 2017 | Appendix B Review of the Implementation of IFC and MIGA PSs Updates in terms of IEG 2010 safeguards evaluation recommendations. After analyzing IEG’s recommendations, IEG developed separate evaluative questions for IFC and MIGA to assess whether the recommendations have been integrated and implemented into IFC 2012 and MIGA 2013 Sustainability Frameworks.10 The review covered both real and financial sector projects; each with a separate set of relevant questions. These evaluative questions (see box B.1) were used for the structured interviews with eight E&S specialists from IFC, MIGA and the Compliance Adviser / Ombudsman, and two senior evaluators from IEG.11 Questions in Assessing IFC and MIGA Implementation of Box B.1. Evaluative the Recommendations in IEG’s 2010 Safeguards Evaluation Evaluative Questions Relating to IFC Real Sector Projects 1. Has IFC established inclusive criteria for category A (for example, including guidance on risk assessment and typical categorizations for industry sectors? Is the current guidance adequate and suf ficient to unambiguously categorize A and B category projects? 2. Have IFC projects been correctly categorized? 3. Has the identification of greenhouse gas (GHG) emissions been effective? In high GHG emission projects, have alternatives been considered and feasible and cost- effective options to reduce GHG emissions been considered? Have emissions been correctly quantified, monitored and reported in projects with more than 25,000 t/a GHG emissions? 4. Have the clients and their Environmental and Social Management Systems (ESMS) systems identified primary supply chain risks and taken remedial actions (for example, change to a certified supplier) related to PS2 (Child and forced labor, life-threatening health and safety situations) and PS6 (biodiversity)? 5. Have IFC inspections during site visits at appraisal addressed supply chains? 6. Has IFC suf ficient tools to address supply chains and define areas of influence in all sectors with a focus on the highest risks, such as child labor, forced labor, and clearing of critical habitats? 7. Has IFC disclosed suf ficiently project supervision information, including the Environmental and Social Action Plan (ESAP) documents and their updates according to Access to Information Policy 2012? 8. What incentives IFC has developed for investment of ficers to share ownership of the Performance Standards and mainstream their implementation? 9. Has IFC used Advisory Services suf ficiently and adequately to enhance positive environmental and social (E&S) impacts in its investment projects and as stand-alone Advisory Services projects? Independent Evaluation Group | World Bank Group 95 Box B.1, continued. 10. To what extent and how systematically IFC uses clean production and energy audits at appraisal using auditors with relevant sector knowledge to improve energy ef ficiency and clean production? 11. Has IFC used loan covenants, including Conditions of Disbursement to enforce compliance with E&S requirements and reporting, if the client has lacked commitment and is continuously out of compliance? Has noncompliance of policy provisions triggered the option for IFC to sell its shares? How and to what extent has IFC encouraged and used third-party or community 12. monitoring and have the reports been disclosed? 13. Do you have any other comments on IFC Sustainability Framework or suggestions to enhance its content or implementation? Evaluative Questions Relating to IFC Financial Intermediary Projects 1. Has IFC established inclusive criteria for categorization of financial intermediary (FI) projects? 2. Has IFC appraisal, including categorization of FI projects and their Environmental & Social requirements followed IFC’s 2012 Sustainability Framework and have IFC’s FI projects been correctly categorized? 3. Have the relevant FI projects complied with the Exclusion List requirements? 4. Have the clients and their ESMS systems identified primary supply chain risks and taken remedial actions (for example, change to a certified supplier) related to Performance Standard (PS) 2 (child and forced labor, life-threatening health and safety situations) and PS6 (biodiversity)? Have IFC inspections during appraisal site visits addressed supply chains? 5. Does IFC have suf ficient tools to address supply chain issues in FI subprojects and define areas of influence in all sectors with a focus on the highest risks, such as child labor, forced labor, and clearing of critical habitats? 6. Has IFC’s supervision of FI projects including site visits and Annual Environmental Performance Report reviews followed IFC’s 2012 Sustainability Framework requirements, including tracking compliance of relevant Performance Standards? 7. What incentives have IFC developed for its investment of ficers to share ownership of IFC’s Sustainability Framework and Performance Standards and mainstream their implementation? 96 Results and Performance of the World Bank Group 2017 | Appendix B Box B.1, continued. 8. Has IFC used Advisory Services suf ficiently and adequately to enhance positive E&S impacts in its FI projects and as stand-alone Advisory Services projects? 9. Have high-risk subprojects been monitored by third-party especially if the financial intermediary supported by IFC lacks its own monitoring capacity? 10. Has IFC used loan covenants, including Conditions of Disbursement to enforce compliance with E&S requirements and reporting, if the client has lacked commitment and is continuously out of compliance? Has noncompliance of policy provisions triggered the option for IFC to sell its shares? 11. Do you have any other comments on IFC Sustainability Framework or suggestions to enhance its content or implementation? Evaluative Questions Relating to MIGA Projects 1. Has MIGA established inclusive criteria for category A (for example, including guidance on risk assessment and typical categorizations for industry sectors), and refined the categorization system to address the bunching of higher and lower risk projects within category B? Is the present guidance including MIGA E&S Policy, Performance Standards and E&S Review Procedure adequate and suf ficient to unambiguously categorize A and B category projects? 2. Have MIGA projects been correctly categorized? 3. Is MIGA’s E&S standards’ implementation framework (E&S staff, resources and procedures) adequate to manage E&S risks in MIGA’s portfolio? How have MIGA’s E&S staff and resources developed from 2010 to 2017? 4. Has the brief project summary disclosed from MIGA Small Investment Program projects been suf ficient to inform stakeholders and public on project-related E&S risks and their mitigation? 5. Are MIGA’s due diligence reviews adequately addressing Social and Environmental Management System of developing-country subsidiaries the project supports, rather than the corporate policies of the parent banks? 6. How long has Japan Trust Fund been operating and what has been the size and eligibility of the Trust Fund for addressing E&S challenges to all low-capacity clients on the basis of need? Has the Trust Fund enhanced client capacity? 7. Has disclosure of project-level environmental and social information from supervision reports been strengthened in the 2013 Sustainability Framework and adequate in terms of meeting project stakeholders’ and affected communities’ needs for information on E&S impacts and their mitigation? Independent Evaluation Group | World Bank Group 97 Box B.1, continued. 8. Are MIGA’s mechanisms to ensure Performance Standards adherence in implementing FI projects credible in terms of (i) adequate project appraisal (categorization and E&S requirements, including Social and Environmental Management System and relevant Performance Standards), (ii) monitoring (quality of Annual Monitoring Reports and their reviews, site visits), (iii) engaging and communicating with stakeholders, and (iv) taking corrective actions if needed. 9. Do you have any other comments on MIGA Sustainability Framework or suggestions to enhance its content or implementation? Evaluative Questions for the Compliance Adviser / Ombudsman 1. Has IFC and MIGA established inclusive criteria for category A (for example, including guidance on risk assessment and typical categorizations for industry sectors)? Is the present guidance adequate and suf ficient to unambiguously categorize A and B category projects? 2. Have IFC and MIGA projects been correctly categorized? 3. Have the clients and their ESMS systems identified primary supply chain risks (also beyond agrocommodities) and taken remedial actions (for example, change to a certified supplier) related to PS2 (Child and forced labor, life-threatening H&S situations) and PS6 (biodiversity)? 4. Have IFC inspections during site visits at appraisal addressed supply chains? 5. Does IFC have suf ficient tools to address supply chains and define areas of influence in all sectors with a focus on the highest risks, such as child labor, forced labor, and clearing of critical habitats? 6. Has IFC disclosed suf ficiently project supervision information, including the ESAP documents and their updates according to Access to Information Policy 2012? 7. What incentives have IFC developed for its investment of ficers to share ownership of the Performance Standards and mainstream their implementation? 8. Has IFC used Advisory Services suf ficiently and adequately to enhance positive E&S impacts in its investment projects and as stand-alone Advisory Services projects? 9. To what extent and how systematically IFC uses clean production and energy audits at appraisal using auditors with relevant sector knowledge to improve energy ef ficiency and clean production? 10. Has IFC used loan covenants, including conditions of disbursement to enforce compliance with E&S requirements and reporting, if the client has lacked commitment and is continuously out of compliance? Has noncompliance of policy provisions triggered the option for IFC to sell its shares? 98 Results and Performance of the World Bank Group 2017 | Appendix B Box B.1, continued. 11. How and in what extent has IFC encouraged and used third-party or community monitoring and have the reports ben disclosed? 12. Is the present reporting procedure on grievance and meditation results of Compliance Adviser / Ombudsman to the President, and communication to the Board effective and independent? 13. Do you have any other comments on IFC and MIGA Sustainability Frameworks or suggestions to enhance its content or implementation? Note: E&S = environmental and social; FI = financial intermediary; IEG = Independent Evaluation Group; IFC = International Finance Corporation; GHG = greenhouse gas; MIGA = Multilateral Investment Guarantee Agency. IEG also applied the above questions in the desk review analysis of ten purposefully sampled IFC projects and six MIGA projects. For IFC, a purposeful sample was developed from all 1,115 investment projects committed in FY12-FY16, weighted more toward category A projects. Category C, category FI-2, and FI-3 projects were omitted from the population to focus only on high and medium risk projects, resulting in a population of 605 projects. Breakdown by E&S Category are presented in table B.6. Table B.5. IFC Sampling Frame, by Environmental and Social Risk Category  Sample to Sample to Sample Population Population Sample Population Population Category (no.) (no. project) (percent) ($, millions) ($, millions) (percent) Category A 5 73 6.8 755.0 47,041 1.6 Category B 2 476 0.4 155.0 39,484 0.4 Category FI-1 3 56 5.4 570.0 6,570 8.7 Total 10 605 1.7 1,480 93,095 1.6 Note: IFC = International Finance Corporation. Similarly, a purposeful sample was created from 80 MIGA projects that were issued from FY12 to FY16, weighted more toward category A projects. Category C, category FI-2, and FI-3 projects were omitted from the universe of projects to focus only on high and medium risk projects, resulting in a population of 61 projects. Breakdown by E&S Category are presented in table B.2 below. Table B.6. MIGA Sampling Frame, by Environmental and Social Risk Category  Percent of Sample Population Sample to Category (no.) (no.) Population Category A 4 15 27 Category B 1 42 2 Category FI-1 1 4 25 Total 6 61 10 Source: Independent Evaluation Group. Note: MIGA = Multilateral Investment Guarantee Agency. Independent Evaluation Group | World Bank Group 99 Analysis of World Bank Safeguards Reporting Quality IEG assessed safeguards-related reporting in completion reports for all 314 World Bank investment lending projects validated by IEG over July 2015-June 2017. The goal was to validate implementation of the requirements in Harmonized Evaluation Guidelines for completion reports. This analysis updates and expands work conducted for RAP16, adding in a second year of data and presenting results on the pooled data set. The analysis documented the following questions: ƒƒ Was the environmental assessment category correctly recorded in both the appraisal document and completion report? ƒƒ Did the completion report accurately describe and list all safeguards that had been trig- gered, based on the appraisal document and any restructuring? ƒƒ Did the completion report clearly identify whether or not safeguards had been complied with? In addition, the analysis looked more deeply into the 41 projects assigned category A, looking also at Implementation Supervision Reports and checking for consistency with completion reports. The analysis of projects covered by RAP16 was adjusted slightly for the combined data set. The interpretation of a statement of compliance was expanded to include reference to Safeguards Compliance Ratings from Implementation Status and Results Reports of moderately satisfactory or higher, as this could be interpreted as compliance. This affected 16 projects. Analysis of IEG Evaluation Recommendations IEG conducted a deep dive on recommendations for IEG evaluations relating to environmental sustainability. IEG reviewed all evaluations disclosed between FY07–16, that is, those in the Management Action Record database for which at least one Management Action Record update had been completed, and identified those relating to environmental sustainability. This produced 59 recommendations, including copies of recommendations across World Bank Group institutions. A typology of recommendations was developed, and recommendation implementation was compared across this typology based on the most recent rating as of November 2017. Each recommendation was reviewed to identify the progress made, the adequacy of information provided by management updates, and the consistency of action plans with recommendations. Limitations: What is Not Covered in the Review of World Bank Group 100 Results and Performance of the World Bank Group 2017 | Appendix B Environmental Sustainability? The RAP report produced by IEG provides a synthesis of existing evaluative materials, supplemented by those from new but limited desk-based reviews. Its purpose is to provide an annual review of recent IEG evaluations and what these reveal about the World Bank Group’s effectiveness in addressing development challenges A special topic is selected for each year but its assessment is limited in scope compared with IEG’s thematic, sector, corporate or other major evaluations. For RAP17, environmental sustainability was selected as the special topic but is not intended to have the scope of a major evaluation in terms of questions, coverage, or evaluative methods. It is limited also by resource costs and feasibility, and has to focus on areas where evidence is available and where additional desk work will be most useful. These limit both the scope of evaluative coverage and the types of conclusions that can be drawn. Two data limitations are particularly important: first, IEG does not directly evaluate or rate World Bank ASA work, so these were not covered in the main detailed portfolio review. Second, the World Bank’s project evaluation methodology (used also by IEG) is objectives-based; this means that project results frameworks, indicators, and ratings generate information focused on the main project development objective—and so are unable to capture effects on environmental sustainability for projects that did not have major environmental goals. Together, these limitations mean that RAP17 assesses results on environmental sustainability only across a partial range of World Bank Group interventions. The following aspects are not covered or only partly covered in this report: ƒƒ World Bank Group partnerships on environment including global programs and con- vening roles are not covered directly, nor were effects from convening roles, standard setting, or other forms of influence. ƒƒ Stand-alone carbon finance projects are not covered; IEG has an evaluation on carbon finance being carried out in parallel which will be delivered at the end of FY18. ƒƒ The World Bank’s ASA projects are not covered in the main portfolio classification exer- cise but are covered through a simple stock-taking. ƒƒ Program-for-Results lending is relatively new and with little evaluative evidence so far, and so will is included only in the main portfolio classification exercise. ƒƒ IFC’s Green Bonds are not covered. ƒƒ The review of project indicators and explanatory factors of project ratings will cover World Bank environment projects only. For RAP17, IEG is piloting the application of quali- tative comparative analysis method to analyze key explanatory factors. Through this pilot application, IEG will explore the potential to apply the qualitative comparative analysis method to IFC and MIGA projects in the future. ƒƒ Environmental outcomes of projects are typically measured and evaluated only for those Independent Evaluation Group | World Bank Group 101 projects with environmental objectives (with the exception of environmental footprint ef- fects, which are measured for IFC and MIGA projects). Hence, much of the RAP focuses on the type of activities that are being reported and on measurement systems, rather than trying to give identify results of the World Bank Group on environmental sustainabil- ity. ƒƒ The report does not assess the adequacy of the new World Bank Environmental and Social Framework or IFC and MIGA PSs, a topic that is more suitable as an IEG thematic evaluation. ƒƒ Activities financed by IFC’s Asset Management Company are not covered. ƒƒ The corporate footprint of the World Bank Group (in terms of energy use, resource use, recycling, and so on). However, some of these topics can be covered indirectly from synthesizing findings from existing IEG evaluations. Possibilities for Future Evaluations During preparation of this concept note, commenters suggested a number of evaluative questions and topics that are beyond the scope of the Results and Performance of the World Bank Group 2017 but would be important questions that could be suitable for future evaluations carried out either by the IEG or by World Bank Group operations. These included the following: ƒƒ What has been the performance of projects with environmental objectives? ƒƒ How do World Bank Group nonlending activities, such as technical assistance, policy dialogue, knowledge and convening role come together to support the lending activities to promote environmental sustainability? ƒƒ How effectively has the World Bank mainstreamed environment in its country and sector policies? What institutional factors have supported or hindered mainstreaming? ƒƒ What have been the longer term and indirect impacts of the World Bank Group on envi- ronmental sustainability? ƒƒ Which projects are more effective in promoting environmental sustainability-stand-alone/ core environment projects, or projects where environmental concerns are main- streamed? ƒƒ Do trade-offs exist between promoting environmental sustainability vs promoting eco- 102 Results and Performance of the World Bank Group 2017 | Appendix B nomic growth and poverty reduction? Are there synergies between these objectives? How well is the World Bank Group positioned to address these? ƒƒ How significant has World Bank Group engagement been with often-weak environmen- tal agencies as compared with other sectoral ministries? ƒƒ Have interventions been sequenced appropriately-for example, for use of analytical work, technical assistance, policy dialogue, investment lending, and policy lending? ƒƒ How effective has World Bank Group analytical and knowledge work been on environ- mental sustainability? ƒƒ How effective has the World Bank Group been in building resilience in client countries? ƒƒ What are explanatory factors of success or failure for World Bank Group environmental performance at the country program level? ƒƒ How effective are monitoring frameworks (structures, systems and processes) at the project level for tracking environmental requirements? ƒƒ What is the track record of World Bank performance in the resolution of safeguards-re- lated concerns and grievances? How effective are the grievance mechanism and accountabilities? ƒƒ What is the verification process of monitoring information of safeguards? How effective have stakeholders, third parties, and nongovernmental organizations been in the verifica- tion process? ƒƒ Given the linkages between climate change and fragility/conflict, are there any differen- tiated lessons concerning the World Bank’s work on environmental sustainability in fragil- ity, conflict, and violence and non-fragility, conflict, and violence contexts? ƒƒ Are the IFC and MIGA PSs adequate for managing E&S risks? How ef ficient and effective are the standards? Are they realistic for all client country situations? ƒƒ Has the World Bank Group focused on the most important environmental issues? ƒƒ Are there trade-offs between E&S goals in natural resource management? How has the World Bank Group managed these trade-offs? ƒƒ Are the World Bank Group’s environmental goals consistent with the Sustainable Devel- opment Goals? ƒƒ Are World Bank interventions environmentally sustainable? ƒƒ How is the World Bank Group managing environmental risks to operational implementa- tion? Independent Evaluation Group | World Bank Group 103 Reference World Bank. 2008. Environmental Sustainability: An Evaluation of World Bank Group Support. Independent Evaluation Group. Washington, DC: World Bank. 1 Endnote 51 in chapter 2 describes in detail the Multilateral Investement Guarantee’s Small Investment Program. 2 Of the Multilateral Investement Guarantee projects reviewed for this exercise, only those with contracts of guarantee (projects insured or projects issued) were included. 3 The share of Advisory Services and Analytics (ASA) outside of the Sustainable Development Practice Group with environmental benefits was likely to be low and not easily identifiable from the project objective and description. Resource constraints made it more ef ficient to focus attention on Sustainable Development ASA. 4 The question was presented in the Results and Performance of the World Bank Group 2017 concept note as part of evaluation subquestion 7b but was modifi d to focus on the outcomes of environmental-related standards. Source: CODE2017-0027. IEG Review of Results and Performance of the World Bank Group 2017 with a Special Focus on Environmental Sustainability. Page 9. 5 The question was presented in the Results and Performance of the World Bank Group 2017 concept note as part of evaluation subquestion 7a. Source: CODE2017-0027. IEG Review of Results and Performance of the World Bank Group 2017 with a Special Focus on Environmental Sustainability. Page 9. 6 Performance Standards were published in International Finance Corporation’s Performance Standards on Social & Environmental Sustainability, 2006; and Multilateral Investment Guarantee Agency’s Performance Standards on Social and Environmental Sustainability, 2007. 7 International Finance Corporation (IFC) and Multilateral Investment Guarantee Agency (MIGA) projects are evaluated when they reached operational maturity—five years after Board approval for IFC and three years after effectiveness for MIGA projects. 8 The Independent Evaluation Group’s environmental and social evaluation methodology for both International Finance Corporation and Multilateral Investment Guarantee Agency projects rates every indicator and sub-indicator for each Performance Standard on a four-point scale (that is, excellent, satisfactory, partly unsatisfactory and unsatisfactory) “at- appraisal” and “at-evaluation.” Ratings of “not applicable” (NA) as appropriate and “no opinion possible” (NOP) may be assigned to individual indicators during evaluation where, after best efforts, the relevant information cannot be obtained. For this review, the “success rate” for each indicator was calculated as a percentage of Excellent and Satisfactory ratings, with NA and NOP ratings excluded from both the numerator and denominator (unless otherwise stated). 9 Multilateral Investment Guarantee Agency evaluated projects include thosereaching operational maturity (on average, three years after effectiveness date) or canceled before reaching operational maturity. 10 The updates comprise of (i) Policy on Environmental and Social Sustainability; (ii) International Finance Corporationg Performance Standards and their guidance notes; (iii) Access to Information Policy; (iv) Environmental and Social Review Procedure; and (v) Environmental and Health and Safety Guidelines. The questions were sent to the interviewees in advance. On average, the structured interviews lasted 1–2 hours and 11 was conducted during the period October 10 to 16, 2017. 104 Results and Performance of the World Bank Group 2017 | Appendix B Appendix C. Additional Findings on Environmental Sustainability Environmental Sustainability in World Bank Group Projects Chapter 1 presented high-level results from a detailed portfolio review of a one-third random sample of World Bank projects approved in fiscal year (FY)08–10 and FY15–17, and appendix B presented the methodology for generating this. This section provides additional detail from this rich data set. Environmental Activities by Practice Group In the World Bank, environmental activities—not surprisingly—are mostly found in the Sustainable Development Practice Group, but other Global Practices’ contribution has increased over time. Two other practice groups—Human Development and Economics and Finance—have become more active, more so in the Clean area. In Human Development, such activities are found in community development, social protection, education, and health projects, with components related to such infrastructure improvements as wastewater treatment, solid waste management, green buildings, off-grid electrification, micro irrigation, and disaster relief. In health, examples are a mine safety project aiming at tackling occupational lung diseases (with the components aimed at reducing industrial air pollution and environmental management of extractive industries), a project involving citizen engagement in building environmental and social management capacity for infrastructure resiliency, and another project establishing a productive and disaster responsive safety net. In Economics and Finance, examples of environmental activities in investment policy financing projects include a Green logistics framework that involves environmentally sustainable design of warehousing, container yards, and truck parks; technical assistance for a water resources management strategic plan; and several disaster and climate-related risk insurance activities. Development policy operations often include environmental actions and pillars such as removing constraints to expansion of renewable energy, minimizing environmental and social costs of irrigation schemes, electricity tariff adjustments, development of environmentally sustainable tourism, and fiscal policy reforms aimed to facilitate a transition toward a green growth strategy. Environmental Activities by Region Regions differ significantly in their involvement in environmental activities of the World Bank: East Asia and Pacific has the highest share of project components that provide environmental benefits, and that share almost doubled from FY08–10 to FY15–17; and Sub-Saharan Africa has the largest absolute number of environmental components (because it is the region with the most projects) but the smallest share of environmental components (figure 1.3 in the main text). In contrast, environmental activity of Latin America and the Caribbean, Europe and Central Asia, and Middle East and North Africa decreased over time, in parallel with the overall decline in the number of World Bank projects in these Regions. In East Asia and Pacific, from FY08–10 to FY15–16, the environmental effort doubled, both in absolute terms and as percentage of the overall East Asia and Pacific portfolio. The increase happened in part because increases in disaster risk management activities (increased resilience to cyclones, floods, 105 and tsunamis; disaster insurance, disaster relief and reconstruction),, installation of power mini grids and solar photovoltaic electrification on islands, and sustainable fisheries management activities. In East Asia and Pacific, environmental effort is largely led by China which is responsible for more than one-half of the regional benefits. Environmental activities in China are mainly in the Clean area in both time periods. However, the structure of the China environmental portfolio changed over time, with a significant increase in climate change mitigation activities in the second period and a decrease of Resilient activities (only three such activities are captured in the second period). In FY08–10, typical World Bank’s environmental activities in China were in energy ef ficiency, coal to gas conversion, water resource management, wastewater treatment, flood control, electrified rail (including urban), sustainable transport and sustainable cities, and environmental management of farms. In FY15–17, the World Bank’s environmental involvement in China was typically in: reduced coal consumption from increased renewable energy and energy ef ficiency, reduced air pollution from pollution abatement measures, green cities, urban rail, green growth strategy, wastewater collection and treatment, and preventing environmental pollution. In Sub-Saharan Africa, the structure of the portfolio also changed from FY08–10 to FY15–17 from the prevalence of Green activities to an increase of Clean ones. Resilient activities increased, from a low base. This largely reflects the trend in the overall World Bank’s environmental portfolio. On a more detailed level, the environmental portfolio evolved from generally focusing on activities in biodiversity, community livelihood, watershed, agricultural sustainability, urban congestion, agriculture value chains, wind electrification and forestry to an emphasis on climate smart agriculture, off-grid, in particular, solar photovoltaic projects, addressing drought, involving carbon tax and CO2 storage, and including activities reducing emissions from deforestation and forest degradation (REDD+). For the International Finance Corporation (IFC), most regions saw little change between periods, but the share of IFC projects with potential environmental benefits in the Independent Evaluation Group (IEG) sample increased in Latin America and the Caribbean and decreased South Asia. The difference across regions was driven in part by relative shares across industry groups: regions with more infrastructure or manufacturing, agribusiness, and industry had larger shares of projects with potential environmental benefits (figure C.1). World Bank Environmental Activities by Country Income Group The World Bank’s environmental activity differs by country income group: overall, the highest share of the benefits is in the upper-middle-income countries; this group also saw a larger increase in that share from FY08–10 to FY15–17. However, the picture at the subsector level is more complicated, below is a description of the findings for the subsectors with the highest number of activities. The largest group of activities by far is climate change mitigation, and in this subsector, FY15–17 shares of environmental benefits in project components were equal across country income groups, a change from FY08–10 when the upper-middle-income group had a larger share than the other two groups. Air pollution abatement activities are still more prevalent in the upper-middle-income countries but 106 Results and Performance of the World Bank Group 2017 | Appendix C the differences among country income groups decreased over time as well. Water and soil pollution abatement activities are still more prevalent in upper middle and lower-middle-income groups as compared with low-income one. Three subsectors where low-income group’s share of environmental benefits in project components is above or equal to other groups are climate change mitigation, water management, and resilience or adaptation (figure C.2). Project Support For Environmental Sustainability Increased in the Figure C.1. IFC Middle East and North Africa and Decreased in South Asia Share of IFC projects in IEG sample with potential environmental benefits (pe cent) Note: Sample sizes are 17–42 across regions, so a small number of projects can drive results. AFR = Africa; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; EAP = East Asia and Pacific; IEG = Independent Evaluation GroupIFC = International Finance Corporation; MENA = Middle East and North Africa; SAR = South Asia. Have the Highest Share and Larger Figure C.2. Upper-Middle-Income-Countries Increase in Environmental Activities, Low-Income Countries Saw Increase over Time from a Low Base a. Share of components with potential environmental b. Share of commitments for components with benefits in orld Bank sample, by country income potential environmental benefits in orld Bank sample, group (percent) by country income group (percent) Note: Sample sizes are 17–42 across regions, so a small number of projects can drive results. AFR = Africa; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. Independent Evaluation Group | World Bank Group 107 World Bank Environmental Activities by Country Size Components with environmental activities are more prevalent in countries with larger World Bank portfolios. The share of components with potential environmental effects is larger in the 10 countries that make up the largest share of commitments in the combined population (all approvals FY08–10 and FY15–17) than in the remaining countries. This may be because environment may be a secondary priority for countries, and so get space in the program only in larger portfolios. These 10 countries collectively constitute around a third of project activities (35 percent of all components in the sample), but roughly half of the project activities with likely environmental effects (49 percent). Table C.1. Environmental Activities Are More Prevalent in the World Bank’s Largest Clients 2008–10 2008–10 2015–17 2015–17 2008–10 2015–17 Comps. Comps. Comps. Comps. with Env. with Env. with Env. with Env. Beneficial Beneficial Beneficial Beneficial All Comps. Activities All Comps. Activities Activities Activities  Country Type (no.) (no.) (no.) (no.) (percent) (percent) Largest 10 countries (by 237 101 202 92 43 46 total commitments) All other countries 639 189 606 205 30 34 Note: The 10 countries in the population with the largest total commitments are India, Mexico, China, Brazil, Indonesia, Vietnam, Arab Rep. Egypt, Kazakhstan, Ethiopia, and Bangladesh. Comps. = components; Env. = environmental. World Bank Environmental Activities by Lending Instrument Most World Bank support for environmental sustainability has been channeled through traditional investment financing. Development policy financing (DPF) played a larger role in the years following the global food crisis and financial crisis. However, the overall share of DPF financing for environmental sustainability decreased very slightly by FY, from 24 percent to 22 percent, and a new instrument Program-for-Results was introduced. The decreasing share of DPF was most significant for resilience activities, as crisis response budget support activities were phased out, and as the emphasis in resilience activities shifted from policy and institutional work in FY08–10 to a greater emphasis on physical infrastructure in FY15–17. A larger share of DPF financing in the Green lending can be explained by some large Green growth lending operations in FY08–10 (figure C.3). 108 Results and Performance of the World Bank Group 2017 | Appendix C Role of Development Policy Lending in Resilient and Green Figure C.3. The Decreased from FY08–10 to FY15–17 World Bank commitment by lending instrument (percent) Note: The Program-for-Results instrument did not exist until 2012. APL = adaptable program loan; DPF = development policy financing; ERL = emergency recovery loan; FIL = financial intermediary loan; IPF = investment project financing; P4R = Program-for-Results; SIL = specific investment loan; SIM = sector investment and maintenance loan. World Bank Environmental Activities by Number of Projects The previous analysis for the World Bank focused on comparisons made at the component level. It is also feasible to aggregate results to the project level. By construction, the share of projects with any environmental activities is larger than the share of components with any environmental activities, as not every component necessarily supports environmental benefi s (for example, a community-driven development program might have investments including renewable energy in one component, but other components focused on governance issues). But the overall patterns are similar: over time there is an increasing share of projects with environment, and increasing Clean and Resilient but decreasing Green. from Comparing by Project Are Similar to Those from Table C.2. Trends Comparing by Component With Env. Beneficial With “Clean” With “Green” With “Resilient” All Activities Activities Activities Activities (no.) [n (percent)] [n (percent)] [n (percent)] [n (percent)] Year Proj. Comp. Proj. Comp Proj. Comp. Proj. Comp. Proj. Comp. 2008–10 282 876 133 (47) 290 (33) 86 (30) 163 (19) 56 (20) 111 (13) 24 (9) 45 (5) 2015–17 278 808 144 (52) 297 (37) 90 (32) 179 (22) 47 (17) 81 (10) 42 (15) 82 (10) Independent Evaluation Group | World Bank Group 109 Environmental Sustainability Cobenefits in World Bank Projects Many activities supported by World Bank projects can have multiple types of environmental benefits. The share of components providing multiple benefits has increased over time. In the IEG samples from FY08–10, 10 percent with any environmental benefit had multiple benefits across Clean, Green and Resilient categories; this increased to 15 percent for FY15–17. Project components with Clean and Green benefits included forest management activities with both climate change mitigation benefits and forest benefits, hydropower activities that supported sustainable water management, and rural infrastructure or community development components that included activities such as waste management, small-scale renewable energy, ecosystem management, sustainable tourism, and soil or water conservation. Project components with Clean and Resilient benefi s included water infrastructure combining sewerage and wastewater assets with and flood management and drainage, and institutional capacity building that covered both climate change mitigation and adaptation. Project components with Green and Resilient benefits included wetlands management with both biodiversity and flood control benefits, water resource management activities with flood management, and food security activities that combined sustainable land management with agricultural resilience benefi s. Share of Components with Multiple Potential Environmental Table C.3. The Benefits Has Increased Number of project components with benefit types in Independent Evaluation G oup sample, FY08–10 and FY2015–17 Benefit Type FY08–10 FY15–17 Clean only 143 149 Green only 90 45 Resilient only 27 57 Clean and Green 11 21 Clean and Resilient 9 8 Green and Resilient 9 11 Clean, Green, and Resilient 1 6 Total 290 297 Share with multiple benefits (% 10 15 Source: Independent Evaluation Group portfolio review. Environmental Activities in Other World Bank Group Analytical Activates The main analysis in this report on environmental activities concentrated on traditional World Bank Group financing activities. But the World Bank Group undertakes many other activities that may also 110 Results and Performance of the World Bank Group 2017 | Appendix C contribute to environmental sustainability. This section presents a simple analysis of environmental sustainability in key product lines excluded above. World Bank Analytical And Advisory Work In addition to its lending projects, the World Bank supports its clients through a range of services. Most prominent in these are technical assistance, reimbursable advisory services, and economic and sector work. Many of these products may include potential environmental benefits, especially those relating to Global Practices in the Sustainable Development Practice Group. IEG conducted a simple review of a sample of these activities for (see Methodology in appendix B). The share of Advisory Services and Analystics (ASA) operations in the Sustainable Development Practice Group with potential environmental benefits was stable between FY08–10 and FY15–17, but the absolute number increased by roughly 60 percent due to the expansion of World Bank analytical and advisory activities over this period. The types of potential environmental benefits from ASA are broadly similar to those for projects. The share of Sustainable Devleopment ASA activities with potential environmental benefits is similar to the share of projects with potential environmental benefits. For Clean, Green and Resilient activities the direction of change is the same as that for projects, but smaller in magnitude. The share of activities with multiple types of benefits (such as an activity with both Clean and Green benefits) has increased over time. Figure C.4. The share of Sustainable Development Advisory Services and Analytics with Potential Environmental Benefits Is Similar to that for Projects a. Share of World Bank Sustainable Development b. Share of World Bank Sustainable Development Analytical and Advisory activities with potential Project components with potential environmental environmental benefit benefit Sample Population Years Sample ASA operations (n) Population ASA operations (N) 2008–10 240 1,049 2015–17 240 1,670 Source: Independent Evaluation Group analysis. Independent Evaluation Group | World Bank Group 111 World Bank Group Carbon Finance Carbon finance has been one of the World Bank Group’s first and longest engagements for mitigating climate change. The World Bank Group’s experience with carbon finance will be evaluated in detail in a forthcoming IEG major evaluation to be published in 2018. Carbon finance is a generic term used for the revenue streams that can be generated by sale of project-based greenhouse gas emission reductions or from payments involving carbon sequestration and storage from forests. The World Bank Group supported carbon finance operations under the Clean Development Mechanism and Joint Implementation, both established under the 1997 Kyoto Protocol. But it has also supported a range of other carbon finance activities relating to the REDD+ process, and more recently has developed a range of innovative mechanisms. The World Bank Group’s carbon finance activities can be categorized into two categories (World Bank 2017a): (i) Carbon market activities for the development of the essential architecture for the functioning of carbon markets and the identification and design of projects for buying carbon credits through Emission Reduction Purchase Agreements and ASA activities; and (ii) advisory services and analytics for capacity building and strengthening global and national partnerships for carbon pricing. Due to the rapidly changing global context and regulatory environment, carbon finance in FY15–17 looks very different to that of FY08–10. Table C.4 presents the numbers and type of newly approved carbon finance activities in these periods. Data on financial commitments at the activity level is dif ficult to capture accurately for ASA activities and so is not presented. Emission Reduction Purchase Agreements activities have declined substantially as the Kyoto-era mechanisms have sunsetted and have not yet been replaced, and there has been a change in composition of ASA activities with a decline in new activities from older activities such as the biocarbon fund, but increases from new mechanisms such as the Partnership for Market Readiness. The Forest Carbon Partnership Facility has provided ongoing support through both periods. Table C.4. Carbon Finance Activities Have Changed in Composition Over Time Number of newly approved carbon finance activities th ough World Bank Group–managed carbon funds and facilities Carbon Finance Activity FY08–10 FY15–17 Energy, waste, and others ERPA 74 4 22 25 25 Forest and landscape ERPA 6 3 22 22 Note: ASA = Advisory Services and Analytics; ERPA = Emission Reduction Purchase Agreements. 112 Results and Performance of the World Bank Group 2017 | Appendix C Environmental activities in IFC Advisory Services Considering only investments does not give the full picture of IFC’s activities, as many of the reviewed investments did not offer environmental benefits directly within the project, but IFC also offered associated advisory services for environmental issues. Advisory services were not assessed by the IFC investment portfolio review. However, IEG conducted a simple analysis by counting the number of advisory services in the Environment, Social, and Governance business line, and adding to these cross-cutting advisory solutions in sectors with potential environmental benefits (see appendix B). This shows that there has been significant variation over time in the share of IFC advisory services with potential environmental benefits, but a small upward trend (figure C.4). Roughly one-fifth of these projects were for advice on environmental and social sustainability, while the remainder were for advice on other aspects in sectors that had environmental benefits. Figure C.5. Share of Advisory Services with Potential Environmental Benefits Share of all IFC AS with potential environmental benefits, FY08–17 (percent) Note: Potential environmental benefits determined only from industry coding and product line data.Covers population of 1,511 Advisory Services projects. Dashed line shows trend. AS = Advisory Services; IFC = International Finance Corporation. Environmental activities in World Bank Group Country Strategies Chapter 1 found that there had been an increase in the share of project components with potential environmental benefits, while it found no pattern in the share of country strategy objectives with environmental benefits. What might explain this discrepancy? ƒƒ The country strategy analysis implicitly weights all countries equally, while the project analysis is weighted by the number of projects or by commitments. As noted above, the concentration of environment activities in the World Bank’s largest clients is larger than that of other countries. ƒƒ Country strategy environmental activities can be supported purely by nonlending instru- ments such as knowledge work, carbon finance, or small technical assistance, which is not captured by the project analysis. ƒƒ The magnitude of change in the project portfolio (4 percentage points) is potentially Independent Evaluation Group | World Bank Group 113 too small to detect compared with the variability over time in country strategies, which depends on which countries happen to have a strategy that year. ƒƒ Country strategy objectives pick up only the highest level goals, and not potential envi- ronmental benefits that might come from a particular subcomponent. ƒƒ There is a time lag for country strategies, as strategies reflect the existing active as well as a short-term pipeline. Chapter 2 noted that just over half of environment outcomes or objectives were achieved where ratings are available. Figure C.5 shows how these varied across environmental areas. Figure C.6. Achievement Rates for Environmental Outcomes or Objectives in World Bank Group Country Strategies in Aggregate Are the Same as for All Outcomes or Objectives Percent of outcomes or objectives in World Bank Group country strategies rated achieved or mostly achieved, strategies prepared FY08–12 Source: Independent Evaluation Group analysis of country strategies. Box C.1 gives examples of how environmental priorities can change over time depending on country circumstances and context. Box C.1. The World Bank Group’s Changing Approach to Environment in Selected Country Strategies Ethiopia: Ethiopia is vulnerable to climate-related shocks, especially drought. Building resilience and reducing vulnerability was a key part of the World Bank’s response, with sustainable land management, along with social safety nets and other social programs, included in all three country strategies. In addition to sustainable land management, the World Bank supported disaster risk management in the fiscal year (FY)13 country partnership strategy (CPS) and forestry in the fiscal year (FY)17 Country Partnership Framework (CPF). Indonesia: The environmental program in Indonesia evolved over the last ten years, from a focus on policies and capacity development in climate change mitigation, 114 Results and Performance of the World Bank Group 2017 | Appendix C Box C.1, continued. forestry, and disaster risk management in the FY09 CPS to specific programs in sectors (forestry, marine protection, and disaster risk management) in the FY13 CPS. While the FY16 CPF included objectives in marine protection and forestry, the World Bank Group program expanded to include capacity building in spatial planning and land administration to better harness the potential of natural assets. At the same time, disaster risk management, which was a key part of the World Bank Group’s program following the 2004 tsunami, was subsumed into landscape management and is no longer a separate outcome or objective in the FY16 CPF. Colombia: The World Bank Group’s strategic priorities in Colombia evolved from policy and capacity enhancement at the national level in environment management, climate change and disaster risk management (FY08 and FY11 CPSs), to area focused objectives, tailored to local needs, in the FY16 CPF to address uneven territorial development which the Systematic Country Diagnostic identified as one the country’s main challenges. Bosnia and Herzegovina: Environmental sustainability was a key challenge in Bosnia and Herzegovina. Environment went from being one of 16 outcomes in the FY08 CPS to one of three pillars in the FY12 CPS and one of three focus areas in the FY16 CPF. The World Bank Group’s program expanded from natural resource management the FY08 CPS to include water pollution in the FY12 CPS and then air pollution and flood protection in the FY16 CPF. Some results take time to achieve: the target for area under formal protection set in the FY08 CPS was only met by the end of the (next) FY12 CPS. Serbia: The environment was one of three pillars of the FY08 CPS which included outcomes in industrial and agricultural pollution, energy ef ficiency and disaster risk management. While the FY12 CPS maintained separate outcomes in pollution and energy ef ficiency, the overall country strategy shifted its focus to competitiveness and social spending following the 2008/09 economic crisis. The FY15 CPF ranked environmental sustainability lower among its priorities and did not include a separate objective focused on the environment. Source: Independent Evaluation Group analysis Measuring Environmental Sustainability World Bank Group Country Strategies Environmental indicators in country strategies are present across the full range of environmental goals, but were most common for disaster risk management or climate change adaptation, and for climate change mitigation (figure C.6). Independent Evaluation Group | World Bank Group 115 Figure C.7. Country Strategy Indicators Covered All Major Environment Subsectors Number of indicators by environment subsector Source: Independent Evaluation Group analysis of country strategies. The indicators were framed as process (such as “disaster and climate risk considerations incorporated into public investments”), outputs (often plans, systems, studies), or outcomes (including greenhouse gas emissions reduced, area brought under enhanced biodiversity protection, households at reduced risk of flooding and erosion). About a quarter (26 percent) of the indicators mirrored the core sector indicators. All the indicators in biodiversity and forestry subsectors were core sector indicators or slightly modifi d versions thereof. Core sector indicators for energy ef ficiency and renewable energy were used in climate change mitigation, with most of the remainder using greenhouse gas emissions, one of the new corporate results indicators, as the unit of measurement. The other indicators were either in areas where core sector indicators were not available (for example, climate change adaptation/disaster risk management, fisheries) or were designed to reflect the specifics of a country program (for example, outputs, links to individual projects). Chapter 2 noted how country strategy indicators were often not able to capture a large portion of the World Bank’s potential environmental results in that country. Box C.2 gives examples of this gap. 116 Results and Performance of the World Bank Group 2017 | Appendix C Box C.2. Examples Where Indicators May Not Capture Full Environmental Results The World Bank Group gives “the number of people provided with capacity building support to improve management of natural resources” as the indicator for the objective “Enhancing sustainable resource management in target areas” (Belize, fiscal year [FY]07). This objective is supported by six World Bank interventions including ongoing projects in biodiversity area management/protection and marine conservation/ climate adaptation, and proposed operations in forest protection and natural resource management. While the number of people provided with capacity building support may be a common numerator among the different operations, it may only captures part of what the World Bank is accomplishing in Belize. The objective “Build institutional capacity to manage and adapt to climate change” in the Guatemala Country Partnership Framework (CPF; FY17) measures reductions in forest greenhouse gas emissions and mitigation support to indigenous and vulnerable peoples to assess results. The World Bank Group program includes ongoing and planned operations in disaster risk management. Disaster risk management results are not reflected in the indicators although disaster risk management is listed as one of the priorities under the Intervention Logic write-up in the CPF. In rating the objective “Adaptation to climate change and reduced risk of natural disasters” on completion of Mozambique’s FY12 country strategy, IEG noted that the World Bank Group operations went beyond the single indicator “accurate weather information system available to stakeholders” and stated that it would have been useful to have indicators to measure progress against the two dimensions of the objective. World Bank Environmental Projects Most Clean projects have indicators that provide adequate information to assess the achievement of outcomes and their objectives. Clean projects usually have indicators to measure pollution abatement and estimated greenhouse gas emission reductions directly.1 For Clean projects, 86 percent had indicators that could demonstrate achievement of outcomes and were plausibly attributable to the project activities. Three factors account for this. First, these projects have clear goals and links between project activities to the achievement of their goals. Second, there are established methodologies for measuring their outcomes within the project life or projecting longer- term outcomes. Third, several projects use payments for results approaches that are contingent on the collection and verification of data on results, creating a built-in incentive to collect and analyze information on results.2 Most resilience projects do not measure their impact on vulnerability, but are able to provide useful evidence on results by tying quality standards into their indicators. Many resilience projects frame their objectives in terms of reduced vulnerability or enhanced resilience, which is not measured directly. However, for resilience projects, 82 percent had indicators that could demonstrate achievement of outcomes and that these were plausibly attributable to the project activities. These Independent Evaluation Group | World Bank Group 117 projects typically measure the production of civil works (Resilient infrastructure, flood control structures) as a proxy for reduced disaster vulnerability. However, these indicators have a degree of quality built into them such as work produced “to a particular design standard” which provides more confidence that the desired goals will be met than would occur from merely measuring the amount of infrastructure completed alone. Green projects often have not generated information that was adequate to demonstrate the achievement of their outcomes. Green projects typically measure outputs or areas where practices are to be implemented, but tend not to measure actual environmental results. For example, tree planting may not lead to forest regeneration if tree survival rates are low, so measures of tree planting are not strong proxies for forest regeneration. Only 30 percent of Green projects had indicators that were clearly adequate to demonstrate the achievement of outcomes.3 For 30 percent of projects the indicators were inadequate to demonstrate achievement of the outcomes. In 40 percent of the cases it was not possible to determine their adequacy because the intended results or indicators were not clearly defined. Several factors contribute to this. Compared with Clean and Resilient projects, Green projects involve more complicated theories of change and their goals are of a longer-term nature (such as to conserve biodiversity or reduce deforestation). Many Green projects aim to achieve objectives that require a sustained change in behavior that can be dif ficult to obtain or demonstrate over the span of a single project, and these projects lacked outcome-level indicators. Indicators commonly measure outputs such as the area in which a treatment is carried out (number of hectares in which new agriculture practices were adopted, number of hectares where trees were planted or fenced off to allow natural regeneration, number of hectares included in a protected area, number of hectares for which a management plan exists, and so on) but not intermediate outcomes that might assess whether interventions were effective (such as tree seedling survival or reduction in fires). Biodiversity projects typically incorporate the Management Effectiveness Tracking Tool,4 which is useful for tracking changes in the management capacity of protected areas but lack additional indicators to record the status and health of flora and fauna. It is rare for projects to provide evidence on biodiversity gains, reduction in forest loss or forest degradation, or changes in the level of land degradation.5 Finally, natural resource management projects often promote practices that are intended to be both economically and environmentally beneficial, yet very few projects provide information on economic aspects. Indicators have had dif ficulty in capturing capacity building across all project types. These indicators typically measure coordination or management tools put in place, trainings conducted, or the number of financing mechanisms identified, but do not measure changes in capacity or capture changes in the performance of the supported institutions. Indicators for capacity building improvements were typically less rigorous, using outputs as proxies for capacity enhancement.6 The review of closed projects found that most indicators planned at appraisal were measured ex post, but some were not. Several projects had results frameworks with design flaws that were not 118 Results and Performance of the World Bank Group 2017 | Appendix C revised during implementation, leading to indicators that were not measured. A few Green projects had indicator designs that could not be implemented because of limited country capacity and resources. Their results frameworks were revised to improve indicator measurability. But this resulted in an overcorrection, with an overemphasis on the ease of measurement and a lack of attention to whether the information generated by these indicators was suf ficient to demonstrate achievement of the project objective. Determinants of Outcome Ratings for World Bank Environmental Projects IEG conducted an analysis of recently evaluated World Bank environmental projects to determine which factors are linked to outcome ratings. The analysis took the universe of evaluated projects closed in FY15–16 that had been tagged as having a primary or secondary environment theme, and then conducted a secondary screening based on whether the projects had an environment- related objective. The analysis compared high-performing projects (those with outcome ratings of satisfactory or highly satisfactory) to low-performing projects (those with outcome ratings of moderately unsatisfactory or below), excluding the midrange projects with moderately satisfactory ratings that could be borderline cases and could muddy any analysis. This yielded a population of 50 projects—of which 21 were from the Environment and Natural Resources Global Practice and the remaining 29 were from other Sustainable Development Global Practices. IEG tested a set of hypotheses relating to outcome ratings for environmental projects based on its own experience in evaluating and validating these projects, and on consultations with the Environment and Natural Resources Global Practice. While some of the examined characteristics are somewhat “generic” factors for ratings, experience suggested that these were especially likely to apply to environmental projects (see appendix B). IEG then coded the 50 projects based on characteristics relating to these hypotheses. The analysis proceeded in two steps: when an association between project characteristics and outcome ratings seemed to emerge based on simple correlational analysis, IEG applied qualitative comparative analysis7 to identify patterns that differentiated low-performing projects from high-performing ones. Two factors stand out as suf ficient to generate low outcome ratings: whether or not core project outputs were successfully produced, and whether or not the project provided suf ficient evidence on outcomes. Of projects with low ratings, 60 percent did not produce their core outputs, and 80 percent did not provide suf ficient evidence on outcomes. In contrast, all highly rated projects that produced their core outputs, and only 10 percent of highly rated projects did not provide suf ficient evidence on outcomes. IEG’s monitoring and evaluation (M&E) quality rating is also very closely related: all projects with substantial or high ratings for quality of M&E also received high outcome ratings, and only a single project received a substantial or high M&E quality rating but did not provide suf ficient evidence on outcomes. This suggests that M&E quality ratings are driven largely by whether the results framework measures outcomes, and not by other M&E failures such as weaknesses in M&E implementation. Independent Evaluation Group | World Bank Group 119 Projects Received Low Outcome Ratings as a Result of Figure C.8. No Overambitious Objectives Note: Numbers = number of project cases in each scenario; + = presence of factor; − = absence of factor. The framing of project objectives in a realistic manner is highly correlated with outcome rating—but is not suf ficient or necessary to generate low outcomes, and does so only in the presence of the previous two factors. 11 of 50 projects had objectives that were overambitious, either because the objective could not realistically be accomplished by the time of evaluation, or because the limited scale of the project could not realistically achieve the stated objectives. The presence of overambitious objectives is correlated with low outcome ratings—10 of the 11 projects with overambitious objectives received low outcome ratings. And yet, overambitious objectives by themselves do not seem suf ficient to cause low outcomes—all of the projects with low outcome ratings and overambitious objectives also either failed to produce their core outputs or did not provide suf ficient evidence on outcomes (see figure C.7). The presence of an economic analysis is strongly related with outcome ratings. Ef ficiency ratings are directly related to outcome by construction—rating assignment rules mean that projects with low ef ficiency ratings cannot receive high outcome ratings. The presence of economic analysis is a major driver of ef ficiency, as ef ficiency is dif ficult to demonstrate without economic analysis. For the 20 projects which provided an economic analysis, of them (65 percent) received high outcome ratings, whereas for the 25 projects which did not provide an economic analysis, only 32 percent received a positive outcome rating.8 Only 11 of 45 projects sought to value environmental benefits—these projects estimated a range of environmental benefits including reductions in pollution, greenhouse 120 Results and Performance of the World Bank Group 2017 | Appendix C gas emissions reduction, nutrient depletion, and natural disaster damages, along with benefi s from ecosystem services and biodiversity conservation. Projects that undertook such valuation efforts were more likely to receive high outcome ratings: 9 of 11 projects (82 percent) which valued environmental benefits received high outcome ratings, whereas only 12 of 34 projects (36 percent) with no such valuation received high outcome ratings. Inadequate client implementation capacity is often present with low outcome ratings—but usually only in cases where core outputs were not produced. Projects where evaluations cite low implementation capacity by clients are more likely to have low outcome ratings (14 of 15 projects have low outcome ratings) than are other projects (15 of 35 have low outcome ratings). However, for 11 of these 14 projects, the project also did not produce its core outputs, and all 6 projects which did not cite weak implementation capacity but did not produce their core outputs still received low outcome ratings. This suggests that there may be a tendency for project evaluations to conclude that implementation capacity was weak because outputs were not produced. Weakness in implementing capacity may well be an important underlying factor behind low project outcomes, but it may not be observable independently of the actual project implementation experience. However, there is evidence that projects with environmental implementation agencies are more likely to note weaknesses in implementation capacity (9 of 21 projects, 56 percent) than projects with nonenvironmental implementing agencies (6 of 29 projects, 21 percent). Other hypotheses tested did not find significant factors. Projects receiving financial support from the Global Environment Facility did not receive lower outcomes than other environment projects.9 Multisector projects (with project implementers across multiple sectors) did not receive lower outcomes than single sector projects. There was no stable relationship between project size and outcome rating as the direction of relationship depended on the project size threshold. There was weak evidence that projects that sought to change behavior at the household level received lower outcome ratings than other projects—but this was largely explained by such projects being less likely to be able to provide suf ficient evidence on outcomes. Findings on Environmental Sustainability from Recent IEG Reports The World Bank Group has made significant progress in improving pollution management, but important gaps remain. An IEG evaluation on pollution abatement efforts found that even though the World Bank Group has had significant engagement on pollution, the magnitude of its engagement has not kept pace with increasing global pollution levels (World Bank 2017c). The evaluation had five recommendations: (i) Improving pollution monitoring would help to provide the data needed to make countries aware of the severity of pollution. (ii) Strengthening country analytical work on pollution and greater integration in Systematic Country Diagnostics would help ensure that pollution concerns were adequately prioritized in country strategies and national policies. (iii) Scaling up the the pollution management portfolio and re-calibrating it toward the pollutants with the most serious consequences (especially air pollution) would increase the impact of the World Bank Group on pollution. (iv) Independent Evaluation Group | World Bank Group 121 Leveraging the climate change portfolio by emphasizing local pollution cobenefits would also support this. (v) Strengthening IFC’s advisory support to investment clients that lack technical knowledge could help to improve client compliance with pollution requirements. The World Bank Group has achieved localized air pollution abatement along key urban transit corridors or systems, but broader environmental benefits could be achieved by combining upstream (policy and sector framework) and downstream (operational) measures. An IEG evaluation of urban transport found that of such operations, many more supported climate change mitigation measures than adaptation measures (World Bank 2017b). Most projects were successful in direct pollution abatement, with 70 percent achieving their targets for reduced emissions of greenhouse gases or air pollutants. But modal shift from vehicles to public transit or nonmotorized transport did not materialize, which confined the environmental benefits to the vicinity of the project area or to the specific transport mode supported by the World Bank Group. The World Bank has been generally successful in supporting clients in issues of water pollution and extreme water stress, but there is little evidence of coordinated efforts on water and sanitation issues within the World Bank or by client governments. An IEG evaluation on water supply and sanitation found that urban growth and self-provision of water have placed increased stress on groundwater quantity and quality, that unregulated wastewater and sludge disposal has resulted in increased water pollution and contamination, and that climate variability is increasing uncertainty in water availability (World Bank 2017d). The evaluation found that two-thirds of World Bank projects supporting wastewater treatment had moderately satisfactory or better performance for relevant performance indicators. Projects that were less successful fell short in institutional arrangements and support, connectivity to waste collections systems, solid waste management, or government commitment. Increasing water scarcity has highlighted the need for wastewater reuse, but facilitating dialogue between irrigation agencies and water utilities has been dif ficult without a well-established policy and regulatory framework. Disaster risk management and climate resilience work has been one of the most successful avenues of World Bank Group support for small states. A Cluster Country Program Evaluation on small states looked in particular at disaster risk management programs in the Organization of Eastern Caribbean States and the Pacific Islands (World Bank 2016). The evaluation found that disaster- related infrastructure investments had been generally successful (although with cost overruns and maintenance concerns). Results on capacity building were mixed: preparedness work and warning systems were generally effective, but capacity building often failed to achieve sustained capacity, and knowledge work was not always used. Modest progress was made on policy improvements, with successes on raising awareness on financial disaster risk management but limited progress on land use and zoning. Despite many successes at the project level, only a modest dent has been made in overall vulnerability. Direct financing of infrastructure will struggle to meet financing needs, so changing incentives and behavior toward risk reduction will be critical. More could be done to address long-term climate change, in addition to adapting to current climate risks. Access to climate finance is still dif ficult and needs to be simplified. 122 Results and Performance of the World Bank Group 2017 | Appendix C Environmental policy lending has supported a variety of policy reforms, but has generated a common set of lessons around issues of political economy, of operation design and preparation, and of institutional strength and capacity. An IEG learning product found that environmental development policy lending is most effective when used in a way that plays to the strengths of the instrument: when policy issues (rather than capacity or enforcement) are the main barriers, when the World Bank has prior knowledge of the country and sector and strong instuonal relaonships, and when the policy issues are those that need addition from high-level decision makers (World Bank 2016). It found that operations were more effective when they could clearly articulate the additionality of the World Bank in improving the country’s policy reforms. And it found that suf ficient analytical work and technical assistance was not always present. Review of IFC and Multilateral Investment Guarantee Agency Environmental-Related Performance Standards IFC support improved the environmental performance of real sector projects; however, one-fifth of the evaluated projects could not meet IFC’s environmental requirements at the time of IEG’s evaluation.10 Sixty-nine percent of the 222 IFC real sector projects evaluated in CY12–16 that trigged either of the three environmental-related Performance Standards had satisfactory or better ratings (excellent or satisfactory) at appraisal. And although one-third of the evaluated projects had less than satisfactory (partly unsatisfactory or unsatisfactory) rating at this early process,11 it simply means that, as a condition of approval, the project sponsors had made a legal commitment to meet IFC’s environmental-related Performance Standards requirements including implementation of an Environmental and Social Action Plan (ESAP) that would be expected to bring the project into compliance. Appendix B describes the methodology in detail. At evaluation, 81 percent were rated positively (that is, excellent or satisfactory) for meeting IFC’s environmental-related Performance Standards after at least five years of operation (table C.5). However, a fifth of the evaluated projects did not meet or implemented fully the requirements of IFC’s environmental-related Performance Standards and the ESAP at the time of evaluation, at least five years after project approval. Of the three environmental-related Performance Standards, IFC had the highest share of satisfactory or better ratings for its compliance with PS6 (Biodiversity Conservation and Sustainable Management of Living Natural Resources) requirements at appraisal and at evaluation. The share of positive ratings (that is, excellent or satisfactory or S+ ratings) in PS3 (Resource Ef ficiency and Pollution Prevention) and in PS1 (Assessment and Management of Environmental and Social Risks and Impacts) follows consecutively. A comparison of the satisfactory or better (S+) ratings at appraisal and at evaluation for each Performance Standards in presented in figure C.9. Independent Evaluation Group | World Bank Group 123 Performance Standards and Ratings of Table C.5. Environment-Related Evaluated IFC Investment Projects, CY12–16 PS3: Pollution Prevention PS6: Biodiversity Conserva- PS1: E&S Assessment & and Abatement tion and Sustainable Natural Management System Resource Management Ratings At Appraisal At Evaluation At Appraisal At Evaluation At Appraisal At Evaluation Excellent (no.) 2 15 1 7 1 2 Satisfactory 112 127 109 140 38 38 (no.) Partly 71 53 53 39 8 2 unsatisfactory (no.) Unsatisfactory 3 4 3 2 1 1 (no.) NOP (no.) 1 4 7 16 1 9 NA (no.) 23 8 39 8 163 160 Share of 61 71 66 78 81 93 Satisfactory or better rating, excluding NA, NOP (percent) Total Real 188 199 166 188 48 43 Sector Projects, excluding NA, NOP (no.) Source: Synthesis Review of the results of the environmental-related Performance Standards of International Finance Corporation and Multilateral Investment Guarantee Projects by the Independent Evaluation Group, 2017.. Note: Success Rate = Excellent or Satisfactory Ratings/Total Number of Projects with Ratings. Excludes projects with not applicable, no rated or no opinion possible data entries. E&S = environmental and social; NA = not applicable; NOP = no opinion possible. Although ratings at appraisal and at evaluation remained the same for nearly three-quarters of the evaluated projects, 22 percent were rated positively for demonstrating improvements at evaluation. Only 5 percent had high ratings (S+) at appraisal but were rated low (partly unsatisfactory or unsatisfactory) at evaluation (table C.6). On balance, therefore, IFC’s support had a positive effect (measured as change in ratings at appraisal and at evaluation) on the project’s environmental performance, particularly with PS1 which had the lowest share of satisfactory or better ratings at appraisal. By environmental and social risk category, category A projects (those with “potential significant adverse impacts) had higher share of satisfactory or better ratings than category B projects (figure C.10). The results seem to be consistent with the profile of category A projects, which typically involves more experienced sponsors and with adequate Environmental and Social Management Systems (ESMS) already developed at the time of IFC’s appraisal. While category B projects, or projects with “limited impacts,” have lower share of S+ ratings than category A projects, the latter showed the most improvement in all three environmental-related Performance Standard. 124 Results and Performance of the World Bank Group 2017 | Appendix C Figure C.9. IFC’s Satisfactory or Better Ratings, at Appraisal and Evaluation, in Three Environmental-Related Performance Standards, CY12-CY16 a. PS1: Social and Environmental Assessment and Management System b. PS3: Pollution Prevention and Abatement c. PS6: Biodiversity Conservation and Sustainable Natural Resource Management Source: Independent Evaluation Group data for International Finance Corporation investments projects. Note: The share of positive ratings presented above does not indicate any significant trends over time, at the aggregate or the Performance Standard specific level due to the short time period. IFC = International Finance Corporation; PS = Performance Standard. Independent Evaluation Group | World Bank Group 125 Table C.6. Impact of IFC Support on Environment-Related Performance Standards Positive Effect Negative Effect No Change Performance Standard [n (%)] [n (%)] [n (%)] PS1: Environmental & Social 46 (25) 11 (6) 127 (69) Assessment and Management System PS3: Pollution Prevention and 32 (20) 5 (3) 121 (77) Abatement PS6: Biodiversity Conservation 6 (16) 1 (3) 31 (82) and Sustainable Natural Resource Management Total Number of Observations 84 (22) 17 (5) 279 (73) Source: Synthesis Review of the results of the environmental-related Performance Standards of International Finance Corporation and Multilateral Investment Guarantee Projects by the Independent Evaluation Group, 2017. Note: These statistics only include projects for which both “at appraisal” and “at evaluation” ratings are available and excludes rating of not applicable and no opinion possible. PS = Performance Standard. By environmental and social risk category, category A projects (those with “potential significant adverse impacts) had higher share of satisfactory or better ratings than category B projects (figure C.10). The results seem to be consistent with the profile of category A projects, which typically involves more experienced sponsors and with adequate Environmental and Social Management Systems (ESMS) already developed at the time of IFC’s appraisal. While category B projects, or projects with “limited impacts,” have lower share of S+ ratings than category A projects, the latter showed the most improvement in all three environmental-related Performance Standard. Greenfield projects have higher share of satisfactory or better ratings compared with brownfield (or existing) projects. However, greenfield projects tend to have lower PS1 ratings at appraisal and although ratings improved at evaluation, the difference in the two sets of ratings is the same as brownfield projects (figure C.11) The reason is likely due to the greenfield project sponsors’ efforts to develop their environmental management systems. But greenfield projects have better ratings in PS3 and PS6 because environmental considerations had been integrated in the design of these new projects and therefore, less challenging to implement compared with brownfield projects, which require retrofitting. Projects with large amount of IFC investments experienced greater improvement in satisfactory or better ratings from appraisal and at evaluation compared with small and medium-size projects.12 In addition, small investments faced challenges in complying with PS3 requirements compared with the medium- and large-size investments. Figure C.12 illustrates the performance at appraisal and at evaluation of the different project sizes. 126 Results and Performance of the World Bank Group 2017 | Appendix C Figure C.10. IFC’s Satisfactory or Better Ratings in Three Environmental-Related Performance Standards, at Appraisal and at Evaluation, by Environmental Risk Category, CY12–16 Environmental performance of environmental risk category A and category B projects Source: Independent Evaluation Group data for International Finance Corporation Investments projects. Satisfactory or Better Ratings in Three Environmental-Related Figure C.11. IFC’s Performance Standards, at Appraisal and at Evaluation, by Type of Project, CY12-CY16 Environmental performance of greenfield and b ownfield p ojects Source: Independent Evaluation Group data for International Finance Corporation investments projects. Note: Greenfield projects refer to newly created or built projects. Brownfield projects refer to existing project enterprises. Independent Evaluation Group | World Bank Group 127 Figure C.12. IFC’s Satisfactory or Better Ratings in Three Environmental- Related Performance Standards, at Appraisal and at Evaluation, by Project Size, CY12-CY16 Environmental performance of small, medium and large-sized investment projects Source: Independent Evaluation Group data for International Finance Corporation investments projects. Note: Project size refers to IFC’s net commitment amount. Project size is considered small if the total amount is equal or less than $4.32 million; medium if it is between $4.32 million and $35.74 million; and large if it is greater than $35.74 million. IFC = International Finance Corporation. In sum, the review suggests that IFC’s results and performance in ensuring adherence to its PS1, PS3 and PS6 requirements is broadly consistent with the environmental risks associated with the projects. The share of S+ ratings are highest in category A projects, where the potential environmental risks are greatest. In terms of specific Performance Standards, the effects of IFC’s support have also been most positive with respect to PS1, where “at appraisal” S+ ratings were lowest. Higher PS1 ratings in projects by the manufacturing, agribusiness, and services industry group (with more than 10 projects each in the review sample), including subindustries Health Care, Information, Chemicals, and Industrial and Consumer Products, can lift IFC’s overall positive effects. Review of Multilateral Investment Guarantee Agency Environmental-Related Performance Standards Multilateral Investment Guarantee Agency (MIGA) support improved the environmental performance of real sector projects; however, one-fifth of the evaluated projects could not meet MIGA’s environmental requirements at the time of IEG’s evaluation.13 Seventy-five percent of the 34 evaluated real sector projects in FY12 to FY16 that trigged either of the three environmental- related Performance Standards had satisfactory or better ratings (excellent or satisfactory) at the due diligence stage. And although 25 percent of the evaluated projects had less than satisfactory rating during this stage (meaning these were not in compliance with MIGA’s environmental- related Performance Standards), it simply indicates that, as a condition of approval, the guarantee holders made a legal commitment to meet MIGA’s environmental-related Performance Standards requirements including implementation of an ESAP that would be expected to bring the project into 128 Results and Performance of the World Bank Group 2017 | Appendix C compliance.14 At evaluation, 82 percent were rated positively (that is, excellent or satisfactory) for meeting MIGA’s environmental-related Performance Standards after at least three years of project operation. Appendix B describes the methodology in detail. The share of satisfactory or better (S+) ratings was highest for PS6 (Biodiversity Conservation), followed by PS3 (Pollution Prevention and Abatement) and PS1 (E&S Assessment and Management Systems). Positive effect is highest with respect to compliance with PS1, with “at due diligence” ratings had been weakest (table C.7). Due to the small number of MIGA projects evaluated, the review was not able to draw conclusions regarding environmental-related performance by environmental risk category, sector or type of project. Performance Standards and Ratings of Table C.7. Environment-Related Evaluated MIGA Projects, CY12–16 PS6: Biodiversity Conser- PS1: E&S Assessment & PS3: Pollution Prevention vation and Sustainable Management System and Abatement Natural Resource Manage- (no.) (no.) ment (no.) Ratings At Appraisal At Evaluation At Appraisal At Evaluation At Appraisal At Evaluation Excellent 0 0 0 1 0 1 Satisfactory 20 26 17 20 5 5 Partly 8 3 7 6 0 0 Unsatisfactory Unsatisfactory 0 0 1 1 0 0 No Opinion 1 4 1 4 2 5 Possible (NOP) Not Applicable 5 1 8 2 27 23 (NA) Share of 71 90 68 75 100 100 Satisfactory or better ratings, excluding NA, NOP (%) Total Number 28 29 25 28 5 6 of Real Sector Projects Evaluated, excluding NA, NOP Source: Synthesis Review of the results of the environmental-related Performance Standards of International Finance Corporation and Multilateral Investment Guarantee Projects by the Independent Evaluation Group, 2017. Note: MIGA = Multilateral Investment Guarantee Agency. IFC and MIGA Implementation of IEG’s Recommendations in the 2010 Safeguards Evaluation In the 2010 safeguards evaluation, IEG had a recommendation to IFC, MIGA, and the World Bank and additional recommendations specific to each institution. This section summarizes first IEG’s Independent Evaluation Group | World Bank Group 129 recommendations in the 2010 evaluation and the findings from IEG’s review of IFC and MIGA implementation of the recommendations. IEG’s review covered a select sample of IFC investment and MIGA projects approved in FY12 to FY16 and supplemented by staff interviews. Details of the methodology used is described in appendix B. Harmonizing environmental and social risk categorization. IEG recommended that IFC, MIGA, and the World Bank jointly adopt and use a shared set of objective criteria to assess social and environmental risks to ensure adequacy and consistency in project categorization across the World Bank Group, using the more inclusive criteria for category A, and refining the categorization system to address the bunching of higher and lower risk projects within the current category B. Finding: IFC and MIGA have developed and applied inclusive criteria for determining the environmental and social category that include guidance on contextual risk assessment to classify projects better into category A and B. IEG’s recent review found that IFC and MIGA had correctly categorized its projects after the approval of the 2012 Sustainability Frameworks (20 for MIGA). With the new approach, the share of IFC category A projects increased from 4 percent between FY06-09 (the period covered by the 2010 Safeguard Evaluation) to percent between FY-16, the period after IFC adopted its 2012 Sustainability Framework. (table 4). The share of MIGA category A projects also increased from 16 percent in FY00-09 period to 26 percent in FY-16. However, there is room for improvement, specifically in distinguishing higher and lower risk projects within the category B, which the World Bank has adopted. Classification of financial intermediary (FIs) are also differed between IFC and MIGA, on one hand, and the World Bank, on other.15 Table C.8. Share of Category A Projects in IFC and MIGA Portfolio E&S Category Institution (Evaluations) Commitment Year A B Total (A+B) Share of A% IFC Pre PS 2006 (IEG SG Evaluation 2010) FY98–05 62 934 996 6.2 Post PS 2006 (SG study 2010) FY06–09 14 378 392 3.6 Post PS 2012 (RAP 2017) FY13–16 73 476 549 13.3 MIGA IEG SG Evaluation 2010 FY00–09 4 21 25 16.0 Post PS 2012 (RAP 2017) FY13–16 15 42 57 26.3 Source: Implementation of IEG Safeguard Evaluation 2010 Recommendations on Performance Standards, World Bank 2017. Note: E&S = environmental and social; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency; RAP = Results and Performance of the World Bank Group; SG = safeguards. IFC-Specific Recommendations and IEG Review Findings Addressing climate change aspects in project appraisal and monitoring. IEG recommended that IFC strengthen the provisions on sustainability to address climate change. 130 Results and Performance of the World Bank Group 2017 | Appendix C Findings: IFC is using an Excel-based Carbon Emission Estimation Tool (CEET) to calculate project- specific emissions. It also reduced the greenhouse gas emission reporting threshold under IFC’s Performance Standard 3 (2012) from 100,000 tons CO2 to 25,000 tons of CO2 per year. IEG’s recent review noted that IFC reassess the purpose of the CEET it has not been applied properly except in power sector projects; thus, it has not been useful for demonstrating development of CO2 intensity in IFC projects. Staff working in other sectors need training to use the CEET. IEG also noted that in projects with potentially high greenhouse gas emissions, identification of alternatives (including technically and financially cost-effective) to carbon-intensive solutions has been inconsistent and needs improvement. Addressing supply chains and their commodity certification. IEG recommended that IFC strengthen its assessment of supply chains and commodity certification. Finding: IFC (and MIGA) had addressed supply chain issues have been more thoroughly addressed in its 2012 (20 for MIGA) Sustainability Frameworks, specifically biodiversity supply chain provisions aspects have been added to PS6.16 The supply chain considerations are mainly focused on primary suppliers’ child and forced labor issues (PS2) and biodiversity (PS6). However, these issues are thoroughly addressed in agribusiness projects mostly and not in other sectors.17 Staff interviews also acknowledged projects involving supply chains for single-source commodities (such as chemical, power and pulp & paper industries) are only loosely monitored. Furthermore, IEG’s portfolio review of high-risk financial intermediary (FI-1) projects noted that information provided by subproject clients was insuf ficient to assess their compliance with IFC’s and MIGA’s Performance Standards. This conclusion is consistent with the Compliance Adviser / Ombudsman’s 20 and 2017 reports, which finds that FI-1 supply chains and subprojects do not provide suf ficient information to assess compliance with Performance Standards. Strengthening disclosure policies and practices. IEG recommended that IFC strengthen its disclosure policies and practices on disclosure, including at the local levels. Finding: Compared with IFC’s 2006 Disclosure of Information Policy, the requirements for the disclosure of project-level information have not changed much compared with the 2012 Access to Information Policy (AIP). The requirements for disclosing SII/SPI and Environmental and Social Review Summary (ESRS) have remained about the same, however 2012 AIP requires Broad Community Support process documents, ESIA and third-party monitoring reports for A and B category projects, and ESAP implementation status to be disclosed. IFC consistently discloses ESAP and impact assessment at appraisal but have not disclosed subsequent ESAP updates, which describe progress toward achieving E&S targets. The ESAP updates describe how the E&S targets have been achieved during the project course and should be disclosed to the stakeholders and public in greater extent, as required in the 2012 AIP. Independent Evaluation Group | World Bank Group 131 Incentives for Performance Standards ownership. IEG recommended that IFC develop incentives for investment of ficers to share ownership of the Performance Standards and mainstream their implementation. Finding: IFC’s incentives to encourage ownership of the Performance Standards include linking investment of ficers’ performance awards to the E&S performance of the projects, ensuring that repeat IFC investments are now conditional on an Environmental and Social Risk Rating of at least satisfactory. While more training is provided in meeting Performance Standards, staff interviews acknowledged that building capacity and awareness on IFC’s Sustainability Framework remains challenging. On the Use of IFC Advisory Services for developing Clean Technology and Environmental Management. IEG recommended that IFC use its advisory services to build social and environmental management systems (SEMS) and implementation capacity, especially among small and medium enterprises, financial intermediaries, and clients in countries and sectors with weak environmental and social management Finding: In its 2012 Sustainability Framework, IFC integrated its advisory and investment products to enhance to enhance the projects’ E&S impacts.18 Staff interviews noted that past instances when IFC investment offered clients advisory services for energy ef ficiency but client uptake is still limited. And in some of the greenfield projects reviewed by IEG, there was no need for environmental advisory services because clean technology is already integrated into the design or project sponsor has strong competency and well-functioning ESMS. Present arrangement has located IFC advisory services within IFC investment departments, which offer more opportunities to systematically identifying opportunities to address a range of client needs. Use of IFC Loan Covenants for Compliance. IEG recommended that IFC make greater use of its loan covenants, including Conditions of Disbursement, to enforce compliance with environmental and social requirements and reporting if the clients lack commitment and are continuously failing to meet IFC’s requirements. Finding: IFC has made greater use of legal documents to ensure compliance. As example, procedures for establishing legal E&S covenants at appraisal and monitoring and supervision are described in IFC’s 2012 Environmental and Social Review Procedures (ESRP). And while IFC’s 2012 Sustainability Framework lacks guidance on enforcing E&S compliance using loan covenants, breaches are usually addressed in the projects’ legal documents. Approval of repeat investments and condition for loan disbursements are based on meeting specific criteria.19 According to staff interviews, IFC has withdrawn from three to four loans on environmental and social grounds since 2010. Although the number of projects reviewed is too small to draw wider conclusions, linking the Environmental and Social Risk Rating to loan disbursement appears to be an effective incentive for ensuring project compliance with E&S safeguards. 132 Results and Performance of the World Bank Group 2017 | Appendix C Third-party and community monitoring. IEG recommended that IFC to make use of independent, third-party, or community M&E for its projects, particularly for projects with involuntary resettlement and higher-risk FI and agribusiness projects.20 Finding: Third-party or community monitoring is addressed in IFC’s 2012 Environmental and Social Policy and specifically in Performance Standard 1 (PS1) and Performance Standard 5 (PS5), which encourage client participation in third-party, government, or community monitoring. The 2012 AIP provides for the disclosure of third-party monitoring reports. In category A projects, community monitoring is used when relevant (for example, in large gas pipeline projects). Community representatives sometimes join monitoring teams and in some projects investigated by the Compliance Adviser / Ombudsman, community monitoring had been proposed. IEG’s present review indicate that while third-party monitoring is used mostly in category A projects, not all high- risk projects apply this type of monitoring. Additionally, third-party nor community monitoring and monitoring reports are not adequately disclosed, leaving some gaps where disclosure could be further enhanced. IFC’s approach to FI projects. IEG recommended that IFC develop more robust approaches to the implementation of the Performance Standards in FI projects, listed equities, and trade finance. Finding: The categorization and supervision requirements of FI projects in the 2012 ESP and ESRP follows a risk-based approach that is more robust and detailed compared with the 2006 Sustainability Framework.21 Of the few FI projects reviewed by IEG, all were correctly categorized to FI-1, as these projects involved large corporate loans or project financing of high and medium risk industry sectors. But not all the projects had suf ficient subproject information to assess compliance and the few FI-1 projects reviewed by IEG also noted the weak assessment of Performance Standards compliance in Annual Environmental Performance Reports. And, IFC’s assessment does not provide consistent and reliable information on how high-risk subprojects with comply with IFC’s Performance Standards requirements. Although this IEG review covered only a few projects, its findings seem consistent with the 20 and 2017 Compliance Adviser / Ombudsman reports on the sustainability of IFC’s FI projects. The reports also noted that IFC’s reporting lacked evidence on compliance with Performance Standards requirements including supply chains in high-risk, large corporate loans and project finance subprojects. In sum, supervision of FI projects’ compliance with IFC Performance Standards continues to be challenging and needs improvement. MIGA-Specific Recommendations and IEG Review Findings Enhancing the capacity of MIGA’s Environmental and Social Unit.22 IEG recommended that MIGA increase the capacity of its Environmental and Social Unit to the level needed to provide credible assurance on performance against the standards for every project. Independent Evaluation Group | World Bank Group 133 Finding: MIGA has increased its resources (E&S staff, budget, procedures) to manage E&S risks in its portfolio. The number of staff in its Environment, Social, and Integrity unit has increased from six to eight specialists and consultants. It introduced Environmental and Social Risk Rating and Knowledge Gap concepts that reduced E&S performance and information deficiencies. Use of the Japan Trust Fund. IEG recommended that MIGA to expand the size and eligibility of the Japan Trust Fund (JTF) for Addressing Environmental and Social Challenges to all low-capacity clients based on need. Finding: MIGA has used JTF from 2006 to address environmental and social needs of low-capacity clients in African countries. In 2010, an additional grant was provided and the Trust Fund operation was extended to 2015. Twenty engagements in building ESMS capacity and enhancing community health and safety were supported in agribusiness, mining, solar and hydropower sectors in 15 countries. When the JTF was closed, the cumulative grant was $1.4 million and the remainder was returned to the donor. There has been no follow-up of this type of funding to support low-capacity clients after the JTF closed. Strengthening MIGA’s disclosure policies and practices. IEG recommended that category B, Small Investment Program (SIP) projects follow the same disclosure requirements as for regular guarantee category B projects, and disclose project-level environmental and social information from supervision reports. Finding: MIGA’s 20 AIP states that for category B projects underwritten through the SIP, it will issue a short summary of the expected environmental and social impacts or risks in the Summary of Proposed Guarantee instead of an ESRS. And it will disclose, as appropriate, project’s environmental and social assessment documentation or an ESAP. IEG found that only a few SIP projects are publicly disclosed. MIGA does not disclose E&S information from supervision reports, but in projects that may have an impact on communities, it requires clients’ regular disclosure of the E&S impacts and mitigation measures to the affected communities. IEG notes that the practices described in its 20 AIP and 20 Performance Standard 1 (PS1) met minimum requirements for regular disclosure of project E&S impacts to communities by the clients, but not the full IEG recommendation for MIGA to disclose its supervision reports. Additionally, ESAPs at the due diligence stage were not disclosed in MIGA Project Information Portal, which it should ensure as stated in the 20 AIP. MIGA’s approach to FI projects. IEG recommended that MIGA to develop a credible mechanism to ensure its financial sector projects comply with its Performance Standards. Finding: MIGA’s approach to FI projects under the 20 Policy on Environmental and Social Sustainability involves regular monitoring via Annual Monitoring Reports submitted by its FI clients. MIGA also conducts site visits to the subsidiary level (project enterprise) and selected subprojects, using a risk-based approach. The ESRP 20 gives detailed requirements for MIGA E&S appraisal and monitoring. These policies are consistent with the original IEG recommendation. Despite these 134 Results and Performance of the World Bank Group 2017 | Appendix C policies, the absence of monitoring information makes it dif ficult to assess the implementation performance of FI-1 (highest risk) projects. Specifically, the lack of Annual Environmental Performance Reports and MIGA review makes impossible to assess the adequacy of FI projects’ implementation of MIGAs requirements. These requirements include developing a SEMS, compliance of high-risk subprojects with MIGA’ s Performance Standards and supply chain management requirements. References World Bank. 2016. Lessons from Environmental Policy Lending. Independent Evaluation Group. Washing- ton, DC: World Bank. ———. 2017a. “Cool Markets” for GHG Emission Reduction in a Warming World: Evaluation of World Bank Group’s Support to Carbon Finance. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017b. Mobile Metropolises: Urban Transport Matters. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017c. Toward a Clean World for All: An IEG Evaluation of the World Bank Group’s Support for Pollution Management. Independent Evaluation Group. Washington, DC: World Bank ———. 2017d. A Thirst for Change An IEG Evaluation of the World Bank Group’s Support for Water Supply and Sanitation with Focus on the Poor, FY 2007–16. Independent Evaluation Group. Washington, DC: World Bank. 1 Common indicators included estimated greenhouse gas emission reductions, the number of contaminated dump sites that are closed, the volume of pollutants that are removed or stored according to environmental safe fashion, and reduction in wastewater discharge. 2 This included several pollution abatement projects that are carried out as Program-for-Results or as on-lending schemes to provide incentives for industry to install systems to abate pollution. The infrastructure and systems to collect data on results is built into projects design. Many of these projects also incorporate third-party verification, enhancing the validity of data. 3 Some fisheries governance projects include good indicators to measure intermediate outcomes such as the extent to which catches remain within regional fisheries quotas. 4 The METT was developed by the IUCN World Commission on Protected Areas and is used in many protected area projects (including those supported by the World Bank and the Global Environmental Facility, GEF). The METT been designed to track and monitor progress towards worldwide protected area management effectiveness over time. The methodology is a rapid assessment based on a scorecard questionnaire that includes all six elements of management identifi d in the IUCN-World Commission on Protected Areas Framework (context, planning, inputs, process, outputs and outcomes), but has an emphasis on context, planning, inputs and processes. The quality of each METT depends on the knowledge and diligence of the assessors, and the integration of information from a diverse range of stakeholders. 5 There were two exceptions of sustainable land and water resources management projects that measured biophysical impacts such as reduction in the level of soil erosion, incremental increases incremental in biomass, and changes in Independent Evaluation Group | World Bank Group 135 groundwater levels. These projects were the China Climate Smart Staple Crop Production project and the India National Watersheds Project (Neeranchal), and were mapped to the agriculture and water GPs, espectively. 6 For example, several projects used the establishment of operational disaster management systems or setup of hydrometeorolgical equipment as proxies for capacity enhancement. A few recent projects included measures of capacity performance, such as improvements in forecasting accuracy or measure changes in forecasting skill against an established criterion. 7 Qualitative comparative analysis (QCA) is a good fit for the analysis of a relatively small number of cases. QCA techniques allow for identifying multiple paths to the same outcome and to identify combinatiosn of causally relevant factors that generate a particular outcome. It is also less sensitive to measurement error than statistical techniques. 8 Note that 5 projects are excluded here, as they were Development Policy Operations, which provide budget support to clients and are not evaluated on ef ficiency. 9 Historically, it may have been the case that projects with Global Environment Facility (GEF ) financing received lower outcomes. This pattern has been observed in other Independent Evaluation Group evaluations which look at a longer time period (such as Independent Evaluation Group evaluation, Toward a Cleaner World for All; World Bank 2017b). However, since a 2014 agreement between Independent Evaluation Group and World Bank management, projects with GEF financing have been evaluated solely against their project development objective and not their GEOs. This may be an important factor in explaining why recently evaluated GEF projects do not have lower outcome ratings than other environmental projects. GEF projects also were no more likely to have overambitious objectives than projects without GEF fi ancing. 10 The review covered all evaluated IFC real sector investment projects in the CY12 to CY16 XPSR Programs that triggered the Performance Standard 1, Assessment and Management of Environmental and Social Risks and Impacts; Performance Standard 3, Resource Ef ficiency and Pollution Prevention; and Performance Standard 6, Biodiversity Conservation and Sustainable Management of Living Natural Resources. Please refer to appendix B for a detailed description of the methodology. 11 Less than satisfactory rating given by the International Finance Corporation (IFC) at appraisal means that the project was not in compliance with IFC’s environmental-related performance standards at this early stage of the project process. 12 International Finance Corporation (IFC) investment size is small if its total net commitment to the project is equal or less than $4.32 million; medium if it is between $4.32 million and $35.74 million; and large if IFC’s net commitment to the project is greater than $35.74 million. The Independent Evaluation Group evaluated the environmental performance of 34 Multilateral Investment Guarantee 13 Agency real sector guarantees during FY12–16. 14 The legal commitment is typically included in the contract of guarantee that the guarantee holder and MIGA signs. 15 The International Finance Corporation and MIGA have three environmental and social risk categories applicable to financial intermediary (FI) projects: FI-1 (the highest risk), FI For FI projects the categories are FI-1 (highest E&S risk), FI-2 and FI-3. In the 2017 World Bank Environmental and Social Framework, the projects including FIs are categorized according to four classifications: high risk, substantial risk, moderate risk, or low risk. 16 The International Finance Corporation developed new tools to address supply chains, the most important of which are: “Good Practice Handbook Assessing and Managing Environmental and Social Risks in an Agro-Commodity Supply Chain (2013)” and “Tip Sheet Performance Standard 2, Addressing Child and Forced Labor in the Supply Chain of Retailers”, which provide detailed guidance on the subject matter. 17 Examples of supply chain considerations in other sectors include health and safety aspects of transport and distribution suppliers, and environmental aspects of mining and power sectors, and supply chain review should be expanded and address primary single source suppliers of relevant commodities and transportation and distribution. 136 Results and Performance of the World Bank Group 2017 | Appendix C 18 The integration, appraisal and monitoring are described in a detailed manner in the International Finance Corporation’s 2012 Environmental and Social Policy and 2012 Environmental and Social Review Procedure. 19 Approval of repeat investments rests on having an environmental and social risk rating 2 (Satisfactory) or better. Loan disbursement is conditional upon having a satisfactory ESAP implementation score. 20 In the Management Action Record for the 2010 Safeguard Evaluation, International Finance Corporation management disagreed on independent/third party monitoring across the board. However, the Independent Evaluation Group’s recommendation on third party monitoring was not intended to cover all projects, but only to projects with involuntary resettlement and higher-risk financial intermediary and agribusiness projects. 21 The 2012 Environmental and Social Policy (ESP) classifi s financial intermediary (FI) projects into three environmental and social categories: FI-1 with highest risks, FI-2 with limited risks, and FI-3 with minimal risks. The environmental and social requirements for FI projects are described in detail in the 2012 ESP and in the guidance note, “Environmental and Social Policy for Financial Intermediaries.” Investments with project finance and corporate loans with tenor of not less than 36 months and funding-defined assets as part of a project amounting to at least $10 million of total capital cost, compliance with all Performance Standards (PSs) is required; for other corporate loans provided to a single client exceeding $5 million, on an aggregated basis over a period of 36 months, the FI should require the client’s Environmental and Social Management Systems and labor practices to be consistent with PS1 and PS2. IFC’s supervision and site visit requirements are defined in detail. 22 At the time of the 2010 Safeguard Evaluation, the Multilateral Investment Guarantee Agency (MIGA) was reforming its Sustainability Framework and during the process, it referred to the 2012 Sustainability Framework of the International Finance Corporation (IFC) in its documents, including MIGA Access to Information Policy and Performance Standards. In 2013 MIGA had the new Sustainability Framework approved by the Board; it reflects several IFC policies and procedures and reliance on other MDBs that rely on their own or IFC policies. MIGA Environmental and Social Review Procedure was created based on IFC version but with adaptions to project mix and cycles. Independent Evaluation Group | World Bank Group 137 Appendix D. Background on World Bank Project Performance Ratings Trends, comparisons and other analyses in this report that look at World Bank project performance often use quantitative or categorical data based on outcome ratings and World Bank performance ratings of World Bank projects. The ratings used to indicate World Bank project performance in this report are based on validation reviews done by the Independent Evaluation Group (IEG) of Implementation Completion and Results Reports (ICRs). This appendix provides background on the methodology behind these ratings as well as some contextual analysis. What Is an ICR? An ICR is one of the main instruments of self-evaluation of the World Bank. It is prepared by the World Bank at the close of every International Development Association (IDA) or International Bank for Reconstruction and Development (IBRD)-funded operation or, in the case of a series of programmatic policy operations, at the end of a series of operations. According to the guidelines to World Bank staff for preparing ICRs for investment project financing (IPF) projects, the ICR is intended to (i) provide accountability and transparency for the performance and results of each operation and (ii) capture and disseminate experience from operational design and implementation to improve the design and implementation of future interventions. An ICR is expected to constitute complete and systematic account of the performance and results of the project. In addition to telling the result story of the project, the ICR contains ratings of the project’s performance. The ratings often used in monitoring are the outcome rating, which is based on the subratings of the project’s relevance, ef ficacy, and ef ficiency, and the World Bank performance rating, which takes into account the World Bank’s performance in ensuring quality at entry as well as the World Bank’s performance in supervision of the project. Ratings scales and criteria were developed through a collaboration between the World Bank’s Operations Policy and Country Services (OPCS) and IEG. What Is an ICR Review? The ICR Review, conducted by IEG, is an independent, desk-based, critical review of the evidence, results, and ratings of the ICR in relation to the project’s design documents. Based on the evidence provided in the ICR and an interview with the last task team leader (task team leader), IEG arrives at its own ratings for the project, based on the same evaluation criteria used by the World Bank. In reviewing the findings and ratings in the ICR, IEG provides an independent view of the results and ratings, conditioned on the evidence presented in the ICR and from the last task team leader for the project. However, IEG is not privy to evidence that was not included in the ICR. The ICR Review is thus an independent validation of the World Bank’s self-evaluation and ratings; it is not an independent evaluation of the project based on evidence collected outside the World Bank’s self-evaluation process. 138 Ratings assigned by IEG in its ICR Reviews are aggregated into the performance percentages reported for World Bank projects in IEG’s RAPs as well as in the World Bank’s Corporate Scorecard. Definitions of the criteria and scales for each rating element are provided in IEG’s Guidelines for Reviewing World Bank Implementation Completion and Results Reports: A Manual for Evaluators (World Bank 2014). How Are ICRs and ICR Reviews Changing? Based in part on response to the recommendation in IEG’s report Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group to avoid excessive focus on ratings and make the ICR system more flexible and geared toward learning, OPCS and IEG undertook a collaborative reform process to simplify the structure of the ICR for IPFs and to improve related guidance (World Bank 2016). OPCS rolled out a new ICR structure on July 1, 2017, to be implemented in a phased manner: Any ICR initiated on July 1, 2017, or later uses the new ICR structure and guidance, while any ICR initiated earlier uses the previous ICR structure and guidance, regardless of the closing date of the project. Ratings data used for this Results and Performance of the World Bank Group (RAP), therefore, are not affected by the ICR reform. Next year’s RAP, however, is expected to include ratings from ICRs and ICR Reviews that use the new structure and guidance as well as ratings based on the previous structure and guidance. Figure D.1 provides a summary of IEG’s ICR Review ratings under the old and new structure. Figure D.1. Ratings Structure for IEG’s ICR Reviews Before and After ICR Reform Independent Evaluation Group | World Bank Group 139 Note: Elements outlined in blue are rated on a 6-point scale (from lowest to highest: highly unsatisfactory [HU], unsatisfactory [U], moderately satisfactory [MS], satisfactory [S], highly satisfactory [HS]). Elements outlined in green are rated on a four-point scale (from lowest to highest: negligible [N], modest [M], substantial [S], high [H]). Elements in a gray box are not rated but do have a discussion. ICR = Impletementation Completion and Results Report; IEG = Independent Evaluation Group; M&E = monitoring and evaluation. Which Projects Are Covered by IEG’s ICR Review Ratings? World Bank policy requires an ICR for each completed financing project to be submitted to the approving authority for that project. For financing projects funded through IBRD or IDA and approved by the World Bank’s Board, ICRs are submitted to the World Bank’s Board and, therefore, also submitted to IEG for validation. Financing projects for which an ICR is sent to an approving authority other than the World Bank’s Board—and that, as such, are not required to be submitted to IEG for validation—include fi ancing projects with the following funding categories: ƒƒ Recipient-Executed Trust Funds (although ICRs of recipient-executed trust funds of $5 million or above are generally submitted to IEG even though not required to do so) ƒƒ Global Environment Fund (medium size) ƒƒ Carbon Initiative ƒƒ Rain forest Initiative ƒƒ Institutional Development Fund ƒƒ Special Fund ƒƒ Debt Reduction Fund The concept note for RAP17 raised the question about the extent to which World Bank projects are covered by self-evaluation (for example, through ICRs or Notes on Canceled Operations [NCOs]) and independent evaluation (for example, through IEG’s ICR Reviews and Project Performance Assessment Reports). In response to this question, a special analysis done for this RAP of the cohort of World Bank financing projects approved in FY06–08 suggests that self-evaluation extending to IEG validation covered 63 percent of projects and 95 percent of volume.1 If cases where ICRs were not required to be submitted to the World Bank’s Board of Executive Directors—and therefore not required to be submitted to IEG—are included as falling within self-evaluation, however, then self- evaluation covered 99 percent of both projects and volume (see figure D.2.) Gap Analysis and Coverage of the World Bank Group Self-Evaluation 140 Results and Performance of the World Bank Group 2017 | Appendix D and Independent Evaluation Systems Although IEG reviews all ICRs it receives, IEG’s ICR Reviews do not cover the entire population of World Bank projects because not all World Bank projects are evaluated at closing and because not all World Bank projects that get evaluated at closing (for example, trust-funded projects) are required to submit those evaluations to the World Bank’s Board and to IEG. Some projects have an Implementation Completion Memorandum at closing rather than an ICR, but these are not reviewed by IEG. Figure D.2 summarizes the coverage of ICRs and ICR Reviews among a cohort of World Bank lending projects. of Self-Evaluation Among World Bank Financing Projects Figure D.2. Coverage Approved FY06–08 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data as of September 7, 2017. Note: CR = completion report; Dropped = Project was dropped (includes only projects dropped after Board of Executive Director approval; projects dropped before Board approval are excluded from the total); ICR = Implementation Completion and Results Report; ICR not sent to IEG = ICR or NCO was found in ImageBank, but has not been submitted to IEG; IEG = Independent Evaluation Group. Lack ICR = ICR required but not found in ImageBank; Not required = Projects for which no ICR was found in ImageBank, and that fell into a project type category for which the requirement is that the ICR is submitted to an approving authority other than theHIa World Bank Board and therefore is not required to be submitted to IEG. Examples include: Recipient-Executed Trust Funds, Global Environment Facility medium size, IDF, Carbon Initiative, Rain forest Initiative, Special Fund, and Debt Reduction Facility. Pipeline = Listed as a pipeline project; Have ICR Review = Projects that went through the complete self-evaluation and independent evaluation process. Still Active = Project is still under implementation; Too soon = projects that closed after June 30, 2016, such that they might plausibly still be working on an ICR. Independent Evaluation Group | World Bank Group 141 Incomplete Coverage of Project Ratings Data for FY16 and FY17 Arising from Waiver for the Republic of Yemen Portfolio On November 23, 2016, World Bank management waived the completion reporting requirement of Operational Policy / Bank Policy 10.0 for 23 projects under implementation in the World Bank’s Republic of Yemen portfolio as of March 2015, when disbursements were suspended due to an ongoing civil conflict. Of these projects 23 projects, an Implementation Completion and Results Report would have been required for 11 of the projects closed in FY16 and FY17 (two projects totaling $53 million and nine projects totaling $183 million in disbursements, respectively). Without an ICR, IEG cannot provide ratings data. General Coverage of Project Ratings Data for this Report IEG validates all ICRs received, whether those ICRs are required to be submitted to the Board or to other approving authorities. Among 734 projects closed in in FY14–16 with ICRs completed and received in IEG, IEG had completed 709 ICR Reviews as of October 2, 2017, for a coverage rate of 97 percent in this report. By comparison, Results and Performance of the World Bank Group 2016 was based on 95 percent coverage. Figure D.3 summarizes the number of ICRs received by IEG for projects closed in each fiscal year, along with the percentage of ICRs received for which IEG has completed the ICR Review. Independent Evaluation Group’s ICR Review Coverage, as of Figure D.3. The October 2, 2017 142 Results and Performance of the World Bank Group 2017 | Appendix D Source: World Bank Business Intelligence; IEG ICR Review process data. Note: This report is based on 97 percent coverage (709 ICR Reviews completed among 734 projects closed in FY14–16 with ICRs completed and received in IEG). Ratings data is as of October 2, 2017. By comparison, Results and Performance of the World Bank Group 2016 was based on 95 percent coverage (792 ICR Reviews completed among 839 projects closed in FY13–15 with ICRs completed and received in IEG, with ratings reported as of October 7, 2016). An important caveat is that the FY of project closing is assigned based on information from the World Bank Business Intelligence system, which relies on accurate completion of the actual final closing date in the Operations Portal record for the project. F Y = fiscal year; ICR = Implementation Completion and Results Report; ICR Review = Implementation Completion and Results Report Review. The algorithm used for calculating the final closing fiscal year of a project is as follows: Calculate the Derived Closing Date based on data availability of three fields in the following sequence: REV_CLOSE_DATE, CLOSING_ DATE, and Deactivation_Date; Create a variable for Derived Closing FY based on the Derived Closing Date Determine the Final Closing FY based on the following: if the project status (PROJECT_STAT) is not “Closed” (that is, status is “Active”, “Dropped”, “Legacy”, or “Pipeline”, then make the Final Closing FY equal to the Evaluation FY (Eval FY); if not (that is, if the project status (PROJECT_STAT) is “Closed”, then: if Derived Closing FY is blank (has no value), then make Final Closing FY equal to the Evaluation FY. If Derived Closing FY has a value, then compare that with the Eval FY as follows: If Derived Closing FY is after Eval FY, then make Final Closing FY equal to Eval FY. If Derived Closing FY is the same as or earlier than Eval FY, then make Final Closing FY equal to Derived Closing FY. References World Bank. 2014. Guidelines for Reviewing World Bank Implementation Completion and Results Reports: A Manual for Evaluators. Independent Evaluation Group. Washington, DC: World Bank. http://ieg. worldbankgroup.org/sites/default/files/Data/ICRR_EvaluatorManualAugust2014.PDF ———. 2016. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Wash- ington, DC: World Bank. 1 For this analysis, the cohort of World Bank lending projects approved in FY06-08 was selected because most projects in approved in this time frame would be expected to have been closed and to have completed self-evaluation as of September 2017 when the data was pulled. This cohort also avoided projects intended to respond to the financial crisis of 2008–09. Independent Evaluation Group | World Bank Group 143 Table E.1. Very Large Projects Closed in FY14–16 (Net Commitment $500 Million or More) 144 Very large projects closed in FY14–16 with IEG ratings World Bank Lending perfor- instru- Global Project revised Rating Outcome mance M&E quality Net commitment Portfolio Project ID Project name ment practice Country closing date type rating rating rating amount P115067 MX Support to IPF SPL Mexico 31-Dec-13 ES S MS HIGH 2,753,759,398 Oportunidades Project P130048 Program for Economic DPF MFM Indonesia 31-Dec-15 ES S MS SUBSTANTIAL 2,000,000,000 Resilience, Investment and Social Assistance in Indonesia P101324 BR-Second Minas Gerais IPF Governance Brazil 31-Oct-14 ES MS MS MODEST 1,434,442,627 Developmentt Partnership P130051 Development Policy DPF MFM Romania 27-Oct-14 ES MS MS SUBSTANTIAL 1,333,300,000 Operation—DDO P130459 Development Policy Loan DPF MFM Poland 30-Jun-14 ES S S MODEST 1,307,800,000 2 P116226 Support to the Social IPF HNP Mexico 31-Dec-13 ES MS MS MODEST 1,250,000,000 Protection System in Health P101653 IN: Power System IPF Energy & India 31-Jul-14 ES S S SUBSTANTIAL 976,864,011 Development Project IV Extractives P115183 AR Basic Protection IPF SPL Argentina 30-Jun-16 ES MS S SUBSTANTIAL 929,989,613 Project P096858 Fourth Export Finance IPF F&M Turkey 31-Dec-14 ES S MS SUBSTANTIAL 899,998,490 Intermediation Loan P113220 ET Productive Safety Nets IPF SPL Ethiopia 30-Jun-15 ES S MS SUBSTANTIAL 850,000,000 (APL III) P127787 Competitiveness and DPF MFM Turkey 30-Jun-14 ES MS MS MODEST 800,000,000 Savings Development Policy Loan (CSDPL) P114107 CN-Wenchuan Earthquake IPF GP SURR China 31-May-16 ES MS MS MODEST 710,000,000 Recovery Project Appendix E. Additional Information on the World Bank World Bank Lending perfor- instru- Global Project revised Rating Outcome mance M&E quality Net commitment Project ID Project name ment practice Country closing date type rating rating rating amount P101590 Second Programmatic DPF MFM Peru 16-Jan-15 PAR MS MS SUBSTANTIAL 700,000,000 Fiscal Mgmt and Competitiveness Table E.1, continued. Development Policy Loan P126351 Bahia Inclusion and DPF MFM Brazil 31-Dec-14 ES MU MU MODEST 700,000,000 Economic Development DPL P094488 HIGHWAY 2 IPF Transport & Azerbaijan 30-Jun-15 ES MS MS MODEST 674,956,064 ICT P106038 BR Sao Paulo Trains and IPF Transport & Brazil 30-Apr-15 ES MS S MODEST 662,910,000 Signaling ICT P145605 CO Enhancing Fiscal DPF Poverty & Colombia 31-Dec-14 ES MS MS MODEST 600,000,000 Capacity DPL Equity P132768 Pernambuco Equity and DPF SPL Brazil 30-Sep-14 ES S S SUBSTANTIAL 550,000,000 Inclusive Growth DPL P078359 SEISMIC RISK IPF GP SURR Turkey 31-Dec-15 ES S S SUBSTANTIAL 544,105,655 MITIGATION P122810 FOURTH NATIONAL IPF GP SURR Indonesia 30-Jun-14 ES MS MS MODEST 531,190,000 PROGRAM FOR COMMUNITY EMPOWERMENT IN RURAL AREA (PNPM IV) P082144 PH-Social Welfare and IPF SPL Philippines 31-Dec-15 ES HS S HIGH 505,000,000 Development Reform P125943 Disaster Risk Management DPF GP SURR Philippines 31-Oct-14 ES S S SUBSTANTIAL 500,000,000 Development Loan with a CAT DDO P118308 Second Turkey Access IPF F&M Turkey 31-Mar-15 ES S S SUBSTANTIAL 500,000,000 Independent Evaluation Group | World Bank Group to Finance for Small and Medium Enterprise 145 World Bank 146 Lending perfor- instru- Global Project revised Rating Outcome mance M&E quality Net commitment Project ID Project name ment practice Country closing date type rating rating rating amount P147695 Enhancing Public DPF Transport & Brazil 31-Jan-16 ES U MU MODEST 500,000,000 Management for Service ICT Delivery in Rio de Janeiro Table E.1, continued. Source: IEG World Bank project ratings data; World Bank Business Intelligence data. Note: DPF = development policy financing; F&M = Finance & Markets Global Practice; GP SURR = Social, Urban, Rural and Resilience Global Practice; HNP = Health, Nutrition, and Population Global Practice; ICRR = Implementation Completion and Results Report Review; IPF = investment policy financing; M&E = monitoring and evaluation; MFM = Macroeconomics and Fiscal Management Global Practice; SPL = Social Protection & Labor Global Practice; HS = highly satisfactory rating; S = satisfactory rating; MS = moderately satisfactory rating; MU = moderately unsatisfactory rating; U = unsatisfactory rating; ES = ICRR rating type; PAR = project performance assessment report rating type. Table E.2. Very Large Projects Closed in FY11–13 Net Commitment $500 Million or More World Bank Lending Perfor- Net Commit- Instru- Global Revised Rating Outcome mance M&E Quality ment Project ID Project Name ment Practice Country Closing Date Type Rating Rating Rating Amount P115608 Mexico Framework for Green DPF Energy & Mexico 31-Dec-10 ES MS MS SUBSTANTIAL 1,503,750,000 Growth Development Policy Extractives Results and Performance of the World Bank Group 2017 | Appendix E Loan P118070 MX Economic Policies in DPF MFM Mexico 31-Dec-10 ES S S SUBSTANTIAL 1,503,750,000 Response to the Crisis DPL P114991 Hungary—Financial Sector and DPF F&M Hungary 31-Dec-10 ES N/A MU N/A 1,413,210,000 Macro Stability Loan (DPL) P102547 India: Elementary Education IPF Education India 30-Sep-12 ES MS MS SUBSTANTIAL 1,350,000,000 (Sarva Shiksha Abhiyan II) P117666 Third Development Policy Loan DPF MFM Poland 31-Dec-10 ES S S HIGH 1,331,300,000 P112495 RESTORING EQUITABLE DPF MFM Turkey 01-Nov-10 ES MS MS NEGLIGIBLE 1,300,000,000 GROWTH AND EMPLOYMENT PROGRAMMATIC DEVEL P095205 First Programmatic DPF Environment Brazil 31-Dec-10 PAR U U NEGLIGIBLE 1,300,000,000 Development Policy Loan for and Natural Sustainable E Resources World Bank Lending Perfor- Net Commit- Instru- Global Revised Rating Outcome mance M&E Quality ment Project ID Project Name ment Practice Country Closing Date Type Rating Rating Rating Amount P111017 Bangladesh Padma IPF Transport & Bangladesh 29-Jun-12 ES N/A S N/A 1,200,000,000 Multipurpose Bridge Project ICT Table E.2, continued. P115426 Energy Efficiency Development DPF Energy & Poland 31-Mar-12 ES MS MS SUBSTANTIAL 1,114,500,000 Policy Loan Extractives P106767 RIO GRANDE DO SUL FISCAL DPF Governance Brazil 31-Dec-10 ES MS MS SUBSTANTIAL 1,100,000,000 SUSTAINABILITY P107661 BOS Knowledge Improvement IPF Education Indonesia 31-Dec-12 ES MS MS MODEST 1,100,000,000 for Transparency and Accountability P111665 Rio de Janeiro Municipality DPF MFM Brazil 30-Jun-13 ES S S SUBSTANTIAL 1,045,000,000 Fiscal Consolidation for Efficienc P112258 Private Housing Finance IPF F&M Mexico 31-Dec-12 ES MS MS MODEST 1,005,471,532 Markets Strengthening Project P119856 Kazakhstan Development Policy DPF MFM Kazakhstan 07-Jan-11 ES MS S MODEST 1,000,000,000 Loan P114154 Public Sector Development DPF MFM Thailand 30-Jun-13 ES MS MS MODEST 1,000,000,000 Policy P127433 First Development Policy Loan DPF MFM Poland 30-Jun-13 ES S S MODEST 991,400,000 P103022 Ethiopia Protection of Basic IPF SPL Ethiopia 07-Jan-13 ES MS MS SUBSTANTIAL 957,830,061 Services Program Phase II Project P115052 Third National Program for IPF GP SURR Indonesia 31-Dec-12 ES MS MS MODEST 785,000,000 Community Empowerment in Rural Areas P082466 CO Integrated Mass Transit IPF Transport & Colombia 31-Dec-12 ES MS MS MODEST 752,648,485 Independent Evaluation Group | World Bank Group Systems ICT P112264 Strengthening the Business DPF F&M Mexico 31-Jan-12 ES S S MODEST 751,879,700 Environment for Enhanced Economic 147 World 148 Bank Lending Perfor- Net Commit- Instru- Global Revised Rating Outcome mance M&E Quality ment Project ID Project Name ment Practice Country Closing Date Type Rating Rating Rating Amount P099110 PK: Pakistan Earthquake ERC IPF GP SURR Pakistan 31-May-11 ES MS MS SUBSTANTIAL 700,000,000 Table E.2, continued. P112262 Upper Secondary Education DPF Education Mexico 30-Jun-11 ES MS MS SUBSTANTIAL 700,000,000 (MUSE) Development Policy Loan P117651 Second Environmental DPF Energy & Turkey 31-Dec-10 PAR S MS MODEST 700,000,000 Sustainability and Energy Sector Extractives Development P123073 Second Restoring EquiGrowth DPF MFM Turkey 31-Dec-11 ES MS MS NEGLIGIBLE 700,000,000 and Employment Programmatic Development Policy Loan P082822 Access to Finance for SMEs IPF F&M Turkey 30-Apr-12 ES S MS SUBSTANTIAL 696,910,000 P101211 Support for the Second Phase IPF SPL Colombia 31-Dec-11 ES MS MS SUBSTANTIAL 636,500,000 of the Expansion of the Program P077856 Lucknow-Muzaffarpur National IPF Transport & India 30-Jun-12 ES U U NEGLIGIBLE 620,000,000 Highway Project ICT Results and Performance of the World Bank Group 2017 | Appendix E P117874 Indonesia Seventh Development DPF MFM Indonesia 31-Mar-11 ES MS MS MODEST 600,000,000 Policy Loan P121651 Environmental Sustainability and DPF Energy & Turkey 30-Jun-13 PAR S MS MODEST 600,000,000 Energy Sector (ESES) DPL3 Extractives P122222 Development Policy Loan 3 DPF MFM Romania 31-Dec-12 ES S HS SUBSTANTIAL 560,600,000 (DPL 3) P105029 Third Programmatic Business DPF F&M Colombia 30-Jun-11 ES MS MS MODEST 550,000,000 Product and Efficiency Development Policy Loan P082977 Second Poverty Alleviation Fund IPF F&M Pakistan 30-Jun-11 ES MS MS MODEST 546,653,183 Project P050668 Mumbai Urban Transport IPF Transport & India 15-Jun-11 PAR MS MS SUBSTANTIAL 532,122,009 Project ICT World Bank Lending Perfor- Net Commit- Instru- Global Revised Rating Outcome mance M&E Quality ment Project ID Project Name ment Practice Country Closing Date Type Rating Rating Rating Amount P105002 ID National Program for IPF GP SURR Indonesia 31-Dec-11 ES MS MS MODEST 531,190,000 Community Empowerment in Table E.2, continued. Rural Areas P086518 SME Financing & Development IPF F&M India 30-Jun-13 ES S S SUBSTANTIAL 520,000,000 P110849 Mexico—Climate Change DPF Environment Mexico 30-May-11 ES MS MS NEGLIGIBLE 501,250,000 Development Policy Loan and Natural Resources P106708 CO DPL on Promoting an DPF SPL Colombia 31-May-11 ES MS MS MODEST 500,000,000 Inclusive, Equitable and Efficient Social Protection System P106753 Expanding Opportunities, DPF Poverty & Brazil 01-Jun-11 ES S MS SUBSTANTIAL 500,000,000 Enhancing Equity in the State Equity of Pernambuco Development Policy Loan P115173 Public Finance and DPF MFM Costa Rica 02-Jun-11 ES S S SUBSTANTIAL 500,000,000 Competitiveness Development Policy Loan with Deferred Drawdown Option P117723 VN-Public Investment Reform 1 DPF Governance Vietnam 03-Jun-11 PAR U U NEGLIGIBLE 500,000,000 P120470 Third Financial Sector DPF F&M Egypt, Arab 04-Jun-11 ES S S SUBSTANTIAL 500,000,000 Development Policy Loan Republic of P126094 Tunisia Governance and DPF MFM Tunisia 05-Jun-11 ES MU MS MODEST 500,000,000 Opportunity DPL S ource: IEG World Bank project ratings data; World Bank Business Intelligence data. Note: DPF = development policy financing; F&M = Finance & Markets Global Practice; GP SURR = Social, Urban, Rural and Resilience Global Practice; HNP = Health, Nutrition, and Population Global Practice; ICRR = Implementation Completion and Results Report Review; IPF = investment policy financing; M&E = monitoring and evaluation; MFM = Macroeconomics and Fiscal Management Global Practice; SPL = Social Protection & Labor Global Practice; HS = highly satisfactory rating; S = satisfactory rating; MS = moderately satisfactory rating; MU = moderately unsatisfactory rating; U = unsatisfactory rating; ES = ICRR rating type; PAR = Project Performance Assessment Report rating type. Independent Evaluation Group | World Bank Group 149 Figure E.1. World Bank Projects: Outcome Ratings over Time, Year by Year a. World Bank lending projects with outcome rated moderately satisfactory or above, percentage of projects b. World Bank lending projects with outcome rated moderately satisfactory or above, percentage of volume Source: World Bank Business Intelligence and IEG World Bank project ratings data. Note: FY = project closing fiscal year; N = number of projects; V = total volume of projects ($, billions). 150 Results and Performance of the World Bank Group 2017 | Appendix E Figure E.2. World Bank Projects: Performance Ratings over Time, Year by Year a. World Bank lending projects with World Bank performance rated moderately satisfactory or above, percentage of projects b. World Bank lending projects with World Bank performance rated moderately satisfactory or above, percentage of volume ($, billions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = project closing fiscal year; N = number of projects; V = total volume of projects ($, billions). Independent Evaluation Group | World Bank Group 151 Figure E.3. World Bank Project Monitoring and Evaluation Ratings over Time, Year by Year a. World Bank lending projects with World Bank performance rated moderately satisfactory or above, percentage of projects b. World Bank lending projects with World Bank performance rated moderately satisfactory or above, percentage of volume ($, billions) Source: World Bank Business Intelligence and IEG World Bank project ratings data. Note: FY = project closing fiscal year; N = number of projects; V = total volume of projects ($, billions). 152 Results and Performance of the World Bank Group 2017 | Appendix E Bank Project Monitoring and Evaluation Ratings over Time, Figure E.4 World Rolling Three-Year Average a. World Bank lending projects with M&E rated moderately satisfactory or above, percentage of projects b. World Bank lending projects with World Bank performance rated moderately satisfactory or above, percentage of volume ($, billions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = project closing fiscal year. Independent Evaluation Group | World Bank Group 153 Figure E.6. Country Strategy World Bank Group Performance Ratings Vary over Time Percentage distribution of World Bank Group performance ratings in CLR Reviews, with percentage of good+ indicated above each period Source: IEG completion and learning review ratings data. CLR = completion and learning review; good+ = good or superior; FCV = fragility, conflict, and violence; N = number of CLRs. Note: Note on World Bank Group Performance rating scale: The scale was converted from a six-point scale to a four-point scale in December 2013 (FY14) when World Bank Group and IEG issued a shared methodology “Assessing Country Partnership Frameworks: A Shared Approach to reconcile differences between teams’ self-evaluation and IEG’s rating. Prior ratings were converted as following: satisfactory = superior, moderately satisfactory = good, moderately unsatisfactory= fair, unsatisfactory = poor. There were no CLRs with World Bank Group Peformance rated as “highly satisfactory” or “highly unsatisfactory” in the entire database. Blue box indicates periods containing earlier CLR Reviews with World Bank Group Performance ratings on the old 6-point scale as well as later CLR Reviews with World Bank Group Performance ratings on the current 4-point scale. FY15 was the first year when all CLR Reviews used the current 4-point scale for the World Bank performance rating. 154 Results and Performance of the World Bank Group 2017 | Appendix E Figure E.7. Country Strategy Development Outcome Ratings Vary across Regions Percentage distrbution of development outcome ratings in CLR Reviews, with percentage of MS+ indicated on the right Source: Independent Evaluation Group Completion and Learning Review ratings data. Note: CLR = completion and learning review; MS+ = moderately satisfactory or above; N = number of CLRs. Independent Evaluation Group | World Bank Group 155 Strategy World Bank Group Performance Ratings Vary Figure E.8. Country across Regions Percentage distrbution of World Bank Group performance ratings in CLR Reviews, with percentage of good+ indicated on the right Source: Independent Evaluation Group Completion and Learning Review ratings data. Note: CLR = completion and learning review; N = number of CLRs. 156 Results and Performance of the World Bank Group 2017 | Appendix E Bank Project Outcome Ratings over Time, Year by Year Figure E.9. World versus Three-Year Rolling a. World Bank lending projects with outcome rated moderately satisfactory or above, percentage of projects b. World Bank lending projects with outcome rated moderately satisfactory or above, percentage of volume ($, billions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = project closing fiscal year. Independent Evaluation Group | World Bank Group 157 Figure E.10. World Bank Project World Bank Performance Ratings over Time, Year by Year versus 3-Year Rolling a. World Bank lending projects with World Bank performance rated moderately satisfactory or above, percentage of projects b. World Bank lending projects with World Bank performance rated moderately satisfactory or above, percentage of volume ($, billions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = project closing fiscal year. 158 Results and Performance of the World Bank Group 2017 | Appendix E Figure E.11. World Bank Outcome and World Bank Performance Ratings, by Region a. Outcome ratings, by number of projects b. Outcome ratings, by volume c. World Bank performance ratings, by number of projects c. World Bank performance ratings, by volume Note: World Bank-wide includes projects not tagged to a Region. Source: World Bank Business Intelligence and IEG World Bank project ratings data. Independent Evaluation Group | World Bank Group 159 Figure E.12. World Bank Project Outcome Ratings, by Global Practice 160 a. By number of projects a. By volume Results and Performance of the World Bank Group 2017 | Appendix E Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Figure E.13. World Bank Project Performance Ratings, by Global Practice a. By number of projects a. By volume Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Independent Evaluation Group | World Bank Group 161 Appendix F. Analysis of Distribution of World Bank Financing Volume Across Projects and Countries An analysis of the distribution of volume across projects and countries found a notably uneven distribution of the portfolio; much of the volume is concentrated in a few projects and in a few countries.1 Among World Bank financing projects closed during FY11–16 with completion reports validated by the Independent Evaluation Group (IEG) and volume data, the first two percent of projects accounted for 21 percent of project volume, and the first four percent of projects accounted for 35 percent of project volume (see figure F.1) Among the 1,719 projects in the group, 441 projects (26 percent) of projects had volume of $100 million or more, but these accounted for 79 percent of total volume. The other 1,278 projects (under $100 million volume) accounted for 74 percent of the projects but only 21 percent of total volume. Figure F.1. A Few Very Large Projects Account for a Large Share of Volume Number of Volume,   Project Size Projects $ million   >= 800 million 28 34,752 >= 500 million and <   800 million 38 22,663 >= 100 million and <   500 million 375 73,440   < 100 million 1278 35,713   TOTAL 1719 166,568 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: Based on 1,719 projects closed during FY11–16 with completion reports validated by the Independent Evaluation Group and volume (net commitment) data available, with a total $166,568 million volume (net commitment). When projects are grouped by country, nine countries accounted for half of project volume. 2 Eight of these nine countries have appeared consistently in International Bank for Reconstruction and Development financial statements during the past five years as the top countries to which the World Bank has credit exposure (table F.1). 162 Table F.1. Eight Countries Were Consistently on List of Highest Country Exposures in Credit Risk Section of IBRD Financial Statements for the Past Five Years Credit Exposure by IBRD Year End ($, billions) Country 6/30/2013 6/30/2014 6/30/2015 6/30/2016 6/30/2017 Brazil 11.6 13.9 15.4 16.1 16.1 Indonesia 12.4 13.6 14.3 14.2 15.4 Mexico 14.9 14.9 14.7 14.7 14.8 China 12.9 12.8 13 13.1 13.3 India 11.9 12 12.5 12.5 12.6 Turkey 12.9 13.6 12.1 10.9 10.9 Colombia 7.8 7.7 8.5 9.7 10.2 Poland 6.7 7.9 7.1 8 8.1 Egypt, Arab Rep. (a) (a) (b) (b) 7.6 Argentina (a) (a) (b) 5.7 6 Peru (a) (a) (b) 5.7 (b) Ukraine (a) (a) 4.6 (b) (b) Source: International Bank for Reconstruction and Development statements as of each date noted above. Note: Data are from the IBRD financial statement for the end of the fiscal year indicated. IBRD = International Bank for Reconstruction and Development. a. not among the top 8 countries listed as highest exposure in IBRD financial statements. b. not among the top 10 countries listed as highest exposure in IBRD financial statements. As of the 6/30/2017 IBRD Financial Statement, the 10 countries with highest exposure accounted for 62 percent of exposure. When projects in the nine large-volume countries were excluded, patterns of volume and performance in the other 121 countries represented in the Results and Performance of the World Bank Group 2017 (RAP17) sample became observable. For example, the analysis of project volume per country and the number of people living in poverty in the country found that project volume was associated with the number of people in poverty in the country, this association was stronger among the 121 small-volume countries (figure F.2). Among development policy financing (DPF) projects only, and among the 121 small-volume countries only, the average World Bank performance ratings across projects in the country showed an association with higher volume in a country as well as with volume per person in poverty in a country. Also among DPFs only, average World Bank performance ratings showed an association with the total net commitment per person in poverty in the country. No correlation was found among investment project financing. Also, for some of the large-volume countries, the average World Bank performance rating is below the World BankWorld Bank-wide average, for both investment project fi ancing and DPFs. Independent Evaluation Group | World Bank Group 163 Figure F.2. Net commitment of World Bank Lending Projects in a Country Shows Correlation with Number of People in Poverty in the Country Number of people in poverty in the country and total net commitment of World Bank lending projects a. All countries (N = 130) b. Countries with small volume (N = 121) R-squared = 0.293573 R-squared: 0.412273 P value < 0.0001 P value: < 0.0001 c. Countries with large volume (N = 9) Source: World Bank Business Intelligence, and Independent Evaluation Group World Bank project ratings. Note: The total net commitment of World Bank lending projects in a country shows correlation with number of people in poverty in the country, with stronger correlation in small-volume countries; relationship may not be linear. P values below 0.05 are significant at the 95 percent level. 164 Results and Performance of the World Bank Group 2017 | Appendix F Figure F.3. DPFs Show Some Association between World Bank Performance Rating And Total Net Commitment per Person in Poverty in the Country IEG World Bank performance rating (average across projects in the country) as a function of total net commitment per person in poverty in the country a. IPFs, countries with small volume (N = 121) b. DPFs, countries with small volume (N = 121) R-squared: 0.0919828 No correlation P value: 0.0056095 c. IPFs, countries with large volume (N = 9) d. DPFs, countries with large volume (N = 9) Source: World Bank Business Intelligence and IEG World Bank Project Rating. Note: P values below 0.05 are significant at the 95 percent level. DPF = development policy financing; IEG = Independent Evaluation Group; IPF = investment project fi ancing. Independent Evaluation Group | World Bank Group 165 References Dollar, David, and Victoria Levin. 2005. “Sowing and Reaping: Institutional Quality and Project Outcomes in Developing Countries.” Policy Research Working Paper 3524, World Bank, Washington, DC. 1 The analysis grouped the projects in the RAP17 cohort (World Bank lending projects closed in FY11–16 with Independent Evaluation Group ratings as of October 2, 2017) by country, and looked at patterns in the size of total project volume (net commitment) in countries, whether countries with more people living in poverty had higher project volume, and whether higher project volume was associated with better project performance ratings. 2 The nine countries that accounted for half of total project volume were Brazil (9 percent of total volume), India (8 percent), Mexico (8 percent), Indonesia (6 percent), China (5 percent), Turkey (5 percent), Vietnam (4 percent), Colombia (3 percent), and Poland (3 percent). 166 Results and Performance of the World Bank Group 2017 | Appendix F Appendix G. Risk and Independent Evaluation Group Outcome Ratings Part 1: Initial Risk Ratings To analyze the role of ex ante risk ratings as they relate to project performance ratings, the Results and Performance of the World Bank Group 2017 (RAP17) team conducted a new analysis based on the data set prepared for RAP13, which focused on risk in World Bank Group operations. RAP13 analyzed the relationship between risk and results of World Bank investment lending projects, using risk ratings data gathered from a review of a sample of project appraisal documents (PADs) from 200 investment policy financing projects closed during fiscal years (FY)06–12. That analysis found that (1) there was not a statistically significant correlation between risk ratings and IEG project outcome ratings and (2) projects in fragility, conflict, and violence (FCV) countries had higher risk ratings than projects in International Bank for Reconstruction and Development countries but had similar outcome ratings. RAP13 also noted that that 77 percent of PADs reviewed recorded original risks (77 percent), with the remaining PADs addressing residual risks (that isthat is, risk after mitigation measures were put into place) or unclassified risks (see appendix F ). The analysis for RAP17 reanalyzed the risk ratings data collected for the RAP13 with additional variables on country characteristics such as Country Policy and Institutional Assessment (CPIA) score; FCV status and project characteristics including problem fl gs, preparation cost, supervision cost, and borrower performance rating. The new analysis found that the risk rating correlates significantly with several country characteristics and project characteristics (table G.1). Correlation Coefficients, Investment Project Financing Table G.1. Pairwise Projects Closed FY06–12 CPIA FCV Risk Rat- Score Status Problem Super. Borrower ing (FY13) (FY13) Project Prep. Cost Cost Perf. CPIA Score −0.2793*** (FY13) FCV Status 0.1701** 0.5155*** (FY13) Problem Project 0.1701** 0.2886*** 0.1488** Preparation Cost 0.1385** 0.0569 −0.1253* 0.0648 Supervision Cost 0.1996*** −0.1350* 0.0506 0.2674 0.4730*** Borrower −0.1261* 0.2575*** −0.0283 −0.3461 −0.0081 −0.1228* Performance Outcome Ratings −0.1037 0.1535** −0.0020 −0.2506 0.0216 −0.0454 0.8080*** Source: World Bank Project Appraisal Documents and Independent Evaluation Group World Bank project ratings data. Note: * for p<0.1, ** for p<0.05, *** for p<0.01. CPIA = Country Policy and Institutional Assessment; FCV = fragility, conflict, and violence; Perf. = performance; Prep. = preparation; Super = supervision. This finding suggests that risk ratings may refl ect the capacity and situation of borrowing countries (as measured by CPIA score and FCV status), and predict future borrower performance rating. Meanwhile, higher risk is taken into consideration via higher project preparation and supervision costs. 167 The RAP17 analysis did not find a significant direct relationship between risk and outcome ratings. Instead, projects with higher risks are more likely to be tagged as a “problem project” during implementation, and projects tagged as a problem project tend to receive a lower outcome rating (table G.2). Table G.2. Risk Rating, Problem Project, Supervision Cost, and Outcome Ratings Risk Rating in Problem Project Outcome Rated MS+ PAD (percent) Problem Project (percent) Low/Moderate 57 No 81 Substantial/high 76 Yes 55 T-test significanc p <0.01 T-test significanc p <0.01 Avg. Supervision Cost Outcome Rating Avg. Supervision Cost Risk rating in PAD ($, thousands) ($, thousands) Low/Moderate 712 MS+ 758 Substantial/High 858 MU− 818 T-test significanc p <0.01 T-test significanc a Source: World Bank project appraisal documents and Independent Evaluation Group World Bank project ratings data. Note: MS+ = moderately succesfful or above; PAD = project appraisal document. a. cost does not correlate with outcome rating directly but was found in Results and Performance of the World Bank Group 2015 as a statistically significant factor in the regression models explaining driving forces behind outcome ratings. The RAP17 analysis also tested a few regression models to examine whether risk ratings explain project outcome ratings. These models included variables on borrower capacity (as measured by CPIA and FCV status) and possible risk mitigation efforts from the project team (using proxies such as the cost of preparation and supervision and problem project tagging). Outcome ratings are assigned on a 6-point scale. The analysis found that risk ratings are not statistically significant in any of the three models. Models of Project Outcome Ratings, Investment Table G.3. Linear Regression Project Financing Projects Closed FY06–12 Model 1 Model 2 Model 3 Variable Coeff. Signif. Coeff. Signif. Coeff. Signif. Risk rating in PAD −0.11 0.26 −0.11 0.32 −0.10 0.34 Preparation cost (log) 0.12 0.21 0.13 0.21 Supervision cost (log) 0.05 0.73 0.07 0.69 Ever listed as a problem project −0.51 0.00 −0.49 0.00 CPIA score in FY13 0.31 0.06 0.20 0.22 FCV status in FY13 0.19 0.46 0.19 0.46 N of observations 196 190 186 Adjusted R2 0.07 0.02 0.07 Source: World Bank project appraisal documents and Independent Evaluation Group World Bank project ratings data. Note: Coeff. = coef ficient; PAD = project appraisal document; signif. = significance. 168 Results and Performance of the World Bank Group 2017 | Appendix G Overall, these findings are consistent with the analysis for RAP13. The relationship between ex ante risk and project outcome ratings does not yield statistical significance in either correlation or regression models. Instead, this analysis suggests that the risk rating reflects borrowing countries’ capacities or the environment in which the projects would be implemented. However, most of the risk ratings in this data set (77 percent) fell into the initial risk category. Given that initial risk ratings do not indicate the extent to which the risk is likely to materialize, a further analysis of residual risk (the risk that remains after mitigation strategies have been put in place) was formulated to see if residual risk might have a more direct linkage to project outcome ratings. Part 2: Residual Risk Ratings To examine the relationship between residual risk and project outcome ratings, the RAP17 team reviewed sections related to risk in PADs of 152 projects that were approved in FY09–10 and have IEG outcome ratings. (This period was selected in part because approvals during this period included many instances where project risk ratings focused on residual risk. Later projects tended not to focus on residual risk, likely because the Operational Risk Assessment Framework launched by the of fice of Operational Policy and Country Services in FY11, required rating risk before mitigation.)1,2 Of the 152 projects in this review, only one overlapped with the set reviewed for RAP 2013. The RAP17 team built a data set of these projects approved in FY09–10 and rated by IEG, consisting of risk rating in PADs, Implementation Completion and Results Report Review ratings data, project preparation and supervision costs, whether the project was ever tagged as a “problem project,” and country CPIA and FCV status data. Seventy-six percent of the risk ratings in this data set focused on residual risk. Similar to the findings in Part 1 (based on the RAP13 data, which included projects that mostly assessed initial risk), the risk ratings for this new set of projects – most of which assessed residual risk – were highly correlated with country conditions, borrower performance rating, and supervision cost. Unlike the findings based mostly on initial risk, in this analysis the linkage with preparation cost no longer held, and there was a statistically significant negative correlation between residual risk and outcome ratings (table G.4). This finding supports the hypothesis that residual risk (that is, the risk remaining after mitigation has taken place) is more directly related to project outcome ratings than initial risk. Independent Evaluation Group | World Bank Group 169 Table G.4. Pairwise Correlation Coefficients, Investment Financing Projects Approved FY09–10 CPIA Score FCV Status Problem Super. Borr. Risk Rating (FY13) (FY13) Project Prep. Cost Cost Perf. CPIA score −0.4567*** (FY13) FCV status 0.3767*** −0.7147*** (FY13) Problem 0.1510* −0.1511* 0.0829 Project Prep. Cost 0.0241 0.2963*** −0.2609*** 0.1132 Super.Cost 0.1773** 0.0415 −0.0846 0.1154 0.6437*** Borrower −0.1957*** 0.0276 0.0398 −0.3878*** −0.2088*** −0.2822*** Perf. Outcome −0.1584** 0.1062 −0.0459 −0.4414*** −0.1000 −0.2039** 0.8368*** Rating Source: World Bank project appraisal documents and Independent Evaluation Group World Bank project ratings data. Note: * for p<0.1, ** for p<0.05, *** for p<0.01. Borr. = borrower; CPIA =Country Policy and Institutional Assessment; Perf. = performance; Prep. = preparation; Super. = supervision. 1 Noted in “Framework for Management of Risk in Operations” issued by the of fice of Operational Policy and Country Services on August 21, 2014, page 1. https://hubs.worldbank.org/docs/ImageBank/Pages/DocProfile. aspx?nodeid=26800423 2 Noted in “Investment Project Financing Operational Risk Assessment Framework Guidance Note” issued by OPSPQ on April 10 2013, page 5. http://siteresources.worldbank.org/PROJECTS/Resources/40940-1365611011935/Guidance_ Note_Risk_Assesment.pdf 170 Results and Performance of the World Bank Group 2017 | Appendix G Appendix H. Factors of Performance of World Bank Projects: Structured Review of Research Methodology The structured literature review evolved through a dynamic process of inquiry and discussion, beginning with an initial review of the development project outcome literature. This review included studies recommended by experts, references from the 2016 publication of the Results and Performance report, and studies identified through searches of academic journal databases. These searches consisted of English-language queries on combinations of terms including “project outcome,” “World Bank,” “International development,” and “success factors” through online journal databases including Google Scholar, the World Bank–International Monetary Fund Library Network, and JSTOR through July 2017. Studies selected for inclusion on the initial list discussed factors associated with the success or failure of international development interventions at a wide range of levels, including Country Assistance Strategies, program-level, project-level, process-level, and broader issues of aid effectiveness. Additional sources were identified by searching work that cited or was cited by the included literature, resulting in a list of 132 studies relevant to explaining development outcomes. During this scoping phase, the team drafted short summaries of each study, and pulled relevant excerpts from text and tables. After review and discussion of the initial list, refining criteria were introduced to narrow the research question. The Cochrane Handbook for Systematic Reviews of Interventions recommends defining a specific research question formulated around a target population, with specific intervention types, outcomes of interest, and a plan for drawing comparisons across variation in research methods (Higgins and Green 2008, section 5.1–5.4). Applying these criteria for specificity to the review, the original question of “what factors explain the outcomes of development interventions?” was narrowed to “what do empirical studies identify as explanatory factors of World Bank project outcomes?” To answer this more specific question, the structured literature review narrowed its criteria for inclusion to empirical studies of factors explaining the development effectiveness of International Bank for Reconstruction and Development (IBRD) and International Development Association (IDA) projects. Development effectiveness is understood in terms of project-level outcome measures, and while many authors measure this in terms of project outcome ratings validated by the Independent Evaluation Group (IEG), its measure is not limited to IEG project outcome ratings. The empirical studies selected are (i) research papers and evaluation reports published by the World Bank (including IEG) or (ii) academic studies published in peer-reviewed journals. To arrive at this narrow formulation of the research question and the inclusion criteria, the review team applied the guidelines from the Cochrane Handbook, identifying the target population, intervention type, scope, and process for drawing comparisons. In this case, the intervention type and target populations were identical, and remained consistent with the original question: development interventions. The scope was narrowed to the project level to avoid conflating factors that explain project-level outcomes with those that explain broader development outcomes such as economic 171 growth, literacy rates, or governance quality. Because these broader outcomes are themselves correlates of project-level outcomes in some analyses, excluding them as dependent variables avoids potential endogeneity problems from comparing findings at the project level with macrolevel analyses. The delivering organization was restricted to the World Bank (IBRD and IDA), allowing for consistent comparison of organizational factors and processes specific to the institution and improving relevance for designing future World Bank operations. Table H.1. Selection of Sources for Structured Literature Review Included Excluded • Empirical studies • Reviews of existing studies • Peer-reviewed academic studies • Conference presentations • World Bank research/evaluations • Analysis of a single World Bank project • Papers without empirical studies • Development projects not operated by World Bank (IBRD/IDA) Source: Independent Evaluation Group analysis. Applying these criteria narrowed the list from the 122 initially surfaced by the broader initial search to 32 studies and reports. Of the 90 sources excluded from the analysis, 52 were not specific to the World Bank (IBRD and IDA), 35 explained development outcomes not at the project level (for example, broader economic growth trends, or World Bank processes), 26 were not empirical studies (for example, reviews of existing studies, anecdotal evidence from a single individual, or purely theoretical analysis), and 6 otherwise fit the criteria but were not published by the World Bank or in a peer-reviewed journal. Note that some studies were excluded on the basis of more than one criteria. For the remaining studies and reports, the review team conducted a more careful reading of each resource to: ƒƒ Summarize the methodology and find ngs; ƒƒ List the factors of project outcome and their direction (that is, positive, negative, insignif- icant); ƒƒ List the methodology type; and ƒƒ Record how each paper measures project outcome (for example, outcome rating from IEG’s review of the project’s Implementation Completion and Results Report, economic rate of return, beneficiary satisfaction, and so on). After compiling the list of factors identified by each author, duplicates were removed and similar terms were consolidated (for example, democracy, regime type, and level of democracy were merged). Working from this list of terms, the factors were organized into a conceptual hierarchy, grouped into three primary categories of country-level factors, World Bank–level factors, and political 172 Results and Performance of the World Bank Group 2017 | Appendix H economy factors. Around this organizational framework, the 55 lowest-level terms were grouped into categories. For example, authors identifi d economic conditions including gross domestic product (GDP) per capita, GDP growth, and the foreign direct investment / GDP ratio as relevant. These were clustered under “Economic Conditions,” which fell under “Socioeconomic and Environmental Conditions” as country-level factors. Table H.2. Categories of Project Outcome Factors Country-Level Factors Political Economy Factors World Bank–Level Factors The elements of a country’s The incentives and interactions of Characteristics or elements of economic and socio-political borrowers and donors to select and preparation and supervision that are environment that may influence execute successful projects. plausibly within the control of the project outcomes. World Bank. Source: Independent Evaluation Group analysis. The hierarchical categorization process was both top-down and bottom up, following from discussions about how the literature described factors and how other publications categorized similar terms. For example, when constructing the list of terms to fall under the “economic management & policies” branch of the governance category, the team referred to categories already established by the World Bank’s Country Policy and Institutional Assessment (CPIA) index and the World Government Indicators, adapting the categories based on discussion of the factors identified by the literature. After creating this initial hierarchy, several rounds of discussion refined and re-organized it to better match considered judgments about the terms and their relationships to each other within the hierarchy. Following agreement on the list of terms and their conceptual hierarchy, the review team returned to the literature for another careful reading of each resource, coding by factors identified. Each factor was coded individually across all studies, as either a one or a zero. Studies that included a factor in their analysis were coded as a “one” on that factor, regardless of its direction or statistical significance. Depending on the sample, methodology, or statistical model there was considerable disagreement between authors on the effects of some factors. For example, some authors found civil liberties protections to correlate with project outcomes, while others found the opposite, and still others found no correlation at all. As part of the coding process, the reviewers added additional detail on each factor, and how the study characterizes its relationship with World Bank project outcomes. This detail can range from a simple account of its direction and significance, for those included in regression tables. However, for factors that receive additional discussion in the text, the coding table includes details on the author’s analysis of the causal mechanism or other linkages (theoretical or observed) between the factor and outcome. While the review team made careful efforts to thoroughly catalog the relevant literature on World Bank project outcomes, it is possible that reports or studies that met our criteria were not included. However, due to the review methodology, the likelihood that a relevant source was included is Independent Evaluation Group | World Bank Group 173 proportional to the number of times that it was cited by another piece of relevant literature. Therefore, while this systematic review may not be comprehensive in cataloging every piece of literature that attempts to explain the success or failure of World Bank projects, the review team believes that it is representative of the prevailing academic literature on the topic. Findings The literature on World Bank projects describes the factors associated with success or failure on three levels: at the country level, through factors of political economy, and at the World Bank level. At the country level, authors identify success factors ranging from economic conditions, to the policy environment, and institutional quality. The political economy of projects also plays a role in outcomes. Ensuring that the project is aligned with borrowers’ political interests and building government commitment are found to be success factors. At the World Bank level, the literature identifies a wide range of relevant factors across the project cycle, from project selection, to design, selection, and implementation. This appendix outlines the broad patterns from the literature, then narrows in on the effects of specific factors. At the factor-level it will discuss the consensus finding on how and why each affects project outcomes, the extent of that consensus, and disagreements within the literature. Additional detail and analysis can be found on the IEG website. One of the most consistent patterns observed through this review is that more effort, both in terms of preparation and supervision, is dedicated to more dif ficult projects—but that additional effort proves insuf ficient to outweigh additional dif ficulty. For both preparation and supervision, additional time and cost are consistently found to be negatively associated with project outcomes. However, when measured in terms of quality or when accounting for complexity, the relationship becomes positive. In terms of actionable findings for improving project outcomes, there is consensus within the literature that several World Bank–level factors contribute to successful projects. Incorporating lessons learned and analytical work, defining a clear results chain, and collaborating with local stakeholders (particularly in the case of community-driven development) are consistently associated with better project outcomes. Furthermore, while assigning a top-tier task team leader to each project is not a realistic option, task team leader track record (the average outcome rating of their past projects) is one of the most frequently-identified positive correlates of project outcome. Meanwhile, at the country level, there is greater consensus on the effects of good economic policy or institutional quality than on economic or social conditions. Distortionary economic policy, whether through subsidies, trade barriers, or artificially inflated exchange rates is consistently identified as negative factor for project outcomes. Alternatively, aside from economic growth, country conditions like GDP per capita, CPIA rating, human capital, and dependence on foreign aid or natural resources tend to have minimal impact on project outcomes. 174 Results and Performance of the World Bank Group 2017 | Appendix H Taken together, these trends within the literature point toward some degree of agency for the World Bank and its clients in determining project outcomes. Projects that are designed and implemented well tend to succeed. While country conditions beyond the control of the World Bank or the borrowing country do matter, studies are less confident regarding their importance, with many finding that their effects are negligible. Factors Associated with World Bank Project Figure H.1. Country-Level Performance, Based on Structured Literature Review Note: Factors in bold are cited by at least three sources; factors followed by a (+) icon are positively associated with World Bank project outcomes, factors followed by a (-) icon are negatively associated with project outcomes, and factors followed by a (=) icon have a neutral or disputed association with project outcomes. At the country level, the factors most consistently identified as meaningful to project outcomes include economic growth, economic distortions, and the presence of early warning fl gs raised by unsatisfactory performance in implementation status reports. On the other hand, many country-level economic indicators have either a minimal or disputed relationship with project outcomes. Population size, human capital levels, ongoing conflict, and aid or natural resource dependence are found to be statistically insignificant in many econometric analyses of project success. Similarly, the lack of consensus around the roles of a country’s GDP per capita, CPIA rating, and its level of economic vulnerability (typically measured in terms of debt ratio), suggest that those factors have either limited or inconsistent relationships with project outcomes. The findings on GDP per capita are evenly split between studies demonstrating a statistically significant association and those finding no significant association (Laajaj and Guillaumont 2006; Limodio 2011; Kilby 2015; Moll, Geli, and Saavedra 2015). Within those finding statistical significance, there was another even split among those finding it positive (Pohl and Mihaljek 1992; Chauvet, Collier, and Duponchel 2010) and negative (Dollar and Levin 2005; Girod and Tobin 2016). Authors speculate that there are competing effects of income level: on one hand, projects in middle-income countries may be more complex as basic needs are already satisfied. On the other hand, low-income countries Independent Evaluation Group | World Bank Group 175 may have weaker implementation capacity. From the variation in findings, it is unclear whether the benefi s of simpler projects outweigh the effects of lower implementation capacity. While GDP per capita has either mixed or negligible effects, GDP growth is consistently positively associated with project outcome ratings (Chauvet, Collier, and Duponchel 2010; Denizer, Kaufmann, and Kraay 2011; Limodio 2011; Smets, Knack, and Molenaers 2013; Blum 2014; Ralston 2014; Presbitero 2016). Authors further identify the capital/labor ratio (Isham and Kaufmann 1999) and global commodity price increases (Pohl and Mihaljek 1992) as positively varying with projects’ economic rate of return (ERR). Economic vulnerability (as measured by the debt-to-GNI ratio) is negatively associated (Kilby 2015), particularly in cases of politically-motivated aid (for example, United Nations Security Council (UNSC) vote buying: Dreher, Klasen, Vreeland, and Werker 2013). Beyond country-level economic conditions, the findings around human development indicators, fragile state status, and environmental conditions tend to cluster around neutral or negative associations with World Bank project outcomes. Neither population size nor human capital (typically measured by the average years of education of the working age population) correlate significantly with project outcomes (Deininger, Squire, and Basu 1998; Isham and Kaufmann 1999; Kilby 2015; Girod and Tobin 2016; World Bank 2016c; Presbitero 2016), and while Girod and Tobin (2016) show the infant mortality rate varying positively with project outcomes, the relationship does not hold with life expectancy (Blum 2014). Likewise, Dollar and Levin (2005) find that tropical location (as proposed by Sachs as a cause of economic underdevelopment) has no statistically significant relationship with project outcome ratings. Findings on the role of political institutions are split evenly between positive and statistically insignificant effects. Four studies find that a country’s CPIA rating is positively correlated with project outcomes, suggesting that with better economic management, structural policies, social inclusion, and public sector management and institutions are better equipped to implement World Bank projects (Geli, Kraay, and Nobakht 2014; Ralston 2014; World Bank 2016c; Presbitero 2016). However, four other studies find that CPIA ratings are unrelated to project outcomes (World Bank 2011; Smets, Knack, and Molenaers 2013; Blum 2014; Moll, Geli, and Saavedra 2015). However, these latter studies tend to focus on development policy lending (Smets et al. 2013; Moll et al. 2015) or other sector-specific projects (social safety nets in the case of World Bank 2011, or public sector projects for Blum 2014). It seems possible that CPIA ratings are simply less relevant for these project types than for investment project financing in general. Similar to the split between positive and statistically insignificant findings for CPIA rating, the literature is split on the correlation between other institutional quality indexes and project outcome. Laajaj and Guillaumont (2006) find no significant association between the International Country Risk Guide index of institutional quality and project outcomes, while Buntaine and Parks (2013) find that the government effectiveness, regulatory quality, and rule of law components of the World Governance Indicators vary positively with project outcomes (though control of corruption and International Country Risk Guide’s bureaucratic quality are not statistically significant). 176 Results and Performance of the World Bank Group 2017 | Appendix H When narrowing analysis of institutional quality to specific indicators rather than aggregate indexes, studies continue to find mixed results regarding the effects of institutional quality. While Dollar and Levin (2005) and the World Bank (2016b) both find that rule of law and civil liberties protections are associated with better project outcomes, the case for positive influences of political rights protections, democracy, accountability, and transparency finds less support. Buntaine and Parks (2013) find no support for a relationship between civil liberties protections and project outcomes, with similar findings for political rights protections by Denizer, Kaufmann, and Kraay (2011) and Dollar and Levin (2005). Findings are likewise split on the effects of democracy, with Blum (2014) demonstrating a positive association (in the case of public sector projects) and Dreher, Klasen, Vreeland, and Werker (2013) find ng no statistical signifi ance. The studies that include analysis of economic policy conclude that poor policy is associated with worse project outcomes. Pohl and Mihaljek (1992) find that economic management quality (as captured by the Agarwala price distortion index) is slightly positively correlated with the project’s ERR. Economic openness (measured by Summers & Heston Openness and Investment data set) is positively correlated with ERR and outcome rating (Deininger, Squire, and Basu 1998). Furthermore, market-skeptical economic ideologies on the part of the borrowing government are positively associated with development policy financing (DPF) project outcome rating and compliance (Smets, Knack, and Molenaers 2013). The authors explain the effect through increased preparation effort and stronger project design on the part of the World Bank, to convince the skeptical borrower of the merits of the proposed reforms. Distortionar y fiscal, monetary, and trade policy is consistently found to be associated with worse project outcomes. Price distortions (via subsidies and/or price controls) are correlated with lower economic rates of return (Pohl and Mihaljek 1992; Isham and Kaufmann 1999), lower project sustainability (Kumar, Narain, and Rubbani 2015), and failure to meet the needs of beneficiaries (Isham and Kaufmann 1999). Public investment project spending has declining marginal returns. Implementing many projects at once undermines outcomes, as local-level implementation resources (for example, government of ficials, firms) are spread thin (Presbitero 2016). Finally, while a public sector surplus does not affect the outcome rating or return on projects, it does correlate with improved sustainability (Deininger, Squire, and Basu 1998). An overvalued exchange rate (along with other economic distortions including subsidies) reduces a project’s ERR (Isham and Kaufmann 1999). Similarly, inflation varies negatively with project outcome and sustainability (Deininger, Squire, and Basu 1998) with unexpected inflation reducing the project’s ERR (Pohl and Mihaljek 1992). Trade barriers (including tariffs and import restrictions), as part of broader economic distortions, are also associated with lower economic rates of return (Pohl and Mihaljek 1992). Independent Evaluation Group | World Bank Group 177 Economy Factors Associated with World Bank Project Figure H.2. Political Performance, Based on Structured Literature Review Note: Factors in bold are cited by at least three sources; factors followed by a (+) icon are positively associated with World Bank project outcomes, factors followed by a (-) icon are negatively associated with project outcomes, and factors followed by a (=) icon have a neutral or disputed association with project outcomes. Studies reviewed find that the political economy factor most consistently associated with project outcomes is the borrowing government’s commitment to the project objectives. While factors like the borrower’s role in international institutions and strategic relationships with major donor states are widely found to affect the approval time of projects delivered in that state, they are unrelated to project outcomes. The relationship between the level of government commitment and project outcomes is observed across several measures of the concept. Project outcomes suffer when the recipient government fails to take concrete actions following process-oriented steps, such as task forces, studies, action plans (Moll, Geli, and Saavedra 2015). Alternatively, when the borrowing government has clear incentives to deliver on the project objectives, particularly through pressure by domestic constituencies, those objectives are more likely to be accomplished (Buntaine and Parks 2013; World Bank 2016b; Blum 2014). Finally, policy-based loans tend to be less successful when the recipient’s policy-making environment is not conducive to change, as in the case of political actors who have an interest in maintaining the status quo (Kumar, Narain, and Rubbani 2015; Girod and Tobin 2016). While literature beyond the scope of this review demonstrates the impact of natural resource dependence on political institutions (for example, Ross 2001), there is little evidence that natural resource dependence plays a role at the project level. Once accounting for CPIA rating, the effect is statistically insignificant (Blum 2014; Presbitero 2016). However, when shifting the focus to development policy lending or conditional loans, countries with greater income from natural resource 178 Results and Performance of the World Bank Group 2017 | Appendix H rents are less likely to comply with loan terms (Girod and Tobin 2016). The authors argue that natural resource revenue insulates governments from potential consequences of noncompliance. The findings on the effect of aid dependence on project outcomes are mixed. Under some statistical models, the effect of aid dependence is negative and significant, but loses significance when using a different model (Dollar and Levin 2005) or looking only at projects completed since 1995 (Presbitero 2016). Blum (2014) finds that aid dependence has a positive association with project success, but only on public sector management projects. Laajaj and Guillaumont (2006), on the other hand, find that aid dependence has a negative effect; they theorize that limited absorptive capacity leads to diminishing marginal returns to aid. Several authors test the theory that project outcomes suffer from close political or economic relationships between borrowers and major donors. Their theoretical claim is that development projects are pushed through the approval process on a political basis, rather than on the merits of recipient need or project design—that is, a quid pro quo relationship in which loans are exchanged for political favors such as votes in the UNSC. However, their findings indicate that, while the nature of the donor-recipient relationship (for example, as allies, trade partners, membership in international bodies such as the UNSC) may affect approval times and a project’s quality at entry rating, they do not have statistically significant effects on the project outcomes (Dreher and others 2013; Smets, Knack, and Molenaers 2013; Kilby 2015; Girod and Tobin 2016). Bank–Level Factors Associated with World Bank Project Figure H.3. World Performance, Based on Structured Literature Review Note: Factors in bold are cited by at least three sources; factors followed by a (+) icon are positively associated with World Bank project outcomes, factors followed by a (-) icon are negatively associated with project outcomes, and factors followed by a (=) icon have a neutral or disputed association with project outcomes. Among the factors plausibly within the World Bank’s control (World Bank–level factors), the literature has a near consensus view that project selectivity, the quality of project design, the incorporation of lessons learned, results chain quality, collaboration with local stakeholders, task team leader track record, and supervision quality are associated with better project outcomes. While there is Independent Evaluation Group | World Bank Group 179 less consensus on what factors contribute to worse project outcomes, some authors point to the increased time and cost of supervision and preparation, along with task team leader turnover rates. Both project duration and delays between approval and disbursement are identified as negatively associated with project outcomes, though agreement is stronger regarding the role of project duration (Chauvet, Collier, and Duponchel 2010; Denizer, Kaufmann, and Kraay 2011; Ralston 2014; Ika 2015; Girod and Tobin 2016). The effect of the disbursement gap is negligible, even in those analyses that find it to be statistically significant (Denizer, Kaufmann, and Kraay 2011; Blum 2014). Although the consensus view of the literature is that a project’s initial budget size is unrelated to its outcome, several studies make the case that the final cost is positively correlated with outcome (World Bank 2015a; World Bank 2016c. Furthermore, even when this higher cost is due to overruns, Pohl and Mihaljek (1992) find that the project does not suffer a penalty to its rate of return. Likewise, Pohl and Mihaljek (1992) and IEG find that project extension is associated with stronger performance (World Bank 2015a). The positive influence of task team leader quality is one of the project outcome literature’s most consistent findings. task team leaders with higher average outcome ratings on past projects are more likely to deliver successful projects in the present (Denizer, Kaufmann, and Kraay 2011; Geli, Kraay, and Nobakht 2014; Ralston 2014; Moll, Geli, and Saavedra 2015; World Bank 2016c). While it may be unsurprising that higher rates of task team leader turnover are a negative correlate of project outcome (Denizer, Kaufmann, and Kraay 2011; World Bank 2016c), Ralston (2014) finds that projects led by task team leaders with more experience are less successful in fragile or conflict-affected situations. She accounts for this counterintuitive finding by theorizing that task team leaders with more experience are assigned to more dif ficult projects. Two studies identify project team performance as a factor of project outcome, focusing on the team’s resources and performance, as well as their access to relevant training. Teams that can devote more resources (time and budget) to problem-solving are more likely to achieve better project outcomes (World Bank 2016c). Likewise, according to Ika (2015), World Bank staff identified their team’s training as a critical success factor for their projects. While the findings on the role of team capacity are less extensively validated than the task team leader-specific conclusions, the research does suggest that project outcomes are better when the project team has greater access to time, budget, and knowledge. Failure to incorporate lessons learned from World Bank work is associated with lower quality at entry ratings and lower project outcomes ratings (World Bank 2016c). Evidence from a Mexican financial inclusion project indicates that its success was due to the World Bank’s long-standing analytical work in that sector (Kumar, Narain, Rubbani 2015). Furthermore, conducting economic and sector work as part of a lending program will increase the program’s project-level outcome ratings by 12–20 percent (Deininger, Squire, and Basu 1998). 180 Results and Performance of the World Bank Group 2017 | Appendix H Unsurprisingly, projects with early warning flags receive lower outcome ratings than those without (Denizer, Kaufmann, and Kraay 2011; Ralston 2014). These flags are assigned to a project when it receives at three or more unsatisfactory outcome ratings in implementation status reports during the first half of implementation. However, a task team leader’s risk assessment of the project is more predictive of its outcome rating than standardized risk flags (Blum 2014). Collaboration across the World Bank and with local stakeholders is another element of successful projects. Within the World Bank, IEG found that the 1997 reorganization to the Matrix (sector and region) system undermined the incentives and ability of staff to collaborate across the institution (World Bank 2012). Conversely, World Bank staff identified internal coordination and attention from senior management as critical success factors for their projects (Ika 2015). In a similar vein, Kumar, Narain, and Rubbani (2015) found that dialogue between senior World Bank staff and country of ficials could help projects overcome political obstacles to success. Collaboration at the local level between the World Bank team and local stakeholders is associated with improved project outcomes. In rural water supply projects, including communities in the decision-making process was associated with greater satisfaction, use, equity in access, time savings, and perceptions of sustainability by beneficiaries (Prokopy 2005). Another study of community-driven development projects found that beneficiaries identified engagement, local ownership, and alignment of the project to community needs as critical success factors (Yalegama, Chileshe, and Ma 2016). Furthermore, engaging with local stakeholders can help the project team to understand the local context and adapt the project accordingly (Kumar, Narain, and Rubbani 2015). The findings around project preparation time and costs trend toward indicating that both are negatively associated with project outcome ratings and ERR, with most authors theorizing that more dif ficult projects receive additional time and attention in the preparation phase (Deininger, Squire, and Basu 1998; Denizer, Kaufmann, and Kraay 2011; Geli, Kraay, and Nobakht 2014; Ralston 2014;). On the other hand, measures of preparation effort, including the IEG rating of quality at entry and the quality of project design, indicate that higher preparation quality is associated with stronger project outcomes (Vawda, Moock, Gittinger, and Patrinos 2003; Chauvet, Collier, and Duponchel 2010; Ika 2015; IEG 2016e; Raimondo 2016). Similar to preparation effort, when measured purely in terms of cost (Denizer, Kaufmann, and Kraay 2011; Ralston 2014) or staff time (Deininger, Squire, and Basu 1998; Ralston 2014; IEG 2016e), supervision tends to be negatively correlated with project outcomes, both in terms of outcome ratings and economic rates of return. When measured in terms of supervision quality (Chauvet, Collier, and Duponchel 2010; Buntaine and Parks 2013; Smets, Knack, and Molenaers 2013; IEG 2016e; Raimondo 2016) or when accounting for dif ficulty (Kilby 2000) supervision is strongly and consistently associated with better outcome ratings. The literature consistently identifies selectivity as a factor of strong project outcomes. However, what constitutes selectivity varies across studies, with some defining it with regards to components or Independent Evaluation Group | World Bank Group 181 objectives (Isham and Kaufmann 1999; World Bank 2015b; Kumar, Narain, and Rubbani 2015) and others with regards to the project selection process (Deininger, Squire, and Basu 1998; Limodio 2011; Yalegama, Chileshe, and Ma 2016). Projects with greater complexity, as assessed on a subsector by subsector basis by OED (Isham and Kaufmann 1999) or case analysis (World Bank 2015b), have lower economic rates of return, or fail to achieve their intended results in the case of the Gambia Gateway Project and the Ghana Land Administration project (World Bank 2015b). In project selectivity, authors argue that selecting well-designed projects minimizes the possibility of failure at the preparation stage; if the project is implemented well, it is likely to succeed (Limodio 2011). Findings on whether IDA financing performs better than IBRD are somewhat mixed, with Buntain and Parks (2013) finding a positive relationship, Kilby (2015) finding a weak but positive relationship, and Moll and others (2015) finding a weak but negative relationship significant at the p < 0.10 level. Given that the findings on GDP per capita are also mixed, it may be dif ficult to disentangle the effects of income with those of the lending instrument. Several papers discuss factors specific to the outcome of DPF projects. Because DPF projects advance policy change and institutional reform, political buy-in from policy-makers – as observed through the share of a project’s activities with tangible outputs – is essential (Moll, Geli, and Saavedra 2015). Getting this buy-in is more likely in democratic countries with programmatic political parties, and those that are more dependent on foreign aid (Blum 2014). When political leadership cannot be held accountable, either by its citizens or its donors, reforms that undermine elite interests are unlikely. In fragile and conflict-affected situations, DPF projects with a focus in a single sector are less likely to achieve their intended outcomes (Ralston 2014). Finally, better quality of design can improve borrower compliance with the terms of DPF projects. Smets et al. (2012) find that development policy recommendations are more successful when the client government holds a left-leaning political ideology, and explain this finding by pointing to the additional effort in the preparation stage. When the project team knows that its client will treat market-liberalization policies with skepticism, they devote more time and resources to project design in an effort to convince them of the merits of their proposed reforms. References Blum, Jurgen Rene. 2014. “What Factors Predict How Public Sector Projects Perform?” Policy Research Working Paper 6798, World Bank, Washington, DC. Buntaine, Mark T., and Bradley C. Parks. 2013. “When Do Environmentally Focused Assistance Projects Achieve Their Objectives? Evidence from World Bank Post-Project Evaluations.” Global Environ- mental Politics 13 (2): 65–88. Chauvet, Lisa, Paul Collier, and Marguerite Duponchel. 2010. “What Explains Aid Project Success in Post-Conflict Situations?” Policy Research Working Paper 5418, World Bank, Washington, DC. 182 Results and Performance of the World Bank Group 2017 | Appendix H Deininger, Klaus, Lyn Squire, and Swati Basu. 1998. “Does Economic Analysis Improve the Quality of For- eign Assistance?” World Bank Economic Review 12 (3): 385–418. Denizer, Cevdet, Daniel Kaufmann, and Aart Kraay. 2011. “Good Countries or Good Projects? Macro and Micro Correlates of World Bank Project Performance.” Journal of Development Economics 105: 288–302. Dollar, David, and Victoria Levin. 2005. “Sowing and Reaping: Institutional Quality and Project Outcomes in Developing Countries.” Policy Research Working Paper 3524, World Bank, Washington, DC. Dreher, Axel, Stephan Klasen, James Raymond Vreeland, and Eric Werker. 2013. “The Costs of Favorit- ism: Is Politically Driven Aid Less Effective?” Economic Development and Cultural Change 62 (1): 157–191. Geli, Patricia, Aart Kraay, and Hoveida Nobakht. 2014. “Predicting World Bank Project Outcome Ratings.” Policy Research Working Paper 7001, World Bank, Washington, DC. Girod, Desha M., and Jennifer L. Tobin. 2016. “Take the Money and Run: The Determinants of Compliance with Aid Agreements.” International Organization 70 (1): 209–239. Higgins, Julian, and Sally Green. 2008. Cochrane Handbook for Systematic Reviews of Interventions. Lon- don: Cochrane Collaboration. Ika, Lavagnon A. 2015. “Opening the Black Box of Project Management: Does World Bank Project Super- vision Influence Project Impact?” International Journal of Project Management 33 (5): 1111–1123. Isham, Jonathan, and Daniel Kaufmann. 1999. “The Forgotten Rationale for Policy Reform: The Productivi- ty of Investment Projects.” Quarterly Journal of Economics 114 (1): 149–184. Kilby, Christopher. 2000. “Supervision and Performance: The Case of World Bank Projects.” Journal of Development Economics 62 (1): 233–259. Kilby, Christopher. 2015. “Assessing the Impact of World Bank Preparation on Project Outcomes.” Journal of Development Economics 115: 111–123. Kumar, Anjali, Sushma Narain, and Swizen Rubbani. 2015. “World Bank Lending for Financial Inclusion: Lessons from Reviews of Select Projects.” Independent Evaluation Group Working Paper Series No. 2015/1, World Bank, Washington, DC. Laajaj, Rachid, and Patrick Guillaumont. 2006. “When Instability Increases the Effectiveness of Aid Proj- ects.” Policy Research Working Paper 4034, World Bank, Washington, DC. Moll, Peter G., Patricia Geli, and Pablo Saavedra. 2015. “Correlates of Success in World Bank Development Policy Lending.” Policy Research Working Paper 7181, World Bank, Washington, DC. Pohl, Gerhard, and Dubravko Mihaljek. 1992. “Project Evaluation and Uncertainty in Practice: A Statistical Analysis of Rate-of-Return Divergences of 1,015 World Bank Projects.” World Bank Economic Review 6 (2): 255–277. Independent Evaluation Group | World Bank Group 183 Presbitero, Andrea F. 2016. “Too Much and Too Fast? Public Investment Scaling-Up and Absorptive Ca- pacity.” Journal of Development Economics 120: 17–31. Prokopy, Linda Stalker. 2005. “The Relationship between Participation and Project Outcomes: Evidence from Rural Water Supply Projects in India.” World Development 33 (11): 1801–1819. Raimondo, Estelle. 2016. “What Difference Does Good Monitoring and Evaluation Make to World Bank Project Performance?” Policy Research Working Paper 7726, World Bank, Washington, DC. Ralston, Laura. 2014. “Success in Dif ficult Environments: A Portfolio Analysis of Fragile and Conflict-Affect- ed States.” Policy Research Working Paper 7098, World Bank, Washington, DC. Smets, Lodewijk, Stephen Knack, and Nadia Molenaers. 2013. “Political Ideology, Quality at Entry and the Success of Economic Reform Programs.” Policy Research Working Paper 6130, World Bank, Washington, DC. Vawda, Ayesha Yaqub, Peter Moock, J. Price Gittinger, and Harry Anthony Patrinos. 2003. “Economic Analysis of World Bank Education Projects and Project Outcomes.” International Journal of Edu- cational Development 23 (6): 645–660. World Bank. 2011. Social Safety Nets: An Evaluation of World Bank Support, 2000–2010, World Bank, Washington, DC. ———. 2012. The Matrix System at Work: An Evaluation of the World Bank’s Organizational Effectiveness. Washington, DC: World Bank. ———. 2015a. Learning Note: Additional Financing for Transport and Information and Communication Technology (ICT). Washington, DC: World Bank. ———. 2015b. Results and Performance of the World Bank Group 2014. World Bank, Washington, DC. ———. 2016a. Lessons from Environmental Policy Lending. Washington, DC: World Bank. ———. 2016b. Lessons from Land Administration Projects: A Review of Project Performance Assessments. Independent Evaluation Group. Washington, DC: World Bank. ———. 2016c. Results and Performance of the World Bank Group 2015: An Independent Evaluation. Washington, DC: World Bank. ———. 2016d. World Bank Group Engagement in Small States: The Cases of the OECS, PICs, Cabo Verde, Djibouti, Mauritius, and the Seychelles. Washington, DC: World Bank. 184 Results and Performance of the World Bank Group 2017 | Appendix H Appendix I. Gender Incorporation in Project Development Objectives, Components, and Indicators In fiscal year (FY)16, the Independent Evaluation Group piloted a systematic documentation of gender dimensions in individual World Bank projects as part of the Implementation Completion and Results Report (ICR) review process.1 ICR reviewers examine the following: ƒƒ Whether gender is an explicit part of the project development objective (PDO) or one of the project components; and ƒƒ Whether the ICR reports sex-disaggregated or female- or male-specific indicators. Building on the pilot analysis in Results and Performance of the World Bank Group 2015 (RAP15) and RAP16, this RAP found that gender was an explicit part of the PDO or at least one component in 68 of 241 projects reviewed (28 percent) during FY14–16. Of the 68 projects explicitly incorporating gender, 50 (73 percent) reported some sex-disaggregated or male- or female-specific indicators at some level in the ICR, or that one of the indicators was the share or percentage of female beneficiaries. Gender was an explicit part of the PDO in 16 projects. Among these, 12 projects (75 percent) reported on gender indicators at the PDO level in the ICR (see for additional details). Beyond the World Bank-wide results reported in chapter 2, the analysis noted variation across Global Practice clusters in the proportion of projects in which gender appeared to be relevant, and the proportion that incorporated gender into a PDO or component. The FY14–16 analysis found that the Human Development cluster had the highest proportion of projects that integrated gender into a PDO or component (45 percent), followed by the Equitable Growth, Finance and Institutions cluster with 32 percent and the Sustainable Development cluster with 19 percent. The differences are statistically significant for Human Development and Sustainable Development (p<.01), but not Equitable Growth, Finance and Institutions. Regional variation was not statistically significant (figure I.1). Within the Human Development and Sustainable Development Global Practice clusters, the particular Global Practices varied in the degree to which they integrated gender. In most cases, variation across GPs was not statistically significant. However, Energy & Extractives (p<0.05), Environment and Natural Resources (p<0.05), and Transport & ICT (p<0.01) had significantly less explicit incorporation of gender than other GPs, while Health, Nutrition, and Population had significantly more (p<0.01; figure I.2). World Bank–wide, among the ICRs for 68 projects that explicitly included gender in the PDO or at least one component, three-fourths reported sex-disaggregated indicators, male- or female-specific indicators, or at least included the share or percentage of female beneficiaries. Figure I.3 provides details by Global Practice cluster and region. 185 Figure I.1. Projects that Incorporated Gender in PDOs or Components, by Global Practice Cluster and Region, FY14–16 Source: Independent Evaluation Group analysis. Note: N = number of projects; PDO = project development objective. 186 Results and Performance of the World Bank Group 2017 | Appendix I Figure I.2. Projects Incorporating Gender in PDOs or Components, within the Human Development and Sustainable Development Global Practices, FY14–16 Source: Independent Evaluation Group analysis. Note: ICT = information and communication technologies; N = number of projects; PDO = project development objective. Independent Evaluation Group | World Bank Group 187 Figure I.3. Presence of Gender-Relevant Indicators among Projects Incorporating Gender in the PDO or at Least One Component, FY14–16 Source: Independent Evaluation Group analysis. Note: The figure excludes Global Practices with zero or one projects that mentioned gender in the PDO or at least in one component. ICT = information and communication technologies; N = number of projects; PDO = project development objective. 188 Results and Performance of the World Bank Group 2017 | Appendix I 1 For the Results and Performance of the World Bank 2015, theIndependent Evaluation Group reviewed project appraisal documents from FY12–14 for integration of gender on a project-by-project basis, finding that 43 percent of projects (N = 231) included gender in either its actions or components. In FY16, Independent Evaluation Group piloted systematic documentation of gender-integration as part of the Implementation Completion and Results Report review process. Implementation Completion and Results Report reviewers examined whether gender is a relevant aspect of the project and whether the Implementation Completion and Results Report reports sex-disaggregated or female- or male-specific indicators. The Results and Performance of the World Bank 2016 analyzed this Implementation Completion and Results Report Review data, finding that gender was explicitly incorporated in 38 percent of projects in FY16 (N = 156). Independent Evaluation Group | World Bank Group 189 Appendix J. Additional Information on the International Finance Corporation Portfolio Results and Performance of International Finance Corporation Investment Projects Result and Performance of the World Bank Group 2017 (RAP17) updated the analysis of the development results of investment projects by the International Finance Corporation (IFC) and IFC’s performance by including findings from 69 projects evaluations completed in the calendar year (CY)16 Expanded Project Supervision Report (XPSR) Program. A total of 236) IFC investment projects, with completed evaluations as of 12 December 2017, were included in the 3-year review period, CY14 to CY16, with approval years spanning fiscal year (FY)07 to FY12 (during and after the global financial crisis) and evaluated in the CY14, CY15 and CY16 XPSR Programs. This appendix presents project results data based on 3-year moving averages. Annual performance is selectively mentioned to provide additional explanation about the trends underpinning the 3-year moving average results. This appendix also presents the updated statistical analysis of external and internal drivers of (and risks to) development outcome. The updated statistical analysis included the results and performance of 69 projects in the CY16 XPSR Program. The CY14–16 results show continued slippage in mostly successful or better (MS+) development outcome ratings, by number of projects or by volume (IFC net commitment amount) (figure J.1). The overall average mostly successful or better MS+ development outcome rating in CY14–16 stands at 50 percent (by number of projects) or 59 percent, if weighted by the amount of IFC net commitment (by volume). By contrast, the share of MS+ development outcome rating in CY11–13 were 60 percent by number of projects or 70 percent, by volume. The decline in successful outcome ratings in the CY14–16 review period reflected continued fall in the share of MS+ (or positive) development outcome ratings since its peak in CY08 (by number of projects). Of the 69 evaluated investment projects in CY16, 29 projects had positive development outcome ratings. Although IFC’s investment outcome and IFC’s role and contribution indicators received higher share of positive ratings in CY14–16, their ratings have also weakened compared with CY11–13 results, by number of projects and volume (figure J.1). Among the four subindicators of development outcome, only the ratings for environmental and social (E&S) effects showed improvement in CY14–16 due to the significant increase in the share of evaluated projects in CY16 with satisfactory or better ratings (figure J.2). Reasons for the improved E&S effects ratings were mentioned in chapter 2. But the falling positive ratings in project business success, economic sustainability raises questions about these projects’ financial viability, contribution to economic growth and in supporting private sector development. The share of satisfactory or better (S+) ratings in these three subindicators reached an all-time low in CY16, especially if weighted by IFC’s net commitment amount, reflecting the low ratings of several large-sized projects in business success, economic sustainability and private sector development. 190 Trend of IFC Investment Projects’ Results and Figure J.1. Long-Term Performance Ratings, by XPSR Program Years (percent) a. Share of positive ratings, by number of evaluated projects, three-year moving average b. Share of positive ratings, by IFC net commitment (in $, billions), three-year moving average Source: Independent Evaluation Group Evaluation Expanded Project Supervision Report Program Data (as of December 12, 2017. Note: Development Outcome of International Finance Corporation investment projects are rated along a six-point scale namely: highly successful, successful, mostly successful (all considered positive) and highly unsuccessful, unsuccessful, and mostly unsuccessful. Investment outcome and IFC’s role and contribution are rated along a four-point scale namely: excellent, satisfactory (all considered positive), partly unsatisfactory, and unsatisfactory. With the revision of XPSR Guidelines in 2015, IFC’s role and contribution indicator was replaced with the indicator, IFC’s additionality. CY = calendar year; IFC = International Finance Corporation; XPSR = Expanded Project Supervision Report. The weak development outcome ratings in CY14–16 compared with CY11–13 results is reflected across different project characteristics: projects in International Development Association (IDA) and non-IDA countries; in fragility, conflict, and violence (FCV) and non-FCV countries; by IFC industry groups; and by IFC regions. (figures J.3 and J.4) However, there are differences in outcome results within each characteristic. As example, evaluated projects in IDA countries have higher share of outcome success ratings than projects in non-IDA countries, by number of projects and especially by volume (figure J.3, panel a). Projects in non-FCV countries have slightly higher share of MS+ outcome ratings than the projects in FCV countries (figure J.3, panel b) both by number of projects and volume. Independent Evaluation Group | World Bank Group 191 However, the outcome success rating of projects in non-FCV countries continues to decline since CY11–13 whereas the outcome success rating of projects in FCV countries showed some modest improvement recently. or Better Ratings of the Four Development Outcome Figure J.2. Satisfactory Indicators Shows that Only the E&S Effects Ratings Improved in CY14–16 (percent) a. By number of projects b. By net commitment amount Source: Independent Evaluation Group Expanded Project Supervision Report Program data (as of December 12, 2017). Note: Development outcome has four elements or indicators: project business success; economic sustainability; environmental and social effects; and private sector development. Each indicator is assessed and rated along a 4-point scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory. 192 Results and Performance of the World Bank Group 2017 | Appendix J Trend of Mostly Successful or Better Development Figure J.3. Long-Term Outcome ratings of IFC Investment Projects in IDA-Eligible and Non-IDA Countries and in FCV and Non-FCV Countries (percent) Share of mostly successful or better development outcome ratings, IDA-eligible and Non-IDA countries, three- year moving average Share of mostly succesfful or better development outcome ratings, FCV and Non-FCV countries, three-year moving average Source: Independent Evaluation Group Expanded Project Supervision Report Program Data (as of December 12, 2017). Note: The Independent Evaluation Group used the World Bank Group country classification and the Harmonized List of Fragile Situations for FY17.International Development Association (IDA)-eligible countries refer to countries classified as IDA and Blend countries. Non-IDA countries consist of International Bank for Reconstruction and Development and high-income countries. FCV = fragile, conflict and violence. Results and performance by industry group, in CY14–16, shows some variation if measured by number of projects or if weighted by the amount of IFC net commitment (figure J.4). By number of projects, infrastructure and natural resources (INR) projects had the highest share of outcome success ratings among the industry groups in CY14–16 (62 percent), an improvement over the CY11–13 development outcome ratings (53 percent). Projects by the telecom, media, technology, and venture funds; group had the lowest share of positive outcome ratings in CY14–16 (38 percent) and remains at par with the CY11–13 outcome results (39 percent). However, projects in the manufacturing, agribusiness, and services (MAS) industry group exhibited the sharpest contraction in the share of outcome success ratings in CY14–16 (46 percent) compared with the with CY11–13 Independent Evaluation Group | World Bank Group 193 results (72 percent). By IFC commitment amount, INR projects had the largest share of outcome success ratings in CY14–16 (74 percent had MS+ outcome ratings). Projects by the Financial Institutions Group (FIG) had the lowest share of outcome success ratings in this review period, with only half of the evaluated projects rated mostly successful or better. Compared with CY11–13 results, MAS and FIG projects experienced sharpest decline in outcome success ratings, by 22 and 21 percentage points, respectively. By contrast, telecom, media, technology, and venture funds; group projects outcome success ratings improved the most in CY14–16 (57 percent) compared with CY11–13 ratings (45 percent). Trend of Mostly Successful or Better Development Figure J.4. Long-Term Outcome Ratings of IFC Investment Projects (percent). Share of mostly succesful or better development outcome ratings, three-year moving average a. By number of projects b. By volume Source: Independent Evaluation Group Expanded Project Supervision Report Program Data (as of December 12, 2017). Note: In CY16, there were only three evaluated CTT and one had a MS+ rating for development outcome, the amount of IFC investment was large that it distorts the annual outcome ratings by volume. CTT = telecom, media, technology, and venture funds; FIG = Financial Institutions Group; IFC = International Finance Corporation INR = infrastructure and natural resources; MAS = manufacturing, agribusiness, and services. By region, South Asia (including Afghanistan and Pakistan) had a higher share of evaluated projects with positive development outcome ratings in CY14–16, by number of projects and volume. The region’s share of positive outcome ratings also improved by 22 percentage points from 49 percent in CY11–13 to 71 percent in CY14–16, by number of projects. The weighted share of positive outcome ratings also increased from 67 percent in CY11–13 to 79 percent in CY14–16. By contrast, evaluated projects in the Middle East and North Africa region (excluding Afghanistan and Pakistan) had the lowest outcome success ratings in CY14–16 and had declined drastically compared with CY11–13 ratings. Appendix L presents detailed analysis of IFC results and performance in each region. 194 Results and Performance of the World Bank Group 2017 | Appendix J of Mostly Successful ot Better Development Outcome Figure J.5. Comparison Ratings of IFC Investment Projects, by Region (percent) a. Share by number of projects, three-year moving average b.Share by volume, three-year moving average Source: Independent Evaluation Group Expanded Project Supervision Report program data (as of December 12, 2017). Note: Note that the IFC regional data aggregation follows that institution’s country classifications as of CY17, which classifies Afghanistan and Pakistan as part of the Middle East and North Africa Region. Starting CY19, IFC will classify these two countries as part of the South Asia Region. AFR = Sub-Saharan Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; IFC = International Finance Corporation; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. Refer to appendix L for the detailed analysis of IFC results and performance in each region. The Independent Evaluation Group (IEG) updated the statistical analysis of the drivers (and risks to) of development outcome using ratings data from 569 (387 real sector and 182 financial intermediary) projects1 that were evaluated in the CY08 to CY16 XPSR Programs. The regression model assumed that the development outcome of an IFC investment project is a function of two sets of factors: external and internal. The sensitivity of development outcome ratings to each external and internal variables was calculated by adding the marginal effects along the 6-point scale of development outcome ratings, which are shown in columns 2 and 4, table J.1. The analysis applied logistic and ordered logistic regression approaches using the following variables: ƒƒ Development Outcome (dependent variable): based on a 2-point scale and on a 6-point scale. 2 ƒƒ Internal risk factors (independent variable):3 screening, appraisal, and structuring work quality; supervision and administration work quality; and IFC role and contribution/IFC additionality. ƒƒ External risk factors (independent variable): management quality risk; profit margin risk, Independent Evaluation Group | World Bank Group 195 for real sector projects4; corporate governance risk for financial intermediary projects;5 Institutional Investor Country Credit Rating (IICCR) rating;6 and the change in IICCR rating indexes between project approval and project evaluation. ƒƒ Project characteristics (independent variable): project size;7 advisory services -invest- ment project link; greenfield and brownfield project; projects of repeat clients and one-off clients; loan instrument only IFC investment; equity-only IFC investments; projects in IDA and non-IDA countries; projects in FCV and non-FCV countries.8 The regression results points to the stronger effects of improved IFC up-front work (screening, appraisal and structuring) and IFC’s role and contribution on development outcome in both real sector and financial intermediary projects.9 Both logistics and ordered logistics results presented in table J.1 underscores the more significant and robust association between development outcome ratings and these two internal factors than other external risk factors such as changes in country risk (measured by changes in IICCR index between approval and evaluation),10 management quality and improved profit margins. While changes in country risk and improvement in profit margins and in client’s management quality are also significantly associated with development outcome ratings, the sensitivity of development outcome ratings to incremental changes in these external factors is slightly less. A real sector project with positive screening, appraisal and structuring rating increases the likelihood (measured in log-odds) of having a positive development outcome rating by 2.39 points, or a 51 percentage points increase in the probability of getting a positive development outcome rating (table J.1, column 1). The ordered logistic regression results also underscore the sensitivity of development outcome ratings to IFC’s up-front work ratings—a one-point increase in screening, appraisal and structuring rating (using the 4-point scale) increases the likelihood of getting a higher development outcome rating by 1.41 points. Within these results, there are slight differences in the external and internal drivers that influence of development outcomes. Among real sector projects, development outcome ratings are most sensitive to external risks to profit margins, changes in country risk, client’s management quality and to a lesser extent, projects with repeat clients. For financial intermediary projects, change in country risk is the only external risk factor that has the likelihood (expressed in log-odd ratio) of affecting development outcome ratings. And for financial intermediary projects IFC’s role and contribution (an identified internal driver) ratings has a more sizable effect on overall development outcome ratings than IFC’s screening, appraisal, and structuring ratings. Among the other internal factors, the association between positive ratings of IFC’s supervision and administration performance with positive development outcome ratings (column 2, table J.1) remain statistically significant. However, development outcome ratings of real sector and financial intermediary projects are not sensitive to the incremental changes in IFC’s supervision and administration ratings.11 Project size (measured by IFC’s net commitment) have a positive effect on DO ratings but the association is less statistically significant and robust. In recent years, the development outcome of projects involving large amount of IFC net commitment had weakened and 196 Results and Performance of the World Bank Group 2017 | Appendix J the regression results reflect this trend. Projects with IFC equity investment have lower likelihood of achieving a successful development outcome rating compared with projects financed by a mix of IFC loans and equity.12 On the other hand, greenfield projects and projects’ location in IDA and FCV countries have no significant effect on development outcome ratings as indicated in table J.1 below. Analysis of External and Internal Risk Factors Affecting Table J.1. Regression Development Outcome Ratings in IFC Investment Projects. Dependent variable: development outcome Real Sector Projects Financial Intermediary Projects (2) (4) Development (3) Develop- Development (1) Development Outcome (6-point ment Outcome Outcome Outcome (2-point Scale) Ordered (2-point Scale) (6-point Scale) Explanatory Variables Scale) Logit Logistic Logit Ordered Logistic SAS (2 pt scale) 2.392***  1.472*** (0.340) (0.458)  SUP (2 pt scale) 1.154*** 0.656  (0.358) (0.494)  ROLE (2 pt scale) 1.436*** 1.594*** (0.448) (0.281) SAS (4 pt scale) 1.412*** 1.414*** (0.194) (0.281) SUP (4 pt scale) 0.286 0.194 (0.181) (0.325) ROLE (4 pt scale) 0.999*** 1.319*** (0.214) (0.270) Management quality −0.840** −1.036*** −0.908 −0.873* (0.416) (0.326) (0.601) (0.527) Profit margi −1.585*** −0.705*** (0.368) (0.250) Corporate governance −0.748 −0.034 (0.491) (0.434) Project size 0.161 0.089* 0.222 0.158 (0.116) (0.054) (0.195) (0.153) AS-IS link 0.928 0.022 −0.769 0.058 (0.842) (0.468) (0.565) (0.475) Loan only −0.308 −0.347 −1.055 −0.798 (0.371) (0.356) (0.931) (0.572) Equity only −1.147*** −1.067** −0.001 −0.059 (0.435) (0.432) (0.902) (0.591) IDA −0.391 −0.119 −0.305 0.083 (0.891) (0.521) (0.689) (0.538) Independent Evaluation Group | World Bank Group 197 Real Sector Projects Financial Intermediary Projects (2) (4) Development (3) Develop- Development (1) Development Outcome (6-point ment Outcome Outcome Outcome (2-point Scale) Ordered (2-point Scale) (6-point Scale) Explanatory Variables Scale) Logit Logistic Logit Ordered Logistic FCV 0.303 0.392 −0.406 0.001 (1.082) (0.700) (1.008) (0.766) Greenfield p ojects −0.485 −0.465* −0.373 −0.482   (0.349) (0.263) (0.455) (0.398) Repeat client 0.559* −0.145 −0.168 0.116 (0.322) (0.241) (0.627) (0.380) IICCR index (at −0.024 −0.004 0.003 0.002 approval) (0.015) (0.011) (0.017) (0.013) Change in IICCR 0.059*** 0.054*** 0.050* 0.057** indexes (at approval and at evaluation) (0.020) (0.015) (0.027) (0.023) Note: Number of real sector projects = 387 and number of financial intermediary projects = 182. Standard errors are in parenthesis. Real sector projects = projects with E&S Category A, B and C. Financial intermediary projects = projects with E&S Category FI, FI-1, FI-2 and FI-3. ***statistically significant at p<.01; **statistically significant at p<.05; *statistically significant at p<.1. SAS = screening, appraisal, and structuring. SUP = supervision and administration. AS-IS Link = Combined IFC advisory and investment support in a project. IDA = International Development Association. FCV = fragile, conflict and violence. IICCR = Institutional Investor Country Credit Rating. The share of satisfactory or better ratings in IFC’s screening, appraisal and structuring in CY14–16 have slightly declined, by number of projects, compared with CY11–13 (figure J.6). Adjusted ratings by the amount of IFC net commitment show improved ratings in CY14–16 (65 percent) compared with CY11–13 (58 percent) due to better ratings of IFC’s up-front work in large projects. However, the share of satisfactory or better ratings in IFC’s role and contribution (the other significant influencer of development outcome ratings) had weakened in CY14–16 compared with CY11–13, both by number of projects and volume, and have affected this reiew period’s development outcome results. As the statistical analysis in table J.1 show, an increase in IFC’s role and contribution rating increases the likelihood of getting a positive (by 1.4 points for real sector and 1.6 points for financial intermediary projects) or high (by 1 point for real sector and 1.4 points for financial intermediary projects) development outcome ratings. of Satisfactory or Better Ratings in IFC Work Quality Figure J.6. Comparison and IFC’s Role and Contribution Indicators (percent) 198 Results and Performance of the World Bank Group 2017 | Appendix J a. Share by number of projects, three-year moving average b. Share by IFC net commitment amount, three-year moving average Source: Independent Evaluation Group Expanded Project Supervision Report program data (as of December 12, 2017). Note: Screening, appraisal and structuring; supervision and administration; and IFC’s Role and Contribution are rated based on a 4-point scale: excellent, satisfactory (all positive), partly unsatisfactory and satisfactory. Screening, appraisal and structuring and supervision and administration are two subindicators of IFC’s work quality. XPSR = Expanded Project Supervision Report. IFC Advisory Projects Results and Performance RAP17 updated the analysis of the development results of advisory projects and IFC’s performance by including findings from 37 projects evaluated under the FY16 Project Completion Report Program. The analysis focused on the FY14–16 results and performance covering 150 IFC advisory projects with completed evaluations and IEG validations as of 12 December 2017. Implementation years of the 150 projects ranged from FY06 to FY15 (before and after the global financial crisis) and evaluated in the FY14 to FY16 Project Completion Report Programs. This part of the appendix J presents details on project results data based on 3-year moving averages; however, annual performance is selectively mentioned to provide additional explanation about the trends underlying the 3-year moving average results. Recent results show further decline among advisory projects with mostly successful or better (MS+) ratings for development effectiveness, whether measured by number of evaluated projects or weighted by IFC expenditures (volume). Forty-nine percent of projects evaluated in FY14–16 had MS+ ratings for development effectiveness, in contrast to the 63 percent share among the projects evaluated in FY11–13. By volume, the average MS+ rating for the FY14–16 fell by 14 percentage points in FY14–16 (52 percent) from FY11–13 (66 percent). The 3-year rolling average was adversely Independent Evaluation Group | World Bank Group 199 affected by results of the projects evaluated in the FY16 Project Completion Program with only 9 of the 35 projects evaluated had mostly successful or better ratings for development effectiveness. By far, these development effectiveness ratings have been the lowest since IEG started evaluating IFC Advisory Services in 2008. Trend of IFC Advisory Projects with Positive Ratings for Figure J.7. Long-Term Development Effectiveness and IFC’s Role and Contribution (percent) a. Share of positive ratings, by number of projects, three-year moving average b. Share of positive ratings, by IFC expenditure, three-year moving average Source: Independent Evaluation Group Project Completion Report program data (as of December 12, 2017). Note: Development Effectiveness is rated on a 6-point scale: highly successful, successful, mostly successful (all positive), mostly unsuccessful, unsuccessful and highly unsuccessful. IFC’s role and contribution is rated on a 4-point scale: excellent, satisfactory (all positive), partly unsatisfactory and unsatisfactory. IFC = International Finance Corporation. IFC’s role and contribution ratings have also weakened. And although an overwhelming majority of evaluated advisory projects had satisfactory or better ratings (74 percent, by number of projects and 71 percent, by IFC expenditure amount) in IFC’s role and contribution, positive ratings fell in FY14–16 compared with FY11–13 results. By volume, the share of positive ratings for IFC’s role and contribution declined by 14 percentage points, from 85 percent in FY11–13 to 71 percent in FY14–16. The percentage decline by number of projects is lesser (7 percentage point) in comparison, which indicate lower ratings for IFC’s role and contribution in large-sized advisory projects in FY14–16. Among the indicators of development effectiveness, output achievement continues to have the highest share of MS+ ratings, indicating the ability of IFC Advisory Services projects to deliver the intended products, services, or advice. However, outcome achievements and ef ficiency were the 200 Results and Performance of the World Bank Group 2017 | Appendix J lagging indicators and in FY14–16, the share of satisfactory or better (S+) ratings fell at an all-time low (fi ure J.7). Ratings Trend of Development Effectiveness and Its Figure J.8. Long-Term Indicators (percent) a. Share of satisfactory or better development effectiveness indicator ratings, by number of projects, three-year moving average b. Share of satisfactory or better development effectiveness indicator ratings, by IFC expenditure, three-year moving average Source: Independent Evaluation Group Expanded Project Supervision Report program data (as of December 12, 2017). Note: The indicators of development effectiveness are rated on a 4-point scale: excellent, satisfactory (all positive), partly unsatisfactory and unsatisfactory. A comparison of the different advisory project profiles (that is, by regions, IDA and non-IDA; and FCV and non-FCV)13 show a deterioration in the share of MS+ outcome ratings across the board in FY14–16. The following charts illustrate the trends across different project characteristics. Among the six regions, evaluated advisory projects in South Asia (including Afghanistan and Pakistan) had the largest share of positive development effectiveness ratings in FY14–16 (58 percent) by number of projects. At the opposite end, evaluated projects in the Africa Region had the smallest share of positive development effectiveness ratings, at 33 percent, in this review period. This poor result was particularly evident in the FY16 PCR program where no evaluated project in the Africa Region (out of 10) was rated positively for development effectiveness. Evaluated projects in Africa in FY14–16 also Independent Evaluation Group | World Bank Group 201 had the lowest share in positive ratings for IFC’s role and contribution (61 percent) compared with the other regions. By volume (measured by IFC expenditure amount), projects in the Middle East and North Africa region had lowest share of positive effectiveness ratings in FY14–16. Only a third of the evaluated projects in Middle East and North Africa region had been rated positively in development effectiveness in this review period, which also fell drastically compared with the FY11–13 ratings (92 percent to 30 percent in FY14–16). Appendix L presents detailed analysis of IFC advisory projects results and performance in each region. Trend of IFC Advisory Projects’ Mostly Successful or Figure J.9. Long-Term Better Development Effectiveness ratings, by IFC Region (percent) a. Share by number of projects, 3-year moving average b. Share by IFC expenditure, 3-year moving average Source: Independent Evaluation Group Project Completion Report Program Data (as of December 12, 2017). Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; IFC = International Finance Corporation; LAC = Latin America and the Caribbean; MENA = Middle East and North Africa; SAR = South Asia. Development effectiveness success ratings of advisory projects in IDA and non-IDA countries weakened further in FY14–16 compared with FY11–13 results, with ratings in IDA countries falling to an all-time low in this review period, by number of projects and by volume (figure J.10). IDA 202 Results and Performance of the World Bank Group 2017 | Appendix J projects’ effectiveness rating in the FY14–16 review period had been adversely affected by the poor development effectiveness results of the FY16 PCR Program, in which only two (located in South Asia) of the 17 evaluated projects in IDA countries were rated positively. Of the remaining 15 evaluated projects that had poor effectiveness ratings, 8 were in the Africa Region and except for one project, the rest had poor ratings for outcome achievement. In fact, all evaluated projects with mostly unsuccessful or worse ratings in development effectiveness had low ratings (partly unsatisfactory or unsatisfactory ratings in outcome achievement, a key indicator of advisory projects’ development effectiveness. Figure J.10. Long-Term Trend of Mostly Successful or Better Development Effectiveness Ratings, by Projects in IDA and Non-IDA Countries (percent) a. Share by number of projects, three-year moving average b. Share by IFC expenditures, three-year moving average Note: IDA = International Development Association; PCR = Program Completion Report. Development effectiveness results of evaluated projects in FCV and non-FCV countries also follow similar pattern as that of IDA and non-IDA countries. Figure J.11. below illustrates the declining trend in mostly successful or better ratings of evaluated projects in FCV and non-FCV countries, by number of projects and by IFC expenditure amount. Development effectiveness success ratings in this review period is at an all-time low, by number of projects and volume, in both FCV and non-FCV countries. Independent Evaluation Group | World Bank Group 203 A joint IEG and IFC working group is currently conducting an in-depth assessment of the reasons for the deteriorating results in IFC advisory projects. Figure J.11. Long-Term Trend of Mostly Successful or Better Development Effectiveness Ratings, by Projects in FCV and Non-FCV Countries (percent) a. Share by number of projects, three-year moving average b. Share by IFC expenditure, three-year moving average Source: Independent Evaluation Group Project Completion Report program data (as of December 12, 2017). Note: Classification of FCV and non-FCV countries used the World Bank Group’s Harmonized List of Fragile Situations 2017. FCV = fragility, conflict and violence. 1 Projects that have missing information in any of the explanatory variables were excluded. Of the 694 investment projects with completed evaluations in CY08 to CY16, 125 had missing information in at least one of the explanatory variables and were excluded from the regression analysis data. 2 The two-point scale consisted of: 0 = mostly unsuccessful or worse; 1= mostly successful or better. The six-point scale included: 1 = highly unsuccessful; 2 = unsuccessful; 3 = mostly unsuccessful; 4 = mostly successful; 5 = successful; and 6 = highly successful. 3 Internal risk factors refer to factors that are within IFC’s direct control. 204 Results and Performance of the World Bank Group 2017 | Appendix J 4 Real sector projects refer to projects with Environmental and Social Risk Codes A, B, and C in IFC’s Management Information System (MIS) database. 5 Financial intermediary projects refer to projects with Environmental and Social Risk Codes FI or FI-1, FI-2, and FI-3 in in IFC’s Management Information System (MIS) database. 6 IICCR rating is an index of external risk as measured by the Institutional Investor Country Credit Risk ratings (IICCR) at the time of project approval and project evaluation. 7 Project size is measured as the natural logarithm of IFC’s net commitment amount (in thousands of dollars) in the project. 8 Fragility, conflict, and violence (FCV) and non-FCV country classifications followed the World Bank Group’s FY17 Harmonized List of Fragile Situations. 9 Although the statistical analysis show strong association between the ratings of IFC screening, appraisal and structuring and of IFC’s role and contribution, the results should does not imply causality. 10 Ordered logistic regression assumes that the coef ficients that describe the relationship between, the lowest versus all higher categories of the response variable are the same as those that describe the relationship between the next lowest category and all higher categories, etc. 11 The effects of International Finance Corporation’s supervision and administration ratings on development outcome ratings are less robust as seen from the results of the ordered logistics method. 12 A project with only equity financing from IFC has lower likehood or log-odds of a having a successful development outcome rating compared to projects with IFC equity and loan financing, or have a 26 percentage point drop in the probability of getting a positive development outcome. 13 Results and performance by International Finance Corporation (IFC) advisory business lines or areas was excluded because of the on-going reclassification of the advisory services business lines, which had expanded from four business lines to nearly 12 business areas. Remapping of projects by IFC is on-going. Independent Evaluation Group | World Bank Group 205 Appendix K. Additional Findings: Management Action Record Tracking Data and Methodology What Is the Management Action Record, and Where Do Its Data Come From? Management Action Record (MAR) is an annual update process to follow up on the implementation of management’s action plans in response to sector, thematic and corporate evaluations performed by the Independent Evaluation Group (IEG). IEG’s recommendations, Management Action Plans and subsequent updates and reviews are stored in a centralized online repository that is accessible only by MAR update participants. The centralized system allows MAR participants access historical data and share updates through a single platform minimizing data loss and increasing collaboration. MAR also aims to create transparency about progress made by World Bank Group management in addressing IEG recommendations and Management Action Plans. Thus, the final MAR data is available on IEG’s public website and is accessible by external audiences. IEG has been tracking management’s implementation progress in response to its recommendations since the late 1990s for the World Bank, since 2003 for the Multilateral Investment Guarantee Agency (MIGA), and since 2004 for the International Finance Corporation (IFC). In the early years of implementation tracking, IEG rated progress of implementation against the outcomes outlined in its recommendations. A reform of the MAR process in 2011 set forward a proposal to eliminate ambiguity over the implementation of recommendations and reduce the inconsistencies in rating between IEG and management. The reform resulted in introducing Management Action Plans that had specific targets and indicators, and changed rating of progress from adoption of IEG’s recommendations to the implementation of action plans. The MAR tracks only those recommendations that are agreed or partially agreed on by management of the World Bank Group institutions. For each agreed recommendation, management provides an action plan, progress against which is then assessed by World Bank Group management and IEG independently and rated for four years, after which the recommendation is “retired” from the MAR review process. IEG provides comments on management’s action plans, but prerogative to finalize the action plans lies with management. This is the first year since the 2011 MAR reform when the entire data set of active recommendations had Management Action Plans, as opposed to the previous years where there were some evaluations that preceded the reform and did not have action plans associated with recommendations mixed with recommendations that had action plans. During the MAR updates, each recommendation may stay in the update cycle for four years to complete and exit the cycle or may be retired earlier than the fourth year. If a recommendation is retired earlier than the fourth year because either Management or IEG considers implementation to be complete, it remains in the set of active recommendations with the status of Inactive and is labeled as retired, but it only exits the cycle at the end of the fourth year. The calculations of the fourth year ratings are based only on the recommendations that stayed in the MAR update cycle until their fourth year. 206 The analysis of this year’s data in chapter 3 was based on the implementation of 146 recommendations drawn from 23 evaluations from FY13-16 and their action plans covering all three institutions.1 Around 66 percent of all recommendations targeted World Bank, 29 percent targeted IFC and 5 percent targeted MIGA. The analysis included both management and IEG’s ratings between FY13–16 and the review of relevant updates. The analysis shows that in the fourth year of tracking recommendations, both IEG and management rate implementation progress for over 90 percent of recommendations as substantial or above, but IEG is more likely to give higher implementation ratings later in the implementation cycle. Since the rating of substantial captures implementation progress in a range from 50-89 percent, in this year’s analysis IEG also focused on recommendations rated High and Complete to reflect implementation progress at 90 percent or above. This year’s analysis shows that only 52 percent of recommendations in their fourth year of implementation were rated as High or Complete by IEG, with 20 percent rated complete. In addition, analysis of recommendations rated both by IEG and management (excluding the recommendations unilaterally retired earlier) shows that management is less likely to rate its progress lower than high or complete in the final years of implementation, which is partially explained because implementation is rated against action plans instead of IEG’s recommendations. of Recommendations Rated Both by IEG and Management Figure K.1. Ratings in the Third and Fourth Year of Implementation Source: Data from the Independent Evaluation Group’s Management Action Record database. All recommendations were analyzed and coded into the categories with most frequently recurring themes. Some recommendations fell in more than one category and categories were not mutually exclusive. In general, over 65 percent of IEG’s recommendations in this year’s set fell in the “corporate and institutional improvement” category and over 40 percent fell in the “knowledge, measurement Independent Evaluation Group | World Bank Group 207 and impact” category. In the latter category, around 20 percent of recommendations were focused on improving internal World Bank Group capacity. Analysis of all active recommendations shows that management receive lower completion ratings for the recommendations in the “implementing programs or projects” category followed by the “knowledge, measurement, and impact” category (Figure K.2). Recommendations on “high-level policy or strategic directions” received high or complete ratings most frequently and the disconnect in ratings between IEG and Managmeent is the smallest in this category. Figure K.2. Recommendations Rated High and Complete, by Category (percent) Source: Data from the Independent Evaluation Group’s Management Action Record database. Methodology Used for Focus Group Discussions In this year’s MAR update cycle, the following studies were in their final year of implementation tracking and will no longer be tracked in the MAR system: Adapting to Climate Change: Assessing the World Bank Group Experience Phase III, Capturing Technology for Development—An Evaluation of the World Bank Group Activities in Information and Communication Technologies, The World Bank’s Support to Global Food Crisis, Biennial Report on Operational Effectiveness of IFC and MIGA (BROE), Knowledge-Based Country Programs: An Evaluation of the World Bank Group Experience, World Bank Group Support to Youth Employment Programs, Managing Forest Resources for Sustainable Development An Evaluation of World Bank Group Experience, Evaluation of the International Finance Corporation’s Global Trade Finance Program, and Improving Institutional Capability and Financial Viability to Sustain Transport An Evaluation of World Bank Group Support Since 2002. To understand how IEG’s recommendations may have influenced policy and operational work of the World Bank Group in the past 4–6 years, IEG conducted focus group discussions on evaluation exiting the MAR update cycle with management counterparts and IEG staff. The 208 Results and Performance of the World Bank Group 2017 | Appendix K focus groups were conducted with IEG and management counterparts who were the original task team leaders of the IEG’s evaluations and management responders to these evaluations or were designated as MAR update responders and managers. In total, IEG held 14 focus group discussions2. To ensure heterogeneity of focus groups and to stimulate candid discussions, IEG held focus groups for IEG staff separately from management representatives and each focus group targeted one evaluation only. All of the focus groups used the same standardized questionnaire (see the standardized questionnaire at the end of this appendix) to ensure uniformity and impartiality of data collection. The discussions probed perceptions of how IEG’s recommendations may or may not have contributed to the change in the given field. The discussions also aimed at identifying bottlenecks with translating IEG’s recommendations into specific Management Action Plans and implementation during the four-year implementation cycle. Some of the key messages from the focus group included rigidity of the MAR tracking process, minimal knowledge about MAR data among staff and lack of clear accountability over implementation. Another recurring issue was that management does not adopt new strategies and methodologies in response to IEG’s evaluations but rather retrofit planned activities and strategies into proposed action plans. Lastly, management participants mentioned that in some cases IEG’s recommendations are not specific and actionable enough that may contribute to misalignment between management’s action plans and IEG’s recommendations. Additional Information on Environmental Recommendations As discussed on chapter 3, implementation of environmental recommendations varied by the type of recommendation: Recommendations on Knowledge, Measurement, and Impact Table K.1. IEG Assessment Have Seen Lower Uptake Types of IEG recommendations on environmental sustainability and their MAR rating Recommenda- Rated Substantial Rated High or tions or Higher Complete Category (no.) (percent) (percent) High-level policy or strategic direction 9 89 56 How to implement programs or projects 12 92 58 Corporate or institutional 16 94 47 Knowledge, measurement, or impact analysis 13 46 31 Note: Three recommendations were not accepted by World Bank Group management and so were not rated, data was not captured by recording systems for two recommendations. IEG = Independent Evaluation Group; MAR = Management Action Record. Independent Evaluation Group | World Bank Group 209 Management Actions in Areas Related to IEG Recommendations on Environmental Sustainability This section describes the progress that has been made and gaps that remain on implementing IEG’s recommendations on environmental sustainability. Environmental mainstreaming: The World Bank Group promoted environmental sustainability through a 2012 environment strategy, consistent with IEG’s 2008 evaluation. Development of the strategy triggered analytical pieces that then helped to frame environmental engagements. There was an initial increase in support for Country Environmental Analysis and Strategic Environmental Assessments, which later declined. The World Bank improved collaboration and knowledge sharing on climate change though climate change coordinators and tracking of country strategies. Progress has been made on building environmental considerations into country diagnostics. Climate change mitigation: The World Bank devoted high-level attention to promoting energy subsidy reforms which could reduce fossil fuel consumption, including through policy lending. The World Bank approved a framework for its guarantee instruments (which could help to support cleaner infrastructure). The World Bank supported specific projects and financing facilities with demonstration and learning goals, but these approaches were not included in most operations. The World Bank had increased the size of its energy ef ficiency portfolio by 2008, but was not able to extend it further; however, IFC was more successful in promoting energy efficiency. Little progress was documented on using energy ef ficiency opportunities and their price savings to ease the burden of subsidy reform. The World Bank did not develop outcome-oriented targets for climate change mitigation, and corporate targets were set in terms of financing shares rather than emission reduction or development goals. Modest progress was made on improving climate change monitoring metrics during the period of MAR review, but subsequently the World Bank has made significant progress on developing greenhouse gas accounting tools and has begun to apply them to new projects. The World Bank moved to promote cross-sectoral cooperation on climate change through creation of the climate change cross-cutting solutions area. Climate change adaptation: The World Bank created a set of tools and guidance for identifying climate risks in projects, and worked iteratively to improve them. IFC has been slower to develop such tools, with assessment tools being rolled out for forestry but not yet for other industries. The World Bank made progress in developing approaches for resilience indicators, but more work will be needed to ensure these are implemented in projects. The World Bank has moved in the right direction on carrying out assessments of resilience impacts, but knowledge gaps remain on the effectiveness of specific interventions, in part because this analysis is seldom built into project evaluation systems. The World Bank has made impressive gains in its support for hydrometeorological systems, with a rapid scale-up of financing commitments along with outreach to partners and knowledge work. More could be done to build considerations of long-term climate change into policy dialogue with highly vulnerable countries, in addition to the work being done to address current adaptation deficits. Natural resource management: The World Bank has developed a set of proxy indicators for the forest sector that could be used to predict interim outcomes that in turn lead to longer-term impacts. 210 Results and Performance of the World Bank Group 2017 | Appendix K However, the indicators have not been adopted by most forest projects—in part because they have been crowded out by requirements to include corporate indicators. The World Bank developed a database and road map that lays out the institutional approach for mainstreaming biodiversity into World Bank Group operations. Increasingly, most protected area projects include land tenure activities. However, only a fraction of projects under implementation track this as part of their results framework. Progress was made on addressing critical water issues in countries with the greatest water stress, and on improving monitoring of groundwater sustainability. Environmental and social risk management: Progress on environmental and social risk management from IEG’s safeguards evaluation was discussed in chapter 1. Progress on IEG’s other recommendations on environmental and social risks covered similar ground to that evaluation, including expansion of Performance Standards to cover biodiversity in supply chains, and increasing capacity within IFC to support environmental and social performance improvements. Standardized Questionnaire with the Main Questions for the Focus Group Discussions 1. What role did you play in regards to this evaluation and its recommendations, including evaluation before/after the [Committee on Development Effectiveness] meeting and during the MAR follow ups? 2. How useful were the findings and conclusions of the evaluation at the time when evaluation was disclosed, in your opinion? 3. Please elaborate if the recommendations were relevant and actionable at the time when the study was disclosed? 4. Do you believe the recommendations remained relevant to the Bank Group’s work in the following four years? Please explain why or why not? 5. Whatfactors have been most instrumental in supporting or obstructing implementation of the recommendations agreed to in the MAR? 6. Has the MAR update process been useful and if yes, in what ways? If not, please explain why not. 7. Probe: Please elaborate on the Action Plan for this study: how they were created and agreed on. 8. Probe: Please share your views how the MAR pdates for the agreed action plans been pro- vided and what have been positive and negative experiences around the updates. 9. What do you think IEG did successfully to ensure that these particular recommendations translated into policy/operational changes in the Bank Group and what it missed? Independent Evaluation Group | World Bank Group 211 1 Note that the methodology for deep dive analysis of Independent Evaluation Group’s recommendations on environmental sustainability is described in appendix B. 2 The evaluation of the International Finance Corporation’s Global Trade Finance Program did not have any focus groups discussions due to unavailability of target audiences. Every effort was made to send multiple meeting requests for each of the study’s target audiences and in some cases two focus groups were held with Management counterparts or additional phone interviews scheduled for a single study to accommodate everyone’s schedules. 212 Results and Performance of the World Bank Group 2017 | Appendix K  219 Project- and Country-Level Performance in the Africa Region 219 The World Bank: Project-Level Performance 219 International Finance Corporation: Project-Level Performance 226 Multilateral Investment Guarantee Agency: Project-Level Performance 232 World Bank Group: Country-Level Performance 234 Evaluation Findings from Project Performance Assessment Reports 235 Evaluation Findings from Country Program Performance 247 East Asia and Pacific Region 253 Project- and Country-Level Performance 253 The World Bank: Project-Level Performance 253 International Finance Corporation: Project-Level Performance 260 Multilateral Investment Guarantee Agency: Project-Level Performance 267 World Bank Group: Country-Level Performance 267 Evaluation Findings from Project Performance Assessment Reports 268 Evaluation Findings from Country Program Performance 272 Europe and Central Asia Region 278 Project- and Country-Level Performance 278 The World Bank: Project-Level Performance 278 International Finance Corporation: Project-Level Performance 286 Multilateral Investment Guarantee Agency: Project-Level Results 292 World Bank Group: Country-Level Performance 293 Evaluation Findings from Country Program Performance 298 Latin America and the Caribbean Region 300 Project and Country-Level Performance in the Latin America and the Caribbean Region 300 The World Bank: Project-Level Performance 300 International Finance Corporation: Project-Level Performance 308 Multilateral Investment Guarantee Agency: Project-Level Results 315 Evaluation Findings from Project Performance Assessment Reports 316 Evaluation Findings from Country Program Performance 322 Middle East and North Africa Region 326 Project- and Country-Level Performance 326 The World Bank: Project-Level Performance 327 The International Finance Corporation: Project-Level Performance 335 Multilateral Investment Guarantee Agency: Project-Level Results 338 World Bank Group: Country-Level Performance 339 Evaluation Findings from Project Performance Assessment Reports 340 Evaluation Findings from Country Program Performance 341 Independent Evaluation Group | World Bank Group 213 South Asia Region 346 Project- and Country-Level Performance 346 The World Bank: Project-Level Performance 347 International Finance Corporation: Project-Level Results and Performance 351 Multilateral Investment Guarantee Agency: Project-Level Performance 358 World Bank Group: Country-Level Performance 358 Evaluation Findings from Project Performance Assessment Reports 359 Evaluation Findings from Country Program Performance 362 List of Summarized Works 363 Figures  220 221 222 223 223 224 225 226 Figure L.9. Africa: Performance of Development Outcome Subindicators of IFC Investment Projects, by XPSR Program Years 228 Figure L.10. Africa: Association Between Development Outcome, IFC Work Quality and IFC’s Role and Contribution, CY14–16 229 Figure L.11. Africa: IFC Work Quality Subindicators, by XPSR Program Years 230 Figure L.12. Africa: Results and Performance Ratings of Evaluated IFC Advisory Projects, by PCR Program Years 231 Figure L.13. Africa: Ratings of Four Indicators of Development Effectiveness of Evaluated IFC Advisory Projects, by PCR Program Years 232 Figure L.14. Africa: Results and Performance Ratings of Evaluated MIGA Projects, by PER Program Years 233 Figure L.15. East Asia and Pacific: Project Outcome Ratings, FY05–16 254 Figure L.16. East Asia and Pacific: World Bank Performance Ratings, FY05–16 255 Figure L.17. East Asia and Pacific: M&E Quality Ratings, FY05–16 256 Figure L.18. East Asia and Pacific: Project Outcome Ratings by Financing Source, FY14–16 257 Figure L.19. East Asia and Pacific: World Bank Performance Ratings by Financing Source, FY14–16 257 Figure L.20. East Asia and Pacific: M&E Quality Ratings by Financing Source, FY14–16 258 Figure L.21. East Asia and Pacific: Development Outcome by Global Practice, FY14–16 259 Figure L.22. East Asia and Pacific: Results and Performance Ratings of Evaluated IFC Investment Projects 261 Figure L.23. East Asia and Pacific: Performance of Development Outcome Subindicators of IFC Investment Projects, by XPSR Program Years 262 Figure L.24. East Asia and Pacific: Association between Development Outcome, IFC Work Quality and IFC’s Role and Contribution, CY14–16 263  264 Figure L.26. East Asia and Pacific: Results and Performance Ratings of Evaluated IFC Advisory Projects, by PCR Program Years 265 214 Results and Performance of the World Bank Group 2017 | Guide to Appendix L Figure L.27. East Asia and Pacific: Ratings of Four Indicators of Development Effectiveness of Evaluated IFC Advisory Projects, by PCR Program Years 266 Figure L.28. Europe and Central Asia: Project Outcome Ratings, FY05–16 279 Figure L.29. Europe and Central Asia: World Bank Performance Ratings, FY05–16 280 Figure L.30. Europe and Central Asia: M&E Quality Ratings, FY05–16 281 Figure L.31. Europe and Central Asia: Project Outcome Ratings by Financing Source, FY14–16 282 Figure L.32. Europe and Central Asia: World Bank Performance Ratings by Financing Source, FY14–16 282 Figure L.33. Europe and Central Asia: M&E Quality Ratings by Financing Source, FY14–16 283 Figure L.34. Europe and Central Asia: Development Outcome by Global Practice, FY14–16 284 Figure L.35. Europe and Central Asia: Results and Performance Ratings of Evaluated IFC Investment Projects 285 Figure L.36. Europe and Central Asia: Performance of Development Outcome Indicators of IFC Investment Projects, by XPSR Program Years 286 Figure L.37. Europe and Central Asia: Association Between Development Outcome, IFC Work Quality and IFC’s Role and Contribution, CY14–16 288 Figure L.38. Europe and Central Asia: IFC Work Quality Subindicators, by XPSR Program Years 289 Figure L.39. Europe and Central Asia: Results and Performance Ratings of Evaluated IFC Advisory Projects, by PCR Program Years 290 Figure L.40. Europe and Central Asia: Ratings of Four Indicators of development effectiveness of Evaluated IFC Advisory Projects, by PCR Program Years 291 Figure L.41. Europe and Central Asia: Results and Performance Ratings of Evaluated MIGA Projects, by PER Program Years 292  300 301 302 303 304 304 305 306 Figure L.50. Latin America and the Caribbean: Results and Performance Ratings of Evaluated IFC Investment Projects 307 Figure L.51. Latin America and the Caribbean: Performance of Development Outcome Indicators of IFC Investment Projects, by XPSR Program Years 308 Figure L.52. Latin America and the Caribbean: Association between Development Outcome, IFC Work Quality and IFC’s Role and Contribution, CY14–16 309 Figure L.53. Latin America and the Caribbean: IFC Work Quality Subindicators, by XPSR Program Years 310 Figure L.54. Latin America and the Caribbean: Results and Performance Ratings of Evaluated IFC Advisory Projects, by PCR Program Years 311 Figure L.55. Latin America and the Caribbean: Ratings of Four Indicators of Development Effectiveness of Evaluated IFC Advisory Projects, by PCR Program Years 312 Figure L.56. Latin America and the Caribbean: Results and Performance Ratings of Evaluated MIGA Projects, by PER Program Years 313  325 Independent Evaluation Group | World Bank Group 215 Figure L.58. Middle East and North Africa: World Bank Performance Ratings, FY05–16 326 Figure L.59. Middle East and North Africa: M&E Quality Ratings, FY05–16 327 Figure L.60. Middle East and North Africa: Project Outcome Ratings by Financing Source, FY14–16 328 Figure L.61. Middle East and North Africa: World Bank Performance Ratings by Financing Source, FY14–16 329 Figure L.62. Middle East and North Africa: M&E Quality Ratings by Financing Source, FY14–16 329 Figure L.63. Middle East and North Africa: Development Outcome by Global Practice, FY14–16 330 Figure L.64. Middle East and North Africa: Results and Performance Ratings of Evaluated IFC Investment Projects 332 Figure L.65. Middle East and North Africa: Development Outcome Subindicators of IFC Investment Projects, by XPSR Program Years 333 Figure L.52. Middle East and North Africa: Association between Development Outcome, IFC Work Quality and IFC’s Role and Contribution, CY14–16 334  335 Figure L.68. Middle East and North Africa: Results and Performance Ratings of Evaluated IFC Advisory Projects, by PCR Program Years 336 Figure L.69. Middle East and North Africa: Ratings of Four Indicators of development effectiveness of Evaluated IFC Advisory Projects, by PCR Program Years 337  345 346 347 348 349 349 350 352 Figure L.78. South Asia Region: Development Outcome Subindicators of IFC Investment Projects, by XPSR Program Years 353 Figure L.79. South Asia: Association between Development Outcome, IFC Work Quality and IFC’s Role and Contribution,CY14–16 354 Figure L.80. South Asia Region: IFC Work Quality Performance, by XPSR Program Years 355 Figure L.81. South Asia Region: Results and Performance Ratings of Evaluated IFC Advisory Projects, by PCR Program Years 356 Figure L.82. South Asia Region: Ratings of Four Indicators of development effectiveness of Evaluated IFC Advisory Projects, by PCR Program Years 357 Tables  234 234 267 268 293 293 314 216 Results and Performance of the World Bank Group 2017 | Guide to Appendix L Table L.8. Latin America and the Caribbean: Country-Level Performance Ratings, FY14–17 314 Table L.9. Middle East and North Africa: Project Outcome Ratings by Country Type, FY14–16 339 Table L.10. Middle East and North Africa: Country-Level Performance Ratings, FY14–17 340 Table L.11. South Asia: Project Outcome Ratings by Country Type, FY14–16 359 Table L.12. South Asia: Country-Level Performance Ratings, FY14–17 359 Independent Evaluation Group | World Bank Group 217 abbreviations AAA Analytical and Advisory Assistance CAS country assistance strategy CDD community-driven development CLR Completion and Learning Report CLRR Completion and Learning Report Review CPF Country Partnership Framework CPS country partnership strategy DPL development policy loan DPO development policy operation DRM disaster risk management E&S environmental and social EGPRC Economic Governance and Poverty Reduction Credit GDP gross domestic product GP Global Practice HEPI Health Equity and Performance Improvement HNP Health, Nutrition, and Population IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report ICT information and communication technologies IDA International Development Association IEG Independent Evaluation Group IFC International Finance Corporation IMF International Monetary Fund ISN interim strategy note M&E monitoring and evaluation MDTF multi-donor trust fund MIGA Multilateral Investment Guarantee Agency OECS Organisation of Eastern Caribbean States PBG policy-based guarantee PCR Project Completion Reports PDR People’s Democratic Republic PFM public financial management PLR performance and learning review PPAR Project Performance Assessment Report PPP public-private partnership PRSC Poverty Reduction Support Credits SAS screening, appraisal, and structuring SME small and medium enterprise SPL Social Protection and Labor SURR Social, Urban, Rural, and Resilience WSS water supply and sanitation 218 Results and Performance of the World Bank Group 2017 | Abbreviations Africa Region Project- and Country-Level Performance in the Africa Region This note presents the results and performance separately for the World Bank, International Finance Corporation (IFC), and Multilateral Investment Guarantee Agency (MIGA) given the different time frames, operating models, and clients of the three institutions. The World Bank: Project-Level Performance For the current review period, fiscal years (FY) 14–16, the development outcome rating for 70 percent of projects in the Africa Region was moderately satisfactory or better (MS+), improving from 63 percent MS+ during the period FY11–13. While this outcome rating is below the 74 percent MS+ rating in all other Regions during FY14–16, projects in the Africa Region have improved to greater extent than those outside of the Region (figure L.1, panel a). By commitment volume, the rating was 80 percent MS+ in the Africa Region during FY14–16, compared with an average 85 percent MS+ rating across all other Regions (figure L.1, panel b). This 80 percent of volume committed to projects rated MS+ represented a notable improvement within the Africa Region, improving from 62 percent MS+ during FY11–13. Overall World Bank performance, a composite of ratings for quality at entry and project supervision, was 68 percent MS+ in the Africa Region during FY14–16, compared with an average 73 percent MS+ rating across all other Regions (figure L.2, panel a). Within the overall World Bank performance rating in the Africa Region, quality at entry was rated 55 percent MS+, compared with an average 62 percent MS+ rating across the other Regions. Supervision was rated 76 percent MS+ in the Africa Region, compared with an average 79 percent MS+ rating across the other Regions. By commitment volume, World Bank performance in the Africa Region during FY14–16 (78 p ercent MS+) was below the average for all other Regions combined (85 p ercent MS+); however, the improvement by volume is signi ficant at th e 5 p ercent confidence level (figure L.2, pane l b). The overall quality of monitoring and evaluation (M&E) for projects in the Africa Region during FY14–16 was 31 percent substantial or better (S+), compared with an average 35 percent S+ rating across all other Regions (figure L.3, panel a). By commitment volume, the quality of M&E during the same period is 34 percent MS+, compared with an average 48 percent S+ rating across all other Regions (figure L.3, panel b). The development outcome rating for projects supported by the International Bank for Reconstruction and Development (IBRD) in the Africa Region during FY14–16 was 80 percent MS+, compared with an average 75 percent MS+ rating across all other Regions (figure L.4, panel a). By commitment, the Independent Evaluation Group | World Bank Group 219 Figure L.1. Africa: Project Outcome Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: MS+ = a rating of moderately satisfactory or better. rating was 49 percent MS+ in the Africa Region, compared with an average 85 percent MS+ rating across all other Regions (figure L.4, panel b). The development outcome rating for projects supported by the International Bank for Reconstruction and Development (IBRD) in the Africa Region during FY14–16 was 80 percent MS+, compared with an average 75 percent MS+ rating across all other Regions (figure L.4, panel a). By commitment, the rating was 49 percent MS+ in the Africa Region, compared with an average 85 percent MS+ rating across all other Regions (figure L.4, panel b). 220 Figure L.2. Africa: World Bank Performance Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: MS+ refers to rating of moderately satisfactory or higher. World Bank performance for IBRD-supported projects in the Africa Region during FY14–16 (80 percent MS+) was slightly higher than the average across all other Regions (75 percent MS+). World Bank performance ratings for IDA-supported projects (70 percent MS+) dip just below average for IDA-supported projects across all other Regions (74 percent MS+: figure L.5). By number of projects, M&E quality for IBRD-supported projects in the Africa Region during FY14–16 was 40 percent S+ rating, compared with 36 percent S+ rating across all other Regions. M&E quality Independent Evaluation Group | World Bank Group 221 Figure L.3. Africa: M&E Quality Ratings, FY05–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: M&E = monitoring and evaluation; S+ = a rating of substantial or better ratings for IDA-supported projects (30 percent S+) are just below the respective ratings for IDA- supported projects across all other Regions (34 percent S+: figure L.6). Global Practices with the largest number of projects rated during FY14–16 performed well. Outcome rating for the 16 Health, Nutrition, and Population (HNP) projects, 12 Social Protection and Labor (SPL) projects, and 22 Transport and Information and Communication Technologies (ICT) projects were 94 percent, 92 percent, and 77 percent MS+, respectively (figure L.7). 222 Figure L.4. Africa: Project Outcome Ratings by Financing Source, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by number Source: Independent Evaluation Group data. Note: IBRD = International Bank for Reconstruction and Development; IDA =International Development Association; MS+ = a rating of moderately satisfactory or better. World Bank Performance Ratings by Financing Figure L.5. Africa: Source, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by number Source: Independent Evaluation Group data. Note: IBRD = International Bank for Reconstruction and Development; IDA =International Development Association; MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 223 Figure L.6. Africa: M&E Quality Ratings by Financing Source, FY14–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: IBRD = International Bank for Reconstruction and Development; IDA =International Development Association; S+ = a rating of substantial or better; M&E = monitoring and evaluation. World Bank performance was particularly strong for Africa Region projects rated for the SPL (92 percent MS+) and HNP (88 percent MS+) Global Practices (GPs) during FY14–16. M&E was strong for the SPL (67 percent substantial or above), Energy and Extractives (57 percent substantial or above), and HNP (50 percent substantial or above) GPs for Africa Region projects closed during FY14–16, but particularly low for the Environment and Natural Resources and Macroeconomics and Fiscal Management GPs (both 9 percent substantial or above). 224 Figure L.7. Africa: Development Outcome by Global Practice, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: “Other” includes countries without International Development Association or International Bank for Reconstruction and Development status or project locations that are above country level. MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 225 International Finance Corporation: Project-Level Performance International Finance Corporation Investments Project-Level Results Of the 50 investment projects in the Africa Region that were evaluated in the current review period, CY14–16, nearly half (or 46 percent) were rated positively (mostly successful or better, MS+) for their development outcome. This share of projects with MS+ ratings for development outcome is slightly higher at 61 percent, if adjusted by IFC net commitment or volume (figure L.8). Compared with the Figure L.8. Africa: Results and Performance Ratings of Evaluated IFC Investment Projects Share of Projects with Positive Ratings a. By number of projects, three-year moving averages b. By IFC net commitment, three-year moving averages Source: Independent Evaluation Group evaluation data for IFC XPSRs as of December 12, 2017. Note: (1) Positive ratings for development outcome consists of mostly successful, successful, or highly successful (aggregated as MS+) ratings. Positive ratings for investment outcome and IFC’s role and contribution consist of excellent or satisfactory (aggregated as S+) ratings. (2) Due to the revision of the XPSR guidelines in 2015, the indicator, IFC’s role and contribution, has been amended starting CY15, and was replaced with the indicator, IFC additionality. IFC = International Finance Corporation; XPSR = Expanded Project Supervision Report. 226 CY11–13 results by volume, however, the share of projects with MS+ ratings this current review period weakened slightly despite the near-doubling of IFC’s net commitments. Despite the decline in positive (MS+) ratings for development outcome in CY14–16, IFC’s return on its investments (measured by IFC’s investment outcome rating) has remained constant. Adjusted by commitment amounts, however, the share of projects rated satisfactory or better (S+) in meeting IFC’s investment return benchmarks are once-again close to historical peak levels in CY14–16. The lower share of positive (satisfactory or better, S+) ratings in three of the four subindicators of development outcome has contributed to the decline in overall development outcome results in CY14–16. By number of projects, the percentage of S+ rated projects for their contribution to private sector development declined even more sharply this current period, although performance has improved if the ratings are weighted by IFC’s commitment amounts, suggesting strong performance by large-sized projects in the Africa Region in CY14–16 (figure L.9). Positive (S+) ratings for environment and social effects remained the same, based on the number of evaluated projects. The result differs slightly if the ratings are weighted by IFC commitment amount. By volume, 58 percent of the projects in CY14–16 had S+ ratings for the projects’ overall environmental and social (E&S) effects, but this share is lower compared with CY11–13. Six large-sized projects evaluated in CY14–16 were rated less than satisfactory (partly unsatisfactory or unsatisfactory rating) for their E&S outcomes, which lowered the overall E&S effects rating for the period. A review of development outcome, IFC work quality and IFC’s role and contribution ratings, which have declined further in CY14–16, indicate strong association among these three indicators of IFC results and performance. Figure L.10 shows the strong association between high development outcome ratings with high IFC work quality ratings, especially if the ratings are weighted by volume. Additionally, the association between low development outcome and low IFC work quality ratings is likely, if the results are measured by number of projects. Positive development outcome ratings are also associated with positive IFC role and contribution ratings or vice-versa. Positive or high IFC work quality ratings are also associated with high IFC investment outcome ratings. Of the two subindicators of IFC work quality, positive (S+) ratings for IFC’s screening, appraisal, and structuring (SAS) of projects in the Africa Region have improved in CY14–16 when compared with CY11–13, whether measured by number of projects or by the amount of IFC net commitment. On the other hand, the Share of Projects with Positive Ratings for IFC’s supervision and administration (SUP) dropped slightly in CY14–16 by number of projects although not as much as the 9-percentage point decline in S+ ratings by commitment amount (figure L.11). This drop in the share of S+ ratings, by volume, points to shortcomings in IFC’s supervision and administration of a few large size projects in the region. Independent Evaluation Group | World Bank Group 227 Figure L.9. Africa: Performance of Development Outcome Subindicators of IFC Investment Projects, by XPSR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving averages b. By IFC net commitment, three-year moving averages Source: Independent Evaluation Group evaluation data for IFC investment projects as of December 12, 2017. Note: (1) S+ = a rating of satisfactory or excellent. (2) E&SE = environmental and social effects; ECSUS = economic sustainability; PBS = project business success; PSD = private sector development. These four subindicators comprise the development outcome dimension of an IFC investment project. (3) No opinion possible or not applicable ratings were excluded. IFC = International Finance Corporation; XPSR = Expanded Project Supervision Report. 228 Figure L.10. Africa: Association Between Development Outcome, IFC Work Quality and IFC’s Role and Contribution, CY14–16 By number of evaluated projects By IFC net commitment ($, millions) a. Association between development outcome and IFC work quality b. Association between development outcome and IFC role and contribution c. Association between work quality and IFC investment outcome Source: Independent Evaluation Group evaluation data for IFC investment projects as of December 12, 2017. Note: (1) IFC work quality in CY14–16 reflects the aggregate ratings of two subindicators: screening, appraisal, and structuring and supervision and administration. (2) High development outcome = highly successful, successful, and mostly successful ratings; low development outcome = mostly unsuccessful, unsuccessful and highly unsuccessful ratings; high work quality = excellent and satisfactory ratings; low work quality = partly unsatisfactory and unsatisfactory ratings; high IFC role and contribution = excellent and satisfactory ratings; low IFC role and contribution = partly unsatisfactory ratings; high IFC investment outcome = excellent and satisfactory ratings; low IFC investment outcome = partly unsatisfactory and unsatisfactory ratings. (3) Due to the revision of the Expanded Project Supervision Report guidelines in 2015, the indicator IFC’s role and contribution has been amended starting CY15, and was replaced with the indicator IFC additionality. Independent Evaluation Group | World Bank Group 229 Figure L.11. Africa: IFC Work Quality Subindicators, by XPSR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC net commitment, three-year moving average Source: Independent Evaluation Group evaluation data for IFC investment projects as of December 12, 2017. Note: Based on the 2015 revised XPSR guidelines, IFC work quality is measured by two indicators: screening, appraisal, and structuring (SAS) and supervision and administration (SUP) in the project. IFC = International Finance Corporation; XPSR = Expanded Project Supervision Report. IFC Advisory Services Project-Level Results Of the 33 IFC Advisory Services projects in the Africa Region evaluated during the FY14–16 period, only a third (33 percent) had a rating of MS+ for overall development effectiveness. This development effectiveness success rating is the lowest since IFC and Independent Evaluation Group (IEG) started evaluating IFC advisory projects in FY08 (figure L.12). A similar downward trend is evident in the percentage of projects rated satisfactory or better (S+) for IFC’s role and contribution in the evaluated advisory projects, especially if weighted by project expenditures. Satisfactory or better (S+) ratings for output achievement show strong improvement in FY14–16, with a 19-percentage point increase in positive ratings from FY11–13 (figure L.13). Among the subindicators of development effectiveness, achievement of project output (for example, a diagnostic 230 Figure L.12. Africa: Results and Performance Ratings of Evaluated IFC Advisory Projects, by PCR Program Years Share of Projects with Positive Ratings a. By number of projects, three-year moving average b. By IFC expenditure ($, millions), three-year moving average Source: Independent Evaluation Group evaluation data for International Finance Corporation PCRs as of December 12, 2017. Note: (1) The reported ratings are based on IEG’s final ratings in the completed PCRs. (2) Positive ratings for development effectiveness consist of highly successful, successful, or mostly successful (aggregated as MS+) ratings. Positive ratings for IFC’s role and contribution consist of excellent or satisfactory (aggregated as S+) ratings. PCR = Project Completion Report. report, delivery of training program or drafting of laws and regulations, and so on.) has consistently outperformed other development effectiveness indicators, especially the outcome achievement subindicator. Success (S+) ratings for outcome achievement among the evaluated advisory projects in the Africa Region in FY14–16 reached their lowest point (39 percent) since the PCR Program started in FY08. The other development effectiveness subindicators, strategic relevance and ef ficiency, also performed poorly. Independent Evaluation Group | World Bank Group 231 Figure L.13. Africa: Ratings of Four Indicators of Development Effectiveness of Evaluated IFC Advisory Projects, by PCR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC expenditure ($, millions), three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC PCRs as of December 12, 2017. Note: (1) The reported ratings are based on IEG’s final ratings in the completed Project Completion Reports (PCRs). (2) Success ratings for the development effectiveness subindicators = excellent or satisfactory (referred to as S+) ratings. Multilateral Investment Guarantee Agency: Project-Level Performance Of the 23 Multilateral Investment Guarantee Agency (MIGA) projects in the Africa Region evaluated in FY11–16, 61 percent were rated satisfactory or better (S+) for their development outcome, which is about the same success rate as past review periods (figure L.14). The share of S+ ratings for MIGA’s role and contribution also showed a similar, near-constant trend. Among the three performance indicators, MIGA’s Assessment, Underwriting and Monitoring, showed considerable improvement in the share of S+ ratings from 50 percent in FY04–09 (the early years of independent evaluation in MIGA) to 70 percent in FY11–16. 232 Figure L.14. Africa: Results and Performance Ratings of Evaluated MIGA Projects, by PER Program Years Share of Projects with S+ Ratings, by Number of PERs a. By main indicators of MIGA project results and MIGA performance, six-year moving average b. Ratings of the four subindicators of development outcome, by PER program years Source: Independent Evaluation Group Evaluation Data for MIGA PERs as of December 12, 2017. Note: (1) PERs = Project Evaluation Reports. (2) Positive or success ratings for development outcome, MIGA’s role and contribution and assessment, underwriting and monitoring refer to excellent or satisfactory (referred to as S+) ratings. (3) BP = business performance; ECSUS = economic sustainability; E&SE = environmental and social effects; PSD = contribution to private sector development. MIGA = Mulitlateral Investment Guarantee Agency. Of the four subindicators of development outcome, only E&S effects improved in FY11–16. Ratings of the three other subindicators of development outcome weakened, particularly the projects’ contribution to private sector development. Over a third (34 percent) of the projects evaluated in FY11–16 had less than satisfactory rating for their contribution to the development of the region’s private sector. Independent Evaluation Group | World Bank Group 233 World Bank Group: Country-Level Performance Projects were rated for 34 IDA and 4 IBRD countries in the Africa Region during FY14–16. Within the IDA complement, 14 countries were designated fragile. By number, project outcome ratings are close to the average for the Region for all three country types; however, by volume, projects in both IDA (84 percent MS+) and IDA fragile (81 percent MS+) countries perform better than those in the smaller number of IBRD counterparts (53 percent MS+; table L.1). Table L.1. Africa: Project Outcome Ratings by Country Type, FY14–16 Countries Projects Volume Evaluated Rated MS+ Total Amount Rated MS+ Country Type (no.) No. (percent) ($, millions) (percent) IDA fragile 14 55 69 2,625 81 IDA nonfragile 20 106 71 7,300 84 IBRD fragile 0 0 n.a. 0 n.a. IBRD nonfragile 4 8 75 302 53 IDA total 34 161 70 9,925 83 IBRD total 4 8 75 302 53 Fragile total 14 55 69 2,625 81 Other n.a. 17 71 1,037 54 Total 38 186 70 11,264 80 Source: World Bank Business Intelligence and Independent Evaluation Group data. Note: One project was excluded when measured by volume due to lack of volume data. “Other” includes countries without IDA or IBRD status or project locations that are above country level. IBRD = International Bank for Reconstruction and Development; IDA =International Development Association; MS+ = a rating of moderately satisfactory or better; n.a. = not applicable. During FY17, IEG produced Completion and Learning Report Reviews (CLRRs) for six countries in the Region, namely Cameroon, Ethiopia, Madagascar, Mauritius, Mozambique, and Togo. IEG rated development outcome moderately satisfactory for two country programs, moderately unsatisfactory for another, and unsatisfactory for two. No rating was provided for Togo due to data limitations (table L.2).1 Table L.2. Africa: Country-Level Performance Ratings, FY14–17 CASCR/ IFC CLR Review World Bank Perfor- MIGA Review FY Country Period Outcome Performance mance Performance 2014 Burkina Faso FY10–12 MS Good n.a. n.a. Ghana FY08–12 MU Fair n.a. n.a. Guinea FY04–13 n.a. n.a. n.a. n.a. Kenya FY10–13 MS Fair n.a. n.a. Liberia FY09–12 MS Good n.a. n.a. 234 Table L.2, continued. CASCR/ IFC CLR Review World Bank Perfor- MIGA Review FY Country Period Outcome Performance mance Performance Mauritania FY07–12 U Poor n.a. n.a. Nigeria FY10–13 MS Good n.a. n.a. Rwanda FY09–13 S Superior n.a. n.a. São Tomé and Príncipe FY06–12 MS Good n.a. n.a. South Africa FY08–12 U Fair n.a. n.a. 2015 Cabo Verde FY09–12 MU Fair n.a. n.a. 2016 Botswana FY09–13 U Fair n.a. n.a. Cóte d’Ivoire FY10–14 MS Good n.a. n.a. Lesotho FY10–14 MU Fair n.a. n.a. Mali FY08–15 U Fair n.a. n.a. Uganda FY11–15 MU Fair n.a. n.a. 2017 Cameroon FY10–14 U Fair n.a. n.a. Ethiopia FY13–16 MS Good n.a. n.a. Madagascar FY07–13 U Poor n.a. n.a. Mauritius FY07–15 MS Fair n.a. n.a. Mozambique FY12–15 MS Fair n.a. n.a. Togo FY08–17 n.a. n.a. n.a. n.a. Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: All World Bank performance ratings were standardized with the following conversions: satisfactory as superior, moderately satisfactory as good, moderately unsatisfactory as fair, and unsatisfactory as poor. CASCR = country assistance strategy completion report; CLR = Completion and Learning Report; IFC = International Finance Corporation; MIGA = Multilateral.Investment Guarantee Agency; MS = moderately satisfactory; MU = moderately unsatisfactory; U = unsatisfactory; n.a. = not applicable. Evaluation Findings from Project Performance Assessment Reports During FY17, IEG completed 17 Project Performance Assessment Reports (PPARs) covering 22 projects in the Africa Region. For development outcome, IEG rated 2 projects highly satisfactory, 1 satisfactory, 7 moderately satisfactory, 10 moderately unsatisfactory, and 2 unsatisfactory. Nations in Africa had accumulated large stocks of obsolete pesticides over the second half of the 20th century. Some of these stocks included persistent organic pollutants, which posed serious long-term threats to human health and the environment. The Africa Stockpiles Program 2005–13 was an attempt to develop and implement cost-effective and ef ficiency inventory, collection, and disposal operations, and to prevent the accumulation of future stocks, through a partnership of international organizations working in collaboration. The World Bank was to coordinate the Africa Stockpiles Program (Ethiopia, Mali, Morocco, South Africa, Tanzania) and manage funds (from a $25 million Global Environment Facility grant), with technical support from the Food and Agriculture Organization, and other support from civil society organizations. Yet the complex international partnership arrangements made coordination very challenging, leading to confusion in implementation and failure to deliver synergies. Independent Evaluation Group | World Bank Group 235 The struggle to design partnership arrangements meant that preparation took four years, with little to show for the effort at the country level and an inadequate level of readiness for implementation. At the core of the challenges encountered were fundamental differences in leadership from the World Bank and from Food and Agriculture Organization. The project delivered several outputs that led to elimination of some pesticide stocks, and project effectiveness varied significantly across countries. But none of the countries could eliminate pesticide stocks and associated waste to the extent originally envisioned, and two countries were not able to eliminate stocks even after project restructuring that reduced the level of ambition. The horizontal adaptable program loan covering six countries at once did not add ef ficiencies in preparation, or synergies in implementation, because there was little opportunity for intercountry interaction, or cooperation. Relatively little was accomplished on preventing future accumulation because project design concentrated on cleanup aspects. Diffuse responsibilities, high turnover of task team leaders, and a lack of sustained attention from the World Bank contributed to weak project supervision. The World Bank’s Implementation Completion and Results Report (ICR) rated the project outcome as moderately unsatisfactory; IEG rated the project as unsatisfactory, based on incomplete implementation of the project. Gravely reminiscent of the 1984 famine, some 20 million people in Ethiopia faced destitution and starvation in 2003, after a sixth poor rainy season in three years. It was in the wake of this crisis that the World Bank, and International Fund for Agricultural Development helped develop and fund a program that would increase the resilience of Ethiopia’s 12 million livestock dependent pastoral and agropastoralist communities. The Pastoral Community Development Programs (Phases I and II), supported the delivery of basic social services, productive activities, an early warning system, and rural savings and lending to reduce vulnerability. IEG found that the two projects made a significant contribution to meeting critical social and economic infrastructure needs, but the community-driven development (CDD) modality was not well adapted to identifying and meeting nonsedentary, or transhumant, needs. Most of the infrastructure provided was sedentary, and the provision was not sensitive to human geography. Overall, the assessment found that project support for pastoral development needs to take place within the context of a sound sector strategy that can offer well-researched, analytical advice about climatic and environmental conditions, and the implications of shifting social, economic, and demographic pressures and trends. Further, a sustainable, livelihood-centered approach would have required a much broader range of coordinated productive investments, taking into account the combination of investments package needed to significantly elevate livelihoods. Given the constraints and challenges of pastoralist communities, a sustainable livelihood approach would be centered on the livestock value chain, rangeland management, access to water and grazing land as well as the development of supplementary sources of income in the nonfarm economy. The World Bank’s ICR rated both phases of the project moderately satisfactory; IEG rated phase I moderately unsatisfactory and phase II moderately satisfactory. 236 The Statistical Development Project in Ghana was approved at a time when the global development community became acutely aware of the need to upgrade and significantly increase the supply of relevant statistics. Quality data was sought in the quest to improve policy making, enhance governance, and monitor progress. The Trust Fund for Statistical Capacity Building, a World Bank- administered, multi-donor trust fund (MDTF) provided the grant to prepare Ghana Statistical Service for institutional reform, and to ensure timely and extensive analysis and dissemination of census data. The MDTF included the Department for International Development of the United Kingdom and the European Commission. The total cost was $5.9 million, with which the MDTF supported legal and institutional reform, Ghana Statistical Service capacity building, improved quality and dissemination of statistical products, and 2010 census postenumeration. In principle, the design of the project should have been clear and straight-forward, but the complexity of institutional reforms, and fragmentation of reform efforts, made reforms dif ficult to implement. This weakness in design offered little assurance of the likelihood for success. In fact, in supporting corporate or institutional reform agendas, it is essential to ensure orderly, ef ficient, and effective implementation to minimize the impact on staff. The project provided two valuable lessons on statistical capacity building projects. First, it highlighted the need for a responsive governing body to give strategic direction to the national statistical of fice to ensure data are relevant and reliable, emphasizing the importance of having present, directors who represent the interest of data users (researchers, development partners, academicians, students and policy makers). Second, the capacity of the World Bank team to deliver results in an area in which it played a large role in mobilizing international support (that is, for statistical development) should not be taken for granted. Even though the World Bank team appeared to be well qualified, and the task team seemed to possess the right skills mix, project results were modest. The main shortcoming was in achieving institutional reforms. At the time of IEG’s evaluation, the Statistics Bill had not yet been approved by parliament. Timely analysis and dissemination of the 2010 census was achieved before the project closed. Both the World Bank’s ICR and IEG’s assessment rated the project outcome unsatisfactory. In the years leading up to the preparation of the first of the two-part Agriculture Sector Development Policy Operation (DPO) Series (the first DPOs prepared specifically for the agriculture sector in Ghana and more broadly, the Africa Region), Ghana experienced a sustained period of economic growth and reductions in national poverty rates. During this period the agricultural sector also experienced strong growth, driven largely by the cocoa subsector, and expansion of the area under cultivation. Most of these gains were achieved in the South and Central regions of the country, where agro- ecological conditions are more favorable; however, poverty remained high in rural areas and was increasingly concentrated in the more arid northern regions. The World Bank prepared the DPO series to support the government with implementing newly developed strategies that aimed at diversifying the economy away from its dependence on cocoa and toward cereals and other cash crops for export markets; addressing regional disparities; and improving the productivity of staple food crops (because of the potential impact on poverty and Independent Evaluation Group | World Bank Group 237 meeting domestic demand). Historically, budget allocations to the agriculture sector from government sources covered salaries only. The government relied predominantly on investments through rural development and agriculture projects to address other needs. Strengthening the sector’s ability to plan and implement using budgetary resources instead of project funding was viewed as a critical element of implementing a coordinated and internally consistent development policy in the sector. There was clear alignment between the DPO prior actions and government policies, and the prior actions were mostly carried out as designed, although expected development outcomes were not achieved. Many prior actions involved upstream adoption of legislative or government decisions, plans and policies that were insuf ficiently linked to implementation. The government fell short in ensuring implementation, and the intended reforms in many areas have been under resourced. Prior actions were also spread across too many policy areas, with insuf ficient depth in any one area to translate into results in the field. The first objective (increasing agriculture’s contribution to growth and poverty reduction) was rated modest. Agriculture sector growth rates have slowed over time and have been significantly lower than that of the overall economy. There is no hard evidence to establish agriculture’s contribution to poverty reduction. The ef ficacy of the second objective (improved management of soil and water resources) was rated negligible. Despite efforts to disseminate information on various sustainable land management GPs, the GPs did not result in high adoption rates of better soil and water GPs. Insuf ficient resources at the district level and uncertain security of land use rights have constrained further progress. The ef ficacy of the first objective of the second series of DPOs (improving farmers’ productivity and market access) was rated modest. Yields of most major crops have been relatively flat and well below their potential. Farmers continue to face significant hurdles that limit the uptake of productivity-enhancing technologies and access to markets. The ef ficacy of the second objective (improved management of the sector) was rated negligible. Changes to budgetary processes have been minimal, and the structures to facilitate coordination have not functioned effectively. Coordination across the Ministry of Food and Agriculture directorates and other agencies with work affecting the sector remains weak, which is a key bottleneck to improving planning and management of the entire sector. The World Bank’s ICRs rated both phases of the series moderately satisfactory. IEG rated outcome for both phases moderately unsatisfactory. Ghana emerged from an economic crisis in 2006 and wanted to diversify its growth for sustainability and achieving middle-income status by 2015. It emphasized the importance of ICT in achieving the country’s objectives of diversified export-led economic growth, increased competitiveness, and transparent, accountable and ef ficient government. The World Bank sought to support the government through a $116 million e-Ghana Project 2006–14 in generating growth and employment by developing the Information Technology Enabled Services industry, and contributing to improved ef ficiency and transparency of selected government functions through e-government applications. The project used IDA financing together with cofinancing from other development partners for the Government Integrated Financial Management Information System. The project created an enabling environment for ICT sector, supported investments in Information Technology Enabled Services, 238 established a Business Process Offshoring center and conducted business incubation activities. The ICT sector grew at 23.3 percent per year in the project period. The project automated the business processes of Ghana Revenue Authority, which led to a 400,000 increase in new tax payers and a tripling of tax collection between 2010 and 2015. The project also automated the Registrar General Department, which led to a 55 percent increase in company registration in 2012–13. The project supported setting up several eServices like birth certification, online marriage registration, company registration, and so on; however, its coverage was only about 10 percent due to low Internet penetration, poor connectivity and lack of citizen awareness. The project implemented a Management Information System covering central government funds that accounted for 66 percent of total public expenditure. However, there was inadequate funding for the operation and maintenance of the Government-Wide Area Network leading to a risk of collapse of all e-Government applications and the management information system. Overall, the World Bank’s ICR rated development outcome satisfactory; IEG’s field-based assessment rated outcome moderately satisfactory due to low coverage of eServices, limited control of the management information system, and insuf ficient impact on employment. IEG assessed DPOs supporting Ghana’s $394 million Economic Governance and Poverty Reduction Credit (EGPRC; 2009–10) and Poverty Reduction Support Credits (PRSCs) 7 ($219 million) and 8 ($100 million), which were disbursed 2011 and 2012. The first poverty reduction credit was stand- alone and sought to support a three-pronged effort to restore budgetary discipline and tackle long- standing public sector and energy issues, while protecting the poor. It worked across many policy areas such as public financial management (PFM), public sector reform, and social protection. IEG rated World Bank performance as satisfactory, based on high relevance of design and objective as well as close coordination, particularly in risk assessment, with an International Monetary Fund (IMF) program, reflected in the stand-alone, two-tranche instrument. IEG’s evaluation found a fall in the fiscal deficit from 8.4 percent in 2008 to 3.2 percent in 2011 and other financial management improvements, reflecting improved budgetary discipline. Actions on wage bill growth were particularly important, though they were insuf ficient in mitigating underlying risks, which later became evident. Public sector reform results were mixed. For example, political interventions undermined efforts to reform the civil service, leading to increased wage hikes, and a continuous increase in the number of civil servants. On the other hand, improved social protection contributed to a substantial drop in poverty across the country supported by the rapid expansion of the Livelihood Empowerment Against Poverty program, although the evaluation also notes that, despite fiscal consolidation, the social sectors budget remained at about 50 percent of total expenditures in 2011. Overall, the World Bank’s ICR rated development outcome satisfactory; IEG’s fi ld-based assessment rated outcome moderately satisfactory. A programmatic series of two PRSCs followed the 2009 stand-alone poverty reduction credit in Ghana. The objectives of PRSCs 7 and 8 were like those of the EGPRC. They included restoring budget discipline and fi ancial stability; tackling long-standing public sector and energy issues; Independent Evaluation Group | World Bank Group 239 protecting the poor; and preparing the economy for the oil era (including managing expected oil revenues). Although objectives were highly relevant, there were shortcomings in design, such as not addressing emerging risks to overall macroeconomic stability from the single-spine salary system implementation, and lack of strong policy content for reforming endowed organizations. Measures supported in PRSCs 7 and 8 to maintain fiscal discipline proved ineffective. Major macroeconomic destabilization ensued, characterized by monetary accommodation of the fiscal deficit, high inflation, and persistent pressure on the exchange rate. Ultimately, macroeconomic stabilization was achieved in 2015 with IMF program support and a new World Bank programmatic DPO series. The situation in public utilities improved somewhat due to tariff adjustments, but previously accumulated debt continues to undermine financial viability. IEG rated protecting the poor as substantial due to considerable progress in the Livelihood Empowerment Against Poverty East Asia and Pacific program (beyond the gains under EGPRC), and ef ficiency gains from joint targeting. However, concerns about expenditure composition remain because administrative expenditures and debt servicing demands appear to crowd out public service expenditures. IEG also rated achievements regarding the objective of preparing the economy for the oil era as substantial. Ghana established institutions that could allow transparent oil revenue management and account for intergenerational considerations by creating the Ghana Heritage Fund for accumulating oil revenues for the next generations. IEG rated the World Bank performance as moderately unsatisfactory given the insuf ficient consideration in project designs to the single- spine salary system and PFM measures, and ongoing ineffective dialogue with authorities, albeit in a dif ficult environment. Lessons from the assessment include, for example, af firmation of the important role that the World Bank can play in its policy lending, helping clients to preserve borrowing space and safeguard sustainability of pro-growth and pro-poor expenditure in the long run; and the criticality of adequate macroeconomic management as a precondition of budget support, and as relevant, the importance of World Bank–IMF cooperation; three objectives. Overall, the World Bank’s ICR rated development outcome moderately satisfactory. IEG’s field-based assessment rated outcome moderately unsatisfactory reflecting modest design for all objectives and substantial ef ficacy for two of the four objectives (social protection and oil revenue management)—fiscal discipline and public sector reforms objectives were not as well realized. In the early 2000s, Kenya’s statistical institutions were in decline; the capacity of Kenya’s National Statistical System to produce timely, high-quality, and relevant statistical information had deteriorated over the two decades prior to the project. Among the causes identified at the time were: declining budgetary support for the production and dissemination of statistics, inadequate professional staff at the management and technical levels, dilapidated statistical infrastructure and ICT equipment, and a loss of relevance due to inadequate engagement with users. The objective of the Development of the National Statistical System Project was, to establish a sustainable national statistical system to provide reliable, timely and accurate data in accordance with international standards. Total project cost was estimated at $50.5 million. In March 2007, an IDA credit of $20.5 million was approved, out of which $11.7 million was disbursed, while $8.8 million was canceled when the project closed in September 2012. The project supported institutional reforms to create an autonomous agency with 240 clear mandates, training, and technical assistance to enhance managerial and professional skills, as well as statistical infrastructure and ICT equipment. The project was largely implemented as designed but experienced delays. Before project approval, the World Bank suspended new operations in Kenya because of concerns about governance. Later, the country suffered civic strife involving an outbreak of violence. The project offered useful lessons for statistical capacity building projects. First, ensuring that institutional reforms (both corporate and legal) were in place before project implementation and capacity building efforts began contributed to success in Kenya – which could be the standard approach to future statistical capacity building efforts. Second, establishing Sector Working Groups (also known as interagency working groups) as a coordinating mechanism proved useful and durable in Kenya and could have wider applicability. A final lesson has broader implications: when the risk of corruption is identified, the mitigation measures need to be realistic and targeted. In the case of this project, World Bank staff maintained that the fiduciary risks were due to capacity limitations. But this view was not corroborated by experience. Overall, the World Bank ICR rated development outcome moderately unsatisfactory, which was upgraded following IEG’s field-based assessment, to moderately satisfactory. IEG observed that substantial progress had been made by the implementing agency, the Kenya National Bureau of Statistics, such that, by June 2016, all three objectives had been substantially achieved. The coverage of source data was enlarged by more frequent and regular surveys, and by an updated registry of business establishments; data are released in a timelier manner; and accessibility of data was advanced by a new dissemination framework put in place by Kenya National Bureau of Statistics management. Most of ficial statistics are now available online and usable as envisaged by the project, including historical data and anonymized microdata sets. Today, Kenya is one of the most advanced countries in Africa in terms of open access to of ficial data. In 2008, when the Malawi Third Social Action Fund Adaptable Program Loan II Project was approved, Malawi was one of the poorest countries in the world with a per capita gross domestic product (GDP) of $302. Over half the population lived below the poverty line, with 22 percent below the food poverty line. Human development indicators and access to services were dismal. The economy was trapped in a vicious cycle of emergency appeals to address economic and climatic shocks, to which the international community responded through humanitarian assistance rather than through support to build sustainable economic development. The Malawi Third Social Action Fund series offered an approach to alleviate poverty through a decentralization agenda to ensure that citizens would engage in planning local development initiatives. Adaptable Program Loan II aimed to deepen the community empowerment processes (participatory planning and implementation) at local level that had begun under Adaptable Program Loan I. The operation aimed to finance small labor-intensive community- level public works schemes; support voluntary community savings-investment groups; build capacity of local authorities to manage grant moneys; and support community participation in district planning and implementation of community subprojects. World Bank performance was moderately satisfactory given some shortcomings in expectations of the decentralization process and in risk analysis. The main messages from IEG’s assessment are Independent Evaluation Group | World Bank Group 241 that (i) CDD that emphasizes longer-term development may not always be the best instrument —it can facilitate decentralization, but requires the right balance between setting community priorities and planning local development; (ii) beneficiary feedback throughout provides important evidence and perspective; and (iii) linking income generation interventions with asset building opportunities can help build economic resilience. The World Bank’s ICR rated overall development outcome satisfactory. IEG’s field-based assessment resulted in a development outcome rating of moderately satisfactory, reflecting substantial achievement in improved livelihoods and capacity building and modest effi iency. Approved 15 years after the cessation of a brutal civil war that caused massive economic upheaval and destruction of rural infrastructure, the Mozambique Market-Led Smallholder Development in the Zambezi Valley Project was designed to increase farm productivity and incomes and to connect small farmers to opportunities within the emerging rural economy. IEG’s assessment found that though the project led to outputs it fell short of achieving its aim of increasing agricultural income for smallholder farmers relative to nonproject farmers. Productivity increased for the treatment areas, and farmers achieved crop diversity, but the project fell short of addressing key binding constraints associated with the unstructured and asymmetric nature of the country’s agricultural value chains. IEG’s field-based assessment found no evidence that the project enabled sustained marketing opportunities for the target groups. Rural savings schemes helped to smooth consumption and respond to shocks, such as flooding, but the schemes did not provide seed capital so were unable contribute to the project’s investment aims. A key project lesson was that market-based approaches to developing the smallholder sector require up-front analysis of skills, knowledge, and capacity to engage in various value chain activities, such as marketing. In Mozambique, owing in part to the nascent nature of the country’s market-led economy, skills in areas such as ef ficient marketing were rare in many of the district-level administrations visited. Another lesson is that simple technologies work best in poor rural communities. The introduction of complex technology into poor farming areas is risky and unlikely to succeed because the maintenance of complex systems is almost always unaffordable for poor smallholder farmers. The World Bank’s ICR rated overall development outcome moderately satisfactory. IEG’s fi ld-based assessment rated development outcome moderately unsatisfactory. Prior Actions for Mozambique’s PRSCs 3–5 were aligned with the challenges facing Mozambique’s economy and were complementary to the concurrent support provided by the IMF, but there was an overreliance on PFM reforms to achieve the broader macroeconomic objective, to the exclusion of indicators of macroeconomic stability. Other activities in relation to governance, were well aligned. However, program design could have paid more attention to the urban/rural distribution of pro-poor expenditure from the national budget, as well as the quality of these expenditures. Macroeconomic performance improved during implementation of the PRSCs 3–5. There was strong GDP growth, and the current account balance of payments deficit declined. After PRSC 5 closed, however, public debt increased to about 40 percent of GDP between 2010 and 2013, and then 242 to 74 percent of GDP in 2015. In addition, although there was a substantial allocation of budget resources to priority sectors, there was no information on the distribution of these funds to the areas where the poor are located. Moreover, the government acknowledged in April 2016 that it had guaranteed undisclosed borrowings of about $1 billion by state enterprises which contravened the program of macroeconomic management agreed with the IMF. The guarantee represented a serious contingent liability for the economy. The IMF suspended disbursements from its support program to Mozambique. There were only modest gains against the program’s decentralization and governance targets. More progress was made in the productive sectors. Growth in GDP from agriculture was between 5 and 6 percent per annum during the three-year implementation period, although it dropped to about 3.5 percent per annum between 2009 and 2014—still a sound and sustained performance given the sector’s structural problems and the incidence of natural disasters. The World Bank’s ICRs rated overall outcome satisfactory for each of the three credits. Following IEG’s field- based assessment, development outcome was rated moderately unsatisfactory for all three credits. Lagos is the largest city in Subsaharan Africa, but for many years has suffered from severe traf fic congestion and fragmented and uncoordinated public transport. The World Bank–supported Lagos Urban Transport Project sought to help the government of Nigeria improve transport services in the Lagos Metropolitan Area through management and road network improvements, enhanced bus services, and the promotion of water and nonmotorized transport. The emphasis was on sustainability and improving the quality of service for the poor. Central to the project was the establishment of a strong transport authority to coordinate and enforce planning and regulation, but the setting up phase took longer than anticipated. A crucial lesson was to learn that urban transport projects, because of their complexity, require longer periods for implementation. The creation of a transport fund was also an important step on the way to financial sustainability. A critical factor was to bring on board small operators and trade unions, and this was eventually achieved through study tours so that key players could observe successful transport models in other countries. The project team displayed considerable flexibility and used windfall savings to finance the design costs of the first phase of a rapid bus transit system, and follow-on projects are now supporting the expansion of this system. However, weaknesses included the failure to implement a results framework in the first few years of the project and lack of progress in attracting private sector investment in ferry services. Nonetheless, poor households reported time and cost savings and enhanced safety when using public transport. Overall, the World Bank ICR rated development outcome highly satisfactory. Following IEG’s field-based assessment, the project was rated moderately satisfactory reflecting the absence of a results framework and adequate indicators during the early stages of the project. The National Water Sector Reform Project 2004–10 project sought to increase access to piped water networks in selected urban areas in Nigeria, and to improve reliability and financial viability of selected urban water utilities, in Kaduna, Kano, and Ogun States of Nigeria. The project, which received $202 million of IDA assistance, also addressed the challenges facing urban water utilities including lack of cost recovery, maintenance, reliability, and low investment in rehabilitation and expansion. Independent Evaluation Group | World Bank Group 243 Targets for new household connections were initially exceeded by 40 percent, and partially achieved (80 percent) for rehabilitated connections, but both have declined in the years following project completion. Overall water production, which had grown from 250 to 865 million liters per year during the project, has since declined to 606 million liters per year. In addition, the urban population benefiting from the project had declined from 5.4 million to 4.1 million. Operations and maintenance coverage from revenues increased from an average of 20 percent across the three states to 60 percent at project completion, but dropped back to 20 percent by late 2016. The project saw two of five planned PPPs (implemented, though these did not involve major investment or risk transfer to the private sector, and did not sustainably advance reforms on a large scale). State water and sanitation policy legislation, which included the establishment of regulatory bodies, were drafted in all three states as planned, but they were not enacted by the time the project closed. Important lessons from the project are that necessary institutional restructuring and incentive mechanisms should be addressed up front in implementing reforms among multiple state and national stakeholders and that reliable and valid monitoring is crucial to improve utility performance and accountability to customers. The World Bank ICR rated overall development outcome moderately satisfactory, and IEG’s independent fi ld-based evaluation rated development outcome moderately unsatisfactory. At the start of the new millennium, malnutrition in Senegal was of great concern. Among children under five years of age, almost one-third (30 percent) were stunted (low height for age), 10 percent were wasted (low weight for height), and 20 percent were underweight (weight for age). IEG’s PPAR, A Decade of World Bank Support to Senegal’s Nutrition Program (2002–12), involved a field-based assessment of three World Bank projects in support of Senegal’s Nutrition program through three projects: (1) the Nutrition Enhancement Program, (2) the Nutrition Enhancement Project in Support of the Second Phase of the Nutrition Enhancement Program, and (3) the Rapid Response Child- Focused Social Cash Transfer and Nutrition Security Project. All three projects performed well in their intervention areas, culminating in greater knowledge, healthier behaviors and GPs, and improved nutrition outcomes. But, even though these intervention areas extended to all of Senegal’s 14 regions and 45 departments, they did not achieve suf ficient coverage of at-risk communities (within regions and departments) to effect major improvements in nutrition outcomes at the national level that were initially anticipated. The World Bank’s strategic and technical contributions to the Nutrition Enhancement Program were important, but the retraction of the World Bank’s financial commitment to provide substantial financial and technical support to three phases of the program was a factor in the failure to achieve a level of coverage that would have had a significant effect on nationwide trends. The financial constraints not only limited the number of intervention areas that the Nutrition Enhancement Program could support, they also affected the frequency and intensity of interventions. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that outcome ratings were highly satisfactory for two operations and satisfactory for the other one, based on increased capacity to develop, monitor, and oversee multisectoral nutrition activities. Senegal’s transport sector is of high strategic importance to its economy, yet urban mobility— particularly in the Greater Dakar Area (GDA)—has been a key impediment to growth. The country’s urban transport sector has been characterized by the government’s inability to meet rapid increases 244 in service demand severe traf fic congestion, poor and deteriorating road conditions, lack of traf fic management, an aging transport fl et, inadequate regulations, and serious gaps in maintenance. The World Bank’s Urban Mobility Improvement Project sought improvements in the safety, ef ficiency and quality of urban mobility in Greater Dakar Area, and road safety in Thies and Kaolack. The interventions aimed at achieving this objective included (i) investments in road rehabilitation, road safety and traf fic management; (ii) upgrading of pedestrian and traf fic safety along the suburban railway line; and (iii) the piloting of a minibus leasing scheme to finance fleet renewal. The overly ambitious project had a highly complex design but was inadequately prepared. It was assigned to a very weak implementing agency that had just been created. Its technical aspects did not adequately capture the continuing rapid growth in vehicular traf fic, which had the effect of canceling out the project’s benefi s. Although the minibus leasing scheme was successful and road safety improved, the road network continues to deteriorate rapidly due to lack of maintenance, scarcity of funding, and continued use by overloaded trucks. Railway-related outcomes were not achieved, and operations were in a worse state when the project closed. The project yielded lessons about the importance of having a competent lead agency backed up by strong and sustained government support, coordination of land use and transport planning at the metropolitan level, and the adequate funds for road maintenance. To enhance the success of minibus leasing schemes, a replicable lesson indicates the need for (i) bus operator inputs at the design stage; (ii) targeted technical assistance to professionalize bus operators and drivers; and (iii) the restructuring of the network of informal operators. Overall, the World Bank’s ICR rated development outcome moderately unsatisfactory, and IEG’s independent field-based assessment rated development outcome unsatisfactory. Faced with very high rates of extreme poverty despite sustained growth, in 2000 the government of Tanzania set its policies to explicitly address income poverty and human capital deficiencies. Community mobilization, empowerment, decentralization, accountability, and improved governance became key policy themes and supported first by a pilot Tanzania Social Action Fund, a CDD approach. The Second Tanzania Social Action Fund, the subject of IEG’s PPAR, scaled up the program to the national level. A $215 million IDA credit financed infrastructure subprojects and public works to generate household-level earnings opportunities. It also introduced elements of a productive safety net that targeted the poorest in the community, including a voluntary group savings/investment mechanism, and grants for income earning opportunities. The achievements of the projects were modest, especially in empowering communities to request, implement, and monitor subprojects. The involvement in and ownership of the project were lower than expected and not much was known about further outcomes. The project invested heavily in health and education facilities, water and sanitation, and economic infrastructure such as markets. This resulted in better student/classroom ratios, more use of preventive care and better access to water and sanitation; however, it was unclear to what extent these investments impacted poverty and household well-being. The project did succeed in improving Independent Evaluation Group | World Bank Group 245 household livelihoods through public works and cash transfers. More than 20,000 savings- investment groups were created—with overwhelmingly female participation. Public works increased the living standards of beneficiary households, including their chance of taking three meals per day. The income generating activities allowed households to increase their livestock and asset holdings substantially, and to generate earnings from them. The conditional cash transfer program increased enrollment in the health insurance scheme and health status of the beneficiaries. It also increased primary school completion. Because the benefits of the projects were obvious only in relation of one of its components and only in the second phase of the project (after restructuring) IEG rated the project moderately unsatisfactory, diverging from the World Bank’s ICR satisfactory assessment of development outcome. The Zambia Public Sector Management Program Support Project aimed to make public service delivery institutions more effective and ef ficient through strengthening PFM and reforming the civil service. The project contributed to some important outputs with regards to the objective of improved public expenditure management. A macrofiscal forecasting and debt management system was set up and successfully implemented. A Financial Management Information System was rolled out across government, assuring system integrity and increasing the speed of transactions. Advanced budgeting and planning modalities were institutionalized. The effectiveness of the public service benefited from upstream investments leading to improved payment and control systems, and an important pay policy was developed and approved. Efforts to introduce performance-based pay were unsuccessful due to insuf ficient staff qualification, staff shortages, and moonlighting in critical service delivery sectors. Lessons from the project were that good analytical work is an important underpinning for program design, but it is not suf ficient, since program design must be strategic, practical, prioritized, and account for the implementation and absorptive capacity of the borrower. Also, project design based on a complex institutional reform agenda would benefit from explicitly recognizing the iterative nature of the reform process. World Bank performance suffered from poor quality at entry, especially due to excessive complexity that made the project dif ficult to implement and manage. Both the World Bank’s ICR and IEG’s independent field-based assessment rated overall development outcome moderately unsatisfactory, reflecting modest design and mixed ef ficacy. In 2015, 90 percent of Zambia’s urban residents had access to improved water supply, but only 41 percent had connections in their house or yard, while 49 percent relied on water kiosks or standpipes. Access to improved sanitation was only 56 percent in urban areas. The Water Sector Performance Improvement Project (2006–13) provided $23 million in IDA assistance, including a $10 million grant for two original objectives—improving access and sustainability of water supply and sanitation (WSS) services in Lusaka and developing a comprehensive institutional structure supporting a coordinated approach to WSS investments—and the two revised objectives of improving the technical ef ficiency and financial sustainability of Lusaka Water and Sanitation Company and improving access to WSS services for urban consumers in Lusaka, Kafue, Chongwe, and Luangwa districts; and strengthening the effectiveness of national WSS planning. Under the 246 project, about 5,000 urban residents were provided access to improved water supply for the first time, and overall water availability increased from 210,000 to 230,000 cubic meters (per day). However, IEG’s focus group discussions with selected beneficiaries indicated reduced levels of satisfaction regarding the adequacy, reliability, and quality of WSS services. The planning capacity of the sector ministry remains weak. The ministry has no clear mechanism to coordinate with or regulate the city councils that own and manage the WSS utilities. Among the lessons learned from this project are that sustainable provision of water supply services in the face of growing population and demand requires active coordination between the authorities responsible for long-term water resource planning, and service providers. Alternative approaches need to be actively explored and adopted for urban and rural sanitation, especially when conventional sewerage may not be financially feasible. The World Bank’s ICR rated overall development outcome moderately satisfactory, and IEG’s independent field-based assessment rated development outcome moderately unsatisfactory. The World Bank’s ICR rated overall development outcome moderately satisfactory; IEG’s independent fi ld-based assessment rated development outcome moderately unsatisfactory. Evaluation Findings from Country Program Performance During FY17, IEG produced CLRRs for six countries in the Africa Region, namely, Cameroon, Lesotho, Madagascar, Mauritius, Mozambique, and Togo. For development outcome, IEG rated two programs moderately satisfactory, one moderately unsatisfactory, and two unsatisfactory. No rating was provided for Togo due to data limitations. The Cameroon—Country Partnership Strategy for the Period FY10–14 for the focused on governance, competitiveness, and public sector services. IEG’s CLRR found the country partnership strategy (CPS) realistically designed for gradual and quite modest improvements. However, and critically in terms of substantiating outcomes, not enough attention was given up front to whether the results could be measured. Under the program, governance was identified as Cameroon’s greatest challenge to growth and development, as the pervasiveness of weak governance affected most economic transactions. However, IEG’s review found very modest progress evident for this theme—governance indicators overrelied on Public Expenditure and Financial Accountability indicators that could not be verified because the government postponed the planned Public Expenditure and Financial Accountability, which was also true of other indicators across the results framework. Other objectives performed somewhat better in terms of achieving targets, for example, access to infrastructure and health services. The Kribi Gas to Power project provided 216 megawatts of electricity through an IDA guarantee project, IFC lending and later a Multilateral Investment Guarantee Agency guarantee, which was indicative of progressive engagement across the institutions. Average transit times for imports from Douala port were reduced, days to register a business declined substantially (although less than planned), and there were some improvements in the use of natural resources. Independent Evaluation Group | World Bank Group 247 The completion rate in primary education increased from 64 percent to 83 percent over the program period, but there were no indicators to measure the quality of education. Knowledge activities provided good support for the preparation of lending operations. The progress report addressed an important stepping-up of supervision and implementation support, and a stronger focus on a few selected operations. World Bank lending, including regional operations, was significantly above program expectations. IFC focused on development and support of the electricity (involving multiple institutions) and financial sector (supporting two microfinance institutions and after the period, helping develop the small and medium enterprise (SME) portfolio in a leading bank). The Completion and Learning Report (CLR) identifies lessons related, for example, to the quality of the results framework, complementing centralized approaches to strengthening governance with decentralized and sector-based approaches, and the need to enhance the focus more directly on poverty reduction and spatial inequality. IEG draws three main lessons: First, programs addressing governance need to provide a mix of interventions commensurate with the nature of the objectives, be structured realistically to conditions in the field and World Bank instruments. Second, indicators need to be designed keeping in mind the ability to monitor progress and to measure and assess results. Third, World Bank country program documents including CLRs need to pay clear attention where there are (as in the case of Cameroon) significant indications of broader underlying fiduciary and governance issues. The CLR rated overall development outcome moderately unsatisfactory, and IEG’s review rated overall development outcome unsatisfactory. The country assistance strategy (CAS) for Lesotho (FY10–14) had three focus areas (fiscal adjustment and public sector ef ficiency, human development and improved service delivery, and enhanced competitiveness and diversification) in line with the government’s priorities. IEG’s CLRR found that the program was successful in improving the quality of national roads and in expanding the mandate of the Lesotho Electricity Authority toward improving regulation of infrastructure services. There was substantial progress in increasing annual visits to the new referral hospital and filter clinics (albeit at great cost), and in connecting targeted population to piped water supply. Targets were achieved in streamlining business procedures at the project level, but the overall ranking of Lesotho in Doing Business dropped by 4 points in 2016. However, there was limited progress in trade facilitation and export diversification. Although progress was noted in the institutional capacity of the Bureau of Statistics, the link between this indicator and better service delivery is tenuous. The envisaged outcomes of fiscal consolidation, reduction in wage bill, and strengthening PFM were not achieved. Finally, the reported achievement of other program objectives (for example, reverse trend in health) could not be verified in certain instances. In other cases, the link between developments and World Bank interventions is weak. IEG noted that the original program was ambitious in scope and coverage, and noted a lack of selectivity for a country with a small lending envelope and limited institutional capacity. The results framework had significant shortcomings, including weak links between outcome indicators and World Bank interventions (for example, human immunodeficiency virus / acquired 248 immune deficiency syndrome and secondary education) and the output oriented nature of some of the outcome indicators. During the CAS progress report, refinements were made in the results framework, but, overall, this did not change the quality and measurability of indicators. Some key pieces of analytical work were undertaken, such as the Public Expenditure Review, although other, equally important planned work on governance and capacity development to anchor all analytical work did not materialize. Planned analytical work on poverty and gender assessment was not undertaken during the CAS period. While there was strategic engagement between the World Bank and IFC through public-private partnership (PPP) in health (Lesotho Hospital Project), there was clearly room for improving collaboration at implementation and understanding of shared results. IEG’s review provides lessons including the following: introducing innovative approaches such as PPPs and performance-based financing in a weak institutional capacity environment requires enhanced and coordinated risk assessment and identification of risk mitigation - in the case of the Lesotho Hospital Project, the full extent of the risk of fiscal burden and sustainability was not considered at the outset; in an environment where the World Bank is not a major player, but contributes to a bigger undertaking involving other donors, it is critical to set program objectives and indicators that can be clearly linked to the World Bank’s interventions and to explicitly discuss the contribution of other donors to the program objectives to ensure proper attribution and set the ambition of the program accordingly. IEG rated overall outcome at program level, moderately unsatisfactory. IEG’s CLRR for Madagascar covered both the CAS (FY07–11) and the interim strategy note (ISN, FY12–13). Both had three focus areas: (i) governance and public sector capacity; (ii) employment and competitiveness; and (iii) vulnerability and resilience. The areas of engagement under the CAS were well aligned with government priorities. Both the CAS and ISN recognized the significant risk to program implementation from both external risks (Madagascar’s vulnerability to economic and natural shocks), and internal risks (fragile macroeconomic environment, possible political instability, and capacity constraints). As it happened, the implementation of the CAS or ISN was affected negatively by political instability, and an associated temporary freeze in disbursements from March 2009 till May 2011. IEG concluded that, of the total 31 objectives, 13 were not achieved, another 7 were rated partially achieved, and 2 were rated as not verified by IEG. The three main lessons drawn by IEG are: (i) the importance of being realistic and focused about what can be accomplished through lending, especially when the resolution to a political crisis is not in sight; (ii) a stronger understanding of the political economy environment is essential in volatile situations; and (iii) flexible and cost-effective delivery modalities, such as community-driven approaches, are crucial to sustain results in the field in a fragile context. IEG rates overall World Bank Group performance as poor as the CAS lacked selectivity in terms of the number of objectives and outcomes articulated, and was not adjusted in line with the contraction of the World Bank’s lending portfolio due to project cancelations following the portfolio-wide restructuring. IEG rates overall development outcome of the CAS/ISN unsatisfactory. The World Bank Group country-level engagement in Mauritius FY07–15 focused on increasing competitiveness while protecting the vulnerable. Sound economic management had helped the Independent Evaluation Group | World Bank Group 249 country successfully weather several adverse external developments, but could not prevent a decline in GDP growth rates in recent years. Both the government and the World Bank agreed that the lower growth rate threatened earlier achievements in poverty reduction and human development that had been driven by economic growth. The CPS was organized around the government’s four focus areas (or pillars) of reform: fiscal consolidation and improved public sector ef ficiency; improving trade competitiveness; improving the investment climate; and democratizing the economy through participation, social inclusion and sustainability. At first, there was good synergy between the government reform plans and the CPS, but a 2010 change in government slowed implementation of the reform agenda. IEG finds that the World Bank Group was slow to recognize this and program alignment suffered accordingly. Few advances were made in infrastructure development, while little progress was made toward realizing environmental objectives and improvements in sanitation or health. On the other hand, there was substantial progress in fiscal management, in reducing the anti-export bias of the trade regime, and in improving the business environment. IEG rates World Bank Group performance fair given the missed opportunity at the performance and learning review (PLR) stage to consolidate and retrofit the results matrix, which lacked quantitative baselines and targets, with sometimes tenuous links between World Bank Group action and outcomes. In addition, critical constraints to development identified in the analytical work and included in the CPS and PLR (that is, infrastructure and primary education) were not pursued in program implementation. PLR lessons emphasize the importance of maintaining a continuous and open dialogue with government as well as good understanding of the political economy of the country, and the need for a base of solid analytical work to help engage all levels of government and to build commitment. IEG underscores the need for the CPS to further engage in policy dialogue through programmatic analytical and advisory services, and identifies the following lessons: the absence of clearly specified and verifiable indicators with baselines and targets in the results matrix makes it very dif ficult to assess or validate achievement of objectives; and despite challenges, the World Bank’s comparative advantage in relation to PPPs should continue to be leveraged in Mauritius. Both the World Bank’s closure report and IEG’s review rated overall development outcome of the CPS as moderately satisfactory. The CPS for Mozambique covered the period FY12–15 and developments in FY16. The program focused on three areas relevant to government strategy: competitiveness and employment, vulnerability and resilience, governance and public sector capacity. IEG’s review found the World Bank Group mostly chose appropriate interventions to deliver the program and to achieve the objectives. Program design had three shortcomings: a fragmented results matrix, with 17 objectives and 30 outcome indicators; some indicators were inappropriate to establish achievement of objectives; and IFC interventions were overlooked in the results matrix. During implementation, the World Bank Group increased its reliance on the country’s financial management systems at state and municipal levels. All the risks identified in the CPS materialized; as donor aid fell, the World Bank stepped up its financial support, but the World Bank did not undertake a PLR, missing the opportunity to correct program course at midterm. Ten of the program’s 17 250 objectives were achieved or mostly achieved (regulatory environment, employment and growth in targeted areas of the tourism sector, road infrastructure, water and sanitation, access to electricity, affordable telecommunications, education and skills of the workforce, improved health services for the vulnerable, improved capacity of local administrations to manage public finances, better conservation of wildlife, and more transparency in extractive industries). The objective of raising agricultural productivity was partially achieved, but objectives in spatial planning, access to electricity, adaptation to climate change, social protection, PFM, and public service monitoring, were not achieved. IEG identified the following findings and lessons noting the World Bank Group must lead on substantive issues; consolidate and integrate the portfolio better; strengthen the political economy and institutional assessments to inform project design and implementation; focus on few implementable interventions; and align objectives and interventions, and improve the selectivity of its interventions. IEG rated overall outcome moderately satisfactory. IEG validated the CLR for the World Bank Group (World Bank engagement in Togo FY08–17 covering ISNs for the periods FY08–10 and FY12–13), as well as activity that continued until FY17 when the first Country Partnership Framework (CPF) for Togo was prepared. The first ISN had the overarching goal to help Togo recover from a long period of instability and suspension of external aid, and was aligned with the Poverty Reduction Strategy Papers adopted in 2008 and 2009 (covering 2009–11). Given continuing uncertainty at the end of the first ISN period, the World Bank Group prepared a second ISN rather than a CPS. The second ISN) updated the first and proposed to use the two-year implementation period to consolidate Togo’s knowledge base and help the country increase its aid absorption capacity. In August 2013, the government adopted a poverty reduction strategy, a Strategy for Boosting Growth and Promoting Employment, which covered 2013–17, but a CPS was not prepared. During the four- year gap before the preparation of a CPF, the World Bank Group continued to approve operations, undertake Analytical and Advisory Assistance (AAA) and work with other development partners, although it is not apparent from the CLR that the FY12 ISN was formally extended. IEG found that program objectives were not selective, especially when account is taken of the respective two-year frameworks, the small IDA envelope, the country’s limited implementation capacity, and the fragile political economy. The World Bank Group program had nine objectives under three focus areas being, and available information suggests uneven progress in the 2008–16 review period. For example, the initial privatization agenda proved ambitious, and key segments of the economy (including telecommunications, phosphates, utilities, and some banks) remain under state control; the intended gains in PPPs lag initial expectations; and whereas gains in primary education completion rates were achieved, and CDD engagements were effective, social protection initiatives remain at a pilot level. From FY08 to FY16, 18 World Bank operations were approved for a total of $436.3 million, of which six were DPOs for $ 267.3 million. Grants amounted to $147 million, with the bulk ($100 million) Independent Evaluation Group | World Bank Group 251 going to education and agriculture. A considerable body of analytical work has now been produced, including the 2016 Systematic Country Diagnostic. Given the limited impact of economic growth on poverty reduction, earlier attention to the drivers of poverty reduction would have helped to fine-tune the programs by sharpening the focus on the most critical priorities for poverty reduction. Strong partnerships have been built with the government and the development community. The ISNs identified risks to the program but the mitigating measures were not suf ficient to contain fiscal pressures that led to increases in the level of public debt. IFC investment helped mobilize private sector resources and resulted in improved infrastructure services and job creation. The CLR identifies lessons regarding the importance of political economy analysis and of taking a long-term approach in the context of political and economic volatility, as well as the importance of selectivity (IEG further emphasizes the need for selectivity in practice rather than simply in principle), realism (IEG further emphasizes the need for simplicity of design in challenging contexts), and intensive supervision. IEG adds two lessons including reference to the challenges that arise from operating through sequential ISNs. In the case of Togo, delays in moving to a full CPS or CPF risked leaving a sizeable program without an adequate supporting results framework. IEG also concurs with the finding of the Compliance Adviser / Ombudsman that IFC’s disclosure and consultation with affected communities in infrastructure projects should be undertaken in a manner that is inclusive and culturally appropriate. In line with the CLR, IEG does not rate the overall development outcome and the World Bank Group’s performance due to data limitations. IEG also produced a Small States Cluster Country Program Evaluation that included coverage of several small states (defined as those that have a population of less than 1.5 million) in the Region, namely Mauritius, the Seychelles, Cabo Verde, and Djibouti (as well as the Pacific Island States, and the small states included under the umbrella of the Organization for Eastern Caribbean States). Though small states differ widely, they share several challenges, including those of strengthening their resilience to economic and natural disaster-related shocks and of remaining competitive to generate vigorous economic growth. Among the strategic and operational approaches that the World Bank Group used to respond to the more constraining parameters that small state country teams confront, were engaging on a regional or multicountry level, addressing country capacity limitations, and using partnerships to engage jointly or in a coordinated way with others. Thematically, the small state programs under review deployed several approaches, and addressed key development issues geared to the client countries’ small size, seeking to strengthen macroeconomic, disaster and climate, and social resilience, as well as enhance competitiveness by facilitating trade, strengthening the financial sector, expanding infrastructure and improving its management, and supporting leading sectors (such as tourism, fisheries, and agriculture). Under the resilience pillar, the evaluations suggest that support for strengthening macroeconomic resilience needs a multifaceted approach to be effective. The support needs to focus on the multiple drivers of fiscal and debt sustainability. The approach to disaster risk management (DRM) of 252 directly supporting resilience building (notably for infrastructure) was helpful but had limits. Making real improvements in vulnerability requires continuing efforts to foster wholesale changes in public and private incentives and behavior, and the long-term risks of climate change (including helping countries access climate financing) need to be addressed. Important focus areas for social resilience include preventing and managing noncommunicable diseases, strengthening social protection, and enhancing mobility to larger countries’ labor markets (as the World Bank helped to do for the Pacific Island Countries (PICs) with Australia and New Zealand). Under the competitiveness pillar, a sharper focus is needed on the most binding business constraints, using lenses specific to the leading sectors—typically fisheries, tourism, and agriculture. Several programs have started such an approach. In the financial sector, key issues for attention include system soundness and risks (particularly in the OECS), access to finance for SMEs, and remittance costs. Finally, it is important for infrastructure work to continue focusing on separating regulatory functions from service delivery functions. Despite the pitfalls, a strong case can be made for continuing to support PPP in infrastructure service delivery and regional service provision and regulatory initiatives. The program evaluations also raise issues for World Bank Group consideration in its engagement with small states—relating to the cross-cutting aspects at both at the country team and institutional levels. For example, at the country team level, contingency planning, including the use of contingent financing instruments, has a key role to play in enhancing World Bank responsiveness to the shocks that small states suffer. At the institutional level, issues that merit consideration include whether small state IDA allocations can be made more responsive to vulnerability, and whether small state–specific World Bank and IFC financing instruments are worth developing. East Asia and Pacific Region Project- and Country-Level Performance This note presents the results and performance separately for the World Bank, IFC, and MIGA given the different time frames, operating models, and clients of the three institutions. The World Bank: Project-Level Performance For the current review period, FY14–16, the development outcome rating for 83 percent of projects in the East Asia and Pacific Region was MS+, improving from a rating of 67 percent MS+ in FY11–13 and exceeding the outcome rating for all other Regions over the latest period (71 percent MS+: figure L.15, panel a). By commitment, the rating was 89 percent MS+ for the East Asia and Pacific Region during FY14–16, compared with an average 83 percent MS+ rating across all other Regions (figure L.15, panel b). Independent Evaluation Group | World Bank Group 253 Figure L.15. East Asia and Pacific: Project Outcome Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: = East Asia and Pacific; MS+ = a rating of moderately satisfactory or better. This 89 percent of commitments to projects rated MS+ also represents a notable improvement from 74 percent MS+ in FY11–13. Overall World Bank performance, a composite of ratings for quality at entry and project supervision, was 83 percent MS+ in the East Asia and Pacific Region during FY14–16, compared with a 70 percent MS+ average rating across all other Regions (figure L.16, panel a). Within the overall World Bank performance rating for the East Asia and Pacific Region, quality at entry was rated 69 percent MS+, compared with a 59 percent MS+ average for quality at entry across the other Regions. 254 Figure L.16. East Asia and Pacific: World Bank Performance Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: EAP = East Asia and Pacific; MS+ = a rating of moderately satisfactory or better. Supervision was rated 87 percent MS+ in the East Asia and Pacific Region, compared with an average rating of 76 percent MS+ across the other Regions. The World Bank performance rating improved between FY11–13 and FY14–16, by both number and commitment (from 68 percent MS+ to 83 percent MS+, and from 75 percent MS+ to 88 percent MS+, respectively; figure L.16, panels a and b). The overall quality of M&E for projects in the East Asia and Pacific Region during FY14–16, was 41 percent S+, compared with an average 33 percent S+ rating across all other Regions and Independent Evaluation Group | World Bank Group 255 Figure L.17. East Asia and Pacific: M&E Quality Ratings, FY05–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: EAP = East Asia and Pacific; M&E = monitoring and evaluation; S+ = a rating of substantial or better. 34 percent substantial rating World Bank–wide (figure L.17, panel a). By number of projects, the M&E rating improved from between FY11–13 (28 percent S+) and FY14–16 (41 percent S+). M&E rating by commitment volume is also strong for the East Asia and Pacific Region (55 percent S+) when compared with the average for all other Regions (43 percent S+) and World Bank–wide (46 percent S+: figure L.17, panel b). By commitment volume, the M&E rating improved between FY11–13 (26 percent S+) and FY14–16 (55 percent S+). The development outcome rating for 52 IBRD-supported projects rated for the East Asia and Pacific Region during FY14–16 is 87 percent MS+, compared with an average 72 percent MS+ rating across 256 Asia and Pacific: Project Outcome Ratings by Figure L.18. East Financing Source, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: EAP = East Asia and Pacific; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; MS+ = a rating of moderately satisfactory or better. Asia and Pacific: World Bank Performance Ratings Figure L.19. East by Financing Source, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: EAP = East Asia and Pacific; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 257 all other Regions (figure L.18, panel a). By commitment ($10.67 billion), the rating was 86 percent MS+, compared with an average 85 percent MS+ rating across all other Regions (figure L.18, panel b). The development outcome rating for 51 IDA-supported projects rated for the East Asia and Pacific Region during FY14–16 is 84 percent MS+, compared with 71 percent MS+ for IDA-supported projects across all other Regions FY14–16 (figure L.18, panel a). By commitment volume, the rating was 96 percent MS+, compared with 79 percent MS+ across all other Regions (figure L.18, panel b). World Bank performance for IBRD-supported projects in the East Asia and Pacific Region during FY14–16 (87 percent MS+) is higher than the average across all other Regions (72 percent MS+: figure L.19, panel a), as is M&E quality (44 percent S+ versus 34 percent S+: figure L.20, panel a). World Bank performance for IDA-supported projects (82 percent MS+) and M&E quality (39 percent S+) ratings are higher than the respective ratings for IDA-supported projects across all other Regions (70 percent MS+ and 31 percent S+: figure L.19, panel a, and Figure L.20, panel a). Several individual GPs performed strongly in the East Asia and Pacific Region. The outcome rating for the largest single practice (by number), Social, Urban, Rural, and Resilience (SURR; 26 projects) was 92 percent MS+, followed by Agriculture (11 projects) at 91 percent MS+, Education (13 projects) at 85 percent MS+, and Transport and ICT (18 projects) rated at 83 percent MS+ for outcome. The Governance (10 projects) and the HNP (9 projects) GPs performed less well, with respective ratings of 60 percent MS+ and 67 percent MS+ for outcome (figure L.21, panel a). Asia and Pacific: M&E Quality Ratings by Financing Figure L.20. East Source, FY14–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: EAP = East Asia and Pacific; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; M&E = monitoring and evaluation; S+ = a rating of substantial or better. 258 Asia and Pacific: Development Outcome by Global Figure L.21. East Practice, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: “Other” includes countries without International Development Association or International Bank for Reconstruction and Development status or project locations that are above country level. EAP = East Asia and Pacific; ICT = Information and Communication Technology; MS+ = a rating of moderately satisfactory or better. The two largest GPs (SURR and Transport and ICT) in terms of commitment comprised almost half of all commitments among projects rated for the East Asia and Pacific Region during FY14–16. Outcome by commitment was strong for both, with 95 percent and 84 percent MS+, respectively (figure L.21, panel b). Independent Evaluation Group | World Bank Group 259 World Bank performance was strong for several GPs in the East Asia and Pacific Region including Finance and Markets, Macroeconomics and Fiscal Management, and SPL with 100 percent of projects rated MS+, followed by Energy and Extractives (93 percent MS+), SURR (92 percent MS+), Agriculture (91 percent MS+), and Transport and ICT (83 percent MS+), but less so for others, for example, Governance (50 percent MS+) and HNP (67 percent MS+). The M&E rating for the Region (41 percent S+) was relatively strong compared with the average for other Regions (33 percent S+). Individual GPs, including Education (77 percent S+), Macroeconomic and Fiscal Management (63 percent S+), HNP (56 percent S+), and SPL, Finance and Markets, and Energy and Extractives (all 50 percent S+) performed particularly well. The SURR (38 percent S+) and Transport and ICT (22 percent S+) GPs that performed strongly for development outcome and World Bank performance did less well for M&E. International Finance Corporation: Project-Level Performance IFC Investments Project-Level Results For the CY14–16 XPSR program year, IEG assessed 28 investment projects with an equivalent $643 million IFC’s net commitment in the East Asia and Pacific Region. Nearly half (46 percent by number of projects and volume amount) of the projects had ratings of MS+ for achieving their intended development outcomes. Compared with CY11–13 results, the decline in success ratings of recently evaluated projects in the region is rather stark, especially if weighted by volume amount. Figure L.22 shows a similar downward trend of the satisfactory or better (S+) ratings for IFC’s investment outcomes and IFC’s role and contribution. The 44-percentage-point decline in S+ ratings for IFC’s investment outcome in CY14–16 compared with CY11–13 has been particularly steep and may call for additional review. Coincidentally, 13 of the 28 projects (equivalent to 37 percent, by volume) evaluated in CY14–16 are in China. Of these 13 evaluated projects in CY14–16, only four had MS+ development outcome ratings and four projects had positive (S+) for IFC investment outcome ratings. Results in three of the four subindicators of development outcome declined sharply in CY14–16 compared with the CY11–13, by number of projects, while the positive ratings for the subindicator E&S effects remained the same. Only two-fifths of the evaluated investment projects in the region met their business performance benchmarks while nearly half of the projects met their economic sustainability objectives and benchmarks and contributed to private sector growth in their respective countries or in the region. Positive ratings for all four development outcome subindicators in CY14–16, by number of projects and by volume, declined sharply compared with CY11–13 ratings, especially private sector development (figure L.23). A review of the development outcome, IFC work quality and IFC’s role and contribution ratings, which fell sharply in CY14–16, indicates strong association between these three indicators of IFC results and performance. Figure L.24 shows strong association between high (MS+) development outcome ratings with high (S+) IFC work quality ratings, and the same pattern is shown for projects with low development outcome and low IFC work quality ratings, especially by number of projects. A similar 260 Figure L.22. East Asia and Pacific: Results and Performance Ratings of Evaluated IFC Investment Projects Share of Projects with Positive Ratings a. By number of projects, three-year moving averages b. By IFC net commitment, three-year moving averages Source: Independent Evaluation Group Evaluation Data for IFC XPSRs as of December 12, 2017. Note: (1) Positive ratings for development outcome consists of mostly successful, successful, or highly successful (aggregated as MS+) ratings. Positive ratings for investment outcome and IFC’s role and contribution consist of excellent or satisfactory (aggregated as S+) ratings. (2) Due to the revision of the XPSR guidelines in 2015, the indicator, IFC’s role and contribution, has been amended starting CY15, and was replaced with the indicator, IFC additionality. relationship is indicated between development outcome ratings and IFC’s role and contribution and investment outcome ratings, by number of projects. If the ratings are weighted by the amount of IFC expenditures, the association is more tenuous between IFC overall work quality and investment outcome ratings.A review of the development outcome, IFC work quality and IFC’s role and contribution ratings, which fell sharply in CY14–16, indicates strong association between these three indicators of IFC results and performance. Figure L.24 shows strong association between high (MS+) Independent Evaluation Group | World Bank Group 261 Figure L.23. East Asia and Pacific: Performance of Development Outcome Subindicators of IFC Investment Projects, by XPSR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving averages b. By IFC net commitment, three-year moving averages Source: Independent Evaluation Group Evaluation Data for IFC Investment Projects as of December 12, 2017. Note: (1) S+ = a rating of Satisfactory or Excellent. (2) PBS = Project Business Success. ECSUS = Economic Sustainability. E&SE = Environmental and Social Effects. PSD = Private Sector Development. These four subindicators comprise the development outcome dimension of an IFC investment project. (3) No Opinion Possible or Not Applicable ratings were excluded. development outcome ratings with high (S+) IFC work quality ratings, and the same pattern is shown for projects with low development outcome and low IFC work quality ratings, especially by number of projects. A similar relationship is indicated between development outcome ratings and IFC’s role and contribution and investment outcome ratings, by number of projects. If the ratings are weighted by the amount of IFC expenditures, the association is more tenuous between IFC overall work quality and investment outcome ratings. Ratings of the two subindicators of IFC work quality point to weak performance in CY14–16 compared with CY11–13 ratings especially IFC’s screening, appraisal, and structuring (SAS) of projects, whether measured by number of projects or by the amount of IFC net commitment (figure L.25). Although positive ratings for IFC’s SUP improved slightly in CY14–16 by number of 262 Asia and Pacific: Association between Development Figure L.24. East Outcome, IFC Work Quality and IFC’s Role and Contribution, CY14–16 By number of evaluated projects By IFC net commitment ($, millions) a. Association between development outcome and IFC work quality b. Association between development outcome and IFC role and contribution c. Association between work quality and IFC investment outcome Source: Independent Evaluation Group evaluation data for IFC investment projects as of December 12, 2017. Note: (1) IFC work quality in CY14–16 reflects the aggregate ratings of two subindicators: screening, appraisal, and structuring and supervision and administration. (2) High development outcome = highly successful, successful, and mostly successful ratings; low development outcome = mostly unsuccessful, unsuccessful and highly unsuccessful ratings; high work quality = excellent and satisfactory ratings; low work quality = partly unsatisfactory and unsatisfactory ratings; high IFC role and contribution = excellent and satisfactory ratings; low IFC role and contribution = partly unsatisfactory ratings; high IFC investment outcome = excellent and satisfactory ratings; low IFC investment outcome = partly unsatisfactory and unsatisfactory ratings. (3) Due to the revision of the Expanded Project Supervision Report guidelines in 2015, the indicator IFC’s role and contribution has been amended starting CY15, and was replaced with the indicator IFC additionality. IFC = International Finance Corporation. Independent Evaluation Group | World Bank Group 263 Figure L.25. East Asia and Pacific: IFC Work Quality Subindicators, by XPSR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC net commitment, three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC Investment Projects as of December 12, 2017. Note: Based on the 2015 revised XPSR guidelines, IFC work quality is measured by two indicators: screening, appraisal, and structuring (SAS) and supervision and administration (SUP) in the project. IFC = International Finance Corporation; XPSR = Expanded Project Supervision Report. projects, the result is different if weighted by volume. Two large projects accounted for the decline in positive ratings for the subindicator supervision and administration, by commitment amount. The two projects accounted for nearly 30 percent of the total volume of evaluated projects in CY14–16 and had major shortcomings with IFC’s supervision and administration. IFC Advisory Services Project-Level Results Of the 20 IFC Advisory Services projects in the East Asia and Pacific Region evaluated in the FY14–16 PCR Program, nearly half (45 percent) had a rating of MS+ for overall development effectiveness. This development effectiveness success rating is on par with the development effectiveness rating when IFC and IEG started evaluating IFC advisory projects in FY08 (figure L.26). A similar downward trend 264 Figure L.26. East Asia and Pacific: Results and Performance Ratings of Evaluated IFC Advisory Projects, by PCR Program Years Share of Projects with Positive Ratings a. By number of projects, three-year moving average b. By IFC expenditure ($, millions), three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC PCRs as of December 12, 2017. Note: (1) The reported ratings are based on IEG’s final ratings in the completed Project Completion Reports (PCRs). (2) Positive ratings for development effectiveness consists highly successful, successful, or mostly successful (aggregated as MS+) ratings. Positive ratings for IFC’s role and contribution consist of excellent or satisfactory (aggregated as S+) ratings. is illustrated in Figure L.26 regarding the share of projects rated satisfactory or better (S+) for IFC’s role and contribution, both by number of projects evaluated and by volume. Among the East Asia and Pacific advisory projects evaluated in FY14–16, the development effectiveness subindicator, outcome achievement seems to influence the aggregate development Independent Evaluation Group | World Bank Group 265 Figure L.27. East Asia and Pacific: Ratings of Four Indicators of Development Effectiveness of Evaluated IFC Advisory Projects, by PCR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC expenditure ($, millions), three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC PCRs as of December 12, 2017. Note: Success ratings for the development effectiveness subindicators = excellent or satisfactory (referred to as S+) ratings. The reported ratings are based on IEG’s final ratings in the completed Project Completion Reports (PCRs). effectiveness rating (figure L.27). Compared with FY11–13 results, positive ratings for outcome achievement in FY14–16 weakened, both by number and volume of evaluated projects. In fact, success (S+) ratings during this review period for outcome achievement reached its lowest point (48 percent) since the PCR Program started in FY08. Ratings for output achievement also declined indicating shortcomings in the delivery of expected project outputs (for example, a diagnostic report, delivery of training program or drafting of laws and regulations, and so on.) to IFC’s clients. While the share of S+ ratings for strategic relevance is about the same, ratings for the ef ficiency in delivering the expected outputs have also declined in FY14–16 compared with FY11–13. 266 Multilateral Investment Guarantee Agency: Project-Level Performance Nine MIGA projects in the East Asia and Pacific Region were evaluated in FY11–16 and five had ratings of satisfactory or better (S+) for development outcome. Among the four subindicators of development outcome, positive rating was weakest for project business performance with four of nine evaluated projects rated satisfactory or better. Seven of the nine projects were rated S+ for MIGA’s role and contribution in the projects. However, only five of the nine projects had S+ ratings for the quality of MIGA’s assessment, underwriting and monitoring. World Bank Group: Country-Level Performance Projects were rated for 11 IDA and 5 IBRD countries in the East Asia and Pacific Region during FY14– 16. Within the IDA complement, 6 countries were designated fragile. By number, outcome ratings are close to the average for the Region (83 percent MS+) for projects in IBRD and IDA nonfragile country types but well below that average for IDA fragile projects (69 percent MS+). On the other hand, by volume, IDA fragile (94 percent MS+) performs above the average (89 percent MS+) and above both IBRD and IDA nonfragile for the Region (table L.3). During FY17, IEG completed CLRRs for eight countries in the Region, including the Lao People’s Democratic Republic (PDR), Vietnam, and six Pacific Island States, namely: Kiribati, the Marshall Islands, the Federated States of Micronesia, Samoa, Tonga, and Tuvalu. IEG rated development Table L.3. East Asia and Pacific: Project Outcome Ratings by Country Type, FY14–16 Countries Projects Volume Evaluated Rated MS+ Total Amount Rated MS+ Country Type (no.) No. (percent) ($, millions) (percent) IDA fragile 6 13 69 515 94 IDA nonfragile 5 26 85 442 89 IBRD fragile 0 0 n.a. 0 n.a. IBRD nonfragile 5 94 85 14,070 88 IDA total 11 39 79 957 92 IBRD total 5 94 85 14,070 88 Fragile total 6 13 69 515 94 Other n.a. 0 n.a. 0 n.a. Total 16 133 83 15,027 89 Source: Independent Evaluation Group data. Note: “Other” includes countries without IDA or IBRD status or project locations that are above country level. IBRD = International Bank for Reconstruction and Development; IDA =International Development Association; MS+ = a rating of moderately satisfactory or better; n.a. = not applicable. Independent Evaluation Group | World Bank Group 267 outcome satisfactory in one case, moderately satisfactory for two country programs, moderately unsatisfactory for three country programs, unsatisfactory for one country program, and in one case, development outcome was rated highly unsatisfactory (table L.4).2 Table L.4. East Asia and Pacific: Country-Level Performance Ratings, FY14–17 CASCR/ CLR Review World Bank IFC MIGA Review FY Country Period Outcome Performance Performance Performance 2014 Philippines FY10–13 MS Good Good n.a. 2016 Cambodia FY05–15 n.a. n.a. n.a. n.a. Indonesia FY13–15 MS Good n.a. n.a. 2017 Kiribati FY11–14 U Fair n.a. n.a. Lao, PDR FY12–16 MS Good n.a. n.a. Marshall Islands FY13–16 HU Fair n.a. n.a. Micronesia, Fed. Sts. FY14–17 MU Fair n.a. n.a. Samoa FY12–16 MU Good n.a. n.a. Tonga FY11–14 S Good n.a. n.a. Tuvalu FY12–15 MU Good n.a. n.a. Vietnam FY12–16 MS Good n.a. n.a. Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: All World Bank performance ratings were standardized with the following conversions: satisfactory as superior, moderately satisfactory as good, moderately unsatisfactory as fair, and unsatisfactory as poor. CASCR = country assistance strategy completion report; CLR = Completion and Learning Report; Fed. Sts. = Federated States; HU = highly unsatisfactory; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency; MS = moderately satisfactory; MU = moderately unsatisfactory; PDR = People’s Democratic Republic; S = satisfactory; U = unsatisfactory; n.a. = not applicable. Evaluation Findings from Project Performance Assessment Reports During FY17, IEG completed PPARs covering seven projects in the East Asia and Pacific Region. In terms of development outcome, IEG rated one project highly satisfactory, four satisfactory, one moderately unsatisfactory, and one unsatisfactory. The Public Financial Management and Accountability Project operated in Cambodia between June 29, 2007, and November 15, 2013. The project aimed to strengthen the mobilization and use of public resources, bolster accountability institutions, and develop the capacity of the civil service. It became effective shortly after the World Bank’s Integrity Vice Presidency found suf ficient evidence to substantiate allegations of fraud and corruption in seven unrelated projects. The World Bank ultimately suspended all lending to Cambodia in 2011 after the already tense relationship was further aggravated over a land settlement dispute. 268 Although Indonesia entered the global financial crisis in late 2008 in a sound economic position, structural weaknesses made it vulnerable to global financial crises, and the increased global risk aversion after September 2008 significantly affected its financial conditions. The World Bank Public Expenditure Support Facility Development Policy Loan (DPL) with Deferred Drawdown Option (March 2009 to December 2010), sought to assist the government in addressing the potential adverse impacts of the global financial crisis on public expenditures by supporting key policy measures in three areas: (i) reassuring fi ancial markets and maintaining fi ancial system stability; (ii) sustaining critical public expenditures while maintaining budget discipline; and (iii) facilitating private investment and supporting exports. As part of a $5.5 billion package, the loan provided a contingent line of credit of $2 billion, which the government committed to use only if it could not meet its financing needs for 2009 by defining specific drawdown triggers based on market conditions. The program was very successful in achieving the goals of calming the financial market at a time of great uncertainty, and of safeguarding critical public expenditures. The more recent drastic reduction of energy subsidies went well beyond expectations when the operation was implemented. On the other hand, there was backtracking in investment regulations, as well as a lack of timely compliance with Extractive Industries Transparency Initiative obligations, although recent policy changes in foreign direct investment regulations and the lifting of Indonesia’s Extractive Industries Transparency Initiative suspension bode well for the continuation of the reform process in these areas. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the outcome rating was satisfactory based on data produced through the program’s monitoring and on additional information. The Poverty Reduction Fund project was approved in 2002 to respond to one of the top priorities of the government of the Lao PDR—reducing poverty in rural and remote areas of the country. The government considered lack of infrastructure to be the root cause of poverty and in response, the project sought to provide rural infrastructure and better access to basic services through a $34 million loan. The project adopted a community based approach, which was aligned with the vision of participation and decentralization expressed in the country strategy, although the approach turned out to be largely incoherent given the highly-centralized nature of government policies. The project supported more than 3,000 subprojects in seven provinces, supporting the construction, for example, of schools, bridges, health dispensaries, rural roads, clean water systems, and irrigation schemes. A large share of first phase subprojects did not go to the poorest districts, and it took some time to adjust targeting. The quality of the infrastructure developed varied, and about one-fourth of subprojects did not meet technical specifications at closing. Rates of use were high, but the impact on poverty and well-being of rural communities could not be determined. Overall, however, higher enrollment rates, higher access to a minimum package of health services, and better availability of clear water were documented in participating villages benefiting from investment. By contrast, only half of rural roads constructed could be used year around, because of the challenges of maintaining them. The project had very modest results in generating local capacity, as communities were not given full responsibility for selecting, implementing, supervising, and assessing subprojects. The integration of the poor was limited as was that of non-Lao-speaking Independent Evaluation Group | World Bank Group 269 groups. In addition, despite attempts to increase women’s participation, decision making at the community level remained largely male dominated. Overall, the World Bank’s ICR rated development outcome satisfactory. Based on the shortcomings outlined above, IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately unsatisfactory. The World Bank has provided support to the basic education sector in Lao PDR since 1993, when the country started its transition to a market economy. After the first project in the sector closed in 2000, the government and the World Bank embarked on developing the Second Education Development Project reflecting the fact that low levels of education continued to act as a major constraint to further development—enrollment in primary education was nearly universal at grade one, but many students did not graduate, and only about one-third of the workforce has secondary education. The project objective broadly sought to achieve universal completion of primary education, and was supported by three components designed to increase access to and completion of primary education in the poorest districts; improve the quality of primary education; and strengthen capacities for policy analysis and management. IEG found that all three components were well designed and supported, and made a clear contribution to the project’s objective. The project became effective in January 2005 and a midterm review was carried out in 2007. Because the project was assessed to be performing well, the World Bank and the government agreed to additional financing to cover more remote communities, extending the planned life of the project from 2010 to 2013 and providing additional financing of $15.5 million, supplementing the original project budget of $13 million, and a trust fund supported by the government of Australia to finance additional infrastructure ($2.9 million) that had been added in 2008 and 2009. IEG also found that the project made an important contribution to improving both net enrollment, and the completion rate at primary level, primarily by increasing coverage in poor and remote areas. Increased participation is directly associated with the increase in the number of classrooms in the provinces that the project financed. IEG found likely moderate increases in competency in the Lao language from 2006 to 2009, although the level of mathematics competence decreased from a low baseline. IEG draws lessons from the project assessment, including the following: direct technical support from World Bank staff can make an important contribution to implementation and outcomes; and sustainable student assessment systems take time to develop and to contribute to the education system. In that latter regard, IEG notes that much of the benefits of assessment set up under the project are being fully demonstrated a decade after the project was approved. As such, IEG rated overall development outcome satisfactory, an improvement on the World Bank’s ICR rating of moderately satisfactory Despite the frequency and severity with which the Philippines suffers from natural hazards, which partially explains the stubbornly high poverty and inequality in the country, there was limited attention to DRM prior to 2009. Since then, the government has sought to develop a coherent, integrated, and proactive approach to DRM, which the World Bank has supported by increasing 270 the focus on resilience to natural disaster in its country strategies. The World Bank DRM DPL with a catastrophe deferred drawdown option (2012–15) aimed to enhance the capacity of the Government of the Philippines to manage the impacts of natural disasters through a policy program targeted at strengthening the institutional capacity for DRM efforts, mainstreaming DRM into development planning; and better managing the government’s fiscal exposure to natural hazard impacts. The operation had a catastrophe deferred drawdown option for disbursement of the loan proceeds to provide quick-release financing in the event of a disaster. The operation was complemented by a technical assistance program from the World Bank Global Fund for Disaster Risk Reduction to provide targeted support in these areas. The program was successful in achieving the goal of enhancing the capacity of the country to manage the impacts of natural disasters. The relevant agencies demonstrated high ownership of the reform agenda, and there was considerable progress in all three areas of the policy program. The weakest spot identified by IEG was in the design and implementation of a monitoring system to track disaster financing where weak institutional capacity handicapped the Of fice of Civil Defense. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded on a satisfactory development outcome rating based on data monitored under the program and other additional, supporting evidence. Rural development in Solomon Islands has performed poorly since the peace settlement that followed tensions there in 2003. The World Bank sought to contribute to improvements in rural development by supporting the first phase of the $37 million Rural Development Program (2007– 15). The program used a CDD approach to ensure improved social services, enhanced rural infrastructure, and amplified commercial opportunities for farmers. It was intended that phase I would provide the basis for a longer-term commitment from Solomon Islands government to ensure increased outreach and visibility for those in isolated regions. The program was successful in introducing the necessary structural and distributional platforms for the provision of enhanced services. In addition, communities embraced the participatory process for decision making and benefited from the direct funding of small scale infrastructure investments. Importantly, the Ministry of Agriculture and Livestock was re-energized by institutional support provided under the program, and nascent extension activities were now being directed at regions with potential agricultural growth opportunities. The project effectively illustrated the benefits of targeted community programs in fragile states, and showed that building sustainable institutional structures for multiphase projects has long-term benefits. In addition, the Rural Development Project showed that capacity building and institutional reform in weak ministerial environments can energize and enhance positive change, but only if there is suf ficient understanding of the gaps and limitations of the department; and that rural enterprise development is dif ficult in isolated regions, requiring a combination of approaches to ensure it can benefit the diverse interests and capabilities of all participants. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the outcome rating was moderately satisfactory, Independent Evaluation Group | World Bank Group 271 based on monitoring data that demonstrated progress in improving rural livelihoods and ensuring greater access to services. The project development objectives for the Public Financial Management Reform Project that was implemented in Vietnam May 22, 2003, to October 31, 2013, (inclusive of a period supported by additional financing) were to strengthen the government’s capacity to plan, execute and report on its budget, and to improve the transparency and accountability of the budgetary systems and processes. The objectives for the supporting MDTF for Public Financial Management Modernization was to strengthen the client’s capacity for increased coordination and transparency in public finance management; improved revenue mobilization; and effective and ef ficient public expenditure management. That project was implemented April 1, 2009, to June 30, 2012, inclusive of extensions. Both projects experienced slow initial starts and various delays, but the bulk of activities and reforms were completed by 2012/13, most notably the implementation of the budget and of the treasury budget management information system that consumed the bulk of financing. Top-down and bottom-up ownership and capacity building proved to be effective and sustainable, while change management actions have continued to be implemented by the government, thereby enhancing impact. The IEG review estimated that 22 PFM reforms were directly attributable to the MDTF II. Similarly, the modern treasury management information systems put in place appeared to be one of the best performing systems (for coverage, functionality, and reliability) implemented under similar World Bank operations. Both operations benefited from strong fundamentals, with suf ficiently ambitious and realistic objectives. Some lack of clarity in the logical framework was rectified in due course. PFM reform project achieved its objectives, principally on the successful implementation of the budget and treasury management information system as well as development of the debt database and piloting of the medium-term expenditures framework in four ministries and four provinces. Overall, the World Bank’s ICR rated development outcome moderately satisfactory for both the PFM reform project and the MDTF. IEG rated the development outcome for the former satisfactory and highly satisfactory for the latter. Evaluation Findings from Country Program Performance During FY17, IEG completed CLRRs for eight countries in the East Asia and Pacific Region, namely, Lao PDR, Vietnam, and six Pacific Island States: Kiribati, the Marshall Islands, the Federated States of Micronesia, Samoa, Tonga, and Tuvalu. In terms of development outcome, IEG rated one program satisfactory, two moderately satisfactory, three moderately unsatisfactory, one unsatisfactory, and one highly unsatisfactory. The World Bank Group’s CPS for Lao PDR FY12–16 comprised three strategic, thematic areas— competitiveness and connectivity, sustainable natural resource management, and inclusive development—which were traversed by a public sector management cross-cutting theme. The CPS and the CPS progress report addressed important priorities, and drew on lessons from the previous 272 program, including on the critical importance of capacity building that needed to be embedded into broader sectoral programs rather than through separate activities. However, IEG found it unclear to what extent the program succeeded in this regard—only a few of the objectives and or outcome indicators in the CPS or CPS progress report results matrix related to capacity building. The program was supported by a limited IDA envelope and complemented by many grants and trust funds, as well as a substantial knowledge program. IDA approved 17 operations (including additional financings) during FY12–16 for a total amount of $331 million, of which the largest were for power grid, Poverty Reduction Fund, and integrated water resource management. IFC played a modest role, primarily focused on Advisory Services. IEG found that the program addressed key priority areas, but given the limited IDA envelope and institutional capacity issues, it was not well focused, and did not have as much depth as would have been desired. IEG concurs with lessons presented in the CLR, including the following: considering political economy factors as drivers of government ownership and commitment is critical for program design; a strong consolidated program of Advisory Services and Analytics is a key strength of the CPS and highly valued by the client; presence and program leadership in the field provide immediate and prompt response to client needs and emerging opportunities; and engagement is critical in community-driven interventions, and it is as important as the provision of infrastructure. IEG draws four main lessons from its review as follows: program design matters, with particular reference to focus and selectivity toward achieving development results, noting that, in this case, the program was probably too broad in focus; World Bank and IFC collaboration on specific operations can be useful to drive reforms forward, but needs to be properly planned and accounted for in the results frameworks; it is preferable for results frameworks to incorporate all major areas of the program, and in reporting on results frameworks, it is equally important to ensure that all results, including numerical results, can easily be accounted; and finally, the World Bank’s safeguard requirements for high risk operations can work, with the help of attentive World Bank supervision, but may require close attention throughout the life of a project. The World Bank’s closure report rated overall development outcome satisfactory, and IEG’s review rated overall development outcome of the CPS as moderately satisfactory reflecting, for example, IEG’s findings regarding lack of focus and the lack of indicators to support demonstration of progress in capacity building. IEG reviewed six CLRs covering the FY11–17 period for the PICs of Kiribati, the Marshall Islands, the Federated States of Micronesia, Samoa, Tonga, and Tuvalu. During the period under review, a stand-alone CAS or CPS was prepared for each country. This was facilitated by the significant partnership relationships that developed between the World Bank Group and the governments of Australia and New Zealand. The strong partnership arrangements resulted in financial support from those governments for bilateral engagement, in contrast to previous engagements that were done under an umbrella regional strategy for the PICs. Except for Tuvalu’s country program, all CPSs were joint programs between the World Bank and IFC, and with the exceptions of Samoa and Tonga, these programs represented the first direct engagement between the countries and World Bank Group. The challenges for the six countries are common to many small islands and include small size, Independent Evaluation Group | World Bank Group 273 relative isolation and remoteness, distance to markets, vulnerability to natural disasters, limited private sector opportunities, and large—and in some cases, inef ficient—public sectors. All the countries share middle-income status with some hardship, but little extreme poverty, and limited administrative capacity. In support of the development plans of the various governments, the World Bank Group prepared country assistance strategies shaped by four overarching themes: (i) strengthening regional and global integration; (ii) building resilience to economic shocks, natural disasters, and climate change; (iii) encouraging economic reform and private sector development; and (iv) improving education, health, and social services. In Kiribati, the two focus areas of the World Bank’s program were (i) building resilience and (ii) mitigating economic isolation; however, the results framework was overly complex, comprising 14 outcomes across six objectives. Certain interventions supporting the objectives were delayed or did not perform as Kiribati got to grips with direct engagement with the World Bank Group. Over the duration of the program, little progress was made on improving the capacity to transport food to remote outer islands, or reducing reliance on imported fuel and increasing energy affordability. Interventions to create greater opportunities for temporary labor migrations were implemented after the CAS period. However, the program showed some success in improving resilience to climate change and natural disasters, and improving transport infrastructure and telecommunication services. The World Bank program for the Marshall Islands sought to liberalize and increase access to the ICT sector; reduce reliance on expensive fuel through significant energy reform; increase the net economic benefits from fisheries; and enhance capacity to manage transition to increased budgetary self-reliance. IEG found that the World Bank underestimated the political economy limitations that shaped the client’s poor commitment to reform in key areas in the program, such as in the energy and ICT sectors. The World Bank had limited prior engagement and limited knowledge of country context. Progress in achieving CPS objectives was also hampered by a lack of staf fing in the field, as well as changes in government. In support of the Federated States of Micronesia’s development plan, the World Bank Group program focused on two themes: (i) strengthening the enabling environment for private sector development; and (ii) promoting a sustainable medium-term fiscal situation to improve service delivery. In this case, IEG found the results framework to be compact and adequate. However, the program—which was to be completed in FY17 after the CLR and IEG’s review—showed limited progress on strengthening the business climate, improving the management of fisheries resources, and managing fiscal adjustment through the compact of transition. No progress was made on improving the ef ficiency of infrastructure and lowering the cost of services. In the case of Samoa, the World Bank supported the government in (i) rebuilding macroeconomic resilience and encouraging inclusive growth; (ii) generating opportunities from global and regional integration; and (iii) strengthening resilience against natural disasters and climate change. IEG found that the use of DPOs was appropriate given Samoa’s strong performance, policy reform experience, and relative familiarity with the World Bank. Some progress was observed on rebuilding macroeconomic 274 resilience and improving agriculture productivity while strengthening opportunities for medium and small scale farmers, but no tangible progress was made in infrastructure and service delivery. The World Bank program for Tonga had three focus areas: (i) building fiscal resilience and stability, (ii) generating opportunities through greater integration with the world, and (iii) building resilience against shocks. As in the case of Samoa, IEG found that the use of DPOs was appropriate given Tonga’s relative familiarity with the World Bank and its commitment to reform in areas such as macroeconomic management, fiscal policy, and energy development. Substantial progress was observed in building fiscal resilience and stability, and in generating opportunities through greater integration with the world, but uneven progress across outcomes was observed in strengthening resilience against natural disasters. The World Bank program for Tuvalu was simply designed around two focus areas: ([i] mitigating economic isolation by encouraging regional and global integration and [ii] building resilience against exogenous shocks) with five targeted outcomes. Progress was made on improving operational safety and oversight of airport infrastructure, and on strengthening fiscal management and improving the fiscal position. However, on the objective of creating opportunities for Tuvaluans from overseas temporary labor migration, IEG finds that the CLR notes progress that is dif ficult to attribute to World Bank interventions. That said, the World Bank had a track record in this area having started to work on labor mobility in 2012 and having advocated for labor mobility on an ongoing basis with the governments of Australia and New Zealand. IEG found World Bank Group performance mixed, with a rating of fair in three countries (Kiribati, the Marshall Islands, and the Federated States of Micronesia), and good in the other three (Samoa, Tonga, and Tuvalu). Programs were generally relevant, and the World Bank Group provided selective support where it had established capacity. However, the most significant design problem was that the World Bank underestimated the issues arising from its first-time direct engagement. Interventions supporting the targeted outcomes were delayed in some cases, or did not perform as several countries were just learning how to implement a World Bank Group program—including dealing with World Bank procedures. Remote locations added to the dif ficulties of program supervision. Another key weakness of design in most programs was the inability to accurately estimate the time required to process and implement new supporting activities that would deliver results. There were no PLRs undertaken for any of the programs in any of the PICs, and IEG notes that PLRs would have helped address some of the issues that appeared during the first few years of program implementation. With reference to procurement, the smaller islands face particularly significant capacity constraints—both on the government side as well as the private sector side. World Bank support is also limited owing to cost, size of projects, and remoteness. IEG agrees with the conclusion reach by the Integrity Vice Presidency that the impact of not building fiduciary capacity is likely to reduce the quality, if not increase the cost of the goods and services procured, or both. Similarly, not building in-country capacity on safeguards is having similar adverse effects. The application of safeguards policies, for example, face similar issues as in the fiduciary area: (i) Independent Evaluation Group | World Bank Group 275 unfamiliarity with World Bank policies and procedures; (ii) extremely low country capacity revealed by considerable skills and technical constraints; and (iii) absence of social specialists during project identification and preparation, leading to extensive delay in operations. During the review period, IFC had a limited role in the PICs. The three IFC Advisory Services projects—there were no investment project–supported microfinance networks in Samoa and Tonga, a hotel in Kiribati, and, during an eight-month period in 2010, post-tsunami rehabilitation of hotels in Samoa—were rated successful or mostly successful for development effectiveness. IEG identifies a number of lessons from its review of the CLRs for the PICs, for example: program design can be regional (owing to scale in some cases) but implementation is always country specific (reflecting issues such as ownership and institutional capacity); broad ownership of proposed reforms, beyond the government, and appropriate preparation of analytical work (for example, political economy of reform) are essential for the success of DPOs; development partners remain an essential component of World Bank success in the PICs, and the World Bank’s convening role needs to be further enhanced; there is room for a more prominent role for IFC in line with some transformational investments made in the past (for example, DIGICEL in Tonga); and PLRs are important program milestones to respond to changing realities in the field. In the case of Samoa, both the CLR and IEG’s review rated overall development outcome satisfactory. IEG upgraded the CLR rating for development outcome from moderately unsatisfactory to moderately satisfactory for both the Federated States of Micronesia and Tuvalu, the rating for Kiribati from unsatisfactory to moderately unsatisfactory, and the rating for the Marshall Islands from highly unsatisfactory to unsatisfactory. The FY12 – calendar year 2016 CPS for Vietnam was the first since Vietnam became a lower- middle-income country in 2009. To assist the country in realizing its strategy and its aspiration to become a successful middle-income country, the CPS had three focus areas: (i) competitiveness, (ii) sustainability, and (iii) opportunity. The program, which saw new approvals of $8.52 billion for 48 operations, was dominated by investment project financing (36 investment projects for a total of $6.61 billion). IFC made long-term investment commitments for 20 projects ($228 million), and extended short-term revolving trade guarantees under the Global Trade Finance Program to three Vietnamese banks ($818 million), which IEG considers to be a quite modest level of investment for a rapidly growing economy, and may possibly be due to delays in the privatization process. The CPS also identified three cross-cutting themes. The first, governance, was well addressed through high level dialogue, the 2012 anticorruption diagnostic, sector reviews and analytical work. The CLR states that 75 percent of the World Bank projects approved in FY16 included a gender analysis (the second cross-cutting theme), and the third (resilience) was captured through specific indicators in the framework. IEG rates World Bank Group performance as good. The World Bank Group played a strong role in policy dialogue and donor relations at a time when several important development partners were phasing down their activities in Vietnam. Lending was complemented by considerable amounts of complementary trust funds and knowledge activities. However, IEG found the program to be somewhat fragmented, lacking real selectivity, and that the results framework had significant weaknesses. The CLR identifies lessons with which IEG mostly agrees, regarding, 276 for example, selectivity through dialogue, the value of cross-sectoral approaches, and the need to better understand the political economy. IEG’s review highlights three lessons that emphasize the need for greater concentration of resources (this also applies to knowledge products) and for fewer but on average larger lending operations; the need to pay more attention to the design of the results framework, including the balance between objectives and their underpinning outcomes and indicators; and the need to recognize that the opportunity to update the results framework at the PLR is an essential element of proactive program management. Both the World Bank’s closure report and IEG’s review rated overall development outcome of the CPS as moderately satisfactory. IEG also produced a Cluster Country Program Evaluation on Small States that included coverage of several small states (defined as those that have a population of less than 1.5 million) in the Region, namely Mauritius, the Seychelles, Cabo Verde, and Djibouti (as well as the Pacific Island States, and the small states included under the umbrella of the Organization for Eastern Caribbean States). Though small states differ widely, they share several challenges, including those of strengthening their resilience to economic and natural disaster-related shocks and of remaining competitive to generate vigorous economic growth. Among the strategic and operational approaches that the World Bank Group used to respond to the more constraining parameters that small state country teams confront, were engaging on a regional or multicountry level, addressing country capacity limitations, and using partnerships to engage jointly or in a coordinated way with others. Thematically, the small state programs under review deployed several approaches, and addressed key development issues geared to the client countries’ small size, seeking to strengthen macroeconomic, disaster and climate, and social resilience, as well as enhance competitiveness by facilitating trade, strengthening the financial sector, expanding infrastructure and improving its management, and supporting leading sectors (such as tourism, fisheries, and agriculture). Under the resilience pillar, the evaluations suggest that support for strengthening macroeconomic resilience needs a multifaceted approach to be effective. The support needs to focus on the multiple drivers of fiscal and debt sustainability. The approach to DRM of directly supporting resilience building (notably for infrastructure) was helpful but had limits. Making real improvements in vulnerability requires continuing efforts to foster wholesale changes in public and private incentives and behavior, and the long-term risks of climate change (including helping countries access climate fi ancing) need to be addressed. Important focus areas for social resilience include preventing and managing noncommunicable diseases, strengthening social protection, and enhancing mobility to larger countries’ labor markets (as the World Bank helped to do for the PICs with Australia and New Zealand). Under the competitiveness pillar, a sharper focus is needed on the most binding business constraints, using lenses specific to the leading sectors—typically fisheries, tourism, and agriculture. Several World Bank Group programs have started such an approach. In the financial sector, key issues for attention include system soundness and risks (particularly in the Organisation of Eastern Caribbean States [OECS]), access to finance for SMEs, and remittance costs. Finally, it is important for infrastructure work to continue focusing on separating regulatory functions from service Independent Evaluation Group | World Bank Group 277 delivery functions. Despite the pitfalls, a strong case can be made for continuing to support PPP in infrastructure service delivery and regional service provision and regulatory initiatives. The program evaluations also raise issues for World Bank Group consideration in its engagement with small states—relating to the cross-cutting aspects at both at the country team and institutional levels. For example, at the country team level, contingency planning, including the use of contingent financing instruments, has a key role to play in enhancing World Bank responsiveness to the shocks that small states suffer. At the institutional level, issues that merit consideration include whether small state IDA allocations can be made more responsive to vulnerability, and whether small state–specific World Bank and IFC financing instruments are worth developing. Europe and Central Asia Region Project- and Country-Level Performance This note presents the results and performance separately for the World Bank, IFC, and MIGA given the different time frames, operating models, and clients of the three institutions. The World Bank: Project-Level Performance For the current review period, FY14–16, the overall development outcome rating for 76 percent of the 98 projects rated in the Europe and Central Asia Region was MS+, compared with a rating of 73 percent MS+ both World Bank–wide and as an average for all other Regions (figure L.28, panel a). By commitment, the rating was 90 percent MS+ in the Europe and Central Asia Region, compared with an average 84 percent MS+ rating World Bank–wide and an 83 percent MS+ rating across all other Regions (figure L.28, panel b). Overall World Bank performance, a composite of ratings for quality at entry and project supervision, was 76 percent MS+ in the Europe and Central Asia Region during FY14–16, ahead of a rating of 72 percent MS+ World Bank–wide and a 71 percent MS+ average rating across all other Regions (figure L.29, panel a). Within the overall World Bank performance rating, quality at entry was rated 64 percent MS+, compared with an average rating of 60 percent MS+ across the other Regions. Supervision was rated 85 percent MS+ in the Europe and Central Asia Region, compared with an average rating of 77 percent MS+ across the other Regions. By commitment volume, World Bank performance in the Europe and Central Asia Region during FY14–16 (92 percent MS+) exceeds both the World Bank–wide average (84 percent MS+) and the average for all other Regions (82 percent MS+: figure L.29, panel b). By volume, World Bank performance in the Europe and Central Asia Region improved between FY11–13 (82 percent MS+) and FY14–16 (92 percent MS+). 278 Figure L.28. Europe and Central Asia: Project Outcome Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: ECA = Europe and Central Asia; MS+ = a rating of moderately satisfactory or better. The overall quality of M&E for projects in the Europe and Central Asia Region during FY14–16 was 44 percent S+, compared with a World Bank–wide average of 34 percent substantial and an average 33 percent S+ rating across all other Regions (figure L.30, panel a). By commitment volume levels, M&E in the Region (53 percent S+) is somewhat better than the World Bank–wide average (46 percent S+) and the average for all other Regions (44 percent S+: figure L.30, panel b). The development outcome rating for IBRD-supported projects in Europe and Central Asia—51 projects rated for the Region during FY14–16 and over $10 billion of the $11 billion in volume—is Independent Evaluation Group | World Bank Group 279 Figure L.29. Europe and Central Asia: World Bank Performance Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: ECA = Europe and Central Asia; MS+ = a rating of moderately satisfactory or better. 80 percent MS+ for the period, compared with an average 74 percent MS+ rating across all other Regions (figure L.31, panel a). By commitment volume, the rating was 93 percent MS+, ahead of an average 83 percent MS+ rating across all other Regions (figure L.31, panel b). The development outcome rating for IDA-supported projects in the Europe and Central Asia Region during FY14–16 is 69 percent MS+, compared with an average 74 percent MS+ rating across all other Regions (figure L.31, panel a). By commitment volume, the rating was 74 percent MS+, compared with an average rating of 83 percent MS+ across all other Regions (figure L.31, panel b). World Bank performance for IBRD-supported projects in the Europe and Central Asia Region during FY14–16 (82 percent MS+) also leads the average across all other Regions (74 percent MS+). World 280 Figure L.30. Europe and Central Asia: M&E Quality Ratings, FY05–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: ECA = Europe and Central Asia; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; MS+ = a rating of moderately satisfactory or better. Bank performance (72 percent MS+) for IDA-supported projects in the Europe and Central Asia Region is the same as the respective average across all other Regions (figure L.32). M&E quality for IBRD-supported projects in the Europe and Central Asia Region during FY14–16 (49 percent S+) leads the average across all other Regions (33 percent S+). M&E quality for IDA- supported projects in the Europe and Central Asia Region is 33 percent S+, compared with 32 percent rated S+ for all other Regions (figure L.33, panel a). Ten or more projects were rated in each of three GPs in the Europe and Central Asia Region during FY14–16: Governance (10); SURR (12); and Transport and ICT (13). These three combined GPs represent 36 percent of all projects rated for the Region. Of those GPs with 10 or more projects, Independent Evaluation Group | World Bank Group 281 Figure L.31. Europe and Central Asia: Project Outcome Ratings by Financing Source, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: ECA = Europe and Central Asia; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; MS+ = a rating of moderately satisfactory or better. and Central Asia: World Bank Performance Ratings Figure L.32. Europe by Financing Source, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: ECA = Europe and Central Asia; MS+ = a rating of moderately satisfactory or better. 282 and Central Asia: M&E Quality Ratings by Financing Figure L.33. Europe Source, FY14–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: ECA = Europe and Central Asia; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; M&E = monitoring and evaluation; S+ = a rating of substantial or better. Independent Evaluation Group | World Bank Group 283 and Central Asia: Development Outcome by Global Figure L.34. Europe Practice, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: “Other” includes countries without International Development Association or International Bank for Reconstruction and Development status or project locations that are above country level. ECA = Europe and Central Asia; MS+ = a rating of moderately satisfactory or better. 284 outcome ratings ranged from 50 percent MS+ for Governance projects, to 85 percent MS+ for Transport and ICT projects, to 92 percent MS+ for SURR projects (figure L.34, panel a). World Bank performance for both the SURR and Transport and ICT GPs was also better than counterpart performance in other Regions. Figure L.35. Europe and Central Asia: Results and Performance Ratings of Evaluated IFC Investment Projects Share of Projects with Positive Ratings a. By number of projects, three-year moving average b. By IFC net commitment, three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC XPSRs as of December 12, 2017. Note: (1) Positive ratings for development outcome consists of mostly successful, successful, or highly successful (aggregated as MS+) ratings. Positive ratings for positive ratings for investment outcome and IFC’s role and contribution consist of excellent or satisfactory (aggregated as S+) ratings. (2) Due to the revision of the XPSR guidelines in 2015, the indicator, IFC’s role and contribution, has been amended s replaced with the indicator, IFC Additionality. Independent Evaluation Group | World Bank Group 285 M&E ratings were substantially better for all three of the GPs when compared with counterpart GPs in other Regions. This is particularly the case for SURR projects in the Region, for which M&E was rated 67 percent S+. International Finance Corporation: Project-Level Performance IFC Investments Project-Level Results Compared with CY11–13 results, the development outcome ratings of the evaluated IFC investment projects in the Europe and Central Asia Region were lower this review period (CY14–16). Of the 55 projects evaluated in CY14–16, 24 (or 44 percent) were rated MS+ for achieving the projects’ intended development outcome (figure L.35). By IFC commitment amount, the share of projects with MS+ outcome ratings was slightly higher (65 percent), which marks an improvement over the CY11–13 outcome results. Positive (S+) ratings for IFC investment outcome and for IFC role and contribution Figure L.36. Europe and Central Asia: Performance of Development Outcome Indicators of IFC Investment Projects, by XPSR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC net commitment, three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC Investment Projects as of December 12, 2017. Note: (1) S+ = a rating of Satisfactory or Excellent. (2) PBS = Project Business Success. ECSUS = Economic Sustainability. E&SE = Environmental and Social Effects. PSD = Private Sector Development. These four subindicators comprise the development outcome dimension of an IFC investment project. (3) No Opinion Possible or Not Applicable ratings were excluded. 286 remain the same, if measured by volume, although the percentages show a decline between the two periods, by number of projects. Except for E&S effects, the positive (satisfactory or better, S+) ratings for the other three subindicators of development outcome declined compared with CY11–13 results. The poor ratings of the projects’ business success, economic sustainability and contribution to private sector development weakened the overall development outcome results in CY14–16, by number of projects. Only one-third of the 55 evaluated projects in the Europe and Central Asia Region in CY14–16 had positive (S+) ratings for meeting their business performance and economic sustainability benchmarks and objectives. Projects with low (partly unsatisfactory or unsatisfactory) development outcome ratings also had low ratings for their business success and economic sustainability. And only about half of the CY14–16 evaluated projects in the region contributed satisfactorily to private sector development (figure L.36). Evaluated projects in the Europe and Central Asia Region in CY14–16 show close association between IFC work quality, development outcome and IFC outcome ratings. Positive development outcome ratings are closely associated with positive work quality ratings. Similarly, projects with low development outcome ratings (mostly unsuccessful or below, MU-) also have low ratings for IFC work quality (figure L.37). A similar association is also observed between positive development outcome ratings and positive ratings for IFC’s role and contribution as well as the low development outcome and low IFC work quality and low development outcome and IFC role and contribution ratings. However, the pattern of association between development outcome and IFC investment outcome ratings seems applicable only to evaluated projects with high development outcome and high investment outcome ratings. By volume, the close association between these three IFC performance indicators only applies to the high-high ratings combination. Ratings of the two subindicators of IFC work quality differ slightly in CY14–16. By number of evaluated projects, half of the evaluated projects had S+ ratings (conversely, half were rated less than satisfactory) for the quality of IFC’s project SAS. By volume, however, the percentage of S+ ratings for SAS improved compared with CY11–13 results because of better SAS ratings in several large projects (figure L.38). However, in CY14–16, the quality of IFC’s SUP of the evaluated projects weakened, both by number of projects and volume, compared with the SUP ratings in CY11–13. Major shortcomings in the supervision and administration of a few evaluated large projects in the tourism, retail and construction subgroup affected the aggregate rating for the quality of IFC’s supervision and administration in CY14–16. Independent Evaluation Group | World Bank Group 287 and Central Asia: Association Between Development Figure L.37. Europe Outcome, IFC Work Quality and IFC’s Role and Contribution, CY14–16 By number of evaluated projects By IFC net commitment ($, millions) a. Association between development outcome and IFC work quality b. Association between development outcome and IFC role and contribution c. Association between work quality and IFC investment outcome Source: Independent Evaluation Group evaluation data for IFC investment projects as of December 12, 2017. Note: (1) IFC work quality in CY14–16 reflects the aggregate ratings of two subindicators: screening, appraisal, and structuring and supervision and administration. (2) High development outcome = highly successful, successful, and mostly successful ratings; low development outcome = mostly unsuccessful, unsuccessful and highly unsuccessful ratings; high work quality = excellent and satisfactory ratings; low work quality = partly unsatisfactory and unsatisfactory ratings; high IFC role and contribution = excellent and satisfactory ratings; low IFC role and contribution = partly unsatisfactory ratings; high IFC investment outcome = excellent and satisfactory ratings; low IFC investment outcome = partly unsatisfactory and unsatisfactory ratings. (3) Due to the revision of the Expanded Project Supervision Report guidelines in 2015, the indicator IFC’s role and contribution has been amended starting CY15, and was replaced with the indicator IFC additionality. IFC = International Finance Corporation. 288 Figure L.38. Europe and Central Asia: IFC Work Quality Subindicators, by XPSR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC net commitment, three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC Investment Projects as of December 12, 2017. Note: Based on the 2015 revised XPSR guidelines, IFC work quality is measured by two indicators: screening, appraisal, and structuring (SAS) and supervision and administration (SUP) in the project. IFC = International Finance Corporation; XPSR = Expanded Project Supervision Report. IFC Advisory Services Project-Level Results Of the 18 IFC Advisory Services projects in the Europe and Central Asia Region evaluated in FY14- FY16, about half (56 percent by number of projects; 66 percent by volume) had ratings of MS+ for overall development effectiveness . This overall development effectiveness rating seems to be the lowest since the PCR Program started in FY08 (figure L.39). The share of success (satisfactory or better, S+) ratings of IFC’s role and contribution in the evaluated projects, remained constant compared with previous review periods. Independent Evaluation Group | World Bank Group 289 Figure L.39. Europe and Central Asia: Results and Performance Ratings of Evaluated IFC Advisory Projects, by PCR Program Years Share of Projects with Positive Ratings a. By number of evaluated projects, three-year moving average b. By IFC expenditure ($, millions), three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC PCRs as of December 12, 2017. Note: (1) The reported ratings are based on IEG’s final ratings in the completed Project Completion Reports (PCRs). (2) Positive ratings for development effectiveness consists highly successful, successful, or mostly successful (aggregated as MS+) ratings. Positive ratings for IFC’s role and contribution consist of excellent or satisfactory (aggregated as S+) ratings.. 290 Overall development effectiveness success rating in FY14–16 seems to reflect the share of MS+ ratings for the subindicator, outcome achievement, by number of projects and by volume. Compared with FY11–13, positive (S+) ratings of the four subindicators of development effectiveness in FY14–16 were lower, including ratings for output achievement, especially by number of projects. By volume, the decline in success ratings affected three of the four development effectiveness subindicators while rating for strategic relevance held constant (figure L.40). Figure L.40. Europe and Central Asia: Ratings of Four Indicators of development effectiveness of Evaluated IFC Advisory Projects, by PCR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC expenditure ($, millions), three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC PCRs as of December 12, 2017. Note: Success ratings for each of the three indicators of development effectiveness = Satisfactory or Excellent (referred to as S+) ratings. The reported ratings are based on IEG’s final ratings in the completed Project Completion Reports (PCRs). Independent Evaluation Group | World Bank Group 291 Multilateral Investment Guarantee Agency: Project-Level Results Of the 27 evaluated MIGA projects in the Europe and Central Asia Region in FY11–16, 16 projects (or 59 percent) had satisfactory or better (S+) development outcome ratings. MIGA’s work quality (assessment, underwriting, and monitoring) rating improved in FY11–16 while positive ratings for MIGA’s role and contribution in the evaluated projects held steady (figure L.41). Of the four subindicators of development outcome, private sector development had the strongest ratings in FY11–16 while about half of the evaluated projects had positive ratings for business performance, economic sustainability and E&S effects. Figure L.41. Europe and Central Asia: Results and Performance Ratings of Evaluated MIGA Projects, by PER Program Years Share of Projects with S+ Ratings, by Number of PERs a. By main indicators of MIGA project results and MIGA performance, six-year moving average b. Ratings of the four indicators of development outcome, by number of projects, six-year moving average Source: Independent Evaluation Group Evaluation Data for MIGA PERs as of December 12, 2017. Note: (1) PERs = Project Evaluation Reports. (2) Positive ratings for development outcome, MIGA’s Role and Contribution and Assessment, Underwriting and Monitoring refer to Satisfactory or Excellent (referred to as S+) ratings. The four subindicators of development outcome refer to BP = Business Performance; ECSUS = Economic Sustainability; E&SE = Environmental and Social Effects; PSD = Contribution to Private Sector Development. 292 Table L.5. Europe and Central Asia: Project Outcome Ratings by Country Type, FY14–16 Countries Projects Volume Evaluated Rated MS+ Total Amount Rated MS+  Country Type (no.) No. (percent) ($, millions) (percent) IDA fragile 1 1 100 10 100 IDA nonfragile 4 25 64 580 76 IBRD fragile 1 5 80 116 82 IBRD nonfragile 16 65 80 10,283 91 IDA total 5 26 65 589 76 IBRD total 17 70 80 10,399 91 Fragile total 2 6 83 125 83 Other n.a. 2 50 26 62 Total 22 98 76 11,014 90 Source: Independent Evaluation Group data. Note: “Other” includes countries without IDA or IBRD status or project locations that are above country level. IBRD = International Bank for Reconstruction and Development; IDA =International Development Association; MS+ = a rating of moderately satisfactory or better; n.a. = not applicable. World Bank Group: Country-Level Performance Projects were rated for five IDA and 17 IBRD countries in the Europe and Central Asia Region during FY14–16. There was one fragile country within each of the IDA and IBRD complements. By number, Table L.6. Europe and Central Asia: Country-Level Performance Ratings, FY14–17 CASCR/ CLR Review World Bank IFC MIGA Review FY Country Period Outcome Performance Performance Performance 2014 Armenia FY09–13 MS Superior n.a. n.a. Georgia FY10–13 S Good Good Fair Kyrgyz Republic FY07–13 MS Good n.a. n.a. Moldova FY09–13 MS Good n.a. n.a. Poland FY09–13 MS Superior n.a. n.a. Romania FY09–13 MS Good n.a. n.a. Tajikistan FY10–14 MS Good n.a. n.a. 2015 Albania FY11–14 MS Good n.a. n.a. Azerbaijan FY11–14 S Good n.a. n.a. Macedonia, FY11–14 MS Fair Good n.a. FYR of Serbia FY12–15 MS Fair n.a. n.a. 2016 Bosnia and FY12–15 MS Good n.a. n.a. Herzegovina Independent Evaluation Group | World Bank Group 293 Table L.6, continued. CASCR/ CLR Review World Bank IFC MIGA Review FY Country Period Outcome Performance Performance Performance Bulgaria FY11–13 U Fair n.a. n.a. Montenegro FY11–15 MS Good n.a. n.a. Uzbekistan FY12–15 MS Good n.a. n.a. 2017 Kosovo FY12–16 MS Good n.a. n.a. Ukraine FY12–16 MS Good n.a. n.a. Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: All World Bank performance ratings were standardized with the following conversions: satisfactory as superior, moderately satisfactory as good, moderately unsatisfactory as fair, and unsatisfactory as poor. CASCR = country assistance strategy completion report; CLR = Completion and Learning Report; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency; MS = moderately satisfactory; S = satisfactory; U = unsatisfactory; n.a. = not applicable. average performance on outcome across the Region during FY14–16 was 76 percent MS+ with projects in IBRD countries as a whole (80 percent MS+) performing better than the average, and projects in IDA countries performing below the average (65 percent MS+). By volume, the average satisfaction rating was 90 percent MS+ with projects in IBRD countries performing just above the average (91 percent MS+), and a rating for projects in IDA countries of 76 percent MS+ (table L.5). During FY17, IEG produced CLRRs for two countries in the Region, namely, Kosovo, and Ukraine. IEG rated development outcome moderately satisfactory for both country programs. World Bank performance was rated good in both instances (table L.6).3 Evaluation Findings from Project Performance Assessment Reports During FY17, IEG completed six PPARs in the Europe and Central Asia Region. In terms of development outcome, IEG rated four projects moderately satisfactory and two projects moderately unsatisfactory. The Danube River Basin is one of Europe’s largest river catchment areas. It suffers from pollution due to a high volume of nutrients, which has substantial negative consequences for soil fertility, agricultural productivity, and conservation of the biological ecosystem. More than half of the nutrient loads originate from agriculture due to intensive fertilizer application and poor manure management. In Croatia, the Danube River and its tributaries drain about 60 percent of the country’s territory, covering large parts of agricultural land. About 40 percent of Croatia’s agricultural holdings raise livestock, with a prevalence of poor manure management practices and lacking awareness of practices to reduce nutrient loads to water bodies. To address pollution in Croatia’s agricultural sector, the Global Environment Facility provided a $5 million grant for the Agricultural Pollution Control Project 2007–12. The project was successful in 294 achieving an increase in the use of environmentally friendly agricultural practices in three agriculture- intensive target counties. The establishment of a mitigation investment fund, which provided grants for 75 percent of the cost of manure storage for medium-scale livestock farmers was key to this outcome, in addition to an extensive field demonstration program on sustainable, cost-effective manure management. However, the project provided only limited outcome evidence on the reduction in nutrient discharge in water bodies from agricultural sources due to insuf ficient monitoring and impact analyses. Implementation benefited from a strong technical and highly engaged project implementation unit and a mostly consistent government commitment. Croatia’s accession to the European Union in 2013 and the related compliance with Nitrates and Water Directives requirements has ensured continued funding for environmental measures in the agriculture sector. Overall, the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory. In 2002, the Kyrgyz Ministry of Education approached the World Bank seeking assistance to address poor education quality, particularly in rural areas. In response, the World Bank developed and approved the $15 million Rural Education Project (2005–11), the World Bank’s first entry into the sector in the Kyrgyz Republic. The project development objective sought to improve learning and learning conditions in primary and secondary schools, with priority attention to rural areas. Some of the six project components had a national focus (for example, the development and provision of textbooks and national level assessment), and others (for example, the development of a formative assessment approach) focused on two, largely rural, pilot oblasts (regions), namely, Issyk-Kul and Talas. Despite the relevance of the objectives and evident client demand IEG found design to be modest, referencing a lack of prudent selectivity and unrealistic scope given known capacity limitations within the system. The objective to improve the quality of education, the core outcome aspired to, was only modestly realized despite the efforts of the government, the World Bank and other donors. The objective to improve the conditions to support learning was substantially achieved. The Rural Education Project contributed to building the momentum for change, not least through the low-cost medium of funding two Program for International Student Assessment assessments, but also through piloting initiatives that are being used to modernize the sector. That said, other relatively expensive interventions (for example, provision of textbooks) had little ongoing, systematic effect, and IEG noted a lack of coherence between interventions supported by the World Bank, a late comer to the sector, and other development partners already working in the education sector in the Kyrgyz Republic. IEG emphasizes that a selective and more realistic approach to design is likely to create a more closely articulated link (attribution) between World Bank interventions and outcomes. In this case, expert and thorough diagnosis of needs and associated challenges was not accompanied by realistic assessment of possibility, taking constraints into account such as the political economy, commitment, capacity, resources (human and financial), and available time. The World Bank’s self-evaluation rated development outcome satisfactory. IEG’s review rated development outcome moderately unsatisfactory, given design faults and limited evidence of improvements in the quality of education. Independent Evaluation Group | World Bank Group 295 The first policy-based guarantee (PBG) for the Former Yugoslav Republic of Macedonia (FYR Macedonia) was approved in 2011 in the face of recession in the Western Balkans, and market uncertainty. Despite solid macro management, the country was vulnerable to the global crisis as well as contagion from the crisis in Greece. The PBG, which combines features of a partial sovereign credit guarantee provided by the World Bank with stand-alone budget suppor t financing through international commercial markets, supported a program of policy reform agreed with the World Bank. The operation aimed to (i) improve the government’s access to international financial markets, (ii) fortify public finances and functioning of labor markets by controlling public expenditures, (iii) strengthen social safety nets by improving ef ficiency and equity of conditional cash transfers, and (iv) support resilience of the financial sector by creating a Financial Stability Committee to coordinate actions across agencies and reduce systemic risk. IEG found the operation benefited from lessons generated from an earlier, pioneering operation that the World Bank had supported in Serbia and this contributed to IEG’s satisfactory assessment of World Bank performance, noting that the team also provided extensive informal technical assistance and encouraged learning to the client. The main messages from the PPAR for this first PBG for the FYR Macedonia are that in addition to direct benefits of enhanced access, volume, and terms of borrowing and support for policy reform, the PBG can help the borrower establish itself in the eyes of investors and broaden the investor base. But large financing may help postpone rather than implement reforms, hence the need for a strong macroeconomic framework and results framework. Also, a PBG works better when informed by relevant knowledge, longer-term engagement, and trust. IEG rated development outcome for the operation moderately satisfactory reflecting substantial achievement on the financial objective and less so on fiscal and safety net objectives. The €155 million Public Expenditure Policy-Based Guarantee 2012–13 was the second PBG for FYR Macedonia in the aftermath of the global crisis. The guarantee, which supported a commercial loan in the amount of €250 million, had three specific objectives: improving the ef ficiency of public expenditures by strengthening PFM practices; improving the ef ficiency of service provision in the health sector; and strengthening social protection and inclusion. Design suffered from a lack of operational focus on the program’s macroeconomic and fiscal risks, and the relatively narrow scope of the PFM reform agenda at a time of heightened risks. This led to mixed results. The program helped to achieve clearance of arears and to introduce changes in treasury practices to prevent accumulation of future arrears. However, the information about arrears at the level of central government is not fully reliable, while arrears at the municipality level and in the health sector remain significant. The ef ficiency of capital expenditures deteriorated, and the replacement of the indexation mechanism for pensions by ad-hoc increases reduced the predictability of PFM. The program helped reduce the unit price of pharmaceuticals and some medical equipment, but savings yielded from these sources was tempered by an increase in the number and cost of prescriptions. An improvement in service provision to underserved areas came through the introduction of a new framework that helped direct more resources to those areas, and the government’s active labor market programs were strengthened by improving access for vulnerable groups. Fiscal resources 296 freed up because of improved targeting of social protection which enhanced the levels of financial assistance to beneficiaries. World Bank performance is rated moderately satisfactory reflecting the emphasis on clearance of arrears, and treasury reforms to alleviate a key constraint affecting the quality of PFM, as well as shortcomings in addressing macro and fiscal risks. Among the key evaluation lessons is that the positive impact of public expenditure measures (wage control, reforms in health procurement, and better targeting of social protection) can be eroded if the overall program lacks an overarching fiscal and debt framework—as in this case, fiscal savings can be dissipated by introducing nonpriority spending such as ad-hoc pension increases. The World Bank’s ICR rated the project highly satisfactory, and IEG’s independent field-based evaluation concluded that the outcome rating was moderately satisfactory refl cting the shortcomings identifi d above. The 2012–13 Financial Sector PBG for Montenegro was prepared by converting the second operation of the programmatic series of the Financial Sector DPL in the amount of $20 million into a guarantee instrument to respond to the large external financing needs of Montenegro with the sole objective of strengthening the banking system. IEG found that many important outcomes achieved in Montenegro’s banking system—banking system largely recovered, market confidence restored, and total deposits surpassing precrisis level—were attributable to the strong reform actions supported by the financial sector DPL, with the Financial Sector PBG having relatively low value-added. World Bank performance was rated moderately unsatisfactory, largely reflecting the poor choice of instrument, and modest program design reflecting the lack of operational focus on the program’s macroeconomic framework, as well as risks and shortcomings in assessing Montenegro’s eligibility for a PBG. The evaluation found the reform program to be incomplete, with gaps evident in crisis management, less than full compliance in banking supervision with the Basel Core Principles for Effective Banking Supervision, and a slower than expected decline in nonperforming loans. Withdrawal of public sector deposits in Prva Banka to ensure market-based funding was reversed in 2014 and 2015, and the World Bank continued to hold a large amount of public sector deposits at the time of the evaluation. These factors contributed to significant risk to development outcome. Among the lessons derived from the evaluation is that the choice of a PBG as an instrument should be based on robust macro and fiscal projections indicating that the financing framework is sustainable, and macro risks are mitigated (highlighting the criticality of World Bank–IMF cooperation). The World Bank’s ICR rated outcome as satisfactory, but IEG’s independent field-based evaluation delivered an outcome rating of moderately unsatisfactory, reflecting IEG’s assessment of the inadequacy of the PBG as an engagement modality, the lack of focus on the macroeconomic framework and risks, and an incomplete banking system reform agenda. As Ukraine began its policy reforms in transition toward a market economy after gaining independence in 1991 from the Soviet Union, the country had a statistical system designed for central planning that relied on full coverage of data collected through administrative mechanisms. Over the years, the government of Ukraine progressively transformed its statistical practices, and in 2003, Independent Evaluation Group | World Bank Group 297 the government approached the World Bank to request assistance for financing comprehensive modernization of the national statistical system. Under the Statistics Capacity Building Program, the World Bank provided $42 million to support the Development of State Statistics System for Monitoring Social and Economic Transformation. The project focused on organizational changes among data producers and the upgrade of data quality, the data dissemination framework, and the ef ficiency of data production. Project implementation benefi ed from several technical assistance activities bundled into one package, and this approach saved time and money for the participating agencies. The project made clear the need for long-term engagement in statistical capacity building to ensure gains in one area are not offset by losses in another. Under the project, progress was made on ef ficiency, accuracy, and timeliness of data, with statistical practices largely harmonized with international standards. On the other hand, operational autonomy of the central statistical agency—the State Statistics Service of Ukraine—was compromised by government measures in 2013. The project also highlighted the need for a governing body for the national statistical system. One of the key recommendations of the Statistics Capacity Building Program, and part of its template, is to establish a governing body responsible for setting strategic direction for the entire national statistical system. In Ukraine, apart from setting strategic direction, the National Statistics Council would have been expected to ensure the operational autonomy and professional independence of the State Statistics Service of Ukraine. Strong safeguards and clear rules for resource allocation and adequate funding for statistics are essential and these were not ensured in this case. The World Bank’s ICR assessed the operation’s outcome as satisfactory, whereas IEG’s evaluation of the project’s performance rated the outcome of the operation as moderately satisfactory, reflecting IEG’s observations on the limited extent to which operational autonomy and professional independence were realized. Evaluation Findings from Country Program Performance During FY17, CLRRs for two countries in the Europe and Central Asia Region, namely, Kosovo and Ukraine, were produced. IEG rated development outcome moderately satisfactory for both country programs, and World Bank performance was rated good in both instances. The Kosovo—Country Partnership Strategy for the Period FY12–15 was the World Bank Group’s first full CPS for Kosovo since it became a member of the World Bank in 2009. The CPS addressed a wide range of critical challenges across two focus areas: (i) accelerate broad-based economic growth and employment generation and (ii) improve environmental management. IEG’s review found program objectives to be broadly relevant to the country’s development goals. Interventions were aligned with the CPS objectives and well-supported by AAA products, although the contribution of some interventions (private sector development, agriculture, and employment) to the country’s development goals was not evident. 298 However, program design was fragmented, with seven CPS objectives, in contrast with the limited number of activities supporting each objective. The dispersed program objectives and results indicators undermined the program’s focus and impact. The results framework was overly complex, with 26 outcomes and 31 indicators, making it dif ficult to assess program results. Some result indicators reflected either inputs or intermediate results, but lacked reference to or measurement of outcome. The program mostly achieved five of seven objectives in infrastructure, education and skills, promoting private sector development, strengthening agricultural development, and environment. It partially achieved objectives in employment, social inclusion, and PFM. IEG presents findings and lessons that suggest the World Bank Group needs to: keep program design simple, with fewer outcomes and results indicators, and a stronger link between outcomes and indicators; and to align the program with the objectives of EU accession, to enhance its effectiveness. The review also noted that public employment is inadequate to solve Kosovo’s long-term problem of low employment. To increase labor force participation, the country should enact labor policy reforms that can generate structural changes in both the demand and supply of labor. IEG rated overall development outcome moderately satisfactory, a step down from the World Bank Group’s self-rating of satisfactory. The Ukraine—Country Partnership Strategy for the Period FY12–16 had two focus areas in support of Ukraine’s 2010–14 Economic Reform Program: support for building relations with citizens by improving public services and fi ance and support for building relations with business by improving policy effectiveness and economic competitiveness. At about mid-way through the CPS period, significant challenges arose in the political and economic context, including macroeconomic imbalances, structural reform priorities, and challenges relating to transparency and anticorruption measures. In response, the World Bank supported major policy and institutional reforms, primarily through development policy funding (public sector governance, business environment, utility subsidies, social protection, and the banking system). IEG’s review found the World Bank Group’s program performed well in several areas, including business climate reforms, banking system stabilization and consolidation, pension reform, and energy sector governance reform. The program fell short of meeting the targets for strengthening ef ficiency of PFM, improving ef ficiency of education and health expenditures, improving governance of municipal infrastructure services, and increasing access to finance for agriculture and agri-business. Both the CLR and the IEG review rated World Bank Group performance as good based on the well- designed, selective program notwithstanding weaknesses in the results framework (for example, indicators that were not suf ficiently linked to the outcomes and objectives, and the lack of program level outcome indicators in several areas). IEG’s review also finds the CPS was not modified to take account of changes in the country’s development challenges and the program, including the DPOs approved in FY14–15. IEG’s review draws the following lessons: the use of a phased approach can be appropriate where there is uncertain client commitment to reform, but a midterm assessment is necessary to assesses progress; and in the case of Ukraine, IFC played a countercyclical role, and helped boost investors’ confidence by maintaining its exposure and making new investments in the agriculture and agribusiness sectors. Both the World Bank’s closure report and IEG’s review rated overall development outcome moderately satisfactory. Independent Evaluation Group | World Bank Group 299 Latin America and the Caribbean Region Project and Country-Level Performance in the Latin America and the Caribbean Region This note presents the results and performance separately for the World Bank, IFC, and MIGA given the different time frames, operating models, and clients of the three institutions. The World Bank: Project-Level Performance For the current review period, FY14–16, the development outcome for 69 percent of all projects in the Latin America and the Caribbean Region was rated MS+, which is just below the average for all other Regions (74 percent) over the same period (figure L.42, panel a). Figure L.42. Latin American and the Caribbean: Project Outcome Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: LCR = Latin America and the Caribbean; MS+ = a rating of moderately satisfactory or better. 300 By commitment volume, the rating was 82 percent MS+ for the Latin America and the Caribbean Region during FY14–16, compared with an average 85 percent MS+ across all other Regions (figure L.42, panel b). Overall World Bank performance, a composite of ratings for quality at entry and project supervision, in the Latin America and the Caribbean Region during FY14–16, was 68 percent MS+, compared with a 73 percent MS+ average rating across all other Regions (figure L.43, panel a). Within the overall World Bank performance rating, quality at entry was rated 58 percent MS+, compared with a 61 percent MS+ average rating for quality at entry across the other Regions. Supervision was rated 76 percent MS+ in the Latin America and the Caribbean Region, compared with an average rating of 79 percent MS+ across the other Regions. America and the Caribbean: World Bank Performance Figure L.43. Latin Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: LCR = Latin America and the Caribbean; MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 301 By commitment volume, World Bank performance is 82 percent MS+ in the Latin America and the Caribbean Region during FY14–16, compared with an average 85 percent MS+ across all other Regions (figure L.43, panel b). The overall quality of M&E for projects in the Latin America and the Caribbean Region during FY14– 16 was 27 percent S+, compared with an average 36 percent S+ rating across all other Regions (figure L.44, panel a). By commitment volume, quality of M&E matched the World Bank–wide rating of 46 percent S+ and was slightly ahead of the average for all other Regions (45 percent S+: figure L.44, panel b). Figure L.44. Latin America and the Caribbean: M&E Quality Ratings, FY05–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: LCR = Latin America and the Caribbean; M&E = monitoring and evaluation; S+ = a rating of substantial or better. 302 The development outcome rating for IBRD-supported projects in the Latin America and the Caribbean Region during FY14–16 was 71 percent MS+, compared with an average 70 percent MS+ rating across all other Regions. By commitment volume, the rating was 82 percent MS+, compared with an average 80 percent MS+ rating across all other Regions (figure L.45, panel a). America and the Caribbean: Project Outcome Ratings Figure L.45. Latin by Financing Source, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: MS+ refers to rating that is “moderately satisfactory” or higher. The development outcome rating for IDA-supported projects in the Latin America and the Caribbean Region during FY14–16 was 78 percent MS+, compared with an average 73 percent MS+ rating across all other Regions. By commitment volume, the rating was 87 percent MS+, compared with an average rating of 82 percent MS+ across all other Regions (figure L.45, panel b). World Bank performance for IBRD-supported projects in the Latin America and the Caribbean Region during FY14–16 (69 percent MS+) lags behind the average across all other Regions (80 percent MS+), as does World Bank performance for IDA-supported projects in the Latin America and the Caribbean Region (67 percent MS+ versus 73 percent MS+ across all other Regions: figure L.46). M&E quality for IBRD-supported projects in the Latin America and the Caribbean Region during FY14–16 (30 percent S+) lags behind the average across all other Regions (41 percent S+), as does M&E quality for IDA-supported projects (27 percent S+ versus 33 percent S+: figure L.47). Development outcome is strong for most of the larger (by number and commitment) GPs in the Latin America and the Caribbean Region. SPL is the largest GP by commitment (around $5 billion) and development outcome rating is 100 percent MS+ by commitment and 88 percent MS+ by number (8 projects).4 Projects rated for the Governance GP are rated 100 percent MS+ by commitment (around Independent Evaluation Group | World Bank Group 303 America and the Caribbean: World Bank Figure L.46. Latin Performance Ratings by Financing Source, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; LCR = Latin America and the Caribbean; MS+ = a rating of moderately satisfactory or better. Figure L.47. Latin America and the Caribbean: M&E Quality Ratings by Financing Source, FY14–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; LCR = Latin America and the Caribbean; M&E = monitoring and evaluation; S+ = a rating of substantial or better. $3 billion) and 85 percent MS+ by number (13 projects).5 For projects rated under the largest single GP by number during FY14–16, Transport and ICT, development outcome was rated 86 percent MS+ by number (2 projects), and 81 percent by commitment (around $3 billion). Ratings for Water (11 projects, 55 percent MS+) and Energy and Extractives (15 projects, 53 percent MS+) are lower than the average for the Region overall (69 percent MS+: figure L.48). 304 America and the Caribbean: Development Outcome by Figure L.48. Latin Global Practice, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: LCR = Latin America and the Caribbean; MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 305 America and the Caribbean: World Bank Performance by Figure L.49. Latin Global Practice, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: LCR = Latin America and the Caribbean; MS+ = a rating of moderately satisfactory or better. 306 Figure L.50. Latin America and the Caribbean: Results and Performance Ratings of Evaluated IFC Investment Projects Share of Projects with Positive Ratings a. By number of projects, three-year moving average b. By IFC net commitment, three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC XPSRs as of December 12, 2017. Note: (1) Positive ratings for development outcome consists of mostly successful, successful, or highly successful (aggregated as MS+) ratings. Positive ratings for positive ratings for investment outcome and IFC’s role and contribution consist of excellent or satisfactory (aggregated as S+) ratings. (2) Due to the revision of the XPSR guidelines in 2015, the indicator, IFC’s role and contribution, has been amended s replaced with the indicator, IFC Additionality. As above, World Bank Performance in the Latin America and the Caribbean Region (68 percent MS+) is lower than the average across all other Regions (73 percent MS+). World Bank performance for several GPs is, in turn, relatively weak (Energy and Extractives, 47 percent MS+ [15 projects]; Water, 55 percent MS+ [11 projects]; and Energy and Extractives, 57 percent MS+ [7 projects]). Other GPs perform above the average for the Region (for example, SPL, 88 percent MS+ [8 projects], Governance, 85 percent MS+ [13 projects], and Transport and ICT, 82 percent MS+ [22 projects]: figure L.49). Independent Evaluation Group | World Bank Group 307 International Finance Corporation: Project-Level Performance IFC Investments Project-Level Results Compared with CY11–13 results, the overall development outcome success rating for IFC investment projects in the Latin America and the Caribbean Region has declined this review period (CY14–16). Of the 50 investment projects evaluated in CY14–16, about half (55 percent by number of projects; 56 percent by volume) had MS+ development outcome ratings (figure L.50). Positive investment outcome ratings have markedly declined compared with CY11–13, both by number of projects and by net commitment amount. Positive ratings of IFC’s role and contribution to projects evaluated in CY14–16 have also declined. Figure L.51. Latin America and the Caribbean: Performance of Development Outcome Indicators of IFC Investment Projects, by XPSR Program Years Share of Projects with Positive Ratings a. By number of evaluated projects, three-year moving average b. By IFC net commitment, three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC Investment Projects as of December 12, 2017. Note: (1) S+ = a rating of satisfactory or excellent. (2) PBS = Project Business Success. ECSUS = Economic Sustainability. E&SE = Environmental and Social Effects. PSD = Private Sector Development. These four subindicators comprise the development outcome dimension of an IFC investment project. (3) No opinion possible or not applicable ratings were excluded. 308 Figure L.52. Latin America and the Caribbean: Association between Development Outcome, IFC Work Quality and IFC’s Role and Contribution, CY14–16 By number of evaluated projects By IFC net commitment ($, millions) a. Association between development outcome and IFC work quality b. Association between development outcome and IFC role and contribution c. Association between work quality and IFC investment outcome Source: Independent Evaluation Group evaluation data for IFC investment projects as of December 12, 2017. Note: (1) IFC work quality in CY14–16 reflects the aggregate ratings of two subindicators: screening, appraisal, and structuring and supervision and administration. (2) High development outcome = highly successful, successful, and mostly successful ratings; low development outcome = mostly unsuccessful, unsuccessful and highly unsuccessful ratings; high work quality = excellent and satisfactory ratings; low work quality = partly unsatisfactory and unsatisfactory ratings; high IFC role and contribution = excellent and satisfactory ratings; low IFC role and contribution = partly unsatisfactory ratings; high IFC investment outcome = excellent and satisfactory ratings; low IFC investment outcome = partly unsatisfactory and unsatisfactory ratings. (3) Due to the revision of the Expanded Project Supervision Report guidelines in 2015, the indicator IFC’s role and contribution has been amended starting CY15, and was replaced with the indicator IFC additionality. IFC = International Finance Corporation. Independent Evaluation Group | World Bank Group 309 Lower positive (satisfactory or better, S+) ratings for three of the four indicators of development outcome contributed to the decline in overall development outcome ratings for projects evaluated in CY14–16. Whether measured by number of projects or adjusted by volume of IFC net commitments, the percentage of S+ ratings for project business success, economic sustainability, and private sector development fell sharply compared with the ratings in the CY11–13 review period (figure L.51). Ratings for E&S effects, the fourth indicator of development outcome, improved. Figure L.53. Latin America and the Caribbean: IFC Work Quality Subindicators, by XPSR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC net commitment, three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC Investment Projects as of December 12, 2017. Note: Based on the 2015 revised XPSR guidelines, IFC work quality is measured by two indicators: screening, appraisal, and structuring (SAS) and supervision and administration (SUP) in the project. IFC = International Finance Corporation; XPSR = Expanded Project Supervision Report. Evaluated projects in the Latin America and the Caribbean Region in CY14–16 indicate a close association between high or positive IFC work quality ratings and high development outcome ratings (figure L.52). High ratings for IFC’s role and contribution are also closely associated with high development outcome ratings. A similar strong association is also evident between high or positive 310 ratings for IFC work quality and IFC investment outcomes among the 53 evaluated projects in the region during this review period. Compared with CY11–13, overall IFC work quality S+ ratings dipped slightly in CY14–16 by number of projects but remained constant when measured using IFC’s commitment amounts (figure L.53). Satisfactory or better (S+) ratings for SAS, one of the two subindicators of IFC work quality either remained the same or weakened slightly compared with CY11–13, by number of evaluated projects. Figure L.54. Latin America and the Caribbean: Results and Performance Ratings of Evaluated IFC Advisory Projects, by PCR Program Years Share of Projects with Positive Ratings a. By number of evaluated projects, three-year moving average b. By IFC expenditure ($, millions), three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC PCRs as of December 12, 2017. Note: (1) The reported ratings are based on IEG’s final ratings in the completed Project Completion Reports (PCRs). (2) Positive ratings for development effectiveness consists highly successful, successful, or mostly successful (aggregated as MS+) ratings. Positive ratings for IFC’s role and contribution consist of excellent or satisfactory (aggregated as S+) ratings. Independent Evaluation Group | World Bank Group 311 Figure L.55. Latin America and the Caribbean: Ratings of Four Indicators of Development Effectiveness of Evaluated IFC Advisory Projects, by PCR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC expenditure ($, millions), three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC PCRs as of December 12, 2017. Note: (1) The reported ratings are based on IEG’s final ratings in the completed Project Completion Reports (PCRs). (2) Success ratings for each of the four indicators of development effectiveness = Satisfactory or Excellent (referred to as S+) ratings. Performance is slightly better for the subindicator SUP, if weighted by volume, which indicates the better tracking of large-sized investment projects. IFC Advisory Services Project-Level Results Of the 27 IFC Advisory Services projects evaluated for the Latin America and the Caribbean Region in FY14-FY16, nearly half (48 percent) are rated MS+ for overall development effectiveness. This overall development effectiveness rating is the lowest since IFC and IEG started evaluating IFC advisory 312 Figure L.56. Latin America and the Caribbean: Results and Performance Ratings of Evaluated MIGA Projects, by PER Program Years Share of Projects with S+ Ratings, by Number of PERs a. By main indicators of MIGA project results and MIGA performance, six-year moving average b. Ratings of the four indicators of development outcome, by number of projects, six-year moving average Source: Independent Evaluation Group Evaluation Data for MIGA PERs as of December 12, 2017. Note: (1) PERs = Project Evaluation Reports. (2) Positive ratings for Development Outcome, MIGA’s Role and Contribution and Assessment, Underwriting and Monitoring refer to Satisfactory or Excellent (referred to as S+) ratings. (3) The four subindicators of development outcome refer to BP = Business Performance; ECSUS = Economic Sustainability; E&SE = Environmental and Social Effects; PSD = Contribution to Private Sector Development. MIGA = Multilateral Investment Guarantee Agency. project (figure L.54). Success (satisfactory or better, or S+) ratings of IFC’s role and contribution in the evaluated projects, remained about the same compared with FY13–15. Compared with FY11–13, the success ratings for the four subindicators of development effectiveness have weakened considerably in the FY14–16 review period. By volume, the 20-percentage point decline in S+ ratings affected all for development effectiveness subindicators presented in figure L.55 Independent Evaluation Group | World Bank Group 313 America and the Caribbean: Project Outcome Ratings by Table L.7. Latin Country Type, FY14–16 Countries Projects Volume Evaluated Rated MS+ Total Amount Rated MS+  Country Type (no.) No. (percent) ($, millions) (percent) IDA fragile 1 8 50 136 49 IDA nonfragile 6 17 82 423 88 IBRD fragile 0  0 n.a. 0 n.a. IBRD nonfragile 16 126 68 19,317 82 IDA total 7 25 72 559 79 IBRD total 16 126 68 19,317 82 Fragile total 1 8 50 136 49 Other n.a. 3 100 50 100 Total 23 154 69 19,927 82 Source: Independent Evaluation Group data. Note: One project was excluded when measured by volume due to lack of volume data. “Other” includes countries without IDA or IBRD status or project locations that are above country level. IBRD = International Bank for Reconstruction and Development; IDA =International Development Association; MS+ = a rating of moderately satisfactory or better; n.a. = not applicable. below, including the achievement of intended project outcomes, ef ficiency in the delivery of the expected outputs and the projects’ strategic relevance to IFC’s clients. America and the Caribbean: Country-Level Performance Table L.8. Latin Ratings, FY14–17 CASCR/ CLR Review World Bank IFC MIGA Review FY Country Period Outcome Performance Performance Performance 2014 Jamaica FY10–13 MS Good n.a. n.a. Mexico FY08–13 MS Good n.a. n.a. 2015 Argentina FY10–14 MS Fair Good n.a. Costa Rica FY12–15 MS Fair n.a. n.a. Dominican FY10–13 MS Fair Good n.a. Republic El Salvador FY10–14 MS Good n.a. n.a. OECS Countries FY10–14 MS Good Fair n.a. Panama FY11–14 MS Fair n.a. n.a. Paraguay FY09–14 MU Fair n.a. n.a. 314 Table L.8, continued. CASCR/ CLR Review World Bank IFC MIGA Review FY Country Period Outcome Performance Performance Performance 2016 Bolivia FY12–15 MU Good n.a. n.a. Colombia FY12–16 MS Fair n.a. n.a. Guyana FY09–12 MS Good n.a. n.a. Haiti FY09–14 U Good n.a. n.a. Honduras FY12–15 MU Fair n.a. n.a. Uruguay FY11–15 S Good n.a. n.a. 2017 Belize FY12–15 U Fair n.a. n.a. Brazil FY12–15 MU Fair n.a. n.a. Guatemala FY13–16 MU Fair n.a. n.a. Peru FY12–16 MS Good n.a. n.a. Source: Independent Evaluation Group data. Note: All World Bank performance ratings were standardized with the following conversions: satisfactory as superior, moderately satisfactory as good, moderately unsatisfactory as fair, and unsatisfactory as poor. CASCR = country assistance strategy completion report; CLR = Completion and Learning Report; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency; MS = moderately satisfactory; MU = moderately unsatisfactory; OECS = Organisation of Eastern Caribbean States; S = satisfactory; U = unsatisfactory; n.a. = not applicable. Multilateral Investment Guarantee Agency: Project-Level Results Of the 10 MIGA projects in the Latin America and the Caribbean Region that were evaluated during the current period, FY11–16, seven were rated satisfactory or better (S+) for their development outcome. MIGA’s work quality (assessment, underwriting, and monitoring) rating also improved in FY11–16, with seven of the ten evaluated projects rated satisfactory or better (figure L.56). Of the four subindicators of development outcome, E&S effects and contribution to private sector development had the strongest ratings while S+ ratings for economic sustainability lagged the other outcome subindicators. World Bank Group: Country-Level Performance Projects were rated for seven IDA and 16 IBRD countries in the Latin America and the Caribbean Region during FY14–16. Within the IDA complement, 1 country was designated fragile. By number and volume, outcome ratings were close to the average outcome rating for the Region (respectively, 69 percent MS+, and 82 percent MS+) for projects in both IDA and IBRD country types noting the eight projects rated for the one IDA fragile country type underperformed by number (50 percent MS+) and volume (49 percent MS+; table L.7). During FY17, IEG produced CLRRs for four country program strategies in the Region, namely Belize, Brazil, Guatemala, and Peru. IEG rated development outcome moderately satisfactory in the case of Peru, moderately unsatisfactory in the case of Brazil, and Guatemala, and unsatisfactory in the case of Belize (table L.7).6 Independent Evaluation Group | World Bank Group 315 Evaluation Findings from Project Performance Assessment Reports During FY17, IEG completed eight PPARs covering 13 projects in the Latin America and the Caribbean Region. In terms of development outcome, IEG rated 4 projects satisfactory, 7 moderately satisfactory, 1 moderately unsatisfactory, and 1 unsatisfactory. When the Buenos Aires Urban Transport Project (1997–2011) was being appraised, Argentina was embarking on large-scale privatization reforms of infrastructure, including transport. The poor state of infrastructure, large operating deficits, and significant investment needs motivated the privatization of the Argentine railway system, including the Buenos Aires subway and suburban rail. The World Bank supported government commitments under the concession contract agreement with Metrovias, a private firm, that included improvements in four subway lines, as well as the Urquiza suburban rail. After the economic crisis in 2001 in Argentina, both the government and Metrovias faltered on their obligations under the concession contract. The concessionaire received compensation for the costs incurred but it lacked incentives for cost control and ef ficiency in urban rail operations. In addition, low fares maintained through government subsidies subsequently contributed to a growing fiscal deficit, while underinvestment in the system continued. The experience of this project demonstrates how a PPP scheme can underperform if sector reforms and the right price signals are not in place. The project also provided institutional support to contribute to the development of an integrated urban transport system for Greater Buenos Aires. The formation of an entity to coordinate metropolitan transport planning in Buenos Aires was supported under this project from its inception, and the Metropolitan Transport Agency was finally established in 2014 under a subsequent World Bank operation and supported political will that brought progress and functionality to the new entity. In addition, the project helped improve safety through construction of road/rail grade separated crossings and the development of integrated urban transport strategies in the metropolitan areas of Cordoba, Mendoza, Posadas, Rosario, and Tucuman. Overall, the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory. IEG prepared a clustered PPAR to assess two rural poverty reduction projects carried out by the World Bank in two Brazilian states during 2001–09: the Ceará Rural Poverty Reduction Project and the Santa Catarina Natural Resource Management Project. The projects shared the objective of improving the economic and social conditions of the rural poor, but under very different conditions. Rural poverty rates in the two states were, and remain, opposites. At design, Ceará had the second highest rural poverty rate among Brazilian states, and Santa Catarina had the lowest. IEG’s assessment found that the Ceará CDD project substantially improved access by the poor to water and electricity and to some extent, productive capacity (mainly tractors), but it had negligible effect on health and education. The Santa Catarina Natural Resource Management project achieved its objective of integrating E&S sustainability into development and poverty reduction strategies. However, it only modestly enhanced local governance and community participation in decision making. The aim of reversing land degradation and protecting natural resources was only modestly 316 achieved based on data obtained on water and soil quality for targeted microcatchments. A key lesson that emerged from Ceará is that CDD projects may not be the best option to provide access to “last mile” public services, such as electricity. The provision of last mile services requires clear participation criteria underpinned by accurate welfare data, and a system of aid allocation that is transparent and accounted for by beneficiaries. The two projects confirm that area poverty targeting arrangements, unaccompanied by clear selection criteria, are likely to result in reaching less of the rural poor than is desired. IEG also found that an integrated poverty-environment approach, as was the case in Santa Catarina, may be a better option than addressing these objectives separately, through stand-alone projects. In Ceará, assistance to farmers, particularly on productive (tractor) subprojects, did not benefit enough from land management techniques, resulting in farming technologies that could have an adverse effect on soil quality. Nevertheless, as water availability and adaptation to the adverse impacts of periodic droughts are the main natural resource-related challenges in Ceará, the project’s support for small water infrastructure partly addressed these challenges. IEG rated overall development outcome moderately satisfactory for the Ceará project, one step down from the self-rating arrived at in the project’s ICR. IEG rated the overall development outcome satisfactory for the Santa Catarina project, one step up from the moderately satisfactory self-rating arrived in that project’s ICR. The Pernambuco Education Results and Accountability Project (2009–15) set out to address challenges identified by the research undertaken by the Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, that is, student scores for fundamental and secondary education in Pernambuco were among the lowest in Brazil. On April 14, 2009, an IBRD loan of $154 million was approved for the State Government of Pernambuco (with a guarantee by the Government of Brazil). The operation financed a portion of the government’s broader program through a sectorwide approach of eligible expenditure programs, such as a school standards program, the State-Level Student Assessment System, literacy, and an overage correction program. The operation also disbursed based on performance conditions. The theory of change for this project needs to be viewed in relation to the broader government program (not just the items financed by the World Bank), as the government simultaneously implemented complementary activities that also supported attainment of the objectives. As such, attribution of results for this operation goes beyond World Bank financing, to the broader government program. There has been continuity in education policies and programs in the state government, despite there being multiple governors. This continuity helped stabilize the performance monitoring system so that data served as a feedback loop to the system. Equity was a stated objective, but it was not adequately addressed, as a performance gap remained between rural and urban schools. Overall, both the World Bank’s ICR and IEG’s independent field-based evaluation assessed development outcome to be satisfactory, based on state and federal data that demonstrated significant improvement in education quality and in student flow (reduced rates of repetition and dropout). Independent Evaluation Group | World Bank Group 317 Over the last 30 years, Colombia has increased its efforts to prevent and mitigate natural hazards and deal ef ficiently with their consequences when they strike. This World Bank DRM DPL (2009–12) sought to strengthen the government’s program for reducing risks resulting from adverse natural events through a policy program targeted at improving risk identification and monitoring, increasing risk prevention and mitigation, strengthening the national system for disaster management and prevention, and reducing the fiscal vulnerability of the state to natural disasters. The budget support operation had a catastrophe deferred drawdown option for disbursement of the loan to facilitate quick-release financing in the event of a disaster. The program was successful in advancing reforms in the four policy areas: significantly expanded disaster monitoring stations made it possible to know more precisely who was at risk and what risks they faced; DRM plans and investments increased across all levels of government; legal and institutional frameworks for disaster management were strengthened; and the government advanced in understanding contingent liabilities from natural disasters, assessing options for public insurances, and improving the legal framework for infrastructure concessions to reduce the fiscal vulnerability to natural disasters. On the other hand, the original approach to resettle the residents from the Galeras volcano zone encountered many dif ficulties and had to be modified. The program also had a deficient M&E framework, which poorly captured the program’s scope and results, and limited learning about the catastrophe deferred drawdown option instrument. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the outcome rating was satisfactory, based on data going beyond the formal monitoring data. Water Supply and Sanitation (WSS) in Colombia has improved in recent decades. Between 1990 and 2010, access to improved sanitation increased from 67 percent to 82 percent, and access to improved water sources increased from 89 percent to 94 percent, but coverage in rural areas continues to lag. IEG completed a clustered PPAR covering three projects—the Cartagena Water Supply, Sewerage and Environmental Management Project (1999–2009); the Water Sector Reform Assistance Project (2001–10); and the Water and Sanitation Sector Support Project (2005–11)— that form part of the second-generation WSS projects in Colombia. These projects benefited from lessons learned in the 1990s from World Bank–supported WSS projects in that country, although somewhat different in reach, focus, nature, and geographical coverage. The first two portray a somewhat logical development in the World Bank’s support to Colombia’s water sector as they tried to replicate the private sector operator model but at a smaller scale in medium and small municipalities. In contrast, the third focused on building infrastructure and support to both public sector as well as private sector run utilities. The World Bank’s design of these projects created a synergy with the changes and developments within Colombia’s water sector and its reforms by focusing on the provision of water supply and sewerage services to the most underserved sections of Colombian society—the poor segments of the income strata, some rural areas, and small and medium-sized cities. While at different scale, the three projects shared components of what was at the time an innovative focus on private sector participation, including (i) sector reform to support private sector participation; and (ii) strengthening 318 of private sector participation entities, in addition to supporting the actual expansion of WSS services. The key lessons from IEG’s assessment are as follows: consolidation of smaller WSS operations servicing poorer neighborhoods can foster economies of scale and cross-subsidization in achieving financial sustainability at the aggregate level; small municipalities with limited service coverage require large capital investments, and because of their lack of financial autonomy, government subsidies would be required until full cost recovery is achieved; it is crucial to phase and/or sequence project activities to ensure achievement of the objectives of utility operators; management capacity of the utility can be strengthened by contracting a competent operator; a carefully designed M&E system and comprehensive planning are essential for success. The objectives for the Cartagena Water Supply, Sewerage and Environmental Management Project (1999–2009) were to (i) improve the water and sewerage services; (ii) facilitate the environmental cleanup of water bodies surrounding the project area and to (iii) improve the sustainability of water and sewerage services through a private sector participation model. IEG rated the overall project outcome satisfactory. The objectives of the Water Sector Reform Assistance Project (2001–10), were to (i) facilitate an increased participation of the private sector in the management and operation of water utilities, (ii) expand the coverage of water supply and sewerage services provided in participating municipalities, (iii) facilitate access to water and sewerage services by the low-income population, (iv) improve environmental protection practices, and (v) define rural waste and sanitation policies and develop adequate methods for increasing coverage for both water and sewerage services in the rural sector. Overall outcome of the project was rated moderately unsatisfactory based on the modest achievement of two of the four objectives, modest design, and modest ef ficiency associated with a long implementation delays and increased budget requirement. The project objective of the Water and Sanitation Sector Support Project (2005–11) was “to improve the access to WSS services in rural and urban communities throughout the Borrower’s territory.” IEG rated the outcome moderately satisfactory. Panama achieved high economic growth in 2000–16 that contributed to significant poverty reduction. During this period, Panama expanded its social assistance programs and in 2006, it introduced the conditional cash transfer Red de Oportunidades (Opportunity Network) with the objective of inserting poor households in the dynamics of national development, incentivizing the use of basic health and education services. The World Bank sought to support the Red de Oportunidades through the Social Protection Project (2007–14). The project was supported by $24 million of IBRD financing, and $22.8 million cofinancing from the Inter-American Development World Bank, and the Government of Panama. The project aimed to improve the management and operation of Red de Oportunidades, increase beneficiaries’ demand for education and maternal and infant health services, expand the provision of health and nutrition services in the indigenous communities covered by the Red de Oportunidades, Independent Evaluation Group | World Bank Group 319 and enhance the government’s capacity to design, implement, and monitor coherent and ef ficient social sector policies and interventions. IEG found that the project improved the management and operation of the Red de Oportunidades, and strengthened the provision of health and nutrition services. However, it was not possible to determine if project support and targeted communication (to improve knowledge among beneficiaries about their rights, how to access services, and their co-responsibilities) improved demand for health and education services. Finally, the improvement of government’s capacity to design, implement, and monitor coherent and ef ficient social sector policies and interventions was rated modest, as the capacity of the Technical Secretariat of the Social Cabinet to coordinate social policies across ministries remained weak throughout the life of the project. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory. The Health Equity and Performance Improvement Project (HEPI: 2008–14) sought to address the relatively poor (compared with the rest of the population) access to health services, and poor health outcomes among rural and indigenous communities in Panama. For example, the national average maternal mortality rate in 2007 was estimated at 59.4 deaths per 100,000 live births, but indigenous and rural communities reported rates that were between 6 to 8 times higher. In 2009, the infant mortality rate in the indigenous communities was almost twice the national average of 12.2 deaths per 1,000 live births. To expand basic health and nutrition services, the Government of Panama introduced the Coverage Extension Strategy, based on contracting-out to nongovernmental organizations for the provision of a Comprehensive Basic Package of Health Services. The HEPI project financed the Coverage Extension Strategy to cover about 180,000 people living in rural communities, and the provision of technical assistance to strengthen the entire primary health services networks of the rural communities and improve the performance of the overall national health system. The HEPI project was supported by $40 million of IBRD financing, with government cofinancing of $18.73 million. The objective to increase access in selected underserved rural communities to quality basic health services known to improve mother and child health was substantially achieved: the mobile health teams achieved all targets related to population coverage, and to the provision of health services. However, improvement in the capacity of the primary health care networks was more modest, as were achievement related to the development of strategic planning, regulatory, and monitoring mechanisms known to improve health system performance. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory IEG completed a clustered assessment of two projects in Peru. The Lima Water Rehabilitation and Management Project (2002–13) received $170 million from the World Bank, while the National Rural Water Supply and Sanitation Project (2002–13) received $110 million. The projects closed eight years and four-and-a-half years, respectively, after their planned closing dates. Under the Lima project, although the plans for privatizing the utility operations were dropped by the government, the alternative approach of upgrading management capacity and introducing performance-based incentives yielded the targeted physical infrastructure and financial results. Expanding services to the 320 urban poor exceeded the revised targets; however, the pilot exercise for using low-cost “condominial” water and sewerage networks did not yield significant results. Under the “Rural” project, the principal objective of increasing the sustainable use of WSS facilities in rural areas and small towns was underachieved. There were no significant differences in desired behavior (hygiene practices) observed between populations in the project control areas and those outside the control areas. Lack of capacity to manage the facilities persisted, and there was a lack of administrative and financial support down the chain of government agencies, municipalities and local government, and rural community water and sanitation services associations. An important lesson from the Lima project is that a strong focus on project outcomes and adequate accountability mechanisms are key factors in achieving significant improvements in service provision. An important lesson from the rural project is that, while community participation in planning and operations underpins sustainability, adequate training support, financial incentives, and contractual arrangements are necessary for continued and effective participation. Overall, the World Bank’s ICR concluded that the Lima project’s outcome rating was satisfactory and the outcome rating for the rural project was moderately unsatisfactory. IEG’s independent field-based evaluation concluded that project outcome rating for the Lima project was moderately satisfactory, and for the rural project, IEG’s assessment rated project outcome unsatisfactory. In 2006, Peru’s economy was performing well. To make a significant leap in its development, the government designed a comprehensive reform program that was supported by the World Bank’s programmatic DPLs—Fiscal Management and Competitiveness Programmatic DPLs (2007–15)—that sought to support improvement in the functioning of Peru’s public sector institutions and business environment. In September 2008, Peru’s financial conditions were temporarily, but significantly, affected by the onset of the global crisis. Consequently, the second loan and its supplement were granted with a deferred drawdown option to allow the government to proactively manage and contain the risk from the global economic crisis, as a preventive measure, while remaining committed to the reform program. The programmatic series was successful in making public sector processes more transparent and accessible, increasing sustainability of fiscal policy, strengthening budget reporting and planning, improving management of public spending, and expanding international trade. However, in many reform areas supported by the program, including the tax system, intergovernmental transfers, cost of doing business, and financial deepening, advances fell short of expectations or were reversed after program closing, which suggests that the root causes of the problems have not been satisfactorily addressed. The extent of the program’s contribution to risk management during the global financial crisis is dif ficult to assess for lack of metrics. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the outcome rating was moderately satisfactory based on data monitored under the program. Independent Evaluation Group | World Bank Group 321 Evaluation Findings from Country Program Performance During FY17, IEG produced CLRRs for country program strategies in four countries in the Latin America and the Caribbean Region, namely Belize, Brazil, Guatemala, and Peru. IEG rated development outcome moderately satisfactory in the case of Peru, moderately unsatisfactory in the case of Brazil and Guatemala, and unsatisfactory in the case of Belize. The Belize CPS FY12–15 had three focus areas: (i) policies and strategies for mainstreaming of natural resources and climate resilience; (ii) institutional capacity strengthening for natural resource management and climate change; and (iii) investments to strengthen climate resilience. IEG’s CLR review found that the World Bank supported the preparation of the National Environmental Policy and Strategy, built technical and fiduciary capacity of national entities to access climate funds, and created awareness on environmental sustainability issues. However, the program was unable to enhance the effectiveness of the Environmental Impact Assessment system, nor could it climate proof key infrastructure, increase ecosystem resilience to climate change impacts, or enhance preparedness through improved DRM institutions. IEG found the focus areas were broadly in line with the government’s priorities, but the program objectives were too broad and could not be fully achieved using the available World Bank interventions. IEG identified the following lessons. First, it is critical that the principle of selectivity is applied not only in the choice of sector, but also to other areas of the program. In the case of Belize, the World Bank was strategic in choosing on the natural resource and climate resilience sector where it had comparative advantage, but its selectivity did not extend to the choice of focus areas and program objectives. Second, a well-designed results framework requires appropriate and timely interventions that support the program objectives. In this case, the results framework did not fully account for the contributions of an ongoing lending operation which delivered most of its results during the CPS period. Finally, the World Bank did not use the PLR to update its critical milestones, though there was the need for it, and this affected the performance of the indicators. Overall, IEG rated the development outcome unsatisfactory noting three of the 11 objectives were mostly achieved, four were partially achieved, and four were not achieved. IEG’s CLRR found that the FY12–15 CPS for Brazil was aligned with government strategies that sought to address the need to accelerate economic growth and strengthen resilience to international shocks; to reduce inequality and offer human development opportunities to all, especially women; and to enhance environmental sustainability and resilience. The CPS was focused on providing support to subnational governments—that absorbed 61.0 percent of lending during the current CPS compared with 32.5 percent for the previous CPS - to help enhance implementation of national policies, while also providing knowledge intensive services at the federal level. The four focus areas of the CPS were (i) ef ficiency of public and private investments; (ii) quality and provision of public services for low-income households; (iii) regional economic development, and (iv) sustainable natural resource management and climate resilience. Compared with the previous CPS, the composition of World Bank operations shifted toward multisector DPOs and subnational lending that covered, 322 for example, public sector management, transportation, water, rural development, health, energy, environment, education, and social protection. IEG found that IFC played an important role in Brazil, particularly in supporting micro, small, and medium enterprises through investments in local financial institutions and through investment and advisory work with PPPs. During the CPS period, Brazil was IFC’s third largest client in terms of its long-term loans and equity investments, and its largest client in Latin America. However, despite IFC’s efforts to provide evidence beyond the indicators of the results matrix, IEG found development contributions are sometimes dif ficult to gauge accurately because of the lack of relevant indicators. Overall World Bank Group performance was rated fair given, for example, alignment and relevance of objectives, and selectivity toward poor states that strengthened the focus on poverty reduction. That said, design would have benefited from a more deliberate programmatic approach to policy lending, more efforts to achieve synergies between the World Bank and IFC, and more attention to the adequacy of outcome indicators. Implementation also faltered largely due to Brazil’s deteriorating economy and political crisis, which constrained fiscal resources and reduced client commitment (although weaknesses in World Bank supervision before FY15 may have also played a role). IEG agrees with CLR lessons, including, for example, the need for strong engagement with central government in a program that is largely focused on subnational governments, and that attention to fiscal sustainability at the subnational level will improve the performance of DPOs that focus on service delivery. IEG provides the following three lessons: DPO lending in a programmatic setting may help the World Bank to better achieve a more consistent sequencing of policies and achievement of welfare outcomes; World Bank-IFC collaboration can be strengthened through more planning and coordination between the two institutions; and finally, indicators that accurately reflect outcomes are critical to enable program evaluation. IEG rates overall development outcome moderately unsatisfactory due to mixed program outcomes (for example, modest progress on public sector management, PPPs, and productivity policies); little evidence of improved quality in health services despite improvements in social assistance, health coverage, and gender mainstreaming; and no progress toward the PPPs on irrigation that IFC sought. The World Bank Group’s CPS for Guatemala during FY13–16 built on the government’s ambitious plan to address key economic challenges, its broader reform agenda, which was supported by three pacts (fiscal reform, zero hunger, and security, peace and justice), and the National Competitiveness Agenda 2012–21. The CPS had five results areas but, at the PLR stage (September 2015), the results framework was significantly revised and consolidated to focus on three specific objectives: (i) enhance tax administration; (ii) improve access to and ef ficiency of social programs; and (iii) increase productive opportunities. However, there was no substantive change in the program itself. Under the first objective, the tax base was widened, but the overall tax-to-GDP ratio did not improve due to deterioration in the collection of indirect taxes, particularly on customs. Under the second objective, there was increased access to social programs in the areas of World Bank intervention, with overall improvements in maternal mortality. However, malnutrition among children remains chronic and the Independent Evaluation Group | World Bank Group 323 impact of institutional reforms on ef ficiency of social spending was not realized. Under objective 3, program performance was poor, with partial achievement of PLR targets. World Bank projects in rural development and SME productivity have not been able to produce sustainable and scalable models of support. While IFC investments contributed to increased lending to micro, small, and medium enterprises, Advisory Services products were not effective, and Guatemala continues to lag behind its peers in terms of financial inclusion. IEG rated World Bank Group performance fair, on the basis that the design of the CPS was too ambitious, beyond the government’s implementation capacity, and the mix of instruments was not consistent with the country’s institutional weaknesses. There was also too much reliance on AAAs, which did not provide the same level of support and engagement as investment policy financing projects. The results framework was poorly constructed, with weak linkages between the actual program and objectives, and outcome indicators that were not well defined. Changes made at PLR stage were late in the life of the program, precluding the opportunity to significantly improve program results. A major IFC investment in ports was put on hold due to ongoing corruption issues, and several planned Advisory Services products were not pursued due to lack of client support. IEG generally agrees with the CLR lessons and recommendations, including the need for World Bank Group to be a steadfast and flexible partner, recognizing that change will be incremental and over the long term, the importance of simple design and a selective program, and the need to consider reputational risk in all aspects of engagement. IEG adds lessons pertaining to: the need to achieve the right mix of World Bank Group instruments aligned with the country’s institutional capacity; and the benefits of implementing a timely delivery of the PLR. The CLR provided a moderately satisfactory rating for overall development outcome which was downgraded to moderately unsatisfactory in IEG’s review. The World Bank Group CPS for Peru (FY12–16) had four focus areas: increased access and quality of social services for the poor; connecting the poor to services and markets; sustainable growth and productivity; and improved public sector performance for greater inclusion. Focus areas I–III performed well. The program improved institutional design in support of access to social protection programs, and delivered quality social services to the poor. A greater proportion of prenatal women received care, and children under age 3 received growth checkups. The rural poor were connected to services and markets, and gained better access to electricity, potable water, and to road networks. The value of production and income increased in targeted regions. The program did less well in promoting new technologies among farmers and in designing instruments for marine protected areas. Focus area IV saw mixed results. It promoted good governance in municipalities and helped to prioritize public investments, but did less well in reducing disaster risk. Design was selective, and addressed the government’s priorities, noting that four development policy financings accounted for 80 percent of lending. The results framework was revised (while retaining the original pillars) at midterm. In focus areas 3 and 4, the links between the results framework and the development policy financings were weak. 324 Despite revisions, the outcomes and indicators at midterm did not adequately reflect the shift in government priorities and the adjustment of the World Bank Group program. IEG rated World Bank Group performance good. The focus areas were well aligned with the government priorities and the proposed mix of instruments of lending and AAAs were appropriate to achieve the desired outcomes. The following lessons are offered: reimbursable advisory services play a key role in providing customized knowledge and advisory services; there needs to be focus at the subnational level to address regional disparities within a framework of effective policies and with complementary support at the national level; remaining flexible and initiating timely mid-course corrections are important, when necessary; and analytical and knowledge services should be combined with lending to achieving overall results. IEG rated overall development outcome for the program moderately satisfactory. Figure L.57. Middle East and North Africa: Project Outcome Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: MS+ refers to rating that is “moderately satisfactory” or higher Independent Evaluation Group | World Bank Group 325 Middle East and North Africa Region Project- and Country-Level Performance This note presents the results and performance separately for the World Bank, IFC, and MIGA given the different time frames, operating models, and clients of the three institutions. Unlike previous Results and Performance of the World Bank Group (RAP) reports, the 2017 RAP (RAP17) excludes East and North Africa: World Bank Performance Figure L.58. Middle Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: Middle East and North Africa = Middle East and North Africa; MS+ = a rating of moderately satisfactory or better. 326 Afghanistan and Pakistan in IFC’s regional data aggregation for the Middle East and North Africa Region. The two countries were classified as part of Middle East and North Africa following past IFC country classification. For RAP17, the two countries are classified as part of the South Asia Region to reflect IFC’s current organizational structure.7 The World Bank: Project-Level Performance For the current review period, FY14–16, development outcome was rated 61 percent MS+ for projects in the Middle East and North Africa Region, below the average for all other Regions Figure L.59. Middle East and North Africa: M&E Quality Ratings, FY05–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: Middle East and North Africa = Middle East and North Africa; M&E = monitoring and evaluation; S+ = a rating of substantial or better. Independent Evaluation Group | World Bank Group 327 (74 percent) and the World Bank–wide average (73 percent MS+) over the same period (figure L.57, panel a). East and North Africa: Project Outcome Ratings by Figure L.60. Middle Financing Source, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; Middle East and North Africa = Middle East and North Africa; MS+ = a rating of moderately satisfactory or better. By commitment volume, the satisfaction rating was 70 percent MS+ for the Middle East and North Africa Region, compared with an average 85 percent MS+ across all other Regions and a World Bank–wide average of 84 percent MS+ (figure L.57, panel b). Overall World Bank performance, a composite of ratings for quality at entry and project supervision, in the Middle East and North Africa Region during FY14–16, was 65 percent MS+, compared with a 73 percent MS+ average rating across all other Regions (figure L.58, panel a). Within the overall World Bank performance rating in the Middle East and North Africa Region, Quality at entry was rated 61 percent MS+ (the same as the average across the other Regions). Supervision was rated 64 percent MS+ in the Middle East and North Africa Region, compared with an average rating of 80 percent MS+ across the other Regions. By commitment volume, World Bank performance was rated 71 percent MS+ in the Middle East and North Africa Region, below the 85 percent MS+ average rating across all other Regions (figure L.58, panel b). 328 East and North Africa: World Bank Performance Ratings by Figure L.61. Middle Financing Source, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: RD = International Bank for Reconstruction and Development; IDA = International Development Association; Middle East and North Africa = Middle East and North Africa; MS+ = a rating of moderately satisfactory or better. East and North Africa: M&E Quality Ratings by Financing Figure L.62. Middle Source, FY14–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; Middle East and North Africa = Middle East and North Africa; M&E = monitoring and evaluation; S+ = a rating of substantial or better. Independent Evaluation Group | World Bank Group 329 East and North Africa: Development Outcome by Global Figure L.63. Middle Practice, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: Middle East and North Africa = Middle East and North Africa; MS+ = a rating of moderately satisfactory or better. 330 The overall quality of M&E for projects in the Middle East and North Africa Region was 31 percent S+, compared with an average 34 percent S+ rating across all other Regions (figure L.58, panel a). By commitment volume, the quality of M&E in the Middle East and North Africa Region were rated 32 percent S+ against an average 46 percent S+ rating for all other Regions. The development outcome rating for IBRD-supported projects in the Middle East and North Africa Region during FY14–16 was 62 percent MS+, compared with an average 77 percent MS+ rating across all other Regions (figure L.59, panel a). By commitment volume, the rating was 77 percent MS+ in the Middle East and North Africa Region, compared with an average 86 percent MS+ rating across all other Regions (figure L.60, panel b). The development outcome rating for IDA-supported projects in the Middle East and North Africa Region during FY14–16 is 47 percent MS+ (figure L.59, panel a), compared with an average 74 percent MS+ rating across all other Regions. By commitment volume, the rating was 51 percent MS+ in the Middle East and North Africa Region, compared with an average rating of 84 percent MS+ across all other Regions (figure L.60, panel b). World Bank performance for IBRD-supported projects in the Middle East and North Africa Region during FY14–16 (65 percent MS+) lags behind the average across all other Regions (76 percent MS+), as does World Bank performance for IDA-supported projects (47 percent MS+ versus 73 percent MS+: figure L.61). M&E quality ratings for IBRD-supported projects in the Middle East and North Africa Region during FY14–16 (31 percent S+) lags behind the average across all other Regions (37 percent S+), as do M&E quality ratings for IDA-supported projects (13 percent S+ versus 33 percent S+: figure L.62). There were 62 projects valued at almost $3.50 billion rated for the Middle East and North Africa Region during FY14–16. The small number of projects rated means correspondingly small numbers of projects within and across the 14 GPs. The average outcome rating for the Region is 61 percent MS+ and for SURR, the largest single GP by number (14 projects), the outcome rating was 50 percent MS+. For the next largest practice by number, Water (9 projects), outcome rating was 33 percent MS+ (figure L.63, panel a). The Water ($637 million) and Finance and Markets ($604 million) GPs are the largest by commitment, and ratings by commitment are, respectively, 29 percent MS+ and 100 percent MS+ (figure L.63, panel b). Independent Evaluation Group | World Bank Group 331 Figure L.64. Middle East and North Africa: Results and Performance Ratings of Evaluated IFC Investment Projects Share of Projects with Positive Ratings a. By number of projects, three-year moving average b. By IFC net commitment, three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC XPSRs as of December 12, 2017. Note: (1) Results for the Middle East and North Africa Region exclude projects in Pakistan and Afghanistan, which have been reclassified as part of South Asia to reflect current IFC organizational structure. In prior years, the two countries were classified under the Middle East and North Africa Region in IFC’s project portfolio databases. (2) Positive ratings for development outcome consists of mostly successful, successful, or highly successful (aggregated as MS+) ratings. Positive ratings for investment outcome and IFC’s role and contribution consist of excellent or satisfactory (aggregated as S+) ratings. (3) Due to the revision of the Expanded Project Supervision Report guidelines in 2015, the indicator, IFC’s role and contribution, has been amended starting CY15, and was replaced with the indicator, IFC Additionality. 332 Figure L.65. Middle East and North Africa: Development Outcome Subindicators of IFC Investment Projects, by XPSR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC net commitment, three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC Investment Projects as of December 12, 2017. Note: (1) Results for the Middle East and North Africa Region exclude projects in Afghanistan and Pakistan, which have been reclassified under the South Asia Region (South Asia Region) to reflect current IFC organizational structure. In prior years, the two countries were classified under the Middle East and North Africa Region in IFC’s project portfolio databases. (2) S+ = a rating of Satisfactory or Excellent. (3) PBS = Project Business Success. ECSUS = Economic Sustainability. E&SE = Environmental and Social Effects. PSD = Private Sector Development. These four subindicators comprise the development outcome dimension of an IFC investment project. (4) No Opinion Possible or Not Applicable ratings were excluded. Independent Evaluation Group | World Bank Group 333 East and North Africa: Association between Development Figure L.52. Middle Outcome, IFC Work Quality and IFC’s Role and Contribution, CY14–16 By number of evaluated projects By IFC net commitment ($, millions) a. Association between development outcome and IFC work quality b. Association between development outcome and IFC role and contribution c. Association between work quality and IFC investment outcome Source: Independent Evaluation Group Evaluation Data for IFC Investment Projects as of December 12, 2017. Note: (1) Results for the Middle East and North Africa Region exclude projects in Afghanistan and Pakistan, which have been reclassified under the South Asia region (South Asia Region) to reflect current IFC organizational structure. In prior years, the two countries were classified under the Middle East and North Africa region in IFC’s project portfolio databases. (2) IFC work quality in CY14–16 reflects the aggregate ratings of two subindicators: screening, appraisal, and structuring and supervision and administration. (2) High development outcome = highly successful, successful, and mostly successful ratings; low development outcome = mostly unsuccessful, unsuccessful and highly unsuccessful ratings; high work quality = excellent and satisfactory ratings; low work quality = partly unsatisfactory and unsatisfactory ratings; high IFC role and contribution = excellent and satisfactory ratings; low IFC role and contribution = partly unsatisfactory ratings; high IFC investment outcome = excellent and satisfactory ratings; low IFC investment outcome = partly unsatisfactory and unsatisfactory ratings. (3) Due to the revision of the Expanded Project Supervision Report guidelines in 2015, the indicator IFC’s role and contribution has been amended starting CY15, and was replaced with the indicator IFC additionality. IFC = International Finance Corporation. 334 Figure L.67. Middle East and North Africa: IFC Work Quality Performance, by XPSR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC net commitment, three-year moving average Source: Independent Evaluation Group evaluation data for IFC investment projects as of December 12, 2017. Note: (1) Results for the Middle East and North Africa Region exclude projects in Afghanistan and Pakistan project results, which have been reclassified under the South Asia Region (South Asia Region) to reflect current IFC organizational structure. In prior years, the two countries were classified under the Middle East and North Africa Region in IFC’s project portfolio databases. (2) Based on the 2015 revised XPSR guidelines, IFC work quality is measured by two indicators: screening, appraisal, and structuring (SAS) and supervision and administration (SUP) in the project. IFC = International Finance Corporation; XPSR = Expanded Project Supervision Report. The International Finance Corporation: Project-Level Performance IFC Investments Project-Level Results Of the 16 investment projects in the Middle East and North Africa Region that were evaluated in the current review period CY14–16, only six projects (38 percent by number of projects, or 30 percent if measured by commitment amount) had MS+ development outcome ratings (figure L.64).8 Satisfactory or better (S+) ratings for IFC investment outcome in these ratings are at an all-time low at 36 percent, if the amount of IFC commitment is considered. IFC’s role and contribution in the evaluated projects Independent Evaluation Group | World Bank Group 335 Figure L.68. Middle East and North Africa: Results and Performance Ratings of Evaluated IFC Advisory Projects, by PCR Program Years Share of Projects with Positive Ratings a. By number of evaluated projects, three-year moving average b. By IFC expenditure ($, millions), three-year moving average Source: Independent Evaluation Group evaluation data for IFC PCRs as of December 12, 2017. Note: (1) Results for the Middle East and North Africa Region exclude projects in Afghanistan and Pakistan, which have been reclassified under the South Asia Region (South Asia Region) to reflect current IFC organizational structure. In prior years, the two countries were classified under the Middle East and North Africa Region in IFC’s project portfolio databases. (2) The reported ratings are based on IEG’s final ratings in the completed Project Completion Reports (PCRs). (3) Positive ratings for development effectiveness consists highly successful, successful, or mostly successful (aggregated as MS+) ratings. Positive ratings for IFC’s role and contribution consist of excellent or satisfactory (aggregated as S+) ratings. was considered poor, with nearly half (56 percent) rated S+, by number of projects but only about a third had S+ ratings, if the ratings are weighted by IFC’s net commitment. Except for the subindicator E&S effects, all three other subindicators of development outcome fell to an all-time low in CY14–16, by number of evaluated projects. If results are weighted by volume, all 336 four outcome subindicators performed poorly compared with CY11–13 (figure L.65). The evaluated projects’ business viability and contribution to economic growth of Middle East and North Africa countries has performed poorly compared with expectations at approval. Only 5 of the 16 evaluated projects in CY14–16 were rated positively for their business viability and economic sustainability. By volume, only 22 percent had positive ratings for these two indicators. Figure L.65 illustrates the precipitous percentage point drop in the S+ ratings for the periods CY11–13 and CY14–16. Among the 16 projects evaluated in CY14–16, low development outcome and low IFC work quality ratings are strongly associated, whether by number of projects or weighted by IFC amounts. The same association is observed between development outcome and IFC role and contribution ratings, especially by IFC volume amount. As shown in Figure L.66, half of the projects with low (mostly unsuccessful or below) development outcome ratings also had low (partly unsatisfactory Figure L.69. Middle East and North Africa: Ratings of Four Indicators of development effectiveness of Evaluated IFC Advisory Projects, by PCR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC expenditure ($, millions), three-year moving average Source: Independent Evaluation Group evaluation data for IFC PCRs as of December 12, 2017. Note: (1) Results for the Middle East and North Africa Region exclude projects in Afghanistan and Pakistan, which have been reclassified under the South Asia Region (South Asia Region) to reflect current IFC organizational structure. In prior years, the two countries were classified under the Middle East and North Africa Region in IFC’s project portfolio databases. (2) The reported ratings are based on IEG’s final ratings in the completed Project Completion Reports (PCRs). (3) Positive or success ratings for each of the subindicators indicators of development effectiveness = Satisfactory or Excellent (referred to as S+) ratings. Independent Evaluation Group | World Bank Group 337 or unsatisfactory) IFC work quality ratings. Nearly half (44 percent by number of projects or 56 percent by volume) of projects that had low IFC role and contribution ratings also had low (mostly unsuccessful or below, MU-) development outcome ratings. Low investment outcome rating is also associated with low development outcome rating, by volume. In sum, improving IFC’s work quality ratings may help achieve better development outcome and investment outcome ratings. Both subindicators of IFC work quality underperformed in CY14–16. Satisfactory or better (S+) ratings for SAS and SUP have weakened further compared with CY11–13 results, which may have affected the aggregate development outcome and the four development outcome subindicators’ ratings (figure L.66). Projects with less than satisfactory (partly unsatisfactory or unsatisfactory) ratings for SAS also had poor ratings for project business success and economic sustainability. IFC Advisory Services Project-Level Results Nine IFC Advisory Services projects in the Middle East and North Africa Region were evaluated in FY14–16, and four projects were rated MS+ for overall development effectiveness or development effectiveness (figure L.68). If adjusted by the amount of IFC expenses in the nine projects, the share of MS+ development effectiveness rating in FY14–16 (36 percent) represents a drop by 17 percentage point compared with FY11–13 results. Positive ratings for IFC’s role and contribution to the projects have also declined, by volume of projects. For the Middle East and North Africa projects evaluated in FY14–16, the overall development effectiveness success rating reflects the positive rating for the subindicator, outcome achievement (figure L.69). Although success (S+) ratings for the subindicators strategic relevance, output achievement, and ef ficiency improved in FY14–16, the overall development effectiveness success rating for the period reflected the success rating for the development effectiveness subindicator outcome achievement, both by number of projects and by volume. Of the nine evaluated projects in the Middle East and North Africa Region in FY14–16, the same four advisory projects had positive development effectiveness ratings as well as outcome achievement ratings. Multilateral Investment Guarantee Agency: Project-Level Results There were only five MIGA projects in the Middle East and North Africa Region that have been evaluated in the FY11–16 period (and since MIGA created an independent evaluation function in FY03). Four of the five projects had satisfactory or better (S+) ratings for development outcome. All 338 five projects had (S+) ratings for the indicator, MIGA’s role and contribution and four of five projects were rated S+ for the quality of MIGA’s work at assessment, underwriting and monitoring. World Bank Group: Country-Level Performance Projects were rated for two IDA and six IBRD countries in the Middle East and North Africa Region during FY14–16. Both IDA countries were designated fragile, as were two of the IBRD-supported projects. Average overall outcome rating for projects rated in the Region during FY14–16 is 61 percent MS+. Projects rated for both the IBRD Fragile and nonfragile countries performed above the average (63 percent MS+ for both) and the IDA fragile projects performed below the average (54 percent MS+). By volume, projects in IBRD Nonfragile countries performed above the relevant average (76 percent MS+ versus 70 percent MS+) whereas projects in both the IDA Fragile and IBRD fragile countries performed below the average (respectively, 48 percent MS+, and 64 percent MS+; table L.9). East and North Africa: Project Outcome Ratings by Country Table L.9. Middle Type, FY14–16 Countries Projects Volume Evaluated Rated MS+ Total Amount Rated MS+  Country Type (no.) No. (percent) ($, millions) (percent) IDA fragile 2 13 54 393 48 IDA nonfragile 0 0 n.a. 0 n.a. IBRD fragile 2 8 63 646 64 IBRD nonfragile 4 30 63 2,304 76 IDA total 2 13 54 393 48 IBRD total 6 38 63 2,951 73 Fragile total 4 21 57 1,039 58 Other n.a. 11 64 152 72 Total 8 62 61 3,496 70 Source: Independent Evaluation Group data. Note: One project was excluded when measured by volume due to lack of volume data. “Other” includes countries without IDA or IBRD status or project locations that are above country level. IBRD = International Bank for Reconstruction and Development; IDA =International Development Association; MS+ = a rating of moderately satisfactory or better; n.a. = not applicable. During FY17, IEG produced CLRRs for country programs in three countries in the Region, namely, Jordan, Lebanon, and the Republic of Yemen. IEG rated development outcome (development outcome) moderately satisfactory for two programs. In the case of the third, the Republic of Yemen, outcome is not rated in either the CLR or in IEG’s review due to data limitations (table L.10).9 Independent Evaluation Group | World Bank Group 339 Table L.10. Middle East and North Africa: Country-Level Performance Ratings, FY14–17 CASCR/ CLR Review World Bank IFC MIGA Review FY Country Period Outcome Performance Performance Performance 2014 Djibouti FY09–13 MU Good n.a. n.a. Morocco FY10–13 MS Good n.a. n.a. 2016 Egypt, Arab n.a. n.a. Republic of FY06–14 U Fair Tunisia FY10–14 MS Good n.a. n.a. 2017 Jordan FY12–15 MS Good n.a. n.a. Lebanon FY11–15 MS Fair n.a. n.a. Yemen, Republic of FY10–15 n.a. n.a. n.a. n.a. Source: Independent Evaluation Group data. Note: All World Bank performance ratings were standardized with the following conversions: satisfactory as superior, moderately satisfactory as good, moderately unsatisfactory as fair, and unsatisfactory as poor. CASCR = country assistance strategy completion report; CLR = Completion and Learning Report; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency; MS = moderately satisfactory; MU = moderately unsatisfactory; U = unsatisfactory; n.a. = not applicable. Evaluation Findings from Project Performance Assessment Reports During FY17, IEG completed PPARs for two projects in the Middle East and North Africa Region. For development outcome, IEG rated one project satisfactory and the other moderately satisfactory. The Arab Republic of Egypt’s rapid population growth combined with its economic development and industrialization policies and weak environmental management have resulted in widespread and severe pollution. The World Bank sought to support pollution abatement from large industrial sources in Cairo and Alexandria through the $166 million Second Pollution Abatement Project that operated 2006–12. The project used IBRD financing and cofinancing from other development partners to support credit lines for industrial pollution abatement investments through an apex World Bank, supported by technical assistance. The credit lines were partly concessional, and so the project sought to encourage firms to make investments by combining this “carrot” with the “stick” of enforcement of existing pollution regulations. The project was successful in generating investments by firms, who invested $145 million of their own funds beyond the project financing. Implementation benefited from support provided by a capable project implementation unit from a predecessor project; from the application of a model where the environment agency provided technical advice to the private World Bank carrying out lending; and from the actions taken by the World Bank and government to help build a pipeline of subprojects even in the wake of diminished demand following economic downturn and political crises. Yet, the credit line mechanism was limited in its ability to target the most serious pollution problems because firms with poor creditworthiness could not receive financing. Program design relies on concessional donor finance, and so it may not be possible to scale up, or replicate the program at a level where 340 it would have a significant impact on ambient pollution. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the outcome rating was satisfactory, based on monitoring data that demonstrated significant reductions in pollution emissions from subprojects. Morocco’s urban transport systems have been under considerable pressure due to rapid urbanization. Public transport has been inadequate, of low quality, and disorganized—particularly the bus-tramway integration in Rabat and Casablanca. These urban mobility issues are partly the result of underinvestment and the lack of financial sustainability, thus hindering the participation of new private operators. The World Bank supported the Urban Transport Sector DPL with $136.7 million in pursuit of three specific policy objectives: (i) to improve the governance of the urban transport sector; (ii) to increase the ef ficiency and supply of urban transport infrastructure and services; and (iii) to improve the sector’s E&S sustainability. The government implemented 10 prior actions under these policy areas. A key finding is that the World Bank’s analytical and advisory work contributed greatly in helping ensure that the prior actions built directly on the government’s own priorities, and were causally linked to the DPL’s targeted outcomes. Nonetheless, additionality and attribution are pertinent issues, since many of the prior actions had already been initiated by the government before the DPL’s approval. The first and third policy objectives were substantially achieved, but the second objective achieved only modest progress in addressing operational ef ficiencies and lack of financial sustainability in the Rabat and Casablanca agglomerations. Lessons learned from the PPAR exercise relate to the need for (i) a strong reform momentum, (ii) early participation by implementing agencies and local governments, and (iii) flexible and adaptive responses. The World Bank clearly contributed in terms of its convening authority and intellectual leadership; however, this sector DPL may have been more effective had it been accompanied in parallel by investment lending and technical assistance operations. Overall, the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory. Evaluation Findings from Country Program Performance During FY17, IEG produced CLRRs for country programs in three countries in the Middle East and North Africa Region, namely, the Jordan, Lebanon, and the Republic of Yemen. IEG rated development outcome moderately satisfactory for two programs. In the case of the third, the Republic of Yemen, outcome is not rated in either the CLR or in IEG’s review due to data limitations. The CPS FY12–15 for Jordan built on the government agenda by focusing on three complementary and mutually reinforcing focus areas: strengthen fis al management and increase accountability; strengthen the foundation for sustainable growth with a focus on competitiveness; and enhance inclusion through social protection and local development. Transparency, accountability, and participation were cross-cutting themes. The CPS progress report increased World Bank support and adjusted the program by incorporating emergency operations to address the adverse effects of the Syrian crisis while maintaining focus on longer-term reforms. Focus areas were revised as follows: Independent Evaluation Group | World Bank Group 341 (i) strengthening resilience to economic shocks; (ii) enhancing competitiveness and shared prosperity; and (iii) mitigating vulnerability to reduce poverty. Progress was achieved under focus area I as measured by Global Integrity Country Scores, in improving public access to information, and in strengthening effectiveness of the Audit Bureau to enhance accountability in the use of public resources (supported by Programmatic DPLs and AAA). However, the reform targeting the social protection system stalled due to unsatisfactory implementation of supporting projects. Focus area II sought to address major infrastructure bottlenecks, environmental sustainability, key constraints to micro, small, and medium enterprises growth, lack of legal framework for PPP, and the mismatch between available skills and labor demand. IFC played an important role in supporting renewable energy, access to finance, and skills development. Four of the seven objectives in this area were achieved or mostly achieved, but three important targets—in transport, PPP, and learning environment—were not achieved or partially achieved. Measures were supported under focus area III to mitigate the adverse impact of the Syrian crisis on access to safety nets and social services, and on the capacity of local governments to provide public services. Support was also provided for longer-term reforms to improve access to social assistance, and to expand coverage to the transient poor. While the emergency projects performed well, projects dealing with longer-term reforms suffered from poor design, lack of government commitment, and unsatisfactory implementation. Both the CLR and IEG’s review rate World Bank Group performance as good. The CPS was appropriately designed with flexibility built into the program in the context of regional and political risks. The CPS progress report made significant revisions to the original results framework to give the program more focus given changing context. Programmatic development policy financing operations set the policy and institutional framework for the reforms, investment policy financing projects supported implementation of various initiatives, and AAA provided both analytical underpinning and technical assistance. IFC Advisory and Investments Services complemented World Bank activities. However, the review concludes that World Bank support to reforms in social protection and social assistance systems lacked a clearly articulated strategy or approach, and the revised results framework was light on outcome indicators. While the share of projects at risk to total number of projects in the portfolio declined in FY14, it doubled in FY15, indicating the need for more sustained efforts at improving implementation. IEG’s review concurs with CLR lessons and findings including: responding to immediate priorities while making progress in addressing long-standing issues; need for careful assessment of implementation capacity and institutional incentives to inform priorities for World Bank support; and, the need to replicate successes achieved in pilot initiatives and innovative approaches. IEG identifies the following lessons: promoting inclusion requires complementing growth-enhancing programs with targeted interventions (with reference, for example, to women and youth employment); projects need to be kept flexible in the context of a rapidly changing environment, and problem projects require intensive implementation support and timely adjustment; it is important to manage the change process, including allowing for enough time and buy-in from stakeholders; and finally, selectivity in project selection is critical to program success, especially in the context of 342 weaknesses in implementation capacity. Both the CLR and IEG’s review rated overall development outcome moderately satisfactory. The areas of focus for the Lebanon CPS FY11–14 reflected the directions and priorities of the government’s Progress and Development program: energy, water, transport, municipal and urban development, social protection, and fiscal management. However, IEG found the initial World Bank program to be overly ambitious because there were insuf ficient World Bank interventions to back up program objectives, or interventions were not commensurate in size and scope with the objectives. Initial ambition was scaled back at progress report stage, although financing under the program increased significantly. Eight out of 12 objectives and their corresponding indicators were revised, and other indicators added at progress report stage. Focus area I, fiscal management, was the most effective intervention area in which success was achieved in generating better information for fiscal policy decisions and enhanced transparency and control in PFM. Focus area II (competitiveness), focus area III (infrastructure), and focus area 4 (human development and social protection) showed mixed results. Access to finance by SMEs and women entrepreneurs, and private sector access to trade finance showed positive developments, and the urban transport system improved, as did the delivery of social services. However, there was less success achieved in relation to improving the business environment, reforming the electricity, water, and telecommunication sectors, provision of enhanced education, or an improved statistical system. The program also failed in its objective to put the economy on a path to sustained high growth. The CLR rated World Bank Group performance good, and IEG’s review rated it fair reflecting IEG’s view that weak selectivity under the program was compounded by a changing environment where implementation became quite uncertain. Program ambition was supported by modest, and sometimes inadequate, World Bank inputs, and some lending projects were poorly sequenced, an issue both of design and implementation. Overall, program design was too complex given local implementation capacity, though the World Bank responded cooperatively and flexibly as problems arose, particularly in response to the effects of the Syrian crisis. IFC played a positive role in partnering with financial sector institutions, but its Advisory Services program with the government fell short of expectations. IEG presents the following lessons: to increase the chances of success in Lebanon, projects need to be simple in scope and allow for flexibility in implementation; to improve chances of success, the World Bank will need to re-double quality at entry efforts, both for emergency and nonemergency operations; emergency operations require additional attention and review steps during implementation, for example, of M&E and risk mitigation; and the World Bank might consider focusing more on grant funding and technical assistance in emergency situations (for example, challenges associated with influx of refugees) given the relative success of such activities over lending interventions in that context. Both the CLR and IEG’s review rate overall outcome moderately satisfactory. Owing to data limitations and in line with the CLR, IEG’s review does not rate overall development outcome or World Bank Group performance for the duration of the Republic of Yemen CAS FY10–13, and the ISN (January 2013–December 2014). The three pillars of the ISN—protecting the poor, Independent Evaluation Group | World Bank Group 343 growth, and governance and public service delivery—were largely derived from the existing CAS results framework, updated to reflect the new country context. As such, both the CLR and IEG’s review also report on indicators from before the ISN. Implementation of the FY10–13 CAS was disrupted in 2011 by popular unrest—part of the so- called Arab Spring—and led to a temporary halt of World Bank operations in the Republic of Yemen, and evacuation of all staff from the capital. This challenging situation resulted in the World Bank suspending disbursements, except for the operating costs of six well-performing project implementation units. The temporary suspension of World Bank Group–funded operations was lifted following the installation of a transitional government in late 2011, but World Bank missions could not go outside the capital, and this limited access to project sites. The World Bank set up third-party monitoring arrangements in response (International Relief and Development). This arrangement worked well until the end of 2014, when the conflict escalated, and the World Bank again suspended most disbursements. This remained in place at the time of the CLR and its review, although IEG noted that the CLR does not discuss how the suspension exceptions under the CAS worked and does not suggest lessons that could be useful under the ISN suspension or future World Bank engagements, except for the need to continue supporting and strengthening resilient, inclusive institutions. During the period of this review, the World Bank program was always operating within some form of preconflict, open conflict, or postconflict environment. In retrospect, both the CAS and ISN were too optimistic about local security conditions. As political and security risks were not mitigated adequately (noting how dif ficult a task this presented), implementation of many operations could not proceed as planned, and most expected results under the program were not achieved. Solid progress was made under one of the 13 objectives, to strengthen competitiveness in fisheries, and agriculture showed solid progress. For the rest, results were mixed at best with few exceptions. Some progress was made under focus area I, protecting the poor and improving basic service delivery, reflecting partnerships with other donors in certain instances (for example, UN agencies). Focus area II on improving economic management generally showed little progress, and there was no progress evident under focus area III, local service delivery. IEG agrees with CLR lessons: helping resilient state institutions is essential to maintain service delivery (for example, the Public Works Program); close collaboration with agencies such as the United Nation Children’s Fund and the World Health Organization is critical and, in this case, allowed the World Bank to lift suspension of disbursements under two World Bank-funded health projects, even in the midst of conflict; and IFC cooperation with nonprofit organizations contains positive lessons for future engagements (for example, cooperation with the Republic of Yemen Education for Employment Foundation). The review also notes that the number of program objectives for both the CAS and ISN appears high for a conflict situation such as the Republic of Yemen, and underscores poor selectivity. IEG highlights four lessons: to improve the relevance of interventions, make use of participatory beneficiary involvement in the planning and design of activities - simple project design that promotes active beneficiary participation resulting in demand based delivery systems are highly appropriate for fragile and conflict states; where infrastructure investments and sector reforms are 344 coupled in the same project, the infrastructure investments tend to outpace sector reforms—the World Bank needs to find ways to tie disbursements for infrastructure with progress on sector reforms, to ensure parallel movement; supervision and monitoring project implementation activities in a country experiencing social and political unrest are virtually impossible without effective third-party monitoring; and during a conflict or political crisis that triggers the suspension of the World Bank’s and other development partners’ disbursements, a proactive emergency implementation plan that identifies transition mechanisms, essential activities, negotiations with active contractors, and support to other development partners that remain in the country, is an important tool that enables swifter revamping of activities in a postcrisis period. Figure L.70. South Asia: Project Outcome Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 345 South Asia Region Project- and Country-Level Performance This note presents the results and performance separately for the World Bank, IFC, and MIGA given the different time frames, operating models, and clients of the three institutions. Unlike previous RAP reports, RAP17 excludes Afghanistan and Pakistan in IFC’s regional data aggregation for Middle East and North Africa Region. The two countries were classified as part of Middle East and North Africa following past IFC country classification. For RAP17, the two countries are classified as part of the South Asia Region (South Asia Region) to reflect IFC’s current organizational structure.10 Figure L.71. South Asia: World Bank Performance Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: MS+ = a rating of moderately satisfactory or better. 346 Figure L.72. South Asia: M&E Quality Ratings, FY05–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: M&E = monitoring and evaluation; S+ = a rating of substantial or better. The World Bank: Project-Level Performance For the current review period, FY14–16, the development outcome for 74 percent of projects in the South Asia Region was MS+, compared with an average 73 percent MS+ rating across all other Regions (figure L.70, panel a). By commitment, the rating was 83 percent MS+ in the South Asia Region, compared with an average 84 percent MS+ rating across all other Regions (figure L.70, panel b). Overall World Bank performance, a composite of ratings for quality at entry and project supervision, in the South Asia Region during FY14–16, is 74 percent MS+, compared with a 72 percent MS+ average rating across all other Regions (figure L.71, panel a). Within the overall World Bank performance rating in the South Asia Region, quality at entry was rated 59 percent MS+, compared with a 61 percent MS+ Independent Evaluation Group | World Bank Group 347 average across the other Regions. Supervision in the South Asia Region was rated 77 percent MS+, compared with an average rating of 78 percent MS+ across the other Regions. The overall quality of M&E for projects in the South Asia Region during FY14–16 was 33 percent S+, compared with an average 34 percent S+ rating across all other Regions (figure L.72, panel a). By commitment, the World Bank performance rating decreased between FY11–13 and FY14–16 (from 58 percent S+ to 40 percent S+) (figure L.72, panel b). The development outcome rating for IBRD-supported projects in the South Asia Region during FY14–16 was 71 percent MS+, compared with an average 75 percent MS+ rating across all other Regions (figure L.73, panel a). By commitment volume, the rating was 84 percent MS+ in the South Asia Region, compared with an average 85 percent MS+ rating across all other Regions (figure L.73, panel b). The development outcome rating for IDA-supported projects in the South Asia Region during FY14–16 was 79 percent MS+, compared with an average 72 percent MS+ rating across all other Regions (figure L.73, panel a). By commitment volume, the rating was 84 percent MS+ in the South Figure L.73. South Asia: Project Outcome Ratings by Financing Source, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; MS+ = a rating of moderately satisfactory or better. 348 Asia: World Bank Performance Ratings by Financing Figure L.74. South Source, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; MS+ = a rating of moderately satisfactory or better. Figure L.75. South Asia: M&E Quality Ratings by Financing Source, FY14–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; S+ = a rating of substantial or better. Independent Evaluation Group | World Bank Group 349 Figure L.76. South Asia: Development Outcome by Global Practice, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: MS+ = a rating of moderately satisfactory or better. 350 Asia Region, and this compares with an average satisfaction rating 82 percent MS+ across all other Regions during FY14–16 (figure L.73, panel b). World Bank performance for IBRD-supported projects in the South Asia Region during FY14–16 (71 percent MS+) is slightly lower than the average across all other Regions (76 percent MS+; figure L.73, panel a). World Bank performance (81 percent MS+) for IDA-supported projects in the South Asia Region during FY14–16 is higher than the average rating for IDA-supported projects across all other Regions (71 percent MS+; figure L.74, panel b). M&E quality for IBRD-supported projects in the South Asia Region during FY14–16 is 14 percent S+, compared with a 37 percent S+ rating across all other Regions. M&E quality for IDA-supported projects in the South Asia Region (42 percent S+) is higher than the average rating for IDA-supported projects across all other Regions (31 percent S+: figure L.75). Seventy-eight projects were rated for the South Asia Region during FY14–16 valued at almost $9 billion in commitments. The largest GPs generally performed well: Agriculture, with 100 percent MS+ (13 projects, $1.6 billion), and Water, with 80 percent MS+ (10 projects, $1.6 billion). Finance and Markets did less well, with development outcome rated 50 percent MS+ (8 projects) and 68 percent MS+ by commitment ($643 million). International Finance Corporation: Project-Level Results and Performance IFC Investments Project-Level Results Development outcome ratings of evaluated IFC investment projects in the South Asia Region improved markedly this review period (CY14–16). Twenty-two of the 31 projects evaluated in CY14–16, (71 percent) had MS+ development outcome ratings (figure L.77). By contrast, half (49 percent) of the investment projects evaluated in CY11–13 had ratings of MS+ for achieving their intended development outcome. The share of projects with MS+ development outcome ratings was also higher in CY14–16 compared with CY11–13 results, by IFC net commitment amount. Large-sized projects evaluated in CY14–16 had high (MS+) development outcome ratings. IFC’s investment outcome and its role and contribution ratings also improved in CY14–16, by number of projects and by volume, from CY11–13. The improvement in the overall development outcome success rating in CY14–16 reflects higher success ratings of the evaluated projects’ business success, economic sustainability and contribution to private sector development. The share of positive (S+) ratings for these three development outcome subindicators improved during this review period, based on the number of evaluated projects. If the results are adjusted by the amount of IFC’s net commitment, only the success ratings pertaining to private sector development showed marked increased in CY14–16. However, the subindicator E&S effects has underperformed this review period. Compared with CY11–13 results, the share of evaluated projects with S+ ratings for E&S effects in CY14–16 declined, by number of projects and by volume. Several large-sized projects evaluated in CY14–16 had poor Independent Evaluation Group | World Bank Group 351 Figure L.77. South Asia Region: Results and Performance Ratings of Evaluated IFC Investment Projects Share of Projects with Positive Ratings a. By number of projects, three-year moving average b. By IFC net commitment ($, millions), three-year moving average Source: Independent Evaluation Group Evaluation Data for IFC XPSRs as of December 12, 2017. Note: (1) Results for the South Asia Region include projects in Pakistan and Afghanistan, which have been reclassified as part of South Asia to reflect current IFC organizational structure. In prior years, the two countries were classified under the Middle East and North Africa Region in IFC’s project portfolio databases. (2) Positive ratings for development outcome consists of mostly successful, successful, or highly successful (aggregated as MS+) ratings. Positive ratings for investment outcome and IFC’s role and contribution consist of excellent or satisfactory (aggregated as S+) ratings. (3) Due to the revision of the XPSR guidelines in 2015, the indicator, IFC’s role and contribution, has been amended starting CY15, and was replaced with the indicator, IFC Additionality. 352 Figure L.78. South Asia Region: Development Outcome Subindicators of IFC Investment Projects, by XPSR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC net commitment ($, millions), three-year moving average Source: Independent Evaluation Group evaluation data for IFC investment projects as of December 12, 2017. Note: (1) Results for the South Asia Region include projects in Pakistan and Afghanistan, which have been reclassified as part of South Asia to reflect current IFC organizational structure. In prior years, the two countries were classified under the Middle East and North Africa Region in IFC’s project portfolio databases. (2) S+ = a rating of satisfactory or excellent. (3) E&SE = environmental and social effects; ECSUS = economic sustainability; PBS = project business success; PSD = private sector development. These four subindicators comprise the development outcome dimension of an IFC investment project. (3) No opinion possible or not applicable ratings were excluded. Independent Evaluation Group | World Bank Group 353 Figure L.79. South Asia: Association between Development Outcome, IFC Work Quality and IFC’s Role and Contribution, CY14–16 By number of evaluated projects By IFC net commitment ($, millions) a. Association between development outcome and IFC work quality b. Association between development outcome and IFC role and contribution c. Association between work quality and IFC investment outcome Source: Independent Evaluation Group evaluation data for IFC investment projects as of December 12, 2017. Note: (1) IFC work quality in CY14–16 reflects the aggregate ratings of two subindicators: screening, appraisal, and structuring and supervision and administration. (2) High development outcome = highly successful, successful, and mostly successful ratings; low development outcome = mostly unsuccessful, unsuccessful and highly unsuccessful ratings; high work quality = excellent and satisfactory ratings; low work quality = partly unsatisfactory and unsatisfactory ratings; high IFC role and contribution = excellent and satisfactory ratings; low IFC role and contribution = partly unsatisfactory ratings; high IFC investment outcome = excellent and satisfactory ratings; low IFC investment outcome = partly unsatisfactory and unsatisfactory ratings. (3) Due to the revision of the Expanded Project Supervision Report guidelines in 2015, the indicator IFC’s role and contribution has been amended starting CY15, and was replaced with the indicator IFC additionality. IFC = International Finance Corporation. 354 Figure L.80. South Asia Region: IFC Work Quality Performance, by XPSR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC net commitment, three-year moving average Source: Independent Evaluation Group evaluation data for IFC investment projects as of December 12, 2017. Note: (1) Results for the South Asia Region include projects in Pakistan and Afghanistan, which have been reclassified as part of South Asia to reflect current IFC organizational structure. In prior years, the two countries were classified under the Middle East and North Africa Region in IFC’s project portfolio databases. (2) Based on the 2015 revised XPSR guidelines, IFC work quality is measured by two indicators: screening, appraisal, and structuring (SAS) and supervision and administration (SUP) in the project. IFC = International Finance Corporation; XPSR = Expanded Project Supervision Report. (partly unsatisfactory or unsatisfactory rating) E&S outcomes causing the average E&S effects ratings to drop 24 percentage-points from CY11–13 ratings. Projects evaluated in CY14–16 with high (MS+) development outcome rating also had high (S+) IFC work quality and IFC outcome ratings. As shown in Figure L.79, nearly 60 percent of the evaluated projects have high development outcome and high IFC work quality ratings, using unweighted (number of projects) and weighted (by IFC net commitment amount) measures. Projects rated highly for development outcome tend to also have high (S+) ratings for IFC role and contribution (nearly 60 percent of evaluated projects). The same close association can be observed in Figure L.79 below Independent Evaluation Group | World Bank Group 355 Figure L.81. South Asia Region: Results and Performance Ratings of Evaluated IFC Advisory Projects, by PCR Program Years Share of Projects with Positive Ratings a. By number of evaluated projects, three-year moving average b. By IFC expenditure ($, millions), three-year moving average Source: Independent Evaluation Group evaluation data for IFC PCRs as of December 12, 2017. Note: (1) Results for the South Asia Region include projects in Pakistan and Afghanistan, which have been reclassified as part of South Asia to reflect current IFC organizational structure. In prior years, the two countries were classified under the Middle East and North Africa Region in IFC’s project portfolio databases. (2) The reported ratings are based on IEG’s final ratings in the completed Project Completion Reports (PCRs). (3) Positive ratings for development effectiveness consists of highly successful, successful, or mostly successful (aggregated as MS+) ratings. Positive ratings for IFC’s role and contribution consist of excellent or satisfactory (aggregated as S+) ratings. 356 among projects with high IFC investment outcome ratings and projects with high ratings for IFC work quality. In sum, improving IFC’s work quality can also improve the likelihood of achieving better development outcome and IFC investment outcomes. Figure L.82. South Asia Region: Ratings of Four Indicators of Development Effectiveness of Evaluated IFC Advisory Projects, by PCR Program Years Share of Projects with S+ Ratings a. By number of evaluated projects, three-year moving average b. By IFC expenditure ($, millions), three-year moving average Source: Independent Evaluation Group evaluation data for IFC PCRs as of December 12, 2017. Note: (1) Results for the South Asia Region include projects in Pakistan and Afghanistan, which have been reclassified as part of South Asia to reflect current IFC organizational structure. In prior years, the two countries were classified under the Middle East and North Africa Region in IFC’s project portfolio databases. (2) The reported ratings are based on IEG’s final ratings in the completed Project Completion Reports (PCRs). (3) Positive or success ratings for each of the subindicators indicators of development effectiveness = satisfactory or excellent (referred to as S+) ratings. Projects with Not Applicable, too early to tell, too early to judge, or cannot verify are excluded from the count. Independent Evaluation Group | World Bank Group 357 Of the two subindicators of IFC work quality, satisfactory or better (S+) ratings for the quality of IFC’s SAS improved markedly in CY14–16 compared with CY11–13 results, both by number of projects and weighted by IFC net commitment. But positive ratings for the subindicator SUP fell drastically during this review period (figure L.80). The 16-percentage point drop in S+ ratings, by volume, for the quality of IFC’s supervision and administration was due to low SUP ratings for four large projects, which account for nearly 25 percent of the total volume of projects evaluated in CY14–16. IFC Advisory Services Project-Level Results Thirty-eight IFC Advisory Services projects in the South Asia Region were evaluated in FY14–16, and 28 projects (58 percent) had MS+ development effectiveness ratings (figure L.81). Adjusting the results by the amount of IFC project expenditures, the share of MS+ rating in FY14–16 is much lower, at 43 percent, due to poor (mostly unsuccessful or below) development effectiveness ratings of a few large IFC advisory projects that were evaluated in FY14–16. Compared with FY11–13 results, development effectiveness and IFC role and contribution success ratings have declined substantially, both by number of projects and by volume. Ratings for the four subindicators of development effectiveness have mostly underperformed in FY14–16 (figure L.82). Success (S+) ratings of the evaluated projects’ strategic relevance, achievement of expected outcomes and ef ficiency in delivering outputs and outcomes fell compared with ratings in FY11–13. Evaluated projects in FY14–16 with development effectiveness ratings of MS+ also have the same share of MS+ ratings for the subindicator, outcome achievement. Multilateral Investment Guarantee Agency: Project-Level Performance Only three MIGA projects in the South Asia Region were evaluated since an independent evaluation function was established in MIGA in FY03 until FY16. The first project involved a MIGA guarantee coverage to a power sector project, the second a telecom project and the third project was in the manufacturing sector. Of the two projects evaluated in the FY11–16 Project Evaluation Report Program, one had satisfactory or better development outcome ratings and for MIGA’s role and contribution in the projects. However, the two projects received less than satisfactory ratings for the quality of MIGA’s assessment, underwriting and monitoring. World Bank Group: Country-Level Performance Projects were rated for six IDA and two IBRD countries in the South Asia Region during FY14–16. Within the IDA complement, 1 country was designated fragile. By number, outcome ratings for projects in IBRD countries exceed the average for the Region (82 percent MS+ versus 74 percent MS+) and projects rated for IDA countries fall below the average (69 percent MS+). By volume, the Region average is 83 percent MS+ and projects in the IBRD countries perform above that average (89 percent MS+). The overall average, by volume, for projects supported in IDA countries is 68 percent MS+ noting that the outcome rating for the eight projects valued at $348 million in the one IDA fragile country in the Region for which projects were rated FY14–16, was 81 percent MS+ (table L.11). 358 Table L.11. South Asia: Project Outcome Ratings by Country Type, FY14–16 Countries Projects Volume Evaluated Rated MS+ Total Amount Rated MS+  Country Type (no.) No. (percent) ($, millions) (percent) IDA fragile 1 8 63 348 81 IDA nonfragile 5 37 70 1,868 66 IBRD fragile 0 0 n.a. 0 n.a. IBRD nonfragile 2 33 82 6,592 89 IDA total 6 45 69 2,216 68 IBRD total 2 33 82 6,592 89 Fragile total 1 8 63 348 81 Other n.a. 0 n.a. 0 n.a. Total 8 78 74 8,808 83 Source: Independent Evaluation Group data. Note: One project was excluded when measured by volume due to lack of volume data. “Other” includes countries without IDA or IBRD status or project locations that are above country level. IBRD = International Bank for Reconstruction and Development; IDA =International Development Association; MS+ = a rating of moderately satisfactory or better; n.a. = not applicable. IEG completed one CLRR for the Region in FY17. Both the World Bank’s CLR and IEG’s review rated overall development for the Sri Lanka Country Program Strategy FY13–16 moderately satisfactory (table L.12).11 Table L.12. South Asia: Country-Level Performance Ratings, FY14–17 CASCR/ CLR Review World Bank IFC MIGA Review FY Country Period Outcome Performance Performance Performance 2014 Pakistan FY10–14 MU Fair n.a. n.a. 2015 Bhutan FY11–14 MS Good n.a. n.a. 2016 Bangladesh FY11–15 MS Good n.a. n.a. 2017 Sri Lanka FY13–16 MS Fair n.a. n.a Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: All World Bank performance ratings were standardized with the following conversions: satisfactory as superior, moderately satisfactory as good, moderately unsatisfactory as fair, and unsatisfactory as poor. CASCR = country assistance strategy completion report; CLR = Completion and Learning Report; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency; MS = moderately satisfactory; MU = moderately unsatisfactory; U = unsatisfactory; n.a. = not applicable. Evaluation Findings from Project Performance Assessment Reports During FY17, IEG completed four PPARs for projects in the South Asia Region. For development outcome, IEG rated one project satisfactory, two moderately satisfactory, and one moderately unsatisfactory. The Bangladesh Social Investment Program Project (March 2003 to December 2012) attracted three rounds of additional financing, although the third additional financing was canceled such that actual Independent Evaluation Group | World Bank Group 359 project cost at closing was $60.3 million. The project supported the construction of 1,734 community infrastructure works, including 2,490 km of earthen roads, 3,160 culverts, and 2,940 hand tube wells. Eighty-five schools were repaired, meaning the objective of improving access to local infrastructure and basic services was substantially met. An impact assessment was positive about improvements supported by the project such as significantly improved mobility for beneficiaries, increased enrollment in schools, and a decrease in waterborne illness over the life of the project. However, the impact assessment did not report on the quality of the services delivered, including, importantly, the quality of the water in areas where arsenic is known to be a contaminant, or the quality of the tube wells regarding their flood resilience. IEG found the original objective (improved access to local infrastructure and basic services, which were intended to be delivered through effective and ef ficient financing mechanisms and by using a CDD approach) to be modestly relevant noting the lack of specificity regarding proposed beneficiaries; however, greater specification was provided in the second phase of the project when it was clarified that the project sought to improve the livelihoods of communities in which there was a high incidence of extreme poverty. IEG rated ef ficiency as modest referencing a lack of clarity as to whether the infrastructure built was flood resilient (considered unlikely in a CDD project), and the absence of a calculation of the returns on health savings, owing to the increased access to potable water. IEG found significant risk to development outcome as only 20 percent of all community institutions supported by the project would be sustainable without ongoing implementing support. At the time of the assessment there was no evidence to suggest such support would be forthcoming. The PPAR also notes that, to be sustainable, there is a need to situate the works provided through CDD projects in Bangladesh within a basin-wide water and disaster management planning approach. IEG drew lessons from the assessment, including the importance of facilitating the formation of strong institutions in rural areas to enable the poor to access services, and the need for thorough assessment of an implementing agency’s capacity to adapt, to manage funding, and to reach disaster-affected populations outside of a project area (notwithstanding the benefits of CDD as an initial platform). Overall, the World Bank’s ICR concluded that the project’s outcome rating was moderately satisfactory. IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately unsatisfactory for development outcome, reflecting the assessment of modest relevance for the objective, design, and ef ficiency despite substantial ef ficacy. Mumbai Metropolitan Region is one of the world’s largest urban areas. Suburban railway services are the backbone of the agglomeration, but appraisal undertaken for the Mumbai Urban Transport Project found them to be grossly overcrowded, especially in the peak hours. The road and bus systems also suffered from bottlenecks and lack of investment. The World Bank supported a comprehensive approach that included improving the capacity and performance of the suburban rail system, an enhanced bus system, the introduction of Area Traf fic Control for the roads to optimize the functioning of traf fic signals, as well as improvements to pedestrian facilities, road improvement schemes and parking controls. 360 The project involved the relocation of over 100,000 people affected by the project, many of whom lived in squatter settlements adjacent to the railway tracks. This aspect turned out to be more complex than expected and grievances expressed by affected persons were not adequately addressed leading to an Inspection Panel investigation. However, by the end of the project, following the implementation of an appropriate action plan, most people were rehoused or compensated. Moreover, the knowledge gained in the resettlement activities enabled the regional development authority to set up a Resettlement Division that has successfully replicated the procedures developed in the Mumbai project, in other non–World Bank financed developments. Both suburban rail and bus fares were revised upward and further rail improvements have been undertaken in a follow-on project. However, at the time of the assessment, the regional government had shied away from more radical policy reforms such as a road user charging, land use reform, and the introduction of parking restrictions, so any capacity improvements are soon dissipated by increased traf fic congestion, while air quality in the region continues to deteriorate. On the other hand, institutional capacity has improved and infrastructure maintenance is adequate. Provision of free, alternative housing with title has had an empowering effect on project-affected households. Overall, the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory, reflecting the achievement of most objectives albeit with some shortcomings. Between 1996–2006, Maoist insurgency plunged Nepal into a prolonged period of political instability that contributed to retardation of economic development. Most rural men had migrated to urban centers, leaving villages largely inhabited by women, children, and the elderly. Water supply coverage was 72 percent and sanitation coverage was only 25 percent. Through the $73.6 million Second Rural Water Supply and Sanitation Project (2004–12), the World Bank sought to support the government in improving rural WSS institutional performance; in mainstreaming the “Fund Board” approach (a demand-driven and community-managed approach in which funds are transferred to communities for procurement and implementation) in government’s systems; and in supporting communities to form inclusive local WSS user groups capable of planning, implementing, and operating drinking water and sanitation infrastructure. The project applied financing sourced from IDA, the government, and local communities to form 1,465 local WSS user groups in 831 villages, benefiting 1,140,892 people in the project area, resulting in an increase in access to tap water from 7.1 percent to 94.6 percent. User groups planned and constructed 64,407 sanitary latrines using project funds, and an additional 72,080 latrines using their own resources and prompted by increased awareness. Selective impact studies showed a significant increase in the use of improved toilets, hand, washing practices, and a decrease (from 78 percent to 14 percent) in the prevalence of diarrheal disease among young children over the life of the project. Sustainable benefits were derived from the decision to place women, the main stakeholders, at the center of planning and implementation. However, the Fund Board approach was not mainstreamed in government systems due to a lack of government ownership, and the failure to realize this objective contributed to IEG’s moderately satisfactory assessment of overall outcome, marking down the World Bank’s self-assessment of satisfactory. Independent Evaluation Group | World Bank Group 361 The Second Community Water Supply and Sanitation Project (2003–10) supported the government of Sri Lanka’s ongoing commitment to expanding rural WSS services. IEG found that, overall, the project contributed to providing adequate, affordable, and relatively sustainable water services, and to ensuring convenience and time saving for the beneficiaries. Through a Sanitation Revolving Fund, the project has also contributed to achieving and maintaining a high percentage of toilet availability for rural households. The first project objective—increasing service coverage of water and sanitation services in rural communities—was modestly achieved partly due to reallocation of funds following the 2004 tsunami, as well as overly ambitious targets for increasing service coverage at the design stage. The second objective—achieving effective and sustained use of water and sanitation services in rural communities—was substantially achieved although there have been some challenges in ensuring reliability and water quality, such as the lack of 24-hour supply and water contamination. Important lessons from the project were that lack of continuity in M&E and its use by an implementing agency beyond project completion undermines sustainability of development outcomes, and that consistent institutional and technical support is needed to achieve sustainable service delivery in community based organization schemes. Overall, the World Bank’s ICR and IEG’s independent field- based evaluation concluded that the project’s outcome rating was moderately satisfactory. Evaluation Findings from Country Program Performance IEG completed one CLRR for the Region in FY17. Both the World Bank’s CLR and IEG’s review rated development outcome for the Sri Lanka Country Program Strategy FY13–16 moderately satisfactory. Sri Lanka’s national policy framework 2010–16 had three goals: doubling per capita income; shifting the structure of the economy to be more knowledge based, globally and internally integrated, environmentally friendly, and increasingly urban; and ensuring improved living standards and social inclusion. The World Bank Group’s ambitious CPS for FY13–16 focused on three matching areas of engagement: facilitating sustained private and public investment; supporting structural shifts in the economy; and improving living standards and social inclusion. A fourth area, improving resilience to climate and disaster risks, was added through the 2014 CPS progress report process. The program as designed had mixed results. The World Bank Group contribution to higher education quality, and progress in skills development was innovative, although there is now a need to complement that progress with labor market and entitlement reforms, and there is opposition to reform from, among others, trade unions and student lobbies. The program also supported improvement in education quality and in health services, but malnutrition continues to be a problem. The World Bank’s support for local authorities to deliver services and local infrastructure in a more responsive and accountable manner is positive, but results on service quality are yet to be assessed, and improved revenue effort remains vital to sustaining service delivery. World Bank Group performance was rated good in the CLR, and fair in IEG’s review. IEG found the CPS was responsive to the government’s development policy framework but could have exercised greater selectivity in its development objectives - several objectives (for example on public investment, and nutrition) were not backed by suf ficient interventions to attribute meaningful 362 results. The CPS progress report brought more realism, but the important Investment Policy Reform DPL was delayed, and three operations (including the Legal and Judicial Reform project) were dropped, indicating weak ownership for policy reforms. Analytical work and technical assistance complemented lending in several sectors, and more recent analytical work on revenue and Public Expenditure Reviews has laid the basis for a more meaningful engagement in the next CPF period. The program demonstrated flexibility by adding a new focus area on climate resilience, but the revisions to the results framework led to a dilution of focus from outcomes to outputs, weakening the ability to assess results. IFC’s Investment and Advisory Services complemented the World Bank’s role on investment climate, access to finance, and climate resilience. IEG broadly concurs with the CLR lessons on flexibility, the continued challenges in relation to the investment climate, and the fiscal space, the importance of social inclusion, the need for a more realistic results framework, and the potential contribution of a robust analytical program, provided it is timely. In addition, IEG identified lessons that emphasized the need for greater selectivity of objectives, and more precise definition of outcome indicators, and the need to preempt critical development constraints through early investment, as well as the need for greater awareness of the role of other development partners. IEG’s review also notes that IFC needs to systematically ensure that investees strengthen M&E of benefits from financial services to the rural poor, particularly women, in postconflict areas, and improve risk management. Both the CLR and IEG’s review rated overall development outcome moderately satisfactory. List of Summarized Works World Bank. 2006. Madagascar—Country Assistance Strategy for the Period FY07–11. Washington, DC: World Bank. ———. 2006. Mauritius—Country Partnership Strategy. Washington, DC: World Bank. ———. 2008. Togo—Interim Strategy Note for the Period FY08–10. Washington, DC: World Bank. ———. 2009. Republic of Yemen—Country Assistance Strategy for the Period FY 2010–2013. Washington, DC: World Bank. ———. 2010. Lebanon—Country Partnership Strategy for the Period FY11–FY14. Washington, DC: World Bank. ———. 2010. Lesotho—Country Assistance Strategy for the Period FY10–14. Washington, DC: World Bank. ———. 2010. Tonga—Country Assistance Strategy for the Period FY11–14. Washington, DC: World Bank. ———. 2011. Belize—Country Partnership Strategy for FY12–FY2015. Washington, DC: World Bank. ———. 2011. Brazil—Country Partnership Strategy (CPS) for the period FY12–15. Washington, DC: World Bank. Independent Evaluation Group | World Bank Group 363 ———. 2011. Kiribati—Country Assistance Strategy for the Period FY11–14. Washington, DC: World Bank. ———. 2011. Togo—Interim Strategy Note for the Period FY12–13. Washington, DC: World Bank. ———. 2011. Tuvalu—Country Assistance Strategy for the Period FY12–15. Washington, DC: World Bank. ———. 2011. Vietnam—Country Partnership Strategy for the Period FY12–16. Washington, DC: World Bank. ———. 2012. Guatemala—Country Partnership Strategy for the Period FY13–16. Washington, DC: World Bank. ———. 2012. Jordan—Country Partnership Strategy for the Period FY11–15. Washington, DC: World Bank. ———. 2012. Kosovo—Country Partnership Strategy for the Period FY12–15. Washington, DC: World Bank. ———. 2012. Lao People’s Democratic Republic—Country Partnership Strategy for the Period FY12–16. Washington, DC: World Bank. ———. 2012. Madagascar— Interim Strategy Note for the Period FY12–13. Washington, DC: World Bank. ———. 2012. Mozambique—Country Partnership Strategy for the Period FY12–15. Washington, DC: World Bank. ———. 2012. Peru—Country Partnership Strategy for the Period FY12–FY16. Washington, DC: World Bank. ———. 2012. Samoa—Country Partnership Strategy for the Period FY12–16. Washington, DC: World Bank. ———. 2012. Sri Lanka—Country Partnership Strategy for the Period FY13–16. Washington, DC: World Bank. ———. 2012. Ukraine—Country Partnership Strategy Ukraine for the Period FY12–16. Washington, DC: World Bank. ———. 2012. Yemen— Interim Strategy Note for the Period FY13–14. Washington, DC: World Bank. ———. 2013. Marshall Islands—Country Partnership Strategy for the Period FY13–FY16. Washington, DC: World Bank. ———. 2014. Micronesia, Federated States of—Country Partnership Strategy for the Period FY14–17. Washington, DC: World Bank. ———. 2016. Bangladesh—Social Investment Program Project. Independent Evaluation Group, Project Performance Assessment Report 106442. Washington, DC: World Bank. http://ieg.worldbank- group.org/sites/default/files/Data/reports/ppar_bangladesh_2016.PDF. ———. 2016. Bangladesh— Rural Transport Improvement Project, Phase I. Independent Evaluation Group, Project Performance Assessment Report 106729. Washington, DC: World Bank. http://ieg.world- bankgroup.org/sites/default/files/Data/reports/ppar_bangaldeshruraltransport_0716.PDF. 364 Results and Performance of the World Bank Group 2017 | Appendix L ———. 2016. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Inde- pendent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/ default/files/Data/Evaluation/files/behindthemirror_0716.PDF. ———. 2016. Brazil—Rural Poverty Reduction and Community Driven Development Lessons from Ceará and Santa Catarina Projects. Independent Evaluation Group, Project Performance Assessment Report 106455. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/reports/ppar_brazil_0716.PDF. ———. 2016. Financial Viability of the Electricity Sector in Developing Countries: Recent Trends and Effectiveness of World Bank Interventions. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/ lp_financial_viabili-ty_electricity_sector.PDF. ———. 2016. Findings from Evaluations of Policy-Based Guarantees. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ lp_policy_based_guarantees_102116.PDF. ———. 2016. Former Yugoslav Republic of Macedonia—Policy-Based Guarantee. Independent Evaluation Group, Project Performance Assessment Report 106177. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_macedonia_policy_0716.PDF. ———. 2016. Ghana—Economic Governance and Poverty Credit, and Seventh and Eighth Poverty Reduction Support Credits. Independent Evaluation Group, Project Performance Assessment Report 106279. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/ files/Data/reports/ppar_ghanaecon_0716.PDF. ———. 2016. Ghana—Statistical Development Project. Independent Evaluation Group, Project Perfor- mance Assessment Report 106283. Washington, DC: World Bank. http://ieg.worldbankgroup. org/sites/default/files/Data/reports/ppar_Ghana_0816_0.PDF. ———. 2016. India—Mumbai Urban Transport Project. Independent Evaluation Group, Project Perfor- mance Assessment Report 106391. Washington, DC: World Bank. http://ieg.worldbankgroup. org/sites/default/files/Data/reports/ppar_india_0716.PDF. ———. 2016. Industry Competitiveness and Jobs: An Evaluation of World Bank Group Industry-Specific Support to Promote Industry Competitiveness and Its Implications for Jobs. Independent Evalua-tion Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/ files/Data/Evaluation/files/industry-competitiveness-and-jobs.PDF. ———. 2016. Lao People’s Democratic Republic—Poverty Reduction Fund Project. Independent Evalu- ation Group, Project Performance Assessment Report 106387. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_laos_0716.PDF. ———. 2016. Learning from IDA Experience: Lessons from IEG Evaluations. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/re- ports/lp_ida_0716.PDF. Independent Evaluation Group | World Bank Group 365 ———. 2016. Lessons from Environmental Policy Lending. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/lp_envi- ronmentalpolicylending_0716_0.PDF. ———. 2016. Malawi— Third Social Action Fund (MASAF 3) APL II (LDF Mechanism) Project. Independent Evaluation Group, Project Performance Assessment Report 106389. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_malawi_0716.PDF. ———. 2016. Montenegro—Financial Sector Policy-Based Guarantee. Independent Evaluation Group, Project Performance Assessment Report 106282. Washington, DC: World Bank. http://ieg.world- bankgroup.org/sites/default/files/Data/reports/ppar_montenegro_0716.PDF. ———. 2016. Morocco—Urban Transport Sector Development Policy Loan. Independent Evaluation Group, Project Performance Assessment Report 106390. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_morocco_2016.PDF. ———. 2016. Mozambique— Poverty Reduction Support Credits 3–5. Independent Evaluation Group, Project Performance Assessment Report 106459. Washington, DC: World Bank. http://ieg.world- bankgroup.org/sites/default/files/Data/reports/ppar_mozambique_0716.PDF. ———. 2016. Niger—Ownership, Capacity, Results Focus and Accountability: Lessons for Enhancing the Bank’s Program Approach to Health and Population. Independent Evaluation Group, Project Per- formance Assessment Report 106254. Washington, DC: World Bank. http://ieg.worldbankgroup. org/sites/default/files/Data/reports/ppar_niger_0716.PDF. ———. 2016. Peru—Lima Water Rehabilitation and Management Project and National Rural Water Supply and Sanitation Project. Independent Evaluation Group, Project Performance Assessment Report 106723. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/ reports/ppar_peru_0716.PDF. ———. 2016. Senegal—Urban Mobility Improvement Project. Independent Evaluation Group, Project Per- formance Assessment Report 108407. Washington, DC: World Bank. http://ieg.worldbankgroup. org/sites/default/files/Data/reports/ppar_Senegal_102016.PDF. ———. 2016. Tanzania—Second Social Action Fund. Independent Evaluation Group, Project Performance Assessment Report 106388. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/ default/files/Data/reports/ppar_tanzania_0716.PDF. ———. 2016. The Role of Political Economy Analysis in Development Policy Operations. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/reports/lp_poleconanalysis_0716.PDF. ———. 2016. The World Bank Group’s Support to Capital Market Development. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/Eval- uation/files/wbgsupporttocapitalmarketdev-full.PDF. 366 Results and Performance of the World Bank Group 2017 | Appendix L ———. 2016. Vietnam—Evaluation of Public Financial Management Reform. Independent Evaluation Group, Project Performance Assessment Report 106635. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_vietnam_0716.PDF. ———. 2016. World Bank Group Engagement in Situations of Fragility, Conflict, and Violence. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/Evaluation/fi es/fcv-full.PDF, ———. 2016. World Bank Group Engagement in Small States: The Cases of the OECS, Pacific Island Countries, Cabo Verde, Djibouti, Mauritius, and the Seychelles — Clustered Country Program Evaluation. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbank- group.org/sites/default/files/Data/reports/ExecSum-SmallStates.PDF. ———. 2016. Zambia— Public Sector Management Program Support Project. Independent Evaluation Group, Project Performance Assessment Report 106280. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_zambia_0716.PDF. ———. 2017. Arab Republic of Egypt—Second Pollution Abatement Project. Independent Evaluation Group, Project Performance Assessment Report 113565. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar-egypt-04282017.PDF. ———. 2017. Argentina—Buenos Aires Urban Transport Project. Independent Evaluation Group, Project Performance Assessment Report 113664. Washington, DC: World Bank. http://ieg.worldbank- group.org/sites/default/files/Data/reports/ppar-argentina-04282017.PDF. ———. 2017. Brazil—Pernambuco Education Results and Accountability Project. Independent Evaluation Group, Project Performance Assessment Report 114046. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar-brazil-07142017.PDF. ———. 2017. Cambodia—Public Financial Management and Accountability Project. Independent Evalu- -ation Group, Project Performance Assessment Report 108359. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_cambodia_12282016.PDF. ———. 2017. Colombia—Disaster Risk Management Development Policy Loan. Independent Evaluation Group, Project Performance Assessment Report 114745. Washington DC: World Bank. http://ieg. worldbankgroup.org/sites/default/files/Data/reports/ppar-colombiadisaster-07182017.PDF. ———. 2017. Colombia—Private Paths to Sustainable Water Supply and Sanitation in Colombia. Indepen- dent Evaluation Group, Project Performance Assessment Report 111404. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_colombia_01132017. PDF. ———. 2017. Croatia—Agriculture Pollution Control Project. Independent Evaluation Group, Project Perfor- mance Assessment Report 115691. Washington, DC: World Bank. http://ieg.worldbankgroup.org/ sites/default/files/Data/reports/ppar-croatia-06272107.PDF. Independent Evaluation Group | World Bank Group 367 ———. 2017. Ethiopia, Mali, Morocco, South Africa, Tanzania, Tunisia—Africa Stockpiles Program. Inde- pendent Evaluation Group, Project Performance Assessment Report 108524. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_africa_102416_2. PDF. ———. 2017. Ghana—Agriculture Development Policy Operations: Phase I–IV. Independent Evaluation Group, Project Performance Assessment Report 112622. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar-ghanaagridpo-3202017.PDF. ———. 2017. Ghana—eGhana Project. Independent Evaluation Group, Project Performance Assessment Report 108359. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/reports/ppar_ghana_01032017_0.PDF. ———. 2017. Higher Education for Development: An Evaluation of the World Bank Group’s Support. Inde- pendent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/ default/files/Data/Evaluation/files/highereducation.PDF. ———. 2017. Indonesia—Public Expenditure Support Facility Development Policy Loan with Deferred Drawdown Option. Independent Evaluation Group, Project Performance Assessment Report 110741. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/re- ports/ppar-indonesia-06262017.PDF ———. 2017. Kenya—Development of the National Statistical System Project. Independent Evaluation Group, Project Performance Assessment Report 110966. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_kenya_01042017.PDF. ———. 2017. Kyrgyz Republic—Rural Education Project. Independent Evaluation Group, Project Perfor- mance Assessment Report 114047. Washington, DC: World Bank. http://ieg.worldbankgroup.org/ sites/default/files/Data/reports/ppar-kyrgyzeducation-07172017.PDF. ———. 2017. Lao People’s Democratic Republic—Second Education Development Project. Independent Evaluation Group, Project Performance Assessment Report 113148. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar-laos-04192017.PDF. ———. 2017. Nepal—Second Rural Water Supply and Sanitation Project. Independent Evaluation Group, Project Performance Assessment Report 114286. Washington, DC: World Bank. http://ieg.world- bankgroup.org/sites/default/files/Data/reports/ppar-nepalwater-07122017.PDF. ———. 2017. Nigeria—National Water Sector Reform Project. Independent Evaluation Group, Project Per- formance Assessment Report 115782. Washington, DC: World Bank. http://ieg.worldbankgroup. org/sites/default/files/Data/reports/ppar-nigeriawater-07122017.PDF. 368 Results and Performance of the World Bank Group 2017 | Appendix L ———. 2017. Panama—Improving Basic Health, Equity, Opportunities among Indigenous and Poor Rural Communities. Independent Evaluation Group, Project Performance Assessment Report 110617. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/reports/ppar-panama-07052017.PDF. ———. 2017. Peru—Fiscal Management and Competitiveness Programmatic Development Policy Loans. Independent Evaluation Group, Project Performance Assessment Report 110742. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_perufis- cal_03062017.PDF. ———. 2017. Philippines—Disaster Risk Management Development Policy Loan with a Catastrophe Deferred Drawdown Option. Independent Evaluation Group, Project Performance Assessment Report 114746. Washington, DC: World Bank. http://ieg.worldbankgroup.org/ sites/default/files/Data/reports/ppar-philippinesdisaster-07172017.PDF. ———. 2017. Program-for-Results: An Early-Stage Assessment of the Process and Effects of a New Lend- ing Instrument. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.world- bankgroup.org/sites/default/files/Data/Evaluation/files/program-for-results-full.PDF. ———. 2017. Public-Private Partnerships in Health: World Bank Group Engagement in Health PPPs—An IEG Synthesis Report. Independent Evaluation Group. Washington DC: World Bank. http://ieg. worldbankgroup.org/sites/default/files/Data/Evaluation/files/lp_Health_PPP_1116.PDF. ———. 2017. Reliable and Affordable Off-Grid Electricity Services for the Poor: Lessons from the World Bank Group Experience. Independent Evaluation Group. Washington, DC: World Bank. http://ieg. worldbankgroup.org/sites/default/files/Data/Evaluation/files/lp_off-grid_electricity_1116.PDF. ———. 2017. Senegal—A Decade of World Bank Support to Senegal’s Nutrition Program. Independent Evaluation Group, Project Performance Assessment Report 110290. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/281111484163839170/PDF/110290-PPAR- P070541-P097181-P115938-PUBLIC.PDF. ———. 2017. Senegal—Une décennie de soutien de la Banque mondiale au programme de nutrition du Sénégal. Independent Evaluation Group, Project Performance Assessment Report 110290. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ ppar-senegal-fr-06052017.PDF. ———. 2017. Solomon Islands—Rural Development Program. Independent Evaluation Group, Project Per- formance Assessment Report 110966. Washington, DC: World Bank. http://ieg.worldbankgroup. org/sites/default/files/Data/reports/ppar_solomonislands_01042017.PDF. ———. 2017. Sri Lanka—Second Community Water Supply and Sanitation Project. Independent Evaluation Group, Project Performance Assessment Report 115781. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar-srilanka-07132017.PDF. ———. 2017. Tax Revenue Mobilization: Lessons from World Bank Group Support for Tax Reform. Inde- pendent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/ default/fi es/Data/Evaluation/fi es/lp-taxrevenue-03152017.PDF. Independent Evaluation Group | World Bank Group 369 ———. 2017. Ukraine—Development of State Statistics System for Monitoring Social and Economic Trans- formation (Devstat) Project. Independent Evaluation Group, Project Performance Assessment Report 113588. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/reports/ppar-ukraine-4192017.PDF. ———. 2017. Women’s Empowerment in Rural Community-Driven Development Projects. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/Evaluation/fi es/lp_genderincdd_01272017.PDF. ———. 2017. World Bank Group Country Engagement: An Early-Stage Assessment of the Systematic Country Diagnostic and Country Partnership Framework Process and Implementation. Inde- pendent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/ default/files/Data/Evaluation/files/scd-cpf.PDF. ———. 2017. World Bank Group Joint Projects: A Review of Two Decades of Experience. Independent Evaluation Group. Washington, DC. https://ieg.worldbankgroup.org/evaluations/world-bank- group-joint-projects-review-two-decades-experience. ———. 2017. Zambia—Water Sector Performance Improvement Project. Independent Evaluation Group, Project Performance Assessment Report 111134. Washington, DC: World Bank. http://ieg.world- bankgroup.org/sites/default/files/Data/reports/ppar_zambia_01052017.PDF. 1 Summaries of the reviews are provided in the section titled Evaluation Findings from Country Program Performance. 2 Summaries of the reviews are provided in the section titled Evaluation Findings from Country Program Performance. 3 Summaries of the reviews are provided in the section titled Evaluation Findings from Country Program Performance. 4 One of eight Social Protection and Labor projects was rated below moderately satisfactory or better; however, it accounted for only $13 million of $4,545 million value of Social Protection and Labor projects rated during FY14—16. 5 One project was rated below moderately satisfactory or better for the Governance Global Practice; however, it accounted for only $8 million of the $2,636 million committed to the projects rated during FY14—16. 6 Summaries of the reviews are provided in the section titled Evaluation Findings from Country Program Performance. 7 The Independent Evaluation Group has remapped all evaluated International Finacne Corporation investment (IFC) and advisory services projects in Afghanistan and Pakistan to South Asia for the 2017 Results and Performance of the World Bank Group (RAP). Hence, IFC project-level results and performance for Middle East and North Africa and South Asia Region in this RAP should not be compared with previous RAPs. 8 For he 2017 Results and Performance of the World Bank Group, Afghanistan and Pakistan are classified as part of South Asia to refl ct the IFC’s current organizational structure. These two countries were classifi d as part of Middle East and North Africa in IFC’s past portfolio database. 10 Summaries of the reviews are provided in the section titled Evaluation Findings from Country Program Performance The Independent Evaluation Group has remapped all evaluated International Finacne Corporation investment (IFC) and 11 advisory services projects in Afghanistan and Pakistan to South Asia for the 2017 Results and Performance of the World Bank Group (RAP). Hence, IFC project-level results and performance for Middle East and North Africa and South Asia Region in this RAP should not be compared with previous RAPs. 370 Results and Performance of the World Bank Group 2017 | Appendix L Independent Evaluation Group | World Bank Group 371 Appendix M. Independent Evaluation Group Note on Practice Group Performance guide to appendix M Equitable Growth, Finance, and Institutions Practice Group 374 Portfolio Performance 374 Evaluation Findings and Lessons 386 Human Development Practice Group 396 Portfolio Performance 396 Evaluation Findings and Lessons 408 Sustainable Development Practice Group 415 Portfolio Performance 415 Evaluation Findings and Lessons 427 List of Summarized Works 444 Figures Figure M.1. EFI Group: Project Outcome Ratings, FY05–16 375 Figure M.2. EFI Group: Project Outcome Ratings by Global Practice, FY14–16 376 Figure M.3. EFI Group: Project Outcome Ratings by Practice Group and Financing Source, FY05–16 377 Figure M.4. EFI Group: Project Outcome Ratings by Practice Group and Lending Instrument, FY05–16 378 Figure M.5. EFI Group: World Bank Performance Ratings by Practice Group, FY05–16 379 Figure M.6. EFI Group: World Bank Performance Ratings by Global Practice, FY14–16 380 Figure M.7. EFI Group: M&E Quality Ratings by Practice Group, FY05–16 381 Figure M.8. EFI Group: M&E Quality Ratings by Global Practice, FY14–16 382 Figure M.9. EFI Group: Project Outcome Ratings for Practice Group by Region, FY14–16 383 Figure M.10. EFI Group: World Bank Performance Ratings for Practice Group by Region, FY14–16 384 Figure M.11. EFI Group: M&E Quality Ratings for Practice Group by Region, FY14–16 385 Figure M.12. HD Group: Project Outcome Ratings, FY05–16 397 Figure M.13. HD Group: Project Outcome Ratings by Global Practice, FY14–16 398 Figure M.14. HD Group: Project Outcome Ratings by Practice Group and Financing Source, FY05–16 399 Figure M.15. HD Group: Project Outcome Ratings by Practice Group and Lending Instrument, FY05–16 400 Figure M.16. HD Group: World Bank Performance Ratings, FY05–16 401 Figure M.17. HD Group: World Bank Performance Ratings by Global Practice, FY14–16 402 Figure M.18. HD Group: M&E Quality Ratings, FY05–16 403 Figure M.19. HD Group: M&E Quality Ratings by Global Practice, FY14–16 404 Figure M.20. HD Group: Project Outcome Ratings for Practice Group by Region, FY14–16 405 Figure M.21. HD Group: World Bank Performance Ratings for Practice Group by Region, FY14–16 406 Figure M.22. HD Group: M&E Quality Ratings for Practice Group by Region, FY14–16 407 Figure M.23. SD Group: Project Outcome Ratings, FY05–16 417 Figure M.24. SD Group: Project Outcome Ratings by Global Practice, FY14–16 418 Figure M.25. SD Group: Project Outcome Ratings by Practice Group and Financing Source, FY05–16 419 Figure M.26. SD Group: Project Outcome Ratings by Practice Group and Lending Instrument, FY05–16 420 Figure M.27. SD Group: World Bank Performance Ratings, FY05–16 421 Independent Evaluation Group | World Bank Group 371 Figure M.28. SD Group: World Bank Performance Ratings by Global Practice, FY14–16 422 Figure M.29. SD Group: M&E Quality Ratings, FY05–16 423 Figure M.30. SD Group: M&E Quality Ratings by Global Practice, FY14–16 424 Figure M.31. SD Group: Project Outcome Ratings for Practice Group by Region, FY14–16 425 Figure M32. SD Group: World Bank Performance Ratings for Practice Group by Region, FY14–16 426 Figure M33. SD Group: M&E Quality Ratings for Practice Group by Region, FY14–16 426 372 Results and Performance of the World Bank Group 2017 | Appendix M abbreviations CDD community-driven development DPF development policy financing DPL development policy loan DPO development policy operation DRM disaster risk management EGPRC Economic Governance and Poverty Reduction Credit ENR Environment and Natural Resources GDP gross domestic product GP Global Practice EFI Human Development HEPI Health Equity and Performance Improvement HNP Health, Nutrition, and Population IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report ICT information and communication technology IDA International Development Association IEG Independent Evaluation Group IFC International Finance Corporation IMF International Monetary Fund IPF investment project fi ancing M&E monitoring and evaluation MDTF multi-donor trust fund MIS management information system PBG policy-based guarantee PFM public financial management PPAR Project Performance Assessment Reports PPP public-private partnership PRSC Poverty Reduction Support Credits SD Sustainable Development SPL Social Protection and Labor WSS water supply and sanitation Independent Evaluation Group | World Bank Group 373 Equitable Growth, Finance, and Institutions Practice Group For perspective, the following introductory paragraphs provide an indication of the relative size, in number and volume, of the three practice groups: Sustainable Development (SD), Human Development (HD), and Equitable Growth, Finance, and Institutions (EFI).1 With 412 projects, the SD Practice Group of Global Practices (GPs) holds more than half of the total 713 projects rated across all practice groups during fiscal years FY14–16. Under the EFI Practice Group, 141 projects were rated, and a further 158 were rated under the HD Practice Group. By volume, the EFI Practice Group is the second largest by volume ($18,748 million). The SD Practice Group is the largest—the 412 rated projects were supported by $35,609 million, and the HD Practice Group was supported with $15,035 million. The largest single GP by number of projects is the Social, Urban, Rural, and Resilience (SURR) GP (93 projects), which sits within the SD Practice Group. The smallest GP by number of projects (6) is Poverty and Equity, which sits under the EFI Practice Group. That GP is also the smallest by commitment volume ($670 million). The Transport and Information and Communication Technology (ICT) GP, under the SD Practice Group, is the largest GP by commitment volume of rated projects ($11,157 million). The largest number of development policy financing (DPF) is found under the EFI Practice Group (52 of 69 operations), also representing the greatest commitment volume for DPFs rated during FY14–16 ($12,785 million). The SD Practice Group holds the largest number of investment project financing (IPF; 399 of a total of 635 projects), including the greatest commitment volume for IPFs rated during FY14–16 ($33,412 million). Portfolio Performance The EFI Practice Group comprises five GPs: ƒƒ Finance and Markets: seeks to develop resilient, ef ficient, and transparent financial sys- tems to help mobilize capital needed for investments in country development priorities. ƒƒ Governance: develops innovative, integrated solutions to pernicious institutional prob- lems using a problem-driven, diagnostic approach that combines knowledge of reform successes and failures with a keen understanding of institutional challenges and oppor- tunities in developing countries. ƒƒ Macroeconomics and Fiscal Management: provides integrative development strate- gies, policy-based lending, macro data, global perspectives, real-time policy analysis, country risk assessments, and innovative projection tools. ƒƒ Poverty: seeks to deliver advice and integrated knowledge to identify key policies and multisectoral solutions that effectively reduce poverty and benefit the less well-off, and to help better understand the relationship between growth, poverty, and inequality. 374 Results and Performance of the World Bank Group 2017 | Appendix M ƒƒ Trade and Competitiveness: helps countries develop more dynamic and integrated economies by boosting trade, enhancing the investment climate, improving competi- tiveness in sectors, and fostering innovation and entrepreneurship. Outcome rating was consistent across GPs within EFI, with each GP near the groupwide rating of 67 percent MS+ during FY14–16 (figure M.2). For the current review period, FY14–16, 67 percent of all 141 projects closed under the EFI Practice Group were rated moderately satisfactory or better (MS+), which is below the average outcome rating (75 percent MS+) for all projects in non-EFI groups over the same period, and the World Bank–wide 73 percent MS+ average (figure M.1, panel a).2,3 At a satisfaction rating of 86 percent MS+, projects in the EFI Practice Group outperform all non-EFI projects (83 percent MS+), and the World Bank– wide average (84 percent MS+), when rated by commitment (figure M.1, panel b). Figure M.1. EFI Group: Project Outcome Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: Fourteen projects were excluded when measured by volume due to lack of volume data. EFI = Equitable Growth, Finance, and Institutions; MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 375 Outcome rating was consistent across GPs within EFI, with each GP near the groupwide rating of 67 percent MS+ during FY14–16 (figure M.2). There was greater variation in performance levels by GP when measured by commitment, ranging from a 99 percent MS+ rating for projects in the Poverty and Equity GP (the smallest of the practices in volume terms [$670 million]), to 64 percent MS+ for the Trade and Competitiveness GP. Performance for the largest GP by volume, Macro Economics and Fiscal Management ($9,881 million), was 87 percent MS+. Figure M.2. EFI Group: Project Outcome Ratings by Global Practice, FY14–16 Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. EFI = Equitable Growth, Finance, and Institutions; MS+ = a rating of moderately satisfactory or better. The overall outcome rating for International Bank for Reconstruction and Development (IBRD) projects (n = 52) in the EFI Practice Group FY14–16 is 83 percent MS+, and the rating for International Development Association (IDA) projects (n = 65) is 60 percent MS+ (figure M.3, panel a). This compares with respective average outcome ratings of 74 percent MS+ and 76 percent MS+ for non- EFI projects rated over the same period (figure M.3, panels b and c). Within the EFI group, DPF outperformed IPF with project outcome ratings at 75 percent MS+ compared with 62 percent MS+ (figure M.4, panel a). The performance of the 86 IPF projects rated in the EFI Practice Group (62 percent MS+) compares with a 75 percent MS+ average rating for IPFs across the other two clusters in the same period (figure M.4, panel b). The outcome rating for the 52 DPF projects rated for the EFI Practice Group during FY14–16 (75 percent MS+) is slightly below the 76 percent non-EFI Practice Group ratings, noting a total of only 17 DPFs across the other two practice groups (figure M.4, panel c). World Bank performance, a composite of ratings for quality at entry and project supervision, was 67 percent MS+ for all projects in the EFI Practice Group compared with 74 percent MS+ for non- EFI groups, and 72 percent MS+ World Bank–wide. By commitment, World Bank performance for projects in the EFI Practice Group (87 percent MS+) is greater than the World Bank–wide average rating (84 percent MS+), and the average rating for non-EFI groups (83 percent MS+; figure M.5). 376 Results and Performance of the World Bank Group 2017 | Appendix M Figure M.3. EFI Group: Project Outcome Ratings by Practice Group and Financing Source, FY05–16 a. IBRD versus IDA: Percentage rated MS+, by number b. IBRD projects—EFI versus other practice groups: Percentage rated MS+, by number c. IDA projects—EFI versus other practice groups: Percentage rated MS+, by number Source: Independent Evaluation Group data. Note: One project was excluded when measured by volume due to lack of volume data. EFI = Equitable Growth, Finance, and Institutions; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 377 Figure M.4. EFI Group: Project Outcome Ratings by Practice Group and Lending Instrument, FY05–16 a. IPF versus DPF: Percentage rated MS+, by number b. IPF projects: EFI versus other practice groups: Percentage rated MS+, by number c. DPF projects: EFI versus other practice groups: Percentage rated MS+, by number Source: Independent Evaluation Group data. Note: One project was excluded when measured by volume due to lack of volume data. DPF = development policy financing; EFI = Equitable Growth, Finance, and Institutions; IPF = investment project financing; MS+ = a rating of moderately satisfactory or better. 378 Results and Performance of the World Bank Group 2017 | Appendix M Figure M.5. EFI Group: World Bank Performance Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: Thirteen projects were excluded when measured by volume due to lack of volume data. EFI = Equitable Growth, Finance, and Institutions; MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 379 Under the EFI Practice Group, World Bank performance is relatively better than performance under non-EFI groups when measured by IBRD agreement type (83 percent MS+ versus 74 percent MS+), but underperforms non-EFI groups when measured by the IDA agreement type (60 percent MS+ versus 75 percent MS+). Likewise, EFI World Bank performance exceeds performance under non-EFI groups when measured by the DPF instrument (85 percent MS+ versus 76 percent MS+), but underperforms non-EFI groups when measured by the IPF instrument (74 percent MS+ versus 56 percent MS+). World Bank performance for the individual GPs within the practice group varied. World Bank performance under Macro Economic and Fiscal Management (the second largest GP with 44 projects) was 82 percent MS+ (by number), and 89 percent MS+ by commitment. For the Governance GP, which had the largest number of rated projects (49), World Bank performance was rated 59 percent MS+, the lowest for any of the practices under the EFI Practice Group in FY14–16 (figure M.6). The quality of monitoring and evaluation (M&E) for EFI projects is 35 percent substantial or better (S+), slightly better than the 34 percent average S+ rating both World Bank–wide, and for non-EFI groups. At a rating of 46 percent S+ during FY14–16, the quality of M&E for EFI projects by commitment matches the World Bank–wide rating, and is a little above the rating for non-EFI groups (45 percent S+; figure M.7). Figure M.6. EFI Group: World Bank Performance Ratings by Global Practice, FY14–16 Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. EFI = Equitable Growth, Finance, and Institutions; MS+ = a rating of moderately satisfactory or better. 380 Results and Performance of the World Bank Group 2017 | Appendix M Figure M.7. EFI Group: M&E Quality Ratings, FY05–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: Nine projects were excluded when measured by volume due to lack of volume data. EFI = Equitable Growth, Finance, and Institutions; M&E = monitoring and evaluation; S+ = a rating of substantial or better. M&E quality for the individual GPs within the group ranges from a high of 45 percent S+ for the 20 projects rated under the Finance and Market GP to a low of 30 percent S+ for the 23 projects rated under the Trade and Competitiveness GP. By commitment, the spread in the ratings was wider, with respective ratings of 58 percent and 56 percent S+ for the Finance and Markets and Macroeconomics and Fiscal Management GPs, to respective M&E ratings of 15 and 8 percent S+ for the Trade and Competitiveness and Poverty and Equity GPs (figure M.8). At the regional level, project outcome ratings under EFI were relatively strong in the East Asia and Pacific (76 percent MS+), Europe and Central Asia (77 percent MS+), and Latin America and the Caribbean (76 percent MS+) Regions, but less so for the Africa (58 percent MS+), Middle East and North Africa (56 percent MS+), and South Asia (50 percent MS+) Regions (figure M.9, panel a). Independent Evaluation Group | World Bank Group 381 Figure M.8. EFI Group: M&E Quality Ratings by Global Practice, FY14–16 Source: Independent Evaluation Group data. Note: Two projects were excluded when measured by volume due to lack of volume data. EFI = Equitable Growth, Finance, and Institutions; M&E = monitoring and evaluation; S+ = a rating of substantial or better. By commitment, the Europe and Central Asia (99 percent MS+), Middle East and North Africa (93 percent MS+), and Latin America and the Caribbean (83 percent MS+) Regions perform strongest under the EFI Practice Group (figure M.9, panel b). World Bank performance for EFI projects ranges from a high of 77 percent MS+ in the Latin America and the Caribbean Region to a low of 50 percent MS+ in the South Asia Region, compared with a World Bank–wide rating of 67 percent MS+ (figure M.10). Against a World Bank–wide rating of 35 percent S+, M&E ratings for EFI projects are particularly strong in the Europe and Central Asia (58 percent S+) and Middle East and North Africa (50 percent S+) Regions but relatively weak in the Africa (24 percent S+), and South Asia (11 percent S+) Regions (figure M.11). 382 Results and Performance of the World Bank Group 2017 | Appendix M Figure M.9. EFI Group: Project Outcome Ratings for Practice Group by Region, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. EFI = Equitable Growth, Finance, and Institutions; MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 383 Group: World Bank Performance Ratings for Practice Group Figure M.10. EFI by Region, FY14–16 Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. “Other” includes projects that are not mapped to any Practice group. EFI = Equitable Growth, Finance, and Institutions; MS+ = a rating of moderately satisfactory or better. 384 Results and Performance of the World Bank Group 2017 | Appendix M Figure M.11. EFI Group: M&E Quality Ratings for Practice Group by Region, FY14–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: Two projects were excluded when measured by volume due to lack of volume data. EFI = Equitable Growth, Finance, and Institutions; M&E = monitoring and evaluation; S+ = a rating of substantial or better. Independent Evaluation Group | World Bank Group 385 Evaluation Findings and Lessons This section summarizes key findings and lessons from Project Performance Assessment Reports (PPARs) completed by the Independent Evaluation Group (IEG) in FY17 for GPs under the EFI Practice Group. No major evaluations or learning products were completed for the EFI Practice Group in FY17. For the EFI Practice Group, IEG completed one PPAR under the Finance and Markets GP, four under the Governance GP covering five projects, seven under the Macroeconomic and Fiscal GP, and two under the Poverty GP. There were no PPARs completed under the Trade and Competitiveness GP during FY17. Finance and Markets The 2012–13 Financial Sector Policy-Based Guarantee for Montenegro was prepared by converting the second operation of the programmatic series of the Financial Sector Development Policy Loan in the amount of $20 million into a guarantee instrument to respond to the large external financing needs of Montenegro with the sole objective of strengthening the banking system. IEG found that many important outcomes achieved in Montenegro’s banking system—banking system largely recovered, market confidence restored, and total deposits surpassing precrisis level—were attributable to the strong reform actions supported by the financial sector development policy loan (DPL), with the Financial Sector Policy-Based Guarantee having relatively low value-added. World Bank performance was rated moderately unsatisfactory, largely reflecting the poor choice of instrument, and modest program design reflecting the lack of operational focus on the program’s macroeconomic framework, as well as risks and shortcomings in assessing Montenegro’s eligibility for a policy-based guarantee (PBG). The evaluation found the reform program to be incomplete, with gaps evident in crisis management, less than full compliance in banking supervision with the Basel Core Principles for Effective Banking Supervision, and a slower than expected decline in nonperforming loans. Withdrawal of public sector deposits in Prva Banka to ensure market-based funding was reversed in 2014 and 2015, and the World Bank continued to hold a large amount of public sector deposits at the time of the evaluation. These factors contributed to significant risk to development outcome. Among the lessons derived from the evaluation is that the choice of a PBG as an instrument should be based on robust macro and fiscal projections indicating that the financing framework is sustainable, and macro risks are mitigated (highlighting the criticality of World Bank– International Monetary Fund [IMF] cooperation). The World Bank’s Implementation Completion and Results Report (ICR) rated outcome as satisfactory, but IEG’s independent field-based evaluation delivered an outcome rating of moderately unsatisfactory, reflecting IEG’s assessment of the inadequacy of the PBG as an engagement modality, the lack of focus on the macroeconomic framework and risks, and an incomplete banking system reform agenda. Governance GP The five projects covered under four PPARs completed by IEG in FY17 under the Governance GP are summarized below. 386 Results and Performance of the World Bank Group 2017 | Appendix M The Public Financial Management and Accountability Project operated in Cambodia between June 29, 2007, and November 15, 2013. The project aimed to strengthen the mobilization and use of public resources, bolster accountability institutions, and develop the capacity of the civil service. It became effective shortly after the World Bank’s Integrity Vice Presidency found suf ficient evidence to substantiate allegations of fraud and corruption in seven unrelated projects. The World Bank ultimately suspended all lending to Cambodia in 2011 after the already tense relationship was further aggravated over a land settlement dispute. To continue engagement, the World Bank appointed an Independent Procurement Agency. The project ultimately made some contribution to revenue generation, though this was predominantly driven by external factors such as a shift in the composition of gross domestic product (GDP), and the formalization of the economy. The project also facilitated improvement in the enabling environment for public financial management, and human resource management, but failed to fully achieve planned inroads. The Financial Management Information System contract award collapsed after five years of failed procurement, and at project end, none of the expected benefits in expenditure management from such a system had materialized meaning there was no input to ongoing design, procurement, and implementation strategy. The treasury single account was consolidated and the accounts codes were updated to reflect international standards. The ICR rated overall development outcome moderately unsatisfactory, which IEG downgraded to unsatisfactory based on an assessment of negligible ef ficiency and modest ef ficacy for three out of four objectives. The Statistical Development Project in Ghana was approved at a time when the global development community became acutely aware of the need to upgrade and significantly increase the supply of relevant statistics. Quality data was sought in the quest to improve policy making, enhance governance, and monitor progress. The Trust Fund for Statistical Capacity Building, a World Bank– administered, multi-donor trust fund (MDTF) provided the grant to prepare Ghana Statistical Service for institutional reform, and to ensure timely and extensive analysis and dissemination of census data. The MDTF included the Department for International Development of the United Kingdom and the European Commission. Total project cost was estimated at $7.37 million. Actual cost was $5.9 million, with which the project supported legal and institutional reform, Ghana Statistical Service capacity building, improved quality and dissemination of statistical products, and 2010 census postenumeration. In principle, the design of the project should have been clear and straight-forward, but the complexity of institutional reforms, and fragmentation of reform efforts, made reforms dif ficult to implement. This weakness in design offered little assurance of the likelihood for success. In fact, in supporting corporate or institutional reform agendas, it is essential to ensure orderly, ef ficient, and effective implementation to minimize the impact on staff. The project provided two valuable lessons on statistical capacity building projects. First, it highlighted the need for a responsive governing body to give strategic direction to the national statistical of fice to ensure data are relevant and reliable, emphasizing the importance of having present, directors who represent the interest of data users (researchers, development partners, Independent Evaluation Group | World Bank Group 387 academicians, students and policy makers). Second, the capacity of the World Bank team to deliver results in an area in which it played a large role in mobilizing international support (that is, for statistical development) should not be taken for granted. Even though the World Bank team appeared to be well qualified, and the task team seemed to possess the right skills mix, project results were modest. The main shortcoming was in achieving institutional reforms. At the time of IEG’s evaluation, the Statistics Bill had not yet been approved by parliament. Timely analysis and dissemination of the 2010 census was achieved before the project closed. Both the World Bank’s ICR and IEG’s assessment rated the project outcome unsatisfactory. The project development objectives for the Public Financial Management Reform Project that was implemented in Vietnam May 22, 2003, to October 31, 2013, (inclusive of a period supported by additional financing) were to strengthen the government’s capacity to plan, execute and report on its budget, and to improve the transparency and accountability of the budgetary systems and processes. The objectives for the supporting MDTF for Public Financial Management Modernization was to strengthen the client’s capacity for increased coordination and transparency in public finance management; improved revenue mobilization; and effective and ef ficient public expenditure management. That project was implemented April 1, 2009, to June 30, 2012, inclusive of extensions. Both projects experienced slow initial starts and various delays, but the bulk of activities and reforms were completed by 2012/13, most notably reform of the treasury budget management information system (MIS) that consumed the bulk of financing. Capacity building proved to be effective and sustainable, and top-down and bottom-up ownership policy actions not yet completed at closing were followed through by the government. The review estimated that 22 public financial management (PFM) reforms were directly attributable to the MDTF II. Similarly, the modern treasury management systems out in place appeared to be one of the best performing systems (for coverage, functionality, and reliability) implemented under similar World Bank operations. Both operations benefited from strong fundamentals, with suf ficiently ambitious and realistic objectives. Some lack of clarity in the logical framework was rectified in due course. PFM reform project achieved its objectives, principally on the successful implementation of the treasury management systems as well as development of the debt database and piloting of the medium-term expenditures framework in four ministries and three regions. Overall, the World Bank’s ICR rated development outcome moderately satisfactory for both the PFM reform project and the MDTF. IEG rated the development outcome for the former satisfactory and highly satisfactory for the latter. The Zambia Public Sector Management Program Support Project aimed to make public service delivery institutions more effective and ef ficient through strengthening PFM and reforming the civil service. The project contributed to some important outputs with regards to the objective of improved public expenditure management. A macro-fiscal forecasting and debt management system was set up and successfully implemented. A Financial Management Information System was rolled out across government, assuring system integrity and increasing the speed of transactions. Advanced budgeting and planning modalities were institutionalized. The effectiveness of the public service benefited from upstream investments leading to improved payment and control systems, and an important pay 388 Results and Performance of the World Bank Group 2017 | Appendix M policy was developed and approved. Efforts to introduce performance-based pay was unsuccessful due to insuf ficient staff qualification, staff shortages, and moonlighting in critical service delivery sectors. Lessons from the project were that good analytical work is an important underpinning for program design, but it is not suf ficient, since program design must be strategic, practical, prioritized, and account for the implementation and absorptive capacity of the borrower. Also, project design based on a complex institutional reform agenda would benefit from explicitly recognizing the iterative nature of the reform process. World Bank performance suffered from poor quality at entry, especially due to excessive complexity that made the project dif ficult to implement and manage. Both the World Bank’s ICR and IEG’s independent field-based assessment rated overall development outcome moderately unsatisfactory, reflecting modest design and mixed ef ficacy. Macroeconomics and Fiscal Management GP The seven PPARs completed by IEG in FY17 for projects under the Macroeconomics and Fiscal Management GP, are summarized below. IEG assessed development policy operations (DPO) supporting Ghana’s $394 million Economic Governance and Poverty Reduction Credit (EGPRC; 2009–10) and Poverty Reduction Support Credits 7 ($219 million) and 8 ($100 million), which were disbursed in 2011 and 2012. The first poverty reduction credit was stand-alone and sought to support a three-pronged effort to restore budgetary discipline and tackle long-standing public sector and energy issues, while protecting the poor. It worked across many policy areas such as PFM, public sector reform, and social protection. IEG rated World Bank performance as satisfactory, based on high relevance of design and objective as well as close coordination, particularly in risk assessment, with an IMF program, reflected in the stand-alone, two-tranche instrument. IEG’s evaluation found a fall in the fiscal deficit from 8.4 percent in 2008 to 3.2 percent in 2011 and other financial management improvements, reflecting improved budgetary discipline. Actions on wage bill growth were particularly important, though they were insuf ficient in mitigating underlying risks, which later became evident. Public sector reform results were mixed. For example, political interventions undermined efforts to reform the civil service, leading to increased wage hikes, and a continuous increase in the number of civil servants. On the other hand, improved social protection contributed to substantial drop in poverty across the country supported by the rapid expansion of the Livelihood Empowerment Against Poverty program, although the evaluation also notes that, despite fiscal consolidation, the social sectors budget remained at about 50 percent of total expenditures in 2011. Overall, the World Bank’s ICR rated development outcome satisfactory; IEG’s field-based assessment rated outcome moderately satisfactory. A programmatic series of two Poverty Reduction Support Credits (PRSCs) followed the 2009 stand- alone poverty reduction credit. The objectives of PRSCs 7 and 8 were like those of the EGPRC. They included restoring budget discipline and fi ancial stability; tackling long-standing public sector and Independent Evaluation Group | World Bank Group 389 energy issues; protecting the poor; and preparing the economy for the oil era (including managing expected oil revenues). Although objectives were highly relevant, there were shortcomings in design, such as not addressing emerging risks to overall macroeconomic stability from the single- spine salary system implementation, and lack of strong policy content for reforming supported organizations. Measures supported in PRSCs 7 and 8 to maintain fiscal discipline proved ineffective. Major macroeconomic destabilization ensued, characterized by monetary accommodation of the fiscal deficit, high inflation, and persistent pressure on the exchange rate. Ultimately, macroeconomic stabilization was achieved in 2015 with IMF program support and a new World Bank programmatic DPO series. The situation in public utilities improved somewhat due to tariff adjustments, but previously accumulated debt continues to undermine financial viability. IEG rated protecting the poor as substantial due to considerable progress in the Livelihood Empowerment Against Poverty program (beyond the gains under EGPRC), and ef ficiency gains from joint targeting. However, concerns about expenditure composition remain because administrative expenditures and debt servicing demands appear to crowd out public service expenditures. IEG also rated achievements regarding the objective of preparing the economy for the oil era as substantial. Ghana established institutions that could allow transparent oil revenue management and account for intergenerational considerations by creating the Ghana Heritage Fund for accumulating oil revenues for the next generations. IEG rated the World Bank performance as moderately unsatisfactory given the insuf ficient consideration in project designs to the single-spine salary system and PFM measures, and ongoing ineffective dialogue with authorities, albeit in a dif ficult environment. Lessons from the assessment include, for example, af firmation of important role that the World Bank can play in its policy lending, helping clients to preserve borrowing space and safeguard sustainability of pro- growth and pro-poor expenditure in the long run; and the criticality of adequate macroeconomic management as a precondition of budget support, and as relevant, the importance of World Bank–IMF cooperation; three objectives. Overall, the World Bank’s ICR rated development outcome moderately satisfactory. IEG’s field-based assessment rated outcome moderately unsatisfactory reflecting modest design for all objectives and substantial ef ficacy for two of the four objectives (social protection and oil revenue management)—fiscal discipline and public sector reforms objectives were not as well realized. Although Indonesia entered the global financial crisis in late 2008 in a sound economic position, structural weaknesses made it vulnerable to global financial crises, and the increased global risk aversion after September 2008 significantly affected its financial conditions. The World Bank Public Expenditure Support Facility DPL with Deferred Drawdown Option (March 2009 to December 2010), sought to assist the government in addressing the potential adverse impacts of the global financial crisis on public expenditures by supporting key policy measures in three areas: (i) reassuring fi ancial markets and maintaining fi ancial system stability; (ii) sustaining critical public expenditures while maintaining budget discipline; and (iii) facilitating private investment and supporting exports. As part of a $5.5 billion package, the loan provided a contingent line of credit of $2 billion, which the government committed to use only if it could not meet its financing needs for 2009 by defining specific drawdown triggers based on market conditions. 390 Results and Performance of the World Bank Group 2017 | Appendix M The program was very successful in achieving the goals of calming the financial market at a time of great uncertainty, and of safeguarding critical public expenditures. The more recent drastic reduction of energy subsidies went well beyond expectations when the operation was implemented. On the other hand, there was backtracking in investment regulations, as well as a lack of timely compliance with Extractive Industries Transparency Initiative obligations, although recent policy changes in foreign direct investment regulations and the lifting of Indonesia’s Extractive Industries Transparency Initiative suspension bode well for the continuation of the reform process in these areas. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the outcome rating was satisfactory based on data produced through the program’s monitoring and on additional information. In the early 2000s, Kenya’s statistical institutions were in decline; the capacity of Kenya’s National Statistical System to produce timely, high-quality, and relevant statistical information had deteriorated over the two decades prior to the project. Among the causes identified at the time were: declining budgetary support for the production and dissemination of statistics, inadequate professional staff at the management and technical levels, dilapidated statistical infrastructure and ICT equipment, and a loss of relevance due to inadequate engagement with users. The objective of the Development of the National Statistical System Project was, to establish a sustainable national statistical system to provide reliable, timely and accurate data in accordance with international standards. Total project cost was estimated at $50.5 million. In March 2007, an IDA credit of $20.5 million was approved, out of which $11.7 million was disbursed, while $8.8 million was canceled when the project closed in September 2012. The project supported institutional reforms to create an autonomous agency with clear mandates, training, and technical assistance to enhance managerial and professional skills, as well as statistical infrastructure and ICT equipment. The project was largely implemented as designed but experienced delays. Before project approval, the World Bank suspended new operations in Kenya because of concerns about governance. Later, the country suffered civic strife involving an outbreak of violence. The project offered useful lessons for statistical capacity building projects. First, ensuring institutional reforms, both corporate and legal, were in place before project implementation and capacity building efforts begin was an important element of success in Kenya and could be the standard approach to statistical capacity building efforts. Second, establishing Sector Working Groups (also known as interagency working groups) as a coordinating mechanism proved useful and durable in Kenya and could have wider applicability. A final lesson has broader implications: when the risk of corruption is identified, the mitigation measures need to be realistic and targeted. In the case of this project, World Bank staff maintained that the fiduciary risks were due to capacity limitations. But this view was not corroborated by experience. Overall, the World Bank ICR rated development outcome moderately unsatisfactory, which was upgraded following IEG’s field-based assessment, to moderately satisfactory. IEG observed that substantial progress had been made by the implementing agency, the Kenya National Bureau of Statistics, such that, by June 2016, all three objectives had been substantially achieved. The coverage of source data was enlarged by more frequent and regular surveys, and by an updated registry of business establishments; data are released in a timelier manner; and accessibility of data was advanced by a new dissemination framework put in place by Kenya National Bureau of Statistics Independent Evaluation Group | World Bank Group 391 management. Most of ficial statistics are now available online and usable as envisaged by the project, including historical data and anonymized microdata sets. Today, Kenya is one of the most advanced countries in Africa in terms of open access to of ficial data. The first PBG for the Former Yugoslav Republic of Macedonia (FYR Macedonia) was approved in 2011 in the face of recession in the Western Balkans, and market uncertainty. Despite solid macro management, the country was vulnerable to the global crisis as well as contagion from the crisis in Greece. The PBG, which combines features of a partial sovereign credit guarantee provided by the World Bank with stand-alone budget support financing through international commercial markets, supported a program of policy reform agreed with the World Bank. The operation aimed to (i) improve the government’s access to international financial markets, (ii) fortify public finances and functioning of labor markets by controlling public expenditures, (iii) strengthen social safety nets by improving ef ficiency and equity of conditional cash transfers, and (iv) support resilience of the financial sector by creating a Financial Stability Committee to coordinate actions across agencies and reduce systemic risk. IEG found the operation benefited from lessons generated from an earlier, pioneering operation that the World Bank had supported in Serbia and this contributed to IEG’s satisfactory assessment of World Bank performance, noting that the team also provided extensive informal technical assistance and encouraged learning to the client. The main messages from the PPAR for this first PBG for the FYR Macedonia are that in addition to direct benefits of enhanced access, volume, and terms of borrowing and support for policy reform, the PBG can help the borrower establish itself in the eyes of investors and broaden the investor base. But large financing may help postpone rather than implement reforms, hence the need for a strong macroeconomic framework and results framework. Also, a PBG works better when informed by relevant knowledge, longer-term engagement, and trust. IEG rated development outcome for the operation moderately satisfactory reflecting substantial achievement on the financial objective and less so on fiscal and safety net objectives The €155 million Public Expenditure Policy-Based Guarantee 2012–13 was the second PBG for FYR Macedonia in the aftermath of the global crisis. The guarantee, which supported a commercial loan in the amount of €250 million, had three specific objectives: improving the ef ficiency of public expenditures by strengthening PFM practices; improving the ef ficiency of service provision in the health sector; and strengthening social protection and inclusion. Design suffered from a lack of operational focus on the program’s macroeconomic and fiscal risks, and the relatively narrow scope of the PFM reform agenda at a time of heightened risks. This led to mixed results. The program helped to achieve clearance of arears and to introduce changes in treasury practices to prevent accumulation of future arrears. However, the information about arrears at the level of central government is not fully reliable, while arrears at the municipality level and in the health sector remain significant. The ef ficiency of capital expenditures deteriorated, and the replacement of the indexation mechanism for pensions by ad-hoc increases reduced the predictability of PFM. The program helped reduce the unit price of pharmaceuticals and some medical equipment, but savings yielded from these sources was tempered by an increase in the number and cost of prescriptions. 392 Results and Performance of the World Bank Group 2017 | Appendix M An improvement in service provision to underserved areas came through the introduction of a new framework that helped direct more resources to those areas, and the government’s active labor market programs were strengthened by improving access for vulnerable groups. Fiscal resources freed up because of improved targeting of social protection enhanced the levels of financial assistance to beneficiaries. World Bank performance was rated moderately satisfactory reflecting the emphasis on clearance of arrears, and treasury reforms to alleviate a key constraint affecting the quality of PFM, as well as shortcomings in addressing macro and fiscal risks. Among the key evaluation lessons is that the positive impact of public expenditure measures (wage control, reforms in health procurement, and better targeting of social protection) can be eroded if the overall program lacks an overarching fiscal and debt framework—as in this case, fiscal savings can be dissipated by introducing nonpriority spending such as ad-hoc pension increases. The World Bank’s ICR rated the project highly satisfactory, and IEG’s independent field-based evaluation concluded that the outcome rating was moderately satisfactory refl cting the shortcomings identifi d above. Prior Actions for Mozambique’s PRSCs 3–5 were aligned with the challenges facing Mozambique’s economy and were complementary to the concurrent support provided by the IMF, but there was an overreliance on PFM reforms to achieve the broader macroeconomic objective, to the exclusion of indicators of macroeconomic stability. Other activities in relation to governance, were well aligned. However, program design could have paid more attention to the urban/rural distribution of pro-poor expenditure from the national budget, as well as the quality of these expenditures. Macroeconomic performance improved during implementation of the PRSCs 3–5. There was strong GDP growth, and the current account balance of payments deficit declined. After PRSC-5 closed however, public debt increased to about 40 percent of GDP between 2010 and 2013, and then to 74 percent of GDP in 2015. In addition, although there was a substantial allocation of budget resources to priority sectors, there was no information on the distribution of these funds to the areas where the poor are located. Moreover, the government acknowledged in April 2016 that it had guaranteed undisclosed borrowings of about $1 billion by state enterprises which contravened the program of macroeconomic management agreed with the IMF. The guarantee represented a serious contingent liability for the economy. The IMF suspended disbursements from its support program to Mozambique. There were only modest gains against the program’s decentralization and governance targets. More progress was made in the productive sectors. Growth in GDP from agriculture was between 5 and 6 percent per year during the three-year implementation period, although it dropped to about 3.5 percent per year between 2009 and 2014—still a sound and sustained performance given the sector’s structural problems and the incidence of natural disasters. The World Bank’s ICRs rated overall outcome satisfactory for each of the three credits. Following IEG’s field-based assessment, development outcome was rated moderately unsatisfactory for all three credits. In 2006, Peru’s economy was performing well. To make a significant leap in its development, the government designed a comprehensive reform program that was supported by the World Bank’s Independent Evaluation Group | World Bank Group 393 programmatic DPLs—Fiscal Management and Competitiveness Programmatic DPLs (2007–15)—that sought to support improvement in the functioning of Peru’s public sector institutions and business environment. In September 2008, Peru’s financial conditions were temporarily, but significantly, affected by the onset of the global crisis. Consequently, the second loan and its supplement were granted with a deferred drawdown option to allow the government to proactively manage and contain the risk from the global economic crisis, as a preventive measure, while remaining committed to the reform program. The programmatic series was successful in making public sector processes more transparent and accessible, increasing sustainability of fiscal policy, strengthening budget reporting and planning, improving management of public spending, and expanding international trade. However, in many reform areas supported by the program, including the tax system, intergovernmental transfers, cost of doing business, and financial deepening, advances fell short of expectations or were reversed after program closing, which suggests that the root causes of the problems have not been satisfactorily addressed. The extent of the program’s contribution to risk management during the global financial crisis is dif ficult to assess for lack of metrics. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the outcome rating was moderately satisfactory based on data monitored under the program. Poverty GP The two PPARs completed by IEG in FY17 under the Poverty GP are summarized below. The Poverty Reduction Fund project was approved in 2002 to respond to one of the top priorities of the government of the Lao People’s Democratic Republic—reducing poverty in rural and remote areas of the country. The government considered lack of infrastructure to be the root cause of poverty and in response, the project sought to provide rural infrastructure and better access to basic services through a $34 million loan. The project adopted a community based approach, which was aligned with the vision of participation and decentralization expressed in the country strategy, although the approach turned out to be largely incoherent given the highly centralized nature of government policies. The project supported more than 3,000 subprojects in seven provinces, supporting the construction, for example, of schools, bridges, health dispensaries, rural roads, clean water systems, and irrigation schemes. A large share of first phase subprojects did not go to the poorest districts, and it took some time to adjust targeting. The quality of the infrastructure developed varied, and about one-fourth of subprojects did not meet technical specifications at closing. Rates of use were high, but the impact on poverty and well-being of rural communities could not be determined. Overall, however, higher enrollment rates, higher access to a minimum package of health services, and better availability of clear water were documented in participating villages benefiting from investment. By contrast, only half of rural roads constructed could be used year around, because of the challenges of maintaining them. The project had very modest results in generating local capacity, as communities were not given full responsibility for selecting, implementing, supervising, and assessing subprojects. The integration of the poor was limited as was that of non-Lao speaking groups. In addition, despite attempts to increase women’s participation, decision making at the 394 Results and Performance of the World Bank Group 2017 | Appendix M community level remained largely male dominated. Overall, the World Bank’s ICR rated development outcome satisfactory. Based on the shortcomings outlined above, IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately unsatisfactory. As Ukraine began its policy reforms in transition toward a market economy after gaining independence in 1991 from the Soviet Union, the country had a statistical system designed for central planning that relied on full coverage of data collected through administrative mechanisms. Over the years, the government of Ukraine progressively transformed its statistical practices, and in 2003, the government approached the World Bank to request assistance for financing comprehensive modernization of the national statistical system. Under the Statistics Capacity Building Program, the World Bank provided $42 million to support the Development of State Statistics System for Monitoring Social and Economic Transformation. The project focused on organizational changes among data producers and the upgrade of data quality, the data dissemination framework, and the ef ficiency of data production. Project implementation benefi ed from several technical assistance activities bundled into one package, and this approach saved time and money for the participating agencies. The project made clear the need for long-term engagement in statistical capacity building to ensure gains in one area are not offset by losses in another. Under the project, progress was made on ef ficiency, accuracy, and timeliness of data, with statistical practices largely harmonized with international standards. On the other hand, operational autonomy of the central statistical agency—the State Statistics Service of Ukraine—was compromised by government measures in 2013. The project also highlighted the need for a governing body for the national statistical system. One of the key recommendations of Statistics Capacity Building Program, and part of its template, is to establish a governing body responsible for setting strategic direction for the entire national statistical system. In Ukraine, apart from setting strategic direction, the National Statistics Council would have been expected to ensure the operational autonomy and professional independence of the State Statistics Service of Ukraine. Strong safeguards and clear rules for resource allocation and adequate funding for statistics are essential and these were not ensured in this case. The World Bank’s ICR assessed the operation’s outcome as satisfactory, whereas IEG’s evaluation of the project’s performance rated the outcome of the operation as moderately satisfactory, reflecting IEG’s observations on the limited extent to which operational autonomy and professional independence were realized. Independent Evaluation Group | World Bank Group 395 Human Development Practice Group For perspective, the following introductory paragraphs provide an indication of the relative size, in number and volume, of the three GP groups: the SD, HD, and EFI.1 With 158 projects, the HD Practice Group holds about 22 percent of the total 713 projects rated World Bank–wide during FY14–16. Under the SD Practice Group there were 412 projects rated, and a further 141 were rated under the EFI Practice Group. By volume, the HD Practice Group is the smallest of the three—the 158 rated projects were supported by $15,035 million. The SD Practice Group is the largest ($35,609 million), followed by the EFI Practice Group ($18,748 million). The largest single GP within the HD group by number of projects rated is Education (65 projects), followed by Health, Nutrition, and Population (HNP; 57 projects), and Social Protection and Labor (SPL; 36 projects). SPL is the largest by commitment volume of rated projects during FY14–16 ($7,342 million), followed by HNP ($4,681 million), and Education ($3,013 million). Under the HD group, most of the projects rated during FY14–16 are IPF (151 projects), supported by $13,950 million. Seven operations are DPF, supported by $1,085 million. Portfolio Performance The HD cluster comprises three GPs: ƒƒ Education: supporting countries that request financing or technical assistance to reach Sustainable Development Goal 4, which calls for access to quality education and lifelong learning opportunities for all by 2030. ƒƒ HNP: promoting universal health coverage and providing fi ancing, advanced analysis, and policy advice to help countries expand access to quality, affordable health care. It also sets priorities for protecting people from falling into poverty or becoming poorer because of illness, and promotes investments in all sectors that form the foundation of healthy societies. ƒƒ SPL: supporting well-functioning social safety nets proven to reduce poverty and in- equality, promoting access to health and education among poor children, empowering women, and supporting sustainable social insurance programs that help cushion the impact of crises on households. It supports promoting effective policies for productive employment that help people access labor markets and accumulate skills in normal times and during recovery from economic crisis. For the current review period, FY14–16, 77 percent of all 158 projects with outcome ratings under the HD Practice Group were rated MS+, which is above the average outcome rating (72 percent MS+) for all projects in non-HD groups over the same period, and the World Bank–wide average of 396 Results and Performance of the World Bank Group 2017 | Appendix M 73 percent MS+.2,3 At a satisfaction rating of 88 percent MS+, projects in the HD Practice Group also outperform the average performance for all non-HD projects (83 percent MS+), and the World Bank–wide average (84 percent MS+) when rated by commitment (figure M.12). All three GPs within the group performed near or above the World Bank–wide average during FY14–16. The SPL GP achieved a 92 percent MS+ rating (36 projects rated), and a 100 percent MS+ outcome rating when measured by commitment (figure M.13). The percent MS+ outcome rating for the HNP and Education GPs was 74 percent and 72 percent, respectively, by number of projects, and 76 percent and 78 percent, respectively, by commitment. Figure M.12. HD Group: Project Outcome Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: HD = Human Development; MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 397 Figure M.13. HD Group: Project Outcome Ratings by Global Practice, FY14–16 Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. HD = Human Development; MS+ = a rating of moderately satisfactory or better. The overall outcome rating for IBRD-supported projects (n = 50) in the HD Practice Group FY14–16 is 80 percent MS+, and the rating for IDA-supported projects (n = 83) is 75 percent MS+. This compares with respective outcome ratings of 74 percent MS+, and 73 percent MS+ for non-HD projects rated over the same period (figure M.14). During FY14–16, 77 percent of 150 IPF projects in the HD Practice Group were rated MS+ for outcome, which compares with a 72 percent MS+ average rating for IPFs across the other two GP groups in the same period. At 86 percent MS+, outcome ratings for the seven DPF projects rated also exceeds the non-HD group ratings (74 percent MS+; figure M.15). World Bank performance, a composite of ratings for quality at entry and project supervision, was 76 percent MS+ for all projects in the HD Practice Group compared with 71 percent MS+ for non-HD groups, and World Bank–wide average of 72 percent MS+. By commitment, World Bank performance for projects in the HD Practice Group (87 percent MS+) is also greater than the World Bank–wide average rating (84 percent MS+), and the average rating for non-HD groups (83 percent MS+; figure M.16). Under the HD Practice Group, World Bank performance for IBRD-supported projects rated during FY14–16 (82 percent MS+) outperforms IDA-supported projects rated over the same period (73 percent MS+). World Bank performance is relatively better than performance under non-HD groups when measured by IBRD agreement type (82 percent MS+ versus 74 percent MS+), and IDA agreement type (73 percent MS+ versus 72 percent MS+). World Bank performance for the individual GPs within the group varied. World Bank performance under SPL was 92 percent MS+ (by number), and 100 percent MS+ by commitment, mirroring 398 Results and Performance of the World Bank Group 2017 | Appendix M Group: Project Outcome Ratings by Practice Group and Figure M.14. HD Financing Source, FY05–16 a. IBRD versus IDA: Percentage rated MS+, by number b. IBRD projects—HD versus other practice groups: Percentage rated MS+, by number c. IDA projects—HD versus other practice groups: Percentage rated MS+, by number Source: Independent Evaluation Group data. Note: HD = Human Development; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 399 Group: Project Outcome Ratings by Practice Group and Figure M.15. HD Lending Instrument, FY05–16 a. IPF versus DPF: Percentage rated MS+, by number b. IPF projects: HD versus other practice groups: Percentage rated MS+, by number c. DPF projects: HD versus other practice groups: Percentage rated MS+, by number Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. DPF = development policy financing; HD = Human Development; IPF = investment project financing; MS+ = a rating of moderately satisfactory or better. 400 Results and Performance of the World Bank Group 2017 | Appendix M Figure M.16. HD Group: World Bank Performance Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: Thirteen projects were excluded when measured by volume due to lack of volume data. HD = Human Development; MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 401 Figure M.17. HD Group: World Bank Performance Ratings by Global Practice, FY14–16 Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. HD = Human Development; MS+ = a rating of moderately satisfactory or better. overall outcome for the GP. World Bank performance (by number) for the HNP and Education GPs was 74 percent and 69 percent MS+, respectively, with similar levels of World Bank performance by commitment (figure M.17). World Bank performance for DPF is 86 percent MS+, compared with 75 percent MS+ for IPF within the HD group. World Bank performance under the HD Practice Group is relatively better than performance under non-HD groups for both DPF (83 percent MS+), and IPF (70 percent MS+). The quality of M&E for HD projects is 44 percent S+, which exceeds the World Bank–wide average (34 percent S+) and the average for non-HD groups (32 percent S+). At a rating of 62 percent S+, the quality of M&E for HD projects by commitment, is also greater than the overall World Bank average (46 percent S+) and the rating for non-HD groups (41 percent S+; figure M.18). M&E quality for the individual GPs within the practice group varied for projects closed in FY14–16, with 61 percent of SPL, 40 percent of HNP, and 37 percent of Education projects rated S+. By volume, 93 percent of SPL, 31 percent of HNP, and 37 percent of Education commitments went to projects rated substantial or higher (figure M.19). 402 Results and Performance of the World Bank Group 2017 | Appendix M Figure M.18. HD Group: M&E Quality Ratings, FY05–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: Nine projects were excluded when measured by volume due to lack of volume data. HD = Human Development; M&E = monitoring and evaluation; S+ = a rating of substantial or better. Independent Evaluation Group | World Bank Group 403 During FY14–16, the better overall M&E ratings for projects under the HD Practice Group holds for both IPF and DPF projects, and for IBRD- and IDA-supported projects when compared with respective non-HD groups. At regional level, project outcomes under HD in the Africa Region are strong, with an overall outcome rating of 85 percent MS+. Satisfaction ratings for the East Asia and Pacific (East Asia and Pacific) Region are 79 percent MS+. By comparison, respective satisfaction ratings for the Europe and Central Asia (Europe and Central Asia) and Middle East and North Africa (Middle East and North Africa) Regions are 65 percent MS+ and 69 percent MS+ (figure M.20, panel a). In addition to strong performance by the East Asia and Pacific and Africa Regions, Latin America and the Caribbean also performs well when measured by commitment volume (figure M.20, panel b). Figure M.19. HD Group: M&E Quality Ratings by Global Practice, FY14–16 Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. HD = Human Development; M&E = monitoring and evaluation; S+ = a rating of substantial or better. 404 Results and Performance of the World Bank Group 2017 | Appendix M Group: Project Outcome Ratings for Practice Group by Figure M.20. HD Region, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. HD = Human Development; MS+ = a rating of moderately satisfactory or better. World Bank performance for HD projects is even across all Regions when measured by number, with each region scoring close to the overall 76 percent MS+ rating for the overall practice group (figure M.21). Differences are more marked when measured by commitment volume. M&E ratings for HD projects measured by number of projects range from 20 percent S+ for the Europe and Central Asia Region to 67 percent S+ for the East Asia and Pacific Region. As above, the overall rating for M&E quality under the HD Practice Group is 44 percent S+ (figure M.22). Independent Evaluation Group | World Bank Group 405 Group: World Bank Performance Ratings for Practice Group Figure M.21. HD by Region, FY14–16 Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. “Other” includes projects that are not mapped to any Practice group. HD = Human Development; MS+ = a rating of moderately satisfactory or better. 406 Results and Performance of the World Bank Group 2017 | Appendix M Figure M.22. HD Group: M&E Quality Ratings for Practice Group by Region, FY14–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: Two projects were excluded when measured by volume due to lack of volume data. HD = Human Development; M&E = monitoring and evaluation; S+ = a rating of substantial or better. Independent Evaluation Group | World Bank Group 407 Evaluation Findings and Lessons This section summarizes, as relevant, key findings and lessons from IEG’s major evaluations, learning products, and project-level field-based assessments (PPARs) completed in FY17 for GPs under the HD Practice Group. Under the Education GP, IEG completed a major evaluation on higher education, and three project- level field-based assessment. Under the HNP GP, IEG completed a summary learning product on public-private partnerships (PPPs) in health, as well as two PPARs. Finally, under the SPL GP, IEG completed three PPARs for projects during FY17. Education GP Higher Education for Development is a major evaluation that evaluates the World Bank Group’s global support for higher education since 2003 (IEG, 2017). While the World Bank Group has long supported higher education, the sector represents a relatively small proportion of the World Bank Group’s overall financing portfolio. The evaluation looked at 54 World Bank operations that focused exclusively on higher education as well as 85 operations that included higher education as a part of larger interventions. The World Bank has provided support in a wide range of countries—in all regions and all income groups, ranging from low-income to high-income countries. The International Finance Corporation (IFC) provided support through 42 operations, mostly in middle-income countries, particularly in Latin America. The World Bank Group’s support is growing and a wide range of countries have expressed interest in future support. The evaluation found that the World Bank supports both improved teaching and research (to increase employability and competitiveness), and that improved governance has been an emerging theme, but a few operations directly focus on equity. Indeed, many World Bank projects implicitly support elite public universities. World Bank projects usually support efforts to increase the autonomy of universities and many projects finance grant programs to allow institutions to finance their own priorities. IFC has a narrow focus on improved employability, in institutions that cater to middle-class and lower-middle-class students. IFC support has been focused almost entirely on middle-income countries. While the World Bank typically has clear objectives, few operations have results frameworks that can measure its contributions in reaching these objectives. This makes it dif ficult to assess the effectiveness of World Bank support. The evaluation also found that the World Bank does not have a strong evidence base to support its interventions, and that there are few rigorous evaluations. This lack of evidence is compounded by the fact that most operations support block grants to universities that often put the World Bank at an arm’s length from the interventions themselves. The evaluation also noted that the World Bank primarily collaborates with traditional donor agencies and development World Banks. However, it does less in collaborating with other, of ficial donors providing support for higher education, including Ministries of Education and Science Foundations in Organization for Economic Co-operation and Development countries. However, IEG notes ongoing efforts to improve both the quality of result frameworks and to broaden collaboration. 408 Results and Performance of the World Bank Group 2017 | Appendix M At the strategic level, IEG recommends that the World Bank Group carry out a strategic review of its support for higher education. The goal of this review is to ensure the World Bank Group’s support is aligned with its strategy and the twin goals. The evaluation also recommends that the World Bank Group increase the use of data and generate evidence to support future interventions. This is aligned with one of the key principles of the World Bank Group’s strategy. Given the importance of increasing employability, which often figures prominently in project objectives, the World Bank Group could work to better incorporate the views of employers in its support. With the large number of GPs supporting higher education within the World Bank Group and the different types of donors, the evaluation recommends greater effort to improve both internal and external coordination. The three PPARs completed by IEG in FY17 for projects under the Education GP, are summarized below. The Pernambuco Education Results and Accountability Project (2009–15) set out to address challenges identified by the research undertaken by the Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, that is, student scores for fundamental and secondary education in Pernambuco were among the lowest in Brazil. On April 14, 2009, an IBRD loan of $154 million was approved for the State Government of Pernambuco (with a guarantee by the Government of Brazil). The operation financed a portion of the government’s broader program through a sectorwide approach of eligible expenditure programs, such as a school standards program, the State-Level Student Assessment System, literacy, and an overage correction program. The operation also disbursed based on performance conditions. The theory of change for this project needs to be viewed in relation to the broader government program (not just the items financed by the World Bank), as the government simultaneously implemented complementary activities that also supported attainment of the objectives. As such, attribution of results for this operation goes beyond World Bank financing, to the broader government program. There has been continuity in education policies and programs in the state government, despite there being multiple governors. This continuity helped stabilize the performance monitoring system so that data served as a feedback loop to the system. Equity was a stated objective, but it was not adequately addressed, as a performance gap remained between rural and urban schools. Overall, both the World Bank’s ICR and IEG’s independent field-based evaluation assessed development outcome to be satisfactory, based on state and federal data that demonstrated significant improvement in education quality and in student flow (reduced rates of repetition and dropout). In 2002, the Kyrgyz Ministry of Education approached the World Bank seeking assistance to address poor education quality, particularly in rural areas. In response, the World Bank developed and approved the $15 million Rural Education Project (2005–11), the World Bank’s first entry into the sector in the Kyrgyz Republic. The project development objective sought to improve learning and learning conditions in primary and secondary schools, with priority attention to rural areas. Some of the six project components had a national focus (for example, the development and provision of Independent Evaluation Group | World Bank Group 409 textbooks and national level assessment), and others (for example, the development of a formative assessment approach) focused on two, largely rural, pilot oblasts (regions), namely, Issyk-Kul and Talas. Despite the relevance of the objectives and evident client demand IEG found design to be modest, referencing a lack of prudent selectivity and unrealistic scope given known capacity limitations within the system. The objective to improve the quality of education, the core outcome aspired to, was only modestly realized despite the efforts of the government, the World Bank and other donors. The objective to improve the conditions to support learning was substantially achieved. The Rural Education Project contributed to building the momentum for change, not least through the low- cost medium of funding two Program for International Student Assessment assessments, but also through piloting initiatives that are being used to modernize the sector. That said, other, relatively expensive interventions (for example, provision of textbooks) had little ongoing, systematic effect, and IEG noted a lack of coherence between interventions supported by the World Bank, a late comer to the sector, and other development partners already working in the Education sector in the Kyrgyz Republic. IEG emphasizes that a selective, and more realistic approach to design is likely to create a more closely articulated link (attribution) between World Bank interventions and outcomes. In this case, expert and thorough diagnosis of need and associated challenges was not accompanied by realistic assessment of possibility, taking constraints into account such as the political economy, commitment, capacity, resources (human and financial), and available time. The World Bank’s self-evaluation rated development outcome satisfactory. IEG’s review rated development outcome moderately unsatisfactory, given design faults and limited evidence of improvements in the quality of education. The World Bank has provided support to the basic education sector in Lao People’s Democratic Republic since 1993, when the country started its transition to a market economy. After the first project in the sector closed in 2000, the government and the World Bank embarked on developing the Second Education Development Project reflecting the fact that low levels of education continued to act as a major constraint to further development—enrollment in primary education was nearly universal at grade one, but many students did not graduate, and only about one-third of the workforce has secondary education. The project objective broadly sought to achieve universal completion of primary education, and was supported by three components designed to increase access to and completion of primary education in the poorest districts; improve the quality of primary education; and strengthen capacities for policy analysis and management. IEG found that all three components were well designed and supported, and made a clear contribution to the project’s objective. The project became effective in January 2005 and a midterm review was carried out in 2007. Because the project was assessed to be performing well, the World Bank and the government agreed to additional financing to cover more remote communities, extending the planned life of the project from 2010 to 2013 and providing additional financing of $15.5 million, supplementing the original project budget of $13 million, and a trust fund supported by 410 Results and Performance of the World Bank Group 2017 | Appendix M the government of Australia to finance additional infrastructure ($2.9 million) that had been added in 2008 and 2009. IEG also found that the project made an important contribution to improving both net enrollment, and the completion rate at primary level, primarily by increasing coverage in poor and remote areas. Increased participation is directly associated with the increase in the number of classrooms in the provinces that the project financed. IEG found likely moderate increases in competency in the Lao language from 2006 to 2009, although the level of mathematics competence decreased from a low baseline. IEG draws lessons from the project assessment, including the following: direct technical support from World Bank staff can make an important contribution to implementation and outcomes; and sustainable student assessment systems take time to develop and to contribute to the education system. In that latter regard, IEG notes that much of the benefits of assessment set up under the project are being fully demonstrated a decade after the project was approved. As such, IEG rated overall development outcome satisfactory, an improvement on the World Bank’s ICR rating of moderately satisfactory. Health, Nutrition, and Population GP IEG’s summary learning product, Public-Private Partnerships in Health, which covered FY04 to FY15, aimed to provide insights into the World Bank Group’s efforts in applying PPP arrangements in the health sector. The report builds on previous IEG work on PPPs and on private sector involvement in health, notably World Bank Group Support to Public-Private Partnerships: Lessons from Experience in Client Countries, FY02–12 (2013) and Improving Effectiveness and Outcomes for the Poor in Health, Nutrition, and Population: An Evaluation of World Bank Group Support since 1997 (2009). The report found that the World Bank Group approved 78 projects that provided support for health- related PPP operations. Capacity building was an increasingly prominent focus of the World Bank’s upstream support—a much-needed effort to strengthen countries’ ability to assess the merits of deploying PPPs in the health sector. IFC Advisory Services focused on providing structuring advice and finance. The World Bank Group supports countries that need its support the most, that is, countries where the enabling environment for PPPs is weak. However, adequate sequencing of World Bank Group interventions is rare. Moreover, World Bank Group advice has not been strategic enough with respect to laying out all options for the provision of health services in client countries, and there is little evidence that fiscal implications are assessed consistently. Self-assessment reports (prepared two years after completion) indicate positive effects of PPPs in some areas, for example, access and quality; but evidence is still limited. A clear majority of World Bank Group–supported PPP interventions have an explicit emphasis on the poor or underserved population in their design. However, suitable M&E indicators or proper baselines and targets are lacking. Despite the need for support in postcontract award management, the World Bank Group rarely provides “aftercare” for contract management. The following lessons are identified: (i) when advising governments on the various models for providing health services, both the World Bank and IFC should act as One World Bank Group and discuss the whole range of options from the public and mixed options to the various types of potential health PPPs, taking account of the sector reform context of a country and its overall maturity and track record in using PPPs; (ii) to be better positioned to deliver such Independent Evaluation Group | World Bank Group 411 strategic advice, the World Bank Group should improve the integration of its sector reform and policy work on the structuring and financing of PPPs; (iii) pro-poor access and affordability need not only be systematically considered at the design stage, they should also be tracked to ensure that the poor actually benefit from PPPs; (iv) the PPP Fiscal Risk Assessment Model tool should be systematically applied to the structuring of PPPs with substantial fiscal implications; and (v) given the long contractual life of PPPs, the new practice of preparing postcompletion reports after the PPPs have gone into operation should be mainstreamed. Two PPARs were completed for projects under the HNP GP. The Health Equity and Performance Improvement Project (HEPI—2008–14) sought to address the relatively poor (compared with the rest of the population) access to health services, and poor health outcomes among rural and indigenous communities in Panama. For example, the national average maternal mortality rate in 2007 was estimated at 59.4 deaths per 100,000 live births, but indigenous and rural communities reported rates that were between 6 to 8 times higher. In 2009, the infant mortality rate in the indigenous communities was almost twice the national average of 12.2 deaths per 1,000 live births. To expand basic health and nutrition services, the Government of Panama introduced the Coverage Extension Strategy, based on contracting-out to nongovernmental organizations for the provision of a Comprehensive Basic Package of Health Services. The HEPI project financed the Coverage Extension Strategy to cover about 180,000 people living in rural communities, and the provision of technical assistance to strengthen the entire primary health services networks of the rural communities and improve the performance of the overall national health system. The HEPI project was supported by $40 million of IBRD financing, with government cofinancing of $18.73 million. The objective to increase access in selected underserved rural communities to quality basic health services known to improve mother and child health was substantially achieved: the mobile health teams achieved all targets related to population coverage, and to the provision of health services. However, improvement in the capacity of the primary health care networks was more modest, as were achievement related to the development of strategic planning, regulatory, and monitoring mechanisms known to improve health system performance. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory. At the start of the new millennium, malnutrition in Senegal was of great concern. Among children under five years of age, almost one-third (30 percent) were stunted (low height for age), 10 percent were wasted (low weight for height), and 20 percent were underweight (weight for age). IEG’s PPAR, A Decade of World Bank Support to Senegal’s Nutrition Program (2002–12), involved a field-based assessment of three World Bank projects in support of Senegal’s Nutrition program through three projects: (1) the Nutrition Enhancement Program, (2) the Nutrition Enhancement Project in Support of the Second Phase of the Nutrition Enhancement Program, and (3) the Rapid Response Child- Focused Social Cash Transfer and Nutrition Security Project. All three projects performed well in their intervention areas, culminating in greater knowledge, healthier behaviors and practices, and improved 412 Results and Performance of the World Bank Group 2017 | Appendix M nutrition outcomes. But, even though these intervention areas extended to all of Senegal’s 14 regions and 45 departments, they did not achieve suf ficient coverage of at-risk communities (within regions and departments) to effect major improvements in nutrition outcomes at the national level that were initially anticipated. The World Bank’s strategic and technical contributions to the Nutrition Enhancement Program were important, but the retraction of the World Bank’s financial commitment to provide substantial financial and technical support to three phases of the program was a factor in the failure to achieve a level of coverage that would have had a significant effect on nationwide trends. The financial constraints not only limited the number of intervention areas that the Nutrition Enhancement Program could support, they also affected the frequency and intensity of interventions. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that outcome ratings were highly satisfactory for two operations and satisfactory for the other one, based on increased capacity to develop, monitor, and oversee multisectoral nutrition activities. Social Protection and Labor GP The three PPARs completed by IEG in FY17 under the SPL GP are summarized below. Panama achieved high economic growth 2000–16 that contributed to significant poverty reduction. During this period, Panama expanded its social assistance programs and in 2006, it introduced the conditional cash transfer Red de Oportunidades (Opportunity Network) with the objective of inserting poor households in the dynamics of national development, incentivizing the use of basic health and education services. The World Bank sought to support the Red de Oportunidades through the Social Protection Project (2007–14). The project was supported by $24 million of IBRD financing, and $22.8 million cofinancing from the Inter-American Development World Bank, and the Government of Panama. The project aimed to improve the management and operation of Red de Oportunidades, increase beneficiaries’ demand for education and maternal and infant health services, expand the provision of health and nutrition services in the indigenous communities covered by the Red de Oportunidades, and enhance the government’s capacity to design, implement, and monitor coherent and ef ficient social sector policies and interventions. IEG found that the project improved the management and operation of the Red de Oportunidades, and strengthened the provision of health and nutrition services. However, it was not possible to determine if project support and targeted communication (to improve knowledge among beneficiaries about their rights, how to access services, and their co-responsibilities) improved demand for health and education services. Finally, the improvement of government’s capacity to design, implement, and monitor coherent and ef ficient social sector policies and interventions was rated modest, as the capacity of the Technical Secretariat of the Social Cabinet to coordinate social policies across ministries remained weak throughout the life of the project. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory. In 2008, when the Malawi Third Social Action Fund Adaptable Program Loan II project was approved, Malawi was one of the poorest countries in the world with a per capita GDP of $302. Independent Evaluation Group | World Bank Group 413 Over half the population lived below the poverty line, with 22 percent below the food poverty line. Human development indicators and access to services were dismal. The economy was trapped in a vicious cycle of emergency appeals to address economic and climatic shocks, to which the international community responded through humanitarian assistance rather than through support to build sustainable economic development. The Malawi Third Social Action Fund series offered an approach to alleviate poverty through a decentralization agenda to ensure that citizens would engage in planning local development initiatives. The Adaptable Program Loan II aimed to deepen the community empowerment processes (participatory planning and implementation) at local level that had begun under Adaptable Program Loan I. The operation aimed to finance small labor-intensive community-level public works schemes; support voluntary community savings-investment groups; build capacity of local authorities to manage grant moneys; and support community participation in district planning and implementation of community subprojects. World Bank performance was moderately satisfactory given some shortcomings in expectations of the decentralization process and in risk analysis. The main messages from IEG’s assessment are that (i) community-driven development (CDD) that emphasizes longer-term development may not always be the best instrument for addressing shocks—it can facilitate decentralization, but requires the right balance between community priority setting and local development planning; (ii) beneficiary feedback throughout provides important evidence and perspective; and (iii) linking income generation interventions with asset building opportunities can help build economic resilience. The World Bank’s ICR rated overall development outcome satisfactory. IEG’s field-based assessment resulted in a development outcome rating of moderately satisfactory, reflecting substantial achievement in improved livelihoods and capacity building, and modest effi iency. Faced with very high rates of extreme poverty despite sustained growth, in 2000 the government of Tanzania set its policies to explicitly address income poverty and human capital deficiencies. Community mobilization, empowerment, decentralization, accountability, and improved governance became key policy themes and supported first by a pilot Tanzania Social Action Fund, a CDD approach. The Second Tanzania Social Action Fund, the subject of IEG’s PPAR, scaled up the program to the national level. A $215 million IDA credit financed infrastructure subprojects and public works to generate household-level earnings opportunities. It also introduced elements of a productive safety net that targeted the poorest in the community, including a voluntary group savings/investment mechanism, and grants for income earning opportunities. The achievements of the projects were modest, especially in empowering communities to request, implement, and monitor subprojects. The involvement in and ownership of the project were lower than expected and not much was known about further outcomes. The project invested heavily in health and education facilities, water and sanitation, and economic infrastructure such as markets. This resulted in better student/classroom ratios, more use of preventive care and better access to water and sanitation; however, it was unclear to what extent these investments impacted poverty and household well-being. The project did succeed in improving household livelihoods through public works and cash transfers. More than 20,000 savings- 414 Results and Performance of the World Bank Group 2017 | Appendix M investment groups were created—with overwhelmingly female participation. Public works increased the living standards of beneficiary households, including their chance of taking three meals per day. The income generating activities allowed households to increase their livestock and asset holdings substantially, and to generate earnings from them. The conditional cash transfer program increased enrollment in the health insurance scheme and health status of the beneficiaries. It also increased primary school completion. Because the benefits of the projects were obvious only in relation of one of its components and only in the second phase of the project (after restructuring) IEG rated the project moderately unsatisfactory, diverging from the World Bank’s ICR satisfactory assessment of development outcome. Sustainable Development Practice Group For perspective, the following introductory paragraphs provide an indication of the relative size, in number and volume, of the three GP groups: SD,HD, and EFI.1 With 412 projects, the SD Practice Group of GPs holds more than half of the total 713 projects rated across all practice groups during FY14–16. Under the HD Practice Group, 158 projects were rated, and a further 141 were rated under the EFI Practice Group. By volume, the SD Practice Group is also the largest—the 412 rated projects were supported by $35,609 million. The EFI Practice Group was the next largest by volume ($18,748 million), and projects rated during FY14–16 under HD were supported with $15,035 million. The largest single GP by number of rated projects is the SURR GP (93 projects), which sits within the SD Practice Group. The smallest GP by number of projects (6) is Poverty and Equity, which sits under the EFI Practice Group. That GP is also the smallest by commitment volume ($670 million), and the Transport and ICT GP, under the SD Practice Group, is the largest GP by commitment volume of rated projects ($11,157 million). The SD Practice Group holds the largest number of IPF: 399 of a World Bank–wide total of 635 IPF projects, including the greatest commitment volume for IPFs rated during FY14–16 ($33, 412 million). The largest number of DPF is found under the EFI Practice Group (52 of 69 operations), also representing the greatest commitment volume for DPFs rated during FY14–16 ($12,785 million). Portfolio Performance The SD Practice Group consists of six GPs and two cross-cutting solutions areas. The six GPs are as follows: ƒƒ Agriculture: aims to link farmers to markets to increase food availability and stimulate general economic growth using a value chain approach, including on-farm inputs, land water, financial services, and postharvest agro-processing. ƒƒ Energy and Extractives: sets priorities for improving the energy investment climate by promoting sector reform and governance, strengthening utilities, enhancing investment Independent Evaluation Group | World Bank Group 415 frameworks, encouraging private participation, and rationalizing subsidies. ƒƒ Environment and Natural Resources: seeks to promote a green, clean, and resilient world in which natural resources are managed to support livelihoods and strong econ- omies, and to share prosperity by transferring wealth from downstream beneficiaries of ecosystem services to upstream communities that carry the opportunity costs of protecting nature. ƒƒ SURR: gives priority to ensuring that marginalized and vulnerable populations have a voice in defining the growth of cities, human settlements, and rural areas, which have local and global implications for sustainability and climate change. ƒƒ Transport and Information and Communication Technologies: seeks to promote virtual and physical connectivity, to improve the movement of people, goods, and information, thus enabling access to economic opportunities, to vital services like healthcare and education, and to knowledge and ideas. ƒƒ Water: aims to ensure that water is a reliable foundation for poverty reduction and shared prosperity by delivering public water goods coupled with private initiatives to add value to water services throughout the water cycle. The two cross-cutting solutions areas are as follows: ƒƒ Climate Change: aims to help country clients accelerate efforts to tackle climate change and deliver on their national climate plans. ƒƒ PPP: provides strategic, leadership, and coordination support for the provision of basic services. PPPs introduce private sector technology, innovation, and funding to public services. For the current review period, FY14–16, 74 percent of all 412 projects closed under the SD Practice Group were rated moderately satisfactory or better (MS+), which is above the average outcome rating (72 percent MS+) for all projects in non-SD groups over the same period, and the World Bank–wide 73 percent MS+ average (figure M.23, panel a).2,3 At a satisfaction rating of 82 percent MS+, average outcome ratings for projects in the SD Practice Group are slightly less that all non-SD projects (87 percent MS+), and the World Bank–wide average (84 percent MS+), when rated by commitment for the FY14–16 period (figure M.23, panel b).There is variation in the performance level of individual GPs within the practice group when compared with the SD group overall rating for development outcome (74 percent MS+). The best-performing GP in FY14–16 was Transport and ICT (82 percent MS+), followed by the Agriculture and SURR GPs (both at 75 percent MS+)—noting the SURR GP had a larger number of rated projects (93 projects) than any other individual GP in the practice group. The poorest performance, 60 percent MS+ for 416 Results and Performance of the World Bank Group 2017 | Appendix M development outcome, was for the Environment and Natural Resources (ENR) GP. Development outcome rating for Water GP projects in FY14–16 was 68 percent MS+ (figure M.24). There was also variation in performance levels by GP when measured by commitment volume levels, ranging from the Agriculture (90 percent MS+) and SURR GPs (87 percent MS+), to the ENR (67 percent MS+) and Water GPs (70 percent MS+; figure M.24). Performance by commitment for the Transport and ICT GP, the largest single GP in commitment levels ($11,157 million), was 82 percent MS+. Figure M.23. SD Group: Project Outcome Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: Thirteen projects were excluded when measured by volume due to lack of volume data. MS+ = a rating of moderately satisfactory or better; SD = Sustainable Development. Independent Evaluation Group | World Bank Group 417 Figure M.24. SD Group: Project Outcome Ratings by Global Practice, FY14–16 Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. MS+ = a rating of moderately satisfactory or better; SD = Sustainable Development. Within the SD group, IDA-supported projects (77 percent MS+) outperformed IBRD-supported projects (72 percent MS+) on rating for project outcome (Figure 25, panel a). The outcome rating for the 151 IBRD supported projects under the SD Practice Group FY14–16 (72 percent MS+) dipped below respective ratings for non-SD IBRD-supported projects over the period (81 percent MS+; figure M.25, panel a). For IDA-supported projects, the outcome rating of the SD Practice Group was greater than that of the non-SD groups (77 compared with 68 percent MS+; figure M.25, panels b and c). During the three-year period FY14–16, 74 percent of 399 IPF in the SD Practice Group was rated MS+ for outcome, which compares with a 70 percent MS+ rating for the 10 DPF operations in SD rated over the period (figure M.26, panel a). For IPF, the SD group has a higher share of MS+ projects (74 percent) than the non-SD GP groups during the same period (71 percent MS+; figure M.26, panel b). For DPF projects, the SD group received lower outcome ratings than non-SD groups (70 percent MS+ compared with 76 percent MS+; figure M.26, panel c). World Bank performance, a composite of ratings for quality at entry and project supervision, was 73 percent MS+ for all projects in the SD Practice Group compared with 72 percent MS+ for non-SD groups, and a World Bank–wide average 72 percent MS+. By commitment, World Bank performance for projects in the SD Practice Group (81 percent MS+) is below the World Bank–wide average rating (84 percent MS+) and the average rating for non-SD groups (87 percent MS+; figure M.27). 418 Results and Performance of the World Bank Group 2017 | Appendix M Group: Project Outcome Ratings by Practice Group and Figure M.25. SD Financing Source, FY05–16 a. IBRD versus IDA: Percentage rated MS+, by number b. IBRD projects—SD versus other practice groups: Percentage rated MS+, by number c. IDA projects—SD versus other practice groups: Percentage rated MS+, by number Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. SD = Sustainable Development; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; MS+ = a rating of moderately satisfactory or better. Independent Evaluation Group | World Bank Group 419 Group: Project Outcome Ratings by Practice Group and Figure M.26. SD Lending Instrument, FY05–16 a. IPF versus DPF: Percentage rated MS+, by number b. IPF projects: SD versus other practice groups: Percentage rated MS+, by number c. DPF projects: SD versus other practice groups: Percentage rated MS+, by number Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. DPF = development policy financing; SD = Sustainable Development; IPF = investment project financing; MS+ = a rating of moderately satisfactory or better. 420 Results and Performance of the World Bank Group 2017 | Appendix M Figure M.27. SD Group: World Bank Performance Ratings, FY05–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: Thirteen projects were excluded when measured by volume due to lack of volume data. MS+ = a rating of moderately satisfactory or better; SD = Sustainable Development. World Bank performance for the individual GPs within the practice group ranges from 58 percent MS+ for ENR, to 80 percent MS+ for Transport and ICT (figure M.28). By commitment, World Bank performance for ENR is 65 percent MS+ compared with 81 percent MS+ for Transport and ICT scores, and 89 percent MS+ for Agriculture, the highest within the SD Practice Group. World Bank performance for IBRD supported projects under the SD Practice Group is relatively lower than IBRD performance under non-SD groups. Among IDA-supported projects, the SD group outperforms non-SD GPs. However, IDA performs better than IBRD within the SD Practice Group itself (76 percent MS+ versus 71 percent MS+). Independent Evaluation Group | World Bank Group 421 Figure M.28. SD Group: World Bank Performance Ratings by Global Practice, FY14–16 Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. MS+ = a rating of moderately satisfactory or better; SD = Sustainable Development. World Bank performance is also better for IPF within the practice group (73 percent MS+ of 399 projects closed in FY14–16 with ratings) when compared with DPF where World Bank performance was rated 70 percent MS+ (10 DPF operations FY14–16).4 The quality of M&E for SD projects is 30 percent S+, below the 34 percent World Bank–wide average rating, and further below the 39 percent S+ rating for non-SD groups. At a rating of 38 percent S+, the quality of M&E for SD projects by commitment is also below the World Bank–wide rating of 46 percent S+, and the rating for non-SD groups (53 percent S+; figure M.29). M&E quality for the individual GPs within the practice group is spread from a high of 53 percent S+ for the 53 projects rated under the Energy & Extractives GP, to 19 percent S+ for the 62 projects rated for the Water GP. By commitment, the spread in the ratings is wider, with a high rating of 64 percent S+ for projects under the Energy & Extractives GP and a low rating of 19 percent S+ for projects under the Water GP (figure M.30). At regional level, overall performance under SD is relatively strong in the East Asia and Pacific (86 percent MS+), South Asia (84 percent MS+), and Europe and Central Asia (82 percent MS+) Regions, but less so for the Middle East and North Africa (60 percent MS+), Africa (69 percent MS+), and Latin America and the Caribbean (65 percent MS+) Regions. The gap between SD and non-SD 422 Results and Performance of the World Bank Group 2017 | Appendix M Figure M.29. SD Group: M&E Quality Ratings, FY05–16 a. Percentage of projects rated S+, by number b. Percentage of projects rated S+, by volume Source: Independent Evaluation Group data. Note: Two projects were excluded when measured by volume due to lack of volume data. M&E = monitoring and evaluation; S+ = a rating of substantial or better; SD = Sustainable Development. Independent Evaluation Group | World Bank Group 423 Figure M.30. SD Group: M&E Quality Ratings by Global Practice, FY14–16 Source: Independent Evaluation Group data. Note: Two projects were excluded when measured by volume due to lack of volume data. M&E = monitoring and evaluation; S+ = a rating of substantial or better; SD = Sustainable Development. GPs is largest in South Asia, where SD performance is notably better (84 compared with 63 percent MS+; figure M.31, panel a). By commitment, East Asia and Pacific (91 percent MS+), South Asia (91 percent MS+), and Europe and Central Asia (86 percent MS+) perform strongest under the SD Practice Group. Overall performance by commitment is lower for the SD group than non-SD groups, with relative performance in the Middle East and North Africa and Latin America and the Caribbean Regions falling well below counterpart performance, although overall performance by commitment is relatively strong in the South Asia, and East Asia and Pacific (figure M.31, panel b). World Bank performance for SD projects ranges from a high of 86 percent MS+ in the East Asia and Pacific Region, to a low of 60 percent MS+ in the Middle East and North Africa Region, with an overall rating of 73 percent MS+ across all Regions. World Bank performance in the South Asia Region is notably better for SD projects than non-SD projects during FY14–16 (figure M.32). With an overall rating of 30 percent S+, M&E ratings for SD projects are particularly strong in the Europe and Central Asia (48 percent S+) and South Asia (35 percent S+), and East Asia and Pacific (35 percent S+) Regions, but relatively weak in the Latin America and the Caribbean (18 percent S+), Africa (28 percent S+), Middle East and North Africa (28 percent S+) Regions (figure M.33). 424 Results and Performance of the World Bank Group 2017 | Appendix M Group: Project Outcome Ratings for Practice Group by Figure M.31. SD Region, FY14–16 a. Percentage of projects rated MS+, by number b. Percentage of projects rated MS+, by volume Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. “Other” includes projects that are not mapped to any Practice group. MS+ = a rating of moderately satisfactory or better; SD = Sustainable Development. Independent Evaluation Group | World Bank Group 425 Group: World Bank Performance Ratings for Practice Group by Figure M32. SD Region, FY14–16 Percentage of projects rated MS+, by number Source: Independent Evaluation Group data. Note: Four projects were excluded when measured by volume due to lack of volume data. “Other” includes projects that are not mapped to any Practice group. MS+ = a rating of moderately satisfactory or better; SD = Sustainable Development. Figure M33. SD Group: M&E Quality Ratings for Practice Group by Region, FY14–16 Percentage of projects rated S+, by number Source: Independent Evaluation Group data. Note: Two projects were excluded when measured by volume due to lack of volume data. M&E = monitoring and evaluation; S+ = a rating of substantial or better; SD = Sustainable Development. 426 Results and Performance of the World Bank Group 2017 | Appendix M Evaluation Findings and Lessons This section summarizes key findings and lessons from learning products and PPARs completed by IEG in FY17 for GPs under the SD Practice Group. There were no major evaluations or learning products completed for the SD Practice Group in FY17. For the SD Practice Group, IEG completed four PPARs for projects under the Agriculture GP; one learning product and one PPAR under the ENR GP; four PPARs under the SURR GP; and six PPARs each under the Transport and ICT and Water GPs. There were two learning products but no project- level evaluations completed in FY17 for the Energy and Extractives GP. Agriculture GP The four PPARs completed by IEG in FY17 under the Agriculture GP are summarized below. Nations in Africa had accumulated large stocks of obsolete pesticides over the second half of the 20th century. Some of these stocks included persistent organic pollutants, which posed serious long-term threats to human health and the environment. The Africa Stockpiles Program 2005–13 was an attempt to develop and implement cost-effective and ef ficiency inventory, collection, and disposal operations, and to prevent the accumulation of future stocks, through a partnership of international organizations working in collaboration. The World Bank was to coordinate the Africa Stockpiles Program (Ethiopia, Mali, Morocco, South Africa, Tanzania) and manage funds (from a $25 million Global Environment Facility grant), with technical support from the Food and Agriculture Organization, and other support from civil society organizations. Yet the complex international partnership arrangements made coordination very challenging, leading to confusion in implementation and failure to deliver synergies. The struggle to design partnership arrangements meant that preparation took four years, with little to show for the effort at the country level and an inadequate level of readiness for implementation. At the core of the challenges encountered were fundamental differences in leadership from the World Bank and from the Food and Agriculture Organization. The project delivered several outputs that led to elimination of some pesticide stocks, and project effectiveness varied significantly across countries. But none of the countries could eliminate pesticide stocks and associated waste to the extent originally envisioned, and two countries were not able to eliminate stocks even after project restructuring that reduced the level of ambition. The horizontal adaptable program loan covering six countries at once did not add ef ficiencies in preparation, or synergies in implementation, because there was little opportunity for intercountry interaction, or cooperation. Relatively little was accomplished on preventing future accumulation because project design concentrated on cleanup aspects. Diffuse responsibilities, high turnover of task team leaders, and a lack of sustained attention from the World Bank contributed to weak project supervision. The World Bank’s ICR rated the project outcome as moderately unsatisfactory; IEG rated the project as unsatisfactory, based on incomplete implementation of the project. Independent Evaluation Group | World Bank Group 427 The Danube River Basin is one of Europe’s largest river catchment areas. It suffers from pollution due to a high volume of nutrients, which has substantial negative consequences for soil fertility, agricultural productivity, and conservation of the biological ecosystem. More than half of the nutrient loads originate from agriculture due to intensive fertilizer application and poor manure management. In Croatia, the Danube River and its tributaries drain about 60 percent of the country’s territory, covering large parts of agricultural land. About 40 percent of Croatia’s agricultural holdings raise livestock, with a prevalence of poor manure management practices and lacking awareness of practices to reduce nutrient loads to water bodies. To address pollution in Croatia’s agricultural sector, the Global Environment Facility provided a $5 million grant for the Agricultural Pollution Control Project 2007–12. The project was successful in achieving an increase in the use of environmentally-friendly agricultural practices in three agriculture- intensive target counties. The establishment of a mitigation investment fund, which provided grants for 75 percent of the cost of manure storage for medium-scale livestock farmers was key to this outcome, in addition to an extensive field demonstration program on sustainable, cost-effective manure management. However, the project provided only limited outcome evidence on the reduction in nutrient discharge in water bodies from agricultural sources due to insuf ficient monitoring and impact analyses. Implementation benefited from a strong technical and highly engaged project implementation unit and a mostly consistent government commitment. Croatia’s accession to the European Union in 2013 and the related compliance with Nitrates and Water Directives requirements has ensured continued funding for environmental measures in the agriculture sector. Overall, the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory. In the years leading up to the preparation of the first of the two-part Agriculture Sector DPO Series (the first DPOs prepared specifically for the agriculture sector in Ghana and more broadly, the Africa Region), Ghana experienced a sustained period of economic growth and reductions in national poverty rates. During this period the agricultural sector also experienced strong growth, driven largely by the cocoa subsector, and expansion of the area under cultivation. Most of these gains were achieved in the South and Central regions of the country, where agro-ecological conditions are more favorable; however, poverty remained high in rural areas and was increasingly concentrated in the more arid northern regions. The World Bank prepared the DPO series to support the government with implementing newly developed strategies that aimed at diversifying the economy away from its dependence on cocoa and toward cereals and other cash crops for export markets; addressing regional disparities; and improving the productivity of staple food crops (because of the potential impact on poverty and meeting domestic demand). Historically, budget allocations to the agriculture sector from government sources covered salaries only. The government relied predominantly on investments through rural development and agriculture projects to address other needs. 428 Results and Performance of the World Bank Group 2017 | Appendix M Strengthening the sector’s ability to plan and implement using budgetary resources instead of project funding was viewed as a critical element of implementing a coordinated and internally consistent development policy in the sector. There was clear alignment between the DPO prior actions and government policies, and the prior actions were mostly carried out as designed, although expected development outcomes were not achieved. Many prior actions involved upstream adoption of legislative or government decisions, plans and policies that were insuf ficiently linked to implementation. The government fell short in ensuring implementation, and the intended reforms in many areas have been under resourced. Prior actions were also spread across too many policy areas, with insuf ficient depth in any one area to translate into results in the field. The first objective (increasing agriculture’s contribution to growth and poverty reduction) was rated modest. Agriculture sector growth rates have slowed over time and have been significantly lower than that of the overall economy. There is no hard evidence to establish agriculture’s contribution to poverty reduction. The ef ficacy of the second objective (improved management of soil and water resources) was rated negligible. Despite efforts to disseminate information on various sustainable land management practices, the practices did not result in high adoption rates of better soil and water practices. Insuf ficient resources at the district level and uncertain security of land use rights have constrained further progress. The ef ficacy of the first objective of the second series of DPOs (improving farmers’ productivity and market access) was rated modest. Yields of most major crops have been relatively flat and well below their potential. Farmers continue to face significant hurdles that limit the uptake of productivity-enhancing technologies and access to markets. The ef ficacy of the second objective (improved management of the sector) was rated negligible. Changes to budgetary processes have been minimal, and the structures to facilitate coordination have not functioned effectively. Coordination across the Ministry of Food and Agriculture directorates and other agencies with work affecting the sector remains weak, which is a key bottleneck to improving planning and management of the entire sector. The World Bank’s ICRs rated both phases of the series moderately satisfactory; IEG rated outcome for both phases moderately unsatisfactory. Rural development in Solomon Islands has performed poorly since the peace settlement that followed tensions there in 2003. The World Bank sought to contribute to improvements in rural development by supporting the first phase of the $37 million Rural Development Project (2007–15). The project used a CDD approach to ensure improved social services, enhanced rural infrastructure, and amplified commercial opportunities for farmers. It was intended that phase I of the project would provide the basis for a longer-term commitment from Solomon Islands government to ensure increased outreach and visibility for those in isolated regions. The project was successful in introducing the necessary structural and distributional platforms for the provision of enhanced services. In addition, communities embraced the participatory process for decision making and benefited from the direct funding of small scale infrastructure investments. Importantly, the Ministry of Agriculture and Livestock was re-energized by institutional support provided under the program, and nascent extension activities were now being directed at regions with potential agricultural growth opportunities. Independent Evaluation Group | World Bank Group 429 The project effectively illustrated the benefits of targeted community programs in fragile states, and showed that building sustainable institutional structures for multiphase projects has long-term benefits. In addition, the Rural Development Project showed that capacity building and institutional reform in weak ministerial environments can energize and enhance positive change, but only if there is suf ficient understanding of the gaps and limitations of the department; and that rural enterprise development is dif ficult in isolated regions, requiring a combination of approaches to ensure it can benefit the diverse interests and capabilities of all participants. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the outcome rating was moderately satisfactory, based on monitoring data that demonstrated progress in improving rural livelihoods and ensuring greater access to services. Energy and Extractives GP In FY17, IEG produced two learning products under the Energy and Extractives GP that are summarized below. The first of the learning products, Financial Viability of the Electricity Sector in Developing Countries: Recent Trends and Effectiveness of World Bank Interventions, FY00–15, addresses the effectiveness of World Bank interventions in supporting client countries to improve the financial performance and long-term viability of their electricity sectors. The World Bank supports client countries in improving the financial performance and viability of their electricity sectors principally through financial or institutional development components and financial covenants in investment loans, and through conditionalities in DPOs. IEG’s analysis shows that most investment loans with financial components and covenants for the electricity sector show a moderately satisfactory or better performance regarding their financial performance objectives. However, most DPOs show moderately unsatisfactory or lower performance in this regard. The analysis of the experience of both investment loans and DPOs leads to several findings and lessons, as follows: Aligning the DPO program’s timeline with that of a government reform program. Most World Bank client countries require deep sector reforms that often operate on different timelines than DPOs, but the government, national utility, and the World Bank might focus excessively on short-term management of crises. The generic lesson learned is that the DPO-supported sector reform program’s time horizon and complementary investment projects should reasonably match the time required for effective government action. Matching the scale of World Bank support to the scope of reforms and political risk. The quantum and scope of World Bank support should equal the scale of reforms and the accompanying political risks—for example, in complex reforms involving large tariff adjustments. Balancing the window of opportunity with achievable targets. When the window of political economy opportunity is small, it is better to set realistic, modest targets that are achievable in a relatively short time. 430 Results and Performance of the World Bank Group 2017 | Appendix M Avoiding the pitfalls of overambitious agendas and excessive conditionalities. An overambitious agenda and too many loan conditionalities in DPOs is seen to impact the performance of sector financial objectives. Retaining focus on the underlying structural causes of sector financial viability. Several investment and multitranche DPO operations gave disproportionate attention to developing financial indicators and targets instead of focusing on the underlying structural causes of the sector’s or national utility’s poor financial performance. Overall, programmatic DPOs showed better outcomes than multitranche and one-off DPOs. A programmatic series of single-tranche policy loans can be an effective way to support a well- specified, medium-term financial recovery program. One-off DPOs are prone to excessive complexity, overdesign, and trying to do too much in a short time, with few exceptions. Multitranche loans tend to be less flexible in their conditionalities. IEG’s second learning product under this GP, Reliable and Affordable Off-Grid Electricity Services for the Poor: Lessons from World Bank Group Experience, draws lessons from the World Bank Group’s support for rapid scale-up of off-grid electrification (“pico-solar” products, individual solar home systems, and micro and mini-grids), in client countries. In recent years, there has been renewed focus on the 1.1 billion people around the world without any access to electricity from either the conventional grid or any off-grid or stand-alone source. In this context, the learning product focuses on experiences geared toward ef ficiently and effectively integrating off-grid electrification scale-up efforts with grid rollout—within a national road map for achieving universal access in a given time frame. The experience and good practices from client countries yield the following insights: Putting people (beneficiaries) first, not technology solutions. The opportunity costs of delay in providing even basic access are very high. Access to even small quantities of modern electricity— suf ficient to power basic modern lighting, cell phone charging, a small radio or television—can trigger a giant step for those that are without any access. Significantly, over time, peoples’ expectations tend to grow, and almost all aspire to electricity access comparable in adequacy, quality, and reliability that is typical of a well-managed utility-run grid system. Grid versus off-grid: a falsely framed binary choice. Grid and off-grid technologies are not necessarily either-or options to be determined simply based on a narrow least-cost calculus. Rather, looking ahead, off-grid access provision is appropriately viewed as playing a complementary and coordinated role alongside grid rollout in achieving universal access. Private and public sector engagement. Two noteworthy features in this regard are (i) the evolving scope and roles of the World Bank and IFC in enabling early stage (“premarket”) development and (ii) assisting client countries in orienting the private and public sector to work in close partnership and to leverage their respective strengths toward the scale-up of access. Independent Evaluation Group | World Bank Group 431 Adoption of a sector wide organizing architecture, and programmatic financing, show promise. Providing access in some form to all households, clinics, and trading centers requires identifying where the beneficiaries are located geospatially. This enables least-cost placement and expansion of existing electricity access infrastructure. Further, sectorwide organizing architecture and programmatic financing can be effective for rallying and orchestrating stakeholder participation in closer alignment with national priorities than otherwise; as well as syndication of program financing of an integrated grid and of-grid rollout. Specifically, a sector wide investment prospectus for grid and off-grid expansion would be anchored by an overall least-cost geospatial rollout plan to achieve the time-bound targets set by the government; together with an enabling policy and institutional framework. Environment and Natural Resources GP The one learning product and one PPAR completed by IEG in FY17 under the ENR GP are summarized below. IEG’s learning product, Lessons from Environmental Policy Lending reviews the World Bank’s experience with DPF in the Environment sector, broadly defined. The report offers lessons from evaluation of this experience and informs stakeholders on how to design and implement this instrument, outlining some of the tensions and trade-offs that must be grappled with. The World Bank’s environmental policy lending has grown rapidly since 2005. These operations have supported policy actions across a broad range of subsectors. Yet despite this variation, operations have tended to generate lessons focusing on a common set of issues around political economy, operation design and preparation, and institutional strength and capacity. Environmental development policy lending is most effective when used in a way that plays to the strengths of the instrument. Environmental policy lending can be most effective when policy issues are the main barrier to improving environmental outcomes, rather than capacity or other issues. It offers advantages for achieving sector wide or multisectoral goals across many ministries. It can be most effective when the World Bank has prior knowledge of the country and sector and strong institutional relationships, which may be developed through use of other instruments. It is useful for those policy issues that need attention from high-level decision makers, especially in financing and planning ministries. Its fl xibility allows the World Bank to take advantage of opportunities as they arise, but this can be a risky approach. It can be most effective when used in combination with other instruments. A few key design and implementation considerations tend to determine the extent to which environmental policy lending can be effective. Policy lending is most effective when there is a clear rationale for how the operation will influence policy outcomes. This can include supporting policy reforms that would not happen without the World Bank operation, but also influence on prioritization, timing, or technical quality. Policy lending is more effective with a strategy for achieving institutional buy-in. The design of results frameworks requires intensive dialogue and debate between World Bank teams and governments. It involves trade-offs and tensions: between ambition and realism, between additionality and country ownership, between depth and breadth. The strongest policy actions are 432 Results and Performance of the World Bank Group 2017 | Appendix M those that are relevant, critical, additional, and measurable. Programmatic series offer advantages, including the ability to induce or support longer-term government commitment to reforms. But they can be more effective if they endeavor to include substantial policy actions from the first operation, if they ensure that spacing between operations matches the time needed to complete reform actions, and if careful considerations are made about decisions to drop indicative triggers from future operations. M&E systems for environmental DPF have often been weak and could be improved: the report offers advice on selection of objectives and indicators. Analytical work plays a key role as a diagnostic and for persuading decision makers with evidence. Technical assistance is often critical for development of reforms and completion of policy actions. Yet suf ficient analytic and advisory work is not always present. This has been caused by tightening budgets, declining availability of trust funds, challenges of timing, and the unwillingness of many governments to borrow for technical assistance. The Arab Republic of Egypt’s rapid population growth combined with its economic development and industrialization policies and weak environmental management have resulted in widespread and severe pollution. The World Bank sought to support pollution abatement from large industrial sources in Cairo and Alexandria through the $166 million Second Pollution Abatement Project that operated 2006–12. The project used IBRD financing and cofinancing from other development partners to support credit lines for industrial pollution abatement investments through an apex World Bank, supported by technical assistance. The credit lines were partly concessional, and so the project sought to encourage firms to make investments by combining this “carrot” with the “stick” of enforcement of existing pollution regulations. The project was successful in generating investments by firms, who invested $145 million of their own funds beyond the project financing. Implementation benefited from support provided by a capable project implementation unit from a predecessor project; from the application of a model where the environment agency provided technical advice to the private World Bank carrying out lending; and from the actions taken by the World Bank and government to help build a pipeline of subprojects even in the wake of diminished demand following economic downturn and political crises. Yet, the credit line mechanism was limited in its ability to target the most serious pollution problems because firms with poor creditworthiness could not receive financing. Program design relies on concessional donor finance, and so it may not be possible to scale up, or replicate the program at a level where it would have a significant impact on ambient pollution. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the outcome rating was satisfactory, based on monitoring data that demonstrated significant reductions in pollution emissions from subprojects. Social, Urban, Rural, and Resilience GP The four PPARs completed by IEG in FY17 under the SURR GP are summarized below. The project objective for the Bangladesh Social Investment Program Project (March 2003 to December 2012) was improved access to local infrastructure and basic services, which were Independent Evaluation Group | World Bank Group 433 intended to be delivered through effective and ef ficient financing mechanisms and by using a CDD approach. IEG found the original objective to be modestly relevant due, among other reasons, to the lack of specificity regarding proposed beneficiaries, although this was corrected in the second phase when it was clarified that the project sought to improve the livelihoods of extreme poor communities. The project attracted three rounds of additional financing, although the third additional financing was canceled such that actual project cost at closing was $60.3 million. The project financed the construction of 1,734 community infrastructure works, including 2,490 km of earthen roads, 3,160 culverts, and 2,940 hand tube wells, while 85 schools were repaired meaning the objective of improving access to local infrastructure and basic services was substantially met. An impact assessment was positive about improvements supported by the project such as significantly improved mobility for beneficiaries, increased enrollment in schools, and a decrease in waterborne illness during the project period. However, the impact assessment did not report on the quality of the services delivered, including, importantly, the quality of the water in areas where arsenic is known to be a contaminant, or the quality of the tube wells regarding flood resilience. IEG rated ef ficiency as modest referencing a lack of clarity as to whether the infrastructure built was flood resilient (considered unlikely in a CDD project), and the absence of a calculation of the returns on health savings, owing to the increased access to potable water. IEG found significant risk to development—only 20 percent of all community institutions supported by the project would be sustainable without continuing implementing support, and evidence to support this was not available at the time of the assessment. The PPAR mission also comments that, to be sustainable, there is a need to situate the works provided through CDD projects in Bangladesh within a basin-wide water and disaster management planning approach. IEG drew several lessons from the assessment including the importance of facilitating the formation of strong institutions in rural areas to enable the poor to access services, and the need for thorough assessment of an implementing agency’s capacity to adapt, to manage funding, and to reach disaster-affected populations outside of a project area (notwithstanding the benefits of CDD as an initial platform). IEG rated the project moderately unsatisfactory for development outcome compared with an ICR rating of moderately satisfactory, reflecting the modest relevance of the objective and of design, substantial ef ficacy, and modest effi iency. IEG prepared a clustered PPAR to assess two rural poverty reduction projects carried out by the World Bank in two Brazilian states during 2001–09: the Ceará Rural Poverty Reduction Project and the Santa Catarina Natural Resource Management Project. The projects shared the objective of improving the economic and social conditions of the rural poor, but under very different conditions. Rural poverty rates in the two states were, and remain, opposites. At design, Ceará had the second highest rural poverty rate among Brazilian states, and Santa Catarina had the lowest. IEG’s assessment found that the Ceará CDD project substantially improved access by the poor to water and electricity and to some extent, productive capacity (mainly tractors), but it had negligible effect on health and education. The Santa Catarina Natural Resource Management project achieved its objective of integrating environmental and social sustainability into development and poverty reduction strategies. However, it only modestly enhanced local governance and community 434 Results and Performance of the World Bank Group 2017 | Appendix M participation in decision making. The aim of reversing land degradation and protecting natural resources was only modestly achieved based on data obtained on water and soil quality for targeted micro-catchments. A key lesson that emerged from Ceará is that CDD projects may not be the best option to provide access to “last mile” public services, such as electricity. The provision of last mile services requires clear participation criteria underpinned by accurate welfare data, and a system of aid allocation that is transparent and accounted for by beneficiaries. The two projects confirm that area poverty targeting arrangements, unaccompanied by clear selection criteria, are likely to result in reaching less of the rural poor than is desired. IEG also found that an integrated poverty-environment approach, as was the case in Santa Catarina, may be a better option than addressing these objectives separately, through stand-alone projects. In Ceará, assistance to farmers, particularly on productive (tractor) subprojects, did not benefit enough from land management techniques, resulting in farming technologies that could have an adverse effect on soil quality. Nevertheless, as water availability and adaptation to the adverse impacts of periodic droughts are the main natural resource-related challenges in Ceará, the project’s support for small water infrastructure partly addressed these challenges. IEG rated overall development outcome moderately satisfactory for the Ceará project, one step down from the self-rating arrived at in the project’s ICR. IEG rated the overall development outcome satisfactory for the Santa Catarina project, one step up from the moderately satisfactory self-rating arrived in that project’s ICR. Over the last 30 years, Colombia has increased its efforts to prevent and mitigate natural hazards and deal ef ficiently with their consequences when they strike. This World Bank Disaster Risk Management DPL (2009–12) sought to strengthen the government’s program for reducing risks resulting from adverse natural events through a policy program targeted at improving risk identification and monitoring, increasing risk prevention and mitigation, strengthening the national system for disaster management and prevention, and reducing the fiscal vulnerability of the state to natural disasters. The budget support operation had a catastrophe deferred drawdown option for disbursement of the loan to facilitate quick-release financing in the event of a disaster. The program was successful in advancing reforms in the four policy areas: significantly expanded disaster monitoring stations made it possible to know more precisely who was at risk and what risks they faced; disaster risk management plans and investments increased across all levels of government; legal and institutional frameworks for disaster management were strengthened; and the government advanced in understanding contingent liabilities from natural disasters, assessing options for public insurances, and improving the legal framework for infrastructure concessions to reduce the fiscal vulnerability to natural disasters. On the other hand, the original approach to resettle the residents from the Galeras volcano zone encountered many dif ficulties and had to be modified. The program also had a deficient M&E framework, which poorly captured the program’s scope and results, and limited learning about the catastrophe deferred drawdown option instrument. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the outcome rating was satisfactory, based on data going beyond the formal monitoring data. Independent Evaluation Group | World Bank Group 435 Despite the frequency and severity with which the Philippines suffers from natural hazards, which partially explains the stubbornly high poverty and inequality in the country, there was limited attention to disaster risk management (DRM) prior to 2009. Since then, the government has sought to develop a coherent, integrated, and proactive approach to DRM, which the World Bank has supported by increasing the focus on resilience to natural disaster in its country strategies. The World Bank Disaster Risk Management DPL with a Catastrophe Deferred Drawdown Option (2012–15) aimed to enhance the capacity of the Government of the Philippines to manage the impacts of natural disasters through a policy program targeted at strengthening the institutional capacity for DRM efforts, mainstreaming DRM into development planning; and better managing the government’s fiscal exposure to natural hazard impacts. The operation had a catastrophe deferred drawdown option for disbursement of the loan proceeds to provide quick-release financing in the event of a disaster. The operation was complemented by a technical assistance program from the World Bank Global Fund for Disaster Risk Reduction to provide targeted support in these areas. The program was successful in achieving the goal of enhancing the capacity of the country to manage the impacts of natural disasters. The relevant agencies demonstrated high ownership of the reform agenda, and there was considerable progress in all three areas of the policy program. The weakest spot identified by IEG was in the design and implementation of a monitoring system to track disaster financing where weak institutional capacity handicapped the Of fice of Civil Defense. Both the World Bank’s ICR and IEG’s independent field-based evaluation concluded on a satisfactory development outcome rating based on data monitored under the program and other additional, supporting evidence. Transport and Information and Communication Technologies GP The six PPARs completed by IEG in FY17 under the Transport and Information and Communication Technologies GP are summarized below. When the Buenos Aires Urban Transport Project (1997–2011) was being appraised, Argentina was embarking on large-scale privatization reforms of infrastructure, including transport. The poor state of infrastructure, large operating deficits, and significant investment needs motivated the privatization of the Argentine railway system, including the Buenos Aires subway and suburban rail. The World Bank supported government commitments under the concession contract agreement with Metrovias, a private firm, that included improvements in four subway lines, as well as the Urquiza suburban rail. After the economic crisis in 2001 in Argentina, both the government and Metrovias faltered on their obligations under the concession contract. The concessionaire received compensation for the costs incurred but it lacked incentives for cost control and ef ficiency in urban rail operations. In addition, low fares maintained through government subsidies subsequently contributed to a growing fiscal deficit, while underinvestment in the system continued. The experience of this project demonstrates how a PPP scheme can underperform if sector reforms and the right price signals are not in place. The project also provided institutional support to contribute to the development of an integrated urban transport system for Greater Buenos Aires. The 436 Results and Performance of the World Bank Group 2017 | Appendix M formation of an entity to coordinate metropolitan transport planning in Buenos Aires was supported under this project from its inception, and the Metropolitan Transport Agency was finally established in 2014 under a subsequent World Bank operation and supported political will that brought progress and functionality to the new entity. In addition, the project helped improve safety through construction of road/rail grade separated crossings and the development of integrated urban transport strategies in the metropolitan areas of Cordoba, Mendoza, Posadas, Rosario, and Tucuman. Overall, the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory. Ghana emerged from an economic crisis in 2006 and wanted to diversify its growth for sustainability and achieving middle-income status by 2015. It emphasized the importance of ICT in achieving the country’s objectives of diversified export-led economic growth, increased competitiveness, and transparent, accountable and ef ficient government. The World Bank sought to support the government through a $116 million eGhana Project 2006–2014 in generating growth and employment by developing the Information Technology Enabled Services industry, and contributing to improved ef ficiency and transparency of selected government functions through e-government applications. The project used IDA financing together with cofinancing from other development partners for the Government Integrated Financial MIS. The project created an enabling environment for ICT sector, supported investments in Information Technology Enabled Services, established a Business Process Offshoring center and conducted business incubation activities. The ICT sector grew at 23.3 percent per year in the project period. The project automated business processes of Ghana Revenue Authority, which led to an 400,000 increase in new tax payers and a tripling of tax collection between 2010 and 2015. The project-automated Registrar General Department led to a 55 percent increase in company registration in 2012–13. The project supported setting up several eServices like birth certification, online marriage registration, company registration, and so on; however, its coverage was only about 10 percent due to low Internet penetration, poor connectivity and lack of citizen awareness. The project implemented a MIS covering central government funds that accounted for 66 percent of total public expenditure. However, there was inadequate funding for the operation and maintenance of the Government-Wide Area Network leading to a risk to the collapse of all e-Government applications and the MIS. Overall, the World Bank’s ICR rated development outcome satisfactory, IEG’s filed based assessment rated outcome moderately satisfactory due to low coverage of eServices, limited control of the MIS, and insuf ficient impact on employment. Mumbai Metropolitan Region is one of the world’s largest urban areas. Suburban railway services are the backbone of the agglomeration, but appraisal undertaken for the Mumbai Urban Transport Project found them to be grossly overcrowded, especially in the peak hours. The road and bus systems also suffered from bottlenecks and lack of investment. The World Bank supported a comprehensive approach that included improving the capacity and performance of the suburban rail system, an enhanced bus system, the introduction of Area Traf fic Control for the roads to optimize Independent Evaluation Group | World Bank Group 437 the functioning of traf fic signals, as well as improvements to pedestrian facilities, road improvement schemes and parking controls. The project involved the relocation of over 100,000 people affected by the project, many of whom lived in squatter settlements adjacent to the railway tracks. This aspect turned out to be more complex than expected and grievances expressed by affected persons were not adequately addressed leading to an Inspection Panel investigation; however, by the end of the project following the implementation of an appropriate action plan, most people were rehoused or compensated. Moreover, the knowledge gained in the resettlement activities enabled the regional development authority to set up a Resettlement Division that has successfully replicated the procedures developed in the Mumbai project, in other non–World Bank financed developments. Both suburban rail and bus fares were revised upward and further rail improvements have been undertaken in a follow-on project. However, at the time of the assessment, the regional government had shied away from more radical policy reforms such as a road user charging, land use reform, and the introduction of parking restrictions, so any capacity improvements are soon dissipated by increased traf fic congestion, while air quality in the region continues to deteriorate. On the other hand, institutional capacity has improved and infrastructure maintenance is adequate. Provision of free, alternative housing with title has had an empowering effect on project-affected households. Overall, the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory, reflecting the achievement of most objectives albeit with some shortcomings. Morocco’s urban transport systems have been under considerable pressure due to rapid urbanization. Public transport has been inadequate, of low quality, and disorganized—particularly the bus-tramway integration in Rabat and Casablanca. These urban mobility issues are partly the result of underinvestment and the lack of financial sustainability, thus hindering the participation of new private operators. The World Bank supported the Urban Transport Sector DPL with $136.7 million in pursuit of three specific policy objectives: (i) to improve the governance of the urban transport sector; (ii) to increase the ef ficiency and supply of urban transport infrastructure and services; and (iii) to improve the sector’s environmental and social sustainability. The government implemented 10 prior actions under these policy areas. A key finding is that the World Bank’s analytical and advisory work contributed greatly in helping ensure that the prior actions built directly on the government’s own priorities, and were causally linked to the DPL’s targeted outcomes. Nonetheless, additionality and attribution are pertinent issues, since many of the prior actions had already been initiated by the government before the DPL’s approval. The first and third policy objectives were substantially achieved, but the second objective achieved only modest progress in addressing operational ef ficiencies and lack of financial sustainability in the Rabat and Casablanca agglomerations. Lessons learned from the PPAR exercise relate to the need for (i) a strong reform momentum, (ii) early participation by implementing agencies and local governments, and (iii) flexible and adaptive responses. The World Bank clearly contributed in terms of its convening authority and intellectual leadership; however, this sector DPL may have been more effective had it been accompanied in parallel by investment lending and technical assistance 438 Results and Performance of the World Bank Group 2017 | Appendix M operations. Overall, the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory. Lagos is the largest city in Sub-Saharan Africa, but for many years it has suffered from severe traf fic congestion and fragmented and uncoordinated public transport. The World Bank–supported Lagos Urban Transport Project sought to help the government of Nigeria improve transport services in the Lagos Metropolitan Area through management and road network improvements, enhanced bus services, and the promotion of water and nonmotorized transport. The emphasis was on sustainability and improving the quality of service for the poor. Central to the project was the establishment of a strong transport authority to coordinate and enforce planning and regulation, but the setting up phase took longer than anticipated. A crucial lesson was to learn that urban transport projects, because of their complexity, require longer periods for implementation. The creation of a transport fund was also an important step on the way to financial sustainability. A critical factor was to bring on board small operators and trade unions, and this was eventually achieved through study tours so that key players could observe successful transport models in other countries. The project team displayed considerable flexibility and used windfall savings to finance the design costs of the first phase of a rapid bus transit system, and follow-on projects are now supporting the expansion of this system. However, weaknesses included the failure to implement a results framework in the first few years of the project and lack of progress in attracting private sector investment in ferry services. Nonetheless, poor households reported time and cost savings and enhanced safety when using public transport. Overall, the World Bank ICR rated development outcome highly satisfactory. Following IEG’s field-based assessment, the project was rated moderately satisfactory reflecting the absence of a results framework and adequate indicators during the early stages of the project. Senegal’s transport sector is of high strategic importance to its economy, yet urban mobility— particularly in the Greater Dakar Area—has been a key impediment to growth. The country’s urban transport sector has been characterized by the government’s inability to meet rapid increases in service demand severe traf fic congestion, poor and deteriorating road conditions, lack of traf fic management, an aging transport fl et, inadequate regulations, and serious gaps in maintenance. The World Bank’s Urban Mobility Improvement Project sought improvements in the safety, ef ficiency and quality of urban mobility in Greater Dakar Area, and road safety in Thies and Kaolack. The interventions aimed at achieving this objective included (i) investments in road rehabilitation, road safety and traf fic management; (ii) upgrading of pedestrian and traf fic safety along the suburban railway line; and (iii) the piloting of a minibus leasing scheme to finance fleet renewal. The overly ambitious project had a highly complex design but was inadequately prepared. It was assigned to a very weak implementing agency that had just been created. Its technical aspects did not adequately capture the continuing rapid growth in vehicular traf fic, which had the effect of canceling out the project’s benefi s. Although the minibus leasing scheme was successful and road safety improved, the road network continues to deteriorate rapidly due to lack of maintenance, scarcity of funding, and Independent Evaluation Group | World Bank Group 439 continued use by overloaded trucks. Railway-related outcomes were not achieved, and operations were in a worse state when the project closed. The project yielded lessons about the importance of having a competent lead agency backed up by strong and sustained government support, coordination of land use and transport planning at the metropolitan level, and the adequate funds for road maintenance. To enhance the success of minibus leasing schemes, a replicable lesson indicates the need for (i) bus operator inputs at the design stage; (ii) targeted technical assistance to professionalize bus operators and drivers; and (iii) the restructuring of the network of informal operators. Overall, the World Bank’s ICR rated development outcome moderately unsatisfactory, and IEG’s independent field-based assessment rated development outcome unsatisfactory. Water GP The six PPARs completed by IEG in FY17 under the Water GP are summarized below. Water supply and sanitation in Colombia has improved in recent decades. Between 1990 and 2010, access to improved sanitation increased from 67 percent to 82 percent, and access to improved water sources increased from 89 percent to 94 percent, but coverage in rural areas continues to lag. IEG completed a clustered PPAR covering three projects—the Cartagena Water Supply, Sewerage and Environmental Management Project (1999–2009); the Water Sector Reform Assistance Project (2001–10); and the Water and Sanitation Sector Support Project (2005–11)—that form part of the second-generation water supply and sanitation (WSS) projects in Colombia. These projects benefited from lessons learned in the 1990s from World Bank–supported WSS projects in that country, although somewhat different in reach, focus, nature, and geographical coverage. The first two portray a somewhat logical development in the World Bank’s support to Colombia’s water sector as they tried to replicate the private sector operator model but at a smaller scale in medium and small municipalities. In contrast, the third focused on building infrastructure and support to both public sector as well as private sector run utilities. The World Bank’s design of these projects created a synergy with the changes and developments within Colombia’s water sector and its reforms by focusing on the provision of water supply and sewerage services to the most underserved sections of Colombian society—the poor segments of the income strata, some rural areas, and small and medium-sized cities. While at different scale, the three projects shared components of what was at the time an innovative focus on private sector participation, including (i) sector reform to support private sector participation; and (ii) strengthening of private sector participation entities, in addition to supporting the actual expansion of WSS services. The key lessons from IEG’s assessment are as follows: consolidation of smaller WSS operations servicing poorer neighborhoods can foster economies of scale and cross-subsidization in achieving financial sustainability at the aggregate level; small municipalities with limited service coverage require large capital investments, and because of their lack of financial autonomy, government subsidies would be required until full cost recovery is achieved; it is crucial to phase and/or sequence project activities to ensure achievement of the objectives of utility operators; management capacity of the 440 Results and Performance of the World Bank Group 2017 | Appendix M utility can be strengthened by contracting a competent operator; a carefully designed M&E system and comprehensive planning are essential for success. The objectives for the Cartagena Water Supply, Sewerage and Environmental Management Project (1999–2009) were to (i) improve the water and sewerage services; (ii) facilitate the environmental cleanup of water bodies surrounding the project area and to (iii) improve the sustainability of water and sewerage services through a private sector participation model. IEG rated the overall project outcome satisfactory. The objectives for the Water Sector Reform Assistance Project (2001–10), were to (i) facilitate an increased participation of the private sector in the management and operation of water utilities, (ii) expand the coverage of water supply and sewerage services provided in participating municipalities, (iii) facilitate access to water and sewerage services by the low-income population, (iv) improve environmental protection practices, and (v) define rural waste and sanitation policies and develop adequate methods for increasing coverage for both water and sewerage services in the rural sector. Overall outcome of the project was rated moderately unsatisfactory based on the modest achievement of two of the four objectives, modest design, and modest ef ficiency associated with a long implementation delays and increased budget requirement. The project objective for the Water and Sanitation Sector Support Project (2005–11) was “to improve the access to WSS services in rural and urban communities throughout the Borrower’s territory.” IEG rated the outcome moderately satisfactory. Between 1996–2006, Maoist insurgency plunged Nepal into a prolonged period of political instability that contributed to retardation of economic development. Most rural men had migrated to urban centers, leaving villages largely inhabited by women, children, and the elderly. Water supply coverage was 72 percent and sanitation coverage was only 25 percent. Through the $73.6 million Second Rural Water Supply and Sanitation Project (2004–12), the World Bank sought to support the government in improving rural WSS institutional performance; in mainstreaming the “Fund Board” approach (a demand-driven and community-managed approach in which funds are transferred to communities for procurement and implementation) in government’s systems; and in supporting communities to form inclusive local WSS user groups capable of planning, implementing, and operating drinking water and sanitation infrastructure. The project applied financing sourced from IDA, the government, and local communities to form 1,465 local WSS user groups in 831 villages, benefiting 1,140,892 people in the project area, resulting in an increase in access to tap water from 7.1 percent to 94.6 percent. User groups planned and constructed 64,407 sanitary latrines using project funds, and an additional 72,080 latrines using their own resources and prompted by increased awareness. Selective impact studies showed a significant increase in the use of improved toilets, hand, washing practices, and a decrease (from 78 percent to 14 percent) in the prevalence of diarrheal disease among young children over the life of the project. Sustainable benefits were derived from the decision to place women, the main stakeholders, at the center of planning and implementation. However, the Fund Board approach was not mainstreamed in government systems due to a lack of government ownership, and the failure to realize this objective Independent Evaluation Group | World Bank Group 441 contributed to IEG’s moderately satisfactory assessment of overall outcome, marking down the World Bank’s self-assessment of satisfactory. The National Water Sector Reform Project 2004–10 project sought to increase access to piped water networks in selected urban areas in Nigeria, and to improve reliability and financial viability of selected urban water utilities, in Kaduna, Kano, and Ogun States of Nigeria. The project, which received $202 million of IDA assistance, also addressed the challenges facing urban water utilities including lack of cost recovery, maintenance, reliability, and low investment in rehabilitation and expansion. Targets for new household connections were initially exceeded by 40 percent, and partially achieved (80 percent) for rehabilitated connections, but both have declined in the years following project completion. Overall water production, which had grown from 250 to 865 million liters per year during the project, has since declined to 606 million liters per year. In addition, the urban population benefiting from the project had declined from 5.4 million to 4.1 million. Operations and Maintenance coverage from revenues increased from an average of 20 percent across the three states to 60 percent at project completion, but dropped back to 20 percent by late 2016. The project saw two of five planned PPP implemented, though these did not involve major investment or risk transfer to the private sector, and did not sustainably advance reforms on a large scale. State water and sanitation policy legislation, which included the establishment of regulatory bodies, were drafted in all three states as planned, but they were not enacted by the time the project closed. Important lessons from the project are that necessary institutional restructuring and incentive mechanisms should be addressed upfront in implementing reforms among multiple state and national stakeholders and that reliable and valid monitoring is crucial to improve utility performance and accountability to customers. The World Bank ICR rated overall development outcome moderately satisfactory, and IEG’s independent fi ld-based evaluation rated development outcome moderately unsatisfactory. IEG completed a clustered assessment of two projects in Peru. The Lima Water Rehabilitation and Management Project (2002–13) received $170 million from the World Bank, while the National Rural Water Supply and Sanitation Project (2002–13) received $110 million. The projects closed eight years and four-and-a-half years, respectively, after their planned closing dates. Under the Lima project, although the plans for privatizing the utility operations were dropped by the government, the alternative approach of upgrading management capacity and introducing performance-based incentives yielded the targeted physical infrastructure and financial results. Expanding services to the urban poor exceeded the revised targets; however, the pilot exercise for using low-cost “condominial” water and sewerage networks did not yield significant results. Under the “Rural” project, the principal objective of increasing the sustainable use of WSS facilities in rural areas and small towns was underachieved. There were no significant differences in desired behavior (hygiene practices) observed between populations in the project control areas and those outside the control areas. Lack of capacity to manage the facilities persisted, and there was a lack of administrative and financial support down the chain of government agencies, municipalities and local government, and rural community water and sanitation services associations. An important lesson from the Lima project is that a strong 442 Results and Performance of the World Bank Group 2017 | Appendix M focus on project outcomes and adequate accountability mechanisms are key factors in achieving significant improvements in service provision. An important lesson from the Rural project is that, while community participation in planning and operations underpins sustainability, adequate training support, financial incentives, and contractual arrangements are necessary for continued and effective participation. Overall, the World Bank’s ICR concluded that the Lima project’s outcome rating was satisfactory and the outcome rating for the Rural project was moderately unsatisfactory. IEG’s independent field-based evaluation concluded that project outcome rating for the Lima project was moderately satisfactory, and for the Rural project, IEG’s assessment rated project outcome unsatisfactory. The $69.1 million Second Community Water Supply and Sanitation Project (2003–10) supported the government of Sri Lanka’s ongoing commitment to expanding rural WSS services. IEG found that, overall, the project contributed to providing adequate, affordable, and relatively sustainable water services, and to ensuring convenience and time saving for the beneficiaries. Through a Sanitation Revolving Fund, the project has also contributed to achieving and maintaining a high percentage of toilet availability for rural households. The first project objective—increasing service coverage of water and sanitation services in rural communities—was modestly achieved partly due to reallocation of funds following the 2004 tsunami, as well as overly ambitious targets for increasing service coverage at the design stage. The second objective—achieving effective and sustained use of water and sanitation services in rural communities—was substantially achieved although there have been some challenges in ensuring reliability and water quality, such as the lack of 24-hour supply and water contamination. Important lessons from the project were that lack of continuity in M&E and its use by an implementing agency beyond project completion undermines sustainability of development outcomes, and that consistent institutional and technical support is needed to achieve sustainable service delivery in community based organization schemes. Overall, the World Bank’s ICR and IEG’s independent field-based evaluation concluded that the project’s outcome rating was moderately satisfactory. In 2015, 90 percent of Zambia’s urban residents had access to improved water supply, but only 41 percent had connections in their house or yard, while 49 percent relied on water kiosks or standpipes. Access to improved sanitation was only 56 percent in urban areas. The Water Sector Performance Improvement Project (2006–13) provided $23 million in IDA assistance, including a $10 million grant for two original objectives—improving access and sustainability of WSS services in Lusaka and developing a comprehensive institutional structure supporting a coordinated approach to WSS investments—and the two revised objectives of improving the technical ef ficiency and financial sustainability of Lusaka Water and Sanitation Company and improving access to WSS services for urban consumers in Lusaka, Kafue, Chongwe, and Luangwa districts; and strengthening the effectiveness of national WSS planning. Under the project, about 5,000 urban residents were provided access to improved water supply for the first time, and overall water availability increased from 210,000 to 230,000 cubic meters (per day). Independent Evaluation Group | World Bank Group 443 However, IEG’s focus group discussions with selected beneficiaries indicated reduced levels of satisfaction regarding the adequacy, reliability, and quality of WSS services. The planning capacity of the sector ministry remains weak. The ministry has no clear mechanism to coordinate with or regulate the city councils that own and manage the WSS utilities. Among the lessons learned from this project are that sustainable provision of water supply services in the face of growing population and demand requires active coordination between the authorities responsible for long-term water resource planning, and service providers. Alternative approaches need to be actively explored and adopted for urban and rural sanitation, especially when conventional sewerage may not be financially feasible. The World Bank’s ICR rated overall development outcome moderately satisfactory; IEG’s independent fi ld-based assessment rated development outcome moderately unsatisfactory. List of Summarized Works World Bank. 2016. Bangladesh— Rural Transport Improvement Project, Phase I. Independent Evaluation Group, Project Performance Assessment Report 106729. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_bangaldeshruraltransport_0716. PDF. ———. 2016. Bangladesh—Social Investment Program Project. Independent Evaluation Group, Project Performance Assessment Report 106442. Washington, DC: World Bank. http://ieg.worldbank- group.org/sites/default/files/Data/reports/ppar_bangladesh_2016.PDF. ———. 2016. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Inde- pendent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/ default/files/Data/Evaluation/files/behindthemirror_0716.PDF. ———. 2016. Brazil—Rural Poverty Reduction and Community Driven Development Lessons from Ceará and Santa Catarina Projects. Independent Evaluation Group, Project Performance Assessment Report 106455. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/reports/ppar_brazil_0716.PDF. ———. 2016. Financial Viability of the Electricity Sector in Developing Countries: Recent Trends and Ef- fectiveness of World Bank Interventions. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/lp_financial_viabili- ty_electricity_sector.PDF. ———. 2016. Findings from Evaluations of Policy-Based Guarantees. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ lp_policy_based_guarantees_102116.PDF. ———. 2016. Former Yugoslav Republic of Macedonia—Policy-Based Guarantee. Independent Evaluation Group, Project Performance Assessment Report 106177. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_macedonia_policy_0716.PDF. 444 Results and Performance of the World Bank Group 2017 | Appendix M ———. 2016. Ghana—Economic Governance and Poverty Credit, and Seventh and Eighth Poverty Reduc- tion Support Credits. Independent Evaluation Group, Project Performance Assessment Report 106279. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/ reports/ppar_ghanaecon_0716.PDF. ———. 2016. Ghana—Statistical Development Project. Independent Evaluation Group, Project Perfor- mance Assessment Report 106283. Washington, DC: World Bank. http://ieg.worldbankgroup. org/sites/default/files/Data/reports/ppar_Ghana_0816_0.PDF. ———. 2016. India—Mumbai Urban Transport Project. Independent Evaluation Group, Project Perfor- mance Assessment Report 106391. Washington, DC: World Bank. http://ieg.worldbankgroup. org/sites/default/files/Data/reports/ppar_india_0716.PDF. ———. 2016. Industry Competitiveness and Jobs: An Evaluation of World Bank Group Industry-Specific Support to Promote Industry Competitiveness and Its Implications for Jobs. Independent Evalua- tion Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/ Evaluation/files/industry-competitiveness-and-jobs.PDF. ———. 2016. Lao People’s Democratic Republic—Poverty Reduction Fund Project. Independent Evalu- ation Group, Project Performance Assessment Report 106387. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_laos_0716.PDF. ———. 2016. Learning from IDA Experience: Lessons from IEG Evaluations. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ lp_ida_0716.PDF. ———. 2016. Lessons from Environmental Policy Lending. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/lp_envi- ronmentalpolicylending_0716_0.PDF. ———. 2016. Malawi— Third Social Action Fund (MASAF 3) APL II (LDF Mechanism) Project. Independent Evaluation Group, Project Performance Assessment Report 106389. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_malawi_0716.PDF. ———. 2016. Montenegro—Financial Sector Policy-Based Guarantee. Independent Evaluation Group, Proj- ect Performance Assessment Report 106282. Washington, DC: World Bank. http://ieg.world- bankgroup.org/sites/default/files/Data/reports/ppar_montenegro_0716.PDF. ———. 2016. Morocco—Urban Transport Sector Development Policy Loan. Independent Evaluation Group, Project Performance Assessment Report 106390. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_morocco_2016.PDF. ———. 2016. Mozambique— Poverty Reduction Support Credits 3–5. Independent Evaluation Group, Project Performance Assessment Report 106459. Washington, DC: World Bank. http://ieg.world- bankgroup.org/sites/default/files/Data/reports/ppar_mozambique_0716.PDF. ———. 2016. Niger—Ownership, Capacity, Results Focus and Accountability: Lessons for Enhancing the Bank’s Program Approach to Health and Population. Independent Evaluation Group, Project Per- Independent Evaluation Group | World Bank Group 445 formance Assessment Report 106254. Washington, DC: World Bank. http://ieg.worldbankgroup. org/sites/default/files/Data/reports/ppar_niger_0716.PDF. ———. 2016. Peru— Lima Water Rehabilitation and Management Project and National Rural Water Supply and Sanitation Project. Independent Evaluation Group, Project Performance Assessment Report 106723. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/ reports/ppar_peru_0716.PDF. ———. 2016. Senegal—Urban Mobility Improvement Project. Independent Evaluation Group, Project Per- formance Assessment Report 108407. Washington, DC: World Bank. http://ieg.worldbankgroup. org/sites/default/files/Data/reports/ppar_Senegal_102016.PDF. ———. 2016. Tanzania—Second Social Action Fund. Independent Evaluation Group, Project Performance Assessment Report 106388. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/ default/files/Data/reports/ppar_tanzania_0716.PDF. ———. 2016. The Role of Political Economy Analysis in Development Policy Operations. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/reports/lp_poleconanalysis_0716.PDF. ———. 2016. The World Bank Group’s Support to Capital Market Development. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/Eval- uation/files/wbgsupporttocapitalmarketdev-full.PDF. ———. 2016. Vietnam—Evaluation of Public Financial Management Reform. Independent Evaluation Group, Project Performance Assessment Report 106635. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_vietnam_0716.PDF. ———. 2016. World Bank Group Engagement in Situations of Fragility, Conflict, and Violence. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/Evaluation/fi es/fcv-full.PDF, ———. 2016. World Bank Group Engagement in Small States: The Cases of the OECS, Pacific Island Countries, Cabo Verde, Djibouti, Mauritius, and the Seychelles — Clustered Country Program Evaluation. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbank- group.org/sites/default/files/Data/reports/ExecSum-SmallStates.PDF. ———. 2016. Zambia— Public Sector Management Program Support Project. Independent Evaluation Group, Project Performance Assessment Report 106280. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_zambia_0716.PDF. ———. 2017. Arab Republic of Egypt—Second Pollution Abatement Project. Independent Evaluation Group, Project Performance Assessment Report 113565. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar-egypt-04282017.PDF. ———. 2017. Argentina—Buenos Aires Urban Transport Project. Independent Evaluation Group, Project Performance Assessment Report 113664. Washington, DC: World Bank. http://ieg.worldbank- group.org/sites/default/files/Data/reports/ppar-argentina-04282017.PDF. 446 Results and Performance of the World Bank Group 2017 | Appendix M ———. 2017. Brazil—Pernambuco Education Results and Accountability Project. Independent Evaluation Group, Project Performance Assessment Report 114046. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar-brazil-07142017.PDF. ———. 2017. Cambodia—Public Financial Management and Accountability Project. Independent Evalu- ation Group, Project Performance Assessment Report 108359. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_cambodia_12282016.PDF. ———. 2017. Colombia—Disaster Risk Management Development Policy Loan. Independent Evaluation Group, Project Performance Assessment Report 114745. Washington DC: World Bank. http://ieg. worldbankgroup.org/sites/default/files/Data/reports/ppar-colombiadisaster-07182017.PDF. ———. 2017. Colombia—Private Paths to Sustainable Water Supply and Sanitation in Colombia. Indepen- dent Evaluation Group, Project Performance Assessment Report 111404. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_colombia_01132017. PDF. ———. 2017. Croatia—Agriculture Pollution Control Project. Independent Evaluation Group, Project Perfor- mance Assessment Report 115691. Washington, DC: World Bank. http://ieg.worldbankgroup.org/ sites/default/files/Data/reports/ppar-croatia-06272107.PDF. ———. 2017. Ethiopia, Mali, Morocco, South Africa, Tanzania, Tunisia—Africa Stockpiles Program. Inde- pendent Evaluation Group, Project Performance Assessment Report 108524. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_africa_102416_2. PDF. ———. 2017. Ghana—Agriculture Development Policy Operations: Phase I–IV. Independent Evaluation Group, Project Performance Assessment Report 112622. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar-ghanaagridpo-3202017.PDF. ———. 2017. Ghana—eGhana Project. Independent Evaluation Group, Project Performance Assessment Report 108359. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/reports/ppar_ghana_01032017_0.PDF. ———. 2017. Higher Education for Development: An Evaluation of the World Bank Group’s Support. Inde- pendent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/ default/files/Data/Evaluation/files/highereducation.PDF. ———. 2017. Indonesia—Public Expenditure Support Facility Development Policy Loan with Deferred Drawdown Option. Independent Evaluation Group, Project Performance Assessment Report 110741. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/re- ports/ppar-indonesia-06262017.PDF ———. 2017. Kenya—Development of the National Statistical System Project. Independent Evaluation Group, Project Performance Assessment Report 110966. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_kenya_01042017.PDF. Independent Evaluation Group | World Bank Group 447 ———. 2017. Kyrgyz Republic—Rural Education Project. Independent Evaluation Group, Project Perfor- mance Assessment Report 114047. Washington, DC: World Bank. http://ieg.worldbankgroup.org/ sites/default/files/Data/reports/ppar-kyrgyzeducation-07172017.PDF. ———. 2017. Lao People’s Democratic Republic—Second Education Development Project. Independent Evaluation Group, Project Performance Assessment Report 113148. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar-laos-04192017.PDF. ———. 2017. Nepal—Second Rural Water Supply and Sanitation Project. Independent Evaluation Group, Project Performance Assessment Report 114286. Washington, DC: World Bank. http://ieg.world- bankgroup.org/sites/default/files/Data/reports/ppar-nepalwater-07122017.PDF. ———. 2017. Nigeria—National Water Sector Reform Project. Independent Evaluation Group, Project Per- formance Assessment Report 115782. Washington, DC: World Bank. http://ieg.worldbankgroup. org/sites/default/files/Data/reports/ppar-nigeriawater-07122017.PDF. ———. 2017. Panama—Improving Basic Health, Equity, Opportunities among Indigenous and Poor Rural Communities. Independent Evaluation Group, Project Performance Assessment Report 110617. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ ppar-panama-07052017.PDF. ———. 2017. Peru—Fiscal Management and Competitiveness Programmatic Development Policy Loans. Independent Evaluation Group, Project Performance Assessment Report 110742. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_perufis- cal_03062017.PDF. ———. 2017. Philippines—Disaster Risk Management Development Policy Loan with a Catastrophe Deferred Drawdown Option. Independent Evaluation Group, Project Performance Assessment Report 114746. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/reports/ppar-philippinesdisaster-07172017.PDF. ———. 2017. Program-for-Results: An Early-Stage Assessment of the Process and Effects of a New Lend- ing Instrument. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.world- bankgroup.org/sites/default/files/Data/Evaluation/files/program-for-results-full.PDF. ———. 2017. Public-Private Partnerships in Health: World Bank Group Engagement in Health PPPs—An IEG Synthesis Report. Independent Evaluation Group. Washington DC: World Bank. http://ieg. worldbankgroup.org/sites/default/files/Data/Evaluation/files/lp_Health_PPP_1116.PDF. ———. 2017. Reliable and Affordable Off-Grid Electricity Services for the Poor: Lessons from the World Bank Group Experience. Independent Evaluation Group. Washington, DC: World Bank. http://ieg. worldbankgroup.org/sites/default/files/Data/Evaluation/files/lp_off-grid_electricity_1116.PDF. ———. 2017. Senegal—A Decade of World Bank Support to Senegal’s Nutrition Program. Independent Evaluation Group, Project Performance Assessment Report 110290. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/281111484163839170/PDF/110290-PPAR- P070541-P097181-P115938-PUBLIC.PDF. 448 Results and Performance of the World Bank Group 2017 | Appendix M ———. 2017. Senegal—Une décennie de soutien de la Banque mondiale au programme de nutrition du Sénégal. Independent Evaluation Group, Project Performance Assessment Report 110290. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ ppar-senegal-fr-06052017.PDF. ———. 2017. Solomon Islands—Rural Development Program. Independent Evaluation Group, Project Per- formance Assessment Report 110966. Washington, DC: World Bank. http://ieg.worldbankgroup. org/sites/default/files/Data/reports/ppar_solomonislands_01042017.PDF. ———. 2017. Sri Lanka—Second Community Water Supply and Sanitation Project. Independent Evaluation Group, Project Performance Assessment Report 115781. Washington, DC: World Bank. http:// ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar-srilanka-07132017.PDF. ———. 2017. Tax Revenue Mobilization: Lessons from World Bank Group Support for Tax Reform. Inde- pendent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/ default/fi es/Data/Evaluation/fi es/lp-taxrevenue-03152017.PDF. ———. 2017. Ukraine—Development of State Statistics System for Monitoring Social and Economic Trans- formation (Devstat) Project. Independent Evaluation Group, Project Performance Assessment Report 113588. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/reports/ppar-ukraine-4192017.PDF. ———. 2017. Women’s Empowerment in Rural Community-Driven Development Projects. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/Evaluation/fi es/lp_genderincdd_01272017.PDF. ———. 2017. World Bank Group Country Engagement: An Early-Stage Assessment of the Systematic Country Diagnostic and Country Partnership Framework Process and Implementation. Inde- pendent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/ default/files/Data/Evaluation/files/scd-cpf.PDF. ———. 2017. World Bank Group Joint Projects: A Review of Two Decades of Experience. Independent Evaluation Group. Washington, DC. https://ieg.worldbankgroup.org/evaluations/world-bank- group-joint-projects-review-two-decades-experience. ———. 2017. Zambia—Water Sector Performance Improvement Project. Independent Evaluation Group, Project Performance Assessment Report 111134. Washington, DC: World Bank. http://ieg.world- bankgroup.org/sites/default/files/Data/reports/ppar_zambia_01052017.PDF. 1 The Practice Group updates refl ct the relevant organizational structure of the World Bank Group as it pertained during the period under review, FY14-16. Subsequent and ongoing change in organizational structure will, as appropriate, be reflected in future editions of the RAP. 2 All data in this section are based on analysis that the Independent Evaluation Group conducted for RAP17. 3 Performance refers to rating by number of projects unless otherwise specified, in which instances we refer, explicitly, to performance by commitment volume. Independent Evaluation Group | World Bank Group 449 4 The Practice Group updates refl ct the relevant organizational structure of the World Bank Group as it pertained during the period under review, FY14-16. Subsequent and ongoing change in organizational structure will, as appropriate, be reflected in future editions of the RAP. 5 All data in this section are based on analysis conducted by Independent Evaluation Group for RAP17. 6 Performance refers to rating by number of projects unless otherwise specified, in which instances we refer, explicitly, to performance by commitment volume. 7 The Practice Group updates refl ct the relevant organizational structure of the World Bank Group as it pertained during the period under review, FY14-16. Subsequent and ongoing change in organizational structure will, as appropriate, be reflected in future editions of the RAP. 8 All data in this section are based on analysis that Independent Evaluation Group conducted for RAP17. 9 Performance refers to rating by number of projects unless otherwise specified, in which instances we refer, explicitly, to performance by commitment volume. 10 Investment project financing projects rated under Sustainable Development comprise 63 percent of all such projects rated FY14–16. 450 Results and Performance of the World Bank Group 2017 | Appendix M Appendix N. Cross-Cutting Evaluation Findings Independent Evaluation Group Note on Corporate and Thematic Evaluations and Learning Products Produced in FY17 During fiscal year (FY)17, the Independent Evaluation Group (IEG) produced several major evaluations and learning products that are not specific to any given Region of the World Bank Group or to any individual Practice Group.1 Nonetheless, these evaluations and learning products have broad relevance from a corporate or thematic perspective. Summaries of these evaluations and learning products produced by the IEG in FY17 are presented in this appendix. Corporate and Process Evaluations Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group Self-evaluation—the formal, written assessment of a project, program, or policy by an entity engaged in that activity—lies at the heart of the Bank Group’s results measurement system, and has been used to assess the outcomes of investments for 40 years. This evaluation sought to assess whether and how the Bank Group’s self-evaluation systems serve their expected purposes through a broad examination of the ways in which the systems operate, including analysis of the behaviors, incentives, and organizational culture surrounding the production and use of self-evaluation. The Bank Group’s self-evaluation systems have expanded since they started 40 years ago, and compliance with requirements is mostly strong. The systems mesh well with the independent evaluation systems for which they provide information, and the systems have been emulated and adapted by other development agencies. However, the self-evaluation systems primarily focus on results reporting and accountability needs and do not provide the information necessary to help the Bank Group transform into a “Solutions Bank” or develop learning to enhance performance. Information generated through the systems is not regularly mined for knowledge and learning except by IEG, and its use for project and portfolio performance management can be improved. The systems produce corporate results measures but need to produce value to staff and line management and to the primary beneficiaries of the Solutions Bank—client governments, implementing agencies, firms, and beneficiaries and citizens. Balance between corporate results reporting and the other uses of the systems needs to be balanced—learning and performance management. Achieving this will require changes in behaviors and attitudes so that instead of engaging with a compliance mind-set, staff use systems with more of a learning mind-set. Changes in policies, guidelines, or templates will not suf fice—culture change requires a sustained long-term effort, with aligned incentives and consistent signals and modeled behaviors from the top. Other recommendations included more adaptability and experimenting in operations, as well as better monitoring and evaluation (M&E), which the report linked to improved project outcomes. The more significant boost to operational learning is more likely to occur outside the mandatory reporting systems, and IEG therefore also recommended more voluntary self- Independent Evaluation Group | World Bank Group 451 evaluation, for example in the form of portfolio-wide retrospectives at country or Practice level, and a more systematic approach to impact evaluation. Program-for-Results: An Early-Stage Assessment of the Process and Effects of a New Lending Instrument At the request of the Board of the Bank Group, IEG undertook and early, process evaluation of the new Program-for-Results (PforR) lending instrument that was introduced in 2012. The evaluation sought to assess experience with design and implementation of PforR operations, and to identify lessons and recommendations to strengthen this new lending instrument. Overall, IEG found that the structure of the World Bank’s assessments for PforRs—technical, fiduciary, and environmental and social—has proven to be appropriate, and the assessments have generally been credible and comprehensive. The results frameworks, indicators, and program action plans are often reasonably coherent, and risks related to PforR operations have generally been well identified and assessed. However, IEG identified areas for improvement in relation to designing the programs to achieve results, and to the monitoring and reporting systems. Because none of the PforR programs has yet closed, it is too early to draw definitive conclusions about how well the instrument is performing, but IEG identified several, early insights from its assessment of the early design and implementation experience. IEG found that, while the programs focus on results more explicitly than other instruments through the introduction of disbursement-linked indicators, these indicators are not always well integrated with the results frameworks. IEG finds that both ownership and partnership are well addressed in the World Bank’s program documents, and the field visits found a considerable degree of government ownership of the programs under implementation. Both the World Bank teams and government counterparts have moved well up the learning curve for this new instrument, and countries have been eager to rely on their own financial management systems and procedures. So far, the World Bank’s average costs for the preparation of new programs have been like those of other investment policy financing operations, while average World Bank implementation costs have been significantly higher than for those operations. The fiduciary assessments have been comprehensive, and cover most of the key aspects that one would expect to find. The assessments are generally thorough in identifying relevant transparency measures such as timely provision of information to stakeholders, disclosure of tender notices and award decisions, and parliamentary oversight. PforR financing cannot normally be used for procurement packages above certain thresholds based on the type of procurement and the level of fiduciary risk. This exclusion has not been a major obstacle in the PforRs reviewed. IEG’s review has found that the allowable exclusion of high environmental and social risk activities has sometimes been interpreted in an overly cautious manner. The implementation of capacity building measures is progressing reasonably well, although with frequent delays. The Bank Group should (i) strengthen the design of the results framework and disbursement- linked indicators to ensure that the project appraisal document presents a clear line of sight to 452 Results and Performance of the World Bank Group 2017 | Appendix N developmental results; (ii) strengthen the design and monitoring of the program action plans; (iii) minimize the overly cautious interpretation of the “high” environmental and social risks exclusion; and (iv) strengthen the monitoring and reporting of results to cover the entire results framework systematically as well as the environmental and social performance of the projects. World Bank Group Country Engagement: An Early-Stage Assessment of the SCD and CPF Process and Implementation Under the new Bank Group strategy, country programs will focus on accelerating progress toward sustainably reducing poverty and building shared prosperity (the “twin goals”). To this end, the Bank Group established a new model for country engagement consisting of two instruments that is, the Systematic Country Diagnostic (SCD), and the Country Partnership Framework (CPF). Responding to a request from the Board, the IEG reviewed the processing and early implementation of the new country engagement model. The evaluation seeks to answer Board members’ request for an early look at how well the SCD/CPF process is working and identify ways to strengthen the new country engagement model as it continues to unfold. IEG found that the new SCD instrument has added considerable value to the quality of the Bank Group’s engagement with countries. The instrument has been well received inside and outside the Bank Group, including by governments and development partners. Most of the SCDs reviewed provided a strong or reasonable analytical basis for the main issues, constraints, and opportunities they identified. The identification of opportunities for economic growth was the strongest of the SCD dimensions explored. The majority of SCDs analyzed the drivers of economic growth and their impact on poverty. All SCDs identified priorities for achieving the twin goals, but clearer guidance is needed on prioritization. The SCD and CPF teams undertook broad consultations with a wide range of stakeholders. Aligning the program priorities of the CPF to the constraints and priorities identified in the SCD was challenging, and overall, the contribution of advisory services and analytics to the Bank Group program was handled unevenly in the CPFs reviewed. The SCDs identified knowledge and data gaps, but the CPFs did not address these gaps effectively. The definition of the results chains in CPFs has improved, but remains work in progress. The results chains can best be described as truncated, with several items missing. The identification of monitorable and measurable indicators for the results was a strong point in many of the CPFs reviewed. CPF preparation and review generally worked well. The corporate review process often helped to enhance the quality of the reports. The timing and processing of the new engagement model—SCDs and CPFs combined—inevitably took longer than the previous country assistance strategy preparation cycle, and the new processes clearly required additional resources. The integration of International Finance Corporation and Multilateral Investment Guarantee Agency into the CPF process has significantly improved under the new country engagement approach. Independent Evaluation Group | World Bank Group 453 The Bank Group should enhance its guidance and ensure more consistent and focused priority setting in SCDs, and more explicit discussion of strategic priorities in CPFs. The Bank Group should disseminate the SCD to further enhance its potential use by other stakeholders in the country, with specific steps to be determined by each country team. The CPFs should explicitly indicate how SCD- identified knowledge and data gaps critical to CPF objectives, will be addressed. The Bank Group should strengthen the results frameworks in CPFs to ensure they more clearly reflect the ongoing and new work programs and results of all types of interventions. The Bank Group should modify the country engagement guidance to ensure consistency with the new gender strategy. Finally, the Bank Group should improve resource management and budget transparency to strengthen “One Bank Group” participation in the new country engagement model. Thematic Evaluations Industry Competitiveness and Jobs: An Evaluation of World Bank Group Industry-Specific Support to Promote Industry Competitiveness and Its Implications for Jobs Productivity, competitiveness, and more and better jobs are key to economic development and are at the top of government development agendas. Supporting industry competitiveness has been an important part of Bank Group activities with a portfolio estimated at $21.6 billion during the 2008–14 period, corresponding to about 6 percent of all operations. In this evaluation, IEG assessed the extent to which the Bank Group has achieved its goal of helping client countries enhance the competitiveness of specific industries (agriculture, manufacturing, ICT, and tourism), with the effect of generating more and better jobs. IEG finds that the Bank Group industry support has been relevant to country priorities, country level of competitiveness, and stages of country development. The two areas of support that appear to be underemphasized are management skills and manufacturing. The World Bank’s development effectiveness is below the institutional average. The Bank Group’s industry-specific support is associated with distinctively positive improvements in competitiveness, but without observable increases in productivity compared with available alternatives. The positive association with accelerated expansion and growth in export markets may indicate that the Bank Group is contributing to accelerated expansion by alleviating industry constraints through provision of critical inputs or financing, but not by improving relative productivity. The Bank Group has not had a distinct, overarching approach to supporting industry-specific competitiveness in the last decade. Furthermore, the new organizational structure has distributed the industry competitiveness portfolio across GPs, creating a greater need for enhanced coordination across and within institutional units. The Bank Group should clarify its approach to support for industry competitiveness and given the multiple points of engagement on competitiveness, management should articulate the Bank Group’s approach in industry-specific competitiveness work and ensure a consistent treatment across the Group. 454 Results and Performance of the World Bank Group 2017 | Appendix N Support to manufacturing has declined in recent years—at a time when deindustrialization is emerging as a big problem for developing countries, particularly in Sub-Saharan Africa. The phenomenon of growing deindustrialization across the developing countries should be reflected by strengthening (in line with the SDG #9.2) the Bank Group’s industry level support to inclusive and sustainable industrialization, taking account of specific country circumstances and challenges faced by low-income countries. Employment is a central aspect of the productivity and competitiveness agenda. The evaluation illustrates the complex effects of productivity improvements on jobs, yet only a small proportion of the Bank Group portfolio specifically references jobs in project objectives, interventions, or indicators. Attention to job quality is even less common. The Bank Group should integrate the jobs perspective in its industry-specific support to competitiveness, by incorporating jobs effects in objectives, design, M&E of its interventions. Given the importance and cross-cutting challenge of employment, with multiple Bank Group units working on jobs, management should articulate the Group’s approach in this area and ensure its consistency across the Group. World Bank Group Engagement in Situations of Fragility, Conflict, and Violence IEG’s evaluation on situations of fragility, conflict, and violence (FCV) sought to assess and report on the quality and results of Bank Group programs and operations in selected middle-income countries that face situations of FCV; provide deductive analysis of the factors that lead to success or failure of Bank Group engagements; and facilitate learning to help inform the design of future strategies and assistance programs (World Bank 2016). The evaluation found that the provision of assistance can be challenging when middle-income countries experience situations of FCV as the government may have its own views on how and when to tackle the underlying issues, or may be reluctant to increase its borrowing to correct what may be seen to be a localized or temporary problem. In that regard, IEG concludes that the Bank Group’s comparative advantage is in supporting countries in tackling longer-term development challenges through early engagement, a sustained presence, as well as continuous dialogue with the parties to violent conflicts, where possible. The evaluation also found that whereas the Bank Group has been adept at adjusting its strategies and analytical support to situations of violence and conflict, its operational response has been constrained by its limited menu of instrument choices. In addition, institutional and staff incentives to engage in conflict situations and to take risks seem to lag the spirit of stated strategy. IEG found the community-drive development (CDD) model has often provided a point of entry in conflict-affected areas, and has been useful as a form of establishing a presence and contextual learning to support more ambitious efforts once peace is restored, noting, however, that CDD projects have not addressed the causes and consequences of violence. Finally, the evaluation found the Bank Group can further enhance its impact in situations of FCV by exploring opportunities beyond supporting livelihoods in conflict-affected communities, including support for private sector development, using its expertise in public financial management more effectively, and improving its M&E frameworks. Independent Evaluation Group | World Bank Group 455 Key recommendations: Based on these findings, IEG made several stratified recommendations as follows. At the strategic level, IEG recommends a review of the institutional setup and interaction of various Bank Group units dealing with FCV matters, to ensure that the FCV cross-cutting solutions area drives substantive thinking on strategy and policy across the Bank Group; development of a new multidimensional publicly disclosed system of markers and fl gs for monitoring and measuring fragility, including in non–fragile and conflict-affected countries, as well as the integration of fragility assessments as part of the SCD/CPF process; and scaling up of regional presence and outlook, including through regional projects and joint analytic and advisory work across relevant countries with sustained Bank Group presence in FCV-affected areas, monitoring the fragility profile. At the operational level, IEG recommends the development of new financial mechanisms or fast- response facilities to be used when outbreaks of violence or external threats jeopardize resilience in middle-income countries; improved strategic alignment and use of global FCV thematic trust funds, while boosting their agility; and within the new SCD/CPF model, developing a broader strategic approach to interventions in FCV-affected areas, expanding sector interventions beyond the CDD model, and improving coordination and information sharing across the Bank Group. Finally, on the subject of partnership in situations of FCV, IEG recommends developing institutional incentives for collaboration and strategic thematic guidance, particularly with the United Nations. Learning Products Learning from IDA Experience: Lessons from IEG Evaluations The International Development Association’s 14th disbursement (IDA14) introduced special themes in 2005 to reinforce the agency’s effectiveness for achieving the Millennium Development Goals. To achieve the overarching goal of maximizing development impact through a strong focus on results measurement and partnership, IDA gives high priority to gender equality, climate change, and fragile and conflict-affected situations, as well as crisis response and inclusive growth. In line with the commitment to learning from the past, this synthesis report presents findings from recent IEG evaluations and analysis to offer evaluation evidence on what has and has not worked in IDA priority areas under IDA16 and IDA17, to support the IDA18 replenishment discussions. In gender equality, the World Bank has strengthened its attention to gender issues at the corporate level, and most World Bank operations display at least some gender awareness. However, these have not necessarily translated into more coherent, outcome-focused gender integration. In climate change, World Bank management has made some progress in taking a systematic view to build client capacity to manage climate change risks and assess the impacts of interventions, albeit at a slow pace in some instances. IDA17 requirements to screen all IDA-supported projects for climate risks represent a good first step forward. In fragile and conflict-affected situations (FCS), since IDA identified support to FCS as a strategic priority, there has been demonstrable improvement in Bank Group support and effectiveness in FCS. Further improvements are being implemented, including the criteria for FCS classification, 456 Results and Performance of the World Bank Group 2017 | Appendix N interventions to address gender-based violence, and fragility assessments. However, the need remains to develop an inclusive growth and jobs strategy that takes advantage of potential synergies across the Bank Group. In crisis response, IEG evaluations highlighted the importance of long-term country engagement, analytic work in key areas, and institutional capacity building in the precrisis or predisaster period, as well as more guidance in the use of instruments and better-quality assurance procedures for crisis response operations. Adequate budgets in times of rapidly expanding programs was particularly important for IDA. Finally, in inclusive growth, IEG evaluations offer lessons in some of the areas IDA chose to focus under the special theme, including youth employment, financial inclusion and support for small and medium enterprises, and agriculture and natural resource management. The overall results are mixed, with significant contribution to expanding the reach and sustainability of the microfinance industry, but few examples of successful interventions in the other areas. World Bank Group Joint Projects: A Review of Two Decades of Experience—Lessons and Implications from Evaluation The learning note on Bank Group joint projects represents IEG’s first systematic stocktaking of joint projects. It provides insights into the benefits of, and challenges in, developing, structuring, supervising, monitoring, and evaluating joint projects. The review identifies the core characteristics of these projects, including the quality and nature of their “jointness”; the push-pull factors at play; their key value-added; how jointness is factored in throughout the project lifecycle; and the enablers or inhibitors of successful Bank Group joint projects. The review found a lack of definitional clarity regarding joint projects, which continue to be relatively scarce across the portfolio. Joint projects fall into cofinancing, parallel, or sequential categories, though these attributes are not clearly identified in the respective project portfolio databases. IEG’s review focused on joint projects cofinanced by two or by all three Bank Group institutions. This category represents, perhaps, the purest form of joint project. Even with this focus, IEG found current project evaluation systems limit the degree to which jointness can be assessed, as each of the institutions independently assesses development effectiveness and work quality from their vantage point, and through their respective, separate systems. Allowing for these limitations, IEG established that joint projects achieved a reduction of risk in client investments across a range of high-risk countries, and facilitated pioneering foreign direct investment. The review also established that information sharing remains a hurdle for the knowledge exchange crucial to successful jointness, and that conflict-of-interest issues surface. Differences and incompatibility in processes, incentives, and organizational cultures persist. However, efforts to streamline processes, standardize transaction documents, and train Bank Group staff are in train to address relevant coordination, policy, and resource challenges. IEG’s review notes that joint projects do not represent a panacea, and realism must temper expectations regarding their ongoing value and application. Such projects require the willingness of public or private sector clients to procure products or services from two or more Bank Group Independent Evaluation Group | World Bank Group 457 institutions. Implementation timelines, different institutional processes, and the adequacy and stability of financial and staff resources present challenges. However, project jointness can be especially helpful in specific contexts with high direct relevance to the Bank Group’s development mission, and can represent powerful and creative tools for development. Any increased use of joint projects implies the need to develop and test approaches for evaluating them, especially from a One Bank Group perspective. Finally, recognizing work on joint projects in staff performance reviews can reinforce staff incentives to work collaboratively across the different institutions. The Role of Political Economy Analysis in Development Policy Operations IEG’s learning product on the role of political economy analysis in development policy operation (DPOs) reviewed the extent to which political economy analysis (PEA) is used to improve the design of DPOs and how effective it has been (World Bank 2016a). The review found that PEA can improve the design of DPOs by identifying implementation risks and mitigating actions—the lack of PEA to support politically sensitive and dif ficult actions tend to reduce the effectiveness of operations. PEA can be used to: support the assessment of the eligibility of the client countries for budget support; provide important insights on the political feasibility of specific reforms supported by DPOs; inform specific design elements, including choices of programmatic instruments versus. stand-alone operations, or front-loading versus. back-loading of important reform actions in a programmatic series; and as part of the self-evaluation process, to support enhanced analysis of factors affecting program effectiveness. The review found that many World Bank analytical products as well as project or program documents contain PEA relevant for DPOs, for example, in studies, such as institutional and governance reviews, country economic memoranda, poverty and social impact assessments, and public expenditure reviews. Risk assessments in operational documents often draw on PEA, and can be useful in revealing the extent to which political economy perspectives have been applied through formal or informal analysis. More recently, SCDs have created new opportunities for assessment of the overall policy environment and political feasibility of specific reforms. The learning product also found the political feasibility of specific reforms supported by DPOs may not receive adequate attention, although DPO teams are generally well informed about the overall policy environment and more general political economy issues. The under-provision of more specific political economy assessment may be explained, for example by the reliance on national development strategies to signal ownership and political feasibility of reform programs, whereas evaluative evidence suggests a need for PEA specific to politically sensitive reforms. More broadly, the review finds the under-provision of PEA could be a symptom of broader issues concerning the results orientation of DPOs with respect to specific reforms. The key metrics of results orientation in DPOs is the extent to which DPOs promote policy and institutional change in specific reform areas as reflected in the policy matrix of the instrument, but other aspects outside the specific policy matrix (for example, maintaining the World Bank’s relevance in the policy dialogue on macrofiscal frameworks, sector policies, and poverty reduction) are also important determinants 458 Results and Performance of the World Bank Group 2017 | Appendix N of the quality of the World Bank’s engagement through DPOs. These “reforming” and “financing” elements of budget support can reinforce each other if World Bank and client priorities are strongly aligned. But there may be trade-offs between the two when there is a lack of alignment, and this can manifest itself in a weaker focus on specific reforms, and an associated under-provision of PEA. Better prioritization of reforms supported by the World Bank, and risk assessment of specific reform actions may strengthen the results orientation in development policy financing with respect to policy and institutional change and increase demand for PEA. Stronger integration of development policy financing with SCDs and CPFs that are underpinned by solid technical and PEA to identify the most critical reforms and to monitor achievement of results can be a way to strengthen the reform pillar of the instrument. In addition, individual risk assessment for all reform actions supported by DPOs may enhance the World Bank’s knowledge about the theory of change in those reform areas and contribute to more informed risk taking. Tax Revenue Mobilization: Lessons from Bank Group Support for Tax Reform The mobilization of domestic resources through reforms in taxation is essential to ensuring sustainable financing of development. Several international initiatives are under way that focus attention on constraints to growth and development, particularly in low-income economies, where domestic taxes and foreign private and market-related borrowing do not expand enough to compensate for declining flows of of ficial development assistance. This report reviews Bank Group support to tax policy and administration reform over FY2005–15, covering DPOs, investment projects, and International Finance Corporation Advisory Services on business taxation. It focused on describing Bank Group support to tax reforms, and on questions related to development effectiveness. Over FY05–15, most World Bank support to tax policy and administration reform (205 projects in 107 countries; total commitments of $28.4 billion) has been provided through programmatic DPOs, with investment projects representing 11 percent of the total. For most of the operations bearing the tax theme code, the tax reform component was a minor part of the operation, predominantly in the 10–14 percent range. Most tax components in World Bank operations were designed to enhance revenue to enable fiscal consolidation or to create or maintain fiscal space for priority expenditure. Some also aimed at improving investment climate or strengthening export competitiveness and improving trade integration. Some success was achieved in increasing revenue through tax administration reform. For instance, Guatemala created a new tax administration after conflict and tax revenue increased from 8.8 percent of gross domestic product in 1998 to 9.8 percent in 2001 and 10.3 percent in 2006. Mozambique improved the ef ficiency of its tax administration, and then strengthened tax legislation, including the fiscal regime for mining and oil sectors. Paraguay improved its tax of fice capacity to audit large taxpayers. On the other hand, little was done in tax policy reforms and with limited success. Tax policy measures aimed at broadening the tax base primarily by controlling tax exemptions. For Independent Evaluation Group | World Bank Group 459 example, in Peru, limited progress was made in eliminating tax exemptions, and in Jamaica, progress in improving uniformity of the tax code was limited. In addition, the review highlights several missed opportunities. Major structural weaknesses of country tax systems were rarely addressed, even in DPO series (for example, Colombia, Guatemala, and Indonesia). Little analytical or diagnostic work was conducted to inform operation design, although this work may have been done by the International Monetary Fund, especially when they had a parallel program. Synergies between DPOs and technical assistance projects were only observed in Guatemala, São Tomé & Príncipe, Pakistan. It is important to ensure and sustain government ownership, to sustain engagement to correct structural and systemic tax issues, and better monitor and evaluate the impact of tax reforms. At a more strategic level, it matters to address other pervasive governance issues (for example, corruption and weak judiciary system), to better exploit synergies between policy operations and technical assistance projects (both in terms of timing and content), to cultivate in-house capacity to do diagnostic and advisory work, and to increase ef ficiency and equity while increasing tax revenue. Women’s Empowerment in Rural Community-Driven Development Projects This report analyzes gender features of long-standing World Bank rural CDD projects, their response to women’s specific constraints, and their impacts on women’s economic, political, and social empowerment. The report was designed to help practitioners who implement CDD interventions more explicitly define, discuss, and integrate gender-relevant elements in the design of CDD projects; be more effective in implementing and monitoring features that may affect men and women differently; and identify meaningful indicators and information to assess gender impacts. The repor t finds that World Bank–supported CDD projects actively pursue and generate high female participation. CDD projects aim to empower communities, particularly excluded groups. Women are increasingly recognized as a group that CDD interventions need to reach out to with specifically designed activities to ensure that they fully benefit. The report highlights the importance of understanding the quality and implications of women’s participation to assess the empowerment impacts of CDD. CDD projects have the potential of enabling women to overcome some of the constraints that they face in the rural space. If these constraints are ignored, CDD projects can have the undesirable effect of leaving women behind and exacerbating gender gaps. Thus, the report stresses the importance of carrying out gender-specific needs assessments to inform project design. Moreover, the report highlights the importance of explicitly emphasizing empowerment in the results chain of CDD projects. Empowerment is a key element of the CDD approach. Yet, CDD projects rarely systematically discuss the dimensions of empowerment (economic, political, or social) they aim to affect, directly or indirectly, and how. Recognizing that empowerment is both a process and an outcome can help CDD interventions to better frame what they can impact both in the short and the long term. The repor t finds that immediate impacts of CDD projects are found in the inclusion of women, and on some of the economic benefits from the project’s activities. Positive impacts are documented for outputs such as participation and engagement, and in some cases for intermediate outcomes such 460 Results and Performance of the World Bank Group 2017 | Appendix N as social capital and women’s increased confidence. These results are mostly restricted to the CDD project sphere. Very little information is found on impacts of CDD on the broader economic, political, and social empowerment of women. The report concludes that by tracking outcomes in addition to outputs (such as the number of subprojects proposed by women or the number of women who received credit or training) and through more systematic assessment, reporting and evaluation, the learning of what works to increase women’s empowerment through CDD projects can be greatly enhanced. List of Summarized Works World Bank. 2016. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/ sites/default/files/Data/Evaluation/files/behindthemirror_0716.pdf. ———. 2016. Industry Competitiveness and Jobs: An Evaluation of World Bank Group Industry-Specific Support to Promote Industry Competitiveness and Its Implications for Jobs. Independent Evalua- tion Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/ Evaluation/files/industry-competitiveness-and-jobs.pdf. ———. 2016. Learning from IDA Experience: Lessons from IEG Evaluations. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ lp_ida_0716.pdf. ———. 2016. The Role of Political Economy Analysis in Development Policy Operations. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/reports/lp_poleconanalysis_0716.pdf. ———. 2016. World Bank Group Engagement in Situations of Fragility, Conflict, and Violence. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/Evaluation/fi es/fcv-full.pdf, ———. 2017. Program-for-Results: An Early-Stage Assessment of the Process and Effects of a New Lend- ing Instrument. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.world- bankgroup.org/sites/default/files/Data/Evaluation/files/program-for-results-full.pdf. ———. 2017. Tax Revenue Mobilization: Lessons from World Bank Group Support for Tax Reform. Inde- pendent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/ default/fi es/Data/Evaluation/fi es/lp-taxrevenue-03152017.pdf. ———. 2017. Women’s Empowerment in Rural Community-Driven Development Projects. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/ Data/Evaluation/files/lp_genderincdd_01272017.pdf. ———. 2017. World Bank Group Country Engagement: An Early-Stage Assessment of the Systematic Country Diagnostic and Country Partnership Framework Process and Implementation. Inde- Independent Evaluation Group | World Bank Group 461 pendent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/ default/fi es/Data/Evaluation/fi es/scd-cpf.pdf. ———. 2017. World Bank Group Joint Projects: A Review of Two Decades of Experience. Independent Evaluation Group. Washington, DC. https://ieg.worldbankgroup.org/evaluations/world-bank- group-joint-projects-review-two-decades-experience. 1 Several major Independent Evaluation Group evaluations and learning products produced in FY17 are specific to a Region or Global Practice, and these are included in relevant Region or Practice Group updates. 462 Results and Performance of the World Bank Group 2017 | Appendix N The World Bank 1818 H Street NW Washington, DC 20433