VOLUME I M AIN R EPOR T Results and Performance of the World Bank Group 2014 A N I N D E PE N D E N T E VA LUATI O N Results and Performance of the World Bank Group 2014 An Independent Evaluation Contents ABBREVIATIONS ................................................................................................................................. III  ACKNOWLEDGMENTS ......................................................................................................................... V  OVERVIEW ........................................................................................................................................... VII  MANAGEMENT COMMENTS ............................................................................................................... XI  1.  THE WORLD BANK GROUP AND THE MDGS ....................................................................... 1  Integrating the MDGs into its Strategies while pursuing a Broader Approach to Poverty Reduction ........................ 3  Providing Key Intellectual Underpinnings to Development Thinking ......................................................................... 8  Translating the MDGs into Country Programs through Country-Based Model ....................................................... 10  Engaging Collectively for Achieving the MDGs ....................................................................................................... 13  Articulating a Results Chain for the World Bank Group’s Role, Approach, and Expected Contribution to the MDGs ................................................................................................................................................................................ 16  Key Messages for the Bank Group’s Engagement with the Post-2015 Agenda ..................................................... 17  References .............................................................................................................................................................. 18  2.  WORLD BANK GROUP OPERATIONS ................................................................................. 23  World Bank Group Commitments Rose .................................................................................................................. 23  Performance of Bank Lending and IFC Investments Continued to Decline ............................................................ 24  DEVELOPMENT OUTCOMES OF BANK-FINANCED PROJECTS DECLINED ............................................................................................. 24  DEVELOPMENT OUTCOMES OF IFC INVESTMENT PROJECTS DECLINED SHARPLY......................................................................... 26  DEVELOPMENT EFFECTIVENESS OF IFC ADVISORY SERVICES GREW STRONGER IN 2014 ............................................................ 29  MIGA GUARANTEES CONTINUED TO ACHIEVE RELATIVELY STRONG DEVELOPMENT OUTCOMES .................................................. 30  Bank and IFC Investment Portfolio Performance can be Improved by Addressing some Work Quality Issues...... 32  BANK PERFORMANCE IS DRIVEN BY WORK QUALITY FACTORS ................................................................................................... 32  UP-FRONT WORK QUALITY IS THE MOST POWERFUL DRIVER OF IFC INVESTMENT PROJECT SUCCESS ......................................... 39  Performance of World Bank Group Country Programs Improved ........................................................................... 43  PERFORMANCE OF BANK GROUP COUNTRY PROGRAMS HAS IMPROVED SINCE 2010 .................................................................. 44  FURTHER IMPROVEMENT CAN BE ACHIEVED BY STRENGTHENING SELECTIVITY OF BANK GROUP COUNTRY STRATEGIES BASED ON CLEAR RESULTS CHAINS......................................................................................................................................................... 45  The Bank Group Implements IEG Recommendations over time ............................................................................ 48  References .............................................................................................................................................................. 49  Boxes Box 1.1. Evolution of the Millennium Development Goals and World Bank Group Response ................. 2  Box 1.2. World Bank Selectivity Framework............................................................................................ 5  Box 1.3. Role of the Bank in Developing the MDG Indicators ................................................................. 8  Box 1.4. Support of the United Nations Development Program and Asian Development Bank for the MDGs .................................................................................................................................................... 13  Box 2.1. IFC Profitability is Only One Part of IFC’s Business Model, Though an Important Part .......... 28  i CONTENTS Tables Table 1.1. Country Coverage in Primary Education, Health, Water, and Sanitation Portfolio, FY 02–04 7  Table 1.2. Risk to Development Outcomes in Primary Education, Health, and Water and Sanitation Projects................................................................................................................................................... 8  Table 2.1. Frequently Identified Work Quality Factors ...........................................................................40  Table 2.2. Frequently Identified Factors Associated with IFC’s Screening, Appraisal, and Structuring Rating ....................................................................................................................................................41  Table 2.3. From Risk Identification to Risk Mitigation ............................................................................41  Table 2.4. Role of Client Contact for IFC Supervision Quality ...............................................................42  Table 2.5. Examples of Different Supervision Practices on Development and Investment Outcomes ...42  Table 2.6. Positive IFC Engagement on Environmental and Social Issues ...........................................43  Table 2.7. Relatively Strong Results Measurement but Weak Results Chains in CPS Results Frameworks ...........................................................................................................................................47  Figures Figure 1.1. Progress Toward Achieving the MDGs (percent).................................................................. 1  Figure 1.2. MDG-Related Bank Commitments and Composition ............................................................ 6  Figure 1.3. MDG-Related AAA and Composition .................................................................................... 9  Figure 1.4. MDG-Related Indicators in Nine Country Strategies, FY97–14 (percent) ............................12  Figure 1.5. MDG-Related Commitments in Nine Countries (percent) ....................................................12  Figure 1.6. Trust Funds Support for MDG-Related Sectors, FY02–14 ..................................................14  Figure 2.1. World Bank Group Financing, FY01–14 ..............................................................................23  Figure 2.2. Bank Lending Success Rate, FY00–13 ...............................................................................25  Figure 2.3. Bank Lending Success Rate by Country Group and Region, FY08–13 ...............................25  Figure 2.4. Bank Lending Success Rate by Global Practice, FY08–13 .................................................26  Figure 2.5. IFC Investments Success Rate, FY00–13 ...........................................................................27  Figure 2.6. IFC Advisory Services Success Rate, FY08–13 (percent)...................................................30  Figure 2.7. MIGA Development Outcome Success Rate by Sector, FY08–13 (percent) .......................31  Figure 2.8. Bank and IFC Performance Success Rate, FY00–13 ..........................................................32  Figure 2.9. Factors Influencing Quality at Entry Ratings ........................................................................34  Figure 2.10. Factors Influencing Quality of Supervision Ratings............................................................37  Figure 2.11. Projects with Low Up-Front Work Quality Approved During a Period of Rapid Change ....39  Figure 2.12. Success Rates of World Bank Group Country Strategies, FY05–14 (percent) ..................44  Figure 2.13. Changes in the Number of Objectives in Consecutive CPSs, FY05–14 ............................46  Figure 2.14. Adoption of IEG Recommendations (percent) ...................................................................48 ATTACHMENT: REGIONAL UPDATES The Appendixes are available online at: https://ieg.worldbankgroup.org/Data/reports/rap2014-app.pdf Evaluation Managers  Caroline Heider Director-General, Evaluation  Nick York Director, IEGCC  Geeta Batra Manager, IEGCC  Xiaolun Sun Task Manager ii Abbreviations A2F Access to Finance AAA Analytical and Advisory Activities ADB Asian Development Bank AMS Agribusiness, Manufacturing, and Services (MIGA) CAS Country Assistance Strategy CPE Country Program Evaluation CPS Country Partnership Strategy CPSCR Country Partnership Strategy Completion Report DAC Development Assistance Committee DPF Development Policy Financing E&S Environmental and social FCS Fragile and Conflict-affected States FIF Financial Intermediary Funds FTI Fast Track Initiative FY Fiscal Year GRPP Global and Regional Partnership Program HIPC Heavily Indebted Poor Country HNP Health, Nutrition, and Population IBRD International Bank for Reconstruction and Development IC Investment Climate ICR Implementation Completion Report IDA International Development Association IDG International Development Goal IEG Independent Evaluation Group IFC International Finance Corporation IPF Investment Project Financing LIC Low-Income Country M&E Monitoring and Evaluation MAR Management Action Record MAS Manufacturing, Agribusiness, and Services (IFC) MCH Maternal and Child Health MDGs Millennium Development Goals MIC Middle-Income Country MIGA Multilateral Investment Guarantee Agency MS Moderately Satisfactory NLTA Non-Lending Technical Assistance iii ABBREVIATIONS OECD Organization for Economic Co-operation and Development PPAR Project Performance Assessment Report PPP Public-Private Partnership PRS Poverty Reduction Strategy PRSC Poverty Reduction Support Credit PRSP Poverty Reduction Strategy Paper RAP Results and Performance of the World Bank Group SBA Sustainable Business Advisory SIP Small Investment Program SDGs Sustainable Development Goals UNDP United Nations Development Program WSS Water Supply and Sanitation All dollar amounts are in U.S. dollars unless otherwise indicated. iv Acknowledgments This report was prepared by a team led by Xiaolun Sun. The analyses were carried out under the guidance of Nick York (Director) and Geeta Batra (Manager), as well as the direction of Caroline Heider (Director-General, Evaluation). Sidney Edelmann coordinated the analyses related to the International Finance Cooperation and Multilateral Investment Guarantee Agency. The team consisted of Maria Ann Baldauf, Nils Fostvedt, Cecilia Jade Kern, Xue Li, Eduardo Fernandez Maldonado, Albert Martinez, Kristin Panier, Carla Lizette Pazce, Swizen Rubbani, Luis Alvaro Sanchez, Anthony Martin Tyrrell, and Yoshine Uchimura. Jacqueline Andrieu, Samia Ausaf, Ken Chomitz, Katia Ondina Diaz Morejon, Hiroyuki Hatashima, and Alex McKenzie provided additional data support. Six Independent Evaluation Group (IEG) Reviews of Regional Updates were prepared concurrently with this report by Dinara Akhmetova, Ismail Arslan, Konstantin Atanesyan, Florence Charlier, Rasmus Heltberg, Takatoshi Kamezawa, Jiro Tominaga, Yoshine Uchimura, Rene Vandendries, and Clay Wescott. Peer reviewers were Amitava Banerjee (former Senior Manager, IEG Private Sector Evaluation) and Homi Kharas (Senior Fellow and Deputy Director, Global Economy and Development program at the Brookings Institution, and former chief economist in the East Asia and Pacific Region). IEG management and colleagues provided helpful guidance and comments, including Marie Gaarder, Emmanuel Jimenez, Midori Makino, Christopher Nelson, Barbara Rice, Pia Schneider, Andrew Stone, Mark Sundberg, William Sutton, Marvin Taylor-Dormond, Stoyan Tenev, Cheryl Marie Toksoz, and Stephan Wegner. Carla F. Chacaltana provided administrative support to the team and, together with Yasmin Angeles and Emelda Cudilla, assisted with report production. v Overview The year 2015 is pivotal in international in development. Its engagement with the development. As the era of the Millennium MDGs highlights several key strengths and Development Goals (MDGs) concludes, work some weaknesses. continues to take forward the unfinished MDG agenda and to define and meet the The Bank Group integrated the MDGs in its more ambitious Sustainable Development strategies while pursuing a broader approach Goals (SDGs). In preparation for the new to poverty reduction. Its approach to challenges, the World Bank Group is supporting the MDGs was to implement the changing. A new strategy in 2013 set two twin pillar strategy by strengthening the ambitious goals: ending extreme poverty by country focus and utilizing the 2030 and boosting shared prosperity for the Comprehensive Development Framework. In bottom 40 percent in a sustainable manner. time, there was greater emphasis on To achieve the twin goals and contribute institutions and governance, global public significantly to the SDGs, it is important to goods, results management, enhanced reflect on the Bank Group’s experience with instruments, domestic resource mobilization, the MDGs to draw lessons for its engagement and expanded analytic work and evaluation with the post-2015 agenda. This report also evidence. International Finance Corporation reviews the effectiveness of the Bank Group’s (IFC) and Multilateral Investment Guarantee portfolio and country programs for delivering Agency (MIGA) strategies did not target the results, and its implementation of MDGs, but sought to contribute to growth by Independent Evaluation Group (IEG) promoting private investment. This holistic recommendations. approach to development was validated at the Busan Aid Effectiveness Forum and The World Bank Group responded to the incorporated into the post-2015 agenda. MDGs in many ways, but there were gaps and lessons The Bank’s analytical work influenced global development thinking and its own strategies The Bank Group clearly saw the value added and approaches. Since 1990 several World and opportunities in the MDGs, but also the Development Reports advanced the thinking risks. In the decade before the Millennium on poverty reduction and achieving the Declaration, the Bank Group started to focus MDGs. They also underpinned many Bank on poverty reduction, with private sector–led Group corporate and sector strategies. growth and social and human development as Drawing from its strong statistical expertise, the twin pillars of its strategies. Despite the Bank played a key role in defining the perceptions that the Bank Group resisted the MDG targets and indicators in 2001, and MDGs, it endorsed them in 2001. The Bank promoted the managing for results agenda. Group saw the MDGs as both a results-based Starting in 2004, the Bank, with the framework for assessing the impact of its International Monetary Fund (IMF), activities and an opportunity for alignment produced the Global Monitoring Reports to with partners and countries. At the same time, provide analysis of the policies and actions the Bank Group recognized that the MDGs needed for achieving the MDGs and related did not sufficiently focus on sustainability, development outcomes. The Bank Group is institutions, and the role of the private sector the largest producer of impact evaluations vii OVERVIEW (IEs), although a recent IEG evaluation found not have the metrics for assessing that the feedback loop between IE production partnerships, trust funded or not. and Bank operations and learning has been Independent reviews and IEG evaluations of weak. global funds and global and regional partnerships identified a number of issues, The Bank’s country-based model was crucial including poor integration of the issue- for translating the MDGs into country focused partnership programs with the Bank’s programs, but it needs strengthening. In country-based model. endorsing the MDGs, the Bank Group emphasized the primacy of country ownership The Bank Group strengthened its results for achieving the global goals. Its country orientation, but has not articulated a clear strategies have sought to link the Bank results chain for the MDGs. As part of the Group’s institutional objectives with the institutional effectiveness reforms in the late countries’ own priorities. This heightened 1990s, a new system of performance country focus was supported by indicators for client responsiveness and organizational changes and country drivers of project outcomes was introduced. instruments and processes that were initiated Results-based country strategies were in the late 1990s. Under the Bank’s country- mainstreamed in 2005, and results-based based model, the MDG themes were financing is now widely used in many sectors. incorporated substantially in Bank Group In 2010 IFC piloted the IFC Development country programs. However, recent IEG Goals in an effort to use development goals evaluation found that although the country- to drive strategies and operational decision based model improved the Bank Group’s making. Since 2011 the Bank’s Corporate responsiveness to client needs, it made it Scorecard includes MDGs in Tier 1 indicators difficult to integrate sector and corporate (and water and sanitation MDG indicators in strategies with countries strategies. Tier 2). However, the Bank Group did not have a clearly articulated results chain for the The Bank Group engaged in partnerships at MDGs, or a systematic process for assessing several levels to support the MDGs, but there its contribution to the global goals. This is much room for improvement. The Bank deficiency created ambiguity internally and Group recognized early on the unprecedented externally about the expected roles, opportunity the MDGs presented for aligning approaches, and contributions of the Bank partner efforts to improve the effectiveness of Group. It also created difficulties in poverty reduction. Its selectivity framework identifying specific MDG areas that might specified the principles for working with need special attention. partner organizations. Trust funds expanded, bringing with it both opportunities as well as The World Bank Group has to address some challenges. The Bank Group played a clear long-standing work quality issues to realize role in getting governments and partners its Solutions Bank ambitions aligned through its programs and the Poverty The Bank Group commitments increased Reduction Strategy Papers and Heavily amid significant organizational changes, but Indebted Poor Country Initiative with the development outcomes of its lending IMF, but there is no indication that the portfolio continued to decline. In fiscal year MDGs were key drivers in this regard. (FY) 14 Bank Group commitments reached Internally, despite the Bank Group strategic $61 billion, consisting of an increase of almost focus on private sector–led growth, 30 percent in Bank lending to $40.8 billion, a cooperation among the Bank, IFC, and 5 percent decline in IFC investments to $17.3 MIGA remains ad hoc. The Bank Group does viii OVERVIEW billion (excluding mobilization of $5.1 billion), important, but to a lesser extent. For IFC, and a 14 percent increase in MIGA guarantees work quality goes beyond risk identification— to $3.2 billion. However, continuing a long- that is, identification and assessment of risks term, steady downward trend, performance of during project appraisal is not sufficient Bank-financed operations and IFC without risk mitigation. investments deteriorated further when comparing projects evaluated in FY11–13 Further improvement in Bank Group country with those in FY08–10. Portfolio program performance requires addressing the performance fell the most in International long-standing issues of poor strategic Bank for Reconstruction and Development selectivity and weak results chains. IEG (IBRD) countries and the East Asia and research found that although the Bank Group Pacific Region (the Bank only), which were paid significant and increasing attention to the good performers in the past. The lack of a selectivity, there is still a lack of common fully functioning self-evaluation system understanding of what defines selectivity and prevents continuous tracking and assessment how to achieve it. Efforts at sharpening the Bank’s knowledge activities’ impacts. selectivity in Bank Group country programs have not led to clear progress. Poorly By contrast, performance of Bank Group articulated results chains that do not clearly country programs and IFC advisory services identify the underlying assumptions and projects grew stronger in FY14, with their transmission channels make it difficult to success rates meeting the respective corporate assess whether the planned interventions can targets for the first time since 2009. MIGA deliver on the proposed objectives. Because projects also realized a relatively strong the tendency is to underestimate the efforts success rate. required for delivering results, deficiencies in the results chains deprives the Bank Group of Some behavior and process factors are key a mechanism to compare the realism of drivers of Bank Group work quality at entry, proposed objectives, leading to selection of during supervision and, ultimately, of project too many engagement areas. development outcomes. Previous work by IEG and management demonstrated a strong The Management Action Record shows that statistical correlation between the quality of the Bank Group increasingly implements IEG work by Bank Group teams at project recommendations. The recommendations preparation and implementation, and the related to the sector or theme being evaluated, ultimate development outcome of projects. but roughly one of every five For both the Bank and IFC, poor work recommendations included a reference to quality was driven mainly by inadequate M&E, pointing to a common challenge across quality at entry, underscoring the importance the Bank Group. The Bank Group reported of getting things right from the outset. For the on the various steps it took to address M&E Bank, applying past lessons at entry, effective weaknesses in response to the risk mitigation, sound monitoring and recommendations, but the concern persisted. evaluation (M&E) design, and appropriate objectives and results frameworks are The World Bank Group can make a strong powerful attributes of well-designed projects. contribution to the post-2015 agenda by During project implementation, the providing incentives and organizational propensity to problem solving is the dominant support for excellence feature of good supervision. Other factors The Bank Group responded appropriately to such as the regularity of missions and the MDGs, but the declining performance of corrective actions at mid-term are also ix OVERVIEW its portfolio raises questions about the Management is aware of these issues, but the effectiveness of its support. Performance of impact of the recent initiatives remains to be MDG-related commitments was on par with seen. Many of the ongoing reforms are geared Bank averages, with many projects facing toward resolving the underlying problems. significant or high risks to development For example, in response to acknowledged outcomes because of issues of financial shortcomings in M&E systems, the Results, sustainability and institutional weaknesses. Measurement, and Evidence Stream network IEG analysis shows that it is possible to was created and dedicated resources were enhance Bank Group portfolio performance made available to strengthen the institutional by paying closer attention to specific factors at skills for M&E. Given the low levels of M&E entry and during supervision. Focus group performance, these initiatives are welcome discussions with staff, however, suggest that and should be pursued vigorously. Overall, incentives for improving work quality—if they however, the ambitious, prolonged change exist—are not well-communicated. There process within the Bank Group has yet to seems to be relatively low awareness, even produce the desired institutional clarity, disbelief that high work quality by one streamlined processes, and enabling individual can contribute to better environment that motivate staff to strive for development outcomes. The need for excellence. Past experience with large developing some soft and hard project organizational changes suggests that these management skills and competencies among problems are common, but must be dealt with staff is also highlighted. decisively as the change process is completed. x Management Comments World Bank Group management welcomes the Independent Evaluation Group (IEG) report Results and Performance (RAP) of the World Bank Group 2014 and the opportunity to respond with comments. The RAP reflects on the Bank Group’s experience with the Millennium Development Goals (MDGs) and draws lessons for its engagement with the post-2015 Sustainable Development Goals (SDGs) agenda. Further, it considers the Bank Group’s work towards achieving its goals of ending extreme poverty by 2030 and promoting shared prosperity for the bottom 40 percent in a sustainable manner. World Bank Group management acknowledges IEG’s recognition of its role in the achievement of the MDGs and its work in helping countries achieve them. Management appreciates the three main lessons of the RAP 2014, namely: (a) lessons from the response to the MDGs; (b) incentives and organizational support to strengthen contributions to the post-2015 agenda; and (c) addressing long-standing work quality issues. World Bank Group management is also pleased with the recognition of its efforts to implement IEG’s recommendations (83 percent by the fourth year of implementation). Early on, the World Bank Group recognized the importance of the post-2015 development agenda. Senior management created a Special Envoy office attached to the President’s office responsible for coordinating the Bank Group’s engagement with the United Nations (UN). The primary focus has been to support the achievement of the remaining MDG agenda issues. This office has also helped advance the new agenda by: (a) exploring how the World Bank Group, in collaboration with other multilateral development banks (MDBs) and the International Monetary Fund (IMF), can leverage its financial resources for development; (b) helping meet the need for data to inform development policy and monitor and evaluate progress toward the goals and targets; and (c) preparing for implementation with an aim to develop integrated solutions. Consequently, the Bank Group has played an active role in the various fora to help the UN shape the new agenda. Suggested Areas for Attention In regards to the Bank Group’s response to the MDGs, IEG has identified five areas for attention: (a) integration of the MDGs into strategies while pursuing a broader approach to poverty reduction; (b) provision of key intellectual underpinnings that support the MDGs through the Bank’s knowledge work; (c) the role of the country- based model in translating the MDGs into country programs; d) collective xi MANAGEMENT COMMENTS engagement to support the MDGs; and (e) the need for a clear articulation of a results chain for understanding and assessing the Bank Group’s role, approach, and expected contribution to the MDGs. Integration of MDGs into Bank Group strategies while pursuing a broader approach to poverty reduction. Management appreciates the relatively broad view of the Bank Group’s experience with the MDGs. The report articulates well the overall alignment between operational engagements and the MDGs. At the same time, it recognizes that achievement of the MDGs requires increased country resources; for example, through economic growth, and improved capacity to manage those resources and deliver MDG-related services. The report also emphasizes the achievements of the MDGs and the Bank’s contributions towards the goals, while highlighting some key areas for improvement as the Bank Group shifts towards the post-2015 agenda. The report has recognized the Bank Group’s partnership efforts, including the Poverty Reduction Strategy Papers (PRSPs) and Heavily Indebted Poor Country (HIPC) Initiative with the IMF, which pre-dated the MDGs but were adapted to support the realization of the goals. In 2005, the HIPC initiative was supplemented by the Multilateral Debt Relief Initiative and was an explicit motivation to help accelerate progress toward the MDGs. In addition, HIPC focused on boosting social spending linked to MDG-focused sectors such as agriculture, health, and education. An IMF study concluded that following the HIPC Initiative, eligible countries have substantially increased their expenditures on health, education, and other social services. Role of the World Bank Group’s country-based model in translating the global MDGs into country programs. Management acknowledges IEG’s recognition that MDG themes have been substantially incorporated in Bank Group country programs. The integration of the MDGs helped governments, in cooperation with donors and multilateral organizations, link the global goals to local priorities in country strategies and results frameworks. Country needs and priorities will always be the key drivers of Bank Group country partnerships. Management believes, therefore, that the measures it has taken to support the achievement of the goals in each country-specific context are sufficient. Management recognizes the need to utilize the country-based model to bring global perspectives into country programs, and welcomes the attention to this issue. The new operating model is designed to better link and disseminate global and local knowledge—within and across sectors. The post-2015 development agenda is anticipated to be more comprehensive and inclusive, with additional topics not on the MDG agenda, such as infrastructure, inclusiveness, and governance. xii MANAGEMENT COMMENTS In response to IEG comments on the need to improve strategic selectivity in Bank Group country programs, in FY15, the World Bank Group introduced a new approach to country engagement. Under the new approach, Country Partnership Frameworks (CPFs) will continue to be based upon a country’s own poverty-focused development program. All CPFs will be underpinned by a Systematic Country Diagnostic (SCD), which provides rigorous analysis to identify, in a strategic and selective manner, the key opportunities and constraints to achieving the goals of eliminating extreme poverty and increasing shared prosperity in a sustainable manner. CPF objectives are derived from those country development goals that align with the findings of the SCD and also fall under the Bank Group’s comparative advantage. As the MDGs and now the SDGs are consistent with the focus on the goals, they will be an integral part of World Bank Group country strategies. Management agrees with the report’s observation that the Bank Group should have better articulated a “results chain for the MDGs,” which must be defined in the context of country development plans (e.g., PRSP for low-income countries). Performance of MDGs. Management acknowledges that the performance of MDG- related commitments were consistent with Bank averages, in spite of significant and high risks. Support to MDGs through knowledge work. Management appreciates the recognition by the IEG report that the Bank’s analytical work has a strong influence on global development thinking and has provided strong and growing support to the MDGs. Ad hoc cooperation. The report states that despite the Bank Group’s strategic focus on private sector-led growth, cooperation among the Bank, International Finance Corporation (IFC), and Multilateral Investment Guarantee Agency (MIGA) remains ad hoc. The above new approach to country engagement is a Bank Group-wide program, which aims to enhance the systematic approach to country engagement across the Bank Group. This is supported by organizational changes, such as the Global Practices (GPs) and the Re-Focused IFC, which provide enhanced opportunities for cooperation. In addition, the IEG assessment also overlooks the formal IFC–MIGA Business Development Partnership that was established in March 2010. In four years, the IFC–MIGA partnership has mobilized US$ 2.1 billion with a focus on investments in International Development Association (IDA) countries and fragile, conflict, and violence-affected situations (FCV). The partnership has established the foundation for stronger Bank Group cooperation to enhance transformational engagements that will contribute to the goals. Results Measurement and Evidence Stream (RMES). Management appreciates the recognition of the recently launched Bank Group-wide community of practice for xiii MANAGEMENT COMMENTS results measurement and evaluation practitioners. With the development of a professional cadre of practitioners on results measurement, this initiative promotes a broad spectrum of evaluative approaches to produce evidence on what works. It also fosters a holistic approach to results and evidence that links all stages of the results’ reporting cycle and promotes the adoption of uniform practices across the World Bank Group. Management aims to support the RMES to advance the frontiers of knowledge to help the World Bank Group adopt cutting-edge practices. Integration of partnership programs. In response to the report’s finding on poor integration of the issue-focused partnership programs with the Bank’s country- based model, management agrees that selectivity in new partnerships and trust fund arrangements is important. Entering selectively into new engagements with partners and trust funds requires close dialogue with development partners and coordinated actions across Bank Group units. It is important that such efforts be more systematically aligned with the strategic priorities of the Bank Group and its clients and consistent with the Bank Group’s comparative advantages. To this end, the Development Finance Vice Presidency (DFi) is working with the Global Practice Directors and Regional Vice Presidents to proactively support partnership engagement efforts with more comprehensive strategies and related opportunity analyses. Regular strategic consultations with development partners are becoming an important tool to monitor progress jointly with Bank Group development partners. The goal is to ensure collaboration across the Bank Group, increase transparency, and support the business units to develop their own strategies and manage partnership and fundraising efforts. Key elements of the proposed approach include: (a) an annual strategic planning exercise on corporate and business-unit level fundraising, which would be discussed across the Bank Group, (b) transparency on individual fundraising initiatives throughout the year via better information sharing and inputs into a Fundraising Activity and Transparency database. To ensure institutional buy-in and successful implementation, the External Resource Mobilization Council (ERMC), an advisory body comprising a cross- section of Bank Group leadership, has been created to provide guidance on new partnership engagements and fundraising to ensure that such activities are consistent with strategic priorities. ERMC meets on a monthly basis and is supported by a DFi secretariat. World Bank Management Specific Comments Quality of operations Performance of country programs. Management agrees that the quality of operations is essential to the success of the implementation of the Bank Group strategy, and is pleased to note IEG’s assessment of improvements in country program performance since 2010. IEG notes that it rated 76 percent of country strategies in FY14 as xiv MANAGEMENT COMMENTS moderately satisfactory or better—exceeding the corporate target for the first time since it was established. Management agrees with the report's findings that high quality results frameworks and selectivity are critical to the success of country programs. Indeed much of the recent improvement in country program performance can be attributed to management’s increased focus on the quality of results frameworks. Going forward, management will intensify its efforts to improve results frameworks and increase selectivity. The new approach to country engagement introduces new templates and guidance to help teams ensure that country programs are results focused and selective. The SCD and the CPF results frameworks play a critical role in improving selectivity by using rigorous analysis to identify, define, and track priorities for helping a country to achieve the goals. Outcome ratings. Management questions the report’s conclusions about the declining quality of the Bank’s lending operations, which is true only when based on the number of projects rated. The same analysis using commitment amounts shows performance has stayed steady or improved, and should be made much more visible in the report. The message is that larger projects seem to achieve better results. It is important to analyze the data to understand (a) what percentage of unsatisfactory projects are concentrated in smaller projects, (b) which GPs seem to have a larger share of such operations, and (c) whether there is a concentration of small, unsatisfactory projects in a group of countries. Management believes further analysis of the country’s role in project performance and ways to manage country factors at different stages of the project cycle would be worth considering, especially for smaller projects. Overall, management believes that with over 80 percent of the commitments delivering satisfactory results, it is inaccurate to say that Bank performance is on a steady decline. This situation, an increase in success by commitments, but decline when the data on exits is looked at by the number of projects, calls for a more detailed analysis. Management is also concerned about the reporting of certain numbers, which may be misunderstood. Specifically,  Figure 2.2, on page 23, reports the success rate of development policy financing (DPF) by the number of projects. This would suggest that in the period FY11-13, some 73 percent of DPF operations by number were successful. The data on which this statistic is based is, however, far from complete. For FY11, only 49 of 76 operations (64 percent) have been reviewed by IEG; for FY12, only 37 of 69 (54 percent); and for FY13, only 7 of 45 (16 percent). Over the whole three-year period FY11- 13, only 93 of the 190 DPF operations, or 49 percent, have been evaluated by IEG. Hence the figure of 73 percent success in FY11-13 is much less secure than are the figures for the preceding years. Nowhere xv MANAGEMENT COMMENTS in the graph or the text is there any indication that the figures for FY11- 13 are based on less than half of the data.  Figure 2.4b, on page 24, gives a graph of DPF success rates, this time broken down by GP, and comparing the period FY08-10 with the period FY11-13. It is noted in the text, only three GPs had more than 20 evaluated development policy financing operations in the period FY08-13. This is a low bar; for if there were only 20 in the period FY08- 13, there may have been at best 10 for FY08-10 and 10 for FY11-13, which is scarcely a basis for an intertemporal comparison. Furthermore, we have shown above that only 49 percent of the DPFs for the period FY11-13 have been evaluated by IEG. It would have been more convincing if the numbers of operations involved in the comparisons in Fig. 2.4b had been candidly stated, together with the percentage of the universe, so that the reader could judge whether the conclusions have a sound statistical foundation. As it stands, with the key numbers withheld, the unsuspecting reader might draw false conclusions. Quality at entry. The report’s conclusion that quality at entry is the main determinant for declining performance needs to be qualified. This correlation holds only when applied to number of projects, not project commitments. Management agrees that quality at entry is important and anticipates a positive shift over time as a result of several recently adopted actions: a new country engagement model, stronger peer review system, better technical support to task teams under the GPs, more systematic use of evaluative evidence and learning from implementation, sharpened focus on results and results measurement, and increased management attention and oversight. Results Results focus. Management agrees with IEG on the need to strengthen the overall results focus and quality of the monitoring and evaluation approach in Bank operations. Several initiatives have been introduced as part of the broader change agenda. The introduction of the new country engagement model put more emphasis on more systematically identifying achievable objectives and the monitoring and evaluation of the achievement of those goals. The approach also places increased attention on adaptive management, including early corrective actions to reflect changed circumstances and learning from implementation. The Bank has also adopted a more coherent approach to solicit beneficiary feedback on the effectiveness of Bank Group-supported programs. It is expected that this will help gain insights on the results that they most value, as well as potential risks and ways to address them. xvi MANAGEMENT COMMENTS Remaining challenges for monitoring and evaluation (M&E). Notwithstanding the above, IEG indicates that results frameworks are the most important drivers of outcome ratings, both in terms of magnitude and statistical significance over FY06- 13. This conclusion reinforces the need for management to focus on better capturing and measuring results at the country program level, in partnership with its clients. Knowledge services. Management appreciates the acknowledgment of the quality of Bank Group knowledge services. Management would like to note that the expenditure on analytical and advisory activities (AAA) in FY14 was not US$ 361 million but rather US$ 611 million. AAA includes economic and sector work, technical assistance, technical engagement, impact evaluation, and project assessments, including externally funded activities and reimbursable advisory services. In regards to the issue of self-evaluation for the Bank’s knowledge activities, management would like to clarify that the World Bank Group’s Advisory Services and Analytics does have self-evaluation in the Activity Completion Summary and in the annual review of programmatic approaches. AAA self-evaluation indicators have fed the Corporate Scorecard for some years and were only recently replaced by client ratings. The self-evaluation ratings of many AAA products were updated in the Operations Portal in September 2014. Need to develop soft and hard project management skills. Management agrees with the need for staff to develop soft and hard project management skills and competencies. The Leadership, Learning, and Innovation (LLI) Vice Presidency is working with other units to focus on developing technical, competency-based learning and soft skills training for staff. LLI is also proposing a set of offerings geared to enhance staff skills in the areas of project management, program design, and stakeholder engagement. This is being done more systematically by developing the learning curriculum and content to address specific needs identified by staff and managers. Incentives for learning and improving these skills are also being put in place. IFC Management Specific Comments Management appreciates the thorough review and analysis provided by IEG. The report provides valuable insights and recommendations to improve IFC’s business and contributions to the Bank Group twin goals via private sector development. Development results. Management is pleased with the strong growth in development effectiveness of IFC’s advisory services in 2014, as reported in the 2014 RAP, with the success rate meeting the IFC corporate target. IFC advisory services is xvii MANAGEMENT COMMENTS further strengthening its program, such as with a pilot workshop emphasizing good fundamentals of project design and post-implementation monitoring. On IFC’s investment services, management notes the drop to 60 percent in the FY11- 13 rolling average success rate of IFC investment development outcomes, as measured in the Expanded Project Supervision Reports (XPSRs). As the report describes, this drop continued the downward trend that had started with the XPSRs completed in FY08-10, recording a 13 percentage point fall since that time. While management takes seriously signs of weakening development outcomes of IFC investments, the level of decrease may be exacerbated by the XPSR-based evaluation framework. Based on IFC’s own internal development outcome tracking system (DOTS), the decline in development outcomes is much more modest. The FY11-13 DOTS development outcome success rate was 67 percent, a modest decline from 71 percent in FY08-10. In management’s observation, the lower decline relative to the XPSRs is due to the net improvement in project outcomes since the XPSRs were conducted—as DOTS is updated annually and is able to better capture recent trends in the portfolio. For example, for financial institution projects, 15 of the low- performing clients at the time of the XPSRs have since improved their performance, while only four clients deteriorated. A similar overall improvement is confirmed for the results for Infrastructure and Natural Resources, based on DOTS. Management attributes the decline in development outcome results still observed in DOTS primarily to the effects from the 2009 financial crisis and to a more limited extent from increased IDA focus. First, IEG’s review found that in about 25 percent of the evaluated investments with a low Project Business Success Rating, the low results were caused primarily by the 2009 global financial crisis, and it is management’s view that there were also additional indirect and other adverse effects from the crisis on private sector performance and IFC’s investments. Second, both the XPSR and DOTS systems show lower development outcome success rates for investments in IDA countries, highlighting higher project and macro challenges prevalent in IDA markets. As IFC continues to enhance its strategic focus on higher risk countries to enhance its overall impact, management is aware of the challenges, and as discussed in the IFC Management Response to the 2013 IEG RAP, continues to innovate to overcome such challenges with targeted programs and more effective organization. The report discusses IEG’s in-depth analysis of IFC investment work quality that shows a decline, particularly in upfront screening, appraisal, and structuring. Based on a detailed analysis of the projects reviewed by IEG, management does not observe a systemic work quality issue in investment executions. Nevertheless, IFC management does expect a number of recent changes will improve IFC’s work quality. The recent “Refocused IFC” launched Global Industry Departments and a xviii MANAGEMENT COMMENTS new Client Engagement Model. The new structure, coupled with the Bank Group changes that introduced GPs, is expected to strengthen effective knowledge sharing and allocation of best expertise to solutions, which should enhance work quality. In addition, the new integrated Corporate Risk and Sustainability Vice Presidential Unit will allow IFC to better focus resources to efficiently address critical project issues, which should also lead to overall improvements in work quality. IFC’s approach to MDGs. Management appreciates the report stating IFC’s focus on growth as a key way to address the MDGs. However, the report could have been more forthcoming regarding the importance of IFC in poverty reduction and other MDGs. It is now widely recognized that the private sector is crucial to achieving poverty reduction and shared prosperity. IFC, as the part of the Bank Group most focused on the private sector, plays a critical role in helping achieve these goals. In addition, IFC has aligned its objectives and strategic focus areas with Bank Group priorities in client countries, and as the IEG report states, IEG’s review of nine countries found that close to 80 percent of the Bank Group strategic pillars supported specific MDGs. IFC has also been explicitly part of sector-based Bank Group strategies that addressed MDGs, such as in education. In other MDG areas, such as water and sanitation, IFC has provided support including through advisory programs. Going forward, the proposed new SDGs, which include many private sector-oriented measures such as in infrastructure, agribusiness, financial sector, jobs, and industrialization, show how the global community has now placed the private sector squarely at the center of development. IFC will be actively considering and supporting SDGs through the SCD and CPF process. IFC’s approach to investments. IFC is pleased with the finding by IEG that even though the simultaneous occurrence of pro-growth and pro-poor results is neither universal nor automatic, IFC’s focus on the developmental contribution of its investments has assured a high coincidence of the two. The report further states that IFC has generally supported projects where project development results and IFC profitability were aligned. Regional operations—Africa. The report shows poor results of infrastructure investments in the Africa Region compared to IFC’s average. In this regard, management is pleased to report that this trend is reversing in more recent data. Further, in the context of IFC’s support for micro, small, and medium enterprises (MSMEs) through financial intermediaries and microfinance institutions, management notes the considerable challenges often faced by MSMEs. We have helped create 19 greenfield microfinance institutions in Sub-Saharan Africa since 2005. While their performance varies, as identified in a previous IEG report, none has failed and we work closely to combine investments and advice to further improve performance. On average, IFC-backed microfinance institutions have a return on assets above 2 percent and together hold about 300,000 loan accounts and xix MANAGEMENT COMMENTS 1.2 million deposit accounts, providing critical financial services to microentrepreneurs and low-income households. Nearly 50 percent of these accounts are held by women. Regional operations—East Asia. With respect to IFC’s East Asia financial market operations, management clarifies that given the nature of business, the environmental and social (E&S) requirements for the referred Chinese fund, engaged in secondary purchase of distressed assets portfolios, are differentiated. For private equity funds, IFC focuses on building capacity of client fund managers, who directly negotiate deals with investee companies. With the referred Vietnamese fund, IFC has co-invested in some investee companies and also invested in the sponsor’s subsequent fund, with both funds demonstrating sound corporate governance practice. This project’s ratings for E&S effects and IFC’s work quality are both satisfactory by IEG and IFC. Advisory support for the financial sector in Indonesia and the Philippines was initially focused broadly on the small and medium enterprises (SME) sector due to the high levels of underserved companies and IFC's assessment that risk management practices for that market segment had significant room for improvement despite the relative sophistication of the clients. However, there was limited interest by the banks to push the boundaries of SME lending given other lucrative market opportunities at the time. IFC was eventually more successful pursuing niche financial products which targeted SMEs but could be branded more effectively, such as in Sustainable Energy Finance and Gender Finance. Finally, in regard to IEG’s finding that IFC’s investment strategy in the Indonesian banking sector in more recent years was less relevant, management stresses that it was IFC’s strategic choice that led to supporting leading banks in order to reach into underserved and frontier regions. The financial inclusion agenda in Indonesia is now the subject of a Bank Group key joint project/program MIGA Management Specific Comments Report design. Overall, MIGA finds that RAP 2014 provides a balanced and fair view of MIGA operations during the review period, based on a representative sample. Unlike other IEG thematic evaluations (fragile and conflict-affected states, SME), RAP 2014 is fully based on completed Project Evaluation Reports (PERs). MIGA-supported projects have performed well with a Development Outcome success rate of 71 percent. Performance of MIGA-supported Europe and Central Asia projects. The report refers to the poor performance of several MIGA-supported banking institutions in the Europe and Central Asia Region during the financial crisis. MIGA notes that MIGA support for the financial sector projects in the Europe and Central Asia Region was part of the broader international financial institution (IFI) response to the global financial crisis and should be viewed in context, as was done in the IEG xx MANAGEMENT COMMENTS evaluation of the Bank Group response to the global financial crisis (2011). MIGA also notes that many of the banking institutions supported by MIGA in the Europe and Central Asia Region were also supported by other IFIs (IFC, European Bank for Reconstruction and Development) and therefore share similar performance characteristics. In other words, the performance of the financial sector projects in Europe and Central Asia seems to be driven primarily by macro than micro factors. xxi 1. The World Bank Group and the MDGs Since the Millennium Declaration in September 2000, much progress has been made toward achieving the Millennium Development Goals (MDGs).1 The world reached the poverty reduction target five years ahead of schedule. However, considerable challenges remain since a large number of MDG targets will not be met by the end of 2015 (figure 1.1). Placing social, economic, and environmental sustainability at its core, the post-2015 agenda will be significantly more ambitious (UN 2013). Figure 1.1. Progress Toward Achieving the MDGs (percent) 100 80 60 40 20 0 Poverty Hunger Primary Gender Infant Under‐5 Maternal Water Sanitation Slums education parity motality mortality mortality Source: World Bank and IMF, 2014. Note: Data shows progress toward meeting the relevant MDG targets; 100% = target met. The analysis below reflects on the World Bank Group’s experience with the MDGs to draw lessons for its engagement with the evolving post-2015 development agenda. Through a review of the Bank Group’s strategies, institutional reforms, support programs, and partnerships at the corporate, sector, and country engagement levels, it identifies five key findings:  The Bank Group integrated the MDGs into its strategies while pursuing a broader approach to poverty reduction;  The Bank’s knowledge work provided key intellectual underpinnings to development thinking and directly supported the MDGs;  The Bank Group’s country-based model was crucial for translating the global MDGs into country programs and needs to be further strengthened;  Collective engagement formed an important part of the Bank Group’s support for the MDGs, but there is much room for improvement;  There was no clear articulation of a results chain for understanding and assessing the Bank Group’s role, approach, and expected contribution to the MDGs. 1 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS Box 1.1. Evolution of the Millennium Development Goals and World Bank Group Response MDGs / United Nations World Bank Group World Development Report on Poverty; Poverty First Human Development Report 1990 Reduction Strategy UN Conferences on education, children, environment, human rights, population, social development, and women IDGs approved by OECD/DAC (Bank part of 1996 IDG formulation); HIPC Initiative launched Strategic Compact initiated organizational 1997 reforms; first World Development Indicator published using IDGs as a framework Bank-Fund PRS Initiative introduced PRSP; 1999 Comprehensive Development Framework launched A Better World For All signed by IMF, OECD, Millennium Declaration 2000 UN, and World Bank; Strategic Directions Paper articulated Bank poverty reduction strategy; Bank-hosted technical meeting on MDGs; MDGs adopted 2001 Strategic Framework endorsed MDGs; PRSC introduced Monterrey Consensus 2002 Strategy Update called for scaled-up activities 2003 and partnerships for achieving the MDGs Global Monitoring Report launched; IDA Results 2004 Measurement System incorporated MDG indicators in Tier 1; 2005 Results-based CASs mainstreamed Final MDG targets and indicators approved 2008 Post-Crisis Directions Paper stepped up support MDG Summit 2010 in IDA/AFR; IFC Development Goals piloted 2 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS 2011 Bank Corporate Scorecard incorporated MDG indicators in Tier 1 and WSS MDG indicators in Tier 2 High-Level Panel Report on Post-2015 Agenda 2013 Twin goals approved 17 SDGs proposed 2014 MDGs end; final SDGs to be adopted 2015 Source: IEG. Note: AFR = Africa Region; CAS = Country Assistance Strategy; DAC = Development Assistance Committee; GMR = Global Monitoring Report; HIPC = Heavily Indebted Poor Country; IDA = International Development Association; IDG = International Development Goal (IDG); MDG = Millennium Development Goal; OECD = Organization for Economic Co-operation and Development; PRS = Poverty Reduction Strategy; PRSC = Poverty Reduction Support Credit; PRSP = Poverty Reduction Strategy Paper; SDG = Sustainable Development Goals; WSS = water supply and sanitation. Integrating the MDGs into its Strategies while pursuing a Broader Approach to Poverty Reduction The Bank Group pursued a more comprehensive approach to poverty reduction than the MDGs by combining private sector–led growth with empowering the poor to participate in development—the twin pillars of Bank Group strategies since 2000 (World Bank 2000b).2 During the 1990s, Bank Group strategies had incorporated the themes embodied in the MDGs. It formally endorsed the MDGs in the 2001 Strategic Framework Paper, which found the MDGs to be a useful results-based framework for assessing development impact, and to provide a context for coordination among donors and for engagement with countries. However, the strategy emphasized that meeting the MDGs was a joint responsibility within the international community and expressed concern about the time frames. The Bank Group’s approach to supporting the MDGs was to implement its twin pillar strategy by strengthening the country focus and utilizing the Comprehensive Development Framework. Although this approach was perceived at times as a lack of vigor in the Bank Group’s response to the MDGs, the 2011 Busan Aid Effectiveness Forum adopted a similar approach by recognizing, among other things, the role of the private sector in contributing to poverty reduction and the importance of strong, sustainable, and inclusive growth. Bank Group strategies evolved to account for the changes in development thinking and lessons from independent evaluations. The twin pillars of the Bank’s strategies were developed with input from past World Development Reports (WDRs) and Independent Evaluation Group (IEG) work on poverty (World Bank 1990, 2000a). Subsequent Bank Group strategies, though broadly following the two-pillar approach, evolved based on 3 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS various management reviews and analytical work. The 2007 strategy, for example, found consistency between the Bank’s twin pillar approach and the MDGs, but called for greater focus on institutions and governance, as well as global public goods to reflect lessons learned and new knowledge on development effectiveness generated during the previous six years.3 Following the United Nations MDG Summit in 2010, the Bank stepped up its MDG efforts by emphasizing results management, enhanced instruments, scaled-up support for agriculture, increased International Development Association (IDA) resources for basic education, sustained infrastructure funding, and expanded analytic work and evaluation evidence on what works. Strategies pursued by International Finance Corporation (IFC) and the Multilateral Investment Guarantee Agency (MIGA) did not target the MDGs, but sought to align their objectives and strategic focus areas with the Bank Group’s strategies4 and to contribute to growth by promoting private investment. Since 2000 IFC’s strategy included a focus on opening frontier markets; micro, small and medium enterprises; supporting high-impact sectors that included health, education, water supply, and sanitation; and differentiating through sustainability (IFC 2000, 2004). MIGA’s strategies gave priority to IDA and fragile and conflict-affected states (FCS), transformational projects, energy efficiency and climate change, and support for South-South investments (MIGA, 2005, 2008, 2011 and 2014). Different sectors within the Bank responded to the MDGs differently, reflecting analysis of the Bank’s value-added in the respective sector and application of the selectivity framework (box 1.2). The responses of the education, health, and water and sanitation sectors to the MDGs are reviewed as illustrative examples.5 In education, the Bank shifted its focus from primary education in the 1990s to a broader agenda starting in 2005 to include learning outcomes, education and labor market linkages, and service delivery systems. Given the massive increase in health-related trust funds, the Bank took a more focused approach in 2007 by emphasizing health systems strengthening. In water supply and sanitation (WSS), the Bank addressed access issues, but emphasized sustainability of sector investments on the basis of an important lesson from the 1980s where “performance by target” led to pursuing quick results at the expense of bypassing more difficult reforms (World Bank 2003). 4 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS Box 1.2. World Bank Selectivity Framework 2001 Strategic Directions Paper  Within Countries. Programs will reflect client priorities, use of Comprehensive Development Framework principles, and strategic partnership among stakeholders  Across Countries. Key criteria are income, poverty, and performance. In poor performers, focus will be on non-lending services to support the development of good policies  Global. Criteria will be comparative advantage, strategic relevance, and expected benefits. 2010 Post-Crisis Directions Paper  At the Global Level. Criteria will include: clear linkage to core institutional objectives and country operational work; proven leveraging and catalytic effect; clear understanding of resources required and potential risks involved  At the Sector Level. Specific roles and focus within a sector are made by considering best practices and the Bank Group’s comparative advantage  At the Country Level. Selectivity to be built around Poverty Reduction Strategy Papers for IDA countries; principles for trade-offs are: a long-term vision and strategy; clear country ownership of development goals and actions; strategic partnership among stakeholders; and accountability for results. Both Papers stated the Bank Group will step back where other institutions have a clear comparative advantage, and will encourage and support leadership roles for partner organizations. Source: World Bank 2001; World Bank 2010b. Bank lending for the MDGs reflected its corporate and sector strategies.6 Within growing overall commitments, the share of MDG-related commitments (mostly investment lending) in total Bank commitments remained stable at about 40 percent since the 1990s (figure 1.2). This reflects, in part, the Bank’s increased attention to the MDG themes during the 1990s: the share of Bank lending to the social sectors increased from 5 percent in the early 1980s to 20 percent in the late 1990s (IEG 2000). This trend continued after adoption of the MDGs, with more Bank resources going to MDGs 2–6. Moreover, consistent with the MDG focus of IDA 14, whose replenishment period (FY05–08) was considered a critical time in the effort of low income countries to reach the MDGs (IDA 2005), the share of IDA in MDG-related commitments increased from 27 percent in FY09–01 to 35 percent in FY02–14, while that of the International Bank for Reconstruction and Development (IBRD) declined from 72 percent to 58 percent. About 70 percent of the Bank’s non-MDG related commitments supported its growth and public sector governance goals, which were important drivers for achieving the MDGs. 5 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS Figure 1.2. MDG-Related Bank Commitments and Composition b. Composition of MDG-related commitments (percent) FY90–01 FY02–14 MDG 1 9 7 MDG 2 5 10 MDG 3 7 11 MDG 4 1 2 MDG 5 1 1 MDG 6 1 3 MDG 7 31 27 MDG 8 45 38 Total 100 100 Source: Business Warehouse. Notes: MDG-related commitments are determined using Operations Policy and Country Services classification, which maps the Bank’s theme codes to specific MDGs. Therefore, the accuracy of the data reported here depends on the quality of Bank projects coding. MDG = Millennium Development Goal. The Bank’s support for education, health, and water and sanitation increased relative to overall commitments and became more focused. Primary education focused on IDA countries: average annual IDA commitments to primary education increased by 26 percent between FY90–01 and FY02–14, but IBRD commitments declined by 28 percent. Almost half of Health, Nutrition, and Population (HNP) commitments went to strengthening health system performance, while close to 70 percent of the reproductive health and disease-related (except HIV/AIDS) projects had health system components. In line with the status of the respective MDG targets—the global water supply target has been met but the sanitation target is not likely to be achieved—Bank lending to sanitation7 increased almost eightfold, while that to water supply8 increased by 40 percent. Country coverage increased in all three sectors, which had operations in almost all countries in the Africa Region and FCS (table 1.1). About two-thirds of the countries covered by maternal and child health (MCH) and sanitation projects were off target, which is consistent with the call to focus on lagging MDGs.9 Primary education and water supply had a lower proportion of off-track countries in their portfolios: support to primary education in the on-target countries focused on quality issues since the Bank’s past experience was that global priorities were not the highest priorities in all countries (World Bank 1999). In water supply, Bank support sought to address institutional strengthening and capacity building.10 6 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS Table 1.1. Country Coverage in Primary Education, Health, Water, and Sanitation Portfolio, FY 02–04 Countries added  Countries covered  Countries off‐track   MDG sectors  (number)  (number)  (percent)  Primary education  26 103 54 Maternal and child health  29 92 64 Water  28 106 51 Sanitation  30 107 66 Source: Business Warehouse. The Bank adopted multi-sector approaches in these three sectors. Public administration was the most prevalent component in primary education and HNP projects. Close to 70 percent of primary education projects approved during FY02-14 had public administration components, with 22 percent having HNP components. In HNP, 62-82 percent of projects in various sub-sectors supporting different MDGs had public administration components, with 9-27 percent having education components. While improved water and sanitation were considered critical to health outcomes, a negligible number of HNP projects had WSS components. Nonetheless, because 95 percent of the countries with HNP projects also had WSS projects, the Country Partnership Strategy was seen as the instrument to ensure synergy between HNP and WSS programs. In the case of WSS, there has been an increase in WSS components in projects managed by other sector boards, mainly in Urban Development and Environment. Existing evaluation evidence is inconclusive with regard to the effectiveness of multi-sector approaches in delivering results.11 More analysis is needed to fully understand how this approach would help in achieving the MDGs. Development outcomes for projects in primary education, HNP, and WSS hovered around Bank averages,12 but many faced significant or high risk to development outcomes due to uncertain financial sustainability and institutional weaknesses (table 1.2). Addressing the financial sustainability risk would require greater attention to improving domestic resource mobilization capacity and, more generally, establishing stronger linkages between direct MDG support and public sector governance initiatives. During FY02–14, about half of the Bank’s non-MDG public sector governance commitments were in public expenditure and tax policy, and administrative and civil service reform. Although MDG-related projects included institutional and capacity building components, the effectiveness of these components would need to be improved. 7 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS Table 1.2. Risk to Development Outcomes in Primary Education, Health, and Water and Sanitation Projects Primary  Risk factors  Education  MCH  HIV/AIDS  WSS  (N=48) (N=59) (N=48)  (N=52) Uncertain Financial Sustainability  29 50 32  26 Institutional/Capacity Weaknesses  27 22 36  33 Lack of Ownership  4 24 4  9 Political/Security Risks  11 7 9  15 Source: IEG reviews of the Implementation Completion Reports of relevant projects approved during FY02–14. Notes: A project may have multiple risk factors. MCH = maternal and child health; WSS = water supply and sanitation. N = number of projects. Providing Key Intellectual Underpinnings to Development Thinking The Bank’s analytical work has influenced global development thinking, Bank Group strategies, and the results agenda. For example, World Development Report 1990: Poverty was instrumental for putting poverty reduction at the core of development. Since 2000 several WDRs, such as World Development Report 2004: Making Services Work for the Poor, advanced the thinking on achieving the MDGs and significantly influenced the Bank Group’s corporate and sector strategies.13 Drawing on its strong statistical expertise, the Bank played a key role in defining the International Development Goals (IDGs) in the late 1990s and the MDGs in 2001 (box 1.3), and pushed forward the Managing for Development Results agenda through a series of international roundtables on the use of quantitative, time-bound indicators for measuring development impact.14 Starting in 2004 the Bank and the International Monetary Fund (IMF) produced the annual Global Monitoring Report, which complemented the UN Millennium Development Goals Reports by providing analysis of the policies and actions needed for achieving the MDGs and related development outcomes. Box 1.3. Role of the Bank in Developing the MDG Indicators In May 1997 the Bank, the UN, and OECD sponsored a seminar for DAC members and development partners to review the concepts, methodologies, and data issues in development indicators. Pulling the various declarations at the UN summits into a coherent framework, they produced a synthesis report that articulates seven International Development Goals (IDGs) with 21 indicators. In 2001 the Bank hosted a seminar on the IDGs, “From Consensus to Action: Making Cooperation on the Goals More Effective,” which was attended by key agencies to reconcile the IDGs with the goals in the Millennium Declaration. This led to the emergence of a unique set of goals, targets, and indicators – the MDGs – that could be quantified and for which there were reasonable data. Source: DIIS 2009; Hulme 2009 8 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS Production of impact evaluations (IEs) in the Bank Group grew rapidly during FY02–14. The Bank produces the most IEs of all of the development institutions (IEG 2012a). Recognizing the role of IEs in furthering the knowledge agenda, the Bank in 2005 created the Development Impact Evaluation Initiative, a global program hosted by the Development Research Group to help guide policy choices and improve results. Through the Spanish Impact Evaluation Trust Fund,15 there has been a more systematic approach to identifying and financing IEs, notably in the human development network. IEG reviews of IEs in some MDGs—maternal and child health (IEG 2013b), nutrition (IEG 2010), and water and sanitation (IEG 2008)—identified lessons about what works and what is needed to expand knowledge about the impact of MDG interventions and programs. However, a 2012 IEG evaluation found that the feedback loop between IE production and Bank project operations and learning has been modest, and recommended improved integration of IEs into the design and review of projects to sharpen their results focus (IEG 2012a). The Bank’s analytical and advisory activities (AAA) played an important and growing role in supporting the MDGs. In line with the Bank’s corporate strategies, MDG-related AAA products grew in both absolute terms (by expenditure and by number) and as a share of total AAA delivered (figure 1.3). Although there was more MDG-related economic and sector work overall, MDG-related, non-lending technical assistance (NLTA) expanded particularly fast. Changes in the composition of MDG-related AAA suggest that the Bank used AAA to support the MDG areas where its MDG-related lending declined, and vice versa (figures 1.2 and 1.3). Figure 1.3. MDG-Related AAA and Composition b. Composition of MDG-related AAA (percent) FY90–01 FY02–14 MDG 1 2 10 MDG 2 12 3 MDG 3 14 8 MDG 4 0 1 MDG 5 0 1 MDG 6 2 3 MDG 7 22 23 MDG 8 48 51 Total 100 100 Source: Business Warehouse. Notes: MDG-related AAA is determined using Operations Policy and Country Services classification, which maps the Bank’s theme codes to the individual MDGs. AAA = analytical and advisory activities; ESW = economic and sector work; FY = fiscal year; MDG = Millennium Development Goal; NLTA = non-lending technical assistance. 9 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS At the country level, IEG evaluation evidence suggests that the continuous use of a combination of instruments over a long period achieved synergies among different instruments within the same sector. For example, in Bangladesh, analytic work, policy lending, investment lending, and capacity-building NLTA contributed to improved social protection outcomes.16 The 2014 IEG Country Program Evaluation of Brazil found that for the equity-related objectives, the Bank’s most significant contribution was its technical and knowledge assistance. An extensive AAA program combined with technical assistance lending helped strengthen the government’s capacity to monitor the progress, impact, and targeting of social policies. Although it is likely that growing AAA was in response to country demand and is useful for achieving the MDGs, this cannot be assessed with precision in the absence of a fully developed monitoring and evaluation system for Bank AAA. Translating the MDGs into Country Programs through Country-Based Model Bank Group country strategies are designed to link its institutional priorities with the clients’ own priorities. The 2002 Monterrey Consensus stated up front that “each country has primary responsibility for its own economic and social development.” (UN 2002, 2014). In endorsing the MDGs, the Bank Group emphasized the primacy of country ownership and stepped up the country focus in the implementation of its strategies. This heightened country focus was supported by organizational changes initiated in the late 1990s, which included decentralization and matrix management. The Bank’s country- based model increased client responsiveness by using the Poverty Reduction Strategy Paper (PRSP) as a key country instrument (in IDA countries) and country budgetary actions under the Heavily Indebted Poor Country (HIPC) Initiative and the subsequent Multilateral Debt Reduction Initiative as sources of funding for the social sectors (IEG 2012b). In Bangladesh, for example, the entry points for Bank engagement were largely to align the Country Partnership Strategy objectives with the national goals outlined in the PRSPs, which also formed the basis for collaboration among development partners that led to major sector-wide approaches in health and education.17 Under the Bank’s country-based operational model, the MDG themes were incorporated substantially into country programs. In the nine countries reviewed,18 MDG-related issues were important components of Bank Group country strategies, even before the introduction of the MDGs. Although the term “MDG” was rarely used (either spelled out or as an acronym), nearly 80 percent of the strategic pillars supported specific MDGs.19 Virtually all country strategies included objectives in education and health areas, and environmental sustainability also received considerable attention. In this regard, there were no significant changes in the country strategies after adoption of the MDGs since the country strategies were already well attuned to MDG priorities. There 10 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS were, however, some examples where the MDGs figured prominently in the strategy formulation, such as the Yemen FY03–05 Country Assistance Strategy (CAS), where the Bank stepped in to address MDG targets that the country was not expected to meet (for example, basic education and gender disparity). The MDGs had a modest impact on the choice of indicators for results monitoring in Bank Group country strategies. In the 40 country strategies prepared between FY97–14 for the nine countries, the use of MDG-related indicators in the results matrices remained stable between 35 and 38 percent of all CPS indicators for measuring achievement of the CPS objectives. This is consistent with the finding that the Bank’s overall MDG-related commitments stayed at about 40 percent of all commitments during FY90–14.20 For MDG-related objectives, formal MDG indicators continued to be less frequently used than indicators that addressed some aspects of an MDG objective; even so, there was a modest increase in the use of formal MDG indicators from 11 to 15 percent between FY01–05 and FY10–14. This suggests a gradual uptake of internationally adopted indicators, where appropriate, for measuring the results of Bank Group country programs. Subject to individual country context, Bank Group country strategies had varying degrees of MDG focus. Measured by the use of MDG-related indicators in the CPS results frameworks, the Bank Group focused more on MDG issues in the two FCSs than the low-income countries (LICs) and middle-income countries (MICs) in the sample (figure 1.4). Significant variability existed among the three MICs: in Indonesia, MDG issues figured prominently in the Bank Group program prior to the adoption of the MDGs, and this emphasis was maintained (though at a lower level) afterward; in Tunisia, the relatively strong MDG focus eroded as time passed, with the most recent CPS addressing mainly macroeconomic and private sector development issues; and in Brazil, attention to the MDGs grew rapidly from a low starting point to make the Brazil program one of the most MDG-intensive in the sample. By contrast, the Bank Group’s country strategy for Malawi focused on the MDGs as well as growth (agricultural productivity, business environment, and infrastructure) and governance (macroeconomic management, public financial management and accountability) issues. There is no evidence that the Bank Group considered the MDGs to be less relevant for FCSs or MICs from this sample. 11 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS Figure 1.4. MDG-Related Indicators in Nine Country Strategies, FY97–14 (percent) 70 60 50 40 30 20 10 0 Nepal Yemen Bangladesh Georgia Malawi Nigeria Brazil Indonesia Tunisia FCS LIC MIC FY97–00 FY01–04 FY05–09 FY10–14 Source: IEG based on World Bank Group Country Assistance/Partnership Strategies and Interim Strategy Notes. Notes: Early CPSs for Nigeria did not have results frameworks; therefore no outcome indicators were defined. FCS = fragile and conflict- affected states; LIC = low-income country; MIC = middle-income country. Bank support for the MDGs in the nine countries was followed up through lending and knowledge support (figure 1.5). The three MICs had a higher proportion of MDG-related lending (42 percent on average) than the LICs (36 percent) and FCSs (34 percent) in the 1990s, and this gap in MDG focus widened after the introduction of the MDGs. To a large extent, this reflects the MICs’ choice to work with the Bank on MDG-related issues. In Brazil, for example, the Bank strongly supported MDG 7 throughout the period, as well as objectives in poverty, education, MCH, and communicable diseases. However, in Nepal, one of the poorest countries in the sample, the MDGs were highlighted in Bank Group strategies, but much of the Bank’s lending program was directed to transport, energy, and mining. The Bank’s AAA in the nine countries was high and increasing— from 63 percent in 2000–04 to 74 percent in 2010–14 (by number), consistent with the Bank’s overall trend (figure 1.5). Figure 1.5. MDG-Related Commitments in Nine Countries (percent) 50 45 40 35 30 25 20 Nepal Yemen Bangladesh Georgia Malawi Nigeria Brazil Indonesia Tunisia FCS LIC MIC FY90–01 FY02–14 Source: IEG reviews of World Bank Group Country Assistance Strategies, Country Partnership Strategies, and Interim Strategy Notes. Note: FCS = fragile and conflict-affected states; LIC = low-income country; MDG = Millennium Development Goal; MIC = middle-income country. 12 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS However, the 2012 IEG evaluation The Matrix System at Work found that the country- based model, while having improved the Bank Group’s responsiveness to client needs, made it difficult to integrate sector and corporate strategies with country strategies. Also, global and regional programs were not well integrated into Bank Group country strategies and country programs (paragraph 1.25). The IEG evaluation recommended, among other things, rebalancing the relationship between the country and sector units to ensure that the country-focused model is complemented by appropriate attention to global priorities. Based on internal reviews and analysis since 2008 (World Bank 2008, 2010a, 2010c) and in response to IEG evaluations, the Bank Group is undertaking substantial changes to strengthen its capacity to offer top-quality global knowledge and to integrate technical expertise with country understanding. The impact of the structural reforms may take some time to realize and is yet to be assessed. Engaging Collectively for Achieving the MDGs The Bank’s selectivity framework stated that it would step back in cases where other institutions had a clear comparative advantage and would encourage and support leadership roles for partner organizations (box 1.2). The 2001 World Bank Group Strategic Framework recognized that “the alignment of the efforts of the development community with the MDGs presented an unprecedented opportunity to improve effectiveness in poverty reduction.” Applying the selectivity framework in the context of other partners’ support programs for the MDGs (box 1.4 provides two examples), the Bank Group engaged in partnerships at several levels in the pursuit of the MDGs. Box 1.4. Support of the United Nations Development Program and Asian Development Bank for the MDGs The Millennium Development Goals (MDGs) shaped the United Nations Development Program (UNDP) to a considerable extent, but the focus of this work changed. Initially UNDP focused on raising awareness about the MDGs through a series of campaigns. At the country level, it aimed to work with national partners to fix national MDG targets, establish monitoring mechanisms, mobilize public support, and plan national MDG reports. After 2005 it focused support on incorporating and mainstreaming the MDGs in national development strategies through policy advisory services. Its work also included data analysis to help governments decide on the relative allocation of resources for health, education, irrigation, transport, and other sectors. Since 2010 UNDP sought to directly support the achievements of selected MDGs through the MDG Acceleration Framework. Asian Development Bank’s (ADB) 1999 Poverty Reduction Strategy (PRS) declared poverty reduction as its overarching goal, with sustainable economic growth and social development as the twin pillars of its poverty reduction framework. ADB announced its commitment to the MDGs in its 2002 Annual Report. A review of the PRS in 2004 concluded that ADB’s comparative advantage was in financing relatively large investments and related policy reform, and made the case for social development through infrastructure investments in water, roads, and energy. The subsequent PRS shifted to a stronger emphasis on poverty reduction at the country level by addressing it in national poverty reduction strategies. ADB’s 2006 strategy showed less progress for non-income MDGs, but deprioritized the related sectors (agriculture, health, and biodiversity). Its subsequent strategy continued to focus on some MDGs while supporting others indirectly. The ADB’s results framework incorporated selected MDGs in its Level 1 indicators. Its direct support for the MDGs is estimated to be 38 percent of total commitments. Support (by number of projects) for MDG 1 and MDG 7, individually, account for more than MDGs 2–6 combined. Source: UNDB 2014; ADB 2013. 13 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS The expansion of trust funds, in particular the financial intermediary funds (FIFs), created opportunities as well as challenges (figure 1.6). Trust fund reforms resulted in the integration of Bank-executed trust funds and recipient-executed trust funds into regular Bank processes (World Bank 2013b). For the FIFs, with all but the Global Environment Facility being established after 2000 and with the expressed objective to support MDG issues, each FIF had its own governance structure and operating modalities. This made it challenging to fully integrate and align the FIF programs with Bank Group country strategies. In interviews conducted for this report, there were concerns about the adverse impact of trust funds on Bank support, with priorities determined by money flows; however, IEG cannot conclude that trust funds diverted attention from priorities. Figure 1.6. Trust Funds Support for MDG-Related Sectors, FY02–14 Bank‐executed trust funds:  Recipient‐executed trust  Financial intermediary  $5.4 billion funds: $30.8 billion funds: $44.6 billion EDU ,  EDU ,  8% 1% HNP,  EDU ,  11% Other 15% s, 20% HNP,  ENV,  14% ENV,  12% 13% Others Others HNP,  , 70% , 65% 66% ENV,  6% Source: Operations Policy and Country Services Concessional Financing. Note: EDU = Education; ENV = Environment; HNP = Health, Nutrition, and Population. The Bank Group followed different sectoral approaches to MDG partnerships. Although trust funds supported all sectors reviewed, there were some notable partnerships that complemented the Bank’s MDG efforts. In education, the Bank relied on the Education for All-Fast Track Initiative to channel donor assistance for primary education in low- income countries, with the Bank managing and supervising the projects.21 This gave space for the Bank to channel its own resources into support for education and labor market linkages and accountability and incentive systems to improve service delivery. In health, disbursement from the health sector FIFs was equivalent to the total HNP sector commitments approved during FY02–14. The Global Fund to Fight AIDS, Tuberculosis, and Malaria, which was established in 2002 to support the MDGs, disbursed $26.7 billion  during 2002–14, about five times the Bank’s disease-related commitments. An IEG 14 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS review found that country-level stakeholders saw The Global Fund as not well integrated into the donor coordination mechanism in the countries (IEG 2012c). Finally, in WSS, the partnerships focused mainly on advocacy, coordination, and knowledge sharing, as with the Sanitation and Water for All partnership. Public-private partnerships were viewed as key to scaling up service delivery in WSS. The Bank and IFC developed various forms of engagement with the private sector (World Bank and PPIAF 2006), and several Bank projects supported private sector participation in urban and rural WSS (IEG 2009). The World Bank Group played a clear role in helping to align governments and donors through the Bank-IDA programs, and through the PRSPs and HIPC with the IMF, but there is no indication that the MDGs were key drivers in World Bank Group country partnerships. There are examples of good institutional cooperation between development partners, such as between the Bank and the Millennium Challenge Corporation (MCC) on education in Georgia, the long-term cooperation between the Bank Group and the U.K. Department for International Development in Nigeria, or the joint program among the four largest development partners in Bangladesh. However, these are examples of cooperation in MDG-related sectors, not on the MDGs as such. Since 2013 the Bank Group worked with the UN on 11 country reviews through the Chief Executive Board in an effort to identify roadblocks to MDG achievement at the country level, to draw lessons for post-2015, and to build models of collaboration.22 Independent reviews and IEG evaluations identified issues with some partnerships and global programs. The Five Year Evaluation of the Global Fund23 in 2009 found that the external financing of the disease control services, although helped to mobilize existing capacity in the affected countries, had stretched the countries’ generally weak health systems to their limits (The Global Fund 2009). IEG’s 2012 Global Program Review: The Global Fund to Fight AIDS, Tuberculosis and Malaria and the World Bank’s Engagement with the Global Fund (IEG 2012c) raised concerns over the long-term sustainability of recipient countries’ programs. Also, in a survey of Bank task team leaders conducted as part of the IEG review, less than 10 percent of respondents characterized the relationship between the Bank and Global Fund staff as collaborative at the country level. IEG’s 2011 evaluations of trust funds and global and regional partnership programs (GRPPs) found that on specific issues such as primary education and child immunization, trust funds facilitated substantive coordination of donor funds, but there was no clear evidence that trust fund resources added to global official development assistance (IEG 2011b), and that half the GRPPs reviewed faced issues of sustainability due to ineffective resource mobilization strategies, poor management, or difficulty in demonstrating results (IEG 2011a). In response, the Bank is pursuing reforms of its trust fund activities (World Bank 2013a, 2013b). 15 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS Articulating a Results Chain for the World Bank Group’s Role, Approach, and Expected Contribution to the MDGs The Bank Group strengthened its results orientation, and as time passed, Bank results monitoring systems and results frameworks utilized MDG indicators. As part of the institutional effectiveness reforms in the late 1990s, a new system of performance indicators for client responsiveness and drivers of project outcomes was introduced and included baselines and time-bound targets. Results-based country strategies were mainstreamed in 2005, and results based financing is now widely used in many sectors. The recently developed Program for Results instrument represents another effort toward linking financing to pre-determined, staged results. The Bank was also an early advocate for the Managing for Development Results agenda (paragraph 1.17). In 2004 the IDA Results Management System incorporated MDG indicators in Tier 1 (country results). In 2010 IFC developed the IFC Development Goals,24 which was an important step forward for private sector–focused international financial institutions to use development goals to drive strategies and operational decision making. Since 2011 the Bank’s Corporate Scorecard included MDGs in their Tier 1 indicators, and the WSS MDG indicators were included in Tier 2 (country results supported by Bank activities). Many Bank Group country programs addressed statistical capacity issues. In most of the countries reviewed, one or more country strategy documents included dialogues and assistance for statistical capacity building, although normally these were not highlighted and not reflected in the objectives or the results frameworks. Support was usually provided through small trust-funded grants (except for a large IBRD loan for Indonesia), but was also included as part of broader economic management programs. In some cases (Yemen for example), the support for statistical capacity building was linked directly to results monitoring under the PRSP and the CAS; in other cases, the support was linked more generally to the need for improved poverty analyses (for example, Bangladesh) or improved domestic resource management (Nigeria, for example). Note, however, that the Bank provided the least statistical support in the poorest countries in the sample—no specific project in Malawi,25 and one recent operation each in Nepal and Yemen. However, the Bank Group did not have a clearly articulated results chain for the MDGs, nor did it have a systematic process for assessing or evaluating its contribution to MDGs. Given the different views and expectations – both externally and internally - on how the Bank Group should address the MDGs, a corporate level results framework and a robust monitoring and evaluation system for the MDGs would have provided stakeholders and partners with better information on the Bank Group’s institutional response and contribution to the MDGs. Although the Bank has a clear selectivity framework (Box 1.2), there is little analysis of its contribution to the MDGs apart from its overall contribution to poverty reduction and economic development. Consequently, there was 16 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS no effective feedback loop that would enable the Bank Group at the corporate level to address specific MDG areas that might need special attention. Identifying intermediate outcomes that can be monitored would have helped link the MDGs to Bank Group activities. The 2001 World Bank Group Strategic Framework emphasized that intermediate goals should take into account the differences in country and regional conditions and develop realistic timeframes. Many Bank and IFC projects (both financing and knowledge services) contribute to the MDGs, but measuring the impact of these projects based on specific MDG indicators focuses on only one facet of the Bank Group’s work; other intermediate outcomes are needed to take into account the long- term nature of the MDGs while clarifying the paths through which the Bank Group contributes to the MDGs. Key Messages for the Bank Group’s Engagement with the Post-2015 Agenda The Bank Group can continue to play a key role in the post-2015 agenda by anchoring its own strategies and approaches firmly on sound analysis and evaluation evidence. Based on major analytical work and IEG evaluation findings, the Bank Group internalized the MDG themes a decade before the Millennium Declaration and initiated organizational reforms to strengthen its capacity to deliver results. This allowed the Bank Group to continue to pursue a broad development agenda and to utilize existing tools and structures to help advance the MDG agenda. Going forward, the Bank Group can play an equally centering role by emphasizing world-class knowledge work, including data, and evidence-based decision making. In a global context of uncertainties, it is important that the Bank Group respond to the complex set of new challenges with intellectual clarity and conviction to achieve maximum development impact. Translating global goals to sustainable national programs remains the key post-2015 challenge. The 2014 Outcome Document of the UN Open Working Group on the SDGs reemphasized the central role played by national governments in development. The Bank Group, with its long-standing country focus, can play a key role in helping national governments assume their responsibilities. However, it will have to address the issues raised in the IEG matrix evaluation, in particular the strengthening of the Bank Group’s ability to bring global perspective to the delivery of its country services. The experience with the MDGs highlights the importance of having a country instrument such as the PRSP to help identify realistic national medium-term targets and programs toward achieving long-term goals and securing ownership among stakeholders. In this process, there is a need to balance “quick wins” and long-term sustainability, manage the constructive tension between country focus and issue focus, address domestic resource mobilization and institutional capacity issues, and build strong partnerships at all levels. 17 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS To deliver on the twin goals and the post-2015 agenda, the Bank Group would benefit from a clearly articulated role, approach, and expected contribution to the evolving SDGs, both externally and internally. With the MDG agenda, there were different external expectations of the Bank Group for its MDG support, while internally management and staff were faced with a multitude of corporate priorities (MDG and non-MDG) that had to be aligned with client priorities. Although a framework of selectivity exists to help the Bank Group determine where and how to engage with the MDGs, there is no assessment on the effectiveness of the application of the framework. As such, it is unclear whether the Bank responded to the lagging MDGs in Tier 1 of the corporate scorecard, and if so, how it responded. Similarly, although IFC and MIGA engaged with the MDGs in various ways, it is not clear how much of this was by design. A clearly articulated results chain and monitoring system for the World Bank Group’s contribution to the SDGs may be one way of providing clarity on the expected roles, approaches, and contribution of the Bank, IFC, and MIGA. References Asian Development Bank (ADB). 2013. Thematic Evaluation Study: ADB’s Support for Achieving the Millennium Development Goals. ADB, Manila, Philippines. Danish Institute for International Studies (DIIS). 2009. Using Indicators to Encourage Development Lessons from the Millennium Development Goals, by Richard Manning. DIIS Report 2009:0, DIIS, Copenhagen. Hulme, David. 2009. “The Millennium Development Goals: A Short History of the World’s Greatest Promise.” Brooks World Poverty Institute Working Paper 100, University of Manchester, Manchester, UK. International Development Association (IDA). 2005. Additions to IDA Resources: Fourteenth Replenishment Working Together to Achieve the Millennium Development Goals. Washington, DC: IDA. IEG (Independent Evaluation Group). 2000. Poverty Reduction in the 1990s: An Evaluation of Strategy and Performance. Washington DC: World Bank. ―――. 2008. What Works in Water Supply and Sanitation? Lessons From Impact Evaluations. Washington DC: World Bank. ―――. 2009. Water and Development An Evaluation of World Bank Support 1997–2007. Washington DC: World Bank. ―――. 2010. What Can We Learn From Nutrition Impact Evaluations? Lessons From A Review of Interventions To Reduce Malnutrition In Developing Countries. Washington DC: World Bank. ―――. 2011a. An Independent Assessment: The World Bank’s Involvement in Global and Regional Partnership Programs. Washington DC: World Bank. ―――. 2011b. Trust Fund Support for Development: An Evaluation of the World Bank’s Trust Fund Portfolio. Washington DC: World Bank. ―――. 2012a. World Bank Group Impact Evaluations: Relevance and Effectiveness. Washington DC: World Bank. 18 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS ―――. 2012b. The Matrix System at Work An Evaluation of the World Bank’s Organizational Effectiveness. Washington DC: World Bank. ―――. 2012c. Global Program Review: The Global Fund to Fight AIDS, Tuberculosis, and Malaria, and the World Bank’s Engagement with the Global Fund. Washington DC: World Bank. ―――. 2013a. Biennial Report on Operations Evaluation (BROE): Assessing the Monitoring and Evaluation Systems of IFC and MIGA. Washington, DC: World Bank. ―――. 2013b. Delivering the Millennium Development Goals To Reduce Maternal and Child Health Mortality. Washington DC: World Bank. International Finance Corporation (IFC). 2000. “IFC Strategic Directions, April 14, 2000.” Strategic Directions Paper, IFC, Washington, DC. ―――. 2004. “IFC Strategic Directions, March 23, 2004.” Strategic Directions Paper, IFC, Washington, DC. Multilateral Investment Guarantee Agency (MIGA). 2005. Strategic Directions FY05–08. Washington, DC: MIGA. ―――. MIGA. 2008. Strategic Directions FY09–11. Washington, DC: MIGA. ―――. MIGA. 2011. Strategic Directions FY12–14 Strategy: Achieving Value-Driven Volume. Washington, DC: MIGA. ―――. MIGA. 2014. Strategic Directions FY15–17. Washington, DC: MIGA. Macro International. 2009. The Five Year Evaluation of the Global Fund to Fight AIDS, Tuberculosis and Malaria: Synthesis of Study Areas 1, 2, and 3. Geneva: The Global Fund. United Nations (UN). 2002. Monterrey Consensus on Financing for Development. New York: UN. ―――. 2013. A New Global Partnership: Eradicate Poverty and Transform Economies Through Sustainable Development, the Report of the High Level Panel of Eminent Persons on the Post-2015 Agenda. New York: UN. ―――. 2014. Open Working Group proposal for Sustainable Development Goals. New York: UN. United Nations Development Programme (UNDP). 2014. “Evaluation of the Role of UNDP in Supporting National Achievement of the MDGs.” Concept Note, March 2014, UNDP. New York: UN. World Bank. 1990. World Development Report 1990: Poverty. Washington, DC: World Bank. ―――. 1999. Education Sector Strategy. Washington DC: World Bank. ―――. 2000a. Poverty Reduction in the 1990s: An Evaluation of Strategy and Performance. Washington DC: World Bank. ―――. 2000b. Strategic Directions Paper for FY01–03: A Framework for Strengthening the Fight Against Poverty. Washington, DC: World Bank. ―――. 2001. “Strategic Directions For FY02–04.” Strategic Directions Paper, World Bank, Washington, DC. ―――. 2003. “Addendum 3—Development Committee 2003 Spring Meetings: Water Supply and Sanitation and the Millennium Development Goals.” Board Report 30359, World Bank, Washington DC. ―――. 2008. “Toward a Global Bank: A Discussion Note” Discussion Note SecM2008-0428, World Bank, Washington, DC. ―――. 2010a. Toward a Global Bank: The Second Update. Washington, DC: World Bank. ―――. 2010b. “New World, New World Bank Group: (I) Post-Crisis Directions.” CD2010-0003, World Bank, Washington, DC. 19 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS ―――. 2010c. “New World, New World Bank Group: (II) The Internal Reform Agenda.” DC2010-0004 , World Bank, Washington, DC. ―――. 2013a. “Management Framework for World Bank Partnership Programs and Financial Intermediary Funds.” Board Report 78512, World Bank, Washington DC. ―――. 2013b. Trust Fund Reforms: Progress to Date and Future Directions. Washington DC: World Bank. World Bank and Public-Private Infrastructure Advisory Facility (PPIAF). 2006. Approaches to Private Participation in Water Services: A Toolkit. Washington DC: World Bank. World Bank and IMF. 2014. Global Monitoring Report 2014: Growth, Poverty and Shared Prosperity. Washington, DC: World Bank. 1 See Appendix A for a brief summary of the origins and evolution of the MDGs. 2See Appendix B for a more detailed review of the Bank Group’s corporate and institutional responses to the MDGs. 3The paper cited World Development Report 2004: Making Services for the Poor and World Development Report 2005: A Better Investment Climate for Everyone as providing important analysis of the twin pillar approach. 4For example, the 2001 IFC Strategic Directions paper was presented in the context of the World Bank Group 2001 Strategic Framework paper, and the 2013 IFC Road Map FY14-16 discussed how the IFC strategic focus areas contributed to the goals of the current World Bank Group strategy. 5 Appendix D provides a fuller review of the three sectors’ response to the MDGs. 6 See Appendix C for more details on Bank Group activities to support the MDGs. 7 Based on commitments under the sector codes for sanitation (WA), wastewater collection and transportation (WT), and wastewater treatment and disposal (WV). 8 Based on commitments under the sector code for water supply (WC). 9 For example, see the 2010 World Bank Group Post-Crisis Directions paper. 10Of the 54 water supply and sanitation investment projects approved during FY02–14 with IEG Implementation Completion Report Reviews, 47 had infrastructure components and 51 had institution building components. 11For example, IEG’s water sector evaluation (Water and Development, An Evaluation of World Bank Group Support, 1997-2007) found that multi-sector approach had better outcome and sustainability than single sector approach, but its recent transport evaluation (Improving Institutional Capability and Financial Viability to Sustain Transport - An Evaluation of World Bank Group Support Since 2002) found that multi-sector approach has not worked as well as single sector approach with regards to sustained services delivery. 12Between FY90–01 and FY02–14, share of Health, Nutrition, and Population projects rated MS+ (moderately satisfactory or better) for outcome achievement improved from 68 to 71 percent, but declined from 84 to 73 percent for primary education, and from 76 to 69 and from 77 to 71 percent, for water and sanitation respectively. 13The 2001 World Bank Group strategies reflected World Development Report 2000/2001: Attacking Poverty and the 2000 IEG evaluation Poverty Reduction in the 1990s: An Evaluation of Strategy and 20 CHAPTER 1 THE WORLD BANK GROUP AND THE MDGS Performance. The 2007 World Bank Group strategy incorporated the findings from World Development Report 2004: Making Services Work for Poor People and World Development Report 2005: A Better Investment Climate for Everyone. World Development Report 2004: Making Services Work for Poor People was an important input to the 2005 Education strategy. 14The 2002 Monterrey Conference recognized the need to measure results throughout the development process. Soon after, the World Bank convened the first International Roundtable on Measuring, Monitoring, and Managing for Results. Subsequent roundtables were held in 2004 (Marrakech), 2007 (Hanoi), and 2008 (Accra). 15The Spanish Impact Evaluation Trust Fund was reconstituted with UK funding as the Strategic Impact Evaluation Fund. 16Background note for the upcoming IEG evaluation The Poverty Focus of Country Programs: Lessons from World Bank Experience. 17Background note for the upcoming IEG evaluation The Poverty Focus of Country Programs: Lessons from World Bank Experience. 18The nine countries include Nepal and Yemen— fragile and conflict-affected states (FCS)— Bangladesh, Malawi, Nigeria, and Georgia (non-FCS low-income countries as of 2000), and Brazil, Indonesia and Tunisia (middle-income countries). See Appendix E for more information. 19Forty country strategies for the nine countries were reviewed, spanning from before the Millennium Declaration to the present. These strategies had a total of 138 pillars or components to reflect different areas of World Bank Group engagement, of which 108 pillars were related to some extent to the Millennium Development Goals (MDGs). 20Note that non-MDG related objectives usually had more outcome indicators than MDG- related objectives in World Bank Group country strategies. 21Fast Track Initiative (FTI) was incorporated into Bank programs and projects because, since its inception, the Bank has hosted the FTI Secretariat and serves as Trustee and Supervising Entity for the Catalytic Fund and the Education Program Development Fund. 22 Board Seminar on the MDGs and Post-2015, September 11, 2014. 23 This is an external evaluation commissioned by the governing body of the Global Fund. 24 See Appendix F for more information on IFC Development Goals. 25According to Malawi country team, the Bank has been providing statistical support; however, the related activities cannot be identified through the Bank’s lending and AAA tracking system. 21 2. World Bank Group Operations Total Bank Group commitments rose sharply to $61 billion in fiscal year (FY) 14. However, Independent Evaluation Group (IEG) portfolio reviews show that the performance of Bank Group lending operations and International Finance Corporation (IFC) investments continued to decline, while Bank Group country programs, IFC advisory services, and MIGA guarantees achieved positive results. The analysis in this chapter sheds new light on the link between Bank Group performance and some work quality issues that are within the institutions’ control, and on the need for better articulation of the results chains in country strategies to facilitate strategic decision making and to improve selectivity. World Bank Group Commitments Rose World Bank commitments increased by almost 30 percent in FY14 to $40.8 billion, with growth in International Development Association (IDA) commitments continuing to outpace International Bank for Reconstruction and Development (IBRD), as shown in figure 2.1. The surge in IDA commitments in FY14 reflects the completion of the IDA16 replenishment, with $8.7 billion (38 percent) going to four large borrowers (India, Pakistan, Bangladesh, and Nigeria). IBRD commitments, although lower than IDA, are above pre- crisis levels. Infrastructure remains a key sector for the Bank, accounting for more than 40 percent of total commitments. Four Global Practices (GPs)—Transport and Information and Communications Technology, Macroeconomics and Fiscal Management (Macro & Fiscal), Energy and Extractive Industries, and Social, Urban, Rural, and Resilience (SURR)— accounted for 57 percent of Bank commitments in FY14. The Bank’s expenditure on analytical and advisory activities (AAA) grew by 71 percent to $361 million in FY14. Figure 2.1. World Bank Group Financing, FY01–14 b. MIGA guarantees ($,  billions) 4.0 3.0 2.0 1.0 0.0 MIGA traditional MIGA new Source: World Bank Group. Note: IBRD = International Bank of Reconstruction and Development; IDA = International Development Association; IFC = International Finance Corporation; IFC LT = IFC long-term commitment; IFC ST = IFC short-term commitment; MIGA = Multilateral Investment Guarantee Agency. 23 CHAPTER 2 WORLD BANK GROUP OPERATIONS IFC’s long-term investment commitments fell in 2014, but were up 27 percent when comparing FY12–14 with FY09–11. Infrastructure and Natural Resources is the only cluster where long-term investment commitments grew in FY14. The Financial Institutions Group’s investment commitments grew over FY12-14 and remained IFC’s largest industry cluster. Commitments to IDA countries grew proportionately with long-term commitments; commitments in fragile and conflict-affected states (FCS) increased by 80 percent, although they still represent only 2–3 percent of IFC’s total. Mobilization of funds from other financiers continues to be an IFC priority, but IFC’s syndicated loan program, which is augmented by its Asset Management Company and other mobilization instruments, grew more modestly than IFC’s long-term commitments. IFC’s short-term finance instruments, mainly its Global Trade Finance Program, continued to grow rapidly, and in FY14 made up about 40 percent of IFC’s total commitments.1 Expenditures for IFC advisory work remain stable, totaling $234 million in FY14 versus $232 million in FY13. MIGA grew its business through supporting larger projects and new guarantee products. Infrastructure accounts for 45 percent of its new guarantee exposure. During the past two years, MIGA increased its exposure in Africa and introduced guarantees against non- honoring of financial obligations by sovereign and sub-sovereign enterprises. The new guarantees represented almost half of MIGA’s new guarantee volume in FY14. Performance of Bank Lending and IFC Investments Continued to Decline Recent portfolio performance of Bank and IFC financing, which are subject to separate and different evaluation methods, is trending downwards. IFC advisory services are the bright spot within the portfolio. The number of evaluated MIGA guarantees is too small to do meaningful trend analysis. Bank knowledge activities do not currently have a fully functioning self-evaluation system, which prevents continuous tracking and assessment of their impact.2 DEVELOPMENT OUTCOMES OF BANK-FINANCED PROJECTS DECLINED IEG reviews of Implementation Completion Reports (ICRs) of Bank lending operations show that the share of projects with moderately satisfactory or better (MS+) development outcome (DO) ratings declined from 73 to 70 percent of rated projects between FY08–10 and FY11–13 (figure 2.2). Continuing a long-term, steady downward trend in overall performance since FY06–08, the decline across the two most recent periods is statistically significant at 90 percent confidence level. Based on commitments, Bank lending performance is holding steady at 80 percent and above, suggesting that larger projects continued to achieve better results than smaller operations. 24 CHAPTER 2 WORLD BANK GROUP OPERATIONS Figure 2.2. Bank Lending Success Rate, FY00–13 b. By commitment (percent) 90 85 80 75 70 65 DPF IPF Bank‐wide Source: IEG. Notes: Success rate is the share of projects rated moderately satisfactory or better for development outcomes. The ratings by number of projects include all IBRD, IDA and trust funded projects, but ratings by commitment include only IBRD and IDA projects. DPF = development policy financing; IPF = investment project financing. DO ratings (by number of projects) fell significantly in IBRD countries and in the East Asia and Pacific Region (figure 2.3). The 6 percentage point drop in IBRD outcome achievement effectively eliminates the previously observed performance difference between IBRD and IDA. FCS performance also worsened. Of particular note is the 11 percentage point decline in the East Asia and Pacific Region performance, which is associated with a sharp drop in outcome achievement for investment project financing (IPF). The Middle East and North Africa Region continues to be the weakest performer among all Regions despite being the only Region with an improving trend in overall DO ratings between FY08–10 and FY11–13. IEG’s Regional Updates provide more detailed analysis of Regional performance (see attachement). Figure 2.3. Bank Lending Success Rate by Country Group and Region, FY08–13 Source: IEG. Notes: AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; FCS = fragile and conflict-affected states; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; LAC = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. * Statistically significant at 90 percent confidence level; ** Statistically significant at 95 percent confidence level. 25 CHAPTER 2 WORLD BANK GROUP OPERATIONS DO ratings by number of projects varied significantly across the 14 GPs (figure 2.4).3 Performances trended upward for four GPs between FY08–10 and FY11–13, but moved in the opposite direction in all others. For most practices, the change was driven almost exclusively by IPF performance, noting that only three GPs had more than 20 evaluated development policy financing (DPFs) during FY08–13. Among the best (and improving) performers is SURR, the largest single practice with 201 evaluated projects, of which 94 percent is investment lending. By contrast, some of the strongest performers in the earlier period such as Water, a large practice for IPFs, experienced more recent decline in outcome achievement. The same is true for Macro & Fiscal, which is the second largest practice (157 projects in FY08–13, of which 91 percent is DPF). The Governance GP appears to have the poorest performance in the last three years for both IPFs and DPFs. Figure 2.4. Bank Lending Success Rate by Global Practice, FY08–13 a. IPF (percent) b. DPF (percent) 90 90 80 80 70 70 60 60 50 50 FY08–10 FY11–13 FY08–10 FY11–13 Source: IEG. Notes: Global Practices with fewer than 20 evaluated projects during FY08–13 are excluded. IPF = investment project financing; DPF = development policy financing; E&E = Energy and Extractives; ENR = Environment and Natural Resources; Mkts = Markets; HNP = Health, Nutrition, and Population; ICT = Information and Communications Technology; Macro & Fiscal = Macroeconomics and Fiscal Management; SPL = Social Protection and Labor; SURR = Social, Urban, Rural, and Resilience. * Statistically significant at 90 percent confidence level; ** statistically significant at 95 percent confidence level. By commitment, these trends hold true broadly except that outcome achievement also trended downward in the Middle East and North Africa Region, while IPF performance in Environment and Natural Resources and Governance GPs appears to be improving between the two periods. Further analysis is required to fully understand the factors that drive the portfolio performance in these cases. DEVELOPMENT OUTCOMES OF IFC INVESTMENT PROJECTS DECLINED SHARPLY Continuing the downward trend reported in the Results and Performance of the World Bank Group 2013, the success rate for IFC investment projects dropped by 13 percentage points between 2008–10 and 2011–13, to only 60 percent for the 232 evaluated projects (figure 26 CHAPTER 2 WORLD BANK GROUP OPERATIONS 2.5).4 Three of the four sub-ratings (project business success, contribution to economic sustainability, and private sector development impact) declined in 2011–13 compared with 2008–10, while ratings for environmental and social (E&S) effects remained strong and stable. Project performance in FCS, IDA countries, infrastructure, education, and agribusiness—all strategic priority areas for IFC—lagged IFC’s overall results, although the performance gap between IBRD and IDA countries narrowed as DO success rates in IBRD countries dropped more sharply. Manufacturing, Agribusiness, and Services (MAS) was the best performing cluster, with some manufacturing projects benefitting from first mover advantages or competitive cost structures that contributed to very strong project business success rates and private sector development results. In contrast, results for the Infrastructure and Natural Resources (INFRA) and Financial Institutions Group (FIG, formerly Financial Markets) clusters continued to drop. Figure 2.5. IFC Investments Success Rate, FY00–13 a. Overall outcome (percent) b. By sub‐indicator (percent) 85 85 80 80 75 75 70 70 65 65 60 60 55 55 50 50 45 45 40 40 35 35 Overall DO IDA & blend non‐IDA PBS ES ESE PSD Source: IEG. Notes: IFC development outcome is a synthesis rating of project business success, economic sustainability), environmental and social effects, and private sector development success ratings. DO = development outcome; ES = economic stability; ESE = environmental and social effects; IDA = International Development Association; PBS = project business success; PSD = private sector development. Project-specific factors and, to a lesser degree, the effects of the global financial crisis contributed to poor results. IFC’s experience with past crises showed that investment projects approved just before a crisis are more vulnerable.5 A review of projects evaluated in 2011–13 showed that about 25 percent experienced severe financial constraints that affected project business success (or their return to financiers) related to the financial crisis (for example, investors walking away and commodity price declines).6 However, in most cases, project business success rates suffered due to non-crisis-related issues (for example, overestimation of the market, offtake agreements gone awry, and poor investments by equity funds) that may have been addressed with better IFC work quality. 27 CHAPTER 2 WORLD BANK GROUP OPERATIONS The success rate of IFC’s investment outcome (profitability of the investment to IFC) in 2011–13 dropped by 14 percentage points during the previous three years to 68 percent, due primarily to lower equity success rates. Loan success rates remained strong. IFC’s investment outcome is important to sustain its ability to deliver its development objectives and contribute to the achievement of the twin goals. However, profitability is only one part of its business model, though an important part (box 2.1). Box 2.1. IFC Profitability is Only One Part of IFC’s Business Model, Though an Important Part Profitability is not identical to development. In FY11 IFC added a profitability indicator to its corporate scorecard and since then has placed increasing focus on the profitability of IFC operations. A profitability focus can bring an important change in incentives that is fully compatible with development outcomes. However, profit is not identical to development. Since IFC presented its triple bottom line in its FY07 annual report, it made it clear that profitability is only one part of its business model, though an important part. It is important to recognize that IFC’s focus is on development under the constraint of client business profitability and IFC’s returns. This indirect approach can be a powerful and effective way to achieve both profitability and development results. Profitability for IFC and profitability of clients are closely associated. According to its business principles,a IFC is expected to fully share the risks with its business partners. Results indicate that IFC’s investment outcomes are closely aligned and typically exceed project business success rates (Figure B2.1.1). Evidence shows that client business profitability and Figure B2.1.1. Project business success and development outcome are closely  aligned with important exceptions  development outcomes Example: Poor investment, good  tend to move in the same developmental impact:   direction. For example, An agribusiness project funded by IFC  when profitability reflects a was not fully implemented. Investors’  returns were lower than expected and  successful response to social below the company’s weighted  needs, a need for average cost of capital. However, more  innovation, or increased than 6,500 jobs were created in a poor  competition, it can be rural part of the country, and an  integrated pest management program  viewed as a proxy for reduced the use of chemical pesticides.  contribution to development. However, profitability derived from economic distortions or unsustainable environmental and social practices, and the like, is less compatible with development.b IEG also showed that a focus on poverty and inclusion need not come at the expense of profitability (IEG 2011). And IFC’s interventions can be simultaneously pro-growth and pro-poor, but this link is neither universal nor automatic. Instead, it has been IFC’s focus on the developmental contribution of its investments that has assured a high coincidence of the two. 28 CHAPTER 2 WORLD BANK GROUP OPERATIONS The positive association between environmental and social sustainability and project business success is less pronounced, indicating a weaker alignment between the interests of financiers and those of the broader society. IEG’s analysis also showed that there is a positive association between sustainability and IFC’s investment outcome. However, there are a significant number of cases where profitable companies do not show good stewardship, and the reverse. Overall, results show both achievements and continued challenges for IFC in promoting good environmental and social practices (Figure B2.1.2). Achieving profitability for IFC and development Figure B2.1.2. Profits and E&S effects: associated, with numerous exceptions outcomes. IFC profitability Example: Profitable, poor  is essential for its stewardship:  sustainability and An IFC‐funded project in Central  accomplishment of its America was financially successful,  benefitting from a long upturn in  mission, but is not enough to commodity prices. However, the  assure its impact on the twin company did not act on identified  goals of the World Bank E&S issues. Wastewater treatment  and community health and safety  Group. As shown above, 13 issues were a continuing concern.  percent of developmentally favorable IFC projects were not profitable, and 16 percent of profitable projects were not environmentally or socially sustainable. As a socially and environmentally responsible development institution, it is vital for IFC to balance the “triple bottom line” instead of focusing single-mindedly on profits. Evaluation data show that IFC has generally not supported projects where there was a tradeoff between project development results and IFC profitability—only 13 percent of evaluated projects have high IFC investment outcome and low development outcome. Yet this is crucially dependent on the IFC’s decisions seeking a coincidence of business success, development impact and environmental and social sustainability. Source: IEG. a. IFC Articles of Agreement, Article III, Section 3. b. This is why IFC’s evaluation system for private sector investment projects captures not only profitability, but also the effects of economic distortions, externalities, and governance issues in the four sub-indicators that are synthesized into its rating or development outcome. DEVELOPMENT EFFECTIVENESS OF IFC ADVISORY SERVICES GREW STRONGER IN 2014 The success rate of development effectiveness of advisory services met the IFC corporate target for the first time since 2009.7 Performance improved most in Latin America and the Caribbean and East Asia and Pacific Regions, but declined in the MNA Region. Among the four business lines, performance of Public and Private Partnership (PPP) Advisory projects improved the most thanks to the stronger political will of the government and stakeholders, and better country conditions. The success rate of Sustainable Business Advisory (SBA) increased overall, but was hampered by 29 CHAPTER 2 WORLD BANK GROUP OPERATIONS weaknesses in project preparation, design, and M&E, and the performance of SBA projects in Africa. The success rate for Access to Finance (A2F) remained above the IFC average; in-depth capacity building efforts for single financial institutions showed better results than those catering to several entities under one project umbrella. Investment climate (IC) success rates improved, yet M&E shortcomings constrained IEG from rating impact in 44 percent of successful projects (figure 2.6). Figure 2.6. IFC Advisory Services Success Rate, FY08–13 (percent) a. By business line b. By sub‐indicator 90 90 80 80 70 70 60 60 50 50 40 40 30 30 20 20 10 10 0 0 FY08–10 FY11–13 FY08–10 FY11–13 FY08–10 FY11–13 FY08–10 FY11–13 FY08–10 FY11–13 FY08–10 FY11–13 FY08–10 FY11–13 FY08–10 FY11–13 FY08–10 FY11–13 A2F IC SBA PPP DE Strategy Output Outcome Impact Source: IEG. Note: A2F = access to finance; IC = investment climate; SBA = sustainable business advisory; PPP = public-private parternship; DE = development effectiveness; Strategy = strategic relevance; Output = output achievement; Outcome = outcome achievement; Impact = impact achievement. Impact assessment is constrained by poor M&E quality and M&E limitations in advisory services projects. Since tangible, measurable impacts take longer and are more difficult to achieve than project outcomes, it is understandable that project impact success rates (35 percent) are lower than project outcome success rates (62 percent) ) at evaluation. Impacts are observable earlier for some business lines (for example, 81 percent of A2F projects compared with 40 percent of PPP projects). Nevertheless, for 12 percent of the successful projects, IEG could not verify impact ratings due to M&E shortcomings—in IC and SBA, this ratio was 30 percent. MIGA GUARANTEES CONTINUED TO ACHIEVE RELATIVELY STRONG DEVELOPMENT OUTCOMES MIGA projects achieved an overall success rate of 70 percent in FY08–13 for development outcomes (figure 2.7). Evaluated projects in the agribusiness, manufacturing, and general services sector performed strongly, even though the majority was implemented in Africa, a Region with below average overall performance (65 percent). Projects in IDA countries slightly outperformed those in IBRD countries (74 30 CHAPTER 2 WORLD BANK GROUP OPERATIONS percent versus 67 percent, respectively) as results in IBRD reflected weaknesses in financial sector projects in the Europe and Central Asia Region and several poor performing infrastructure projects in middle-income countries. Driven almost entirely by the poor performance of several MIGA-supported banking institutions in the Europe and Central Asia Region during the financial crisis, projects in the financial sector had the lowest success rate (64 percent). This reflects several weaknesses: poor financial performance as a consequence of increased country risk due to the financial crisis and specific characteristics of the financial institutions; a contraction of the loan portfolio in some of the institutions instead of the expected expansion; high leverage in some projects; and shortcomings in the environmental and social performance in some cases. Figure 2.7. MIGA Development Outcome Success Rate by Sector, FY08–13 (percent) 100 80 60 40 20 0 AMS (n=13) Financial (n=15) Infrastructure Mining (n=1) MIGA Overall (n=14) Source: IEG data. Notes: AMS = Agribusiness, Manufacturing, and Services. n = number of projects. Projects supported by the Small Investment Program (SIP) had poorer results. IEG’s programmatic evaluation of SIP in FY14 found that the program was highly relevant for expanding MIGA support in priority areas (for example, IDA, Sub-Saharan Africa, FCS, and South-South investments), but the financial and economic viability of the projects for which IEG was able to determine outcomes faced country risk (location in FCS countries), company risk (small firm size), commercial risks (cost and demand projections, lack of local infrastructure), and client risk (client quality and capacity). The development outcome of SIP projects is thus uncertain: although MIGA devoted significant resources to underwriting SIP projects, only two out of fourteen projects for which outcome data were available had clearly achieved positive development outcomes. IEG recommended that MIGA rethink and radically reform its support to small investments to improve performance and impact. 31 CHAPTER 2 WORLD BANK GROUP OPERATIONS Bank and IFC Investment Portfolio Performance can be Improved by Addressing some Work Quality Issues The proportion of projects rated MS+ for Bank performance8 and IFC work quality declined during the past few years, driven mainly by deterioration of quality at entry (figure 2.8). Previous work by IEG (IEG 2013a, 2013b) and ongoing World Bank Group research (Geli, Kraay, and Nobakht 2014; Denizer, Kaufmann, and Kraay 2013) identified a strong statistical correlation between factors that are within management’s control during project preparation and implementation and the project’s ultimate development outcome. These factors are rated in IEG portfolio reviews under Quality at Entry (QAE) and Quality of Supervision (QOS) for Bank-financed projects, and under Screening, Appraisal, and Structuring, Supervision and Administration, and IFC Roles and Contribution for IFC investment projects. Also, IEG assesses IFC’s environmental work quality as an input to all other IFC work quality ratings. Figure 2.8. Bank and IFC Performance Success Rate, FY00–13 b. IFC performance (percent) a. Bank performance (percent) 90 85 90 80 85 80 75 75 70 70 65 65 60 60 55 55 50 50 45 40 Bank performance Quality at entry IFC work quality Screening, appraisal & structuring Quality of supervision Supervision and adminstration IFC role & contribution Source: IEG. Note: World Bank and IFC performance ratings are not comparable due to differences in evaluation methodology. The figures above are meant to show the trends in the two institutions. BANK PERFORMANCE IS DRIVEN BY WORK QUALITY FACTORS Textual analysis of the relevant sections of 109 Project Performance Assessment Reports (PPARs) covering 203 Bank projects9 that were completed by IEG between FY08 and FY13 identifies key factors that affect QAE and QOS ratings.10 Quality at Entry Ratings are Strongly Affected by Application of Past Lessons, Effective Risk Mitigation, and Design of Objectives and Results Frameworks Regression analysis carried out on data generated through the application of text analytics to PPARs shows that drawing on past lessons at entry, effective risk mitigation, 32 CHAPTER 2 WORLD BANK GROUP OPERATIONS M&E design, and design of objectives and results frameworks are powerful factors both in magnitude and statistical significance in the context of QAE rating. These results broadly hold when the factors are regressed against development outcome. The most frequently identified factors that positively affect Bank performance include active use of lessons learned from earlier engagement in project design, adequate attention to technical issues, and good relationships with stakeholders (figure 2.9). In 41 percent of all projects, the application of lessons from earlier projects in a series, earlier projects within a given country or region, or projects implemented by the Bank more generally 11 had a positive influence at new project design. Among the 148 projects rated positively for overall development outcome, 71 (48 percent) successfully used past lessons in project design compared with just 8 (15 percent) of the 55 projects where overall outcome was adjudged to be unsatisfactory. Other factors found to positively contribute to project design include adequate initial analysis of critical technical aspects, such as procurement or safeguards, and good working relationships with key stakeholders, including the client, agencies, civil society, and others. Almost half of well-designed projects (QAE rated MS+) incorporated past lessons or paid adequate attention to technical aspects of project design, compared with less than a quarter of the projects rated moderately unsatisfactory or worse (MS-) at entry. There are more examples of combined positive factors in well-designed projects than in poorly designed projects. 33 CHAPTER 2 WORLD BANK GROUP OPERATIONS Figure 2.9. Factors Influencing Quality at Entry Ratings Source: IEG. Note: M&E = monitoring and evaluation; NEG = negative; POS = positive. The most frequently identified shortcomings in Bank support at entry are deficiencies in M&E design, failure to adequately consider technical aspects in project design, and poor definition of objectives and the results framework. The prominence of poor M&E confirms the consistently poor ICR review ratings (30 percent MS+) for World Bank projects in that regard, even in projects that are rated positively for QAE. Of the 131 PPARs that included a rating for M&E, M&E was rated substantial or high in 49 (37.5 percent) instances. These projects were much more likely to be rated positively for QAE (90 percent versus 64 percent for projects with modest or negligible M&E) and for overall outcome (94 percent versus 62 percent. However, overall, the identified deficiencies were more frequently found in projects rated poorly for project preparation; this is especially true when several negative factors are co-present. 34 CHAPTER 2 WORLD BANK GROUP OPERATIONS Some less-frequently-observed factors are still important since they are closely associated with positive or negative QAE ratings. These include analysis of the political economy, ensuring government commitment, and to a lesser extent, M&E design; well- developed objectives and results frameworks; and good risk mitigation (bottom right quadrant of figure 2.9). Where these factors are present, they are almost invariably associated with a positive QAE rating. Similarly, some factors such as failure to mitigate for risks and poor analysis of the political economy—although they occur infrequently— are readily associated with projects rated MU. Most of the well-designed projects that were found to use past lessons more frequently draw on lessons embedded in specific country context and less frequently on generalized (that is, regional and global) lessons. Lessons from experience or analytical work that are specific and easier to operationalize appear to be most relevant. The design of the 2005 Productive Safety Nets Project APL I in Ethiopia was based on different pieces of analytical work, including one on the role of a productive safety net in Ethiopia, and another that involved a risk and vulnerability assessment for Ethiopia. Both helped define the project scope and overall approach, moving away from emergency or crisis management to a more stable, predictable safety nets regime, and helped identify capacity limitations that would likely impact project implementation. The PPAR notes “the World Bank had enough experience to be familiar with the operation requirements of a cash-based public works program,” which complements specific knowledge acquired from analytical work undertaken in Ethiopia. Adequate technical analysis—including the adequacy of economic analysis, fiduciary arrangements, and other project or sector-specific technicalities—is an important contributing factor in achieving positive QAE ratings. For example, in the 2001 Gujarat State Highways Project in India, several elements of a good technical analysis were present. At entry, the Bank’s support contributed to robust economic analysis through realistic cost estimates for civil works, and improved procurement practices through the consideration of competitive tendering in the arrangements; at the same time, the engineering, environmental, and social factors associated with building the highway were detailed and proper. Strong technical analysis, together with other positive factors like use of lessons from similar projects in the country and strong collaboration with the state government and other stakeholders, resulted in a satisfactory rating for Bank support at entry. Typically, both positive and negative factors are present in every project, but positive factors outweigh negative factors in projects with good QAE ratings. For example, both positive and negative factors are found in the 2000 Health Sector Reform project in Lesotho (rated Satisfactory for QAE). The project team carried out appropriate analytical work (Institutional Capacity Assessment, Health Sector Review) and engaged in wide 35 CHAPTER 2 WORLD BANK GROUP OPERATIONS consultation with stakeholders (including donors and political opposition). The team conducted political economy analysis, which drew lessons from the previous Population, Health, and Nutrition Project (1989–1998) to ensure a focus on government commitment and institutional strengthening. However, there were shortcomings as well. The project design suffered from too much focus on outputs instead of outcomes, insufficient attention to developing the relationship between the government and a key non-governmental provider, and insufficient attention to the ongoing professionalization of civil servants. But on balance, the positive aspects of the effort were judged to have significantly outweighed the negative, more peripheral aspects for this project. Negative factors tend to concentrate in projects with poor QAE ratings. The 2000 Klaipeda Port project in Lithuania (rated highly unsatisfactory for QAE) illustrates the much less nuanced composition of a project rated poorly at entry. The PPAR identifies a failure to take referenced lessons on board regarding the ability of poor communities to implement complex projects, and finds that there was no analytical work documenting issues arising with earlier strategies (to support the new strategy of formal transfers of schools to community management), and no assessment of important risk factors. Systemic obstacles were not identified and objectives were poorly articulated. Quality of Supervision Ratings are Strongly Affected by the Bank’s Problem-Solving Abilities, as well as the Regularity of Missions and the Attention to Corrective Actions Regression analysis carried out on data generated through the application of text analytics to PPARs supports the high level of statistical significance and importance in magnitude of a number of variables under QOS, including, primarily, positive and negative problem solving. It also references the benefit of regular missions and the negative impact of the failure to take corrective action and to use persuasion in support of projects. The most important factor in Bank performance during supervision is the capacity of the Bank team to solve problems encountered during project implementation (figure 2.10). This refers to real time engagement with a wide range of possible issues (for example, procurement, financial management, or staffing) that can impede progress and influence impact.12 The quality of relationships with stakeholders, the regularity and frequency of missions, and the expertise of the Bank team are also found to contribute positively. Inability to overcome implementation difficulties—failure to address problems—is the single most important factor for poor quality of supervision. Once present, it tends to pull supervision into an MU rating. 36 CHAPTER 2 WORLD BANK GROUP OPERATIONS Figure 2.10. Factors Influencing Quality of Supervision Ratings Source: IEG. Note: NEG = negative; POS = positive; TTL = Task Team Leader A key support during project implementation is the extent to which the Bank team identifies appropriate solutions to a wide range of problems that may arise. For example, there may be deficiencies in project design or exogenous shocks that affect project assumptions and estimates. The Bank may also be required to take more extreme measures and restructure the project mid-way, or to intervene to clarify matters and persuade the government or other stakeholders when miscommunication, disagreements, or constraints arise. However, the Bank’s ability to solve problems is the dominant trait in projects with satisfactory supervision quality. During the 37 CHAPTER 2 WORLD BANK GROUP OPERATIONS implementation of the 2002 Social Safety Net Project in Jamaica (rated Satisfactory for QOS), the Bank addressed gaps in design and implementation readiness by fine-tuning M&E deficits (outcome indicators and targets) and helped the implementing agency strengthen its procurement, financial management, and audit functions. The 2002 Brazil Family Health Extension APL1 (rated highly satisfactory for QOS) faced country capacity constraints and high staff turnover among project counterparts when the government changed. In response, the team intensified supervision efforts, involving the task team leader and senior operations officers based in Brasilia. That responsiveness to unforeseen events, together with timely restructuring of the project in response to requests from the new government, contributed to the good QOS rating. In projects with good QOS ratings, problem solving is frequently supported by other positive factors such as constructive working relationships with key stakeholders and regular, timely missions. The PPAR for the 1999 National Functional Literacy project in Ghana (rated satisfactory for QOS) finds that the positive relationship among the project team, the government counterpart, and the implementing agency not only helped the Bank team identify capacity limitations, but also facilitated close interaction to resolve issues related to procurement. The Bank (project team, country director, and others) also acted as advocates for the project in a difficult political context that threatened the project’s future. The PPAR for the 2000 Privatization Support operation in Bulgaria (rated highly satisfactory for QOS) emphasizes positive relationships (strengthened by missions) and problem solving, which gave the Bank credibility to play a convening role. In this case, the Bank provided technical support for the negotiation of privatization agreements and helped clarify stakeholder expectations by exercising its objectivity, independence, and convening power. In poorly supervised projects, problems arising during implementation are often neglected because of inaction in several areas. The PPAR for the 1997 Planning for Adaptability to Climate Change project in the Caribbean (rated unsatisfactory for QOS) notes the Bank’s poor problem-solving efforts—slow reaction to evident problems (disbursements, performance, fiduciary issues) caused delays, waiting until the mid- term to take action when the problems were “too obvious to ignore.” Infrequent missions (in this case, seven in five years) also suggest an overall lack of Bank attention to this project. The PPAR for the 2000 Agricultural Technology and Rural Education APL in Nicaragua (rated unsatisfactory for QOS) notes inadequate problem solving evidenced by lack of Bank advocacy to establish an M&E system at the relevant Ministry, and poor management of internal personnel change that impacted the Bank’s credibility and ongoing relationships with donors. Furthermore, when the government diverted funds to a program that was not linked to project objectives, no corrective action was taken. 38 CHAPTER 2 WORLD BANK GROUP OPERATIONS Again, some factors that are infrequently mentioned have significant influence on QOS ratings. The positive use of persuasion and continuity of staff involvement, for example, are closely associated with positive QOS ratings, whereas infrequent missions and poor relationships with stakeholders tend to have a pronounced negative effect on QOS ratings. UP-FRONT WORK QUALITY IS THE MOST POWERFUL DRIVER OF IFC INVESTMENT PROJECT SUCCESS13 Ratings for the quality of IFC’s up-front work have fallen significantly from their peak in 2009. The decline corresponds to a period of rapid growth and frequent institutional change, including the transformation of IFC’s regional departments into strategy and business development units with almost all of their staff in the field, and the departure of more than one hundred IFC staff when skills requirements changed (figure 2.11). IFC, as indicated in their comments on RAP 2013, launched various initiatives to improve work quality, focusing on increasing IFC’s local knowledge base,14 but the effect has not been evident in project evaluations. Textual analysis of the Expanded Project Supervision Reports and associated IEG validations (Evaluative Notes) completed in 2011–13 (covering 232 IFC projects) shed more light on the factors affecting IFC work quality.15 Figure 2.11. Projects with Low Up-Front Work Quality Approved During a Period of Rapid Change Source: IEG. Risk identification and client assessment are the most frequently cited work quality factors.16 Risk identification is reported as a work quality factor in nearly 80 percent of 39 CHAPTER 2 WORLD BANK GROUP OPERATIONS projects evaluated during 2011–13 (table 2.1). For projects rated high for overall work quality, good risk identification was reported almost 90 percent of the time, compared with only 54 percent of the time for projects with low overall work quality. Risks identified in project documents often include project risk (such as completion, technology, cost overrun, cash generation, and so on), country risk (for example, macroeconomic, regulatory, political, exchange rate/currency, and others), market risk (for example, competition, customer concentration, cyclical prices, sales seasonality, and the like), and client/sponsor risk (such as capacity, commitment, reputation, financial strength, strategic interests, and so on). Poor client assessments were a factor in almost half of the projects with low overall work quality ratings, compared with only 7 percent of projects with high overall work quality ratings, indicating the importance of a strong client assessment by IFC. In contrast, E&S factors were generally positive whether or not projects are rated high or low for overall work quality. Table 2.1. Frequently Identified Work Quality Factors High WQ Rating (n=148)  Low WQ Rating (n=58)     No. of  Identified  Rank  No. of  Identified  Rank  Difference  Work Quality Factors Identified in  projects   as positive  projects   as positive  (%)  Project Documents  (%)  (%)  Risk identification  120  88  1  52  54  1  34  Client assessment  92  88  5  46  41  2  47  E&S supervision and follow‐up  98  90  2  37  95  3  5  IFC team advice  95  88  3  34  59  5  29  E&S client engagement and follow‐up  94  99  4  28  89  8  10  Source: IEG. Notes: Work quality factors are listed in the order of their overall rank for positive occurrences; Rank = order of the number of times a factor was identified as positive with the largest number being 1; E&S = environmental and social; IFC = International Finance Corporation; WQ = work quality; n = number of projects. Different work quality factors are observed with different frequencies across clusters. The MAS cluster was more likely to report factors related to client engagement and E&S issues, including E&S supervision. FIG reported most frequently on factors related to providing advice and assessing clients. INFRA focused mostly on factors associated with client relationships and engagement around E&S issues. In general, all clusters focused on work quality factors related to client engagement, which is an important element in IFC’s refocus. Up-Front Work Quality is Most Affected by Risk Mitigation Up-front work quality is driven by IFC’s success in mitigating risk. For projects with high up-front work quality where risk mitigation was a factor, it was done appropriately in 98 percent of cases, compared with 45 percent for projects that were rated low for up-front work quality (table 2.2). Regression analysis shows that risk mitigation for IFC projects, when it was identified as a work quality factor (whether positive or negative), was strongly 40 CHAPTER 2 WORLD BANK GROUP OPERATIONS correlated with up-front work quality ratings, and statistically significant in the context of the development outcome rating. Risk identification, though strongly correlated with up- front work quality, was not statistically significant in the context of development outcome. Therefore, IFC must go beyond risk identification to mitigation to improve up-front work quality (table 2.3).17 Table 2.2. Frequently Identified Factors Associated with IFC’s Screening, Appraisal, and Structuring Rating             High SAS Rating (n=117)  Low SAS Rating (n=89)     No. of  Identified  Rank  No. of  Identified  Rank  Difference  projects  as   projects  as  (%)  Work Quality Factors Identified in  positive  positive  Project Documents  (%)  (%)  Risk identification  91  95  1  76  59  1  36  Client assessment  71  99  2  58  50  2  49  E&S assessment, analysis, and  48  94  5  44  75  3  19  categorization  Risk mitigation  56  98  3  35  46  7  52  Market and country  48  94  6  42  55  4  39  assessment/analysis  Source: IEG. Note: Work quality factors are listed in the order of their overall rank for positive occurrences; Rank = order of the number of times a factor was identified as positive with the largest number being 1; E&S = environmental and social; SAS = screening, appraisal, and structuring; n = number of projects. Table 2.3. From Risk Identification to Risk Mitigation Risks Identified  Mitigation  Result  Sponsors of a fund were financially  Supplement expertise through capacity  The external fund manager initiated a  strong, but operationally weak.   development and with the assistance of  new fund and did not engage fully with  an external, experienced fund manager.  the IFC client. Four of nine initial deals  performed poorly.  Risks associated with the regulatory  A rapid upgrade and expansion was  Competition was much more intense  and competitive environment were  expected to mitigate risks associated  than expected. Changes on the  identified for a mobile phone  with the project.  regulatory side were unpredictable  upgrade and expansion project.   and adversely affected the project.   IFC mainly relied on SEC  IFC did not conduct a thorough  The “poison pills” reduced  requirements to reduce the risk of its  corporate governance due diligence,  management’s incentive to improve  equity investment in an integrated  and later learned that the project had  operations and reduced IFC’s options  gas‐to‐power project.   adopted anti‐takeover protection to  on how to deal with the problem.  prevent replacement of management  by active investors.   IFC recognized that the EP contractor  An experienced local firm was to be  The change of plans resulted in a 2.5  for a run‐of‐the‐river hydro power  hired to complete the work, but a deal  year delay in project start‐up and a 50  plant did not have the experience to  could not be reached, and the EP  percent project cost overrun.   complete a complex operation.   contractor took on the work.   Source: IEG. Note: EP = Engineering Procurement Construction. 41 CHAPTER 2 WORLD BANK GROUP OPERATIONS IFC’s Overall Supervision Work Quality Remained High The success rates of IFC’s supervision quality remained above 80 percent since 2004–06 (table 2.4), but can still be improved. Specific factors most often identified for IFC’s project supervision are desk supervision and E&S supervision and follow-up. Weak client contact is often associated with poor IFC supervision quality, whereas positive client contact was associated with nearly half of IFC projects with good supervision quality. The examples in table 2.5 show that strong IFC supervision quality can help IFC anticipate and manage issues related to developmental, environment and social, and financial aspects of the project. High-quality supervision may also lead to follow-on investments. Poor supervision, including insufficient contact with IFC clients, perfunctory reviews or pursuit of client reports, and inadequate follow-up by IFC, could result in reputational risk, missed opportunities, and sub-optimal financial and developmental results for IFC. Table 2.4. Role of Client Contact for IFC Supervision Quality Positive SUP Rating (n=173)  Negative SUP Rating (n=33)     Work Quality Factors Identified in  No. of  Identified as  Rank  No. of  Identified as  Rank  Difference  Project Documents  projects   positive (%)  projects   positive (%)  (%)  Supervision—general  85  99  1  18  39  1  60  E&S supervision and follow‐up  82  93  3  16  81  2  12  Supervision—client contact  83  100  2  14  57  3  43  Supervision—desk  62  97  4  14  64  4  33  Client / partner relationship  62  98  5  9  78  5  20  Source: IEG. Notes: Work quality factors are listed in the order of their overall rank for positive occurrences; Rank = order of the number of times a factor was identified as positive with the largest number being 1; SUP = Supervision and administration; E&S = environmental and social; n = number of projects Table 2.5. Examples of Different Supervision Practices on Development and Investment Outcomes Factors  Positive Example  Negative Example  Desk  The team stayed informed about the sub‐ IFC's supervision seems lacking in care and diligence.  supervision  projects' and the Bank's performance in all  To illustrate, sections of PSRs issued during four years  material areas. The Bank's PSRs and CRRs  were identical, although the circumstances of the  analysis accurately monitored the its financial  project underwent material change. IFC may have  performance, main risks, and events, and  missed out on the collateral benefits of proactive  properly examined the country’s economic and  supervision, such as gaining market intelligence, and  political situation.  access to investment opportunities in the junior mining  sector—all of which was part of the rationale for IFC to  invest in this project.  Client contact  Supervision included regular site visits and  Supervision was from a nearby country office, but the  meetings with senior management, at least  relationship was managed from headquarters. The  twice a year. As a result, IFC was always aware  portfolio officer had little to no direct contact with the  of the evolution of the company's operations  client. A subsequent change in supervision  and its performance, and could identify follow‐ responsibility and location helped identify  on investments.  deterioration in the Bank, which lost more than $20  million in only 15 months.  42 CHAPTER 2 WORLD BANK GROUP OPERATIONS E&S  IFC's supervision resulted in the identification of  AMRs submitted by the client were below average in  supervision  additional areas for improving the company's  quality and contained data and information  and follow‐ social and environmental performance (for  deficiencies (for example, reporting on air emissions  up  example, labor conditions for workers engaged  and lost time accidents). AMRs were not duly  on dredgers), and the company accepted this  reviewed. Overall, IFC did not diligently follow up with  feedback.  the client through site visits and by directly soliciting  information.  Sources: Various IFC self evaluations and IEG evaluative notes for IFC investment projects. Notes: AMR = Annual Environmental and Social Monitoring Report; CRR = Credit Risk Rating; PSR = Project Supervision Report. IFC Role and Contribution are Affected by a Wide Range of IFC Activities IFC’s role and contribution rating18 had few work quality factors (table 2.6). High ratings for IFC’s role and contribution are most frequently associated with positive engagement on environmental and social issues and providing advice. These include a wide range of IFC activities such as the provision of technical expertise to assist clients with decisions related to technology, environmental, and social considerations, providing advisory services to help clients achieve global standards, assisting in the development of management and technical expertise, setting up effective governance structures, and being on the client’s board of directors. Low IFC role and contribution ratings are applied when IFC falls short in a material area, or there is no plausible additionality associated with IFC’s involvement. IEG’s factor analysis did not identify specific factors that were frequently associated with low role and contribution ratings. Table 2.6. Positive IFC Engagement on Environmental and Social Issues High R&C Rating (n=155)       Low R&C Rating (n=51)     No. of  Identified  Rank  No. of  Identified  Rank  Difference  Work Quality Factors Identified in  projects  as positive  projects  as positive  (%)  Project Documents  (%)  (%)  E&S client engagement and follow‐up  76  99  1  15  87  1  12  IFC team advice  67  91  2  15  67  2  24  IFC team corporate governance  34  82  3  11  45  3  37  Client‐partner relationship  30  100  4  2  50  10  50  E&S supervision and follow‐up  22  86  5  9  100  4  14  Source: IEG. Notes: Work quality factors are listed in the order of their overall rank for positive occurrences; Rank = order of the number of times a factor was identified as positive with the largest number being 1; R&C = role and contribution; n = number of projects. Performance of World Bank Group Country Programs Improved Recent Bank Group country program performance is on an upward trend. The seeming discrepancy between portfolio and country program results is driven by many factors, including the fact that in any given year, fewer than 20 percent of the evaluated projects were from the reviewed country programs. Performance of country programs also depends on previously closed projects, the active portfolio, non-lending products, and external factors. 43 CHAPTER 2 WORLD BANK GROUP OPERATIONS PERFORMANCE OF BANK GROUP COUNTRY PROGRAMS HAS IMPROVED SINCE 2010 In FY14 the Bank Group satisfactorily completed 76 percent of the country strategies, meeting the corporate target for the first time since 2009. The improvement reflects steady progress among both IBRD and IDA countries since 2010, when the impact of the financial crisis led to an extremely low success rate for Bank Group country programs— just 28 percent19 (figure 2.12). Programs in IBRD countries continue to achieve better results than those in IDA countries, with the gap between the two groups remaining above 25 percentage points. Performance of the FCS programs improved sharply in FY12–14, and the impressive progress was accompanied by an increase in the number of reviewed FCS country programs in FY13–14.20 Yet the relatively low number of reviewed FCS programs (the most was six in FY13), and Bank Group country programs in general (the most was 22 in FY14), remain an important caveat in the analysis. Figure 2.12. Success Rates of World Bank Group Country Strategies, FY05–14 (percent) a. Overall Outcome b. Bank Performance 100 100 80 80 60 60 40 40 20 20 0 0 Bank IBRD IDA FCS Bank IBRD IDA FCS Source: IEG data. Notes: IDA includes Blend countries, all of which received more resources from IDA than from IBRD, and non-FCS IDA countries; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; FCS = fragile and conflict-affected states. IEG reviews of the Country Partnership Strategy Completion Reports for FCS programs in FY12–14 shed light on the good outcome and performance achievements in FCS countries. 21 Good supervision and responsiveness to crisis and other changing circumstances are key strengths that are underscored in 7 of the 10 FCS programs. However, these efforts did not always lead to reduced portfolio risk, and procurement was often raised as an issue. Good partnership with governments and other donors was another notable strength for the FCS programs. However, program design remained problematic: although the strategic objectives were relevant and well aligned with the countries’ own goals, in most cases (8 out of 10) they were overly ambitious, diffuse, and did not adequately reflect the post-conflict situation, even though in three countries 44 CHAPTER 2 WORLD BANK GROUP OPERATIONS selectivity improved after the mid-term reviews. Complex project/program design was noted in most programs, and nearly all results frameworks were weak. FURTHER IMPROVEMENT CAN BE ACHIEVED BY STRENGTHENING SELECTIVITY OF BANK GROUP COUNTRY STRATEGIES BASED ON CLEAR RESULTS CHAINS To operationalize the twin goals, the Bank Group introduced the Country Partnership Framework (CPF) as a more evidence-based, selective country engagement model. The new architecture seeks to address recurrent concerns about relevance, selectivity, and weak results chains. Findings from IEG’s recent research into these issues can inform the process underway (IEG 2014a). Although selectivity has been a long-standing concern for the Bank Group, there is no common understanding of what defines selectivity or how to achieve it. Concern with selectivity in the Bank goes back at least to Operations Policy and Country Services’ 1998 Country Assistance Strategies: Retrospective and Outlook. A majority and increasing number of country strategies in FY09–13 discussed selectivity when addressing the upcoming country engagements. The definitions of strategic selectivity, however, were typically based on qualitative judgments, leading to divergent criteria for defining and exercising selectivity across the Bank Group. More than 30 different definitions were identified in the strategies during the five years, and the Country Partnership Strategies (CPSs) often define selectivity in more than one dimension, making it difficult to monitor progress in the exercise of selectivity. Selectivity, or lack of it, is often identified by IEG as a key factor affecting the outcomes of the Bank Group’s country engagement. In one of its earliest Country Assistance Evaluations (CAE), the 2000 Tanzania CAE, IEG showed that the lack of emphasis on strategic selectivity was a major weakness in the Bank’s support program. More than a decade later, in its most recent Country Program Evaluation (CPE), the 2014 Tunisia CPE, IEG discussed at length why the Bank Group program’s unsatisfactory outcomes were attributable to a lack of selectivity. Also, IEG’s reviews of the CPSCRs frequently refer to selectivity as a driver of results. For example, the FY14 reviews for Georgia and Rwanda explicitly referenced selectivity as a key factor for the satisfactory delivery of results. Controlling for other factors, statistical analysis confirms that selectivity, as measured by the number of objectives in a strategy and the number of sectors in which the Bank Group engages during a strategy period, is positively correlated with the performance of Bank Group country programs. It is especially important in countries with high levels of extreme poverty. Despite efforts to sharpen selectivity in country strategies, the Bank Group found it difficult to achieve greater selectivity in practice. In the 200 CPSs prepared during FY05– 14,22 the number of stated objectives varied widely from zero (Mexico 2008) to 53 45 CHAPTER 2 WORLD BANK GROUP OPERATIONS (Morocco 2010), with 50 percent falling between 10 and 18. There is no discernible change in the sector spreads of Bank Group country programs during this time. In countries with two or more CPSs, there were almost as many instances where the number of objectives increased as those where the number decreased, and most countries with three CPSs experienced a reversal in the trend (figure 2.13). Efforts to improve selectivity sometimes focused only on reducing the number of objectives and resorted to “relabeling” or “regrouping” the objectives without a corresponding streamlining of the supporting programs. This proved to be unsustainable. Transforming an indiscriminate program into a selective one is not easy and may take longer than expected to implement, which is especially true where the circumstances encourage expanding engagement and where development needs are overwhelming. Figure 2.13. Changes in the Number of Objectives in Consecutive CPSs, FY05–14 Continuous Decrease ,  10% Decrease then  Decrease ,  Increase, 19% 27% Two CPSs Increase ,  23% Increase then  Decrease, 15% Continuous  Increase, 6% Source: IEG 2014b. Note: CPS = Country Partnership Strategy Weak results chains make it difficult to assess whether the proposed Bank Group program could actually deliver on the proposed objectives. Various CPEs conducted by IEG since 2005 point to a lack of realism in the formulation of CPS objectives. A key finding is that the absence of credible results chains further aggravates the tendency to underestimate the efforts needed to produce results. Currently, the CPS results frameworks essentially serve a results measurement function, which neglects their purpose as a tool for strategic decision making. Of 25 CPSs approved in FY14, the vast majority (90 percent) has measurable indicators; even if most indicators are only partially aligned with the corresponding CPS objectives23 (table 2.7). However, in just a little more than 40 percent of the cases is there a clear discussion of how pipeline projects by the Bank, IFC, and MIGA fit together to deliver results; there is even less discussion of how proposed Bank Group interventions 46 CHAPTER 2 WORLD BANK GROUP OPERATIONS complement or are complemented by other partners’ actions for achieving the CPS objectives.24 Such partial results chains make it difficult to identify the design and implementation risks associated with proposed Bank Group interventions, and thus decisions to mitigate these risks. Because the tendency is to underestimate the efforts required for delivering results, strategies tend to include too many objectives and/or set objectives at unrealistic levels. Table 2.7. Relatively Strong Results Measurement but Weak Results Chains in CPS Results Frameworks Results Measurement Results Chain  Category  Measurability of  Scope of  Synergy of Bank  Collaboration with  indicators  Indicators  Group instruments  partners  (percent)  (percent)  (percent)  (percent)  Absencea  1  1 32 18 b Partial presence   8  59 26 50 c Strong presence   90  38 42 32 NAd  1  2 0 0  Source: IEG. Note: The 25 CPSs approved in FY14 have a total of 304 objectives. For each objective, the strength of its results measurement system is assessed by the extent to which the CPS indicators are measureable and cover the full scope of the stated objective; the strength of its results chain is assessed by the clarity of the discussion on how World Bank Group instruments and partnerships will be harnessed to achieve the objective. a. The indicators are not measurable or have no baseline/targets; the indicators are not aligned with the stated objectives; no discussion of how selected World Bank Group instruments fit together; or no discussion of how the World Bank Group works with the government and other partners to achieve results. b. Some, but not all, of the key elements are provided. c. The indicators are measurable with all baseline and targets defined; the indicators have the appropriate scope for measuring achievement of the objectives; there is a convincing story of how World Bank Group instruments complement one another to deliver results; and there is a clear discussion of how the World Bank Group collaborates with partners, with the key assumptions identified. d. The indicators will be defined later. Selecting country engagement areas and setting strategic objectives in the absence of results chains frequently led unselective country programs with objectives set at inappropriate levels. IEG reviews of the CPSCRs found that many CPSs have too wide a scope and inappropriately defined objectives, either overambitious given the instruments available and the country context, or trivial for lack of a clearly defined mechanism to achieve broader, transformational impact. Finding the appropriate balance calls for an iterative process centered on the articulation of a results chain for each objective being considered. This process may reveal that the achievement of some objectives depends on assumptions that are not likely to be realized (for example, reforms that the government is not ready to undertake, donor funding that is highly uncertain). Such clarity would facilitate setting the objectives at more appropriate levels and/or selecting among potential engagement areas. Examination of consecutive CPSs shows that selectivity is best exercised when the CPSCR of the concluding strategy analyzes the Bank Group’s past experience with selectivity in the country context. The continuity across strategies allows for increasing clarity over time of what is truly 47 CHAPTER 2 WORLD BANK GROUP OPERATIONS important and how to prioritize. In this context, the new Systematic Country Diagnostic will have an important role to play in providing the objective evidence base to facilitate the process of prioritization and selectivity of country engagement. The Bank Group Implements IEG Recommendations over time The Bank Group and IEG are tracking 133 recommendations from 23 corporate and sector/thematic evaluations completed between FY10 and FY13 in the Management Action Record (MAR).25 IEG rated 35 percent of the recommendations as substantially adopted or better in the first year of follow-up, improving to 83 percent by the fourth year (figure 2.14). Annual ratings are influenced by progress on specific recommendations. Differences in ratings between IEG and the Bank Group decreased as time passed, and often arose from ambiguity on what constituted adoption. As part of MAR reform, IEG and Bank Group management are working to provide clarity on adoption of each recommendation.26 Figure 2.14. Adoption of IEG Recommendations (percent) 100 50 0 MAR Year I (n=133) MAR Year II (n=72) MAR Year III (n=50) MAR Year IV (n=12) IEG Management Source: http://mar.worldbank.org. Note: n = number of recommendations. The MAR reveals some of the challenges facing the Bank Group and the areas where improvements could be made. About one-third of the recommendations in the MAR were on designing and preparing Bank Group activities. Recommendations on Bank Group strategic directions and operational policies accounted for another one-third. Most of the recommendations related specifically to the sector or theme being evaluated. These challenges are being addressed since the outstanding issues tend to be sector/theme specific. 27 Similar recommendations are made across several evaluations over different years, which may point to common challenges for the Bank Group. Improvement in M&E was recommended by multiple evaluations covering different sectors and themes. Roughly 48 CHAPTER 2 WORLD BANK GROUP OPERATIONS one of every five recommendations included a reference to M&E at both the sector strategy and project levels. There was a shift in the recommendations as time passed, with the earlier evaluations calling for appropriate indicators, adequate baselines, and other basic improvements, though more recent recommendations are focusing on better outcomes and outcome indicators, and improving the impact of Bank operations. Underlying these recommendations is a need to get a better handle on the results chain to better understand the links among inputs, outputs, and outcomes. Other recommendations pointed to the importance of knowledge and learning in designing effective Bank Group activities through better diagnostics in project preparation, and exploiting synergies within the Bank Group and between different instruments. The response to these common challenges has been mixed. On M&E, management reported on guidance to staff (for example, a Good Practice Note on Results in Development Policy Lending), the relevant parts of the Bank’s overall effort to improve results reporting (such as the Corporate Scorecard and core sector indicators), and improvement to sectoral results frameworks (for example, gender, governance and anti- corruption). However, Management also considered that more needed to be done, generally rating itself lower than average for recommendation adoption. On learning, Management reported on the reviews and quality assurance it was conducting on project preparation, but did not address the question of incorporating learning, other than in agriculture. Finally, Management reported on various mechanisms it set up to further coordination and collaboration, but it is not sure what effect these had. References Denizer, Cevdet, Daniel Kaufmann, and Aart Kraay. 2013. “Good Countries or Good Projects? Macro and Micro Correlates of World Bank Project Performance.” Journal of Development Economics 105 (November): 288–302. Geli, Patricia, Aart Kraay, and Hoveida Nobakht. 2014. “Predicting World Bank Project Outcome Ratings.” Policy Research Working Paper No. 7001, World Bank, Washington, DC. IEG (Independent Evaluation Group). 2011. Assessing IFC’s Poverty Focus and Results. Washington, DC: World Bank. ―――. 2013a. Biennial Report on Operations Evaluation (BROE): Assessing the Monitoring and Evaluation Systems of IFC and MIGA. Washington, DC: World Bank. ―――. 2013b. Results and Performance of the World Bank Group 2013. Washington, DC: World Bank. ―――. 2014a. Selectivity in Country Strategies: the Evidence. ―――. 2014b. The Big Business of Small Enterprises. Evaluation of the World Bank Group Experience with Targeted Support to Small and Medium-Size Enterprises. 2006–2012. Washington, DC: World Bank.  1IEG evaluated IFC’s Global Trade Finance Program (GTFP) in 2013 and recommended that IFC adapt its method of reporting GTFP to reflect its distinct nature. In FY15 IFC plans to report 49 CHAPTER 2 WORLD BANK GROUP OPERATIONS IFC’s short-term fund business based on the average annual outstanding portfolio. AS GTFP will no longer report its commitment volume, rather its outstanding portfolio balance, it will no longer contribute to IFC’s Commitment Volume total. As such IFC’s reported commitment volume will only include long-term finance. 2The Bank has introduced a client feedback mechanism for AAA as part of the quality assurance process. 3The analysis uses the OPCS mapping to determine each closed project’s GP affiliation. Since further adjustments to the GP continue to be made, this analysis is meant to provide a baseline for future comparisons. 4 This report analyzes IEG evaluation ratings, not IFC’s Development Outcome Tracking System (DOTS) indicators. As a monitoring system, DOTS indicators are not independently validated, nor do they cover closed projects or have stringent quality control as do evaluations that are independently validated by IEG. The variance between DOTS indicators and IEG ratings grew from just over one percent (64.6 percent vs 63.3 percent) in CY2012 to 17.2 percent (74.1 percent vs 56.9 percent) in CY2013. Analysis of the reasons for this variance is beyond the scope of this report. 5For example, in Argentina none of the projects approved just prior to the 2001 crisis was successful. 6 IEG reviewed the evaluations for all projects with low Development Outcome and Project Business Success (PBS) ratings to determine the cause of the low PBS ratings. In only about 25 percent of the evaluations reviewed was a low PBS rating primarily caused by the global financial crisis. 7The Development effectiveness success rate for advisory services projects in 2013 was 69 percent compared to IFC’s corporate target of 65 percent. 8The decline in overall Bank Performance and, within it, the decline in QAE ratings between FY08–10 and FY11–13 is statistically significant at the 99 percent confidence level. 9Project Performance Assessment Reports (PPARs) are IEG’s most in-depth, project-level evaluations of World Bank projects. Some PPARs cover more than one project. 10 Appendix G provides a fuller explanation of the approach and methodology for this exercise. 11Of the set of projects reviewed, only two are noted to have incorporated findings and recommendations from IEG evaluations into their design: the 2001 Gujarat State Highways Project, and the 2003 Technical Engineering Education Quality Improvement Project stemming from a PPAR of a similar project in India and a study on technical education and vocational training in India, respectively. Lessons from IEG were codified as “other type” of lessons together with lessons from analytical and advisory work. Only 16 percent drew from experiences in previous projects as part of the series, perhaps reflecting the limited number of programmatic projects in the sample. 12The results of the text analysis of Quality of Supervision are less nuanced than those yielded through the analysis of Quality at Entry. This is mainly due to the higher quality of the data on Quality at Entry than those on Supervision, which in turn reflects the tendency of the PPARs to focus less on process-related issues even though they are fundamental to project management and supervision. It may also reflect the typical abundance of formal documentation available to 50 CHAPTER 2 WORLD BANK GROUP OPERATIONS evaluators concerning the entry phase, and the relative paucity of documentation associated with the ongoing supervision effort. 13Results and Performance of the World Bank Group 2013 (RAP) found statistically significant associations among external risks, IFC’s work quality, and the development outcome for IFC investments. The marginal effects of up-front work quality issues within IFC’s control were larger than external risk factors. 14Initiatives include the establishment of an operational hub in Istanbul, increasing IFC’s local presence, placing more experienced staff with global knowledge nearer clients, and hiring staff with local knowledge. 15When there were directional differences in the assessment of work quality factors by IEG and IFC, IEG’s assessment was used. 16 See Appendix H for a complete list and ranking of work quality factors. 17IEG recognizes that not all risks identified can be fully mitigated. As a developmental institution, IFC may consciously accept risks or choose not to fully mitigate risks in order to do business. 18Role and contribution captures three basic principles: Additionality/Special Contribution Principle, Business Principle, and Catalytic Principle. 19See RAP 2013 for more analysis of the crisis impact on World Bank Group country program performance. 20This indicates a greater use of the results-based Country Partnership Strategy instead of the Interim Strategy Notes by the World Bank Group to guide its engagement in FCS, since the Country Partnership Strategy Completion Reports (CPSCRs) are only prepared and reviewed by IEG for the concluding country strategies when the new country strategy is a CPS. 21In 25 percent of the CPSCR reviews, IEG rated Bank and IFC performance separately, while in all other cases IEG rated either Bank performance alone or World Bank Group performance for all three institutions together. The Joint IEG-Operations Policy and Country Services Methodology for CPSCR and CPSCR reviews provide guidelines for rating Bank, IFC, and MIGA performance together as One World Bank Group. 22Forty-five countries had one Country Assistance Strategy (CAS), 31 countries had two CASs, and another 31 countries had three CASs. 23Misalignment between CPS objectives and indicators is an issue frequently raised in IEG’s reviews of the CPSCRs, and has been a major cause of disagreement between IEG and management, who often assesses only whether the targets have been met. 24 Detailed IEG ratings of the results frameworks of the 25 CPS can be found in Appendix J. 25 IEG assesses the effectiveness of World Bank Group operations and makes recommendations on areas for improvement through it sector/thematic and corporate evaluations. It then follows up on the implementation of these recommendations annually, with World Bank Group Management reporting on the actions it took for each recommendation. Management and IEG then rate the level of adoption based on a four-point scale: High (fully adopted), Substantial (largely adopted, but not fully incorporated into policy, strategy, or operations), Medium (adopted in some operational and policy work, but not to a significant degree in key areas), and 51 CHAPTER 2 WORLD BANK GROUP OPERATIONS Negligible. IEG often agrees with Management’s ratings, but also “downgrades” a rating if it considers Management’s action to be inadequate. The data is presented by the year of follow- up, not calendar years. So the recommendations from an evaluation in its fourth year of follow- up would factor in all four years, while recommendation from an evaluation in the first year will only factor in Year One. Therefore, the number of recommendations included each year goes down, from 139 in Year One to 12 in Year Four. 26 World Bank Group Management and IEG agreed to have one common system across the Bank Group, clarified what constitutes adoption by specifying action and timelines for recommendation implementation, and introduced an electronic tracking system for the recommendation. The changes were implemented in FY12, and the tracking system was launched in FY13. 27 All the recommendations with less substantial adoption in the fourth (and last) year of follow- up in 2014 were in water. 52 Attachment: Regional Updates Africa REGIONAL CHALLENGES AND EVALUATION FINDINGS FROM WORLD BANK GROUP OPERATIONS IN THE REGION Sub-Saharan Africa continued to grow strongly in 2013. Output increased by 4.7 percent over 2012, supported by investments in the resource sector and in public infrastructure. Excluding South Africa, growth was 6 percent, well above the global gross domestic product (GDP) average of 2.4 percent. The Region is expected to remain one of the world’s fastest growing, with its GDP projected to expand above 5 percent in 2015–16. However, the Ebola outbreak severely disrupted economic activity in Guinea, Liberia, and Sierra Leone, causing their growth to slow.1 The poverty level remains high. In 2010, 48.5 percent of the population was living on less than $1.25 a day. To achieve the World Bank Group’s twin goals of ending extreme poverty and boosting shared prosperity, the Africa Regional Strategy focuses on two pillars: (i) competitiveness and employment, and (ii) vulnerability and resilience over the foundation of good governance and public sector capacity.2 The World Bank’s operational performance in the Sub-Saharan Africa Region has been consistently below the Bank average (Figure 1). However, in relative terms, the gap has narrowed due to the deterioration of portfolio performance in better performing Regions. The portfolio performance has been steady in the Region in recent years. At the same time, in a few sectors, such as Social Protection and Labor and Social, Urban, Rural and Resilience Global Practices, project performance surpassed Bank averages with substantive Bank involvement. Of the projects that exited the Bank portfolio during FY11–13, outcome of 64 percent (by number of projects) was rated moderately satisfactory or better (MS) by the Independent Evaluation Group (IEG) compared to a Bank average of 70 percent (Table 1). Over the same period, IEG prepared 19 Project Performance Assessment Reviews (PPARs) for the Africa Region, of which about half showed MS or better outcomes. Portfolio performance in the Africa Region by sector was uneven. Public Sector Governance was by far the worst performing: only 7 of 20 projects were rated MS or better. Other poorly performing sectors included Education as well as Finance and Markets. In contrast, outcomes in Social Protection and Labor; Social, Urban, Rural, and Resilience; and Macroeconomics and Fiscal Management were very good, exceeding Bank averages. For example, all eight exiting Social Protection and Labor projects were reviewed and rated by IEG as MS or better (Table 1). An important factor in relatively weak project outcome ratings was a steady decline in quality at entry. As in previous years, this finding was consistent in Implementation 53 ATTACHMENT: REGIONAL UPDATES Completion Report Reviews (ICRRs) and in PPARs. Projects failed due to over- ambitiousness and complexity; a poor assessment of country conditions and capacity (not recognized or well addressed in project design); and a deficient results framework. An example of poor design in the Public Sector Governance Sector was the 2008 Cameroon Transparency and Accountability Capacity Building Project. Limited consideration was given to sequencing reforms and change management aspects. The project design did not sufficiently take into account the implications of inadequate government commitment to reforms and the power of vested interests. Another example of a project with weak design and unsatisfactory quality at entry was the 2006 Ghana Multi-sectoral HIV/AIDS Program. It was characterized by an overly optimistic risk assessment, the lack of a clear link between development objectives and performance indicators, and the lack of a clear definition of responsibility for achievement of objectives. There was also no effort to design a robust monitoring and evaluation (M&E) system. Stakeholder consultations focused on potential suppliers of project interventions rather than on beneficiaries. M&E is essential for the delivery of results. However, the quality of M&E systems has been weak in the Region and Bank-wide. The percentage of projects with M&E rated high or substantial was 26 percent—only one project in every four was rated in the top half of the rating scale in the Africa Region, against a Bank average of 30 percent. For example, the 2003 Mauritania Multi-sector HIV/AIDS Control Project was designed with unrealistic and difficult-to-measure key indicators without taking into account the capacity of implementing agencies. An external agency was to be contracted for the M&E system, but this did not happen. A background analysis with a rapid assessment of targeted, high-risk groups and beneficiaries, meant to feed into the results framework, was initiated at the time of appraisal when it was too late to inform the project design on baseline indicators and outcomes. As a result, the project failed. The outcomes of two projects in two countries in different sectors strengthened the main evaluation finding that successful projects depend on good design. Both projects had satisfactory outcomes: the 2008 Rwanda Second Rural Sector Support Project and the 2008 Côte d’Ivoire Education and Training Project. In both cases, the results frameworks were logical and relevant. High efficacy was gained from the clear results chain and confirmed by impact assessments. Quality at entry benefited from several strategic studies funded by grants. In addition, good quality M&E systems contributed to the projects’ success. PPARs prepared during FY11–13 also provide ample evidence of poorly designed projects and subsequently less than satisfactory outcomes. But, in addition, these PPARs provide examples of how to avoid complexity and be realistic. The Gambia Gateway 54 ATTACHMENT: REGIONAL UPDATES Project and the Ghana Land Administration Project had moderately unsatisfactory (MU) outcomes. In both cases, the complexity of project design and implementation arrangements contributed to the failure to achieve the desired results. In contrast, the Ethiopia Productive Safety Net Project and the Malawi Rural Land Development Project had satisfactory outcomes. In both cases, the Bank worked through existing agencies and established programs (however imperfect) rather than attempting to impose new structures. The overall success rate of the International Finance Corporation’s (IFC) investment projects) in the Africa Region declined from 68 percent during FY08–10 to 54 percent in FY11–13. The gap between the Region and the overall IFC success rate was relatively unchanged at 5 percentage points in FY08–10 compared to 6 percentage points in FY11– 13 (Table 2). Working with the Bank, IFC had a successful transformational project in the Region by leveraging its convening power and global knowledge: open access to key telecom infrastructure for 20 African countries was achieved by combining the strength and expertise of both IFC and the Bank for policy development, transaction structuring, and co-financing from other development financing institutions. However, IFC’s poor appraisal work quality at times led to low development outcomes in the Region. Examples of shortcomings in appraisal work include inadequate estimates of capital requirements for a greenfield microfinance institution, poor due diligence on the capacity of a financial intermediary, and a lack of proper assessment of land-acquisition risks. These factors resulted in development outcome ratings of mostly unsuccessful or below, and IFC failed to achieve its objective such as an upturn in financing to small and medium enterprises (SMEs). Performance of IFC advisory services (AS) continues to be below IFC’s average. The Sub-Saharan Africa Region had a 50 percent success rate for AS projects, while IFC’s overall success rate was 63 percent (Table 3). For Investment Climate business line, the region had a 75 percent sucesss rate, which was higher than IFC’s overall success rate of 67 percent. A low success rate (24 percent) for the Sustainable Business Advisory (SBA) is the main reason for below average performance of AS projects in the Region. Various factors led to low development outcomes of SBA projects, which include lack of ownership by the IFC counterpart, optimistic project outcome targets, and staff changes. For the Multilateral Investment Guarantee Agency (MIGA), the success rate of the Africa Region was on a par with the overall average of 64 percent (Table 2). For infrastructure development in Africa, MIGA was effective, for example, in supporting an Islamic finance facility and also partnering with institutions such as the African Development Bank. Projects funded by the International Development Association (IDA) have performed as well as projects funded by the International Bank for Reconstruction and 55 ATTACHMENT: REGIONAL UPDATES Development (IBRD) in the Region. The IEG portfolio performance rating was 63 percent MS or better for IDA and 62 percent for IBRD (Table 4). Bank assistance to Africa reached a record high with lending commitments of $9.8 billion in FY14. The leading sectors were Agriculture; Transport; Water; and Social, Urban, Rural, and Resilience (Table 5). FINDINGS FROM THEMATIC, GLOBAL, AND CORPORATE EVALUATIONS Several recent IEG evaluations are of high relevance for the Africa Region. IEG’s evaluation of World Bank and IFC support for youth employment programs found that although the Bank’s response to government requests for analytical work was timely, economic and sector work was not necessarily conducted in countries where youth employment is a pressing issue.3 Analytic work in youth employment in Africa tends to focus on formal employment in urban areas, where political interest and data can be found. Few reports examine youth employment in rural low-income areas. Impact assessments showed that in Africa, it is most likely that youth employment will continue to be created in agriculture, rural non-farm work, or self-employment (Sierra Leone, Zambia). The Bank’s analytical work proposes easing regulations for hiring youth, but does not provide guidance for implementation (Ghana labor market report). IEG’s evaluation of the World Bank Group’s support to public-private partnerships (PPP) found that PPP interventions in Senegal helped address critical factors constraining growth.4 Significant resources were devoted to developing a toll highway, given the urgent need to expand economic activity out of the Dakar peninsula. The evaluation found that upstream work strongly supports the countries that need it the most, that is those with a nascent enabling environment or those that are in the process of building up. These are low-income countries that were mostly located in Sub-Saharan Africa. For example, in Ghana, efficiency gains through private sector involvement were considered substantial in the electricity sector, which became the priority sector for PPPs. Success of PPP projects depends on a strong regulatory framework, and the framework must balance market requirements with technical principles and public policy objectives, such as pro poor access, as well as optimal public sector risk exposure. IEG’s evaluation of the World Bank Group’s support to health financing showed that the Bank helped raise or protect public revenues for health against budget cuts during economic crises.5 Equity in pooling increased where the Bank assisted governments in subsidizing compulsory contributions to various health insurance plans for low-income groups. This type of Bank support was provided through lending and policy dialogue in Benin, Ghana, and Rwanda. The evaluation also found that government commitment ensured the success of projects supporting health financing reforms in Ghana and Rwanda. 56 ATTACHMENT: REGIONAL UPDATES A recent IEG learning product, which synthesized findings and lessons from IEG’s evaluations of avian influenza and pandemic preparedness projects from 2006 to 2013, as well as other evaluations helped to provide key lessons relevant to dealing with the Ebola outbreak in West African countries.6 They include: (i) the demand for rapid response needs to be balanced with the requirement for robust design. Deferring key decisions until after approval can lead to long lag times before project effectiveness and initial procurement. (ii) Investments in physical infrastructure such as laboratories and equipment can outstrip the human capacity of agencies to operate and maintain the systems, both technically and financially. Establishing a strategy for funding surveillance and diagnostic systems may make them more sustainable. (iii) It can be more effective to work through and expand existing health and emergency response institutions rather than creating new ones in parallel. (iv) Monitoring and evaluation system design should not be neglected in the rush to provide a rapid response. Intermediate outcome indicators that track the performance of particular functions against benchmarks can be useful. Surveys should assess behavioral practice, not just knowledge. EVALUATION FINDINGS FROM COUNTRY PROGRAM PERFORMANCE IN THE REGION IEG reviewed 21 Country Assistance Strategy Completion Reports (CASCRs) from FY12-14 for African countries. The country program outcomes of only nine (less than half) were rated MS or better while the outcomes of the remaining eleven were rated MU by IEG; five of the eleven MU ratings were downgraded from the Region’s ratings. The most common critical findings in all low-performing programs were that they were overly ambitious and spread sparsely over many areas, the outcome indicators were not clearly linked to the objectives, and risk assessments and mitigating measures were inadequate. In those cases where the outcome was rated MU, problems were severe and aggravated by shortcomings such as limited government capacity (Gabon), a critical lack of information (the Republic of Congo), or a lack of mid-term corrections in strategy (Malawi). In Mauritania, there were too many ambitious objectives (simultaneously pursuing 27 objectives across all five program pillars) set out in the county assistance program given the complex political environment. Relatively successful country programs were often associated with broad-based support, a good understanding of the local context and conditions (Benin), strong stakeholder engagement (Nigeria), and a sound results framework (Rwanda). The Bank Group’s country program in Rwanda, which is the only country program rated by IEG as fully satisfactory during the FY12–14 period, suggests other valuable lessons as well: (i) a highly selective program could help focus the design of the Country Partnership Framework in key strategic areas where the Bank Group would add value; (ii) capacity building is an important element of the program; ( iii) program selectivity is easier 57 ATTACHMENT: REGIONAL UPDATES when the country—even a low-income country—has clear ideas of the areas where it wants the Bank to concentrate; and (iv) strong risk assessment is an essential element of a successful timely program. CASCR reviews found examples of successful collaboration among Bank Group institutions and ways in which IFC and MIGA can play important roles in the country or Regional context. IFC and MIGA have been successful in building long-term partnerships and facilitating investments by South African companies across Africa. During the Nigeria banking crisis in 2009, both the Bank and IFC remained a significant source of support to the financial sector. CASCR reviews also found weaknesses in CAS results frameworks for IFC and MIGA. For instance, although IFC and MIGA’s programs were discussed in the Country Partnership Strategy and Country Partnership Strategy Progress Report text for South Africa in FY08–12, neither the programs nor their impacts were captured by the results framework. Figure 1. IEG Development Outcome Ratings for Africa Operations Relative to World Bank Average, FY00-13 100% 90% 80% 70% Percentage 60% 50% 40% 30% 20% 10% 0% % of ICRR in all ICR AFR IEG outcomes MS or higher Bank‐wide IEG outcomes MS or higher Source: Business Warehouse Note: AFR=Africa Region; ICR=Implementation Completion Report; ICRR=Implementation Completion Report Review; IEG=Independent Evaluation Group; MS=moderately satisfactory.   Table 1. Africa Region: IEG Development Outcome Ratings by Global Practice for Operations Exiting in FY11-13) Global Practices Africa Region World Bank Average 58 ATTACHMENT: REGIONAL UPDATES MS or higher Total MS or higher Total Number of Number of Number % Percentage of evaluated Number % Percentage evaluated total projects of total projects commitment commitment Agriculture 13 59 35 22 30 64 63 47 Education 14 52 43 27 51 67 88 76 Energy & Extractives 9 64 80 14 38 66 81 58 Environment & Natural 9 50 54 18 33 59 64 56 Resources Finance & Markets 11 58 70 19 42 71 94 59 Governance 7 41 45 17 34 60 76 57 Health, Nutrition & 20 67 63 30 52 71 83 73 Population Macro Economics & 20 74 77 27 45 73 86 62 Fiscal Management Social Protection & Labor 8 100 100 8 27 79 93 34 Social, Urban, Rural and 21 91 88 23 81 79 81 103 Resilience Global Practice Transport & ICT 10 59 39 17 60 78 72 77 Water 8 80 60 10 33 62 63 53 Other 0 0 0 2 3 50 96 6 Grand Total 150 64 68 234 529 70 81 761 Source: Business Warehouse Note: MS=moderately satisfactory; "Other" represents global practices with fewer than five projects within the AFR region. As relates to Bank- wide figures, other includes the same global practices with fewer than five projects, as in AFR, in addition to global practices which were not represented in AFR but which were present Bank-wide. Figures for percentage of total commitment relate solely to IBRD/IDA funding and exclude projects funded through trust funds. 1World Bank. 2014. Africa’s Pulse. “Decades of Sustained Growth is Transforming Africa’s Economies.” October 7, 2014, Vol. 10. 2 Africa: Investing in Shared Growth, Regional Update 2014. 3IEG. 2012. Youth Employment Programs: An Evaluation of World Bank and International Finance Corporation Support. Washington, DC: World Bank. 4IEG. 2014. World Bank Group Support to Public-Private Partnerships: Lessons from Experience in Client Countries, FY02–12. Washington, DC: World Bank. 5 IEG. 2014. World Bank Group Support to Health Financing. Washington, DC: World Bank. 6IEG. 2014. Responding to Global Public Bads: Learning from Evaluation of the World Bank Experience with Avian Influenza 2006-13. Washington, DC: World Bank. 59 ATTACHMENT: REGIONAL UPDATES Table 2. IEG Development Outcome Ratings of IFC Investment and MIGA Guarantee Operations in Africa Region and Overall, FY11-13 IFC Africa IFC Overall Commit of XPSRs Commit of XPSRs/PES Commit of Commit of mostly Success /PES mostly Success No. of evaluated as Success evaluated No. of Success evaluated successful rate by evaluated successful rate by Cluster XPSRs/PES mostly rate operations XPSRs/PES rate operations or higher commit as mostly or higher commit evaluated successful (%) (US$ evaluated (%) (US$ (US$ (%) successful (US$ (%) or higher millions) millions) millions) or higher millions) Financial Markets 17 10 59 266.8 184.7 69 106 59 56 3,274 2,219 68 Infrastructure & 8 2 25 153.0 47.7 31 48 25 52 1,850 1,293 70 Natural resources Manufacturing, Agribusiness & 10 7 70 188.2 163.1 87 78 56 72 1,915 1,462 76 Services Total 35 19 54 608.0 395.5 65 235 140 60 7,039 4,974 71 MIGA Africa MIGA Overall PERs PERs No. of MIGA No. of MIGA evaluated Success evaluated Success rate projects projects satisfactory rate (%) satisfactory (%) evaluated evaluated or higher or higher MIGA Total 16 11 69 43 30 70 Source: IEGPE Database (XPSR and PES Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. MIGA ratings are for the FY08-13 period. MIGA's rating criteria follow a four-point rating scale: excellent; satisfactory; partly unsatisfactory; and unsatisfactory. Data includes project ratings finalized up to December 31, 2014. 60 ATTACHMENT: REGIONAL UPDATES Table 3. IEG Ratings of IFC Advisory Projects: Africa Region and IFC Overall, 2011-2013 Africa IFC Overall Business Line Number of PCRs Success Number of PCRs Success PCRs evaluated as rate PCRs evaluated rate evaluated mostly (percentage) evaluated as mostly (percentage) successful or successful higher or higher Access to 15 9 60 58 39 67 Finance Investment 12 9 75 42 28 67 Climate Public Private 6 4 67 29 17 59 Partnership Sustainable 21 5 24 78 46 59 Business Advisory Total 54 27 50 207 130 63 Source: IEGPE Database (PCR Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. Includes preliminary ratings. Table 8. IEG Development Outcome Ratings by Country, FY11-13 World Bank Projects IFC-IEG XPSR/PES Ratings IFC-IEG PCR Ratings MS+ Total MS+ MS+ Number Total Total Number of Number Number Number Number of evaluated of of rated of of rated Projects % projects Projects % projects Projects % projects IBRD Angola 1 33 3 Botswana 1 100 1 0 2 Gabon 1 33 3 Mauritius 1 100 1 Namibia 1 50 2 Seychelles 1 100 1 South Africa 2 100 2 1 25 4 0 3 IBRD Subtotal 7 58 12 2 40 5 0 5 Blend Cabo Verde 1 100 1 Cameroon 2 50 4 0 1 0 1 Congo, Republic of 2 67 3 Nigeria 9 82 11 4 100 4 0 1 Zimbabwe 1 100 1 Blend Subtotal 15 75 20 4 80 5 0 2 IDA Benin 6 60 10 Burkina Faso 6 75 8 1 100 1 1 100 1 Burundi 4 67 6 0 1 Central African Republic 2 67 3 Chad 0 4 0 1 Comoros 2 100 2 Congo, Democratic Republic of 5 71 7 0 1 Cote d'Ivoire 4 67 6 Eritrea 1 50 2 61 ATTACHMENT: REGIONAL UPDATES World Bank Projects IFC-IEG XPSR/PES Ratings IFC-IEG PCR Ratings MS+ Total MS+ MS+ Number Total Total Number of Number Number Number Number of evaluated of of rated of of rated Projects % projects Projects % projects Projects % projects Ethiopia 4 50 8 1 100 1 Gambia, The 3 100 3 Ghana 3 33 9 1 33 3 1 50 2 Guinea 2 67 3 Guinea-Bissau 2 100 2 Kenya 2 25 8 0 3 2 33 6 Lesotho 2 100 2 1 100 1 Liberia 3 100 3 0 1 2 50 4 Madagascar 6 67 9 1 50 2 1 100 1 Malawi 5 71 7 1 100 1 1 50 2 Mali 1 14 7 1 100 1 Mauritania 3 50 6 Mozambique 5 83 6 2 100 2 0 1 Niger 6 75 8 0 1 Rwanda 6 100 6 2 67 3 2 100 2 Sao Tome and Principe 3 100 3 2 100 2 Senegal 4 57 7 1 100 1 1 50 2 Sierra Leone 2 67 3 0 1 2 67 3 South Sudan 6 50 12 Sudan 4 80 5 1 50 2 Tanzania 6 55 11 0 1 2 100 2 Togo 3 100 3 Uganda 5 83 6 1 100 1 Zambia 4 67 6 0 3 IDA Subtotal 120 63 191 9 47 19 22 54 41 Regional 8 73 11 4 67 6 5 83 6 Grand Total for Africa Region 150 64 234 19 54 35 27 50 54 Source: IEGPE Database (XPSR Evaluation Notes, PCR Evaluation Notes Note: Success rate refers to projects rated as mostly successful or higher. Includes preliminary ratings.  Table 5. Africa Region: New World Bank Lending Commitments by Global Practice, FY10-14 (US$ millions) Global Practice FY10 FY11 FY12 FY13 FY14 Agriculture 492 580 480 740 1,107 Education 246 288 127 290 705 Energy & Extractives 4,745 925 993 1,072 916 Environment & Natural Resources 55 172 560 57 71 Finance & Markets 555 134 95 40 330 Governance 230 206 297 237 356 Health, Nutrition & Population 484 370 534 354 452 Macro Economics & Fiscal Management 815 914 790 623 835 Poverty 132 245 440 5 Social Protection & Labor 875 515 761 1,244 824 62 ATTACHMENT: REGIONAL UPDATES Social, Urban, Rural and Resilience Global Practice 442 609 641 654 1,229 Trade & Competitiveness 97 445 70 335 360 Transport & ICT 1,691 1,063 326 1,874 1,471 Water 335 596 963 666 1,096 Grand Total 11,194 7,060 7,077 8,185 9,757 Source: Business Intelligence as of October 15, 2014 Note: The New World Bank Lending Commitments are the sum of IBRD and IDA commitments for PE projects approved between FY10-14. Table 6. Africa Region: IEG Ratings of Project M&E Quality, FY11-13 M&E Quality Substantial or Better Total number of rated projects Number of projects Percentage IBRD 5 38 13 Blend 8 40 20 IDA 45 24 191 Regional 2 18 11 Grand Total for Africa Region 60 26 235 Grand Total Bank-wide 224 29 765 Table 7. Africa Region: CASCRR Outcome and Performance Ratings, FY12-14 FY of review Country CASCR review period Outcome rating Bank performance IFC Performance 2012 Gabon FY05-11 MU U Unrated Mozambique FY08-11 MU MU MS 2013 Benin FY09-12 MS MS Unrated Burundi FY09-12 MS MS MU Congo, Democrat FY08-12 MS S Unrated Congo, Republic FY10-12 MU MS Unrated Ethiopia FY08-12 MS MS Unrated Gambia, The FY08-12 MU MS Unrated Malawi FY07-12 MU MS Unrated Niger FY08-11 MU MS Unrated Senegal FY07-CY10 MU MS MS Zambia FY08-12 MU MU Unrated 2014 Burkina Faso FY10-12 MS MS Unrated Ghana FY08–12 MU MU Unrated Guinea5 FY04-FY13 NR NR Unrated Liberia FY09-12 MS MS Unrated Mauritania FY07-12 U U Unrated South Africa FY08-12 U MU Unrated Nigeria FY10-13 MS MS Unrated Kenya FY10-14 MS Fair Unrated Rwanda FY10-13 S Superior Unrated 63 ATTACHMENT: REGIONAL UPDATES East Asia and the Pacific REGIONAL CHALLENGES AND EVALUATION FINDINGS FROM WORLD BANK GROUP OPERATIONS IN THE REGION The East Asia and Pacific (EAP) Region remained the fastest-growing Region in the world in 2013, accounting for more than 40 percent of the increase in global growth. China’s annual growth remained 7.7 percent, while growth in other developing countries slowed to 5.3 percent, down from 6.3 percent in 2012. Total growth in the Region is expected to remain about 6.9 percent in 2014 and 2015.1 The proportion of people living in poverty in the Region has steadily declined over the past 25 years. However, nearly 140 million (7 percent) of the Region’s 2 billion people still live on less than $1.25 a day and another 300 million (15 percent) live on incomes between $1.25 and $2.00 a day. Challenges include rising inequality; gaps in infrastructure and services; threats from natural disasters and climate change; and effective job creation. To address these challenges and achieve the World Bank Group’s twin goals of ending extreme poverty and boosting shared prosperity, EAP strategic priorities include inclusion and empowerment; jobs and private sector-led growth; governance and institutions; infrastructure and urbanization; and climate change and disaster risk management.2 Overall performance of World Bank operations in the Region has continued to deteriorate and now is about average. Outcome ratings at exit in FY11–13 were 68 percent moderately satisfactory (MS) or better compared to 70 percent Bank-wide in terms of the number of projects. Expressed as a share of commitment volumes, outcomes ratings were 79 percent MS or better compared to 81 percent Bank-wide (Figure 1 and Table 1). Relative to other Regions, sectors with projects having weak performance included Social, Urban, Rural, and Resilience; Transport; Information and Communications Technologies; Energy; and Extractives Industries—which together accounted for more than half of the Regional commitments in the FY10–14 period. The portfolio performance of these sectors was poorer than the Bank average, both in terms of project numbers and commitment volumes. Slightly more than half of the Project Performance Assessment Reports (PPARs), 53 percent, rated outcomes as MS or better. The steady decline in the quality-at-entry rating was an important factor in the decline of project outcome ratings in the EAP Region. Quality at entry showed a huge drop from 78 percent in FY08–10 to 63 percent in FY11–12, while quality of supervision declined modestly to 76 percent. For example, the Thailand Highway Management Project was not properly appraised and the project design was overly ambitious in its expectation of introducing a performance-based contract pilot without sufficient discussion with the Department of Highways about both commitment and interest. As a result, the project achieved very few of its objectives. At the same time, there are examples of satisfactory projects demonstrating that operations with complex structures can succeed if design is flexible enough, and the project is backed by strong government 64 ATTACHMENT: REGIONAL UPDATES commitment. For example, the Indonesia Second Urban Poverty Project had a complex design, but its development objectives were clearly stated, and there was a strong results chain between the activities and the intended outcomes. The government’s commitment was also strong, and it used the project as a vehicle for delivering support to poor communities throughout Indonesia. Finally, lessons from earlier projects can contribute greatly to improvements of project design as was the case of the China Pearl River Watershed Rehabilitation Project. Objectives were straightforward and design was well focused on the objectives. The monitorable indicators were practical and nicely captured the project’s achievements and outcomes. As a result, the project was a success story for China. The quality of monitoring and evaluation (M&E) frameworks in Bank-financed projects was well below the Bank average, with only 24 percent of the projects rated substantial or better compared to 30 percent for the Bank. As with the outcome ratings, International Development Association (IDA) countries (30 percent) and International Bank for Reconstruction and Development (IBRD) countries (27 percent) had better M&E frameworks than blend countries (13 percent). Common shortcomings included weak results chains; poorly operationalized indicators; lack of baselines and targets; and indicators focused on outputs rather than outcomes. For example, the Thailand Highway Management Project did not have the baseline data for major indicators in the initial period, the quality of collected data was very low, and the project failed to measure achievement of outcomes. IEG’s PPARs prepared during FY12–14 also provide ample evidence of poorly designed projects and consequently unsatisfactory outcomes. Government ownership and good assessment of the policies and institutions at project appraisal are pre-conditions for successful projects. The Indonesia Land Management Project3 and the Second Land Titling Project4 of the Lao People’s Democratic Republic had unsatisfactory outcomes. In both cases the project design assigned priority to land titling without properly assessing the sufficiency of legal and administrative frameworks as well as government ownership. The PPARs on two Vietnam energy projects5 demonstrate the importance of appropriate sector policies for successful energy access projects. The China Chongqing Small Cities Infrastructure Project6demonstrated that the Bank can maintain the effectiveness of its support by tailoring its services and instruments to the more localized needs and capacity of the client. In the case of the International Finance Corporation (IFC), both by number of projects and investment commitments, EAP Region success rates were similar to IFC averages. IFC’s development outcome ratings was affected since IFC took the major restructuring of the portfolios in FY10-FY11 in the EAP region to improve development results for the investment operations. For FY11–13, IEG validated 31 investment projects and assigned 65 ATTACHMENT: REGIONAL UPDATES its mostly successful or higher rating to 19 projects (61 percent success rate), which was on a par with IFC’s overall average of 60 percent. In terms of commitment, the Region’s success rate was 70 percent, just 1 percentage point below IFC’s overall number of 71 percent. IFC’s financial market operations had a low success rate of 46 percent, which was 22 percentage points below the IFC average of 68 percent with respect to the commitments. IFC’s poor work quality during the supervision of its investment is one of the factors that led to lower development outcomes for financial market operations. Corporate governance and environment and social (E&S) performance standards are the areas where IFC can add value as a development finance institution, but IFC was short in delivering its additonality in some of its investments in the Region. With limited resources to cover the Asian fund portfolio, IFC had little impact on the Chinese fund’s E&S risk management policies and was not successful in improving corporate governance of investee companies. During the supervision period, IFC was also short in supporting the first time Vietnamese fund manager in the areas of deal negotiations, corporate governance, and E&S performance standards. During the review period, development outcome ratings of IFC’s advisory services (AS) projects in the Region was 57 percent, lower than IFC’s overall average of 63 percent. This lower rating was in part due to comprehensive portfolio overhaul during FY10- FY11 to rationalize the programming and improve the design/development effectiveness of AS operation, including the closure of many AS operations. In the Philippines, high-quality preparatory work and due diligence led to the first successful public-private partnerships (PPP) transaction at the municipality level. In China, active involvement of the IFC advisory service team from the beginning of the project led to a successful greenfield microfinance project. But, IFC’s development impact was not high when it pushed its own development agenda and demand for its services was weak. In Indonesia where the banking sector is sophisticated and small and medium-sized enterprises (SME) practices are already in place, IFC’s AS team struggled to communicate their value proposition to the banking sector on SME issues and achieve the development results it aimed to accomplish. In the Philippines, IFC was not able to achieve its overall objective of addressing the existing SME finance gap because of the lack of market interest and failure to find SME champions to enhance SME lending in the country, although IFC added values in niche areas of the SME market in agro- finance and gender. During FY11–13, IEG validated two Multilateral Investment Guarantee Agency (MIGA) projects in the Region and found that both projects delivered successful development outcomes. In Mongolia, IEG found that MIGA was effective in achieving close coordination with the World Bank, IFC, and the Asian Development Bank during the 66 ATTACHMENT: REGIONAL UPDATES privatization process of the state-owned bank and contributed to the development of the Mongolian banking system. FINDINGS FROM THEMATIC, GLOBAL, AND CORPORATE EVALUATIONS Several findings from IEG evaluations are relevant to emerging trends in the Bank Group’s support strategy, including promoting private sector growth and improving the investment climate. These lessons are valuable in achieving EAP’s strategic pillar on “jobs and private sector growth.” The evaluation of the Bank Group’s experience with targeted support to SMEs found that interventions have been successful, however, the impact was hard to identify because of attribution issues.7 The evaluation notes that IFC’s relevance is greater when it operates near the frontier in countries. For example, in Indonesia, IFC’s interventions were highly relevant in the post-1998 financial crisis period, and a number of the banks became IFC’s long-term partners, which benefited from the support for targeted SME projects. In recent years, the banking community in Indonesia has actively competed for SME clients, has known how to assess risk and manage portfolios, and has had access to private capital. However, IFC continued to finance these banks on favorable terms, even though several of them had grown and are now among the country’s leading banks. This evaluation suggests that a more relevant IFC strategy might have been to move to second- or third-tier banks; focus on underserved regions, locations, and clients; or shift capital to countries with less mature capital markets. IEG’s evaluation of investment climate reform showed that the Regional EAP portfolio supporting investment climate reform is one of the smallest among Bank Regions.8 However, the Bank Group’s support was successful in helping to enact laws and streamline the process, although the impact on investments, jobs, and growth was mixed. In a few countries, including Cambodia, interventions focused on specific areas such as trade promotion and guidance on entry to the World Trade Organization; provided support for the development of an alternative dispute resolution system for commercial disputes; helped establish a legal and institutional framework for special economic zones; and supported reductions in trade and investment-related processes and procedures through a single window system. The small number of completed and evaluated investment climate projects in fragile and conflict-affected states (FCS) suggests that effectiveness in FCS is significantly lower than in non-FCS. Evidence from country case studies shows mixed results and indicates the importance of political feasibility and institutional capacity building. Thus in Lao PDR, taking into account local capacity limitations, the Bank increased technical assistance during the progression of its budget support operations leading to positive outcomes. IEG’s thematic evaluation of the Bank Group’s support to PPPs found evidence that a single PPP transaction may have broader effects by setting an example of a pioneering 67 ATTACHMENT: REGIONAL UPDATES project in a relatively untested environment through demonstration effects.9 After an initial exploration phase by Chinese authorities to assess whether PPPs were the right tool, IFC’s support added value by introducing safeguards standards in the wastewater sector in China. Not only did this project introduce environmental and social standards, but six to eight private investors entered the market within five years of IFC's investment. The evaluation also found that PPPs have mainly worked in the water sector in China, as other infrastructure sectors are dominated by state-owned enterprises. EVALUATION FINDINGS FROM COUNTRY PROGRAM PERFORMANCE IN THE REGION Over the period FY12–14, IEG reviewed eight Country Assistance Strategy Completion Reports (CASCRs) for the EAP countries. The country program outcomes of six were rated MS or better while the outcome of the remaining two were rated unsatisfactory (Papua New Guinea) and moderately unsatisfactory (Timor-Leste). In the case of Timor Leste, the environment in which the World Bank Group program was delivered was fragile, rapidly changing and uniquely challenging. Successful country programs were often associated with sustained, innovative support for key development agendas. In Lao PDR, IDA took calculated risks in supporting the implementation of the transformative Nam Theun 2 Project as an example of an area- based, sustainable, natural resource development program that contributes to growth, improved social outcomes, capacity development, and stronger partnerships. In the Philippines, impressive progress in social protection is a good illustration of an area where Bank expertise provides added value to its clients. In China, the Bank’s financial and knowledge support in health, education, and social security systems over a long period— with projects piloting new local approaches and analytic and advisory activities (AAA) providing higher-level inputs for policy reforms— contributed effectively to helping to reduce poverty, inequality, and social exclusion. IFC made significant contributions to improving access to finance, using advisory services to improve institutions and a combination of investments and advisory services to develop or strengthen financial institutions. On the other hand, weak country programs could have benefited from greater selectivity and stronger results frameworks. In Papua New Guinea and Timor-Leste, overly broad and diffuse strategies critically impeded the Bank Group’s capacity to achieve the expected results. By attempting to cover a wide range of issues in a context of weak institutional capacity and modest IDA resources, the programs ended with diluted priorities and the under-achievement of strategic objectives. In addition, most country strategies in the EAP Region lacked useful results frameworks. 68 ATTACHMENT: REGIONAL UPDATES Key lessons for EAP countries from CASCR reviews can be drawn. The Bank Group has an important role in middle-income countries regarding knowledge transfer, whether from its own AAA or through promoting South–South exchanges. With limited presence in these countries, a strategy focused on the few key areas where the Bank Group has comparative advantage may have more impact than a broader but more diffuse strategy. Strong ownership by the client country is essential for progress. Evaluations also indicate the importance of frequent monitoring and evaluation, so that adjustments and fine-tuning can be made in a timely manner. An absence of a results framework that includes a robust M&E system can handicap the country team in delivering development results. Figure 2. IEG Development Outcome Ratings for East Asia and Pacific Operations Relative to World Bank Average, FY00-13 100% 90% 80% 70% Percentage 60% 50% 40% 30% 20% 10% 0% % of ICRR in all ICR EAP IEG outcomes MS or higher Bank‐wide IEG outcomes MS or higher Source: Business Warehouse Note: EAP=East Asia and Pacific Region; ICR=Implementation Completion Report; ICRR=Implementation Completion Report Review; IEG=Independent Evaluation Group; MS=moderately satisfactory. 69 ATTACHMENT: REGIONAL UPDATES Table 8. East Asia and Pacific Region: IEG Development Outcome Ratings by Global Practice for Operations Exiting in FY11-13 East Asia and Pacific Region World Bank Average MS or higher Total MS or higher Total Number % Percentage Number Number % Percentage Number of total of of total of commitment evaluated commitment evaluated Global Practices projects projects Agriculture 4 80 77 5 30 64 63 47 Education 9 82 99 11 51 67 88 76 Energy & Extractives 4 44 38 9 38 66 81 58 Environment & Natural Resources 5 71 0 7 33 59 64 56 Finance & Markets 6 86 100 7 42 71 94 59 Governance 3 60 46 5 34 60 76 57 Health, Nutrition & Population 7 70 64 10 52 71 83 73 Macro Economics & Fiscal Management 6 75 99 8 45 73 86 62 Social, Urban, Rural and Resilience Global Practice 17 68 82 25 81 79 81 103 Transport & ICT 11 69 65 16 60 78 72 77 Water 4 44 61 9 33 62 63 53 Other 2 100 100 2 30 75 93 40 Grand Total 78 68 79 114 529 70 81 761 Source: Business Warehouse Note: MS=moderately satisfactory; "Other" represents global practices with fewer than five projects within the EAP region. As relates to Bank- wide figures, other includes the same global practices with fewer than five projects, as in EAP, in addition to global practices which were not represented in EAP but which were present Bank-wide. Figures for percentage of total commitment relate solely to IBRD/IDA funding and exclude projects funded through trust funds. 1 Global Economic Prospects, June 2014. 2 EAP Regional Update, October 2014. 3IEG (2014). Project Performance Assessment Report for the Indonesia Land Management and Policy Development Project. 4IEG (2013). Project Performance Assessment Report for the Lao People’s Democratic Republic Second Land Titling Project, IEG Public Sector Evaluation. 5IEG (2013). Project Performance Assessment Report for the Vietnam Transmission, Distribution, and Disaster Reconstruction Project and the Vietnam Rural Energy Project. 62014. IEG. Project Performance Assessment Report for the China Chongqing Small Cities Infrastructure Improvement Project. 7IEG. 2014. The Big Business of Small Enterprises: Evaluation of the World Bank Group Experience with Targeted Support to Small and Medium-Size Businesses, 2006–12. Washington, DC: World Bank. 8IEG. 2014. Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms of Business Regulations. Washington, DC: World Bank. 9IEG. 2014. World Bank Group Support to Public-Private Partnerships: Lessons from Experience in Client Countries, FY02–12. Washington, DC: World Bank. 70 ATTACHMENT: REGIONAL UPDATES Table 9. IEG Development Outcome Ratings of IFC Investment and MIGA Guarantee Operations in East Asia and Pacific Region and Overall, FY11-13 IFC EAP IFC Overall Cluster No. of XPSRs/PE Success Commit Commit of Success Number of XPSRs/PES Succes Commit of Commit of Success XPSRs/PE S rate of mostly rate by XPSRs/PE evaluated as s rate evaluated mostly rate S evaluated (%) evaluated successful commit S mostly (%) operations successful by evaluated as mostly operation or higher (%) evaluated successful (US$ or higher commit successful s (US$ (US$ or higher millions) (US$ (%) or higher millions) millions) millions) Financial Markets 15 7 47 494 227 46 106 59 56 3,274 2,219 68 Infrastructure & Natural 6 4 67 512 439 86 48 25 52 1,850 1,293 70 resources Manufacturing, 10 8 80 282 237 84 78 56 72 1,915 1,462 76 Agribusiness & Services Total 31 19 61 1,288 903 70 232 140 60 7,039 4,974 71 MIGA EAP MIGA Overall No. of PERs Success No. of PERs Success MIGA evaluated rate MIGA evaluat rate projects satisfactor (%) projects ed (%) evaluated y or higher evaluated satisfac tory or higher MIGA Total 4 4 100 43 30 70 Source: IEGPE Database (XPSR and PES Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. MIGA ratings are for the FY08-13 period. MIGA's rating criteria follow a four-point rating scale: excellent; satisfactory; partly unsatisfactory; and unsatisfactory. Data includes project ratings finalized up to December 31, 2014. 71 ATTACHMENT: REGIONAL UPDATES Table 10. IEG Ratings of IFC Advisory Projects: East Asia and Pacific Region and IFC Overall, 2011- 2013 East Asia and Pacific IFC Overall Business Line Number of PCRs rated as Success rate Number of PCRs rated as Success rate PCRs rated mostly (percentage) PCRs rated mostly (percentage) successful or successful or higher higher Access to Finance 15 11 73 58 39 67 Investment Climate 6 2 33 42 28 67 Public Private 3 1 33 29 17 59 Partnership Sustainable 11 6 55 78 46 59 Business Advisory Total 35 20 57 207 130 63 Source: IEGPE Database (PCR Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. Includes preliminary ratings. Table 11. IEG Development Outcome Ratings by Country, FY11-13 World Bank Projects IFC-IEG XPSR/PES Ratings IFC-IEG PCR Ratings MS+ Total MS+ Total MS+ Total Number % Number Number % Number Number % Number of of of of rated of of rated Projects evaluated Projects projects Projects projects projects IBRD China 20 83 24 9 69 13 6 75 8 Indonesia 16 67 24 1 25 4 1 50 2 Malaysia 1 100 1 Philippines 3 33 9 5 100 5 2 40 5 Thailand 1 50 2 IBRD Subtotal 41 68 60 15 68 22 9 60 15 Blend Mongolia 5 71 7 1 100 1 Papua New Guinea 1 100 1 1 33 3 Timor-Leste 0 4 1 100 1 Vietnam 14 74 19 0 2 4 80 5 Blend Subtotal 20 65 31 1 33 3 6 67 9 IDA Cambodia 6 75 8 1 100 1 2 67 3 Kiribati 0 1 Lao People's Democratic Republic 6 75 8 1 50 2 1 33 3 Myanmar 1 100 1 Samoa 1 100 1 1 100 1 Solomon Islands 1 100 1 Tonga 1 50 2 0 1 Vanuatu 0 2 IDA Subtotal 16 73 22 2 67 3 4 40 10 Regional 1 100 1 1 33 3 1 100 1 Grand Total for East Asia and Pacific Region 78 68 114 19 61 31 20 57 35 Source: IEGPE Database (XPSR Evaluation Notes, PCR Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. Includes preliminary ratings. 72 ATTACHMENT: REGIONAL UPDATES Table 12. East Asia and Pacific Region: New World Bank Lending Commitments by Global Practice, FY10-14 (US$ millions) Global Practice FY10 FY11 FY12 FY13 FY14 Agriculture 535 5 17 80 360 Education 741 83 50 345 438 Energy & Extractives 508 1,655 490 676 810 Environment & Natural Resources 263 110 280 219 134 Finance & Markets 22 22 100 20 Governance 650 542 150 430 Health, Nutrition & Population 146 170 100 150 126 Macro Economics & Fiscal Management 810 1,890 2,411 1,312 835 Social Protection & Labor 655 180 60 Social, Urban, Rural and Resilience Global Practice 2,087 859 1,873 2,088 1,054 Trade & Competitiveness 66 300 Transport & ICT 810 1,797 776 631 1,541 Water 289 865 481 400 205 Grand Total 7,517 7,997 6,628 6,247 6,313 Source: Business Intelligence as of October 15, 2014 Note: The New World Bank Lending Commitments are the sum of IBRD and IDA commitments for PE projects approved between FY10-14. Table 13. East Asia and Pacific Region: IEG Ratings of Project M&E Quality, FY11-13 M&E Quality Substantial or Better Total number of rated Number of projects Percentage projects IBRD 16 27 60 Blend 4 13 31 IDA 7 30 23 Regional 1 100 1 Grand Total for East Asia and Pacific 28 24 115 Region Grand Total Bank-wide 224 29 765 Table 9. . East Asia and Pacific Region: CASCRR Outcome and Performance Ratings, FY12-14 FY of review Country CASCR review period Outcome rating Bank performance IFC Performance 2012 Lao People's Democratic FY05-11 S MS NA Mongolia CY05–11 MS MS MS Vietnam FY07-11 MS MS S 2013 China FY07-12 S S S Indonesia FY09-12 MS MS NA Papua New Guinea FY08-12 U MU S Timor Leste FY06-11 MU MU NA 2014 Philippines FY10-13 MS Good Good 73 ATTACHMENT: REGIONAL UPDATES Europe and Central Asia REGIONAL CHALLENGES AND EVALUATION FINDINGS FROM WORLD BANK GROUP OPERATIONS IN THE REGION Growth in Europe and Central Asia (ECA) remains weak with regional gross domestic product projected to grow 1.8 percent in 2014, improving slightly to 2.1 percent in 2015- 1. Recovery in the new European Union (EU) member states remains tepid, while growth is stagnating in Russia and is slowing down in some neighboring countries. The conflict in Ukraine has developed into a deep crisis, but so far has had limited spill-over effect on other countries. According to the ECA Regional Strategy, going forward, the Region should emphasize improving governance and the investment climate, strengthening competitiveness, ensuring the stability of the financial sector, and maintaining a sound macro-economic framework. To achieve the World Bank Group’s twin goals of ending extreme poverty and boosting shared prosperity, ECA Regional Strategy focuses on two pillars: (a) competitiveness and shared prosperity through jobs; and (b) environmental, social, and fiscal sustainability.2 The performance of World Bank operations in ECA continues to be better than the performance of the Bank as a whole in terms of the number of projects: outcome ratings at exit in FY11-13 were 76 percent moderately satisfactory or better (MS+) for ECA compared to the Bank average of 70 percent. In commitment amounts, 88 percent were rated MS+ compared to 81 percent Bank-wide (Table 1). Energy and Extractive Industries; Finance and Markets; and Macro Economics and Fiscal Management, which accounted for 72 percent of the commitments during FY10-14 (Table 5), performed better than the Regional average. Education; Environment and Natural Resources; and Social Protection and Labor, on the other hand, performed below Bank-wide averages. Many countries in the Balkans performed below the Regional average, with less than two-thirds of projects rated MS+ at exit (Table 4).3 Portfolio performance could have improved with better project preparation. Most projects with less than satisfactory outcomes were also rated less than satisfactory for quality-at-entry. The most common weaknesses were overly ambitious development objectives relative to the project components and/or time frame, overly complex project design involving multiple components and implementing agencies, and over-estimation of client buy-in and implementing capacity. Adequate assessment of the capacity of key implementing agencies is important. A strong implementing agency with proven experience in dealing with local governments and communities contributed to the success of the Albania Secondary and Local Roads Project. Experience from successful projects shows that it is possible to set ambitious objectives and design projects with a wide range of activities as long as there is leadership and institutional capacity to manage the task. While government administrative changes resulted in delays in implementation, the Russia Customs Development Project succeeded in modernizing 74 ATTACHMENT: REGIONAL UPDATES customs administration and updating the legal framework due in great part to a strong commitment to project objectives and technical and administrative capacity in the implementing agency. Successful projects also incorporated learning from previous projects or similar projects in other countries, e.g. including technical assistance to overcome institutional shortcomings identified in earlier operations. A clear causal link between project activities, output, outcomes and development objectives was another common feature among successful projects. Monitoring and evaluation (M&E) continues be a concern in ECA. While better than the Bank-wide average, only 36 percent of projects in ECA had high or substantial ratings for M&E design, implementation, and use in FY11-13 (Table 6). Problems start with the design of the M&E framework. The most common issues were the lack of outcome indicators, indicators not directly relevant to the project objectives, indicators not directly attributable to the project activities, indicators where data was hard to obtain, and a lack of baseline data. Some of the issues with the M&E framework were fixed during implementation with more appropriate indicators substituted and base line data developed. But data often would be collected and reported only on selected indicators, reducing the effectiveness of the M&E systems. International Finance Corporation (IFC) investments performed worse than in other Regions – a 51 percent success rate by number and 59 percent by commitment in ECA compared to 60 percent and 71 percent for IFC overall (Table 2). Performance has been poor across the board in financial markets, but especially so among smaller projects in commercial banking, and in telecommunications within infrastructure. Shortcomings in appraisal work, such as unsuccessful negotiation of equity exit arrangements and a failure to reach proper investment fund size, contributed to investment projects with development outcomes of mostly unsuccessful (MU) or lower in the Region. IFC’s advisory service (AS) projects fared better with 74 percent rated mostly successful or better, compared to the IFC-wide average of 63 percent (Table 3). IFC’s work quality is also an important contributing factor for successful AS projects across different AS business lines. In its Sustainable Business Advisory (SBA) business line, IFC was successful in securing commitments from the Georgian government in corporate governance reforms by having a major investor to the country as a project donor. For a sub-national level investment climate (IC) project in Montenegro, IFC achieved demonstration effects through replication of reforms in other municipalities as it succeeded in identifying champions for IC reforms. The Multilateral Investment Guarantee Agency’s (MIGA) guarantee projects in the Region were on par with MIGA’s overall success rate. The success rate of MIGA projects evaluated by IEG was 64 percent for Europe and Central Asia for FY08-13 (Table 2), which was lower than the overall MIGA success rate of 70 percent. 75 ATTACHMENT: REGIONAL UPDATES FINDINGS FROM THEMATIC, GLOBAL, AND CORPORATE EVALUATIONS Improving the business climate and promoting private sector growth will be critical to achieving ECA’s first strategic pillar: “competitiveness and shared prosperity through jobs.” Recently completed IEG evaluations of investment climate reform4 and targeted support to small and medium enterprises5 pointed to the need for better coordination within the Bank Group. The salient findings are as follows:  Investment Climate Reform: Among the Regions, ECA had the highest share of successful investment climate interventions and ranked second, after Sub- Saharan Africa, in the number of interventions6. IEG found that the Bank Group succeeded in helping to enact laws, streamline process and time, and provide simple cost savings for private firms, but their impact on investment, jobs, business formation, and growth was not clear. Simplicity of design and good risk assessment contributed to satisfactory outcomes. At the same time, political instability and lack of political commitment remained major problems. The Bank Group needed to expand coverage of current diagnostic tools and integrate them to produce comparable indicators and develop a differentiated approach to identify the social effects of regulatory reforms on all groups to be affected by them beyond the business community. Coordination within the Bank Group remained mostly informal, relying mainly on personal contacts, and synergies within the group needed improvement.  Targeted support for Small and Medium Enterprises (TSME): TSME is big business for the Bank Group, averaging around $3 billion a year in commitments, expenditures, and gross exposure over FY06-12. ECA played a major role in the Bank Group’s TSME operations, coming in first among the Regions in IFC’s TSME portfolio, MIGA’s regular guarantees targeting SMEs, and the World Bank’s TSME lending portfolio. Bank-Group wide, IEG found that a clear, strategic approach was lacking and concluded that TSME needed to be more firmly rooted in a clear, evidence-based understanding of what defines an SME and how the proposed support would remove constraints to the ability of SMEs to contribute to employment, growth, and economic opportunity in developing economies. Inconsistencies and lack of coordination across the Bank Group resulted in missed opportunities, requiring a harmonized approach to SMEs. At the country level, TSME needed to be supported by analytic work that was adapted to local conditions and grounded in the Country Partnership Strategy (taking advantage of complementarity between the World Bank, IFC and MIGA) and include an M&E framework to maximize learning. A fiscally sustainable health care system is one of the elements of ECA’s second strategic pillar: “environmental, social, and fiscal sustainability. ”Experience in ECA has been mixed. For example, the Bank helped reduce out-of-pocket expenses by increasing hospital reimbursements, but was not able to achieve the objective of increasing access 76 ATTACHMENT: REGIONAL UPDATES for the poor in Armenia.7. In Romania, the Bank supported the implementation of a copayment system to promote the use of health services in a rational way and prepare the ground for the introduction of private health insurance, but a substantial share of health service users were still exempted from enrolling in the system.8. A recent IEG evaluation, World Bank Group Support to Health Financing (2014), concluded that the Bank had some notable successes in improving revenue collection, risk pooling, and purchasing, and helped increase government health budgets and protect health spending during economic crisis. Common success factors included government commitment and capacity; high quality analytical work; capacities and collaboration within the Bank Group, integration of health financing functions; and sound M&E. EVALUATION FINDINGS FROM COUNTRY PROGRAM PERFORMANCE IN THE REGION Overall outcomes of Bank Group country programs in ECA have improved. Country outcomes were rated MS+ in only 43 percent of IEG Country Assistance Strategy Completion Report (CASCR) reviews in FY12 (Table 7). Subsequently, country outcome ratings improved to 50 percent in FY13 and to 100 percent in FY14. The most successful (S- rated) Georgia country program benefited from a selective program which focused on areas where the Bank had a comparative advantage, and from a strong results framework with clearly defined objectives and outcomes and measurable indicators that provided a clear results chain. Less successful (MU-rated) country programs, on the other hand, often had broad or overly ambitious objectives which were not supported by Bank Group operations, and poorly developed results frameworks. Not surprisingly, the countries with less than satisfactory country outcomes in FY12-13 are also countries with less than average project portfolio performance, other than Belarus which only had one project evaluated (Table 4). Social programs were a challenge even in successful country programs. Bank Group country programs are divided into strategic pillars with IEG rating the outcomes for each pillar. The seven country programs reviewed in FY14 contained a total of 20 strategic pillars of which two were rated MU by IEG. Both related to social sector programs. In Moldova, IEG rated the pillar “Minimizing Social and Environmental Risks, Building Human Capital, and Promoting Human Capital” MU due to slow and partial progress in improving the efficiency of social protection and education systems, and a lack of analytic input and insufficient evidence of results in water and environment. In the Kyrgyz Republic, IEG rated the pillar “Building Sustainable Human and Social Capital” MU because no progress had been made in improving the targeting of social benefits, a lack of evidence of improvement in the fiscal sustainability of the pension system, mixed results in improving the efficiency of water utilities, and a lack of evidence of increased access to safe and clean water. Targeting social assistance to the poor has also proved to be difficult in other countries: Belarus, Croatia,9 Ukraine, Russia, and Bosnia and Herzegovina. 77 ATTACHMENT: REGIONAL UPDATES FY14 CASCR reviews bring out lessons for analytic and advisory activities (AAA), which play an important role in the Region. The Polish experience highlighted the need to put in place mechanisms for systematic planning, management, and evaluation of AAA and technical assistance (TA) programs. Good analytic work needs to underpin Bank operations (Georgia) and the Bank Group is most effective when it combines policy dialogue, knowledge, and investment (Moldova). Demand-based advisory services can be a very effective tool in middle-income countries (Romania). However, good, but supply driven, AAA should not be confused with government ownership of the evidence and recommendations. Figure 3. IEG Development Outcome Ratings for Europe and Central Asia Operations Relative to World Bank Average, FY00-13 100% 90% 80% 70% Percentage 60% 50% 40% 30% 20% 10% 0% % of ICRR in all ICR ECA IEG outcomes MS or higher Bank‐wide IEG outcomes MS or higher Source: Business Warehouse Note: ECA=Europe and Central Asia Region; ICR=Implementation Completion Report; ICRR=Implementation Completion Report Review; IEG=Independent Evaluation Group; MS=moderately satisfactory. 78 ATTACHMENT: REGIONAL UPDATES Table 15. Europe and Central Asia Region: IEG Development Outcome Ratings by Global Practice for Operations Exiting in FY11-13 Europe and Central Asia Region World Bank Average MS or higher Total MS or higher Total Number Number Percentage of Percentage of of total evaluated of total evaluated Global Practices Number % commitment projects Number % commitment projects Agriculture 5 63 62 8 30 64 63 47 Education 6 60 17 10 51 67 88 76 Energy & Extractives 15 79 92 19 38 66 81 58 Environment & Natural Resources 6 50 57 12 33 59 64 56 Finance & Markets 10 100 100 10 42 71 94 59 Governance 8 89 97 9 34 60 76 57 Health, Nutrition & Population 7 78 76 9 52 71 83 73 Macro Economics & Fiscal Management 8 80 97 10 45 73 86 62 Social Protection & Labor 8 67 71 12 27 79 93 34 Social, Urban, Rural and Resilience Global Practice 19 83 91 23 81 79 81 103 Transport & ICT 13 93 80 14 60 78 72 77 Water 9 64 71 14 33 62 63 53 Other 1 100 100 1 3 50 96 6 Grand Total 115 76 88 151 529 70 81 761 Source: Business Warehouse Note: MS=moderately satisfactory; "Other" represents global practices with fewer than five projects within the ECA region. As relates to Bank- wide figures, other includes the same global practices with fewer than five projects, as in ECA, in addition to global practices which were not represented in ECA but which were present Bank-wide. Figures for percentage of total commitment relate solely to IBRD/IDA funding and exclude projects funded through trust funds. 1 Regional Economic Update, Oct. 8, 2014 (ECA presentation for the Annual Meetings). 2 ECA Regional Update 2014. 3 Albania, Bosnia and Herzegovina, Montenegro, Serbia, and Kosovo had project outcomes ratings that fell below the regional average of 76 percent MS+. Croatia and Romania, countries which are located in the Balkans but are members of the EU, also had below regional average project performances. These seven countries accounted for more than half (56 percent) of the projects rated less than satisfactory (below MS+) in ECA. 4 IEG, Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms of Business Regulations, 2014. 5 IEG, The Big Business of Small Enterprises: Evaluation of the World Bank Group Experience with Targeted Support to Small and Medium- Size Enterprises, 2006-12, 2014. Targeting means focusing benefits on one size-class of firms to the exclusion of others. 6 At around 80 percent, ECA had the highest share of successful interventions for both WB and IFC. For the period of FY07-13 the region accounted for 24 per cent of WB and 17 per cent of IFC investment climate interventions. 7 Armenia, CASCR Review. 8 Romania, CASCR Review 9 IEG’s review of the FY09-12 CASCR rated the social pillar (“Improving the Quality and Efficiency in Social Sectors”) as moderately unsatisfactory, noting that “The Economic Recovery DPL (FY11) supported the enactment of the Social Welfare Law in 2011, which aimed to reduce social exclusion and improve the efficiency of the system including through better 79 ATTACHMENT: REGIONAL UPDATES targeting. However, it is unclear whether these activities have led to efficiency gains in Croatia’s social programs. IEG’s ICR review of the SWDP (Social Welfare Development Program) rated its development outcome as moderately unsatisfactory, noting that the project did not implement activities intended to improve the targeting of benefits and services. There is no information on the implementation of the Social Welfare Law in 2011”. The Croatia country team, however, states that targeting social assistance to the poor is comparatively good in Croatia. ATTACHMENT: REGIONAL UPDATES Table 16. IEG Development Outcome Ratings of IFC Investment and MIGA Guarantee Operations in Europe and Central Asia Region and Overall, FY11-13 IFC Europe and Central Asia IFC Overall Cluster No. of XPSRs/PES Success Commit of Commit of Success Number of XPSRs/PES Success Commit of Commit of Success XPSRs/PES evaluated as rate evaluated mostly rate by XPSRs/PES evaluated rate evaluated mostly rate by evaluated mostly (%) operations successful commit evaluated as mostly (%) operations successful commit successful (US$ or higher (%) successful (US$ or higher (%) or higher millions) (US$ or higher millions) (US$ millions) millions) Financial Markets 27 10 37 1,004 622.8 62 106 59 56 3,274 2,219 68 Infrastructure & 6 2 33 297.6 100.3 34 48 25 52 1,850 1,293 70 Natural resources Manufacturing, 22 16 73 573.4 391.8 68 78 56 72 1,915 1,462 76 Agribusiness & Services Total 55 28 51 1,875.4 1,114.8 59 232 140 60 7,039 4,974 71 MIGA Europe and Central Asia MIGA Overall No. of MIGA PERs Success No. of MIGA PERs Success projects evaluated rate projects evaluated rate evaluated satisfactory (%) evaluated satisfactory (%) or higher or higher MIGA Total 14 9 64 43 30 70 Source: IEGPE Database (XPSR and PES Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. MIGA ratings are for the FY08-13 period. MIGA's rating criteria follow a four-point rating scale: excellent; satisfactory; partly unsatisfactory; and unsatisfactory. Data includes project ratings finalized up to December 31, 2014. 81 ATTACHMENT: REGIONAL UPDATES Table 17. IEG Ratings of IFC Advisory Projects: Europe and Central Asia Region and IFC Overall, 2011-2013 Europe and Central Asia IFC Overall Business Line Number of PCRs rated as Success rate Number of PCRs rated as Success rate PCRs rated mostly (percentage) PCRs rated mostly (percentage) successful or successful or higher higher Access to Finance 6 5 83 58 39 67 Investment Climate 8 6 75 42 28 67 Public Private 29 17 59 Partnership Sustainable Business 13 9 69 78 46 59 Advisory Total 27 20 74 207 130 63 Source: IEGPE Database (PCR Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. Includes preliminary ratings. Table 18. IEG Development Outcome Ratings by Country, FY11-13 World Bank Projects IFC-IEG XPSR/PES Ratings IFC-IEG PCR Ratings MS+ MS+ MS+ Number Total Total Total of Number of Number Number Number Number Project evaluated of of rated of of rated s % projects Projects % projects Projects % projects IBRD Albania 9 69 13 3 100 3 Armenia 11 100 11 2 67 3 2 100 2 Azerbaijan 6 100 6 1 50 2 2 100 2 Belarus 1 100 1 1 100 1 Bosnia and 4 67 6 0 1 2 50 4 Herzegovina Bulgaria 1 100 1 2 50 4 Croatia 4 50 8 1 100 1 Georgia 8 89 9 1 100 1 2 100 2 Hungary 0 1 Kazakhstan 5 71 7 0 2 Latvia 2 100 2 Macedonia, former 7 88 8 1 100 1 Yugoslav Republic of Montenegro 2 50 4 1 50 2 Poland 3 100 3 Romania 5 63 8 1 50 2 Russian Federation 6 86 7 8 53 15 2 67 3 Serbia 5 63 8 0 1 0 1 Turkey 5 56 9 8 73 11 Ukraine 1 33 3 3 43 7 2 67 3 IBRD Subtotal 85 74 115 28 55 51 15 71 21 Blend Moldova 8 80 10 Uzbekistan 2 67 3 2 100 2 82 ATTACHMENT: REGIONAL UPDATES World Bank Projects IFC-IEG XPSR/PES Ratings IFC-IEG PCR Ratings MS+ MS+ MS+ Number Total Total Total of Number of Number Number Number Number Project evaluated of of rated of of rated s % projects Projects % projects Projects % projects Blend Subtotal 10 77 13 2 100 2 IDA Kosovo 2 67 3 0 1 Kyrgyz Republic 10 100 10 Tajikistan 7 88 8 IDA Subtotal 19 90 21 2 67 3 Regional Central Asia 0 1 0 4 1 100 1 South Eastern 1 100 1 Europe and Balkans Regional Subtotal 1 50 2 Grand Total for Europe 115 76 151 28 51 55 20 74 27 and Central Asia Region Source: IEGPE Database (XPSR Evaluation Notes, PCR Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. Includes preliminary ratings. Table 19. Europe and Central Asia Region: New World Bank Lending Commitments by Global Practice, FY10-14 (US$ millions) Global Practice FY10 FY11 FY12 FY13 FY14 Agriculture 36 18 50 239 Education 2 29 17 57 54 Energy & Extractives 852 1,811 1,320 391 852 Environment & Natural Resources 30 44 60 Finance & Markets 3,157 344 368 381 300 Governance 70 33 122 86 Health, Nutrition & Population 7 149 10 145 554 Macro Economics & Fiscal Management 4,987 1,549 3,035 2,278 2,356 Poverty 20 10 Social Protection & Labor 191 924 61 21 Social, Urban, Rural and Resilience Global Practice 315 341 221 145 400 Trade & Competitiveness 24 10 76 50 Transport & ICT 896 260 1,248 890 45 Water 314 199 73 273 510 Grand Total 10,816 5,725 6,381 4,862 5,527 Source: Business Intelligence as of October 15, 2014 Note: The New World Bank Lending Commitments are the sum of IBRD and IDA commitments for PE projects approved between FY10-14. 83 ATTACHMENT: REGIONAL UPDATES Table 20. Europe and Central Asia Region: IEG Ratings of Project M&E Quality, FY11-13 M&E Quality Substantial or Better Total number of rated Number of projects Percentage projects IBRD 41 36 115 Blend 4 31 13 IDA 9 43 21 Regional 0 2 Grand Total for Europe and Central 54 36 151 Asia Region Grand Total Bank-wide 224 29 765 Table 21. Europe and Central Asia Region: CASCRR Outcome and Performance Ratings, FY12-14 FY of review Country CASCR review period Outcome rating Bank performance IFC Performance 2012 Bosnia and Herzegovina FY08-11 MS MS MS Kazakhstan FY05–11 MU MU MS Russian Federation FY07-11 MS MS MS Serbia FY08-11 MU MS MS Turkey FY08-11 MS MS S Ukraine FY08-11 MU S MS Uzbekistan FY08-11 MU MU MS 2013 Belarus FY08-12 MU MS NA Croatia FY09-13 MS MS NA 2014 Kyrgyz Republic FY07-CY12 MS MS NA Moldova FY09-13 MS MS NA Poland FY09-13 MS S NA Armenia FY09-13 MS S NR Georgia FY10-13 S Good Good Romania FY09–13 MS Good NR Tajikistan FY10-14 MS Good NR 84 ATTACHMENT: REGIONAL UPDATES Latin America and the Caribbean REGIONAL CHALLENGES AND EVALUATION FINDINGS FROM WORLD BANK GROUP OPERATIONS IN THE REGION Growth in the Latin America and the Caribbean Region (LAC) has declined over the period 2011-2013 from 4.1 percent to 2.4 percent, with a further decline to 1.9 percent expected in 2014. This reflects the initial recovery from the global financial crisis, followed by the impact of weak export growth, declining commodity prices no longer supported by a surging China, the slow growth recovery of other trading partners such as the United States, and domestic challenges.1 The LAC Regional Update for 2014 highlights that there has been good progress in poverty reduction and shared prosperity since the early 2000s. However, it also notes that LAC is still the most unequal Region in the world and the decline in inequality over the last decade has now stopped with a slowdown in growth rates. As the tailwinds from economic growth abate, addressing structural weaknesses such as low productivity, high logistics costs, low investment for infrastructure, and limited competition in domestic and external markets is important. In addition, increasing violence, environmental vulnerabilities (particularly for Central America and Small Island States), and unmet expectations in terms of the quality of public services including education and health are the key challenges for LAC countries. New World Bank lending commitments in the Region have sharply declined over the period from $9.63 billion in FY11 to $5.07 billion in FY14 (Table 5), as the demand for crisis-related lending has dropped off. The outcome of 74 percent of projects in LAC that exited during FY11-FY13 was rated moderately satisfactory or better, compared to the Bank average of 70 percent (Figure 1). The shares of operations rated moderately satisfactory or better were substantially higher than the average of other Regions for those under the education; macroeconomics and fiscal management; and transport/ICT global practices (Table 1), which accounted for 43 percent of the commitments during FY10-14 (Table 5). Projects that received favorable ratings had features such as the following:  An appropriate level of ambition and flexibility. The Argentina National Highway Asset Management Project focused on a few critical achievable goals and allowed for additional flexibility to accommodate evolving needs. The Guyana Water Sector Consolidation Project also had flexibility in project design and appropriate conditionality and targets. The Honduras Judicial Branch Modernization Project piloted a unified court management model which could be scaled up later, while supporting the nationwide rollout of an integrated financial management system for monitoring the institutional performance of courts. 85 ATTACHMENT: REGIONAL UPDATES  Long-term engagement by the Bank. The Guyana Water Sector Consolidation Project built on 10 years of Bank engagement in the sector. In the case of the Guatemala Integrated Financial Management III Project, the 15-year period of Bank engagement — from preparation of the initial operation to the closure of this one —gave sufficient time for reforms to take hold, although there is still a large capacity gap to fill. This case shows that even in a difficult situation of a country with a high level of conflict, Bank support can be valuable when working with committed partners  Internalization of monitoring and evaluation (M&E) by clients. In the case of the Brazil Espirito Santo Water & Coastal Pollution Management Project, the implementing agency internalized M&E for its own purposes, improving decision making and resource allocation processes, and ensuring sustainability. It also published results to ensure accountability with the public. The Argentina Provincial Maternal-Child Health Project had an M&E design that was innovative in combining data needs for the project M&E with the needs for the health system. In the case of the Mexico Environmental Services Project, the results measurement and lesson-learning from the selected pilot areas were designed to provide valuable guidance for the replication of such programs elsewhere in Mexico. Projects rated moderately unsatisfactory or below often supported actions at the wrong scale of ambition needed to achieve success. The Rio de Janeiro Metropolitan Urban and Housing Development Project supported some actions that were not related to the objectives, were at a disparate scale from the intended result, or overestimated the capabilities of the client state government. The project also shows the difficulties of working with agencies that present uneven levels of institutional maturity, technical competency, and leadership. The Honduras Financial Sector Technical Assistance Project supported one outcome, loan quality of top corporate borrowers of banks that was not within the scope of the project to control. Other key indicators were unobservable or difficult to measure. The LAC Region delivered the largest investment program for the International Finance Corporation (IFC) with $3.87 billion of net commitment during FY14, representing 23.4 percent of IFC’s total net commitment ($16.5 billion). With regard to the outcomes rating, out of 56 IFC investment projects validated by IEG during FY11-13, IEG assigned 41 projects with development outcome ratings of mostly satisfactory or better (73 percent). This share was higher than IFC’s overall average of 60 percent. Among three investment clusters, the Manufacturing, Agribusiness & Services (MAS) Cluster had the highest success rate of 82 percent by the number of projects. Sound IFC work quality contributed to better development outcomes in the MAS cluster. In 86 ATTACHMENT: REGIONAL UPDATES addition to financing, IFC was able to add value to its client companies by helping them improve environmental, health, and safety standards. IFC clients improved their environment and social standards by implementing the Environmental and Social Action Plan agreed with IFC during the appraisal. These improvements were possible with the proper identification of gaps in environment and social standards, effective follow-ups during supervision, and commitments by IFC clients. As for MIGA, out of seven projects validated by IEG, five were rated satisfactory or higher. Seventy percent of IFC’s advisory service projects reviewed by IEG from FY11-13 were rated as mostly successful or higher. Of particular interest is the Public-Private Partnership Business Line. All six evaluated projects in this business line were rated successful. The engagement for a major highway project in Colombia is a good example of how IFC delivered valued support in a number of different areas. As a transaction advisor, IFC suggested eliminating several inefficient mechanisms in the concession structures, bidding, and awarding processes, and secured the participation of domestic and international financial service providers, including the Colombian pension fund. The project is expected to help address a capacity bottleneck in the Colombian infrastructure including a potential to benefit nearly 5.25 million people or 11 percent of Colombia’s population from improved transportation services. FINDINGS FROM THEMATIC, GLOBAL, AND CORPORATE EVALUATIONS IEG’s The World Bank Group’s Response to the Global Economic Crisis, Phase II found that conditional cash transfer programs that had been expanded in LAC countries such as El Salvador, Guatemala, Jamaica, and Mexico since the late 1990s, were put to the test. However, these were geared mainly to help the chronically (rather than transitory) poor, so could not easily bring on large numbers of additional poor stemming from labor market contractions. The Bank addressed this in El Salvador and Mexico by launching or scaling up labor-intensive public works and income-support programs. The Bank also supported conditional cash transfer programs in Guatemala, Jamaica, and Mexico to help households keep their children in school and continue health care. IEG’s World Bank Group Support to Health Financing found a need to better prioritize health subsidies based on evidence. For example, in Nicaragua, public spending on social services is not pro-poor—it benefits all socioeconomic groups about equally. However, the Bank supported the government in bringing poor communities into the health sector by strengthening the use of locally collected data, including technical and social audits. In Mexico, a Bank study found that a risk pooling program, Seguro Popular, improves access to care, but incentivizes workers to move to the informal sector to receive health services. This could be addressed by consolidating Seguro Popular with the formal sector social health insurance program. 87 ATTACHMENT: REGIONAL UPDATES The Bank’s effectiveness as a health finance advisor depends on finding willing partners to engage with. In the early 2000s, for example, Bank lending and analytical work guided much of the policy discussion in health financing reform in Bolivia. Yet following the change of government in 2006, the new government no longer engages with the Bank for technical discussions in the health sector. In Brazil, the Bank has found willing partners at the municipal level, and worked to achieve implementation of the health budget. IEG’s World Bank Group Assistance to Low-Income Fragile and Conflict-Affected States An Independent Evaluation considered the Bank-administered financial intermediary trust fund in Haiti, an international response to the 2010 earthquake. It found some weaknesses, in comparison to trust funds in other fragile and conflict-affected countries. These included:  Complicated governance arrangements that caused friction among stakeholders; and  A trust fund structure that required all implementing partners to comply with the Bank’s operating procedures, often with significant delays. The same evaluation found that the community-driven development services for urban slum dwellers in Haiti were relatively well targeted, with good outreach to women. However, the reliance on project financing makes it difficult to build sustainable local institutions, and to promote downward accountability. IEG’s recent PPP evaluation concludes that the lack of bankable PPP projects is one of the major challenges to expand the use of the modality to fill in infrastructure financing gaps. Unsolicited proposals can help address this upstream deal gap as it can add bankable deals. In Brazil, unsolicited proposals are widespread across all sectors without having a formal framework to either reject or manage them. In Colombia, the newly approved PPP law covers unsolicited proposals for all PPP sectors, which is likely to trigger an increase in such proposals. IEG’s evaluation called for establishing Bank Group-wide practice on how to handle unsolicited bids. The Region will benefit from increasing transparency and practices of infrastructure financing through unsolicited proposals. EVALUATION FINDINGS FROM COUNTRY PROGRAM PERFORMANCE IN THE REGION Between FY12 and FY14, IEG reviewed 10 Country Assistance Strategy Completion Reports (CASCRs). Eight were rated moderately satisfactory for outcome, and two moderately unsatisfactory (Table 7). One Country Program Evaluation was carried out for Brazil, with an overall rating of moderately satisfactory. 88 ATTACHMENT: REGIONAL UPDATES A key lesson is the need for results frameworks with realistic, clearly defined objectives that can be achieved within the life of the strategy, effectively monitored, and adjusted as necessary based on progress being made on the ground (Bolivia, Mexico). Simplification of project design can help reduce delays in achieving outcomes, along with close collaboration with other development partners (Jamaica). Weak information on public sector management at the sub-national level, and in the areas of anticorruption, transparency, results monitoring and the judicial system in Colombia remains even though the International Bank for Reconstruction and Development (IBRD) provided support in setting up a sophisticated M&E system over 15 years ago. Objectives that are well monitored using verifiable data show the best development results (Peru). Some clients lacking capacity to design a coherent and results-focused strategy face numerous challenges hindering the Bank’s support, including inability to qualify for budget support to facilitate policy dialog, weak donor coordination, and poor data for M&E. One approach in such settings is an expanded role for analytic and advisory activities support, focusing on a few areas where advances on the technical level can be upgraded to higher-level policy dialog and coordination (Nicaragua). A related point is that the Bank needs a better system to monitor and align the work of IFC with the overall Bank Group strategy, to better take advantage of good initiatives that can be scaled up to benefit the countries (Brazil, Peru). IFC also needs exit strategies and the ability to execute them to help cut losses from non-performing investments (Mexico). During the FY12-14 period, IEG rated IBRD/International Development Association (IDA) performance as moderately satisfactory or better for seven out of ten programs. A key challenge for the three programs that were rated moderately unsatisfactory was the weakness in monitoring the progress and outcomes of the country program. Unrealistic objectives in the results matrix were also pointed out. IFC’s performance was rated separately for six countries in the Region. Except for a moderately unsatisfactory rating for the Bolivia program due to limited new activities developed during the review period, IEG assigned moderately satisfactory or better ratings to IFC performance to five other countries (Table 7). 89 ATTACHMENT: REGIONAL UPDATES Figure 4. IEG Development Outcome Ratings for Latin America and Caribbean Operations Relative to World Bank Average, FY00-13 100% 90% 80% 70% Percentage 60% 50% 40% 30% 20% 10% 0% % of ICRR in all ICR LCR IEG outcomes MS or higher Bank‐wide IEG outcomes MS or higher Source: Business Warehouse Note: LCR=Latin America and Caribbean Region; ICR=Implementation Completion Report; ICRR=Implementation Completion Report Review; IEG=Independent Evaluation Group; MS=moderately satisfactory. Table 22. Latin America and Caribbean Region: IEG Development Outcome Ratings by Global Practice for Operations Exiting in FY11-13 Latin America and Caribbean Region World Bank Average MS or higher Total MS or higher Total Number % Percentage Number Number % Percentage Number of total of of total of commitment evaluated commitment evaluated Global Practices projects projects Agriculture 5 63 85 8 30 64 63 47 Education 10 83 97 12 51 67 88 76 Energy & Extractives 4 67 94 6 38 66 81 58 Environment & Natural Resources 9 60 52 15 33 59 64 56 Finance & Markets 7 78 98 9 42 71 94 59 Governance 9 60 96 15 34 60 76 57 Health, Nutrition & Population 7 64 91 11 52 71 83 73 Macro Economics & Fiscal Management 11 85 94 13 45 73 86 62 Social Protection & Labor 4 80 98 5 27 79 93 34 Social, Urban, Rural and Resilience Global Practice 10 77 45 13 81 79 81 103 Transport & ICT 13 93 99 14 60 78 72 77 Water 4 80 72 5 33 62 63 53 Other 1 100 100 1 3 50 96 6 Grand Total 94 74 91 127 529 70 81 761 Source: Business Warehouse Note: MS=moderately satisfactory; "Other" represents global practices with fewer than five projects within the LCR region. As relates to Bank- wide figures, other includes the same global practices with fewer than five projects, as in LCR, in addition to global practices which were not represented in LCR but which were present Bank-wide. Figures for percentage of total commitment relate solely to IBRD/IDA funding and exclude projects funded through trust funds. 1 Global Economic Prospects, June 2014. 90 ATTACHMENT: REGIONAL UPDATES Table 23. IEG Development Outcome Ratings of IFC Investment and MIGA Guarantee Operations in Latin America and Caribbean Region and Overall, FY11-13 IFC Latin America and the Caribbean IFC Overall Cluster No. of XPSRs/PES Success Commit of Commit of Success No. of XPSRs/PES Success Commit of Commit of Success XPSRs/PES evaluated rate evaluated mostly rate by XPSRs/PES evaluated rate evaluated mostly rate by evaluated as mostly (%) operations successful commit evaluated as mostly (%) operations successful commit successful (US$ or higher (%) successful (US$ or higher (%) or higher millions) (US$ or higher millions) (US$ millions) millions) Financial Markets 24 16 67 732 572 78 106 59 56 3,274 2,219 68 Infrastructure & Natural resources 15 11 73 497 390 79 48 25 52 1,850 1,293 70 Manufacturing, Agribusiness & Services 17 14 82 444 372 84 78 56 72 1,915 1,462 76 Total 56 41 73 1,673 1,334 80 232 140 60 7,039 4,974 71 MIGA Latin America and the Caribbean MIGA Overall No. of MIGA PERs Success No. of MIGA PERs Success projects evaluated rate projects evaluated rate evaluated satisfactory (%) evaluated satisfactory (%) or higher or higher MIGA Total 7 5 71 43 30 70 Source: IEGPE Database (XPSR and PES Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. MIGA ratings are for the FY08-13 period. MIGA's rating criteria follow a four-point rating scale: excellent; satisfactory; partly unsatisfactory; and unsatisfactory. Data includes project ratings finalized up to December 31, 2014. 91 ATTACHMENT: REGIONAL UPDATES Table 24. IEG Ratings of IFC Advisory Projects: Latin America and Caribbean Region and IFC Overall, 2011-2013 Latin America and Caribbean IFC Overall Business Line Number of PCRs rated as Success rate Number of PCRs rated as Success rate PCRs rated mostly (percentage) PCRs rated mostly (percentage) successful or successful or higher higher Access to Finance 12 7 58 58 39 67 Investment Climate 7 5 71 42 28 67 Public Private 6 6 100 29 17 59 Partnership Sustainable Business 15 10 67 78 46 59 Advisory Total 40 28 70 207 130 63 Source: IEGPE Database (PCR Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. Includes preliminary ratings. Table 25. IEG Development Outcome Ratings by Country, FY11-13 World Bank Projects IFC-IEG XPSR/PES Ratings IFC-IEG PCR Ratings MS+ Total MS+ Total MS+ Total Number % Number Number % Number Number % Number of of of of rated of of rated Projects evaluated Projects projects Projects projects projects IBRD Argentina 8 89 9 4 100 4 Brazil 12 52 23 11 79 14 6 75 8 Chile 3 100 3 1 33 3 Colombia 5 100 5 10 91 11 5 100 5 Costa Rica 1 100 1 0 Dominican Republic 3 75 4 2 100 2 El Salvador 4 67 6 0 1 Guatemala 2 100 2 1 100 1 Jamaica 2 67 3 1 100 1 Mexico 10 91 11 4 80 5 1 20 5 Panama 0 1 1 100 1 Peru 5 71 7 5 63 8 5 63 8 Trinidad and Tobago 1 100 1 0 1 Uruguay 5 100 5 1 100 1 IBRD Subtotal 61 76 80 36 75 48 22 69 32 Blend Bolivia 2 67 3 1 100 1 Dominica 1 100 1 Grenada 3 75 4 St. Lucia 1 50 2 0 1 St. Vincent and the Grenadines 1 50 2 Blend Subtotal 8 67 12 1 100 1 0 1 IDA Guyana 2 100 2 Haiti 5 63 8 1 100 1 1 100 1 Honduras 4 44 9 2 100 2 Nicaragua 4 80 5 2 67 3 0 1 92 ATTACHMENT: REGIONAL UPDATES IDA Subtotal 15 63 24 3 75 4 3 75 4 Regional Caribbean 2 100 2 Central America 2 67 3 Latin America 3 100 3 OECS Countries 3 100 3 Regional Subtotal 10 91 11 1 33 3 3 100 3 Grand Total for Latin America and 94 74 127 41 73 56 28 70 40 Caribbean Region Source: IEGPE Database (XPSR Evaluation Notes, PCR Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. Includes preliminary ratings. Table 26. Latin America and Caribbean Region: New World Bank Lending Commitments by Global Practice, FY10-14 (US$ millions) Global Practice FY10 FY11 FY12 FY13 FY14 Agriculture 188 90 388 580 290 Education 1,563 39 626 290 523 Energy & Extractives 2,004 822 50 110 50 Environment & Natural Resources 74 75 710 Finance & Markets 300 755 100 Governance 65 134 128 1,100 286 Health, Nutrition & Population 2,073 1,107 80 220 10 Macro Economics & Fiscal Management 2,732 1,650 1,747 796 1,215 Poverty 50 500 623 Social Protection & Labor 868 2,178 75 615 72 Social, Urban, Rural and Resilience Global Practice 190 844 427 714 307 Trade & Competitiveness 461 32 480 20 350 Transport & ICT 2,532 949 1,015 701 932 Water 858 905 304 58 410 Grand Total 13,907 9,629 6,629 5,204 5,068 Source: Business Intelligence as of October 15, 2014 Note: The New World Bank Lending Commitments are the sum of IBRD and IDA commitments for PE projects approved between FY10-14. Table 27. Latin America and Caribbean Region: IEG Ratings of Project M&E Quality, FY11-13 M&E Quality Substantial or Better Total number of rated Number of projects Percentage projects IBRD 30 37 81 Blend 1 8 12 IDA 5 21 24 Regional 0 11 Grand Total for Latin America and 36 28 128 Caribbean Region Grand Total Bank-wide 224 29 765 93 ATTACHMENT: REGIONAL UPDATES Table 28. Latin America and Caribbean Region: CASCRR Outcome and Performance Ratings, FY12- 14 FY of review Country CASCR review period Outcome rating Bank performance IFC Performance 2012 Bolivia FY10-11 MU MU MU Brazil FY08-11 MS MU MS Colombia FY08-11 MS MS MS Costa Rica FY09-11 MS MU NA Honduras FY07-11 MU MS NR Peru FY07-11 MS MS S 2013 Guatemala FY09-12 MS MS MS Nicaragua FY08-12 MS S S 2014 Mexico FY08-13 MS MS NR Jamaica FY10-13 MS Good NR 94 ATTACHMENT: REGIONAL UPDATES Middle East and North Africa REGIONAL CHALLENGES AND EVALUATION FINDINGS FROM WORLD BANK GROUP OPERATIONS IN THE REGION Developing countries in the Middle East and North Africa (MNA) Region suffered from a turbulent and highly insecure geopolitical context, and weak export demand over the period, with gross domestic product (GDP) growth per annum going from -0.8 percent in 2011, to -0.1 percent in 2013. A pickup in oil production, manufacturing, and exports is forecast to increase growth to 1.9 percent for 2014. Structural reforms are being delayed in many countries to await political transitions, with recent or planned elections in Tunisia, Egypt, Lebanon, Iraq, Algeria, Yemen, and Libya. Fiscal deficits exceeded 7 percent of GDP for 2012-13 to make up for declining revenues, coupled with increased spending on public sector wages and general subsidies. Foreign direct investment was only $13.4 billion in 2013, less than half the amount in 2008. Remittances also declined in 2013 by 2 percent, mainly because of more stringent enforcement of labor regulations in Saudi Arabia, leading to the return of 300,000 migrants back to Egypt.1 This is down from an average 12 percent per annum growth over the previous three years (World Bank, 2014: 61-7). The recent sharp drop in oil prices will have a mixed effect on developing countries in the Region, helping the oil importing countries Djibouti, Arab Republic of Egypt, Jordan, Lebanon, Morocco, Tunisia, and West Bank and Gaza, but hurting oil exporters Algeria, the Islamic Republic of Iran, Iraq, Libya, the Syrian Arab Republic, and the Republic of Yemen. New World Bank lending commitments in the Region have been uneven over the period with $2.07 billion in FY11, $1.51 billion in FY12, and $2.06 billion in FY13. There was an increase in FY14 to $2.79 billion (Table 5). The Bank’s strategic framework focuses on solutions tailored to meet the needs of clients, doing different things and doing things differently to reduce poverty and enhance shared prosperity, stepping up engagement on gender, and addressing governance reforms and accountability deficits. The Bank’s lending operations performed somewhat worse in MNA than in other Regions, with 60 percent of projects that exited during FY11-13 rated moderately satisfactory or better, compared to the Bank average of 70 percent (Table 1). The International Finance Corporation’s (IFC) net commitment in the Region increased from $581 million in FY11 (5.1 percent of total IFC net commitment), to $1.29 billion in FY12 (9.0 percent), and declined to $1.19 billion in FY13 (6.8 percent)2. IFC's strategic framework supports a level playing field for the private sector and investment climate reforms to enable inclusive growth, jobs, and prosperity. 95 ATTACHMENT: REGIONAL UPDATES Projects receiving favorable ratings had features such as the following:  Effective monitoring and evaluation (M&E) suitable for the context. In the Morocco Rural Roads Project, there was a well-defined "measure of accessibility" to all-weather roads by the rural population. All stakeholders understood and internalized the measure based on yearly accessibility reports that were distributed to all provinces.  Synergy through multiple Bank engagements. The success of the Egypt Mortgage Finance Project built on the 2002 Financial Sector Assessment Program, which analyzed challenges and provided policy recommendations. There was also a series of Financial Sector Development Policy Operations supporting reforms over the period 2004−12. These operations helped with key building blocks of the legal and institutional framework of the mortgage market.  Clear output and outcome indicators. The West Bank & Gaza Emergency Services Support Program Multi-donor Trust Fund closely monitored implementation through clear output and outcome indicators. An external auditor carried out spot-check audits that helped take the place of mid-term reviews and other formal performance assessments.  The Lebanon Municipal Infrastructure Project helped improve living conditions in the affected areas by restoring basic services and rebuilding priority public infrastructure. It supported local economic recovery in the most affected municipalities, and helped put in place systems that would improve municipal finances. The scope of the project was kept simple and allowed maximum flexibility. It used an experienced project unit rather than creating a new structure, and benefitted from a technically qualified Bank team, some stationed in the field.  The Yemen Education for all Fast-track Initiative Catalytic Fund Phase III Project found that direct supervision of project activities by the Bank team was made impossible by the civil unrest; in this setting, a third party monitoring agency helped to verify implementation of the quality of civil works, and to make recommendations for follow-up actions. These processes helped to sustain project momentum in spite of the fragile context. Projects with unfavorable ratings had weaknesses in design and monitoring.  The Tunisia Employment Development Policy Operation (DPO) was based on previous Bank diagnostics that identified several factors in the labor, education, and business markets that affect employment including: oversupply of university graduates and shortages of vocational technicians; rigid employment laws; and constraints to the expansion of small businesses. However, the design did not adequately address these weaknesses and the related legal changes were 96 ATTACHMENT: REGIONAL UPDATES too small to have an impact on the efficiency and effectiveness of job creation, or on key labor market constraints.  In the case of the Gaza Emergency Water Project, support to the Coastal Municipalities Water Utility should have been based on improving its financial standing and improving cost recovery. Yet the estimated value of the monitoring indicator for revenue collection was too imprecise to determine if the target was achieved or not and if the actual value decreased or increased.  The Tunisia cultural heritage project was limited by an M&E system that was unable to measure important desired results such as institutional strengthening for conservation, management, and promotion of cultural heritage, civil participation in cultural asset protection, and the resulting economic gain. Although the monitoring indicators were restructured during the mid-term review, they still remained output oriented, and didn't capture the intended outcomes of Bank support. IFC’s investment operations performed about the same in MNA as in other Regions, with 61 percent of projects evaluated during FY11-13 rated moderately satisfactory or better, compared to the average for all Regions of 60 percent (Table 2). This was a decline from the MNA rate for FY08-10 of 80 percent, as the global financial crisis and regional turmoil starting with the Arab Spring began to impact the investment performance and development results of some IFC projects. For example, IFC was not able to meet the targets of its manufacturing sector project in Tunisia, though IFC’s financial support helped the company to continue its commercial operation despite weak demand and increases in operational costs due to the regional crisis.3 Despite the increasingly difficult environment in the region, IFC succeeded in helping a regional player in the water and wastewater treatment industry to transition from a family-owned entity to a global player, and added value in terms of global knowledge in the industry and the improvement of corporate governance. The success rate of IFC’s advisory service projects declined from 66 percent for 2008-10 to 50 percent in 2011-13, compared to an overall IFC success rate of 63 percent (Table 3). IFC took risks to implement these projects in difficult environments in the Region with mixed results. Despite the challenging country context, IFC was able to help strengthen the capacity of an investment promotion agency in Yemen, increasing the number of investment leads in the country. Due to the lack of government ownership, the public- private partnership (PPP) reform agenda was not implemented in Yemen. IFC had an important role in the initial stages of the PPP advisory project in Syria by fostering the first PPP project, but the project needed to be terminated with the onset of civil unrest in the country from 2011. 97 ATTACHMENT: REGIONAL UPDATES FINDINGS FROM THEMATIC, GLOBAL, AND CORPORATE EVALUATIONS In responding to the global economic crisis, there were good examples of cooperation among international partners. For example, IEG’s The World Bank Group’s Response to the Global Economic Crisis, Phase II found that in both Egypt and Morocco, the International Monetary Fund (IMF), World Bank, and African Development Bank staff worked together on technical issues although there was no IMF program involvement or financial support. Most countries that received DPOs with financial sector content during the crisis did not have a significant deterioration in country conditions or a financial systems crisis. There were setbacks in some areas, such as an increase in sovereign debt spreads in Egypt, compounded by the dollarization of public debt facing reductions in nonresident holdings. Yet because of significant earlier reforms, some with Bank assistance, Egyptian banks did not suffer from excessive leverage. Conditions for women remained a challenge in the Region, with mixed results, according to IEG’s World Bank Group Support to Health Financing. For example, the majority of microfinance clients in Yemen were women, in line with the Region’s commitment to step up engagement on gender. The Social Fund for Development supported these programs by providing financial services (loans, savings) to low- income clients, especially women, focusing on improved living standards and increased income and economic activity. However, Yemen’s legal and judicial system received little support, and women continue to face severe legal constraints. Yemen became a pilot country for the Bank’s governance and anticorruption initiative, aimed at strengthening governance through innovative approaches. While Yemen achieved some progress in governance due to its Extractive Industries Transparency Initiative accession, Bank results were disappointing overall in this area. Civil service reform has yet to show results, and public financial management reforms have had minimal impact. These efforts were perhaps too complex, technocratic, without sufficient political support, and insufficiently focused on the need for broad coalition building. In another sector, Bank support in Yemen for growth and jobs had some success, but important objectives were not achieved. Notably, there was little diversification of the economy away from oil and inadequate creation of sustainable jobs. Migration to neighboring countries is an important livelihood strategy for youth, but has received no attention. Yemen continues to face daunting challenges in view of rapid population growth, a lack of access to land, and an urgent need for economic diversification. In Egypt, Bank support has strengthened management and information systems in the health sector. Supervision of health facilities improved, as did information and 98 ATTACHMENT: REGIONAL UPDATES reporting systems and the validity of routine data. In addition, performance-based payment positively affected provider behaviors toward patients, reduced staff turnover, and helped increase the number of patients treated by medical providers from three to 16 per day. However, there has been discord within the government over health sector reforms; the government didn't scale up the performance-payment component after donor funding ended, even though it was perceived as successful. The Bank has since found it hard to reengage. A proposed Program 4 Results (P4R) Healthcare Support Project under preparation would (i) expand equitable access to family health services; (ii) improve health system response to neonatal and obstetric emergencies; and (iii) improve patient and blood safety. It would also introduce grievance and redress mechanisms and incorporate customer satisfaction as part of the pay-for-performance criteria. IEG’s World Bank Group Support to Public-Private Partnerships: Lessons from Experience in Client Countries, FY02-12 found that one of the core strengths of the World Bank Group is the ability to offer the comprehensive “PPP solution package” to client countries. Both the Bank and IFC’s Advisory Services were involved in the development of the 500 MW Ouarzazate solar power plant through a PPP. IFC acted as a transaction advisor of the Moroccan Agency for Solar Energy (MASEN) to provide, during the prequalification phase, general advice on design of the operation. The Bank assisted MASEN in financing its power purchasing agreement with the concessionaire company by partially covering the incremental cost of concentrated solar power over conventional technologies. However, IFC’s engagement on PPP projects in Morocco faced challenges due to the lack of readiness by the government to implement PPP projects during the CPS period. EVALUATION FINDINGS FROM COUNTRY PROGRAM PERFORMANCE IN THE REGION There were three Country Assistance Strategy Completion Reports (CASCRs) completed during the period, with two (Jordan and Morocco) rated moderately satisfactory, and one (Djibouti) rated moderately unsatisfactory. One Country Program Evaluation was carried out for Tunisia, with an overall rating of unsatisfactory. The Jordan strategy shows how institutional change can be supported through timely analytical work and technical assistance, thus enabling governance reforms in line with the Regional framework for engagement. Yet there is a need to better analyze political economy risks and the alternative approaches that the World Bank Group may take when those risks materialize. Within this spirit, CASCRs seem particularly appropriate to deal with those risks when they materialize, keeping the Executive Board informed and abreast of developments. The Jordan strategy also shows good collaboration in strategy preparation between the International Bank for Reconstruction and Development (IBRD) and IFC, where there is a clear division of labor, allowing each 99 ATTACHMENT: REGIONAL UPDATES institution to focus on their strengths and experiences while exploiting important synergies. For example, IBRD, together with the Multilateral Investment Guarantee Agency (MIGA), supported construction of the Amman East Power Plant, while IFC provided its financial support for the distribution network since Jordan privatized all electricity distribution through participating in the privatization of the public distribution company. The Bank and IFC cooperated well, including a joint IBRD/IFC mission to assess the impact of the global financial crisis on the country as per the request of the government. During the Country Assistance Strategy (CAS) period, IFC also substantially expanded its operations in Jordan, growing from around $45 million in FY06 to $432 million in FY10. The World Bank’s program in Djibouti shows the importance of government ownership, effective supervision, and close alignment between program interventions and objectives. Sound results and monitoring frameworks are also essential. The Bank Group's positive relationship with the government of Morocco and other stakeholders pointed out good practice in risk management, and a coherent and informed approach to its lending through analytical work in areas such as climate change, renewable energy, and targeting social protection, and "how to" guidance with respect to an energy supply strategy, urban development, transport, pension reform, and other relevant areas. The Bank exercised good selectivity in instrument choice, combining investment lending with an extensive DPO program, including multi-sector operations. However, the inclusion of better defined, high-level objectives in the results framework would have helped to strengthen the program and lending choices made. The Tunisia Country Program Evaluation highlighted the importance of selectivity, and using DPOs to achieve institutional objectives when there is an appropriate policy framework and client buy-in. It is also critical to implement new legislation to build on financial and institutional capacity, to carry out candid risk assessment with exit strategies, and to protect the Bank’s reputation for honest, unbiased, and high-quality reporting and analysis. This was in line with the Region’s commitment to address governance reforms. The IFC helped improve investor confidence in Morocco during its recovery from the regional turmoil by increasing its investment commitments and taking risks with substantial equity investments (over 95 percent of IFC’s commitments) together with IFC’s Asset Management Company. IFC also provided valuable advisory services, particularly with respect to microfinance and dispute resolution. 100 ATTACHMENT: REGIONAL UPDATES Figure 5. IEG Development Outcome Ratings for Middle East and North Africa Operations Relative to World Bank Average, FY00-13 100% 90% 80% 70% Percentage 60% 50% 40% 30% 20% 10% 0% % of ICRR in all ICR MNA IEG outcomes MS or higher Bank‐wide IEG outcomes MS or higher Source: Business Warehouse Note: MNA=Middle East and North Africa Region; ICR=Implementation Completion Report; ICRR=Implementation Completion Report Review; IEG=Independent Evaluation Group; MS=moderately satisfactory. Table 29. Middle East and North Africa Region: IEG Development Outcome Ratings by Global Practice for Operations Exiting in FY11-13 Middle East and North Africa Region World Bank Average MS or higher Total MS or higher Total Number Number Number % Percentage of Number % Percentage of of total evaluated of total evaluated commitment projects commitment projects Global Practices Education 4 50 61 8 51 67 88 76 Finance & Markets 4 67 91 6 42 71 94 59 Social Protection & Labor 5 83 0 6 27 79 93 34 Social, Urban, Rural and Resilience Global Practice 5 71 95 7 81 79 81 103 Water 3 33 21 9 33 62 63 53 Other 13 62 21 21 295 68 79 436 Grand Total 34 60 40 57 529 70 81 761 Source: Business Warehouse Note: MS=moderately satisfactory; "Other" represents global practices with fewer than five projects within the MENA region. As relates to Bank- wide figures, other includes the same global practices with fewer than five projects, as in MENA, in addition to global practices which were not represented in MENA but which were present Bank-wide. Figures for percentage of total commitment relate solely to IBRD/IDA funding and exclude projects funded through trust funds. 101 ATTACHMENT: REGIONAL UPDATES 1See http://www.worldbank.org/content/dam/Worldbank/GEP/GEP2014b/GEP2014b_MENA.p df page 64. 2To ensure comparability with the composition of MENA from a World Bank Group perspective, Pakistan and Afghanistan are not included in IFC data for this regional update, although IFC’s classification of MENA includes both countries. 3 Management states that the IFC client has strongly recovered since IEG visit. ATTACHMENT: REGIONAL UPDATES Table 30. IEG Development Outcome Ratings of IFC Investment and MIGA Guarantee Operations in Middle East and North Africa Region and Overall, FY11-13 IFC Middle East and North Africa IFC Overall Cluster No. of XPSRs/PE Success Commit of Commit of Success Number of XPSRs/PE Success Commit Commit of Success XPSRs/PE S rate evaluated mostly rate by XPSRs/PE S rate of mostly rate by S evaluated (%) operations successfu commit S evaluated (%) evaluated successful commit evaluated as mostly (US$ l or higher (%) evaluated as mostly operation or higher (%) successful millions) (US$ successful s (US$ (US$ or higher millions) or higher millions) millions) Financial Markets 14 10 71 559.2 420.8 75 106 59 56 3,274 2,219 68 Infrastructure & Natural 4 2 50 123.0 87.0 71 48 25 52 1,850 1,293 70 resources Manufacturing, Agribusiness 5 2 40 50.6 20.0 40 78 56 72 1,915 1,462 76 & Services Total 23 14 61 732.8 527.8 72 232 140 60 7,039 4,974 71 MIGA Middle East and North Africa MIGA Overall No. of PERs Success No. of PERs Success MIGA evaluated rate MIGA evaluated rate projects satisfactory (%) projects satisfactory (%) evaluated or higher evaluated or higher MIGA Total 1 0 0 43 30 70 Source: IEGPE Database (XPSR and PES Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. MIGA ratings are for the FY08-13 period. MIGA's rating criteria follow a four-point rating scale: excellent; satisfactory; partly unsatisfactory; and unsatisfactory. Data includes project ratings finalized up to December 31, 2014. 103 ATTACHMENT: REGIONAL UPDATES Table 31. IEG Ratings of IFC Advisory Projects: Middle East and North Africa Region and IFC Overall, 2011-2013 Middle East and North Africa IFC Overall Business Line Number of PCRs rated as Success rate Number of PCRs rated as Success rate PCRs rated mostly (percentage) PCRs rated mostly (percentage) successful or successful or higher higher Access to Finance 3 2 67 58 39 67 Investment Climate 3 1 33 42 28 67 Public Private 3 0 29 17 59 Partnership Sustainable 5 4 80 78 46 59 Business Advisory Total 14 7 50 207 130 63 Source: IEGPE Database (PCR Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. Includes preliminary ratings. Table 32. IEG Development Outcome Ratings by Country, FY11-13 World Bank Projects IFC-IEG XPSR/PES Ratings IFC-IEG PCR Ratings MS+ Total MS+ Total MS+ Total Number % Number Number % Number Number % Number of of of of rated of of rated Projects evaluated Projects projects Projects projects projects IBRD Egypt, Arab Republic of 1 25 4 2 100 2 Iran, Islamic Republic of 0 2 Iraq 4 57 7 Jordan 1 50 2 0 1 0 1 Lebanon 1 25 4 3 100 3 Morocco 8 100 8 0 1 Syrian Arab Republic 0 1 Tunisia 5 56 9 1 50 2 1 100 1 IBRD Subtotal 20 56 36 6 67 9 1 33 3 .. West Bank and Gaza 8 80 10 .. Subtotal 8 80 10 IDA Djibouti 1 50 2 Yemen, Republic of 5 56 9 2 40 5 IDA Subtotal 6 55 11 2 40 5 Regional 7 100 7 0 1 Grand Total for Middle East 34 60 57 13 82 16 3 33 9 and North Africa Region Source: IEGPE Database (XPSR Evaluation Notes, PCR Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. Includes preliminary ratings. 104 ATTACHMENT: REGIONAL UPDATES Table 33. Middle East and North Africa Region: New World Bank Lending Commitments by Global Practice, FY10-14 (US$ millions) Global Practice FY10 FY11 FY12 FY13 FY14 Agriculture 36 252 3 203 Education 73 40 6 166 Energy & Extractives 676 445 591 61 Environment & Natural Resources 300 Finance & Markets 1,300 100 100 720 Governance 100 12 16 5 205 Health, Nutrition & Population 100 35 7 10 Macro Economics & Fiscal Management 550 570 250 500 500 Social Protection & Labor 70 50 305 155 150 Social, Urban, Rural and Resilience Global Practice 60 139 388 130 14 Trade & Competitiveness 160 52 Transport & ICT 402 467 40 591 Water 370 500 185 Grand Total 3,737 2,065 1,513 2,058 2,788 Source: Business Intelligence as of October 15, 2014 Note: The New World Bank Lending Commitments are the sum of IBRD and IDA commitments for PE projects approved between FY10-14. Table 34. Middle East and North Africa Region: IEG Ratings of Project M&E Quality, FY11-13 M&E Quality Substantial or Better Total number of rated Number of projects Percentage projects IBRD 5 14 36 Blend 6 60 10 IDA 3 27 11 Grand Total for Middle East and North 14 25 57 Africa Region Grand Total Bank-wide 224 29 765 Table 35. Middle East and North Africa Region: CASCRR Outcome and Performance Ratings, FY12- 14 FY of review Country CASCR review period Outcome rating Bank performance IFC Performance 2012 Jordan FY06-11 MS S S 2014 Djibouti FY09-13 MU MS NR Morocco FY10-13 MS Good NR 105 ATTACHMENT: REGIONAL UPDATES South Asia REGIONAL CHALLENGES AND EVALUATION FINDINGS FROM WORLD BANK GROUP OPERATIONS IN THE REGION Regional gross domestic product (GDP) in South Asia (SAR), measured at market prices, grew an estimated 4.7 percent in 2013, down from 5.0 percent in 2012. Regional growth in 2013 was 2.6 percentage points below average growth in 2003-12, mainly due to subdued manufacturing performance and a sharp slowing of investment growth in India.1 Extreme poverty remains high in South Asia, which accounts for 42 percent of the world’s poor (571 million people in the Region survive on less than $1.25 a day and half a billion people lack access to electricity). Work in the Region supports the Bank’s overarching goals to reduce poverty and boost shared prosperity and is based on five pillars: increasing employment and accelerating growth; enhancing human development and social welfare; strengthening governance and accountability; reducing weather, disaster, and food vulnerability; and enhancing Regional integration and cooperation. World Bank lending commitments in the Region in FY14 reached the pre-FY12 level ($10.1 billion) after a nearly 50 percent fall in FY13 mainly as a result of increased commitments in energy; social, urban, rural and resilience; transport; and water. The performance of Bank operations in the Region was somewhat above the Bank average in FY2011-13. Some 74 percent of projects exiting in FY11-13 were rated moderately satisfactory or better (MS+) compared to 70 percent Bank-wide (Table 1). Education, water, agriculture, and transport and ICT —which together account for more than half of commitments—performed better than the Regional and global averages. Projects in governance and finance and markets performed below the Bank average. Projects in International Development Association (IDA) countries performed below the Regional average (67 percent MS+). Better project design could have improved portfolio performance. There is a strong correlation between weak quality-at-entry and eventual weak performance on outcomes. Some projects failed to conduct an adequate assessment of conditions in the sector or failed to internalize lessons from previous similar operations. Many projects were rated poorly as a result of weak design — for example the absence of activities to address institutional weaknesses in an India highways project or to address security concerns in a private sector development project in Afghanistan. The same Afghanistan project was also affected by low quality of the economic analysis, which resulted in higher than expected cost of fuel choice and equipment. Experience from successful projects shows that a good design allows for combining investment, policy interventions, and capacity building supported by strong government commitment. Bhutan’s Sustainable Land Management Project is an example of this: the project combined investment with institutional support for better policies and legislation with 106 ATTACHMENT: REGIONAL UPDATES adoption of those policies. Its design benefited from sound economic analysis and Global Environmental Facility (GEF) involvement, which facilitated the incorporation of lessons from other environmental projects and other agencies. Likewise, in the Bangladesh Rural Electrification and Renewable Energy Development Project, government commitment and capacity; lessons from similar projects in other countries and from gender studies; and relevant analytic and advisory activities (AAA) contributed to better project design. The quality of monitoring and evaluation (M&E) continues to be an issue in the Region. Although higher than the Bank-wide average, only 41 percent of projects in SAR had high or substantial ratings for M&E quality. The common issues are: lack of indicators, irrelevant indicators, focus on output indicators, and weak client M&E capacity. India’s Lucknow Highway Project had a strong monitoring and information system, but did not use it for M&E. Some projects construct M&E without a causal chain between inputs, outputs, and outcomes. For example, the absence of a results framework linking activities in a logical chain to objectives affected the performance of the Afghanistan Short-term Urban Water Supply and Sanitation Project. Strong M&E was found in the India Third National HIV/AIDS Control Project where the appraisal document contained a clear results framework, a tight set of indicators, complete baseline data, mid-term and end-line targets, specification of data sources, and frequency and responsibility for data collection. The International Finance Corporation’s (IFC) performance in SAR is above average. Measured by commitment volume, IFC investments had a success rate of 81 percent, 10 percentage-points higher than the global average. Measured by number of projects, the success rate of IFC in SAR was 59 percent, similar to the global average. Stronger IFC work quality during appraisal and supervision could have enhanced development outcomes. For example in India, IFC failed to adequately factor in the foreign exchange risks of a power sector company and failed to play an active role on the advisory committee of a fund where the IFC could have added value. IFC advisory services continued to have above-average success rates (77 percent successful compared to a global average of 63 percent). Sustainable Business Advisory projects were rated MS or better 90 percent of the time, well above the IFC average of 59 percent. IFC’s ability to find strong and capable local partners drove this strong performance. FINDINGS FROM THEMATIC, GLOBAL, AND CORPORATE EVALUATIONS Promoting private sector growth and improving the investment climate will be critical to achieving South Asia’s strategic pillar on “increasing employment and accelerating growth.” Findings from IEG’s recently completed evaluations of investment climate reform and targeted support to small and medium enterprises (TSME) include: 2 107 ATTACHMENT: REGIONAL UPDATES  SAR had one of the largest shares of successful investment climate interventions, and the Bank Group succeeded in helping to enact laws and streamline processes. Support to long-term regulatory reform agendas with clear government commitment was successful. Diagnostic analysis and consultations with enterprises and other stakeholders was a strength of the Bank’s approach in SAR. Bank-IFC coordination was mainly informal, except in India.  TSME is big business for the Bank Group, averaging around $3 billion a year in commitments, disbursements, and gross exposure. SAR plays a major role in the TSME portfolio, with 15 funds alone located in India. Globally, the overall performance of TSME funds worsened over time. The main finding of the evaluation was that a strategic approach was lacking on how to define SMEs and distinguish them from other enterprises as well as on how support might remove obstacles to SME development. IEG found a need for a clearer understanding of the role of SMEs in employment and economic growth; harmonized approaches; and better M&E frameworks to enhance learning. Better healthcare systems are essential for the achievement of human development. As in other Regions, experience in healthcare financing in South Asia was mixed. In Afghanistan, Bank support contributed to increased equity in health service use for rural populations and women and improved quality of care in some areas. However, while coverage and quality of care have improved significantly, coverage of important preventive and curative services remains low in Afghanistan by global standards, including the use of modern contraceptives, antenatal care, assisted delivery, and diphtheria, pertussis, and tetanus immunizations. Overall, factors critical for Bank success included: government commitment and capacity, high quality analytical work, collaboration within the Bank Group, sound M&E, and the integration of health financing functions.3 IEG’s evaluation of IFC’s global trade finance program (GTFP) found that GTFP grew from 2 percent of IFC commitments in South Asia in 2005 to 24 percent in 2012, with India and Pakistan among the top ten GTFP countries by volume. The program has been a relevant response to demand for trade finance risk mitigation, and has improved IFC’s engagement in trade finance from its past efforts by introducing an open, global network of banks and a quick and flexible response platform to support the supply of trade finance. The program helped connect local banks with global banks. It has also helped global banks extend their capacity to operate in developing countries. IFC provided short-term credit guarantees to 13 Pakistani banks. The program had high additionality because Pakistan’s banking sector has also been perceived as high-risk due to poor credit quality, concerns over political interference in loan recovery, and political and macroeconomic instability.4 108 ATTACHMENT: REGIONAL UPDATES EVALUATION FINDINGS FROM COUNTRY PROGRAM PERFORMANCE IN THE REGION Over the last three years, IEG has reviewed Country Assistance Strategy Completion Reports (CASCRs) for India and Sri Lanka (both rated moderately satisfactory) and Pakistan (rated unsatisfactory for outcomes) and published a full Country Program Evaluation for Afghanistan which rated the program outcome moderately satisfactory. The cross-cutting lessons from these reports reiterate that: (1) country ownership matters—results are stronger where government strategies and Bank objectives coincide; (2) strong dialogue with governments, backed up by analytical work, increases effectiveness; (3) donor coordination enhances results; and (4) the results framework needs to be realistic and ensure the quality of M&E. Pakistan’s Country Assistance Strategy (CAS) was particularly weak with a complex results framework and inadequate M&E, and did not achieve the majority of outcomes. In India, the program contributed to the strong national outcomes in agricultural productivity and community development in rural areas—contributing to reduced food insecurity—and made substantial progress in education, energy, transport, and rural water and sanitation. In Sri Lanka, the program made progress in education, infrastructure, health, and strengthening the investment climate. Afghanistan’s Country Program Evaluation points to achievements in fiscal management, health, infrastructure, microfinance, and primary education. In Pakistan, outcomes achieved or mostly achieved were in the areas of safety nets, rural livelihoods, and urban services. Shortcomings were also noted. For example, efforts on public-private partnerships (PPPs), public sector governance, and disaster risk management in India were not always successful, and the fragmented approach to address child malnutrition merits further attention. In Sri Lanka, the strategic relevance and efficiency of public spending did not improve as expected, neither did progress in irrigation. In Pakistan, the majority of program outcomes were not achieved; moreover several objectives were not monitored because of the complex results framework and weak M&E. Shortcomings in Bank-IFC collaboration were also noted. India’s CAS called for stronger Bank-IFC collaboration but did not elaborate on how it was to take place in delivering specific outcomes. In Pakistan, it took several years to achieve better exchange of information at the sector and task team levels. 109 ATTACHMENT: REGIONAL UPDATES Figure 6. IEG Development Outcome Ratings for South Asia Operations Relative to World Bank Average, FY00-13 100% 90% 80% 70% Percentage 60% 50% 40% 30% 20% 10% 0% % of ICRR in all ICR SAR IEG outcomes MS or higher Bank‐wide IEG outcomes MS or higher Source: Business Warehouse Note: SAR=South Asia Region; ICR=Implementation Completion Report; ICRR=Implementation Completion Report Review; IEG=Independent Evaluation Group; MS=moderately satisfactory. Table 36. South Asia Region: IEG Development Outcome Ratings by Global Practice for Operations Exiting in FY11-13 South Asia Region World Bank Average MS or higher Total MS or higher Total Number % Percentage Number Number % Percentage Number of total of of total of commitment evaluated commitment evaluated Global Practices projects projects Education 8 100 100 8 51 67 88 76 Energy & Extractives 4 67 99 6 38 66 81 58 Finance & Markets 4 50 92 8 42 71 94 59 Governance 3 43 52 7 34 60 76 57 Health, Nutrition & Population 8 80 94 10 52 71 83 73 Social, Urban, Rural and Resilience Global Practice 9 75 87 12 81 79 81 103 Transport & ICT 10 83 68 12 60 78 72 77 Water 5 83 100 6 33 62 63 53 Other 7 78 98 9 138 67 84 205 Grand Total 58 74 86 78 529 70 81 761 Source: Business Warehouse Note: MS=moderately satisfactory; "Other" represents global practices with fewer than five projects within the SAR region. As relates to Bank- wide figures, other includes the same global practices with fewer than five projects, as in SAR, in addition to global practices which were not represented in SAR but which were present Bank-wide. Figures for percentage of total commitment relate solely to IBRD/IDA funding and exclude projects funded through trust funds. 1Global Economic Prospects, June 2014 2IEG. 2014. Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms of Business Regulations; IEG. 2014. The Big Business of Small Enterprises: 110 ATTACHMENT: REGIONAL UPDATES Evaluation of the World Bank Group Experience with Targeted Support to Small and Medium- Size Enterprises, 2006-12 3 IEG.2014. World Bank Group Support to Health Financing 4 IEG. 2013. Evaluation of the International Finance Corporation’s Global Trade Finance Program, 2006–12. ATTACHMENT: REGIONAL UPDATES Table 37. IEG Development Outcome Ratings of IFC Investment and MIGA Guarantee Operations in South Asia Region and Overall, FY11- 13 IFC South Asia IFC Overall Cluster No. of XPSRs/PES Success Commit of Commit of Success No. of XPSRs/PE Success Commit Commit of Success XPSRs/PES evaluated rate evaluated mostly rate by XPSRs/PE S rate of mostly rate by evaluated as mostly (%) operations successful commit S evaluated (%) evaluated successful commit successful (US$ or higher (%) evaluated as mostly operation or higher (%) or higher millions) (US$ successful s (US$ (US$ millions) or higher millions) millions) Financial Markets 5 3 60 161 139 86 106 59 56 3,274 2,219 68 Infrastructure & Natural 8 4 50 263 228 87 48 25 52 1,850 1,293 70 resources Manufacturing, Agribusiness 14 9 64 377 278 74 78 56 72 1,915 1,462 76 & Services Total 27 16 59 802 645 81 232 140 60 7,039 4,974 71 MIGA South Asia MIGA Overall No. of MIGA PERs Success No. of PERs Success projects evaluated rate MIGA evaluate rate evaluated satisfactory (%) projects d (%) or higher evaluated satisfacto ry or higher MIGA Total 1 1 100 43 30 70 Source: IEGPE Database (XPSR and PES Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. MIGA ratings are for the FY08-13 period. MIGA's rating criteria follow a four-point rating scale: excellent; satisfactory; partly unsatisfactory; and unsatisfactory. Data includes project ratings finalized up to December 31, 2014. 112 ATTACHMENT: REGIONAL UPDATES Table 38. IEG Ratings of IFC Advisory Projects: South Asia Region and IFC Overall, 2011-2013 South Asia IFC Overall Business Line Number of PCRs rated as Success rate Number of PCRs rated as Success rate PCRs rated mostly (percentage) PCRs rated mostly (percentage) successful or successful or higher higher Access to Finance 5 5 100 58 39 67 Investment Climate 4 3 75 42 28 67 Public Private Partnership 11 6 55 29 17 59 Sustainable Business 10 9 90 78 46 59 Advisory Total 30 23 77 207 130 63 Source: IEGPE Database (PCR Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. Includes preliminary ratings. Table 39. IEG Development Outcome Ratings by Country, FY11-13 World Bank Projects IFC-IEG XPSR/PES Ratings IFC-IEG PCR Ratings MS+ Total MS+ Total MS+ Total Number Number Number Number of % of Number % of rated Number % of rated Projects evaluated of projects of projects projects Projects Projects IBRD India 19 83 23 15 60 25 13 68 19 IBRD Subtotal 19 83 23 15 60 25 13 68 19 Blend Pakistan 7 88 8 1 14 7 1 100 1 Sri Lanka 4 80 5 3 100 3 Blend Subtotal 11 85 13 1 14 7 4 100 4 IDA Afghanistan 12 67 18 2 100 2 Bangladesh 9 69 13 3 100 3 Bhutan 4 67 6 0 1 Maldives 0 1 1 100 1 3 100 3 Nepal 3 75 4 0 1 1 100 1 IDA Subtotal 28 67 42 1 50 2 9 90 10 Grand Total for 58 74 78 17 50 34 26 79 33 South Asia Region Source: IEGPE Database (XPSR Evaluation Notes, PCR Evaluation Notes) Note: Success rate refers to projects rated as mostly successful or higher. Includes preliminary ratings. 113 ATTACHMENT: REGIONAL UPDATES Table 40. South Asia Region: New World Bank Lending Commitments by Global Practice, FY10-14 (US$ millions) Global Practice FY10 FY11 FY12 FY13 FY14 Agriculture 445 490 1,991 226 381 Education 1,454 471 1,250 595 1,498 Energy & Extractives 1,460 837 1,212 221 2,393 Environment & Natural Resources 286 357 100 Finance & Markets 3,752 57 30 130 57 Governance 107 370 119 117 Health, Nutrition & Population 296 400 192 855 100 Macro Economics & Fiscal Management 14 25 36 450 Social Protection & Labor 208 537 150 613 84 Social, Urban, Rural and Resilience Global Practice 899 560 592 831 1,433 Trade & Competitiveness 68 142 200 Transport & ICT 1,056 4,594 505 749 2,142 Water 1,290 1,291 523 1,204 Grand Total 11,334 10,130 6,446 4,474 10,059 Source: Business Intelligence as of October 15, 2014 Note: The New World Bank Lending Commitments are the sum of IBRD and IDA commitments for PE projects approved between FY10-14. Table 41. South Asia Region: IEG Ratings of Project M&E Quality, FY11-13 M&E Quality Substantial or Better Total number of rated Number of projects Percentage projects IBRD 12 52 23 Blend 7 50 14 IDA 13 31 42 Grand Total for South Asia 32 41 79 Region Grand Total Bank-wide 224 29 765 Table 42. South Asia Region: CASCRR Outcome and Performance Ratings, FY12-14 FY of review Country CASCR review period Outcome rating Bank performance IFC Performance 2012 Sri Lanka FY09-12 MS MS MS 2013 India FY09-12 MS S NA 2014 Pakistan FY10-14 MU Fair NR 114