LE A R N ING PR O DUC T Learning from IDA Experience: Le s s ons fr om IEG Eva lu ations © 2016 International Bank for Reconstruction This work is a product of the staff of The World RIGHTS AND PERMISSIONS and Development / The World Bank Bank with external contributions. The findings, The material in this work is subject to copyright. 1818 H Street NW interpretations, and conclusions expressed in Because The World Bank encourages Washington DC 20433 this work do not necessarily reflect the views dissemination of its knowledge, this work Telephone: 202-473-1000 of The World Bank, its Board of Executive may be reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they represent. noncommercial purposes as long as full attribution to this work is given. The World Bank does not guarantee the accuracy of the data included in this work. Any queries on rights and licenses, including The boundaries, colors, denominations, and subsidiary rights, should be addressed to other information shown on any map in this World Bank Publications, The World Bank work do not imply any judgment on the part Group, 1818 H Street NW, Washington, DC of The World Bank concerning the legal 20433, USA; fax: 202-522-2625; e-mail: status of any territory or the endorsement pubrights@worldbank.org. or acceptance of such boundaries. Learning from IDA Experience: Lessons from IEG Evaluations Contents ABBREVIATIONS AND ACRONYMS .................................................................................................. IV ACKNOWLEDGMENTS......................................................................................................................... V OVERVIEW ......................................................................................................................................... VI WORLD BANK GROUP MANAGEMENT COMMENTS ..................................................................... XIII 1. IDA AND THE SPECIAL THEMES .................................................................................... 1 IDA Commitments and Resource Mobilization .......................................................................................................... 2 IDA Special Themes ................................................................................................................................................. 4 Bank Group Support in IDA Special Thematic Areas ................................................................................................ 5 Portfolio Performance in IDA Countries .................................................................................................................... 7 References ................................................................................................................................................................ 9 2. EVALUATION FINDINGS IN IDA THEMATIC AREAS ................................................... 11 Inclusive Growth...................................................................................................................................................... 11 Enhance Youth Employment ............................................................................................................................................... 12 Promote Financial Inclusion and Targeted Support for SMEs ............................................................................................ 14 Develop Agriculture and Natural Resource Sectors for Poverty Reduction ........................................................................ 18 Gender Equality ...................................................................................................................................................... 21 Gender in IDA Country Strategies ....................................................................................................................................... 22 Gender in IDA Operations ................................................................................................................................................... 24 Gender in FCS Context ....................................................................................................................................................... 27 Climate Change ...................................................................................................................................................... 28 Fragile and Conflict-Affected States........................................................................................................................ 32 Crisis Response ...................................................................................................................................................... 37 References .............................................................................................................................................................. 43 3. MANAGING FOR RESULTS ........................................................................................... 46 Results and Learning .............................................................................................................................................. 46 Cross-Cutting Issues ............................................................................................................................................... 49 References .............................................................................................................................................................. 51 APPENDIX A. COUNTRY PROGRAM EVALUATIONS FROM IDA AND BLEND COUNTRIES ........ 53 APPENDIX B. THEMATIC EVALUATIONS AND LEARNING PRODUCTS ........................................ 54 APPENDIX C. LIST OF PROJECT PERFORMANCE ASSESSMENT REPORTS, FY05–14 .............. 56 ii CONTENTS Boxes Box 1.1. Treatment of Climate Change in Mozambique Country Partnership Strategy FY12 –15 ........... 6 Box 2.1. Youth Employment Interventions in Afghanistan, Bolivia, Liberia, and Timor-Leste................ 13 Box 2.2. Microfinance in Afghanistan, Bangladesh, Bolivia, and Liberia ............................................... 15 Box 2.3. Targeted Support for SMEs in Bolivia, Mongolia, Mozambique, and Zambia ......................... 17 Box 2.4. Poverty Reduction Interventions in Bank Group Country Programs in Afghanistan, Bolivia, Cambodia, Liberia, and Uganda............................................................................................................ 20 Box 2.5. Gender Interventions in Nepal, Bolivia, and Zambia ............................................................... 22 Box 2.6. Gender in IEG Thematic Evaluations ...................................................................................... 24 Box 2.7. Examples of Informing Project Design with Analysis of Potential Gender Impact ................... 25 Box 2.8. Gender Monitoring in Bank Group Support to Electricity Access ............................................ 27 Box 2.9. Gender Interventions in Afghanistan, Liberia, and Nepal ........................................................ 27 Box 2.10. Managing Climate Change Impacts in Bangladesh, Bolivia, Mongolia, and Zambia ............. 30 Box 2.11. FCS Classification and Bank Group Country Strategies for Nepal ........................................ 33 Box 2.12. Fragility Risks and Bank Group Country Strategies for Timor-Leste ..................................... 34 Box 2.13. CDD Programs in Afghanistan and Nepal ............................................................................. 35 Box 2.14. Crisis Response in Afghanistan, Mongolia, and Timor-Leste ................................................ 40 Box 2.15. Crisis Response Operations ................................................................................................. 41 Box 3.1. Challenges with Monitoring and Evaluation ............................................................................ 47 Tables Table 1.1. Resources Mobilized during Most Recent Replenishment .................................................... 4 Table 1.2. Presence of IDA Special Themes in IDA Country Strategies Approved during FY12 –15 ....... 5 Table 1.3. Evaluated Bank Projects, FY09–14 ........................................................................................ 8 Figures Figure 1.1. Summary of Commitments among International Financial Institutions by Type .................... 3 Figure 1.2. World Bank Group Performance by Country Group .............................................................. 7 Abbreviations and Acronyms ADB Asian Development Bank ADF Asian Development Fund AfDB African Development Bank AfDF African Development Fund AF additional financing AIDS acquired immune deficiency syndrome CDD community-driven development CAS country assistance strategy CPE country program evaluation CPF country partnership framework CPIA Country Policy and Institutional Assessment CPS country partnership strategy CRW Crisis Response Window DPF Development Policy Financing ECD early childhood development FCS fragile and conflict-affected states HIV human immunodeficiency virus IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report IDA International Development Association IEG Independent Evaluation Group IFAD International Fund for Agricultural Development IFC International Finance Corporation IFI international financial institutions ISR Implementation Status and Results Report LIC low-income country M&E monitoring and evaluation MDG Millennium Development Goal MIC middle-income country MIGA Multilateral Investment Guarantee Agency PAF Poverty Alleviation Fund PPAR Project Performance Assessment Report RAP Results and Performance of the World Bank Group RMS Results Measurement System RRDC resource-rich developing country SCD Systematic Country Diagnostic SDG Sustainable Development Goal SME small and medium enterprises SSA Sub-Saharan Africa SSN social safety net TA technical assistance All dollar amounts are U.S. dollars unless otherwise indicated. iv Acknowledgments This paper was prepared by an Independent Evaluation Group (IEG) team composed of Xiaolun Sun (Task Team Leader), Anis Dani, R. Pablo Guerrero, Xue Li, and Anthony Martin Tyrrell under the guidance of Geeta Batra, Monika Huppi (previous Managers), and Marie Gaarder (Manager), Nick York (Director), and the overall direction of Caroline Heider (Director-General, Evaluation). Ismail Arslan prepared the approach paper with support from Nils Fostvedt. Barbara Rice edited the report. Carla Fabiola Chacaltana and Helen Joan Mongal provided administrative support. Peer reviewers are Henock Kifle (former Chief Economist for the African Development Bank and Permanent Secretary of the Ministry of Agriculture in Ethiopia) and Johannes Linn (former Vice President for the Europe and Central Asia Region). The team wishes to thank many colleagues within the IEG for their comments and data support in preparing this document. v Overview At the United Nations General completed programs and projects, the Assembly in September 2015, world report draws on World Bank Group leaders endorsed 17 Sustainable experience prior to IDA16 and IDA17 Development Goals (SDGs) as a for the most part. In many cases, Bank successor framework to the Group management has taken actions Millennium Development Goals to address some of the issues (MDGs). Placing economic, social, and identified in IEG evaluations. This environmental sustainability at the report acknowledges the new center of development, the new initiatives, but is not in a position to agenda has the potential for a historic comment on their effectiveness. shift in achieving the goal of ending poverty and promoting shared IDA Special Themes prosperity. IDA14 introduced special themes in Within this changed and changing 2005 to reinforce the agency’s context, the operations and modus effectiveness for achieving the MDGs. operandi of the International In the ensuing years, although IDA’s Development Association (IDA) are strategic priorities evolved, there has being examined. As the world’s largest been high continuity in its overarching provider of financial resources to the goal of maximizing development poorest countries, it is expected to impact through a strong focus on deliver greater results in the new results measurement and partnership. development paradigm. In particular, IDA gives high priority to gender equality, climate change, In line with the commitment to and fragile and conflict-affected states learning from the past, this synthesis (FCS), as well as crisis response and report presents findings from recent inclusive growth. These special evaluations and analysis from the thematic areas are fully consistent Independent Evaluation Group (IEG). with the Bank Group’s twin goals and Focusing on the special themes under the SDGs. IDA16 and IDA17, it aims to offer evaluation evidence on what has and IEG Evaluation Evidence in IDA has not worked in IDA priority areas Special Thematic Areas in order to support the IDA18 Many recent IEG evaluations address replenishment discussions. issues that are relevant to IDA and its To the extent that evaluations are special themes. Although most of designed to be ex-post assessments of these evaluations transcend IDA16 vi OVERVIEW and IDA17 effectiveness periods and to inform policy and program design are not solely focused on IDA, they for youth. Management is taking offer important lessons for IDA going action to support clients in the forward. collection of labor market outcome data for youth; while the jobs INCLUSIVE GROWTH diagnostics include analyses for youth. IDA17 introduced the special theme of Financial inclusion and targeted inclusive growth to support the Bank support for small and medium Group’s twin goals. Among the wide enterprises (SMEs). IEG’s 2015 range of issues that directly and evaluation on financial inclusion finds indirectly contribute to the agenda, that the Bank Group has contributed IDA chose to focus on strengthening significantly to progress in financial the knowledge base and tools in the inclusion, helping to expand the reach areas of job creation, financial and sustainability of the microfinance inclusion, efficiency of public finance, industry. With the International and natural resource management in Finance Corporation (IFC) playing a IDA countries. Some important leading role, Bank Group support aspects of inclusive growth, such as targets countries with low inclusion gender, are dealt with separately. rates and reflects patterns of over- Recent IEG evaluations offer lessons in indebtedness. However, the links some of these areas. The overall results between access and usage, and are mixed with some moderate success between finance and poverty but mostly hard earned and valuable reduction, remain uncertain; and lessons. innovative delivery models are required to reach the remaining Youth employment. IEG’s 2013 youth unbanked population. employment evaluation found that Bank Group country strategies paid IEG’s 2014 evaluation on SMEs finds little attention to this issue even where few examples of successful targeted youth unemployment was serious. SME support programs. In particular, Although youth employment was the World Bank does not have an addressed in some sector strategies institutional definition of SME, and (e.g., education and social protection), this lack of clarity on what constitutes a comprehensive approach was an SME, and when and how to missing in Bank programs, and few intervene, have resulted in missed youth employment projects were opportunities. In response, implemented by multisectoral teams management is taking action to define despite the multisectoral nature of the and harmonize SME support issue. This calls for strong diagnostics approaches across the World Bank, IFC, and MIGA. vii OVERVIEW Agriculture and natural resource GENDER EQUALITY management for poverty reduction. In Over the past few years, the Bank practice, the Bank Group has often Group has strengthened its attention targeted agriculture and rural to gender issues at the corporate level. development for poverty reduction in Country strategies show a greater its country programs. IEG’s 2011 level of gender integration in IDA than evaluation of agriculture and in International Bank for agribusiness productivity noted a shift Reconstruction and Development in the World Bank’s approaches to countries, and most Bank operations agricultural development, from display at least some gender agricultural production to community- awareness. However, these have not driven development (CDD). This shift necessarily translated into more was particularly evident in Sub- coherent, outcome-focused gender Saharan Africa. Client commitments, integration. capacity, and resources often affected Bank Group effectiveness, as did Gender issues in FCS are even more internal constraints on staffing and acute than in other IDA countries. cross-institutional coordination. While the Bank has been relatively Management reports that the Bank effective in mainstreaming gender Group’s recent reorganization is within the health and education geared toward addressing many of the portfolios and in CDD, it has not issues. responded adequately to gender- based violence against women or their For countries endowed with natural economic empowerment. resource wealth, IEG’s 2015 evaluation of resource-rich developing countries To effectively address gender issues, identifies five areas where the Bank IEG’s report on Bank Group results Group needs to focus more intensively and performance suggests that gender to reduce poverty significantly. These must be an integral part of IDA include a consistent poverty focus country programs, as is envisaged in especially in the upward swing of the the new Gender Strategy. This will resource cycle; fiscally sustainable and entail detailed gender analysis at well-targeted social transfers; quality country and project levels to assess the education and skills development; likely impact of the proposed reaching the rural poor to avoid the interventions on women in the local development of a dual economy; and context. Attention is also required to attention to environmental integrate gender across key sectors, implications of extractive industries and to design a monitoring and through regulations and monitoring. evaluation (M&E) system that allows viii OVERVIEW for full reporting on gender-related several of them into IDA17 outcomes. commitments. Actual implementation is in early stages of planning and CLIMATE CHANGE design. For example, work has started IDA countries are especially to redo the criteria for FCS vulnerable to the adverse impacts of classification and interventions to climate change even though they are address gender-based violence are not the main drivers of climate being piloted with trust funds. The changes. IDA17 requirements to work on fragility assessments appears screen all IDA projects for climate to be well advanced, which could help risks represent a major achievement integrate fragility and conflict risks in and a good first step forward. the design of country assistance strategies and programs and help Recent IEG evaluations on climate monitor those risks. change and related issues, such as adaptation and disaster risk However, the need remains to develop management, emphasize the need to an inclusive growth and jobs strategy take a systemic view to build client based on a cross-sectoral approach capacity to manage climate change that takes advantage of potential risks and assess the impacts of synergies across the Bank Group. An interventions. Management has important lesson of the evaluation was generally accepted IEG that projects in FCS will need more recommendations and made some support because they are prepared progress in implementing them (e.g., more rapidly and have weaker client screening all IDA projects for climate capacity. While IDA allocation for FCS risks, developing approaches to increased recently, the average measure socioeconomic vulnerability administrative budget to support IDA and climate resilience), albeit at a slow operations in FCS declined and the pace in some instances. gap in unit costs with non-FCS IDA countries grew. The drop in self- FRAGILE AND CONFLICT-AFFECTED STATES assessment ratings of project performance in FCS could indicate a Since IDA identified support to FCS as declining confidence among task a strategic priority, there has been teams about implementation demonstrable improvement in Bank effectiveness, or a more realistic Group support and IDA effectiveness assessment. in FCS. IEG’s 2014 comprehensive FCS evaluation offers many lessons for The current trends of increased further improvement. Bank population displacement and the management agreed with IEG horizontal spread of fragility and recommendations and integrated conflict to middle-income countries ix OVERVIEW have also increased the relevance of restructured projects and additional the lessons from IEG’s FCS evaluation. finance. Finally, for the Bank Group as While some of these issues are beyond a whole and IDA in particular, the the range of IDA interventions, the need for adequate resource analytical and operational lessons mobilization to ensure adequate from IDA FCS countries may be budgets in times of rapidly expanding relevant for non-IDA fragile situations. programs was highlighted. CRISIS RESPONSE The food, fuel, and financial crises of 2008–10 underscored the urgency of IDA16 established a Crisis Response developing social safety net (SSN) Window (CRW) to facilitate IDA programs in all countries. IEG’s 2011 countries’ timely access to additional SSN evaluation found that the World resources in the face of severe Bank needed to engage consistently economic and food crises and natural during stable times to help countries disasters. IEG has reviewed two of the develop SSNs that address poverty 27 CRW-funded projects and found and protect the poor and vulnerable one to have satisfactorily achieved its from individual and systemic shocks. objectives and the other not, with both To this end, efforts needed to be facing high risk to the sustainability of directed to the two most common development outcomes. constraints for Bank support in IDA Between FY06-12, IEG conducted countries: weak institutions and several evaluations of Bank Group inadequate data. responses to natural disasters, the Managing for Results global financial crisis, and the global food crisis. These evaluations IDA has been a pioneer in results highlighted the importance of long- measurement. IEG’s evaluation of term country engagement, analytic Bank Group self-evaluation systems work in key areas, and institutional finds that its comprehensive self- capacity building in the pre-crisis or evaluation system adheres to relevant pre-disaster period. Past experience good practice standards, but the focus also called for more guidance in the on corporate results measurement and use of instruments and better quality reporting has sidelined other purposes assurance procedures for the Bank’s such as using the system for crisis response operations. In this performance management, strategic regard, Bank management has made decision-making, and learning. While progress in preparing guidelines to the existing systems mesh well with strengthen ex-ante quality assurance independent evaluation systems and for food crisis response programs and have been emulated by other in ensuring appropriate M&E in development agencies, they are not x OVERVIEW used to their full potential. To build a Strategic choice needs also to be stronger results culture in the Bank exercised at country level to take into Group, management needs to create account the nature and dynamics of more incentives for staff to engage the development challenges in the with the M&E systems with candor, local context and the key constraints. conduct thoughtful analysis of both In this regard, the SCD/CPFs for IDA successes and failures, and move from countries could play a key role in a compliance mindset to a learning assessing the trade-offs and making perspective. strategic decisions, while client ownership of the IDA agenda could be Despite the different focuses of the strengthened through a coherent, IDA special themes, relevant IEG dynamic, and well-sequenced strategy evaluation findings point to a core set by IDA recipients. of lessons that is applicable to whatever theme that IDA is pursuing. Finally, despite IDA’s strong effort to These include the need for (i) the strengthen results measurement for special theme to be part of the both accountability and learning underlying Bank Group knowledge purposes, the current results work; (ii) the agenda to be owned by monitoring systems remain under- client countries and supported by utilized. To improve the results successive governments; (iii) IDA’s culture in the Bank Group requires country strategies to specifically target clear signals from the top of the and fund actions to pursue them; and importance of adequate and timely (iv) results of the thematic targeting to results measurement, coupled with be monitored as part of the strategy consistent incentives and adequate implementation process. training and support to staff. In tandem with this, more attention In this context, a strategic choice has to needs to be paid to supporting the be exercised at the corporate level as to development of country capacity and which thematic areas are relatively country M&E systems. more important for IDA. The themes need to be assessed within the context of the changing development landscape, Bank Group strategic directions, IDA’s value proposition, and the need for enhanced learning, innovation, and results -- so that the selected themes are not simply “add- ons.” xi World Bank Group Management Comments Management welcomes the Learning Product of the Independent Evaluation Group (IEG), Learning from IDA Experience: Lessons from IEG Evaluations. Management appreciates the active and collaborative engagement with IEG during the preparation of the report, which provides important inputs for the IDA18 Replenishment discussion. In response to the final Learning Product, management wishes to make the following comments. Coverage and Scope This report presents a useful summary of relevant IEG evaluations in the recent past. While it is not a comprehensive review of IDA’s performance or effectiveness, and draws, for the most part, on projects and programs carried out before IDA16, many of the observations made still warrant continued attention. Management appreciates the report’s recognition of the follow-up actions taken and progress made in many areas. Because of the report’s coverage and time lag, some caution is needed in interpreting the findings and applying the lessons for the future. For example, recent lessons from climate change are based on a limited number of reports; many evaluations cover both IDA and International Bank for Reconstruction and Development (IBRD) countries; and there are no recent evaluations for some key sectors (e.g., water). In several cases conclusions seem to be drawn from a small number of projects—for example, the lessons on supporting small and medium-sized enterprises in fragile and conflict-affected states (FCS) come from one project, and those on the Crisis Response Window are based on two projects. IDA Thematic Areas The five thematic areas covered by the report—gender equality, climate change, FCS, crisis response, and inclusive growth—are fully aligned with the World Bank Group’s corporate goals and priorities, and have increasingly been at the forefront of its attention. This institutional commitment and effort are reflected in, among other things, the recent reorganization that elevated these cross-cutting areas, development of a Bank Group-wide gender strategy, and greater emphasis on FCS and crisis response in resource allocation decisions. xiii MANAGEMENT COMMENTS Management fully agrees with the need for “strategic choice” at both the corporate and country levels to determine the relative importance of these themes in each context. The new country engagement framework—whose the Systematic Country Diagnostic and Country Partnership Framework add increased rigor and a systematic approach to the Bank Group’s country-led business model—is focused on informing the client and the Bank Group in making such strategic decisions. Managing for Results The report notes that the World Bank Group has comprehensive self-evaluation systems that “adhere to relevant good practice standards” and are “as good as or better than those in comparable organizations,” but that the systems and incentives are focused more on accountability than on performance management or learning. While management agrees there is room for a more balanced approach, given the inherent tension between the accountability and learning objectives, it may not be feasible or realistic to expect one instrument to serve all objectives equally. For instance, accountability is and will remain a critical function of the Implementation Completion and Results Report (ICR). Learning, while it can be enhanced through the ICRs, could also be accomplished with other activities, such as impact evaluations and other voluntary reviews. Hence, in reforming the ICR approach, management will explore options to maximize learning while maintaining the ICR’s accountability function. Management agrees with the report’s conclusion that the Bank Group’s current results monitoring systems and incentives have room for improvement, and is already taking steps with a view to deepening the results culture and using the self- evaluation systems more strategically for knowledge and learning. Management has been undertaking a number of activities, and plans to further intensify its effort, to strengthen the focus of the institution on results in the following areas: (i) improved results measurement and reporting, with stronger links among project, country, and corporate levels (e.g., the ongoing Core Sector Indicator reform, the planned ICR reform, and updates of the Corporate Scorecards and IDA Results Measurement System indicators in line with the Sustainable Development Goals (SDGs)) (ii) strengthened capacity in results measurement and monitoring and evaluation (M&E) (e.g., various activities through the Results Measurement and Evidence Stream, enhanced training in results and M&E); and xiv MANAGEMENT COMMENTS (iii) improved guidance and knowledge management for results and M&E (e.g., strengthened guidance, good practice notes, and knowledge-sharing tools and network). Finally, as the report notes, the importance of country capacity and M&E systems cannot be understated. More effort is needed to enhance them, especially with a view to monitoring progress against the SDGs. xv 1. IDA and the Special Themes At the United Nations General Assembly in New York in September 2015, world leaders endorsed 17 Sustainable Development Goals (SDGs), an ambitious agenda to end poverty, promote prosperity, and protect the environment. The recent IEG report for the Addis Ababa conference on financing for development underscores the potential for a historic shift in the understanding of development (IEG 2015a). Developing countries are taking a stronger stance in the trajectory of their own development. The group of stakeholders is expanding to include a growing number of countries that have become donors bilaterally or through multilateral development banks. The private sector is increasingly involved as a key partner in providing solutions and finance. The voice of citizens and civil society has been more deeply integrated into the conversation. As the world’s largest provider of financial resources to the poorest countries, the International Development Association (IDA) was a cornerstone of the global effort to achieve the Millennium Development Goals (MDGs). It will continue to play a key role for attaining the SDGs by 2030. The 18th IDA Replenishment (IDA18) will be negotiated in the context of significant changes in the direction of international development and of an evolving World Bank Group. Meanwhile, change is in the air within IDA itself: its non-concessional borrowing policy is being reviewed and updated, and options for broader IDA reform are under consideration by the Board, with the Development Committee agreeing to take a medium- to long-term view of the future of the Bank Group and its role in the international financial architecture (World Bank Group 2015). Within this changed and changing context of development goals, emphasis, modus operandi, and resource implications, the Bank Group, and IDA in particular, will be under increasing pressure to deliver results. Numerous IEG evaluations address issues that are relevant to IDA’s development impact. Although most of these evaluations transcend the IDA16 and IDA17 effectiveness periods and are not targeted at IDA specifically, they contain relevant information on what has and has not worked when the IDA special themes were addressed. In line with the commitment to learning from the past, this synthesis report presents findings from recent IEG country program, thematic, and project evaluations, learning products, and relevant follow-up information from the Management Action Record (MAR), to help inform the discussions for IDA18. 1 CHAPTER 1 IDA AND THE SPECIAL THEMES Through a desk review without engaging in new research, this report brings together insights from recent IEG evaluations relevant for IDA and IDA countries. It is not an evaluation of IDA performance, its progress in relation to the replenishment commitments, or its effectiveness in addressing the special thematic issues. Instead, it aims to offer evaluation evidence, based on past Bank Group experience, on what has worked for delivering results in IDA priority areas. Recent Bank Group initiatives are noted, but not elaborated on because their effectiveness has yet to be evaluated. As the focus of the report is on IDA, interventions by International Finance Corporation (IFC) and the Multilateral Investment Guarantee Agency (MIGA) are not discussed independently, but only in the context of their interactions with IDA. IDA Commitments and Resource Mobilization Like other international financial institutions (IFIs), IDA replenishment commitments fall under four main categories: institutional effectiveness; financial management or financial instruments; operational reforms to strengthen policies, strategies, or portfolio performance; and results management, including monitoring and evaluation. Comparing the number and composition of these commitments, there is a convergence among the IFIs in the total number of commitments in the last replenishment cycle (i.e., around 50), and IDA stands out for being able to pay more attention to operational commitments than most comparators. This focus has increased even further in IDA17 (figure 1.1). The operational commitments at IDA and International Fund for Agricultural Development (IFAD) were related to implementation of the strategic priorities, while at the African Development Fund (AfDF) many of the operational commitments related to preparation of new policies and strategies. IDA has been effective in designing its commitment matrix in a manner that furthers its strategic and operational objectives. 2 CHAPTER 1 IDA AND THE SPECIAL THEMES Figure 1.1. Summary of Commitments among International Financial Institutions by Type 60 50 40 Number 30 20 10 0 ADF X ADF XI AfDF-12 AfDF-13 IDA 16 IDA 17 IFAD 8 IFAD 9 (2009-2012) (2013-2016) (2011-2013) (2014-2016) (2011-2014) (2014-2017) (2010-2012) (2013-2015) Institutional Financial Operational Results & M&E Note: ADF X and XI = ninth and tenth replenishments of the Asian Development Fund; AfDF 12 and 13 = African Development Fund 12th and 13th Replenishments; IFAD8/IFAD9 = Eighth and Ninth Replenishments of the International Fund for Agricultural Development; M&E = monitoring and evaluations. Calendar year is used for all institutions except IDA, which uses fiscal year. Source: ADB (2015); AfDB (2015). The replenishment volume for IDA17 increased by 5.5 percent from IDA16 (table 1.1). While total donor contributions decreased marginally by 1.7 percent, given the size of IDA resources, they are still more than twice as large as the other three IFIs together. Moreover, donor countries contributed 50 percent of the IDA17 replenishment, which is a much larger share than donor contributions to the Asian Development Fund (ADF) and IFAD. Among the IFIs, only AfDF 13 received a larger share from donor countries. The larger share of ADF’s resources is now coming from reflows. This has encouraged the Asian Development Bank to rethink how it mobilizes and leverages its ADF resources, resulting in a merger of the balance sheets of ADF with its ordinary capital resources in order to increase the volume of concessional funding it will be able to provide to its low-income recipient member countries on a sustained basis. 3 CHAPTER 1 IDA AND THE SPECIAL THEMES Table 1.1. Resources Mobilized during Most Recent Replenishment ADF XI AfDF 13 IDA17 IFAD9 Replenishment, billions SDR 7.9 UA 4.9 SDR 34.6 US$3.53 Replenishment, US$ billions 12.4 7.3 52.1 3.53 Change from previous replenishment 11.1% -16.9% 5.5% 17.7% Change in donor contributions 11.5% 1.3% -1.7% 27.3% Donor share of replenishment 36.7% 79.8% 50.0% 39.7% Note: ADF XI = tenth replenishment of the Asian Development Fund; AfDF 13 = African Development Fund 13th Replenishment; IDA17 = 17th Replenishment of the International Development Association; IFAD9 = Ninth Replenishment of the International Fund for Agricultural Development; SDR = International Monetary Fund Special Drawing Rights; US$ = U.S. dollar; UA is the official currency for AfDB projects (1 UA = 1 SDR). Sources: ADB (2015); AfDB (2015). IDA Special Themes IDA first introduced special themes in 2005 with IDA14 to “reinforce the effectiveness of IDA assistance in ensuring lasting reduction in poverty and making progress toward the MDGs” (World Bank 2005). The recommendations of the IDA14 report centered on private sector led growth, debt sustainability and grants, results measurement, and working together at country level. During the IDA15 discussions, it was realized that despite some progress, many MDGs were likely not to be met in South Asia and Sub-Saharan Africa. In this context, IDA15 focused on the agency’s role in the global aid architecture to achieve results at the country level (IDA 2008). Within the special themes of global aid architecture, country-level effectiveness, and fragile states, IDA defined its support for climate change actions, debt management, gender outcomes, results measurement, and harmonization and alignment. The IDA16 discussions took place in the aftermath of the global food, fuel, and economic crises. As IDA countries faced new and growing challenges, delivering development results became an overarching theme for IDA16, with crisis response, gender, climate change, and fragile and conflict-affected states (FCS) included as special themes (IDA 2011). IDA’s Results Measurement System (RMS) would be expanded to track IDA’s operational and organizational effectiveness. IDA17 maintained the strong focus on results of the previous IDA replenishment periods as well as the special themes of gender equality, climate change, and FCS (IDA 2014). More attention was paid to IDA’s results delivering and monitoring mechanism, and inclusive growth was added as a fourth special theme “given its centrality for achieving the World Bank Group goals.” Although the various special themes evolved in response to the pressing issues over time, there has been high continuity in IDA’s overarching goal of achieving 4 CHAPTER 1 IDA AND THE SPECIAL THEMES development impact through a strong focus on results measurement and partnership as well as special attention to gender equality, climate change, FCS, and broadly shared economic growth. These areas are consistent with the Bank Group’s twin goals of ending extreme poverty and boosting shared prosperity, and with the SDGs. As such, they will continue to be important for IDA going forward. Bank Group Support in IDA Special Thematic Areas The Bank Group has incorporated the IDA special themes into its country strategies to a varying extent (table 1.2). For example, nearly 90 percent of the 64 Bank Group country strategies1 approved for IDA countries in FY12–15 have some discussion of gender issues in the country context, and half of them provide detailed analysis either in a dedicated section of the main strategy document or in an annex. Analysis of specific gender challenges in the country is less common. Almost all of the strategies reflect gender on a cross-cutting basis in stated objectives and contain indicators to measure progress. Table 1.2. Presence of IDA Special Themes in IDA Country Strategies Approved during FY12– 15 Analysis Objectives Results Matrix Country context Specific challenges (percent) (percent) (percent) (percent) Gender 87.5 18.8 98.4 98.4 Climate change 45.3 39.1 81.3 51.6 Inclusive growth 57.8 43.8 85.9 46.9 Fragility and conflict FCS countries 80.8 26.9 53.8 3.8 Non-FCS countries 39.5 10.5 13.2 0.0 Note: Table shows the percentage of country strategies addressing each theme in different sections of the strategy document. Specific challenges = analysis of a country’s particular gender issues. FCS = fragile and conflicted-affected states. Source: IEG staff calculation. For climate change and inclusive growth themes, although over 80 percent of the strategies define objectives or outcomes in these areas, they are not always guided by an analysis of the issues and the specific challenges in the country, nor followed up with measurable indicators to track Bank Group performance. This lack of follow-up is a recurring finding from IEG country and thematic evaluations that is discussed in Chapter 2. Box 1.1 offers an example of how the special theme of climate change is addressed systematically in an IDA country program. 5 CHAPTER 1 IDA AND THE SPECIAL THEMES Box 1.1. Treatment of Climate Change in Mozambique Country Partnership Strategy FY12–15 The Mozambique FY12–15 Country Partnership Strategy (CPS) takes as its starting point the government’s Poverty Reduction Strategy, the overarching theme of which is inclusive and broad-based growth. It has three pillars: (i) competitiveness and employment; (ii) vulnerability and resilience; and (iii) governance and public sector capacity. An analysis of climate change and natural disaster is provided under the Country Context and Development Agenda section of the CPS that draws on earlier economic and sector work (World Bank 2010). The analysis notes that Mozambique ranks third among African countries most exposed to risks from extreme weather events in the form of floods, cyclones, and droughts. In regional projections, climate change is expected to increase the frequency and magnitude of weather events. Under Pillar II of the CPS, climate change mitigation and adaptation activities are introduced as an important new business line. Broader environmental concerns are also evident across the other pillars: agriculture, water, and energy under Pillar I and promoting of greater contribution of wildlife conservation to the economy under Pillar III. IDA support will be rooted in a programmatic Development Financing Policy (DPF) series to support policy reforms for mainstreaming climate change throughout the policy- making process and a coordinated approach to planning. The DPFs are intended to complement the portfolio of investment projects co-financed by the Pilot Program for Climate Resilience, a newly established Climate Investment Fund. By FY15, operational follow through includes nine projects totaling $292 million that have a particular, explicit emphasis on climate change. IFC will implement private sector components under Building Africa’s Resilience to Climate Change, a pre-pipeline advisory services program that will channel $10 million in funding to catalyze private sector investments to enhance climate resilience across all economic sectors with a focus on agriculture and forestry. The IFC Mozambique Forestry program will provide advisory services to develop sustainable private forestry investments and to take advantage of potential carbon offsets. The results matrix defines two outcome indicators under Pillar II: (i) adaptation to climate change and reduced the risk of natural disasters; and (ii) accurate weather information system available to stakeholders in 70 percent of Central and Southern regions on at least a 12-hour basis (from 0 percent in 2010 to 70 percent in 2015). It also defines four related milestones for measuring project outputs. Source: World Bank (2012). With regard to FCS, what is most surprising is that about 20 percent of Bank Group strategies in FCS countries (FY12-15) reviewed for this exercise, did not have any kind of analysis of fragility, violence, or conflict to frame the ensuing strategic objectives and support programs. Although more than half of the strategies have specific conflict-related objectives or actions, the expected outcomes are not identified for close monitoring in the results matrix. Out of 26 country strategies for 6 CHAPTER 1 IDA AND THE SPECIAL THEMES FCS countries, only the Democratic Republic of Congo has a conflict-related indicator. Portfolio Performance in IDA Countries The 2014 and 2015 reports on Results and Performance of the World Bank Group (IEG 2015b and IEG 2016) note an improvement in the performance of Bank Group country programs in IDA for both FCS and non-FCS countries. As measured by outcome achievement, the performance of IDA lending projects in IDA is catching up with those supported by the International Bank for Reconstruction and Development (IBRD) (see figure 1.2) (IEG 2016). Nevertheless, IDA country program performance remains below that of IBRD (by over 25 percentage points) and portfolio performance in IDA FCS is on a downward trend after an earlier improvement. Figure 1.2. World Bank Group Performance by Country Group a. Bank Group country programs b. Bank lending projects 100 100 80 80 Percent Percent 60 60 40 40 20 20 0 0 IBRD IDA Non-FCS IDA FCS IBRD IDA Non-FCS IDA FCS FY08-FY11 FY12-FY15 FY09-FY11 FY12-FY14 Note: The Bank’s lending project success rate is calculated by number of projects. Source: IEG database. Projects in IDA countries are different from IBRD projects in many respects. They are typically smaller, less likely to involve development policy lending, less likely to be programmatic, more likely to operate in a volatile environment such as a crisis or disaster, and more likely to be implemented where there is less local capacity (table 1.3). Empirical analysis carried out for RAP 2013 looked at entry risks and outcome achievement in Bank projects (IEG 2014). It found that less risky projects tended to have better outcomes, but 46 percent of the sampled investment projects had substantial or high entry risks. This was driven by IDA countries, especially FCS, where entry risks averaged 69 percent (versus 43 percent for IBRD-only countries). In choosing to fund more projects in IDA than in IBRD countries, the Bank may be less risk averse than often perceived. This finding suggests it is important that the 7 CHAPTER 1 IDA AND THE SPECIAL THEMES Bank help countries in difficult environments convert high risks into high rewards, and that quality management in IDA countries be enhanced to improve project outcome achievement. Table 1.3. Evaluated Bank Projects, FY09–14 Average Project Commitment Value Number of Share of DPF Country Group Size ($ billion) Projects (percent) (US$, million) IBRD 99.44 562 176.94 51.20 IDA 48.09 824 58.37 19.92 Non-FCS 38.36 627 61.18 17.68 FCS 9.74 197 49.43 28.73 Note: DPF = Development Policy Financing; FCS = fragile and conflicted-affected states; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association. Source: World Bank Business Warehouse. RAP 2014 undertook an in-depth analysis of the continuous decline in quality at entry (a key correlate of overall performance) and quality of supervision (IEG 2015b). It finds that well-designed projects tend to draw on past lessons at entry, have effective risk mitigation measures, and adopt appropriate objectives and Results Frameworks. During project implementation, the propensity to solve problems, facilitated by regular missions, determines the supervision quality of Bank projects. These findings suggest that the success of Bank projects depends critically on factors within Bank management control. RAP 2014 also echoes the findings of earlier reports in the series regarding the perennially poor ratings for monitoring and evaluation: 30 percent and 29 percent rated substantial for FY08–10 and FY11–13, respectively.2 Focusing on the typically better performance of larger operations than smaller projects, RAP 2015 found that mid-course corrections matter more than project size (IEG 2016). That is, project performance has more to do with what happens during the course of project implementation—such as addition or cancellation of funds— than with the initial size of the project. Therefore, although project size and development outcome ratings are correlated, better performance associated with large projects may reflect the practice of directing additional resources to projects that are performing well during implementation (and discontinuing those that are not).3 This is consistent with the findings of a recent IEG report on additional financing (AF) (IEG 2015a), which finds that projects with AF had relatively better overall outcome ratings compared with the rest of the portfolio.4 Lower project outcome ratings were associated with turnover of the task team leader, regardless of 8 CHAPTER 1 IDA AND THE SPECIAL THEMES whether the project was ever designated a “problem project,” and regardless of longer preparation time. References AfDB (African Development Bank). 2015. Independent Evaluation of General Capital Increase VI and African Development Fund 12 and 13 Commitments: Overarching Review, Summary Report. Abidjan, Côte d’Ivoire: African Development Bank Group. http://idev.afdb.org/sites/default/files/documents/Independent%20Evaluation%20of%20 General%20Capital%20Increase%20VI%20and%20African%20Development%20Fund%2012% 20and%2013%20Commitment%20-%20Overarching%20Review.pdf. ADB (Asian Development Bank). 2015. Asian Development Fund X and XI Operations: Opportunity Amid Growing Challenges. Corporate Evaluation Study. Manila, Philippines: ADB. http://www.adb.org/sites/default/files/evaluation-document/174392/files/ces-adf.pdf. IEG (Independent Evaluation Group). 2014. Results and Performance of the World Bank Group 2013. Washington, DC: World Bank. ______. 2015a. Financing for Development: Using Independent Evaluation to Turn Aspirations into Achievements. Washington, DC: World Bank. ______.2015b. Results and Performance of the World Bank Group 2014. Washington, DC: World Bank. ______. 2016. Results and Performance of the World Bank Group 2015. Washington, DC: World Bank. IDA (International Development Association). 2008. Report from the Executive Directors of the International Development Association to the Board of Governors, Additions to IDA Resources: Fifteenth Replenishment, IDA: the Platform for Achieving Results at the Country Level. Washington, DC: World Bank. ______. 2011. Report from the Executive Directors of the International Development Association to the Board of Governors, Additions to IDA Resources: Sixteenth Replenishment, IDA16: Delivering Development Results. Washington, DC: World Bank. ______. 2014. Report from the Executive Directors of the International Development Association to the Board of Governors, Additions to IDA Resources: Seventeenth Replenishment, IDA17: Maximizing Development Impact. Washington, DC: World Bank. World Bank. 2005. Additions to IDA Resources: Fourteenth Replenishment (IDA14) —Working Together to Achieve the Millennium Development Goals. Washington, DC: World Bank. ______. 2010. Economics of Adaptation to Climate Change: Mozambique. Washington, DC: World Bank. ______. 2012. Mozambique—Country Partnership Strategy for the Period FY2012–FY2015. Washington, DC: World Bank. World Bank Group. 2015. 2015 Shareholding Review: Report to Governors September 2015. Washington, DC: World Bank. 1 Including interim strategy notes. 2Monitoring and evaluation for IDA projects as a whole was rated 25 percent for FY08–10 and 29 percent for FY11–13. 9 CHAPTER 1 IDA AND THE SPECIAL THEMES 3The analysis also found that, when controlling for country, sector, and project characteristics, higher outcome ratings were associated with projects in countries with higher Country Policy and Institutional Assessment ratings (a measure of institutional capacity in the country) or larger populations. 4However, the provision of additional resources is no guarantee of success since 13 percent of the projects receiving additional resources were rated negatively for overall outcomes at project closure. 10 2. Evaluation Findings in IDA Thematic Areas Many IEG evaluations offer lessons that are relevant to one or more IDA special themes. While all the country program evaluations (CPEs) and the Project Performance Assessment Reports (PPARs) selected for this synthesis report target IDA countries,1 IEG’s thematic evaluations do not always distinguish between IDA and IBRD specific issues. Nevertheless, it is important to note that strategies and operations in IDA and FCS countries are often subject to the same factors that impact the success of those in IBRD countries. More generic issues are at play across the portfolio around how the Bank Group does business that need to be taken into account as well as factors that fall more sharply into focus in the context of IDA and the special themes. Moreover, because IDA countries, especially FCS, often have weaker institutions than their IBRD counterparts, the issues tend to be accentuated. To the extent that evaluations are designed to be ex-poste assessment of completed programs and projects, findings from recent IEG evaluations are based on Bank Group experience prior to IDA 16 and 17 for the most part. Some of the challenges identified in these evaluations may have been, or are being, addressed by the Bank Group in one form or another. This synthesis report notes many of the new initiatives, but overall, is not in a position to comment on their effectiveness. As development is a long-term endeavor, the lessons distilled from IEG evaluations remain valid and deserve continuous attention. Finally, although this synthesis report is based on a comprehensive review of all recent IEG evaluations relevant to IDA, it is not a comprehensive review of the IDA special themes. The evidence gaps point to where Bank Group and IEG attention is need in the coming years. Inclusive Growth IDA17 introduced the special theme of inclusive growth in support of the Bank Group’s twin goals. Focusing on building the knowledge base and tools for understanding country specific constraints to inclusive growth, IDA committed to devote special attention to job creation, financial inclusion, public financial management, and natural resource management as a way to promote inclusive growth. Since then, efforts have concentrated on making the theme central to country strategies and filling knowledge gaps. Some important aspects of inclusive growth, such as gender, are dealt with separately. Over the last decade, IEG evaluations have paid increasing attention to the issues of inclusive growth and poverty reduction. They offer lessons on three types of 11 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS interventions that aim to (i) enhance youth employment; (ii) promote financial inclusion and provide targeted support for small and medium enterprises (SMEs); and (iii) develop agriculture and natural resource sectors. These topics are taken up in several IEG thematic evaluations and all of the CPEs to some extent. The overall results are mixed with some moderate success but mostly hard earned and valuable lessons. This conclusion is not unexpected given the challenges presented by many IDA countries that have achieved limited progress toward poverty reduction despite solid growth. IEG’s evaluation of the poverty focus of country programs calls for greater attention to the mix between indirect poverty interventions (inclusive growth) and direct poverty interventions (basic services and social safety nets) (IEG 2015a). It also highlights the need to comprehensively address poverty data development and reporting in the Systematic Country Diagnostic (SCD) and country policy dialogue. To this end, the evaluation suggests that the Bank should advocate and support sustainable improvement of the capability of national statistical agencies, take a stronger lead in strengthening mechanisms for quality and transparency of poverty data, motivate country compliance, and regularly disseminate data. The Bank should also ensure that its poverty diagnostic work incorporates relevant social and political dimensions of poverty analysis and offer actionable priorities for policy interventions. Finally, explicit evaluation protocols for piloted interventions are needed to capture lessons for future scaling-up and to ensure evaluability at project inception. ENHANCE YOUTH EMPLOYMENT IEG’s evaluation of youth employment programs found that “although youth employment is addressed in the education, social protection, and labor strategies, it is not recognized as an issue in most country strategies, even where youth unemployment is serious” (IEG 2013). This finding is confirmed by the limited attention paid to the issue in Bank Group country programs. Among the eleven CPEs reviewed, only in Afghanistan, Bolivia, Liberia, and Timor-Leste did youth employment receive any attention, and with the exception of Liberia, none reported promising results (box 2.1). A comprehensive approach was missing in the Bank’s youth employment programs—most projects included isolated interventions, and collaboration across sector teams is limited. 12 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS Box 2.1. Youth Employment Interventions in Afghanistan, Bolivia, Liberia, and Timor-Leste In Liberia, the Bank Group supported basic education under the Education for All Initiative and assisted vocational and technical education through the Youth Employment and Skills Project. The Human Development, Sustainable Development, and Private Sector Development teams collaborated on several studies to inform the government’s youth empowerment and employment strategy. This was followed up with a technical assistance project in FY13 to assist the government in designing concrete interventions in these areas. These activities helped to expand employment to absorb the unemployed or under-employed youth, who include a large number of ex-combatants, which in turn helped Liberia manage social tensions, conflict, and political instability. In Bolivia, the Bank had a clearly articulate strategy to increase the percentage of low- income youth in urban and peri-urban areas with regular employment and improved labor incomes, but the interventions did not lead to expected results for several reasons. The Bank carried out a series of activities that included a poverty and vulnerability analysis, Social Safety programs to assess the roles of subnational agencies, a mapping of existing programs, and technical assistance to national and local governments to help implement the Social Protection strategy, and a Youth Employment Program for Urban Areas. However, as the government continued to change and interest in supporting local governments waned, the Bank’s potential for impact declined as well. Although these analytic and advisory activities and technical assistance efforts evolved into financial support for the Investing in Children and Youth Project, the impact was not realized. Such concerted efforts were less evident in other IDA countries. In Afghanistan, the Skills Development Project was the main intervention for addressing employment issues. While the project was analytically sound and responded to concerns of youth unemployment, reduced employment prospects for Afghan citizens, and re-enrollment of children in school, it suffered from the challenges of coordinating across 11 ministries and implementation problems from staff turnover and shortcomings in government leadership. Similarly in Timor-Leste, although Bank Group goals included temporary employment creation, particularly in rural areas, and engagement of alienated youth in productive community-based activities. Except for planned economic and sector work on youth, the country assistance strategy included no projects to address specific concerns of youth or flaws in education. Instead, youth issues were to be mainstreamed in all of the Bank Group‘s other activities planned over the period, which meant, in effect, little was done. Source: IEG (2011b, 2012b; 2013a; 2015a). At the project level, the evaluation found the Bank’s lending portfolio for youth employment to be relatively small and include mainly skills development and school-to-work transition projects.2 Some operations included interventions to foster job creation and work opportunities for youth. Few youth employment projects were implemented by multisectoral teams even though youth employment is a multisectoral issue. The limited experience in this area means evidence on what 13 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS works in youth employment is scarce. The Bank’s few impact evaluations on youth employment examined short-term effects, found limited positive results, and did not calculate the cost-effectiveness of interventions. This calls for strong diagnostics to inform policy and program design for youth. More recent analytical work, such as that by Christopher Blattman and Laura Ralston to evaluate a number of programs on youth unemployment in fragile states,3 provides a base to bridge the knowledge gap. The evaluation identified several factors that contribute to success, including a comprehensive approach with participation of the private sector, complementary interventions (e.g., combined training with job search and placement assistance, rather than isolated interventions), and monitoring and follow-up of individual participants. Programs that combine smoothing the transition from school to work with work-based skills development were found to be most effective for youth employment and earnings, although the applicability of this model to developing countries with a large informal sector and rural areas is limited. In high- unemployment environments, wage subsidies, skills training, and job search support are of little impact; and demand-side interventions are needed. For example, in rural low-income areas, where most youth are active in agriculture and nonfarm employment or self-employment in the informal sector, stimulating the market for the growth of farms, nonfarm household enterprises, and rural agribusinesses is essential. The Bank and IFC could further support the expansion of rural and urban jobs in these sectors. The paucity of data limits the ability to assess the impact of Bank Group support to youth employment. Few projects included tracer studies to track the subsequent employment history of youth or provided information on youth as a beneficiary group. Hence there was a critical need to support government to collect labor market outcome data for youth in relevant surveys, monitor the employment situation by age groups, and use the relevant statistics to inform country strategies. PROMOTE FINANCIAL INCLUSION AND TARGETED SUPPORT FOR SMES These two themes are intricately related. A significant portion of Bank Group interventions to support SMEs has focused on improving their access to finance, and this is usually done in tandem with financing through microcredits or providing microfinance. IEG’s evaluation on financial inclusion finds that the Bank Group “contributed significantly to progress in financial inclusion globally and in client countries” (IEG 2015b). In particular, the Bank Group reached a substantial share of the microfinance industry and contributed to the sustainability of microfinance services. Moreover, 14 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS Bank Group resources devoted to advancing financial inclusion were not only well aligned with countries’ needs, targeting countries with low inclusion rates, but also reflected patterns of over-indebtedness, providing advisory and knowledge work rather than with funding to markets at risk of over-indebtedness. However, the link between access and usage is not automatic, and that between finance and poverty reduction is neither certain nor well understood. To ensure that improved financial access translate into financial inclusion and poverty alleviation, the Bank Group needs to clarify its approach on financial inclusion by making it more evidence based and comprehensive, focusing on financial services with benefits for the poor and specifying when and how to use subsidies. Critical to its work is how the Bank Group systematically finds and replicates innovative delivery models that lower transaction costs and improve financial inclusion. An appropriate, holistic, and systematic diagnostic tool4 is important, as are country- level strategies for financial inclusion to guide its work. As it experiments with new ways to achieve the envisaged universal access goal, it is of particular importance to have in place a well-established M&E system that takes account of Results Frameworks established in country financial inclusion strategies and follows a practical and cost-effective approach to improving measures of beneficiary impact. Finally, this very recent evaluation finds that the Bank Group needs to strengthen partnerships by advocating clear strategies, Results Frameworks, and M&E arrangements. Nine of the eleven CPEs reviewed discuss financing or other types of support for SMEs. Some of the objectives pursued in this regard were limited to studies to understand the problem; in other cases specific lending programs were provided. The results were mixed with over half of the CPEs reporting some degree of unsatisfactory performance (box 2.2). IFC has played a leading role in these areas. Box 2.2. Microfinance in Afghanistan, Bangladesh, Bolivia, and Liberia In Afghanistan, the Bank Group’s interventions in the microfinance sector completely transformed the sector. Its early activities supported the microfinance industry and improved knowledge of the financial sector. The Bank Group took a two-pronged approach: the Bank provided lending and non-lending technical assistance to transform the local microfinance institutions and help them achieve operating self-sufficiency; IFC invested in a commercial bank that specializes in micro lending which provided an additional mechanism for mobilizing funds and delivering microfinance services. The Bank also invested in the Microfinance Investment Support Facility for Afghanistan to create an apex organization that finances and monitors microfinance institutions in Afghanistan, while IFC also provided advisory services for developing the sector by helping establish the First Microfinance Bank of Afghanistan (FMBA) with an initial 15 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS investment of $1 million. The FMBA is now the leading microfinance bank in the country and highly regarded in the banking sector through its continued and improving financial performance. Such a two-pronged approach was also adopted in Bangladesh, where efforts by the Bank to extend the country’s “unique and pioneering microcredit system to the poorest" was supported by two projects, including the Learning and Innovation Loan—Financial Services for the Poorest. The Bank’s effort was hailed as innovative and very successful, and efforts to scale up the model were underway. Meanwhile, IFC worked with commercial banks to increase their lending to SMEs, although direct attribution to IFC activities is difficult. Bolivia has a vibrant and mature microfinance market, but a relatively small number of SMEs and rural producers have access to it. The Bank and IFC used several tools to help expand financial access to that market. The Bank conducted two studies to explore ways to provide collateral and a financial sector review to provide input to the government’s reforms. IFC explored various forms of intervention to expand SME and rural finance. However, most of these efforts had no follow-up because the government was not interested, the clients did not need IFC resources, or insurmountable regulatory obstacles existed. In Liberia, the Bank Group focused on microfinance to support financial sector development. An IFC Advisory Service project conducted a pre-feasibility study on microfinance and recommended revisions to the banking law and related regulations. This led to IFC investing in a new microfinance bank, the Access Bank Liberia, and providing technical assistance that enabled the bank to develop good practices in microfinance. IFC's work addressed the lack of access among a large segment of private businesses, and the process introduced innovations in the microfinance market by using a private commercial bank as the funding and delivery mechanism. Source: IEG (2009b, 2012b; 2013a, 2015a). Overall, these evaluations highlight the importance of having in place a managerial, accounting, and financial capacity in the countries where the Bank Group wants to support SMEs and microfinance. They point to the fallacy of overestimating demand for credit given the high transaction costs of borrowing through Bank Group supported financial intermediaries. Having a coherent strategy and complementary interventions by different Bank Group institutions enhances the delivery of results. On the other hand, poor understanding of the political and market conditions threatens the Bank Group’s efforts to promote financial inclusion. With regard to targeted support for SMEs, IEG’s evaluation finds that how SMEs are defined establishes their relevance to the Bank Group’s twin goals, but the literature offers little guidance on the actual efficacy of the most common forms of targeted SME support, either for direct beneficiaries or, more broadly, for markets and economies, much less the appropriate sequencing and complementarities of 16 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS interventions (IEG 2014a). For SMEs to be a meaningful category of enterprises, they should be differentiated from others by the way they experience particular policy, institutional, or market failures, or the way they benefit the economy and the poor. Targeted support for SMEs needs to be firmly rooted in a clear, evidence-based understanding of how the proposed support will sustainably remove the problems that constrain SMEs’ ability to contribute to employment, growth, and economic opportunity. As revealed by the evaluation, the World Bank lacked an institutional definition of SMEs,5 leading to project-specific definitions that lacked solid links to the underlying rationale for assistance. The commitment level of the World Bank SME lending portfolio was low in IDA countries compared to upper-middle-income countries, which raises questions of its relevance for reaching the frontier and building markets and market institutions where they are weakest. In addition, the lack of institutional consensus on a range of questions (e.g., what constitutes an SME? when it is appropriate to support them? what forms it will take? how it relates to systemic reform? what constitutes success? and how it will be monitored and evaluated?) have resulted in missed opportunities. The CPEs reviewed provide very few examples of successful targeted support for SMEs (box 2.3). The Bank Group is taking actions to address some of these issues. For example, IFC and the Jobs Group are developing tools for including jobs data in the administrative data collected by banks. Box 2.3. Targeted Support for SMEs in Bolivia, Mongolia, Mozambique, and Zambia In Bolivia, there was no well-articulated Bank Group strategy for private sector development in general, partly because of a strenuous relationship between the Bank Group and the government. Although IFC rightly focused on SMEs and explored various ways to support local businesses and income generation opportunities for indigenous communities, most of the interventions never went beyond the pilot stage with little broader impact. In Mongolia, Bank support increased the amounts of term lending available to SMEs, but the mismatch between the SMEs’ need for long-term assets and their access to only short- term loans was not solved. IFC’s investments contributed to improving access to finance, especially for micro, small, and medium enterprises. This included a trade financing facility in the cooperating banks, although the facilities were reportedly underused due to scant demand since IFC’s funds were more expensive to use. In addition, Bank lending and analytic work appeared to have been operating independently of each other rather than developing synergies, yielding poor overall results. 17 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS In Mozambique, the Bank program successfully provided some business extension services to SMEs and some technical assistance to strengthen a couple of government agencies, but its line of credit faltered and had to be altered to allow larger firms to borrow. IFC’s direct support to SMEs was unsuccessful in most cases because of the small firms’ lack of management, accounting and financial capacity and know-how. As a result, only a limited number of SMEs were supported, mainly through Advisory Services, thus failing to achieve the goal of developing capacity among a large group of local SMEs. In Zambia, Bank Group efforts to promote SMEs were largely unsuccessful. They continued to focus on lines of credit that were not in demand because they did not address the underlying issues of creditworthiness and collateral. High yield on government bonds and imposition of interest rate caps also restrained lending to SMEs. IFC extended support for local SMEs to improve their operations to meet the procurement requirements of multinational companies, building on its successful pilot in Konkola Copper Mines. Training was provided to 170 SMEs, but only one out of five participating multinational companies replaced imports of goods with local products supplied from SMEs. Source: IEG (2011a, 2015a,c, d). There is one relevant PPAR that assesses the performance of the Enterprise Development Project in Mozambique (IEG 2014a). It finds that in post conflict countries with weak institutions and serious capacity constraints, a programmatic approach rather than a single project with multiple components could have provided opportunities to learn from each phase as inputs to designing the next. Moreover, matching grant programs for business services can potentially be an effective way of providing incentives to firms, although such programs have to be well managed to be successful. On the other hand, credit line operations that are designed solely to increase the supply of funds do not increase sustainable access to finance if other institutional, legal, and regulatory constraints to financial sector development are not addressed. DEVELOP AGRICULTURE AND NATURAL RESOURCE SECTORS FOR POVERTY REDUCTION IEG’s agriculture and agribusiness evaluation notes that, following a strategic shift in the 1990s in its approach to agricultural development, many Bank-supported rural projects adopted community-driven development (CDD) models in which agricultural production was one of many priorities (IEG 2011b). This trend was particularly pronounced in Sub-Saharan Africa (SSA) where the share of rural projects focusing explicitly on improving agricultural growth and productivity was the lowest. IFC had relatively little engagement in agribusiness in SSA because of difficult business environments, a shortage of indigenous entrepreneurs, small size of the potential investments, lack of access to markets, and the discouraging 18 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS experience of working directly with small-scale sponsors. Inconsistent client commitment and weak capacity limited the effectiveness of Bank Group support in agriculture-based economies, while constraints on staffing (e.g., decline in agriculture-related skills and analytic work, especially in some of the poorer IDA countries) and limited internal coordination within the Bank Group were also found to have hurt outcomes. Financial sustainability was further constrained by insufficient government funding and the difficulty of maintaining agricultural services and infrastructure. To meet the worldwide demand for food and reduce poverty, particularly in the poorest developing countries, the evaluation suggests greater effort is needed to connect sectoral interventions (e.g., among transport, finance, and agriculture) and achieve synergies from public and private sector interventions; to build capacity and knowledge exchange; to take stock of experience in rain-fed agriculture; to ensure attention to financial sustainability and cross-cutting issues of gender, environmental, and social impacts and climate; and to better integrate Bank Group support at the global and regional levels with that at the country level. Management agreed with IEG recommendations, noting that several recommended actions had been initiated after the IEG evaluation period. The MAR indicates that the recent Bank Group reorganization has the potential to improve collaboration on agribusiness across relevant departments and institutions as well as the movement of technical staff to ensure better matching of technical specialists with demand. IFC reports increasing investment in SSA through a more coordinated approach with Financial Markets, Infrastructure, and Bank activities. In addition, expenditure on analytical work in agriculture-based countries has increased, an Action Plan was prepared in FY15 to improve quality of knowledge activities, and progress in South– South knowledge exchange continues. On the other hand, it is not clear how the newly instituted M&E review mechanism differs from the review mechanisms in place prior to the reorganization for improving the M&E framework in agriculture interventions. Inclusive growth is one of three interrelated focus areas for IEG’s evaluation of resource-rich developing countries (RRDCs) (IEG 2015e). Drawing from four CPEs (Bolivia, Kazakhstan, Mongolia, and Zambia), the evaluation finds that the implications of resource revenues for poverty reduction and social development are not qualitatively different in RRDCs from those in other developing countries but often manifest themselves with particular intensity. In this context, the evaluation suggests Bank Group programs should pay special attention to: (i) a consistent poverty focus, especially in the upward swing of the resource cycle when RRDC governments can become complacent about whether the benefits of growth are 19 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS reaching the poor; (ii) fiscally sustainable and well-targeted social transfers because RRDC governments are often under pressure to support cash transfers that are poorly designed and do not meet their objectives; (iii) improved quality in education and skills development to ensure that the investment in human development will have a significant impact on productivity, competitiveness, and poverty reduction; (iv) engagement with the rural poor to avoid the development of a dual economy where a relatively prosperous urban economy coexists with a rural sector that is stagnant, and where poverty is concentrated; and (v) a special focus on the environmental implications of extractive industries through effective regulations and monitoring. Addressing a key constraint to growth and poverty reduction, IEG’s evaluation on electricity access finds that given the scale of the challenge and associated targets to which the Bank Group is committed (through the Sustainable Electricity for All initiative and the SDGs), it should move from a predominantly project-by-project approach—which lacks the scale and speed to achieve universal access by 2030 in low-access countries—to a far greater use of a sector-wide organizing framework and process for mainstreaming the sustained engagement needed for implementing rapid access scale-up (World Bank 2015). The new strategic framework and country plans, and the Bank Group’s operational engagement going forward should be informed by the core principles and strategic drivers underlying best practice programs, which are systematic implementation of national electricity access, enabling sector policies and regulation, commercial viability of service providers, affordability of connections costs for the poor, and overarching government commitment and leadership. In practice, the Bank Group has often targeted agriculture and rural development for poverty reduction in its country programs (box 2.4). These examples show that support for rural activities and agriculture and provision of affordable credit are the most direct way to enable improvements in the income of the poor. To ensure that positive outcomes are associated with poverty reduction programs, country strategies should address governance and, in some cases, budget leakages and corruption. Finally, all poverty reduction strategies and programs must specifically include monitoring of poverty reduction impacts. Box 2.4. Poverty Reduction Interventions in Bank Group Country Programs in Afghanistan, Bolivia, Cambodia, Liberia, and Uganda In Afghanistan, the primary drivers of the rural economy are the massive donor expenditures, which provide significant short-term employment for many rural people, and the illicit opium industry that provides substantial cash income to small farmers. 20 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS While rural poverty reduction suggests a positive effect on livelihoods, results are uneven, reflecting varying support levels across provinces. In Bolivia, the Rural Alliances Project was the anchor of World Bank assistance for agriculture and rural development. In designing the program, the Bank listened to government officials and drew from its experience with what farmers needed to succeed. By targeting poor farmers with productive potential instead of the poorest farmers without such potential, the Bank was able to separate social assistance programs from income-enhancing programs, thereby showing how effective the latter can be in reducing poverty and improving living standards in a sustained manner, even in traditional agriculture. In Cambodia, agriculture is the most important sector for sustaining economic growth and alleviating poverty, but budgetary expenditure on agriculture and rural development was low. Hence additional budgetary support by the Bank could have enhanced resources for these sectors, while sustained macroeconomic stability and economic growth are prerequisites for reducing poverty. In Liberia, the Bank’s interventions in the forest sector did not adequately take into account the intended beneficiaries’ needs and capacity, including their ability to make a deal and monitor the actions of logging companies. As a result, little gains accrued to them. A new strategic framework for growth is needed and its pro-poor focus could be enhanced by integrating the role of indigenous communities and civil society in the design of interventions. In Uganda, IDA’s economic and sector work helped the government put poverty reduction and social development at the center of the country strategy. By documenting service performance and expenditure leakages, IDA helped the government, civil society, and its own team recognize the importance of service delivery and governance for poverty reduction. Besides two programs with explicit poverty reduction objectives, an agricultural sector adjustment credit proved to be a potent poverty eradication instrument because the associated coffee and cotton marketing reforms raised the incomes of poor farmers. Nevertheless, much higher or more broadly based growth is required for a more rapid decline in poverty. Source: IEG (2009a; 2010a; 2012b; 2013a; 2015a), Gender Equality Over the past few years, the Bank Group has made steady progress in moving the gender agenda forward. Concrete actions include the creation of the Gender Labs, the production of an increasing number of impact evaluations, the preparation and implementation of Regional Gender Action Plans, and the development of a new World Bank Group Gender Strategy. These efforts have brought greater focus on gender issues at various levels; however, the results of that increased attention are not yet evident. 21 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS RAP 2015 provides a detailed analysis of the system currently in place for promoting gender integration into Bank Group strategies and operations (IEG 2016). It finds that the Bank has implemented several recommendations from the 2010 IEG gender evaluation (IEG 2010a),6 including the institutionalization of gender reporting systems (e.g., a gender flag at project design); intensified efforts to produce gender- disaggregated data and impact evaluation evidence; and progress in strengthening results and accountability mechanisms (e.g., gender tracked in IDA Results Frameworks and Corporate Scorecard). Nevertheless, the increased attention to gender issues at corporate level has not necessarily translated into more coherent, outcomes-focused gender integration in practice, and poor measurement persists. GENDER IN IDA COUNTRY STRATEGIES IDA country strategies have a greater level of gender integration than those for IBRD countries. Among the country strategies completed in FY12–14, 40 percent reflected gender in an objective or a pillar; of the 559 strategic objectives pursued in these country strategies, five directly focused on gender (rather than on a cross- cutting basis) and all five instances are associated with an IDA country. Recent IEG CPEs for IDA countries confirm a high level of attention to gender issues in Bank Group country strategies: nine out of eleven IDA CPEs note that gender equality was addressed in some way. Such attention in IDA country strategies coincides with, or reflects, the fact that almost all these strategies were developed in the context of the MDGs, which called for improvements in this area (e.g., increasing schooling of girls or electing women to parliament). Also, since 2000, the Bank's Country Policy and Institutional Assessment (CPIA) began to capture gender equality as a factor for allocating resources across IDA countries. Nevertheless, the Bank has found it difficult to mainstream gender interventions in general. In some cases, Bank interventions achieved positive results. In others, the efforts produced mixed outcomes, or the recognition of gender issues in the country strategies was not followed up with adequate support programs (box 2.5). Box 2.5. Gender Interventions in Nepal, Bolivia, and Zambia In Nepal, IDA support focused more on economic than social inclusion, but contributed substantially to bringing social inclusion issues to the center of the development strategy. It accomplished this by raising awareness and understanding of the issues, and by supporting critical measures such as equal access to education, civil service diversity and Poverty Alleviation Fund targeting based on gender, caste, and ethnicity. Progress on access and equity was considerable, and the efforts were especially effective in education as well as for a large number of health outcomes and health care services utilization. 22 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS Nevertheless, very little progress was made in education quality, and despite improvement, huge inequalities remain in education provision. In Bolivia, Bank Group strategies paid increasing attention to gender issues over time, and a number of Bank credits incorporated gender in their project design and results measurement. The attention to gender was manifested most explicitly in the interventions on maternal and child health, but the expected outcomes were not achieved as the health projects failed. Most of the Bank’s interventions in agriculture and rural development had a gender dimension and produced good gender results as the projects succeeded overall. Although IFC executed several advisory services projects that targeted women entrepreneurs, most IFC projects did not explicitly incorporate gender dimensions, even when working with indigenous and farming communities as well as small and medium enterprises, by including gender-disaggregated data for project beneficiaries, or tracking project impact on women. In Zambia, successive country strategies have recognized the magnitude of gender inequalities in the country, noting the specific challenges faced by women in health, education, labor force participation, agriculture, and access to land. However, it is not clear how this recognition is translated into operations. In the agricultural sector, for example, women do most of the farm work, but do not have control over or access to resources; yet the emphasis on gender disparities has not been translated into an increased focus on women in agriculture. A gender review in 2012 recommended that project design make better use of analytical work to incorporate country and sector level gender issues, especially in infrastructure and water sector projects. It also pointed to education and gender-based violence as two areas where gender relevant work was absent. More recent sector interventions are better at integrating gender aspects, but the lack of a gender-targeted approach makes it challenging to address and track gender based outcomes. Source: IEG (2011g, 2015a, d). RAP 2015 finds that even when country strategies address gender issues, they do not clearly define the goal of gender integration or the rationale for the proposed approach (IEG 2016). In particular, when gender is treated as a “cross-cutting” issue, as is most often the case, few strategies adequately explain what that designation means operationally. In some cases, treating gender as cross-cutting becomes a means to avoid articulating a clear rationale and elaborating a results chain for the (undefined) gender goals. Consequently, the gender indicators in most country strategies are intermediate in nature and derived from existing operations. They are insufficient for monitoring gender outcomes. To effectively address gender issues, gender must be an integral part of IDA country programs, as is currently envisaged in the SCD/CPF process and the new Gender Strategy. To this end, it will be necessary to have a detailed gender analysis at country and project levels that allows for an assessment of the differentiated impact 23 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS of the proposed strategies and actions on men and women as well as boys and girls. This may require prior technical and analytical work to ensure the availability of reliable data, the inclusion of relevant voices in design, and buy-in for proposals. The findings from the analysis should inform IDA’s country strategies and project design, with ongoing consultation throughout the life of the strategy and project to ensure that it is delivering as intended. Attention is needed to integrate gender across key sectors and address marginalized communities, not simply where women have traditionally worked. GENDER IN IDA OPERATIONS At the operational level, RAP 2015 finds a frequent inclusion of at least some discussion of gender-related issues in project documents, but the rationale for their presence is often not apparent (IEG 2016). Typically, project documents forego any analysis of the relationship between the gender elements included and project objectives. In fact, the current “gender flag” system may have inadvertently supported the implicit notion that gender should be relevant for all Bank Group operations. The lack of explicit discussion of gender relevance to main project objectives may have led to important gender dimensions being overlooked. Several IEG thematic evaluations provide examples of such missed opportunities (box 2.6). Box 2.6. Gender in IEG Thematic Evaluations The systematic review of impact evaluations of social safety nets (SSNs) and gender finds that interventions almost inevitably produce gender-differentiated impacts whether those impacts were expected or not (IEG 2014d). However, the Bank’s SSN projects rarely capture these impacts by expecting the unintended consequences and incorporating the impact of gender differences at the design stage. Most projects include limited discussion of intra-household dynamics and the gender-relevant context of the supported intervention. Gender is often missing from monitoring and evaluation frameworks, except for tracking female beneficiaries, and this is observed even in projects that include an extensive discussion of gender and have incorporated that discussion in project design. There is also evidence that SSN interventions can contribute to empowering women, if only economically, although more evidence is needed to determine whether empowerment should be an explicit goal of Bank projects. The early childhood development (ECD) evaluation finds that synergies have not been established between the Bank’s work in gender and ECD despite the evidence of a nexus between women’s economic empowerment, girls’ education, and the development of children through quality childcare. An internal structure does not yet exist to coordinate sectoral interventions and capitalize on opportunities. When cross-sector coordination occurred, it was based on staff initiative rather than organizational practices. The lack of childcare is often a constraint on female participation in the labor force, but the Bank has 24 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS not been prominent in supporting childcare (World Bank 2015; O’Sullivan and others 2014).7 The investment climate evaluation by IEG (2015j) finds that explicit targeting of women entrepreneurs is not common, even in projects that more directly impact small entrepreneurs and act on constraints more likely to impact women (e.g., reforms dealing with registering property, land administration, permits, tax regulations, agriculture, licensing, access to land, and property rights). Many of the projects that target women entrepreneurs are small in size and mostly focused on capacity-building, which treat women as participants of training or consultative working groups rather than as entrepreneurs (or potential entrepreneurs). Sources: IEG (2014d; 2015g; 2015j); World Bank (2015). On the other hand, some IEG project level evaluations offer insight on what works to promote gender equality in Bank operations. They point to the need to inform project design with analysis of the likely impact on women’s welfare, women’s role in society, their special needs, and the local context. Box 2.7 presents some positive examples of integrating gender consideration into project design and paying attention to gender impact monitoring. Box 2.7. Examples of Informing Project Design with Analysis of Potential Gender Impact The Rural Electricity and Renewable Energy Development Project in Bangladesh had a gender-informed design by including analysis of the likely impact on women’s security, income generation opportunities, and knowledge via access to radio and television (IEG 2014b). It had indicators for measuring gender outcomes, such as the number of hours that girls study at night, access to news by women, improved reproductive health, and increased human immunodeficiency virus (HIV) and acquired immune deficiency syndrome (AIDS) information and awareness. An associated technical assistance project helped build the client’s administrative and project implementation capacity by developing a monitoring and evaluation framework and a methodology for evaluating socioeconomic and gender related impacts on electrification projects. The Fadama II Project in Nigeria was grounded in a deep understanding of context and of what motivates people in order to change behavior (IEG 2014c). The project design was informed by relevant analytical and technical work, including a gender assessment. The project’s social guidelines encouraged the formation of marginalized groups (e.g., youth, widows, physically challenged, people with HIV or AIDS) as a way to ensure that their voices were fully heard and that their interests were reflected in local development planning. Five years after project close, female participation in local economic development planning increased, but greater attention to women’s specific needs is required as there is evidence to suggest that women (and other vulnerable groups) were often not able to afford or obtain the assets they needed, or that the equipment they acquired was being operated by hired hands who benefited more from the assets than the women themselves. To address this, the project’s mid-term review recommended 25 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS lowering the beneficiary contribution for women and vulnerable groups, but the recommendation was not taken up for the next phase of the project. The National Social Action Project (2003–10) and the Food Crisis Response Development Policy Grant (2008-09) in Sierra Leone also point to the need to pay special attention to women’s particular needs in order not to compromise the effectiveness of a cash-for-work program targeted at them (IEG 2013b). In this instance, women’s participation could have been improved through inclusion of more projects other than roads (where physical strength is a primary attribute for the job), or activities more suitable for women (fetching water, gravel, cooking, etc.), and arrangements for childcare. Sources: IEG (2013b; 2014b,c). There is a need to design an M&E system that allows for full reporting on gender- related outcomes. RAP 2015 finds that most projects with gender indicators were in education and health and the indicators selected tend to measure access or coverage rather than quality (IEG 2016). Very few indicators measure gender dimensions of employment and entrepreneurship, or of agricultural and rural development, while indicators of voice and agency are essentially absent. The drawbacks in the Bank Group’s M&E framework, especially with respect to gender monitoring, are highlighted by IEG’s evaluation of electricity access (box 2.8). Specific gender targets are needed, although this does not necessarily require adding multiple indicators or gender disaggregating all existing ones, but rather identifying the key indicators that can be monitored to assess the distributional impacts of the intervention. 26 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS Box 2.8. Gender Monitoring in Bank Group Support to Electricity Access The evaluation finds that attention to welfare and gender-related outcomes of electricity access interventions appears to be increasing in Bank projects and with satisfactory impacts. However, most projects limit themselves to tracking the “number of female beneficiaries (percent),” and there is a long way to go for welfare and gender impacts to be mainstreamed in IDA and IBRD electricity projects. To improve key performance indicators for gender, the evaluation calls for: (i) a clear definition of beneficiaries versus users, since they may be different groups; (ii) tracking of outputs and outcomes, not only headcount figures; and (iii) measures of outcomes. Source: IEG (2015f). GENDER IN FCS CONTEXT Gender issues in FCS are often even more acute than in other IDA countries. Women are more vulnerable to gender-based violence and often face greater economic burden than in more stable societies. IEG’s FCS evaluation (IEG 2014c) finds that the Bank has been relatively effective in mainstreaming gender in FCS within the health and education portfolios and in CDD projects, but has not responded adequately, or in a timely manner, to conflict-related violence against women (box 2.9). Consequently, a significant increase in school enrollment of girls was evident in many FCS countries, and gender-related health indicators also showed significant improvement. However, most assistance strategies did not address gender-based violence or constraints affecting the economic empowerment of women. Most of the demobilization, disarmament, and reintegration programs were not gender sensitive and provided disproportionate benefits to ex-combatants rather than the victims of violence. Gender analysis has often been delayed. Box 2.9. Gender Interventions in Afghanistan, Liberia, and Nepal In Afghanistan, there was a concerted effort in the IDA strategy to reverse the gender exclusionary policies of the Taliban. Because ownership of gender issues was an integral part of the country program, they were addressed systematically through support for the country's National Solidarity Program objectives on gender equality and a Transitional Support Strategy. The Bank adopted programs that involved community groups in making gender-related decisions. Although women’s voices are yet heard, they have a seat at the table. Although access to microfinance has not yet empowered women economically, it enhanced their social status. Substantial results have been achieved in terms of service delivery (e.g., women’s access to health services and girls’ access to schooling) and increased economic and social opportunities for women. 27 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS However, the relevance of the Bank’s analytical work on gender was compromised by the fact that the Country Gender Assessment and other gender-related analytic and advisory activities were prepared entirely by Bank staff and consultants. Inputs from the government were limited, which means weak country ownership. Moreover, the assessment did not provide clear guidance, which reduced its usefulness both for the Bank and the country. Only after funds were obtained from the Bank’s Gender Action Plan was an effort made to operationalize the assessment’s recommendations. In Liberia, the Bank’s strategic objectives included gender equality in a broad sense, but the design of its assistance focused narrowly on women’s economic empowerment. The strategies identified gender-based violence as a serious problem affecting women and girls, but provided no assistance. Without alleviating the threat of physical assaults, the objective of women’s economic empowerment was compromised. Gender sensitive design appeared in more recent Bank projects. Nepal was a notable exception where a grant from the State and Peacebuilding Fund was used to finance a Peace Support Project with a gender-sensitive approach providing cash benefits to widows of conflict-affected families who are particularly burdened and disproportionately affected as a consequence of war. A thorough gender analysis also enabled Nepal to address gender inequality in livelihoods programs, local government programs, and parliamentary participation. Source: World Bank (2011g, 2012b, 2013a). The evaluation recommends that in FCS countries, programs addressing gender issues be more responsive to the conflict context. This would involve timely gender analysis to assess the effects of conflict and violence, and implementation of measures to address conflict-related violence against women and legal constraints against women’s engagement in economic activities. In particular, the Bank should:  Frame country strategies with assessment of the nature, dynamics, and impact of conflict on women to help prevent and mitigate adverse effects;  Increase women’s involvement in post-conflict efforts, such as peace processes, to improve program design and implementation;  Collect gender-disaggregated data for monitoring program results;  Target female-headed households for income generation, employment, and human development programs after the conflict ends; and  Prioritize services for medical and psychosocial conditions and legal, education, and livelihood needs to overcome the effects of violence. Climate Change IDA countries are especially vulnerable to the adverse impacts of climate change. IDA17 requirements to screen the entire IDA portfolio for climate risks represent a 28 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS major achievement and a good first step toward helping countries better assess and prioritize their risk mitigation needs. Other efforts, such as developing an approach to measure socioeconomic vulnerability at the national level, multisectoral investment plans and projects, and a streamlined set of comparable indicators to measure climate resilience are also positive moves. In recent years, IEG undertook three evaluations of the Bank Group’s support for climate change and related issues such as water, natural disasters, and environmental sustainability. Together, they provide a way forward for engaging effectively at the country level to pursue energy policy reforms, low-carbon development, and climate change adaptation. The evaluations emphasize the need to take a systemic view for assessing interventions’ impacts and alternatives. Focusing on greenhouse gas emissions from the energy sector, the first IEG evaluation (IEG 2009b) offered four recommendations for the Bank: (i) systematically promote the removal of energy subsidies; (ii) emphasize policies that induce improvement in energy efficiency; (iii) provide incentives to address climate change issues through cross- sectoral approaches; and (iv) invest more to improve metrics and monitoring for motivation and learning at the global, country, and project levels. Bank management accepted and substantially implemented these recommendations. To promote low-carbon development, IEG’s second evaluation (IEG 2010b) concluded that the Bank Group needed to leverage its resources to support four interlinked actions: (i) place greater emphasis on large-scale, potentially catalytic, and energy efficiency scale-up and improve coordination between the Bank and IFC; (ii) foster technology transfer and identify innovations that have potentially high returns; (iii) enhance the delivery of its guarantee products by improving policies, procedures, incentives, flexibility, and skills for the deployment of the products; and (iv) measure its projects’ economic and environmental impact during execution and after closure for feedback and learning (IEG 2010b). Much progress has been achieved with regard to the Bank’s guarantee instruments, but progress in the other areas is partial. Renewable energy is receiving far more emphasis than energy efficiency, which has potentially higher returns. For instance, in contrast to the Clean Technology Fund for IBRD countries, which finances both renewables and energy efficiency, the Scaling-up Renewable Energy Program does not support energy efficiency measures in IDA countries. This is a missed opportunity. To focus the Bank Group and its development partners on climate adaptation results, IEG’s third evaluation (IEG 2012b) suggested to develop reference guidelines for incorporating climate risk management into project and program design, 29 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS appraisal, and implementation; test territorial and national-level measures of adaptation-related outcomes and impacts for inclusion in an improved Results Framework; and pilot approaches to better assess the costs, benefits, sustainability, and impact of activities with presumed resilience benefits. It also called for support to improve client countries climate monitoring services and information sharing within and between countries. IDA has initiated support for improving hydromet systems in several IDA countries (e.g., Malawi and Ethiopia). There has been continued progress on mainstreaming climate risk screening, although it is too soon to tell whether this screening is leading to improved project design. The development of a national level resilience metric represents a shift toward an outcome-oriented approach. However, the efforts so far appear to focus more on adaptation to current climatic conditions and less on preparing vulnerable countries for the climate change they are likely to face in the future. This is especially important for low lying coastal areas that will be increasingly at risk from rising sea levels. At country program level, few of the CPEs reviewed pay a significant attention to the subject— climate change issues were touched on in only four of the eleven countries (box 2.10). This lack of climate change consideration in IDA country strategies may be partially explained by the fact that the evaluated country programs cover the period FY87–13, before climate change was identified as a priority area for IDA assistance. In these country strategies, climate change and disaster risks are a small part of the Bank Group’s overall country program. All the Country Partnership Frameworks (CPFs) prepared/under preparation, on the other hand, have incorporated climate and disaster risk in their programs. Box 2.10. Managing Climate Change Impacts in Bangladesh, Bolivia, Mongolia, and Zambia In Bangladesh, consecutive IDA country strategies included general objectives to improve disaster management capacities and called for an increased emphasis on mitigating the climate change effects in donor programs to protect the interests of the poor, especially in view of the expected impact of climate change on dry-season water availability. However, the Bank was remiss in not following up more proactively by matching its assistance to strategy objectives. Although climate change was a threat to which the country is highly vulnerable, it was not adequately addressed in Bank Group programs. Bolivia’s topography makes it highly vulnerable to climate change. There has been increased frequency and intensity of floods and droughts, such as those associated with El Niño and La Niña as well as glacier retreat. Bank Group strategies targeted climate change and disaster risk management. The assistance, however, was small. It contributed to building institutions and organizations, including laws required to protect the environment and to deal with natural hazards, but it has had little impact on reducing the overall environmental pressures. Continued forest losses have reduced biodiversity and 30 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS the potential for carbon sequestration and now constitute the largest source of greenhouse gas emissions in Bolivia. In Mongolia, IDA strategy identified a wide range of internal and external risks that could jeopardize development outcomes, including factors such as climate change and weather variability. Climate change has increased the frequency of extreme winters (dzuds) that typically cause significant losses of livestock herds in some areas. To further improve pasture management and reduce herder’s vulnerability to climate shocks, the Bank is supporting the Emergency Disaster Management Program, which builds capacity at the National Agency for Meteorology, Hydrology, and Environmental Monitoring to institutionalize and fully deploy a livestock early warning system. The program also pilots an emergency feed and fodder production system to help reduce the impact of dzuds on rural livelihoods. This system has provided accurate forecasting of weather conditions and has become a major source of information for the herder community. Zambia is very sensitive to recurrent floods and droughts, which are likely to increase in scale and frequency with climate change. The latest CPS incorporates climate variability risks as one of the vulnerability factors for poverty reduction. However, building resilience to climate change, which had been promoted under an Emergency Recovery Project, was subject to a lengthy eight-year gap when no progress was made. Meanwhile, forest degradation and deforestation was not addressed and continued unabated. Under the Climate Investment Funds (CIF), the Pilot Program for Climate Resilience began a two-phase program to address Zambia’s climate vulnerability in such areas as agriculture, energy, food security, forestry, health, water, and wildlife. The self-evaluation by the CIF suggests that there was progress in mainstreaming climate risks and opportunities within a number of sector strategies. An important institutional set-up that came out of this activity is the Secretariat for Climate Change under the Ministry of Finance. The overall effect, however, is not clear due to a lack of information. Source: IEG (2009b; 2015a, c, d). At project level, an IEG evaluation on agriculture and agribusiness concluded that Bank projects had the potential to improve the readiness of countries to deal with the effects of climate change on agriculture (IEG 2011b). In particular, focused analytical work could be important in helping clients identify the direct links between agricultural production and climate change, a rising priority across countries. IEG’s project level evaluations have identified several good practice examples for addressing environmental issues. For example, global initiatives like the Biodiversity and Agricultural Commodities Program and the Extractive Industries Transparency Initiatives responded to the need for major economic sectors to address negative externalities associated with their activities by supporting the development and implementation of voluntary global standards. The disaster risk management projects in the Caribbean and Guatemala addressed the risk of adverse 31 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS natural events such as earthquakes, hurricanes, and excess rainfall by providing immediate funds to allow governments to respond to a natural disaster. The Weather Index Insurance projects in Kenya and Rwanda aimed to address the risk of smallholder farmers sliding back into extreme poverty as a result of crop damage caused by adverse weather conditions. By reducing risk to farmers, the projects aimed to increase farm investment and income, ultimately improving food security and reducing poverty. Taking into consideration the complexity of climate change issues, the main lesson to emerge from the CPEs and the thematic evaluations is that Bank Group strategies must be implemented through concrete projects that are monitored and followed up in order to produce positive results. Especially when capacity building is required, too small a level of assistance generates weak thrust. Lessons from IEG project level evaluations are relatively limited and refer to addressing environmental issues in general, but with resonance in the climate change context. These include: (i) when tackling climate change issues that, by their nature, require long term engagement, it is necessary to plan programmatically over time and to limit processing requirements; (ii) programmatic approaches may need complementary technical assistance and other support to address fundamental constraints; and (iii) donor harmonization has trade-offs—while it can help to provide a unified platform for sector reform, it has the risk of reducing the flexibility of programs and posing significant challenges without prior investments to build country capacity for donor coordination. In the context of limited IDA resources, the Crisis Response Window (CRW) provides a supplementary channel of funding when the financing needs exceed the IDA allocations for the country, as in the case of the $80 million assistance provided to Vanuatu to help recover from cyclone Pam. The CRW, as designed, is expected to be used to rebuild infrastructure to an enhanced resilience standard. Relief efforts after a shock often present a policy opportunity to strengthen institutional capacity. A small investment, particularly in small states, whose IDA allocations are very small, could help build country capacity and better prepare them for risk management of future shocks, or for donor coordination, harmonization, and co- financing of concessional climate finance. Fragile and Conflict-Affected States IEG’s comprehensive evaluation of low-income FCS highlighted significant progress achieved in some areas and identifies gaps and lessons for future actions (IEG 2014c).8 It found demonstrable improvement in Bank Group support and IDA 32 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS effectiveness in FCS during 2007–12, when IDA identified support to FCS as a strategic priority, and as a result of these efforts, portfolio performance in IDA FCS had improved since 2001, with the gap between FCS and non-FCS IDA countries being closed. More recently, fragility assessments are being undertaken more frequently, enhancing IDA’s ability to address fragility and conflict drivers within the Country Partnership Frameworks in FCS. The procurement policy has been revised, and the revised framework is being implemented. Stronger partnerships are reported with nine Joint Implementation Plans prepared in FCS by Bank Group institutions. An exceptional allocation for “turnaround” regimes has been introduced, although no countries have received funding from the exceptional allocation midway through IDA17. Bank Group engagement in FCS is a long-term agenda with many challenges and constraints yet to be overcome. On organizational effectiveness, there have been efficiency gains in the interval from concept note to disbursement, which is 16 months in FCS compared to 21 months in FY15 in IDA (IDA 2015). Operationally, there has been a noticeable drop in the self-evaluation score for satisfactory implementation in FCS, which reflects the extent of confidence among task teams about implementation effectiveness. IEG’s evaluation had commended the World Bank for recognizing that operations in FCS need more support, and for allocating higher administrative budgets for project preparation and implementation support in FCS. This positive trend was not sustained: average project implementation support cost to IDA FCS operations declined in FY14, and the gap with non-FCS IDA countries grew further in FY15.9 One key lesson from IEG’s evaluation was that the classification system to identify FCS countries was inadequate since it was based on CPIA indicators of past policy and institutional performance.10 While governance is highlighted in these indicators, conflict and violence risks are not. The threshold defining FCS status was determined in the early 2000s when conflict was concentrated among poor countries with very weak institutions. Consequently, countries like Sri Lanka, which suffered several decades of conflict but had higher CPIA ratings, was never classified as FCS. The Bank Group’s strategy can be seriously affected by whether or not the existence of fragility and conflict risks are recognized (box 2.11). Box 2.11. FCS Classification and Bank Group Country Strategies for Nepal After Nepal’s conflict intensified in 2001, the Bank did not move to classify Nepal as an FCS. Instead, the Bank’s focus was predominantly on long-term development and poverty reduction, and Nepal’s Country Policy and Institutional Assessment remained above 3.0. As a result, the Bank’s then nascent toolkit for engaging in FCS was not applied and most 33 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS of the Bank’s strategy objectives were not achieved. Nepal was classified as FCS only after the peace agreement was signed and United Nations peacekeeping was initiated in 2006. Subsequently, the Bank became more responsive to conflict risks and even financed a project to help conflict-affected communities. Source: IEG (2011g). A second lesson was that in the absence of systematic analysis of fragility, conflict, and political economy risks, country assistance strategies lacked tailoring to fragility and conflict drivers. Although the Bank was responsive to FCS with reconstruction programs in the early years after conflict, strategies often lacked realism in the medium-term and did not have contingencies based on political economy and conflict risks to adjust objectives and results if risks materialize. Assistance to Timor- Leste is one such example (box 2.12). In many FCS, fragility risks were treated as a secondary issue instead of a cornerstone for developing a longer-term strategy to exit fragility. Box 2.12. Fragility Risks and Bank Group Country Strategies for Timor-Leste As an FCS, Timor-Leste has received one of the highest levels of financial support on a per capita basis. Yet, poverty increased during the period of the largest aid inflow. Learning from the experience in other FCS countries, the Bank's CAS correctly underscored the importance of aligning interventions with the political realities and the institutional capacity constraints in the country. The principles of engagement focused on results on the ground, country ownership of the development program, effective partnership with other donors, and the efficacy of good communication. Over time, however, the Bank's program lost focus and overemphasized long-term institutional objectives, placing very little emphasis on short-term interventions that would yield immediate benefits to the population. This led to a diminution of program relevance, a loss of country ownership, and safeguard violations, all of which affected strategy outcomes. Source: IEG (2011b). A third lesson came from the Bank Group’s success at the operational level. Although Bank Group projects in FCS were more resource intensive, enhanced financial and staff resources and greater managerial attention led to rapid improvement in FCS portfolio performance and closing of the gap with nonfragile IDA countries. Deployment of senior staff at country level,11 made possible by human resources reforms in 2007, allowed relatively rapid scaling up of programs in a low capacity environment. Other key determinants of better operational quality included increased budget allocations to meet the higher cost of doing business in 34 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS FCS, introduction of simplified procedures and implementation mechanisms to address capacity constraints, and greater reliance on partnerships and support for national programs in FCS. In Afghanistan and Liberia, for example, the Bank adjusted for capacity constraints by introducing an outsourcing model to deliver health services in partnership with civil society organizations. A related lesson was that Bank Group support for state-building needs to be sustained through careful sequencing, prioritization of long-term reforms, and a mix of predictable, programmatic budget support, investment projects, and technical assistance. This message was reinforced by the PPAR for the Programmatic Support for Institution Building in Afghanistan, which referred to the need to tailor efforts to local context in institution building activities and highlighted the importance of empowering senior government officials (via analytical work and parallel technical assistance) to determine and direct administrative reforms, public financial management, and other institutional processes (IEG 2013c). The Afghanistan Reconstruction Trust Fund was used effectively to finance investment projects complemented by predictable budget support in the form of an incentives program to encourage policy and institutional reforms. This judicious mix of instruments and financial modalities significantly increased the effectiveness and impact of IDA investments. The PPAR for Investing in Social Capital in Sri Lanka, although not classified as an FCS, highlights the importance of continuity of engagement and the programmatic approach that involve multiple stages of lending require iterative analysis of national priorities, particularly in areas or countries affected by fragility or conflict, for achieving results in FCS (IEG 2015h). A fifth lesson emerged from the use of CDD programs, which are useful for short- term assistance to local communities, but in the absence of a mechanism to ensure sustainability their long-term viability is at risk. As shown in the examples in box 2.13, what is appropriate for conflict-risk management needs to evolve into more sustainable institutional systems when the short-term crisis is over. Box 2.13. CDD Programs in Afghanistan and Nepal In Afghanistan, the National Solidarity Program has been an effective delivery mechanism to mitigate conflict by allocating funds for small-scale schemes identified by local communities. However, the Community Development Councils were not linked to local government organizations, and there were no incentives for their sustainability beyond the implementation of the schemes. An impact evaluation showed high interest at mid-point but interest in the Councils waned once the schemes were completed. In Nepal, the Poverty Alleviation Fund (PAF) was effective in providing grants for short- term schemes and revolving credit grants to poor communities in the conflict-affected 35 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS districts. However, it had no linkages and synergies with the microfinance industry. Representatives of microfinance institutions expressed concern that a large portion of PAF loans were not effectively targeted and were distorting the microfinance sector by damaging the discipline and sustainability of established Savings and Credit Groups. Source: IEG (2011g, 2013a). A sixth lesson was that inclusive growth and job creation were difficult to achieve in FCS. The Bank provided targeted support for short-term jobs through CDD and public works programs. Support for long-term jobs focused on investment climate reforms, which are necessary but not sufficient for private sector development, given infrastructure and capacity constraints. To be effective, an inclusive growth and jobs strategy would need to be based on a cross-sectoral approach consisting of prioritized interventions that take advantage of potential synergies across the Bank Group The final lesson was that resources matter. Other than FCS countries that are well endowed with natural resources, most FCS tend to be dependent on donor financing. Due to the constraints of performance-based allocation criteria for IDA, the global shift in aid flows toward fragile states was not matched by IDA during IEG’s evaluation period, and despite the large volumes of exceptional post-conflict allocations to Afghanistan, Liberia, Sudan, and Timor-Leste, FCS received less aid per capita from IDA than other low-income countries (LICs). This was in contrast to the increased share of overall official development assistance to FCS, largely because of increases in bilateral assistance to FCS. In this context, multi-donor trust funds could be a source of additional finance, but also play a central role in donor coordination, policy dialogue, and institution building. Recent changes in the IDA resource allocation framework has reportedly led to an increase in FCS allocation (IDA 2015). To enhance the relevance and effectiveness of its assistance to FCS, the evaluation recommended that the Bank Group adjust its strategy, approach, and product mix. Specifically, the Bank Group should (i) develop a more suitable and accurate mechanism to determine FCS by integrating indicators of political economy and conflict risks within the current system; (ii) tailor country assistance strategies better to FCS and monitor fragility risks to make strategic adjustments, when necessary; (iii) increase support to reform-oriented FCS for state capacity building at national and subnational levels through predicable budget support, TA, and investment lending; (iv) develop and implement a plan for institutional sustainability of CDD programs; (v) ensure that gender-oriented programs in conflict-affected countries are more responsive to the conflict context and address gender-based violence and 36 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS legal constraints on economic empowerment of women; and (vi) develop a more realistic medium- to long-term framework for inclusive growth and jobs in FCS and ensure synergies across the Bank Group. This would also imply adaptation of IFC and MIGA business models, risk tolerances, and product mix to be responsive to the special needs of the private sector in FCS. Bank management agreed with and committed to implement these recommendations, and integrated several of them into IDA17 commitments. Actual implementation is in early stages of planning and design, which has not yet been integrated within operations and therefore not been evaluated. This appears to be the case for the analytical tool being prepared to develop an inclusive growth and jobs strategy, the planning being done to implement measures for institutional sustainability of CDD programs, the trust-funded pilots to address gender-based violence, and the work initiated to revise the criteria for FCS classification. In comparison, the work on fragility assessments appears to be more advanced. This could serve as a useful instrument to integrate fragility and conflict risks in the design of country assistance strategies and programs and help monitor those risks. Two recent trends—increased population displacement and the horizontal spread of fragility and conflict to middle-income countries (MICs)—have increased the relevance of the lessons from IEG’s FCS evaluation. While some of the issues are beyond the range of IDA interventions, the fragility challenges they face suggest that analytical and operational lessons learned from IDA-eligible FCS countries may well be relevant for these non-IDA fragile situations. What is evident is that these trends have exposed the underlying fragility in several apparently stable and economically thriving states and made the need for fragility and political economy analysis critical, not just for countries classified as FCS but for neighboring countries and even for MICs at risk. Crisis Response In the post-crisis context of IDA16 discussions, a dedicated CRW was establisehd to facilitate IDA countries’ timely access to additional resources in response to the impact of severe economic and food crises and natural disasters. Since then, nearly half of the resources set aside for the purpose have been utilized. IEG has reviewed two of the 27 CRW-funded projects and found one satisfactorily achieved its objectives and the other not, while both faced high risk to development outcomes. Several IEG thematic evaluations have looked at the Bank Group’s crisis responses to natural disasters, global financial crisis, global food crisis, and other humanitarian 37 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS crisis. The evaluation brief on natural disaster response of the World Bank and others (IEG 2011c) found that although the short-term response to disasters tended to draw most attention, work done during the pre-disaster phase (in relation to disaster response and post-disaster phases) in terms of institution and systems building could ensure a simpler and more effective response during a given disaster. This echoes the findings of an earlier IEG evaluation (IEG 2006), which showed that efforts at reducing risks and increasing preparedness within national development strategies of countries prone to disasters could deliver substantial gains in the effectiveness of the response when disasters occur. During the disaster response phase itself, the evaluation brief pointed to the need to (i) keep project design as simple and realistic as possible, and streamline procedures for quick delivery; (ii) base project deadlines on a specific analysis of likely post- disaster (rather than “normal”) capacity; (iii) consult with a wide range of stakeholders (including the poor and the private sector) while focusing on rapid development and processing of interventions; and (iv) begin the longer term process of developing disaster preparedness, being cognizant of the longer term financing and implementation commitment involved. Post-disaster, recovery efforts that involved developing the capacity of existing (or new) institutions and community driven approaches could be very effective, while housing reconstruction programs that implicated homeowners and required adequate grievance and complaint mechanisms worked better than purely contracted arrangements. The need for flexibility and strong M&E arrangements was also emphasized. IEG’s two evaluations of the global financial crisis found that Bank Group lending increased to unprecedented level in response to the global economic crisis (IEG 2011e, 2012e). Three factors contributed to the Bank’s readiness: knowledge of poverty impact, long-term relationship with country authorities, and IBRD’s inherited financial headroom. In comparison, IDA’s additional financial support was more modest, reflecting an inelastic envelope and performance based resource allocation. In particular, the evaluations found that it was vital for the Bank to continue country engagement, policy dialogue, and analytic work in key areas (e.g., in financial sector) to maintain expertise. This would help the Bank Group anticipate crises, play a proactive role in providing early warnings and alerts to clients and to the broader international community, and be ready to act quickly (e.g., Mongolia, see box 2.7). More guidance was needed in the use of instruments and policy conditionality of crisis operations. In this regard, a review of the Bank’s overall financial position could usefully include a review of its financial products, assessing the flexibility for price adjustments on standard loan products during crisis and the usefulness of 38 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS explicit countercyclical loans (e.g., greater spread, shorter maturities). To protect the vulnerable during crises, the evaluations found that Bank interventions reached a small number of formal workers by scaling up unemployment insurance and the structurally poor through cash transfer programs, but left a “missing middle” of transient and near poor unprotected. This highlighted the need for the Bank to progress toward the adoption of a system-wide approach to social protection and risk management beyond social safety nets to ensure that data and programs be available to cope with crises. Finally, the role of IDA in the Bank’s resource-mobilization activity was emphasized: while IDA fast-tracking helped to speed the processing of eligible operations, it was no substitute for increased resources. Although MICs were more affected by the crisis given their greater global linkages, LICs were far less able to bear the costs of the crisis and thus needed more Bank proactivity on their behalf. IEG’s global food crisis evaluation generated a number of sobering conclusions and recommendations about IDA-supported crisis response programs (IEG 2014d). First, operational resources (budget and staff) were not expanded sufficiently for program preparation and supervision to match the increased and accelerated volume of operations. This highlights the need to ensure that country-driven food crisis response programs have adequate administrative budgets to facilitate effective preparation and supervision of food crisis mitigation operations. Second, the Bank acquiesced with, or supported, policies and actions that were inconsistent with its own food crisis-related policy advice or were not aligned with the country context. This pointed to the need to develop quality assurance procedures for food crisis response programs. Third, several food crisis operations exclusively supported programs for longer-term resilience, many of them in agriculture, with no component to address the immediate effects of the crisis (e.g., Afghanistan and Timor-Leste, see box 2.14). The evaluation recommended that IDA better target those most vulnerable to a food price crisis (especially children under two and pregnant and breastfeeding women) with adequate nutrition interventions in their mitigation programs. 39 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS Box 2.14. Crisis Response in Afghanistan, Mongolia, and Timor-Leste In Afghanistan, two relatively small operations—the Food Crisis Response Project ($8 million) and the Avian Flu Project ($5 million)—were completed, but only the first was able to achieve its objectives. The main impact of IDA funding was to increase the scope of an ongoing irrigation rehabilitation program, but its impact on food prices in the short term was negligible because the activities were planned to take two years. Mongolia was badly affected by the Asian financial crisis of 1998. In 2008–09, the Bank reacted promptly to realign its portfolio to help the government cope with the global economic crisis. This involved shifting the majority of new IDA resources to development policy credits and intensifying its analytic and advisory activities to deliver real-time policy advice. The Bank’s economic updates started during the crisis and was initially produced monthly, then quarterly, and now biannually. Although specific outcomes cannot be attributed to them, they were considered valuable contributions and provided a significant input into the general economic analysis and debate in Mongolia. As the economy rapidly recovered from the global crisis, however, the impetus for implementing the newly adopted laws and pursuing the reforms collapsed, and the policies introduced in the wake of the crisis were not implemented. In Timor-Leste, while the Bank (and other donors) did well to avert a major food crisis in the years following 1999, the interventions in later years focused unduly on adopting frameworks and laws that did not necessarily translate into improvements in productivity or incomes in the rural sector. Enforcing full compliance with the Bank‘s fiduciary requirements caused serious delays in Bank project implementation, including projects that were meant to provide immediate short-term benefits to the poor and where speedy implementation was essential. These delays also negated the other efforts of the Bank to respond to crisis situations and accelerate project preparation and approvals. Source: IEG (2011b; 2013a, and 2015c). Given that data scarcity was acute for most LICs, few recipient countries assessed the impact of food crisis on the poor, and most projects did not specify actual and expected program “coverage” to assess the likely contribution of the project to the population most in need of assistance. This called for greater IDA effort to work with client countries and development partners to identify practical mechanisms (including indicators) for monitoring nutritional and welfare outcomes and impacts of food crises and mitigation programs, and work with them to implement those mechanisms and report the results. There was also a need to ensure that food crisis response components, processed as re-structured projects, additional or supplemental finance operations, include appropriate M&E arrangements. 40 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS The food, fuel, and financial crises of 2008–10 underscored the urgency of developing Social Safety Net (SSN) programs in all countries. IEG’s SSN evaluation pointed to the need for the Bank to engage consistently during stable times to help countries develop SSNs that address poverty and protect the poor and vulnerable from individual and systemic shocks (IEG 2011e). Stronger demand for SSN support in MICs led to significantly stronger engagement there than in IDA countries.12 The two most common constraints for Bank support, particularly in LICs, were weak country institutions and inadequate data. Throughout the decade, the Bank’s lending, analytical, and capacity-building support for SSNs were significantly more concentrated in MICs than in IDA countries, and this may have perpetuated the latter’s low level of attention to SSNs. Countries that had developed SSN programs or institutions during “stable times” were better positioned to scale up—and the World Bank was better able to help them—than those that had not (Ethiopia, see box 2.15). Box 2.15. Crisis Response Operations The Food Crisis Response Development Policy Grant Project for Burundi emphasizes the need to fully understand local context (IEG 2012a). In this case, the choice of instruments and policy measures for an emergency food crisis intervention took into account the local causes of poverty and hunger which were identified through recent sector work. However, the operation was framed as a short-term response to spikes in world commodity prices, and was not designed to address the fundamental problem of low agricultural productivity, the root of Burundi‘s food crisis. A budget support to Djibouti in response to the global food crisis underscores the need to recognize possible trade-offs between speed and targeting accuracy in preparing emergency response operations (IEG 2012b). It also shows that emergency response can be fruitfully combined with the development of a longer-term program following a programmatic approach. Through Ethiopia’s Productive Safety Net Program, the largest SSN supported by the Bank in an IDA country, the World Bank and other donors supported the government’s transition from a situation of annual emergency food appeals to the development of a predictable SSN able of providing work opportunities and direct assistance to needy families during the annual drought season (IEG 2011e). A three-phase adaptable program loan enabled the Bank to support Ethiopia continuously through financing and analytical and technical assistance as it prepared, supervised, and evaluated each stage. Evaluation evidence indicates that program workfare beneficiaries are more likely to be food secure, borrow for productive purposes, use improved agricultural technologies, and operate nonfarm business activities. The program prevented beneficiary households from sliding deeper into poverty and selling household assets, thus protecting them from the worst effects of the shocks. Ethiopia has focused narrowly (and effectively) on food insecurity in 41 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS rural areas but is now engaged in the preparation of a broader social protection strategy that will address other sources of vulnerability as well. The National Social Action Project and the Food Crisis Response Development Policy Grant in Sierra Leone point to the benefit of adding emergency components to existing projects with an established institutional base (IEG 2013b). In these projects, the addition of the cash-for-work program to the Social Action Project benefitted from an experienced, field-oriented agency operating in all the country’s districts and having done a small pilot for cash-for-work giving it at least a minimal base for scale-up. Sources: IEG (2011e; 2012 a,b; 2013b). To improve its effectiveness in helping countries respond to shocks, the evaluation urged the Bank to maintain steady engagement through lending, knowledge work, and dialogue during stable times to build SSN institutions and systems; increase SSN engagement in LICs; improve the Results Frameworks of Bank supported SSNs to more clearly identify the needs of the poor or vulnerable and how project objectives fit within longer term objectives for developing country SSNs; assess the cost effectiveness of policy alternatives; and improve internal coordination. The priorities of the Bank’s social protection strategy reinforced these recommendations, and the Bank has made a substantial effort to increase its efforts in IDA countries, with a particular emphasis on building systems and institutional capacity. While data and capacity constraints remain, many IDA countries are better positioned to scale up SSNs to address economic crises and natural disasters than they were prior to IDA16. IEG’s project level evaluations highlight the need to understand local context in order to inform the approach to be taken and to ensure that actions address binding constraints and be well targeted (box 2.15). They also identify the need to balance different trade-offs in designing and implementing crisis response operations, which may require a quality assurance procedure applicable to crisis situations that helps mitigate potentially adverse effects associated with rapid preparation and implementation. 42 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS References IDA (International Development Association). 2015. Maximizing Development Impact, IDA 17 Implementation and Results Progress Report. Washington, DC: World Bank. IEG (Independent Evaluation Group). 2006. Hazards of Nature, Risks to Development: an IEG Evaluation of World Bank Assistance for Natural Disasters. Washington, DC: World Bank. ______.2009a. Uganda Country Assistance Evaluation 2001-07. Washington, DC: World Bank. ______. 2009b. Bangladesh Country Assistance Evaluation. Washington, DC: World Bank. ______. 2009c. Climate Change and the World Bank Group —Phase I: An Evaluation of World Bank Win-Win Energy Policy Reforms. Washington, DC: World Bank. ______. 2010a. Cambodia, An IEG Country Assistance Evaluation 1999-2006. Washington, DC: World Bank. ______. 2010b. An Evaluation of World Bank Support, 2002–08: Gender and Development. Washington, DC: World Bank. ______. 2010c. Climate Change and the World Bank Group—Phase II: The Challenge of Low-Carbon Development. Washington, DC: World Bank. ______. 2011a. Mozambique Country Program Evaluation. Washington, DC: World Bank. ______. 2011b. Timor-Leste Country Program Evaluation, 2000-2010. Washington, DC: World Bank. ______. 2011c. Growth and Productivity in Agriculture and Agribusiness: Evaluation Lessons from World Bank Group Experience. Washington, DC: World Bank. ______. 2011d. Natural Disaster Response—Lessons from Evaluations of the World Bank and Others. Washington, DC: World Bank. ______. 2011e. The World Bank Group's Response to the Global Economic Crisis: Phase 1. Washington, DC: World Bank. ______. 2011f. Social Safety Nets: An Evaluation of World Bank Support 2000–2010. Washington, DC: World Bank. ______. 2011g. The World Bank in Nepal, 2003-2008 (Country Program Evaluation). Washington, DC: World Bank ______. 2012a. Burundi—Food Crisis Response Development Policy Grant Project. PPAR No. 67471. Washington, DC: World Bank. ______. 2012b. Liberia Country Program Evaluation: 2004-2011. Washington, DC: World Bank. ______. 2012c. Adapting to Climate Change: Assessing World Bank Group Experience—Phase III of the World Bank Group and Climate Change. Washington, DC: World Bank. ______.2012d. Djibouti—Food Crisis Response Development Policy Grant Project. PPAR No. 67189. Washington, DC: World Bank. ______. 2012e. The World Bank Group's Response to the Global Economic Crisis: Phase II. Washington, DC: World Bank. ______.2013a. Afghanistan Country Program Evaluation, 2002-2011. Washington, DC: World Bank. ______. 2013b. Sierra Leone—National Social Action Project. PPAR No. 76197. Washington, DC: World Bank. 43 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS ______. 2013c. Afghanistan—First, Second, and Third Programmatic Support for Institution Building Project and Strengthening Institutions Development Policy Grant Project. PPAR No. 78801. Washington DC: World Bank Group. ______. 2013d. Youth Employment Programs: An Evaluation of World Bank and International Finance Corporation Support. Washington, DC: World Bank. ______. 2014. The Big Business of Small Enterprises: Evaluation of the World Bank Group Experience with Targeted Support to Small and Medium-Size Enterprises, 2006–12. Washington, DC: World Bank. ______. 2014a. Mozambique—Enterprise Development Project. PPAR No. 85596. Washington, DC: World Bank. ______. 2014b. Bangladesh—Rural Electrification and Renewable Energy Development; and Power Sector Development Technical Assistance Project. PPAR No. 88546. Washington, DC: World Bank. ______. 2014c. Nigeria—Second National Fadama Development Project. PPAR No. 88958. Washington, DC: World Bank. ______. 2014d. Social Safety Nets and Gender: Learning from Impact Evaluations and World Bank Projects. Washington, DC: World Bank. ______. 2014e. World Bank Group Assistance to Low-Income Fragile and Conflict-Affected States. Washington, DC: World Bank. ______. 2014f. The World Bank Group and the Global Food Crisis: An Evaluation of the World Bank Group Response. Washington, DC: World Bank. ______. 2015a. The Plurinational State of Bolivia, Country Program Evaluation FY05-13. Washington, DC: World Bank. ______. 2015b. Financial Inclusion: A Foothold on the Ladder toward Prosperity? An Evaluation of World Bank Group Support for Financial Inclusion for Low-Income Households and Microenterprises. Washington, DC: World Bank. ______. 2015c. Mongolia, Country Program Evaluation FY05-13. Washington, DC: World Bank. ______. 2015d. Zambia, Country Program Evaluation FY04-13. Washington, DC: World Bank. ______. 2015e. World Bank Group Engagement in Resource-Rich Developing Countries: The Cases of the Plurinational State of Bolivia, Kazakhstan, Mongolia, and Zambia, Clustered Country Program Evaluation Synthesis Report. Washington, DC: World Bank. ______. 2015f. World Bank Group Support to Electricity Access, FY2000 –2014. Washington, DC: World Bank. ______. 2015g. World Bank Support to Early Childhood Development. Washington, DC: World Bank. ______. 2015h. Sri Lanka—Community Development and Livelihood Improvement Gemi Diriya Project. PPAR No. 92788. Washington, DC: World Bank. ______. 2015i. The Poverty Focus of Country Programs: Lessons from World Bank Experience. Washington, Dc: World Bank. ______. 2015j. Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms and Business Regulations. Washington, DC: World Bank. doi:10.1596/978-1-4648-0628- 5. License: Creative Commons Attribution CC BY 3.0 IGO. ______. 2016. Results and Performance of the World Bank Group 2015. Washington, DC: World Bank. O'Sullivan, M., A. Rao; R. Banerjee, K. Gulati, and V. Margaux. 2014. Levelling the Field: Improving Opportunities for Women Farmers in Africa. Washington, DC: World Bank Group. 44 CHAPTER 2 EVALUATION FINDINGS IN IDA THEMATIC AREAS World Bank. 2015. Gender Equality Progress Report: IDA17 Mid-Term Review. Washington, DC: World Bank. 1 Includes 11 country assistance and program evaluations completed in FY08–15, 20 IEG thematic evaluations since 2010, and 49 Project Performance Assessment Reports completed in FY12–15 covering 83 projects in IDA countries. See appendices A-C for the list of evaluations reviewed. 2 Three Project Performance Assessment Reports touched on youth employment issues in the context of temporary work in post-war Sierra Leone and skills development and curriculum updating for tertiary education in Sri Lanka. 3 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2622220. 4 The Global Financial Inclusion Database, known as Global Findex, was presented during the 2015 annual spring meetings of the International Monetary Fund and World Bank. 5 The International Finance Corporation and the Multilateral Investment Guarantee Agency have standardized definitions, but the evaluation found them to be ill tuned to many local contexts. 6 The evaluation recommended the World Bank Group institutionalize the management accountability framework, develop a monitoring system to assess how well the Bank’s work program addresses gender-related concerns, establish a Results Framework to facilitate adoption of an outcome focused approach to gender integration, and restore a broader requirement for gender integration at the project level. 7 The progress report notes that drawing from the findings of O’Sullivan and others (2014), the Bank’s Growth Poles Project in the Democratic Republic of Congo started to test the recommended policy action of providing rural childcare to help women spend more time on agricultural work (World Bank 2015). 8 Many of the lessons on FCS are addressed under separate IDA special themes (e.g., gender). This section focuses on more systemic issues facing FCS countries. 9 The average implementation support cost per project for FCS countries is now $155,000, compared to $175,000 for all International Development Association (IDA) countries. See IDA17 Implementation and Results Progress Report, November 2015, p. 71. 10 The Country Policy and Institutional Assessment indicators were developed as an instrument to determine Performance-Based Allocations of IDA resources. 11 Staff deployed in FCS country offices increased by 68 percent over the evaluation period, twice the increase of nonfragile IDA countries, and the number of senior staff (GH+) based in country offices doubled. 12 The Bank provided $11.5 billion for social safety nets (SSNs) in 83 countries in FY00–10, half of it during the last two years of that decade in the immediate aftermath of the crises. However, 70 percent of the SSN lending portfolio in FY00–10 went to 10 countries which, with the exception of Ethiopia and Pakistan, were middle-income countries. 45 3. Managing for Results IDA has been a pioneer in results measurement. In 2002, IDA13 adopted the two- tiered RMS to track the progress of IDA countries and IDA-supported development results. It was the first time for a multilateral development institution to use a framework with quantitative indicators to monitor results and performance. In 2010, IDA16 expanded the RMS from two to four tiers to track IDA operational and organizational effectiveness. It also introduced indicators to track progress on the IDA special themes. These activities inspired the World Bank in 2011 to adopt the Corporate Scorecard, which drew heavily from the RMS indicators. The IDA16 process also included a Working Group on Development Results. Based on its success, IDA17 re-established the group with the objective of providing forward-looking suggestions for how IDA might improve its impact on the lives of the poor and its accountability. Deliberations of the working group have identified three sets of issues for further discussion in IDA18: IDA Results Framework; results culture, and learning from results. IEG evaluations on Bank Group results and learning, as well as those on IDA special themes, offer relevant lessons. Results and Learning IEG’s evaluation of Bank Group self-evaluation systems, Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group, finds that the Bank Group has put in place comprehensive self-evaluation systems the design and operation of which adhere to relevant good practice standards (IEG 2016). Consisting of cascading indicators, some of which draw on information from project supervision and completion reports, the system provides a framework and data for corporate results reporting to the Board and other stakeholders. The systems mesh well with the independent evaluation systems for which they provide information. Many evaluation experts consider the design of the Bank Group’s systems to be as good as or better than those in comparable organizations. However, the existing self-evaluation systems are found to focus primarily on meeting the need for accountability and do not well serve the emphasis placed by the 2013 Bank Group Strategy on the “Solutions Bank” and learning to enhance performance. That is, the focus on corporate results measurement and reporting has sidelined other purposes, such as using the system for performance management, strategic decision making, and learning. For example, when the Bank’s Implementation Status and Results Report (ISR) system works as intended, flags are 46 CHAPTER 3 MANAGING FOR RESULTS raised at the right time,1 and teams and managers act on them to turn problem projects around. Yet ISR ratings and indicators derived from them are not always precise because of weak project monitoring and optimistic reporting. The ISR would be a more effective early warning tool if team leaders had incentives to proactively acknowledge issues and raise risk flags. Better early warning needs to be combined with timely action. IEG’s second evaluation of learning and results in World Bank operations (IEG 2015a) also notes a lack of candor in ISR ratings as the supervision record often understates the severity of the problems a project encounters.2 A detailed analysis of IEG’s Implementation Completion and Results Report Reviews finds that commonly occurring supervision issues, such as inadequate attention to M&E (45 percent), excessively optimistic ISR ratings (30 percent), and weak project management (28 percent) are far more prevalent among unsuccessful than successful projects.3 IEG thematic and project evaluations highlight some long-standing shortcomings in the M&E system that generate the underlying evidence for results (box 3.1). Weak M&E is a factor in 70 percent of all downgrades of project completion reports. At the country strategy level, the introduction of results frameworks in 2005 has improved country program self-evaluations, but results measurement continues to be an issue. IEG’s learning note on the results framework in country strategies (IEG 2015b) finds that of the 25 country partnership strategies approved in FY14, less than 50 percent were fully aligned with the objectives although 90 percent had measurable indicators. Also, the country strategy results frameworks are primarily based on Bank project level M&E frameworks and in many cases lack country-level indicators. This leads to a substantial gap between Bank Group strategic objectives and the indicators provided to measure program impact. Because of this interrelationship, weaknesses in project M&E are carried over to country program results measurement. Box 3.1. Challenges with Monitoring and Evaluation Monitoring and evaluation (M&E) is a problem that cuts across multiple evaluations. Close to two-thirds of the thematic evaluations by the Independent Evaluation Group (IEG) include a recommendation to improve M&E. Many recommendations focus on strengthening Results Frameworks and indicators, especially outcome indicators, for the World Bank Group to be in a better position to assess the impact of its interventions and determine whether they are meeting their development objectives. Some recommendations point to working with the client to build their data collection and M&E capacity. M&E systems for new and cross-cutting areas, such as innovation and entrepreneurship, public-private partnerships, and procurement, are also vital. 47 CHAPTER 3 MANAGING FOR RESULTS Bank Group management generally agrees with IEG’s recommendations but cites many challenges such as difficulty in collecting data especially when relying on the countries to generate the necessary information, methodological issues in assessing project impacts and in developing outcomes, and the time lag for outcomes to materialize, sometimes several years after a Bank Group intervention is completed. In its annual updates, management acknowledges the difficulties in strengthening M&E and rates only one-half the M&E related recommendations as “substantial” or better for implementation by the fourth year of follow-up, compared to an average of 89 percent. Source: IEG (2015c). Having all operational units write substantive end-of-project reports is a noteworthy accomplishment that not many other organizations can boast. In practice, however, the two recent evaluations of Learning and Results in World Bank Operations (IEG 2014 and 2015a) and Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group (IEG 2016) find that knowledge from the Bank Group systems is not much used, and there is limited effort to extract and synthesize evidence and lessons to inform operations. Staff are more likely to rely on tacit knowledge than on written information from the self-evaluation systems. There is some individual learning, but few benefits of this learning accrue beyond the authors, leaving the potential of the systems for organizational learning unfulfilled. Learning has taken a backseat to accountability. The systems’ focus on accountability drives the shape, scope, timing, and content of reporting, and limit the usefulness of the exercise for learning. The Learning and Results evaluations (IEG 2014 and 2015a) and Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group (IEG 2016) find that the observed shortcomings are associated with incentives and behaviors rather than templates and processes. Incentives created inside and outside systems are not conducive to conducting high- quality self-evaluation, and most staff do not view the self-evaluation systems as a source of timely, credible, and comprehensive information. Staff engage with the systems with a compliance mindset, where candor and thoughtful analysis of drivers of results and failures suffer. Parts of the system not focused on corporate reporting, such as impact evaluations and other voluntary self-evaluations, are more valued by respondents. An increasing number of impact evaluations are carried out at project level. These optional, often self-selected evaluations are seen as technically credible and regarded as a valuable tool to increase development effectiveness. Demand for such learning evaluations is stronger, but they need to be deployed more strategically to understudied areas, and the feedback loop between evaluation findings and operations needs to be strengthened. 48 CHAPTER 3 MANAGING FOR RESULTS Despite a strong commitment to deliver results, these earlier evaluations have highlighted that the culture and systems of the organization, the incentives offered to staff, and the signals from managers are not as effective as they could be. Many teams do not feel rewarded for good M&E and for identifying and fixing problems in real time. Management needs to create more incentives for staff to engage with the self-evaluation systems with candor, conduct thoughtful analysis of both successes and failures, and move from a compliance mindset to a learning perspective. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group (IEG 2016) identifies three broad causes of misaligned incentives for using self-evaluation systems: excessive focus on ratings, attention to volume that overshadows attention to results, and low perceived value of the knowledge created. It then puts forward three preliminary suggestions:  Make the ICR system more compatible with innovation and course corrections by reducing the barriers to innovation and experimentation, and devoting greater attention to evaluability and adaptability in order to balance the excessive focus on ratings.  Strengthen leadership signals about the value of self-evaluation by making better institutional use of the knowledge generated.  Enhance the value of self-evaluations by using self-evaluations strategically to bridge knowledge gaps, developing frameworks for capturing, storing, and acting on evaluative lessons, tracking projects with high-quality results frameworks, and testing approaches to uphold both accountability and learning needs. Cross-Cutting Issues Despite differences in the five IDA special themes, the relevant IEG evaluation findings point to a core set of generally applicable lessons that suggest positive development impact will be enhanced when:  The special themes are part of the underlying Bank Group knowledge work;  The agenda is owned by the client countries and supported by successive governments;  IDA’s country strategies specifically target and fund actions to pursue them; and  Results of the thematic targeting are monitored as part of the strategy implementation process. 49 CHAPTER 3 MANAGING FOR RESULTS It is important to note that IEG evaluations are prepared to help the Bank Group deliver the best possible development results in a particular area. Each may thus suggest additional efforts for achieving that goal. Clearly, a strategic choice has to be exercised at the corporate level as to which thematic areas are relatively more important for IDA at each moment. The themes need to be assessed within the context of the changing development landscape, Bank Group strategic directions, IDA’s value proposition, and the need for enhanced learning, innovation, and results -- to ensure that they are not simply add-ons. For example, the viability of the themes should be the demonstrated through socioeconomic and technical analysis. This would help manage the tension between the pressure to add to the development agenda for political expediency and the realities of IDA’s resource (staff and budget) constraints. Strategic choice also needs to be exercised at country level to take into account the nature and dynamics of the issues in the local context and the key constraints to resolving them. As demonstrated in the country examples referenced above, earlier evaluations indicated room for better alignment between IDA’s thematic aspirations and country strategies. The SCD/CPF process could play an important role for assessing the trade-offs and making strategic decisions on the Bank Group side. However, country ownership of IDA’s development agenda determines the ultimate outcomes. Just as the Heavily Indebted Poor Country Initiative was tied to the Poverty Reduction Strategy Papers, which also proved to be a useful instrument for advancing the MDGs (IEG 2015c), future IDA strategic and programmatic support could be strengthened through a coherent, dynamic, and well-sequenced strategy by IDA recipients. Learning from past experiences and correcting previous mistakes, IDA could assist its clients to address the IDA special themes within the context of the country’s specific priorities, constraints, funding requirements, and M&E framework. The issue with project and country program M&E is worth reiterating. Despite IDA’s strong effort to strengthen results measurement for both accountability and learning purposes, the current results monitoring systems remain underutilized for either objective. This is not a problem unique to IDA or the Bank Group; all other development agencies face the same problem. It is particularly important to increase the focus on learning from projects that have follow-on projects or that are being considered for adoption in other contexts, and to ensure that those involved in designing these mine the existing evidence. In this context, clear signals from the top are critical for creating a robust results culture, including an emphasis on the importance of adequate and timely results measurement, consistent incentives, and adequate training and support to staff as well as more attention to supporting the development of country capacity and country M&E systems. 50 CHAPTER 3 MANAGING FOR RESULTS References Independent Evaluation Group. 2014. Learning and Results in World Bank Operations: How the World Bank Learns, Evaluation 1. Washington, DC: World Bank. ______. 2015a. Learning and Results in World Bank Operations: Toward a New Learning Strategy, Evaluation 2. Washington, DC: World Bank. ______. 2015b. Results Frameworks in Country Strategies: Lessons from Evaluations. Washington, DC: World Bank. ______. 2015c. Results and Performance of the World Bank Group 2014 . Washington, DC: World Bank. ______. 2016. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Washington, DC: World Bank. 1 Some of these ISR warning flags were removed in ISR simplification. 2In poorly supervised projects, task team issues, such as expertise, frequent changes in task team leader, timely succession, and coordination within the Bank team, were identified. 3 Stratified random sample of 144 Investment Projects Financing closed FY12–14. 51 Appendix A. Country Program Evaluations from IDA and Blend Countries Lending Delivery FY Country Region Category 2009 Uganda AFR IDA 2009 Bangladesh SAR IDA 2009 Nepal SAR IDA 2010 Cambodia EAP IDA 2010 Mozambique AFR IDA 2011 Timor-Leste EAP Blend 2012 Liberia AFR IDA 2013 Afghanistan SAR IDA 2015 Mongolia EAP Blend 2015 Bolivia LCR Blend 2015 Zambia AFR IDA Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; FY = fiscal year; IDA = International Development Association; LCR = Latin America and the Caribbean; SAR = South Asia. 53 Appendix B. Thematic Evaluations and Learning Products Evaluation Type of Fiscal Year Evaluation Evaluation 2009 Climate Change and the World Bank Group—Phase I: An Evaluation of World Thematic Bank Win–Win Energy Policy Reforms 2010 An Evaluation of World Bank Support, 2003–08: Gender and Development Thematic 2010 World Bank Group's Response to the Global Economic Crisis, Phase I Thematic 2010 The Challenge of Low-Carbon Development Thematic 2011 Social Safety Nets: An Evaluation of World Bank Support 2000–2010 Thematic 2011 Natural Disaster Response: Lessons from Evaluations of the World Bank and Thematic Others 2011 Growth and Productivity in Agriculture and Agribusiness: Evaluation Lessons Thematic from World Bank Group Experience. 2012 World Bank Group's Response to the Global Economic Crisis, Phase II Thematic 2012 Adapting to Climate Change: Assessing World Bank Group Experience—Phase Thematic III of the World Bank Group and Climate Change. 2013 World Bank Group's Response to the Global Food Crisis Thematic 2013 Adapting to Climate Change: Assessing World Bank Group Experience Thematic 2013 Youth Employment Programs: An Evaluation of World Bank and International Finance Corporation Support 2014 World Bank Group Assistance to Low-Income Fragile and Conflict-Affected Thematic States 2014 The Big Business of Small Enterprises: Evaluation of the World Bank Group Thematic Experience with Targeted Support to Small and Medium-Size Enterprises, 2006– 12 2014 Learning and Results in World Bank Operations: How the Bank Learns, Thematic Evaluation 1 2014 World Bank Group Support to Health Financing Thematic 2014 Investment Climate Reforms: An Independent Evaluation of World Bank Group Thematic Support to Reforms and Business Regulations 2014 Social Safety Nets and Gender: Learning from Impact Evaluations and World Learning Bank Projects Product 2014 The World Bank Group and the Global Food Crisis: An Evaluation of the World Thematic Bank Group Response 2014 Results and Performance of the World Bank Group 2013: An Independent Thematic Evaluation 2015 Results and Performance of the World Bank Group 2014: An Independent Thematic Evaluation 2015 The Poverty Focus of Country Programs: Lessons from World Bank Experience Thematic 54 CHAPTER 3 MANAGING FOR RESULTS 2015 World Bank Group Engagement in Resource-Rich Developing Countries: The Thematic Cases of the Plurinational State of Bolivia, Kazakhstan, Mongolia, and Zambia, Clustered Country Program Evaluation Synthesis Report. 2015 World Bank Support to Early Childhood Development Thematic 2015 Learning and Results in World Bank Operations: Toward a New Learning Thematic Strategy, Evaluation 2 2015 World Bank Group Support to Electricity Access, FY2000–2014 Learning product 2015 Financial Inclusion: A Foothold on the Ladder toward Prosperity? An Evaluation Thematic of World Bank Group Support for Financial Inclusion for Low-Income Households and Microenterprises 2015 The Poverty Focus of Country Programs: Lessons from World Bank Experience Thematic 2015 Review of Instruments: DPO (5 notes) Learning product 2015 Financial Inclusion Thematic 2016 Results and Performance of the World Bank Group 2015 Thematic 2016 Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Thematic Group 55 Appendix C. List of Project Performance Assessment Reports, FY05–14 Evaluation Lending No. Fiscal Year Category Region Country Project Name 1 2012 IDA AFR Burundi BI - Food Crisis Response DP Grant - P113438 2 2012 TF LCR Caribbean GEF 6R-CRB Planning for Adapt Clim Chang - P040739 3 2012 TF LCR Caribbean GEF MACC 6R Mainstreaming Adapt to Clima - P073389 4 2012 IDA AFR Congo, Dem. DRC Econ Recovery Credit SAL (FY02) - P057293 Rep. 5 2012 IDA AFR Congo, Dem. DRC - Post-Reunification - ERC - P082443 Rep. 6 2012 IDA MNA Djibouti Djibouti - Food Crisis Response Dev. - P112017 7 2012 TF EAP Mekong 4M-Mekong Water Utilization - P045864 8 2012 IDA AFR Mozambique MZ-Roads and Bridges MMP (FY02) - P001785 9 2012 IDA AFR Mozambique MZ-Railway and Port Restr (FY00) - P042039 10 2012 Blend SAR Pakistan BSRPP - P055292 11 2012 Blend SAR Pakistan PK Banking Sector Dev. Policy Credit - P083079 12 2012 Blend AFR Congo, Rep. Congo, Rep. Post Conflict–Econ. Rehab - P073316 13 2012 Blend AFR Congo, Rep. CG-Econ Recovery Credit ERL (FY05) - P083627 14 2012 Blend EAP Vietnam VN - Mekong Delta Water - P004845 15 2012 Blend EAP Vietnam VN-3 Cities Sanitation - P051553 16 2013 IDA SAR Afghanistan AF - Programmatic Support for Inst. Building - P078618 17 2013 IDA SAR Afghanistan AF Program. Support for Inst. Building II - P090829 18 2013 IDA SAR Afghanistan Program. Support for Inst. Building III - P102709 19 2013 IDA SAR Afghanistan Afg Strengthening Institutions DPG - P107921 20 2013 IDA AFR Gambia, The GM-Gateway SIL (FY02) - P057394 21 2013 IDA AFR Ghana GH Land Administration (FY04) - P071157 22 2013 IDA AFR Malawi MW-Com Based Rural Land Dev (FY04) - P075247 23 2013 IDA SAR Nepal NP: Avian Flu - P100342 24 2013 Blend AFR Nigeria Avian Influenza Emergency ERL (FY06) - P100122 25 2013 IDA AFR Sierra Leone SL-Natl Social Action (FY03) - P079335 26 2013 IDA AFR Sierra Leone SL-DPL-Food Crisis Response - P113219 27 2013 Blend SAR Sri Lanka Relevance and Quality of Undergraduate Education - P050741 28 2013 Blend SAR Sri Lanka LK: Education Sector Development Project - P084580 29 2013 IDA ECA Tajikistan Avian Flu - TJ - P100451 30 2013 IDA AFR Tanzania TZ-PRSC 1 (first) - P074072 31 2013 IDA AFR Tanzania TZ-PRSC2 (intermediate) - P074073 32 2013 IDA AFR Tanzania TZ-PRSC 3 (last) - P087256 33 2013 IDA AFR Tanzania TZ-PRSC 4 DPL (first of second series) - P095509 56 APPENDIX C LIST OF PROJECT PERFORMANCE ASSESSMENT REPORTS, FY05–14 34 2013 IDA AFR Tanzania TZ-PRSC 5 (second of second series) - P095657 35 2013 IDA AFR Tanzania TZ-PRSC 6 (third of second series) - P101229 36 2013 IDA AFR Tanzania TZ-PRSC 7 (fourth in second series) - P101230 37 2013 IDA AFR Tanzania TZ-PRSC 8 (fifth and last in second series) - P116666 38 2014 IDA SAR Bangladesh Primary Education Development Project II 39 2014 IDA SAR Bangladesh Rural Electricity and Renewable Energy Development Project (P071794) 40 2014 IDA SAR Bangladesh Power Sector Development Technical Assistance Project (P78707 41 2014 IDA SAR Bangladesh Power Sector Development Policy Loan (P107797) 42 2014 IDA AFR Ghana Trade Gateway and Investment Project 43 2014 IDA AFR Ghana Rural Finance Services SIL 44 2014 IDA AFR Ghana First Natural Resource and Environmental Governance Projects (IDA-44260, IDA-46270, IDA-47460) 45 2014 IDA AFR Ghana Second Natural Resource and Environmental Governance Projects (IDA-44260, IDA-46270, IDA-47460) 46 2014 IDA AFR Ghana Third Natural Resource and Environmental Governance Projects (IDA-44260, IDA-46270, IDA-47460) 47 2014 IDA EAP Lao PDR Second Land Titling Project (P132897) 48 2014 IDA AFR Mozambique Decentralization Planning and Fin SIL 49 2014 IDA AFR Mozambique Enterprise Development 50 2014 IDA AFR Mozambique Public Sector Reform Project 51 2014 Blend AFR Nigeria Nigeria State Education Sector Project 51 2014 Blend AFR Nigeria Nigeria Community Based Poverty Reduction Project 52 2014 Blend AFR Nigeria Fadama II 53 2014 IDA AFR Senegal Energy Sector Recovery Development Policy 54 2014 IDA AFR Senegal Electricity Sector Efficiency–Phase 1–Apl 1 55 2014 Blend SAR Sri Lanka Renewable Energy for Rural Economic Dev. Project (P077761) 56 2014 Blend LCR St. Lucia Economic and Social Development Loan 57 2014 IDA AFR Tanzania Public Sector Reform 58 2014 IDA AFR Uganda Local Governance Development 2 59 2014 Blend EAP Vietnam Transmission, Dist., and Disaster Recon. Project (P045628) 60 2014 Blend EAP Vietnam Systems Energy Equitization 61 2014 Blend EAP Vietnam Rural Energy Project (P056452) 62 2015 IDA LCR Nicaragua Education Project (P078990) 63 2015 IDA AFR Rwanda Decentralization and Community Development 64 2015 IDA AFR Senegal Senegal Long Term Water Sector Project P041528 65 2015 IDA AFR Senegal Supporting Access to On-site Sanitation P102478 66 2015 IDA AFR Senegal Integrated Marine and Coastal Resource Management Project (P086480) 67 2015 IDA AFR Senegal Sustainable Management of Fish Resources (P105881) 68 2015 Blend SAR Sri Lanka Sri Lanka Cluster–Gemi Dirlya I (P074872) 69 2015 Blend SAR Sri Lanka Sri Lanka Cluster–North-East Irrigated Agric. (P058070) 57 APPENDIX C LIST OF PROJECT PERFORMANCE ASSESSMENT REPORTS, FY05–14 70 2015 IDA AFR Tanzania Tanzania Government Support Project (P070736) 71 2015 IDA AFR Uganda Uganda DPO–2 PRSC series (PRSCs 5–7 and 8–9) + 1 financial DPO (truncated series) 71 2015 Blend ECA Uzbekistan Uzbekistan Rural Water Supply and Sanitation Project (P009121) 72 2015 Blend ECA Uzbekistan Bukhara and Samarkand Water Supply Project (P049621) 73 2015 Blend EAP Vietnam Second Higher Education Project 74 2015 Blend EAP Vietnam Vietnam Poverty Reduction Support Credit 6 P101724 75 2015 Blend EAP Vietnam Vietnam Poverty Reduction Support Credit 7 P105287 76 2015 Blend EAP Vietnam Vietnam Poverty Reduction Support Credit 8 P111164 77 2015 Blend EAP Vietnam Vietnam Poverty Reduction Support Credit 9 P111182 78 2015 Blend EAP Vietnam Vietnam Poverty Reduction Support Credit 10 P111183 79 2015 Blend EAP Vietnam VN-Public Investment Reform 1 P117723 80 2015 Blend EAP Vietnam VN-Public Investment Reform 2 P120946 81 2015 Blend EAP Vietnam VN-Program 135 Phase 2 Support Credit P104097 82 2015 Blend EAP Vietnam Second Program 135 Phase 2 Support P107062 83 2015 Blend EAP Vietnam Third Program 135 Phase 2 Development Policy Support Operation P117610 Note: AFR = Africa; DPL = Development Policy Lending; DPO = Development Policy Operation; EAP = East Asia and Pacific; ECA = Europe and Central Asia; IDA = International Development Association; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia; TF = Trust Fund. 58