Independent Evaluation Group Validation of the Management Action Record 2023 © 2023 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION Please cite the report as: World Bank. 2023. Independent Evaluation Group Validation of the Management Action Record 2023. Independent Evaluation Group. Washington, DC: World Bank. COVER PHOTO Shutterstock / MADEINEDEN EDITING AND PRODUCTION Amanda O’Brien GRAPHIC DESIGN Luísa Ulhoa This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclu- sions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Independent Evaluation Group Validation of the Management Action Record 2023 October 19, 2023 Contents Abbreviations ..................................................................................................................................................... v Acknowledgments ........................................................................................................................................... vi Overview ............................................................................................................................................................vii 1. Introduction ...................................................................................................................................................1 2. Validation Framework ................................................................................................................................2 3. Progress toward Achieving Intended Outcomes of Recommendations .................................6 Quality of Evidence .....................................................................................................................................6 Progress toward Outcomes ...................................................................................................................... 7 Retirement .................................................................................................................................................... 12 4. Progress on the Implementation of Recommendations, by Evaluation............................... 14 Investing in Human Capital .................................................................................................................... 14 Promoting Climate and Resilience ....................................................................................................... 16 Crowding in Private Finance .................................................................................................................. 24 Boosting Long-Term Economic Growth ............................................................................................ 27 Improving World Bank Group Capacity to Meet Its Goals .......................................................... 35 5. Conclusions and Suggestions for Further Enhancement of the Management Action Record ................................................................................................................................................................ 41 References ........................................................................................................................................................ 43 Box Box 3.1. The Implementation of Carbon Finance Recommendations: Responding to External Change ..................................................................................................................................................................9 Figure Figure 2.1. Management Action Record Process, Fiscal Year 2023 ...................................................3 Tables Table 2.1. Management Action Record Assessment Framework, Fiscal Year 2023 ......................3 Table 3.1. Independent Evaluation Group Assessment of Implementation Progress of Recommendations, by Fiscal Year of CODE Discussion........................................................................6 Table 3.2. Evaluations from Fiscal Years 2021 and 2022 Showing Limited Evidence of Progress .............................................................................................................................................................. 10 iii Contents Table 3.3. Recommendations Proposed for Retirement (Management Action Record 2023) ............................................................................................................................................................................... 13 Table 4.1. Health Services, Fiscal Years 2019–23 .................................................................................... 14 Table 4.2. Undernutrition, Fiscal Years 2022–26 ................................................................................... 15 Table 4.3. Carbon Finance, Fiscal Years 2019–23 .................................................................................. 16 Table 4.4. Urban Resilience, Fiscal Years 2020–24................................................................................ 17 Table 4.5. Renewable Energy, Fiscal Years 2021–25 ............................................................................. 19 Table 4.6. Natural Resource Degradation and Vulnerability, Fiscal Years 2021–25 .................. 20 Table 4.7. World Bank Engagement in Situations of Conflict, Fiscal Years 2022 –26 ................. 21 Table 4.8. Municipal Solid Waste Management, Fiscal Years 2022–26 ........................................ 22 Table 4.9. Urban Spatial Growth, Fiscal Years 2022–26 ..................................................................... 23 Table 4.10. Private Capital Mobilization, Fiscal Years 2021 –25 ........................................................ 24 Table 4.11. Private Investment in Fragile and Conflict-Affected Situations, Fiscal Years 2022– 26 ......................................................................................................................................................................... 25 Table 4.12. Regional Integration, Fiscal Years 2019–23 ...................................................................... 27 Table 4.13. Trade Facilitation, Fiscal Years 2019–23 ............................................................................. 27 Table 4.14. State-Owned Enterprises, Fiscal Years 2021–25.............................................................. 28 Table 4.15. Public Financial and Debt Management, Fiscal Years 2021 –25 ................................. 29 Table 4.16. Mobilizing Technology for Development, Fiscal Years 2021 –25 ............................... 30 Table 4.17. Sustainable Development Finance Policy, Fiscal Years 2022 –26............................... 32 Table 4.18. Doing Business Report and Business Ready, Fiscal Years 2022–26 ........................... 34 Table 4.19. Citizen Engagement, Fiscal Years 2019–23 ....................................................................... 35 Table 4.20. Convening Power, Fiscal Years 2020–24 .......................................................................... 37 Table 4.21. Outcome Orientation, Fiscal Years 2021–25 .................................................................... 38 Table 4.22. Global Footprint Effectiveness, Fiscal Years 2022–26................................................... 39 Appendix Appendix A. Summary Table of Progress Levels................................................................................ 46 iv Abbreviations ASA advisory services and analytics CCDR Country Climate and Development Report CODE Committee on Development Effectiveness CPF Country Partnership Framework DSA debt sustainability analysis ESF Environmental and Social Framework FCS fragile and conflict-affected situation FCV fragility, conflict, and violence FY fiscal year GP Global Practice IDA International Development Association IEG Independent Evaluation Group IFC International Finance Corporation MAR Management Action Record MIGA Multilateral Investment Guarantee Agency NRDV natural resource degradation and vulnerability PC progress constrained PCM private capital mobilization PFDM public financial and debt management PPA performance and policy action SOE state-owned enterprise All dollar amounts are US dollars unless otherwise indicated. v Acknowledgments This report was led by Stephen Porter under the supervision of Lily L. Chu. Arjun Kaushik coproduced the report. Overall direction was provided by Oscar Calvo- Gonzalez and Sabine Bernabe. Independent Evaluation Group contributors who provided critical inputs to this report include Anahit Aghumian, Elena Bardasi, Mariana Branco, Soniya Carvalho, Jeff Chelsky, Jenny Gold, Mitko Grigorov, Rasmus Heltberg, Stephen Hutton, Ramachandra Jammi, Jennifer Keller, Lauren Kelly, Marialisa Motta, Raghavan Narayanan, Maria-Elena Pinglo, Estelle Raimondo, Heinz Rudolph, Andrew Stone, Aurora Medina Siy, Mercedes Vellez, Victor Vergara, and Stephan Wegner. Zorayda Opana provided administrative support. The collaborative approach between the Independent Evaluation Group and management led to rich discussions and concrete proposals to enhance the quality of the Management Action Record process and product. The Independent Evaluation Group is thankful to Operations Policy and Country Services, International Finance Corporation, and Multilateral Investment Guarantee Agency colleagues, including Hiroyuki Hatashima, Arvinth Booma Jayakumar, Sparsh Kansal, Alan Andres Kaufman, Debra Ladner, Cristina Marosan Ling, Liane Asta Lohde, Diego Garrido Martin, Lisandro Martin, Maurya West Meiers, Ayesha Muzaffar, Vijay Pillai, Diego Carrillo Purin, Vlajko Senic, and Peixuan Zhou for their close collaboration. We also thank the more than 130 technical focal points and respective managers for their commitment to providing the expanded evidence base and enhanced engagement process. vi Overview This report provides the Independent Evaluation Group (IEG) validation of World Bank Group management’s report Learning and Adapting for Outcomes through the Management Action Record 2023: A World Bank Group Management Report on Implementation of IEG Recommendations for the period July 2022 to June 2023. The purpose of the Management Action Record (MAR) assessment system is to support accountability, learning, and adaptation for the Bank Group’s implementation of recommendations from IEG evaluations. This validation document presents IEG’s assessment of progress toward achieving the intended outcomes of evaluations and the evidence in management’s MAR report. Quality of Evidence Building from last year, the MAR report for fiscal year (FY)23 provides a thorough review with improved assessment of progress toward fulfilling IEG’s recommendations. To enable improved learning and adaptation on recommendations, IEG and management enhanced the assessment framework, information flow, and dialogue on recommendations. The MAR process engaged 22 IEG evaluators and more than 130 Bank Group technical focal points for the 22 evaluations under review. Management provided for each evaluation useful evidence that describes progress along a results chain for each recommendation. This evidence draws on limited portfolio reviews, analysis of commitments, examples of practice, and documents. In the report, categories of progress are applied that aid shared assessment toward recommendations’ intended outcomes. Management and IEG applied the same assessment categories to assess progress on recommendations for the MAR. Compared with previous years, the definitions and criteria for these categories were sharpened through the application of results-chain thinking. The four jointly agreed categories to assess progress are (i) limited evidence of progress, (ii) emerging evidence of progress, (iii) change of direction, and (iv) progress constrained. To help management collect evidence, IEG suggested a results chain aligned with these categories for each recommendation. To help make decisions on the level of progress, two or three criteria were defined for each category. With these changes, IEG and management were generally better aligned on assessing progress. IEG and management agreed on the level of progress in 4 out of 5 recommendations. In 13 recommendations, IEG and management assessed progress differently. In 9 recommendations, IEG assessed evidence as showing a lower level of progress, and in 4 recommendations, IEG saw evidence of more progress than gauged by management. vii Overview Progress in Implementing Independent Evaluation Group Recommendations The Bank Group made steady progress in implementing IEG recommendations through delivering internal products and adapting processes; in some cases, it has achieved meaningful change of direction that shows that the outcomes of recommendations are being achieved. This validation assesses the evidence for all 22 IEG evaluations included in the MAR—that is, all evaluations reviewed by the Board Committee on Development Effectiveness between FY19 and FY22. These 22 evaluations contain 59 recommendations. Steady emerging evidence of progress was shown with the delivery of outputs in the recommendations of four out of five evaluations, such as putting guidance and processes into practice and providing training valued by staff. Evidence of outcomes— changes in behavior or systems—is far less prevalent than evidence of delivering new products or processes linked to recommendations. At the end of four years of MAR tracking, numerous evaluations show no change of direction for many of their recommendations. Recommendations related to climate and resilience and long-term economic growth showed important progress this year. Climate change, building resilience, and reversing the erosion of economic growth are critical areas targeted as part of the Bank Group’s evolution (World Bank 2023a). For example, the Bank Group has progressed to integrating renewable energy into clients’ power systems through core analytics, such as Country Climate and Development Reports, and lending operations. In addition, contributions to reversing the erosion of economic growth receive support through progress on recommendation 1 of the trade facilitation evaluation. The Bank Group has made changes so that trade reforms are systematically reinforced among advisory services and analytics, International Finance Corporation (IFC) advisory services, IFC investments, development policy financing, and investment project financing. The third recommendation from the evaluation of private investment in fragile and conflict-affected situations is not validated in this report by management or IEG. The recommendation focuses on IFC and the Multilateral Investment Guarantee Agency identifying and agreeing on targets specific to fragile and conflict-affected situations in their corporate scorecards (World Bank 2022c). In response to the evaluation, the Committee on Development Effectiveness encouraged IEG, IFC, and the Multilateral Investment Guarantee Agency to continue conversations regarding adopting a wider set of metrics. In the follow-up conversation, IFC and the Multilateral Investment Guarantee Agency stated that they disagreed with the recommendation; however, IEG considers this recommendation important. This change is highlighted in this report because it was not previously reported to the Committee on Development Effectiveness. viii Overview There continues to be important unfinished business on recommendations that are being automatically retired. In FY23, many recommendations lagged in achieving their outcomes after four years of MAR tracking. Of the five evaluations set to be automatically retired after four years of tracking, four would have remained active into next year. For example, the citizen engagement and carbon finance evaluations’ recommendations are areas of corporate priority, but still have not demonstrated a change of direction. Achieving a change of direction more frequently may require earlier adaptation of implementation, engagement with IEG, or a more thorough change in management’s approach to implementing some recommendations. The citizen engagement recommendations did not result in a change of direction because needed activities were not implemented early enough to achieve results. For example, as the MAR report shows, an enhanced analytical framework for citizen engagement project monitoring was developed and piloted in FY22, whereas expanded training modules that link citizen engagement, the Environmental and Social Framework, and stakeholder engagement plans were launched in FY23. Engagement with IEG can support identification of issues or the confirmation of progress on recommendations. For example, this year, IEG suggested a methodology for conducting a portfolio review related to health quality indicators, which helped to show progress in a manner not previously demonstrated in earlier reporting cycles. A fresh analysis is warranted when outcomes are unlikely to be met even with intensified efforts. For the outcome orientation evaluation, for example, completing the activities reported on by management with high-level outcomes to improve implementation is unlikely to address the evaluation’s findings on the limited use of the country-level results system for adaptive management. Retiring Recommendations The main retirement actions proposed by IEG are as follows: • Retire 14 recommendations; 10 recommendations are due for automatic retirement because they have been tracked for four years, 3 recommendations are proposed for early retirement by management and IEG, and 1 recommendation is assessed by IEG as suitable for early retirement that management did not. • One recommendation from the citizen engagement evaluation, on leveraging the Environmental and Social Framework, is set to be automatically retired at progress constrained; we anticipate future IEG evaluations will need to revisit this issue. ix Overview • IEG does not agree with management’s proposal to retire recommendation 3 from the urban resilience evaluation because no criteria for retirement have been met. • IEG agrees with management’s proposal to extend by a year the tracking of two recommendations, one in the citizen engagement and one the regional integration evaluations. The latter has demonstrated a change of direction, and management contends that one more year of reporting would help develop a track record of implementation. Suggestions for Further Enhancement of the Management Action Record and Validation Process The gap between delivering outputs linked to recommendations and achieving outcomes needs to be bridged more often. Many recommendations that show emerging evidence of progress do not advance to a change of direction over the four years of tracking. In some instances, additional management actions are needed for implementation. For other recommendations, a change in approach in working toward their intended outcomes is required, and this can be informed by evidence from IEG evaluations. A final consideration is that recommendations will not achieve their intended outcomes, and a fresh analysis is needed through management’s own review or one of IEG’s products. To tackle these issues, IEG suggests three areas for enhancing MAR effectiveness: • Clear processing of recommendations. Action, dialogue, and assessment of progress on recommendations are a shared responsibility between IEG and management that is hindered when outcomes and agreements are unclear. Management could review and adjust the consistency of how it agrees, disagrees, and partially agrees to recommendations. For example, management agrees to some recommendations with caveats, which may not be fully defined; thus, management considers the implementation more advanced than does IEG. Further, IEG is available to jointly review with management areas for improvement for evaluation recommendations—for example, their number and whether they are stated to best support progress toward intended outcomes. • Quicker implementation of recommendations. The suggestion by management to designate champions and define pathways to implementing recommendations could help the swift implementation of recommendations. IEG is available to work with management to jointly outline the form and function of champions to support the adoption of recommendations. The discussion could also include refinement of the criteria for their retirement or extension and consider the evolution of the Bank Group. x Overview • Constructive dialogue as part of the MAR process. The MAR provides a regular account that helps consolidate learning and adapt the implementation of recommendations to current operational realities. The FY23 MAR process’s constructive dialogue helped clarify recommendations, IEG’s expectations of evidence, and Bank Group teams’ implementation challenges. Early dialogue on recommendations—soon after evaluations are completed—would smooth implementation. xi 1. Introduction 1.1 This report provides the Independent Evaluation Group (IEG) validation of the World Bank Group’s report Learning and Adapting for Outcomes through the Management Action Record 2023: A World Bank Group Management Report on Implementation of IEG Recommendations for the period July 2022 to June 2023. The purpose of the Management Action Record (MAR) is to support accountability, learning, and adaptation for the Bank Group’s implementation of recommendations from IEG evaluations. The MAR is a key element of the Bank Group’s wider knowledge management framework. This validation document provides IEG’s assessment of the Bank Group’s progress on the agreed recommendations and takes a position on the evidence of their implementation, advancement, and retirement. As such, this is not an evaluation and does not provide IEG’s evidence and perspectives on the larger development issues covered in the MAR’s evaluations. The principal audiences for this validation are the Board’s Committee on Development Effectiveness (CODE) and the teams and managers in the Bank Group who implement changes informed by IEG recommendations. In this fourth validation since the MAR reform in 2020, IEG continues to adopt a forward-looking approach to assessing progress against the intended outcomes of evaluations, as embodied in their recommendations. 1.2 The structure of this validation is as follows: (i) the framework for the MAR and validation, (ii) progress toward intended outcomes of recommendations, (iii) the record of agreements and disagreements on progress for the 22 evaluations containing 59 recommendations in the MAR’s scope this year, and (iv) conclusions and suggestions on how to continue to improve the MAR’s effectiveness. 1 2. Validation Framework 2.1 IEG and Bank Group management agreed to an updated focused framework for the fiscal year (FY)23 MAR process. Building on recent years, the framework aims to deepen dialogue between IEG and Bank Group management on IEG recommendations. Apprised by CODE’s comments on the FY22 MAR process, the after-action review identified changes that would improve candor, consolidate the use of good evidence, and enhance engagement. To this end, the framework for FY23 establishes transparent criteria and definitions that strengthen the consistency of the MAR and validation process. The three elements of the framework are (i) a five-step process for conducting and validating the MAR, (ii) categories and criteria to assess progress against recommendations, and (iii) criteria for recommendation retirement. 2.2 The MAR is grounded in technical dialogue between IEG and management (figure 2.1). First, we conducted discussions among 22 IEG evaluators and more than 130 focal points—technical staff from operations and corporate units in charge of providing implementation evidence in their areas of work. This first dialogue clarified expectations and provided an initial discussion of progress toward implementing IEG recommendations. Second, the focal points collated evidence on implementation progress using a template defined by management. IEG-management exchange on technical evidence continued at the working level. Third, management produced the MAR report by reviewing and synthesizing the inputs they received from the technical focal points. IEG provided informal feedback on the contents and the technical evidence. Fourth, IEG conducted the validation of management’s MAR report and shared it with CODE for discussion in conjunction with the report. The final step in the annual MAR process is a planned after-action review, where management and IEG can reflect on the previous cycle and define areas for improvement. 2 Chapter 2 Validation Framework Figure 2.1. Management Action Record Process, Fiscal Year 2023 4. IEG MAR 1. Upstream 2. Evidence 3. MAR 5. After-action validation report engagements, “What collection by TFPs report review and CODE review good evidence looks like” 22 Meetings Aggregated between Report to IEG by MAR report and validation January–March 2023 March–June 2023 beginning of July 2023 to CODE by August 2023 Source: Independent Evaluation Group. Note: CODE = Committee on Development Effectiveness; IEG = Independent Evaluation Group; MAR = Management Action Record; TFP = technical focal point. 2.3 Categories with defined criteria enabled consistent assessment of progress toward recommendations’ intended outcomes (table 2.1). IEG and management used four categories to assess progress: (i) limited evidence of progress, (ii) emerging evidence of progress, (iii) change of direction of progress, and (iv) progress constrained (PC). Compared with previous years, the definitions and criteria for these categories were sharpened through results-chain thinking to better assess progress toward IEG recommendations’ intended outcomes. The PC category, which should be used judiciously, identifies recommendations that require intensified support and alerts management and CODE to delays in implementation. To help management collect evidence, IEG provided a results chain aligned with these categories for each recommendation. Table 2.1. Management Action Record Assessment Framework, Fiscal Year 2023 Category Criteria Progress constrained • No progress. • Valid evidence on progress is unavailable or inadequate. • Reporting in year three or four continues to show insufficient evidence of progress. Limited evidence: activities • Activities delivered and knowledge generated are linked to the recommendation. conducted • Skills developed are linked to the recommendation. • Limited new evidence of progress since the previous MAR. Emerging evidence: • Evidence of developed capacities or application or use of outputs—such as processes, demonstrating application information technology systems, and guidance implemented—is supported by limited of outputs  examples. • Anecdotal evidence of changes in behavior. 3 Chapter 2 Validation Framework Category Criteria Change of direction: • Meaningful change in behavior in the intended outcome of the recommendation that is demonstrating systematic likely to be sustained. behavior changes • Implemented systems changes, for example, incentives, financing mechanisms, processing, and new standards being applied across relevant portfolio. Source: Independent Evaluation Group. Note: MAR = Management Action Record. 2.4 Four “retirement” criteria were used. Recommendations can be retired, or exited, from MAR reporting when they meet any one of four criteria. The retirement criteria have evolved across recent MARs. This year’s MAR consolidated and applied all previously agreed criteria, resulting in the following criteria: • Time. The MAR reports on recommendations for up to five years, starting from the date of their review by CODE. Reporting usually takes place over four years, starting in the FY after an IEG evaluation was discussed by CODE. • Change of direction. Management achieves either a change in behavior or systemic change for the intended outcome of the recommendation. • Change in external context. The outcome of the recommendation is no longer considered relevant, for example, through force majeure or a refinement of global development priorities requiring a change in Bank Group strategy. • Change in internal context. Management can make no further progress on or no longer agrees with the intended outcome of a recommendation, or IEG has conducted a fresh evaluation that supersedes the earlier recommendation. 2.5 Retirements are always subject to CODE’s guidance. Especially when IEG and management disagree on whether to retire or continue tracking a recommendation, CODE provides guidance on the final decision. 2.6 The MAR’s approach had some limitations. The consistent application of the jointly agreed elements outlined above—expanded and enhanced technical dialogue, refined progress categories with defined and clarified criteria, and consolidated retirement criteria—brought clear strengths and enabled an approach shared by IEG and management. However, some challenges emerged when implementing the MAR framework: • Turnover of technical focal points. The MAR generally provides better quality evidence when focal points have experience with the process and understand the requirements of recommendations. Focal points, however, can change between years; consequently, knowledge of the process and progress on 4 Chapter 2 Validation Framework recommendations need to be rebuilt. This is mitigated by maintaining the evidence base and the focal points who remain in their roles. • The use of mainly secondary evidence. Technical focal points collate and summarize existing evidence and may conduct exercises such as portfolio reviews. This sometimes limits the depth of evidence but enables timely completion of the MAR. Coordination on recommendations across units. Most recommendations require coordination across the Bank Group to instigate change. If these units do not regularly work together or there is no defined lead unit, it can take time to organize roles and responsibilities. Defined champions at the management level may help coordinate the implementation of recommendations and subsequent reporting of evidence. 5 3. Progress toward Achieving Intended Outcomes of Recommendations 3.1 The Bank Group made steady progress in implementing IEG recommendations through delivering internal products and adapting processes, but only rarely has it achieved progress that shows a change of direction. The recommendations of four out of five evaluations delivered outputs, such as putting guidance into practice and providing training valued by staff. Table 3.1 shows that even in mature evaluation recommendations, evidence of a change of direction is far less prevalent than evidence of delivering new products or processes. At the end of four years of MAR follow-up, a substantial number of evaluations still do not report changes in direction across all their recommendations. Table 3.1. Independent Evaluation Group Assessment of Implementation Progress of Recommendations, by Fiscal Year of CODE Discussion FY Discussed Change of Emerging Limited Progress by CODE Direction Evidence Evidence Constrained Total FY19 2 9 — 1 12 FY20 2 5 — — 7 FY21 2 11 3 — 16 FY22 — 9 12 3 24 Total 6 34 15 4 59 Source: Independent Evaluation Group. Note: CODE = Committee on Development Effectiveness; FY = fiscal year; — = not available. Quality of Evidence 3.2 The identification of progress in management’s MAR report improved with the implementation of the shared assessment framework described in chapter 2. The MAR report draws on a reasonable evidence base, including limited portfolio reviews, analysis of financial commitments, examples of practice, and documents. The report’s evidence describes change according to a results-chain logic. For example, a progress rating of emerging evidence indicates how changes made by management have led to the enhanced quality of products. Detailing evidence in this way improves the clarity of reporting progress. For example, management assigned limited evidence to 19 percent of recommendations in FY23, whereas the category was applied to only 2 percent in FY22. 3.3 The refined assessment framework used in the identification of progress helps flag challenges management faces when implementing recommendations. In line with CODE’s request of the Bank Group to establish an early-warning system for 6 Chapter 3 Progress toward Achieving Intended Outcomes of Recommendations recommendations, management applied the PC flag for three recommendations. This system helps candidly highlight implementation challenges. Using the updated framework for progress, IEG and management agreed on the level of progress in four out of five recommendations. In 13 recommendations, IEG and management assessed progress differently. In 9 recommendations, IEG determined evidence of a lower level of progress, and in 4 recommendations, IEG saw evidence of more progress than assessed by management. The openness of management to identify areas for adaptation in the implementation of recommendations provides a sound basis for optimizing learning and ensuring that the MAR tool is perceived as a space for task teams to seek support. Progress toward Outcomes 3.4 Implementation advanced for most recommendations tracked in both FY22 and FY23. Fifteen recommendations saw an upgrade in their assessment by IEG from FY22 to FY23. Nine recommendations moved up from limited evidence to emerging evidence. Five recommendations advanced from emerging evidence to change of direction, and one recommendation progressed from limited evidence to change of direction. The citizen engagement and mobilizing technology for development evaluations were the only evaluations whose recommendations received a lower assessment of progress in FY23 compared with FY22. In citizen engagement, one recommendation was flagged as PC. In mobilizing technology for development, one recommendation was downgraded from emerging evidence to limited evidence because of the rapidly changing context that requires coordinated and routine engagement with disruptive technology. 3.5 Recommendations related to climate and resilience and long-term economic growth showed important progress this year. Climate change, building resilience, and reversing the erosion of economic growth are critical areas targeted as part of the Bank Group’s evolution (World Bank 2023a). The World Bank has developed tracking systems and diagnostics to better analyze progress toward building the resilience of cities and considers resilience in its urban infrastructure work, as recommended by the urban resilience evaluation. Similarly, the Bank Group has progressed to integrating renewable energy into clients’ power systems through core analytics, such as the Country Climate and Development Reports (CCDRs), and lending operations. Contributions to reversing the erosion of economic growth receive support through progress on recommendation 1 of the trade facilitation evaluation. The Bank Group has made changes so that trade reforms are systematically reinforced among advisory services and analytics (ASA), International Finance Corporation (IFC) advisory services, IFC investments, development policy financing, and investment project financing. Further, systematic changes have been implemented in the 20th Replenishment of the International 7 Chapter 3 Progress toward Achieving Intended Outcomes of Recommendations Development Association (IDA) Regional Policy Window to improve the assessment of cross-border spillover effects, as recommended by the regional integration evaluation. 3.6 A change of direction can be realized after two or three years by gradually working toward implementing recommendations and acting on issues that have an elevated corporate focus. The renewable energy and urban resilience evaluations both have recommendations that show a change of direction, respectively, in their second and third years of tracking. For the urban resilience evaluation, management showed gradual progress year by year by implementing the Resilience Rating System. The increasing focus across the Bank Group on renewable energy also provided an opportunity to draw on evidence that enables two recommendations to be retired this year. 3.7 The implementation of many recommendations continues to lag in achieving their intended outcomes after five FYs of implementation. Previously in FY22, four of the nine evaluations automatically retired saw notable delays in implementing their recommendations. Likewise, this year, multiple recommendations lag in their implementation in their final year of tracking. Of the seven evaluations IEG completed in FY19, four of these would remain active next year if not for automatic retirement. For example, although the citizen engagement and carbon finance recommendations are linked to corporate priorities, they still have not demonstrated a change of direction. The citizen engagement evaluation recommended that the Environmental and Social Framework (ESF) leverage citizen engagement mechanisms, but progress on this has lagged over several years and is flagged as PC this year. Moreover, although there was progress on implementing the health services and trade facilitation evaluations’ recommendations, the evidence did not yet show systemic changes in practice for conducting diagnostics and monitoring. The changing context for carbon finance required the World Bank to adapt its approach and indicates emerging evidence in progress (box 3.1). 3.8 Earlier adaptations in implementation of recommendations and engagement with IEG could facilitate changes in direction in evaluations due for automatic retirement. The citizen engagement recommendations did not result in a change of direction because needed activities were implemented late. For example, as the MAR report shows, an enhanced analytical framework for citizen engagement project monitoring was developed and piloted in FY22, whereas expanded training modules that link citizen engagement, ESF, and stakeholder engagement plans were launched in FY23. Health services recommendation 1 on health services quality was assessed to have reached emerging evidence of progress, with a portfolio review of indicators related to 8 Chapter 3 Progress toward Achieving Intended Outcomes of Recommendations quality showing progress in FY23. Conducting portfolio reviews for previous MAR reports may have helped demonstrate sustained change over time. Engagement between IEG and management this year was helpful in both these cases. IEG and the Citizen Engagement and Social Accountability Global Solutions Group together reviewed indicators, whereas for the health services evaluation, IEG suggested a methodology for conducting a portfolio review related to health quality indicators. Box 3.1. The Implementation of Carbon Finance Recommendations: Responding to External Change The World Bank is operating in a rapidly changing external environment where corporate action far outpaces government action related to carbon markets, and there is a lack of clarity on what constitutes integrity and high quality. Although private voluntary markets have grown rapidly, the bottom-up framework introduced by the Paris Agreement implies that to participate in carbon markets, countries must increase preparation significantly. They need to build the infrastructure and institutional mechanisms required to engage in compliance markets and develop pricing strategies that fully consider the opportunity cost of implementing nationally determined contributions to the Paris Agreement. It remains to be seen how soon countries complete such readiness activities. The World Bank’s programs have responded to countries’ needs while recognizing that carbon markets are one tool in a country’s climate policy mix and are not expected to form the sole strategy for achieving climate goals. The World Bank’s efforts in supporting carbon markets have focused primarily on creating the necessary enabling environment for market participants and playing a facilitative role, including using and scaling up carbon finance instruments to attract and mobilize finance that supports transformational activities and leverages private investments, which is the focus of the Independent Evaluation Group recommendation. Sources: Independent Evaluation Group; Vives 2023. 3.9 Management could consider changing their approach to implementing some recommendations or conducting a fresh analysis when outcomes are unlikely to be met, even with intensified efforts. For the outcome orientation evaluation, for example, completing the activities reported on by management is unlikely to achieve the outcomes envisioned in the recommendation. Implementation and reporting currently focus on including high-level outcomes in new Country Partnership Frameworks (CPFs). However, the use of high-level outcomes is unlikely to address the evaluation findings on the limited use of the country-level results system for adaptive management. In their evaluation response, management committed to develop complementary approaches to monitoring, evaluation, and learning to be explored and piloted, but reporting has not discussed these issues. Management could consider a change in approach to take new actions to improve the utility of the country-level results system for country teams. 9 Chapter 3 Progress toward Achieving Intended Outcomes of Recommendations 3.10 Attention is needed in nine evaluations that show limited evidence in the first two years of tracking (table 3.2). In these cases, there are gaps in demonstrating whether recommendations’ implementation activities have yet led to outputs. It is urgent to enhance the implementation of recommendations for evaluations that entered the MAR system in FY21; these are at the midterm of their tracking and should be targeting a change of direction. For FY22 evaluations, it is expected that progress can start at limited evidence and then advance. The Bank Group can resolve the limited evidence of these evaluation recommendations through follow-up on existing implementation activities or plans. 3.11 IEG assesses progress as constrained in two evaluations completed in FY22. In the case of the Doing Business evaluation, both IEG and management assessed two recommendations as PC. The first Business Ready report, which replaces Doing Business, is expected in March–April 2024. Progress on two lessons related to the treatment of indicators relies on the behaviors of various Bank Group units and the guidance for global indicator processes, not just the unit in charge of implementing the recommendations. For Sustainable Development Finance Policy, IEG assessed progress as constrained on recommendation 1, related to expanding the countries covered by the Debt Sustainability Enhancement Program. Review processes have been updated, and changes are planned. However, importantly, management has not applied an additional filter that can expand coverage of the program, which is the key aspect of the recommendation. Table 3.2. Evaluations from Fiscal Years 2021 and 2022 Showing Limited Evidence of Progress Year Evaluation and Recommendation Management IEG Considerations FY21 Mobilizing Technology for Development: An Assessment EE LE A systemic approach to of World Bank Group Preparedness disruptive technologies is Recommendation 1: The World Bank Group avails itself yet to emerge, and it is of opportunities and addresses risks posed by becoming more urgent in disruptive technologies. Recommendation 2: Build skills the context of increased to harness disruptive technology opportunities and technological risk. mitigate risks. World Bank Support for Public Financial and Debt LE LE There is no evidence yet that Management in IDA-Eligible Countries changes in guidance and Recommendation 2: Prioritize and sequence World process helped develop Bank support to public financial and debt capacity or enable management. sequenced reforms. FY22 The Development Effectiveness of the Use of Doing LE, PC LE, PC Other units of the Bank Business Indicators, Fiscal Years 2010–20 Group and global indicator Lesson 2: Avoid indicators as reform objectives. Lesson 4: guidance affect the ability to Develop mechanisms and safeguards for Bank Group realize lessons; mechanisms reports and communication.a 10 Chapter 3 Progress toward Achieving Intended Outcomes of Recommendations Year Evaluation and Recommendation Management IEG Considerations to help manage this risk will need to be defined. Enhancing the Effectiveness of the World Bank’s Global EE LE Demonstrate the Footprint applications of new Recommendation 1: Define expected outcomes of guidance, measurement decentralization. processes, and tools. World Bank Support to Reducing Child Undernutrition LE LE Demonstrate the Recommendation 1: Adjust nutrition programming. applications of new Recommendation 2: Strengthen nutrition support in guidance, measurement Global Practices. processes, and tools. Managing Urban Spatial Growth: World Bank Support to LE LE Demonstrate the Land Administration, Planning, and Development applications of new Recommendation 2: Use preventive approaches. guidance, measurement Recommendation 3: Collect location and land market processes, and tools. data. The International Finance Corporation’s and Multilateral EE LE Provide evidence on new Investment Guarantee Agency’s Support for Private initiatives undertaken since Investment in Fragile and Conflict-Affected Situations, the evaluation. Fiscal Years 2010–21 Recommendation 1: Review financial risk and enhance capabilities to address nonfinancial risk. Recommendation 2: Recalibrate business models, client engagements, and instruments to continuously adapt them to the needs and circumstances of fragile and conflict-affected situations. The International Development Association’s Sustainable EE, LE, LE PC, LE, Demonstrate that planned Development Finance Policy LE actions have been Recommendation 1: Expand the countries covered by implemented. the Debt Sustainability Enhancement Program. Recommendation 2: Ensure that performance and policy actions emanate from an up-to-date assessment of country-specific debt stress. Recommendation 3: Take performance and policy actions that aim for long- lasting institutional reforms rather than relying on one- time actions. World Bank Engagement in Situations of Conflict EE LE Demonstrate the Recommendation 3: Address factors that dissuade applications of new World Bank engagement. guidance, measurement processes, and tools. Sources: Independent Evaluation Group; World Bank 2021a, 2021b, 2021c, 2021e, 2021f, 2021g, 2022a, 2022b, 2022c. Note: EE = emerging evidence of a change in the direction of travel; IEG = Independent Evaluation Group; LE = limited evidence of a change in the direction of travel; PC = progress constrained. a. Due to the suspension of the Doing Business report, IEG updated the evaluation with lessons that draw out issues to be considered by the follow-on project. In its report to the Board, the Committee on Development Effectiveness noted that although the Doing Business flagship report was discontinued, enabling a business environment remains a relevant task for the World Bank and the International Finance Corporation. The Committee on Development Effectiveness further noted that important lessons could be drawn from the rich and high-quality analysis in the IEG evaluation to shape the strategic focus and design of the new approach. 11 Chapter 3 Progress toward Achieving Intended Outcomes of Recommendations Retirement 3.12 IEG proposes retiring 14 recommendations. Of the 12 recommendations that are due for automatic retirement at the end of four reporting cycles, IEG agrees with retiring 10 recommendations. IEG concurs with management’s proposal to extend tracking of 2 final-year recommendations. IEG also agrees with the early retirement of 3 recommendations proposed by management and proposes retiring an additional recommendation. Table 3.3 shows IEG’s justifications for recommendations proposed for retirement. 3.13 IEG agrees with extending MAR tracking of the citizen engagement and regional integration evaluations, which would otherwise be retired, having completed four years of tracking. On regional integration, IEG agrees with management’s proposal to continue tracking the remaining recommendation, related to regional spillover effects. Although the change of direction has been achieved, management contends that it would be beneficial to show continued progress on this corporate priority through additional narrow reporting in FY24. On citizen engagement, IEG agrees with management’s proposal to continue tracking recommendation 3, which relates to strengthened monitoring. The citizen engagement evaluation’s recommendation 4, focused on the ESF leveraging citizen engagement mechanisms (World Bank 2018b), has been flagged as PC. IEG agrees to retire this recommendation, even though it is marked as PC, because IEG will follow up on this issue through the upcoming ESF evaluation in IEG’s work program. IEG is also content to retire health services recommendation 1, focused on measuring the quality of health care (World Bank 2018c), and notes that IEG will follow up on this issue in upcoming evaluations. 3.14 IEG does not agree with retiring one recommendation proposed by management. Management proposes retiring the urban resilience evaluation’s recommendation 3, which focuses on crime and violence as a resilience risk (World Bank 2019a), and assessed progress as PC. Management suggests retirement because this recommendation was only partially agreed, and its retirement would allow sharper focus on several other recommendations on urban issues. IEG finds that none of the established four criteria for retirement (see chapter 2) has been met in this instance. 12 Chapter 3 Progress toward Achieving Intended Outcomes of Recommendations Table 3.3. Recommendations Proposed for Retirement (Management Action Record 2023) Level of Evidence Recommendations Reason for for Retiring Theme IEG Evaluation to Retire (n = 14) Retirement Recommendations Human World Bank Group Support to Health Recommendations 1 Time Both EE Capital Services: Achievements and Challenges and 3 Climate and Carbon Markets for Greenhouse Gas Recommendation 2 Time EE Resilience Emission Reduction in a Warming World: An Evaluation of the World Bank Group’s Support to Carbon Finance Climate and Building Urban Resilience: An Recommendation 1 IEG and Both CD Resilience Evaluation of the World Bank Group’s management Evolving Experience (2007–17) agree to a CD assessment. Recommendation 5 IEG proposes CD. Climate and Renewable Energy: Evaluation of the Recommendations 1 IEG and Both CD Resilience World Bank Group’s Support for and 3 management Electricity Supply from Renewable agree to a CD Energy Resources, 2000–2017 assessment. Long-Term Grow with the Flow: An Evaluation of Recommendations 1, Time Recommendation 1: Economic World Bank Group Support to 2, and 3 CD; Growth Facilitating Trade 2006–17 Recommendations 2 and 3: EE Improving Engaging Citizens for Better Recommendations 1, Time Recommendations 1, World Bank Development Results 2, 4, and 5 2, and 5: EE; Group Recommendation 4: Capacity PC Sources: Independent Evaluation Group; World Bank 2018a, 2018b, 2018c, 2019a, 2019b, 2020a. Note: CD = evidence substantiates a change of direction of travel, with reversals unlikely; EE = emerging evidence of a change in the direction of travel; IEG = Independent Evaluation Group; PC = progress constrained. 13 4. Progress on the Implementation of Recommendations, by Evaluation 4.1 This chapter contains IEG’s validation of implementation progress for all evaluations and recommendations in this year’s MAR. Investing in Human Capital Table 4.1. Health Services, Fiscal Years 2019–23 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 1: Improve measurement of the quality of EE EE CD EE health services and the distributional effects of health service projects. Recommendation 3: For sustainable capacity to address CD EE EE EE pandemics and systematically integrate preparedness plans and governance frameworks for pandemic control within the client country’s own health system in World Bank Group–financed projects and advisory services. Sources: Independent Evaluation Group; World Bank 2018c. Note: CD = evidence substantiates a change of direction of travel, with reversals unlikely; EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group. 4.2 The quality of health service measurement is developing, and further tracking in upcoming IEG evaluations is suggested to provide evidence that changes in measurement practice are maintained (recommendation 1; World Bank 2018c). The intent of this recommendation is to improve how the World Bank supports clients to understand and respond to challenges in health service quality.1 A review by the World Bank showed that all projects focusing on improving the quality of health services had at least one indicator on both health service quality and distributional effects. The incorporation of these indicators was facilitated by the training of peer reviewers and a module focused on quality indicators integrated into the flagship learning program by the Health, Nutrition, and Population Global Practice (GP). Further, work undertaken through the Primary Health Care Performance Initiative in 20 countries helps develop capacities to assess health service quality. However, there is a need for further evidence that a systemic change has taken place on the extent of indicators’ measurement of the different dimensions of quality (structural, process, and outcome) and the distributional effects. Considering there is no other mechanism for reporting to CODE, IEG suggests that Bank Group efforts on health service quality be assessed as part of the upcoming evaluations. Further review of progress in this upcoming evaluation will help IEG and CODE understand whether World Bank projects track all three dimensions of health quality—which is essential for advancing toward this recommendation’s intended 14 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation outcome—and whether the proportion of indicators measuring process quality and outcome quality has increased over time. 4.3 The World Bank is expanding its capacity for pandemic preparedness and response, and further tracking of progress on this unmet recommendation will be based on the recommendations of IEG’s COVID-19 response evaluations (recommendation 3). Building on the evidence from last year, the COVID-19 response has been shown to help restructure health systems and develop pandemic preparedness and response capacity at the country level. For example, the World Bank has reported supporting 37 IDA countries in implementation of their pandemic preparedness plans. Moreover, the focus on pandemic preparedness is likely to be maintained through the Pandemic Fund, Health Emergency Preparedness and Response Umbrella Program, and continued IDA and International Bank for Reconstruction and Development financing. Progress on pandemic preparedness will also continue to be tracked through follow-up on the recommendations of the recent IEG COVID-19 response evaluations (World Bank 2022e, 2023b). To help verify progress outside of MAR tracking, there is an IEG evaluation planned for FY26 on pandemic preparedness and One Health. This can provide further analysis of recommendation 3 issues if needed. Table 4.2. Undernutrition, Fiscal Years 2022–26 Management IEG Recommendation FY23 FY23 Recommendation 1: Adjust nutrition programming in country portfolios to (i) give LE LE more priority to institutional strengthening for coordination and implementation of multisectoral nutrition interventions and (ii) increase focus on subnational targeting of interventions to reflect areas of greatest disadvantage and persistency of need. Recommendation 2: Strengthen nutrition support in GPs to (i) rebalance LE LE investments to have greater emphasis on nutrition-specific interventions and (ii) increase focus on social norms, interventions, and behavior changes, with more attention to tracking expected achievements to improve nutrition determinants. Sources: Independent Evaluation Group; World Bank 2021g. Note: FY = fiscal year; GP = Global Practice; IEG = Independent Evaluation Group; LE = limited evidence of a change in the direction of travel. 4.4 To adjust and strengthen nutrition support, the World Bank has undertaken activities to fine-tune new projects, reviewed the portfolio of nutrition projects, developed tools, and shared knowledge (recommendations 1 and 2; World Bank 2021g). These activities are appropriate building blocks in the first year of tracking these recommendations. Management’s review of project financing showed that the nutrition portfolio has grown from a total International Bank for Reconstruction and Development and IDA commitment of $312 million in FY20 to an estimated $937 million in FY23; however, nutrition-sensitive interventions tailed off in FY23. Artificial intelligence–based tools are planned to be used in FY24 to allow for a more granular 15 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation portfolio assessment that can identify support for social norms and behavioral change in nutrition interventions. In addition, the World Bank is disseminating knowledge and evidence through various initiatives (including webinars, brown-bag lunches, articles, and Nutrition Financing Week and other events), some of which address aspects of gender and behavioral change. Moreover, the Nutrition Global Solutions Group provides continued support to country operations and ASA in subnational targeting through the Optima Nutrition learning tool. IEG looks forward to evidence on the results of these activities in the next MAR. IEG also anticipates greater depth of evidence on institutional strengthening and engagement of social norms change. Promoting Climate and Resilience Table 4.3. Carbon Finance, Fiscal Years 2019–23 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 2: The World Bank Group should increase its EE LE EE EE use of carbon finance instruments to attract and mobilize finance that supports transformational activities and leverages private investments. Sources: Independent Evaluation Group; World Bank 2018a. Note: EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group; LE = limited evidence of a change in the direction of travel. 4.5 The World Bank has a maturing approach to carbon finance instruments that seeks to navigate changing conditions in carbon markets and increase its leverage through private investments.2 In responding to a changing context, the World Bank identified and started to tackle challenges in the functioning of carbon markets, using two carbon finance and pricing trust funds. The Partnership for Market Implementation Facility is undertaking ASAs, such as the Climate Warehouse Phase II, which, in reference to the 2015 Paris Agreement, seeks to support the establishment of infrastructure, policies, and institutional arrangements to ensure greater transparency and reduce the risk of double counting emission reductions. The Scaling Climate Action by Lowering Emissions trust fund aims to provide results-based financing for emissions reduction credits that it defines as transformative. The evidence provided also shows the implementation of emission-linked bonds, trust funds, implemented guidance, and the development of country capacity and new instruments. There is not yet evidence of systematic processes that will leverage private investment and support transformational activities to contribute to Paris Agreement results. This recommendation will be retired because it is in its final year of reporting. IEG anticipates Board attention through the World Bank evolution process, where IEG suggests that management reviews assess 16 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation whether the existing carbon finance instruments effectively leverage private sector investments at the scale, scope, and speed needed for transformational change. Table 4.4. Urban Resilience, Fiscal Years 2020–24 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 1: The Bank Group should systematically EE EE CD CD identify and track progress of interventions that build urban resilience to chronic stresses and acute shocks, across its institutions. Recommendation 2: The design and implementation of World CD LE EE EE Bank projects that build urban resilience should systematically incorporate resilience characteristics and articulate their application throughout the project cycle. These should include the following: (i) design standards in line with resilience risks, (ii) cost-benefit analysis in line with resilience risks, (iii) city and interjurisdictional coordination, and (iv) inclusive approaches for vulnerable people. Recommendation 3: In urban areas where the client has EE LE PC EE identified crime and violence as a resilience risk, the World Bank’s support should be based on a localized typology of crime and violence that is informed by relevant analytic work. This approach should be supported by an assessment of the mechanisms most effective at reducing crime and violence within operations. Recommendation 5: IFC should support its public and private EE EE EE CD sector Cities Initiative clients through available resilience risk assessment and mitigation tools to strengthen development impacts. Sources: Independent Evaluation Group; World Bank 2019a. Note: CD = evidence substantiates a change of direction of travel, with reversals unlikely; EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group; IFC = International Finance Corporation; LE = limited evidence of a change in the direction of travel; PC = progress constrained. 4.6 The Bank Group has developed systematic tracking systems and diagnostics to better analyze progress toward building the resilience of cities (recommendation 1; World Bank 2019a). The World Bank seeks to contribute to enhanced urban resilience through the Resilience Rating System, which helps integrate resilience considerations into CCDRs, climate co-benefits into urban financing operations, and urban green spaces and nature-based solutions into projects. The system’s dual-track approach supports monitoring and measuring of both how interventions plan to enhance resilience and how resilience outcomes manifest. The Water and Transport GPs have also integrated urban resilience into project design. IFC has improved the identification and tracking of resilience outcomes in its projects. Based on these Bank Group–wide changes, IEG agrees with the retirement of this recommendation. 17 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation 4.7 The World Bank incorporates resilience characteristics throughout the project cycle, with a focus on design standards and cost-benefit analysis (recommendation 2). Grant financing, frameworks, and diagnostic tools are used to promote and track urban resilience initiatives. For example, transport and infrastructure investments increasingly incorporate resilience characteristics throughout the project cycle. Project teams use cost- benefit and economic analyses to prioritize urban resilience—enhancing infrastructure investments, ensuring that key objectives, such as risk reduction, are met. Progress toward a change of direction assessment could be shown in the next MAR, if systemic change is demonstrated in how urban projects incorporate in their design and implementation interjurisdictional coordination and inclusive approaches to vulnerable people. 4.8 IFC’s Cities Initiative has demonstrated systematic change in its approach to urban resilience through the ongoing development of tools, refinement of its theory of change, inclusion of resilience design standards in its urban infrastructure work, and articulation of impacts (recommendation 5). IFC’s Cities Initiative integrates resilience aspects into project design and its theory of change. For example, the Kerala Infrastructure Investment Fund Board Green Capacity Building project has been designed to integrate climate resilience into new infrastructure assets, reduce investment risk, minimize damage, and extend operating life. Moreover, the Building Resilience Index program, a web-based hazard mapping and resilience assessment framework for the building sector, has grown from being a pilot to being applied to about 2 million square meters of building space in the Philippines and 100,000 square meters in the Dominican Republic, and expanded to Vietnam in late FY23. 4.9 Evidence shows that the World Bank is working with clients to address crime and violence as a priority risk through analysis (recommendation 3). The recent CPF for Brazil includes important commitments to gender-based violence work. The FY22 Honduras CPF included a governance and violence lens for new lending operations. Further, in Tajikistan and Bangladesh, as well as West Africa, crime and violence issues have also been addressed. Moreover, the fragility, conflict, and violence (FCV) Mid-Term Review reinforces the importance of ending crime and violence and recommends the expansion of analytical work on urban and criminal violence. IEG suggests that in the future, management further identify and describe the results of these analytical initiatives from across the Bank Group that help address crime and violence issues. Management proposes retiring this recommendation because it is partially agreed, and its retirement would allow sharper focus on several other recommendations on urban issues. IEG proposes that the recommendation continue to be tracked—the evidence shows that the 18 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation World Bank continues to contribute to localized crime typologies through its work on urban crime, and no criteria for retirement have been met. Table 4.5. Renewable Energy, Fiscal Years 2021–25 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 1: The World Bank Group to prioritize EE EE CD CD interventions that focus on the integration of renewable energy sources into the power systems of client countries, to facilitate progress in their clean energy transitions. Recommendation 2: The Bank Group to support renewable energy EE EE EE EE scale-up through comprehensive, long-term country engagements, with coordinated Bank Group solutions, based on the comparative advantages of each institution, to address barriers, aided by robust upstream diagnostics. Recommendation 3: The Bank Group to continually upgrade the EE EE CD CD pool of specialized skills to help clients address their pressing and rapidly evolving challenges to scale up renewable energy. Sources: Independent Evaluation Group; World Bank 2020a. Note: CD = evidence substantiates a change of direction of travel, with reversals unlikely; EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group. 4.10 The integration of renewable energy into clients’ power systems is being prioritized by fully embedding it in core analytics (such as the CCDRs) and lending operations, as well as through corporate commitments (recommendation 1; World Bank 2020a). The evidence highlights that the Bank Group addresses barriers to renewable energy integration in operations through grid strengthening, system planning, and storage investments. CCDRs support these processes by identifying priorities for the integration of renewable energy in 29 client countries. IFC promotes renewable energy integration through climate finance, hybrid renewable energy projects, and battery storage initiatives. The Renewable Energy Catalyst Trust Fund from the Multilateral Investment Guarantee Agency (MIGA) provides political risk insurance and risk management instruments to mobilize finance for renewable energy projects. The integration of renewable energy into power systems, with a focus on network infrastructure, will remain a corporate priority given policy commitments outlined in the 20th Replenishment of IDA. 4.11 The Bank Group institutions are coordinating their action on renewable energy (recommendation 2). Based on the comparative advantages of each institution, the World Bank, IFC, and MIGA have been working together in countries such as Niger, Togo, and Uzbekistan on the Scaling Solar program. Other collaborative initiatives include the Offshore Wind Development Program and the Distributed Access through Renewable Energy Scale-Up Platform. The extent to which this coordination is sustained 19 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation and expanded in a defined system would be a useful area for further evidence in the next MAR. 4.12 The Bank Group is expanding renewable energy skills by increasing the pool of staff with specialized knowledge (recommendation 3). IFC is restructuring the teams in line with renewable energy priorities, and all institutions are constantly fostering staff’s knowledge in this area. The World Bank and IFC have expanded staff skills in renewable energy through additional hiring and training. In the World Bank, 28 professionals with renewable energy skills have been hired. More than 50 delegates from 15 countries joined the World Bank’s Offshore Wind Study Tour, and more than 70 participants from 12 countries and four international and regional organizations joined a two-day hydropower workshop. Table 4.6. Natural Resource Degradation and Vulnerability, Fiscal Years 2021–25 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 1: The World Bank should identify and analyze EE LE EE EE natural resource degradation and vulnerability nexus issues and leverage this knowledge in Systematic Country Diagnostics and in country engagements where such issues matter for achieving sustainable poverty reduction and shared prosperity. Recommendation 2: World Bank operations that address natural EE LE EE EE resource degradation should direct attention to resource governance challenges and use a mix of resource management practices and financial incentives appropriate for the relevant socioecological systems. Recommendation 3: World Bank Global Practices involved in EE EE LE EE addressing natural resource degradation and associated vulnerability should share knowledge, improve measurement, and enhance coordination in the design and implementation of their projects to optimize development effectiveness. Sources: Independent Evaluation Group; World Bank 2021d. Note: EE = emerging evidence of a change in the direction of travel; FY = fiscal year; LE = limited evidence of a change in the direction of travel; IEG = Independent Evaluation Group. 4.13 The World Bank integrated natural resource degradation and vulnerability (NRDV) nexus issues through a variety of analytical, knowledge sharing, and lending instruments in country engagements and operations (recommendations 1, 2, and 3; World Bank 2021d). The evidence highlights the establishment of processes to identify and address NRDV nexus issues derived from global databases, such as the Changing Wealth of Nations and the Knowledge, Information & Data Services Helpdesk. These data sources are used by task teams in considering NRDV issues in diagnostic reports and country partnership documents, such as CPFs. The World Bank uses various instruments and trust funds that support analytical work and coordination in 20 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation operations. Cooperation arrangements and knowledge-sharing initiatives are ongoing between the Environment, Natural Resources, and Blue Economy GP and other GPs. IEG concurs with management on the importance of establishing more structured collaboration in coming years. For the FY24 MAR, evidence on deeper collaboration and the results of process changes to strengthen NRDV would be useful, especially evidence showing how measurement and reporting of resources- and vulnerability-related outcomes of different approaches are being improved. Table 4.7. World Bank Engagement in Situations of Conflict, Fiscal Years 2022–26 Management IEG Recommendation FY23 FY23 Recommendation 1: To enhance the conflict sensitivity of World Bank EE EE engagement, ensure that politically sensitive, confidential analysis is generated, retained, and managed so that it can be used by select future staff working on that country. Recommendation 2: Ensure that country engagements are informed by timely EE EE analyses of conflict dynamics and risks. Recommendation 3: Address factors that dissuade World Bank engagement in EE LE conflict-affected areas. Recommendation 4: In conflict-affected countries, rethink what success looks LE EE like. Sources: Independent Evaluation Group; World Bank 2021e. Note: EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group; LE = limited evidence of a change in the direction of travel. 4.14 Conflict analysis and risk monitoring is conducted by the World Bank through assessments, monitoring, briefings, and portfolio analyses that are available to staff, feed into country engagements, and help define success (recommendations 1, 2, and 4; World Bank 2021e). Risk and Resilience Assessments serve as a diagnostic tool to identify FCV drivers and resilience factors and help define success in country engagement and programming. The assessments have been integrated into key country engagement documents, such as Systematic Country Diagnostics, CPFs, Country Engagement Notes, and Performance and Learning Reviews. Risk and Resilience Assessments (17 full assessments and two summary notes) are available to staff on the Bank Group intranet. Further, the use of various tools for FCV risk monitoring and analytics—including crisis risk monitoring, conflict tracking activities, FCV portfolio analysis, and FCV project lenses—contributes to the understanding of conflict dynamics and risks. In Myanmar, the Country Management Unit uses regular briefings to better understand the FCV dynamics and their implications for World Bank engagements. To help adapt, Completion and Learning Reviews have been used to draw lessons that inform future country programming. Future MAR reporting could include examples of how the 21 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation systems in place helped key staff working on FCV issues in country and also how the tools described have been further developed. 4.15 The World Bank has introduced changes to remove constraints for engagement in conflict-affected areas, but further evidence of their quality and application in FCV countries would be useful (recommendation 3). The World Bank has updated policies, rolled out flexibilities and introduced tools, invested in third-party implementation and monitoring, increased staffing, and provided new support for staff security and well- being. There is also new guidance on security considerations in the project cycle that clarifies the security coverage. Future evidence could provide examples of how the processes established since the evaluation have helped relieve constraints in FCV contexts. Table 4.8. Municipal Solid Waste Management, Fiscal Years 2022–26 Management IEG Recommendation FY23 FY23 Recommendation 1: To achieve more sustainable and scalable outcomes in EE EE municipal waste management, World Bank Group technical and financial support to clients should give clear priority to the adoption and implementation of waste hierarchy practices, in line with client needs and capabilities for MSWM. Recommendation 2: To support the low-income countries where municipal solid EE EE waste is growing most rapidly, the Bank Group should identify constraints on demand and investments and leverage external partnerships to implement context- specific MSWM solutions. Recommendation 3: To bring prominence to and spur action on the global EE EE municipal solid waste agenda, the Bank Group should take up a clear leadership position, collaborating and convening with developmental partners. Sources: Independent Evaluation Group; World Bank 2022d. Note: EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group; MSWM = municipal solid waste management. 4.16 The Bank Group has progressed in its client support on waste hierarchy practices and context-specific solutions in low-income countries (recommendations 1 and 2; World Bank 2022d). Stand-alone projects that address the entire waste sector chain (collection, transport, recycling, recovery, and disposal) have increased and make up just over a quarter of all projects that address solid waste, compared with about a fifth when IEG conducted its evaluation. The China Plastic Waste Reduction Project (Shaanxi) incorporates the waste sector chain, applying a novel approach with the objective to demonstrate economies of scale, professionalization of service delivery, and reduction in plastic pollution. In IFC, Circularity Plus is a global platform designed to assist solid waste management companies with investment and advisory solutions that accelerate the waste-to-value approach. A 2023 World Bank survey of 20 Country Management Units identified constraints on municipal solid waste management, although it had 22 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation limited coverage of low-income countries. The survey has prompted further discussions on how the Bank Group can enhance its support for low-income countries. In the next MAR reporting period, it would be useful to see continued expansion of waste hierarchy practices by the World Bank, confirmation that the IFC Circularity Plus platform continues operation, and follow-up on the survey (with expansion into more low- income countries). 4.17 IFC provides examples of its leadership in municipal solid waste management in countries and collaboration with development partners, whereas the World Bank’s evidence of engagement is in one Region so far (recommendation 3). IFC shows that it actively engages with developmental partners in multiple spaces to promote the adoption of sustainable waste management. For example, IFC and the Alliance to End Plastic Waste are collaborating on a platform to address bottlenecks in scaling plastics recycling. IFC is also fostering engagement through sharing its experience in the waste- to-value approach with the International Solid Waste Association’s member countries and private sector representatives. The World Bank through the Southeast Asia Regional Program on Combating Marine Plastics has convened development partners to harmonize policies, develop innovative solutions, share knowledge, leverage finance and investment, and elevate dialogue to address the plastic waste management challenges. Future MAR reporting could show continued deepening of IFC’s partnerships. Further, reporting could also show how the World Bank expands partnerships, such as collaborations with other multilateral development banks. Table 4.9. Urban Spatial Growth, Fiscal Years 2022–26 Management IEG Recommendation FY23 FY23 Recommendation 1: Adopt a framework that links the determinants of urban EE EE expansion to pathways for managing urban spatial growth and that contributes to the achievement of SDGs 1 and 11. Recommendation 2: Support World Bank clients with anticipating and preparing for LE LE urban spatial growth using preventive approaches, not just curative ones. Recommendation 3: Strengthen and ensure implementation of the World Bank’s LE LE protocol to identify and record precise project locations and collect land market data necessary to support clients with managing urban spatial growth. Sources: Independent Evaluation Group; World Bank 2021b. Note: EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group; LE = limited evidence of a change in the direction of travel; SDG = Sustainable Development Goal. 4.18 The World Bank’s adoption and use of planning frameworks demonstrates an evolving commitment to informed decision-making for urban spatial growth (recommendation 1; World Bank 2021b). The Urban, Disaster Risk Management, Resilience, and Land GP employed frameworks to help city leaders and inform global 23 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation discussions, such as at the Group of Twenty. Recent examples highlight the application of frameworks within projects in India, Pakistan, and Senegal to improve land administration and land development. Moreover, the new City Planning Labs program supports the use of tools to guide project teams in conducting spatial analysis for upstream dialogue with clients and project identification. Future MAR reporting on progress could provide evidence of continued implementation of the different initiatives and frameworks and their links to the achievement of the Sustainable Development Goals 1 and 11. Building on this, reporting could also show how project implementation improved managing urban spatial growth. 4.19 Tools, reports, and guidance have been developed to support preventive approaches for urban spatial growth (recommendation 2), and protocols are being implemented to record precise project locations (recommendation 3). The World Bank has provided guidance and support to help address urban informality, sprawl, and expansion that are linked to issues such as flood risk and environmental degradation. For example, the City Resilience Program Framework Agreement for Risk-Informed Land Use Planning is being established to help clients use preventive approaches. The Geo-Enabling Initiative for Monitoring and Supervision portfolio mapping exercise has started to provide geospatial project data at the level of subproject implementation sites, with at least 24 Urban, Disaster Risk Management, Resilience, and Land GP projects mapping more than 4,300 individual subproject sites (although it is not clear what proportion of the portfolio the mapping covers). In the next MAR, it would be useful to see the ongoing use of these tools and guidance, with more evidence of the preventive value of projects and of geospatial data. Crowding in Private Finance Table 4.10. Private Capital Mobilization, Fiscal Years 2021–25 Management Management IEG Recommendation FY22 IEG FY22 FY23 FY23 Recommendation 2: Expand private capital mobilization World Bank: LE; World EE EE platforms, guarantees, and disaster risk management products IFC and MIGA: Bank: LE; commensurate with project pipeline development (for the EE IFC and World Bank Group). MIGA: EE Recommendation 3: Develop new products and improve EE EE EE EE product alignment with the needs of new investor groups and partners (for IFC and MIGA). Sources: Independent Evaluation Group; World Bank 2020c. Note: EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group; IFC = International Finance Corporation; LE = limited evidence of a change in the direction of travel; MIGA = Multilateral Investment Guarantee Agency. 24 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation 4.20 The Bank Group expanded private capital mobilization (PCM) platforms, guarantees, and disaster risk management products (recommendation 2) and developed new products (recommendation 3; World Bank 2020c). The World Bank has executed swaps, catastrophe insurance bonds, and green bonds, as well as related advisory services. Moreover, IFC has looked to strengthen project designs to leverage more private capital and estimate the PCM effects within Project Appraisal Documents and regularly discuss the pipeline at the management level. IFC’s measure of PCM, core mobilization, committed as of June 30, 2023, $15 billion, which is the highest level for any FY. IFC’s Managed Co-Lending Portfolio Program continues to expand. Its newest facility, Managed Co-Lending Portfolio Program One Planet, received $2.5 billion in total investor contributions and committed $639 million within nine months. Managed Co-Lending Portfolio Program One Planet uses IFC’s best impact and environmental, social, and governance practices to expand investor partnerships and support further growth in PCM. IFC has also expanded its mobilization product offerings to better meet the needs of investors; for example, it prioritized mobilizing banks, institutional investors, and impact investors into groundbreaking sustainability-focused products. MIGA delivered in FY23 gross guarantee issuance of $6.4 billion, all of which mobilized private capital and operates in the private reinsurance market. It would be helpful for future reporting to show increased use of instruments and demonstrate that the use of guarantees to support new project financing or refinancing efforts is systematic. Table 4.11. Private Investment in Fragile and Conflict-Affected Situations, Fiscal Years 2022–26 Management Recommendation FY23 IEG FY23 Recommendation 1: IFC and MIGA should continue to review their financial risk, make EE LE more explicit the implications of IFC’s portfolio approach for FCS, and enhance capabilities to address nonfinancial risks to ensure they align with achieving business growth targets and impacts in FCS. Recommendation 2: To focus on the development of bankable projects, IFC and MIGA EE LE should further recalibrate their business models, client engagements, and instruments to continuously adapt them to the needs and circumstances of FCS and put in place mechanisms to track their effectiveness for real-time learning. Recommendation 3: IFC and MIGA should identify and agree on FCS-specific targets in Not validated Not their corporate scorecards to focus their efforts and track progress in implementing validated the World Bank Group FCV strategy for the private sector. Sources: Independent Evaluation Group; World Bank 2022c. Note: EE = emerging evidence of a change in the direction of travel; FCS = fragile and conflict-affected situation; FCV = fragility, conflict, and violence; FY = fiscal year; IEG = Independent Evaluation Group; IFC = International Finance Corporation; LE = limited evidence of a change in the direction of travel; MIGA = Multilateral Investment Guarantee Agency. 4.21 IFC and MIGA continue to mitigate financial and nonfinancial risks in fragile and conflict-affected situations (FCS), but the extent of updating the approach since 25 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation IEG’s evaluation is unclear (recommendation 1; World Bank 2022c). In the MAR report, IFC describes further progress in applying its contextual risk framework, the use of the IDA Private Sector Window to guide its investment decisions in FCS, the development of blended finance in its portfolio, and initiatives in the environmental, social, and governance space. MIGA’s report emphasizes its strengthened portfolio supervision, use of blended finance, and an increase in staff who address environmental and social risks in FCS. It is not clear from reporting, however, whether IFC and MIGA have taken additional measures beyond existing practices to review financial and nonfinancial risks and then adapted their business model in FCS. Without this evidence, it is difficult to assess whether the increases in portfolio have been supported by changes linked to the recommendation. For next year’s reporting, it would be useful to further explain what has changed since the evaluation in how the portfolio approach assesses trade-offs between investments in FCS and other markets and how nonfinancial risks are considered beyond issues related to environmental, social, and governance space. Further, MIGA’s response seems to emphasize keeping the claims ratio low by managing preclaims; this may be at odds with increasing its engagement in FCS, and further evidence would help IEG understand how trade-offs are managed. 4.22 In FCS, supporting the development of bankable projects in IFC and MIGA has mainly relied on upstream and risk mitigation instruments (recommendation 2). IFC has continued to invest in upstream instruments and deployed new types of project engagements with the objective to increase the pipeline of bankable projects. IFC’s proactive initiative to develop local champions can help expand IFC’s client base into new local and regional clients. MIGA has used existing instruments, such as the IDA Private Sector Window and trust funds, to support investments in FCS. With these efforts alone, IEG sees limited progress in recalibrating client engagement and their business models. An important area for further development and evidence next year is the tracking of progress and effectiveness from the implementation of new initiatives and tools in FCS described this year, and of correcting course as needed. 4.23 The third recommendation from this evaluation is not validated in this report by management or IEG. In response to the evaluation, CODE encouraged IEG, IFC, and MIGA to continue conversations regarding adopting a wider set of metrics. In the follow-up conversation, IFC and MIGA stated that they disagreed with the recommendation, although IEG considers this recommendation important. This change is highlighted in this report because it was not previously reported to CODE. 26 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation Boosting Long-Term Economic Growth Table 4.12. Regional Integration, Fiscal Years 2019–23 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 5: Strengthen the design of IDA’s Regional EE LE CD CD Policy Window–supported projects to improve the assessment of spillover effects and generate evidence based on robust indicators. Sources: Independent Evaluation Group; World Bank 2019c. Note: CD = evidence substantiates a change of direction of travel, with reversals unlikely; EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IDA = International Development Association; IEG = Independent Evaluation Group; LE = limited evidence of a change in the direction of travel. 4.24 Building on change during FY23, systematic processes have been implemented in the 20th Replenishment of IDA Regional Policy Window to improve the assessment of spillover effects. Following the introduction of Guidance Notes and processes, all approval stages for projects integrate the requirement to track spillovers effects. This has led to all projects that access the Regional Policy Window including at least one program development objective– or project development objective–level indicator to track transboundary spillover effects of the introduction of indicators. For example, the Africa Centres for Disease Control and Prevention Support Program to Combat Current and Future Public Health Threats aims to address substantial positive externalities that stem from disease surveillance at the regional and continental levels. Specific project indicators measure how this project will build the human and technical capacity of the countries while constructing a network of regional and continental connected disease surveillance laboratories. IEG agrees with management’s proposal to continue tracking this recommendation through the MAR; although the recommendation achieved a change of direction, management contends that continued reporting on spillover effects would help provide a track record of implementation. Table 4.13. Trade Facilitation, Fiscal Years 2019–23 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 1: The World Bank Group should promote CD EE CD CD an approach of complementary (simultaneous and/or sequential) interventions in trade facilitation reforms. Recommendation 2: The Bank Group should identify and EE LE EE EE mitigate political economy constraints to trade facilitation reform. 27 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 3: The Bank Group systematically apply a EE EE EE EE differentiated approach to identify and monitor, where relevant, the public policy objectives of trade regulations relating to public health, safety, the environment, good governance, and formality and the rule of law. Sources: Independent Evaluation Group; World Bank 2019b. Note: CD = evidence substantiates a change of direction of travel, with reversals unlikely; EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group; LE = limited evidence of a change in the direction of travel. 4.25 The implementation of all recommendations from the trade facilitation evaluation (World Bank 2019b) advanced in their final year of tracking, with recommendation 1 demonstrating a change of direction. Building on last year, the evidence shows that the Bank Group undertakes systematic complementary trade facilitation reforms between ASA, IFC advisory services, IFC investments, development policy financing, and investment policy financing and is informed by an annual dialogue between Country Management Units and the Global Trade Unit. There is also the useful application of tools for diagnostics on political economy constraints and the monitoring of trade regulation. In comparison with last year, evidence demonstrates an increasing application of diagnostics to mitigate political economy constraints; the diagnostics inform action plans that sequence the implementation of reforms. However, for both recommendations 2 and 3, there is unfinished business. IEG encourages management to continue the systematization of tools to mitigate political economy constraints and monitor trade regulations through, for example, the Global Trade Facilitation team. Table 4.14. State-Owned Enterprises, Fiscal Years 2021–25 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 1: The World Bank Group should apply a EE EE EE EE selectivity framework for SOE reform support that considers country governance conditions, control of corruption, and sector- and enterprise-level competition. Recommendation 2: The Bank Group should apply the EE EE EE EE Maximizing Finance for Development and its embedded Cascade approach for SOE reform. Sources: Independent Evaluation Group; World Bank 2020b. Note: EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group; SOE = state-owned enterprise. 4.26 The Bank Group applies a range of tools and assessments to assist in the selection of state-owned enterprise (SOE) reforms and the Cascade approach (recommendations 1 and 2; World Bank 2020b). Progress on recommendation 1 this year shows that the 28 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation World Bank uses tools, such as the Integrated SOE Framework, and core diagnostics, such as Country Economic Memorandums, Country Private Sector Diagnostics, the Infrastructure Sector Assessment Program 2.0, and the Financial Sector Assessment Program. IFC is facilitating enterprise-level reform processes, supporting SOE governance, and mitigating related risks through its SOE Corporate Governance Assessments. The Bank Group’s evidence for recommendation 2 includes examples of operations, shared tools, and intensified dialogue between IFC, MIGA, and the World Bank that support the Cascade approach. For example, IFC’s investment in Banque de l’Habitat du Sénégal forms part of a multicomponent intervention in Senegal’s housing market that follows the Cascade approach. Furthermore, the World Bank has increased the number of development policy operations that contribute to SOE reform. In its MAR reporting, the World Bank provided limited evidence on improved public governance reform of SOEs, focusing instead on corporate governance. To build toward changes in practice, it will be important to show the effect of changes implemented on operations and thus demonstrate improved public governance in SOE reform and the development of a coordinated approach by IFC, MIGA, and the World Bank for Maximizing Finance for Development in SOE reform across a range of sectors. Table 4.15. Public Financial and Debt Management, Fiscal Years 2021–25 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 1: The World Bank should regularly EE EE EE EE monitor the quality of the key pillars of PFDM for each IDA- eligible country, possibly through a centralized country- specific PFDM assessment. Recommendation 2: Actively use the previously described EE LE LE LE assessment (recommendation 1) to prioritize and sequence World Bank support for PFDM capacity building and reform in IDA-eligible countries. Sources: Independent Evaluation Group; World Bank 2021f. Note: EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IDA = International Development Association; IEG = Independent Evaluation Group; LE = limited evidence of a change in the direction of travel; PFDM = public financial and debt management. 4.27 The Equitable Growth, Finance, and Institutions Practice Group continues to work on a more integrated approach to identifying and acting on public financial and debt management (PFDM), which can translate into capacity building and reform support in IDA-eligible countries (recommendations 1 and 2; World Bank 2021f). Building on previously reported progress, Equitable Growth, Finance, and Institutions continued to implement a four-pronged approach to addressing the need for more cohesive monitoring of key pillars of PFDM in IDA-eligible countries.3 On the basis of this approach, the World Bank has completed a stocktaking of the tools it applies. 29 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation Frameworks have been updated with metrics capturing PFDM results. A results-based monitoring framework has also been elaborated to include relevant indicators from Debt Management Performance Assessments, the Public Expenditure and Financial Accountability framework, and debt transparency assessments. There are ongoing initiatives to support engagement across GPs (for example, a joint Governance– Macroeconomics, Trade, and Investment mechanism for renewed monitoring of public financial management and debt management). However, the World Bank provides limited evidence of how updated assessments have helped develop capacity and shows no examples of sequenced reforms in country. Given the progress on monitoring the quality of PFDM pillars, it will be important to provide examples of how the approach being followed will develop from internal country reforms to systemic efforts to develop country capacity. If limited capacity for change in recommendation 2 continues, future reporting could consider whether a change of approach is required. Table 4.16. Mobilizing Technology for Development, Fiscal Years 2021–25 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 1: Where DTT offer opportunities to make EE EE EE LE progress on the twin goals more effectively or efficiently, ensure that the World Bank Group avails itself of those opportunities and addresses, in particular, the risks posed by DTT. Recommendation 2: Build a Bank Group workforce with the EE LE EE LE skills required to harness DTT opportunities and mitigate DTT risks by identifying DTT-relevant skills, determining gaps in these skills, and filling these gaps. Recommendation 3: Improve the effectiveness and efficiency of EE EE EE EE World Bank procurement for complex technology projects (World Bank only). Sources: Independent Evaluation Group; World Bank 2021c. Note: DTT = disruptive and transformative technologies; EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group; LE = limited evidence of a change in the direction of travel. 4.28 Since the completion of the evaluation on mobilizing technology for development (World Bank 2021c), the context in which these recommendations are implemented has changed rapidly. The size and scale of potential risks and opportunities has grown in applying disruptive technology, for example, with the increasing integration of artificial intelligence into the workplace. Despite these changes, the MAR lacks evidence of the World Bank’s increasing the intensity of its efforts on disruptive technologies compared with last year. 4.29 The World Bank has made limited progress on mobilizing opportunities and addressing risks posed by disruptive technologies, whereas IFC has established multiple 30 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation initiatives to develop its lending framework (recommendation 1). The World Bank has made progress through several initiatives, such as the Digital Development Partnership 2.0; diagnostics, such as the Digital Economy for Africa Initiative; the multidonor trust fund for cybersecurity; and the tools for identifying the accountability of artificial intelligence in World Bank operations. However, there is little evidence of how adequate or effective these initiatives are in relation to the size and scale of potential risks and opportunities. IFC has implemented new programs to help digital start-ups access financing, provided advisory services in digital transformation strategies, and increased its investment in digital infrastructure projects. Further evidence is required from the World Bank on how the initiatives individually or collectively link together and how evidence on disruptive technology is being considered in review meetings. 4.30 On recommendation 2, IFC has further reorganized its operations function to better align its workforce with disruptive and transformative technologies opportunities, whereas the World Bank has supported training for staff, organized several knowledge events, and produced many publications. In IFC, there has been a 20 percent increase in staff in its Disruptive Technologies and Funds Department, which sought to build expertise in a broad range of strategic themes, including climate tech, health tech, agritech, edtech, fintech, artificial intelligence, and machine learning. The World Bank has launch multiple initiatives across GPs; however, IEG does not see evidence of a coordinated approach. The trainings and increases in skills are potentially useful, although further evidence on how a systemic approach to identifying and filling relevant skill gaps would be helpful. 4.31 The World Bank is taking several measures to improve the effectiveness and efficiency of procurement processes for complex technology projects (recommendation 3). The implementation of guidance and a roster of experts on complex procurement is advancing, with countries using ProcurementiNet.org for capacity building. The Health, Nutrition, and Population GP is contributing technical experts to World Bank health sector operations and working on a position paper to address the challenges of digital health procurements. The Geo-Enabling Initiative for Monitoring and Supervision portfolio mapping exercise is providing guidance on effective and cost-efficient technology solutions, whereas the GovTech group is proactively engaging in improving procurement practices. Future MAR reports could focus on examples of the adoption by World Bank staff of well-tailored procurement practices. 31 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation Table 4.17. Sustainable Development Finance Policy, Fiscal Years 2022–26 Management IEG Recommendation FY23 FY23 Recommendation 1: Consideration should be given to expanding the countries EE PC covered by the Debt Sustainability Enhancement Program beyond those at moderate or high levels of debt distress or in debt distress. A low level of debt distress alone should not be sufficient for exclusion from the Debt Sustainability Enhancement Program, and IEG recommends applying an additional filter. Recommendation 2: PPAs should emanate from an up-to-date assessment of country- LE LE specific debt stress and be set explicitly within a longer-term reform agenda. Recommendation 3: Where PPAs support actions that need be taken regularly (for LE LE example, debt reporting to parliament), they should aim for long-lasting institutional reforms rather than relying on one-time actions. PPAs should seek to institutionalize good practice in fiscal and debt management by supporting the establishment of statutory requirements, the existence of which can help depoliticize future decisions. Sources: Independent Evaluation Group; World Bank 2021a. Note: EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group; LE = limited evidence of a change in the direction of travel; PC = progress constrained; PPA = performance and policy action. 4.32 The World Bank has started to strengthen, review, and plan for further changes in the process of the debt sustainability analysis (DSA) in IDA countries (which is a key tool for defining the coverage of the Debt Sustainability Enhancement Program). Since the IEG’s evaluation (World Bank 2021a), action has been taken to strengthen the DSA through, for example, increased scrutiny of data and projections, and the involvement of senior management. A review of DSAs conducted this year identified no issues in the coverage of debt data. However, the review did not provide an analysis of debt data quality (which is an important area that can lead to challenges in identifying debt issues). Adaptations to filters that exclude countries from the Debt Sustainability Enhancement Program are at the planning stage. In the coming year, management plans to revisit checklists to give greater attention to data coverage when reviewing future DSAs. In addition, options for handling countries at low risk of debt distress with severe data coverage and data quality issues are planned to be presented to the Sustainable Development Finance Policy committee. 4.33 Progress is assessed as constrained for recommendation 1 because an additional filter has yet to be applied that can expand coverage of the Debt Sustainability Enhancement Program for some countries at low risk of debt stress based on the DSA. IEG maintains that an additional filter is warranted due to cases of rapid debt deterioration in IDA-eligible countries assessed to be at low risk of debt distress. One- third of IDA-eligible countries that experienced an elevation in their risk of debt distress over the past decade experienced a two-level deterioration within three years. The evaluation of the Sustainable Development Finance Policy also found that DSA growth 32 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation assumptions have tended to have an optimistic bias, with downside risks often underestimated, particularly regarding contingent liabilities of SOEs or in assessing the potential impact of a compounding of vulnerabilities. Although management notes that exceptions can exist to changing a country’s performance and policy action (PPA) eligibility status, no examples of applying this exception are provided. A risk remains that DSAs based on poor quality or incomplete data serve as a basis for excluding a country from participation in the Debt Sustainability Enhancement Program. IEG suggests that exemption from the Debt Sustainability Enhancement Program also be accompanied by confirmation from World Bank management on the adequacy of data used in DSAs. There are several ways that such an assurance can be provided, including drawing on relevant dimensions of a current Debt Management Performance Assessment and compliance with Debtor Reporting System requirements. IEG sees such options as complementary to the commitment of management to give greater attention to data quality and coverage in response to the recommendations of The World Bank’s Role in and Use of the Low-Income Country Debt Sustainability Framework evaluation (World Bank 2023c). 4.34 The World Bank has implemented activities to improve the extent to which PPAs emanate from up-to-date assessments of the drivers of country-specific debt stress and risks and ensure that PPAs are set explicitly within a longer-term reform agenda (recommendation 2). Management reports that PPAs are increasingly framed as programmatic, are institutionalized, and draw on sound analytics. Moreover, the PPA template has been modified to help teams flag, for example, programmatic characteristics more explicitly. To provide evidence of ongoing progress for this recommendation, future reporting could show the proportion of PPAs that address identified drivers and risks of debt distress and demonstrate that PPAs are situated within results chains that describe subsequent and complementary actions needed for impact. 4.35 Some progress has been made toward ensuring that PPAs institutionalize reforms (recommendation 3). Management reports that in FY21 and FY22, 15 countries each institutionalized debt publication, improving frequency and timeliness. Several countries have engaged in a programmatic approach, which introduces intermediate steps that can contribute to the fulfillment of institutional reforms over the medium term (for example, in Ghana, Guyana, the Kyrgyz Republic, Pakistan, Tajikistan, and Tuvalu). In the next MAR, it would be helpful for evidence to describe systematic processes being put in place to support the introduction of regulatory and institutional frameworks to manage debt levels prudently. 33 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation Table 4.18. Doing Business Report and Business Ready, Fiscal Years 2022–26 Management IEG Recommendation FY23 FY23 Lesson 1: Recognizing the powerful motivational effect of reform indicators, especially LE LE those that facilitate country rankings, this evaluation notes the limitations in the coverage and guidance offered by any single indicator set on its own and advocates integrating them with complementary analytic tools and indicators. Lesson 2: Recognizing the granularity and specificity of individual reforms in any given PC PC country context, the findings from this evaluation suggest that it is better to avoid using business regulatory or similar global indicators as explicit reform objectives or monitoring indicators in World Bank Group projects and country strategies focused on improving the business environment. This does not preclude the use of primary data to agreed targets that track and measure critical Bank Group institutional commitments. Lesson 3: Global indicators coverage and specifications are improved if, at regular and LE LE predictable intervals, they are updated to reflect learning from research and field experience to (i) improve links to important development outcomes; (ii) strengthen relevance to the experience of the subject of coverage; and (iii) adapt to technological changes in the areas covered by the indicators. Lesson 4: The Doing Business experience indicates the need for mechanisms and PC PC safeguards to ensure the accuracy and validity of Bank Group global indicator-based reports and related communications, using robust and transparent standards of evidence. Sources: Independent Evaluation Group; World Bank 2022a. Note: FY = fiscal year; IEG = Independent Evaluation Group; LE = limited evidence of a change in the direction of travel; PC = progress constrained. 4.36 A range of mechanisms has been adopted by the Doing Business replacement— Business Ready (B-READY), that commendably took account of the lessons provided by IEG in this first year of reporting. B-READY’s intended increased coverage, specifications, and candor on the limitations of the indicators can help avoid focus being placed on any single indicator and provide the opportunity for updates (lessons 1 and 3). Management’s evaluation response highlights a range of safeguards that have been defined to assure B-READY’s accuracy and validity and to reduce the risk of its use to set reform objectives (lessons 2 and 4). Progress is reported as constrained for these two lessons because ongoing implementation requires the observance of Country Management Units (lesson 2) and the development of guidance to depict, consolidate, and codify rules and protocols in a comprehensive way for global indicator processes for B-READY (lesson 4). Evidence provided in future MARs would benefit from continued reporting on progress, coverage, and adaptations of the mechanisms being put in place in the B-READY Concept Note. Management could consider a mechanism to interact with other units that affect the success of B-READY. 34 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation Improving World Bank Group Capacity to Meet Its Goals Table 4.19. Citizen Engagement, Fiscal Years 2019–23 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 1: As it defines future corporate priorities for CD EE EE EE citizen engagement, the World Bank should reflect in those priorities the need to achieve greater depth and quality of the citizen engagement activities it supports. Recommendation 2: The World Bank should encourage and EE EE EE EE support efforts of its regional, country, and Global Practices teams to establish, where appropriate, “thick” citizen engagement that is regular and continuous, uses multiple tools, and is embedded in country systems. Recommendation 3: The World Bank should strengthen the EE LE EE EE monitoring of its citizen engagement activities by systematically adopting results framework indicators that are results oriented. Recommendation 4: The World Bank should seize the EE LE EE PC opportunity of the implementation of the ESF to leverage citizen engagement mechanisms—beyond consultations and grievance redress mechanisms—to reach the objectives of managing social risks, strengthening country systems, and promoting social inclusion. Recommendation 5: IFC should ensure that its clients’ EE LE EE EE stakeholder engagement activities required by Performance Standard 1 in projects with affected communities are carried out during appraisal and supervision of the projects and systematically documented. Sources: Independent Evaluation Group; World Bank 2018b. Note: CD = evidence substantiates a change of direction of travel, with reversals unlikely; EE = emerging evidence of a change in the direction of travel; ESF = Environmental and Social Framework; FY = fiscal year; IEG = Independent Evaluation Group; IFC = International Finance Corporation; LE = limited evidence of a change in the direction of travel; PC = progress constrained. 4.37 This is the final year of tracking for the citizen engagement evaluations, and there is no evidence that the recommendations’ intended outcomes are yet being met. IEG recognizes that over the past two years, the Citizen Engagement and Social Accountability Global Solutions Group piloted new monitoring approaches, revitalized its engagement with external experts, and conducted a strategic review. These are promising but have not yet yielded results. For learning purposes, in a fresh analysis, it would be useful to explain what has delayed change on these recommendations. 4.38 The World Bank continues to develop the quality and monitoring of citizen engagement through innovations, tools, and training (recommendations 1, 2, and 3; World Bank 2018b) but does not yet demonstrate “thick” engagement. The World Bank is piloting an enhanced framework for tracking citizen engagement. This framework is designed to encourage institutional change toward more intensive and holistic citizen 35 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation engagement throughout the project cycle. Seventy-three percent of the FY19 portfolio of projects (145 of 200 investment policy financing projects) advanced in assessing progress on beneficiary feedback indicators. There are examples of innovative forms of citizen engagement in work on climate change and in FCS that can be further built on. For example, the Stabilization and Recovery in Eastern Democratic Republic of Congo Project will implement multiple mechanisms aimed at giving citizens a meaningful say in decision-making despite the challenges of operating in a conflict-affected area. Moreover, the World Bank has now launched a Green Accountability Call for Proposals aimed at creating demonstration cases of meaningful participation of civil society and local communities in ensuring accountable public climate finance. IEG agrees with management’s suggestion to continue tracking of recommendation 3 in the MAR. 4.39 As this is the final year of tracking recommendation 4, given the initial progress on activities reported on leveraging citizen engagement through the ESF, IEG flags the recommendation as PC. The activities underway have yet to show outputs. Internal training on the links between citizen engagement, the ESF, and stakeholder engagement has been implemented, but no evidence is presented on their quality or reach. The Bank Group reports ongoing efforts to intensify synergies between the ESF and citizen engagement, for example, through collaborations on digital grievance redress mechanisms. The World Bank and IFC are also exploring ways to conduct complementary and coherent actions between stakeholder engagement plans and citizen engagement. IEG suggests retiring this recommendation and conducting a fresh analysis on leveraging citizen engagement through the ESF in the FY24 IEG evaluation. Ongoing tracking of this recommendation could additionally fall under management’s proposal to continue tracking recommendation 3. 4.40 IFC has also made a range of efforts to strengthen stakeholder engagement activities through training and tools (recommendation 5). A variety of trainings have been developed or implemented; for example, in FY23, 186 IFC environmental and social staff and consultants attended internal trainings on stakeholder engagement—a significant additional effort compared with the previous year when 75 staff attended one training. Staff who attended the training found the content pertinent to their work in assessing IFC clients’ efforts during project appraisal and supervision. IFC also updated its environmental and social questionnaires and the Annual Monitoring Report templates in FY23. This change aims to realize significant improvement in standardization, coverage, and consistency in analysis across regions and sectors on stakeholder engagement. 36 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation Table 4.20. Convening Power, Fiscal Years 2020–24 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 1: Scope engagements and contributions EE EE EE EE to major global convening initiatives more deliberatively. Recommendation 2: Enhance how the World Bank and IFC’s EE EE EE EE internal systems and processes support managing major convening initiatives over their life cycle. Recommendation 3: Improve links between the World Bank’s EE EE EE EE global and country work. Sources: Independent Evaluation Group; World Bank 2020e. Note: EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group; IFC = International Finance Corporation. 4.41 The Bank Group has shown an increasingly selective and deliberative approach in its global convening initiatives over the past three MAR reports, especially in relation to trust funds and financial intermediary funds. In FY21, the Bank Group began to define and apply rigorous criteria to engage in new convening activities. In FY22 and FY23, evidence related to the World Bank’s trust fund and financial intermediary fund reforms convey a more deliberative and selective approach to major global convening initiatives. The current MAR shows examples of selectivity of convening efforts in response to priority global development issues, such as climate change, debt, pandemics, food security, and FCV, that are in line with the evolution roadmap. The Strategic External Financing Framework serves as a tool for greater strategic alignment, coordination, and selectivity. Moreover, IFC convenes global audiences around major thematic priorities, such as affordable housing and climate, to support the development of a specific set of actionable investment priorities and private sector solutions. To demonstrate a change of direction next year, it would be helpful to maintain these advances and present evidence on how management at various levels defines a system for prioritization of convening. 4.42 The Bank Group has made progress on managing major convenings both over their life cycle (recommendation 2; World Bank 2020e) and in linking global and country work (recommendation 3). The World Bank is implementing new frameworks to help manage convening. The Financial Intermediary Fund Management Framework, for example, has established a risk-based approach to the World Bank’s life cycle management. Further, work on Maximizing Finance for Development with the Group of Twenty members was driven by a secretariat that engaged both the World Bank and IFC. IFC has recently combined the partnerships unit with the blended finance department to strengthen the focus of donor fundraising on blended finance and facilitate the development of new partnerships with academia and think tanks. New tools, diagnostics, and templates have been introduced to link global convening to 37 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation country programs; for example, CCDRs serve as an input into Systematic Country Diagnostics. Moreover, at the 27th Conference of the Parties to the United Nations Framework Convention on Climate Change, CCDRs were discussed by several governments that convened events for partners. In the next MAR, it would be useful to focus on the results achieved through the trust fund life cycle management process and on how global convenings link to country programs; this would help show that a change of direction has occurred. Table 4.21. Outcome Orientation, Fiscal Years 2021–25 Management IEG Management IEG Recommendation FY22 FY22 FY23 FY23 Recommendation 1: Reform the country-level results system EE EE EE EE to ensure that it accurately captures the Bank Group’s contribution to country outcomes and usefully informs decision-making on country engagements. Sources: Independent Evaluation Group; World Bank 2020d. Note: EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group. 4.43 The Bank Group has advanced on an aspect of reforming country-level results systems based on its approach that articulates high-level outcomes. Articulating high- level outcomes in CPFs may help better capture the contribution of the Bank Group to development outcomes. Building on progress reported last year, all CPFs delivered in FY23 include higher-level outcome statements, up from 4 out of 10 in FY22. As high- level outcomes are not being mandated for already approved CPFs, it will take some years before all countries have them in place. The revised country engagement guidance is being implemented to enable IFC’s and MIGA’s inputs to better embed their perspectives in Bank Group documents, such as Systematic Country Diagnostics and CPFs. The Bank Group has made changes to the Completion and Learning Reviews that seek to enable lessons from previous CPFs to benefit at an earlier stage the ones under development. Continued training activities aim to contribute to the assessment of outcomes in country results frameworks, engagement, and partnership frameworks. Guidance on capturing the contribution of knowledge products to high-level outcomes is also being developed. 4.44 The outcome orientation changes may not be going far enough and fast enough against the recommendation and in a world beset with multiple crises. Management updates provide limited evidence on how much reforms increase the accuracy and utility of the country-level results system for country teams and support adaptive management. The limitations on progress relate to the decision to continue reliance on results frameworks, whereas the evaluation recommended shifting to a country-level monitoring, evaluation, and learning plan. Moreover, the evidence does not discuss 38 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation management’s commitment to pilot monitoring, evaluation, and learning plans. The outcome orientation evaluation contains useful evidence and examples of how teams practice adaptive management through tacit knowledge and how plans could be put in place to support changes in practice that respond swiftly to local challenges. These adaptive practices could be considered by the Bank Group in an updated approach to enhancing outcome orientation that helps better respond to the multiple crises noted in management’s MAR report. 4.45 In FY24, MAR reporting on outcome orientation will enter its third year, and IEG anticipates examples of changed practice that move toward enhanced learning from country-level results systems. Evidence could include what difference the approach has made for country teams in their learning and adaptation practices. Examples could also cite whether Performance and Learning Reviews and Completion and Learning Reviews have better evidence of IFC’s contribution; how it contributes to market creation; and how its advisory work contributes to market, industry, or institutional change. Robust feedback mechanisms that reflect the country teams’ experience with the outcome orientation reforms could strengthen evidence for the MAR. Table 4.22. Global Footprint Effectiveness, Fiscal Years 2022–26 Management IEG Recommendation FY23 FY23 Recommendation 1: The World Bank should refine its current approach to EE LE managing its staffing global footprint by clearly specifying decentralization’s expected outcomes and adopting principles to guide and adjust decentralization decision-making based on evidence. Recommendation 2: The World Bank should mitigate the risks to knowledge flow EE EE brought about by decentralization and put in place safeguards to avoid developing country and regional silos. Recommendation 3: The World Bank should establish clear and structured paths to EE EE systematically promote locally recruited staff’s professional and career growth within its overall approach to improving the effectiveness of its global footprint. Sources: Independent Evaluation Group; World Bank 2022b. Note: EE = emerging evidence of a change in the direction of travel; FY = fiscal year; IEG = Independent Evaluation Group; LE = limited evidence of a change in the direction of travel. 4.46 The World Bank has progressed in its management of its global footprint through workforce planning and mobility exercises but has yet to specify and measure expected outcomes (recommendation 1; World Bank 2022b). The annual workforce planning and corporate recruitment and mobility exercises implement tools that help guide decisions on decentralization. The recommendation of the evaluation, however, goes beyond these changes and looks for clear outcomes to be defined; they are then measured. Clear outcomes for decentralization have yet to be defined; without this step, it is difficult to identify potential contributions of decentralization to development 39 Chapter 4 Progress on the Implementation of Recommendations, by Evaluation effectiveness. Ongoing reporting on this recommendation would usefully reflect progress in defining and measuring the outcomes on what the decentralization seeks to achieve. IEG can engage with management on this issue, if helpful. 4.47 Enhanced platforms, training, and frameworks have been implemented to address barriers to knowledge flow and provide structured paths for locally recruited staff’s professional and career growth (recommendations 2 and 3). To manage knowledge risks, learning continues to be strengthened by the expansion of online courses. Learning weeks have resumed and include participation of locally recruited staff. Each Region has an ongoing program for knowledge and learning to support flow between and within countries and Regions, complementing the global sector learning initiatives led by GPs. For example, Middle East and North Africa and Eastern and Southern Africa implement their own knowledge and learning strategies. The Career Development and Mobility Framework is aimed at helping promote locally recruited staff’s professional and career growth. Further, ongoing efforts seek to expand opportunities and support staff’s career development. Future MARs could build on this evidence and show how these efforts continue to develop and adapt, including collecting data on the movement of internationally recruited staff across the Regions and headquarters. 40 5. Conclusions and Suggestions for Further Enhancement of the Management Action Record 5.1 The Bank Group continues to make steady progress in implementing IEG recommendations; however, it is infrequent that a change of direction is demonstrated. IEG validates that just under 60 percent of the 59 recommendations from the 22 IEG evaluations in this year’s MAR have been implemented to the level of emerging evidence of progress. This includes, for example, implementation of guidance, processes, and anecdotal examples of change in behavior. For the evaluations in their first year of tracking, just under 4 out of 10 recommendations are found to be making progress at emerging evidence level and above. In the second-to-fourth year of tracking, 9 out of 10 recommendations are making progress at emerging evidence level and above. However, for only 6 recommendations, there is evidence that the Bank Group has changed its practices or systems in response to IEG’s recommendations. Therefore, up to five years after their discussion in CODE, changes envisioned by many IEG evaluations remain unfinished in the sense that the MAR report does not demonstrate changes in systems or practices. 5.2 The gap between delivering outputs linked to recommendations and achieving outcomes needs to be bridged more often. Many recommendations that show emerging evidence of progress do not advance to a change of direction over the four years of tracking. In some instances, additional actions by management are needed for implementation. For other recommendations, a change in approach in working toward their intended outcomes is needed, and this change can be informed by evidence from IEG evaluations. A final issue is that some recommendations’ intended outcomes will not be achieved, and a fresh analysis is needed through management’s review of one of IEG’s products. To tackle these issues, IEG suggests three areas of action to enhance the effectiveness of the MAR: • Clear processing of recommendations. Action, dialogue, and assessment of progress on recommendations are a shared responsibility between IEG and management that is hindered when outcomes and agreements are unclear. Management could review and adjust the consistency of how it agrees, disagrees, and partially agrees to recommendations. For example, management agrees to some recommendations with caveats, which may not be fully defined; thus, management considers the implementation more advanced than does IEG. Further, IEG is available to jointly review with management areas for improvement for evaluation recommendations—for example, their number and whether they are stated to best support progress toward intended outcomes. 41 • Quicker implementation of recommendations. The suggestion by management to designate champions and define pathways to implementing recommendations could help the swift implementation of recommendations. IEG is available to work with management to jointly outline the form and function of champions to support the adoption of recommendations. The discussion could also include refinement of the criteria for their retirement or extension and consider the evolution of the Bank Group. • Constructive dialogue as part of the MAR process. The MAR provides a regular account that helps consolidate learning and adapt the implementation of recommendations to current operational realities. The FY23 MAR process’s constructive dialogue helped clarify recommendations, IEG’s expectations of evidence, and Bank Group teams’ implementation challenges. Early dialogue on recommendations—soon after evaluations are completed—would smooth implementation. 42 References Vives, Gemma Torras. 2023. “Why Data Infrastructure Is Key for a Transparent Carbon Market.” World Bank Blogs (blog), March 7, 2023. https://blogs.worldbank.org/climatechange/why- data-infrastructure-key-transparent-carbon-market. World Bank. 2018a. Carbon Markets for Greenhouse Gas Emission Reduction in a Warming World: An Evaluation of the World Bank Group’s Support to Carbon Finance. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018b. Engaging Citizens for Better Development Results. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018c. World Bank Group Support to Health Services: Achievements and Challenges. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019a. Building Urban Resilience: An Evaluation of the World Bank Group’s Evolving Experience (2007–17). Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019b. Grow with the Flow: An Evaluation of World Bank Group Support to Facilitating Trade 2006–17. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019c. Two to Tango: An Evaluation of World Bank Group Support to Fostering Regional Integration. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2020a. Renewable Energy: Evaluation of the World Bank Group’s Support for Electricity Supply from Renewable Energy Resources, 2000–2017. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2020b. State Your Business! An Evaluation of World Bank Group Support to the Reform of State-Owned Enterprises, FY08–18. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2020c. World Bank Group Approaches to Mobilize Private Capital for Development . Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2020d. The World Bank Group Outcome Orientation at the Country Level . Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2020e. The World’s Bank: An Evaluation of the World Bank Group’s Global Convening . Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021a. The International Development Association’s Sustainable Develop ment Finance Policy: An Early-Stage Evaluation. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021b. Managing Urban Spatial Growth: World Bank Support to Land Administration, Planning, and Development. Independent Evaluation Group. Washington, DC: World Bank. 43 References World Bank. 2021c. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021d. The Natural Resource Degradation and Vulnerability Nexus: An Evaluation of the World Bank’s Support for Sustainable and Inclusive Natural Resource Management (2009–19). Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021e. World Bank Engagement in Situations of Conflict: An Evaluation of FY10–20 Experience. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021f. World Bank Support for Public Financial and Debt Management in IDA-Eligible Countries. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021g. World Bank Support to Reducing Child Undernutrition. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2022a. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2022b. Enhancing the Effectiveness of the World Bank’s Global Footprint. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2022c. The International Finance Corporation’s and Multilateral Investment Guarantee Agency’s Support for Private Investment in Fragile and Conflict-Affected Situations, Fiscal Years 2010–21. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2022d. Transitioning to a Circular Economy: An Evaluation of the World Bank Group’s Support for Municipal Solid Waste Management (2010–20). Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2022e. The World Bank’s Early Support to Addressing COVID -19: Health and Social Response—An Early-Stage Evaluation. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2023a. Evolution of the World Bank Group—A Report to Governors. Washington, DC: World Bank. World Bank. 2023b. The World Bank Group’s Early Support to Addressing the COVID-19 Economic Response, April 2020–June 2021: An Early-Stage Evaluation. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2023c. The World Bank’s Role in and Use of the Low-Income Country Debt Sustainability Framework. Independent Evaluation Group. Washington, DC: World Bank. 44 Notes Chapter 4 1 The Management Action Record report from management also discusses the overall quality of monitoring and evaluation in Health, Nutrition, and Population Global Practice projects. This is a separate issue and is not considered in the assessment of progress toward the recommendation; it is possible to have substantial monitoring and evaluation quality at a portfolio level but have poor monitoring of health quality in important projects. 2In the Management Response, only the World Bank committed to taking action on this recommendation; there is no validation of evidence from the International Finance Corporation. 3 The approach includes (i) conducting global stocktaking of public finance tools, (ii) enhancing the frequency and timeliness of public financial and debt management assessments, (iii) strengthening metrics for capturing public financial and debt management results, and (iv) enhancing the focus on results. 45 Appendix A. Summary Table of Progress Levels Evaluation Management IEG Topic Recommendation FY23 FY23 Status Health Services Recommendation 1: Improve measurement of the quality CD EE Retired of health services and the distributional effects of health services projects. Recommendation 3: To develop sustainable capacity to EE EE Retired address pandemics and systematically integrate preparedness plans and governance frameworks for pandemic control within the client country’s own health system in World Bank Group–financed projects and advisory services. Undernutrition Recommendation 1: Adjust nutrition programming in LE LE Active country portfolios to (i) give more priority to institutional strengthening for coordination and implementation of multisectoral nutrition interventions and (ii) increase focus on subnational targeting of interventions to reflect areas of greatest disadvantage and persistency of need. Recommendation 2: Strengthen nutrition support in LE LE Active Global Practices to (i) rebalance investments to have greater emphasis on nutrition-specific interventions and (ii) increase focus on social norms interventions and behavior changes, with more attention to tracking expected achievements to improve nutrition determinants. Carbon Finance Recommendation 2: The Bank Group should increase its EE EE Retired use of carbon finance instruments to attract and mobilize finance that supports transformational activities and leverages private investments. Urban Resilience Recommendation 1: The Bank Group should CD CD Retired systematically identify and track progress of interventions that build urban resilience to chronic stresses and acute shocks, across its institutions. Recommendation 2: The design and implementation of EE EE Active World Bank projects that build urban resilience should systematically incorporate resilience characteristics and articulate their application throughout the project cycle. These should include the following: (i) design standards in line with resilience risks, (ii) cost-benefit analysis in line with resilience risks, (iii) city and interjurisdictional coordination, and (iv) inclusive approaches for vulnerable people. Recommendation 3: In urban areas where the client has PC EE Active identified crime and violence as a resilience risk, the World Bank’s support should be based on a localized typology of crime and violence that is informed by relevant analytic work. This approach should be supported by an assessment of the mechanisms most effective at reducing crime and violence within operations. Recommendation 5: IFC should support its public and EE CD Retired private sector Cities Initiative clients through available 46 Appendix A Summary Table of Progress Levels Evaluation Management IEG Topic Recommendation FY23 FY23 Status resilience risk assessment and mitigation tools to strengthen development impacts. Renewable Recommendation 1: The Bank Group to prioritize CD CD Retired Energy interventions that focus on the integration of renewable energy sources into the power systems of client countries, to facilitate progress in their clean energy transitions. Recommendation 2: The Bank Group to support EE EE Active renewable energy scale-up through comprehensive, long- term country engagements, with coordinated Bank Group solutions, based on the comparative advantages of each institution, to address barriers, aided by robust upstream diagnostics. Recommendation 3: The Bank Group to continually CD CD Retired upgrade the pool of specialized skills to help clients address their pressing and rapidly evolving challenges to scale-up renewable energy. Natural Resource Recommendation 1: The World Bank should identify and EE EE Active Degradation and analyze natural resource degradation and vulnerability Vulnerability nexus issues and leverage this knowledge in Systematic Country Diagnostics and in country engagements where such issues matter for achieving sustainable poverty reduction and shared prosperity. Recommendation 2: World Bank operations that address EE EE Active natural resource degradation should direct attention to resource governance challenges and use a mix of resource management practices and financial incentives appropriate for the relevant socioecological systems. Recommendation 3: World Bank Global Practices involved LE EE Active in addressing natural resource degradation and associated vulnerability should share knowledge, improve measurement, and enhance coordination in the design and implementation of their projects to optimize development effectiveness. World Bank Recommendation 1: To enhance the conflict sensitivity of EE EE Active Engagement in World Bank engagement, ensure that politically sensitive, Situations of confidential analysis is generated, retained, and managed Conflict so that it can be used by select future staff working on that country. Recommendation 2: Ensure that country engagements are EE EE Active informed by timely analyses of conflict dynamics and risks. Recommendation 3: Address factors that dissuade World EE LE Active Bank engagement in conflict-affected areas. Recommendation 4: In conflict-affected countries, rethink LE EE Active what success looks like. Municipal Solid Recommendation 1: To achieve more sustainable and EE EE Active Waste scalable outcomes in municipal waste management, Bank Management Group technical and financial support to clients should give clear priority to the adoption and implementation of 47 Appendix A Summary Table of Progress Levels Evaluation Management IEG Topic Recommendation FY23 FY23 Status waste hierarchy practices, in line with client needs and capabilities for municipal solid waste management. Recommendation 2: To support the low-income countries EE EE Active where municipal solid waste is growing most rapidly, the Bank Group should identify constraints on demand and investments and leverage external partnerships to implement context-specific municipal solid waste management solutions. Recommendation 3: To bring prominence to and spur EE EE Active action on the global municipal solid waste agenda, the Bank Group should take up a clear leadership position, collaborating and convening with developmental partners. Urban Spatial Recommendation 1: Adopt a framework that links the EE EE Active Growth determinants of urban expansion to pathways for managing urban spatial growth and that contributes to the achievement of SDGs 1 and 11. Recommendation 2: Support World Bank clients with LE LE Active anticipating and preparing for urban spatial growth using preventive approaches, not just curative ones. Recommendation 3: Strengthen and ensure LE LE Active implementation of the World Bank’s protocol to identify and record precise project locations and collect land market data necessary to support clients with managing urban spatial growth. Private Capital Recommendation 2: Expand private capital mobilization EE EE Active Mobilization platforms, guarantees, and disaster risk management products commensurate with project pipeline development (for the Bank Group). Recommendation 3: Develop new products and improve EE EE Active product alignment with the needs of new investor groups and partners (for IFC and MIGA). Private Recommendation 1: IFC and MIGA should continue to EE LE Active Investment in review their financial risk, make more explicit the FCS implications of IFC’s portfolio approach for FCS, and enhance capabilities to address nonfinancial risks to ensure they align with achieving business growth targets and impacts in FCS. Recommendation 2: To focus on the development of EE LE Active bankable projects, IFC and MIGA should further recalibrate their business models, client engagements, and instruments to continuously adapt them to the needs and circumstances of FCS and put in place mechanisms to track their effectiveness for real-time learning. Regional Recommendation 5: Strengthen the design of IDA CD CD Active Integration Regional Window–supported projects to improve the assessment of spillover effects and to generate evidence based on robust indicators. 48 Appendix A Summary Table of Progress Levels Evaluation Management IEG Topic Recommendation FY23 FY23 Status Trade Facilitation Recommendation 1: To enhance effectiveness, the World CD CD Retired Bank Group should promote an approach of complementary (simultaneous and/or sequential) interventions in trade facilitation reforms in countries where trade is a client priority and the Bank Group has a comparative advantage, substantiated by consistent diagnostics. This also requires collaboration between the Bank Group institutions under Maximizing Finance for Development to allow better use of their assets and resources to plan and support reforms that advance the trade facilitation agenda in client countries. Recommendation 2: The Bank Group should identify and EE EE Retired mitigate political economy constraints to trade facilitation reform implementation through systematic application of its tools for stakeholder analysis and consultation (including public private dialogue). This would allow the Bank Group to more consistently use its tools to address risks and build a broad base of support for trade facilitation reforms. Recommendation 3: The Bank Group should EE EE Retired systematically apply a differentiated approach to identify and monitor, where relevant, the public policy objectives of trade regulations relating to public health, safety, the environment, good governance, and formality and the rule of law. State-Owned Recommendation 1: The Bank Group should apply a EE EE Active Enterprises selectivity framework for state-owned enterprise reform support that considers country governance conditions, control of corruption, and sector- and enterprise-level competition. Recommendation 2: The Bank Group should apply the EE EE Active Maximizing Finance for Development and its embedded Cascade approach for state-owned enterprise reform. Public Financial Recommendation 1: World Bank should regularly monitor EE EE Active and Debt the quality of the key pillars of public financial and debt Management management for each IDA-eligible country, possibly through a centralized country-specific public financial and debt management assessment. Recommendation 2: Actively use the previously described LE LE Active assessment to prioritize and sequence World Bank support for public financial and debt management capacity building and reform in IDA-eligible countries. Disruptive Recommendation 1: Where DTT offer opportunities to EE LE Active Technology make progress on the twin goals more effectively or efficiently, ensure that the Bank Group avails itself of those opportunities and addresses, in particular, the risks posed by DTT. Recommendation 2: Build a Bank Group workforce with EE LE Active the skills required to harness DTT opportunities and 49 Appendix A Summary Table of Progress Levels Evaluation Management IEG Topic Recommendation FY23 FY23 Status mitigate DTT risks by identifying DTT-relevant skills, determining gaps in these skills, and filling these gaps. Recommendation 3: Improve the effectiveness and EE EE Active efficiency of World Bank procurement for complex technology projects (World Bank only). Sustainable Recommendation 1: Consideration should be given to EE PC Active Development expanding the countries covered by the Debt Finance Policy Sustainability Enhancement Program beyond those at moderate or high levels of debt distress or in debt distress. IEG recommends applying an additional filter. Recommendation 2: PPAs should emanate from an up-to- LE LE Active date assessment of country-specific debt stress and be set explicitly within a longer-term reform agenda. Recommendation 3: Where PPAs support actions that LE LE Active need be taken regularly (for example, debt reporting to parliament), PPAs should aim for long-lasting institutional reforms rather than relying on one-time actions. Doing Business Lesson 1: Recognizing the powerful motivational effect of LE LE Active reform indicators, especially those that facilitate country rankings, this evaluation notes the limitations in the coverage and guidance offered by any single indicator set on its own and advocates integrating them with complementary analytic tools and indicators. Lesson 2: Recognizing the granularity and specificity of PC PC Active individual reforms in any given country context, the findings from this evaluation suggest that it is better to avoid using business regulatory or similar global indicators as explicit reform objectives or monitoring indicators in Bank Group projects and country strategies focused on improving the business environment. This does not preclude the use of primary data to agreed targets that track and measure critical Bank Group institutional commitments. Lesson 3: Global indicators coverage and specifications LE LE Active are improved if, at regular and predictable intervals, they are updated to reflect learning from research and field experience to (i) improve links to important development outcomes; (ii) strengthen relevance to the experience of the subject of coverage; and (iii) adapt to technological changes in the areas covered by the indicators. Lesson 4: The Doing Business experience indicates the PC PC Active need for mechanisms and safeguards to assure the accuracy and validity of Bank Group global indicator- based reports and related communications, using robust and transparent standards of evidence. Citizen Recommendation 1: As it defines future corporate EE EE Retired Engagement priorities for citizen engagement, the World Bank should reflect in those priorities the need to achieve greater depth and quality of the citizen engagement activities it supports. 50 Appendix A Summary Table of Progress Levels Evaluation Management IEG Topic Recommendation FY23 FY23 Status Recommendation 2: The World Bank should encourage EE EE Retired and support efforts of its regional, country and Global Practices teams to establish, where appropriate, “thick” citizen engagement that is regular and continuous, uses multiple tools, and is embedded in country systems. Recommendation 3: The World Bank should strengthen EE EE Active the monitoring of its citizen engagement activities by systematically adopting results framework indicators that are results oriented. Recommendation 4: The World Bank should seize the EE PC Retired opportunity of the implementation of the ESF to leverage citizen engagement mechanisms—beyond consultations and grievance redress mechanisms—to reach the objectives of managing social risks, strengthening country systems, and promoting social inclusion. Recommendation 5: IFC should ensure that its clients’ EE EE Retired stakeholder engagement activities required by Performance Standard 1 in projects with affected communities are carried out during appraisal and supervision of the projects and systematically documented. Convening Recommendation 1: Scope engagements and EE EE Active Power contributions to major global convening initiatives more deliberatively. Recommendation 2: Enhance how the World Bank and EE EE Active IFC’s internal systems and processes support managing major convening initiatives over their life cycle. Recommendation 3: Improve links between the World EE EE Active Bank’s global and country work. Outcome Recommendation 1: The Bank Group should reform the EE EE Active Orientation country-level results system to ensure that it accurately captures the Bank Group contribution to country outcomes and usefully informs decision-making on country engagements. Global Footprint Recommendation 1: The World Bank should refine its EE LE Active Effectiveness current approach to managing its staffing global footprint by clearly specifying decentralization’s expected outcomes and adopting principles to guide and adjust decentralization decision-making based on evidence. Recommendation 2: The World Bank should mitigate the EE EE Active risks to knowledge flow brought about by decentralization and put in place safeguards to avoid developing country and regional silos. Recommendation 3: The World Bank should establish EE EE Active clear and structured paths to systematically promote locally recruited staff professional and career growth within its overall approach to improving the effectiveness of its global footprint. 51 Appendix A Summary Table of Progress Levels Sources: Independent Evaluation Group; World Bank 2018a, 2018b, 2018c, 2019a, 2019b, 2019c, 2020a, 2020b, 2020c, 2020d, 2020e, 2021a, 2021b, 2021c, 2021d, 2021e, 2021f, 2021g, 2022a, 2022b, 2022c, 2022d. Note: CD = evidence substantiates a change of direction of travel, with reversals unlikely; DTT = disruptive and transformative technologies; EE = emerging evidence of a change in the direction of travel; ESF = Environmental and Social Framework; FCS = fragile and conflict-affected situation; FY = fiscal year; IDA = International Development Association; IEG = Independent Evaluation Group; IFC = International Finance Corporation; LE = limited evidence of a change in the direction of travel; MIGA = Multilateral Investment Guarantee Agency; PC = progress constrained; PPA = performance and policy action; SDG = Sustainable Development Goal. References World Bank. 2018a. Carbon Markets for Greenhouse Gas Emission Reduction in a Warming World: An Evaluation of the World Bank Group’s Support to Carbon Finance. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018b. Engaging Citizens for Better Development Results. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018c. World Bank Group Support to Health Services: Achievements and Challenges . Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019a. Building Urban Resilience: An Evaluation of the World Bank Group’s Evolving Experience (2007–17). Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019b. Grow with the Flow: An Evaluation of World Bank Group Support to Facilitating Trade 2006–17. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019c. Two to Tango: An Evaluation of World Bank Group Support to Fostering Regional Integration. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2020a. Renewable Energy: Evaluation of the World Bank Group’s Support for Electricity Supply from Renewable Energy Resources, 2000–2017. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2020b. State Your Business! An Evaluation of World Bank Group Support to the Reform of State-Owned Enterprises, FY08–18. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2020c. World Bank Group Approaches to Mobilize Private Capital for Development. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2020d. The World Bank Group Outcome Orientation at the Country Level . Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2020e. The World’s Bank: An Evaluation of the World Bank Group’s Global Convening. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021a. The International Development Association’s Sustainable Development Finance Policy: An Early-Stage Evaluation. Independent Evaluation Group. Washington, DC: World Bank. 52 Appendix A Summary Table of Progress Levels World Bank. 2021b. Managing Urban Spatial Growth: World Bank Support to Land Administration, Planning, and Development. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021c. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021d. The Natural Resource Degradation and Vulnerability Nexus: An Evaluation of the World Bank’s Support for Sustainable and Inclusive Natural Resource Management (2009–19). Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021e. World Bank Engagement in Situations of Conflict: An Evaluation of FY10 –20 Experience. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021f. World Bank Support for Public Financial and Debt Management in IDA-Eligible Countries. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021g. World Bank Support to Reducing Child Undernutrition. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2022a. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2022b. Enhancing the Effectiveness of the World Bank’s Global Footprint. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2022c. The International Finance Corporation’s and Multilateral Investment Guarantee Agency’s Support for Private Investment in Fragile and Conflict-Affected Situations, Fiscal Years 2010–21. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2022d. Transitioning to a Circular Economy: An Evaluation of the World Bank Group’ s Support for Municipal Solid Waste Management (2010–20). Independent Evaluation Group. Washington, DC: World Bank. 53 The World Bank 1818 H Street NW Washington, DC 20433