A WORLD BANK GROUP MANAGEMENT REPORT ON IMPLEMENTATION OF IEG RECOMMENDATIONS FY23 LEARNING AND ADAPTING FOR OUTCOMES through the Management Action Record MAR © 2023 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION. Please cite the report as: World Bank. 2023. Learning and Adapting for Outcomes through the Management Action Record 2023: A World Bank Group Management Report on Implementation of IEG Recommendations. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/099453010192327211/IDU0cdec48fe0b73304eb20a44d0dfa31b12 19ae This work is a product of the staff of the World Bank Group. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS. The material in this work is subject to copyright. Because the World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Management has prepared this FY23 MAR. The Independent Evaluation Group reviews this report and prepares its own validation report, the Independent Evaluation Group Validation of the Management Action Record 2023. See more information about the historic Management and Validation reports at https://ieg.worldbankgroup.org/management-action-record. MAR REPORT About the Fiscal Year 2023 (FY23) Management Action Record (MAR) Following the FY20 MAR reform process, this year’s MAR continues to make improvements to provide better information and in a more accessible format. In this document, the overview reflects on the context in which the MAR has been developed with a discussion on the overall assessment and lessons learned in this year’s MAR. This is followed by the highlights of evidence of progress towards outcomes of evaluation recommendations, including the Management self-assessment for each evaluation. The MAR Report ends with conclusions and a look ahead. CONTENTS MAR REPORT Abbreviations iii Acknowledgments v Executive Summary vi Chapter 1. Overview of Performance 1 Chapter 2. Evidence of Progress and Self-Assessment: Highlights 6 Chapter 3. Conclusions and Looking Ahead 23 ANNEXES Annex I. FY23 MAR Management Assessment for Each Recommendation I Annex II. Proposals for Recommendation Retirement V Annex III. Methodology VI TABLES Table O.1. Evaluations by Theme viii Table 1.1. Aggregate Management Assessment of Levels of Progress Achieved 2 Table III.I MAR Assessment Framework FY23 VI FIGURES AND CHARTS Chart 1.1. Self-Assessment by Reporting Year 3 Figure 1.1. FY23 MAR Thematic Groupings and Evaluation Topics 5 Chart 2.1. Positive increase in the Outcome & M&E Quality ratings amongst all HNP 6 quality focused projects Chart 2.2. Significant increase in Renewable Energy Generation and Commitments 9 Chart 2.3. Substantial increase in IFC Upstream Performance 12 Chart 2.4. Increase in share of RW-financed projects with PDO indicators 14 Chart 2.5. Substantial improvement in monitoring Citizen Engagement 19 Chart 2.6. Progress towards reduction in number of Trust Funds 21 Chart 2.7. Increased presence of staff in FCV countries 22 BOXES Box 1.1. The MAR’s Contribution to the WBG Evolution 2 Box 1.2. Pathways of Change: Examples of How Recommendations Are 4 Implemented Box 2.1. PMIF Innovation Pillar - Capturing the Forward-looking Work Related to 8 Carbon Markets Box III.I Criteria for the Retirement of Recommendations for FY23 VI ii ABBREVIATIONS AEPW Alliance to End Plastic Waste GRID Green, Resilient, and Inclusive Development AFE East and Southern Africa Region, WB GSG Global Solutions Group AFW West and Central Africa Region, WB HLO High Level Outcomes AIMM Anticipated Impact Measurement and Monitoring HNP Health, Nutrition, and Population ASA Advisory services and analytics IBRD International Bank for Reconstruction and ASEAN Association of Southeast Asian Nations Development CAFEF Conflict-Affected and Fragile Economies Facility IEG Independent Evaluation Group CCDR Country Climate and Development Report IDA International Development Association CD Change of Direction IFC International Finance Corporation CDMF Career Development and Mobility Framework InfraSAP Infrastructure Sector Assessment Program CE Citizen Engagement IPF Investment Project Financing CEN Country Engagement Note iSOEF Integrated State-Owned Enterprise Framework CESA Citizen Engagement and Social Accountability ISWA International Solid Waste Association CF Carbon Finance ITSTI ITS Technology and Innovation Lab CLR Completion and Learning Review LCR Latin America and the Caribbean Region, WB CMU Country Management Unit LE Limited Evidence CN Concept Note LGAF Land Governance Assessment Framework CODE Committee on Development Effectiveness LIC Low-Income Country CoP Community of Practice LMIC Lower-Middle Income Country COP Conference of Parties LRS Locally-recruited staff COVID-19 Coronavirus Disease 2019 M&E Monitoring and Evaluation CPF Country Partnership Framework MAR Management Action Record CPL City Planning Labs MCPP Managed Co-Lending Portfolio Program CPSD Country Private Sector Diagnostic MDB Multilateral Development Bank CRP City Resilience Program MDTF Multi-donor Trust Fund CSF Climate Support Facility MFD Maximizing Finance for Development DARES Distributed Access through Renewable Energy Scale-Up MIGA Multilateral Investment Guarantee Agency DDP Digital Development Partnership MoU Memorandum of Understanding DE4A Digital Economy for Africa MPA Multiphase Programmatic Approach DFI Development Finance, WB MSWM Municipal Solid Waste Management DPF Development Policy Financing MVP Matrix Vice Presidents DPO Development Policy Operations MW Megawatt DSEP Debt Sustainability Enhancement Program NHSOE Non-honoring state-owned enterprise DT4D Disruptive Technologies for Development Initiative NRM Natural Resources Management DTT Disruptive and Transformative Technologies OPCS Operations Policy and Country Services, WB EE Emerging Evidence PA Prior Action EBRD European Bank for Reconstruction and Development PASA Programmatic Advisory Services & Analytics EEX Energy and Extractives Global Practice PC Progress Constrained EFI Equitable, Growth, Finance, and Institutions PCF Planning, Connecting and Financing EMDE Emerging Markets and Developing Economies PCM Private Capital Mobilization ERC Emission Reduction Credit PDO Project Development Objective E&S Environment and Social PEFA Public Expenditure and Financial Accountability ESF Environmental and Social Framework PER Public Expenditure Review ESG Environmental, Social, and Governance PFDM Public Financial and Debt Management FCS Fragile and conflict-affected situation PforR Program for Results FCV Fragility, Conflict and Violence PFM Public Financial Management FY Fiscal Year PIU Project Implementation Unit GBV Gender-based Violence PLR Performance and Learning Review GCRF Global Crisis Response Framework PMIF Partnership for Market Implementation Facility GEM Global Emerging Market PPA Power Purchase Agreement GEMS Geo-Enabling Initiative for Monitoring and Supervision PPR Pandemic Preparedness and Response GM Grievance Mechanism PS Performance Standards GP Global Practice PSW Private Sector Window GPG Global Public Good RBCF Results-based Climate Finance iii MAR REPORT FY23 RE Renewable Energy TPI Third-Party Implementation RECTF Renewable Energy Catalyst Trust Fund TPM Third-Party Monitoring RI Regional Integration UN United Nations RMS Results Measurement System UNHCR United Nations High Commissioner for Refugees RRA Risk and Resilience Assessment VPU Vice-Presidential Unit RRS Resilience Rating System WB World Bank SAR South Asia Region, WB WBG World Bank Group SCALE Scaling Climate Ambition by Lowering Emissions WHO World Health Organization SCD Systematic Country Diagnostic WPA Work Program Agreement SDFP Sustainable Finance Debt Policy WTO World Trade Organization SE Stakeholder Engagement SEFF Strategic External Funding Framework All dollar amounts are US dollars unless otherwise indicated. SEP Stakeholder Engagement Plan SDG Sustainable Development Goal SOE State-Owned Enterprise SPJ Social Protection and Jobs SRT Sustainability Rating Tool TA Technical Assistance TF Trust Fund TFA Trade Facilitation Agreement TOR Terms of Reference iv MAR REPORT FY23 ACKNOWLEDGMENTS This report is jointly prepared by a team from the World Bank (WB), International Finance Corporation (IFC), and Multilateral Investment Guarantee Agency (MIGA). The work was done under the overall supervision of: Paloma Anós Casero (Director, OPSRL) and Lisandro Martin (Manager, OPSSR), both of Operations Policy and Country Services (OPCS) in the WB; Aisha Williams (Director, CBFDR), and Liane Lohde (Manager, CSMCS) of IFC; and Merli Baroudi (Former Director, MIGES) and Hiroyuki Hatashima (Chief Evaluation Officer and Acting Director, MIGES) in MIGA. Lead specialists guiding the process were Vijay Pillai (OPCS) in the WB and Ayesha Muzaffar (CSMCS) in IFC. The Task Team Leaders from the WB responsible for the overall update and the collaboration with IEG were Diego Garrido Martin and Maurya West Meiers (OPCS), joined by Cristina Marosan Ling, Senior Operations Officer on special assignment to OPCS. Coordinating the update for IFC were Peixuan Zhou (Strategy Analyst, on special assignment to CSMCS) and Arvinth Booma Jayakumar (Strategy Officer, on special assignment to CSMCS); and for MIGA, Vlajko Senic (Consultant, MIGES). In OPCS, Alan Kaufman and Sparsh Kansal provided data analysis, visualization, and overall coordination support; Francesca Gentile was responsible for the overall design; Jacob Bathanti led the copyediting; and Mimoza Velo managed general coordination and scheduling. Debra Ladner and Diego Carrillo, who were OPCS Task Team Leaders for the FY22 MAR, ensured a smooth transition for the FY23 team. The MAR coordination team worked closely with IEG counterparts in a highly productive collaboration, including Stephen Porter, Task Team Leader of the IEG MAR Validation, Arjun Kaushik of the core IEG MAR team, and 22 IEG evaluators. The MAR is informed by inputs of various contributors from across the WBG for the 22 evaluations under review. They include managers and technical focal points across more than 55 units. This report would not have been possible without the technical focal points who met with the MAR coordination team and IEG colleagues to discuss the progress for each recommendation, and then provided qualitative and quantitative inputs to substantiate progress in the implementation of IEG recommendations. v MAR REPORT FY23 EXECUTIVE SUMMARY Following the 2020 Management Action Record (MAR) reforms, World Bank Group (WBG) Management prepares an annual self-assessment of its progress toward implementing the full range of outstanding Independent Evaluation Group (IEG) evaluation recommendations. In turn, IEG reviews the Management’s self-assessment, judging the progress toward achieving the outcomes of active recommendations. As part of the accountability function of the MAR, both Management’s self-assessment and IEG’s review is discussed with the Committee on Development Effectiveness (CODE) annually. The 2023 Fiscal Year (FY23) MAR reports on progress in implementing 59 recommendations from 22 evaluations. 1. The MAR monitors the implementation of evaluation recommendations, which contain relevant insights on priorities for the WBG Evolution. The recommendations involve actions to enhance operational modalities, inform risk taking, improve guidance for staff, and improve results measurement systems. Additionally, the MAR process contributes to the proposed adjustments to the WBG operating model by enhancing the way operational teams internalize lessons from evaluations to maximize the development impact of their interventions. 2. Management responded to feedback from last year’s MAR by assessing progress in the implementation of recommendations in a more rigorous manner. This is reflected in a larger share of the recommendations discussed in the self-assessment described as “limited evidence (LE)” and a smaller share achieving a “change in direction” (CD). Compared to FY22, self-assessment across 59 recommendations shows that 28 percent have either “limited evidence (LE)” or demonstrate “change of direction (CD),” while a larger share, at 68 percent, are in the middle category of “emerging evidence” (EE). Management notes a higher share of LE and lower share of CD assessments in FY23, reflecting higher standards to demonstrate institutionalization of progress. Aggregate Management Assessment of Levels of Progress Achieved (# of Recommendations) Assessment Category FY22 Assessment FY23 Assessment Progress Constrained (PC) NA 3 Limited Evidence (LE) 3 11 Emerging Evidence (EE) 38 39 Change of Direction (CD) 13 6 See Annex III on Methodology for a description of the assessment framework. 3. Management has taken a cautious approach to retiring recommendations. In fast-changing times, the relevance of recommendations for MAR reporting should be reviewed on an ongoing basis. There is also a need to manage the overall volume of recommendations reported in the MAR due to the addition of new evaluations. In FY23, 12 recommendations are due for “automatic” retirement. Some recommendations are proposed to be extended beyond the stage of “automatic” retirement to achieve better outcomes through continued implementation. Management also proposes early retirement of recommendations when the context or priorities have evolved. 4. Implementing evaluation recommendations complements other ongoing WBG efforts to improve development effectiveness. Experience with the MAR process confirms that the intent of the MAR reforms vi MAR REPORT FY23 is being met by contributing to the development effectiveness of the WBG’s work. IEG evaluations identify areas where progress is being made by the WBG, and flag areas for improvement. When recommendations are being well implemented, this strengthens the overall development effectiveness of WBG efforts, and this report shows several examples. 5. Management is implementing measures to further optimize learning from the MAR process. Effective implementation of evaluation recommendations requires that operational teams and Managers understand what is required to demonstrate implementation progress. Starting in FY24, Management will appoint “champions” for each evaluation to identify key milestones to ensure succ ess in implementing recommendations. Identifying champions has an extra value-added for those recommendations that entail cross-World Bank (WB) and cross-WBG efforts, which require multiple players to act in a coordinated manner. To maintain the learning focus of the MAR, it will also be crucial to keep an eye on maintaining a manageable number of active recommendations to monitor. The FY22 MAR had reported on 18 evaluations and 55 recommendations, while the FY24 MAR is expected to report on 26 evaluations and 79 recommendations. Management will also make the MAR process nimbler, and the FY23 MAR after action review will consider this with a view to implementing improvements in the FY24 cycle. vii MAR REPORT FY23 TABLE O.1. EVALUATIONS BY THEME Theme Evaluation  Title # Recommendations Orgs  World Bank Group Support to Health Services: Health Services (FY19) Achievements and Challenges 2  Bank  Investing in Human Capital World Bank Support to Reducing Child Undernutrition (FY22) Undernutrition 2  Bank  Carbon Markets for Greenhouse Gas Emission Carbon Finance (FY19)  Reduction in a Warming World 1  Bank    Building Urban Resilience: An Evaluation of the Urban Resilience (FY20) World Bank Group's Evolving Experience (2007- 4  Bank, IFC  2017) Evaluation of the World Bank Group's support for Renewable Energy (FY21) electricity supply from renewable energy resources, 3  Bank, IFC, MIGA  2000-2017 Promoting Natural Resource Degradation and Vulnerability Climate and Natural Resource Degradation Nexus: An Evaluation of the World Bank's Support for Resilience Sustainable and Inclusive Natural Resource 3  Bank  (FY21)  Management (2009-2019) WBG Engagement in World Bank Engagement in Situations of Conflict: An 4  Bank  Situations of Conflict (FY22)  Evaluation of FY10-20 Experience Managing Urban Spatial Growth World Bank Support Urban Spatial Growth (FY22) to Land Administration, Planning, and Development 3  Bank  Municipal Solid Waste Transitioning to a Circular Economy 3  Bank, IFC, MIGA  Management (FY22) Private Capital Mobilization The World Bank Group's Approach to the Mobilization of Private Capital for Development: An 2  Bank, IFC, MIGA  Crowding in (FY21)  IEG Evaluation Private The International Finance Corporation’s and Finance Private Investment in Multilateral Investment Guarantee Agency’s Support for Private Investment in Fragile and Conflict 2  IFC, MIGA    FCS (FY22) Affected Situations, Fiscal Years 2010–21 Two to Tango: An Evaluation of World Bank Group Regional Integration (FY19) Support to Fostering Regional Integration 1  Bank  Grow with the Flow: An Independent Evaluation of Trade Facilitation (FY19)  World Bank Group Support to Facilitating Trade 3  Bank, IFC  2006-17 State Your Business! An Evaluation of World Bank SOE Reforms (FY21) Group Support to the Reform of State-Owned 2  Bank, IFC, MIGA  Boosting Long- Enterprises, FY08-18 Term Public Finance and Debt World Bank Support for Public Financial and Debt Economic Management in IDA-Eligible Countries 2  Bank  Management (FY21) Growth Mobilizing Technology for Development: An Disruptive Technology (FY21) Assessment of World Bank Group Preparedness 3  Bank, IFC  Sustainable Development The International Development Association’s Sustainable Development Finance Policy: An Early- 3  Bank  Finance Policy (FY22) Stage Evaluation The Development Effectiveness of the Use of Doing Doing Business Report (FY22) Business Indicators, Fiscal Years 2010–20 4  Bank, IFC  Citizen Engagement (FY19) Engaging Citizens for Better Development Results 5  Bank, IFC  Improving The World's Bank: An Evaluation of the World Bank Convening Power (FY20) Group's Global Convening 3  Bank, IFC  WBG Capacity to Meet Its The World Bank Group Outcome Orientation at the Outcome Orientation (FY21) County Level: An Independent Evaluation 1  Bank, IFC, MIGA  Goals Global Footprint Effectiveness Enhancing the Effectiveness of the World Bank’s Global Footprint 3  Bank  (FY22)  viii MAR REPORT FY23 1. OVERVIEW OF PERFORMANCE A Dynamic External Environment and an Evolving Internal Context  1.1 The 2023 Fiscal Year (FY23) Management Action Record (MAR) was carried out during a challenging and uncertain period for World Bank Group (WBG) clients. Management undertook the FY22 MAR process when the number of global COVID-19 cases was unacceptably high. The decline in infections during recent months led the World Health Organization (WHO) to declare in May 2023 that COVID-19 was no longer a pandemic. However, various interlinked and reinforcing factors that cut across borders, such as Russia’s invasion of Ukraine, the global food crisis, climate effects, reversed progress on human development, rising inflation, slowdown in growth, and the possibility of a “hard landing” from the economic crisis, contributed to a difficult and uncertain period for World Bank (WB) clients—both governments and private sector. The crises are straining developing countries’ capacities and resources and causing disturbing reversals in development gains. This difficult context reinforces the need for even sharper attention to ensuring the effectiveness of domestic and external resources—by focusing on development priorities and achieving lasting impact. 1.2 The WBG provided a significant crisis response to support clients to cushion the impact of the multiple crises. By the end of FY23, the WBG delivered $199 billion, much higher than the planned $170 billion promised in the Global Crisis Response Framework (GCRF) during the period April 2022 to June 2023. This reflects a sizeable growth in commitments (over 50 percent for the WB) since the period before the pandemic—thus making critical financing available to client countries and companies at a time when global flow of capital has been declining to the developing world. At the same time, WB portfolio quality improved, with a record $51 billion in disbursements during FY23, an uptick in the share of projects rated by the Independent Evaluation Group (IEG) as marginally satisfactory or above (from 86 percent to 91 percent during the last two FYs) and a similar improvement in development outcomes of Country Partnership Framework (CPFs). The achievement of WBG results in the Corporate Scorecard demonstrates the scale of development results. These are indicators of stepped-up performance by the WBG and balancing support between immediate and medium-term development needs. 1.3 The MAR monitors the implementation of evaluation recommendations that contain relevant insights on key priorities for the WBG Evolution. In response to the rapidly changing nature of global development challenges, WBG shareholders asked for an Evolution process to respond to these challenges and provide scaled-up finances. The WBG Evolution emphasizes dealing with pandemics, climate change, cross-border challenges, and private capital mobilization (PCM), while also strengthening the operating model to address these global challenges within the WBG’s country-based model. Some of the evaluations cover these topics, including pandemics, global convening, renewable energy (RE), disaster risk, energy efficiency, regional integration, conflict, PCM, the work of the International Finance Corporation (IFC) and Multilateral Investment Guarantee Agency (MIGA) on PCM in Fragile, Conflict and Violence (FCV) environments, and the WBG’s global footprint. The recommendations in these evaluations relate to strategic issues, operational improvements, and better monitoring of results—all of which are relevant to taking forward the WBG Evolution priorities (see Box 1.1). 1 MAR REPORT FY23 Emerging Picture from the FY23 Overall MAR Self-Assessment 1.4 The WBG has made good implementation progress in FY23 despite the challenges of the polycrises and increased standards for demonstrating progress in the MAR. Management has scrutinized evidence of progress more carefully, which has resulted in a more balanced distribution along the assessment categories1 (see Table 1.1). Compared to FY22, Management notes a higher share of “limited evidence” (LE) assessments and lower share of “change of direction” (CD) assessments in FY23, reflecting a higher bar to demonstrate institutionalization of progress. This year’s assessment includes a new category of “progress constrained” (PC), following a request from the Committee on Development Effectiveness (CODE). Self-assessment of LE in Year 2 or later points to the need for closer attention to implementation performance. See Annex I for the self-assessment for each recommendation and Annex III, on Methodology, for a description of the assessment framework. TABLE 1.1. Aggregate Management Assessment of Levels of Progress Achieved (# of Recommendations) Assessment Category FY22 Assessment FY23 Assessment Progress Constrained (PC) NA 3 Limited Evidence (LE) 3 11 Emerging Evidence (EE) 38 39 Change of Direction (CD) 13 6 1  The MAR assessment framework contains four categories. See Annex III for definitions of criteria and how they are applied. Briefly explained, progress constrained (PC) means that evidence on progress is unavailable or inadequate; limited evidence of progress (LE) is generally used when initial activities linked to the recommendation are delivered and some knowledge generated; emerging evidence of progress (EE) describes cases in which new capacities and systems are being developed and have started to be implemented; and change of direction of progress (CD) shows cases with meaningful changes in behavior and systems that are likely to be sustained in the future. 2 MAR REPORT FY23 1.5 Analysis of the “time-effect” of implementation progress shows a nuanced picture—not all recommendations need five years to reach CD and neither do all recommendations reach CD at the end of five years. This year’s MAR update further confirms that the implementation of recommendations travels at variable speeds over time, with initial activities leading to outputs and early outcomes, and eventually progressing to systems change, but with the CHART 1.1. Self-Assessment by Reporting Year amount of time for these stages differing. Chart 1.1 shows the assessment grouped by the reporting Number of Recommendations year for the evaluations. Of CD the 24 recommendations that are in Year 1 of 13 EE reporting, 11 are either in 2 the LE or PC category— LE which is to be expected. 3 However, a more nuanced 12 PC 9 1 picture emerges by 9 examining the assessments 5 in Years 2 and 3 of the 2 2 1 1st Year 2nd Year 3rd Year 4th Year reporting cycle with more in the EE category and fewer in LE. 1.6 Management identified various “pathways” toward change that meet the realities in each context. While by no means exhaustive, this MAR cycle highlights four pathways toward change that have been implicitly followed by various teams (these pathways have been formalized as described in Box 1.2). A typical “step-by-step” pathway of implementation step by step involves the following steps: preparation of new guidance (e.g., GP issuing formal guidance), training of staff on the new guidance, initial application of the new guidance (i.e., trialing period), and scaling-up for broader coverage (when they meet the “change of direction” criteria). Another pathway, “riding the wave” would be most relevant when an evaluation recommendation benefits from considerable institutional momentum behind an agenda, so that just “riding the wave” could deliver quick results (e.g., the recommendations from the RE evaluation). Yet another pathway, “being opportunistic” could work when hooking a railcar to a moving train—that is, when teams leverage other initiatives that are already underway—could help with steady implementation success (e.g., regional integration recommendations benefiting from International Development Association (IDA) policy changes). For recommendations dependent on “cross-unit collaboration,” the pathway may be less clear, notwithstanding progress being made (e.g., the evaluation on convening power). On a related note, introduction of the new core diagnostic on the Country Climate Development Report (CCDR) has made a direct contribution to strengthening implementation of several recommendations across evaluations—demonstrating how a corporate initiative can “lift many boats.” 3 MAR REPORT FY23 BOX 1.2. Pathways of Change: Examples of How Recommendations Are Implemented Retirement of Recommendations 1.7 Management follows a pragmatic while cautious approach to monitoring and tracking implementation efforts and making proposals for retirement. A recommendation is retired when an assessment of CD is reached, or the recommendation has been implemented for five years, or a significant change in external or internal context makes the recommendation redundant and is proposed for early retirement. There are 12 recommendations in five evaluations that have been implemented for five years and are due for automatic retirement. These evaluations are for health services, carbon finance (CF), regional integration, trade facilitation, and citizen engagement. In all five evaluations, the proposals for retirement of 14 recommendations are as follows: a) of the 12 recommendations due for automatic retirement, Management proposes to retire 10 and extend two others for another year of reporting; b) retirement of three other recommendations upon reaching a CD assessment; and c) early retirement of one recommendation due to strategic prioritization issues (for details, please see Annex II). Despite the careful approach to retirement, Management notes the challenge with the growing number of recommendations in the MAR cycles. The FY24 MAR update is expected to report on 26 evaluations and 79 recommendations—representing an over 50 percent increase in a three-year MAR reporting cycle (FY22- FY24)—making the lesson-learning objective of the MAR more challenging. 1.8 Management’s proposals for retirement are based on considerations of alignment with the WBG Evolution and other strategic priorities. The two recommendations being proposed for another year of reporting beyond the automatic retirement stage pertain to citizen engagement and regional integration. The recommendation on citizen engagement is about strengthening the use of results-oriented indicators in operations, which resonates with Management’s emphasis on strengthening the overall results architecture of the WBG and would allow the recommendation to reach CD before it is fully retired next year. The recommendation on regional integration being proposed for another year of reporting is about strengthening the measurement of cross-border spillover benefits. Even though the self-assessment shows “change of direction,” thus qualifying for retirement, Management is proposing reporting for another year on an exceptional basis as it sees value in this recommendation contributing to the WBG Evolution by providing important learning on measurement of cross-border spillover benefits (including for Global Public Goods (GPGs)). 4 MAR REPORT FY23 FIGURE 1.1. FY23 MAR THEMATIC GROUPINGS AND EVALUATION TOPICS CROWDING IN BOOSTING PRIVATE FINANCE LONG-TERM Two evaluations: on ECONOMIC PRIVATE private capital FINANCE GROWTH mobilization, and on Seven evaluations: on private investment in regional integration, FCS. trade facilitation, SOE LONG-TERM ECONOMIC reforms, public finance PROMOTING GROWTH and debt management, CLIMATE AND disruptive technology, RESILIENCE and doing business, and Seven evaluations: on sustainable carbon finance, urban CLIMATE development finance resilience, renewable AND RESILIENCE policy. energy, natural resource degradation, urban spatial growth, municipal solid waste management, CORPORATE EFFECTIVENESS and WBG engagement in situations of conflict. IMPROVING WBG CAPACITY TO MEET ITS GOALS INVESTING IN HUMAN Four evaluations: on HUMAN CAPITAL CAPITAL citizen engagement, Two evaluations: on convening power, health services, and on outcome orientation, undernutrition. and global footprint effectiveness. THEMES 5 MAR REPORT FY23 2. EVIDENCE OF PROGRESS AND SELF-ASSESSMENT: HIGHLIGHTS Throughout this section, for each evaluation, the main topic of each recommendation is shown in bold text. World Bank Group Support to Health Services: Achievements and Challenges FY19. Two of the four recommendations were retired having reached CD rating. Final year of reporting. World Bank only. 2.1 Management assesses both recommendations on health services can be retired, and further progress on pandemic preparedness will be reported in future MAR updates under the COVID-19 evaluation. Improvements in measurement of quality of health services and distributional effects is based on demonstrated progress in a) increased uptake—all FY23 projects on improving the quality of health services have at least one indicator for service quality and distributional effects by gender and/or low- income area; b) portfolio performance on Monitoring and Evaluation (M&E) quality of health quality- oriented projects—which improved from 41 percent before the evaluation to 96 percent (See Chart 2.1); c) further customized provision of guidance and synergy between global and country efforts; d) training of clients and staff—around 1,000 staff and clients trained in improved M&E skills. On pandemic preparedness, the World Bank (WB) mounted the largest multilateral response to COVID-19 and was pragmatic in balancing emergency response with strengthening medium-term country capacity on preparedness. Management assessment is based on the strength of the efforts to build surveillance, testing and tracing capacity, scaled-up financing for country and regional efforts, and fostering of partnerships. The COVID-19 Multiphase Programmatic Approach (MPA) operations have been successful in building Pandemic Preparedness and Response (PPR) capacity by strengthening national surveillance systems for infectious diseases in 25 countries and improving prevention and response planning through the development of national emergency contingency plans in 26 countries. This will be a priority under the WBG Evolution. 6 MAR REPORT FY23 World Bank Support to Reducing Child Undernutrition FY22. Two recommendations. First year of reporting. World Bank only.    2.2 Management assesses that both recommendations on undernutrition are making progress consistent with Year 1 reporting. There has been increased attention to institutional strengthening as demonstrated by a) performance of the FY22-23 operations—of the 57 projects with nutrition components, 86 percent prioritized institutional strengthening, and b) analytical work and preparation of guidance on capacity-building efforts and improved sub-national targeting underpinned by a recent Health, Nutrition, and Population (HNP) Global Practice (GP) discussion paper on “Operationalizing Multisectoral Nutrition Programs to Accelerate Progress: A Nutrition Governance Perspective ” and the Optima Nutrition Learning Tool. A good example of a country program taking a systematic multisector approach to undernutrition is Rwanda, which has three projects led by HNP, Social Protection and Jobs (SPJ), and Agriculture, and is further supported through a Development Policy Operation (DPO). The assessment on increased nutrition- specific interventions and focus on social norms is based on a) analysis of FY23 operations—total commitments for IBRD/IDA nutrition-specific projects steadily increased from $312 million in FY20 to an estimated $937 million in FY23, b) analytical work and preparation of guidance and capacity building efforts—including updating the 2017 Investment Framework on Nutrition to reflect new evidence, and c) delivered operations and Advisory Services and Analytics (ASA) on social and behavioral changes in nutrition and gender across regions. Carbon Markets for Greenhouse Gas Emission Reduction in a Warming World FY19. Four of the five recommendations have been retired having reached CD assessment. Final year of reporting. World Bank only. 2.3 The CF agenda is set to receive a further boost under the WBG Evolution, and Management assesses there is strong basis to retire the remaining recommendation for the WB. Management assesses that there is “Emerging Evidence” (EE) of progress in attracting and mobilizing finance, as the reforms proposed in the evaluation have mostly been carried out. Management notes that the building blocks for moving to the next stage of attracting investments are in place, through building readiness and implementation experience with carbon pricing; piloting Emission Reduction Credits (ERCs) from climate mitigation projects; supporting the creation of CF market infrastructure; building country capacity to make informed decisions related to the role of markets; providing a market signal of the quality of ERCs; and exploring financing structures that could enhance the impact of results-based climate finance (RBCF) and carbon revenues. Three umbrella trust funds were created and are being capitalized to consolidate and aid in the WB’s CF efforts, including to leverage private investments: Partnership for Market Implementation 7 MAR REPORT FY23 Facility (PMIF), Scaling BOX 2.1. PMIF Innovation Pillar – Capturing the Forward-looking Work Related Climate Ambition by to Carbon Markets Lowering Emissions (SCALE), and Climate Support Facility (CSF). See 2.1 for information about the PMIF Innovation Pillar and how it captures the forward-looking work related to carbon markets. Building Urban Resilience: An Evaluation of the World Bank Group's Evolving Experience (2007-2017) FY20. One of the five recommendations was retired having reached CD assessment. Third year of reporting. World Bank and IFC. 2.4 Given rapid levels of urbanization and the challenge with resilient growth, the WB and the International Finance Corporation (IFC) have systematically followed up on strengthening the quality of engagement to address urban resilience. The main tools and approaches for tracking interventions and building resilience into all stages of the project cycle at the WB include the Resilience Rating System (RRS), the CCDRs, the IDA Policy Commitments and Results Measurement System (RMS) Indicators, and the Climate Co-Benefits Dashboard. For the indicator “People provided with improved urban living conditions,” which includes “people living in urban areas that have been provided with access to improved…resilience and/or urban environmental conditions,” results for the 18th replenishment of IDA (IDA18) showed 15.6 million people reached over three years, IDA19 held steady with 10-12 million over two years, and IDA20 expects 15-20 million over three years, a significant increase. Thirteen CCDRs drew on analysis of city resilience challenges and climate-related hazards, covering 128 cities. Through a variety of approaches, the WB is mainstreaming and incorporating resilience characteristics throughout the project cycle—on design standards, cost-benefit analyses, and inclusive approaches for vulnerable people—while noting that more could be done on interjurisdictional coordination. Management recommends retirement of the recommendation on crime and violence as it would allow sharper focus on several other recommendations on urban issues. For IFC, three Investment and seven Upstream/Advisory projects that support urban resilience were identified and tracked through IFC’s Cities Project Database. An internal guidance note was developed to improve the identification and articulation of resilience elements in IFC projects and a training on the guidance was conducted to promote its consistent use. IFC continues to offer support to strengthen urban infrastructure resilience for its clients, including through its Cities Initiatives, the global Building Resilience Index (BRI) program, or other tools and approaches, such as an upstream resilience methodology to identify resilience-related projects. 8 MAR REPORT FY23 Evaluation of the World Bank Group's support for electricity supply from renewable energy resources FY21. Three recommendations. Second year of reporting. World Bank, IFC, and MIGA.  2.5 Renewable energy (RE) continues to be a strategic institutional priority and the WB, IFC, and the Multilateral Investment Guarantee Agency (MIGA) have respectively made rapid progress across all recommendations. The progress in RE at the WB is reflected in over 60 percent of the Energy and Extractives (EEX) GP’s FY23 lending operations that supported RE integration (see Chart 2.2 related to the WB’s increased RE commitments and projects, including battery storage development). During the reporting period, IFC committed more than $2 billion in climate finance, much of which will support RE development. IFC increased review of several hybrid renewable energy and battery storage projects across diverse markets. As of March 2023, six projects with a cumulative size of 1,189 megawatts (MW) renewables and at least 127MW/471 megawatt hours (MWh) of battery storage have been mandated; three projects with a cumulative size of 334MW of solar photovoltaic (PV) and 334MW/344MWh of battery storage have passed concept review. In FY23, MIGA committed $1.528 billion in climate finance, most of which is supporting integration of RE sources. Moreover, MIGA’s Renewable Energy Catalyst Trust Fund (RECTF) is now fully operational, with a dedicated window to support renewable energy projects. For example, the RECTF enabled MIGA’s support of a solar plus storage project in Somalia. On scaling up support for RE, the combination of diagnostics and financing at the WB fosters the establishment of enabling policy, regulatory and institutional conditions for RE, to mitigate project-level risks and to provide financing support consistent with the Maximizing Finance for Development (MFD) approach. IFC scaled up its investment in RE with approximately $1.1 billion (own- account and mobilization) in solar and wind so far in FY23, and $244 million in commercial and industrial distributed generation as new ways to further expand RE. IFC’s innovative finance offerings, such as green and sustainability-linked finance in the energy space, reached $1.5 billion (own-account mobilization), showing a 79 percent increase from FY22. Based on the comparative advantages of each institution, WB, IFC, and MIGA have been working together on several initiatives in multiple countries to further scale up RE (e.g., through the Scaling Solar program, the Offshore Wind Development program, and the Distributed Access through Renewable Energy Scale-Up (DARES) platform). The WBG is simultaneously expanding the pool of staff with specialized RE knowledge (EEX hired 28 GG and GF level positions, which included the requirement of RE skills and/or highlighted RE as part of job responsibilities in the Terms of Reference (TORs) and is structuring the teams in line with RE priorities). All institutions are fostering RE knowledge through continually developing knowledge products and offering learning activities. 9 MAR REPORT FY23 The Natural Resource Degradation and Vulnerability Nexus: An Evaluation of the World Bank's Support for Sustainable and Inclusive Natural Resource Management (2009-2019) FY21. Three recommendations. Second year of reporting. World Bank only.  2.6 There was mixed progress on implementing recommendations in Natural Resources Management (NRM), with more progress in integrating resource governance challenges into operations. On leveraging analytical work into country engagement products, there was headway in reflecting the resource vulnerability nexus in CCDRs (with inputs provided to 14 of the 24 FY23 CCDRs) and provision of inputs into Systematic Country Diagnostics (SCD)/CPF cycle. The country engagement products will continue to reflect client demand and selectivity. On integrating natural resource governance, there was advancement in systematically using Green, Resilient, and Inclusive Development (GRID) DPOs as primary instruments to support governments in adopting policies that address governance issues and provide incentives for improved NRM, with five DPOs with over a $1.25 billion approved during the last 12 months. In the agricultural sector, the WB is seeking to repurpose environmentally and socially damaging subsidies toward nature-positive and climate-resilient practices. The WB continues to promote cross-GP engagement on NRM to break silos, share knowledge, enhance measurement, and improve coordination, although this area will require additional efforts to establish more structured collaboration in the coming years. World Bank Engagement in Situations of Conflict: An Evaluation of FY10-20 Experience FY22. Four recommendations. First year of reporting. World Bank only. 2.7 The WB made progress across all recommendations in the work on fragile and conflict affected situations (FCS). To enhance conflict sensitivity in its engagement, the WB strengthened the Risk and Resilience Assessments (RRAs), which are the WB’s primary diagnostic tool to identify FCV drivers and enhance FCV sensitivity. The WB made RRAs accessible to all staff working in FCV environments, compared to earlier years, by publishing all 19 RRAs conducted since January 2021 on the intranet. RRAs are carried out on a regular basis to inform country engagement products like SCDs and CPFs. An ex-post review of all IDA FCS SCDs, CPFs, Country Engagement Notes (CENs) and Performance and Learning Reviews (PLRs) covering FY21 and FY22 show that all 11 CPFs/CENs/PLRs benefited from updated RRAs. To complement the RRAs, the WB developed a suite of tools including crisis risk monitoring, conflict tracking activities (with the International Crisis Group (ICG), for example) and FCV portfolio analysis to inform country operations. The WB enacted changes at the policy, project, and staff levels, to better position itself to engage in situations of conflict by removing bottlenecks and disincentives. The updated operational policy on FCV removes previously perceived ambiguities about the mandate to engage in conflict situations. Operational flexibilities were rolled out for FCV contexts in response to paragraphs 12 and 13 of the Policy on Investment Project Financing addressing Situations of Urgent Need of Assistance or Capacity Constraints and the revised procurement guidance. In recent years, the WB invested heavily in third-party implementation (TPI) and monitoring (TPM). The Work Program Agreement (WPA) budget for FCS projects is higher than regular projects and Regions routinely allocate more resources to Country Management Units (CMUs) covering FCS countries to account for higher operating costs. WB revised its approach to what success means in conflict-affected countries, with the new approach embedded in the roadmap on Strengthening World Bank Group Outcome Orientation in June 2021 through a new lens on country engagement including guidance on FCV countries. 10 MAR REPORT FY23 Transitioning to a Circular Economy: An Evaluation of the WBG’s Support for Municipal Solid Waste Management (2010-20) FY22. Three recommendations. First year of reporting. World Bank, IFC, and MIGA.  2.8 The WBG made progress on improving municipal solid waste management (MSWM) in client countries. To promote the adoption and implementation of waste hierarchy practices, the WB developed 22 projects with MSWM components since FY21, six of which are stand-alone projects, which have specific reference to waste hierarchy. The steady growth in stand-alone MSWM projects addressing the entire waste sector chain is taking place in an integrated and incremental manner in line with client needs and capabilities. IFC already implemented the Waste-to-Value approach (Circularity Plus Platform) to assist solid waste management companies with tailored investment and advisory solutions. Support from the Circularity Plus platform was provided to IFC clients in various countries, with strong demand emerging from countries where local policy and regulations are pushing for higher circularity, though markets are still nascent. The WB’s policy work and IFC’s upstream engagements aim to reduce constraints on cross-border private sector participation, supporting potential investors with MIGA guarantees at the project level. To identify constraints for implementing context specific MSWM solutions in Low-Income Countries (LICs), the WB completed a survey and analysis of MSWM with CMUs (country director, country manager, or operations manager) in LICs and Lower Middle-Income Countries (LMICs). Responses from seven LICs and 13 LMICs were used to develop follow-up actions on what ASAs can do to facilitate discussions on prioritizing and investing in the MSWM sector. IFC completed an upstream study in partnership with the French development finance institution PROPARCO to screen opportunities to improve solid waste management in selected cities in 17 African countries, including LICs such as Mozambique, Rwanda, and Zambia. Results from this study will assist in exploring context-specific solutions to address the collection and treatment issues of municipal waste in Africa. With regard to taking a leadership role with development partners in MSWM, the WB established the Southeast Asia Regional Program on Combating Marine Plastics (SEAMAP) to convene the Association of Southeast Asian Nations (ASEAN) member states and regional institutions around a shared interest in combating plastic waste pollution in Southeast Asia. IFC is engaging with development partners, such as the International Solid Waste Association (ISWA) and the Alliance to End Plastic Waste (AEPW), to promote the transition and adoption of sustainable waste management in the context of climate change and in specific areas. Managing Urban Spatial Growth: World Bank Support to Land Administration, Planning, and Development FY22. Three recommendations. First year of reporting. World Bank only. 2.9 Management assesses that, for this first year of review, progress was made on all recommendations and there is likelihood of further growth in the coming years. On having a framework to help city leaders make informed decisions for sustainable urban development, the WB has three strategic documents (frameworks) that work in concert to this end: (1) Planning, Connecting and Financing (PCF) Framework; (2) Housing Strategy/Framework; and (3) Land Governance Assessment Framework (LGAF). The WB helps clients prepare for urban spatial growth through preventative, not just curative, approaches through a series of interventions, including: the launch of City Planning Labs (CPL) Global, a new initiative that uses a Municipal Spatial Data Infrastructure approach and scales up the original CPL approach that was developed in Indonesia to at least seven cities around the world, based on client demand from India, Nepal, Kenya, Vietnam and others; the newly designed City Resilience Program (CRP) Framework Agreement for Risk-Informed Land Use Planning; and the City Climate Finance GAP fund, which is supporting an increasing number of urban teams to conduct upstream analysis around city spatial growth to inform lending operations. Moreover, various global knowledge products such as “Mobilizing Finance through Anticipating 11 MAR REPORT FY23 the Economic Impact of Urban Infrastructure” provide project teams with case studies and distinct methodologies To strengthen implementation of the protocol to identify and record precise project locations and collect land market data to support clients with managing urban spatial growth, including in the expansion and use of the Geo-Enabling Initiative for Monitoring and Supervision (GEMS), the GEMS Portfolio Mapping constitutes the only systematic approach to collect granular spatial project data in the WBG (ITS also runs a “spatial” project platform, called “Global Reach,” which is connected to the portal). GEMS significantly increased its work over the past year and expanded beyond FCS to cover all types of client country globally. As of March 2023, GEMS supported over 1,000 projects across sectors in 100+ countries (with at least 85 projects in 40 countries as part of the Urban, Disaster Risk Management, Resilience and Land GP (GPURL) portfolio), with Project Implementation Units (PIUs) trained in using cost- free and simple open-source technology for geo-tagged project data. Beyond GEMS, geospatial data acquisition and image processing is at the center of GPURL efforts to modernize project preparation and implementation support. World Bank Group Approaches to Mobilize Private Capital for Development. An Independent Evaluation. FY21. Two recommendations. Second year of reporting. World Bank, IFC, and MIGA.  2.10 There is continued progress on PCM across the WB, IFC, and MIGA. This is a priority in the WBG Evolution. The WB’s self-assessment is based on the progress in designing PCM platforms and associated analytics (e.g., Infrastructure Sector Assessment Program (InfraSAP)), new guidance to staff, promoting new scaling approaches including in renewables, new risk sharing facilities for small and medium infrastructure development, and new operations aligned with the PCM approach. IFC committed $15.0 billion in core mobilization as of June 30, 2023, which is the highest level for any fiscal year. IFC expanded PCM platforms (i.e., MCPP One Planet) and is developing innovative platforms, such as the Warehouse-Enabled Securitization Program (WESP), to unlock institutional capital for development finance at scale. IFC rapidly scaled its Upstream capacity- and operations- enhancing project pipeline development with a 500 percent increase in the Upstream pipeline over the past three fiscal years (see Chart 2.3). By February 2023, IFC reached $32.3 billion in own-account investment potential and an additional $33.7 billion in mobilization potential. Of this pipeline, climate represents 47 percent and IDA17-FCS markets represent 37 percent, providing a strong foundation for investment delivery in the near term. All MIGA guarantees contribute 12 MAR REPORT FY23 to mobilizing private capital. MIGA mobilized $5.6 billion of private capital in FY23, of which approximately 90 percent was direct and 10 percent was indirect. MIGA also catalyzes private capital through reinsurance with the private reinsurance market, and in FY23, $5.1 billion of MIGA’s gross issuance of $6.4 billion was reinsured. IFC and MIGA developed new mobilization products. Specifically, IFC prioritized mobilizing banks, institutional investors, and impact investors into groundbreaking sustainability-focused products, and is leveraging its world-class standards in impact measurement and reporting, and in Environmental and Social (E&S) risk management to attract more investors and allocate more funds to Emerging Markets and Developing Economies (EMDEs). IFC is also working with development partners that make up the Global Emerging Markets (GEMs) Risk Database Consortium to conduct a survey of private investors and develop a plan to make the GEMs investment risk data more broadly available to the market. MIGA also is working to advance its innovation agenda by adapting its PRI product to catalyze the carbon credits market, collaborating with private sector insurers to develop a parametric insurance product, and innovating PRI structures to attract institutional investors by working with other Multilateral Development Banks (MDBs). The International Finance Corporation’s and Multilateral Investment Guarantee Agency’s Support for Private Investment in Fragile and Conflict Affected Situations, Fiscal Years 2010–21 FY21. Two recommendations. First year of reporting. IFC and MIGA. 2.11 IFC and MIGA demonstrated growth in their business in FCS, challenges notwithstanding. IFC committed $930 million in Long-Term Finance (own-account) and $2.994 billion in Short-Term Finance in FCS in FY23. Additionally, IFC pursues a proactive advisory role to increase investible opportunities. For example, the flagship “Local Champions Initiative” in the Sahel and West African sub-region aims to create a pipeline of investable transactions through tailored technical assistance. MIGA's guarantee issuance in FCS reached $1.232 billion in FY23, its highest single year volume in MIGA’s history. On reviewing financial risks, IFC conducts regular assessments and engages with the Board on issues related to financial risk, cost of doing business, and capital requirements, all of which could potentially influence IFC’s capacity to grow its business in line with FCS targets. Additionally, IFC utilized blended finance solutions and its portfolio approach to mitigate perceived high risks in FCS. To mitigate financial risks in FCS countries, MIGA used its underwriting expertise to carefully structure projects, appropriately used blended finance, especially the IDA Private Sector Window (PSW) and the Conflict-Affected and Fragile Economies Facility (CAFEF), and reinsured FCS projects in the private reinsurance market. To better address non-financial risks, IFC improved its contextual risk screening and analysis to identify and mitigate broader country risks as part of project E&S due diligence. Notably, IFC Environmental, Social, and Governance (ESG) Advisory has in-depth country-level interventions to tackle systemic ESG risks at firm, market and/or regulatory levels in FCS countries. Country Private Sector Diagnostics (CPSDs) serve as a key tool to identify near-term opportunities for private sector engagement, and to develop recommendations of reforms and policy actions to mobilize private investment and drive solutions to key development challenges in FCS. CPSDs are increasingly considering political economy and conflict, with guidance notes on how to address such issues in CPSDs under development. In addressing non-financial risks, MIGA increased its E&S staff, including staff with strong expertise in Gender-based violence (GBV); enhanced its tools for mitigating GBV risks; and continued to deploy its contextual risk screening applications. To focus on the development of bankable projects in FCS, IFC expanded its Upstream instruments. As of Q3FY23, FCS accounted for $3.283 billion of the pipeline compared to $2.2 billion in March 2021. Upstream developed and deployed additional instruments to fill gaps in IFC’s offering to test new ideas more nimbly (Upstream Seeds projects), convene stakeholders to help build actionable pipelines of opportunities (ideation workshops), standardize, and deploy services at 13 MAR REPORT FY23 scale to grow the pipeline rapidly and efficiently (globally replicable products), and deploy IFC resources for project preparation and development (Collaborations and Co-developments). “Upstream Snapshots” integrate data from advisory, investment, and budgetary systems to enable continuous and dynamic monitoring. To adapt to the realities and dynamic challenges in FCS, IFC launched the Africa Fragility Initiative, a $74 million five-year program to scale-up support for the development and implementation of investment, upstream, and advisory programs in 32 FCS countries. Other IFC initiatives helped identify private sector investment opportunities in challenging FCS regions, such as CPSDs in FCS, and IFC’s partnership with the United Nations High Commissioner for Refugees (UNHCR). MIGA continues its efforts to develop projects in FCS countries by partnering closely with the WB and IFC, and with other MDBs (e.g., European Bank for Reconstruction and Development (EBRD) in support of Ukraine), and by ensuring that year-end staff awards include a focus on FCS projects. Two to Tango: An Evaluation of World Bank Group Support to Fostering Regional Integration FY19. Five of the six recommendations have been retired. Fourth year of reporting. World Bank only. 2.12 The Regional Integration evaluation prompted a helpful exercise to codify spillover effects of regional programs. The reporting is at three levels. At the IDA-level: the IDA20 Regional Window Implementation Guidelines were updated to add a strict eligibility criterion for every operation seeking funding from the Regional Window (RW) to include at least one Project Development Objective (PDO) level indicator tracking a transboundary spillover effect. This is verified at the time of the Decision Meeting when Development Finance (DFI) clears the project for access to window financing. The change builds on a detailed Guidance Note on Demonstrating Regional Externalities, which was produced and distributed in FY22, and annexed to the IDA20 Regional Window Implementation Guidelines. At the Operational level: since the start of FY23, DFI cleared projects for RW financing if they have at least one PDO- level indicator tracking spillovers, with the consequence that the entire FY23 portfolio of RW-supported projects now 14 MAR REPORT FY23 has at least one PDO-level indicator. This system was established during FY22, and Chart 2.4 shows the progress made since FY21. At the Regional level: the 2021 Update to the Africa Regional Integration Strategy provides a framework for measuring spillover benefits of the RI program, as does the South Asia Regional Integration, Cooperation, and Engagement (RICE) Approach . While IDA fully implemented this recommendation since the start of FY23, and for some projects in earlier fiscal years, it is important to remain realistic about the degree to which spillover outcomes can be attributed to WBG interventions. The self-assessment gives it a CD, which satisfies the typical retirement criteria. However, given the importance of cross-border issues in the WBG Evolution, IDA can offer several lessons for the rest of the WBG on measuring the spillover benefits. Thus, as an exceptional case, Management proposes to report on the latter part of the recommendation, generating evidence based on robust indicators, for another year, which would allow the WBG to learn further lessons on spillover benefits of regional programs. Grow with the Flow: An Independent Evaluation of World Bank Group Support to Facilitating Trade FY19. One of the four recommendations was retired after reaching CD. Final year of reporting. World Bank and IFC. 2.13 The WBG’s trade facilitation work progressed well considering the context of the adverse global trade situation driven by the effects of the COVID-19 pandemic and wider geopolitical developments. The WBG program mainstreamed the agenda of complementary interventions. A good example is the Competitive Pacific for Economic Transformation Programmatic Advisory Services and Analytics (PASA) which includes multiple trade facilitation interventions, complementing trade policy and finance activities to improve competitiveness. Since the FY22 MAR update, in FY23 13 additional programmatic interventions commenced, bringing the total to 22 (adding PASA, Development Policy Financing (DPF), and IFC Advisory interventions) since the IEG evaluation. Projects build on previous IFC and WB engagements to increase alignment between project objectives and international standards including the World Trade Organization (WTO) Trade Facilitation Agreement (TFA). Diagnostic tools such as the WTO TFA Gap Assessment and the Time Release Study helped to identify and mitigate political economy (and other) constraints that can hinder trade facilitation reform. Since 2015, the WBG undertook 41 full TFA Gap Assessments. This diagnostic is now institutionalized as a baseline assessment to identify constraints for effective implementation of reforms. The WBG continues to expand efforts to systematically identify and monitor relevant public policy dimensions of trade regulations. The Public Policy Monitoring Screening Framework involves standard screening questions to identify projects where monitoring public policy objectives of trade regulations relating to public health, safety, the environment, good governance, formality, and the rule of law would be relevant—five new engagements were identified in FY22. State Your Business! An Evaluation of World Bank Group Support to the Reform of State-Owned Enterprises, FY08-18 FY21. Two recommendations. Second year of reporting. World Bank, IFC, and MIGA.  2.14 WBG support for State-Owned Enterprise (SOE) reforms demonstrated substantial progress. Analysis of SOE reforms are increasingly being incorporated into WB strategy documents and mainstreamed through core diagnostics and the Integrated SOE Framework (iSOEF). SOE engagement in IDA countries often focuses on the foundational aspects of SOE governance, reflecting the need for selectivity, institutional capacities, and the specific governance context. To this end the infrastructure sectors continued using the Infrastructure Assessment Program (INFRA-SAP) to focus SOE reforms on foundational aspects. Sector and enterprise-level competition, which is a key aspect of IEG’s recommendation, is also addressed through core diagnostics such as Country Economic Memorandum 15 MAR REPORT FY23 (CEM), CPSDs, the iSOEF Markets module, and in the case of the financial sector, the Financial Sector Assessment Program (FSAP). SOE-related analyses in core diagnostics and ASAs increasingly inform Investment Project Financing (IPF), Program for Results (PforR), and DPF operations, calibrating the policy measures to factor in country capacity and control of corruption as well as sector competition. IFC focuses on facilitating enterprise-level reforms by offering a combination of investments and Upstream/Advisory engagements to support SOEs in increasing their commercial orientation and strengthening their corporate governance standards. The WB and IFC coordinated multi-component interventions in Senegal’s housing market demonstrating that the WBG Cascade approach to SOE reform helps enhance internal coordination and mobilize private financing and capacity. Since 2018, the WB increasingly supported the SOE reform agenda through the implementation of DPOs, many of which are highly relevant to the MFD agenda. MIGA also intensified its dialogue and cooperation with the WB, IFC, and other DFIs to address the challenges faced by SOEs. Since IEG’s Report, MIGA issued over $4.0 billion of Non-honoring State-Owned Enterprise (NHSOE) guarantees, including over $2.0 billion in FY23. MIGA’s NHSOE guarantees helped SOEs access long-term commercial financing. MIGA further enhanced the development impact of several of these guarantees by supporting inclusion and climate-related activities. World Bank Support for Public Financial and Debt Management in IDA-Eligible Countries FY21. Two recommendations. Second year of reporting. World Bank only. 2.15 There is evidence of progress on monitoring the quality of countries’ Public Finance and Debt Management (PFDM) systems and a more mixed picture on that assessment shaping WB support. The following progress can be reported on monitoring PFDM and with some early examples of how the assessment is used to prioritize and sequence WB support: a) the launch of the Global Stocktake of Public Financial Management Tools, which plays a key role in improving monitoring and availability of country specific PFDM data, b) continued efforts to undertake 20-30 country level Public Expenditure and Financial Accountability (PEFA) assessments per year; the Equitable, Growth, Finance, and Institutions (EFI) Practice Group issued the second Global Report on Public Financial Management (PFM) in FY23, including a crisis budgeting section, which established a relationship between crisis PFM system, Public Investment Management (PIM) systems and debt management; c) completion of 18 Debt Management Performance Assessments (DeMPAs) in IDA countries since 2020 and support provided to eight countries in creating debt reform plans, which is a good example of the usage of the PFDM monitoring data. EFI established several databases that capture country specific debt management outcomes for IDA countries increasing the alignment of PFDM tools; d) 39 IDA-eligible countries finalized a Public Expenditure Review (PER) since 2022 and PER policy recommendations are used to inform reform priorities and the formulation of Power Purchasing Agreements (PPAs) and as analytical underpinnings of Prior Actions (PAs) in DPFs; e) the 2022 Global Stocktake of PFM diagnostic tools provides an enhanced framework for more effective interactions between various diagnostic tools, ongoing PFM–PIM–Debt Management reforms and WB support to these reforms; f) strong and regular coordination on debt management work (including TA and training) on IDA- eligible countries with the IMF and through the Debt Management Facility; and g) use of DeMPAs, LIC-DSAs, PERs, PIMAs, and PEFAs to inform preparation of 89 PPAs on Debt Management and Debt Transparency for FY23 Sustainable Debt Finance Policy (SDFP) across 42 IDA countries. 16 MAR REPORT FY23 Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness FY21. Three recommendations. Second year of reporting. World Bank and IFC.   2.16 The WB and IFC continue to mobilize Disruptive and Transformative Technologies (DTT). On DTT opportunities and risks, a range of activities have taken place. The Digital Development Partnership (DDP) has in 2022-2023 undergone a new strategic restructuring of the Umbrella Trust Fund, re-branded as DDP 2.0. The Digital Economy for Africa (DE4A) program is working to ensure that all people, businesses, and governments in Africa are digital enabled by 2030, with similar digital economy assessments expanding to South Asia (SAR) and Latin America and the Caribbean (LAC). The Disruptive Technologies for Development initiative (DT4D) awarded pilots and grants. The Mainstreaming Digital and Disruptive Technologies PASA (a DDP 2.0 window) supports nearly a dozen initiatives across the WB DTT initiatives. The cybersecurity multi-donor trust fund (MDTF) launched 20 country-level TA activities, convened engagements, and promoted an increasingly structured approach to cybersecurity at the WBG through working groups with LEG, ITS, Digital Development Practice Group, Operations Policy and Country Services (OPCS), and the IFC. At the GP level, and among technology-focused communities of practice (CoPs), key efforts are led by BlueTech (ENB GP), GovTech Global Partnership (GTGP), GovTech Global Solutions Group (GSG), HNP GP, Digital Development in INF, and GEMS. Finally, an initiative to address the risks of AI-powered technologies embedded within WB lending projects is the “Tools for Identifying the Human Rights Impact and Algorithmic Accountability of Artificial Intelligence in World Bank Operations,” under the leadership of LEG and supported by the Human Rights, Inclusion, and Empowerment Trust Fund. On skilling up the WBG workforce on DTT, nearly all the initiatives, GPs, and programs referenced above have engaged in skills development activities to varying degrees. DT4D hosted knowledge events and prepared publications. The Digital Development GP’s Digital Safeguards group developed a consultant database of experts to support TTLs to find cyber skills for their projects. ITS hosts a cyber intranet page. ITS Technology and Innovation Lab (ITSTI) conducted various activities to develop WBG staff skills in other DTT topics including Web 3.0 technologies, central bank digital currencies, tokenization, non-fungible tokens (NFTs), and other emerging technology themes. GEMS also contributed to technology and data analysis skills among WBG staff. At the GP level, knowledge sharing and skilling up activities are led by, among others, the Digital Development GP, Health GP, GovTech team, and the GovTech CoP in the GOV GP. There are related efforts through the BlueTech initiative with most of these knowledge activities being delivered through the WBG’s Online Learning Campus platform. Procurement was flagged for priority attention in the Cybersecurity MDTF Strategy/Annual Report, to ensure best practices regarding digital infrastructure and cybersecurity. IFC utilized DTT opportunities to intensify its support to digitalization in financial services, venture capital and ecosystem, infrastructure, health, and education to promote innovative solutions to global challenges. IFC established strategic initiatives to develop lending frameworks (balance sheet lending and asset-based lending) to fund later-stage innovative digital lenders in emerging markets, such as Nubank in Colombia, with the aim of providing access to credit for underserved populations, especially among women and minority groups. On venture capital, the IFC Startup Catalyst Program (ISC) supports incubators, accelerators, and seed funds in the most nascent venture ecosystem to drive tech innovation in climate, gender and inclusion. It doubled its size in FY23 with a new pool of $60 million, of which IFC expects to commit 100 percent of ISC Expansion II to Seed Stage Funding Mechanisms (SSFMs) in IDA/FCS markets, challenging geographies, and/or SSFMs investing in areas that are strategic priorities for IFC such as Climate and gender. With regard to Telecom, Media, and Technology, IFC has delivered for the fourth year in a row its highest-ever levels of funding to support digital infrastructure (a cumulative amount of almost $6 billion in FY20-FY23) targeting mobile operators, particularly in IDA/FCS countries (e.g., Ethiopia), on-ramping data center capacity and shared infrastructure (fiber and towers) to lower infrastructure deployment costs and support climate transition. Furthermore, IFC implemented Upstream projects across digital infrastructure sectors (e.g., mobile networks, towers, broadband, cloud, and data) and geographical regions. An IFC 17 MAR REPORT FY23 InfraTech framework and implementation approach was approved in FY23 to expand IFC investment to include next-generation network technologies to deliver enhanced connectivity and enable digital economy applications such as sensor networks, AI-based models to reduce energy consumption, and 5G, in partnership with the Disruptive Technology and Funds (CDF) department. Advisory services were provided to IFC clients in health and education sectors to develop and implement digital transformation strategies. In the meantime, IFC takes a risk-based approach to assessing cybersecurity risks and tiers of projects. IFC further reorganized its operations function to align its workforce with DTT opportunities and provided regular and in-depth training to enhance staff capacity. IDA’s Sustainable Development Finance Policy: An Early-Stage Evaluation FY21. Three recommendations. First year of reporting. World Bank only.    2.17 The Sustainable Development Finance Policy (SDFP) was approved in June 2022 to support IDA- eligible countries in their efforts to achieve and maintain debt sustainability and there is progress in the first year of implementing IEG recommendations. All IDA countries are screened annually for their debt vulnerabilities and are covered by Debt Sustainability Enhancement Programs (DSEPs), including countries with low risk of debt distress. Different tools and protocols can be applied to deal with changes in the debt distress situation and Sustainable Finance Debt Policy (SDFP) guidelines do not preclude low-risk countries from the preparation of PPAs. The number of countries preparing PPAs expanded (from 55 in FY21, to 58 in FY22, and to 60 in FY23). PPAs were designed to consider countries’ debt vulnerabilities through an intensive dialogue with countries’ authorities, with no standard PPAs applied across all countries. The SDFP helps address debt transparency, fiscal sustainability, and debt management challenges systematically and proactively over a medium- to long-term horizon and the policy pays attention to Small States and FCV countries. Country teams helped authorities articulate focused, but impactful, PPAs, while considering the country’s capacity to implement these PPAs. Up-to-date Core diagnostics inform all PPAs (e.g., DSA, Public Expenditure Review, DeMPA, Debt Reporting Heatmap), which focus on immediate priorities with a medium- to long-term perspective. Given that SDFP was only established a couple years ago, the adoption of a programmatic approach is a work in progress. Several countries have already engaged in a programmatic approach and the results will be assessed in the coming years. A programmatic approach introduces intermediate steps that contribute to the fulfillment of results over the medium term. There was also progress in the implementation of PPAs institutionalizing debt transparency in the first two years of PPAs implementation. In March 2023, the WB undertook the SDFP IDA20 Mid-Term Implementation Review, which analyzed implementation of the policy in detail. The review includes detailed evidence on which types of analysis were used to develop PPAs, which types of PPAs were put in place, and which countries engaged in programmatic approaches. The Development Effectiveness of the Use of Doing Business Indicator: Fiscal Years 2010 –20 FY22. There are four recommendations. First year of reporting. World Bank and IFC. After the CODE discussion on the evaluation, Management confirmed plans for a new approach to assessing progress on enabling environment (now called “B-READY”) and is reporting on how the new approach starts to reflect evaluation lessons. 2.18 There has been a good start to the new approach to assessing progress on business enabling environment (now called “B-READY”) after discontinuation of Doing Business. The B-READY Concept Note (CN) was published in December 2022, and the project will operate as a pilot during the initial three-year period with phased inclusion of countries. The first report will be published in Spring 2024, and two additional reports during the pilot phase in 2025 and 2026. This sequencing during the pilot phase is necessary to produce a baseline for the complete B-READY data set, while allowing the methodology to 18 MAR REPORT FY23 fully develop and for Management to consider best ways of using it to inform client engagement. The B- READY CN states that it will be one of many benchmarking exercises and regularly update indicators to reflect learning (i.e., Lessons 1 and 3 from the evaluation have more to report on the technical and methodology aspects and the CN confirms there is LE in these areas, while Lessons 2 and 4, are largely about WBG practices in how it uses B-READY to engage with clients and the overall messages that are communicated broadly). Given that the first B-READY report will only be published in April 2024, it is not possible for the FY23 MAR to report on the latter aspect, and thus Management has assessed these two recommendations to be “Progress Constrained” (i.e., “valid evidence of progress unavailable”). Engaging Citizens for Better Development Results FY19. Five recommendations. Fourth year of reporting. World Bank and IFC. 2.19 There was progress across all recommendations in the work on Citizen Engagement (CE) for the WB and IFC. There was steady improvement in implementing “thick CE” as currently defined.2 Compared to the 27 percent baseline in FY22, 73 percent of FY19 portfolio projects (145 out of 200 IPFs) reported on or made credible progress toward reporting on Beneficiary Feedback indicator(s) by the third year of implementation (Chart 2.5). Eighty-one percent of CE-related indicators met their targets by project completion. The WB seeks to enhance its focus on the depth and quality of Citizen Engagement and Social Accountability (CESA) in both incentives and implementation. In FY22, the CESA GSG developed and piloted an Enhanced Analytical Framework for CE Project Monitoring (CE enhancement framework)3 2Thick CE is defined as “citizen engagement that is regular and continuous, uses multiple tools, and is embedded in country systems” on page 57 of the IEG Report Engaging Citizens for Better Development Results. 3 The CE enhancement framework was first piloted for FY22 Q4 projects and is currently being used in FY23 to routinely collect and use evidence on CE quality and implementation and to inform future implementation, strategy, and decisions. Detailed analysis of this pilot will be published with the FY22 Citizen Engagement Annual Report. For summary, please see Figure 1. Project Classification based on the CE Intensity Scale (FY22Q4). 19 MAR REPORT FY23 that goes well beyond previous “Yes/No” corporate compliance tracking to assess the quality of CE across multiple parameters during project design. The pilot found that a little more than half (53 percent) of 159 projects approved during the last quarter of FY22 had engagements identified at the level of consultations, and 6 percent were at the level of empowering citizens. The analysis suggests modest improvements in results-oriented monitoring of CESA, with potential to better target and monitor systems-level results. To build on ongoing progress, the WB aims to engage in a more cohesive and client-focused approach to Environmental and Social Frameworks (ESF) and CE frameworks moving forward and enhance synergies between the two agendas. To this end, the CESA GSG is leading stock-taking research focused on better understanding Stakeholder Engagement Plans (SEPs) as a channel to strengthen CE and the nature of operational synergies between the two. The WB is also exploring ways to strengthen the monitoring of SEPs and ensure complementarity and coherence between SEP and CE monitoring. Most importantly, to provide new evidence on the WB’s CESA agenda, systems, and insights for designing the future CESA architecture, the WB commissioned a CESA Strategic Review. This will propose a series of updates for FY24 onwards to achieve more strategic and systems-oriented CESA, and an institutional support system that is fit-for- purpose and incentivizes high-quality and results-oriented approaches that will be captured in the corporate scorecard. These recommendations will inform a Strategic Options paper that will be submitted to IBRD and IDA management by December 2023. Three training events for E&S specialists were held by IFC in FY23, in addition to other existing stakeholder engagement (SE)/grievance mechanism (GM) training programs aimed at a) strengthening E&S specialists’ capacity in reviewing and assessing those components of clients’ E&S management systems and b) building clients’ capacity. In FY23, a total of 186 IFC E&S staff and consultants attended internal training on SE, a significant additional effort when compared to the previous three years (2020-2022) during which six training events were attended by 197 specialists cumulatively. Evaluation forms, completed for six of the training events by attendees, provided good reviews in terms of quality and effectiveness. Regarding efforts aimed at client capacity building, eLearning modules are being developed and will be available for clients to improve the quality and outcomes of effective SE/GM, consistent with IFC Performance Standard 1. Based on feedback received from clients and stakeholders during IFC monitoring of portfolio projects and community visits, training on SE resulted in improved clients’ SE processes. Moreover, IFC is improving its review and rating systems to provide for enhanced systematic metrics and documentation of the effectiveness of SE/GM. Newly developed tools, such as the Sustainability Rating Tool (SRT) and MALENA (Machine Learning ESG Analyst), are being piloted across sectors and regions, will support systematic documentation and review of SE/GM implementation at project appraisal and supervision and will allow IFC to monitor trends at the portfolio level over time. The World's Bank: An Evaluation of the World Bank Group's Global Convening FY20. Third year of reporting. Three recommendations. World Bank and IFC. 2.20 Progress was made on the convening power agenda, which is a critical part of the WBG Evolution discussion. The WB employed deliberative approaches in global convening initiatives, through greater selectivity in the implementation of TF reforms. For the last three years, there has been greater transparency and oversight around fundraising plans and purposes in the annual WB-wide Strategic External Financing Framework (SEFF). Progress in achieving TF consolidation (Chart 2.6) is integrated into regular Matrix Vice Presidents (MVP) reporting and management Memorandum of understanding (MoUs). There is greater Management oversight on the life cycle of TFs, with regular stocktaking of institutional priorities to address pressing global development issues. At the global, regional, and country levels, the Bank’s convening efforts and engagement in partnerships consider its comparative advantage, value addition and role in other engagements. Guided by Senior Management, WBG initiatives have focused on priority areas including climate; debt; pandemic prevention, preparedness, and response; food security; 20 MAR REPORT FY23 and FCV. TF Directives, Procedures and Guidance were formally issued, including external versions made available to donors on the external TF Reform website—aiding internal systems and processes. To strengthen links between global and country work, as part of the Trust Fund Integration Initiative, new processes and systems were put in place to ensure early CMU endorsement of proposals to global trust funds. The WB is aligning global priorities with country support through its corporate and IDA commitments and using its organizational levers. For example, it used a multiphase programmatic approach to help countries access financing as part of the global COVID-19 health response. Another example is the WB’s commitment to achieve 100 percent of new financing operations aligned to the goals of the Paris Agreement from July 1, 2023. IFC places deliberate focus on the scope of its engagements to maximize operational and development impact. IFC is strategically partnering with organizations to convene global audiences around major thematic priorities, such as climate and affordable housing, to support the development of actionable private sector solutions. On enhancing internal systems and processes to support convening power, IFC announced two organizational adjustments in FY23 to streamline its organizational structure (effective July 1, 2023). The first creates a Strategy & Operations Management Department, housed alongside other corporate support departments, thereby strengthening the link between strategy, operational support and corporate resources. The second is the newly combined Partnerships & Blended Finance Department which will strengthen the focus of donor fundraising on Blended Finance. This new combined department will be housed alongside research and thought leadership functions to facilitate the development of new partnerships with academia and think tanks. IFC also improved coordination among Vice-Presidential Units (VPUs) to align convening efforts that are cross-cutting or comprise a major thematic priority. The World Bank Group Outcome Orientation at the County Level: An Independent Evaluation FY21. Second year of reporting. One active recommendation. World Bank, IFC, and MIGA.  2.21 The WBG made progress in implementing the outcome orientation work in CPFs following the guidance issued in 2021. This enables a more strategic narrative of the WB’s contribution to high level outcomes and aids learning and adaptation. The new guidance also facilitates IFC and MIGA’s inputs to ensure the private sector perspective is fully embedded in WBG documents. As of March 2023, all 17 CPFs include High Level Outcomes (HLOs) which is an improvement over FY22, when only 40 percent of CPFs included HLOs. There is close coordination across the WB matrix in contributing to this agenda (e.g., regional teams are providing training and backstopping to CPF teams to strengthen the outcome orientation). The revised country engagement guidance includes several features, in addition to the HLOs, that are improving the country level results system, as seen in early evidence of some of the new CPFs. First, it is increasing the emphasis on capturing the contribution of knowledge, policy-related interventions, and advisory services and analytics. Second, it is strengthening the use of lessons learned from the completion and learning review (CLR) to inform new CPF design. Third, it is making the CPF objectives under HLOs more specific, realistic, and attributable to the WBG interventions, compared to the objectives in older CPFs. IFC outcome orientation at a country level 21 MAR REPORT FY23 continues through tracking IFC’s project results, including market outcomes, through its Anticipated Impact Measurement and Monitoring (AIMM) results measurement system. MIGA realizes deeper links between country objectives and MIGA's strategic focus through a stronger engagement with WB and IFC colleagues in each CPF. Enhancing the Effectiveness of the World Bank’s Global Footprint FY22. First year of reporting. Three recommendations. World Bank only. 2.22 The WB continues to make progress in enhancing staff field presence, tailoring support to country and program specific needs. Through its field presence, the WB aims to support country’s development needs and deliver on intended development outcomes effectively and efficiently. That is through adjusting to the evolving country context and emerging issues and considering the available budget. The WB uses tools such as the annual workforce plan carried out on a three-year rolling basis and the internal corporate recruitment/mobility exercises for decisions. Adjustments to the WBG field presence are being guided by a set of general principles. The implementation of the annual workforce plan and outcomes of the internal recruitment and mobility exercises are closely monitored by Management, and People and Culture (PAC) VPs. As of end-FY22, 48 percent of staff were based in the field, and there is an increased presence in FCV countries (see Chart 2.7). Overall, 47 percent of staff were based in non-US locations, including 57 percent of staff in Operations and 35 percent of staff in Institutional, Governance, and Administrative (IG&A) units. Corporate efforts to enhance knowledge flow across the WB are ongoing at the global, regional, and country levels, guided by the Strategic Framework for Knowledge (SFK). To address concerns of regional silos and loss of knowledge due to staff movement, knowledge flow is supported through investments in systems and technology to facilitate access to information and knowledge wherever staff are located; through ongoing support (e.g., knowledge exchange, briefings, outreach initiatives); and through staff development including the enhanced Career Development and Mobility Framework (CDMF). Professional and career growth for Locally Appointed staff continues to be enhanced through contextualized resources and interventions. Various programs in PAC are in place to complement the actions Locally Appointed staff and their managers are taking with respect to professional development: the FCV Leadership and Effectiveness Programs; expanded WBG Language Program partnering with Regions (LCR, AFW and soon AFE); mentoring as a complement to other forms of staff development; and Career Seminar opportunities. At the regional level, management teams also continue to implement career-focused initiatives, which are complemented by GP initiatives for their respective sectors. 22 MAR REPORT FY23 3. CONCLUSIONS AND LOOKING AHEAD Conclusions 3.1 Several evaluations contain relevant insights on priorities for the WBG Evolution like climate, pandemics, cross-border challenges, and private sector financing. These involve actions to address the enabling environment, enhanced operational modalities, informed risk taking, improved guidance for staff, and stronger results measurement systems. Additionally, the MAR process contributes to the proposed adjustments to the WBG operating model by enhancing the direct feedback conduit for evaluations to inform WBG’s policy, operational and investment work. The MAR process would continue to contribute to the implementation of the Evolution priorities. 3.2 The FY23 MAR shows that implementation of evaluation recommendations was embedded in the WBG crisis response. Management previously highlighted three elements of how the WBG institutions responded effectively to the recent multiple crises through a) scaled-up financing, b) improvements in portfolio quality, and c) continued delivery of the scale and quality of development results. Additionally, the FY23 MAR self-assessment confirms a strong performance in the implementation of the evaluation recommendations—adding an important lesson learning element to the overall WBG crisis-response efforts and signaling the effective balancing by the WBG between immediate and medium-term development priorities. 3.3 Management responded to feedback from last year’s MAR by raising the bar in assessing progress in the implementation of recommendations. This is reflected in a larger share of the self-assessment acknowledging “limited progress” and smaller share achieving a “change in direction.” Compared to FY22, the self-assessment across 59 recommendations shows that 28 percent have either “limited evidence” (LE) or demonstrate “change of direction” (CD) and a larger share, 68 percent, in the middle category of “emerging evidence” (EE). Management notes a higher share of LE and lower share of CD assessments in FY23, reflecting a higher bar to demonstrate institutionalization of progress. 3.4 Management has taken a careful approach to retiring recommendations. In an era of multiple crises and change, the relevance of recommendations for MAR reporting should be reviewed on an ongoing basis. There is also a need for managing the overall volume of recommendations reported in the MAR due to the addition of new evaluations to the MAR reporting cycle every year. In FY23, 12 recommendations are due for “automatic” retirement. Some recommendations are proposed to be extended beyond the stage of “automatic” retirement to achieve better learning and outcomes through continued implementation. Management also proposes early retirement of recommendations when the context and priorities have evolved. 3.5 The FY23 MAR identifies “pathways” used for implementing the recommendations, recognizing that they differ in the degree of difficulty and resource intensity. The 59 recommendations being reported on this year vary in terms of their nature and intent—some entail more gradual uptake while others are more amenable to faster implementation. The FY23 MAR made a start in identifying “pathways” through which IEG recommendations are being implemented—opening the opportunity for implementation plans 23 MAR REPORT FY23 to be more explicit and understood by staff. Codifying and explaining these pathways could help strengthen all stages of the evaluation life cycle. 3.6 Implementing evaluation recommendations complements other ongoing WBG efforts to improve development effectiveness. Experience with MAR implementation confirms that the intent of the MAR reforms is being met by contributing to the development effectiveness of the WBG’s work. All IEG evaluations identify areas where progress is being made by the WBG, and flag areas for improvement. When the recommendations are being well implemented, this strengthens the overall development effectiveness of WBG efforts—the report shows several examples of this. Looking Ahead 3.7 Management is implementing measures to further optimize learning from the MAR process. Effective implementation of evaluation recommendations requires that operational teams and Managers understand what is required to demonstrate implementation progress. Starting in FY24, Management will appoint “champions” for each evaluation to identify key milestones to ensure success in implementing recommendations. Identifying “champions” has an extra value-added for those recommendations entailing cross-WB and cross-WBG efforts, requiring multiple players acting in a coordinated manner. To maintain the learning focus of the MAR, it will also be crucial to keep an eye on maintaining a manageable number of active recommendations to monitor. The FY22 MAR had reported on 18 evaluations and 55 recommendations, while the FY24 MAR is expected to report on 26 evaluations and 79 recommendations. Management will also make the MAR process nimbler, and the FY23 MAR “after action review” will consider this with a view to implementing improvements in the FY24 cycle. 24 MAR REPORT FY23 ANNEX I. FY23 MAR MANAGEMENT ASSESSMENT FOR EACH RECOMMENDATION Evaluation / Recommendations Management Assessment Investing in Human Capital FY19 World Bank Group Support to Health Services: Achievements and Challenges FY22 FY23 1 Improve measurement of the quality of health services and the distributional effects of health services projects. EE CD 2 To develop sustainable capacity to address pandemics, systematically integrate, in World Bank Group–financed CD EE projects and ASA, awareness and preparedness plans and governance frameworks for pandemic control with the client country’s own health system. FY22 World Bank Support to Reducing Child Undernutrition. FY22 FY23 1 Adjust nutrition programming in country portfolios to (i) give more priority to institutional strengthening for N.A. LE coordination and implementation of multisectoral nutrition interventions; and (ii) increase focus on subnational targeting of interventions to reflect areas of greatest disadvantage and persistency of need. 2 Strengthen nutrition support in Global Practices to (i) rebalance investments to have greater emphasis on N.A. LE nutrition-specific interventions and (ii) increase focus on social norms interventions and behavior changes, with more attention to tracking expected achievements to improve nutrition determinants. Climate and Resilience FY19 Carbon Markets for Greenhouse Gas Emission Reduction in a Warming World FY22 FY23 1 The WBG should increase its use of CF instruments to attract and mobilize finance that supports transformational EE EE activities and leverages private investments FY20 Building Urban Resilience: An Evaluation of the World Bank Group's Evolving Experience (2007-2017) FY22 FY23 1 The Bank Group should systematically identify and track progress of interventions that build urban resilience to EE CD chronic stresses and acute shocks, across its institutions (for WB and IFC). 2 The design and implementation of World Bank projects that build urban resilience should systematically CD EE incorporate resilience characteristics and articulate their application throughout the project cycle. These should include the following: (i) design standards in line with resilience risks, (ii) cost-benefit analysis in line with resilience risks, (iii) city and interjurisdictional coordination, and (iv) inclusive approaches for vulnerable people. 3 In urban areas where the client has identified crime and violence as a resilience risk, the World Bank’s support EE PC should be based on a localized typology of crime and violence that is informed by relevant analytic work. This approach should be supported by an assessment of the mechanisms most effective at reducing crime and violence within operations. 4 IFC should support its public and private sector Cities Initiative clients through available resilience risk assessment EE EE and mitigation tools to strengthen development impacts (only IFC). FY21 Evaluation of the World Bank Group's support for electricity supply from renewable energy resources, FY22 FY23 2000-2017 1 WBG to prioritize interventions that focus on the integration of RE sources into the power systems of client EE CD countries, to facilitate progress in their clean energy transitions (for WB/IFC/MIGA). 2 WBG to support RE scale-up through comprehensive, long-term country engagements, with coordinated WBG EE EE solutions, based on the comparative advantages of each institution, to address barriers, aided by robust upstream diagnostics (for WB/IFC/MIGA). 3 WBG to continually upgrade the pool of specialized skills to help clients address their pressing and rapidly EE CD evolving challenges to scale-up RE (for WB/IFC/MIGA). I MAR REPORT FY23 FY21 The Natural Resource Degradation and Vulnerability Nexus: An Evaluation of the World Bank's FY22 FY23 Support for Sustainable and Inclusive Natural Resource Management (2009-2019) 1 The World Bank should identify and analyze natural resource degradation and vulnerability nexus issues and EE EE leverage this knowledge in SCDs and in country engagements where such issues matter for achieving sustainable poverty reduction and shared prosperity. 2 World Bank operations that address natural resource degradation should direct attention to resource governance EE EE challenges and use a mix of resource management practices and financial incentives appropriate for the relevant socioecological systems. 3 World Bank Global Practices involved in addressing natural resource degradation and associated vulnerability EE LE should share knowledge, improve measurement, and enhance coordination in the design and implementation of their projects to optimize development effectiveness. FY22 World Bank Engagement in Situations of Conflict: An Evaluation of FY10-20 Experience FY22 FY23 1 To enhance the conflict sensitivity of World Bank engagement, ensure that politically sensitive, confidential N.A. EE analysis is generated, retained, and managed so that it can be used by select future staff working on that country 2 Ensure that country engagements are informed by timely analyses of conflict dynamics and risks. N.A. EE 3 Address factors that dissuade World Bank engagement in conflict affected areas. N.A. EE 4 In conflict affected countries, rethink what success looks like. N.A. LE FY22 Managing Urban Spatial Growth: World Bank Support to Land Administration, Planning, and FY22 FY23 Development 1 Adopt a framework that links the determinants of urban expansion to pathways for managing urban spatial N.A. EE growth and that contributes to the achievement of Sustainable Development Goals (SDGs) 1 and 11. 2 Support World Bank clients with anticipating and preparing for urban spatial growth using preventive N.A. LE approaches, not just curative ones. 3 Strengthen and ensure implementation of the World Bank’s protocol to identify and record precise project N.A. LE locations and collect land market data necessary to support clients with managing urban spatial growth. FY22 Transitioning to a Circular Economy: An Evaluation of the World Bank Group's Support for Municipal FY22 FY23 Solid Waste Management (2010-20) 1 To achieve more sustainable and scalable outcomes in municipal waste management, Bank Group technical and N.A. EE financial support to clients should give clear priority to the adoption and implementation of waste hierarchy practices, in line with client needs and capabilities for MSWM (for WB/IFC/MIGA). 2 To support the LICs where municipal solid waste is growing most rapidly, the Bank Group should identify N.A. EE constraints on demand and investments and leverage external partnerships to implement context-specific MSWM solutions (for WB/IFC/MIGA). 3 To bring prominence to and spur action on the global municipal solid waste agenda, the Bank Group should take N.A. EE up a clear leadership position, collaborating and convening with developmental partners (for WB/IFC/MIGA). Private Finance FY21 The World Bank Group's Approach to the Mobilization of Private Capital for Development: An IEG FY22 FY23 Evaluation 1 Expand PCM platforms, guarantees, and disaster risk management products commensurate with project pipeline LE-WB; EE development (for WB/IFC/MIGA). EE-IFC & MIGA 2 Develop new products and improve product alignment with the needs of new investor groups and partners (for EE EE IFC and MIGA). FY22 The International Finance Corporation’s and Multilateral Investment Guarantee Agency’s Support for FY22 FY23 Private Investment in Fragile and Conflict Affected Situations, Fiscal Years 2010 –21 1 IFC and MIGA should continue to review their financial risk, make more explicit the implications of IFC’s portfolio N.A. EE approach for FCS, and enhance capabilities to address nonfinancial risks to ensure they align with achieving business growth targets and impacts in FCS (for IFC and MIGA). II MAR REPORT FY23 2 To focus on the development of bankable projects, IFC and MIGA should further recalibrate their business N.A. EE models, client engagements, and instruments to continuously adapt them to the needs and circumstances of FCS and put in place mechanisms to track their effectiveness for real-time learning (for IFC and MIGA). Long Term Economic Growth FY19 Two to Tango: An Evaluation of World Bank Group Support to Fostering Regional Integration FY22 FY23 1 Strengthen the design of IDA Regional Window supported projects to improve the assessment of spillover effects EE CD and to generate evidence based on robust indicators. FY19 Grow with the Flow: An Independent Evaluation of World Bank Group Support to Facilitating Trade FY22 FY23 2006-17 1 In order to enhance effectiveness, the World Bank Group should promote an approach of complementary CD CD (simultaneous and/or sequential) interventions in trade facilitation reforms in countries where trade is a client priority and World Bank Group has a comparative advantage, substantiated by consistent diagnostics (for WB and IFC). 2 The World Bank Group should identify and mitigate political economy constraints to trade facilitation reform EE EE implementation through systematic application of its tools for stakeholder analysis and consultation (including public private dialogue) (for WB and IFC). 3 The World Bank Group should systematically apply a differentiated approach to identify and monitor, where EE EE relevant, the public policy objectives of trade regulations relating to public health, safety, the environment, good governance, formality and the rule of law (for WB and IFC). FY21 State Your Business! An Evaluation of World Bank Group Support to the Reform of State-Owned FY22 FY23 Enterprises, FY08-18 1 The World Bank Group should apply a selectivity framework for SOE reform support that considers country EE EE governance conditions, control of corruption, and sector and enterprise-level competition (for WB/IFC/MIGA). 2 The World Bank Group should apply the MFD and its embedded Cascade approach for SOE reform (for EE EE WB/IFC/MIGA). FY21 World Bank Support for Public Financial and Debt Management in IDA-Eligible Countries FY22 FY23 1 World Bank should regularly monitor the quality of the key pillars of PFDM for each IDA-eligible country, possibly EE EE through a centralized country-specific PFDM assessment 2 Actively use the previously described assessment to prioritize and sequence World Bank support for PFDM EE LE capacity building and reform in IDA-eligible countries FY21 Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness FY22 FY23 1 Where DTT offer opportunities to make progress on the twin goals more effectively or efficiently, ensure that the EE EE Bank Group avails itself of those opportunities and addresses, in particular, the risks posed by DTT (for WB and IFC). 2 Build a Bank Group workforce with the skills required to harness DTT opportunities and mitigate DTT risks by EE EE identifying DTT-relevant skills, determining gaps in these skills, and filling these gaps (for WB and IFC). 3 Improve the effectiveness and efficiency of World Bank procurement for complex technology projects. EE EE FY22 The International Development Association’s Sustainable Development Finance Policy : An Early-Stage FY22 FY23 Evaluation 1 Consideration should be given to expanding the countries covered by the DSEP beyond those at moderate or high N.A. EE levels of debt distress or in debt distress. IEG recommends applying an additional filter. 2 PPAs should emanate from an up-to-date assessment of country-specific debt stress and be set explicitly within a N.A. LE longer-term reform agenda. 3 Where PPAs support actions that need be taken regularly (for example, debt reporting to parliament), PPAs N.A. LE should aim for long-lasting institutional reforms rather than relying on one-time actions. III MAR REPORT FY23 FY22 The Development Effectiveness of the Use of Doing Business Indicator: Fiscal Years 2010–20 FY22 FY23 1 Recognizing the powerful motivational effect of reform indicators, especially those that facilitate country N.A. LE rankings, this evaluation notes the limitations in the coverage and guidance offered by any single indicator set on its own and advocates integrating them with complementary analytic tools and indicators (for WB and IFC). 2 Recognizing the granularity and specificity of individual reforms in any given country context, the findings from N.A. PC this evaluation suggest that it is better to avoid using business regulatory or similar global indicators as explicit reform objectives or monitoring indicators in Bank Group projects and country strategies focused on improving the business environment. This does not preclude the use of primary data to agreed targets that track and measure critical Bank Group institutional commitments (for WB and IFC). 3 Global indicators coverage and specifications are improved if, at regular and predictable intervals, they are N.A. LE updated to reflect learning from research and field experience to (i) improve links to important development outcomes; (ii) strengthen relevance to the experience of the subject of coverage; and (iii) adapt to technological changes in the areas covered by the indicators (for WB and IFC). 4 The DB experience indicates the need for mechanisms and safeguards to assure the accuracy and validity of Bank N.A. PC Group global indicator–based reports and related communications, using robust and transparent standards of evidence (for WB and IFC). Improving WBG Capacity to Meet Its Goals FY19 Engaging Citizens for Better Development Results FY22 FY23 1 As it defines future corporate priorities for citizen engagement, the World Bank should reflect in those priorities CD EE the need to achieve greater depth and quality of the citizen engagement activities it supports. 2 The World Bank should encourage and support efforts of its regional, country and Global Practices teams to EE EE establish, where appropriate, “thick” citizen engagement that is regular and continuous, uses multiple tools, and is embedded in country systems. 3 The World Bank should strengthen the monitoring of its citizen engagement activities by systematically adopting EE EE results framework indicators that are results oriented. 4 The World Bank should seize the opportunity of the implementation of the ESF to leverage citizen engagement EE EE mechanisms—beyond consultations and Grievance Redress Mechanisms—to reach the objectives of managing social risks, strengthening country systems, and promoting social inclusion. 5 IFC should ensure that its clients’ stakeholder engagement activities required by Performance Standard 1 in EE EE projects with affected communities are carried out during appraisal and supervision of the projects and systematically documented (only IFC). FY20 The World's Bank: An Evaluation of the World Bank Group's Global Convening FY22 FY23 1 Scope engagements and contributions to major global convening initiatives more deliberatively (for WB and IFC). EE EE 2 Enhance how the World Bank and IFC’s internal systems and processes support managing major convening EE EE initiatives over their life cycle (for WB and IFC). 3 Improve links between the World Bank’s global and country work. EE EE FY21 The World Bank Group Outcome Orientation at the County Level: An Independent Evaluation FY22 FY23 1 The Bank Group should reform the country level results system to ensure that it accurately captures the Bank EE EE Group contribution to country outcomes and usefully informs decision making on country engagements (for WB/IFC/MIGA). FY22 Enhancing the Effectiveness of the World Bank’s Global Footprint FY22 FY23 1 The World Bank should refine its current approach to managing its staffing global footprint by clearly specifying N.A. EE decentralization’s expected outcomes and adopting principles to guide and adjust decentralization decision- making based on evidence. 2 The World Bank should mitigate the risks to knowledge flow brought about by decentralization and put in place N.A. EE safeguards to avoid developing country and Regional silos. 3 The World Bank should establish clear and structured paths to systematically promote LRS professional and N.A. EE career growth within its overall approach to improving the effectiveness of its global footprint. IV MAR REPORT FY23 ANNEX II. PROPOSALS FOR RECOMMENDATION RETIREMENT Rec.  Full text  Level Comments  Citizen 1.) As it defines future corporate priorities for citizen engagement, the World Bank EE  End of cycle. Retire. Engagement   should reflect in those priorities the need to achieve greater depth and quality of the citizen engagement activities it supports.  2.) The World Bank should encourage and support efforts of its regional, country and EE  End of cycle. Retire. Global Practices teams to establish, where appropriate, “thick” citizen engagement that is regular and continuous, uses multiple tools, and is embedded in country systems.  3.) The World Bank should strengthen the monitoring of its citizen engagement EE Management activities by systematically adopting results framework indicators that are results proposes to extend oriented.  this for another year 4.) The World Bank should seize the opportunity of the implementation of the ESF to EE End of cycle. Retire. leverage citizen engagement mechanisms—beyond consultations and Grievance Redress Mechanisms—to reach the objectives of managing social risks, strengthening country systems, and promoting social inclusion.  5.) IFC should ensure that its clients’ stakeholder engagement activities required by EE  End of cycle. Retire. Performance Standard 1 in projects with affected communities are carried out during appraisal and supervision of the projects and systematically documented.  Health 1.) Improve measurement of the quality of health services and the distributional CD  End of cycle. Retire. Services   effects of health services projects.  Pandemic capacity 2.) To develop sustainable capacity to address pandemics, systematically integrate, in EE  will be monitored World Bank Group–financed projects and ASA, awareness and preparedness plans moving forward and governance frameworks for pandemic control with the client country’s own under the COVID-19 health system.  eval.   Carbon 1.) The WBG should increase its use of CF instruments to attract and mobilize finance EE  End of cycle. Retire.  Finance  that supports transformational activities and leverages private investments  Facilitating 1.) In order to enhance effectiveness, the World Bank Group should promote an CD End of cycle. Retire.  Trade  approach of complementary (simultaneous and/or sequential) interventions in trade facilitation reforms in countries where trade is a client priority and World Bank Group has a comparative advantage, substantiated by consistent diagnostics.  2.) The World Bank Group should identify and mitigate political economy constraints EE  End of cycle. Retire.  to trade facilitation reform implementation through systematic application of its tools for stakeholder analysis and consultation (including public private dialogue).  3.) The World Bank Group should systematically apply a differentiated approach to EE End of cycle and EE identify and monitor, where relevant, the public policy objectives of trade regulations level achieved relating to public health, safety, the environment, good governance, formality and the rule of law.  Urban 1.) The Bank Group should systematically identify and track progress of interventions CD  CD level achieved  Resilience   that build urban resilience to chronic stresses and acute shocks, across its institutions.  3.) In urban areas where the client has identified crime and violence as a resilience PC  Partially agreed and risk, the World Bank’s support should be based on a localized typology of crime and demand-driven topic. violence that is informed by relevant analytic work. This approach should be Limited progress in supported by an assessment of the mechanisms most effective at reducing crime and year 3.  FY23 violence within operations.  retirement proposed. Renewable 1.) WBG to prioritize interventions that focus on integrating renewable energy CD  CD level achieved  Energy sources into the power systems of client countries, to facilitate progress in their clean energy transitions   3.) WBG to continually upgrade the pool of specialized skills to help clients address CD CD level achieved  their pressing and rapidly evolving challenges to scale-up RE V MAR REPORT FY23 ANNEX III. METHODOLOGY As part of the process of continuing improvements for the FY23 MAR, Management and IEG worked collaboratively on refining the FY23 assessment framework and criteria. The assessment framework continues to use the three primary categories as in FY22—limited evidence of progress (LE), emerging evidence of progress (EE), and change of direction of progress (CD)—but the associated definitions for these criteria and how to apply them have been sharpened (see Table III.I). An important and CODE-requested addition was made to the assessment framework involving an early warning system to identify and flag recommendations that are not on track to deliver results and to review the assessment framework for ensuring greater candor in the self-assessment. The jointly chosen approach involves the introduction of a new category to the assessment framework called “progress constrained (PC)” and it is intended to be used judiciously. Table III.I: MAR Assessment Framework FY23 Progress Levels Description • No progress Progress Constrained • Valid evidence on progress unavailable or inadequate (PC) • Reporting in year 3 or 4 continuing to show insufficient evidence of progress • Activities delivered and knowledge generated linked to the recommendation. Limited evidence (LE) • Skills developed linked to recommendation Activities conducted • Limited new evidence of progress since previous MAR • Evidence of developed capacities or application/utilization of outputs – such as Emerging evidence (EE) processes, IT systems, and guidance implemented, supported by limited Demonstrating examples. application of outputs  • Anecdotal evidence of changes in behavior Change in direction (CD) • Meaningful change in behavior in the intended outcome of the Demonstrating recommendation that is likely to be sustained systematic behavior • Implemented systems changes e.g., incentives, financing mechanisms, changes processing, new standards being applied across relevant portfolio. Box III.I: Criteria for the Retirement of Recommendations for FY23 CHANGE IN TIMING CHANGE IN EXTERNAL CHANGE IN INTERNAL DIRECTION CONTEXT CONTEXT Change in direction The reporting period for The outcome of the Management can make no rating achieved. recommendations is up to recommendation is no longer further progress or no longer five years. Reporting usually relevant, for example, agree with the outcome of a takes place over four years, through force majeure or a recommendation; IEG has starting one fiscal year after change in global priorities. conducted a fresh evaluation a report was discussed at that will supersede earlier CODE. Reporting can start recommendations. early if decided by WBG Management. VI MAR REPORT FY23 FY23 MAR VII