Validation of Management’s Progress on Independent Evaluation Group Recommendations An Independent Evaluation © 2021 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION Please cite the report as: World Bank. 2021. Validation of Management’s Progress on Independent Evaluation Group Recommendations. Independent Evaluation Group. Washington, DC: World Bank. COVER PHOTO Adapted from shutterstock/Tom Gowanlock EDITING AND PRODUCTION Amanda O’Brien GRAPHIC DESIGN Luísa Ulhoa This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The bound- aries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissem- ination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Validation of Management’s Progress on Independent Evaluation Group Recommendations An Independent Evaluation September 13, 2021 Contents Abbreviations ..................................................................................................................................................... v Acknowledgments ..........................................................................................................................................vii Overview ............................................................................................................................................................. ix Report to the Board from the Committee on Development Effectiveness ................................ x 1. Introduction ...................................................................................................................................................1 2. Process and Methods .................................................................................................................................2 3. IEG’s Assessment of Management’s MAR Report ...........................................................................3 Outcome Focus ............................................................................................................................................ 3 Quality of Evidence .....................................................................................................................................4 Implementation Progress ......................................................................................................................... 11 4. Proposals to Retire Specific Recommendations............................................................................ 14 5. Conclusions and Suggestions .............................................................................................................. 16 References ........................................................................................................................................................ 18 Boxes Box 3.1. Independent Evaluation Group’s Criteria for Assessing Management Action Record Evidence................................................................................................................................................................ 9 Box 3.2. Assessing Implementation Progress ......................................................................................... 12 Tables Table 3.1. Intended Outcomes, Implementation Progress, and Evidence Quality .......................5 Appendixes Appendix A. Scope of Management Action Record 2021 ............................................................. 20 iii Abbreviations FY Fiscal Year IEG Independent Evaluation Group IFC International Finance Corporation MAR Management Action Record v Acknowledgments This report was led by Rasmus Heltberg, under the supervision of Sophie Sirtaine and Alison Evans. Luis Alvaro Sanchez and Onno Ruhl were members of the core team. Many Independent Evaluation Group team leaders made valuable contributions. Zorayda Opana and Gaby Loibl provided administrative support. vii Overview This document is the validation by the Independent Evaluation Group (IEG) of the report entitled Using Evaluative Evidence to Deliver Development Outcomes: A World Bank Group Management Report on Implementation of IEG Recommendations FY17–21, which is management’s report on the Management Action Record (MAR; World Bank 2021). The MAR system tracks follow-up to IEG evaluations. There are 15 evaluations in this year’s MAR report. The document covers IEG’s synthesis of progress toward achieving IEG evaluations’ intended outcomes and its assessment of the approach and evidence in management’s MAR report. Progress has been made toward making the MAR more useful, focused on outcomes, and aligned with the World Bank Group’s broad strategic priorities, though gaps in evidence remain. The MAR report’s evidence suggests good implementation progress has been made on many IEG evaluations, including capital markets, citizen engagement, International Finance Corporation client engagement, forced displacement, health services, knowledge flow and collaboration, and regional integration. There has also been implementation progress on many other evaluations, including core diagnostics related to the creation of markets and urban resilience evaluations; carbon fund consolidation related to the carbon finance evaluation; and analytics, technical assistance, and trust fund consolidation for pollution control work. More progress can be made on the quality, depth, and type of evidence. The evidence in the 2021 MAR report on progress toward key outcomes or outputs is broadly satisfactory for about half of the 15 covered evaluations. For many of the remaining evaluations, the evidence quality makes it hard to understand the extent of implementation progress toward outcomes. Evidence on implementation and outcomes is particularly important for recommendations proposed to be retired. Areas for future improvement of the MAR therefore include greater clarity on intended and achieved outcomes, more understanding of plausible links from implementation actions to intended outcomes, better evidence regarding implementation actions and trends, and candid dialogue and learning. This assessment makes suggestions for how to progress in these directions. IEG can accept most of the proposals to retire specific recommendations, with some qualifications, but suggests continued reporting on carbon fund coordination. ix Report to the Board from the Committee on Development Effectiveness The Committee on Development Effectiveness met to consider the World Bank Group management report Using Evaluative Evidence to Deliver Development Outcomes—A World Bank Group Management Report on Implementation of Independent Evaluation Group Recommendations FY17–21 together with the Validation of Management’s Progress on Independent Evaluation Group Recommendations: An Independent Evaluation. The committee welcomed the World Bank Group management’s Management Action Record (MAR) report (MAR) and Independent Evaluation Group (IEG) validation report, and the constructive and collaborative IEG–management engagement. Members highlighted that the MAR process was moving in the right direction and was leading to building feedback loops; refining the quality, practicality, and strategic relevance of IEG's recommendations; and improving the way management responds to, tracks, and reports on progress toward the intended outcomes of IEG evaluations and recommendations. Yet, members noted there was still room to improve the way plausible causal links from actions to outcomes are defined and tested and the way the evidence is used to discern the extent of progress. Also, there is a need to reach a shared understanding of the evaluations’ intended outcomes. Members agreed with the need for IEG and management to continue collaborating closely to develop a systematic approach to improve evidence collection, define outcome-oriented progress milestones, and ensure a consistent framework to report on progress and assess evidence of progress on the implementation of IEG recommendations. Members appreciated management’s efforts to develop a systematic approach for reporting on IEG recommendations, the thematic grouping of management’s actions along IEG’s six workstreams, and aligning them with corporate strategic priorities. They welcomed management’s and IEG’s continuous efforts to improve development outcomes based on strong evidence and knowledge sharing and were pleased to learn that the World Bank is taking a leading role in the Multilateral Development Bank Working Group on Managing for Development Results. Some highlighted the importance of the MAR process to provide enough flexibility to adjust to the post- coronavirus pandemic reality and enable IEG and management to prioritize the most relevant recommendations. They encouraged management to consider reviewing the strategic priorities or framework in situations where there is lackluster client demand, to remain responsive to clients. The committee supported Bank Group management’s and IEG’s proposal to retire 20 recommendations and to continue monitoring and demonstrating impact on the x Report to the Board from the Committee on Development Effectiveness remaining 39 recommendations, including those related to carbon finance funds. Some expressed concern regarding the retirement of recommendations related to knowledge management, underscoring that the best way to report progress on the knowledge agenda is through a concrete, actionable, and measurable Bank Group knowledge management action plan with timebound deliverables, as requested during the committee meeting on the Knowledge Strategic Framework. They welcomed management’s commitment to engage with committee on the Knowledge Strategic Framework follow up matrix. Members welcomed IEG’s and management’s commitment to continue working to address the issues raised by members, such as plausible links from actions to outcomes, the evidence regarding the progress, and the need to reach a shared understanding of the evaluations’ intended outcomes. xi 1. Introduction 1.1 The MAR is a key element of the World Bank Group’s accountability framework. The MAR supports accountability in the follow-up of Independent Evaluation Group (IEG) recommendations by enabling meaningful tracking, dialogue, and self-assessment of Bank Group management’s implementation of IEG recommendations. 1.2 The 2020 MAR reform was grounded in a common vision of how IEG recommendations can contribute to enhanced Bank Group development effectiveness through more strategic recommendations, more decisive management actions in response, and better tracking of recommendation implementation and the ensuing outcomes. It also aimed to achieve increased engagement of senior management and members of the Board of Executive Directors. As part of the MAR reform, many recommendations were “retired,” meaning there was no further reporting on them. IEG and management agreed that there will be fewer IEG recommendations in the future and they will be more strategically focused; management responses will clearly state agreements and disagreements with the recommendations; and management will report annually on the progress made toward desirable outcomes intended by IEG recommendations. 1.3 This document is IEG’s validation of management’s 2021 MAR report and is intended to complement the report, Using Evaluative Evidence to Deliver Development Outcomes: A World Bank Group Management Report on Implementation of IEG Recommendations FY17–21 (World Bank 2021). Both reports will be discussed simultaneously by the Committee on Development Effectiveness. This document presents IEG’s assessment of the approach used and evidence found in management’s MAR report; its synthesis of progress toward outcomes for the 15 evaluations in this year’s MAR cycle; its agreements and disagreements regarding proposals to retire specific recommendations; and its suggestions for how to continue improving the MAR’s usefulness. 1 2. Process and Methods 2.1 This is the second reporting cycle after the MAR reform. Given this, some aspects of process and method remain in flux. IEG, Operations Policy and Country Services, and the International Finance Corporation (IFC) had constructive engagements, with meetings to discuss the scope, timeline, and overall approach to outcome-based reporting. Seven meetings were held to present and discuss implementation progress in various thematic areas. IEG also had the opportunity to comment on an earlier version of the MAR report. 2.2 Dialogue between evaluators and operational staff is essential for evaluations to foster learning and deeper understanding and foster buy-in and implementation. This is true before, during, and after evaluations are completed. The thematic meetings mentioned above involved IEG teams, technical teams in charge of implementing the recommendations, and corporate units. The meetings discussed implementation progress and promoted understanding of the outcome-based reporting approach, what evidence IEG staff would like to see, and what actions management has taken and why. The direct personal engagement allowed for candid exchanges and enhanced understanding, but the meetings could have been more useful if they had been held earlier in the process, when there was more time to inform the final product. 2.3 This document is based on the evidence in management’s MAR report and the presentations and discussions during the meetings with management. IEG assessed this evidence against the technical and evaluative understandings that underpin the original recommendations. IEG arrived at its assessment through discussion, commenting, and deliberation among IEG staff. The assessment’s main methodological weakness is the reliance on evidence of mixed depth and quality in management’s MAR report and the absence of agreed-on benchmarks for assessing implementation progress. 2 3. IEG’s Assessment of Management’s MAR Report 3.1 Management’s MAR report demonstrates clear progress in implementing the MAR reform. The report’s framing around outcomes and its linking of IEG recommendations to Bank Group strategic priorities hold promise. The report gives the IEG recommendations much of the credit for influencing the Bank Group’s direction and development effectiveness (World Bank 2021). Good analysis has gone into grouping IEG recommendations functionally and thematically. The analysis that places evaluation follow-ups in the Bank Group’s broad strategic priority areas is likely to be useful to members of the Board of Executive Directors and management. The grouping by strategic priority areas makes synergies across evaluations explicit, stimulates thinking on strategic directions, and aligns with IEG’s work program. This more analytical and outcome-oriented approach is an improvement on the more tedious reporting on action plan outputs that was used before the MAR reform. The Bank Group’s new approach to MAR is likely to spark interest and replication among other multilateral development banks. 3.2 Yet the approach can be further improved in future years, which management recognizes. IEG agrees with the directions for future improvement outlined in the concluding section of management’s MAR report. Specifically, there is room for more conceptual clarity on outcomes, better data and evidence on outcomes achievement, and more candor. Candid discussion based on sound evidence regarding when management has acted on IEG recommendations, and when it has not, is essential for the MAR’s usefulness. IEG and management should collaborate to further develop the approach in future MAR cycles. Outcome Focus 3.3 Future MAR reports could strive for more clarity on the type and levels of outcomes to report. The MAR report moves between the long-term outcomes the Bank Group pursues through its broad strategic themes and the intermediate outcomes and actions implemented in response to specific IEG recommendations. The “Outcome Frameworks” in the MAR report’s tables 1–5 may have been helpful organizing devices for the report authors, but they may not be as helpful to many readers. The detailed sections of management’s MAR report tend to focus on activities, processes, and outputs, often with little evidence of outcomes. For example, the report describes some trust fund consolidations, which are processes, without discussing whether these resulted in more coherent or increased funding for the respective areas of work, which would be relevant outcomes. This area will need continued work and dialogue. 3 Chapter 3 IEG’s Assessment of Management’s MAR Report 3.4 It can be useful to focus future MAR reporting on the intended outcomes identified in IEG evaluations. Table 3.1 lists such outcomes for the 15 evaluations in this year’s MAR report in the form of brief summaries of what, in IEG’s view, the Bank Group would do differently if it fully implemented the evaluations’ recommendations. For example, the outcome orientation at the country level evaluation lays out a vision in its concluding chapter for a renewed country-level results system with tools, accountability, and incentives that better capture the Bank Group’s contributions to countries’ outcomes (World Bank 2020a). The vision that tied the knowledge flow and collaboration evaluation’s six recommendations together was to improve the functioning of the World Bank’s matrix system; the recommendations specified IEG’s view of the actions management should take to get there—namely, improving the approach to knowledge, ensuring robust connectors between the two sides of the matrix (directors and program leaders), adjusting incentives for collaboration, having more contestability in quality assurance, and paying continuous attention to the World Bank’s organizational effectiveness. 3.5 For existing evaluations, the descriptions of evaluations’ intended outcomes are explanatory vision statements, and they do not replace IEG recommendations. IEG recommendations focus on specific actions and are written to support the MAR’s accountability. IEG extracted the outcome descriptions from the evaluations’ conclusions, synthesized them from the recommendations, or adapted them from management’s MAR report. In future evaluations, IEG will endeavor to clarify the intended outcomes of its recommendations to support MAR reporting. Quality of Evidence 3.6 In IEG’s assessment, the evidence on progress toward key outcomes or outputs is broadly satisfactory for half of the covered evaluations. For these, IEG was able to make a qualitative assessment of implementation progress. The evidence is either partially satisfactory or unsatisfactory for the remainder of the evaluations. In many instances, IFC provided more detail than the World Bank. Table 3.1 shows IEG’s assessment of the evidence’s quality and relevance, and box 3.1 describes the criteria and considerations IEG used to assess the evidence. 4 Chapter 3 IEG’s Assessment of Management’s MAR Report Table 3.1. Intended Outcomes, Implementation Progress, and Evidence Quality Progress toward Outcomes and Description of the Implementation of Key Actions Quality of Evidence in Evaluation Evaluation’s Intended as Reported in Management’s Management’s MAR Topic Outcomea MAR Report Reportb Capital More coherent and integrated Substantial progress, including via Satisfactory evidence. The markets capital market development the Joint Capital Markets initiative, information is relevant and programs across the World which seeks to foster coherent adequate and covers Bank Group underpinned by capital market development outputs with likely links to diagnostics, improved programs. intended outcomes. There knowledge management, and is quantitative data on the adequate funding. number of countries that have been assisted with integrated capital market development programs and amounts mobilized. Rural nonfarm World Bank interventions will It is not clear what and how much Unsatisfactory evidence. economy better target context-specific action management has taken on Major elements of the constraints to nonfarm identifying context-specific recommendations were employment, income, and constraints to rural poverty not covered. productivity faced by rural reduction, though there has been poor people, women, and apparent progress on knowledge youth. products, including rural income diagnostics. Also, the MAR report focuses on a generic screening process that applies to every single World Bank operation and does not discuss how rural project interventions align to target relevant constraints or whether monitoring and evaluation practices have changed. Pollution Increased World Bank efforts The World Bank has taken many Partially satisfactory help countries be more aware actions regarding analytics, evidence. The MAR report of the most serious pollution technical assistance, and trust fund could include trend data issues and how to monitor and consolidation for pollution control to discern change over address these, leveraging the work. IFC has provided advice on time and evidence on climate change portfolio as compliance with performance progress toward outcomes well to combat pollution. IFC standards. such as whether Bank clients enhance compliance Group–supported with performance standards on analytical work is helping pollution. countries establish pollution control priorities. Trade More effective Bank Group Cannot discern progress on Unsatisfactory evidence. facilitation promotion of trade facilitation implementing the Very limited evidence and via a programmatic approach recommendations. no quantitative data were that addresses binding provided, making it constraints to reforms and impossible for the monitors public policy impacts. Independent Evaluation Group (IEG) to assess implementation progress. 5 Chapter 3 IEG’s Assessment of Management’s MAR Report Progress toward Outcomes and Description of the Implementation of Key Actions Quality of Evidence in Evaluation Evaluation’s Intended as Reported in Management’s Management’s MAR Topic Outcomea MAR Report Reportb Forced The relevant IFC has made solid progress on Satisfactory evidence. The displacement recommendation’s intended advisory services, knowledge, evidence is relevant, albeit outcome is for the Bank Group partnerships, and investment at the activity level. More to be more engaged and projects, though more time is outcome evidence would collaborative in supporting needed for outcomes to emerge. be expected as private sector development The retirement last year of implementation that can benefit displaced recommendations to the World progresses over time. people and host communities. Bank has led to IFC reporting on its own activities, with no reporting on Bank Group collaboration on supporting private sector development for displaced people and host communities. Carbon Align different carbon finance The presented evidence points to Partially satisfactory finance initiatives and instruments, substantial progress on evidence. While some mainstream them in country consolidating and mainstreaming elements of the programs, and use them in carbon financing instruments, an recommendations are support of transformational important output. covered well, there is approaches with poverty limited evidence on reduction co-benefits. recommendations regarding finance mobilization, greenhouse gas emission reduction, and poverty reduction co- benefits. As implementation progresses, MAR reporting could provide evidence on progress toward intended outcomes, including aligning the funds’ visions, governance, rules, and results frameworks. Citizen Apply a deeper, more strategic This evaluation has influenced a Satisfactory evidence, engagement approach to engaging with dynamic agenda that continues to including on activities with citizens when opportunities enjoy strong ownership. The MAR likely strong links to arise. report describes important activities, outcomes. Future MAR outputs, and processes, including a reporting could focus policy commitment in the 19th more on evidence of Replenishment of the International progress toward the Development Association on evaluation’s intended establishing citizen-engagement outcome of deeper, country platforms. Such platforms “thicker,” and more hold promise to advance the deeper strategic citizen and more strategic citizen engagement. engagement at the recommendations’ core. Regional Increase regional integration There has been strong uptake of Satisfactory evidence. The integration initiatives in underserved this report’s findings and MAR report relevantly and regions, based on tailored, recommendations, including in this adequately covers the 6 Chapter 3 IEG’s Assessment of Management’s MAR Report Progress toward Outcomes and Description of the Implementation of Key Actions Quality of Evidence in Evaluation Evaluation’s Intended as Reported in Management’s Management’s MAR Topic Outcomea MAR Report Reportb evidence-informed International Development recommendations’ major approaches. Association cycle, via high-level elements. strategic commitments in all Regions, and in the recent Africa regional strategy. IFC client Leverage strategic clients to Satisfactory evidence. The engagement mobilize financing and After a slow start due to IFC’s MAR report adequately enhance development impact organizational adjustments, work is covers actions and in priority and underserved in progress on creating staff outputs. More evidence on sectors. incentives and accountabilities, how these activities lead simplifying approval processes, and to outcomes would be defining an approach for selection welcome. and segmentation of clients according to strategic priorities. IFC also has updated human resources competencies and strengthened diagnostic work and staff incentives for upstream engagements. Health Provide better measurement of There has been substantial Satisfactory evidence. The services health service quality and implementation progress. The MAR MAR report has relevant equity, stronger World Bank– report describes important activities descriptions of progress IFC synergies, more support to with the potential to promote on most clients in improving pandemic outcomes in all of the evaluation’s recommendations, though preparedness capacity, and four outcome areas, including with a focus on more strategic alignment and pandemic preparedness. diagnostics and other selectivity of Bank Group activities, limited global partnership program quantitative data, and no engagements. evidence of outcomes. A planned IEG evaluation will assess some relevant outcomes, including on pandemic preparedness. Knowledge Improve the functioning of the Most key actions were completed, Satisfactory evidence. The flow and World Bank’s matrix system via including via two rounds of staff MAR report has relevant collaboration an improved approach to realignment. descriptions of key actions. knowledge, robust connectors The outcome assessment between the two sides of the (that is, How well does the matrix (directors and program operating model leaders), incentives that function?) will await a promote collaboration, more planned future IEG contestability in quality evaluation. assurance, and continuous attention to organizational effectiveness. Creating Look for and act on There has been progress on Country Partially satisfactory markets opportunities more Private Sector Diagnostics and using evidence. The description systematically to create these to influence country is only partially relevant markets by either investing partnership frameworks’ design. and is confusing in places. The description has limited evidence of deeper 7 Chapter 3 IEG’s Assessment of Management’s MAR Report Progress toward Outcomes and Description of the Implementation of Key Actions Quality of Evidence in Evaluation Evaluation’s Intended as Reported in Management’s Management’s MAR Topic Outcomea MAR Report Reportb directly or working on the engagement and enabling environment. outcomes beyond undertaking Country Private Sector Diagnostics. It is not clear that the report addresses the recommendation about IFC’s financial sustainability. Urban Strengthen the Bank Group’s There has been progress on key Partially satisfactory resilience capacity to support client diagnostics for urban resilience such evidence. The description countries in building urban as the City Scan tool and the focuses on activities and resilience to chronic stresses Resilience Rating System. lacks clarity. Future MAR and disaster shocks. reporting could focus on how the activities and diagnostic tools link to the intended outcome and progress toward integrating resilience considerations in operations’ cost-benefit analysis. World Bank Ensure more effective Bank With regard to the World Bank, it isUnsatisfactory evidence Group global Group convening on global unclear if there has been progress for the World Bank. The convening issues through more selective toward more intentional scoping focus on trust fund and scoping, life cycle management and stronger management of Financial Intermediary of convening initiatives, and convening besides trust fund and Funds reforms aligns stronger links with country Financial Intermediary Funds partially with the programs. reforms. evaluation’s focus on Progress has been made in ensuring convening activities but IFC convening efforts are strategic does not cover the and selective. recommendation on better aligning of convening initiatives with country activities. Partially satisfactory evidence for IFC, with a relevant focus on IFC’s management of convening activities though with some gaps. Outcome Improve accuracy, utility, and The World Bank’s actions have a Satisfactory evidence, with orientation at outcome orientation of the number of positive elements in line key actions adequately the country country-level results system via with the evaluation’s described. level tools, principles, and incentives recommendation, such as that better capture the Bank articulating a line of sight to higher- Group’s contribution to level outcomes over a longer time country outcomes. period, adjusting evaluation timing, and offering more flexibility for teams to report on contributions to 8 Chapter 3 IEG’s Assessment of Management’s MAR Report Progress toward Outcomes and Description of the Implementation of Key Actions Quality of Evidence in Evaluation Evaluation’s Intended as Reported in Management’s Management’s MAR Topic Outcomea MAR Report Reportb outcomes. Management has yet to address the evaluation’s other findings regarding the results system, including its skewing of incentives toward approvals, commitments, and outputs rather than outcomes; low utility; weak evidence on achievement of country programs’ objectives; and low use by teams and clients. Dialogue on how to continue progressing toward the evaluation’s outcome is ongoing. Source: Independent Evaluation Group; World Bank 2016, 2017a–b, 2018a–d, 2019a–f, 2020a–b, 2021. Note: IEG = Independent Evaluation Group; IFC = International Finance Corporation; MAR = Management Action Record. a. The column shows IEG’s reconstruction of the intended outcomes implicit in the 15 evaluations in this year’s MAR cycle. IEG extracted the outcome statements from the evaluations’ conclusions, synthesized the recommendations, or adapted language from management’s MAR report. b. The column shows IEG’s assessment based on the criteria described in box 3.1. c. The evaluation was retired in fiscal year 2020 for the World Bank, while IFC continues reporting on the recommendation to the Bank Group to catalyze private sector solutions to promote resilience of displaced and host communities. The entry relates to IFC’s reporting. Box 3.1. Independent Evaluation Group’s Criteria for Assessing Management Action Record Evidence The Independent Evaluation Group’s (IEG) technical assessment of implementation progress requires (at a minimum) a solid account of how recommendations have been acted on and, ideally, how the implementation of recommendations may plausibly contribute to desirable outcomes. IEG found the evidence provided in management’s Management Action Record (MAR) report on implementation and progress toward outcomes to be less than fully satisfactory in half the cases, as described below: • Satisfactory evidence. The presented evidence is relevant to, and adequately covers, the recommendations. The evidence relates to outcomes—or to activities and outputs with plausibly strong links to outcomes—and covers the major elements of the recommendations. For example, the MAR report allowed IEG to discern the type and range of actions taken to implement the recommendations made in the capital markets and the outcome orientation of country programs evaluations. Establishing the Joint Capital Markets initiative and related tools and mechanisms for enhanced Bank Group coherence is a relevant and important outcome for the capital markets evaluation. For the outcome orientation of country programs’ evaluations where actions are recent and no outcomes can yet be expected, the description is sufficient for IEG to discern what elements of the recommendation Bank Group management has or has not yet acted on. For the citizen engagement, knowledge flow and collaboration, and regional integration evaluations, management’s MAR report describes relevant and important strategy and 9 Chapter 3 IEG’s Assessment of Management’s MAR Report International Development Association commitments with plausible links to important outcomes. • Partially satisfactory evidence. The presented evidence is partially relevant to the recommendations or partially covers their major elements. For example, the MAR report’s coverage of the carbon finance evaluation leaves some recommendations unaddressed, including those on finance mobilization, greenhouse gas emission reductions, and poverty reduction co-benefits. Likewise, for the creating markets evaluation, the MAR report does not address actions taken in response to recommendations to enhance access to markets for underserved groups and monitor how market creation activities affect poor people. • Unsatisfactory evidence. The presented evidence is not relevant to the recommendations or does not cover major aspects of the recommendations. For example, the MAR report’s coverage of the convening (World Bank portion), rural nonfarm economy, and trade facilitation evaluations leaves many aspects of the recommendations unaddressed, making it impossible for IEG to assess management’s implementation progress or even the direction of travel. For the convening, the World Bank’s coverage centers on new frameworks for managing trust funds and Financial Intermediary Funds, which is only partially relevant to the evaluation’s focus on how global initiatives are selected, managed, and linked to country programs, regardless of how they are financed. Source: Independent Evaluation Group. 3.7 More quantitative data would support statements about progress. For many evaluations, management presents single-point data even as trend data would be needed to discern whether efforts have increased. For example, management’s MAR report states, “Bank Group support for trade facilitation is increasingly based on a programmatic approach” (World Bank 2021, 29), but it does not indicate the number of countries with programmatic approaches or how this number changed after the trade facilitation evaluation. Also, specifying what was done during the past year and what actions were taken earlier would help indicate the pace of implementation progress. 3.8 Stronger reporting is particularly important for evaluations that are in their last year of MAR reporting or proposed for retirement. Ideally, management would sum up key actions taken, progress toward the evaluation’s intended outcomes and Bank Group strategic priorities, any lessons, and whether reporting will continue via other channels. Stronger reporting is especially appropriate for evaluations that management proposes to retire early. However, for some evaluations, outcome data are not yet available, as reforms need time before bearing fruit. For other evaluations, the intended outcome is complex and may require more extensive evaluative work than what is feasible for the MAR to assess. This is the case for knowledge flow and collaboration, for example, where the intended outcome is more effective functioning of the World Bank’s matrix management system. IEG can agree to retire this evaluation’s recommendations in part 10 Chapter 3 IEG’s Assessment of Management’s MAR Report because it has a planned follow-up evaluation in its work program that would assess the matrix management system’s functioning. Implementation Progress 3.9 Implementation of IEG recommendations continues to help influence the direction and development effectiveness of the Bank Group. Though the magnitude of progress is often hard to assess, progress toward intended outcomes can be detected for the capital markets, citizen engagement, health services, knowledge flow and collaboration, and regional integration evaluations. This is according to evidence in the MAR report, as summarized in table 3.1. For example, establishing the Joint Capital Markets initiative and related tools and mechanisms has enhanced the coherence of the Bank Group’s capital market development programs, a key intended outcome of the capital markets evaluation. The International Development Association policy commitment on establishing citizen-engagement country platforms holds promise to advance deeper and more strategic citizen engagement, which is core to the citizen engagement evaluation’s recommendations. Staff realignments, the creation of regional director positions, and the development of the Strategic Framework for Knowledge are major efforts in the spirit of the knowledge flow and collaboration evaluation’s recommendations, with potential to improve the World Bank operating model’s functioning. Regarding the health services evaluation, the World Bank has taken important steps, including by strengthening pandemic preparedness and improving alignment and selectivity of its partnerships in public health. There has been strong uptake and use of the regional integration evaluation, including in the recent Africa regional strategy update, which explicitly builds on this evaluation. There has been clear progress on implementing outputs and activities related to many other evaluations’ recommendations, though outcomes have not yet materialized. For example, there has been progress on core diagnostics relevant to the creating markets and urban resilience evaluations’ recommendations, as well as on trust fund consolidation related to the carbon finance evaluation. The evaluation of outcome orientation at the country level has sparked recent guideline changes and dialogue on how to make the country-level results system more useful. IFC has acted on IEG recommendations to ensure that IFC’s convening efforts are strategic and selective, it better leverages strategic clients, and it is more engaged in supporting private sector development that can benefit displaced people and host communities. 3.10 IEG cannot validate management’s claim that “[the] quantitative and qualitative evidence demonstrates that the Bank Group has taken substantial action in 95 percent of recommendations, and roughly half of those actions are already delivering outcomes” (World Bank 2021, 6). The claim is based on an unvalidated assessment framework. The 11 Chapter 3 IEG’s Assessment of Management’s MAR Report MAR report’s annex 3, which covers the World Bank only, operates with four assessment categories: substantial outcomes, moderate outcomes, substantial outputs, and moderate outputs. To validate claims about the share of recommendations where management has taken “substantial action” or reached outcomes, there would need to be a clearer understanding of, and agreement on, the criteria that distinguish outcomes from outputs and moderate outputs and outcomes from substantial outputs and outcomes. A credible methodology to distinguish between small steps in the right direction and substantial outputs or outcomes has not been established (box 3.2). However, IEG appreciates management’s statements about the many areas where IEG evaluations have helped influence the Bank Group’s actions, confirming that independent evaluation plays an important role for the Bank Group’s organizational change and learning. Box 3.2. Assessing Implementation Progress The 2020 Management Action Record (MAR) reform aimed for increased focus on progress toward desirable outcomes at the strategic level. The Independent Evaluation Group (IEG) and management agreed that there should be a deliberate break with the previous system, which focused people’s attention on ratings of action plan implementation progress and ratings disputes and was based on granular data on management’s action plans, with sometimes tenuous links to the desirable outcomes intended by IEG recommendations. Instead, the objective was to have meaningful dialogue and reporting on progress toward IEG recommendations’ desirable outcomes. IEG has concerns regarding the World Bank’s framework for self-assessing progress toward outcomes in the 2021 MAR report’s annex 3. Because there are no transparent criteria and targets for the complex strategic outcomes that the MAR reform aspires to focus on, it is hard to establish a rigorous methodology for assessing progress. For IEG to validate management’s assessment of progress, the methodology would need to be jointly agreed to. Further, there is the risk of derailing focus from desirable outcomes, lessons learned, and stakeholder dialogue to ratings and ratings disputes. Work remains to be done in future MAR cycles on improving the evidence and developing criteria for assessing progress toward desirable outcomes. Source: Independent Evaluation Group; World Bank 2021. 3.11 Some IEG evaluations have clearly seen more traction. Judging from this and last year’s management MAR report, and the technical MAR discussions, the capital markets; citizen engagement; health services; knowledge flow and collaboration; Program-for-Results; regional integration; Systematic Country Diagnostics and Country Partnership Framework; and World Bank Group engagement in situations of fragility, conflict, and violence evaluations have seen much uptake and informed strategic, conceptual, and organizational changes. These evaluations were timely. Many of them were early assessments of programs or processes and arrived in time to inform strategic decisions or process adjustments. They achieved buy-in from champions in Bank Group 12 Chapter 3 IEG’s Assessment of Management’s MAR Report management, thanks to outreach and engagement efforts, and their recommendations pointed in a direction Bank Group management was willing to go. 3.12 Candor is important for the MAR process to function well. Candor is needed to foster learning on how, when, and why IEG’s evaluations influence concepts, thinking, learning, strategies, and organizational action and change—and when they do not. IEG appreciates when management’s MAR report is clear on where little progress has been made so far, as is the case on monitoring the public policy objectives of trade regulations and efforts to build urban resilience by strengthening mechanisms to address crime and violence. There are also other cases, however, where IEG would argue that the implementation progress has been limited and where candid discussion might promote learning and understanding. This includes, for example, some of the recommendations on scoping and managing convening initiatives more intentionally; creating budget systems that better incentivize collaboration; having stronger coordination across carbon funds; and implementing project interventions that better target the context-specific constraints to nonfarm employment, income, and productivity faced by rural poor people. 13 4. Proposals to Retire Specific Recommendations 4.1 Management proposes to retire 21 recommendations from future MAR reporting. IEG appreciates that management’s proposal reflects its earlier feedback, to a degree. IEG has the following specific remarks on management’s proposals and the justifications behind the proposals. Overall, IEG agrees to retire all proposed recommendations, except one related to carbon finance: • Capital markets evaluation. Due to retire after the standard four-year reporting cycle, IEG is pleased with the progress toward desirable outcomes. • Carbon finance evaluation. Management proposes to retire three recommendations on country focus and co-benefits, enhanced emphasis on the greenhouse gas emission reduction impact of carbon pricing, and coordination across different carbon finance funds. IEG agrees to retire the first two recommendations, given the work done. IEG suggests continued reporting on the recommendation on coordination across different carbon finance funds. Management has recently consolidated the carbon finance fund architecture. This action is in the recommendation’s spirit but is, in itself, insufficient to reach the desirable outcomes described in the recommendations, such as shared vision, common governance systems, simpler rules, and well-functioning and consistent results frameworks across funds. It would make sense to report on how the consolidation affects coordination across funds, enables streamlining of approaches, improves synergy, and supports responding to the Paris Agreement, as well as on how management avoids future fragmentation of funds. IEG’s position would therefore be to continue reporting on recommendations 1,2, and 5 and to retire recommendations 3 and 4, using the evaluation’s original numbering. • Health services evaluation. IEG agrees to retiring the two recommendations on World Bank and IFC synergy in public-private interactions and stronger alignment and selectivity of partnership engagements. IEG appreciates the continued reporting on the recommendations for improved measurement of health service quality and pandemic preparedness. As the World Bank works to address the COVID-19 crisis, it will be important to focus on strengthening health systems’ preparedness for future pandemics as well. • IFC client engagement. IFC proposes to retire the recommendation on accountability and incentives for strategic client relationship management because it has created a new human resources competency framework and new investment processing initiatives. IEG would have liked to have seen evidence 14 Chapter 4 Proposals to Retire Specific Recommendations on client relationship outcomes from these actions but can agree with management’s proposal to retire the recommendation. • Knowledge flow and collaboration evaluation. World Bank management realigned the matrix system’s reporting lines and formulated a new strategic framework for knowledge, among other things. IEG does not believe that there has been much progress on reforming World Bank budget systems to better incentivize collaboration as stated by management. Even so, IEG does not see any reason to oppose management’s proposal to retire the evaluation from reporting, given plans to conduct future evaluations on collaboration and knowledge-related aspects. • Rural nonfarm economy evaluation. IEG and management have struggled to reach a shared understanding of the extent of the progress in implementing the evaluation’s recommendations. The MAR report has evidence on many outputs and activities but not on how these are making a difference in how the World Bank addresses rural poverty. There is no shared understanding of the evidence that would be needed to make a compelling case for implementation progress and achievement of intended outcomes. IEG continues to see merit in the recommendations but acknowledges that management does not see value in continued reporting on an evaluation whose recommendations are unlikely to see further action. 15 5. Conclusions and Suggestions 5.1 This validation report concludes that there has been progress toward making the MAR more useful, focused on outcomes, and aligned with the Bank Group’s broad strategic priorities. IEG can accept most of management’s proposals to retire specific recommendations for a number of reasons, including the completion of the standard four annual reporting cycles, the completion of key actions, evidence of or plausible links to outcomes, and plans to conduct future IEG evaluations that will assess outcomes in some areas. The one exception is the recommendation on coordination across different carbon finance funds, which IEG proposes continued reporting on to track progress toward more streamlined funds after the recent carbon fund consolidation. 5.2 IEG believes, however, that more progress is feasible and desirable regarding the quality, depth, and type of evidence and what management provides in its MAR report. IEG is particularly concerned about the quality of evidence for some of the recommendations proposed to be retired and has outstanding concerns about the justification and evidence behind some of the proposals. These concerns point to areas for future improvement of the MAR, such as greater clarity on intended and achieved outcomes, more detailed and relevant descriptions of implementation actions, more discussion of outcomes or plausible links to outcomes, more quantitative data to discern whether efforts increased, and more candid dialogue. 5.3 Moving forward, IEG makes the following suggestions based on its analysis and assessment of the MAR process this year and in previous years: i. Maintain the organization of the MAR reporting by strategic priority areas, which is helpful, while some methodological aspects should be further improved. ii. Continue technical dialogue between evaluators, teams in charge of implementing the recommendations, and corporate units. Technical dialogues are essential for trust, understanding, and learning and could be made a structured part of the MAR process to be held at key junctures. iii. Ensure collaboration between IEG and management on ways to improve evidence collection and frameworks for assessing implementation progress. This collaboration could take the form of a learning-oriented pilot for a few selected evaluations. The objectives of the collaboration would include more aligned expectations on desirable and feasible evidence; improved evidence collection systems; and shared criteria for assessing progress toward outcomes. 16 Chapter 5 Conclusions and Suggestions iv. Have IEG and management do more to reach a shared understanding of evaluations’ intended outcomes. Some of the different views between IEG and management on progress and evidence may in fact stem from different visions of the intended outcomes. v. Hold candid discussions on progress, evidence, intended outcomes, and areas of agreement and disagreement to foster learning about how IEG’s evaluations can best contribute to the Bank Group’s development effectiveness. 17 References World Bank. 2016. The World Bank Group’s Support to Capital Market Development. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2017a. Growing the Rural Nonfarm Economy to Alleviate Poverty: An Evaluation of the Contribution of the World Bank Group. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2017b. Toward a Clean World for All: An IEG Evaluation of the World Bank Group’s Support to Pollution Management. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018a. Carbon Markets for Greenhouse Gas Emission Reduction in a Warming World. Independent Evaluation Group. Washington DC: World Bank. World Bank. 2018b. Engaging Citizens for Better Development. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018c. The International Finance Corporation’s Approach to Engaging Clients for Increased Development Impact. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018d. World Bank Group Support to Health Services: Achievements and Challenges. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019a. ‘Creating Markets’ to Leverage the Private Sector for Sustainable Development and Growth: An Evaluation of the World Bank Group’s Experience through 16 Case Studies. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019b. Building Urban Resilience: An Evaluation of the World Bank Group’s Evolving Experience (2007–17). Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019c. Grow With the Flow: An Independent Evaluation of the World Bank Group’s Support to Facilitating Trade 2006–17. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019d. Knowledge Flow and Collaboration under the World Bank’s New Operating Model. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019e. Two to Tango: An Evaluation of World Bank Group Support to Fostering Regional Integration. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019f. World Bank Group Support in Situations Involving Conflict-Induced Displacement. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2020a. The World Bank Group Outcome Orientation at the Country Level. Independent Evaluation Group. Washington, DC: World Bank. 18 References World Bank. 2020b. The World’s Bank: An Evaluation of the World Bank Group’s Global Convening. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021. Using Evaluative Evidence to Deliver Development Outcomes: A World Bank Group Management Report on Implementation of IEG Recommendations FY17–21. Washington, DC: World Bank. 19 Appendix A. Scope of Management Action Record 2021 Table A.1 outlines the reports included in management’s MAR report and summarizes management’s proposed action. Table A.1. Scope of Management Action Record 2021 CODE Management’s Topic Discussion Year Scope of Reporting in 2021 Proposed Action Capital markets FY17 All recommendations included. Retire all four recommendations (age and other reasons). Rural nonfarm FY17 Recommendations for IFC were Retire the remaining five economy retired in 2020; others included. recommendations. Pollution FY18 All recommendations included. Trade facilitation FY19 All recommendations included. Forced displacement FY18 One recommendation included, for IFC (World Bank Group to catalyze private sector solutions to promote resilience of displaced and host communities). Carbon finance FY18 Recommendations pertaining to Retire three the World Bank included. recommendations.a Citizen engagement FY18 All recommendations included. Regional integration FY18 Recommendations 1 and 3–5 for the World Bank included. IFC client engagement FY18 Recommendation 2 retired and Retire one additional recommendations 3 and 6 recommendation. merged for reporting purposes. Health services FY19 All recommendations to the Retire two World Bank included; retired for recommendations. IFC. Knowledge flow and FY19 All recommendations included; Retire all six collaboration one recommendation partially recommendations.b agreed. Creating markets FY19 Recommendation 2 retired for IFC. Building urban FY20 First year of reporting; one resilience recommendation disagreed and four recommendations partially agreed. 20 Appendix A Scope of Management Action Record 2021 CODE Management’s Topic Discussion Year Scope of Reporting in 2021 Proposed Action World Bank Group FY20 First year of reporting. global convening Outcome orientation FY21 First year of reporting. of country programs Source: Independent Evaluation Group. Note: CODE = Committee on Development Effectiveness; FY = fiscal year. a. Management’s Management Action Record (MAR) report annex 4 table lists four recommendations, but the text clearly indicates the proposal to retire three. b. Management’s MAR report annex 4 indicates five recommendations, but the evaluation made six recommendations. 21 The World Bank 1818 H Street NW Washington, DC 20433