Results and Performance of the World Bank Group 2022 © 2022 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION Please cite the report as: World Bank. 2022. Results and Performance of the World Bank Group 2022. Independent Evaluation Group. Washington, DC: World Bank. COVER PHOTO Adapted from shutterstock/Sweet Art EDITING AND PRODUCTION Amanda O’Brien GRAPHIC DESIGN Luísa Ulhoa This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The bound- aries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissem- ination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Results and Performance of the World Bank Group 2022 December 7, 2022 Contents Abbreviations vii Acknowledgments viii Glossary of World Bank Group Ratings ix Overview xiii Management Comments xxvi 1. Background and Methodology�������������������������������������������������������������������������������� 1 Context 2 Methods and Data Sources 2 Main Evaluation Questions and Report Structure 4 World Bank Project Portfolio Performance�������������������������������������������������������������7 2.  Project Rating Trends and Patterns 9 Project Performance during the COVID-19 Pandemic 14 3.  International Finance Corporation and Multilateral Investment Guarantee Agency Project-Level Performance������������������������������������������������������������������������������������15 International Finance Corporation Investment Projects 17 International Finance Corporation Advisory Services 28 Multilateral Investment Guarantee Agency 33 Development Outcome and Other Ratings at the Country Program Level��������43 4.  Ratings Trends and Patterns 45 Analysis of World Bank Group Support and Country Partnership Framework Objectives 50 IFC and MIGA Contributions to Country Partnership Framework Objectives and Performance 55 World Bank Group Performance at the Country Program Level�������������������������62 5.  Country Partnership Framework Design and Implementation 66 Line of Sight 85 Advisory Services and Analytics in Country Programs���������������������������������������90 6.  ii Advisory Services and Analytics and Development Outcome 92 Advisory Services and Analytics and Bank Performance 99 7. Future Directions��������������������������������������������������������������������������������������������������107 Enhancing the Effectiveness of World Bank Group Support at the Country Program Level107 Strengthening Risk Identification and Mitigation 108 Monitoring Advisory Services and Analytics Use or Influence 108 Bibliography������������������������������������������������������������������������������������������������������������� 109 Boxes Enhancements in the Revised Country Engagement Guidance of July 2021 Box 5.1.   64 Criteria for Examining the Performance of the World Bank Group at the Box 5.2.  Country Level 65 Box 6.1. Criteria for the Examination of Advisory Services and Analytics 91 Importance of Integrating Advisory Services and Analytics with Other Bank Box 6.2.  Support 97 Figures Figure 2.1. Distribution of the World Bank’s Project Outcome Ratings 10 Bank Performance, Quality at Entry, and Quality of Supervision for World Figure 2.2.  Bank Projects: Investment Project Financing and Program-for-Results Projects 11 Bank Performance Ratings for World Bank Projects: Development Figure 2.3.  Policy Financing 12 Outcome and Bank Performance Rated MS+ for World Bank Projects: Figure 2.4.  Investment Project Financing and Program-for-Results 12  istribution of the World Bank’s Project Monitoring and Evaluation Quality Figure 2.5. D Ratings 13 Figure 3.1. Overall Development Outcome Ratings, 2012–21 17 iii Share of MS+ Development Outcome Ratings, by IFC Region, 2012–21 21 Figure 3.2.   evelopment Outcome Ratings of Projects in IDA and Non-IDA Countries, Figure 3.3. D 2012–21 22 Development Outcome Ratings of Projects in FCS and Non‑FCS, Figure 3.4.  2012–21 24 Development Outcome, Investment Outcome, Work Quality, and IFC’s Figure 3.5.  Additionality Ratings in FCS and Non-FCS 26 IFC Investment Projects with Advisory Services Projects Tend to Have Figure 3.6.  Better Development Outcomes and IFC’s Additionality 28 IFC Advisory Services Projects’ Development Effectiveness Ratings Figure 3.7.  29 Development Effectiveness Ratings of Projects in FCS and Non-FCS, Figure 3.8.  2012–21 31 Development Effectiveness Ratings of Projects in IDA and Non-IDA Figure 3.9.  Countries, 2012–21 32 MIGA’s Average and Disaggregated Development Outcome Ratings, Figure 3.10.  2012–21 34 Development Outcome Ratings of MIGA Projects in FCS and Non-FCS Figure 3.11.  and IDA and Non-IDA Countries, 2012–21 36 Development Outcome, Multilateral Investment Guarantee Agency’s Figure 3.12.  Role and Contribution, and Work Quality Ratings, 2012–21 38 Figure 4.1. Country Program Ratings, Fiscal Years 2011–21 45 Figure 4.2. Country Program Ratings by Region, Fiscal Years 2011–21 47 Moderately Satisfactory or Higher Development Outcome Ratings in the Figure 4.3.  Country’s Latest Completion and Learning Review (CLR) Review 48 Development Outcomes and Bank Group Performance Ratings (number Figure 4.4.  of Completion and Learning Review [CLR] Reviews) 49 Shift in Development Outcome and World Bank Group Performance Figure 4.5.  Ratings in Countries’ Latest Two Completion and Learning Review (CLR) Reviews 49 World Bank Group Performance at the Country Program Level Figure 5.1.  67 Figure 5.2. Dropped and Canceled Projects 85  eporting of Development Outcome in Advisory Services and Analytics 93 Figure 6.1. R Advisory Services and Analytics and Bank Performance Figure 6.2.  100 iv Tables Table O.1. Line of Sight Analysis xxii Country Partnership Framework Objectives in Key Areas of International Table 4.1.  Finance Corporation Activities 52 International Finance Corporation and Multilateral Investment Guarantee Table 4.2.  Agency Contributions by Country Partnership Framework Objective 56 International Finance Corporation and Multilateral Investment Guaran- Table 4.3.  tee Agency Support to Country Partnership Framework Objectives by Instrument 57 Performance of Objectives with International Finance Corporation and Table 4.4.  Multilateral Investment Guarantee Agency Contributions 59 Line of Sight Analysis Based on Completion and Learning Review (CLR) Table 5.1.  Reviews 86 Appendixes Appendix A. Random Sampling Methodology 116 Appendix B. Data Sources and Coverage 119 Appendix C. Methodology for Identifying Drivers of Performance at the Project Level 122 Appendix D. Project Rating Trends and Patterns 127 Appendix E. The International Finance Corporation and the Multilateral Investment Guarantee Agency 137 Appendix F. World Bank Group Support for Country Partnership Framework Objectives 163 Appendix G. Identifying Country Partnership Framework Priorities 170 Appendix H. World Bank Group Corporate Approaches 171 Appendix I. Building on the New Country Engagement Guidance 173 Appendix J. Risk Mitigation: A Continuing Problem 175 Appendix K. Integrating the International Finance Corporation and the Multilateral Investment Guarantee Agency in Country Programs 177 Appendix L. Trend Analysis of Advisory Services and Analytics 179 Appendix M. Unplanned Advisory Services and Analytics 185 v Abbreviations AAA analytic and advisory activities AIMM Anticipated Impact Measurement and Monitoring ASA advisory services and analytics CLR Completion and Learning Review CPE Country Program Evaluation CPF Country Partnership Framework CPIA Country Policy and Institutional Assessment CPS Country Partnership Strategy DPF development policy financing E&S environmental and social ESW economic and sector work EU European Union FCS fragile and conflict-affected situations FY fiscal year HLO high-level outcome IBRD International Bank for Reconstruction and Development IDA International Development Association IEG Independent Evaluation Group IFC International Finance Corporation Independent Evaluation Group World Bank Group    vii IMF International Monetary Fund IPF investment project financing ISN Interim Strategy Note M&E monitoring and evaluation MIGA Multilateral Investment Guarantee Agency P4R Program-for-Results PER Project Evaluation Report PLR Performance and Learning Review RAP Results and Performance of the World Bank Group RAS reimbursable advisory services RSF Risk-Sharing Facilities All dollar amounts are US dollars unless otherwise indicated. Acknowledgments This report was prepared by an Independent Evaluation Group (IEG) team co-led by Soniya Carvalho, lead evaluation officer, and Aurora Medina Siy, senior evaluation officer, under the overall direction of Alison Evans, Director-General, Evaluation, and with the guidance and supervision of Galina Sotirova, manager, Corporate and Human Development, and Oscar Calvo-Gonzalez, director, Human Development and Economic Management. The team included Shonar Lala Chinoy, Asita De Silva, Ellen Goldstein, Breda Griffith, Poonam Gupta, John Heath, Doruk Yarin Kiroglu, Yingjia Liu, Gaby Loibl, Albert Martinez, Daniel Palazov, Carla Pazce, Xiaoxiao Peng, Xiaolun Sun, Dileep Wagle, and Howard White. The report benefited from the advice of three external reviewers: Tamar Manuelyan Atinc (nonresident senior fellow, Brookings Institution); Monika Huppi (principal adviser, Office of Evaluation and Oversight, Inter-American Development Bank); and Barbara Scott (deputy director general, external cooperation management and Results and Performance of the World Bank Group 2022  Acknowledgments project development, Planning Institute of Jamaica). It also benefited from consultations with IEG colleagues, including past Results and Performance of the World Bank Group task team leaders, the IEG methods team, Implementation Completion and Results Report (ICR) Review coordinators, Completion and Learning Review (CLR) Review coordinators, IEG colleagues working on International Finance Corporation and Multilateral Investment Guarantee Agency, and IEG staff and consultants involved in the validation of self-evaluations. Contributions by Bank Group colleagues are gratefully acknowledged. viii Glossary of World Bank Group Ratings World Bank Ratings Projects » Outcome. The outcome rating is derived from relevance, efficacy (the extent of achievement of objectives), and efficiency. This criterion is rated on a six- point scale: highly satisfactory, satisfactory, moderately satisfactory, moder- ately unsatisfactory, unsatisfactory, highly unsatisfactory. » Bank performance. The Bank performance rating is derived from two sub- ratings, quality at entry and quality of supervision. This criterion is rated on a six-point scale: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, highly unsatisfactory. Country Programs » Development outcome. The development outcome rating is derived from the extent of achievement of the Country Partnership Framework (CPF) objectives, hence the use of “CPF objectives/development outcomes” in this Independent Evaluation Group World Bank Group    ix Results and Performance of the World Bank Group. Individual CPF objectives are rated on a five-point scale: achieved, mostly achieved, partially achieved, not achieved, not verified. Development outcome is rated on a six-point scale: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatis- factory, unsatisfactory, highly unsatisfactory. » World Bank Group performance. This rating is derived from the overall per- formance of the World Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA). This criterion is rated on a four-point scale: superior, good, fair, poor. International Finance Corporation Ratings Investment Projects » Development outcome. The project’s development outcome is measured across four indicators: project business performance, economic sustainabil- ity, environmental and social effects, and private sector development. This criterion is rated on a six-point scale: highly successful, successful, mostly successful, mostly unsuccessful, unsuccessful, highly unsuccessful. Each indicator of development outcome is rated on a four-point scale: excellent, satisfactory, partly satisfactory, and unsatisfactory. » IFC’s role and contribution (until calendar year 2015 Expanded Project Supervision Report program). This rating assesses how well the Interna- Results and Performance of the World Bank Group 2022  Glossary of World Bank Group Ratings tional Finance Corporation fulfilled this developmental role based on three operating principles—additionality/special contribution principle, business principle, and catalytic principle—as well as IFC’s timeliness, efficiency, and client satisfaction; relevance within IFC’s country strategy; and governance and environment. This criterion is rated on a four-point scale: excellent, sat- isfactory, partly unsatisfactory, unsatisfactory. » IFC’s additionality (starting calendar 2016 Expanded Project Supervision Report program) assesses the benefit or value addition IFC brings that a client would not otherwise have. This criterion is rated on a four-point scale: excellent, satisfactory, partly unsatisfactory, unsatisfactory. » IFC’s investment outcome assesses the extent to which IFC has realized at the time of evaluation and expects to realize over the remaining life of the investment the loan income and/or equity returns that were expected at ap- proval (four-point scale: excellent, satisfactory, partly unsatisfactory, unsatis- factory). » IFC’s work quality assesses IFC’s operational performance, including in re- lation to environmental and social aspects, with respect to precommitment work in (i) screening, appraisal, and structuring, and (ii) its supervision and administration after project approval by the Board of Executive Directors and subsequent IFC investment commitment. This criterion is rated on a four-point scale: excellent, satisfactory, partly unsatisfactory, unsatisfactory). x Advisory Services » Development effectiveness is a synthesis of the following five dimensions: (i) strategic relevance, (ii) output achievement, (iii) outcome achievement, (iv) impact achievement, and (v) efficiency. This criterion is rated on a six- point scale: highly successful, successful, mostly successful, mostly unsuc- cessful, unsuccessful, highly unsuccessful. » IFC role and contribution (starting fiscal year [FY]15) assesses the extent to which IFC added value or made a special contribution to the advisory services project. This criterion is rated on a four-point scale: excellent, satisfactory, partly unsatisfactory, unsatisfactory. » IFC overall work quality assesses the extent to which services provided ensured quality at entry and supported effective implementation through appropriate supervision and execution toward the achievement of devel- opment objectives. The Independent Evaluation Group assesses IFC overall work quality by assessing and rating two dimensions: (i) project preparation and design and (ii) project implementation. This criterion is rated on a four- point scale: excellent, satisfactory, partly unsatisfactory, unsatisfactory. Multilateral Investment Guarantee Agency Ratings » Development outcome is the bottom-line assessment of the project’s results on the ground, as measured across four indicators: project business performance, economic sustainability, environmental and social effects, and Independent Evaluation Group World Bank Group    xi foreign investment effects (starting in 2019) or contribution to private sector development (before 2019). Since FY20, developement has been rated on a six-point scale: highly successful, successful, mostly successful, mostly un- successful, unsuccessful, highly unsuccessful. Before FY20, a four-point scale was used: excellent, satisfactory, partly unsatisfactory, unsatisfactory. » MIGA’s role and contribution assesses the benefits and value added that MIGA, as a development institution and member of the Bank Group, brings to the client, the project, or the political risk insurance industry. It is rated on a four-point scale: excellent, satisfactory, partly unsatisfactory, unsatisfactory. » MIGA’s work quality addresses two aspects: (i) the quality of MIGA’s due dil- igence and underwriting processes, including of risk assessment and mitiga- tion, and (ii) the quality of MIGA’s monitoring after the issuance of the MIGA guarantee. This criterion is rated on a four-point scale: excellent, satisfactory, partly unsatisfactory, unsatisfactory. Results and Performance of the World Bank Group 2022  Glossary of World Bank Group Ratings xii Overview This year’s Results and Performance of the World Bank Group (RAP), like pre- vious RAPs by the Independent Evaluation Group (IEG), updates the analysis of project portfolio performance. As a new contribution, this 12th edition of the RAP analyzes outcomes and performance at the country program level. The country is the primary unit of engagement of the Bank Group, and this RAP presents the effectiveness of the Bank Group’s support at the country program level. It covers World Bank lending and advisory services and an- alytics (ASA) and International Finance Corporation (IFC) investment and advisory services and Multilateral Investment Guarantee Agency (MIGA) guarantee operations over a 10-year period (for World Bank, fiscal year [FY]13–22; for IFC advisory services and MIGA guarantee projects, FY12–22; for IFC investment services, calendar years 2012–22). The Bank Group revised its country engagement guidance in July 2021 to address weaknesses in its approach to country programs, including in the results frameworks of its Country Partnership Frameworks (CPFs). This revised guidance did not yet apply to the country programs analyzed in this RAP. Instead, this RAP establishes a baseline against which the country program performance resulting from the new guidance can be assessed in the future. Building on RAP 2020 and RAP 2021, this RAP updates the project Independent Evaluation Group World Bank Group    xiii ratings trends since then and carries forward the analysis of outcome orien- tation at the country level and of the line of sight from Bank Group support to high-level outcomes (HLOs; defined by the Bank Group as a sustained improvement in the well-being of the poorest and most vulnerable people). Project Portfolio Performance The analysis of the project portfolio performance draws on IEG’s project data- bases of Implementation Completion and Results Report Reviews and Project Performance Assessment Reports (for World Bank), IEG’s Evaluation Notes of IFC’s Expanded Project Supervision Reports and Project Completion Reports and IEG’s Project Evaluation Summaries (for the International Finance Cor- poration), and IEG’s Validation Notes of MIGA Project Evaluation Reports and IEG’s Project Evaluation Reports (for the Multilateral Investment Guarantee Agency). World Bank. At the time of RAP 2021, 88 percent of projects that closed in FY20 had an outcome rating of moderately satisfactory or higher. However, additional projects that closed in FY20 have been evaluated since the publi- cation of RAP 2021, resulting in a downward revision of the share to 84 per- cent. For FY21, the percentage of projects rated moderately satisfactory or higher rose marginally, to 85 percent. Among projects rated moderately satisfactory or higher, the proportion of projects rated satisfactory or highly satisfactory increased from 54 percent in FY20 to 58 percent in FY21 (imply- ing a decline in the share of projects rated moderately satisfactory). So far, no decline is observed in the outcome ratings for projects that closed during the COVID-19 pandemic (during FY20 and FY21). For fragile and conflict-af- fected situations (FCS), the percentage of projects rated moderately satisfac- tory or higher in FY21 was 74 percent, compared with 82 percent in FY20. The Bank performance rating and its constituent ratings for quality at entry and quality of supervision have maintained their upward trend. Between FY20 and FY21, Bank performance continued to be strong—the percent- Results and Performance of the World Bank Group 2022  Overview age of projects rated moderately satisfactory or higher increased from 86 to 91 percent (80 percent to 85 percent for quality at entry and 86 percent to 92 percent for quality of supervision). Among the projects rated moder- ately satisfactory or higher, the proportion of projects rated satisfactory or highly satisfactory increased from 41 percent to 45 percent. Furthermore, monitoring and evaluation (M&E) quality—an important dimension of Bank performance—registered clear improvement. Although 57 percent of projects in FY20 were rated high or substantial on M&E quality, 64 percent were rated high or substantial in FY21. M&E ratings for projects in FCS had a similar increase, from 52 percent in FY20 to 57 percent in FY21. However, the Bank performance rating in FCS in FY21 was much lower than the Bank performance rating for all countries in that fiscal year (81 percent moderate- ly satisfactory or higher in FCS, compared with 91 percent for all countries) and, comparing FY21 with FY20, the percentage of projects rated moderately satisfactory or higher declined marginally, from 82 percent to 81 percent. xiv In addition to reporting on development outcomes, this year’s RAP presents for the first time IFC and MIGA performance on project dimensions such as work quality, role and contribution, or additionality, and for IFC’s investment projects, investment outcome ratings at the project level. The purpose is to assess the performance of IFC and MIGA projects in meeting other institu- tional objectives, the relationships among these dimensions, and the devel- opment outcomes of IFC and MIGA projects. International Finance Corporation investments. Development outcome rat- ings of recently evaluated IFC investment projects have improved. Fifty-two percent of investment projects evaluated between calendar years 2019 and 2021 were rated on average mostly successful or higher, compared with 48 percent between 2018 and 2020. Using IFC’s new regional classification, projects in Central Asia and Türkiye and in South Asia outperformed projects in other regions, especially projects in Africa, whose development outcome success ratings fell from 57 percent in 2012–14 to 31 percent in 2019–21. The low development outcome rating of projects in Africa constrains IFC’s overall development outcome results in International Development Association (IDA) countries. Projects in non-IDA Africa countries have better development outcome ratings. Development outcome ratings of projects in FCS also weakened between 2019 and 2021. Low development outcome ratings of projects in FCS can be observed across all industries, by size of investment commitment and type of IFC investment instrument, and across the four indicators of development outcome. IFC’s Independent Evaluation Group World Bank Group    xv additionality ratings in FCS have also declined. IDA countries and FCS are both IFC corporate priorities. Overall, IFC’s investment outcome success ratings have remained high compared with the ratings on development outcomes, suggesting that even projects with less successful development outcomes ratings still met IFC’s investment returns benchmarks. However, when investment projects per- form poorly in delivering development outcomes, IFC’s bottom line also suffers. IFC’s role and contribution or additionality ratings remain stable and closely associated with development outcome ratings. IFC considers addi- tionality essential to achieving development impact, whether in the form of delivering positive outcomes to stakeholders or helping create or develop markets. In 2019–21, the percentage of projects rated satisfactory or higher for IFC’s additionality reached 61 percent. Most of the projects with high ad- ditionality ratings benefited from a combination of financial and nonfinan- cial additionality. IFC’s financial additionality is delivered by way of financial structuring of the investment, while nonfinancial additionality took the form of improved business environment, corporate governance, and environmen- tal and social standards. IFC investment projects delivered along with IFC advisory services, either sequentially or simultaneously, have better development outcome ratings than stand-alone IFC investment projects. Fifty-five percent of combined investment and advisory services projects have better development outcome ratings, compared with 49 percent for development outcome ratings of IFC investment projects only. However, investment outcome ratings in projects with combined IFC investment and advisory services support are lower, implying that offering these two types of support in a project needs careful consideration and greater selectivity, particularly for advisory services proj- ects that are paired with only equity investment from IFC. International Finance Corporation advisory services. Development effec- tiveness ratings of evaluated IFC advisory services projects have steadily Results and Performance of the World Bank Group 2022  Overview improved and stabilized after dropping to their lowest level in 2015–17. Six- ty-two percent of projects evaluated in 2019–21 were rated mostly successful or higher, compared with 38 percent in 2015–17. Performance of IFC advisory services projects in IDA countries was better than projects in non-IDA countries, and development effectiveness ratings of projects in FCS continue to improve, although they remain lower than ratings for projects in non-FCS. As with IFC investment projects, IFC work quality matters in improving the development effectiveness of advisory services proj- ects. Some of the changes made in the advisory services business line after the joint IEG and IFC work quality study in 2017 seem to be making a difference, especially in stronger governance and increased emphasis on lesson learning. Work quality and development effectiveness ratings are closely associated. Forty-three percent of projects rated satisfactory or higher on IFC work quality are also rated highly on development effectiveness. No advisory services proj- ects were rated highly successful on development effectiveness if IFC work xvi quality was poor, and vice versa. Projects with high ratings on IFC role and contribution also have high rat- ings on development effectiveness, and the two indicators are closely associ- ated. More than half of advisory services projects rated satisfactory or higher on IFC role and contribution were also rated highly on development effec- tiveness. The opposite is also true—if IFC role and contribution is rated low, the chance of getting a high development effectiveness rating is slim (only 1 percent likelihood). Multilateral Investment Guarantee Agency. Development outcome ratings of MIGA projects have improved further, with the share rated satisfactory or better averaging 70 percent of evaluated projects during the six-year period of 2016–21. With the addition of two self-evaluated and validated projects in 2021, the development outcome success rate improved substantially from 66 percent in 2015–20, and it is now one percentage point higher than MIGA’s high success rate in 2013–18. Development outcome ratings of projects in FCS remained the same, but ratings of projects in IDA countries were slightly lower than projects in FCS, non-IDA countries, and non- FCS. MIGA’s work quality ratings are also relatively stable in the six-year periods 2015–20 and 2016–21, with 57 percent and 55 percent of projects rated satisfactory or better, respectively. Work quality also matters in improving MIGA’s development results. Forty-six percent of projects rated highly in MIGA’s work quality also have high development outcome ratings. What MIGA brings to the investment also matters in ensuring positive Independent Evaluation Group World Bank Group    xvii development outcomes. MIGA’s role and contribution rating reached an all-time high of 88 percent of the 66 self-evaluated and validated projects in 2016–21 rated satisfactory or better (with the caveat that the 2021 MIGA program is ongoing, with only 2 out of 8 projects rated so far). MIGA’s key contributions were in political risk mitigation, enabling provision of long- term financing (and, in some cases, facilitating local currency financing when not available from domestic or international financing sources), mobilizing finance, and increasing foreign direct investment flows. MIGA has also contributed positively to improving the environmental and social aspects of 9 out of 10 recently evaluated projects. The likelihood of getting a high development outcome rating is 66 percent when the rating for MIGA’s role and contribution is also high. Both work quality and role and contribution are aspects within MIGA’s control. Effectiveness of World Bank Group Support at the Country Program Level The primary data sources for the analysis at the country program level are IEG’s 152 Completion and Learning Review (CLR) Reviews and 9 Country Program Evaluations completed during FY13–22 (with a cut-off date of March 7, 2022). A stratified random sample of 50 countries—representative of the 108 countries covered by the FY13–22 CLR Reviews—was chosen for in-depth qualitative analysis. There is an important caveat to this analysis. IEG’s evaluation of outcome orientation at the country level pointed out that the Bank Group’s CLRs, which IEG’s CLR Reviews validate and rate, provide a partial picture of country-level development outcomes because of their “overemphasis on those results that can be [quantitatively] measured and on results from lending projects” (World Bank 2020b, xiii). Furthermore, the “CLR rarely captures complementarities across instruments or institutions and so is not able to establish whether the Bank Group’s contribution to country outcomes amounts to more than the sum of its parts” (World Bank 2020b, xiii). Development outcome ratings at the country program level paint a favorable Results and Performance of the World Bank Group 2022  Overview picture of the Bank Group’s effectiveness. Development outcome ratings have continued to improve since FY14. Among CLR Reviews covered in FY19, FY20, and FY21, all country programs were rated moderately satisfactory or higher, with the caveat that these ratings are drawn from a small number of CLR Reviews (21 CLR Reviews that contained ratings). Regarding COVID-19, the analysis finds that the prepandemic CLR Review ratings are a good predictor of project performance during the COVID-19 period. For FCS, the trend is not reported in this RAP given the very small number of CLR Re- views available for each year. World Bank Group performance ratings have also continued to improve, though more gradually, and they remain below the development outcome rating. Seventeen of the 21 CLR Reviews, which contained ratings and were covered in FY19, FY20, and FY21, rated World Bank Group performance as good or higher. As noted in the previous paragraph, the FCS trend is not reported in this RAP, given the very small number of CLR Reviews available xviii for each year. IFC and MIGA contributed 23 percent of the 780 objectives in the 65 CLR Reviews assessed. The share of objectives with IFC and MIGA contribution in recent CLR Reviews increased (from 19 percent to 31 percent) because more attention was paid to the integration of IFC and MIGA activities in the CPFs. Enabling a better business environment and access to finance accounted for half of the CPF objectives with IFC and MIGA contributions. Energy and infrastructure accounted for another quarter of the objectives. IFC uses both investments and advisory services to support the objectives, with business environment objectives supported mainly by advisory services. However, IFC activities that contributed to the CPF objectives represented only a portion of IFC’s active portfolio in a country. In all but one of the objectives, MIGA’s contribution was accompanied by IFC investment or advisory services. MIGA’s business development agreement with IFC emphasizes leveraging IFC’s more extensive global footprint, ensuring collaboration. The objectives with combined IFC and MIGA contributions in energy, finance, and agri- culture performed well: 61 percent to 72 percent of rated objectives were assessed as achieved or mostly achieved, compared with 37 percent for busi- ness environment. The combined IFC and MIGA contributions for all CPF objectives was assessed as 75 percent achieved or mostly achieved, compared with the 55 percent achieved or mostly achieved rating for all CPF objectives with only IFC or MIGA contributions. The overall favorable trend in the Bank Group’s effectiveness at the country Independent Evaluation Group World Bank Group    xix program level is tempered by the following findings: » When individual CPF objectives were considered over the 10-year period (FY13–22), nearly half were rated partially achieved or not achieved (based on 113 CLR Reviews for which this information was available). » There were issues with country program relevance. These included lack of selectivity (such as too many CPF objectives or CPF objectives that were too broad); adaptiveness (such as insufficient preparedness to respond to chang- es in country conditions, government commitment, or Bank Group priorities); and realism in programs and projects (such as operations that overestimated implementation capacity and underestimated political economy challenges). Furthermore, the CPFs and their results frameworks relied overwhelmingly on the World Bank lending portfolio, and they insufficiently integrated and leveraged ASA and the support provided by IFC and MIGA. » The One Bank Group approach, wherein support from the World Bank, IFC, and MIGA complement each other, remained a work in progress. There has been a lack of attention in CLRs and CLR Reviews to a discussion of the implementation of the Bank Group’s Mobilizing Finance for Development agenda (formerly Maximizing Finance for Development) and the Cascade approach. With greater emphasis by both IFC and MIGA on the need for col- laboration in their corporate strategies, the lack of substantive monitoring of the Mobilizing Finance for Development agenda and the Cascade approach is a missed opportunity for learning. » A historical lack of ASA monitoring raises questions about the Bank Group’s ambitions to be a “knowledge bank” and its ability to strategical- ly use ASA to improve country-level impact. Analysis of the latest CLR Reviews for a random sample of 50 countries indicated that these CLR Reviews reported, on average, on only about one-third of the ASA pro- gram in terms of use or influence. » The CPFs sometimes did not adapt sufficiently and quickly enough to changes in context, such as when government commitment or Bank Group priorities changed, implementation capacity was weaker than expected, or planned lending failed to materialize, including because of dropped or can- Results and Performance of the World Bank Group 2022  Overview celed projects. Even when country programs were adapted to reflect chang- ing country circumstances during CPF implementation, the corresponding Performance and Learning Reviews or results frameworks were not neces- sarily modified enough to fully reflect the changes. However, the reasons for adaptation are important, and potential trade-offs between adaptation and shifting goalposts need to be recognized and managed. » The Bank Group was good at identifying risks, especially macroeconomic risks and risks associated with external shocks. It did less well at identify- ing implementation capacity and political economy risks. Overall, the Bank Group fared less well on risk mitigation than on risk identification. For residual risks and risks that could not be fully mitigated (such as political upheaval risks), the Bank Group lacked procedures or guidance to anticipate possible risk scenarios and propose appropriate program adjustments when those scenarios materialized. Consequently, the Bank Group’s response to xx residual risk was ad hoc and not adequately informed by how the Bank Group had responded across countries facing similar circumstances. Line of Sight The Bank Group defines the line of sight as “a clear path connecting an activity with its ultimate desired outcome” (World Bank Group 2021c, iv). Within the Bank Group context, the line of sight involves two stages: (i) from Bank Group support to CPF objectives/development outcome (the extent of achievement of CPF objectives determines the development outcome rating), and (ii) from CPF objectives/development outcome to HLOs. The line of sight requires two conditions to be met at both stages: relevance (a necessary con- dition) and contribution (a sufficient condition). In the first stage, the RAP found that Bank Group support was not always fully relevant to CPF objectives/development outcome due to a lack of se- lectivity, adaptiveness, or realism. Contribution also had some weaknesses in this first stage in that lending often fell short (that is, planned lending did not materialize or was dropped or canceled, or disbursements were delayed), and adequate restructuring of the country program and the associ- ated results framework was not undertaken at the Performance and Learning Review stage. Stated CPF objectives/development outcome then tended not to be achieved. Independent Evaluation Group World Bank Group    xxi The RAP found that both relevance and contribution were less evident in the second stage, even though CPF objectives/development outcome generally mapped well to HLOs. Establishing relevance and contribution in the second stage tends to be difficult because extraneous influences increase as one moves toward outcomes at the final stages, and confounding factors from actions originating outside the Bank Group program come into play. Table O.1 summarizes the RAP’s analysis of the line of sight. Table O.1. Line of Sight Analysis First Stage Second Stage From World Bank Group support to CPF objectives/ From CPF objective/develop� Criterion development outcome ment outcomes to HLOs Relevance Weaknesses in relevance Relevance less evident » There is a lack of selectivity, » CPF objectives/development adaptiveness, and realism outcome generally map well to in country programs, such HLOs. as insufficient attention to » However, issues remain with implementation capacity and the measurement of relevance. political risks. Determining Bank Group rel- » There are weaknesses in CPFs evance is complicated by the and their results frameworks need to also account for the that insufficiently integrate actions of other development and leverage ASA and IFC actors. and MIGA support. Contribution Weaknesses in contribution Contribution less evident » Lending often falls short— » Establishing contributions planned lending does not has tended to be difficult— materialize, projects are extraneous influences dropped or canceled, or dis- increase as one moves toward bursements are slower than outcomes at the final stages, Results and Performance of the World Bank Group 2022  Overview planned. and confounding factors from » Adequate restructuring is actions originating outside the not undertaken at the PLR Bank Group program come into stage—CPF objectives/devel- play. opment outcome then tend not to be achieved. Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; CPF = Country Partnership Framework; IFC = International Finance Corporation; HLO = high-level outcome; MIGA = Multilateral Investment Guarantee Agency; PLR = Performance and Learning Review. Paying More Attention to the Performance of the World Bank Group This RAP supports an increased focus on the Bank Group’s performance at both the project and country levels. This is especially important for the World Bank’s project portfolio in FCS, where the Bank performance rating (81 percent in FY21) trails the Bank performance rating for the entire portfo- xxii lio (91 percent in FY21). World Bank Group performance, unlike development outcome, is directly within the Bank Group’s control; therefore, a consistent- ly higher rating of Bank performance (for World Bank projects) and World Bank Group performance (at the country level) therefore would not be an unreasonable target. This RAP highlights several factors that can enhance the Bank Group’s performance, including greater selectivity, adaptiveness, and realism in country programs and CPFs and their results frameworks that integrate and leverage ASA and IFC and MIGA support. World Bank Advisory Services and Analytics This RAP undertook a qualitative analysis of the use or influence of ASA based on the CLR Reviews for the random sample of 50 countries. More than 80 per- cent of cases exhibited a good match between ASA topics and the topics cov- ered by government policies and programs, and between ASA topics and topics of CPF objectives/development outcome and World Bank operations. However, when reviewing the CLR Reviews for higher levels of ASA influence (such as ASA influence on policy dialogue, uptake in government programs and poli- cies, and uptake in CPF objectives and World Bank operations), less than half of the CLR Reviews reported on these higher levels of ASA influence. Of those that did, ASA were found to be influential in most cases. Furthermore, this RAP finds a significant lack of reporting on client and stakeholder ownership and engagement and on dissemination. There were Independent Evaluation Group World Bank Group    xxiii also a few instances of less positive client and stakeholder ownership and engagement. These process issues are important to ensure that the knowl- edge generated by ASA actually feeds into decision-making. Without such uptake of ASA in decision-making, the World Bank risks being a “report bank,” not yet a “knowledge bank.” Becoming a “knowledge bank” calls for the use of knowledge, which stakeholder engagement and dissemination can foster. Beyond this, knowledge translation or knowledge brokering can help country clients determine which knowledge is most relevant to the decisions they face and how best to tailor global knowledge to local cir- cumstances (White 2019). Future Directions Although the enhancements now included in the International Bank for Reconstruction and Development, IFC, MIGA, and IDA country engagement guidance are expected to help address some of the issues identified in this RAP (World Bank Group 2021a), we offer three future directions for the Bank Group’s consideration. Enhancing the Effectiveness of World Bank Group Support at the Country Program Level » Improve the selectivity and framing of CPF objectives and the realism of Bank Group country programs, especially regarding implementation capacity and political challenges. » Prioritize implementation of the One Bank Group approach (including the Mobilizing Finance for Development agenda and the Cascade approach), a long-standing aim of the Bank Group. » Ensure that the Bank Group adapts and keeps pace with changing circum- stances, such as shifts in government commitment or changes in Bank Group Results and Performance of the World Bank Group 2022  Overview priorities, while being mindful not to lower the objectives to make up for the lack of progress. One option would be to consider the application, at the country level, of the M&E system currently applied to restructured projects at the World Bank. » Report in a timely way, in relevant documents (particularly in the Perfor- mance and Learning Review), any adaptations made to the country program and update the results frameworks accordingly. » Ensure that the Bank Group’s country program benefits from all elements of Bank Group support collectively and that the sum of the parts of Bank Group support is monitored. Strengthening Risk Identification and Mitigation » Maintain the Bank Group’s current good performance on identifying macro- economic risk and risks associated with external shocks while improving the xxiv identification of risks associated with implementation capacity and political economy. » Identify up front the possible key risk scenarios and outline the course of ac- tion to address those scenarios so that timely action can be taken, depending on which scenario unfolds. » Expand and update the current country engagement guidance to include possible key risk scenarios and the responses to each to facilitate better risk mitigation. Monitoring Advisory Services and Analytics Use or Influence » Systematically monitor and evaluate whether ASA are achieving their intended influence. » Consider introducing self-evaluation of analytical work across all Bank Group institutions. Independent Evaluation Group World Bank Group    xxv Management Comments Management of the World Bank Group welcomes the Independent Evalu- ation Group (IEG) report Results and Performance of the World Bank Group 2022 (RAP 2022) and thanks the IEG team for having taken on board several comments previously provided. Management is pleased with IEG’s overall positive findings on performance at the project and country level. The re- port’s main findings, analysis, and lessons provide a source of learning that informs strategic decision-making for Bank Group management. World Bank Management Comments Overall Management welcomes the report’s overall positive findings and is pleased that satisfactory project outcome ratings are at a historical high. Management is also satisfied with the stable and positive performance on quality at entry and quality of supervision at the project level. Management also welcomes the increased effectiveness of Bank Group support at the country level, with Results and Performance of the World Bank Group 2022  Overview more than 70 percent of country programs rated moderately satisfactory or higher, accompanied by a steady increase in inclusion of International Finance Corporation (IFC) and Multilateral Investment Guarantee Agency (MIGA) activities throughout the full country engagement cycle. While the report has a few methodological shortcomings stemming from its innovative nature, management broadly concurs with the areas for future direction, which had already been identified in other IEG products, most notably in the World Bank Group Outcome Orientation at the Country Level evaluation (World Bank 2020b). The valuable lessons offered by the report are well aligned with management’s outcome orientation agenda and therefore are embedded in current efforts. As some key initiatives in this regard have only recently been launched, some of the findings are not yet reflected in the cohort of products covered by the report. Management notes IEG’s intention to use this report as the baseline against which future country programs will be assessed. xxvi While reassured by the positive performance, management notes the ex- traordinarily different global development context in recent years. The Results and Performance of the World Bank Group (RAP) analysis covers FY13– 22. The past two years offer a very early view of the COVID-19 response, as well as the current context of compounding crises (post–COVID-19 recovery; the impact of war in Ukraine; inflation; food and energy insecurity; reces- sion; the increasing prevalence of fragility, conflict, and violence situations; and climate events). In the RAP’s preliminary assessment, COVID-19 does not seem to have an immediate implication for the reviewed results. Nev- ertheless, management remains vigilant of how the complex global context impacts all World Bank’s clients in diverse ways and to varying degrees, and what the consequences for future World Bank engagements would be. Man- aging risks is paramount in this volatile environment and the report offers insights to avoid reversals of progress made over the past two decades. Country Engagement Guidance Management notes IEG’s assertion that the country engagement guidance revised in 2021 is intended to address most of the identified shortcomings and finds the lessons of the report useful for effective implementation of this guidance. Management adjusted the country engagement guidance to offer teams a more flexible approach to articulate and monitor their contribution to Independent Evaluation Group World Bank Group    xxvii selected high-level outcomes (HLOs), including in the results framework. The changes are providing a more structured approach for setting HLOs anchored in the country’s development priorities and articulating the Bank Group’s col- lective contributions to those outcomes, including through indirect pathways and through a coordinated One Bank Group approach. The revised narrative describes more clearly the role and complementarities of investments, poli- cies, and institutions, and contributions from the public and private sectors. The time horizon of HLOs extends beyond the Country Partnership Framework (CPF) cycle as long as it remains relevant as high-level development goal that the Bank Group seeks to contribute to, on the basis of country priorities and demands for Bank Group engagement, Systematic Country Diagnostic find- ings, and other strategic considerations. At the end of the CPF cycle, the Com- pletion and Learning Review (CLR) contextualizes and puts in perspective the results achieved under the CPF objectives—including results attributable to projects and interventions predating the current CPF cycle—and prog- ress observed toward the selected HLOs. These changes, accompanied by the Outcome Orientation Roadmap, are ambitious and therefore require consistent implementation and handholding. One significant challenge is establishing a credible and measurable line of sight to HLOs, and management finds the RAP’s conceptual framework for this useful. HLOs are a new feature in the revised guidance, as the report points out, and the analysis of past CPFs based on HLOs may not give an appropriate depiction about CPF achievements and performance. Previously, CPFs used focus areas and references to the twin goals, Sustainable Development Goals, and individual country development goals. The revised guidance includes explicit reference to the contribution of CPF objectives to HLOs whereby progress toward HLOs is measured by selected indicators over multiple CPF cycles. CPF results frameworks gain increased relevance, as they provide an empirical basis to substantiate the Bank Group’s contributions to HLOs. This will better align the country program instruments with the Bank Group’s business model of helping client countries achieve HLOs over time, while still maintaining a strong degree of accountability within the CPF cycle. Management will purposefully use the unpacked concept of line of sight pro- Results and Performance of the World Bank Group 2022  Overview posed by IEG to continue refining the way it defines HLOs. Future Directions Management values the proposed future directions as an opportunity for further fine-tuning recent reforms, particularly during implementation. The paragraphs in the following sections offer both some clarifications and steps forward. Enhancing the Effectiveness of World Bank Group Support at the Country Program Level Management concurs with the report’s emphasis on the need to further advance adaptive management throughout the country engagement cycle but would like to reiterate that flexibility occurs at multiple levels. In its 2021 outcome orientation paper, management explained that “…in the way xxviii the WBG [Bank] Group aims for outcomes, adaptation and course corrections can be strategic or tactical. Strategic adaptations involve higher level chang- es, for example adding, removing, or adjusting a development objective in response to new knowledge or emerging constraints, priorities, or opportuni- ties. […] Tactical adaptations focus on course corrections to navigate imple- mentation challenges and ensure the achievement of existing objectives. […] These decisions are typically made at the project level and reflected in the ISRs [Implementation Status and Results Report] and ICRs [Implementa- tion Completion and Results Reports]” (World Bank Group 2021c, 8–19). The revised country engagement guidance encourages teams to be more proactive to reflect changes in the results matrix, with the explicit re- quirement to explain the impact of such changes on the objectives and the HLOs. Management is working with country teams to create the necessary training, incentives, and space for this to happen effectively. A strengthened CPF Academy (revamped in 2022) is offering dedicated guidance on this matter. Tactical adaptations are organic to the way the World Bank aims for outcomes, as reflected for example by the large number of operations that were adjusted and repurposed during the COVID-19 response. Management shares the view that clients are often better served through a ho- listic Bank Group approach but cautions against overgeneralization. Given the many nuances, the treatment of experience of One Bank Group in the report seems unnuanced. The three Bank Group institutions develop country-specific collaboration and programming approaches depending on country needs, Independent Evaluation Group World Bank Group    xxix sectoral landscape, institutional and other risks, and timing of the respec- tive interventions. Not all CPF objectives lend themselves for collaboration. Within the Bank Group, the World Bank’s comparative advantage lies in the ability of do “systemwide” engagement. Notwithstanding the need for selectivity based on comparative advantages, management has emphasized through the revised country engagement guidance the importance of joint work, and it believes that the introduction of HLOs helps Bank Group insti- tutions identify common ground to deliver longer-term results by combining and sequencing multiple products. Strengthening Risk Identification and Mitigation Management concurs with the importance of more explicit risk management in country engagements but does not find it feasible to include alternative scenarios in CPFs. The RAP makes the broad assertion that the Bank Group was good at identifying (some) risks but fared less well on risk mitigation. Management however believes that the best measure of effective suggesting that both risk identification and mitigation are also improving. The report concludes that “the response to crises as known risks materialize is often ad- hoc and inconsistent across countries and over time” (74). The application of standardized approaches is not desirable as the World Bank’s responses to crises are typically country and situation-specific and regional or country management is best placed to assess the level and intensity of engagement in an unstable situation. The RAP suggests the inclusion of scenarios in CPFs in advance so that corrective action can be taken in a timely manner, depending on which scenario unfolds. Management believes it is not realis- tic to expect a publicly available CPF to outline some of the major potential risk scenarios—such as, for example, political upheaval—as doing so could be counterproductive. While CPFs cannot be expected to engage in scenario planning on all political risk factors, World Bank management agrees that Results and Performance of the World Bank Group 2022  Overview there needs to be systematic discussion about different situations during the CPF period. This discussion must happen organically throughout the country engagement cycle, as those situations directly influence the client’s ability to achieve the desired HLOs. Finally, as evidenced through the current context of compounding crises, external risks can never be fully mitigated. Monitoring the Use and Influence of Advisory Services and Analytics Management shares the appreciation of the importance of advisory ser- vices and analytics (ASAs) to deliver outcomes—as highlighted in the Strategic Framework for Knowledge—as well as the need for better mea- surement of their effects, although the report shows an incomplete picture of ASA performance. While the implementation of the Strategic Framework for Knowledge will certainly be influenced by the RAP 2022 conclusions, management has reservations on the way the report oversimplifies the xxx varying objectives and use of ASAs. The report does not distinguish be- tween various types of ASAs and how and when they influence develop- ment outcomes at the country level: (i) a significant number of ASAs are intentionally designed to inform narrow aspects of operational design and so are never intended to have a larger impact at the country level, and their value is reflected through project-level results; (ii) some ASAs conducted during a CPF cycle, such as advocacy work, could inform and influence activ- ities and operations in future engagement cycles; (iii) other core ASAs in the country engagement cycle such as Country Climate and Development Re- ports do not seem to have been recognized in the process; and (iv) global, re- gional, and thematic ASAs that can have significant influence across several countries are not considered. Recognizing this variety, management is mak- ing strides to better align ASAs with the outcome orientation agenda. While coverage is not comprehensive, it is encouraging to see that the impact is high for that portion of the portfolio which has been assessed. The revised country engagement guidance also addresses this issue through (i) increased emphasis on capturing indirect pathways in the results frameworks, which will influence future country engagement, and (ii) encouraging teams to include the contribution of ASAs in discussing development outcomes in CLRs. Dedicated activities in the Outcome Orientation Roadmap will provide guidance to operational teams on how to capture the contribution of indirect pathways—of which ASA is an important part—to HLOs. International Finance Corporation Management Independent Evaluation Group World Bank Group    xxxi Comments IFC management welcomes IEG’s flagship report Results and Performance of the World Bank Group 2022. This year, the RAP goes beyond updating project-level portfolio performance trends to reporting on the Bank Group’s country-level outcomes and performance. We welcome IEG’s review and the establishment of the baseline for future reviews in this regard. In addition, the RAP 2022 presents helpful first-time analysis regarding IFC’s perfor- mance on project dimensions, such as work quality, role and contribution, additionality, and investment outcome ratings. IFC management acknowl- edges the IEG analysis on projects at the extreme end of the spectrum— highly successful and highly unsuccessful—which has great potential for future learning. Management is pleased to see the continued positive trends in IFC’s develop- ment results, particularly in the context of compounding crises and uncertain- ty. As detailed in the RAP 2021 report (World Bank 2021b), IFC management has implemented a deliberate, multifaceted effort over the years to strengthen the institutional focus on development results. This included, among others, establishing the Economics and Private Sector Development Vice Presidential Unit to strengthen country, market and country assessments and launching the Anticipated Impact Measurement and Monitoring (AIMM) framework to improve project assessment, selection, and design as well as providing addi- tional resources for staff to focus on project evaluations and placing greater emphasis on work quality and candor for operational staff. This approach is now firmly embedded in IFC’s investment and advisory operations, and we are pleased to see these efforts bearing fruit. Overall, the development out- come of evaluated IFC investments has improved to 52 percent in 2019–21 from 47 percent in 2017–19 (reported in RAP 2021) and a sustained recov- ery from the lowest level of 41 percent in 2016–18. While the full effects of the COVID-19 pandemic on portfolio performance have yet to materialize, Results and Performance of the World Bank Group 2022  Overview management is pleased to see increasing IFC development outcome ratings and is optimistic that the reversal in declining results will be sustained. Similarly, with respect to IFC advisory projects’ development effectiveness ratings, 62 percent of projects evaluated in 2019–21 were rated mostly successful or higher, a continuous improvement from 52 percent in 2017–19 (reported in RAP 2021). As the report recognizes, these development effec- tiveness ratings demonstrate a significant continuous recovery from the lowest level at 38 percent in 2015–17. While noting the marked improvement in development effectiveness ratings for advisory projects in International Development Association (IDA) / fragile and conflict-affected situations (FCS) contexts, management acknowledges, with concern, the poor outcome ratings for IFC investments in Africa, as well as in FCS markets more broadly, and commits to undertake a review of the key drivers. IFC 3.0, launched in 2017, enshrined IFC’s commitment to supporting private sector development in the most challenging markets, with Africa being designated as one of IFC’s xxxii three strategic focus regions. IFC also set ambitious goals through both the capital package and as part of the Bank Group fragility, conflict, and violence strategy to accelerate its program in fragile and conflict-affected markets, where private sector activity is constrained by both financial and nonfinancial risks. Advisory services play a critical role in IFC’s programs in both project preparation and capacity building in these regions, and man- agement is pleased to note the stark improvement in development effec- tiveness ratings for advisory projects in IDA/FCS markets (from 22 percent mostly successful or higher in 2015–17 to 56 percent in 2019–21). However, development outcome ratings for investment projects in Africa and IDA/FCS markets continue to lag behind the IFC average, particularly in the areas of work quality and additionality. Given the centrality of Africa as well as IDA/ FCS markets to IFC’s mission, management is keen to better understand the underlying drivers of investment outcome ratings in these areas and com- mits to an in-depth, internal review with a view to improving development performance in priority markets. To complement the above, management would also like to highlight im- portant organizational, managerial, and business decisions that have been taken in recent years to scale up both investment and development impact in these Africa and IDA/FCS markets. While we strongly agree that more work needs to be done, we also point to the solid progress that has been made and that may not yet be reflected in the most recent IEG ratings Independent Evaluation Group World Bank Group    xxxiii because of the lag between project approval and when projects are rated. In FY22, IFC invested $9.4 billion across 36 countries in Africa—the largest ever annual commitment for the continent—including $2.6 billion in mo- bilization and $3.0 billion in much needed trade and short-term finance. IFC’s average AIMM score in Africa was 55 and in FCS–low-income coun- tries–IDA17 markets was also 55, higher than the IFC average of 53. IEG’s FY22 evaluation The International Finance Corporation’s and Multilateral Investment Guarantee Agency’s Support for Private Investment in Fragile and Conflict-Affected Situations, Fiscal Years 2010–21 and the associated management response provided further detail on IFC’s efforts launched over the past three years to improve impact and work quality in Africa and across IDA/FCS markets. These include the following: » Adding experienced, senior resources, including locating five Regional directorships in Africa; » Expanding IFC’s overall footprint across Africa, including in FCS markets. IFC has increased the staff in FCS locations by 88 percent (from 89 staff members in FY19 to 167 in FY22) during COVID-19. IFC has also increased the incentives for staff working in FCS. For example, in FY21, 22 percent of corporate awards were given to teams working on FCS, and almost 50 percent (14 of 30) of the staff receiving IFC top 30 individual corporate awards in FY21 were recognized for multiyear efforts in FCS; » Scaling up dedicated platforms such as the Africa Fragility Initiative (AFI) which supports responsible private sector-led growth and job creation across 32 African FCS countries; » Institutionalizing IFC’s systematic approach to Upstream project develop- ment and market creation with special focus on IDA/FCS—The Upstream pipeline in IDA17+FCS markets increased by close to US$ 1.3 billion over FY22 to reach US$ 9.2 billion as of FY22-end » Leveraging blended finance resources, such as IDA Private Sector Window (PSW) to help mitigate financial risks, and deployment of tools such as the Results and Performance of the World Bank Group 2022  Overview FCS & LIC IDA Risk Envelope; » Enhancing tools to address non-financial risks, such as: (i) development of Contextual Risk Framework–a diagnostic framework used to better understand country context, risks, and fragility drivers to inform strategy and operations in FCS markets; (ii) expansion of ESG advisory services; and (iii) development of a dedicated e-learning module for IFC staff on conflict sensitivity; » Development of a Guidance Note on incorporating FCS considerations into Country Private Sector Diagnostics; » Prioritization of private equity and venture capital funds in IDA/FCS markets under IFC’s Equity Strategy; and » Launching a course on “Tools for Investing in FCS and Low-Income Countries (LICs),” targeted at staff working in these markets. xxxiv The aforementioned improvement in development effectiveness ratings for advisory projects in IDA/FCS markets demonstrates the results of sustained effort and managerial focus. With the above initiatives and advisory support in place, IFC is proactively aiming to improve the identification, selection, and delivery of investments in Africa and IDA/FCS to increase investment volume and development impact in these priority markets. IFC also closely collaborates with the World Bank and MIGA and hopes that collectively, our efforts would contribute to improving long-term development outcomes in IDA/FCS countries. Management requests more clarification on the evidence base for the RAP’s characterization of IFC first-loss guarantee facilities as unsuc- cessful. Chapter 3 of RAP 2022 (the International Finance Corporation Investment Projects section) describes IFC’s experience so far in the use of first-loss guarantee facilities with local banks as “challenging and unsuccessful” (20). This assessment is made in the context of Financial Institutions Group projects in Africa that were rated unsuccessful as part of the 2019–21 reporting cycle. However, this finding is based on a sam- ple of facilities that is not representative of IFC’s broader experience with the small and medium enterprises (SMEs) first-loss guarantee risk-shar- ing product. The report leverages five Risk-Sharing Facilities (RSFs) that were rated mostly unsuccessful or lower in the 2019–21 reporting cycle, Independent Evaluation Group World Bank Group    xxxv of which, four were with the same client group and all were in Africa. This results in an undue concentration of ratings, which are more reflective of the performance of a sponsor group than performance of RSF as a product. In contrast, IFC has committed 46 SME RSFs (some under a programmatic ap- proach) since FY15 with a total IFC commitment of $442 million directed to underserved SMEs. These IFC investments have a multiplier effect in terms of onlending. The average ex ante AIMM rating for this asset class commit- ted since 2019 is 59. To further support its summary findings, beyond the aforementioned five cases, the report also references the IEG evaluation of IFC investments in K–12 private schools. However, this evaluation cites four RSFs in the K–12 sector, also all in Africa, that were committed in FY05–08, and it is not clear what other RSFs are included (World Bank 2022a). RSFs are a key tool for IFC to support financial inclusion by allowing IFC and local banks to form a partnership with the goal of expanding the bank’s lending with target market segments. IFC management has recognized the challeng- es with RSF, especially as IFC is targeting more difficult markets. As a result, IFC has reviewed the underlying performance drivers and taken action based on lessons learned. This has included changes to RSF structure, more focused portfolio management, and enhancements of client capacity through a ded- icated and programmatic approach like the Small Loan Guarantee Program. Small Loan Guarantee Program reach, use rates, and development results have shown positive trends in terms of effectiveness, with 15 projects com- mitted and over 4,900 SMEs reached so far. Greater clarity on the evidentiary basis underlying IEG’s assessment would be very helpful to support ongoing learning. Management appreciates different cuts in the analysis of results and is interested in more detail to better understand underlying drivers. For example, management notes that IFC investment projects that are paired with advisory services have higher results than stand-alone IFC invest- ment projects with respect to development outcomes. Fifty-five percent of investment services projects that are paired with advisory services projects have development outcome ratings of mostly successful or higher, com- pared with 49 percent for development outcome ratings of stand-alone IFC Results and Performance of the World Bank Group 2022  Overview investment projects. The RAP 2022 notes that most of the joint investment services and advisory projects with low investment outcome ratings were eq- uity investments and that this partly explains the result. Management would appreciate a more detailed analysis of the causality between joint invest- ment services and advisory services equity investments and weak investment outcome ratings. More granularity would be helpful in this regard, as IFC has worked to address weak outcomes from equity investments since adopting a new approach to such investments in FY19. This approach has included greater specialization among staff, active portfolio management, systematic assessment of and attention to macroeconomic risks among others. Finally, IEG presents analysis of projects rated at both ends of the rating spectrum (highly successful and highly unsuccessful), noting that diversity in project characteristics limits generalizability of results drivers. IFC would like to work with IEG to refine the analysis over time, as it has potential to yield great learning. xxxvi Management requests that future RAPs reflect the external operating envi- ronment more systematically when presenting findings, as some of the past RAPs have done. With the COVID-19 pandemic exacerbating preexisting weaknesses, the private sector in emerging markets has faced unprece- dented crises in recent years, resulting in depressed and changing patterns of demand, reduced access to capital, rising bankruptcies, and the arrival of persistent uncertainty. IFC management thanks IEG for their construc- tive partnership and collaboration in agreeing to postpone evaluation of IFC projects with high COVID-19 impact exposure. However, projects that faced more moderate COVID-19 impacts were evaluated. More broadly, management maintains that factoring in the external context, including systemic exogenous shocks such as the COVID-19 pandemic, can ensure that results and performance trends are viewed in the appropriate context and that narratives are more nuanced. As acknowledged by the IEG team in their response to Bank Group’s draft-stage comments, providing analysis of the impact of the operating environment brought by volatile macro and mar- ket conditions on project results and performance in future RAPs would be indeed helpful. To assist management and the Board in interpreting the report findings, management requests that in future RAPs, IEG clearly explain that the reported Bank Group performance results are subject to a degree of fluctu- Independent Evaluation Group World Bank Group    xxxvii ation until IEG validation reviews are complete. Management understands that Executive Directors requested that the Board review the RAP as early as possible after FY22-end and that the latest available Bank Group results and performance data be used. If this approach is carried forward, it may have the unintended effect of decreasing the number of projects that can be included in the report as not all projects from the recent fiscal year will have been validated by IEG before the release of the RAP. Specifically, at the time of developing these comments, only 51 percent of the FY21 advisory projects sample has been validated, and thus the development effectiveness rating reported in the RAP is based on this partial sample. Furthermore, this rep- resents only 31 percent of the underlying FY21 advisory portfolio, but this serves as the cohort from which lessons are generalized. The lower coverage of FY21 projects implies a stronger bias on FY20 and FY19 evaluations in the latest three-year rolling grouping. Similarly for investments, the calendar year 2021 validation coverage is 33 percent of the underlying portfolio. While we appreciate that IEG wishes to provide up-to-date findings to the Board, management would like to safe- guard the quality and relevance of findings. To support the interpretation of report findings, management requests that in future RAPs, IEG clearly reference the status of validation and explain that the reported Bank Group performance results fluctuate until IEG validation reviews are complete. Finally, management acknowledges the country-level outcomes and appreci- ates the future directions shared for consideration with the Bank Group in chapter 7. Management takes note of the strong Bank Group development outcome and performance ratings at the country program level and acknowl- edges IFCs contribution to CPF objectives to be concentrated in the areas of business environment and access to finance with IFC advisory being the main instrument to contribute to business environment CPF objectives. Results bear out that if World Bank and IFC jointly deliver on objectives, there is a (slightly) better record of achievement, which points to the ben- efits and potential for collaboration. Management will consider the future directions presented by IEG with World Bank and MIGA colleagues as the Bank Group further builds on its country engagement programs. With xxxviii Results and Performance of the World Bank Group 2022  Overview respect to “Monitoring Advisory Services and Analytics Use or Influence” IFC would like to register that regarding advisory services, IFC has a robust process for evaluation, which was developed jointly with IEG and which val- idates self-evaluations. IFC management is happy to share its experiences if this is to be developed for others. Multilateral Investment Guarantee Agency Management Comments MIGA welcomes IEG’s RAP 2022 report and finds it useful and important. MIGA commends IEG for the report’s new contribution to the effectiveness of the Bank Group’s support at the country program level, based on the analysis of Bank Group outcomes and performance. MIGA thanks IEG for the productive engagement during the drafting and finalization of the report. Historically high MIGA development outcome performance. The report presents many useful findings, and MIGA appreciates IEG’s observations. In particular, the report notes the steady increase in the development outcome success rates of MIGA guarantee projects over the past 10 years. The devel- opment outcome success rate for the period under review, FY16–21, reached the MIGA-historic high of 70 percent by the number of projects; though not stated explicitly in the report, RAP 2022 also marked historically high success rates of 86 percent for environmental and social effects (E&S) and 72 percent for foreign investment effects, based on the RAP 2022 database. The historic high development outcome success rate was also driven by the historic high success rate of 88 percent for MIGA’s role and contribution, as discussed in the report. RAP 2022, as well as RAP 2021, recognized MIGA’s solid and steadfast ef- forts for strengthening self-evaluations as a key factor behind the steady increase in development outcome success rates. MIGA has strived to over- come the challenges of collecting evaluative information through various measures, including evaluation missions to nearly all projects selected for evaluations by the self-evaluation teams in the pre–COVID-19 era and intensifying the collection of projects’ E&S performance information. Over the years, MIGA has also progressed in undertaking self-evaluations of all guarantee projects—including canceled guarantees—which IEG then vali- dates. In addition, MIGA notes that the high development outcome perfor- mance of MIGA guarantee projects has been built on the solid foundation of Independent Evaluation Group World Bank Group    xxxix the Results Measurement Systems that have been established in MIGA for more than 10 years, together with MIGA’s increased emphasis on underwrit- ing impactful projects under challenging settings and significantly greater attention given to monitoring, evaluation, and learning. Historically high MIGA’s role and contribution performance. The report finds that MIGA’s role and contributions were fundamental for successful devel- opment outcomes, related to enabling long-term financing, facilitating local currency financing, and improving E&S performance. MIGA’s Financial and Non-Financial Additionality, as outlined in MIGA’s FY21–24 Strategy and Business Outlook, provides a holistic framework for assessing MIGA’s role and contributions: (i) availability of insurance, (ii) increasing tenors, (iii) competitive pricing, (iv) access to funding, (v) lowering borrowing cost, (vi) mobilizing reinsurance capacity, (vii) regulatory capital relief for banks to increase lending headroom, (viii) resolving disputes, (ix) knowledge sharing, and (x) standard setting (for example, E&S, integrity, and corporate gover- nance). Historically high E&S performance. MIGA welcomes the report’s recognition of MIGA’s contributions to building the E&S capacity of clients and improv- ing the E&S effects of guarantee projects. MIGA notes that E&S effects were the highest-rated development outcome indicator during 2016–21, reaching a historically high success rate of 86 percent. MIGA notes that the strong E&S results in 2016–21 have been on account of MIGA’s enhanced E&S mon- itoring and supervision efforts of its guarantee projects. The rapid strides made in MIGA’s E&S monitoring work followed the issuance of MIGA’s Policy on Social and Environmental Sustainability (2007), the update and enhance- ments to the policy as reflected in MIGA’s Policy on Environmental and Social Sustainability (2013), and the development of supportive E&S Review Procedures (2014). MIGA notes the good example cited in the RAP 2022 re- garding MIGA’s involvement in a financial intermediary project where MIGA helped develop E&S policies and procedures at the bank’s subsidiaries that were supported by a MIGA guarantee, with MIGA also assisting in establish- ing the bank’s overall Environmental and Social Management System. Results and Performance of the World Bank Group 2022  Overview MIGA contributions at the country program level and Bank Group collabora- tion in country engagement. MIGA’s country-level engagement is vital for MIGA’s work and development impact. MIGA welcomes the Report’s as- sessment of the Agency’s contributions at the country program level. The report also highlighted the importance of World Bank Group collaboration in the country engagement process, including the quality of the discus- sion on Bank Group internal collaboration in the CLR Reviews. MIGA made significant progress in integrating MIGA teams into the country engage- ment process, including Systematic Country Diagnostic, Country Partner- ship Frameworks, and CLRs. The analysis in RAP was based on the historic country-level assessment before the introduction of the revised Bank Group country engagement guidance adopted in July 2021. The new approach will capture the distinct and joint contributions of the three Bank Group insti- tutions to country outcomes based on the unique mandates and purposes of the Bank Group institutions and their attendant business models. xl 1 | Background and Methodology Highlights This Results and Performance of the World Bank Group updates the annual analysis of project portfolio performance using the Independent Evaluation Group’s database of project evaluations (Implementation Completion and Results Report Reviews and Project Performance Assessment Reports). It focuses on the Bank Group’s effectiveness at the country program level and the use or influence of advisory services and analytics. The new country engagement guidance adopted by the Bank Group in 2021 did not yet apply to the country programs analyzed in this report. Instead, this Results and Performance of the World Bank Group establishes a baseline against which country program performance resulting from the new guidance can be measured in the future. The primary data sources for the country-level analysis are the Independent Evaluation Group’s Completion and Learning Review (CLR) Reviews and Country Program Evaluations. 1  Context This year’s Results and Performance of the World Bank Group (RAP) report updates the project portfolio performance undertaken in previous RAPs (World Bank 2020a, 2021b) while focusing on the country level. It presents an annual review of evidence from Independent Evaluation Group (IEG) evaluations and validation work on the development effectiveness of the Bank Group.1 The 2022 RAP is the 12th in an annual series. The 2022 RAP aligns with the Bank Group’s outcome orientation agenda by focusing on the country level.2 This country-level focus also responds to the interest of members of the Bank Group’s Board of Executive Directors for reporting specifically on country-level performance. Accordingly, this RAP presents a snapshot of the Bank Group’s effectiveness at the country program level, covering lending and advisory services and analytics (ASA). A 10-year period (from fiscal year [FY]13 to FY22) was chosen because this is the first time such country-level analysis is being undertaken. The revised country engagement guidance that the Bank Group adopted in July 2021 did not yet apply to the country programs analyzed in this RAP. Instead, this RAP establishes a baseline against which country program performance re- Results and Performance of the World Bank Group 2022  Chapter 1 sulting from the new guidance can be measured in the future. This RAP also focuses on the performance of the Bank Group with regard to the World Bank Group performance rating (for the country level) and the Bank performance rating (for the project level) as well as a qualitative analysis of performance, and it aims to bring attention to this dimension of development effectiveness. This is because the performance of the Bank Group is fully within the Bank Group’s control and can yield direct insights into what the Bank Group can do to enhance the future effectiveness of its lending and ASA support. Methods and Data Sources The primary data sources for this year’s RAP are IEG’s Completion and Learning Review (CLR) Reviews and Country Program Evaluations (CPEs). All CLR Reviews were examined for the ratings trends analysis. A stratified random sample of 50 countries was chosen for in-depth qualitative analysis 2 from a population of 108 countries with 152 CLR Reviews completed during FY13–22 (with a cut-off date of March 7, 2022).3 IEG reviewed the most recent CLR Reviews for each of these 50 countries (appendix A). All 9 completed CPEs were also examined for the same analysis.4 Furthermore, the findings from CPEs and CLR Reviews were complemented by country- level insights from a purposive sample of sector and thematic evaluations (including corporate evaluations), Project Performance Assessment Reports, and other relevant evaluative material. For the International Finance Corporation (IFC) and the Multilateral Investment Guarantee Agency (MIGA), the findings from CPEs and CLR Reviews were complemented by country-level insights from sector highlights, learning engagements, and synthesis notes. Appendix B provides a count of available IEG evaluations by year. An important caveat relates to the RAP’s use of IEG’s CLR Reviews, which are based on the Bank Group’s Completion and Learning Reviews (CLRs). IEG’s evaluation of outcome orientation at the country level pointed out that the Bank Group’s CLRs, which IEG’s CLR Reviews validate and rate, provide a partial picture of the Bank Group’s development outcomes at the country level because of their “overemphasis on those results that can be [quantitatively] measured and on results from lending projects” (World Bank 2020b, xiii). Furthermore, the “CLR rarely captures complementarities across instruments or institutions and so is not able to establish whether the Bank Independent Evaluation Group World Bank Group    3 Group’s contribution to country outcomes amounts to more than the sum of its parts” (World Bank 2020b, xiii). At the project level, the project ratings trends were updated from the previous RAP based on IEG’s project ratings database. For the World Bank, the main data sources at the project level were Implementation Completion and Results Report Reviews and Project Performance Assessment Reports. For IFC and MIGA, the main data sources at the project level were IFC Expanded Project Supervision Report Evaluation Notes and IEG Project Evaluation Summaries for IFC investments, IFC Project Completion Report Evaluation Notes for IFC advisory services, MIGA Project Evaluation Report (PER) Validation Notes, and IEG PERs. For the World Bank, the RAP identified some drivers of performance based on common characteristics of projects using unsupervised machine learning. This analysis was exploratory, and the results are provided in appendix C. For IFC investments and advisory services projects and MIGA guarantee projects, the RAP identified some drivers of performance at the two ends of the ratings spectrum (for example, highly successful and highly unsuccessful for IFC investments and advisory services projects and excellent and un- satisfactory for MIGA projects).5 Past RAPs have conducted qualitative and manual text analysis, which was complemented by quantitative analyses (regression) analyses. The focus on the two ends of the ratings spectrum in this RAP aimed to elicit any new insights on what worked well and not so well. Because of the small number of evaluated IFC and MIGA projects with development outcome ratings at the two ends of the ratings spectrum, the RAP reviewed the project documents manually. The small number of projects rated at the two ends of the rating spectrum makes it challenging to draw firm conclusions and limits generalizability. Main Evaluation Questions and Report Structure Results and Performance of the World Bank Group 2022  Chapter 1 The main evaluation questions answered by this RAP and the corresponding chapters are as follows: » At the project level for the World Bank, how have IEG project ratings (out- come ratings and Bank performance ratings) changed over time and across types of countries and operations? (See chapter 2.) » At the project level for IFC and MIGA, what explains variations in the ratings at the two ends of the rating spectrum (highly successful and highly unsuc- cessful for IFC and excellent and unsatisfactory for MIGA)? For IFC invest- ment and advisory services and MIGA guarantee projects (in addition to the previous question), what was the relationship between development outcome and work quality and between development outcome and IFC’s additional- ity and MIGA guarantees role and contribution? For IFC investments, what was the relationship between development outcome and IFC’s investment outcome? For IFC, what was the relationship between IFC advisory services 4 and investments (particularly those IFC investment projects that had prior or concurrent advisory services)? (See chapter 3.) » At the country level for the Bank Group, how have IEG country program ratings trends and patterns evolved over time and across types of countries? To what extent did the timing, type, and performance of Bank Group interventions have an impact on the achievement of Country Partnership Framework (CPF) objectives? (See chapter 4.) » At the country level for the Bank Group, to what extent did the Bank Group’s support (that is, project portfolio and ASA) contribute to the achievement of the CPF’s objectives and the development outcome? To what extent was there a line of sight between the development outcome and high-level out- comes? (See chapter 5.) » For ASA for the World Bank, how was the use or influence of ASA reported at the country program level? (See chapter 6.) Independent Evaluation Group World Bank Group    5 The World Bank Group consists of the World Bank (International Bank for Reconstruction 1  and Development and the International Development Association), the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) for the purpos- es of this Results and Performance of the World Bank Group (RAP) report. This RAP does not cover the International Centre for Settlement of Investment Disputes, which is also part of the Bank Group. 2  World Bank (2020b) assessed the Bank Group’s country-level results system, specifically the extent to which it supports the organization’s monitoring, evaluation, and learning needs to effectively manage country engagements (it did not assess the project-level results system). This 2022 RAP does not duplicate but rather complements that evaluation by reviewing avail- able evidence on the development effectiveness of the Bank Group’s support at the country and project levels. 3  The 152 Completion and Learning Review (CLR) Reviews cover 108 countries with at least one CLR Review conducted since fiscal year (FY)13 and up through March 7, 2022. The actual population size is 103 countries because the 6 Organisation of Eastern Caribbean States countries are treated as one, given that they received aggregate instead of individual ratings and all belong to the same stratum of income level and fragile and conflict-affected situations (FCS) status. For each country, its income level and FCS status are tagged based on the ending Results and Performance of the World Bank Group 2022  Chapter 1 FY of the latest Country Partnership Framework (CPF) period that the Independent Evalua- tion Group (IEG) reviewed. 4  The nine completed Country Program Evaluations (CPEs) include two clustered CPEs: (i) one that covers Bolivia, Kazakhstan, Mongolia, and Zambia, with a separate, full CPE for each country (World Bank 2015b); and (ii) another that addresses six Organisation of Eastern Ca- ribbean States countries, nine Pacific island countries, and four African countries (Cabo Verde, Djibouti, Mauritius, and the Seychelles), with full CPEs for Organisation of Eastern Caribbean States countries and Pacific island countries at the regional level and case studies for Mauri- tius and the Seychelles (World Bank 2016d). 5  The development outcomes of IFC investment projects and the development effectiveness of IFC advisory services projects are rated on a six-point scale: highly successful, successful, mostly successful, mostly unsuccessful, unsuccessful, and highly unsuccessful. Until FY19, evaluated MIGA projects were rated excellent, satisfactory, partly unsatisfactory, and unsat- isfactory. MIGA and IEG adopted a six-point rating scale in FY20 for consistency with IFC and the World Bank. 6 2 | World Bank Project Portfolio Performance Highlights Project outcome ratings in fiscal year [FY]21 rose marginally to 85 percent of projects rated moderately satisfactory or higher, compared with 84 percent in FY20. No decline is observed in the outcome ratings for World Bank projects that closed during the COVID-19 pandemic (FY20 and FY21). At the time of the 2021 Results and Performance of the World Bank Group (RAP), 88 percent of projects had a rating of moderately satisfactory or higher. However, since more projects that closed in FY20 have been reviewed after publication of the 2021 RAP, that figure is now 84 percent. The percentage of projects in fragile and conflict-affected situa- tions rated moderately satisfactory or higher for outcome in FY21 was 74 percent, compared with 82 percent in FY20. Among projects rated moderately satisfactory or higher on out- come, the proportion rated satisfactory or highly satisfactory increased from 54 percent in FY20 to 58 percent in FY21, implying a decline in the share of projects rated moderately satisfactory. Projects in two of the four Practice Groups (Human Development and Sustainable Development) averaged higher outcome ratings in FY21 than in FY20, but ratings declined in the Infrastructure and the Equitable Growth, Finance, and Institutions Practice Groups. Four of the seven Regions (Eastern and Southern Africa, East Asia and Pacific, Europe and Central Asia, and South Asia) received higher project outcome ratings in FY21 than in FY20. 7  Bank performance continued to be strong, with performance in 91 percent of projects rated moderately satisfactory or higher in FY21, up from 86 percent in FY20, among which the proportion of projects rated satisfactory or highly satisfactory rose from 41 per- cent to 45 percent. Bank performance ratings for fragile and conflict-affected situations declined marginally to 81 percent of projects rated moderately satisfactory or higher in FY21 (from 82 percent in FY20). Monitoring and evaluation quality registered a clear improvement: the share of projects rated high or substantial increased from 57 percent in FY20 to 64 percent in FY21. 8   Project Rating Trends and Patterns This RAP updates the project ratings, building on previous RAPs. The lend- ing project outcome ratings remained high in FY21 after the increase in FY20. Among investment project financing (IPF) and Program-for-Results (P4R) projects closed in FY21, 85 percent were rated moderately satisfactory or higher, the highest since FY11 (figure 2.1, panel a). The average outcome rating is 4.4 on a six-point scale, compared with 4.3 in FY20. There are more fluctuations in development policy financing (DPF) project ratings, with the average rating improving from 4.0 in FY20 to 4.3 in FY21 and the percentage rated moderately satisfactory or higher declining from 88 percent in FY20 to 86 percent in FY21 (figure 2.1, panel b). However, such shifts are unlikely to be stable, given the small number of DPF proj- ects in both periods (24 projects in FY20 and 7 in FY21). A more detailed presentation of project outcome ratings can be found in appendix D. Bank performance ratings, including both quality at entry and quality of supervi- sion, maintained their upward trend. Between FY20 and FY21, the percent- age of IPF and P4R projects with moderately satisfactory or higher ratings improved from 85 to 90 percent for Bank performance, 80 to 85 percent for quality at entry, and 85 to 92 percent for quality of supervision (figure 2.2). A similar increase was seen in ratings for projects in the Human Development and Sustainable Development Practice Groups, the Eastern and Southern Africa Region, and non–fragile and conflict-affected situations (FCS); there was a similar decline for projects in the Infrastructure and Equitable Growth, Independent Evaluation Group World Bank Group    9 Finance, and Institutions Practice Groups, the Middle East and North Af- rica Region, and FCS. Bank performance ratings also improved among DPF projects, from 92 percent rated moderately satisfactory or higher in FY20 to 100 percent in FY21 (figure 2.3). Project outcome and Bank performance show a strong positive correlation, with both trending upward (figure 2.4). Meanwhile, there are a few outliers. For example, in Western and Central Africa, the percentage of projects with Bank performance rated moderately satisfactory or higher increased from 80 percent in FY20 to 83 percent in FY21, while the outcome decreased from 84 to 81 percent. Figure 2.1. Distribution of the World Bank’s Project Outcome Ratings a. Investment project financing and Program-for-Results projects b. Development policy financing projects Results and Performance of the World Bank Group 2022  Chapter 2 Source: Independent Evaluation Group data. 10  ank Performance, Quality at Entry, and Quality of Figure 2.2. B Supervision for World Bank Projects: Investment Project Financing and Program-for-Results Projects a. Projects rated MS+ 100 Quality of supervision Bank performance Quality at entry 80 Projects rated MS+ (%) 60 40 20 0 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 Fiscal year of project closing b. Projects rated S+ 100 80 Quality of supervision Projects rated S+ (%) Independent Evaluation Group World Bank Group    11 60 Quality at entry 40 Bank performance 20 0 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 Fiscal year of project closing Source: Independent Evaluation Group data. Note: MS+ = moderately satisfactory or higher; S+ = satisfactory or higher.  ank Performance Ratings for World Bank Projects: Figure 2.3. B Development Policy Financing Source: Independent Evaluation Group data.  utcome and Bank Performance Rated MS+ for World Bank Figure 2.4. O Projects: Investment Project Financing and Program-for- Results Results and Performance of the World Bank Group 2022  Chapter 2 100 Bank performance Outcome 80 Projects rated MS+ (%) 60 40 20 0 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 Fiscal year of project closing Source: Independent Evaluation Group data. Note: MS+ = moderately satisfactory or higher. 12 Among IPF and P4R projects, the steadily upward march in monitoring and evaluation (M&E) quality ratings continued. The percentage of projects with M&E quality ratings of high or substantial increased from 57 percent in FY20 to 64 percent in FY21 (figure 2.5), but the increase is driven by only two Practice Groups: Human Development (up from 76 to 97 percent) and Sustainable Development (up from 51 to 62 percent). A significant increase in M&E quality ratings is also found in projects of $25–100 million and in non-FCS. All Regions experienced the improvement except for Latin America and the Caribbean, where the percentage of projects rated high or substan- tial on M&E quality plummeted from 63 percent in FY20 to 46 percent in FY21, with 24 projects rated in both fiscal years. East Asia and Pacific had the most pronounced increase in percentage of projects with the higher ratings (rising from 54 percent in FY20 to 74 percent in FY21). Despite the decrease in outcome rating, the percentage of projects rated high or substantial on M&E quality in Western and Central Africa increased from 59 percent in FY20 to 69 percent in FY21.  istribution of the World Bank’s Project Monitoring and Figure 2.5. D Evaluation Quality Ratings a. Investment project financing and Program-for-Results projects Independent Evaluation Group World Bank Group    13 b. Development policy financing projects b. Development policy financing projects Source: Independent Evaluation Group data. Note: There are no development policy financing projects with monitoring and evaluation quality ratings in 2021. Project Performance during the COVID-19 Pandemic Results and Performance of the World Bank Group 2022  Chapter 2 Among the projects rated by IEG, just 10 projects were identified as respond- ing to COVID-19, of which 7 received satisfactory outcome ratings and 3 received a moderately satisfactory outcome rating (appendix D). The RAP also found that the pre–COVID-19 CLR Review ratings are a good predictor of project performance during the COVID-19 time period (appendix D). 14 3 | International Finance Corporation and Multilateral Investment Guarantee Agency Project-Level Performance Highlights Development outcome ratings of recently evaluated International Finance Corporation (IFC) investment projects have improved slightly, from 47 percent in 2017–19 to 48 percent in 2018–20 and to 52 percent in 2019–21. By region, investment projects in South Asia and Central Asia and Türkiye outperformed projects in other regions, especially Africa, whose development outcome ratings declined further in 2019–21. Development outcome ratings of projects in fragile and conflict-affected situations (FCS) have weakened, with only 21 percent of evaluated projects rated mostly successful or higher on development outcome. Development effectiveness ratings of IFC advisory services projects have improved in 2019–21. Development effectiveness ratings of projects in International Development Association countries outperformed projects in non–International Development Association countries, while success ratings of advisory services projects in FCS continue to improve, though projects in non-FCS have a higher development effectiveness rating. Development outcome ratings of recently evaluated Multilateral Investment Guarantee Agency (MIGA) guarantee projects improved substantially, with 70 percent of projects rated satisfactory or better in 2016–21, compared with 66 percent in 2015–20 (from a previous high of 69 percent in 2013–18), but only two out of eight projects were rated in 2021 program. Development outcome ratings of projects in FCS have also held steady since 2015–20, 15  but development outcome ratings of projects in International Development Association countries continue to weaken. Work quality and role and contribution are both important aspects for IFC and MIGA in improving development results. MIGA’s role and contribution rating reached an all-time high in 2016–21 (with the caveat that the 2021 MIGA self-evaluation and validation pro- gram is ongoing, with only two out of eight projects rated so far). For IFC investments, IFC’s returns also suffer when projects have poor development outcomes. 16   International Finance Corporation Investment Projects Development outcome ratings of recently evaluated IFC investment projects have improved slightly. Fifty-two percent of investment projects evaluated during the calendar years 2019–21 were rated, on average, mostly successful or higher, compared with 48 percent during the previous three-year period (2018 to 2020) and 47 percent in 2017–19. The overall improvement was aided by an increase in the share of projects rated successful and the gradual decline in projects rated unsuccessful from 2015 to 2017 (figure 3.1). However, the overall development outcome ratings at the lower end of the rating scale (highly unsuccessful) increased in the 2019–21 period. On an annual basis, the share of projects rated highly unsuccessful in 2020 and 2021 reached 13 percent and 10 percent, respectively—a substantial increase from the 4 percent of projects with a similar rating in 2019 and 2018 (appendix E). The share of projects rated highly unsuccessful in 2012–21 averaged 9 percent of all evaluated projects during that period. By contrast, the average share of projects with highly successful development outcome ratings held steady at 2 percent of all projects evaluated in 2012–21. Figure 3.1. Overall Development Outcome Ratings, 2012–21 100 Share of projects rated (%) 80 Independent Evaluation Group World Bank Group    17 60 MS+ 40 20 0 2012–14 2013–15 2014–16 2015–17 2016–18 2017–19 2018–20 2019–21 Three-year rolling average by calendar year of self-evaluation Highly unsuccessful Unsuccessful Mostly unsuccessful Mostly successful Successful Highly successful Source: Independent Evaluation Group data. Note: This figure is based on 809 Expanded Project Supervision Report Evaluation Notes completed between 2012 and 2021, of which 14 projects were rated highly successful and 69 were rated highly un- successful on development outcome. One project has a rating of no opinion possible and was excluded from the count. MS+ = mostly successful or higher. Projects with highly successful development outcome ratings also have high ratings on IFC’s work quality, on IFC’s role and contribution or additionality to the project and the client, and in exceeding IFC’s expected investment returns. Only 14 of 808 evaluated projects in 2012–21 are rated highly successful,1 compared with 69 projects with highly unsuccessful development outcome ratings. Highly successful projects have excellent ratings on their business performance and economic sustainability and received either an excellent or satisfactory rating on the projects’ environmental and social (E&S) effects and in promoting development of the private sector. Eight of the projects rated highly successful were medium-size investments.2 Three of the 4 equity investments projects with highly successful development outcome ratings involved greenfield projects, which goes against the historic underperformance of projects involving greenfield investments.3 Five of the 14 projects rated highly successful were financed by a combination of IFC loan and equity, 5 projects received IFC loans, and IFC provided only equity investment in 4 projects. However, unlike most projects financed by equity investments (which tend to have low development outcome ratings), in the 4 highly successful equity projects, IFC had the right strategic partner who maintained good synergy and a good Results and Performance of the World Bank Group 2022  Chapter 3 working relationship with IFC throughout the investment and benefited from IFC’s close supervision. Highly successful equity projects also gained from an enabling policy environment or at least neutral regulatory framework and from more open market access. These aspects point to the high work quality of the IFC investment team at appraisal and structuring and during monitoring and supervision. IFC’s financial additionality through sound structuring of IFC financing and mobilizing financing were most common, and IFC’s nonfinancial value added was mainly in the form of the introduction of new and better standards of corporate governance, E&S compliance, and operational aspects. Projects with highly successful development outcome ratings are also rated highly for exceeding IFC’s investment return expectations. Conversely, projects with highly unsuccessful development outcome ratings were also rated poorly on IFC’s work quality, IFC’s role and contribution, and the lower-than-expected investment return to IFC. Projects with 18 highly unsuccessful development outcome ratings were rated unsatisfactory for their project business performance and economic sustainability and either unsatisfactory or partly unsatisfactory in fostering private sector development. Highly unsuccessful projects are also noncompliant with IFC’s E&S Performance Standards. Most of the projects involved either equity investments (32 of 69 projects) or loans (28 or 69 projects) that did not meet IFC’s expected gross returns.4 In contrast with equity investment projects with highly successful outcome ratings, the quality of IFC’s screening, appraisal and structuring, and supervision and administration of projects rated highly unsuccessful were mostly rated either unsatisfactory or partly unsatisfactory. Although IFC took a risk in investing equity in projects with sponsors that had weak track records, adequate risk mitigation measures were lacking. Poor due diligence of the project sponsor, coupled with a weak risk assessment and risk mitigation plan by the IFC investment team, is common among equity investment projects with highly unsuccessful development outcome ratings. Highly unsuccessful equity investment projects have poorly defined project scope, with implementation timelines that resulted in considerable time and cost overruns and the sponsors taking a much more aggressive investment strategy that caused financial and operational problems. Corporate governance problems were also common in several equity investment projects rated highly unsuccessful. Government policy distortions weakened the business and operating environment of most equity projects rated highly unsuccessful. The low ratings in IFC’s role and contribution to the project and the client reflect the failure to address the weaknesses inherent in these Independent Evaluation Group World Bank Group    19 projects. Despite some contrasting features, investment projects rated at the extreme ends have diverse characteristics, which limits generalizability. The invest- ment projects rated at the extreme ends are too diverse to identify patterns in addition to the few unique characteristics described in the previous two paragraphs. Projects rated highly successful and highly unsuccessful are in FCS, International Development Association (IDA), non-FCS, or non-IDA countries or in the same region or same industry group. Most of the projects rated at the extreme ends involved medium-size investments (8 of 14 proj- ects rated highly successful and 41 of 69 projects rated highly unsuccessful). Five of 14 projects rated highly successful are in Africa, and at the opposite end, 23 percent (16 of 69) of projects rated highly unsuccessful are also in the Region. The Latin America and the Caribbean Region had the highest share of projects rated highly unsuccessful (17 of 69, or 25 percent), followed by projects in Africa. By IFC industry group, 6 of 14 projects rated highly suc- cessful were in Infrastructure and Natural Resources, but 42 percent (29 of 69) of projects rated highly unsuccessful were also in that industry group, which has the highest share of projects with highly unsuccessful develop- ment outcome ratings by industry group. Five of 14 projects rated highly successful were in the Manufacturing, Agribusiness, and Services industry group; at the same time, 21 of 69 (30 percent) projects rated highly unsuc- cessful were also in this industry group. A review of the project approval dates for projects rated at the extreme ends shows that both sets of projects share similar approval and commitment years. For example, there were proj- ects rated highly successful and highly unsuccessful approved in 2007, 2008, 2009, and 2010 (the years of the global financial crisis). Using IFC’s new regional classification, projects in Central Asia and Türkiye and in South Asia outperformed projects in other regions, especially projects in Africa, whose development outcome ratings have declined further in the past three years. In 2019–21, 7 of the 9 evaluated projects in Central Asia Results and Performance of the World Bank Group 2022  Chapter 3 and Türkiye and 20 of 26 projects (77 percent) of evaluated projects in South Asia were rated mostly successful or higher on their development outcome (figure 3.2). By contrast, 31 percent of evaluated projects in Africa in 2019– 21 were rated mostly successful or higher on development outcome, down slightly from 34 percent in 2018–20 but a significant decline from 57 percent in 2012–14. In 2019–21, nearly half of the mostly unsuccessful or lower rated projects in Africa were in the Manufacturing, Agribusiness, and Services industry group; most of these projects are in the health, education, and life sciences subsector.5 Additionally, more projects by the Financial Institutions Group in Africa were rated mostly unsuccessful or lower on development outcome in 2019–21. Several of the Financial Institutions Group projects involved IFC special programs (particularly IFC’s Risk-Sharing Facility) with local banks to provide local currency financing to small and medium enter- prises. Experience so far indicates that the use of this first-loss guarantee facility has been challenging and unsuccessful.6 20 The declining development outcome rating of projects in the Africa Region constrains overall IDA outcome ratings. The overall development outcome success rates of projects in IDA held steady in 2017–19 because of improvements in the development outcome ratings of projects in non- IDA Africa. Projects in non-IDA Africa have better development outcome ratings than projects in IDA Africa. Development outcome ratings of projects in non-IDA Africa countries have recovered from 44 percent in 2015–17 to 70 percent in 2019–21 because of improvements in the development outcome ratings of IFC’s equity investments and small-size investment projects in non-IDA Africa countries.7 If IDA Africa projects ratings are excluded, the development outcome ratings of projects in IDA would be higher, which suggests that an improvement in the development outcome ratings of projects in IDA Africa is likely to make a difference in the results and performance of IFC projects in IDA countries (figure 3.3).  hare of MS+ Development Outcome Ratings, by IFC Region, Figure 3.2. S 2012–21 100 Share of projects rated MS+ (%) 80 60 40 20 Independent Evaluation Group World Bank Group    21 0 2012–14 2015–17 2018–20 2019–21 Three-year rolling average by calendar year of self-evaluation AFR CAT EAP EUR LAC ME SA Source: Independent Evaluation Group data. Note: Excludes 22 evaluated investment projects implemented across different regions or world projects. Afghanistan and Pakistan are grouped in the Middle East region in IFC’s September 2021 regional classification. AFR = Africa; CAT = Central Asia and Türkiye; EAP = East Asia and the Pacific; EUR = Europe; IFC = International Finance Corporation; LAC = Latin America and the Caribbean; ME = Middle East; MS+ = mostly successful or higher; SA = South Asia.  evelopment Outcome Ratings of Projects in IDA and Non- Figure 3.3. D IDA Countries, 2012–21 100 Share of projects rated MS+ (%) 80 IDA, excluding 60 IDA-Africa All IDA 40 20 0 2012–14 2013–15 2014–16 2015–17 2016–18 2017–19 2018–20 2019–21 Three-year rolling average by calendar year of self-evaluation Non-IDA Source: Independent Evaluation Group data. Note: Of the 809 investment projects evaluated in calendar years 2012–21, 270 are in IDA countries, of which 158 are in non-IDA Africa countries and 539 evaluated projects are in non-IDA countries. IDA country classification is based on the year of International Finance Corporation investment commitment. IDA = International Development Association; MS+ = mostly successful or higher; AFR = Africa. IFC’s work quality ratings of projects in the Africa Region have plateaued Results and Performance of the World Bank Group 2022  Chapter 3 recently and lag projects in other regions. IFC’s work quality represents an assessment of its operational performance, including assessment of E&S work; IFC’s up-front work in screening, appraisal, and structuring; and its monitoring and supervision after project approval by the Board of Executive Directors and subsequent IFC commitment. Although the work quality rating of projects in Africa has recovered from its low point in 2015, it still lags other regions (see appendix E, figure E.3, panel b). Ratings in the quality of IFC’s project screening, appraisal, and structuring—especially financial sector projects—are low, with 36 of the 86 projects rated partly unsatisfactory or lower and followed by projects in the agribusiness and forestry subsector (12 of 86 projects have low ratings). By contrast, more projects in Central Asia and Türkiye and in South Asia are rated satisfactory or higher for IFC’s work quality,8 especially in the quality of screening, apprais- al, and structuring of projects. Work quality affects projects’ development outcome ratings. Among all projects evaluated in 2012–21, the likelihood of the project receiving a 22 successful development outcome rating is low (7 percent) when the project has a poor work quality rating (partly unsatisfactory or lower; appendix E, figure E.4). Regarding the two dimensions of IFC’s work quality, a high rating on the quality of supervision and administration increases the likelihood of a successful development outcome by 44 percent, and the chance of a suc- cessful development outcome rating is slim (at 6 percent) if the quality of monitoring and supervision is also rated unsatisfactory. Effective monitoring and supervision ensure that risks are addressed during implementation and that the client and project meet IFC’s requirements and provide IFC with a chance to offer its knowledge and expertise to the project and client. How- ever, if the rating on IFC’s screening, appraisal, and structuring is low, there is a 34 percent chance that a project will also receive a low development outcome rating. This aspect of IFC’s work quality reflects IFC’s due diligence conducted on the project and the sponsor, as well as the structuring of the project in a manner that mitigates risks and ensures that the sponsor meets IFC’s requirements. Therefore, IFC’s work quality matters both at the front end (screening, appraisal, and structuring) and at implementation (super- vision and administration). The association is stronger in projects that are rated at the low end of the rating scale—84 percent of projects rated low (un- satisfactory) on work quality are rated highly unsuccessful on development outcome (appendix E, figure E.4), and there is no chance of getting a high development outcome rating if IFC’s work quality is rated unsatisfactory. Independent Evaluation Group World Bank Group    23 Development outcome ratings of projects in FCS have weakened. Despite the small number projects in evaluated between 2012 and 2021,9 the develop- ment outcome trend for projects in FCS has been declining in recent years (figure 3.4). Low development outcome ratings of projects in FCS, an IFC cor- porate priority, can be observed across all industries, by size of investment commitment and type of IFC investment instrument. Projects in FCS have weak ratings across all four indicators of development outcome—business performance, economic sustainability, E&S effects, and private sector devel- opment—compared with projects in non-FCS. World Bank (2022b) found that more infrastructure projects in FCS evaluated during 2010–17 were rated successful because of project sponsors or project developers with global experience and broader access to capital and that the larger the investment size, the higher the likelihood of achieving positive development outcomes. This finding holds true for FCS projects in earlier years, but there has been a gradual reversal of this trend since 2017, although IFC has introduced sev- eral initiatives and instruments in FCS. In the past five years, more large infrastructure projects in FCS were rated mostly unsuccessful or lower, thus contributing at least partially to the decline in the overall development out- come ratings of projects in FCS. By contrast, large infrastructure projects in non-FCS showed improved development outcome ratings, which exacerbated the divergence in development outcome ratings between projects in FCS and non-FCS in the past five years.  evelopment Outcome Ratings of Projects in FCS and Figure 3.4. D Non‑FCS, 2012–21 100 Share of projects rated MS+ (%) 80 60 Non-FCS 40 Results and Performance of the World Bank Group 2022  Chapter 3 20 FCS 0 2012-14 2013-15 2014-16 2015-17 2016-18 2017-19 2018-20 2019-21 Three-year rolling average by calendar year of self-evaluation Source: Independent Evaluation Group data. Note: Of the 809 investment projects evaluated in calendar years 2012–21, 56 are in FCS and 753 are in non-FCS. Nineteen of the 216 investment projects evaluated in 2019–21 are in FCS and 197 are in non- FCS. Nineteen projects in FCS were evaluated in 2019–21, of which 4 are rated MS+. FCS = fragile and conflict-affected situations; MS+ = mostly successful or higher. IFC’s additionality rating remains stable and closely associated with devel- opment outcome ratings. IFC considers additionality essential to achieving development impact, whether in the form of delivering positive outcomes to stakeholders or helping create or develop markets. Sixty percent of evaluated investment projects during the 10-year period 2012–21 benefited from both financial and nonfinancial value added from IFC, while nearly one-third had only financial additionality from IFC, mostly in the form of financial struc- 24 turing of the transaction. In 2019–21, projects rated satisfactory or higher for IFC’s additionality reached 61 percent, compared with 58 percent in 2018–21.10 Results for 2019–21 indicate that projects in the Europe, Central Asia and Türkiye, and East Asia and the Pacific regions; the Manufacturing, Agribusiness, and Services and the Infrastructure and Natural Resources industry groups; non-FCS; and projects involving large investments financed by IFC loans have a higher share of satisfactory or higher IFC’s additionality ratings than their respective cohorts. On average, there is only a 7 percent likelihood that projects with low IFC’s additionality ratings will also have suc- cessful development outcomes (appendix E, figure E.7, panel a). However, the association between development outcome and IFC’s additionality ratings is closer among projects rated at the two extreme ends of the ratings scale (ap- pendix E, figure E.7, panel b), where there is a 91 percent chance of receiving a low development outcome rating if the additionality that IFC planned to deliver is rated low, or a zero chance of rating successful on development outcome if IFC’s additionality is rated unsatisfactory. IFC’s additionality ratings of projects in FCS have also declined, while work quality ratings remained steady. IFC’s additionality is the unique benefit or value addition that IFC brings to a private sector client, a project, or a cli- ent country that is not typically offered by commercial sources of finance.11 A premise of IFC’s support to FCS is that it can provide value added through financial and nonfinancial means that other financiers are unable to provide. However, only 32 percent of projects in FCS during 2019–21 are Independent Evaluation Group World Bank Group    25 rated satisfactory or higher for IFC’s unique value added that it planned to deliver to the project and the client, compared with 62 percent of projects rated satisfactory or higher among non-FCS projects (figure 3.5). Half of the projects in FCS benefited from a combined financial and nonfinancial value added from IFC, mostly in the form of financial structuring of the IFC investment and in assisting clients with adopting new or better business, corporate governance, and E&S standards.12 Projects in FCS also have lower ratings than projects in non-FCS in supervision and administration, one of two indicators of IFC’s work quality, and that has a greater effect on overall work quality and IFC’s investment outcome ratings.  evelopment Outcome, Investment Outcome, Work Quality, Figure 3.5. D and IFC’s Additionality Ratings in FCS and Non-FCS 100 Share of projects rated MS+ or S+ (%) 80 60 40 20 0 FCS Non-FCS FCS Non-FCS FCS Non-FCS FCS Non-FCS Development IFC’s investment IFC’s work quality IFC’s additionality outcome outcome XPSR project rating dimension 2017–19 2018–20 2019–21 Source: Independent Evaluation Group data. Note: Development outcome is rated on a six-point rating scale (highly successful, successful, mostly successful, mostly unsuccessful, unsuccessful and highly unsuccessful). IFC’s investment outcome, work quality, and additionality are rated on a four-point rating scale (excellent, satisfactory, partly unsat- isfactory, and unsatisfactory). FCS = fragile and conflict-affected situations; IFC = International Finance Corporation; MS+ = mostly successful or higher; S+ = satisfactory or higher; XPSR = Expanded Project Supervision Report. Results and Performance of the World Bank Group 2022  Chapter 3 Overall IFC’s investment outcome success ratings are higher than the ratings for development outcomes, suggesting that even projects with weak development outcomes met IFC’s investment returns benchmarks. At the project level, IFC’s investment outcome measures the extent to which IFC has realized and expects to realize, over the remaining life of the investment, the loan or equity returns that were expected at approval.13 In 2019–21, 64 percent of projects were rated satisfactory or higher for their investment outcome, an improvement from the 57 percent of projects rated satisfactory or higher in 2018–20 (appendix E, figure E.9, panel a). By contrast, 52 percent of projects were rated mostly satisfactory or higher on their development outcome in 2019–21, an improvement from the 48 percent success rate in 2018–20. The association between IFC’s investment outcome and development outcome ratings is reflected in appendix E, figure E.10, panel a, which shows that among all evaluated projects in 2012 to 2021, there is a 41 percent likelihood that developmentally successful projects also bring good returns to IFC, but 26 there is only an 8 percent likelihood that projects can be rated successful on development outcome if IFC’s investment outcome ratings in projects is low. The association between development outcome and IFC’s investment outcome ratings is closer among projects rated at the two extreme ends of the ratings scale (appendix E, figure E.10, panel b), indicating that the chance of achieving a successful development outcome is zero if the investment returns to IFC are low. Projects in higher-income countries have better investment outcome ratings compared with projects in IDA countries, FCS, and Africa (particularly Sub-Saharan Africa), and returns from the former group offset low returns in the latter—the premise of IFC’s portfolio approach. The improving share of investment outcome satisfactory or higher ratings can be explained partly by the significant improvement in investment outcome ratings of projects in the East Asia and the Pacific region—from 45 percent rated satisfactory or higher in 2015–17 to 72 percent in 2019–21, which is now on par with the investment outcome ratings of projects in the Europe and South Asia regions. Investment projects with IFC advisory services support slightly improved the investment project’s development outcome and additionality ratings but not IFC’s investment outcome rating. During 2012–21, 69 evaluated IFC invest- ment projects benefited from prior or simultaneous implementation with IFC advisory services support. On average, 55 percent of these linked IFC investment services and advisory services projects were rated mostly satis- factory or higher, compared with the 49 percent ratings for all the evaluated stand-alone IFC investment projects in the same period (figure 3.6). Work Independent Evaluation Group World Bank Group    27 quality of evaluated IFC investment projects implemented with advisory ser- vices projects have, on average, a slightly higher percentage of satisfactory or higher ratings (61 percent), compared with 59 percent for all IFC investment projects during the 10-year period. On average, investment services and advisory services linked projects show 65 percent satisfactory or higher IFC’s additionality ratings compared with 60 percent satisfactory or higher ratings for stand-alone IFC investment projects. However, an IFC investment project paired with an IFC advisory services project before or simultaneously with an IFC investment has a lower IFC’s investment outcome rating, suggesting that deploying advisory services may help enhance development outcome and IFC’s additionality but has some trade-offs when it comes to realizing IFC’s expected investment returns and therefore must be considered selectively, particularly advisory projects that are paired with only equity investment from IFC. Most of the investment services–advisory services projects with low in- vestment outcome ratings (16 of 36 projects) received only equity investment from IFC, which partly explains the weak investment outcome ratings. FC Investment Projects with Advisory Services Projects Figure 3.6. I Tend to Have Better Development Outcomes and IFC’s Additionality 100 Share of projects rated MS+ or S+ (%) 80 60 40 20 0 IS only IS-AS IS only IS-AS IS only IS-AS IS only IS-AS Development IFC’s work quality IFC’s additionality IFC’s investment outcome outcome XPSR project rating dimension Results and Performance of the World Bank Group 2022  Chapter 3 Source: Independent Evaluation Group data. Note: Development outcome is rated on a six-point rating scale (highly successful, successful, mostly successful, mostly unsuccessful, unsuccessful and highly unsuccessful). IFC’s investment outcome, work quality and additionality are rated on a four-point rating scale (excellent, satisfactory, partly unsat- isfactory, and unsatisfactory). AS = advisory services; IS = investment services; IFC = International Finance Corporation; MS+ = mostly successful or higher; S+ = satisfactory or better; XPSR = Expanded Project Supervision Report. International Finance Corporation Advisory Services Development effectiveness ratings of evaluated IFC advisory services projects have stabilized since the 2015–17 period. In 2019–21,14 62 percent of advisory services projects were rated mostly successful or higher on development effectiveness, a significant recovery from their lowest level of 38 percent in 2015–17 (figure 3.7). The improved development effectiveness success rating comes mainly from the increase in the share of projects rated mostly successful or higher. The share of projects rated mostly unsuccessful on development effectiveness has declined. Projects rated at the extreme 28 ends (highly successful and highly unsuccessful) and the share of projects rated successful have remained relatively stable and are thus unlikely to have contributed to the recent positive trend. Year-on-year development effectiveness ratings also recovered from a 23 percent success rate (mostly successful or higher) in 2016 to 65 percent in 2019 before declining to 56 percent in 2021 (appendix E, figure E.11). FC Advisory Services Projects’ Development Effectiveness Figure 3.7. I Ratings 100 80 Share of projects (%) MS+ 60 40 20 0 2012–14 2013–15 2014–16 2015–17 2016–18 2017–19 2018–20 2019–21 Three-year rolling average by fiscal year of self-evaluation Highly unsuccessful Unsuccessful Mostly unsuccessful Mostly successful Successful Highly successful Source: Independent Evaluation Group data. Note: Based on 574 IFC advisory services evaluated between fiscal years 2012 and 2021, as of July 7, 2022. Eight projects are rated highly successful, and 13 are rated highly unsuccessful on development effectiveness. One project had a rating of not applicable and was not included in this and other graphs. Independent Evaluation Group World Bank Group    29 There were 154 projects evaluated in fiscal years 2019–21, of which 95 were rated mostly successful or higher on development effectiveness. IFC = International Finance Corporation; MS+ = mostly successful or higher. The South Asia and Central Asia and Türkiye regions have a higher share of projects rated mostly successful or higher on development effectiveness than other regions. In 2019–21, 18 of 20 evaluated projects in South Asia and 7 of 9 evaluated projects in Central Asia and Türkiye are rated mostly successful or higher on development effectiveness.15 Advisory services projects in Latin America and the Caribbean lagged other regions, with 47 percent of projects rated mostly successful or higher in 2019–21 (appendix E, figure E.12). Development effectiveness ratings of projects in FCS continue to improve. Eighteen of 32 projects (56 percent) evaluated in 2019–21 are rated mostly successful or higher on their development effectiveness, a stark contrast to the 22 percent success rate in 2015–17 (figure 3.8). The recent result suggests that some of the factors that IEG has previously identified as positively affecting development effectiveness have a larger effect on the FCS projects cohort. Past RAPs have reported the factors that negatively affect development effectiveness, such as large project size, longer project durations, and high team leadership turnover. Lack of client engagement or ownership, poorly executed M&E framework, and weak supervision practices were also important contributors to the decline in development effectiveness ratings. These drivers of low success ratings reversed in the last five years, notably the improvement in IFC work quality ratings since 2016. Actions taken in response to the IEG-IFC joint study on the work quality ratings of IFC advisory services projects indicate positive effects on development effectiveness ratings,16 including during the most recent period, 2019–21. The shorter duration and smaller size of recently approved advisory services projects also reduced the chance that exogenous shocks resulting from conflict will affect the projects’ development effectiveness, although this also applies to projects in non-FCS. Additionally, a recent IEG evaluation (The International Finance Corporation’s and Multilateral Investment Guarantee Agency’s Support for Private Investment in Fragile and Conflict- Results and Performance of the World Bank Group 2022  Chapter 3 Affected Situations, Fiscal Years 2010–21; World Bank 2022b) found that locating IFC advisory services team leaders in the field may have reduced the frequent turnover among team leaders. Frequent team leader turnover, especially in complex projects, has negatively affected the development effectiveness of advisory services projects. Development effectiveness ratings of projects in IDA countries outperformed projects in non-IDA countries. Sixty-five percent of evaluated projects in IDA countries in 2019–21 received mostly successful or higher ratings on their development effectiveness—a significant improvement from the 38 percent of projects rated mostly successful or higher in 2015–17 (figure 3.9). Suc- cess rates in non-IDA countries in 2019–21 are lower (56 percent) but have improved slightly compared with the 51 percent and 52 percent success rates in the 2017–19 and 2018–20 periods, respectively. Development effectiveness and IFC role and contribution ratings are also strongly correlated. Projects with satisfactory or higher ratings on IFC role 30 and contribution also have high ratings on development effectiveness. In 2012–21, 54 percent of projects with satisfactory or higher ratings on IFC role and contribution were also rated highly on their development effectiveness. However, there is only a 1 percent chance of receiving a high development effectiveness rating if IFC role and contribution is rated low. The association is more evident for advisory services projects rated at the opposite ends of the development effectiveness and IFC role and contribution rating scales. Half of the projects with low ratings on IFC role and contribution were also rated low on development effectiveness (appendix E, figure E.20).  evelopment Effectiveness Ratings of Projects in FCS and Figure 3.8. D Non-FCS, 2012–21 100 Share of projects rated MS+ (%) 80 Overall MS+ 60 40 20 0 2012–14 2013–15 2014–16 2015–17 2016–18 2017–19 2018–20 2019–21 Three-year rolling average by fiscal year of self-evaluation FCS Non-FCS Independent Evaluation Group World Bank Group    31 Source: Independent Evaluation Group data. Note: FCS = fragile and conflict-affected situations; MS+ = mostly successful or higher. Of the 573 self-evaluated and validated International Finance Corporation advisory services projects in fiscal years 2012 to 2021, 88 are in FCS, and 485 are in non-FCS. Forty of the 88 projects are rated mostly successful or higher on development effectiveness.  evelopment Effectiveness Ratings of Projects in IDA and Figure 3.9. D Non-IDA Countries, 2012–21 100 Share of projects rated MS+ (%) 80 Overall MS+ 60 40 20 0 2012–14 2013–15 2014–16 2015–17 2016–18 2017–19 2018–20 2019–21 Three-year rolling average by fiscal year of self-evaluation IDA Non-IDA Source: Independent Evaluation Group data. Note: Of the 573 advisory services projects that were self-evaluated and validated advisory in fiscal years 2012–21, 350 are in IDA countries, and 223 are in non-IDA countries; 127 of the 350 projects are rated MS+. IDA = International Development Association; MS+ = mostly successful or higher. A high rating in IFC work quality increases the likelihood of a successful Results and Performance of the World Bank Group 2022  Chapter 3 development effectiveness rating. IFC’s improved work quality and moni- toring of development results were important factors in the improvement of development effectiveness ratings (World Bank 2021b). Forty-three percent of projects rated satisfactory or higher on IFC work quality are also rated mostly successful or higher on development effectiveness. The association is more evident in projects with ratings at the two extreme ends of the rating scale (that is, highly successful and highly unsuccessful). Sixty-nine percent of advisory services projects with unsatisfactory ratings on IFC work quality are also rated highly unsuccessful on development effectiveness. Crucially, there is zero chance of getting a highly successful rating on development ef- fectiveness if IFC work quality is rated unsatisfactory, and vice versa (appen- dix E, figure E.18). Development effectiveness and IFC role and contribution ratings are also strongly correlated. Projects with satisfactory or higher ratings on IFC role and contribution also have high ratings on development effectiveness. In 32 2012–21, 54 percent of projects with satisfactory or higher ratings on IFC role and contribution were also rated highly on their development effective- ness. However, there is only a 1 percent chance of receiving a high devel- opment effectiveness rating if IFC role and contribution is rated low. The association is more evident for advisory services projects rated at the oppo- site ends of the development effectiveness and IFC role and contribution rat- ing scales. Half of the projects with low ratings on IFC role and contribution were also rated low on development effectiveness (appendix E, figure E.20). Multilateral Investment Guarantee Agency Development outcome ratings of recently evaluated MIGA guarantee proj- ects showed considerable improvement. Including the two projects that were self-evaluated by MIGA and validated by IEG in the FY21 Project Evaluation Report (PER) program,17 70 percent of evaluated projects in FY16–21 are rated satisfactory or better, compared with 66 percent in FY15–20, 68 per- cent in FY14–19, and MIGA’s previous high development outcome rating of 69 percent in FY13–18.18 Disaggregated ratings on development outcome in figure 3.10 illustrate the stability of the ratings based on a six-year rolling average. On a year-on-year basis, however, evaluated projects in 2018, 2019, and 2020 show an increase in the share of projects rated unsatisfactory (from 6 percent in 2018 to 8 percent and 18 percent in 2019 and 2020, respective- ly). The share of projects rated satisfactory and partly unsatisfactory also declined during these three years (2018, 2019, and 2020). In 2021, the two Independent Evaluation Group World Bank Group    33 self-evaluated and validated projects in the FY21 PER program were both rated successful. With the small number of MIGA evaluated projects, ratings from one project can make a big difference in the success rates.  IGA’s Average and Disaggregated Development Outcome Figure 3.10. M Ratings, 2012–21 100 80 Share of projects (%) Overall S+ 60 40 20 0 2012–17 2013–18 2014–19 2015–20 2016–21 Six-year rolling average by fiscal year of self-evaluation Unsatisfactory Partly unsatisfactory Satisfactory Excellent Source: Independent Evaluation Group data. Note: Based on 108 projects evaluated in fiscal year (FY)12–21, of which 5 projects were rated excellent and 8 projects were rated unsatisfactory. Starting in FY20, MIGA projects are rated on development outcome on a six-point rating scale (a change from the previous four-point rating scale). There are only 2 of 8 projects that were self-evaluated and validated in the FY21 MIGA Project Evaluation Report Results and Performance of the World Bank Group 2022  Chapter 3 program included in this report. FY = fiscal year; MIGA = Multilateral Investment Guarantee Agency; S+ = satisfactory or better. MIGA projects rated at the extreme ends of the development outcome rating scale are few, with some distinguishing features, though they are not generalizable to the entire portfolio. The 13 projects with either excellent or unsatisfactory development outcome ratings are in similar sectors and Re- gions, but these two sets of projects have a few distinguishing features. Among projects rated highly on development outcome, early and consistent engage- ment with the guarantee holder throughout the life of the MIGA guarantee ensures buy-in and collaboration, especially on E&S aspects of the project. In three of five projects with excellent development outcome ratings, MIGA collaborated with other development finance institutions to enhance capacity building and promote corporate, industry, and transparency good practices. Information and knowledge sharing with other partners, including within the Bank Group, proved helpful in building E&S capacity and in promoting private sector development, especially in transition economies. Lessons from 34 8 projects rated unsatisfactory on development outcome included the impor- tance of ascertaining at appraisal the guarantee holder and project enterprise buy-in on every aspect of the project and throughout the effectiveness period of the MIGA guarantee. Early engagement and buy-in are especially important for complex projects such as those in the oil and gas subsectors and projects in countries with governance challenges. A lesson from another project rated un- satisfactory on development outcome notes the importance during appraisal of ensuring that complex projects are supported by an engineering, procure- ment, and construction company that knows how to implement MIGA’s E&S requirements. Engaging knowledgeable independent engineers or external experts can also help mitigate E&S and operational risks, a lesson from anoth- er project rated unsatisfactory on development outcome. All 8 projects rated unsatisfactory also have poor business performance because of the sponsor’s inexperience, limited financial resources to meet business and operational challenges, overoptimistic assumptions, or, in one financial sector project, de- pendence on the parent company for financial support because of asset quality issues. A common takeaway from these 8 projects is the importance of assess- ing at appraisal the risks that are likely to affect development outcomes from the aforementioned lessons. Development outcome ratings of projects in FCS and IDA countries also held steady, although the recent trend indicates a slight drop in IDA ratings. Of the 14 rated projects in FCS that were self-evaluated and validated between 2012 and 2021, 10 (71 percent) were rated satisfactory or better on develop- Independent Evaluation Group World Bank Group    35 ment outcome. In the same period, 27 of the 42 (64 percent) self-evaluated and validated MIGA projects in IDA countries were rated high on develop- ment outcomes. In relation to the six-year average trend, there has been a slight decline in development outcome ratings among projects in IDA coun- tries in the most recent period, 2016–21 (figure 3.11).  evelopment Outcome Ratings of MIGA Projects in FCS and Figure 3.11. D Non-FCS and IDA and Non-IDA Countries, 2012–21 100 Share of projects rated S+ (%) 80 60 40 20 0 FCS Non FCS IDA Non-IDA Country type 2012–17 2013–18 2014–19 2015–20 2016–21 Source: Independent Evaluation Group data. Note: Of the 108 evaluated MIGA projects in fiscal years 2012–20, 14 rated projects are in FCS and 42 are in IDA countries. Classification of projects by IDA and FCS used the World Bank Group List of Economies and Harmonized List of Fragile and Conflict-Affected Situations at the time of the MIGA contract effectiveness date. FCS = fragile and conflict-affected situations; IDA = International Development Results and Performance of the World Bank Group 2022  Chapter 3 Association; MIGA = Multilateral Investment Guarantee Agency; S+ = satisfactory or better. Development outcome ratings of projects in East Asia and the Pacific im- proved considerably in 2016–21 compared with 2015–20. In 2016–21, three of four evaluated projects in East Asia and the Pacific were rated satisfactory or better. Development outcome success rates of projects in Europe and Cen- tral Asia, Latin America and the Caribbean, and Africa held steady during the 2015–20 and 2016–21 periods. There were two evaluated projects in South Asia in 2012–20, and both have satisfactory or better ratings. Development outcome success ratings of projects in the Middle East and North Africa de- clined in 2016–21 from 2015–20, although the success rate is on par with the 75 percent satisfactory or better development outcome ratings of projects in East Asia and the Pacific in 2016–21. Six of eight (75 percent) evaluated projects in the Middle East and North Africa were also rated satisfactory or better on development outcome during the same period. Finance and Capital Markets projects have made substantial gains in their 36 development outcome ratings, from 58 percent in 2015–20 to 71 percent satisfactory or better ratings in 2016–21. Similarly, development outcome ratings of Agriculture, Manufacturing and General Services projects also showed considerable improvement in 2016–21 with the addition of 1 self-evaluated and validated project in 2021. Although the Energy and Extractive Industries sector has outperformed other MIGA sectors in previous years, its development outcome ratings have declined recently. Energy and Extractive Industries projects had higher average development outcome ratings than other sectors until 2014–19, when they declined to 80 percent from 86 percent in 2013–18 and subsequently to 67 percent in 2015–20 and 60 percent in 2016–21. Six of 10 self-evaluated and validated Energy and Extractive Industries projects in 2016–21 are rated satisfactory or better on development outcomes, compared with 8 of 12 projects in 2015–20. Recent 2016–21 development outcome results indicate that Infrastructure sector projects have a slightly higher share of satisfactory or better development outcome ratings (73 percent) than projects in other sectors (appendix E, figure E.22). Development outcome is closely associated with MIGA’s role and contribution but less so with MIGA’s work quality. Satisfactory or higher ratings on MIGA’s role and contribution outpace the ratings on development outcome and the quality of MIGA’s assessment, underwriting, and monitoring (figure 3.12). Trends in these three dimensions have held steady in earlier years until 2014–19. In 2016–21, MIGA’s role and contribution ratings Independent Evaluation Group World Bank Group    37 increased further to 88 percent from 84 percent in 2015–20, with the addition of two projects in 2021 that have satisfactory or higher ratings. High ratings on MIGA’s role and contribution are due to satisfactory or higher ratings in providing political risk mitigation (which enables provision of long-term financing and, in some cases, local currency financing when financing is not available from domestic or international financing sources), mobilizing finance, and increasing foreign direct investment flows. MIGA’s role and contribution also received high ratings for building capacity on E&S and improving corporate governance. Nearly all the 13 projects rated excellent on MIGA’s role and contribution got high marks for two contributions, which relates mostly to MIGA’s political risk mitigation role and improving projects’ E&S effects. Nine of 10 recently evaluated projects were rated satisfactory on MIGA’s role and contribution in improving E&S aspects of the projects. MIGA’s role and contribution in improving the projects’ E&S aspects (self-evaluated and validated between 2017 and 2021) began at the due diligence stage by work- ing with the MIGA guarantee holder and project enterprise in developing the Environmental and Social Action Plan. During project implementation, MIGA provided support and feedback to the guarantee holder and project en- terprise through its review of annual monitoring reports, periodic site visits, and preparation of back-to-office reports. One example is MIGA’s involve- ment in a financial intermediary project that helped the bank’s subsidiaries develop E&S policies and procedures. MIGA also assisted in establishing the bank’s Environmental and Social Management System. By contrast, the only project rated partly unsatisfactory on MIGA’s role and contribution in E&S indicated that MIGA’s role and contribution would have been more substan- tial if the occupational health safety risks had been adequately identified at due diligence and if proper actions had been included in the Environmental and Social Action Plan and in the contract of guarantee between MIGA and the guarantee holder. Results and Performance of the World Bank Group 2022  Chapter 3  evelopment Outcome, Multilateral Investment Guarantee Figure 3.12. D Agency’s Role and Contribution, and Work Quality Ratings, 2012–21 100 Share of projects rated S+ (%) 80 Development outcome 60 40 20 0 2012–17 2013–18 2014–19 2015–20 2016–21 Six-year rolling average by fiscal year of self-evaluation MIGA’s work quality MIGA’s role and contribution Source: Independent Evaluation Group data. 38 Note: MIGA = Multilateral Investment Guarantee Agency; S+ = satisfactory or better rating. In 3 of the 10 projects, MIGA and IFC collaborated in joint investment projects where IFC led the projects’ due diligence and supervision activities. Although MIGA’s role and contribution in E&S were somewhat limited in these joint investments, MIGA continued contributing through its annual monitoring reviews and site visits to ensure that the project enterprise and guarantee holder complied with the E&S conditions of MIGA’s longer-term contract of guarantee conditions, even after the IFC loans are repaid. Independent Evaluation Group World Bank Group    39 1  As of July 7, 2022, 809 Expanded Project Supervision Reports were completed (with com- pleted Validation Notes from IEG). One project has a rating of no opinion possible in develop- ment outcome and was excluded from the totals. 2  Small-size investment projects refer to investments with IFC total net commitment equal to or less than $4.3 million. Medium-size investment projects have IFC net commitment greater than $4.35 million and up to $35.74 million. Large-size investment projects have IFC net commitment greater than $35.74 million. 3  Historical performance of IFC investment projects indicates that equity-only investments and greenfield investments tend to have lower development outcome ratings than projects involving expansion or modernization (also known as brownfield) because such projects tend to be riskier. 4  Five projects have a rating of no opinion possible on IFC’s investment outcome. 5  In 2019–21, 33 of the 48 evaluated investment projects in Africa had an unsuccessful or lower rating on development outcome. Fifteen projects were in the Manufacturing, Agribusiness, and Services industry group; 13 were projects by the Financial Institutions Group; and 5 were from Infrastructure and Natural Resources. Seven of the less successful Manufacturing, Agribusiness, and Services projects are in health, education, and life sciences, Results and Performance of the World Bank Group 2022  Chapter 3 projects to which IFC provided equity investment. 6  For an analysis of IFC’s Risk-Sharing Facility, see World Bank (2022a), box 2.2. 7  Development outcome ratings of equity projects in International Development Association coun- tries outside of the Africa Region have improved substantially, from 33 percent in 2015–17 to 75 percent in 2018–20. In the same period, equity investments in International Development Association countries in Africa remained flat, moving from 14 percent to just 17 percent in the same period. 8  The result for the Central Asia and Türkiye region may not be statistically significant because of the few projects evaluated in 2019–21 (only 9). Between 2012 and 2021, there were only 47 evaluated investment projects in Central Asia and Türkiye. 9  Fifty-six of the 809 investment projects evaluated in calendar years 2012–21 are in FCS, and 753 projects in non-FCS. In 2019, 6 FCS projects were evaluated, and 2 projects were rated mostly successful or higher on development outcome. In 2020, of the 9 evaluated projects in FCS, 1 project was rated mostly successful or higher on development outcome. In 2021, of the 40 4 evaluated projects in FCS, 1 project was rated mostly successful or higher on development outcome. 10  Measurement and rating of IFC’s additionality began in 2015. Before this calendar year, the indicator of IFC’s role and contribution was used. IFC’s additionality is a broader concept than role and contribution. 11  IFC’s additionality is a subset of its role that is unique to IFC and that cannot be filled by the client or any commercial financier. IFC’s additionality in a project consists of two types: financial and nonfinancial additionality. IFC’s additionality in its investment projects is rated based on four criteria: excellent, satisfactory, partly unsatisfactory, or unsatisfactory. 12  Thirty percent of evaluated projects in FCS in 2012–20 (42 of 139 ex post additionalities) identified the presence of IFC’s support in the financial structuring of the investment as IFC’s financial additionality or contribution to the sponsor. Regarding nonfinancial IFC’s addition- ality, the introduction of new and better standards was identified as the most present among evaluated projects in FCS (31 of the 139 ex post additionalities, or 22 percent). The pattern is also similar for evaluated projects in non-FCS, where 60 percent of projects have received both financial and nonfinancial additionality from IFC. Financial structuring of the project is the most present IFC financial additionality in projects in non-FCS (32 percent), and the new and better standards were prevalent among IFC’s nonfinancial additionality (23 percent). An ongoing IEG evaluation of IFC’s additionality in middle-income countries, due in FY23, can provide greater and deeper insights about IFC’s additionality. 13  The assessment of the IFC’s investment outcome at the project level considers the quali- Independent Evaluation Group World Bank Group    41 tative observations of the performance of each investment associated with the project and data from IFC’s resource management system to gauge the likely contribution after consid- ering costs. If IFC made only a loan or only an equity investment in support of the evaluated project, then the overall investment outcome rating is the same as that for the loan or equity indicator, as applicable. However, if IFC made both loan and equity investments in support of the evaluated project, they are rated separately, and IFC’s investment outcome is rated on a four-point scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory. 14  A total of 185 IFC advisory services projects were selected for evaluation in FY19, FY20, and FY21. All 54 projects selected for evaluation in 2019 and 68 projects selected for 2020 were completed as of June 30, 2022. Of the 63 advisory services projects selected for evaluation in FY21, 32 were completed (52 percent) as of June 30, 2022. 15  Afghanistan and Pakistan are grouped with Middle East region in IFC’s regional classifica- tion as of September 2021. 16  The IEG-IFC joint study on the link between IFC advisory services projects’ work quality and development outcome found that weakness in project design and the monitoring and evaluation (M&E) framework in the evaluated IFC advisory services projects were the main shortcomings across all sectors. Project design was the most frequent driver of low development effective- ness ratings. 17  There are 8 projects in MIGA’s FY21 Project Evaluation Report program. Of these, 2 were self-evaluated and validated as of the RAP 2022 cut-off date of June 30, 2022. Another 2 self-evaluated projects were submitted to IEG for validation on June 30, 2022, too late to be included in the RAP 2022. The self-evaluations of the 4 remaining projects in the FY21 Project Evaluation Report program have not been submitted to IEG for validation as of the end of August 2022. 18  See Results and Performance of the World Bank Group 2021 (World Bank 2021b), which describes initiatives by MIGA to enhance its self-evaluation. In addition, MIGA’s ability to collect information and track development results is inherently limited by its role as a guarantee provider, with an indirect relationship with the project enterprise through the guarantee holder. It has sought to overcome these challenges, including by having teams Results and Performance of the World Bank Group 2022  Chapter 3 (prior to the COVID-19) visit nearly all projects that are subject to self-evaluation. Over the years, MIGA has made efforts to undertake self-evaluations of all guarantee projects— including canceled guarantees—which are then validated by IEG. MIGA has also been deferring evaluations for projects that are not yet fully operational and projects with political risks until those issues are resolved and its criteria for determining which projects are eligible for evaluation are more firm. 42 4 | Development Outcome and Other Ratings at the Country Program Level Highlights The share of country programs rated moderately satisfactory or higher on development outcome and good or higher on World Bank Group performance has risen since fiscal year (FY)14 and exceeded 70 percent since FY17, with the caveat that the ratings since FY19 are drawn from a small number of Completion and Learning Review (CLR) Reviews. The two Africa Regions (Eastern and Southern Africa and Western and Central Africa) were rated substantially lower than the other five Regions on development outcome at the country level. With regard to the World Bank Group performance rating, Eastern and Southern Africa and Latin America and the Caribbean ranked the lowest. With regard to the World Bank Group performance rating, lending group (International Development Association and International Bank for Reconstruction and Development) is the most significant discriminator: for International Bank for Reconstruction and Devel- opment countries, 68 percent were rated good or higher, compared with 56 percent of International Development Association countries. This suggests that a stronger performance by the Bank Group is called for in International Development Association countries. International Finance Corporation (IFC) and Multilateral Investment Guarantee Agency (MIGA) contributions were mainly in business environment and access to finance, which accounted for about half of the Country Partnership Framework objectives with IFC and MIGA contributions. More than 70 percent of Country Partnership 43  Framework objectives in which both IFC and MIGA contributed had ratings of mostly achieved or achieved. Greater attention to collaboration between the World Bank, IFC, and MIGA seems necessary, even though collaboration appears strong between IFC and MIGA. The success of the Bank Group’s Mobilizing Finance for Development agenda (formerly Maximizing Finance for Development) through its Cascade approach, as well as that of IFC 3.0 (creating markets) and MIGA’s strategy, hinges on greater and more effective collaboration with the World Bank. Monitoring such collaboration in country programs both in terms of coverage and depth is currently lacking in Completion and Learn- ing Reviews and Completion and Learning Review (CLR) Reviews but needs to be a priority. 44   Ratings Trends and Patterns There has been an upward trend in Bank Group country program ratings in both development outcome and World Bank Group performance since FY14 (figure 4.1). The share of development outcome ratings of moderately sat- isfactory or higher and World Bank Group performance ratings of good or higher has exceeded 70 percent since FY17. Among CLR Reviews covered in FY19, FY20, and FY21, development outcome was rated moderately satisfac- tory or higher for all countries, with the caveat that these ratings are drawn from a small number of CLR Reviews. Regarding the World Bank Group performance rating, the improvement is more gradual and remains below the development outcome rating. Seventeen of the 21 CLR Reviews that contained ratings and were covered in FY19, FY20, and FY21 rated World Bank Group performance good or higher—though the caveat about the small number of CLR Reviews holds here too.1 Figure 4.1. Country Program Ratings, Fiscal Years 2011–21 100 100 100 Development 100 outcome (MS+ %) 92 100 World Bank Group performance 80 74 80 (Good+ %) 69 71 79 66 66 67 66 76 72 Percentage (%) 70 60 63 64 64 60 61 99 97 98 98 Independent Evaluation Group World Bank Group    45 88 100 40 69 80 Number 49 60 20 27 40 16 20 6 2 0 0 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 Fiscal year covered by CLR Review CLR Reviews (no.) Source: Independent Evaluation Group data. Note: The data are reported with the smoothing approach adopted since World Bank (2020a), with the year referring to every year in the period reviewed by the CLR Reviews. Six Organisation of Eastern Caribbean States countries were reviewed together. CLR = Completion and Learning Review; MS+ = moderately satisfactory or higher. Country program ratings vary significantly among Regions (figure 4.2). For development outcome, South Asia, Europe and Central Asia, and Middle East and North Africa received high ratings at 89, 83, and 82 percent rated moder- ately satisfactory or higher, respectively, and Eastern and Southern Africa and Western and Central Africa had the lowest, with 49 percent rated moderately satisfactory or higher in both Regions. For World Bank Group performance, Europe and Central Asia outperformed other Regions, while Eastern and Southern Africa had the lowest rating. The development outcome rating is strongly correlated with income level, lending group, FCS status, and Country Policy and Institutional Assessment (CPIA) score (figure 4.3). IEG reviewed all countries’ latest CLR Review rat- ings and found that development outcome was rated moderately satisfactory or higher in all high-income countries, and for low-income countries, only 52 percent of ratings were in this outcome bracket. By disaggregating coun- tries into quartiles based on their CPIA scores, IEG also found that develop- ment outcome rating rises in step with the CPIA score. Seventy-nine percent of countries placing in the top quartile by CPIA score had CLR Review develop- ment outcome ratings of moderately satisfactory or higher; for countries in Results and Performance of the World Bank Group 2022  Chapter 4 the bottom quartile, the corresponding figure was 50 percent. It is notewor- thy that among FCS, 59 percent had development outcome ratings of moder- ately satisfactory or higher, compared with 66 percent of non-FCS. The gap in development outcome performance is wider when International Bank for Reconstruction and Development (IBRD) countries are compared with IDA countries. With respect to World Bank Group performance, lending group (IDA and IBRD) is the most significant discriminator: for IBRD countries, 68 per- cent rated good or higher, compared with 56 percent for IDA countries. This suggests that a stronger performance by the Bank Group is called for in IDA countries. The World Bank Group performance rating assesses the design and implementation quality of the country program, focusing on issues within the Bank Group’s control, such as the selection of CPF objectives, portfolio man- agement, results framework, risk identification and mitigation, attention to safeguards, fiduciary diligence, ownership and flexibility, internal cooperation, and coordination with other development partners (World Bank Group 2021a). 46 Given that factors beyond the Bank Group’s control could affect the develop- ment outcome rating, differences between the development outcome rating and World Bank Group performance rating are not surprising (figure 4.4). Figure 4.2. Country Program Ratings by Region, Fiscal Years 2011–21 a. Development outcome South Asia 89 Europe and Central Asia 83 Middle East and North Africa 82 Latin America and 75 the Caribbean East Asia and Pacific 69 Eastern and Southern Africa 49 Western and Central Africa 49 0 20 40 60 80 100 Share of CLR Reviews rated moderately satisfactory or higher (%) b. World Bank Group performance Europe and Central Asia 83 Independent Evaluation Group World Bank Group    47 Middle East and North Africa 76 East Asia and Pacific 71 South Asia 64 Western and Central Africa 63 Latin America and 51 the Caribbean Eastern and Southern Africa 44 0 20 40 60 80 100 Share of CLR Reviews rated good or higher (%) Source: Independent Evaluation Group data. Note: The data are reported with the smooth approach adopted since World Bank (2020a), with the year referring to every year in the period reviewed by IEG’s Completion and Learning Review (CLR) Review. Six Organisation of Eastern Caribbean States countries were reviewed together.  oderately Satisfactory or Higher Development Outcome Figure 4.3. M Ratings in the Country’s Latest Completion and Learning Review (CLR) Review a. Income level b. Country Policy and Institutional Assessment ratings c. Lending group d. Fragile and conflict-affected situation status Sources: Independent Evaluation Group data; World Bank Enterprise Data Catalog; World Bank Opera- Results and Performance of the World Bank Group 2022  Chapter 4 tions Policy and Country Services; World Bank Classification of Fragile and Conflict-Affected Situations. Note: Income level, lending group, Country Policy and Institutional Assessment ratings, and fragile and conflict-affected situations status are assigned based on the ending fiscal year of the latest Country Partnership Framework period that the Independent Evaluation Group reviewed. Six countries in the Organisation of Eastern Caribbean States were reviewed together. FCS = fragile and conflict-af- fected situations; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association. On development outcome ratings, the proportion of countries trending downward is smaller than the proportion trending upward, with the opposite found in World Bank Group performance ratings, based on a review of the latest two CLR Review ratings of countries with more than one CLR Review (figure 4.5). 48  evelopment Outcomes and Bank Group Performance Figure 4.4. D Ratings (number of Completion and Learning Review [CLR] Reviews) Source: Independent Evaluation Group data. Note: Six countries in the Organisation of Eastern Caribbean States were reviewed together. MS+ = mod- erately satisfactory or higher; MU−= moderately unsatisfactory or lower; WBG = World Bank Group.  hift in Development Outcome and World Bank Group Figure 4.5. S Performance Ratings in Countries’ Latest Two Completion and Learning Review (CLR) Reviews Development Outcome 16 56 28 World Bank Group Independent Evaluation Group World Bank Group    49 33 55 12 Performance 0 20 40 60 80 100 Shift (%) Downward No change Upward Source: Independent Evaluation Group data. Note: Six Organisation of Eastern Caribbean States countries were reviewed together. Analysis of World Bank Group Support and Country Partnership Framework Objectives The 2018 country engagement guidance states that “Bank Group support is aimed squarely at helping clients achieve CPF objectives, which in turn con- tribute to HLOs [high-level outcomes]” (World Bank Group 2018, 4). As such, the success of Bank Group support—both lending and ASA—during a CPF period depends critically on the achievement of each of the CPF objectives. This is reflected in the Bank Group’s CLR and IEG’s CLR Review ratings, with the development outcome rating directly linked to the extent to which the CPF objectives have been achieved (World Bank Group 2018, para. 78). This section explores the relationship among the timing, type, and performance of Bank Group interventions and the achievement of CPF objectives. When individual CPF objectives were considered over the 10-year period (FY13–22), nearly half were rated partially achieved or not achieved (based on 113 CLR Reviews for which this information was available). The analysis in the following section draws from 55 CLR Reviews between 2016 and 2022,2 which contain 593 CPF objectives. Each objective is assessed Results and Performance of the World Bank Group 2022  Chapter 4 as achieved, mostly achieved, partially achieved, not achieved, not rated, or not verified. Although the last three ratings have the same influence on a CPF’s overall development outcome rating, not achieved is considered separately in this analysis because it indicates that the Bank Group support failed to deliver the expected outcomes, while not rated and not verified are often a result of poor results monitoring, and thus IEG is not able to deter- mine the achievement of an objective. Excluding 12 objectives for which the CLR Reviews do not clearly identify the supporting Bank Group operations, the findings in the following section are thus based on Bank Group support for 581 CPF objectives. The observations presented point to areas for further exploration, not conclusions or lessons on what the Bank Group should do. Legacy Projects versus New Operations Half of the CPF objectives were supported by a mix of legacy and new oper- ations. In the 55 countries, about 14 percent of the CPF objectives were sup- ported exclusively by operations already active at the start of the CPF period, 50 and more than one-third of the objectives were underpinned entirely by new projects approved during the CPF period. For objectives that were supported by both old and new interventions, new projects dominated in nearly 80 per- cent of cases. DPFs figured more prominently in the Bank Group’s portfolio for the objectives that were supported entirely and mostly by new projects. Countries that had a large presence of mature projects in the portfolio but also included new operations that could better respond to new realities had a good record on achievement of CPF objectives (see appendix F, figure F.1). About 60 percent of the objectives fully supported by pre-CPF-approved Bank Group projects and 72 percent of the objectives primarily supported by pre-CPF-approved Bank Group projects were either fully achieved or mostly achieved, compared with 52 and 47 percent, respectively, of the objectives mostly and entirely supported by new projects launched during the CPF period. The shorter period of implementation for the new projects may have led to poorer delivery of the expected outcomes among the latter group, resulting in a higher share of partially achieved objectives. However, the CPF, as a forward-looking document, risks losing relevance if it does not anticipate upcoming change. Appendix G provides guidance on how to avoid this risk. One Bank Group Approach The World Bank supported 98 percent of all CPF objectives in the 55 countries, but IFC was involved in only 20 percent of them; four out of five objectives Independent Evaluation Group World Bank Group    51 were supported only by World Bank operations (see appendix F, table F.3), even though most CPFs are joint strategies for the World Bank, IFC, and MIGA (appendix H). The CLR Reviews frequently note that the CPF results frame- work fails to reflect IFC interventions adequately. In some cases, IFC provided an add-on complement to World Bank operations (for example, South Africa in 2021, The Gambia in 2018, and Bulgaria in 2016). In general, IFC and MIGA do not play a big role in public sector reforms, which explains why they were absent from Bank Group support for some CPF objectives. But their nonin- volvement in support of objectives concerning private sector–led growth can be considered a missed opportunity—which is even more striking, given that these objectives constitute a key pillar or focus area for most CPFs. Even for objectives supporting the business environment and private and fi- nancial sector development, where IFC has significant expertise and well-de- veloped tools, it still played a smaller role than the World Bank (table 4.1). The CLR Reviews note IFC activities in a wide range of areas. IFC is particu- larly active in supporting business environment, private and financial sector development, energy, agrifood, and health care services. Compared with the objectives in other areas, Bank Group support for these objectives involved IFC more often, yet IFC could contribute much more. For instance, CLR Reviews offer many examples of successful World Bank–IFC-MIGA collabo- ration in all aspects of energy generation, transmission, and distribution, in- cluding renewable energy development (for example, Nigeria in 2021, China in 2020, Kazakhstan in 2020, Myanmar in 2020, and Senegal in 2020). How- ever, the World Bank and IFC worked together (collaboratively or in parallel) in only 26 percent of the energy objectives, and the World Bank worked alone in two-thirds of the cases. The same is true for agrifood objectives, where the World Bank dominated the Bank Group’s sector support. Overall, CPF objectives that were supported by World Bank and IFC opera- tions together registered a slightly better record of achievement than those Results and Performance of the World Bank Group 2022  Chapter 4 supported by World Bank operations alone. Combined World Bank and IFC support translated into more mostly achieved and partially achieved CPF ob- jectives and fewer not achieved objectives but not more fully achieved objec- tives (see appendix F, figure F.2). In the five areas where IFC has significant experience and was most active, the objectives that were supported by both World Bank and IFC operations did not perform any better than those sup- ported by the World Bank alone (see appendix F, figure F.3). In fact, contrary to expectations, the World Bank and IFC seem to have derived more benefits from working together in areas other than those in which they collaborat- ed more extensively: only 19 percent of the objectives were achieved and 32 percent mostly achieved in the five areas of key IFC activities. The compa- rable figures in other areas were 34 percent achieved and 38 percent mostly achieved. The objectives supported by IFC alone seem to have performed very well, although the small number of observations (11) makes it difficult to generalize. 52  ountry Partnership Framework Objectives in Key Areas of Table 4.1. C International Finance Corporation Activities Business Environment, Private Sector Development, Agriculture Financial Sector Health Care and Food Share (%) Development Energy Services Share (%) Share (%) Share (%) (no.) (no.) (no.) (no.) Operations Total 46 96 47 34 World Bank + 15 33 45 47 12 26 2 6 IFC operations World Bank 31 67 45 47 31 66 32 94 operations only IFC operations n.a. n.a. 6 6 4 9 n.a. n.a. only Source: Independent Evaluation Group. Note: IFC = International Finance Corporation. The same projects often support the business environ- ment, private sector, and financial sector development at the same time through different components. The “Knowledge Bank” Half of the CPF objectives were supported by a combination of Bank Group Independent Evaluation Group World Bank Group    53 financing (World Bank lending and IFC investments) and knowledge (ASA and nonlending technical assistance) operations (see appendix F, ta- ble F.5). Financing underpinned more than 90 percent of all CPF objectives, but only 9 percent of the objectives relied solely on the Bank Group’s ana- lytical and technical support. Indeed, unlike Bank Group financing, which typically leads to measurable changes in some ways (for example, roads built or banks capitalized), Bank Group analytical work is often intended to inform rather than to deliver development outcomes directly. Advisory services and technical assistance, however, have a more discernible impact on the achievement of CPF objectives: they are less diffuse in their effect because they target specific CPF development outcomes. Combined Bank Group financing and knowledge support translated into more mostly achieved CPF objectives and fewer not achieved objectives, though not more fully achieved objectives (see appendix F, figure F.4). In the 55 countries, about 23 percent of the objectives were fully achieved, regardless of the knowledge content of the Bank Group interventions (that is, whether the support was financing only, knowledge only, or a mix of both). However, when Bank Group financing was combined with knowledge support, they had a higher incidence of delivering most of the expect- ed outcomes (33 percent) than did financing or knowledge alone (27 and 22 percent, respectively) and a lower tendency to fail completely (8 percent versus 17 and 14 percent, respectively). Furthermore, combined support seemed to support better results monitoring, reducing the share of objectives whose achievement could not be verified. Outcome of World Bank Operations A positive relationship exists between the achievement of CPF objectives and the outcome rating of the supporting operations, which is particularly nota- ble at the higher end. Consistent project ratings are available only for World Results and Performance of the World Bank Group 2022  Chapter 4 Bank lending operations. For the 522 objectives supported either partially or entirely by World Bank lending, the data confirm the expected: a CPF objec- tive is more likely to be achieved when there are more successful projects in the country program. This is especially true for fully achieved objectives (see appendix F, figure F.5). DPFs were associated with better performance than IPFs in outcome achievement at the project level, but their contribution to CPF objective achievement was mixed. IPFs formed the backbone of the World Bank’s lending programs. Nevertheless, of the 581 CPF objectives from the 55 countries, 125 were supported by at least one DPF.3 This included 21 objec- tives that were supported exclusively by DPFs, although for most objectives, DPFs made up less than half of the Bank Group portfolio. A heavy presence of policy operations was associated with more successful operations (project outcomes rated moderately satisfactory or higher) in the Bank Group’s sup- port program but not better achievement of CPF objectives (see appendix F, figure F.6). The 21 objectives supported entirely by DPFs are notable: they 54 performed exceedingly well, with 90 percent of the DPFs successfully imple- mented and 60 percent of the objectives were fully achieved. IFC and MIGA Contributions to Country Partnership Framework Objectives and Performance IFC and MIGA contributions were mainly in business environment and access to finance, which accounted for about half of the CPF objectives with IFC and MIGA contribution. Energy accounted for another 17 percent, and other infrastructure (almost equal shares for telecommunications, transport, and water and waste- water) accounted for another 10 percent. Agriculture, agribusiness, and rural development accounted for 11 percent, and the social sectors (divided almost equally between health and education) accounted for about 10 percent. Climate change accounted for most of the remaining objectives (table 4.2). IFC used both investments and advisory services to contribute to the objec- tives. Table 4.3 shows that IFC contributed to almost all the objectives with IFC and MIGA contribution. In about one-third of the objectives, IFC com- bined investments and advisory services. Advisory services may be provided to investment clients directly (for example, capacity building for lending to woman-owned small and medium enterprises). Accompanying advisory services support focuses on improving broader business environment con- Independent Evaluation Group World Bank Group    55 straints or financial infrastructure gaps (for example, microfinance institu- tion investment accompanied by support to credit bureaus). MIGA projects accounted for about 11 percent of the objectives with IFC and MIGA contributions. In all but one of these objectives, MIGA’s contribu- tion was accompanied by IFC investment or advisory services or both. This is partly due to MIGA’s leveraging of IFC’s more extensive global footprint. MIGA’s business development agreement with IFC since 2009 provides both marketing support (acting as a broker on behalf of MIGA) and sharing of due diligence (for example, environment and social assessments and oth- er information on common clients). MIGA provides value added by reducing noncommercial risks in joint projects with IFC, an example of jointness that has become increasingly important because of the Bank Group’s Mobilizing Finance for Development agenda (formerly Maximizing Finance for Develop- ment) and the Cascade approach. nternational Finance Corporation and Multilateral Investment Table 4.2. I Guarantee Agency Contributions by Country Partnership Framework Objective Share of Share Combined Share of IFC of MIGA IFC and MIGA Contribution Contribution Contribution by CPF by CPF by CPF CPF Objective Objective Objective CPF Objective Objectives Classification Classification Classification Classification (no.) (%) (%) (%) Business 49 26 10 26 environment Access to finance 44 24 10 24 Energy 32 17 45 17 Agriculture, 21 11 10 11 agribusiness, and Results and Performance of the World Bank Group 2022  Chapter 4 rural develop- ment Infrastructure 18 9 20 10 Health 9 5 5 5 Education 7 4 0 4 Other 6 4 0 3 Total 186 100 100 100 Source: Independent Evaluation Group. Note: CPF = Country Partnership Framework; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency. IFC support to access to finance objectives was mainly through investments in financial institutions. IFC investments accounted for 62 percent of the support, either as sole contributor (32 percent) or combined with IFC advi- sory services (30 percent). Investments in financial institutions accounted for a significant part of the IFC portfolio, in some cases supported by capac- ity-building programs, especially in microfinance. A significant part of the advisory services is in strengthening the financial infrastructure, including 56 credit information and collateral systems. nternational Finance Corporation and Multilateral Investment Guarantee Agency Support to Country Table 4.3. I Partnership Framework Objectives by Instrument Objectives Supported by Objectives Combined IFC Supported by Investments Objectives with Objectives Objectives Combined IFC Objectives and Advisory Combined IFC Supported by Supported by Investments Supported by Services and and MIGA Con� IFC Investment IFC Advisory and Advisory MIGA Guaran� MIGA Guaran� tributions Only Services Only Services tee Projects tee Projects Number 186 59 71 55 20 19 Distribution of IFC and n.a. 32 38 30 11 10 MIGA instruments as share of total objectives (%) Source: Independent Evaluation Group. Note: IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency. Independent Evaluation Group World Bank Group    57 IFC investments and MIGA guarantee projects were the main instruments supporting energy-related objectives. IFC investments accounted for almost 70 percent of the energy objectives with IFC and MIGA contributions, of which 25 percent were a combination of investments and advisory services. MIGA ac- counted for 28 percent of the energy objectives, all of which were also support- ed by IFC investments or advisory services or both. Many of the MIGA projects supported IFC investments (for example, Kribi Gas Power Project in Cameroon) or were enabled by IFC advisory services (for example, Myingyan Independent Power Producer Project in Myanmar). IFC investments and MIGA projects were also primary contributors to other Infrastructure-sector objectives in the CPF. IFC investments accounted for more than 70 percent of the infrastruc- ture objectives with IFC and MIGA contributions, which had 28 percent com- bined investments and advisory services support from IFC. MIGA contributed to more than 20 percent of the infrastructure objectives. IFC supported the Kenya-Uganda Railway, the EASSy cable regional project in East Africa/Kenya, and the Dakar Diamniadio Toll Road Project in Senegal. IFC also provided pub- lic-private partnership advisory support in several countries. IFC advisory services were the main instrument used to contribute to the Results and Performance of the World Bank Group 2022  Chapter 4 CPF objective of improving the country’s business environment. IFC sup- ported public sector institutions in developing and implementing invest- ment climate policies and reforms that focused on reducing the cost of doing business. About 85 percent of the business environment objectives with IFC and MIGA contribution included IFC advisory services either as the sole con- tributor (60 percent) or combined with IFC investments (25 percent). But of all the CPF objectives rated in the CLR Reviews, more than 55 percent were assessed as not achieved or partially achieved. Of the total rated objectives in low-income and FCS, about 60 percent were assessed as not achieved or partially achieved. About 75 percent of CPF objectives with combined IFC and MIGA contribu- tions were assessed as mostly achieved or achieved.4 In most CLR Reviews, the results of IFC and MIGA interventions were incorporated in indicators that contribute to the rating of CPF objectives, such as the number of small and medium enterprises with credits secured by moveable property, in- creased energy generation, and reduction of regulatory procedures. Other 58 objectives with IFC or MIGA contribution, such as in improving the business environment and in health and education, did not perform as well, with more than 63 and 50 percent rated as partially achieved or not achieved, respec- tively (table 4.4). The IFC investments and advisory services projects identified as contribut- ing to the CPF objectives were typically only a portion of IFC’s portfolio in the country. IFC’s active investments and advisory services projects in the country were described in the CLR Reviews and listed in the appendixes. However, the CLR Reviews did not discuss the relevance of the IFC portfolio to the CPFs. Additionally, the CLR Reviews (and, by extension, the CLRs) did not discuss and incorporate IFC country strategies and their links to the CPFs. The CLR Reviews assessed for this RAP exercise did not mention pri- vate finance mobilization, an important pillar of IFC’s and MIGA’s strategies.  erformance of Objectives with International Finance Table 4.4. P Corporation and Multilateral Investment Guarantee Agency Contributions Rated Objectives with Rated Objectives with Combined IFC and MIGA Combined IFC and MIGA Contributions Contributions in LICs and FCS Share of Share of not Share of Share of not achieved achieved achieved achieved CPF Objec- or mostly or partially or mostly or partially Independent Evaluation Group World Bank Group    59 tives Classifi- achieved achieved achieved achieved cation ratings (%) ratings (%) ratings (%) ratings (%) Business 37 63 44 56 environment Access to 61 39 63 37 finance Energy 72 28 89 11 Agri-rural 63 38 50 50 Infrastructure 53 47 83 17 Health and 55 45 NA NA education Source: Independent Evaluation Group. Note: CPF = Country Partnership Framework; FCS = fragile and conflict-affected situations; IFC = Inter- national Finance Corporation; LIC = low-income country; MIGA = Multilateral Investment Guarantee Agency. Although collaboration is strong between IFC and MIGA, there is scope for improving the quality of the Bank Group Internal Collaboration sec- tion of the CLR Reviews. The Bank Group’s Mobilizing Finance for De- velopment agenda, through its Cascade approach, is missing in the CLR Reviews assessed for this report. The success of IFC’s 3.0 (creating markets) and MIGA’s strategies hinge on greater and more effective collaboration with the World Bank. Improving the section on collaboration and assessing the coherence of the mix of various Bank Group instruments in the CLR Reviews can provide helpful feedback and lessons that can enhance intra–Bank Group collaboration in achieving HLOs. Results and Performance of the World Bank Group 2022  Chapter 4 60 1 The total number of CLR Reviews covering fiscal years 2019, 2020, and 2021 was 16, 6, and 2 respectively, including CLR Reviews without ratings.  2  See appendix F for more information about sample construction. 3  Program-for-Results projects supported 23 objectives (less than 0.1 percent). 4  The assessment refers to the Bank Group’s achievement of each of CPF objectives and not a rating of IFC and MIGA contribution in the CPF. Independent Evaluation Group World Bank Group    61 5 | World Bank Group Performance at the Country Program Level Highlights This Results and Performance of the World Bank Group found weaknesses in country program relevance with regard to selectivity (for example, Country Partnership Framework [CPF] objectives that were too numerous or too broad), adaptiveness (for example, insufficient preparedness to respond to changes in country conditions, government commitment, or Bank Group priorities), and realism (for example, operations overestimating implementation capacity and underestimating political challenges). Furthermore, CPFs and their results frameworks also often relied overwhelmingly on the lending portfolio and insufficiently integrated and leveraged advisory services and analytics and International Finance Corporation and Multilateral Investment Guarantee Agency support. Risks are identified well up front, especially regarding macro- economic risks and risks associated with external shocks, but they are not identified as well regarding implementation capacity and political economy risks. A timelier and ongoing reevaluation of risks, adaptation of pro- grams, and updating of results frameworks is needed, certainly at the time of the Performance and Learning Review. However, the reasons for adaptation are important, and potential trade-offs between adaptation and shifting goalposts need to be recognized and managed. Risk mitigation is often partial, and the Bank Group lacks proce- dures at the country level, such as scenario planning, for acknowl- edging risk that cannot be fully mitigated. 62   Completion and Learning Review (CLR) Reviews provide limited evidence on the quality of implementation support, including in middle-income countries, where challenges are emerging at the subnational level. The potential for a One Bank Group approach has not been fully realized, and the contributions of the International Finance Corporation and the Multilateral Investment Guarantee Agency have been captured poorly within CPFs. Partnership with donors is seen as a Bank Group strength, especially in low-income countries. 63  World Bank Group (2021c) and World Bank (2020a) pointed to weakness- es in the Bank Group’s approach to country programs.1 The Bank Group subsequently revised its country engagement guidance in July 2021 (World Bank Group 2021a). Box 5.1 summarizes the main enhancements in that new guidance. These enhancements did not yet apply to the country programs covered by this RAP. Instead, this RAP establishes the baseline against which the country program performance resulting from the new country engage- ment guidance can be assessed in the future. Appendix I discusses some aspects of the new guidance that can be strengthened.  nhancements in the Revised Country Engagement Guidance Box 5.1. E of July 2021 The World Bank Group revised its country engagement guidance, country engage- ment procedure, and the Systematic Country Diagnostic guidance to strengthen outcome orientation in the Bank Group’s approach to country engagement. The revisions were informed by ongoing discussions with the Committee on Development Effectiveness and the Independent Evaluation Group report on outcome orientation at the country level (about how to best capture Bank Group contributions to long-term Results and Performance of the World Bank Group 2022  Chapter 5 country outcomes). The revised country engagement guidance introduces high-level outcomes (HLOs) in Country Partnership Frameworks (CPFs). HLOs are defined as a sustained improve- ment in the well-being of the poorest and most vulnerable people—for example, their health, security, mobility, opportunity, livelihood, or standard of living. These HLOs are primarily drawn from a client country’s own development strategy and are meant to be aligned with the Sustainable Development Goals. HLOs are typically achieved over a time horizon that extends beyond a single Country Engagement Cycle, are set at a higher level than CPF objectives, and typically result from the combined effort of multiple partners. The revised Systematic Country Diagnostic guidance now includes guidance for teams to identify long-term development outcomes and HLOs that are critical to the achievement of the twin goals and to articulate constraints and opportunities for achieving them. (continued) 64  nhancements in the Revised Country Engagement Guidance of Box 5.1. E July 2021 (cont.) During CPF preparation, Bank Group teams, in consultation with clients, will select relevant HLOs and CPF objectives for their country programs. The updated guidance became effective on July 1, 2021, and applies to all country engagement products that have Concept Note reviews after the effective date. Source: World Bank Group 2021a. This chapter provides a qualitative analysis of the performance of the Bank Group at the country program level based on the RAP’s analysis of the CLR Reviews of the 50 randomly sampled countries.2 Box 5.2 describes the criteria used.  riteria for Examining the Performance of the World Bank Box 5.2. C Group at the Country Level Country Partnership Framework (CPF) Design and Implementation 1. Relevance of country program (for example, selectivity and framing of CPF objec- tives, choice of interventions, adaptiveness, and realism of program design) Independent Evaluation Group World Bank Group    65 2. Quality of results framework 3. Risk identification 4. Risk mitigation 5. Bank Group support to implementation (including technical assistance) 6. One Bank Group approach 7. Partnership with donors 8. Additional elements (support for twin goals, “knowledge bank,” dropped or canceled projects) (continued)  riteria for Examining the Performance of the World Bank Group Box 5.2. C at the Country Level (cont.) Line of Sight 1. From Bank Group support to CPF objectives/development outcome 2. From CPF objectives/development outcomes to high-level outcome (which are meant to be primarily drawn from a client country’s own development strategy and are meant to be aligned with the Sustainable Development Goals) Sources: Independent Evaluation Group; World Bank Group 2021a. Country Partnership Framework Design and Implementation Country Program Relevance This RAP’s definition of relevance includes a larger set of dimensions than is traditionally the case. Relevance includes, for example, selectivity and fram- Results and Performance of the World Bank Group 2022  Chapter 5 ing of CPF objectives, choice of interventions, adaptiveness, and realism of program design. Using this definition of relevance, this RAP’s analysis found that the CLR Reviews for just 21 of the 50 sampled countries pointed to good country program relevance (figure 5.1). 66  orld Bank Group Performance at the Country Program Level Figure 5.1. W Source: Independent Evaluation Group. Note: The figure shows the number of CLR Reviews. Criterion discussed means that information provided in the CLR Review addressed the specific criterion. Good performance on criterion means that infor- mation provided in the CLR Review addressed the specific criterion. CLR = Completion and Learning Review. Framing of Country Partnership Framework Objectives Poorly framed CPF objectives include those that are excessively broad and monitorable only infrequently or at the countrywide level (for example, Independent Evaluation Group World Bank Group    67 increasing rural incomes). Other poorly framed CPF objectives are those that are excessively narrow, often project related and output based (for example, doubling the number of rural microloans). Some objectives are defined in ways that fail to address the highest needs in that area. The Sri Lanka CPF (FY13–16) had 18 objectives, some of which could have been combined. For example, objective 7 (increased connectivity) focused exclusively on road connections, thereby duplicating objective 12 (improved quality and sustainability of roads). In the Tajikistan CPF (FY15–18), the results framework set appropriate targets and baselines, but objectives were framed poorly, providing little support for implementation and results monitoring. According to the CLR Review, most, if not all, of the objectives were pitched either at the project level (for example, objectives 8, 11, and 19) or in the form of indicators (for example, objectives 1, 2, 3, 5, 6, 7, 10, 12, 13, 14, 16, 17, 20, and 21). In the São Tomé and Príncipe CPF (FY06–09), for instance, two outputs (number of donor meetings and a new development strategy submitted to the National Assembly) were the stated CPF objectives. In Senegal, a 2014–15 enterprise survey found that tax rates and tax admin- istration were the primary constraints for firms, but the CPF objective to enhance the investment climate did not reflect this. Choice of Interventions As indicated by the CLR Reviews that the RAP examined, sometimes the Bank Group’s choice of interventions was not adequate to achieve the stated program objectives. In the São Tomé and Príncipe CPF (FY06–09), a microcredit feasibility study was not a sufficient intervention for strengthening financial intermediation and broadening access to credit. In the Sri Lanka CPF, the activities proposed by the Bank Group to enhance accountability and transparency of public funds alone were not adequate support for the government’s development goal of increasing fiscal space and increasing efficiency of public spending by increasing revenue and lowering expenditures. And in the Mauritius CPF (FY07–15), 3 of 21 objectives were not sufficiently supported by lending or nonlending activities. Results and Performance of the World Bank Group 2022  Chapter 5 The Brazil CPE pointed to the need for Bank Group support to be more cat- alytic, meaning that programs and projects should have an impact extend- ing beyond any one intervention. Examples of a catalytic response include support for reforms that create enabling environments and incentives for other actors, activities to enhance demonstration effects and replicate pos- itive results, and engagements that leverage the Bank Group’s knowledge base and its convening role to facilitate cross-sectoral dialogue. However, the 2017 CLR Review noted that the catalytic effect was blunted somewhat by the sheer size of the country, which led the Bank Group to spread its resourc- es too thinly across 20 subnational borrowers plus the federal government. In Mexico, the CPE (World Bank 2018) found that the Bank Group needed to engage more fully with lagging regions, a recommendation that applies to all middle-income countries with large regional income disparities and pockets of extreme poverty. Although there may be obstacles to subnational lending, 68 there is often scope for engaging effectively through advisory work and more effective joint programing among IBRD, IFC, and MIGA. Selectivity of Objectives When programs have too many objectives, implementation may be com- promised. In the Kosovo CPF, the program had 26 outcomes and 31 indi- cators and was far too ambitious relative to the size of the IDA envelope. Objective 5 (promoting sustainable employment and social inclusion) was too compound in nature, lacking the focus needed to guide interventions. In Madagascar, the number of CPF objectives and outcomes needed to be reduced when projects were canceled, but no adjustment was made. In the Romania CPF, the number of focus areas remained unchanged throughout the program, but the number of objectives increased from 7 to 14, with the number of associated indicators rising from 16 to 43. The Argentina CPF (FY15–18) included a compound four-in-one objective to increase access to electricity, safe drinking water, housing and sanitation supported by a wide range of interventions. Three of four areas were not achieved under this objec- tive, with only a narrow housing pilot achieved during CPF implementation. The Rwanda CPF is a good example of selectivity. With just 12 well-defined objectives, the program was sufficiently selective and played to the Bank Group’s strengths in supporting infrastructure, agriculture, and governance. Independent Evaluation Group World Bank Group    69 In general, CPFs with too many objectives (typically more than 12 objectives or with multiple compound objectives) tended toward poorer development outcomes than CPFs that were more selective. Factoring in Flexibility and Adaptation An a priori assessment of government commitment to the planned Bank Group program is an essential aspect of CPF design. Nonetheless, during CPF implementation, government commitment can change, and governments can also change. Fiscal space and debt burdens evolve, affecting attitudes toward planned Bank Group lending or choice of lending instruments. When circumstances change, Bank Group teams need to respond flexibly and adapt appropriately, adjusting the CPF program and the underlying results frame- work as necessary. Such adaptation may not be without its challenges given that multiple stakeholders may need to be involved within the client country. Yet adaptation is critical for maintaining the relevance of the country program. Furthermore, at the country program level, potential trade-offs between adap- tation and shifting goalposts need to be recognized and managed. This RAP’s review of 50 sampled countries’ CLR Reviews suggests two inter- related problems: failure to adjust the program sufficiently and in a timely manner to respond to changing circumstances, and failure to revise the CPF results framework to keep pace with changes in the Bank Group program. The first problem reduces the program’s relevance and effectiveness. The second problem leads to a missed opportunity to report the Bank Group con- tribution stemming from changes to the program and for learning. In the Bhutan Country Partnership Strategy (CPS), the government ultimate- ly decided against planned investment projects and P4R operations, opting instead for a smaller number of policy-based operations. In value terms, only about half of the planned IDA lending program was delivered through policy-based operations. The shift to quick-disbursing policy loans, without more traditional projects, weakened efforts at longer-term institutional ca- pacity building and state modernization. The turn away from lending shifted Results and Performance of the World Bank Group 2022  Chapter 5 Bank Group efforts into a large volume of ASA (45 products delivered during the CPF period versus only 4 loans), plus trust-funded technical assistance. The high volume of ASA and technical assistance only partially compensated for the longer-term capacity building designed into project lending and chal- lenged absorptive capacity—the CLR Review in Bhutan described the ASA program as being too large and dispersed among sectors, given the coun- try’s limited absorptive capacity. Furthermore, the shift toward policy-based operations and ASA was not fully addressed at the Mid-Term Review of the CPF, with only marginal adjustment of the initial objectives and indicators that were rooted in the planned investment lending. As a result, 4 of the 7 program objectives were only partially achieved at the end of the CPF period. The Bolivia CPF is an example of where the lending program that support- ed 16 CPF objectives was only half delivered. Expected CPF development outcomes and indicators were not adjusted sufficiently at midterm to compensate for the lending shortfall and lack of projects in key areas of the 70 Bank Group program. As a result, 8 of the 16 objectives were not achieved, partially achieved, or not verifiable at the end of the CPF period, and the overall development outcome was deemed moderately unsatisfactory. Quality of Results Frameworks The performance of the Bank Group was weakest in the quality of the re- sults frameworks underpinning the CPFs, as also indicated by World Bank (2020b). In many results frameworks, the objectives were poorly articulated (too vague, too broad, or too narrowly project related) and not accompanied by adequate indicators that could assess the achievement of these objectives and provide a link to the program’s interventions. This RAP’s analysis found that the CLR Reviews for 4 of the 50 sampled countries had a good results framework (figure 5.1). A significant weakness in CPF results frameworks was the inappropriate choice of indicators. Failings included (i) indicators that were nationwide in scope (too broad to assess the Bank Group’s contribution); (ii) indicators that could not be easily quantified; (iii) indicators that were irrelevant for measuring progress toward the desired outcome; (iv) indicators that mea- sured inputs, outputs, or intermediate outcomes rather than the relevant program outcome; (v) indicators that were not available or not verifiable at ex post evaluation of the CPF; (vi) indicators that were not generated by Bank Group programs if not otherwise done by government; (vii) indicators that had never been tracked by the country; and (viii) importantly, indica- tors reported individually when a set of indicators needed to be considered Independent Evaluation Group World Bank Group    71 together to determine the achievement of a CPF objective. In the Senegal CPF, for instance, one indicator was the increase in commer- cial cases tried by courts—not necessarily the best measure of progress to- ward the objective of an improved investment climate. Milestones or output measures are often inappropriately substituted for outcome measures. The South Africa CPF is a case in point: the results framework contained 73 mile- stones and outputs but few outcome indicators with baselines and targets. In the Ukraine CPF, there were no indicators to track the contribution of project results to country outcomes such as control of corruption, business environ- ment, agricultural productivity, access to finance, and financial stability. In the Bhutan CPS, IEG found that for four of the seven objectives, the chosen indicators were not relevant to measure progress toward the stated objec- tive. Indicators on revenue generation and the size of fiscal deficit were an insufficient measure of progress toward the objective of strengthening fiscal efficiency. Indicators on the volume of foreign direct investment were poorly defined and not directly relevant to the objective of improving the regulatory framework. In the Brazil CPF, the indicators for early childhood, primary and secondary education, and for access to water and sanitation, were all nationwide in scope and thus too broad to capture the contributions made by Bank Group inter- ventions. Also, the chosen agriculture sector indicators (number of producers supported and number of states promoting climate-smart agriculture) were not relevant for monitoring progress toward the stated objective of enhanc- ing market access and boosting small farmers’ adoption of climate-smart farming practices. Risk Identification Handled Well This RAP’s analysis found that CLR Reviews for 32 of the 50 sampled Results and Performance of the World Bank Group 2022  Chapter 5 countries show good risk identification (figure 5.1). In Senegal, limited gov- ernment capacity was correctly identified as a major impediment to program implementation. Institutional capacity constraints were also well anticipated in the CPFs for the Philippines, Rwanda, and São Tomé and Príncipe. Most CPFs did a good job of identifying risks, especially macroeconomic risks and risks associated with external shocks, but they were less adept at assess- ing risks from limited implementation capacity and lack of local commit- ment and were overoptimistic in assessing political risks.3 These risks can be particularly detrimental to achieving policy reform in a country and should also be addressed at the country program level rather than just at the project or other levels. Uganda is a case in point. Eight of the 12 objectives of the CPF (FY11–15) were either partially achieved or not achieved, suggesting that the country program design was based on an overoptimistic reading of Uganda’s insti- tutional capacity. In Sri Lanka, the CPF overestimated the government’s 72 willingness to carry out policy reforms. The World Bank underestimated the political economy limitations that shaped the Marshall Islands’s poor com- mitment to reforming key areas in the program, such as in the energy and information and communication technology sectors. The CPF program in Montenegro missed the risk of insufficient government ownership, failing to anticipate the corresponding obstacles to implementing reforms in areas such as environmental and public health management. In Pakistan, the World Bank collaborated with the International Monetary Fund (IMF) on the Stand- By Arrangement that closed in September 2011, leading the policy dialogue on tax administration, electricity, and social protection issues. However, it took more than two years to reach agreement on a new IMF Extended Fund Facility Arrangement because the political risks identified at the outset did materialize, and the Stand-By Arrangement was already off track at the time of closing. The program in Romania underestimated the risks to planned lending from changes in government and administrative weaknesses. It also underestimated the extent to which the government’s focus on the European Union (EU) and its access to EU grant funds would influence the scope for Bank Group intervention. Risk is constantly changing, but the Bank Group does not systematically update its CPF risk analysis on a timely basis. In many cases, IEG found that when country circumstances changed significantly for political, economic, or other reasons, a Performance and Learning Review (PLR) was either not Independent Evaluation Group World Bank Group    73 issued when it seemed warranted or the program was not adequately adjust- ed at midterm. In Senegal, the World Bank, faced with new risks, did not use the opportunity presented at the PLR stage to revise the overambitious set of 15 objectives and 27 indicators in the CPS. In Uganda, emerging governance and devel- opment risks led to changes in the Bank Group’s portfolio but not in the Country Assistance Strategy objectives. Although the Ukraine CPS objectives remained relevant throughout the program period, a PLR should have been completed during the time period of the program (FY14–16) as noted by the CLR Review, given the changes in the political and economic environment, the increased size and different instruments of the World Bank program, and new risks faced by the Bank Group. By contrast, the Bank Group responded flexibly in São Tomé and Príncipe, adapting the program to the uncertainties related to potential inflow of oil revenues and the government’s renewed emphasis on eliminating constraints to growth. Risk Mitigation Handled Less Well World Bank teams were adept at identifying risks, but they were less success- ful in mitigating them. This RAP’s analysis found that CLR Reviews for 48 countries discussed risk mitigation, with just 18 of those showing good risk mitigation (figure 5.1).4 The Bank Group relies on the Mid-Term Review (the PLR) to adjust the CPS to a changing risk profile. The Bank Group’s PLR guidance states that a PLR should be undertaken every two years or at the CPF’s midpoint, although there is some flexibility in the guidance to consider relevant country circum- stances in deciding on the frequency and duration of the PLR (World Bank Group 2021a). However, this flexibility is not always used. For example, when government commitment to reform slackens, the Bank Group needs to adjust its pipeline, especially for policy-based operations, but the Bank Group may fail to act or act in a timely fashion. Results and Performance of the World Bank Group 2022  Chapter 5 ASA work on mitigation measures needs to be responsive and timely. In the Tajikistan CPF implementation, ASA on the political economy drivers of policy reform came too late in the program to help shape mitigation measures. Similar delays and consequences occurred in Sri Lanka: the revenue inci- dence analysis, the enhancing competitiveness note, and the Public Expen- diture Review were delivered too late to mitigate the fiscal risks they were intended to address. CPFs lack procedures for acknowledging known risks that cannot be fully mitigated and proposing appropriate program adjustments as these risks materialize. The response to crises as known risks materialize is often ad hoc and not adequately informed by how the Bank Group had responded across countries facing similar circumstances (appendix J). The programs in Burundi and Myanmar illustrate this shortcoming. The Burundi CPF (FY13– 16) is an example of an FCS country where the Bank Group was unable to fully mitigate the effects of instability and violence resulting from the 2015 74 political crisis, even when they were clearly identified as a risk. If risk cannot be fully mitigated, then the Bank Group needs to think through responses and scenarios carefully. This requires thoughtful political economy analysis as a background to CPS design. The CLR also argues that continued World Bank engagement during a political crisis can enhance the relationship of trust and the World Bank’s capacity to conduct dialogue on critical develop- ment issues. Staying appropriately engaged rather than vacillating between engagement and disengagement is essential to maintaining influence on key issues related to poverty and social welfare during times of crisis. In Myanmar’s portfolio, risks and capacity constraints are higher than in the average Bank Group–FCS client country, calling for greater-than-normal selectivity and more emphasis on capacity building. The main risk that mate- rialized was ethnic conflict (the stalling of the peace process in 2016) and re- newed violence in the Kachin, Rakhine, and Shan states. In the Myanmar CPF (FY15–19), the treatment of social inclusion and administrative capacity was uneven, undercutting efforts to respond to conflicts. Portfolio risks increased as the number of new projects increased from 9 in 2016 to 13 in 2018. Timely political economy analysis is needed to inform the CPF design pro- cess. Program design needs to candidly acknowledge the limits of the Bank Group’s ability to mitigate these risks and consider in advance possible ad- justments to the Bank Group program as risks materialize. This lesson is also emphasized in the Tunisia CPE, which argued for Independent Evaluation Group World Bank Group    75 greater adaptability to evolving risks, particularly political economy risks. It noted that the use of multisector development policy operations, underpinned by sound analysis, helped focus support, reinforce coordina- tion across donors, and increase responsiveness to the government’s needs. But improved streamlining and timing of measures would have been useful, particularly by mid-2012, when the political context became more volatile. The World Bank may have overestimated the government’s commitment to reform. The CPE recommended that program design acknowledge political economy risks and provide for mitigation. It also suggested that the World Bank could have tried to build public support for reforms in the interim— before political order was restored—using its rich analytic work to better inform and build the capacity of such groups as trade unions, think tanks, civil society organizations, and parliament (World Bank 2014c). During program implementation, the Bank Group needs to modify its am- bitions and adjust its focus and timetable when risks materialize. In Poland, when most of the planned investment lending did not materialize (partly because EU grant funding substituted for World Bank loans), additional ASA were commissioned as an alternative to help achieve the stated development outcomes. The results framework was not adjusted to reflect this change. In Jamaica, the government changed midway through the program. The new administration was more supportive of the reforms that the World Bank, the IMF, and other partners advocated. The CPS was adjusted accordingly, although institutional and capacity constraints argued for a smaller program than the one approved later. The CPE for resource-rich developing countries emphasized the need for risk mitigation in the face of volatile commodity prices. It recommended that the World Bank and the IMF maintain a dialogue on macroeconomic and fiscal policies and help countries build fiscal buffers while prices were high, remaining prepared to offer exceptional budget support in the event of a downturn. Implementation Capacity and Implementation Results and Performance of the World Bank Group 2022  Chapter 5 Support—A Relatively Neglected Area for Relieving Capacity Constraints, Including at the Subnational Level CLR Reviews were weakest in providing evidence of Bank Group support for implementation. In two-thirds of sampled countries, it was not possible to assess the quality of support for implementation because of a paucity of evidence. Of the CLR Reviews for the 17 countries that provided evidence on this criterion, only 11 countries had good implementation capacity and implementation support (figure 5.1). Haiti (CPF FY09–14) is a good example of strong implementation support in a low-income and FCS country. The World Bank was quick to respond to the 2010 earthquake with nonlending technical assistance, timely adjustments to the World Bank program, a damage assessment, and additional commitments for recovery and reconstruction. The Haiti CPF and Interim Strategy Notes (ISNs) before and after the devastating 2010 earthquake are a good example of strong implementation support by the Bank Group team. Especially in 76 the aftermath of the earthquake, the Bank Group complemented its in- creased financial support with a high degree of technical assistance. Twen- ty-eight of 31 ASA tasks were technical assistance support supplemented by trust-funded technical assistance. This helped Haiti maintain strong dis- bursement rates and good implementation of various emergency operations in the aftermath of the earthquake, when institutional capacity was at its weakest. Although capacity constraints tend to be better recognized within low-in- come and FCS contexts, the Bank Group also faces emerging challenges in middle-income countries. The need to support implementation capac- ity is particularly acute for subnational governments in middle-income countries aiming to decentralize and deconcentrate government. These middle-income clients often will not borrow on IBRD terms to support soft interventions such as capacity building and technical assistance, further complicating the design of lending operations at the subnational level. Bank Group support for building subnational government capacity was insufficient in the Brazil program, for example. The Bank Group respond- ed to the government’s need for this type of support, but interventions were spread too thinly, covering 20 subnational borrowers plus the federal government. Portfolio performance lagged regional and World Bank–wide averages as the share of subnational lending in the overall portfolio rose. Implementation problems were most acute in sectors and programs fo- Independent Evaluation Group World Bank Group    77 cused at the subnational level, and many CPF indicators defined at the subnational level were not verifiable or not achieved. Two lessons emerge from this example: Innovative efforts are needed to identify financing for capacity building at subnational levels, and a good dialogue with and com- mitment from national government is necessary to work effectively at sub- national levels. In Nigeria, the World Bank used four P4Rs at the state level to support institutional changes in fiscal governance, education, health, and growth. Programs under the same project worked independently as separate operations requiring 170 project implementation units—more than half the number of World Bank–supported project implementation units in the Sub-Saharan Africa Region. Although the World Bank pro- moted state coordination mechanisms through annual country portfolio performance reviews, further progress will require helping to reduce the number of project implementation units and facilitating synergies. The One Bank Group Approach—Still a Work in Progress Despite the implementation of several initiatives since IEG’s 2017 Learning Engagement, which reviewed two decades of experience with the One Bank Group approach (World Bank 2017b), the One Bank Group approach still remains a work in progress.5 The contributions of IFC and MIGA have been poorly leveraged within CPFs. This RAP’s analysis found that CLR Reviews for 45 countries discussed the One Bank Group approach criterion, but just 18 countries had good performance on this criterion (figure 5.1). In Mexico, IEG found that the World Bank, IFC, and MIGA could work together more effectively to crowd in other sources of private and official finance for greater development impact. In Pakistan, the World Bank and IFC agreed to cooperate in several areas, but synergies were hard to attain within the Bank Group because parallel financing between IFC and the World Bank proved problematic due to different timelines and operating procedures. Collaboration between the World Bank and IFC was slow but Results and Performance of the World Bank Group 2022  Chapter 5 improving. In Nicaragua, the joint IDA-IFC work that was planned in edu- cation, health services, and financial innovation failed to materialize, and the World Bank and IFC proceeded on parallel but separate tracks. In Tajik- istan, the World Bank and IFC worked together on reforms to the business environment—IFC through advisory services and the World Bank through the Private Sector Competitiveness Project. It was envisioned that a series of development policy operations would help remove barriers to private sector development, opening the way for IFC to invest between $10 million and $20 million per year. But the development policy operation series was dropped due to a difficult macroeconomic environment, and the One Bank Group approach’s potential was not realized. Bank Group strategies for the resource-rich developing countries failed to convey how the World Bank and IFC might jointly promote much-needed economic diversification. The expected dividends from cooperation also failed to materialize in Arme- nia, Bhutan, and Brazil. World Bank teams need to calibrate the One Bank Group approach to define meaningful collaboration in countries where the 78 policy environment is not conducive to private sector investment or expected reforms fail to materialize. There are two related issues worth highlighting about a One Bank Group approach. The first is that collaboration and coordination across the Bank Group in the field is still a work in progress. The second is that the CPF and its accompanying results framework often fail to capture existing collabora- tion, coordination, and contribution by the institution’s private sector arms. This is partly a weakness in the initial CPF’s design and partly a reflection of a failure to capture unplanned IFC and MIGA opportunities that arise during CPF implementation. The Côte d’Ivoire CPF (FY10–14) is an example of where the results framework underreports on both IFC and MIGA, and the PLR does not adequately adjust to reflect an upswing in IFC and MIGA activ- ity. IFC made 20 investments during the CPF period, representing a tenfold increase in the value of the IFC portfolio, and MIGA made four guarantees consistent with corporate efforts to engage more in FCS. Yet, as reported in the CLR Review, the results matrix says little about IFC activities and the CLR does not discuss the IFC portfolio. In the Uganda CPF, the results framework facilitated the monitoring of prog- ress in addressing inefficiencies in land registration through World Bank in- terventions but overlooked the equally important contribution of IFC’s work on simplification of licensing and online filing for taxpayers, including small Independent Evaluation Group World Bank Group    79 and medium enterprises. The complementarity of these interventions in support of the business environment was not reflected adequately. Likewise, in Sri Lanka, IFC’s development outcome could have been better reflected in the results framework if appropriate indicators had been identified. A simi- lar oversight was manifest in the Montenegro program. In the Pakistan CPS results framework, many indicators lacked benchmarks, particularly those related to IFC, because IFC contributions were not adequately targeted when the strategy was prepared. As recognized in the completion report, although the results framework covered IFC activities as part of the array of CPS im- plementation instruments, IFC input into the formulation of outcomes and milestones was not evident. Three good examples of the One Bank Group approach hold lessons for the future. Senegal is an example of superior synergies across the Bank Group. Cooperation among the World Bank, IFC, and MIGA helped push complex reforms through in the energy sector, improve the investment climate, and strengthen financial infrastructure. The Dakar-Diamniadio Toll Road Proj- ect and the Scaling Solar Program were other instances in which the One Bank Group approach succeeded. The toll road project was the first of its kind in West Africa to involve a public-private partnership sponsored by the Bank Group. Uganda also showcased the potential of the unified approach. Support to the Bujagali Hydropower Project was provided through an IDA partial risk guarantee, an IFC loan, and a 20-year MIGA guarantee. IFC and MIGA also cooperated well in supporting Umeme, the electricity distribution company. Argentina is another example of capturing synergies across the Bank Group effectively. The Mobilizing Finance for Development approach was used to help develop a market for renewable energy. All three Bank Group institutions contributed to the program with several joint or com- plementary activities. Operationalizing this approach and replicating it for other relevant programs in Argentina requires systematic coordination among the Bank Group institutions and with counterparts. But according to the CLR, the prerequisites for a successful leveraging of private sector financing for renewable energy need to be spelled out so that others may Results and Performance of the World Bank Group 2022  Chapter 5 replicate Argentina’s success. Overall, IFC and MIGA need to be integrated more fully in the design and implementation of the Bank Group’s overall program. Additionally, their contribution needs to be made explicit in the results framework at the initial design stage and adjusted at the PLR stage so that the effectiveness of their support for development outcomes can be assessed properly (appendix K). Partnering for Results—Promising Progress The Bank Group performed well on partnering for results. This RAP’s analysis found that CLR Reviews for 47 countries discussed it, of which 31 countries showed good partnering for results (figure 5.1). The Bank Group has coordinated its work effectively with other development partners in most countries and has played a central role in donor coordina- tion in other countries. In Uganda, for instance, the Bank Group has served 80 as both permanent co-chair of the former Joint Budget Support Framework (which included the 12 budget support development partners) and chair of the Local Development Partners’ Group. In Rwanda, the World Bank is also well integrated into the donor coordination system, which operates through quarterly and annual meetings, including sector working groups and de- velopment partner consultative groups. Likewise, in Tajikistan, the World Bank chairs the Development Coordination Council, which comprises 13 working groups and 29 development partners. And in Ukraine, the Bank Group coordinated its program and activities with many development part- ners, including the IMF, the EU, the United States Agency for International Development, and the United Nations. Donor coordination has helped enhance the credibility and selectivity of World Bank programs (Senegal); helped reconstruction and reconciliation efforts in Mindanao Island (the Philippines); helped support reforms for a new government as part of a larger international package including World Bank policy-based operations (Ukraine); helped support two state-owned banks prepare for privatization (Serbia); brought in other donors in ener- gy (Uganda); and resulted in jointly agreed procedures on procurement, financial management, safeguards, and project implementation support (Tajikistan). This is a solid record of achievement, but within this are many examples where donor partnership failed to materialize or was ineffective. In Afghanistan, for example, contradictory advice and competing donor pro- grams compounded conflicting views among different government agencies, Independent Evaluation Group World Bank Group    81 leading to a “missing middle” in the absence of agreement on subnational governance, as noted in IEG’s 2013 CPE for Afghanistan. Additional Elements of World Bank Group Performance Support for the Twin Goals—Greater Attention Needed, Especially in the Face of Shocks Changing country circumstances and Bank Group responses can produce un- intended consequences that may undermine the twin goals. In implementing the Uganda CPF (FY11–15), a decision was made to cancel budget support that had been a regular feature of the program for many years. The cancella- tion reflected concerns about transparency, accountability, and corruption. Pro-poor spending by the government subsequently contracted, squeez- ing social services in the most vulnerable areas. The CLR Review suggests that the Bank Group could have made more effort to protect poor people in Uganda without the budget support through better dialogue on domes- tic resource mobilization and pro-poor spending or through compensating mechanisms within its own portfolio and pipeline. Specifically, the CLR Re- view advocated that in the event of a resumption of policy-based lending, the World Bank should assess the pro-poor nature of public expenditure care- fully and ensure that the macrofiscal framework that it supports includes measures to raise Uganda’s low revenue-to–gross domestic product ratio and does not penalize pro-poor spending. In Ukraine, the World Bank program was adjusted to protect poor people after utility tariff increases. But despite appropriate support from investment projects and policy-based operations, targeting accuracy declined because indicators on health and education expenditures were omitted. Similar problems arose in Sri Lanka, where the CPF objective to reduce the prevalence of malnutrition received too little support from the Bank Group to have an impact on the government’s goal of reducing the malnutrition rate and the country program’s objective in this regard was not achieved. These examples show that good intentions must be Results and Performance of the World Bank Group 2022  Chapter 5 accompanied by good design of measures intended to protect vulnerable and poor people from negative impacts associated with changing country cir- cumstances or essential reforms. The Haiti CPF and ISNs before and after the 2010 earthquake are a good example of the Bank Group strengthening its support for the twin goals in a crisis. In the aftermath of the 2010 earthquake, the Bank Group restructured its existing portfolio for immediate relief and recovery efforts. Country cir- cumstances changed so much and the portfolio restructuring was so exten- sive that it warranted preparation of a new country strategy in the form of two successive ISNs. These ISNs superseded the uncompleted CPF, for which most of the objectives were never achieved. Instead, the ISNs doubled down on reaching poor people and most disaster-affected populations with basic infrastructure and social services, including housing and education. 82 Realizing the “Knowledge Bank” The CPF framework is less well suited to clients with more limited demand for IBRD and IDA lending or limited demand for investment lending versus policy-based lending. Resource-rich countries are an example of the former. IEG’s CPEs found that there was no consistent framework for engaging these countries—no common approach to dealing with clients whose defining characteristics were a rich endowment with nonrenewable natural resourc- es and dependence on revenues from their exploitation. In these countries, the challenge for the Bank Group is how to best leverage its knowledge and global experience, and lending is less relevant. EU member states such as Bulgaria and Romania are also examples where demand for lending tends to be limited (except in economic crises), and the Bank Group relies increas- ingly on reimbursable advisory services (RAS), a form of fee-for-service ASA delinked entirely from lending. Bhutan is an example of a shift away from investment lending toward policy-based lending linked to high volumes of ASA. The team did not adequately adjust the CPF results framework to capture outcomes linked to policy-based lending and dissemination and dialogue on ASA—the only outcomes that could arise from implementation of the Bhutan CPS. Overall, the Bank Group does not have a standardized M&E system for ASA products. CPF results frameworks tend to be dominated by objectives and indi- Independent Evaluation Group World Bank Group    83 cators related to the lending portfolio, regardless of its relative importance in the mix of interventions in the Bank Group program. Evaluation of CPF pro- grams that tilt heavily toward ASA (including RAS-based programs) are likely to undervalue the influence of the Bank Group program within the country. Dropped and Canceled Projects—May Be Necessary for Adaptability but Could Involve Possible Trade-Offs with Efficiency It is expected that some projects planned at the start of the CPF will be dropped or canceled because of changing circumstances, notably a change in government priorities or commitment. Dropped or canceled projects can be a sign of adaptation and innovation. However, in some instances, it is possible that they may represent an inefficiency in the time or money that may have been spent preparing them. Over the period covered by this RAP, more than one-third of projects (35 percent) have been dropped or canceled (equivalent to 27 percent of the planned lending volume). However, this figure is an underestimate because it applies only to projects that were assigned a project identification number,6 while other projects never reach that stage. The share of dropped and can- celed projects has fluctuated modestly across the RAP period, with no clear trend (figure 5.2, panel a). The reason for projects being dropped is noted in nearly three-quarters of cases (figure 5.2, panel b). The most common reason is that the government has a change in priorities, which may include a central agency (likely finance) blocking the loan from moving forward, even though a line ministry has in- terest. There is a relatively small, though not a negligible, number of projects for which the World Bank feels that the project no longer fits its strategy. The possible downsides to dropped projects include the following: » Project planning takes time and money for World Bank staff and government Results and Performance of the World Bank Group 2022  Chapter 5 representatives. Projects that are planned but are never identified or are canceled or dropped after they are identified may represent an inefficient use of resources. » Dropped projects weaken the line of sight from planned Bank Group activities to CPF objectives. The World Bank (2020b) found that more than 80 percent of CPF indicators were dropped or revised at the PLR stage. This high propor- tion is partly driven by the large number of dropped projects. » The analysis of dropped projects in CLRs and CLR Reviews currently does not go much beyond stating the numbers. CLRs and CLR Reviews could usefully examine the reasons for dropped projects and the associated costs in more detail to help inform a discussion on how to ensure that such projects do not represent an inefficiency. 84 Figure 5.2. Dropped and Canceled Projects a. Share of dropped and canceled projects (%) b. Reasons why projects are dropped Independent Evaluation Group World Bank Group    85 Source: World Bank Enterprise Data Catalog. Note: The drop reason is available for 73 percent of dropped projects. ID = identification number. Line of Sight The Bank Group defines the line of sight as “a clear path connecting an activity with its ultimate desired outcome” (World Bank Group 2021c, iv). Within the Bank Group context, the line of sight involves two stages: (i) from Bank Group support to CPF objectives/development outcome; and (ii) from CPF objectives/development outcome to HLOs (defined in box 5.1). The line of sight requires two conditions to be met at both stages: relevance (a neces- sary condition) and contribution (a sufficient condition). Regarding the first stage, the RAP found that Bank Group support was not fully relevant to CPF objectives/development outcome. Contribution also had some weaknesses in this first stage in that lending often fell short (that is, planned lending did not materialize and was dropped or canceled, or disbursements were delayed), and adequate restructuring of the country program and the associated results framework was not undertaken even at the PLR stage. Stated CPF objectives/development outcome then tended not to be achieved. The RAP found that both relevance and contribution were less evident in the second stage, from CPF objectives/development outcome to HLOs, although CPF objectives/development outcome generally mapped well into HLOs. Establishing relevance and contribution tends to be difficult because extra- neous influences increase as one moves toward final-stage outcomes, and confounding factors from actions originating outside the Bank Group pro- gram come into play. Results and Performance of the World Bank Group 2022  Chapter 5 Table 5.1 summarizes the RAP’s analysis of the line of sight.  ine of Sight Analysis Based on Completion and Learning Table 5.1. L Review (CLR) Reviews First Stage Second Stage From World Bank Group support to CPF objectives/ From CPF objectives/ Criterion development outcomes development outcomes to HLOs Relevance Weaknesses in relevance Relevance less evident » There is a lack of selectivity, » CPF objectives/development adaptiveness, and realism outcomes generally map well to in country programs, such HLOs. as insufficient attention to » However, issues remain with the implementation capacity and measurement of relevance. Deter- political risks. mining Bank Group relevance is » There are weaknesses in CPFs complicated by the need to also and their results frameworks account for the actions of other that insufficiently integrate development actors. 86 and leverage ASA and IFC and MIGA support. (continued) First Stage Second Stage From World Bank Group support to CPF objectives/ From CPF objectives/ Criterion development outcomes development outcomes to HLOs Contribution Weaknesses in contribution Contribution less evident » Lending often falls short— » Establishing contributions has planned lending does not ma- tended to be difficult—extraneous terialize, projects are dropped influences increase as one moves or canceled, or disbursements toward outcomes at the final are slower than planned. stages, and confounding factors » Adequate restructuring is not from actions originating outside undertaken at the PLR stage— the Bank Group program come CPF objectives/development into play. outcomes then tend not to be achieved. Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; CPF = Country Partnership Framework; IFC = International Finance Corporation; HLO = high-level outcome; MIGA = Multilateral Investment Guarantee Agency; PLR = Performance and Learning Review. Overall, the first stage of the line of sight, where relatively more weaknesses are observed, is under the Bank Group’s control. Bank Group actions that ad- dress weaknesses identified in this RAP can help improve both the relevance and contribution of country programs in this first stage. Improving the relevance and contribution in the second stage is more Independent Evaluation Group World Bank Group    87 difficult, given the influence of other development actors, including the government, civil society, and other development partners. For the sec- ond stage, further work is needed on approaches to consider the actions of other development actors and on measurement. Regarding relevance in the second stage, it will need to be established up front that every single country program consider the complementary actions of multiple actors that are needed to achieve the HLOs. This is necessary to ensure that the Bank Group is not supporting duplicative or redundant activities or activi- ties that may fail without complementary action. Regarding contribution in the second stage, in-depth, rigorous measurement on a selective case study basis is needed, rather than attempting to measure contribution for every single country program. 1  For example, IEG’s evaluation of outcome orientation at the country level pointed out that the Bank Group’s “country-level results system does not capture the Bank Group’s contribu- tion to country outcomes well, because it relies on results frameworks premised on metrics, attribution, and time-boundedness that do not fit the nature of country programs well” (World Bank 2020b, x). 2  During the sample period of FY13–22, the country program was sometimes referred to as the Country Assistance Strategy, Country Partnership Strategy, or CPF. The associated progress reports were referred to as Country Assistance Strategy Completion Report, Country Part- nership Strategy Progress Report, and Performance and Learning Review, respectively. These terms are used interchangeably in this RAP. 3  It is recognized that it may not be possible to discuss some sensitive topics in the CPF document. For such topics, it would still be useful for the Bank Group to develop a more transparent internal process. All other risks can be discussed in the CPF in terms of a change in commitment on key issues such as inclusion or economic reform. The CPF would also identify how the country program would be adjusted if government commitment did not match expectations. The RAP emphasizes that these risks can and should be addressed in the CPF/ Performance and Learning Review. The RAP gives examples of these risks not being well addressed in country programs, suggesting that the issue is not just one of inadequacies in Results and Performance of the World Bank Group 2022  Chapter 5 documentation. 4  It would be useful to examine in the future if weaknesses in risk mitigation explain, for example, the RAP’s finding that when individual CPF objectives/development outcome were considered over the 10-year period of FY13–22, almost half were partially achieved or not achieved. It would also be useful to examine in the future if weaknesses in country-level risk mitigation explain the gap between project ratings and country program ratings in CLR Reviews. The RAP provides examples of country programs where risks could have been better mitigated. With inadequate risk mitigation, future outcome ratings could come under pressure. The RAP also found examples where program changes (possibly in response to risks materializing) were insufficiently documented. 5  Clearly, the same degree of “oneness” will not be appropriate in every instance—the magnitude and nature of the role that each of the three institutions will need to play will vary from case to case. Senior management signaling and staff incentives are likely to be critical for ensuring well-coordinated, well-complemented, and well-sequenced actions across the three institutions. 88 6  A project identification number is generated for a new project once the World Bank team creates the Activity Initiation Summary in the World Bank’s Operations Portal. This happens during the identification and preparation stage and before the Concept Review stage. Independent Evaluation Group World Bank Group    89 6 | Advisory Services and Analytics in Country Programs Highlights Analysis of Completion and Learning Review (CLR) Reviews for a random sample of 50 countries found a good match in more than 80 percent of cases between advisory services and analytics (ASA) topics and government policies and programs and between ASA topics and CPF objectives/development outcome and World Bank operations. Less than half of the CLR Reviews reported on higher levels of ASA influence (such as ASA influence on policy dialogue, uptake in gov- ernment programs and policies, and uptake in World Bank Country Partnership Framework objectives and World Bank operations). In those CLR Reviews that did report on higher levels of ASA influ- ence, ASA were found to be influential in most cases. The CLR Reviews for the random sample of 50 countries reported, on average, on only about one-third of the ASA program in terms of use or influence. CLR Reviews do not seem to report on ASA that are not used. Ex- amining the reasons for the likely nonuse of ASA—particularly any institutional constraints within the Bank Group that hamper their use—may provide lessons for enhancing ASA use in the future. A historical lack of ASA monitoring raises questions about the Bank Group’s ambitions to be a “knowledge bank” and its ability to stra- tegically use ASA to improve country-level impact. 90   This chapter presents a qualitative analysis of the use or influence of ASA based on the CLR Reviews for this RAP’s random sample of 50 countries (budgetary trends are shown in appendix L).1 Unlike IFC, the World Bank does not conduct a self-evaluation of each ASA; therefore there is no valida- tion by IEG. The World Bank spends significant amounts of money on ASA: the annual ASA expenses in absolute terms have increased from $498 million in FY13 to $783 million in FY22, reaching a peak of $811 million in FY19.2 ASA expens- es as a share of total World Bank project expenses (both actual) were within the 40–50 percent range during FY13–22, with a peak of 50 percent in FY17. In this context, the examination of the CLR Reviews for this RAP’s random sample of 50 countries indicated that CLR Reviews reported, on average, on about one-third of the ASA program in terms of use or influence. The find- ings presented must be interpreted with this caveat in mind. The examination of ASA in this RAP is based on criteria distilled from a number of IEG sources (box 6.1). Box 6.1. Criteria for the Examination of Advisory Services and Analytics ASA and Development Outcome » Match between ASA topics and topics of government policies and programs » Extent to which ASA facilitated policy dialogue with government Independent Evaluation Group World Bank Group    91 » ASA uptake in government programs and policies » ASA uptake in World Bank Group CPFs or World Bank operationsa or both ASA and Bank Performanceb » Match between ASA topics and topics of CPF objectives or World Bank opera� tions or both » Extent to which ASA facilitated client and stakeholder ownership and engagement (continued)  riteria for the Examination of Advisory Services and Analytics Box 6.1. C (cont.) » Extent to which ASA were disseminated by the World Bank » An additional element of Bank performance: monitoring and evaluation and results frameworks for ASA Sources: Criteria derived from World Bank 2018 and other Independent Evaluation Group work. The derived criteria reflect the varied objectives and channels of ASA influence, which this Results and Performance of the World Bank Group report emphasizes need to be explicitly stated and put in place up front, then implemented and monitored. Note: ASA = advisory services and analytics; CPF = Country Partnership Framework. a. Two other criteria that are important—that is, the extent to which ASA help convene stakehold- ers in different agencies both internally and among donors, and the extent to which ASA influence knowledge and policy approaches in other countries—are not used in this Results and Performance of the World Bank Group report because the Completion and Learning Review (CLR) Reviews contain limited information on them. b. Two other criteria that are important—that is, the quality of the ASA and the World Bank inputs and processes pertaining to ASA—are not used in this Results and Performance of the World Bank Group report because the Completion and Learning Review (CLR) Reviews contained virtually no information on them. Advisory Services and Analytics and Development Outcome Results and Performance of the World Bank Group 2022  Chapter 6 Figure 6.1 presents the findings from the qualitative analysis of ASA and development outcome based on the criteria outlined in box 6.1. 92  eporting of Development Outcome in Advisory Services and Figure 6.1. R Analytics Source: Independent Evaluation Group. Note: This figure shows the number of Completion and Learning Review (CLR) Reviews. ASA = advisory services and analytics; CLR = Completion and Learning Review; CPF = Country Partnership Framework. Criterion discussed with respect to ASA means the CLR Review mentions specific criterion. Good influ- ence of ASA on criterion means that the CLR Review provided information or examples showing that a specific criterion was addressed. Match between Advisory Services and Analytics Topics and Topics of Government Policies and Programs Of the 50 randomly sampled countries for this RAP, CLR Reviews for 49 of Independent Evaluation Group World Bank Group    93 the countries reported on this criterion. Of these 49 CLR Reviews, 43 sug- gested a good match between ASA topics and the topics of government policies and programs (figure 6.1). For example, the ASA discussed in the CLR Review for Argentina reported that the World Bank expanded ASA after the 2015 election to provide analytical support in the government’s prior- ity areas of infrastructure, agriculture, and the environment. In Bulgaria, the provision of knowledge and advisory services in innovation, education, public finance and administration, competitiveness, green growth, and social inclusion aimed to further the government’s National Reform Programme goals. An increase in the number of ASA in Kazakhstan came from the government’s demand for policy advice and technical assistance during the macroeconomic crisis. The World Bank delivered 126 items of ASA in the Philippines, encompassing in-depth sector diagnostics that focused on the key constraints to poverty reduction and economic growth, in line with the government program. ASA were critical in filling knowledge gaps in Myanmar, where ASA covered a broad range of topics, particularly topics related to inclusion that were high- ly relevant after the 2016 Rakhine crisis. In Guyana, the proposed program of analytic and advisory activities (AAA) included on-demand economic monitoring and analysis of policy options to manage the impact of the crisis. The analytics were congruent with the Poverty Reduction Strategy Paper in Rwanda, as noted in the CLR Review. The ASA program reported in the CLR Review for the Comoros helped close a significant part of the knowledge gap by delivering core diagnostic tasks, including PERs, poverty assessment, and debt sustainability analysis. Extent to Which Advisory Services and Analytics Facilitated Policy Dialogue with Government This RAP’s analysis found that the CLR Reviews for 21 of the 50 sampled coun- tries mentioned policy dialogue with government (figure 6.1). Of these 21, 19 Results and Performance of the World Bank Group 2022  Chapter 6 suggested that ASA facilitated good policy dialogue. ASA were used to engage the (new) government in Argentina, for example, on important policy issues such as increasing agricultural productivity and sustaining dialogue to enable longer-term engagement on complex reforms. The CLR Review for Belarus notes that technical assistance supported policy dialogue with the government after the preparation of the 2014 Public Expenditure and Financial Account- ability report, which approved a public finance management reform strategy in the areas of medium-term budgeting, program budgeting, treasury, debt management, and accounting. In Kenya, the World Bank, with support from the Australian Agency for International Development (now Australian Aid), ran a large AAA program to provide policy and institutional reform advice to the government on the transition from a centralized to a more decentralized structure. The CLR Review for Poland notes that the ASA program benefited from strong dialogue with the authorities, and 19 of the ASA products were RAS. In Nicaragua, ASA provided the basis for country dialogue, which then helped the government and donors develop rural development strategies. In 94 Rwanda, ASA provided the basis for policy dialogue in social protection and in agriculture. In China, the CLR Review noted that joint flagship reports between the government and the World Bank were important for deepen- ing the dialogue on policies and World Bank programs. In the Democratic Republic of Congo, economic and sector work (ESW) was the main instru- ment to engage the government on policy issues in the areas of economic diversification, public expenditure management (including in the human development area), and mining. In Bosnia and Herzegovina, in addition to informing and stimulating debate on policy priorities, AAA products were designed to help maintain policy dialogue and World Bank presence in areas where the lack of policy consensus ruled out lending.3 The CLR Reviews for Panama and Nepal note that policy notes formed the basis for dialogue with the government, providing the basis for discussion on noncommunicable dis- eases in the former and on investment, infrastructure, and inclusion for the latter. In two of the CLR Reviews, the reporting of ASA noted that although IDA used ASA products as a basis for country dialogue (in Guinea), there was no information on which products were used, especially because no core di- agnostic analyses were produced during the CPS period. In Burundi, the CLR Review noted that policy notes lost traction because of the political crisis; thus the opportunity for dialogue was lost. According to IEG’s 2014 CPE for Brazil, the Bank Group made significant contributions when it served as a trusted adviser, providing analytical inputs and exchanging views on immediately relevant policy issues.4 IEG’s 2013 Independent Evaluation Group World Bank Group    95 CPE for Afghanistan pointed out that ASA were an effective complement to lending. Knowledge services were an important part of Bank Group support and showed the value of analytical work, even in areas where the Bank Group may opt out of direct project financing. Advisory Services and Analytics Uptake in Government Programs and Policies This RAP’s analysis found that the CLR Reviews for just 14 of the 50 sam- pled countries mentioned the uptake of ASA in government programs and policies. All of these reported the uptake as being good (figure 6.1). In the Bulgaria CLR Review, AAA recommendations were incorporated into the government’s higher education strategy, and Bulgaria’s National Roma Integration Strategy was prepared through the RAS and disseminated in the country. In Mauritius, World Bank AAA helped the government in de- veloping a new proxy means test for social assistance, and a social registry was launched to improve the information on clients of social assistance. In China, the World Bank’s ASA work on access to finance contributed to ad- vances in financial inclusion, including in rural areas.5 On energy efficiency and greenhouse gas emissions, ASA contributions in China helped integrate climate change considerations into energy regulation and develop energy efficiency financing mechanisms, including working with banks to finance energy efficiency and with client companies to reduce their carbon imprint. In Belarus, the ASA on directed lending helped guide the establishment of a comprehensive database at the central and local government levels. The CLR Review for Côte d’Ivoire reported that the World Bank’s analytical work was crucial in helping the government design and implement the eco- nomic recovery and growth policies after many years of civil strife. In addi- tion, it noted that considerable technical assistance helped the country reach its heavily indebted poor countries completion point. In the Arab Republic of Egypt, the AAA on education led to the development of new approaches Results and Performance of the World Bank Group 2022  Chapter 6 to university entrance and secondary education graduation exams. In São Tomé and Príncipe, the production of the Country Economic Memorandum provided the core analytical underpinnings for the preparation of the second Poverty Reduction Strategy Paper. Furthermore, the ASA assessing the gov- ernment’s debt management capacity provided support for the development of a debt management reform plan, leading to strengthened fiscal controls. In Pakistan, World Bank AAA addressed social safety net issues by focusing on the design of cash transfers and consolidating and integrating various social protection instruments. In the low-income country of Nepal, the ASA program informed the develop- ment of the country program by helping to prepare the scaling up of hydro- power development and informing the financial sector agenda. In Burundi, the technical assistance on social protection covered social protection policy and operational aspects and helped expand safety nets. The uptake of ASA findings in government programs and policies can be 96 helped by integrating ASA with other World Bank support (box 6.2). mportance of Integrating Advisory Services and Analytics with Box 6.2. I Other Bank Support Helping governments effectively achieve policy reforms typically takes sustained World Bank Group lending and nonlending support. A World Bank report noted that sustained follow-up (beyond one-off events) in the form of lending or nonlending technical assistance is a factor that improves the effectiveness of World Bank economic and sector work and technical assistance (World Bank 2008). The World Bank (2016c) recommended that the Bank Group ensure appropriate links among economic and sector work, nonlending technical assistance, and projects. The World Bank (2010) also recommended that the integration of the Poverty and Social Impact Analysis (a form of advisory services and analytics) into the World Bank’s country assistance program be improved. Similarly, the World Bank (2021c) found that a growing body of analytical work on aging is only partially reflected in the Systematic Country Diagnostics and Country Partnership Frameworks of countries that are aging. One reason for the fragmentation of advisory services and analytics on aging was the lack of coordination and collaboration across Global Practices such that issues and risks were often considered in isolation. Furthermore, the outcome orientation pilots, presented in World Bank Group 2021c, revealed that pairing lending programs with technical assistance, impact evaluations, and dialogue was successful in Punjab, Pakistan. The World Bank followed an indirect pathway, pairing World Bank lending with high-level dialogue and technical support Independent Evaluation Group World Bank Group    97 and impact evaluation, thus helping the government of Punjab bring about a long- term impact in Punjab’s education system. In the first phase, the World Bank’s analyti- cal work on private school subsidies, a Public Expenditure Review, and an impact eval- uation on conditional cash transfers for girls’ enrollment fed into government policies and built evidence for implementing new strategies. A longitudinal panel survey the World Bank financed also helped provide a deep understanding of the challenges in school enrollment and quality. In the second phase, the World Bank shifted to invest- ment loans but continued impact evaluations on school vouchers in public-private partnership schools, private school participation, and teacher incentives that led to evidence-based strategies. Source: World Bank 2008, 2010, 2016c, 2021c Advisory Services and Analytics Uptake in World Bank Group Country Partnership Framework Objectives and World Bank Operations This RAP’s analysis found that the CLR Reviews for just 15 of the 50 sampled countries discussed the uptake of ASA in in Bank Group CPF objectives or World Bank operations. Of these, uptake was reported to be good in 9 countries (figure 6.1). The CLR Review for Bosnia and Herzegovina noted that the World Bank had undertaken a series of core diagnostic ESW reports in the 18–24 months before preparation of the CPS, helping inform CPS priorities and, in some cases, the design of individual lending projects. Delivery of half of the CPS objectives for Poland was supported exclusively by ASA products, and the CLR Review also notes that a number of these were quite innovative in the areas of fiscal rules and tax administration. In Argentina, ASA contributed to new operations and technical assistance projects in agriculture and the environment, augmenting the IPF operations in supporting CPS objectives. In Jamaica, ESW provided the analytical underpinning of many operations. Results and Performance of the World Bank Group 2022  Chapter 6 Sector-level ASA work in education, health, poverty, and social protection in the Madagascar CPF paved the way for the preparation of the new CPF for the postcrisis period. In Nepal, ASA work on governance and fiscal issues (Public Expenditure and Financial Accountability) informed the preparation of the fiscal reform development policy credit, while the ASA work on the financial sector (Financial Sector Assessment Program) aided the design and implementation of the financial sector development policy credit support project. A review of urbanization and work on urban development informed the design of new urban projects, as reported in the CLR Review for Senegal. In a few cases, the ASA program was not as effective. The CLR Review for Bhutan noted that the large ASA program required to support the CPS re- sults areas and build the knowledge base to strengthen policies and institu- tions lacked focus. In Mauritius, major obstacles to long-term development that had been identified in the analytical work were not included in the CPS and PLR. In Tunisia, the CPE noted that project design was often flawed because critical bottlenecks identified in ESW were not addressed; many of 98 the operations delivered change that was necessary but insufficient to ac- complish Bank Group objectives without supporting reforms to remove core obstacles. Timeliness is an important factor for uptake but proved a negating factor in a few cases. There were concerns about the timeliness of ASA in Brazil, Kazakhstan, Rwanda, and Sri Lanka. In Brazil, the ASA program matched well with the CPS focus areas, but ASA’s attention to structural issues was belat- ed, discouraged by weak government demand, and it was conducted only in the final two years of the CPS period. The CLR Review notes that this belated attention may have reduced the Bank Group’s preparedness for addressing the impacts of recession on operations. In Kazakhstan, the CLR Review notes that the World Bank did not act in a timely fashion to reduce capacity risk, which it addressed only when the crisis exploded by stepping up efforts to ensure that officials could implement the projects better, faster, and with fewer mistakes, but this was probably too little, too late. In Sri Lanka, the CLR Review notes that timeliness is a key determinant of the effective- ness of AAA. Some of the AAA tasks (PER, revenue analysis note) came too late to make a substantial contribution to the CPS cycle. The development policy loan had to be delayed because the AAA was not being undertaken until FY15 and FY16, and the policy dialogue on reforms had to be deferred beyond the CPS period. In Rwanda, the CPE emphasized the need to use analytical work more strategically to draw timely attention to insufficiently Independent Evaluation Group World Bank Group    99 addressed binding constraints or policy errors that will need to be over- come if Rwanda is to continue smoothly and sustainably on its path toward becoming a middle-income country and eventually achieving high-income country status. Advisory Services and Analytics and Bank Performance Figure 6.2 presents the findings from the qualitative analysis of ASA and Bank performance based on the criteria outlined in box 6.1.  dvisory Services and Analytics and Bank Performance Figure 6.2. A Source: Independent Evaluation Group. Note: The figure shows the number of Completion and Learning Review (CLR) Reviews. Criterion dis- cussed with respect to ASA means the CLR Review mentions the specific criterion. Good performance relating to ASA on criterion means the CLR Review provided information or examples that the specific criterion was addressed. Match between Advisory Services and Analytics Topics and Topics of Country Partnership Framework Objectives or World Bank Operations Results and Performance of the World Bank Group 2022  Chapter 6 All CLR Reviews from the 50 randomly sampled countries for this RAP discussed the match between ASA topics and CPF objectives or World Bank operations, and a good match was noted for 41 of the countries (figure 6.2). The ASA mentioned in the CLR Review for Kazakhstan covered topics that corresponded with the CPS and PLR objectives, and their distribution corre- sponded with that of World Bank operations, with the program intensive in areas where financing was small. In Jordan, the CLR Review noted that the CPS was generally selective, with the focus areas and interventions chosen and designed based on analytical work. The CLR Review for Armenia report- ed that completed ASA activities corresponded with most areas where the World Bank provided operational support.6 In Pakistan, AAA corresponded with all the pillars of the CPS, particularly the governance pillar.7 In Nige- ria, ASA overall followed a programmatic approach for the main themes of engagement that were clearly identified in the CPS, indicating strong match- ing between the ASA program and the CPS pillars. Similarly, as noted in the 100 CLR Review for Bhutan, the ASA covered issues that were related to the CPS objectives. The CLR Review for Kenya noted that the AAA included major reports covering the range of sectors covered by lending. Where the match was not as good, the CLR Reviews commented on an ab- sence of core ASA documents, such as a Country Economic Memorandum to update growth analyses and an assessment of political economy risks (CLR Review for Burundi), and an absence of the Country Economic Memorandum or poverty assessment (CLR Review for Guinea). In addition, the CLR Review for Burundi reported that the ASA added during the program (for example, on debt management) did not closely match the Country Assistance Strategy objectives, and similarly, the ASA on trade did not closely match the Country Assistance Strategy objective on private investment. Planned ASA were dropped for Guinea, and the CLR Review noted that at least one planned task, the agricultural growth and competitiveness task, might have provid- ed relevant input into CPS efforts to raise productivity. The CLR Review for Guinea also noted that the CLR paid inadequate attention to the role of ASA, while the CLR Review for the Comoros noted that the ASA program was too fragmented, with about half of the ASA not linked to any CPS objective. The CLR Review for the Marshall Islands noted that more robust analytical work was required given that it was the World Bank’s first direct engagement in the country. Independent Evaluation Group World Bank Group    101 Extent to Which Advisory Services and Analytics Facilitated Client and Stakeholder Ownership and Engagement This RAP’s analysis of CLR Reviews for the random sample of 50 countries found that just 11 discussed the extent to which ASA facilitated client and stakeholder ownership and engagement. Of these, 8 reported that ASA had good client and stakeholder ownership and engagement (figure 6.2). The CLR Reviews reported on good coordination and cooperation between the World Bank and development partners in technical assistance work and producing joint analytical work. The CLR Review for Jordan reported on the coordination with development partners for several technical assistance projects. In Kosovo, the CLR Review noted that the World Bank worked with other development partners (the Danish International Development Agency and the Swiss Agency for Development and Cooperation) on joint analytical work and with bilateral and multilateral donors in Lesotho. The CLR Review for Panama noted good coordination on analytical work between the World Bank and other development partners in areas where there was overlap. In Pakistan, the World Bank worked with the government of Pakistan and development partners (African Development Bank, the EU, and the United Nations) in carrying out the postconflict needs assessment. In Bulgaria, World Bank ASA directly supported the Ministry of Finance in identifying expenditure opti- mization programs that supported the government in tackling public sector wages and employment, leading to a sharp reduction in the fiscal deficit. In three other CLR Reviews that discussed client and stakeholder ownership and engagement, the findings were less positive. Client and stakeholder ownership and engagement are important for ensuring that the knowledge generated by ASA actually feeds into decision-making. Without such uptake of ASA in decision-making, the World Bank risks being a “report bank,” not a “knowledge bank.” Becoming a “knowledge bank” calls for the use of knowl- edge, which stakeholder engagement and dissemination can foster. Beyond Results and Performance of the World Bank Group 2022  Chapter 6 this, knowledge translation or knowledge brokering (White 2019) can help country clients determine which knowledge is most relevant to the decisions they face and how best to tailor global knowledge to local circumstances. Extent to Which the World Bank Disseminated Advisory Services and Analytics Just 18 of the CLR Reviews reported on the dissemination of ASA by the World Bank, with just 3 indicating good dissemination (figure 6.2). In Nigeria, the CLR Review notes that the Growth and Employment Study set a good prac- tice example in dissemination with extensive seminars and conferences and because the report was serialized in the local press. The CLR Review for Pakistan noted the “good dissemination” of several AAA reports. Six CLR Reviews note that that ASA were disseminated through the Open Knowledge Repository. Although this alone is a good thing, it does not pro- vide information on usage and thus was considered modest. The CLR Review 102 for Belarus notes that the World Bank could have done more in the consulta- tion and dissemination of ASA to a broader nongovernment audience. Sim- ilarly, in Uganda, the CLR Review criticizes the CLR for not describing what had been done to disseminate the ESW. The Bank Group will need to pay greater attention to the adequacy of knowledge transfer and will need to treat ASA as a process, not just a product, as noted in World Bank (2003). Furthermore, a stronger relationship with the client correlates with more learning and better results as noted in World Bank (2005). The 2003 Guatemala Poverty Assessment was designed as a multiyear program of analytical work and technical assistance involving the establishment of long-term working relations with in-country organizations. Regional staff commented that this collaborative process helped improve the realism of the study’s analysis and the relevance of its recommendations. Moreover, the Bank Group’s convening power as an independent generator and broker of global knowledge allows it to inform development policy makers and take a lead role in setting the agenda for global discussions on development, according to the World Bank Group (2021b). This report also notes that the Bank Group’s involvement in financing and knowledge enables it to generate development solutions, thereby enabling it to become a solutions bank (World Bank Group 2021b). An Additional Element of Bank Performance: Monitoring and Evaluation and Results Frameworks for Advisory Independent Evaluation Group World Bank Group    103 Services and Analytics The World Bank has introduced some provisions for monitoring the influence of ASA (such as an ex ante identification of the primary clients of the ASA, who can then be surveyed ex post to determine their perception of ASA impact). However, more comprehensive M&E of the use or influence of ASA (beyond perception surveys) will be needed. Evidence from an examination of CLR Reviews for this RAP suggests that M&E are weak aspects of ASA and that the results framework for the World Bank’s ASA needs to be tightened. For example, in Poland, there has been inadequate attention to ASA’s policy impact, which has not been reflected in the results framework. In Romania, the results framework had weak links between stated objectives and their related indicators. The CLR Review finds that “it is important to design the results framework to ensure strong linkages between RAS, CPS objectives, and results and ensure appropriate monitoring and reporting of results. In Egypt, the CLR would have benefited from more evidence on whether particular AAA analyses either influenced in-country thinking on the issues they were designed to illuminate or informed the design of World Bank operations. Several past IEG evaluations have flagged weaknesses in M&E and results frameworks for ASA, indicating that this has been a long- standing issue. For example, the World Bank (2008) pointed out that the World Bank’s results-tracking framework for ESW and technical assistance was poor and recommended it be improved, including by incorporating client feedback. The World Bank (2010) recommended that the Bank Group clarify the operational objectives (that is, objectives relating to intended effects and influence) of each Poverty and Social Impact Analysis, how the Bank Group intended to meet them, and how the achievement of those objectives would be monitored and evaluated. Furthermore, the World Bank (2015a) pointed out that project M&E frameworks—how objectives are defined, indicators are derived, and baselines are deployed—help determine how effectively results are reported. The World Bank (2016c) noted that the monitoring of the results of the World Bank’s knowledge services was weak for both individual Results and Performance of the World Bank Group 2022  Chapter 6 activities and country programs. Unplanned ASA can be beneficial to the country program when responding to changed circumstances, but it also needs to be monitored. The flexibil- ity adopted at the Country Assistance Strategy Progress Report stage in Sri Lanka through adding unplanned AAA enabled the country program to refocus its attention on core AAA. The AAA portfolio witnessed a great deal of volatility, with 11 planned AAA dropped at the Country Assistance Strategy Progress Report stage and an additional 15 unplanned AAA tasks added, including core AAA that were essential for informing strategy choic- es. In Pakistan, the CPS planned on completing 36 nonlending tasks, and the Country Partnership Strategy Progress Report dropped 5 of these and added an additional 26. In total, about 85 actual tasks were delivered. The World Bank addressed the flood that occurred at the beginning of the CPS period through reallocation of existing disaster funding and an increase in project and AAA disaster prevention and management activities. In Bulgaria, 104 a number of new AAA were added during the CPS period and appear to have been consistent with the objectives of the CPS and the priorities laid out in the National Reform Programme. In Lesotho, the World Bank also provided several analytical pieces and technical assistance in response to the govern- ment’s request. At the same time, unplanned ASA added or dropped during the CPF without adjusting the results framework risked reducing relevance. In Panama, the number of ASA was increased at the PLR stage in response to government requests, but the impact of these—especially the stand-alone AAAs—was not clear because they were not incorporated into the results framework. In Nicaragua, unplanned ASA covered some of the areas targeted by the CPS objectives as well as areas that were not part of the CPS, with even some critical ASA (poverty assessments) dropped. In Romania, the expanded RAS program at the PLR stage caused the program to lose some of its earlier focus and selectivity.8 In The Gambia, only one of the six technical assistance delivered was in the original list. Other planned ASA—such as the ESW on energy and technical assistance on trade logistics, public-private partner- ship, human development, financial sector, and social protection—did not materialize, and no explanation was given in the CLR Review. The World Bank (2016c) noted that demand-driven, knowledge-based programs can be useful instruments for strengthening partnerships. However, their effectiveness can be limited by an inadequate M&E framework, a lack of disclosure of policy rec- ommendations, and insufficient engagement by local partners. Furthermore, Independent Evaluation Group World Bank Group    105 the resulting ASA program encompassing unplanned ASA can also lead to a fragmented ASA program. This was noted in several CLR Reviews. For exam- ple, in the Comoros, the ASA program was fragmented, with about half of the products not linked to any specific CPS objectives. The CLR Review for South Africa noted the absent to weak role of ASA in providing a knowledge base for government policies and described the ASA program as relatively scat- tered. Appendix M provides the reasons for unplanned ASA. Stronger M&E and results frameworks can help avoid these risks. 1  During the sample period FY13–22, advisory services and analytics (ASA) also covered economic and sector work or analytic and advisory activities (AAA). These terms are used in- terchangeably in this RAP. The RAP’s coverage of ASA follows the World Bank’s identification, so, for example, if the World Bank tagged project-specific ASA as “ASA,” this RAP included it. 2  The project expense data are extracted from the Enterprise Data Catalog’s Project Expenses Details data set and aggregated at an annual level. 3  A lack of policy consensus that ruled out lending included, for example, public expenditure reform, public financial management, and financial sector reform. The most important was the public expenditure reform, where a planned development policy loan series had been dropped, but the dialogue was maintained through a number of analytic and advisory activ- ities, such as a Report on the Observance of Standards and Codes, support for debt manage- ment on social spending, and smart safety nets. 4  Examples are support for Bolsa Família, student learning outcomes, pension reforms, and subnational results-based management systems. 5  The World Bank supported the People’s Bank of China in the preparation of the Financial Sector Inclusion Plan. 6  In addition to other topics not linked to Country Partnership Strategy objectives (that is, Results and Performance of the World Bank Group 2022  Chapter 6 tasks on gender dynamics and vulnerability), although relevant from a broader development perspective 7  According to the CLR Review, the impact of ASA was not assessed in a meaningful way ex- cept for noting its role in governance and disaster risk. 8  Changes in the Performance and Learning Review witnessed a sharp reduction in lending because of changing government priorities and a strong preference for reimbursable advisory services. 106 7 | Future Directions The enhancements now included in World Bank Group (2021a) are ex- pected to help address some of the issues identified in this RAP, but further enhancements to the guidance are identified in appendix I. We provide three future directions for the Bank Group’s consideration. Enhancing the Effectiveness of World Bank Group Support at the Country Program Level » Improve the selectivity and framing of CPF objectives and the realism of Bank Group country programs, especially regarding implementation capacity and political challenges. » Prioritize implementation of the One Bank Group approach (including the Mobilizing Finance for Development agenda and the Cascade approach), a long-standing aim of the Bank Group. » Ensure that the Bank Group adapts and keeps pace with changing circum- stances, such as shifts in government commitment or changes in Bank Group priorities, while being mindful not to lower the objectives to make up for the lack of progress. One option would be to consider the application, at the country level, of the M&E system currently applied to restructured projects at the World Bank. » Report in a timely way, in relevant documents (particularly in the Perfor- mance and Learning Review), any adaptations made to the country program and update the results frameworks accordingly. » Ensure that the Bank Group’s country program benefits from all elements of Bank Group support collectively and that the sum of the parts of Bank Group support is monitored. Strengthening Risk Identification and Mitigation » Maintain the Bank Group’s current good performance on identifying macro- 107 economic risk and risks associated with external shocks while improving the identification of risks associated with implementation capacity and political economy. » Identify up front the possible key risk scenarios and outline the course of ac- tion to address those scenarios so that timely action can be taken, depending on which scenario unfolds. » Expand and update the current country engagement guidance to include possible key risk scenarios and the responses to each to facilitate better risk mitigation. Strengthening Risk Identification and Mitigation » Maintain the Bank Group’s current good performance on identifying mac- roeconomic risk and risks associated with external shocks while improving the identification of risks associated with implementation capacity and political economy. » Identify up front the possible key risk scenarios and outline the course of ac- tion to address those scenarios so that timely action can be taken, depending on which scenario unfolds. » Expand and update the current country engagement guidance to include Results and Performance of the World Bank Group 2022  Chapter 7 possible key risk scenarios and the responses to each to facilitate better risk mitigation. Monitoring Advisory Services and Analytics Use or Influence » Systematically monitor and evaluate whether ASA are achieving their intended influence. » Consider introducing self-evaluation of analytical work across all Bank Group institutions. 108 Bibliography White, Howard. 2018. “Theory-Based Systematic Reviews.” Journal of Development Effectiveness 10 (1): 17–38. White, Howard. 2019. “The Twenty-First Century Experimenting Society: The Four Waves of the Evidence Revolution.” Palgrave Communications 5 (1): 47. World Bank. 2003. Sharing Knowledge: Innovations and Remaining Challenges. Wash- ington, DC: World Bank. https://openknowledge.worldbank.org/bitstream/ handle/10986/15043/280400PAPER0Sharing0Knowledge.pdf?sequence=1&isAl- lowed=y. World Bank. 2005. Capacity Building in Africa: An OED Evaluation of World Bank Support. Washington, DC: World Bank. https://openknowledge.worldbank.org/ bitstream/handle/10986/7468/343510PAPER0AF101OFFICIAL0USE0ONLY1. pdf?sequence=1&isAllowed=y. World Bank. 2008. Using Knowledge to Improve Development Effectiveness: An Evalu- ation of World Bank Economic and Sector Work and Technical Assistance, 2000– 2006. Independent Evaluation Group. Washington, DC: World Bank. https:// openknowledge.worldbank.org/bitstream/handle/10986/6317/485970PUB- 0know10Box338915B01PUBLIC1.pdf?sequence=1&isAllowed=y. Independent Evaluation Group World Bank Group    109 World Bank. 2010. Analyzing the Effects of Policy Reforms on the Poor: An Evaluation of the Effectiveness of World Bank Support to Poverty and Social Impact Analyses. Independent Evaluation Group. Washington, DC: World Bank. https://open- knowledge.worldbank.org/handle/10986/2484. World Bank. 2013. Afghanistan Country Program Evaluation, 2002–11: Evaluation of the World Bank Group Program. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/ files/AfghanistanCPE_2002_11.pdf. World Bank. 2014a. Brazil Country Program Evaluation, FY2004–11: Evaluation of the World Bank Group Program. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/ files/brazil_cpe_updated.pdf. World Bank. 2014b. Learning and Results in World Bank Operations: How the Bank Learns, Evaluation I. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/learning-and-results. World Bank. 2014c. Tunisia Country Program Evaluation, FY2005–13: Evaluation of the World Bank Group Program. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/ files/tunisia_cpe_updated.pdf. World Bank. 2015a. Learning and Results in World Bank Operations: Toward a New Learning Strategy, Evaluation 2. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/reports/ chapters/LR2_full_report_revised.pdf. World Bank. 2015b. World Bank Group Engagement in Resource-Rich Developing Coun- tries: The Cases of the Plurinational State of Bolivia, Kazakhstan, Mongolia, and Zambia. Clustered Country Program Evaluation Synthesis Report. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evalu- ation/files/ccpe-synthesis_full.pdf. World Bank. 2016a. Enhancing Competitiveness in Sri Lanka. Washington, DC: World Results and Performance of the World Bank Group 2022  Bibliography Bank. https://openknowledge.worldbank.org/handle/10986/24927. World Bank. 2016b. Growing the Rural Nonfarm Economy to Alleviate Poverty: An Evaluation of the Contribution of the World Bank Group. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/de- fault/files/Data/Evaluation/files/RuralNonFarm.pdf. World Bank. 2016c. Knowledge-Based Country Programs: An Evaluation of World Bank Group Experience. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/knowledge-based-country-programs. World Bank. 2016d. World Bank Group Engagement in Small States: The Cases of the OECS, Pacific Island Countries, Cabo Verde, Djibouti, Mauritius, and the Sey- chelles—Clustered Country Program Evaluation. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/ files/Data/reports/ExecSum-SmallStates.pdf. World Bank. 2017a. World Bank Group Country Engagement: An Early-Stage Assess- 110 ment of the Systematic Country Diagnostic and Country Partnership Framework Process and Implementation. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/ files/scd-cpf.pdf. World Bank. 2017b. World Bank Group Joint Projects: A Review of Two Decades of Expe- rience: Lessons and Implications from Evaluation. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/ files/Data/Evaluation/files/lp_wbgjointprojects.pdf. World Bank. 2018. Mexico: Country Program Evaluation: An Independent Evaluation of the World Bank Group’s Support to Mexico (2008–17). Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/de- fault/files/Data/Evaluation/files/Mexico_CPE.pdf. World Bank. 2019a. Rwanda Country Program Evaluation FY09–17: An Independent Evaluation. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/Rwan- daCPE.pdf. World Bank. 2019b. The World Bank Group Partnership with the Philippines, 2009–18: Country Program Evaluation. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/ files/PhilippinesCPE.pdf. World Bank. 2020a. Results and Performance of the World Bank Group 2020. Indepen- Independent Evaluation Group World Bank Group    111 dent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbank- group.org/sites/default/files/Data/Evaluation/files/RAP2020.pdf. World Bank. 2020b. The World Bank Group Outcome Orientation at the Country Level: An Independent Evaluation. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/ files/WBG_OutcomeOrientation.pdf. World Bank. 2021a. Albania Country Program Evaluation. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/de- fault/files/Data/Evaluation/files/Albania_CPE.pdf. World Bank. 2021b. Results and Performance of the World Bank Group 2021. Indepen- dent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbank- group.org/sites/default/files/Data/Evaluation/files/RAP2021.pdf. World Bank. 2021c. World Bank Support to Aging Countries: An Independent Evaluation. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.world- bankgroup.org/sites/default/files/Data/Evaluation/files/AgingCountries.pdf. World Bank. 2022a. An Evaluation of International Finance Corporation Investments in K–12 Private Schools. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/ IFCSupport_K12PrivateEducation.pdf. World Bank. 2022b. The International Finance Corporation’s and Multilateral In- vestment Guarantee Agency’s Support for Private Investment in Fragile and Con- flict-Affected Situations, Fiscal Years 2010–21. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/ files/Data/Evaluation/files/IFC_MIGA_FCS.pdf. World Bank Group. 2018. “IBRD/IDA/IFC/MIGA Guidance: Country Engagement.” OPS5.01-GUID.110, World Bank Group, Washington, DC. https://ppfdocuments. azureedge.net/1e77ac41-2540-44d7-ad2c-1384abff3b63.pdf. World Bank Group. 2020. Revised Classification of Fragility and Conflict Situations for World Bank Group Engagement. Washington, DC: World Bank Group. https://the- Results and Performance of the World Bank Group 2022  Bibliography docs.worldbank.org/en/doc/333071582771136385-0090022020/original/Classifi- cationofFragileandConflictAffectedSituations.pdf. World Bank Group. 2021a. “IBRD/IFC/MIGA/IDA Guidance: Country Engagement.” OPS5.01-GUID.113, World Bank Group, Washington, DC. https://ppfdocuments. azureedge.net/1d0c9176-2efc-4b9e-8148-f6149b5f8c75.pdf. World Bank Group. 2021b. “Realizing the World Bank Group’s Knowledge Potential for Effective Development Solutions: A Strategic Framework.” Board Report 157822, World Bank Group, Washington, DC. http://documents.worldbank.org/ curated/en/309981617140869469/. World Bank Group. 2021c. Strengthening World Bank Group Outcome Orienta- tion: A Roadmap. Washington, DC: World Bank Group. https://documents1. worldbank.org/curated/en/170591626919261063/pdf/Strengthen- ing-World-Bank-Group-Outcome-Orientation-A-Roadmap.pdf. 112 APPENDIXES Independent Evaluation Group Results and Performance of the World Bank Group 2022 Appendix A. Random Sampling Methodology The population consists of 108 countries, each with at least one CLR Review conducted since fiscal year 2013. CLR Reviews without ratings of development outcome or Bank Group Performance are excluded. Each coun- try’s income level and fragile and conflict-affected situations (FCS) status is tagged based on the ending fiscal year of the latest Country Partnership Framework period reviewed by the Independent Evaluation Group. The sam- pling process starts with dividing countries into strata based on income level and FCS status, as shown in table A.1. Note that the total size of the popu- lation is 103 because six Organisation of Eastern Caribbean States countries are treated as one, given that they received aggregate ratings instead of individual ratings, and all belong to the same stratum. Table A.1. Strata of Population Strata Number Percent Results and Performance of the World Bank Group 2022  Appendix A Upper-middle-income or above and non-FCS 35 34 Lower-middle-income and non-FCS 33 32 Low income and non-FCS 15 15 Upper-middle-income or above and FCS 4 4 Lower-middle-income and FCS 9 9 Low income and FCS 7 7 Total 103 100 Source: Independent Evaluation Group data. Note: FCS = fragile and conflict-affected situations. The total sample size is calculated based on confidence level and margin of error. The numbers to be considered are listed in table A.2. 116 Table A.2. Total Sample Size Sample Size Confidence Level (%) Margin of Error (%) 82 95 5 75 90 5 50 95 10 42 90 10 Source: Independent Evaluation Group data. Under proportionate sampling, the sample size of each stratum is calculated based on the subgroup’s proportion in the population. Table A.3 shows the sample size for each subgroup under different total sample sizes. The sam- ple size of 50 is selected under consideration of both analysis workload and sample quality. Because of rounding of the calculated sample size for each subgroup, the actual sample size is 49 under the intended sample size of 50. Given the high interest in low-income FCS and the small population of only 7 countries, the analysis decided to select 1 additional country for this stra- tum, making an actual sample of 50 low-income FCS. Table A.3. Calculated Strata Sample Size Sample Size Strata 82 75 50 42 Independent Evaluation Group World Bank Group    117 Upper-middle-income or above and non-FCS 28 25 17 14 Lower-middle-income and non-FCS 26 24 16 13 Low income and non-FCS 12 11 7 6 Upper-middle-income or above and FCS 3 3 2 2 Lower-middle-income and FCS 7 7 4 4 Low income and FCS 6 5 3 3 Total 82 75 49 42 Source: Independent Evaluation Group data. Note: FCS = fragile and conflict-affected situations. Random sampling is conducted to select countries for each stratum. Table A.4 provides the countries sampled for each subgroup under the total sample size of 50. Table A.4. Fifty Sampled Countries Income Level Non-FCS FCS N = 17 (6 with 2 CLR Reviews) N=2 High- and upper- Argentina a Montenegro Bosnia and Herzegovina middle income Belarusa OECS countries Marshall Islands, The Brazil Panama Bulgaria Peru China a Polanda Jordan Romaniaa Jamaica Serbia Kazakhstan South Africaa Mauritius N = 16 (7 with 2 CLR Reviews) N=4 Lower-middle income Armeniaa Nicaraguaa Côte d’Ivoire Bhutana Nigeriaa Kosovo Bolivia Pakistan Micronesia, Fed. Sts. Cameroon Philippines a Myanmar Congo, Rep.a São Tomé and Príncipe Results and Performance of the World Bank Group 2022  Appendix A Egypt, Arab Sri Lanka Rep. Guyana Tajikistana Lesotho Ukraine N = 7 (2 with 2 CLR Reviews) N = 4 (2 with 2 CLR Reviews) Low income Guinea Rwandaa Burundia Kenya Senegala Comoros Madagascar Uganda Gambia, Thea Nepal Haiti Source: Independent Evaluation Group data. Note: CLR = Completion and Learning Review; FCS = fragile and conflict-affected situations. a. These countries have two CLR Reviews. 118 Table B.1. Data Sources and Coverage, World Bank Source FY13 FY14 FY15 FY16 FY17 FY18 FY19 FY20 FY21 FY22 Total CPEs 1 2 0 1 1 0 2 2 0 0 9 CLR Reviews 19 23 13 22 23 16 15 11 8 2 152 Sector, thematic, and 5 6 10 11 11 10 9 6 10 7 85 Coverage corporate evaluations PPARs 33 47 36 43 35 61 41 20 28 15 359 ICRRs 322 374 413 283 331 306 237 276 221 147 2,910 Source: Independent Evaluation Group data. Note: For CPEs and sector, thematic, and corporate evaluations, the fiscal year refers to the publication or document disclosure fiscal year. For CLR Reviews, PPARs, and ICRRs, the fiscal year refers to the evaluation fiscal year. The final number of evaluations will be updated as of June 30, 2022. CLR = Completion and Learning Review; CPE = Country Program Evaluation; FY = fiscal year; ICRR = Implementation Completion and Results Report Review; PPAR = Project Performance Assessment Report. Appendix B. Data Sources and Independent Evaluation Group World Bank Group    119 120 Results and Performance of the World Bank Group 2022  Appendix B Table B.2. Data Sources and Coverage, International Finance Corporation Investment Services Source CY12 CY13 CY14 CY15 CY16 CY17 CY18 CY19 CY20 CY21 Total IFC XPSR EvNotes 88 65 80 90 79 92 99 81 85 50 809 and PES Source: Independent Evaluation Group data. Note: Based on completed project validations as of June 30, 2022. CY = calendar year; EvNote = Evaluation Note; IFC = International Finance Corporation; PES = Project Evaluation Summary; XPSR = Expanded Project Supervision Report.  ata Sources and Coverage, International Finance Corporation Advisory Services Table B.3. D FY12 FY13 FY14 FY15 FY16 FY17 FY18 FY19 FY20 FY21 Total IFC PCRs EvNotes 66 70 54 61 39 45 85 54 68 32 574 Source: Independent Evaluation Group data. Note: Based on completed project validations as of June 30, 2022. EvNote = Evaluation Note; FY = fiscal year; IFC = International Finance Corporation; PCR = Project Completion Report.  ata Sources and Coverage, Multilateral Investment Guarantee Agency Table B.4. D FY12 FY13 FY14 FY15 FY16 FY17 FY18 FY19 FY20 FY21 Total MIGA PER Validation 12 8 6 16 14 11 16 12 11 2 108 Notes and IEG PERs Source: Independent Evaluation Group data. Note: Based on completed validations as of June 30, 2022. FY = fiscal year; IEG = Independent Evaluation Group; MIGA = Multilateral Investment Guarantee Agency; PER = Project Evaluation Report.  ountries with Completion and Learning Review (CLR) Reviews Figure B.1. C since Fiscal Year 2013 Sources: World Bank Enterprise Data Catalog; World Bank Classification of Fragile and Conflict-Affected Situations. Note: Country characteristics are determined based on the ending fiscal year of the latest Country Partnership Framework period that the Independent Evaluation Group reviewed. IDA = International Development Association; IBRD = International Bank for Reconstruction and Development; FCS = fragile and conflict-affected situations. Independent Evaluation Group World Bank Group    121 Appendix C. Methodology for Identifying Drivers of Performance at the Project Level Past Results and Performance of the World Bank Group (RAP) reports analyzed drivers of performance by conducting regression analysis and identifying factors such as project characteristics (for example, length, volume, and quality of monitoring and evaluation); country context (measured by indicators such as Country Policy and Institutional Assessment score); and team leader experience. However, regression analysis could not provide sufficient insights on how these factors affect project outcome and was unable to investigate factors lacking measurability. Therefore, to better understand what matters for project performance, the RAP 2022 team decided to review the Lessons section of Implementation Completion and Results Report Reviews (ICRRs) for investment project financing projects closed since fiscal year 2013. The Lessons section provides reflections on Results and Performance of the World Bank Group 2022  Appendix C what works, what does not work, and why it matters for project effectiveness. The RAP used a mixed methods approach, applying unsupervised machine learning and content analysis on the extracted lessons from ICRR documents. The following sections detail the methodology. Portfolio identification. The analysis focuses on World Bank investment project financing projects closed since fiscal year 2013 and evaluated by the Independent Evaluation Group as of April 21, 2022. The portfolio does not cover projects rated by the Independent Evaluation Group but with- out Outcome ratings. In addition, development policy financing and Pro- gram-for-Results projects were excluded, given their different nature. A total of 1,882 projects were identified covering all Global Practices and Regions. With the significant variance in text volume between ICRRs and Project Performance Assessment Reports, the RAP team limited the evaluation docu- ments to ICRRs only to avoid overrepresentation of projects with Project Performance Assessment Reports. 122 Data preparation. There are several steps in preparing the text data for analysis. The RAP team first conducted REST API query to acquire the URL links to all ICRR documents on the World Bank’s external site for publications, Documents and Reports. Then, the team conducted web scraping combined with regular expression to extract the Lessons section from each document. Finally, in the data preprocessing step, the RAP team removed stop words (which are commonly used words and sector-specific words irrelevant for the analysis) and lemmatized words to their base forms. Identification of topics and keywords with machine learning. Given the large size of the text data, it was impractical to review all text data manually. Therefore, the RAP team conducted text mining to identify possible topics and keywords in lessons from ICRRs, deploying three unsupervised machine learning algorithms: Latent Dirichlet Allocation, KeyBERT, and BERTopic. A popular topic modeling method, Latent Dirichlet Allocation is a generative statistical model that treats a document as a mixture of topics attributable to the probability of words. KeyBERT and BERTopic are competitive novel approaches developed in 2020 that both involve Bidirectional Encoder Representations from Transformers word embeddings, but they differ in dimensionality reduction approach.1 During the analysis, the team tested various scenarios on the maximum number of words in a phrase, the number of topics, and the degree of diversity among topics. The machine learning produced outputs including keywords and keyphrases (all algorithms) Independent Evaluation Group World Bank Group    123 and representative sentences for topics (Latent Dirichlet Allocation and BERTopics). Drill-down into topics for further examination. After reviewing extract- ed keywords and keyphrases and representative sentences, several topics emerged as drivers of project performance. Although these topics are not new, the findings serve to “validate” existing knowledge about the drivers of project success and point to future actions. The following topics were identi- fied by machine learning: » Local engagement » Realism » Adaptability and flexibility » Sustainable outcome » Collaboration and partnership among stakeholders and donors » Multisector approach » Performance-based mechanism » Monitoring and evaluation quality » Procurement efficiency » Funding availability and adequacy » Innovation Drill-down into topics for further examination. Of these topics, the RAP team purposively selected three for a deeper drill-down: local engagement, realism, and adaptability and flexibility. The general criterion that guided this selection was the strategic importance of the topics in terms of overlap with findings from other Independent Evaluation Group evaluations while being relatively underresearched. To gain a deeper understanding of each topic, the team conducted content analysis combining deductive and induc- Results and Performance of the World Bank Group 2022  Appendix C tive approaches. The analysis started with searching keywords relevant to each selected topic among all project lessons, including words identified by machine learning and words added manually. The search was iterative, with more words added and the search condition refined after reviewing search results. After the search, the team manually reviewed the identified text seg- ments and derived several key themes under each topic: » Local engagement. Engaging local stakeholders and utilizing local expertise can enable projects to identify and target the needs in the field. Effective and sufficient communications and consultation with communities, with respect for local languages and customs, are essential for avoiding misunderstanding, promoting awareness, and building trust and buy-in. » Realism. Early preparatory work (such as through feasibility studies and capacity assessments) is critical for ensuring implementation readiness. In-depth sector knowledge and understanding of the political economy, especially in countries experiencing instability, are necessities at the 124 appraisal stage. Projects with too many implementing parties could involve cumbersome administrative procedures and coordination. An incremental phased approach can help avoid project complexity and improve the chance of success. » Adaptability and flexibility. Flexibility and contingency planning built into the project design are crucial for interventions in a dynamic context, espe- cially in fragile and conflict-affected situation countries. They can allow rapid adaptation to shifts in client needs and shifts resulting from new policies and institutional structures when a new government is elected. Actively tracking progress and detecting issues in the field can allow timely corrective action, avoid delays, and reduce efficiency losses. A stable World Bank team is critical to maintaining project continuity and ensuring appropriate course correction. Advantages and Disadvantages of Using Machine Learning This RAP presents an exploratory analysis of unsupervised machine learning to understand the drivers of project performance. It also notes the pros and cons of using machine learning for identifying performance drivers. In addition to identifying less-quantifiable factors, the main advantage of the unsupervised machine learning approach used in this RAP is that it is a neutral way of identifying topics since no human specification or coding of possible drivers or topics was required, whereas human coders may be subject to unconscious biases. Furthermore, machine learning can also be Independent Evaluation Group World Bank Group    125 a cost-effective way to examine large amounts of digital data. A possible drawback is that machine learning may miss topics if there are synonyms used in different reports to refer to the same topic. As the Lessons section in ICRRs becomes more robust, this RAP’s experience shows machine learning to be a promising approach that can be used for more extensive analysis of drivers of performance in the future. 1  Both KeyBERT and BERTopic use Bidirectional Encoder Representations from Transformers embeddings, a deep learning technique developed and released by Google in 2018 for natural language processing. After embeddings, KeyBERT applies cosine similarity to identify words and phrases that are the most similar to the documents, and BERTopic clusters embeddings with UMAP and HDBSCAN and extracts topics using class-based trust fund-IDF procedure. UMAP refers to Uniform Manifold Approximation and Projection, a dimension reduction tech- nique constructed based on Riemannian geometry and algebraic topology. HDBSCAN stands for Hierarchical Density-Based Spatial Clustering of Applications with Noise, a hierarchical clustering algorithm. Trust fund-IDF, short for term frequency-inverse document frequency, measures how relevant a word is to a document in a collection of documents. Results and Performance of the World Bank Group 2022  Appendix C 126 Appendix D. Project Rating Trends and Patterns The lending project Outcome ratings remained high in fiscal year (FY)21 after the jump in FY20. Among investment project financing (IPF) and Pro- gram-for-Results (P4R) projects closed in FY21, 85 percent were rated mod- erately satisfactory or higher, the highest since FY11 (figure 2.1, panel a). The average Outcome rating is 4.4 on a six-point scale, compared with 4.3 in FY20. There are more fluctuations in development policy financing project ratings, with the average rating improving from 4.0 in FY20 to 4.3 in FY21 and the percentage rated moderately satisfactory or higher declining from 88 percent in FY20 to 86 percent in FY21 (figure 2.1, panel b). However, such shifts are unlikely to be stable, given the small number of development poli- cy financing projects in both periods (24 projects in FY20 and seven in FY21). Outcome ratings also improved at the two ends of the rating spectrum. Con- tinuing the trend from FY19 to FY20, among IPF and P4R projects, there is an increase in projects rated satisfactory or higher, from 48 percent in FY20 to 49 percent in FY21. Meanwhile, projects rated unsatisfactory or lower dropped from 5 percent to 2 percent, with no projects rated highly unsatis- factory in FY21. For development policy financing projects, 40 percent of the Independent Evaluation Group World Bank Group    127 projects were rated satisfactory or higher in FY21, compared with 25 percent in FY20 (figure D.1, panel b). Of the four Practice Groups, only Human Development and Sustainable De- velopment projects registered an improvement in the mean outcome rating between FY20 and FY21 (figure D.2, panel a). By contrast, ratings declined in both the Infrastructure and the Equitable Growth, Finance, and Institutions Practice Groups (figure D.2, panel b). As shown in figure C.3, the decomposi- tion analysis revealed that each Practice Group’s contribution to the overall change in ratings is greatly affected by the shift in its relevant portfolio size. For example, the contribution from Human Development projects is amplified by a vastly expanded portfolio share, while for Sustainable Development, a reduced portfolio size outweighs its improvement in rating. Figure D.1. Distribution of the World Bank Project Outcome Ratings Results and Performance of the World Bank Group 2022  Appendix D Source: Independent Evaluation Group data. 128  orld Bank Investment Project Financing and P4R Projects Figure D.2. W Outcome Rated Moderately Satisfactory or Higher by Prac- tice Group Independent Evaluation Group World Bank Group    129 Source: Independent Evaluation Group data. Note: The graphs include only investment project financing and Program-for-Results projects. EFI = Equitable Growth, Finance, and Institutions; HD = Human Development; INFRA = Infrastructure; IPF = investment project financing; P4R = Program-for-Results; SD = Sustainable Development.  ontributions to the Shift in World Bank IPF and P4R Projects Figure D.3. C Outcome Ratings between Fiscal Years 2020 and 2021, by Practice Group Source: Independent Evaluation Group data. Note: The graphs include only investment project financing and Program-for-Results projects. EFI = Equitable Growth, Finance, and Institutions; HD = Human Development; INFRA = Infrastructure; IPF = investment project financing; P4R = Program-for-Results; SD = Sustainable Development. Of the seven Regions, four (Eastern and Southern Africa, East Asia and Pacif- ic, Europe and Central Asia, and South Asia) received higher ratings in FY21 than in FY20, with Eastern and Southern Africa making the biggest leap. As Results and Performance of the World Bank Group 2022  Appendix D for contribution to the shift in the overall trend, the increase in ratings in Europe and Central Asia is overshadowed by a large drop in portfolio size (from 18 to 11 percent of the overall portfolio), while the ratings rise in East- ern and South Africa and South Asia is complemented by a growth in portfo- lio share (figure D.5). 130  hift in World Bank IPF and P4R Projects Outcome Rated Figure D.4. S Moderately Satisfactory or Higher by Region between Fiscal Years 2020 and 2021 Source: Independent Evaluation Group data. Note: The graphs include only IPF and Program-for-Results projects. IPF = investment project financing; P4R = Program-for-Results.  ontributions to the Shift in World Bank IPF and P4R Projects Figure D.5. C Outcome Ratings between Fiscal Years 2020 and 2021 by Region Independent Evaluation Group World Bank Group    131 Source: Independent Evaluation Group data. Note: The graphs include only IPF and P4R projects. IPF = investment project financing; P4R = Pro- gram-for-Results. The share of IPF and P4R projects with Outcome ratings of MS+ declined marginally from 84 percent in FY20 to 83 percent in FY21 in Internation- al Development Association (IDA) countries, reversing the improvement between FY19 and FY20 (figure D.6, panel a). The share of projects rated moderately satisfactory or higher rose from 84 to 89 percent in non–frag- ile and conflict-affected situations (FCS) countries but declined from 82 to 74 percent in FCS (figure D.6, panel b). The widening gap between FCS and non-FCS suggests a greater influence of country context on project perfor- mance in FY21. It also aligns with the correlation between project Outcome rating and Country Policy and Institutional Assessment country score, which is statistically significant in FY21 but not in FY20. The decline of Outcome ratings in IDA and FCS occurred along with the portfolio expansion in these countries. From FY20 to FY21, projects from IDA countries rose from 49 percent to 51 percent of the total IPF and P4R portfo- lio, and the share of projects from FCS also grew from 22 percent to 24 per- cent. Regarding the overall level rating shift from FY20 to FY21 (figure D.7), projects from FCS IDA countries still contributed positively, with the jump in portfolio size offsetting the decline in rating. Results and Performance of the World Bank Group 2022  Appendix D 132  orld Bank IPF and P4R Projects Outcome Rated Moderately Figure D.6. W Satisfactory or Higher by Country Type Independent Evaluation Group World Bank Group    133 Source: Independent Evaluation Group data. Note: The graphs include only IPF and P4R projects. Regional projects and projects from countries with- out lending group status are excluded. Note not all countries with World Bank projects have lending group type defined. FCS = fragile and conflict-affected situations; IBRD = International Bank for Re- construction and Development; IDA = International Development Association; IPF = investment project financing; P4R = Program-for-Results.  ontributions to the Shift in World Bank IPF and P4R Projects Figure D.7. C Outcome Ratings between Fiscal Years 2020 and 2021 by Lending Group and FCS Status Source: Independent Evaluation Group data. Note: The graphs include only IPF and P4R projects. Regional projects and projects from countries with- out lending group status are excluded. Note not all countries with World Bank projects have lending group type defined. FCS = fragile and conflict-affected situations; IBRD = International Bank for Re- Results and Performance of the World Bank Group 2022  Appendix D construction and Development; IDA = International Development Association; IPF = investment project financing; P4R = Program-for-Results. Although large projects outperformed small ones (in size), the shift in Out- come ratings from FY20 to FY21 appeared different (figure D.8). Project Outcome ratings increased among small and medium-size projects, from 77 to 88 percent of projects, 83 to 86 percent, and 87 to 93 percent for proj- ects in the range of less than $25 million, $25 million to $50 million, and $50 million to $100 million, respectively. The opposite occurred for large projects ($100 million or more), with ratings declining from 89 percent of projects to 86 percent. Projects in the $25 million to $50 million range contributed positively to the overall shift of ratings from FY20 to FY21 with both improved ratings and an expanded portfolio. Projects of $100 million or more also made a positive contribution because of the rise in portfolio size, despite the decrease in rating. 134  ontributions to the Shift in World Bank IPF and P4R Projects Figure D.8. C Outcome Ratings between Fiscal Years 2020 and 2021 by Project Size Source: Independent Evaluation Group data. Note: The graphs include only IPF and Program-for-Results projects. IPF = investment project financing; P4R = Program-for-Results. Projects rated by Independent Evaluation Group so far are inadequate to explain the COVID-19 pandemic’s impact on project performance. The anal- ysis covered projects that were a response to COVID-19 if meeting any one of the following conditions: tagged with the “COVID” emergency response code, tagged with a “COVID” crisis response code, tagged with the “Pandemic Response” theme code, or having “covid” or “corona” in the project name or objective or components. Among the projects rated by the Independent Eval- Independent Evaluation Group World Bank Group    135 uation Group, just 10 projects were identified as responding to COVID-19, of which 7 received satisfactory Outcome ratings, and 3 received moderate- ly satisfactory Outcome ratings. There may be some other projects that do not meet these criteria but which nevertheless contain activities addressing issues relevant to COVID-19. Identifying them would require a detailed re- view of project documents, which is beyond the scope of the analysis for this Results and Performance of the World Bank Group (RAP) report. The RAP also examined the relationship between the pre-COVID-19 CLR Review ratings and the project performance ratings for all projects (not just COVID-19 response projects) during the COVID-19 time period. This was done by first identifying the latest development outcome and Bank Group Performance ratings from CLR Reviews covering a period before FY21. Next, the RAP identified the latest Implementation Status and Results Report (ISR) ratings on progress toward achievement of project development objective and overall implementation progress from ISRs completed since February 1, 2020, to June 2, 2022. The project-level data were aggregated at the country level as the average ratings and share of projects rated satisfactory or high- er (S+). The RAP found that there is a strong positive correlation between the CLR Review ratings and project ISR ratings, with the highest correlation coefficient and statistical significance found between country development outcome and project ISR development outcome ratings, as shown in ta- ble D.1. Overall, the pre-COVID-19 CLR Review ratings are a good predictor of project performance during the COVID-19 time period.  orrelation between Completion and Learning Review (CLR) Table D.1. C Review Ratings and ISR Ratings CLR Review Rating World Bank Group Development Outcome Performance Coefficient p-value Coefficient p-value DO average 0.349 0.000 0.233 0.014 Results and Performance of the World Bank Group 2022  Appendix D ISR rating DO S+ (%) 0.418 0.000 0.221 0.020 IP average 0.315 0.001 0.247 0.009 IP S+ (%) 0.324 0.001 0.226 0.017 Sources: World Bank Enterprise Data Catalog; Independent Evaluation Group data. Note: CLR = Completion and Learning Review; DO = development outcome, IP = implementation prog- ress; ISR = Implementation Status and Results Report; S+ = satisfactory or higher; WBG = World Bank Group. 136 Appendix E. The International Finance Corporation and the Multilateral Investment Guarantee Agency International Finance Corporation and Multilateral Investment Guarantee Agency Contribution to World Bank Group’s Outcome at the Country Level Methodology. The assessment of the contributions of the International Finance Corporation (IFC) and the Multilateral Investment Guarantee Agency (MIGA) to the World Bank Group’s country-level outcomes was based on the assessment of 65 CLR Reviews of 47 countries. The sampling was described in the Concept Note for the Results and Performance of the World Bank Group (RAP) 2022 report. The CLR Reviews for each sampled country were assessed for each objective in the Country Partnership Frameworks of the IFC and MIGA projects that were deemed to have contributed to the achievement of the Country Partnership Framework objective. The performance of these ob- jectives and the characteristics of the IFC and MIGA projects were analyzed, Independent Evaluation Group World Bank Group    137 and the findings synthesized. In addition to reviewing each objective that mentioned IFC and MIGA, the CLR Reviews sections on Bank Group Internal Cooperation were also assessed. There were 780 objectives in the 65 CLR Reviews reviewed, an average of 12 objectives per CLR Review. More than 60 percent of the CLR Reviews in the sample had programs ending in fiscal year (FY)16 or earlier. The older CLR Reviews with programs that ended in FY16 or earlier had an average of 13 objectives per CLR Review, compared with 11 for the latter ones. Tables E.1 and E.2 present the breakdown of CPF objectives and the respective contri- butions of IFC and MIGA. 138 Results and Performance of the World Bank Group 2022  Appendix E nternational Finance Corporation Contributions to Country Table E.1. I Partnership Framework Objectives Number of Number Number CPF Objectives of CPF of CPF Supported by Share of IFC Share to Objectives Objectives Combined IFC Contribution by Number Total CPF Supported Supported by Investments Total Number CPF Objective CPF Objective of CPF Objectives by IFC IFC Advisory and Advisory of IFC Classification Classification Objectives (%) Investments Services Services Contribution (%) Business Envi- 49 26 7 29 13 49 26 ronment Access to Fi- 44 24 16 15 13 44 24 nance Energy 32 17 14 10 8 32 17 Agriculture, 21 11 6 6 8 20 11 Agribusiness and Rural De- velopment Infrastructure 18 10 8 4 5 17 9 Health 9 5 4 3 2 9 5 Education 7 4 3 1 3 7 4 Other 6 3 1 3 3 7 4 Total 186 100 59 71 55 185 100 Source: Independent Evaluation Group data. Note: CPF = Country Partnership Framework; IFC = International Finance Corporation.  ultilateral Investment Guarantee Agency Contribution to Table E.2. M Country Partnership Framework Objectives CPF Objectives Share of MIGA Share of Total Supported Contribution by CPF Number by MIGA CPF Objective CPF Objective Objectives of CPF Guarantees Classification Classification (no.) Objectives (%) (no.) (%) Business 49 26 2 10 Environment Access to 44 24 2 10 Finance Energy 32 17 9 45 Agriculture, 21 11 2 10 Agribusiness and Rural Development Infrastructure 18 10 4 20 Health 9 5 1 5 Education 7 4 0 0 Other 6 3 0 0 Total 186 100 20 100 Source: Independent Evaluation Group data. Independent Evaluation Group World Bank Group    139 Note: CPF = Country Partnership Framework International Finance Corporation and Multilateral Investment Guarantee Agency Project-Level Performance International Finance Corporation Investments The graphs and matrixes in this section of the appendix provides additional information on the results and performance of IFC investment projects eval- uated in calendar years 2012 to 2021, which total 809 projects with complet- ed Validation Notes from the Independent Evaluation Group. One project is rated no opinion possible on development outcome, IFC’s investment out- come, and in IFC’s supervision and administration and was not included in the count. Data are based on the Expanded Project Supervision Report rating database as of June 30, 2022. Figure E.1 presents the annual development outcome ratings of IFC investment projects evaluated in calendar year 2012 to 2021.  ear-on-Year Disaggregated Development Outcome Ratings, Figure E.1. Y by Number of Projects, Calendar Year 2012–21 Results and Performance of the World Bank Group 2022  Appendix E Source: Independent Evaluation Group data. Note: HS = highly successful; HU = highly unsuccessful; MS+ = mostly successful or higher; MS = mostly successful; MU = mostly unsuccessful; SU = successful; US = unsuccessful. IFC industry group projects evaluated in calendar years 2012 to 2021 included 52 in Disruptive Technologies and Funds, 296 in the Financial Institutions Group, 191 in Infrastructure and Natural Resources, and 270 in Manufacturing, Agribusiness, and Services. Figure E.2 presents the trend of evaluated IFC investment projects rated mostly successful or higher on development outcome from calendar year 2012 to 2021. An Infrastructure and Natural Resources project has a rating of no opinion possible on development outcome and was excluded from the rating count. 140  evelopment Outcome Ratings, by International Finance Figure E.2. D Corporation’s Industry Group, 2012–21 Source: Independent Evaluation Group data. Note: CDF = Disruptive Technologies and Funds; FIG = Financial Institutions Group; IFC = International Finance Corporation; INR = Infrastructure and Natural Resources; MAS = Manufacturing, Agribusiness, and Services; MS+ = mostly successful or higher. IFC’s Work Quality and Work Quality Indicators Ratings Figure E.3, panels a, b, and c present ratings of IFC’s work quality and its two subindicators: screening, appraisal, and structuring; and supervision and administration for all investment projects evaluated in 2012–21. IFC’s work quality; screening, appraisal, and structuring; and supervision and adminis- tration are rated on a four-point scale: excellent, satisfactory, partly unsatis- factory, and unsatisfactory. A total of 804 evaluated projects in 2012–21 have Independent Evaluation Group World Bank Group    141 IFC’s work quality; screening, appraisal, and structuring; and supervision and administration ratings. The five projects that have a no opinion possible rating on IFC’s work quality; and screening, appraisal, and structuring; and the six projects that have no opinion possible rating in supervision and administration were excluded from the graphs. IFC’s work quality; screening, appraisal, and structuring; and supervision and administration are rated on a four-point scale: excellent, satisfactory, partly unsatisfactory, and unsatis- factory. nternational Finance Corporation’s Work Quality Ratings, Figure E.3. I 2012–21 Results and Performance of the World Bank Group 2022  Appendix E Source: Independent Evaluation Group data. Note: AFR = Africa; CAT = Central Asia and Türkiye; EAP = East Asia and Pacific; EUR = Europe; IFC = Inter- national Finance Corporation; FCS = fragile and conflict-affected situations; IFC = International Finance Corporation; LAC = Latin America and the Caribbean; ME = Middle East; S+ = satisfactory or higher rating; SA = South Asia; SAS = screening, appraisal, and structuring; SUP = supervision and administration; XPSR 142 = Expanded Project Supervision Report; WQ = IFC’s work quality. Afghanistan and Pakistan are grouped in the Middle East region in IFC’s September 2021 regional classification. Figure E.3, panel c compares IFC’s work quality rating and its two indicators (screening, appraisal, and structuring; and supervision and administration) for evaluated projects in fragile and conflict-affected situations (FCS) and non-FCS. IFC’s work quality and its two indicators are rated on a four-point scale: excel- lent, satisfactory, partly unsatisfactory, and unsatisfactory. Relationship between Development Outcome and International Finance Corporation’s Work Quality Total project count for figure E.4, panel a is 803, of which 14 are rated highly successful and 69 highly unsuccessful in development outcome. For fig- ure E.4, panel b, five projects are rated highly successful in development outcome and excellent work quality, and 27 projects have highly unsuccess- ful and unsatisfactory work quality ratings. development outcome is rated on a six-point scale from highly successful to highly unsuccessful, while IFC’s work quality is rated on a four-point scale: excellent, satisfactory, partly un- satisfactory, and unsatisfactory. nternational Finance Corporation’s Development Outcome Figure E.4. I and Work Quality Independent Evaluation Group World Bank Group    143 Source: Independent Evaluation Group. Note: E = excellent; HS = highly successful; HU = highly unsuccessful; IFC = International Finance Corpo- ration; U = unsuccessful. International Finance Corporation’s Role and Contribution Ratings The total number of evaluated IFC investment projects in 2012–21 with ratings on role and contribution or additionality is 803. Five projects have ratings of no opinion possible and were excluded. IFC’s role and contribution is rated on a four-point rating scale: excellent, satisfactory, partly unsatisfac- tory, and unsatisfactory. Figure E.5 presents the rating trend with the break- down of each rating on an annual basis.  isaggregated International Finance Corporation’s Role and Figure E.5. D Contribution Ratings, Year-on-Year, 2012–21. Results and Performance of the World Bank Group 2022  Appendix E Source: Independent Evaluation Group data. Note: E= excellent; PU = partly unsatisfactory; S+ = satisfactory or higher; S = satisfactory; U = unsatisfac- tory. Figure E.6 compares the share of satisfactory or higher ratings of IFC invest- ment projects in FCS and non-FCS self-evaluated and validated in calendar years 2012 to 2021. The total number of self-evaluated and validated IFC investment projects reviewed in figure E.6 is 809, of which 56 are in FCS and 753 are in non-FCS. IFC’s role and contribution is rated on a four-point scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory. 144 Figure E.6. I nternational Finance Corporation’s Role and Contribution Ratings, FCS and Non-FCS, 2012–21 Source: Independent Evaluation Group data. Note: FCS = fragile and conflict-affected situations; S+ = satisfactory or higher. Relationship Between Development Outcome and International Finance Corporation’s Role and Contribution or Additionality Total project count for figure E.7, panel a is 803, of which 14 are rated highly successful and 69 highly unsuccessful on development outcome. For fig- ure E.7, panel b, there are three projects with highly successful development Independent Evaluation Group World Bank Group    145 outcome and excellent IFC’s additionality ratings and 32 projects with highly unsuccessful development outcome and unsatisfactory IFC’s role and contri- bution or additionality ratings. Development outcome is rated on a six-point scale from highly successful to highly unsuccessful, while IFC’s role and contribution or additionality is rated on a four-point scale: excellent, satis- factory, partly unsatisfactory, and unsatisfactory.  evelopment Outcome and International Finance Figure E.7. D Corporation Additionality/Role and Contribution, 2012–21 Source: Independent Evaluation Group. Note: E = excellent; HS = highly successful; HU = highly unsuccessful; IFC = International Finance Corpo- ration; U = unsuccessful. Results and Performance of the World Bank Group 2022  Appendix E International Finance Corporation’s Investment Outcome Ratings The total number of evaluated IFC investment projects in calendar years 2012–21 with ratings on IFC’s investment outcome is 808. One project has a rating of no opinion possible and was excluded. IFC’s investment outcome is rated on a four-point rating scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory. Figure E.8 presents the rating trend with the breakdown of each rating on an annual basis from calendar years 2012 to 2021. 146  isaggregated IFC’s Investment Outcome Ratings, Year-on- Figure E.8. D Year, 2012–21 Source: Independent Evaluation Group data. Note: E = excellent; PU = partly unsatisfactory; S = satisfactory; U = unsatisfactory; S+ = satisfactory or higher. Figure E.9, panel a presents IFC’s investment outcome rating trend of evaluated IFC investment projects from calendar years 2012 to 2021 on a three-year rolling average. The total number of evaluated IFC investment projects reviewed in this graph is 808 (one project was excluded because of a rating of no opinion possible). IFC’s investment outcome is rated on a four-point rating scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory. Independent Evaluation Group World Bank Group    147 nternational Finance Corporation’s Investment Outcome Figure E.9. I Ratings, 2012–21 Results and Performance of the World Bank Group 2022  Appendix E Source: Independent Evaluation Group data. Note: E = excellent; FCS = fragile and conflict-affected situations; PU = partly unsatisfactory; S+= satisfac- tory or higher; S = satisfactory; U = unsatisfactory. Figure E.9, panel b compares the 10-year rating trend of IFC’s investment outcome of evaluated investment projects in FCS and non-FCS on a three- year rolling average. Total count of evaluated IFC investment projects com- pleted in 2012–21 and reviewed for this graph is 808, of which 56 are in FCS and 752 in non-FCS. IFC’s investment outcome is rated on a four-point rating scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory. 148 Relationship Between Development Outcome and International Finance Corporation’s Investment Outcome The total count of evaluated projects in figure E.10, panel a is 808, of which 14 are rated highly successful and 69 highly unsuccessful on development outcome. For figure E.10, panel b, there are 7 projects with highly successful development outcome and excellent IFC’s investment outcome ratings and 58 projects with highly unsuccessful development outcome and unsatisfac- tory IFC’s investment outcome ratings. Development outcome is rated on a six-point scale from highly successful to highly unsuccessful, and IFC’s investment outcome is rated on a four-point scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory.  elationship between International Finance Corporation’s Figure E.10. R Development Outcome and Investment Outcome Independent Evaluation Group World Bank Group    149 Source: Independent Evaluation Group. Note: E = excellent; HS = highly successful; HU = highly unsuccessful; IFC = International Finance Corpo- ration; U = unsuccessful. International Finance Corporation Advisory Services The graphs and matrixes in this section of the appendix provide additional information on the results and performance of IFC advisory services projects self-evaluated by IFC and validated by Independent Evaluation Group in fiscal years 2012 to 2021, which totals 573 projects. One project is rated Not Applicable on Development Effectiveness and was not included in the count. Data are based on the Project Completion Reports Rating database as of June 30, 2022. Not all Project Completion Reports for 2020 and 2021 Project Completion Reports Programs have been delivered as of the RAP 2022 cut-off date.  isaggregated Development Effectiveness Ratings, Year-on- Figure E.11. D Year, 2012–21 Results and Performance of the World Bank Group 2022  Appendix E Source: Independent Evaluation Group data. Note: Eight projects are rated highly successful, and 13 are rated highly unsuccessful on their devel- opment effectiveness. HS = highly successful; HU = highly unsuccessful; MS = mostly successful; MU = mostly unsuccessful; SU = successful; US = unsuccessful. MS+ = mostly successful or higher By region. Figure E.12 presents the Development Effectiveness ratings of evaluated IFC advisory services projects from fiscal year 2012 to 2021 based on IFC’s eight new region classifications. In 2012–21, the number of self-evalu- ated and validated projects by region are as follows: Africa (165), Central Asia and Türkiye (18), East Asia and Pacific (91), Europe (59), Latin America and the Caribbean (83), Middle East (51), and South Asia (89). The graph excludes 17 projects implemented in several regions or globally (World projects). 150 Figure E.12. Development Effectiveness Rating, By Region, 2012–21 Source: Independent Evaluation Group data. Note: AFR = Africa; CAT = Central Asia and Türkiye; EAP = East Asia and Pacific; EUR = Europe; LAC = Latin America and the Caribbean; ME = Middle East; SA = South Asia. Afghanistan and Pakistan were grouped in the Middle East region in International Finance Corporation’s new regional classification.  hare of MS+ Development Effectiveness Rating, by FCS and Figure E.13. S Non-FCS Independent Evaluation Group World Bank Group    151 Source: Independent Evaluation Group data. Note: FCS = fragile and conflict-affected situations; MS+ = mostly successful or higher. Of the 573 advisory services projects evaluated in fiscal year 2012–21, 350 are in International Development Association (IDA) countries, and 223 are in non-IDA countries. Figure E.14 compares the development effectiveness rating trend for evaluated projects in IDA and non-IDA countries.  hare of MS+ Development Effectiveness Rating, by IDA and Figure E.14. S Non-IDA Source: Independent Evaluation Group data. Note: IDA = International Development Association; MS+ = mostly successful or higher. International Finance Corporation Advisory Services Work Quality Ratings The work quality rating of IFC advisory services projects began in 2015. Fig- Results and Performance of the World Bank Group 2022  Appendix E ures E.15, E.16, and E.17 are based on 346 evaluated advisory projects from 2015–21. Assessing and rating the work quality of IFC advisory services proj- ects began in 2015. IFC work quality is rated on a four-point scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory. 152  isaggregated International Finance Corporation Work Figure E.15. D Quality Ratings, Year-on-Year, 2012–21 Source: Independent Evaluation Group data. Note: E = excellent; IFC = International Finance Corporation; PU = partly unsatisfactory; S+ = satisfactory or higher; S = satisfactory; U = unsatisfactory. nternational Finance Corporation Work Quality Ratings, Figure E.16. I Share of Satisfactory or Higher Ratings, by IFC Regions, 2012–21 Independent Evaluation Group World Bank Group    153 Source: Independent Evaluation Group data. Note: AFR = Africa; CAT = Central Asia and Türkiye; EAP = East Asia and Pacific; EUR = Europe; IFC = Inter- national Finance Corporation; LAC = Latin America and the Caribbean; ME = Middle East; S+ = satisfac- tory or higher; SA = South Asia. Afghanistan and Pakistan were grouped under the Middle East region in IFC’s September 2021 regional classification. nternational Finance Corporation Work Quality Ratings, Figure E.17. I Share of Satisfactory or Higher Ratings, FCS and Non-FCS, 2012–21 Source: Independent Evaluation Group data. Note: FCS = fragile and conflict-affected situations; IFC = International Finance Corporation; S+ = satisfac- tory or better. Relationship between Development Effectiveness and International Finance Corporation Work Quality Results and Performance of the World Bank Group 2022  Appendix E The work quality rating of IFC advisory services projects began in 2015. Total project count for figure 3.18, panel a is 346, of which 5 have highly success- ful and 12 highly unsuccessful ratings on development effectiveness. For figure 3.18, panel b, 4 projects are rated highly successful on development effectiveness and excellent in IFC work quality, and 9 projects have highly unsuccessful ratings on development effectiveness and unsatisfactory work quality ratings. Development effectiveness is rated based on a six-point rat- ing scale from highly successful to highly unsuccessful, and IFC work quality rating is based on a four-point scale (excellent, satisfactory, partly unsatis- factory and unsatisfactory). 154  elationship between Development Effectiveness and Figure E.18. R International Finance Corporation Work Quality Source: Independent Evaluation Group. Note: The two extreme ends of the development effectiveness rating scale are highly successful and highly unsuccessful. The two extreme ends of IFC work quality rating scale are excellent and unsat- isfactory ratings. E = excellent; HS = highly successful; HU = highly unsuccessful; IFC = International Finance Corporation; U = unsuccessful. International Finance Corporation Role and Contribution Figure E.19 presents the IFC role and contribution rating trend from 574 self-evaluated and validated IFC advisory services projects from fiscal years 2012 to 2021. IFC role and contribution is rated along a four-point scale: Independent Evaluation Group World Bank Group    155 excellent, satisfactory, partly unsatisfactory, and unsatisfactory.  isaggregated International Finance Corporation Role and Figure E.19. D Contribution Rating, Year-on-Year, 2012–21 Source: Independent Evaluation Group data. Note: E = excellent; IFC = International Finance Corporation; PU = partly unsatisfactory; S+ = satisfactory or higher; S = satisfactory; U = unsatisfactory. Relationship Between Development Effectiveness and Results and Performance of the World Bank Group 2022  Appendix E International Finance Corporation Role and Contribution The total project count for figure E.20, panel a is 573, of which 8 have highly successful and 13 highly unsuccessful ratings on development effectiveness. For figure E.20, panel b, 7 projects are rated highly successful on develop- ment effectiveness and excellent in IFC role and contribution, and 8 projects have highly unsuccessful ratings on development effectiveness and unsat- isfactory role and contribution ratings. development effectiveness is rated based on a six-point rating scale, and IFC role and contribution rating is based on a four-point scale. 156  elationship between Development Effectiveness and Figure E.20. R International Finance Corporation Role and Contribution Source: Independent Evaluation Group. Note: The two extreme ends of the development effectiveness rating scale are highly successful and highly unsuccessful. The two extreme ends of IFC work quality rating scale are excellent and unsat- isfactory ratings. E = excellent; HS = highly successful; HU = highly unsuccessful; IFC = International Finance Corporation; U = unsuccessful. Multilateral Investment Guarantee Agency The graphs and matrixes in this section of the appendix provide additional information on the results and performance of MIGA projects evaluated in FY12 to FY21, which totals 108. Two projects that were self-evaluated by MIGA Independent Evaluation Group World Bank Group    157 and validated by Independent Evaluation Group in the FY21 Project Evalua- tion Report program were included in the analysis of the 2016–21 results. 66 evaluated projects were reviewed for the 2016–21 MIGA project results. Starting in FY20, development outcome of MIGA projects were rated on a six-point scale similar to the development outcome ratings of IFC invest- ments projects. Before fiscal year 2020, development outcome was rated on a four-point scale: excellent, satisfactory, partly unsatisfactory, and unsat- isfactory. Figure E.21 presents the year-on-year breakdown of development outcome ratings of MIGA projects self-evaluated and validated in FY12–21, with the total number of MIGA projects evaluated indicated per FY.  isaggregated Development Outcome Ratings, Year-on- Figure E.21. D Year, 2012–21 Source: Independent Evaluation Group data. Note: Project Evaluation Reports of 108 MIGA projects evaluated in fiscal years 2012 to 2021 were reviewed in the analysis of MIGA’s project-level results and performance. E = excellent; PU = partly unsatisfactory; S+ = satisfactory or higher; S = satisfactory; U = unsatisfactory. Development Outcome Ratings by Multilateral Investment Results and Performance of the World Bank Group 2022  Appendix E Guarantee Agency Sector Group Figure E.22 presents the development outcome ratings of MIGA evaluated projects from FY12 to FY21, by MIGA sector group. 158  hare of Satisfactory or Higher Development Outcome Figure E.22. S Ratings, by MIGA Sector Group, 2012–21. Source: Independent Evaluation Group data. Note: AGS = Agribusiness, Manufacturing, and General Services; DO = development outcome; EEI = Energy and Extractive Industries; FINCAP = Finance and Capital Markets; INFRA = Infrastructure; MIGA = Multilateral Investment Guarantee Agency; S+ = satisfactory or higher.  hare of Satisfactory or Higher Development Outcome Figure E.23. S Ratings, by Region, 2012–21 Independent Evaluation Group World Bank Group    159 Source: Independent Evaluation Group data. Note: Of the 108 self-evaluated and validated in 2012–21 as of June 30, 2022, only two projects are in the South Asia Region. Both have satisfactory or higher development outcome ratings. EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MENA = Middle East and North Africa; S+ = satisfactory or higher; SAR = South Asia; SSA = Sub-Saharan Africa. Figure E.24 presents the ratings on MIGA’s role and contribution based on 106 MIGA evaluated projects from FY12 to FY21, by MIGA sector group. This excludes one project rated no opinion possible in 2014. MIGA’s role and con- tribution is rated on a four-point rating scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory.  hare of Satisfactory or Higher Rating, MIGA’s Role and Con- Figure E.24. S tribution, by Sector Group, 2012–21. Results and Performance of the World Bank Group 2022  Appendix E Source: Independent Evaluation Group data. Note: Of the 108 self-evaluated and validated projects in 2012–21 as of June 30, 2022, one evaluated project was rated no opinion possible on MIGA’s role and contribution in 2014. AGS = Agribusiness, Man- ufacturing and General Services; DO = development outcome; EEI = Energy and Extractive Industries; FINCAP = Finance and Capital Markets; INFRA = Infrastructure; MIGA = Multilateral Investment Guarantee Agency; S+ = satisfactory or higher. Figure E.25 presents the ratings on MIGA’s role and contribution based on 108 MIGA evaluated projects from FY12 to FY21, by Region. MIGA’s role and contribution is rated on a four-point rating scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory. 160  hare of Satisfactory or Higher Rating, MIGA’s Role and Con- Figure E.25. S tribution, by Region, 2012–21. Source: Independent Evaluation Group data. Note: Of the 108 evaluated projects in 2012–21, one evaluated project was rated no opinion possible on MIGA’s role and contribution in 2014. Only two projects in the South Asia Region were self-evaluated and validated from fiscal years 2012–21. Both have satisfactory or higher ratings on MIGA’s role and con- tribution. One self-evaluated and validated “World” project was excluded from figure E.25. EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MENA = Middle East and North Africa; MIGA = Multilateral Investment Guarantee Agency; S+ = satisfactory or higher; SAR = South Asia region; SSA = Sub-Saharan Africa. Relationship Between Development Outcome and Work Quality and Development Outcome and Multilateral Investment Guarantee Agency’s Role and Contribution Independent Evaluation Group World Bank Group    161 The total project count for figure E.26, panel a is 108, of which only 4 proj- ects were rated excellent on development outcome and on MIGA’s role and contribution. No project was rated unsatisfactory in development outcome and in MIGA’s role and contribution. For figure E.26, panel b, of the 108 evaluated projects with ratings, only 1 project was rated excellent in both de- velopment outcome and in MIGA’s appraisal, underwriting, and monitoring (referred to in this report as MIGA’s work quality). No project had a rating of unsatisfactory on development outcome and in MIGA’s work quality. For this RAP 2022 report, rating on development outcome used the four-point rating scale: excellent, satisfactory, partly satisfactory, and unsatisfactory. MIGA’s work quality is also based on a similar four-point rating scale.  elationship between Development Outcome and Work Figure E.26. R Quality and Development Outcome and MIGA’s Role and Contribution, Fiscal Years 2012–21 Source: Independent Evaluation Group. Note: MIGA = Multilateral Investment Guarantee Agency. Results and Performance of the World Bank Group 2022  Appendix E 162 Appendix F. World Bank Group Support for Country Partnership Framework Objectives The 55 Completion and Learning Review (CLR) Reviews that provide the database for the analysis include 36 reviews (one for each country) from the representative sample of 50 countries between 2016 and 2021, and 19 reviews (one for each country) from outside the sample between 2019 and 2022 (see tables F.1 and F.2). The inclusion of the 19 CLR Reviews from outside the sample is in response to the exclusion of 31 CLR Reviews for 14 of the sample countries for lack of either assessment for the achievement of objec- tives or nonidentification of World Bank Group projects to support each Country Partnership Framework objective. Although it makes the resulting 55 countries nonrepresentative of Bank Group clients, it ensures better capture of the current Bank Group practices by including all the CLR Reviews during the past four years.  ompletion and Learning Review (CLR) Reviews from the Sam- Table F.1. C ple of 50 Countries Country 2013 2014 2015 2016 2017 2018 2019 2020 2021 Independent Evaluation Group World Bank Group    163 Argentina No 2 Yes Armenia No 1 Yes Belarus No 1 Yes Bhutan No 2 Yes Bolivia No 2 Bosnia and Herze- No 2 govina Brazil Yes Bulgaria Yes Burundi No 1 Yes Cameroon Yes China No 1 Yes Comoros Yes Congo, Rep. of No 1 Yes Côte d’Ivoire No 2 (continued) Country 2013 2014 2015 2016 2017 2018 2019 2020 2021 Egypt, Arab Rep. of No 2 Gambia, The No 1 Yes Guinea Yes Guyana No 2 Haiti No 2 Jamaica No 1 Jordan No 2 Kazakhstan Yes Kenya No 2 Kosovo Yes Lesotho Yes Madagascar Yes Marshall Islands, Yes The Mauritius Yes Micronesia, Fed. Yes Sts. Montenegro Yes Myanmar Yes Nicaragua No 1 Yes Nigeria No 1 Yes Results and Performance of the World Bank Group 2022  Appendix F OECS countries No 2 Pakistan No 2 Panama No 2 Peru Yes Philippines No 2 Yes Poland No 1 Yes Romania No 2 Yes Rwanda No 2 Yes Samoa Yes São Tomé and No 2 Príncipe Senegal No 1 Yes Serbia No 2 South Africa No 1 Yes Sri Lanka Yes Tajikistan No 2 Yes (continued) 164 Country 2013 2014 2015 2016 2017 2018 2019 2020 2021 Uganda Yes Ukraine Yes Number of CLR 5 10 6 4 7 4 Reviews included Source: Independent Evaluation Group. Note: “No 1” means objectives not rated, CLR Review excluded; “No 2” means World Bank Group proj- ects not identified, CLR Review excluded; Yes means objectives rated and Bank Group projects iden- tified, CLR Review included. CLR = Completion and Learning Review; OECS = Organisation of Eastern Caribbean States. Table F.2. Other Recent Completion and Learning Review (CLR) Reviews Country 2019 2020 2021 2022 Cabo Verde Yes Croatia Yes Djibouti Yes Ghana Yes India Yes Indonesia Yes Kyrgyz Republic Yes Liberia Yes Malawi Yes Mexico Yes Mongolia Yes Independent Evaluation Group World Bank Group    165 Morocco Yes Nepal Yes North Macedonia Yes Papua New Guinea Yes Paraguay Yes Solomon Islands, The Yes Timor‐Leste Yes Zambia Yes Number of CLR Reviews included 11 3 4 1 Source: Independent Evaluation Group. Note: “Yes” means objectives rated and World Bank Group projects identified, CLR Review included. CLR = Completion and Learning Review  ountry Partnership Framework Objectives by Timing of Bank Table F.3. C Group Operations Timing Number Share (%) All operations approved before CPF period 79 14 More than 50% of operations approved before CPF period 66 11 More than 50% of operations approved during CPF period 241 41 All operations approved during CPF period 195 34 Total 581 100 Source: Independent Evaluation Group based on World Bank data. Note: CPF = Country Partnership Framework.  ountry Partnership Framework Objectives by World Bank Table F.4. C Group Institutions Institutions Number Share (%) World Bank and IFC operations 106 18 World Bank operations only 464 80 IFC operations only 11 2 Results and Performance of the World Bank Group 2022  Appendix F Total 581 100 Source: Independent Evaluation Group based on World Bank data. Note: IFC = International Finance Corporation.  ountry Partnership Framework Objectives by Type of World Table F.5. C Bank Group Operations Operations Number Share (%) Financing and Knowledge operations 290 50 Financing operations only 240 41 Knowledge operations only 51 9 Total 581 100 Source: Independent Evaluation Group. 166  chievement of Country Partnership Framework Objective Figure F.1. A and Timing of World Bank Group Operations Source: Independent Evaluation Group. Note: “All pre-CPF” means all World Bank Group operations started before the CPF period; “> 50 percent Pre-CPF” means the majority of Bank Group operations started before the CPF period; “> 50 percent during CPF” means the majority of Bank Group operations approved during the CPF period; “all during CPF: means all Bank Group operations approved during the CPF period. A = ACHIEVED; CPF = Country Partnership Framework; MA = mostly achieved; NA = not achieved; NRV = not rated or not verified; PA = partially achieved.  chievement of Country Partnership Framework Objectives Figure F.2. A and World Bank Group Institutions Independent Evaluation Group World Bank Group    167 Source: Independent Evaluation Group. Note: “World Bank + IFC operations” means CPF objectives supported by both World Bank and IFC op- erations; “World Bank operations only” means CPF objectives supported only by World Bank operations; “IFC operations only” means CPF objectives supported only by IFC operations. A = achieved; IFC = Inter- national Finance Corporation; MA = mostly achieved; NA = not achieved; NRV = not rated or not verified; PA = partially achieved.  chievement of Country Partnership Framework Objectives Figure F.3. A and International Finance Corporation Engagement Areas Source: Independent Evaluation Group. Note: “Key IFC strength areas” means business environment, private and financial sector development, energy, agrifood, and health. A = achieved; CPF = Country Partnership Framework; IFC = International Finance Corporation; MA = mostly achieved; NA = not achieved; NRV = not rated or not verified; PA = partially achieved.  chievement of Country Partnership Framework Objectives Figure F.4. A by Type of World Bank Group Operations Results and Performance of the World Bank Group 2022  Appendix F Source: Independent Evaluation Group. Note: A = achieved; MA = mostly achieved; NA = not achieved; NRV = not rated or not verified; PA = par- tially achieved. 168  chievement of Country Partnership Framework Objectives Figure F.5. A and Successful World Bank Operations Source: Independent Evaluation Group data. Note: The horizontal axis refers to the share of World Bank projects with moderately satisfactory or high- er Outcome ratings in the total number of World Bank lending operations supporting Country Partner- ship Framework objectives (for example, “< 25 percent MS+” means less than 25 percent of the projects supporting a Country Partnership Framework objective is rated moderately satisfactory or higher). A = achieved; MA = mostly achieved; MS+ = moderately satisfactory or higher; NA = not achieved; NRV = not rated or not verified; PA = partially achieved.  evelopment Policy Financing, Project Performance, and Figure F.6. D Country Partnership Framework Objective Achievement Independent Evaluation Group World Bank Group    169 Source: Independent Evaluation Group data. Note: “All DPF” means objective supported entirely by DPF; “>/< 50 percent DPF” means DPFs made up more or less than 50 percent of the projects supporting an objective; “no DPF” means no DPF in support program. A = achieved; CPF = Country Partnership Framework; DPF = development policy financing; MA = mostly achieved; MS+ = moderately satisfactory or higher; NA = not achieved; NRV = not rated or not verified; PA = partially achieved. Appendix G. Identifying Country Partnership Framework Priorities The Independent Evaluation Group’s evaluation of Systematic Country Diagnostics (SCDs) and Country Partnership Frameworks (CPFs) pointed out that the World Bank Group guidance identifies three criteria for the selection of program priorities: (i) alignment with SCD priorities; (ii) align- ment with the government’s own priorities; and (iii) areas where the Bank Group has a comparative advantage (World Bank 2017). World Bank (2017) found that most of the CPFs reviewed referred to these criteria, and some discussed each of them. But in practice, country teams also used priorities based on the Bank Group’s ongoing operational program, whether or not these were consistent with the three criteria. Although most CPFs have a table or a paragraph comparing the SCD and CPF priorities, they usually did not clearly explain the reasons for the variances among them. The evaluation pointed out that SCD priorities may not always match the Results and Performance of the World Bank Group 2022  Appendix G ongoing lending program, especially where the SCDs identified new priori- ties. Some country teams responded by defining a long list of priorities that would be consistent with almost any set of Bank Group-supported programs, while others stated CPF objectives in general terms that would accommodate both the ongoing program and the SCD priorities (World Bank 2017). World Bank (2017) identified as good practice the CPFs that clearly separated the ongoing program from what was needed to support the new program, whether through modifications of the existing program, new operations in the outer years, or through a redesigned program of analytic work and technical support. Reference World Bank. 2017a. World Bank Group Country Engagement: An Early-Stage Assess- ment of the Systematic Country Diagnostic and Country Partnership Framework Process and Implementation. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/ 170 files/scd-cpf.pdf. Appendix H. World Bank Group Corporate Approaches One Bank Group Approach “Operating as One Group, the Bank Group intends to work with the public and private sectors in partnership to end extreme poverty and boost shared prosperity sustainably” (World Bank Group 2014, 2). “Working as One WBG [World Bank Group] is central to the WBG Strategy. Collaboration across the WBG will be increased systematically, and planning and budgeting processes will be better coordinated at the corporate level… The new CPF [Country Partnership Framework] will mainstream joint busi- ness planning as the backbone for strengthening operational collaboration” (World Bank Group 2014, 4). Mobilizing Finance for Development Agenda and Cascade Approach “The WBG is intensifying and systemizing its commitment to Maximizing [now Mobilizing] Finance for Development (MFD). The March 2017 Forward Look: A Vision for the World Bank Group in 2030—Progress and Challenges in- Independent Evaluation Group World Bank Group    171 troduced the ‘Cascade Approach’ as a concept to guide the Bank Group’s ef- forts to leverage the private sector for growth and sustainable development. This approach asks the Bank Group to help countries maximize their devel- opment resources by drawing on private financing and sustainable private sector solutions to provide value for money and meet the highest environ- mental, social, and fiscal responsibility standards, and reserve scarce public financing for those areas where private sector engagement is not optimal or available” (World Bank Group 2017, 1). References World Bank Group. 2014. “World Bank Group Strategy: October 2013.” World Bank Group, Washington, DC. https://openknowledge.worldbank.org/bitstream/han- dle/10986/16095/32824_ebook.pdf?sequence=5&isAllowed=y. World Bank Group. 2017. “Maximizing Finance for Development: Leveraging the Private Sector for Growth and Sustainable Development.” World Bank Group, Washington, DC. https://www.devcommittee.org/sites/dc/files/download/Docu- mentation/DC2017-0009_Maximizing_8-19.pdf. Results and Performance of the World Bank Group 2022  Appendix H 172 Appendix I. Building on the New Country Engagement Guidance Timely adjustment of the Country Partnership Framework (CPF) and its results framework. The 2021 country engagement guidance underscores that “during the CPF period, the Bank Group carries out a continuous process of monitoring, learning from implementation, and adapting” (World Bank Group 2021), and it confirms the Performance and Learning Review (PLR) as the tool for this adaptation and adjustment. This Results and Performance of the World Bank Group found that PLRs (i) were not always done on a timely basis, (ii) did not necessarily adjust CPF objectives or results frameworks sufficiently to reflect actual changes in the Bank Group country program, or (iii) both. Despite the new guidance, more is needed to change the incentives for teams to produce timely and appropriately adapted PLRs when needed. Failure to reflect changes in the CPF objectives or results frameworks leads to a mismatch between what was intended, what was measured, and actual activities in the field, thereby limiting the Bank Group’s contribution. Incorporating the contribution of advisory services and analytics into the CPF. The latest guidance underscores the importance of using in the CPF both quantitative and qualitative indicators and “considering both direct Independent Evaluation Group World Bank Group    173 pathways (investments in infrastructure and services) and indirect pathways (strengthening institutions, generating and applying knowledge, and creat- ing markets and stimulating foreign investment)” (World Bank Group 2021). Although this encourages a more robust description of advisory services and analytics (ASA) in the CPF and incorporation of ASA in the results frame- work, it does not address two important issues highlighted in this RAP: (i) lack of a self or independent evaluation system for ASA; and (ii) lack of coverage of ASA in CLR Reviews reflecting a similar lack in CLRs; analysis of the latest CLR Reviews for a random sample of 50 countries indicated that these CLR Reviews reported on average on only about one-third of the ASA program in terms of use or influence. The World Bank may wish to introduce self-evaluation (and potentially independent evaluation) for ASA. Reflecting the contribution of the International Finance Corporation (IFC) and the Multilateral Investment Guarantee Agency (MIGA) to country outcomes. The new guidance acknowledges that the private sector mod- el for IFC and MIGA does not lend itself as readily to ex ante definition of activities, but it reconfirms the One Bank Group approach to preparation of a joint CPF, with full integration of IFC and MIGA objectives and activities into CPF objectives and programming and the accompanying results framework. Review of past CPFs and CLR Reviews indicates that inclusion of IFC and MIGA in the CPF description of the country program has been variable, with adequate inclusion in the results framework quite rare, even in PLRs that substantially revise the results framework during implementation. Although acknowledging the problem, the guidance could call for IFC and MIGA indi- cators at inception whenever their contribution to a CPF objective is antici- pated to be significant. Planning for materialization of risks that cannot be mitigated. The new guidance underscores that many risks are residual and cannot be fully mitigated through Bank Group actions, especially in countries affected by fragility, conflict, and violence and other high-risk environments. The guid- ance assumes that teams do not develop programming scenarios but rather Results and Performance of the World Bank Group 2022  Appendix I describe possible adjustments if risks materialize. Our analysis found little evidence that the Bank Group discusses ex ante adjustments to the program when identified risks materialize. Rather, the Bank Group’s responses to risks that cannot be mitigated—often political risks—tended to be ad hoc and were not adequately informed by how the Bank Group had responded across countries facing similar circumstances. Management may want to consid- er including scenario planning to minimize these ad hoc and inconsistent responses to known risks. Reference World Bank Group. 2021. “IBRD/IFC/MIGA/IDA Guidance: Country Engage- ment.” World Bank Group, Washington, DC. https://ppfdocuments.azureedge. net/1d0c9176-2efc-4b9e-8148-f6149b5f8c75.pdf. 174 Appendix J. Risk Mitigation: A Continuing Problem An analysis of Completion and Learning Review (CLR) Reviews for this Results and Performance of the World Bank Group found that several of the issues flagged by the 2017 Independent Evaluation Group evaluation of Systematic Country Diagnostics and Country Partnership Frameworks are still present, such as inadequate risk mitigation. That evaluation noted that “all CPFs [Country Partnership Framework] used the Systematic Operations Risk-Rating Tool (SORT), which was introduced in 2014 to provide a more structured method of rating risks. This tool was originally designed for use in projects, and some elements have little relevance for country-level risk analysis. The country teams have been conservative in their risk assessments. However, the descriptions of risks were often generic, and the risk mitigation or risk adaptation strategy was often vague, possibly because of the broad way in which the risks themselves had been framed. The presentation and discussion of risks also needs to draw a clearer distinction between World Bank and International Finance Corporation programs, since these have very different risk profiles. Overall, the Systematic Operations Risk-Rating Tool may need to be adjusted to better support the CPF through more focus on country-level risks and how Independent Evaluation Group World Bank Group    175 the World Bank Group can address systemic operational risks” (World Bank 2017a). Although the guidance for the tool has been updated to provide more guidance on residual risk, fuller ex ante risk assessments and identification of proposed corrective actions are still needed at the country level. The new country engagement guidance acknowledges that teams working in countries affected by fragility, conflict, and violence face high risks and en- courages the use of analytic work such as Risk and Resilience Assessments to better identify and manage such risks. “Teams avail themselves of diagnos- tics such as Bank Group Risk and Resilience Assessments or other analytical work on fragility and resilience, which inform the SCD [Systematic Country Diagnostic]” (World Bank Group 2021a). The new guidance also underscores that many risks are residual and cannot be fully mitigated through Bank Group actions. However, it provides insufficient guidance on how to prepare for and mitigate residual risks when they become a reality. Our review found that without ex ante planning, Bank Group responses to risks that cannot be mitigated—often political risks—tended to be ad hoc and were not adequate- ly informed by how the Bank Group had responded across countries facing similar circumstances. References World Bank. 2017a. World Bank Group Country Engagement: An Early-Stage Assess- ment of the Systematic Country Diagnostic and Country Partnership Framework Process and Implementation. Independent Evaluation Group. Washing- ton, DC: World Bank. https://openknowledge.worldbank.org/bitstream/ handle/10986/32123/World-Bank-Group-country-engagement-an-early- stage-assessment-of-the-systematic-country-diagnostic-and-country-partner- ship-framework-process-and-implementation.pdf?sequence=5&isAllowed=y. World Bank Group. 2021a. “IBRD/IFC/MIGA/IDA Guidance: Country Engage- ment.” World Bank Group, Washington, DC. https://ppfdocuments.azureedge. net/1d0c9176-2efc-4b9e-8148-f6149b5f8c75.pdf. Results and Performance of the World Bank Group 2022  Appendix J 176 Appendix K. Integrating the International Finance Corporation and the Multilateral Investment Guarantee Agency in Country Programs The Independent Evaluation Group’s outcome orientation evaluation noted that CPFs do not serve the strategic needs of the International Finance Corporation (IFC) well. The evaluation pointed out that this mo- tivated IFC to launch its own internal country strategies and other country products. It found that early experience with these IFC country strategies showed that they offer potential for increasing IFC’s focus on country out- comes, but close alignment with World Bank and World Bank Group country engagement is a condition of success and a condition of avoiding the risks of duplicated efforts (World Bank 2020, xii). The new country engagement guidance acknowledges that the private sector business model for IFC and the MIGA does not lend itself as readily to ex ante definition of activities, but it nonetheless reconfirms the One Bank Independent Evaluation Group World Bank Group    177 Group approach to preparation of a joint CPF, with full integration of IFC and MIGA objectives and activities into CPF objectives and programming and the accompanying results framework. The new guidance acknowledges preparation of a separate IFC country strategy as an internal document that may be used to integrate IFC objectives into the broader CPF (World Bank Group 2021a). However, the Bank Group will need to ensure that a separate internal IFC country strategy document does not dilute the One Bank Group approach. Review of past CPFs and CLRs indicates that inclusion of IFC and MIGA in the CPF description of the country program has been variable, with adequate inclusion in the results framework quite rare, even in Performance and Learning Reviews, which substantially revise the results framework during implementation. Although acknowledging the problem, the guidance falls short on requiring IFC and MIGA indicators at inception whenever their contribution to a CPF objective is anticipated to be significant. References World Bank. 2020. The World Bank Group Outcome Orientation at the Country Level. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.world- bankgroup.org/sites/default/files/Data/Evaluation/files/WBG_OutcomeOrienta- tion.pdf. World Bank Group. 2021a. “IBRD/IFC/MIGA/IDA Guidance: Country Engage- ment.” World Bank Group, Washington, DC. https://ppfdocuments.azureedge. net/1d0c9176-2efc-4b9e-8148-f6149b5f8c75.pdf. Results and Performance of the World Bank Group 2022  Appendix K 178 Appendix L. Trend Analysis of Advisory Services and Analytics The 2022 Results and Performance of the World Bank Group report com- piled a data set of advisory services and analytics (ASA) activities com- pleted during fiscal years (FYs)13–22 in the 50 sampled countries to analyze ASA’s role. (Data selection is not affected by the CLR Review period of each country). The total number of ASA activities increased from FY13 to FY16, reaching a peak of 544 in FY16 and declining afterward (figure L.1). As for volume, both the World Bank budget and trust fund costs increased during FY13–16, before falling and then rebounding in FY16–19 (figure L.2). Since FY19, World Bank budget costs have fallen. Trust fund costs rose continuously, peaking at $136 million in FY20 but dropping thereafter. Given that the data were captured in April 2022, the actual number and volume of ASA activities in 2022 would be somewhat larger, suggesting a less-steep downward slope from FY21 to FY22.  umber and Volume of ASA Activities in 50 Sampled Coun- Figure L.1. N tries, FY13–22 Independent Evaluation Group World Bank Group    179 Source: World Bank Enterprise Data Catalog. Note: ASA = advisory services and analytics; FY = fiscal year.  orld Bank Budget and Trust Fund Volume of ASA Activities Figure L.2. W in 50 Sampled Countries, FY13–22 Source: World Bank Enterprise Data Catalog. Note: ASA = advisory services and analytics; BB = World Bank budget; FY = fiscal year; TF = trust fund. Among ASA activities, the 2022 Results and Performance of the World Bank Results and Performance of the World Bank Group 2022  Appendix L Group team also reviewed the trends among activities focusing on the appli- cation of knowledge, including activities from the expired nonlending tech- nical assistance and external training product lines and the advisory task under the current advisory services and analytics product line.1, 2, 3 The AA product line was launched in FY17 to replace old ASA product lines, includ- ing both technical assistance and external training.4 The pattern in technical assistance, external training, and AA advisory activities resembles the pattern at the overall level. The number of activi- ties rose from 160 in FY13 to 374 in FY16 but declined continuously after- ward (figure L.3). World Bank budget and trust fund costs showed similar trends during FY13–17, increasing from FY13 to FY16 and dropping from FY16 to FY17 (figure L.4). Afterward, the World Bank budget cost fell from $31 million in FY17 to $16 million in FY22. At the same time, the trust fund cost experienced more shifts, with an increase from $38 million in FY17 to $89 million in FY20, followed by a sharp drop to $13 million in FY22. 180  umber and Volume of Technical Assistance, External Train- Figure L.3. N ing, and Advisory Services and Analytics Activities in 50 Sam- pled Countries, FY13–22 Independent Evaluation Group World Bank Group    181 Source: World Bank Enterprise Data Catalog. Note: AA = advisory services and analytics; FY = fiscal year; TA = technical assistance; TE = external train- ing.  orld Bank Budget and Trust Fund Volume of Technical Figure L.4. W Assistance, External Training, and AA Advisory Activities in 50 Sampled Countries, FY13–22 Results and Performance of the World Bank Group 2022  Appendix L Source: World Bank Enterprise Data Catalog. Note: AA = “product line” of advisory services and analytics; BB = World Bank budget; FY = fiscal year; TE = external training; TF = trust fund. A further review of the composition of ASA cost found a decline in the World Bank budget’s share and rise in trust fund share. There is a convergence be- tween the shares of World Bank budget and trust fund during FY13–19, with the difference of 33 percentage points decreasing to six percentage points or below (figure L.5). Since FY20, the trust fund ratio continued to rise, leading 182 to a reversed gap between World Bank budget and trust fund, with trust fund surpassing World Bank budget by 40 percentage points in FY20 and 34 in FY21. The gap between World Bank budget and trust fund appeared to be re- duced again in FY22, with World Bank budget counting 49 percent and trust fund 51 percent.  hare of World Bank Budget and Trust Fund in the Volume of Figure L.5. S ASA Activities in 50 Sampled Countries, FY13–22 Source: World Bank Enterprise Data Catalog. Note: ASA = advisory services and analytics; BB = World Bank budget; FY = fiscal year; TF = trust fund. Independent Evaluation Group World Bank Group    183 1  According to Operations Policy and Country Services, technical assistance (nonlending) are services to assist clients to implement a policy or program, develop or strengthen institutions, or to facilitate knowledge exchange. To qualify as technical assistance, an activity must (i) have the primary intent of enabling an external client to implement reform (that is, how to), strengthen institutions, or both; and (ii) be linked to a World Bank unit with clear account- ability for the service provided. 2  External training activities are defined as those that must (i) involve the design and delivery of events that focus on awareness raising or skills enhancement, particularly on exchang- ing good practices and experiences or on developing relevant skill sets for practitioners; (ii) include curriculum and pedagogical design, testing, and fine-tuning of learning materials; and (iii) target an audience that is external to the World Bank (although World Bank staff may participate); and (iv) be linked to a Bank unit, with clear accountability. 3  As defined by Operations Policy and Country Services, analytical and advisory are two types of advisory services. An advisory task has a development objective focusing on the application of preexisting knowledge, with varying degrees of customization. Examples of advisory deliv- erables include direct support for institutional capacity building or knowledge exchange and convening services concerned with the application of knowledge. Results and Performance of the World Bank Group 2022  Appendix L 4  According to an Operations Policy and Country Services announcement on July 11, 2016, the advisory services and analytics (AA) product line replaced five advisory services and analytics product lines: economic and sector work, technical assistance, impact evaluation, external training, and programmatic approach. 184 Appendix M. Unplanned Advisory Services and Analytics Table M.1 summarizes the reasons for unplanned advisory services and analytics (ASA) and their outcome as described in CLR Reviews. Of the 14 countries in which unplanned ASA was undertaken, in most of them (9 countries), it was in response to a World Bank decision and not to govern- ment demand. Furthermore, in most cases (8 countries), this unplanned ASA resulted in greater relevance of the ASA program to the Country Partnership Framework or government program. However, there were 3 countries where the unplanned ASA resulted in fragmentation or lack of correspondence with the Country Partnership Framework objectives, and there were 3 where the outcome was unknown. Independent Evaluation Group World Bank Group    185 186 Results and Performance of the World Bank Group 2022  Appendix M  easons for Unplanned Advisory Services and Analytics and Their Outcome Table M.1. R Reason for Unplanned ASA Outcome of Unplanned ASA Response To Became More Relevant Became Fragmented Or Not Noted In CLR Government World Bank To CPF Or Government Did Not Correspond With Review Or Don’t Country Demand Decision Program CPF Objectives Know Argentina ✓ ✓ Bulgaria ✓ ✓ Burundi ✓ ✓ Comoros ✓ ✓ Gambia, The ✓ ✓ Kazakhstan ✓ ✓ Lesotho ✓ ✓ Nicaragua ✓ ✓ Panama ✓ ✓ Pakistan ✓ ✓ Romania ✓ ✓ Senegal ✓ ✓ Sri Lanka ✓ ✓ South Africa ✓ ✓ Source: Based on an Independent Evaluation Group’s desk review of CLR Reviews. Note: ASA = Advisory Services and Analytics; CLR = Completion and Learning Review; CPF = Country Partnership Framework. The World Bank 1818 H Street NW Washington, DC 20433