Approach Paper Early-Stage Evaluation of the Multiphase Programmatic Approach June 10, 2024 1. Background and Evaluation Rationale 1.1 The multiphase programmatic approach (MPA) is a way of structuring a long, large, or complex engagement—typically over 8–10 years—either as a set of smaller linked operations or phases using investment project financing (IPF) or Program-for- Results under one program or as a multicountry program, sometimes designed to deliver a global or regional public good. Since its introduction in fiscal year 2018, there has been a steady increase in its use (figure 1.1). A total of US$18 billion was approved under the World Bank’s COVID-19 response, of which US$10.8 billion has been committed and US$8.5 billion disbursed. A further US$28.8 billion has been approved under other MPA programs, of which US$12.6 billion has been committed. The MPA is also expected to support the World Bank’s response to global challenges. The lending pipeline for the next 18 months includes 46 MPA engagements, of which 23 are expected to support global challenge objectives. Most of these are in Energy and Extractives, Transport, and Water. 1 Figure 1.1. Multiphase Programmatic Approach Approvals and Commitments a. Multiphase programmatic approach engagements by fiscal b. COVID-19 multiphase programmatic approach phase year commitments by fiscal year 15 2019 2020 2021 2022 2023 12 20 80 Projects (no.) 71 Commitment (US$, 10 7 7 15 60 Projects (no.) 4 4 4 5 billions) 5 2 4 1 2 10 40 3 1 1 0 8.2 5 20 15 2017 2018 2019 2020 2021 2022 2023 2024 6 0.7 0 1.9 0 Year 2020 2021 2022 Horizontal Vertical Year Commitment Projects c. Multiphase programmatic approach approvals by fiscal year 8 Approval (US$, billions) 6 4.0 4 3.1 2.5 1.9 0.9 2.7 2 0.2 0.2 0.6 1.3 1.4 0.4 2.3 0.1 0.1 0.6 1.3 1.2 0 0.1 0.1 0.2 2018 2019 2020 2021 2022 2023 2024 Fiscal year Horizontal Vertical Source: World Bank 2024. Note: Data cover the International Bank for Reconstruction and Development and the International Development Association only and exclude canceled multiphase programmatic approach programs. Fiscal year 2024 is reported only up to the end of the second quarter. Data snapshot is as of December 31, 2023. Panels a and c: Global COVID-19 data are presented separately; 2021 saw a significant decline as the pipeline stalled because of COVID-19 as the priority. Panel b: Data represent the number and volumes of COVID-19 projects under the global multiphase programmatic approach. 1.2 The motivation behind the MPA was to provide continuity of engagement, allow more flexibility in responding to changed circumstances, encourage adaptive learning, and support stepwise progress toward a long-run development objective; it was also intended to reduce processing costs and allow the World Bank to better manage its capital. 1 By signaling a willingness to engage in pursuit of a long-term development objective, it was envisaged that the MPA would strengthen the coherence of World Bank–financed interventions and contribute to building consensus around them on the 1Committing financing in phases rather than up front would enable the World Bank to reduce undisbursed balances and allow borrowers to save on commitment fees; it would also reduce the processing costs of follow-up phases relative to stand-alone operations, especially for guarantees. 2 client side, while also diminishing the likelihood of interruptions in support between phases. Compared with a single large operation, it would also make adaptation to circumstances easier by incorporating multiple opportunities for reflection and course correction, including the possible cancellation of follow-up phases in the event of a change in country circumstances or priorities. In addition, the MPA would encourage self-reflection by requiring teams to articulate a forward-looking learning agenda. Finally, by sending a signal of long-term support to the government, it was hoped that the approach would mobilize private sector investment, particularly for the energy transition. 1.3 The MPA was originally conceived in two forms: (i) a vertical or single-country form in which a series of sequential or partially overlapping phases allows the World Bank and a client to address a sectoral problem that requires more than one project cycle to achieve results (typically over 8–10 years), the value of the MPA lying in its combination of continuity and flexibility, and (ii) a horizontal or regional form in which each phase supports a country (or a group of countries) with a common theory of change, results framework, and activities with a short- to medium-term focus (4–6 years), the value of the approach consisting in the speed of preparation and implementation enabled by the standardization of operations (figure 1.2). The World Bank used the horizontal MPA in its response to the COVID-19 and locust emergencies. There have also been 11 regional MPAs in East Africa, Europe, South Asia, and West Africa, each involving anything from 4 to 10 countries. 2 2The regional multiphase programmatic approach differs from a standard regional operation in that (i) not all phases require the Board of Executive Directors’ approval (nor does the addition of a preidentified borrower) and (ii) phases can use Program-for-Results and investment project financing. 3 Figure 1.2. Types of Multiphase Programmatic Approaches Source: World Bank 2024. Note: PrDO = program development objective. 4 1.4 An important characteristic of the MPA is delegated authority. Whereas the Board of Executive Directors approves the program development objective, overall financing envelope, and details of the first phase, the commitment of financing to subsequent phases is, with some policy exceptions, delegated to management. 3 In return for this delegation, Operations Policy and Country Services (OPCS) established certain expectations for how the MPA would be used. These were (i) the existence of an outcome-focused program development objective with stated impact on final beneficiaries (not system strengthening or capacity building); (ii) a robust forward- looking learning agenda, which specified what would be tested during each phase and how; and (iii) an expectation that the first phase will achieve meaningful results and be viable even as a stand-alone operation because there could be no guarantee of continuation to the next. 1.5 The 2017 MPA Board paper also acknowledged that the approach was not without risk and might not suit all borrowers (World Bank 2017). Even if it simplified the World Bank approval process for follow-up phases, the same was not necessarily true on the client side, where legislative consent might be required for all phases. Previous experience with adaptable program loans—a forerunner to the MPA— indicated that client support for long-term programs often turned out to be more subject to electoral cycles or other shifts in priorities than initially presumed. It was also envisaged that the availability and terms of World Bank financing might change during implementation (a country eligible for the International Development Association, for example, might graduate or become a blend country). This could lead a borrower to prefer to lock in current terms and conditions with a single, long-term stand-alone project. 1.6 OPCS oversaw decisions on the choice of the approach through the issuance of formal written guidance, support to task teams, and comments at review meetings. Out of the 153 proposals it received between 2017 and 2022, only 24 were eventually approved as MPAs. 4 The novelty of the MPA, particularly when used in support of the World Bank’s response to the COVID-19 emergency, demanded an unusual degree of management attention. The COVID-19 evaluation by the Independent Evaluation Group emphasizes the effort it took to establish a program blueprint, provide guidance to teams, streamline approval processes, and coordinate internally (World Bank 2022). The Independent Evaluation Group has not evaluated the MPA, although it has evaluated 3These exceptions are for investment project financing phases that involve high or substantial environmental and social risks, Program-for-Results phases that introduce new disbursement-linked indicators, changes to the program development objective or the original financing envelope, or the addition of a new borrower under a horizontal program. 4Most of those rejected were approved instead as stand-alone investment project financing or as series of projects. 5 the World Bank’s COVID-19 response and the role of MPAs in supporting learning in lending. 1.7 There has also been an informal OPCS briefing to the Executive Directors. This showed that the COVID-19 Strategic Preparedness and Response Program MPA programs took less time to prepare and disburse than emergency non-MPA IPFs, that preparation times for the first phases of MPAs were similar to those for regular IPFs, and that follow-up phases for MPAs were quicker to prepare. It also claimed that MPAs were achieving higher-level results earlier than comparable non-MPA engagements. This evaluation will validate these claims, as outlined in chapter 4. Consequently, OPCS proposed that MPA policy be changed to (i) allow the inclusion of development policy financing and International Finance Corporation and Multilateral Investment Guarantee Agency instruments and (ii) delegate commitment authority to management for phases involving high- and substantial-risk IPFs and Programs-for-Results that introduced new disbursement-linked indicators. 2. Purpose and Audience 2.1 The objective of this evaluation is to understand whether the use and effectiveness of the MPA have met the expectations of the 2017 Board paper. This is particularly timely given the potential use of MPAs to support the global challenge programs. Its audience is the World Bank’s Board of Executive Directors, its Committee on Development Effectiveness, and World Bank Group management and staff working on MPAs. The evaluation has been requested by the Committee on Development Effectiveness (for submission in the first quarter of fiscal year 2025) to inform the Board’s ongoing discussions with management on the MPA. 3. Evaluation Scope and Questions 3.1 The evaluation scope is determined both by the short turnaround time for this exercise and by the fact that the MPA portfolio is young. This exercise, therefore, will not seek to address whether the MPA is more likely to achieve long-term outcomes, will not assess the uptake of MPAs or whether there have been “missed opportunities” to apply it, and will not evaluate the suitability of the instrument where it has not been applied. The evaluation will also not address whether expectations of improved coherence on the client side were met or whether to expand the policy scope of the MPA as suggested by OPCS. The pros and cons of delegation of authority in general are outside the scope of this evaluation. 3.2 The evaluation scope is limited to approved nonemergency MPAs as of December 31, 2023. For MPAs approved after that cutoff, rather than address all three proposed evaluation questions, the Independent Evaluation Group may assess design 6 elements from the perspective of compliance with Board expectations. Out of the 40 nonemergency MPAs, 11 are horizontal and 29 are vertical; 7 have progressed beyond the first phase (3 vertical and 4 horizontal), but none has closed. The exclusion of the emergency response MPAs is mainly because (i) the short time horizon of this evaluation will not permit an expansion of scope to emergency circumstances, which would call for a significantly different evaluation approach, and (ii) the Committee on Development Effectiveness has sufficient information on how the MPA supported crisis response. 3.3 The evaluation questions are proposed as follows: 1. To what extent has the design of MPAs followed Board expectations and management guidance? a. To what extent have the objectives of MPA operations been oriented toward high-level impacts? b. To what extent have MPAs been designed to support institutional development, adaptive management, and learning? c. To what extent do MPAs conform to either the horizontal or vertical models? 2. To what extent have the mechanisms embedded in the MPA worked as expected to achieve its objectives? (See table 3.1.) a. To what extent have they improved the coherence of interventions? b. To what extent have they supported programmatic continuity? c. To what extent have they facilitated and supported monitoring and learning within or across phases? d. To what extent have they supported adaptation to changing circumstances and priorities? 3. Under what circumstances or enabling conditions has the MPA worked as intended? a. To what extent have client-side conditions enabled or prevented the MPA from working as intended? b. To what extent have conditions within the Bank Group enabled or prevented the MPA from working as intended? 7 Table 3.1. Hypothesized Multiphase Programmatic Approach Mechanisms Associated with Development Effectiveness Observable Implications Objective Mechanism Vertical Horizontal Program coherence Agreement on long-run Articulation of long-term objectives that Identification of a global (World Bank 2017) objectives and constraints strengthens consensus around them public good whose across Global Practices within World Bank team and outcome provision is being orientation in CPFs and PLRs; greater supported that cross-sector collaboration on the World encourages cross- Bank and client side (for example, country learning and common monitoring arrangements across collaboration, including line agencies and division of phases cross-sector across World Bank Global Practices) collaboration as in the relative to a set of independent case of vertical MPAs operations or single large operation Continuity Greater likelihood of long- Greater likelihood that client will adopt term funding and invest in long-term objectives relative to a set of independent operations Learning Requirement of learning World Bank supervision more oriented More parallel learning plan in PAD backed by toward learning than compliance; more across World Bank teams monitoring and capture of self-evaluation by vertical MPA clients and clients than in a set lessons learned than in a single large operation of independent operations Adaptation Multiple points for Earlier cancellation or restructuring in reflection (Mid-Term response to changed circumstances and Review and the end of lessons learned than in a single large each phase) that enable operation restructuring or cancellation of activities Source: Independent Evaluation Group. Note: The mechanisms will be refined and expanded during the evaluation. CPF = Country Partnership Framework; MPA = multiphase programmatic approach; PAD = Project Appraisal Document; PLR = Performance and Learning Review. 4. Methods 4.1 The evaluation will use (i) (semi)structured interviews with key informants (World Bank task team leaders, practice managers, country directors, regional directors, directors of strategy and operations and vice presidents, and, time permitting, stakeholders on the client side); (ii) a portfolio review of the 40 approved MPAs and a set of comparator non-MPA operations; and (iii) a desk-based document review (table 4.1). 4.2 Two groups of comparator projects will be selected to minimize the influence of country- and project-specific confounders. They include (i) the most similar operations in the same country for vertical or region for horizontal MPAs and (ii) the most similar operations in a similar context, as measured by the public administration Country Policy and Institutional Assessment (fiscal years 2018–22). Project similarity is calculated as the 8 cosine distance of mean text embeddings from the project description section of Project Appraisal Documents from that of reference projects. Table 4.1. Basic Evaluation Matrix Evaluation Subquestion Source of Information Methods Limitations Design and compliance with expectations To what extent have the PDO indicators align with • Portfolio analysis: We will assess compliance with objectives of MPA or clearly build toward document review expectations set out in the operations been oriented PrDOs. • Use of methodology 2017 Board paper (World Bank toward high-level impacts? Theories of change laid out developed for 2017) at the design stage only; in the project document gauging outcome thus, it will limit some of the are conducive to pursuing orientation in RAP conclusions we can draw with long-term objectives. 2021 respect to elements of compliance, but they may be • Structured and relevant later in the MPA semistructured process. interviews to validate any unusual findings To what extent have MPAs PADs, ISRs, and data from • Document review been designed to support structured interviews will • IEG evaluation of institutional development, be used to interrogate learning from lending adaptive management, and alignment of learning? implementation, M&E, and institutional arrangements with MPA objectives. To what extent do MPAs PADs and interviews will be conform to either the used. horizontal or vertical models? Expected mechanisms associated with achieving objectives To what extent have these PADs, CPFs, and PLRs • Desk review of General: Most MPAs are at an mechanisms improved the should indicate a logic strategic documents early stage of implementation, coherence of interventions? framework that articulates and selected advisory with only seven having the link among country services and analytics graduated to the second or analytics, strategy, and third phase; thus, we cannot MPA. CPFs and PLRs should assess longer-term results or demonstrate stronger effectiveness, which will limit outcome orientation. some of the conclusions we To what extent have these PDO and PrDO indicators, • Desk review of can draw. mechanisms supported institutional arrangements project documents To mitigate this issue and be programmatic continuity? for M&E, and interview • Portfolio analysis able to draw meaningful data should indicate conclusions with respect to • Interviews consistent measurable results, we will focus on progress toward long-term observable implications within development objective(s) earlier phases of the MPAs. and long-term assurance of Interviews: The team will take support leading to greater care to mitigate potential continuity on the client side interview biases (for example, in terms of staffing and selection bias, social capacity building. desirability bias, and confirmation bias) via proper 9 Evaluation Subquestion Source of Information Methods Limitations To what extent have these • Learning plans • Review of learning selection of interviewees, mechanisms facilitated and articulated in PADs plans, project projects, and interview supported monitoring and • ISR reporting of documents, ISRs, and questions. learning within or across implementation of aide-mémoire With respect to project phases? learning plans • Structured and selection, there are many • Interview data semistructured variables along which MPA interviews projects differ, some of which • Information on IEG evaluation of are confounders for institutional • arrangements for learning from lending effectiveness outcomes. We would ideally like to control for generating, managing, these as much as possible but and communicating are limited by a small number knowledge of MPA projects. Given the To what extent have these • Restructuring papers • Document review small number, we cannot mechanisms supported • Interview data • Structured and stratify the data based on too adaptation to changing • Mid-Term Review aide- semistructured many variables. circumstances and mémoire for data on interviews To mitigate this, we will create priorities? progress toward long- • Document review a typology based on critical run development and coding of results dimensions of variation, then objective, ambition, indicators for rely on random selection relevance, and efficacy operations that have within each cell. of PDO indicators completed the first The short project timeline will phase also limit the overall number of interviews we can perform. We will rely on a points of saturation strategy to make sure we obtain all needed information with a smaller number of interviews. Enabling conditions To what extent have client- • Data on country and • Document review Data on sector and country side conditions enabled or sector context • Data and portfolio context, including capacity, are prevented the MPA from • Assessment of client analysis at the country level, whereas working as intended? capacity and demand in • Structured and our outcomes of interest are at PADs and other project semistructured the project level. Depending documents interviews on how the projects are • Implementation-related clustered geographically, there data in ISRs may be too little variation in the country context variables • Interview data to draw meaningful conclusions. We will rely on more granular information obtained during interviews to mitigate some of these concerns. 10 Evaluation Subquestion Source of Information Methods Limitations To what extent have • Interview data • Structured and Social desirability bias is a conditions within the World • Data on budget semistructured potential concern because we Bank Group enabled or allocations versus interviews with task are trying to obtain prevented the MPA from spending for horizontal team leaders and information on internal working as intended? MPAs practice managers conditions. • Data on rejected To mitigate this concern, we proposals, including will formulate interview whether these questions that highlight proposals were “process” as opposed to developed into stand- “opinions.” alone operations Source: Independent Evaluation Group. Note: The evaluation matrix will be reviewed from the perspective of tractability and feasibility given four months from launch to World Bank management submission. CPF = Country Partnership Framework; IEG = Independent Evaluation Group; ISR = Implementation Status and Results Report; M&E = monitoring and evaluation; MPA = multiphase programmatic approach; PAD = Project Appraisal Document; PDO = project development objective; PLR = Performance and Learning Review; PrDO = program development objective; RAP = Results and Performance of the World Bank Group. 4.3 The overall limitations of this evaluation are (i) that no MPA has closed (that is, we cannot evaluate its achievement of program development objectives) and (ii) that only seven MPAs have progressed from the first to the second or third phase (that is, we are constrained in evaluating the mechanisms on which their effectiveness depends). 5. Quality Assurance and Resources The evaluation will be led by Rashmi Shankar, under the supervision of Theo Thomas. Team members are Andrea Rojas Hosse, Sengphet Lattanavong, Diana Stanescu, Marwane Zouaidi, and Harsh Anuj. Rasmus Heltberg will be advising the team. The team will engage with Estelle Raimondo (the head of methods) on a regular basis, given the tight timeline of the evaluation. The completed report is scheduled to be submitted to the Committee on Development Effectiveness in the first quarter of fiscal year 2025. The estimated budget is US$250,000. 11 References World Bank. 2017. “Multiphase Programmatic Approach.” Board Report 117742, World Bank, Washington, DC. https://documents.worldbank.org/en/publication/documents- reports/documentdetail/203081501525641125/multiphase-programmatic-approach. World Bank. 2022. The World Bank’s Early Support to Addressing COVID-19: Health and Social Response—An Early-Stage Evaluation. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2024. “Multiphase Programmatic Approach.” Technical Briefing to the Board of Executive Directors, January 2024, World Bank, Washington, DC. 12 Appendix A. Interview Design and Selection Strategy We plan to leverage semistructured and structured interviews for all evaluation questions either to validate any unexpected results that come up from desk review or to dig deeper into mechanisms or uncover pathways of change where other sources of data are scarce. Interview data will be rigorously coded and presented. Given the short timeline, we will start with a low number of interviews and increase it if we do not reach saturation with respect to the types of answers we obtain. We plan to focus on the 40 nonemergency multiphase programmatic approach (MPA) projects, selecting respondents for all horizontal MPAs (4 out of 11 horizontal MPAs have progressed beyond the first phase) and from a sample of the vertical MPAs. We will use a purposive sampling strategy to select vertical MPAs for interviews as follows. First, we will create a typology of projects based on two dimensions of variation: (i) level of institutional maturity and (ii) nature of intervention. We will classify projects based on qualitative assessment because the scope and timeline of the evaluation (and limited number of MPA projects) render a more complex, machine learning–based classification exercise not optimal. Second, we will select two to three projects from each cell (institutional maturity and nature of intervention) to maximize our chances of gathering evidence around our evaluation questions. For example, we plan to include all projects that have advanced beyond the first phase. For the subset of questions where we intend to highlight points of divergence or convergence in task team leaders’, country directors’, and clients’ perspective, we want to obtain triangulated information on three to four questions, with the goal of obtaining quantifiable data by stakeholder group. In practice, these structured questions would approximate a mini-survey (as opposed to open questions). We would be asking questions such as, “If you were to tell me on a scale of 1–5 how well the MPA served its pursuit of long-term outcomes…?” Additional Considerations • In regard to stakeholders, we plan to start with task team leaders with experience on both the MPA and non-MPA side. We want to make sure to interview task team leaders, country directors, and stakeholders on the client side. • Group interviews may be useful, particularly on the operating environment. 13 Appendix B. Initial Multiphase Programmatic Approach Portfolio Breakdown This evaluation will assess 40 nonemergency multiphase programmatic approaches (MPAs), of which 29 are vertical and 11 are horizontal. In terms of regional distribution of the broader MPA portfolio, Africa leads the Regions with 23 MPAs, followed by Europe and Central Asia and South Asia with 5 each (figure B.1). The Energy and Extractives Global Practice leads the portfolio with 10 MPAs, followed by the Health, Nutrition, and Population; Transport; and Water Global Practices leading 6 MPAs each. Figure B.1. Overall Multiphase Programmatic Approach Portfolio, by Region and Global Practice EAE HNP WAT TT Global Practice AGR SPJ EDU IDD URL SSI FCI 0 1 2 3 4 5 6 7 8 9 10 11 MPA (no.) Africa EAP ECA LAC MENA SAR Source: Independent Evaluation Group staff analysis of World Bank data. Note: AGR = Agriculture and Food; EAE = Energy and Extractives; EAP = East Asia and Pacific; ECA = Europe and Central Asia; EDU = Education; FCI = Finance, Competitiveness, and Innovation; HNP = Health, Nutrition, and Population; IDD = Digital Development; LAC = Latin America and the Caribbean; MENA = Middle East and North Africa; MPA = multiphase programmatic approach; SAR = South Asia; SPJ = Social Protection and Jobs; SSI = Social Sustainability and Inclusion; TT = Transport; URL = Urban, Disaster Risk Management, Resilience, and Land; WAT = Water. In terms of program financing, as demonstrated in figure B.2, the Africa Region leads the MPA portfolio with 74 percent of total funding (US$20,854.1 million), followed by South Asia with 13 percent (US$3,731.45 million) and Latin America and the Caribbean with 7 percent (US$1,905 million). The Energy and Extractives Global Practice leads the Global Practices by funding, with 30 percent of total MPA funding, followed by the Health, Nutrition, and Population and the Agriculture and Food Global Practices. 14 Figure B.2. Multiphase Programmatic Approach Total Commitments at Board Approval, by Region and Global Practice a. Program financing by Global Practice (percent) b. Program financing by Region (percent) 2 1 1 2 1 EAE HNP 1 13 AFR 4 AGR 30 7 EAP TDD 14 WAT 3 ECA SPJ 2 LAC DD 14 EDU MENA 74 16 URL SA 15 SSI FCI Source: Independent Evaluation Group staff analysis of World Bank project data. Note: AFR = Africa; AGR = Agriculture and Food; DD = Digital Development; EAE = Energy and Extractives; EAP = East Asia and Pacific; ECA = Europe and Central Asia; EDU = Education; FCI = Finance, Competitiveness, and Innovation; HNP = Health, Nutrition, and Population; LAC = Latin America and the Caribbean; MENA = Middle East and North Africa; SA = South Asia; SPJ = Social Protection and Jobs; SSI = Social Sustainability and Inclusion; TDD = Transport and Digital Development; URL = Urban, Disaster Risk Management, Resilience, and Land; WAT = Water. Vertical A total of 25 out of 29 vertical MPAs are in countries eligible for the International Development Association. In addition, about half the portfolio of vertical MPAs are in lower-middle-income countries (14 out of 29), whereas 31 percent are in upper-middle- income countries (9 out of 29) and 21 percent in low-income countries (6 out of 29). Horizontal Africa leads all Regions (figure B.3) with 72 percent of the horizontal portfolio (8 out of 11), followed by Europe and Central Asia with 18 percent (2 out of 11) and South Asia with 10 percent (1 out of 11). 15 Figure B.3. Regional Distribution of Multiphase Programmatic Approaches, by Category 16 14 12 10 MPAs (no.) 8 6 4 2 0 Africa SAR ECA LAC EAP MENA Region Horizontal MPA Vertical MPA Source: Independent Evaluation Group staff analysis of World Bank project data. Note: EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MENA = Middle East and North Africa; MPA = multiphase programmatic approach; SAR = South Asia. 16