Approach Paper Country Program Evaluation World Bank Group Support to Somalia, Fiscal Years 2013–22 March 2, 2023 The Somalia Country Program Evaluation (CPE) will assess the evolution of the World Bank Group’s support over fiscal years (FY)13–22 and the extent to which the Bank Group adequately prepared for an eventual normalization of relations with Somalia, tailored its support to the conflict and fragility situation in Somalia and evolving circumstances and country priorities, and learned from experience. It will seek to inform the preparation of the next Somalia Country Partnership Framework (CPF) and may be relevant to broader Bank Group engagement in countries affected by fragility, conflict, and violence (FCV). 1. Country Context and Main Development Challenges 1.1 Having suffered from a decades-long civil war and recurrent droughts and floods, Somalia saw much of its human and physical capital destroyed and had long been considered the quintessential example of a failed state. At the start of the evaluation period in 2013, despite overall improvements in Somalia’s security environment since the 1990s, the country remained mired in conflict and internal displacement, adding to economic and environmental challenges. Much of the country was under Al-Shabaab control.1 1.2 Somalia is today among the poorest and most fragile countries in the world, facing myriad development challenges related to ongoing conflict, climate change, food insecurity, natural disasters, and displacement. Overlapping crises related to the COVID-19 pandemic, a prolonged drought, and macroeconomic shocks from rising food and fuel costs have worsened socioeconomic conditions (World Bank 2022). Seventy-one percent of Somalis lived in extreme poverty in 2021, compared with 28 percent for Sub- Saharan Africa (World Bank 2021). Average life expectancy was 57.4 years, and maternal mortality stood at 734 for every 100,000 births (World Bank 2018d). The country’s Sustainable Development Goal ranking was 160th out of 163. As a result of displacement, Somalia has experienced rapid urbanization; close to 75 percent of the 2.6 million internally displaced persons lived in urban centers in 2018. This unplanned urban growth has led to slum expansion and environmental degradation throughout Somali cities (World Bank 2020c). 1.3 Existing crises are compounded by the country’s high vulnerability to natural disasters. Many Somalis are nomadic or seminomadic herders, and livestock production 1 provides income to 60 percent of the population (Famine Early Warning Systems Network 2020). These traditional rural livelihoods are highly vulnerable to drought. Despite contributing just 0.08 percent of global greenhouse gas emissions, Somalia is ranked the second-most climate-vulnerable country in the world (ICRC 2021). Drought has displaced another 1 million people in Somalia since January 2021 (UNHCR and NRC 2022). The number of people in Somalia facing crisis hunger levels is expected to rise to more than 7 million (44 percent of the population) in the coming months. In 2021, almost half the population required humanitarian assistance (World Bank 2022). 1.4 With little ability to finance core state functions and services, Somalia is highly dependent on external assistance. In 2020, external grants accounted for 57 percent of federal government revenue (Federal Government of Somalia 2021). In the same year, official development assistance was 43 percent of the gross domestic product, and remittances accounted for 31 percent. Domestic revenue mobilization is extremely low, with a tax–to–gross domestic product ratio of 4 percent. Although Somalia reportedly has a vibrant private sector, reported foreign direct investment flows in 2020 are relatively small at 9 percent of gross domestic product (World Bank 2022). 1.5 Somalia’s security and political situation remains highly volatile and subject to setbacks. In 2021, a political crisis emerged as Somalia’s then-president delayed national elections, which led to demonstrations. Consequently, the World Bank triggered Operational Policy 7.30, which governs engagement with de facto governments.2 After repeated delays and mounting pressure, presidential elections were held in May 2022, allowing a new government to be formed by President Hassan Sheikh Mohamud. Even so, urban centers have suffered frequent violence perpetrated by Al-Shabaab, which continues to pose the main threat to Somalia’s stability (Faruk 2022). In addition, Somalia’s strategic location in the Horn of Africa and abutting the Red Sea and the Gulf of Aden also makes it the subject of competing interests by international actors. 1.6 The federal government struggles to perform state functions and provide services and competes for legitimacy. A key challenge to state building in Somalia is the fact that the federal government of Somalia armed nonstate actors (primarily Al- Shabaab, but also the Islamic State and local militias), and semi-independent subnational governments compete for legitimacy in the eyes of the Somali people. The ultimate success of state building in Somalia depends on the ability of the Somali government to present itself as the most credible provider of state functions, of which the most essential is security. Weak domestic revenue mobilization hinders the federal government’s ability to restore and supply basic services (such as education and policing), which have been fragmented and privatized by conflict. Although the federal government’s domestic revenue collection stood at US$221 million in 2020, it is estimated that Al- Shabaab’s annual revenue collection is about US$180 million (Hiraal Institute 2020). 2 1.7 State building is also challenged by the competition between the central government and the periphery. Under the 2012 constitution, Somalia is a federal republic, and the federal member states (FMS) enjoy a degree of autonomy over regional affairs. Although the federation is intended to bind together Somalia’s disparate ethnic and linguistic groups, there are significant tensions between the federal government and the FMS and among the FMS. Following the collapse of the central government in early 1991, Somaliland unilaterally declared independence, but it is not currently recognized by any United Nations member state. The relationships between the federal government and the FMS, especially with Somaliland, but also with the other FMS, remain challenging in the face of competition for authority and resources, including security sector management and the control of key economic assets and revenue-generating infrastructure and aid distribution. 1.8 Although Somalia faces substantial challenges, the country also has significant sources of resilience. The weakening or absence of a central state since 1991 increased reliance on personal networks as an important source of resilience. Strong intragroup support and family relations across the Horn of Africa have often provided the necessary safety net to stave off the most severe impacts of drought, conflict, and displacement (World Bank 2018b). Somalia’s private sector is another source of resilience, and it provides basic services. Already three-quarters of Somalis over the age of 16 years use mobile money (World Bank 2018c). Somalia’s strategic location along international shipping lanes and close to growing markets in the Red Sea and the Persian Gulf presents opportunities for trade and regional integration (World Bank 2018a). 2. World Bank Group Engagement in Somalia 2.1 A turning point for Somalia’s relations with the international community was the adoption of the New Deal Compact for Somalia in 2013 (figure 2.1). The compact was endorsed at a conference hosted by the federal government and the European Union, paving the way for limited engagement by the international community, including international financial institutions.3 The compact’s goals were (i) legitimate and inclusive politics, (ii) security, (iii) justice, (iv) economic foundations, and (v) revenue and services. Aided by the ousting of Al-Shabaab from Mogadishu in 2011–12, the federal government was inaugurated in 2012, based on the provisional constitution. This was the first permanent central government in the country since 1991, when the civil war started. 3 Figure 2.1. Timeline of Events in Somalia Source: Independent Evaluation Group. Note: CPF = Country Partnership Framework; HIPC = heavily indebted poor countries; IDA = International Development Association; OP = Operational Policy. 2.2 The Bank Group’s engagement evolved from small, trust-funded operations to a mostly normalized financing relationship after Somalia cleared its arrears to international financial institutions in 2020 (figure 2.2). Since the endorsement of the New Deal Compact, Bank Group engagement was guided by two country strategies—an Interim Strategy Note (FY14–16) and a CPF (FY19–22, subsequently extended through FY23; World Bank 2013, 2018d). These strategies reflect two distinct phases of Bank Group engagement. During the first phase, 2014–20, the World Bank was prohibited from using its resources to support Somalia because of large-scale arrears to itself, the International Monetary Fund, and other external creditors. Over this period, World Bank support to Somalia (largely in the form of analytical and diagnostic work) was financed by trust funds (the Somalia Multi-Partner Fund and the State and Peacebuilding Fund).4, 5 The second phase, the period after arrears clearance, represented the normalization of the relationship between international financial institutions and Somalia. It was during this period that Somalia gained access to regular funding from major development institutions, including the International Development Association (IDA). 4 Figure 2.2. Evolution of World Bank Engagement in Somalia 2013: New Deal Compact 2013: Multi-Partner Fund Roadmap for promoting state building and Building and testing 2019: Pre-arrears clearance financing peace building government systems and capacity to lead 2020: IDA development Increase support to Provides a strategic plan complete reforms Developing World Bank toward stability and necessary for HIPC, Develop national Group ability to engage peace across Somalia, demonstrate reforms programs, deepen in the unique operating with a focus on inclusive dividends, increase reforms, invest in private context political processes, near-term resilience to sector–led growth, security, justice, • Channel for shocks improve human economic foundations, coordinating development outcomes and revenue and development services resources • Coordination of policy dialogue • Recognized mechanism for distributing resources across FMS Source: Adapted from Aleph Strategies 2019, supported by the World Bank Group. Note: FMS = federal member states; HIPC = heavily indebted poor countries; IDA = International Development Association. 2.3 Arrears clearance and compliance with an International Monetary Fund staff– monitored program allowed Somalia to reach the decision point under the Heavily Indebted Poor Countries Initiative. Before arrears clearance, Somalia’s external debt stood at US$5.2 billion (as of end-2018), which was expected to be reduced to US$557 million (in net present value) after reaching the heavily indebted poor countries completion point (World Bank 2020a). With the help of a US$366 million bridge loan from the Norwegian government and other support from IDA, Somalia was able to clear its arrears, thus normalizing its financial relationship with the World Bank (World Bank 2020b). The clearance gave Somalia access to resources from IDA and, alongside a period of compliance with designated reforms under an International Monetary Fund staff–monitored program, paved the way to receive debt relief under the Heavily Indebted Poor Countries and Multilateral Debt Relief Initiatives. The government subsequently secured additional resources of US$1.9 billion during FY20–22 from the 18th Replenishment of IDA Turnaround Regime and the 19th Replenishment of IDA Turnaround Allocation and the IDA Regional Integration and Crisis Response Windows. The World Bank’s increased engagement included a first development policy 5 loan for Somalia, approved in FY20, which enabled the country to repay the Norwegian bridge loan for arrears clearance (figure 2.3). Figure 2.3. Evolution of World Bank Financing from Trust Funds to International Development Association 1,000 900 800 Amount (US$, millions) 700 600 500 400 300 200 100 0 2014 2015 2016 2017 2018 2019 2020 2021 2022 Fiscal year Development policy loan (IDA) Investment project financing (IDA) Investment project financing (trust fund) Sources: World Bank and Independent Evaluation Group databases. Note: The amount of investment project financing (trust fund) in 2014 was US$6,500,000; in 2017, it was US$500,000; no operations were approved in fiscal year 2013. IDA = International Development Association. Interim Strategy Note Period, 2014–19 2.4 As the international community and the World Bank reengaged with Somalia, an Interim Strategy Note was developed for FY14–16 with a focus on strengthening economic institutions and expanding economic opportunity. The Interim Strategy Note had two priorities, each of which had two focus areas (table 2.1). Table 2.1. Somalia Interim Strategy Note, Fiscal Years 2014–16 Priority Area Focus Area 1. Strengthening core a. Improving economic governance, including macrofiscal and public finance economic institutions management, and statistical capacity. b. Supporting the establishment and implementation of a sound regulatory framework in strategic economic sectors. 2. Expanding economic a. Supporting job creation through private sector investment through public- opportunity private partnerships, public-private dialogue, and targeted financial support. b. Financing basic rehabilitation of infrastructure for employment generation and productivity. Source: World Bank 2013. 6 2.5 After a long period of limited engagement, 2014 represented the turning point for Bank Group engagement in Somalia. During FY15–19, the average number of lending and nonlending projects and activities per year rose to four, with the commitment size of investment projects increasing manifold. Financing came from US$19.5 million committed under the State and Peacebuilding Fund, with additional resources mobilized from international partners through the World Bank–administered Multi-Partner Fund. There was an increase in analytical activities, which helped inform the eventual Systematic Country Diagnostic (SCD), published in late 2018 (World Bank 2018c). In this period, financing primarily focused on public sector management, starting in 2014 with the Somalia Public Financial Management Capacity Strengthening Project, the Recurrent Cost and Reform Financing Project, and the Somalia Capacity Injection Project (the so-called Troika programs). Until 2016, projects were implemented mostly in Mogadishu and less conflict-affected areas of Somalia, primarily in the north. The Special Financing Facility for Local Development approved in FY16 was the first project to be implemented across the south. 2.6 During FY13–22, the World Bank approved 48 financing operations totaling US$2.2 billion, with a shift in emphasis over time from public sector governance to human capital. Most operations were in public administration, which accounted for 17 projects representing US$556 million in commitments. This was followed by social protection, with nine operations totaling US$630 million. Most of the social protection commitments (US$555 million) have been approved since FY20 as part of IDA’s Somalia Crisis Recovery Project and the Shock Responsive Safety Net for Human Capital Project. During the same period, the World Bank delivered 63 advisory services and analytics, 30 of which were in public administration (appendix B). Country Program Framework Period, Fiscal Years 2019–22 2.7 In 2018, the World Bank prepared a Risk and Resilience Assessment (RRA) for Somalia, which identified two interlinked cycles of fragility. The RRA identified dual fragility “traps”: a political-fiscal trap and a longer-term vulnerability-resilience trap. It homed in on weak domestic revenues as a key driver of fragility, as these were seen to exacerbate the federal government’s inability to provide basic services (such as education and policing) to the population. This in turn contributed to Somali citizens’ mistrust of the state. This political-fiscal trap was closely connected to a vulnerability- resilience trap in which environmental degradation and climate change led to the degradation of livelihoods, displacement, unsustainable urbanization, and land conflict. In the absence of government capacity, humanitarian intervention was required, further weakening institutions and markets and increasing the long-term vulnerability of Somali citizens (figure 2.4). 7 Figure 2.4. Interlinked Cycles of Fragility Identified by the Risk and Resilience Assessment a. Political-fiscal trap (6- to 12-month cycle) b. Vulnerability-resilience trap (3- to 5-year cycle) Vulnerability Weak to natural domestic disasters revenues Undermining Mistrust of Fiscal Increasing of public constraint forced institutions institutions on delivery displacement and markets Continued Unsustainable Unsustainable privatization Short-cycle urbanization humanitarian of politics and land relief assistance institutions conflict Source: World Bank 2018b. 2.8 By 2019, the World Bank had transitioned from the Interim Strategy Note to a full CPF that covered FY19–22. The CPF emphasized institution building and maintained the Interim Strategy Note focus on generating economic opportunities. The 2018 SCD used a framework similar to the RRA’s dual fragility traps, renaming them the fragility trap and the resilience trap, and structured its priority areas in relation to them (World Bank 2018c). The RRA and SCD informed the FY19–22 CPF (World Bank 2018d). The CPF had two focus areas. Focus area 1, building institutions to deliver services, corresponds to the first SCD priority area. Focus area 2, restoring economic resilience and opportunities, addresses SCD priority areas 2 and 3. The Performance and Learning Review (FY23) realigned the focus areas as follows: (i) strengthening institutions and financing for social service delivery; (ii) enabling inclusive, private sector–led growth; and (iii) strengthening resilience (World Bank 2022). 2.9 Since clearing its arrears in 2020, Somalia has seen a significant increase in World Bank financing driven by access to IDA funds. In the transition from smaller trust- funded operations, commitments increased substantially beginning in FY20. These include the US$420 million Somalia Reengagement and Reform Support development policy financing operation approved in FY20. This budget support operation, together with its US$55 million supplemental operation, accounted for 20 percent of all financing received by Somalia in FY14–22 (figure 2.5). 8 Figure 2.5. Evolution of World Bank Lending and Nonlending, Fiscal Years 2013–22 12 1,000 Commitment (US$, millions) 10 800 8 Projects (no.) 600 6 400 4 200 2 0 0 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 Fiscal year Lending by commitment (DPF) World Bank lending by commitment (exc. DPF) World Bank lending by projects World Bank ASA by projects Source: Independent Evaluation Group staff analysis using World Bank operations data. Note: The majority of the loan amount of the 2020 DPF financed the repayment of the bridge loan supplied to Somalia by the government of Norway. ASA = advisory services and analytics; DPF = development policy financing; exc. = excluding. 2.10 The International Finance Corporation portfolio in Somalia was limited to advisory services. The International Finance Corporation conducted seven advisory services activities, most of them approved during FY19–21. The majority focus on market creation by addressing the investment climate and related institutions. 3. Purpose, Objectives, and Audience 3.1 The Somalia CPE seeks to inform the preparation of the next Bank Group– supported strategy for Somalia. It may also be of relevance to Bank Group reengagement in other FCV situations and the implementation of the Bank Group’s FCV strategy (2020–25). The CPE will assess the extent to which the Bank Group has tailored its support to the specific circumstances in Somalia, evolving country priorities and capacity, and learning from experience. Particular attention will be given to the period before arrears clearance to assess the extent to which World Bank support helped prepare for the eventual normalization of relations with Somalia and the associated scale-up in financing. 3.2 The CPE will focus on two key strategic objectives of the Bank Group in Somalia—increasing the capacity of Somalia to govern and manage its financial resources and improving Somalia’s ability to deliver services to citizens. It will also assess the extent to which engagement in Somalia was informed by the 2020 FCV strategy, which committed the Bank Group to enhance its effectiveness to support FCV 9 countries through a differentiated approach, including by helping address the drivers of conflict and strengthening resilience. 3.3 The primary audiences for this evaluation are the Bank Group Boards of Executive Directors and Bank Group staff and management working on Somalia and in the FCV Group. Although its focus is on assessing the Bank Group’s performance (rather than that of Somalia), it will likely also be of interest to Somalia’s national and subnational governments. The evaluation will seek to generate lessons that can serve Bank Group staff considering and designing future engagements with the country. The evaluation aims to help Bank Group management and the Boards of Executive Directors in their deliberations concerning the future CPF. It may also be of interest to a broader audience, both in and outside Somalia, including development partners, civil society organizations or researchers working on the country, and Somali citizens. 3.4 Although the Independent Evaluation Group (IEG) typically seeks to conclude CPEs before or at the start of work on a new CPF, this evaluation will be prepared in parallel with the next CPF for Somalia. This reflects the strong desire of the Somalia Country Management Unit to obtain the IEG’s input into its future engagement in Somalia. The timing will have implications for the way the evaluation is conducted because the CPE will seek entry points to inform the CPF. This engagement will involve the sharing of early findings from the evaluation to inform the preparation of the CPF. This approach has been closely coordinated with the Country Management Unit. 4. Evaluation Scope and Questions 4.1 The CPE will assess the relevance and effectiveness of Bank Group engagement with Somalia throughout the evaluation period. This will include an assessment of the sequencing of its support activities and its coordination with development partners. It will address three major cross-cutting questions of a strategic nature: 1. To what extent did the Bank Group seek to adapt and succeed in adapting its support for Somalia’s evolving development challenges, risks, and fragility context? 2. To what extent were the Bank Group’s modalities of engagement and program design appropriate for Somalia’s situation of fragility and conflict? 3. To what extent has the Bank Group contributed to building the capacity of the state to support Somalia’s longer-term development? 4.2 The first question will involve assessing how well the World Bank identified and prioritized its support in Somalia based on a sound understanding of Somalia’s challenges and constraints. It will allow IEG to look at how well the Bank Group 10 prepared itself (through diagnostic and analytical work to build country knowledge, choice of interventions, and strategic partnerships) and how well it helped prepare the client for the normalization of relations with IDA, including an eventual resumption of financing (for example, support for capacity building, strategic engagement with key government counterparts and reform champions, dialogue about development priorities, and laying the foundations for implementation arrangements). This question will also allow the evaluation to reflect on the outcome of the international community’s decision to rebuild Somalia by strengthening country systems instead of relying on parallel institutions. 4.3 The second question is intended to examine how the Bank Group adapted its approaches and program design to the situation of fragility and conflict in Somalia. IEG will address whether and how the Bank Group’s engagement in Somalia was appropriately tailored to the context—that is, was the Bank Group’s approach fit for purpose to engage in a country experiencing ongoing conflict with unclear and contested governance structures? This will also include an assessment of instruments and instrument choice, policies and procedures, programs and implementation modalities, stakeholder interests, degree of local presence, staffing and other resources, risk identification and mitigation, and ability to learn. It will also assess the use of strategic partnerships, including to support Somalia’s participation in regional integration activities. Finally, it will examine the impact on relationships with partners once Somalia became eligible for IDA financing. 4.4 The third question reflects the criticality of state building in Somalia, which has teetered on the brink of being a failed state. The evaluation will examine the extent to which the Bank Group contributed to strengthening state capacity, especially in the core state functions of public financial management, domestic revenue mobilization, and delivery of services to citizens. Methodology 4.5 Tackling the above questions will require assessing the relevance and early results of Bank Group support to Somalia and its challenges (including content, modalities, sequencing, and adequacy of analytical underpinnings). The evaluation will seek to identify and assess the early results of Bank Group support during the evaluation period. This will include an assessment of implementation modalities and its appropriateness to the country context and the use of partnerships. To the extent possible, IEG will seek to assess impact and outcomes; however, the relative newness of much of the portfolio means that there are few completed projects for which results have been assessed. Nevertheless, the evaluation will examine how analytical, advisory, and 11 diagnostic work informed the Bank Group’s efforts to achieve the stated objectives of its engagement. 4.6 The evaluation will seek to assess the contributions of the Bank Group to observed outcomes. The team will identify intermediate steps and pathways needed to prepare for the eventual normalization of relations with Somalia (including for building the economic capacity of the state and for the government’s focus on arrears clearance) and link these to project outputs and the extent to which these outputs contribute to those outcomes. For recent engagements, the evaluation will focus on learning from aspects of relevance, design, and early implementation. This will include coherence and consistency of the program with the needs of the country, adaptation of design to the situation in the field, implementation arrangements, and early implementation experience and emerging lessons. 4.7 The evaluation will apply several methods adapted to the circumstances of Somalia. Appendix A provides a detailed design matrix, which maps evaluation questions to methods. The team envisions using the following methods: • Structured literature review. The team will analyze contemporaneous literature and diagnostic work prepared by the World Bank, its partners, and relevant external sources to identify key constraints and development priorities in Somalia. This will allow the team to compare World Bank diagnostics with those of other agencies. • Stakeholder analysis. The team will map stakeholders to assess the role and interests of different stakeholders within government, the private sector, civil society organizations and think tanks, partner organizations, and key categories of citizens. The stakeholder mapping will help identify the interlocutors to interview and the interests of citizen groups and government and nongovernment actors that will need to be considered. The mapping will also inform the composition of possible stakeholder workshops for the evaluation. • Analytical framework for state capacity building. The team will develop a framework to assess contributions to state capacity building. The framework will unpack the different areas of state capacity, including public financial management, domestic revenue mobilization, and service delivery, and identify measures to assess the Bank Group’s contributions in each area. • Semistructured interviews with Bank Group staff, partners, government officials, and key stakeholders in civil society organizations. The team will conduct and analyze interviews with a broad range of stakeholders, including Bank Group staff, humanitarian and development partners, implementing 12 agencies, government officials, civil society organizations, and private sector representatives. The CPE will conduct a field mission to Nairobi to interview Country Management Unit staff and management, partners, and other key stakeholders. If feasible, it will carry out a mission to Mogadishu to engage with senior government officials, civil society organizations, the private sector, and other local stakeholders. • Expert and stakeholder workshops. The team will explore the use of a panel of representatives of partner organizations active in Somalia and of key stakeholders to help shape its understanding of key development challenges and conflict drivers as a benchmark to assess the relevance and prioritization of World Bank support. The evaluation team will also explore the possibility of convening workshops with stakeholder representatives, such as think tanks, civil society organizations, private sector, and citizen groups. • Analysis of Bank Group and external data and surveys. Within the limitations of available data, the team will analyze data from World Bank and external sources. These include data on conflict, socioeconomic trends, institutional capacity, the Bank Group portfolio, and internal Bank Group data related to staffing and resources. The team will also draw on existing evaluations of trust funds or partner programs in Somalia and available survey data. Three sources of perception data have been identified: (i) Somali Perceptions Survey (2017), commissioned by the United States Agency for International Development to provide its Somali office with information to inform its policy and project implementation; (ii) Perception Survey of Aid Recipients in Somalia (2020), conducted by Ground Truth Solutions; (iii) the Somali Pulse, which provides access to data on livelihoods, consumption patterns, and perceptions of Somali citizens. • Document analysis of advisory services and analytics and operations. The team will conduct a detailed analysis of the Bank Group’s advisory services and analytics to assess relevance to key development challenges and the extent to which they informed World Bank support and helped the World Bank prioritize and adapt its engagement. For investment projects and budget support operations, the team will analyze deliberative, approval, supervision, and evaluation documents (both published and unpublished). This will help assess the degree of adaptation, awareness of risk, and adaptation of project design and implementation to Somalia’s circumstances. • Geospatial analysis. The team will explore the usefulness of geospatial analysis to map the footprint of Bank Group operations to data on population and 13 poverty to assess the consistency of support with areas in greatest need of development assistance. 4.8 The evaluation will use several benchmarks and data sources to address the evaluation questions: • For question 1. The team will examine the consistency of World Bank diagnostics with contemporaneous external diagnostic work and views of experts, the consistency of World Bank priorities with the findings of diagnostic work, and the consistency with its comparative advantage and the work of other partners. • For question 2. The team will use criteria such as staffing and staff location, and the frequency and nature of staff-client interactions and will review Implementation Status and Results Reports and back-to-office reports and the few Implementation Completion and Results Reports and Implementation Completion and Results Report Reviews to assess the adequacy of approaches and program design. • For question 3. Evaluation criteria will include progress in implementation and early results identified in the analytical framework for state capacity building; available perception survey data on the availability, reach, and quality of government services; World Bank and analytical evidence, such as Country Policy and Institutional Assessment and Public Expenditure and Financial Accountability assessments; and benchmarks available from governance indicators, such as the World Governance Indicators. Limitations 4.9 The evaluation faces several challenges and limitations: • The security environment in Somalia limits direct fieldwork. Because much of the Bank Group’s support to Somalia is conducted from Nairobi, this will be the focus of the team’s mission travel and interviews with key stakeholders. Security conditions permitting, the team plans to visit Mogadishu to ground the analysis in the local context, carrying out face-to-face meetings with government officials, key project implementation staff, and representatives of civil society organizations, think tanks, and the private sector. • Somalia suffers from a significant lack of data, and existing data are often unreliable. Despite efforts by the World Bank and others to increase the availability of socioeconomic data, the data environment remains very 14 inadequate. The analysis will consider the adequacy of the Bank Group’s investment in data capacity and the extent to which Somalia was able to benefit from regional statistical capacity-building initiatives while paying due attention to potential constraints imposed by the availability, completeness, and quality of relevant data (including subnational and gender-disaggregated data). To increase the consistency and validity of the analysis, the evaluation will make maximum use of secondary data while attempting to triangulate evidence to the extent feasible. • Limited evidence from closed projects and the difficulty in attributing results to Bank Group interventions will add to the evaluation’s challenges. Seven World Bank projects have been assessed to date by IEG, limiting the ability of the CPE to examine their effectiveness. In addition, given the presence of many development partners, attributing broader changes to any one intervention or actor may be difficult. A preliminary review of Implementation Completion and Results Report Reviews showed that several lending operations use output indicators to track their results. This may be appropriate given the complex circumstances of project implementation in Somalia, but it will make it more difficult to link World Bank projects to country- or higher-level outcomes. To address this, the team will look at the contribution of the more mature portfolio to higher-level outcomes and assess coherence, design, and early implementation issues for recent operations. 5. Quality Assurance Process 5.1 The evaluation will follow IEG’s standard quality assurance and external review process. The evaluation will be subjected to review by IEG management and peer reviews. Three external reviewers will provide guidance and quality assurance for the evaluation: • Damon Bristow, development director, Somalia (since 2019), United Kingdom Foreign, Commonwealth & Development Office, Nairobi, Kenya. • Samira Gaid, executive director of the Hiraal Institute, a Mogadishu-based think tank. She is a regional and security analyst with extensive experience and has served as the special adviser to the prime minister of Somalia, Hassan Ali Khaire, from May 2017 to July 2020, during which she led transformation and reform efforts of the Somali security sector. • Deborah Wetzel, former World Bank senior director, Governance Global Practice, and former director for regional integration, Africa and Middle East and North Africa. 15 6. Timeline and Team 6.1 IEG will prepare an evaluation report with key findings and lessons to be submitted to the Board’s Committee on Development Effectiveness in the fall of 2023. The task team leader for the evaluation is Stephan Wegner (senior evaluator), and core team members are Mees van der Werf (economist) and Anis Dani, Daniel Palazov, Yongmei Zhou, and Marwane Zouaidi (consultants). The work will be conducted under the guidance of Jeffrey Allen Chelsky (manager), Carmen Nonay (director, Finance, Private Sector, Infrastructure, and Sustainable Development Department, and acting director, Human Development and Economic Management Department), and Oscar Calvo-Gonzalez (acting Director General, Evaluation). 1Al-Shabaab, a fundamentalist group, assumed control over much of southern and central Somalia after the Somalia War (2006–09). 2World Bank Operational Policy 7.30, Dealing with De Facto Governments, is triggered when a de facto government comes into power by means outside of a country’s constitution, such as a coup d’état, revolution, or suspension of the constitution, or when no government is in place. When this policy is triggered, the World Bank suspends disbursements on projects under implementation. When a country is under this policy, the World Bank will generally not approve new operations. This lasts until the World Bank determines that a proper legal framework is in place and that all parties have carried out their obligations as agreed with the World Bank. 3To develop the compact, the Somali president established the High-Level Task Force, chaired by the Somali minister of finance and comprising senior representatives from the Somali government, the g7+ Focal Point, the United Nations, and the European Union (Hearn and Zimmerman 2014). 4The Somalia Multi-Partner Fund was established in 2013 to support government-led state building, economic growth, and urban development, supported by Denmark, the European Union, Finland, Germany, Italy, Norway, the State and Peacebuilding Fund, Sweden, Switzerland, the United Kingdom, and the United States. The Multi-Partner Fund is one of the funding windows to support implementation of Somalia’s ninth National Development Plan. 5 The State and Peacebuilding Fund, established in 2008, is a multidonor trust fund to provide catalytic funding to fragility, conflict, and violence situations. As of December 31, 2020, its active portfolio included 56 grants with over US$39 million in commitments. The net value of the fund is over US$371 million. Its donors are the International Bank for Reconstruction and Development and nine development partners—Australia, Denmark, France, Germany, the Netherlands, Norway, Sweden, Switzerland, and the United Kingdom. 16 Bibliography Aleph Strategies. 2019. Multi-Partner Fund for Somalia Mid-Term Review: Final Report. Whyteleafe: Aleph Strategies. Famine Early Warning Systems Network. 2020. “Somalia Livestock Price Bulletin.” Situation Report, Famine Early Warning Systems Network. https://reliefweb.int/report/somalia/somalia-livestock-price-bulletin-june-2020. Faruk, Omar. 2022. “Somali Forces End Hotel Attack That Left 21 Dead, 117 Hurt.” Associated Press, August 21, 2022. Hearn, Sarah, and Thomas Zimmerman. 2014. A New Deal for Somalia: The Somali Compact and Its Implications for Peacebuilding. New York: Center on International Cooperation, New York University. https://cic.nyu.edu/resources/a-new-deal-for-somalia-the-somali-compact- and-its-implications-for-peacebuilding. Hiraal Institute. 2020. A Losing Game: Countering Al-Shabab’s Financial System. Mogadishu: Hiraal Institute. https://hiraalinstitute.org/wp-content/uploads/2020/10/A-Losing-Game.pdf. ICAI (Independent Commission for Aid Impact). 2017. UK Aid in a Conflict-Affected Country: Reducing Conflict and Fragility in Somalia. London: ICAI. https://icai.independent.gov.uk/review/somalia. ICRC (International Committee of the Red Cross). 2021. “‘If Your Animals Die, You Die with Them’—Somali Herders Defenceless against the Climate Crisis.” September 13, 2021. https://www.icrc.org/en/document/somalia-herders-climate-change-conflict-crisis. Somalia, Federal Government. 2021. FGS Quarterly Financial Report: Quarter Four 2020. Mogadishu: Office of the Accountant General, Ministry of Finance. https://mof.gov.so/index.php/publications/fgs-quarterly-financial-report-quarter-four- 2020. Somalia, Ministry of Planning, Investment and Economic Development. 2020. Somalia National Development Plan 2020 to 2024. Mogadishu: Federal Government of Somalia. https://andp.unescwa.org/sites/default/files/2020- 10/Somalia%20National%20Development%20Plan%202020%20to%202024.pdf. UNHCR (United Nations High Commissioner for Refugees) and NRC (Norwegian Refugee Council). 2022. “One Million People Displaced by Drought in Somalia.” Press Release, August 11, 2022. https://www.unhcr.org/en-us/news/press/2022/8/62f4c3894/million- people-displaced-drought-somalia.html. World Bank. 2013. Somalia—Interim Strategy Note for the Period FY14–16. Washington, DC: World Bank. https://documents.worldbank.org/en/publication/documents- reports/documentdetail/795741468103478907/somalia-interim-strategy-note-for-the- period-fy14-fy16. 17 World Bank. 2018a. Rebuilding Resilient and Sustainable Agriculture in Somalia. Washington, DC: World Bank. https://documents1.worldbank.org/curated/en/781281522164647812/pdf/124651- REVISED-Somalia-CEM-Agriculture-Report-Main-Report-Revised-July-2018.pdf. World Bank. 2018b. Risk and Resilience Assessment for Somalia: Input to the Somalia Country Partnership Framework FY19–22. Washington, DC: World Bank. World Bank. 2018c. Somalia. Systematic Country Diagnostic. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/30416. World Bank. 2018d. Somalia—Country Partnership Framework for the Period FY19–22. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/941361537906125793/Somalia-Country- Partnership-Framework-for-the-Period-FY19-FY22. World Bank. 2019. Somali Poverty and Vulnerability Assessment: Findings from Wave 2 of the Somali High Frequency Survey. Washington, DC: World Bank. https://openknowledge.worldbank.org/entities/publication/2869359f-a8b7-5a1a-aee4- f6ba7177a5ce. World Bank. 2020a. “Reengaging with Somalia for Long-Lasting Development Outcomes.” News, October 1, 2020. https://www.worldbank.org/en/news/feature/2020/10/01/reengaging- with-somalia-for-long-lasting-development-outcomes. World Bank. 2020b. “Somalia Clears Arrears to World Bank Group.” Press Release, March 5, 2020. https://www.worldbank.org/en/news/press-release/2020/03/05/somalia-clears- arrears-to-world-bank-group. World Bank. 2020c. Somalia Urbanization Review: Fostering Cities as Anchors of Development. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/35059. World Bank. 2021. Investing in Health to Anchor Growth. Washington, DC: World Bank. https://documents1.worldbank.org/curated/en/926051631552941734/pdf/Somalia- Economic-Update-Investing-in-Health-to-Anchor-Growth.pdf. World Bank. 2022. Somalia—Performance and Learning Review of the Country Partnership Framework. Washington, DC: World Bank. 18 Appendix A. Evaluation Design Matrix Table A.1 indicates the key questions, information required, sources, and data collection and analysis methods. Table A.1. Evaluation Design Matrix Key Questions Information Required Data Collection Methods 1. To what extent did Development challenges and Structured analysis of contemporaneous literature the World Bank evolving risks in Somalia and diagnostic (World Bank and external) regarding Group seek to adapt Appropriateness of areas of key constraints and succeed in prioritization and selectivity of World An expert panel, complementary interviews, or both adapting its support Bank support informed by with key stakeholders in the World Bank, among for Somalia’s diagnostic work and by an partners, government officials, civil society evolving understanding of the role of organizations, and think tanks development partners and its comparative Analysis of existing macro, fragility, and development challenges, risks, and advantage data (WDI, ACLED, and so on) fragility context? Alignment of World Bank support Analysis of World Bank ASA to determine coherence with priorities identified in with diagnostic work (focused on right issues?) and diagnostic work the degree to which they informed or influenced and were coherent with subsequent World Bank support (advice, policy dialogue, lending) Analysis of World Bank portfolio approval, supervision, and evaluation documents, where available Geospatial analysis mapping World Bank support within the country, mapping activities versus needs, if deemed useful to provide additional evaluative insights 2. To what extent Data on roles and interests of key Stakeholder mapping; stakeholder workshops with were the Bank stakeholders and citizen groups key interlocutors and groups Group’s modalities Data on staffing, staff location, Interviews with World Bank staff (current and former of engagement and staffing levels, and frequency of CMU, regional management, and TTLs), development program design staff-client interactions; instrument partners (including implementing agencies), appropriate for use, implementation modalities, and government officials, and other stakeholders Somalia’s situation other aspects pertaining to Review of relevant World Bank databases on staffing, of fragility and adaptations of the World Bank’s budgets, resources, risks, and so on conflict? business model Structured review of portfolio and of operations (ISRs, Data on the extent to which the BTORs, ICRRs) to assess the degree of adaptation and Bank Group adapted its approach to flexibility in programming, implementation, and facilitate partnerships and monitoring and to assess course correction due to implementation arrangements to shifting circumstances. For ASA, the team will rely on expand the Bank Group’s footprint evidence of collaborative work and a mix of and effectiveness within Somalia diagnostic studies and just-in-time policy notes or advisory activities. Review of trust funds and implementation arrangements for a subset of activities 3. To what extent Analytical framework for state Analysis of macro and development data trends, has the Bank Group building capacity including on governance contributed to Data on relevance and effectiveness Analysis of existing perception survey data and building the of World Bank support to select feedback from stakeholder workshops capacity of the aspects of state building and their 19 Key Questions Information Required Data Collection Methods state to support intermediate steps, including on Review of ASA and lending portfolios and approval, Somalia’s longer- design and implementation of supervision, and evaluation documents, where term development? support and, where feasible, available contributions to increased capacity Analysis of Bank Group contributions to state capacity and reach of state institutions building Review of available literature and diagnostic work on the public sector in Somalia Stakeholder interviews with government officials, development partners, and civil society organizations; interviews with World Bank staff (CMU, TTLs) Source: Independent Evaluation Group. Note: ACLED = Armed Conflict Location & Event Data; ASA = advisory services and analytics; BTOR = back-to-office report; CMU = Country Management Unit; ICRR = Implementation Completion and Results Report Review; ISR = Implementation Status and Results Report; TTL = task team leader; WDI = World Development Indicators. 20 Appendix B. Preliminary Portfolio Review The Independent Evaluation Group conducted a preliminary identification and classification of the World Bank Group portfolio in Somalia over fiscal years (FY)13–22. The Somalia Country Program Evaluation (CPE) portfolio covers the Bank Group’s engagement in the country based on activities that were approved during this period. Portfolio identification is based on information extracted from the World Bank’s Enterprise Data Catalog (World Bank financing and advisory services and analytics) and the management information system and advisory services databases of the International Finance Corporation (IFC; investments and advisory services). For both World Bank financing and IFC, activities approved and closed before the start of FY13 (before July 1, 2012) and approved after the end of FY22 (on or after July 1, 2022) are thus excluded from the respective portfolios. For World Bank advisory services and analytics, any activity approved before FY13 that is no longer active or was signed off after FY22 is also excluded. The Multilateral Investment Guarantee Agency had no guarantees in Somalia during the evaluation period and is therefore not covered by this CPE. World Bank Lending During the evaluation period (FY13–22), 48 financing operations were approved for US$2,208 million. These include 36 new commitments and 12 additional financings. Most operations were in public administration, which accounted for 17 projects representing US$556 million in commitments, excluding the two development policy loans. This was followed by social protection, with nine operations totaling US$630 million. Most of the social protection commitments (US$555 million) have been approved since FY20 as part of the International Development Association’s Somalia Crisis Recovery Project and the Shock Responsive Safety Net for Human Capital Project. Figure B.1 shows the distribution of World Bank financing by sector. The federal government is listed as the primary government implementing agency in 40 of the 48 projects, with the Ministry of Finance involved in over a third of the projects. Only a single project was implemented by an international nongovernmental organization (the International Rescue Committee). The six federal member states and several of their ministries (Education, Energy, Finance, Health) are listed as implementing agencies for the remaining 8 out of 48 projects. 21 Figure B.1. World Bank Financing to Somalia a. World Bank financing by major sector, fiscal years 2013–22 (excluding development policy operations) 700 20 Commitment (US$, millions) 18 600 16 500 14 Projects (no.) 400 12 10 300 8 200 6 4 100 2 0 0 Sector Commitment b. World Bank financing by sector share, fiscal years 2013–22 (excluding development policy operations) (US$, millions) Transportation;Water/Sanit/Waste, $6 Transportation;Industry & Trade/Ser, $5 Agriculture, $44 Education, $65 Energy & Extractives, Financial Sector, $68 $158 Financial Sector;Industry & Trade/Ser, $3 Social Protection, $630 Health, $140 Industry & Trade/Ser, $29 Info & Communication, $32 Public Admin, $554 Source: Independent Evaluation Group staff analysis of World Bank operations data. Note: Admin = Administration; Info = Information; Sanit = Sanitation; Ser = Services. 22 World Bank Advisory Services and Analytics The World Bank delivered 63 advisory services and analytics activities during the CPE period (FY13–22), with 38 completed in FY13–19 and 25 since FY20 (figure B.2). Public Administration was the largest sector in both periods, with 15 activities each, for 30. At the same time, there was a shift away from social protection (5 activities in the earlier period, and none since FY20). 23 Figure B.2. World Bank Advisory Services and Analytics by Major Sector, Somalia, Fiscal Years 2013–22 a. Fiscal years 2013–19 (total 38) Energy and Extractives, 4 Social Protection, 5 Industry and Trade/Services, 1 Agriculture, 3 Health, 2 Public Administration; Social Protection, 1 Education, 3 Financial Sector, 3 Public Administration, Industry and 15 Trade/Services; Financial Sector, 1 b. Fiscal years 2020–22 (total 25) Water/Sanitation/Waste, 2 Health, 1 Information and Communication, 1 Industry and Trade/Services, 1 Agriculture, 1 Public Administration; Social Protection, 1 Education, 1 Financial Sector, 2 Public Administration, 15 Source: Independent Evaluation Group staff analysis using World Bank operations data. 24 Figure B.3. Evolution of World Bank Lending and Nonlending, Fiscal Years 2014–22 12 1,000 900 10 800 Commitment (US$, millions) 700 8 Projects (no.) 600 6 500 400 4 300 200 2 100 0 0 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 Fiscal year World Bank lending by commitment World Bank lending by projects World Bank ASA by projects Source: Independent Evaluation Group staff analysis using World Bank operations data. Note: ASA = advisory services and analytics. Evaluated Projects Seven projects that closed after FY10 have been evaluated by the Independent Evaluation Group, six of which were approved during the CPE period (FY13–22). Of these, all were investment projects. Bank performance was satisfactory across all evaluated projects (table B.1). 25 Table B.1. Evaluated World Bank Financing Projects Approval Overall Quality at Quality at Overall Bank Project Fiscal Evaluation Major Sectors Outcome Efficacy Entry Supervision Performance ID Project Name Year Fiscal Year (%) Rating Rating Rating Rating Rating P148428 Somalia Recurrent 2015 2016 • Public Administration (80) Moderately Substantial Satisfactory Satisfactory Satisfactory Cost and Reform • Education (10) satisfactory Financing Facility • Health (10) P150374 Somali Urban 2016 2023 • Public Administration (20) Satisfactory Substantial Satisfactory Satisfactory Satisfactory Investment Planning • Transportation (40) • Water/Sanitation/Waste (40) P152241 Somali Core 2016 2021 • Industry and Trade/Services (73) Satisfactory Substantial Satisfactory Satisfactory Satisfactory Economic Institutions • Financial Sector (18) and Opportunities • Transportation (9) Program P152358 ICT Sector Support in 2015 2021 • Information and Communication (70) Satisfactory Substantial Satisfactory Satisfactory Satisfactory Somalia Phase II • Education (20) • Public Administration (10) P156257 Special Financing 2016 2021 • Public Administration (48) Moderately Substantial Satisfactory Satisfactory Satisfactory Facility for Local • Water/Sanitation/Waste (13) satisfactory Development • Transportation (13) • Health (13) • Education (13) P163857 Somalia Urban 2019 2023 • Social Protection (34) Moderately Substantial Satisfactory Satisfactory Satisfactory Resilience Project • Transportation (33) satisfactory • Public Administration (33) Source: Independent Evaluation Group. Note: ICT = information and communication technologies. 26 International Finance Corporation Activities The IFC portfolio in Somalia remains limited to advisory services. IFC conducted seven advisory services activities for US$18.25 million in funds managed by IFC, all except one approved during FY19–21. The majority focus was on market creation by addressing the investment climate and related institutions. In terms of primary business line, three were in Regional Advisory, two in the Financial Institutions Group, and one each in the Financial Institutions Group and in Equitable Growth, Finance, and Institutions. IFC did not commit to nor had any outstanding investments in Somalia during its reviewed period. None of the seven IFC advisory services have been evaluated by the Independent Evaluation Group. 27 Appendix C. Independent Evaluation Group and Other Evaluations on Somalia These evaluations are in addition to Independent Evaluation Group validations of project- and operation-specific Implementation Completion and Results Reports and Country Partnership Framework Completion and Learning Reviews. Somalia Case Studies in Independent Evaluation Group Thematic Evaluations, Fiscal Years 2010–22 • World Bank Group Support in Situations Involving Conflict-Induced Displacement (2019) • World Bank Engagement in Situations of Conflict: An Evaluation of FY10–20 Experience (2021) • Enhancing the Effectiveness of the World Bank’s Global Footprint (2022) Project Performance Assessment Reports, Fiscal Years 2010–22 • No Project Performance Assessment Reports were prepared for projects closing during the evaluation period. Other Evaluations on Somalia, 2010–22 • United States Agency for International Development: Mid-Term Performance Evaluation of the Somalia Partnership for Economic Growth Program (2014) • United Kingdom Department for International Development: Shocks & Hard Knocks: The Impact of Resilience Programming in Somalia (2019) • Independent Commission for Aid Impact: UK Aid in a Conflict-Affected Country: Reducing Conflict and Fragility in Somalia (2017) • United Nations Development Programme: Independent Country Programme Evaluation: Somalia (2019) • Aleph Strategies: Multi-Partner Fund for Somalia Mid-Term Review: Final Report (2019) 28