Approach Paper Kyrgyz Republic Country Program Evaluation December 6, 2021 The Country Program Evaluation (CPE) for the Kyrgyz Republic seeks to assess the World Bank Group’s efforts to help the Kyrgyz Republic address its main development challenges. The evaluation will cover fiscal years (FY)14–21 and is timed to inform the next Country Partnership Framework (CPF) between the Kyrgyz Republic and the Bank Group. The evaluation will assess (i) how relevant was the Bank Group’s strategy and how it evolved over time, given changes in the country context and lessons from experience; (ii) the extent to which Bank Group support helped the Kyrgyz Republic foster increased private sector–led growth to reduce economic vulnerability; (iii) the extent to which Bank Group support helped improve central government governance and institutional capacity; and (vi) the extent to which Bank Group support improved local governance and the quality of, and access to, local public services. 1. Context and Development Challenges Country Context 1.1 The Kyrgyz Republic is a lower-middle-income country, highly dependent on remittances, with a vibrant civil society. It is one of the poorest countries in the Europe and Central Asia Region. Its per capita gross domestic product (GDP) in 2020 was US$1,174 (in current dollars); before the coronavirus (COVID-19) crisis, per capita GDP had grown slowly for six years (at between 1 and 3 percent per year) before falling by 10.5 percent in 2020. Remittances represented approximately 30 percent of GDP during 2013–19, making the Kyrgyz Republic one of the world’s most remittance-dependent economies. The economy is also heavily dependent on gold mining, with the Kumtor mine (nationalized in June 2021) scheduled to close in 2031. The country is the only democracy in Central Asia. Although it undertook economic liberalization soon after independence in 1991, reform momentum slowed over the past two decades because of political instability (and its underlying drivers). 1.2 The country is landlocked, and its mountainous geography is difficult to traverse. One of the world’s highest mountain ranges separates the northern and southern regions of the country. Hundreds of mountain peaks dot the rest of the territory. Economic activity is concentrated around cities in flat regions of the country while other regions, 1 particularly in the south of the country, continue to lag. Approximately two-thirds of the population live in rural areas on steep terrain, making travel among communities difficult. 1 1.3 Consistency in economic policy and follow-through on reform implementation have been major challenges due to persistent political instability. Figure 1.1 presents a brief timeline of major political and economic events and indicators. Since independence, the country has experienced three revolutions—in 2005, 2010, and 2020— that led to change in leadership and were accompanied by violence and looting. The 2010 revolution was followed by violent clashes between the Kyrgyz and Uzbek communities in the south. In 2021, an armed conflict erupted between the Kyrgyz Republic and Tajikistan. Since 2011, there have been 10 changes of government; the average tenure of the cabinet of ministers is less than one year. Political instability and civil unrest have been inextricably linked to perceptions of corruption and misgovernance; the 2020 revolution was also driven by public dissatisfaction with the ability of the government and the legislature to create a long-term sustainable political environment (World Bank 2018a). In 2019, the number of firms ranking corruption and political instability as the top constraints to doing business was almost twice the average in the Europe and Central Asia Region and in lower-middle-income countries (World Bank 2020a, 4). 2 Figure 1.1. Kyrgyz Republic Economic and Political Timeline Source: Independent Evaluation Group. Note: GDP = gross domestic product. 1.4 Although poverty declined from 37 percent in 2013 to 20 percent in 2019, roughly 65 percent of the population is vulnerable to falling into poverty, and poverty remains high in areas outside of Bishkek (World Bank 2018a). 3 A large portion of the population live just above the poverty line, making them vulnerable to shocks (World Bank 2018a). 2 Since independence, the country has seen a pattern of severe shocks, including natural disasters, volatile global and regional food prices, economic crises, government instability, and ethnic conflict (USAID 2015). The ongoing coronavirus pandemic is an additional shock. Table 1.1 presents selected economic and social indicators. Table 1.1. Selected Economic and Social Indicators 2020 Indicator 2013 2014 2015 2016 2017 2018 2019 (est.) GDP growth (annual %) 10.9 4.0 3.9 4.3 4.7 3.8 4.6 (8.6) GDP per capita growth (annual %) 8.7 2.0 1.8 2.2 2.7 1.7 2.4 (10.5) GDP per capita (current US$) 1,282 1,280 1,121 1,121 1,243 1,308 1,374 1,174 Current account balance (% of GDP) (13.8) (17.4) (15.8) (11.6) (7.0) (11.6) (12.0) — General government gross debt (% of GDP) 47.1 53.6 67.1 59.1 58.8 54.8 54.1 74.3 Overall fiscal balance (net lending or borrowing, % of GDP) (3.7) (3.1) (2.5) (5.8) (3.7) (0.6) (0.1) (6.8) Remittances (% of GDP) 31.1 30.0 25.3 29.3 32.3 32.5 27.2 28.4 Rural population (% of total 64.5 64.4 64.2 64.1 63.9 63.6 63.4 63.1 population) Poverty headcount ratio at national 37.0 30.6 32.1 25.4 25.6 22.4 20.1 — poverty lines (% of population) Gini index (World Bank estimate) 28.8 26.8 29.0 26.8 27.3 27.7 29.7 — Sources: World Development Indicators (World Bank) and World Economic Outlook Database (International Monetary Fund). 2020 figures are estimates. Note: — = n.a.; GDP = gross domestic product. Main Development Challenges Governance and Institutional Weaknesses 1.5 Although the Kyrgyz Republic is notable in Central Asia by the openness of its political process and scope for voice and agency, governance weaknesses are pervasive. The country was a parliamentary republic until April 2021, when a constitutional referendum abolished the post of prime minister and concentrated power in the presidency, reducing the number of independent ministries or agencies from 48 to 16. It has a functioning democracy, enjoys a vibrant civil society and relatively free media, and compared with regional peers is advanced in terms of respect for human rights and fundamental freedoms. However, it has been unable to capitalize on these assets and continues to suffer from weak governance, corruption, inefficient rule of law, and north- south tensions. The country performs well below the average for lower-middle-income economies in terms of rule of law, control of corruption, and political instability (figure 1.2). In 2020, the Kyrgyz Republic ranked 124th out of 180 economies in Transparency International’s Corruption Perceptions Index, with a score of 31 out of 100. 3 Figure 1.2. Performance on Worldwide Governance Indicators, Kyrgyz Republic versus Lower-Middle-Income Economies, 2019 Control of corruption 60 40 Voice and accountability Government effectiveness 20 0 Political stability and Rule of law absence of violence or terrorism Regulatory quality Kyrgyz Republic Lower-middle-income economies Source: Worldwide Governance Indicators (WGI). The WGI are a research data set summarizing the views on the quality of governance provided by a large number of enterprises and citizen and expert survey respondents in industrial and developing countries. These data are gathered from a number of survey institutes, think tanks, nongovernmental organizations, international organizations, and private sector firms. The WGI do not reflect the official views of the World Bank, its executive directors, or the countries they represent. The WGI are not used by the Bank Group to allocate resources. Note: A lower score (toward the center of the figure) indicates worse performance. 1.6 Governance has been identified by the Bank Group as the main priority for fostering better development results in the Kyrgyz Republic. In a Bank Group client survey undertaken in early 2013, almost 40 percent of respondents said that public sector governance was the country’s top development priority, followed by 35 percent who cited corruption as its biggest challenge. Nearly 70 percent of firms cite governance as the major constraint to doing business, and corruption contributes to gaps between de jure design of regulation and de facto implementation (World Bank 2018c). The 2018 Bank Group Systematic Country Diagnostic identified 11 constraints to advancing the twin goals in the Kyrgyz Republic and stressed that progress on governance would leverage gains in virtually all other areas (World Bank 2018a). 4 Need to Diversify Sources of Private Sector–Led Economic Growth 1.7 Despite the Kyrgyz Republic being an early leader among Central Asian countries in economic liberalization and democratization, broad-based economic growth remains elusive. The country depends on gold (primarily from the Kumtor mine) for over 40 percent of exports and 10 percent of GDP (World Bank 2020a). The economy has become more dependent on remittances over time, from 12 percent in 2005 to nearly one-third of GDP in 2018, the second-highest in the world. 5 Productivity is generally low 4 and has grown by 0.5 percent per year on average since 2000 (World Bank 2018a). The country has a large informal sector estimated at over 30 percent of GDP in 2015 (World Bank 2020a). The financial sector is underdeveloped; the ratio of private credit to GDP was 16 percent in 2013, 6 and although it has increased since (to 28 percent as of 2020), only 9 percent of firms finance investment through bank loans (IMF 2019). 1.8 To enhance economic resilience, the Kyrgyz Republic needs to diversify sources of private sector–led economic growth. However, endowments apart from gold have not been exploited, and political instability and weak rule of law deter investment. The country has the potential to develop hydropower and tourism (IFC 2021), but the first has not been exploited and the second is in an incipient stage. Cross-cutting reforms, including in information and communications technologies, business regulation, investment policy, and financial infrastructure and regulation may open up opportunities across other sectors (IFC 2021). Less than 7 percent of land is arable, and rural workers tend to prefer emigration to the Russian Federation; thus, the rural reservation wage approximates the construction wage in Russia. Future sources of growth will depend on the quality of the investment climate and the private sector’s ability to capitalize on opportunities. Firms—both foreign and domestic—are reluctant to invest because of political and policy instability, which make the returns on investment and appropriability of those returns uncertain. The uncertainty comes largely from frequent changes in government combined with a weak rule of law. 1.9 Foreign direct investment and small and medium-size enterprises play undersized roles in the economy. The Kyrgyz Republic has struggled to attract and retain foreign direct investment beyond the extractives sector, and there is little evidence of knowledge and technology spillover or backward links to the economy (IFC 2021). There is also a “missing middle” in the domestic economy; small and medium enterprises, which can be important sources of innovation and job creation, play a more limited role than in comparator countries. In 2018, there were fewer than 800 medium- size enterprises in the country, representing only 0.1 percent of the total number of enterprises, a very low proportion. 7 Large enterprises (10 percent of which are state- owned enterprises) are mostly concentrated in mining, energy, banking, and communications. Formally registered small and medium enterprises accounted for only 3.6 percent of employment and 10.5 percent of GDP, and individual entrepreneurs accounted for 16.8 percent of employment and 23.3 percent of GDP (World Bank 2020a). 8 This is well below the 50–60 percent share in employment and 65–75 percent share in GDP seen in advanced economies and below regional comparators with relatively open economies (figure 1.3). 5 Figure 1.3. Share of GDP and Employment by Firm Size a. Share of GDP b. Share of employment Source: World Bank 2020a. Note: For the Kyrgyz Republic, the “small” category includes small enterprises and individual entrepreneurs (those licensed under the “patent” regime). EU = European Union; GDP = gross domestic product. 1.10 There are also gender differences in access to economic opportunities. The Kyrgyz Republic ranks 91st out of 189 countries in the 2017 Gender Inequality Index. The 2012 Gender Diagnostic conducted by the World Bank identified gender differences in access to economic opportunities and in human capital indicators. The FY19–22 CPF notes that “the main gender-related challenges are threefold: reversing the still elevated maternal mortality rate, closing the gap between male and female labor force participation, and reducing the disparity in wages” (World Bank 2018b, 8). 9 Inadequate Provision of Public Services at the Local Level 1.11 Despite a reduction in overall poverty, living standards for much of the population remain low. Access to and quality of basic services such as water, sewerage, heating, paved roads, and electricity is limited. Piped water supply outside of the capital is intermittent and commonly disrupted by shortages or by noncompliant or unsafe water. Access to district heating is limited to approximately 17 percent of the 1.1 million Kyrgyz households. The remaining 907,000 households resort to individual solutions to meet their heating needs during winter (World Bank 2017). In 2020, the vast majority of citizens lacked access to sewerage systems, and their number is growing—from 91 percent in 2018 to 93 percent in 2020 (Development Policy Institute 2020). 10 1.12 Basic public services are the responsibility of local governments, many of which lack capacity to deliver on the mandates assigned them. The Kyrgyz Republic inherited a reasonably developed, if basic, infrastructure and social service system that has rapidly deteriorated since independence. The disappearance of Soviet structures and systems left significant voids in infrastructure maintenance, drinking water provision, sanitation services, health care, childcare, and social facilities (World Bank 2018e). In 2008, 6 responsibility for the delivery of most basic public services was turned over to local self- governments, many of which have been unable to carry out their new responsibilities because of the lack of funding, technical means, and institutional capacity. The country’s mountainous terrain exacerbates the challenge of delivering local infrastructure and basic public services. 1.13 Failure to provide basic services contributes to frustration and lack of confidence in government, which has been a trigger for conflict. Public frustration over the deterioration in social services and infrastructure (and corruption and rising inflation) contributed to the political and social upheaval in 2005 and 2010. In a 2019 survey asking voters for the most important issue facing their village or town, substandard roads and streets, lack of clean drinking water, and problems with kindergarten, electricity, and garbage were among the highest-ranked responses, along with unemployment (Center for Insights in Survey Research 2019). The low quality of local infrastructure and public services also constrains private sector development. 1.14 There are large disparities in service provision among rural and urban areas and across regions. Mountain oblasts generally have fewer services than other regions, and, within each region, rural areas are less likely to have services than urban areas. Forty- two percent of the population of the capital has access to piped sewerage, compared with only 3 percent of the predominantly rural population of Batken region. In small towns, 25–50 percent of the population have regular solid waste service, compared with about 96 percent of the population in Bishkek and 60 percent in Osh. Over the past decade, the gap between the poorest and richest regions has widened. In 2010, the richest region had a gross regional product per capita four times higher than the poorest region; by 2015, this had risen to a factor of six (World Bank 2020c). 11 Increasing regional disparities poses a threat to social cohesion and contribute to political instability. Impact of COVID-19 1.15 The challenges facing the Kyrgyz Republic have been compounded by the COVID-19 pandemic. Real GDP declined by 8.6 percent in 2020, and 40,000 jobs were lost (World Bank 2021b). The budget deficit increased to 4.2 percent of GDP (from 0.5 percent in 2019). The poverty rate increased to 25.3 percent in 2020, from 20.1 percent in 2019. In an April 2020 survey of businesses, 80 percent of respondents reported a decrease of more than 75 percent in revenues, and almost half of respondents had put their staff on leave without pay. Nearly two-thirds of respondents did not have enough reserves to cover operational expenses for more than one month (World Bank 2020a). The COVID-19 crisis also put a strain on local health services that were already suffering from substandard conditions, staff shortages, and a weak arsenal of diagnostics (World Bank 2021b). 7 2. World Bank Group–Supported Program 2.1 After the 2010 revolution, the Bank Group’s work was guided by the 2010 Joint Economic Assessment and the FY12–13 Interim Strategy Note. 12 Immediate reconstruction and humanitarian and budget needs were supported by the Bank Group and other development partners based on the Joint Economic Assessment (Asian Development Bank, International Monetary Fund, and World Bank 2010). The Bank Group’s FY12–13 Interim Strategy Note recognized that corruption, nepotism, and the misuse of public assets had been fundamental causes of the 2010 crisis and focused on three themes: governance; economic adjustment; and social stabilization, especially in the south of the country (World Bank 2011a). 2.2 The FY14–17 Country Partnership Strategy (CPS) set improved governance as its strategic goal and focused on three dimensions of the relationship between the state and its citizens. The state was characterized as “provider and consumer of essential public services, as enabler and developer of private business, and as steward and user of natural resources and physical infrastructure and key public goods” (World Bank 2013b, 20). The CPS was informed by studies that indicated that inadequate public administration, lack of transparency, and weak public services were binding constraints on development, and that future stability and achievement of the twin goals (of reducing poverty and promoting shared prosperity) were highly dependent on the population seeing tangible improvements in governance. 13 The CPS also underscored that the Kyrgyz Republic’s economy was vulnerable to volatility—with its reliance on gold and remittances—and that “a strong private sector development agenda, under the broad theme of governance, could unleash the potential of the economy” (World Bank 2013b, 7). 2.3 Thus, the three pillars of the FY14–17 CPS were public administration and public service delivery, business environment and investment climate, and management of natural resources and physical infrastructure. The CPS described the selection of these as the most important areas in the following ways: • Public administration and public service delivery: “more efficient public administration and public services—especially those relating to lowering poverty, improving accountability, enhancing human capital, and reducing ethnic, gender, and social disparities—are preconditions for political stability, effective public expenditure, human capital formation, and thus for long-term growth and poverty reduction” (World Bank 2013b, 2). • Business environment and investment climate: “an improved business environment and investment climate, including a more transparent, rules-based relationship between the state and the private sector, are prerequisites for 8 promoting competitiveness, diversification, and productivity as well as job creation” (World Bank 2013b, 2). • Management of natural resources and physical infrastructure: “better management of natural resources and physical infrastructure as essential public goods, while at the same time taking into account climate change, environmental sustainability, and regional considerations, is crucial for improved poverty outcomes, in particular for the almost two-thirds of the Kyrgyz population that live in isolated rural and mountainous areas” (World Bank 2013b, 2). 2.4 The 2018 Systematic Country Diagnostic (SCD) argued that the most pressing need in the Kyrgyz Republic is to adjust its economic development model to one with more diverse sources of activity and growth driven by the private sector. The SCD describes the challenge as follows: “Growth has been driven by an ad hoc, opportunistic adaptation to constraints.… This ‘growth model’ led to significant welfare gains, as it rode on the back of the commodity super-cycle.… But its lack of coherence in addressing key constraints has implied that vulnerabilities remain widespread” (World Bank 2018a, 4); therefore, a new development model was necessary. The SCD also confirms that governance remains an important constraint, and that pursuing a new development model will “require a fundamental departure from haphazard policy making, whereby macrofiscal policies have failed to support greater poverty reduction and resilience, and weak governance has undermined the effectiveness of reforms” (World Bank 2018a, 5). To successfully pursue this new development model, the SCD sets forth seven core constraints to development and growth that Bank Group should focus on in the short- term: 14 macrofiscal stress; business environment weaknesses; weak external competitiveness; obstacles to mining and hydropower development; low agricultural productivity; lack of targeting in social protection programs; and policy uncertainty and weak rule of law, which impede progress in addressing all other constraints. 15 2.5 The FY19–22 CPF aims to help the Kyrgyz Republic promote “diversified, export-oriented, inclusive, and sustainable growth” (World Bank 2018b, 13). The focus draws on the SCD, government program (Vision 2040 and medium-term implementation plan for 2018–22), and Bank Group comparative advantages. The focus areas of the CPF are the following: • Focus area 1: Strengthen foundations for inclusive, private sector–led growth. This comprises the following objectives: strengthen institutions for improved macroeconomic management; enhance conditions for private investment and diversification; and enhance financial deepening and inclusion. 9 • Focus area 2: Raise productivity and build connectivity. This comprises the following objectives: enhance growth of natural resource sectors, especially hydropower; promote digitization and development of e-economy; and build transport connectivity. • Focus area 3: Enhance economic opportunities and resilience. This comprises the following objectives: develop human capital, support regional development, and enhance resilience to climate change and disaster risks. 2.6 Figure 2.1 shows how the FY14–17 CPS, priorities identified in the SCD, and the FY19–22 CPF are related. The FY19–22 CPF continues many of the objectives of the FY14–17 CPS, but approaches them through the lens of economic growth. In line with the findings of the SCD, governance remains a defining challenge for the Kyrgyz Republic, but the CPF focuses on a narrower set of priority reforms. The Bank Group also drew on the government’s Sustainable Development Strategy 2040 and Medium- Term Action Plan 2018–22 as a selectivity filter for CPF priorities. 16 2.7 Governance is approached as key to enabling more dynamic economic growth and delivering better services to citizens. The SCD constraints of policy uncertainty and weak rule of law and low accountability and capacity of public officials are subsumed within the CPF objective of strengthening institutions for improved macroeconomic management. The focus on judicial reform was dropped in the FY18–22 CPF, citing commitment issues and delays in preparation of the judicial reform project in the FY14– 17 CPS. Focus on macroeconomic management became more explicit, given a deterioration in the country’s fiscal stance in the years leading up to the CPF. 17 2.8 Regional development became a stand-alone objective, informed by the government’s Sustainable Development Strategy 2040 and regional development plan for 2018–22. World Bank analysis pointed to unbalanced spatial patterns of growth and “persistent spatial disparities in living standards and opportunities … often pose a threat to social and political cohesion” (World Bank 2018b, 26). Also noted were urban overpopulation, youth unemployment and marginalization, and the growing specter of religious radicalization. Although the FY14–17 CPS sought to expand access, improve the quality of essential public services, and increase the efficiency and competitiveness of agriculture, the FY19–22 CPF placed these within a strategic goal of advancing regional development in two areas identified by Bank Group staff as “poor but [of] high potential” (World Bank 2018b, 26). 18 10 Figure 2.1. Mapping of FY14–17 CPS, Priorities in the SCD, and FY19–22 CPF Source: Independent Evaluation Group. Note: CPF = Country Partnership Framework; CPS = Country Partnership Strategy; FY = fiscal year; Obj. = objective; SCD = Systematic Country Diagnostic; WSS = water supply and sanitation. 11 2.9 The context for improving the management of natural resource sectors was narrowed from the CPS, with the reference to natural resource sectors either made explicitly economic or directed toward enhancing resilience to climate change and disaster risks. Promoting the development of e-economy was a new objective, related to the constraint of infrastructure bottlenecks identified in the SCD. 3. Purpose, Objectives, and Audience Objective and Audience 3.1 The objective of this CPE is to assess how the Bank Group helped the Kyrgyz Republic address the main challenges that constrained its development. The evaluation will cover FY14–21; 19 it is timed to inform the Kyrgyz Republic’s next CPF, which is currently expected in late 2022. 3.2 The primary audience for the CPE is the Bank Group Board of Executive Directors, its Committee on Development Effectiveness, and Bank Group management and staff working on the Kyrgyz Republic. The evaluation will also be relevant for the government of the Kyrgyz Republic in informing its future use of Bank Group support. The evaluation may be of interest to elements of civil society, which at times have been vocal about Bank Group–supported programs, and to other development partners working in the country. The report will be sent to the government for comments, and once finalized, will be made public (including government comments). 4. Evaluation Scope and Questions Evaluation Scope 4.1 The CPE will have a high-level focus on the relevance of the Bank Group– supported program and its adaptation over time, including an examination of the Bank Group’s COVID-19 response. The CPE will also take an in-depth look at Bank Group efforts (and their evolution over the evaluation period) to support private sector development, governance, and the provision of local public services. These are the major development challenges identified in a literature review and interviews conducted during the scoping stage of this evaluation and were also identified in the SCD and reflected in the focus areas of the FY14–17 CPS and in the Context and Development challenges identified in the FY19–22 CPF. The CPE will encompass all World Bank, International Finance Corporation (IFC), and Multilateral Investment Guarantee Agency work relevant to these areas. 4.2 Taken together, the three issue areas and the COVID-19 response cover a large proportion of the Bank Group lending and trust fund portfolio during the evaluation 12 period. The three focus areas and the COVID-19 response capture 81 percent of International Development Association lending; the International Development Association grant and trust fund portfolio is equivalent to US$1,079 million (see appendix B on the Bank Group portfolio). The issue areas cover all of the IFC investments active during the period (33 investments with a total project size of US$127 million) and all of the IFC advisory services active during the period (35 tasks). The one Multilateral Investment Guarantee Agency guarantee, with a gross exposure of US$5.8 million, active until 2017, is also included. 20 The issue areas also cover 53 of the 73 World Bank advisory services and analytics tasks carried out during FY14–21. Appendix B presents a preliminary portfolio review of the projects and tasks relevant to each of the three issues. Evaluation Questions 4.3 The CPE will address four evaluation questions: • How relevant to the development needs of the Kyrgyz Republic was the Bank Group–supported strategy, and did it evolve appropriately over time, given changes in the country context and lessons from experience? • To what extent did Bank Group assistance help the Kyrgyz Republic increase private sector–led economic growth to reduce the country’s economic vulnerability? • To what extent did Bank Group assistance help improve central government governance and institutional capacity? • To what extent did Bank Group assistance enhance local governance and the provision of basic local public services? 4.4 To answer the evaluation questions, the evaluation will address the dimensions of relevance, effectiveness, learning and adaptability, and the effectiveness of partnerships (with civil society and development partners). 4.5 Within three issue areas, the CPE will assess the extent to which the Bank Group made progress on relevant special themes under the International Development Association, including the following: • Jobs and economic transformation. Issue area 1 on private sector development will address this directly. 13 • Governance and institutions. Issue area 2 on governance is fully aligned with this theme. Issue area 3 on provision of local-level services will consider the extent to which institutions that deliver these services have been strengthened. • Climate change. The impact of climate change in the Kyrgyz Republic is important for the agriculture sector, generation of hydroelectricity, and disaster risk management. Issue area 1 on private sector development and issue area 3 on provision of local services will consider the extent to which Bank Group irrigation operations have strengthened farmers’ resilience to climate change. • Gender. Issue area 1 on private sector development will examine the extent to which the Bank Group was successful in supporting improvements in women’s access to economic opportunities. • Fragility. Issue area 3 on local governance and service delivery will include an assessment of fragility risks as part of the analysis of relevance. 4.6 The CPE will examine the relevance of the Bank Group’s early response to the COVID-19 pandemic in the Kyrgyz Republic as part of its assessment of the evolution of Bank Group support. The World Bank’s COVID-19 response will also figure in the assessment of support for private sector development (support for micro, small, and medium enterprises) and public local services (vaccine support). 5. Evaluation Design Methodology 5.1 The evaluation will consider the full range of Bank Group engagement, including lending and investments, convening, and analytical work. The evaluation will also explore the extent to which adaptation and learning over time took place. The evaluation will identify areas to which the Bank Group contributed and what lessons can be drawn to inform future Bank Group support to the country. 5.2 The CPE will use a range of methods to triangulate evidence to answer the evaluation questions. The evaluation will conduct a virtual mission to Kyrgyz Republic because of COVID-19–related travel restrictions. The methodological approaches outlined below are described in the design matrix in appendix A. • Portfolio review and analysis, including review of project and task-related documents. The evaluation will conduct a portfolio review to identify the support delivered by the Bank Group during FY14–21 that is relevant to the evaluation’s areas of focus. A preliminary review has already been conducted 14 and has informed this Approach Paper. Based on the portfolio identified, the evaluation will carry out a structured document review of lending and nonlending portfolios (including IFC and the Multilateral Investment Guarantee Agency) to extract, code, and analyze relevant data and information. Projects, operations, and tasks will be analyzed to isolate activities that correspond to the evaluation’s three areas of interest. Document review will include Project Appraisal Documents, Implementation Completion and Results Reports and Implementation Completion and Results Report Reviews, Implementation Status and Results Reports, Expanded Project Supervision Reports, Project Completion Reports, aide-mémoire, review meeting minutes, and outputs documenting technical and policy dialogue. • Existing evaluative evidence. The evaluation will build on existing internal and external evaluations. This includes several Project Performance Assessment Reports and relevant case studies on the Kyrgyz Republic conducted for previous Independent Evaluation Group (IEG) evaluations (box 5.1). Box 5.1. Existing Independent Evaluation Group Evaluations of World Bank Group Support to the Kyrgyz Republic Learning Products, Working Papers, Others “Lessons from Land Administration Projects: A Review of Project Performance Assessments” Project Performance Assessment Reports “Kyrgyz Republic—Rural Education Project” “Kyrgyz Republic—Village Investment Project and Second Village Investment Project” “Kyrgyz Republic—Governance and Competitiveness Development Policy Operation” Country Case Studies Engaging Citizens for Better Development Results “Creating Markets” to Leverage the Private Sector for Sustainable Development and Growth Two to Tango: An Evaluation of World Bank Group Support to Fostering Regional Integration Source: Independent Evaluation Group. • Review of World Bank and IFC analytical work. Relevant sources include investment climate assessments and the Country Private Sector Diagnostic; public expenditure reviews, Public Expenditure and Financial Accountability assessments, and Financial Sector Assessment Program assessments; country economic memorandums; regular economic updates; and country opinion surveys. 15 • Review of external analytical work and data. The evaluation will review analytical work relevant to the evaluation questions from other development partners, civil society organizations, academia, and others as relevant. 21 Government strategies will be reviewed to help in understanding the strategic approach of the government. External (to Bank Group) sources of data and information will be used to get a more complete picture of how the country’s performance on the issue areas has evolved. 22 • Structured and semistructured interviews and focus groups. The evaluation team will conduct semistructured virtual interviews with Bank Group staff, government officials, development partners, and relevant academics and members of Kyrgyz civil society. For each set of interviews, a template with questions or topics will be consistently applied as appropriate. Issues covered in the interviews will include the rationale, nature, and extent of Bank Group support; the roles of the different Bank Group institutions; the adequacy of the Bank Group’s assessment of the Kyrgyz Republic’s development challenges; complementarity with other activities; coordination with donors and counterpart agencies with emphasis on the Bank Group’s comparative advantage and value added; views on the efficacy of Bank Group support; how the program adjusted to the country context; and other information required to address the dimensions of the evaluation questions. Focus groups may also be held to cover selected topics. 5.3 The country has a vibrant civil society that the evaluation team will leverage. The view of stakeholders in the field will be key to assessing to what extent the Bank Group program has made a difference in the areas it expected to influence—primarily governance, private sector development, and local public services. The evaluation will consult independent grassroots stakeholders, taking into account the cultural context in interpreting the findings of interviews with domestic stakeholders, and triangulating interviews with other evidence. The CPE team includes two short-term consultants who are knowledgeable about local conditions and counterparts. • Geospatial analysis. Contingent on data availability, the evaluation will explore the potential to conduct geospatial analysis to inform the assessment of Bank Group support for the provision of basic local public services. This analysis would be conducted by geolocating investment project sites. Project geolocations will be overlaid with georeferenced data on outcomes related to basic services. Information would be sought from the relevant national statistical agencies, along with supplementary information available from remotely sensed data and other global information sources. The use of geospatial analysis will be further refined in consultation with the IEG methods team. 16 • Mapping of relevant government institutional processes and how they have changed over the evaluation period. This method can be used to compare how certain government processes have become more efficient or less discretionary. This method may be used to assess reforms in private sector development related to interactions between firms and government; in governance, for instance, reforms related to public financial management or procurement; and in processes governing local public services. • Media analysis. The Kyrgyz Republic has an active and vibrant media, including news outlets and social media. Mixed-media analysis may be useful to understand the private sector’s view of the business environment and perceptions of the quality of governance in the country. The evaluation team will explore this further with the IEG methods team. Limitations of the Methodology 5.4 The evaluation design has two main limitations, starting with the inability to conduct field missions because of the COVID-19 pandemic. Although the evaluation team plans to use digital technologies to interview staff, donors, and other stakeholders, it is possible that not all interlocutors will be reachable, particularly outside of major urban centers, which will potentially inject a degree of selection bias into the qualitative findings. An additional challenge is the high turnover of government counterparts over the evaluation time frame. To address these limitations, the evaluation team plans to make use of local consultants where feasible. The evaluation team includes local consultants with extensive institutional memory of many of the World Bank’s counterparts, and with strong knowledge of the civil society sector and other relevant actors. 6. Quality Assurance Process 6.1 Quality assurance will take place through peer review arrangements, close collaboration with IEG management, and periodic consultations with the Bank Group staff who have worked on or are currently working on the country. This Approach Paper and the final report will be peer-reviewed by former International Monetary Fund mission chiefs for the Kyrgyz Republic (Eddy Gemayel for the Approach Paper and Christian Josz for the final report); Mahabat Baimyrzaeva, associate professor, Middlebury Institute of International Studies at Monterey; and Askar Sydykov, executive director of the International Business Council, a private sector association in the Kyrgyz Republic. The work will be conducted under the guidance and quality assurance of Jeffrey Allen Chelsky (manager), Oscar Calvo-Gonzalez (director), and the overall direction of Alison Evans (Director-General, Evaluation). 17 7. Expected Outputs and Outreach 7.1 The main output of this evaluation will be a report that presents relevant findings and lessons that can inform the next CPF for the Kyrgyz Republic. The CPE is planned for e-submission in the fourth quarter of FY22. This should provide sufficient time for it to inform the preparation of the next CPF, currently planned for no earlier than the third quarter of FY23. 7.2 The CPE team will plan a virtual dissemination event, to include, as appropriate, the Country Management Unit, government representatives, other relevant civil society organizations, and others, for early FY23. 8. Resources 8.1 The evaluation team will be led by April Connelly (senior natural resources specialist) and Melissa Metz (senior economist). Team members include Patricia Acevedo (program assistant), Konstantin Atanesyan (senior evaluation officer), Asita Da Silva (consultant—financial sector), Corky De Asis (program assistant—portfolio), Lev Freinkman (consultant–governance), Alisher Khamidov (local consultant), Sengphet (Anna) Lattanavong (extended-term consultant), Lilia Saetova (local consultant), and others in the process of being confirmed. 8.2 The estimated budget for delivering this evaluation is US$450,000 including US$10,000 for dissemination in FY23. 1Bishkek and its surrounding cities in the Chui valley and Osh city, Jalal-Abad city, and the surrounding areas on the Ferghana valley. 2 See also the 2019 Enterprise Survey, https://microdata.worldbank.org/index.php/catalog/3566. 3 See also the World Development Indicators, https://data.worldbank.org/country/kyrgyz- republic and the National Statistical Committee of the Kyrgyz Republic, table 5.04.00.25, “Poverty rate,” http://stat.kg/en/bazy-dannyh/. 4Eliminating poverty and promoting shared prosperity (improving living standards of the bottom 40 percent of a country’s population by income distribution). 5 See the World Development Indicators, https://data.worldbank.org/country/kyrgyz-republic. 6 See the World Development Indicators, https://data.worldbank.org/country/kyrgyz-republic. 7As of 2018, there were 305 large enterprises, 769 medium enterprises, 14,520 small enterprises, and 401,658 individual entrepreneurs. Data on large enterprises is from International Finance Corporation (2021). Other data are from the National Statistical Committee of the Kyrgyz 18 Republic, table 1.1, “Number, number of employees and the amount of the gross added value of small and medium-sized enterprises,” http://stat.kg/en/statistics/maloe-i-srednee- predprinimatelstvo/, accessed June 29, 2021. The latest data available are from 2018. 8 See also the National Statistical Committee of the Kyrgyz Republic, table 1.1, “Number, number of employees and the amount of the gross added value of small and medium-sized enterprises.” http://stat.kg/en/statistics/maloe-i-srednee-predprinimatelstvo/, accessed June 29, 2021. 9The maternal mortality ratio is extremely high, at 76 deaths per 100,000 live births. Women’s economic participation remains low at 48.2 percent compared with 75.7 percent for men. Women’s political empowerment is also low, with only 16 percent of seats in Parliament in 2018. 10 The number of households is growing, but the sewerage capacity is not keeping pace. 11As a reference, in the European Union leading regions have, on average, 2.3 times the gross domestic product (GDP) per capita of the poorest region in their country (Farole et al. 2017 cited in World Bank 2020c). 12The Joint Economic Assessment was prepared jointly by the World Bank, Asian Development Bank, and International Monetary Fund, with the participation of the Eurasian Development Bank, the European Bank for Reconstruction and Development, the European Commission, the International Finance Corporation, and the United Nations. 13These studies were Asian Development Bank, International Monetary Fund, and World Bank (2010) and World Bank (2011b, 2013a, 2014a, 2014b). Projects and consultations: Transparency and Accountability Project; stakeholder consultations at the national and local levels during March–April 2013. 14The Systematic Country Diagnostic uses as prioritization criteria magnitude of expected impacts on twin goals; actual potential for (medium-term) reform; adequacy of balance between short-term and longer-term impacts; extent to which essential preconditions for a productive life are addressed (such as ensuring a basic minimum standard of living for all); existence of links and complementary effects; and soundness of the evidence base, for identifying problems and suggesting solutions. 15Other constraints that the Systematic Country Diagnostic identifies but does not prioritize as core for focus in the short-term are low accountability or capacity of public officials; infrastructure bottlenecks, particularly for connectivity; quality deficit in social services provision; inadequate management of natural resources; and exposure to climate and disaster risks. 16Government priorities in the Sustainable Development Strategy were (i) closing infrastructure gaps; (ii) creating an attractive business climate; (iii) public-private partnerships and improved management of public sector assets; (iv) financial sector strengthening; (v) human capital; and (vi) regional development. 17After picking up in 2013, nongold growth decelerated markedly in 2014–15. This was driven by economic slowdown and currency devaluations in the region (particularly the Russian Federation), falling gold prices and remittances, and uncertainties and delays surrounding 19 accession to the Eurasian Economic Union. The som lost almost half of its value against the US dollar between June 2013 and October 2015; this also contributed to inflation. Fiscal buffers were reduced by counter cyclical fiscal policies and a public investment program that was largely financed by donors (IMF 2016).The fiscal deficit deteriorated to 5.8 percent of GDP by 2016. However, the macroeconomic situation has improved more recently. The economy experienced steadier growth, averaging 4 percent per year, inflation was lower, fiscal deficits were lower, and the exchange rate was stable (World Bank 2021a). The fiscal balance improved, to −0.6 percent of GDP in 2018 and −0.1 percent of GDP in 2019. However, fiscal risk and debt vulnerability have increased, and the banking sector has become more vulnerable in the face of high dollarization and a weak institutional and regulatory framework (World Bank 2018a). 18 Issyk-Kul in the north and Osh, Jalalabad, and Batken in the south. 19The evaluation period begins in 2013, after the completion of work to rebuild after the 2010 revolution and as the country looked to the future. 20All International Finance Corporation and Multilateral Investment Guarantee Agency activities are relevant to the private sector development issue area. 21This will include International Monetary Fund, Asian Development Bank, and European Bank for Reconstruction and Development reports, country analysis, and diagnostics (for example, Public Investment Management Assessment and Article IV Consultation staff reports and program documents); Ministry of Finance and central bank reports; white papers and other documents on perceptions and constraints from private sector associations; research reports from development partners, nongovernmental organizations, and academics on the state of public service delivery at the local level; review of local media sources; reputable opinion surveys (for example, relevant surveys from the International Republican Institute, https://www.iri.org/country/kyrgyzstan). 22Relevant data sources include the Enterprise Survey (https://www.enterprisesurveys.org/), the National Statistical Committee of the Kyrgyz Republic database (includes data on small and medium enterprises, economic sectors, local living standards, and many more), scores on relevant indicators of the Global Competitiveness Report, Economist Intelligence Unit data, and data from Transparency International and Open Budget. 20 Bibliography Asian Development Bank, International Monetary Fund, and World Bank. 2010. The Kyrgyz Republic Joint Economic Assessment: Reconciliation, Recovery and Reconstruction. Asian Development Bank, International Monetary Fund, and World Bank. Center for Insights in Survey Research. 2019. “Public Opinion Poll Residents of Kyrgyzstan: November 21–December 3, 2019.” International Republican Institute, Washington, DC. https://www.iri.org/sites/default/files/final_kygyzstan_slides.pdf. Development Policy Institute. 2020. How Can Local Self-Government of the Kyrgyz Republic Inspire Citizens? Bishkek, Kyrgyz Republic: Development Policy Institute. High Representative of the Union for Foreign Affairs and Security Policy. 2020. “The EU Special Incentive Arrangement for Sustainable Development and Good Governance (‘GSP+’) Assessment of the Kyrgyz Republic Covering the Period 2018–2019.” Joint Staff Working Document, European Commission, Brussels. Joint Report to the European Parliament and the Council, October 2020. Holzhacker, Hans, and Dana Skakova. 2019. Kyrgyz Republic Diagnostic. London: European Bank for Reconstruction and Development. IFC (International Finance Corporation). 2021. Creating Markets in the Kyrgyz Republic: Unleashing the Private Sector to Rebuild Development Success. Country Private Sector Diagnostic. Washington, DC: International Finance Corporation. IMF (International Monetary Fund). 2007. “Kyrgyz Republic: Poverty Reduction Strategy Paper— Country Development Strategy (2007–2010).” Country Report 07/193, IMF, Washington, DC. IMF (International Monetary Fund). 2016. “Kyrgyz Republic: Staff Report for the 2015 Article IV Consultation.” Country Report 16/55, IMF, Washington, DC. IMF (International Monetary Fund). 2019. “Kyrgyz Republic: Staff Report for the 2019 Article IV Consultation.” Country Report 19/208, IMF, Washington, DC. OECD (Organisation for Economic Co-operation and Development). 2018. Anti-Corruption Reforms in Kyrgyzstan: Fourth Round of Monitoring of the Istanbul Anti-Corruption Action Plan. Paris: OECD. Reuters. 2021. “Kyrgyzstan in Full Control of Kumtor Gold Mine as Centerra Takes Legal Action.” June 1, 2021. Reuters Business. https://www.reuters.com/business/kyrgyzstan- full-control-kumtor-gold-mine-centerra-takes-legal-action-2021-06-01/. Schwab, Klaus. 2019. The Global Competitiveness Report 2019. Geneva: World Economic Forum. 21 UNDP (United Nations Development Programme). 2013. Human Development as the Focus of Local Self-Governance: Here and Now! National Human Development Report of the Kyrgyz Republic 2012. Bishkek, Kyrgyz Republic: UNDP. UNCTAD Investment Dispute Settlement Navigator. USAID (US Agency for International Development). 2015. “Kyrgyz Republic: Country Development Cooperation Strategy, FY2015–FY2020.” USAID, Washington, DC. World Bank. 2011a. Kyrgyz Republic—Interim Strategy Note, FY12–13. Washington, DC: World Bank. World Bank. 2011b. “Kyrgyz Republic—Judicial System Diagnostic: Measuring Progress and Identifying Needs.” Poverty Reduction and Economic Management Unit, Europe and Central Asia Region, Report 61906-KG, World Bank, Washington, DC. World Bank. 2013a. Kyrgyz Republic—Country Economic Memorandum. Washington, DC: World Bank. World Bank. 2013b. Kyrgyz Republic—Country Partnership Strategy, FY14–17. Washington, DC: World Bank. World Bank. 2013c. “Kyrgyz Republic—The World Bank Country Survey FY 2013: Report of Findings.” World Bank, Washington, DC. World Bank. 2014a. Kyrgyz Republic Conflict Filter: Enhancing Sensitivity to Conflict Risks in World Bank–Funded Activities—Lessons from the Kyrgyz Republic. Washington, DC: World Bank. World Bank. 2014b. “Kyrgyz Republic Public Expenditure Review.” Policy Notes, World Bank, Washington, DC. World Bank. 2015. “Kyrgyz Republic—Third Village Investment Project.” Project Appraisal Document 1124, World Bank, Washington, DC. World Bank. 2017. “Kyrgyz Republic—Heat Supply Improvement Project.” Project Appraisal Document 2164, World Bank, Washington, DC. World Bank. 2018a. Kyrgyz Republic. Systematic Country Diagnostic. Washington, DC: World Bank. World Bank. 2018b. Kyrgyz Republic—Country Partnership Framework, FY19–22. Washington, DC: World Bank. World Bank. 2018c. “Kyrgyz Republic—Livelihoods for Youth.” Project Appraisal Document 2784, World Bank, Washington, DC. 22 World Bank. 2018d. “Kyrgyz Republic—Review of Country Partnership Strategy for FY14–17.” Country Partnership Strategy Completion and Learning Report Review 78500, Independent Evaluation Group, World Bank, Washington, DC. World Bank. 2018e. “Kyrgyz Republic—Village Investment Project and Second Village Investment Project.” Project Performance Assessment Report 123089, Independent Evaluation Group, World Bank, Washington, DC. World Bank. 2020a. Kyrgyz Republic—Country Economic Memorandum. Washington, DC: World Bank. World Bank. 2020b. “Kyrgyz Republic—Proposed Additional Finance for the Village Investment Project.” Project Paper 4053, World Bank, Washington, DC. World Bank. 2020c. “Kyrgyz Republic—Regional Economic Development.” Project Appraisal Document 3480, World Bank, Washington, DC. World Bank. 2021a. “Kyrgyz Republic.” Macro-Poverty Outlook. April. World Bank, Washington, DC. https://pubdocs.worldbank.org/en/189651492011105317/mpo-kgz.pdf. World Bank. 2021b. “One Year Later in the Kyrgyz Republic’s Battle against COVID-19.” News, March 17, 2021. https://www.worldbank.org/en/news/feature/2021/03/17/one-year-later- in-the-kyrgyz-republic-s-battle-against-covid-19. 23 Appendix A. Evaluation Design Matrix Table A.1. Evaluation Design Dimensions Approach and Methods Data Sources How relevant was the World Bank Group’s overall strategy, and did it evolve appropriately over time given changes in the country context and lessons from experience? Relevance Document review Relevant analytical or diagnostic work on PSD in How relevant Bank Group strategies (FY14–17 CPS and Semistructured interviews with Bank Group team, Kyrgyz Republic from Bank Group and other FY19–22 CPF) were to the country’s priority constraints government representatives, and other development partners (for example, IMF, ADB, to development; whether certain country priorities development partners and key informants (for EBRD); Bank Group country engagement documents: were excluded from the strategies, and the extent to example, private sector associations) CPS, SCD, CPF, CLR, CLRR which the reasons are justified (that is, lack of Reports from other development partners (for comparative advantage or division of labor with other example, the 2010 Joint Economic Assessment from development partners); whether the design of the the IMF, ADB, and EBRD) strategies consistent with the country context; the Government development strategies and other strengths of the Bank Group’s approach and any relevant documents missed opportunities Adaptability How well the Bank Group strategies adapted to major developments at the country level and to lessons learned in strategy implementation. 25 How well has the Bank Group helped the Kyrgyz Republic increase private sector–led economic growth and reduce the country’s economic vulnerability? Relevance Identify the country’s main development Relevant analytical or diagnostic work on PSD in How relevant the Bank Group–supported program was challenges related to PSD and identify which Kyrgyz Republic from Bank Group and other to the country’s main PSD challenges; whether certain challenges the Bank Group program attempted to development partners (for example, IMF, ADB, and relevant priorities were excluded from the program, address and the reform areas Bank Group worked EBRD) and the rationale for doing so (that is, lack of on Reports from private sector associations, the comparative advantage or division of labor with other Enterprise Survey, international indexes, and development partners); whether the Bank Group Document analysis reputable opinion surveys program design was consistent with the country Semistructured interviews, focus groups, or both context, the magnitude of the challenges, and Government strategies PRA of Bank Group interventions supporting PSD evidence of what works; the strengths of the Bank (lending, analytics, policy dialogue) and mapping Bank Group strategies, project or ASA documents Group’s approach and any missed opportunities against country needs and priorities (including IFC and Multilateral Investment Guarantee Agency), and key project or ASA outputs as relevant Interviews with CSOs or key informants (for example, private sector associations), other development partners, government representatives, and Bank Group staff 26 Effectiveness Identify how the country’s performance on the IMF, ADB, and EBRD reports, country analysis, and The development outcomes the Bank Group program most relevant aspects of PSD has evolved. In the diagnostics; white papers and other documents on contributed to and how; the factors behind successes reform areas that Bank Group worked on, perceptions and constraints from private sector or lack thereof; effectiveness of Bank Group understand the main development outcomes and associations; research reports from development collaboration in achieving development outcomes; the how they were achieved and the Bank Group partners, NGOs, and academics; reputable opinion extent to which Bank Group–supported improvements contribution to this achievement. Identify how they surveys (for example, relevant surveys from IRI— increased women’s access to economic opportunities link to the evolution of the issues at the country https://www.iri.org/country/kyrgyzstan); the level. Review gender-disaggregated data. Identify Enterprise Survey which Bank Group interventions targeted gender, (https://www.enterprisesurveys.org/); the National how, and what the results were. Identify how and Statistical Committee of the Kyrgyz Republic why results were or were not achieved. The database (includes data on SMEs, economic sectors, approach will examine the relevant reform areas, scores on relevant indicators of the Global considering packages of interventions, going Competitiveness Report, Economist Intelligence Unit beyond project-level results. data, and other available data) Bank Group project completion reports, aide- Document and data analysis mémoire, and notes from policy dialogue, and so on, Semistructured interviews and focus groups as and evaluative evidence (IEG validations, PPARs, appropriate CLRRs, PCRs, XPSRs, EvNotes). PRA of Bank Group interventions supporting PSD Interviews with CSOs or key informants (for example, (lending, analytics, policy dialogue) private sector associations), other development partners, government counterparts, Bank Group staff Mapping of institutional processes and how they have changed Possibly mixed-media analysis to help in understanding the private sector’s view of the business environment 27 Learning and Adaptability Identify whether and how the Bank Group’s Review of Bank Group program and project How well the Bank Group program learned and program in the area changed over time to adapt to documents and documents analyzing changes in adapted to changing circumstances and experience country circumstances (for example, political country context instability and changing counterparts, capacity). Interviews with Bank Group staff, government This will examine (i) how Bank Group programs counterparts, and others as necessary to identify adjusted to changing circumstances; (ii) whether factors that drove how well the programs adjusted the design and implementation of later projects and incorporated lessons, and explore “why” and incorporated lessons learned from earlier projects, “how” particularly in areas where reform was slow or not forthcoming. Identify particular reform areas in which progress was made and where progress was not made and examine them in depth to identify drivers and lessons for the future Partnerships Identify the partnerships that existed relevant to Bank Group project and strategy documents, and How successful the Bank Group was in engaging with the reform areas in which Bank Group worked and strategy and project documents from development and benefiting from partnerships with CSOs and the outcomes of those partnerships. Examine partners development partners factors contributing to success or lack thereof. Interviews with Bank Group staff, other development Review of Bank Group project documents to partners, government counterparts, CSOs identify explicit partnerships. Desk review of strategic document and thematic analysis from other key development partners. Comparison of donors’ work against Bank Group work to understand implicit and explicit division of labor. Interviews with key officials from the Bank Group, other donor organizations, and government officials to understand their views with respect to the division of labor and coordination To what extent did Bank Group assistance help improve central government governance and institutional capacity? 28 Relevance Identify the country’s main development Bank Group analytical work (for example, PEFA, PER) How relevant the Bank Group–supported program was challenges related to governance and identify and other development partners (for example, IMF to the country’s main central government governance which challenges the Bank Group program [Article IV consultations, PIMA, others], ADB, EBRD, and institutional capacity challenges; whether certain attempted to address and the reform areas Bank UN, DFID, SECO), government strategies, white relevant priorities were excluded from the program, Group worked on papers and other documents from CSOs, and the rationale for doing so (that is, lack of Desk review of World Bank and external strategy international indexes, reputable opinion surveys, and comparative advantage or division of labor with other documents and analytical work others development partners); whether the Bank Group Interviews with CSOs, other development partners, program design was consistent with the country government representatives, and Bank Group staff context, the magnitude of the challenges, and evidence of what works; the strengths of the Bank Group’s approach and any missed opportunities 29 Effectiveness Identify how the country’s performance on the Bank Group project documents (completion reports, The development outcomes the Bank Group program most relevant aspects of governance has evolved. major outputs, aide-mémoire, and notes from policy contributed to and how; the factors behind successes In the reform areas that Bank Group worked on, dialogue, and so on) and evaluative evidence (IEG or lack thereof; the effectiveness of Bank Group understand the main development outcomes, validations, PPARs, CLRR) collaboration in achieving development outcomes how they were achieved, and the Bank Group Data and information sources will vary by reform contribution to this achievement. Identify how area. They may include the following: they link to the evolution of the issues at the • Macroeconomic management capacity (fiscal, country level. The approach will examine the debt, and so on): Bank Group documents relevant reform areas, considering packages of (SCD, CPF, CEM, DPL PDs, other ASA and interventions, going beyond project-level results project documents); IMF reports; ADB Desk review of relevant project documents reports; government (MOF, CB) reports Analysis of relevant national statistics (including debt management); other external sources—Economist Intelligence, and so on; Interviews with government officials, CSOs, interviews with government officials and development partners, and others donor counterparts, academics, and local Mapping of institutional processes and how they experts; local media review have changed • Statistical capacity: Bank Group documents Possibly mixed-media analysis to understand (project completion reports, ASA perceptions of the quality of governance presentations, and so on); IMF reports; macroeconomic management capacity (fiscal, government (MOF) reports debt, and so on): ASA and technical assistance • PFM (incl. PIM, tax administration, budget, effectiveness surveys (simple questionnaire). procurement): Bank Group documents (PEFA, Possible comparison of outputs, before and after PER, and so on); IMF reports (PIMA); other World Bank intervention(s)—that is, compare donors—ADB, EU, UN, DFID, SECO, and so quality of reporting and statistical analysis; other on; government (MOF) reports external sources; interviews with government officials, donor counterparts; academia and local • Procurement capacity (separately or experts; possible online survey of beneficiaries absorbed into the PFM section, TBD): Bank (civil servants) and users; dynamics on the use of Group documents (including CPAR if official statistics, accessibility, and quality available); other external sources, including Transparency International, Open Budget, Statistical capacity: Comparative analysis; and so on; interviews with government interviews with Kyrgyz Republic government officials and donor counterparts, academics, officials, donor counterparts, academics, and local and local experts experts 30 Procurement capacity (separately or absorbed • Anticorruption: Bank Group documents into the PFM section; TBD): Possible online survey (including country opinion survey dynamics of beneficiaries (civil servants) and users (vendors); on corruption), Worldwide Governance dynamics of public procurement (resources saved, Indicators; external sources on corruption accessibility of the system, participants) perception, including Transparency Anticorruption: Roundtable with NGOs and International, Open Budget, EU, UN, and so partners (if possible to do remotely). Possible on; interviews with government officials and online survey of population; media review; donor counterparts; academics, and local government reports—prosecution of corruption experts (NGOs) cases, and so on; assessment(s) of institutional capacity, mapping of anticorruption institutions, or both (for example, who does what, results, relevance) Learning and Adaptability Identify whether and how the Bank Group’s Review of Bank Group program and project How well the Bank Group program learned and program in the area changed over time to adapt to documents; review of documents analyzing changes adapted to changing circumstances and experience country circumstances (for example, political in country context instability and changing counterparts, capacity). Interviews with Bank Group staff, government This will examine (i) how Bank Group programs counterparts, and others as necessary to identify adjusted to changing circumstances; (ii) whether factors that drove how well the programs adjusted the design and implementation of later projects and incorporated lessons, and explore why and how incorporated lessons learned from earlier projects, particularly in areas where reform was slow or not forthcoming. Identify particular reform areas in which progress was or was not made and examine them in depth to identify drivers and lessons for the future 31 Partnerships Identify the partnerships that existed relevant to Bank Group project and strategy documents; How successful the Bank Group was in engaging with the reform areas in which the Bank Group worked strategy and project documents from development and benefiting from partnerships with CSOs and and the outcomes of those partnerships. Examine partners development partners factors contributing to success or lack thereof. Interviews with Bank Group staff, other development Review of Bank Group project documents to partners, government counterparts, CSOs identify explicit partnerships. Desk review of strategic document and thematic analysis from other key development partners. Comparison of donors’ work against Bank Group work to understand implicit and explicit division of labor. Interviews with key officials from the Bank Group, other donor organizations, and government officials to understand their views with respect to the division of labor and coordination To what extent did Bank Group assistance enhance local governance and the provision of local public services? Relevance Identify the country’s main development Analytical work from Bank Group and other How relevant the Bank Group program was to the challenges related to local governance and the development partners (for example, ADB, EBRD, country’s main challenges in local governance and provision of basic services and identify which USAID, UNDP), government strategies, documents provision of local services and to country context; the challenges the Bank Group program attempted to from CSOs, international indexes, reputable opinion extent to which it prioritized the Kyrgyz Republic’s address and the reform areas Bank Group worked surveys on local governance and satisfaction with main development challenges; whether certain on, drawing on the following: public services country priorities were excluded from the program, • Desk review of government plans and and the rationale for doing so (that is, lack of strategies, Bank Group strategies, and comparative advantage or division of labor with other project and ASA documents (including development partners); whether the Bank Group self-evaluation and independent program design was consistent with the country validation and evaluation documents) context, the magnitude of the challenges (including • Review of key documents from other the extent to which it took into account the drivers of donors fragility), and evidence of what works; the strengths of • PRA of Bank Group interventions Bank Group’s approach and any missed opportunities • Interviews with CSOs, other development partners, government representatives, and Bank Group staff 32 Effectiveness Desk review of Bank Group project and ASA Bank Group project documents, including IEG project The development outcomes the Bank Group program documents (including self-evaluation and validations, PPARs, and relevant thematic evaluations contributed to and how; the factors behind success or independent validation and evaluation documents) (for example, Citizen Engagement) failure Possible mixed-media analysis to explore Reputable opinion surveys on local governance and perceptions on the responsiveness of local satisfaction with public services governance to citizens’ demands Bank Group staff and external stakeholders and Semistructured interviews with World Bank staff, experts (including government, IMF, donors, private government officials, and CSOs sector, CSOs, academia) Focus groups with beneficiaries, local government Bank Group database of georeferenced project units, and CSOs; if feasible, geospatial analysis to locations help in understanding the regional distribution of Subnational Human Development Index, Gender Bank Group support and the extent of success of Inequality Index, Multidimensional Poverty Index, such support (contingent on adequacy of available Local Self-Governance Capacity Index data) Learning and Adaptability Identify whether and how the Bank Group’s Bank Group country strategies and project How well the Bank Group program learned from and program in the area changed over time to adapt to documents adapt to changing circumstances. How did it balance country circumstances (for example, political Interviews with Bank Group staff, government adapting with maintaining focus on the most strategic instability and changing counterparts, capacity)— counterparts, and relevant development partners issues for the country? through document review and interviews. This will examine (i) how Bank Group programs adjusted to changing circumstances; (ii) whether the design and implementation of later projects incorporated lessons learned from earlier projects, particularly in areas where reform was slow or not forthcoming Partnerships Desk review of strategic document and thematic Bank Group project and strategy documents, How successful the Bank Group was in engaging with analysis from other key development partners. strategy and project documents from development and benefiting from partnerships with CSOs and Comparison of donors’ work against Bank Group partners development partners work to understand implicit and explicit division of Bank Group staff, other development partners, labor. Interviews with key officials from the Bank government officials, CSOs Group, other donor organizations, and government to understand their views with respect to the division of labor and coordination. Identify the outcomes of those partnerships and examine factors contributing to success or failure 33 Source: Independent Evaluation Group. Note: ADB = Asian Development Bank; ASA = advisory services and analytics; CB = central bank; CEM = Country Economic Memorandum; CLR = Completion and Learning Review; CLRR = Completion and Learning Review Report; CPAR = Country Procurement Assessment Review; CPF = Country Partnership Framework; CPS = Country Partnership Strategy; CSO = civil society organization; DFID = Department for International Development (UK); DPL = development policy loan; EBRD = European Bank for Reconstruction and Development; EU = European Union; EvNote = Evaluation Note; IEG = Independent Evaluation Group; IFC = International Finance Corporation; IMF = International Monetary Fund; IRI = International Republican Institute; MOF = ministry of finance; NGO = nongovernmental organization; PCR = Project Completion Report; PD = project document; PEFA = Public Expenditure and Financial Accountability; PER = Public Expenditure Review; PFM = public financial management; PIM = public investment management; PIMA = Public Investment Management Assessment; PPAR = Project Performance Assessment Report; PRA = Portfolio Review Analysis; PSD = private sector development; SCD = Systematic Country Diagnostic; SECO = State Secretariat for Economic Affairs (Switzerland); SMEs = small and medium enterprises; TBD = to be determined; UN = United Nations; UNDP = United Nations Development Programme; USAID = United States Agency for International Development; XPSR = Expanded Project Supervision Report. 34 Appendix B. Preliminary Portfolio Review The team has reviewed the World Bank, International Finance Corporation (IFC), and Multilateral Investment Guarantee Agency portfolios for fiscal years (FY)14–21 (including projects active as of FY14) and identified projects and tasks relevant to each of the three issue areas: private sector development, governance, and local public services, and the coronavirus pandemic (COVID-19) response. The graphs below show the proportion of the portfolio for each issue area identified. Lending Overview The three issue areas and the COVID-19 response capture 81 percent of the lending portfolio, equivalent to US$1,079 million in International Development Association lending, grants, and trust funds (see figure B.1). Of this, US$122 million, or just over 10 percent, is development policy financing. All five development policy operations (DPOs) active during the period had prior actions related to governance or private sector development (and four of them relate to both areas). Of the remainder of the portfolio, 25 percent of lending and trust funds (US$338 million) is directly relevant to private sector development, 5 percent (US$67 million) to governance, and 20 percent (US$260 million) to the delivery of local public services. The evaluation will also cover the COVID-19 response, representing 5 percent of lending and trust funds over the period (US$70 million). An additional 9 percent of lending and trust funds (US$121 million) went to roads projects that may be relevant to private sector development; the Country Program Evaluation will examine them to the extent that they are relevant to this issue area. Likewise, an additional 8 percent of lending and trust funds (US$102 million) went to national-level health and education reform that may affect what is done at the local level; the Country Program Evaluation will examine them to the extent that they are relevant to the delivery of health and education services at the local level. 35 Figure B.1. Coverage of FY14–21 Lending and Trust Fund Portfolio by Issue Area (percentage of commitments) Not included Private sector 19% development 25% COVID-19 response 5% National health Roads and education 9% reform 8% Governance 5% Local public DPOs (focus on services PSD and 20% governance) 9% Source: Independent Evaluation Group. Note: DPO = development policy operation; FY = fiscal year; PSD = private sector development. DPOs supported policy reforms related to private sector development, governance, and the energy sector. Four DPOs from FY14 through FY19 contained prior actions relevant to governance and competitiveness, and several policy areas were supported in multiple operations (see table B.1 below). One DPO contained prior actions targeting the energy sector. DPOs in FY14–21 consisted of the following: • A programmatic, unnamed DPO series, with two operations of US$25 million each, approved in FY14. These operations contained prior actions related to reforms in the following areas: o Governance: anticorruption, budget discipline and transparency; transparency in public procurement; increasing energy sector transparency, governance, and accountability; and judicial reform o Private sector development: starting a business, corporate governance, risk- based firm inspections, banking sector supervision, and deposit protection • The Governance and Competitiveness DPO, US$24 million, approved in FY17. This was originally planned to be a two-DPO programmatic series, but the 36 second operation was canceled. The operation supported prior actions related to reforms in the following areas: o Governance: anticorruption; transparency in public procurement; increasing energy sector transparency, governance, and accountability; and public sector service delivery standards o Private sector development: risk-based firm inspections, tax policy for exporters, tax administration, and cross-border telecommunications • The Economic Governance DPO, US$24 million, approved in FY19, contained prior actions related to reforms in the following areas: o Governance: budget discipline and transparency, budget oversight of state- owned enterprises, revenue mobilization, regulating conflicts of interest in the public sector (anticorruption), efficiency of public procurement o Private sector development: food safety standards, tax administration, and cross-border telecommunications and digital economy • The Energy Sector DPO, US$24 million, approved in FY15. This DPO supported prior actions to improve the energy sector’s financial viability; enhance the sector’s governance, transparency, and accountability; and manage the impact of power shortages on impoverished regions. 37 Table B.1. Policy Areas Supported by Multiple Development Policy Operations Increasing Energy Sector Transparency Transparency, Budget or Efficiency in Governance, Risk-Based Discipline and Public and Judicial Firm Tax Cross-Border Anticorruption Transparency Procurement Accountability Reform Inspections Administration Telecommunications DPO 1 (FY14) X X X X X X Programmatic X X X X X X DPO 2 (FY14) Governance and X X X X X X Competitiveness DPO Economic X X X X X Governance DPO Energy Sector X DPO Source: Independent Evaluation Group. Note: DPO = development policy operation; FY = fiscal year. 38 The preliminary portfolio review has identified that six projects do not fall within any of the issue areas. These represent 19 percent of lending and trust funds (US$252 million). They are primarily in the education, agriculture, energy and extractives, and water and sanitation sectors. Despite being mapped to some sectors that are relevant, the projects at the level of their project development objectives, components, and activities are not relevant to the issue areas. They include projects to help the country recover from the violence that occurred after the 2010 revolution; develop an early warning system for emergency situations; strengthen country’s capacity to respond to disasters; improve forest ecosystem management; improve electricity supply; and improve the accuracy and timeliness of hydrometeorology services in Central Asia. If the team finds that any of these projects are indeed relevant to the issue areas during the evaluation, they will be included through the appropriate lens of the relevant issue area. Thus, indicating that they are out of scope at this stage does not have a material impact on the evaluation. International Finance Corporation Investments and Multilateral Investment Guarantee Agency Guarantees The issue areas cover all IFC advisory and investments, and all Multilateral Investment Guarantee Agency activity over the period: 33 IFC investments with a total project size of US$127 million, 35 IFC advisory services, and one Multilateral Investment Guarantee Agency guarantee with a gross exposure of US$5.8 million. These are all relevant to the private sector development issue area. Advisory Services and Analytics The issue areas capture a significant amount of the World Bank advisory services and analytics (ASA) conducted over the period. Figure B.2 shows the number of ASA tasks within the period that fall within each issue area. 39 Figure B.2. Coverage of FY14–21 ASA Portfolio in Issue Areas, by Number of Tasks Not included, 20 Private sector development, 22 COVID-19, 1 Local public services, 2 Private sector development and governance, 2 Governance, 26 Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; FY = fiscal year. The preliminary portfolio review has identified that 20 ASAs do not fall within any of the issue areas. These include seven tasks in the health sector; four in the public administration sector; three in social protection; one in public administration and social protection; two in education; and one each in energy and extractives, transportation, and agriculture and forestry. As with the lending and trust fund portfolio, although these ASAs may appear relevant from the perspective of the sectors they are mapped to, they address topics that are outside the scope of the issue areas. For instance, although public administration is key to governance, the ASAs that the team has identified as out of scope include one on climate change adaptation, two on disaster resilience, one on safety nets, and one on the environmental and social framework. As with lending, if the evaluation identifies that any of these are indeed relevant to the Bank Group’s work in the issue areas, they will be integrated into the evaluation through the appropriate lens. Therefore, excluding these ASAs from the relevant portfolio after this initial portfolio review will not have a material impact on the evaluation. Lending by Economic Sector As indicated in figure B.3, the largest sectors of investment project financing lending and trust funds are education, agriculture, energy and extractives, and health. 40 Figure B.3. Lending, Grants, and Trust Fund Portfolio by Economic Sector (excluding DPOs), FY14–21 180 Portfolio size (US$, millions) 160 17 3 140 12 60 120 0 1 100 80 150 151 21 60 4 129 115 99 3 0 109 40 69 3 63 50 55 20 38 0 IDA Lending and Grants Trust Funds Source: Independent Evaluation Group. Note: DPO = development policy operation; FY = fiscal year; IDA = International Development Association. • Education projects include three community-driven development projects (and additional financing for one of them) that are within the scope of the issue area on local public services. It includes three national-level education reform programs that the team will review further to identify any links with the delivery of education services at the local level. It also includes two projects that focus on disaster response and are outside the scope of the issue areas (as discussed above). • Agriculture projects include nine projects that fall within the private sector development issue area and an Integrated Forest Ecosystem Management project that falls outside of the scope of the focus areas. This sector has received substantial funding from trust funds, particularly in the form of a US$38 million grant from the Global Agriculture and Food Security Program to support the Agriculture Productivity and Nutrition Improvement Project and smaller trust funds for four other projects. • Energy and extractives include heating, urban development, and community- driven development (village improvement) projects that are included in issue area of local public services; an emergency recovery project that supported the 41 recovery from the 2010 revolution and is out of scope of the issue areas; and an electricity supply project that is out of the scope of the issue areas. Many of the health; water, sanitation, or waste; and social protection projects address local public services. The health sector also includes a national-level reform program that the team will review further to identify any links with the delivery of local services. The financial sector; industry, trade, and services; and information and communications address private sector development. Public administration includes 13 projects that fall within the governance issue area. It includes one project on emergency situations, which refers to the 2010 revolution and is outside the scope. The transportation projects (which are all for roads) will be examined from the perspective of their contribution to private sector development. Compared with that in FY14–17, lending in FY18–21 focused more on education, less on agriculture, and less on energy and extractives (figure B.4). It also had no new commitments on transportation or water, sanitation, or waste. This partially reflects the reduction in focus on the energy sector because reforms had not gained the traction the World Bank wished to see. Although the FY18–21 Country Partnership Framework included transport connectivity, no new projects were committed in FY18 and beyond. Indeed, the Country Partnership Framework did not include new projects in these areas, and the ongoing commitments as of the Country Partnership Strategy were already substantial. The new Country Partnership Framework included digital connectivity, which is reflected in the lending on information and communications. 42 Figure B.4. Lending and Trust Fund Commitments by Economic Sector under FY14–17 CPS versus FY18–21 CPF 160 140 Commitments (US$, millions) 120 0 55 0 1 100 0 1 80 0 60 125 12 3 115 115 95 0 3 100 40 86 3 73 17 17 56 50 56 50 20 4 40 2 4 29 1 17 25 13 21 17 0 0 - - FY14–17 FY18–21 FY14–17 FY18–21 FY14–17 FY18–21 FY14–17 FY18–21 FY14–17 FY18–21 FY14–17 FY18–21 FY14–17 FY18–21 FY14–17 FY18–21 FY14–17 FY18–21 FY14–17 FY18–21 FY14–17 FY18–21 AGR EDU E&E FIN HEA ITS I&C PUB SOC TRA WAT IDA Trust funds Source: Independent Evaluation Group. Note: AGR = agriculture; CPF = Country Partnership Framework; CPS = Country Partnership Strategy; E&E = energy and extractives; EDU = education; FIN = finance; FY = fiscal year; HEA = health; I&C = information and communications; IDA = International Development Association; ITS = industry, trade, and services; PUB = public administration; SOC = social protection; TRA = transportation; WAT = water, sanitation, and waste. 43