Approach Paper Country Program Evaluation World Bank Group Support to Georgia, Fiscal Years 2014–23 June 8, 2023 1. Evaluation Purpose and Objectives 1.1 The Country Program Evaluation (CPE) will assess the performance of the World Bank Group’s support of Georgia in achieving its development objectives. It will evaluate how the Bank Group program has adapted over time to changing conditions and priorities. The evaluation covers the period from fiscal year (FY)14 to FY23, starting with the country’s graduation from financing by the International Development Association (IDA), the publication of Social-Economic Development Strategy of Georgia— Georgia 2020, and the accession to Deep and Comprehensive Free Trade Area of the European Union (EU) following the implementation of an ambitious reform program. 1.2 The evaluation will seek to assess how well the Bank Group–supported strategy was aligned with Georgia’s main development challenges and how effective the Bank Group’s support was in addressing these challenges. It will assess the use of analytical, advisory, diagnostic, and evaluative work to inform the Bank Group’s strategic directions and project selection, design, and implementation. Given the sizable involvement of other donors, the CPE will assess the use of partnerships, convening power, and complementarity with the work of other donors and the Bank Group’s ability to adjust to changes in the economic and political environment. 1.3 The evaluation seeks to identify lessons that support the further adaptation and refinement of Bank Group engagement in support of the country’s development priorities. It intends to inform management and the Board of Executive Directors in the design of the upcoming Country Partnership Framework (CPF) expected in 2024, and the government of Georgia in its engagement with the Bank Group. The audience of the evaluation includes Bank Group boards, Bank Group staff and management working on Georgia, and the government of Georgia. The evaluation may also be of interest to a broader set of stakeholders, including international donors coordinating their efforts with the Bank Group; and local and international civil society, research organizations, and citizens. 1 2. Country Context and Development Challenges Country Context 2.1 Georgia has a track record of successful economic management and governance reform, starting after the 2003 Revolution of Roses. For much of the past two decades, the Bank Group supported economic reform in Georgia, which also took place in a context of increasing alignment with the EU acquis communautaire, starting with the adoption of an EU-Georgia Action Plan in 2006. Economic reforms contributed to one of the lowest levels of corruption in the Europe and Central Asia region, according to Transparency International’s Corruption Perception Index,1 with levels that were better than some EU accession countries. Georgia also experienced significant improvement in the Worldwide Governance Indicators,2 improving from the bottom 30th percentile to the top 25th percentile for regulatory quality between 2005 and 2014.3 Economic reforms were also informed by the World Bank’s Doing Business Indicators, on which Georgia’s rankings improved from 100 in 2007 to 6 in 2019.4 Facing various shocks, including the 2007–08 global economic and financial crisis and an armed conflict with the Russian Federation in 2008, the country pursued a successful countercyclical fiscal policy (World Bank 2018). 2.2 The Georgian political environment was relatively stable for most of the period since the 2003 Revolution of Roses, with only one change in governing party. However, polarization is increasing between the two major political camps and within the governing coalition, and the country’s geographic location is a source of instability. An armed conflict with Russia in 2008 led to the temporary classification of Georgia as a fragile and conflict-affected state between 2010 and 2012 and resulted in more than 200,000 internally displaced persons.5 Georgian Dream, a coalition led by entrepreneur Bidzina Ivanishvili, won the 2012 election against the United National Movement, which was in power since 2004. Georgian Dream continued the previous government’s reform agenda, though with an increased focus on social spending. Despite these relative stable governments, the World Bank’s Enterprise Surveys consistently identified political instability as the primary concern raised by firms with respect to the business environment.6 The Fragile States Index flagged factionalized elites as a concern,7 reflecting polarization between Georgian Dream and the opposition and a history of divisions within the Georgian Dream coalition. In 2020, the country adopted election reforms that included stronger elements of representative democracy, reducing parties’ ability to win disproportionate majorities. Risk and uncertainty have increased after Russia’s invasion of Ukraine in 2022. 2.3 The decade prior to 2014 was characterized by strong economic growth, but high rates of unemployment and inequality persisted. Annual gross domestic product (GDP) 2 growth averaged 5.5 percent in the 10 years leading up to 2014, while unemployment remained high at about 15 percent. Inequality was also high compared with other countries in the region, with a Gini coefficient of 0.39 (World Bank 2014). The main drivers of growth during that period were capital accumulation and public spending, largely financed through external borrowing. Productivity growth and foreign direct investment were concentrated in nontradable sectors, particularly construction and real estate. 2.4 Georgia has continued its economic expansion since its graduation from IDA in 2014, but progress on poverty reduction has been limited. The country graduated from IDA in 2014 and reached upper-middle-income status in 2015. Since then, annual real GDP per capita grew by 4.2 percent on average until the COVID-19 pandemic (figure 2.1). Growth during that period was primarily driven by domestic demand, tourism, remittances, and capital inflows that benefited infrastructure, construction, and finance. Remittances increased from 11.3 percent of GDP in 2014 to 14.2 percent in 2021. Reforms helped open the economy, although the complexity of the export basket remains low, and success in creating jobs was limited (World Bank 2022). Georgia has seen additional improvements in some areas of the Worldwide Governance Indicators since 2014, while progress in others, such as the rule of law and accountability, was partially reversed. The national poverty rate declined from 21.6 percent in 2014 to 17.5 percent in 2021 after a temporary increase in the aftermath of the COVID-19 pandemic. A 2018 World Bank report found that to graduate to high-income-country status, Georgia would need to continue on a path of structural transformation and expand its manufacturing sector (Posadas et al. 2018). 2.5 Georgia’s population is decreasing, but vulnerability and welfare disparities remain high. The country’s population has been in decline since the end of the Soviet Union, and labor force participation is relatively low, especially among women and youth. A targeted social assistance program and the pension system have helped reduce extreme poverty, but a lack of labor market opportunities—especially for less skilled workers—is threatening the sustainability of poverty eradication efforts. High levels of informality contribute further to a risk of a large share of the population falling back into poverty. Welfare disparities persist across geographic regions and along the urban-rural divide. 3 Figure 2.1. Development GDP Growth, GDP per Capita, and Poverty Rate (2011–20) a. GDP growth and GDP per capita, 2011–21 20 5,500.00 5,000.00 GDP per capita (US$) 10 GDP growth (%) 4,500.00 4,000.00 0 3,500.00 -10 3,000.00 Year GDP growth GDP per capita b. Poverty rate, 2011–20 70 60 50 Poverty rate (%) 40 30 20 10 0 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 Year US$6.85 per day National poverty line US$2.15 per day Source: Panel a: World Development Indicators database (World Bank), https://databank.worldbank.org; panel b: Poverty and Inequality Platform (World Bank), https://pip.worldbank.org/home. Note: GDP = gross domestic product. 2.6 Georgia has made efforts to deepen integration into global markets, with mixed success. The country acceded to the EU’s Deep and Comprehensive Free Trade Area in 2014. In 2018, a free trade agreement with China entered into force, followed in 2021 by 4 the UK-Georgia Strategic Partnership and Cooperation Agreement. Georgia submitted a formal application for EU membership in March 2022. Exports of goods and services have risen from 40 percent of GDP in 2014 to 55 percent in 2019, driven primarily by travel and tourism and agricultural products, minerals, and metals.8, 9 Although increasing in magnitude, Georgia’s exports have failed to evolve in a significant way toward more value added. According to the World Bank, the country is exploiting only 4.2 percent of the market potential of its export structure, with services (primarily tourism) accounting for 70 percent of export value added (World Bank 2022). 2.7 Georgia’s current growth model comes with risks and limitations. Although Georgia benefits from a significant volume of foreign direct investment (8.7 percent of GDP on average since 2014), investment remains concentrated in infrastructure, real estate, and services. Georgia has experienced large current account deficits, averaging 10 percent of GDP over the past decade. Balance sheet exposure to foreign currency fluctuations is significant as dollarization levels remained relatively high. Central government debt increased from 31 percent in 2014 to a peak of about 60 percent of GDP during the pandemic and subsequently decreased to 40 percent as of the end of 2022, resulting from both fiscal consolidation and currency appreciation in 2022. Domestic savings have increased over the past decade, helping reduce reliance on external financing. Contingent liabilities related to quasi-fiscal activities of state-owned enterprises and power purchasing agreements with hydropower operators are also a source of fiscal risk (World Bank 2022), although authorities have introduced some mitigating measures over the past two years. Main Development Challenges 2.8 The government’s Georgia 2020 plan, adopted in 2014, identified private sector competitiveness, development of human resources, and access to finance as main priorities. These priorities were intended to be underpinned by a sound macroeconomic policy framework and efficient public administration (Georgia 2014). 2.9 In its 2014 Country Economic Memorandum, Georgia: Seizing the Opportunity to Prosper, the Bank Group determined that Georgia needed to strengthen its income- generating capacity by increasing export potential and by targeting job growth and the job readiness of its population. The limited domestic market size, regional disparities, and a poorly developed tradable sector required a sustained increase in firm and labor productivity combined with a strengthened capacity to trade. Georgia’s ability to benefit from a more sophisticated export structure was undermined by firms’ inability to grow durable export relationships and to diversify past a single product and destination. Job readiness was identified as a problem disproportionately affecting households in the bottom 40 percent of the income distribution (World Bank 2014). 5 2.10 Georgia’s ability to generate economic growth is increasingly facing headwinds. Annual growth has been on a declining trend, and the World Bank expects growth to slow to 3 percent by 2030 and to decrease further after that, assuming a business-as- usual scenario (World Bank 2022). Capital accumulation accounted for the largest share of productivity growth between 2010 and 2019. Investment remained concentrated in physical infrastructure. A high share of small and informal firms with relatively low sophistication and capacity is hampering the country’s ability to innovate and to participate in global supply chains (World Bank 2018). Firms face a series of cross- cutting constraints, including with respect to an underdeveloped logistics sector, gaps in multimodal transport infrastructure, access to financial services, and the quality of institutions. Georgia is lagging in adopting sophisticated digital technology (Baller, Dutta, and Lanvin 2016; World Bank 2022). Limited competition in a highly concentrated banking system with a low reach of the nonbank financial sector and underdeveloped capital markets has contributed to a shortage of stable, long-term funding for the private sector (IMF 2021). Poor transportation infrastructure hampered spatial integration and trade during the evaluation period, although there are encouraging signs because the country’s performance in the Logistics Performance Index improved from rank 119 in 2018 to 79 in 2023.10 2.11 A declining population, compounded by poor education and health outcomes, places a burden on employers. Georgia’s population has declined sharply since the early 1990s, driven by low fertility and high emigration rates in the 1990s and 2000s. Fertility rates recovered to more than 2.0 children per woman after 2013 and peaked at 2.2 children per woman in 2015,11 but the lagging effect of low birthrate cohorts in reproductive age will cause population decline to persist. Gross emigration has averaged approximately 2.5 percent of Georgia’s population per year between 2016 and 2019, but this was largely offset by immigration, resulting in an average net emigration of 0.07 percent during that period. More recently, Georgia experienced an influx of Russian migrants in the context of the Russian invasion of Ukraine. Reasons for emigration are complex, with recent emigration being influenced primarily by economic and social factors (State Commission on Migration Issues 2021). The United Nations estimates that in addition to the 3.7 million Georgians residing in the country, approximately 860,000 Georgians born in Georgia reside abroad .12Georgia’s Human Capital Index score was 56.9 in 2020,13 meaning that the future earnings potential of children born today will be less than 60 percent of what it could have been with complete education and full health. The main challenges resulting from demographic change at the start of the evaluation period were documented in the 2011 South Caucasus Programmatic Poverty Assessment, which identified the need for increased labor force participation (especially by women) and productivity, highlighting that analytical work was required to identify policy options (World Bank 2011). 6 2.12 The contribution of labor and skills to total factor productivity development has been relatively modest since 2010 (World Bank 2022). The country suffers from a skill mismatch, with 38 percent of workers with tertiary education working in semiskilled occupations. The share of employment in low-productivity agriculture is relatively high at 19 percent of total employment. 3. World Bank Group in Georgia Evolution of Strategic Engagement 3.1 In the decade leading up to Georgia’s IDA graduation, Bank Group engagement focused on income- and employment-generating growth, human development and social protection outcomes, and strengthened efficiency of public service delivery. The World Bank–supported program of investment projects focused heavily on transport to contribute to the growth agenda, while development policy financing supported reforms in public administration, industry, and trade, and spending on social protection and health. Activity of the International Finance Corporation (IFC) was concentrated in the financial and energy sectors. 3.2 Bank Group engagement over the evaluation period can be grouped into three interrelated areas: growth and job creation, inclusion and equity, and sustainability and resilience. The strategy for Bank Group engagement since IDA graduation (figure 3.1) was laid out in two documents: the Country Partnership Strategy (CPS) for FY14–17 and the CPF for FY19–22, the latter of which was informed by the 2018 Systematic Country Diagnostic (SCD). The SCD concluded that Georgia did not need a new growth paradigm to achieve the twin goals of poverty reduction and shared prosperity but that it did need to adjust and refine its current model. The CPF emphasizes a One Bank Group approach with a focus on maximizing finance for development using the Cascade approach. Growth and Job Creation 3.3 Bank Group strategy to support growth and job creation was largely consistent throughout the evaluation period, focusing on enhancing productivity and competitiveness. Priorities that remained in effect since 2014 include improving labor market outcomes through education and skills matching, upgrading infrastructure, increasing access to financial services for firms, supporting agricultural modernization, and promoting innovation. The growth and jobs agenda intersected with other major strategy areas. It applied a spatial lens, with an emphasis on decreasing the disparity between urban and rural areas and supporting social integration as jobs offered a way to escape poverty and reduce the strain on tax-funded social assistance. 7 Figure 3.1. Evolution of World Bank Group Engagement in Georgia since 2014 Source: Independent Evaluation Group. Note: CPF = Country Partnership Framework; CPS = Country Partnership Strategy; FY = fiscal year; ICT = information and communication technology; SCD = Systematic Country Diagnostic, TSA = targeted social assistance. 8 3.4 The 2018 SCD highlighted the need to focus on increasing integration into the global economy and value chains. Within the infrastructure agenda, the 2019–22 CPF increased the emphasis on infrastructure services to support integration, especially logistics and digital infrastructure. The private sector–related reform agenda shifted from supporting general legal and regulatory reform toward “last mile” reforms focused on attracting private investment, with a stronger emphasis on advisory services and analytics (ASA) support. The SCD also revealed firm-level constraints and identified four sectoral opportunities in Georgia: agribusiness, hydropower, textiles and apparel, and tourism. Of these opportunities, the CPF had a dedicated objective on agribusiness and viewed tourism as an opportunity to increase economic participation in the region. Inclusion and Equity 3.5 Both the 2014–17 CPS and the 2019–22 CPF supported human capital formation. Within health care, focus shifted from expanding access to universal coverage to enhancing the efficiency of health care delivery. The Bank Group–supported strategy in education focused on improving education system quality and making preprimary education universal. Bank Group support complemented the programs of other donors. The CPF introduced additional objectives related to teaching quality and vocational education, both of which the SCD identified as issues. The CPF states that gender considerations will inform the program implementation. 3.6 Spatial integration was reflected as a cross-cutting priority throughout the evaluation period, with infrastructure and social services applying a spatial lens to reduce urban-rural inequality. The SCD reiterated this challenge and emphasized the importance of connectivity to reduce spatial inequality. It identified tourism and agriculture as potential growth engines to reduce the urban-rural divide. The CPS also aimed to increase social accountability by strengthening participatory local development, a theme that was continued in the CPF with community-led investments to strengthen infrastructure and tourism. 3.7 The theme of expanding and strengthening social protection remained in effect throughout the decade and was complemented in the CPF with attention to financial inclusion. The Bank Group prioritized support for an expansion of Georgia’s targeted social assistance program, with a focus on the bottom 20 percent throughout the review period. The World Bank’s support for pension reform reflected progress, with the CPS supporting the development of a pension reform road map and the CPF focusing on the enactment of relevant legislation and the establishment of a pension agency. Additionally, the CPF supported the operationalization of a deposit insurance agency to strengthen households’ financial resilience. 9 Sustainability and Resilience 3.8 The SCD considered an increasingly constrained fiscal space and environmental concerns as primary threats to sustainability and resilience. Concerns with the longer- term sustainability of public finances relating to both high infrastructure investment and increasing social spending were addressed in the CPS with objectives relating to the identification, prioritization, and implementation of public investments and the adoption of modern public sector and fiscal management systems. The CPF retained a focus on these objectives in its support to improve macrofiscal management and mitigate risk from contingent liabilities of state-owned enterprises and power purchasing agreements, as well as subnational financial management. Environmental issues were not explicitly reflected in the results framework of the 2014–17 CPS. The SCD identified deforestation, air pollution, and vulnerability to climate change as challenges. The CPF subsequently included priorities related to sustainable land management, land degradation, and disaster risk management. World Bank Group–Supported Programs 3.9 Between FY14 and FY18, the World Bank approved 10 investment projects for US$538 million and five development policy operations for US$313 million (figure 3.2; table 3.1). Investment projects were concentrated in transportation, which held 57 percent of the project value. The focus of budget support operations shifted from public sector management reform—which accounted for about half of the development policy operation prior actions in the previous cycles—to greater attention to human development and private sector development in the CPF. During the CPS, the World Bank undertook 56 analytical and advisory activities that covered a range of areas, with public administration (24 percent); social protection (15 percent); and industry, trade, and services (14 percent) among the more prominent. IFC committed US$380 million in long-term own-account financing and mobilized US$825 million during the CPS period, with slightly more than half of own-account financing benefiting financial sector projects. The remainder of the financing consisted of a relatively sizable investment in energy and health care. IFC supported 10 advisory projects for a total amount of funds managed of $31.1 million. The Multilateral Investment Guarantee Agency issued a US$63 million guarantee during the period to support 1 energy project in collaboration with IFC and increased the amount of an existing guarantee by US$27.8 million. Out of 8 World Bank projects evaluated during the period, 2 scored satisfactory for overall development outcome, 3 scored moderately satisfactory, and 2 scored moderately unsatisfactory.14 The Bank performance rating was moderately satisfactory for 6 projects and satisfactory for 2 projects. 10 Figure 3.2. Evolution of World Bank Lending by Commitments Approved and Number of Projects, Fiscal Years 2014–22 700 18 Total commitment (US$, millions) 600 16 14 500 12 Projects (no.) 400 10 300 8 6 200 4 100 2 0 0 2014 2015 2016 2017 2018 2019 2020 2021 2022 Fiscal year DPF IPF and P4R World Bank lending (no. of projects) World Bank ASA (no. of projects) Source: World Bank Business Intelligence database Note: ASA = advisory services and analytics; DPF = development policy financing; IPF = investment project financing; P4R = Program-for-Results. Table 3.1. World Bank Investment Operations—Instrument Composition, Fiscal Years 2014–23 FY14–18 FY19–23 Number of Number of projects/ Commitment projects/ Commitment Source of Financing operations (US$, millions) operations (US$, millions) IPF 10 524 10 651 DPF 5 313 2 98 P4R 0 0 1 400 Total 15 837 13 1,149 Source: World Bank Business Intelligence database Note: DPF = development policy financing; FY = fiscal year; IPF = investment project financing; P4R = Program-for-Results. 3.10 The Bank Group portfolio grew significantly between FY19 and FY23, driven by a large increase in investment project financing and one large Program-for-Results. Project approvals reached US$1.1 billion during the CPF period, nearly half of which was for education projects, including a US$400 million Program-for-Results. Development policy operation approvals fell to US$98 million, with prior actions primarily focusing on public sector management and private sector development. Since the start of the CPF cycle, the World Bank undertook 27 ASAs for US$11 million. Approximately one-third of this budget was tagged to public administration, one-third 11 to the financial sector, and one-third to sizable expenditures on education. The average size per ASA more than doubled as the program evolved to larger, more programmatic multiyear ASAs. IFC investments decreased to US$99.14 million in long-term, own- account finance and US$8.2 million in mobilization. All of these projects were in the finance and insurance sector. IFC supported three advisory projects for US$9.7 million in funds managed. The Multilateral Investment Guarantee Agency issued a €47.5 million guarantee to a bank focusing on serving small and medium enterprises, complementing International Bank for Reconstruction and Development support that consisted of both investment and advisory services. 4. Country Program Evaluation Special Themes Private Sector Development 4.1 Georgia has been celebrated as a star reformer since it implemented an ambitious private sector development reform agenda after the Revolution of Roses in 2003. Since then, private sector development has been a priority for the government of Georgia and major development partners such as the European Bank for Reconstruction and Development and the Asian Development Bank, and for the Bank Group. In the years since then, the country has successfully attracted investment in infrastructure and the financial sector and experienced a period of impressive economic growth. However, growth was driven primarily by consumption and high rates of investment in enabling infrastructure, with mostly negative contributions from net exports (World Bank 2018). 4.2 There is a need to refine Georgia’s growth model. The World Bank estimates that “under a business-as-usual scenario, potential GDP per capita growth could slow from 4–5 percent currently to 3.5 percent in 2030 and to 1.9 percent by 2050” (World Bank 2022, 5). Reasons for the slowed growth include limited ability to sustain past levels of investment, given the need to stabilize debt in the medium term; limited availability of skills in the economy; and firms that are not increasing in productivity and sophistication. Reflecting these constraints, 44 percent of the increase in productivity between 2010 and 2019 stemmed from increases in capital stock. One-third of the increase resulted from an increase in total factor productivity, but this resulted primarily from a move into services from agriculture, with limited contributions from within- sector increases in productivity. Labor and human capital formation had a relatively modest impact on growth (14 percent and 9 percent respectively; World Bank 2022). 4.3 Despite reforms and major trade agreements, including with China and the EU, Georgia has not been able to fully capitalize on trade opportunities. Although Georgia has been exporting more products to more markets in the period since 2014, export growth before the COVID-19 pandemic has been concentrated in tourism. In 2019, 12 tourism represented 42.1 percent of exports after growing on average 12.8 percent per year between 2014 and 2019. By contrast, goods exports grew at 1.25 percent per year during the same period, with agriculture and minerals (the two sectors with the least embedded economic complexity) making up the largest share of goods exported. As a result, the overall complexity of Georgia’s export basket decreased between 2014 and 2019.15 4.4 The CPE will examine in depth the relevance and efficacy of Bank Group support for creating a competitive private sector that contributes to job creation and growth. This special theme will review the evolution of Bank Group support for Georgia’s private sector development–related reform agenda throughout the evaluation period— including support for relevant enabling sectors such as infrastructure and financial markets—and the role of gender in private sector development. The CPE will pay special attention to the adequacy of the analytical and diagnostic underpinnings that guided engagement. The analysis will include an assessment of Bank Group internal coordination and complementarity between the strategies and activities of the World Bank, IFC, and the Multilateral Investment Guarantee Agency, and with development partners. Human Capital and Labor Skills 4.5 Sustained low fertility rates and persistent out-migration—particularly of qualified workers—have led to a decline in Georgia’s population. Although fertility rates have recovered to more than 2.0 children per woman after 2013, and emigration has been offset by immigration to a large degree,16 these changes have not been enough to overcome the trend toward a shrinking and aging population (figure 4.1). The medium-term impact of a recent inflow of young, relatively well-educated Russians and capital in the context of the Russian invasion of Ukraine is not clear. 4.6 At the same time, too few Georgians hold well-paying jobs in the formal sector, and gender disparities in labor force participation are significant. Periods of high economic growth have not translated into commensurate net job creation, and female labor force participation remains low. A structural shift from informal, low-productivity agriculture toward services and, to a lesser degree, manufacturing has resulted in some productivity gains. Additionally, less than 20 percent of Georgia’s working-age population are private sector wage workers, while 34 percent are self-employed (primarily in unpaid and low-productivity activities), 26 percent are not in the labor force, and 11 percent are employed by the public sector (World Bank 2018). As a result, opportunities to raise living standards have been limited, and the burden of poverty reduction has fallen primarily on social transfer systems. 13 Figure 4.1. Population Distribution in Georgia a. 2004 population distribution 100+ 90–94 80–84 70–74 60–64 Age 50–54 40–44 30–34 20–24 10–14 0–4 10.0 8.0 6.0 4.0 2.0 0.0 2.0 4.0 6.0 8.0 10.0 Population (%) Male Female b. Projected 2025 population distribution 100+ 90-94 80-84 70-74 60-64 Age 50-54 40-44 30-34 20-24 9-14 0-4 10.0 8.0 6.0 4.0 2.0 0.0 2.0 4.0 6.0 8.0 10.0 Population (%) Male Female Source: United Nations Department of Economic and Social Affairs, Population Division, World Population Prospects: The 2022 Revision. https://population.un.org/dataportal/data/indicators/46/locations/268/start/2004/end/2025/pyramid/pyramidagesexplotsi ngle 14 4.7 Skills mismatch and a shortage of skilled labor place a burden on firms. Despite broad access to primary and secondary education and relatively high enrollment rates compared with peer countries, Georgia performed significantly lower than other countries in the region in reading, math, and science in OECD’s Programme for International Student Assessment.17 Tertiary and vocational education often fail to confer relevant skills, including specialized technical skills and generalized skills such as critical thinking or sociobehavioral skills (World Bank 2022). As a result, employers cite an inadequately educated workforce as third among the top 10 business environment constraints (World Bank 2019). 4.8 The CPE will evaluate the relevance and efficacy of Bank Group support to addressing human capital constraints. The special theme will assess whether Bank Group support to address human capital constraints has focused on the correct areas (in coordination with development partners), including the extent to which Bank Group interventions have been informed by credible analytical and diagnostic work. The evaluation will trace relevant education outcomes to which the Bank Group has contributed, including with respect to gender aspects. It will also aim to identify factors underpinning migration patterns that have eroded human capital and to assess the impact of Bank Group support in retaining Georgia’s skilled labor. 5. Evaluation Questions 5.1 The CPE will assess the relevance and effectiveness of Bank Group engagements in Georgia after IDA graduation in 2014. Specific evaluation questions are as follows: 1. How relevant was the Bank Group–supported strategy for Georgia to the country’s development needs, and did it evolve over time to address changing priorities, country context, and lessons learned? To what extent was the strategy implemented, and how successful was it in achieving its objectives? 2. To what extent has the Bank Group’s support for private sector development and the business climate contributed to increased competitiveness, growth, and job creation? 3. To what extent has Bank Group support for human capital development created the conditions needed for the emergence of a Georgian labor force with complete education, full health, and necessary skills for a growing and entrepreneurial economy? Did Bank Group support help reverse the persistent out-migration of skilled workers? 15 6. Evaluation Design and Methods 6.1 Evaluation question 1 will employ a mixed methods approach that combines document review, analysis of available data, and qualitative analysis of semistructured interviews. Evaluation design related to this question focuses on reviewing the Bank Group’s and other donors’ diagnostic and analytical products and key country indicators to validate the relevance of the Bank Group strategy and the evolution of the strategy over time in response to changing conditions. To establish the relevance of design and objectives, the degree to which the program was informed by sound analysis, and the effectiveness of interventions, the Independent Evaluation Group (IEG) will perform content analysis of ASAs, project preparation, supervision, and (self-) evaluation documents, and of other evaluation documents. Semistructured interviews with key stakeholders—including Bank Group staff and management familiar with Georgia; government officials and private sector clients involved in Bank Group– supported activities in Georgia; and informed representatives of civil society, think tanks, and development partners—will help triangulate findings. We will select interview partners based on a mapping of relevant stakeholders, with a focus on the inclusion of all relevant interest groups. 6.2 Work to answer evaluation questions 2 and 3 will explore how Bank Group interventions contributed to addressing specific challenges. We selected private sector development and human capital for more detailed analysis because of their importance to Georgia’s development during the evaluation period and in the future. The CPE will aim to reconstruct the theory of change underpinning some areas of Bank Group support for these themes and causal chains to analyze and test the most plausible change pathways of how Bank Group interventions have supported engagement in these areas. 6.3 Wherever possible, the evaluation will link Bank Group support to country development outcomes. We will review strategies, portfolios, and evaluation documents of other donors to assess whether Bank Group operations achieved complementarity with the support of other donors. The CPE will pay special attention to whether the structuring, design, and sequencing of the country program and collaboration with development partners promoted complementarities that contributed to program results. Limitations 6.4 Only a few Bank Group–supported projects and operations approved during the evaluation period are closed and were fully evaluated and validated by IEG. This may constrain the assessment of impact. The Georgia lending portfolio is relatively young, with only 8 projects closed out of 28 projects approved during the evaluation period. Of 16 these, only 7 have self-evaluations that IEG has validated. Nevertheless, the set of interventions that will inform this evaluation will be larger and will include the assessment of the extent to which the design of all interventions reflected country context, sound analytical underpinnings, and lessons learned. 6.5 The availability of stakeholders with direct involvement in strategy formulation and program implementation may be limited. We may not be able to locate and interview all staff and officials in the Bank Group, government, and development partners who were directly involved in Georgia during the evaluation period. 7. Quality Assurance Process 7.1 The evaluation will go through IEG’s standard quality assurance process to ensure credibility of the evidence, validity, and utility of the findings. The process include internal review meetings with members of the leadership team. We will conduct the work under the guidance of Jeffrey Allen Chelsky (manager, Economic Management and Country Programs Unit); Carmen Nonay (acting director, Human Development and Economic Management); and Oscar Calvo-Gonzalez (acting director-general, Evaluation). In addition, three external reviewers will be involved in the quality assurance process: • Mercedes Vera Martin, Georgia mission chief 2016–20, International Monetary Fund • Koba Turmanidze, president, Caucasus Research Resource Center • Ari Perdana, evaluation specialist, Asian Development Bank 8. Resources 8.1 The evaluation will be prepared with an estimated budget of $525,000. The evaluation task team leader is Lars Johannes (senior economist), and core team members are Gabriela Chamartin (consultant), Konstantin Atanesyan (senior evaluation officer), and Patricia Acevedo (program assistant). Additional team members include Peter Freeman (consultant), Anna Sengphet Lattanavong (consultant), Aleksander Olechnowicz (consultant), Yiorgo Polenakis (consultant), and Disha Zaidi (consultant). 17 1To learn more about the Corruption Perceptions Index, see the Transparency International website at https://www.transparency.org/en/cpi/2022. 2The Worldwide Governance Indicators are a research data set summarizing the views on the quality of governance provided by a large number of enterprise, citizen, and expert survey respondents in industrial and developing countries. These data are gathered from a number of survey institutes, think tanks, nongovernmental organizations, international organizations, and private sector firms. The Worldwide Governance Indicators do not reflect the official views of the World Bank, its Board of Executive Directors, or the countries they represent. The Worldwide Governance Indicators are not used by the Bank Group to allocate resources. 3For information on the Worldwide Governance Indicators, see the World Bank’s website at https://info.worldbank.org/governance/wgi/. 4Data are from the Doing Business Indicators (database), World Bank, Washington, DC (accessed February 2, 2023), https://archive.doingbusiness.org/en/doingbusiness. 5For more information about the conflicts, see the United Nations High Commissioner for Refugees website at https://www.unhcr.org/georgia.html. 6For information and data about the World Bank’s Enterprise Surveys, see https://www.enterprisesurveys.org/en/data/exploreeconomies/2019/Georgia. 7For information about the Fragile States Index, see the Fund for Peace website at https://fragilestatesindex.org/. 8To learn more, see the Atlas of Economic Complexity at the Center for International Development at Harvard University at https://atlas.cid.harvard.edu/. 9Data are from the World Development Indicators (database), World Bank, Washington, DC (accessed February 2, 2023), https://databank.worldbank.org/source/world-development- indicators. For more information, please refer to the Logistics Performance Index, World Bank, 10 Washington, DC. https://lpi.worldbank.org/sites/default/files/2023-04/LPI_2023_report.pdf 11Data are from Fertility Rate, Total (births per woman)—Georgia (database), World Bank, Washington, DC (accessed February, 2, 2023), https://data.worldbank.org/indicator/SP.DYN.TFRT.IN?locations=GE. 12Data are from the International Migrant Stock (database), United Nations Population Division (accessed March 28, 2023) https://www.un.org/development/desa/pd/content/international- migrant-stock. 13Georgia’s score is from Human Capital Index (HCI) (scale 0–1)—Georgia (database), World Bank, Washington, DC (accessed February 2, 2023), https://data.worldbank.org/indicator/HD.HCI.OVRL?locations=GE. Additional information can be found from this database. 14 One project evaluated only Bank Group performance. 18 15To learn more, see the website of the Atlas of Economic Complexity at the Center for International Development at Harvard University at https://atlas.cid.harvard.edu/. Average emigration was 2.5 percent of the total population between 2016 and 2020, while 16 average immigration was 2.41 percent (State Commission on Migration Issues 2021). 17To learn more about Georgia’s performance in these areas, see the Organisation for Economic Co-operation and Development’s Education GPS at https://gpseducation.oecd.org/CountryProfile?primaryCountry=GEO. 19 References Baller, S., S. Dutta, and B. Lanvin, eds. 2016. The Global Information Technology Report 2016: Innovating in the Digital Economy. Geneva: World Economic Forum. Georgia (government of). 2014. Social-Economic Development Strategy of Georgia: Georgia 2020. Tbilisi: government of Georgia. IMF (International Monetary Fund). 2021. “Georgia: Financial Sector Assessment Program.” IMF Country Report 21/222, International Monetary Fund, Washington, DC. Posadas, J., M. Makovec, R. F. Jaef, C. Gruen, and M. I. Ajwad. 2018. Georgia at Work: Assessing the Jobs Landscape. Washington, DC: World Bank. State Commission on Migration Issues. 2021. Migration Profile of Georgia (2021). Tbilisi: government of Georgia. World Bank. 2011. “Georgia Demographic Change: Implications for Social Programs and Poverty.” Report 63156, World Bank, Washington, DC. World Bank. 2014. “Georgia: Seizing the Opportunity to Prosper.” Country Economic Memorandum 89608, World Bank, Washington, DC. World Bank. 2018. Georgia—From Reformer to Performer. Systematic Country Diagnostic. Washington, DC: World Bank. World Bank. 2022. “Georgia: Charting Georgia's Future.” Country Economic Memorandum 173357, World Bank, Washington, DC. 20 Appendix A. Evaluation Design Matrix Table A.1. Evaluation Design Matrix Methods ToC and ToA Question Trend analysis (EQ2 and EQ3) Relevance Effectiveness 1. How relevant to the development needs of Analysis of key country Content analysis of Content analysis of Content analysis of (self-) Georgia was the Bank Group–supported strategy indicators related to strategy documents (to evaluation documents to evaluation documents for Georgia, and did it evolve over time to address sectors the Bank Group document limited ToCs), extract evidence on the and other evaluation changing priorities, country context, and lessons aimed to address with an additional focus relevance of design and documents to capture learned? To what extent was the strategy successful Review of Bank Group’s on complementarity and objectives of operations evidence of contributions in achieving its objectives? and other donors’ sequencing of projects that are part of packages to outcomes diagnostic and analytical Interviews with Bank Interviews with clients Interviews with clients products documenting Group management, and partners to gauge and partners to gauge and analyzing trends in sector leaders, task team opinions about the opinions and collect the relevant sectors leaders (World Bank) and relevance and adaptation additional or Interviews with experts investment officers, and of Bank Group (dis)confirming evidence 2. To what extent has the Bank Group’s support for to validate trends and task leaders (IFC) to operations over the of the Bank Group private sector development and business climate findings validate and refine ToC decade contribution contributed to increased competitiveness, growth, and ToA Contribution analyses to and job creation? Stakeholder mapping test the most plausible and analysis of political change pathways 3. To what extent has Bank Group support for economy to identify human capital development created the conditions constraints to reforms needed for the emergence of a Georgian labor force with complete education, full health, and the skills necessary for a growing and entrepreneurial economy? Did Bank support helped reverse the persistent out-migration of skilled workers? Source: Independent Evaluation Group. Note: EQ = evaluation question; IFC = International Finance Corporation; ToA = theory of action; ToC = theory of change. 21 Appendix B. Methods The Independent Evaluation Group will follow a mixed methods approach to answer the evaluation questions. The primary methods of analysis include the following: • Descriptive review and analysis of the portfolio: We will describe the World Bank Group’s lending and nonlending portfolio for Georgia. Besides serving descriptive purposes, the portfolio analysis will constitute the basis for more in- depth content analysis. • Document review: In addition to the descriptive review and analysis of the portfolio, we will extract relevant information from Bank Group project documents, minutes of review meetings, advisory services and analytics reports, country engagement documents, and relevant documents and analysis from the government of Georgia, development partners, and credible external commentators. The team will code, classify, and compare evidence to address relevance and effectiveness questions. • Qualitative analysis of semistructured interviews: We will interview Bank Group management and staff familiar with Georgia and government officials and private sector clients involved in Bank Group–supported activities in Georgia, along with informed representatives of civil society, think tanks, and development partners. The identification of interviewees will aim to mitigate the risk of confirmation bias. We will conduct interviews virtually and during an in- person mission to Georgia, adhering to local health guidelines. • Analysis of survey and administrative data: We will use national statistics, recognized international data sources (for example, International Monetary Fund, World Bank, and official government statistics), and other reliable sources of data to analyze the trends in country and sectoral outcomes that the Bank Group has sought to influence. • Reconstruction and refinement of causal theories of change: The team will conduct a theory-based analysis under the two themes of private sector development and human capital development. The analysis will be more in- depth with the use of additional documentary and interview-based evidence within the selected areas of work. 22 Appendix C. Preliminary World Bank Group Portfolio Review The Independent Evaluation Group conducted a preliminary identification and classification of the World Bank Group portfolio in Georgia over the fiscal years (FY)14– 23. Portfolio identification is based on information extracted from the World Bank Business Intelligence database (World Bank financing and advisory services and analytics) and the management information system and advisory services databases of the International Finance Corporation and the Multilateral Investment Guarantee Agency (investments and advisory services). World Bank Lending Portfolio Between FY14 and FY23, the World Bank approved 28 financing operations valued at US$1.986 billion (tables C.1 and C.2) and 83 nonlending activities. This included commitments of $1.575 billion for investment projects (79 percent of total commitments) and $412 million for development policy financing operations (budget support). Excluding budget support operations, lending was focused on the education and transportation sectors, accounting for 28 and 21 percent, respectively, of total new commitments in the period FY14–23 (figure C.1). For development policy operations approved during the evaluation period, we reviewed prior actions to determine the nature of the reforms supported. Public financial management accounted for 11 percent of all total prior actions (figure C.2). 23 Figure C.1. World Bank Non–Development Policy Operation Lending by Sectors (% of total commitments, fiscal years 2014–23) 17.7 Education 28.3 Transportation 6.8 Energy and extractives Public administration 7.5 Financial Health 9.4 20.8 Other 9.4 Source: World Bank Business Intelligence database (accessed October, 10, 2022) Note: Other = industry, trade and services; information and communications technologies agriculture; social protection; and water, sanitation, and waste management. Figure C.2. Georgia’s Development Policy Operations Prior Actions by Theme (% of total prior actions, fiscal years 2014–22) Education 13.2 28.3 Public financial management 11.3 Public administration Business enabling 11.3 environment 7.5 Information and communication 7.5 11.3 technology 9.4 Financial stability Source: Development Policy Actions Database, World Bank, Washington, DC (accessed April, 25, 2023), Note: “Other” includes data development and capacity building, social protection, enterprise development, fiscal policy, energy, financial infrastructure and access, trade, and nutrition and food security. Data are based on six development policy operations and 53 prior actions. 24 Table C.1. World Bank Investment Projects Approved in Georgia, Fiscal Years 2014–23 Bank Total Outcome Performance Approval Closing Commitment Rating Rating Project Name Status FY FY (US$, millions) (IEG) (IEG) Irrigation and Land Market Active 2014 2024 50 — — Development Project Transmission Grid Strengthening Active 2014 2023 60 — — Project Second Regional and Municipal Active 2015 2025 30 — — Infrastructure Development Project Third Secondary and Local Roads Closed 2015 2022 75 MS MS Project East-West Highway Corridor Active 2016 2024 140 — — Improvement Secondary Road Asset Management Active 2016 2023 40 — — Project Third Regional Development Project Active 2016 2025 60 — — Georgia National Innovation Active 2016 2023 40 — — Ecosystem Project Second Regional Development Closed 2016 n.a. 9 — — Project Additional Financing East-West Highway Corridor Active 2018 n.a. 20 — — Improvement Project Additional Financing Georgia I2Q—Innovation, Inclusion, Active 2019 2026 103 — — and Quality Additional Financing for the Second Active 2020 n.a. 41 — — Regional and Municipal Infrastructure Development Project Energy Supply Reliability and Active 2019 2025 71 — — Financial Recovery Log-In Georgia Active 2021 2026 40 — — Energy Supply Reliability and Active 2019 2025 50 — — Financial Recovery Guarantee Additional Financing for Irrigation Active 2020 n.a. 20 — — and Land Market Development Project Kakheti Connectivity Improvement Active 2022 2028 109 — — Project Georgia Emergency COVID-19 Active 2020 2023 80 — — Response Project Georgia Relief and Recovery for Active 2021 2026 103 — — Micro, Small, and Medium Enterprises 25 Bank Total Outcome Performance Approval Closing Commitment Rating Rating Project Name Status FY FY (US$, millions) (IEG) (IEG) Georgia Human Capital Program Active 2022 2029 400 — — AF for Georgia Emergency COVID-19 Active 2021 n.a. 35 — — Response Project Source: Business Intelligence database, World Bank, Washington, DC (accessed October, 10, 2022), Note: AF = additional financing; FY = fiscal year; IEG = Independent Evaluation Group; MS = moderately satisfactory; n.a. = not applicable.; — = not available. Table C.2. World Bank Development Policy Operations in Georgia, Fiscal Years 2014–23 Bank Total Outcome Performance Approval Closing Commitment Rating Rating Project Name Status FY FY (US$, millions) (IEG) (IEG) Georgia Competitiveness and Closed 2014 2015 93 MS MS Growth DPO3 Inclusive Growth DPO Closed 2015 2016 60 MU MS Private Sector Competitiveness Closed 2015 2016 60 S S DPO1 Private Sector Competitiveness Closed 2018 2019 50 S S DPO2 Georgia Inclusive Growth DPO2 Closed 2017 2018 50 MU MS Economic Management and Closed 2020 2021 50 MS MS Competitiveness DPO Economic Management and Active 2020 n.a. 49 – – Competitiveness DPO: COVID19 Supplemental Financing Source: Business Intelligence database, World Bank, Washington, DC (accessed October, 10, 2022 ), Note: DPO = development policy operation; FY = fiscal year; IEG = Independent Evaluation Group; MS = moderately satisfactory; MU = moderately unsatisfactory; n.a. = not applicable.; S = satisfactory; – = not available. World Bank Advisory Services and Analytics Between FY14 and FY23, the World Bank delivered 83 nonlending activities. Although the largest share of advisory services and analytics focused on public administration (figure C.3), the two largest projects in terms of total expenditure were in the financial and education sectors. Strengthening Teacher Quality in Vocational Education and Training (VET) was a US$1.9 million advisory project to support the development of industry-led skills for vocational education and training graduates and strengthen the quality of professional development for their teachers. Georgia Financial Sector Deepening and Inclusion (P159890)was another US$1.9 million advisory project aiming to deepen and diversify the financial sector through the implementation of various reforms. The number of advisory services and analytics under implementation 26 decreased from 70 to 27 in the period FY19–22, with only 14 projects starting during the COVID-19 pandemic. Public administration was the largest sector across periods, representing 26 percent in FY14–18 and 37 percent in FY19–20. Figure C.3. World Bank Advisory Services and Analytics, Number of Projects by Sector, Fiscal Years 2014–23 60 50 Energy and extractives 40 Financial Projects (no.) Industry, trade, and services 30 Public administration Social protection 20 Transportation Other 10 0 2014–18 2019–23 Source: Business Intelligence database, World Bank, Washington, DC (accessed October, 10, 2022 ), Note: “Other” includes agriculture; fishing and forestry; education; health; information and communication technology; and water, sanitation, and waste management. International Finance Corporation During the evaluation period, the International Finance Corporation committed financing for 19 new operations in the amount of US$79 million (figure C.4). More than half of the investment projects were concentrated in the finance and insurance sector, and the largest project was in the electric power sector. The International Finance Corporation supported Georgia with 13 advisory projects, 9 of which were in infrastructure and financial institutions. Most advisory projects were implemented before the COVID-19 pandemic. 27 Figure C.4. Total Commitment of International Finance Corporation Investment Projects to Georgia by Sector, Fiscal Years 2014–23 300 12 Total commitment (US$, millions) 250 10 200 8 Projects (no.) 150 6 100 4 50 2 0 0 Accommodation and Electric power Food and beverages Nonmetallic mineral tourism services product manufacturing Sector Projects Total commitment Source: Business Intelligence database, World Bank, Washington, DC (accessed October, 10, 2022), Multilateral Investment Guarantee Agency During the review period, the Multilateral Investment Guarantee Agency issued guarantees for two projects: Adjaristsqali Hydro Project for US$63 million, and ProCredit Mandatory Reserves Coverage for €47.5 million. In 2018, the Multilateral Investment Guarantee Agency increased the amount of the ProCredit Group Central Bank Mandatory Reserves Coverage issued in 2010, covering the risk of expropriation of funds for mandatory reserves held by the subsidiary in the central bank. 28 Appendix D. Development Partners The World Bank was the third largest donor to Georgia, accounting for 13 percent of the total committed amount. The Asian Development Bank accounted for 20 percent of the total commitments during the evaluation period. Table D.1 provides the list of the top 10 development partners and their commitments during the evaluation period (calendar years 2014–21). The European Bank for Reconstruction and Development and the Asian Development Bank made loans to support the construction of the Adjaristsqali Hydro Project that started in 2014 and was completed in 2020. The International Finance Corporation provided a loan of US$70 million and an equity of US$31 million. Additionally, the Multilateral Investment Guarantee Agency provided a guarantee to cover an equity investment. Table D.2 shows that the top sectors of development partner activities in Georgia were road transport and formal sector financial intermediaries, accounting for one-quarter of total commitment. Table D.1. Top 10 Development Partners Active in Georgia by Commitment Amount of Official Development Assistance, 2014—21 Commitment Proportion of Amount Total Organization (US$, Commitment No. Organization Name Type millions) (%) 1 Asian Development Bank Multilateral 2,953 20 2 European Bank for Reconstruction and Development Multilateral 2,349 16 3 World Bank Group Multilateral 1,865 13 4 KfW—Germany Government 1,215 8 5 European Commission Other public 1,198 8 sector 6 European Investment Bank Other public 977 7 sector 7 Agence Française de Développement—France Government 712 5 8 US Agency for International Development—United Government 435 3 States 9 Asian Infrastructure Investment Bank Multilateral 364 3 10 Overseas Private Investment Corporation—United Government 214 1 States 237 Total 14,456 100 Sources: Creditor Reporting System Aid Activity (database), Organisation for Economic Co-operation and Development, (accessed January 9, 2023), https://stats.oecd.org/Index.aspx?DataSetCode=crs1; Independent Evaluation Group. 29 Table D.2. Top Sectors of Development Partner Activities in Georgia by Commitment Amount of Official Development Assistance, 2014—21 Proportion of Commitment Total Amount Commitment No. Sector Category (US$, millions) (%) 1 Road transport 1,976 14 2 Formal sector financial intermediaries 1,691 12 3 Public sector policy and administrative management 828 6 4 Energy policy and administrative management 711 5 5 Urban development and management 657 5 6 Electric power transmission and distribution (centralized grids) 480 3 7 Water supply and sanitation—large systems 478 3 8 Social protection 442 3 9 Public finance management 326 2 10 Civilian peace building and conflict prevention and resolution 257 2 195 Total 14,456 100 Sources: Creditor Reporting System Aid Activity (database), Organisation for Economic Co-operation and Development, (accessed January 9, 2023), https://stats.oecd.org/Index.aspx?DataSetCode=crs1; Independent Evaluation Group. Note: Sectors are based on OECD CRS three-digit purpose code. CRS = Creditor Reporting System; OECD = Organisation for Economic Co-operation and Development. 30