Approach Paper An Evaluation of World Bank Group Support to Jobs and Labor Market Reform through International Development Association Financing March 2, 2023 1. Background and Context 1.1 The International Development Association (IDA) has included jobs as a special theme since the 17th Replenishment of IDA (IDA17) in 2014, when it explicitly recognized the role played by labor markets in intermediating between growth and inclusion. This acknowledgment of jobs marked a shift in IDA’s inclusive growth strategy. Before the IDA17 strategy paper, IDA emphasized growth and the use of social safety nets to mitigate the effects of poverty. Beginning in 2014, however, jobs became more central to IDA’s strategy for inclusive growth and for achieving the twin goals. IDA17, the 18th Replenishment of IDA, and the 19th Replenishment of IDA established specific policy commitments and results indicators under the jobs-related special theme. At the same time, the World Bank Group expanded and deepened its attention to jobs, resulting in an increasingly multidimensional jobs agenda characterized by a growing body of lending, technical assistance and diagnostics, and a strong focus on IDA-eligible countries, including through use of the Country Private Sector Diagnostic and IDA’s private sector window. 1.2 This evaluation will assess IDA’s support for jobs-related objectives over fiscal years (FY)14–22, the period covering three IDA replenishments during which jobs became an IDA special theme (IDA17, the 18th Replenishment of IDA, and the 19th Replenishment of IDA). The objectives of this assessment are to interrogate the contribution of IDA’s Bank Group financing to improving outcomes related to more, better paying, and more inclusive jobs; the role of IDA’s jobs strategy at the corporate, country, and operational levels in this context; and the analytical underpinnings of jobs- related interventions. The evaluation will provide lessons and recommendations to inform the design of the Bank Group’s future multidimensional jobs support and enhance IDA’s effectiveness in this space based on eight years of strategic, diagnostic, and operational experience.1 1.3 For the purposes of this evaluation, jobs are defined broadly as including both wage employment and self-employment in the formal and informal sectors (see box 1.1). Relevant jobs-related objectives are more and better jobs, with better jobs defined as those that are better paid and more inclusive (of women, youth, the bottom 40 percent of the population’s income distribution, and individuals from marginalized groups). To 1 keep the evaluation tractable, it will focus on IDA-supported interventions that directly address constraints on more and better jobs, rather than on projects that support broad- based growth or structural transformation. World Bank support to jobs in middle- income countries could potentially be included in the Independent Evaluation Group’s future work program. Box 1.1. Defining Jobs and Jobs Objectives For the purposes of this evaluation, jobs is defined to include both wage employment and self- employment in both the formal and informal sectors. This evaluation recognizes that the International Development Association’s support for jobs embodies several distinct objectives, including the pursuit of more jobs and better jobs, with better jobs characterized by higher development payoff (see box 1.2; World Bank 2012), improved productivity and incomes, and better working conditions. For ease of reference, jobs is used interchangeably with jobs agenda and jobs or jobs-related objectives. For clarity, this evaluation adopts the following International Labour Organization definitions in the jobs space: • Employment: all those of working age who, during a short reference period, were engaged in any activity to produce goods or provide service for pay or profit • Job: a set of tasks and duties performed, or meant to be performed, by one person, including for an employer or in self-employment • Unemployment: all those of working age who were not in employment, carried out activities to seek employment during a specified recent period, and were currently available to take up employment given a job opportunity • Underemployment: all persons in employment who, during a short reference period, wanted to work additional hours, had working time in all jobs less than a specified hours threshold, and were available to work additional hours given an opportunity for more work • Labor force participation: a measure of the proportion of a country’s working-age population that engages actively in the labor market, either by working or looking for work • Not in education, employment, or training rate: the percentage of the population of a given age group and sex that is not employed and not involved in further education or training • Informal employment: jobs in unregistered and/or small-scale, private, unincorporated enterprises that produce goods or services meant for sale or barter Source: ILO 2016. 1.4 The Bank Group and IDA corporate jobs strategy was informed by World Development Report 2013: Jobs (World Bank 2012; box 1.2), which first highlighted the multidimensional nature of jobs, presenting the idea that jobs are more than a by- product of economic growth. There are multiple jobs objectives—higher job growth is 2 one, but the development dividend from jobs is also critical. This dividend can take many forms: Jobs help poor people work their way out of poverty and provide higher earnings and economic empowerment to women, who subsequently invest more in their children. Jobs lead to improved efficiency as workers get better at what they do and as more productive jobs appear. Jobs improve social cohesion by bringing people from different backgrounds together and helping reduce conflict (Filmer et al. 2014; Urdal 2006). Although the World Development Report on jobs noted that high unemployment and unmet job expectations among youth are immediate concerns, it emphasized that growth rarely occurs without also creating jobs in developing economies. Unemployment rates can be low, particularly where dependence on agriculture is high or the private sector is largely informal and characterized by self-employment, but earnings are also low. Therefore, the number of jobs is not all that matters: jobs with high development payoffs—not just more jobs but better jobs—are needed. Development policies that support economic transformation are critical to these payoffs. 1.5 It is obvious that improved jobs outcomes are critical from a development policy perspective, given the dominance of labor income over other sources of household income. Both income and consumption inequality are largely driven by inequality in labor earnings (Fields 2011). Underemployment, not unemployment, is the main challenge for many low-income countries (Merotto, Weber, and Aterido 2018). Underemployment and low labor productivity are related in most IDA client countries where poor people, in particular, are not without work but are unable to earn sufficient income to lift themselves out of poverty. On average, 40 percent of employed workers work fewer than 35 hours per week in low- and middle-income countries. However, approximately one-third of employed people in low-income countries work more than 45 hours a week, indicating that hourly productivity and income are low, so survival calls for longer working hours. Demographic pressures may dictate that more jobs will be needed, but for poverty and inequality to fall, labor income—whether from wages or self-employment—must also increase. 1.6 The recent finding of the World Bank (2022b) that poverty and shared prosperity indicators have deteriorated in IDA-eligible countries after the COVID-19 pandemic underscores the criticality of pursuing jobs objectives more deliberately. Without sufficient social safety nets and the fiscal space to cushion workers from shocks, the bottom 40 percent of the population’s income distribution are highly vulnerable to job loss caused by crises. Jobs and higher earnings from jobs, social protection to support workers through transitions, the elimination of gender gaps in employment, voice and protection of marginalized groups, and support for core labor standards are critical pathways to better social, economic, and development outcomes and greater household resilience to adverse events (ILO 1999; World Bank 2012). 3 Box 1.2. World Development Report 2013 on Jobs The World Development Report 2013: Jobs focused on both policy solutions to achieve jobs- related objectives and, based on global experience, on identifying which jobs have the most development payoff. The report identified three types of developmentally desirable jobs: (i) jobs that support living standards—including jobs for women, youth, poor people, and the marginalized—and that do not come at the expense of others; (ii) jobs that support social cohesion by promoting a sense of fairness, connectedness, and social identity; and (iii) jobs that support productivity, are environmentally benign, encourage participation in global markets, and support and are supported by functional cities. A country’s level of development, institutional strengths, endowments, and demography define where the development payoff from jobs is greatest. The jobs agenda in one country may thus be different from that in another country. The challenges facing countries as they move along the development path vary. Agrarian economies may need to focus primarily on raising agricultural productivity to support better incomes for a large share of the population. As countries urbanize, greater focus is needed on functional and economically dynamic cities that can absorb workers shifting out of agriculture and rural areas. Job opportunities for women tend to emerge in light manufacturing and have significant impact on intrahousehold resource allocation. This phase would ideally see greater access to global markets and improving productivity. As cities become congested, jobs that are environmentally benign become more desirable. As economies formalize, skills premiums and wage gaps may emerge, and there may be tensions among the objectives of livelihood, social cohesion, and productivity. Agglomeration and scale economies may be a challenge in small island nations that may need to rely on out- migration and remittances. Resource-rich economies may need to focus on export diversification to support better jobs, and conflict-affected countries may need jobs primarily to support social cohesion and livelihoods. Multiple contexts may exist in one country at a time—for example, an urbanizing economy may have high youth unemployment and may be attempting to formalize. The report advances a three-stage approach to help governments address jobs-specific, jobs- related obstacles: (i) addressing policy fundamentals—including but not limited to macroeconomic stability, an enabling business environment, investments in human capital, and the rule of law—to promote growth and private sector development and thereby facilitate job creation; (ii) developing well-designed public policies to ensure that growth translates into employment opportunities; and (iii) strategically identifying and prioritizing which jobs interventions would do the most for development, given the specific country context. Source: World Bank 2012. 2. Evaluation Context: International Development Association’s Strategic Shift in Response to Rising Inequality 2.1 The recognition that the labor market is a key intermediary between growth and inclusion and that jobs are the main pathway out of poverty and inequality marked a shift in strategy between the 16th Replenishment of IDA and IDA17 (table 2.1). IDA17 first explicitly identified jobs as a subtheme under inclusive growth, drawing on the conclusions of the World Bank (2012) and noting that “jobs are the most important 4 determinant of living standards around the world” (World Bank 2012, 76). Macroeconomic and microeconomic fundamentals were critical because private sector development was essential to jobs outcomes. Furthermore, based on an analysis of country-specific constraints to employment through well-developed diagnostics, specific policy interventions could be identified to support jobs with greater development outcomes (World Bank 2012). This framework was reflected in how IDA’s jobs strategy evolved and in the policy commitments and results indicators used under IDA’s jobs and economic transformation special theme. Table 2.1. International Development Association and Jobs IDA17 IDA18 IDA19 (July 1, 2014, to June 30, 2017) (July 1, 2017, to June 30, 2020) (July 1, 2020, to June 30, 2023) Policies before IDA17 centered on IDA18 put job creation in the context IDA19 formally presented the new growth with social safety nets. IDA16 of structural transformation, focusing strategic framework of jobs and special themes included gender and on the role of the private sector in economic transformation, placing the recognized the need to raise female creating good jobs in labor-intensive private sector at the heart of job labor force participation rates. sectors (partly drawing on World creation. IDA17 was the first document to Bank 2012). The focus was on jobs measurement, explicitly identify jobs as a subtheme. A continued focus on policy skills and employability, and Policy commitments under IDA commitments was placed on connecting workers to jobs. focused on jobs diagnostics. diagnostics and on impact The Jobs CCSA was created in 2014. evaluations to support improved M&E. Sources: World Bank 2012, 2014, 2017a, 2020a. Note: CCSA = cross-cutting solutions area; IDA = International Development Association; IDA16 = 16th Replenishment of IDA; IDA17 = 17th Replenishment of IDA; IDA18 = 18th Replenishment of IDA; IDA19 = 19th Replenishment of IDA; M&E = monitoring and evaluation. 2.2 The adoption of jobs-related objectives in IDA17 was largely motivated by the view that rising inequality was undermining progress in poverty reduction and other Sustainable Development Goals (World Bank 2014). Although poverty reduction in IDA- eligible countries between 2000 and 2014 had been unprecedented, extreme poverty remained widespread. Where growth was robust, it did not always contribute to the creation of higher labor productivity and better paying jobs for the poorer segments of society (Dollar and Kraay 2002). Even now, less than half of women globally hold paid jobs, compared with almost four-fifths of men, and women’s jobs tend to be lower paid than men’s jobs. 2.3 IDA’s first jobs policy commitment was on diagnostics—15 jobs diagnostics were to be completed under the supervision of the newly created Jobs Group in IDA-eligible countries, and these diagnostics were intended to inform country strategies and guide project-level interventions (table 2.2). This evaluation will therefore also look at strategic coherence and analytical grounding of IDA-supported Country Partnership Frameworks, strategies, and operational activities. IDA established a series of results indicators as part of its Results Measurement System (described in table 2.2) to track the 5 impact of policy commitments under the jobs-related special theme over the eight years since 2014. Part of this evaluation will assess the relevance and adequacy of these indicators to determine the impact of IDA operations and policy commitments. 2.4 Informality is a significant challenge to better jobs for many people in emerging markets and developing economies, where informal jobs have been estimated to account for 70 percent of employment and about one-third of the gross domestic product (Ohnsorge and Yu 2022). Informal workers are more likely to lose their jobs or suffer severe income losses during crises such as the COVID-19 pandemic, and they tend to be largely excluded from social safety nets and have been associated with broader development challenges (World Bank 2019a, 2020b). Regardless of the nature and causes of informality, countries with larger informal sectors tend to have less access to finance for the private sector, lower labor productivity, slower physical and human capital accumulation, and smaller fiscal resources (Docquier, Müller, and Naval 2017; La Porta and Shleifer 2014). Informality is also associated with higher income inequality and poverty and less progress toward the Sustainable Development Goals (Chong and Gradstein 2007; Loayza, Servén, and Sugawara 2010). The informal sector is on average less productive than the formal sector because it tends to employ more low-skilled workers; has more restricted access to funding, services, and markets; and lacks economies of scale (Amaral and Quintin 2006; Loayza 2018). As a result, informal workers have lower savings and assets buffers and significantly less resilience and access to coping mechanisms than formal workers. Given the importance of the informal sector and the potentially critical link between formality and more and better jobs, this evaluation will pay special attention to how IDA’s jobs-related interventions approach informality. 2.5 IDA has maintained a focus on women and youth in its jobs strategies (World Bank 2014, 2017a, 2020a). Youth unemployment and underemployment are serious challenges in many developing economies, and significant gender gaps remain for both workforce participation rates and labor income (ILO 2017; World Bank 2014). IDA results indicators pertaining to gender and youth seek to track progress in implementing IDA’s policy commitments on jobs (box 2.1). Women are also more likely than men to be in the informal sector. This evaluation will therefore include a focus on IDA’s contribution to achieving jobs objectives for women and youth. Box 2.1. Focus on Gender and Youth in International Development Association Strategies The evolution of International Development Association (IDA) strategies in the 17th, 18th, and 19th Replenishments of IDA (IDA19) mirrors elements of the 2016–23 World Bank Group gender 6 strategy, which identified the lack of gender equity in employment as a costly challenge and noted the need for more and better jobs for women in its second pillar. On gender, special theme 2 in the 17th Replenishment of IDA focused on strengthening gender policy dialogue and integrating gender diagnostics into IDA operations and was captured by two Results Measurement System indicators: employment to population ratio for women and number of IDA countries with discriminatory laws against women. The 18th Replenishment of IDA introduced new special themes that promoted competitiveness and jobs—particularly for women and youth—by supplementing country engagements with operational guidelines that integrated interventions for youth and women. Under the gender theme, female labor force participation rates were to be supported by strengthened connectivity and digital skills training. IDA19 recognized the need for sustained efforts to realize benefits across the full spectrum of jobs, including by raising the quality of jobs in the informal sector and ensuring inclusive access to earnings opportunities. With regard to gender development, the commitment was to help close the gaps in employment and leadership between women and men. At least 60 percent of IDA19 operations for digital skills development was to support women’s access to higher- productivity jobs, including online work, and at least 30 percent of IDA19 infrastructure operations was to include actions to create employment opportunities for women in medium- and high-skilled jobs in these sectors. Sources: World Bank 2014, 2017a, 2020a. 2.6 The 18th Replenishment of IDA also put job creation in the context of structural shifts in the economy, paving the way for a new framework in “IDA 19: Jobs and Economic Transformation” (World Bank 2019c). Policies that facilitated connectivity to market opportunities and built the capabilities of firms and individuals to benefit from these opportunities would support the creation of more and better jobs. The private sector’s role at the heart of creating diverse formal sector wage employment was explicitly recognized. Table 2.2. International Development Association Policy Commitments and Results Measurement System Indicators Replenishment Policy Commitments RMS Indicators IDA17 Establish a more evidence-based country engagement • Median income growth for bottom model: 40% • Roll out a jobs diagnostic tool in at least 15 IDA- • Employment ratios eligible countries (at least 5 FCS countries), plus key • Youth employment strategic priorities on jobs. • Women’s employment ratios Address drivers of fragility and conflict: • Undertake analytical work on job creation in FCS countries. IDA18 Use analytics to inform current operations: • Female labor force ratios • Conduct 10 global value chain analyses to • Youth employment ratios understand how they create jobs, as well as spatial mapping to assess jobs dynamics. 7 Replenishment Policy Commitments RMS Indicators Use learning and measurement to inform future • Beneficiaries of job-focused operations: intervention in IDA-eligible • Conduct a systematic impact analysis of small and countries medium enterprises and entrepreneurship in IDA- eligible countries to understand outcomes for women and youth. IDA19 Create and connect to markets: • Female labor force ratios • Implement interventions in 10–15 countries to help • Youth employment ratio remove bottlenecks in sectors with high potential • Beneficiaries of job-focused for private sector–led job creation. intervention in IDA-eligible Build capacities and connect workers to jobs: countries • Support at least 15 IDA-eligible countries with programs to improve skills and employability toward more and higher quality jobs. Focus on gender and development: • Ensure that at least 60 percent of IDA19 financing operations for digital skills development will support women’s access to higher productivity jobs, including online work. Develop cross-cutting commitments: • Conduct 20 pilot programs in economic transformation in IDA-eligible countries to estimate indirect or induced jobs. • Have IFC track direct jobs and estimates of indirect jobs associated with all IFC PSW investments, and ensure that jobs reporting, where feasible, will be disaggregated by the poorest quintile, gender, FCS, disability, and youth. Sources: World Bank 2017b, 2018, 2019c. Note: FCS = fragile and conflict-affected situation; IDA = International Development Association; IDA17 = 17th Replenishment of IDA; IDA18 = 18th Replenishment of IDA; IDA19 = 19th Replenishment of IDA; IFC = International Finance Corporation; PSW = private sector window; RMS = Results Measurement System. 2.7 As the 20th Replenishment of IDA kicks off, with continued attention to jobs as a pathway to inclusive recovery, this evaluation will have a wide audience and is timely. The evaluation is expected to generate findings useful to IDA, the IDA executive board, and senior management as strategy-setters and to clients and the World Bank task teams. The evaluation’s findings could also provide inputs to the IDA Mid-Term Review and the Mid-Term Review of the Jobs Umbrella Multi-Donor Trust Fund, as well as to ongoing advisory initiatives at the World Bank and the International Finance Corporation (IFC) that are directly relevant to achieving and refining jobs objectives— for example, IFC’s gender and inclusion work and the Better Work collaboration between IFC and the International Labour Organization. 8 3. Drivers of More and Better Jobs: Analytical Underpinnings 3.1 IDA’s approach to jobs is grounded in the sense that not enough high-quality private sector jobs are being created in low-income countries. Although often stated as a response to the need for “more and better jobs,” this approach effectively embodies more than one objective, which suggests multiple dimensions for pursuing the jobs agenda in IDA-eligible countries, as described in figure 3.1. Demand for labor from the private sector can potentially be the largest source of formal wage employment in an economy and is therefore critical to achieving IDA’s jobs objectives. The Bank Group’s jobs diagnostics find that almost 80 percent of labor productivity growth in low-income countries came from the reallocation of labor from lower-productivity agriculture into relatively higher-productivity services and industry (Merotto, Weber, and Aterido 2018). However, low-income economies are usually unable to absorb all the workers released from agriculture into higher-productivity and higher-paying wage employment. This suggests that labor market flexibility and worker employability are important intervention channels to support jobs objectives. Targeted interventions that facilitate transitions from low-paying to higher-paying employment, from informal to formal employment, and from self-employment to wage employment are critical for inclusive economic transformation (Fields 2011; World Bank 2014, 2017a, 2020a). Reflecting these various channels of impact, this evaluation will focus on IDA support to policies and operations that affect labor demand, labor supply, and labor market flexibility. 9 Figure 3.1. Channels for Achieving More and Better Jobs Source: Independent Evaluation Group staff analysis. Note: MSME = micro, small, and medium enterprise. 4. Evaluation Scope and Methods Evaluation Scope 4.1 Based on the received literature, the World Bank Development Report 2013 (World Bank 2012), early findings of World Bank jobs diagnostics, IDA jobs strategies, and consultations with experts, this evaluation identifies three interrelated channels through which IDA support for jobs objectives is pursued: labor demand (including efforts to expand the demand of private sector firms for workers and create the conditions for reliable and adequately remunerated self-employment); labor supply (including efforts to improve marketable skills of workers, expand women’s labor force participation, and support youth employment); and labor market flexibility (including policies to facilitate the movement of workers from lower-productivity to higher-productivity activities within sectors and from lower-value-added to higher-value-added sectors) and geographic mobility where it is identified as a significant constraint in the jobs diagnostic. All three channels play a role in achieving IDA’s objectives in the jobs space (see figure 4.1). 10 Figure 4.1. Evaluation Scope Source: Independent Evaluation Group staff analysis. Note: IDA = International Development Association; PSD = private sector development. 4.2 Although the structural change associated with economic transformation is linked to jobs and growth, the scope of this evaluation is limited to the three main pathways or channels for achieving IDA jobs objectives. It will therefore not evaluate the implementation and effectiveness of the jobs and economic transformation agenda, given its broad purview. Figure 4.1 presents the scope of this evaluation, with the light gray rectangles seen as relevant to the broader development agenda but outside the remit of this evaluation. 4.3 Given the proposed scope, the evaluation will address the following questions: 1. What was the impact of the Bank Group’s IDA-financed support for jobs and labor market reforms? 2. Were jobs-related interventions based on sound analysis and adapted to country context and priorities? 3. To what extent is IDA’s corporate strategy on jobs clear, coherent, and grounded in sound analytics and lessons learned from country engagement? 11 Evaluation Methods 4.4 The evaluation will have a three-tier structure: corporate level, country level, and intervention level. The evaluation team will undertake a review of IDA’s strategic positioning and a stock take of commitments related to jobs to assess clarity and coherence. This stage will include a review of corporate strategies and relevant policy papers and background material, starting with IDA17. The adequacy of analytical foundations will be assessed, drawing on a structured literature review. Case studies will provide the platform for assessing how IDA’s jobs-related objectives are pursued at the country level, including how they inform policy dialogue and country portfolios. The relevance and impact of individual interventions (lending and nonlending) will be assessed at both the portfolio level (to the extent possible, given data constraints) and in the context of country case studies. Portfolio Identification and Analysis 4.5 The evaluation team will pursue a two-phase approach to identify the relevant IDA portfolio. First, we will consider the universe of lending projects identified by using World Bank–assigned jobs and labor markets theme codes (World Bank 2016b). Appendix B presents definitions and details of the specific theme codes. This process will extract all IDA projects approved during FY14–22 mapped to the level 2 (22) or 3 (221, 222, 223) jobs-related theme codes, plus labor market theme codes mapped to level 2 (66) or 3 (661, 662, 663). The IDA jobs and labor market–themed portfolio over the evaluation period (all approved projects during FY14–22) will be filtered further and categorized using text analytics to drill down on specific subthemes—for example, gender, youth employment, working conditions, and COVID-19 response. Appendix C presents some basic portfolio data. 4.6 The evaluation will cover all financing instruments. The approach described initially identified 599 investment project financing projects and Program-for-Results operations (167 of which are closed projects). The Operations Policy and Country Services prior action database will be used to identify development policy operations with jobs-related prior actions, using both thematic codes and a relevant word search. Theme codes alone identified 73 operations (54 of which are closed). Country Case Studies 4.7 To ensure a balanced and representative sample, we will follow a three-stage approach to identify countries for more in-depth study: 1. Country context 12 a. Female labor force participation rate: top 15 countries with substantial increases in the female labor force participation rate over the evaluation period and 15 countries that show negligible progress in the female labor force participation rate b. Youth employment ratio: top 15 countries with substantial increases in the youth employment rate over the evaluation period and 15 countries with negligible progress over the evaluation period 2. Diagnostic work: IDA-eligible countries (identified in step 1) reviewed to determine which ones have had a jobs diagnostic; the existence of a Country Private Sector Diagnostic also viewed favorably in country selection 3. Existence of support to jobs: countries that have a critical mass of jobs-related operations, including private sector window–financed investments 4.8 Using these criteria, we identified 13 countries that reflect the regional breakdown of IDA’s lending and are a mix of low-income, fragile, and lower-middle- income countries. Six of these countries will be subject to a more comprehensive deep dive assessment of Country Partnership Strategies and Country Partnership Frameworks, diagnostics and analytics, and the jobs and labor market portfolio (table 4.1). The evaluation team will draw on this material to assess how IDA jobs commitments and relevant analytical work informed strategy and were reflected in project design, with the aim of assessing the impact of jobs-related interventions. 4.9 The country case studies are a critical component of the evaluation and will be structured consistently to ensure comparability across countries. The following elements will be reflected in case study design: • Interrogation of quality and availability of relevant analytical work • Identification of gaps in relevant jobs data • Assessment of how country strategies and project and intervention design draw on analytical work and IDA corporate strategy • Evidence of learning from experience within and across countries • Evaluation of collaboration across the World Bank; between the World Bank, IFC, and Multilateral Investment Guarantee Agency (as a participant in one of four private sector window facilities); and with development partners, authorities, and key stakeholders 13 • Expected and actual job effects of IDA private sector window projects (The underlying model of Anticipated Impact Measurement and Monitoring will not be assessed.) • Outcomes and impacts of jobs-related interventions • Lessons learned 4.10 The deep dives will also review private sector window projects through a jobs lens and assess the role of the Country Private Sector Diagnostic in identifying sectors with jobs potential consistent with the IDA objectives. One deep-dive case study will involve a country mission to further refine the approach, which will then be applied to the other five cases, for which in-country interviews with relevant stakeholders will be conducted virtually. The remaining seven country case studies will be based on desk review only. Table 4.1. Country Case Study Selection Bank Youth Portfolio PSW FLFP: Change Unemployment: (Lending Jobs Projects (2009–19) or Change (2009–19) or with Country Diagnostic (no.) CPSD Level (2019) Level (2019) Themes) FCV Deep dive Bangladesh Yes 3 Yes Change: 3.2% .. 23 No increase Côte d’Ivoire Yes 2 Yes .. Change: −3.9% 12 No Ghana Yes 1 Yes .. Change: −1.5% 14 No Kenya Yes 3 Yes Level: 72% .. 13 No Kyrgyz Republic Yes 3 Yes .. Change: −1.6% 12 No Sierra Leone Yes 3 No .. .. 8 Yes Desk review Haiti Yes 2 Yes .. Level: 32% 4 Yes Honduras Yes 0 Yes .. .. 8 No Malawi No 2 Yes .. .. 8 No Nepal Yes 3 Yes .. .. 16 No Rwanda Yes 3 Yes .. .. 14 No Tanzania Yes 4 No .. .. 14 No Democratic Yes 3 Yes .. .. 13 Yes Republic of Congo Source: Independent Evaluation Group analysis and calculations using World Bank data. Note: .. = negligible; CPSD = Country Private Sector Diagnostic; FCV = fragility, conflict, and violence; FLFP = female labor force participation; PSW = private sector window. 14 Limitations 4.11 The evaluation is subject to noteworthy limitations. First, portfolio identification is challenging because few operations focus exclusively on jobs, while many operations mention jobs, even when their content is only peripherally jobs related. Although jobs and labor market theme codes are rigorously assigned and validated, there is potential for both inclusion and exclusion error. This risk can be mitigated to a degree in deep- dive country case studies, given the ability to look more closely at the full portfolio. Second, measurement of job impact (both direct and indirect) is methodologically challenging, making the assessment of jobs created imprecise at best. Third, employment data are generally incomplete or of poor quality in lower-income countries, further complicating the assessment of impact. Finally, there are tensions and trade-offs among IDA’s various jobs objectives that are not often explicit. For example, raising productivity can often improve compensation (that is, create better jobs), but it may reduce the number of jobs available. 5. Quality Assurance Process 5.1 The evaluation will follow standard Independent Evaluation Group (IEG) quality assurance processes, including external peer review. The Approach Paper and final report will be peer reviewed by Dr. Gordon Betcherman (professor, Faculty of Social Sciences, School of International Development and Global Studies, University of Ottawa); Dr. Abebe Shimeles (director of research, African Economic Research Consortium, Nairobi, Kenya); and Corinne Deléchat (division chief, Regional Office for Asia and the Pacific, International Monetary Fund). 6. Timeline, Team, and Resources 6.1 The draft evaluation report is expected to be ready for internal IEG review by the end of July 2023 and for e-submission to the Policy and Operations Unit by early September 2023.2 This timeline would allow this evaluation to provide input to the 20th Replenishment of IDA Mid-Term Review that is expected to be held in late fall 2023. The evaluation team includes Harsh Anuj (data scientist); Soniya Carvalho (lead evaluation officer); Dung Thi Kim Chu (program assistant); Gary Fields (consultant, senior technical adviser); Pablo Fleiss (technical adviser); Lars Johannes (senior economist); Jennifer Keller (senior economist); Doruk Yarin Kiroglu (portfolio analyst); Denita Pious (research analyst, consultant); Andrea Rojas (consultant, evaluator); Rashmi Shankar (task team leader, lead economist); Sandra Jane Wark (consultant, evaluator); Andrei Wong (portfolio analyst, consultant); and Marwane Zouaidi (extended term consultant and project officer). Estelle Rosine Raimondo (methods adviser) will advise on methodological aspects of the evaluation. The evaluation will be produced under the 15 direction of Jeffrey Allen Chelsky (manager, Economic Management and Country Programs Unit, IEG) and Carmen Nonay (acting director, Human Development and Economic Management Department, IEG). The evaluation is projected to cost $750,000, including dissemination 1Given the breadth and complexity of the jobs topic, no single evaluation could do credit to the full extent of the World Bank Group’s contribution to its clients. For this reason, we have narrowed the scope of the evaluation to support through IDA financing, making non –private sector window and International Finance Corporation advisory projects outside our scope of evaluation. 2 Policy and Operations Department of the Corporate Secretariat. 16 Bibliography Amaral, P., and E. Quintin. 2006. “A Competitive Model of the Informal Sector.” Journal of Monetary Economics 53 (7): 1541–53. Aterido, Reyes, Mary Hallward-Driemeier, and Carmen Pagés. 2007. “Investment Climate and Employment Growth: The Impact of Access to Finance, Corruption, and Regulations across Firms.” IZA Discussion Paper 3138, Institute for the Study of Labor, Bonn, Germany. https://docs.iza.org/dp3138.pdf. Chong, A., and M. Gradstein. 2007. “Inequality and Informality.” Journal of Public Economics 91 (1–2): 159–79. Docquier, F., T. Müller, and J. Naval. 2017. “Informality and Long -Run Growth.” The Scandinavian Journal of Economics 119 (4): 1040–85. Dollar, David, and Aart Kraay. 2002. “Growth Is Good for the Poor.” Journal of Economic Growth 7 (3): 195–225. http://www.jstor.org/stable/40216063. Fields, G. S. 2011. “Poverty and Low Earnings in the Developing World.” IRL Working Paper, Cornell University School of Industrial and Labor Relations, Ithaca, NY. https://ecommons.cornell.edu/handle/1813/74734. Fields, G. S. 2012. Working Hard, Working Poor: A Global Journey. New York: Oxford University Press. Filmer, D., L. Fox, K. Brooks, A. Goyal, T. Mengistae, P. Premand, D. Ringold, S. Sharma, and S. Zorya. 2014. Youth Employment in Sub-Saharan Africa. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/16608. ILO (International Labour Organization). 1999. Decent Work. Report of the Director-General, Geneva. https://www.ilo.org/public/english/standards/relm/ilc/ilc87/rep-i.htm. ILO (International Labour Organization). 2016. Key Indicators of the Labour Market, Ninth Edition. Geneva: International Labor Organization. https://www.ilo.org/wcmsp5/groups/public/-- -dgreports/---stat/documents/publication/wcms_498929.pdf. ILO (International Labour Organization). 2017. Decent Work and the 2030 Agenda for Sustainable Development. International Labor Organization, Geneva. http://ilo.org/wcmsp5/groups/public/---dgreports/--- dcomm/documents/publication/wcms_436923.pdf. La Porta, R., and A. Shleifer. 2014 . “Informality and Development.” Journal of Economic Perspectives 28 (3): 109–26. 17 Loayza, Norman V. 2018. “Informality: Why Is It So Widespread and How Can It Be Reduced?” Research and Policy Brief 20, World Bank, Washington, DC. https://documents1.worldbank.org/curated/en/130391545228882358/pdf/Informality- Why-Is-It-So-Widespread-and-How-Can-It-Be-Reduced.pdf. Loayza, Norman, Luis Servén, and Naotaka Sugawara. 2010. “Informality in Latin America and the Caribbean.” In Business Regulation and Economic Performance, edited by N. Loayza and L. Servén, 157–96. Washington, DC: World Bank. https://documents1.worldbank.org/curated/en/647141468324528897/pdf/528590PUB0Busi 000public00BOX358355B.pdf. Merotto, Dino 2020. Jobs Diagnostics: A Step-by-Step Guide. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/33434. Merotto, Dino, Michael Weber, and Reyes Aterido. 2018. “Pathways to Better Jobs in IDA Countries: Findings from Jobs Diagnostics.” World Bank, Washington, DC. https://openknowledge.worldbank.org/handle/10986/30594. Ohnsorge, Franziska, and Shu Yu, eds. 2022. The Long Shadow of Informality: Challenges and Policies. Washington, DC: World Bank. https://openknowledge.worldbank.org/bitstream/handle/10986/35782/The-Long-Shadow- of-Informality-Challenges-and-Policies.pdf?sequence=5&isAllowed=y. Urdal, H. 2006. “A Clash of Generations? Youth Bulges and Political Violence.” International Studies Quarterly 50 (3): 607–29. World Bank. 2008. “Additions to IDA Resources: Fifteenth Replenishment.” World Bank, Washington, DC. https://thedocs.worldbank.org/en/doc/415131d585283fb45eeb7348db847da0- 0410012016/original/finalreportmarch2008.pdf. World Bank. 2010. “Special Themes for IDA16.” World Bank, Washington, DC. https://documents1.worldbank.org/curated/en/407511468147850201/pdf/549540BR0IDA1 S101Official0Use0Only1.pdf. World Bank. 2011. “Additions to IDA Resources: Sixteenth Replenishment—Delivering Development Results.” Board Report 60575, World Bank, Washington, DC. https://ida.worldbank.org/en/replenishments/ida16-replenishment. World Bank. 2012. World Development Report 2013: Jobs. Washington, DC: World Bank. https://documents1.worldbank.org/curated/en/263351468330025810/pdf/73068-v1- english-revised-PUBLIC-PUBDATE-10-15-12.pdf. World Bank. 2013. “Special Themes for IDA17.” Board Report 78555, World Bank, Washington, DC. https://documents1.worldbank.org/curated/en/514971468147870158/pdf/785550BR0IDA0 S00Box0377330B00OUO090.pdf. 18 World Bank. 2014. “Additions to IDA Resources: Seventeenth Replenishment—IDA17: Maximizing Development Impact.” Board Report 86434, World Bank, Washington, DC. https://documents1.worldbank.org/curated/en/410401468160173357/pdf/864340BR0IDA0 R0C0discl0osed04010140.pdf. World Bank. 2016a. “IDA18 Special Theme: Jobs and Economic Transformation.” Board Report 106106, World Bank, Washington, DC. https://documents1.worldbank.org/curated/en/500371467996716841/pdf/106106-BR-IDA- SecM2016-0110-PUBLIC.pdf. World Bank. 2016b. “Theme Taxonomy and Definitions.” World Bank, Washington, DC. https://thedocs.worldbank.org/en/doc/275841490966525495- 0290022017/original/NewThemeTaxonomyanddefinitionsrevisedJuly012016.pdf. World Bank. 2017a. “Additions to IDA Resources: Eighteenth Replenishment—Towards 2030: Investing in Growth, Resilience, and Opportunity.” Board Report 112728, World Bank, Washington, DC. https://documents1.worldbank.org/curated/en/348661486654455091/pdf/112728-correct- file-PUBLIC-Rpt-from-EDs-Additions-to-IDA-Resources-2-9-17-For-Disclosure.pdf. World Bank. 2017b. “IDA17 Results Measurement System (RMS).” World Bank, Washington, DC. https://thedocs.worldbank.org/en/doc/362611508859552486- 0290022017/original/IDA17ResultsMeasurementSystemOCTOBER2017.pdf. World Bank. 2018. “IDA18 Results Measurement System (RMS).” World Bank, Washington, DC. https://thedocs.worldbank.org/en/doc/61eddd3dcba42bd006ac6f6bc602ed97- 0410012020/original/ida18_rms_2018.pdf. World Bank. 2019a. Global Economic Prospects, January 2019: Darkening Skies. Washington, DC: World Bank. https://documents1.worldbank.org/curated/en/307751546982400534/pdf/Global- Economic-Prospects-Darkening-Skies.pdf. World Bank. 2019b. “IDA19 Special Theme: Jobs and Economic Transformation.” Board Report 139725, World Bank, Washington, DC. https://documents1.worldbank.org/curated/en/381791564075012778/pdf/Special-Theme- Jobs-and-Economic-Transformation.pdf. World Bank. 2019c. “The IDA19 Results Measurement System.” Board Report 139521, World Bank, Washington, DC. https://documents1.worldbank.org/curated/en/719071563779842522/pdf/IDA19-Second- Replenishment-Meeting-The-IDA19-Results-Measurement-System.pdf. 19 World Bank. 2020a. “Additions to IDA Resources: Nineteenth Replenishment—Ten Years to 2030: Growth, People, Resilience.” Board Report 146261, World Bank, Washington, DC. https://documents1.worldbank.org/curated/en/459531582153485508/pdf/Additions-to- IDA-Resources-Nineteenth-Replenishment-Ten-Years-to-2030-Growth-People- Resilience.pdf. World Bank. 2020b. Global Economic Prospects, June 2020. Washington, DC: World Bank. https://documents1.worldbank.org/curated/en/502991591631723294/pdf/Global- Economic-Prospects-June-2020.pdf. World Bank. 2022a. “Additions to IDA Resources: Twentieth Replenishment—Building Back Better from the Crisis: Toward a Green, Resilient, and Inclusive Future.” Board Report 168910, World Bank, Washington, DC. https://documents1.worldbank.org/curated/en/163861645554924417/pdf/IDA20-Building- Back-Better-from-the-Crisis-Toward-a-Green-Resilient-and-Inclusive-Future.pdf. World Bank. 2022b. Poverty and Shared Prosperity 2022: Correcting Course. Washington, DC: World Bank. https://documents1.worldbank.org/curated/en/099750110062235316/pdf/IDU07dfa30ac03 4520434108ecd0db3a26cd8be6.pdf. 20 Appendix A. Evaluation Design Matrix Table A.1 presents the evaluation’s key questions, sources, data collection, and analysis methods. Table A.1. Evaluation Design Matrix Key Questions Information Sources Data Collection and Analysis Methods I. What was the impact of the World Bank Group’s IDA- financed support for jobs and labor market reforms? 1. How has Bank Group lending contributed to jobs Structured literature review • Establish transparent criteria for effectiveness based objectives? Within lending, has DPF been used to its Portfolio analysis findings on structured literature review and empirical full potential? evidence • Assess the performance of the portfolio of closed projects 2. What has been the implementation and Country case studies Complement data collection and analysis methods performance of the Bank Group’s lending portfolio? with findings from case studies to evaluate the Bank Group’s role and contribution on jobs objectives. 3. Did IDA’s RMS indicators cascade into country- Structured interviews with World Bank and Deep dives through case studies on an in-depth review level results frameworks? If not, where and why? IFC staff of country strategy (both World Bank and IFC), World Bank–IFC collaboration, within World Bank cross– Global Practice collaboration, MIGA PSW engagements, and country portfolio of ASA and lending projects (including CPSDs) II. Were jobs-related interventions based on sound analysis and adapted to country circumstances and priorities? 1. How was the establishment of jobs as a special IDA CPFs, PLRs, and CPE review for selected Portfolio review analysis focus reflected in the Bank Group’s country countries strategies? 2. Did country-level ASA identify constraints on job Structured literature review Deep dives through country case studies on ASA growth, assess data adequacy, and generate Review of core diagnostics for selected quality and relevance and strategic coherence evidence-based and finite policy recommendations? countries (including CPSDs) Were these reflected in country strategies? Deep dives through country case studies on lending portfolio, both World Bank and IFC, as well as MIGA PSW engagements where relevant 21 Key Questions Information Sources Data Collection and Analysis Methods 3. Did projects that aimed to improve jobs outcomes Broad portfolio review analysis Assessment of engagement on data adequacy and anchor their design in diagnostics and fully account Structured interviews with World Bank and M&E on jobs projects for data quality and availability? IFC staff, counterparts, and relevant stakeholders III. To what extent is IDA’s strategy on jobs clear, coherent, and grounded in sound analytics? 1. To what extent did the strategy evolve in line with IDA special themes strategy papers Review strategic documents, policy commitments, and findings from jobs diagnostics? associated findings and reports 2. How did the pilots on measurements inform Jobs diagnostics Structured interviews with IDA and relevant staff corporate strategy and the results framework? How IDA policy commitments and associated engaged at the strategic level in both the World Bank was the issue of data adequacy tackled at the reports as articulated in the and IFC strategic level? documentation under various Replenishment rounds 3. Where there were multiple objectives to IDA Consultations with relevant World Bank Structured literature review strategies, how were potential tensions resolved and management and staff prioritization established? Structured literature review Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; CPE = Country Program Evaluation; CPF = Country Partnership Framework; CPSD = Country Private Sector Diagnostic; DPF = development policy financing; IDA = International Development Association; IFC = International Finance Corporation; M&E = monitoring and evaluation; MIGA = Multilateral Investment Guarantee Agency; PLR = Performance and Learning Review; PSW = private sector window; RMS = Results Measurement System. 22 Appendix B. World Bank Theme Codes Relevant to Jobs and Labor Markets The purpose of the jobs code is to identify interventions explicitly focused on addressing jobs-related challenges and track progress over time, as well as to enable operational teams to access best practice resources from the jobs-coded portfolio and technical support from jobs experts in the World Bank.1 The jobs code tracks all World Bank Group activities that have an explicitly stated and substantive link to creating more, better, and/or inclusive jobs. The words explicit and substantive refer to activities selected or designed with a focus on jobs. Activities intended to analyze or diagnose labor market and jobs-related challenges in a country, program, or specific project may also be included, even if they do not aim to create jobs. Specifically, the activity’s first-order objective or expected development impact must be to create better and/or inclusive jobs. This means that either direct and indirect job creation or improvement can be covered if it is a part of the operation’s focus (lending and nonlending) or investment. This condition is not met if job creation or improved job quality is a by-product or a secondary result of an activity with another primary objective or expected development impact. The jobs code is used when there is a reference to addressing a jobs-related challenge in one or more of the following: project title, development objective, expected development impact, results framework, and/or components. Typical Examples of Coded Activities Activities are coded if the project aims directly to do any of the following: • Expand employment or labor force participation • Increase employability of unemployed, poor people/bottom 40 percent, women, or youth (inclusive jobs) • Increase firm entry and employment; strengthen entrepreneurship; improve micro, small, and medium enterprise finance—for example, through small and medium enterprise banking, microfinance, secured transactions, and credit reporting • Improve labor productivity, earnings, or livelihoods • Improve working conditions and worker benefits • Develop value chains so that they channel more benefits downstream to create more, better jobs 23 • Analyze jobs, employment, and labor market issues Examples of Excluded Activities The jobs code is not used if job effects are a by-product rather than an explicit part of the project. The jobs code is not appropriate if the project aims more broadly to • Expand growth without reference to employment • Strengthen human capital in general • Improve the business environment, expand access to finance through broader financial sector development and/or infrastructure services, unless explicitly done to expand jobs • Reduce poverty without looking at contributions of labor incomes or jobs • Improve social protection system • Expand value chain links upstream and downstream to capture new sales or margin Job creation code: Projects with interventions that relate to the creation of more direct, indirect, and/or inclusive jobs for project target beneficiaries, whether short or long term. Job quality code: Projects addressing worker productivity, working conditions and benefits, earning outcomes, sustained jobs, and any social benefits accruing from jobs. Youth employment code: Projects with interventions supporting financial and jobs outcomes for youth, often through youth employment programs; includes results for disadvantaged youth (for example, orphans, street kids, substance abuse, and so on). Normally defined for the age group 15–29, but age range may also depend on national definitions. Labor market policy and programs code: Activities intended to improve the functioning of labor markets, facilitate transitions into employment, improve the earnings opportunities of individuals, and manage risks facing workers. The following are typical examples of coded activities: • Labor market assessments • Employment services, including counseling and job search assistance • Active labor market programs, including training, employment incentives (wage subsidies and reductions in labor taxes), sheltered and supported employment 24 and rehabilitation; direct job creation (public works); and start-up incentives that promote entrepreneurship • Out-of-work income maintenance and support, including unemployment benefits and early retirement benefits • Labor regulations, including enforcement mechanisms • Labor market information systems Examples of excluded activities: activities focused broadly on social protection and labor policy and systems—including assessments and analytical work, capacity building, institutional and policy reform, and knowledge sharing with the objective of building the administration or policies for social protection and labor systems across all social protection and labor programs—should be coded under social protection. Skills development code: Skills development is a lifelong process that proceeds in stages (each building on the previous one) and includes foundational skills and socioemotional skills. Job-specific skills (hard skills) describe a person‘s technical skill set and ability to perform specific job-related tasks. Labor market institutions code: laws, practices, policies, and conventions that fall under the umbrella of labor market institutions that determine, for example, what kinds of employment contracts are permissible; set boundaries for wages and benefits, hours, and working conditions; define the rules for collective representation and bargaining; proscribe certain employment practices; and provide for social protection for workers Active labor market programs code: programs that stimulate labor demand, support job searches, and improve employability. Bibliography World Bank. 2016. “Theme Taxonomy and Definitions.” World Bank, Washington, DC. https://thedocs.worldbank.org/en/doc/275841490966525495- 0290022017/original/NewThemeTaxonomyanddefinitionsrevisedJuly012016.pdf. 1Each World Bank project, operation, and analytical work is assigned a set of sector and theme codes to reflects its content and focus (World Bank 2017b). 25 Appendix C. Portfolio Analysis Figure C.1. Regional Distribution of IDA IPF and Program-for-Results Portfolio on Jobs and Labor Markets by Commitment 18% 3% 2% 4% 5% 68% Africa East Asia and Pacific Europe and Central Asia Latin America and the Caribbean Middle East And North Africa South Asia Source: Independent Evaluation Group. Note: IPF and Program-for-Results jobs-related IDA commitments equal US$51.1 billion. IDA = International Development Association; IPF = investment project financing. Figure C.2. Regional Distribution of IDA IPF and Program-for-Results Portfolio on Jobs and Labor Markets by Number of Projects 4% 13% 5% 8% 58% 12% Africa East Asia and Pacific Europe and Central Asia Latin America and the Caribbean Middle East And North Africa South Asia Source: Independent Evaluation Group. Note: The number of IPF and Program-for-Results jobs-related IDA projects is 599. IDA = International Development Association; IPF = investment project financing. 26 Table C.1. Regional Distribution of Active and Closed Projects with Jobs or Labor Market Content (IDA Only, Fiscal Years 2015–22, IPF plus Program-for-Results) Share of Jobs or Share of Jobs or LM-Related LM-Related Jobs-Related Commitments Jobs or LM- Projects Out of Commitments Out of Total Related Projects Total Region (US$, millions) Commitments (%) (no.) Commitments (%) Africa 34,729.3 29 347 31 East Asia and Pacific 2,311.8 17 69 26 Europe and Central Asia 1,861.2 28 47 35 Latin America and the 1,205.3 29 32 28 Caribbean Middle East and North Africa 1,781.9 45 24 41 South Asia 9,195.1 22 80 29 Total 51,084.6 27 599 31 Source: Independent Evaluation Group. Note: Jobs and labor market theme codes were applied as noted in the text. IDA = International Development Association; IPF = investment project financing; LM = labor market. Figure C.3. Outcome Ratings for Jobs or Labor Market–Related IPF and Program-for- Results Projects 16 14 World Bank Group projects (no.) 12 10 8 6 4 2 0 Highly Unsatisfactory Moderately Moderately Satisfactory Highly unsatisfactory unsatisfactory satisfactory satisfactory Outcome ratings Source: Independent Evaluation Group Data Hub. Note: The ratings are based on the 29 validated Implementation Completion and Results Reports available for the 167 closed projects. The Implementation Completion and Results Report Reviews cover projects that are not additional financings and were closed at least 12 months earlier. IPF = investment project financing. 27 Table C.2. Development Policy Operations with at Least One Jobs or Labor Market– Related Prior Action), by Region DPOs with at Least One Share of DPOs with at Least Jobs or LM-Related One Jobs or LM-Related Region Prior Action Total DPOs Prior Action (%) Africa 31 163 19 East Asia and Pacific 17 54 31 Europe and Central Asia 8 15 53 Latin America and the Caribbean 5 23 22 Middle East and North Africa 0 0 0 South Asia 12 48 25 Total 73 303 24 Source: Independent Evaluation Group staff analysis. Note: Jobs-related DPOs include all DPOs that were assigned jobs and labor market theme codes. DPO = development policy operation; LM = labor market. 28