World Bank Support to Jobs and Labor Market Reform through International Development Association Financing A First-Stage Evaluation © 2024 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION Please cite the report as: World Bank. 2024. World Bank Support to Jobs and Labor Market Reform through International Development Association Financing: A First-Stage Evaluation. Independent Evaluation Group. Washington, DC: World Bank. COVER PHOTO Marwane Zouaidi EDITING AND PRODUCTION Amanda O’Brien GRAPHIC DESIGN Luísa Ulhoa Vonda Witley This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The bound- aries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissem- ination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. World Bank Support to Jobs and Labor Market Reform through International Development Association Financing A First-Stage Evaluation January 4, 2024 Contents Abbreviations v Acknowledgments vi Overview vii Management Response xvi Report to the Board from the Committee on Development Effectiveness xx 1. Introduction�������������������������������������������������������������������������������������������������������������� 1 The Evolving Jobs Agenda in International Development Association Countries 4 Evaluation Scope and Framework 7 2. The International Development Association Jobs Strategy��������������������������������� 15 Analytical Underpinning 16 Operational Relevance 18 Results Measurement 22 3. Jobs Interventions�������������������������������������������������������������������������������������������������� 26 Portfolio Evolution 27 Design of Jobs Interventions 35 Performance 42 4. Conclusions and Recommendations�������������������������������������������������������������������� 50 Recommendations 52 Bibliography��������������������������������������������������������������������������������������������������������������� 55 ii Boxes Jobs in Fragile and Conflict-Affected Situations Box 2.1.  20 Example of Apprenticeship Interventions Box 3.1.  38 Figures Trends in Labor Force and Employment Ratios for IDA Countries Figure 1.1.  Compared with IBRD Countries 3 Evolution of International Development Association Jobs Strategy, Figure 1.2.  FY15–22 6 Conceptual Framework Figure 1.3.  8 Evaluation Scope Figure 1.4.  9 IDA Jobs-Related Investment Project Financings and Figure 3.1.  Programs-for-Results before and after the Shift in IDA Jobs Strategy 28 Change in Proportion of Different Types of Interventions in Figure 3.2.  “Synthetic” Jobs Portfolio 30 Top Areas of Interventions for Prior Actions across Figure 3.3.  IDA Replenishments 34 Project Counts by Outcome Ratings in Implementation Figure 3.4.  Completion and Results Report Review 43 iii Tables IDA Jobs-Related Portfolio by Lending Instrument and Project Status, Table 1.1.  FY15–22 10 Geographic Distribution of IDA Jobs-Related Investment Project Table 1.2.  Financing and Program-for-Results Portfolio, FY15–22 11 Practice Groups and Global Practices Managing IDA Jobs-Related Table 1.3.  Investment Project Financing and Program-for-Results Portfolio, FY15–22 12 Breakdown of Outcome-Level Versus Output-Level Indicators of Table 2.1.  Jobs Interventions 24 Share of IDA Jobs-Related Investment Project Financing and Table 3.1.  Program-for-Results Portfolio in Total IDA Projects over IDA Replenishment Cycles, FY15–22 28 Cross–Global Practice Collaboration on Jobs-Related Projects Table 3.2.  by Selected Global Practices 32 Collaboration across Practice Groups on Jobs-Related Projects Table 3.3.  33 Types of Jobs Interventions in the Portfolio and Strength of Evidence 35 Table 3.4.  Indicator Analysis Table 3.5.  43  xtent of Problem Projects in the Active IDA-Financed Table 3.6. E Jobs-Related and Nonjobs-Related Portfolios, FY15–22 45 Examples of Jobs Outcome Indicators by Types of Interventions Table 3.7.  48 Appendixes Appendix A. Methods 64  ortfolio Identification and Analysis Appendix B. P 73  tructured Literature Review Appendix C. S 91 Examples of Jobs Prior Actions Appendix D.  102  indings from Related Independent Evaluation Group Appendix E. F Evaluations 106 International Development Association Jobs Strategy Appendix F.  Policy Commitments and Results Indicators 112 iv Abbreviations CPF Country Partnership Framework DPO development policy operation FCV fragility, conflict, and violence FY fiscal year GP Global Practice IDA International Development Association IDA17 17th Replenishment of the International Development Association IEG Independent Evaluation Group IFC International Finance Corporation IPF investment project financing PforR Program-for-Results RMS Results Measurement System SME small and medium enterprise TVET technical and vocational education and training All dollar amounts are US dollars unless otherwise indicated. Independent Evaluation Group World Bank Group    v Acknowledgments This evaluation was carried out by an Independent Evaluation Group team led by Rashmi Shankar (lead economist). The report’s main author is Estelle World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Acknowledgments Raimondo (program manager). The task was conducted under the guidance of Jeffrey Allen Chelsky (manager), the direction of Theo Thomas (director), and the overall guidance of Sabine Bernabè (Director-General, Evaluation). The core evaluation team included Patricia Acevedo (program assis- tant); Harsh Anuj (data scientist); Dung Thi Kim Chu (program assistant); Shahrzad Mobasher Fard (consultant, evaluator); Professor Gary Fields (consultant, senior technical adviser); Pablo Fleiss (technical adviser); Lars Johannes (senior economist); Thomas Kenyon (senior economist); Doruk Yarin Kiroglu (portfolio analyst); Melissa Metz (senior economist); Denita Pious (research analyst and consultant); Andrea Rojas (consultant and evaluator); Rashmi Shankar (task team leader and lead economist); Xiaolun Sun (senior evaluation officer); Sandra Jane Wark (consultant and evalua- tor); Andrei Wong (portfolio analyst and consultant); and Marwane Zouaidi (extended-term consultant and project officer). The report was peer-reviewed by Dr. Gordon Betcherman (professor, Faculty of Social Sciences, School of International Development and Global Studies, University of Ottawa) and Abebe Shimeles (director of research, African Economic Research Consortium, Nairobi, Kenya). The team is grateful to the Independent Evaluation Group Methods team and to colleagues from the Jobs Group, Global Practices, International Finance Corporation, development finance institutions, Operations Policy and Country Services, and Regions, as well as external stakeholders for ex- tensive consultations. vi Overview Key Messages This evaluation represents the first stage of the Independent Evaluation Group’s assessment of the World Bank’s performance in supporting more, better, and more inclusive jobs through International Development Association (IDA) financing. The scope of this evaluation is relatively narrow, focusing on direct IDA support to the achievement of IDA’s jobs objectives. Additional evaluations would be required to consider the broader jobs and economic transformation agenda. The evaluation assesses the analytical underpinnings and opera- tional relevance of the evolving IDA jobs strategy and the extent to which it has been translated into well-designed and well-perform- ing jobs interventions that directly address the objectives of more, better, and more inclusive jobs. The IDA jobs strategy successfully stimulated country analytics and diagnostics to fill knowledge gaps. IDA policy commitments adapted well to reflect learning from jobs diagnostics. Most jobs diagnostics have incorporated recommen- dations that (i) feed into operational work and improve how Country Partnership Frameworks articulate jobs-related objectives and theories of change and (ii) help make jobs more central to engage- ments with countries affected by fragility, conflict, and violence. The enhanced focus on jobs in IDA’s overall strategy has not been associated with a significant increase in the size of the jobs portfo- lio but has led to a change in the mix of jobs interventions. The design of most jobs interventions was well grounded in analysis (including impact evaluation), and projects adequately vii  combined multiple interventions. The IDA jobs strategy prioritization of youth and women’s employ- ment resulted in more focus on these two beneficiary groups in jobs interventions. The use of jobs-relevant development policy financing prior actions has increased steadily, yet they remain infrequent and have had an impact only in one-third of validated operations. However, the IDA jobs strategy’s promise of improved results mea- surement has yet to be fulfilled. Although impact evaluations are more frequent, results frameworks continue to track outputs rather than outcomes, and many countries’ labor market statistics are still lacking. Overall, the performance of jobs interventions under implementation is on track. Two-thirds of jobs-related indicators in projects that have passed their Mid-Term Reviews were on track to meet their targets. However, little is known about the effectiveness of the interventions because of weaknesses in results measurement or in country-level statistical systems measuring labor market developments. viii   More, better, and more inclusive jobs are critical for poverty reduction and shared prosperity in countries eligible for International Development Association (IDA) financing. Since 2014, IDA has included jobs as a special theme, and subsequent IDA Replenishments have had what this evalua- tion terms an “IDA jobs strategy,” with explicit objectives, a series of policy commitments to achieve them, and results indicators to track them. This evaluation is the first in a potential series to address the World Bank’s con- tribution to the jobs agenda, and as such has a focused scope. It assesses the implementation of the IDA jobs strategy and answers two questions. The first is to what extent IDA’s strategy on jobs was grounded in sound analyt- ics, adaptive, and operationally relevant. The second is to what extent the strategy has been translated into relevant and effective jobs interventions that directly address the objectives of more, better, and more inclusive jobs. The evaluation is based on a conceptual framework that identifies three direct, interrelated channels through which IDA support for jobs objectives is pursued: acting on labor demand (including efforts to expand the demand of private sector firms for workers and create the conditions for reliable and adequately remunerated self-employment); increasing labor supply (including efforts to improve the marketable skills of workers, expand the labor force participation of women, and support youth employment); and improving labor market flexibility (including policies to facilitate the move- ment of workers from lower-productivity to higher-productivity activities within sectors, and from lower-value-added to higher-value-added sectors) and geographic mobility where lack of mobility is identified as a significant Independent Evaluation Group World Bank Group    ix constraint in jobs diagnostics. The evaluation focuses on IDA-supported interventions that directly sup- port jobs objectives across the three Replenishment cycles from fiscal years 2015 to 2022. Although the structural change associated with economic transformation is linked to jobs and growth, the scope of this evaluation is limited to the three main channels for achieving IDA jobs objectives. Therefore, we do not evaluate the implementation and effectiveness of the economic transformation agenda, given (i) its very broad purview and (ii) the difficulties inherent in measuring the volume and impact of World Bank financing for economic transformation and how it is intermediated through results indicators at the project or Country Partnership Framework levels. The contribution through the IDA Private Sector Window is also outside the scope of this evaluation, which is being presented alongside the evaluation of the Private Sector Window by the Independent Evaluation Group. International Development Association’s Evolving Jobs Strategy The evaluation finds that, through the deployment of jobs diagnostics, the World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Overview 17th Replenishment of IDA (IDA17) and IDA18 policy commitments helped address significant knowledge gaps. Previous country analytics did not focus on jobs, with little attention to the factors that influence labor mar- ket outcomes in IDA countries. Most jobs diagnostics provided actionable recommendations. In two-thirds of cases, these were tailored to country con- texts and offered guidance on prioritization, even if very few distinguished between short-term and long-term interventions. There was also a robust feedback loop between evidence from jobs diagnos- tics and subsequent IDA cycles. Policy commitments evolved over time to reflect learning from country evidence and experience—for example, with respect to the importance of complementing interventions aimed at raising productivity in agriculture (to improve earnings) with interventions that promote agribusiness (to create new jobs) in IDA18. The IDA jobs strategy and jobs diagnostics contributed to a better articula- tion of jobs-related objectives, theories of change, and results frameworks in Country Partnership Frameworks. Whereas previously support for jobs had been implicit, typically occurring through business environment reforms, access to finance, or improvement in agricultural productivity, the strategy nudged Country Partnership Frameworks toward a more explicit and coher- ent statement of the jobs agenda. The sharper focus on jobs in country strategies is associated with a slight increase in the relative size of the portfolio. The estimated average annual shares of the IDA jobs-related portfolio in the total project count and value of commitments of all IDA projects increased slightly over the evaluation period. There was also a pronounced effect on engagements with countries affected by fragility, conflict, and violence (FCV). IDA17 policy commitments under x the FCV pillar led to the development of a specific integrated jobs strategy for countries affected by FCV. As a result, the jobs portfolio in countries af- fected by FCV put greater emphasis on youth employment and the economic inclusion of displaced people. However, the IDA jobs strategy’s promises of improved results measurement have not yet been fulfilled. Although impact evaluations are more frequent, results frameworks continue to track outputs rather than outcomes, and most countries’ labor market statistics are still lacking. There is an inherent tension between the IDA jobs strategy’s ambition of better capturing jobs outcomes and the systems and incentives underlying results frameworks. Although IDA Results Measurement System Tier 2 indicators capture in- termediate outcomes from IDA financing,1 at the project level, teams may be discouraged from including appropriate outcome-level indicators in results frameworks, partly over concerns about attribution. Drawing on the experience with improving the focus on gender in operations, IDA Results Measurement System Tier 3 indicators could also better capture how well IDA operations articulate and track contributions to jobs outcomes. World Bank analysis has also highlighted some of the technical difficulties in esti- mating the indirect impact of IDA’s interventions on jobs created, which is complex and data intensive.2 Jobs Interventions Independent Evaluation Group World Bank Group    xi IDA commitments provoked a change in the mix of jobs interventions in the portfolio. First, consistent with the IDA19 commitment to help remove bottle- necks in sectors with high potential for private sector–led job creation, there was an increase in projects addressing labor demand in formal firms, in some cases accompanied by an explicit focus on economic diversification through support for specific value chains. Second, there was a shift toward support for agribusiness and agricultural value chains, consistent with IDA’s priority of growing jobs in the food system. Third, there was an increase in the proportion of jobs-relevant projects that sought the participation of youth and women and of projects specifically seeking to increase youth employment. Conversely, interventions that specifically sought to improve women’s employment re- mained rare in the portfolio. That said, reinforcing commitments made under the gender strategy and IDA policy commitments resulted in wider use of gender-disaggregated indicators in jobs-focused projects. Development policy financing was used in several countries to contribute to jobs outcomes through changes in the labor market regulations. In the small sample of development policy financing Implementation Completion and Results Report Reviews, the Independent Evaluation Group deemed jobs-related prior actions to be relevant, but only one-third of operations achieved targets World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Overview for results indicators, primarily because of a mismatch between the ambition of the reform supported and the institutional capacity for implementation. The design of jobs interventions was well informed by evidence. The evaluation triangulated information from the portfolio review, the struc- tured literature review, Independent Evaluation Group–validated project self-evaluations, and case studies to assess the strength of the analytical underpinning of jobs-related interventions and the quality of their de- sign. Most jobs interventions supported by IDA were grounded in a strong evidence base. There was also evidence of learning and adaptation of op- erational design within and across countries. Youth employment and some productive inclusion projects, for example, showed a strong trend toward the bundling of supply-side interventions, marking an improvement since the Independent Evaluation Group’s 2012 evaluation. However, fully integrating supply- and demand-side interventions—as recommended by many jobs diagnostics—has proven difficult. Although collaboration between the Agriculture and Food and the Finance, Competitiveness, and Innovation Global Practices (GPs) was an important factor behind the growth of agribusiness and value chain projects, inter- views with country teams suggest obstacles to working across sectors. Collaboration was hindered, for example, by corporate incentives favoring certain GPs and task team leaders’ responsibilities, including through great- er control of budgetary resources. There were also significant differences in perspectives between GPs on how best to address jobs objectives. Finally, the jobs agenda is not managed in a centralized manner within most client governments, which can contribute to a fragmented policy dialogue, which is not conducive to a more integrated approach within the World Bank, where different GPs may have different government interlocutors. xii Based on the limited available data, the performance ratings of the closed projects in the evaluation portfolio were slightly better than those of the rest of the IDA portfolio. Given that the evaluation period starts in fiscal year 2015, there are relatively few closed projects with validated outcome ratings available. A comparison between jobs- and nonjobs-related closed investment projects with IDA financing indicates relatively better performance of the jobs-related projects. Two-thirds of jobs-related indicators in projects that have passed their Mid-Term Reviews were on track to meeting their targets, but, as noted, there were shortcomings in the many of the underlying indicators. However, we can say little about the jobs outcomes of IDA-financed in- terventions. This is because the IDA strategy has not been successful at promoting better measurements of jobs outcomes, with more attention needed to enhance the systems for capturing labor market impact and out- comes. In most cases, the impact of interventions on jobs was only weakly captured through project development objective indicators, with project teams incentivized to focus more narrowly on what is directly attributable and tending therefore to measure the volume of outputs or number of ben- eficiaries instead. Examples include number of persons completing a skills training program or number of public employment centers upgraded. Recommendations On the basis of the evidence and findings presented in this evaluation, we Independent Evaluation Group World Bank Group    xiii make the following two recommendations: Recommendation 1. IDA could strengthen the measurement of its contribu- tion to the achievement of jobs objectives. This can be achieved through better and more relevant corporate-level indicators, better project monitoring and evaluation, and enhanced support for country statistical systems for measur- ing labor market outcomes. Implementation of this recommendation has the potential to significantly improve learning, adaptation, and accountability. Recommendation 2. IDA could draw more systematically on jobs diagnos- tics to inform country-level operational engagement. This can be achieved by strengthening the ownership and use of jobs diagnostics and the integrated approach contained therein to inform policy dialogue, Country Partnership Framework priorities, and operational design. For example, Country Management Units could better incentivize cross-GP collaboration by drawing on diagnostics to establish an integrated and contextualized vision of policy priorities to be the basis of policy dialogue and project design. World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Overview xiv 1  The Results Measurement System of the International Development Association (IDA) includes three tiers of indicators. Tier 1 indicators measure high-level outcomes, Tier 2 indicators capture intermediate outcomes from IDA support, and Tier 3 indicators gauge organizational effectiveness. 2  The policy commitments of the 19th Replenishment of IDA included “conduct[ing] 20 pilots in ‘economic transformation IDA projects’ to estimate indirect and/or induced jobs” (World Bank 2020a, 90), which led to the development of a model-based estimation methodology, not currently used for project selection or for monitoring of jobs outcomes. Independent Evaluation Group World Bank Group    xv Management Response Management of the World Bank thanks the Independent Evaluation Group World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Management Response for the report World Bank Support to Jobs and Labor Market Reform through International Development Association Financing: A First-Stage Evaluation and the opportunity to provide comments. The evaluation assesses the imple- mentation of the International Development Association (IDA) jobs strategy through three interrelated channels: labor demand, labor supply, and labor market flexibility. Provision of jobs is a critical agenda for IDA countries, which are grappling with burgeoning youth populations and low-quality jobs with limited earnings. It is also particularly relevant to the evolution discussion as the World Bank scales up its impact on key development out- comes, including jobs, and is developing a new scorecard to better measure its results. Management welcomes the report’s overall positive findings of the World Bank’s performance across several areas during three IDA cycles from fiscal year (FY) 15 to FY22. The report has useful insights to inform the Mid-Term Review discussions of the 20th Replenishment of IDA (IDA20) scheduled for early December. World Bank Management Comments Overall Management is pleased with the report finding that, since IDA17, there has been a clear IDA jobs strategy with strong analytical underpinnings and a discernible influence on country strategies. The report finds that most jobs diagnostics undertaken during the period have provided actionable recom- mendations tailored to country contexts, offered guidance on prioritization, and successfully informed Country Partnership Frameworks (CPFs). The report recognizes that there was significant engagement with countries affected by fragility, conflict, and violence through the development of a jobs framework for conflict-affected and fragile countries and projects emphasiz- ing youth employment and the economic inclusion of displaced people. The report points to youth employment and productive inclusion projects bun- xvi dling labor market supply-side interventions, representing an improvement since Independent Evaluation Group’s 2012 evaluation. The report also highlights a robust feedback loop between evidence from jobs diagnostics informing policies of subsequent IDA cycles. Management appreciates the report’s finding regarding the strong uptake of learning, particularly from impact evaluations. Also welcome is the increasing inclusion of gender-dis- aggregated indicators in jobs-focused projects. Management welcomes the report’s appreciation of the operational changes in IDA programs and notes the areas for improvement. The report finds that there has been only a slight increase in the total number of jobs-related projects in the IDA portfolio. Management notes that the total commitments related to jobs-related projects have increased from $6.5 billion (IDA17) to $9.7 billion (IDA19)—a 50 percent increase. Management also notes the increase in the estimated number of jobs enabled by relevant prior actions in development policy operations and concurs with exploring the increased use of this instrument for the jobs agenda. The benefits of cross–Global Practice collaboration have been identified in the report, particularly among Agriculture and Finance, Competitiveness, and Innovation Global Practices, and the need for doing this on a broader scale is well recognized. Other Comments Management believes that comprehensive approaches that address both labor market supply- and demand-side constraints are more promising than Independent Evaluation Group World Bank Group    xvii stand-alone job-focused interventions, and IDA is moving in that direction. It is widely accepted that projects focused on labor supply-side interven- tions alone would not effectively lead to more and better jobs. Likewise, labor demand-side interventions alone may fail to achieve intended jobs outcomes, especially for the vulnerable and excluded. This recognition is leading to IDA projects taking a more comprehensive approach following the evidence from well-evaluated labor market interventions. Traditional projects that had relied only on support to technical and vocational education training institutions have shifted toward interventions that either bundle labor demand and labor supply interventions or are closely aligned with labor demand dynamics, or both. The report points to limited outcome orientation in the intermediate IDA results measurement indicators. Management notes that Tier 2 of the IDA Results Measurement System (RMS) captures outputs or results achieved to date, not high-level outcomes. More broadly, some caution may be required with respect to the ability of individual projects to influence jobs outcomes at scale, given the number of factors that are at play in labor market outcomes. World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Management Response For CPFs, management notes that, with the implementation of the new CPF guidance since FY22, more attention is being paid to outcomes (high-level outcomes) rather than outputs, including in IDA countries. The proposed new World Bank Group Scorecard includes a jobs indicator that will incorporate a detailed methodology to capture direct job creation and estimate job induced impacts and labor outcomes enabled by policy reforms supported by develop- ment policy operations and Bank Group upstream support. The nature and complexity of the jobs challenge in IDA countries is daunting and requires sustained cross-sector reform efforts. The immediate challenge for IDA countries is in improving the quality, job security, productivity, and earnings from jobs in “lower tier” or easy-access informal self-employment. These types of jobs may need fewer skills and little capital but they are seldom “good” jobs. Many self-employed workers in IDA countries are involuntarily self-employed as “entrepreneurs out of necessity,” working long hours for limited pay. Higher-paid jobs supported by improved productivity and techno- logical innovation are good for long-term growth and are also more likely to encourage women and youth to join the labor force and learn on the job. While apprenticeships have been effective in some contexts, as noted in the report, they may not reach the most vulnerable workers in the informal sector or self-employed workers. Management relies on expertise from across the World Bank to address this complex agenda and will continue to explore more effec- tive ways to address the agenda at the country portfolio level, underpinned by several (single or multi–Global Practice) projects. Recommendations and the Way Forward Management concurs with the recommendation to strengthen IDA’s RMS, which is in line with ongoing work on the new Bank Group Scorecard and asso- ciated upgrading of the Bank Group results architecture. The report proposes a set of relevant outcome indicators and improved monitoring and evaluation systems (both are priorities for the new Scorecard and associated work) and xviii support for strengthening the country’s statistical systems (this has been an important policy commitment in IDA, and progress is being made, particu- larly in Sub-Saharan Africa). Management notes that the lack of good quality jobs data (which impacts the choice of indicators for operations and the Jobs and Economic Transformation RMS more broadly) may influence the pace of operationalization of the recommendation. As part of the Bank Group’s efforts to enhance the outcome orientation of corporate reporting, a results indica- tor called the “number of new or better jobs” has been proposed in the new Scorecard. A Bank Group working group has been set up to define a detailed methodology and how the various measurement challenges will be addressed, including capturing indirect or induced jobs, and estimated labor outcomes enabled by policy reforms supported by development policy operations and Bank Group upstream work. Once developed, this new Scorecard indicator will be embedded in relevant CPFs and project-level results frameworks. As part of the IDA21 strategic discussions in the coming year, management will explore opportunities to align IDA’s RMS with the new Scorecard. Management agrees with the second recommendation on increasing the use of jobs diagnostics. As part of the effort toward strengthening the analytical basis, alongside the existing jobs diagnostics, the revamped Country Economic Memorandum, to be renamed Country Growth and Jobs Report, will be a core advisory services and analytics to help client countries design and implement a strategy to accelerate economic growth and job creation. The Country Growth and Jobs Report will include a diagnostic of the Independent Evaluation Group World Bank Group    xix macroeconomic drivers of economic growth as well as the drivers of business performance and growth. These diagnostics will be followed by an in-depth analysis of key cross-cutting topics that influence economic growth and job creation. The report will conclude with options for policy reforms required to achieve sustainable and inclusive growth and will provide estimates of potential effects on growth and jobs. The Jobs Group will complement the Country Economic Memorandum findings with detailed jobs diagnostics feeding into country policy dialogue and CPFs and plans to roll out a strategic jobs framework for the forthcoming Jobs for Development report. Report to the Board from the Committee on Development Effectiveness World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Report to the Board The Committee on Development Effectiveness met to consider the Independent Evaluation Group (IEG) evaluation World Bank Support to Jobs and Labor Market Reform through International Development Association Financing: A First-Stage Evaluation and the draft management response. The committee welcomed the report and noted that, while limited in scope, IEG’s first-stage early assessment of World Bank support to the jobs and la- bor market through International Development Association (IDA) financing provided a timely narrative and helpful inputs to inform the upcoming Mid- Term Review of the 20th Replenishment of IDA (IDA20) slated for December 6–8, 2023. Members welcomed IEG’s findings, noting that IDA support has helped address significant knowledge gaps, that jobs interventions were also well designed and relevant to the jobs objectives, and that IDA policy com- mitments adapted well to reflect learning from jobs diagnostics that have contributed to a better articulation of jobs-related objectives in Country Partnership Frameworks and operational work. From IEG’s findings, mem- bers also noted that performance ratings of closed projects were slightly better than those of the rest of the IDA portfolio; that integrating supply- and demand-side interventions—as recommended by jobs diagnostics and IDA Strategy—has proven challenging; that the IDA jobs strategy’s promise of improved results measurement has yet to be fulfilled; and that IEG can say little about jobs outcomes of IDA-financed interventions. Members commended management’s preparedness to address the challeng- es identified by the evaluation and the commitment to enhance support for job creation in IDA countries. They noted their agreement with IEG’s rec- ommendations aimed at strengthening measurement of IDA’s contribution to the jobs agenda given the identified weaknesses in the results frame- work, which largely track outputs rather than outcomes. They expressed xx concerns regarding the inadequate measurement of results and limitations in data availability and welcomed the World Bank’s response to this includ- ing efforts to enhance support for countries’ statistical systems in building capacity to strengthen data coverage and quality to effectively measure jobs and labor market outcomes. They also encouraged management to work closely with development partners in this regard. Given that the adoption of jobs-related objectives in IDA17 was largely moti- vated by the view that rising inequality was undermining progress in poverty reduction and other Sustainable Development Goals, and considering that extreme poverty and inequality remain widespread, members highlighted the importance of systematically leveraging jobs diagnostics to inform IDA’s interventions on job creation, and appreciated management’s response to addressing this issue including through efforts to capture direct job creation in the new Corporate Scorecard. In addition, they urged management to enhance incentives underlying the results frameworks as part of the ongoing work on outcome orientation and on the Corporate Scorecard and to give greater attention to country statistical capacity to improve the quality of data needed to estimate labor market developments and to inform related policy. They expressed support for management’s plan to rename the World Bank’s detailed analysis of a country’s economic developments, prospects, and policy agenda from Country Economic Memorandum to Country Growth and Jobs Report, to strengthen the link between jobs diagnostics and coun- Independent Evaluation Group World Bank Group    xxi try-level operations. 1 | Introduction Highlights In countries eligible for International Development Association (IDA) financing, the challenges related to jobs are multifaceted and have compounded effects on poverty and inequality. Since 2014, IDA has included a special theme on jobs, and, over the subsequent three Replenishments, it enhanced its policy commit- ments to realize its stated jobs objectives. This evaluation focuses on IDA-supported interventions that di- rectly address constraints to the key objectives of more, better, and more inclusive jobs across three IDA cycles in a portfolio of 257 IDA-financed World Bank projects approved during fiscal years 2015–22. This evaluation assesses the analytical underpinnings and op- erational relevance of the IDA jobs strategy and the design and performance of jobs interventions. 1  In countries eligible for International Development Association (IDA) fi- nancing, jobs challenges are multifaceted and have compounded effects on poverty and inequality. These countries face six major types of challeng- es. First, both income and consumption inequality are driven by inequality in labor earnings (Fields 2011a) and inequality of opportunities based on gender, ethnicity, or age. Second, underemployment, not unemployment, is the main challenge for many low-income countries (Merotto, Weber, and Aterido 2018), where poor people have work but are unable to earn sufficient income to World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 1 lift themselves out of poverty. This is particularly true of women and youth, whose labor force participation is much higher than in International Bank for Reconstruction and Development countries but who tend to work in informal and low-paid jobs (figure 1.1, panel a). Third, informality can be a significant challenge to better jobs for many people in emerging markets and developing economies, where low-productivity and low-value-added informal jobs have been estimated to account for 70 percent of employment and about one-third of the GDP (Ohnsorge and Yu 2022). Informal workers are more likely to lose their jobs or suffer severe income losses during crises, such as the COVID-19 pandemic, and they tend to be largely excluded from social safety nets. Informal jobs have also been associated with broader development chal- lenges (World Bank 2019b, 2020b). Fourth, high unemployment and unmet job expectations among youth, as well as significant gender gaps for both workforce participation rates and labor income, are immediate concerns (ILO 2017; World Bank 2014a). Fifth, almost 80 percent of labor productivity growth in low-income countries came from the reallocation of labor from lower-productivity agriculture into relatively higher-productivity services and industry (Merotto, Weber, and Aterido 2018). However, low-income economies are usually unable to absorb all the workers released from agri- culture into higher-productivity and higher-paying wage employment. This suggests that labor market flexibility and worker employability are import- ant intervention channels to support jobs-related objectives. Sixth, there can be trade-offs among various interventions supporting jobs objectives; for example, although raising agricultural productivity can increase earnings, it often leads to decreased employment in the sector. 2  rends in Labor Force and Employment Ratios for IDA Figure 1.1. T Countries Compared with IBRD Countries a. Female and youth labor force participation 55 53 51 Participation rate (%) 49 47 45 43 41 39 37 35 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 Calendar year IDA: Female labor force IDA: Youth labor force IBRD: Female labor force IBRD: Youth labor force b. Employment in agriculture 70 Employment category (%) 60 50 40 30 20 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 Independent Evaluation Group World Bank Group    3 Calendar year IDA: Employment in agriculture IBRD: Wage and salaried workers IBRD: Employment in agriculture IDA: Wage and salaried workers Source: International Labour Organization 2021. Note: Data presented are modeled estimates by the International Labour Organization. Female labor force participation rate is the ratio of female labor force to female population 15 years of age and older, and that for youth is the ratio of youth 15–24 years of age in the labor force to total population 15–24 years of age. Employment in agriculture and wage and salaried workers is the relative ratio of employ- ment in these categories to total employment. IDA and IBRD aggregates are for IDA-only and IBRD-only countries and are based on weighted averages as published in World Bank’s World Development Indicators database. IBRD = International Bank for Reconstruction and Development; IDA = International Development Association. More, better, and more inclusive jobs are critical for poverty reduction and shared prosperity in IDA-eligible countries. “Inclusive jobs” refers to both more jobs and improved access to jobs and opportunities for populations. The World Development Report 2013: Jobs (World Bank 2012a) highlighted the importance of pursuing multiple jobs-related objectives, including higher jobs growth and improved development dividend from jobs. This dividend can take many forms: for example, jobs help poor people work their way out of poverty and provide higher earnings and economic empowerment specif- ically to women, who subsequently invest more in their children. Jobs lead to improved efficiency as workers get better at what they do and as more productive jobs appear. Jobs improve social cohesion by bringing people from different backgrounds together and helping to reduce conflict (Filmer World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 1 et al. 2014; Urdal 2006). The World Development Report on jobs noted that high unemployment and unmet job expectations among youth are immedi- ate concerns and that growth can also be “jobless” (World Bank 2012a, 99). Unemployment rates can be low, particularly where dependence on agri- culture is high or the private sector is largely informal and characterized by self-employment, but earnings are also low. As such, promoting full and productive employment and decent work for all is at the core of Sustainable Development Goal 8. The Evolving Jobs Agenda in International Development Association Countries IDA has included a special theme on jobs since 2014, and over the subse- quent three Replenishments, it enhanced its policy commitments to realize its jobs-related objectives. In the 17th Replenishment of IDA (IDA17), IDA explicitly recognized the role played by labor markets in intermediating between growth and inclusion. Before the IDA17 strategy paper, IDA em- phasized growth and the use of social safety nets to mitigate the effects of poverty. This acknowledgment of jobs marked a shift in IDA’s inclusive growth strategy, and jobs became more central to its strategy for inclu- sive growth and for achieving the twin goals. IDA17, IDA18, and IDA19 established specific policy commitments and results indicators under the jobs-related special theme. The two main policy commitments under IDA17 were the establishment of the jobs diagnostics to be implemented through the newly constituted Jobs Group and the preparation of an integrated jobs strategy for countries classified as fragile and conflict-affected situations. IDA18 established a confluence between jobs and economic transforma- tion and highlighted the role of the private sector in creating “good” jobs in 4 labor-intensive sectors. IDA19 and IDA20 preserved the jobs special theme (figure 1.2). In this evaluation, the IDA jobs strategy includes three main components: a strategic framework, policy commitments, and the Results Measurement System. The strategic framework sets out jobs-related objectives and poten- tial intervention channels; policy commitments set the stage for country contextualized engagements (both lending and nonlending) by establishing jobs priorities and setting a course toward higher-level outcomes; and the IDA Results Measurement System measures outcomes and effectiveness across three tiers. Tier 1 indicators measure high-level outcomes in IDA- eligible countries, Tier 2 indicators try to capture intermediate outcomes from IDA financing, and Tier 3 indicators aim to gauge institutional effec- tiveness. Figure 1.2 presents the evolution of the IDA jobs strategy and its analytical underpinning across IDA Replenishments. Appendix F presents the detailed policy commitments and Results Measurement System indica- tors related to jobs. Independent Evaluation Group World Bank Group    5 6 World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 1  volution of International Development Association Jobs Strategy, FY15–22 Figure 1.2. E Evolution of Policy Commitments IDA17 (FY15–17)   IDA18 (FY18–20)   IDA19 (FY21–23) ▪ Performed JDs in at least 15 IDA countries (at least 5 ▪ Performed 10 GVC and spatial mapping analyses to understand how jobs ▪ Helped 10–15 countries address bottlenecks in sectors with high FCS countries), plus key strategic priorities on jobs are created potential for private sector through interventions ▪ Expanded coverage of the Global Financial Inclusion ▪ Integrated operational guidelines for youth employment programs ▪ Ensured that at least 66% of agriculture projects in IDA countries Database to better measure innovative payments ▪ Used operational instruments to improve risk sharing in projects and support VCs ▪ Established key strategic priorities on jobs and crowd in private capital in high-risk investment environments ▪ Supported 15 IDA countries in developing their primary and secondary reported on the priorities and targets  ▪ Performed systematic impact analyses of SME and entrepreneurship cities ▪ Undertook analytical work on job creation in FCS programs across IDA countries to assess their overall impacts for women ▪ Incorporated digital financial services or digital entrepreneurship countries, including by rolling out a “JDs tool” in at and youth elements in 50% of MSME projects least 5 FCS countries  ▪ Assessed how JDs informed operations targeting JET ▪ Supported at least 15 IDA countries with programs or policies to  ▪ Incorporated gender considerations into the ▪ Used ex ante measurement tools and systems to assess the impacts of improve skills and employability analysis, content of the program, and the results large-scale public and PPP investments ▪ Began embedding a JET focus in all IDA country programs and the framework of all IDA CPFs ▪ Adopted a “migration lens” in IDA countries where migration has an design of operations as appropriate, informed by SCDs and CPSDs ▪ Supported 10 IDA countries in meeting their financial impact ▪ Supported at least 10 IDA countries in developing a coherent vision inclusion targets and priorities through financing and ▪ Developed and integrated spatial perspectives into analysis of migration and a set of actions for JET and 20 pilots in “economic transformation technical assistance and urbanization trends and the impacts of infrastructure on JET IDA projects” 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 Jobs S4YE five-year SEJLS program A JET focus was embedded in all IDA19 country cross-cutting strategy An Integrated Integration: A launched programs, including for Bangladesh, Chad, Guyana, WDR 2013: solutions area launched  Framework for New Approach Haiti, Honduras, Kenya, Lesotho, Maldives, Jobs established Jobs in Fragile to Youth Mozambique, Pakistan, São Tomé and Principe, and Conflict Employment Theoretical and Uganda Situations Programs underpinnings Jobs umbrella of JDs MDTF and WDR 2019: S4YE Pathways to Better Jobs The Changing established IEG's youth Nature of Work in IDA Countries: employment Findings from 75 impact evaluations performed evaluation Jobs by Regional Gender Innovation Diagnostics Labs Analytical work Policy commitment JDs were published for Afghanistan; Bangladesh; GVC analyses were completed in 12 IDA countries: Burkina Faso; Côte d’Ivoire; Congo, Dem. Rep.; Ghana; Bangladesh, Benin, Chad, Ethiopia, Haiti, Lesotho, World Bank Group programs Haiti; Kenya; Kosovo; Kyrgyz Republic; Moldova; Mali, Mongolia, Mozambique, Nepal, Pakistan, Rwanda; Sierra Leone; Tajikistan; and Zambia and Uganda Evolution of Analytical Work Source: Independent Evaluation Group. Note: CPF = Country Partnership Framework; CPSD = Country Private Sector Diagnostic; FCS = fragile and conflict-affected situation; FY = fiscal year; GVC = global value chain; IDA = International Development Association; IDA17, IDA18, IDA19 = 17th, 18th, and 19th Replenishments of IDA; IEG = Independent Evaluation Group; JD = jobs diagnostic; JET = Jobs and Economic Transformation; MDTF = multidonor trust fund; MSME = micro, small, and medium enterprise; PPP = public-private partnership; S4YE = solutions for youth employment; SCD = Systematic Country Diagnostic; SEJLS = Supporting Effective Jobs Lending at Scale; SME = small and medium enterprise; VC = value chain; WDR = World Development Report. Evaluation Scope and Framework This evaluation focuses on IDA-supported interventions that directly address constraints to the key objectives of more, better, and more inclusive jobs over three IDA cycles. For the purposes of this evaluation, jobs are defined broadly as including both wage employment and self-employment in the formal and informal sectors. Relevant jobs-related objectives are more, better, and more inclusive jobs, which lead to improved productivity and incomes and better working conditions. Better jobs are defined as those that are better paid, and more inclusive jobs are defined as those employing women, youth, the bottom 40 percent of the population’s income distribution, and individuals from marginalized groups. For ease of reference in this evaluation, “jobs” is used interchangeably with “the jobs agenda” and “jobs-related objectives.” On the basis of the reviewed literature, the early findings of World Bank jobs diagnostics, IDA jobs strategies, and consultations with experts, we identify three interrelated channels through which IDA support for jobs objectives is pursued: acting on labor demand (including efforts to expand the demand of private sector firms for workers and create the conditions for reliable and adequately remunerated self-employment); increasing labor supply (including efforts to improve the marketable skills of workers, expand the labor force participation of women, and support youth employment); and improving labor market flexibility (including policies to facilitate the move- ment of workers from lower-productivity to higher-productivity activities Independent Evaluation Group World Bank Group    7 within sectors, and from lower-value-added to higher-value-added sectors) and geographic mobility where lack of mobility is identified as a significant constraint in jobs diagnostics. All three channels play a role in achieving IDA’s objectives in the jobs space as per the conceptual framework laid out in figure 1.3.  onceptual Framework Figure 1.3. C Through Channels of Objectives interventions Outcomes interventions to Reduce barriers to work, Improve worker skills and with focus on women, expand labor force youth, and marginalized Labor supply World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 1 participation of women groups; improve quality and youth and accessibility of training to build marketable skills Expand private sector Reduce impediments to demand for workers, MSME market access, More, better, and Labor demand including by removing investment, and regulatory more inclusive disincentives to formal disincentives to formal sector hiring hiring jobs Facilitate the movement of Reduce unproductive Labor market workers from lower- to restrictions to labor higher-value-added mobility across sectors flexibility activities within and across and regions sectors and regions Source: Independent Evaluation Group. Note: MSME = micro, small, and medium enterprise. The evaluation focuses on IDA support to policies and operations that directly affect labor demand, labor supply, and labor market flexibility. Although the structural change associated with economic transformation is linked to jobs and growth, the scope of this evaluation is limited to the three main pathways or channels for achieving IDA jobs objectives. Therefore, we do not evaluate the implementation and effectiveness of the economic transformation agenda, given (i) its very broad purview and (ii) the difficul- ties inherent in measuring the volume and impact of World Bank financing on economic transformation and how it is intermediated through results indicators at the project or Country Partnership Framework (CPF) level. Figure 1.4 presents the scope of this evaluation, with the light gray rectangle outline seen as being relevant to the broader job development agenda but outside the remit of this evaluation. This evaluation focuses on IDA financ- ing implemented by the World Bank. Real sector IDA Private Sector Window transactions of the International Finance Corporation (IFC) are outside the scope of this evaluation. On the basis of a thorough portfolio review, the team also developed a typology of jobs interventions to guide data collection 8 and analysis.  valuation Scope Figure 1.4. E Poverty Reduction and Shared Prosperity Private PSD, governance, More, better, and more inclusive jobs investment and infrastructure, and growth macro policy Beyond the scope Beyond the scope of this evaluation Channels of impact for of this evaluation IDA support to jobs Gender and Youth Labor Demand Labor Supply Labor Market Flexibility Expand private sector Improve worker skills and Facilitate the movement of demand for workers, expand labor force workers from lower- to including by removing participation of women higher-value-added disincentives to formal and youth activities within and across sector hiring sectors and regions Source: Independent Evaluation Group. Note: IDA = International Development Association; PSD = private sector development. The evaluation portfolio consisted of 257 IDA-financed projects approved during fiscal years (FY)15–22 that supported at least one of the three IDA jobs-related objectives. The Independent Evaluation Group (IEG) identified these projects from the set of all IDA-financed projects by using a combi- nation of data on the World Bank’s theme taxonomy (World Bank 2016), text mining, and review of project documentation. Manual review of oper- ations was necessary because many operations mentioned jobs even when Independent Evaluation Group World Bank Group    9 their content was only peripherally jobs related. As shown in table 1.1, the portfolio contained 216 investment project financings (IPFs) and 11 Programs-for-Results (PforRs) with IDA commitments of $24 billion and $1.3 billion, respectively. There were also 30 development policy operations (DPOs) with at least one jobs-relevant prior action. A summary of the portfo- lio based on lending instruments and project status is presented in table 1.1. DA Jobs-Related Portfolio by Lending Instrument and Project Table 1.1. I Status, FY15–22 Projects with ICRR Projects IDA Commitments Ratings (no.) (US$, millions) (no.) Lending World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 1 Instrument  Active Closed Total Active Closed Total Closed DPO  0  30  30  n.a.  n.a.  n.a.  16  IPF  144  72  216  18,569  5,392  23,961  18  of which new  131  43  174  17,327  4,100  21,427  18  of which AF  13  29  42  1,242  1,292  2,534  0  PforR  7  4  11  932  419  1,351  2  of which new  6  3  9  782  373  1,155  2  of which AF  1  1  2  150  46  196  0  Total  151  106  257  19,501  5,811  25,312  36  Source: Independent Evaluation Group. Note: AF = additional financing; DPO = development policy operation; FY = fiscal year; ICRR = Implementation Completion and Results Report Review; IDA = International Development Association; IPF = investment project financing; n.a. = not applicable (because DPO financing does not directly sup- port individual prior actions); PforR = Program-for-Results. Most jobs projects took place in the Eastern and Southern Africa Region. The geographic distribution of project counts and commitments for the jobs-related portfolio is shown in table 1.2, in absolute terms and relative to total IDA projects and commitments. The regional distribution of the jobs portfolio is broadly aligned with the overall distribution of IDA resources. The Eastern and Southern Africa Region had the highest proportion of IDA financing going toward projects with jobs-related interventions. At the coun- try level, the countries with the highest proportion (more than 25 percent) of total IDA commitments in the evaluation portfolio, in descending order, were Honduras, Rwanda, the Democratic Republic of Congo, Cameroon, Ethiopia, Mali, Ghana, and Sierra Leone, and the countries with the lowest proportions (of less than 6 percent), in ascending order, were Afghanistan, Madagascar, Somalia, the Marshall Islands, Chad, the Lao People’s Democratic Republic, and Moldova. 10  eographic Distribution of IDA Jobs-Related Investment Table 1.2. G Project Financing and Program-for-Results Portfolio, FY15–22 Projects IDA Commitments As a share Total As a share of total (US$, of total World Bank Region Total (no.) IDA (%) millions) IDA (%) East Asia and Pacific  23  10  1,227  10  Eastern and Southern Africa  66  14  10,870  18  Europe and Central Asia  13  10  640  9 Latin America and the 7  7  435  12  Caribbean  Middle East and North Africa  7  13  480  12  South Asia  36  15  5,323  15  Western and Central Africa  75  15  6,336  13  Total  227  13  25,312  15  Source: Independent Evaluation Group. Note: Total IDA project counts include all projects in the Region with IDA financing, and total IDA commitments include all IDA financing to the Region. FY = fiscal year; IDA = International Development Association. Most of the portfolio was led by three Global Practices (GPs): Agriculture and Food; Finance, Competitiveness, and Innovation; and Social Protection and Jobs. Table 1.3 provides the distribution of project counts and commitments (both in absolute terms and relative to total IDA) by the lead GP managing Independent Evaluation Group World Bank Group    11 the projects and by the Practice Groups under which the respective GPs are mapped. The three GPs with the greatest focus on projects with jobs-related interventions, both in terms of IDA financing and share in total IDA financ- ing, were Agriculture and Food, Social Protection and Jobs, and Finance, Competitiveness, and Innovation, which were spread across three Practice Groups (Sustainable Development, Human Development, and Equitable Growth, Finance, and Institutions).  ractice Groups and Global Practices Managing IDA Jobs- Table 1.3. P Related Investment Project Financing and Program-for-Results Portfolio, FY15–22 Projects IDA Commitments As a share As a share Practice Group or Lead GP of total IDA Total (US$, of total IDA World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 1 Managing Project Total (no.) (%) millions) (%) Equitable Growth, Finance, and 34  30  3,229  31  Institutions  Finance, Competitiveness, and 34  30  3,229  31  Innovation  Human Development  78  13  8,602  15  Education  34  22  2,654  15  Health, Nutrition, and Population  2  1  153  1  Social Protection and Jobs  42  26  5,795  30  Infrastructure  11  6  1,856  9  Digital Development  10  22  1,356  46  Transport  1  1  500  3  Sustainable Development  103  18  11,611  21  Agriculture and Food  78  53  9,304  60  Environment, Natural Resources, 6  9  637  11  and Blue Economy  Social Sustainability and Inclusion  11  19  497  10  Urban, Disaster Risk Management, 7  4  1,040  6  Resilience, and Land  Water  1  1  134  1  Total  226  16  25,219  18  Source: Independent Evaluation Group. Note: Total IDA project counts include all projects managed by a GP with IDA financing, and total IDA commitments include all IDA financing managed by the GP. One project mapped to the erstwhile Trade and Competitiveness GP is excluded from the calculations. Governance and Poverty GPs under the Equitable Growth, Finance, and Institutions Practice Group had no projects in the portfolio. Projects not mapped to any of the GPs listed in the table are excluded from the calculation of total IDA commit- ments. Where more than one GP is involved in a project, the full amount is assigned to the “lead” GP; these figures may therefore understate the extent of involvement of some GPs. FY = fiscal year; GP = Global Practice; IDA = International Development Association. 12 This evaluation seeks to answer two main sets of questions. The first set of questions is addressed in chapter 2, which assesses the adequacy of the IDA jobs strategy through two main evaluation criteria: the suitability of the ana- lytical underpinnings and operational relevance. The second set of questions is answered in chapter 3, which focuses on the portfolio of jobs interventions in IDA-eligible countries and assesses the relevance of design and perfor- mance of the operations. » Evaluation question 1: To what extent is the IDA jobs strategy grounded in sound analytics, adaptive, and operationally relevant? » Evaluation question 2: To what extent has the IDA jobs strategy been trans- lated into relevant and effective jobs interventions? The evidence to answer the evaluation questions stems from five main components. IEG conducted the following analysis (for more details, see appendix A): » A structured literature review of the impact evaluation literature on jobs interventions. » A portfolio review and analysis of 257 projects, using a mix of text mining and content analysis techniques to identify, classify, and assess jobs interven- tions and performance indicators. » A before-and-after analysis of the number, commitment volumes, and inter- Independent Evaluation Group World Bank Group    13 ventions content of jobs-related projects based on “synthetic” portfolio data. » Two structured country-level document reviews: (i) all 22 jobs diagnostics of IDA-eligible countries were manually coded to assess the strength of their analytical underpinning and the relevance of their recommendations, and (ii) 66 country strategies in 33 IDA-eligible countries were reviewed to compare jobs-related objectives, theories of change, results frameworks, prioritization of youth, and gender trends before and after IDA17. » Thirteen in-depth country case studies, including within and cross-case analysis, based on a review of Project Appraisal Documents, Implementation Completion and Results Reports, and Implementation Completion and Results Report Reviews. » IEG conducted nearly 100 interviews with staff from GPs, the Jobs Group, IFC, Country Management teams, the International Labour Organization, and donors and government counterparts (deep-dive case studies only). World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 1 14 2 | The International Development Association Jobs Strategy Highlights Jobs diagnostics addressed significant knowledge gaps. Through a robust feedback loop with the International Development Association (IDA) cycle, jobs diagnostics informed the policy commitments of the 18th and 19th Replenishments of IDA. The IDA jobs strategy and jobs diagnostics improved the articu- lation of jobs-related objectives, theories of change, and results frameworks in Country Partnership Frameworks. The evolving IDA jobs strategy had a pronounced effect on en- gagements with countries affected by fragility, conflict, and violence, with an increased focus on youth employment. The IDA jobs strategy did not successfully promote better mea- surements of jobs outcomes. 15  This chapter assesses the IDA jobs strategy’s analytical underpinnings and operational relevance. IEG examined whether policy commitments evolved over time to better reflect learning from country evidence and experience. IEG also evaluated the operational relevance of the policy commitments and the extent to which they were translated into country strategies. To assess the strength of the feedback loop between the IDA jobs strategy and country engagements, IEG (i) reviewed each of the three strate- World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 2 gy components; (ii) carried out a structured review of 66 CPFs from a sample of 33 IDA countries to compare jobs-related objectives, results indicators, and theories of change before and after IDA17; (iii) performed content analy- sis of 22 jobs diagnostics; and (iv) interviewed country teams in 13 countries. Overall, we found that the IDA jobs strategy had a strong analytical under- pinning and successfully stimulated country analytics and diagnostics to fill knowledge gaps. In turn, policy commitments adequately evolved across replenishments to reflect learning. The jobs strategy also had a discernable impact on country strategies, bringing jobs-related objectives into sharper focus and improving the coherence of the theories of change. However, the strategy failed to stimulate better measurement of jobs outcomes. Analytical Underpinning Jobs diagnostics helped address significant knowledge gaps. Jobs diagnostics were mandated by specific policy commitments in IDA17 and have provided an impetus for unpacking jobs constraints in IDA countries, filling an im- portant knowledge gap. Conventional growth diagnostics and other country analytics did not focus on jobs, and much was unknown about the factors that influence jobs outcomes in IDA countries. The strategy thus called for a multisectoral approach and integrated diagnostics covering issues such as access to markets, inputs, capital, technology, skills, and matching supply and demand. A structured review of all 22 jobs diagnostics in IDA-eligible countries carried out for this evaluation found that the jobs diagnostics’ analysis of labor market trends was thorough, often broken down by sector, region, gender, and age. In all cases, there was a reference to gender gaps, and in 14 out of 22 cases, the jobs diagnostics also provided a thorough assessment of gender-differentiated outcomes and constraints to enhanced female labor force participation or formal employment. In most cases, youth 16 employment challenges were well motivated and discussed, although in nine cases the analysis was rather shallow. The jobs diagnostics also offered a de- tailed analysis of the shifts in employment out of agriculture into services or manufacturing, as well as patterns of internal migration across regions and between rural and urban areas. Most jobs diagnostics usefully incorporated recommendations for action. The structured review of 22 jobs diagnostics also analyzed the extent to which jobs diagnostics offered country-specific recommendations to realize jobs-related objectives. The review found that for active labor market poli- cies, constraints to female employment, and barriers to youth employment, almost two-thirds of jobs diagnostics reviewed offered recommendations that were specifically tailored to country contexts and jobs challenges and to a specific strategy for prioritization. However, only 9 out of 22 jobs diag- nostics included sector-specific recommendations to enhance labor demand, and only 2 jobs diagnostics divided recommendations by time horizon, giving a sense of how to combine short-run and long-run interventions to maximize outcomes. For example, under small and medium enterprise (SME) support, the Nepal jobs diagnostic suggests that in the short term, measures be taken to facilitate the growth of commercially sustainable firms by de- veloping business support services and lower-hanging fruit to encourage financial access, whereas in the medium to longer term, deeper reforms could support foreign direct investment and its linkages with SMEs in addi- tion to strengthening digital infrastructure and the enabling environment for digital solutions. Independent Evaluation Group World Bank Group    17 A robust feedback loop existed between evidence from jobs diagnostics and subsequent IDA cycles. The policy commitments adequately evolved over time to reflect learning from country evidence and experience. The jobs diagnostic was seen as necessary, given that traditional growth diagnostics and other country analytics did not focus enough on jobs. For example, a paper synthesizing early lessons from 15 pilot jobs diagnostics highlighted the fragmentation of the private sector in IDA countries, which are dominat- ed by self-employment and household enterprises and heavily dependent on low-productivity agriculture (Merotto, Weber, and Aterido 2018). The findings informed the IDA18 jobs strategy. They clarified IDA’s jobs-related objectives, adding a specific objective on inclusive jobs, balancing the need to promote formal sector jobs and raising income in the informal sector, and focusing attention on the importance of raising productivity in agriculture and promoting agribusiness. Jobs diagnostics also revealed issues with labor force data and evidence, which was the subject of several IDA18 policy com- mitments to improve data availability and support evidence-based project design. In turn, IDA19 pivoted toward targeted policy commitments to ad- dress specific constraints that had been found to be particularly challenging World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 2 in jobs diagnostics and other country analytics. For example, IDA19 com- mitted to having at least 66 percent of agriculture, and agribusiness projects in IDA countries include support for participation in value chains that were seen to have high potential for growth and job creation. Operational Relevance The IDA jobs strategy and jobs diagnostics improved the articulation of jobs-related objectives, theories of change, and results frameworks in CPFs. IEG conducted a before-and-after analysis on five dimensions of the CPF for 33 IDA-eligible countries. For each country, we compared the articulation of jobs-related CPF objectives, theories of change, results framework indica- tors, youth, and gender prioritization in country strategies designed before and after IDA17. Moreover, the CPFs now have clearer and more explicit statements of jobs-related objectives (22 out of 33) than before IDA17 (15 out of 33). The IDA strategy also nudged a better articulation of both direct and indirect pathways to jobs outcomes. Before IDA17, CPFs that had a clear and coherent theory of how the World Bank’s interventions would contribute to better jobs were rare (2 out of 33 had a fully laid-out theory of change, and 10 had a partial one), whereas they are now much more common (15 have a fully laid-out theory of change, and 8 have a partial one). For example, in Sierra Leone, the FY21 CPF placed jobs at its center, through a theory of change linking the entire portfolio to employment generation. The specification of how the portfolio of interventions can address barriers to more, better, and more inclusive jobs is particularly well done in countries with a jobs diagnostic. Previously, support for jobs had been implicit—usu- ally taking place through business environments and education reforms, as in Bangladesh, or as support for a more transparent, rules-based relation- 18 ship between the state and the private sector, as in the Kyrgyz Republic, or improvements in agricultural productivity, as in most other countries. The improved theories of change are also more routinely reflected in results frameworks, which now more often include indicators to capture progress toward more or better jobs (in 21 cases in the post-IDA17 CPF versus 15 prior). However, most of these indicators continue to be output, rather than outcome, oriented as discussed in this chapter. There was no significant change in the prioritization of gender and youth as cross-cutting themes because this was already common before IDA17. The IDA jobs strategy had a pronounced effect on engagements with coun- tries affected by fragility, conflict, and violence (FCV), especially with an increased focus on youth employment. Before 2014, the World Bank lacked a concrete framework to address inclusive jobs constraints in fragile and conflict-affected states (World Bank 2014b). To address this issue, IDA17 policy commitments under the FCV pillar led in 2015 to the development of a specific integrated jobs strategy for FCV countries (box 2.1). The de- velopment of a specific jobs framework for FCV countries came out of the realization that carrying out full-fledged jobs diagnostics in these countries was impractical because of the data requirements. The framework recog- nized the importance of jobs for social cohesion and strengthened the focus on jobs-related objectives in these countries. Out of $21.4 billion in IDA financing via (nonadditional financing) IPF, $5.2 billion (32 percent) went to countries that were on the World Bank’s FCV lists at the time of project Independent Evaluation Group World Bank Group    19 approval. This proportion was slightly lower, at 27 percent of total financing, in the nonjobs-related IDA-financed portfolio of nonadditional financing IPF during the same period. The jobs-related projects in countries classified as fragile and conflict-affected situations tended to have a higher share of total commitments allocated to jobs components as compared with non-FCV countries. This could explain why, out of 10 IDA-eligible countries with no jobs interventions during the evaluation period, 6 were on the FCV list.  obs in Fragile and Conflict-Affected Situations Box 2.1. J “An Integrated Framework for Jobs in Fragile and Conflict Situations” (Von der Goltz et al. 2016) was the main document supporting the International Development Association’s jobs objectives in countries affected by fragility, conflict, and violence. The strategy emphasized the importance of jobs for social cohesion and stability and highlighted two main points: World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 2 1. Although jobs objectives remain relevant in countries classified as fragile and conflict-affected situations, additional goals may be of more importance in the near term: a. Reintegrating ex-combatants and the displaced, reconnecting people’s net- works torn apart by conflict and violence, and giving everybody—women, men, and excluded groups—a stake in society. b. Restoring livelihoods affected by conflict and violence and jump-starting eco- nomic activity and domestic investment. c. Building confidence in institutions and the social contract. 2. The constraints on achieving jobs objectives are compounded in countries classi- fied as fragile and conflict-affected situations. Destruction of physical capital, loss of customers, an ineffective business environment, slow investment and firm growth, erosion of opportunities for young people, and loss of access to skills training, active labor market policies, and finance all limit job quality and job growth. Elite capture and the concentration of investment and returns in a small number of sectors further reduce the scope of broad-based growth and limit avenues for workers, especially young workers. The Independent Evaluation Group identified four main intervention channels to address these constraints: 1. Longer-term structural reform, with particular focus on governance and macrofiscal stability, finance and markets, trade, competitiveness, and infrastructure. 2. Livelihood support, including through labor-intensive public works, and active labor market policies to better connect people with jobs and self-employment opportunities. (continued) 20  obs in Fragile and Conflict-Affected Situations (cont.) Box 2.1. J 3. Support to the private sector, through both short-run targeted interventions and longer-run enabling reforms. 4. Support to the forcibly displaced and their host communities to find productive live- lihoods and opportunities. All four channels were to be underpinned by engagement to build systems of identifica- tion and registration of the most vulnerable segments of the population, strengthen data collection, gather more evidence from impact evaluations, avoid distortionary incentives and elite capture, and lay the groundwork for longer-run structural reforms. The current World Bank Group Strategy for Fragility, Conflict, and Violence 2020–2025 missed an opportunity to update and strengthen jobs focus based on lessons learned from jobs diagnostics between 2015 and 2020. Although there are multiple references to jobs and economic transformation and to the importance of youth as a source of resilience, the strategy did not undertake a stocktaking of what worked and what did not work in the contexts of countries affected by fragility, conflict, and violence. Source: Von der Goltz et al. 2016. The jobs portfolio in FCV countries placed greater emphasis on operations designed to improve youth employment and the economic inclusion of Independent Evaluation Group World Bank Group    21 displaced people. The strategy highlighted the importance of youth econom- ic activity for stability, and this imperative was translated into the design of operations that targeted youth. The share of youth employment inter- vention in FCV countries is also slightly higher than in the rest of the IDA portfolio. Of jobs-related projects in FCV countries, 68 percent had at least one youth-focused intervention (versus 46 percent of projects in non-FCV countries). The portfolio also contained 14 projects that specifically tar- get refugees and displaced persons, with a focus on economic integration into host communities. For example, the Kenya Development Response to Displacement Impacts Project (P166266, ongoing) and the Socio-economic Inclusion of Refugees and Host Communities in Rwanda Project (P164130) aim to improve access to basic social services, expand economic opportu- nities, and enhance environmental management for communities hosting refugees. In Kenya, this includes activities supporting capacity building, technology access in farm and nonfarm households among host commu- nities, and grants that are expected to support higher incomes among beneficiaries. In Rwanda, this includes support to market and educational infrastructure accessible to displaced persons, as well as jobs skills training for both refugees and host communities to improve employability. However, as explained in the Results Measurement section in chapter 2, because of World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 2 measurement issues, it was sometimes unclear whether these operations led to improved outcomes for these groups. Results Measurement The IDA jobs strategy did not successfully promote better measurements of jobs outcomes. There is an inherent tension between the IDA jobs strategy’s ambition of better capturing jobs outcomes and the systems and incentives underlying results frameworks. IDA Results Measurement System Tier 2 indicators are meant to capture intermediate outcomes from IDA financing. The main Tier 2 indicator related to jobs is an output indicator (number of beneficiaries in IDA countries of jobs-focused interventions). It does not adequately capture progress on addressing constraints to jobs creation or measuring jobs quality and sustainability. Formal Operations Policy and Country Services guidance is that “an operation’s objective focuses on out- comes for which the project can reasonably be held accountable. It neither encompasses higher-level objectives beyond the purview of the project, nor is it a restatement of the project’s activities.” Interviews with task teams sug- gest that this inconsistency in internal expectations disincentivizes teams to capture jobs outcomes or invest in systems that can measure them. Moreover, unlike gender-disaggregated measurement, IDA Results Measurement System Tier 3 indicators do not capture how well IDA operations articulate and track contributions to jobs outcome. There are also technical difficulties in the complex and data-intensive task of estimating the direct and indirect impact of IDA interventions on jobs created. IDA19 committed to experimenting with model-based estimation of jobs impact on 20 projects. Twenty-three pilots were completed in FY22, but the methodology has not been used for project selection or as a basis for project design and monitoring. 22 Overall, we know little about the jobs outcomes of IDA-funded interventions or their effect on welfare and inclusion. In most cases, the impact of inter- ventions on jobs was only weakly captured through project development objective indicators, with projects tending instead to measure the volume of outputs or number of beneficiaries. Table 2.1 shows the shares of a sample of results framework indicators relevant to jobs and whether they captured outputs or outcome in a stratified random sample of indicators based on text searches. The share of outcome-level indicators is overall quite low (only one-third), except for indicators that seek to enhance youth employment, which tend to capture outcomes (48 percent) more systematically. The outcome-orientation of jobs objectives in results frameworks was particularly weak with respect to access to finance, value chain, and agri- business projects. Only a few, such as the Integrating Innovation for Rural Competitiveness in Honduras—COMRURAL II project, measured jobs created or sustained and instead focused on either increases in yields or technol- ogy adoption or the value of sales, exports, and financing mobilized. Value chain projects similarly tended to focus on outcomes for firms rather than workers, although the Bangladesh Export Competitiveness for Jobs project, the Democratic Republic of Congo SME Development and Growth Project, and the Empowering Women Entrepreneurs project captured wage growth and jobs created. The Emergency Support for Micro, Small, and Medium Enterprises Project for the Kyrgyz Republic and the Access to Finance for Independent Evaluation Group World Bank Group    23 Recovery and Resilience Project for Rwanda also measured the survival and employment rates of supported relative to unsupported firms. Conversely, impacts on individuals, whether employment or income, were better tracked through skills development and livelihoods programs (although not always and, in the former case, generally only up to six months after the provision of assistance).  reakdown of Outcome-Level Versus Output-Level Indicators Table 2.1. B of Jobs Interventions Share of Directly Share of Indicators Jobs-Related Sample Identified as Being Indicators Classified as Size Directly Jobs Related Measuring Outcomes World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 2 Intervention Type (no.) (%) (%) Female employment 62 84 44 Youth employment 60 90 48 Jobs—general 60 87 22 Value chains 33 3 n.a. Access to finance— 31 10 n.a. MSME TVET 21 100 36 Agriculture 21 0 n.a. Apprenticeship 20 95 18 LIPW 16 81 25 Labor intermediation 4 100 n.a. Skills—MSME 1 100 n.a. Total 329 67 33 Source: Independent Evaluation Group. Note: Classification of nine indicators as measuring outcomes or outputs was omitted for intervention types where the sample size was less than 10. LIPW = labor-intensive public works; MSME = micro, small, and medium enterprise; n.a. = not applicable; TVET = technical and vocational education and training. Conversely, the dedicated efforts to incentivize youth-focused and gender-disaggregated measurements of IDA jobs interventions paid off. Reinforcing commitments made under the gender strategy, IDA policy com- mitments set out to increase gender-disaggregated data in projects, including jobs-related operations. For example, in IDA17, at least seven Tier 3 indica- tors tracked how well projects articulated or tracked gender results. As shown in table 2.1, the majority of sampled indicators in projects with female em- ployment interventions were relevant to jobs, and almost half were outcome oriented. These monitoring and measurement efforts were complemented by 24 significant investments in generating impact evaluation results, particularly through Regional Gender Innovation Labs mobilized to carry out more than 70 impact evaluations—most of them in IDA-eligible countries in Africa—to test various interventions seeking to close the gender gap in earnings, pro- ductivity, assets, and agency. Learning from impact evaluations was evident in the portfolio, with several examples of successful interventions replicated in other IDA countries. For example, a 2016 impact evaluation of noncog- nitive skills development in Togo demonstrated that women who received innovative entrepreneurial training—focused on personal initiative—saw profits increase by 40 percent, compared with 5 percent for those who had received traditional business training. This intervention has since been repli- cated in projects in Ethiopia, Mauritania, and Mozambique. Independent Evaluation Group World Bank Group    25 3 | Jobs Interventions Highlights The shift in the International Development Association (IDA) jobs strategy led to a change in the mix of jobs interventions in the portfolio. Focus on demand-side interventions increased, and, in line with the evidence literature, bundling of interventions is now common practice. Overall, the choice and design of jobs interventions was well in- formed by evidence. The prioritization of youth and women’s employment in the IDA jobs strategy also resulted in more focus on these two beneficiary groups in jobs interventions. However, despite a growing body of evidence from impact evaluations, interventions that specifically sought to improve women’s employment were infrequent in the portfolio, whereas youth employment projects were more common. Overall, the performance of jobs interventions under implemen- tation is on track. Two-thirds of jobs-related indicators in projects that have passed their Mid-Term Reviews were on track to meeting their targets. However, there were shortcomings in many of the underlying indicators. Validated ratings for the limited number of closed projects sup- porting the jobs agenda were slightly better than those of the rest of the IDA portfolio. However, because of shortcomings in the underlying indicators, little is known about the effectiveness of the interventions, with IDA having fallen short of stimulating mean- ingful improvements in results measurement or in country-level statistical systems measuring labor market developments. 26   This chapter assesses the extent to which the shift in the IDA jobs strat- egy has influenced the portfolio of jobs interventions and whether it has been translated into relevant and well-performing jobs interventions. To answer this question, we (i) performed a portfolio review and analysis of 257 projects; (ii) carried out a structured literature review of the im- pact evaluations about jobs interventions; (iii) analyzed 13 country case studies; and (iv) ran a content analysis of the 18 available IPF and PforR Implementation Completion and Results Report Reviews. Overall, we found that the enhanced focus on jobs in the IDA strategy has been associated with only a slight increase in the size of the jobs portfolio but has led to a more pronounced change in the mix of jobs interventions. Jobs interventions are for the most part well designed, grounded in evidence, and often effectively combined within projects. The IDA jobs strategy’s prioritization of youth and women’s employment also resulted in more focus on these two beneficiary groups in jobs interventions. However, because of the inadequate results measurements, little is known about the effectiveness of the portfolio. Portfolio Evolution The enhanced focus on jobs in IDA strategy is associated with a slight in- crease in the size of the jobs portfolio. Figure 3.1 shows that IDA17 and IDA18 had higher estimated average annual shares of the IDA jobs-related portfolio in the total count and commitments of all IDA projects than IDA15 Independent Evaluation Group World Bank Group    27 and IDA16. The share of the IDA jobs-related portfolio in the total count and commitments of all IDA projects remained stable across the three IDA Replenishment rounds examined (table 3.1). DA Jobs-Related Investment Project Financings and Figure 3.1. I Programs-for-Results before and after the Shift in IDA Jobs Strategy 140 120 Index (IDA15–16 = 100) 100 World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 3 80 60 40 20 0 Average annual jobs-related project Average annual jobs-related project counts as a proportion of total IDA commitments as a proportion of total IDA Measure IDA15–16 IDA17–18 Source: Independent Evaluation Group. Note: Calculations presented are based on “synthetic” IDA jobs-related portfolios created for the pur- poses of comparison with the pre-evaluation period, and therefore they are not directly comparable with those for the “actual” evaluation portfolio presented in table 3.1. Thus, values are presented as an index with the first period being equal to 100 to show the change. IDA = International Development Association; IDA15, IDA16, IDA17, IDA18 = 15th, 16th, 17th, and 18th Replenishments of IDA.  hare of IDA Jobs-Related Investment Project Financing and Table 3.1. S Program-for-Results Portfolio in Total IDA Projects over IDA Replenishment Cycles, FY15–22 Projects IDA Commitments As a share As a share IDA Replenishment of total IDA Total (US$, of total IDA Round Total (no.) (%) millions) (%) IDA17  75  13 6,486  13  IDA18  76  12  9,140  15  IDA19  76  14  9,686  15  Total  227  13  25,312  15  Source: Independent Evaluation Group. 28 Note: FY = fiscal year; IDA = International Development Association; IDA17, IDA18, IDA19 = 17th, 18th, and 19th Replenishments of IDA. IPF and PforR projects worked primarily through the labor demand and supply channels to achieve jobs-related objectives. Supply-side interventions tended to focus on skills and training for formal sector workers, and the use of small grants, microloans, and capacity building to support the self-employed and microentrepreneurs, who were more likely to be informally or vulnerably employed workers. Demand-side interventions included support to micro, small, and medium enterprise employers in both the formal and informal sec- tors through managerial, leadership, and psychosocial training or digital skills development. Such projects also focused on strengthening financial access through grants or venture capital to support better-paying jobs and invest- ments in upskilling or reskilling workers to increase their productivity. When projects specifically targeted women or youth, they tended to include some form of labor intermediation, such as job fairs, matching services, or transport subsidies. Interventions to enhance labor market flexibility were rarer. They included interventions to improve labor information systems or labor code. Within agriculture, projects targeted on-farm productivity through a com- bination of technology transfer, capacity building, and subsidized inputs to improve outcomes for both formal and informal agricultural workers. Other projects targeted the development of the agribusiness value chain to increase production for processing or experts. These interventions aimed to improve the earnings of smallholder farmers—a critical jobs objective in itself—or the formalization of agricultural and food systems jobs. Independent Evaluation Group World Bank Group    29 The shift in the IDA jobs strategy led to a change in the mix of jobs inter- ventions in the portfolio. The evolution of the portfolio shows three distinct patterns, which reflected the shift in the IDA jobs strategy. First, there was a notable increase in the proportion of jobs-related projects that worked through the labor demand channel, such as the promotion of value chains, especially agricultural value chains, and business support to micro, small, and medium enterprises. Second, there was a notable increase in the propor- tion of jobs-related projects that sought to ensure the participation of youth and women in project activities. Third, the share of jobs-related projects that worked through the supply side remained stable across periods (figure 3.2). In line with the evidence literature, bundling labor demand and labor supply interventions has now become common practice in the portfolio. In the IPF and PforR portfolio, 87 projects (40 percent) integrated both supply- and demand-side interventions. The literature clearly finds that interventions tended to have more impact when combined. Program effectiveness for youth was generally higher if the interventions combined multiple forms of support and offered personalized assistance and follow-up services (Kluve et al. 2019). Several such examples were found in the portfolio. In the Solomon Islands Rural Development Program II, there was a dual focus on both sup- ply- and demand-side interventions, with income-generating skills courses World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 3 and market linkages for farmers and agribusinesses. The project met both its objectives and surpassed both its targets for its indicators.  hange in Proportion of Different Types of Interventions in Figure 3.2. C “Synthetic” Jobs Portfolio Youth employment Female employment Value chains Agriculture Intervention type Access to finance—MSME Skills: microenterprises Skills: SME TVET Apprenticeship Labor intermediation LIPW 0 100 200 300 400 500 600 700 800 Proportion of jobs-related projects with focus on topic (index; access to finance—MSME for IDA15–16 = 100) IDA15–16 IDA17–18 Source: Independent Evaluation Group. Note: Values presented are indexed, with the value for the intervention type “access to finance—MSME” in IDA15–16 as the base (=100). For example, the value of the intervention type “youth employment” in IDA15–16 is 300 (that is, it was three times that of “access to finance—MSME” in the same period); similarly, the value for the intervention type “female employment” in IDA17–18 was 500, implying that it was five times that of “access to finance—MSME” in IDA15–16. The actual proportions are not reported because they are based on “synthetic” portfolios and are therefore not fully aligned with values com- puted for the actual evaluation portfolio for the IDA17–18 period. IDA15, IDA16, IDA17, IDA18 = 15th, 16th, 17th, and 18th Replenishments of the International Development Association; LIPW = labor-intensive public works; MSME = micro, small, and medium enterprise; SME = small and medium enterprise; TVET = technical and vocational education and training. 30 However, working across sectors to fully integrate supply- and demand-side interventions—as recommended by jobs diagnostics—has been hard to translate into practice. The objective of the jobs diagnostic is to guide pol- icy makers and development practitioners in the design of country-specific jobs strategies comprising policy reforms, regulations, and investments to improve labor incomes and working conditions (Lachler and Merotto 2020). However, the portfolio review revealed that various GPs continued to tar- get different beneficiary pools and pursue different jobs-related objectives rather than exploiting synergies and seeking scale. The extent of estimat- ed cross-GP collaboration for projects in the evaluation portfolio varied depending on the specific pairs of GPs considered, with pockets of strong collaboration along with some that could potentially be strengthened. For the four GPs that led most jobs projects (Agriculture and Food; Education; Finance, Competitiveness, and Innovation; and Social Protection and Jobs), table 3.2 shows the proportions of their projects supported by the other GPs. For some GP pairs, there is also a higher degree of reciprocity than for others. For example, Finance, Competitiveness, and Innovation supported 28 percent of the Agriculture and Food GP’s projects, and the latter support- ed 21 percent of the former. GPs could better leverage each other’s expertise to design interventions with more impact. Collaboration data, and evidence from case studies on the prevalence of demand- and supply-side interventions in proj- ects, show that collaboration between the Agriculture and Food and the Finance, Competitiveness, and Innovation GPs, which collaborated in about Independent Evaluation Group World Bank Group    31 one-quarter of projects, was an important factor in the uptake of agribusi- ness and value chain projects. For example, the jointly designed Nepal Rural Enterprise and Economic Development Project strengthened rural market linkages, incentivized regulatory improvements, and financed services and infrastructure in support of producer organizations, agribusiness SMEs, and agritech start-ups. One potential barrier to cross-GP collaboration was likely to be obstacles in the way of (or lack of incentives for) collaboration with GPs across the Practice Groups, the administrative divisions (vice presidential units) within which the various GPs operate. The extent of cross–Practice Group collaboration, based on the same measures used for cross-GP collaboration, is presented in table 3.3 for the entire evaluation portfolio. As can be seen from this table, for the Human Development and Sustainable Development Practice Groups, there was more collaboration within than across Practice Groups. Interviews with country teams pointed to barriers to exploiting synergies across jobs projects through a more integrated approach. Collaboration was hindered, for example, by corporate incentives favoring certain GPs and task team leaders lead responsibilities, including through greater control of World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 3 budgetary resources. There were also significant differences in perspectives between GPs on how best to address jobs objectives. Finally, the jobs agenda is not managed in a centralized manner within most client governments, which can contribute to a fragmented policy dialogue, which is not condu- cive to a more integrated approach within the World Bank, where different GPs may have different government interlocutors.  ross–Global Practice Collaboration on Jobs-Related Projects Table 3.2. C by Selected Global Practices Supporting GP Finance, Social Agriculture Competitiveness, Protection Projects and Food Education and Innovation and Jobs Lead GP (no.) (%) (%) (%) (%) Agriculture and 78 n.a. 0 28 5 Food Education 34 9 n.a. 12 29 Finance, 34 21 15 n.a. 9 Competitiveness, and Innovation Social Protection 42 10 17 21 n.a. and Jobs Source: Independent Evaluation Group. Note: GP = Global Practice; n.a. = not applicable. 32  ollaboration across Practice Groups on Jobs-Related Table 3.3. C Projects Practice Groups for Supporting GP Equitable Growth, Human Sustainable Practice Groups for Projects Finance, and Development Development Lead GP (no.) Institutions (%) (%) (%) Equitable Growth, 34 24 24 38 Finance, and Institutions Human Development 78 24 35 21 Sustainable 103 29 17 52 Development Source: Independent Evaluation Group. Note: The newly formed Infrastructure Practice Group is excluded from this table (N = 11). GP = Global Practice. Although the use of development policy financing in direct support of jobs-related objectives was infrequent, the number of prior actions supporting jobs-related objectives has increased steadily across IDA Replenishments. Policy reforms mainly focused on labor regulation, gender discrimination, financial access, and agriculture, with DPO engagement on agriculture tapering off in IDA19. Figure 3.3 describes DPO prior actions by Independent Evaluation Group World Bank Group    33 type of jobs intervention, especially through reforms targeting gender-based discrimination and youth development. For instance, in Niger, the DPO supported technical and vocational education and training (TVET) reform and introduced a national policy on dual apprenticeship training to promote youth employment, including through prior actions supporting the adoption of a new labor code that describes the legal framework for dual apprentice- ships. In terms of mapping to channels of intervention, the overwhelming majority of DPO prior actions relate to labor demand, including those classi- fied as agriculture-related interventions. IEG found no evidence of a shift in mapping of prior actions across IDA Replenishment rounds.  op Areas of Interventions for Prior Actions across IDA Figure 3.3. T Replenishments 35 30 25 Prior actions (no.) 20 World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 3 15 10 5 0 IDA17 IDA18 IDA19 Labor regulation Agriculture Skills and capacity building Business support Source: Independent Evaluation Group. Note: IDA17, IDA18, IDA19 = 17th, 18th, and 19th Replenishments of the International Development Association. Development policy financing was used adequately in several operations to contribute to jobs outcomes through necessary changes in labor market reg- ulations. Concrete, well-articulated prior actions often contributed to jobs outcomes. For example, a DPO for Uzbekistan supported both labor mobility and formalization of work contracts for part-time and temporary workers through a prior action to simplify contracting procedures. The associated results indicator tracked the number of part-time and temporary employees with formal contracts. In Bangladesh, the jobs development policy credit built on collaboration with IFC and the International Labour Organization to support improved workplace safety and an improved labor code, which introduced, among other important provisions, penalties for sexual harass- ment and gender-based violence. The associated results indicator related to the number of complaints dealt with, including those related to sexual harassment and gender-based violence. In Senegal, the instrument was used to outlaw discrimination in the workplace based on gender, pregnancy, or lactation. The associated results indicator tracked the increase in share of female-led firms awarded public procurement contracts. 34 Appendix D contains additional examples of DPO jobs-related prior actions and associated results indicators. Design of Jobs Interventions Overall, the choice and design of jobs interventions was well informed by analysis. The evaluation triangulated information from the portfolio review, the structured literature review, IEG validations of staff self-evaluations, and case studies to assess the strength of the analytical underpinning of inter- ventions and the quality of their design. Table 3.4 summarizes the findings and shows that the majority of jobs interventions supported by IDA were backed by evidence from impact evaluations.  ypes of Jobs Interventions in the Portfolio and Strength of Table 3.4. T Evidence Strength of Projects Channels Type of Jobs Interventions Evidence (no.) Labor Skills: training and skills building for microentre- Mixed 80 supply preneurs; TVET; apprenticeships evidence Measures to support labor force participation Mixed 10 (for example, childcare; outreach to women and evidence marginalized groups) Labor regulations (for example, antidiscrim- Limited 3 inatory rules; antiharassment; incentivizing evidence formalization) Labor Support to agriculture value chains and agri- Strong 28 demand business evidence Independent Evaluation Group World Bank Group    35 Support to agricultural productivity (for example, Strong 43 inputs, seeds, and technology) and extension evidence services Business support and capacity building of SMEs Strong 33 evidence Access to finance for microentrepreneurs, Strong 65 SMEs, and MSMEs (through small loans and evidence grants) Labor-intensive public works, including with Mixed 12 a design focused on productive community evidence assets Both Combination of multiple labor supply and Strong 87 demand interventions (for example, skills and evidence extension services, technology transfer, and finance) (continued) Strength of Projects Channels Type of Jobs Interventions Evidence (no.) Labor Labor market information systems; labor inter- Limited 19 flexibility mediation services; employment exchanges; job evidence search assistance; transport subsidies Labor codes; worker safety and health Limited 1 evidence World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 3 Source: Independent Evaluation Group. Note: The table overlays evidence from the portfolio analysis and the structured literature review. The strength of evidence was assessed based on the search in the literature related to jobs objectives. MSME = micro, small, and medium enterprise; SME = small and medium enterprise; TVET = technical and vocational education and training. Labor Demand Interventions The shift in jobs strategy led to an increase in interventions to enhance labor demand, especially support to sector-specific value chains. Consistent with recommendations from jobs diagnostics, the World Bank increased its efforts to remove bottlenecks in sectors with high potential for private sector–led job creation. Portfolio data show that 177 projects (69 percent of the port- folio) supported labor demand interventions. A deep dive into the evolution of country portfolios in 13 IDA countries also reveals an increased incidence of value chain and sector-specific projects. For example, the value chain operations in Côte d’Ivoire, Ghana, Kenya, and Sierra Leone were the first of their kind in the countries. These projects typically combined regulato- ry simplification with business support services and access to finance, and their objective was to enhance economic diversification through targeted support to specific value chains. They were often appropriately tested in situ before committing to financing. The Kenya Industry and Entrepreneurship Project was informed by a prior impact assessment of an in-country short- term, applied, intensive technology bootcamp and drew on findings from a trust-funded open innovation pilot. A notable shift toward support to agribusiness took place to realize IDA’s priority of increasing jobs in the food system. Twenty-eight projects in the portfolio focused on promoting agribusiness or agricultural value chains. Since IDA18, country portfolios tend to combine projects with more conven- 36 tional interventions to enhance agricultural productivity (through inputs and technology transfer), with complementary projects specifically targeting agribusiness SMEs. This shift is supported by evidence from the literature (Creevey, Dunn, and Farmer 2011; Dunn 2014; Humphrey and Navas-Alemán 2010; Rutherford et al. 2016). Examples from case studies show the benefits of combining both approaches. For example, in Sierra Leone, the Smallholder Commercialization and Agribusiness Development Project was complement- ed by the Agro-Processing Competitiveness Project, whose objective was to improve the enabling environment for agribusiness by simplifying regula- tions and providing business development services and capital investment grants to firms. Several factors that would support better results for this type of interven- tion can be inferred from IEG validations of staff self-evaluations. When both farmers and processors were involved along the entire agricultural value chains, ownership and results tended to be better. Flexibility in the implementation of agricultural value chain projects was crucial to meet the needs of farmers, especially in the formation of producer groups, which play a crucial role in facilitating market access for agricultural products. The use of direct payments of grants to beneficiaries through mobile money can enhance local ownership and accessibility. Supporting nonfarm enterprises empowered beneficiaries to shift their household income away from insecure and low-paid wage employment, providing a pathway out of poverty. Labor Supply Interventions Independent Evaluation Group World Bank Group    37 Projects addressing labor supply constraints did not change fundamental- ly, with a prevalence of TVET and stand-alone skills development projects, despite mixed evidence that such interventions have sustained impact on jobs outcomes. In the portfolio, 80 projects focused on enhancing the skills and capacities of (future) workers through apprenticeship, or TVET, or on- the-job training. Typical skills enhancement interventions, especially TVET, continue to rely on public training institutes and emphasize curriculum de- velopment through consultation with private employers without attempting to stimulate markets for training providers or strengthen their accountabil- ity for results. For example, TVET projects implemented before IDA17 in low-income and lower-middle-income countries, such as Bangladesh and Ghana, had limited impact, yet newer projects largely replicated the same design. The literature shows that getting technical and vocational education right requires adequate institutional capacity (World Bank, UNESCO, and ILO 2023), and in many lower-middle-income countries, the link between techni- cal and vocational education and the labor market is currently broken. Case studies show that learners, especially women, face obstacles when looking for adequate jobs after completing technical and vocational education. They are often trapped in low-skilled jobs because of occupational segregation or World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 3 cultural and social norms. There are limited incentives for the providers of technical and vocational education to respond to the needs of the labor mar- ket because their accountability to learners and teachers remains limited. Conversely, some interventions, such as apprenticeships, show evidence of having achieved targeted outcomes but were rare in the portfolio, except in Western and Central Africa. The portfolio had only 27 apprenticeship projects, two-thirds in Western and Central African countries. A series of evaluations reported the positive effects of apprenticeships on a range of socioeconomic and well-being outcomes (Alfonsi et al. 2020; Crépon and Premand 2019), although effects for young women are less encouraging (Cho et al. 2013). The skills acquired during apprenticeship programs seem to lead to better employment outcomes. For example, the Bangladesh Recovery and Advancement of Informal Sector Employment project supported on-the-job training under informal apprenticeships through stipends to apprentices, training in life skills, testing and certification of apprentices’ knowledge, and payments to expert craftspeople for hosting apprentices (box 3.1).  xample of Apprenticeship Interventions Box 3.1. E The Niger Skills Development for Growth Project aims to improve the effectiveness of formal technical and vocational training, short-term skills development, and appren- ticeship programs in priority sectors. This additional financing scales up the parent project approved in November 2013 and focuses mainly on agriculture and livestock. The project development objective (PDO) indicator capturing progress on improving short-term skills development and apprenticeship programs has been exceeded. The percentage of out-of-school youth completing dual apprenticeship training programs (continued) 38  xample of Apprenticeship Interventions (cont.) Box 3.1. E as a result of the project’s activities was targeted to be 70 percent by November 2023 and is currently 81 percent. The share of women and girls among these youth was targeted to be 40 percent and is currently 48 percent. The Central African Republic Investment and Business Competitiveness for Employment Project aims to implement reforms to enable investment, improve access to credit, and support targeted small and medium enterprises and young workers. The PDO indicator—certified apprentices with an active economic activity six months after the program completion—measures progress against supporting workers and captures the share of trainees who secure employment at the business in which they were apprenticed. No progress has been reported so far—targets for March 2027 are 50 percent overall, of which youth are targeted to be 50 percent and women and girls 40 percent. The Chad Skills Development for Youth Employability Project aims to improve access to skills training and labor market outcomes for project beneficiaries and strength- en the technical and vocational education and training sector in Chad. Component 2 supports internships to improve the school-to-work transition and the expansion of opportunities for apprenticeships. The PDO indicator—share of beneficiaries of skills development programs who are employed (wage or self-employment) within six months of completion—captures progress toward the objectives through all activities, including apprenticeships and internships. The Ghana Jobs and Skills Project aims to support skills development and job creation. Independent Evaluation Group World Bank Group    39 Component 1 supports apprenticeship training for jobs. The PDO indicator capturing progress against the objective is the share of apprenticeship training program partici- pants who complete the program and have jobs at least six months after completion. Targets do not seem ambitious relative to baseline, with an increase of only 10 per- centage points envisaged, including in terms of share of women and girls successfully completing the program. Source: Independent Evaluation Group based on Project Appraisal Documents. In line with the literature, skills enhancement projects routinely bundled mul- tiple interventions to enhance effectiveness. In the nine IEG validations of staff self-evaluations that assessed the performance of skills enhancement projects, the design of labor supply interventions was rated high, with clear lines of sight directly to jobs in seven out of nine projects and with appropriate bundling of multiple interventions. Some projects also used technology adequately to address access barriers. For example, in Pakistan, training sessions reached World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 3 396,530 farmers by using YouTube to stream agricultural training videos through mobile agricultural cinemas. Two other factors were found to enhance the likelihood of implementation success in skills interventions. First, a strong outreach and recruitment effort is deemed necessary to ensure the mean- ingful participation of youth in programs, and community engagement with local leaders is needed to promote the involvement of minority communities. Second, a strong public-private collaboration is needed for the development and maintenance of relevant skills programs, and performance-based contracts can enhance dialogue, transparency, and results. Women and Youth Employment Interventions The IDA jobs strategy prioritization of youth and women’s employment also resulted in more focus on ensuring the participation of these two beneficia- ry groups in jobs interventions. Jobs interventions more routinely seek to improve the gender balance among beneficiaries. Projects now systematical- ly seek to reach women in jobs interventions, and they more routinely use specific incentive mechanisms to do so. For example, in Kenya and Sierra Leone, skills development projects used payment bonuses to service pro- viders to try to reach a 50 percent female beneficiary ratio. In Côte d’Ivoire, the competitive value chain operation contained a small module to finance childcare services and personal initiative training for women. The emphasis on tracking gender-disaggregated indicators also means that more is known about whether targets are achieved or not. Despite a growing body of evidence from impact evaluations, and the mag- nitude of the challenge, women-specific employment interventions remain infrequent in the portfolio. The Africa Gender Innovation Lab and the World Bank’s Gender team have conducted several studies to collate evidence on 40 interventions that improve female labor force participation rates, support female entrepreneurship, and reduce gender differentials in income and employment vulnerability (Halim, O’Sullivan, and Sahay 2023; Sahay 2023; Ubfal 2023). These evaluations show promising evidence of successful women-specific projects. For example, the Sahel Adaptive Social Protection Project, which covers multiple countries and is mainly focused on wom- en, found positive, significant, and persistent impact on off-farm business revenues and savings. Multiple interventions were undertaken, such as cash grants, savings associations, and training (life skills, micro entrepreneurship, and coaching and mentoring), as well as community sensitization on aspi- rations and social norms. Impact was high across different contexts—from rural Niger and Mauritania to urban Senegal. The Africa Gender Innovation Lab conducted randomized controlled trials in Ethiopia, the Republic of Congo, and Uganda to test interventions seeking to encourage women entrepreneurs to enter male-dominated sectors and showed that specif- ic information-sharing practices about better opportunities had the most impact. The provision of childcare has been shown to lead to significantly higher female participation, employment, financial resilience, and savings. Yet women-specific interventions remain rare in the portfolio, with only 10 having specific interventions to ensure childcare and very few working on rules and social norms to address gender gaps in employment. The World Bank has used accrued evidence from impact evaluations to effectively adapt the design of youth employment interventions. IEG’s 2012 Independent Evaluation Group World Bank Group    41 evaluation on youth employment noted the absence of a comprehensive approach to youth employment projects (World Bank 2012b). Since then, significant progress has been made in ensuring that youth employment projects bundle multiple supply-side interventions in line with what the literature recommends. Interventions that target youth tend to combine skills development, TVET, and enterprise support. Although the benefits of training programs on youth are rather weak when training is used alone (Fox and Kaul 2018; Kluve et al. 2019), a recent meta-analysis reviewing estimates from more than 200 studies found that training programs com- bining multiple forms of support and offering personalized assistance and follow-up services are very effective (Kluve et al. 2019; Puerto et al. 2022). Examples of such combinations are increasingly common in the portfolio, with a trend toward bundling cash transfer, labor-intensive public works, and entrepreneurship programs, and a growing recognition of the need to encourage graduation from safety nets through job creation (for example, Ghana, the Kyrgyz Republic, and Sierra Leone). In FCV countries, youth employment interventions are increasingly common and getting better at ensuring fairness and avoiding elite capture in beneficiary selection (for example, the Democratic Republic of Congo). Performance World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 3 Based on the limited available data, the performance ratings of the closed projects in the evaluation portfolio were slightly better than those of the rest of the IDA portfolio. Given that the evaluation period starts in FY15, rela- tively few closed projects with validated outcome ratings are available. As shown in table 1.1, there were only 43 closed IPF projects, and IEG-validated outcome ratings were only available for 18 of these at the time of writing. Of the 18, 15 (83 percent) were rated moderately satisfactory or above (fig- ure 3.4). In comparison, 72 percent of the 105 nonjobs-related closed IPF projects (with IDA financing) with IEG-validated outcome ratings were rated as moderately satisfactory or above. This indicates a relatively better performance of the jobs-related project portfolio compared with the rest of the IDA portfolio. To complement the evidence from the limited set of IEG validations of staff self-evaluations, we also conducted an analysis of proj- ect indicators and found that 75 percent of jobs-related indicators in closed projects achieved their targets (table 3.5). 42  roject Counts by Outcome Ratings in Implementation Figure 3.4. P Completion and Results Report Review 13 12 11 IDA-financed projects (no.) 10 9 8 7 6 5 4 3 2 1 0 Highly Satisfactory Moderately Moderately Unsatisfactory satisfactory satisfactory unsatisfactory ICRR development outcome rating PforR IPF DPF Source: Independent Evaluation Group. Note: DPF = development policy financing; ICRR = Implementation Completion and Results Report Review; IPF = investment project financing; PforR = Program-for-Results. ndicator Analysis Table 3.5. I No Not on Projects Progress Track On Track Achieved Independent Evaluation Group World Bank Group    43 Indicator Type (no.) (%) (%) (%) (%) By indicator category: All projects Outcome 42 29 14 7 50 Output 85 20 11 22 47 Total 127 2 12 17 48 By indicator category: Closed projects Outcome 11 9 9 0 82 Output 29 3 0 24 72 Total 40 5 3 18 75 (continued) No Not on Projects Progress Track On Track Achieved Indicator Type (no.) (%) (%) (%) (%) By indicator category: Active projects Outcome 31 35 16 10 39 Output 56 29 16 21 34 World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 3 Total 87 31 16 17 36 By indicator type: All projects Intermediate results 63 16 11 14 59 indicator PDO indicator 64 30 13 20 38 Total 127 23 12 17 48 Source: Independent Evaluation Group. Note: The sample of indicators included in this analysis covers 46 projects total: 33 active and 13 closed. The methodology is described in appendix B. PDO = project development objective. The implementation performance of the active projects followed a similar pattern. To understand the performance of active projects in jobs-related versus nonjobs-related IDA-financed projects, IEG analyzed the proportion of projects in the portfolio that were classified as “actual problem projects” at any time during the 24 months between July 2021 and June 2023. The measure, calculated for a subset of projects from each group where data were available for all 24 months, is presented in table 3.6. It can be seen that the set of active jobs-related projects fared slightly better than the rest of the IDA active portfolio. However, the indicator analysis showed that targets for only half of the jobs-related indicators in active projects that were past their Mid-Term Review were on track to be achieved, and about one-third of indi- cators had made no progress toward their targets. 44  xtent of Problem Projects in the Active IDA-Financed Jobs- Table 3.6. E Related and Nonjobs-Related Portfolios, FY15–22 Projects Classified Proportion of Projects as “Actual Problem Classified as “Actual Projects” Any Time Problem Projects” Any Projects during June 2021 and Time during June 2021 Type (no.) June 2023 (no.) and June 2023 (%) Jobs 105  26  25  related  Not jobs 624  245  39  related  Source: Independent Evaluation Group. Note: FY = fiscal year; IDA = International Development Association. Several factors helped the implementation of jobs interventions in IDA. Based on case studies evidence and mining of Implementation Completion and Results Report Reviews, the following factors, not all unique to jobs interventions, emerged as enabling successful implementation. Projects with strong institutional support for public-private partnerships and dialogue paved the way for effective collaboration between government and pri- vate sector interests. Partnerships are not limited to the public and private spheres only. Extensive diagnostic work and collaboration among donors Independent Evaluation Group World Bank Group    45 also provided a solid foundation for project planning and implementa- tion. Having a balance between short- and long-term reform needs careful identification of achievable targets based on different timelines. The imple- mentation of ambitious reforms could involve phasing in key interventions and reforms. When lessons from previous projects informed subsequent phases, especially in low-capacity or FCV environments, projects were more likely to succeed, especially when they adapted to specific security and fragility challenges. For example, in Côte d’Ivoire, the World Bank’s involve- ment began largely with labor-intensive public works and cash transfers in the immediate postconflict period but then evolved to include entrepreneur- ship training and business development services for individuals to encourage graduation from the social safety net. Three factors were recurrent in jobs interventions with failed implemen- tation. Weaknesses in project preparedness led to the abandonment or delay of interventions, mostly because of unforeseen implementation costs, underestimated staffing costs, or capacity issues in implementation agen- cies. Similarly, frequent changes in management, staff, and personnel also yielded the same result. Next, political risks related to political uncertain- ties, especially during election cycles, were not fully accounted for and had World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 3 a significantly negative impact on projects. At times, high reliance on the government or implementing agencies with low capacity to implement com- plex projects resulted in failure. Finally, the pandemic significantly affected project implementation and coordination because of lockdowns, breaks in operational continuity, and health risk factors. Jobs-related prior actions in DPOs were deemed relevant, but only one-third of operations achieved the targets associated with their results indicators. A total of 33 DPOs fell within the scope of this evaluation (that is, those with at least one jobs-related prior action approved under IDA17, IDA18, or IDA19), and of those, 22 operations have been validated or evaluated by IEG. The majority of jobs-related prior actions in these DPOs were rated satis- factory for relevance, with almost all rated at least moderately satisfactory. This suggests that prior actions supporting jobs-related objectives were part of a coherent and well-articulated results chain and represented meaning- ful progress along the results chain to the associated jobs objective. With regard to results indicators measuring progress toward jobs objectives, 16 out of 22 results indicators were rated at least moderately satisfactory for relevance. However, just over one-quarter of results indicators were given a rating of moderately unsatisfactory or below for relevance, implying that they either inadequately captured the impact of the associated prior action or inadequately measured progress toward the associated objective. In terms of results, over one-third (38 percent) of jobs-related results indicators had an achievement rating of high (that is, targets were either fully or mostly achieved), whereas just over half (56 percent) had an achievement rating of modest or negligible, suggesting little impact from the associated prior action on progress toward the associated objective. Several factors contribute to shortcomings in DPO achievements in reaching 46 jobs objectives. In several DPOs, there was a mismatch between the ambition of the reform and the institutional capacity to implement it. For example, in Tanzania, the Business Environment and Competitiveness for Jobs DPO pursued too many objectives, and in the absence of a strong political backing for the reform agenda from the newly elected administration, achievements were modest, and the subsequent operations in the series were canceled. In Burkina Faso, the Fourth Growth and Competitiveness Credit DPO had a large number of unrelated prior actions, which undermined the achievement of results and made it difficult to monitor, prioritize, and focus implementa- tion on the most critical areas. Shortcomings in the data available to measure progress inhibited the as- sessment of outcomes from World Bank–supported jobs interventions. An assessment of the results indicators in projects and operations approved during the evaluation period highlights that the majority of indicators con- tinue to capture outputs and not outcomes. The lack of comprehensive labor market information on a national scale also complicates robust tracking of progress toward jobs-related objectives and the measurement of indirect im- pacts of interventions beyond direct project beneficiaries. This suggests a clear need for enhanced measurement strategies and data collection mechanisms to adequately capture progress toward jobs-related objectives. A detailed review of indicators by type of intervention reveals the limitations of indicators in capturing jobs outcomes. Table 3.7 includes examples of outcome-oriented indicators as follows: Independent Evaluation Group World Bank Group    47 » For skills development projects, indicators typically tracked graduation or com- pletion from training and occasionally tracked the probability of employment around the time of or within six months of project closing. A relatively small number of projects tracked the private sector relevance of TVET programs—for example, through curriculum design with industry participation—but typically did not track improvements in job quality (salary, benefits, or contractual secu- rity), private sector provision, or co-financing of training. » For business and employer support programs, indicators mainly tracked fi- nancing or investment raised and business performance (sales, revenue, and number of beneficiaries). » For agriculture support and agribusiness projects, indicators mainly tracked productivity (for example, yield per hectare, fall in harvesting losses, increas- ing share of processed commodities, increase in profitability, and adoption of technology and innovation); improving value chains (for example, market linkages, value of exports, yield increases in specific value chains, percentage of farmers selling produce in the market, and percentage of farmers selling produce in value-added form); access to finance (for example, new accounts opened by farmers or firms at financial institutions and SME action plans receiving funding for business expansion); and complying with standards (for example, SMEs compliant with international standards). Indicators tracking World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 3 productivity were by far the most common.  xamples of Jobs Outcome Indicators by Types of Table 3.7. E Interventions Indicator Interventions Value chain or » Average wage growth for beneficiary firms in targeted sectors MSME support (Bangladesh) » Number of new jobs created in targeted sectors (Bangladesh) » Number of direct full-time equivalent jobs facilitated (Bangladesh) » Postcrisis survival rates of beneficiary firms (Kyrgyz Republic) Skills » Employment rates of graduates from project-supported short cours- development es (Bangladesh) » Share of youth employed or employed within six months of program completion (Côte d’Ivoire) » Share of apprenticeship or entrepreneurship program participants within jobs six months after completion (Ghana) » Share of increase in monthly average income of project beneficiaries 12 months after graduation (Côte d’Ivoire) » Percentage of youth trained or interns employed six months after completion of training (Kenya) Entrepreneurship » Number of beneficiaries with income increase of at least 30% from and livelihoods income-generating activities (Bangladesh) » Share of households reporting an increase in earnings (female; Côte d’Ivoire) » Share of beneficiaries who initiated or expanded a household enter- prise (Ghana) » Beneficiaries of economic development activities reporting an in- crease in income (Kenya) » Beneficiaries of economic opportunity activities reporting an increase in income (Rwanda) Agriculture and » Full-time jobs created under business plans financed by the project agribusiness (Honduras) 48 Source: Independent Evaluation Group. Note: MSME = micro, small, and medium enterprise. Similar challenges affect monitoring of the impact of several DPO pri- or actions, with results frameworks having significant shortcomings. In Afghanistan’s 2020 Incentive Program Development Policy Grant, however, data scarcity at the time of project evaluation meant that detailed infor- mation on results indicators was lacking for several key areas, even areas where regular policy dialogue was already happening. Similarly, in Benin, the First Fiscal Reform and Growth Credit did not integrate monitoring and evaluation into government systems because of the lack of proper defini- tion of some indicators from the outset. In Burkina Faso, the Fourth Growth and Competitiveness Credit DPO saw several delays in the implementation and resourcing of the monitoring and evaluation framework, which in turn resulted in the delay of the identification of problems and therefore the ap- plication of solutions. Independent Evaluation Group World Bank Group    49 4 | Conclusions and Recommendations This evaluation assessed the adequacy of the analytical underpinning and operational relevance of the IDA jobs strategy. IEG examined its effects on a portfolio of interventions implemented by the World Bank that directly supported jobs-related objectives. The IDA17 Replenishment cycle marked a turning point in the World Bank’s jobs agenda. Until then, jobs were seen primarily as the by-product of growth, and the main objective was to pursue jobs growth. In IDA17, jobs be- came a special theme, with an explicit recognition of the critical role played by labor markets in intermediating growth and inclusion. Since then, IDA Replenishments have had what this evaluation terms an “IDA jobs strategy,” with explicit objectives, a series of policy commitments to achieve them, and results indicators to track them. This evaluation is IEG’s first stage in tackling the jobs agenda. It focuses on the operational relevance of the evolving IDA jobs strategy and its trans- lation into World Bank support to jobs through labor supply, demand, and flexibility interventions that directly affect the labor market. This evaluation did not assess the rest of the jobs and economic transformation agenda or indirect support to jobs through private sector growth. The work of IFC in support of jobs through the IDA Private Sector Window was also outside the scope of this evaluation. A total of 257 projects were identified that con- tained jobs interventions between FY15 and FY22, for which only 18 IPF and 15 development policy financing Implementation Completion and Results Report Reviews were available. IEG found that the shift in the IDA jobs strategy had a strong analytical underpinning and successfully stimulated country analytics—through the introduction and rollout of country jobs diagnostics—to bridge important knowledge gaps. In turn, the feedback loop between the country evidence and the subsequent adjustment to the IDA jobs strategy was strong. The 50   strategy had a discernible effect on country strategies, which now better articulate their jobs objectives and the theories of change to achieve them. However, this is not yet fully reflected in the operationalization of country strategies. The jobs agenda also became more central to engagement with FCV countries, especially to promote youth employment as a key mechanism for social stability. However, the IDA jobs strategy’s promises of improved results measurement have not yet been fulfilled. Although impact evaluations are more frequent, results frameworks continue to track outputs rather than outcomes, and many countries’ lack comprehensive labor market statistics. There is an in- herent tension between the IDA jobs strategy’s ambition of better capturing jobs outcomes and the systems and incentives underlying results frame- works. Although IDA Results Measurement System Tier 2 indicators capture intermediate outcomes from IDA financing,1 at the project level, teams may be discouraged from including appropriate outcome-level indicators in results frameworks, partly over concerns about attribution. Drawing on the experience with improving the focus on gender in operations, IDA Results Measurement System Tier 3 indicators could also better capture how well IDA operations articulate and track contributions to jobs outcomes. World Bank analysis has also highlighted some of the technical difficulties in esti- mating the indirect impact of IDA’s interventions on jobs created, which is complex and data intensive.2 The enhanced focus on jobs in IDA’s strategy has been associated with a Independent Evaluation Group World Bank Group    51 slight increase in the size of the jobs portfolio funded by IDA. The share of total IDA commitments that went to projects supporting jobs averaged 13 percent. However, not all components in jobs-relevant projects were fo- cused on jobs. IEG estimated that about 60 percent of commitment amounts in jobs-related projects were used for jobs interventions, averaging 8 percent of the IDA commitment amount. The use of development policy financing prior actions in direct support of jobs-related objectives has increased steadily. However, they remain infre- quent and had impact in only one-third of validated operations. The IDA jobs strategy has led to a more substantial change in the mix of interventions. Demand-side interventions, such as business development and agriculture value chain support, have become more commonplace. The prioritization of youth and women’s employment in the IDA jobs strategy also resulted in a focus on these two beneficiary groups in jobs interven- tions. However, despite a growing body of evidence from impact evaluations, interventions that specifically seek to improve women’s employment remain rare in the portfolio, whereas youth employment projects are more common. Working across GPs to fully integrate supply- and demand-side interven- tions—as recommended by jobs diagnostics—has been hard to translate World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 4 into operations. In line with the evidence literature, bundling labor demand and labor supply interventions has now become common practice in the portfolio. However, the portfolio review revealed that various GPs contin- ue to target different beneficiary pools and pursue different jobs-related objectives rather than exploiting synergies and seeking scale. The extent of estimated cross-GP collaboration for projects in the evaluation portfolio varied depending on the specific pairs of GPs considered, with pockets of strong collaboration along with some that could potentially be strengthened. GPs could better leverage each other’s expertise to design interventions with higher impact. Where strong collaboration exists, there is evidence of improved project design and jobs focus—for example, on agriculture value chains between the Agriculture and Food and the Finance, Competitiveness, and Innovation GPs. Recommendations Based on the evidence and findings presented in the report, the evaluation makes the following two recommendations: Recommendation 1. IDA could strengthen the measurement of its contribu- tion to the achievement of jobs objectives. This can be achieved through better and more relevant corporate-level indicators, better project monitoring and evaluation, and enhanced support for country statistical systems for measur- ing labor market outcomes. Implementation of this recommendation has the potential to significantly improve learning, adaptation, and accountability. Currently, IDA’s Tier 2 indicators capture outputs, whereas Tier 3 indicators have yet to be used to incentivize better results frameworks in jobs projects, as was successfully done to encourage gender-disaggregated measurement.3 52 It would be necessary to amend guidance and procedures that discourage teams from including appropriate outcome-level indicators in results frame- works. IDA could also improve its mechanisms to support country statistical and monitoring and evaluation systems for measuring jobs outcomes. Recommendation 2. IDA could draw more systematically on jobs diagnostics to inform country-level operational engagement. This can be achieved by strengthening the ownership and use of jobs diagnostics and the integrated approach contained therein to inform policy dialogue, CPF priorities, and operational design. For example, Country Management Units could better incentivize cross-GP collaboration by drawing on diagnostics to establish an integrated and contextualized vision of policy priorities to be the basis of policy dialogue and project design. Independent Evaluation Group World Bank Group    53 1  The Results Measurement System of the International Development Association (IDA) includes three tiers of indicators. Tier 1 indicators measure high-level outcomes, Tier 2 indicators capture intermediate outcomes from IDA support, and Tier 3 indicators gauge organizational effectiveness. 2  19th Replenishment of IDA policy commitments included “conduct[ing] 20 pilots in ‘eco- nomic transformation IDA projects’ to estimate indirect and/or induced jobs” (World Bank World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Chapter 4 2020a, 90), which led to the development of a model-based estimation methodology, not currently used for project selection or for monitoring of jobs outcomes. 3  See also World Bank Group Gender Strategy Mid-Term Review: An Assessment by the Independent Evaluation Group (World Bank 2021b). Tier 3 commitments under several IDA Replenishments, along with commitments under the Corporate Scorecard, have strengthened and helped maintain country teams’ focus on gender. 54 Bibliography Alfonsi, Livia, Oriana Bandiera, Vittorio Bassi, Robin Burgess, Imran Rasul, Munshi Sulaiman, and Anna Vitali. 2020. “Tackling Youth Unemployment: Evidence from a Labor Market Experiment in Uganda.” Econometrica 88 (6): 2369–414. Amaral, Pedro S., and Erwan Quintin. 2006. “A competitive model of the informal sector.” Journal of Monetary Economics 53 (7): 1541–53. Capasso, Salvatore, Franziska Ohnsorge, and Shu Yu. 2022. “Informality and Financial Development: A Literature Review.” The Manchester School 90 (5): 587–608. Cho, Yoonyoung, Davie Kalomba, Ahmed Mushfiq Mobarak, and Victor Orozco. 2013. “Gender Differences in the Effects of Vocational Training: Constraints on Women and Drop-out Behavior.” Policy Research Working Paper 6545, World Bank, Washington, DC. Chong, Alberto, and Mark Gradstein. 2007. “Inequality and Institutions.” The Review of Economics and Statistics 89 (3): 454–65. Creevey, Lucy, Elizabeth Dunn, and Elisabeth Farmer. 2011. “Outreach, Outcomes and Sustainability in Value Chain Projects.” microReport 171, United States Agency for International Development, Washington, DC. Independent Evaluation Group World Bank Group    55 Crépon, Bruno, and Patrick Premand. 2019. “Direct and Indirect Effects of Subsidized Dual Apprenticeships.” Discussion Paper 12793, Institute of Labor Economics, Bonn. Datta, Namita, Angela Elzir Assy, Johanne Buba, Samantha Watson, Gloria La Cava, Alexandre Hugo Laure, Iftikhar Mostafa, Raphaela Beatrice Karlen, Rene Antonio Leon Solano, Branka Minic, Sara Johansson de Silva, and Ewa Sobczynska. 2018. Integration: A New Approach to Youth Employment Programs— General Guidelines for Project Teams. Washington, DC: World Bank. Docquier, Frédéric, Tobias Müller, and Joaquín Naval. 2017. “Informality and Long- Run Growth.” The Scandinavian Journal of Economics 119 (4): 1040–85. Dollar, David, and Aart Kraay. 2002. “Growth Is Good for the Poor.” Journal of Economic Growth 7 (3): 195–225. Dunn, Elizabeth. 2014. “Smallholders and Inclusive Growth in Agricultural Value Chains.” FIELD Report 18, United States Agency for International Development, Washington, DC. Fields, Gary S. 2011a. “Poverty and Low Earnings in the Developing World.” ILR World Bank Support to Jobs and Labor Market Reform through International Development Association Financing Bibliography Working Paper, Cornell University School of Industrial and Labor Relations, Ithaca. https://ecommons.cornell.edu/handle/1813/74734. Fields, Gary S. 2011b. Working Hard, Working Poor: A Global Journey. New York: Oxford University Press. Filmer, Deon, Louise Fox, Karen Brooks, Aparajita Goyal, Taye Mengistae, Patrick Premand, Dena Ringold, Siddharth Sharma, and Sergiy Zorya. 2014. Youth Employment in Sub-Saharan Africa. Washington, DC: World Bank; Paris: Agence Française de Développement. https://openknowledge.worldbank.org/han- dle/10986/16608. Fox, Louise, and Upaasna Kaul. 2018. “The Evidence Is in: How Should Youth Employment Programs in Low-Income Countries Be Designed?” Policy Research Working Paper 8500, World Bank, Washington, DC. Halim, Daniel, Michael B. O’Sullivan, and Abhilasha Sahay. 2023. “Increasing Female Labor Force Participation.” World Bank Group Gender Thematic Policy Notes Series: Evidence and Practice Note, World Bank, Washington, DC. Humphrey, John, and Lizbeth Navas-Alemán. 2010. “Value Chains, Donor Interventions and Poverty Reduction: A Review of Donor Practice.” IDS Research Reports 2010 (63): 1–106. ILO (International Labour Organization). 1999. “Decent Work.” Report of the Director-General, ILO, Geneva. https://www.ilo.org/public/english/standards/ relm/ilc/ilc87/rep-i.htm. ILO (International Labour Organization). 2016. Key Indicators of the Labour Market. 9th ed. Geneva: ILO. https://www.ilo.org/wcmsp5/groups/public/---dgreports/-- -stat/documents/publication/wcms_498929.pdf. 56 ILO (International Labour Organization). 2017. Decent Work and the 2030 Agenda for Sustainable Development. Geneva: ILO. http://ilo.org/wcmsp5/groups/public/--- dgreports/---dcomm/documents/publication/wcms_436923.pdf. ILO (International Labour Organization). 2021. “ILO Modelled Estimates and Projections Database (ILOEST).” ILOSTAT. https://ilostat.ilo.org/data. Kluve, Jochen, Susana Puerto, David Robalino, Jose M. Romero, Friederike Rother, Jonathan Stöterau, Felix Weidenkaff, and Marc Witte. 2019. “Do Youth Employment Programs Improve Labor Market Outcomes? A Quantitative Review.” World Development 114 (February): 237–53. Lachler, Ulrich, and Dino Merotto. 2020. “Theoretical Underpinnings of Jobs Diagnostics.” Working Paper 39, World Bank, Washington, DC. La Porta, Rafael, and Andrei Shleifer. 2014. “Informality and Development.” Journal of Economic Perspectives 28 (3): 109–26. Loayza, Norman V., Luis Serven, and Naotaka Sugawara. 2009. “Informality in Latin America and the Caribbean.” Policy Research Working Paper 4888, World Bank, Washington, DC. Loayza, Norman V. 2018. “Informality: Why Is It So Widespread and How Can It Be Reduced.” Research & Policy Brief 20, World Bank, Washington, DC. Merotto, Dino, Michael Weber, and Reyes Aterido. 2018. “Pathways to Better Jobs Independent Evaluation Group World Bank Group    57 in IDA Countries: Findings from Jobs Diagnostics.” Jobs Series 14, World Bank, Washington, DC. Ohnsorge, Franziska, and Shu Yu, eds. 2022. The Long Shadow of Informality: Challenges and Policies. Washington, DC: World Bank. Puerto, Susana, Chiara Curcio, Jonas Bausch, Jonathan Stöterau, and Michael Weber. 2022. The Impact of Active Labour Market Programmes on Youth: An Updated Systematic Review and Meta-Analysis. Geneva: International Labour Organization; Washington, DC: World Bank Group; The Hague: Ministry of Foreign Affairs. Rutherford, Diana Duff, Holly M. Burke, Kelly K. Cheung, and Samuel H. Field. 2016. “Impact of an Agricultural Value Chain Project on Smallholder Farmers, Households, and Children in Liberia.” World Development 83 (July): 70–83. Sahay, Abhilasha. 2023. “Closing Gender Gaps in Earnings.” World Bank Group Gender Thematic Notes Series: Evidence and Practice Note, World Bank, Washington, DC. Townsend, Robert, Rui Benfica, Ashesh Prasann, and Maria Lee. 2017. “Future of Food: Shaping the Food System to Deliver Jobs.” Working Paper 114394, World Bank, Washington, DC. World Bank Support to Jobs and Labor Market Reform through International Development Association Financing Bibliography Ubfal, Diego Javier. 2023. “What Works in Supporting Women-Led Businesses?” World Bank Group Gender Thematic Notes Series: Evidence and Practice Note, World Bank, Washington, DC. Urdal, Henrik. 2006. “A Clash of Generations? Youth Bulges and Political Violence.” International Studies Quarterly 50 (3): 607–29. Von der Goltz, Jan, Mary Hallward-Driemeier, Nadia Piffaretti, Michal Rutkowski, and Saroj Kumar Jha. 2016. “An Integrated Framework for Jobs in Fragile and Conflict Situations.” Working Paper 108777, World Bank, Washington, DC. World Bank. 2008. “Additions to IDA Resources: Fifteenth Replenishment.” Report from the Executive Directors of the International Development Association to the Board of Governors, World Bank, Washington, DC. https://thedocs.world- bank.org/en/doc/415131d585283fb45eeb7348db847da0-0410012016/original/ finalreportmarch2008.pdf. World Bank. 2010. “Special Themes for IDA16.” Board Report 54954, World Bank, Washington, DC. https://documents1.worldbank.org/curated/ en/407511468147850201/pdf/549540BR0IDA1S101Official0Use0Only1.pdf. World Bank. 2011. “Additions to IDA Resources: Sixteenth Replenishment— Delivering Development Results.” Board Report 60575, World Bank, Washington, DC. World Bank. 2012a. World Development Report 2013: Jobs. Washington, DC: World Bank. https://documents1.worldbank.org/curated/en/263351468330025810/pd- f/73068-v1-english-revised-PUBLIC-PUBDATE-10-15-12.pdf. World Bank. 2012b. Youth Employment Programs: An Evaluation of World Bank and IFC Support. Independent Evaluation Group. Washington, DC: World Bank. 58 World Bank. 2013. “Special Themes for IDA17.” Board Report 78555, World Bank, Washington, DC. https://documents1.worldbank.org/curated/ en/514971468147870158/pdf/785550BR0IDA0S00Box0377330B00OUO090.pdf. World Bank. 2014a. “Additions to IDA Resources: Seventeenth Replenishment— Maximizing Development Impact.” Board Report 86434, World Bank, Washington, DC. https://documents1.worldbank.org/curated/ en/410401468160173357/pdf/864340BR0IDA0R0C0discl0osed04010140.pdf. World Bank. 2014b. World Bank Group Assistance to Low-Income Fragile and Conflict- Affected States. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/fcs_ eval.pdf. World Bank. 2016. “IDA18 Special Theme: Jobs and Economic Transformation.” Board Report 106106, World Bank, Washington, DC. https://documents1. worldbank.org/curated/en/500371467996716841/pdf/106106-BR-IDA- SecM2016-0110-PUBLIC.pdf. World Bank. 2017a. “Additions to IDA Resources: Eighteenth Replenishment— Towards 2030: Investing in Growth, Resilience, and Opportunity.” Board Report 112728, World Bank, Washington, DC. https://documents1.worldbank.org/ curated/en/348661486654455091/pdf/112728-correct-file-PUBLIC-Rpt-from- EDs-Additions-to-IDA-Resources-2-9-17-For-Disclosure.pdf. Independent Evaluation Group World Bank Group    59 World Bank. 2017b. IDA17 Results Measurement System (RMS). Washington, DC: World Bank. https://thedocs.worldbank.org/en/doc/362611508859552486- 0290022017/original/IDA17ResultsMeasurementSystemOCTOBER2017.pdf. World Bank. 2018a. IDA17 Retrospective: Maximizing Development Impact—Leveraging IDA to Meet Global Ambitions and Evolving Client Needs. Washington, DC: World Bank. World Bank. 2018b. IDA18 Results Measurement System (RMS). Washington, DC: World Bank. https://thedocs.worldbank.org/en/doc/61eddd3dcba42bd006ac6f- 6bc602ed97-0410012020/original/ida18_rms_2018.pdf. World Bank. 2019a. “IDA19 Results Measurement System.” Board Report 139521, World Bank, Washington, DC. https://documents1.worldbank.org/curated/ en/719071563779842522/pdf/IDA19-Second-Replenishment-Meeting-The- IDA19-Results-Measurement-System.pdf. World Bank. 2019b. “IDA19 Special Theme: Jobs and Economic Transformation.” Board Report 139725, World Bank, Washington, DC. https://documents1. World Bank Support to Jobs and Labor Market Reform through International Development Association Financing Bibliography worldbank.org/curated/en/381791564075012778/pdf/Special-Theme-Jobs-and- Economic-Transformation.pdf. World Bank. 2019c. World Development Report 2019: The Changing Nature of Work. Washington, DC: World Bank. World Bank. 2020a. “Additions to IDA Resources: Nineteenth Replenishment— Ten Years to 2030: Growth, People, Resilience.” Board Report 146261, World Bank, Washington, DC. https://documents1.worldbank.org/curated/ en/459531582153485508/pdf/Additions-to-IDA-Resources-Nineteenth- Replenishment-Ten-Years-to-2030-Growth-People-Resilience.pdf. World Bank. 2020b. World Bank Group Strategy for Fragility, Conflict, and Violence 2020–2025. Washington, DC: World Bank. World Bank. 2021a. Poverty and Shared Prosperity 2022: Correcting Course. Washington, DC: World Bank. World Bank. 2021b. World Bank Group Gender Strategy Mid-Term Review: An Assessment by the Independent Evaluation Group. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2022a. “Additions to IDA Resources: Twentieth Replenishment— Building Back Better from the Crisis: Toward a Green, Resilient, and Inclusive Future.” Board Report 168910, World Bank, Washington, DC. https://documents1.worldbank.org/curated/en/163861645554924417/pdf/ IDA20-Building-Back-Better-from-the-Crisis-Toward-a-Green-Resilient-and- Inclusive-Future.pdf. World Bank. 2022b. Implementing World Bank Jobs Agenda in Agriculture Projects. Independent Evaluation Group. Washington, DC: World Bank. 60 World Bank. Forthcoming. Jobs for Development: Reframing Perspectives on Inclusive Growth for Better Jobs. Washington, DC: World Bank. World Bank, UNESCO (United Nations Educational, Scientific, and Cultural Organization), and ILO (International Labour Organization). 2023. Building Better Formal TVET Systems: Principles and Practice in Low- and Middle-Income Countries. Washington, DC: World Bank; Paris: UNESCO; Geneva: ILO. Independent Evaluation Group World Bank Group    61 APPENDIXES Independent Evaluation Group World Bank Support to Jobs and Labor Market Reform through International Development Association Financing Appendix A. Methods The evaluation answered two main evaluation questions: » Evaluation question 1: To what extent is the jobs strategy of the International Development Association (IDA) grounded in sound analytics, adaptive, and World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix A operationally relevant? » Evaluation question 2: To what extent has IDA’s strategy on jobs been trans- lated into relevant and effective jobs interventions? The evidence used to answer these evaluation questions stems from eight types of analysis. Findings emerged from the triangulation of evidence across these various types of analyses. Figure A.1 summarizes the evaluation design, and the rest of the appendix provides more details on each of the components.  ummary of Evaluation Design Figure A.1. S Source: Independent Evaluation Group. Note: CPF = Country Partnership Framework; EQ = evaluation question; FY = fiscal year; IDA = International Development Association. 64 Strategy-Level Analyses To answer the first evaluation question, the team carried out the following analyses. Structured Document Reviews Three types of structured document reviews, using customized templates, evalu- ation criteria, and ratings, were conducted to answer evaluation question 1: 1. A structured review of the 22 jobs diagnostics conducted in IDA countries was performed. Each jobs diagnostic was rated against the following criteria: » How specifically did the jobs diagnostic analyze constraints to jobs? » How thorough was the labor market analysis? » How thorough was the diagnostic for women’s employment issues? » How thorough was the diagnostic for youth employment issues? » How specific were the jobs diagnostic sector-level recommendations? » How specific were the jobs diagnostic active labor market policy–related recommendations? » How specific were the jobs diagnostic women’s employment–related recom- Independent Evaluation Group World Bank Group    65 mendations? » How specific were the jobs diagnostic youth employment–related recom- mendations? » Did the jobs diagnostic propose a prioritization strategy by sector? » Did the jobs diagnostic propose a prioritization strategy by time horizon? » Did the jobs diagnostic propose a prioritization strategy for women? » Did the jobs diagnostic propose a prioritization strategy for youth? » Did the jobs diagnostic present an integrated view of jobs constraints and solutions? 2. A coherence analysis of IDA jobs strategies across the three IDA Replenishments was conducted: we reviewed all the elements of the IDA jobs strategies (objectives, policy commitments, and results indicators) and mapped them against evidence and recommendations from jobs diagnostics to assess the strength of the analytical underpinning and the feedback loops between country analytics and subsequent iterations of the strategy. The elements of the strategies were also overlaid with evidence World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix A from the structured before-and-after review of the Country Partnership Frameworks (CPFs) to assess the level of operational relevance of the strategy. 3. An analysis of 66 CPFs in a sample of 33 IDA countries (13 case study countries and 20 randomly selected countries, 10 with and 10 without jobs diagnostics), seeking to compare the documents before and after IDA17 on five dimensions: » Does the CPF have explicit objectives on jobs? » Does the CPF Results Framework include indicators to capture progress on jobs outcomes (more, better, and more inclusive jobs)? » Does the CPF propose a coherent theory of change underpinning each jobs objective (more, better, and more inclusive jobs)? » Does the CPF have an explicit treatment of youth unemployment? » Does the CPF have an explicit treatment of female labor force participation or women’s employment challenges? Interviews We conducted interviews with 97 interviewees from Global Practices, the Jobs Group, the International Finance Corporation, Country Management teams, task team leaders, the International Labour Organization, and donors and government counterparts (deep-dive case studies only). 66 Intervention-Level Analyses To answer the second evaluation question, the evaluation team triangulated evidence from three types of analyses. Structured Literature Review The literature review is based on the conceptual framework and comprises the following interventions: technical and vocational education, training of workers and entrepreneurs, productive inclusion, labor-intensive public works, support to businesses, job search and matching, agricultural inter- ventions, and interventions focused on women.1 First, for each policy area, a recent summary or review article on the topic and all the relevant impact evaluations it cited were identified. Second, any additional studies that were mentioned in these impact evaluations but were not included in the original review articles were taken into consideration. Finally, the team performed a Google search (for example, through Google Scholar or using commonly used repositories, such as IZA [Institute of Labor Economics] or NBER [National Bureau of Economic Research]) to identify any missing articles. The review consisted mainly of studies that aimed at identifying a causal effect of the policy intervention at the center of the analysis through the construction of a counterfactual scenario, using either experimental or quasi-experimental methods (randomized controlled trials now account Independent Evaluation Group World Bank Group    67 for most evaluations on the impact of job search support and training). However, some of the included studies were observational and conducted at the country level. A few qualitative or descriptive reports were also included, although their findings should not be interpreted causally. The primary goal of the review was to analyze evidence from low-income countries. Whenever enough evidence on a given policy intervention was available, the review focused on studies from this group of countries. This was possible for worker and entrepreneur training, productive inclusion, public works, job search support, and agricultural interventions. For support to businesses, evidence from low-income countries had to be complemented with evidence from middle-income economies. For technical and vocational education, the available evidence mostly came from high-income countries. We then classified interventions across three categories, depending on the availability and strength of evidence underpinning them: (i) strong evidence characterizes interventions for which there were multiple impact evalua- tions confirming positive effects on jobs objectives in several low-income country settings; (ii) mixed evidence characterizes interventions for which there were both positive and negative findings from impact evaluations, depending on the country context; and (iii) limited evidence characterizes World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix A interventions for which there were either mostly negative findings or, more commonly, a limited number of studies available to judge impact. Portfolio Identification and Analysis Evaluation portfolio identification: Few IDA operations focus exclusive- ly on jobs, although many mention them even when their content is only peripherally jobs related. There is therefore potential for both inclusion and exclusion error. The identification of the evaluation portfolio addressed this by (i) applying the World Bank’s level 3 theme codes aligned with the conceptual framework (Job Creation, Job Quality, Youth Employment, Skills Development, Labor Market Institutions, and Active Labor Market Programs) to all projects approved in IDA-eligible countries over fiscal years (FY)15–22; (ii) manually reviewing the operations identified via theme codes to ensure that their content was relevant to the evaluation framework; (iii) conduct- ing keyword searches (followed by manual review) on part of the remaining portfolio, corresponding to the case study countries listed in the Country Case Study Selection and Analysis section in this appendix, to identify any additional operations that had been omitted from step (i); and (iv) soliciting inputs from operations (Global Practice) teams on the identified portfolio to incorporate any relevant projects that were still missing. This led to a final evaluation portfolio consisting of 216 investment project financings, 11 Programs-for-Results, and 30 development policy operations. “Synthetic” portfolio identification: In addition, to allow before-and-after analysis, the team constructed a synthetic portfolio consisting of investment project financing and Program-for-Results projects approved between FY09 and FY20, using text mining and supervised machine learning (develop- ment policy operations, additional financing, and the 19th Replenishment 68 of IDA were excluded). This was derived by (i) applying the same level 3 theme codes to all projects approved in IDA-eligible countries in that period, supplemented by keyword searches, to identify 1,472 projects from a pool of 2,004 and (ii) developing and applying a text classification model based on the manual review conducted for the evaluation portfolio and using the evaluation portfolio as a training model, which led to the identification of 334 jobs-relevant projects for the entire period (142 for FY09–14 and 192 for FY15–20). Evaluation portfolio analysis: The evaluation portfolio was used to analyze (i) project performance through project development objective outcome ratings for closed projects and the incidence of problem projects for active project, both compared with the IDA portfolio of nonjobs-related proj- ects over the same period; (ii) the composition of project development objective and intermediate indicators through manual classification into jobs- and nonjobs-related indicators, followed by the Generative Pre-trained Transformer (GPT) classification of jobs-related indicators as output or outcome; (iii) achievement of results based on the target and achieved values for project development objective and intermediate indicators; and (iv) the incidence of cross–Global Practice collaboration through analysis of “practice area (lead)/practice area (contributing)” from Project Appraisal Document data sheets. Synthetic portfolio analysis: The synthetic portfolios were used to perform a before-and-after comparison of (i) the share of jobs projects in the IDA Independent Evaluation Group World Bank Group    69 portfolio, looking at total financing from projects in either synthetic jobs portfolio as total IDA financing, and (ii) the shift in interventions between the pre-evaluation and evaluation period as the percentage of projects con- taining keywords related to search terms derived from the intervention types specified in table 2.1 of the main text. Country Case Study Selection and Analysis Thirteen in-depth country case studies were conducted to inform the an- swers to both the first and second evaluation questions. Selection of country cases: Qualitative findings were derived largely from a manual review of project documents for jobs-related projects between FY10 and FY22 for 13 countries (Bangladesh, Côte d’Ivoire, the Democratic Republic of Congo, Ghana, Haiti, Honduras, Kenya, the Kyrgyz Republic, Malawi, Nepal, Rwanda, Sierra Leone, and Tanzania). These countries were selected according to the following criteria: (i) top 15 countries with either substantial or negligible increases in female labor force participation rates over the evaluation period, (ii) top 15 countries with substantial increases or negligible increases in the youth employment rate over the evaluation period, (iii) existence of a jobs diagnostic, and (iv) existence of direct support World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix A to the achievement of jobs and labor market objectives. Case study analysis: The case study analysis investigated (i) how well coun- try strategies evolved to reflect the IDA strategy (based on a desk review of CPFs, Country Assistance Strategies, or comparable documents for 33 coun- tries since FY10); (ii) how well country portfolios evolved to reflect either the IDA strategy or country circumstances (based on a desk review of Project Appraisal Documents for all relevant projects approved in the 13 coun- tries since FY10); (iii) whether there was any credible evidence of learning in operational design (based again on a desk review of Project Appraisal Documents supplemented by Implementation Completion and Results Reports and Implementation Completion and Results Report Reviews for closed projects in the 13 countries); and (iv) whether there has been any credible evidence of impact (based on Implementation Completion and Results Reports, Implementation Completion and Results Report Reviews, and in a few cases, the most recent interim project monitoring reports for projects approved since FY15 in the 13 countries). Limitations The evaluation was subject to several noteworthy limitations: » Portfolio identification was challenging because few operations focused exclusively on jobs, although many operations mention jobs, even when their content was only peripherally jobs related. Although jobs and labor market theme codes are rigorously assigned and validated, there is potential for both inclusion and exclusion error. This risk was mitigated by a manual review of the portfolio identified and elimination of inclusion errors. The portfolio was also shared with Global Practice counterparts to avoid exclusion errors. 70 » The evaluation portfolio was relatively young, with a substantial share of projects still being implemented or having only recently been completed. This significantly reduced the team’s ability to generate findings on impact. To address this limitation, the team reviewed implementation progress based on information in the portal, Implementation Completion and Results Reports, and Implementation Supervision Reports. » Design and implementation details are often missing from project docu- ments, especially for interventions embedded in components that are not primarily about jobs. Bibliography McKenzie, David. 2017. “How Effective Are Active Labor Market Policies in Developing Countries? A Critical Review of Recent Evidence.” The World Bank Research Observer 32 (2): 127–54. Suri, Tavneet, and William Jack. 2017. “The Long-Run Poverty and Gender Impacts of Mobile Money.” Science 354 (6317): 1288–92. Independent Evaluation Group World Bank Group    71 1  We excluded other labor market areas (such as employment protection legislation, minimum wages, and labor inspection) because they do not constitute frequent areas of technical sup- port from the World Bank Group. World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix A 72 Appendix B. Portfolio Identification and Analysis This appendix describes in detail the approach to the identification and analysis of relevant projects as part of the evaluation’s methods. Evaluation Portfolio Identification The following steps were used to identify the set of jobs-related projects financed by the International Development Association (IDA) from all 2,012 IDA-financed projects approved during the evaluation period of fiscal years (FY)15–22. 1. Identification of relevant theme codes from the World Bank’s theme code taxonomy. The following level 3 theme codes were identified as being aligned with the evaluation conceptual framework: Job Creation, Job Quality, Youth Employment, Skills Development, Labor Market Institutions, and Active Labor Market Programs. 2. Identification of all IDA-financed lending projects mapped to relevant theme codes. Data on key project attributes were collected from the World Bank’s operations data systems, including on assignment of codes from the theme taxonomy. The set of all projects mapped to the afore- Independent Evaluation Group World Bank Group    73 mentioned theme codes was extracted. 3. Manual review of projects. A manual review was conducted to remove false positives from the over 700 projects identified based on theme codes. In total, 190 investment project financings (IPFs), 9 Programs-for-Results (PforRs), and 17 development policy operations (DPOs) were identified at this stage. 4. Text mining and inputs from operations. To identify any false negatives when relying on projects’ theme codes, string searches were conducted on project text fields (such as project development objectives [PDOs], com- ponent titles, prior actions, key document abstracts, and so on).1 Feedback from relevant operations units on the initial set of identified projects was also sought. In total, 26 IPFs, 2 PforRs, and 13 DPOs were identified and added to the evaluation portfolio. Based on the above steps, a final evaluation portfolio comprising 216 IPFs, 11 PforRs, and 30 DPO projects was identified, with commitments totaling $23.96 billion, $1.35 billion, and $4.39 billion, respectively. It should be noted that the process followed was not linear, with iterations between the different steps to arrive at a refined list of projects aligned with the evalua- tion conceptual framework. World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix B Evaluation Portfolio Description Descriptive summaries of the evaluation portfolio were produced for various key facets of interest to inform the portfolio analysis. Figure B.1 provides basic descriptive summaries of the evaluation portfolio (DPO commitments are excluded, and therefore commitment totals equal $25.31 billion in all figures).  escriptive Summaries of Evaluation Portfolio Figure B.1. D a. Project counts by instrument (no.) b. Project commitments by instrument (US$, millions) c. Project counts by instrument over time d. Project commitments by instrument over time Source: Independent Evaluation Group. Note: DPF = development policy financing; IDA = International Development Association; IPF = invest- ment project financing; PforR = Program-for-Results. 74  roject Counts by Instrument and Global Practice Figure B.2. P Source: Independent Evaluation Group. Note: DPF = development policy financing; IDA = International Development Association; IPF = invest- ment project financing; PforR = Program-for-Results.  roject Commitments by Instrument and Global Practice Figure B.3. P Independent Evaluation Group World Bank Group    75 Source: Independent Evaluation Group. Note: IDA = International Development Association; IPF = investment project financing; PforR = Program- for-Results.  roject Counts and Commitments by Status Figure B.4. P a. Project counts by status (no.) b. Project commitments by status (US$, millions) World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix B Source: Independent Evaluation Group.  roject Counts by Development Outcome Rating Figure B.5. P Source: Independent Evaluation Group. Note: DPF = development policy financing; ICRR = Implementation Completion and Results Report Review; IDA = International Development Association; IPF = investment project financing; PforR = Program-for-Results. 76  roject Commitments by Development Outcome Rating Figure B.6. P Source: Independent Evaluation Group. Note: ICRR = Implementation Completion and Results Report Review; IDA = International Development Association; IPF = investment project financing; PforR = Program-for-Results.  roject Counts by Overall Bank Performance Rating Figure B.7. P Independent Evaluation Group World Bank Group    77 Source: Independent Evaluation Group. Note: DPF = development policy financing; ICRR = Implementation Completion and Results Report Review; IDA = International Development Association; IPF = investment project financing; PforR = Program-for-Results.  roject Commitments by Overall Bank Performance Rating Figure B.8. P World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix B Source: Independent Evaluation Group. Note: ICRR = Implementation Completion and Results Report Review; IDA = International Development Association; IPF = investment project financing; PforR = Program-for-Results. 78  roject Counts by Development Outcome Rating and Figure B.9. P Country Independent Evaluation Group World Bank Group    79 Source: Independent Evaluation Group.  rojects by Fragility, Conflict, and Violence Status Figure B.10. P a. Project counts by FCV status (no.) b. Project commitments by FCV status (US$, millions) Source: Independent Evaluation Group. Note: Country FCV status is based on the respective year of project approval and the relevant annual World Bank FCV lists. FCV = fragility, conflict, and violence. Evaluation Portfolio Analysis Based on the descriptive summaries of the evaluation portfolio, evaluation questions, and emerging findings from other sources of evidence, analyses of specific aspects of the evaluation portfolio were carried out. The analyses are not duplicated in this appendix and can be found in tables 1.1, 1.2, 1.3, 3.1, 3.2, 3.3, and 3.5 in the main text of this evaluation. World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix B Triangulation of Projects Mapping to Intervention Types As part of the qualitative portfolio review and analysis conducted of the evaluation portfolio, a typology of interventions was developed. This typol- ogy is presented in table 3.4. Projects from the evaluation portfolio were mapped to this typology using a qualitative review of the text on project design (components and subcomponents), which was collected from project design documents. To systematically triangulate and fill any potential gaps in the coding performed manually, the following steps were used. 1. Structured search taxonomy. A taxonomy was developed that mapped a set of keywords to the various categories in a simplified version of the interventions typology. This search taxonomy included rules to only select results where certain specific combinations of words were detected within a sentence. 2. Data collection. Text on project design collected during the manual cod- ing stage, along with project text fields (such as PDOs, component titles, prior actions, key document abstracts, and so on) were collected and toke- nized into sentences. 3. Text mining. String searches were conducted to identify which sentences satisfied the relevant conditions from the search taxonomy and to map such sentences to the corresponding intervention categories. 4. Input for manual review. The results from the text mining exercise were reviewed by the team to triangulate and enhance the manual coding through the identification presence of false positives and false negatives. 80 Classification of Project Indicators as Outcomes or Outputs To understand which indicators for IPF projects in the portfolio were relat- ed to the various categories of the intervention typology, and whether they measured jobs-related outcomes or outputs, the following steps were used. 1. Data collection. The list of all PDO and intermediate outcome indicators for all 216 IPF projects in the evaluation portfolio was extracted from the World Bank’s operations data platforms, resulting in 5,492 indicators. 2. Text mining. The structured search taxonomy developed to map the inter- ventions typology to a set of keywords and Boolean rules was adapted and applied to the indicators text. A total of 815 indicators were identified. 3. Manual review. For intervention categories where the text search identi- fied more than 60 indicators, a random sample of 60 indicators was drawn for manual review. For categories with fewer than 60 indicators, all were included. A total of 329 indicators were selected for manual review. The manual review process identified 193 as being jobs related. 4. Artificial intelligence–based classification and validation. The capability of the generative artificial intelligence tool—Chat Generative Pretrained Transformer (ChatGPT)2 with GPT-4—to classify indicators as measuring outcomes or outputs was assessed on a sample of 109 indicators from the Independent Evaluation Group World Bank Group    81 evaluation portfolio that were not included in the set of 193 selected in the above sample.3 GPT-4’s classification was found to be the same as hu- man classification in 94 out of 109 cases (86 percent), and in the 15 cases where the model disagreed with human coding, it was found to have cor- rected the human coding in 8 cases (based on review by different coders). Therefore, the overall performance of the model was assessed as being satisfactory, with a margin of error likely to be equal to or even lower than that arising from inaccurate coding or disagreement in human coding. Finally, GPT-4 was leveraged to classify whether the set of jobs-related indicators measured outcomes or outputs.4 Assessment of Project Results To understand the performance of IPF projects with respect to the achieve- ment of stated targets in their results frameworks, an analysis of project indicator data was conducted using the following steps. 1. Data collection. The list of all PDO and intermediate outcome indicators World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix B for all 174 nonadditional financing IPF projects in the evaluation portfolio was extracted from the World Bank’s operations data platforms, resulting in 4,782 indicators. These data included information on the baseline value, target, and latest achieved value (as per the latest Implementation Status and Results Report) for each indicator. 2. Text mining. The structured search taxonomy developed to map the inter- ventions typology to a set of keywords and Boolean rules was adapted and applied to the indicators text.5 A total of 1,001 indicators were identified. 3. Portfolio subset. A subset of the portfolio was selected for this exercise, since the achievement of results in projects is related to its age (that is, time since project effectiveness). This subset included all closed projects and all active projects for which the Mid-Term Review date was before October 11, 2023, or, if the Mid-Term Review date was unavailable, proj- ects that were at least halfway between project effectiveness (or appraisal, if this date was missing as well) and closing. A total of 577 indicators from 102 projects were retained. 4. Manual review. All 577 indicators from the preceding step were selected for manual review, which identified 179 as being jobs related. 5. Indicators subset. A subset of the indicators identified from the preced- ing step was developed, which included only those indicators for which the baseline, target, and latest achieved values were all available and were numeric. This resulted in the final set of 127 results indicators. 6. Artificial intelligence–based classification. GPT-4 was leveraged to classify whether the set of jobs-related indicators measured outcomes or outputs. This resulted in the classification of 42 (33 percent) as outcome indicators and 85 (67 percent) as output indicators. 82 7. Typology of results achievement. For each indicator, the proportion of the target value achieved as per the latest achieved was calculated, and these proportions were mapped to the below typology. The results of the analysis are also presented in this appendix. See tables B.1, B.2, and B.3.  ypology of Results Achievement Table B.1. T Range for Proportion of Target Achieved (%) Type 0 No progress 1–49 Not on track 50–99 On track 100 or more Achieved Source: Independent Evaluation Group.  chievement of Investment Project Financing Project Table B.2. A Indicator Targets No Projects Progress Not on On Track Achieved Indicator Type (no.) (%) Track (%) (%) (%) Independent Evaluation Group World Bank Group    83 By indicator category: All projects Outcome 42 29 14 7 50 Output 85 20 11 22 47 Total 127 2 12 17 48 By indicator category: Closed projects Outcome 11 9 9 0 82 Output 29 3 0 24 72 Total 40 5 3 18 75 (continued) No Projects Progress Not on On Track Achieved Indicator Type (no.) (%) Track (%) (%) (%) By indicator category: Active projects Outcome 31 35 16 10 39 Output 56 29 16 21 34 World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix B Total 87 31 16 17 36 By indicator type: All projects Intermediate results 63 16 11 14 59 indicator PDO indicator 64 30 13 20 38 Total 127 23 12 17 48 Source: Independent Evaluation Group. Note: PDO = project development objective.  chievement of Investment Project Financing Project Table B.3. A Indicator Targets by Intervention Type—All Projects No Projects Progress Not on On Track Achieved Intervention Type (no.) (%) Track (%) (%) (%) Skills—general 92 20 13 12 55 Jobs—general 78 27 13 21 40 Youth employment 42 17 7 24 52 Women’s employment 17 41 18 35 6 Source: Independent Evaluation Group. 84 Assessment of Global Practice and Practice Group Collaboration To understand the extent of collaboration between Global Practices (GPs) and Practice Groups on jobs-related projects, information on the GP leading a project, along with those supporting it, was extracted from World Bank operations data systems for all 227 IPF projects in the evaluation portfolio. Information on the administrative mapping of GPs to Practice Groups was also extracted from the World Bank data systems and was used to identify the Practice Groups to which the lead and supporting GPs were mapped. A summary of the portfolio based on the lead GP and Practice Group is pre- sented in table 1.3. Values for contingency tables were computed based on the lead GP and Practice Group (rows) and the supporting GPs and Practice Groups (columns) and are presented in tables 3.2 and 3.3. These tables con- tain information on what proportion of projects led by a particular GP and Practice Group involved collaboration with other GPs and Practice Groups. For example, the tables show that 29 percent of the jobs-related projects led by the Education GP involved collaboration with the Social Protection and Jobs GP, and that 52 percent of projects led by GPs mapped to the Sustainable Development Practice Group involved collaboration with other GPs in the Sustainable Development Practice Group. Although the absence of collaboration among GPs does not necessarily imply Independent Evaluation Group World Bank Group    85 the lack of a multisectoral approach within projects, it serves as a useful proxy in the absence of a detailed analysis of cross-support provided by dif- ferent sector experts in project teams (this was not conducted as part of the evaluation). Thus, although this analysis presents an incomplete picture of cross-sectoral collaboration, it does provide useful insights, especially given that staff with specific specializations tend to be mapped under respective GPs. Furthermore, as shown by the scatterplot in figure B.11,6 there is a pos- itive association between the number of GPs collaborating on a project and the number of level 1 sector codes mapped to the project.  catter Plot for Count of Collaborating Global Practices Figure B.11. S and Count of Sectors World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix B Source: Independent Evaluation Group. Note: N = 126. Intercept is set to zero. GP = Global Practice. “Synthetic” Portfolio Identification To understand changes in the portfolio of IDA-financed jobs-related IPF and PforR projects after the 17th Replenishment of IDA (IDA17), it was necessary to identify this set of projects for the pre-evaluation period FY09–14 that encompassed the IDA15 and IDA16 Replenishment rounds. However, the methodology used to identify the portfolio within the evalua- tion period FY15–22 (that is, corresponding to the IDA17, IDA18, and IDA19 Replenishment rounds) could not feasibly be used for the pre-evaluation period because it involved a manual review of project documentation for hundreds of projects. Therefore, a simplified and innovative methodology relying on text mining and supervised machine learning (that is, artificial intelligence) was used to identify the FY09–22 portfolio in a manner that would preserve the internal validity of the comparison between the two subperiods. This portfolio is referred to as the “synthetic” portfolio. The fol- lowing steps were followed. 1. Theme codes and text mining. The set of projects with the relevant theme codes was extracted (see the Evaluation Portfolio Identification section in this appendix). A simplified search taxonomy aligned with the evaluation conceptual framework (and based on the one used to support 86 the identification of the evaluation portfolio) was also developed. A text search was conducted of relevant project text fields (such as PDOs, com- ponent titles, key document abstracts, and so on) to identify the set of all potentially relevant projects. In total, 1,472 projects were identified in this step from a pool of 2,004 nonadditional financing projects. 2. Development of training and test sets. The manual review of projects conducted by the evaluation team to identify projects in the evaluation portfolio (see the Evaluation Portfolio Identification section in this appen- dix) was used to build the training set for a text classification model. This training set included the project identifiers for all the projects manually reviewed by the team, concatenated text fields for each project, and a binary outcome variable indicating whether a project was relevant or not. The training set included 571 projects (observations), of which 314 had been manually labeled as relevant and 257 as not relevant. 3. Development of a text classification model. The concatenated text fields for projects in the training set were preprocessed using a text prepro- cessing pipeline that included stopword removal, lemmatization, and part-of-speech tagging. This preprocessed text was then converted to numerical representation using the term frequency–inverse document frequency (TF-IDF) weighting scheme.7 A random sample of 10 percent of the training set (57 observations) was kept aside as a test set to calculate the unbiased accuracy measure. The document term matrix with TF-IDF weights, along with the binary outcome variable, was then used to train Independent Evaluation Group World Bank Group    87 a support vector machine model.8 The model achieved an accuracy of 85 percent on the test set that had not been viewed by the model during training. 4. Development of a prediction set. Concatenated text fields for all 1,472 projects identified in step 1 were preprocessed using the text preprocess- ing pipeline and were converted to a TF-IDF weighted document term matrix. These preprocessing steps were identical (as is necessary) to the steps followed for the training set. 5. Deployment of the trained text classification model. Finally, the trained text classification model was applied to the prediction set to derive the model predicted probabilities for each project being relevant. A probabil- ity value cutoff of 0.5 was used to convert these probabilities to a binary outcome variable. Based on the model predictions, 456 (438 IPF and 18 PforR) out of 1,472 projects were included in the synthetic jobs-related portfolio for FY09–22. Although there were limitations to this approach (see the Synthetic Portfolio Analysis section in this appendix), it was grounded in methodology used to identify the evaluation portfolio, as it leveraged the same data sets (theme codes and project text fields), taxonomies (that is, theme codes and keywords), World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix B and also incorporated via supervised machine learning the human judgment on project inclusion or exclusion applied to the evaluation portfolio. Synthetic Portfolio Analysis The synthetic portfolio identified as described in the “Synthetic” Portfolio Identification section was used to assess whether there was a significant change in the counts and commitments of IDA-financed jobs-related projects after the commencement of the IDA17 Replenishment period in FY15. The results of this analysis are presented in figure 3.1. The synthetic portfolio was also used to map projects to the typology of interventions, the results of which are presented in figure 3.2. This was done using the same approach as described for the triangulation of projects mapping to intervention types for the evaluation portfolio. However, there were two differences. First, the text from project design documents manually compiled for the evaluation portfo- lio was excluded from this search because it did not include projects from the pre-evaluation period. Second, a manual review of the results of text mining for the 456 projects was impractical and therefore was not conducted. For both analyses carried out using the synthetic portfolio, two limitations were imposed to ensure a reasonable degree of alignment with the analysis carried out using the evaluation portfolio. The first was with respect to the presentation of the results. Because the various values and ratios calcu- lated for the evaluation period using the synthetic portfolio are different from those for the evaluation portfolio, the former is reported as indexes in figures 3.1 and 3.2. This is done to highlight their usefulness in understand- ing the change between the two periods rather than the actual values for the respective periods per se. The second limitation imposed pertained to the divergence between the synthetic portfolio and the evaluation portfolio for 88 the IDA19 period (FY21–22). Descriptive summaries of the synthetic portfo- lio revealed a much higher number of projects (and commitment amounts) for the IDA19 period, which was inconsistent with the trend observed in the evaluation portfolio. Therefore, the IDA19 period was excluded from the analysis of the synthetic portfolio. This also provided consistency in terms of the number of years compared between the two time periods (IDA15–16 and IDA17–18). Finally, another difference from the evaluation portfolio was that the synthetic portfolio excluded additional financing projects because of limitations in the application of the above portfolio identification meth- odology to accurately identify such projects based on their text fields and without a manual review of project papers. Independent Evaluation Group World Bank Group    89 1  A string search refers to a sequence of characters and spaces. 2  It should be noted that all the projects in the evaluation portfolio are disclosed, and their key disclosed project documents, such as Project Appraisal Documents, Implementation Status and Results Reports, and Implementation Completion and Results Reports, contain the full list of indicators. Therefore, the information provided to Chat Generative Pretrained Transformer (ChatGPT) was disclosed information, in line with the requirements of the World World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix B Bank’s information policies. 3  This coding was conducted for a separate analysis not included in this report. 4  This was done in batches of 20, with each batch entered in a new chat to avoid issues related to hallucination, which are more likely to arise in longer, multiturn conversations. 5  The search taxonomy used was a slightly expanded version of the one used for the preceding classification of indicators as outputs or outcomes. 6  Although a scatterplot with a linear regression line is not ideal to show the association between two count variables and does not provide proof of causation or accurate estimates of the regression coefficient, it is used to indicate evidence of a positive association between the two variables. 7  Text classification models work by first converting text into such numerical representations (with words as the independent variables) and then using these numerical representations to learn the association between them and the outcome variables. 8  Several trials with the generally high-accuracy decision tree models were conducted, but the support vector machine was found to have higher accuracy in this application. 90 Appendix C. Structured Literature Review  ummary of the Evidence by Type of Intervention Table C.1. S Channels Type of Jobs Intervention Strength of Evidence Literature Survey Labor supply  Skills: Training and skill build- Mixed evidence  Available studies on skills training find a positive but small effect on ing for microentrepreneurs, labor market outcomes, with some countries reporting impact only on TVET, and apprenticeships formal employment and wages. Comparisons of returns on vocation- al education in relation to academic learning are mixed, and where on-the-job training leads to higher returns, these advantages dissipate over time. Evidence is mixed on the effectiveness of scaling up TVET through publicly financed initiatives. Despite the widespread diffu- sion of competency-based training, evidence on its effectiveness is scarce. Significant knowledge gaps exist, including differences across countries in terms of the definition of competency-based training. Dual apprenticeship programs, focused on younger workers, have generat- ed positive impacts on employment outcomes, particularly earnings (Alfonsi et al. 2020; Attanasio et al. 2017; Crépon and Premand 2019). There is evidence of positive impact of skills and training programs on youth but more mixed results for women and girls, possibly because of the predominance of social norms. (continued) Independent Evaluation Group World Bank Group    91 92 World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix C Channels Type of Jobs Intervention Strength of Evidence Literature Survey   Measures to support labor Mixed evidence  Growing evidence demonstrates that daycare and preschool interven- force participation (for exam- tions can relax women’s time constraints and circumvent social and ple, childcare and outreach cultural norms related to women’s care roles, although the magnitude to women and marginalized of effects varies by context, as does perception of childcare services groups)  (Ajayi, Dao, and Koussoubé 2022; Evans, Jakiela, and Knauer 2021; Halim, Perova, and Reynolds 2021; Martínez and Perticará 2017). GIL reports positive impact in Burkina Faso from World Bank project.   Labor regulations Limited evidence  Limited evidence and impact and limited engagement in the World (for example, antidiscrimina- Bank Group. tory rules, antiharassment, incentivizing formalization)  (continued) Channels Type of Jobs Intervention Strength of Evidence Literature Survey Labor demand  Support to agriculture value Strong evidence  In Kenya, providing farmers with access to a package of services, chains and agribusiness  including access to credit, training, and marketing, led to large gains in agricultural output, yields, and profits (Deutschmann, Bernard, and Yameogo 2020) and in exports and income of small farmers (Ashraf, Giné, and Karlan 2009). Positive impact was seen in Senegal and Benin as well (Arouna, Michler, and Lokossou 2021). Increased focus was on agriculture value chains likely because of implications for sustainable development, including positive results at the farm or enterprise level, with large outreach and positive spillover effects (Creevey, Dunn, and Farmer 2011; Dunn 2014; Humphrey and Navas-Alemán 2010; Kidoido and Child 2014; Rutherford et al. 2016). According to the World Development Report 2020 (World Bank 2020), agriculture value chains are a particularly powerful factor in poverty reduction because they integrate rural households and smallholder farmers into supply chains (Madagascar and Senegal). However, this is an area where there is a marked absence of rigorous impact evaluations (Engelbert et al. 2023). (continued) Independent Evaluation Group World Bank Group    93 94 World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix C Channels Type of Jobs Intervention Strength of Evidence Literature Survey   Support to agricultural pro- Strong evidence  Interventions aimed at tackling knowledge constraints among farmers ductivity (for example, inputs, generally report positive effects (Aker and Jack 2021; Bernard et al. seeds, and technology) and 2017). Positive results are reported by Jayne et al. (2018), who find that extension services input subsidy programs increase national food production and raise both agricultural yield and household income. A meta-analysis by Hemming et al. (2018) of 31 cases found generally that included stud- ies provide evidence linking fertilizer and seed subsidies to increased use of the subsidized inputs, higher agricultural yields, and increased income among farm households but limited evidence relating to effects on poverty. Models simulating subsidy effects show the intro- duction or increase in subsidies generally results in positive effects for consumers and wider economic growth. However, the review also indicates the importance of program implementation and wider contextual factors. There is also a relatively small evidence base of both experimental and quasi-experimental studies, and econometric modeling studies, mainly from Sub-Saharan Africa, especially Malawi.   Business support and capaci- Strong evidence  Training entrepreneurs can be effective in lower-middle-income ty building of SMEs  countries, where the vast majority of enterprises are of small size and fail to follow standard business practices, which affects business per- formance (McKenzie 2021; McKenzie and Puerto 2021). Another form of business training is to provide entrepreneurs with mentorship or soft- skills training. Evaluation of these programs shows promising results (Brooks, Donovan, and Johnson 2018; Campos et al. 2017). There are positive results for women and for youth. (continued) Channels Type of Jobs Intervention Strength of Evidence Literature Survey   Access to finance for micro- Strong evidence  Success of interventions to increase firms’ access to capital seems to entrepreneurs, SMEs, and depend on the size of the targeted enterprise—larger grants targeting MSMEs (through small loans growing firms increased firms’ survival rates, including a range of firm and grants) performance indicators in Nigeria (McKenzie 2017). Kersten et al. (2017) review 16 impact evaluations on the topic and report that increas- ing access to finance has a positive impact on capital investment, employment, and firm performances of targeted SMEs. Grants outper- form microloans. Small grants to microenterprises had no significant effects on profits in Ghana (Fafchamps et al. 2014; Karlan, Knight, and Udry 2015). There is very limited evidence on the effectiveness of oth- er forms of capital access (Grover and Imbruno 2020), and this mostly comes from high-income countries. Positive results are reported from Ethiopia (Alibhai et al. 2019, 2022).   Labor-intensive public works, Mixed evidence  Positive short-term effects, especially for women and on consump- including with a design tion. Mixed results in the longer term and for employment (Azam 2012; focused on productive com- Berg et al. 2018; Bertrand et al. 2021; Pandey, Gupta, and Gupta 2019; munity assets  Zimmermann 2012). Design may matter (more impact when focus is on community productive assets and environmental projects). (continued) Independent Evaluation Group World Bank Group    95 96 World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix C Channels Type of Jobs Intervention Strength of Evidence Literature Survey Bundled Combination of multiple Strong evidence  Banerjee, Karlan, and Zinman (2015) found that multiple interventions interventions, supply- and demand-side with randomly selected households receiving a cash transfer, together including interventions primarily (for with training, life skills support, access to saving accounts, and health productive example, skills and extension information, led to improved well-being along 10 different dimen- economic services, technology transfer, sions in six countries. Similar results are obtained by Bedoya et al. inclusion and finance) (2019), who study the effects of a big-push intervention in Afghanistan. Recent evidence from a productive inclusion program delivered by the World Bank in Burkina Faso, Mauritania, Niger, and Senegal also shows positive results (Premand 2022). Bandiera et al. (2017) found that effects persisted in Bangladesh even after seven years. Such pro- ductive economic inclusion projects often target women effectively (as in the Sahel Productive Safety Nets Project). In general, findings are positive where multiple interventions target multiple constraints (especially when the objective is inclusion of women and youth). As noted under agriculture, multiple interventions were found to have more impact on output, yields, and in some cases, exports. Women benefit more consistently from a combination of access to credit and training (Bastian et al. 2018). Labor flexibility  Labor market information Limited evidence  Results vary by country context; evidence is very weak and mixed on systems, labor intermedia- public employment exchanges. tion services, employment exchanges, job search assistance, and transport subsidies    Labor codes, worker safety, Limited evidence  Evidence and engagement are limited in the World Bank Group. and health Source: Independent Evaluation Group. Note: GIL = Gender Innovation Lab; MSME = micro, small, and medium enterprise; SME = small and medium enterprise; TVET = technical and vocational education and training. Bibliography Acevedo, Paloma, Guillermo Cruces, Paul Gertler, and Sebastian Martinez. 2017. “Living Up to Expectations: How Job Training Made Women Better Off and Men Worse Off.” Working Paper 23264, National Bureau of Economic Research, Cambridge. Ajayi, Kehinde F., Aziz Dao, and Estelle Koussoubé. 2022. “The Effects of Childcare on Women and Children: Evidence from a Randomized Evaluation in Burkina Faso.” Policy Research Working Paper 10239, World Bank, Washington, DC. Aker, Jenny C., and Kelsey Jack. 2021. “Harvesting the Rain: The Adoption of Environmental Technologies in the Sahel.” Working Paper 29518, National Bureau of Economic Research, Cambridge. Alfonsi, Livia, Oriana Bandiera, Vittorio Bassi, Robin Burgess, Imran Rasul, Munshi Sulaiman, and Anna Vitali. 2020. “Tackling Youth Unemployment: Evidence from a Labor Market Experiment in Uganda.” Econometrica 88 (6): 2369–414. Alibhai, Salman, Niklas Buehren, Michael Frese, Markus Goldstein, Sreelakshmi Papineni, and Kathrin Wolf. 2019. “Full Esteem Ahead? Mindset-Oriented Business Training in Ethiopia.” Policy Research Working Paper 8892, World Bank, Washington, DC. Alibhai, Salman, Rachel Cassidy, Markus Goldstein, and Sreelakshmi Papineni. 2022. Independent Evaluation Group World Bank Group    97 “Evening the Credit Score? Impact of Psychometric Loan Appraisal for Women Entrepreneurs.” Policy Research Working Paper 10230, World Bank, Washington, DC. Arouna, Aminou, Jeffrey D. Michler, and Jourdain C. Lokossou. 2021. “Contract Farming and Rural Transformation: Evidence from a Field Experiment in Benin.” Journal of Development Economics 151 (June): 102626. Ashraf, Nava, Xavier Giné, and Dean Karlan. 2009. “Finding Missing Markets (and a Disturbing Epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya.” American Journal of Agricultural Economics 91 (4): 973–90. Attanasio, Orazio, Arlen Guarín, Carlos Medina, and Costas Meghir. 2017. “Vocational Training for Disadvantaged Youth in Colombia: A Long-Term Follow-up.” American Economic Journal: Applied Economics 9 (2): 131–43. Azam, Mehtabul. 2012. “The Impact of Indian Job Guarantee Scheme on Labor Market Outcomes: Evidence from a Natural Experiment.” Discussion Paper 6548, Institute of Labor Economics, Bonn. Bandiera, Oriana, Robin Burgess, Narayan Das, Selim Gulesci, Imran Rasul, and Munshi Sulaiman. 2017. “Labor Markets and Poverty in Village Economies.” The Quarterly Journal of Economics 132 (2): 811–70. World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix C Banerjee, Abhijit, Dean Karlan, and Jonathan Zinman. 2015. “Six Randomized Evaluations of Microcredit: Introduction and Further Steps.” American Economic Journal: Applied Economics 7 (1): 1–21. Bastian, Gautam, Iacopo Bianchi, Markus Goldstein, and Joao Montalvao. 2018. “Short-Term Impacts of Improved Access to Mobile Savings, with and without Business Training: Experimental Evidence from Tanzania.” Working Paper 478, Center for Global Development, Washington, DC. Bedoya, Guadalupe, Aidan Coville, Johannes Haushofer, Mohammad Isaqzadeh, and Jeremy P. Shapiro. 2019. “No Household Left Behind: Afghanistan Targeting the Ultra Poor Impact Evaluation.” Working Paper 25981, National Bureau of Economic Research, Cambridge. Berg, Erlend, Sambit Bhattacharyya, D. Rajasekhar, and R. Manjula. 2018. “Can Public Works Increase Equilibrium Wages? Evidence from India’s National Rural Employment Guarantee.” World Development 103 (March): 239–54. Bernard, Tanguy, Alain de Janvry, Samba Mbaye, and Elisabeth Sadoulet. 2017. “Expected Product Market Reforms and Technology Adoption by Senegalese Onion Producers.” American Journal of Agricultural Economics 99 (4): 1096–115. Bertrand, Marianne, Bruno Crépon, Alicia Marguerie, and Patrick Premand. 2021. “Do Workfare Programs Live Up to Their Promises? Experimental Evidence from Côte d’Ivoire.” Policy Research Working Paper 9611, World Bank, Washington, DC. Brooks, Wyatt, Kevin Donovan, and Terence R. Johnson. 2018. “Mentors or Teachers? Microenterprise Training in Kenya.” American Economic Journal: Applied Economics 10 (4): 196–221. 98 Campos, Francisco, Michael Frese, Markus Goldstein, Leonardo Iacovone, Hillary C. Johnson, David McKenzie, and Mona Mensmann. 2017. “Teaching Personal Initiative Beats Traditional Training in Boosting Small Business in West Africa.” Science 357 (6357): 1287–290. Creevey, Lucy, Elizabeth Dunn, and Elisabeth Farmer. 2011. “Outreach, Outcomes and Sustainability in Value Chain Projects.” microReport 171, United States Agency for International Development, Washington, DC. Crépon, Bruno, and Patrick Premand. 2019. “Direct and Indirect Effects of Subsidized Dual Apprenticeships.” Discussion Paper 12793, Institute of Labor Economics, Bonn. Deutschmann, Joshua W., Tanguy Bernard, and Ouambi Yameogo. 2020. Contracting and Quality Upgrading: Evidence from an Experiment in Senegal. Agricultural Technology Adoption Initiative. https://cega.berkeley.edu/wp-content/ uploads/2021/03/Contracting-and-quality-upgrading-Evidence-from-an- experiment-in-Senegal.pdf. Dunn, Elizabeth. 2014. “Smallholders and Inclusive Growth in Agricultural Value Chains.” FIELD Report 18, United States Agency for International Development, Washington, DC. Engelbert, Mark, Zafeer Ravat, Katherine Quant, Maciej Respekta, Fiona Kastel, Carolyn Huang, Daniel Frey, Faez Ahmed, Binyang Song, Kristen Marie Edwards, Independent Evaluation Group World Bank Group    99 Jaron Porciello, and Birte Snilstveit. 2023. “Agriculture-Led Growth in Low- and Middle-Income Countries: An Evidence Gap Map.” Evidence Gap Map Report 26, International Initiative for Impact Evaluation, Washington, DC. Evans, David K., Pamela Jakiela, and Heather A. Knauer. 2021. “The Impact of Early Childhood Interventions on Mothers.” Science 372 (6544): 794–96. Fafchamps, Marcel, David McKenzie, Simon Quinn, and Christopher Woodruff. 2014. “Microenterprise Growth and the Flypaper Effect: Evidence from a Randomized Experiment in Ghana.” Journal of Development Economics 106 (January): 211–26. Grover, Arti, and Michele Imbruno. 2020. “Using Experimental Evidence to Inform Firm Support Programs in Developing Countries.” Policy Research Working Paper 9461, World Bank, Washington, DC. Halim, Daniel, Elizaveta Perova, and Sarah Reynolds. 2021. “Childcare and Mothers’ Labor Market Outcomes in Lower- and Middle-Income Countries.” Policy Research Working Paper 9828, World Bank, Washington, DC. Hemming, David J., Ephraim W. Chirwa, Andrew Dorward, Holly J. Ruffhead, Rachel Hill, Janice Osborn, Laurenz Langer, Luke Harman, Hiro Asaoka, Chris Coffey, and Daniel Phillips. 2018. “Agricultural Input Subsidies for Improving World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix C Productivity, Farm Income, Consumer Welfare and Wider Growth in Low- and Lower-Middle-Income Countries: A Systematic Review.” Campbell Systematic Reviews 14 (1): 1–153. Humphrey, John, and Lizbeth Navas-Alemán. 2010. “Value Chains, Donor Interventions and Poverty Reduction: A Review of Donor Practice.” IDS Research Reports 2010 (63): 1–106. https://doi.org/10.1111/j.2040-0217.2010.00063_2.x. Jayne, Thomas S., Nicole M. Mason, William J. Burke, and Joshua Ariga. 2018. “Review: Taking Stock of Africa’s Second-Generation Agricultural Input Subsidy Programs.” Food Policy 75 (February): 1–14. Karlan, Dean, and Martin Valdivia. 2011. “Teaching Entrepreneurship: Impact of Business Training on Microfinance Clients and Institutions.” The Review of Economics and Statistics 93 (2): 510–27. Karlan, Dean, Ryan Knight, and Christopher Udry. 2015. “Consulting and Capital Experiments with Microenterprise Tailors in Ghana.” Journal of Economic Behavior & Organization 118 (October): 281–302. Kersten, Renate, Job Harms, Kellie Liket, and Karen Maas. 2017. “ Small Firms, Large Impact? A Systematic Review of the SME Finance Literature.” World Development 97 (September): 330–48. Kidoido, Michael, and Keith Child. 2014. Evaluating Value Chain Interventions: A Review of Recent Evidence. Nairobi: International Livestock Research Institute. Martínez, Claudia, and Marcela Perticará. 2017. “Childcare Effects on Maternal Employment: Evidence from Chile.” Journal of Development Economics 126 (May): 127–37. 100 McKenzie, David. 2017. “How Effective Are Active Labor Market Policies in Developing Countries? A Critical Review of Recent Evidence.” The World Bank Research Observer 32 (2): 127–54. McKenzie, David. 2021. “Small Business Training to Improve Management Practices in Developing Countries: Re-Assessing the Evidence for ‘Training Doesn’t Work.’” Oxford Review of Economic Policy 37 (2): 276–301. McKenzie, David, and Susana Puerto. 2021. “Growing Markets through Business Training for Female Entrepreneurs: A Market-Level Randomized Experiment in Kenya.” American Economic Journal: Applied Economics 13 (2): 297–332. Pandey, Vivek, Abhishek Gupta, and Shivani Gupta. 2019. “Labor and Welfare Impacts of a Large-Scale Livelihoods Program: Quasi-Experimental Evidence from India.” Policy Research Working Paper 8883, World Bank, Washington, DC. Premand, Patrick. 2022. “Scaling Effective Economic Inclusion Programs.” In Driving Revolutionary Ideas into Practice: Infrastructure for Climate Change, Poverty Reduction 2.0, Human Development on Mobile Government, Disrupted, 24–7. Washington, DC: World Bank. Rutherford, Diana Duff, Holly M. Burke, Kelly K. Cheung, and Samuel H. Field. 2016. “Impact of an Agricultural Value Chain Project on Smallholder Farmers, Households, and Children in Liberia.” World Development 83 (July): 70–83. World Bank. 2020. World Development Report 2020: Trading for Development in the Independent Evaluation Group World Bank Group    101 Age of Global Value Chains. Washington, DC: World Bank. Zimmermann, Laura V. 2012. “Labor Market Impacts of a Large-Scale Public Works Program: Evidence from the Indian Employment Guarantee Scheme.” Discussion Paper 6858, Institute of Labor Economics, Bonn. 102 World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix D Appendix D. Examples of Jobs Prior Actions  rior Actions by Type of Interventions Table D.1. P Intervention Examples of Prior Actions Results Indicators Business, entrepreneurship, » Financial inclusion: Access to credit for SMEs; support to leasing » Number of start-ups registered and employer support and warehousing; COVID-19 related: emergency response through » Share of SME finance in total finance employment-linked wage subsidies (formal and informal) and » Financial institutions providing leasing capitalization of credit guarantees and partial credit guarantees to and warranty services support crisis response. » Number of SMEs registered under » Risk-sharing facilities: Credit guarantee and partial credit guarantee turnover tax scheme funds (governance and capitalization) » Funding to support income-generating » Informality: Strengthened policies and guidelines for women activities for women entrepreneurs, budget allocation for schemes targeting women or youth, and regulatory reform supporting start-up funding to » Gender-disaggregated tracking of incentivize formalization by adopting a turnover-based tax regime credit access of female entrepreneurs for SMEs and self-employed (continued) Intervention Examples of Prior Actions Results Indicators Agriculture » Technology and skills: Adoption of a national policy on extension » Beneficiaries (of subsidized inputs, services to support technology use by farmers, including provision or using improved technologies or for private operators; policy for strengthened technology transfer receiving extension or e-extension through e-extension services services), typically gender disaggre- » On-farm productivity: Nationwide rollout of subsidized inputs gated and equipment through vouchers and e-vouchers; regulation to » Share of fertilizers or seeds made improve governance of fertilizer sector; regulation of irrigation available through private providers associations to define mandates and accountabilities for main- » Average increase in share of pro- tenance and management of public irrigation infrastructure; cessed crop measures to support private sector participation in production and » Share of crop produced under sustain- distribution of seeds and fertilizers able standards or other established » Value chain development: Setting up development agencies to standards support the development of agricultural value chains in identified » Usage of standardized or certified zones and to support agro-processing, funding and process estab- inputs; additional irrigated area man- lished for recruitment for trainers to provide technical assistance, aged by irrigation associations adoption of specific warehousing schemes, and establishment of windows for easing financing constraints on businesses along » Increase in value of exports or increase the value chain; adoption of regulations for the establishment of in yield standards (for example, phytosanitary or other export-related stan- » Number of financial institutions that dards); adoption of strategies for the development of specific value offer leasing and warrantage products chains; reduction or elimination of price controls to either increase farmer share in agricultural sales or incentivize exports » Financial access: Leasing law to support financial access for SMEs in agribusiness and in rural areas; funding of research in agricultural technology and strengthening links between research and farmers (continued) Independent Evaluation Group World Bank Group    103 104 World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix D Intervention Examples of Prior Actions Results Indicators Skills and training » Legal support and institutional reform: Comprehensive legal frame- » Results indicators: Number of workers work for technical and vocational education and training; National trained in partnership with private Skills Development Policy and implementing regulations to con- industries solidate a fragmented approach to skills; performance contracts » Beneficiaries of training programs in TVET; revised TVET policy with focus on inclusion of women; » Share of funding for training and num- private sector focus ber of TVET graduates » Budgeting for training: Rollout of training programs for crisis-affected » Number of NEET youth registered in sectors; strategy, including prioritization for funding for green skills dual apprenticeship programs training adopted (continued) Intervention Examples of Prior Actions Results Indicators Labor regulation » Gender related: Elimination of restrictions on women relating to » Legal assistance for expats pregnancy or lactation; prohibition of discrimination against women » Number of women at risk safely expa- based on pregnancy or childcare; improved regulation or provision triated of childcare; improved protection against sexual harassment and » Number of labor and safety complaints GBV, including through broader reforms to the labor code; labor addressed, of which percent pertain- code reforms; protections to survivors of GBV; elimination of vari- ing to sexual harassment and GBV ous legal barriers to women or other forms of discrimination » Average cost of migration through » Safety: Improved workplace safety, including from natural hazards, formal channels, licensed daycare climate change–related disasters; protection of workers, including centers, increase in share of migrants: improved regulation of work permits; legal requirement female-owned companies receiving for all workers to have written contracts, policies to promote labor public procurement contracts mobility, mainly by facilitating migration, removing restrictions and costs related to migration (external or internal) » Number of employees with part-time and temporary formal contracts » Informality: Elimination of restrictions on formal employment for example on part-time formal employment or temporary formal » Increase in formal employment based employment on payroll tax information » Indicators disaggregated by gender where available Source: Independent Evaluation Group. Note: GBV = gender-based violence; NEET = not in education, employment, or training; SME = small and medium enterprise; TVET = technical and vocational education and training. Independent Evaluation Group World Bank Group    105 Appendix E. Findings from Related Independent Evaluation Group Evaluations World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix E This appendix draws on a small number of complementary Independent Evaluation Group (IEG) evaluations. Youth Employment Programs: An Evaluation of World Bank and International Finance Corporation Support (World Bank 2012) It is related to discussion in chapter 3 on increased focus on youth and the almost universal usage of multiple interventions in projects. IEG noted in its 2012 evaluation of World Bank support to youth employment that a comprehensive, cross-sector approach embedding multiple interven- tions had been missing in the World Bank’s programs and that evidence-based diagnostics would be necessary to inform policy and program design. As such, IEG points out that the participation of the private sector in skills building, monitoring and follow-up of individual program participants, and complementary interventions, such as training with job search and place- ment assistance, tends to contribute to better jobs prospects. Interventions focused on smoothing the transition from school to work with work-based skills development tend to be most effective for youth employ- ment and earnings. However, structural factors need to be accounted for: a large informal sector, a rural context, and a low- or no-growth economy will impede generating more and better jobs, even with the World Bank’s inter- ventions. As noted in IEG’s evaluation, supporting the expansion of rural and urban jobs in the farm, nonfarm household enterprise, and rural agribusi- nesses sectors could potentially counter this. It should be noted that overall, the results framework used by projects is weak and constrains the analysis of labor market outcomes for the evaluated projects. 106 Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity (World Bank 2018) It is related to findings on data to support results measurement. According to IEG’s data for development report, the World Bank Group played a key role in strengthening clients’ statistical systems but was little involved in developing their monitoring, evaluation, and results manage- ment systems. IEG noted that the Bank Group was among the world’s largest providers of development cooperation to build the capacity of governments’ national statistical offices and country statistical systems. This type of support is mainstreamed across country engagements. All sampled Country Partnership Frameworks, except one, clearly articulated the Bank Group’s support, through projects and advisory services and analytics, for statistical capacity. In interviews, Bank Group staff said that the Bank Group does not rely on country-owned results frameworks to supply outcome evidence but instead relies on World Bank–funded project-level results frameworks, which it considers higher capacity. IEG could not identify any Country Partnership Framework that clearly uses country systems for monitoring and evaluations. Interviewed clients highlight that most development partners similarly rely on their own systems, which leads to a highly fragmented monitoring and evaluation landscape. As noted in IEG’s evaluation, this is a well-known and well-documented issue in the Sustainable Development Goal era, when Independent Evaluation Group World Bank Group    107 country-owned results frameworks and underlying monitoring and evalua- tion systems are supposed to be a core mechanism for measuring Sustainable Development Goals, yet are underperforming and overlooked by the donor community. Toward Productive, Inclusive, and Sustainable Farms and Agribusiness Firms: An Evaluation of the World Bank Group’s Support for the Development of Agrifood Economies (2010–20) (World Bank 2022b) It is related to findings on the potential for stronger reliance on agriculture to support all three jobs objectives, provided results frameworks and associ- ated data collection are strengthened. The evaluation found that Bank Group interventions supporting agrifood system development reached many countries that needed this support, including to increase agricultural productivity, improve social inclusion, and mitigate and adapt to climate change. However, support to increase access to agricultural finance, improve the enabling business environment, and enhance food safety standards was not commensurate with needs in terms of reach or intensity. Although the targeting of countries with relevant inter- World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix E ventions works quite well overall, it could be improved. In fact, although the intensity of Bank Group support (the number of interventions per country) for enhancing food safety standards, social inclusion, and climate mitiga- tion is commensurate with need, the intensity of Bank Group support to increase agricultural productivity, enhance access to agricultural finance, and improve the enabling business environment is not. In addition, only about two-thirds of countries with multiple constraints on agrifood system development received the appropriate mix of interventions. For example, the Bank Group often provided productivity-enhancing measures without support for agricultural finance in countries that needed both types of sup- port. In some cases, other development partners may be providing support in one of the two areas (or in other complementary areas), but the evaluation’s analysis was limited to Bank Group activities. Furthermore, the evaluation notes that in Western and Central Africa, only 49 percent of projects were successful in achieving their productivity outcomes because of the fragility context, ineffective extension and service delivery systems, weak producer groups and implementing capacity, inadequate market infrastructure and underdeveloped supply chains, weak midstream value-adding sectors, and high risks because of climatic shocks or conflict. Developing agrifood sys- tems and value chains to meet jobs objectives has been an explicit priority of the International Development Association since 2017. Implementing World Bank Jobs Agenda in Agriculture Projects (World Bank 2022a) It is also related to the need for strengthened data collection and results frameworks to capture the importance of the International Development Association’s investments in agriculture for jobs objectives. 108 IEG comes to the following main conclusion about World Bank–supported agricultural projects: “Crop agriculture projects have implicitly supported a JET [jobs and economic transformation] framework approach to sec- tor growth and development. The reasons are obvious—agricultural value chains provide employment opportunities to the majority of the labor force in low and lower-middle-income countries. Yet [World] Bank–supported agricultural projects have not always recognized, identified, and tracked these outcomes, despite their importance to project beneficiaries and gov- ernments” (World Bank 2022a, 13). In particular, jobs objectives are implicit, often described in terms of land productivity. Rather, jobs objectives should be explicit and data on income and yield per worker collected along with data on yield per hectare. For agricultural value chain projects, the starting point for the design of a value chain project is the value chain survey, which measures and maps the value added at each processing stage, from raw material to consumer. This involves the use of farm and firm (enterprise) survey data, as well as conducting specialized surveys of firms in the sector. This background work should help uncover, for each production stage off-farm, (i) the industry structure—how many firms there are, what the cost structure of the firms is, and how competitive they are (including a distribution of productivity), and (ii) where the jobs are, compared with the value added. World Bank Engagement in Situations of Conflict: An Evaluation of FY10–20 Independent Evaluation Group World Bank Group    109 Experience (World Bank 2021) It is related to the nature of jobs support in countries classified as fragile and conflict-affected situations (FCS). On the use of labor-intensive public works (LIPW) to support political stabiliza- tion and peace-building goals in FCS, IEG’s evaluation concludes the following. There is a need to test assumptions about the contribution of LIPW to wider stabilization and peace-building efforts in different conflict-affected settings. LIPW was the most frequently cited modality used to support stabilization aims (among many other proactive measures), including in Afghanistan, the Central African Republic, the Democratic Republic of Congo, Niger, Nigeria, and the Republic of Yemen. LIPW are used to increase state presence and dampen grievances, and, in some cases, they are intended to be used as a tool to dis- suade potential fighters (for example, youth) from joining insurgent groups. For example, the 2014 Democratic Republic of Congo Eastern Recovery Project is using 61 LIPW to prevent youth in high-risk zones from being recruited into armed groups. Although the World Bank indicates that short-term employment programs are only one small piece of the puzzle in working toward longer-term stabilization, the evidence that exists on the links between LIPW and stabili- World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix E zation goals shows only modest effects. As such, these theories and activities should be further tested and refined. As noted in the main text and in the lit- erature survey, the evidence suggests that LIPW do not support jobs objectives except in the very near term. World Bank Group Assistance to Low-Income Fragile and Conflict-Affected States (World Bank 2014) It is related to discussion in chapter 3 on the need to distinguish between jobs and short-term income stabilization in FCS countries: The Bank Group lacks a strategic and effective framework for inclusive growth and job creation in FCS: Bank Group support for long-term jobs has focused on investment climate reforms, which are necessary but not sufficient for private sector development. Synergies across the Bank Group are lacking, and frag- mented interventions reduce the potential effect on long-term employment generation. The [World] Bank has focused targeted support for jobs mainly on short-term jobs through projects supporting community-driven development and public works programs over the FY [fiscal year]01–12 period. International migration is another important livelihood strategy in many IDA [International Development Association] countries—especially in the short-term when the lo- cal economy cannot provide a sufficient number of jobs. (World Bank 2014, xx) 110 References World Bank. 2012. Youth Employment Programs: An Evaluation of World Bank and International Finance Corporation Support. Independent Evaluation Group. Washington, DC: World Bank. https://openknowledge.worldbank.org/entities/ publication/8df5da74-81fa-54c9-a698-82b7db45dd89. World Bank. 2014. World Bank Group Assistance to Low-Income Fragile and Conflict-Affected States. Independent Evaluation Group. Washington, DC: World Bank. https://openknowledge.worldbank.org/entities/publica- tion/1972bd46-a1e1-5538-85d0-b3fbd23ad579. World Bank. 2018. Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/data-for-development. World Bank. 2021. World Bank Engagement in Situations of Conflict: An Evaluation of FY10–20 Experience. Independent Evaluation Group. Washington, DC: World Bank. https://openknowledge.worldbank.org/entities/publication/579fc1e8-03d 3-522d-a1a1-c561ce96b4ad. World Bank. 2022a. Implementing World Bank Jobs Agenda in Agriculture Projects. Independent Evaluation Group. Washington, DC: World Bank. https:// documents.worldbank.org/en/publication/documents-reports/documentde- tail/099251212152216409/idu0bb992b6703586044900ad0702ea3ae848d3f. World Bank. 2022b. Toward Productive, Inclusive, and Sustainable Farms and Independent Evaluation Group World Bank Group    111 Agribusiness Firms: An Evaluation of the World Bank Group’s Support for the Development of Agrifood Economies (2010–20). Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/to- ward-productive-inclusive-and-sustainable-farms-and-agribusiness-firms. Appendix F. International Development Association Jobs Strategy Policy Commitments and Results Indicators World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix F This appendix summarizes the main components of the International Development Association jobs strategy, its relevant policy commitments, and its results indicators (tables F.1 through F.5).  7th Replenishment of IDA Policy Commitments, 2014–17 Table F.1. 1 IDA Special Theme Policy Commitment IDA17 Inclusive growth Roll out a new “jobs diagnostic tool” in at least 15 IDA countries (of which at least 5 are FCS), using multidisci- plinary micro- and macro-level data. Expand coverage of the Global Financial Inclusion da- tabase and other World Bank Group surveys, including to better measure innovative payments, mobile phone banking, and financial literacy. Establish key strategic priorities on jobs and report on the priorities and targets. Support at least 10 IDA countries to meet their financial inclusion targets and priorities through financing and technical assistance, including through the new Financial Inclusion Support Framework. Gender equality All IDA Country Partnership Frameworks incorporate gender considerations into the analysis, the content of the program, and the results framework. FCS Undertake analytical work on job creation in FCS coun- tries, including by rolling out a jobs diagnostic tool in at least 5 FCS countries (see details on the jobs diagnostic tool in the policy commitments for inclusive growth). Source: International Development Association annual reports and strategy papers. Note: FCS = fragile and conflict-affected situation; IDA = International Development Association; IDA17 = 17th Replenishment of IDA. 112  8th Replenishment of IDA Policy Commitments under Jobs Table F.2. 1 and Economic Transformation, 2017–20 IDA Special Theme Policy Commitment IDA18 JET The World Bank Group will deploy tools and resources from IDA and IFC to undertake 10 inclusive global value chain analyses in IDA countries to understand how they can contribute to economic transformation and job creation, including through growth in agribusinesses, manufacturing, and services and will use this analysis to inform activities within the IDA portfolio. The Bank Group will systematically carry out impact analyses of SME and entrepreneurship programs across IDA countries to assess their overall impacts and dif- ferentiated outcomes for women and youth and will develop operational guidelines to inform future opera- tions. The Bank Group will prepare operational guidelines for integrated youth employment programs with a focus on connecting to demand-side interventions and support- ing labor market integration to inform the design of a new generation of youth employment programs in IDA countries. The Bank Group will enhance existing and introduce new operational instruments to improve risk sharing in proj- ects and crowd in private capital in high-risk investment environments, including through the introduction of an Independent Evaluation Group World Bank Group    113 IFC-MIGA PSW. The Bank Group will adopt a “migration lens” in IDA countries where migration has a significant economic and social impact (including home, host, and transit countries); this will include analytics that close critical knowledge gaps and, where there is explicit country de- mand, support for operations that focus on job creation, managing legal economic migration, and integrating young people and economic migrants. The Bank Group will develop and make available for use in IDA countries a set of ex ante measurement tools and systems to assess the impacts of large-scale public and PPP investments targeting infrastructure and economic transformation on jobs, including pilot assessments on gender outcomes. (continued) IDA Special Theme Policy Commitment The Bank Group will catalog learnings from the jobs di- agnostics, assess how jobs diagnostics are informing the design and implementation of operations in IDA coun- tries targeting job creation and economic transformation, and recommend any changes necessary to improve the impact of the tool. World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix F The Bank Group will develop and integrate spatial perspectives into analysis of migration and urbanization trends and the impacts of infrastructure on jobs and eco- nomic transformation; this will include piloting of spatial inventory of infrastructure in five IDA countries, urban jobs accessibility assessments of 10 cities in IDA coun- tries, and spatial assessment of trends in job creation and destruction in five countries. Source: International Development Association annual reports and strategy papers. Note: IDA = International Development Association; IDA18 = 18th Replenishment of IDA; IFC = International Finance Corporation; JET = jobs and economic transformation; MIGA = Multilateral Investment Guarantee Agency; PPP = public-private partnership; PSW = Private Sector Window; SME = small and medium enterprise. 114  elected Policy Commitments under the 19th Replenishment Table F.3. S of IDA, 2021–23 IDA Special Theme Policy Commitment IDA19 JET The World Bank Group will undertake interventions in 10–15 countries to help them address bottlenecks in sectors with high potential for private sector–led job creation and economic transformation, which will be country specific and could include sectors such as agri- business, manufacturing, and others. Proposed World Bank Group actions will be grounded in diagnostics, such as the Country Private Sector Diagnostics findings and jobs diagnostics, and selected in agreement with country authorities. At least 66% of agriculture and agribusiness projects in IDA countries include support for participation in value chains with high potential for growth and jobs creation, through connecting producers to markets, technical assistance for meeting international standards and regulations, adoption of modern technology, supporting logistics, and reducing trade costs. IDA will support at least 15 IDA countries to develop their primary and secondary cities through an integrated package of support to deliver sustainable, inclusive, and productive cities with a focus on JET, including through climate-smart development, strengthening urban land management, and development of enabling infrastruc- Independent Evaluation Group World Bank Group    115 ture for job creation. Of entrepreneurship and micro, small, and medium enterprise projects, 50% will incorporate digital financial services or digital entrepreneurship elements and ensure that they address particular constraints facing women and people with disabilities. IDA will support at least 15 IDA countries, including at least 12 of those among the 30 with the lowest Human Capital Index, with programs or policies to improve skills and employability toward more and higher-quality jobs, considering the differential constraints facing young women, men, and people with disabilities. (continued) IDA Special Theme Policy Commitment IDA will embed a JET focus in all IDA country programs and the design of operations as appropriate, informed by diagnostics, such as Systematic Country Diagnostics and Country Private Sector Diagnostics, and reflected in all new IDA Country Partnership Frameworks and Performance and Learning Reviews, including enhanced World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix F use of JET results indicators. Where relevant, IDA country programs and design of operations will be informed by migration diagnostics. Under country government leadership, IDA will actively participate in country platforms to collaborate and coor- dinate with partners and stakeholders (including MDBs, development finance institutions, bilaterals, the private sector, and so on) in at least 10 IDA countries toward developing a coherent vision, a set of actions for JET, and mobilization of private finance. IDA will conduct 20 pilots in “economic transformation IDA projects” to estimate indirect or induced jobs. IFC will track direct jobs and estimates of indirect jobs associat- ed with all IFC PSW investments. Where feasible, jobs reporting will be disaggregated by the poorest quintile, gender, FCS, disability, and youth. Gender and At least 60% of IDA19 financing operations for digital development skills development will support women’s access to higher-productivity jobs, including online work. At least 30% of IDA19 infrastructure operations (trans- port, energy, and water) will include actions to create employment opportunities for women in medium- and high-skilled jobs in these sectors. Source: International Development Association annual reports and strategy papers. Note: FCS = fragile and conflict-affected situation; IDA = International Development Association; IDA19 = 19th Replenishment of IDA; IFC = International Finance Corporation; JET = jobs and economic transfor- mation; MDB = multilateral development bank; PSW = Private Sector Window. 116  elected 20th Replenishment of IDA Policy Commitments Table F.4. S under Jobs and Economic Transformation, 2023–Present IDA Special Theme Policy Commitment IDA20 JET Creating better jobs and sustainable, inclusive eco- nomic transformation in high potential sectors: Support interventions to address market failures and remove constraints in sectors with high potential for the private sector to drive sustainable and inclusive economic transformation and create better jobs, or where women and youth disproportionately work, in 20 IDA countries, of which 5 are FCS, including through upstream activi- ties, informed by data and private sector development diagnostics, such as the joint IFC–World Bank Country Private Sector Diagnostics, and selected in agreement with country authorities. Boosting agriculture productivity, value chains, and food security: Improve agricultural productivity, including through the promotion of climate-smart agriculture, and strengthen sustainable agribusiness value chains with high potential for growth and better jobs addressing modernization and food and nutrition security in 15 IDA countries, including 5 FCS, in ways that are inclusive, ex- panding training for agricultural workers to access these better jobs, and encouraging private sector opportunities. Expanding broadband access and usage for jobs of the Independent Evaluation Group World Bank Group    117 future: To close the connectivity gap, IDA will support 17 IDA countries, including those that will benefit from IFC’s support under the IDA PSW to develop digital infrastruc- ture, to increase inclusive, secure, and affordable access to and usage of broadband connectivity, among which are six landlocked countries and four small states. Gender and Scaling up on productive economic inclusion: development Incorporate specific productive economic inclusion components (for example, producer cooperatives and associations, digital finance and savings and service delivery, entrepreneurship support, social care services, regulatory frameworks, or links to market support) for women in at least 35 IDA social protection, jobs, agricul- ture, urban, or community development projects. (continued) IDA Special Theme Policy Commitment Expanding childcare: Support at least 15 IDA countries to expand access to quality, affordable childcare, especially for low-income parents. Supporting medium- and high-skilled employment opportunities for women: At least 35% of IDA20 infra- World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix F structure operations (transport, energy, and water) will include actions to create employment opportunities for women in medium- and high-skilled jobs in these sectors. Closing the gap in digital technology: At least 30 IDA20 operations in digital development, financial inclusion, and agriculture will increase women’s access to and usage of digital technology to close gender gaps in access and usage. Source: Independent Evaluation Group. Note: FCS = fragile and conflict-affected situation; IDA = International Development Association; IDA20 = 20th Replenishment of IDA; IFC = International Finance Corporation; JET = jobs and economic transfor- mation; MDB = multilateral development bank; PSW = Private Sector Window. 118 DA Results Measurement System Indicators Related to Jobs Table F.5. I Corporate Scorecard No. Indicators Tier IDA17 IDA18 IDA19 IDA20 2022 1 Median income 1 Y growth rate of bottom 40% 2 Growth rates of house- 1 Y hold expenditure or income per capita among the bottom 40% of the population and the total popula- tion (SDG 10.1.1) 3 Median growth rate 1 Y Y Y of consumption and income per capita of the bottom 40% 4 GDP per person em- 1 Y Y Y Y ployed (constant 2011 PPP, US$) 5 Nonagricultural sec- 1 Y Y Y Y tors, value added (as share of GDP) 6 Number of IDA 1 Y Independent Evaluation Group World Bank Group    119 countries without any discriminatory laws against women 7 Legal changes that 1 Y Y Y Y support gender equality over the past two years (no. of legal gender changes) 8 Ratio of female to 1 Y Y Y male labor force par- ticipation rate (%) 9 Employment-to- 1 Y population ratio 10 Employment-to- 1 Y population ratio in FCS countries (continued) Corporate Scorecard No. Indicators Tier IDA17 IDA18 IDA19 IDA20 2022 11 Youth employment-to- 1 Y Y Y Y Y population ratio (ages 15–24; %) » Youth employment-to- World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix F population ratio (ages 15–24), women (%) » Youth employment-to- population ratio (ages 15–24), men (%) 12 Youth employment-to- 1 Y Y Y Y Y population ratio (ages 15–24; %) in FCS countries » Youth employment-to- population ratio (ages 15–24), women (%) » Youth employment-to- population ratio (ages 15–24), men (%) 13 Proportion of adults 1 Y Y Y Y (15 years and older) with an account at a bank or other financial institution or with a mobile money service provided (SDG 8.10.2) 14 Proportion of adults 1 Y (15 years and older) with an account at a bank or other financial institution or with a mobile money service provided, bottom 40% (%) 15 Percentage of popu- 1 Y Y Y Y Y lation below US$1.25 a day, including in FCS countries (continued) 120 Corporate Scorecard No. Indicators Tier IDA17 IDA18 IDA19 IDA20 2022 16 Percentage of IDA 1 Y countries where growth in average income of the bottom 40% is positive and greater than growth in average income of the population 17 Bank accounts per 1 Y 1,000 adults (gender disaggregated) 18 Countries with growth 1 Y Y Y concentrated in the bottom 40% (%) 19 Farmers adopting 2 Y Y Y Y improved agricultural technology (millions) 20 Area provided with 2 Y Y Y Y Y new or improved irrigation or drainage services (hectares) 21 Beneficiaries reached 2 Y Y Y Y with financial services supported by World Independent Evaluation Group World Bank Group    121 Bank operations: » People (individuals) » Businesses 22 Beneficiaries in IDA 2 Y Y Y Y countries of jobs- focused interventions (millions) 23 People with enhanced 2 Y Y Y access to transporta- tion services (millions) 24 People provided with 2 Y Y Y Y improved urban living conditions (millions) (continued) Corporate Scorecard No. Indicators Tier IDA17 IDA18 IDA19 IDA20 2022 25 Number of active 2 Y microfinance loan accounts 26 Female beneficiaries 2 Y World Bank Support to Jobs and Labor Market Reform through International Development Association Financing  Appendix F from agriculture and rural development projects 27 Active microfinance 2 Y loan accounts for women 28 Percentage of IDA- 3 Y Y Y Y Y supported projects that demonstrate a results chain by linking gender gaps identified in analysis to spe- cific actions that are tracked in the results framework (%) 29 Percentage of IDA op- 3 Y erations that integrate gender into analysis, design, and monitoring (%) 30 Percentage of IDA- 3 Y supported operations reporting gender re- sults at completion (%) 31 Proportion of IDA 3 Y operations with core sector indicators that can be gender disag- gregated that report such data (%) 32 For projects with 3 Y gender monitoring in project design, the share that report on such indicators during implementation (%) 122 (continued) Corporate Scorecard No. Indicators Tier IDA17 IDA18 IDA19 IDA20 2022 33 IDA operations that 3 Y integrate gender into analysis, design, and monitoring in FCS countries (%) 34 For projects with 3 Y gender monitoring in project design, the share that report on such indicators during implementation in FCS countries (%) Source: Independent Evaluation Group. Note: FCS = fragile and conflict-affected situation; IDA = International Development Association; IDA17, IDA18, IDA19, IDA20 = 17th, 18th, 19th, and 20th Replenishments of IDA; PPP = purchasing power parity; SDG = Sustainable Development Goal. Independent Evaluation Group World Bank Group    123 The World Bank 1818 H Street NW Washington, DC 20433