JOBS GUIDE Issue 8 Measuring Jobs Impacts: A Decision Framework and Available Methods November 2023 MEASURING JOBS IMPACTS: A DECISION FRAMEWORK AND AVAILABLE METHODS Huw Lloyd-Ellis, Bahman Kashi and Brett Crowley © 2023 International Bank for Reconstruction and Development / The World Bank. 1818 H Street NW, Washington, DC 20433, USA. Telephone: 202-473-1000; Internet: www.worldbank.org. Some rights reserved This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) http://creativecommons.org/licenses/by/3.0/igo. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: Huw Lloyd-Ellis, Bahman Kashi and Brett Crowley. “Measuring Jobs Impacts: A Decision Framework and Available Methods” World Bank, Washington, DC. License: Creative Commons Attribution CC BY 3.0 IGO. Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations—If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Views and opinions expressed in the adaptation are the sole responsibility of the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third-party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to re-use a component of the work, it is your responsibility to determine whether permission is needed for that re-use and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. All queries on rights and licenses should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Images: © World Bank. Further permission required for reuse. Measuring Jobs Impacts: A Decision Framework and Available Methods Final Report Prepared for: World Bank Group Date: April 5, 2023 Measuring Jobs Impacts Final Report - 2023-04-05 Table of Contents Table of Contents 2 Authors & Acknowledgements 4 Executive Summary 5 Acronyms 6 Glossary 7 1. Introduction 9 A Jobs-focused Theory of Change 9 Approaches and Techniques 11 Ex Ante vs. Ex Post Evaluation 11 Counterfactuals 12 Road map 12 Table 1 A summary of the procedure for evaluating of Job Impacts 13 2. A Standardized Decision Framework for Evaluating Job Impacts 14 A: Describe the context 14 1. Baseline information 14 2. Establish the Treated entity(ies) and sector(s) 15 3. Characterize the counterfactual scenario 15 B: Specify a Jobs-focused Theory of Change for the Intervention 15 1. Identify the Job outcomes to be measured 16 2. Specify the Job Channels Framework (JCF) 17 3. Identify the key dimensions of the jobs impact measurement 19 C: Selecting and implementing the techniques to quantify expected job impacts 21 C1. Reduced-form estimation approach 22 C2. Parameterized CJC approach 24 C3. Equilibrium and multiplier modeling approach 26 3. Approaches, Techniques and Timing Considerations 31 Techniques for Quantifying job impacts 31 C1 Techniques for reduced-form estimation 31 C2 Techniques for a parameterized CJC approach 36 C3 Equilibrium and Multiplier modeling techniques 39 Ex ante vs. ex post evaluation 46 4. Application to Pilot Studies 49 PS1 Transport infrastructure and structural change in the Lake Chad region 49 PS4 IFC Investment in Mozambique Agribusiness 53 PS5 Transport infrastructure and connectivity project in Lesotho 56 Page 2 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 PS8 Energy Sector Reforms in Rwanda 58 PS9 Dar es Salaam bus rapid transit system in Tanzania 60 PS10 Pathways to Sustainable Oceans in Tonga 64 PS15 Strengthening the National Social Protection System in Angola 66 PS17 Second Agricultural Growth Project (AGPII) in Ethiopia 68 PS19 Promote access to finance, entrepreneurship and employment project in Mali 71 5. Concluding Remarks 74 Appendix A: Empirical Estimation Frameworks 75 Quasi-experimental methods requiring baseline and endline observations 75 Quasi-experimental methods based on differential treatment 76 Appendix B: Model Frameworks 79 Static models 79 Models with endogenous dynamics 84 Appendix C: Pilot Studies 87 Appendix D: Recent Related Studies 90 References 91 Page 3 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Authors & Acknowledgements Limestone Analytics Limestone is a consulting firm based in the US and Canada, specializing in the evaluation of public policy, social programs, and international development projects. The firm is recognized for combining academic rigor, state of the art methods, and international development experience to provide customized evaluation and economic analysis services and to help their clients incorporate evidence to improve the design, financing, and implementation of their projects. Information about our current and past projects can be found at: limestone-analytics.com. Authors Huw Lloyd-Ellis, PhD Senior Academic Economic Advisor, Limestone Analytics Professor of Economics, Queen’s University Bahman Kashi, PhD President, Limestone Analytics Adjunct Assistant Professor, Queen’s University Brett Crowley, BA.Sc Project Coordinator, Limestone Analytics Acknowledgements The authors acknowledge the technical and operational insights provided by the World Bank’s Jobs Group, and in particular by Theresa Osborne and Jose Romero. All errors remain the sole responsibility of the authors. Page 4 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Executive Summary What are the impacts of a particular intervention in a specific context on employment, wages, job characteristics, and related labor market outcomes? To answer these and related questions, economists and policy analysts have devised a wide variety of evaluation techniques. However, determining whether an evaluation technique is well-suited to any particular intervention is a complex exercise that requires a thorough understanding of intervention design and context, labor market structure and dynamics, and the universe of available impact measurement techniques and methodologies. Given these complexities, if a labor market impact evaluation is designed without following a systematic procedure for selecting an appropriate approach, the evaluation may fail to provide rigorous and comprehensive findings. This report introduces a standardized and transparent decision-making procedure for choosing appropriate approaches and techniques for evaluating the labor market impacts of an intervention. The decision-making procedure consists of three steps. Step A: Describing the Context. Evaluators first gather information on the baseline situation and identify the economic problems that the intervention is intended to address. The entities and sectors targeted by the intervention are clearly identified. Moreover, a counterfactual scenario is also outlined (i.e., a broad description of what is or was expected to happen in the absence of the intervention?). Step B: Specifying a Jobs-focused Theory of Change (JToC). In this step, evaluators first identify the job outcome indicators to be measured through the evaluation. Then they apply a standardized framework to describe the expected labor market impact channels and specify the sectoral, temporal, and spatial scope and granularity of the evaluation. Step C: Selecting and implementing techniques for Quantifying job impacts. The final step is a decision tree that structures the process for selecting one or a combination of labor market impact evaluation approaches, using the information gathered in steps A and B. There are three broad approaches and, within each approach, a number of techniques. Typically, one technique from each approach will be most appropriate, reflecting the inputs from A and B. However, it may often be useful to combine two, or even all, approaches. The decision-making procedure was developed by drawing on lessons from a number of recent pilot studies undertaken and financed by the World Bank as part of a policy commitment made in connection with its IDA19 replenishment. To demonstrate its utility, the decision-making procedure is implemented for several of the pilot studies, both to rationalize the choices made by their authors and to highlight potential improvements. Page 5 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Acronyms AIMM Anticipated Impact Measurement and Monitoring CBA Cost-benefit analysis CJC Core job channel DSGE Dynamic Stochastic general equilibrium EPIQ Economy-wide Private Impact Quantification FPIO Fixed-proportions input-output GE General equilibrium GTAP Global Trade Analysis Project IDA International Development Agency IFC International Finance Corporation ILO International Labour Organization I-O Input-Output PS Pilot study RS Related study SAM Social accounting matrix JToC Jobs-focused Theory of Change WBG World Bank Group WIOD World Input-Output Database Page 6 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Glossary Backward supply chain jobs impacts: Job impacts in entities supplying either treated entities or producers using or reliant on treated factor, input, or condition. Core job channels: The channel through which job impacts may be experienced by the entity treated by an intervention, the entities that directly use the goods and services produced or provided by the treated entry, and the entities that directly supply inputs to the treated entity. Consumption spillover jobs: Effects on jobs in other markets resulting from changed consumption demand by those with changed labor income. Direct jobs impacts: Jobs impacts in/for treated entity(ies) or an integrated public service. Empirical estimation: The use of data and statistical techniques to estimate the effects of an intervention on a particular outcome of interest, such as job impacts. Endogenous/exogenous variables: Variables that are assumed to be influenced by the intervention being evaluated (endogenous), and those that are assumed not to be influenced by the intervention (exogenous). Equilibrium and multiplier channel: The use of economic models that simulate the long-term and broader effects of an intervention on various sectors and markets, and their multiplier effects on the overall economy. Ex ante and ex post economic evaluation: The evaluation of an intervention before it is implemented (ex ante), and after it has been implemented (ex post), to determine its impact on a particular outcome of interest, such as job impacts. Experimental and quasi-experimental techniques: Statistical techniques and methods used to estimate the effects of an intervention using a control group or natural experiment. Extrapolation: The use of data and statistical techniques to project future outcomes based on past trends. Factor markets: The market for inputs or factors of production, such as labor, capital, land, and natural resources. Forward factor usage: Jobs impacts in/for productive entities using or experiencing treated factor, input, or condition. Page 7 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Indirect job impacts: Jobs impacts beyond those in/for treated entity(ies) or an integrated public service. Jobs-focused Theory of Change: A standardized framework used to identify the main channels through which jobs impacts might occur and the key dimensions along which they may be measured, and develop a strategy to measure jobs impacts. Jobs impacts: The change in job-related outcomes (such as employment, wages, and working conditions) that is attributable to an intervention. Job impacts measurement approach: The three approaches are reduced-form empirical estimation, core job channels analysis, and the application of equilibrium and multiplier models. These approaches are not mutually exclusive and capture different but overlapping aspects of the job impacts. Each approach may adopt one or more analytical techniques. Job impacts measurement strategy: The selection of one or more approaches used to measure the jobs impacts of a particular intervention. Job impacts measurement technique: A specific analytical procedure used to estimate the jobs impacts of a particular intervention. Key measurement dimensions (sectoral, temporal, spatial): Contextual details specific to an intervention which help to determine the selection of a suitable jobs impact measurement strategy. Labor market impacts: The change in labor market-related outcomes (such as labor force participation and labor productivity) that is attributable to an intervention. Non-experimental techniques: Statistical techniques and methods used to estimate the effects of an intervention without a control group or randomization. Primary jobs impact channel: The jobs channel through which an intervention is expected to have the greatest impact on jobs outcomes. Quantified economic models: Economic models that simulate the impacts of interventions on labor market outcomes. Supply chain: The chain of businesses or suppliers that provide goods or services to a company or industry. Page 8 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 1. Introduction What are the impacts of a particular intervention in a specific context on employment, wages, job characteristics, and related labor market outcomes? In their attempts to answer these and related questions, economists and policy analysts have adopted or devised a wide variety of evaluation approaches. However, determining whether an evaluation approach is well-suited to any particular intervention is a complex exercise that requires a thorough understanding of intervention design and context, labor market structure and dynamics, and the universe of available impact measurement approaches and techniques. Given these complexities, if a labor market impact evaluation is designed without following a systematic procedure for selecting appropriate techniques, the evaluation may fail to provide rigorous and comprehensive findings. This report proposes a standardized and transparent decision-making procedure for choosing appropriate techniques for evaluating the labor market impacts of an intervention.1 The decision-making procedure was developed by drawing on lessons from a number of recent pilot studies undertaken and financed by the World Bank as part of a policy commitment made in connection with its IDA19 replenishment.2 Of particular interest here is the quantification of job impacts over and above those experienced by the entity(ies) in the sectors and region(s) receiving treatment. Such indirect job impacts include those arising along the supply chains connected to directly affected sectors and geographical areas, as well as those stemming from resulting changes in spending patterns, re-allocations of labor and economy-wide price adjustments. These impacts are inherently difficult to measure directly and often require knowledge of the details of the production processes involved and characterizations of the behavioral responses of households and employers.3 A Jobs-focused Theory of Change An important part of any evaluation design is the specification of the intervention’s Theory of Change (ToC). The ToC describes the main channels through which impacts are 1 While there exist other methodological overviews (e.g. ILO, 2020), this report is intended as an accessible, non-technical survey that focuses on the validity of different approaches in varying contexts and the thought-process involved in selecting the most appropriate methods. 2 Policy Commitment 12 reads: IDA will conduct 20 pilots in ‘economic transformation IDA projects’ to estimate indirect and/or induced jobs. The IFC will track direct jobs and estimates of indirect jobs associated with all IFC PSW investments. Where feasible, jobs reporting will be disaggregated by the poorest quintile, gender, FCS, disability and youth. 3 Most of the pilot studies discussed here were not originally designed specifically to obtain estimates of indirect effects or decompose overall effects into direct and indirect ones. Page 9 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 expected to occur, sets limits on the likely extent of significant economic “ripple effects,” and identifies the dimensions along which those impacts should be measured. However, in the pilot studies reviewed here and in other evaluations, ToCs are specified in a wide variety of ways. In some cases, they are extremely detailed whereas in others they are loosely described, or even non-existent. The various direct and indirect job outcomes are not always systematically delineated. The labor market literature employs a broad range of terms to describe labor market impacts, which can result in ambiguous descriptions of labor market impacts. To address these issues we first propose a template for systematically developing a Jobs-focused Theory of Change (JToC). This consists of three components: a specification of the job outcomes to be measured; a summary of the expected channels through which these outcomes are expected to be affected; and a specification of the key dimensions (sectoral, spatial and temporal) over which they are to be evaluated. The expected channels of impact are summarized by a Job Channels Framework (JCF), which is depicted on Figure 1-1 below. The JCF consists of two categories of job impact channels: core job channels (CJC) and equilibrium and multiplier job channels. Figure 1-1. Job Channels Framework Page 10 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 A clearly specified and well-structured JToC is necessary to assess the appropriateness of alternative evaluation approaches. Note that ‘appropriateness’ is defined here in terms of the dimensions and the likely validity of key embedded assumptions, rather than by comparing estimated impacts against the “true impacts”, which are ultimately unknown. Approaches and Techniques There are three broad approaches to quantifying job impacts all of which are used in the pilot studies: reduced-form empirical estimation, core job channels (CJC) analysis, and the application of equilibrium and multiplier models. These approaches are not mutually exclusive and capture different but overlapping aspects of the job impacts. As a result, it may be necessary to implement two, or even all three, of the broad approaches in order to fully assess the relevant job impacts. Within each broad approach there is a wide variety of techniques for quantifying impacts. The appropriateness of a technique in a specific context depends on both the specification of the JToC and the availability of resources (i.e. data, time, human resources). Ex Ante vs. Ex Post Evaluation An important distinction between economic evaluations is whether they are being conducted prior to the intervention (ex ante) or after the intervention has taken place (ex post). Obviously, the key difference is the potential existence of new data and information arising from the impacts of the intervention itself. However, there are several reasons why this difference may not always be as stark as one might expect. First, not all outcomes that are deemed important can easily be measured. Secondly, even those that are measured may not be measured very precisely. Thirdly, due to significant time lags, measurement may often take place before all (future) impacts could have occurred. Finally, in order to correctly measure the impacts of an intervention it is necessary to have some estimate of what would have happened to those affected in its absence. Not all techniques are well suited to establish a clear counter-factual. A consequence of these issues is that, in practice, all three approaches for quantifying job impacts remain relevant for both ex ante and ex post evaluations. Ex post evaluations should benefit from more relevant and credible empirical estimates based on data from the actual intervention. However, to estimate the full job impacts they must still rely to some extent on assumptions, models and imported parameter estimates. The balance between the application of direct empirical estimation and the use of model frameworks for ex post evaluation ultimately depends on the quality and availability of relevant data gathered before, during and after the intervention. The results of high quality ex post evaluations of interventions can and should be an important input into ex ante evaluations of subsequent similar interventions. In their absence, evaluators must identify alternative, and often less Page 11 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 compelling, sources of evidence and rely on strong assumptions to bridge data gaps. Conversely, the process of developing a well-structured ex ante evaluation can provide useful knowledge for subsequent ex post evaluations. Counterfactuals A theme that arises throughout this document is the importance of counterfactuals in assessing the impacts of interventions. That is, what would have been expected to happen to the various job outcomes in the absence of the intervention. This is not typically just the baseline situation. Would employers have expanded production and hired more workers over the period of a subsidy, for example, even if that subsidy had not been provided? This is an important question that is not always clearly specified in the pilot studies reviewed here, but should be. As will become apparent, how counterfactuals are dealt with varies with the approach. In applying quasi-experimental empirical techniques discussed there is often an implicit set of counterfactual assumptions “built in” that may or may not be valid, depending on the context. In the standard analysis of core job channels and some fixed-price multiplier models, the estimated impacts are essentially multiples of the estimated direct job impacts. So it is important that a quantified counterfactual for these direct effects is clearly specified, even when endline data are captured through ex post surveys. Finally, for models incorporating equilibrium and economy-wide impacts, a path for the economy in the absence of the intervention(s) should be specified and the impacts of the intervention should be measured in comparison to that path. Road map The remainder of this document is structured as follows. In Section 2, the standardized procedure is laid out which provides a transparent structure to guide jobs impact evaluation design. The main elements of this procedure are outlined in Table 1. Throughout, we link key references and decision nodes with more detailed and technical discussion of the various approaches and techniques that is contained in Section 3, and Appendices A and B. To demonstrate the utility of the decision framework, Section 4 applies the procedure to a number of representative pilot studies and uses it to rationalize the choices made by their authors and to highlight potential improvements. Relatedly, Appendix C lists all the pilot studies that have informed this report, identifying the main approach(es) taken in each case. Finally, Appendix D references a few other studies that used some of these techniques that were not exemplified by the pilot studies, but which are referenced in the document. Page 12 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Table 1: A summary of the procedure for evaluating of Job Impacts A: Describe the Context 1. Baseline information and context 2. Identifying the treated entities and sectors 3. Counterfactual scenario B: Specify a Jobs-focused Theory of Change (JToC) for the Intervention 1. Job outcomes to be measured 2. Job Channels Framework a. Core job channels b. Equilibrium and multiplier channels 3. Key dimensions for jobs impact measurement a. Sectoral dimensions b. Temporal dimensions c. Spatial dimensions C: Select and implement techniques for Quantifying job impacts C1 Reduced-form estimation approach 1. Selection of estimation technique 2. Extrapolation C2 Parameterized CJC framework approach 1. Direct jobs impact 2. Forward factor usage jobs impact 3. Backward supply chain jobs impact C3 Equilibrium/Multiplier modeling approach 1. Endogenous/exogenous variables 2. Model Selection/development 3. Implementation Page 13 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 2. A Standardized Decision Framework for Evaluating Job Impacts In this section, we outline a standardized decision framework (summarized in Table 1) which emphasizes the job impacts of an intervention. Section 3 provides further supporting information about the various techniques, including associated definitions, underlying assumptions and examples. Throughout Section 2 we provide links to the specific information relevant foreach reference. Section 4 demonstrates the utility of the decision procedure by applying it to several of the pilot studies, as representative examples. Before outlining the decision framework, it is worth noting that the ultimate nature of the evaluation will face several constraints determined by institutional capacity and the availability of data and resources. These include: ● Availability and accessibility of relevant data sources (e.g. Demographic and Health Surveys, business surveys, labor force surveys). A good knowledge of what’s available and how to access it may yield ideas as to how to assess potential impacts. ● The knowledge, skills and previous experience of the evaluation team members or of any consultants used. If team members/consultants have undertaken an evaluation previously using similar methodologies, this will reduce the costs and time involved. ● The budget available. ● The timeline for the evaluation. A: Describe the context 1. Baseline information Summarize the initial relevant information understood and data collected prior to the start of the intervention. This baseline information is a minimal requirement for assessing the effect of the intervention and allowing a comparison of what happens before and after it has been implemented. Importantly, this baseline description should minimally include: ● An assessment of existing labor market conditions. Is unemployment high in the economy or sector, especially for those workers likely to be affected? Is there a significant degree of labor mobility across sectors and geographically? ● An assessment of recent relevant trends affects the region and sector and the factors causing them, including seasonal impacts, global cycles, climate change etc. ● A description of the constraints or market imperfections that the intervention is intended to address. Page 14 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 2. Establish the Treated entity(ies) and sector(s) Where are the funds being spent and/or what are the institutions being modified? In what sector(s) of the economy are these changes taking place? Treated entities are the households, producers or public sector organizations that directly receive the investments and/or experience the institutional changes associated with the intervention. Getting this straight initially will avoid confusion regarding subsequent nomenclature. It is helpful to note that the treated entity is very often not that which experiences the most significant job impacts. That means that their “direct” jobs impacts may not be an important aim of the intervention, but rather the other jobs impacts – which are indirect – are more important to quantify. This is especially true in the case of infrastructure investments or institutional changes. For interventions that involve multiple components and institutional changes, identifying the treated entities can be more complex. 3. Characterize the counterfactual scenario Broadly speaking, what would have been or will be expected to occur in the absence of this intervention? This is a qualitative description of the likely factors that would have affected the treated entity and sector and the outcomes observed in the economy more generally. This descriptive scenario will be used to generate quantitative counterfactual scenarios that reflect the alternative evaluation approaches discussed below. Ultimately, impacts should be measured relative to the implications of the counterfactual scenario. Where estimated impacts are incremental, short term and ignore equilibrium effects, however, the details of this counterfactual scenario may be less important (see below). B: Specify a Jobs-focused Theory of Change for the Intervention Here we outline a systematic and structured approach to developing a JToC that emphasizes job impacts and clarifies the dimensions along which they are to be measured. Specifying a JToC in this way helps to clarify subsequent choices regarding the appropriate techniques and models for quantifying job impacts. Note that the various components distinguished below will likely need to be specified simultaneously, since they interact with each other. Depending on the nature of the intervention, it may often be necessary to specify more than one JToC. For example, this would be the case if the intervention has multiple components with very different impact channels. Even if there is just one component, it may also be useful to separate out JToCs that apply to short and long run effects because the respective channels of impact differ qualitatively. Page 15 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 1. Identify the Job outcomes to be measured On what job outcomes is it desirable and feasible to focus? This will likely depend on what can be measured empirically and existing labor market conditions. Job outcomes can be conceived and measured in a variety of ways. The JToC should specify the outcomes of focus and the measures to be used. Measures of job quality are often more important than the numbers of jobs created but are sometimes ignored. For an ex ante evaluation, this will ultimately be constrained by data availability and influenced by the approach ultimately taken to quantifying impacts. The main potential dimensions of job outcomes are: Employment: This could be measured in terms of the change in the number of people employed, or the number of hours worked, or both. It could also be measured in terms of the shares of overall employment in different sectors. In some cases, it may be possible to decompose these impacts by demographic characteristics (below). Wages/income: This could be measured in various ways, including the change in total annual employment income or hourly/weekly wages. Work conditions: These could include indicators of employment contract status; health insurance; retirement benefits; leave policies; etc. Part-time/Seasonality: An important dimension of job quality, especially in rural areas, is whether the increase in jobs is for part time versus full time and seasonal versus regular. Permanence: Are the jobs being created likely to remain after the intervention is completed? Productivity: How does the intervention affect output per worker? While some of any increase in productivity may be received as a wage, the remainder will generate income for the business operator. It may be possible to estimate productivity gains even in the absence of household surveys. Demographic decomposition: Job impacts generally vary across the affected population in ways that depend on the specific characteristics of individuals. In some cases, it may be possible and desirable to break these impacts down by age and education, by gender, by occupation or some other characteristic. Typically, the feasibility of doing this will depend on the availability of household survey data or census information. Page 16 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 2. Specify the Job Channels Framework (JCF) The core job channels Describe the nature of the job impacts that are expected for each of the core job channels. Are they expected to be substantial? Which of these is expected to be the primary jobs impact channel? As portrayed in Figure 1-1, the core job channels (CJCs) consist of the expected “first-round” job impacts on the treated entity (direct jobs impacts) plus the entities that directly rely on or use the resources, goods and services produced, or enabling conditions provided by the treated entry (factor usage), and the entities that directly supply inputs to the treated entity (supply chain).4 By focusing on the core job channels the evaluator takes all production technologies as given and ignores broader, cross-industry interactions, labor supply reallocations and equilibrium price impacts. The primary jobs impact channel is the one through which labor demand is expected to be increased most significantly by the intervention. The direct jobs channel refers to those jobs created, destroyed, or whose quality or terms of employment change within the entity(ies) or sector(s) receiving the treatment. This includes jobs to operate and maintain any project assets / services over the anticipated life of the investment or new/improved service or function. Note that, depending on the nature of the intervention, the direct jobs channel may not always be the primary channel through which jobs are affected. Forward factor usage channels refer to job outcomes that occur when a change in the available supply, quality, or cost of a factors (which may include an input or productive factor) or condition that causes a change in either (a) the supply of labor by workers who utilize the factor; or (b) demand for labor by producers who utilize the factor. In many cases, especially for infrastructure projects and institutional changes, this is the primary jobs impact channel. Backward supply chain channels refer to impacts that arise due to (a) changes in the demand for locally produced inputs by entities directly impacted by the intervention and (b) those enterprises using the goods or services treated. While such channels almost always exist, it may be challenging to identify the most relevant ones for overall job impacts. 4 These indirect channels can potentially impact an array of sectors. If the treated entity is road transport, for example, the forward factor-usage jobs would include those resulting from the use of improved transport services across all sectors of the economy. Page 17 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Equilibrium and multiplier channels Describe the potential substantial job impacts that might occur beyond those implied by the CJC channels. While consumption spillover impacts are always potentially present, are they expected to be large compared to the other direct or indirect job impacts? How might prices of goods and factors of production (land, labor, capital) be expected to change? How substantial are these effects expected to be? A focus on the CJCs limits the measurement of potential indirect job impacts to “first-round” suppliers and users of the goods/services provided by the treated entity. While this may be appropriate if it is expected to capture most of the job impacts, in many cases other indirect channels will need to be considered to obtain a full assessment. It is useful to delineate these potential “ripple effects” as follows: Narrow consumption spillovers refer to job impacts that are due to changes in the demand for goods and services on the part of the people experiencing a change in income from direct jobs, forward factor usage jobs, and backward supply chain jobs impacts. The term “narrow” reflects the limitation to the employment impacts resulting from increased demand by households affected by the core job channels. As discussed below, there may be additional consumption and other demand spillovers resulting from broader impacts. Broad cross-industry interactions and demand spillovers: While they may capture the impacts on immediate suppliers and users, the CJCs ignore broader upstream and downstream impacts, including some that may “reflect back” to the original treated entities. All these effects combined could, in turn, have substantial additional impacts through additional consumption-spillovers, as well as through other demand spillovers, including greater spending on investment goods arising from increased firm operating profits or increased government spending arising from greater tax revenue. As with the CJCs, modeling these effects while holding prices constant may be reasonable in the short run or in contexts where supply constraints are not binding (e.g. unemployment is high). We discuss these issues in more detail in Section 3. In general it may be difficult to determine ex ante whether these effects are likely to be significant or not. One way to determine this might be to use an indicator of sectoral (and regional) “centrality” based on input-output tables.5 5 The economy is a network of activity, where sectors are “nodes” that produce and trade intermediate output in the process of producing final goods and services. The output of some sectors is used more intensively as intermediate inputs than others. As a result, they are more “central” to the economic network, because other sectors depend on them. For example, suppose there were just four sectors in the economy: A, B, C and D. Sector A provides intermediate output to Sectors B, C and D; Sectors B and C supply to D; C and D supply to consumers (as final demand); and Sector D supplies only to final demand. Here, Sector A has high centrality because it supplies to all Page 18 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Equilibrium price impacts: Neither the CJCs nor the short-run multiplier channels described above account for endogenous equilibrium price responses which, in turn, may affect the resulting job impacts. Such effects could range from those due to increased wages which offset job creation in the directly affected market (partial equilibrium effects), to the job impacts of more broadly experienced price changes in the markets for goods/services and other factors of production (general equilibrium effects). Price adjustments and their consequences for job impacts can arise in different ways over time and space, depending on the supply constraints faced by the economy, trading frictions, price rigidities and factor mobility. These equilibrium effects may often offset the job impacts acting through other channels. Labor supply and production reallocation across sectors and space: If they are able to do so, households may be expected to eventually respond to price changes by reallocating their labor supply to the locations and the sectors of the economy where the return on their efforts is highest. The speed and scale with which such adjustments may occur depend on a multitude of factors and their consequences for both the level of and distribution of job impacts across the economy can be complex. Likewise, the distribution of production, and therefore labor demand, across sectors and locations is likely to adjust in response to price changes, at least in the long run. 3. Identify the key dimensions of the jobs impact measurement The approaches taken to quantify jobs impacts depend on several key dimensions. It is therefore important to clearly specify how impacts are expected to occur and be measured along these dimensions as part of the JToC.6 The specification of these dimensions reflects a tradeoff between comprehensiveness and resource availability (i.e., time, capacity, financing, and data). The most comprehensive JToC possible would involve the maximum breadth, disaggregation and detail along all dimensions. However, such comprehensiveness is not generally feasible, nor is it typically necessary. In practice, the specification of the JToC should proceed by simplifying and focusing on the dimensions that are expected to be most important for the particular intervention and context. It is likely that the specification of some of these may need to be revisited as more is learned. Here we identify the key dimensions and the associated questions that must be addressed. How these dimensions (specifically, the responses to the questions emphasized below) feed into the choice of techniques for quantifying job impacts is then explained in part C. nodes in the supply chain. In contrast, D has low centrality because it is the most downstream (or outermost) node in the production network. See Carvalho and Tahbaz-Salehi (2019). 6 It is these aspects in particular that are often left unclearly specified by typical ToCs. Page 19 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Sectoral dimensions Sectoral Breadth reflects the expected ripple effects of the intervention beyond the treated sector(s). For example, are the job impacts expected to be experienced significantly in a few sectors or in multiple sectors? Sectoral Disaggregation refers to the decomposition of measured job impacts by sector. Will they be measured at the aggregate level, or at the summary sector level or by more detailed sub-sectors? Are the job impacts expected to be very different across sectors? If not, it may be more parsimonious to focus on impacts aggregated across sectors. Cross-sectoral labor mobility captures the degree and speed with which workers can or wish to move from one sector to another in response to labor market changes. How easy is it for workers to acquire the skills and qualifications necessary to obtain employment in the affected sectors? Are there significant barriers to sectoral mobility other than these? For example: policy restrictions, working conditions, geographical immobility, etc. Where the skills involved are not sector-specific it may be that the workforce can adjust fairly quickly in response to changing incentives. However, where retraining or upgrading of skills is required such reallocations may take a long time. Input-output structure: Are the primary job impacts expected to be experienced uniformly across multiple sectors? In some cases, it may be important to explicitly specify the input-output structure of production, whereas in others it is not necessary. If the primary impact of an intervention takes place in a specific sector (e.g., agriculture) but there are expected to be major effects on upstream (fertilizer and other inputs) or downstream sectors (agro-processing), it is likely important to carefully characterize this value chain structure. However, if the primary impacts are expected to be distributed in a roughly uniform way across multiple sectors, the value of specifying and parameterizing the input-output structure is less clear. Informal sector: Is it important to explicitly distinguish between the formal and informal sectors as part of the JToC? In many developing nations, the informal sector constitutes a large share of the economy but can pose problems for measurement of job impacts. Some interventions are expected (and often intended) to have significant impacts in the informal sector. Temporal dimensions The time horizon: Over what period is the intervention expected to have its major impacts? This could be an explicit number of years or a less precise horizon such as “the immediate future” or “the long run”. Page 20 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Temporal disaggregation: To what extent is it feasible and desirable to measure impacts year-by-year over the full time horizon? Is it important to inform M&E targets or the economic analysis, for example? Endogenous dynamics: Are there impacts of an intervention occurring in one period that themselves induce further job impacts in subsequent periods, over and above those directly due to the intervention? Is it feasible and desirable to specify explicitly how these mechanisms work, or can they be ignored without significantly affecting the results? Even if it is agreed that these are important, are they quantifiable? Spatial dimensions Spatial Breadth: What is the geographic area over which significant impacts are expected to take place? For example, are they expected to be largely concentrated in a particular city or region, or are they national? To a large extent the appropriate choice of geographical area over which to measure impacts is likely to depend on the scale and specificity of the intervention. Spatial disaggregation: Is it desirable and feasible to decompose impacts by location, or is it sufficient to measure the impact aggregated across locations? This is likely to depend on the nature of spatial trading frictions that determine the costs of and barriers to transferring labor and goods across locations: ● Labor mobility frictions arise when there are substantial moving or commuting costs, or other barriers to worker mobility to take up jobs. ● Goods trading frictions refer to large transport costs, legal restrictions or other barriers to moving goods between sellers and buyers. In general, the existence of these frictions implies spatial heterogeneity in the job impacts of an intervention that is location-specific. If these costs and barriers are relatively small, it may be reasonable to assume that impacts in different locations are proportional to the aggregate impacts (and effectively ignore them). Otherwise, they will need to be incorporated into the evaluation framework. C: Selecting and implementing the techniques to quantify expected job impacts The following describes a decision making process that will help determine which techniques are feasible and desirable and which it may be reasonable to omit. There are three broad approaches to quantifying the actual or potential impacts of an intervention. They are not mutually exclusive and typically yield estimates reflecting different but Page 21 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 overlapping aspects of the job impacts. Within each approach (labeled C1, C2 and C3 in Figure 2-1), there are multiple techniques to quantify impacts that could potentially be applied. The objective of the decision framework is to help identify which approaches and which techniques are likely to be most appropriate given the JToC specified above. While one technique will typically be most appropriate within each approach, depending on data availability and resources it might be possible and important to implement two approaches, or even all three. A set of decision flow charts visualizing this decision process is included in this section. Figure 2-1. Overall decision flow for selecting approaches to quantify job impacts. Decision flows within each of C1, C2 and C3 are depicted in Figures 3, 4 and 5, respectively. C1. Reduced-form estimation approach This approach encompasses a variety of techniques to obtain empirical estimates of the overall impacts on the job outcomes of households or employers, without imposing an explicit theoretical structure on underlying behavioral, production and equilibrium Page 22 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 relationships. Supporting information regarding the features of specific estimation techniques referenced here is provided in Section 3 and Appendix A. 1. Obtaining Estimates ● Have similar interventions taken place in the past for which ex post empirical evaluations have been undertaken using experimental or quasi-experimental methods? Ideally, these would be in the focus country, but could also be in other countries with similar technologies and institutions. If the answer is “yes”, use these estimates and skip to step 2. Otherwise continue. ● Is there data available that allows for a new ex post empirical evaluation of a past similar intervention using quasi-experimental methods using observations before and after the intervention both on households or employers who were affected and those unaffected by the intervention? If the answer is “yes”, use these data and the appropriate methods to estimate impacts and move to step 2. Otherwise continue. ● Is there data available that would allow the attribution of impacts to observed differential treatments of households or employers that have arisen for other reasons (other than past interventions)? Note that the potential for estimation biases due to non-random assignment will need to be addressed. If the answer is “yes”, use these data and the appropriate methods to estimate impacts and move to step 2. Otherwise continue. ● If the answer to all of these is no, move to approach C2. 2. Using the estimates to quantify job impacts of the current intervention ● Is it possible to use the estimates from step 1, combined with other information, to quantify any kind of job impact due to the intervention? If the answer is “no”, move to approach C2. Otherwise continue. ● Do the estimates from step 1 yield the primary job impacts only? This will likely be the case if employer survey data was used. In this case, developing a parameterized CJC framework may be needed to obtain overall job impacts. If the answer is “yes”, move to approach C2 and use these estimates where appropriate. Otherwise continue. ● Do the estimates from step 1 yield job impacts that combine the various direct and indirect impacts? This may often be the case if household survey data was used. In this case, developing a CJC framework is not necessary to obtain overall CJC job impacts. If the answer is “yes”, move to approach C3. Otherwise continue. Page 23 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Figure 2-2. Selection of techniques to obtain reduced-from empirical estimates C2. Parameterized CJC approach Given estimates of the direct jobs impact, this approach uses estimates of the relationships between labor inputs at different stages along supply chains to quantify the various CJCs identified by the JToC. If the primary job impacts are expected to be fairly uniform across multiple sectors a parameterized CJC framework is not likely to be appropriate. In this case, move on to approach C3. If, instead, the primary job impacts are expected to be Page 24 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 sector-specific, the subsequent decision flow for the CJC approach is depicted in Figure 2-3. Potential sources of estimates for the following impacts are discussed in Section 3: 1. Direct job impacts: The relative importance of direct jobs in the overall jobs impact depends on the nature of the intervention and the adjustment horizon. It may be quite negligible or even negative. For an ex ante evaluation, the direct jobs impact will often be part of the project proposal or monitoring plan in the first place. For an ex post evaluation, it may be possible to directly determine direct job impacts through surveys of employers or households following the intervention. 2. Forward factor usage job impacts: If this is the primary jobs impact channel, estimates may coincide with those obtained via reduced-form estimation (C1). Otherwise, forward factor usage job impacts could be determined by making use of estimates of their production relationship to direct job impacts. For example, how many downstream jobs would be created due to the improved services associated with an additional job in the treated entity/sector? 3. Backward supply chain job impacts: As with forward factor usage jobs, backward supply chain impacts can be determined by making use of estimates of their production relationship to direct job impacts. For example, how many upstream jobs would be created due to the increased demand associated with an additional job in the treated entity/sector? Page 25 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Figure 2-3 Selection of techniques to obtain estimates of job impacts along the core job channels C3. Equilibrium and multiplier modeling approach This approach attempts to capture the equilibrium and/or broader net effects of an intervention on jobs beyond the CJCs, using alternative quantitative model frameworks that incorporate explicit assumptions regarding household behavioral and production relationships. Supporting information regarding the features of specific classes of model frameworks referenced here is provided in Section 3 and Appendix B. Page 26 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 If equilibrium and multiplier effects beyond the CJC are not expected to be significant, then stop here. Otherwise, model selection, quantification and implementation proceeds as follows. 1. Exogenous/endogenous variables Classify whether the following variables are to be treated as exogenous or endogenous: ● Labor supply across sectors and locations ● Prices (including other wages and other factor prices) by sector and location ● Spatial location of production ● Input mix proportions ● Accumulation of assets: physical and human capital ● Labor productivity by sector and location ● Demographic characteristics of the workforce The temporal horizon and spatial breadth described in the JToC will help determine the classification of these and other variables. 2. Model selection/development Figure 2-4 depicts the decision flow for model class selection. Are the primary job impacts expected to be sector specific? If yes, a model with an explicit input-output structure is likely to be appropriate. If, in addition, the evaluation is for a short run period, and/or if there is significant excess capacity, a fixed proportions IO/SAM multiplier framework could be appropriate. Otherwise, for longer run analysis and contexts of labor market tightness, a flexible-price GE model with an embedded input-output structure might be needed. If, instead, the primary job impacts will be experienced uniformly across multiple sectors, models with an explicit I-O structure may be less relevant and a horizontal production structure could be adopted.7 7 A horizontal production structure directly specifies final value added as a function of factors of production, thereby abstracting from the details of the underlying I-O structure. Page 27 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Figure 2-4 Selection of equilibrium/multiplier model features to obtain estimates of job impacts Are the primary job impacts expected to vary substantially by the location of households and employers? If yes, a model that allows for spatial heterogeneity is likely needed. Relatedly, are the associated spatial trading frictions due to costs of Page 28 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 labor mobility or goods transportation costs? If the spatial breadth is a single metropolitan area and the main spatial frictions are commuting costs, a spatial city model would likely be used. For example, this would be the case for an improvement to an urban transport system. If the spatial breadth is regional or national, a spatial GE model with appropriate price determination would likely be more valid. Are endogenous dynamics expected to play a significant role? For example, in addition to greater demand today, a cash transfer may induce greater investment in childrens’ education which affects incomes and employment in the future. A model with recursive dynamics could capture these impacts over time but would require additional assumptions and parameters regarding the accumulation processes. At this point, the type of model developed would also depend on the nature of assumptions regarding the expectations of households and employers. 3. Implementation Parameterization: Model parameter values may be pinned down directly from existing empirical estimates, or calibrated/estimated to match targets from various potential sources including: ● The reduced form estimates from C1 ● The various CJC channel estimates from C2 ● Aggregate economic accounts available from the national statistical bureau ● Demographic and Households Surveys data ● Business Survey data ● Estimates made by other researchers for similar contexts. Quantitative counterfactual: Specify a quantitative “business-as-usual” parameterization representing what is expected to happen in the absence of the intervention. This is required to account for changes relative to the baseline situation which would have happened anyway. All job impacts should then be measured relative to this counterfactual. Proximate impacts of the intervention: Specify how the intervention be directly reflected in terms of the parameters or variables of the model. In the simplest case, this would be the expected direct jobs impact. In the case of infrastructure or institutional changes, however, where the primary job impacts are forward factor usage impacts, it may be reflected in reduced prices, lower costs or higher productivity. For example, improved road infrastructure might be modeled as a reduction in transport costs. Page 29 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Decomposition: Is it possible and desirable to decompose the overall job impacts into direct and the various indirect channels? This is likely only possible if an explicit input-output structure has been specified as part of the model. Sensitivity: Identify the parameters whose values are most uncertain and assess how the quantified job impacts vary when they take on alternative values within an empirically-relevant reasonable range. To assess robustness, one could also consider alternative counterfactual scenarios that might arise due to previously unanticipated economic changes. Page 30 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 3. Approaches, Techniques and Timing Considerations This section provides key information and discussion needed to support the decision framework described above. Specifically, it provides a detailed description and discussion of the various approaches and techniques that are commonly used to estimate job impacts. Moreover, it highlights the embedded assumptions and data requirements of each and provides examples drawn from the pilot studies and other related studies. Subsequently, the differences and similarities in the techniques that are appropriate and feasible for ex ante and ex post evaluations are discussed. Techniques for Quantifying job impacts C1: Techniques for reduced-form estimation In this section we discuss the techniques that are most commonly implemented under this approach. In fact, as will become apparent, the same techniques can also be used to estimate the values of parameters that are then used to quantify key relationships within more structured frameworks, including the relationships between direct and indirect job impacts implicit in the CJC approach and key parameters of equilibrium models. Social experiments To measure the impact of an intervention on impacted households or employers, a social experiment would directly construct a control (or comparison) group of households who are not impacted by the intervention, which consists of a randomized subset of the eligible population.8 However, it can be difficult to ensure that the experimental conditions have been met. Moreover, for many interventions it is unlikely that such experiments would be feasible (either for cost or ethical reasons). In any case, none of the pilot studies considered here involved social experiments. Well-executed randomized control trials (RCTs) do, however, provide an “optimal benchmark” against which alternative methods can be compared. 8 In principle, social experiments could be designed to provide estimates of all the various job channels and not just reduced-form estimates of the overall impact. In this sense, they could provide more detail on the underlying structural relationships amongst the CJCs. Page 31 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Example: Experimental evidence on the impact of public employment programs in India RS1: In the study by Muralidharan et al. (2020), the authors worked with government of the Indian state of Andhra Pradesh to randomize the order in which 157 sub-districts introduced a new system (biometric “Smartcards”) for making payments in the National Rural Employment Guarantee Scheme. Using this exogenous source of randomization, they are able to identify causal effects of the program on various outcomes. Quasi-experimental designs9 If households or firms have been treated differently with respect to an intervention or some other change, it may be possible to obtain valid estimates of related job impacts, even in the absence of a social experiment. Commonly used econometric frameworks designed to provide estimates of the impact of policy interventions or other sources of differential treatment are summarized in the following table with further details provided Appendix A. The appropriateness of each technique depends on the nature of the intervention or other sources of differential treatment, the context and, most critically, the extent and structure of the data collected. These, in turn, inevitably depend on the resources and time available. Quasi-experimental designs to measure job impacts require microeconomic survey data on households, employers or both. Moreover, the data need to be structured in such a way that it is possible to clearly distinguish between households or employers that have been impacted by an intervention from those that have not (or between those who are impacted more or less). The appropriate framework applied depends critically on the “assignment rule”: what determines whether or not, or how much, a household or employer is impacted by a given intervention? Is it random? Is it the result of choices made by the individual household or employer before or after the intervention? Does it just depend mechanically on observable characteristics of households/employers? When baseline data is unavailable, as is the case with many of the pilot studies reviewed here, it can be very challenging to provide compelling evidence of causal relationships using these methods. 9 In a social experiment the researcher randomly assigns subjects to control and treatment groups and designs the treatment. In a quasi-experiment, some other non-random method is used to assign subjects to groups. The researcher does not have control over the treatment, but instead studies pre-existing groups that received different treatments after the fact. A natural experiment is a particular type of quasi-experiment in which an external event or situation (“nature”) results in the random or similar assignment of subjects to the treatment group in a manner that avoids selection bias. Page 32 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Quasi-experimental designs requiring baseline and endline observations Difference-in-difference: Applied in situations when some households/ employers are affected by an intervention (or some other related, exogenous change) and others are not – at least not at the same time – and we have data on them before and after an intervention (or other related change) takes place. The impact is then measured by how much job outcomes change for the impacted (“treatment”) group relative to the non-impacted (“control”) group. A key assumption is that in the absence of the intervention (or other related change), the outcomes for the impacted group (the counterfactual) would have had the same trend as for the non-impacted group. This assumption is typically considered more valid when random or random-like assignment occurs. Triple difference: It may not be valid to assume that workers/employers with different characteristics would have experienced similar counterfactual employment trends. In this case, if one can argue that the employment ratio of different household types would have followed similar trends across regions in the absence of the intervention, one could estimate the impact of an intervention by how much this ratio would have changed in the region where the intervention took place relative to the other region. As an example, this approach is taken as part of one of the pilot studies evaluating the potential impacts of an improved bus rapid transit system in Dar es Salaam, Tanzania (PS9). Quasi-experimental designs based on differential treatment Matching methods: This method aims to construct the counterfactual outcomes that would have been experienced by the impacted households/employers in the absence of the intervention by pairing each impacted household/employer with ex ante observationally equivalent non-impacted households. The only remaining difference between the two groups is assumed to be the impact of the intervention. With a finite sample, it may be impossible to find a non-impacted household with exactly the same characteristics as each impacted household. Propensity score matching substitutes these individual characteristics with the likelihood that a household is impacted by the intervention, conditional on their characteristics. As an example, this approach is taken as part of one of the pilot studies evaluating the potential job impacts of improved access to credit by employers in Mali (PS19). Instrumental variables (or other similar approaches): These methods rely on finding variables which do not directly impact the outcome (i.e., which are not correlated with the second stage equation’s error term when omitted from that stage), but which determine whether or not households/employers are impacted by the intervention. If the potential impact of the intervention can be assumed to be the same across households/employers, the IV estimator identifies the impact removed of all the biases that emanate from a Page 33 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 nonrandomized control. Otherwise, the IV estimator will only identify the average impact under strong assumptions. As an example, this approach is taken as part of one of the pilot studies evaluating the potential job impacts of improved transport infrastructure in the Lake Chad region (PS1). Discontinuity design: In some cases, the probability of being impacted by an intervention changes discontinuously with some continuous variable. One can use this discontinuous dependence to identify a local average treatment effect even when the variable does not satisfy the assumptions needed for IV estimation. Any discontinuity in the relationship between the outcome and the continuous variable across households is attributed to a discontinuous change in impact due to the intervention. None of the pilot studies adopted this technique. For a related example, see Zimmert and Zorn (2021) ,which studies the impact of “direct payments” on on-farm employment in the Swiss agricultural sector (RS7). Estimation of direct/indirect job impacts using quasi experiments Given sufficiently well-structured data it is, in principle, possible to obtain valid empirical estimates of the impact of an intervention on employment outcomes in various sectors and locations using a quasi-experimental approach. In practice, however, the data requirements for obtaining estimates of all the separate direct and indirect employment impacts are demanding. For example, we might need employment outcomes for households and/or employers in every relevant sector/location, some of which were impacted (directly or indirectly) by the intervention and some which were not. One also needs to be able to credibly assume that households/employers who were impacted would have experienced similar employment outcomes to their “observationally-equivalent” counterparts who were not. To avoid “spillover” impacts on the employment outcomes of “untreated” households, one would likely require geographical separation. In most cases, therefore, it is challenging to obtain separate empirical estimates of the impacts of each of the various indirect channels on job outcomes. Micro-empirical studies of the employment outcomes of households will typically yield estimates of the overall employment impact (combining all direct and indirect impacts) in the given region. Empirical studies of employers are likely to yield estimates of direct or forward factor usage job impacts (perhaps by sector) but not other indirect ones. In practice, estimating or isolating all direct and indirect employment impacts is typically going to require some kind of model framework with imported parameter values where necessary. On the other hand, with household or employer survey data it is often possible to estimate overall impacts for several different job outcomes (e.g. wages, type of employment, etc.) and to decompose these impacts spatially and along several demographic dimensions. Page 34 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Pilot study examples: Household vs. employer surveys PS5: Data from household surveys in Lesotho and information on existing bridges from government reports are combined to estimate the effect of footbridge proximity (defined as being within 5 kms) on household outcomes, controlling for demographic and geographic variables. This estimate would be expected to include the aggregate of all job impacts, direct and indirect, on households. PS19: Employer data from the World Bank Enterprise Survey (WBES) for Mali is used to estimate the causal impact of credit access on employment growth, controlling for multiple factors. Propensity score matching is used to address the potential bias in estimates due to the non-random allocation of credit. This estimate would be expected to reflect job impacts due to the forward-factor usage channel (i.e., the improved credit access of the employers) only. Non-experimental ex post designs In many cases, it is not possible to cleanly separate out impacted households or employers from non-impacted ones and thereby apply quasi-experimental techniques. In these cases, one is forced to adopt alternative approaches. In many situations, in fact, one may be more interested in the overall impacts on employment in a given market and region, rather than the distribution across households or employers. If survey data were available to undertake the types of analyses described above, a preferred approach would be to aggregate up from micro-estimates. But, if data is limited, and time and resources are scarce, it may be necessary to take a less detailed, market-level approach. Assuming employment numbers in different sectors can be observed (e.g. via a labor force survey), to determine the impact of an intervention ex post, an estimate of the counterfactual time path of employment in each sector (which may differ significantly from the baseline level) is required. Counterfactual paths could be constructed in various ways. The simplest is to forecast outcomes based on past trends and any additional relevant information (e.g. seasonal factors). Although none of the pilot studies quantified counter factual paths in such a manner, it is a common approach in the economics literature. Such an approach might become less valid, however, if the economy faces multiple shocks whose impacts are difficult to control for. A better approach, if it were possible, might be to compare outcomes to those of a “similar” region facing similar “macroeconomic shocks”, but which was not impacted by the intervention. This requires that other initial differences across regions could be adequately controlled for and that there are no significant spillovers. Page 35 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Related Study Example: Estimating counterfactual employment outcomes RS4: In estimating the employment impacts of lock-down policies intended to limit the spread of COVID-19 in Canada, the authors quantify counterfactual paths for each industry and province based on existing forecasts and historical season patterns. Finally, if multiple “similar” interventions or other changes have taken place and if one can argue that the (unobserved) characteristics of households or employers in the regions impacted are uncorrelated with these treatments, then one could try to base impact estimates on cross-sectional empirical regressions. If the allocation of interventions or other differences is considered likely to have been endogenous, then some kind of cross-region “macro” instrumental variables approach might be possible. Using past empirical estimates to quantify impacts of interventions Using reduced-form estimates based on past data to quantify the implications of on-going or future interventions inevitably requires strong assumptions regarding external validity. In particular, such reduced-form estimates likely reflect an underlying structural equilibrium relationship that could potentially change in response to future interventions. The context may be different; and the intervention itself may differ from those for which existing estimates exist, even in subtle but potentially important ways. In certain cases, therefore, it may be important to more fully assess and quantify the possible impacts of shifts in factor or output markets by constructing and parameterizing an equilibrium model (see below). Moreover, given that reduced-form empirical estimates are also often for one time period, additional assumptions must be made in order to quantify impacts over time. In practice, the exact value of a reduced-form estimate need not be interpreted as a precise quantification of potential impacts. Rather they provide credible, empirical evidence that the impacts are likely to go in the expected direction and are economically and statistically significant. C2: Techniques for a parameterized CJC approach Although restricting attention to the CJC channels may potentially miss some of the broader job impacts of the intervention or overstate effects by ignoring supply responses, there are some benefits compared to using multiplier and equilibrium models: ● They may be relatively less resource-intensive to obtain. This will be the case especially if high quality M&E data have already been collected as part of previous, related projects, or if detailed household or employer surveys exist that facilitate micro-estimation. Page 36 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 ● The estimates do not require investigators to lay out in detail the behavior of households and firms and the interactions between sectors of the economy.10 Nor does it require the construction and solution of potentially complex models. Moreover, if broader impacts are expected to be relatively small, ignoring them may not matter much. Even if they are expected to be large, obtaining CJC estimates could be a useful step towards parameterizing equilibrium and multiplier models (see below). This approach often implicitly assumes “fixed-proportions” technology and no supply constraints, so that input-output relationships are fixed as production expands. This can be valid if the intervention is relatively small-scale and value chain-specific. A more general approach is to incorporate estimated supply elasticities that reflect more realistic, flexible relationships between output and employment in a given sector.11 These estimates could, in principle, be computed and disaggregated along various spatial and temporal dimensions. For example, one could focus on a very localized impact for one year, or a broader geographical impact over several years, a decision which may depend on the intervention.12 Direct job impacts For an ex ante evaluation, the direct jobs impact should be the most straightforward to quantify and will often be part of the project proposal or monitoring plan in the first place. For an ex post evaluation, it may be possible to directly determine direct job impacts through surveys of employers or households following the intervention. In this case, it must be clear which jobs were created as a result of the intervention and which would have been created even in its absence (i.e. the counterfactual). This can be challenging. For example, even if it is known how many jobs were paid for by an employer subsidy, it is possible that the employers would have hired some of these workers anyway. Thus, to assess the true jobs impact of the intervention requires a quantified counterfactual that specifies the expected path of outcomes with no intervention. Forward factor usage job impacts Possible sources of estimates for these impacts include: 10 To make inferences made based on these estimates, however. still requires strong implicit assumptions, as noted above. 11 For example, an innovative feature of the jobs estimation module within IFC’s AIMM framework is that it uses such estimated supply elasticities from previous studies in order to quantify the relationships between output and jobs. 12 However, the focus on core job channels may become less credible as we consider broader geographic and longer run impacts. Page 37 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 ● Empirical estimates that yield the impact of past similar changes or interventions on the production and job outcomes of the input/service users in the focus region/country. Ideally, impacts should be attributed using quasi-experimental methods and controlling for other relevant factors. ● Empirical estimates drawn from other studies of similar interventions, in other regions or countries, ideally with similar production technologies and institutions. ● Estimates using Supply-Use relationships provided by the focus country’s statistical bureau or that of other countries with similar production technologies and institutions; firm surveys, or combined data sources. Such estimates are likely to be relatively crude as they only allow for the output relationships between aggregated sectors.13 These could then be translated into employment effects using sector-level labor productivity or output elasticity estimates. ● Estimates of the responsiveness of production and employment to changes in factor usage according to industry experts based on anecdotal evidence. Backward supply chain jobs impact Possible sources of backward supply chain estimates will typically come from similar types of sources: ● Empirical estimates of the increase in inputs resulting from the increased production by the same or similarly treated entities in the focus country. Ideally, attribution should be achieved by using quasi-experimental methods. ● Empirical estimates drawn from other studies of similar interventions, either in the focus country or in other countries with similar production technologies and institutions. ● If the sectors of the suppliers are known, crude estimates could be made using Supply-Use tables, translated to employment effects using sector-level labor productivity or output elasticities estimates. ● Estimates of the responsiveness of the production of inputs and the associated employment to changes in production by the treated entities according to industry experts. 13 The technologies, sales and input requirements of treated entities may not be representative of the overall sector of which they are part. Page 38 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Pilot Study Example: Parameterizing a CJC framework PS10: One aspect of a sustainable fisheries intervention in Tonga involves institutional investments to counteract illegal overfishing and declining fish stocks. The main CJC impacts were estimated as follows: a. Forward factor usage jobs impact: Increased hours of employment due to an improvement in fishing stocks, estimated based on an input-output job multiplier estimate from fisheries sectors in Latin America. This is the primary impact. b. Backward supply chain jobs impact: Also based on input-output job multiplier estimates from Latin America. In a follow-up report, economists from the IFC, using the AIMM framework, provide alternative estimates using job multipliers derived from a SAM from Fiji. They also estimate narrow consumption-spillover impacts using the same source. C3: Equilibrium and Multiplier modeling techniques Possible economic models range from relatively simple partial equilibrium models focusing on a single sector or region to complex general equilibrium models incorporating demographic, geographic and sectoral heterogeneity.14 Specifying or adopting such models can be costly, time-consuming and requires specific technical skills, but has several potential benefits: ● They provide controlled environments within which the potential impacts of interventions can be demonstrated, while literally holding other factors constant. In particular, the counterfactual path, specifying outcomes in the absence of the intervention, can be clearly laid out according to some “business as usual” scenario. ● They allow for a decomposition of potential impacts (e.g. by sector and region) and allow one to isolate the key linkages between interventions and household well-being. 14 It may be possible to estimate impacts arising from narrow consumption spillovers without developing a structural model. In the face of limited resources, data and time, it may be more reasonable to import “consumption multipliers” estimated by researchers who have already developed and quantified such models for the focus economy or for those of other countries with similar technologies and institutions. See, for example, PS10 (IFC follow-up) and PS15. Page 39 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 ● They can be constructed to take into account market interactions and one or more frictions, distortions, or policy changes at once, to capture dynamics with adjustment costs, and/or different assumptions regarding expectations. ● Although the results and implications ultimately depend on the underlying modeling choices, assumptions and parameter estimates, the sensitivity of these conclusions can, in principle, be assessed through robustness checks. Here we outline and discuss the issues and decisions involved in this approach to quantifying job impacts. 1. Exogenous vs. endogenous variables An important determinant of the appropriate model framework is the classification of those variables which are treated as endogenous (determined within the model framework and, hence, impacted by the intervention) and those which are treated as exogenous (determined outside the model framework). In reality, all variables could be thought of as endogenous to some extent. But for a particular impact evaluation, with a specific adjustment horizon, sectoral breadth and geographic area, it is reasonable to treat marginally impacted variables as exogenous.15 The classification of variables as endogenous or exogenous depends, in part, on the temporal dimensions specified by the JToC. Specifically, as the adjustment horizon increases following an intervention, we might expect the qualitative nature of the impact on certain variables to change: ● Short run: When the demands for labor inputs first increase, it may be reasonable to assume that most workers are not immediately able to change location, sector or occupation.16 Thus, in the short run, worker location and sector might reasonably be treated as exogenous. Similarly, the relative proportions in which inputs to production are used cannot initially be changed significantly immediately following 15 For example, local temperatures are determined by regional weather systems that reflect global climate. Large scale, “integrated assessment models” have been developed which incorporate the global impact of economic activity on climate and the consequent feedback effects on regional productivity via temperature change. However, when evaluating the impacts of subsidizing a producer in an individual sector and region, say, it makes sense to adopt a much narrower model framework that treats regional temperatures and their consequences for productivity as exogenous. This does not imply that environmental consequences of the intervention are being ignored. The impacts on greenhouse gas emissions and estimates of their marginal cost can still be an endogenous outcome. 16 Of course, if the intervention directly aims to create mobility between specific regions or sectors, this short-run restriction would apply only to other regions and sectors of the economy. Page 40 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 an intervention.17 In assessing initial impacts, it is also common to treat prices and wages as being determined exogenously. In some respects, this reflects the Keynesian perspective that price responses take time to unfold and may reflect uncertainty and incomplete information on the part of employers.18 ● Medium run: Subsequently, in response to rising demands, we might expect wages and the prices of some goods and services to adjust endogenously.19 In the presence of frictions and informational limitations such adjustments may still be incomplete. At the same time, it may start to become possible for producers to change the mix of inputs they require and, in response to price and wage changes, they may try to do so. Moreover, while there may exist costs and barriers to factor mobility, we might expect workers to start to move endogenously across sectors and regions in response to these market signals. Similarly, over this horizon, capital investments across sectors and regions may start to respond endogenously to changes in profit opportunities and expectations of demand shifts. ● Long run: In the absence of other exogenously-imposed constraints (e.g., price controls) or persistent institutional failures (e.g., credit market problems), prices and wages should eventually be expected to endogenously adjust towards their market clearing levels. Such a situation would be associated with full input and factor adjustment and endogenous regional labor reallocation. Over this horizon, the interactions of the intervention with physical and human capital accumulation, technological and demographic change may start to play a more important role. In particular, the impacts of the intervention on these factors may need to be endogenized. A common hybrid approach allows for longer time horizons by modeling a series of short-run scenarios and then allow key variables such as prices, supply constraints and technological parameters to evolve exogenously.20 While this is somewhat ad hoc, it allows for the impact of factors that are deemed important drivers of these variables but that are not explicitly treated endogenously in the model. No tractable model is comprehensive enough to capture everything and such assumptions cede greater control to modelers over the nature of alternative dynamic scenarios being considered. Another hybrid approach combines short and medium run impacts in a single model framework by holding 17 This assumption of “fixed proportions technology” implies that little substitution between inputs is initially possible in response to changing conditions. The higher the level of sectoral aggregation, the more valid this assumption will tend to be. 18 This assumption is reasonable when there is excess capacity and fixed proportions technology. 19 The size of such adjustments will depend on the size of the intervention and the linkages assumed. 20 For example, an input-output framework may have fixed prices in a given year, but these could be allowed to change exogenously over time according to observed data. Page 41 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 prices/wages constant for a period but allowing them to adjust over time according to a “frictional” market clearing process. Other approaches can be viewed as combining medium and long run impacts by explicitly accounting for factor accumulation and frictional reallocation of labor. Similarly, the spatial dimensions specified by the JToC also determines the classification of certain variables as exogenous or endogenous which, in turn, should be reflected in the model selection/development: ● Regional: Suppose the major employment impacts are expected to be confined to a particular sub-national region. For example, suppose an investment in a particular agricultural crop may be expected to have few effects outside a specific rural area. In this case, an appropriate model would be an agricultural household production model which treats prices outside the area as exogenous but those within it as endogenous. Alternatively, to evaluate the main employment impacts of public investments in transit infrastructure in a given city, an urban spatial model, which treats variables beyond the city boundaries as exogenous would likely capture the main long-run effects. ● Multi-regional: Where the direct impacts of an intervention are sufficiently widespread or where the scale is sufficiently large, it is necessary to account for the endogenous impacts on employment across regions. If labor were highly mobile, this could be accomplished by broadening the definition of the “region” being studied. However, where significant costs and other barriers to mobility are expected, a model framework which explicitly endogenizes cross-regional flows would be more appropriate. ● National: In cases where the intervention is on a national scale, directly affecting multiple sectors simultaneously (e.g., fiscal policy changes), it may be appropriate to specify a macroeconomic framework that abstracts from specific regional impacts and inter-regional flows. Such a framework would emphasize total direct and indirect employment impacts in each sector aggregated across regions. While this implies a loss of regional resolution, such simplified macroeconomic models can be relatively straightforward to construct and implement with existing data and standard assumptions. 2. Selecting and/or developing an appropriate model Model selection and development proceeds by focusing on the features that are expected to be most important for the particular intervention and context, as described by the JToC. The most comprehensive model framework would feature a detailed input-output Page 42 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 structure, formal and informal sectors, high geographical resolution with a realistic characterization of the costs of moving labor and goods across heterogenous locations, as well as endogenous dynamics characterizing the evolution of key assets and rigidities over time. In practice, however, developing or modifying such a comprehensive framework for each evaluation is infeasible, impractical, and unnecessary. The features of the appropriate quantitative model framework will also inevitably depend on what data is available to parameterize it. The following is a brief summary of the broad classes of model frameworks commonly used in economic evaluations. Within each of these classes, there is a wide variety of model frameworks with their own particular focus, features and assumptions.21 Appendix B provides more detailed descriptions of models within each class. Classes of Equilibrium and Multiplier Model Frameworks Static Models The defining feature of this sub-group is that they do not incorporate an endogenous mechanism through which decisions and outcomes in one period impact, or are impacted by, those in subsequent periods. They can still, however, be used to study impacts over time by specifying relevant changes to key variables in an exogenous fashion. Partial equilibrium models: These frameworks incorporate endogenous output and/or input price adjustment within the sector and/or region directly affected by the intervention. This requires a quantified specification of supply and demand schedules in directly impacted output and/or input markets, holding other prices and quantities constant. This class includes agricultural household production models (e.g. RS2), local multiplier models (e.g. RS3) and urban spatial models (e.g. PS9). Fixed-price multiplier models: A common approach to assessing broader supply-chain impacts and household expenditure responses is to make use of the input-output tables and other national accounts data that is supplied by national statistical agencies. Models based on these accounting frameworks alone allow for multi-sectoral and macroeconomic production impacts but are not “equilibrium models” because they treat output and input prices as exogenous. Models in this class include traditional fixed-proportions (Leontief) input-output models and Social Accounting Matrix (SAM) models (e.g. PS4, PS6, PS13, and PS20). 21 There are multiple ways in which one could group economic models. The particular delineation adopted here reflects the key impact dimensions in the JToC discussed above and how dynamics are treated. Page 43 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Static General Equilibrium (GE) models: The defining feature of GE model frameworks is the endogenous determination of prices in multiple output and input markets. GE models typically also allow for endogenous substitution amongst final goods and services and amongst factor inputs. Some GE models include a specification of the input-output structure of the economy, while others abstract from these details. There is a wide-variety of GE models, some assuming well-functioning, flexible markets and others incorporating various kinds of imperfections and frictions. This class encompasses regional GE models, national GE models and multi-regional GE models (e.g. PS1), with and without spatial frictions. Models with Endogenous Dynamics This sub-group incorporates endogenous mechanisms through which decisions and outcomes today impact, or are impacted by, those in the future. For example, decisions made today may affect the future via various types of asset accumulation (e.g. physical capital or education), technological changes, regional or sectoral shifts in labor supply, or government budgets. One challenging issue here is the extent to which these dynamics are assumed to be accounted for by decision-makers when making current decisions: that is, how should peoples’ expectations be modeled? Models with non-rational expectations: This class of model frameworks either treats choices that might reflect expectations about the future as exogenous, or models expectations by extrapolating from past empirical relationships in the data. Again, there is a wide variety of frameworks that take this approach including those with no asset accumulation but other kinds of intertemporal linkages and those with physical or human capital accumulation but where the rate of savings or investment is exogenously determined. As an example, a recursive GE model is used to inform one of the pilot studies evaluating the impacts of agricultural sector reforms in Ethiopia (PS17). The IFC have recently developed a hybrid dynamic GE framework, referred to as the Economy-wide Private Impact Quantification (EPIQ) model (RS5). Rational expectations dynamic GE models: The assumption of rational expectations implies that economic actors form expectations about the future that are consistent with assumptions of the model framework. That is, expectations are assumed to be endogenously determined. Developing dynamic GE models with this feature can be challenging and implementing them may require sophisticated computational techniques and coarse approximations. Although such frameworks are used for macroeconomic research by many central banks, they are not commonly used for project evaluations. Page 44 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 3. Implementation Parameterizing the model: The more comprehensive is a model framework, the more challenging it is to pin down all the relevant parameter values. This is another reason for focusing on more simplified model frameworks. An advantage of fixed-proportions, IO/SAM multiplier models, for example, is that they can often be easily parameterized using available government statistics. Other structural equilibrium models (such as GE and urban spatial models) typically require multiple additional parameter values which may be harder to obtain. Some of these may be found in the existing literature or estimated using available data. The process of obtaining reduced-form or CJC estimates may also yield some useful parameter values or calibration targets that can be used to quantify the model. In the academic literature, new models are proposed to understand a variety of phenomena, and parameters are often estimated using simulated methods and selection criteria (such as by minimizing squared deviations from key outcomes). Specifying a quantitative counterfactual: The descriptive counterfactual discussed above can be used to generate a more quantitative comparison by running a “business as usual” scenario through the model. This is an important step in determining impact and tries to account for the likely trends and changes that would have affected outcomes in the absence of the intervention. Inputting the proximate effects of the intervention: This involves translating the change created by the intervention into exogenous “shocks” to the model. This is not generally straightforward, especially if the intervention consists of institutional changes or infrastructure investments, and/or the model is not explicit with respect to the actual intervention and there may be multiple ways in which this could be done. Decomposing the job outcome impacts: Equilibrium and multiplier models typically deliver aggregate impacts on job outcomes. However, it should be possible to decompose these into those arising from direct and the various indirect channels. This could be done by setting parameter values that effectively “shut down” some channels and using the restricted, counterfactual parameterization to generate the impacts without them. Assessing the sensitivity of the results: Most model frameworks impose strong assumptions and incorporate parameter values about which there is often significant uncertainty. Moreover, it is not possible to control, in advance, for all potential contingencies and confounding factors that might impact the outcomes. Considering the implications of alternative assumptions, parameter values and scenarios provides a sense of how important these are to the overall impacts. It also highlights the limitations of the analysis and areas where greater realism and precision may be beneficial. Page 45 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 Ex ante vs. ex post evaluation An important distinction between economic evaluations is whether they are being conducted prior to the intervention (ex ante) or after the intervention has taken place (ex post). The primary objective of an ex ante evaluation is to assess the likely or possible impacts of a future intervention on various outcomes. This may be used to justify funding for the intervention, to compare alternative interventions and/or provide guidance to stakeholders on what they might expect as a result of the intervention. In contrast, an ex post evaluation attempts to assess what the actual impacts of the intervention were and the extent to which they conform to prior expectations. Obviously, the key difference is the existence of new data and information arising from the impacts of the intervention itself. However, there are several reasons why this difference may not always be as stark as one might expect: ● Not all outcomes that are deemed important can easily be directly measured. ● Even those that are measured may not be measured very precisely. ● Due to significant time lags, measurement may often take place before all (future) impacts could have occurred. ● To correctly measure the impacts of an intervention it is necessary to have some estimate of what would have happened to those affected in its absence. The issue of counterfactuals is never straightforward. A consequence of these issues is that, in practice, all three of the approaches to quantifying job impacts discussed above remain relevant for both ex ante and ex post evaluations. Ex post evaluations should benefit from more relevant and credible empirical estimates based on data from the actual intervention. However, to estimate the full job impacts they will still need to rely to some extent on additional assumptions, model frameworks and imported parameter estimates. The balance between the application of direct empirical estimation and the use of models for ex post evaluation ultimately depends on the quality and availability of relevant data gathered before, during and after the intervention. In practice, an aspect of ex post estimation that typically always relies on assumptions, even with well-structured data, is the specification of the counterfactual. Moreover, it is rare that enough variation in the data can be acquired to go much beyond estimating average marginal impacts of interventions (average treatment effects) on households, workers (or other economic agents) relative to the counterfactual. If significant heterogeneity is expected across households, for example, it may not be possible to estimate a distribution of marginal impacts. This may limit the extent to which Page 46 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 investigators can assess the general validity of any model assumptions made ex ante and use them to generate estimates of additional unmeasured outcomes.22 Data Collection The more that the collection of data – before, during and after the intervention – has been structured to enable the application of empirical estimation methods, the more reliable the estimates will be. First, this enables the application of methods with weaker identifying assumptions and estimates of a greater range of outcomes. Moreover, with better and more extensive data, estimators can rely less on quantitative model frameworks using parameters that cannot be estimated using data associated with the intervention. Monitoring and evaluation (M&E) data is commonly collected as part of the overall project intervention. Ideally, the M&E data should have baseline values and will enable before and after comparisons for treated entities and beneficiaries. The World Bank Group’s “Jobs M&E Toolkit”, for example, provides resources to be used throughout the project cycle and recommends they are best applied ex ante in the design of projects and their M&E systems (see WBG, 2017). The WBG’s Jobs M&E toolkit provides standardized definitions of job indicators and guidance on how to measure them, as well as templates for jobs data collection forms. Jobs indicators include measures of job creation, job quality and job access, and the beneficiaries may be workers and/or firms. In addition, it is also common to collect information on intermediate outcomes that facilitate improved job indicators (e.g. human capital accumulation, increased firm performance, etc.), which can be important to understand whether the intervention’s theory of change is borne out and/or whether changes in outcomes can be attributed to the intervention. Even with detailed, precise data and information, a key challenge is to identify the job impacts that result from the intervention and that would not have occurred in its absence. For this, a quantified counterfactual path of job outcomes is needed. Even if it is clear, for example, which jobs were funded by the intervention, some of these may have been created in its absence. Connecting ex ante and ex post evaluations Ex ante and ex post evaluations should not be thought of as separate undertakings. It seems clear that the results of high quality ex post evaluations of previous interventions 22 Economic models often make assumptions that imply marginal impacts which vary with household characteristics. But empirically we may only be able to estimate the average impact. As a result, although one can compare the mean impacts generated by the model with that estimated, one cannot fully validate all underlying assumptions of the economic model. Page 47 of 92 Measuring Jobs Impacts Final Report - 2023-04-05 can and should be an important input into ex ante evaluations of subsequent similar interventions. In their absence, evaluators must identify alternative, and often less compelling, sources of causal evidence and rely on strong assumptions to bridge data gaps. This is the case for many of the pilot studies reviewed here: the authors had to develop various creative ways to obtain valid empirical estimates of impact. What may be less clear is how an ex ante evaluation can help improve a subsequent ex post evaluation. The process of developing and implementing an ex ante evaluation can yield a more precise (and formal) understanding of the way in which the intervention is likely to lead to impacts on various outcomes and the relative size of these impacts. It can help identify data gaps and key assumptions to inform subsequent M&E design. In undertaking an ex ante evaluation, key questions inevitably arise: are the assumptions reasonable for this particular context/intervention? What information/data, were it available, would help to make the ex ante evaluation more credible? How could an ex post evaluation be structured to provide a valid measure of the impact of the intervention? How can the project’s M&E framework be adjusted in order to increase the validity of ex post evaluation? If these questions are addressed and their responses recorded as part of the ex ante evaluation process, they can be used to inform data collection during implementation and thereby improve ex post evaluation. Page 48 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 4. Application to Pilot Studies It is instructive to illustrate how the standardized procedure described above could have been implemented for several of the pilot studies. In most cases, the studies already follow some parts of this procedure (the standardized procedure is, after all, based on a review of the pilot studies) and it is useful to consider the reasoning behind the choices that were made by the evaluations teams. In each case, we describe the project estimations as they were done, rather than how they would have been done following the procedure, but the procedure is also used to highlight additional elements which might have been implemented but were not.23 In some cases, this may have been due to a lack of data/resources, in others because they were judged to be of less importance. The pilot studies that are mapped into the generalized procedure below were selected to be representative of the themes and approaches characterizing the twenty studies. PS1: Transport infrastructure and structural change in the Lake Chad region Summary This intervention consists of investments to improve the rail/road Intervention transport corridor in the Lake Chad region. Theme Transport infrastructure Makes use of household survey data that has been georeferenced, new spatial infrastructure data, and district characteristics. New information was collected on road network expansions, access to the electricity Institutional capacity network, and access to Internet fiber backbone. The team includes members that have capabilities and experience in both empirical methods and equilibrium modeling. A: Context 1. Baseline: (i) Trends in share of employment in agriculture in the region. (ii) Details of existing and planned rehabilitation and extension of the transport (road and rail) corridor. (iii) Summary of available data on employment outcomes and infrastructure indicators. The justification for the intervention is presumably that road/rail infrastructure is a public good. 23 Comments regarding these elements are in italics. Page 49 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 2. Treated entities and sectors: The treated entities are private contractors in the road/rail construction sector and also some public sector activities. 3. Counterfactual scenario(s): Based on estimates of existing (baseline) travel speeds (reflecting road conditions and border delays). B: Jobs-focused Theory of Change 1. Job Outcomes: Share of employment by occupation and industrial aggregate measured at the district and industry level; Wages and land prices by sector; Welfare. No demographic breakdown is reported, but estimated impacts control for education. 2. The Job Channels Framework a. Expected CJCs Direct: Increased labor demands in transport infrastructure construction in the short term plus maintenance in the longer term - these are not discussed or estimated here. Forward factor usage: Reduction in transport costs leads to reduced overall production costs, increased entry and greater labor demand across multiple sectors. This translates into proportionally higher employment in non-agricultural sectors. This is expected to be the primary jobs impact channel, at least in the long run. Backward supply chain: Possible supply chain impacts are not discussed and there is no explicit I-O structure. This ignores short run impacts via suppliers of construction firms. Long run impacts via suppliers of primary beneficiaries of the road network are implicitly included in the overall impact. b. Expected equilibrium/multiplier channels: Firms and workers are expected to respond endogenously to changing transport costs by potentially switching their choice of location and sector of employment. As a result, locations that are not directly affected by such infrastructure investments can indirectly benefit or lose through increases or decreases in economic activity, employment and wages in nearby connected locations. Equilibrium adjustments are expected to occur through utility equalization across locations in response to wages and land prices. 3. Key dimensions for jobs impact measurement a. Sectoral dimensions Breadth: All industries Disaggregation: By 3 aggregate industries - agriculture, manufacturing, services. Consumption spillovers: Yes - increases in income result in greater demand for all goods and services. Page 50 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Input-output structure: No - the focus is on value added which reflects employment in each sector. This reflects the expectation that the primary job impacts will be experienced broadly across multiple sectors. Explicit informal sector: No. b. Temporal dimensions Adjustment horizon: The long run Disaggregation: Single period - the focus is on the long run and not the short run Endogenous dynamics: No – this is a static perspective representing the long run c. Spatial dimensions Breadth: National/multi-national, reflecting the affected transport corridor. Disaggregation: Multiple locations - census districts in multiple countries. Labor mobility frictions: Perfect labor mobility within countries. Zero mobility across them. Goods/service transports frictions: Yes – “iceberg costs” across locations with location-specific primary impacts resulting from infrastructure changes. C: Selecting and implementing techniques to quantify job impacts C1 Reduced-form empirical estimates 1. Obtaining estimates: From Cameroon DHS and satellite data. Two sets of estimates: a. Individual logit regression: impact of various measures of the availability of good quality roads and overall “market access” due to reduced transport times on the likelihood of survey respondents working in various occupations. b. District level regressions: impact on employment shares in different occupations. An instrumental variables approach is taken to address causation/attribution issues. 2. Extrapolation: Combines estimates of the predicted impact of improved road networks on transport times between locations with the share of the population in each location to obtain a measure of market access. This is combined with the parameter estimates on market access from the individual logit regressions to yield estimates of the impacts of road improvements on employment shares by sector. Both aggregate and district impacts are estimated. C2 Parameterized CJC framework: Not implemented in this study. Short run direct impacts are not considered, and all indirect impacts are implicitly included in the overall Page 51 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 impacts. This reflects the fact that the primary job impacts are experienced broadly across multiple sectors. C3 Equilibrium/Multiplier model framework 1. Endogenous/exogenous variables: The focus is on the long-run, so prices of goods, labor and land are treated endogenously and are determined by market clearing. The allocation of labor across locations and sectors within countries is treated endogenously assuming indifference of households across location. However, that across countries is exogenous. Technologies and national populations are exogenously fixed. 2. Model Selection: A static, three-sector spatial general equilibrium model (Lebrand, 2022) with multiple locations and transport costs between them, but no input-output structure. This choice of model reflects the following aspects: a. benefits of the transport network infrastructure depend on its proximity to production, b. the focus is on the long-run with endogenous prices and market clearing, c. the primary job impacts are experienced broadly across multiple sectors, and d. the impacts are expected to differ across these three aggregated sectors. 3. Implementation: a. Parameterization: Parameters that are held constant across regions are drawn from other studies, mostly for African countries. Productivities for each sector in each location are set so that the observed distribution of population, employment, and land is an equilibrium given a parameterization of trade costs. b. Counterfactual: Baseline transport costs calibrated to reflect estimated travel speeds. c. Inputting proximate impacts: Effects of road improvements are modeled via changes in transport costs which are estimated to reflect both travel time and border delays between specific locations. d. Decomposition: Employment effects in the model implicitly reflect all combined indirect channels. There is no decomposition provided. e. Sensitivity: Multiple scenarios are considered. Comments: Extrapolation based on reduced-form estimates predict a shift of employment out of agriculture in Cameroon and into agriculture in Chad. In contrast, the equilibrium model suggests a net shift of employment into agriculture in both countries (at least when border delays are included). In general, it is hard to compare these two sets of estimates: 1. A potential problem with the reduced form estimates (well-understood by the report’s authors) is that it seems likely that districts that are growing economically for many Page 52 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 reasons (e.g. improvements in agricultural productivity, greater commercialization, improved credit markets, etc.) would experience declining shares of the workforce in agriculture and improved transport infrastructure. A correlation driven by such confounding factors is not causal. While some efforts have been made to address this issue, it is hard to overcome in a fully compelling way. Time and district fixed effects included in both the OLS and multinomial logit regressions may control for some of these confounding effects, but not if they impact districts differently over time. The instrumental variables approach is reasonably compelling for electricity infrastructure but seems rather opaque and hard to evaluate for roads. Another complementary approach, if feasible, may be to include more district and time-varying drivers of economic growth as additional controls in the regressions. 2. While the impacts implied by the GE model are clearly causal and are measured holding other potential confounding factors constant, they rest on numerous maintained explicit assumptions and parameter values. In such a complex model, it is not easy to assess how robust the estimated impacts are to reasonable variations in these assumptions/values. This is why a full sensitivity analysis is warranted, especially with regard to those assumptions/values over which there is substantial uncertainty. Such a “what if” analysis can also be illuminating with regard to the workings of the model, and the key mechanisms and parameter values driving the quantitative results. Note that there are also short-run job impacts, due to the construction activity, which are being ignored here, primarily because these are not a primary objective of the project (and would not be lasting). PS4: IFC Investment in Mozambique Agribusiness Summary This intervention consists of an IFC Investment in Westfalia Fruto Intervention Mozambique (WFM), an avocado producer. Theme Private Sector Development IFC assessment consists of both an ex ante evaluation based on IO/SAM modeling and a midline evaluation based on direct surveys of WFM and Institutional capacity employers. This allows for a possible validation of ex ante assumptions. Relies on access to the IFC’s AIMM assessment framework, including economic and financial modeling and data. Page 53 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 A: Context 1. Baseline: Key issues include disincentives to invest due to risks associated with conflict and state-fragility. Although unemployment is low, the predominance of low-productivity subsistence farming suggests an important scope for improving agricultural labor incomes. Mozambique ranks 61st out of 101 countries in the world in the World Bank’s “Enabling the Business of Agriculture” index. 2. Treated entities and sectors: The treated entity is the avocado producer. 3. Counterfactual scenario: Baseline. No counterfactual scenario is specified. B: Jobs-focused Theory of Change 1. Job Outcomes measured: Total employment by sector and job type (management vs. non-management; permanent vs. seasonal); wages; working conditions. All outcomes are decomposed by gender. 2. The Job Channels Framework a. Expected CJCs Direct: Increased labor demand in the WFM orchard, including that associated with on-site production and picking and bulking. This is expected to be the primary jobs impact channel. Forward factor usage: Increased labor demand in packaging, quality control, cold storage, processing, marketing and distribution, wholesale and retail trade. Backward supply chain: Increased demand for services of seed traders and distributors, and producers of fertilizer and farming equipment. A significant fraction of chemical inputs are imported (esp. From South Africa). b. Expected equilibrium/multiplier channels: Broad impacts are expected through cross-sectoral and consumption-spillover multiplier channels. No major equilibrium price effects due to the relatively small scale of investment. 3. Key dimensions for jobs impact measurement a. Sectoral dimensions Breadth: All industries, especially backward supply chain. Disaggregation: Multiple summary sectors Consumption spillovers: Yes Input-output structure: Yes. This is a sector-specific intervention with significant expected upstream and downstream impacts. Explicit informal sector: No b. Temporal dimensions Adjustment horizon: Six years Disaggregation: Year-by-year Page 54 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Endogenous dynamics: No c. Spatial dimensions Breadth: National Disaggregation: None Labor mobility costs: None Goods/service transports costs: None C: Selecting and implementing techniques to quantify job impacts C1 Reduced-form empirical estimates: Not implemented in this study. C2 Parameterized CJC framework: a. Direct jobs impact: Ex ante estimates based on project proposal. Ex post estimates obtained via a detailed midline survey. No counterfactual appears to be specified, so it is unclear which of these jobs would have been created in the absence of intervention. b. Forward factor usage jobs impact: Ex ante estimates based on relationships implied by supply-use tables. Ex post estimates are not available because avocado trees have not fully matured and are expected to start producing in 2023. c. Backward supply chain jobs impact: Ex ante estimates based on relationships implied by supply-use tables. Ex post estimates obtained via a detailed midline survey of WFMs suppliers. No counterfactual appears to be specified, so it is unclear which of these jobs would have been created in the absence of intervention. C3 Equilibrium/Multiplier model framework 1. Endogenous/exogenous variables: This is a very short-run evaluation of a sector- and location-specific intervention. Consequently, the following are treated as exogenous: labor supply across sectors and locations; prices (including other wages and other factor prices) by sector and location; spatial location of production; input mix proportions; accumulation of assets; labor productivity by sector and location; demographic characteristics of the workforce. 2. Model Selection: A fixed-price multiplier SAM/IO model is adopted. The investee is thought to be a small actor, so that the impact on prices is negligible. 3. Implementation a. Parameterization: SAM data is available and already in use as part of AIMM. b. Counterfactual: No counterfactual path is specified. Estimates are relative to baseline. Page 55 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 c. Inputting proximate impacts: Direct employment and input expenditures are provided by the Client. The latter are assumed to result in an equal increase in the intermediate inputs by sector demanded in the model. d. Decomposition: Appendix provides detailed decomposition of job impacts implied by ex ante modeling. e. Sensitivity: Three alternative scenarios are considered, reflecting different assumptions about the share of chemical inputs imported. Comments: The midline survey faced multiple challenges as described in the appendices. PS5: Transport infrastructure and connectivity project in Lesotho Summary This intervention consists of building 41 footbridges in order to improve Intervention rural access to social services and markets. Theme Rural Infrastructure Makes use of fairly detailed existing household surveys and data on Institutional capacity bridge construction. A: Context 1. Baseline: The main problem is the uneven distribution of the existing road network and the lack of bridges in highland areas. The footbridges constructed are expected to have a significant impact by enhancing connectivity to essential infrastructure services and agricultural and labor markets. 2. Treated entities and sectors: Bridge builders or authorities building bridges. 3. Counterfactual scenario: The current baseline situation. (No specification of underlying growth trends in Lesotho.) The estimates below are incremental. B: Jobs-focused Theory of Change 1. Job Outcomes measured: Travel time; overall employment rate; paid employment rate; agricultural employment rate; job quality characteristics; poverty, consumption and wages; occupational skill levels; agricultural production for sale. Although demographic decomposition is feasible, the focus is on aggregate impacts and impacts amongst sub samples with differential requirements for health and educational services. Page 56 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 2. Job Channels Framework a. Expected CJCs Direct: Footbridge construction and maintenance - not discussed presumably as this is expected to be relatively minor. Forward factor usage: New footbridges expected to reduce transportation constraints, especially in highlands during the rainy seasons when those populations are isolated. This is expected to increase trade in agricultural markets and stimulate labor demand and supply. This is expected to be the primary jobs impact channel. Backward supply chain: Increased trade of agricultural production inputs will generate increased demand for labor in those input sectors. b. Expected equilibrium/multiplier channels: The rise in the income per capita in the benefited villages will increase demand for different goods and services and, consequently, create new jobs (mostly serving this local demand). It seems plausible that households may relocate in response to changes in travel costs and changing economic activity. This could enhance the overall benefits of the bridges. If there is little evidence of this relocation in the past, perhaps it is a relatively unimportant channel. 3. Key dimensions for jobs impact measurement a. Sectoral dimensions Breadth: All industries in region Disaggregation: Agricultural and non-agricultural Consumption spillovers: Yes. Input-output structure: No. This reflects the expectation that the primary job impacts will be experienced broadly across multiple sectors. Explicit informal sector: No explicit distinction. b. Temporal dimensions Adjustment horizon: Long run - this reflects the focus on indirect job impacts Disaggregation: Single period. Endogenous dynamics: No c. Spatial dimensions Breadth: National Disaggregation: 360 Census enumeration areas Labor mobility costs: Yes - implicit but not measured Goods/service transports costs: Yes - implicit but not measured Page 57 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 C: Selecting and implementing techniques to quantify job impacts C1 Reduced-form empirical estimates 1. Obtaining estimates: Household data from Lesotho Continuous Multipurpose Household Survey/ Household Budget Survey (CMS/HBS) 2017-2018 and bridge data from Environmental and Social Management Plan (ESMP) Report. Estimates based on the effect of footbridge proximity (defined as being within 5 kms) on household outcomes, controlling for 17 demographic and geographic variables. A decomposition is provided based on requirements for health/educational services. 2. Extrapolation: Statistically significant estimates from reduced-form estimates are used to directly compute ex ante expected long-run incremental impacts of the bridges to be built on the specified outcomes. C2 Parameterized CJC framework: Not implemented. Short run direct impacts are assumed to be negligible and all indirect impacts are implicitly included in the overall impacts estimated above. C3 Equilibrium/Multiplier model framework: Not implemented. Comments: 1. In the reduced-form estimation, there are other potential attribution issues due to the non-random distribution of households. For example, having a higher income may make living closer to a bridge more affordable. It might be possible to address these using instrumental variables estimation. 2. The reduced-form estimates take household locations as given. It is possible that these might change in response to new bridges, new trade flows and new opportunities. One possibility might be to build and quantify a spatial equilibrium model with labor mobility and costly goods trade (as in PS1). If there is little evidence of households re-locating in response to bridges built in the past, however, then such a (potentially costly) modeling strategy would be unwarranted. PS8: Energy Sector Reforms in Rwanda Summary This intervention consists of investments in electricity services to Intervention reduce its fiscal cost and increase efficiency, affordability and accountability. Page 58 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Theme Power infrastructure Institutional capacity - A: Context 1. Baseline: - 2. Treated entities and sectors: The Rwanda Energy Group (REG). 3. Counterfactual scenario(s): None specified (see comments below) B: Jobs-focused Theory of Change 1. Job Outcomes: Aggregate national employment by occupation, sector and gender. Distribution of establishments by sector. 2. Job Channels Framework a. Expected CJCs Direct: Potential minor changes in employment within the energy sector. Forward factor usage: Indirect employment resulting from new connections and new economic activity by employers in all sectors of the economy. This is expected to be the primary jobs impact channel, in the long run. Backward supply chain: Possible supply chain impacts are not discussed and there is no explicit input-output structure. Long run impacts via suppliers of primary beneficiaries of the road network are implicitly included in the overall impact. b. Expected equilibrium/multiplier channels: 3. Key dimensions for jobs impact measurement a. Sectoral dimensions Breadth: All industries Disaggregation: None Consumption spillovers: Yes - increases in income result in greater demand for all goods and services. Input-output structure: No - This reflects the expectation that the primary job impacts will be experienced broadly across multiple sectors. Explicit informal sector: No. b. Temporal dimensions Adjustment horizon: The focus is on the period from 2017-2020. Disaggregation: Annual Endogenous dynamics: Not specified c. Spatial dimensions Breadth: National. Page 59 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Disaggregation: Rural versus urban (in appendix). Labor mobility frictions: Not specified Goods/service transports frictions: Not specified C: Selecting and implementing techniques to quantify job impacts C1 Reduced-form empirical estimates 1. Obtaining estimates: Data sources: Rwandan labor force survey; Rwandan establishment census; World Bank Enterprise Survey; National Accounts of Rwanda; administrative statistics on electricity; program documents; and stakeholder interviews. No formal attempt made to attribute impacts of employment. Instead, conclusions are based on an informal discussion of relevant evidence. 2. Extrapolation: This is an ex post evaluation. C2 Parameterized CJC framework: Not implemented in this study. This reflects the fact that the primary job impacts are expected across multiple sectors. C3 Equilibrium/Multiplier model framework: Not implemented in this study. Comments: According to the report, sectors that use electricity more intensively experienced a decline in employment between 2017 and 2020, while those that use it less intensively experienced an increase. One possible explanation for this is that lower tariffs and improved services were explicitly targeted disproportionately to low income, low-usage customers that tend to be in sectors using electricity less intensively. The authors also suggest that, based on surveys, most employers were previously relatively unconstrained by access to electricity. Note that no counterfactual is specified so it is unclear what would have happened in different sectors in the absence of these reforms to the electrical power grid. Moreover, it is possible that the time horizon is too short to observe the eventual impacts. These limitations are noted in the conclusion. PS9: Dar es Salaam bus rapid transit system in Tanzania Summary This intervention consists of government investment in a bus rapid Intervention transit system in Dar Es Salaam. Theme Transport infrastructure Makes use of several household surveys conducted over multiple years, Institutional capacity combined with existing data to provide a detailed panel data set. The team includes members that have capabilities and experience in both Page 60 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 empirical methods and equilibrium spatial urban modeling. A: Context 1. Baseline: With a rapidly growing population, commuters in Dar es Salaam were experiencing severe challenges due to congestions by 2016. A primary justification for the intervention is that improved BRT infrastructure will reduce travel times thereby resulting in greater labor market efficiency and access to services. The BRT infrastructure was to be developed in six phases. At the time the baseline survey was undertaken, Phase 1 had been completed. 2. Treated entities and sectors: The treated entity is the Dar es Salaam Transit Agency (DART) and the construction/maintenance activities it funds. 3. Counterfactual scenario(s): Based on estimates of existing (baseline) travel speeds between locations within the region prior to Phase 2. B: Jobs-focused Theory of Change 1. Job Outcomes: Transport modes; travel times and costs by gender; satisfaction; rents; incomes; consumption; employment and wages by gender. 2. Job Channels Framework a. Expected CJCs Direct: Increased labor demands in transport infrastructure construction in the short term plus maintenance in the longer term - these are not the main focus here. Forward factor usage: Job seekers are expected to find jobs that better match their skill set, as reductions in commute time increase the set of employers they can access. Ultimately, this should induce a more efficient allocation of labor and an increase in overall productivity. This is expected to be the primary jobs impact channel, in the long run. Backward supply chain: Possible supply chain impacts are not discussed and there is no explicit input-output structure. Long run impacts via suppliers of primary beneficiaries of the road network are implicitly included in the overall impact. b. Expected equilibrium/multiplier channels: Households are expected to respond endogenously to changing commuting costs by potentially switching their choice of location within the region. As a result, locations that are not directly affected by such infrastructure investments can indirectly benefit or lose through increases or decreases in economic activity in nearby connected locations. Equilibrium adjustments are expected to occur through Page 61 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 utility equalization for each household type across locations in response to wages, amenities and rents. 3. Key dimensions for jobs impact measurement a. Sectoral dimensions Breadth: All industries Disaggregation: None Consumption spillovers: Yes - increases in income result in greater demand for all goods and services. Input-output structure: No - the focus is on value added which reflects overall employment. This reflects the expectation that the primary job impacts will be experienced broadly across multiple sectors. Explicit informal sector: No. b. Temporal dimensions Adjustment horizon: The long run Disaggregation: Single period - the focus is on the long run and not the short run Endogenous dynamics: No – this is a static perspective representing the long run c. Spatial dimensions Breadth: Single city/region, reflecting the affected transit corridor. Disaggregation: Multiple locations - neighborhoods. Labor mobility frictions: Commuting costs between residential and work locations. Goods/service transports frictions: Yes – services provided by “amenities” implicitly must be consumed locally. C: Selecting and implementing techniques to quantify job impacts C1 Reduced-form empirical estimates 1. Obtaining estimates: Data sources: three household surveys at baseline, midline and endline; a baseline travel time survey and endline estimates using GoogleMaps and data from DART; population data from most recent Census (2012). Estimates are based on a “triple-difference’’ regression design estimating how changes in the outcomes experienced by households differed depending on their proximity to a previous phase (Phase 1) of the BRT. 2. Extrapolation: This was not possible because no statistically significant differences in the observed outcomes of households at different distances from Phase 1 of the BRT were identified. Page 62 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 C2 Parameterized CJC framework: Not implemented in this study. Short run direct impacts are not considered, and all indirect impacts are implicitly included in the overall impacts. This reflects the fact that the primary job impacts are experienced broadly across multiple sectors. C3 Equilibrium/Multiplier model framework 1. Endogenous/exogenous variables: The focus is on the long-run, so prices of goods, labor and land are treated endogenously and are determined by market clearing. The residential locations of heterogeneous households within the city are treated endogenously, assuming indifference of households of each type across locations. 2. Model Selection: A static, urban spatial commuting model with heterogeneous home and work locations (Balboni et al., 2020). This choice of model reflects the following aspects: a. benefits of the transit network infrastructure depend on its proximity to residential and work locations, b. the focus is on the long-run with endogenous prices and market clearing, c. the primary job impacts are experienced broadly across multiple sectors, and d. the impacts are expected to be largely experienced within the city region. 3. Implementation: a. Parameterization: Many of the structural parameters can be estimated based on regression specifications that are implied by the model for households of a given type across neighborhoods (this is a desirable feature of the model that makes it very tractable). Others are drawn from aggregate statistics and estimates in the literature. b. Counterfactual: Baseline commuting costs calibrated to reflect estimated travel times. c. Inputting proximate impacts: Effects of BRT improvements are modeled via changes in the commuting costs between specific locations. d. Decomposition: Employment effects in the model implicitly reflect all combined indirect channels. There is no decomposition provided. e. Sensitivity: Multiple scenarios are considered. Comments: The quantified model ultimately predicts little impact on productivity due to a more efficient labor market allocation. Most of the welfare impact is directly due to lower commuting costs. The full structural analysis has not yet been completed. Page 63 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 PS10: Pathways to Sustainable Oceans in Tonga Summary This intervention mainly consists of two components: Intervention ● Various investments to increase production of Mabe Pearls; and ● Institutional investments in Onshore and Offshore fisheries. Theme Industry-support Team members have acquired detailed knowledge of the production Institutional capacity process. The original evaluation was complemented by a follow-up evaluation from the IFC using their AIMM framework. A: Context 1. Baseline: Fisheries sector is 3% of GDP. Commercial fisheries employ 2% of the workforce but 83% of the rural population are involved in inshore fishing. Relatively tight labor market. ○ Constraint on pearl production is the capacity of the hatchery ○ Current fish stocks are precarious due to illegal overfishing. 2. Treated entities and sectors: (1) Government hatchery and pearl producing firms, (2) Sustainable Management Areas (SMAs) and Ministry of Fisheries (MoF) 3. Counterfactual scenario: (1) No change in pearl production; (2) Decline in fish stock due to continued illegal overfishing and a decline in production. B: Jobs-focused Theory of Change 1. Job Outcomes: Numbers employed. No demographic decomposition. 2. Job Channels Framework a. Expected CJCs Direct: i. Increased labor demand in pearl production. This is expected to be the primary jobs impact for component (1). ii. Increased labor demand at the MoF. Forward factor usage: iii. Increased labor demand for jewelry producers; iv. Increased production and hence greater labor demand in the domestic fishing industry due to reduction in illegal overfishing. This is expected to be the primary jobs impact for component (2). Backward supply chain: v. Increased demand for services and inputs by fishers and jewelry producers; Page 64 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 vi. Increased demand for services and inputs by fishers. b. Equilibrium/multiplier channels: Not specified in the original evaluation. However, the IFC follow-up specifies a narrow consumption-spillover impact in addition to the CJCs. 3. Key dimensions for jobs impact measurement a. Sectoral dimensions Breadth: core job channels only Disaggregation: Several sub-sectors within CJC Consumption spillovers: No Input-output structure: Yes. The primary job impact is sector specific Explicit informal sector: No b. Temporal dimensions Adjustment horizon: 10 years Disaggregation: Year-by-year Endogenous dynamics: None. c. Spatial dimensions Breadth: National Disaggregation: None Labor mobility costs: Not specified Goods/service transports costs: Not specified C: Selecting and implementing techniques to quantify job impacts C1 Reduced-form empirical estimates: Not implemented by this study. Comment: It is likely that the nature of the interventions preclude differential treatment of households and producers. Therefore it is hard to learn from past cross-sectional estimates based on surveys. C2 Parameterized CJC framework: Well-understood CJC input-output production structure based on published estimates from industry experts. 1. Direct jobs impact: a. Increase in labor demand due to increased hatchery capacity. No anticipated impact on wages due to the small scale of intervention relative to the industry. b. Small impact on employment 2. Forward factor usage jobs impact: a. Increased employment in jewelry production. No anticipated impact on wages. b. Increased hours of employment due to offset in decline of fishing stocks, estimated based on an input-output job multiplier estimate from fisheries Page 65 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 sectors in Latin America (Simas and Wiebe, 2021). The IFC follow-up report proposes using job multipliers based on the SAM from Fiji. 3. Backward supply chain jobs impact: a. Increased employment in input production. No anticipated impact on wages. b. Based on input-output estimates from Latin America, The IFC follow-up report proposes using job multipliers based on the SAM from Fiji. C3 Equilibrium/Multiplier model: Not implemented in the original study. However, the IFC follow-up report estimates consumption-spillovers based on the SAM from Fiji. Comments: Labor demand is generated in other industries (e.g. tourism), so there could be some multiplier impacts that could be assessed using an IO/SAM model. However, given the relatively small size of the industry in overall GDP and the small scale of the intervention, multiplier effects are expected to be small. PS15: Strengthening the National Social Protection System in Angola Summary This intervention is aimed at strengthening the national protection system by ● Expanding a cash transfer system - this is the main focus of the Intervention evaluation ● Creating an institutional structure that provides a permanent safety net Theme Human development Institutional capacity - A: Context 1. Baseline: Low development indicators for Angola. Large informal labor market and low productivity jobs. Existing constraints: inefficient credit market and no social insurance. 2. Treated entities and sectors: Household-producers that receive the transfers. 3. Counterfactual scenario: Estimates are incremental. B: Jobs-focused Theory of Change 1. Job Outcomes measured: Total employment. No demographic decomposition. 2. Job Channels Framework Page 66 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 a. Expected CJCs Direct: By relieving credit constraints, transfers will enable greater investment in productive assets and an increase in demand for labor, mostly supplied via self-employment. This is considered to be the primary jobs impact channel. Forward factor usage: Not specified - output is mainly for subsistence Backward supply chain: Not specified - inputs are self-produced(?) b. Expected equilibrium/multiplier channels: Increased self-employment income will result in increased demand for goods/services and further increases in labor demand (narrow consumption spillover impacts). Other broader impacts are not specified as they are expected to be small 3. Key dimensions for jobs impact measurement a. Sectoral dimensions Breadth: All industries (mostly informal work) Disaggregation: None Consumption spillovers: Yes Input-output structure: No Explicit informal sector: This is all part of the informal sector b. Temporal dimensions Adjustment horizon: Unclear Disaggregation: Potentially Endogenous dynamics: Potentially. c. Spatial dimensions Breadth: National Disaggregation: None Labor mobility costs: None Goods/service transports costs: None C: Selecting and implementing techniques to quantify job impacts C1 Reduced-form empirical estimates: Not implemented in this study. C2 Parameterized CJC framework: Not explicitly specified but the following information can be determined: 1. Direct jobs impacts: Increase in labor demand is proportional to cash transfers. Alternative sources are used to estimate this proportion. 2. Forward factor usage impacts: Not estimated - subsistence 3. Backward supply chain impacts: Not estimated Page 67 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 C3 Equilibrium/Multiplier model framework: Uses estimates of the propensity to consume food out of income and labor demand elasticities in food production from Ghana to estimate narrow consumption spillovers. No model implemented. Comments: Although a dynamic household production model with borrowing constraints is specified, it is not really being used in the simulation. In the end, the expected increase in labor demand is assumed to be proportional to the cash transfer. PS17: Second Agricultural Growth Project (AGPII) in Ethiopia Summary Interventions consist of financial and institutional support in 5 main areas: (1) Agricultural Public Support Services; (2) Agricultural Research; Intervention (3) Small Scale Irrigation; (4) Agricultural Marketing and Value Chain, and (5) Program Management, Capacity building and Monitoring and Evaluation. Theme Rural infrastructure Institutional capacity - A: Context 1. Baseline: AGPII is ongoing since 2017. 2. Treated entities and sectors: Primary goal is to increase productivity and commercialization of the agricultural sector. 3. Counterfactual scenario(s): “Business-as-usual” with zero productivity growth. B: Jobs-focused Theory of Change 1. Job Outcomes: Job creation broken down by types: temporary vs. permanent; low-skilled vs. high skilled; part-time vs. full-time; women and youth. 2. Job Channels Framework a. Expected CJCs Direct: There may be jobs created as part of the provision of increased services to farmers and other agricultural producers. Forward factor usage: Increased agricultural productivity, due to the greater services provided and institutional improvements, is expected to increase labor demand. This is expected to drive up wages and improve working Page 68 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 conditions and create new jobs for some unskilled workers. This is expected to be the primary jobs impact. Backward supply chain: Increased demand for upstream inputs into agricultural production is expected to increase labor demand further. b. Expected equilibrium/multiplier channels: Increased incomes in agriculture is expected to increase demand for agricultural final products and those of other sectors. The agricultural sector is a large part of the Ethiopian economy and the scale of the interventions are significant and regionally widespread. As a result, impacts are expected beyond the agricultural sector. 3. Key dimensions for jobs impact measurement a. Sectoral dimensions Breadth: All industries Disaggregation: Multiple sub-sectors within production supply chain and multiple summary sectors Consumption spillovers: Yes - increases in income result in greater demand for all goods and services. Input-output structure: Yes Explicit informal sector: No. b. Temporal dimensions Adjustment horizon: 10 years Disaggregation: Year-by-year Endogenous dynamics: Yes - asset accumulation, but exogenous expectations. c. Spatial dimensions Breadth: National Disaggregation: None - policy interventions are not treated as region specific Labor mobility frictions: No Goods/service transports frictions: No C: Selecting and implementing techniques to quantify job impacts C1 Reduced-form empirical estimates: Not implemented yet, but framework discussed in principle. C2 Parameterized CJC framework: a. Direct jobs impact: Although the survey report describes the primary jobs impact as direct jobs, they appear to be forward factor usage jobs resulting from improved services and productivity associated with AGPII. Page 69 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 b. Forward factor usage jobs impact: Ex post survey directly asks sampled employers about workers employed under AGPII. Estimates broken down by multiple categories. No counterfactual appears to be specified, so it is unclear which of these jobs would have been created in the absence of AGPII. c. Backward supply chain jobs impact: Not determined directly, but implicitly part of equilibrium/multiplier model framework. C3 Equilibrium/Multiplier model framework 1. Endogenous/exogenous variables: The following are to be treated as endogenous: ● Labor supply across sectors and locations is perfectly mobile ● Prices (including wages and other factor prices) by sector, except the unskilled wage ● Input mix proportions ● Accumulation of physical capital 2. Model Selection: Recursive computable general equilibrium model (IFPRI, Diao and Thurlow) with an input-output structure, three types of workers (high skilled, medium skilled, unskilled), land and capital. All factor markets clear, except for unskilled workers. 3. Implementation a. Parameterization: Makes use of detailed SAM data for Ethiopia constructed by the Policy Studies Institute. b. Counterfactual: A business as usual (BAU) scenario is used for the counterfactual, assuming zero agricultural productivity growth. Is this reasonable? c. Inputting proximate impacts: AGPII is assumed to increase productivity growth in multiple agricultural sectors covered by the policy changes. Multiple scenarios are considered with different rates of growth. d. Decomposition: Employment impacts by sector; income impacts by household type. Various other decompositions are possible. e. Sensitivity: Based on the impacts of various productivity growth scenarios. No other parameter variations are considered. Comments: There was a lot of discussion of possible alternative empirical estimation methods (baseline report) and local multiplier models, but so far the evaluations appear to consist of estimates based on a direct survey (with no counterfactual), and a dynamic quantitative GE model of the entire economy. Page 70 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 PS19: Promote access to finance, entrepreneurship and employment project in Mali Summary This intervention consists of two broad components: ● Microfinance Institution (MFI) enhancement, and improving access by Micro, Small and Medium Enterprises (MSMEs) to Intervention finance, and ● direct financing for income-generating activities (IGAs) and short-term employment opportunities through labor intensive public works (LIPWs) for selected beneficiaries. Theme Finance Institutional capacity - A: Context 1. Baseline: (i) Fragility of economy; (ii) poverty; (iii) high youth unemployment (17%); (iv) dominance of agricultural employment (62%). No clear description of pre-existing underlying credit constraints. 2. Treated entities and sectors: (1) MFIs plus public sector training programs for MSMEs; (2) directly financed household/workers, plus training 3. Counterfactual scenario(s): Baseline B: Jobs-focused Theory of Change 1. Job Outcomes: Change in numbers employed. 2. Job Channels Framework a. Expected CJCs Direct: (1) Increased hiring by MFIs and public sector (2) Increased demand for workers from IGAs and LIPWs. This is expected to be the primary jobs impact channel for component (2). Forward factor usage: (1) The credit channel - Increased credit use will allow greater production and therefore greater demand for labor. This is expected to be the primary jobs impact channel for component (1). (2) None specified – could come from inputs supplied to other producers Backward supply chain: Page 71 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 (1) None specified - could come from inputs purchased from others but sectors of beneficiaries us not known precisely (2) None specified - could come from inputs purchased from others b. Expected equilibrium/multiplier channels: (1) None specified - could come from the same estimate as for component (2) (2) Higher incomes of the poor are spent on goods and services that generate further increases in labor demand (narrow consumption spillover). 3. Key dimensions for jobs impact measurement a. Sectoral dimensions Breadth: All industries – no restrictions Disaggregation: No Consumption spillovers: Yes Input-output structure: No - primary job impacts experienced broadly across multiple sectors. Explicit informal sector: No. b. Temporal dimensions Adjustment horizon: The short run Disaggregation: Single period Endogenous dynamics: No. c. Spatial dimensions Breadth: National. Disaggregation: None. Labor mobility frictions: No Goods/service transports frictions: No C: Selecting and implementing techniques to quantify job impacts C1 Reduced-form empirical estimates 1. Obtaining estimates: (1) Data from the World Bank Enterprise Survey (WBES) for Mali is used to estimate the causal impact of credit access on employment growth, controlling for multiple factors. Propensity score matching is used to address the potential bias in OLS estimates due to the non-random allocation of credit. (2) Given estimates of the total “transfer” of income to program recipients, estimates of the indirect consumption-spillover impact are obtained using time-series analysis of aggregate (national) data. This involves estimating the marginal propensity to consume out of income and the impact of an increase in household consumption on productive activities. 2. Extrapolation: Page 72 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 (1) Multiplying the estimated impact of credit access on employment by the expected increase in the number of loans (under a range of alternative assumptions) yields an estimate of the total impact of the intervention on employment. (2) Not attempted – time-series estimation yielded statistically insignificant results C2 Parameterized CJC framework: Not explicitly specified but the following information can be determined: 1. Direct jobs impacts: a. negligible; b. directly measured 2. Forward factor usage impacts: a. can be derived from reduced-form estimates above; b. not estimated 3. Backward supply chain impacts: a. Not estimated - sectors of recipients is not precisely known b. Not estimated 4. Consumption spillover: - C3 Equilibrium/Multiplier model framework: 1. Not implemented 2. Narrow consumption spillover impact derived using an average “consumption multiplier” from other studies (Beegle et al. 2018). A $1 transfer to beneficiaries is estimated to yield a $.30 increase in income of non-beneficiaries. This is converted to a short-term employment impact by dividing by aggregate labor productivity. Page 73 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 5. Concluding Remarks Based on a review of up to 20 pilot studies currently being undertaken and financed by the World Bank, this report has identified some key lessons for determining the feasibility and appropriateness of alternative methodologies for quantifying the jobs impact of interventions in varying contexts. These insights have been used to develop a standardized and transparent decision-making framework for choosing approaches and quantitative methods for evaluating job impacts, either ex ante or ex post. There are myriad interacting decisions involved which are, in large part, constrained by the institutional capacity of specific countries and the financial and human resources of the evaluation teams. Nevertheless, a common, core decision-making procedure for choosing the most appropriate quantification methods can be constructed. We have demonstrated the utility of this standardized procedure by implementing it for several representative pilot studies and using it to highlight the likely reasoning behind choices made and the possible additions or improvements that might be considered. Page 74 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Appendix A: Empirical Estimation Frameworks Econometricians have developed various methods to provide estimates of the impact of policy interventions in situations where social experiments are not possible. The following is a non-technical and non-exhaustive outline of the most common methods. All of these methods take advantage of “quasi experiments” arising because of the way the intervention has been undertaken and/or the available data is structured. For a more in-depth and technical overview, a useful reference is Blundell and Costa Dias (2009).24 In each case, we highlight the key assumptions and data requirements. We also provide links to the pilot studies and other related evaluations that have adopted each framework. Quasi-experimental methods requiring baseline and endline observations Difference-in-difference estimation This approach is applied in situations when certain households or employers are affected by an intervention and others are not and we have data on them before and after the intervention takes place. Consider an example based on two groups of households and two periods. In the first period, neither group is affected by the intervention. In the second period, only one of the groups is affected by the intervention, but not the other. The impact of the intervention is then measured by how much the outcome variable (e.g. employment) changes for the impacted (“treatment”) group relative to the non-impacted (“control”) group. Key Assumptions: While this approach does not require that the groups have the same employment (or other) outcomes to start with, it does assume that their outcomes would have evolved in the same way (would have had parallel trends) in the absence of the intervention. Data requirements: Data is needed before and after the intervention, not only on those directly or indirectly impacted by the intervention but also on households that are not impacted by the intervention (even indirectly) and whose employment outcomes would have been expected to change in the same way as those impacted in the absence of the intervention. This may not be possible for households interacting in the same labor market or even for those working for indirectly impacted sectors. A common approach is to rely on 24 For a practical guide, see https://dimewiki.worldbank.org/Quasi-Experimental_Methods. Page 75 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 some kind of geographical separation that mitigates spillovers between groups. But if the intervention is sufficiently large, this may require comparisons across counties, states, provinces or even countries. Triple difference estimation The basic difference-in-difference approach is invalid if we cannot argue that workers or employers with different characteristics would have experienced similar counterfactual employment trends. In this case, if we can distinguish between two household types within each region by whether they are likely to be impacted by the intervention, we may still be able to obtain valid estimates. Specifically, if we can argue that the employment ratio of the two household types (e.g., formal vs. informal) would have followed similar trends across regions in the absence of the intervention, we could estimate the impact by how much this ratio (or difference) would have changed in the region where the intervention took place relative to the other region. This is an example of a “triple difference” estimation. Key Assumptions: The triple difference approach requires weaker assumptions than the diff-in-diff approach but has additional data requirements. Data requirements: We need data before and after the intervention for two regions/areas. Within each region/area we need to be able to identify some households who are impacted and some who are non-impacted (even indirectly) by the intervention. For many types of interventions that have multiple impacts working through labor and goods markets, this seems like a tall order. Example: PS9 Quasi-experimental methods based on differential treatment Matching methods Matching estimators The main purpose of matching is to re-establish the conditions of an experiment when no randomized control group is available. The matching method aims to construct the counterfactual outcomes that would have been experienced by the impacted households in the absence of the intervention by pairing each impacted household with groups of ex ante observationally equivalent non-impacted households. Under the matching assumption, the only remaining difference between the two groups is the impact of the intervention. Note that multivariate regression can be a simple linear example of matching. Page 76 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Key Assumptions: (1) That we can observe all the observable information that simultaneously characterizes whether or not a household was impacted by the intervention and predicts the counterfactual outcome if not impacted. (2) The observable information does not predict exactly whether a household is impacted. Data requirements: Assumption (1) potentially imposes such significant data needs to ensure that all relevant factors are observed (including ability, potential, social networks, and the like) that it makes this approach infeasible. Propensity score matching As noted above, a big problem for matching estimators can be the number of observable variables needed in order to satisfy the assumptions above. With a finite sample, it may be impossible to find an exact match for each impacted household. The idea of propensity score matching is to replace these variables with the likelihood that a household is impacted by the intervention, conditional on these observables (the household’s propensity score). This likelihood must be estimated separately (e.g. using a Logit estimation) and the comparison group for each impacted household must be decided using pre-specified criteria of proximity between the propensity scores for impacted and controls (e.g. the closest ten). Key Assumptions: That the data used to estimate the likelihood has “sufficient” explanatory power and that the criteria for specifying the comparison group are reasonable. Data requirements: A finite set of relevant baseline observations on households who were ultimately impacted and those not impacted by an intervention. Examples: PS19, RS6 Note: In the presence of longitudinal or repeated cross-section data, matching and difference-in-difference can be combined to weaken the underlying assumptions of both methods. Instrumental Variables This approach relies on finding variables which do not directly impact the outcome (exclusion restriction), but which is a determinant of whether or not households or employers are impacted by the intervention (the assignment rule). If the potential impact of the intervention can be assumed to be the same across households, the IV estimator identifies the impact removed of all the biases that emanate from a nonrandomized control. However, if the potential impact varies in unobservable ways across households, the IV estimator will only identify the average impact under strong assumptions (and ones that are unlikely to hold in practice). Page 77 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Key Assumptions: That the variable has explanatory power in determining assignment while clearly satisfying the exclusion assumption. This combination is often hard to satisfy and the exclusion restriction may be hard to test. Data requirements: More than one instrumental variable is needed to test the exclusion restriction. When infeasible, the restriction is often assumed anyway. Example: PS1 Note: There are other econometric methods for identifying causal effects where there is no valid instrument as well, which may be feasible in certain situations as well. For example, Klein and Vella (2010) show that under certain assumptions regarding the dependence of the distribution of the error terms on exogenous variables, exclusion restrictions need not be required. Discontinuity Design Discontinuity design exploits situations where the probability of being impacted by an intervention changes discontinuously with some continuous variable. The discontinuity design estimator uses this discontinuous dependence to identify a local average treatment effect even when the instrument does not satisfy the assumptions needed for IV estimation discussed above (esp. exclusion). Any discontinuity in the relationship between the outcome and the continuous variable across households is attributed to a discontinuous change in impact due to the intervention. Imbens and Lemieux (2008) provide a practical guide to this commonly used approach. Key Assumptions: That the probability with which a household is impacted depends on whether they are below or above some cut-off value of a continuous variable that can be observed across households. Moreover, there are no major differences between people close to the cut off. Data requirements: A continuously measured characteristic of households, or of their situation, for which there is some cut-off that determines whether or not they are impacted by the intervention. Example: RS7 Page 78 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Appendix B: Model Frameworks Here we provide a brief overview of the various model frameworks that are commonly used for evaluating economic impacts. There is a wide variety of frameworks to choose from depending on the context. In each case, we highlight the key assumptions and data requirements. We also provide links to the pilot studies and other related studies that have adopted each framework. Static models Many models used to evaluate impacts are essentially static in nature. They can, however, be used to think about dynamics by comparing static situations over time and allowing key variables to adjust in exogenously specified ways.25 However, in these frameworks there are no endogenous impacts resulting from choices or changes made today on outcomes tomorrow. Consequently, for example, these frameworks do not directly incorporate the effects of changes in savings behavior, investments in education or capital accumulation etc. in response to the intervention. Static partial equilibrium models Partial equilibrium models typically focus on one labor market in one region taking most other prices and quantities as being determined exogenously. In principle, these models could be linked to broader I-O or GE models to generate estimates of indirect effects. Agricultural household models This approach builds on the methodology first developed by Deaton (1989) to study the predicted impact on household welfare of changes in the price of rice on rural and urban Thai families. The analysis often incorporates an equilibrium farm-household production model (Singh et al., 1986), which can allow for income from wage labor, profits or subsistence. Pre-existing household survey data is used to estimate the key relationships between expenditures, incomes and prices. Then price and other changes associated with interventions are “fed in” to generate estimates from microsimulations. Context: Usually rural but could be applied more generally. Assumptions: Tradable goods prices determined exogenously (perhaps with pass-through restrictions). 25 Moreover, such models are often used to estimate the net present value of an intervention over multiple periods. Page 79 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Data requirements: Pre-existing and ongoing household micro surveys, prices of relevant goods and services. Example: PS14, RS2 Urban spatial models A large literature in urban economics has generated a variety of spatial urban models. These frameworks incorporate land density, local amenities and commuting possibilities and may distinguish the residential and work locations of households. This allows for a characterization of the location-specific costs faced by urban residents, which typically help to determine their employment outcomes. Such frameworks are useful for understanding and potentially quantifying the impacts of interventions whose effects are concentrated in urban areas. For example, such interventions might include new commuter links, new amenities and changes in zoning regulations. These models do not typically distinguish sectoral impacts and often focus on overall welfare impacts. Context: Densely populated urban and suburban areas. Assumptions: The main market is the urban land market (rents). Labor markets can be incorporated but sectors of employment are not typically distinguished. Data requirements: Household and labor market survey data with locational indicators (e.g. postal codes). Examples: PS9 Local Multipliers The local multiplier approach estimates the jobs created in the (local) economy when one job is created in a particular sector in the same municipality. A panel database is constructed with information on sectoral employment measured at the local level over time. The impact of the creation of one job in a given location and subsector on another subsector is estimated via reduced-form regressions. The size of the effect could reflect various unspecified transmission channels, such as the demand for local goods, labor intensity and local supply chains, as well as wage and price inflation. Although this can be viewed as an empirical methodology it is still a kind of (non-structural) ex ante modeling approach in that it assumes the resulting estimated multipliers based on past data remain valid for the intervention. Context: Typically an agglomeration, municipality or district defined in the census. Assumptions: Estimated multipliers based on past data remain valid for the intervention. Page 80 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Data requirements: Information on geography, employment and industry contained in the population and housing census. Requires a panel database with information on sectoral employment measured at the local level over time. The cross-section dimension is conveyed by the spatial information contained in the census. Example: RS3 Fixed-price multiplier models These models focus more broadly on multiple sectors in the economy but are not equilibrium models because prices are treated as fixed, and the constraints imposed by factor (labor, capital or land) markets are ignored. Fixed proportions Input-Output (FPIO) multiplier models These models capture the input-output structure of the entire economy under the assumption that the quantity of inputs required by each industry is a constant proportion of the quantity of output that it produces. This yields a linear relationship between the final outputs demanded, which are treated exogenously, and the total gross output produced by each industry. Given a known relationship between output and employment in each sector, the I-O model can, in principle, yield a full accounting of the macro indirect supply-chain employment impacts across sectors arising from an intervention in a single sector. In fact, the model’s implications can be represented by a simple summary formula of these direct and indirect impacts, known as a multiplier matrix. Context: Usually national but can be regionalized (with enough assumptions). Key Assumptions: Fixed-proportions technology with no responsiveness to impacts of intervention. All goods and factor prices fixed, no factor market constraints, consumption expenditures are treated as fixed exogenously, impact on employment usually based on proportionality (but could make use of estimated elasticities, e.g. AIMM). These assumptions imply that markets do not generally clear. This is justified by viewing the estimates as related to short-run impacts in which prices don’t change much. Data requirements: Most countries have publicly available Supply-Use tables that are created by their statistical bureaus, which can be translated into symmetric input-output tables. This is usually only available at the national level but sometimes also at the provincial/state level (e.g. Canada). There are also multi-country standardized I-O tables available (e.g. WIOD). This framework also requires data on labor productivity by sector plus, if possible, elasticities of output with respect to labor input. Examples: PS4, PS20 Page 81 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Social Accounting Matrix (SAM) multiplier models SAM multiplier models typically consist of a production side characterized by a FPIO multiplier model combined with a more fully-specified demand side, thereby making use of additional data usually collected by statistical agencies. In particular, the demand for goods from each sector reflects the aggregate of the demand coming from households, typically allowing for income heterogeneity across them. These incomes depend in part on wage payments received from producers, which in turn reflect employment demand across sectors. Note that SAM multiplier models thereby incorporate “indirect consumption- spillover” employment effects. Context: Usually national, but can be regionalized Key Assumptions: Prices fixed but consumption expenditures are endogenous – household incomes reflect employment outcomes (but not supply choices). Impact of heterogeneity in households’ characteristics (demographics) may be accounted for and this can be used to forecast future demand based on demographic projections. Data requirements: In addition to those for the I-O multiplier models, we need household expenditure shares by sector and household type, the fraction of incomes consumed by household type and non-wage income sources. Examples: PS6, PS13 Static General Equilibrium models GE models are typically distinguished by market clearing and flexible price determination in both non-tradable goods sectors and in factor markets. Tradable goods prices may be determined exogenously with excess demand (supply) being imported (exported). While the nature of GE models can vary along many dimensions, key distinctions are their spatial dimensions (breadth, disaggregation and frictions), the extent to which they incorporate an explicit input-output structure and the extent to which household heterogeneity is accounted for. Regional GE models Regional GE models limit the measurement of impacts from an intervention to a single region (e.g., state or province). Prices of inter-regional and international tradeables are treated exogenously, while local non-tradable goods’ prices and wages are endogenously determined. The demand system is typically generated from a static household optimization with a particular preference specification (e.g., Stone-Geary). Local labor supply may be treated exogenously or may be determined as part of the household’s Page 82 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 optimization. Most regional GE models do not incorporate an explicit input-output structure due to lack of regional data. Context: Limiting impacts to a single region may be appropriate for localized interventions Data requirements: Regional input-output tables do not typically exist, so regionalization would be needed if one wants to break down employment impacts into direct and indirect effects by region. Example: PS14 National GE models National GE models measure impacts at a national level. Prices of international tradeables are exogenous, while domestic non-tradable goods’ prices and wages are endogenously determined. National labor supply may or may not be treated as exogenous. These models implicitly assume that labor is mobile and that goods trade is frictionless across regions (and perhaps sectors) , so they can abstract from regional aspects. The demand system is typically generated from a static household optimization problem with a particular preference specification. Many national GE models incorporate an explicit IO structure. Context: Appropriate for interventions directly impacting most regions/sectors with national repercussions (e.., fiscal policy changes). Assumptions: Given their aggregative nature, GE models make strong assumptions regarding functional forms (e.g., utility and production relationships). In many cases the functional forms assumed reflect past observation and experience, but they also reflect the need for tractability and uniqueness of equilibrium outcomes. Data requirements: Parameterization of any GE model may require a substantial amount of data, depending on the number of parameters that must be pinned down. In some countries/regions, ranges of estimates of key parameters may already exist from past studies. In others, they may need to be estimated which, as usual, may be resource intensive. For those incorporating an IO structure, many countries provide adequate IO tables at least at a summary level. However, such models may require a lot of information regarding elasticities of substitution across intermediate goods and services. Example: PS17 Multi-regional spatial GE models Multi-regional spatial GE models also typically measure impacts at a regional or national level. But these models incorporate meaningful spatial effects by incorporating frictions in Page 83 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 labor allocation and goods markets. Prices of international tradable goods and services are exogenous, while domestic interregional and non-tradable goods’ prices and wages are endogenously determined. National labor supply is often exogenous, but labor and/or goods trade is imperfectly mobile across locations, reflecting moving costs, transport or other barriers. Context: These models are likely to be appropriate for thinking about fairly large-scale interventions in one or more regions that are likely to have significant impacts on the location of workers across regions (e.g. rural versus urban). Key Assumptions: prices of internationally tradable goods exogenous; inter-regional tradable goods and local non-tradable goods prices are endogenous; often endogenous allocation of labor across regions (with mobility frictions). Data requirements: Combines those of regional and national GE models. Example: PS1 Models with endogenous dynamics The impacts of many interventions take time to unfold and we may be interested in distinguishing shorter-run impacts from longer-run ones. Moreover, the transitional dynamics may derive from the accumulation or decumulation of various kinds of assets (e.g., wealth, physical and human capital, knowledge, natural resources) that will affect future outcomes. A central issue when modeling such endogenous dynamics is the extent to which the choices of households and firms today reflect their expectations of what the future holds (e.g. as a result of interventions) and how such expectations might reasonably be incorporated into modeling frameworks. Models with non-rational expectations In a dynamic context, households and firms take actions today that reflect their expectations about what might happen in the future. This creates a challenge for modeling since it requires assumptions specifying how these expectations are formed. Common simplified modeling approaches in applied work include: ● pinning down key choice variables exogenously (e.g. constant savings rate), ● assuming that the expected value of a variable is equal to its current value, or are essentially extrapolated from empirical estimates of its past behavior. ● assuming expected values are a weighted average of current and steady state values Page 84 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Models with no capital investment or reallocation Some models abstract from dynamic portfolio decisions but still need to specify expectations that affect dynamic production decisions. Short-run dynamic IO/SAM models (Avelino and Hewings, 2019), for example, incorporate dynamics due to supply-chain constraints as the economy transitions to its steady state (Example: RS4). Models with capital investment but with exogenous saving rates These models also abstract from dynamic portfolio decisions but do specify dynamic transition equations for capital assets that relate periods in a consistent fashion. These include Recursive Dynamic GE models (Example: PS17) and hybrid dynamic input-output models. The latter assume goods and factor prices are fixed in the short run but are a function of the resulting excess supply/demand in the long run (Example: RS5). Rational expectations models The assumption of “rational expectations” imposes the requirement that expectations regarding future outcomes are formulated optimally, taking into account the equilibrium of the model. While ensuring internal consistency of the modeling, this level of sophistication on the part of households and firms appears to be viewed as unrealistic or too restrictive by many practitioners. Nevertheless, it is a core feature of most dynamic macroeconomic models discussed by academics and some policymakers (e.g., central banks). In part this is because it removes a degree of arbitrariness that can have significant effects on model predictions. Moreover, it is often justified by appealing to the idea that, while most people do not formulate their own models of the economy, they may instead base their expectations on the forecasts of more sophisticated “experts” who do. Dynamic partial equilibrium models Of particular relevance here are models which characterize the dynamic investment decisions of household-producers that have limited access to capital markets and who supply their own labor to production. Such a model may be taken to represent informal and/or rural agricultural producers and the focus is on optimal investment decisions in the presence of a household’s need to smooth consumption over time. Context: Informal/agricultural self-employed household-producers with limited access to capital markets Key Assumptions: Incomplete capital markets; consumption-smoothing motive. Data requirements: Targets to parameterize utility and production functions Example: PS15 Page 85 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Dynamic General Equilibrium (GE) models with no capital investment Some dynamic forward-looking GE models abstract from capital and other traditional assets, in order to focus on labor market frictions and goods trade.26 In this case, the relevant “assets” of a household consist of the characteristics of the job (wage, tenure, locations, sector) that it currently holds at a point in time. Examples include labor-market search and matching models and multi-regional GE models with mobility and trading frictions. Recent examples include Caliendo, Parro and Rossi-Hansberg (2018) and Caliendo, Dvorkin and Parro (2019). Dynamic General Equilibrium (GE) models with endogenous capital investment Fully specified dynamic GE models typically incorporate optimal portfolio decisions of households, consumption expenditure allocations across goods and services, labor supply choices, entry/exit decisions of firms, capital (and other asset) accumulation. Few of these models explicitly specify a complete input-output structure. Examples include neoclassical growth models, endogenous growth models and Dynamic Stochastic General Equilibrium (DSGE) models (see Christiano, Eichenbaum and Trabandt, 2018). 26 The computational complexity of dynamic rational expectations GE models often imposes constraints on the number of state variables that can reasonably be incorporated and/or may require fairly coarse approximations. Page 86 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Appendix C: Pilot Studies Table C1. Brief Summary of Pilot Studies (Expected) Study Author(s) Timeline Frameworks Budget PS1 Chad, Mathilde Completed ● Extrapolation from reduced-form N/A Cameroon - Lebrand (2021) estimates using instrumental Infrastructure variables approach and Structural ● Multi-regional spatial general Change in the equilibrium model Lake Chad region PS2 Senegal ● Dynamic growth model with informal sector ● CGE modeling proposed PS3 Kenya - N/A N/A ● Extrapolation from reduced form N/A Digital Economy estimates using past data Acceleration project PS4 Mozambique N/A 2021 - 2022 ● Ex ante IO/SAM modeling $125,000 - IFC investment ● Ex post case study, including a USD in Agribusiness survey instrument and interviews targeting the suppliers of the IFC client firm and, if feasible, their “second-tier” suppliers (details N/A) PS5 Lesotho - N/A N/A ● Extrapolation from reduced form N/A Transport estimates of impact of existing Infrastructure bridges across individuals and Connectivity Project PS6 Uganda - N/A N/A ● Ex ante estimation of the net $185,000 Investing in impact at the intervention level USD Forests and against an estimated Protected Areas counterfactual for ● Ex ante estimation of the broader Climate-Smart economy-wide employment Development impact. (i) “business as usual”, (ii) Project net direct employment impact and (iii) indirect value chain impacts. Impacts coming from consumption spillovers to be assessed using “standard multipliers”. Page 87 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 (Expected) Study Author(s) Timeline Frameworks Budget PS7 Uzbekistan - N/A N/A ● Telephone opinion survey N/A Livestock Sector Development Projects PS8 Rwanda - N/A N/A ● Qualitative evaluation based on N/A Energy Sector interviews Development policy PS9 Tanzania - Juliana Completed ● Reduced-form ex post empirical N/A Dar es Salaam Aguilar-Restrep (2020) evaluation using a Bus Rapid Transit o, Clare Balboni, triple-difference approach System (BRT) Gharad Bryan, ● Urban spatial model for ex ante Melanie assessment of future investments Morten, Bilal Siddiqi PS10 Tonga - N/A N/A ● Aquaculture: core value-chain N/A Pathway to analysis of investments to set up a Sustainable pearl farm Oceans ● For offshore and in-shore fisheries, indirect employment impacts per dollar invested based on multipliers from similar investments PS11 Nigeria - N/A N/A ● N/A N/A Bus Rapid Transit PS12 Kiribati - N/A N/A ● N/A. Considering simple $170,000 Pacific Islands bottom-up simulation models of USD Regional fishery value chains, and possibly Oceanscape input-output/SAM models Program PS13 Bangladesh N/A 2022 - 2027 ● Ex ante assessment jobs impacts of NA - Private investments in spatial area Investment and influenced by the SEZ (details N/A) Digital Enterprise ● Ex post survey monitoring of job (PRIDE) project impacts of the SEZ and its area of influence, during project implementation (2022-27) and beyond (details N/A) PS14 Cambodia - J. Edward N/A ● Agricultural producer-household N/A Sustainable Taylor, Heng model parameterized using micro Landscape and Zhu survey data Ecotourism ● General equilibrium model of the Project local regional economy, parameterized using estimated Page 88 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 (Expected) Study Author(s) Timeline Frameworks Budget production functions and input-output information PS15 Angola - N/A N/A ● A dynamic household producer N/A Strengthening model of a borrowing constrained the National household is specified, but is not Social Protection really used. Increased labor System demand is assumed proportional to cash transfers. PS16 Bangladesh N/A N/A ● Extrapolation based on past N/A - Enhancing estimates of elasticity of indirect Digital employment w.r.t. direct Government & employment Economy Project PS17 Ethiopia - N/A N/A ● Survey to directly measure direct N/A Second job estimates Agricultural ● Recursive general equilibrium Growth Project model (AGP2) PS18 Kenya - N/A N/A ● Core job channels analysis N/A National and Rural Inclusive Growth Project PS19 Mali - N/A N/A ● Extrapolation from reduced form N/A Promote Access estimates of impact of credit to Finance, access on employment, using Entrepreneurship matching methods and Employment ● Estimation of impact of additional Project income on employment at aggregate level PS20 Ghana - N/A 2021 - 2022 ● Ex ante IO/SAM modeling $125,000 IFC investment in ● Ex post case study, including a USD manufacturing survey instrument and interviews sector targeting the suppliers of the IFC client firm and, if feasible, their “second-tier” suppliers (details N/A) Page 89 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Appendix D: Recent Related Studies RS1 Muralidharan, K., P. Niehaus and S. Sukhtankar (2020). “General equilibrium effects of (improving) public employment programs: experimental evidence from India”. Poverty Action Lab Working Paper. https://www.povertyactionlab.org/sites/default/files/research-paper/General-Equilibriu m-Effects-of-Public-Employment_Muralidharan-Niehaus-Sukhtankar_December2019.pdf RS2 Louhichi, K., Tillie, P., Ricome, A., and Gomez y Paloma, S (2020), “Modelling Farm-household Livelihoods in Developing Economies: Insights from three country case studies using LSMS-ISA data”. Joint Research Center Technical Report #118822, European Commission. https://publications.jrc.ec.europa.eu/repository/bitstream/JRC118822/jrc118822-online.p df RS3 Charpe, M. (2019). “Sectoral employment multipliers in Rwanda: Comparing local multipliers and input-output analysis.” STRENGTHEN Publication Series Working Paper No. 13. International Labour Organization. https:/ /www.ilo.org/wcmsp5/groups/public/---ed_emp/---ifp_skills/documents/publi cation/wcms_723283.pdf RS4 C. Cotton, B. Kashi, H. Lloyd-Ellis F. Tremblay and B. Crowley (2022), "Quantifying the Economics Impacts of COVID-19 Policy Responses on Canada's Provinces in (Almost) Real Time". Canadian Journal of Economics, vol. 55. https://www.econ.queensu.ca/sites/econ.queensu.ca/files/Lloyd-Ellis/caje_12567_Rev3. pdf RS5 IFC (2018). “Economy-wide Private Impact Quantification Model EPIQ Ethiopia Pilot.” https://www.jobsanddevelopment.org/wp-content/uploads/2018/10/EPIQ-Ethiopia.pdf RS6 H. Shin, K.H. Kim, J. Kim and E. Lee (2020). “Adolescent Employment, Mental Health, and Suicidal Behavior: A Propensity Score Matching Approach.” International Journal of Environmental Research and Public Health, vol. 17, issue 18. https://doi.org/10.3390/ijerph17186835 RS7 F. Zimmert and A. Zorn, Alexander (2021) “Direct payments and on-farm employment: evidence from a spatial regression discontinuity design” German Association of Agricultural Economists (GEWISOLA) conference paper. https://ageconsearch.umn.edu/record/317052/ Page 90 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 References Avelino, A. F. T., and G. J. D. Hewings (2019). “The challenge of estimating the impact of disasters: Many approaches, many limitations and a compromise,” in Y. Okuyama and A. Rose, eds., Advances in Spatial and Economic Modeling of Disaster Impacts, pp. 163–189, Switzerland AG: Springer Nature. Beegle, Kathleen; Coudouel, Aline; Monsalve, Emma (2018). Realizing the Full Potential of Social Safety Nets in Africa. Africa Development Forum;. Washington, DC: World Bank. © World Bank. https:/ /openknowledge.worldbank.org/handle/10986/29789 License: CC BY 3.0 IGO. Bonach, Peter (1987). “Power and centrality: A family of measures.” American Journal of Sociology, vol. 92 (5). https://doi.org/10.1086/228631. Blundell, R., Dias, M.C. (2009). “Alternative Approaches to Evaluation in Empirical Microeconomics.” The Journal of Human Resources , Summer, 2009, Vol. 44, No. 3 (Summer, 2009), pp. 565-640. https:/ /www.jstor.org/stable/20648911 Caliendo, L., M. Dvorkin and F. Parro (2019). “Trade and labor market dynamics: General equilibrium analysis of the China trade shock.” Econometrica, vol. 87 (3), pp. 741-835. Carvalho, V.M., and A. Tahbaz-Salehi (2019). “Production Networks: A Primer.” Annual Review of Economics, vol. 11, pp. 635–63. https://doi.org/10.1146/annurev-economics-080218030212. Christiano, L. J., M. S. Eichenbaum, M. S. and M. Trabandt (2018). “On DSGE models”, in Journal of Economic Perspectives, Vol. 32, No. 3, pp. 113–40. DOI: 10.1257/ jep.32.3.113. Deaton, A. (1989), “Rice Prices and Income Distribution in Thailand: A Non-parametric Analysis,” Economic Journal, vol. 99, issue 395, pp. 1-37. https://www.jstor.org/stable/2234068 ILO (2020). “Reference Guide for Employment Impact Assessment.” Employment Policy Department. https://www.ilo.org/wcmsp5/groups/public/---ed_emp/---ifp_skills/documents/publi cation/wcms_750484.pdf Imbens, G. W. and T. Lemieux (2008). “Regression discontinuity designs: A guide to practice.” Journal of Econometrics, vol. 142, issue 2, pp. 615-635. https://www.sciencedirect.com/science/article/abs/pii/S0304407607001091 Page 91 of 92 Measuring Jobs Impacts Draft Final Report - 2022-12-05 Klein, R. and F. Vella (2010). “Estimating a class of triangular simultaneous equations models without exclusion restrictions.” Journal of Econometrics, vol. 154, issue 2, pp. 154-164. Magli, M. (2021). “The Direct and Indirect Effect of Services Offshoring on Local Labour Market Outcomes.” Center for Economic Studies Ifo Institute for Economic Research Working Paper No. 8413. https://www.cesifo.org/en/publikationen/2020/working-paper/direct-and-indirect-effe ct-services-offshoring-local-labour-market Singh, I., L. Squire and J. Strauss (1986). Agricultural Household Models, Extensions, Applications and Policy, The World Bank and The Johns Hopkins University Press. Taylor, J.E., Filipski, M.J. (2014). “Beyond Experiments in Development Economics: Local Economy-Wide Impact Evaluation.” Oxford Scholarship Online. DOI 10.1093/acprof:oso/9780198707875.001.0001 World Bank Group Jobs Group (2017). “Monitoring and evaluation for job operations.” Jobs M&E Toolkit, Volume 1, June 2017. Page 92 of 92 Address: 1776 G St, NW, Washington, DC 20006 Website: http://www.worldbank.org/en/topic/jobsanddevelopment Twitter: @WBG_Jobs Blog: https://blogs.worldbank.org/jobs/