Publication: Program Evaluation and Spillover Effects
Loading...
Files in English
1,163 downloads
Published
2015-04
ISSN
Date
2015-05-04
Author(s)
Editor(s)
Abstract
This paper is a practical guide for researchers and practitioners who want to understand spillover effects in program evaluation. The paper defines spillover effects and discusses why it is important to measure them. It explains how to design a field experiment to measure the average effects of the treatment on eligible and ineligible subjects for the program in the presence of spillover effects. In addition, the paper discusses the use of nonexperimental methods for estimating spillover effects when the experimental design is not a viable option. Evaluations that account for spillover effects should be designed such that they explain the cause of these effects and whom they affect. Such an evaluation design is necessary to avoid inappropriate policy recommendations and neglecting important mechanisms through which the program operates.
Link to Data Set
Citation
“Angelucci, M.; Di Maro, Vincenzo. 2015. Program Evaluation and Spillover Effects. Policy Research Working Paper;No. 7243. © World Bank. http://hdl.handle.net/10986/21857 License: CC BY 3.0 IGO.”
Digital Object Identifier
Associated URLs
Associated content
Other publications in this report series
Publication The Macroeconomic Implications of Climate Change Impacts and Adaptation Options(Washington, DC: World Bank, 2025-05-29)Estimating the macroeconomic implications of climate change impacts and adaptation options is a topic of intense research. This paper presents a framework in the World Bank's macrostructural model to assess climate-related damages. This approach has been used in many Country Climate and Development Reports, a World Bank diagnostic that identifies priorities to ensure continued development in spite of climate change and climate policy objectives. The methodology captures a set of impact channels through which climate change affects the economy by (1) connecting a set of biophysical models to the macroeconomic model and (2) exploring a set of development and climate scenarios. The paper summarizes the results for five countries, highlighting the sources and magnitudes of their vulnerability --- with estimated gross domestic product losses in 2050 exceeding 10 percent of gross domestic product in some countries and scenarios, although only a small set of impact channels is included. The paper also presents estimates of the macroeconomic gains from sector-level adaptation interventions, considering their upfront costs and avoided climate impacts and finding significant net gross domestic product gains from adaptation opportunities identified in the Country Climate and Development Reports. Finally, the paper discusses the limits of current modeling approaches, and their complementarity with empirical approaches based on historical data series. The integrated modeling approach proposed in this paper can inform policymakers as they make proactive decisions on climate change adaptation and resilience.Publication South Africa’s Fragmented Cities: The Unequal Burden of Labor Market Frictions(Washington, DC: World Bank, 2026-01-08)Using high-resolution administrative, census, and satellite data, this paper shows that South African cities are characterized by spatial mismatches between where people live and where jobs are located, relative to 20 global peers. Areas within 5 kilometers of commercial centers have 9,300 fewer residents per square kilometer than expected, which is 60 percent below the global median. Poor, dense neighborhoods are most affected. In Johannesburg, a 10-percentile increase in distance from the nearest business hub corresponds to a 3.7-percentile drop in asset wealth (a proxy of household wellbeing) and 4.9-percentile drop in employment. In Cape Town, the declines are 4.0 and 3.7 percentiles, respectively. Employment is 87 percent lower in the poorest decile than the richest in Johannesburg and 61 percent lower in Cape Town. These findings suggest that South Africa’s spatial organization of people and economic activity constrains agglomeration and reinforces inequality. This methodology provides a scalable and standardized data-driven framework to analyze spatial accessibility and agglomeration frictions in complex, data-constrained urban systems.Publication The Evolution of Local Participatory Democracy in Nepal(Washington, DC: World Bank, 2025-11-05)Nepal is, according to its constitution, among the world’s most decentralized countries, with a long and complex tradition of local-level public participation. This paper traces the evolution of Nepal’s modern participatory institutions, examining the extent to which they are “induced” by external interventions versus being “organically” rooted in indigenous practices. The paper identifies three broad phases: an initial focus on participation in project implementation; a subsequent phase that expanded citizen engagement; and a third phase of citizen empowerment, culminating in the 2015 federal constitution, which granted unprecedented local autonomy. The analysis yields five key findings. First, over the past 50 years, successive reforms have progressively expanded opportunities for citizens to influence local decision-making. Second, these reforms have integrated traditional participatory mechanisms into formal institutions of local government. Third, although central-level initiatives exist, most participatory platforms continue to operate at the local level. Fourth, the federal constitution has created a new landscape of local democracy, embedding autonomy and accountability. Fifth, although they are still valued in many ethnic and territorial communities, traditional participatory practices are gradually disappearing. The paper concludes by offering policy recommendations to help donor agencies and governments strengthen Nepal’s democratic trajectory. It argues that effective interventions should build on Nepal’s deep participatory traditions while recognizing the constitutional reality of far-reaching local autonomy.Publication Institutional Capacity for Policy Implementation: An Analytical Framework(Washington, DC: World Bank, 2026-01-07)State capacity is an important prerequisite for policy implementation, yet at the country level it is difficult to measure, assess, and reform. This paper proposes a focus on institutional capacity: the ability of public institutions to implement the specific policy mandates for which they are responsible. Based on a review of existing literature, the paper defines the different dimensions that compose institutional capacity and groups them into two cross-cutting categories: organizational dimensions (personnel, financial resources, information systems, and management practices) and governance dimensions (transparency, independence, and accountability). The paper proposes measures for organizational and governance dimensions using existing data, shows intra-institutional variation of these measures within countries, and discusses how new data could be collected for better measurement of these concepts. Finally, the paper illustrates how the framework can be used to diagnose the sources of common problems related to weak policy implementation.Publication Closing the Gender Gap in Entrepreneurship: Overcoming Challenges in Law and Practice for Female Entrepreneurs(Washington, DC: World Bank, 2026-01-07)Despite significant strides toward gender equality, women around the world continue to encounter systemic obstacles that hinder their entrepreneurial success. This paper systematically reviews the literature on the barriers female entrepreneurs face and the solutions proposed to overcome these challenges. It discusses institutional factors, financial factors, human capital factors, and social and cultural factors. The literature overview is complemented by a series of stylized facts that illustrate how overcoming some of these existing barriers is correlated with improved women’s entrepreneurship and female labor force participation, drawing on the World Bank’s Women, Business and the Law database as well as the World Bank’s Enterprise Surveys. The findings underscore the need for creating an enabling environment where women can thrive as entrepreneurs.
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication Population Health and Economic Growth(World Bank, Washington, DC, 2008)Health is a direct source of human welfare and also an instrument for raising income levels. The authors discuss a number of mechanisms through which health can affect income, focusing on worker productivity, children's education, savings and investment, and demographic structure. As well as the impact of current illness, health may have large effects on prospective life spans and life cycle behavior. Studies suggest there may be a large effect of health and nutrition in uteri, and in the first few years of life, on physical and cognitive development and economic success as an adult. Macroeconomic evidence for an effect on growth is mixed, with evidence of a large effect in some studies. However, there is a possibility that gains from health may be outweighed by the effect of increased survival on population growth, until a fertility transition occurs. The low cost of some health interventions that have large-scale effects on population health makes health investments a promising policy tool for growth in developing countries. In addition, higher priority could be given to tackling widespread 'neglected' diseases that is, diseases with low mortality burdens that are not priorities from a pure health perspective, but that do have substantial effects on productivity.Publication Environmental Health and Child Survival : Epidemiology, Economics, Experiences(Washington, DC : World Bank, 2008)This report complements Repositioning Nutrition as Central to Development by looking at environmental health issues that affect child health broadly, while also exploring the links through malnutrition. This report argues that environmental health interventions are preventive measures that are imperative to improve child survival with sustainable results in the long term. Preventive measures, such as improving environmental conditions are effective in reducing a child's exposure to a disease agent and thereby averting infection. The overall aim of this report is to provide information to decision-makers on the optimal design of policies to help reduce premature deaths and illness in children under five years of age. To protect the health, development, and wellbeing of young children, decision-makers must identify and reduce environmental risk factors by providing appropriate interventions that prevent and diminish exposures. This study is intended to advance the understanding of what those risk factors are, when and how to reduce children's exposure to them, and how to mitigate their consequent health impact.Publication The Contribution of Traditional Herbal Medicine Practitioners to Kenyan Health Care Delivery : Results from Community Health-seeking Behavior Vignettes and a Traditional Herbal Medicine Practitioner Survey(World Bank, Washington, DC, 2011-09)This study examines the role that Traditional Herbal Medicine Practitioners (THMPs) play in Kenya in the context of its human resources for health crisis. Two surveys were carried out to obtain evidence. The first documented the choices and perceptions of households in 36 communities on seeking medical assistance for eight common illnesses. The second survey asked 258 THMPs in five provinces to identify their knowledge sources, training, common illnesses treated, forms of payment, challenges, and concerns. Community-derived data show that households make reasonable decisions when faced with difficult circumstances: they prefer hospitals when these are affordable and seek care at clinics and health centers when hospitals are too far away. There is significant self-care and use of pharmacies, although THMPs are preferred for worms and lower respiratory problems. In general, THMPs provide an important though diminishing role in the provision of health care; they are not sought out in situations when inadequate care is dangerous, specifically infant diarrhoea and potential TB. Whilst Human Resources for Health (HRH) policies are urgently required to strengthen the conventional health workforce and increase their accessibility for the poor, policies should not ignore the findings from this study: many of the rural poor currently receive services from a traditional health workforce not linked to, or regulated by, the national government. This paper argues that formal recognition of their role by the government and by the conventional medical associations, and a targeted strategy to strengthen and build on the positive qualities evident in many traditional medicine practices may be beneficial to safeguarding the well-being of the poor.Publication Small Cash Rewards for Big Losers(World Bank, Washington, DC, 2015-06)This paper examines the sustainability of weight loss achieved through cash rewards and, for the first time, the potential of monetary incentives to prevent weight cycling. In a three period randomized controlled trial, about 700 obese persons were assigned to two treatment groups, which were promised different cash rewards contingent on the achievement of an individually assigned target weight, and to a control group. Successful participants were subsequently allocated to two treatment groups offered different monetary incentives for maintaining the previously achieved target weight and to a control group. This is the first experiment of this kind that finds sustainable effects of weight loss rewards on the body weight of the obese even 18 months after the rewards were removed. Additional incentives to maintain an achieved body weight improve the sustainability of weight loss only while are in place.Publication Conditional Cash Transfers, Adult Work Incentives, and Poverty(World Bank, Washington, DC, 2006-08)Conditional cash transfer (CCT) programs aim to alleviate poverty through monetary and in-kind benefits, as well as reduce future levels of poverty by encouraging investments in education, health, and nutrition. The success of CCT programs at reducing poverty depends on whether, and the extent to which, cash transfers affect adult work incentives. The authors examine whether the PROGRESA program of Mexico affects adult participation in the labor market and overall adult leisure time, and they link these effects to the impact of the program on poverty. Using the experimental design of PROGRESA's evaluation sample, the authors find that the program does not have any significant effect on adult labor force participation and leisure time. Their findings on adult work incentives are reinforced further by the result that PROGRESA leads to a substantial reduction in poverty. The poverty reduction effects are stronger for the poverty gap and severity of poverty measures.
Users also downloaded
Showing related downloaded files
Publication What Matters Most for School Autonomy and Accountability(World Bank, Washington, DC, 2015-03-30)This framework paper provides an overview of what matters most for school autonomy and accountability. The focus is on public schools at the primary and the secondary level. This paper begins by grounding School Autonomy and Accountability in its theoretical evidence base (impact evaluations, lessons learned from experience, and literature reviews) and then discusses guiding principles and tools for analyzing country policy choices. The goal of this paper is to provide a framework for classifying and analyzing education systems around the world according to the following five policy goals that are critical for enabling effective school autonomy and accountability: 1) level of autonomy in the planning and management of the school budget; 2) level of autonomy in personnel management; 3) role of school councils in school governance; 4) school and student assessment, and 5) accountability to stakeholders. This paper also discusses how country context matters to school autonomy and accountability and how balancing policy goals matters to policy making for improved education quality and learning for all.Publication Road Capacity, Domestic Trade and Regional Outcomes(World Bank, Washington, DC, 2021-09)What is the impact on intranational trade and regional economic outcomes when the quality and lane capacity of an existing paved road network is expanded significantly? This paper investigates this question for the case of Turkey, which undertook a large-scale public investment in roads during the 2000s. Using spatially disaggregated data on road upgrades and domestic transactions, the paper estimates a large positive impact of reduced travel times on trade as well as local manufacturing employment and wages. A quantitative exercise using a workhorse model of spatial equilibrium implies heterogeneous effects across locations, with aggregate real income gains reaching 2–3 percent in the long run. Reductions in travel times increased the local employment-to-population ratio but had no effect on local population. The model is extended by endogenizing the labor supply decision to capture this finding. The model-implied elasticity of employment rates to travel time reductions captures about one-third of the empirical elasticity.Publication Digital Africa(Washington, DC: World Bank, 2023-03-13)All African countries need better and more jobs for their growing populations. "Digital Africa: Technological Transformation for Jobs" shows that broader use of productivity-enhancing, digital technologies by enterprises and households is imperative to generate such jobs, including for lower-skilled people. At the same time, it can support not only countries’ short-term objective of postpandemic economic recovery but also their vision of economic transformation with more inclusive growth. These outcomes are not automatic, however. Mobile internet availability has increased throughout the continent in recent years, but Africa’s uptake gap is the highest in the world. Areas with at least 3G mobile internet service now cover 84 percent of Africa’s population, but only 22 percent uses such services. And the average African business lags in the use of smartphones and computers as well as more sophisticated digital technologies that catalyze further productivity gains. Two issues explain the usage gap: affordability of these new technologies and willingness to use them. For the 40 percent of Africans below the extreme poverty line, mobile data plans alone would cost one-third of their incomes—in addition to the price of access devices, apps, and electricity. Data plans for small- and medium-size businesses are also more expensive than in other regions. Moreover, shortcomings in the quality of internet services—and in the supply of attractive, skills-appropriate apps that promote entrepreneurship and raise earnings—dampen people’s willingness to use them. For those countries already using these technologies, the development payoffs are significant. New empirical studies for this report add to the rapidly growing evidence that mobile internet availability directly raises enterprise productivity, increases jobs, and reduces poverty throughout Africa. To realize these and other benefits more widely, Africa’s countries must implement complementary and mutually reinforcing policies to strengthen both consumers’ ability to pay and willingness to use digital technologies. These interventions must prioritize productive use to generate large numbers of inclusive jobs in a region poised to benefit from a massive, youthful workforce—one projected to become the world’s largest by the end of this century.Publication Global Logistics Indicators, Supply Chain Metrics, and Bilateral Trade Patterns(World Bank, Washington, DC, 2005-11)Past research into the determinants of international trade highlighted the importance of the basic spatial gravity model augmented by additional variables representing sources of friction. Studies modeled many sources of friction using various proxies, including indices based on expert judgment in some cases. This paper focuses on logistics friction and draws on a data set recently compiled by the World Bank with specific quantitative metrics of logistics performance in terms of time, cost, and variability in time. It finds that the new variables that relate directly to logistics performance have a statistically significant relationship with the level of bilateral trade. It also finds that a single logistics index can capture virtually all of the explanatory power of multiple logistics indicators. The findings should spur public and private agencies that have direct or indirect power over logistics performance to focus attention on reducing sources of friction so as to improve their country's ability to compete in today's global economy. Moreover, since the logistics metrics are directly related to operational performance, countries can use these metrics to target actions to improve logistics and monitor their progress.Publication Monetary Policy Strategies for Latin America(World Bank, Washington, DC, 2001-10)The authors examine possible monetary policy strategies for Latin America that may help lock in the gains the region attained in the fight against inflation in the 1990s. Instead of focusing the debate about the conduct of monetary policy on whether the nominal exchange rate should be fixed or flexible, the focus should be on whether the monetary policy regime appropriately constrains discretion in monetary policymaking. Three basic frameworks deserve serious discussion as possible long-run strategies for monetary policy in Latin America. The authors examine the advantages and disadvantages of a hard exchange-rate peg, monetary targeting, and inflation targeting, in light of monetary policy's recent track record in several Latin American countries, looking for clues about which of the strategies might be best suited to economies in the region. The answer: It depends on the country's institutional environment. Some countries appear not to have the institutions to constrain monetary policy if discretion is allowed. In those countries, there is a strong argument for hard pegs, including full dollarization, that allow little or no discretion to monetary authorities. In countries such as Chile, which can constrain discretion, inflation targeting is likely to produce a monetary policy that keeps inflation low yet appropriately copes with domestic and foreign shocks. Monetary targeting as a strategy for Latin America is not viable because of the likely instability of the relationship between inflation and monetary aggregates, of which there is ample international evidence. No monetary strategy can solve the basic problems that have existed in Latin American economies for a long time. The authors welcome the recent move in Latin American countries toward inflation targeting, but say no policy will succeed unless government policies also create the right institutional environment.