Publication: An Evaluation of the Performance of Regression Discontinuity Design on PROGRESA
Loading...
Published
2004-09
ISSN
Date
2013-06-24
Author(s)
Buddelmeyer, Hielke
Editor(s)
Abstract
While providing the most reliable method of evaluating social programs, randomized experiments in developing and developed countries alike are accompanied by political risks and ethical issues that jeopardize the chances of adopting them. In this paper the authors use a unique data set from rural Mexico collected for the purposes of evaluating the impact of the PROGRESA poverty alleviation program to examine the performance of a quasi-experimental estimator, the Regression Discontinuity Design (RDD). Using as a benchmark the impact estimates based on the experimental nature of the sample, we examine how estimates differ when we use the RDD as the estimator for evaluating program impact on two key indicators: child school attendance and child work. Overall the performance of the RDD was remarkably good. The RDD estimates of program impact agreed with the experimental estimates in 10 out of the 12 possible cases. The two cases in which the RDD method failed to reveal any significant program impact on the school attendance of boys and girls were in the first year of the program (round 3). RDD estimates comparable to the experimental estimates were obtained when we used as a comparison group children from non-eligible households in the control localities.
Link to Data Set
Citation
“Buddelmeyer, Hielke; Skoufias, Emmanuel. 2004. An Evaluation of the Performance of Regression Discontinuity Design on PROGRESA. Policy Research Working Paper;No.3386. © World Bank. http://hdl.handle.net/10986/14153 License: CC BY 3.0 IGO.”
Digital Object Identifier
Associated URLs
Associated content
Other publications in this report series
Publication Climate and Social Sustainability in Fragility, Conflict, and Violence Contexts(Washington, DC: World Bank, 2026-01-07)Climate change is widely recognized as a driver of violent conflict, but its broader social effects remain less understood. Ignoring these dimensions risks a vicious cycle where climate policies might undermine socially just adaptation. Evidence is still limited on how climate shocks influence political participation, trust, or migration. This paper helps fill that gap by examining links between climate change, conflict, and social sustainability, with a focus on inclusion, resilience, cohesion, and legitimacy. Using secondary data from 2019–24, the study applies simple correlation-based methods to test three hypotheses on the nature, severity, and composition of these associations. The analysis combines multiple climate impact measures, new conflict classifications, recent social sustainability frameworks, and controls for population and geography. The results reveal strong correlations—not causation—between climate events and contexts of fragility, conflict, and violence. Climate impacts are most pronounced in both national and subnational conflict settings. The study also finds robust links between fragility, conflict, and violence and low levels of social sustainability, reflecting its role as both a driver and consequence of conflict. Some dimensions—such as violent events and insecurity—appear weaker in areas most affected by climate shocks. Two of the hypotheses are supported, and one remains inconclusive.Publication The Macroeconomic Implications of Climate Change Impacts and Adaptation Options(Washington, DC: World Bank, 2025-05-29)Estimating the macroeconomic implications of climate change impacts and adaptation options is a topic of intense research. This paper presents a framework in the World Bank's macrostructural model to assess climate-related damages. This approach has been used in many Country Climate and Development Reports, a World Bank diagnostic that identifies priorities to ensure continued development in spite of climate change and climate policy objectives. The methodology captures a set of impact channels through which climate change affects the economy by (1) connecting a set of biophysical models to the macroeconomic model and (2) exploring a set of development and climate scenarios. The paper summarizes the results for five countries, highlighting the sources and magnitudes of their vulnerability --- with estimated gross domestic product losses in 2050 exceeding 10 percent of gross domestic product in some countries and scenarios, although only a small set of impact channels is included. The paper also presents estimates of the macroeconomic gains from sector-level adaptation interventions, considering their upfront costs and avoided climate impacts and finding significant net gross domestic product gains from adaptation opportunities identified in the Country Climate and Development Reports. Finally, the paper discusses the limits of current modeling approaches, and their complementarity with empirical approaches based on historical data series. The integrated modeling approach proposed in this paper can inform policymakers as they make proactive decisions on climate change adaptation and resilience.Publication Institutional Capacity for Policy Implementation: An Analytical Framework(Washington, DC: World Bank, 2026-01-07)State capacity is an important prerequisite for policy implementation, yet at the country level it is difficult to measure, assess, and reform. This paper proposes a focus on institutional capacity: the ability of public institutions to implement the specific policy mandates for which they are responsible. Based on a review of existing literature, the paper defines the different dimensions that compose institutional capacity and groups them into two cross-cutting categories: organizational dimensions (personnel, financial resources, information systems, and management practices) and governance dimensions (transparency, independence, and accountability). The paper proposes measures for organizational and governance dimensions using existing data, shows intra-institutional variation of these measures within countries, and discusses how new data could be collected for better measurement of these concepts. Finally, the paper illustrates how the framework can be used to diagnose the sources of common problems related to weak policy implementation.Publication South Africa’s Fragmented Cities: The Unequal Burden of Labor Market Frictions(Washington, DC: World Bank, 2026-01-08)Using high-resolution administrative, census, and satellite data, this paper shows that South African cities are characterized by spatial mismatches between where people live and where jobs are located, relative to 20 global peers. Areas within 5 kilometers of commercial centers have 9,300 fewer residents per square kilometer than expected, which is 60 percent below the global median. Poor, dense neighborhoods are most affected. In Johannesburg, a 10-percentile increase in distance from the nearest business hub corresponds to a 3.7-percentile drop in asset wealth (a proxy of household wellbeing) and 4.9-percentile drop in employment. In Cape Town, the declines are 4.0 and 3.7 percentiles, respectively. Employment is 87 percent lower in the poorest decile than the richest in Johannesburg and 61 percent lower in Cape Town. These findings suggest that South Africa’s spatial organization of people and economic activity constrains agglomeration and reinforces inequality. This methodology provides a scalable and standardized data-driven framework to analyze spatial accessibility and agglomeration frictions in complex, data-constrained urban systems.Publication Investment in Emerging and Developing Economies(Washington, DC: World Bank, 2026-01-07)The world faces a pressing challenge to meet key development objectives amid slowing growth and rising macroeconomic and geopolitical risks. With the number of job seekers rising rapidly, infrastructure shortfalls continuing to be large, and climate costs mounting, the case for a significant investment push has never been stronger. Yet the capacity to respond in many emerging markets and developing economies has eroded. Since the global financial crisis, investment growth has slowed to about half its pace in the 2000s, with both public and private investment weakening. Foreign direct investment inflows—a critical source of capital, technology, and managerial know-how—have also fallen sharply and become increasingly concentrated, leaving low-income countries with only a marginal share. The risks of further retrenchment are significant, as trade tensions, policy uncertainty, and elevated debt levels continue to weigh on investment. Reigniting momentum will require ambitious domestic reforms to strengthen institutions, rebuild macro-fiscal stability, and deepen trade and investment integration—the foundations of a supportive business climate. At the same time, international cooperation is indispensable. A renewed commitment to a predictable system of cross-border trade and investment flows, combined with scaled-up financial support and sustained technical assistance, is essential to help emerging markets and developing economies—especially low-income countries and economies in fragile and conflict situations—bridge financing gaps and implement the domestic reforms needed to restore investment as an engine of growth, jobs, and development.
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication How Do Ex Ante Simulations Compare with Ex Post Evaluations? Evidence from the Impact of Conditional Cash Transfer Programs(2011-06-01)This paper compares the ex ante simulation of the impacts of conditional cash transfer programs against the ex post estimates of impacts obtained from experimental evaluations. Using data on program-eligible households in treatment areas from the same baseline surveys that are used for experimental evaluations of conditional cash transfer programs in Mexico and Ecuador, the authors use a micro-simulation model to derive ex ante estimates of the impact of the programs on enrollment rates and poverty. The estimates reveal that ex ante predictions of certain impacts of conditional cash transfer programs match up well against the benchmark estimates of ex post experimental studies. The findings seem to support the use of this model to assess the potential impact and cost efficiency of a conditional cash transfer program ex ante, in order to inform decisions about how the program would be designed.Publication Evaluating the Impact of Conditional Cash Transfer Programs : Lessons from Latin America(World Bank, Washington, DC, 2003-08)Unlike most development initiatives, conditional cash transfer programs recently introduced in the Latin America and the Caribbean region have been subject to rigorous evaluations of their effectiveness. These programs provide money to poor families, conditional on certain behavior, usually investments in human capital-such as sending children to school or bringing them to health centers on a regular basis. Rawlings and Rubio review the experience in evaluating the impact of these programs, exploring the application of experimental and quasi-experimental evaluation methods and summarizing results from programs launched in Brazil, Honduras, Jamaica, Mexico, and Nicaragua. Evaluation results from the first generation of programs in Brazil, Mexico, and Nicaragua show that conditional cash transfer programs are effective in promoting human capital accumulation among poor households. There is clear evidence of success in increasing enrollment rates, improving preventive health care, and raising household consumption. Despite this promising evidence, many questions remain unanswered about the impact of conditional cash transfer programs, including those concerning their effectiveness under different country conditions and the sustainability of the welfare impacts.Publication Gender Equality, Poverty and Economic Growth(World Bank, Washington, DC, 2007-09)This paper reviews empirical findings from economic analyses of the role of gender equality and women's empowerment in reducing poverty and stimulating growth. Going beyond the large literature documenting the impact of female education on a range of development outcomes, the paper presents evidence on the impact of women's access to markets (labor, land, and credit) and women's decision-making power within households on poverty reduction and productivity at the individual and household level. The paper also summarizes evidence from studies examining the relationship between gender equality and poverty reduction and growth at the macro level. Although micro level effects of gender equality on individual productivity and human development outcomes have been well documented and have important ramifications for aggregate economic performance, establishing an empirical relationship between gender equality and poverty reduction and growth at the macro level has proven to be more challenging. The paper concludes by identifying priority areas for future research.Publication A Gender (R)evolution in the Making? Expanding Women's Economic Opportunities in Central America : A Decade in Review(Washington, DC, 2012-01)A Gender Revolution in the Making was produced by the World Bank as a special report on the dichotomy of the economic participation of women in Central America from 1997-2007. Central America made considerable progress during 1997-2006: stronger economic and political stability, overall positive albeit modest economic growth and improvements in socio-economic outcomes. Nonetheless, the panorama for women in the region, and in particular their ability to work and generate income is mixed. Labor force participation of women in Central America remained unchanged over the decade, with one in every two women of working age still not participating economically. This is lower than the rest of Latin America, a region with the lowest labor force participation in the world, although it has improved significantly in recent years. And yet, women in the region have made major advances in closing the gender earnings gaps and increasing their ability to earn income. Women's contribution to overall income generation in Central America rose significantly over the decade. This report aims to identify the factors behind these three trends. However, stagnant labor force participation rates for women contrasted with narrowing earnings gaps and a greater contribution to overall income apparent in Central America from 1997 to 2006.Publication Evaluating the Impact of Mexico's Quality Schools Program : The Pitfalls of Using Nonexperimental Data(World Bank, Washington, DC, 2006-10)The authors evaluate whether increasing school resources and decentralizing management decisions at the school level improves learning in a developing country. Mexico's Quality Schools Program (PEC), following many other countries and U.S. states, offers US$15,000 grants for public schools to implement five-year improvement plans that the school's staff and community design. Using a three-year panel of 74,700 schools, the authors estimate the impact of the PEC on dropout, repetition, and failure using two common nonexperimental methods-regression analysis and propensity score matching. The methods provide similar but nonidentical results. The preferred estimator, difference-in-differences with matching, reveals that participation in the PEC decreases dropout by 0.24 percentage points, failure by 0.24 percentage points, and repetition by 0.31 percentage points-an economically small but statistically significant impact. The PEC lacks measurable impact on outcomes in indigenous schools. The results suggest that a combination of increased resources and local management can produce small improvements in school outcomes, though perhaps not in the most troubled school systems.
Users also downloaded
Showing related downloaded files
Publication Breaking the Conflict Trap : Civil War and Development Policy(Washington, DC: World Bank and Oxford University Press, 2003)Most wars are now civil wars. Even though international wars attract enormous global attention, they have become infrequent and brief. Civil wars usually attract less attention, but they have become increasingly common and typically go on for years. This report argues that civil war is now an important issue for development. War retards development, but conversely, development retards war. This double causation gives rise to virtuous and vicious circles. Where development succeeds, countries become progressively safer from violent conflict, making subsequent development easier. Where development fails, countries are at high risk of becoming caught in a conflict trap in which war wrecks the economy and increases the risk of further war. The global incidence of civil war is high because the international community has done little to avert it. Inertia is rooted in two beliefs: that we can safely 'let them fight it out among themselves' and that 'nothing can be done' because civil war is driven by ancestral ethnic and religious hatreds. The purpose of this report is to challenge these beliefs.Publication Design Thinking for Social Innovation(2010-07)Designers have traditionally focused on enchancing the look and functionality of products.Publication Government Matters III : Governance Indicators for 1996-2002(World Bank, Washington, DC, 2003-08)The authors present estimates of six dimensions of governance covering 199 countries and territories for four time periods: 1996, 1998, 2000, and 2002. These indicators are based on several hundred individual variables measuring perceptions of governance, drawn from 25 separate data sources constructed by 18 different organizations. The authors assign these individual measures of governance to categories capturing key dimensions of governance and use an unobserved components model to construct six aggregate governance indicators in each of the four periods. They present the point estimates of the dimensions of governance as well as the margins of errors for each country for the four periods. The governance indicators reported here are an update and expansion of previous research work on indicators initiated in 1998 (Kaufmann, Kraay, and Zoido-Lobat 1999a,b and 2002). The authors also address various methodological issues, including the interpretation and use of the data given the estimated margins of errors.Publication Governance Matters VIII : Aggregate and Individual Governance Indicators 1996–2008(2009-06-01)This paper reports on the 2009 update of the Worldwide Governance Indicators (WGI) research project, covering 212 countries and territories and measuring six dimensions of governance between 1996 and 2008: Voice and Accountability, Political Stability and Absence of Violence/Terrorism, Government Effectiveness, Regulatory Quality, Rule of Law, and Control of Corruption. These aggregate indicators are based on hundreds of specific and disaggregated individual variables measuring various dimensions of governance, taken from 35 data sources provided by 33 different organizations. The data reflect the views on governance of public sector, private sector and NGO experts, as well as thousands of citizen and firm survey respondents worldwide. The authors also explicitly report the margins of error accompanying each country estimate. These reflect the inherent difficulties in measuring governance using any kind of data. They find that even after taking margins of error into account, the WGI permit meaningful cross-country comparisons as well as monitoring progress over time. The aggregate indicators, together with the disaggregated underlying indicators, are available at www.govindicators.org.Publication Governance Matters IV : Governance Indicators for 1996-2004(World Bank, Washington, DC, 2005-06)The authors present the latest update of their aggregate governance indicators, together with new analysis of several issues related to the use of these measures. The governance indicators measure the following six dimensions of governance: (1) voice and accountability; (2) political instability and violence; (3) government effectiveness; (4) regulatory quality; (5) rule of law, and (6) control of corruption. They cover 209 countries and territories for 1996, 1998, 2000, 2002, and 2004. They are based on several hundred individual variables measuring perceptions of governance, drawn from 37 separate data sources constructed by 31 organizations. The authors present estimates of the six dimensions of governance for each period, as well as margins of error capturing the range of likely values for each country. These margins of error are not unique to perceptions-based measures of governance, but are an important feature of all efforts to measure governance, including objective indicators. In fact, the authors give examples of how individual objective measures provide an incomplete picture of even the quite particular dimensions of governance that they are intended to measure. The authors also analyze in detail changes over time in their estimates of governance; provide a framework for assessing the statistical significance of changes in governance; and suggest a simple rule of thumb for identifying statistically significant changes in country governance over time. The ability to identify significant changes in governance over time is much higher for aggregate indicators than for any individual indicator. While the authors find that the quality of governance in a number of countries has changed significantly (in both directions), they also provide evidence suggesting that there are no trends, for better or worse, in global averages of governance. Finally, they interpret the strong observed correlation between income and governance, and argue against recent efforts to apply a discount to governance performance in low-income countries.