Publication: Beyond Baseline and Follow-up : The Case for More T in Experiments
Loading...
Published
2011-04-01
ISSN
Date
2012-03-19
Author(s)
Editor(s)
Abstract
The vast majority of randomized experiments in economics rely on a single baseline and single follow-up survey. If multiple follow-ups are conducted, the reason is typically to examine the trajectory of impact effects, so that in effect only one follow-up round is being used to estimate each treatment effect of interest. While such a design is suitable for study of highly autocorrelated and relatively precisely measured outcomes in the health and education domains, this paper makes the case that it is unlikely to be optimal for measuring noisy and relatively less autocorrelated outcomes such as business profits, household incomes and expenditures, and episodic health outcomes. Taking multiple measurements of such outcomes at relatively short intervals allows the researcher to average out noise, increasing power. When the outcomes have low autocorrelation, it can make sense to do no baseline at all. Moreover, the author shows how for such outcomes, more power can be achieved with multiple follow-ups than allocating the same total sample size over a single follow-up and baseline. The analysis highlights the large gains in power from ANCOVA rather than difference-in-differences when autocorrelations are low and a baseline is taken. The paper discusses the issues involved in multiple measurements, and makes recommendations for the design of experiments and related non-experimental impact evaluations.
Link to Data Set
Citation
“McKenzie, David. 2011. Beyond Baseline and Follow-up : The Case for More T in Experiments. Policy Research working paper ; no. WPS 5639. © World Bank. http://hdl.handle.net/10986/3403 License: CC BY 3.0 IGO.”
Digital Object Identifier
Associated URLs
Associated content
Other publications in this report series
Publication The Macroeconomic Implications of Climate Change Impacts and Adaptation Options(Washington, DC: World Bank, 2025-05-29)Estimating the macroeconomic implications of climate change impacts and adaptation options is a topic of intense research. This paper presents a framework in the World Bank's macrostructural model to assess climate-related damages. This approach has been used in many Country Climate and Development Reports, a World Bank diagnostic that identifies priorities to ensure continued development in spite of climate change and climate policy objectives. The methodology captures a set of impact channels through which climate change affects the economy by (1) connecting a set of biophysical models to the macroeconomic model and (2) exploring a set of development and climate scenarios. The paper summarizes the results for five countries, highlighting the sources and magnitudes of their vulnerability --- with estimated gross domestic product losses in 2050 exceeding 10 percent of gross domestic product in some countries and scenarios, although only a small set of impact channels is included. The paper also presents estimates of the macroeconomic gains from sector-level adaptation interventions, considering their upfront costs and avoided climate impacts and finding significant net gross domestic product gains from adaptation opportunities identified in the Country Climate and Development Reports. Finally, the paper discusses the limits of current modeling approaches, and their complementarity with empirical approaches based on historical data series. The integrated modeling approach proposed in this paper can inform policymakers as they make proactive decisions on climate change adaptation and resilience.Publication South Africa’s Fragmented Cities: The Unequal Burden of Labor Market Frictions(Washington, DC: World Bank, 2026-01-08)Using high-resolution administrative, census, and satellite data, this paper shows that South African cities are characterized by spatial mismatches between where people live and where jobs are located, relative to 20 global peers. Areas within 5 kilometers of commercial centers have 9,300 fewer residents per square kilometer than expected, which is 60 percent below the global median. Poor, dense neighborhoods are most affected. In Johannesburg, a 10-percentile increase in distance from the nearest business hub corresponds to a 3.7-percentile drop in asset wealth (a proxy of household wellbeing) and 4.9-percentile drop in employment. In Cape Town, the declines are 4.0 and 3.7 percentiles, respectively. Employment is 87 percent lower in the poorest decile than the richest in Johannesburg and 61 percent lower in Cape Town. These findings suggest that South Africa’s spatial organization of people and economic activity constrains agglomeration and reinforces inequality. This methodology provides a scalable and standardized data-driven framework to analyze spatial accessibility and agglomeration frictions in complex, data-constrained urban systems.Publication The Evolution of Local Participatory Democracy in Nepal(Washington, DC: World Bank, 2025-11-05)Nepal is, according to its constitution, among the world’s most decentralized countries, with a long and complex tradition of local-level public participation. This paper traces the evolution of Nepal’s modern participatory institutions, examining the extent to which they are “induced” by external interventions versus being “organically” rooted in indigenous practices. The paper identifies three broad phases: an initial focus on participation in project implementation; a subsequent phase that expanded citizen engagement; and a third phase of citizen empowerment, culminating in the 2015 federal constitution, which granted unprecedented local autonomy. The analysis yields five key findings. First, over the past 50 years, successive reforms have progressively expanded opportunities for citizens to influence local decision-making. Second, these reforms have integrated traditional participatory mechanisms into formal institutions of local government. Third, although central-level initiatives exist, most participatory platforms continue to operate at the local level. Fourth, the federal constitution has created a new landscape of local democracy, embedding autonomy and accountability. Fifth, although they are still valued in many ethnic and territorial communities, traditional participatory practices are gradually disappearing. The paper concludes by offering policy recommendations to help donor agencies and governments strengthen Nepal’s democratic trajectory. It argues that effective interventions should build on Nepal’s deep participatory traditions while recognizing the constitutional reality of far-reaching local autonomy.Publication Institutional Capacity for Policy Implementation: An Analytical Framework(Washington, DC: World Bank, 2026-01-07)State capacity is an important prerequisite for policy implementation, yet at the country level it is difficult to measure, assess, and reform. This paper proposes a focus on institutional capacity: the ability of public institutions to implement the specific policy mandates for which they are responsible. Based on a review of existing literature, the paper defines the different dimensions that compose institutional capacity and groups them into two cross-cutting categories: organizational dimensions (personnel, financial resources, information systems, and management practices) and governance dimensions (transparency, independence, and accountability). The paper proposes measures for organizational and governance dimensions using existing data, shows intra-institutional variation of these measures within countries, and discusses how new data could be collected for better measurement of these concepts. Finally, the paper illustrates how the framework can be used to diagnose the sources of common problems related to weak policy implementation.Publication Closing the Gender Gap in Entrepreneurship: Overcoming Challenges in Law and Practice for Female Entrepreneurs(Washington, DC: World Bank, 2026-01-07)Despite significant strides toward gender equality, women around the world continue to encounter systemic obstacles that hinder their entrepreneurial success. This paper systematically reviews the literature on the barriers female entrepreneurs face and the solutions proposed to overcome these challenges. It discusses institutional factors, financial factors, human capital factors, and social and cultural factors. The literature overview is complemented by a series of stylized facts that illustrate how overcoming some of these existing barriers is correlated with improved women’s entrepreneurship and female labor force participation, drawing on the World Bank’s Women, Business and the Law database as well as the World Bank’s Enterprise Surveys. The findings underscore the need for creating an enabling environment where women can thrive as entrepreneurs.
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication In Pursuit of Balance : Randomization in Practice in Development Field Experiments(World Bank, Washington, DC, 2008-10)Randomized experiments are increasingly used in development economics, with researchers now facing the question of not just whether to randomize, but how to do so. Pure random assignment guarantees that the treatment and control groups will have identical characteristics on average, but in any particular random allocation, the two groups will differ along some dimensions. Methods used to pursue greater balance include stratification, pair-wise matching, and re-randomization. This paper presents new evidence on the randomization methods used in existing randomized experiments, and carries out simulations in order to provide guidance for researchers. Three main results emerge. First, many researchers are not controlling for the method of randomization in their analysis. The authors show this leads to tests with incorrect size, and can result in lower power than if a pure random draw was used. Second, they find that in samples of 300 or more, the different randomization methods perform similarly in terms of achieving balance on many future outcomes of interest. However, for very persistent outcome variables and in smaller sample sizes, pair-wise matching and stratification perform best. Third, the analysis suggests that on balance the re-randomization methods common in practice are less desirable than other methods, such as matching.Publication Replicating Replication : Due Diligence in Roodman and Morduch’s Replication of Pitt and Khandker (1998)(World Bank, Washington, DC, 2012-11)"The Impact of Microcredit on the Poor in Bangladesh: Revisiting the Evidence," by David Roodman and Jonathan Morduch (2011) is the most recent of a sequence of papers and postings that seeks to refute the findings of the Pitt and Khandker (1998) article "The Impact of Group-Based Credit on Poor Households in Bangladesh: Does the Gender of Participants Matter?" that microcredit for women had significant, favorable effects on poverty reduction. In this paper the authors show that these latest Roodman and Morduch claims are based on seriously flawed econometric methods and theory and a lack of due diligence in formulating models and interpreting output from packaged software. On the basis of Roodman and Morduch's preferred two-stage least squares regression, an alternative calculation of the standard errors would lead one to conclude that the problem with Pitt and Khandker is that they underestimate the positive and statistically significant effect of women's credit on household consumption. As in their previous efforts, the methods of Roodman and Morduch are shown to bias the findings in the direction of rejecting the results of Pitt and Khandker. We also further examine two aspects of our instrumental variable approach that have been attacked by Roodman and Morduch. The first is the validity of the exclusion restrictions underlying the use of interactions between program choice and the set of exogenous variables (including the village fixed effects) as instruments. The second is the application of the "one-half acre" program eligibility rule. The authors show that identification does not require both of these, and present new results dropping each assumption in turn. The results originally reported in the Pitt and Khandker paper hold up extremely well in this new analysis.Publication Brazil within Brazil : Testing the Poverty Map Methodology in Minas Gerais(World Bank, Washington, DC, 2008-02)The small-area estimation technique developed for producing poverty maps has been applied in a large number of developing countries. Opportunities to formally test the validity of this approach remain rare due to lack of appropriately detailed data. This paper compares a set of predicted welfare estimates based on this methodology against their true values, in a setting where these true values are known. A recent study draws on Monte Carlo evidence to warn that the small-area estimation methodology could significantly over-state the precision of local-level estimates of poverty, if underlying assumptions of spatial homogeneity do not hold. Despite these concerns, the findings in this paper for the state of Minas Gerais, Brazil, indicate that the small-area estimation approach is able to produce estimates of welfare that line up quite closely to their true values. Although the setting considered here would seem, a priori, unlikely to meet the homogeneity conditions that have been argued to be essential for the method, confidence intervals for the poverty estimates also appear to be appropriate. However, this latter conclusion holds only after carefully controlling for community-level factors that are correlated with household level welfare.Publication Using Repeated Cross-Sections to Explore Movements in and out of Poverty(2011-01-01)Movements in and out of poverty are of core interest to both policymakers and economists. Yet the panel data needed to analyze such movements are rare. In this paper, the authors build on the methodology used to construct poverty maps to show how repeated cross-sections of household survey data can allow inferences to be made about movements in and out of poverty. They illustrate that the method permits the estimation of bounds on mobility, and provide non-parametric and parametric approaches to obtaining these bounds. They test how well the method works on data sets for Vietnam and Indonesia where we are able to compare our method to true panel estimates. The results are sufficiently encouraging to offer the prospect of some limited, basic, insights into mobility and poverty duration in settings where historically it was judged that the data necessary for such analysis were unavailable.Publication Missing(ness) in Action : Selectivity Bias in GPS-Based Land Area Measurements(World Bank, Washington, DC, 2013-06)Land area is a fundamental component of agricultural statistics, and of analyses undertaken by agricultural economists. While household surveys in developing countries have traditionally relied on farmers' own, potentially error-prone, land area assessments, the availability of affordable and reliable Global Positioning System (GPS) units has made GPS-based area measurement a practical alternative. Nonetheless, in an attempt to reduce costs, keep interview durations within reasonable limits, and avoid the difficulty of asking respondents to accompany interviewers to distant plots, survey implementing agencies typically require interviewers to record GPS-based area measurements only for plots within a given radius of dwelling locations. It is, therefore, common for as much as a third of the sample plots not to be measured, and research has not shed light on the possible selection bias in analyses relying on partial data due to gaps in GPS-based area measures. This paper explores the patterns of missingness in GPS-based plot areas, and investigates their implications for land productivity estimates and the inverse scale-land productivity relationship. Using Multiple Imputation (MI) to predict missing GPS-based plot areas in nationally-representative survey data from Uganda and Tanzania, the paper highlights the potential of MI in reliably simulating the missing data, and confirms the existence of an inverse scale-land productivity relationship, which is strengthened by using the complete, multiply-imputed dataset. The study demonstrates the usefulness of judiciously reconstructed GPS-based areas in alleviating concerns over potential measurement error in farmer-reported areas, and with regards to systematic bias in plot selection for GPS-based area measurement.
Users also downloaded
Showing related downloaded files
Publication The US-China Trade War and Global Reallocations(World Bank, Washington, DC, 2022-01)This paper studies global trade responses to the US-China trade war. It estimates the tariff impacts on product-level exports to the US, China, and rest of world. On average, countries decreased exports to China and increased exports to the US and rest of world. Most countries export products that complement the US and substitute China, and a subset operate along downward-sloping supplies. Heterogeneity in responses, rather than specialization, drives export variation across countries. Surprisingly, global trade increased in the products targeted by tariffs. Thus, despite ending the trend towards tariff reductions, the trade war did not halt global trade growth.Publication Growth in Syria(Taylor and Francis, 2021-07-05)This paper addresses three questions: (1) what would have been the growth and income trajectory of Syria in the absence of war; (2) given the war, what explains the reduction in economic growth; and (3) what potential growth scenarios for Syria there could be in the aftermath of war. Conflict impact estimates point to negative GDP growth of −12% on average over 2011–2018, with output contracting to about one-third of the 2010 level. In post-conflict simulation scenarios, the growth drivers are affected by the assumed levels of reconstruction assistance, repatriation of refugees, and productivity improvements associated with three political settlement outcomes: a baseline (Sochi-plus) moderate scenario, an optimistic (robust political settlement) scenario, and a pessimistic (de facto balance of power) scenario. Respectively for these scenarios, GDP per capita average growth in the next two decades is projected to be 6.1%, 8.2%, or 3.1%, assuming a final and stable resolution of the conflict.Publication Global Seaweed(Washington, DC: World Bank, 2023-08-14)With its ability to sink carbon, sustain marine biodiversity, employ women, and unlock value chains, seaweed farming demonstrates how development, climate, and nature work together to generate value and uplift communities. Seaweed farming can help build a world free of poverty on a livable planet and has enormous growth potential. This report has identified ten global seaweed markets with the potential to grow by an additional USD 11.8 billion by 2030. Yet, much of the seaweed sector’s value remains untapped - it has clear growth potential beyond its current markets. Today, most farmed seaweed is used for direct human consumption, as fresh feed in aquaculture, or as hydrocolloids. However, seaweed-farmed products may be able to displace fossil fuels in sectors such as fabrics and plastics; can provide ecosystem services, such as carbon sequestration and nitrogen cycling; and can generate socioeconomic benefits in fragile coastal communities. Further, the market is currently dominated by a handful of Asian countries, which produce 98 percent of farmed seaweed by volume globally. Opportunities for growth in new regions and applications are high.Publication Comparative Situational Analysis of Comprehensive Abortion Care in Four Southern African Countries(Taylor and Francis, 2025-01-08)We report on a comparative situational analysis of comprehensive abortion care (CAC) in Botswana, Eswatini, Lesotho and Namibia. We conducted systematic literature searches and country consultations and used a reparative health justice approach (with four dimensions) for the analysis. The following findings pertain to all four countries, except where indicated. Individual material dimension: pervasive gender-based violence (GBV); unmet need for contraception (15−17%); high HIV prevalence; poor abortion access for rape survivors; fees for sexual and reproductive health (SRH) services (Eswatini). Collective material dimension: no clear national budgeting for SRH; over-reliance on donor funding (Eswatini; Lesotho); no national CAC guidelines or guidance on legal abortion access; poor data collection and management systems; shortage and inequitable distribution of staff; few facilities providing abortion care. Individual symbolic dimension: gender norms justify GBV; stigma attached to both abortion and unwed or early pregnancies. Collective symbolic dimension: policy commitments to reducing unsafe abortion and to post-abortion care, but not to increasing access to legal abortion; inadequate research; contradictions in abortion legislation (Botswana); inadequate staff training in CAC. Political will to ensure CAC within the country’s legislation is required. Reparative health justice comparisons provide a powerful tool for foregrounding necessary policy and practice change.Publication Digital Africa(Washington, DC: World Bank, 2023-03-13)All African countries need better and more jobs for their growing populations. "Digital Africa: Technological Transformation for Jobs" shows that broader use of productivity-enhancing, digital technologies by enterprises and households is imperative to generate such jobs, including for lower-skilled people. At the same time, it can support not only countries’ short-term objective of postpandemic economic recovery but also their vision of economic transformation with more inclusive growth. These outcomes are not automatic, however. Mobile internet availability has increased throughout the continent in recent years, but Africa’s uptake gap is the highest in the world. Areas with at least 3G mobile internet service now cover 84 percent of Africa’s population, but only 22 percent uses such services. And the average African business lags in the use of smartphones and computers as well as more sophisticated digital technologies that catalyze further productivity gains. Two issues explain the usage gap: affordability of these new technologies and willingness to use them. For the 40 percent of Africans below the extreme poverty line, mobile data plans alone would cost one-third of their incomes—in addition to the price of access devices, apps, and electricity. Data plans for small- and medium-size businesses are also more expensive than in other regions. Moreover, shortcomings in the quality of internet services—and in the supply of attractive, skills-appropriate apps that promote entrepreneurship and raise earnings—dampen people’s willingness to use them. For those countries already using these technologies, the development payoffs are significant. New empirical studies for this report add to the rapidly growing evidence that mobile internet availability directly raises enterprise productivity, increases jobs, and reduces poverty throughout Africa. To realize these and other benefits more widely, Africa’s countries must implement complementary and mutually reinforcing policies to strengthen both consumers’ ability to pay and willingness to use digital technologies. These interventions must prioritize productive use to generate large numbers of inclusive jobs in a region poised to benefit from a massive, youthful workforce—one projected to become the world’s largest by the end of this century.