Publication: In Pursuit of Balance : Randomization in Practice in Development Field Experiments
No Thumbnail Available
Date
2009
ISSN
19457782
Published
2009
Author(s)
Editor(s)
Abstract
We present new evidence on the randomization methods used in existing experiments, and new simulations comparing these methods. We find that many papers do not describe the randomization in detail, implying that better reporting is needed. Our simulations suggest that in samples of 300 or more, the different methods perform similarly. However, for very persistent outcome variables, and in smaller samples, pair-wise matching and stratification perform best and appear to dominate the rerandomization methods commonly used in practice. The simulations also point to specific recommendations for which variables to balance on, and for which controls to include in the ex post analysis.
Link to Data Set
Associated content
Other publications in this report series
Journal
Journal Volume
Journal Issue
Citations
Collections
Related items
Showing items related by metadata.
Publication In Pursuit of Balance : Randomization in Practice in Development Field Experiments(World Bank, Washington, DC, 2008-10)Randomized experiments are increasingly used in development economics, with researchers now facing the question of not just whether to randomize, but how to do so. Pure random assignment guarantees that the treatment and control groups will have identical characteristics on average, but in any particular random allocation, the two groups will differ along some dimensions. Methods used to pursue greater balance include stratification, pair-wise matching, and re-randomization. This paper presents new evidence on the randomization methods used in existing randomized experiments, and carries out simulations in order to provide guidance for researchers. Three main results emerge. First, many researchers are not controlling for the method of randomization in their analysis. The authors show this leads to tests with incorrect size, and can result in lower power than if a pure random draw was used. Second, they find that in samples of 300 or more, the different randomization methods perform similarly in terms of achieving balance on many future outcomes of interest. However, for very persistent outcome variables and in smaller sample sizes, pair-wise matching and stratification perform best. Third, the analysis suggests that on balance the re-randomization methods common in practice are less desirable than other methods, such as matching.Publication Eliciting Probabilistic Expectations with Visual Aids in Developing Countries: How Sensitive Are Answers to Variations in Elicitation Design?(2011)Eliciting subjective probability distributions in developing countries is often based on visual aids such as beans to represent probabilities and intervals on a sheet of paper to represent the support. We conduct an experiment in India which tests the sensitivity of elicited expectations to variations in three facets of the elicitation methodology: the number of beans, the design of the support (predetermined or self-anchored), and the ordering of questions. Our results show remarkable robustness to variations in elicitation design. Nevertheless, the added precision offered by using more beans and a larger number of intervals with a predetermined support improves accuracy.Publication Surveying Migrant Households: A Comparison of Census-Based, Snowball and Intercept Point Surveys(2009)New representative surveys of households of migrants exist, limiting our ability to study the effects of international migration on sending families. We report the results of an experiment that was designed to compare the performance of three alternative survey methods in collecting data from Japanese-Brazilian families, many of whom send migrants to Japan. The three surveys that were conducted were households selected randomly from a door-to-door listing using the Brazilian census to select census blocks, a snowball survey using Nikkei community groups to select the seeds and an intercept point survey that was collected at Nikkei community gatherings, ethnic grocery stores, sports clubs, and other locations where family members of migrants are likely to congregate. We analyse how closely well-designed snowball and intercept point surveys can approach the much more expensive census-based method in terms of giving information on the characteristics of migrants, the level of remittances received and the incidence and determinants of return migration.Publication Beyond Baseline and Follow-up : The Case for More T in Experiments(2011-04-01)The vast majority of randomized experiments in economics rely on a single baseline and single follow-up survey. If multiple follow-ups are conducted, the reason is typically to examine the trajectory of impact effects, so that in effect only one follow-up round is being used to estimate each treatment effect of interest. While such a design is suitable for study of highly autocorrelated and relatively precisely measured outcomes in the health and education domains, this paper makes the case that it is unlikely to be optimal for measuring noisy and relatively less autocorrelated outcomes such as business profits, household incomes and expenditures, and episodic health outcomes. Taking multiple measurements of such outcomes at relatively short intervals allows the researcher to average out noise, increasing power. When the outcomes have low autocorrelation, it can make sense to do no baseline at all. Moreover, the author shows how for such outcomes, more power can be achieved with multiple follow-ups than allocating the same total sample size over a single follow-up and baseline. The analysis highlights the large gains in power from ANCOVA rather than difference-in-differences when autocorrelations are low and a baseline is taken. The paper discusses the issues involved in multiple measurements, and makes recommendations for the design of experiments and related non-experimental impact evaluations.Publication Testing the Importance of Search Frictions, Matching, and Reservation Prestige through Randomized Experiments in Jordan(World Bank Group, Washington, DC, 2014-09-01)Unemployment rates for tertiary-educated youth in Jordan are high, as is the duration of unemployment. Two randomized experiments in Jordan were used to test different theories that may explain this phenomenon. The first experiment tested the role of search and matching frictions by providing firms and job candidates with an intensive screening and matching service based on educational backgrounds and psychometric assessments. Although more than 1,000 matches were made, youth rejected the opportunity to even have an interview in 28 percent of cases, and when a job offer was received, they rejected this offer or quickly quit the job 83 percent of the time. A second experiment built on the first by examining the willingness of educated, unemployed youth to apply for jobs of varying levels of prestige. Youth applied to only a small proportion of the job openings they were told about, with application rates higher for higher prestige jobs than lower prestige jobs. Youth failed to show up for the majority of interviews scheduled for low prestige jobs. The results suggest that reservation prestige is an important factor underlying the unemployment of educated Jordanian youth.
Users also downloaded
Showing related downloaded files
No results found.