Publication: Measuring the Measurement Error: Method to Qualitatively Validate Survey Data
Loading...
Files in English
255 downloads
Published
2016-05
ISSN
0304-3878
Date
2016-05-09
Editor(s)
Abstract
Empirical social science relies heavily on self-reported data, but subjects may misreport behaviors, especially sensitive ones such as crime or drug abuse. If a treatment influences survey misreporting, it biases causal estimates. We develop a validation technique that uses intensive qualitative work to assess survey misreporting and pilot it in a field experiment where subjects were assigned to receive cash, therapy, both, or neither. According to survey responses, both treatments reduced crime and other sensitive behaviors. Local researchers spent several days with a random subsample of subjects after surveys, building trust and obtaining verbal confirmation of four sensitive behaviors and two expenditures. In this instance, validation showed survey underreporting of most sensitive behaviors was low and uncorrelated with treatment, while expenditures were under reported in the survey across all arms, but especially in the control group. We use these data to develop measurement error bounds on treatment effects estimated from surveys.
Link to Data Set
Digital Object Identifier
Associated URLs
Associated content
Other publications in this report series
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication Reducing Crime and Violence(World Bank, Washington, DC, 2016-04)The paper shows that self-control, time preferences, and values are malleable in adults, and that investments in these skills and preferences reduce crime and violence. The authors recruited criminally-engaged Liberian men and randomized half to eight weeks of group cognitive behavioral therapy, fostering self-regulation, patience, and noncriminal values. They also randomized $200 grants. Cash alone and therapy alone dramatically reduced crime and violence, but effects dissipated within a year. When cash followed therapy, however, crime and violence decreased by as much as 50 percent for at least a year. They hypothesize that cash reinforced therapy's lessons by prolonging practice and self-investment.Publication Assessing Workplace Social, Emotional, and Behavioral Skills in Africa(Washington, DC: World Bank, 2025-05-21)Social, emotional, and behavioral skills are critical for success across life domains, yet research is constrained by a lack of internationally validated measures for adult populations. Existing tools often assess isolated skills and are predominantly validated in Western, school- aged samples. To address these limitations, this study developed and validated the Effective Socio-emotional skills To gain Economic EMpowerment framework, comprising 14 distinct social, emotional, and behavioral skills with prior demonstrated relevance to economic outcomes. The framework’s self-report scales were tested among adults in six Sub-Saharan African countries (Côte d’Ivoire, Nigeria, the Republic of Congo, Rwanda, South Africa, and Tanzania), spanning diverse cultural and linguistic contexts (English, French, Hausa, Swahili, and Yoruba). The results confirm the psychometric validity of the scales, supporting their utility in both research and practice. The framework categorizes skills as intrapersonal or interpersonal, awareness or management, and agentic or communal, providing a robust tool to unpack which skills matter for employment and earnings and how this differs by gender. By enabling exploration of social, emotional, and behavioral skills in underrepresented and cross-cultural contexts, use of the Effective Socio-emotional skills To gain Economic EMpowerment self-report scales advances theoretical and practical understanding of social, emotional, and behavioral skills in adult populations.Publication Sex, Lies, and Surveys(World Bank, Washington, DC, 2019-02)This paper examines how easily observable interviewer characteristics, such as gender and physical attractiveness, and more difficult to observe characteristics, such as attitudes and beliefs, affect adolescent girls' disclosure of sexual behavior during a baseline survey for an adolescent girls program in Liberia. The study finds that girls are more likely to report sexual activity to better-looking interviewers, and less likely to do so to interviewers holding more discriminatory gender attitudes and greater expectations about the program. The study finds no evidence that the gender of the interviewer matters.Publication The Limits of Commitment(World Bank, Washington, DC, 2018-10)Working with a private bank in Ghana, this study examines the impacts of a commitment savings product designed to help clients taking repeated overdrafts break their debt cycles. Overall, the product significantly increased savings with the bank without increasing overdrafts. However, after accounting for other sources of savings, the study finds that clients with above-median baseline overdraft histories do not accrue new savings during the commitment period. Rather, they draw down other savings to offset the committed amount and take on new debt. In contrast, individuals with below-median overdraft histories significantly increase savings during and after the commitment period.Publication Missing(ness) in Action : Selectivity Bias in GPS-Based Land Area Measurements(World Bank, Washington, DC, 2013-06)Land area is a fundamental component of agricultural statistics, and of analyses undertaken by agricultural economists. While household surveys in developing countries have traditionally relied on farmers' own, potentially error-prone, land area assessments, the availability of affordable and reliable Global Positioning System (GPS) units has made GPS-based area measurement a practical alternative. Nonetheless, in an attempt to reduce costs, keep interview durations within reasonable limits, and avoid the difficulty of asking respondents to accompany interviewers to distant plots, survey implementing agencies typically require interviewers to record GPS-based area measurements only for plots within a given radius of dwelling locations. It is, therefore, common for as much as a third of the sample plots not to be measured, and research has not shed light on the possible selection bias in analyses relying on partial data due to gaps in GPS-based area measures. This paper explores the patterns of missingness in GPS-based plot areas, and investigates their implications for land productivity estimates and the inverse scale-land productivity relationship. Using Multiple Imputation (MI) to predict missing GPS-based plot areas in nationally-representative survey data from Uganda and Tanzania, the paper highlights the potential of MI in reliably simulating the missing data, and confirms the existence of an inverse scale-land productivity relationship, which is strengthened by using the complete, multiply-imputed dataset. The study demonstrates the usefulness of judiciously reconstructed GPS-based areas in alleviating concerns over potential measurement error in farmer-reported areas, and with regards to systematic bias in plot selection for GPS-based area measurement.
Users also downloaded
Showing related downloaded files
Publication Migrants, Towns, Poverty and Jobs(World Bank, Washington, DC, 2018-04-01)For a long time, the urbanization and development discourse has coincided with a focus on economic growth and big cities. Yet, much of the world's new urbanization is taking place in smaller urban entities (towns), and the composition of urbanization may well bear on the speed of poverty reduction. This paper reviews the latter question within the context of Tanzania. It starts from the observation that migration to towns contributed much more to poverty reduction than migration to cities because many more (poor) rural migrants ended up in Tanzania's towns than its cities, despite larger welfare gains from moving to the city. Drawing on the findings from a series of studies, looking at this from different angles (theoretical and empirical, quantitative and qualitative), the paper shows how towns are better at enabling the rural poor to access off-farm employment and exit poverty because they are more nearby. It concludes with a call for greater consideration of the role of towns in accelerating Africa's poverty reduction.Publication Digital Africa(Washington, DC: World Bank, 2023-03-13)All African countries need better and more jobs for their growing populations. "Digital Africa: Technological Transformation for Jobs" shows that broader use of productivity-enhancing, digital technologies by enterprises and households is imperative to generate such jobs, including for lower-skilled people. At the same time, it can support not only countries’ short-term objective of postpandemic economic recovery but also their vision of economic transformation with more inclusive growth. These outcomes are not automatic, however. Mobile internet availability has increased throughout the continent in recent years, but Africa’s uptake gap is the highest in the world. Areas with at least 3G mobile internet service now cover 84 percent of Africa’s population, but only 22 percent uses such services. And the average African business lags in the use of smartphones and computers as well as more sophisticated digital technologies that catalyze further productivity gains. Two issues explain the usage gap: affordability of these new technologies and willingness to use them. For the 40 percent of Africans below the extreme poverty line, mobile data plans alone would cost one-third of their incomes—in addition to the price of access devices, apps, and electricity. Data plans for small- and medium-size businesses are also more expensive than in other regions. Moreover, shortcomings in the quality of internet services—and in the supply of attractive, skills-appropriate apps that promote entrepreneurship and raise earnings—dampen people’s willingness to use them. For those countries already using these technologies, the development payoffs are significant. New empirical studies for this report add to the rapidly growing evidence that mobile internet availability directly raises enterprise productivity, increases jobs, and reduces poverty throughout Africa. To realize these and other benefits more widely, Africa’s countries must implement complementary and mutually reinforcing policies to strengthen both consumers’ ability to pay and willingness to use digital technologies. These interventions must prioritize productive use to generate large numbers of inclusive jobs in a region poised to benefit from a massive, youthful workforce—one projected to become the world’s largest by the end of this century.Publication World Bank Annual Report 2023(Washington, DC: World Bank, 2023-09-28)This annual report, which covers the period from July 1, 2022, to June 30, 2023, has been prepared by the Executive Directors of both the International Bank for Reconstruction and Development (IBRD) and the International Development Association (IDA)—collectively known as the World Bank—in accordance with the respective bylaws of the two institutions. Ajay Banga, President of the World Bank Group and Chairman of the Board of Executive Directors, has submitted this report, together with the accompanying administrative budgets and audited financial statements, to the Board of Governors.Publication Taxes, Spending, and Equity: International Patterns and Lessons for Developing Countries(Washington, DC: World Bank, 2025-11-17)Taxes and public spending underpin the basic administration of government and finance the human capital and infrastructure investments needed for economic growth. They can also have a significant and immediate impact on poverty and inequality. The question of how public finance can support longer-term growth objectives while promoting equity has become even more important in recent years, given the high fiscal deficits and debt levels most countries emerged with in the aftermath of the COVID-19 pandemic. These included the increasing cost of debt and the need to restart environmentally sustainable growth while helping households address the learning losses and other social scars caused by the pandemic. This paper examines the global evidence on which households pay which taxes and who benefits from what spending, and critically, the net effect on different households across the income distribution. The aim is to identify the patterns and lessons that emerge for designing progressive fiscal policies. A global dataset of 96 countries is assembled, spanning all regions of the world and all national income levels, grounded in the Commitment to Equity (CEQ) approach to fiscal incidence.Publication Monitoring COVID-19 Impacts on Firms in Ethiopia, Report No. 9(World Bank, Washington, DC, 2020-10-15)For the current study, six rounds of the HFPS-F survey are used, mainly focusing on the business closure module from round six. The six rounds are roughly three weeks apart and were implemented between April 15 and September 8, 2020 in Addis Ababa. The sampling strategy is explained in detail in a companion technical note. Six survey briefs highlighting the impact of the pandemic on firms’ operations and labor dynamics are already produced along with two special topic briefs focusing on the gendered difference of the pandemic and how firms were affected by the State of Emergency. The current brief examines firm closure in relation to firms direct or indirect exposure to the social and economic impacts of the pandemic. The good news is that most of the firms that are closed have stopped production or services temporarily with less than 2 percent of firm closures reported to be permanent in R6 (roughly August). The following analysis thus focuses on temporary closures and mostly rely on data from 436 firms, of which 108 were temporarily closed and 328 were open at the time of the R6 survey.