Publication: Measuring the Measurement Error: Method to Qualitatively Validate Survey Data
Loading...
Files in English
250 downloads
Published
2016-05
ISSN
0304-3878
Date
2016-05-09
Editor(s)
Abstract
Empirical social science relies heavily on self-reported data, but subjects may misreport behaviors, especially sensitive ones such as crime or drug abuse. If a treatment influences survey misreporting, it biases causal estimates. We develop a validation technique that uses intensive qualitative work to assess survey misreporting and pilot it in a field experiment where subjects were assigned to receive cash, therapy, both, or neither. According to survey responses, both treatments reduced crime and other sensitive behaviors. Local researchers spent several days with a random subsample of subjects after surveys, building trust and obtaining verbal confirmation of four sensitive behaviors and two expenditures. In this instance, validation showed survey underreporting of most sensitive behaviors was low and uncorrelated with treatment, while expenditures were under reported in the survey across all arms, but especially in the control group. We use these data to develop measurement error bounds on treatment effects estimated from surveys.
Link to Data Set
Digital Object Identifier
Associated URLs
Associated content
Other publications in this report series
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication Reducing Crime and Violence(World Bank, Washington, DC, 2016-04)The paper shows that self-control, time preferences, and values are malleable in adults, and that investments in these skills and preferences reduce crime and violence. The authors recruited criminally-engaged Liberian men and randomized half to eight weeks of group cognitive behavioral therapy, fostering self-regulation, patience, and noncriminal values. They also randomized $200 grants. Cash alone and therapy alone dramatically reduced crime and violence, but effects dissipated within a year. When cash followed therapy, however, crime and violence decreased by as much as 50 percent for at least a year. They hypothesize that cash reinforced therapy's lessons by prolonging practice and self-investment.Publication Assessing Workplace Social, Emotional, and Behavioral Skills in Africa(Washington, DC: World Bank, 2025-05-21)Social, emotional, and behavioral skills are critical for success across life domains, yet research is constrained by a lack of internationally validated measures for adult populations. Existing tools often assess isolated skills and are predominantly validated in Western, school- aged samples. To address these limitations, this study developed and validated the Effective Socio-emotional skills To gain Economic EMpowerment framework, comprising 14 distinct social, emotional, and behavioral skills with prior demonstrated relevance to economic outcomes. The framework’s self-report scales were tested among adults in six Sub-Saharan African countries (Côte d’Ivoire, Nigeria, the Republic of Congo, Rwanda, South Africa, and Tanzania), spanning diverse cultural and linguistic contexts (English, French, Hausa, Swahili, and Yoruba). The results confirm the psychometric validity of the scales, supporting their utility in both research and practice. The framework categorizes skills as intrapersonal or interpersonal, awareness or management, and agentic or communal, providing a robust tool to unpack which skills matter for employment and earnings and how this differs by gender. By enabling exploration of social, emotional, and behavioral skills in underrepresented and cross-cultural contexts, use of the Effective Socio-emotional skills To gain Economic EMpowerment self-report scales advances theoretical and practical understanding of social, emotional, and behavioral skills in adult populations.Publication Sex, Lies, and Surveys(World Bank, Washington, DC, 2019-02)This paper examines how easily observable interviewer characteristics, such as gender and physical attractiveness, and more difficult to observe characteristics, such as attitudes and beliefs, affect adolescent girls' disclosure of sexual behavior during a baseline survey for an adolescent girls program in Liberia. The study finds that girls are more likely to report sexual activity to better-looking interviewers, and less likely to do so to interviewers holding more discriminatory gender attitudes and greater expectations about the program. The study finds no evidence that the gender of the interviewer matters.Publication The Limits of Commitment(World Bank, Washington, DC, 2018-10)Working with a private bank in Ghana, this study examines the impacts of a commitment savings product designed to help clients taking repeated overdrafts break their debt cycles. Overall, the product significantly increased savings with the bank without increasing overdrafts. However, after accounting for other sources of savings, the study finds that clients with above-median baseline overdraft histories do not accrue new savings during the commitment period. Rather, they draw down other savings to offset the committed amount and take on new debt. In contrast, individuals with below-median overdraft histories significantly increase savings during and after the commitment period.Publication Missing(ness) in Action : Selectivity Bias in GPS-Based Land Area Measurements(World Bank, Washington, DC, 2013-06)Land area is a fundamental component of agricultural statistics, and of analyses undertaken by agricultural economists. While household surveys in developing countries have traditionally relied on farmers' own, potentially error-prone, land area assessments, the availability of affordable and reliable Global Positioning System (GPS) units has made GPS-based area measurement a practical alternative. Nonetheless, in an attempt to reduce costs, keep interview durations within reasonable limits, and avoid the difficulty of asking respondents to accompany interviewers to distant plots, survey implementing agencies typically require interviewers to record GPS-based area measurements only for plots within a given radius of dwelling locations. It is, therefore, common for as much as a third of the sample plots not to be measured, and research has not shed light on the possible selection bias in analyses relying on partial data due to gaps in GPS-based area measures. This paper explores the patterns of missingness in GPS-based plot areas, and investigates their implications for land productivity estimates and the inverse scale-land productivity relationship. Using Multiple Imputation (MI) to predict missing GPS-based plot areas in nationally-representative survey data from Uganda and Tanzania, the paper highlights the potential of MI in reliably simulating the missing data, and confirms the existence of an inverse scale-land productivity relationship, which is strengthened by using the complete, multiply-imputed dataset. The study demonstrates the usefulness of judiciously reconstructed GPS-based areas in alleviating concerns over potential measurement error in farmer-reported areas, and with regards to systematic bias in plot selection for GPS-based area measurement.
Users also downloaded
Showing related downloaded files
Publication Lebanon Economic Monitor, Fall 2022(Washington, DC, 2022-11)The economy continues to contract, albeit at a somewhat slower pace. Public finances improved in 2021, but only because spending collapsed faster than revenue generation. Testament to the continued atrophy of Lebanon’s economy, the Lebanese Pound continues to depreciate sharply. The sharp deterioration in the currency continues to drive surging inflation, in triple digits since July 2020, impacting the poor and vulnerable the most. An unprecedented institutional vacuum will likely further delay any agreement on crisis resolution and much needed reforms; this includes prior actions as part of the April 2022 International Monetary Fund (IMF) staff-level agreement (SLA). Divergent views among key stakeholders on how to distribute the financial losses remains the main bottleneck for reaching an agreement on a comprehensive reform agenda. Lebanon needs to urgently adopt a domestic, equitable, and comprehensive solution that is predicated on: (i) addressing upfront the balance sheet impairments, (ii) restoring liquidity, and (iii) adhering to sound global practices of bail-in solutions based on a hierarchy of creditors (starting with banks’ shareholders) that protects small depositors.Publication Classroom Assessment to Support Foundational Literacy(Washington, DC: World Bank, 2025-03-21)This document focuses primarily on how classroom assessment activities can measure students’ literacy skills as they progress along a learning trajectory towards reading fluently and with comprehension by the end of primary school grades. The document addresses considerations regarding the design and implementation of early grade reading classroom assessment, provides examples of assessment activities from a variety of countries and contexts, and discusses the importance of incorporating classroom assessment practices into teacher training and professional development opportunities for teachers. The structure of the document is as follows. The first section presents definitions and addresses basic questions on classroom assessment. Section 2 covers the intersection between assessment and early grade reading by discussing how learning assessment can measure early grade reading skills following the reading learning trajectory. Section 3 compares some of the most common early grade literacy assessment tools with respect to the early grade reading skills and developmental phases. Section 4 of the document addresses teacher training considerations in developing, scoring, and using early grade reading assessment. Additional issues in assessing reading skills in the classroom and using assessment results to improve teaching and learning are reviewed in section 5. Throughout the document, country cases are presented to demonstrate how assessment activities can be implemented in the classroom in different contexts.Publication Argentina Country Climate and Development Report(World Bank, Washington, DC, 2022-11)The Argentina Country Climate and Development Report (CCDR) explores opportunities and identifies trade-offs for aligning Argentina’s growth and poverty reduction policies with its commitments on, and its ability to withstand, climate change. It assesses how the country can: reduce its vulnerability to climate shocks through targeted public and private investments and adequation of social protection. The report also shows how Argentina can seize the benefits of a global decarbonization path to sustain a more robust economic growth through further development of Argentina’s potential for renewable energy, energy efficiency actions, the lithium value chain, as well as climate-smart agriculture (and land use) options. Given Argentina’s context, this CCDR focuses on win-win policies and investments, which have large co-benefits or can contribute to raising the country’s growth while helping to adapt the economy, also considering how human capital actions can accompany a just transition.Publication World Development Report 2006(Washington, DC, 2005)This year’s Word Development Report (WDR), the twenty-eighth, looks at the role of equity in the development process. It defines equity in terms of two basic principles. The first is equal opportunities: that a person’s chances in life should be determined by his or her talents and efforts, rather than by pre-determined circumstances such as race, gender, social or family background. The second principle is the avoidance of extreme deprivation in outcomes, particularly in health, education and consumption levels. This principle thus includes the objective of poverty reduction. The report’s main message is that, in the long run, the pursuit of equity and the pursuit of economic prosperity are complementary. In addition to detailed chapters exploring these and related issues, the Report contains selected data from the World Development Indicators 2005‹an appendix of economic and social data for over 200 countries. This Report offers practical insights for policymakers, executives, scholars, and all those with an interest in economic development.Publication Digital Africa(Washington, DC: World Bank, 2023-03-13)All African countries need better and more jobs for their growing populations. "Digital Africa: Technological Transformation for Jobs" shows that broader use of productivity-enhancing, digital technologies by enterprises and households is imperative to generate such jobs, including for lower-skilled people. At the same time, it can support not only countries’ short-term objective of postpandemic economic recovery but also their vision of economic transformation with more inclusive growth. These outcomes are not automatic, however. Mobile internet availability has increased throughout the continent in recent years, but Africa’s uptake gap is the highest in the world. Areas with at least 3G mobile internet service now cover 84 percent of Africa’s population, but only 22 percent uses such services. And the average African business lags in the use of smartphones and computers as well as more sophisticated digital technologies that catalyze further productivity gains. Two issues explain the usage gap: affordability of these new technologies and willingness to use them. For the 40 percent of Africans below the extreme poverty line, mobile data plans alone would cost one-third of their incomes—in addition to the price of access devices, apps, and electricity. Data plans for small- and medium-size businesses are also more expensive than in other regions. Moreover, shortcomings in the quality of internet services—and in the supply of attractive, skills-appropriate apps that promote entrepreneurship and raise earnings—dampen people’s willingness to use them. For those countries already using these technologies, the development payoffs are significant. New empirical studies for this report add to the rapidly growing evidence that mobile internet availability directly raises enterprise productivity, increases jobs, and reduces poverty throughout Africa. To realize these and other benefits more widely, Africa’s countries must implement complementary and mutually reinforcing policies to strengthen both consumers’ ability to pay and willingness to use digital technologies. These interventions must prioritize productive use to generate large numbers of inclusive jobs in a region poised to benefit from a massive, youthful workforce—one projected to become the world’s largest by the end of this century.