Publication: Using Paradata to Assess Respondent Burden and Interviewer Effects in Household Surveys: Evidence from Low- and Middle-Income Countries
Loading...
Published
2023-06-15
ISSN
Date
2023-06-15
Author(s)
Editor(s)
Abstract
Over the past decade, national statistical offices in low- and middle-income countries have increasingly transitioned to computer-assisted personal interviewing and computer-assisted telephone interviewing for the implementation of household surveys. The byproducts of these types of data collection are survey paradata, which can unlock objective, module- and question-specific, actionable insights on respondent burden, survey costs, and interviewer effects. This study does precisely that, using paradata generated by the Survey Solutions computer-assisted personal interviewing platform in recent national household surveys implemented by the national statistical offices in Cambodia, Ethiopia, and Tanzania. Across countries, the average household interview, based on a socioeconomic household questionnaire, ranges from 82 to 120 minutes, while the average interview with an adult household member, based on a multi-topic individual questionnaire, takes between 13 to 25 minutes. Using a multilevel model that is estimated for each household and individual questionnaire module, the paper shows that interviewer effects on module duration are significantly larger than the estimates from high-income contexts. Food consumption, household roster, and non-farm enterprises consistently emerge among the top five household questionnaire modules in terms of total variance in duration, with 5 to 50 percent of the variability being attributable to interviewers. Similarly, labor, health, and land ownership appear among the top five individual questionnaire modules in terms of total variance in duration, with 6 to 50 percent of the variability being attributable to interviewers. These findings, particularly by module, point to where additional interviewer training, fieldwork supervision, and data quality monitoring may be needed in future surveys.
Link to Data Set
Citation
“Hasanbasri, Ardina; Kilic, Talip; Koolwal, Gayatri; Moylan, Heather. 2023. Using Paradata to Assess Respondent Burden and Interviewer Effects in Household Surveys: Evidence from Low- and Middle-Income Countries. Policy Research Working Papers; 10456. © World Bank. http://hdl.handle.net/10986/39878 License: CC BY 3.0 IGO.”
Associated URLs
Associated content
Other publications in this report series
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication LSMS+ Program(World Bank, Washington, DC, 2021-05-05)Established in 2016, the World Bank living standards measurement study - plus (LSMS+) program works to enhance the availability and quality of intra-household, self-reported, individual-disaggregated survey data collected in low- and middle-income countries on key dimensions of men’s and women’s economic opportunities and welfare. This report presents an overview of the LSMS+ program and provides operational guidance regarding individual-disaggregated data collection in large-scale household surveys, based on the experience with and analysis of the national surveys that have been implemented by the respective national statistical offices (NSOs) in Cambodia, Ethiopia, Malawi, Tanzania over the period 2016-2020, with support from the LSMS+ program.Publication LSMS+ Program in Sub-Saharan Africa(World Bank, Washington, DC, 2021-05-05)Established in 2016, the World Bank living standards measurement study - plus (LSMS+) program works to enhance the availability and quality of intra-household, self-reported, individual-disaggregated survey data collected in low- and middle-income countries on key dimensions of men’s and women’s economic opportunities and welfare. This report presents findings on gender differences in labor market outcomes and ownership of physical and financial assets in Sub-Saharan Africa, based on the national surveys that have been implemented by the respective national statistical offices (NSOs) in Ethiopia, Malawi, and Tanzania over the period 2016-2020, with support from the LSMS+ program.Publication Individual Wealth and Time Use(World Bank, Washington, DC, 2021-08)A better understanding of how individual wealth and time use are linked—across paid, unpaid, and leisure activities —is important for targeting widespread gender inequalities in time allocation, as well as in accessing economic opportunities. The lack of reliable, individual-level data on asset ownership across different subpopulations, however, has limited discussions of these issues in the literature. Using a unique nationally representative survey from Cambodia, this paper shows that individual wealth, as measured through self-reported ownership of physical and financial assets, is significantly associated with time allocation to different activities. The role of asset ownership in time use is also stronger, particularly among women, vis-à-vis the competing proxies for socioeconomic status. Ownership of financial accounts, motorized vehicles, and mobile phones —all of which can improve access to networks, markets, and services—is associated with less time in unpaid work, and in some cases greater time in paid work, specifically among women in off-farm jobs. There are also distinct gender differences in how men and women shift their time away from leisure and childcare, highlighting the importance of social norms in choices over time use. The analysis highlights the utility of integrated, intra-household, individual-disaggregated data collection on asset ownership, time use, and employment in lower-income contexts.Publication Multidimensionality of Land Ownership among Men and Women in Sub-Saharan Africa(World Bank, Washington, DC, 2022-01)Across Sub-Saharan African countries with customary tenure systems and low levels of documented land ownership, there are limited nationally representative insights on men and women landowners’ rights over land. Variations in institutions and norms governing land ownership further complicate cross-country comparisons. Using machine learning techniques and nationally representative, intrahousehold survey data elicited in private from men and women on their ownership of assets, this paper creates unique profiles of landowners in Ethiopia, Malawi, and Tanzania, anchored in a range of constructs related to self-reported rights and control over land parcels. The analysis reveals a high degree of cross-country consistency in the new insights. Landowners, particularly women, often do not have full rights and decision-making power over land. Multiple correspondence analysis demonstrates that transfer rights (rights to bequeath, sell, rent out, and use as collateral) contribute the most to the variation in the composition of the constructs related to rights and control over land. Hierarchical clustering shows that landowners can effectively be clustered into three categories: (1) owners with mostly exclusive transfer rights, (2) owners with mostly joint transfer rights, and (3) owners with no/limited transfer rights. Owners with transfer rights tend to have all other rights and measures of control. Women are overrepresented in the cluster of landowners with no/limited transfer rights, and in moving from the cluster with mostly joint transfer rights to the one with mostly exclusive transfer rights, the increase in the share of individuals not needing permission to exercise any right is considerably greater among women than men.Publication Are You Being Asked? Impacts of Respondent Selection on Measuring Employment(World Bank, Washington, DC, 2020-02)Accurate estimates of men's and women's employment are at the heart of understanding sources of productivity and economic growth and designing well-targeted, gender-sensitive labor policies. How respondent selection in household and labor force surveys affects these estimates is a key question, for which experimental evidence outside high-income settings is limited. Leveraging two concurrent, national surveys in Malawi that differed in their approach to respondent selection, the analysis shows that, compared with the best practice of privately interviewing adults about their employment outcomes, the common "business-as-usual" approach that permits the use of proxy respondents and non-private/group interviews leads to significant underreporting of employment across a range of wage and self-employment activities, with stronger effects for women and for a longer (12-month) recall period. Under the business-as-usual approach, the main factors linked to underreporting include household wealth, proxy reporting, and potential difficulties associated with interpreting/answering questions on household non-farm enterprises.
Users also downloaded
Showing related downloaded files
Publication Sourcebook on the Foundations of Social Protection Delivery Systems(Washington, DC: World Bank, 2020-07-30)The Sourcebook synthesizes real-world experiences and lessons learned of social protection delivery systems from around the world, with a particular focus on social and labor benefits and services. It takes a practical approach, seeking to address concrete “how-to” questions, including: How do countries deliver social protection benefits and services? How do they do so effectively and efficiently? How do they ensure dynamic inclusion, especially for the most vulnerable and needy? How do they promote better coordination and integration—not only among social protection programs but also programs in other parts of government? How can they meet the needs of their intended populations and provide a better client experience? The Sourcebook structures itself around eight key principles that can frame the delivery systems mindset: (1) delivery systems evolve over time, do so in a non-linear fashion, and are affected by the starting point(s); (2) additional efforts should be made to “do simple well”, and to do so from the start rather than trying to remedy by after-the-fact adding-on of features or aspects; (3) quality implementation matters, and weaknesses in the design or structure of any core system element will negatively impact delivery; (4) defining the “first mile” for people interface greatly affects the system and overall delivery, and is most improved when that “first mile” is understood as the weakest link in delivery systems); (5) delivery systems do not operate in a vacuum and thus should not be developed in silos; (6) delivery systems can contribute more broadly to government’s ability to intervene in other sectors, such as health insurance subsidies, scholarships, social energy tariffs, housing benefits, and legal services; (7) there is no single blueprint for delivery systems, but there are commonalities and those common elements constitute the core of the delivery systems framework; (8) inclusion and coordination are pervasive and perennial dual challenges, and they contribute to the objectives of effectiveness and efficiency.Publication Digital Africa(Washington, DC: World Bank, 2023-03-13)All African countries need better and more jobs for their growing populations. "Digital Africa: Technological Transformation for Jobs" shows that broader use of productivity-enhancing, digital technologies by enterprises and households is imperative to generate such jobs, including for lower-skilled people. At the same time, it can support not only countries’ short-term objective of postpandemic economic recovery but also their vision of economic transformation with more inclusive growth. These outcomes are not automatic, however. Mobile internet availability has increased throughout the continent in recent years, but Africa’s uptake gap is the highest in the world. Areas with at least 3G mobile internet service now cover 84 percent of Africa’s population, but only 22 percent uses such services. And the average African business lags in the use of smartphones and computers as well as more sophisticated digital technologies that catalyze further productivity gains. Two issues explain the usage gap: affordability of these new technologies and willingness to use them. For the 40 percent of Africans below the extreme poverty line, mobile data plans alone would cost one-third of their incomes—in addition to the price of access devices, apps, and electricity. Data plans for small- and medium-size businesses are also more expensive than in other regions. Moreover, shortcomings in the quality of internet services—and in the supply of attractive, skills-appropriate apps that promote entrepreneurship and raise earnings—dampen people’s willingness to use them. For those countries already using these technologies, the development payoffs are significant. New empirical studies for this report add to the rapidly growing evidence that mobile internet availability directly raises enterprise productivity, increases jobs, and reduces poverty throughout Africa. To realize these and other benefits more widely, Africa’s countries must implement complementary and mutually reinforcing policies to strengthen both consumers’ ability to pay and willingness to use digital technologies. These interventions must prioritize productive use to generate large numbers of inclusive jobs in a region poised to benefit from a massive, youthful workforce—one projected to become the world’s largest by the end of this century.Publication Dominican Republic Poverty Assessment 2023(Washington, DC: World Bank, 2023-11-08)In recent decades, economic growth in the Dominican Republic (DR) has been steady. However, growth has not occurred in such a way as to make the benefits widely and evenly available. In fact, although the DR economy grew faster than that of other LAC countries before the Covid-19 pandemic, its poverty rates and social outcomes remain broadly similar to them. This report seeks to explain this conundrum, as well as to expand the knowledge base to improve the effectiveness of ongoing poverty reduction policies in the DR. The Poverty Assessment draws primarily on new analytical work conducted in the DR, structured around four background notes on: (i) trends in monetary poverty and inequality, as well as the key drivers of those changes; (ii) nonmonetary poverty and its spatial dimensions; (iii) social assistance programs and their role in mitigating poverty; and (iv) climate change and its interaction with poverty. By helping to reduce the evidence gap in each of these areas, our analysis hopes to inform government policies and the national dialogue on poverty reduction. In addition, the note integrates existing analytical work and evidence produced inside and outside the Bank, including from its operations in the country.Publication Classroom Assessment to Support Foundational Literacy(Washington, DC: World Bank, 2025-03-21)This document focuses primarily on how classroom assessment activities can measure students’ literacy skills as they progress along a learning trajectory towards reading fluently and with comprehension by the end of primary school grades. The document addresses considerations regarding the design and implementation of early grade reading classroom assessment, provides examples of assessment activities from a variety of countries and contexts, and discusses the importance of incorporating classroom assessment practices into teacher training and professional development opportunities for teachers. The structure of the document is as follows. The first section presents definitions and addresses basic questions on classroom assessment. Section 2 covers the intersection between assessment and early grade reading by discussing how learning assessment can measure early grade reading skills following the reading learning trajectory. Section 3 compares some of the most common early grade literacy assessment tools with respect to the early grade reading skills and developmental phases. Section 4 of the document addresses teacher training considerations in developing, scoring, and using early grade reading assessment. Additional issues in assessing reading skills in the classroom and using assessment results to improve teaching and learning are reviewed in section 5. Throughout the document, country cases are presented to demonstrate how assessment activities can be implemented in the classroom in different contexts.Publication Business Ready 2024(Washington, DC: World Bank, 2024-10-03)Business Ready (B-READY) is a new World Bank Group corporate flagship report that evaluates the business and investment climate worldwide. It replaces and improves upon the Doing Business project. B-READY provides a comprehensive data set and description of the factors that strengthen the private sector, not only by advancing the interests of individual firms but also by elevating the interests of workers, consumers, potential new enterprises, and the natural environment. This 2024 report introduces a new analytical framework that benchmarks economies based on three pillars: Regulatory Framework, Public Services, and Operational Efficiency. The analysis centers on 10 topics essential for private sector development that correspond to various stages of the life cycle of a firm. The report also offers insights into three cross-cutting themes that are relevant for modern economies: digital adoption, environmental sustainability, and gender. B-READY draws on a robust data collection process that includes specially tailored expert questionnaires and firm-level surveys. The 2024 report, which covers 50 economies, serves as the first in a series that will expand in geographical coverage and refine its methodology over time, supporting reform advocacy, policy guidance, and further analysis and research.