Global Poverty Monitoring Technical Note 40 How Improved Household Surveys Influence National and International Poverty Rates Daniel Gerszon Mahler, Elizabeth Foster, and Samuel Tetteh-Baah September 2024 Keywords: Poverty; survey design, comparability. Development Data Group Development Research Group Poverty and Equity Global Practice Group GLOBAL POVERTY MONITORING TECHNICAL NOTE 40 Abstract To effectively address poverty, it is essential that countries have the tools and means to accurately measure people’s living standards. Most countries rely on data collected from household surveys to measure monetary poverty, defining households as poor when their consumption is below the national poverty line. When countries improve the quality and scope of their household surveys, as some have done in recent years, they often capture consumption that is overlooked in previous surveys, thus leading to higher measured consumption. With better data, countries redefine their national poverty line, which on average balances out the higher consumption, leading to minimal change in national poverty rates. However, the international poverty line is fixed at any given point in time. Thus, when measured consumption increases, international poverty rates fall, sometimes dramatically. For this reason, international poverty rate comparisons over time should be done with caution when countries implement improved household surveys. All authors are with the World Bank. Corresponding author: Daniel Gerszon Mahler (dmahler@worldbank.org). The authors are grateful for comments from Barbara Balaj, Carolina Diaz- Bonilla, Christoph Lakner, Francis Mulangu, Maria Eugenia Genoni, Paul Anthony Clare, Rose Mungai, and Salman Zaidi. The authors gratefully acknowledge financial support from the UK Government through the Data and Evidence for Tackling Extreme Poverty (DEEP) Research Program. The Global Poverty Monitoring Technical Note Series publishes short papers that document methodological aspects of the World Bank’s global poverty estimates. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Global Poverty Monitoring Technical Notes are available at https://pip.worldbank.org/publication. Most countries rely on data collected from household surveys to measure poverty. In household surveys, a representative sample of households of a country are asked a range of questions covering different topics. In low- and middle-income countries, where the value of consumption or consumption expenditure is typically used to inform households’ poverty status, many questions relate to the consumption and spending of households. The questionnaire design of household surveys greatly influences the data collected. Numerous studies have shown that the design of household surveys impacts measured consumption (Beegle and others 2012; De Weerdt and others 2020). For example, when measuring food consumption, it matters whether households are asked to keep a diary of their consumption or to recall their consumption. In the latter case, it is of consequence whether they are asked to recall their consumption over the past seven days or 30 days. For both food and non-food consumption, it matters how disaggregated the categories are. Also of importance are whether the questionnaire allows for the computation of food consumed outside of the home (known as food away from home); the value of goods that deliver services repeatedly through an extended period over time, such as cars and household appliances (known as durables); and whether the spending on housing is included for renters and can be imputed for homeowners (known as imputed rent). Improved household surveys often lead to large increases in measured consumption. Innovations in surveys often mean that more consumption is captured due to previously overlooked food consumption, as well as the inclusion of durables and/or imputed rent. We have identified 12 countries with large improvements in the quality of their household surveys (Table 1). The ways in which the surveys improved differ, and some of the improvements may not relate to the questionnaire, such as a switch from pen-and-paper interviewing (PAPI) to computer-assisted personal interviewing (CAPI) (Caeyers and others 2012). On average across these 12 countries, mean consumption increased by 46 percent in real terms from the older survey to the newer survey (Figure 1). Part of this phenomenon could be explained by general progress occurring between the two survey rounds and other changes, such as whether price differences within countries is only accounted for in one of the rounds. However, the vast majority can be explained by the inclusion of previously overlooked consumption. When countries implement new surveys, they often revise their national poverty line upwards. Typically, national poverty lines in low- and middle-income countries are based on the cost of obtaining a set amount of calories per day — for example, 2200 calories per day — plus the non-food consumption of households close to this threshold. For the national poverty line to reflect the newly measured consumption, the line is often revised when the survey design changes. The effect of adding more food items and/or changing the recall period on the national poverty line is ambiguous, depending on whether these modifications increase the measured cost per calorie. Adding non-food items, however, mechanically increases the national poverty line (Ravallion 1994). We have data on national poverty lines in ten of the 12 countries, lacking data for Mongolia and Senegal. Across the ten countries, the national poverty line increased by an average of 50 percent following the implementation of improved household surveys (Figure 2). 1 Table 1: Cases Identified with Large Figure 1: Impact of Improved Surveys on Improvements in Household Surveys Mean Consumption Country First survey Second survey Bangladesh 2016 HIES 2022 HIES Benin 2015 EMICOV 2018-19 EHCVM Bhutan 2017 BLSS 2022 BLSS Burkina Faso 2014 EMC 2018-19 EHCVM China 2012 CNIHS 2013 CNIHS Cote d’Ivoire 2015 ENV 2018-19 EHCVM Guinea 2012 ELEP 2018 EHCVM Guinea-Bissau 2010 ILAP 2018-19 EHCVM Mali 2009-10 ELIM 2018-19 EHCVM Mongolia 2018 HSES 2022 HSES Senegal 2011-12 ESPS 2018-19 EHCVM Togo 2015 QUIBB 2018-19 EHCVM Figure 2: Impact of Improved Surveys on Figure 3: Impact of Improved Surveys on National Poverty Lines National Poverty Rates Source: World Bank’s Poverty and Inequality Platform and World Development Indicators. Note: BEN = Benin; BFA = Burkina Faso; BGD = Bangladesh; CHN = China; CIV = Côte d'Ivoire; GIN = Guinea; GNB = Guinea-Bissau; MLI =Mali; MNG = Mongolia; SEN = Senegal; TGO = Togo. In Figure 1, mean consumption is expressed in real terms, which means that general inflation is not influencing the growth in consumption observed. In Figure 2, the poverty lines are converted to 2017 purchasing power parity (PPP) adjusted U.S. dollars. 2 The higher national poverty line often offsets the increased measured consumption, leading to minimal impact on national poverty rates. Consequently, even with more accurate consumption data, the proportion of people considered poor according to the new national poverty line may not decrease. In fact, across the 10 countries studied, implementing high-quality surveys did not lead to a systematic change in national poverty rates (Figure 3). The offsetting increase in national poverty lines and measured consumption is not innocuous: With these two simultaneous changes, it can be difficult to decompose what is driven by changes to the definition of poverty and what is driven by changes to real welfare. The ideal scenario is to have two comparable welfare aggregates with a fixed national poverty line. The international poverty line (IPL) is fixed at a given point in time; thus, when new surveys reveal large increases to measured consumption, international poverty rates often drop drastically. Currently set at US$2.15 per day in 2017 purchasing power parity (PPP) adjusted prices, the IPL is defined as the median national poverty line of low-income countries around 2017 (Joliffe and others, forthcoming). This line is applied consistently across countries and over time, even if new household surveys report higher consumption. Across the 12 countries, there are cases where the international poverty rate of the old survey is more than twice that of the new survey. In Guinea-Bissau, the rate decreased from 67 to 22 percent with the newer survey (Figure 4). In China, the number of people below the IPL fell by 31 million people in 2013 due to the inclusion of imputed rent and other enhancements (World Bank 2016). International poverty rate comparisons over time should be done with caution when countries implement improved household surveys because it often overstates declines in poverty. This is not related to the specific value of the IPL. When new rounds of international purchasing power comparisons are completed, the IPL is often updated to account for these new data. At the same time, the latest sample of national poverty lines that form the backbone of the IPL are used. If countries have changed the real value of their national poverty line in the meantime (perhaps in response to more measured consumption), the real value of the IPL would likely change as well. Yet, even if the line would be higher with a new PPP round, it would still exaggerate the decline in poverty because it remains fixed over time for temporal analyses. The challenge lies with any poverty line that remains fixed over time while the household survey design changes. For countries that have recently implemented new household surveys, the IPL in 2017 PPPs may not accurately reflect poverty prevalence as defined by the typical low-income country. The IPL set at the US$2.15 line was derived when most recently improved household surveys were not available. Hence, the IPL largely reflects the national poverty line of the typical low-income country before this improvement. If re-estimated with the latest sample of national poverty lines, the IPL would increase to US$2.53. Including only countries with a household survey conducted in or after 2017, the IPL would rise further to US$2.73 (Figure 5). The next time the IPL is revised, it may increase in real value and imply that the line broadly reflects the typical national poverty line used in low-income countries with high quality household surveys. Yet, this new line would be used for both older and newer surveys, meaning that the decline in international poverty rates arising from improved household surveys would persist. 3 Figure 4: Impact of Improved Surveys on Figure 5: Impact of Improved Surveys on the International Poverty Rates International Poverty Line Source: World Bank’s Poverty and Inequality Platform and World Development Indicators. Note: BEN = Benin; BFA = Burkina Faso; BGD = Bangladesh; CHN = China; CIV = Côte d'Ivoire; GIN = Guinea; GNB = Guinea-Bissau; MLI = Mali; MNG = Mongolia; SEN = Senegal; TGO = Togo. IPL= International Poverty Line; NPL = National Poverty Line; PPP = Purchasing Power Parity. Cross-country comparisons of international poverty rates should be done with caution if countries measure consumption very differently. It is more useful to compare countries that have implemented high-quality household surveys with one another, and separately compare countries using older-standard surveys. This ensures more accurate and meaningful comparisons by accounting for variations in data collection methods. This issue also applies to time-invariant differences in how countries measure poverty. One country may use income to measure poverty while another country may use consumption. Even the countries do not change their survey instrument, poverty rate comparisons between them should be done with caution. It is vital that countries implement high-quality household surveys to improve the quality and comparability of poverty estimates across countries. As long as some countries have high- quality household surveys and others do not, the IPL can be misleading and cross-country comparisons can be challenging. Thus, it is imperative that all countries possess the technical, financial, and political capital necessary to measure the living standards of their citizens using state-of-the-art methods. Countries can mitigate the impact of changes to survey design on temporal poverty rate comparisons using various strategies. When estimating international poverty rates, countries face a trade-off between accuracy, i.e. using the best methods to measure consumption, and temporal comparability (Jolliffe and others 2023). New surveys are needed to improve accuracy, however changes to survey design should be avoided if the gains in accuracy are outweighed by 4 the lack of ability to create a comparable time trend. This trade-off can be weakened by ex-ante thinking. If the design of a forthcoming survey will change, countries can ensure a smaller sample follows the old questionnaire. Alternatively, countries can use imputation models to predict poverty in one of the two surveys by relying on indicators that are comparable in both surveys. They can then use this model to predict poverty in the other survey (Yoshida and others 2022). Both options will allow for a comparable trend to be constructed. Cross-country modelling can mitigate the impact of changes to survey design on cross- country comparisons. The mitigation strategies that exist at the country level are difficult to implement systematically across countries. For example, none of the mitigation strategies above can make poverty estimates based on consumption comparable to poverty estimates based on income. When comparing poverty rates across countries, a trade-off exists between staying close to countries’ own estimates of international poverty rates vis-à-vis ensuring cross-country comparability. Some producers of cross-country databases prefer the former, and hence have estimates consistent with how countries themselves estimate the variable of interest. Others use predictions from econometric or machine-learning models to ensure comparability. For international poverty rates, an intermediate strategy could be to remove the most troublesome incomparabilities through survey-to-survey imputation or use scaling factors that remove large differences between selected welfare aggregates. Other key social and economic statistics undergo similar revisions due to quality improvements. When Nigeria rebased its gross domestic product (GDP) in 2014, the size of its economy increased by 89 percent, equivalent to US$510 billion (Economist 2014), thereby making it the largest economy in Sub-Saharan Africa. Revisions of this magnitude also occur in high- income countries. For example, Saudi Arabia’s 2022 census revised population estimates down by 8 percent, equivalent to about 3 million fewer people (Callen 2023). In contrast to improvements to measured poverty, revisions to population and GDP numbers are often done retroactively by rebasing the entire series to a consistent baseline, thereby masking large changes. In summary, the design of household surveys is critical to accurately measuring consumption and assessing poverty. The choice of recall periods, the level of disaggregation in consumption categories, and the inclusion of durables and imputed rent are pivotal factors that influence the data collected. Enhanced survey methodologies have led to significant increases in measured consumption, which is highly desirable if those changes reflect more accurate and updated consumption patterns. However, these improvements have implications for analyzing trends in poverty rates. Usually, national rates remain relatively unchanged due to the recalibration of poverty thresholds. Yet, international poverty rates can show marked declines due to the fixed nature of the International Poverty Line. This underscores the importance of high-quality, comparable household surveys for accurate poverty estimation. Ensuring all countries have the means to conduct high-quality surveys is essential for reliable and meaningful poverty metrics and comparisons across nations. 5 References Beegle, Kathleen, Joachim De Weerdt, Jed Friedman, and John Gibson. 2012. "Methods of household consumption measurement through surveys: Experimental results from Tanzania." Journal of Development Economics 98(1):3–18. Callen, Tim. 2023. “A Smaller Saudi Population Puts Key Economic Indicators in a More Favorable Light�. The Arab Gulf States Institute, Washington, DC. July 24. Caeyers, Bet, Neil Chalmers, and Joachim De Weerdt. 2012. "Improving consumption measurement and other survey data through CAPI: Evidence from a randomized experiment." Journal of Development Economics 98(1): 19-33. De Weerdt, Joachim, John Gibson, and Kathleen Beegle. 2020. "What can we learn from experimenting with survey methods?" Annual Review of Resource Economics 12: 431–447. Economist. 2024. “Africa’s New Number One�. April 12. Jolliffe, Dean ⓡ Daniel Gerszon Mahler ⓡ Malarvizhi Veerappan ⓡ Talip Kilic ⓡ Philip Wollburg. 2023. "What Makes Public Sector Data Valuable for Development?" World Bank Research Observer 38 (2): 325–346. Jolliffe, Dean ⓡ Daniel Gerszon Mahler ⓡ Christoph Lakner ⓡ Aziz Atamanov ⓡ Samuel Kofi Tetteh Baah. Forthcoming. "Assessing the Impact of the 2017 PPPs on the International Poverty Line and Global Poverty." World Bank Economic Review. Ravallion, M. 1994. “Poverty Comparisons.� Chur, Switzerland: Harwood Academic Press. World Bank. 2016. “Poverty and Shared Prosperity 2016: Taking on Inequality.� Washington, D.C.: World Bank. Yoshida, Nobuo, Shinya Takamatsu, Kazusa Yoshimura, Danielle Victoria Aron, Xiaomeng Chen, Silvia Malgioglio, Shivapragasam Shivakumaran, and Kexin Zhang. 2022. “The Concept and Empirical Evidence of SWIFT Methodology.� Equitable Growth, Finance, and Institutions Insights, World Bank. 6