Publication: Machine Learning Imputation of High Frequency Price Surveys in Papua New Guinea
Loading...
Date
2023-09-28
ISSN
Published
2023-09-28
Author(s)
Editor(s)
Abstract
Capabilities to track fast-moving economic developments re-main limited in many regions of the developing world. This complicates prioritizing policies aimed at supporting vulnerable populations. To gain insight into the evolution of fluid events in a data scarce context, this paper explores the ability of recent machine-learning advances to produce continuous data in near-real-time by imputing multiple entries in ongoing surveys. The paper attempts to track inflation in fresh produce prices at the local market level in Papua New Guinea, relying only on incomplete and intermittent survey data. This application is made challenging by high intra-month price volatility, low cross-market price correlations, and weak price trends. The modeling approach uses chained equations to produce an ensemble prediction for multiple price quotes simultaneously. The paper runs cross-validation of the prediction strategy under different designs in terms of markets, foods, and time periods covered. The results show that when the survey is well-designed, imputations can achieve accuracy that is attractive when compared to costly–and logistically often infeasible–direct measurement. The methods have wider applicability and could help to fill crucial data gaps in data scarce regions such as the Pacific Islands, especially in conjunction with specifically designed continuous surveys.
Link to Data Set
Citation
“Andrée, Bo Pieter Johannes; Pape, Utz Johann; Andree, Bo, Pieter Johannes. 2023. Machine Learning Imputation of High Frequency Price Surveys in Papua New Guinea. Policy Research Working Papers; 10559. © World Bank. http://hdl.handle.net/10986/40410 License: CC BY 3.0 IGO.”
Associated URLs
Associated content
Other publications in this report series
Publication Disentangling the Key Economic Channels through Which Infrastructure Affects Jobs(Washington, DC: World Bank, 2025-04-03)This paper takes stock of the literature on infrastructure and jobs published since the early 2000s, using a conceptual framework to identify the key channels through which different types of infrastructure impact jobs. Where relevant, it highlights the different approaches and findings in the cases of energy, digital, and transport infrastructure. Overall, the literature review provides strong evidence of infrastructure’s positive impact on employment, particularly for women. In the case of electricity, this impact arises from freeing time that would otherwise be spent on household tasks. Similarly, digital infrastructure, particularly mobile phone coverage, has demonstrated positive labor market effects, often driven by private sector investments rather than large public expenditures, which are typically required for other large-scale infrastructure projects. The evidence on structural transformation is also positive, with some notable exceptions, such as studies that find no significant impact on structural transformation in rural India in the cases of electricity and roads. Even with better market connections, remote areas may continue to lack economic opportunities, due to the absence of agglomeration economies and complementary inputs such as human capital. Accordingly, reducing transport costs alone may not be sufficient to drive economic transformation in rural areas. The spatial dimension of transformation is particularly relevant for transport, both internationally—by enhancing trade integration—and within countries, where economic development tends to drive firms and jobs toward urban centers, benefitting from economies scale and network effects. Turning to organizational transformation, evidence on skill bias in developing countries is more mixed than in developed countries and may vary considerably by context. Further research, especially on the possible reasons explaining the differences between developed and developing economies, is needed.Publication Economic Consequences of Trade and Global Value Chain Integration(World Bank, Washington, DC, 2025-04-04)This paper introduces a new approach to measuring Global Value Chains (GVC), crucial for informed policy-making. It features a tripartite classification (backward, forward, and two-sided) covering trade and production data. The findings indicate that traditional trade-based GVC metrics significantly underestimate global GVC activity, especially in sectors like services and upstream manufacturing, and overstate risks in early trade liberalization stages. Additionally, conventional backward-forward classifications over-estimate backward linkages. The paper further applies these measures empirically to assess how GVC participation mediates the impact of demand shocks on domestic output, highlighting both the exposure and stabilizing potential of GVC integration. These new measures are comprehensively available on the World Bank’s WITS Platform, providing a key resource for GVC analysis.Publication Participation in Pension Programs in Low- and Middle-Income Countries(Washington, DC: World Bank, 2025-04-24)Low- and middle-income countries are aging rapidly but stagnation of growth in participation in pension programs, due to widespread informal employment, presents a major fiscal challenge. Some claim that improving the design of pension program rules can encourage more pension contributions, while others push for universal non-contributory pensions. This paper reviews the recent academic literature on the determinants of active participation in pension systems in high- informality settings. An emerging body of evidence shows that participation responds significantly to financial incentives as well as nonfinancial obstacles. At the same time, pensions are imperfect substitutes for other strategies to cover longevity risk, including support through the family, which will remain crucial for many older people in fiscally constrained environments. Therefore, policy makers should integrate the design of contributory pensions, social pensions, and policies that facilitate other forms of elderly support and consider how all three interact. To inform such efforts, these interactions must be more systematically investigated, and the empirical evidence must be expanded beyond a small number of middle-income countries.Publication Capitalizing on Digital Transformation to Enhance the Effectiveness of Property Institutions(Washington, DC: World Bank, 2025-04-14)Property registries have long been a pillar of state capacity and a basis for private market activity. While registry establishment and operation traditionally were costly and time consuming, digital technology makes low-cost registry operation and wide outreach easier. To guide developing countries aiming to establish such registries and measure progress, this paper develops indicators (in terms of digital coverage, interoperability, and property taxation for local service delivery and public land management) of effective digital registry service provision. Data from 85 countries highlight vast differences and provide suggestions for strategic reforms as well as a basis for measuring progress over time. Expanding geographical coverage and collecting these indicators on a regular basis could provide guidance to improve the way in which, by protecting property, the state creates the basis for widely shared prosperity and a livable environment.Publication Bridging the Gap(Washington, DC: World Bank, 2025-04-21)This paper examines tax revenue shortfalls in South Asian countries. On average during 2019–23, South Asian revenues totaled 18 percent of GDP—well below the average 24 percent among emerging market and developing economies (EMDEs). Econometric estimates from stochastic frontier analysis, which control for tax rates and the size of potential tax bases, suggest that tax revenues in the region are 1 to 7 percentage points of GDP below potential, with shortfalls in five of the region’s eight countries larger than in the average EMDE. Even after controlling for country characteristics, such as widespread informal economic activity outside the tax net and large agriculture sectors, sizable tax gaps remain, suggesting the need for improved tax policy and administration. The paper discusses and provides evidence from international experience with reforms to raise government revenues.
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication Machine Learning Guided Outlook of Global Food Insecurity Consistent with Macroeconomic Forecasts(World Bank, Washington, DC, 2022-10)Motivated by the deterioration in global food security conditions, this paper develops a parsimonious machine learning model to derive a multi-year outlook of global severe food insecurity from macro-economic projections. The objective is to provide forecasts that are internally consistent with wider economic assessments, allowing both food security policies and economic development policies to be informed by a cohesive set of expectations. The model is validated on holdout data that explicitly test the ability to forecast new data from history and extrapolate beyond observed intervals. It is then applied to the World Economic Outlook database of April 2022 to project the severely food insecure population across all 144 World Bank lending countries. The analysis estimates that the global severely food insecure population may remain above 1 billion through 2027 unless large-scale interventions are made. The paper also explores counterfactual scenarios, first to investigate additional risks in a downside economic scenario, and second, to investigate whether restoring macroeconomic targets is sufficient to revert food insecurity back to pre-pandemic levels. The paper concludes that the proposed model provides a robust and low-cost approach to maintain reliable long-term projections and produce scenario analyses that can be revised systematically and interpreted within the context of available economic outlooks.Publication Estimating Food Price Inflation from Partial Surveys(World Bank, Washington, DC, 2021-12)The traditional consumer price index is often produced at an aggregate level, using data from few, highly urbanized, areas. As such, it poorly describes price trends in rural or poverty-stricken areas, where large populations may reside in fragile situations. Traditional price data collection also follows a deliberate sampling and measurement process that is not well suited for monitoring during crisis situations, when price stability may deteriorate rapidly. To gain real-time insights beyond what can be formally measured by traditional methods, this paper develops a machine-learning approach for imputation of ongoing subnational price surveys. The aim is to monitor inflation at the market level, relying only on incomplete and intermittent survey data. The capabilities are highlighted using World Food Programme surveys in 25 fragile and conflict-affected countries where real-time monthly food price data are not publicly available from official sources. The results are made available as a data set that covers more than 1200 markets and 43 food types. The local statistics provide a new granular view on important inflation events, including the World Food Price Crisis of 2007–08 and the surge in global inflation following the 2020 pandemic. The paper finds that imputations often achieve accuracy similar to direct measurement of prices. The estimates may provide new opportunities to investigate local price dynamics in markets where prices are sensitive to localized shocks and traditional data are not available.Publication Climate Shocks and Their Effects on Food Security, Prices, and Agricultural Wages in Afghanistan(Washington, DC: World Bank, 2024-12-17)This study examines the effects of climate and weather shocks on Afghanistan's agricultural economy, with an emphasis on food security, prices, and wages. By utilizing a dynamical model and a unique data set that includes monthly global and local food prices, agricultural wages, unofficial exchange rates, and local climate data, the research provides econometric estimates of the impacts of droughts and floods. The findings reveal that both flooding and drought significantly increase food insecurity, directly and indirectly. Directly, these climatic shocks are linked to heightened risks of food insecurity in the following months, even when controlling for price and wage fluctuations. Indirectly, droughts and floods drive up food prices and depress agricultural wages, further exacerbating food insecurity. The study suggests that enhancing climate resilience in the agriculture sector could mitigate these risks, stabilize local food prices and wages, and strengthen food security and the broader agricultural economy. The results also show that price data effectively capture food security shocks from various non-economic sources, and can serve as a versatile monitoring tool in situations where detailed data on food security are unavailable.Publication 2023 Food security monitoring in Papua New Guinea - Insights from high frequency phone surveys(Washington, DC: World Bank, 2024-05-06)The objective of the Pacific Observatory is to improve welfare for the poor and vulnerable in Papua New Guinea and the Pacific Island Countries through expanding socio-economic information for better data-driven policymaking.Publication Are International Food Price Spikes the Source of Egypt's High Inflation?(World Bank, Washington, DC, 2012-08)This paper examines whether domestic inflation spikes in Egypt during 2001-2011 were primarily the result of external food price shocks. To estimate the pass-through of international food price inflation to domestic price inflation, two different methodologies are used: a two-step regression model estimates the pass-through in the long run, and a vector autoregression model provides the short-run estimates. The empirical evidence confirms that pass-through is high in the short term, but not in the long run. More precisely, the results show that (i) long-run pass-through to domestic food inflation is relatively low, lying between 13 and 16 percent, while the long-term spill-over from domestic food inflation to core inflation is moderate, lying around 60 percent; (ii) in the short term, pass-through is relatively high, estimated around 29 percent after 6 months and around two-thirds after a year, but the spill-over effect to core inflation is limited; (iii) international food price shocks explain only a small portion of domestic inflation shocks in both the short and long terms; and (iv) international price inflation has asymmetric effects on domestic prices.
Users also downloaded
Showing related downloaded files
Publication Women, Business and the Law 2024(Washington, DC: World Bank, 2024-03-04)Women, Business and the Law 2024 is the 10th in a series of annual studies measuring the enabling conditions that affect women’s economic opportunity in 190 economies. To present a more complete picture of the global environment that enables women’s socioeconomic participation, this year Women, Business and the Law introduces two new indicators—Safety and Childcare—and presents findings on the implementation gap between laws (de jure) and how they function in practice (de facto). This study presents three indexes: (1) legal frameworks, (2) supportive frameworks (policies, institutions, services, data, budget, and access to justice), and (3) expert opinions on women’s rights in practice in the areas measured. The study’s 10 indicators—Safety, Mobility, Workplace, Pay, Marriage, Parenthood, Childcare, Entrepreneurship, Assets, and Pension—are structured around the different stages of a woman’s working life. Findings from this new research can inform policy discussions to ensure women’s full and equal participation in the economy. The indicators build evidence of the critical relationship between legal gender equality and women’s employment and entrepreneurship. Data in Women, Business and the Law 2024 are current as of October 1, 2023.Publication Global Economic Prospects, January 2025(Washington, DC: World Bank, 2025-01-16)Global growth is expected to hold steady at 2.7 percent in 2025-26. However, the global economy appears to be settling at a low growth rate that will be insufficient to foster sustained economic development—with the possibility of further headwinds from heightened policy uncertainty and adverse trade policy shifts, geopolitical tensions, persistent inflation, and climate-related natural disasters. Against this backdrop, emerging market and developing economies are set to enter the second quarter of the twenty-first century with per capita incomes on a trajectory that implies substantially slower catch-up toward advanced-economy living standards than they previously experienced. Without course corrections, most low-income countries are unlikely to graduate to middle-income status by the middle of the century. Policy action at both global and national levels is needed to foster a more favorable external environment, enhance macroeconomic stability, reduce structural constraints, address the effects of climate change, and thus accelerate long-term growth and development.Publication Making Headway in the Water Sector(Washington, DC: World Bank, 2025-01-03)The Global Water Security and Sanitation Partnership (GWSP) was launched in 2017 as an international partnership to support countries to meet the targets related to water and sanitation under the Sustainable Development Goals (SDGs), particularly those of SDG 6. GWSP is a platform for collaboration and support, providing client countries and development partners with global knowledge, innovations, and country-level technical support while leveraging World Bank resources and financial instruments. GWSP-funded knowledge and technical assistance influence the design and implementation of government policies and programs, as well as water sector investments and reforms carried out by client countries with the support of the World Bank and other partners. The GWSP Results Framework streamlines the tracking and reporting of results using standardized indicators grouped into three blocks. The third block -Block C- includes qualitative and quantitative assessments of the influence and impact of knowledge and technical assistance on lending operations of the Global Department for Water in nine priority countries, based on agreed-upon indicators, at intervals over the life of the GWSP. These countries represent the geographic and socioeconomic diversity of GWSP’s portfolio. From fiscal year (FY) 2018 to FY24, these countries included the Arab Republic of Egypt, Bangladesh, Benin, Bolivia, Ethiopia, Haiti, Pakistan, the Socialist Republic of Viet Nam, and Uganda. In FY24, country teams completed end-of-term monitoring forms to document results and the contributions of GWSP-funded knowledge and technical assistance in strengthening institutions, regulatory and policy environments, and investments in water, sanitation, hygiene, climate-resilient irrigation, and water resources management. This report summarizes the results achieved as of June 30,2024.Publication Why Have Traffic Fatalities Declined in Industrialized Countries? Implications for Pedestrians and Vehicle Occupants(World Bank, Washington, DC, 2005-08)This paper examines whether the relationship between traffic fatalities and per capita income is the same for different classes of road users and investigates the factors underlying the decline in fatalities per vehicle kilometer traveled (VKT) observed in high-income countries over recent decades. Formal models of traffic fatalities are developed for vehicle occupants and pedestrians. Reduced-form approximations to these models are estimated using panel data for 32 high-income countries over 1964-2002. The results suggest that the downward-sloping portion of the curve relating traffic fatalities per capita to per capita income is due primarily to improved pedestrian safety. The more detailed models shed light on some factors influencing pedestrian fatalities per VKT, but much of the reduction in pedestrian fatalities remains unexplained. Increased motorization and a reduction in the proportion of young drivers in the population, however, clearly played a role. In contrast to pedestrian fatalities, occupant fatalities do not show a significant decline with income. What does explain declines in occupant fatalities per VKT are reductions in alcohol abuse and improved medical services, and a reduction in young drivers. The importance of demographic factors suggests that in countries where young persons (between 15 and 24 years of age) comprise an increasing share of the driving population, adopting policies to improve young driver education and reduce speeds will be crucial.Publication Business Ready 2024(Washington, DC: World Bank, 2024-10-03)Business Ready (B-READY) is a new World Bank Group corporate flagship report that evaluates the business and investment climate worldwide. It replaces and improves upon the Doing Business project. B-READY provides a comprehensive data set and description of the factors that strengthen the private sector, not only by advancing the interests of individual firms but also by elevating the interests of workers, consumers, potential new enterprises, and the natural environment. This 2024 report introduces a new analytical framework that benchmarks economies based on three pillars: Regulatory Framework, Public Services, and Operational Efficiency. The analysis centers on 10 topics essential for private sector development that correspond to various stages of the life cycle of a firm. The report also offers insights into three cross-cutting themes that are relevant for modern economies: digital adoption, environmental sustainability, and gender. B-READY draws on a robust data collection process that includes specially tailored expert questionnaires and firm-level surveys. The 2024 report, which covers 50 economies, serves as the first in a series that will expand in geographical coverage and refine its methodology over time, supporting reform advocacy, policy guidance, and further analysis and research.