Publication: Machine Learning in Evaluative Synthesis: Lessons from Private Sector Evaluation in the World Bank Group
Loading...
Date
2023-07-20
ISSN
Published
2023-07-20
Editor(s)
Abstract
This resource discusses the use of machine learning (ML) techniques in evaluation research and their potential to automate the process of extracting and classifying large amounts of texts. ML methods can accelerate the process of extracting and classifying content in evaluation research provided that practitioners train the extraction tool properly. In practical terms, such an approach can offer evaluators a powerful analytical tool for a range of evaluative purposes, for example, for better understanding the various determinants of project success, potential challenges to project implementation, and practical lessons for future projects, among others. With the above goal in mind, the paper provides an overview of ML and discusses relevant applications in the field of evaluation. This is supported by the case of the Finance and Private Sector Evaluation Unit of the Independent Evaluation Group as an example to illustrate the benefits of ML for text classification in evaluation. The paper concludes by offering a summary of the results of this experiment and a brief discussion of potential next steps.
Link to Data Set
Citation
“Bravo, Leonardo; Hagh, Ariya; Joseph, Roshin; Kambe, Hiroaki; Xiang, Yuan; Vaessen, Jos. 2023. Machine Learning in Evaluative Synthesis: Lessons from Private Sector Evaluation in the World Bank Group. IEG Methods and Evaluation Capacity Development Working Paper Series. © World Bank. http://hdl.handle.net/10986/40054 License: CC BY-NC 3.0 IGO.”
Associated URLs
Associated content
Other publications in this report series
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication What Can We (Machine) Learn about Welfare Dynamics from Cross-Sectional Data?(World Bank, Washington, DC, 2018-08)This paper implements a machine learning approach to estimate intra-generational economic mobility using cross-sectional data. A Least Absolute Shrinkage and Selection Operator (Lasso) procedure is applied to explore poverty dynamics and household-level welfare growth in the absence of panel data sets that follow individuals over time. The method is validated by sampling repeated cross-sections of actual panel data from Peru. In general, the approach performs well at estimating intra-generational poverty transitions; most of the mobility estimates fall within the 95 percent confidence intervals of poverty mobility from the actual panel data. The validation also confirms that the Lasso regularization procedure performs well at estimating household-level welfare growth between two years. Overall, the results are sufficiently encouraging to estimate economic mobility in settings where panel data are not available or, if they are, to improve panel data when they suffer from serious non-random attrition problems.Publication Applying Machine Learning and Geolocation Techniques to Social Media Data (Twitter) to Develop a Resource for Urban Planning(World Bank, Washington, DC, 2020-12)With all the recent attention focused on big data, it is easy to overlook that basic vital statistics remain difficult to obtain in most of the world. This project set out to test whether an openly available dataset (Twitter) could be transformed into a resource for urban planning and development. The hypothesis is tested by creating road traffic crash location data, which are scarce in most resource-poor environments but essential for addressing the number one cause of mortality for children over age five and young adults. The research project scraped 874,588 traffic-related tweets in Nairobi, Kenya, applied a machine learning model to capture the occurrence of a crash, and developed an improved geoparsing algorithm to identify its location. The project geolocated 32,991 crash reports in Twitter for 2012-20 and clustered them into 22,872 unique crashes to produce one of the first crash maps for Nairobi. A motorcycle delivery service was dispatched in real-time to verify a subset of crashes, showing 92 percent accuracy. Using a spatial clustering algorithm, portions of the road network (less than 1 percent) were identified where 50 percent of the geolocated crashes occurred. Even with limitations in the representativeness of the data, the results can provide urban planners useful information to target road safety improvements where resources are limited.Publication Poverty Mapping in the Age of Machine Learning(World Bank, Washington, DC, 2023-05-04)Recent years have witnessed considerable methodological advances in poverty mapping, much of which has focused on the application of modern machine-learning approaches to remotely sensed data. Poverty maps produced with these methods generally share a common validation procedure, which assesses model performance by comparing subnational machine-learning-based poverty estimates with survey-based, direct estimates. Although unbiased, survey-based estimates at a granular level can be imprecise measures of true poverty rates, meaning that it is unclear whether the validation procedures used in machine-learning approaches are informative of actual model performance. This paper examines the credibility of existing approaches to model validation by constructing a pseudo-census from the Mexican Intercensal Survey of 2015, which is used to conduct several design-based simulation experiments. The findings show that the validation procedure often used for machine-learning approaches can be misleading in terms of model assessment since it yields incorrect information for choosing what may be the best set of estimates across different methods and scenarios. Using alternative validation methods, the paper shows that machine-learning-based estimates can rival traditional, more data intensive poverty mapping approaches. Further, the closest approximation to existing machine-learning approaches, using publicly available geo-referenced data, performs poorly when evaluated against “true” poverty rates and fails to outperform traditional poverty mapping methods in targeting simulations.Publication Machine Learning in International Trade Research(World Bank, Washington, DC, 2021-04)Modern trade agreements contain a large number of provisions besides tariff reductions, in areas as diverse as services trade, competition policy, trade-related investment measures, or public procurement. Existing research has struggled with overfitting and severe multicollinearity problems when trying to estimate the effects of these provisions on trade flows. This paper builds on recent developments in the machine learning and variable selection literature to propose novel data-driven methods for selecting the most important provisions and quantifying their impact on trade flows. The proposed methods have the advantage of not requiring ad hoc assumptions on how to aggregate individual provisions and offer improved selection accuracy over the standard lasso. The analysis finds that provisions related to technical barriers to trade, antidumping, trade facilitation, subsidies, and competition policy are associated with enhancing the trade-increasing effect of trade agreements.Publication Program Targeting with Machine Learning and Mobile Phone Data(World Bank, Washington, DC, 2022-12)Can mobile phone data improve program targeting By combining rich survey data from the baseline of a “big push” anti-poverty program in Afghanistan implemented in 2016 with detailed mobile phone logs from program beneficiaries, this paper studies the extent to which machine learning methods can accurately differentiate ultra-poor households eligible for program benefits from ineligible households. The paper shows that machine learning methods leveraging mobile phone data can identify ultra-poor households nearly as accurately as survey-based measures of consumption and wealth; and that combining survey-based measures with mobile phone data produces classifications more accurate than those based on a single data source.
Users also downloaded
Showing related downloaded files
Publication IFC Annual Report 2005 : Investing in Progress with Experience, Innovation, and Partnership, Volume 1(Washington, D, 2005)The International Finance Corporation is at the forefront of private sector development: it is redefining how poverty can be reduced and lives improved through a stronger private sector in emerging markets. Accomplishing this goal means reaching people, regions, and sectors that have not yet shared in the overall growth of emerging markets. It means innovation-forging new partnerships with governments and other multilateral institutions, identifying new roles for the private sector, creating products that develop financial markets, and making it easier for disadvantaged people to launch a business or own a home. It means building on significant strengths in many countries and industries-helping established enterprises become more competitive and sustainable as they expand their operations or extend their reach into new markets. It also means bringing to developing economies proven products and techniques, both from industrialized countries and, increasingly, from other developing countries. Above all, it means tailoring global expertise to local needs. This annual report outlines the strategic objectives of the institution in expanding access to finance, increasing private participation in key sectors, helping successful enterprises grow, focusing where needs are greatest, and ensuring sustainability.Publication 2006 Annual Report on Operations Evaluation(Washington, DC: World Bank, 2006)The 2006 Annual Report on Operations Evaluation (AROE) updates the actions taken since the 2004 and 2005 AROEs to strengthen the results focus in monitoring and evaluation (M&E). First, it analyzes the implications of managing for results on Bank operations. Second, it assesses the extent to which the Bank's M&E systems provide staff with the information they need to better manage for results. Third, the report evaluates products and services of the Independent Evaluation Group-World Bank (IEG-WB) as part of a continuous process of self-evaluation and assesses their quality, influence, and use among both internal and external audiences. The report also raises some unresolved issues for further consideration. Finally, it makes recommendations on how M&E can be strengthened to increase the World Bank's effectiveness.Publication Private Solutions for Infrastructure in Rwanda : A Country Framework Report(Washington, DC: World Bank, 2005)This report aims to provide an objective assessment of the condition of Rwanda's infrastructure sectors and of the institutional and policy frameworks that are associated with them. It also provides a clear route map for infrastructure sector reform, as well as highlighting both the opportunities that exist for the private sector and the role that the donor community can play in assisting the Government with establishing priorities in infrastructure.Publication World Bank-Civil Society Engagement(World Bank, Washington, DC, 2013)World Bank relations with civil society continued to expand throughout the institution during 2010–12. This evolution was experienced across the spectrum of the "engagement continuum," which includes information disclosure, policy dialogue, strategy consultations, operational collaboration, and institutional partnerships.Publication Poverty Traps, Aid, and Growth(World Bank, Washington, DC, 2005-06)The authors examine the empirical evidence in support of the poverty trap view of underdevelopment. They calibrate simple aggregate growth models in which poverty traps can arise due to either low saving or low technology at low levels of development. They then use these models to assess the empirical relevance of poverty traps and their consequences for policy. The authors find little evidence of the existence of poverty traps based on these two broad mechanisms. When put to the task of explaining the persistence of low income in African countries, the models require either unreasonable values for key parameters, or else generate counterfactual predictions regarding the relations between key variables. These results call into question the view that a large scaling-up of aid to the poorest countries is a necessary condition for sharp and sustained increases in growth.