Publication: Machine Learning in Evaluative Synthesis: Lessons from Private Sector Evaluation in the World Bank Group
Loading...
Published
2023-07-20
ISSN
Date
2023-07-20
Editor(s)
Abstract
This resource discusses the use of machine learning (ML) techniques in evaluation research and their potential to automate the process of extracting and classifying large amounts of texts. ML methods can accelerate the process of extracting and classifying content in evaluation research provided that practitioners train the extraction tool properly. In practical terms, such an approach can offer evaluators a powerful analytical tool for a range of evaluative purposes, for example, for better understanding the various determinants of project success, potential challenges to project implementation, and practical lessons for future projects, among others. With the above goal in mind, the paper provides an overview of ML and discusses relevant applications in the field of evaluation. This is supported by the case of the Finance and Private Sector Evaluation Unit of the Independent Evaluation Group as an example to illustrate the benefits of ML for text classification in evaluation. The paper concludes by offering a summary of the results of this experiment and a brief discussion of potential next steps.
Link to Data Set
Citation
“Bravo, Leonardo; Hagh, Ariya; Joseph, Roshin; Kambe, Hiroaki; Xiang, Yuan; Vaessen, Jos. 2023. Machine Learning in Evaluative Synthesis: Lessons from Private Sector Evaluation in the World Bank Group. IEG Methods and Evaluation Capacity Development Working Paper Series. © World Bank. http://hdl.handle.net/10986/40054 License: CC BY-NC 3.0 IGO.”
Digital Object Identifier
Associated URLs
Associated content
Other publications in this report series
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication What Can We (Machine) Learn about Welfare Dynamics from Cross-Sectional Data?(World Bank, Washington, DC, 2018-08)This paper implements a machine learning approach to estimate intra-generational economic mobility using cross-sectional data. A Least Absolute Shrinkage and Selection Operator (Lasso) procedure is applied to explore poverty dynamics and household-level welfare growth in the absence of panel data sets that follow individuals over time. The method is validated by sampling repeated cross-sections of actual panel data from Peru. In general, the approach performs well at estimating intra-generational poverty transitions; most of the mobility estimates fall within the 95 percent confidence intervals of poverty mobility from the actual panel data. The validation also confirms that the Lasso regularization procedure performs well at estimating household-level welfare growth between two years. Overall, the results are sufficiently encouraging to estimate economic mobility in settings where panel data are not available or, if they are, to improve panel data when they suffer from serious non-random attrition problems.Publication CLARE: A Causal machine Learning Approach to Resilience Estimation(Washington, DC: World Bank, 2026-01-12)This paper proposes a new resilience index, CLARE (Causal machine Learning Approach to Resilience Estimation), which is rooted in an impact evaluation framework and causal machine learning algorithms applied to longitudinal household survey data. The indicator is model-agnostic, data-driven, scalable, and normatively anchored to wellbeing thresholds, and can be either shock-specific or a general-purpose resilience metric. The paper provides an empirical demonstration of constructing the CLARE resilience index, leveraging more than 28,000 household observations from 19 nationally representative, longitudinal, multi-topic surveys that were implemented by the national statistical offices in Malawi, Nigeria, Tanzania, and Uganda over 2009–20 in partnership with the World Bank Living Standards Measurement Study. Although the paper centers on measuring resilience to drought, the proposed index is applicable to any type of shock. The analysis shows that CLARE outperforms existing resilience metrics and alternative approaches to predict food insecurity out-of-sample—both in the future (dynamic forecasting) and in held-out countries (cross-sectional prediction). The index can be decomposed to causally identify the relative importance of resilience capacities that can insulate populations from shocks. Thus, it can be operationalized in designing, targeting, and monitoring policies and investments that aim to strengthen resilience. CLARE’s deployment—paired with continued investments in national longitudinal survey platforms—can boost the effectiveness of early-warning systems and resilience-building interventions, while allowing the transfer of resilience policy advice from data-rich contexts to data-poor environments that may not immediately provide the requisite longitudinal survey data for index estimation.Publication Applying Machine Learning and Geolocation Techniques to Social Media Data (Twitter) to Develop a Resource for Urban Planning(World Bank, Washington, DC, 2020-12)With all the recent attention focused on big data, it is easy to overlook that basic vital statistics remain difficult to obtain in most of the world. This project set out to test whether an openly available dataset (Twitter) could be transformed into a resource for urban planning and development. The hypothesis is tested by creating road traffic crash location data, which are scarce in most resource-poor environments but essential for addressing the number one cause of mortality for children over age five and young adults. The research project scraped 874,588 traffic-related tweets in Nairobi, Kenya, applied a machine learning model to capture the occurrence of a crash, and developed an improved geoparsing algorithm to identify its location. The project geolocated 32,991 crash reports in Twitter for 2012-20 and clustered them into 22,872 unique crashes to produce one of the first crash maps for Nairobi. A motorcycle delivery service was dispatched in real-time to verify a subset of crashes, showing 92 percent accuracy. Using a spatial clustering algorithm, portions of the road network (less than 1 percent) were identified where 50 percent of the geolocated crashes occurred. Even with limitations in the representativeness of the data, the results can provide urban planners useful information to target road safety improvements where resources are limited.Publication Comparison of the Monitoring and Evaluation Systems of the World Bank and the Global Fund(World Bank, Washington, DC, 2012)The purpose of this study is to document the approaches of the World Bank and the Global Fund to monitoring and evaluation (M&E), and to systematically and objectively compare the principles and objectives of the M&E systems and how these systems are implemented and used in practice at the country level. The report also discusses the relationship of the M&E systems to the two different business models of the World Bank and the Global Fund. The first goal is to identify whether and how the conclusions that emerge from their M&E systems on the effectiveness of their respective global health activities can be compared. The second goal is to contribute to the ongoing process of identifying good practices for developing M&E policies for global health programs, setting up M&E frameworks, planning and programming evaluations, and using M&E results more effectively to manage programs and strengthen the health policy process in partner countries. The report is organized as follows. Section two summarizes the World Bank's stated policies and approach to monitoring and evaluation. Each element of standard M&E systems framework; system of indicators and performance measurement; data collection and analysis; feedback and use of monitoring findings; and evaluation is described for the World Bank's approach in this section. Section three summarizes the Global Fund's stated M&E approach and policies according to the same structure. Section four compares the application of the approach to M&E of the two agencies in Burkina Faso, Lesotho and Russia. Section five provides a summary of the comparison between the two approaches to M&E. Section six identifies conclusions and lessons learned.Publication Some Small Countries Do It Better : Rapid Growth and Its Causes in Singapore, Finland, and Ireland(World Bank, 2012)This book is an outcome of a series of study visits to Singapore for African policy makers initiated by Jee-Peng Tan in 2005 with support from Tommy Koh in Singapore and Birger Fredriksen, Yaw Ansu, and Dzingai Mutumbuka at the World Bank. Starting in the 1960s-earlier if Japan is included-a number of East Asian economies began achieving growth rates well above the average and were able to maintain that pace until nearly the end of the 1990s. Countries, large and small, have struggled to imitate the industrial prowess of the East Asian pacesetters and to exploit the opportunities presented by globalization to expand exports. But approximating the East Asian benchmarks has proven difficult, and growth accelerations have tended to be remarkably transient.
Users also downloaded
Showing related downloaded files
Publication Kyrgyz Republic Country Climate and Development Report(Washington, DC: World Bank, 2025-11-03)This Country Climate and Development Report (CCDR) on the Kyrgyz Republic aims to support the country’s development goals amid a changing climate. The CCDR considers two policy scenarios up to 2050: the business-as-usual (BAU) and high-growth scenarios. As it quantifies the likely impacts of climate change on the Kyrgyz economy between now and 2050, the report highlights key government actions to best prepare for and adapt to climate impacts (referred to as “with adaptation” measures), with a particular focus on the time horizon up to 2030. The CCDR also outlines a path to net zero emissions by 2050 (referred to as “with mitigation” measures, “decarbonization,” or, simply, “net zero 2050”), highlighting associated development co-benefits.Publication Jobs in a Changing Climate: Insights from World Bank Group Country Climate and Development Reports Covering 93 Economies(Washington, DC: World Bank, 2025-11-05)The World Bank Group’s Country Climate and Development Reports (CCDRs) provide a crosscutting look at how countries’ development prospects, and the job opportunities they offer to their people, can be threatened by climate impacts and supported by climate policies. Climate change and policies affect jobs through impacts on productivity, energy and material efficiency, and physical, human, and natural capital. They can also transform employment opportunities, especially through complementary measures that help workers and firms adapt to and benefit from new technologies and production practices. Prepared by the World Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA), CCDRs integrate country perspectives, climate science and economic modeling, private sector information, and policy analysis to assess how countries can successfully grow and develop their economies and create jobs despite increasing climate risks and while achieving their climate objectives and commitments. Each CCDR starts from the country’s development priorities, opportunities, and challenges, and is developed in close consultation with governments, businesses, and civil society, ensuring the recommendations reflect national priorities. By combining evidence on adaptation, resilience, and emissions pathways, CCDRs highlight where climate action can reinforce development and job creation, and where targeted policies are needed to manage risks and smooth labor market transitions. Taken together, these elements can help create local jobs, ensure economic transitions are just and inclusive, and equip workers and firms to navigate the disruptions and opportunities of a changing climate and changing technologies.Publication Comoros Country Climate and Development Report(Washington, DC: World Bank, 2025-06-18)The Union of the Comoros (The Comoros) has significant vulnerability to climate change-related risks but has considerable opportunities to strengthen preparedness and resilience against these challenges. According to the Notre Dame Global Adaptation Index, the Comoros is the 29th-most vulnerable country to climate change and the 163rd most ready to adapt (out of 191). The Comoros archipelago is exposed to many natural hazards that adversely affect the country’s natural capital, people, and physical infrastructure. In 2014, the economic cost of climate-related disasters was estimated at 5.7 million dollars annually, equivalent to 9.2 percent of Gross Domestic Product (GDP). Between 2018 and 2023, as many as 11 tropical depressions or cyclones impacted the country, with Cyclone Kenneth causing the greatest damage, equivalent to 14 percent of GDP, resulting in total economic growth falling from 3.6 percent in 2018 to 1.9 percent in 2019. More than 345,000 people (40 percent of the population) were affected by the cyclone, with 185,000 people experiencing severe impacts and 12,000 people displaced. However, there is an opportunity for the country to grow more robust and shock-responsive, and to establish pre-positioned funding mechanisms to enhance future crisis response efforts. For the Comoros, adaptation and climate-resilient development are the key climate change focus areas, with the country projected to face 836 million dollars 2050 in additional costs due to climate-related impacts. Current plans to adapt to the impacts of climate change in the Comoros include efforts to improve water management, strengthen coastal protection, and develop climate-smart agriculture practices. Given the country’s reliance on its natural resource base for economic growth and mobility, protection of these resources from climate change will be essential for promoting resilient growth and development. In addition to growing the adaptive capacity of the country’s natural resource sectors, strategic economic diversification will be important to help minimize future climate impacts, and development activities will need to be undertaken in such a way as to attract low-carbon co-benefits. The Union of the Comoros is committed to addressing climate change through its Nationally Determined Contribution (NDC) and national priorities. The country’s NDC (which was revised in 2021 for a ten-year horizon) sets ambitious targets, with a goal of reducing greenhouse gas emissions by 23 percent by 2030. The country also plans to significantly increase the share of renewable energy in its energy portfolio, reaching 33 MW by 2030. This will not only promote low-carbon development but also reduce the country’s dependency on imported oil and coal, which currently make up 95 percent of the energy mix. Additionally, the Comoros has declared its intention to increase CO2 removals by 47 percent by 2030, compared to BAU.Publication Guinea-Bissau Country Climate and Development Report(Washington, DC: World Bank, 2024-10-23)Guinea-Bissau is endowed with a wealth of natural resources, with the highest natural capital per capita in West Africa (US3,874 dollars per capita), which could be leveraged for sustainable and resilient growth. However, Guinea-Bissau faces significant development hurdles, such as high poverty rates, political instability, and economic challenges, including an over-reliance on cashew nuts. Rural poverty has increased, and the nation's infrastructure, education, and health care systems are underdeveloped. Climate change poses a severe threat, potentially impacting agriculture, fisheries, and infrastructure. Without adaptation, it could lead to a significant cut in real GDP per capita (minus 7.3 percent by 2050) and increase in poverty (with up to over 200,000 additional poor by 2050, that is, 5 percent of the expected population, in the worst scenario). The country's low greenhouse gas emissions are expected to rise, mainly due to agriculture and land-use changes, with deforestation being a major contributing factor. Although Guinea-Bissau is a low emitter, it has high mitigation ambitions, targeting a 30 percent reduction in greenhouse gas emissions by 2030. The Nationally Determined Contribution outlines significant climate actions, with initiatives focused on forest conservation, sustainable agriculture, and community development. However, the country's political instability, institutional weaknesses, and limited financial resources pose challenges to implementing these climate commitments, which depend heavily on external funding. The financial sector's underdevelopment and vulnerability to external shocks limit its ability to support green investments, though reforms could enhance resilience. Guinea-Bissau must consider its climate financing as development financing and vice-versa, engage the private sector, and integrate climate goals with national development plans to ensure a sustainable future. Concessional climate financing is vital due to the underdeveloped financial sector and the government’s limited borrowing capacity. Addressing Guinea-Bissau's vulnerability to climate change and its structural issues requires a cohesive approach that integrates development and climate strategies. This could involve improving governance, diversifying the economy, protecting natural capital, developing human capital, and investing in sustainable agriculture and infrastructure. The transition to a more sustainable and inclusive development pathway that supports economic growth is possible, but requires focusing on key strategic sectors, enhancing institutional capacity, and creating the conditions to mobilize finance. As a highly vulnerable country, there are myriad needs in the different sectors; however, to be more efficient and effective, Guinea-Bissau should prioritize actions in a few sectors, especially actions on biodiversity, agriculture, and social protection. Low carbon development, especially in energy and forestry sectors, could provide cost-efficient solutions and attract climate finance, including from the private sector, which will support the overall development agenda.Publication Mongolia Country Climate and Development Report(Washington, DC: World Bank, 2024-10-22)Mongolia’s development prospects are uniquely challenged by both the impacts of climate change and the global shift toward a low-carbon economy. The country’s efforts toward decarbonization pose significant challenges given the structurally high-emission intensity of its economy. While challenging, climate action also presents Mongolia with opportunities to achieve important development benefits. The effects of climate risks and the shift away from coal will have diverse impacts across different regions, communities, and socioeconomic levels. The report assesses the critical interconnections between Mongolia’s development ambitions and climate change action and identifies ways to transition to a more economically diversified, inclusive, and resilient development path. It highlights key climate and transition risks affecting Mongolia’s future development and presents a pathway to enhance climate mitigation and adaptation. The report also makes a case for strengthening policies to enhance resilience to climate change and ensure a just transition, particularly for the most vulnerable. The report is structured as follows: section 1 gives introduction. Section 2 delves into the linkages between development and climate in Mongolia and presents model-based findings on the economic and poverty impacts of climate change under different scenarios. Section 3 covers four in-depth sectoral analyses. The first two mainly focus on adaptation to climate change in the agriculture and water sectors. The third considers prospects for the extraction sector, while the fourth sectoral analysis focuses on decarbonizing power and heat generation. Section 4 shifts the focus to how the government can boost resilience for climate-vulnerable populations. Section 5 outlines options for mobilizing private and public financing and private investments to support the green transition. Section 6 examines the existing institutional and governance structure for climate action and presents recommendations to improve its effectiveness, and section 7 concludes with a framework for prioritizing the policy actions outlined in this report.