Publication: Using Large Language Models for Qualitative Analysis can Introduce Serious Bias
Loading...
Date
2023-11-08
ISSN
Published
2023-11-08
Author(s)
Editor(s)
Abstract
Large Language Models (LLMs) are quickly becoming ubiquitous, but the implications for social science research are not yet well understood. This paper asks whether LLMs can help us analyse large-N qualitative data from open-ended interviews, with an application to transcripts of interviews with displaced Rohingya people in Cox’s Bazaar, Bangladesh. The analysis finds that a great deal of caution is needed in using LLMs to annotate text as there is a risk of introducing biases that can lead to misleading inferences. Here this refers to bias in the technical sense, that the errors that LLMs make in annotating interview transcripts are not random with respect to the characteristics of the interview subjects. Training simpler supervised models on high-quality human annotations with flexible coding leads to less measurement error and bias than LLM annotations. Therefore, given that some high quality annotations are necessary in order to asses whether an LLM introduces bias, this paper argues that it is probably preferable to train a bespoke model on these annotations than it is to use an LLM for annotation.
Link to Data Set
Citation
“Ashwin, Julian; Chhabra, Aditya; Rao, Vijayendra. 2023. Using Large Language Models for Qualitative Analysis can Introduce Serious Bias. Policy Research Working Papers; 10597. © World Bank. http://hdl.handle.net/10986/40580 License: CC BY 3.0 IGO.”
Associated URLs
Associated content
Other publications in this report series
Publication Geopolitics and the World Trading System(Washington, DC: World Bank, 2024-12-23)Until the beginning of this century, the GATT/WTO system worked. Economic research provided a compelling explanation. It showed that if governments maximize the well-being of their own countries broadly defined, GATT/WTO principles would facilitate mutually beneficial cooperation over their trade policy choices. Now heightened geopolitical rivalry seems to have undermined the WTO. A simple transposition of the previous rationalization suggests that geopolitics and trade cooperation are not compatible. The paper shows that this is only true if rivalry eclipses any consideration of own-country well-being. In all other circumstances, there are gains from trade cooperation even with geopolitics. Furthermore, the WTO’s relevance is in question only if it adheres too rigidly to its existing rules and norms. Through measured adaptation to the geopolitical imperative, the WTO can continue to thrive as a forum for multilateral trade cooperation in the age of geopolitics.Publication Innovative Financial Instruments and Their Role in the Development of Jurisdictional REDD+(Washington, DC: World Bank, 2025-05-08)Achieving global net zero carbon emissions requires stopping deforestation and making full use of tropical forests as carbon sinks. Market instruments for the sale and purchase of emission outcomes coming from Reducing Emissions from Deforestation and Forest Degradation framework programs could play a very significant role in achieving this goal. The development of these markets has been insufficient so far: their scale as of today is much lower than what would be required to generate meaningful resources for the countries that host tropical forests, and the quality of existing instruments is generally insufficient to allow a scaling up in demand. However, efforts to improve the transparency and integrity of these instruments are accelerating, particularly around jurisdictional Reducing Emissions from Deforestation and Forest Degradation framework programs. In parallel with these efforts, innovations in financial instruments suited for the framework’s carbon markets are also taking place, but their scale is limited so far. This paper looks beyond the current state of the framework’s carbon markets to consider a set of innovative financial instruments that would allow completing the infrastructure of emissions trading, enhancing its utility for both issuers and buyers of carbon credits in the framework’s jurisdictional programs. The paper shows how a combination of forest carbon bonds, where countries sell forward (or commit) their emission reduction outcomes, as well as call and put options can be used to de-risk and encourage early investment in jurisdictional Reducing Emissions from Deforestation and Forest Degradation framework programs. To quantify the value of these innovations, the paper evaluates the potential scale of these instruments for the case of Brazil. The estimates suggest that the amounts that could be mobilized would represent a critical contribution to effective forest conservation. The proposed instruments and methods can be used by other tropical nations that are prepared to implement a large-scale jurisdictional program. Although the paper acknowledges that the current state of carbon markets would still not allow their deployment in the short term, the conclusion is that these instruments have significant potential, and their future development could be an important contribution to the establishment of successful markets for the conservation of tropical forests.Publication Disentangling the Key Economic Channels through Which Infrastructure Affects Jobs(Washington, DC: World Bank, 2025-04-03)This paper takes stock of the literature on infrastructure and jobs published since the early 2000s, using a conceptual framework to identify the key channels through which different types of infrastructure impact jobs. Where relevant, it highlights the different approaches and findings in the cases of energy, digital, and transport infrastructure. Overall, the literature review provides strong evidence of infrastructure’s positive impact on employment, particularly for women. In the case of electricity, this impact arises from freeing time that would otherwise be spent on household tasks. Similarly, digital infrastructure, particularly mobile phone coverage, has demonstrated positive labor market effects, often driven by private sector investments rather than large public expenditures, which are typically required for other large-scale infrastructure projects. The evidence on structural transformation is also positive, with some notable exceptions, such as studies that find no significant impact on structural transformation in rural India in the cases of electricity and roads. Even with better market connections, remote areas may continue to lack economic opportunities, due to the absence of agglomeration economies and complementary inputs such as human capital. Accordingly, reducing transport costs alone may not be sufficient to drive economic transformation in rural areas. The spatial dimension of transformation is particularly relevant for transport, both internationally—by enhancing trade integration—and within countries, where economic development tends to drive firms and jobs toward urban centers, benefitting from economies scale and network effects. Turning to organizational transformation, evidence on skill bias in developing countries is more mixed than in developed countries and may vary considerably by context. Further research, especially on the possible reasons explaining the differences between developed and developing economies, is needed.Publication Economic Consequences of Trade and Global Value Chain Integration(World Bank, Washington, DC, 2025-04-04)This paper introduces a new approach to measuring Global Value Chains (GVC), crucial for informed policy-making. It features a tripartite classification (backward, forward, and two-sided) covering trade and production data. The findings indicate that traditional trade-based GVC metrics significantly underestimate global GVC activity, especially in sectors like services and upstream manufacturing, and overstate risks in early trade liberalization stages. Additionally, conventional backward-forward classifications over-estimate backward linkages. The paper further applies these measures empirically to assess how GVC participation mediates the impact of demand shocks on domestic output, highlighting both the exposure and stabilizing potential of GVC integration. These new measures are comprehensively available on the World Bank’s WITS Platform, providing a key resource for GVC analysis.Publication Labor Market Scarring in a Developing Economy(Washington, DC: World Bank, 2025-05-08)This paper estimates the magnitude of labor market scarring in a developing economy, a setting that has been understudied by the labor scarring literature dominated by advanced economies. The paper assesses the contributions of “stigma” versus “lost human capital,” which cause earnings losses among displaced workers relative to non-displaced workers. The findings indicate that job separations caused by plant closings result in sizable and long-lasting reductions in earnings, with an average decline of 7.5 percent in hourly wages over a nine-year period. The estimate for one year after a plant closing is larger, at a decline of 10.8 percent. In a common sample, after controlling for unobserved, time-invariant individual characteristics, the impact of a plant closing declines from 11.9 to 8.2 percent. These results imply that stigma in the labor market due to imperfect information about workers (captured by unobservable worker characteristics) accounts for 30.8 percent of the average earnings losses, whereas lost employer-specific human capital explains the remaining 69.2 percent. The paper explores the effects of job separations due to plant closings on other labor market outcomes, including hours worked and informality, and provides estimates across genders and levels of education.
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication A Method to Scale-Up Interpretative Qualitative Analysis, with an Application to Aspirations in Cox’s Bazaar, Bangladesh(World Bank, Washington, DC, 2022-05)The qualitative analysis of open-ended interviews has vast potential in economics but has found limited use. This is partly because the interpretative, nuanced human reading of text and coding that it requires is labor intensive and very time consuming. This paper presents a method to simplify and shorten the coding process by extending a small set of interpretative human-codes to a larger, representative, sample using natural language processing and thus analyze qualitative data at scale. It applies it to analyze 2,200 open-ended interviews on parent’s aspirations for children with Rohingya refugees and their Bangladeshi hosts. It shows that studying aspirations with open-ended interviews extends the economics focus on material goals to ideas from philosophy and anthropology that emphasize aspirations for moral and religious values, and the navigational capacity to achieve these aspirations. The paper shows how to assess the robustness and reliability of this approach and finds that extending the sample of interviews, rather than the human-coded training set, is likely to be optimal.Publication Central America : Big Data in Action for Development(Washington, DC, 2014)This report stemmed from a World Bank pilot activity to explore the potential of big data to address development challenges in Central American countries. As part of this activity we collected and analyzed a number of examples of leveraging big data for development. Because of the growing interest in this topic this report makes available to a broader audience those examples as well as the underlying conceptual framework to think about big data for development. To make effective use of big data, many practitioners emphasize the importance of beginning with a question instead of the data itself. A question clarifies the purpose of utilizing big data, whether it is for awareness, understanding, and/or forecasting. In addition, a question suggests the kinds of real-world behaviors or conditions that are of interest. These behaviors are encoded into data through some generating process which includes the media through which behavior is captured. Then various data sources are accessed, prepared, consolidated and analyzed. This ultimately gives rise to insights into the question of interest, which are implemented to effect changes in the relevant behaviors. Utilizing big data for any given endeavor requires a host of capabilities. Hardware and software capabilities are needed for interaction of data from a variety of sources in a way which is efficient and scalable. Human capabilities are needed not only to make sense of data but to ensure a question-centered approach, so that insights are actionable and relevant. To this end, cooperation between development experts as well as social scientists and computer scientists is extremely important.Publication Integrating Qualitative Methods into Investment Climate Impact Evaluations(World Bank Group, Washington, DC, 2014-12)Incorporating qualitative methods into the evaluation of development programs has become increasingly popular in recent years, both for the distinctive insights such approaches can bring in their own right and because of their capacity to complement the strengths -- and where necessary correct some of the weaknesses -- of quantitative approaches. Some initial work deploying mixed methods has been undertaken in the assessment of investment climate reforms, but considerable room for expansion exists. This paper summarizes some of the key principles and practices underpinning mixed methods evaluations in development, highlight some notable examples of how such work has been conducted (and the particular contributions it has made), and offers some guidelines for those seeking to increase the sophistication and utility of qualitative methods in the evaluation of investment climate reforms.Publication Toward Greater Social Inclusion in Poland : A Qualitative Assessment in Three Regions(Washington, DC, 2014-05)In Poland, addressing the situation of the remaining poor groups is likely to become much harder over time as their problems are likely to be deeper and their situation more complex. A social inclusion approach that tackles their multiple disadvantages will be needed. This study aims to contribute to Poland's social inclusion debate by providing policy makers and civil society with evidence from the field about (1) what population groups are currently 'socially excluded;' (2) what are the driving factors of their exclusion; and (3) the success and failure of current social inclusion policies and programs. The ultimate goal of this work is to make current social inclusion interventions more effective by learning from what has been tried. The findings are particularly relevant now that a new EU funding cycle has started, with part of the funds earmarked for tackling social inclusion. The study was conducted in three regions: Malopolskie, Podkarpackie, and Mazowieckie (in Radom County only). The first two are among Poland's poorest regions in terms of income poverty. The part of Mazowieckie in which the research was conducted also has a higher than average poverty rate; in addition, the unemployment rate there (31 percent) is much greater than the national average (about 13 percent in 2013). Capitals of the other two regions were excluded from the research.Publication Measurement and Meaning : Combining Quantitative and Qualitative Methods for the Analysis of Poverty and Social Exclusion in Latin America(Washington, DC: World Bank, 2001-12)This report consists of a collection of case studies from Latin America combining qualitative and quantitative research methods for the analysis of poverty within a social exclusion framework. The first chapter provides an overview of the differences between quantitative and qualitative methods, and the gains from using both types of methods in applied work. The other chapters are devoted to three case studies on reproductive health in rural Argentina, the targeting of social programs in Chile, and social exclusion in urban Uruguay. Each case study was prepared within the broader context of country-specific economic and sectoral work at the World Bank.
Users also downloaded
Showing related downloaded files
Publication Digital Progress and Trends Report 2023(Washington, DC: World Bank, 2024-03-05)Digitalization is the transformational opportunity of our time. The digital sector has become a powerhouse of innovation, economic growth, and job creation. Value added in the IT services sector grew at 8 percent annually during 2000–22, nearly twice as fast as the global economy. Employment growth in IT services reached 7 percent annually, six times higher than total employment growth. The diffusion and adoption of digital technologies are just as critical as their invention. Digital uptake has accelerated since the COVID-19 pandemic, with 1.5 billion new internet users added from 2018 to 2022. The share of firms investing in digital solutions around the world has more than doubled from 2020 to 2022. Low-income countries, vulnerable populations, and small firms, however, have been falling behind, while transformative digital innovations such as artificial intelligence (AI) have been accelerating in higher-income countries. Although more than 90 percent of the population in high-income countries was online in 2022, only one in four people in low-income countries used the internet, and the speed of their connection was typically only a small fraction of that in wealthier countries. As businesses in technologically advanced countries integrate generative AI into their products and services, less than half of the businesses in many low- and middle-income countries have an internet connection. The growing digital divide is exacerbating the poverty and productivity gaps between richer and poorer economies. The Digital Progress and Trends Report series will track global digitalization progress and highlight policy trends, debates, and implications for low- and middle-income countries. The series adds to the global efforts to study the progress and trends of digitalization in two main ways: · By compiling, curating, and analyzing data from diverse sources to present a comprehensive picture of digitalization in low- and middle-income countries, including in-depth analyses on understudied topics. · By developing insights on policy opportunities, challenges, and debates and reflecting the perspectives of various stakeholders and the World Bank’s operational experiences. This report, the first in the series, aims to inform evidence-based policy making and motivate action among internal and external audiences and stakeholders. The report will bring global attention to high-performing countries that have valuable experience to share as well as to areas where efforts will need to be redoubled.Publication The Container Port Performance Index 2023(Washington, DC: World Bank, 2024-07-18)The Container Port Performance Index (CPPI) measures the time container ships spend in port, making it an important point of reference for stakeholders in the global economy. These stakeholders include port authorities and operators, national governments, supranational organizations, development agencies, and other public and private players in trade and logistics. The index highlights where vessel time in container ports could be improved. Streamlining these processes would benefit all parties involved, including shipping lines, national governments, and consumers. This fourth edition of the CPPI relies on data from 405 container ports with at least 24 container ship port calls in the calendar year 2023. As in earlier editions of the CPPI, the ranking employs two different methodological approaches: an administrative (technical) approach and a statistical approach (using matrix factorization). Combining these two approaches ensures that the overall ranking of container ports reflects actual port performance as closely as possible while also being statistically robust. The CPPI methodology assesses the sequential steps of a container ship port call. ‘Total port hours’ refers to the total time elapsed from the moment a ship arrives at the port until the vessel leaves the berth after completing its cargo operations. The CPPI uses time as an indicator because time is very important to shipping lines, ports, and the entire logistics chain. However, time, as captured by the CPPI, is not the only way to measure port efficiency, so it does not tell the entire story of a port’s performance. Factors that can influence the time vessels spend in ports can be location-specific and under the port’s control (endogenous) or external and beyond the control of the port (exogenous). The CPPI measures time spent in container ports, strictly based on quantitative data only, which do not reveal the underlying factors or root causes of extended port times. A detailed port-specific diagnostic would be required to assess the contribution of underlying factors to the time a vessel spends in port. A very low ranking or a significant change in ranking may warrant special attention, for which the World Bank generally recommends a detailed diagnostic.Publication Interoperability Between Central Bank Digital Currency Systems and Fast Payment Systems(Washington, DC: World Bank, 2024-07-02)Central banks around the world are actively researching and investigating the benefits, challenges, and design options of wholesale and retail central bank digital currencies (CBDCs). Since CBDCs are one of the most critical components of a national payment system (NPS), it is important that their interoperability with other payment systems is one of the key considerations in the design process. The ITS Technology and Innovation (ITSI) team, in collaboration with the World Banks’s Finance Competitiveness and Innovation (FCI) Global Practice, has conducted technology design experiments on two specific scenarios regarding CBDC system interoperability with fast payment systems (FPS). In the first scenario, the experiment investigated the option of settling FPS obligations in a wholesale CBDC system, including the option to reserve funds to guarantee the settlement of FPS net obligations. In the second scenario, the team investigated the interoperability between users within the FPS and retail CBDC users, including the transfer of funds among both types of users, using common services such as address resolution services. This experiment illustrated how CBDC systems can interoperate with retail payment systems through an interlinking bridge that was used to route messages and application programming interface (API) calls among different systems. The programmability features of distributed ledge technology (DLT) were used to link the settlement in CBDC to the transfer of funds in the FPS. The technical applicability for this type of interoperability was demonstrated through the experiments, with the caveat that these experiments do not take into account complexities that may be involved with live systems.Publication World Development Report 2014(Washington, DC, 2013-10-06)The past 25 years have witnessed unprecedented changes around the world—many of them for the better. Across the continents, many countries have embarked on a path of international integration, economic reform, technological modernization, and democratic participation. As a result, economies that had been stagnant for decades are growing, people whose families had suffered deprivation for generations are escaping poverty, and hundreds of millions are enjoying the benefits of improved living standards and scientific and cultural sharing across nations. As the world changes, a host of opportunities arise constantly. With them, however, appear old and new risks, from the possibility of job loss and disease to the potential for social unrest and environmental damage. If ignored, these risks can turn into crises that reverse hard-won gains and endanger the social and economic reforms that produced these gains. The World Development Report 2014 (WDR 2014), Risk and Opportunity: Managing Risk for Development, contends that the solution is not to reject change in order to avoid risk but to prepare for the opportunities and risks that change entails. Managing risks responsibly and effectively has the potential to bring about security and a means of progress for people in developing countries and beyond. Although individuals’ own efforts, initiative, and responsibility are essential for managing risk, their success will be limited without a supportive social environment—especially when risks are large or systemic in nature. The WDR 2014 argues that people can successfully confront risks that are beyond their means by sharing their risk management with others. This can be done through naturally occurring social and economic systems that enable people to overcome the obstacles that individuals and groups face, including lack of resources and information, cognitive and behavioral failures, missing markets and public goods, and social externalities and exclusion. These systems—from the household and the community to the state and the international community—have the potential to support people’s risk management in different yet complementary ways. The Report focuses on some of the most pressing questions policy makers are asking. What role should the state take in helping people manage risks? When should this role consist of direct interventions, and when should it consist of providing an enabling environment? How can governments improve their own risk management, and what happens when they fail or lack capacity, as in many fragile and conflict-affected states? Through what mechanisms can risk management be mainstreamed into the development agenda? And how can collective action failures to manage systemic risks be addressed, especially those with irreversible consequences? The WDR 2014 provides policy makers with insights and recommendations to address these difficult questions. It should serve to guide the dialogue, operations, and contributions from key development actors—from civil society and national governments to the donor community and international development organizations.Publication Global Economic Prospects, January 2025(Washington, DC: World Bank, 2025-01-16)Global growth is expected to hold steady at 2.7 percent in 2025-26. However, the global economy appears to be settling at a low growth rate that will be insufficient to foster sustained economic development—with the possibility of further headwinds from heightened policy uncertainty and adverse trade policy shifts, geopolitical tensions, persistent inflation, and climate-related natural disasters. Against this backdrop, emerging market and developing economies are set to enter the second quarter of the twenty-first century with per capita incomes on a trajectory that implies substantially slower catch-up toward advanced-economy living standards than they previously experienced. Without course corrections, most low-income countries are unlikely to graduate to middle-income status by the middle of the century. Policy action at both global and national levels is needed to foster a more favorable external environment, enhance macroeconomic stability, reduce structural constraints, address the effects of climate change, and thus accelerate long-term growth and development.