Publication:
Using Large Language Models for Qualitative Analysis can Introduce Serious Bias

Simple item page
dc.contributor.authorAshwin, Julian
dc.contributor.authorChhabra, Aditya
dc.contributor.authorRao, Vijayendra
dc.date.accessioned2023-11-08T15:40:46Z
dc.date.available2023-11-08T15:40:46Z
dc.date.issued2023-11-08
dc.description.abstractLarge Language Models (LLMs) are quickly becoming ubiquitous, but the implications for social science research are not yet well understood. This paper asks whether LLMs can help us analyse large-N qualitative data from open-ended interviews, with an application to transcripts of interviews with displaced Rohingya people in Cox’s Bazaar, Bangladesh. The analysis finds that a great deal of caution is needed in using LLMs to annotate text as there is a risk of introducing biases that can lead to misleading inferences. Here this refers to bias in the technical sense, that the errors that LLMs make in annotating interview transcripts are not random with respect to the characteristics of the interview subjects. Training simpler supervised models on high-quality human annotations with flexible coding leads to less measurement error and bias than LLM annotations. Therefore, given that some high quality annotations are necessary in order to asses whether an LLM introduces bias, this paper argues that it is probably preferable to train a bespoke model on these annotations than it is to use an LLM for annotation.en
dc.identifierhttp://documents.worldbank.org/curated/en/099433311072326082/IDU09959393309484041660b85d0ab10e497bd1f
dc.identifier.doi10.1596/1813-9450-10597
dc.identifier.urihttps://openknowledge.worldbank.org/handle/10986/40580
dc.languageEnglish
dc.language.isoen
dc.publisherWorld Bank Washington, DC
dc.relation.ispartofseriesPolicy Research Working Papers; 10597
dc.rightsCC BY 3.0 IGO
dc.rights.holderWorld Bank
dc.rights.urihttps://creativecommons.org/licenses/by/3.0/igo/
dc.subjectLARGE LANGUAGE MODELS (LLMS)
dc.subjectSOCIAL SCIENCE RESEARCH
dc.subjectROHINGYA PEOPLE
dc.subjectTEXT AS DATA
dc.subjectCHATGPT
dc.subjectQUALITATIVE ANALYSIS
dc.subjectLLAMA 2
dc.subjectMACHINE BIAS
dc.subjectANNOTATION
dc.titleUsing Large Language Models for Qualitative Analysis can Introduce Serious Biasen
dc.typeWorking Paper
dspace.entity.typePublication
okr.crossref.titleUsing Large Language Models for Qualitative Analysis can Introduce Serious Bias
okr.date.disclosure2023-11-07
okr.date.lastmodified2023-11-07T00:00:00Zen
okr.doctypePolicy Research Working Paper
okr.doctypePublications & Research
okr.docurlhttp://documents.worldbank.org/curated/en/099433311072326082/IDU09959393309484041660b85d0ab10e497bd1f
okr.guid099433311072326082
okr.identifier.docmidIDU-99593933-9484-4166-b85d-ab10e497bd1f
okr.identifier.doi10.1596/1813-9450-10597
okr.identifier.doihttp://dx.doi.org/10.1596/1813-9450-10597
okr.identifier.externaldocumentum34192813
okr.identifier.internaldocumentum34192813
okr.identifier.reportWPS10597
okr.import.id2248
okr.importedtrueen
okr.language.supporteden
okr.pdfurlhttp://documents.worldbank.org/curated/en/099433311072326082/pdf/IDU09959393309484041660b85d0ab10e497bd1f.pdfen
okr.region.countryBangladesh
okr.topicInformation and Communication Technologies::ICT Applications
okr.topicInformation and Communication Technologies::ICT Policy and Strategies
okr.topicMacroeconomics and Economic Growth::Economic Theory & Research
okr.unitDECRG: Poverty & Inequality (DECPI)
relation.isSeriesOfPublication26e071dc-b0bf-409c-b982-df2970295c87
relation.isSeriesOfPublication.latestForDiscovery26e071dc-b0bf-409c-b982-df2970295c87
Files
Original bundle
Now showing 1 - 2 of 2
Thumbnail Image
Name:
IDU09959393309484041660b85d0ab10e497bd1f.pdf
Size:
771.03 KB
Format:
Adobe Portable Document Format
Download
No Thumbnail Available
Name:
IDU09959393309484041660b85d0ab10e497bd1f.txt
Size:
272.02 KB
Format:
Plain Text
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.71 KB
Format:
Plain Text
Description:
Download
Collections
Policy Research Working Papers