Reducing COVID-19 Vaccine Hesitancy with Social and Behavior Change Communication Strategies on Social Media Reducing COVID-19 Vaccine Hesitancy with Social and Behavior Change Communication Strategies on Social Media DECEMBER 2024 Table of Contents Executive Summary  4 1. Introduction  6 2. Project Overview  10 3. Impacts  21 4. Lessons Learned  28 5. Recommendations for Future Projects  32 6. Next Steps  35 7. Conclusion  39 REDUCING COVID-19 VACCINE HESITANCY WITH SOCIAL AND BEHAVIOR CHANGE COMMUNICATION STRATEGIES ON SOCIAL MEDIA 4 Executive Summary The COVID-19 pandemic significantly disrupted daily life and exacerbated global inequalities, leading policymakers to seek innovative, evidence-based solutions to mitigate its impacts. By March 2023, the pandemic had resulted in approximately 6.88 million reported deaths and 14.83 million excess deaths globally. The development and approval of COVID-19 vaccines changed the pandemic’s trajectory, but disparities in vaccine supply and demand led to persistent global vaccine coverage inequalities. Vaccine hesitancy, particularly in low- and middle-income countries (LMICs), was a major driver of these disparities. To address vaccine hesitancy, the Alliance for Health Online (AHO) and the World Bank’s Mind, Behavior, and Development (eMBeD) Unit within Development Impact (DIME) developed a global research program leveraging social media and behavioral science. This program aimed to diagnose barriers to vaccine uptake, identify effective behaviorally informed communications, and develop public goods to support while introducing innovative approaches such as chatbot functionality and fostering partnerships with the private sector The program involved rapid diagnostics and message testing through online experiments, reaching 230,000 respondents across 25 countries, supporting an estimate of $700m in World Bank vaccine operations in these 25 countries. Additional activities included boosting resistance to misinformation with a WhatsApp chatbot game and capacity building for behaviorally informed social and behavioral change communications (SBCC) interventions. The project achieved significant impacts, including: • Increased vaccine confidence through behaviorally informed messaging and the identification of effective messengers. • 91% of World Bank project leads found the support effective. • 82% noted increased confidence in their work. • 64% reported improved intervention design. • 46% confirmed operational goal achievement. • 82% indicated a likelihood to apply behavioral insights in future projects. • Moderate stakeholder buy-in remains an area for growth. • All World Bank project leads expressed interest in exploring social media and behavioral science to enhance impact further. REDUCING COVID-19 VACCINE HESITANCY WITH SOCIAL AND BEHAVIOR CHANGE COMMUNICATION STRATEGIES ON SOCIAL MEDIA 5 The use of social media for data collection and testing KNOWLEDGE PRODUCTS proved to be fast and economical, with Facebook data showing comparable results to other survey 13 modalities. The project also highlighted the importance THE NUMBER OF RESEARCH PAPERS / of tailoring communication to distinct personas and WORKING PAPERS PRODUCED evolving situations. In summary, the AHO and World Bank’s research program demonstrated the potential of combining social media REACH and behavioral science to address vaccine hesitancy and promote public health. Building on these achievements, 213,562 THE NUMBER OF PEOPLE REACHED VIA the recommended next steps include: THE RESEARCH • Developing AI-powered tools to provide real-time, personalized guidance for Social and Behavioral Change Communication (SBCC) interventions, 7,277 improving the cultural relevance and effectiveness THE NUMBER OF PEOPLE WHO WATCH THE of health messaging. EVENT LIVE AND AFTERWARD • Implementing cost-effectiveness analyses to optimize the return on investment (ROI) for social and behavior change communications for health, 8,300 THE NUMBER OF VIEWS AND DOWNLOADS helping to prioritize strategies in low-resource OF PUBLICATIONS, KNOWLEDGE PRODUCTS settings. AND BLOGS • Utilizing digital platforms to deliver low-cost, personalized training for healthcare workers, enhancing continuous learning and access to up- to-date protocols. INFLUENCE POLICY • Deploying conversational tools, such as chatbots, MAKERS / OPERATIONS to assist community health workers in addressing 25 specific health concerns, including vaccine THE NUMBER OF COUNTRIES AND TERRITORIES confidence, in local languages via accessible THE RESEARCH/ PROGRAMS TOUCHED platforms like WhatsApp. • Leveraging digital platforms and mobile applications to support community members in managing non-communicable diseases (NCDs) through personalized health tips and behavior tracking, reducing the burden on healthcare systems. 1. Introduction INTRODUCTION 7 COVID-19 upended daily life and exacerbated global inequalities, prompting policymakers to seek innovative, evidence-based solutions to mitigate impacts of the pandemic by promoting vaccine acceptance and take-up. The pandemic resulted in about 6.88 million reported deaths globally by March 2023 (Johns Hopkins University, 2023) and 14.83 million excess deaths including those directly caused by the crisis and those resulting from indirect impacts like healthcare disruptions (Msemburi et al., 2023). The development and regulatory approval of COVID-19 vaccines changed the trajectory of the pandemic, but differences in both vaccine supply and demand resulted in global vaccine coverage inequalities that persisted throughout the pandemic. Vaccine hesitancy was an important driver of disparities in vaccination including in low- and middle-income countries (LMICs). Among LMICs, surveys found that Europe and Central Asia showed the highest vaccine hesitancy (58%), followed by the Middle East and North Africa (47%), East Asia and Pacific (26%), sub-Saharan Africa (15%), and Latin America and the Caribbean (8%) (Dayton Eberwein et al., 2023). Moreover, there is emerging evidence that hesitancy toward COVID-19 vaccines has spilled over to attitudes toward routine and general vaccines, posing new challenges to immunization programs (Lazarus et al., 2024; Lunz Trujillo et al., 2024). Given the prominence of vaccine hesitancy during the COVID-19 pandemic and its lingering effects, policymakers came to recognize that promoting greater vaccine confidence was essential to combatting the pandemic. Policymakers operating in LMICs often had limited evidence of what worked to promote vaccine confidence in their local contexts, but rapid data collection through social media offered an essential tool to closing these gaps in knowledge. Social media offers several comparative advantages in addressing vaccine hesitancy globally, primarily through its extensive reach, personalization, testing capabilities, and low costs (see Box 1). These advantages, which enable rapid online survey experimentation to promptly identify and address vaccine hesitancy drivers, is particularly valuable in fast-evolving environments like the COVID-19 pandemic. Pairing the use of social media with a behavioral science- based approach can further amplify the impact of each individually for promoting vaccine confidence and uptake. Behavioral science considers the various structural, psychological, and social factors that contribute to behaviors like vaccine take-up, following a process from diagnosing underlying drivers of the behavior to scaling effective solutions tailored to addressing barriers to issues like take-up. While social media is a tool that can facilitate data collection and testing, behavioral science can provide insights into which beliefs and characteristics are important to measure, the promising messages that should be tested to address drivers of vaccine hesitancy, and which messengers may be powerful to leverage in communication. INTRODUCTION 8 BOX 1. Advantages of Leveraging Social Media to Promote Vaccine Take Up Extensive Reach: There are over 5 billion smartphone users and over 3 billion Facebook daily users globally (Statista, 2023), and the rate of social media usage even in low- and middle- income countries continues to grow. Unlike traditional media, social media platforms allow for real-time dissemination of accurate information and the rapid countering of misinformation, enabling public health messages to spread quickly and widely. Testing Capabilities: Importantly, social media enables not only one-way communication, but many platforms also have features for testing what messages “work” and for whom. In the context of vaccine hesitancy, these capabilities allow for rapid feedback and experimental testing of different messages for promoting greater confidence in the effectiveness and safety of vaccines before governments engage in costly scaling activities. Personalization: Social media’s ability to target users with specific demographics and individual characteristics with tailored content can facilitate culturally and individually relevant messaging that better resonates with diverse audiences. Cost: Finally, social media platforms can be leveraged at a relatively low cost compared to some traditional media formats like television or in-person engagements, facilitating large reach and opportunities for rapid iteration in communication. On a global scale, paid social media ads generate 1.7 times the short-term return on investment (ROI) of television ads, despite brands investing approximately one-third less in them (Nielsen, 2022). INTRODUCTION 9 Recognizing the urgent need for evidence to inform policymaker decision making during the COVID-19 pandemic and the opportunities afforded by social media, the Alliance for Health Online (AHO) and the Mind, Behavior, and Development (eMBeD) Unit within Development Impact (DIME) developed a global research program to support governments with insights into how to promote vaccine confidence with behaviorally informed communications. With additional funding of over $1,000,000 from World Bank operations, the project leveraged the World Bank’s widespread access to policymakers and in-depth understanding of local contexts, eMBeD’s behavioral science expertise, the operational health experience of World Bank health teams, and social media expertise of AHO to develop a research program that could directly inform government initiatives to combat COVID-19 vaccine hesitancy with actionable insights derived from surveys and experiments conducted through social media. In the remaining sections of this report, we provide an overview of the project, lessons learned and impact from the project, and provide recommendations to inform future efforts to leverage the strengths of social media and behavioral science for development. 2. Project Overview PROJECT OVERVIEW 11 2.1 Global Research Program Objectives In January 2021, eMBeD launched a global research program with funding from the AHO of $2,000,000 to inform countries’ efforts to understand and tackle vaccine hesitancy by leveraging behavioral science and social media. The objectives of this research program were to: 1. Diagnose key structural and behavioral barriers to vaccine take-up to inform country-tailored social and behavioral change communications (SBCC); 2. Identify behaviorally informed communications that can promote intentions to get vaccinated; 3. Develop proof of concepts and public goods to support practitioners to leverage social media and behavioral science for public policy. With implementation lasting from January 2021 through March 2024, the research program involved close collaboration with Ministries of Health from over 25 countries. The project adopted an agile approach, evolving to respond to policymaker needs as their specific challenges from the pandemic shifted over time. 2.2 Activities and Collaborations Reflecting the evolving priorities of policymakers throughout the pandemic, the project undertook several key activities and in-depth country engagements: • Rapid Diagnostics and Message Testing with Online Experiments to Inform Country-Tailored SBCC COVID-19 Interventions • Boosting Resistance to Misinformation with a WhatsApp Chatbot • Capacity Building and Development of Public Goods • Proof of Concepts Applying Social Media Interventions Beyond COVID-19 A. Rapid Diagnostics and Message Testing with Online Experiments to Inform Country-Tailored SBCC COVID-19 Interventions As the core activity funded by the grant, the team led efforts to conduct rapid diagnostics and test behaviorally informed messages aimed at promoting vaccine uptake through social media platforms. Data was collected via an online social media survey that explored beliefs, attitudes, and PROJECT OVERVIEW 12 expectations regarding COVID-19 vaccination, trust in institutions, and potential access issues. To provide timely insights to policymakers during the pandemic, the team developed a flexible survey design and protocols that could be quickly adapted to individual country contexts based on input from health sector experts and government counterparts. By measuring behavioral characteristics alongside demographic information, the team identified distinct behavioral phenotypes, or personas, which offered a deeper understanding of vaccine hesitancy. This allowed for messages to be tailored to address the specific concerns of different population segments more effectively. The team conducted rapid online surveys across 23 countries and territories, reaching 230,000 respondents through Facebook. These insights provided crucial, real-time guidance to governments and international partners, such as UNICEF and WHO, helping improve social and behavior change communication (SBCC) efforts for COVID-19 vaccinations. Furthermore, four countries (Lebanon, Iraq, Jordan, and Tunisia) conducted two or more rounds of the diagnostics to adjust the messaging as more COVID-19 vaccines became available but as at the same, some misinformation began circulating. In addition, A/B testing of messages were conducted in Lebanon and Iraq to guide the governments into selecting the appropriate message framing to the different audiences. Table. COVID-19 Vaccine Hesitancy Country Diagnostics Middle East & Latin America & Europe & North Africa Caribbean Africa East Asia & Pacific Central Asia Lebanon 2 Honduras Chad Papua New Guinea North Macedonia Iraq 2 Beliza Cameroon Kosovo Jordan 2 Haiti The Republic of Congo Serbia Tunisia 3 Jamaica Zambia Ukraine The West Bank & Gaza Gambia Romania Libya Kuwait Saudi Arabia Dijbouti 2 3 Multiple rounds of surveys AB message experiment Pipeline - in progress Custom support Whatsapp misinformation chatbot PROJECT OVERVIEW 13 Based on the success of the social media data collection, the team also engaged in follow-on, special country engagements in several contexts. The purpose of these special country engagements was to extend findings derived from data collected through social media, building on insights by focusing on different populations and measurement methods. Importantly, these efforts highlight the complementarity of insights gathered from social media and data collected through other means—the quick insights derived from social media data was highly valuable in providing an initial evidence base to guide more resource intensive intervention efforts and data collection. Addressing Vaccine Hesitancy Among Primary Care Physicians in Romania In Romania, we applied our behavioral lens to understanding vaccine hesitancy among primary care physicians—a critical population for communicating about the safety and effectiveness of the vaccine. As a result of this engagement, the team provided insights to a professional association of Romanian primary care physicians on levels of vaccine hesitancy among primary care physicians themselves as well as the barriers physicians face in trying to communicate about COVID-19 vaccines with their patients (e.g., time constraints during appointments). Vaccine Promotion through Kingdom of Saudi Arabia Health App In the KSA, the team worked with government counterparts to test behaviorally informed communications delivered via the government’s communication channels, and provided technical assistance to the government to explore the impact of SBCC to real-world behaviors tracked through the government’s health app. Social Media vs. Phone Vaccine Hesitancy Surveys in Zambia Finally, in Zambia, the team collaborated with colleagues measuring vaccine hesitancy through other survey modalities (i.e., in-person surveys) to compare findings based on modality—finding that social media surveys and in-person surveys yielded similar conclusions, even if the social media participants were significantly different from the households who responded to the in-person survey. KEY INNOVATIONS Social media chatbot survey modality Conducting a survey via a chatbot implemented on a social media platform holds several advantages because it leverages real-time, conversational interaction to create a more engaging and personalized experience for respondents. Moreover, since the participant recruitment and survey were all conducted within one social media platform, it seemed more trustworthy than when being prompted to exit the platform to another website. We found that survey completion rates for this chatbot modality averaged 84%, higher than the 77% typically seen with self-administered surveys on other platforms. However, caution is needed in interpreting these completion rates, as they come from surveys PROJECT OVERVIEW 14 conducted in different countries and at different times. The project’s success in leveraging this method for implementing surveys was widely recognized at our institution as innovative, and continues to garner a large amount of interest from other teams to apply themselves across different domains. Quota-based dynamic ads for more population-representative participant recruitment Participants were recruited via Facebook using quota sampling to reflect the country’s population demographics based on census data (region, age, gender, and education level). Recruitment ads targeted specific demographic groups, inviting them to interact with a Facebook Messenger chatbot in their preferred language to complete the survey for a chance to win a lottery reward. To ensure proper representation, ad placements were updated in real time using tools like Virtual Lab (Rao et al., 2020) and RevealBot (RevealBot, 2024). If a subgroup was underrepresented due to higher attrition or non-response, the ad budget for that subgroup was automatically increased to meet sample quotas.1 This dynamic recruitment process facilitated automated quota sampling, ensuring balanced representation across demographic groups while reducing resource demands. By aligning the sample with national demographics, this approach strengthened the study’s relevance, enhanced its credibility with government stakeholders, and improved confidence in its findings. DELIVERABLES • 28 briefings to Ministries of Health summarizing insights into drivers of vaccine hesitancy and recommendations for behaviorally informed messages to promote vaccine take-up • 1 external brief summarizing regional findings from the rapid diagnostics and message testing • Integration of the diagnostic survey instrument into OpenWHO course “COVID-19 Behavioral Data Collection” which enrolled more than 13,000 learners. • 4 working papers reporting on (i) drivers of vaccine hesitancy in Papua New Guinea, (ii) correspondence in results when comparing social media survey, in-person surveys, and household phone survey methodologies in Zambia, (iii) an individual participant data meta- analysis summarizing results of survey experiments across 28 studies, and (iv) findings on COVID-19 misinformation chatbot games intervention in Jordan. • A scholarly journal article reporting results on promoting vaccine take-up in the Kingdom of Saudi Arabia. 1 For example, we recruited participants in Zambia targeting ads based on a combination of gender (2 groups), age group (4 groups), and region (3 groups). This combination created 24 strata for recruitment with each stratum reflecting a combination of these three characteristics. For instance, one set of ads targeted women aged 40-49 who lived in Lusaka while another set of ads targeted men aged 40-49 who lived in the Lusaka with quota targeting based on the latest available population data. During data collection for the study, if the proportion of participants from this demographic group in the study sample was lower than its proportion in the broader population, the recruitment ads were dynamically adjusted to spend more money to better target this population and increase its proportion in the study sample. PROJECT OVERVIEW 15 B. Boosting Resistance to Misinformation with a WhatsApp Chatbot Game As the first global pandemic in the digital age, the spread of mis- and disinformation about COVID-19 vaccines posed a major challenge to public health responses to the pandemic. The proliferation of mis- and disinformation threatened to erode the credibility of public institutions and limited their capacity to implement policies that enhance public health. For example, one survey found that 60% of Jordanians believed the COVID-19 virus was manmade and a substantial minority believed the vaccines caused infertility (Sallam et al., 2021). In our own survey in The Gambia, we found that over 74% of respondents heard that the COVID-19 vaccine was being introduced to reduce population size while 33% believed or were unsure whether the coronavirus was a myth. While misinformation represented an urgent global challenge, relatively little research had examined solutions in low- and middle-income countries. To help close this gap, our team designed and piloted a WhatsApp chatbot game that inoculates people against misinformation techniques. A version of the chatbot aiming to build resistance to COVID-19 misinformation was piloted first in The Gambia and then in Jordan with 2,851 participants with promising results. Similar to the diagnostic surveys and message testing, participants were recruited through Facebook ads, but then directed to WhatsApp to launch the game. The game taught lessons about common misinformation techniques like appealing to emotions, the use of fake experts, and manipulation of documents and images online. Through WhatsApp, the team tested the impact of an active, gamified as well as more passive but less time-intensive exposure to infographics, finding differences in impact on ability to detect misleading headlines and intentions to share misinformation. Given the encouraging findings from the pilots focused on COVID-19, the team was asked to expand the misinformation chatbot game to the domain of smoking misinformation targeting youths. Building on the success of the online launch, the team worked in partnership with the Royal Health Awareness Society in Jordan to integrate the misinformation chatbot into their healthy lifestyle programming delivered to students. The approach involved facilitators introducing the chatbot to students and students playing the game at home as part of homework since most students did not have a phone with them at school. A pilot trial in 13 schools with 324 students successfully launched with plans to disseminate findings to government counterparts and other partners during a roundtable session. Following the engagement in Jordan, the government has requested a new project with the World Bank that would include strengthening the Ministry of Health’s (MoH) capabilities in social and behavioral change (SBC) by building capacity in behavioral science, social media, and experimentation. PROJECT OVERVIEW 16 KEY INNOVATIONS Gamification delivered by a platform more accessible to low-resource contexts Prior research had found evidence that gamified inoculation against health misinformation could be effective, but existing games (e.g., GoViral!) required moderate-to-high data resources, which may discourage some users in low- and middle-income countries where mobile data can pose a significant expense and where connection speeds often remain slow. Our WhatsApp chatbot game was innovative by complementing these more data-intensive efforts with an engaging, effective inoculation game that leverages a data-light modality through mostly text-based messaging. Despite the shift to a simpler, less graphically intensive medium, we were still able to show encouraging, positive impacts on ability to recognize misleading (vs. credible) information and reduce intentions to share such misleading information. Multisectoral extensions to health domains beyond COVID As the response to COVID-19 lessened in urgency and priority among Ministries of Health, the team developed extensions of the game to tackle misinformation in other domains. The topics covered by the chatbot have grown beyond COVID-19, addressing misinformation around smoking among youth. By partnering with schools, the Ministry of Education, and other development partners, we tailored the chatbot to reach students and youth populations effectively. Specifically, in Jordan, the chatbot was integrated into smoking prevention and cessation curriculum to help students identify false claims about tobacco products. By working across sectors, including education, we created scalable, flexible tools that continue to promote positive behaviors in different public health and development contexts. DELIVERABLES • Two chatbot games • 1 reporting misinformation chatbot game results from Jordan C. Capacity Building and Generation of Public Goods As evidenced by the COVID-19 pandemic, delivery of behaviorally informed SBCC interventions is crucial for the success of important health initiatives but many governments lack capacity to design and deploy such interventions. Behaviorally informed communications, which leverage insights from behavioral science about effective messaging, allow agencies to design and disseminate health messages that effectively influence and sustain positive health behaviors. This approach is vital for preventing and controlling diseases, as it ensures that interventions are not only accurate but also resonate with the target audience’s motivations and barriers. By engaging communities through context-sensitive and tailored messages, public health agencies can foster trust, encourage PROJECT OVERVIEW 17 active participation, and ensure that health programs are well-received and sustained. However, many governments lack the skills and knowledge to implement effective behaviorally informed communications. In response to the need for capacity building, our team developed comprehensive curriculum and public goods to support governments to leverage social media for the delivery of behaviorally informed SBCC interventions. The team built on best practices and lessons learned from the COVID-19 project to develop step-by-step guidance to apply a behavioral lens and develop effective SBCC to promote better health, with a focus on opportunities afforded by digital platforms and tools. Capacity Building Curriculum Lessons on opportunities to incorporate digital tools into SBCC campaigns including data collection and message testing were integrated into the team’s standard behavioral science introductory course material. This new material was piloted in Jordan as part of a capacity building workshop focused on improving tobacco control interventions with behavioral science. Feedback on the workshop was positive and the team continues to collaborate with the Ministry of Health on this topic, exploring continued needs for additional capacity building. In particular, the government has indicated strong demand for enhancing SBCC effectiveness and leveraging innovative digital tools like AI chatbots to better address health information and service needs among the Jordanian public. Social Media Methodology Note A practical methodology note was developed to summarize best practices and implementation guidance for how to leverage social media for data collection and experimentation. The note addressed key questions to assess whether social media is the right medium for data collection for a given project and highlighted strengths and weaknesses. The note included a “Frequently Asked Questions” section based on inquiries that other project teams posed about opportunities with social media, touching on representativeness among other issues. The target audience of this note is other teams at the World Bank who could benefit from incorporating social media into their own research and activities. Toolkits and Guides The team worked closely with the World Bank Health team in Yemen, along with WHO and UNICEF to develop a guide tailored to the context of developing SBCC campaigns in the country. While this activity was fully funded by the World Bank project, the team leveraged the development of the first behaviorally informed SBCC toolkit to develop a more robust and comprehensive one that would be global outlining practical steps and best practices for designing and evaluating SBCC campaigns. The final toolkit is designed for practitioners with limited expertise in behavioral science, aiming for a wide audience. PROJECT OVERVIEW 18 Public Event The team organized an event on Social Media and Behavioral Science in Development on April 1st, 2024, called “Influence and Impact: The Power of Social Media and Behavioral Science in Development”. The event attracted around 475 participants both online and in person and consisted of panel discussions with leading experts such as Jay van Bavel, Kang-Xing “KX” Jin, as well as World Bank and governmental experts. Furthermore, six practitioners delivered a ‘lightning talk’ showcasing innovative, scalable, and impactful policy solutions that use behavioral science principles, social media, and cutting-edge digital technologies across various development domains to demonstrate how these tools can effectively tackle big policy challenges for impact. Since then, the event page has been viewed over 6,800 times. Global Publication on Behavioral Science in Public Health The team partnered with WHO to publish the third volume of Behavioral Science Around the World: Public Health. The publication was launched online during the World Health Assembly 2025 meetings. It explores how behavior influences health outcomes and highlights the historical context and current focus on behavioral science in public health and examines the models used by over 40 behavioral insights units in 26 countries to address health challenges. The publication has received over 3,000 downloads the first three months. DELIVERABLES • Methodology Note • SBCC Toolkit (forthcoming) • Social Media and Development Dissemination Event • Behavioral Science around the World: Public Health PROJECT OVERVIEW 19 D. Proof of Concepts Applying Social Media Interventions Beyond COVID-19 In Madagascar, we extended our experiences leveraging social media platforms and chatbots to broader development contexts, continuing our efforts from the COVID-19 social media interventions to promote prosocial behaviors. Building on our expertise in using behaviorally- informed interventions on social media, this initiative sought to test how we can nudge the privileged toward prosocial behaviors and cooperation for broader societal good. Social media was identified as an ideal setting for the intervention because the project specifically targeted the privileged in Madagascar, a group overrepresented on social media platforms and a group often associated with ‘elite capture’. Elite capture reflects unwillingness to share financial gains with others, and has exacerbated poverty and inequality in the country. Prosocial interventions with social media recruitment We designed a trust game with over 10,000 participants, who believed they were interacting with other Malagasy Facebook users, but were actually engaging with a programmed chatbot. The experiment tested various behavioral interventions, including providing information about positive descriptive norms for sharing with others, reminders of cultural values for harmony and kinship in Malagasy culture, self-affirmation, and perspective-getting narratives that related a story of a low- income Malagasy person benefitting from cash assistance and investing the money in productive uses. Our goal was to investigate how these interventions could increase prosocial behaviors—such as sharing resources and promoting cooperation—among privileged groups by addressing various social and psychological mechanisms that have increased cooperation in other contexts. PROJECT OVERVIEW 20 Key findings revealed that descriptive norms messages and self-affirmation significantly increased prosocial behaviors, encouraging participants to share more resources. By drawing on those lessons, we were able to help explore solutions for poverty and equity issues in Madagascar, fostering a more inclusive environment through behaviorally-informed digital interventions. The study underscored the potential for scalable, automated digital tools to influence elite behavior and mitigate elite capture. DELIVERABLES • One trust game chatbot with interventions • 1 working paper reporting the findings from the interventions to encourage cooperation and prosocial behaviors in Madagascar (forthcoming) 3. Impacts IMPACT 22 As the team implemented Just-In-Time technical support to the countries and World Bank teams, notable impacts were achieved. 3.1 Financial Impacts Table 1. AHO EFO Key Achievements and Reach Overview KNOWLEDGE PRODUCTS 13 THE NUMBER OF RESEARCH PAPERS / WORKING PAPERS PRODUCED REACH 213,562 THE NUMBER OF PEOPLE REACHED VIA THE RESEARCH 7,277 THE NUMBER OF PEOPLE WHO WATCH THE EVENT LIVE AND AFTERWARD 8,300 THE NUMBER OF VIEWS AND DOWNLOADS OF PUBLICATIONS, KNOWLEDGE PRODUCTS AND BLOGS INFLUENCE POLICY MAKERS/OPERATIONS 25 THE NUMBER OF COUNTRIES AND TERRITORIES THE RESEARCH/ PROGRAMS TOUCHED IMPACT 23 3.2 Operational Impacts Knowledge outputs have been accessed and utilized by policymakers and the public who require expert-led information on responding to the pandemic. Feedback from 14 World Bank project leads was gathered through an online survey designed to evaluate the behavioral insights support’s effectiveness provided during the COVID-19 pandemic. The survey included questions about the perceived impact on decision-making, intervention design, stakeholder engagement, and operational goals, and the behavioral insights’ applicability to future projects.  The World Bank project leads’ feedback has been overwhelmingly positive, with 91% finding the assistance effective and impactful. Also, 82% reported increased confidence in their decision- making due to the support, and 64% felt it improved the design of their interventions. There is, however, room for improvement in stakeholder buy-in, with 36% of respondents indicated that  the support influenced changes in stakeholder behaviors and decision making (e.g., increased engagement, adoption of new practices), whereas 45% indicating a moderate level of agreement.  Furthermore, 46% of respondents confirmed that the support helped them achieve their operational goals, and 27% observed improvements in achieving project goals due to the behavioral science support (see examples listed in the next section).  Encouragingly, 82% of participants indicated they are likely to apply the behavioral insights gained to other projects, and 55% believe social media plays an important role in enhancing the impact of their operations. All respondents expressed interest in further exploring how social media and behavioral science can enhance their operations’ effectiveness. Some examples of specific feedback: • Ukraine: The project was well-received and aimed at supporting the government’s COVID-19 vaccine rollout. However, with the onset of the Russian invasion, COVID-19 priorities shifted, and the initiative was deprioritized. • Honduras: The engagement led to better targeting of messages, identification of effective communication channels, and profiling of beneficiaries to improve outreach. • Papua New Guinea (PNG): Results highlighted the critical role of local health workers in addressing vaccine hesitancy. This insight prompted the national COVID-19 vaccine task force to shift its communication campaign focus toward training local health workers in addressing community concerns face-to-face. Previously, the campaign primarily targeted reducing misinformation on social media. • Zambia: This work had significant policy impact, with the Ministry of Health adopting a new “opt-out” approach to COVID-19 vaccination, replacing the original “opt-in” system to increase uptake. Additionally, there were adjustments to the consent process, moving from requiring written to verbal permission for vaccination, improving accessibility. IMPACT 24 3.3 Interventions’ Impacts: Behaviorally-Informed Messaging Vaccine confidence can be increased with behaviorally-informed messaging A meta-analysis of our message experiments revealed a notable increase in participants’ COVID-19 vaccination intentions following exposure to short messages. MAIN EFFECT Choosing the right messenger is as important as choosing the right message Our surveys identified a clear preference among participants for health experts, particularly doctors, as sources of COVID-19 vaccination information. Our meta-analysis further demonstrated that incorporating credible information sources, including experts, celebrities, and religious leaders, significantly enhanced vaccination intentions. Understanding distinct personas and tailoring communication to their unique concerns can strengthen the impact of messaging Customizing messages to address individuals’ vaccine concerns has proven effective in heightening vaccination intentions. Notably, people with safety concerns respond especially strongly to messaging that emphasizes that experts endorse the vaccines as safe and effective. IMPACT 25 SAFETY CONCERNS At the same time, people who have already had COVID (and may be less concerned about re-infection) are more persuaded by messages highlighting the prosocial impact of vaccination. HAD COVID ALREADY HAD IMPACT 26 Our surveys unveiled distinct challenges faced by internally displaced people (IDP), refugees, and non-nationals, including obstacles related to vaccine access and information gaps. Communication should also be tailored to the dynamics of an evolving situation As the scientific understanding of the COVID-19 threat and vaccine efficacy evolved over time, effective communication strategies must also adapt to the changing landscape and not remain static. This necessitates agility in message development and testing. For instance, we successfully developed and tested new message frames that center on endemic scenarios and COVID-19 variants, proving their effectiveness. 3.4 Intervention Impacts: WhatsApp Misinformation Chatbot Games Inoculation against misleading techniques can help increase resistance to misinformation In our misinformation inoculation game experiment conducted in Jordan, participants who were unvaccinated and resistant to vaccination displayed poorer performance in misinformation detection compared to their vaccinated counterparts. Nevertheless, the inoculation game significantly enhanced the performance of the treatment group, bringing their misinformation detection capabilities closer to those who had received vaccinations. IMPACT 27 4. Lessons Learned LESSONS LEARNED 29 4.1 Methods and Technologies Using social media (e.g., Facebook ads) can help collect data and test interventions fast and economically On average, our data collection through Facebook ads spanned 21 days (SD = 9.4) and garnered approximately 6,500 responses (SD = 4,744). The cost per completed response was US$ 1.36 (M = 2.34, SD = 2.22). Using social media also helped us collect data from participants who are difficult to reach otherwise. For example, in Nigeria, SMS surveys cost approximately US$ 7 per survey, and panel surveys around US$ 2.60 per survey—both higher than the Facebook response cost average across our study countries. Despite sample variations, Facebook data exhibited comparable results to phone surveys (Papua New Guinea), in-person surveys (Zambia), and UNICEF’s U-Report SMS messages (Zambia). However, using Facebook ads presents several challenges, such as frequent policy changes, the risk of ad rejections, and potential page blocks. Also, survey platforms might experience technical errors that can disrupt data collection efforts. To prevent potential problems, the data collection team must ensure that all ads and page content adheres to Facebook’s guidelines, while actively monitoring and addressing any issues that arise, including managing user comments promptly. Chatbots can help make interventions conversational and interactive Chatbots offer a conversational and interactive user experience, distinguishing them from traditional survey platforms. While these rates should be interpreted cautiously, as the surveys were conducted in different countries, survey completion rates for this Messenger chatbot averaged 84%, surpassing the 77% typical for self-administered survey platforms and suggesting higher engagement levels. However, data collection through chatbots comes with several caveats. Compliance with Facebook policies, including restrictions on personally identifiable information, is imperative. Chatbots are limited in their ability to pose certain types of questions, such as matrix questions or sliders, commonly employed in traditional survey platforms, although the technology might advance and improve the user experience in the future. Additionally, there are constraints on presenting media (e.g., images, videos) to participants. Given these limitations and associated risks, researchers must carefully select between chatbots and other survey platforms. Connectivity is the key to successful digital interventions In general, the success of digital interventions depends on connectivity, such as Internet penetration, smartphone penetration, and the number of active social media uses. Collecting data in regions with lower connectivity demands more resources in terms of time and cost. For example, in areas with low Internet penetration and limited social media users, such as Cameroon (Internet users: 34%, social media users: 16%), the process took US$6.72 and 17 days. Similarly, in Congo-Brazzaville (Internet users: 32%, social media users: 15%), it required US$7.67 and 18 days, LESSONS LEARNED 30 and in the Gambia (Internet users: 51%, social media users: 18%), the cost and time were US$8.75 and 32 days, respectively. Conversely, in regions with high Internet penetration and substantial social media users, such as Libya (Internet users: 46.2%, social media users: 86%), the process took US$0.42 and 14 days. In Jordan (Internet users: 66%, social media users: 61%), it required US$0.58 and 12 days, in Lebanon (Internet users: 89%, social media users: 75%), US$0.58 and 13 days, and in Serbia (Internet users: 79%, social media users: 52%), US$0.67 and 21 days. Still, even in countries with low digital connectivity, social media still provided unique opportunities to engage with the target audience. For instance, in Madagascar, with the strong correlation between social media usage and wealth indicators in the country, Facebook ads allowed the recruitment of over 10,000 participants in just four days, reaching more privileged groups for interventions encouraging cooperation and prosocial behaviors. Similarly, in Yemen, despite infrastructure challenges, Facebook was effectively used to test message effectiveness for public health campaigns before expanding to other media. 4.2 Operations Governments, including public health authorities, are increasingly institutionalizing behavioral science functions to enhance their capacity for preventing, responding to, and recovering from health emergencies A growing number of governments have been integrating behavioral science insights into public policy, particularly within the realm of public health. The inception of the first behavioral insights unit in the UK in 2010 marked the beginning of this trend, with over 200 public organizations adopting behavioral insights by 2018. In the context of public health, our survey with the World Health Organization identified 35 behavioral insight units operating in 23 countries, focusing on public health topics. Institutionalizing behavioral science requires strong leadership and organizational buy-in, a thorough assessment of current capabilities and gaps, and ongoing capacity-building efforts, among other critical steps. Countries have highlighted the need for capacity building in social listening and analysis to identify and address issues undermining vaccine confidence, especially in light of the rise in misinformation and disinformation during the COVID-19 pandemic. Governments recognize the importance of leveraging digital tools to reach the different targeted audiences Tajikistan and Uzbekistan used mass text messaging for COVID-19 response. Notably, Jordan has actively pursued the development of digital capacities for social listening, monitoring, intervention testing, and risk communication, as evidenced by a dedicated capacity-building workshop. However, it can be prohibitively expensive for governments to subscribe to necessary programs and applications, hire skilled personnel, or adequately train employees in using digital tools. Collaborating with external LESSONS LEARNED 31 units or third-party support may be an option, but it requires governments to possess sufficient skills and knowledge to effectively guide these partners. Information literacy and trust in public health systems are critical components of pandemic preparedness To effectively combat the proliferation of health misinformation in future pandemics, there is a critical need to bolster individuals’ information literacy and confidence in public health systems. Various misinformation experiments, including our own, have demonstrated that proactively cultivating individuals’ literacy skills (prebunking) proves more effective than the reactive approach of fact- checking each instance of misinformation (debunking) in safeguarding individuals from misinformation. Leveraging administrative data can enhance the identification of intervention effectiveness Many behavioral science experiments in the COVID-19 pandemic response only measured behavioral, often without observing or tracking actual behaviors. Utilizing government-administered data, such as vaccination status, offers a unique opportunity to assess the real impact of interventions on tangible vaccination behaviors. For instance, the Kingdom of Saudi Arabia could analyze the effectiveness of their COVID-19 messages by examining their influence on adult vaccination rates and the vaccination of children by their parents. Collaborating with governments and international organizations enhances the efficiency and effectiveness of programs Given the extensive efforts of stakeholders, including governments and international organizations, in addressing public health challenges, close collaboration with them becomes instrumental in bridging gaps and maximizing the application of behavioral science in public health. For instance, our team has collaborated with the World Health Organization on noncommunicable diseases and with UNICEF on maternal and child nutrition during conflicts in Yemen. Through these partnerships, we have refined programs, making them more efficient and effective. 5. Recommenda- tions for Future Projects RECOMMENDATIONS FOR FUTURE PROJECTS 33 Building on the lessons learned and impacts, we discuss key recommendations for future projects across three areas: tools, interventions, and operations. By focusing on enhancing digital tools, refining intervention strategies, and addressing operational challenges, these recommendations aim to strengthen the effectiveness and impact of intervention efforts, particularly in low-resource settings. 5.1 Tools Investment in Digital Tools Continuous investment in digital tools like social media listening platforms, online data collection methods, and digital interventions is crucial as technology rapidly evolves. Many governments lack the skills and capacity to manage dynamic online campaigns effectively, including message testing, content performance tracking, and adjusting strategies in real time. Additionally, teams often lack a foundational understanding of behavioral science, which is essential for impactful messaging. Future projects should invest in updating digital tools and training teams to use them efficiently. Governments may also need to collaborate with external service providers to bridge capacity gaps and stay agile in their communication efforts. Administrative Data Administrative data is critical for tracking actual health behaviors and providing more accurate measures of intervention impacts than self-reported intentions. However, challenges such as the lack of digitized and integrated local health systems, and limited staff training, hindered its full use. Future projects should prioritize partnerships with governments to improve access to administrative data by promoting digitization and integration of health systems, along with staff training in digital tools to enhance service delivery and policy-making. 5.2 Interventions Adaptive Risk Communication Strategies Communication strategies must evolve with scientific understanding. Future projects should prioritize flexibility in message development and testing to ensure relevance and alignment with the latest health guidance. RECOMMENDATIONS FOR FUTURE PROJECTS 34 Misinformation Resistance Preemptive approaches to misinformation (e.g., prebunking, fact-checking, and public corrections) are essential. Future projects should include misinformation resistance components to enhance public resilience and information literacy. 5.3 Operations Needs Assessment Governments should identify gaps in skills and resources. Governments can decide to address them through in-house expertise or external partnerships, once identified. Local Partnerships Strengthening partnerships with local health authorities and organizations is essential for ensuring interventions are relevant and impactful. Future projects can partner with local organizations for interventions’ quality control and adaptations. Digital Solutions and Continuous Training Governments should be encouraged to develop and adopt digital solutions for monitoring, testing interventions, and communicating with the public to meet health needs effectively. Regular updates and training on digital tools and platforms are necessary to maintain effective digital interventions. Institutional Capacities As governments increasingly institutionalize behavioral science, capacity-building is crucial. Governments should assess their needs in areas like behavioral science, social media monitoring, online research, and digital interventions. Steps can be taken to enhance skills, engage external experts, or build in-house capabilities. Implementation Challenges Issues with social media and digital platforms, such as policy changes and platform errors, highlight the need for contingency plans. Future projects should establish protocols for compliance, expert reviews of public materials, monitoring responses, and developing alternative data collection strategies. 6. Next Steps RECOMMENDATIONS FOR FUTURE PROJECTS 36 We highlight immediate and actionable next steps to build upon the project’s successes and further capitalize on the comparative advantages of social media and digital tools (as outlined in Box 1) to address pressing global health challenges. These steps focus on harnessing AI for personalized interventions, optimizing cost-effectiveness through ROI modeling, and using advanced technologies to improve the training of healthcare workers and the health behaviors of community members. 6.1 SBCC intervention guidance using AI Building on the unique strengths of digital tools for personalization, we are exploring developing AI- powered guidance for Social and Behavioral Change Communication (SBCC) interventions, drawing from best practices and existing social and behavioral science literature. By leveraging AI’s ability to analyze large datasets and behavioral patterns, combined with insights from proven methodologies, we could offer real-time feedback and create tailored interventions that more effectively address vaccine hesitancy or other health challenges. This approach would allow us to support governments with identifying promising interventions and support personalization of messaging, ensuring that interventions are both culturally relevant and behaviorally informed, while being grounded in evidence-based practices. 6.2 Cost-effectiveness (ROI/ Marketing mix modelling) Recognizing the potential benefits in cost savings through use of social media for public outreach, we might consider implementing marketing mix modeling to explore the return on investment (ROI) with public health authorities for various types of health interventions, particularly those delivered via social media. By using cost-effectiveness analysis, we could better understand which strategies offer the most value for money. This would help us optimize investments in digital tools, media and channels, message testing, and behavioral interventions, with potential applications in low-resource settings. Integrating relative cost-effectiveness analyses enables comprehensive comparisons between investments in social and behavioral change (SBC) interventions and other pharmaceutical (e.g., vaccine supply) and non-pharmaceutical interventions (e.g., physical distancing, mask-wearing, lockdowns). For instance, the WHO global investment case for tobacco cessation evaluates both population- level interventions (e.g., mCessation, national toll-free quitlines, brief advice) and pharmacological treatments (e.g., nicotine replacement therapy, Bupropion, Varenicline), providing insights into their respective impacts and cost-effectiveness. Similarly, the WHO Framework Convention on Tobacco Control’s (FCTC) Investment Case for Tobacco Control in Jordan compares the return on investment of various tobacco control measures, such as cigarette taxes, warning labels, plain packaging, advertising and promotion bans, and mass media campaigns. Applying a similar framework to vaccination efforts RECOMMENDATIONS FOR FUTURE PROJECTS 37 would involve analyzing the incremental impact of each dollar spent on SBC initiatives—including social media campaigns—relative to expenditures on vaccine supply. This holistic assessment ensures that resources are allocated efficiently to maximize vaccination rates and overall public health outcomes. 6.3 Using technologies to improve training of healthcare workers Healthcare workers’ knowledge gaps, refreshers training, and use of device The need for more frequent training and feedback is a common barrier to health worker performance in low- and middle-income countries, but the costs of in-person training can be prohibitive for governments. Moreover, trainings are typically delivered in group settings that limit the ability to tailor curriculum and lessons to individual learner needs. Use of digital platforms offers opportunities to develop low-cost, personalized training for healthcare workers on the topics where they need the most review and practice. We can explore conducting needs assessments to identify specific knowledge gaps among healthcare workers. With these insights, we could develop refresher training programs that integrate mobile devices or tablets into the learning process. This could help facilitate continuous learning by providing access to up-to-date protocols, guidelines, and interactive tools for communicating with patients and the public, managing misinformation, and improving patient care. Conversational tools to help healthcare workers (CHWs) address the specific concerns We could consider deploying conversational chatbots that assist community health workers (CHWs) in addressing specific health concerns, such as vaccine confidence, smoking prevention and cessation, or non-communicable disease prevention. These chatbots could be explored in local languages and made easily accessible via smartphones, Facebook, and/or WhatsApp, enabling CHWs to provide timely and accurate information to their communities. Moreover, an additional benefit of WhatsApp compared to SMS, is that audio and video messages can easily be sent to users. Leveraging audio and video, we can develop content that can be referenced by CHWs and played for beneficiaries directly during consultations, even if the target population has low literacy levels. Such tools could help standardize the educational counselling and other services implemented by CHWs. 6.4 Using technologies to improve community members’ prevention and management of non-communicable diseases (NCD) We can explore leveraging digital platforms and mobile applications to disseminate information about non-communicable diseases (NCDs) to community members, given countries’ strong interests in addressing NCDs. In addition, we could use social media, alongside traditional and new media, to shape attitudes, perceptions, social norms, and behaviors through SBCC campaigns. By engaging influencers and community leaders, we could amplify health messages and create a more impactful reach. Interactive tools such as apps, SMS-based services, and social media platforms could provide personalized health tips, track behaviors, and improve adherence to treatment plans, empowering individuals to manage their health more proactively while reducing the burden on healthcare systems. 6.5 Strengthening capacity in SBC with policymakers and government officials  To sustain and expand the impact of behavioral interventions, we are exploring opportunities to strengthen partnerships with policymakers and government officials, such as Ministries of Health, to build capacity in SBC planning, design, implementation, and evaluation, with a particular focus on social media and digital interventions. For example, ongoing collaboration with the Ministry of Health in Jordan serves as a model for integrating digital tools and SBCC interventions to address public health issues such as vaccine hesitancy and youth smoking. Similar partnerships are currently being explored with countries in MNA, AFR, and SAR to institutionalize SBC capacity-building efforts, supported by additional investments and knowledge-sharing within the World Bank. This approach could enhance the effectiveness of SBC spending, particularly for social media campaigns, by enabling policymakers to achieve more efficient and impactful outcomes. Scaling this model of collaboration would further elevate the role of behavioral science in public health interventions globally. 7. Conclusion CONCLUSIONS 40 In conclusion, the project demonstrated the powerful impact of integrating social media and behavioral science in addressing vaccine hesitancy, offering a scalable model for public health communication. The initiative’s success highlights the critical need for ongoing investment in evidence-based research, capacity-building, and the development of public goods to support practitioners globally. As we look ahead, the role of AI presents exciting opportunities to enhance rapid diagnostics, optimize training for health workers, and more effectively counter misinformation, making SBCC interventions even more targeted and responsive. We extend our sincere gratitude to all collaborators—AHO, World Bank Health teams, Ministries of Health from over 25 countries, UNICEF, and WHO—whose contributions were instrumental in achieving these outcomes. REDUCING COVID-19 VACCINE HESITANCY WITH SOCIAL AND BEHAVIOR CHANGE COMMUNICATION STRATEGIES ON SOCIAL MEDIA 41 Annex 1. Summary of Published Knowledge Products and Public Goods Title  Country Publication Outlet  REPORT, GUIDE, AND PEER-REVIEWED JOURNAL ARTICLES World Bank Report Behavioral Science Around the World Vol. 3. Health  Global (WHO co-publication) Behavioral science and COVID-19: An interactive solutions guide for better Global World Bank Report policy design Behaviorally Informed Messages Increase COVID-19 Vaccination Intentions: World Bank Policy Research Global Insights from a Global Meta-Analysis Working Paper Find the Fake: Boosting Resistance to Health Misinformation in Jordan with a World Bank Policy Research Jordan WhatsApp Chatbot Game Working Paper Behavioral insights for COVID-19 vaccine uptake in the Caribbean: Rapid Belize, Haiti, World Bank Report surveys from Belize, Haiti, and Jamaica  Jamaica Yemen behavioral campaign diagnostics & design guide Yemen WHO internal publication World Health Organization Using social media surveys and interventions to address vaccine hesitancy in Saudi Arabia (WHO),  Eastern Mediterranean Saudi Arabia Health Journal (EMHJ)  Intra-household dynamics and attitudes toward vaccines: Experimental and World Bank Policy Research Zambia survey evidence from Zambia Working Paper Addressing vaccine hesitancy in developing countries: Survey and Papua New Guinea PLoS One experimental evidence WORLD BANK BLOGS Vaccine hesitancy: 10 lessons from chatbotting about COVID-19 in 17 countries Global World Bank Blogs Being behaviorally savvy in vaccine communication Global World Bank Blogs Preparing for the next pandemic? Here are lessons learned from the Global World Bank Blogs last two years How to communicate about the monkeypox virus: Global World Bank Blogs Five key lessons from COVID-19 Jordan capacity building workshop blog post Jordan World Bank Blogs Harnessing the power of behavioral science and Global World Bank Blogs digital social tools for development How are governments integrating behavioral science Global World Bank Blogs in public health?