Policy Research Working Paper 10933 Find the Fake Boosting Resistance to Health Misinformation in Jordan with a WhatsApp Chatbot Game Michelle Dugas Daniel Pinzón JungKyu Rhys Lim Renos Vakis Zeina Afif Takahiro Hasumi Diya Elfadel Poverty and Equity Global Practice A verified reproducibility package for this paper is Health, Nutrition and Population available at http://reproducibility.worldbank.org, Global Practice & click here for direct access. Development Impact Group September 2024 Policy Research Working Paper 10933 Abstract The proliferation of misinformation and disinformation impact of the intervention, the study assesses two main threatens to erode the credibility of public institutions outcomes: (1) ability to discern accurately headlines using and limit their capacity to implement policies that enhance misinformation techniques and headlines that do not use public well-being. While health misinformation represents misinformation techniques, and (2) discernment in shar- an urgent global challenge, relatively little research has exam- ing the two types of headlines. Compared to the placebo ined solutions in low- and middle-income countries. This group, the comprehensive game significantly improved dis- study experimentally tests the impact of a novel WhatsApp cernment of misinformation and reduced the likelihood of chatbot game pre-bunking inoculation intervention in sharing misleading headlines. A brief version of the game Jordan to boost capacity to identify common misinfor- yielded weaker effects on discernment of misinformation, mation techniques and reduce the likelihood of sharing but similarly reduced intentions to share misleading head- misleading headlines with others, effectively “inoculating” lines. In contrast, exposure to infographics teaching similar them against health misinformation. A sample of 2,851 techniques showed no significant impacts on discernment participants was recruited online and randomly assigned of misinformation, and marginal effects on intention to to five study arms: (1) comprehensive game-based inocu- share misleading headlines. These findings suggest that lation, (2) brief game-based inoculation that highlighted games can effectively inoculate the public against misin- examples of only misinformation, (3) infographics-based formation in the context of a middle-income country in inoculation, (4) exposure to placebo infographics unrelated the short term. Future research is needed to explore the to misinformation, and (5) pure control. To evaluate the boundary conditions of the findings. This paper is a product of the Poverty and Equity Global Practice, the Health, Nutrition and Population Global Practice and the Development Impact Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at ____________@ worldbank.org. A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click here for direct access. RESEA CY LI R CH PO TRANSPARENT ANALYSIS S W R R E O KI P NG PA The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Find the Fake: Boosting Resistance to Health Misinformation in Jordan with a WhatsApp Chatbot Game Michelle Dugas, World Bank, mdugas@worldbank.org, Daniel Pinzón, World Bank, dpinzonhernandez@worldbank.org, JungKyu Rhys Lim, World Bank, rhyslim@worldbank.org Renos Vakis, World Bank, rvakis@worldbank.org Zeina Afif, World Bank, zafif@worldbankgroup.org Takahiro Hasumi, World Bank, thasumi@worldbank.org Diya Elfadel, World Bank, delfadel@worldbank.org JEL classification: D.90 Micro-based behavioral-economics, General Acknowledgments From the World Bank Team, the study was led by Michelle Dugas, Daniel Alejandro Pinzón Hernández, JungKyu Rhys Lim, Zeina Afif, Renos Vakis at the Mind, Behavior & Development (eMBeD) Unit in the World Bank’s Poverty and Equity Global Practice and Takahiro Hasumi and Diya Elfadel at the World Bank’s Health, Nutrition and Population Global Practice. Additional contributing team members include Ellen Moscoe, Mohamad Chatila, Jana Kontar, and Haena Kim. The authors express their gratitude to all who contributed to the implementation of this project, especially country government partners. The team is grateful to Victor Orozco (Senior Economist, DIME2), Erwin van Knippenberg (Economist, EMNPV), and Samuel Fraiberger (Data Scientist, DIME0), for their valuable comments as peer reviewers. This study was partially funded by the Advancing Health Online (AHO) Initiative, a fiscally sponsored project of Global Impact and funded by Meta and MSD to advance public understanding of how social media and behavioral sciences can be leveraged to improve the health of communities around the world. The findings, conclusions, and recommendations expressed in this article are those of the authors and do not necessarily reflect the views of the collaborators. 2 Find the Fake: Boosting Resistance to Misinformation in Jordan with a WhatsApp Chatbot Game With the increasing use of social media, misinformation has proliferated and traveled farther, wider, and faster than ever before, impacting wide-ranging topics including climate change, politics, and vaccines (Vosoughi et al., 2018). Accordingly, misinformation and disinformation are considered among the 32 most severe global risks 1 (World Economic Forum [WEF], 2023), with direct costs estimated at about US$ 78 billion across different countries and sectors (Cavazos, 2019). During the coronavirus disease (COVID-19) pandemic, misinformation emerged as a particularly salient threat to public health given its association with vaccine hesitancy (Loomba et al., 2021; Roozenbeek et al., 2020; Zhao et al., 2023). To tackle widespread misinformation, researchers have developed several psychological solutions: fact-checks or corrections of specific misinformation claims (i.e., debunking), priming to be cautious of accuracy (i.e., accuracy primes or nudges), and psychological inoculation against misinformation techniques (i.e., prebunking) (American Psychological Association, 2023 [APA]; Lim et al., 2022; Roozenbeek & van der Linden, 2022; Zhao et al., 2023). Evidence suggests that all these solutions can be effective at reducing the spread of misinformation (Lim et al., 2022; Pennycook et al., 2023; Zhao et al., 2023), but teaching common misinformation techniques as a form of inoculation holds the advantage of being both proactive and effective in combatting misinformation. In contrast, although debunking showed better non-misinformation credibility assessment and sharing intentions than prebunking in some research (Lu et al., 2023), debunking requires significant resources to detect and respond to specific claims and there 1 According to World Economic Forum (2023), “Global risk” is defined as the possibility of the occurrence of an event or condition which, if it occurs, would negatively impact a significant proportion of global GDP, population or natural resources. 3 remains significant debate about potential backfire effects of making misinformation claims more salient and memorable (APA, 2023; Ecker et al., 2022; Nan et al., 2022). In addition, effectiveness of accuracy primes may be limited by challenges in reaching all platforms where people are exposed to misinformation at scale (Roozenbeek & van der Linden, 2022; Traberg et al., 2022; van Bavel et al., 2021). While there is a growing body of evidence supporting the effectiveness of inoculation against misinformation, this research remains largely limited to Western, educated, industrialized, rich, and democratic (WEIRD) societies (Henrich et al., 2010; Nan et al., 2022) with a few notable exceptions (e.g., Arechar et al., 2023; Athey et al., 2022; Bilo-Thomas et al., 2021; Harjani et al., 2023). However, many threats from misinformation are found in other regions, including the Middle East. For example, a survey of Jordanian Arab residents found that over 60% believed that COVID-19 is a man-made virus, and about 25% believed that the COVID-19 vaccine is a way to implant microchips into people and that vaccines cause infertility, exacerbating vaccine hesitancy (Sallam et al., 2020, 2021). Such misinformation has been circulated extensively through social media (Al-Jalabneh & Safori, 2020; Alkhwaldeh & Emam, 2020; Habes et al., 2023), yet solutions for this cultural context are underexplored. To address this knowledge gap, we test the effectiveness of a chatbot-based inoculation game with comprehensive and brief versions (i.e., active inoculation), infographics (i.e., passive inoculation), and a placebo compared to no inoculation through a randomized experiment delivered by WhatsApp in Jordan (N = 2,851). Specifically, participants learn common misinformation techniques through chatbot-based games or infographics. Then, we measure whether participants can accurately discern headlines using misinformation tactics and their 4 likelihood of sharing misinformation. By doing so, we reveal which inoculation interventions could be more effective to tackle misinformation in this unique context. Literature Review How Misinformation Can Be Stopped from Spreading Like vaccination against a virus, inoculation theory poses that people can be psychologically “vaccinated” against misinformation by providing people with tools to counter persuasive attacks (McGuire, 1964). Inoculation can be achieved in two ways: (i) narrow- spectrum inoculation based on specific arguments and issues prior to exposure of a persuasive claim, or (ii) broad-spectrum inoculation against commonly used misinformation techniques (e.g., extreme emotions, fake experts) rather than specific claims (van der Linden et al., 2021; Basol et al., 2021). In both cases, the inoculation approach has two components: a pre-warning of the persuasive claim and a pre-emptive refutation (Compton, 2012). Such refutational pre- emptions help people build “cognitive antibodies” against persuasive arguments (Banas & Rains, 2010). These components, a pre-warning and a pre-emptive argument refutation, contribute to motivational readiness to defend oneself from manipulation and prepare individuals with arguments against misinformation that they might encounter. A meta-analysis of earlier evidence found that inoculation interventions against persuasion showed a medium effect size (d = 0.43) on average with decaying resistance after two weeks (Banas & Rains, 2010). Also, a recent meta-analysis also found that inoculation interventions improved credibility assessment for misinformation (d = -0.36) and non- misinformation (d = 0.20) and increased intentions to share non-misinformation (d = 0.18), yet did not significantly decrease misinformation sharing intentions (d = -0.35, p = .12) (Lu et al., 5 2023). Consistent with the notion of broad-spectrum inoculation, the meta-analysis also found that inoculation treatments can create resistance not only to the target persuasive message included in the inoculation treatment, but also novel persuasive messages, providing a “blanket of protection” (Banas & Rains, 2010; Compton & Pfau, 2005, p. 105). Recent research has also supported the efficacy of inoculation against various types of health misinformation (Compton et al., 2016; Isles et al., 2021). In the context of vaccination specifically, prior research found that inoculations before exposure to misinformation were more effective at increasing vaccination intentions than debunking (Jolley & Douglas, 2017). This is consistent with evidence that, once exposed, it is difficult to remove misinformation’s impact; people may continue to be influenced by misinformation or seek misinformation aligned with their social identities (van Bavel et al., 2021). Together, these findings support the use of inoculation to equip the public with the knowledge and skills needed to recognize and stop the spread of misinformation. Behavioral Inoculation Approaches to Misinformation The inoculation approach can use various modalities, such as text, infographics (i.e., passive inoculation), videos, and games (i.e., active inoculations), with varying degrees of activeness and passiveness (Roozenbeek & van der Linden, 2022; Lewandowsky & van der Linden, 2021). Passive and active inoculations have distinct tradeoffs for potential impact on reducing susceptibility to misinformation. For instance, passive inoculations might be easy to manage and distribute widely (Roozenbeek et al., 2022), yet they may be less engaging, eliciting smaller effect sizes than games (e.g., Basol et al., 2021). Conversely, game-based, or active inoculations, may elicit large effect sizes and their impacts may last longer, but playing games 6 takes more time than reading infographics or watching videos, posing potential challenges for scale (e.g., Maertens et al., 2021). Some studies found that, across various topics including climate change and health, people better identified misinformation (Basol et al., 2021; Roozenbeek et al., 2020), had more confidence to identify misinformation (Basol et al., 2020, Saleh et al., 2021), and lower willingness to share misinformation after exposure to game-based inoculation than viewing infographics (Basol et al., 2021; Roozenbeek & van der Linden, 2020). People also better identified misinformation after reading infographics compared to no inoculation, yet with slightly lower accuracy compared to those exposed to game-based inoculation (Basol et al., 2021). Positive impacts for game-based inoculation have been supported in different languages (e.g., French, German; Basol et al., 2021; Roozenbeek et al., 2020) and such impacts have lasted at least about three months (Maertens et al., 2021), though evidence is mixed on how to most effectively implement prebunking (Athey et al., 2022). Given prior research findings that both active game-based and passive inoculations can effectively increase the accuracy of discerning misinformation, with some indication that passive inoculations yield smaller impacts than active inoculations, we pose the following hypotheses: H1: Active inoculations (i.e., game-based) (H1a) and passive inoculations (i.e., infographics) (H1b) will increase the accuracy of discerning misinformation, compared to the placebo group. H2: Active inoculations (i.e., game-based) will significantly increase the accuracy of discerning misinformation compared to passive inoculations (i.e., infographics). 7 H3: Active inoculations (i.e., game-based) (H3a) and passive inoculations (i.e., infographics) (H3b) will increase discernment in sharing information, compared to the placebo group. H4: Active inoculations (i.e., game-based) will increase discernment in sharing information compared to passive inoculations (i.e., infographics). Prior research found that those who played inoculation games were more willing to share the intervention than those who read related infographics (Basol et al., 2021). Thus, we pose the following hypothesis: H5: Active inoculations (i.e., game-based) will increase the willingness to recommend the chatbot. Notably, prior research suggested that playing prebunking games does not impact the ability to discern non-misinformation but instead makes participants respond conservatively, more frequently judging any news item as ‘false’, including non-misinformation (Basol et al., 2021; Modirrousta-Galian & Higham, 2023a, 2023b). While increased skepticism may reduce the spread of misinformation, some argue that non-discrimination between credible and non- credible information could be harmful in the long run (Guay et al., 2023). Given the criticisms that inoculation may increase skepticism of all information rather than improve discernment between true news and misinformation (Modirrousta-Galian & Higham, 2023a), we aim to explore mechanisms that could mitigate backfire effects on credible information. To this end, we test whether comprehensive training that includes examples of information that is misleading and information that is not misleading is critical to improved discernment. We contrast the comprehensive inoculation against a brief active inoculation 8 condition (i.e., short games) that includes only examples of misinformation, an approach similar to early inoculation theory studies (e.g., Banas & Rains, 2010; McGuire & Papageorgis, 1962) that included only attack messages and refutational pre-emptions. Given no prior research, we explore the extent to which the impact of the brief active inoculation (i.e., short game) on our outcomes, and consider how its impact differs from that of the comprehensive active inoculation. Method We conducted a randomized experiment using WhatsApp. This study was approved by the Health Media Lab Institutional Review Board (#2118). We closely coordinated with the Ministry of Health, the Hashemite Kingdom of Jordan during the design, implementation, and analyses of the study. Recruitment We conducted this study between October 27, 2022, and November 23, 2022. Participants were recruited through Facebook advertisements targeting users aged 18 years or older and located in Jordan with WhatsApp installed. The advertisements (see Figure 1) marketed our chatbot-based game under the title “Find the Fake”, inviting people to play a challenge related to the spread of misinformation online for a chance to win 70 Jordanian dinars (roughly equivalent to US$ 100). 9 Figure 1. Facebook recruitment ads After participants clicked the ads, they were automatically directed to a WhatsApp business line, and the chatbot began after participants sent an initial message to the line. In response to the first message sent by participants, the chatbot replied with a message briefly describing the game and participants were asked if they wanted to continue. Participants who opted in were then provided additional background information about the study and contact information for the researchers, completing the informed consent protocol. Experimental Design After providing informed consent, participants were randomly assigned to one of five study arms summarized in Table 1. 2 Participants were exposed to educational content, if assigned to a treatment group, then completed a series of questions to evaluate the impact of the intervention, and finally a brief survey assessing attitudes toward the chatbot and collecting basic 2 At the launch of the study on October 27, 2022, no pure control was included in the study arms. Instead, a second version of the infographics was tested that also provided examples of people communicating with misinformation techniques. However, due to high levels of attrition (~50%), this arm was dropped, and the pure control was introduced four days after launch on November 1, 2022. 10 demographics. The evaluation and survey were presented directly following the intervention implementation; therefore, outcomes represent measures of immediate term effects. Table 1. Summary of Study Arms Study Arm Description Comprehensive active inoculation Completed three lessons on misinformation (appeal to emotions, false experts, digital manipulation of documents/images), including examples of what misinformation looks like and what misinformation does not look like. Users received feedback on their performance in the form of scores and badges. Brief active inoculation Completed three lessons on misinformation (appeal to emotions, false experts, digital manipulation of documents/images), including only examples of misinformation. Users received feedback on their performance in the form of scores and badges. Passive inoculation Exposed to three infographics about misinformation (appeal to emotions, false experts, and how to spot false content), and asked to complete attention checks. Placebo Exposed to three infographics on an unrelated topic (road safety) and asked to complete attention checks. Control No training or interaction before evaluation. Comprehensive Active Inoculation Participants assigned to the comprehensive active inoculation arm completed three rounds of training that addressed three common misinformation techniques: use of content design to elicit strong emotions (e.g., use of very negative words like ‘horrible’), reference to false experts (e.g., appealing to authorities who are expert in a domain unrelated to the issue at hand), and the digital manipulation of documents/images with fake content. In each round, participants were presented with different vignette-based scenarios reflecting domains where misinformation is commonly spread including political and health issues. After a brief description of the scenario, participants were presented with messages 11 shared by different characters and asked to indicate whether they believed the messages were spreading misinformation or not. In the comprehensive version of the game, messages included examples using misinformation spreading techniques and examples of messages that did not use misinformation techniques. After each question, participants would get feedback about whether their responses were correct or incorrect, and the chatbot leveraged several gamification strategies. For example, participants received visual feedback about their scores that persisted throughout each round (e.g., stars for correct responses and blank squares for each incorrect response). In addition, participants who correctly identified the message spreading misinformation in a round received a virtual ‘badge’. At the end of the game, participants received a recap of their performance and the various badges they earned. See Appendix A for screenshots from the game. Brief Active Inoculation The brief active inoculation condition was the same as the comprehensive inoculation game except participants were exposed to only examples of messages that used misinformation techniques. To shorten the game, all neutral examples were excluded from the training rounds. Passive Inoculation Participants assigned to passive inoculation were sent three infographics adapted from the UNESCO resources on media and information literacy (UNESCO, 2022) with imagery and text referencing COVID-19 removed. The three infographics (see Appendix B) covered information that paralleled the lessons taught in the three rounds of gamified training. Specifically, they addressed: (1) that disinformation was often designed to trigger emotional reactions, (2) that false experts are often used to spread manipulative information, and (3) how to spot false content and rumors. The infographics were described to users as tips that would help them perform well 12 on the ‘game’ that began after they learned the tips, which actually consisted of our evaluation questions. After each infographic, participants were asked to answer an attention check question (e.g., ‘Please reply with “3”. Do not reply with other options.’). The attention check served as a placebo for interaction similar to the game experience, but without actively testing users’ understanding of the misinformation lesson. Placebo The procedure of the placebo arm followed the same protocol as the infographics study arm except all infographics were unrelated to misinformation. The featured infographics were published by the World Health Organization (WHO, 2022) and instead focused on three topics of road safety: (1) traffic injury facts; (2) speed management; (3) how to keep children safe (see Appendix C for placebo infographics). This activity was framed as a practice round for users in which they could learn the game mechanics of how to respond to questions, with the ‘real’ game (i.e., evaluation) beginning after the practice. Control In the control condition, there was no exposure to any material, and participants proceeded directly to the outcome evaluation. The outcome evaluation was framed as the game for participants in this study arm. Measures Evaluation Outcomes. To evaluate the impact of the misinformation treatments, the primary outcomes of interest were: (i) rates of accurately discerning between headlines that use misinformation tactics and those that do not, and (ii) discernment in sharing headlines (see Appendix D for evaluation measures). These outcomes were assessed immediately after the implementation of the intervention. 13 Misleading Headlines. All headlines focused on COVID-19 as a theme, and we generated new headlines based on existing headlines rather than using headlines from published articles. This approach was adopted to ensure that all headlines were completely novel to participants whereas participants could vary in their familiarity with real-world headlines. The designed headlines were adapted from prior research testing the effectiveness of inoculation against COVID-19 misinformation (Basol et al., 2021), and reflect the three common tactics used in misinformation that were also the focus of the inoculation training (i.e., emotional appeals, false experts, and fake documents/images). Additionally, the misleading headlines were designed to address themes similar to misinformation that had spread online according to various fact-checking sources (e.g., AFP Fact Check, Africa Check) to enhance ecological validity. Judgements of Misinformation. Adapted from Roozenbeek et al. (2022), participants were presented with six headlines: three using misinformation tactics and three that did not use common misinformation tactics. When presented with each headline, participants were asked to respond to the question ‘Does this headline use any misinformation techniques?’ on a 4-point scale: Definitely is misinformation, Probably is misinformation, Probably is not misinformation, Definitely is not misinformation. For ease of interpretation, ratings were scored such that higher scores represent stronger belief that a headline was misinformation (Definitely is misinformation = 4 and Definitely is not misinformation = 1). We chose phrasing that highlighted the term misinformation to increase relevance to policymakers and practitioners, but it is worth noting that much prior research on inoculation measures perceptions of misleadingness rather than misinformation classifications per se (e.g., Roozenbeek et al., 2022). In line with prior literature, we compute three scores to assess accuracy in misinformation detection (Basol et al., 2021; Maertens et al., 2021). First, we calculate a measure of 14 discernment, defined as a participant’s average misinformation scores for misleading headlines minus their average score for headlines without misleading content. With this operationalization, discernment scores could range from -3 to +3 where a score of +3 indicates a participant rated all misinformation headlines as ‘Definitely misinformation’ and all non-misinformation headlines as ‘Definitely not misinformation’ for perfect discernment. We also examine the disaggregated discernment score including the average ratings for the three misleading headlines and ratings for the three non-misleading headlines. As higher ratings correspond to judgements that a headline is using misinformation tactics, more accurate scores would be represented by higher scores on the misleading headlines (representing true positives) and lower scores on the non-misleading headlines (representing true negatives). Sharing Misinformation. Adapted from Basol et al. (2021) and Roozenbeek and van der Linden (2020), sharing of misinformation was assessed with two headlines: one that did not use misinformation tactics and a headline that used misinformation tactics. Each participant was randomly assigned to one of two misleading headlines, a headline that used extreme emotion or a headline that used a false expert. Participants were asked to rate their likelihood of sharing each of the headlines on a four-point scale: (1) Very unlikely to share, (2) Unlikely to share, (3) Likely to share, (4) Very likely to share. As with judgements of misinformation, we report three scores for sharing—discernment of sharing, likelihood of sharing misleading headlines, and likelihood of sharing non-misleading headlines. Discernment of sharing was calculated as the sharing score for non-misleading headline minus the sharing score for the misleading headline. Accordingly, participants with high positive discernment of sharing have lower intentions of sharing misinformation, relative to sharing more non-misinformation. 15 Self-Report Outcomes. Complementing the evaluation of the intervention’s impact on detection and sharing of misinformation, we examined differences in attitudes toward the chatbot game. Confidence. Participants were asked to indicate the extent to which they felt more, the same, or less confident in detecting misinformation after completing the game. Perceived Difficulty. Participants were asked to report whether they thought the game was too difficult, the right level of difficulty, or too easy. Recommending the Game. Participants were asked to indicate the extent to which they would recommend the game to others with three possible response categories: yes, maybe, and no. Participants A total of 2,851 participants completed the study. Of them, 63% identified as male and 33% as female; 49% reported having completed secondary education (3% with no education, 12% with primary, and 3% with tertiary education); 53% reported being between the ages of 18 and 29, 25% in their 30s, 13% in their 40s, and 5% over the age of 50. Finally, 85% reported being vaccinated for COVID-19, 5% unvaccinated but willing to vaccinate, and 5% unvaccinated and unwilling to vaccinate. None of the treatment arms reported significant differences in demographics and vaccination status compared to the control group. See Appendix E for sample composition and randomization balance and Appendix F for age and gender distributions of users who started WhatsApp conversations according to Facebook Ad Manager data. Results Attrition and Study Completion Time 16 Figure 2 summarizes attrition at each stage of the study by treatment group and Table 2 summarizes results of attrition analysis in which we use a linear probability model to examine whether assignment to study arms is associated with higher levels of attrition compared to the placebo, the comparison group for our main results. Results reveal that study completion was more likely in the control, brief active inoculation, and passive inoculation conditions compared to the placebo arm. However, study completion rates were similar for both the comprehensive active inoculation and placebo arms, although attrition was more likely to occur at the intervention stage in the comprehensive active inoculation arm. Attrition is mainly explained by differences in the time required to complete the intervention. The median time to complete the comprehensive active inoculation intervention was 10 minutes, 7 minutes for the brief active inoculation, 4 minutes for the passive inoculation, and 5 minutes for the placebo intervention (see Appendix G for more details on time for completion by different stages of the study). Figure 2. Flowchart of experimental design and attrition 17 Table 2. Results of attrition analysis (1) (2) (3) (4) (5) Completed Completed Started Completed Started survey treatment evaluation evaluation survey (end) Control 0.293*** 0.274*** 0.235*** 0.240*** 0.231*** (0.015) (0.016) (0.020) (0.021) (0.021) Comprehensive Active -0.045** -0.039* -0.022 -0.016 -0.013 Inoculation (0.022) (0.022) (0.023) (0.023) (0.023) Brief Active Inoculation 0.027 0.032 0.043* 0.044** 0.046** (0.021) (0.021) (0.022) (0.022) (0.022) Passive Inoculation 0.068*** 0.072*** 0.050** 0.054** 0.054** (0.020) (0.021) (0.022) (0.022) (0.022) Controls No No No No No Observations 4,150 4,150 4,150 4,150 4,150 R-squared 0.060 0.050 0.031 0.031 0.028 Note. OLS estimates. Base comparison group is the placebo condition. Robust standard errors are shown in parenthesis. *** p<0.01, ** p<0.05, * p<0.1. Discernment in Misinformation Judgements Before running analyses to test our hypotheses, we summarize the descriptive statistics for our three scores of misinformation judgements (see Table 3). As expected, misleading headlines tended to be perceived as using misinformation tactics to a greater extent than non- misleading headlines. Violin plots depicting the full distribution of scores are also reported in 18 Appendix H. Table 3. Descriptive statistics of misinformation discernment scores by study arms Misinformation Misinformation Misinformation Judgements of Judgements of Discernment Misleading Non-Misleading Headlines Headlines M SD M SD M SD Placebo 0.89 1.06 3.08 0.72 2.19 0.86 Control 0.96 1.06 3.11 0.72 2.15 0.85 Comprehensive Active Inoculation 1.25 1.03 3.22 0.69 1.98 0.84 Brief Active Inoculation 1.09 1.01 3.34 0.72 2.24 0.92 Passive Inoculation 0.90 1.05 3.14 0.69 2.23 0.85 Effect of Active and Passive Inoculations on Misinformation Discernment (H1, H2) H1 predicted that active inoculations (i.e., game-based) (H1a) and passive inoculations (i.e., infographics) (H1b) will increase the accuracy of discerning misinformation, compared to the placebo group. H2 predicted that active inoculations (i.e., game-based) will significantly increase the accuracy of discerning misinformation compared to passive inoculations (i.e., infographics). To test H1 and H2, we conducted Ordinary Least Squares (OLS) regressions on the three scores to assess misinformation discernment across study arms, controlling for observable demographic characteristics (see Table 4). To address the possibility that misinformation discernment results are due to the experience of the chatbot, differential attrition, or some other external factor, we use the placebo group (instead of the control group) as the comparison group as it followed the same protocol as the infographic group but was not exposed to the misinformation mechanism and learning process. This placebo group mimics the experience of the treatment arms by showing information on road safety and asking follow-up questions to ensure that respondents could learn game mechanism. 2 Impact of active inoculation on misinformation discernment compared to placebo (H1a). Consistent with H1a, we find that participants assigned both comprehensive and brief active inoculation groups are better at discerning between misinformation and non- misinformation than the placebo group. More specifically, the comprehensive active inoculation arm increased the accuracy of discerning misinformation by 0.29 SD compared to the placebo group. Disaggregating this score, we find misinformation ratings increased 0.16 SD for misleading headlines and decreased 0.22 SD for non-misleading headlines compared to the placebo group, suggesting increased accuracy in classifying both types of information. The brief active inoculation group that completed the short version of the game scored 0.14 SD higher than the placebo group on misinformation discernment (see Table 4 and 2 Figure 3, Panel A). However, disaggregating the discernment score, we find this effect is driven by improvements in accurately judging misleading headlines as using misinformation tactics, but there is no improvement in judgements of non-misleading headlines. Specifically, the brief active inoculation, containing only examples of misinformation, increased the probability of detecting misinformation by 0.31 SD but did not decrease misinformation ratings of non-misleading headlines (see Table 4 and 2 Figure 3, Panel B). Impact of passive inoculation on misinformation discernment compared to placebo (H1b). Contrary to H1b, we find that the passive inoculation did not significantly improve discernment (see Table 4, Panel A and 2 Figure 3). Disaggregating the discernment score, there is no difference in ratings of misinformation for either misleading or non-misleading headlines compared to the placebo group. Impact of active inoculation on misinformation discernment compared to passive inoculation (H2). Consistent with H2, the active inoculation groups are better at misinformation discernment than the passive inoculation group (See Table 4, Panel B). More specifically, participants of the comprehensive version of the game score 0.33 SD higher on misinformation discernment, and participants of the brief version of the game score 0.18 SD higher, than the passive inoculation infographic group. 2 Table 4. Regression results for misinformation discernment scores. (1) (2) (3) Misinformation Misinformation Misinformation Judgements of Judgements of Non- Discernment Misleading Misleading Headline Headline Panel A: Placebo as base group Control -0.067 -0.054 0.037 (0.059) (0.061) (0.058) [0.828] [0.828] [0.828] Comprehensive Active 0.285*** 0.155*** -0.218*** Inoculation (0.057) (0.058) (0.056) [0.000] [0.026] [0.000] Brief Active Inoculation 0.140** 0.311*** 0.084 (0.056) (0.059) (0.059) [0.035] [0.000] [0.258] Passive Inoculation -0.041 0.037 0.080 (0.056) (0.057) (0.054) [0.975] [0.975] [0.607] Controls Yes Yes Yes Observations 2,851 2,851 2,851 R-squared 0.090 0.040 0.069 Panel B: Passive Inoculation as base group Control -0.027 -0.091 -0.042 (0.058) (0.060) (0.057) [0.909] [0.563] [0.909] Placebo 0.041 -0.037 -0.080 (0.056) (0.057) (0.054) [0.975] [0.975] [0.607] Comprehensive Active 0.325*** 0.119** -0.297*** Inoculation (0.056) (0.056) (0.055) [0.000] [0.098] [0.000] Brief Active Inoculation 0.181*** 0.274*** 0.005 (0.055) (0.057) (0.058) [0.006] [0.000] [0.938] Controls Yes Yes Yes Observations 2,851 2,851 2,851 R-squared 0.090 0.040 0.069 Note. OLS estimates of treatment effects. Control variables are age, education, gender, and vaccination status for COVID-19. Robust standard errors are shown in parenthesis, and Westfall-Young stepdown adjusted p-values in brackets (1,000 bootstrap replications). *** p<0.01, ** p<0.05, * p<0.1. 2 Figure 3. Impact of treatments on misinformation discernment. A. Overall Discernment Note. Misinformation discernment is defined as a participant’s average misinformation scores for misleading headlines minus their average score for headlines without misleading content. B. Misleading and Non-Misleading Headlines Note. Misinformation Judgements scores range from 1 (‘Definitely not misinformation’) to 4 (‘Definitely misinformation). 2 Sharing Discernment (H3, H4) Table 5 summarizes the descriptive statistics of the three scores for intentions to share misinformation by study arm. Overall, self-reported likelihood of sharing headlines is low, even for neutral headlines. However, the descriptive statistics are consistent with lower likelihood of sharing misleading headlines than non-misleading headlines. Violin plots depicting the full distribution of scores are also reported in 2 Appendix H. Effect of Active and Passive Inoculations on Sharing Discernment (H3, H4) Table 5. Descriptive statistics of intention to share misinformation by study arm. Sharing Misleading Non-Misleading Discernment Headline Sharing Headline Sharing M SD M SD M SD Placebo 0.40 1.37 2.16 1.15 2.56 1.13 Control 0.57 1.28 2.09 1.17 2.66 1.09 Comprehensive Active Inoculation 0.87 1.43 1.74 1.08 2.61 1.14 Brief Active Inoculation 0.64 1.32 1.70 1.07 2.35 1.21 Passive Inoculation 0.55 1.31 1.96 1.13 2.51 1.16 H3 predicted that active inoculations (i.e., game-based) (H3a) and passive inoculations (i.e., infographics) (H3b) will increase discernment in sharing information, compared to the placebo group. Table 6 summarizes the regression results. Impact of active inoculation on sharing intentions compared to placebo (H3a). Contrary to what we found for detecting misinformation in the headlines, only the comprehensive active inoculation arm increased discernment in sharing misinformation compared to the placebo group, partially supporting H3a. 3 Specifically, the comprehensive active inoculation treatment increased sharing discernment by 0.23 SD. Disaggregating the sharing discernment score, we find that the comprehensive inoculation treatment reduces the likelihood of sharing misleading headlines by 0.30 SD and does not impact sharing of non-misleading headlines. However, the brief inoculation treatment reduces likelihood of sharing misleading headlines by 0.33 SD and also reduces sharing of non-misleading headlines by 0.26 SD (See 3 We did not find differences in outcomes for the headline that uses extreme emotion and the headline that uses a false expert. 27 Table 6, Panel A; Figure 4, Panel B). This finding is consistent with the notion that the brief active inoculation may increase overall conservative responding. Impact of passive inoculation on sharing intentions compared to placebo (H3b). Contrary to H3b, the passive inoculation group performs similarly to the placebo group, and the difference between the passive inoculation and placebo group is not statistically significant when using Westfall-Young stepdown adjusted p-values to control for the multiple hypothesis testing. Impact of active inoculations on sharing intentions compared to passive inoculation (H4). H4 predicted that active inoculations (i.e., game-based) will increase discernment in sharing information compared to passive inoculations (i.e., infographics). To test H4, we also run an OLS regression treating the passive inoculation study arm as the base level (see Table 6, Panel B). We find that the comprehensive active inoculation improves sharing discernment by 0.24 SD compared to passive inoculation, a pattern driven by a 0.19 SD reduction in sharing misleading headlines. We also find that there is no significant difference between the brief active inoculation and the passive inoculation in sharing discernment. However, this null effect seems to be driven by a 0.14 SD reduction in sharing non-misleading headlines, as the brief inoculation significantly reduced sharing of misleading headlines by 0.22 SD. 28 Table 6. Regression results for sharing misinformation. (1) (2) (3) Sharing Misleading Non-Misleading Discernment Headline Sharing Headline Sharing Panel A: Placebo as base group Control -0.126** 0.068 -0.080 (0.059) (0.061) (0.058) [0.245] [0.828] [0.717] Comprehensive Active 0.232*** -0.304*** -0.027 Inoculation (0.060) (0.058) (0.057) [0.000] [0.000] [0.629] Brief Active Inoculation 0.062 -0.334*** -0.257*** (0.056) (0.057) (0.058) [0.258] [0.000] [0.000] Passive Inoculation -0.003 -0.118** -0.120** (0.055) (0.057) (0.055) [0.999] [0.224] [0.202] Controls Yes Yes Yes Observations 2,851 2,851 2,851 R-squared 0.025 0.057 0.028 Panel B: Passive Inoculation as base group Control -0.123** 0.186*** 0.040 (0.059) (0.059) (0.059) [0.233] [0.016] [0.909] Placebo 0.003 0.118** 0.120** (0.055) (0.057) (0.055) [0.999] [0.224] [0.202] Comprehensive Active 0.235*** -0.186*** 0.093 Inoculation (0.060) (0.056) (0.058) [0.001] [0.005] [0.115] Brief Active Inoculation 0.065 -0.216*** -0.137** (0.056) (0.055) (0.059) [0.439] [0.000] [0.089] Controls Yes Yes Yes Observations 2,851 2,851 2,851 R-squared 0.025 0.057 0.028 Note. OLS estimates of treatment effects. Control variables are age, education, gender, and vaccination status for COVID-19. Robust standard errors are shown in parenthesis, and Westfall-Young stepdown adjusted p-values in brackets (1,000 bootstrap replications). *** p<0.01, ** p<0.05, * p<0.1. 29 Figure 4. Impact of treatments on sharing discernment scores. A. Overall Discernment Note. Sharing discernment is defined as the difference between sharing a headline that did not use misinformation tactics and a headline that used misinformation tactics. B. Misleading and Non-Misleading Headlines Note. Sharing intention scores range from 1 (rating as ‘Very unlikely to share’) to 4 (rating all as ‘Very likely to share’) 30 Impacts on Recommending the Intervention (H5), Confidence, and Difficulty H5 poses active inoculations will increase the willingness to recommend the intervention more than passive inoculations. Users in the brief active inoculation arm were more willing to recommend the intervention than the passive inoculation group by 0.15 SD, and users in comprehensive inoculation arm were more willing to recommend the intervention by 0.12 SD (See Table 7). Thus, H5 is supported. Additionally, both the comprehensive and brief active inoculation treatments lead to more positive attitudes toward the chatbot relative to the placebo group. For instance, participants in the comprehensive and brief active inoculation groups rated that their ability to detect misinformation increased after the chatbot experience by 0.15 SD and 0.20 SD, respectively, compared to the placebo group. They also found the chatbot more challenging. There were no significant differences between the passive inoculation group and the placebo group in any of these self-reported outcomes. See Figure 5. 31 Table 7. Willingness to Recommend Interventions, Attitudes Toward the Chatbot (H5) (1) (2) (3) Confidence on identifying Recommend the Game difficulty misinformation game after the game Panel A: Placebo as base group Control -0.055 0.015 -0.069 (0.061) (0.062) (0.059) [0.828] [0.828] [0.828] Comprehensive Active 0.152*** 0.286*** 0.097* Inoculation (0.059) (0.057) (0.055) [0.029] [0.000] [0.140] Brief Active Inoculation 0.201*** 0.122** 0.133** (0.058) (0.060) (0.052) [0.001] [0.117] [0.035] Passive Inoculation 0.007 -0.007 -0.018 (0.058) (0.058) (0.056) [0.999] [0.999] [0.991] Controls Yes Yes Yes Observations 2,851 2,851 2,851 R-squared 0.041 0.026 0.092 Panel B: Passive Inoculation as base group Control -0.062 0.022 -0.050 (0.060) (0.061) (0.062) [0.868] [0.909] [0.909] Placebo -0.007 0.007 0.018 (0.058) (0.058) (0.056) [0.999] [0.999] [0.991] Comprehensive Active 0.145** 0.294*** 0.116** Inoculation (0.057) (0.056) (0.057) [0.044] [0.000] [0.098] Brief Active Inoculation 0.194*** 0.129** 0.151*** (0.056) (0.059) (0.055) [0.005] [0.102] [0.026] Controls Yes Yes Yes Observations 2,851 2,851 2,851 R-squared 0.041 0.026 0.092 Note. OLS estimates of treatment effects. Control variables are age, education, gender, and vaccination status for COVID-19. Robust standard errors are shown in parenthesis, and Westfall-Young stepdown adjusted p-values in brackets (1,000 bootstrap replications). *** p<0.01, ** p<0.05, * p<0.1 32 Figure 5. Impacts of Treatments on Attitudes toward the Intervention Note. Scores for confidence on identifying misinformation after the game range from 1 (‘Less confidence than before playing’) to 3 (‘More confident than before playing’); scores for game difficulty range from 1 (‘Too easy’) to 3 (‘Too difficult’); scores from recommend the game range from 1(‘No I would not recommend it’) to 3(‘Yes I would recommend it’) Robustness Checks and Attention Check To ensure the validity and robustness of our previous findings, we perform a series of robustness checks. First, attention checks were introduced in the passive inoculation and placebo study arms to mimic the experience of the active inoculations’ treatments in terms of interactions and time spent. Forty percent of participants in the passive inoculation group and 37% of participants in the placebo group successfully answered the three attention checks presented. We found that participants who correctly answered the attention checks scored higher in both detecting 33 misinformation and not sharing misinformation, than those who missed any of the questions (see Appendix I). Attentiveness proved to be a relevant factor when using self-paced unsupervised digital tools. Further research is needed to determine the role of attention checks in active inoculations. Second, to test the stability of the results, we compare the impact of treatments on discernment in judgements and sharing of misinformation with and without covariates (see Appendix J, Table J1). We find that our results are consistent for different model specifications. Third, the evaluation outcomes include the use of edited documents/images, which was directly taught in the active inoculation treatments (the game) but only indirectly addressed in the infographic of the passive inoculation study arm. To ensure that the null impact of the passive inoculation arm is not driven by this difference, we also test the impact of the interventions with only the appeal to emotions and false expert misinformation tactics, which were directly addressed in both the active and passive inoculation arms. Appendix J, Table J2 reports these results, showing that the pattern of findings is consistent with the exclusion of the document manipulation item. To explore the potential impact of the treatment in Jordan, we included population weights in our analysis to better reflect the population in Jordan. We found that the effect of the active inoculations was no longer significant after these adjustments (Appendix J, Table J3). This result is not unexpected, as adjusting for population weights can dilute the true impact of the intervention due to the under-representation of offline segments of the population in the sample study. We focused on the online population because this group is more exposed to misinformation, making it a critical and distinct subgroup for our investigation. Moreover, it may be reasonable that effects of a digital chatbot game are greater among populations who generally 34 show interest in such digital tools, with this not being a one-size-fits-all tool Future research can test prebunking inoculation interventions in offline settings. Finally, to gain deeper insight into the mechanisms driving the observed effects, we examined demographic and relevant sample characteristics for heterogeneity. Given the focus of the chatbot on COVID-19, we also investigated variation in treatment effects across subgroups based on COVID-19 vaccination status. Appendix K presents our findings, which do not find statistically significant interactions, though we have a small sample of unvaccinated respondents in the study. Discussion We test the effectiveness of active and passive inoculations on detecting and sharing misinformation. First, we found that both comprehensive and brief active inoculation groups detect misinformation better than the passive inoculation and placebo. Second, we found that both comprehensive and brief active inoculation groups are less willing to share misinformation than the placebo and control groups. Third, we found that the passive inoculation did not significantly increase the ability to detect misinformation or lower the willingness to share misinformation, compared to the placebo. Yet, only the comprehensive active inoculation group is significantly less willing to share misinformation than the infographic group. Moreover, compared to the comprehensive active inoculation group, the brief inoculation group was less accurate in identifying non-misleading headlines. Both active inoculation groups have higher confidence, found the intervention more challenging, and are more willing to recommend the intervention than passive inoculation group. Finally, we demonstrated that our results are robust to potential biases and different specifications, although differential rates in attrition across 35 treatment arms imply that results need to be interpreted as upper bounds of impact with appropriate caution. These results are consistent with previous evidence that people are better at identifying misinformation after active inoculations, such as games, and that passive inoculations, such as infographics, have smaller or no effect (Basol et al., 2021; Roozenbeek et al., 2020). However, these results were contrary to the meta-analysis findings that both active and passive inoculations improved misinformation detection (Lu et al., 2023). Our results indicate that it is critical to not only educate people about what misinformation looks like but equally what misinformation does not look like to better discern between misleading and credible news. In our study, the brief active inoculation group did not see any examples of non-misinformation and improved only in their classification of misleading headlines. These findings may highlight one mechanism for mitigating overall skepticism, a challenge that some have argued is a consequence of prebunking games (Modirrousta-Galian & Higham, 2023a, p. 3). While other games for inoculation against misinformation, including Go Viral! and Bad News, featured examples of misleading and non- misleading messages, studies testing the effectiveness of those games did not focus on features that may exacerbate or mitigate potential backfire effects on perceptions of credible information (Basol et al., 2021; Roozenbeek & van der Linden, 2021). While brief versions of gamified inoculation could be promising to keep participants engaged, as attrition rates were lower in the brief than comprehensive version of the game, showing both misinformation and non-misinformation examples may be crucial to better discerning non-misinformation. To minimize attrition during the intervention, future research may explore other methods of strengthening engagement while maintaining examples to contrast true and ‘fake’ news. 36 Unlike prior research that found passive inoculation effectively improved the ability to detect misinformation (Basol et al., 2021; Lu et al., 2023), we found that passive inoculation was not effective at improving the discernment of misinformation detection. Contrary to the prior research that inoculations did not decrease misinformation sharing intentions (Lu et al., 2023), our results showed that all inoculation treatments significantly decreased intentions to share misinformation, whereas only brief active inoculations decreased the intentions to share non- misinformation. These results also add to the literature that accuracy judgements and sharing behavior are not always aligned (Pennycook et al., 2021), underscoring the need to treat these outcomes as distinct. In summary, passive inoculations (i.e., infographics) may fall short for improving the accuracy in detecting misinformation compared to the active inoculations (i.e., games) but they also reached a greater number of participants, as evidenced by lower attrition rates. Further, passive inoculations may still be useful for stopping the spread of misinformation, through reductions in sharing behavior, though this effect did not persist after adjusting for multiple hypothesis testing. Widespread deployment of the infographics may require sufficiently low resources to expose large numbers of individuals at a scale such that these small effects are nonetheless worthwhile in some contexts. Policy Implications As policymakers continue to grapple with the spread of mis- and disinformation in an increasingly digitally connected world, the findings presented in this study have important policy implications. This research provides new tools for policymakers and practitioners to effectively boost misinformation detection in the public and stop the spread of misinformation by improving the public’s information literacy and critical thinking skills. In Jordan, the findings are particularly 37 relevant as the latest available data suggests that coverage of COVID-19 vaccinations remained relatively low, under 50% of the population as of March 2023 (Coronavirus Resource Center, 2023). Survey studies highlighted the prevalence of COVID-19 misinformation as a driver of vaccination status in Jordan along with other barriers like perceived financial cost (Sallah et al., 2022) although the government provided the vaccine free of cost and incentivized vaccination through introduction of an online portal that allowed for greater mobility among the vaccinated (Wolters & Abir, 2022). Moreover, the public continues to encounter mis- and dis-information related to vaccination, including growing rumors about potential harms of the recently introduced measles and rubella vaccine for school-aged children in September 2023 (Barakat et al., 2023; Kuttab, 2023). Beyond vaccination, the inoculation game tested in this study could be effective in stopping the spread of misinformation in other health domains that require active stakeholder engagements to promote accurate health information, such as healthy behaviors to reduce the growing burden of non-communicable diseases (e.g., cardiovascular diseases, diabetes mellitus) and associated risk factors (e.g., hypertension, overweight/obesity, hyperglycemia). For example, misinformation about tobacco, vaping, and e-cigarettes is growing increasingly common online (Al-Rawi et al., 2023; Romer et al., 2020), representing a new threat to global anti-smoking and tobacco control efforts. With such extensions, future research will need to identify the most relevant misinformation techniques and incorporate them into inoculation tools tailored to these domains. This research also adds to the growing body of evidence of social media interventions (e.g., Donati et al., 2022) demonstrating the value of leveraging digital communication platforms like WhatsApp to implement low-cost interventions and evaluation in environments where resources for program implementation and data collection are limited. In particular, the increasing ease of 38 developing chatbots to reach beneficiaries directly through widely used platforms provides public policy implementers, health service managers, and other stakeholders with an additional channel to give health information to citizens in a user-friendly manner that may better reach those with lower levels of digital literacy than traditional digital resources like websites. Importantly, the inoculation approach tested in this research is only one of a range of solutions considered effective to combat misinformation (APA, 2023). As such, other tools like the use of accuracy prompts (Pennycook et al., 2021) and carefully designed debunking (Chan et al., 2017) offer complementary approaches to that should be considered as part of a comprehensive set of policy initiatives to combat misinformation. Limitations Findings from this study may represent an upper bound of this intervention’s impact due to a number of study limitations and future research is needed to explore the boundary conditions of our findings. Notably, the study only examined the immediate impact of the intervention, as outcomes were measured directly following inoculation. Other research has found evidence of lasting impact from game-based inoculation of up to 3 months with regular training but significant decay without refresher training (Maertens et al., 2021), suggesting a need for additional research to assess medium- and long-term effects our chatbot-based game. Critically, the higher rates of attrition in the comprehensive inoculation arm, compared to the brief and passive inoculation arm, could account for some of the results in our study. Although we largely found balance in demographic characteristics across study arms for those participants who completed the study, it is possible that participants nonetheless differed on unmeasured characteristics that could be relevant to misinformation discernment, such as analytical and critical thinking skills (Arechar et al., 2023). As such, our findings have to be interpreted with this context in mind and future research 39 should seek to mitigate these limitations by testing the intervention with participants that are less likely to drop out and measuring relevant characteristics and including them as covariates in analyses. It is also important to note that the results of the study should be interpreted in the context of its sampling limitations. As a widely used communication platform in low- and middle-income countries, the use of WhatsApp also offers unique opportunities for scale at a global level. While WhatsApp is a popular global platform, the sample of this study skewed male, young, and educated compared to the national population. Moreover, the ad for the study mentioned the topic of misinformation, which could have impacted self-selection into the study. While the ad also emphasized the game-like experience and highlighted an incentive to participate in an effort to attract a broader audience, results may not be generalizable to a more representative audience. Future research could overcome these limitations by recruiting target populations in offline settings like schools or other public spaces, leveraging broader sampling frames and less biased sampling strategies. More studies are also needed to test the effectiveness of the intervention in other countries with different contexts, particularly other low- and middle-income countries. Conclusion Rampant misinformation has become a major global risk exacerbating critical challenges such as climate change and public health (APA, 2023; WEF, 2023). This study contributes to the growing literature that finds that broad-based inoculation by teaching about common manipulation techniques is an effective approach to reducing susceptibility to misinformation. Our study suggests that active inoculation through games may yield stronger effects than more passive approaches, such as infographics, however, the infographic intervention yielded less dropout in the study, highlighting the trade-offs between the two approaches. Moreover, our 40 study sheds light on specific features that might maximize the interventions’ effectiveness. The findings from this study provide encouraging evidence that game-based inoculation should be incorporated into policy initiatives to slow down the spread of misinformation and improve global health. The next step is to test and scale up active inoculation in broader contexts. 41 References Al-Jalabneh, A. A., & Safori, A. O. (2020). “Fake news” threats at the time of a global pandemic: Facebook use as a news platform in Jordan, Journal of Critical Reviews, 7(19), 9437-9446. https://doi.org/10.31838/jcr.07.19.1053 Al-Rawi, A., Blackwell, B., Zemenchik, K., & Lee, K. (2023). Twitter Misinformation Discourses About Vaping: Systematic Content Analysis. Journal of Medical Internet Research, 25, e49416. American Psychological Association [APA]. (2023). Using psychological science to understand and fight health misinformation: An APA consensus statement. https://www.apa.org/pubs/reports/misinformation-consensus-statement.pdf Alkhwaldeh, A., & Emam, D. A. (2021). 569 rumors in Jordan in 2020, 31% concerning health sector, Jordanian Media Credibility Monitor. https://akeed.jo/en/post/2730/569_rumors_in_Jordan_in_2020_31_concerning_health_se ctor Arechar, A. A., Allen, J., Berinsky, A. J., Cole, R., Epstein, Z., Garimella, K., … & Rand, D. G. (2023). Understanding and combatting misinformation across 16 countries on six continents. Nature Human Behaviour, 7(9), Article 9. https://doi.org/10.1038/s41562- 023-01641-6 Athey, S., Cersosimo, M., Koutout, K., & Li, Z. (2023). Emotion-versus Reasoning-based Drivers of Misinformation Sharing: A field experiment using text message courses in Kenya. Available at SSRN 4489759. Banas, J. A., & Rains, S. A. (2010). A meta-analysis of research on inoculation theory. Communication Monographs, 77(3), 281-311. https://doi.org/10.1080/03637751003758193 Barakat, M., Abdaljaleel, M., Atawneh, N., Alkhazaleh, R., Aburumman, D., Hamed, E., & Sallam, M. (2023). Pervasive parental hesitancy and resistance towards measles rubella vaccination in Jordan. Vaccines, 11(11), 1672. https://doi.org/10.3390/vaccines11111672 Basol, M., Roozenbeek, J., & Van der Linden, S. (2020). Good news about bad news: Gamified inoculation boosts confidence and cognitive immunity against fake news. Journal of cognition, 3(1), 1-9. https://doi.org/10.5334/joc.91 Basol, M., Roozenbeek, J., Berriche, M., Uenal, F., McClanahan, W. P., & Linden, S. V. D. (2021). Towards psychological herd immunity: Cross-cultural evidence for two prebunking interventions against COVID-19 misinformation. Big Data & Society, 8(1), 20539517211013868, 1-18. https://doi.org/10.1177/20539517211013868 Bilo-Thomas, P., Hogan-Taylor, C., Yankoski, M., & Weninger, T. (2021). Pilot study suggests online media literacy programming reduces belief in false news in Indonesia. arXiv preprint arXiv:2107.08034. 42 Cavazos, R. (2019). The economic cost of bad actors on the Internet: Fake news, CHEQ. https://s3.amazonaws.com/media.mediapost.com/uploads/EconomicCostOfFakeNews.pd f Chan, M. P. S., Jones, C. R., Hall Jamieson, K., & Albarracín, D. (2017). Debunking: A meta- analysis of the psychological efficacy of messages countering misinformation. Psychological Science, 28(11), 1531-1546. Compton, J. (2012). Inoculation theory. In J. P. Dillard, & L. Shen. (Eds). The SAGE handbook of persuasion: Developments in theory and practice, 2, 220-237. Compton, J., Jackson, B., & Dimmock, J. A. (2016). Persuading others to avoid persuasion: Inoculation theory and resistant health attitudes. Frontiers in Psychology, 7, 122, 1-9. https://doi.org/10.3389/fpsyg.2016.00122 Compton, J. A., & Pfau, M. (2005). Inoculation theory of resistance to influence at maturity: Recent progress in theory development and application and suggestions for future research. Annals of the International Communication Association, 29(1), 97-146. https://doi.org/10.1080/23808985.2005.11679045 Donati, D., Orozco-Olvera, V., & Rao, N. (2022). Using social media to change gender norms: An experiment within Facebook messenger in India. World Bank Policy Research Working Paper. 10199. http://hdl.handle.net/10986/38113 Ecker, U. K. H., Lewandowsky, S., Cook, J., Schmid, P., Fazio, L. K., Brashier, N., Kendeou, P., Vraga, E. K., & Amazeen, M. A. (2022). The psychological drivers of misinformation belief and its resistance to correction. Nature Reviews Psychology, 1(1), Article 1. https://doi.org/10.1038/s44159-021-00006-y Guay, B., Berinsky, A., Pennycook, G., & Rand, D. (2023). How to think about whether misinformation interventions work. Nature Human Behavior, 7, 1231-1233. https://doi.org/10.31234/osf.io/gv8qx Habes, M., Elareshi, M., Mansoori, A., Pasha, S., Salloum, S. A., & Al-Rahmi, W. M. (2023). Factors Indicating Media Dependency and Online Misinformation Sharing in Jordan. Sustainability, 15(2), 1474, 1-15. https://doi.org/10.3390/su15021474 Harjani, T., Basol, M.-S., Roozenbeek, J., & Linden, S. van der. (2023). Gamified inoculation against misinformation in India: A randomized control trial. Journal of Trial & Error. https://doi.org/10.36850/e12 Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world?. Behavioral and Brain Sciences, 33(2-3), 61-83. https://doi.org/10.1017/S0140525X0999152X Jolley, D., & Douglas, K. M. (2017). Prevention is better than cure: Addressing anti‐vaccine conspiracy theories. Journal of Applied Social Psychology, 47(8), 459-469. https://doi.org/10.1111/jasp.12453 43 Lewandowsky, S., & Van Der Linden, S. (2021). Countering misinformation and fake news through inoculation and prebunking. European Review of Social Psychology, 32(2), 348- 384. Lim, J. R., Vakis, R., Cameron, C., & Dugas, M. (2022). Behavioral Science and COVID-19: An Interactive Solutions Guide for Better Policy Design. World Bank. https://doi.org/10.1596/37816 Loomba, S., de Figueiredo, A., Piatek, S. J., de Graaf, K., & Larson, H. J. (2021). Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA. Nature Human Behaviour, 5(3), 337-348. https://doi.org/10.1038/s41562-021-01056-1 Lu, C., Hu, B., Li, Q., Bi, C., & Ju, X. D. (2023). Psychological inoculation for credibility assessment, sharing intention, and discernment of misinformation: Systematic review and meta-analysis. Journal of Medical Internet Research, 25, e49255, 1-18. https://doi.org/10.2196/49255 Maertens, R., Roozenbeek, J., Basol, M., & van der Linden, S. (2021). Long-term effectiveness of inoculation against misinformation: Three longitudinal experiments. Journal of Experimental Psychology: Applied, 27(1), 1-16. https://doi.org/10.1037/xap0000315 McGuire, W. J. (1964). Inducing resistance to persuasion: Some contemporary approaches. Advances in Experimental Social Psychology, 1, 191-229. https://doi.org/10.1016/S0065- 2601(08)60052-0 McGuire, W. J., & Papageorgis, D. (1961). The relative efficacy of various types of prior belief- defense in producing immunity against persuasion. The Journal of Abnormal and Social Psychology, 62(2), 327-337. https://doi.org/10.1037/h0042026 McGuire, W. J., & Papageorgis, D. (1962). Effectiveness of forewarning in developing resistance to persuasion. Public Opinion Quarterly, 26(1), 24-34. https://doi.org/10.1086/267068 Modirrousta-Galian, A., & Higham, P. A. (2023a). Gamified inoculation interventions do not improve discrimination between true and fake news: Reanalyzing existing research with receiver operating characteristic analysis. Journal of Experimental Psychology: General, 152(9), 2411–2437. https://doi.org/10.1037/xge0001395 Modirrousta-Galian, A., Higham, P. A., & Seabrooke, T. (2023b). Effects of inductive learning and gamification on news veracity discernment. Journal of Experimental Psychology: Applied, 29(3), 599–619. https://doi.org/10.1037/xap0000458 Nan, X., Wang, Y., & Thier, K. (2022). Why do people believe health misinformation and who is at risk? A systematic review of individual differences in susceptibility to health misinformation. Social Science & Medicine, 314, 115398, 1-17. https://doi.org/10.1016/j.socscimed.2022.115398 44 Pennycook, G., Berinsky, a., Bhargava, P., Cole, R., Goldberg, B., Lewandowsky, S., & Rand, D. G. (2023, August 21). Misinformation inoculations must be boosted by accuracy prompts to improve judgments of truth. https://doi.org/10.31234/osf.io/5a9xq Pennycook, G., Epstein, Z., Mosleh, M., Arechar, A. A., Eckles, D., & Rand, D. G. (2021). Shifting attention to accuracy can reduce misinformation online. Nature, 592(7855), 590- 595. Romer, D., Reese, Z., & Jamieson, P. E. (2020). Misleading tobacco content is on the rise on YouTube. Harvard Kennedy School Misinformation Review, 1(2). Roozenbeek, J., Maertens, R., McClanahan, W., & van der Linden, S. (2021). Disentangling item and testing effects in inoculation research on online misinformation: Solomon revisited. Educational and Psychological Measurement, 81(2), 340-362. https://doi.org/10.1177/0013164420940378 Roozenbeek, J., & van der Linden, S. (2020). Breaking Harmony Square: A game that “inoculates” against political misinformation. Harvard Kennedy School Misinformation Review, 1(8), 1-26. https://doi.org/10.37016/mr-2020-47 Roozenbeek, J., & van der Linden, S. (2022). How to combat health misinformation: A psychological approach. American Journal of Health Promotion, 36(3), 569-575. https://doi.org/10.1177/08901171211070958 Roozenbeek, J., Schneider, C. R., Dryhurst, S., Kerr, J., Freeman, A. L., Recchia, G., … & Van Der Linden, S. (2020). Susceptibility to misinformation about COVID-19 around the world. Royal Society Open Science, 7(10), 201199, 1-15. https://doi.org/10.1098/rsos.201199 Roozenbeek, J., van der Linden, S., & Nygren, T. (2020). Prebunking interventions based on “inoculation” theory can reduce susceptibility to misinformation across cultures. Harvard Kennedy School Misinformation Review, 1(2), 1-23. https://doi.org/10.37016//mr-2020- 008 Saleh, N. F., Roozenbeek, J. O. N., Makki, F. A., McClanahan, W. P., & Van Der Linden, S. (2021). Active inoculation boosts attitudinal resistance against extremist persuasion techniques: A novel approach towards the prevention of violent extremism. Behavioural Public Policy, 1-24. https://doi.org/10.1017/bpp.2020.60 Sallam, M., Dababseh, D., Eid, H., Al-Mahzoum, K., Al-Haidar, A., Taim, D., ... & Mahafzah, A. (2021). High rates of COVID-19 vaccine hesitancy and its association with conspiracy beliefs: a study in Jordan and Kuwait among other Arab countries. Vaccines, 9(1), 42, 1- 16. https://doi.org/10.3390/vaccines9010042 Sallam, M., Ghazy, R. M., Al-Salahat, K., Al-Mahzoum, K., AlHadidi, N. M., Eid, H., Kareem, … & Al Awaidy, S. T. (2022). The role of psychological factors and vaccine conspiracy beliefs in influenza vaccine hesitancy and uptake among Jordanian healthcare workers 45 during the COVID-19 pandemic. Vaccines, 10(8), 1355. https://doi.org/10.3390/vaccines10081355 Traberg, C. S., Roozenbeek, J., & van der Linden, S. (2022). Psychological inoculation against misinformation: Current evidence and future directions. The Annals of the American Academy of Political and Social Science, 700(1), 136-151. https://doi.org/10.1177/00027162221087936 UNESCO. (2022). UNESCO produced visuals, graphic and social media messages to counter disinformation, fight discrimination, and promote best practices. https://en.unesco.org/covid19/communicationinformationresponse/visualresources Van Bavel, J. J., Harris, E. A., Pärnamets, P., Rathje, S., Doell, K. C., & Tucker, J. A. (2021). Political psychology in the digital (mis) information age: A model of news belief and sharing. Social Issues and Policy Review, 15(1), 84-113. https://doi.org/10.1111/sipr.12077 Vosoughi, S., Roy, D., & Aral, S. (2018). The spread of true and false news online. Science, 359(6380), 1146-1151. https://doi.org/10.1126/science.aap9559 WHO. (2022). Advocacy materials related to road traffic injuries. https://www.who.int/teams/social-determinants-of-health/safety-and-mobility/advocacy- materials Wolters, N., & Abir, M. (2022). Jordan: Challenges and successes in COVID-19 pandemic response. The RAND Blog. https://www.rand.org/pubs/commentary/2022/06/jordan- challenges-and-successes-in-covid-19-pandemic.html? World Economic Forum. (2023). The global risks report 2023 18th edition: Insight report. https://www3.weforum.org/docs/WEF_Global_Risks_Report_2023.pdf Zhao, S., Hu, S., Zhou, X., Song, S., Wang, Q., Zheng, H., ... & Hou, Z. (2023). The prevalence, features, influencing factors, and solutions for COVID-19 vaccine misinformation: Systematic review. JMIR Public Health and Surveillance, 9(1), e40201, 1-23. https://doi.org/10.2196/40201 46 Appendix A Active Inoculation (Game) Stimuli 47 Appendix B Passive Inoculation (Infographics) Stimuli Below, we provide the infographics in the English and Arabic. 48 Appendix C Placebo Stimuli 49 Appendix D Evaluation Items Accuracy in Detecting Misinformation Tactics Respondents were shown the below headlines and asked ‘Does this headline use any misinformation techniques?’ on the following scale: (1) Definitely is misinformation (2) Probably is misinformation (3) Probably not misinformation (4) Definitely not misinformation 1. Security Guard at Vaccine Supplier: 'All COVID Vaccinated People at Risk of Heart Failure’ (Probably or Definitely Misinformation accepted as correct; Fake expert) 2. Masks linked to horrific spike in COVID cases among helpless pregnant women (Probably or Definitely Misinformation accepted as correct; Emotional appeal) 3. City's public health official announces 752 new COVID cases this week, up from 600 last week (Probably not or Definitely not Misinformation accepted as correct) 4. Ministry of Health Promotes Fifth Shot for all Citizens in Facebook Ad (see image above) (Probably or Definitely Misinformation accepted as correct; Document manipulation) 5. Testing requirements to be dropped for vaccinated travelers (Probably not or Definitely not Misinformation accepted as correct) 6. More research needed to understand risks of long-term effects of COVID (Probably not or Definitely not Misinformation accepted as correct) Likelihood of Sharing Respondents were shown the below headlines and asked ‘How likely are you to share this headline with others?’ on the following scale: (5) Very unlikely to share (6) Unlikely to share (7) Likely to share (8) Very likely to share 1. More people are getting COVID-19 twice, suggesting immunity weakens quickly in some 2. Randomly presented one of two types of misinformation: • Military general admits toxic vaccine ingredients are harming children worldwide (Fake experts) • COVID-19 vaccines cause shocking, untreatable changes to human DNA (emotional appeals) 50 Appendix E Balance Table 2- 3-Brief 0- 1- Comprehensive 4-Passive Active Diff Diff Diff Diff All Control Placebo Active Inoculation Inoculation (0)-(1) (2)-(1) (3)-(1) (4)-(1) (n=508) (n=603) Inoculation (n=618) (n=574) (n=548) COVID Vaccination Status= 0.85 0.85 0.85 0.84 0.87 0.84 -0.01 -0.01 0.02 -0.02 Vaccinated (0.36) (0.36) (0.35) (0.37) (0.34) (0.37) (0.02) (0.02) (0.02) (0.02) COVID Vaccination Status= 0.05 0.05 0.05 0.05 0.04 0.05 0.00 0.00 0.00 0.00 Unvaccinated, but plan to (0.21) (0.21) (0.21) (0.22) (0.2) (0.21) (0.01) (0.01) (0.01) (0.01) COVID Vaccination Status= Unvaccinated, and don't 0.05 0.04 0.05 0.06 0.03 0.05 -0.01 0.01 -0.02 0.00 want to (0.21) (0.19) (0.22) (0.24) (0.18) (0.22) (0.01) (0.01) (0.01) (0.01) COVID Vaccination Status= 0.06 0.07 0.05 0.05 0.05 0.07 0.02 0.01 N/A (0.23) (0.25) (0.22) (0.21) (0.23) (0.25) (0.01) (0.01) (0.01) (0.01) Gender= Male 0.63 0.62 0.66 0.6 0.62 0.65 -0.03 -0.06** -0.03 -0.01 (0.48) (0.49) (0.48) (0.49) (0.49) (0.48) (0.03) (0.03) (0.03) (0.03) Gender= Female 0.33 0.34 0.31 0.36 0.33 0.32 0.03 0.06** 0.02 0.01 (0.47) (0.47) (0.46) (0.48) (0.47) (0.47) (0.03) (0.03) (0.03) (0.03) Gender= Other 0.04 0.04 0.04 0.04 0.05 0.03 0.00 0.00 0.01 0.00 (0.19) (0.19) (0.19) (0.19) (0.21) (0.18) (0.01) (0.01) (0.01) (0.01) Age= 18-29 0.53 0.54 0.5 0.54 0.56 0.51 0.04 0.04 0.06* 0.01 (0.5) (0.5) (0.5) (0.5) (0.5) (0.5) (0.03) (0.03) (0.03) (0.03) Age= 30-39 0.25 0.24 0.25 0.25 0.25 0.27 -0.01 -0.01 -0.01 0.02 (0.43) (0.43) (0.44) (0.43) (0.43) (0.44) (0.03) (0.03) (0.03) (0.03) Age= 40-49 0.13 0.14 0.16 0.12 0.11 0.13 -0.02 -0.04* -0.05** -0.03 (0.34) (0.35) (0.37) (0.33) (0.32) (0.34) (0.02) (0.02) (0.02) (0.02) Age= 50-59 0.04 0.03 0.04 0.04 0.03 0.04 -0.01 0.00 -0.01 0.00 (0.19) (0.18) (0.2) (0.2) (0.16) (0.19) (0.01) (0.01) (0.01) (0.01) Age= 60+ 0.01 0.01 0.01 00 0.01 0.01 0.00 -0.01** -0.01 0.00 (0.1) (0.12) (0.11) (0.04) (0.08) (0.1) (0.01) (0.01) (0.01) (0.01) Age= N/A 0.04 0.04 0.03 0.05 0.05 0.04 0.01 0.01 0.02 0.01 (0.2) (0.19) (0.18) (0.21) (0.22) (0.2) (0.01) (0.01) (0.01) (0.01) Education Level= No 0.03 0.03 0.02 0.04 0.04 0.03 0.01 0.01 0.01 0.01 (0.18) (0.18) (0.16) (0.19) (0.19) (0.18) (0.01) (0.01) (0.01) (0.01) Education Level= Primary 0.12 0.09 0.13 0.13 0.12 0.13 -0.03* 0.00 -0.01 0.00 (0.32) (0.29) (0.33) (0.34) (0.33) (0.33) (0.02) (0.02) (0.02) (0.02) Education Level= 0.49 0.49 0.5 0.45 0.49 0.5 -0.01 -0.06* -0.01 -0.01 Secondary (0.5) (0.5) (0.5) (0.5) (0.5) (0.5) (0.03) (0.03) (0.03) (0.03) Education Level= Tertiary 0.3 0.31 0.3 0.32 0.29 0.28 0.01 0.03 0.00 -0.01 (0.46) (0.46) (0.46) (0.47) (0.46) (0.45) (0.03) (0.03) (0.03) (0.03) Education Level= N/A 0.06 0.07 0.05 0.06 0.05 0.06 0.02 0.01 0.00 0.01 (0.24) (0.26) (0.22) (0.24) (0.23) (0.24) (0.01) (0.01) (0.01) (0.01) Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 51 Appendix F Age and Gender Distributions for Users Who Started WhatsApp Conversations Figure F1. Age and Gender Distributions for Users Who Started WhatsApp Conversations Note. Data was directly observed on the Facebook Ad Manager. Figure F2. Age and Gender Distributions for Study Participants 52 Appendix G Table G1. Median Completion Time (Minutes) Comprehensive Brief Passive Overall Control Placebo Active Active Inoculation Inoculation Inoculation Intervention 5 N/A 5 9 6 4 Evaluation 4 5 4 5 4 4 Survey 2 2 2 2 2 2 Total 13 9 13 19 15 12 Note. Participants may have engaged in the intervention when they were available, as they were allowed to continue to do so. Completion times reflect median of all participants who completed a given stage, not the final sample that completed the study. 53 Appendix H Score Distributions 54 Appendix I Attention Check Results 55 Appendix J Robustness Checks Table J1. Impact of treatments on discernment in judgements and sharing of misinformation for with and without covariates (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) Confidence on Misinformation Misinformation Misinformation Misleading Headline Non-Misleading identifying Judgements of Judgements of Non- Sharing Discernment Game difficulty Recommend the game Discernment Sharing Headline Sharing misinformation after Misleading Headline Misleading Headline the game Control -0.068 -0.067 -0.051 -0.054 0.040 0.037 -0.129** -0.126** 0.063 0.068 -0.090 -0.080 -0.063 -0.055 0.024 0.015 -0.085 -0.069 (0.061) (0.059) (0.061) (0.061) (0.059) (0.058) (0.059) (0.059) (0.061) (0.061) (0.058) (0.058) (0.062) (0.061) (0.062) (0.062) (0.063) (0.059) [0.731] [0.828] [0.795] [0.828] [0.795] [0.828] [0.208] [0.245] [0.793] [0.828] [0.624] [0.717] [0.795] [0.828] [0.795] [0.828] [0.685] [0.828] Comprehensive Active - - - - 0.274*** 0.285*** 0.154*** 0.155*** 0.223*** 0.232*** -0.041 -0.027 0.150** 0.152*** 0.288*** 0.286*** 0.081 0.097* Inoculation 0.206*** 0.218*** 0.307*** 0.304*** (0.059) (0.057) (0.058) (0.058) (0.057) (0.056) (0.059) (0.060) (0.058) (0.058) (0.057) (0.057) (0.059) (0.059) (0.056) (0.057) (0.058) (0.055) [0.000] [0.000] [0.033] [0.026] [0.000] [0.000] [0.000] [0.000] [0.000] [0.000] [0.496] [0.629] [0.033] [0.029] [0.000] [0.000] [0.309] [0.140] - - - - Brief Active Inoculation 0.124** 0.140** 0.311*** 0.311*** 0.103* 0.084 0.055 0.062 0.200*** 0.201*** 0.132** 0.122** 0.115** 0.133** 0.339*** 0.334*** 0.271*** 0.257*** (0.057) (0.056) (0.059) (0.059) (0.059) (0.059) (0.056) (0.056) (0.057) (0.057) (0.059) (0.058) (0.058) (0.058) (0.060) (0.060) (0.056) (0.052) [0.142] [0.035] [0.000] [0.000] [0.142] [0.258] [0.376] [0.258] [0.000] [0.000] [0.000] [0.000] [0.001] [0.001] [0.142] [0.117] [0.142] [0.035] Passive Inoculation -0.056 -0.041 0.029 0.037 0.093* 0.080 -0.014 -0.003 -0.113* -0.118** -0.128** -0.120** 0.001 0.007 -0.002 -0.007 -0.031 -0.018 (0.058) (0.056) (0.057) (0.057) (0.056) (0.054) (0.055) (0.055) (0.058) (0.057) (0.056) (0.055) (0.058) (0.058) (0.058) (0.058) (0.058) (0.056) [0.842] [0.975] [0.984] [0.975] [0.382] [0.607] [0.984] [0.999] [0.382] [0.224] [0.113] [0.202] [0.984] [0.999] [0.984] [0.999] [0.984] [0.991] Age = 30-39 0.110** 0.059 -0.085* 0.013 -0.031 -0.015 0.014 0.007 -0.012 (0.045) (0.046) (0.045) (0.045) (0.045) (0.046) (0.045) (0.044) (0.043) - Age = 40-49 0.196*** 0.099* 0.140** -0.068 0.097* 0.039 -0.097* 0.120** 0.156*** (0.056) (0.059) (0.057) (0.057) (0.056) (0.058) (0.057) (0.058) (0.051) - Age = 50-59 0.238*** -0.063 0.143 0.068 0.237** 0.009 -0.228** 0.158* 0.340*** (0.090) (0.089) (0.096) (0.102) (0.103) (0.108) (0.097) (0.096) (0.081) Age = 60+ 0.101 -0.006 -0.127 0.290** 0.068 0.409** -0.218 -0.174 0.174 (0.164) (0.199) (0.173) (0.146) (0.208) (0.192) (0.215) (0.222) (0.160) Age = N/A 0.229** 0.181* -0.130 0.112 -0.240** -0.105 -0.049 -0.024 -0.402** (0.114) (0.104) (0.117) (0.125) (0.109) (0.113) (0.126) (0.129) (0.160) - Education = Primary 0.294*** -0.215 0.091 0.247** 0.351*** -0.099 0.068 0.024 0.532*** (0.112) (0.135) (0.127) (0.114) (0.120) (0.124) (0.129) (0.145) (0.119) - Education = Secondary 0.513*** -0.051 0.136 0.172 0.330*** -0.083 0.164 0.044 0.663*** 56 (0.103) (0.124) (0.117) (0.104) (0.110) (0.113) (0.119) (0.136) (0.111) - Education = Tertiary 0.954*** 0.165 0.279*** -0.081 0.248** -0.087 0.145 -0.080 1.022*** (0.105) (0.126) (0.118) (0.106) (0.112) (0.115) (0.120) (0.138) (0.114) - Education = N/A 0.568*** 0.042 0.098 -0.068 0.049 -0.310** 0.176 -0.399** 0.654*** (0.135) (0.148) (0.144) (0.138) (0.134) (0.139) (0.153) (0.170) (0.165) - Gender = Female -0.104** 0.101** -0.028 0.137*** 0.103** 0.028 -0.007 0.120*** 0.153*** (0.039) (0.041) (0.040) (0.040) (0.040) (0.040) (0.041) (0.041) (0.037) - - Gender = Other -0.048 -0.036 0.029 0.023 -0.077 -0.050 0.324*** 0.440*** 0.637*** (0.096) (0.095) (0.103) (0.120) (0.099) (0.103) (0.113) (0.123) (0.143) Vaccinated for COVID-19 -0.030 -0.090 -0.037 -0.066 0.255*** 0.175** -0.153* 0.107 0.002 = No, but plan to (0.088) (0.086) (0.081) (0.085) (0.091) (0.082) (0.092) (0.083) (0.076) Vaccinated for COVID-19 -0.179** -0.151* 0.094 -0.192** 0.178** -0.050 -0.213** -0.022 -0.231** = No, and don't want to (0.089) (0.090) (0.093) (0.086) (0.088) (0.089) (0.087) (0.091) (0.093) Vaccinated for COVID-19 - - - -0.058 -0.137* -0.149* 0.258*** 0.080 0.225** = N/A 0.253*** 0.364*** 0.354*** (0.078) (0.078) (0.078) (0.084) (0.086) (0.079) (0.098) (0.093) (0.117) - Constant -0.054 -0.089** -0.064 -0.008 0.766*** -0.022 -0.200* 0.133*** -0.007 0.106*** -0.242** -0.064 0.071 -0.084** -0.227 -0.022 0.003 0.675*** (0.041) (0.108) (0.041) (0.128) (0.040) (0.122) (0.038) (0.109) (0.042) (0.115) (0.039) (0.118) (0.042) (0.124) (0.041) (0.141) (0.040) (0.116) Controls No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes Observations 2,851 2,851 2,851 2,851 2,851 2,851 2,851 2,851 2,851 2,851 2,851 2,851 2,851 2,851 2,851 2,851 2,851 2,851 R-squared 0.016 0.090 0.017 0.040 0.012 0.069 0.013 0.025 0.025 0.057 0.009 0.028 0.010 0.041 0.012 0.026 0.005 0.092 Note. OLS estimates of treatment effects. Base comparison groups are the Control condition, Age=18-29, Education=No education, Gender=Male, Vaccinated for COVID- 19Control variables are age, education, gender, and vaccination status for COVID-19. Robust standard errors are shown in parenthesis, and Westfall-Young stepdown adjusted p- values in brackets (1,000 bootstrap replications). *** p<0.01, ** p<0.05, * p<0.1 57 Table J2. Impact of treatments on misinformation discernment for items measuring appeal to emotions and false experts. (1) Misinformation Discernment Control -0.091 (0.059) Comprehensive Active Inoculation 0.306*** (0.056) Brief Active Inoculation 0.149*** (0.056) Passive Inoculation -0.044 (0.056) Controls Yes Observations 2,851 R-squared 0.100 58 Table J3. Impact of treatments on discernment in judgements and sharing of misinformation with country weights (1) (2) (3) (4) (5) (6) (7) (8) (9) Misinformation Misinformation Confidence on Misinformation Judgements of Judgements of Sharing Misleading Non-Misleading identifying Recommend the Game difficulty Discernment Misleading Non-Misleading Discernment Headline Sharing Headline Sharing misinformation game Headline Headline after the game Control -0.127 -0.015 0.141 0.027 -0.068 -0.035 0.138 -0.252 -0.228** (0.120) (0.168) (0.095) (0.146) (0.175) (0.132) (0.219) (0.204) (0.096) [0.855] [0.993] [0.769] [0.993] [0.983] [0.993] [0.957] [0.831] [0.443] Comprehensive Active 0.038 -0.096 -0.124 0.118 -0.314*** -0.172 0.239 0.172 0.015 Inoculation (0.108) (0.111) (0.090) (0.094) (0.121) (0.130) (0.193) (0.153) (0.070) [0.934] [0.819] [0.819] [0.819] [0.214] [0.819] [0.819] [0.819] [0.934] Brief Active Inoculation -0.204 -0.034 0.220** 0.022 -0.053 -0.027 0.465* -0.183 0.030 (0.161) (0.172) (0.103) (0.096) (0.175) (0.142) (0.251) (0.244) (0.050) [0.760] [0.993] [0.514] [0.993] [0.990] [0.993] [0.564] [0.948] [0.959] Passive Inoculation -0.111 -0.094 0.058 0.008 0.060 0.069 0.188 -0.184 -0.133** (0.104) (0.103) (0.100) (0.085) (0.125) (0.097) (0.173) (0.162) (0.067) [0.862] [0.862] [0.904] [0.926] [0.904] [0.882] [0.862] [0.862] [0.477] Controls Yes Yes Yes Yes Yes Yes Yes Yes Yes Observations 2,585 2,585 2,585 2,585 2,585 2,585 2,585 2,585 2,585 R-squared 0.207 0.073 0.165 0.040 0.047 0.080 0.082 0.052 0.044 OLS estimates of treatment effects. Base comparison group is the Placebo condition. Control variables are age, education, gender, and vaccination status for COVID-19. Interactions with age, education and gender are omitted. Population weights used following recent Labor data on Gender, Age and Education. Robust standard errors are shown in parenthesis and Westfall-Young stepdown adjusted p-values in brackets (1,000 bootstrap replications). *** p<0.01, ** p<0.05, * p<0.1 59 Appendix K Heterogeneous Treatment Effects Impact of treatments on accuracy in misinformation detection and sharing misinformation with heterogeneous effects for vaccination status on COVID-19 (1) (2) (3) (4) (5) (6) Misinformation Misinformation Misleading Non-Misleading Misinformation Judgements of Judgements of Sharing Headline Headline Discernment Misleading Non-Misleading Discernment Sharing Sharing VARIABLES Headline Headline Control 0.156 0.635* 0.330 -0.550* -0.086 -0.734** (0.316) (0.334) (0.399) (0.308) (0.275) (0.349) Comprehensive Active Inoculation 0.177 0.397 0.110 0.070 -0.046 0.037 (0.356) (0.336) (0.406) (0.311) (0.287) (0.387) Brief Active Inoculation 0.002 -0.095 -0.080 -0.549* 0.749** 0.094 (0.335) (0.382) (0.393) (0.318) (0.313) (0.382) Passive Inoculation 0.116 0.262 0.074 -0.635* 0.661** -0.094 (0.372) (0.361) (0.378) (0.363) (0.327) (0.361) Age = 30-39 -0.000 0.062 0.051 -0.071 0.095 0.010 (0.100) (0.106) (0.100) (0.095) (0.106) (0.098) Age = 40-49 0.152 0.218* -0.005 0.059 0.067 0.136 (0.122) (0.124) (0.117) (0.117) (0.122) (0.114) Age = 50-59 0.091 -0.092 -0.186 0.168 0.091 0.287 (0.180) (0.202) (0.161) (0.155) (0.198) (0.196) Age = 60+ 0.208 0.264 -0.036 0.024 0.164 0.191 (0.252) (0.284) (0.319) (0.156) (0.179) (0.273) Age = N/A -0.050 0.115 0.155 -0.002 -0.093 -0.094 (0.299) (0.235) (0.275) (0.224) (0.246) (0.271) Control#Age = 30-39 0.076 -0.073 -0.152 0.189 -0.239 -0.013 (0.149) (0.152) (0.145) (0.143) (0.154) (0.147) Control#Age = 40-49 -0.042 -0.274 -0.173 0.064 -0.185 -0.107 (0.188) (0.196) (0.181) (0.183) (0.181) (0.172) Control#Age = 50-59 0.127 -0.052 -0.197 0.015 -0.047 -0.029 60 (0.250) (0.274) (0.236) (0.294) (0.334) (0.323) Control#Age = 60+ 0.093 -0.110 -0.203 0.358 -0.570 -0.142 (0.374) (0.515) (0.405) (0.318) (0.440) (0.407) Control#Age = N/A 0.344 -0.122 -0.516 0.222 -0.302 -0.038 (0.421) (0.366) (0.398) (0.348) (0.349) (0.372) Comprehensive Active Inoculation#Age = 30-39 0.271* 0.191 -0.172 -0.092 -0.038 -0.146 (0.142) (0.146) (0.141) (0.145) (0.145) (0.143) Comprehensive Active Inoculation#Age = 40-49 0.120 -0.077 -0.208 0.253 -0.306* -0.004 (0.174) (0.175) (0.168) (0.187) (0.173) (0.186) Comprehensive Active Inoculation#Age = 50-59 -0.046 -0.026 0.035 -0.460 -0.122 -0.662** (0.289) (0.289) (0.305) (0.284) (0.296) (0.292) Comprehensive Active Inoculation#Age = 60+ -0.428 -0.673* -0.032 -0.483 -1.321*** -1.877*** (0.328) (0.362) (0.379) (0.299) (0.301) (0.351) Comprehensive Active Inoculation#Age = N/A 0.422 0.216 -0.334 -0.152 -0.053 -0.232 (0.369) (0.295) (0.375) (0.331) (0.328) (0.351) Brief Active Inoculation#Age = 30-39 0.271* 0.140 -0.214 0.275** -0.198 0.128 (0.140) (0.140) (0.149) (0.135) (0.141) (0.148) Brief Active Inoculation#Age = 40-49 0.204 0.035 -0.219 -0.009 -0.031 -0.041 (0.176) (0.187) (0.191) (0.175) (0.181) (0.182) Brief Active Inoculation#Age = 50-59 0.034 0.007 -0.035 -0.019 0.202 0.177 (0.259) (0.353) (0.315) (0.302) (0.330) (0.366) Brief Active Inoculation#Age = 60+ -0.597 -0.878 0.006 0.875*** 0.069 1.099*** (0.679) (0.637) (0.500) (0.337) (0.466) (0.385) Brief Active Inoculation#Age = N/A 0.315 0.109 -0.293 0.301 -0.117 0.239 (0.353) (0.321) (0.350) (0.379) (0.339) (0.373) Passive Inoculation#Age = 30-39 -0.019 -0.199 -0.139 0.059 -0.205 -0.133 (0.138) (0.145) (0.134) (0.135) (0.142) (0.137) Passive Inoculation#Age = 40-49 -0.036 -0.262 -0.171 0.092 -0.192 -0.082 (0.169) (0.176) (0.170) (0.164) (0.171) (0.169) Passive Inoculation#Age = 50-59 0.582** 0.154 -0.579** 0.335 -0.067 0.329 (0.257) (0.245) (0.232) (0.271) (0.295) (0.276) Passive Inoculation#Age = 60+ -0.134 -0.418 -0.179 0.242 0.454 0.734** (0.382) (0.418) (0.583) (0.387) (0.518) (0.328) Passive Inoculation#Age = N/A 0.279 0.227 -0.152 0.158 -0.372 -0.182 (0.400) (0.330) (0.388) (0.371) (0.335) (0.369) Education = Primary 0.371 -0.071 -0.508 -0.175 0.655*** 0.442 61 (0.297) (0.270) (0.325) (0.251) (0.225) (0.306) Education = Secondary 0.542** 0.028 -0.635** -0.273 0.605*** 0.277 (0.275) (0.235) (0.305) (0.232) (0.194) (0.290) Education = Tertiary 0.977*** 0.288 -0.949*** -0.072 0.303 0.215 (0.280) (0.242) (0.309) (0.236) (0.206) (0.295) Education = N/A 0.706** 0.342 -0.576* -0.039 0.113 0.066 (0.313) (0.261) (0.345) (0.313) (0.251) (0.347) Control#Primary -0.333 -0.433 0.049 0.310 0.256 0.618 (0.362) (0.385) (0.434) (0.358) (0.324) (0.388) Control#Secondary -0.199 -0.636* -0.279 0.349 0.203 0.613* (0.320) (0.338) (0.401) (0.308) (0.277) (0.352) Control#Tertiary -0.226 -0.592* -0.211 0.313 0.340 0.705* (0.326) (0.346) (0.404) (0.315) (0.289) (0.360) Control#Education = N/A -0.661 -1.052** -0.059 -0.112 0.690* 0.551 (0.410) (0.417) (0.483) (0.418) (0.366) (0.435) Comprehensive Active Inoculation#Primary -0.248 -0.672* -0.250 0.034 -0.145 -0.103 (0.390) (0.378) (0.434) (0.346) (0.323) (0.414) Comprehensive Active Inoculation#Secondary -0.016 -0.270 -0.201 0.217 -0.211 0.048 (0.360) (0.337) (0.408) (0.313) (0.285) (0.387) Comprehensive Active Inoculation#Tertiary 0.019 -0.325 -0.290 0.237 -0.264 0.017 (0.365) (0.344) (0.411) (0.318) (0.294) (0.394) Comprehensive Active Inoculation#Education = N/A 0.096 -0.264 -0.333 -0.009 0.077 0.066 (0.427) (0.384) (0.489) (0.422) (0.374) (0.470) Brief Active Inoculation#Primary 0.027 0.388 0.285 0.497 -0.994*** -0.399 (0.364) (0.416) (0.424) (0.347) (0.350) (0.414) Brief Active Inoculation#Secondary 0.154 0.450 0.182 0.762** -1.146*** -0.237 (0.336) (0.379) (0.394) (0.316) (0.311) (0.382) Brief Active Inoculation#Tertiary 0.049 0.352 0.228 0.556* -0.996*** -0.331 (0.345) (0.387) (0.398) (0.325) (0.320) (0.388) Brief Active Inoculation#Education = N/A 0.032 0.120 0.059 0.169 -0.784** -0.576 (0.400) (0.433) (0.457) (0.419) (0.388) (0.470) Passive Inoculation#Primary 0.007 -0.065 -0.062 0.496 -0.872** -0.278 (0.396) (0.396) (0.405) (0.383) (0.356) (0.385) Passive Inoculation#Secondary -0.187 -0.079 0.162 0.707** -0.846*** -0.004 (0.371) (0.357) (0.378) (0.359) (0.324) (0.359) Passive Inoculation#Tertiary -0.069 -0.161 -0.048 0.619* -0.804** -0.066 62 (0.375) (0.363) (0.383) (0.365) (0.333) (0.367) Passive Inoculation#Education = N/A -0.305 -0.432 0.017 0.518 -0.613 0.003 (0.468) (0.422) (0.456) (0.457) (0.395) (0.436) Gender = Female -0.108 -0.135 0.020 0.053 0.091 0.153* (0.091) (0.097) (0.084) (0.087) (0.093) (0.084) Gender = Other -0.060 -0.030 0.048 0.009 -0.272 -0.259 (0.206) (0.182) (0.246) (0.177) (0.205) (0.210) Control#Female -0.004 0.003 0.008 0.166 -0.080 0.116 (0.132) (0.139) (0.125) (0.130) (0.135) (0.125) Control#Other gender 0.235 0.078 -0.222 0.287 0.181 0.518* (0.285) (0.278) (0.313) (0.351) (0.306) (0.304) Comprehensive Active Inoculation#Female -0.008 0.086 0.080 -0.159 0.055 -0.133 (0.125) (0.131) (0.121) (0.129) (0.127) (0.122) Comprehensive Active Inoculation#Other gender 0.378 0.379 -0.148 -0.008 0.243 0.231 (0.282) (0.247) (0.320) (0.334) (0.285) (0.301) Brief Active Inoculation#Female -0.069 0.182 0.233* -0.247** -0.018 -0.309** (0.124) (0.132) (0.129) (0.123) (0.127) (0.128) Brief Active Inoculation#Other gender -0.342 -0.148 0.294 -0.025 0.249 0.218 (0.288) (0.301) (0.318) (0.327) (0.306) (0.323) Passive Inoculation#Female -0.140 -0.100 0.087 -0.151 0.259** 0.079 (0.123) (0.131) (0.119) (0.122) (0.128) (0.121) Passive Inoculation#Other gender -0.178 -0.330 -0.054 -0.234 0.408 0.129 (0.315) (0.271) (0.365) (0.360) (0.312) (0.334) Vaccinated for COVID-19 = No, but plan to -0.060 0.037 0.103 -0.119 0.408* 0.264 (0.225) (0.207) (0.175) (0.198) (0.214) (0.179) Vaccinated for COVID-19 = No, and don't want to -0.016 -0.059 -0.029 -0.084 0.038 -0.062 (0.190) (0.180) (0.203) (0.189) (0.178) (0.191) Vaccinated for COVID-19 = N/A -0.074 -0.204 -0.077 -0.258 0.174 -0.132 (0.192) (0.184) (0.169) (0.157) (0.201) (0.166) Control#Unvaccinated but plan to 0.072 -0.080 -0.153 -0.145 -0.228 -0.396 (0.292) (0.293) (0.246) (0.257) (0.297) (0.257) Control#Unvaccinated and don't want to -0.452 -0.319 0.287 -0.351 0.325 -0.092 (0.318) (0.296) (0.311) (0.244) (0.289) (0.280) Control#Vaccinated for COVID-19 = N/A -0.043 0.015 0.064 0.194 0.176 0.402* (0.249) (0.252) (0.243) (0.229) (0.261) (0.227) 63 Comprehensive Active Inoculation#Unvaccinated but plan to 0.269 0.194 -0.168 0.229 -0.373 -0.099 (0.286) (0.260) (0.255) (0.310) (0.284) (0.262) Comprehensive Active Inoculation#Unvaccinated and don't want to 0.095 -0.101 -0.198 0.299 -0.096 0.258 (0.255) (0.250) (0.260) (0.263) (0.235) (0.272) Comprehensive Active Inoculation#Vaccinated for COVID-19 = N/A 0.028 -0.033 -0.061 0.177 0.129 0.336 (0.265) (0.265) (0.241) (0.272) (0.292) (0.251) Brief Active Inoculation#Unvaccinated but plan to -0.343 -0.754** -0.202 -0.011 -0.077 -0.090 (0.316) (0.297) (0.268) (0.280) (0.316) (0.268) Brief Active Inoculation#Unvaccinated and don't want to -0.693*** -0.339 0.562 -0.549** 0.278 -0.372 (0.247) (0.330) (0.349) (0.234) (0.315) (0.319) Brief Active Inoculation#Vaccinated for COVID-19 = N/A 0.361 -0.179 -0.585** 0.135 -0.131 0.029 (0.248) (0.257) (0.235) (0.282) (0.279) (0.270) Passive Inoculation#Unvaccinated but plan to 0.151 0.011 -0.174 0.250 -0.167 0.130 (0.267) (0.247) (0.238) (0.243) (0.272) (0.236) Passive Inoculation#Unvaccinated and don't want to -0.073 0.126 0.192 -0.172 0.353 0.147 (0.265) (0.245) (0.279) (0.277) (0.270) (0.260) Passive Inoculation#Vaccinated for COVID-19 = N/A -0.011 0.155 0.141 0.138 0.183 0.343 (0.256) (0.238) (0.239) (0.237) (0.277) (0.228) Constant -0.684** -0.197 0.668** 0.168 -0.433** -0.231 (0.277) (0.239) (0.309) (0.238) (0.202) (0.294) Observations 2,851 2,851 2,851 2,851 2,851 2,851 R-squared 0.112 0.067 0.086 0.047 0.079 0.055 OLS estimates of treatment effects. Base comparison groups are the Control condition, Age=18-29, Education=No education, Gender=Male, Vaccinated for COVID-19. Control variables are age, education, gender, and vaccination status for COVID-19. Robust standard errors are shown in parenthesis). *** p<0.01, ** p<0.05, * p<0.1 64