Policy Research Working Paper 11063 Does Free Sound Too Cheap? The Adverse Effect of a Randomized Text Message Campaign on Program Take-up Jeannie Annan Estelle Koussoubé Joséphine Tassy Léa Rouanet Clara Delavallade David Evans Africa Region Gender Innovation Lab February 2025 Policy Research Working Paper 11063 Abstract This study conducted a randomized experiment to improve their contact was also female. Qualitative findings suggest participation in a youth employment program in Côte that distrust among unfamiliar contacts contributed to d’Ivoire by testing text message outreach methods. Send- this decline. The study highlights the importance of tailor- ing text messages highlighting that the program was free ing communication strategies in job training programs to only to eligible youth had no impact, but messages sent to increase effectiveness, considering recipients’ relationships both youth and trusted contacts led to reduced enrollment. and trust. This negative effect was smaller for women, and null when This paper is a product of the Gender Innovation Lab, Africa Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at jtassy@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Does Free Sound Too Cheap? The Adverse Effect of a Randomized Text Message Campaign on Program Take-up∗ Jeannie Annan ○ e○ r Estelle Koussoub´ r Jos´ ephine Tassy ○ r Clara Delavallade ○ ea Rouanet ○ r L´ r David Evans Keywords: youth employment, text message incentives, program take-up. JEL Codes: O15, J16, J24, D83 ∗ Annan: International Rescue Committee, jeannie.annan@rescue.org. Koussoub´ e: World Bank, mkoussoube@worldbank.org. Tassy: World Bank, jtassy@worldbank.org. Rouanet: World Bank, lrouanet@worldbank.org. Delavallade: World Bank, cdelavallade@worldbank.org. Evans: Inter-American Development Bank, devans@iadb.org. The order of authors’ names was randomized. This paper is a prod- uct of the Gender Innovation Lab, within the Office of the World Bank’s Africa Region Chief Economist. We gratefully acknowledge financial support from and the Wellspring Philanthropic Fund and the World Bank’s Umbrella Facility for Gender Equality (UFGE). The UFGE is a multi-donor trust fund administered by the World Bank to advance gender equality and women’s empowerment through experimentation and knowledge creation to help governments and the private sector focus policy and programs on scalable solutions with sustainable outcomes. The findings, interpretations, and conclusions expressed in this work do not neces- sarily reflect the views of The World Bank Group, its Board of Executive Directors, or the governments it represents. All errors and omissions are our own. 1 Introduction With Sub-Saharan Africa’s working-age population expected to reach 1.3 billion by 2050, making up 25% of the global labor force (World Bank, 2023), and 20%-30% of youth cur- rently not in employment, education, or training (UNICEF, 2022), there is an urgent need to equip young people with the skills required to secure meaningful employment. Although there is significant heterogeneity across programs, the average effects of technical and voca- tional education and training (TVET) programs on youth employment have been positive in both higher and lower income country settings (Hanushek et al., 2017; Tripney and Hombra- dos, 2013). However, low take-up rates significantly challenge the effectiveness of training programs in such settings (McKenzie and Woodruff, 2014). Improving uptake of TVET pro- grams in Africa is thus a potentially important step in introducing skilled individuals into the labor market and reducing youth unemployment. The range of TVET offerings in Africa is extensive, with participant costs varying signifi- cantly (Van Lieshout and Mehtha, 2017). Removing cost barriers has been shown to increase product usage in low- and middle-income countries (Cohen and Dupas, 2010; Kremer and Miguel, 2007) as well as participation in basic education (Duflo et al., 2023). Analogously, reducing costs could boost enrollment in training programs. One tool that has proven effec- tive at shifting behaviors at low cost in some contexts is sending informational text messages acio and Vicente 2021; Rodr´ (Gr´ ıguez and Saavedra 2019), making use of growing mobile phone penetration in low and middle income countries. This study uses a randomized controlled trial to test the impact of sending text messages (i.e., short message service, or SMS messages) to youth and—in some cases—a trusted contact the youth have identified in order to increase uptake of a youth employment program ote d’Ivoire. We find that SMS messages that make salient the fact that the training in Cˆ program is free can discourage enrollment. Specifically, we show that the negative impact on enrollment is significant only when both the potential applicant and their contact are sent SMS messages. This finding is consistent with the theoretical literature that suggests 1 pricing can serve as a signal of quality, with higher prices indicating better product quality ote d’Ivoire, (Bagwell and Riordan, 1991). Furthermore, in collectivist cultures such as Cˆ youth career choices are heavily influenced by family expectations (Akosah-Twumasi et al., 2018). The support and approval from family members are pivotal, and youth are more likely to choose careers that are perceived to enhance the family’s social standing and fulfill familial obligations. Moreover, our evidence shows that this influence varies according to the gender of both the youth and their contacts. The discouragement effect is significantly less for women, and it is not statistically significant for women whose contact is also a woman. We complement our quantitative results with qualitative interviews with the youths’ contacts, which demonstrate that the perception of the program by those contacts plays a role. The decision to enroll in a training program is based on both cost and the program’s reputation. We conclude the paper with a discussion on the broader implications of our findings and potential areas for future research. 2 Intervention, Empirical Strategy and Data This study evaluates the impact of an SMS intervention to increase interest and take-up in the PRO-Jeunes youth employment program, which was implemented in urban and peri- ote d’Ivoire by the International Rescue Committee. The PRO-Jeunes program urban Cˆ was extensively publicized through a comprehensive communication strategy, including a public relations campaign, direct marketing, media activities, and community mobilization. The public relations campaign engaged local authorities, professional training centers, em- ployment agencies, social centers, and private enterprises. Media activities included press releases, advertisements, and direct broadcasts on the PRO-Jeunes website and on social media platforms (Twitter, YouTube, Facebook), as well as on local and regional television and radio. Additionally, community mobilization initiatives involved meeting with commu- nity leaders, distributing flyers, and informing communities about the project launch and 2 enrollment sites. Recruitment for the PRO-Jeunes program was carried out in two stages: (i) interested youth first registered, expressing their interest in the program; (ii) after screening all the applications, the project team invited eligible applicants to come to an “enrollment meet- ing.” According to data from the first two cohorts from the program, a significant share of eligible applicants who had expressed their interest in stage 1 did not attend the subsequent enrollment meeting in stage 2. The intervention evaluated in this study targeted the next cohort, and took place between the two stages, aiming to increase take-up from registration to enrollment. Eligible applicants were randomly selected to receive an SMS reminding them that the program was free.1 Youth were already informed that the program was free at pre-registration and the message served as a reminder. Randomization was stratified by gender and geo- graphic location. The SMS message read as follows: “Come participate in an information session about PRO-Jeunes, a program offering free training, support for self-employment and employment opportunities.” In the first treatment, the SMS was randomly sent to the youth only. In the second, treatment, the SMS was sent to the youth and their listed contact person. At pre-registration, youth were asked to list a person to contact in case they could not be reached. It was specified that youth who did not live by themselves should provide the name of a person they lived with. Our study uses administrative data collected in 2019 on a sample of 2,926 eligible ap- plicants to the PRO-Jeunes program who had completed the pre-registration stage, dur- ing which they provided identification and contact information for their contact person, as described in Table 2. The data include variables such as date of birth, phone number, gender and geographic location for youth, and the relationship to the youth and phone number for contacts. Contact gender was inferred from the relationship variable. Listed re- 1 A second type of SMS emphasizing the long-term career benefits of the program was also tested. Results were inconclusive and are described in the appendix. The analysis presented in this paper focuses on the message emphasizing the free nature of the program. 3 lationships were mother/father, uncle/aunt, brother/sister, grandparents, cousin, guardian, spouse, friend and other. The average age of youth in our sample is 23 years (25th percentile: 20 years; 75th percentile: 26 years), and half of the sample are women. In terms of contacts, about half of the youth listed a parent as their contact; another 9% provided a spouse as a contact. Women are 18.5% less likely than men to choose women as contacts. 15.6% of women choosing their husband as a contact, compared to only 1.7% of men choosing their wife (Table 1). These statistics are balanced across treatment groups (Table 2). To complement the quantitative data, we conducted a qualitative survey to better un- derstand the quantitative findings, including 12 individual interviews and 4 focus group discussions among youth and their contacts in August-September 2023. In the focus group discussions, men and women were interviewed separately to better understand the gender differences in the observed results. We use the following specification for our main estimates: Enrolledi = β0 + β1 W omani + β2 T1i + β3 T1i · W omani + (1) β4 T2i + β5 T2i · W omani + Xi′ + λe,c + εi Where Enrolledi is a binary variable indicating whether individual i has completed the enrollment process. T1i is a binary variable indicating whether the individual was in the first treatment group, where an SMS was sent to youth only. T2i is a binary variable indicating whether the individual was in the second treatment group, where an SMS was sent to youth and their contact. W omani is a binary variable taking the value of one when the individual is a woman. Xi′ is a vector of control variables, including age and geographic location. We also run a heterogeneity test with the same specification, comparing two sub-groups: youth with male contacts versus youth with female contacts. 4 3 Results 3.1 Overall impact on enrollment Table 3 presents the treatment effects of sending the SMS to youth only or to youth and their contact on enrollment. We find that sending the SMS to youth only has no statistically significant impact on enrollment. Sending the SMS to both youth and their contact signifi- cantly decreases the probability of enrolling for both men (from 43.8% to 25.2%) and women (from 38.5% to 30.7%), equivalent to a 42.4% decrease for men and a 20.2% decrease for women, both significant at the 5% level. The decrease in enrollment is significantly greater for men, with men’s enrollment decreasing by 10.8 percentage points more than women’s. The fact that we only see a decrease when the contacts are included, coupled with our qualitative evidence, point to contacts being an important influence over youth’s decisions. About half of youth listed a parent as their contact (Table 2). Qualitative interviews indicate that youth select the contact based on either personal relationships (family members or friends) or the contact’s professional experience, and that youth expect their contact to provide significant guidance. Qualitative interviews also indicate that while the program being free was perceived positively by both youth and contacts, it can also signal either poor quality or potential fraud. When asked what could improve the message’s perception, interviewees suggested that it should illustrate the reputability of the organization providing the program. Both contacts and youth mentioned the importance of trusting the organization. This could ex- plain why sending the SMS to youth and their contact had a worse effect on enrollment, compared to sending it to youth only. When the SMS were sent, youth had already attended a pre-registration information session, and therefore already had some information about the organization, whereas contacts had no prior interactions with the implementing organi- zation. Contacts had no reason to have trust in the program, which could have led to them discouraging youth to register. 5 3.2 Impact on enrollment by gender of contact Table 4 shows the treatment effects of sending SMS to youth only or to youth and their contact on program enrollment for two subsamples: column (1) shows the impact on youth whose contacts are men, and column (2) shows the impact on youth whose contacts are women. It presents a more nuanced picture when we consider the gender of the contact. Sending SMS to male youth and their contact always has a negative impact, regardless of the contact’s gender: the effect corresponds to a 36.2% decrease in enrollment for men with male contacts and to a 52.3% for men with female contacts. The impact on women’s enrollment of the SMS to youth and their contact is driven by women with male contacts. Women with male contacts are 30.3% less likely to enroll, whereas women with female contacts are the only group which does not exhibit a significant decrease in enrollment. This is not explained by differences in delivery rates (Figure 1). Our results are robust to restricting the analysis to the subsample of youth whose contact is a parent: young women whose mothers were sent the contact SMS are the only ones not discouraged by the SMS. This may suggest that women value the opportunity cost of the program for young women less than men, either because they discount less the expected quality of the program based on the free message or because they are more aware than men of the limitations in job market opportunities for young women. However, these findings may be subject to selection bias in contact choice. The observed differences in effects by contact gender could be partially attributed to unobserved characteristics of youth who choose male versus female contacts, rather than solely to the gender of the contact itself. 4 Discussion Low take-up is a major barrier to the effectiveness of a large number of development programs and social policies. In some cases, that low take-up may reflect insufficient information regarding the program. This study sought to boost the uptake of a youth employment 6 program by making the free nature of the program salient through SMS reminders sent to both applicants and their contacts. We provide quantitative evidence that this had no significant impact on program enrol- ment when youth were contacted alone and a discouraging effect when youth were contacted along with a person of reference. This discouraging effect was found for all youth, except women with a female contact, and is significantly stronger among men. Complementary qualitative evidence highlights the key role played by trust in the pro- gram in shaping recipients’ perception of the SMS messages. A message highlighting the program’s cost-free aspect can inadvertently backfire if the target audience lacks familiarity with the offering, leading to skepticism about quality rather than generating interest. This finding underscores the value of building trust in the program before promoting it, ensuring that recipients are familiar with and confident in its value. Additionally, our results indicate that contacts are critical in influencing youth’s take-up decisions. While they can be effective channels for information dissemination, trust needs to be cultivated not only with the youth, but also with the contacts, especially for male participants and male contacts, among whom the discouraging effect was strongest. Focusing solely on the free nature of the program may not be enough to attract par- ticipants and can even lead to distrust when interpreted as a low-quality signal. Future communication strategies should highlight the quality and value of the program to align with the expectations and aspirations of the youth and the contacts influencing their de- cisions. Program coordinators could actively build trust with youth and their contacts by providing them with detailed, transparent information about the program’s value and quality, and incorporating this type of information in messaging strategies. Additionally, messaging strategies could be tailored to align with the specific expectations of both youth and their contacts, potentially involving pre-engagement initiatives to familiarize them with the program before direct outreach. Gender dynamics further complicate the enrollment process, particularly among female 7 recipients. The analysis indicates that the deterrent effect of the messaging is significantly less pronounced among women with female contacts than among women with male contacts. This heterogeneity suggests that the perception and impact of the program’s promotional messages are influenced by broader social and cultural factors. However, this finding should be interpreted with caution given the potential selection bias in contact choice. Future re- search could address this by randomly assigning suggested contact gender to establish causal relationships, collecting more detailed data on reasons for contact selection, and exploring the interaction between youth characteristics and contact gender. Taken together, our mixed-method study demonstrates the challenges of promoting the uptake of youth employment programs, highlighting the need for nuanced context-specific approaches. Increasing youth enrollment in skills development programs is a critical chal- lenge, often hindered by issues of trust and perceptions of program benefits. It is crucial for program coordinators to focus on building trust and familiarity before emphasizing cost-free aspects. Messaging strategies should account for the complex dynamics between youth, their contacts, and gender. Leveraging existing trusted networks, such as religious or educational institutions, could enhance program credibility. By addressing the complexities of trust, so- cial networks, and gender dynamics, program organizers can develop more effective strategies to improve registration outcomes and, ultimately, youth employment opportunities. 8 References Akosah-Twumasi, P., Emeto, T. I., Lindsay, D., Tsey, K., and Malau-Aduli, B. S. (2018). A systematic review of factors that influence youths career choices—the role of culture. In Frontiers in Education, volume 3, page 58. Frontiers Media SA. Bagwell, K. and Riordan, M. H. (1991). High and declining prices signal product quality. The American Economic Review, pages 224–239. Cohen, J. and Dupas, P. (2010). Free distribution or cost-sharing? evidence from a random- ized malaria prevention experiment. The Quarterly Journal of Economics, 125(1):1–45. Duflo, E., Dupas, P., and Kremer, M. (2023). The impact of secondary school subsidies on career trajectories in a dual labor market: Experimental evidence from ghana. Gr´acio, M. and Vicente, P. C. (2021). Information, get-out-the-vote messages, and peer influence: Causal effects on political behavior in mozambique. Journal of Development Economics, 151:102665. Hanushek, E. A., Schwerdt, G., Woessmann, L., and Zhang, L. (2017). General educa- tion, vocational education, and labor-market outcomes over the lifecycle. Journal of human resources, 52(1):48–87. Kremer, M. and Miguel, E. (2007). The illusion of sustainability. The Quarterly Journal of Economics, 122(3):1007–1065. McKenzie, D. (2021). Small business training to improve management practices in developing countries: re-assessing the evidence for ‘training doesn’t work’. Oxford Review of Economic Policy, 37(2):276–301. McKenzie, D. and Woodruff, C. (2014). What are we learning from business training and entrepreneurship evaluations around the developing world? The World Bank Research Ob- server, 29(1):48–82. ıguez, C. and Saavedra, J. E. (2019). The persistent effects of youth savings reminders: Rodr´ Experimental evidence from text-message campaigns in colombia. Journal of Development Economics, 139:135–156. Rokicki, S., Cohen, J., Salomon, J. A., and Fink, G. (2017). Impact of a text-messaging program on adolescent reproductive health: a cluster–randomized trial in ghana. American Journal of Public Health, 107(2):298–305. Tripney, J. S. and Hombrados, J. G. (2013). Technical and vocational education and training (tvet) for young people in low-and middle-income countries: a systematic review and meta- analysis. Empirical research in vocational education and training, 5:1–14. UNICEF (2022). Global neet data: Where’s it all gone? Accessed: 2024-09-30. Van Lieshout, S. and Mehtha, P. (2017). The Next 15 Million: Start and Improve Your Business Global Tracer Study 2011-15. ILO. 9 World Bank (2023). World development indicators. Accessed: 2024-09-30. 10 Tables Table 1: Descriptive statistics by gender Men Women T-test (1) (2) (1)-(2) Variable N Mean/SE N Mean/SE Age 1504 23.245 1422 23.035 0.210 [0.097] [0.101] Contact is a woman 1504 0.398 1422 0.489 -0.090*** [0.013] [0.013] Contact is youth’s mother 1504 0.277 1422 0.329 -0.052*** [0.012] [0.012] Contact is youth’s father 1504 0.250 1422 0.160 0.090*** [0.011] [0.010] Contact is youth’s mother or father 1504 0.527 1422 0.489 0.038** [0.013] [0.013] Contact is youth’s spouse 1504 0.017 1422 0.156 -0.139*** [0.003] [0.010] Notes: The value displayed for t-tests are the differences in the means across the groups. *** p < 0.01, ** p < 0.05, * p < 0.1 11 Table 2: Descriptive statistics by treatment group (1) (2) (3) SMS to youth Control SMS to youth t-test t-test t-test & contact N Mean/SE N Mean/SE N Mean/SE (1)-(2) (1)-(3) (2)-(3) Woman 1466 0.487 731 0.487 729 0.483 0.000 0.004 0.004 [0.013] [0.018] [0.019] Age 1466 23.128 731 23.064 729 23.252 0.064 -0.124 -0.188 [0.098] [0.142] [0.139] Contact is a woman 1475 0.452 736 0.413 737 0.438 0.039* 0.014 -0.025 [0.013] [0.018] [0.018] Contact is youth’s mother 1475 0.310 736 0.283 737 0.299 0.027 0.011 -0.016 [0.012] [0.017] [0.017] Contact is youth’s father 1475 0.199 736 0.196 737 0.225 0.004 -0.026 -0.030 [0.010] [0.015] [0.015] Contact is youth’s mother or father 1475 0.509 736 0.478 737 0.524 0.031 -0.015 -0.045* [0.013] [0.018] [0.018] Contact is youth’s spouse 1475 0.087 736 0.087 737 0.076 -0.000 0.011 0.011 [0.007] [0.010] [0.010] Notes: The value displayed for t-tests are the differences in the means across the groups. *** p < 0.01, ** p < 0.05, * p < 0.1 Table 3: Treatment effects on enrollment rates Completed enrollment process Woman -0.052** p-value 0.033 sharpened q-value 0.035 SMS to youth -0.023 p-value 0.431 sharpened q-value 0.275 SMS to youth x Woman 0.015 p-value 0.724 sharpened q-value 0.408 SMS to youth & contact -0.186*** p-value 0.000 sharpened q-value 0.001 SMS to youth & contact x Woman 0.108** p-value 0.011 sharpened q-value 0.024 SMS to youth: p(T + T*W)=0 0.784 SMS to youth & contact: p(T + T*W)=0 0.011 Control mean 0.438 R-squared 0.067 Observations 2926 Notes: T=Treatment. W=Woman. *** p < 0.01, ** p < 0.05, * p < 0.1 Sharpened q-values are p-values that have been adjusted for the False Discovery Rate (FDR). Controls: Age and geographic location . Table 4: Treatment effects on enrollment rates - by contact gender Completed enrolment process Contact is Contact is T-test a man a woman (1)=(2) (1) (2) Woman -0.035 -0.081** 0.249 p-value 0.296 0.026 sharpened q-value 1.000 0.027 SMS to youth -0.007 -0.050 0.514 p-value 0.846 0.285 sharpened q-value 1.000 0.166 SMS to youth x Woman -0.011 0.058 0.379 p-value 0.840 0.366 sharpened q-value 1.000 0.172 SMS to youth & contact -0.158*** -0.229*** 0.191 p-value 0.000 0.000 sharpened q-value 0.001 0.001 SMS to youth & contact x Woman 0.036 0.196*** 0.033 p-value 0.532 0.002 sharpened q-value 1.000 0.004 SMS to youth: p(T + T*W)=0 0.652 0.852 0.553 SMS to youth & contact: p(T + T*W)=0 0.005 0.452 0.090 Control mean 0.438 0.438 R-squared 0.065 0.076 Observations 1632 1294 Notes: T=Treatment. W=Woman. *** p < 0.01, ** p < 0.05, * p < 0.1. Sharpened q-values are p-values that have been adjusted for the False Discovery Rate (FDR). Controls: Age and geographic location. 14 Figure Figure 1: SMS Delivery Rates by Treatment Group 15 Appendix SMS on long-term benefits of the program Another type of SMS, emphasizing the long-term benefits of the program, was sent to either youth only or youth and their contacts. Similarly to the SMS highlighting that the program was free, it had no impact on enrollment when sent exclusively to youth. However, when sent to both youth and their contacts, the impact differed by gender: it decreased enrollment for men but increased enrollment for women. Due to the inconclusive nature of these results, we have chosen to focus on the first type of SMS in this paper. 16