Coding Bootcamps for Female Digital Employment IN FOCUS Evidence from a Randomized Control FINANCE, COMPETITIVENESS & Trial in Argentina and Colombia INNOVATION Julian Aramburu, Ana Goicoechea FIRMS, ENTREPRENEURSHIP & INNOVATION © 2021 The World Bank Group 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org All rights reserved. This volume is a product of the staff of the World Bank Group. The World Bank Group refers to the member institutions of the World Bank Group: The World Bank (International Bank for Reconstruction and Development); International Finance Corporation (IFC); and Multilateral Investment Guarantee Agency (MIGA), which are separate and distinct legal entities each organized under its respective Articles of Agreement. We encourage use for educational and non- commercial purposes. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Directors or Executive Directors of the respective institutions of the World Bank Group or the Governments they represent. The World Bank Group does not guarantee the accuracy of the data included in this work. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. All queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@worldbank.org. The research team consists of Julian Aramburu, PhD Candidate, Yale University; Ana Goicoechea, Senior Economist, World Bank; and Mushfiq Mobarak, Professor, Yale University. The team acknowledges with gratitude the contributions of field coordinators: Isabel Miranda, Agustina Suaya, and Ines Taylor and the World Bank project team, as well as Andrew Beath, Sebastian Hurtado, Sarah Lenoble, and Victor Mulas. This randomized control trial is registered in the American Economic Association RCT Registry (ID AEARCTR-0003850), and is supported by infoDev, a World Bank Group multi-donor program. Design and Layout: Amy Quach / FPS Groups Photo Credits: Freepik.com Table of Contents THE POLICY ISSUE 3 BOOTCAMPS DESIGNED TO ADDRESS WOMEN’S CONSTRAINTS 4 ATTRACTING SUFFICIENT CANDIDATES WITH A POTENTIAL TO GRADUATE 5 CHARACTERISTICS OF ELIGIBLE WOMEN 7 THE LOTTERY: USING RANDOM ASSIGNMENT TO MEASURE THE IMPACT 7 FROM HIGH SATISFACTION TO IMPACT 9 CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT | 1 2 | CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT The Policy Issue The increase in female labor force participation is among the most salient economic and social transformations in the world over the last fifty years, and Latin America is no exception. In this regard, the gender gap in educational attainment has not only narrowed, but it has reversed itself in most countries of the region (OECD 2018). Despite this, two important gaps remain. First, wages of female workers are, on average, 30 percent lower than those of males (World Economic Forum 2018). Second, a high degree of occupational and educational segregation remains, with men and women being concentrated in different fields of occupations and study (Blau and Kahn 2017). These two gaps are closely related. Indeed, Buenos Aires, Argentina and Bogotá, Colombia. gender wage differences can be attributable in Bootcamps have become a policy instrument to part to the occupations and sectors in which men tackle the following two objectives. First, they and women work (Blau and Kahn 2017). An provide training on coding skills at a time when example of this can be found in the information the rapid spread of new digital technologies is technology (IT) industry. Although this is a high- increasing demand for such skills. Second, when paying sector (ILO 2014), it is characterized training on coding skills is specifically targeted by a large imbalance in gender composition. toward women, it reduces the gender gap in terms According to a study by the International Labour of access to effective training on coding skills. Organization (ILO) that analyzes the IT sector in Motivated by previous literature that finds that a 38 countries, women comprise, on average, just higher presence of women in a male-dominated 20 percent of the total professionals in the sector. sector improves the individual performance of For Argentina and Colombia, this share is 22 and females (Botswick and Weinberg, 2018; Shan, 20 percent, respectively (ILO 2014). Likewise, 2020), a secondary component was embedded in this with respect to the acquisition of the skills needed intervention. It consists of encouragement to form to work in the technology industry, women are a study group with another female classmate. The vastly underrepresented in science, technology, objective of this additional intervention is to shed engineering, and mathematics (STEM) education light on whether exposure to female peers affects (Nimmesgern 2016). program enrollment, education and employment A natural question then arises: Why are women so outcomes by generating a more female-friendly under-represented in a high-paying sector such as environment that encourages women to overcome IT? Moreover, why aren’t more women acquiring social norms when entering STEM fields. the skills needed to land a technology job? In particular, the main questions this pilot aims to Guided by these questions and previous evidence, answer are: (i) Does the acquisition of programming this study pilots a comprehensive female-targeted skills demanded by the IT sector have an impact on computer programming training (bootcamp) in female educational and labor market outcomes? (ii) CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT | 3 Does exposure to another female peer increase the similar across bootcamp locations, the instructional probability of acquiring bootcamp skills, thereby methods varied and were decided by each school improving female labor market outcomes? based on their infrastructure, past practices, and local context. In Bogotá, four workshops were Despite the importance and prevalence of offered throughout the duration of the bootcamp bootcamps as a policy tool — and more generally (including LinkedIn, resume development, job of technical, vocational, and educational training interviews, and teamwork),1 followed by up to two (TVET) programs — empirical evidence about 30-minute personal mentoring sessions. In Buenos their effectiveness is not only scarce, but also Aires, students were given access to a webinar inconclusive (Kluve and others 2019; McKenzie platform with short instructional videos on various 2017; World Bank 2018). This pilot will contribute topics regarding job hunting in the digital industry. to this literature by providing evidence about After completing these videos, students could the causal effects of a female-targeted youth request up to three 45-minute personal sessions employment program on educational and labor with a coach. Students in both countries were market outcomes. also given the opportunity to apply for various jobs connected to the bootcamp provider through Bootcamps Designed to Address events (Bogotá and Buenos Aires) and an online Women’s Constraints platform (Buenos Aires). Table 1 presents more details about the duration and number of sessions The research team supported the design and offered in each country. implementation of two ready-to-work bootcamps in Bogotá and Buenos Aires. These were based on an A third component of the bootcamps was the intensive, part-time rapid-skills training that prepares exposure to peers. When eligible women were women to qualify for employment shortly after invited to enroll in the program, they were provided completion. A group of eligible women received with the contact information of a peer (another training on computational and soft skills, and they woman planning to enroll with a similar educational were encouraged to work with their peer classmates. background). During this communication, peers were encouraged to meet and work together The computational component entailed training in during the training. Later, during the bootcamp, basic programming skills based on current market 2 reminders were sent again encouraging peers to demands, including JavaScript, Hypertext Markup work together. Language (HTML), Cascading Style Sheets (CSS), and other platforms. The theoretical module was The bootcamps were designed to respond to common taught using a combination of online and face-to- constraints that women have encountered in these face classes held by 23 qualified instructors (13 in cities, and the differences observed correspond to Bogotá and 10 in Buenos Aires). Students were also adaptations to their local contexts. Sessions were required to complete various practical projects in a offered at various locations (two locations per city) workshop setting. and shifts (each city had morning, afternoon and evening shifts). Women could choose between only The soft-skills component entailed professional face-to-face or blended modalities (that is, a mix of development workshops focusing on confidence- face-to-face and online classes), and childcare was building, leadership, personal initiative, available in Bogotá. communications and presentation skills, and teamwork, among others. Although the contents of The World Bank team engaged with private this professional development component were very providers in each country and provided scholarships 1 To accommodate participant demands and the agenda, these extracurricular workshops were offered about four times each throughout the duration of the course. 4 | CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT Table 1. Features of Bootcamp Computational and Soft-skills Components Bootcamp Features Bogotá, Colombia Buenos Aires, Argentina Provider Bogotá Institute of Technology (BIT) Digital House Timing • 14 weeks of instructionAugust • 20 weeks of instruction • December 2019 • February - September 2019 Computational module • 170 hours • 210 hours (mandatory) Soft skills module (optional) • 4 types of workshops, with a total • 16 types of online webinars of 15 offered (1.5 hours each) • 45 minutes of on-demand personal • 30 minutes of on-demand personal coaching sessions mentoring sessions • 4 recruiting and networking events • 3 recruiting and networking events • In-house job search tool Source: Authors’ elaboration. to cover 80 percent of tuition for about 150 women application processes, communicating to candidates in Bogotá and 65 percent of tuition for about 150 that scholarships were going to be made available women in Buenos Aires.2 Students were responsible through a lottery. for the remainder of the tuition and could opt for upfront or installment payments. In Buenos Aires, In Bogotá, the advertising campaign was developed a zero interest pre-approved loan was also offered. by a marketing firm working in close collaboration with the bootcamp provider. The first step in this advertising communications campaign consisted of Attracting Sufficient Candidates publicizing the bootcamp through social networks with a Potential to Graduate (Facebook, Instagram, LinkedIn) and news outlets. One of the challenges of conducting a rigorous The campaign included concise banners or images evaluation of the impacts of the bootcamps was to accompanied by a link to a landing page with more ensure that a sufficient number of women enroll in detailed program information. The campaign worked the program. In other words, if only a few women through news outlets and included a brief description enroll, it would not be possible to assess the effects of of the program, eligibility criteria, and a link to the the bootcamps using impact evaluation techniques. landing page. The campaign led to 19,343 unique- For instance, previous pilots implemented by the person visits to the landing page. Of those, 4,465 project only included 13 students in Lebanon and women demonstrated an intention to participate in 16 in Kenya, making it unfeasible to implement the program and left their contact information. any quantitative evaluation (World Bank 2018). In Buenos Aires, the advertising campaign included Another pilot in Colombia included a total number Facebook, Instagram, Google and YouTube Ads. The of 281 enrollees, of which 120 were selected campaign led to 13,125 unique-person visits to the as beneficiaries, and 161 as non-beneficiaries. landing page. Of these, 4,867 showed an intention to The impact evaluation for this pilot did not find participate and submitted their contact information. significant results of the program pertaining to the main outcomes of interest. This is likely due to a A second challenge was to identify women with sample with an insufficient size (World Bank 2018). the potential to succeed, that is, women who could undergo the intense training, graduate, To address this participation issue in the present and who were likely to use those skills for career pilot, considerable effort was put into defining and advancement through jobs or additional education. monitoring strong advertising campaigns and clear The total tuition for the program in Bogotá is about US$1,000 and about US$3,000 in Buenos Aires. 2 CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT | 5 Toward this goal, the application process included computer knowledge required to benefit from a a series of steps that aimed to filter those applicants coding bootcamp. This module was not coding- with greater potential. It consisted of the following: specific since no previous coding experience was required for this program. Participants had about • Comply with basic requirements: Women with 3 weeks to complete this module. high school diplomas. • Pass a final exam: A final exam was conducted • Complete an online questionnaire: This on-site in Colombia (written exam) and Argentina questionnaire was about 50 minutes long (oral exam with interview format). This final and included inquiries about demographic exam had the intention of evaluating the minimum characteristics, education, and the employment knowledge acquired through the pre-work module. situations of the candidates, among other relevant information. It aimed at gathering baseline Figure 1 shows the number of women that passed information for the impact evaluation. each step in each country. At the end of the process, 410 women in Bogotá and 393 in Buenos Aires • Complete a pre-work assignment: This satisfied the eligibility criteria, entitling them to consisted of a self-learning module to ensure participate in the lottery.3 This results in a total that all participants had the minimum level of sample size of 803 women, significantly larger than any previous pilots implemented by the program. Figure 1. Number of Women at Each Step of the Application Process 4,465 786 521 420 410 Interested in Completed Finished Passed Eligible for Bogotá Questionnaire Pre-work Exam Lottery 4,867 848 532 393 393 Interested in Completed Finished Passed Eligible for Buenos Aires Questionnaire Pre-work Exam Lottery Source: Authors’ elaboration using project data. Table 2. Characteristics of Eligible Population (Averages) Buenos Aires, Pooled Sample Bogotá, Colombia Argentina Age (years) 29 29 30 Married 14% 15% 13% Mothers 19% 22% 17% College Degree or More 61% 62% 61% Studied STEM Degree 29% 40% 16% Employed 83% 82% 84% Salaried Worker 79% 78% 80% Experience in Digital Jobs 45% 51% 38% Monthly Labor Income (US Dollars) $465 $535 $392 Source: Bootcamp Baseline Survey, World Bank. 3 In Bogotá, the reduction from 420 to 410 in the last step was due to participants who did not satisfy the additional requirements (either education, work experience, or outstanding score). The outstanding score was to 80/100. 6 | CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT Characteristics of Eligible Women The Lottery: Using Random Baseline data were collected for 803 applicants Assignment to Measure the Impact and are presented in Table 2. The average applicant The main objective of the impact evaluation is to was 29 years old. Most applicants were employed, assess the effects of the bootcamps on participants’ receiving a salary, unmarried, and had no children. labor market and educational outcomes, as well as Whereas about 60 percent of the eligible applicants to shed light on the value added of the exposure have a college degree, less than half of those (29 to the peer. More precisely, the following questions percent) studied for a STEM degree. About half of are evaluated: the applicants were working at a job at which they applied coding skills or which would have benefited Does acquiring the programming skills demanded • from coding skills. by the IT sector have an impact on participants’ educational and labor market outcomes? Table 3 compares educational and labor market indicators between the sample of eligible Does exposure to another female peer increase • participants of the program with the universe the probability of acquiring bootcamp skills, of working-age women in each country (base thereby improving labor market and educational population).4 Base population data was obtained outcomes? from the Organisation for Economic Co-operation The labor market outcomes in the analysis and Development (OECD) (2018) country include having a job after graduation, type of job profiles. Regarding education (employment) (IT sector, freelance, informal), working hours, outcomes, the base population consists of women income, expected salary, and job-seeking behavior. from 25-34 (25-54) years of age. In both Educational outcomes include actual and planned cities, program participants show a considerably enrollment in bootcamps, post-graduate education, higher educational attainment compared to the or formal technical education, and IT skills. base population, as well as a slightly higher employment rate. These differences are mostly due A randomized control trial (RCT) is being used to to the selection criteria imposed at the application assess the impact evaluation questions. Soon after process stage, which aimed to identify those identifying the cohort of 803 eligible women, the applicants with more potential to profit from the research team ran a lottery to randomly assign them bootcamp and later succeed. into beneficiary and control groups. All eligible Table 3. Comparison of Eligible Population to Universe of Working-Age Women in Each Country (Percentage) Bogotá, Colombia Buenos Aires, Argentina Indicator Eligible Candidates Base Population Eligible Candidates Base Population College Degree or More 62 33 61 45 Employed 82 64 84 68 Source: OECD (2018) for population data. World Bank, Bootcamp Baseline Survey for Eligible Participants (Bogotá 2019, Buenos Aires 2018). Individuals in the sample of eligible participants reside in Buenos Aires and Bogotá. Due to data limitations, their educational and 4 labor market indicators are compared relative to national averages for working-age women in each country. In Argentina, 34 percent of all women in the country reside in Buenos Aires (INDEC 2010), and in Colombia, 15 percent of all women in the country reside in Bogotá (DANE 2018). CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT | 7 women had the same chance of being selected female peer increases the probability of acquiring into any of these groups. Women under Group A bootcamp skills — and ultimately impacting the represented about 50 percent of the full sample, and main outcomes of interest — comparisons and they received an offer to enroll in the bootcamp with an analysis were made between Groups A.1 and a scholarship. Within this group, half of the women A.2. A brief theory of change that motivates this (Group A.1) also received encouragement to contact additional comparison is as follows. First, by a peer classmate. The contact information of this receiving the encouragement to contact another peer was provided together with the scholarship female peer, beneficiaries of the lottery may offer, and included examples of how they could perceive that the bootcamps are female friendlier, take advantage of such partnerships. The other half hence having a positive impact on the decision to of the beneficiary group (Group A.2) received a enroll. Second, the possibility of obtaining another scholarship offer as well, but they did not receive female “study mate” during the course may help this peer encouragement. Lastly, women in Group students perform better, and ultimately improve the B formed the control group and did not receive primary outcomes of interest. anything, that is, neither a scholarship nor peer encouragement. Figure 2 illustrates the groups, as The lottery was implemented in two stages to ensure well as the number of women assigned to each. sufficient enrollment. The World Bank committed to support 150 scholarships in each country, and it Comparing Groups A and B allows for the was necessary for the statistical power of the impact identification of causal impacts of the bootcamps, evaluation to achieve this target. At the same time, that is, it helps to answer the first of the two it was anticipated that some women would not questions. To analyze whether exposure to another enroll even after being selected. Figure 2. Eligible Candidates Randomly Assigned to Different Treatment Groups Eligible Women N-803 Group A Group B Scholarship Offer No Scholarship, No Peer N-402 N-401 Group A.1 Group A.2 Scholarship Only Scholarship Offer + Peer N-210 N-192 Source: Authors’ elaboration. Table 4. Enrollment Rates Buenos Aires, Pooled Sample Bogotá, Colombia Argentina Scholarships offered 402 205 197 Scholarships accepted Number 287 139 148 Percentage 71 68 75 Source: World Bank, Bootcamp Monitoring Data, 2019. 8 | CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT The solution was to conduct a lottery in two rounds. closely with the research team to ensure the quality In each country, a first round of the lottery identified of the data. Moreover, qualitative observational 150 participants as beneficiaries of the lottery, and data were collected by field consultants with the the remaining eligible participants were waitlisted. objective of gathering qualitative information The 150 beneficiary women were given two weeks concerning the courses. These data were based on to secure the payment of the remaining tuition costs. classroom visits and collected using reports from Enrollment for this first round totaled 106 women “Teach: Observer Manual” (World Bank, 2018), as in Bogotá (71 percent of the 150 who received the adapted to the local contexts. offer), and 111 in Buenos Aires (74 percent of the 150 who received the offer). A second round of the Participation in Bootcamp Components: lottery was performed with those on the waiting list Computational, Soft-Skills, and Peers to assign the remaining scholarships. At this stage, the providers communicated to those in the control Overall, although most of the enrolled women group that they were not selected in the lottery to complied with the minimum class attendance, receive scholarships. participation in soft-skill training and connecting with peers was lower. The participation figures for After completing the two rounds of the lottery, a each component of the bootcamps are presented in total of 402 women received the scholarship offer. Table 5. Of these, 71 percent (287) enrolled and successfully started their bootcamps. Table 4 presents these Regarding the computational component, 83 numbers for the pooled sample according to city. percent of enrolled women attended over 80 percent of the classes offered. Most absences were recorded in the later weeks of the bootcamp (Box 1). From High Satisfaction to Impact During the bootcamps, a team of three field Box 1. Using Data for Timely Course consultants was charged with monitoring Correction implementation of the program. Monitoring activities consisted of conducting a series of spot In Bogotá, a minimum class attendance of 83 percent checks to ensure that sessions took place according of offered classes was required in order to graduate. In practice, students could have up to eight absences. to design. The research team worked together with Using the monitoring data for timely course correction the consultants and the providers in each city to was key to ensuring that women achieved this gather detailed administrative data regarding class benchmark. If they accumulated seven absences, they were approached by the provider and supported through attendance and completion, student performance tutoring. Of the 29 women with seven absences, 19 results on tests and projects, as well as student were able to comply with the attendance requirement. satisfaction. The team of field consultants worked Table 5. Participation in Bootcamp Components Bogotá, Buenos Aires, Bootcamp Features Pooled Sample Colombia Argentina Computational module: Percentage of women enrolled who 83% 85% 80% attended over 80 percent of classes offered. Soft skills: Percentage of women enrolled who attended at least 58% 80% 37% one workshop or webinar. Soft skills: Percentage of women enrolled who attended at least 45% 76% 15% one mentoring/coaching session. Peers: Percentage of women exposed to peers who reported 48% 47% 50% pursuing a relationship. Source: World Bank, Bootcamp Monitoring Data, 2019. CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT | 9 Although most women did not participate in all Buenos Aires, Argentina soft-skill sessions available to them, 58 percent took • Webinars: Students were given access to a webinar advantage of at least one workshop or webinar, and platform with short instructional videos on various 45 percent participated in a session with a coach topics regarding job hunting in the digital industry. or mentor. Participation in this component varies Topics included: personal branding, curriculum significantly across city. vitae (CV) development, LinkedIn and social Bogotá, Colombia media strategies, and job interview preparation. Thirty-seven percent of the students watched at • Workshops: Four workshops were offered least one of these webinars. throughout the duration of the bootcamp (LinkedIn, resume development, job interviews, • Coaching/Mentorship: Students who requested and teamwork).5 Eighty percent of the students this session needed to indicate their objectives attended at least one of workshop, with the for the meeting in advance (such as preparing for most popular being the LinkedIn workshop. It an interview, getting personalized feedback on introduced students to LinkedIn, instructing them their CV, or other topics). Mentors were analysts on how to use it. It also offered useful insights from the alumni department. These one-on-one for taking full advantage of the platform. The sessions gave personalized support for students workshop helped students create their mission who felt they needed more help in finding a job. statement, showed them how to search for jobs, Only 15 percent of students took advantage of how to add multimedia features to their pages, these sessions.6 and how to write short introductions to potential employers. With respect to peer component, about half of the enrolled women were encouraged to connect with • Coaching/Mentorship: This component was a pre-assigned student with similar characteristics structured into two types of mentoring. The first at the beginning of the bootcamp. They were also type of mentoring included one-on-one meetings reminded about the importance of peers. Overall, led by junior-level mentors. The objective of these 48 percent of women (of the 192 women paired meetings was to help students make progress with peers) reported having had some sort of active with their final practical projects. This support interactions with their assigned peer. When asked was intended to focus on branding the practical about the activities with the peer, the most frequent project to future employers, how to manage answer was teamwork, meaning that they worked teamwork during the project, and how to succeed on the course material together. The second most in meeting the requirements of this project. The frequent activity included sharing and preparing second week of mentoring was led by top-level job search activities, with the third being the mentors, and it was aimed at providing career establishment of friendships. counseling to students. This second type of mentorship was intended to provide successful role models to inspire, empower and connect Performance, Graduation, and students with top leaders in the IT sector, while Satisfaction also sharing relevant information on navigating Field consultants in charge of monitoring the the challenges of a technology career as a woman. program also tracked data about performance in Overall, 76 percent of students enrolled in at least theoretical quizzes and practical projects. Table one coaching session. 6 presents the average performance. In both 5 In Bogotá, the reduction from 420 to 410 in the last step was due to participants who did not satisfy the additional requirements (either education, work experience, or outstanding score). The outstanding score was to 80/100. 6 According to information collected by field coordinators, this 15 percent attendance rate among program beneficiaries is even higher than the attendance rate observed for this component among regular students (that is, those who pay full tuition). 10 | CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT 3 A Lasso regression is a machine learning technique used, among other things, to prediction model only those that have high explanatory power. Bogotá and Buenos Aires, average grades were • Inclusive and positive space for women. Several above 70 percent. factors contributed to making the bootcamps a more inclusive and positive space for women, Interestingly, the implementing partner in Buenos including having female teaching assistants in Aires was able to provide anonymous performance many of the commissions as an example and role data for 128 female students who were not part model; having a higher number of women in the of the program and who had enrolled paying full classroom than usual as a result of the outreach of tuition. The average of theory tests for this cohort the program; and having “women in tech” events was 73.255 (standard deviation 12.525). Therefore, along with the course. the beneficiaries seem to have outperformed the regular students by scoring about 6 percentage • Avoidance of gender stereotypes. Providers points higher. This difference is statistically did not promote or develop gender stereotypes significant at a 5 percent confidence level. in the classroom. Materials and examples did not promote stereotypes. In fact, students were Overall, 84 percent of those enrolled graduated very aware of gender inequalities. In this regard, from the bootcamps (specifically 128 of 139 women the projects often geared their applications in Bogota, and 113 of 148 women in Buenos specifically to women. Aires). Most dropouts were due to personal reasons or incompatibilities with work schedules. These • Relationships between students, instructors occurred during the first month of the bootcamp. and bootcamp providers. Students felt comfortable raising concerns and asking An anonymous, short survey was implemented questions. Students also felt free to provide just before graduation. This survey inquired about feedback to providers about things that they did the overall level of satisfaction with the course, not like. Both the providers in Bogotá and Buenos as well as information about future professional Aires showed an interest in making adjustments to and educational plans. On average, 84 percent of better meet students demands. respondents were satisfied or very satisfied with the bootcamp (specifically, 76 percent in Bogotá and Special follow-up actions for particular • 94 percent in Buenos Aires). students. Students who requested extra office hours or extra support were given the requisite Lessons assistance. Based on detailed monitoring reports as well as • Calendar adjustments. Providers were flexible the graduation survey, it is important to highlight regarding student needs and adapted the timeline the following key strengths and elements of good accordingly if there was a valid reason for practice of the bootcamps: adjustment. They also extended project deadlines Table 6. Average Grades for Tests and Projects Evaluated on a Scale of 0-100 (Standard Deviation) Bogotá, Colombia Buenos Aires, Argentina Theory Tests 71.239 77.501 (10.399) (13.359) Projects Score 63.203 N/A (21.891) Number 128 113 Source: World Bank, Bootcamp Monitoring Data, 2019. CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT | 11 select from a long list of covariates used to fit a when students felt unprepared. The priority was Next Steps in Assessing the Impact of the to create an environment where students would Bootcamps be successful. Additional data collection and analysis are expected The following areas for improvement were also in order to assess the impact of the bootcamp on identified: labor market and educational outcomes, as well as the value added of the peer component. Emphasis on soft skills. Bootcamps did not • emphasize the development of soft skills or A brief online survey was successfully implemented goal setting into the curriculum very thoroughly. during May and June 2020, achieving a response Qualitative observations by field coordinators rate of 90 percent of the original sample.7 The highlighted a disconnect between the technical objective of this online survey was threefold. First, content and applying it in a meaningful way it was important to gather information about labor outside of the classroom. market outcomes of program participants right before the COVID-19 global pandemic affected the Logistics of soft-skills activities. There is room • labor markets in both cities. As such, information for improvement regarding the logistics of the was collected dating back to February 2020. soft-skills activities. The main issues are to Second, information about educational outcomes choose more optimal times to encourage larger and skills (both computational and soft skills) attendance, and to better communicate about the was collected in order to measure the short-term objectives and benefits of each workshop. Early impacts of the program on these indicators. Third, and clear communications about the expectations a module regarding the resilience of women to and benefits of the program may promote higher the COVID-19 pandemic was collected to assess participation among students. whether bootcamp participants are better positioned Job search events. These were not organized • than non-participants to tackle the crisis. The data with sufficient preparation. There were also some from the online survey are being analyzed and last-minute date changes, which were only shared preliminary results will be available in 2021. A more retrospectively or as they happened. This meant comprehensive follow-up, face-to-face survey was that on occasion some events were missed due to postponed due to the COVID-19 pandemic. lack of communication or timing conflicts. 7 A response rate of 90 percent compares very favorably to the response rates found in other vocational training studies (McKenzie 2017). 12 | CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT References Beede, David N, Tiffany A Julian, David Langdon, George McKittrick, Beethika Khan, and Mark E Doms. 2011. “Women in STEM: A Gender Gap to Innovation.” Economics and Statistics Administration Issue Brief. Blau, Francine D. and Lawrence M. Kahn. 2017. “The Gender Wage Gap: Extent, Trends, and Explanations.” Journal of Economic Literature 55 (3):789–865. Bostwick, Valerie K. and Bruce A. Weinberg. 2018. “Nevertheless She Persisted? Gender Peer Effects in Doctoral STEM Programs.” Technical report. National Bureau of Economic Research. DANE (Departamento Administrativo Nacional de Estadística). 2018. Censo Nacional de Población y Vivienda de Colombia. ILO (International Labour Organization). 2014. “Global Assessment of Sex Disaggregated ICT Employment Statistics.” Technical report. INDEC (Instituto Nacional de Estadística y Censo). 2010. Censo Nacional de la Población Argentina. Kluve, Jochen, S. Puerto, D. Robalino, J. Romero, F. Rother, J. Stöterau, F. Weidenkaff, and M. Witte. 2019. “Do Youth Employment Programs Improve Labor Market Outcomes? A Quantitative Review.” World Development 114: 237–253. McKenzie, David. 2017. “How Effective are Active Labor Market Policies in Developing Countries? A Critical Review of Recent Evidence.” The World Bank Research Observer 32(2): 127–154. Nimmesgern, H. 2016. “Why are Women Underrepresented in STEM Fields? Chemistry – A European Journal 22(11): 3529–3530. OECD (Organisation for Economic Co-operation and Development). 2018. Education at a Glance 2018: OECD Indicators. Paris: OECD Publishing. Shan, Xiaoyue. 2020. “Does Minority Status Drive Women Out of Male-Dominate Fields?” Unpublished Working Paper. World Bank. 2018. “Coding Bootcamps for Youth Employment: Evidence from Colombia, Lebanon, and Kenya.” Technical Report 1. World Economic Forum (WEF). 2018. “The Global Gender Gap Report.” Technical report. CODING BOOTCAMPS FOR FEMALE DIGITAL EMPLOYMENT | 13