CODING BOOTCAMPS FOR YOUTH EMPLOYMENT EVIDENCE FROM COLOMBIA, LEBANON, AND KENYA B CONTENTS ACKNOWLEDGEMENTS iii EXECUTIVE SUMMARY 1 ABBREVIATIONS 3 INTRODUCTION 5 IMPACT EVALUATION IN MEDELLÍN, COLOMBIA 9 Context 9 Program Background 11 Evaluation Design 15 Sample 19 Results 23 QUALITATIVE STUDY IN BEIRUT, LEBANON 35 Context 35 Program Background 36 Research Design 36 Sample 38 Results 39 QUALITATIVE STUDY IN NAIROBI, KENYA 43 Context 43 Program Background 44 Research Design 44 Sample 46 Results 48 MAIN FINDINGS 53 A. Employment Impact 53 B. Educational Impact 54 C. Bootcamp Programs 54 LESSONS FOR FUTURE IMPACT EVALUATIONS 55 BIBLIOGRAPHY 58 APPENDIX A: Research Design: Randomized Controlled Trial in Colombia 59 APPENDIX B: Research Design: Qualitative Study 62 APPENDIX C: Instrumental Variable Regression Tables 66 APPENDIX D: Surveys and Questionnaires 68 i ii ACKNOWLEDGEMENTS This impact evaluation report has been developed by the Finance, Competitiveness and Innovation Global Practice with the Education Global Practice of the World Bank Group. The report is part of the Decoding Bootcamp initiative funded by the World Bank’s Jobs Umbrella Trust Fund, which is supported by the United Kingdom’s Department for International Development, and the governments of Norway, Germany, Austria, the Austrian Development Agency (ADA), and the Swedish Development Agency (SIDA). Victor Mulas and Cecilia Paradi-Guilford lead the Decoding Bootcamp initiative. The Colombia impact evaluation analysis and its section was led by Pedro Cerdan-Infantes. The above mentioned are the main authors of this report, together with Elene Allende Letona, Domoina Rambeloarison, Erick Ramos Murillo, Luis Felipe Martínez Gómez, Francisco Zavala and Kathy Qian. The following people provided substantive inputs through comments to early drafts, as well as participation in several aspects of the activity: Caio Piza, Paola Vargas González, Yegana Baghirova, Zhenia Viatchaninova Dalphond, Farah Manji, Viviana Mora, Samhir Vasdev, Hallie Applebaum and Marta Khomyn.Yael Hochberg and Eric Floyd designed the experiment and provided technical advice to the team for the RCT in Colombia and qualitative impact evaluations in Kenya and Beirut. The team in Ruta N Corporation, the World Bank’s main partner, was led by Ruben Villegas and Ana Maria Ospina. The World Bank also acknowledges the institutions and individuals that participated in the surveys, Focus Group Discussions and interviews, as well as students who participated in the bootcamp cohorts pertaining to this study. Peer reviewers of the final document were Leonardo Iacovone, Marcio Cruz and Siddhartha Raja. iii iv EXECUTIVE SUMMARY Coding bootcamps are intensive short-term programs methodologies, namely qualitative focus groups in Beirut designed to train participants in programming skills to and Nairobi, and a randomized controlled trial (RCT) in make them immediately employable (Meng 2013). They Medellín. The selection of methodology took into account combine characteristics of traditional vocational training the different market conditions (for example, availability of programs with the intensity of military bootcamps for new sample size for the experiment timeline and/or immaturity recruits, intermingling socioemotional and tech skills learning of the bootcamp providers’ market) in each location. All in an intense and experiential manner, in what could be referred locations used the same baseline and final surveys, making to as “skills accelerators.” We refer to coding bootcamps in findings complementary. Together, these three studies this report as the Ready-to-Work model. This model follows a provide a good understanding of the impact of coding structured process with three main characteristic features: 1) bootcamp programs in developing countries, beyond self- intense rapid-skills training, 2) experiential learning approach, reporting figures. and 3) curricula based on, and continuously adapting to, industry’s demand. Depending on the organization, the The findings from the three impact evaluations suggest model has yielded job placement rates ranging from 60 to that coding bootcamps may have three specific effects: 100 per cent (ITU 2016). Thus, coding bootcamps provide a 1. Employment: Coding bootcamps do not have a mix of technical and practical skills directly connected to particular impact on providing access to employment industry demand, making them a potentially effective tool for generically, but they may have an impact on providing the requirements of the new tech-led economy. This makes access to high-quality jobs (particularly high-quality coding bootcamps a potential tool for developing economies tech jobs). in building a talent pipeline ready to face disruptions in the employment and skills landscape arising from the so- 2. Business creation (self-employment): Coding called Fourth Industrial Revolution (The World Economic bootcamps may have a positive effect on business Forum 2016). (that is, startup) creation for those with low incomes, suggesting that bootcamps could also be leveraged To understand the potential of coding bootcamps for to provide tech-related self-employment for those developing countries, the World Bank launched the segments of the population that may face job access Decoding Bootcamps initiative, the objective of which structural barriers in developing countries. is four-fold: (i) to assess the impact of coding bootcamps on local, young jobseekers to secure quick employment 3. Education: Coding bootcamps seem to support and income generation opportunities; (ii) to compare the completion of tertiary educational programs, employment patterns, and employability in new-economy suggesting that they could potentially play a jobs, between bootcamp participants and those who have complementary role and that there may be a need not received the training; (iii) to identify key success factors to incorporate some of its methodologies in existing of coding bootcamps for emerging economies and devise a tertiary educational programs. toolkit for designing a coding bootcamp from scratch based on an overview of existing tools and best-practice methods; Coding bootcamps are not easy to implement in the and (iv) to inform policy makers in emerging markets on context of developing countries, but they may be how to support the establishment, implementation, and catalyzed with policy interventions. In particular, it growth of demand-driven rapid tech skills training to combat was found that: (i) bootcamp programs are difficult to youth unemployment. implement and require links with potential employers, (ii) not all bootcamps are the same and quality has a The report addresses these objectives by presenting significant impact on results, and, (iii) bootcamp programs the findings from an impact evaluation of coding can be catalyzed through policy intervention. Appendix D bootcamps in Medellín complemented with qualitative provides a guide for implementing bootcamp programs studies in Beirut (Lebanon) and Nairobi (Kenya) -three in developing countries, aiming to inform public policy developing country cities-. The studies followed different interventions and bootcamp providers. 1 2 ABBREVIATIONS API Application Programming Interface BPO Business Process Outsourcing CAQDAS Computer Aided Qualitative Data Analysis Software CSS Cascading Style Sheets DANE National Administrative Department of Statistics (Colombia) EPM Empresas Públicas de Medellín (Medellín Public Enterprises) HTML HyperText Markup Language ICT Information and CommunicationTechnologies iOS iPhone Operating System IT Information Technology ITT Intent to Treat ITU International Telecommunication Union M&E Monitoring & Evaluation MNC Multinational Corporations MOOC Massive Open Online Course MVC Model-View-Controller OLS Ordinary Least Squares ORM Object-relational Mapping PHP Hypertext Preprocessor (originally Personal Home Page) RCT Randomized Controlled Trial SME Small and Medium Enterprises SSOM Standard Student Outcomes Methodology UVA Unidad de Vida Articulada (Articulated Life Vehicle) WBG World Bank Group WTM World Tech Makers All dollar amounts are U.S. dollars unless otherwise indicated. 3 4 INTRODUCTION The recent rise of coding bootcamps across Africa, Asia, intense rapid-skills training, 2) experiential learning approach, and Latin America can be explained in part by the global and 3) curricula based on, and continuously adapting to, shortage in technology skills (ITU 2016). Their founders industry’s demand. Depending on the organization, the seek to leverage local talent for work at international model has yielded job placement rates ranging from 60 to 100 companies with coding-related outsourcing needs. They are percent (ITU 2016). Thus, coding bootcamps provide a mix of typically technology entrepreneurs who are embedded in the technical and practical skills directly connected to industry local technology industry. In turn, this allows them to better demand, making them a potentially effective tool for the assess industry demand, optimize curriculum development, requirements of the new tech-led economy. For developing and develop a strong network of potential employers for economies, they are particularly useful in building a talent bootcamp participants. pipeline ready to face disruptions in the employment and skills landscape arising from the Fourth Industrial Revolution Coding bootcamps were designed to address the gaps in (World Economic Forum 2016). For more information on formal education in tech skills by providing young people coding bootcamps, see the first report of the Decoding an accelerated path to developing coding skills that are Bootcamp initiative (Mulas and others 2017). increasingly important globally (ITU 2016). Bootcamp graduates appear to have a stronger path to employment in The objective of the Decoding Bootcamps initiative is tech-related jobs than those of alternative training options four-fold: (i) to assess the impact of coding bootcamps in this field, such as online tutorials or massive open online on local, young jobseekers to secure quick employment courses (MOOCs). Coding bootcamps focus on developing and income generation opportunities; (ii) to compare software development skills, which are predicted to be in employment patterns, and employability in new-economy high demand. jobs, between bootcamp participants and those who have not received the training; (iii) to identify key success factors This report is part of the Decoding Bootcamps initiative of coding bootcamps for emerging economies and devise a funded through the Jobs Umbrella Multi-Donor Trust toolkit for designing a coding bootcamp from scratch based on an overview of existing tools and best-practice methods; Fund. The initiative aims to collect and share examples and and (iv) to inform policy makers in emerging markets on how lessons of bootcamps in emerging markets, and measure to support the establishment, implementation, and growth the impact of bootcamp training on youth employment of demand-driven rapid tech skills training to combat youth in selected countries. The program seeks to establish a unemployment. framework of best practice for future projects in technology upskilling in the developing world. The World Bank piloted The report highlights the results of a randomized this initiative between March 2016 and July 2017 in three controlled trial (RCT) carried out in Medellín (Colombia), cities: Beirut, Lebanon; Medellín, Colombia; and Nairobi, Kenya. complemented with qualitative studies in Beirut These cities were selected because of their vibrant local tech (Lebanon) and Nairobi (Kenya). In Medellín, government innovation ecosystems, the relevant size of the low-income support, local industry demand for technology talent, and the youth population, and high youth unemployment. ability of the bootcamp provider to scale, enabled the team to secure a higher sample size sufficient for the RCT. (For more Coding bootcamps are intensive short-term programs information on the RCT Design, see Appendix A.) In Beirut designed to train participants in programming skills to make and Nairobi, the market conditions (that is, lack of availability them immediately employable (Meng 2013). They combine of sample size necessary for the experiment timeline and/ characteristics of traditional vocational training programs or immaturity of the bootcamp providers’ market) were not with the intensity of military bootcamps for new recruits, conducive to an RCT experimental design because of the intermingling socioemotional and tech skills learning in an infeasibility of randomization. Thus, the World Bank carried intense and experiential manner, in what could be referred out qualitative studies in these two additional locations to as “skills accelerators.” We refer to coding bootcamps in to gain a deeper understanding of the effects of coding this report as the Ready-to-Work model. This model follows a bootcamps on participants and provide additional insights structured process with three main characteristic features: 1) from two additional developing country locations. This report 5 complements the initial findings of the Decoding Bootcamps the activity targeted existing bootcamps providers, although initiative’s first publication (Mulas and others 2017), which they were still maturing their business models and adapting presented preliminary evidence of the impact of coding their methodologies to their local markets. The activity also bootcamps in the context of emerging economies. targeted Medellín, a city where coding bootcamps did not exist, despite market demand. The local government The emphasis of the two qualitative studies is on the in Medellín demonstrated a clear commitment to attract experiences (emotional, behavioral, and educational such training providers to the city, after having identified an adjustments) of bootcamp participants as a result of important skills gap in entry-level tech jobs (Ruta N Medellín their exposure to training and post-training employment and others 2015). This made Medellín an ideal location to test patterns. They explain why in some cases participating in the implementation of coding bootcamps in a developing bootcamps is a career promoter and a life-changing experience, country context where demand existed but providers were while in others it is not. The current research develops theories not yet present. The activity’s experimental bootcamp built of the determinants of positive outcomes of bootcamp on a market-fit pilot implemented by Ruta N Corporation, the training. An observation-focused qualitative case study local innovation agency, with 25 students in Medellín. This design was implemented to explain the exact mechanisms allowed to test the potential for implementation of the coding that improve post-training quality of life outcomes for some bootcamp methodology in an emerging economy, while students over others. (For more information on the Qualitative identifying challenges and requirements. Building on the Research Design, see Appendix B.) lessons of this additional applied research, this report presents policy recommendations to create an enabling environment Students filled out the same baseline and final surveys for bootcamp attraction and growth, and a methodological in the three countries. The coding bootcamps were similar toolkit for practitioners, based on best-practice cases (see in length, using a learning-by-doing approach. To address Appendix D). the issue of skills mismatch, they leveraged their connection to the local IT industry, ensuring that coding languages For this initiative, the World Bank worked closely with taught in each location reflected specific industry needs. local research partners and government agencies to Bootcamp participants (and the control group in Medellín) execute the training and conduct data collection. Research were monitored for six months after its end to understand partners advised on program design, collected relevant data, the impact of the training on their employability (defined and launched calls for applications for bootcamp providers through access to new job opportunities, better quality of based on their knowledge of local tech ecosystems and youth- jobs, entrepreneurship opportunities, and so on), employment related issues. In Beirut and Nairobi, these were key nodes in (following employment history), and salary. the innovation and technology entrepreneurship ecosystem with demonstrated research capability, and they secured Coding bootcamps are a recent phenomenon, with many partnerships with local bootcamp providers. In Medellín, providers being startups with little or no implementation the World Bank relied primarily on government agencies, experience. At the time of implementation of this initiative, which provided space, equipment, as well as critical outreach coding bootcamps were only just starting to appear in support for data collection. developing countries. Providers were still experimenting with curricula and methodologies, and it was not clear how the The experiment aimed to be inclusive, primarily targeting bootcamp methodology could be best implemented in the the low-income population in Medellin. This posed a context of an emerging economy. Thus, this initiative focused challenge in terms of recruitment, student engagement on testing this rapid skills training methodology in locations through the end of the training, and outreach to participants where either local coding bootcamps existed or local (treatment and control in the case of Medellín) for the final providers could potentially implement the training, having an survey. As mitigating measures, the World Bank and the understanding of local conditions and the capacity to adapt providers identified instructors who could help students who the methodology accordingly. were lagging behind in class, and hired a local survey firm that followed up on the final survey with participants. The Decoding Bootcamp initiative was designed to also test whether these providers could implement Table 1.1 summarizes the intervention in each of the three bootcamps in emerging economies. In Beirut and Nairobi, cities selected for this program. 6 Table 1.1: Summary of interventions Medellín Beirut Nairobi Impact Evaluation Randomized Qualitative study Qualitative study (surveys, Methodology Controlled Trial (surveys, interviews, interviews, focus group (baseline and final focus group discussions) discussions) surveys) Year of first bootcamp May 2014 (Bogotá) March 2016 (Beirut) January 2015 (Nairobi) in the city May 2016 (Medellín)a Bootcamp provider World Tech Makers SE Factory Moringa School Rationale for Competitive selection Comparative review of Comparative review of choosing provider process (there were bootcamps in Lebanon bootcamps in Nairobi no bootcamps in (selected bootcamp (selected bootcamp used Medellín at the time used the model in line the model in line with this project was with the goals of the the goals of the research defined) research experiment) experiment) Other providers of Cymetria, Make It Real LeWagon [now closed] Nairobi Developer School model in the city at (both Bogotá)b (DevSchool) [now closed] time of experiment Bootcamp program May–August 2016 (12 July–October 2016 (12 April–August 2016 (16 implementation dates weeks) weeks) weeks) Coding language Ruby, Rails, HTML, Full-stack web Android, Python, UI and JavaScript, etc. development (Apache, UX, HTML and CSS, and SQL, PHP, HTML / CSS, JavaScript JavaScript, etc.) Audience Youth between 18 Computer science People with some and 28 years of age. students or graduates programming basics and No previous studies from less privileged that passed the intensive or coding knowledge backgrounds selection process (included required coding challenges, motivational questions…) Number of students 120 students (and 161 15 students (13 18 students (16 participated participants in the participated in the in the study); no control control group) study); no control group group Price per student $750-1,000, $100d $2,500e (subsidy) depending on the socioeconomic stratac Research partners Ruta N Corporation, Berytech iHub Secretariat of Youth (Municipality of Medellin) Note: a. Bootcamp in Medellín was catalyzed by the activity; b. Coding bootcamps active by mid-2017; c. Tuition was subsidized by this activity for the bootcamp in Medellin. Regular tuition fee in Colombia is $2,000; d. SE Factory bootcamp follows a non-for-profit model, where tuition fees are subsidized; e. After this cohort, Moringa School in Kenya changed their bootcamp structure, and reduced the pricing to $1,200. 7 The next chapter describes the intervention in Medellín, including the experimental allocation of training slots to the bootcamp. Chapters 3 and 4 present the qualitative studies in Beirut and Nairobi. The main findings from the three interventions are presented in Chapter 5, and lessons for future impact evaluations are described in Chapter 6. Notes: 1. For more detail on this rapid skills training program, see http://www.decodingbootcamps.org. 2. http://www.worldbank.org/en/topic/jobsanddevelopment. 3. The model is the traditional approach to coding bootcamps (ITU, 2016). It typically refers to an intensive 12 to 24 weeks full- or part-time rapid skills training programs that prepare people to qualify for employment as junior developer, either working for a company or as freelancers, shortly after the training ends. 4. The World Economic Forum (2016) forecasts strong employment growth in the Architecture and Engineering and Computer and Mathematical job families by 2020. The Future of Jobs Survey has identified big data analytics, the Internet of things, and mobile internet and cloud technology as important drivers of change of this growth. 5. A Randomized Controlled Trial (RCT) is an experimental form of impact evaluation in which the population receiving the program or policy intervention is chosen at random from the eligible population, and a control group is also chosen at random from the same eligible population. It tests the extent to which specific, planned impacts are being achieved. The distinguishing feature of an RCT is the random assignment of units (e.g. people, schools, villages, etc.) to the intervention or control groups (UNICEF, https://www.unicef-irc.org/KM/IE/ impact_7.php). 6. For more information on the structure of coding bootcamps, see Mulas and others 2017. 7. An entry-level technology job is a job that is normally designed or designated for recent graduates of a technological discipline and typically does not require prior experience in the field or profession. 8. A startup company is an entrepreneurial venture which is typically a newly emerged, fast-growing business that aims to meet a marketplace need by developing a viable business model around an innovative product, service, process, or a platform. 8 IMPACT EVALUATION IN MEDELLÍN, COLOMBIA This chapter describes the randomized controlled trial (RCT) Figure 2.1: Youth unemployment rate in evaluation of the coding bootcamp program in Medellín. The RCT used baseline data as well as survey information collected Colombia (2006-2017) six to seven months following the end of the bootcamp. The most notable impacts of the intervention are on participants’ Youth Unemployment rate educational outcomes, as well as post training type of (age 14 to 28) employment. 25 Unemployment Rate % Context 20 15 Why Colombia? 10 Recent trends suggest that, to continue growing, 5 Colombia needs to shift from high dependency on natural resources to a more knowledge-based economy. This 0 hinges on strengthening human capital, which has become Dec-06 Jun-07 Dec-07 Jun-08 Dec-08 Jun-09 Dec-09 Jun-10 Dec-10 Jun-11 Dec-11 Jun-12 Dec-12 Jun-13 Dec-13 Jun-14 Dec-14 Jun-15 Dec-15 Jun-16 Dec-16 Jun-17 a centerpiece of the country’s development strategy and educational policy efforts. The most recent results of the Programme for International Student Assessment (PISA) show that, in 2015, Colombian students were developing better skills than their peers in 2012 and 2009. Dec-06 Dec-07 Dec-08 Dec-09 Dec-10 Dec-11 Dec-12 Dec-13 Dec-14 Dec-15 Dec-16 Jun-17 While unemployment, particularly among young people, has declined during a large expansionary period, there are signs of a reverse trend. Though Colombia’s unemployment Colombia 19.4 16.5 18 19 18.4 16.2 16.1 13.9 14.3 13.6 14.4 15.6 rate stood at 9.1 percent in 2014, the country has been unable Medellín 19.8 17.6 17 18.2 17.4 to generate employment in line with the increase in the labor force. The informal sector is also growing, representing about Source: DANE, and Medellín Cómo Vamos 2016. 70 percent of those who are employed. A constant growth rate has allowed Colombia to make significant headway in its socioeconomic indicators. However, the fall in oil prices in recent years has reversed this trend. Figure 2.1 shows that support for early-stage startups. In addition, ProColombia, youth unemployment had steadily declined between 2006 the government body responsible for the promotion of trade, and 2013. In the past three years, however, it has been on the foreign investment, and tourism, now aims to build upon these rise, indicating that young Colombians are having a relatively foundations to transform the country into a business process hard time finding a job. Medellín’s youth face a similar outsourcing (BPO) powerhouse not just in Latin America, but challenge, with a slightly higher unemployment rate than the on a global scale. rest of the country. As a result, Colombia has seen a fivefold increase in Over the past few years, public and private efforts to its software and information technology (IT) market support tech entrepreneurship have grown in number and between 2003 and 2015. The country is now among scale. That success is also being nurtured by a constellation the top IT services providers in Latin America, and has of support organizations, incubators, and accelerators, many been lauded as one of the region’s most promising tech of which receive direct government support. Government- hubs. Colombia is turning itself into an attractive market by sponsored programs like INNpulsa and Apps.co have been incentivizing hard science and IT-related education, and this touted as having played an important role in providing is achieving results. According to International Data Collection 9 (IDC), the Colombian IT sector grew 14.1 percent (CAGR the market. In addition, more investor confidence is needed to 2003-2016) and is the fourth largest IT market in the region attract private capital in Medellín and the supply of adequately . Technology companies are concentrated in Colombia’s trained professionals to meet the demand from IT and BPO largest metropolitan areas, most notably Bogotá and companies is not sufficient. Medellín (World Bank and Endeavour Insight 2015). The digital industry in Medellín, a city with a population of In this context, Medellín has made a strong commitment 2.5 million, is increasingly growing because of the recent to diversifying its economic structure through its Medellín use of digital tools in various individual, entrepreneurial, Ciudad Innovadora (Medellín Innovative City) program, and industrial processes. Established in 2010, the with policies to attract IT companies to the city. The Mayor’s center of Medellín’s innovation scene is Ruta N, which Office has instigated several programs for this purpose. Ruta provides incubation, landing services, and office space N Medellín, the local innovation agency, conducted a study for innovative startups and service providers. It has concluding that there is a high need for IT skills in the city, generated some 2,900 jobs by attracting 73 companies especially for those skills required for entry-level positions. The . Another important ecosystem player with a presence in implementation of coding bootcamps is a relevant strategy both Bogotá and Medellín is coworking space Atom House. to tackle this skills gap. According to employment agency El Empleo, 6.05 percent of the total of new offers in its web portal are in the systems and technology field. Colombia has a Medellín’s Startup Ecosystem relatively high demand for skilled workers in these fields. The general areas with higher demand are: commercial and sales As of 2015, the number of tech firms in Colombia (24.6 percent), administrative and financial (13.6 percent), and numbered at least 678, and these tech companies customer service (8.9 percent). There is still a gap between currently employ an estimated 20,000 people (World applicants and job offers: only 4.2 percent of the applicants Bank and Endeavour Insight 2015). This talent is largely on the portal had an adequate profile for the systems and concentrated in a few large cities, and the vast majority of technology field. connections between entrepreneurs involve either Bogotá or Medellín. The recent uptick in entrepreneurial activity has Moreover, Medellín still faces the same challenges as produced an ecosystem that is now growing at a rate of 15 the rest of the country, concerning poverty, inequality, percent annually. If the sector as a whole continues to grow and crime. In the city, the divide between rich and poor at this rate, it will, by 2020, the number of people it employs is wide, as is the case in many parts of the country. Though will double (World Bank and Endeavour Insight 2015). poverty in Colombia has declined markedly since the late 1990s (from 50 percent in 2002 to 28.5 percent in 2014), the Named Innovative City of the Year by the Wall Street benefits of stronger growth have not resulted in equally Journal in 2013, Medellín has therefore developed a strong reductions in income inequality. The Gini coefficient vibrant tech industry, and 18 percent of the IT companies declined only from 57.2 percent in 2002 to 53.8 percent in from Medellín generate 80 percent of the employment in 2014, and inequality in Colombia remains among the highest the Antioquia region (World Bank and Endeavour Insight in the world (International Monetary Fund 2015). With over 2015). However, Medellín only has a few startup success six million Colombians still living in poverty, strategies that stories. It still lags behind Bogotá, and has some way to go help reduce this number are very relevant for the Colombian before it can be compared to other global tech hubs. One context. A key strategy to reduce poverty and consolidate explanation for this is the mismatch between the demand the middle class is education. Education will allow people to and supply of labor with technical skills. As captured in a increase their long-term income, and improve their quality 2015 study by Ruta N Medellín, software developers are in of life. high demand in Medellín, with junior developers being the most sought after by companies (47 percent of the total Given the potential of the tech industry to create jobs companies surveyed and 73 percent of large companies and tackle youth unemployment, the World Bank has surveyed reported requiring junior software developers). been actively supporting the tech startup ecosystem in However, there appear to be technical skills gaps: Medellín, most recently through its Decoding Bootcamps companies surveyed in the study reported that sourcing initiative in partnership with the Municipality of Medellín. web developers was difficult, while SMEs also additionally In conjunction with Ruta N Corporation and the Secretariat reported struggling to hire senior mobile app developers in of Youth, the World Bank expanded the scope of bootcamps 10 in Medellín to train low-income youth, offering subsidies Box 2.1: Local partnerships according to participants’ income levels. The World Bank Group supported Ruta N on the design of the program to teach, in the short run, entry-level programming skills with The World Bank partnered with Ruta N Corporation the goal of increasing employability and job satisfaction of and the Secretariat of Youth for this activity. EPM young people in Medellín. The World Bank also designed and and Microempresas de Colombia provided in-kind developed the impact evaluation to generate high-quality contributions. evidence of the effectiveness of coding bootcamps in large Ruta N Corporation is a public joint venture between cities in developing countries. the Mayor’s Office of Medellin, UNE Telco (UNE), and the public utilities company, EPM. To develop Medellín’s Program Background innovation ecosystem, Ruta N focuses on boosting talent, access to capital, infrastructure, and innovative business Medellín did not have any bootcamp providers serving development. In addition, its Landing Program facilitates the market. The first step for implementing the activity was access to a working space for local and international to test the feasibility of a bootcamp program in Medellín. The companies. Ruta N first developed the coding bootcamp local government actively supported this effort, particularly concept in Medellín. As the World Bank’s main partner, it through Ruta N and the Secretariat of Youth of the Municipality provided monetary and in-kind contributions. of Medellín (Box 2.1). The Secretariat of Youth of the Municipality of Medellín Ruta N catalyzed the establishment of these programs is the agency responsible for equipping young people in the city. Ruta N conducted a competitive selection with knowledge, training, and citizen participation process to establish and develop a coding bootcamp in the opportunities to transform them into agents of change. city. The selection criteria included: (i) proven experience The Secretariat of Youth provided advice and coordinated providing coding bootcamps, including a complementary the activity’s communications campaign. online platform and a component on socioemotional skills; (ii) capacity to work with a minimum of 100 students; and, Empresas Públicas de Medellín (EPM, Medellín Public (iii) a contextualized curriculum that ensured the relevance Enterprises) was established as a residential public utilities of acquired skills to the local market in Medellín. World Tech company, which initially served the residents of Medellín, Makers (WTM) was awarded the contract out of four proposals, and has now expanded to 11 Colombian regions and establishing and conducting an initial coding bootcamp Panama. EPM provides electricity, gas, water, sanitation, for 25 participants. This pilot tested the feasibility of the and telecommunications. EPM provided the bootcamp coding bootcamps model in the city. The activity launched a locations (UVAs) in the city of Medellín. competitive process to conduct a larger scale bootcamp to conduct the randomized controlled trial (RCT). Microempresas de Colombia is a savings and credits association, which aims to stimulate national savings as In April 2016, WTM launched the call for applications real insurance for the future. The association also provided for the RCT bootcamp program, mainly through digital space for one of the bootcamps. and mass media publications. Over a 20-day period, the bootcamp was advertised as an incentive to find a job in the The Secretariat of Youth, Ruta N, and WTM notified those participants ICT sector. News stories on local television channels and other that had been selected and were invited to the Launch Event in Ruta outreach activities, ranging from posts of the partners’ (see Box N’s building on May 24, 2016. 2.1) social media accounts to WTM’s visits to local universities, attracted 903 applicants to the program. As part of the process, Students’ participation fees were determined according students were asked to explain their motivation for applying. to their socioeconomic strata (see Box 2.2). The World Bank, Eligibility requirements included: (i) being Medellín residents Ruta N Corporation, and the Secretariat of Youth agreed on a between 18 and 28 years of age, (ii) showing an intent to subsidy scheme. The program was free for students in strata find a job after the bootcamp, (iii) having basic computer 1 and 2. Those in strata 3 paid Col$300,000 (about $100), and and internet skills, and (iv) attending training in the assigned those in strata 4 through 6 paid a total of about Col$750,000 facilities during the entire duration of the program. (about $250). 11 Box 2.2: Socioeconomic strata in Colombia Figure 2.2: Selection process and program uptake The Colombian government has implemented a socioeconomic stratification system to 903 PEOPLE APPLIED classify urban populations into different strata TO PARTICIPATE IN THE BOOTCAMP with similar economic characteristics. It is in accordance with DANE’s real estate property classification, which evaluates real estate 417 FULFILLED THE MINIMUM PARTICIPATION units based on poverty levels, public services, CRITERIA location, and indigenous population. This system determines tax levels, public services 120 SELECTED RANDOMLY (water, energy, phone and gas) fees, access to TO PARTICIPATE IN THE BOOTCAMP, BEGAN THE free health services, fares at public universities, PROGRAM. access to poverty alleviation programs, and so on. In most cases strata 1 and 2 are subsidized by 99 COMPLETED AT LEAST the upper strata 5 and 6. 80% OF THE BOOTCAMP AND RECEIVED THE COMPLETION It classified areas on a scale from 1 to 6, as follows: DIPLOMA 1. Low-low The program took place from May to August 2016 in six 2. Low different local government training center locations 3. Medium-Low (Unidades de Vida Articuladas, UVAs). 120 students participated, 108 of whom completed the program but 4. Medium only 99 received a completion certificate (see Figure 2.2). A certificate was awarded to students who attended more 5. Medium-High than 80 percent of the classes. There were six classes of 20 people each (two in the morning, two in the afternoon, and 6. High two in the evening). The in-person training program was Sources: DANE, http://dane.gov.co; Congress of carried out in six UVAs with desktops, donated by EPM and Colombia. Law 142 from 1994 (July 11), article 102. the Secretariat of Youth, across the city. The Secretariat of Youth assigned students based on the UVA’s proximity to their homes. However, the use of the UVAs as bootcamp training locations proved to be problematic in some instances. As public spaces, UVAs may hold other activities in the same training space, causing distraction among students. In the first weeks of the program, a number of students were robbed at the UVAs that were in less safe locations. This was corrected by the Secretary of the Youth who arranged local police escorts for students’ transit between the UVA and the closest metro/ bus station. Table 2.1 summarizes the main characteristics of the intervention in Medellín. 12 Table 2.1: Decoding Bootcamps Program RCT in Medellín Impact Evaluation Methodology Randomized controlled trial (RCT) Coding Bootcamp Bootcamp provider World Tech Makers Implementation May–August 2016 dates Cost to participants Subsidies provided up to $250 (varied depending on the socioeconomic situation of the student) Bootcamp Ruby, Rails, HTML, JavaScript, etc. curriculum Number of 6 classes of 20 students each bootcamps Final class size 120 students (and 161 participants in the control group) Participants’ profile Age 18-20: 24 percent 21-25: 62 percent 25-28: 13 percent Gender Male: 72 percent Female: 28 percent Socioeconomic Strata 1-2 (65 percent), strata 3 (31 percent), strata 4-6 (4 percent) standing at baseline Employed (18 percent); Unemployed (82 percent) High school and Baccalaureate (35 percent); Technical diploma (38 percent); University students or graduates (28 percent) Source: Authors. 13 Theory of Change The “increased/enhanced skills” pillar aimed to increase: (i) coding skills acquisition, (ii) socioemotional skill The stated objective of the Decoding Bootcamps development, and (iii) job readiness. More than 90 percent program was to reduce youth unemployment, and provide of the bootcamp was dedicated to learning entry-level coding access to good quality jobs and higher job satisfaction, skills. As coding is taught through an experiential and project while promoting the technology sector and contributing to centered approach, students needed to develop certain skills the local economy’s transformation. The program provided to thrive, including socioemotional skills. They learned about training to young people in Medellín, defined as those teamwork, how to implement large projects, and how to work between 18 and 28 years old. The program was designed in a dynamic environment. around two main pillars: (i) providing participants with skills The labor market component included two main activities. relevant for the labor market (that is, coding, job readiness and Through an end-of-bootcamp project, students developed socioemotional skills); and (ii) establishing linkages between a minimum viable product for an external organization of graduates and employers (see Figure 2.3). Through this their choice. On September 19, 2016, WTM hosted a demo design, the program was expected to equip graduates with day at Ruta N’s headquarters, where some students presented skills to access entry-level IT jobs, which would progressively their websites. More than 400 people, including bootcamp turn into good quality jobs. An alternative path was to equip students, IT companies as potential employers, civil society, graduates with the skills and motivation to continue studies governmental institutions and other partners, were in in the IT area. The ultimate objective was to provide graduates attendance. Prior to the demo day, all students received an with better long-term employment options. eight-hour training on the elevator pitch methodology. Figure 2.3: Theory of change of coding bootcamps CODING BOOTCAMP INCREASED / ENHANCED SKILLS ACCESS TO TECH EMPLOYERS ACTUAL TREATMENT Increased / Increased / Increased / JobPresentation Expand Network (COMBINED) Enhanced Enhanced Job Enhanced /Job Hunting of Tech Peer Coding Skills Readiness * Socioemotional Skills Skills for Tech** “Professionals” *** HYPOTHESIS Access to Higher Level of Tech Self-Employment Low Entry-Level Tech (ALTERNATIVE Education/Higher Quality (Creation of Tech Startup/ Job (Junior developer) OPTIONS) Program on Tech or STEM Tech Freelancing) HYPOTHESIS (ALTERNATIVE Access to Job with higher AND COMBINED Long-Term Salary Potential Higher Job Satisfaction OPTIONS) * Practical Jobs Skills (eg simulations of real-work environment coding projects) ** Demo Day preparation and Demo Day presentations *** Assuming cohort peers will join tech work force Source: Authors. 14 WTM also provided job opportunities in technology related Table 2.2 shows the detail of the hypothesized impacts companies. WTM’s Medellín Office is based at Ruta N, where of bootcamp participation on job, educational, and many of the IT companies and startups in the city are located. socioemotional outcomes. This connection to the ecosystem is essential to get continuous and real-time inputs from companies on the tech skills that All outcomes in Table 2.2 are relevant for the economic they require to link graduates to employment opportunities. development and well-being improvement in low and For the experiment, all bootcamp participants were trained in middle-income countries. Jobs are the main pathway to the same coding languages (Ruby, Rails, HTML, JavaScript, and financial stability, savings, and higher consumption; education so on) determined by the local bootcamp provider, WTM. improves the odds of higher income and social mobility, reduces risky behaviors, and improves health outcomes; and finally, socioemotional skills are associated with general EVALUATION DESIGN better life satisfaction. Research Questions Table 2.2’s structure shows in the first column the three main categories of impacts of participation on coding bootcamps, The impact evaluation of the program has the purpose in the second column the outcomes (dependent variables) of answering the following research questions: does to be assessed, and in the third column the precise research bootcamp participation improve job and educational question that is going to be answered by this evaluation. outcomes? Does bootcamp participation lead to improved socioemotional skills? Table 2.2: List of variables and research question, by category Category Outcomes Research questions Are there differences between the unemployment rates of bootcamp Job participants, and members in the control group? Are there differences on the reported job satisfaction between Job satisfaction bootcamp participants and members in the control group? Are there differences on the probability of having job benefits between Job outcomes Job benefits bootcamp participants and members in the control group? Are there differences on the business creation rate between bootcamp Business creation participants and members in the control group? High-quality job Are there differences on the high-quality employment rate between statusa bootcamp participants and members in the control group? Program Are there differences on the program completion rate between completion bootcamp participants and members in the control group? Educational outcomes Are there differences on the program initiation rate between Program initiation bootcamp participants and members in the control group? Socioemotional Are there differences on socioemotional skills, as measured by GRIT, GRIT outcomes between bootcamp participants and members in the control group? Note: In the baseline and follow-up surveys the team asked participants’ title of their current job position. Jobs were considered to be high quality if they were tech-related or usually, in the Colombian context, required a four-year program. 15 The GRIT is a continuous scale that measures people’s Most of the outcomes created from Table 2.3 were analyzed determination, courage, and strength of character, used here as dichotomous variables. In the case of the variable of High- as proxy of participants’ long-term commitment, perseverance, quality Job Status, the title of the job position provided by and drive for success, that will be explained in more detail in the respondents was modified into a dichotomous variable the next section. differentiating high-quality jobs from the rest. There was only one exception: the GRIT measurement was left as a continuous variable, as reported, and it was analyzed in that way. Research Design To measure the socioemotional variable, participants The evaluation was designed as a randomized controlled were assessed on the 17 item GRIT questionnaire and the trial. This means that participants were selected randomly Review of Personal Effectiveness with Locus of Control among eligible candidates to belong to the treatment and (ROPELOC) instrument at endline. Impacts on GRIT were control groups to achieve two main goals: 1) participants in the analyzed and on the 15 items measured by ROPELOC (Active treatment and control groups would have similar observable Involvement, Cooperative Teamwork, Leadership Ability, Open characteristics on average, and therefore 2) outcomes from Thinking, Quality Seeking, Self-Confidence, Self-Efficacy, Social bootcamps participation could be causally associated with Effectiveness, Stress Management, Time Efficiency, Coping participation in the program. with Change, Overall Effectiveness, Internal Locus of control, and External Locus of Control). The version of GRIT used for Once the treatment and control groups were defined, this exercise had 17 items, self-reporting how likely it is that the field instruments were applied to both groups to respondents will react in a certain way when facing certain collect the necessary information to set up the variables situations, and participants earn between 1 and 5 points of interest. Table 2.3 describes the principal questions asked depending on their answers. These scores are then added to participants in treatment and control groups to assess the to generate a value between 17 and 85. In the end, a person hypothesized outcomes of bootcamp participation. As shown with a higher score is evaluated with a higher GRIT. On the in the table; these questions are related to the job, educational, other hand, ROPELOC measures each construct mentioned in and socioemotional outcomes derived from the research the parenthesis with three items. The score for these items is questions in Table 2.3 which shows all the questionnaires also added. used in the Colombian project. Table 2.3: Questions to determine experiment outcomes Category Outcomes Questions Job status Are you currently working? Job To what degree are you satisfied with your current job? (scale from 1 to 4, satisfaction 1=really unsatisfied, 4=really satisfied) Does your job provide any benefits? Health insurance, paid vacation, Job benefits Job outcomes training, pension fund, etc. Business After participating in the bootcamp, did you create your own business? creation High-quality What is the job title for your position at your current job? job status Program Since May 2016 have you completed any education program, besides the Educational completion coding bootcamp? outcomes Program Since May 2016, have you applied to any education program? initiation Socioemotional GRIT The 17 item GRIT questionnaire was applied. outcomes 16 Model To estimate the impact of the program on the outcomes described above, the following ANOVA model was used: The simple framework for the experimental design is OUTCOMEijk=μ+α_TREAT+β_OUTCOME AT BASELINE described in Equation 1 as an Analysis of Variance (ANOVA) +αβTREAT*OUTCOME AT BASELINE+ϵijk to determine, with the highest statistical power, the mean differences between treatment and control groups. Where the outcomes are those described in Table 2.10 to measure changes in labor, education, and socioemotional variables. Equation 1: ANOVA test for treatment impact To measure the average impact of the training program, intention-to-treat (ITT) effects were first estimated by ANOVA μ_(treat.pre-treatment control value)= to test the treatment effects on the hypothesized outcomes μ_(control.pre-treatment control value) (labor and educational outcomes, and socioemotional skills) on the random assignment to treatment variable. Then, a similar The means of the variables of interest for the treatment analysis was done to test the effects on the same outcomes of group after treatment were compared taking into bootcamp completion. Unconditional ANOVA will show if there account the values of these variables before treatment. are unconditional treatment effects on each of the desired labor Randomization of the individuals participating in the field and educational outcomes. experiment ensures that the bootcamp program, and no other confounding factors, explains the difference To validate this analysis Table 2.4 compares the people who in outcomes (randomization ensures that that baseline finished the bootcamp with those who did not. As shown in this values are similar between these groups). For more table, there were no statistically significant differences between information on the RCT research design, see Appendix A. participants who finished the program, interpreted as attending more than 80 percent of the sessions, and those who did not. Table 2.4: Comparison of participants who dropped out of the bootcamp and those who finished Variable Finished Dropped N Finished N Dropped P Value Bootcamp Out Out Age 24.6 24.1 170 109 0.16 Gender (female) 32.3% 28.4% 170 109 0.49 High socioeconomic level 35.9% 34.9% 170 109 0.86 Rural location 10.6% 11.9% 170 109 0.72 Baseline job status 19.4% 18.3% 170 109 0.82 Baseline job benefits 47.1% 52.9% 34 17 0.69 Baseline job satisfaction 34.4% 33.3% 32 15 0.94 Has work experience at baseline 80.6% 82.6% 170 109 0.67 Finished high school at baseline 28.8% 32.1% 170 109 0.56 Has some tertiary at baseline 67.6% 66.1% 170 109 0.78 Has a high quality or it job at baseline 43.5% 45% 33 20 0.97 Has own business at baseline 4.1% 5.5% 170 109 0.59 Source: authors’ calculations. 17 Then, the multifactor ANOVA other variables that are b) Treat*Female: Colombia is a very unequal society for women; usually associated with our chosen outcomes were women are less likely to get jobs and are more likely to included: age, gender, socioeconomic stratum, rural location, receive a smaller salary. In the context of this evaluation, it mother’s education, and maximum education level achieved. is in our interest to focus on programs that address these Table 2.5 shows these variables and the hypothesized differences in Colombia. We want that education and association with education and labor outcomes. employment programs have better or similar impacts on men and women, so gender gaps in Colombia are reduced. Table 2.5: Hypothesized relationship This interaction will help us identify the differential impact between control variables and labor and of the program for men and women. education outcomes c) Treat*High stratum: As with people with higher education levels, people from higher income backgrounds, in general, Variable Education Labor have better odds at having jobs, and having better paid Age Positive Inverted U ones. Another strong source of inequality in Colombia is family-income level. Colombia is the country with the Gender (female) Positive Negative second largest gap in Latin America between the poor and the rich, as measured by the GINI index, as well as a High socioeconomic level Positive Positive strong correlation between academic results and income Rural location Negative Negative level. We hope that educational and job programs are more beneficial for the low-income population than for the rest Mother’s education Positive Positive of the population; these interactions will allow us to identify this difference. Higher education level Positive Positive Table 2.6, Table 2.7 and Table 2.8 show that the balance Finally, a set of relevant interactions variables were added among interaction groups was broadly achieved. Table 2.6 to the model to identify differential effects of bootcamp shows that the distribution for women between socioeconomic participation among specific groups of the population: strata was not completely balanced. On average, there were treat*tertiary, treat*female, and treat*high stratum. It was more women in the low-income group than in the high-income hypothesized that these groups were more likely to be one. Caution should be used when interpreting the results for impacted by the program, for the following reasons: this interaction since it is likely that its effects will be more likely associated with strata than gender. a) Treat*Tertiary: People with tertiary education are more likely to find jobs; in Colombia, there are many recent graduates Table 2.7 and Table 2.8 show a more balanced scenario. who take more than six months to find a job (80.4 percent of The distribution of women and education level was better people who earned a tertiary education degree in 2013 were distributed than in the previous one, showing a small employed in 2014). This makes relevant the differentiation of asymmetry towards women being more educated. Again, the impact between people who have already completed interpretation of this interaction should take this into account this education level and people who have not. Since people and highlights that, for this interaction, the association between who do not have a tertiary education degree in Colombia are gender and the outcomes could be biased by education level. more likely to have lower incomes, lower quality jobs, and Table 2.8 shows an almost perfectly balanced distribution be unemployed. between education and strata. Table 2.6: Population balance between women and high strata Strata\gender Male Female TOTAL Low socioeconomic strata 117 (65%/60.2%) 63 (35%/73.3%) 180 (100%/64.5%) High socioeconomic strata 76 (76.8%/39.4%) 23 (23.2%/26.7%) 99 (100%/35.5%) TOTAL 193 (69.2%/100%) 86 (30.8%/100%) 279 Note: row percentage/column percentage. 18 Table 2.7: Population balance between women and tertiary education Gender\Tertiary Education No Tertiary Education Some Tertiary Education TOTAL Male 71 (36.8%/77.2%) 122 (63.2%/65.2%) 193 (100%/69.2%) Female 21 (24.2%/22.8%) 65 (75.6%/34.8%) 86 (100%/30.8%) TOTAL 92 (33%/100%) 187 (67%/100%) 279 Note: row percentage/column percentage. Table 2.8: Population balance between tertiary education and high strata Strata/Tertiary Education No Tertiary Education Some Tertiary Education TOTAL Low Socioeconomic Strata 59 (32.8%/64.1%) 121 (67.2%/64.7%) 180 (100%/64.5%) High Socioeconomic Strata 33 (33.3%/35.9%) 66 (66.7%/35.3%) 99 (100%/35.5%) TOTAL 92 (33%/100%) 187 (67%/100%) 279 Note: row percentage/column percentage. SAMPLE Data Collection Based on anticipated sample size requirements, Concerning statistical power, achieving one of 0.8 and a including attrition estimates, initial power calculations statistical significance of 0.05, the team selected a sample estimated that 120 participants were required for the size of 280-300 participants, with 120 participants in the treatment group, and a minimum of 120 participants treatment group. Power tests were conducted for a dichotomous for the control group. In other words, 120 participants outcome variable (employed versus not employed). The results of were randomly selected to receive the bootcamp training the power test supported the initial calculations for effect sizes of (the treatment group) and the remainder, which consisted 20-35 percent improvement from baseline. Significant attrition in of more than 120 people, were assigned to the control either would lead to a severely underpowered experiment that group. All the treatment and control participants would be would prevent detection of even significant impacts and would monitored for 6-9 months after the end of the bootcamp. therefore likely be inconclusive. Figure 2.4: Timeline of pilot activities in Colombia RUTA N RANDOMIZATION BOOTCAMP’S BOOTCAMP MAY 17, 2016 END SURVEY AUG 19, 2016 DEC 10-22, 2016 ADDITIONAL DATA BASELINE COLLECTION APPLICATION SURVEY (9-MONTHS AFTER GRADUATION) PROCESS MAY 30, 2016 JUL 26 – AUG 15, 2017 APR 16 - MAY 13 BOOTCAMP’S DATA COLLECTION START (6-MONTHS AFTER MAY 30, 2016 GRADUATION) FEB 20 – MAR 19, 2017 2016 2017 19 Data was collected at baseline (April-May 2016) through the The endline survey took place in March 2017, between six and program online application form, which included questions on seven months after the end of the bootcamp. Beneficiaries demographics, employment status, and education outcomes, and control group participants were asked to fill out a survey as well as salary aspirations and previous knowledge of which aimed to collect information on labor market and programming languages. All participants were made aware education outcomes; information for 239 participants was of the nature of the study and consented to participate. To be collected at the endline. eligible, participants needed to be willing to provide information regardless of their group assignment. In total, information of 420 Owing to the low response rate from the control group, eligible participants was collected at this stage. since there were many missing values in key variables in the endline survey, a survey firm was hired to make another Table 2.9 shows the differences between those who refused attempt to collect the missing data on both treatment and treatment and those who did not. Even though the groups are control groups. This was done through phone calls and in- not identical, in their observable characteristics, the differences person visits for the participants who initially refused to between the groups are not statistically significant. This means provide certain data. The next section describes in more that treatment refusal most likely happened because of detail the methodological implications of this and other unobserved characteristics. adjustments made in the evaluation. Table 2.9: Characteristics of participants compared to those who refused Variable Participant Refused N Participant N Refused P Value Age 24.4 23.9 279 140 0.09 Gender (female) 30.8% 32.9% 279 140 0.67 High socioeconomic strata 35.5% 33.6% 279 140 0.69 Rural location 11.1% 12.1% 279 140 0.75 Is working at baseline 19% 22.1% 279 140 0.44 Has job benefits at baseline 49% 50% 51 30 0.93 Is satisfied at baseline 34% 46.7% 47 30 0.27 Has work experience at baseline 81.4% 75% 279 140 0.13 Highest level of education is high 30.1% 27.1% 279 140 0.52 school at baseline Has some tertiary education at 67% 70.7% 279 140 0.44 baseline Job generates income at baseline 87.5% 80.6% 48 31 0.41 Has a high quality or IT job at 45.3% 41.9% 53 31 0.76 baseline Owns a company at baseline 4.7% 5.7% 279 140 0.64 Source: authors’ calculations. 20 Methodological Limitations been selected.This implies that those who did participate had one of the following characteristics: either they had The implementation of this evaluation was not free from time available and little chance of finding an occupation, problems. In this sense, it is important to mention at least or they were highly motivated to participate in the two types of concern: the first is related to the adjustments bootcamp. Each of these scenarios generates a possible in the selection strategy of treatment and control group from bias. For instance, people who were available could have the original plan; and the second is related to the limitation of low income, a poor network, and a lack of access to the endline data collection. employment or education opportunities; this could lead to negative bias in our results, because, in this scenario, In the first case, there were three factors leading to participants could be very vulnerable, and this was not changes in the structure of the treatment and control reflected in the data, leading to unobserved hardship on groups. First, participants were not allowed enough time to achieving positive employment and education outcomes. determine whether they wanted, and were able, to commit to On the other hand, if participation is associated with a three-month intensive bootcamp. This led many participants people who have more drive and motivation, it is likely to drop out. The second source of structural changes in the that our results are positively biased since, perhaps, treatment group had to do with bootcamp location; some participants’ success is more likely to be explained by participants were assigned to locations that were far away their own motivation than by program participation. This from their home, leading them to abandon the program issue is more important than the previous one, since it is before starting because of transport difficulties. The third hard to determine, with the existing data, the direction source was associated with the replacement strategy, and and magnitude of the bias (see Table 2.9). the magnitude of the replacements needed in the program. Whenever a participant refused the program, a person in • Finally, participants refused to provide all the the control group, belonging to a randomly created list, was information requested, and there were also some offered participation. This, added to the fact that 140 people problems in the accuracy of the information who were offered participation refused, altered the initial plan provided. Income information was difficult to collect: not of the construction of the treatment group. all participants accurately provided this information, and some of those who provided it did so in an inaccurate These problems with data collection and the construction way. Also, all the information was self-reported, so bias of the treatment and control groups could lead to biases, in the data provided, collected, and analyzed is likely. mostly because of omitted variables, for the following This source of bias is minor as the data collected was not reasons: tied to incentives. In Colombia, people usually do not like reporting their income: DANE works around this problem • Given that a large number of treatment group through proxies. participants were replaced from the intended control group, it is likely that the initial randomization could There were additional challenges concerning the data have led to essentially different treatment and control collection that could lead to possible biases. The surveys groups. Even though similarity between these groups specific to the impact evaluation were not the only ones that was confirmed, it was still possible that they differed on participants were asked complete, as other public entities nonobserved characteristics. In this case, this was not involved – partners of the WBG for this activity – also carried considered to be a significant problem, since the waitlist out their own surveys. This led to survey fatigue among was also randomly assigned. It is unlikely that participants participants. Some of the control group participants that had similar nonobserved characteristics, given that did not respond had their phone numbers deactivated after observable ones were still evenly distributed between participation, which was a cause for nonresponse. treatment and control groups. The team also tried contacting these participants • High program dropout was also another possible through other means (such as home visits or social source of bias. Some people dropped out because media) but some of these attempts at communication they were too busy by the time they were notified of were not successful. Finally, all 239 participants, for whom participation, because the location of the bootcamps data was incomplete, were contacted to provide the was far from their home, or because they never missing information. This greatly reduced the number of responded to e-mails and calls notifying that they had missing values. 21 Balance at Baseline It was assumed that no external factors other than Table 2.11 shows that participants from the sample were participation in the bootcamp would explain any more likely to have completed tertiary education, to be differences in outcomes between the treatment and male urban residents, and to come from slightly lower comparison groups. To validate this assumption, eligible strata than the average population of Medellin. However, participants in the respective groups were tested as to it is important to mention that the data for the experiment whether they had similar characteristics at the baseline. group in Medellin only included participants between the Random allocation of treatment assignment produced ages of 18 and 28. balance across the treatment and control groups, with no significant differences across 12 variables, including covariates and outcomes, as shown in Table 2.10. Table 2.10: Outcomes and covariates of treatment and control groups Main characteristics Treatment group Control group N Treat N Control p-value Outcome variables Job status (employed) 17% 20% 119 160 0.68 Job benefits (Yes) 8% 9% 18 33 0.25 Job satisfaction (satisfied) 31% 35% 16 31 0.61 Looking for a job (Yes) 88% 85% 119 160 0.26 Socioeconomic variables Age (Average) 24.2 24.6 119 160 0.88 Females 28% 33% 119 160 0.83 Average socioeconomic strata 2.3 2.4 119 160 0.89 Rural population 11% 11% 119 160 0.53 Mothers who finished high school 55% 60% 119 160 0.80 Only finished high school 33% 28% 119 160 0.2 Have some tertiary education 66% 68% 119 160 0.67 Have previous work experience 82% 81% 119 160 0.35 Source: authors’ calculations. 22 Table 2.11: Descriptive statistics of the sample Sample (Treatment and Control) Medellín Average age 24.4 34.9 Female (between 18 -24) 30.8% 55% Rural population 11.1% n.a. Father finished high school 53.7% 18.8%a Mother finished high school 58.4% 13.4%a Only completed high school education 30.1% 25.2% Have some level of tertiary education 67% 21.3% Average number of years of education 12.8 11.8 Average socioeconomic strata 2.3 2.2 Source: authors’ calculations and DANE (GEIH and ENCV 2015). Note: a. This data is for the Department of Antioquia in urban areas. Antioquia is Colombia’s most populated province and the country’s largest economy after the capital district of Bogota. Its capital is Medellín. RESULTS Table 2.12 shows that none of the treatment coefficients, linking job status and treatment, are This section describes the results of the empirical statistically significantly different from zero. In other specifications structured around the program’s theory of words, this means that the likelihood of being employed change. Results on labor market outcomes are presented first, was not significantly different between treatment and particularly the probability of being employed, and having a comparison groups on average. Models 1 through 3, job with benefits and job satisfaction. The section then explores which do not consider statistical interactions have average the program’s impact on acquiring high-quality or technical treatment effects that are not statistically different from 0. jobs. The impact of the program on education outcomes was This means that participation in the treatment does not then analyzed, particularly the probability of dropping out of a have an impact, in any direction, on job status. program, or starting a new program. Finally, tentative evidence of the impact of the program on socioemotional skills is explored. When interacting the treatment variable with a dummy variable for being a woman (column 4), having tertiary Given that the impact evaluation tests multiple hypotheses, education (column 5) or belonging to a household that the team made adjustments to prevent finding statistically lives in socioeconomic strata 3 or higher (column 6), no significant results by chance using the Bonferroni significant differential treatment effects between the adjustment. To make sure that our results are statistically robust, groups was observed, since none of the coefficients for the chosen alpha level to accept statistical significance is 5%. these interaction variables are statistically significant. Considering that the impact evaluation tests for seven different Further, when interacting both gender and socioeconomic outcomes, after doing the Bonferroni adjustment, statistical strata, there are no significant differences. significance is reached if p-values are smaller than 0.007. This means that treatment not only lacks an effect on job status (that is, being employed), on average, but Impact on Labor Market Outcomes also does not generate better odds for finding a job, in general, for people belonging to different population The impact of bootcamp participation on job status, job groups. There is no differential impact among people satisfaction, job benefits, business creation, and high-quality depending on their gender, socioeconomic strata, and employment status are analyzed below. Each section presents education level. the ANOVA results and a brief description of them. 23 Table 2.12: Treatment impact on job status   (1) (2) (3) (4) (5) (6) (7) Variables Unconditional Unconditional Regression Controls and Controls and Controls and Controls and Intention- Treatment on with interaction interaction interaction interaction to-treat the treated Controls between between between between gender and tertiary strata and gender, strata treatment education and treatment and treatment treatment Treatment 0.00545 0.00674 0.0496 -0.0309 -0.00916 -0.00910   (0.0554) (0.0556) (0.0673) (0.101) (0.0696) (0.0585) Completed -0.0312 80 percent of Bootcamp (0.0561) Treatment* -0.136 Female (0.121) Treatment* 0.0541 Tertiary Education (0.121) Treatment* 0.0443 High Strata (0.116) Treatment* 0.136 Female* Strata Observations 239 239 239 239 239 239 239 Note: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. 24 Results on job satisfaction show similar patterns. The Similarly, none of the interactions are statistically different from probability of being satisfied with their job is shown in Table zero even when they have positive coefficients. No impact 2.13. The average treatment effects for this variable (column 1) is observed, on average or for the previously mentioned show no impact on average. This also holds for the treatment population groups, on employee satisfaction resulting from effect of bootcamp completion; thus, it cannot be concluded participation in the program. These results may be influenced that treatment has an impact on job satisfaction. by the short timeframe for the population to become employed after the program (in some cases less than 6 months), which could make difficult to understand the satisfaction of a specific job compared with being employed for the first time. Table 2.13: Treatment impact on job satisfaction   (1) (2) (3) (4) (5) (6) (7) Variables Unconditional Unconditional Regression Controls and Controls and Controls and Controls and Intention-to- Treatment on with interaction interaction interaction interaction treat the treated Controls between between tertiary between between gender, gender and education and strata and strata and treatment treatment treatment treatment Treatment -0.141 -0.156 -0.0848 -0.162 -0.375 -0.170 (0.157) (0.178) (0.226) (0.369) (0.276) (0.190) Completed -0.144 80 percent of Bootcamp (0.163) Treatment* -0.243 Female (0.468) Treatment* 0.00767 ertiary Education (0.427) Treatment* 0.378 High Strata (0.364) Treatment* 0.129 Female* Strata Observations 279 279 279 279 279 279 279 Note: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. 25 Table 2.14 also shows that the average treatment effect is services, retirement savings, and work-related risk insurance).  not significant on the proportion of participants reporting to This is because of the high costs of formal labor, understanding have job benefits. On average, participants who attended and formal labor as having a “contrato laboral” which includes completed the bootcamp did not report higher job benefits than health benefits, retirement savings, work-related risk insurance, those who did not. Even when the coefficients for treatment are vacations, and reimbursements. Because of these rigidities, the negative, they are not statistically significantly different from zero. fact that a worker has additional skills may be irrelevant for the type of contract that they are offered. The lack of significant impacts in Colombia may be due in part to labor market rigidities, which leads companies to hire workers Regarding the interaction variables, the same lack of statistical with service contracts (prestación de servicios) without formal significance as in the previous models was observed. There benefits rather than as employees. In this kind of contract, people were no differential impacts of bootcamp participation on the only receive payment for services rendered, and in addition they acquisition of job benefits relative to people’s gender, strata, or need to arrange and pay for benefits themselves (mainly, health education level. Table 2.14: Treatment impact on job benefits   (1) (2) (3) (4) (5) (6) (7) Controls and Controls and Controls and Controls and interaction interaction Unconditional Unconditional Regression interaction interaction between between Variables Attempt to Treatment on with between between tertiary gender, Treat the Treated Controls gender and strata and education and strata and treatment treatment treatment treatment Treatment -0.0493 -0.0840 -0.0852 -0.257 -0.181 -0.107   (0.143) (0.161) (0.198) (0.346) (0.244) (0.171) Completed -0.0818 80 percent of Bootcamp (0.149) Treatment* 0.00418 Female (0.401) Treatment* 0.223 Tertiary Education (0.394) Treatment* 0.169 High Strata (0.318) Treatment* Female* Strata 0.171 Observations 279 279 279 279 279 279 279 Note: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. 26 Results on the probability of creating a business, in The same result was found for all the other models, Table 2.15, show a similar pattern. There was no impact considering interactions. As in the previous cases, there on average on participants; none of the coefficients for were no statistically significant impacts on business the unconditional models, or the controlled model with creation from bootcamp participation for the analyzed interaction, was statistically significant. This means that, population groups. on average, bootcamp participation had no impact on business creation. Table 2.15: Treatment impact on business creation   (1) (2) (3) (4) (5) (6) (7) Variables Unconditional Unconditional Regression Controls and Controls and Controls and Controls and Attempt to Treatment on with interaction interaction interaction interaction Treat the Treated Controls between between between between gender, gender and tertiary strata and strata and treatment education and treatment treatment treatment Treatment 0.0423 0.0490 0.0172 -0.00278 0.0935 0.0547   (0.0461) (0.0473) (0.0572) (0.0862) (0.0591) (0.0499) Completed -0.00406 80 percent of Bootcamp (0.0469) Treatment* 0.102 Female (0.103) Treatment* 0.0746 Tertiary Education (0.104) Treatment* -0.124 High Strata (0.0988) Treatment* -0.0493 Female* Strata Observations 239 239 239 239 239 239 239 Note: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. 27 As previously mentioned, the team also used a qualitative • 0 otherwise. variable to complement the analysis on job status and employment type. A dummy variable was created As Table 2.16 shows, there was no statistically significant of high job quality, as indicated by the job titles/positions impacts of bootcamp participation and completion on reported by participants. This variable was defined using the acquisition of high-quality or tech-related jobs. As in the following criteria: previous cases, there were no statistically significant effects of bootcamp participation on this variable. Even though, the • 1 if: position contained the word engineer, developer, treatment coefficients for ANOVA were positive, they were director (head or coordinator), programmer, analyst, not statistically significant, thus it cannot be concluded that advisor, or the name of a profession that usually requires bootcamp participation has an impact on acquisition of high- a four-year college degree, or if the name of the position quality jobs. had anything to do with the IT sector. 60.1% of participants had a job fitting these criteria. Table 2.16: Treatment impact on high quality and IT jobs   (1) (2) (3) (4) (5) (6) (7) Controls Controls and Controls Controls and and interaction and interaction Treatment Regression Intention- interaction between interaction between Variables on the with to- treat between tertiary between gender, Treated Controls gender and education and strata and strata and treatment treatment treatment treatment Treatment 0.0423 0.0490 0.0172 -0.00278 0.0935 0.0547   (0.0461) (0.0473) (0.0572) (0.0862) (0.0591) (0.0499) Completed -0.00406 80 percent of Bootcamp (0.0469) Treatment* 0.102 Female (0.103) Treatment* 0.0746 Tertiary Education (0.104) Treatment* -0.124 High Strata (0.0988) Treatment* Female* Strata -0.0493 Observations 279 279 279 279 279 279 279 Note: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. 28 The results on the impact of the program do not show statistically significant job outcomes. The program did Impact on Education Outcomes not have an effect on any of the job variables. This has to do The impact of this program on education outcomes can with two main reasons: the first (which will be developed in be observed through at least two channels: (1) returning the Summary of Results) has to do with the statistical power to school, and (2) leading students to consider education of the sample; the second is associated with the Bonferroni more relevant by not dropping out from the program. The adjustment, which was needed because of the multiple bootcamp sought to provide practical and technical skills, hypotheses testing. emulating the work environment. The lack of average impacts has at least three explanations. These activities could have two main effects on students’ First, the program may not be providing the skills required perception of higher education: first, to help them in the labor market; thus, it does not help employment determine their interest, and second, to help them see outcomes, or it may not be providing the skills that different the usefulness of those skills. This may have motivated populations need. Different populations enroll in the participants to successfully complete their studies if they were bootcamp with different skills; and bootcamp participation enrolled in a program. The widely documented shortcomings does not necessarily provide the needed skills for everyone. To in the quality assurance system, especially for technical and bolster the effectiveness of bootcamps, greater focus could be technological programs (which a majority of beneficiaries placed on the skills needed by particular population groups. were studying) may mean that participants benefit from a standardized, practical, market-relevant component to Second, even if the program is providing the right skills, motivate the continuation of their studies. labor market rigidities may prevent the application of these skills to job requirements. For example, well- On the other hand,participants not enrolled in any program documented constraints such as the relatively high costs of may have been motivated to return to school to pursue firing workers or the cost of formal employment may actually further studies. This could happen because, given high- be binding in the Colombian case. According to Clavijo, Vera, quality content and relevant skills being taught, participants Cuellar, and Rios (2015), the additional costs of formality in noticed that pursuing further studies would improve their Colombia amount to 49% of contract value. skills and job market opportunities, showing participants the importance and real-life applicability of these skills. It is also Third, the evaluation period may be too short, and the likely that bootcamp participants in Colombia do not need to targeted population may take more time to find formal immediately enroll in additional programs since they consider jobs. Most participants were studying at baseline, and the the bootcamps a formal education program. The following bootcamp was provided during the summer, with many section presents the results for these two variables. students returning to their programs after it ended. Timing was a constraint in the design of this evaluation. Thus, while The results on the impact of the program on educational short-term impacts may not be seen, it does not mean that outcomes are not statistically significant either. Table long-term outcomes would not improve. However, the impact 2.17 shows the impact of bootcamp participation on formal on educational outcomes, which are discussed next, can be education program completion. used as a proxy for future labor market opportunities. The study considers two educational outcomes: the probability of continuing with an existing educational program (broadly defined), and the probability of starting a new program. 29 Table 2.17: Treatment impact on dropping out from education programs   (1) (2) (3) (4) (5) (6) (7) Variables Unconditional Unconditional Regression Controls and Controls and Controls and Controls and Attempt to Treatment on with interaction interaction interaction interaction Treat the Treated Controls between between between between gender and tertiary strata and gender, strata treatment education and treatment and treatment treatment Treatment -0.157 -0.168 -0.159 -0.336* -0.118 -0.164   (0.0999) (0.102) (0.128) (0.193) (0.122) (0.107) Completed -0.189* 80 percent of Bootcamp (0.102) Treatment* -0.0264 Female (0.215) Treatment* 0.232 Tertiary Education (0.227) Treatment* -0.176 High Strata (0.230) Treatment* -0.0480 Female* Strata Observations 239 239 239 239 239 239 239 Note: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. Average treatment effects are not statistically significantly On the other hand, the impact on starting a program upon different from 0, after adjusting p-values with the Bonferroni completion of the bootcamps behaves in the same way as the adjustment. As in the previous cases, there are no statistically impact on dropping out from formal education programs (see significant impacts of bootcamp participation on educational Table 2.18. Once again, as in the previous cases conclusions program drop out. Some of the ANOVA coefficients are cannot be drawn on the impact of bootcamp participation on statistically significant at the 10% confidence level (before education outcomes. Bonferroni adjustment) and in the expected direction. This leads to the conclusion that the sample lacked the necessary power required to find statistically significant results. 30 Table 2.18: Treatment impact on starting a postsecondary education program   (1) (2) (3) (4) (5) (6) (7) Controls Controls and Controls Controls and and interaction and interaction Unconditional Unconditional Regression interaction between interaction between Variables Attempt to Treatment on with between tertiary between gender, Treat the Treated Controls gender and education and strata and strata and treatment treatment treatment treatment Treatment -0.0959 -0.102* -0.0805 -0.137 -0.0986 -0.0917   (0.0583) (0.0593) (0.0718) (0.108) (0.0744) (0.0625) Completed -0.0719 80 percent of Bootcamp (0.0593) Treatment*F -0.0682 emale (0.129) Treatment* 0.0500 Tertiary Education (0.130) Treatment* -0.00890 High Strata (0.124) Treatment* Female* Strata -0.0862 Observations 239 239 239 239 239 239 239 Note: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. 31 Impact on Socioemotional Skills measures people’s determination, courage, and strength of character. It has been used to determine people’s general Bootcamps are project-based, experiential, and executed in soft skills. dynamic and close to real-life environments. Even though the provider did not execute the socioemotional skills component, The regression outcomes show that there was no statistically it is likely that bootcamp participation could lead to improved significant impact of bootcamp participation on GRIT socioemotional skills. In this sense, measurement was focused (shown in Table 2.19). Also, no statistically significant impacts on the GRIT scale, since it is a reliable measurement of the were found on any of the ROPELOC constructs (thus, this is socioemotional skills most relevant to success. The GRIT scale not reported). Table 2.19: Treatment impact on GRIT   (1) (2) (3) (4) (5) (6) (7) (8) Controls Controls Controls and Controls and and interaction and Uncontrolled Uncontrolled Regression interaction interaction between interaction Attempt Variables Attempt to Treatment on with between between tertiary between to Treat Treat the Treated Controls gender, gender and education strata and strata and treatment and treatment treatment treatment Treatment -0.0612 -0.0391 0.0374 0.112 -0.0157 -0.0269 -0.0612   (0.0765) (0.0776) (0.0949) (0.142) (0.0977) (0.0822) (0.0765) Completed -0.0854 80 percent of Bootcamp (0.0768) Treatment* -0.234 Female (0.168) Treatment* -0.214 Tertiary Education (0.169) Treatment* -0.0643 High Strata (0.163) Treatment* Mother -0.0962 completed Highschool (0.209) Treatment* Female* -0.0612 -0.0391 0.0374 0.112 -0.0157 -0.0269 -0.0612 Strata Observations 207 207 207 207 207 207 207 207 Treatment 0.003 0.006 0.034 0.044 0.042 0.035 0.049 0.035 Note: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. 32 Summary of Results Sample Power The main finding from the impact evaluation in Colombia In general, the results presented in this section do not show is not that bootcamps do not help low income populations’ significant outcomes in the key three domains usually socioemotional skills, job status, or education. Rather, in order impacted by bootcamps. This section briefly summarizes have sound results, larger samples are needed to achieve and analyzes the results for each domain. Appendix E greater power. shows regressions that account for the lack of full program completion of the treatment group using an instrumental There are two sources of reduced power for the experiment. variable approach. The results do not change. The first is associated with the multiple impacts that bootcamps typically have (at least three) on jobs, education, Job Outcomes and socioemotional skills. This condition led to a Bonferroni This was the main goal of bootcamp participation. In this regard, adjustment to all the p-values, to ensure that statistically it would be accurate to state that the goal of the intervention significant outcomes were not found by chance. This made was not achieved. On average, bootcamps participants did not p-values smaller than usual, thus leading to the need for a report higher job satisfaction, job benefits, employment, high larger sample. quality jobs, or business creation. This means that participants’ The second source is associated with the sample’s power, and general job outcomes were not improved by bootcamp its associated minimum detectable effect (MDE), calculated participation. with the collected data. The team did ex post calculations As previously discussed, there are several possible reasons for to determine the MDE for each of the variables, given the these results. It is possible that program implementation and sample and each variable’s descriptive statistics. Table 2.20 curriculum, or treatment group identification design could shows the MDE for the model in general and for each of the have led to different outcomes than expected. The importance job status outcomes in percentage points, without taking into of better targeting women is highlighted, since this group was consideration Bonferroni adjustments. Impacts of the program underrepresented in the training. need to be quite large to be detectable with the sample used. Education Outcomes The secondary goal of bootcamp participation was to provide Table 2.20: MDE for impact evaluation useful skills and to show trainees that these skills were useful. This was not assessed, but the likelihood of dropping out from Variable Minimum Detectable Effect current formal education programs and of starting new formal education programs was measured, but no impact was found. General 0.20 This is most likely because of the time frame; perhaps these participants will enroll in other formal education programs in the future. Job Status 0.13 Socioemotional Outcomes Job Satisfaction 0.16 This part of the program was not fully implemented and, unfortunately, did not provide statistically significant results. Job Benefits 0.17 On average, participants did not have stronger GRIT than their peers. This is one of the key theoretical outcomes of bootcamp Business creation 0.06 participation. This most likely means that participants need the metacognition process to fully develop these skills. In the Colombian case, participants only worked collaboratively High Quality Jobs 0.09 in the dynamic bootcamp environment, but did not have workshops to develop and reflect on the process. 33 Key recommendations for next steps include: 6. Ruta N Corporation’s Landing Program provides business space to companies to quickly start operations in Medellín. • Focus intervention on the inclusion of women in IT-related jobs and training. The Colombian case shows that women 7. Even though when doing a randomized experiment, were underrepresented and had lower outcomes than their control variables are not necessary. male peers. 8. Observatorio Laboral, Colombian Ministry of Education. • Develop a platform for business creation support for lower income populations, and use the lessons of this training 9. This variable was considered, however, it has several program for future interventions in Colombia to guarantee possible sources of bias. In the baseline survey, people better implementation. were asked their approximate date of formal education. Six participants in the treatment group were already planning • Use larger samples, and choose providers who have to finish their formal education programs after the experience with larger bootcamps. bootcamp, while only four were planning to do so in the control group. Additionally, participants were asked the • Focus the research question on fewer outcomes, to prevent program finished by the endline. Among the differences large impacts of multiple hypotheses testing. between the treatment and control group were: (a) only five participants in the control group reported finishing “other” education programs. Among these programs the Notes: answers provided were: “diploma” and “English class” , and (b) 20 people in the treatment group reported finishing 1. http://www.intellectualcapitals.com/colombia-leads-in- “other” education programs. Six of them reported the office-outsourcing. bootcamp as the education program. Among the answers provided for this “other” classification was the “diploma” and 2. http://www.investincolombia.com.co/sectors/services/ “English class, but also: “certificate,” “analytics certificate,” software-and-services-it.html. “course,” and “seminars.” All of these categories could also 3. http://www.investincolombia.com.co/sectors/services/ be confounding variables for the bootcamp. software-and-services-it.html. 10. 0.05/7 = 0.007. 4. https://www.fdiintelligence.com/Locations/Americas/ 11. These are jobs that in Colombia usually require a degree. Colombia/Medellin-s-renaissance-how-tech-investment- For example, school teachers people a four-year college is-transforming-Colombia-s-second-city. degree or a normal school degree. These requirements are 5. See https://www.epm.com.co/site/nuestros-proyectos/ usually enforced more rigorously in cities proyecto-uva. 34 QUALITATIVE STUDY IN BEIRUT, LEBANON To complement the impact evaluation in Medellín, the rate – 34 percent— is alarmingly high, partially explained by World Bank carried out two qualitative case studies: one low domestic market demand for educated but not highly in Beirut (Lebanon) and another in Nairobi (Kenya). This experienced labor. Overall, the country lacks the capacity chapter describes the qualitative case study in Beirut, which to accommodate and use this specific human capital. The was framed around focus group discussions with bootcamp shortage of quality jobs is pushing youth, and others who are students before the program and following their graduation. It disillusioned with the status quo, to emigrate. According to was complemented by interviews with graduates’ employers an independent study that included 25 percent of emigrants in the IT industry, as well as a baseline, midline, and exit surveys as a sample, 26.6 percent of respondents had engineering that shed light on students’ perceptions and employment and technology degrees, 9.8 percent had mathematics and situation. computer science degrees, and 26.5 percent were specialized in business management (World Bank 2015). CONTEXT In 2013, the revenues of Lebanon’s ICT services sector, three- quarters of which is dominated by communications services, Why Beirut? contributed 2.8 percent to GDP (World Bank 2015). The tech ecosystem can therefore help diversify the ICT services sector Lebanon has continued to face security challenges since from traditional telecom services and equipment sales toward the end of the Lebanese civil war in 1990. More recently, the consumer-oriented applications development. country has been unable to escape unscathed from conflicts in its neighborhood, most notably the Syrian civil war, which has rendered Lebanon the largest host of Syrian refugees in The Lebanese Startup Ecosystem proportion to its population: the four million strong country is home to some two million registered and unregistered Syrian ICT startup entrepreneurship in Lebanon epitomizes the refugees. On the upside, Lebanon has managed to bypass the private sector’s solutions to the country’s socioeconomic domestic political deadlock, which culminated in the election problems of stagnating growth, a young workforce subject of a president after a two-year leadership vacuum. to unemployment, or brain drain. The startup ecosystem also facilitates Lebanon’s development into an innovative Lebanon is a regionally acclaimed leader in the provision of knowledge economy. high-quality education. Beirut, in particular, is also known for its growing multilingual entrepreneurial society. The city Lebanon’s tech scene is becoming increasingly attractive, has already developed some key elements of a technology driven by successful startups that have tapped into regional ecosystem, including incubators, venture capital firms, clusters, and global markets. The ArabNet Start-up Database identifies and a number of successful startups that reach out to regional about 170 tech startups that are currently operational. Among and global markets. them, 33 specialize in mobile app development, 24 in software, 19 in e-commerce, and 13 in entertainment, including games. The country and its capital, however, face a considerable urban Lebanon’s tech ecosystem includes a number of coworking unemployment challenge. Lebanon’s population is 88 percent spaces (for example, AltCity, Coworking+961, Cloud5, urban, and nearly one third of its population live in Beirut and DigiHive); business incubators (Business Incubation (World Bank 2015). The poverty level in Beirut is estimated at Association of Tripoli (BIAT)); startup accelerators (for example, 16 percent, the lowest in a country that has an overall poverty UK Lebanon Tech Hub, Speed@BDD, Berytech, and Flat6 rate of 27 percent (Central Administration for Statistics and Labs); networking and event organizers (for example, Bader, World Bank 2015). ArabNet, and Wamda); mentorship and support nonprofits (for example, Endeavor and Lebanon for Entrepreneurs (LFE)); Before the Syrian crisis, Lebanon’s unemployment rate as well as universities with technology and entrepreneurship was 11 percent. Today, over half of Beirut’s labor force (55 programs (for example, Lebanese University, Saint Joseph percent) are wage employees, 25 percent are self-employed, University (USJ), American University of Beirut (AUB), and and 13 percent are unemployed (Central Administration for Beirut Arab University (BAU)). Statistics and World Bank 2015). The youth unemployment 35 The Beirut Digital District (BDD), a large redevelopment SE Factory operates as a nonprofit organization and only project in the Bachoura neighborhood in the center of the accepts computer science students or graduates from less- city, aims to become the heart of Lebanon’s tech ecosystem. privileged backgrounds in order to provide them with the The project is a public-private partnership between ZRE, a real most up-to-date programming languages and tools to make estate firm, Berytech, and the Ministry of Telecommunications. them employable as junior web developers. SE Factory became BDD consists of several new and renovated buildings covering operational in March 2016. To date, it has trained 23 students three city blocks aimed at hosting established ICT firms in two cohorts. Its first cohort consisted of eight students from and startups, as well as other organizations involved in the the low-income strata, selected from 50 eligible applicants. ecosystem. By 2016, 55 companies and 700 employees were The second cohort trained 15 students. For a detailed profile expected to be working at BDD (World Bank 2015). of SE Factory, see Mulas and others (2017). The Lebanese government is addressing financial and entrepreneurial constraints to startups’ development. It RESEARCH DESIGN facilitates access to commercial bank funding, guarantees, and tax exemptions for new technology SMEs through institutions Table 3.1: Decoding Bootcamps Program such as Kafalat and the Investment Development Authority in Beirut of Lebanon. In addition, in August 2013, the Central Bank of Impact Evaluation Lebanon (Banque du Liban) issued an Intermediate Circular no. 331 that guarantees up to 75 percent of investment in Methodology Qualitative study (surveys, Lebanese startups (Mulas and others 2016, 51). According interviews, focus group discussions) to Banque du Liban, local and regional equity investment firms have invested over $10 million in Lebanese startups Coding Bootcamp since 2010. Other investment options include venture capital Bootcamp provider SE Factory firm investments (such as Berytech Fund II and IMPACT Implementation dates July–October 2016 Fund), grants and other financial mechanisms provided by international organizations and donors, and international Cost to participants $100 crowdfunding efforts (Mulas and others 2016, 51). Bootcamp curriculum Full-stack web development (Apache, SQL, PHP, HTML / CSS, The World Bank has been actively supporting the tech startup JavaScript, etc.) ecosystem in Lebanon. Current World Bank projects include Number of bootcamps 1 cohort assistance to support innovation for SMEs. Together with the Ministry of Telecommunications, the World Bank also devised Final class size 15 students the Mobile Internet Ecosystem Project (MIEP), which was later Final number of study 13 students cancelled as the leadership vacuum prevented its ratification. participants The project, however, was instrumental in helping ecosystem Participants’ profile stakeholders prepare and test the design of the Mobile Age 21-22: 46 percent Innovation Hub (MiHub), which became an unofficial forum for programs and events coordination within the ecosystem 23-24: 31 percent (Mulas and others 2016, 56). 25-26: 23 percent Gender Male: 69 percent PROGRAM BACKGROUND Female: 31 percent In 2016 and the first part of 2017, the World Bank partnered with Socioeconomic Employed (15 percent); Berytech, as research partner, and SE (Software Engendering) standing at baseline unemployed (62 percent); Factory, a coding bootcamp, to develop this assessment in economically inactive (23 percent). Beirut (see Table 3.1). The cost of the bootcamp was $100 per student, and partners agreed on a pay-per-survey scheme to Undergraduate student (23 fund up to $90 per student. SE Factory was selected because percent); Technical diploma it was the only bootcamp provider that was affordable to all (8 percent); Bachelor/Master income brackets of the population. degree holders (70 percent). Source: Authors. 36 Methodology: The qualitative research for the Beirut pilot program and confirmed their ability to commit to attending the was based on focus group discussions (FGDs) with bootcamp bootcamp on a full-time basis. Successful applicants who passed students (at the beginning of the bootcamp and four months the online application round were invited for interview, which after graduation) and interviews with bootcamp graduates’ were conducted over the phone or in person.The interviews were employers in the IT industry. In addition, the qualitative research aimed at assessing the applicants’ motivation level. was complemented by three quantitative surveys. All three pilots that are part of this study (Medellín, Beirut, and Nairobi) Curriculum: The SE Factory curriculum focuses on technical included the same baseline and exit surveys, and there was also skills (80 percent of the program content), complemented a midline survey in Beirut and Nairobi to better understand with soft skills necessary for web developers. The curriculum students’ perceptions. was developed in-house by combining best practice from international bootcamps (for example, Dev Bootcamp, Hack Owing to the small sample size (13 students), the conclusions Reactor) with local market demand. The team emphasized the drawn from the analysis in Beirut are not statistically significant. need to adapt the curriculum to the local context rather than However, they complement the results from the randomized simply replicating the approach of U.S. or European bootcamps. controlled trial (RCT) in Medellín. For example, SE Factory teaches PHP (server-side programming language for web development, which can also be used in For more information on the qualitative study design, see general-purpose programming) rather than Ruby (object- Appendix B. oriented general-purpose programming language) since the demand for PHP programmers is much higher in Lebanon. Selection criteria and process: On May 1, 2016, SE Factory opened applications for potential participants to the bootcamp, The SE Factory bootcamp runs for 12 weeks, Monday to Friday, announcing it through various channels, including universities, 10 am–7 pm. Berytech offers free transportation for accepted job boards, social media, information sessions, and mass mailing. students. This intensive training schedule requires a full-time commitment to the program, and students typically cannot To satisfy the minimum selection criteria, bootcamp applicants pursue work or other studies concurrently. were required to: (i) have low income; (ii) have a university degree in computer science or engineering; (iii) have basic English During the last two weeks of the bootcamp, students work on proficiency; (iv) be able to commit to full-time attendance of the individual projects, in which they develop a web application of bootcamp; and (v) be highly motivated to pursue the bootcamp their choice. The program ends with a demo day, where students education (as assessed through interviews). Conformance present their projects to partner companies. The bootcamp with these criteria was tested through an online application aims to facilitate students’ job search through introductions to form, covering the applicant’s educational and professional potential employers, and partner companies have first-hand background, including a personal assessment of various software access to these graduates. development skills, soft skills (for example, project management), and English language skills. The online application also Timeline of activities: Figure 3.1 illustrates the timeline of the contained a questionnaire about applicant expectations of the pilot’s data collection activities. Figure 3.1: Timeline of pilot activities in Beirut BOOTCAMP’S START BOOTCAMP’S END JUL 18, 2016 OCT 14, 2016 APPLICATION BASELINE PROCESS SURVEY MID-SURVEY JUL 18, 2016 NOV 14-30, 2016 INTERVIEW OF FOCUS EMPLOYERS GROUPS #2 JUL 18-29, 2016 FEB 10, 2017 FOCUS EXIT SURVEY GROUPS #1 MAR 13-17 AUG 12, 2016 2016 2017 37 The pilot activities comprised the following: • Students: Eight students participated in the FGD, which aimed to gain insight into how such bootcamps 4. Baseline survey: This survey included questions about could be more effective, and how they could be made students’ backgrounds and motivation for joining the a more useful tool for students. bootcamp. It was administered during the first day of the training. • Interested/prospective students: As the research aimed to gauge the effectiveness of bootcamps in 5. Midline survey: An online midline survey focused advancing the careers of students, this discussion was on graduates’ impressions of the bootcamp. It was held with six participants who did not go through the administered after the training was completed. bootcamp but were interested in the field. 6. Exit survey: This survey measured the impact of the • Bootcamp graduates’ employers: This FGD consisted coding bootcamp in terms of students’ employment of a group discussion with four employers of former and educational opportunities. It also compared their bootcamp students. It was aimed at understanding initial expectations with their progress six months after their perspectives of the competencies of bootcamp graduation. graduates, as well as understanding if there was a skills gap, and what their needs as employers were. 7. Interviews with employers: A Berytech researcher conducted six semistructured interviews with managers of firms who were potentially interested in hiring bootcamp graduates. In addition, four companies were surveyed through an online survey form. SAMPLE 8. Initial Focus Group Discussions (FGDs): The initial round Of 56 applicants to the bootcamp, a total of 15 students of three FGDs consisted of two-hour long sessions with who successfully completed the prescreening were invited the following: to participate in the second cohort and were included as participants in the World Bank study. Of the 15 participants, • Bootcamp students: 10 students participated in a FGD one did not fill out the midline survey and another refused to capture what the quantitative survey did not show to complete the follow-up survey; thus, this pilot followed the and to elicit information about how students felt at progress of 13 individuals. Throughout the course of the study, the beginning of the bootcamp. and out of the 13 participants, one student dropped out of the program because of family reasons. • Interested/prospective students: This session’s participants included those who had applied to the The group of respondents was composed of nine male (69.2 bootcamp but were not accepted, those who were percent) and four female students (30.8 percent). All were interested in applying but for various reasons were single and without children. Their average age was 24.5 years, unable to join the program, and/or those that were the youngest being 21, and the oldest being 26. At the time about to attend to an upcoming cohort. The aim of the baseline survey, nine students (69.2 percent) were was to have the viewpoints of these individuals in residents of Beirut (metropolitan area), two lived in another comparison to the responses provided by students large city (15.4 percent), and two lived in a rural area (15.4 enrolled in the bootcamp. percent). Eight of these students (61.5 percent) grew up in a different area and had moved to Beirut for education or • Mixed session: This FGD was a mix of three students, apprenticeship purposes. three interested/prospective students, and six experts/ representatives from the ICT industry, academia, Most students had a university degree: nine (69.2 percent) and the donor community. The aim was to initiate a had a Bachelor or Master’s degree, three (23.1 percent) were dialogue between the different stakeholders from the still attending an undergraduate program, and just one had ICT innovation ecosystem. a technical diploma as their highest level of education. In all cases, the students’ area of study was related to computer 9. Final Focus Group Discussions: The final round of three science or computer engineering. FGDs consisted of similar two-hour long sessions. The only difference in the format of these final FGDs was in Regarding their employment status, only two students (15.4 the composition of the third grouping: percent) were working at the time the bootcamp started, 38 while eight (61.5 percent) were unemployed (actively looking for a job), and three (23.1 percent) were economically inactive RESULTS (not working nor looking for a job). Of those not working, The following findings emerged from the data collected three (23.1 percent) had no previous work experience, and through the surveys, interviews with employers, and FGDs. for the ten participants (76.9 percent) who had previous The findings are grouped according to the following themes: work experience, six (46.2 percent) had experience in the IT- related field. Perception of Bootcamps Most students were familiar with Java and HTML, and some had previous knowledge of C++, iOS, Android, and Python. Perspectives of surveyed Lebanese youth: Both students In addition, most of their previous coding knowledge and and interested/prospective students mainly heard about skills were obtained at university, and only two students the bootcamp from referrals, in some cases from friends (15.4 percent) mentioned MOOCs as the source of obtaining (students that had participated in the previous cohort) programming skills. or from professors at their respective universities (e-mails were sent from the Dean’s Offices and SE Factory organized As the group was homogeneous, the research also examined information sessions), as well as from social networks. Those students’ households in order to characterize initial students referred to SE Factory by universities said that they endowments. As illustrated in Figure 3.2, almost 50 percent of were excited to apply to the program when they received the students’ parents had attained some form of higher education e-mail, and about a third of members of their class applied (Superior Technical Diploma, Bachelor/Master’s degree), to the bootcamp. Owing to the competitive process, they but the distribution was uneven by gender as 54 percent of appeared glad to have been accepted into the program and fathers had attained higher education and only 38 percent to have the opportunity to further their technical skills. Those of mothers had reached this level. In terms of employment who heard about SE Factory from friends who participated in status, in most of the households there is at least one member unemployed (see Figure 3.3). Figure 3.2: Highest level of education Figure 3.3: Work status of family obtained by parents of bootcamp members in Beirut respondents in Beirut 10 Bachelor Degree /Master Degree TOTAL PERSONS IN HOUSEHOLD 8 Superior Technical Diploma 6 Secondary Education /Technical Diploma 4 Intermediate Education 2 Primary Education 0 0 1 2 3 1 2 3 4 5 6 7 8 9 10 11 12 13 PARTICIPANTS' FAMILIES Total persons Unemployed Employed Mother Father in household 39 a previous cohort were encouraged by their experience and For small companies, it is important to get quality hires as they their improved job prospects following the training. have less resources to invest in training new employees. Two employers stated that they had to create new positions as they Word-of-mouth appears to be the best means of marketing the were not able to choose from graduates that applied for one bootcamp. Young people are increasingly hearing more about job. In comparison to top graduates from the best universities bootcamps from managers in programming companies, and in Lebanon, which might sooner or later be contacted by being a graduate of the bootcamp is associated with better leading multinational IT companies, employers feel bootcamp prospects of being hired. graduates bring a high level of commitment to the company and are looking to work for a purpose. When joining the bootcamp, students reported mixed reactions from their families and friends, although most were positive and supportive, with some families providing Bootcamp experience financial support. One of the participants noted that his friends disapproved of him quitting his paid full-time job. Another The impressions of the students can be summarized by one participant needed more time to explain to his family why he student’s response, who said, “The bootcamp exceeded my was applying for the training program and not seeking a job expectations because not only did I learn how to code, I instead. learned how to learn by myself.” Students felt that an intensive, full-time training could prepare Most of the students (85 percent) valued technical coding skills them for the job market. However, some young people did not as the most important skills imparted during the bootcamp, apply to the bootcamp because of its duration. Given that most while the remaining students said socioemotional skills were of those interested in bootcamps were university students the most valuable skills they acquired. However, all students or recent graduates, they either had school commitments agreed that both sets of skills were crucial to their success or were looking for a job and, therefore, found it difficult to during and after the bootcamp. participate in the program. Technical coding skills: Students reported that SE Factory Of the interested/prospective students, those who were had provided them with relevant skills that the IT industry unable to join SE Factory were committed to reapplying required right now, thus enabling them to get hired in the to future batches, in some cases because they did not like sector. Students felt that this sets bootcamps apart from their current job and wanted to be programmers. Some universities, which only provide students with a basic level of participants had been using online resources to learn understanding of how programming languages work. more coding, but they stated that they would like to gain a bootcamp experience. They also saw the intensive training as Socioemotional skills: When it comes to socioemotional a way to get two years’ worth of experience in three months’ skills, both industry representatives and students stressed the time. In general, they considered that if they had attended the value of communication skills, teamwork, flexibility, passion, bootcamp, they would have different opportunities and been adaptability, and problem solving. Students considered that better equipped with technical skills that would have made they gained a lot in this area, and now appreciated that most them more employable. reputable companies value these skills. Another skill they reportedly gained was related to the way of learning and Perspectives from the tech industry: Interviews revealed researching; there are many programming languages in use that potential employers exercise a “wait and see” strategy and every company has its own specific needs, so learning towards bootcamps as they are still a new method of how to learn is important. Students said that they now had acquiring coding skills in Beirut. A coding bootcamp that is a complete and different mindset that gives them the ability well connected to the ecosystem usually brings in industry to learn anything they want. They did not just learn how to stakeholders to be instructors, mentor the students, or provide code, but they learned how to solve problems. In addition, sessions on topics of interest for the students. Employers said the bootcamp reportedly provided students with networking this is a good opportunity to get a sense of their potential hires opportunities, exposing them to people such as startup as they not only see how skilled the students are in a particular founders, managers, and other top software engineers in technology but also how interested and passionate they are. Lebanese or foreign companies. Some recruitments have been done through connections like these. Students also get excited when companies visit SE Job hunting: Some students noted they had negative and Factory and see the work they have done. stressful experiences while looking for jobs after graduating from university. By contrast, after the bootcamp, students 40 began getting job interviews, which also consisted of percent) reported university graduation dates between 2016 technical exams, and they were proud of how they faced them. and 2017, meaning that they would have been entering the They were interviewed on familiar topics, and this reinforced job market in the short-term anyway. Other factors should their confidence in the training provided by SE Factory. At be considered in analyzing whether the current employment the beginning of the bootcamp, students were told that the situation of students was directly related to the bootcamp bootcamp was going to be a tough and intensive experience, training they received. Students reported that they received based on the military bootcamp methodology. Now that some several job interviews after the training, and six of them (46.1 of these students have started to work, they appreciate that percent) received job offers at a multinational company in the the bootcamp prepared them for their work lives. Moreover, healthcare sector. thanks to the portfolio and CVs they prepared during their training, companies were able to ascertain the students’ skills Eight respondents (61.5 percent) used both the coding skills and knowledge. Students also felt that succeeding in the and soft skills acquired in the bootcamp in their current jobs, bootcamp required not only persistence and support from while three of them (23.1 percent) only used the coding the instructors, but also teamwork and dedication. skills. Similarly, nine respondents (69.2 percent) thought they received their current job because of the knowledge acquired in the bootcamp. Beyond that, the quality of employment is Tuition Fees relevant here: four former students (30.8 percent) were not satisfied with their current jobs because of the pay, absence of As the coding bootcamp in Lebanon was highly subsidized benefits, or long commute to work. (students only paid the equivalent of $100), the topic of tuition costs was not brought into the discussion. At the beginning of the bootcamp, students expected it would be difficult to find a job as employers usually preferred applicants with experience. When the baseline survey and Employability initial FGDs were conducted, students were asked where they aspired to work after completing the training: five wanted a The companies that participated in the FGDs noted that when job in a large private company (38.5 percent), three aspired looking for new hires in the programming field,they usually look to a job in a startup (23.1 percent), two wanted to work in a for soft skills and interpersonal skills as well as an applicant’s multinational corporation (15.4 percent), and two wanted to aptitude and passion for technology. Some companies still start their own businesses (15.4 percent). Students declared prefer university level education. The predominant majority that working for a company was the safest path, but that of their hires are young people between the ages of 15 to creating a startup was most exciting. 35 years. These companies consider the average salary of an entry-level programmer ($1,000–1,500 per month) to be Of the 11 graduates (84.6 percent) who were working at the acceptable. time of the follow up survey, five were employed in a small private company (38.5 percent), five were working in a large All students except for one (92.3 percent), including the private company (38.5 percent), and only one was working in a student who dropped out, felt confident about their future startup (7.7 percent). Only four graduates (30.8 percent) were employment prospects having completed the bootcamp. They working in a company they aspired to, and two of them (15.4 believed that the experience they gained during the training percent) were not satisfied with their current jobs because of had strengthened their portfolios and equipped them with the salary and absence of benefits. the skills to problem solve and even learn new languages if need be. In addition, 9 out of 13 respondents (69.2 percent) Regarding entrepreneurship, students said that they had reported a high level of motivation in terms of looking for a been encouraged to create their own startups and that job after the training, and none experienced a drop in this they did have some ideas to start with; however, they still motivation level. needed some experience before they could embark on becoming entrepreneurs. Though SE Factory gave them a Postbootcamp, 11 out of the 13 respondents (82.6 percent) good foundation, they also needed to learn the technical and were currently employed; thus, the bootcamp may have business aspects involved in starting and running their own improved the employability of students, given that eight of companies. Among the difficulties they perceived in terms of them (61.5 percent) were unemployed at baseline. However, becoming an entrepreneur were competition, funding, and when analyzing the employability of students, the “diploma marketing requirements. effect” should be taken into account as nine of them (69.2 41 Bootcamps versus Employers pointed to the country’s brain drain as a challenge. There was no agreement on the numbers University Education though, as some said that about 80 percent of Lebanese universities’ graduates were emigrating to neighboring Most of the student respondents agreed that universities and countries (especially in the Gulf ) or were being retained bootcamps complement one another. They felt that bootcamps by multinational companies. University deans testified that played a role in bridging the gap between finishing university about 20 percent of their graduates leave the country. In any and entering the job market. Also, many felt that university case, brain drain seems to be one of the major challenges on education was not always geared towards market needs; IT the supply side. languages change a lot, and universities do not adapt as fast. Moreover, the respondents noted that universities did not Most interviewed companies that hire entry-level provide the orientation needed in terms of contextualizing programmers require new hires to go through a lengthy, coding, software development, and engineering to the three to six-month training program in the workplace. In this wider market. light, the previous knowledge of the trainee does not matter that much as they are trained for the new position. However, Skills Gap companies are becoming reluctant to pay for this training. They need graduates to commit to the company for a long Employers reported that the software engineering field is period and prevent them from moving to competitors after currently one of Lebanon’s biggest assets, with many companies having been trained. developing rapidly and jobs being created. Employers and SE Factory is trying to narrow this gap between supply and industry experts also stated that even American companies demand of the available talent.The talent needs to constantly recruit their engineering teams from Beirut because its talent evolve, and the tech industry feels that bootcamps such as could become world-class and was very affordable. At the same SE Factory are making a difference, having already noticed time, demand for software developers in Lebanon is increasing an increase in the quality of their new hires. As opposed to because more startups are emerging and the ecosystem SE Factory, experts believe that other coding schools that is growing. train students with no programming background rarely The challenge companies face is on the supply side; there are not prepare them to be fully professional junior or midlevel enough students enrolled in computer science and computer programmers. There are many computer science graduates engineering programs to meet the market’s needs. However, from universities who know the computer science theory companies acknowledged that there was more talent available but that information is rarely related to the direct needs of now compared to a few years ago, when it was very scarce. the industry, so SE Factory covers this gap. 42 QUALITATIVE STUDY IN NAIROBI, KENYA As in the case of Lebanon, the World Bank carried out a However, Nairobi also faces many development challenges, qualitative study in Nairobi. It was framed around focus group including poverty and youth unemployment, as the most discussions with bootcamp students before the program populous county and one of the most expensive cities in Africa. and following their graduation. It was complemented by Inhabited by 3.2 million people, Nairobi is home to a quarter interviews with graduates’ employers in the IT industry, as of all urban dwellers in the country. Moreover, half of Nairobi well as a baseline, midline, and exit surveys that shed light on residents live in slums and informal settlements (UN Habitat students’ perceptions and employment situation 2014). According to the 2005-2006 Kenya Integrated Household Budget Survey (KIHBS), 22 percent of Nairobi’s population CONTEXT lives below the poverty line, which is considered one of the lowest rates in Kenya. However, Nairobi’s total number of poor (632,373 people) is significant. Why Nairobi? Along with the counties of Garissa and Mandera, Nairobi has Kenya’s technology sector has been one of the fastest growing one of the highest unemployment rates in the country at 7.6 in the world over the past decade. It has become a model for percent. Among Nairobi’s residents, 66.2 percent have attained technology investment in developing countries, including in secondary education or higher, but only 51 percent are Africa. The birthplace of M-Pesa, the highly successful mobile- employed. To compound the problem, over half of all jobs in based money transfer service, and Ushahidi, a crowdsourcing the capital city (51.4 percent) are still in the informal sector (UN platform to track violence in real time and gather data from Habitat 2014). Young people aged 15-34 years old constitute the public using SMS text messaging, Kenya’s capital city, 49 percent of Nairobi’s total population (UN Habitat 2014); Nairobi, is recognized as a hub for technology innovation. this can partially be explained by youth migrating from rural areas in search of jobs. This high proportion of youth further Three factors contributed to Kenya’s emergence as contributes to employment challenges in the capital city. a technology innovator in Africa. First, the increased recognition that the technology sector can help improve both commerce and civic participation has led to the Nairobi’s Startup Ecosystem establishment of facilities to support innovation among Kenyan programmers, entrepreneurs, and civil society Today, Nairobi has a vibrant mobile technology startup professionals. The most distinguishable among them is ecosystem, which includes over 120 startups (for example, the iHub, an innovation space and a catalyzer for the tech mFarm, Start-up Digital Kenya, Silicon Ridge Tech, Duma community in Nairobi. Second, the Government of Kenya has Works, and SafePay Solutions), about seven coworking spaces spearheaded a number of political and economic reforms (for example, Nairobi Garage, iHub, Nexus), twelve startup to support the ICT sector and spur technological changes, accelerators (for example, 88mph, Merck, GrowthAfrica, and such as building Kenya’s own submarine communication Village Capital), ten business incubators (for example, Nailab, cable (The East African Marine System, TEAMS) in 2009, iLab Africa, m:lab, and C4D Lab), and ten IT consulting firms (for which boosted bandwidth, increased the number of example, Bizzlab Kenya Holdings Ltd, UX KENYA, and Intercom Internet users, and significantly cut prices for end users. The Microsystems). Branches and research centers of multinational government has also adopted Vision 2030, a strategy which corporations (for example, Google, Microsoft, GSMA, Nokia, promotes science, technology, and innovation as the main IBM), as well as a host of international donors also participate implementation instruments for social development and in supporting this ecosystem. In January 2017, m:lab East crosscutting solutions to challenges faced by others sectors Africa and the World Bank initiated Traction Camp, a training of the economy. Third, the groundbreaking success of M-Pesa and coaching program that aims to help digital and mobile and Ushahidi further catalyzed Kenya’s computer literate entrepreneurs in the region. In March 2017, Nairobi Tech Week, (and mostly young) population to leverage technology for Sub-Saharan Africa’s largest tech event, took place in Nairobi, the creation of innovative solutions targeting local problems. powered by Moringa School in partnership with Facebook.1 43 About a dozen organizations in this rising tech entrepreneurship scene in Nairobi (also called the “Silicon RESEARCH DESIGN Savannah”) specialize in offering coding training to local Table 4.1: Decoding Bootcamps Program tech talent. These programs vary in terms of timeframe, mode of operation, coding languages, curriculum, students’ in Nairobi background, mentoring, employment support programs, Impact Evaluation and so on. Such organizations include AkiraChix, Moringa School, and Andela, which each provide coding training at Methodology Qualitative study (surveys, different skill levels. Akirachix targets young women and interviews, focus group discussions) girls to develop basic digital skills whereas Moringa offers Coding Bootcamp a coding bootcamp to men and women to become job- ready, entry-level developers in Kenya. Andela’s bootcamp Bootcamp Moringa School model includes a fellowship program through which provider trainees acquire on-the-job experience with mostly large Implementation April–August 2016 U.S. companies. It should be noted that reliable data on dates Kenya’s supply of skilled programmers and its growing demand is not currently available. Cost to $2,500, with a $250 subsidy for participants surveys (prices have now changed) This chapter aims to offer a qualitative assessment of Bootcamp Android, Python, UI and UX, HTML a coding bootcamp in Nairobi, shedding light on the structure and CSS, and JavaScript potential impact of this type of training in the context of Kenya. A qualitative approach, leveraging surveys Number of 1 cohort and focus groups, was applied for the preparation of bootcamps this chapter, given that the sample size was too limited Final class size 18 students at the time of the research to conduct a fully-fledged randomized control trial impact evaluation. Final number of 16 students study participants PROGRAM BACKGROUND Participants’ profile In 2016 and the first part of 2017, the World Bank Age 20-24: 44 percent partnered with iHub Research and the Moringa School, a 25-28: 50 percent coding bootcamp, to conduct an assessment of a coding 29-32: 6 percent bootcamp training program in Nairobi (see Table 4.1). Moringa School was selected because it was the only Gender Male: 75 percent coding bootcamp provider in Nairobi using the model. Female: 25 percent The cost of the bootcamp was $2,500 per student and the World Bank provided a subsidy for willing bootcamp Socioeconomic Employed (6 percent); unemployed students based on a pay-per-survey scheme to collect standing at (44 percent); economically inactive data for the research, which amounted to up to $250 per baseline (50 percent). student. Since January 2015, Moringa has trained eight Undergraduate student (25 full-time cohorts, graduating 216 students. In addition to percent); University graduates (38 the 19-week intensive full-time course on which this study percent); University dropouts (31 did research, the school also offers Moringa Prep, a five- percent); Vocational training (6 week programming course for beginners, either full-time percent). University graduates in IT/ or part-time. For a detailed profile of Moringa School, see Engineering fields (93 percent). Mulas and others (2017). Source: Authors. 44 Methodology: The qualitative research for the Nairobi pilot Owing to the small sample size (16 participants), the was based on focus groups with bootcamp students (at the conclusions drawn from the analysis in Nairobi are not beginning of the bootcamp and five months after graduation) statistically significant. However, they complement the results and interviews with bootcamp graduates’ employers in from the randomized controlled trial (RCT) in Medellín. the IT industry. In addition, the qualitative research was complemented with three surveys. All three pilots that are For more information on the qualitative study design, see part of this study (Medellín, Beirut, and Nairobi) included the Appendix B. same baseline and exit surveys, and there was also a midline Timeline of activities: Figure 4.1 illustrates the timeline of survey administered in Nairobi and Beirut to understand data collection activities: students’ perceptions. Figure 4.1: Timeline of pilot activities in Nairobi BASELINE SURVEY APR 4, 2016 BOOTCAMP’S START BOOTCAMP’S END APR 4, 2016 AUG 5, 2016 APPLICATION MID-SURVEY PROCESS NOV 14-30, INTERVIEW OF MAR 2016 2016 EMPLOYERS APR 4-18, 2016 FOCUS GROUPS #2 DEC 2016 FOCUS GROUPS #1 EXIT SURVEY APR 22-23, 2016 MAR 13-17, 2017 2016 2017 45 The activities comprised the following: was in the composition of the third grouping: 1. Baseline survey: This survey included questions about • Students: Ten students participated in the FGD, which students’ backgrounds and motivation for joining the aimed to gain insight into how bootcamps could be bootcamp. It was administered during the first day of more effective, and how they could be made a more the training. useful tool for students. 2. Midline survey: An online midline survey focused • Interested/prospective students: As the research on graduates’ impressions of the bootcamp. It was aimed to gauge the effectiveness of bootcamps in administered after the training was completed. advancing the careers of students, this discussion was held with seven participants who did not go through 3. Exit survey: This survey measured the impact of the the bootcamp but who were interested in the field. coding bootcamp in terms of students’ employment and educational opportunities. It also compared their • Bootcamp graduates’ employers: This FGD involved initial expectations with their progress six months after five employers of former bootcamp students. It was graduation. aimed at understanding their perspectives of the competencies of bootcamp graduates, as well as 4. Interviews with employers: Over the initial two weeks understanding if there was a skills gap, and what their of training, a researcher conducted five semistructured needs as employers were. interviews with managers of IT firms who were potentially interested in hiring bootcamp graduates. These interviews aimed to establish employers’ perceptions of the Kenyan IT sector and their opinion of bootcamps. SAMPLE 5. Initial Focus Group Discussions (FGDs): The initial In February-March 2016, Moringa School selected 18 applicants round of three FGDs consisted of two-hour long sessions to its sixth cohort. The preinterview process was focused with the following: on the basics of programming: students had to complete specific tasks on the SoloLearn website, which provides free • Bootcamp students: 10 students participated in a FGD online programming courses and a certificate of completion to capture what the quantitative survey did not show to Moringa School. Upon completing this stage, applicants and to elicit information about how students felt at were invited for in-person interviews, which evaluated their the beginning of the bootcamp. personality traits and motivation via behavioral questions, and • Interested/prospective students: This session’s 10 also included coding challenges to assess applicants’ problem- participants included those who had applied to the solving skills. Twenty-seven applicants were preselected from bootcamp but were not accepted, those who were a larger pool of applicants (70) and invited to complete a one- interested in applying but were unable to join the month prebootcamp offsite training. This course covered the program, and/or those who applied and had been foundations of major high-level programming languages, accepted into an upcoming cohort. The aim was and only those who successfully completed their weekly to compare the viewpoints of these individuals to assignments became eligible for the actual bootcamp. the responses provided by students enrolled in the The accepted students formed part of the sixth cohort at bootcamp. Moringa and received the intensive 16-week bootcamp • Mixed session: This FGD was a mix of five students, training that ran five days a week from 8:30 am to 8:00 pm. five interested/prospective students, and five experts/ In the first three weeks, the curriculum covered the basics of representatives of the ICT industry, academia, and the front-end web development (HTML/CSS), while weeks four to donor community. The aim was to initiate a dialogue seven were devoted to back-end web development (Python/ between the different stakeholders from the ICT Django); weeks eight to eleven focused on learning how innovation ecosystem. to build Android applications, and the final five weeks were dedicated to students’ individual projects. 6. Final Focus Group Discussions: The final round of three FGDs consisted of similar two-hour long sessions. The A total of 18 students who had successfully completed only difference in the format of these final focus groups Moringa School’s precoursework were invited to participate 46 in the sixth cohort and included as participants in the World engineering, or mathematics. Only one student had secondary Bank bootcamp research. During the course of the research, education as their highest level of education. one student dropped out from the training because they had difficulty in keeping up and refused to complete the follow- On their employment status, only one student was working up survey. The data from another student’s baseline survey when the bootcamp started (in a field unrelated to was lost, so this study ended up following 16 individuals over technology), while seven (43.8 percent) were unemployed the training period. and actively looking for a job, and eight (50 percent) were economically inactive. Ten students (62.5 percent) had prior Within the sample of bootcamp participants, 12 were male work experience and, of those, five (31.3 percent) had worked (75 percent) and four were female (25 percent). The average in the IT sector. All students claimed to be able to code in at age of the students was 23 years (the youngest being 20 and least one language, which was usually HTML5, Java, or C++. the oldest 32). When the baseline survey was implemented, all students, except for one, were residents of Nairobi, although As the group was homogeneous, the research also examined ten grew up in another city and moved to Nairobi for the students’ households in order to characterize initial education/training purposes. endowments. As illustrated in Figure 4.2, almost 70 percent of students’ parents had attained higher education (vocational, Most participants had a university education: six (37.5 university or postgraduate), but the distribution was uneven by percent) were university graduates in the IT or engineering gender as 92 percent of fathers had attained higher education field (two had also attended postgraduate courses), four while only 50 percent of mothers had reached this level. In (25 percent) were about to finish university before the terms of employment status, in 50 percent of households, bootcamp started, and five (31.3 percent) attended university at least half of the members were unemployed (see Figure but did not graduate. Of the 15 participants that had gone 4.3), which could imply a high economic dependency ratio2 through university education (attended and graduated), 14 that pressures students to get any job rather than jobs that (87.5 percent) had studied a subject related to technology, necessarily draw on their IT qualifications and skill sets. Figure 4.2: Highest level of education Figure 4.3: Work status of family members obtained by parents of bootcamp students in Nairobi in Nairobi 10 Post-graduate studies TOTAL PERSONS IN HOUSEHOLD 8 Vocational Education University 6 Secondary Education 4 Elementary education 2 Other 0 0 2 4 6 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 NUMBER OF PARENTS PARTICIPANTS' FAMILIES Total persons Unemployed Employed Mother Father in household 47 RESULTS bootcamps will not replace university education. What they look for in new hires is theoretical knowledge, internship The following findings emerged from the data collected experience, aptitude, and soft skills, which for them seem through the surveys, interviews with employers, and FGDs. to be demonstrated more through university success than The findings are grouped according to the following themes: coding bootcamps. Employers that are aware of bootcamps considered them Perception of Bootcamps to be a good way to introduce people to programming. Graduates of bootcamps are not experts, but they gain the Perspectives of Surveyed Kenyan Youth tools for continuous learning on their own. These employers noted that the bootcamp’s technical training seems to be Young Kenyans3 heard about the bootcamp from several quite in-depth. Employers also valued the soft-skills training sources. Some of the participants in the focus group that bootcamps provide (how to send an e-mail, how to discussions (FGDs) had attended a short (two-ten day) structure a conversation, how to solve a problem, and so on). bootcamps or hackathons, which had aroused their interest They saw bootcamps as a launchpad for those students with a in participating in a longer bootcamp. Others found out willingness and curiosity to learn, which helps employers avoid about the coding bootcamp by referral, through the iHub “bad hires.” In addition, through bootcamps, students establish or the Internet, where Moringa School is well positioned. networks that are critical for professional development. Students listed job placement opportunities, networking with employers, and the right advertisement as the main reasons The employers interviewed as part of the research stated that why they decided to join the bootcamp. they want to be in touch with bootcamp providers in order to have an opportunity to advise them on specific skills needs Young Kenyans see bootcamps as a good way of entering on an ongoing basis. Bootcamp providers would benefit from the ICT sector for several reasons. First, as bootcamps are such dialogue because they would receive constant input condensed in a short amount of time, they find them attractive, from the sector while companies would benefit from an especially given that many have gone through four-year improved supply of trained employees. university course already. Second, bootcamps offer practical experience in programming as opposed to some IT masters programs, which are perceived to be mostly theoretical. Third, Bootcamp Experience they feel that bootcamp content is industry-centered, making it dynamic and flexible. Finally, another perceived advantage The coding bootcamp offered by Moringa School not only of bootcamps is that they help graduates in job placement and includes training on coding skills but also builds students’ they create a professional peer network. For those who do not socioemotional skills and prepares them for job search. have a background in ICT but are looking to find employment According to the data collected, ten of the interviewed in the growing sector, a bootcamp can be a good option for students (62.5 percent) valued the bootcamp’s coding skills as rapid upskilling and a good entry point to this sector. the most important skills imparted, while five students (31.25 percent) reported that they most valued the networking and However, students who participated in the bootcamp noted soft-skills training offered, while just one participant valued that it was not possible to combine bootcamps with any both sets of skills equally (see Figure 4.4). other commitment, including working, because of the steep learning curve and intensity of the program. Some had to Technical coding skills: Students felt that the learning curve quit their jobs or drop other commitments in order to fully in the bootcamp was steep. They appreciated the teaching concentrate on the bootcamp. Some students also expected methodology whereby bootcamp instructors did not teach that a lot of previous coding knowledge was required to enter the content itself but introduced it to students to work on it such training, which might hinder other young people from independently, nurturing creative and collaborative problem- applying to such programs. solving abilities. Students also reported that they preferred bootcamps to university programs because the latter focused Perspectives of Employers primarily on theory and outdated coding languages while bootcamps were perceived to afford students more flexibility FGDs and interviews reveal that potential employers have and agility to learn the languages employers were looking different perceptions about the value of bootcamps compared for. On the other hand, some students seemed to agree that to students. Bootcamps were not yet on the radar of most IT a basic foundation in coding was needed to be successful in companies interviewed, and these companies believe that the bootcamp.4 48 Socioemotional skills: Both industry representatives and All students bar one felt confident about their future students stressed the value of communication skills teamwork, employment prospects after completing the bootcamp. as well as aptitude. Students also valued public speaking and Thirteen respondents (81 percent) reported a high level business writing skills, self-discipline, confidence, ability to take of motivation when looking for jobs after completing criticism, and business ethics as important skills they developed the bootcamp, and only two graduates (13 percent) throughout the bootcamp. Students affirmed they were happy to experienced a drop in their motivation level. receive this kind of training during the bootcamp, which they had hitherto not received in their prior education, and they realized its In short, eight (50 percent) of the surveyed students importance in preparing them to join the job market. They were believed that the bootcamp was sufficient to raise their also trained on how to communicate through e-mail, the right competitiveness in the job market, while three students etiquette to use, and how to speak to a group of people (voice (19 percent) felt it was insufficient, and five (31 percent) intonation, eye contact, and so on), all of which they valued and were unsure (see Figure 4.4). This might have to do with thought had made them better professionals. the content of the training as half the surveyed students mentioned that they would change the coding languages Job hunting: An advantage of the bootcamp offered by Moringa offered for future students, but this might also be about School is that it helps graduates in the job placement process, and strengthening job placement support and networking this was seen as beneficial for students. Bootcamps help students since half the students would also change those aspects develop a portfolio to showcase their actual work to potential of the bootcamp.5 employers during the last weeks of training at a careers fair. Figure 4.4: Nairobi bootcamp experience: skills and employment prospects Which of the skills acquired in the How confident do you feel about your future bootcamp do you value the most? employment now that you have completed the training? VERY CODING CONFIDENT SKILLS 10 62% CONFIDENT COFING & SOFT NETWORKING 2 SKILLS 1 13% CONFIDENT 6% SOFT SKILLS 3 19% VERY CONFIDENT CONFIDENT VERY CONFIDENT 49 Tuition Fees When the baseline survey was conducted, students were asked where they aspired to work after completing the Many bootcamp participants and interested students stated bootcamp (see Figure 4.5): five students wanted a job in a that their families had questions and concerns about the cost startup (31.3 percent), three wanted to be employed in a of the bootcamp. Parents found it difficult to understand why large private company (18.8 percent), and three aspired to their son or daughter needed the bootcamp training when work in a multinational corporation (18.8 percent). Of the they should be looking for a job, and they wondered by how 16 graduates, four (25 percent) were working in the type of much they could raise their income given the short duration company in which they aspired to work. of the training. In FGDs held with students, they declared that the cost of the Figure 4.5: Category of current employer of bootcamp was perceived to be too high at the outset, but surveyed bootcamp graduates in Nairobi after joining the training, most of them believed that the value added was worthwhile. By contrast, a few felt that they could get the same training by paying a private instructor. GOVERNMENT/PUBLIC SECTOR However, with the average salary of a junior developer ranging MERCH PRIVATE between $400-500 per month (as per students’ estimations), it COMPANY would take a student over a year to repay the loan with interest LARGE PRIVATE as well as the employment placement support fee if they COMPANY used half their earnings to repay the debt. Potential students LARGE PRIVATE therefore need to have a clear understanding of the costs COMPANY involved. Students were offered financing through Moringa’s partner, KIVA, to enable them to pay the commitment fee and STARTUP obtain financing for the following 24 months.6 UNEMPLOYED AND ACTIVELY LOOKING FOR A JOB Employability 0 1 2 3 4 5 6 While only one student was working before the bootcamp started,7 12 students (75 percent) were employed within six During FGDs, students were asked about their ideal job after months of graduating from the bootcamp. Nine (56.3 percent) the bootcamp. Some reported that they wanted to work reported using the acquired coding skills in their jobs, while at international corporations, especially in the IT sector, two were not satisfied with their jobs because they were not because wages in Kenya were only good if one worked for coding related. The four graduates who were unemployed (25 an international company. Others saw entrepreneurship percent) said they were actively looking for a job where they as an option post-training as it would give them freedom could apply the skills learned in the bootcamp. However, when and help them create employment. Some students voiced analyzing the employability of the participants, the “diploma their aspirations to work as remote developers. A handful of effect” should be taken into account, as nine students (56.3 students were also open about working in IT jobs at nontech percent) were about to finish their formal education programs companies, such as banks or hospitals. right before the bootcamp started so they would have been entering the job market in the short term in any case. This means that employment results cannot be directly attributed to the bootcamp itself, and other factors should be considered. 50 Bootcamps versus University Skills Gap Education Employers stated that there was demand in Kenya for skilled The debate about the value of IT degrees from Kenyan IT personnel, programmers, as well as business development universities versus the value of coding bootcamps was brought and marketing professionals. However, startups, SMEs, and even up in most of the FGDs. There was no consensus between large companies faced challenges in the supply side. Although the participants, with some arguing that the theoretical there were plenty of potential recruits available, their skills did education provided by universities was no longer useful, not match demand. Employers can find people with standard and others reiterating that a university education was still skillsets in terms of software development, but the biggest important. The latter agreed that universities and bootcamps challenge is to bring on board people who are well conversant could complement each other; being in university placed one with a specific technology. Coding bootcamps like Moringa in a better position while the bootcamp emulated the real School tailor their training based on industry demand and work environment. Students also valued the fact that Moringa graduates come into the job market with adequate technical School helps connect them to internship opportunities, which skills and job-ready soft skills. most universities do not do. Employers also reported that when it comes to recruiting There was also no consensus on the employers’ side. Some personnel, there are some tradeoffs between expectations and thought that it was culturally and socially important to have a realities. While it may be quite easy to find a good developer, university degree in Kenya in order to get a job. Others looked finding a developer with good social skills and one who is at what candidates could do and how they performed, not just flexible is not as easy. It would therefore be optimal if there were what their CVs said, so the portfolio developed by students more well-rounded professionals. during the bootcamp was crucial. Others were adamant On the demand side, companies sometimes do not know which that bootcamps could not replace university education technical skills they require, so may hire developers who turn because not everybody was prepared to be a programmer out not to be the right fit for these companies. This highlights an in the IT world. As an example, in a university class of 60 opportunity for bootcamp providers to help better identify and students, only 15- 20 students would be interested and good translate employer needs. at programming, meaning that the remaining 45 students would be good at other aspects such as design or business development; therefore, bootcamps were only useful for those who wanted to be programmers while other students Notes could still benefit from university education. Some employers 1. http://moringaschool.com/. also argued that universities build core fundamentals, which bootcamps then go on to refine. 2. Number of employed persons relative to unemployed and inactive family members. Students perceived that employers sometimes get frustrated 3. Those that participated in the surveys and FGDs, both with university graduates who have been trained heavily in students and interested/prospective candidates. theory, and are instead looking for employees who could do practical work. These students felt that universities taught old 4. After the analyzed cohort finished the bootcamp training, coding languages while employers expected them to know Moringa School created Moringa Prep, a one-month prebootcamp that covers the basics of programming and HTML, JavaScript, Angular, and so on. Thus, the only way for levels the knowledge of students so that they can attend the them to learn newer languages was by attending bootcamps longer coding bootcamp. where they could learn a new language quickly. 5. Note that Moringa School bases its curriculum on real- time demand and constantly updates the coding language taught in the bootcamp. 6. Students first pay a commitment fee to Moringa, and then Kiva directly pays Moringa the remaining tuition fee on behalf of the students. Students then have 24 months to repay Kiva. 7. This student returned to their previous job in nursing after the bootcamp ended. 51 52 MAIN FINDINGS The three interventions in Colombia, the Lebanon and that undergoing a bootcamp program does not improve Kenya provide a measurement of the impact of bootcamps the likelihood of being employed. There is no differential in a developing country context. The main findings are in this outcome based on gender, socioeconomic strata, or summarized below: educational level. A. EMPLOYMENT IMPACT 2. Bootcamps may have an impact on accessing high-quality jobs. The Medellín RCT results show that participating in 1. Bootcamp programs do not seem to have a direct or a bootcamp program could provide higher chances of immediate impact on access to employment. Despite the accessing high-quality IT jobs. This is highly relevant and high rates of employment reported by coding bootcamps warrants further research since these high-quality IT jobs providers in general (60-100 percent), and also the high are connected to what are the future jobs as the economy rates of employment achieved by the analyzed bootcamp moves towards more tech-related activities across sectors, students (73 percent in the RCT in Colombia, 75 percent potentially making coding bootcamps a tool for training (or in Kenya and 82.6 percent in Lebanon in the qualitative retraining) potential employees for this type of job through studies), the RCT evaluation in Medellín (Colombia) showed rapid intervention (that is, four to six month courses). Employment and Educational Impact • Bootcamps may have an impact on accessing high-quality jobs. • And… on providing self-employment tools for low income populations. Employment • But… bootcamp programs do not seem to have a direct or immediate impact on access to employment. Job benefits and • No significant observable effects in terms of job benefits and job satisfaction satisfaction within the term of the experiment (6-9 months after graduation) • Women are underrepresented and have fewer opportunities to find high-quality Gender jobs after bootcamp participation. • Bootcamps seem to support the completion of current educational programs Education • But… bootcamp participation does not seem to lead to enrollment in additional formal education within the short term (6-9 months after graduation) Implementation of Coding Bootcamps • Bootcamps programs can be catalyzed through policy intervention. • But… bootcamp programs are difficult to implement and require links with potential employers. • And… quality of bootcamp provider and type of bootcamp program matters substantially. 53 3. There were no significant observable effects in terms 7. However, bootcamp participation does not seem to of job benefits and job satisfaction. We could not lead to enrollment in additional formal education. determine whether bootcamp programs provide access This suggests that bootcamp participants do see the to jobs with better benefits or job satisfaction, probably need to continue their education to attain their short- because of the short timeframe passed from completion term employment goals, especially when bootcamp of the program and the data collection. Further research participation leads to high-quality jobs. For instance, in will be needed to explore whether jobs accessed following Beirut, most participants (85 percent) placed a premium bootcamp participation result in more formal employment on the ability to acquire coding skills in a compressed with higher satisfaction. time (gaining two years’ worth of experience in only three months). Further research is needed to understand this 4. Bootcamps provide self-employment tools for low- result and whether it is related to the short timeframe of income populations. For the bootcamp students in the the evaluation period. three cities, the entrepreneurship rate increased to 17.4 percent in Medellín and 6.25% in Kenya, but was nonexistent in Lebanon. Although in the RCT no evidence was found that C. BOOTCAMP PROGRAMS bootcamps had an effect on business (that is,startup) creation 8. Bootcamp programs are difficult to implement and in general, it did seem to have a positive effect on the low- require links with potential employers. Implementing a income population. This suggests that bootcamps may bootcamp program is not easy. Providers are still maturing be leveraged to provide self-employment, leveraging and many of them still operate as startups and are learning tech opportunities for those segments of the population how to best to implement their programs. Experience from that may face structural barriers to employment in Medellín, Beirut, and Nairobi, shows that the link to local developing countries. Further research would be needed employers is critical in developing the right tech-skills to understand the full potential of this possibility and if large curriculum and high-quality employment for participants. extrapolations are possible. 9. Not all bootcamps are the same and quality among 5. There are clear signs that gender is a determinant of them matters. Bootcamps differ in terms of the quality good quality tech-related employment. Women are of implementation and this really matters. The bootcamp not only generally underrepresented in the tech industry program implemented in Medellín showed variation and participation in bootcamp programs is low (about in quality of implementation among different UVA 20 to 30 percent, World Bank (2017)), but they also have locations. Also, there was limited provision of training in fewer opportunities to find a high-quality job following socioemotional skills, limiting the potential impact of this bootcamp participation. This seems to suggest that there program. In contrast, bootcamp programs in Beirut and is a need for more female-centered interventions to Nairobi emphasized socioemotional skills and preparation attract women to the tech sector in general and these of participants for job hunting and future working kinds of training programs in particular. environments. B. EDUCATIONAL IMPACT 10. Bootcamp programs can be catalyzed through policy intervention (Mulas and others 2017). The experience 6. Bootcamps seem to support the completion of current in Medellín shows that bootcamp programs can be educational programs. From the Medellín RCT (although catalyzed through government-led policies. The leading with certain sample limitations), it was observed that role of Ruta N in the tech ecosystem of the city and the bootcamp participants were more likely to complete connection with potential employees and labor demand formal tertiary educational programs in which they were played a crucial role in the success of the implementation already enrolled. This suggests that bootcamps are a of the bootcamp program in the city. Appendix D provides complement to tertiary education, potentially showing a guide for implementing bootcamp programs in a need to incorporate some of its methodologies in developing countries to inform public policy interventions existing tertiary educational programs. Beirut’s findings and bootcamp providers. support this interpretation, as most participants saw the skills acquired in bootcamps as a bridge between academic education and practical employment. 54 LESSONS FOR FUTURE IMPACT EVALUATIONS Previous chapters highlight the different challenges and lessons from the current evaluations. Table 6.1 summarizes the challenges and expands on the valuable lessons that can be applied to future interventions. Most points make reference to the RCT in Medellín. Table 6.1: Lessons for future impact evaluations Item Best practice/ideal Potential or actual issues Lessons learned 1) Study design Location Locations have Two locations (Beirut and Nairobi) Outside providers are a feasible conditions for did not have mature providers, and in solution but close implementation a successful Medellín an external supplier had to be supervision and quality control is still implementation, such as brought in required high capacity and ability to achieve the desired sample size Sample size Required sample sizes Difficulty reaching the desired number Early engagement with supplier are achievable of participants. to incorporate best practices and marketing efforts to reach required sample size. Potential cost impacts (see “Integrity of See “Integrity of treatment and treatment and control groups” below) control groups” below Unrealistically high expectations about Align participants’ expectations via required coding experience marketing materials and interviews 2) Integrity of treatment and control groups Baseline data Full baseline data Missing values from respondents Early engagement in data collection collection collection for initial because of changes in addresses analysis, collection and consistency randomization and telephone numbers, and survey via survey company or in-the-field fatigue, resulting in need to hire person additional survey company later on Completeness for Refusal from applicants to report When available, use of proxies such relevant survey fields income as geographical socioeconomic stratification 55 Attrition Low attrition and Low participation and selection bias Allow enough time for candidates to crossovers because of lack of time for decision decide their participation and make to participate (e.g. the initial group of arrangements. Standard company accepted participants were given 10 practice when resigning is two weeks’ days to confirm their participation in notice and an extra week should be the bootcamp) allowed to make arrangements Potential cost impact for participation Clear cost expectations should be provided early on Charge applicants in advance, with non-refundable fees in case they are assigned to the treatment group Lower-than-expected enrollment A case could arise where preannounced incentives for survey participation for those in the control group attracts applicants. This was not so for the cases in this study but it is important to keep in mind for future RCTs Dropping out because of location Early selection and evaluation safety concerns and distance of location (safety, distance from participants and connectivity) Unrealistically low expectations about Ensure applicants are fully aware of level of effort needed for program program requirements via marketing completion (e.g., some students had and interviews to quit jobs or other activities to fully concentrate on the bootcamp Endline data High baseline data Survey fatigue: parallel data-gathering Discourage additional data collection collection collection took place as well as a midline survey efforts, monitor implementation and in Nairobi and Beirut eliminate midline surveys. Difficulty to reach out participants Ensure participants are aware of (phone lines disconnected after follow-up surveys and provide participation). sufficient means to be reached 3) Intervention Consistency Intervention quality Higher-than-expected instructors’ Related to previous points, monitor remains consistent rotation (intra and within centers) implementation and require across time and assurances such as instructors’ participants certification and contracts (e.g., interviews for instructors) 56 4) Impact and results reporting Comparability Results allow for Short-term results (six-months) may A shorter-term horizon with its comparison with similar underestimate the longer-term impact respective comparable instruments alternative interventions of the intervention (e.g., other training) should be sought. It is important to establish that the scope of similar studies can only assess short-term effects 5) Scalability of intervention Scalability The chosen intervention Low participation numbers sent mixed Choose location with local industry potential has the potential to be signals about potential demand and demand for technology talent scaled-up scalability Ensure providers are not niche players and have business models that allow for replication and scaling up Cost-benefit Costs and benefits are There is partial information on the Need to establish benefit analysis clearly measured and benefits side measurement and adequate presented. For scalability, evaluation horizons costs are lower than benefits It is important to note that, in addition to the considerations an effect of a certain size from a random occurrence. A study mentioned above, practitioners should first evaluate whether might easily detect a large effect from an intervention but the conditions for an RCT are present before undertaking detecting a small one is much less likely. Thus, it is important to such task. have a clear idea of the magnitude of the desired effect to be detected: the smaller the variation that needs to be detected, Common problems relate to external validity (generalization the higher the power needed and consequently the larger the beyond the evaluation sample), internal validity (degree required sample size. of comparability between treatment and control groups) and implementation issues (for example, when it is not If smaller sample sizes are present, statistical power will be practical, easy, or ethical to restrict treatment access to some diminished and only large effects will be able to be detected. participants). Within this context, one of the main factors for a Thus, it may still be possible to conduct an RCT but, given a successful randomized control trial is sample size. In the cases restricted sample size, it may be necessary to reduce the presented in this report, reaching a large sample size was an power of a test or to entertain just being able to detect large ongoing implementation concern. Among other reasons, if impacts. By the same token, if the sample is small, caveats the sample is too small, statistical tests may not be able to about limited or niche extrapolation to more general samples detect treatment impacts, affecting the validity of the results. should be considered and made explicit. As guidance, sample sizes are related to a desired statistical Similarly, alternative designs such as randomized promotion power, which is typically set at 0.80 for most studies. Simply and qualitative focus groups are to be considered when RCTs put, the power of a study is the likelihood that it will distinguish are not feasible. 57 BIBLIOGRAPHY Central Administration of Statistics (CAS) and the Medellín Cómo Vamos. 2016. “Informe de Calidad de Vida World Bank. 2015. Measuring Poverty in Lebanon Using 2011 de Medellín.” Medellín: Medellín Cómo Vamos. HSB, Technical Report, December 8. http://www.cas.gov. lb/images/Excel/Poverty/Measuring percent20poverty Meng, V. 2013. “The Ultimate Guide to Coding Bootcamps: percent20in percent20Lebanon percent20using The Most Selective Bootcamps.” SkilledUp. http://archive.fo/ percent202011 percent20HBS_technical percent20report. YCwEA#selection-145.0-145.68. pdf. Mulas, V., M. Minges, and E. Allende. 2016. “Tech Start-up Clavijo, S., A. Vera, E. Cuellar, and A. Rios. 2015. “Costos no Ecosystem: The Case of Lebanon.” In A Geo-Economy of Risks Salariales en Colombia Pos-Ley 1607 de 2012” , Asociación and Reward, edited by W. Harake and others. Washington, Nacional de Instituciones Financieras. Bogotá. DC: World Bank. http://documents.worldbank.org/curated/ en/652591468179100109/A-geo-economy-of-risks-and- El empleo. 2016. “Informe de Tendencias Laborales reward. Segundo Trimestre 2016.” http://contenido.elempleo.com/ informes/Informe_elempleo-segundo_trimestre_2016.pdf. Mulas, V., C. Paradi-Guilford, E. Allende Letona, and Z.V. Dalphond. 2017. Coding Bootcamps: Building Future-Proof Skills Government of the Republic of Kenya. 2007. Kenya through Rapid Skills Training. Washington, DC: World Bank. Vision 2030: The Popular Version. http://www.vision2030. https://openknowledge.worldbank.org/handle/10986/28218. go.ke/531/vision-2030-team-maintains-course-deliver- economic-goals/. Ruta N Medellín and others. 2015. “Investigación de Mercado Laboral en el Sector de las Tecnologías de la International Monetary Fund. 2015. IMF Country Información (TI) Medellín.” Report: Colombia. Report No. 15/143. https://www.imf.org/ external/pubs/ft/scr/2015/cr15143.pdf. UN Habitat. 2014. Background Paper. https://unhabitat.org/ wp-content/uploads/2014/07/WHD-2014-Background-Paper.pdf. ITU. 2016. Coding Bootcamps: A Strategy for Youth Employment. International Telecommunications Union. World Bank and Endeavour Insight. 2015. “The http://www.itu.int/en/ITU-D/Digital-Inclusion/Youth-and- Colombian Tech Ecosystem: A Study of Connections Among Children/Documents/CodingBootcamps_E.pdf. Entrepreneurs with Recommendations for Growth.” Kenya Integrated Household Budget Survey (KIHBS) World Economic Forum. 2016. The Future of Jobs: (2015-2016): “District Poverty Data 2015-2016. Kenya Open Employment, Skills and Workforce Strategy for the Fourth Data.” https://www.knbs.or.ke/2015-16-kenya-integrated- Industrial Revolution. Global Challenge Insight Report, household-budget-survey-kihbs-progress-report- accessed at: http://www3.weforum.org/docs/WEF_Future_of_ october-2015. Jobs.pdf. 58 APPENDIX A: RESEARCH DESIGN: The simple framework for our experimental design was thus: RANDOMIZED CONTROLLED Pretreatment Posttreatment TRIAL IN COLOMBIA Treatment X0 X1 Control Y0 Y1 Research Question The treatment effect, (X1-X0)-(Y1-Y0) represents the causal effect of the intervention (X and Y are sample averages). In We are interested in testing the efficacy of coding other words, the difference between job and wage metrics bootcamps on labor market outcomes in developing for the treatment group after treatment were compared with countries around the world. the difference in job and wage metrics for the control group This research project addressed whether coding bootcamps over the same period. Randomization of the individuals impact the employment and employability of youth in participating in the field experiment ensured that the developing countries, and if so, to what extent. measured effect could be attributed to bootcamp training instead of other correlated variables that plague inference in naturally occurring data (randomization ensured that X0=Y0). Experimental Design and Model Pretreatment levels were measured by conducting an initial In order to determine the effects of technology training assessment of participating individuals for both the treatment programs, a group of eligible students were randomly and control. Observing pretreatment levels was important given assigned to bootcamps. In other words, for a sample that there were limitations to our sample size. Pretreatment data of participants, some were randomly selected to receive was collected through entry surveys (see Appendix C). bootcamp training (the treatment group) and the rest were Key to the experimental design was the ability to observe assigned to the control group. Y1. This required the collection of information from To achieve this, in qualifying pilot location(s) a group individuals that did not participate in the bootcamp, and of individuals were first identified who were willing to hence participants were recruited based on their willingness participate in an experiment tracking job and wage to participate in a study on job and wage outcomes more outcomes. The individuals were willing to participate generally, rather than on the promise of receiving bootcamp irrespective of whether or not they received bootcamp training, training. To ensure participation of the control group over the and were also willing to complete the bootcamp training at long run, a simple incentive scheme was proposed, but the local the discounted price if randomized into the treatment group. government advised against it. The experiment was set up and presented to potential participants as a nine-month study on The participant group had to meet minimum requirements income and employment. Thus, incentives were not offered (speak basic English, read, write, have basic computer for completing an intake survey, but those who completed the skills, be between 18 and 28 years of age and live in the follow-up survey were entered into a raffle to win an iPad. By Medellín area) to ensure a sample that was conducive to following this structure, all enrolled participants were agreeing our experiment in order to limit attrition and contagion to participate in the research over the six months after the end effects. In other words, an initial sample of individuals was of the training either in return for participation in a bootcamp identified who showed interest in participating in the bootcamp or receiving valuable incentives. This was done to ensure that training. They had to be motivated and less likely to drop out data was obtained from individuals regardless of whether they if they receive a discounted fee to enroll in the bootcamp.1 participated in a bootcamp, while also achieving randomization. Choosing participants based on their willingness to complete the bootcamp was indicative of our target population for future Based on anticipated sample size requirements (see the intervention – we were interested in the effects of bootcamps section below; in Colombia, the requirements were 120 on those that would voluntarily participate in a bootcamp and people in the treatment group and 120 people in the not necessarily the whole population.2 control group, including attrition estimates), the research 59 design was only implementable in Colombia. Other locations, Power tests for a dichotomous outcome variable (employed Lebanon and Kenya, were not conducive to experimental versus not employed) were conducted. The results of the power design when this activity started, because of the infeasibility test supported the following conclusion: if 120-150 participants of randomization. As a consequence, qualitative studies were could be recruited into both the treatment and control groups, particularly important for these locations and helped to provide an effect that represented a 20-35 percent improvement from context to the experimental results in Colombia. the baseline rate was likely to be detected. The relatively large minimum detectable effect also took into account potential Sample crossover (control group taking coding bootcamp classes through other sources) of approximately 15 percent. Participating Individuals In order to minimize attrition, a pool of participants was Results for other variables in which the sample variances identified that had a minimum acceptable level of motivation. were relatively small were more optimistic. In either case, the At the same time, it was a concern that motivated individuals power tests confirmed three important considerations for: 1) at might then choose to attend a different bootcamp if not selected current sample size estimates, the model required the rapid to receive the bootcamp through this study. Control participants skills training to produce a large treatment effect. Thus, the were specifically asked in the follow-on surveys whether they more participants that could be recruited, the greater chance took a bootcamp or classes, and this was taken into account in our of detecting an effect because the minimum detectable effect models. In addition to incentives for partaking in the surveys, one would become smaller; 2) unequal sample sizes between possibility to account for this with the control group participants treatment and control groups could be considered to account was to offer the possibility to receive the training at a lower cost for differing sample costs. Increasing the size of the control once the experiment was over (in about one year’s time). This group, without increasing the size of the treatment group could would reduce the incentive for members in the control group provide more power and minimize the problem of losing control to seek another bootcamp in the interim (one not requiring any units in different rounds of data collection. This was especially screening after the final survey) while still providing a treated and important since bootcamps were capped in size; 3) alternative a control group during the tracking period. However, this could outcome variables could be considered. Unfortunately, without have had the unfortunate effect of deterring the incentives of some pre-existing or pilot data, it was extremely difficult to the control group to seek job opportunities while they waited to speculate on how promising this approach could be. participate in the bootcamp. This effect would have biased our results by invalidating the control group. Given these competing Initial estimates of samples sizes in Colombia suggested effects, it was decided not to offer the control group a delayed potentially recruiting a sample of 220-240. This number bootcamp and instead deal with the possibility of the control of participants, given the discussion above, would give the group taking bootcamps elsewhere by quantifying the effect. opportunity to detect a relatively large treatment effect. In Although such possibility could have limited the study, it was the event that the treatment effect was small, other statistical unlikely to influence it, given the fact that the predominant methods may have been appropriate. For example, it may majority of Medellín’s youth would not be able to afford the have been found that a treatment effect was significant at the training, and other bootcamp providers were unlikely to step in 10 percent level. Though this effect was not large enough for for the same reason. conventional standards to attribute the effect as nonrandom, Bayesian methodologies, which incorporate priors, would To better understand sample size considerations, several still consider the information in the study useful. If needed, power calculation tests were conducted. The goal of a power it is possible to explore this approach as well (see Floyd and test is to identify sample sizes required to detect a prespecified List 2016). For outcomes with relatively high autocorrelation, treatment effect (also called minimum detectable effect) at an ANCOVA specification could be used to increase power specified levels of power and statistical significance. In our (McKenzie 2012). case, as is consistent with common practice, samples sizes were considered for a specified power of 0.8 and statistical Importance of Randomization significance of 0.05. The key element of the sample design was that the individuals were randomized into a treatment and a control The power calculations also require estimates of variances for group. This element of the design is what allows us to ensure treated and controlled samples. Often, these are obtained from that the education program, and no other confounding factors, pre-existing data or are acquired from pilot studies. Since neither explains the difference in outcomes. Indeed, qualitative results of these data sources was available, additional assumptions were in Kenya and Lebanon are much more interesting having made on reasonable estimates for these values.3 conducted a successful RCT in Colombia. 60 To illustrate this, first consider the case in which there is To check whether short and medium run effects of the no control group. In this design, a simple comparison is program were distinct, a fully interactive model was run: being made between a group of individuals before and after the treatment. The major risk is that there are omitted Yit=α+β1 Treat+β2 End+β3 TreatEnd+∑γjControlsit+εit (2) variables that change over time that could impact the where End is a dummy for the endline survey. The parameters outcome variable. For example, imagine we were interested of interest are β3 and β5. In case of imperfect compliance, the in measuring the impact of training on salary levels. To do parameters of interest in (1) and (2) informed the Intent-to- so, salaries one year before the bootcamp and one year after Treat (ITT) effect, that is, the impact of the training on those the bootcamp could be compared. an increase in salaries was randomly assigned to bootcamp regardless actual take-up. found, this could be because of the effect of the bootcamp or any other macroeconomic factors that changed salaries over Identification Concerns the time period. A control group would be needed to control for these time effects. There were two major concerns with this analysis that were considered (and reiterated). The first was that the Second, consider the case in which a control group was treatment effect might have been driven by a placebo. In incorporated, but the control group was not random. other words, if it was the case that those selected by the For example, suppose that the treatment group was chosen lottery “felt better” about themselves, then an improvement based on how motivated they were to complete the in outcomes might be because of a placebo effect and not bootcamp. If the outcomes of these groups were compared, the training per se. This could also be true if the bootcamp then inference would be confounded by the fact that the two provided a certification mechanism. The second major groups were fundamentally different: one group was highly concern was whether those that were randomized into the motivated and the other was not. If this selection effect was control group were more likely to seek out other bootcamps correlated with the outcome variable, then the impact of the after the completion of the program. If the pool of participants bootcamp could not be recovered. Continuing with salaries utilized were those that were actively seeking a bootcamp, in our example, the experimental design would be equally then it could be that individuals would make sure they attend likely to capture that highly motivated individuals make a bootcamp regardless of whether it was acquired through more money rather than an effect of being in the bootcamp. our experiment. Then the control group would start to look more like the treatment group over time. If all control group Analysis individuals eventually received bootcamp training through a Statistical Model different source, then our treatment would only capture the outcome differences between our bootcamp and a bootcamp The experiment was designed in anticipation of analysis provided by another source. using the following empirical specification: Yit=α+β1 Treat+β2 Post+β3 TreatPost+∑γj Controlsit+εit(1) Notes where Yit is the outcome of individual i in time t,Treat is a dummy 1. In practice, the minimum requirements to participate in a variable that is equal to 1 if the individual is randomly assigned bootcamp are likely be very low to encourage maximum to participate in the bootcamp and zero otherwise, Post is a participation. Nevertheless, participants that do not meet time dummy that takes the value of 1 at the endline and zero at basic requirements were excluded from the randomization sample. the baseline. Equation (1) is a simple difference-in-differences design that captures the effect of the intervention on outcomes 2. This potentially induces a Treatment Specific Selection by comparing the treated group relative to the control group Bias (Al-Ubaydli and List 2013) and therefore limits the around the implementation date4. The parameter of interest, generalizability of our experiment but helps to ensure internal validity was achieved. β3 , is estimated by comparing the change in outcomes for individuals that received the boot camp over time relative to 3. One way of circumventing this problem is to work with the control sample. Pooling the three waves of data collection standardized variable. In that case the standard deviation and estimating one single treatment effect maximized power will be 1. and hence the chance to detect a statistically significant result. 4. Other empirical methodologies, such ANCOVA may be Seeking to improve precision, a vector of control variables was appropriate to use as well. Specifically, ANCOVA may afford added in the specification. more power if the autocorrelation in the variables of interest is low. 61 APPENDIX B: RESEARCH DESIGN: • What causes people, and young people in particular, to join a bootcamp? QUALITATIVE STUDY • What are the implications of bootcamp training on the technological sectors and local economies? Purpose of the Study The main purpose of the qualitative study was to Methodological Framework gain deeper understanding of the effects that coding bootcamps have on participants in terms of influencing The current research developed theories of the their quality of life, including employment patterns and determinants of positive outcomes of bootcamp training; salary levels. The qualitative case studies were used to an observation-focused qualitative case study design was compare pre-existing attitudes with potential postbootcamp implemented in order to explain the exact mechanisms attitude changes among research participants to help that improve post-training quality of life outcomes for determine who is most likely to succeed as a result of attending some students over others. bootcamp training. As a secondary purpose, the interviews Based on motives, attitudes, and beliefs of training and focus groups also provided insight into the effects of participants and their association with bootcamps, bootcamps on the technological sectors and local economies. this research identified both the candidate causes for The study aimed to bring to the attention of governments participating in a bootcamp and the effects of participating and wider public sector stakeholders in the developing as modulated by dispositional characteristics. These countries, as well as related donor communities, the causes and effects were framed in general terms into theories potential positive social impact of bootcamps on local so that they could be tested against other evidence in order communities and related labor market outcomes and to eventually create a generalizable model to explain why youth employment. In addition, the study aimed to help individuals in countries with similar economic and cultural private sector stakeholders to envision the growth potential contexts decide to participate in bootcamps and allow for of the related industries and local technology ecosystem. predictions about which “types” of participants are most likely to benefit from that participation. Research Questions The choice of a case study research methodology is primarily justified by its potential for analytical The emphasis of the study was on the experiences generalization (linking specific findings from a case to a (emotional, behavioral, and educational adjustments) more generalized theory) and providing an opportunity of bootcamp participants as a result of their exposure to understand mechanisms rather than metrics (in other to training and post-training employment patterns. The words, answering questions about what is done, how, study helps explain why in some cases the participation in and why compared to is this done, how much/many and bootcamps becomes a career promoter and a life-changing how often?). The case study methodology provides one of experience, while in other cases it is not. Within this context, the best approaches to “a first-hand understanding of people the study addressed the following question and subquestions: and events” (Yin 2004, 3) and “actual, real-life cases” (Yin 2004, 7). Importantly, this methodology does not require a large • How does bootcamp training impact students’ ability to number of cases, and theories can be constructed from one to compete in the local market? a few. Two holistic case studies were selected for this research, namely the pilot bootcamps in Kenya (the Moringa School) • Which mechanisms explain why specific attitudinal and Lebanon (Le Wagon Beirut). characteristics toward training (causes) result in better post-training employment chances and higher salary levels The Delphi method and congruence procedures were (effects) for certain participants? used to infer theories from the cases, which involved a two- stage process: first, a group of respondents was given the 62 opportunity to provide forecasts, then a researcher synthesized preferably with the highest demand for junior developers. The and summarized those forecasts and had the group respond local research partners in each case study helped to get access to the same kinds of questions in light of the new collective to these lead companies and also helped collect objective and knowledge uncovered by the first round. The Delphi method measurable demographic comparison data (for example, age, mined the views of case participants for drawing hypotheses gender, salary level). from a triangulated data collection technique that cannot be made from observations alone. The congruence procedures looked for within case correlations between the study variable Data Collection and other phenomena, such as possible independent variables A triangulation approach was used to gather data in the new hypothesis (causes of effects). from multiple sources: three longitudinal surveys, semistructured interviews with industry leaders and Sampling experts, and two focus groups (pre- and post-training) that combined bootcamp participants, industry leaders, Kenya and Lebanon pilot locations were selected as case and other parties with vested interest in bootcamps. This studies for this research based on local, but also regional, raw data helped develop hypotheses and theories that would opportunities provided by these locations and the level explain social processes related to bootcamps based on of maturity of their bootcamp programs. In addition, the elements of experience of research participants. World Bank research team had close cooperation with local tech ecosystems in Kenya and Lebanon and identified suitable Surveys local research partners to assist with the bootcamps study. Three standardized longitudinal questionnaire surveys Within each case study, a theoretical (purposeful) sampling were introduced to the bootcamp participants in order to approach was used for elaboration and refining variables obtain data essential for the study but not available from for hypotheses and eventual theory development. In terms existing records (see Appendix C for the baseline survey of the number of participants, it meant that the sample size and sample questions to be included in the follow-up was determined during the research process depending surveys). The surveys were administered before the training on which additional data was needed and the diminishing for treatment group starts, after the completion of the training returns of additional information (saturation). In other words, (subject to individual bootcamp’s length), and six months participants were added to the sample until selected variables after the completion of the training. Survey administration were saturated and no new data relative to the variable was was conducted in person or through e-mails or follow-up discovered. In the end this was not needed. In both Kenya and telephone calls. The survey participants were offered small Lebanon, each cohort comprised 18 students. monetary incentives to complete the questionnaire in full and on time. The surveys measured demographic variables, as well Bootcamp participants in each case study were recruited as attitudes, beliefs, and behavior related to bootcamps and to the bootcamps following the training advertisement. employment outcomes and income. The surveys also provided They had to pass minimum eligibility criteria developed by data comparable across all three pilot locations of the World the bootcamp providers in agreement with local research Bank bootcamps study, which included quantitative research partners (iHub in Kenya and Berytech in Lebanon) and the based on randomized control trials in Medellín, Colombia, World Bank research team. These criteria included basic and two qualitative case studies in Kenya and Lebanon. The logic, mathematics, and cognitive skills, as well as minimum administration of surveys was provided by the research computer skills. partners, iHub in Kenya and Berytech in Lebanon. Case studies did not assume the use of control group, Semistructured Interviews bearing in mind that case study researchers should avoid controlling any real-life events (Yin 2004, 8). For Semistructured in-depth interviews were used to allow the same reason, any manipulation of treatment group was exploration of “the subjective values, beliefs and thoughts not acceptable. To help analyze the findings from each case of the individual respondent” (Valentine 1997, 112) on the study and help identify opportunity cost for the participants demand side. For this purpose, two or more industry leaders of the bootcamp training, a modest amount of comparative and, potentially, public or academic experts in the subject data was needed. This comparative data was identified from area, were interviewed to obtain the primary data about information provided by human resources departments demand size and type, as well as key features of successful of selected industry leaders (two to three local companies, entry-level employees. The local research partners helped 63 identify these industry leaders and facilitated access to them. The length of the focus groups sessions varied from one to The sample semistructured interview questions are shown in two hours, with coffee breaks if needed. The local research Appendix C. Some general open-ended questions were also partners helped identify locations and other logistics for included to re-engage interviewees if the interviewer noticed the focus group sessions. The local research partners also that respondents were exhibiting boredom or annoyance advertised the focus groups and recruited the participants with the lengthy interviews (ideally, not less than one-hour among bootcamp students; that is, those who were long). Interviewers was advised to use “sensitizing concepts,” interested but never participated in bootcamp trainings, that is, intuitive concepts that are built on the participant’s industry lead representatives, academic representatives, own existing knowledge and interest in the study area, to donor community representatives, government, and civil frame questions that would serve as “ice breakers” during society representatives (depending on local context and interviews (Charmaz 2006, 16-17). Permission was obtained culture). from interviewees to audio-record each session. The local research partners helped the World Bank Focus groups research team to identify an experienced focus group facilitator and a skilled rapporteur. The facilitator was in Focus groups provide access to the perspectives of a constant dialogue with at least one researcher from the World greater number of participants compared with interviews. Bank research team. The rapporteur transcribed the focus For the purpose of this study, focus groups were formed taking group discussions verbatim based on video recordings. into account both the needs of the World Bank research team and the interests of the participants. The aim was to bring Most of the focus group participants were provided together a group of people who have a shared understanding monetary incentives for taking part in the focus groups. of the purpose of bootcamps and were comfortable talking The incentives were provided by the World Bank and to each other. As such, the study included two focus group administered by the local research partners. sessions in each location. The initial focus group sessions were conducted immediately after the start of the training. Informed consent was also received from participants The follow-up focus groups were held six months after the reflecting their agreement to video recording. Participants completion of the training. The format and composition of were reassured that recordings would not be shared with the focus groups was determined based on cultural issues. third parties and that their names and affiliations would not be mentioned in the output reports and publications. Rather, The focus groups consisted of a series of discussions of data gathered through the focus groups would be presented different sample sizes and composition throughout a day. in aggregated format. For example, the morning and afternoon sessions included smaller groups of discussants: the morning session comprised The content of questions also met the needs of both the 10 bootcamp students and the afternoon session comprised World Bank research team and the research participants. 10 people (“non-students”) who expressed interest but never To encourage a livelier discussion dynamic that would lead participated in bootcamps for various reasons (for example, to active conversation on the study topic, initial questions high tuition rate, no time for full-time study), including one were oriented toward participants’ interests. The sample list or two applicants with a similar level of intention but who of questions for facilitated focus group discussions can be failed the prescreening. The evening session incorporated a viewed in Appendix C. maximum of 15 people from a combination of students (one- third), nonstudents (one-third), and a group of experts (one- third) comprising: one representative of the industry; one Data Management representative of the donor community; one representative Detailed transcripts followed each interview and focus from an academic institution; one representative of the group meeting. Once video recordings were transcribed, government; and one representative from civil society the World Bank research team used NVivo, a Computer Aided (depending on local context and culture). In the final Qualitative Data Analysis Software (CAQDAS) package as a raw focus group discussions, the third session was formed by data management system and as a tool to validate inferences representatives of companies that had hired a bootcamp about the data during analysis. NVivo was combined with graduate. At least one World Bank researcher played a role as manual data management and analysis techniques in order to both a participant in all focus group discussions to reply to manually upload meeting memos in NVivo and link them with other participants’ questions and as a cofacilitator that could the transcripts from data collection. pose additional questions to the participants. 64 Bibliography Charmaz, Kathy. 2006. Constructing Grounded Theory. A Practical Guide Through Qualitative Analysis. London: Sage. George, Alexander and Andrew Bennett. 2005. Case Studies and Theory Development in the Social Sciences. Cambridge, MA and London: MIT Press. King, Gary, Robert Keohane and Sidney Verba. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press. Valentine, Gill. 1997. “Tell Me About …: Using Interviews as a Research Methodology.” In Methods in Human Geography: A Guide for Students Doing Research Projects, edited by Robin Flowerdew and David Martin, 110-126. Harlow: Longman. Van Evera, Stephen. 1997. Guide to Methods for Students of Political Science. Ithaca and London: Cornel University Press. Yin, Robert K. 2004. “Case Study Methods.” Complementary Methods for Research in Education. American Educational Research Association. Washington, DC. Yin, Robert K. 2014. Case Study Research: Design and Methods. 5th Edition. Thousand Oaks, CA: Sage. 65 APPENDIX C: INSTRUMENTAL VARIABLE REGRESSION TABLES Instrumenting with Assignment The tables below estimate the treatment on the treated instrumenting completion by treatment to account for lack of completion in the treatment group. As stated above, the impact results do not change. Abridged tables are shown to display the relevant impact coefficient but all full regressions include the following variables: age, gender, strata, location type (rural or urban), mother’s high-school completion, own completion of high school, tertiary education and previous experience. Table E.1. Instrumental variable regressions with program completion only   (1) (2) (3) (4) Variables Job Status (being employed) Job Benefits Job Satisfaction Business Creation Instrumented completion -0.0389 -0.0109 0.0818 0.0608 (0.0676) (0.0648) (0.0662) (0.0541) Observations 239 279 279 239 R-squared 0.052 0.054 0.040 0.002 Note: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. Table E.2. Instrumental variable regressions with program completion and interaction between treatment and strata   (1) (2) (3) (4) Variables Job Status (being Job Benefits Job Satisfaction Business Creation employed) Instrumented -0.0364 -0.0437 0.0793 0.115* completion (0.0867) (0.0824) (0.0843) (0.0695) Treatment*High Strata -0.00591 0.0766 0.00584 -0.130 (0.126) (0.116) (0.118) (0.101) Observations 239 279 279 239 R-squared 0.052 0.057 0.040   Note: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. 66 Table E.3. Instrumental variable regressions with program completion and interaction among treatment, strata and gender   (1) (2) (3) (4) Job Status (being Variables employed) Job Benefits Job Satisfaction Business Creation Instrumented completion -0.0667 -0.0488 0.0806 0.0651 (0.0710) (0.0681) (0.0700) (0.0570) Treatment*Female*Strata 0.206 0.298* 0.00949 -0.0317 (0.165) (0.161) (0.165) (0.132) Observations 239 279 279 239 R-squared 0.060 0.067 0.040 0.002 Note: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. Table E.4. Instrumental variable regressions with program completion and interaction among treatment, and tertiary education   (1) (2) (3) (4) Variables Job Status (being employed) Job Benefits Job Satisfaction Business Creation Instrumented -0.0784 -0.0126 0.0656 -0.0136 completion (0.126) (0.118) (0.121) (0.0998) Treatment*Tertiary 0.0492 0.00216 0.0202 0.0927 Education (0.131) (0.121) (0.124) (0.104) Observations 239 279 279 239 R-squared 0.055 0.054 0.041 0.019 Note: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. 67 APPENDIX D: SURVEYS AND QUESTIONNAIRES Baseline Survey Administered to the bootcamp students in Medellín, Beirut and Nairobi, before the beginning of the training. In the case of Medellín, it was administered to the 903 people who applied to the bootcamp (includes treatment and control groups, and those that did not meet the minimum eligibility criteria). A Bootcamp Preferences A1 What is your preference for in-person Bootcamp hours? (Times may slightly vary): Morning: 8:00 am to 12:00 pm (2 groups) Afternoon: 2:00 pm to 6:00 pm (2 groups) Night: 5:00 pm to 9:00 pm (2 groups) A2 What is your motivation to do the Bootcamp? B Personal, Family, and Household Information B1 Full name: B2 Birth date (day/month/year): B3 Sex: Male (1) Female (2) B4 Address B5 Strata B6 (Mobile and/or home) phone number: B7 Email: 68 B8 Describe your current place of residence: Metropolitan (capital city) area (1) Large city (2) Small town (3) Rural area (4) B9 Do you live and/or work in the same area where you grew up? Yes (1) GO TO A11 No (2) GO TO A9 B10 Describe your original place of residence (where you grew up): Metropolitan (capital city) area (1) Large city (2) Small town (3) Rural area (4) Another country (5) Name of the country:____________ B11 What was the main reason for moving to your current residence? To accompany family (1) For education/training/apprenticeship (2) To work/for employment-related reasons (3) Other reasons (99) Specify reason:_____________ B12 What is your current marital status? Never married (1) GO TO A13 Engaged to be married (2) GO TO A13 Married (3) GO TO A12 Separated/divorced (4) GO TO A13 Widowed (5) GO TO A13 B13 What does your spouse currently do (choose the most relevant activity)? Attending education/training (1) Works for salary/wage with an employer (2) Self-employed (3) Unemployed & actively looking for job (4) Does not work (5) Engaged in home duties (including childcare) (6) Unable to work due to sickness or disability (7) Other (99) Specify activity/reason:________________ B14 Do you have any children? Yes (1) GO TO A14 No (2) GO TO A15 B15 How many children do you have? 69 B16 What is the highest level of education obtained by your mother and/or father (circle what applies for each parent)? Father (A) Mother (B) No schooling/Pre-primary education 0 0 Primary education 1 1 Lower secondary education 2 2 Upper secondary education 3 3 Post-secondary non tertiary education 4 4 First stage of tertiary education 5 5 Second stage of tertiary education 6 6 Other 7 7 B17 What are the occupations of your father and/or mother (select main occupation of each parent)? Father (A) Mother (B) Professional, technical, and related worker 1 1 Administrative, clerical, or managerial worker 2 2 Clerical worker 3 3 Agricultural worker 4 4 Sales worker 5 5 Government/public sector worker 6 6 Factory/production worker 7 7 Armed forces 8 8 House-based/subcontractor worker 9 9 Other service worker 10 10 Unpaid family worker 11 11 Student 12 12 Unemployed/looking for job 13 13 Retired 14 14 Disabled 15 15 Parent deceased 16 16 Other 17 17 B18 What is the number of persons in your household (including yourself)? B19 On average, what is the total income of your household? Per Year: _______ Per Month: _______ B20 How many persons (including children?) in the household work for a salary/wage? B21 How many persons are in the household who are without work and actively look for work? 70 C Education C1 What is your highest level of education? Father (A) Mother (B) No schooling/Pre-primary education 0 0 Primary education 1 1 Lower secondary education 2 2 Upper secondary education 3 3 Post-secondary non tertiary education 4 4 First stage of tertiary education 5 5 Second stage of tertiary education 6 6 Other 7 7 C2 If you attended University, what was your area of study (major)? C3 When did you finish your latest studies? (approximate time when you plan to complete your studies if you are currently studying) Month__________ Year__________ C4 If you attended a level of schooling but did not graduate, what was the main reason for stopping your education? Failed examination (1) Did not enjoy schooling (2) Wanted to start working (3) To get married (4) Parents did not want me to continue schooling (5) Economic reasons (could not afford/needed to earn money to support family) (6) Other (99) Specify other reasons: __________ 71 D Employment D1 Are you currently employed? Yes (1) GO TO C2 No (2) GO TO C12 D2 Please describe your current work: Work in public sector (1) Work in private company (2) Work in non-profit organization (3) Work on farm (4) Work in family business (5) Work in informal (black) economy (6) Community volunteer work (7) Internship/apprenticeship in public sector (8) Internship/apprenticeship in private company (9) Self-employed/own your business (10) Other (99) Please specify:______________ D3 How many employees are there at your current employer (approximately)? <10 employees (1) 10-20 employees (2) 20-100 employees (3) 100-500 employees (4) 500+ employees (4) D4 Please provide information about your current employer: Company name:__________________________ Division or Department:____________________ D5 What is the job title for your position at your current job? 72 D6 How many hours per week do you work? 5-10 hours (1) 10-20 hours (2) 20-30 hours (3) 40 hours (full-time) (4) >40 hours (5) D7 Does your work offer benefits (health, paid vacation, education support, pension fund, etc.)? Yes (1) No (2) D8 Is your current work: Paid (1) GO TO C9 Unpaid (2) GO TO C10 D9 What is your current wage? Per Year: _______ Per Month: _______ Per Hour:________ D10 To what extent are you satisfied with your current job? Satisfied (1) GO TO C12 Unsatisfied (2) GO TO C11 D11 Which of the following best describes why you are unsatisfied with your current job? It is temporary (1) Low salary/wage (2) Low-level work (3) Problems with management (4) Routine, not interesting/challenging enough (5) Low promotion/salary increase perspective (6) Absence of benefits or their limit (7) Long commute to work (8) Other (99) Please specify:______________ 73 D12 Are you actively looking for a new job? Yes (1) GO TO C13 No (2) GO TO C14 D13 What is your job seeking strategy? Online search (1) Registering at public employment office (2) Registering at private employment agency (3) Attending career fairs (4) Using personal connections and assistance from friends, relatives, colleagues (5) Word of mouth (6) Placing and answering newspaper advertisements (7) Other (99) Please specify:_____________________ D14 Do you have any prior work experience (either paid or unpaid)? Yes (1) GO TO C15 No (2) GO TO C16 D15 Please describe your past work experience: Employer // Job Title/Role // Paid/Unpaid // Time period (month/year) D16 Which of the following type of work would you prefer if you complete the coding bootcamp training? Start your own business (1) Work for a large private company (2) Work for a startup (3) Work for the government/public sector (4) Work for a bank/financial sector (5) Work for a multinational corporation (6) Work for a non-profit organization (7) Work online/self-employment (8) Other (99) Please specify:__________ D20 What would be your ideal salary/wage per month if you complete the coding bootcamp training? $200-$500 (1) $600-1000 (2) $1100-2000 (3) $3000-4000 (4) >4100 (5) 74 E Basic Technical Skills E1 What is your typing score (words per minute)? (you can check it at https://www.cursomeca.com/test.php) E2 Do you know how to code in any of the following languages? Coding Language Yes No Ruby (1) Python (2) Java (3) HTML5 (4) Android (5) iOS (6) C++ (7) Other (99) Please specify:___________ E4 Where did you learn these coding languages? at secondary/high school (1) at university (2) at vocation school (3) from Massive Open Online Course (MOOC) (Lynda, Coursera, Udemy, Udacity, edX, iTunes U, etc.) (4) from a friend (5) Other (99) Please specify:_______________ E5 What is your English language proficiency (reading and writing)? Fluent (1) Intermediary (2) Fair (3) None (4) 75 MID- SURVEY (END OF TRAINING) Qualitative survey only applied in Beirut and Nairobi. Name: Email: 1. Do you think the bootcamp training you just received will be sufficient for you to become competitive in the job market? A. Yes B. No C. Not sure 2. What do you value about what you learnt at the bootcamp? (Check all that applies) A. Coding skills B. Soft skills (communication, presentation, teamwork, adaptability, problem solving) C. Jobs placement support D. Network with other students E. Network/meeting with professionals in the industry F. Other (Please specify)_______________ 3. Which of the values learnt in the bootcamp you value the most: A. Coding skills B. Soft skills (communication, presentation, teamwork, adaptability, problem solving) C. Jobs placement support D. Network with other students E. Network/meeting with professionals in the industry F. Other (Please specify)_______________ 4. Has the bootcamp met your expectations? A. Yes B. No C. Unsure (Please specify more)____________________ 76 5. What was your motivation level before the training and now, after the training, for looking for jobs? A. High before the training and high after the training B. High before the training and medium (ok) after the training C. High before the training and low after the training D. Low before the training and high after the training E. Low before the training and medium (ok) after the training F. Low before the training and low after the training G. Medium (ok) before the training and high after the training H. Medium (ok) before the training and medium (ok) after the training I. Medium (ok) before the training and low after the training 6. How confident you feel now about your future employment after you have completed the training? A. Very confident B. Confident C. Not so sure (Please specify more)___________________ D. Not confident (Please specify why)__________________ 7. What would you have changed in the training if you could? 8. What did you like in the training the most? 9. What did you dislike in the training the most? 10. Was it worth your time, money, and other efforts? A. Yes B. No C. Not sure (Please specify more)______________ 11. What was painful/too hard (Please check all that applies)? A. Learning coding languages B. Learning soft skills (communication, presentation, teamwork, adaptability, problem solving) C. Jobs placement support D. Networking with other students E. Networking/meeting with professionals in the industry F. Other (Please specify)_______________ 77 12. It was painful/too hard due to what reason (check all that applies)? A. Fatigue B. Long commute C. Shortage of time D. Shortage of money because you did not work E. Too hard to study F. Other (Please specify)_______________ 13. What was easy? (Please check all that applies)? A. Learning coding languages B. Learning soft skills (communication, presentation, teamwork, adaptability, problem solving) C. Jobs placement support D. Networking with other students E. Networking/meeting with professionals in the industry F. Other (Please specify)_______________ 14. It was easy due to what reason (Please check all that applies)? A. I was very motivated/inspired with the opportunities it brings B. I like learning new things C. Instructors were great D. The study ambience/environment was great E. The study materials were very explanatory F. Homework assignments helped solidify the knowledge gained in class G. I am just good at it H. Other (Please specify)___________________ 15. What would you change in the bootcamp for future students (Please check all that applies)? A. Coding languages (Please specify)__________ B. Soft skills (communication, presentation, teamwork, adaptability, problem solving) (Please specify)_______________ C. Jobs placement support (Please specify)___________ D. Networking with other students (Please specify)___________ E. Networking/meeting with professionals in the industry (Please specify)___________ F. Other (Please specify)_______________ 78 FINAL SURVEY Administered in the three cities, 6 months after the end of the bootcamp. A Personal, Family, and Household Information A1 Full name: A2 Address: A3 Mobile and/or home phone number: A4 Email: A5 Did you move during the last 6 months? Yes (1) GO TO A6 No (2) GO TO A8 A6 Where did you move (from which location -city, village, etc.- to which city, village, etc.)? A7 What was the main reason of your move? To accompany the family (1) For education/training/apprenticeship purposes (2) To work/for employment-related reasons (3) Other (99) Please specify: _____________ A8 Has you marital status changed since May 2016? Yes (1) GO TO A9 No (2) GO TO A12 A9 What is your current marital status? Never married (1) GO TO A12 Engaged to be married (2) GO TO A13 Married (3) GO TO A12 Separated/divorced (4) GO TO A13 Widowed (5) GO TO A13 A10 Has the employment status of your spouse changed since May 2016? Yes (1) GO TO A11 No (2) GO TO A12 79 A11 What does your spouse currently do? (Choose the most relevant activity) Attending education/training (1) Works for salary/wage with an employer (2) Self-employed (3) Unemployed & actively looking for job (4) Does not work (5) Engaged in home duties (including childcare) (6) Unable to work due to sickness or disability (7) Other (99) Specify activity/reason: ________________ A12 Have the number of children in your household changed since May 2016? Yes (1) GO TO A13 No (2) GO TO A14 A13 How many children do you currently have? A14 Have the number of persons in your household (including yourself) changed since May 2016? Yes (1) GO TO A15 No (2) GO TO A16 A15 How many persons are currently in your household (including yourself)? A16 Have the total income of your household changed since May 2016? Yes (1) GO TO A14 No (2) GO TO A15 A17 On average, what is the total income of your household? Per Year: _______ Per Month: _______ A18 Have the number of persons in your household who currently work for a salary/wage changed since May 2016? Yes (1) GO TO A19 No (2) GO TO A20 A19 Currently, how many persons in your household currently work for a salary/wage? A20 Have the number of persons in your household who are without work and actively looking for work changed since May 2016? Yes (1) GO TO A21 No (2) GO TO B1 A21 Currently, how many persons in your household who are without work and actively look for work? 80 B Education B1 Since May 2016, have you completed any education program? Yes (1) GO TO B2 No (2) GO TO B3 B2 Which education program you have completed since May 2016? No schooling/Pre-primary education (0) Primary education (1) Lower secondary education (2) Upper secondary education (3) Post-secondary non-tertiary education (4) First stage of tertiary education (5) Second stage of tertiary education (6) Other (99) Specify which one: _______________ B3 Since May 2016 have you applied to any new education program? Yes (1) GO TO B4 No (2) GO TO B5 B4 Which new education program you have applied for since May 2016? No schooling/Pre-primary education (0) Primary education (1) Lower secondary education (2) Upper secondary education (3) Post-secondary non-tertiary education (4) First stage of tertiary education (5) Second stage of tertiary education (6) Other (99) Specify which one: _______________ B5 Since May 2016 have you dropped out of an education program? Yes (1) GO TO B6 No (2) GO TO B7 81 B6 If since May 2016 you dropped out of an educational program, what was the main reason for stopping your education? Failed examination (1) Did not enjoy schooling (2) Wanted to start working (3) To get married (4) Parents did not want me to continue schooling (5) Economic reasons (could not afford/needed to earn money to support family) (6) Other (99) Specify other reasons: __________ B7 If since May 2016 you are studying at university, what is your area of study (major)? B8 If you have started your latest education after May 2016, what was the start day? B9 If you are not working or studying, are you looking for a job? What are you planning to do for the next 6 months? [for Treatment Group ONLY] B10 If after the completion of a bootcamp training you had applied and got accepted to a university program, do you think the bootcamp training contributed to your acceptance to a university program? Yes (1) GO TO B11 No (2) GO TO B13 I did not apply to any university program B11 The contribution of the bootcamp training to your acceptance to a university program was: Essential (5) Very significant (4) Significant (3) Somehow significant (2) Of little significance (1) B12 What was the most contributing factor of the bootcamp training that contributed to your acceptance to a university program? Coding skills (1) Soft skills (2) Increased self-confidence due to the completion of the bootcamp program (3) Getting a job that allows you to finance your education (4) Getting a merit-based scholarship thanks to the coding skills (5) Other (99) Please specify: ____ 82 C Employment C1 Are you currently employed? Yes (1) GO TO C2 No (2) GO TO C13 C2 Please describe your current work: Work for the government/public sector (1) Work in a large private company (2) Work in a small private company (3) Work in a startup (4) Work in non-profit organization (5) Work on a farm (6) Work in a family business (7) Work in informal (black) economy (8) Community volunteer work (9) Internship/apprenticeship in public sector (10) Internship/apprenticeship in private company (11) Self-employed/own your business (12) Other (99) Please specify: ______________ C3 How many employees are there at your current employer (approximately)? <10 employees (1) 10-20 employees (2) 21-100 employees (3) 101-500 employees (4) 500+ employees (4) C4 Please provide information about your current employer: Company name: __________________________ Division or Department: ____________________ C5 What is the job title for your position at your current job? C6 How many hours per week do you work? Less than 24 hours (1) 24 hours (2) 25-48 hours (3) 48 hours (4) >48 hours (5) 83 C7 Does your work offer benefits (health, paid vacation, education support, pension fund, etc.)? Yes (1) No (2) C8 Is your current work: Paid (1) GO TO C9 Unpaid (2) GO TO C10 C9 What is your current wage per month? Less than a SMLV1 (1) 1 SMLV (2) Between 1 and 2 SMLV (3) Between 2 and 4 SMLV (4) Between 4 and 6 SMLV (5) More than 8 SMLV (6) C10 Which of the following skills do you use at your current job (indicate all that apply):            Coding skills (1)             Soft skills (communication, presentation, teamwork, adaptability, problem solving…) (2)             Others (99)  Please specify: _________             None (0) C11 To what extent are you satisfied with your current job? Satisfied (1) GO TO C13 Unsatisfied (2) GO TO C12 C12 Which of the following best describes why you are unsatisfied with your current job? It is temporary (1) Low salary/wage (2) Low-level work (3) Problems with management (4) Routine, not interesting/challenging enough (5) Low promotion/salary increase perspective (6) Absence of benefits or their limit (7) Long commute to work (8) Other (99) Please specify: ______________ 1 SMLV stands for minimum salary in Colombia. The SMLV was: • 2016: 689.455 COP (240 USD) • 2017: 737.717 COP (257 USD) 84 C13 Are you actively looking for a new job? Yes (1) GO TO C14 No (2) GO TO C15 C14 What is your job seeking strategy? Online search (1) Registering at public employment office (2) Registering at private employment agency (3) Attending career fairs (4) Using personal connections and assistance from friends, relatives, colleagues (5) Word of mouth (6) Placing and answering newspaper advertisements (7) Other (99) Please specify:_____________________ C15 Since May 2016, have you created your own business? Yes (1) GO TO C16 No (2) GO TO D1 C16 Can you provide a description of your own business? C17 Including you, how many employees are there at your own business? 1 employee (1) 2-3 employees (2) 4-6 employees (3) +6 employees (4) 85 D Bootcamp Experience [For Control Group Only]: D1 Since May 2016, have you taken any technical skills training or coding bootcamp? Yes (1) GO TO D2 No (2) GO TO D3 D2 What kind of technical or bootcamp training have you attended since May 2016? Full-time and intensive 2-4 months long bootcamp (1) Short-term bootcamp (2) Online bootcamp (3) Short programming courses (4) Other (99) Please specify: ________________ [For Treatment Group Only]: ONLY IF YOU ARE CURRENTLY WORKING D1 Do you use the skills you learned in the bootcamp at your current job? Yes (1) GO TO D2 No (2) GO TO D4 D2 Do you think you got your current job because you had the knowledge acquired in the bootcamp? Yes (1) No (2) Please explain: ___________ D3 Are you planning to get a job where you can apply the skills acquired in the bootcamp? Yes (1) No (2) Please explain: ___________ 86 COMPLEMENTARY SURVEY II Only administered in Medellín, to get additional data on employment and type of job. A Basic Information A1 Nombre completo: A3 Número de celular: Número de teléfono fijo: A4 Email: B Basic Information B1 ¿Usted se encuentra actualmente laborando? Puede ser por cuenta ajena o por cuenta propia. í (1) VAYA A C4 No (2) VAYA A D0 (TRATAMIENTO) O FIN B4 ¿Cuál es el nombre de la entidad para la cual trabaja? B5 ¿Cuál es el nombre del cargo que usted desempeña en esa entidad? Queremos analizar si es un trabajo relacionado con el bootcamp B9 ¿Cuál es su salario por mes? (en pesos colombianos) El Salario Mínimo Legal Vigente para el año 2017 está en $737.717 COP (Pesos Colombianos) [SÓLO grupo de tratamiento] C0 ¿Cada cuánto practica las habilidades aprendidas durante el bootcamp, tanto en casa como en su empleo (si tiene un empleo)? Varias veces por semana (1) Una vez por semana (2) Una vez por mes (3) Nunca (4) 87 SEMI-STRUCTURED INTERVIEWS SAMPLE QUESTIONS Only administered for the qualitative studies in Beirut and Nairobi. These items focus on the CONTENT of the questions; based on the cultural expertise of the professionals conducting the interviews, the specific language of the questions and prompts might be changed considerably. Questions for everyone: 1. Tell me a little about the local landscape in terms of tech jobs – demand and opportunity. 2. What would you say are the main reasons someone would apply for (one of these jobs)? 3. What do people do today to help improve their chances of getting these jobs? 4. What are some failed strategies people employ to get these jobs? Why do they try them? 5. How did you first learn about the bootcamp program? What was your initial reaction when you heard about it? 6. What do your peers think about the bootcamp program? Do you talk about it much with friends, coworkers, or peers? Does the media or online ads cover it? [How salient is the program.] 7. If you had to make a list of people who have a vested interest in the bootcamp program, whom (all) would you include in that list? 8. What is your prediction for the overall outcome of this pilot bootcamp program? What reasons do you have for this prediction? What are you basing this prediction on? 9. What would you say the general feelings of people in your community are about the bootcamp program – the average across the citizens near you, across all social classes? [If not specified – do they have particular expectations for the program in terms of local tech and economic growth?] 10. When you think about the effects of a bootcamp program, how wide-ranging do you think it will be geographically? Citywide, regional, countrywide, or worldwide? 11. When you think about the effects of these programs, how broad do you think they will be in terms of social spheres? Will it just affect the tech industry? Will it improve or worsen things more broadly, so all citizens will benefit? Somewhere in between? 12. What would you say is the number one positive effect you foresee from this program? 13. What would you say is the number one negative effect you foresee from this program? 14. Would you encourage a friend or family member to participate in a bootcamp? Would it depend on their personality or…? 88 15. What percentage of the participants in the bootcamp do you think will have an easier time finding employment after? What percentage do you think will have higher incomes? 16. If you had to guess, what would you think would distinguish someone who “succeeds” in a bootcamp over someone who fails? 17. What are some of the unique challenges you predict for running a bootcamp here in [location] compared to another place? [A possible follow up: right now, the World Bank study includes pilot programs in Kenya, Lebanon, and Colombia. Do you think there is anything different about your location compared to the others that will make a difference in the effectiveness of the program? For example, local conditions, local cultures, specific differences in the types of people who hire or go to one of these programs?] For firms if they do hiring at all: 18. What characteristics do you look for when hiring an entry-level tech employee? 19. If you had to pick, what is the #1 thing you consider as the highest priority, all else being equal? 20. How much weight do you think you would give to having done a bootcamp in making your hiring decisions? For managers, HR experts, or just experts in industry: 21. What distinguishes an “ok” employee from an “outstanding” employee in an entry-level position? 22. What is the most typical pattern for a new hire, in terms of career progression and wages? 89 FOCUS GROUP SAMPLE QUESTIONS Only administered for the qualitative studies in Beirut and Nairobi. [Note: if there is a sizeable female population of participants, make sure at least one focus group is all-woman.] A large portion of the focus group prompts will be created from the one-on-one interviews. Other questions will be based on patterns drawn from the questionnaires, particularly in post-training focus groups. These will be the strongest questions or topics for discussion, including those at the end of the list. These items focus on the CONTENT of the questions; based on the cultural expertise of the professionals conducting the focus groups, the specific language of the questions and prompts might be changed considerably. Pre-training focus groups: 1. How did you hear about the bootcamp? [Might be better in a questionnaire.] 2. [Going around the room, in one or two words] What was your feeling when you realized you would be able to be part of this bootcamp? [Look for words like “optimistic,” “relieved,” or “stressed,” “worried,” etc. Also, see if it is uniformly positive, or if some people have negative reactions]. Was anyone surprised to hear any of those reactions? [Try to get a conversation between two people who had opposing reactions.] [This could be complicated a bit because some of the people are bootcamp peers and the others – who did not get to do a bootcamp. Assuming that this is before they do a bootcamp training but after the selection of participants from a larger pool of applicants.] 3. If you hadn’t been able to get into this bootcamp, what would you have done instead? [Possible directed follow up: anything you would have done to help you get a better job or more income?] [If they just say they would be doing all kinds of applications, or staying at home, etc. Basically… what alternate paths do they say people take to get ahead in the market?] 4. What are some of the experiences you are looking forward to as part of the bootcamp? 5. What personal changes do you expect to result from this experience? 6. What are some examples of changes in your schedule or daily life you are going to have to make to attend the bootcamp? 7. Tell us a bit about what your friends and family think about you attending the bootcamp. 8. Describe the type of person you think would be most likely to do well in the future because he/she has participated in this bootcamp. 9. Describe the type of person you think would do awfully in the job market even after he/she has completed this bootcamp. 10. Is there any kind of person you think would not benefit from this at all – something that makes them a success (or failure) no matter what? [If they are only focused on attitudes or personal qualities, propose the following: Let’s take a step back. What are some of the things you think affect how well a person does when trying to get a job or a higher salary that are important – but have nothing to do with their personality?] 11. Share with us some examples of the kinds of jobs you hope to get after completing this bootcamp. 12. What has it been like looking for a job/working where you work up to now? 90 13. Is there anything you feel has already changed for you now that you have been selected to participate in the bootcamp? 14. Why do you think (this location) was chosen as the place to run a pilot program? 15. Are your friends jealous that you are able to do something like this? 16. What are some of the things you had to do to get enough money to enter a training program like this one? Would any or all of you actually have done all that? 17. These bootcamps take a lot of time and energy. What is the main inspiration you are going to use to make it through the program? Post-training focus groups: 1. What is most different about you now, compared to before you took the training? 2. Do you look at anything differently now that you have gone through the training? 3. What were the hardest things about the training? 4. What was your favorite part about the training? 5. What was it like working with the bootcamp (leader/trainer/life coach/classmate)? 6. Have you made any “industry” connections as a result of the bootcamp training? 7. Do you think you have a better job now that you had have gotten if you had not taken the bootcamp? 8. What new opportunities do you think you have now that you would not have had if you hadn’t done the bootcamp? 9. What did you do/are you doing now to build off of the bootcamp? 10. How hard/easy it was to get into a job after the bootcamp? How long it took? 11. For those employed after the bootcamp: what is one thing you like in you job the most? What do you like the least? 12. What is your future career plans? [if we are curious if it involves moving out of the country.] 13. Do you feel like others treat you any differently now that you have completed the bootcamp? 14. What are some of the changes you have seen in your location in the past year? [does not matter whether they choose a city- or countrywide reference point – just see what they say.] 15. Who here has experienced a significant increase in their earnings over the past six months? What has changed in your lives now that you have more money? 16. What is one positive thing that happened after you attended boot camp that you never would have predicted beforehand? 17. What is your most memorable experience from participating in the bootcamp? 18. What was the best thing about bootcamp overall? 19. What was the worst thing about bootcamp overall? 20. Was there any negative consequence to you doing the bootcamp? 21. Do you feel like you are smarter after having done the bootcamp? 91 22. Had participating in the bootcamp affected your confidence at all? What experiences did you have that made you a more confident person? 23. [Around the room] Would you say generally that the bootcamp was a success for you, a disappointment for you, or something in between? What would you say was the biggest reason you think that this was the case? Was it something about you, or about the bootcamp? 24. What would you say is the most important quality a person should have to do well DURING the bootcamp? 25. What would you say is the most important quality a person should have to do well become successful AFTER the bootcamp? 26. What would be the worst quality for a person to have as a bootcamp participant - something that would make sure they do not do well during the training? [This question is tricky and sensitive because it might be insulting to people who did not do well in the bootcamp, to hear other people say XYZ about them.] 27. As part of this research, we have interviewed experts in your field, such as [general description of who they were]. We woud like to tell you what THEY thought about bootcamps, and see if you guys agree with them or disagree with them [here, take some of the consensus items from the interviews]. 28. Sometimes the people we interviewed disagreed with each other. We would like to present you with their opinions, and you tell us which perspectives you agree with and why – or if you disagree with all the opinions! 92 SOCIOEMOTIONAL QUESTIONNAIRE The following are the measurement scales for the prioritized non-cognitive variables. All scales in this annex are psychometric instruments that have been validated. A. Review of Personal Effectiveness and Locus of Control (ROPELOC) (Richards, et al. 2000) NOT LIKE ME LIKE ME 01. When I have spare time I always use it to paint. 1 2 3 4 5 6 7 8 02. I like cooperating in a team. 1 2 3 4 5 6 7 8 03. No matter what the situation is I can handle it 1 2 3 4 5 6 7 8 04. I can be a good leader. 1 2 3 4 5 6 7 8 05. My own efforts and actions are what will determine my future. 1 2 3 4 5 6 7 8 06. I prefer to be actively involved in things. 1 2 3 4 5 6 7 8 07. I am open to different thinking if there is a better idea. 1 2 3 4 5 6 7 8 08. In everything I do I try my best to get the details right. 1 2 3 4 5 6 7 8 09. Luck, other people and events control most of my life. 1 2 3 4 5 6 7 8 10. I am confident that I have the ability to succeed in anything I want to do. 1 2 3 4 5 6 7 8 11. I am effective in social situations. 1 2 3 4 5 6 7 8 12. I am calm in stressful situations. 1 2 3 4 5 6 7 8 13. My overall effectiveness in life is very high. 1 2 3 4 5 6 7 8 14. I plan and use my time efficiently. 1 2 3 4 5 6 7 8 15. I cope well with changing situations. 1 2 3 4 5 6 7 8 16. I cooperate well when working in a team. 1 2 3 4 5 6 7 8 17. I prefer things that taste sweet instead of bitter. 1 2 3 4 5 6 7 8 18. No matter what happens I can handle it. 1 2 3 4 5 6 7 8 19. I am capable of being a good leader. 1 2 3 4 5 6 7 8 20. I like being active and energetic. 1 2 3 4 5 6 7 8 93 21. What I do and how I do it will determine my successes in life. 1 2 3 4 5 6 7 8 22. I am open to new thoughts and ideas. 1 2 3 4 5 6 7 8 23. I try to get the best possible results when I do things. 1 2 3 4 5 6 7 8 24. When I apply myself to something I am confident I will succeed. 1 2 3 4 5 6 7 8 25. My future is mostly in the hands of other people. 1 2 3 4 5 6 7 8 26. I am competent and effective in social situations. 1 2 3 4 5 6 7 8 27. I can stay calm and overcome anxiety in almost all situations. 1 2 3 4 5 6 7 8 28. I am efficient and do not waste time. 1 2 3 4 5 6 7 8 29. Overall, in all things in life, I am effective. 1 2 3 4 5 6 7 8 30. When things around me change I cope well. 1 2 3 4 5 6 7 8 31. I am good at cooperating with team members. 1 2 3 4 5 6 7 8 32. I can handle things no matter what happens. 1 2 3 4 5 6 7 8 33. I solve all mathematics problems easily. 1 2 3 4 5 6 7 8 34. I am seen as a capable leader. 1 2 3 4 5 6 7 8 35. I like to get into things and make action. 1 2 3 4 5 6 7 8 36. I can adapt my thinking and ideas. 1 2 3 4 5 6 7 8 37. If I succeed in life it will be because of my efforts. 1 2 3 4 5 6 7 8 38. I try to get the very best results in everything I do. 1 2 3 4 5 6 7 8 39. I am confident in my ability to be successful. 1 2 3 4 5 6 7 8 40. I communicate effectively in social situations. 1 2 3 4 5 6 7 8 41. My life is mostly controlled by external things. 1 2 3 4 5 6 7 8 42. I am calm when things go wrong. 1 2 3 4 5 6 7 8 43. I am efficient in the way I use my time. 1 2 3 4 5 6 7 8 44. I cope well when things change. 1 2 3 4 5 6 7 8 45. Overall, in my life I am a very effective person. 1 2 3 4 5 6 7 8 94 B. Escala Grit de Duckworth (Duckworth et al., 2007) (Duckworth and Quinn, 2009) 0- Not like me at all 1- Not much like me 2- Somewhat like me 3- Mostly like me 4- Very much like me 1. New ideas and projects sometimes distract me from previous ones.* 01234 2. Setbacks don’t discourage me. 01234 3. I have been obsessed with a certain idea or project for a short time but later lost interest.* 01234 4. I am a hard worker. 01234 5. I often set a goal but later choose to pursue a different one.* 01234 6. I have difficulty maintaining my focus on projects that take more than a few months to complete.* 0 1 2 3 4 7. I finish whatever I begin. 01234 I am diligent. 0 1 2 3 4 8. Scoring: For questions 2, 4, 7 and 8 assign the following points: 5 = Very much like me 4 = Mostly like me 3 = Somewhat like me 2 = Not much like me 1 = Not like me at all For questions 1, 3, 5 and 6 assign the following points: 1 = Very much like me 2 = Mostly like me 3 = Somewhat like me 4 = Not much like me 5 = Not like me at all Add up all the points and divide by 8. The maximum score on this scale is 5 (extremely gritty), and the lowest score on this scale is 1 (not at all gritty). 95 This work is available under the Creative Commons Attribution Non-Commercial 3.0 IGO license (CC BY NC 3.0 IGO). Under the Creative Commons Attribution Non-Commercial license, you are free to copy, distribute, transmit, and adapt this work for non-commercial purposes, under the following conditions: Attribution—Please cite this work as follows: World Bank. 2018. Coding Bootcamps for Youth Employment, Evidence from Colombia, Lebanon, and Kenya. License—Creative Commons Attribution Non-Commercial CC 3.0 IGO Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations—If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Responsibility for the views and opinions expressed in the adaptation rests solely with the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third-party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to re-use a component of the work, it is your responsibility to determine whether permission is needed for that re-use and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. All queries on rights and licenses should be addressed to the Publishing and Knowledge Division, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@ worldbank.org