RESULTS-BASED FINANCING RBF EDUCATION EVIDENCE GLOBAL Can Results-Based Financing Improve the Effectiveness of Short-Term Education and Skills Training Programs? JUNE 2021 The study explored whether the use of financial incentives for training providers and participants improved the impact of short-term education and skills training programs on out-of-school and unemployed youth. Photo courtesy of Kashan Mustafa The Results in Education for All Children (REACH) Trust Fund supports and disseminates research on the impact of results-based financing on learning outcomes. The EVIDENCE series highlights REACH grants around the world to provide empirical evidence and operational lessons helpful in the design and implementation of successful performance-based programs. No jobs No school Globally, an estimated 267 million with more practical workplace youth aged between 15 and 24 are training and are offered by Many youth are neither in school nor employed. neither in education nor employed,1 governments, often in partnership with most of these youth residing in with the private sector. These Sub-Saharan Africa and South Asia. programs are becoming more This number is likely to increase important as job requirements substantially due to the COVID-19 change and increasingly demand pandemic. Short-term education more advanced cognitive skills. and skills training programs can Given the large amount of money help unemployed and out-of-school that governments channel into RBF incentives may help youth acquire the qualifications such programs, there is a growing improve engagement and and skills they need to secure and interest in using results-based successful completion of maintain productive jobs. Such financing (RBF) to maximize training programs. training programs typically combine returns on their investment. elements of classroom education This note was adapted from Clarke, Marguerite M.; Sharma, Meghna; Bhattacharjee, Pradyumna. 2021. Review of the Evidence on Short-Term Education and Skills Training Programs for Out-of-School Youth with a Focus on the Use of Incentives. Washington, D.C. : World Bank Group. 2 RFB EDUCATION | EVIDENCE Training programs can better results, is increasingly popular measuring whether participants succeed at providing out-of-school among governments as a way to were employed or in further training youth with relevant skills if they provide these kinds of incentives within a few months of completing use the right incentives to motivate in education and skills training the program. Data on longer-term providers and participants. Incentives programs.2 The Results in Education impacts are much more limited. can encourage providers to identify for All Children (REACH) Trust Fund In addition, despite the general and recruit trainees from the groups supported a team of World Bank increased interest in RBF in most in need of support; to invest researchers to conduct a study3 on education, few studies have explored adequate resources into their the use of incentives, particularly the inclusion and impact of RBF programs and deliver high-quality RBF, in short-term education and in short-term education and skills training; to ensure that participants skills training programs for out-of- training programs. Given this, there attend training regularly and school and unemployed youth was a clear need for a single study participate fully; and to provide them around the world. to review and analyze the available with the support needed to graduate evidence on the use of RBF in short- with the right qualifications and skills Despite the growth in the number term training programs. to find good jobs. Incentives can of such training programs globally also motivate participants to enroll in programs, attend every day, complete in recent years, few have undergone any kind of rigorous evaluation. HOW WAS THE the training, and search for and obtain employment. Results-based Although a small number of programs have data on program STUDY DESIGNED? financing (RBF), which links financial outcomes, these data tend to The study presents a review of payments to specific, certifiable address short-term outcomes, the global literature on short-term education and skills training programs for youth who are out of school and unemployed. It addresses three main questions: (i) What kinds of programs have been implemented around the world and what is the evidence on their effectiveness? (ii) How do these programs incentivize participants and providers, and what do we know about the relative effectiveness of Most of the 267 those that use an RBF approach? (iii) What are some implications for million youth that the future design of these programs? are neither in school After an extensive literature search, nor employed reside 10 studies were selected to be included in the review based on the in Sub Saharan following criteria: (i) the program Africa or South Asia uses education and/or skills training to improve the educational outcomes or workforce participation of out-of- school youth, preferably those GLOBAL 3 who did not complete secondary schooling; (ii) the program reported data on program outputs and outcomes; and (iii) there is evidence of program impact through the use of treatment and control groups (preferably in the form of a randomized control trial). This methodology had limitations. It may have left out some relevant published literature that was missed during the team’s initial search. It may also have left out relevant programs whose results were not published because the studies did not find Photo courtesy of Jan Shikshan Sansthan, Malappuram significant impacts or because they replicated existing findings. WHAT WERE THE FINDINGS? For each selected study, the The programs covered in the separately examined five studies researchers categorized the ways selected studies were from a from that region that had outcomes in which the program provided wide range of countries, but data but no control group.5 incentives to training providers, they targeted similar kinds of trainees, or other stakeholders, with participants. The studies covered The evaluated programs differed an emphasis on the program’s use programs in Brazil, Colombia, the in whether they offered remedial of RBF. The team disaggregated and Dominican Republic, Liberia, Nepal, education in addition to vocational analyzed the use of RBF at three Uganda, the United Kingdom, and skills training. Of the 10 programs, stages of program implementation: the United States. Despite this five focused on vocational skills during the recruitment of participants; geographical diversity, the programs training (which might also include during the delivery of training; and evaluated had many similar life skills and business skills), with an after the training, when participants characteristics. Most accepted emphasis on equipping participants were looking for or entering work. individuals between 16 and 30 for work. Four included both work- years old who were unemployed, focused skills training and an element and targeted those from poor urban of broader education, typically Carefully calibrated or minority neighborhoods or who targeting basic literacy and numeracy had an income below a certain skills. One program, the Economic RBF incentives defined level. Most of the programs Empowerment of Adolescent Girls can help providers were financed by government or and Young Women (EPAG) program international development agencies, in Liberia, began with a focus on avoid pitfalls such although two included private-sector skills training but later expanded to as failing to recruit financing. All but two were started include basic literacy and numeracy in the 2000s.4 Because no suitable education so that less educated youth participants from more literature was found from East Asia could participate. disadvantaged groups. and the Pacific, the researchers 4 RFB EDUCATION | EVIDENCE Five of the 10 evaluated programs more disadvantaged groups such successful placement of the most were more explicit in their use as widows, disabled women, and disadvantaged trainees—combining of RBF. Three of these programs, former combatants. Two programs into a single payment incentives to in Colombia, Nepal, and the used RBF for providers during reach disadvantaged groups and to United States, only used RBF for training, primarily to incentivize them place such trainees in employment. training providers; one, also in the to ensure that trainees completed In the Job Training Partnership Act United States, only used RBF for the course. In these cases, the (JTPA) program in the United States, participants; and one, in Liberia, government retained a share of payments were linked to trainees’ used RBF for both providers and the fee it paid to providers for each employment status and their level of participants. participant until the participant earnings 13 weeks after completion completed the course. of the program. This was an unusual RBF for training providers was program as RBF was used at two used during all three stages At the post-training stage, three levels: The federal government of program implementation. programs used RBF to incentivize tied state-level funding to the Two of the evaluated programs providers to ensure that graduates achievement of performance goals, used RBF for training providers were placed in employment or, in and the states tied the financing of during the recruitment stage, to some cases, further study. This training providers to similar goals, incentivize them to target the typically meant that part of the with an emphasis on participants’ most disadvantaged youth among payment per trainee was withheld employment and earnings. those eligible to participate. For until the trainee was confirmed to be example, Nepal’s Adolescent Girls employed three to six months after The four programs that used RBF Employment Initiative (AGEI) allowed the completion of training. Another for training providers achieved training providers to choose their approach was to issue a bonus positive impacts, but given the own trainees from the population of payment to the provider based on design of the program evaluations, unemployed and out-of-school youth, the number of trainees who secured these impacts cannot be definitively but paid a higher fee to providers jobs. For AGEI in Nepal, the size of attributed to the effect of RBF. Of that agreed to train youth from these bonuses was greater for the those programs, three were found RBF was inserted into training in different ways Recruitment Training Post-training Incentives to target Incentives to ensure Incentives to ensure disadvantaged applicants completion of course placement of graduates GLOBAL 5 Photo courtesy of Kashan Mustafa to have had significant positive stage, none of the evaluated programs Evidence was also limited on how impacts on participants’ chances of used RBF explicitly to attract the use of RBF impacted trainee being in employment and three were participants to the program. One behavior and performance. Among found to have significantly increased unusual intervention, used in Uganda’s the programs that provided results- participants’ earnings. Two of the Youth Opportunities Program, offered based incentives to trainees, the programs achieved larger benefits groups of unemployed youth a cash Year Up program in the United States for female participants than male grant at the start of the program to significantly increased participants’ participants. In some cases, the finance a training and employment wages compared to the control programs that were rewarded for project proposed by them. The group, while the EPAG program ensuring employment of attendees program then supported the youth to in Liberia generated benefits to also achieved other significant complete their project over a specified participants in terms of employment benefits such as increasing women’s period of time. In terms of the training and earnings and additional benefits control over their household stage, four of the 10 programs involving food security as described resources (in Liberia and to a evaluated offered stipends to trainees above. However, neither study lesser extent, Nepal) or improving to cover transportation and other provided experimental evidence of participants’ household food security costs related to participation. In two the specific impact of the incentives (in Liberia). However, none of the of these four programs, the stipends on participants. studies provided rigorous evidence were reduced if the trainees were of the specific contributions from absent, frequently arrived late, or Overall, the team concluded that the use of RBF to these impacts, for failed to complete assignments on while a number of the evaluated example by comparing providers that time, effectively making the stipends programs used RBF and were were subject to RBF to a randomized a form of RBF. One of these programs, generally impactful, it did not group of providers that were not. the EPAG in Liberia, also paid a bonus have enough evidence to draw of US$20 each to the participants conclusions about the specific RBF directed at participants (girls ages 16–27) who completed impact of the use of RBF on these was mostly used during training the program. None of the programs programs or their participants. to incentivize attendance and provided rewards to trainees during participation. At the recruitment the post-training stage. 6 RFB EDUCATION | EVIDENCE WHAT WERE (with the latter receiving small bonus payments for successfully recruiting regression-based formula to adjust for the impact of various individual THE LESSONS suitable applicants). and contextual variables on program pass rates. Another approach could LEARNED? Designers of training programs should pay close attention to how be to use a weighted formula to allocate more funding to providers RBF is a useful tool but it’s not RBF can affect the selection of who enroll, train, and place the most necessarily the only way to achieve students for training. The neediest disadvantaged applicants. desired program outcomes. students, who stand to benefit most For example, while five of the 10 from training, may also face the RBF needs to be carefully linked to evaluated programs used RBF, none biggest challenges in achieving the indicators to ensure a proper reward of them saw it as a way to encourage desired results, both in terms of for impact. As described above, the youth to enroll. Instead, the preferred learning and obtaining employment. JTPA program in the United States recruitment approach was to use This could create trade-offs for tied payments to the employment carefully targeted communications training providers. For example, and wages of trainees 13 weeks after and outreach strategies to address RBF that rewards providers based the training. However, researchers any concerns and build intrinsic on the number of trainees who found that these short-term motivation among targeted youth complete the course or obtain jobs employment outcomes were weakly and their families. A good example might discourage providers from and negatively related to participants’ of this is the AGEI in Nepal, which recruiting participants from more earnings and employment 18 to struggled to recruit women in the disadvantaged groups. However, 30 months after they completed first year, but ended up successfully carefully calibrated incentives can training.6 Where possible, RBF should reaching women in subsequent years help avoid such pitfalls. To ensure be applied to medium- to long-term through a customized radio and that facilitators do not choose less outcomes to ensure that providers television campaign and partnerships challenging participants to work are incentivized to develop long- with community-based organizations with, program designers might use a lasting skills in participants. Photo courtesy of Kashan Mustafa GLOBAL 7 Table 1: Overview of Training Programs for Out-of-School Youth Program  Beneficiaries  Content  Use of RBF for Results  Providers Participants Brazil: Male and female youth aged • 180 hours of academic and basic Employment: +  Galpão Aplauso under 29 from households skills education; Earnings: +  (2009– 2013)  with monthly incomes below • 300 hours of vocational training; minimum wage, screened Personality traits: no effect • 120 hours of life skills training through interviews and math and language tests  Colombia: Unemployed male and • Three months of classroom During Employment: + Jóvenes en Acción female youth aged between training on vocational skills; (larger effects for women)  (Youth in Action) 18 and 25 from lowest • On-the-job training Earnings: (2001–2005)  socioeconomic strata in Women: +18% urban areas Men: +8% Dominican Male and female high school • Basic skills training; Overall employment: Republic: dropouts aged between • Vocational training no significant impact Juventud y 16 and 29; living in poor Quality of employment: Empleo (Youth neighborhoods; unemployed, + (males) and Employment underemployed, or inactive; Program) (2001– not holding an identity card  Earnings: + (women) not specified)  Teenage pregnancy: – Perceptions and expectations: + (stronger impact for females and younger participants) Socioemotional skills: +  Liberia: Women aged 16 to 27 with • Six months of classroom-based After During Employment: + Economic basic literacy and numeracy training; Earnings: + Empowerment of skills, not enrolled in school, • Six months of job placement Adolescent Girls and living in one of nine target support Control over monetary resources: + and Young Women communities  Food security: + (EPAG) (2010–2014)  Attitudes towards gender norms: +  Nepal: More than 4,000 women aged • Basic life skills training; Before Employment: + Adolescent Girls between 16 and 24 over a • Technical/vocational skills During Earnings: + (larger impact for women) Employment three-year period training; After Initiative (AGEI) Empowerment and self-confidence: • Basic business skills training no consistent impact (2010–2012)  Control over earnings: + Uganda: Youth Poor, underemployed male • Cash grants offered to groups Earnings: +73% (women); +29% Opportunities and female youth, aged 16– for self-employment; vocational (men) Program (YOP) 35, from Uganda’s conflict- training (for two-thirds of Capital/assets: +  (2006–2008)  affected North participants) Tendency to practice skilled trade: +  United Kingdom: 16- to 17-year-old unemployed • Apprenticeship training in Time it took to find any job: – Youth Training youth  industry Ability to find a “good” job: + (women) Scheme (1983–1989)  no significant impact (men) United States: Adults who lack high • Basic skills instruction; Educational outcomes: + Integrated Basic school-level skills enrolling • Tertiary occupational courses Wages and hours worked: no Education and in community and technical significant impact Skills Training colleges Program (I-BEST) (2004–not specified)  United States: Unemployed individuals (over • Basic education leading with Before Earnings: modest + (overall for both Job Training age 16) meeting income- high school equivalency After male and female youth); – (adult males) Partnership eligibility limits credential; Educational attainment: Act (Title II-A) • Classroom training in + (females and male dropouts) (1982)/Workforce occupational skills; Investment Act Welfare receipt: • On-the-job training; no significant impact (JTPA) (1998)  • Job search assistance United States: Male and female adults aged • Skills training for tertiary education During Earnings: +  Year Up (2000– 18 to 24 from low-income including reading and writing; College attendance: no significant not specified)  urban communities  • Six months of training in impact  information technology and investment operations; • Six-month internship Source: Clarke, Marguerite, Meghna Sharma, and Pradyumna Bhattacharjee. 2021. “Review of the Evidence on Short-Term Education and Skills Training Programs for Out-of- School Youth with a Focus on the Use of Incentives.” Social Protection & Jobs Discussion Paper No. 2103, World Bank, Washington, DC. Note: + = positive effect; - = negative effect. CONCLUSION training, and placement of graduates in employment. For participants, Short-term education and skills RBF was used to incentivize full training programs for youth who are attendance and completion of out of school and unemployed are courses. The programs that used increasingly common around the RBF had significant positive impacts world, but there is little evidence on on participants’ employment, the use of RBF in these programs. earnings, and in some cases The REACH Trust Fund supported education outcomes. However, none Photo courtesy of Kashan Mustafa a team of researchers to conduct a of the reviewed studies incorporated The programs that used global review of the use of incentives a research design that provided in such programs. Out of 10 rigorous rigorous evidence of the specific RBF had significant evaluation studies of suitable contribution of RBF to the program’s positive impacts programs, the team found that five outcomes. Further research is featured training programs that needed to test the specific benefits on participants’ used RBF at some stage to target of different uses of RBF within employment, earnings, training providers or participants. particular programs, and to connect RBF for training providers was used these findings to the broader and in some cases to incentivize recruitment of suitable theoretical literature on incentives education outcomes. trainees, delivery of high-quality and motivation in education. 1 ILO (International Labour Organization). 2020. “Global Employment Trends for Youth 2020.” International Labour Organization, Geneva. 2 World Bank. 2019. Results-Based Financing In Education: Learning from What Works. Washington, DC: World Bank. 3 Clarke, Marguerite, Meghna Sharma, and Pradyumna Bhattacharjee. 2021. “Review of the Evidence on Short-Term Education and Skills Training Programs for Out-of-School Youth with a Focus on the Use of Incentives.” Social Protection & Jobs Discussion Paper No. 2103, World Bank, Washington, DC. 4 The remaining two programs were early examples of this kind of program from the 1980s: the Youth Training Scheme from the United Kingdom and the Job Training Partnership Act from the United States. 5 The additional five studies from East Asia and the Pacific looked at programs in China, East Timor/Timor Leste, and the Philippines. 6 Heckman, James J., Carolyn Heinrich, and Jeffrey Smith. 2002. “The Performance Of Performance Standards.” Journal of Human Resources 37 (4, Fall), 778–811. RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) worldbank.org/reach REACH is funded by the Government of Norway through NORAD, the Government of the United States of America through USAID, and the Government of Germany reach@worldbank.org through the Federal Ministry for Economic Cooperation and Development.