from EVIDENCE to POLICY Learning what works for better programs and policies October 2022 KENYA: Can scripted schooling improve learning? Before the COVID pandemic, more than half of children in low- for-profit company that codified and standardized both pedagogy and middle-income countries suffered from learning poverty: and teacher monitoring across more than 400 schools across the they either were out of school or failed to learn to read with com- country. Teachers in Bridge schools were equipped with a basic prehension by age 10. At the same time, numerous studies have tablet computer (or e-reader) containing detailed lesson plans cre- documented serious challenges related to the quality of education ated centrally by staff located in the U.S. and Nairobi, Kenya. A services, particularly for those serving poor students. In a country scholarship program for Bridge had almost 30,000 applicants for like Kenya, for example, teachers exhibit low levels of content and 10,000 scholarships. To deal with this over-subscription, scholar- pedagogical knowledge. Previous research has shown that highly ships were allocated through a lottery, and researchers tracked ap- EDUCATION plicants for two years. Preprimary and primary students induced to enroll in a Bridge school by the scholarship learned much more compared to students who did not get scholarships, with test score impacts among the highest observed in the international educa- tion literature; they also made more timely grade progression, per- formed better on the primary school leaving exam, and exhibited gains in cognitive development not captured by subject-matter tests. Low-achieving students benefited more from the program, and impacts were uniform across schools. It is not possible to attribute the learning gains of Bridge students solely to the intense scripting followed in schools, as Bridge schools differ from government schools along multiple di- mensions. Bridge employs teachers with fewer credentials, pays Photo: World Bank / Sarah Farhat them much less, and monitors them much more closely than structured teaching guides could improve literacy, but scripted les- civil service teachers. Their class sizes are smaller and school days sons are not without critics, who worry that teachers will not be are longer. Physical facilities are more basic. In addition to the able to adapt content to student’s needs. In places where teachers lesson plans, other processes were standardized across all Bridge may be less prepared to tailor high quality lessons to their students, schools, such as teacher recruitment and monitoring, the payment however, scripting may offer a way to standardize a minimum level of school fees, and the construction of school buildings. Neverthe- of quality at scale. less, these results suggest that scripting may be a promising way to The World Bank’s Strategic Impact Evaluation Fund sup- improve and standardize the quality of education at scale. Future ported an evaluation of a scholarship program that provided experimentation is needed to see if the impressive gains observed funding for preprimary and primary students to attend private can replicate in the public sector and in other countries. schools in Kenya operated by Bridge International Academies, a This policy note is based on Guthrie Gray-Lobe, Anthony Keats, Michael Kremer, Isaac Mbiti, and Owen Ozier, ”Can Education be Standardized? Evidence from Kenya,” BFI Working Paper, Development Innovation Lab, June 2022. Context Primary education is nearly universal in Kenya, as in many other At the time of the study, Bridge hired teachers with less formal lower-middle-income countries, and the private sector serves ap- education and experience than public school teachers, paid them proximately a third of preprimary students and 16 percent of pri- much less, and required them to work longer hours per week, al- mary students. At the end of primary school, students take the though working hours and pay were comparable to those in other high-stakes Kenya Certificate of Primary Education exam, which private schools serving the same population. Only 23 percent of plays a role in determining which, if any, secondary schools stu- Bridge’s primary grade teachers had more than secondary school dents can attend. education. Bridge head teachers earned roughly $100 per month, At the time of the study, Bridge International Academies was while civil service teachers earned several times as much. a private education company operating in multiple countries, not Class sizes were also smaller in Bridge schools. At the time without controversy. Teachers’ unions and international and lo- of the study, the pupil-teacher ratio in preprimary classes was 13 cal non-governmental organizations expressed concerns about in Bridge schools, 25 in other private schools, and 31 in pub- the working conditions of teachers, safety of children, and the lic schools. At the primary level, the pupil-teacher ratio was 20 ethics of charging poor families school fees. While simple com- in Bridge schools, 17 in other private schools, and 34 in public parisons showed that Bridge students in Kenya had above-average schools. test scores in the national primary-school exit examination, this Box: What does a scripted lesson look like? pattern could simply have reflected sorting or selection bias; for example, parents with academically talented children may have disproportionately enrolled at Bridge. In Kenya, Bridge had more than 400 schools serving almost 100,000 students and charging approximately $100 per year. Bridge standardizes lessons in all grades through centrally-devel- oped and highly-detailed lesson guides delivered to teachers using tablet computers. The guides even provide teachers with detailed instructions on classroom management and pupil engagement (see box for a sample lesson). School heads are trained and moni- tored to observe teachers twice daily, recording information on adherence to the detailed teaching plans and their interactions with students. School heads also must follow detailed scripts for giving teachers feedback, which include tallying the number of times a teacher goes off-script for more than 10 seconds, skips a line in the script, or rephrases a line or translates a line to a local language. Evaluation Researchers set up an evaluation to test whether attending a Bridge what happens when you offer scholarships to Bridge. This is called school improved children’s learning. An NGO (UnitedWeReach) the intention-to-treat effect in the impact evaluation literature, started a scholarship program for Bridge schools for the 2016 and and this estimate tells us what would happen to average outcomes 2017 school years. Almost 30,000 students applied, including of applicants if there were a scholarship program for Bridge. Be- children already attending Bridge schools without financial assis- cause not everyone offered a scholarship takes it up and because tance. Since the program could only fund 10,000 scholarships, a some applicants will go to Bridge even without a scholarship, this lottery was used to allocate the scholarships. does not tell us the impact of actually attending Bridge. To esti- Comparing scholarship recipients to non-recipients tells us mate the impact of attending Bridge, the intention-to-treat esti- mate needs to be scaled by the proportion of children who react to To measure outcomes, the study used information collected the scholarship offer and attend Bridge. In this case, the research- through phone calls with caregivers and home-based interviews ers exploit the experimental design to estimate the causal impact with children. This included information on school enrollment, of the scholarship on Bridge enrollment. They then use these es- grade level, national-curriculum-aligned assessment scores, and timates to create a Bridge attendance variable that just captures performance on cognitive and non-cognitive tasks. For children the variation in attendance induced by the scholarship program, old enough to have completed primary school, the study also used as this is the variation that is random (and can be used to estimate (pupil-reported data on) primary school leaving exam scores. The causal effects), as opposed to variation that arises from differences study focused on children’s learning and did not collect data on among households in how much value they place on attending a teachers beyond their credentials, experience, and tenure. Thus, Bridge school. They then use this adjusted Bridge attendance vari- this evaluation cannot shed light on teacher wellbeing, which is able to measure the impact of Bridge attendance on learning and one focus of concern for some of Bridge’s international and local grade progression. critics. Findings Two-year scholarships increased the probability that more likely to be in their projected grade compared to appli- applicants attended Bridge schools. cants not offered the scholarship; primary applicants were 20 percentage points more likely to be in their projected grade. In the first year of the study, 19 percent of primary school ap- plicants to Bridge schools who were not offered a scholarship Enrolling in a Bridge school dramatically improved through the lottery enrolled in Bridge schools anyway. Getting a learning for both preprimary and primary school stu- scholarship increased enrollment by 37 percentage points, nearly dents, with test score effects among the largest re- tripling enrolment in Bridge schools among applicants. Among corded in the international education literature. preprimary applicants, the scholarship more than doubled enrol- ment compared to applicants who did not receive scholarships, After two school years, students who attended Bridge because increasing Bridge attendance by 34 percentage points (over and of the scholarship program demonstrated a large test-score ad- above the 28 percent of applicants who didn’t receive a schol- vantage over their counterparts who were not offered schol- arship but chose to attend Bridge anyway). While the increase arships on tests that covered English, Kiswahili, Math, Social in Bridge attendance among primary school students mainly Studies, and Science. Average test score gains were 1.35 stan- came from students who would have otherwise attended public dard deviations for the preschool cohorts and 0.81 standard schools, the increase among preprimary applicants came from deviations for the primary school cohorts. To get a sense of students switching out of both public and private options. As the magnitude of these effects, the authors translate them into almost all applicants enrolled in school, the scholarship had no “equivalent years of schooling” or the years of schooling it effect on overall school enrolment. Effects on enrolment were of would take the control group to make the same learning gains. the same magnitude in the second year of the scholarship. When students attended Bridge because of the scholarship, preprimary students learned the equivalent of an additional Enrolling in Bridge schools improved timely grade pro- 1.48 years of schooling over and above the control group, and gression. primary school students learned the equivalent of an addition- al 0.89 years. These learning gains persist even when the au- At endline, in both the preprimary and primary samples, 74 thors restrict their focus to exam questions not easily answered percent of applicants who did not get scholarships were in through rote memorization. Children’s skills in non-subject the grade they would be projected to be in if they were not matter domains like fluid intelligence, working memory, and held back. Preprimary applicants induced to enroll in a Bridge receptive vocabulary also improved. school because of the scholarship were 18 percentage points Enrolling in a Bridge school also increased the The consistency of effects across teachers and likelihood that students took and passed the national sites suggests greater standardization of students’ school leaving exam. classroom experiences. Those induced to enroll in a Bridge school because of the Just as all students appeared to gain, regardless of their aca- scholarship were 15 percentage points more likely (from a base demic achievement, researchers also could not detect any sta- of 74 percent) to take the Kenya Certificate of Primary Educa- tistically significant heterogeneity in impacts across different tion exam, which determines admissions in secondary school. school locations. Students’ learning gains were also unrelated They were also 17 percentage points more likely (from a base to teacher characteristics like experience, tenure at Bridge, or of 41 percent) to pass the exam, in part due to the fact that teachers’ scores on the Kenya Certification of Secondary Edu- they were more likely to take the exam on schedule. cation exam. Lower achieving students gained the most. Despite the large gains in learning, some challenges remained for students attending Bridge schools. While both high-achieving and low-achieving students made significant learning gains, in both the preprimary and primary Although student-reported corporal punishment went down samples, the gap between the scholarship recipients and non- in Bridge schools relative to the non-Bridge schools students recipients was greater at the lower end of the distribution of would otherwise have attended, it remained high. Those en- test scores. In fact, in both samples, inequality in learning out- rolled at Bridge were 6 percentage points less likely to report comes decreased, as the standard deviation of test scores was seeing the practice compared to a base of 83 percent among smaller among scholarship recipients, indicating that the scores students who did not receive scholarships. Those students who were less spread out than what was observed among applicants enrolled in Bridge after winning a scholarship and their care- who had not been offered the scholarship. givers were 8 percentage points more likely to report hazards in playing fields, compared to a base of 34 percent among those not receiving scholarships. Conclusion The scholarship lottery for Bridge schools in Kenya dem- of treatment effects in the international education literature. onstrated that economically disadvantaged students can make Bridge schools differ from government schools along multiple very large gains in learning at both the preprimary and primary dimensions, and while it is not possible to isolate the most levels and that these gains can materialize at scale and uni- effective element of Bridge’s approach to pedagogy and man- formly across different schools and different teacher profiles. agement, the highly scripted nature of lessons and monitoring The estimated learning impacts exceed the 90th percentile shows promise for future experimentation. The Strategic Impact Evaluation Fund, part of the World Bank Group, supports and disseminates research evaluating the impact of development projects to help allevi- ate poverty. The goal is to collect and build empirical evidence that can help governments and development organizations design and implement the most appropriate and effective policies for better educational, health, and job opportunities for people in low and middle income countries. For more information about who we are and what we do, go to: http://www.worldbank.org/sief. The Evidence to Policy note series is produced by SIEF with generous support from the British government’s Foreign, Commonwealth and Devel- opment Office and the London-based Children’s Investment Fund Foundation (CIFF). THE WORLD BANK, STRATEGIC IMPACT EVALUATION FUND 1818 H STREET, NW, WASHINGTON, DC 20433