Policy Research Working Paper 10843 Learning Loss as a Result of COVID-19 Evidence from a Longitudinal Survey in Malawi Salman Asim Sajitha Bashir Ravinder Casley Gera Education Global Practice July 2024 Policy Research Working Paper 10843 Abstract School closures from COVID-19 have resulted in large higher than the loss documented in high income contexts. learning losses, from 0.05 to 0.17 standard deviations in Decomposing this loss, the findings show that students lost high income countries, equivalent to two to six months of 0.25 standard deviations of existing knowledge during the lost learning. However, the extent of primary-level learning closure, and a further 0.23 standard deviations in foregone loss in low-income countries remains unclear, studies lack learning compared to the expected trajectory had schools information on individual students’ learning trajectories, remained open. Further loss comes from a slowdown in and most do not include students who dropped out. This learning after schools reopened, with students gaining 7 paper uses representative survey data from Malawi that points’ less new knowledge in math per 100 days, the major- includes unique longitudinal data on individual students ity of which is not explained by increased dropout. Our (grade 4 at baseline), including those who dropped out, at findings are relevant for other low-income and lower-mid- three points in time: pre-COVID; 1–12 months before dle income contexts: remote learning during school closure the seven-month school closures; and 14–20 months after was in general ineffectual, necessitating urgent action to schools reopened. Across math, English, and Chichewa, the remediate lost learning; and children who dropped out had local language, the average learning loss amounts to 18 the highest learning losses and now require out-of-school months (78 points, 0.78 standard deviations), significantly learning opportunities. This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at sasim@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Learning Loss as a Result of COVID-19: Evidence from a Longitudinal Survey in Malawi* † ‡ Salman Asim, Sajitha Bashir, and Ravinder Casley Gera§ * JEL Classification: I21; I28; C59. Key words: Education Quality; Learning; Primary School; Education Policy; Field Experiment; Modelling † Senior Economist, Education Global Practice, World Bank: sasim@worldbank.org. ‡ Global Education Advisor § Education Specialist, Education Global Practice, World Bank Contents Introduction 3 Context 5 Theory of Change 7 Methodology 8 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Data 8 Sample 9 School Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Student Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Identification strategy 11 Results 12 Sub-group analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Discussion 18 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 References 21 Tables 23 Appendices 39 A. Malawi Longitudinal School Survey . . . . . . . . . . . . . . . . . . . . . . . . 39 Instruments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 B. Supplementary Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 2 Introduction School closures associated with COVID-19 pandemic have resulted in substantial loss in learn- ing globally (Asim and Casley Gera, 2024). Moscowicz and Evans (2022) find in a systematic review of 17 countries, that there was consistent learning loss in high-income countries of be- tween 0.05 and 0.17 standard deviations (s.d.) – several months of lost learning. Patrinos et al. (2022) similarly find an average loss of 0.17 s.d. in their review of 36 countries. However, there is limited evidence on the extent of learning loss at the primary level in low- and middle-income countries. In South Africa, Grade 2 students lost between 57 and 70 percent of a year of learn- ing compared to their pre-pandemic peers, and Grade 4 students lost 62 to 81 percent of a year of learning (Ardington et al., 2021). In Uganda, where schools were closed for two full years, surveys suggest that the share of those who could not even read letters of the alphabet doubled in number (Uwezo, 2021; Sandefur, 2022). Evidence from Pakistan suggests that the closure of schools following a major earthquake in 2005 led to between 1.5 and 2 years’ learning loss among primary school age children (Andrabi et al., 2020). In Tamil Nadu, India, panel data suggests that the closure of schools led to initial learning loss of 0.7 s.d. in math and 0.34 s.d. in reading and writing, although around two-thirds of this loss was recovered in the first six months after schools reopened (Singh et al., 2022). In Ethiopia, by contrast, longitudinal data suggests that following the reopening of schools, students’ trajectories of learning were between one-half and one-third of the pre-pandemic pace (Kim et al., 2022). Assessing data from more than 20 countries, Schady et al. (2023) estimates that students lost an average of 6.2 months of learning during the closure of schools. While cohort analyses of student performance can provide broad estimates of COVID-related learning loss, only longitudinal data can enable the specific tracing of the learning trajectories of individual students1 , enabling estimation of learning loss during the closure of schools and the pace of learning following their reopening. In addition, because of high rates of dropout during the pandemic - rising to as high as 35 percent in some cases (Moscowicz and Evans, 2022) - panel studies that use public examination data or only test enrolled students are likely to suffer from selection effects arising from the exclusion of dropped-out students. However, dynamic analysis of learning trajectories of the same student is not possible in many low-income countries for a lack of longitudinal learning data. In this paper, we exploit longitudinal data from a nationally representative survey of schools in Malawi, a low-income country in Eastern Africa, and present analysis of students’ learning profiles before, during and after the closure of 1 See, for example, the Young Lives study, which traces learning outcomes and other indicators for 12,000 children in Ethiopia, India, Peru and Viet Nam. 3 schools. In Malawi the government closed all public schools because of COVID for a total of seven months between March 2020 and February 20212 . The Malawi Longitudinal School Survey conducts regular learning assessments with a nationally representative sample of students who were in Grade 4 of primary school when the first test was administered, between May 2016 and September 2018. Students were retested between November 2018 and February 2020, and tested for a third time between April and December 2021. This means that students were tested twice prior to the pandemic and closure of schools, and once after reopening of schools. Figure 1 summarizes the timeline. Figure 1: Study Timeline Exploiting this and measuring the pace of learning between rounds, adjusted for time and the closure of schools, we can estimate the loss in learning, compared to the expected trajectory of learning without the pandemic. We find that on average, students’ learning in mid-2021 was 78 points (0.78 s.d.) below where we would project if the pandemic had not taken place (on an item response theory scale centered at 500). This is equivalent to around 18 months of lost learning. Exploiting the difference in time between assessments, we then estimate the share of learning loss that occurred during the closure of schools. Immediately after the reopening of schools, students had 48 points (0.48 s.d.) lower learning than expected on the previous trajectory. Adjusting for the lost schooling time, we are able to decompose this loss and find that, in addition to foregone learning because of the closure of schools, students actually experienced a 25 point one-off reduction in knowledge during the closure. Simply put, in the same test 2 This included a one-month period of normal school closure in August 2020. Schools reopened in October 2020 but closed again for a period of four weeks in January-February 2021. 4 administered prior to the closure the same student had performed around half a year better than the post-reopening version of herself. Remote schooling appears to have failed severely in Malawi with students not even retaining the same levels of knowledge as they possessed prior to COVID. Moreover, once students returned to school, their learning did not return to the pre-COVID pace. The learning loss from the closure of schools is less than half the total average learning loss from COVID; the remainder – 30 points – is the result of a slow-down in learning after schools reopened. The level of learning achieved by students in 100 days of schooling is around one-quarter lower following the reopening of schools. This in part reflects the slower learning of students who dropped out as a result of the pandemic, but also reflects slower learning among those students who remained in school. The slowdown is driven by a substantial reduction in the pace of learning in math, where students gain 7.2 points less knowledge per 100 days after the reopening than prior to the closure. Even when excluding students who dropped out, this effect persists, with students who returned to school gaining 5.8 points less knowledge per 100 days than prior to the closure. This suggests that schools have not successfully adjusted their teaching to support students to return to a pre-pandemic pace of learning, let alone to catch up lost learning. If this trend continues, we may see a growing gap in learning over time with some students completely left out while others fall further behind their expected trajectories of learning. Strikingly, girls were most heavily affected by the one-off shock from the closure of schools, but did not experience a significant slowdown in learning once schools reopened. The findings suggest that more research is needed to estimate not only the one-off learning loss from the closure of schools, but the change in the pace of learning in schools after they reopened. There is a need for a concerted focus on remedial learning, responding to the loss of learning experienced by students, if countries are to restore pre-pandemic trajectories of learning – let alone catch up the learning that has been lost. Context Malawi is an extremely poor low-income country and although it has achieved high levels of access to primary education, learning outcomes are poor as a result of overcrowded classrooms3 and a lack of staff in many schools (Asim and Casley Gera, 2024). As of 2020 only 68 percent of students entering primary education remained enrolled until Grade 5 and only 53 percent to 3 The average class in Grade 1 has about 150 students; and in Grade 2, about 125 students (Asim and Casley Gera, 2024). 5 Grade 8. Evidence from the MLSS baseline confirms that learning levels prior to the COVID pandemic were already low: students in Grade 4 obtained an average of only 44 percent in items aligned with the Grade 3-4 curricula in Chichewa 4 . In Malawi schools were closed as a result of COVID from March through October 2020 and again from January-February 2021. Measures were taken to provide distance learning to stu- dents during the closure of schools through a range of methods including radio, TV and online, supported by a grant of US$10 million to the Government of Malawi from the Global Partner- ship for Education. However, the reach of these services appears to have been limited. Data from a household survey collected as part of MLSS in 2022 suggests that 92 percent of house- holds did not access any remote learning materials during the closure of schools (World Bank, 2022). Three-quarters of headteachers said that access to radio (the main modality of distance learning) was a constraint for their students (Kadzamira et al., 2022). Following the reopening of schools, they operated with a number of adjustments to normal operating procedures, including a reduced school day and the use of double shifts to reduce class sizes. The 2019/20 school year was extended by three months to partially account for the closure of schools, running from September 2019 to December 2020, providing approximately seven months’ school time during the academic year. The shortened day was maintained dur- ing the 2020/21 school year; schools returned to a normal day beginning in the 2021/22 school year, which owing to previous delays was reduced in length to only nine months.5 Despite the shortening of the school day and year, the gradewise curricula from the pre-pandemic period were maintained, effectively adopting an accelerated learning approach to deliver three years’ worth of learning in a total of three calendar years despite the seven-month closure of schools. These efforts were supported by dedicated remedial and accelerated learning classes, provided primarily in Grade 8 for students preparing for the high-stakes Primary School Leaving Cer- tificate of Education (PSLCE) examination, but to a varying extent in other grades (UNICEF Malawi, 2022b). Training was provided to all primary school teachers on accelerated and re- medial teaching by Government, with supplemental guidance materials shared with schools (UNICEF Malawi, 2022a). 4 Theequivalent scores for English and math were 32 percent and 52 percent, respectively. 5 The 2020/21 year ran from January – December 2021 with modified school operations. The 2021/22 year runs from January to September 2022 with a normal school day but continuing use of double and overlapping shifts in crowded schools. 6 Theory of Change The COVID pandemic could have affected students’ learning in several ways. 1. Slowdown in learning during the closure of schools. Students are unlikely to have maintained their pre-pandemic learning pace during the closure of schools. Children learn most effectively when information is delivered by a knowledgeable and skilled teacher, with discipline maintained in the classroom, and opportunities available to learn from their peers (National Scientific Council on the Developing Child, 2009). The first of these was generally unavailable during the closure of schools, given the evidence of poor access to distance learning; the second and third are also likely to have been largely unavailable owing to instruction taking place at home where children lack classmates and parents may struggle to maintain discipline. This loss of learning during the closure of schools can further be decomposed into two types: foregone learning, where students do not gain new knowledge because schools are closed; and forgotten learning, where students lose learning they have previously gained (Schady et al., 2023). 2. Increased dropout. Evidence from countries including Ghana, Ethiopia, Liberia, Sene- gal and Pakistan (Moscowicz and Evans, 2022) suggests that COVID has led many stu- dents to drop out of school, as a result of economic pressure on families as well as students simply losing engagement with education during the closure of schools. Dropped-out stu- dents are likely to learn at a slower pace than those still enrolled in school as a result of the same factors that may lead to slow-down in learning during the closure of schools. 3. Changes in learning pace following the reopening of schools. Once schools reopened, accelerated and remedial teaching was intended to produce a rapid pace of learning to ‘catch up’ learning lost during the closure of schools. However, disruptions to the delivery of schooling, such as the reduced school day, as well as ongoing pressures from the pandemic on schools, students and households, may mean that the pace of learning was not be faster following the reopening of schools than it was pre-pandemic – and may even have been slower. Employing the MLSS assessments, and information from associated interviews with tested stu- dents, we are able to decompose these dynamics. Exploiting the fact that students underwent three rounds of testing, two prior to the closure of schools and one several months following the reopening, we are able to measure the change in learning trajectories, adjusting for differences in time between tests and the closure of schools, and estimate the impacts of the pandemic and associated closures on learning. 7 Methodology We estimate trajectories of learning prior to and following the pandemic by comparing baseline and midline test scores, and midline and endline test scores, for the same students (Figure 1). Within each round, the sampled schools are not visited at the same time and therefore, the time between the three rounds varies for each sampled student. To account for this, we control for the length of time the first and second assessment, and second and third assessment, scaled by 100 days for ease of interpretation. We run three different models. • The first model measures the average impact of the one-off COVID shock on students’ test scores after the pandemic, controlling for length of time differences. • The second model incorporates the interaction between the COVID shock and scaled time difference. That helps to divide the average impact of the COVID shock from the previ- ous model into two parts: the one-off shock associated with the closure of schools, and any change in learning pace that students experience following the reopening of schools. The time difference interaction allows us to estimate the change in gradience in learning trajectory before and after the closure of schools. • The third model is similar to the second one, but excludes the time during which schools were closed. That allows to measure the extent of learning loss which came from forget- ting of existing knowledge, rather than foregone learning from the closure of schools. Limitations The model makes a conservative assumption that the learning trajectory of students is linear – every additional year of learning adds as much to learning as the earlier years. It is generally assumed that learning trajectories accelerate parabolically in upper primary for students who remain in school; however, such parabolic trajectories may be less likely in low-income settings. We therefore adopt the conservative assumption, with the result that our projections of students’ learning trajectories in the absence of COVID can be considered lower bound estimates. The resulting estimates of learning loss, derived by comparing actual learning to these projections, can therefore also be considered lower bound estimates. Data We draw on data from the Malawi Longitudinal School Survey (MLSS). Implemented by local research firms and managed by the World Bank in partnership with the Government of Malawi, 8 the MLSS is an independent, nationally representative survey that provides data on students, teachers, schools and school communities. Data is collected through observations and inter- views – for example, observations of school and classroom facilities; observation of lessons and teaching practices; interviews with Head Teachers, teachers, and members of community committees; and interviews and testing of Grade 4 students, as well as a longitudinal sample of students in later rounds. Visits to schools are unannounced in order to obtain a true picture of everyday conditions and practices. We utilize data from the MLSS learning assessments, which are norm referenced tests that cover competencies from Grade 1 through Grade 6 in English, math, and Chichewa (the ma- jority local language and official language of instruction at primary level). Designed by local psychometricians with input from teachers, the MLSS learning assessments are designed to measure student’s mastery of the local curricula and are scaled for comparisons over time. Stu- dent’s scores are converted to knowledge scores using Item Response Theory (IRT) according to the distribution of correct answers across rounds, using a scale centered at 500 with an s.d. of 100. Sample School Sample In this analysis, we utilize data from the 299 schools within the larger MLSS sample which were subject to all three rounds of data collection.6 The full MLSS sample is representative of Malawi’s public schools at the division (sub-region) level; however, not all schools in the sample were included in each of the three MLSS rounds. These 299 schools which were included in all three rounds are all found within 12 districts which were the focus of interventions as part of the Malawi Education Sector Improvement Project (MESIP), a government investment program that operated from 2016-21 for which the MLSS serves as a source of evaluation data.7 Our sample of 299 schools is representative of the entire population of schools in these 12 districts 6 The full MLSS sample, including the nationally representative sample and additional schools included to support impact evaluations, includes 1214 schools. The sample was completed using sampling proportional to size at the division level. For details of the sampling process, see the Appendix. 7 MESIP, financed by the Global Partnership for Education, operated primarily in eight districts: Chikwawa, Dedza, Kasungu, Lilongwe Rural West, Machinga, Mangochi, Mzimba South, and Thyolo. A sister project, MESIP-Extended, conducted the same interventions in a further four districts: Dowa, Mulanje, Nkhotakota, Rumphi. 9 (Table 1 in Tables section).8 Student Sample At each school, a gender-balanced sample of 25 Grade 4 students per school (13 girls and 12 boys) was randomly selected randomly at baseline.9 We then randomly selected 15 of these (8 girls and 7 boys) as the longitudinal sample for retesting at midline and endline. Students who had dropped out or transferred to other schools were traced to the new schools or their homes to complete the learning assessment. Of 4,512 students in the original longitudinal sample, we were able to trace 3,604 at midline (87 percent); of these, we were able to trace 2,832 at endline (79 percent). Table 2 shows the balance on key characteristics between attritors and non-attritors. Attritors are slightly older than non-attritors, and are less likely to have a pre-primary education or a literate parent, and are slightly more likely to have been absent any day in the previous week. In addition, attritors have slightly lower baseline scores in math and English. This suggests that the attriting sample may be expected to have lower longitudinal learning outcomes than the non-attriting sample if reached. Dropped-out students. We exclude students from our analysis who dropped out prior to the closure of schools, and those who dropped out following the reopening of schools for non- COVID-related reasons. To differentiate between “regular” dropouts and those who dropped out because of the pandemic, we match students on several observable covariates using propensity score matching. This method allows us to identify those students who dropped out following the closure of schools and among those, identify those with similar characteristics to those pre- COVID. After excluding these regular dropouts, we are left with a dropout group with distinct characteristics and consider them to have dropped out due to the pandemic. To capture the full impact of the pandemic on learning, we keep these students in the analytical sample. Therefore, our final sample consists of students would have remained in the school absent the pandemic. Of 2,832 students present in all three rounds of MLSS, 802 were in a dropped-out status at the time of endline. Of these, 458 had data available on the year in which they dropped out. For the remaining 344 students, owing to a limitation of the survey instruments, we did not have data 8 750 schools were included in the baseline rounds conducted from May 2016 to September 2018. Of these, 388 underwent midline data collection between November 2018 and February 2020 and 299 underwent endline data collection between April and December 2021. 9 Students were selected randomly from the largest Grade 4 class. 10 on the year in which they dropped out. We imputed their year of dropout using their standard at dropout and the date of interview in which their dropout status was confirmed. Following this process, we found that of the 802 dropped-out students, 488 had dropped out following the closure of schools. This is equivalent to 19 percent of the students still enrolled at the time of the closure. A further 314 had dropped out prior to the closure of schools; these were excluded from the sample. Although this is an extremely high rate of dropout following the closure of schools, not all is attributable to the pandemic. The students who dropped out after 2020 had significantly different characteristics, on average, than those who dropped out prior to 2020. Specifically, they had lower baseline scores on average, were substantially more likely to be male, and more likely to have repeated a grade (see Table D in Appendix B). Using propensity score matching, we then identified 300 students among this group whose characteristics matched those of students who dropped out prior to the closure of schools (see Table E in Appendix B); we treated those as non-pandemic related dropouts and excluded them from the sample. That left 188 students who were identified as having dropped out after 2020 as a result of COVID-19; these were, as expected, substantially more likely to be male, had lower baseline scores, and were more likely to have repeated a class, than those who were found to have dropped out for non-COVID reasons (see Table F in Appendix B). These 188 students were retained in the sample. This left us with a sample for our analysis of 2218 students who had either returned to school, or dropped out as a result of COVID (Table 3). Of these, 1675 remained enrolled at the same school at endline as they had been at baseline; 355 had been tracked to another school; and 188 had dropped out as a result of COVID. This means that of the 19 percent of students dropping out during or after the closure, seven percentage points can be attributed to COVID, which is a lower bound and conservative estimate. Table 4 confirms that this final sample is broadly representative of the larger non-attriting sample on key characteristics. Identification strategy Our estimates are based on multivariate regression analysis with outcome variable defined as the difference between test scores of student (∆Yi ) measured as the difference between scores in two consecutive periods. As not all students were tested on the same date at each time interval – we control for student-specific time interval by normalizing it to 100 calendar days. The interpretation of the coefficient on time difference is the pace at which the student learns every 100 days. The interaction term with COVID dummy gives us the change in pace post-COVID. 11 Therefore, the following three regressions were estimated: ∆Yi = α sc + β1 sc Xi + β2 sc ∆T i + e sci (1) ∆Yi = αlc + β1lc Xi + β2l ∆T i + β3l Xi ∆T i + elci (2) ∆Yi = αln + β1ln Xi + β2l ∆T i + β3l Xi ∆T i + elni (3) where the outcome variable ∆Yi represents a difference in test scores between two points of measurement, Xi is the dummy variable of interest standing for the COVID period, ∆T i stands for time spent between exams, all for each student i. One regression, denoted short s, was estimated without interaction term between the time difference and COVID dummy, and two subsequent regressions denoted long l, were estimated including this interaction term. In addi- tion to that, long regression denoted c contains time variable counting for school closure and the long regression denoted n contain time variable excluding school closure. Our coefficient of interest is β1 and β3 . As can be spotted in the long equations, coefficients, β2 and β3 is not affected by exclusion of school closure time. Only β1 , coefficient identifying COVID shock, would be affected by this change. This is due to fact that school closure does not affect time variables estimated before COVID period. Finally, α represents the constant term and e the error term for each specification. Results Table 5 shows findings from our three models excluding dropouts not as a result of COVID. Overall learning loss. Our first model measures the average impact of the COVID shock on students’ test scores after the pandemic, controlling for time spent between two exams for each student. We estimate the total learning loss from COVID at 78 points (0.78 s.d.) for the average student. This is equivalent to around 1.6 years’ learning. One-off learning shock. The second model incorporates the interaction between the COVID shock and the time spent between the midline and endline learning assessments for each stu- dent. That helps to differentiate the one-off shock in learning from the closure of schools from any change in learning following their reopening. We find that students immediately after the reopening of schools had 48 points (0.48 s.d.) lower learning than would be expected had pre-pandemic rates of learning continued – around one year’s lost learning from the closure of 12 schools alone. The third model enables us to decompose this 48 point static shock in learning into two compo- nents - foregone learning, meaning the loss of learning that would be expected from an absence from school equivalent in time to the closure of schools; and forgotten learning, meaning where students’ knowledge levels have not merely progressed as they would normally, but actually reduced. We find that students returned to schools with 25 points lower learning, on average, than they had prior to the closure - meaning that forgotten learning, rather than merely foregone learning, accounts for the majority of the one-off learning loss. Change in trajectory of learning. Accounting for the one-off learning loss from the closure of schools, there remains 30 points in lost learning unaccounted for. The second model allows for measuring the difference in learning trajectories – the level of learning achieved in 100 days – in schools following the reopening compared to that before the closure. We find that students achieved 14 points of learning per 100 days prior to the closure, but following the reopening, the pace of learning was lower by three points per 100 days. In other words, the level of learning achieved by students in 100 days of schooling is around one-quarter lower following the reopening of schools. This in part reflects the slower learning of students who dropped out as a result of the pandemic, but also reflects slower learning among those students who remained in school. Figure 2 summarizes the findings. The blue line shows how learning would have progressed if the pre-COVID learning trajectory had continued. The red line shows the actual learning trajectory post-COVID, reflecting both the one-off COVID shock and the slowdown in learning following the reopening of schools. The gap between the blue and red lines at the far left shows the one-off impact of the closure of schools.10 The green line shows the learning trajectory post-COVID, adjusted for the 48-point expected loss of learning from the closure. 10 The gap between the red and green lines shows the explained difference in learning (14 points) as a result of the physical time lost from closure of schools. The gap between the green and blue lines on the y-axis shows the unexplained drop in learning (26 points) that may reflect students losing previous mastery over foundation concepts. 13 Figure 2: Learning levels and trajectories pre- and post-COVID closure Learning loss by subject. The dynamics of learning loss are different across subjects. Table 6, Table 7, and Table 8 present the same regression for math, Chichewa, and English scores, respectively. Overall learning loss is greatest for math (Table 6) at 92 points and lowest for Chichewa (Table 7) at 56 points. In Chichewa and English (Table 8), learning loss is dominated by the one-off shock from the closure of schools and the slowdown in learning following the reopening schools is not significant. In math, however, the slowdown in learning following the reopening is large and significant at 7.2 points per 100 days (while the one-off shock is not significant). This suggests that the slowdown in learning observed in total scores is driven by this effect in math. Impact of dropout. As noted, our analysis includes both students who returned to school after the reopening and those who dropped out as a result of COVID-19. To what extent is the slowdown in learning in math after the reopening of schools driven by dropout, and to what extent does it reflect an actual slowdown in learning in schools? Table 9 presents findings for math solely for the 2030 students who returned to school following COVID-19. The slowdown in learning, although smaller than when the COVID-driven dropouts are included, remains large 14 and significant at 5.8 points per 100 days. This confirms that students who returned to school nevertheless learned at a slower pace in math than they had prior to the pandemic. Sub-group analysis Analysis of data from the MLSS baseline reveals that there are large variations in learning levels within students in the same grade in Malawi, with girls, overage students, and those without a literate parent achieve lower learning outcomes on average in Malawi’s primary schools (Asim and Casley Gera, 2022). To explore the differential impacts of COVID on learning, we decom- pose our sample and conduct sub-group analysis by (i) gender; (ii) students’ score at baseline; (iii) age (aged 11, the median age at baseline, or below, versus 12 or older); (iv) household asset holdings; and (v) parental literacy. See Tables 10-15. Gender. Table 10 shows the analysis by gender. The overall learning loss is higher for females than for males at 85 points (versus 72). Strikingly, this takes the form of a large one-off shock in learning of 64 points (versus 31 for boys). The pace of learning was slightly higher for girls than for boys prior to the closure, at 15.4 points per 100 days versus 13.8 for boys, meaning that the loss of schooling during the closure was associated with a greater shock to learning. However, for girls, the slowdown in learning following the reopening of schools is small and statistically insignificant at 2.4 points per 100 days (versus 4.5 for boys), suggesting that girls have been able to substantially resume their previous pace of learning. To what extent do these gender differentials reflect any disparities in dropout? To explore this finding, Table 11 shows the characteristics of the 188 COVID-related dropouts and the 2030 students who remained in school. The dropouts are substantially more likely to be male, with only 28 percent female versus 54 percent of those who remained in school. Dropouts were also older on average. We hypothesize that older male students were pulled into the labor force as a result of economic stress caused by the pandemic and were less likely to return to school as a result. Table 12 presents findings for those students who remained enrolled in school, disaggregated by gender. It confirms that the overall learning loss for females is larger; however, it also demonstrates that the decomposition of learning loss is broadly similar between female and male students who remained enrolled, with a large one-off shock and no significant slowdown in learning. This confirms that the gender differentials observed in Table 10 are driven by differentials in dropout. Score at baseline. There are large variations in learning among students within the same age cohort in Malawi, with students in a single Grade 4 class at the MLSS baseline typically having a gap of two years’ learning between the highest- and lowest-performing students (Asim and 15 Casley Gera, 2024). Table 13 breaks down learning loss by students’ score at baseline. The learning loss for students whose scores were above the 60th percentile at baseline (that is, in the top 40 percent of students) was substantially lower at 40 points (versus 99 for students who were below the 60th percentile at baseline). For students with higher scores at baseline, the learn- ing loss can be entirely attributed to the one-off shock from the closure of schools, while the pace of learning following the reopening is not significantly different from prior to the closure. For students with baseline scores below the 60th percentile, although a large share of the total learning loss comes from the one-off shock (59 points), there remains a large share from slow- down in learning following the reopening of schools (4.4 points per 100 days). This suggests that teachers, who already faced challenges to meet the needs of large classes of varying ability prior to COVID, have struggled to reach the lowest-performing students since the reopening of schools. This may reflect the lack of success of remedial initiatives or the ineffective targeting of these interventions to the neediest students. Age. Table 14 shows the analysis by age. We find that the overall learning loss was slightly higher for students aged 12 or above, with total learning loss (compared to pre-pandemic tra- jectory) of 80 points versus 77 points for the younger students. For older students, the one-off shock was smaller at 31 points, but the slowdown in learning following the reopening was larger at 5.4 points loss per 100 days (versus 57 points one-off shock and 2.1 points loss per 100 days for younger students). Parental literacy. Table 15 shows the analysis by parental literacy.11 We find that the overall learning loss was higher for students who do not live with a literate parent, with a total of 86 points’ loss versus 76 for students who live with at least one. As with older students, students without a literate parent at home experienced a lower one-off shock – 31 points (versus 52 for students with a literate parent) – but a more rapid slowdown in learning following the pandemic, of 6.2 points per 100 days (versus 2.6 points). Figure 3 summarizes the COVID-19 impacts on overall learning trajectories and the sub-group analyses. 11 We measure parental literacy by asking students first whether they live with their mother, father, or both; then by asking if they have seen their mother and/or father reading a newspaper, bible, or book at home. 16 Figure 3: COVID-related learning loss: sub-group analysis 17 Discussion Our findings suggest the following conclusions: Learning loss is large and will worsen unless the pace of learning returns to its pre- pandemic level. The total learning loss experienced by students – an average of 78 points (0.78 s.d.) – is equivalent to 1.6 years’ lost learning. In the Malawian context where learning levels were already low, this is a huge setback for students and for Malawi’s goals of raising overall learning levels and broader levels of human capital. Efforts to support remote learning during the closure of schools appear to have failed. The level of learning loss during the closure of schools in Malawi - with students returning to school with around one year’s less learning than before the closure - suggests that the government’s substantial investments in remote learning during the closure of schools, via radio, TV and online, were not successful. Combined with evidence of challenges of access to even radio, the most widely available of the various remote learning channels, as well as evidence of substantial learning loss in comparable countries such as Uganda, this suggests that such remote learning efforts may not be the most efficient way to maintain learning during school closures for the lowest-income countries. COVID and the closure of schools led to substantial extra dropout, particularly for older boys. Our estimations suggest that 7 percent of those students who were still enrolled prior to the closure of schools dropped out as a result of COVID-19 and the closure, in a conservative estimate. This is in addition to a further 12 percent who dropped out for non-COVID-related reasons. The COVID-driven dropouts are primarily male - more than three-quarters - and older on average than those who remained enrolled, suggesting substantial loss of enrolled students to the labor market. Learning has been slower in math following the reopening of schools, even for students who returned to school. In terms of total scores, around 40 percent - 30 points out of 78 points’ total learning loss - can be attributed to this slowdown in learning. The effect is driven by math where the majority of the total learning loss - 65 out of 93 points’ total loss - can be attributed to this slowdown. Most concerningly, this remains the case even when we look solely at students who returned to school after the reopening, with 42 out of 92 points’ loss stemming from the slowdown in learning. This is in contrast to the findings of Andrabi et al. (2020) that only around 10 percent of total learning loss from the 2005 earthquake in Pakistan and associated closure of schools was a direct result of the closure, the remainder coming from slower learning following reopening. If this trend continues, we may see a growing gap in learning over time with students affected by COVID falling further behind their expected trajectories of learning, 18 even if they have re-enrolled. Learning loss, and the slowdown in learning after schools reopened, is much worse for weaker students. The level of learning loss for students who were in the top 40 percent of scores at baseline was less than half that of students in the lower half of the score distribution. In other words, loss of learning has been much greater among those who were already struggling to reach expected standards of learning for their age and grade. This will exacerbate the already large inequities in learning between students in Malawi’s schools. Girls, older students, and those without a literate parent, all of whom tended to have lower scores before the pandemic, have all suffered a greater level of learning loss than their peers, further demonstrating the equity crisis induced by the pandemic. Recommendations 1. Urgently review curricula and teaching methods to reorient the primary school system around remedial and catch-up learning. The findings suggest that schools have not suc- cessfully adjusted their teaching to support students to catch up lost learning – or even to learn at the same pace as before the disruption of the pandemic. The largely business-as- usual approach adopted by the government – where schools largely continued teaching the regular curriculum following the reopening of schools, at an accelerated pace – does not appear to be fit for purpose in the pandemic context. There is an urgent need for countries to streamline curricula to focus on foundational learning and enable students to re-master core concepts before continuing to more advanced learning. A ‘big push’, with teaching substantially reoriented towards remedial and catch-up learning for an aca- demic term or more, may be needed to address the loss in learning from the closure of schools and enable students to keep up with a pre-pandemic pace of teaching.12 The ap- parent success in Tamil Nadu, where widespread remediation supported rapid regaining of lost learning, particularly by the poorest students (Singh et al., 2022), demonstrates the potential for comprehensive remediation to make catch-up learning a reality. 2. Mobilize communities to urgently re-enroll dropped-out students and retain students in school. The dropout shock from COVID appears to have been large, with an additional seven percent of students dropping out since the closure of schools compared to what would be expected from previous dropout rates, and this is a key contributor to learning 12 Such an approach was recently introduced in Kenya, where the government has announced a two-year acceler- ated catch-up program (World Bank, 2021) 19 loss. Urgent efforts are required to re-enroll students who have not enrolled since the reopening of schools; and to identify and monitor students at risk of dropping out, to ensure that rates of dropout do not remain at an elevated rate. 3. Deepen support for girls, overage students, students without a literate parent, and low- performing students. The disproportionate impact of the pandemic on these groups rep- resents a deepening of an existing crisis of equity in Malawi’s school system. Dedicated efforts to support these groups, and to build school cultures which meet the need of all students, are vital if the blows to learning are to be recouped. 4. Conduct further analysis to decompose learning loss in other countries. Much of the literature on COVID-related learning loss so far has focused on the one-off shock of lost learning during the closure of schools, with an implicit assumption that the pace of learn- ing returned to pre-pandemic levels – or improved on them – following the reopening of schools. Malawi is unlikely to be the only country in which the opposite is the case. Us- ing assessments conducted following the reopening of schools as a starting point, there is an urgent need to develop longitudinal panel assessments to trace students’ learning over time and evaluate the extent to which countries have been able to restore pre-pandemic trajectories of learning. 5. Reconsider investment priorities during future school closures. The apparent failure of the government’s remote learning efforts during the closure of schools suggests that a dif- ferent approach may be required for future pandemics or other extensive school closures. This may include investing more in printed remote learning materials and/or empower- ing local communities to invest in locally identified solutions to advance learning. For example, in India the NGO Pratham adopted an approach of using text messages to sup- port parents to engage children in learning (Pratham, 2022). Direct instruction via mobile phone may be another promising opportunity: in Botswana, provision of math problems by SMS and phone call reduced learning loss by 0.29 s.d., while SMS alone reduced it by 0.16 s.d. (Angrist et al., 2020). In addition, it may be appropriate to rebalance future investment away from attempts to maintain learning during the closure of schools and more heavily towards supporting remedial and accelerated learning following reopening. 20 References Andrabi, T., B. Daniels, and J. Das (2020). "Human Capital Accumulation and Disasters: Evidence from the Pakistan Earthquake of 2005." RISE Working Paper 20/039. https://riseprogramme.org/sites/default/files/2020-11/RISE_WP- 039_Adrabi_Daniels_Das.pdf. Accessed 15th September 2022. Angrist, N., P. Bergman, C. Brewster, and M. Matsheng (2020). "Stemming Learning Loss During the Pandemic: A Rapid Randomized Trial of a Low-Tech Intervention in Botswana”. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3663098. Accessed 12th December 2022. Ardington, C., G. Willis, and J. Kotze (2021). "COVID-19 learning losses: Early grade reading in South Africa". International Journal of Educational Development 86. Asim, S. and R. Casley Gera (2024). "What Matters for Learning in Malawi? Evidence from the Malawi Longitudinal School Survey". Kadzamira, E., J. Mazalale, E. Meke, I. Mwale, F. Jimu, L. Mozcowicz, and J. Rossiter (2022). "What Happened to Student Participation After Two Rounds of School Closures in Malawi—and How Have Schools Responded?". https://www.cgdev.org/blog/what-happened- student-participation-after-two-rounds-school-closures-malawi-and-how-have. Accessed 15th September 2022. Kim, J., P. Rose, D. Tiruneh, R. Sabates, and T. Woldehanna (2022). "Learn- ing Inequalities Widen Following COVID-19 School Closures in Ethiopia.”. https://riseprogramme.org/blog/learning-inequalities-widen-COVID-19-Ethiopia. Accessed 15th September 2022. MoEST (2015). Education Management Information System 2014-15 [database]. Moscowicz, L. and D. Evans (2022). "Learning Loss and Student Dropouts during the COVID- 19 Pandemic: A Review of the Evidence Two Years after Schools Shut Down.” Center for Global Development Working Paper 609. https://www.cgdev.org/sites/default/files/learning- loss-and-student-dropouts-during-covid-19-pandemic-review-evidence-two-years.pdf. Ac- cessed 15th September 2022. National Scientific Council on the Developing Child (2009). "Young children Develop in an Environment of Relationships." Center on the Developing Child Working Pa- per 1. https://developingchild.harvard.edu/wp-content/uploads/2004/04/Young-Children- Develop-in-an-Environment-of-Relationships.pdf. Accessed 1st December 2022. 21 Patrinos, H., E. Vegas, and R. Carter-Rau (2022). "An Analysis of COVID- 19 Student Learning Loss." World Bank Policy Research Working Paper 10033. https://documents1.worldbank.org/curated/en/099720405042223104/pdf/ IDU00f3f0ca808cde0497e0b88c01fa07f15bef0.pdf. Accessed 15th September 2022. Pratham (2022). "Lessons from the Pandemic". https://prathamorg.blob.core.windows.net/data/ Pratham%20during%20Pandemic%20Lessons%20from%20the%20community.pdf. Ac- cessed 15th September 2022. Sandefur, J. (2022). "Uganda’s Record-Breaking Two-Year School Closure Led to. . . No De- cline in the Number of Kids Who Can Read?". https://www.cgdev.org/blog/ugandas-record- breaking-two-year-school-closure-led-to-no-decline-number-kids-who-can-read. Accessed 15th September 2022. Schady, N., A. Holla, S. Sabarwal, J. Silva, and A. Chang (2023). Collapse and Recovery: How The Covid-19 Pandemic Eroded Human Capital And What To Do About It. forthcoming. Singh, A., M. Romero, and K. Muralidharan (2022). "COVID-19 Learn- ing Loss and Recovery: Panel Data Evidence from India.” RISE Work- ing Paper 22/112. https://riseprogramme.org/sites/default/files/2022-09/COVID- 19_Learning_Loss_Recovery_Panel_Data_Evidence_India.pdf. Accessed 15th October 2022. UNICEF Malawi (2022a). email correspondence. UNICEF Malawi (2022b). "Malawi Promising Practices – Recovering Learning Losses through Remedial Teaching and Learning”. Mimeo. Uwezo (2021). "Are Our Children Learning? Illuminating the Covid-19 Learn- ing Losses and Gains In Uganda." Uwezo National Learning Assessment Re- port, 2021. https://uwezouganda.org/wp-content/uploads/2022/01/Uwezo-2021Report- FINAL_Version.pdf. Accessed 15th September 2022. World Bank (2021). Accelerating Learning Recovery. https://thedocs.worldbank .org/en/doc/75bdb5f2c03f19f0642db1c941193f8d-0140042021/related/Recovery-updated- 09-27-2021.pdf. Accessed February 15, 2023. World Bank (2022). "MLSS Household Survey Data”. https://riseprogramme.org/blog/learning-inequalities-widen-COVID-19-Ethiopia. Accessed 15th September 2022. 22 Tables Table 1: Balance Check: School Sample v. 12 Districts Schools (1) (2) T-test Schools in Study Sample All schools in 12 districts P-value Variable N Mean/SE N Mean/SE (1)-(2) PqTR 296 99.904 2652 99.821 0.985 (4.311) (1.058) PqTR in Lower Primary 296 141.044 2627 138.435 0.728 (7.266) (1.896) PCR 296 130.450 2641 120.304 0.174 (7.313) (1.548) PCR Lower Primary 287 194.247 2586 157.669 0.244 (31.153) (4.029) Repetition Rate 296 0.243 2602 0.244 0.834 (0.007) (0.002) Repetition Rate Lower Pri- 296 0.249 2599 0.256 0.329 mary (0.007) (0.002) Dropout Rate 296 0.046 2602 0.048 0.620 (0.004) (0.001) Dropout Rate Lower Primary 296 0.042 2599 0.045 0.428 (0.004) (0.001) Total Enrolment (C1) 297 985.185 2655 869.460 0.019** (48.094) (11.770) Data source: EMIS 2016. Notes: The value displayed for t-tests are p-values. Standard errors are robust. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level. 23 Table 2: Students in MLSS Baseline: Non-Attritors vs Attritors between BL, ML and EL (1) (2) T-test Non-attritor Attritor P-value Variable N Mean/SE N Mean/SE (1)-(2) Student Age 2818 11.302 1667 11.806 0.000*** (0.029) (0.041) Student SES Index 2832 -0.030 1680 -0.259 0.000*** (0.030) (0.038) Student is female 2832 0.535 1680 0.519 0.290 (0.009) (0.012) Student has one or more years 2832 0.502 1680 0.467 0.022** of pre-primary education (0.009) (0.012) Number of times student has 2832 1.093 1680 1.120 0.365 repeated any grade (Std 1-4) (0.019) (0.024) Student has at least one par- 2832 0.790 1679 0.732 0.000*** ents literate (0.008) (0.011) Student speaks Chichewa at 2830 0.757 1679 0.773 0.224 home (0.008) (0.010) Student receives help from 2832 0.598 1680 0.576 0.154 parents on homework at home (0.009) (0.012) Student is absent at any day 2832 0.298 1677 0.338 0.006*** last week (0.009) (0.012) English Knowledge Score 2832 491.680 1680 484.067 0.041** (2.352) (2.889) Math Knowledge Score 2832 495.137 1680 484.302 0.009*** (2.569) (3.289) Chichewa Knowledge Score 2832 491.938 1680 487.110 0.259 (2.625) (3.372) Overall Knowledge Score 2832 492.918 1680 485.160 0.027** across subjects (2.179) (2.747) Notes: Non-attritors are students observed in baseline and subsequently in midline and endline. Attritors are students observed only in baseline. The value displayed for t-tests are p-values. Standard errors are robust. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level. 24 Table 3: Student sample Type Maintained in sample Dropped from sample Still enrolled at original school at endline 1675 - Enrolled at different school at endline 355 - Dropped out prior to 2020 - 314 Dropped out during/after 2020 - 300 for non-COVID-related reasons Dropped out during/after 2020 188 - for COVID-related reasons Total analysis sample 2218 - 25 Table 4: Students in MLSS Baseline: Final sample vs Non-Attritors in wider sample (1) (2) T-test Final sample (n=2218) Wider non-attrited sample (n=2832) P-value Variable N Mean/SE N Mean/SE (1)-(2) Student Age 2208 10.982 2818 11.302 0.000*** (0.031) (0.029) Student SES Index 2218 0.101 2832 -0.030 0.004*** (0.034) (0.030) Student is female 2218 0.516 2832 0.535 0.168 (0.011) (0.009) Student has one or more years 2218 0.523 2832 0.502 0.139 of pre-primary education (0.011) (0.009) Number of times student has 2218 1.036 2832 1.093 0.041** repeated any grade (Std 1-4) (0.020) (0.019) Student has at least one par- 2218 0.801 2832 0.790 0.324 ents literate (0.008) (0.008) Student speaks Chichewa at 2216 0.745 2830 0.757 0.303 home (0.009) (0.008) Student receives help from 2218 0.607 2832 0.598 0.494 parents on homework at home (0.010) (0.009) Student is absent at any day 2218 0.283 2832 0.298 0.236 last week (0.010) (0.009) English Knowledge Score 2218 495.592 2832 491.680 0.274 (2.697) (2.352) Math Knowledge Score 2218 495.622 2832 495.137 0.901 (2.943) (2.569) Chichewa Knowledge Score 2218 492.101 2832 491.938 0.967 (2.973) (2.625) Overall Knowledge Score 2218 494.438 2832 492.918 0.646 across subjects (2.490) (2.179) Notes: The value displayed for t-tests are p-values. Standard errors are robust. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level. 26 Table 5: Total Scores vs School Exposure (1) (2) (3) Model 1 Model 2 Model 3 b/se b/se b/se Covid -78.528*** -48.405*** -25.165** (2.83) (13.65) (10.80) Time 14.358*** 14.616*** (0.44) (0.46) Time & Covid -3.334** (1.52) Time 14.616*** (Excluding Closure) (0.46) Time & Covid -3.334** (Excluding Closure) (1.52) Observations 4436.000 4436.000 4436.000 R-sqr 0.224 0.224 0.224 Data Source: MLSS 2016/18/19/20/21. Note: Scores are for students from Longitudinal sample. Excludes 614 dropouts from before closure of schools, or from after reopening of schools but not associated with COVID-19. Two observations per student (comparison of baseline-midline and midline-endline). Standard Errors are in parenthesis. Model 1 implies an overall covid effect. Model 2 divides shock between static shock and change in learning pace. Model 3 measures the learning pace in periods that schools were open. * p<0.10, ** p<0.05, *** p<0.01 27 Table 6: Math Scores vs School Exposure (1) (2) (3) Model 1 Model 2 Model 3 b/se b/se b/se Covid -92.935*** -27.920 -7.860 (3.64) (18.05) (14.27) Time 16.376*** 16.933*** (0.56) (0.59) Time & Covid -7.196*** (2.00) Time 16.933*** (Excluding Closure) (0.59) Time & Covid -7.196*** (Excluding Closure) (2.00) Observations 4436.000 4436.000 4436.000 R-sqr 0.193 0.195 0.195 Data Source: MLSS 2016/18/19/20/21. Note: Standard Errors are in parenthesis. See Tables 2 and 5 for details of the sample and models. * p<0.10, ** p<0.05, *** p<0.01 28 Table 7: Chichewa Scores vs School Exposure (1) (2) (3) Model 1 Model 2 Model 3 b/se b/se b/se Covid -56.436*** -44.603** -22.837 (3.96) (18.93) (15.02) Time 11.774*** 11.876*** (0.63) (0.66) Time & Covid -1.310 (2.11) Time 11.876*** (Excluding Closure) (0.66) Time & Covid -1.310 (Excluding Closure) (2.11) Observations 4436.000 4436.000 4436.000 R-sqr 0.080 0.080 0.080 Data Source: MLSS 2016/18/19/20/21. Note: Standard Errors are in parenthesis. See Tables 2 and 5 for details of the sample and models. * p<0.10, ** p<0.05, *** p<0.01 29 Table 8: English Scores vs School Exposure (1) (2) (3) Model 1 Model 2 Model 3 b/se b/se b/se Covid -86.082*** -75.211*** -46.502*** (3.68) (18.64) (14.78) Time 15.046*** 15.140*** (0.58) (0.61) Time & Covid -1.203 (2.05) Time 15.140*** (Excluding Closure) (0.61) Time & Covid -1.203 (Excluding Closure) (2.05) Observations 4436.000 4436.000 4436.000 R-sqr 0.161 0.161 0.161 Data Source: MLSS 2016/18/19/20/21. Note: Standard Errors are in parenthesis. See Tables 2 and 5 for details of the sample and models. * p<0.10, ** p<0.05, *** p<0.01 30 Table 9: Math Scores vs School Exposure (only enrolled students) (1) (2) (3) Model 1 Model 2 Model 3 b/se b/se b/se Covid -91.780*** -39.480** -15.172 (3.72) (18.17) (14.38) Time 17.143*** 17.589*** (0.58) (0.60) Time & Covid -5.789*** (2.02) Time 17.589*** (Excluding Closure) (0.60) Time & Covid -5.789*** (Excluding Closure) (2.02) Observations 4060.000 4060.000 4060.000 R-sqr 0.206 0.208 0.208 Data Source: MLSS 2016/18/19/20/21. Excludes all (802) dropouts. Note: Standard Errors are in parenthesis. See Tables 2 and 5 for details of the sample and models. * p<0.10, ** p<0.05, *** p<0.01 31 Table 10: Total Scores vs School Exposure by Gender Female Male (1) (2) (3) (4) (5) (6) Model 1 Model 2 Model 3 Model 1 Model 2 Model 3 b/se b/se b/se b/se b/se b/se Covid -84.878*** -63.580*** -36.675*** -72.076*** -31.729 -12.479 (3.86) (17.36) (13.80) (4.13) (21.20) (16.70) Time 15.234*** 15.414*** 13.467*** 13.818*** (0.62) (0.66) (0.61) (0.63) Time & Covid -2.353 -4.473* (1.94) (2.35) Time 15.414*** 13.818*** (Excluding Closure) (0.66) (0.63) Time & Covid -2.353 -4.473* (Excluding Closure) (1.94) (2.35) Observations 2288.000 2288.000 2288.000 2148.000 2148.000 2148.000 R-sqr 0.255 0.256 0.256 0.194 0.195 0.195 Data Source: MLSS 2016/18/19/20/21. Note: Standard Errors are in parenthesis. See Tables 2 and 5 for details of sample and models. * p<0.10, ** p<0.05, *** p<0.01 32 Table 11: COVID-related dropouts vs still enrolled students (1) (2) T-test 0 1 P-value Variable N Mean/SE N Mean/SE (1)-(2) Total Score at BL 188 341.597 2030 402.420 0.000*** (5.576) (1.938) Student Age at BL 188 12.266 2030 10.872 0.000*** (0.105) (0.030) Student is a female 188 0.277 2030 0.538 0.000*** (0.033) (0.011) Student above median SES 188 0.559 2030 0.539 0.606 (0.036) (0.011) Student having at least one 188 0.755 2030 0.809 0.077* parent literate (0.031) (0.009) Student speaking Chichewa 188 0.660 2030 0.752 0.006*** at home (0.035) (0.010) Student absent at any day last 188 0.324 2030 0.279 0.189 week (0.034) (0.010) Student repeated any class at 188 0.926 2030 0.675 0.000*** least once (0.019) (0.010) Notes: Data Source: MLSS 2016/18. The value displayed for t-tests are p-values. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level. 33 Table 12: Total Scores vs School Exposure by Gender (only enrolled students) Female Male (1) (2) (3) (4) (5) (6) Model 1 Model 2 Model 3 Model 1 Model 2 Model 3 b/se b/se b/se b/se b/se b/se Covid -83.400*** -68.377*** -39.581*** -71.725*** -54.875*** -28.129* (3.83) (17.42) (13.85) (4.17) (19.30) (15.25) Time 15.513*** 15.638*** 14.706*** 14.853*** (0.62) (0.65) (0.63) (0.67) Time & Covid -1.660 -1.869 (1.95) (2.16) Time 15.638*** 14.853*** (Excluding Closure) (0.65) (0.67) Time & Covid -1.660 -1.869 (Excluding Closure) (1.95) (2.16) Observations 2288.000 2288.000 2288.000 2148.000 2148.000 2148.000 R-sqr 0.255 0.256 0.256 0.194 0.195 0.195 Data Source: MLSS 2016/18/19/20/21. Note: Standard Errors are in parenthesis. Excludes all (802) dropouts. See Tables 2 and 5 for details of sample and models. * p<0.10, ** p<0.05, *** p<0.01 34 Table 13: Total Scores vs School Exposure by Baseline Scores Above 60th percentile Below 60th percentile (1) (2) (3) (4) (5) (6) Model 1 Model 2 Model 3 Model 1 Model 2 Model 3 b/se b/se b/se b/se b/se b/se Covid -39.895*** -41.877** -14.928 -99.347*** -59.469*** -38.623** (4.06) (17.71) (13.96) (3.61) (19.08) (15.11) Time 12.879*** 12.863*** 14.172*** 14.540*** (0.57) (0.60) (0.59) (0.62) Time & Covid 0.219 -4.420** (1.98) (2.12) Time 12.863*** 14.540*** (Excluding Closure) (0.60) (0.62) Time & Covid 0.219 -4.420** (Excluding Closure) (1.98) (2.12) Observations 1716.000 1716.000 1716.000 2720.000 2720.000 2720.000 R-sqr 0.221 0.221 0.221 0.252 0.253 0.253 Data Source: MLSS 2016/18/19/20/21. Note: Standard Errors are in parenthesis. See Tables 2 and 5 for details of sample and models. * p<0.10, ** p<0.05, *** p<0.01 35 Table 14: Total Scores vs School Exposure by Age at Baseline Less than 11 More than 11 (1) (2) (3) (4) (5) (6) Model 1 Model 2 Model 3 Model 1 Model 2 Model 3 b/se b/se b/se b/se b/se b/se Covid -77.195*** -57.758*** -31.311** -80.049*** -31.243 -14.041 (3.44) (16.48) (13.09) (4.85) (23.82) (18.74) Time 14.817*** 14.987*** 13.356*** 13.762*** (0.55) (0.58) (0.72) (0.74) Time & Covid -2.149 -5.412** (1.83) (2.67) Time 14.987*** 13.762*** (Excluding Closure) (0.58) (0.74) Time & Covid -2.149 -5.412** (Excluding Closure) (1.83) (2.67) Observations 2832.000 2832.000 2832.000 1604.000 1604.000 1604.000 R-sqr 0.237 0.237 0.237 0.203 0.205 0.205 Data Source: MLSS 2016/18/19/20/21. Note: Standard Errors are in parenthesis. See Tables 2 and 5 for details of sample and models. * p<0.10, ** p<0.05, *** p<0.01 36 Table 15: Total Scores vs School Exposure by Baseline Parental Literacy At least one literate No literate parent at home parent at home (1) (2) (3) (4) (5) (6) Model 1 Model 2 Model 3 Model 1 Model 2 Model 3 b/se b/se b/se b/se b/se b/se Covid -76.486*** -52.307*** -27.721** -86.565*** -31.167 -14.032 (3.12) (15.18) (12.02) (6.62) (31.94) (25.13) Time 14.401*** 14.604*** 14.001*** 14.522*** (0.48) (0.50) (1.03) (1.08) Time & Covid -2.668 -6.203* (1.68) (3.64) Time 14.604*** 14.522*** (Excluding Closure) (0.50) (1.08) Time & Covid -2.668 -6.203* (Excluding Closure) (1.68) (3.64) Observations 3568.000 3568.000 3568.000 868.000 868.000 868.000 R-sqr 0.229 0.229 0.229 0.208 0.210 0.210 Data Source: MLSS 2016/18/19/20/21. Note: Standard Errors are in parenthesis. See Tables 2 and 5 for details of sample and models. * p<0.10, ** p<0.05, *** p<0.01 37 Table 16: Total Scores vs School Exposure by Household Asset Holdings Above 60th percentile Below 60th percentile (1) (2) (3) (4) (5) (6) Model 1 Model 2 Model 3 Model 1 Model 2 Model 3 b/se b/se b/se b/se b/se b/se Covid -72.237*** -75.564*** -44.173*** -83.305*** -28.931 -11.951 (4.13) (21.48) (16.98) (3.85) (17.89) (14.18) Time 14.897*** 14.870*** 13.768*** 14.263*** (0.62) (0.64) (0.61) (0.65) Time & Covid 0.368 -6.020*** (2.35) (2.01) Time 14.870*** 14.263*** (Excluding Clo- (0.64) (0.65) sure) Time & Covid 0.368 -6.020*** (Excluding Clo- (2.35) (2.01) sure) Observations 1942.000 1942.000 1942.000 2494.000 2494.000 2494.000 R-sqr 0.241 0.241 0.241 0.214 0.217 0.217 Data Source: MLSS 2016/18/19/20/21. Note: Standard Errors are in parenthesis. See Table 2 for details of sample and models. * p<0.10, ** p<0.05, *** p<0.01 38 Appendices A. Malawi Longitudinal School Survey The Malawi Longitudinal School Survey (MLSS) collects extensive data on school, classrooms, teachers, Grade 4 students, community members and parents. This Appendix provides summary details. For additional details, see Supplementary Data. Instruments The survey contains the following instruments: 1. Observation of school and classroom facilities 2. Lesson observation 3. Head Teacher interview, including details of teachers and committee members; informa- tion about their background and procedures; and school information from records 4. Student interview 5. Student learning assessment 6. Teacher interview 7. Teacher knowledge assessment 8. Community interviews with members of the School Management Committee, Parent- Teacher Association Executive Committee, and Mother Group 9. Group Village Headman interview All instruments were included in all rounds, except the Group Village Headman interview, which was not included in the 2016 phase of baseline.13 The MLSS instruments are based on similar tools used as part of the Service Delivery Indicators (SDI) survey implemented by the World Bank. The SDI instruments were adapted with addi- tional indicators which were appropriate for the Malawian context and/or specific to the MESIP 13 A Group Village Headman is an intermediary-level official in Malawi’s Traditional Authority structure, broadly analogous to a Village Chief 39 program and related impact evaluations. Learning assessments: MLSS includes learning assessments in English, Chichewa, and math. These subjects not only allow for capturing of students’ literacy and numeracy skills but also allow to test on wide range of cognitive skills. The curriculum of these subjects is relatively standardized across schools. The inclusion of math and English also makes the test comparable with other standardized international tests. The assessments are targeted to Grade 4 students but contain items aligned with the curricula for Grades 1-6. The test was designed by a psycho- metrician in collaboration with experts who had prior experience in designing similar tests and were familiar with the primary school syllabus of Malawi. Teachers in government schools were also consulted during the design and formulation of the test items so as to keep the structure of questions as close as possible to textbooks. Items were developed in reference to interna- tional tests and adapted for Malawian context. Student percentage scores are converted to a mean-centered scale, centered at 500, using Item Response Theory. Sampling The sampling frame for the MLSS was derived from the most up-to-date list of schools available prior to baseline, the 2015 EMIS (MoEST, 2015).14 Twenty percent of urban schools that were mainly concentrated in the four major cities of the country (i.e. Blantyre, Lilongwe, Mzuzu and Zomba cities) were randomly selected. For rural schools, a stratified probability proportional to size (PPS) sampling was used, with strata defined based on the six educational divisions. From each stratum, a random sample of schools was selected using PPS, using the number of schools in each stratum as measure of size15 . At the next stage, for districts that had few schools selected using the first round of PPS sampling, random oversampling was conducted to increase the final number of schools in each district to about 24. This oversampling allows district specific analysis. The urban and the rural samples were then combined to form the final survey sample of 924 schools. As the MLSS is a longitudinal study, it employs both a cohort and longitudinal sample for students. A gender-balanced random sample of 25 Grade 4 students per school (13 girls and 14 The original sample frame contained 5,738 schools with identifier variables such as division, district and zone. 323 private schools were removed from the sample frame, which leaves the frame with 5,415 primary schools subordinated to government or religious agencies. 15 Number of schools is used as a measure of stratum size instead of enrollment as the EMIS enrollment data was found unreliable in many instances. 40 12 boys) is selected at baseline. We then select 15 of these at random (8 girls and 7 boys) for resurvey at endline. Students who have dropped out or transferred to other schools are traced to the new schools or their homes and complete learning assessment and a modified version of the student interview. Two students are additionally selected as a reserve sample to replace students who have died, left Malawi, or cannot be traced. In addition, a new cohort of 15 Grade 4 students is surveyed at endline. For teachers, a primarily longitudinal sample is used. Ten teachers per school are selected at baseline, using a protocol which ensures representation of lower and upper primary and of female and male teachers while maintaining random selection. All of the sampled teachers are resurveyed at endline if eligible. Teachers who have transferred to new schools are tracked and administered a modified version of the interview, while those who have died, left Malawi, left teaching, or cannot be traced are replaced with teachers who have more recently joined the school or were not selected at baseline. B. Supplementary Data Supplementary data associated with this article can be found in the Online Annex at: https://bit.ly/Malawi_learning_loss. 41