Policy Research Working Paper 11144 Effects of COVID-19 on Student Learning Assessing Learning Losses Using Adaptive Technology Juan Barón José Mola Astrid Camille Pineda Paola Patricia Polanco Santos Education Global Department June 2025 Policy Research Working Paper 11144 Abstract This paper quantifies learning losses between 2020 and show no evidence of learning losses in our analysis sample. 2022 in the Dominican Republic, an upper-middle-income However, the paper documents concerningly low learning country. The paper uses data from a sample of ninth-grade levels, with the average student mastering only 45 percent students who benefited from computer adaptive learning of pre-requisite topics for their grade. These results should software during this period. This study is among a few be interpreted with caution, as they are based on a select to measure actual losses among secondary school stu- sample of urban schools and may not fully reflect broader dents, and it is the first to use detailed data on students’ educational trends across the country. mastery of individual math topics to do so. The findings This paper is a product of the Education Global Department. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at jbaron@worldbank.org, apineda@worldbank.org, jmolaa@uninorte.edu.co, and ppolancosantos@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Effects of COVID-19 on Student Learning: Assessing Learning Losses Using Adaptive Technology* Juan Barón,a José Mola,b Astrid Camille Pineda,a Paola Patricia Polanco Santosa a World Bank, Washington, DC, USA b Universidad del Norte, Barranquilla, Colombia Keywords: Learning assessment, edtech, education in emergencies, distance learning, education quality, JEL codes: I21, I24, I26 ∗ We thank Monica Yanez-Pagans, Katia Herrera-Sosa and Ancell Scheker for their support in making this work possible. Diego Luna-Bazaldua, Harry Patrinos and Halsey Rogers provided valuable feedback. The views expressed here are those of the authors and do not necessarily reflect the views of the World Bank. 1. Introduction As a result of the COVID-19 pandemic, schools in the Dominican Republic (DR) were ordered to close in March 2020. With school shutdowns, in-person classes were suspended for the remaining 2019-20 school year and most of the 2020-21 school year. During this period, the Ministry of Education of the Dominican Republic (MINERD) implemented a remote learning education strategy, which included the distribution of booklets targeting a prioritized curriculum for each grade, and the implementation of classes on TV and radio. A recent report found that the share of students who watched classes on TV fluctuated from 62 percent to 25 percent between November 2020 and April 2021 (World Bank, 2022). Moreover, students who watched these lessons did so for less than 12 percent of the duration of the lesson. These numbers suggest that efforts to provide remote instruction during school closures may not have been effective in minimizing learning disruptions throughout this period. In September 2021, 18 months after the first closure, schools in the DR resumed in-person instruction. Nonetheless, the lack of in-person instruction for most of two academic years is expected to have had negative consequences for the younger generations. Until recently most studies measuring learning losses had been based on ex-ante simulations (World Bank, 2021; Angrist et al., 2021; Kaffenberger, 2021). However, in the past two years, several studies have focused on measuring actual losses using data collected after school closures (e.g., Gajderowski et al., 2022; Lichand and Doria, 2022; Hevia et al., 2022; Engzell et al., 2022, Jakubowski et al., 2023; Jakubowski et al., 2024; Iqbal and Patrinos, 2023). A recent meta-study summarizing 40 empirical studies finds evidence of learning losses for most countries analyzed, although for low- and middle-income countries, the evidence is mixed (Moscoviz and Evans, 2022), with two studies having null or positive findings (Uwezo, 2021; Crawfurd et al, 2021). 1 A more recent paper, looking at the case of Uzbekistan, finds no evidence of learning losses when comparing 2022 results in a national assessment to a projection based on 2019 results (Iqbal and Patrinos, 2023). In this report, we aim to quantify learning losses between 2020 and 2022 in the Dominican Republic, an upper-middle-income country, using data from a sample of 9th-grade students who benefited from computer adaptive learning (CAL) software in 2020 and 2022. Among studies that have measured learning losses using data collected after school closures, most have focused on the primary level (Singh et al., 2022; Adeniran et al., 2022; Engzel et al., 2021; Moscoviz and Evans, 2022; Jakubowski et al., 2023). This is one of few studies to measure actual losses among secondary school students in a developing country, and the first one to use rich data on students’ mastery of individual math topics to do so. 2 1 An earlier review found similar results (Donnelly and Patrinos, 2021). 2 Two other studies have measured learning losses among high school-aged students in developing countries. In the first one, Lichand and Doria (2022) find that by the end of 2021, students in Rio de Janeiro were 55 percent behind what they would have learned under normal circumstances over the two previous years. In the second one, Hevia et al. (2022) find evidence of losses in fundamental learning amounting to 0.34-0.45 SD in reading and 0.62-0.82 SD in math, in the Mexican states of Campeche and Yucatán. Both studies used short tests consisting of 10 or fewer items to measure performance Our findings suggest that overall learning levels in the DR are concerningly low, with the average 9th-grade student mastering only 45 percent of prerequisite topics for their grade in 2022. When considering all topics included in the software, mastery levels for the average student in 2022 fall to 14 percent. Nonetheless, learning levels remain stable compared to 2020, and we find no significant evidence of learning losses in this context. We consider two potential explanations for these findings: 1) changes in the composition of students taking the tests, and 2) the widespread implementation of remedial programs during the beginning of the 2021-2022 school year. Because the sample of schools included in this analysis is not representative of the average school in the country, these results should be interpreted with caution. For example, participant schools benefitted from better infrastructure, were more likely to offer an extended school day, and scored higher on a socioeconomic index. Considering these differences between schools in our sample and the average school in the country, our results should only be extrapolated to students in schools that fit this profile. Nonetheless, the main takeaways from this analysis are consistent with performance trends observed in representative national and international assessments among Dominican students in similar age cohorts. Between 2019 and 2022, there were no statistically significant differences in the average performance of 9th-grade students on national assessments (MINERD, 2023a). When comparing PISA results in 2018 and 2022, the Dominican Republic is one of only four countries whose average performance in math, reading, and science increased during this period. However, the proportion of students scoring below a baseline level of proficiency did not change significantly (OECD, 2023). 1.1 Computer adaptive learning (CAL) in the Dominican Republic The Ministry of Education of the Dominican Republic (MINERD), with the support of the World Bank (WB), implemented Programate in 2019 and 2020 in a sample of public secondary schools in the country. This initiative sought to improve math learning among 9th-grade students using adaptive learning software. The software used evaluates students’ initial learning level, walks them through math exercises targeted to their level, provides feedback, and then reassesses students’ learning. Through this cycle, the software ensures that the math content provided to students is appropriate for their level while generating high-frequency data on students’ progress on more than 600 math topics. A small pilot was implemented in 2018 to identify the necessary conditions for schools to benefit from this type of technology and to assess the Ministry’s capacity to implement it at scale. This pilot was implemented in 5 public schools and benefited approximately 400 6th-grade students. The intervention was associated with improvements in math learning, as measured by the platform’s share of mastered topics (Baron et al., 2018). Students who spent more time using the software performed better three months after the start of the intervention. However, insufficient computers and lack of internet connectivity were recurring challenges during implementation. In the following years (2019 and 2020), the intervention was implemented in a larger sample of public secondary schools selected by the Ministry of Education. According to the Ministry’s records, these schools had been verified to have internet access and computer equipment, thus alleviating the two main challenges identified in 2018. Between 2019 and 2020, more than 7,000 9th-grade students benefitted from the software in 62 schools (see Table I). However, the 2020 intervention was interrupted shortly after it started, as schools closed in mid- March due to COVID-19. All students participating in Programate were instructed to use the platform for at least 90 minutes per week, as part of extra-curricular activities. At school, students worked on the platform with the support of their math and ICT teachers. They were also encouraged to use the platform on their own at home. Preliminary results from a quasi-experimental evaluation suggest that the intervention had positive results on students’ performance in end-of-year standardized assessments (Baron et al., unpublished). As part of a more recent effort to measure learning losses after COVID-19, the Ministry of Education, with the support of the World Bank, distributed 5,500 licenses of the ALEKS CAL software among 45 schools in 2022. The sample of schools selected had participated in Programate for at least one academic term between 2018 and 2020. Approximately 80 percent of 9th-grade students in these schools completed an initial diagnostic assessment administered by the platform in 2020 and 82 percent completed it in 2022 (see Table I). In this analysis, we combine results from this assessment with 2020 results to measure changes in student learning among 9th-grade students before and after school closures. We limit the analysis sample to schools that participated in both years to improve comparability. 3 Table I. Students and schools benefitting from CAL Participating Students who completed Year Schools Grade students the diagnostic test 2018 5 6th 418 392 (94%) 2019 51 9th 5,812 4,744 (82%) 2020 33 9th 3,087 2,467 (80%) 2022 45 9th 5,558 4,557 (82%) 2020* 24 9th 2,323 1,798 (77%) 2022* 24 9th 2,502 2,270 (91%) Source: Authors’ own elaboration based on implementation records. Note: The 4th column accounts for the 9th-grade student population in participating schools. The 5th column accounts for students who completed the software’s diagnostic test, the first of the activities presented by the software. *This is the analysis sample used in this study, as we only keep schools that participated in both 2020 and 2022 for the analysis. ALEKS, developed by McGraw Hill, was the adaptive learning platform implemented as part of Programate and this more recent effort to measure learning losses. The platform uses predictive algorithms to offer students personalized learning through adaptive assignments and assessments. 4 These assessments are used to identify the math topics students know, do not know, and/or are ready to learn, based on a field of study known as Knowledge Space Theory (KST). According to KST, the knowledge domain of a subject can be represented by a particular subset of questions or problems that the subject can solve (Doignon and Falmagne, 1985; Falmagne et al., 1990). In this sense, ALEKS determines students’ mastery of a topic based on their ability to solve problems that are representative of that topic. 3 Our conclusions do not change when we replicate the main analysis using the full sample (see Appendix 5.3). 4 Retrieved from ALEKS’s Quick Start Guide: https://www.aleks.com/highered/math/New_IM_HE_Math_Quick_Start_Guide.pdf. When students log into ALEKS for the first time, they complete an Initial Knowledge Check (also referenced here as “diagnostic assessment”). This Initial Knowledge Check has between 11 and 45 items. 5 Each question is selected using students’ answers to previous questions. Based on students’ performance on this Initial Check, the platform predicts which of the 612 topics in the platform are mastered by the student and which are not. The same ALEKS course, with the same list of topics, was used during both years. 6 After completing the Initial Check, students then go on to practice the topics they are most ready to learn, a list of topics that is regularly updated as students acquire mastery in new topics. In this analysis, we use students’ share of mastered topics, as identified in the Initial Knowledge Check, to assess changes in math performance between 2020 and 2022. 7 We focus on examining data from the platform’s diagnostic assessment only, as we believe this provides a “baseline” estimate of students’ math knowledge before any impact the platform may have had on students’ performance. 2. Data and Methodology To examine student performance in math in a post-COVID-19 context, we leverage detailed data on students’ mastery of math topics from the ALEKS diagnostic assessments. This data includes information on students’ proficiency in 612 math topics, classifying each topic as “mastered”, “ready to learn”, or “remaining”. 8 We also observe student performance at more aggregate levels, such as math “areas” and “categories”. 9 Predictions of students’ proficiency in these topics are based on student performance on the platform’s diagnostic assessment. The latter were completed during the second semester of the 2019-2020 and 2021-2022 school years. In 2020, 85 percent of students who completed the diagnostic assessment did so between January and February, and by March 2020, all students had completed it. In 2022, 95 percent of students completed the assessment in April and May, and all students had completed it by June. One important limitation of our data is representativity. The schools in our sample are largely concentrated in the largest and wealthier provinces in the country: Santo Domingo and Santiago. These schools have a larger student population than the average, with 130 students compared to 83 students in the 9th grade, as well as longer school days. The schools also benefitted from better infrastructure – they had access to computer equipment and internet, whereas almost half the schools in the country do not have access to these resources. 10 Although this 5 Examples of questions included in the Initial Knowledge Check can be shared upon request. 6 The list of 612 topics included in this ALEKS course has not changed since 2020. 7 Topics are considered “mastered” when students have demonstrated mastery and retention of a topic through regular Knowledge Checks. The software establishes that students are “ready to learn” a topic based on the students’ mastery of pre-requisite topics. “Remaining” topics are those for which students have not yet mastered pre-requisite topics. Source: https://www.aleks.com/highered/math/New_ALEKS_Student_Module_Reference_Guide.pdf 8 A full list of the 612 topics included in the platform is available upon request. 9 The 612 topics included in the platform can be aggregated into 13 math areas and 61 categories. 10 Retrieved from: https://www.elcaribe.com.do/panorama/pais/el-desafio-de-llevar-la-educacion-a-comunidades-olvidadas-en-republica- dominicana/. does not affect the internal validity of our results, it may have some implications for our ability to extrapolate these to the rest of the country. To examine internal validity, we start by comparing baseline covariates of students in the analysis sample across years. As shown in Table II, there are no statistically significant differences across baseline covariates for students who completed the diagnostic assessment in 2020 and those who completed it in 2022. Unfortunately, the only variables we have available are student gender and classroom size. Table II. Covariate balance across years in the analysis sample benefitting from CAL Covariate 2020 2022 Difference Share of female students 0.59 0.56 0.03 9th grade classroom size (average per school) 31.82 32.92 -1.10 Number of 9th grade classrooms (average per school) 3.04 3.17 -0.13 Source: Authors’ own elaboration based on administrative data. Students’ gender is estimated using a name-based classifier (genderize.io). *p<0.10 ** p<0.05 ***p<0.01. In 2020 and 2022, between 77 and 91 percent of 9th-grade students (target population) in the analysis schools completed the platform’s diagnostic assessments. On average, students took between 59 and 66 minutes to complete these assessments, as shown in Table III. 11 Table III. Students and schools in the analysis sample benefitting from CAL Participating Students who completed Average time spent on Year students the diagnostic test diagnostic test (minutes) 2020 2,323 1,798 (77%) 65.7 2022 2,502 2,270 (91%) 59.2 Source: Authors’ own elaboration based on implementation records. Using student-level data from these assessments, we first examine variations in student learning before and after COVID-19 school closures, comparing the performance of 9th-graders who completed diagnostic assessments using the adaptive platform in 2020 or 2022. 12 To improve comparability across years, we focus on a set of 24 schools whose students received access to the CAL software in both years. We also look at differences by gender and socioeconomic status. 13 We use the following specification to assess differences in students’ performance between 2020 and 2022: = + 2022 + 11 There was no time limit for completing these assessments. 12 We compare students who took the ALEKS diagnostic assessment in January-February 2020 to students who took the ALEKS diagnostic assessment in April-May 2021. 13 We predict students’ gender based on their names using the free API Genderize.io. The API could not identify the gender of approximately 20 percent of students in our sample based on their names. For these students, the research team performed a manual classification. To validate the API’s classification, we looked at a random sample of 100 students and obtained an error rate of less than 3 percent, excluding students who were not classified by the API. Because we do not have socioeconomic information for all students in our sample, we use as a proxy the socioeconomic status assigned to the school in 2019. This is based on a categorical variable ranging from 1 to 5 developed by the Ministry of Education, where 1 is the lowest socioeconomic level and 5 is the highest. The schools in our sample cover categories 2 to 4. Where represents student ’s average mastery of the platform’s topics, defined as the share of mastered topics in the diagnostic assessment. 2022 takes the value of 1 if student took the assessment in 2022 and 0 if the student took it in 2020. Standard errors are clustered at the school level. We use this specification when looking at differences at the topic, category, and area levels. 3. Results 3.1 Variations in student learning: Before and after COVID-19 Our results suggest that, in our sample of schools, math performance among 9th-grade students did not change between 2020 and 2022. As shown in Table IV, the overall share of mastered topics stayed constant at 13.6 percent between 2020 and 2022, when comparing 9th-grade students who took the ALEKS diagnostic assessment in 2020 with 9th-grade students who took it in 2022. Table IV. Differences in student learning, 2020-2022 2020 2022 Difference Total 13.60 13.61 0.01 Source: Authors’ own elaboration using data from the CAL software. N=4053. *p<0.10 ** p<0.05 ***p<0.01. 3.2 Variations in student learning by gender When looking at differences by gender, we find a small positive difference in students’ mastery among males and a negative difference among females between 2022 and 2020; however, these differences are smaller than one percentage point (or 0.01 standard deviation) and are not statistically significant, as shown in Table V. Table V. Differences in learning by gender, 2020-2022 Difference Difference Gender 2020 2022 (p.p.) (SD) Females 13.96 13.66 -0.30 -0.01 Males 13.14 13.62 0.49 0.01 Total 13.60 13.61 0.01 0.00 Source: Authors’ own elaboration using data from the CAL software. N for males=1714; N for females=2300, N for total=4053. *p<0.10 ** p<0.05 ***p<0.01. 3.3 Variations in student learning by socioeconomic status We then look at differences by socioeconomic status and find a small positive difference of 0.49 percentage points (0.01 SD) in students’ mastery among low-SES students and a negative difference of 0.80 percentage point (-0.02 SD) among high-SES students between 2020 and 2022; however, these differences are not statistically significant, as shown in Table VI. Table VI. Differences in learning by socioeconomic status, 2020-2022 Difference Difference Socioeconomic status 2020 2022 (p.p.) (SD) Low 13.04 13.54 0.49 0.01 Middle 14.93 14.49 -0.44 -0.01 High 14.46 13.66 -0.80 -0.02 Total 13.60 13.61 0.01 0.00 Source: Authors’ own elaboration using data from the CAL software. N for Low=453; N for Middle=639, N for High=1828, N for total=4053. *p<0.10 ** p<0.05 ***p<0.01. 3.4 Variations in student learning by math area When looking individually at the 13 math areas included in the platform, we see small differences ranging from –0.51 to 0.78 percentage points (Table VII), or -0.01 to 0.01 standard deviations. 14 However, these differences are not statistically significant for any of the 13 math areas covered by the platform. Table VII. Differences in student learning across math areas, 2020-2022 Diff Lower Upper Diff Area 2020 (p.p.) 2022 CI CI (SD) Course preparation 45.19 -0.51 44.68 -3.56 2.53 -0.01 (1.47) Linear equations 7.61 8.39 0.78 -1.25 2.81 0.01 (0.98) Linear inequalities 3.23 3.37 0.14 -1.22 1.50 0.00 (0.66) Exponential and polynomial functions 1.94 2.44 0.49 -0.93 1.92 0.01 (0.69) Equations and factoring 0.96 1.05 0.09 -0.88 1.05 0.00 (0.47) Rational expressions 0.60 0.67 0.07 -0.47 0.60 0.00 (0.26) Radical expressions 1.54 1.83 0.28 -0.92 1.49 0.01 (0.58) Linear functions 4.29 4.08 -0.21 -0.90 0.49 -0.01 (0.34) Polygons 4.93 5.10 0.17 -1.12 1.46 0.00 (0.62) Complex numbers and quadratic functions 0.19 0.10 -0.09 -0.33 0.14 -0.01 (0.11) Systems of equations 0.41 0.29 -0.13 -0.55 0.29 -0.01 (0.20) Sets 0.23 0.15 -0.08 -0.25 0.09 -0.02 (0.08) Logic 0.10 0.07 -0.03 -0.11 0.04 -0.01 (0.04) Note: Authors’ own elaboration using data from the CAL software. N=4053. Clustered standard errors are shown in parentheses. * p<0.10 ** p<0.05 ***p<0.01. 14 The 13 areas included in the platform are, in order of appearance: 1) course preparation, 2) linear equations, 3) linear inequalities, 4) exponentials and polynomials, 5) equations and factoring, 6) rational expressions, 7) radical expressions, 8) linear functions, 9) polygons, 10) complex numbers and quadratic functions, 11) systems of equations, 12) sets, and 13) logic. 3.5 Post-COVID-19: How do students fare? The math area with the lowest difficulty level in the ALEKS platform is labeled as “course preparation.” This area includes pre-requisite topics from earlier grades that students are expected to master in order to succeed in 9th-grade math. On average, in 2022, students mastered 44.7 percent of topics included in this area, compared to an average of 2.3 percent across the other 12 areas. When compared to 2020, we do not observe statistically significant differences in this or other math areas included in the platform. Even if students performed relatively better in this area than in others, 44.7 percent is a low number considering that these are topics that students are expected to have mastered before entering this grade. For females, the difference across years in the “course preparation” category is -1.11 percentage points (-0.01 standard deviation), and for males, it is 0.46 percentage points (0.01 standard deviation). These differences are not statistically significant at the 5% level, as shown in Table VIII. Table VIII. Differences in student learning across math areas by gender, 2020-2022 Difference Difference Difference Difference Males Females Males Females Area (p.p.) (p.p.) (SD) (SD) Course preparation 0.46 -1.11 0.01 -0.01 (1.87) (1.58) Linear equations 1.43 0.33 0.02 0.01 (1.39) (1.00) Linear inequalities 0.54 -0.12 0.02 0.00 (0.81) (0.75) Exponential and polynomial functions 0.87 0.25 0.02 0.01 (0.84) (0.73) Equations and factoring 0.34 -0.09 0.02 0.00 (0.50) (0.52) Rational expressions 0.25 -0.06 0.02 0.00 (0.31) (0.27) Radical expressions 0.68 0.01 0.03 0.00 (0.65) (0.63) Linear functions 0.00 -0.36 0.00 -0.02 (0.46) (0.39) Polygons 0.54 -0.06 0.02 0.00 (0.84) (0.71) Complex numbers and quadratic functions -0.01 -0.15 0.00 -0.02 (0.08) (0.16) Systems of equations 0.05 -0.25 0.01 -0.02 (0.19) (0.26) Sets -0.18 -0.01 -0.04 0.00 (0.11) (0.10) Logic 0.01 -0.06 0.01 -0.02 (0.04) (0.06) Note: Authors’ own elaboration using data from the CAL software. N for males=1714; N for females=2300. Clustered standard errors in parentheses. * p<0.10 ** p<0.05 ***p<0.01. When examining differences by socioeconomic status across years, we find a difference of 1.53 percentage points (0.01 SD) for low-income students, 0.90 percentage points (0.01 SD) for middle-income students, and -3.25 percentage points (-0.03 SD) for high-income students in the “course preparation” category. These differences are not statistically significant at the 5 percent level, as shown in Table IX. Table IX. Differences in student learning across math areas by socioeconomic status, 2020-2022 Difference Difference Difference Difference Difference Middle Difference Area Low Middle High Low (SD) (SD) High (SD) Course preparation 1.53 0.90 -3.25 0.01 0.01 -0.03 (6.26) (3.30) (2.24) Linear equations 0.85 -0.08 -0.24 0.01 0.00 0.00 (2.83) (2.59) (1.74) Linear inequalities -0.20 -0.95 -0.22 -0.01 -0.02 0.00 (1.39) (2.06) (1.18) Exponential and polynomial functions 0.02 -1.68 0.64 0.00 -0.03 0.01 (0.75) (2.39) (1.20) Equations and factoring 0.06 -1.84 0.43 0.01 -0.03 0.01 (0.36) (2.16) (0.68) Rational expressions 0.17 -0.88 0.16 0.04 -0.03 0.01 (0.19) (1.00) (0.45) Radical expressions 0.33 -1.64 0.39 0.04 -0.03 0.01 (0.42) (2.25) (0.99) Linear functions -0.18 -0.92 -0.42 -0.01 -0.03 -0.02 (0.75) (1.46) (0.47) Polygons 0.09 -0.77 -0.09 0.00 -0.01 0.00 (1.44) (2.32) (0.97) Complex numbers and quadratic functions 0.00 -0.54 0.03 0.00 -0.03 0.00 (0.03) (0.62) (0.13) Systems of equations -0.07 -1.13 0.11 -0.02 -0.04 0.01 (0.14) (1.12) (0.22) Sets -0.09 -0.30 0.01 -0.11 -0.03 0.00 (0.04) (0.46) (0.09) Logic 0.07 -0.17 0.00 0.05 -0.07 0.00 (0.07) (0.11) (0.01) Note: Authors’ own elaboration using data from the CAL software. N for Low=453; N for Middle=639, N for High=1828. Robust standard errors in parentheses. * p<0.10 ** p<0.05 ***p<0.01. When looking at the five segments covered by this area (Table X), we find small but not significant changes when compared to 2020. Across segments, students’ mastery remains low in 2022, ranging from 16 percent for Real Numbers to about 73 percent for Natural Numbers. The largest differences across years are observed for Exponents and Order of Operations (-1.24 percentage points or -0.01 standard deviations) and for Natural Numbers (-0.89 percentage points, -0.01 standard deviations). These differences are not statistically significant at the 5 percent level. Table X. Students’ mastery of prerequisite segments in 2022 Difference Difference Prerequisite segments 2020 2022 (p.p.) (SD) Natural numbers 73.73 72.84 -0.89 -0.01 Exponents and order of operations 37.41 36.17 -1.24 -0.01 Fractions 22.66 23.26 0.61 0.01 Decimals 37.54 36.74 -0.80 -0.01 Real numbers 15.62 15.76 0.14 0.00 Note: Authors’ own elaboration using data from the CAL software. N=4053. * p<0.10 ** p<0.05 ***p<0.01. We find differences in the distribution of students’ performance across prerequisite topics between 2020 and 2022. Table XI suggests that a higher share of students mastered between 0 and 25 percent of pre- requisite topics in 2022 —13.4 percent compared to 10.5 percent in 2020, and a higher share mastered more than 75 percent of topics —10.9 percent in 2022 compared to 8.1 percent in 2020. Table XI. Share of students across levels of performance in prerequisite topics Score range 2020 2022 Difference 0-25 10.5 13.4 2.9 25-50 58.4 56.1 -2.3 50-75 23.0 19.6 -3.4 >75 8.1 10.9 2.8 Note: Authors’ own elaboration using data from the CAL software. N=4053. * p<0.10 ** p<0.05 *** p<0.01. 3.6 Why do learning outcomes remain constant despite school closures? We consider two potential explanations for these findings: 1) changes in the composition of students taking the tests, and 2) the widespread implementation of remedial programs during the beginning of the 2021-2022 school year. 3.6.1 Changes in the composition of students taking the tests In Section 2, we show that although a higher share of students in participant classrooms completed the diagnostic assessments in 2022 compared to 2020 (91 versus 77 percent), the students who took it in 2022 are very similar to those who took it in 2020. We also show that participant schools do not seem to have changed significantly between 2020 and 2022, based on the limited covariates available (i.e., school size and the number of 9th-grade classrooms available). We do not have socioeconomic data for the schools and students in our sample for 2022. However, a recent report by MINERD compared the socioeconomic composition of a representative sample of schools in 2019 and 2022 and found few significant changes across years (MINERD, 2023b). Between 2019 and 2022, 47 percent of 9th-grade schools remained in the same socioeconomic quantile, while only 15 percent experienced a shift of more than one category from their 2019 standing. However, a critical question remains: has the composition of students enrolled in the school system shifted? Evidence indicates that COVID-19 has increased dropout rates globally, particularly in the developing world (Moscoviz and Evans, 2022). This dropout trend has predominantly impacted the most disadvantaged students, who also tend to be the ones with the largest learning gaps. Consequently, if a large share of low-performing students had left the school system by 2022, our findings may be skewed. That is because excluding the lowest achievers could artificially elevate the average learning levels observed in the 2022 sample. In Section I, we find that the number of eligible students for the study (students enrolled in the 9th grade) is very similar across schools (2,323 in 2020 vs 2,502 in 2022). This is consistent with national enrollment trends, as shown in Figure I. Using data from the Central Bank’s National Labor Force Survey, we find that between the first quarter of 2020 and the first quarter of 2021, enrollment rates in surveyed households fell from 89 percent in 2020 to 85 percent in 2021. However, by the first quarter of 2022, enrollment rates had nearly reached their pre-pandemic levels (88 percent), suggesting that the dropout observed during school closures was only temporary. 15 Figure I. Enrollment rates between 2016 and 2022 0.95 0.90 0.90 0.90 0.89 0.89 0.90 0.88 0.85 0.85 0.80 2016 2017 2018 2019 2020 2021 2022 Notes: Author’s own estimates using data from the Dominican Republic Central Bank’s Continuing National Labor Force Survey. Overall, our findings suggest that differences in the composition of test-takers across years do not seem to account for the absence of learning losses found in our study. 3.6.2 Remedial education in the Dominican Republic post-COVID-19 A second hypothesis is that learning losses may have been offset by the implementation of a national strategy designed to address learning losses in the Dominican Republic at the beginning of the 2021-22 academic year. This remediation plan was established through Resolution 03-2021, which establishes, in Article 21, that “a remediation plan would be set for all students of each grade and level of the Dominican educational system.” According to the official school calendar for 2021-22, during the first month of classes, after prolonged school closures, schools in the country were largely focused on measuring and remediating learning. The timeline for these activities is shown in Table XII. Table XII. Timeline for MINERD’s remediation strategy for the 2021-22 school year Date Activity September 20th, 2021 Start of the 2021-22 school year September 27th-30th, 2021 Application of diagnostic evaluations to all students October 4 and 29 , 2021 th th Leveling process based on results from diagnostic evaluations Source: Official school calendar for the 2021-22 school year (MINERD, 2021). This remediation strategy and its activities were established by the Ministry of Education and all schools received detailed guidelines on how to implement them, including the periods in which these were to take 15 In 2021-2022, the Ministry of Education implemented the program “I Want You in Secondary School” (“Te Quiero en Secundaria”) with the goal of achieving the re-entry and retention of students who withdrew from the educational system during the last 3 years. In 2022, the program largely focused on training regional coordinators on feasible, viable, and successful approaches to achieve this goal (MINERD, 2022). place, as shown in Table XII. The initial phase of the strategy involved assessing students to identify their skills in grade-relevant topics, with the Ministry’s technical teams set to offer guidance and support throughout the evaluation process. School management teams were expected to organize the evaluations, provide resources, and assist teachers with test corrections and analysis. Teachers were tasked with preparing materials, conducting evaluations, and incorporating the results into student portfolios. The subsequent phase aimed to address learning gaps through remedial activities, with national, regional, and district technicians overseeing and monitoring their implementation. Management teams and teachers were to play crucial roles in organizing remedial activities, engaging families, and tracking student progress with regular assessments. This strategy was covered by the local media and by regional organizations. 16,17 Unfortunately, we have limited information regarding the actual implementation of this strategy and the extent to which schools in our analysis sample adhered to its prescribed activities. 4. Discussion Overall, our findings suggest that, on average, there were no significant changes in the math performance of 9 -grade students between 2020 and 2022, based on this sample of urban schools. These findings do not seem to th be explained by changes in student composition across schools, as by 2022, enrollment rates at the secondary level appeared to have fully recovered and we do not find observable differences in students’ and school characteristics across years. Nonetheless, it is important to highlight several limitations of this analysis. First, our sample of schools is not representative of the average public school in the country. The schools in our sample are largely concentrated in the largest and wealthier provinces in the country: Santo Domingo and Santiago. These schools have a larger student population as well as longer school days. The schools also benefitted from better infrastructure ––they had access to computer equipment and internet, whereas almost half of schools in the country do not have access to these resources. Considering these differences between schools in our sample and the average school in the country, our results should only be extrapolated to students in schools that fit this profile. We cannot dismiss the possibility that students from more disadvantaged contexts and whose schools might have been less prepared for the transition to remote instruction, including those living in remote communities, may have experienced substantial learning losses. However, our results align with the outcomes of national diagnostic assessments conducted in 2022, which similarly did not reveal statistically significant differences when compared to an equivalent cohort who completed the assessments in 2019. The last round of PISA assessments also revealed that the Dominican Republic was one of few countries that did not experience learning losses between 2018 and 2022. Another potential explanation for the lack of variation in student performance may be that learning losses during school closures were offset by an “accelerated” learning strategy that took place in the fall of 2021. Under this plan, teachers were to first identify the topics in which students needed most support, as part of a national 16 https://socialdigital.iadb.org/es/edu/covid-19/respuesta-regional/6081. 17 https://www.diariolibre.com/actualidad/educacion/refuerzo-y-nivelacion-tareas-para-el-primer-trimestre-del-ano-escolar-KK19024248. consultation on student learning, and then dedicate one month to remedial activities centered around these topics. However, more information is needed to understand the extent to which teachers followed the Ministry’s instruction and, if they did, the type of activities they engaged in during this period. Lastly, we cannot reject that our inability to detect changes in performance within our sample could be attributed to “floor effects,” by which large shares of low-performing students skew the distribution of scores and hinder our ability to detect significant differences. This hypothesis is reinforced by a heavily skewed performance distribution, as evidenced by Figure A1 in Appendix 5.2. To further explore the reasons behind this observed lack of variation in student learning, we have identified a list of activities for future work, including the administration of surveys aimed at understanding the involvement of principals and teachers in remediation efforts within the framework of schools' recovery strategies. 5. Appendix 5.1 Topics, areas and categories included in the platform AREAS CATEGORIES (61) TOPICS (612) EXAMPLES EXAMPLES 1. Course preparation Natural numbers, fractions Introduction to exponents, equivalent fractions Properties of operations, Identifying like terms, additive property of 2. Linear equations introduction to linear equations equality with natural numbers Linear inequalities with one Translating a sentence by using an inequality 3. Linear inequalities variable, absolute value equations symbol and inequalities Introduction to the product rule of exponents, 4. Exponents and polynomials Exponent rules, negative exponents evaluating expressions with exponents of zero Factoring through common factors, Factoring a linear binomial, factoring out a 5. Equations and factoring factoring by grouping binomial from a polynomial: basic 6. Rational expressions Simplifying expressions Restriction on a variable in a denominator Square root of a rational perfect square, Finding nth roots of perfect nth 7. Radical expressions converting between radical form and exponent powers, rational exponents form The coordinate plane, plotting lines Reading a point in the coordinate plane, graphing 8. Functions and lines and intersections a linear equation of the form y = mx + c Perimeter and area of a rectangle, Perimeter of a polygon, acute, obtuse and right 9. Polygons angles angles, right triangles 10. Complex numbers and Complex numbers, quadratic Using i to rewrite quadratic roots of negative quadratic functions equations and functions numbers Identifying solutions to a system of linear Linear systems with two variables, 11. Systems of equations equations, introduction to solving a 3x3 system of linear systems with three variables linear equations Identifying elements of sets for a real world 12. Sets Sets, subsets situation, writing subsets Identity propositions, symbolic representation of Propositions, conjunctions and 13. Logic negations, conjunctions and disjunctions: basic disjunctions level 5.2 Distribution of student performance in 2020 and 2022 Figure A1. Distribution of student performance in 2020 and 2022 5.3 Replication of the main results using the full sample Our conclusions do not change when we replicate the main analysis using the full sample. Table A1. Differences in student learning, 2020-2022 Difference 2020 2022 (p.p.) N 1.08 Full Sample 13.74 14.81 7168 (1.49) 0.01 Analysis Sample 13.60 13.61 4053 (0.68) Source: Authors’ own elaboration using data from the CAL software. * p<0.10 ** p<0.05 *** p<0.01. 6. Bibliography Angrist, N., de Barros, A., Bhula, R., Chakera, S., Cummiskey, C., DeStefano, J., ... & Stern, J. (2021). Building back better to avert a learning catastrophe: Estimating learning loss from COVID-19 school shutdowns in Africa and facilitating short-term and long-term learning recovery. International Journal of Educational Development, 84, 102397. Azevedo, J. P. W. D., Rogers, F. H., Ahlgren, S. E., Cloutier, M. H., Chakroun, B., Chang, G. C., ... & Bergmann, J. L. (2021). The state of the global education crisis: A path to recovery. Baron, J., Adelman, M., Mola, J., Pineda, A & Polanco, P. (unpublished). Can Computer-Assisted Adaptive Instruction Improve Student Outcomes in Public School Systems? Baron, J. D., Taveras, C. M., & Zúñiga, J. C. (2018, December 19). Adaptive Technology to Help Improve Math Learning in the Dominican Republic. Education for Global Development. World Bank. https://blogs.worldbank.org/education/adaptive-technology-help-improve-math-learning-dominican-republic Cosyn, E., Uzun, H., Doble, C., & Matayoshi, J. (2021). A practical perspective on knowledge space theory: ALEKS and its data. Journal of Mathematical Psychology, 101, 102512. Crawfurd, L., Hares, S., & Minardi, A. L. (2021). New data on learning loss in Pakistan. Center for Global Development. https://www.cgdev.org/blog/new-data-learning-loss-pakistan Doignon, J. P., & Falmagne, J. C. (1985). Spaces for the assessment of knowledge. International journal of man- machine studies, 23(2), 175-196. Donnelly, R., & Patrinos, H. A. (2021). Learning loss during Covid-19: An early systematic review. Prospects, 1-9. Engzell, P., Frey, A., & Verhagen, M. D. (2021). Learning loss due to school closures during the COVID-19 pandemic. Proceedings of the National Academy of Sciences, 118(17), e2022376118. Falmagne, J.-C., Koppen, M., Villano, M., Doignon, J.-P., & Johannesen, L. (1990). Introduction to knowledge spaces: How to build, test, and search them. Psychological Review, 97(2), 201–224. https://doi.org/10.1037/0033- 295X.97.2.201 Gajderowicz, T., Jakubowski, M., Patrinos, H. A., & Wrona, S. (2022). Capturing the Educational and Economic Impacts of School Closures in Poland. Hevia, F.J., S. Vergara-Lope Tristan, A. Velásquez-Durán and D. Calderón Martín del Campo. (2021). Estimation of the fundamental learning loss and learning poverty related to COVID-19 pandemic in Mexico, International Journal of Educational Development. https://www.sciencedirect.com/science/article/pii/S0738059321001681 Iqbal, Syedah and Patrinos, Harry Anthony, Learning During the Pandemic: Evidence from Uzbekistan. IZA Discussion Paper No. 16232, Available at SSRN: https://ssrn.com/abstract=4479174 or http://dx.doi.org/10.2139/ssrn.4479174 Jakubowski, M., Gajderowicz, T., & Patrinos, H. A. (2023). Global learning loss in student achievement: first estimates using comparable reading scores. Economics Letters, 232, 111313. Jakubowski, M., Gajderowicz, T., & Patrinos, H. A. (2024). COVID-19, School Closures, and Student Learning Outcomes: New Global Evidence from PISA. Kaffenberger, M. (2021). Modelling the long-run learning impact of the Covid-19 learning shock: Actions to (more than) mitigate loss. International Journal of Educational Development, 81, 102326. Ministry of Education of the Dominican Republic (MINERD, 2021). General Directorate for Primary Education. Guidelines for the implementation of diagnostic evaluations for the 2021-2022 school year. Ministerio de Educación de la República Dominicana. (MINERD, 2023a). Informe Evaluación Diagnóstica 2022: 3ro de Secundaria. Ministerio de Educación de la República Dominicana. Retrieved from: https://www.ministeriodeeducacion.gob.do/docs/direccion-de-evaluacion-de-la-calidad/zA9H-informe-evaluacion- diagnostica-2022-3ro-de-secundariapdf.pdf Ministerio de Educación de la República Dominicana. (MINERD, 2023b). Informe Evaluación Diagnóstica 2022: 3ro de Secundaria. Ministerio de Educación de la República Dominicana. Retrieved from: https://www.ministeriodeeducacion.gob.do/docs/direccion-de-evaluacion-de-la-calidad/yn1z-informe-tecnico- evaluacion-diagnostica-nacional-2022pdf.pdf Moscoviz, L. and D.K. Evans. 2022. Learning Loss and Student Dropouts during the COVID-19 Pandemic: A Review of the Evidence Two Years after Schools Shut Down. CGD Working Paper 609. Washington, DC: Center for Global Development. OECD (2023), PISA 2022 Results (Volume I): The State of Learning and Equity in Education, PISA, OECD Publishing, Paris, https://doi.org/10.1787/53f23881-en Patrinos, H. A., Vegas, E., & Carter-Rau, R. (2022). An analysis of COVID-19 student learning loss. Lichand, G., Doria, C. A., Leal-Neto, O., & Fernandes, J. P. C. (2022). The impacts of remote learning in secondary education during the pandemic in Brazil. Nature Human Behaviour, 6(8), 1079-1086. Uwezo. (2021). Are Our Children Learning? Illuminating the COVID-19 Learning Losses and Gains in Uganda https://uwezouganda.org/wp-content/uploads/2022/01/Uwezo-2021Report-FINAL_Version.pdf World Bank. 2021. Acting Now to Protect the Human Capital of Our Children: The Costs of and Response to COVID- 19 Pandemic’s Impact on the Education Sector in Latin America and the Caribbean. World Bank, Washington, DC. © World Bank. https://openknowledge.worldbank.org/handle/10986/35276 World Bank. (2022). Two Years After: Saving a Generation. Washington, DC: World Bank. © World Bank. https://openknowledge.worldbank.org/handle/10986/37586