LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 01 Learning in the Shadow of the Pandemic: COVID-19 Learning Loss and Widening Learning Disparities in Indonesia Anna Hata, Seil Kim, and Shinsaku Nomura June 2024 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 02 Learning in the Shadow of the Pandemic: COVID-19 Learning Loss and Widening Learning Disparities in Indonesia This report was prepared by: Anna Hata, Seil Kim and Shinsaku Nomura. Acknowledgement This work is a product of the staff of The World Bank, supported by funding from the Australian Government through the Australian Department of Foreign Affairs and Trade (DFAT). The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions © 2024 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for non-commercial purposes as long as full attribution to this work is given. All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Attribution Please cite this work as follows: Hata, Anna; Kim, Seil; Nomura, Shinsaku. 2024. Learning in the Shadow of the Pandemic: COVID-19 Learning Loss and Widening Learning Disparities in Indonesia. Washington, D.C.: World Bank. Contact Information The authors can be contacted at snomura@worldbank.org. Photo Credit: Queenmoonlite/Envato LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 03 Acknowledgements This report was written by Anna Hata (Consultant), Seil Kim (Consultant), and Shinsaku Nomura (Senior Economist) from the Education Global Practice, East Asia and Pacific Region (HEAED). The authors are grateful for the overall guidance provided by Cristian Aedo (Practice Manager, HEAED). The report benefited from peer review comments from Harry Patrinos (Senior Advisor, HEDDR), Marie-Helene Cloutier (Senior Economist, HEDGE), and Adelle Pushparatnam (Senior Education Specialist, HMNED), as well as the wider Indonesia team working on education. The team appreciates Sylvia Njotomihardjo for the operational support to the team and Sheila Town for editorial support. The authors greatly appreciate a series of discussions, consultations and written feedback received from the Government of Indonesia including the Ministry of Education, Culture, Research, and Technology (MoECRT) and the Ministry of Religious Affairs (MoRA). The work received support from the Australia-World Bank Indonesia Partnership (ABIP), which is financed by the Australian Government through the Australian Department of Foreign Affairs and Trade (DFAT). The findings and recommendations are those of the authors and do not necessarily represent the views of the Executive Directors of the World Bank or of the countries they represent. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 04 Table of Contents Acknowledgement 03 Table of Contents 04 Acronyms 06 Executive Summary 07 1. Introduction 12 2. Contextual Background: Indonesia’s Education System, Challenges 13 and The Covid-19 Pandemic 3. International Studies of Post-Pandemic Learning Loss 16 3.1 Magnitude of learning loss and variance across countries 16 3.2 Heterogeneous effects of covid-19 on different social groups 17 4. Objectives and Methodologies of The Study 19 4.1. Objective of the study 19 4.2. Data and sampling process 20 4.3. Measures 21 5. Analytical Results of Learning Losses Between 2019 and 2023 24 5.1. Estimating learning loss in language and mathematic skills 24 5.2. Learning loss disparities across socioeconomic backgrounds 26 5.3. Learning loss disparities related to school environments 28 6. Analysing the Characteristics Related to Learning Disparities in 2023 30 6.1. Correlations between school practices and students’ learning 30 6.2. Examining learning disparities by school practices during the pandemic 33 6.2.1. Urban-rural disparities in school practices and government support 33 6.2.2. An assessment of internet access on learning outcomes in urban and 34 rural areas 6.2.3. Comparing the different ministries’ responses to the school closures 36 6.3. Teachers’ characteristics and teaching practices and students’ learning 37 6.4. Individual-level characteristics and students’ learning 38 6.4.1. Widening learning disparities across groups of different socioeconomic 38 groups 6.4.2. Unpacking socioeconomic disparities in learning environments 41 7. Conclusions and Policy Recommendations 42 References 46 Appendices 50 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 05 Appendices Appendix A: Technical note on the learning loss survey 2023 50 Appendix B: Plausible values calculation and analytical approach 54 Appendix C: Variable descriptions 56 Appendix D: Average student test score by background characteristics, 2023 57 Appendix E: Detailed outputs from the HLM analysis 60 Appendix F: Learning outcomes and individual backgrounds by year 62 Appendix G: Teach instrument and analysis of teachers by gender 64 Appendix H: Distribution of test scores and key parameters 66 Appendix I: Learning loss literature by country with estimated effect and analytical approach 67 List of Tables Table ES1. Changes in Standardized Test Scores Before and After the Pandemic Across Individual 10 and School Characteristics, from 2019 to 2023 Table ES2. Post-Pandemic Learning Outcomes Across School Practices and Household 11 Backgrounds in 2023 Table 1. Estimated Learning Loss between 2019 and 2023 25 Table 2. Average language and mathematics scores among different social groups, 2019 and 2023 27 Table 3. Analysis of Learning Loss by School Factors: Comparing Pre- and Post-COVID-19 Periods 28 Table 4. Between-school and Within-school Variances on Students’ Learning Outcome by Year 29 Table 5. HLM Regression Analysis of School and Individual Factor on Learning Outcomes in 2023 32 Table 6. Comparison of Teacher and Principal Practices and Government Support by School Location 34 Table 7. Comparison of Teacher and Principal Practices and Government Support by School Type 37 Table 8. Association between Teach Scores and Students’ Learning Outcomes 38 Table 9. Descriptive Statistics of Average Household Characteristic by Socioeconomic Groups 39 List of Figures Figure 1. Comparison of 2019 and 2023 Score Distributions by Subject 25 Figure 2. Fitted Test Scores among Groups of Availability of Internet and Locations 35 Figure 3. Fitted Test Scores by Groups of Internet-related Government Interventions 36 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 06 Acronyms BOS School Operational Assistance Fund (Bantuan Operasional Sekolah) GoI Government of Indonesia HIC High income country HiFy survey High-frequency phone-monitoring survey HLM Hierarchical Linear Modeling HIC High Income Country IT Information Technology IRT Item Response Theory LIC Low Income Country LMIC Lower middle income country MIC Middle Income Country MoECRT Ministry of Education, Curriculum, Research and Technology MoRA Ministry of Religious Affairs NER Net enrolment rate NGO Non-government organization PISA Programme for International Student Assessment PV Plausible Value SD Standard Deviation SDI Service Delivery Indicator SE Standard Error SES Socio-economic status SUSENAS Indonesia Socioeconomic Survey UMIC Upper Middle Income Country LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 07 Learning in the Shadow of the Pandemic: 07 COVID-19 Learning Loss and Widening Learning Disparities in Indonesia Executive Summary The COVID-19 pandemic and associated school closures led to significant learning disruption around the world (Patrinos, 2023; Schady et al, 2023), and there is a need to understand the relationship between this disruption and student learning outcomes in Indonesia. Indonesia has been particularly affected, as it has the world’s fourth largest education system, meanwhile low learning quality has been an issue since before the COVID-19 pandemic and resulting school closures. Indonesia’s schools closed for up to 21 months, a relatively long period compared to low and middle income countries (World Bank, 2023a). Since 2020, the Government of Indonesia (GoI) has implemented a series of interventions to mitigate anticipated learning losses. Analysis is important to identify the magnitude and disparities of learning losses across students and to inform evidence-based policy making (Betthäuser et al., 2023). However, research on COVID-19 induced learning losses has mostly been restricted to estimation,1 so more empirical evidence is required in Indonesia to support the GoI’s ongoing effort for learning recovery. This study builds on a recent World Bank report,2 and aims to conduct a rigorous and comprehensive analysis of associated factors related to learning loss and disparity in Indonesia. This study draws on empirical data including both students’ individual characteristics and structural factors such as governmental support related to the pandemic, school-based management and teacher quality. This study employed surveys from 2019 and 2023 to estimate the learning loss among 6,693 primary school students following the COVID-19 school closures.3 Additionally, it identified school and individual factors that promoted improvement in student learning during the pandemic. It found that students examined in March 2023 experienced a learning loss of 0.265 standard deviations in language (equivalent to approximately 10.6 months) and 0.276 standard deviations in math (11 months), compared to same-aged students in 2019.4 1 Some empirical studies include Learning Gap Study (INOVASI and MoECRT PSKP, 2022) and Bounce Back Stronger: Learning Recovery After the Pandemic: A Case Study of INOVASI Partner Schools (Sukoco et al., 2023). 2 World Bank (2023a). Indonesia Economic Prospects, June 2023: The Invisible Toll of COVID-19 on Learning, World Bank Group, https://documents1.worldbank. org/curated/en/099062323023530087/pdf/P179556020fd80030087730cbc843df07de.pdf 3 The study did not aim to measure the impact of the interventions by the GoI’s learning loss mitigation policies. 4 The analysis of learning loss in this study is updated by using a refined methodology, although the message is consistent with the earlier study. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 08 This study reveals increased disparity in learning outcomes across different groups. Students with lower socioeconomic status (SES) experienced larger learning losses than others. The poorest 20 percent of households experienced approximately two years of learning loss in both language and math, and students with less educated parents suffered from greater learning losses, including up to two years in language. Pre-existing gender and regional gaps in learning outcomes persisted during the pandemic, with boys underperforming girls, and rural schools underperforming urban schools in both 2019 and 2023. This increased disparity in learning outcomes is correlated with various factors, organized into four sets of key factors including (i) teacher, (ii) school and (iii) government and (iv) individual levels. More effective teacher practices were associated with higher learning outcomes in both subjects. Frequency of assignment provision, promptness of assignment submission requirement, and overall teaching quality measured by Teach scores were all associated with improvements in student learning outcomes. Better school management contributes to improved learning outcomes. When principals monitored teaching practices frequently and schools had internet access prior to the pandemic, learning outcomes improved. Government support for internet connectivity during the pandemic, and online learning resource provision correlated positively with improved student learning outcomes. However, government support tended to focus on to urban rather than rural schools, so this uneven support to internet connection, online resources, school principal and teacher training, possibly contributed to increasing inequalities in learning outcomes during the pandemic. At student level, learning disparities by student SES increased in 2023, compared to 2019. Students from low SES had less family support for completing assignments, fewer educational resources at home and they often lived in rural areas. These students also tended to take on more household responsibilities, which potentially hampered their learning time. Loss of family members or someone close during the pandemic led to significant decline of learning scores, equivalent to approximately a 6-7 month decline. Students from low SES also reported higher incidences of losing someone close than other students. Hence, widening learning disparities by students’ SES possibly occurred due to these cumulative disadvantages and the lack of systems to provide appropriate support. These findings suggest key policy actions to address widening learning disparities: Reorient resources to target Focus on evidence-based effective disadvantaged areas and students practices aiming to improve by both MoECRT and MoRA, student learning outcomes especially focusing on rural schools and experiences, rather than and students from low SES. increasing inputs only. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 09 The following four action areas detail and integrate the above recommendations, focusing on how to improve the capacities of teachers, school principals, national and subnational governments and students and families. The recommendations mainly focus on learning recovery in a way that is relevant to the current post-pandemic context, to strengthen the system to respond to the needs of teachers, students and other key stakeholders, in order to make the education system more resilient to future shocks. Enhance the effectiveness of teacher training by focusing on teacher behavior change, including rural teachers. Teacher training should focus more on practical skills and how to promote behavior change, instead of focusing purely on knowledge accumulation. For example , the e mpirical e vide nce suggests Te aching at the Right Level can improve Indonesia’s student learning outcomes, and has been adopted by GoI’s education reforms. Teacher training should develop teachers’ skills to improve teacher-student interactions, covering not only teaching methods but also teachers’ socio-emotional skills and assessment design and support skills. It should also provide specific training and support for teachers in rural areas. These can be done by building on GoI’s EdTech teacher training platforms such as Platform Merdeka Mengajar (and its offline version, Awan Penggerak) and Madrasah E-Learning. Strengthen school principal training especially related to monitoring and accountability, including rural schools. School principal training should focus on how to improve school management to bring about changes in student learning outcomes. Principal training should cover key management skills linked to better teaching and learning and include a focus on rural schools. Use digital transformation to promote equity of learning opportunities by providing support especially in rural areas. Bridging the digital divide requires strengthened support for rural areas. Investment in internet accessibility and IT hardware for rural schools can be effective in enhancing student learning, since urban schools often already have these resources. Provision of material resources can support student learning, by simultaneously providing technical and teacher support to ensure effective practices. Alternative support mechanisms for remote areas should also be cultivated to improve equity in education. Strengthen support for students and parents from low SES and rural areas. Education support by the government and school should address the economic, educational and psychological challenges faced by students from low SES. Parental support is particularly needed for students from economically and socially disadvantaged backgrounds. Table 1 shows the factors associated with the learning loss in language and mathematics, including individual and school characteristics. Table 2 shows the parameters associated with learning outcomes in 2023, including individual and school characteristics, as well as school access to government support during the learning disruptions caused by COVID-19 pandemic, focusing on information technology (IT)hardware, internet connection, and online learning resources. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 10 Table ES1. Changes in Standardized Test Scores Before and After the Pandemic Across Individual and School Characteristics, from 2019 to 2023 Language Mathematics 2019 Mean 2023 Mean Diff 2019 Mean 2023 Mean Diff All 0 -0.265 -0.265** 0 -0.276 -0.276** Household Asset Quintile 1-Lowest -0.28 -0.85 -0.571** -0.242 -0.683 -0.440** 2 -0.317 -0.482 -0.165 -0.399 -0.481 -0.081 3 -0.122 -0.255 -0.133 0.099 -0.333 -0.433** 4 -0.122 -0.111 -0.420** 0.156 -0.152 -0.308 5-Highest 0.242 0.139 -0.103 0.138 0.065 -0.073 Father's Education Primary or less -0.255 -0.781 -0.525** -0.328 -0.677 -0.349** Lower Secondary -0.106 -0.408 -0.302** -0.061 -0.403 -0.342** Upper Secondary 0.062 -0.043 -0.105 0.168 -0.127 -0.295* Tertiary 0.356 0.337 -0.020 0.446 0.254 -0.192 Mother's Education Primary or less -0.136 -0.745 -0.609** -0.376 -0.68 -0.304** Lower Secondary -0.016 -0.376 -0.360** 0.157 -0.411 -0.567** Upper Secondary -0.04 -0.067 -0.027 0.115 -0.125 -0.240 Tertiary 0.188 0.326 0.139 0.423 0.299 -0.124 Gender Male -0.183 -0.46 -0.277** -0.156 -0.412 -0.256** Female 0.181 -0.071 -0.252** 0.154 -0.146 -0.301** Area Rural -0.214 -0.522 -0.308** -0.284 -0.557 -0.273** Urban 0.136 -0.13 -0.266*** 0.18 -0.133 -0.313** School Ownership Public -0.031 -0.358 -0.326*** -0.043 -0.326 -0.282*** Private 0.109 -0.019 -0.128 0.151 -0.144 -0.295* School Type MoECRT -0.005 -0.32 -0.315*** 0.008 -0.28 -0.288*** MoRA 0.017 -0.087 -0.104* -0.027 -0.262 -0.235*** Source: Authors’ analysis using SDI Survey 2019 and Learning Loss Survey 2023 Note: Weighted descriptive statistics consider variations in student and school demographics. The 'Diff' column shows the change in test scores before and after the pandemic, calculated using regression analysis with weighted methods and standard errors clustered by school. * p<0.05, ** p<0.01, *** p<0.001 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 11 Table ES2. Post-Pandemic Learning Outcomes Across School Practices and Household Backgrounds in 2023 Language Mathematics Average score Average score All -0.265 -0.276 I. Household level Attending preschool No -0.820 -0.659 Yes -0.189 -0.226 Close One's Death No -0.203 -0.211 Yes -0.382 -0.407 Household chores Non-economic -0.188 -0.231 Economic (e.g., selling goods, farming) -0.502 -0.426 II. School level Pre-COVID Access to Internet No -0.624 -0.522 Yes -0.136 -0.191 Assignment submission time Not on the same day -0.356 -0.323 On the same day -0.092 -0.193 Principals monitoring frequency Not Everyday -0.353 -0.368 Everyday -0.119 -0.133 III. Government support during the pandemic IT Hardware No -0.208 -0.279 Yes -0.411 -0.274 Internet Connection No -0.342 -0.34 Yes -0.036 -0.096 Online learning resources No -0.306 -0.314 Yes -0.004 -0.057 Source: Authors’ analysis using SDI Survey 2019 and Learning Loss Survey 2023 Note: Weighted descriptive statistics consider variations in student and school demographics. The 'Diff' column shows the change in test scores before and after the pandemic, calculated using regression analysis with weighted methods and standard errors clustered by school. * p<0.05, ** p<0.01, *** p<0.001 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 12 Learning in the Shadow of the Pandemic: 12 COVID-19 Learning Loss and Widening Learning Disparities in Indonesia 1 Introduction COVID-19-induced learning loss and learning interventions including a new curriculum and a series recovery in post-COVID is a global concern, especially of teacher and school support which continued in low-income countries (LICs) and middle-income through the pandemic. However, while Indonesia's countries (MICs) where low learning quality was a new curriculum has a greater focus on foundational key issue even before the pandemic (Patrinos, 2023; skills to support learning recove ry, there is still a Schady et al, 2023). Learning losses and learning need for greater examination of the kind of student outcomes are not merely related to students’ individual support, teaching practices, school management characteristics, but also to structural factors at school and government interventions which can accelerate level, such as the quality of teachers and school-based learning recovery post-COVID. management. Globally, studies show the importance of teacher quality and school-based management The objective of the report is to analyze patterns of in improving students’ learning outcomes (Darling- learning losses and associated school and individual Hammond et al., 2017; OECD, 2020). However, more characteristics in Indonesia, focusing on primary research is needed to understand how teacher quality school students.5 This study builds on a recent and school management relate to student learning report by the World Bank (2023a) which provided outcomes and COVID-19-induced learning losses. foundational descriptive analysis of learning losses following the COVID-19 pandemic in Indonesia. In Indonesia, low learning quality was a key issue Building on this earlier work, this report focuses on prior to the pandemic. With the onset of the more detailed analysis of learning losses in correlation pandemic in 2020 and resulting school closures, with students’ characteristics, teacher quality, school the Government of Indonesia (GoI) responded with management and government support during the a range of educational reforms. GoI implemented pandemic. This study provides evidence and policy educational reforms to address the issues with recommendations. learning quality and learning losses through various 5 In this study, learning loss consists of foregone learning, or learning that will not occur due to school closure, and forgetting, which means already acquired learning that students forgot or lost during school closures caused by disengagement with the education system. For more information, see World Bank et al (2021) and World Bank (2023a). LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 13 Learning in the Shadow of the Pandemic: 13 COVID-19 Learning Loss and Widening Learning Disparities in Indonesia 2 Contextual background: Indonesia’s Education System, Challenges and the COVID-19 Pandemic Indonesia’s education system reaches nearly 53 schooling, while students attend schools for 12.4 years million children and is delivered through two on average by 18 years old.8 53 percent of primary ministries. The formal education system in Indonesia school children suffer from learning poverty.9 Further, is governed by both the Ministry of Education, the learning outcomes as measured in PISA tests in Culture, Research and Technology (MoECRT) and the secondary education have not improved in the past Ministry of Religious Affairs (MoRA). Together they 20 years. The latest 2022 results of the Programme serve nearly 53 million children between Grades 1 to for International Student Assessment (PISA), released 12 and about 3.3 million teachers. The net enrolment in December 2023, marked a decline on average rate (NER)6 increased from 89 to 93 percent in primary compared to 2018, and 2022 results were among education and from 53 to 79 percent in secondary the lowest ever measured by PISA in all subjects in education between 2005 and 2018 as a result of the Indonesia, including language, mathematics and GoI’s commitment to expanding access to education.7 science.10 Moreover, equity issues also remain. Improvements in access to education have not GoI has implemented multiple interventions to been followed by increased quality, and inequities mitigate learning losses resulting from school remain. The GoI has made efforts to increase access closures during the pandemic. In Indonesia, to education for diverse populations, however, low approximately 53 million students between Grades learning quality has been a long-lasting education 1 to 12 were affected by the COVID-19-induced issue in Indonesia. Before the pandemic, actual nationwide school closures (full and partial) between learning was estimated to reflect only 7.8 years of March 2020 to December 2021. A total of 644 days of 6 Net Enrolment Rate means the ratio of children of official school age who are enrolled in school to the population of the corresponding school age group of children. 7 World Bank Data. 8 World Bank Data. 9 World Bank and UNESCO (2024). Learning poverty refers to being unable to read and understand an age-appropriate text by age ten. 10 OECD (2023) https://www.oecd.org/publication/pisa-2022-results/country-notes/indonesia-c2e1ae0e/ LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 14 " closure was relatively long compared to other MICs and Southeast Asian countries (UNICEF, 2021a; 2021b; World Bank, 2023a).11 As the The poorest 20 percent of COVID-19 cases increased, the GoI took measures to mitigate the impacts of the closures, including through issuing joint ministerial households experienced decrees on distance learning guidance12 and teaching and learning approximately two years during the pandemic,13 programs for hybrid, flexible and simplified curriculum, increased use of EdTech, provision of teacher support of learning loss in both and student subsidies for internet data, and student learning language and math, and assessment modules for online and hybrid learning (World Bank, 2022a). 14 Distance learning guidelines by the central government students with less educated provided different options, depending on school contexts and connectivity. Also, in line with the decentralized nature of the parents suffered from education system in Indonesia, some sub-national governments greater learning losses, developed their own platforms for online learning.15 including up to two years in However, service delivery has been a challenge, especially " language. concerning equity of education during the pandemic. Based on the HiFy survey,16 access to online schooling via internet and learning resources was limited, especially among students in the bottom 40 percent by income in outside Jakarta, due to no or limited internet access and lack of devices and lack of a supporting environment at home. While the GoI made efforts to improve interventions to address the issues with internet access, the effect of this on student learning outcomes has not been fully examined. Student dropouts were reported from the first year of COVID-19, mainly due to financial reasons, including household income drops and unpaid work. 17 By September 2021, 25,430 children in Indonesia reportedly had lost one or both parents due to COVID-19, and 20 percent of 15-24 years old reported that they had been depressed or lost interest in doing any activities.18 MoECRT has implemented a series of reforms to promote better quality learning. The Merdeka Belajar (Emancipated Learning) policy was launched in 2019 and became the main initiative for learning recovery in the face of large learning disruption due to 11 A joint regulation on school reopening guidelines was issued by MoECRT, MoRA, MoH and MoHA on December 22, 2021, that changed the standard for school operation from remote learning or partial opening into full opening. 12 Ministerial Circular Letter No. 4/2020 by MoECRT; Circular letter No B-686.1/DJ.I/ Dt.I.I/PP.00/03/2020 and Circular letter No B-699/Dt.i.i/ PP.03/03/2020 by MoRA. 13 GoI implemented various remote learning strategies including educational TV, radio programs, online teacher training, a platform for peer- to-peer communications among others. 14 More information on the response to COVID-19 by the GoI in the education sector can be found in Butcher et al (2021). 15 Suriastini et al (2020). 16 The HiFy survey is a high-frequency phone-monitoring survey of households conducted by the World Bank to collect data on the socio- economic impacts of the COVID-19 in Indonesia. Randomly selected between 3,000-4,000 panel households across the country participated in a 7-round panel survey between May 2020 and April 2022. Only around 40 percent of students reported using mobile learning apps/ accessed online schooling from May to June 2020. While MoECRT started the first internet data subsidy for students between September and December 2020, the proportion of students using online resources did not improve by November 2020 according to the survey. See World Bank (2023a) for more information. 17 UNICEF (2021b). 18 UNICEF (2021a). LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 15 " the pandemic.19 The so-called ‘Emancipated Curriculum’ or Kurikulum Merdeka is part of this reform, launched by MoECRT and adapted by MoRA since 2022, to improve learning outcomes. Pre-existing gender and It focuses on literacy, numeracy and character education. However, the effectiveness of the Merdeka Belajar reform and regional gaps in learning the new curriculum on mitigating learning losses remains to be examined. outcomes persisted during the pandemic, with Previous research on learning losses in Indonesia has mainly been restricted to estimation and early evaluation of the boys underperforming implications of school closures.20 More detailed empirical, girls, and rural schools national evidence is required. The recent analytical work by the World Bank (2023a) is one of the first studies that addressed underperforming urban this knowledge gap and analyzed COVID-19-associated learning losses at a national level in Indonesia after school reopening. schools in both 2019 and The study found that the Grade 4 students in primary schools in 2023 lost approximately 11 months in language and math, in comparison with students in 2019. Students from poor households suffered from quality education, losing 27.2 and 18.1 months of learning in literacy and numeracy, which 2023. " indicated a widened inequality in learning outcomes. The economic impact of learning loss is great, and the estimated lifetime loss of earnings will be 30.9 percent among men and 39.2 percent among women.21 Drawing on this foundational evidence of learning losses at a national level, more research is required to examine associated factors of learning losses at individual and structural levels to understand multi-level factors that relate to these losses. 19 Merdeka Belajar consists of 24 highlighted interventions, including Kurikulum Merdeka, new National Assessment, a series of programs called Penggerak that focus on improving quality of teachers and principals, among others (MoECRT, 2023). The new National Assessment measures literacy, numeracy character and learning environment, and started in 2021. The Penggerak program consists of Program Organisasi Guru Penggerak (to improve quality of teachers and school principals through training by NGO/development partners), Program Pendidikan Guru Penggerak (to produce instructional leaders who encourage student-centred learning), and Program Sekolah Penggerak (to improve competencies of existing school leaders and teachers). For more information, see http://merdekabelajar.kemdikbud.go.id While this study included the use of Kurikulum Merdeka as one of the factors related to the reform in the analysis, this report acknowledges the limitation of the analysis of the Merdeka Belajar reform given the context that the reform had only just started to be implemented at the time of data collection used in this report. 20 The exception to this has been studies conducted by the Innovation for Indonesia’s School Children (INOVASI) that identified learning gaps between what the standards set for students to learn and actual student attainment. See for example, the four-series Learning Gap study (INOVASI and MoECRT PSKP, 2022) and Sukoco et al (2023). 21 World Bank (2023a) LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 16 Learning in the Shadow of the Pandemic: 16 COVID-19 Learning Loss and Widening Learning Disparities in Indonesia 3 International Studies of Post-Pandemic Learning Loss 3.1 Magnitude of Learning Loss and Variance across Countries Meta-analyses and systematic reviews have intensifying in the subsequent year (Gambi & De Witte, consistently shown that the pandemic affected 2021; Maldonado & De Witte, 2022). Comparable student learning. The se findings showe d that the trends were observed in Germany and Italy, with magnitude of learning loss during the COVID-19 reduced test scores in reading and math (Schult et pandemic was approximately 0.14 to 0.19 standard al. 2022; Contini et al.; 2022). The United Kingdom, deviations across various countries (Betthäuser et al., with a substantial body of research on this topic, 2023; Di Pietro, 2023; Moscoviz & Evans, 2022). This consistently reported learning losses across primary significant learning deficit was observed not only in and secondary education (Blainey & Hannay, 2021). high-income countries such as the U.S., Germany, and The Unite d State s mirrored the se findings, with Italy but also in low- and middle-income countries national and state-level studies indicating pervasive including Mexico, Brazil, and South Africa. This decline learning losses (Domingue et al., 2022; Lewis et al., suggests that students have lost the equivalent of 2021; Pier et al., 2021). roughly 0.3 to 0.6 years of typical academic progress within a school year (Refer to Appendix I for country- MICs and LICs exhibited a greater variability in specific learning loss details). learning loss. In Bangladesh and Brazil, the effects were more pronounced, especially at the secondary Despite the presence of advanced remote learning level (Asadullah & Bhattacharjee, 2022; Lichand et infrastructure, significant learning losses were al., 2022). In Brazil, secondary students experienced observed in High Income Countries (HICs). For a significant decline in learning during 2020 as instance, in Belgium, standardized test performance compared to 2019 (Lichand et al. 2022). This research in language and math for primary students following indicated a reduction in test scores by 0.32 standard a nine-week closure declined, with language deficits deviations (SD), implying that students had attained LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 17 only about a quarter of the educational content they would have typically absorbed through traditional in- 3.2 Heterogeneous effects of person classes.22 Mexico's children also experienced substantial losses in reading and math (Hevia et al., COVID-19 on different social 2022), while in South Africa and Ethiopia, early grade groups primary students exhibited severe learning losses (Araya et al., 2022; Ardington et al., 2021; Shepherd The COVID-19 pandemic has exacerbated educational et al., 2021). In Thailand, the study also reveals disparities around the world, with pronounced that kindergarten children in lockdown provinces learning loss emerging particularly among children experienced significant losses in cognitive skills, from lower socio-economic strata. The literature particularly in mathematics and working memory, employs a variety of indicators to evaluate socio- compared to their peers in non-lockdown provinces economic status, including parental income, parental (Kilenthong et al., 2023). In Kenya, over half of education level, eligibility for free school meals, or the students in Grades 4 through 8 suffered math school socio-economic conditions (e.g., Blainey & skill losses (Whizz Education, 2021), while Uganda Hannay, 2021; Engzell et al., 2021; Lewis et al., 2021). presented a bimodal distribution of literacy outcomes Despite the differences in these measures, the findings (Sandefur, 2022; Uwezo Uganda, 2021). In Quanzhou, are consistent, and it is clear that students from less China, a study of mock exam scores among high affluent backgrounds have been disproportionately school students revealed a significant decline in impacted by school closures compared to their academic performance in 2020, compared to students more afflue nt pe ers. The disparity is multiface ted, from previous years, with a particularly pronounced encompassing more limited access to and proficiency deficit among those from rural and lower-income in digital learning technologies, lack of suboptimal households (Cai et al., 2023). A study in rural Tamil home learning environments, diminished educational Nadu, India found significant deficits after 18 months support from teachers and parents, and obstacles to of school closures, with primary school-aged children self-regulated learning. experiencing a learning loss of approximately 0.7 SD in mathematics and 0.34 SD in language (Singh et al., Studies in HICs observed widening achievement 2024). gaps between students from schools with low and high poverty rates. Spe cifically, in the United Overall, the pandemic has exacerbated global-level States, Kuhfeld et al. (2022) reported that the initial educational disparities, with MICs and LICs facing achievement gap of about 1 standard deviation in 2019 more pronounced challenges. MICs tended to have increased by an additional 0.20 standard deviations longer school closures and fewer resources for remote in math and 0.13 standard deviations in reading learning than HICs. The learning loss in MICs is notably among elementary school students. This expansion larger than in HICs, and the mean difference in learning represents an approximate increase of 20 percent in loss between these groups is statistically significant. math and 15 percent in reading. In the Netherlands, In conclusion, the pandemic has underscored the students with less educated parents experienced up fragility of educational progress in the face of global to 60 percent greater educational setbacks (Engzell crises, particularly for less affluent countries. e t al. 2021), and similar tre nds we re obse rved in Italy (Contini et al, 2022). This corre lation extends to other vulnerability indicators such as poverty (Haelermans et al. 2022). Belgian students from low socioeconomic backgrounds, whose parents have lower education levels and receive financial support, experienced learning losses ranging from 17 to 20 22 This cited study uses 0.44 standard deviations as a benchmark for traditional classroom learning to evaluate the effectiveness of remote learning during COVID-19. It calculates a reduction of 0.32 standard deviations, corresponding to a 72.5 percent loss (0.32/0.44 = 0.725). In other words, students attained only about a quarter (25 percent) of the educational content they would typically absorb through traditional in-person classes. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 18 percent compared to their peers from more affluent be minimal. Studies from the Netherlands (Engzell et and educated families, with the extent of these losses al., 2021; Haelermans et al., 2022), Ghana (Wolf et al., varying by subject (Maldonado and De Witte, 2020). 2021), and Mexico (Hevia et al., 2022) found that boys and girls suffered similar setbacks. No gender-based Learning inequalities among students in MICs also disparity was observed in Uganda (Sandefur, 2022). became wider. For instance, in Mexico, low socio- However, preliminary data from Pakistan suggested economic status students lost more than twice the that boys might be experiencing greater learning loss reading ability compared to their wealthier peers than girls (Crawfurd et al., 2021). (Hevia et al., 2022). Between 2019 and 2021, low-SES students exhibited a decline of 0.47 SD in reading, There are indications of increased dropout rates while high-SES students showed a drop of 0.22 SD, due to the need for children to earn income. The highlighting a disproportionate impact on lower-SES pandemic-induced educational disruptions have learners. In Pakistan, the poorest children fell behind in also exacerbated the dropout risk among older subtraction skills, unlike their wealthier counterparts students, particularly in low-income countries where (Crawfurd et al., 2021). However, it is important to these students often shoulder greater economic note the lack of evidence from MICs, which may face responsibilities (e.g., Abre h e t al., 2021; Mbaye e t unique challenges that are not as well-documented or al., 2021; Zulaika et al., 2022). The financial setbacks understood due to disparities in data collection and caused by COVID-19 have compelled many households reporting. to prioritize immediate income generation over education, further jeopardizing their academic Other heterogeneous effects, including those continuity. However, the scarcity of available data on associated with age, gender, and ethnicity, remain learning deficits constrains our ability to ascertain inconclusive and underscore the need for further with certainty whether there are differential impacts research to investigate these dimensions. Gender between primary and secondary students on learning. differences in learning loss were generally found to "The pandemic-induced educational disruptions have also exacerbated the dropout risk among older students, particularly in low-income countries where these students often shoulder " greater economic responsibilities LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 19 Learning in the Shadow of the Pandemic: 19 COVID-19 Learning Loss and Widening Learning Disparities in Indonesia 4 Objectives and Methodologies of the Study 4.1 Objective of the study The primary purpose of this report is to support language-related barriers, regional variations, and the GoI in improving their evidence-based policy gender-related challenges contribute to learning interventions. Additionally, this report contributes to deficits. This comprehensive approach enhances the global debate on learning losses due to COVID-19 our understanding of educational disparities and associated school closures by contributing to the provides policymakers with insights to design following knowledge generation. targeted interventions addressing linguistic diversity, regional inequalities, and gender- • Enhancing Understanding in MIC Contexts: responsive education policies, not only in This study's concentration on Indonesia, a Indonesia but also in similar contexts worldwide. middle-income country with unique challenges and resources, serves as a valuable addition • Contextual Interventions: The study's focus on to the literature. The findings can inform not investigating interventions, including teacher only Indonesia's educational policies but also support, learning modes, and student-teacher guide other low- and middle-income nations in interactions, addresses a critical gap in the mitigating the unde sirable consequences of the literature. It acknowledges the limited existing pandemic-induced school closures in education. evidence focusing on such interventions and emphasizes the necessity of designing strategies • Heterogeneity of Learning Disparities: This study tailored to the challe nges and re source s within significantly contributes to the existing literature the Indonesian educational context. This practical by exploring the heterogeneity of learning approach could benefit Indonesia and serve as a disparities among Indonesian students, with a model for other middle-income countries facing particular focus on three overlooked dimensions: similar challenges. It underscores the importance socioeconomic disparities, regional disparities, of adapting interventions to suit the unique needs and gender-specific differences during the of each region or country, which could support pandemic. By e xamining the se spe cific aspe cts, the educational recovery efforts in the post-pandemic research offers a nuanced understanding of how era. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 20 This study proposes two main research questions. RQ 2. What factors are associated with students' (1) To what extent has the COVID-19-induced school learning outcomes during and after the COVID-19 closure influenced student learning loss in language pandemic? and mathematics in Indonesia? and (2) What factors o RQ 2.1: Which practices by teachers and principals have associated with students' learning outcomes are associated with learning outcomes? during and afte r the COVID-19 pande mic? The first research question examines the extent of learning o RQ 2.2. Which types of government support are loss during the COVID-19 pandemic among Indonesian linked to student learning outcomes? students in language and mathematics by comparing o RQ 2.3: Which individual-level factors correlate 2023 and 2019 data. This study builds on previously with learning outcomes? published work using the same dataset (World Bank, 2023a), and enhances the robustness of the technical approach, e xtending the scope of the analysis. The secondary research question aims to investigate key 4.2 Data and Sampling Process factors associated with student learning outcomes during and after the COVID-19 pandemic. Structural This research employs data from two distinct level factors are categorized into two principal phases: surveys: the Service Delivery Indicators (SDI) survey during and post-pandemic. (i) In the pandemic phase, conducted in 2019 and the Learning Loss Survey attention centers on school environments (e.g., school carried out in 2023. The SDI survey provided a near- area, internet access), and the practices adopted by nationally representative23 assessment of student teachers and principals when schools were closed; learning outcomes in language (Bahasa Indonesia) (ii) In the post-pande mic phase , the focus shifts to and math at Grade 4 level, serving as a pre-pandemic the teaching practices after schools reopened, as benchmark for educational attainment in this study. measured by the Teach classroom observation tool in The Learning Loss Survey of 2023 in Indonesia builds early 2023. Individual level factors focus on students’ upon the foundation established by the SDI 2019. The characteristics such as their socioeconomic status, and 2023 survey replicated the key features of the 2019 illness or death of students’ families and somebody survey, including revisiting the same educational close during the pandemic. The study endeavors to institutions to allow for a direct comparison of Grade provide a nuance d unde rstanding of the diffe rent 4 student performance and expanding the sample educational outcomes across various socio-economic to include more schools for national representation. strata and institutional contexts. In addition, the Learning Loss Survey 2023 was conducted during the same February to March period Research Question (RQ) 1. To what extent have as the SDI survey, allowing for a controlled comparison the COVID-19-induced school closures influenced of student performance over the four-year interval student learning loss in language and mathematics in between 2019 and 2023. The survey used identical Indonesia? assessment tools for language and mathematics, and the test administration guidelines were followed as in o RQ 1.1. To what extent did the COVID-19 induced the previous survey. school closures influence learning loss for Indonesian students in 2023 compared to students Furthermore, the 2023 survey was enhanced with additional modules for school principals, in 2019? teachers, and students to gather multidimensional o RQ 1.2. Has learning disparity across schools and information related to learning during the individuals been exacerbated? pandemic. The additional modules aimed to assess teaching practices by using the World Bank’s standard 23 See Appendix A for more details. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 21 classroom observation instrument, Teach, and the multiple imputed scores for each student, addressing support provided by government during COVID-19 measurement error and variability in true abilities (See school closures. Additionally, they captured the Appendix B). The study standardizes these outcomes characteristics of students at the household level, to have a mean of zero and a standard deviation of one, such as illness among close relatives and increased based on the 2019 data, which serves as a baseline household chores during the pandemic. for evaluating the performance of the 2023 cohort against the backdrop of educational disruptions Both surveys employed a random selection process caused by COVID-19 (e.g., Hevia et al., 2022; Kuhfeld et to choose ten Grade 4 students from each school. al., 2022). This standardization approach, employing Consistently, one Grade 4 teacher, typically the class z-scoring, allows for a direct comparison between teacher, was selected for observation and interview, the two cohorts, offering insights into any deviations aligned with the student sample. However, in schools in academic performance over time, potentially employing subject-based teaching methods, two attributable to the pandemic (See Appendices C for the Grade 4 teachers were selected for observation and description of variables used and average test scores interviews (See Appendix A for technical notes on for groups of each characteristic). sampling, matching, weighting, and the analytical approach of this study). School-level variables To assess the educational disruptions caused by the The school-level variables can be divided: (i) school pandemic, this research compares the performance environment, including ownership, location, of the 2023 cohort with the baseline established administrative oversight, and the availability of by the Grade 4 student outcomes in 2019. It aims internet access before and during COVID-19; (ii) to uncover the extent of learning loss by comparing practices adopted by teachers and principals during the two cohorts while considering variations due COVID-19 school closures; and (iii) post-pandemic to different educational institutions and individual teaching practices after school closures, as evaluated characteristics such as socioeconomic background. by the Teach classroom observation tool.23 This approach is intended to provide insights into the varied relationships between the school closures due School Environment: School-related variables such as to the pandemic and student learning at a national type, ownership and area are considered. The type of level. school is categorized based on administrative oversight (MoECRT, MoRA), while ownership is distinguished between public and private. Administrative oversight 4.3 Measures by MoECRT versus MoRA is characterized by differences in curricula, resources, and teaching methods applied Outcome variables in response to the pandemic. Ownership distinguishes between public and private schools, reflecting Learning Outcomes: In this study, learning outcomes disparities in funding, resource availability, and are asse ssed through language and math score s, potentially the educational experience, relating to reflecting students' academic proficiency. Plausible student learning. The school area, categorized as rural Values (PV) scores, derived using Item Response Theory or urban, underscores environmental differences, with (IRT), estimate student abilities by analyzing their urban schools often having better access to resources responses to selected test items and incorporating and rural schools facing challenges such as limited background information.22 This method generates educational materials and connectivity. 22 The World Bank (2023a) used the standardized raw score without using IRT method. The advantage of using IRT is its ability to provide more precise estimates of individual abilities by taking into account the varying difficulty levels of each item. 23 See Appendix G for more details. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 22 Pre-pandemic Internet Access: The availability of dynamic interactions between students, teachers, and internet access prior to the pandemic is pivotal in administrators, highlighting the operational aspects identifying disparities in digital accessibility, which that contribute to student learning experiences. carry profound implications for remote learning. It enables an evaluation of how pre-pandemic internet Pandemic Government Support: During the COVID-19 access, or the lack thereof, could influence individuals’ pandemic, the Ministry of Education, Culture, Research ability to engage in educational activities that shifted and Technology (MoECRT) and the Ministry of Religious online due to health guidelines and restrictions. Affairs (MoRA), implemented a series of measures to support educational continuity and recovery for Pandemic School Practices During School Closures: schools. This metric covers the support provided School practices by teachers and principals include by government to schools during school closures, several key metrics aimed at assessing instructional including internet connectivity, online learning and administrative strategies during the pandemic. resources, IT hardware, etc. These variables examine First, the frequency of assignments is quantified by whether such support have mitigated students’ counting the number of assignments distributed learning loss (See Appendix C). to stude nts within a single we ek during the school closures, providing insight into the academic workload Post-Pandemic Teaching Practices: While this study and expectations placed on students. The assignment assesses school environment and practices during the submission time distinguishes between assignments pandemic, teaching practices after school reopening that teachers require students to submit on the same are also observed and provide valuable information. day they are assigned (coded as 1) and assignments The se we re evaluate d in the Le arning Loss Surve y that allow for a longer completion timeframe (coded through the Teach classroom observation tool in early as 0). Lastly, the principal’s monitoring frequency is 2023, designed to assess the quality of teaching. This measured by identifying whether the principal engages tool focuses on three critical dimensions: Classroom in daily oversight of academic and administrative Culture, Instruction, and Socio-emotional Skills (World activities (coded as 1) or if such monitoring occurs Bank, 2022b). The se are as are asse ssed on a scale less frequently (coded as 0). This metric sheds light on from 1 (lowest) to 5 (highest). The study leverages the the level of administrative engagement and oversight scores across these Teach domains to investigate the within the school, potentially relating to the school’s relationship between teaching practices and student academic climate and operations. Collectively, these learning outcomes (See Appendix G for more details measures of school practices aim to capture the about Teach). LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 23 Individual-level variables Household Background: In assessing socioeconomic Additional measures: The study also examines the factors, this research primarily focuses on two association between children’s experiences with key elements: the education level of parents and the death or illness of close ones (family members, the household asset index as a proxy for student relatives) during the pandemic and student learning economic background. Parental education, identified outcomes. Moreover, the study considers children’s by the educational level of both parents, is divided involvement in household economic activities, such into four different categories: those with primary as farming or selling goods, to assess how their education or less, lower secondary education, upper involvement in these activities associate with their secondary education, and those with tertiary or higher educational achievements. These elements are education. This study also employs the quintiles of examined to grasp the broader context influencing the house hold asse t index, ranging from the first learning, exploring how external pressures and quintile (representing the lowest economic status) emotional experiences might be associated with to the fifth quintile (indicating the highest economic academic performance. This study also incorporates status). The house hold asse t index was de veloped a measure of the number of books read as a metric to using data from the Indonesia Socioeconomic Survey evaluate the association between reading habits and 2022 (SUSENAS), a nationally representative survey of learning outcomes. households in Indonesia. The index was established using eleven household asset variables, such as fridges Individual Background: In this study, individual and smartphones, that were consistent across both backgrounds are explored through a combination of surveys. This index serves as a proxy for the economic educational exposure and gender. The educational status of the students’ households. Additionally, the aspect is reflected in preschool attendance, recorded study includes the number of household members. as a binary variable to indicate whether a student has had the advantage of early educational engagement. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 24 Learning in the Shadow of the Pandemic: 24 COVID-19 Learning Loss and Widening Learning Disparities in Indonesia 5 Analytical Results of Learning Losses between 2019 and 2023 5.1 Estimating Learning Loss in Language and Mathematic Skills To e stimate the magnitude of le arning loss ove r identified through the utilization of PV scores in this the period, this research employed standardized study closely aligns with findings from our previous PV scores of 2019 as a benchmark. This approach report (World Bank, 2023a),26 underscoring the facilitated a comparative analysis of student robustness and reliability of PV scores as a metric academic performance between the years 2019 and for assessing educational outcomes. Considering 2023, providing a robust counterfactual framework the international benchmark, which sets the average for evaluating the extent of learning loss within this annual learning progression in a developing country timeframe. at 0.300 SD27, Indonesian students have experienced learning losses equivalent to approximately 10.6 Learning losses in Indonesia are substantial, in months in language and 11 months in mathematics. parallel with findings in other MICs. Table 1 shows Give n that the data we re collecte d in early 2023, a notable decline in both language and mathematic following the full reopening of schools after a competencies post-COVID-19. Statistically, student 21-month closure, our findings align with previous language outcomes in 2023 exhibited a significant case studies (INOVASI 2021;2022), which indicated reduction of 0.265 standard deviations (SD) from a delay in the learning progression from Grade 1 to those 2019, while the mathematic score demonstrated Grade 2 of approximately five to six months, observed a decrease of 0.276 SD. The degree of learning loss one year after the school closures in Indonesia. 26 The previous report revealed that the national average of Grade 4 student performance in language in 2023 was 0.271 SD lower than the average score of Grade 4 students in 2019, and math was also 0.281 SD lower. 27 See for example, Woessmann, 2016. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 25 Table 1. Estimated Learning Loss between 2019 and 2023 Language Mathematics PV scores Standardized Z-score PV scores Standardized Z-score Post-COVID -0.265*** -0.273*** -0.276*** -0.284*** (0.056) (0.054) (0.058) (0.060) Constant 0.000 0.000 0.000 0.000 (0.048) (0.048) (0.058) (0.058) N 6,693 Source: Authors’ analysis using SDI Survey 2019 and Learning Loss Survey 2023 Note: Standard errors are corrected for clustering within schools. Both scores are standardized with a mean of 0 and a standard deviation of 1. Standard errors are included in parentheses. * p<0.05, ** p<0.01, *** p<0.001. The learning loss between 2019 Figure 1. Comparison of 2019 and 2023 Score Distributions by Subject and 2023 obtained in our surveys is slightly larger in math than language Language 0.5 although the language score distribution seems have a longer 0.4 2019 tail on the lower end.28 The pattern 2023 of larger learning loss on math than 0.3 language is consistent with systemic and meta-analytic studies (Betthäuser 0.2 et al., 2023; Di Pietro, 2023). This could be attributed to parents’ greater ability 0.1 to assist their children with reading and the potential for children to enhance 0 their reading skills through leisure -4 -2 0 2 4 reading outside of school, whereas similar opportunities for improving Mathematics 0.5 mathematical skills may be limited. 0.4 2023 2019 2019 Howe ver, Figure 1 shows that math score distribution shifted towards left 2023 (lower) overall whereas the language 0.3 score distribution be came flatte r in 2023 than 2019 with a longer tail on 0.2 the le ft (lowe r side). This hints that all students, irrespective of whether 0.1 they are top-performing or bottom- performing, were affected on math, 0 -4 -2 0 2 4 but on language, the top-performing group was less affected, while students Source: Authors’ analysis using SDI 2019 and Learning Loss Survey 2023 who were on the middle to lower Note: The figure presents two subplots comparing the distributions of distribution curve were more affected. language and math scores for the years 2019 and 2023. Each curve represents (See Appendix H for more discussion the probability density function of the standardized PV scores, with weighted of score distribution for 2023). means and standard deviations used to account for sample weights. 28 The two-tailed t-test comparing regression results in two subjects using standardized PV scores revealed a δ difference of -0.011 (p < 0.001, χ² = 31.58), indicating significant disparities between subjects. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 26 5.2 Learning Loss Disparities Across Socioeconomic Backgrounds Socioeconomic disparities in learning outcomes closures and remote learning, has affected both have widened. Students from economically genders similarly. Our study identified no significant disadvantaged backgrounds experienced larger change in gender disparities. learning losses. Table 2 shows learning outcomes among students across various household asset Regarding regional disparities, both urban and quintile groups from 2019 to 2023. Students from rural students experienced significant learning loss. the poorest 20 percent of households had a notable During the pandemic (Table 2), rural students saw a drop in their school performance. Statistically, they decline of -0.308 SD in language scores (from -0.214 lost 0.571 SD in language skills, which corresponds to -0.522), while urban students experienced a similar to nearly two years of learning in language. Similarly, loss of -0.266 SD (from 0.136 to -0.130). In math, rural students in the lowest quintile experienced a loss of students saw a decline of -0.273 SD (from -0.284 to 0.440 SD in mathematic skills, equivalent to over one -0.557), while urban students experienced a similar and half years of learning loss. On the other hand, loss of -0.313 SD (from 0.180 to -0.133). The findings students from the richest 20 percent did not have suggest that the educational setbacks resulting from statistically significant difference in scores (although school closures were comparably experienced across the re point e stimate shows lowe r numbe rs). Such rural and urban settings. Nonetheless, entrenched findings suggest that disparities in learning outcomes disparities in academic achievement between these have widened by students’ socio-economic status. demographic groups endured, underscoring the persisting educational inequities. A consistent trend was observed with parental education levels, where students with less educated " parents encountered substantial drops in their learning. For instance, children whose mothers or fathers have primary education or less experienced The magnitude of learning loss the highest loss in language scores, with a -0.609 SD loss for mothers and a -0.525 SD decline for fathers, is relatively consistent across corresponding to 24 and 21 months of learning, genders. Concerning the trends respectively. These findings underscore the profound role of parental socioeconomic status on children’s in gender difference, the result learning during periods of crisis, highlighting the need reveals that both male and for targeted interventions. female students experienced The magnitude of learning loss is relatively consistent across genders. Concerning the trends in a significant decline in both gender difference, the result reveals that both male language and mathematics and female students experienced a significant decline in both language and mathematics performance performance from 2019 to " from 2019 to 2023. Males showed a marginally greater 2023. reduction in language performance than mathematics, with scores dropping by -0.277, compared to a -0.256 decrease in mathematics. Conversely, females exhibited a more substantial decline in mathematics, with a -0.301 drop, than in language, where scores fell by -0.252. However, the magnitude of learning loss observed remains relatively consistent across genders, indicating that the challenges posed by the COVID-19 pandemic, including factors such as school LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 27 Table 2. Average language and mathematics scores among different social groups, 2019 and 2023. Language Mathematics 2019 2023 Diff 2019 2023 Diff (SE) Mean (SD) Mean (SD) Mean (SD) Mean (SD) Household Asset Quintile 1-Lowest -0.28 -0.85 -0.571** -0.242 -0.683 -0.440** (0.814) (1.309) (0.114) (0.848) (1.014) (0.108) 2 -0.317 -0.482 -0.165 -0.399 -0.481 -0.081 (0.991) (1.184) (0.221) (0.961) (1.004) (0.181) 3 -0.122 -0.255 -0.133 0.099 -0.333 -0.433** (1.112) (1.087) (0.153) (0.915) (0.895) (0.114) 4 0.309 -0.111 -0.420** 0.156 -0.152 -0.308 (0.742) (1.015) (0.117) (0.966) (0.947) (0.178) 5-Highest 0.242 0.139 -0.103 0.138 0.065 -0.073 (0.774) (0.965) (0.111) (0.893) (0.943) (0.133) Father’s Education Primary or less -0.255 -0.781 -0.525** -0.328 -0.677 -0.349** (1.113) (1.342) (0.148) (0.863) (0.977) (0.109) Lower Secondary -0.106 -0.408 -0.302** -0.061 -0.403 -0.342** (0.743) (1.096) (0.107) (0.877) (0.932) (0.118) Upper Secondary 0.062 -0.043 -0.105 0.168 -0.127 -0.295* (0.901) (0.976) (0.128) (1.004) (0.941) (0.136) Tertiary 0.356 0.337 -0.020 0.446 0.254 -0.192 (0.716) (0.821) (0.132) (0.881) (0.915) (0.157) Mother’s Education Primary or less -0.136 -0.745 -0.609** -0.376 -0.68 -0.304** (1.117) (1.316) (0.137) (0.801) (0.981) (0.100) Lower Secondary -0.016 -0.376 -0.360** 0.157 -0.411 -0.567** (0.792) (1.130) (0.122) (0.907) (0.940) (0.113) Upper Secondary -0.04 -0.067 -0.027 0.115 -0.125 -0.240 (0.822) (0.983) (0.116) (0.930) (0.913) (0.134) Tertiary 0.188 0.326 0.139 0.423 0.299 -0.124 (0.905) (0.813) (0.149) (0.945) (0.899) (0.151) Gender Male -0.183 -0.46 -0.277** -0.156 -0.412 -0.256** (1.075) (1.218) (0.087) (1.026) (1.023) (0.075) Female 0.181 -0.071 -0.252** 0.154 -0.146 -0.301** (0.883) (1.047) (0.058) (0.949) (0.943) (0.080) Area Rural -0.214 -0.522 -0.308** -0.284 -0.557 -0.273** (1.068) (1.179) (0.070) (0.995) (1.000) (0.072) Urban 0.136 -0.13 -0.266*** 0.18 -0.133 -0.313** (0.929) (1.114) (0.072) (0.960) (0.957) (0.076) Total 0 -0.265 -0.265** 0 -0.276 -0.276** (1.000) (1.152) (0.056) (1.000) (0.993) (0.060) Source: Authors’ analysis using SDI Survey 2019 and Learning Loss Survey 2023 Note: Weighted descriptive statistics consider variations in student and school demographics across institutions. Regression analysis, using weighted methods and clustering standard errors by school, accounts for these demographic differences. “Diff in Mean” shows score differences between periods, indicating performance trends. For example, a “Diff in Mean” of -0.571 shows the degree of learning loss when comparing the lowest quintile groups from 2019 to 2023. * p<0.05, ** p<0.01, *** p<0.001. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 28 5.3 Learning Loss Disparities Related to School Environments Varying degrees of learning loss were observed faced larger and statistically significant learning losses across MoRA and MoECRT schools as well as between in both language and math, with decreases on average public and private schools.29 Schools governed by by 0.326 SD in language and 0.315 SD in math (See different educational structures, such as public versus Table 3). MoRA schools also experienced declines private ownership status and school type (MoRA in both subjects, albeit with less pronounced drops versus MoECRT), may have had varying responses than MoECRT schools, with 0.104 SD in language and to and been influenced differently by the COVID-19 0.235 SD in math, corresponding to four and nine school closures. Table 3 shows that public schools months, respectively. The results show that students witnessed significant declines in both language and under MoECRT schools experienced a more significant math competencies, with a loss of 0.326 SD and 0.282 de cline in language outcome during the transition SD, respectively. Private schools also faced significant from 2019 to 2023 than their MoRA counterparts. The substantial declines of 0.295 SD in mathematic analysis of 2023 data alone also show that the average outcomes, mirroring the challenges faced by their score of students at MoECRT is lower in language, public school counterparts. The analysis by school type although the difference is not statistically significant indicates that students enrolled in MoECRT schools in math. Table 3. Analysis of Learning Loss by School Factors: Comparing Pre- and Post-COVID-19 Periods Language Mathematics 2019 Mean 2023 Mean Diff in mean 2019 Mean 2023 Mean Diff in mean (SD) (SD) (SE) (SD) (SD) (SE) School status Public -0.031 -0.358 -0.326*** -0.043 -0.326 -0.282*** (1.003) (1.158) (0.066) (0.996) (0.986) (0.066) Private 0.109 -0.019 -0.128 0.151 -0.144 -0.295* (0.981) (1.098) (0.090) (1.000) (0.955) (0.115) Diff in mean 0.14 0.338*** 0.194 0.182* (SE) (0.093) (0.087) (0.121) (0.085) School type MoECRT -0.005 -0.32 -0.315*** 0.008 -0.28 -0.288*** (1.003) (1.178) (0.071) (1.005) (0.982) (0.074) MoRA 0.017 -0.087 -0.104* -0.027 -0.262 -0.235*** (0.988) (1.042) (0.044) (0.983) (0.976) (0.049) Diff in mean 0.022 0.233** -0.035 0.018 (SE) (0.078) (0.071) (0.091) (0.073) Source: Authors’ analysis using SDI Survey 2019 and Learning Loss Survey 2023 Notes: MoRA schools include madrasahs and other religious institutions as described in Appendix A. Weighted descriptive statistics consider variations in student and school demographics across institutions. Regression analysis, using weighted methods and clustering standard errors by school, accounts for these demographic differences. “Diff in Mean” shows score differences between periods, indicating performance trends. * p<0.05, ** p<0.01, *** p<0.001 29 While this report used weighted methods to ensure that the data credibly represent both ministries nationwide, this report also acknowledges the limitation of the dataset. See the Appendix A for a detailed explanation of techniques used in this report. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 29 Changes in between-school and within-school variances signify inequalities and differences in " educational outcomes over the years. Between- school variance points to inequities in resources Disparities in and educational quality across different schools, while within-school variance can reveal disparities in language outcomes student achievement within a single school. have become more A widening disparity in language outcomes among students is evident from increases in both within- school and between-school variances. Table 4 shows an increase in these variances, suggesting that disparities in language outcomes have become more pronounced over time. Specifically, the percentage pronounced over time. " of between-school variance for language rose slightly from 22 percent to 24 percent, indicating that nearly a quarter of the variability in literacy outcomes can be attributed to differences between schools. In contrast, mathematic outcomes show relative stability in between- and within-school variances, with a slight reduction from 2019 to 2023. Despite this, the share of between-school variance in mathematics (27 percent) remains higher than that of language (24 percent), pointing to a sustained disparity in the quality of math education among different schools. Table 4. Between-school and Within-school Variances on Students’ Learning Outcome by Year 2019 2023 Total Language Within-school variance 0.85 1.01 0.91 Between-school variance 0.23 0.32 0.27 Total 1.08 1.33 1.01 % share of between-school variance 22% 24% 23% Mathematics Within-school variance 0.75 0.73 0.74 Between-school variance 0.31 0.27 0.30 Total 1.06 1.00 1.04 % share of between-school variance 30% 27% 29% Source: Authors’ analysis using SDI Survey 2019 and Learning Loss Survey 2023 Note: Weighted descriptive statistics consider variations in student and school demographics across institutions. The sample includes 293 schools that were observed in both 2019 and 2023, ensuring a consistent basis for comparing changes over time. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 30 Learning in the Shadow of the Pandemic: 30 COVID-19 Learning Loss and Widening Learning Disparities in Indonesia 6 Analysing the Characteristics Related to Learning Disparities in 2023 This section explores the school-, teacher- and individual- level characteristics that are associated 6.1 Correlations between School with student learning outcomes following the school Practices and Students’ closures prompted by the pandemic. First, this section examines how school practices and government Learning support during school closures were associated with the le arning disruptions cause d by the closure s. To begin with teaching practices during the COVID-19 Furthermore, we investigate how individual-level school closures, the results show that frequent factors are associated with learning performance, such assignments provided by teachers are associated as the loss of a family member or close acquaintance, with improved learning outcomes in both language and engagement in household chores. We utilized HLM and mathematics across the models in 2023. Holding to account for the hierarchical data structure within other variables constant, the allocation of one schools, where shared experiences, teachers, and more assignment within a week is found to increase resources are linked to student performance.30 Results acade mic outcome by about 0.045 in language and are presented in Table 5. This HLM regression analysis 0.056 in mathematics (Column (4)). This improvement starts by examining data at the individual student is substantial, equivalent to an increase of 1.6 to level (1), moves on to factors at the school level (2), 2 months of learning. This insight highlights the incorporates various types of government support significance of maintaining consistent engagement (3), and ultimately integrates all these aspects in the through assignments, emphasizing their role in comprehensive full model (4). Appendix D provides the improving academic achievement during disruptions average test scores of 2023 by different background to standard classroom instruction. Previous research characteristics that are used for the analysis in this has shown that assignments can significantly enhance section. learning outcomes in remote settings (Dynan & Cate, 30 This study validated the use of the HLM model in our empirical models by conducting a likelihood ratio test and measuring the intra-class correlation (ICC). Additional details can be found in Appendix B. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 31 2009; Rayburn & Rayburn, 1999). This underscores the In our survey, about 42 percent of principals importance of assignments that foster student-teacher monitored student learning daily, highlighting a interaction in addressing the challenges of remote high level of engagement and oversight. The findings education during the COVID-19 pandemic (Yuliansyah indicate a positive correlation between the frequency & Ayu, 2021). of these principal monitoring and improved language outcomes (Model 2 in Table 5). This suggests that In our survey, the approximately 33 percent of more active and frequent involvement of principals in students who reported that their teachers required overseeing instructional practices is linked with better them to submit their assignments on the same language scores among students. These results imply day performed better, particularly in language. that frequent monitoring of teachers can contribute This prompt submission requirement enhanced positively to student learning outcomes (World Bank, student engagement in learning during the COVID-19 2020a). However, the introduction of other variables school closures. The students who were subjected to into the analysis re duce s the statistical significance immediate submission deadlines exhibited notably of this relationship, but shows a positive association, better outcomes in language studies, with a significant suggesting that principal monitoring on language score increase (0.121) compared to students who achievements might be associated with additional did not have such submission mandates (Column 4). factors. This improvement, equating to an extra 4.8 months of learning, underscores how learning outcomes Internet access prior to school closures is also for students subjected to same-day assignment associated with better learning outcomes. It is evident submission can be considerably enhanced, that pre-existing internet connectivity in educational highlighting the positive role of stringent submission settings enabled a more efficient transition to remote policies on educational achievement. A similar trend learning modalities during the COVID-19 pandemic, was observed in mathematics, but the association was thereby mitigating potential disruptions to academic marginal and did not reach statistical significance. continuity (Azubuike et al., 2021; Fauzi & Khusuma, 2020). Consistent with previous findings, the results Principals’ monitoring of teaching during school show that having access to the internet before the closures was important for their students’ learning as pandemic was associated with better performance it ensured accountability and quality in the delivery in both language and math subje cts, e ven afte r of remote education. By actively overseeing teacher accounting for individual, school, and government- activities, principals can help maintain instructional standards, provide timely support, and identify areas level variables. Statistically, access to the internet prior where additional resources or adjustments may be to the pandemic increased learning outcomes by 0.223 needed, ultimately benefiting students (Kafa, 2023; in language and 0.143 in math, which corresponds to an Tan, 2023). Principals’ monitoring activities of teaching advancement of approximately 8.9 months in language during the pande mic we re varie d in Indone sia, skills and 5.7 months in mathematical proficiency (See including random drop-ins on online classes, presence Model 4 in Table 5). in classroom messaging groups, checking assignments or assessments via online, and direct communication or meetings with teachers. Table 5. HLM Regression Analysis of School and Individual Factor on Learning Outcomes in 2023 Language Mathematics (1) (2) (3) (4) (1) (2) (3) (4) Individual School Government Full Individual School Government Full I. Individual Level Student’s gender (Female=1, Male=0) 0.356*** 0.351*** 0.261*** 0.257*** (0.046) (0.046) (0.038) (0.038) Household size -0.068*** -0.065*** -0.022** -0.020* (0.016) (0.015) (0.011) (0.011) Attending Preschool (Yes=1, No=0) 0.150* 0.139 0.046 0.036 (0.085) (0.088) (0.084) (0.085) Close One’s Death (Yes=1, No=0) -0.172*** -0.166*** -0.158*** -0.153*** (0.051) (0.052) (0.039) (0.041) Help house chores (Economic=1, No or non-economic=0) -0.076 -0.075 0.020 0.018 (0.062) (0.061) (0.047) (0.048) Number of Books Read during Last Month 0.021*** 0.020*** 0.014** 0.013** (0.006) (0.006) (0.006) (0.006) II. School Level School area (Urban =1, Rural=0) 0.106** 0.114** 0.220*** 0.229*** (0.053) (0.053) (0.052) (0.053) Pre-COVID Access to Internet (Yes=1, No=0) 0.276*** 0.223*** 0.150*** 0.143** (0.063) (0.063) (0.058) (0.058) Assignment frequency (Number of assignments in a week) 0.052*** 0.045*** 0.056*** 0.051*** (0.017) (0.016) (0.014) (0.015) Assignment submission time (On the same day=1, Otherwise=0) 0.138*** 0.121*** 0.015 0.007 (0.046) (0.043) (0.043) (0.042) Principals monitoring frequency (Everyday=1, Not Every day =0) 0.088* 0.058 0.084 0.064 (0.053) (0.052) (0.052) (0.052) III. Government support during the pandemic Internet Connection (Yes=1, No=0) 0.192*** 0.113** 0.129* 0.065 (0.058) (0.053) (0.066) (0.064) Online learning resources (Yes=1, No=0) 0.215*** 0.051 0.131 -0.017 (0.083) (0.078) (0.087) (0.089) IT Hardware (Yes=1, No=0) -0.050 0.001 0.091 0.100* (0.061) (0.057) (0.065) (0.059) LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA Family SES as control Yes Yes Yes Yes Yes Yes Yes Yes Constant -0.782*** -1.360*** -0.843*** -1.247*** -0.794*** -1.200*** -0.777*** -1.251*** (0.133) (0.080) (0.063) (0.132) (0.107) (0.068) (0.051) (0.104) Between province and school variance Between schools 0.285*** 0.260*** 0.249*** 0.243*** 0.268*** 0.237*** 0.230*** 0.228*** (0.022) (0.018) (0.021) (0.017) (0.019) (0.016) (0.019) (0.016) Within schools 0.861*** 0.900** 0.854*** 0.852*** 0.615*** 0.630*** 0.657*** 0.609*** (0.046) (0.048) (0.040) (0.045) (0.021) (0.021) (0.024) (0.020) Source: Authors’ analysis using Learning Loss Survey 2023 Note: The HLM analysis use weighted methods and clustering standard errors, accounts for these demographic differences. * p<0.05, ** p<0.01, *** p<0.001 See Appendix E for the alternative specification for variables related pre-COVID internet access. 32 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 33 6.2 Examining learning disparities by school practices during the pandemic The evidence from the HLM analysis illuminates key Moreover, urban schools received more support teaching practices and school factors that enhance from Government during the pandemic compared student learning outcomes. The key factors identified to rural schools. Urban schools received an average include the responsibility of teachers for distributing of 2.8 forms of assistance, compared to 2.4 in rural and collecting homework, the supervision of teaching schools. Urban schools re ceive highe r pe rcentage s activities by principals, the availability of distance of ministerial support in most categories compared learning infrastructure, such as internet access before to rural schools, except for additional funds. Notably, the pandemic, and Government support for distance urban schools demonstrated higher percentages of learning resources during the pandemic. Drawing on support in critical areas for remote learning, such as these findings, this section examines which schools internet connection, online learning resources, and IT demonstrated key teaching practices and had access hardware, with disparities of 7, 12, and 1 percentage to adequate remote learning resources. Table 6 point, respectively. These disparities are pivotal as illustrates the distribution of key teaching practices they directly influence educators’ practices to interact and G=government support across school areas with students, such as providing and monitoring between urban and rural areas. assignments, which consequently makes differences in students’ learning outcomes. This highlights the disadvantages that rural students faced during the 6.2.1. Urban-Rural Disparities pandemic period. in School Practices and Government Support Urban teachers assigned more homework than their rural counterparts, and urban schools were more likely to have internet access prior to the pandemic. Urban teachers averaged 4.14 assignments per week compared to 3.80 by rural teachers. This difference of 0.34 assignments per week is a significant variation in educational engagement given the 21-month school closures in Indonesia. In addition, 38 percent of urban teachers required students to submit assignments on the same day, while 30 percent of rural teachers did so. One possible explanation for urban teachers assigning more homework and requiring prompt assignment submission could be that they have greater access to resources such as IT technology and online learning resources. This is supported by our survey findings, which reveal a significant contrast in pre-COVID internet access between urban and rural schools. Specifically, urban schools demonstrated a notably higher proportion of pre-COVID internet access at 82 percent, compared to 53 percent in rural schools, representing a significant regional disparity of 28 percentage points. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 34 Table 6. Comparison of Teacher and Principal Practices and Government Support by School Location Urban (a) Rural (b) Diff (a-b) Assignment frequency (Number of assignments in a week) 4.14 3.8 0.34** % of students submitting assignments on the same day 38 30 8** % of principals monitoring classrooms every day 40 35 5 % of schools having access to internet before COVID-19 82 53 28*** % of schools receiving the following support from the Government IT Hardware 24 23 1 Internet Connection 25 17 7 Online learning resources 16 5 12*** Additional Funds 24 29 -6 Books and other physical learning materials 14 12 1 Principal training 53 49 4 Teacher training 67 56 11* Information dissemination 55 47 7 Average Amount of Government Support received per school 2.77 2.39 0.27 Source: Authors’ analysis using Learning Loss Survey 2023 Note: ‘Diff’ represents the differential impact of each factor between urban and rural schools, calculated using a weighted regression analysis. Marginal effects for binary outcomes (e.g., assignment submission time) are derived from probit regression analysis, indicating the change in percentage points for these categorical variables. *p<0.05, **p<0.01, ***p<0.001 6.2.2. An Assessment of Internet for the full regression results). The analysis highlights the better performance of schools that had internet Access on Learning Outcomes access prior to the pandemic. Comparing rural schools in Urban and Rural Areas that had internet access or not before the pandemic, the schools that had internet access exhibit higher An analysis of the impact of internet availability language scores (by 0.260 SD) and higher math scores on learning outcomes indicate that pre-COVID (by 0.154 SD). Likewise, urban schools with pre-COVID internet access markedly enhances student learning internet access show better learning outcomes than outcomes, especially in rural areas. Figure 2 shows urban schools without pre-COVID internet access. the student’s average scores on language and math This suggest that internet access in rural areas could by urban and rural areas and availability of internet contribute to mitigating some of the educational before COVID-19, after controlling for all other factors disadvantages typically associated with rural using a regression mode l (see Appe ndix E Table E1 schooling. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 35 Figure 2. Fitted Test Scores by Availability of Internet and Locations Language Mathematics 0.0 0.0 -0.2 -0.2 -0.188 -0.194 -0.272 -0.315 -0.4 -0.358 -0.4 -0.414 -0.6 -0.533 -0.6 -0.569 -0.8 -0.8 -1.0 -1.0 No Internet No Internet No Internet No Internet Internet Internet Internet Internet Rural Urban Rural Urban Source: Authors’ analysis using Learning Loss Survey 2023 Note: This graph shows imputed weighted average test scores of urban and rural students based on their pre-COVID internet access, derived from the HLM regression model. See Appendix E, Table E1 for the full regression results, including all variables. For example, for language, the rural, no internet group’s score is -0.533 and with internet group is -0.272. The difference is 0.261, which is the same as Table E1, rural schools having pre-COVID internet access (0.260, with a difference due to rounding at third point below decimal). The analysis further assesses whether the provision provision of internet during the pandemic contributed of internet access during the pandemic helped to learning. Consistent with the previous analysis student learning. Figure 3 shows average test scores (Figure 1, Appendix E1), access to the internet before by (1) groups of schools which had no internet access the pandemic was correlated with a more pronounced before and did not receive any support, (2) groups difference of 0.268 in language scores and 0.195 in that did not have internet access prior but received math scores, both statistically significant. Schools it during the pandemic, (3) had internet prior to the with pre-COVID-19 internet access might have teachers pandemic but did not receive additional internet who are better trained on the use of internet. The support, and (4) had internet access before and also government’s additional support for internet access received additional support during the pandemic. to schools with internet prior to COVID-19 may benefit This shows that the group that did not have internet the students better through better hardware (such as before but received access during the pandemic routers or Wi-Fi extenders), or subsidies to reduce the gained higher average scores in language (0.170 cost of internet access, which could increase the use SD) although the difference in math score was not and stability of internet-based teaching and learning. statistically significant (See Table E2 in Appendix E for Statistically, students in this group achieved higher the regression results). Although this is not a causality, scores by 0.419 SD in language and 0.296 SD in math. this can offer a potential counterfactual case of how LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 36 Figure 3. Fitted Test Scores by Type of Internet-related Government Interventions Language Mathematics 0.0 0.0 -0.062 -0.2 -0.2 -0.182 -0.213 -0.310 -0.283 -0.4 -0.4 -0.367 -0.6 -0.481 -0.6 -0.478 -0.8 -0.8 -1.0 -1.0 No Pandemic-Era Pre-COVID Both No Pandemic-Era Pre-COVID Both Internet Goverment Internet Internet Goverment Internet Internet Access Internet Access Support Support Source: Authors’ analysis using Learning Loss Survey 2023 Note: This graph shows imputed weighted average test scores of urban and rural students based on their pre-COVID internet access, derived from the HLM regression model. See Appendix E, Table E2 for the full regression results, including all variables. 6.2.3. Comparing the Different In response to the pandemic, both MoRA and MoECRT Ministries’ Responses to the actively extended various forms of support to schools. This support included training for teachers and School Closures principals, provision of IT hardware, ensuring internet connectivity, and supplying educational resources. MoRA teachers assigned an average of 4.26 While schools unde r diffe rent ministries received a assignments per week, compared to 3.95 comparable amount of support overall, the nature of assignments by MoECRT teachers during the school that support varied by ministry, in terms of the specific closure. The diffe rence of 0.31 is substantial give n interventions focused on in this report. MoRA schools the re latively longer periods of school closure s in we re more likely to re ce ive support for te ache r (57 Indonesia. In addition, principals at MoRA schools percent) and principal (64 percent) training, suggesting also reported a higher rate of daily monitoring at 54 a strategy focused on enhancing instructional and percent, exceeding the 34 percent reported in MoECRT leadership capabilities. Conversely, MoECRT schools schools, by a statistically significant margin of 19 received more infrastructural support, particularly percentage points. Additionally, a notable disparity in in IT hardware (36 percent) and internet access (23 pre-pandemic internet access was observed, with 76 pe rcent). The approach by the MoECRT appe ars to percent of MoRA schools having internet connectivity be an effort to mitigate the digital divide that existed as opposed to 57 percent of MoECRT schools, marking prior to the pandemic, as evidenced by their targeted an 18 percentage point gap.31 support for remote learning infrastructure.32 31 This information was collected through the principal questionnaire module of the Learning Loss survey of 2023,asking about the status of internet connectivity before the pandemic. 32 This report acknowledges the number of interventions that were introduced by MoECRT and MoRA as well as local governments during the pandemic, which could not all be captured in this report. These results focus on specific interventions and therefore need to be interpreted with caution. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 37 Table 7. Comparison of Teacher and Principal Practices and Government Support by School Type MoRA (a) MoECRT (b) Diff (a-b) Assignment frequency (Number of assignments in a week) 4.26 3.95 0.31** % of students submitting assignments on the same day 39 34 5 % of principals monitoring classrooms every day 54 34 19*** % of schools having access to internet before COVID-19 76 57 18*** % of schools receiving the following support from the Ministry IT Hardware 12 36 -23*** Internet Connection 19 23 -5 Online learning resources 10 10 1 Additional Funds 31 22 10* Books and other physical learning materials 10 16 -5 Principal training 57 44 13** Teacher training 64 58 6 Information dissemination 51 51 0 Total Amount of Ministry Support 2.59 2.57 -0.01 Source: Authors’ analysis using Learning Loss Survey 2023 Note: Diff’ represents the differential impact of each factor between MoRA and MoECRT schools, calculated using a weighted regression analysis. Marginal effects for binary outcomes (e.g., assignment submission time) are derived from probit regression analysis, indicating the change in percentage points for these categorical variables. *p<0.05, **p<0.01, ***p<0.001 6.3 Teachers’ Characteristics and Teaching Practices and Students’ Learning The Learning Loss Survey 2023 collected information demonstrated a statistically significant positive on teaching practices by observing teachers’ correlation, with a coefficient of 0.214, equating to an teaching practices in the classrooms using the Teach improvement of 8.6 months (See Table 8). It is worth instrument. Before the COVID-19 outbreak in 2019, the highlighting that teaching practices that promoted SDI survey found that fewer than one in five teachers socioemotional skills of students were found be surveyed met the minimum subject knowledge positively associated with language scores, with a requirements, and this lack of teacher competency coefficient of 0.218 (p<0.05). Such teaching practices was adversely associated with student achievement include providing more opportunities to take on roles (World Bank, 2020b). The Learning Loss Survey 2023 in the classroom, recognizing students’ efforts, and also collected information of teaching practices but encouraging collaborative interactions among peers. with a newer instrument. Due to the change of the instrument, this analysis intends to present analysis In math, higher scores across all Teach areas are of 2023 teaching practices and learning outcomes as a significantly linked to improved learning outcomes, single-year data analysis. with socioemotional skills being most positively correlated with learning. For instance, the Instruction The averaged Teach score suggests that overall domain – encompassing elements such as lesson teaching quality correlates with higher language facilitation, checking for understanding, feedback, skills. For language, the ave raged Te ach score and encouraging critical thinking –exhibited a positive LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 38 association of 0.104 with mathematic outcomes. However, there is a limited number of teachers who That is, teachers who frequently engage in checking employ key teaching practices measured in the Teach students' understanding and providing feedback tend score in Indonesia. In fact, only 20 percent of teachers to foster higher math performance. Socioemotional demonstrated key teaching practices through skills again emerged as a significant contributor, the achievement of scores exceeding three out of with a positive correlation of 0.210 at the 0.001 five points in the Teach score afte r the pande mic significance level, underscoring their role in math (World Bank, forthcoming). In particular, teachers skills development. with socioemotional teaching skills were very limited in number, with only one out of ten teachers In sum, good teaching practices are associated demonstrating key teaching practices (three points with better learning outcomes of students in a or higher) in socioemotional skills. More attention post-pandemic context. These are only observed is required to teaching practices in socioemotional correlations and do not indicate causality, however skills, given the consistent and positive relationship these patterns demonstrate the importance of with students’ learning outcomes as demonstrated in good teaching practices for better learning, such this report. Therefore, it is crucial to enhance teacher as providing choices to students, acknowledging support in guiding these key and holistic teaching students’ efforts, fostering a positive attitude, practices to improve student learning outcomes. See encouraging goal setting, promoting collaboration, Appendix G for additional analysis on teachers by and developing interpersonal skills among students. gender. Table 8. Association between Teach Scores and Student Learning Outcomes (1) (2) (3) (4) Averaged Classroom Instruction Socioemotional Teach Score Culture Skill Language 0.214** 0.057 0.078 0.243*** (0.087) (0.065) (0.063) (0.065) Mathematics 0.243*** 0.108* 0.104* 0.210*** (0.079) (0.057) (0.056) (0.062) Note: Regression analysis uses a weighted methodology to account for these variations, with standard errors clustered within schools. * p<0.05, ** p<0.01, *** p<0.001. 6.4 Individual-level Characteristics and Students’ Learning 6.4.1. Widening learning disparities groups. Students from the wealthiest households, the highest asset quintile, demonstrated a 0.52 SD across groups of different advantage in language and a 0.38 SD advantage in socioeconomic groups math scores compared to their counterparts in the lowest quintile group in 2019 (See Appendix F for the full Following the school closures during the pandemic, table of regression results). In contrast, the 2023 results pre-existing socioeconomic disparities in learning indicate that stude nts from the highe st house hold have widened. With re spe ct to learning loss over asset quintile demonstrated significantly higher the years, there is a clear and observable increase in performance, scoring 0.99 SD in language and 0.75 SD learning disparities across different socioeconomic in math, compared to the lowest quintile counterparts. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 39 This difference is roughly equivalent to a two- to three- pandemic, points to the foundational role of early year advancement in learning. Similarly, students education in shaping academic skills. with parents educated to the tertiary level exhibited superior performance, with a score increase of 0.93 to However, the disparity in preschool attendance 1.2 SD, corresponding to approximately three years of underscores an early educational divide. The survey additional learning. This suggests a widening gap in findings reveal notable differences in preschool educational outcomes based on socioeconomic status attendance based on parental socioeconomic and parental education level over the period studied. backgrounds (Table 9). Preschool attendance rates show a marked increase from the lowest consumption Preschool attendance is positively correlated with quintile at 71 percent to the highest at 95 percent. learning outcomes. Our main finding in Table 5 also Children from higher socioeconomic backgrounds reveals that, even after considering socioeconomic gain the benefits of early learning experiences that are status, students who attended preschool tend to foundational for future academic achievements. The achieve slightly higher language outcomes post- financial barrier to preschool attendance for lower- pandemic (See Model I in Table 5). It also suggests income families in Indonesia suggests a systemic that early childhood education has a lasting positive challenge where the benefits of early education are association with students’ learning outcomes that disproportionately accessible to those with more extends beyond the immediate preschool years. resources. This inequity not only affects individual The resilience of these outcomes, even in the face children but can perpetuate cycles of educational of significant disruptions like those caused by the disadvantage and socio-economic stratification. Table 9. Descriptive Statistics of Average Household Characteristics by Socioeconomic Groups Students Close One’s Urban Attending Number of Students helping Receiving Death Areas Preschool Books with Chores Assignment (%) (%) (%) Read per (%) Support from Month parents (%) Household Asset Quintile 1-Lowest 73 36 44 71 33 2.5 2 78 35 56 88 24 2.6 3 77 34 66 88 24 2.4 4 78 34 71 94 25 2.6 5-Highest 77 32 82 95 17 2.9 Father’s education Primary or less 72 38 52 78 29 2.5 Lower Secondary 80 37 56 87 29 2.8 Upper Secondary 78 32 74 93 22 2.5 Tertiary 76 24 79 96 12 3.0 Mother’s education Primary or less 71 38 52 77 30 2.6 Lower Secondary 78 39 56 90 25 2.5 Upper Secondary 79 30 77 93 24 2.6 Tertiary 81 29 77 96 12 3.0 Total 77 34 65 88 24 2.6 Source: Authors’ analysis using Learning Loss Survey 2023 Note: Weighted descriptive statistics consider variations in student and school demographics across institutions. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 40 Poorer children frequently take on household language and -0.153 in mathematics, translating into responsibilities, potentially hindering their learning declines of approximately 6.6 months and learning. In the shift away from conve ntional in- 6.12 months, respectively (See Table 5). This suggests person schooling, students frequently encountered that experiencing a death during the school closure additional household and economic responsibilities, period had a considerable and negative association hindering their participation in educational pursuits. with students’ learning. Delving into the psychological Our main analysis shows that participation in underpinnings, Grant et al. (2004) highlights how household chores, particularly those involving bereavement-related psychological distress can economic tasks such as farming or selling goods, detract from a student’s academic performance. does not have a direct association with student Furthermore, Christ (2000) explores the broader learning. Nonetheless, when other factors are not ramifications of such loss on a child’s educational controlle d in the re gre ssion analysis, there appe ars journey, including diminished motivation, lowered to be a negative association between participation in academic aspirations, and a disengagement from household chores and language learning outcomes. school activities. This disengagement is particularly In the absence of traditional face-to-face schooling, detrimental to core academic skills development, such economically vulnerable students often have to take as language and mathematics, which are foundational on household economic responsibilities, which can to a student’s overall academic trajectory. The negatively influence their engagement in learning bereavement experienced by students during the activities (Blunde ll e t al., 2020). Our surve y also pandemic is not just a personal tragedy but also reveals a notable trend wherein students from lower a significant educational concern. The negative socioeconomic backgrounds tend to be more involved influence on language and mathematics proficiency in household chores (Refer to Table 9). Table 9 showed points to the need for targeted support mechanisms a notable de cline in the pe rcentage of childre n to address the psychological and academic needs of contributing to household chores as SES increases, bereaved students. from 33 percent among those in the lowest quintile to 17 percent in the highest quintile. This trend suggests Further, the study findings suggest that children that children from lower SES families might have less from lower SES backgrounds are more susceptible time available for homework and studying due to their to the compounded challenges of bereavement household responsibilities, potentially relating to their and educational disruption. Previous studies have academic performance negatively and contributing to found that individuals with lower socioeconomic learning loss. Consequently, heightened participation status experienced higher rates of COVID-19 fatalities in domestic responsibilities within economically (Khanijahani et al., 2021; Magesh et al., 2021). This disadvantaged households serves as an indirect suggests that the socioeconomic disparities in mediator contributing to the deterioration of e xperiencing the de ath of a close one during the educational outcomes among this group. Specifically, pandemic shed light on an additional layer of inequality children from these households allocate more time impacting student learning. Our survey also reveals a to house hold chore s, de tracting from the time and stark contrast in the bereavement experiences among energy available for academic pursuits and school- students from different socioeconomic backgrounds, related tasks. This diversion of resources ultimately with children from lower socioeconomic backgrounds impairs their academic performance. reporting higher incidences of losing someone close to them. For children with fathers who have only a Our evidence indicates that the loss of a family primary education or less, 38 percent reported the member or someone close during the pandemic is death of someone close to them during the pandemic negatively and significantly associated with student (Table 9). This pe rcentage gradually de cre ases as academic performance. Students experiencing the the father’s education level increases, dropping to loss of a family member or someone close during the 24 percent for those with fathers who attained a pandemic had significantly lower language and math tertiary education. A similar pattern is observed with scores compared to those who did not report such the mother’s education, with 38 percent of children an experience. These disparities were statistically experiencing a close one’s death when the mother has significant, corresponding to a reduction of -0.166 in only primary education, compared to 29 percent for LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 41 those whose mothers have a tertiary education. The Furthermore, children from lower socioeconomic higher incidence of bereavement in these groups likely backgrounds often face less conducive environments contributes to the exacerbated learning loss observed for educational success, residing predominantly among them. in rural areas. Only 44 percent of students from the lowest asset quintile live in urban areas, whereas 88 6.4.2. Unpacking Socioeconomic percent of students from the highest asset quintile Disparities in Learning reside in urban are as. Rural are as typically lack access to well-equipped schools and advanced Environments internet infrastructure, which can hinder educational opportunities and exacerbate learning loss for children Parental involvement tends to be greater when from low SES backgrounds. parents are educated and socioeconomically advantaged. The e xtant studie s note that the The results have illuminated the profound and importance of parental involvement in children’s multifaceted ways in which socioeconomic education in mitigating le arning loss due to the disparities have exacerbated learning among COVID-19 pandemic (e.g., World Bank et al, 2021). This Indonesian students during the COVID-19 pandemic includes helping children with home-based learning, across schools and across students. The widening providing emotional support, and fostering an gap in educational outcomes between students from environment conducive to learning. Table 9 shows that different socioeconomic backgrounds, underscored socioeconomically advantaged students are associated by differences in parental involvement in children’s with increased parental or household member support learning, access to educational resources, and living in completing assignments. This is evidenced by the conditions, paints a bleak picture of inequality. incremental increase in assignment support from Particularly, the disparities in preschool attendance, lower to higher SES brackets. For example, 81 percent household support for learning, and the differential of mothers with tertiary education provided support burdens of household chores and bereavement to their children, while only 71 percent of mothers with e xperience s lay bare the critical challe nge s facing primary education or less did so. socioeconomically disadvantaged students. As also seen in the analysis of variance (Table 4), such Moreover, children from lower socioeconomic performance gaps also grew across schools (for backgrounds face challenges in accessing language). The scores across well performing schools educational resources. Challenges with access are and poor performing schools are stark (See Appendix indicated by the lower number of books reported H for the analysis of score distribution across schools). in households with parents having primary or less The se findings not only unde rscore the urge ncy of education compared to those with tertiary educated addressing these disparities but also highlight the parents. Children with parents who have primary need for targeted interventions that can mitigate the education or le ss re porte d reading an ave rage of adverse effects of such inequalities on educational 2.5 books per month, whereas those with tertiary- outcomes. As Indonesia and the global community educated parents reported reading an average of three strive to recover from the educational disruptions books. This contrast emphasizes the challenges faced caused by the pandemic, it is imperative to prioritize by children from lower socioeconomic backgrounds, policies and practices that ensure equitable learning who have restricted access to learning materials and opportunities for all students, especially those who experience lower levels of educational engagement at are socioeconomically disadvantaged. home. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 42 Learning in the Shadow of the Pandemic: 42 COVID-19 Learning Loss and Widening Learning Disparities in Indonesia 7 Conclusions and Policy Recommendations Examination of the association between These findings suggest key policy actions to address governmental and school efforts and student widening learning disparities: learning informed key potential factors to improve • Re-orient resources to disadvantaged areas and student learning outcomes. This study found students by the MoECRT and MoRA, especially rural associations between school practices and factors schools and students from low SES. and learning losses and learning situations of Indonesian students. The following evidence emerges • Focus on supporting evidence-based effective from this analysis on which factors are correlated practices aiming to improve student learning with student’s learning. These are: adequate use of outcomes and experiences, rather than just learning assessment by teachers and teachers’ overall increasing inputs. quality, frequency of school principal’s monitoring of teaching activities, and government’s provision The following four action areas detail and integrate of internet access and online learning materials. the above recommendations, focusing on how to However, evidence also shows that learning disparity improve capacities of teachers, school principals, has been widened by students’ SES. These findings national and subnational governments and students have significant implications for the detailed and families. The recommendations mainly focus on understanding of who experienced learning losses learning recove ry as re levant to the curre nt post- and how much, and how the government and school pandemic context. It is also critical to strengthen the stakeholders can ensure quality of student learning system to respond to the needs of teachers and other post-pandemic. Drawing on these results and existing key stakeholders, in order to make the education frameworks for learning recovery, this report provides system more resilient to other, future shocks. policy recommendations for Indonesia. Policy recommendation 1: [Teacher] Enhance the effectiveness of teacher training by focusing on behavior change, including that of rural teachers. Teacher training should focus more on practical skills and how to bring about behavior change, instead of only focusing on knowledge accumulation. Improved teacher training should emphasize teachers’ skills to improve teacher-student interactions, covering not only teaching methods but also teachers’ socio-emotional skill and assessment design and support skill, with additional training and support for rural teachers, given the urban-rural gaps in teacher practices. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 43 Recommendation 1a: Recommendation 1b: Reinforce instructional approaches through more Strengthen training and support to improve effective use of assessment, harnessing education teachers’ pedagogical practices to foster students’ technology. Effective use of formative and summative socioemotional skills. Teacher training and support is assessment should be highlighted to boost the quality key to ensure effectiveness of teaching, and this report of teaching. This study found that students who proposes that the education system needs to support completed assignments and more tasks assigned by teachers in four focus areas including resilience, teachers during the pandemic showed significantly instruction, technology and socioemotional skills. This higher learning outcomes. The empirical evidence study found that teacher quality, especially concerning suggests that teaching and assessing according to the socio-emotional skills, improved student learning students’ ability (known as Teaching at Right Level) outcomes in the post-pandemic context. This is an (World Bank, 2020a; 2022b) can improve student important addition to the existing knowledge which learning outcomes in the Indonesian context. This emphasizes how resilience, instruction and technology study also highlights the importance of the timing can increase the efficiency of teacher instruction during and frequency of assignment/feedback to increase the pandemic (World Bank, 2020a). This report adds to student learning outcomes. This is consistent with this existing knowledge by highlighting the importance global evidence on remote learning in response to of teachers’ pedagogical practices to foster students’ the pandemic era school closures that highlights the socio-emotional skills. These four points should be importance of ensuring the engagement of students, strongly incorporated in teacher training programs effective teachers and adoption of suitable technology in MoECRT and Peer Working Groups in MoRA. This (Munoz-Najar et al., 2021). These insights can guide promising teaching framework can be used to improve learning post-pandemic. The use of assessment for teacher Continuous Professional Development learning including teaching at the right level has been modules for teacher professional development in a key principle in the new national curriculum policy Indonesia. of GoI, such as Permendikbudristek 12/2024 and its official guidelines (e.g., the Panduan Pembelajaran dan Asesmen).33 Similarly, recent MoECRT policy has The GoI has made efforts to foster students’ social focused on the use of technology to support teacher emotion skills, especially focusing on teacher training. learning via the Platform of Merdeka Mengajar, as Online courses in Platform Merdeka Mengajar and well as an offline version Awan Penggerak to ensure the guidelines on Projek Penguatan Profil Pelajar teachers in remote areas access the same high quality Pancasila focus on socioemotional skills, such as training materials.34 These recent initiatives are in the the use of positive discipline to foster self-regulated right direction, and implementation, effectiveness and learning. However, an impact evaluation of the Guru practical challenges of these policies and guidelines at Penggerak teacher training showed positive impact on the school level would need to be properly evaluated. classroom culture, but no evidence of improvement Further, EdTech applications could be used more was found in differentiated instruction or in teachers’ effectively by teachers and students to improve socio-emotional skills in the classroom (Khairina et al., teacher-student interactions, assignment submission 2023). Ensuring implementation of such practices in and provision and receipt of feedback. As incentives classrooms should follow, along with practical support for teachers, training sessions and follow up should by the government to help teachers and schools be conducted, focusing on the practical use of EdTech implement policies and guidelines in class. to enhance student learning outcomes through more effective assessment/feedback and teacher-student interactions. 33 More information is available from https://kurikulum.kemdikbud.go.id/ 34 https://guru.kemdikbud.go.id/ LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 44 Policy recommendation 2: [School principals] Strengthen school principal training especially related to monitoring and accountability, including rural schools. The quality of school principal training can be improved by focusing on how to improve the capacity of school principals on school management to bring about changes in student outcomes and experiences. Active monitoring by principals is shown to enhance the performance of teachers and students. Further, where school principals have limited capacity, they can become system-level barriers to implementing and sustaining teacher policies (World Bank, 2023c). Thus, appropriate guidance and support for school principals is needed. This recommendation is consistent with the practical framework set by the World Bank (2023c) to identify the barriers inhibiting teacher policies and to develop strategies to overcome them. Principal training should cover the key management skills linked to better teaching and learning and include a focus on rural schools. In Indonesia, while the Guru Penggerak program in Merdeka Belajar includes school principals as target beneficiaries, interventions need to be strengthened by highlighting evidence-based practices to improve student learning. A more targeted approach would be needed for schools suffering from large learning losses such as rural and public schools. Other key differentiating factors related to promoting learning recovery in Indonesia include principals’ use of obtained data, implementation of school-level education programs for students suffering from the pandemic, and budget allocations for student learning recovery.35 School principals who give clear directions and instructions can motivate teachers to continue their teaching and provide individual support for students post-pandemic. It is important for school principals to provide flexible and equitable support, for example by allowing teachers to conduct house visits when online learning is difficult, supervising and discussing with teachers to check students’ progress. Moreover, school principals have crucial role in sustaining and expanding school collaboration with local NGOs, communities, universities and experts in their areas. Policy recommendation 3: [Government] Use digital transformation to promote equity of learning opportunities by providing support especially in rural areas. This study found that internet connectivity is the key governmental intervention associated with improved learning outcomes, and rural areas benefit the most. The combination of internet connectivity and access to online learning resources led to higher student learning outcomes. These findings are important since previous studies tended to focus on either provision of equipment or internet access.36 This result implies a need for a more strategic governmental intervention in the future, especially during emergencies, to provide alternative learning models such as remote learning. GoI’s investment in a strong digital infrastructure is essential to facilitate remote, hybrid and innovative face-to-face learning in a way that can ensure equitable access to education for all and mitigate learning disruptions. Bridging the digital divide requires strengthened support particularly for rural areas. Investment in internet accessibility and IT hardware for rural schools can be effective in enhancing student learning, while urban schools often already have these resources. Alternative support mechanisms for remote areas should also be cultivated to improve equity in education. In addition, understanding gaps not only in digital infrastructure, but also user capacity and selection of different programs for different purposes can be further considered for continuous advancement of digital transformation. 35 INOVASI, 2022 36 INOVASI, 2022 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 45 Policy recommendation 4: [Students and families] Strengthen support for students and parents from low SES, rural areas, and for those who experience the loss of someone close during emergency situations such as the pandemic. Education support by the government and school should address the economic, educational and psychological challenges faced by students from low SES. Support for parents is particularly needed for students from economically and socially disadvantaged backgrounds. Recommendation 4a: Recommendation 4b: Provide adequate budget allocations for further Provide targeted learning, health and social support financial and learning support for students from for vulnerable students including those who low socio-economic status. This study found that experience loss of family members and/or somebody students from low SES experienced larger learning close during the pandemic. This study found that the losses compared to other students between 2019 and death of family members or somebody close during 2023, consistent with a previous study (Haelermans et the pandemic decreased student learning outcomes al. 2022). While the government’s additional support in both literacy and numeracy. It is necessary for GoI to BOS (School Operational Funds) allocations to to further develop psychosocial health and well-being disadvantage d re gions since 2021 has made an support and conduct targeted interventions to reach important contribution, this may not be enough these students to ensure that every child is ready to to address the profound challenges faced by these learn following such loss, as proposed by the RAPID students who suffer from low SES. Boys persistently learning recovery framework (UNICEF et al, 2022; World underperformed girls in both subjects in 2019 and Bank et al, 2021). In addition, financial and specific 2023, and this may be related to the greater economic support should be provided for these students, as responsibilities borne by older students due to the well as other vulnerable children such as children with financial setbacks caused by the pandemic (Zulaika et disabilities and out of school children. al, 2022). Financial support by the government, such as scholarships for students, should be considered, Recommendation 4c: especially for those who experienced large disruptions in the ir e ducation due to pove rty through the Strengthen relations between schools and pandemic. Studies show the positive effects of financial communities for a more resilient education system. interventions in improving schooling decisions for Results underscored the profound influence of students and families from low SES (Burland et al. parental education level in supporting children’s 2023), and educational intervention to improve learning during the pandemic, which implies the need reading skills of primary school students targeting for targeted interventions. Provision of support to students who face difficulties in reading (Lopes et al. parents especially for those from low socio-economic 2024). Program Indonesia Pintar is an existing program status is needed to support their child’s learning, and by MoECRT to provide scholarship to students from the re is a promising cost-e ffective intervention to poor and disadvantaged backgrounds, and leveraging improve child’s learning using low-tech for parents in such existing programs effectively would help deal MICs (Angrist, Bergman, and Matsheng 2022). Sound with specific challenges and reach the most vulnerable relations between schools and communities are key students. to strengthening education in Indonesia beyond the pandemic (Butcher et al, 2021). Increased assistance and support from non-governmental organizations (NGOs), other professionals, social workers should be provided to schools, teachers, families, and students. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 46 Learning in the Shadow of the Pandemic: 46 COVID-19 Learning Loss and Widening Learning Disparities in Indonesia References Abreh, M. K., Agbevanu, W. K., Alhassan, A. J., Ansah, F., Bosu, R. S., Crawfurd, L., Mills, C. A., Minardi, A. L., & Nyame, G. (2021). What happened to dropout rates after COVID-19 school closures in Ghana. CDG Blog. Angrist, N., Bergman, P., Matsheng, M. (2022). Experimental Evidence on Learning Using Low-Tech When School Is Out. Nature Human Behaviour 6 (7): 941–50. Araya, M., Rose, P., Sabates, R., Tiruneh, D. T., & Woldehanna, T. (2022). Learning Losses during COVID-19 Pandemic in Ethiopia: Comparing Student Achievement in Early Primary Grades before School Closures, and after They Reopened. RISE Insight Series, 44. Ardington, C., Wills, G., & Kotze, J. (2021). COVID-19 learning losses: Early grade reading in South Africa. International Journal of Educational Development, 86, 102480. Arenas Jal, A., & Gortazar, L. (2022). Learning loss one year after school closures: Evidence from the Basque Country. IEB Working Paper 2022/03. Asadullah, M. N., & Bhattacharjee, A. (2022). Digital divide or digital provide? Technology, time use, and learning loss during COVID-19. The Journal of Development Studies, 58(10), 1934–1957. Azubuike, O. B., Adegboye, O., & Quadri, H. (2021). Who gets to learn in a pandemic? Exploring the digital divide in remote learning during the COVID-19 pandemic in Nigeria. International Journal of Educational Research Open, 2, 100022. Bazoli, N., Marzadro, S., Schizzerotto, A., & Vergolini, L. (2022). Learning loss and students’ social origins during the covid-19 pandemic in Italy. FBK-IRVAPP Working Papers, 3, 2022. Betthäuser, B. A., Bach-Mortensen, A. M., & Engzell, P. (2023). A systematic review and meta-analysis of the evidence on learning during the COVID-19 pandemic. Nature Human Behaviour, 7(3), 375–385. Birkelund, J. F., & Karlson, K. B. (2023). No evidence of a major learning slide 14 months into the COVID-19 pandemic in Denmark. European Societies, 25(3), 468–488. Blainey, K., & Hannay, T. (2021). The impact of school closures on autumn 2020 attainment. RS Assessment from Hodder Education. Burland, E., Dynarski, S., Michelmore, K., Owen, S., Raghuraman, S. (2023). The Power of Certainty: Experimental Evidence on the Effective Design of Free Tuition Programs. American Economic Review: Insights 5 (3): 293–310. Butcher, N., Khairina, N. N., Kumala, C., & Loots, S. (2021). The struggle against COVID-19 in Indonesian Education: Responses, requirements, and policy needs for learning recovery. Washington, D.C.: World Bank Group. Christ, G. H. (2000). Healing children’s grief: Surviving a parent’s death from cancer. Oxford University Press. Contini, D., Di Tommaso, M. L., Muratori, C., Piazzalunga, D., & Schiavon, L. (2022). Who lost the most? Mathematics achievement during the COVID-19 pandemic. The BE Journal of Economic Analysis & Policy, 22(2), 399–408. Crawfurd, L., Hares, S., & Minardi, A. L. (2021). New data on learning loss in Pakistan. Center for Global Development. Di Pietro, G. (2023). The impact of Covid-19 on student achievement: Evidence from a recent meta-analysis. Educational Research Review, 100530. Domingue, B. W., Dell, M., Lang, D., Silverman, R., Yeatman, J., & Hough, H. (2022). The effect of COVID on oral reading fluency during the 2020–2021 academic year. AERA Open, 8, 23328584221120254. Dynan, L., & Cate, T. (2009). The impact of writing assignments on student learning: Should writing assignments be structured or unstructured? International Review of Economics Education, 8(1), 64–86. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 47 Engzell, P., Frey, A., & Verhagen, M. D. (2021). Learning loss due to school closures during the COVID-19 pandemic. Proceedings of the National Academy of Sciences, 118(17), e2022376118. Fauzi, I., & Khusuma, I. H. S. (2020). Teachers’ elementary school in online learning of COVID-19 pandemic conditions. Jurnal Iqra’: Kajian Ilmu Pendidikan, 5(1), 58–70. Gambi, L., & De Witte, K. (2021). The resiliency of school outcomes after the COVID-19 pandemic. Standardised test scores and inequality one year after long term school closures. FEB Research Report Department of Economics. Gore, J., Fray, L., Miller, A., Harris, J., & Taggart, W. (2021). The impact of COVID-19 on student learning in New South Wales primary schools: An empirical study. The Australian Educational Researcher, 48(4), 605–637. Grant, K. E., Compas, B. E., Thurm, A. E., McMahon, S. D., & Gipson, P. Y. (2004). Stressors and child and adolescent psychopathology: Measurement issues and prospective effects. Journal of Clinical Child and Adolescent Psychology, 33(2), 412–425. Haelermans, C., Korthals, R., Jacobs, M., de Leeuw, S., Vermeulen, S., van Vugt, L., Aarts, B., Prokic-Breuer, T., Van der Velden, R., van Wetten, S., & de Wolf, I. (2022). Sharp increase in inequality in education in times of the COVID- 19-pandemic. Plos One, 17(2), e0261114. Hallin, A. E., Danielsson, H., Nordström, T., & Fälth, L. (2022). No learning loss in Sweden during the pandemic: Evidence from primary school reading assessments. International Journal of Educational Research, 114, 102011. Haelermans, C., Jacobs, M., van der Velden, R., van Vugt, L., van Wetten, S. (2022). Inequality in the Effects of Primary School Closures Due to the COVID-19 Pandemic: Evidence from the Netherlands. AEA Papers and Proceedings 112 (May): 303–7. Hevia, F. J., Vergara-Lope, S., Velásquez-Durán, A., & Calderón, D. (2022). Estimation of the fundamental learning loss and learning poverty related to COVID-19 pandemic in Mexico. International Journal of Educational Development, 88, 102515. INOVASI. (2021). Learning Recovery: Time for Action Policy Brief—August 2021. INOVASI. (2022). Build Back Stronger: Post-Pandemic Learning Recovery. Kafa, A. (2023). Teachers’ perceptions of school principals’ role in tackling the pandemic crisis. International Journal of Educational Management, 37(2), 350–360. Kilenthong, W. T., Boonsanong, K., Duangchaiyoosook, S., Jantorn, W., & Khruapradit, V. (2023). Learning losses from school closure due to the COVID-19 pandemic for Thai kindergartners. Economics of Education Review, 96(2), 102455. Khairina, N. N., Yarrow, N., Cilliers, E. J. P., and Dini, I.S.Z. (2023). Improving Teachers and School Leadership in Indonesia: Impact Evaluation of Guru Penggerak Program at the Primary Level. Washington, D.C.: World Bank Group. Khanijahani, A., Iezadi, S., Gholipour, K., Azami-Aghdash, S., & Naghibi, D. (2021). A systematic review of racial/ethnic and socioeconomic disparities in COVID-19. International Journal for Equity in Health, 20, 1–30. Kim, J., Rose, P., Tiruneh, D. T., Sabates, R., & Woldehanna, T. (2021). Learning inequalities widen following covid-19 school closures in Ethiopia. Research on Improving Systems of Education (Rise), 4. Kuhfeld, M., Soland, J., & Lewis, K. (2022). Test score patterns across three COVID-19-impacted school years. Educational Researcher, 51(7), 500–506. Lewis, K., Kuhfeld, M., Ruzek, E., & McEachin, A. (2021). Learning during COVID-19: Reading and math achievement in the 2020-21 school year. Center for School and Student Progress. Lichand, G., Doria, C. A., Leal-Neto, O., & Fernandes, J. P. C. (2022). The impacts of remote learning in secondary education during the pandemic in Brazil. Nature Human Behaviour, 6(8), 1079–1086. Locke, V. N., Patarapichayatham, C., & Lewis, S. (2021). Learning loss in reading and math in US schools due to the COVID-19 pandemic. Retrieved from https://www.istation.com/hubfs/Content/downloads/studies/COVID-19_Learning_ Loss_USA.pdf LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 48 Lopes, J., Martins, P.S., Oliveira, C., Ferreira, J., Oliveira, J. T., Crato, N. (2024). From A to Z: Effects of a 2nd-Grade Reading Intervention Program for Struggling Readers. Revista de Psicodidáctica (English Ed.) 29 (1): 57–68. Magesh, S., John, D., Li, W. T., Li, Y., Mattingly-App, A., Jain, S., Chang, E. Y., & Ongkeko, W. M. (2021). Disparities in COVID-19 outcomes by race, ethnicity, and socioeconomic status: A systematic review and meta-analysis. JAMA Network Open, 4(11), e2134147–e2134147. Maldonado, J. E., & De Witte, K. (2022). The effect of school closures on standardized student test outcomes. British Educational Research Journal, 48(1), 49–94. Mbaye, S., Nestour, A. L., Moscoviz, L., & Chery, J. (2021). What Happened to Senegalese Students after the COVID-19 School Closure. CDG Blog. Moscoviz, L., & Evans, D. K. (2022). Learning loss and student dropouts during the covid-19 pandemic: A review of the evidence two years after schools shut down. Retrieved from https://www.cgdev.org/sites/default/files/learning-loss- and-student-dropouts-during-covid-19-pandemic-review-evidence-two-years.pdf Munoz-Najar, A., Gilberto, A., Hasan, A., Cobo, C., Azevedo, J. P., & Akmal, M. (2021). Remote Learning During COVID-19: Lessons from Today, Principles for Tomorrow. World Bank. Retrieved from https://doi.org/10.1596/36665 OECD. (2023). PISA 2022 Results: Factsheets Indonesia. Patrinos, H. A. (2023). The Longer Students Were Out of School, the Less They Learned. Washington, DC.: World Bank. Pier, L., Christian, M., Tymeson, H., & Meyer, R. H. (2021). COVID-19 Impacts on Student Learning: Evidence from Interim Assessments in California. Policy Analysis for California Education, PACE. Randall, R., Sukoco, G. A., Heyward, M., Purba, R., Arsendy, S., Zamjani, I., & Hafiszha, A. (2022). Reforming Indonesia’s curriculum: How Kurikulum Merdeka aims to address learning loss and learning outcomes in literacy and numeracy. INOVASI. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Sage. Rayburn, L. G., & Rayburn, J. M. (1999). Impact of course length and homework assignments on student performance. Journal of Education for Business, 74(6), 325–331. Sandefur, J. (2022). Uganda’s record-breaking two-year school closure led to… no decline in the number of kids who can read. Center for Global Development. Schady, N., Holla, A., Sabarwal, S., Silva, J., & Chang, A. Y. (2023). Collapse and Recovery: How the covid-19 pandemic eroded human capital and what to do about it. Washington, DC.: World Bank. Schult, J., Mahler, N., Fauth, B., & Lindner, M. A. (2022). Did students learn less during the COVID-19 pandemic? Reading and mathematics competencies before and after the first pandemic wave. School Effectiveness and School Improvement, 33(4), 544–563. Schuurman, T. M., Henrichs, L. F., Schuurman, N. K., Polderdijk, S., & Hornstra, L. (2023). Learning loss in vulnerable student populations after the first COVID-19 school closure in the Netherlands. Scandinavian Journal of Educational Research, 67(2), 309–326. Singh, A., Romero, M., & Muralidharan, K. (2024). COVID-19 Learning loss and recovery: Panel data evidence from India. Journal of Human Resources. DOI: https://doi.org/10.3368/jhr.0723-13025R2 Shepherd, D., Mohohlwane, N., Taylor, S., & Kotzé, J. (2021). Changes in education: A reflection on COVID-19 effects over a year. National Income Dynamics Study (NIDS)-Coronavirus Rapid Mobile Survey (CRAM). Snijders, T. A., & Bosker, R. (2011). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage Publications Sukoco, G.A., Arsendy, S., Purba, R.E., Zulfa, A.H. (2023). Bounce Back Stronger: Learning Recovery After the Pandemic: A Case Study of INOVASI Partner Schools. Jakarta: INOVASI LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 49 Suriastini, W., Pujiastuti, S., Herawati, F., Naryanta, Moddilani, G., & Pellini, A. (2020). COVID-19 and the education response in Indonesia: exploring the learning crisis. Case study. EdTech Hub. Retrieved from https://edtechhub.org/ Tan, C. Y. (2023). Influence of principal leadership across contexts on the science learning of students. Asia Pacific Education Review, 25, 1-14. Tomasik, M. J., Helbling, L. A., & Moser, U. (2021). Educational gains of in‐person vs. Distance learning in primary and secondary schools: A natural experiment during the COVID‐19 pandemic school closures in Switzerland. International Journal of Psychology, 56(4), 566–576. Uganda, U. (2021). Are Our Children Learning? Illuminating the Covid-19 Learning Losses and Gains in Uganda. Uwezo National Learning Assessment Report, 2021. Kampala: Uwezo Uganda. UNICEF. (2021a). Annual Report 2021. UNICEF. Retrieved from https://www.unicef.org/indonesia/media/13756/file/ Annual%20Report%20 2021%20-%20Spread.pdf UNICEF. (2021b). Issue Brief: The Impact of the COVID-19 Pandemic on Children’s Learning in Indonesia. UNICEF. UNICEF, UNESCO, & World Bank. (2022). Where we are in education recovery? Retrieved from https://www.unicef.org/ media/117626/file/Where%20are%20we%20in%20Education%20Recovery?.pdf Whizz Education. (2021). Measuring the impact of COVID-19 on learning in rural Kenya. London: Whizz Education. Retrieved from https://www.whizzeducation.com/wp-content/uploads/Kenya-Covid-Impact-SCREEN.pdf. Wolf, S., Aurino, E., Suntheimer, N., Avornyo, E., Tsinigo, E., Jordan, J., … & Behrman, J. R. (2021). Learning in the Time of a Pandemic and Implications for Returning to School: Effects of COVID-19 in Ghana. Consortium for Policy Research in Education (CPRE). Woe ssmann, L. (2016). The Importance of School Systems: Evidence from International Difference s in Student Achievement. Journal of Economic Perspectives, 30(3), 3–32. https://doi.org/10.1257/jep.30.3.3 World Bank (2020a). Cost-Effective Approaches to Improve Global Learning: What Does Recent Evidence Tell Us Are ‘Smart Buys’ for Improving Learning in Low and Middle Income Countries? Washington, DC.: World Bank. World Bank (2020b). Measuring the Quality of MORA's Education Services. Washington, DC.: World Bank. World Bank (2022a). Framework for Learning Recovery during the COVID-19 Pandemic and Beyond. Washington DC: World Bank. World Bank (2022b). Teach Primary: Observer Manual (Vol. 2): Observation Sheet (English). Teach Primary Washington, D.C.: World Bank Group. World Bank (2023a). Indonesia Economic Prospects, June 2023: The Invisible Toll of COVID-19 on Learning. Washington, DC.: World Bank Group. World Bank (2023b). World Bank country classifications by income level: 2022-2023. Retrieved from https://datahelpdesk. worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups World Bank and UNESCO (2024). Indonesia Learning Poverty Country Brief (English). Learning Poverty Brief. Washington, D.C. : World Bank Group. http://documents.worldbank.org/curated/en/099082924151529593/ P17920918b1c2f0f61abb41e1a5d85e5dd2 World Bank, UNESCO, & UNICEF. (2021). The State of the Global Education Crisis: A Path to Recovery. Retrieved from https:// documents1.worldbank.org/curated/en/416991638768297704/pdf/The-State-of-the-Global-Education- Crisis-A-Path-to-Recovery.pdf Yuliansyah, A., & Ayu, M. (2021). The implementation of project-based assignment in online learning during covid-19. Journal of English Language Teaching and Learning, 2(1), 32–38. Zulaika, G., Bulbarelli, M., Nyothach, E., van Eijk, A., Mason, L., Fwaya, E., Obor, D., Kwaro, D., Wang, D., & Mehta, S. D. (2022). Impact of COVID-19 lockdowns on adolescent pregnancy and school dropout among secondary schoolgirls in Kenya. BMJ Global Health, 7(1), e007666. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 50 Learning in the Shadow of the Pandemic: 50 COVID-19 Learning Loss and Widening Learning Disparities in Indonesia Appendices Appendix A: Technical Note on the Learning Loss Survey 202337 Context Sampling, Matching and Weighting The Learning Loss Survey 2023 was designed as a The sampling framework has been adjusted from follow-up to the Service Delivery Indicators (SDI) that of the 2019 SDI survey. The main objective of the survey, last performed in Indonesia in early 2019.38 original 2019 SDI survey was to obtain a nationally The instruments employed in the SDI survey included representative sample of madrasahs under MoRA, a measure of student learning outcomes in numeracy and it included a sample of 253 madrasahs across the and literacy in Grade 4 for an almost nationally country and a sample of 87 MoECRT schools located within the immediate vicinity of sampled madrasahs representative sample, offering an exceptional (a total sample size was 350 educational institutions, portrait of the education system immediately prior including additional ten non-Islamic MoRA to the pandemic. The Learning Loss Survey 2023 was institutions). This suggests that the 2019 sample did conducted from February to March 2023 to maintain not represent a true distribution of MoECRT schools, the consistency of the timing of assessment with the as it differs with that of MoRA madrasahs. Because the SDI 2019 survey, which was completed during the objective of 2023 learning loss survey was to obtain same months in 2019. An adapted set of SDI 2019 a nationally representative sample of all educational instruments was employed, mainly to capture learning institutions (i.e. schools under MoECRT and practices during the pandemic and allow the fieldwork madrasahs under MoRA) in the country, adjustment to be accomplished within a compact timeline. to the sampling frame was made to incre ase the coverage of MoECRT schools. Instruments Statistical adjustments were made to ensure that the Some modifications were made to the original 2019 data would be nationally representative by employing SDI survey. The revised instrument removed questions matching and weighting techniques. To account for that are less relevant for the assessment of learning the re pre sentation issue of the MoECRT schools in losses and included additional questions on learning the 2019 survey, matching is performed using the practices during the pandemic. Test items for math Mahalanobis Distance Matching technique, to adjust and language remained identical to SDI 2019 to allow characteristics of the sample schools to be consistent direct comparison of items. The 2023 used a new with the national distribution. The list of provinces and classroom observation tool called TEACH,39 which was districts selected are shown in Figure A1 and Table A1. developed by the World Bank to replace the former SDI The matching was mainly accomplished using school classroom observation module. The methodology of standardized accreditation scores available from the collecting household data changed from house visits Indonesian National Accreditation Body for Schools and Madrasahs (BAN SM). This process generated a to a questionnaire to the students, in order to facilitate matching weight. data collection. 37 This appendix is an excerpt from World Bank (2023a). 38 Further information on the 2019 SDI survey can be found on the following link: https://openknowledge.worldbank.org/entities/publication/ f8d78c37-968a-55e2-96b4-9fc6067fce65 39 Further information on the instrument can be found in the following link: https://www.worldbank.org/en/topic/education/brief/teach- helping-countries-track-and-improve-teaching-quality LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 51 The survey also employs student weights to account unless the number of Grade 4 students in the sample for different number of students across educational institution is less than ten. The total number of institutions, with a school weight applied to take into students interviewed and tested is 3,863 for 2023 (see account the different number of schools each sample Table A3).41 represents. These three weights are then multiplied together to yield a single individual weight employed In each educational institution, at least one teacher in individual-level analysis, while analysis at the school was selected to be observed and interviewed. In level uses a consolidated matching and school weight. educational institutions that practice the thematic The final data set includes the following: (a) a panel teaching system—where a single teacher, usually the sample (i.e. revisiting) of 296 educational institutions, homeroom teacher, teaches a set of basic education which includes 200 MoRA madrasahs randomly subjects, including language and mathematics—one selected from the 253 original SDI sample, 87 MoECRT Grade 4 teacher was randomly selected. In educational schools, and nine non-Islamic MoRA schools; and (b) a institutions where teachers teach by subject, two newly identified 113 non-panel MoECRT schools that Grade 4 teachers were selected, one for each of the would ensure MoECRT schools would be nationally subjects. In both cases, ten Grade 4 students were representative.40 The final sample size of 2023 survey randomly selected for the assessment and given a is 409 education institutions (see Table A2). At each short questionnaire from the pool of students taught educational institution visited, randomly selected 10 by the teachers. In schools with less than ten students, Grade 4 students are included for student assessment all Grade 4 students were included in the sample. Figure A1: Coverage of geographical areas under the 2023 Learning Loss survey Source: World Bank (2023a) 40 The 113 non-panel MoECRT schools were selected with the aim of making the combined sample of MoECRT schools nationally representative. Several balance checks were performed to maintain representativeness, achieved through balancing across several covariates on school characteristics, including school status, size, and sanitation facilities. The data to perform the exercise was obtained from the MoECRT administrative data bank (Data Pokok Pendidikan or Dapodik), using the 2019 point-in-time data to maintain consistency with the original survey. 41 The SDI 2019 data focused on 3,368 Grade 4 students and assessed their learning results. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 52 Table A1: List of surveyed province and districts Province Districts Aceh Pidie, Aceh Tenggara Bali Karangasem, Bangli Banten Kota Serang DKI Jakarta Kota Jakarta Pusat Jambi Sarolangun, Tanjung Jabung Barat Jawa Barat Karawang, Pangandaran, Kota Bekasi, Bandung Barat, Sumedang Jawa Tengah Wonogiri, Blora, Temanggung, Karanganyar Jawa Timur Jombang, Trenggalek, Ponorogo Kalimantan Barat Ketapang, Sambas, Sekadau, Landak Kalimantan Selatan Tabalong, Hulu Sungai Tengah Kalimantan Tengah Katingan, Kotawaringin Timur Kepulauan Bangka Belitung Bangka Selatan Kepulauan Riau Bintan Lampung Tanggamus Nusa Tenggara Timur Alor, Flores Timur Sulawesi Selatan Pangkajene Dan Kepulauan, Kepulauan Selayar, Gowa, Tana Toraja Sulawesi Tengah Sigi, Tojo Una-Una Sulawesi Tenggara Buton Tengah, Buton Selatan, Konawe Sulawesi Utara Kep.Sangihe, Bolaang Mongondow Utara, Bolaang Mongondow Selatan, Minahasa Sumatera Selatan Ogan Komering Ilir, Muara Enim Sumatera Utara Asahan, Nias Barat, Tapanuli Utara, Samosir Source: World Bank (2023a) LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 53 Table A2: Number of educational institutions surveyed in 2023 and number of SDI 2019 institutions used for the Learning Loss analysis Public Private Total Urban Rural Sub-Total Urban Rural Sub-Total 2019 MoECRT 37 42 79 4 2 6 85 MoRA Islamic 17 25 42 64 95 159 201 MoRA non-Islamic 7 2 9 9 Total 54 67 121 75 99 174 295 2023 MoECRT 83 93 176 19 4 23 199 MoRA Islamic 17 25 42 64 95 159 201 MoRA non-Islamic 7 2 9 9 Total 100 118 218 90 101 191 409 Source: World Bank (2023a) Table A3: Number of students surveyed in 2023 and SDI 2019 students used for the Learning Loss analysis Public Private Total Urban Rural Sub-Total Urban Rural Sub-Total 2019 MoECRT 362 395 757 40 20 60 817 MoRA Islamic 170 249 419 632 902 1,534 1,953 MoRA non-Islamic 46 14 60 60 Total 532 644 1,176 718 936 1,654 2,830 2023 MoECRT 799 852 1,651 190 37 227 1,878 MoRA Islamic 170 250 420 621 892 1,513 1,933 MoRA non-Islamic 39 13 52 52 Total 969 1,102 2,071 850 942 1,792 3,863 Source: World Bank (2023a) LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 54 Appendix B: Plausible Values calculation and Analytical Approach Calculation of Plausible Values (PVs) layers: it allows us to examine not just how each student performs on their own but also how the Following the World Bank (2023a), the first part of school they attend relates to that performance. this study compares students from 2019 and 2023 to Esse ntially, HLM separate s the e ffects of individual estimate learning losses. This study uses the Plausible student characteristics from the effects of the school Values (PV) of the students’ test scores, which were environment, enabling a clearer understanding of de rive d by using Ite m Re sponse The ory (IRT), e stimating both personal and institutional influences on student student abilities by analysing their responses to outcomes. To ascertain the suitability of including selected test items and incorporating background the HLM model in our final models, we performed a information. This method generates multiple imputed likelihood ratio test. The comparison between the scores for each student, addressing measurement traditional model and the two-level model yielded LRT error and variability in true abilities estimated based on chi-square values of 682.12 in language and 1093.93 in item difficulties. It then standardizes these scores to a math, both statistically significant ( p<0.001). A p-value mean of 0 and a standard deviation of 1 based on 2019 of less than .001 signifies that the incorporation of data, serving as a baseline to evaluate the 2023 cohort's these random effects is justified and enhances the performance against the backdrop of educational model's fit to the data (Snijders & Bosker, 2011). disruptions, such as those caused by COVID-19. This approach facilitated a comparative analysis of student In addition, a fully unconditional HLM model was academic performance between the years 2019 and generated to comprehend the variance at both the 2023, providing a robust counterfactual framework within-school and between-school levels (Table B1 & for evaluating the extent of learning losses within this B2). It helps to establish a foundation for understanding timeframe. how much of the variation in student performance can be attributed to differences among students within the same school versus differences across schools. Hierarchical Linear Modelling (HLM) Examining the between-school variance can provide insights into the influence of school-level factors, such This paper also adopted hierarchical linear modelling as resources, teaching quality, and school climate, (HLM) to analyze the ke y contributors to le arning on student achievement. Conversely, understanding outcomes in 2023. Students learn as part of a larger within-school variance can reveal individual-level community within their schools, where they share factors, such as student characteristics, family experiences, teachers, and resources with their background, and personal experiences, which peers. Therefore, their performance can reflect not contribute to learning outcomes. Results from the just their individual abilities but also the influence of unconditional model revealed an ICC coefficient of the school's resources, teaching quality, and cultural .30 for language and .33 for mathematics, indicating environment. In other words, the educational data that 30 pe rce nt and 34 pe rcent of the variation in from this survey are also organized in a hierarchical or language and mathematics scores was attributed multilevel structure, meaning students are grouped to differences between schools. This implies that a within schools. This arrangement leads to a breach of considerable portion of the differences in student the assumption that observations are independent, performance in both subjects is associated with due to intra-class correlation (ICC) among subjects the school characteristics, teaching practices, and within the same school (Raudenbush & Bryk, 2002). resources of the schools they attend. Thus, schools play a significant role in shaping student outcomes To address the analytical challenges posed by such in language and mathematics, highlighting the a structure, HLM was employed in this study. HLM importance of understanding and addressing school- handle s the se challenges by treating the data in level factors to improve educational outcomes. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 55 Table B1. One-level model versus Two-level HLM Model Log Likelihood LR Test P-Value (1) Language One level -10335.634 - - Two level (school level) -9994.573 682.12 0.000 (2) Math One level -9521.4927 Two level (school level) -8974.5269 1093.93 0.000 Table B2. HLM unconditional model results (2) (4) Language Math Constant -0.373*** -0.397*** (0.032) (0.028) Between province and school variance Between Schools 0.406*** 0.325*** (0.030) (0.022) Within Schools 0.953* 0.647*** (0.051) (0.022) N of students 3,863 N of schools 409 Proportion of variance between schools (ICC) 0.299 0.335 Note: Standard errors in parentheses, * p < 0.05, ** p < 0.01, *** p < 0.001 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 56 Appendix C: Variable Descriptions Table C1. Description of Variables Variable Variable description Learning outcomes Standardized PV scores in language and mathematics Individual level Parental education level Father’s and mother’s education level: Primary education or less, lower secondary education, upper secondary education, and those with tertiary or higher education Household Asset Index Household asset index quintile (1-5): Presence or absence of certain household items Student gender Female, Male Attending preschool Participation in Early Childhood Care and Education (ECCE): Attended, Not attended Number of Books Read The total books students have read in the last month, represented as a numerical count. Health Incidents Incidence of illness or demise in close contacts during school closures: Yes or No House chores Participation in household economic activities (e.g., farming, selling goods) School Level School type MoRA, MoECRT School ownership Public, Private School area Urban, Rural Pre-COVID Access to Internet School had internet access prior-to COVID-19 pandemic Assignment frequency Number of assignments in a week during the COVID-19 school closures Assignment submission time Timeliness of a teacher's assignment submission during the COVID-19 school closures, with 1 indicating submission on the same day and 0 for submissions on subsequent days. Principals daily monitoring How often school principals engaged in monitoring activities during the COVID-19 school closures, with 1 indicating daily monitoring by the principal, and 0 for monitoring that occurred less frequently. Kurikulum Merdeka Kurikulum Merdeka is binary, with a value of 1 indicating schools that use the Kurikulum Merdeka, and 0 for those that do not. Government Support Type of Assistance provided by the Government during the pandemic Additional Funds, IT Hardware, Internet Connection, Online learning resources, Books and other physical learning, Principal training, Teacher training, and Information dissemination. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 57 Appendix D: Average Student Test Score by Background Characteristics, 2023 Table D1: Average Student Test Scores for Groups with Different Characteristics, 2023 Language score Mathematics Score Count (N) Mean (SD) Mean (SD) Household level Consumption Quintile 1-Lowest -0.849 (1.310) -0.682 (1.014) 810 2 -0.481 (1.185) -0.48 (1.005) 760 3 -0.254 (1.088) -0.332 (0.896) 766 4 -0.109 (1.016) -0.151 (0.948) 767 5-Highest 0.14 (0.966) 0.067 (0.943) 760 Father's education Primary or less -0.78 (1.343) -0.676 (0.978) 1217 Lower Secondary -0.407 (1.097) -0.402 (0.933) 808 Upper Secondary -0.042 (0.977) -0.125 (0.941) 1392 Tertiary 0.339 (0.821) 0.256 (0.916) 350 Mother's education Primary or less -0.744 (1.317) -0.679 (0.981) 1262 Lower Secondary -0.375 (1.131) -0.41 (0.940) 865 Upper Secondary -0.065 (0.984) -0.124 (0.913) 1238 Tertiary 0.329 (0.814) 0.3 (0.900) 437 Student's gender Male -0.459 (1.219) -0.411 (1.024) 1930 Female -0.069 (1.048) -0.145 (0.944) 1933 Attending preschool No -0.82 (1.346) -0.659 (0.970) 454 Yes -0.189 (1.103) -0.226 (0.985) 3409 Language at home Non-bahasa -0.374 (1.151) -0.37 (0.987) 2295 Bahasa -0.154 (1.145) -0.186 (0.991) 1568 Close One's Death No -0.203 (1.126) -0.211 (0.992) 2597 Yes -0.382 (1.195) -0.407 (0.982) 1266 Close One's Illness No -0.255 (1.124) -0.272 (0.979) 1915 Yes -0.271 (1.178) -0.282 (1.006) 1948 House chores Non-economic -0.188 (1.121) -0.231 (0.995) 2924 Economic -0.502 (1.219) -0.426 (0.972) 939 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 58 Language score Mathematics Score Count (N) Mean (SD) Mean (SD) School Level School Type MoECRT -0.319 (1.179) -0.279 (0.995) 1878 MoRA -0.085 (1.043) -0.273 (0.988) 1985 School Ownership Public -0.356 (1.159) -0.326 (1.000) 2071 Private -0.018 (1.099) -0.15 (0.964) 1792 School Area Rural -0.52 (1.180) -0.556 (1.000) 2044 Urban -0.128 (1.115) -0.131 (0.958) 1819 Pre-COVID Access to Internet No -0.624 (1.237) -0.522 (1.003) 1213 Yes -0.136 (1.094) -0.191 (0.975) 2650 Assignment submission time Not on the same day -0.356 (1.174) -0.323 (1.008) 2587 On the same day -0.092 (1.092) -0.193 (0.960) 1276 Principals monitoring frequency Not Everyday -0.353 (1.198) -0.368 (0.982) 2228 Everyday -0.119 (1.061) -0.133 (0.994) 1635 Kurikulum Merdeka No Merdeka -0.263 (1.183) -0.297 (0.992) 2723 Merdeka -0.264 (1.113) -0.252 (0.995) 1140 Government level Additional Funds No -0.269 (1.180) -0.285 (1.002) 2873 Yes -0.242 (1.047) -0.248 (0.961) 990 IT Hardware No -0.208 (1.138) -0.279 (0.970) 2947 Yes -0.411 (1.179) -0.274 (1.053) 916 Internet Connection No -0.342 (1.190) -0.34 (1.005) 3038 Yes -0.036 (1.005) -0.096 (0.935) 825 Online learning resources No -0.306 (1.159) -0.314 (0.991) 3468 Yes -0.004 (1.080) -0.057 (0.978) 395 Books and other learning materials No -0.25 (1.153) -0.273 (0.986) 3369 Yes -0.344 (1.148) -0.304 (1.035) 494 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 59 Language score Mathematics Score Count (N) Mean (SD) Mean (SD) Principal training No -0.256 (1.140) -0.26 (0.997) 1904 Yes -0.272 (1.168) -0.299 (0.989) 1959 Teacher training No -0.256 (1.158) -0.225 (0.996) 1500 Yes -0.268 (1.150) -0.312 (0.990) 2363 Information dissemination No -0.247 (1.165) -0.212 (1.005) 1913 Yes -0.28 (1.140) -0.344 (0.976) 1950 Total -0.265 (1.153) -0.277 (0.993) 3863 Note: Weighted statistics, encompassing means, standard deviations, minimum, and maximum values, have been utilized to account for variations in student and school populations. LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 60 Appendix E: Detailed Outputs from the HLM Analysis These tables are related to the analysis of Section 6.2.2. Table E1: HLM Analysis of the Interaction between School Location and Internet Access (1) (2) Language Mathematics I. Individual Level Student's gender (Female=1, Male=0) 0.351*** 0.257*** (0.046) (0.038) Household size -0.065*** -0.020* (0.015) (0.011) Attending Preschool (Yes=1, No=0) 0.139 0.036 (0.088) (0.085) Close One's Death (Yes=1, No=0) -0.166*** -0.153*** (0.052) (0.041) Helping house chores (Economic=1, No or non-economic=0) -0.075 0.018 (0.061) (0.048) Number of Books Read during Last Month 0.020*** 0.013** (0.006) (0.006) II. School Level Rural schools with No pre-COVID access to internet as reference Rural schools with pre-COVID internet access 0.260*** 0.154** (0.076) (0.067) Urban schools without pre-COVID internet access 0.193* 0.253*** (0.105) (0.096) Urban schools with pre-COVID internet access 0.344*** 0.375*** (0.079) (0.074) Assignment frequency (Number of assignments in a week) 0.045*** 0.051*** (0.016) (0.015) Assignment submission time (On the same day=1, Otherwise=0) 0.121*** 0.007 (0.043) (0.042) Principals monitoring frequency (Everyday=1, Not Every day =0) 0.054 0.063 (0.051) (0.052) III. Government Level Internet Connection (Yes=1, No=0) 0.115** 0.066 (0.053) (0.064) Online learning resources (Yes=1, No=0) 0.055 -0.016 (0.078) (0.090) IT Hardware (Yes=1, No=0) -0.003 0.098* (0.057) (0.059) Family SES as control Constant -1.265*** -1.256*** (0.135) (0.108) Between province and school variance Between Schools 0.242*** 0.228*** (0.017) (0.016) Within schools 0.852*** 0.609*** (0.045) (0.020) Note: The HLM analysis uses weighted methods and clustering standard errors, accounts for these demographic differences. * p<0.05, ** p<0.01, *** p<0.001 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 61 Table E2: HLM Analysis of Pre-COVID Internet Access and Government Internet Support Interaction (1) (2) Language Mathematics I. Individual Level Student's gender (Female=1, Male=0) 0.356*** 0.231*** (0.056) (0.041) Household size -0.058*** -0.024* (0.015) (0.013) Attending Preschool (Yes=1, No=0) 0.144 0.049 (0.101) (0.094) Close One's Death (Yes=1, No=0) -0.135** -0.153*** (0.055) (0.047) Helping with house chores (Economic=1, No or non-economic=0) -0.101 0.025 (0.071) (0.049) Number of Books Read during Last Month 0.023*** 0.014** (0.007) (0.007) II. School Level No pre-COVID access to internet and No ministry support as reference Ministry Support for Internet Connection only 0.170* 0.111 (0.096) (0.138) Pre-COVID Access to Internet only 0.268*** 0.195*** (0.071) (0.065) Both pre-COVID internet access and ministry support for internet access. 0.419*** 0.296*** (0.084) (0.088) Assignment frequency (Number of assignments in a week) 0.041** 0.049*** (0.019) (0.016) Assignment submission time (On the same day=1, Otherwise=0) 0.119** 0.009 (0.053) (0.048) Principals monitoring frequency (Everyday=1, Not Every day =0) 0.051 0.083 (0.055) (0.056) III. Government Level Online learning resources (Yes=1, No=0) -0.013 0.103 (0.058) (0.064) IT Hardware (Yes=1, No=0) 0.005 -0.024 (0.088) (0.091) Family SES as control Yes Yes Constant -1.172*** -1.183*** (0.145) (0.127) Between province and school variance Between Schools 0.266*** 0.268*** (0.020) (0.018) Within schools 0.772*** 0.555*** (0.043) (0.022) Note: The HLM analysis uses weighted methods and clustering standard errors, accounts for these demographic differences. * p<0.05, ** p<0.01, *** p<0.001 Appendix F: Learning Outcomes and Individual Backgrounds by Year Appendix tables F1 and F2 provides regression analysis of 2019 and 2023 by key household and individual characteristics to show the differences in language and math scores across different groups. Appendix F1. Individual Background and Language Outcomes by Year (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) 2019 2023 2019 2023 2019 2023 2019 2023 2019 2023 Consumption Quintile 2 -0.037 0.368*** (0.237) (0.089) 3 0.158 0.595*** (0.172) (0.092) 4 0.557*** 0.740*** (0.132) (0.091) 5-highest 0.522*** 0.990*** (0.144) (0.087) Father's Education Lower Secondary 0.165 0.373*** (0.198) (0.090) Upper Secondary 0.334 0.738*** (0.182) (0.082) Tertiary 0.629*** 1.118*** (0.179) (0.099) Mother's Education Lower Secondary 0.098 0.369*** (0.179) (0.089) Upper Secondary 0.097 0.679*** (0.192) (0.084) LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA Tertiary 0.324* 1.073*** (0.158) (0.099) Student's gender (Female=1, Male=0) 0.361*** 0.390*** (0.078) (0.054) School area (Urban =1, Rural=0) 0.348*** 0.392*** (0.091) (0.078) Constant -0.278** -0.849*** -0.270 -0.780*** -0.135 -0.744*** -0.182* -0.459*** -0.212** -0.520*** (0.097) (0.072) (0.140) (0.078) (0.132) (0.074) (0.077) (0.056) (0.068) (0.054) adj. R2 0.063 0.085 0.058 0.105 0.015 0.098 0.033 0.029 0.029 0.026 Note: The HLM analysis uses weighted methods and clustering standard errors, accounts for these demographic differences. * p<0.05, ** p<0.01, *** p<0.001 62 Appendix F2. Individual Backgrounds and Mathematic Outcomes by Year (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) 2019 2023 2019 2023 2019 2023 2019 2023 2019 2023 Consumption Quintile 2 -0.158 0.201* (0.181) (0.082) 3 0.341* 0.349*** (0.136) (0.069) 4 0.398* 0.530*** (0.190) (0.078) 5-highest 0.380* 0.748*** (0.158) (0.081) Father's Education Lower Secondary 0.267 0.274*** (0.153) (0.068) Upper Secondary 0.496** 0.551*** (0.158) (0.062) Tertiary 0.774*** 0.932*** (0.168) (0.095) Mother's Education Lower Secondary 0.533*** 0.269*** (0.141) (0.068) Upper Secondary 0.491** 0.555*** (0.159) (0.067) Tertiary 0.799*** 0.979*** (0.169) (0.087) Student's gender (Female=1, Male=0) 0.310*** 0.265*** (0.088) (0.041) School area (Urban =1, Rural=0) 0.464*** 0.424*** (0.102) (0.071) Constant -0.242** -0.682*** -0.328*** -0.677*** -0.376*** -0.680*** -0.156* -0.412*** -0.284*** -0.557*** (0.093) (0.062) (0.098) (0.054) (0.083) (0.060) (0.070) (0.048) (0.069) (0.050) LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA adj. R2 0.048 0.070 0.093 0.091 0.106 0.107 0.024 0.018 0.051 0.041 Note: The reference groups are the lowest quintile group, primary or less educated parents. Regression analysis uses a weighted methodology to account for the variations in school and student weights. * p<0.05, ** p<0.01, *** p<0.001. 63 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 64 Appendix G: Teach Instrument and Analysis of Teachers by Gender Description of Teach instruments Teach Primary (hereafter “Teach”) is a classroom observation tool developed by the World Bank that has been used in over 30 middle income countries across the world. It captures the quality of teaching practices by measuring (i) time on task: the time teachers spend on learning and the extent to which students are on task, and (ii) the quality of teaching practices measured by three primary areas: Classroom Culture, Instruction, and Socio-emotional Skills. The tool underwent a rigorous development and validation process which met the appropriate psychometric criteria of reliability.42 As part of the Time on Task component, three “snapshots” of 1–10 seconds are used to record both the teacher’s actions and the number of students who are on task throughout the observation. The quality of teaching practices is evaluated in three areas: Classroom Culture, Instruction, and Socio-emotional Skills. These areas have nine corresponding elements that point to twenty-eight behaviors. The behaviors are characterized as low, medium, or high, based on the evidence observed in this classroom. These preliminary scores are translated into a five-point scale, which quantifies the teacher’s practices as captured in two, 15-minute observations. This study employs Teach to investigate the current landscape of teaching practices in Indonesia. A total of 993 observations were collected, encompassing 501 teachers at two time points during their classes—the first and last 15 minutes. It included 405 primary schools, with 54 percent from MoECRT and 46 percent from MoRA, strategically chosen to ensure national representativeness. Grade 4 primary school classrooms were observed across subjects, including Mathematics (46 percent), Language (Bahasa Indonesia, 31 percent), and other subjects (23 percent). Analysis of Teaching Practices by Gender of the Teachers An additional analysis is conducted to see if there are any patterns of teaching practices by gender of teachers. From the interviews, Table G1 shows the frequency of assignments and the percentage of students submitting the required assignments, and this pattern is compared by teachers’ gender. This shows no statistically significant difference in practices by gender. Table G1. Teaching Practice during the Pandemic by Gender Female Male Diff (SE) Mean (SD) Mean (SD) (a-b) (a) (b) Assignment frequency (Number of assignments in a week) 4.10 (1.83) 3.97 (1.94) 0.27 (0.21) % of students submitting assignments on the same day 0.36 (0.48) 0.25 (0.43) 0.04 (0.05) Note: 'Diff' represents the differential impact of each factor between male and female teachers, calculated using a weighted regression analysis. Marginal effects for binary outcomes (e.g., assignment submission time) are derived from probit regression analysis, indicating the change in percentage points for these categorical variables. *p<0.05, **p<0.01, ***p<0.001 42 Teach Primary: Helping Countries Track and Improve Teaching Quality LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 65 Table G2 shows the gender dimension of the TEACH score. This analysis shows there are statistically significant differences, where female teachers perform constantly higher than male teachers on total Teach score and all three areas. However, a regression analysis shows there are not statistically significant differences in students’ test scores in language and math between the gender of teachers (Table G3). Table G2. Teach Score by Teachers’ Gender Female Male Diff (SE) Mean (SD) Mean (SD) (a-b) (a) (b) Overall Teach score 2.698 (0.435) 2.634 (0.423) 0.064*** (0.001) A. Classroom Culture 3.453 (0.55) 3.391 (0.585) 0.061*** (0.001) B. Instruction 2.522 (0.61) 2.498 (0.608) 0.024*** (0.001) C. Socioemotional Skills 2.12 (0.575) 2.014 (0.575) 0.106*** (0.001) Note: The Teach score difference represents the disparity between teacher’s genders as measured by a weighted regression analysis. * p<0.05, ** p<0.01, *** p<0.001. Table G3. Analysis of Teachers’ Gender on Students’ Learning Outcome (Simply Model) (1) (2) Language Math Teachers' Gender (Female=1, Male =0) 0.038 0.153 (0.058) (0.098) Constant -1.224*** -1.326*** (0.135) (0.117) Between province and school variance Between schools 0.237*** 0.233*** (0.016) (0.017) Within schools 0.845** 0.608*** (0.044) (0.020) Note: After controlling for the variables in Table 5.2, the regression analysis of teacher gender on student learning outcomes is performed. It uses weighted methods and clustered standard errors to account for these demographic differences. * p<0.05, ** p<0.01, *** p<0.001 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 66 Appendix H: Distribution of Test Scores and Key Parameters Table H1 provides shows different perspectives of relationship between the test scores and key parameters analysed. Schools are categorized into 5 groups, with schools at 20 percent of test score distribution belonging to Quintile 5 and bottom 20 percent of test score distribution belonging to Quintile 1. Then, the average characteristics of Quintile 1 to 5 groups are presented in Table H1. This shows very large performance gaps in language (1.74 SD) between quintiles 1 (average score of -1.29) and quintile 5 (average score of 0.45). The distribution of school characteristics overall follows consistent patterns with the analyses in the main text although these are not causality but observed correlations. Table H1: Distribution of test scores by quintile and average school characteristics by quintile Quintile 1 Quintile Quintile Quintile Quintile 5 (Bottom Total 2 3 4 (Top 20%) 20%) Language Average Test Scores -1.29 -0.67 -0.31 0.04 0.45 -0.27 % Urban 43 49 54 81 83 65 % MoECRT 86 83 78 76 64 76 % Public 82 80 83 75 51 72 % Pre-COVID Access to Internet 40 72 64 89 87 73 % Assignment submission on the same day 35 21 32 37 35 73 % Principals monitoring every day 23 29 32 48 52 38 % Kurikulum Merdeka 38 47 42 54 36 43 % Government Support IT Hardware 16 25 31 16 23 22 Internet Connection 38 24 45 15 20 27 Online learning resources 8 23 23 30 36 25 Additional Funds 4 4 19 19 18 14 Books and other physical learning materials 13 17 17 15 10 14 Principal training 48 34 52 56 38 46 Teacher training 57 55 66 75 49 60 Information dissemination 56 45 45 54 48 50 Math Average Test Scores -1.16 -0.71 -0.41 -0.09 0.41 -0.28 % Urban 41 44 59 78 83 65 % MoECRT 80 66 80 77 76 76 % Public 79 69 80 79 61 72 % Pre-COVID Access to Internet 60 56 64 86 86 73 % Assignment submission on the same day 38 25 33 39 28 32 % Principals monitoring every day 27 27 37 41 50 38 % Kurikulum Merdeka 41 25 53 53 42 43 % Government Support IT Hardware 19 31 22 18 23 22 Internet Connection 29 32 28 17 29 27 Online learning resources 9 13 21 36 36 25 Additional Funds 5 12 13 14 20 14 Books and other physical learning materials 18 19 11 7 16 14 Principal training 41 59 48 32 50 46 Teacher training 61 74 58 57 57 60 Information dissemination 58 49 54 51 42 50 Note: This analysis includes all sampled schools (MoECRT and MORA) and is regrouped by test score distribution. Appendix I: Learning Loss Literature by Country with Estimated Effect and Analytical approach Country Income Region Short Effect National Months Grade Analytical approach Level Reference size Sample Closed Australia HIC East Asia & Gore et al. 0.04 SD No 4.25 Primary Cohort comparison Pacific (2021) education (DiD approach with linear mixed models) Belgium HIC Europe & Maldonado & –0.18 SD No 4 Primary Cohort comparison Central Asia De Witte (2022) education (DiD approach) Brazil UMIC Latin America Lichand et al. –0.31 SD No 5.75 High school Cohort comparison & Caribbean (2022) students (DiD approach) Burkina Faso LIC Sub-Saharan UNESCO Reading: +3.2pp Yes 3.25 Grade 6 Cohort comparison Africa (2022) Math: +5.8pp (descriptive analysis) Burundi LIC Sub-Saharan UNESCO Reading: −0.2 pp Yes 0 Grade 6 Cohort comparison Africa (2022) Math: −3.5 pp (descriptive analysis) Côte d’Ivoire LMIC Sub-Saharan UNESCO Reading: +0.4 pp Yes 3.25 Grade 6 Cohort comparison Africa (2022) Math: +1.4 pp (descriptive analysis) Denmark HIC Europe & Birkelund & 0.02 SD Yes 5.5 Grade 2, 4, 6, Cohort comparison Central Asia Karlson (2023) and 8 (difference-in-differences approach) Germany HIC Europe & Schult et al. Overall: –0.07 No 3.75 Grade 5 Cohort comparison Central Asia (2022) Reading: –0.07 SD (DiD approach) Numbers: –0.03 SD Italy HIC Europe & Contini et al. –0.19 SD No Not specified Grade 2 and 3 Cohort comparison Central Asia (2022) (Primary) (difference-in-differences approach) Italy HIC Europe & Bazoli et al. –0.16 SD Yes Not specified Grade 5, 8, 13 Cohort comparison Central Asia (2022) (difference-in-differences approach) Kenya LMIC Sub-Saharan Wizz Education Math: −3.5 months lost No 9.25 NA Cohort Growth at different time points Africa (2021) LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA Mexico UMIC Latin America Hevia et al. Overall: –0.54 SD Yes 14.5 10 to 15 Cohort comparison & Caribbean (2022) Reading: 0.34–0.45 SD years old (Cohen's d) Math: 0.62–0.82 SD Netherlands HIC Europe & Schuurman et al., –0.22 SD No Not specified Grade 3 - 5 Piecewise growth models Central Asia (2023) Netherlands HIC Europe & Engzell et al. –0.07 SD Yes 2.5 Aged 8 to 11 Cohort comparison Central Asia (2021) (DiD approach) Pakistan LMIC Central Asia Crawfurd et al. No difference No 13.75 Primary Sample cohort compared South Asia 2021) education at different time points Senegal LMIC Sub-Saharan UNESCO Reading: −1.4 SD N/A 5.25 Grade 6 Cohort comparison Africa (2022) Math: −0.6 SD (descriptive analysis) 67 South Africa UMIC Sub-Saharan Ardington et al. –0.65 SD No 4.75 Grade 2 & 4 Cohort comparison Africa (2021) (DiD approach) Country Income Region Short Effect National Months Grade Analytical approach Level Reference size Sample Closed Spain HIC Europe & Arenas Jal & –0.04 SD Yes Not specified Primary and Cohort comparison Central Asia Gortazar (2022) secondary (DiD approach) education Sweden HIC Europe & Hallin et al. Primary Cohort comparison 0.07 SD Yes 0 Central Asia (2022) education (descriptive analysis) Switzerland HIC Europe & Tomasik et al. –0.07 SD No 2 Primary Piecewise growth models Central Asia (2021) education Uganda LIC Sub-Saharan UWEZO Uganda Literacy and numeracy: Yes 18.25 Grades 3 to 7 Cohort comparison Africa (2021) +6.1 pp (descriptive analysis) United Kingdom HIC Europe & Kim et al. Overall: –0.16 SD Yes 4.5 Grade 1 to 2 Cohort comparison at Central Asia (2021) Reading: −0.17 SD different time points Math: −0.14 SD United Kingdom HIC Europe & Blainey & Hannay, Overall: –0.19 SD Yes 4.5 Primary Piecewise growth models Central Asia (2021) Math: −0.09 to −0.02 SD education Reading: −0.17 SD to −0.05 SD United States HIC North America Locke et al. –0.14 SD Yes Not specified Primary Cohort comparison (descriptive analysis), (2021) education piecewise growth models United States HIC North America Lewis et al. –0.14 SD Yes Not specified Grade 2 to 5 Cohort comparison (2021) (DiD approach) United States HIC North America Pier et al. –0.14 SD No 14.5 Primary and Cohort comparison (2021) secondary (DiD approach) education United States HIC North America Lewis et al. –0.10 SD Yes Not specified Grade 3 to 8 Cohort comparison (2021) (Descriptive analysis) United States HIC North America Domingue et al. –0.09 SD No 4.75 NA Piecewise growth models (2022) LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA Zambia LMIC Sub-Saharan UNESCO (2022) Reading: 0.5 SD Yes 6.75 Grade 5 Cohort comparison Africa Math: −1.4 SD (Descriptive analysis) Note: The income levels mentioned throughout this document are classified according to the World Bank's income categories: HICs: High-Income Countries. LICs: Low-Income Countries. LMICs: Lower-Middle-Income Countries. UMICs: Upper-Middle-Income Countries. 68 LEARNING IN THE SHADOW OF THE PANDEMIC: COVID-19 LEARNING LOSS AND WIDENING LEARNING DISPARITIES IN INDONESIA 69