The digital future of teacher training in Indonesia: What’s Next? February 2022 II The digital future of teacher training in Indonesia: What’s Next? The digital future of teacher training in Indonesia: What’s Next? This report was prepared by Noah Yarrow, Noviandri N. Khairina, Jacobus Cilliers, and Indah S.Z. Dini. This work is a product of the staff of The World Bank. Financial support for this work was generously provided by the Australian Government under the Australia-World Bank Indonesia Partnership (ABIP). The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions © 2022 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for non-commercial purposes as long as full attribution to this work is given. All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Attribution Please cite the work as follows: Yarrow, Noah; Khairina, Noviandri; Cilliers, Jacobus; Dini, Indah. 2022. The digital future of teacher training in Indonesia: What’s Next?. World Bank, Jakarta. © World Bank. Contact Information The authors can be contacted at nkhairina@worldbank.org, nyarrow@worldbank.org, ejc93@georgetown.edu, idini@worldbank.org. Photo Credits Cover Photo: Freepik.com Photo Page 2: Freepik.com Photo Page 8: Freepik.com The digital future of teacher training in Indonesia: What’s Next? III Table of Contents Figures and Tables IV Abbreviations IV Abstract 01 1 Introduction 03 2 Literature review: Effective Online 05 Teacher Professional Development 3 Indonesian context 07 3.1 Teacher training in Indonesia 08 3.2 Online Training Provider Mapping 09 4 Methodology 10 4.1 Sample 10 4.2 Sampling strategy 10 4.3 Survey protocol and survey response rates 12 4.4 Survey weights 12 4.5 Description of the survey 12 5 Results 13 5.1 Study Limitations 18 6 Conclusion and Policy Recommendations 19 References 23 Annex 1: Provider Mapping 25 Annex 2: Summary Statistics 27 Annex 3: Survey Instruments 29 IV The digital future of teacher training in Indonesia: What’s Next? Figures Figure 1 Theory of change linking teacher training programs and student learning. 05 Figure 2 Teacher participation in online training 13 Figure 3 The proportion of teachers who participated in online training, by age and location. 14 Figure 4 Perceptions of teachers who did not participate in online training 14 Figure 5 Who organized and who paid for training, by teacher type 15 Figure 6 Topics covered during training 16 Figure 7 Content of online training 16 Figure 8 Profile of teachers who completed training 17 Figure 9 Problems in Implementing what was learned during the training 17 Figure 10 Perceptions of online training 18 Tables Table 1 Summary statistics 11 Table 2 Predictors for completing the training 17 Abbreviations Balitbangdiklat Research Center, Education, and Training Agency (part of MoRA) DGHE Directorate General of Higher Education DGQITEP Directorate General of Quality Improvement of Teachers and Education Personnel ECED Early Childhood Education E-Pensa Spatial Reasoning (under PPPPTK Math Program) ETT Structured E-Training (under PPPPTK Math Program) GPK Training for Special Education Teachers (GTK Program) HYLITE Hybrid Learning for Indonesian Teachers ICT Information and Communication Technology IDL Indonesia Digital Learning (PGRI Program) MoECRT Ministry of Education, Culture, Research and Technology MoH Ministry of Health MoHA Ministry of Home Affairs MoRA Ministry of Religious Affairs PGP Program Guru Penggerak (under PPPGTK Program) PGRI Indonesian Teachers Association PPG Pendidikan Profesi Guru (under PPPGTK Program) PPPGTK Directorate of Professional Education and Development of Teachers and Education Personnel (part of MoECRT) PPPPTK Math The Center for Development and Empowerment of Mathematics Teachers and Educational Personnel Pusdatin Center for Data and Information Technology (part of MoECRT) SEAMOLEC Southeast Asian Ministers of Education Organization Regional Open Learning Center ToT Training of Trainers TPD Teacher Professional Development WAVES Workshop and Virtual E-Learning Series (PGRI Program) The digital future of teacher training in Indonesia: What’s Next? 01 Abstract We describe Indonesia’s online teacher training ecosystem with unique data collected from both teachers and providers before and during the COVID-19 period. First, we conducted a detailed mapping of the eight largest providers of online teacher training in Indonesia, covering 25 programs. We find that the majority of programs are short in duration and focus on digital literacy skills and remote learning. Training programs were mostly provided using online lectures, few provided individual coaching, while none provided opportunities for personalized learning. Second, we conducted a nationally representative phone survey of 435 primary and junior secondary teachers spanning 30 provinces across Indonesia (66 percent of whom are female teachers). The teacher survey was conducted between February and March 2021 and covered teachers under both the Ministry of Education, Research, and Technology (MoECRT) and the Ministry of Religious Affairs (MoRA). We find that 44 percent of primary and junior secondary teachers participated in online learning during the pandemic, and that three quarters of these teachers had never participated in online training prior to the pandemic. Training completion rates were high (80 percent), and higher if training was linked to a promotion. These training sessions were typically short, one to three days, and were provided through online seminars or downloadable materials. They rarely included components that would support teachers to apply these lessons, such as providing opportunities for practice or ongoing follow-up or support. Many training participants reported challenges in implementing what they learned from online training. Most of the teachers who participated (88 percent) would like to continue receiving training online even after the pandemic ends. These results suggest that demand for online training is expected to persist, but more can be done to improve their quality. 02 4 The digital future of teacher training in Indonesia: What’s Next? The digital future of teacher training in Indonesia: What’s Next? 03 1 Introduction Prior to 2007, training opportunities for Indonesian in 2020, but due to COVID-19, almost all schools were teachers tended to be limited to in-person training closed, and classroom observations as part of the courses which required teachers to travel often evaluation were not possible. The team decided to long distances, taking time out of teaching develop a preliminary study to examine existing practices activities. In 2007, both the Ministry of Education, of online teacher training and teacher motivation during Culture, Research and Technology (M0ECRT) and the the pandemic, which may affect the design of the planned Ministry of Religious Affairs (MoRA) began to adopt a impact evaluation. hybrid-model of training with a strong Information and Communication Technology (ICT) -based component as We conducted a phone-based survey in February-March a scalable approach to expand access to different types 2021 on a nationally representative sample of 435 of in-service training . Despite these efforts, based on 1 primary and junior secondary teachers in 30 different existing models and available delivery systems, in early provinces across Indonesia. The survey aimed to better 2021, MoECRT predicted that all teachers would only understand the level of teacher participation in online receive the specific in-service training (Program Profesi training programs, as well as their motivation and the Guru, PPG) they required for official certification by challenges they experienced. Survey questions focused on 2035 (MoECRT, 2021). teachers’ prior participation in online training programs, their reasons for participating (or not participating), and Like many countries around the world, Indonesia their experiences of the training itself. We performed was hit by the COVID-19 pandemic in March 2020. stratified random sampling, so our sample is meant to be COVID-19 has challenged almost every element of representative at a national level, with the caveat (discussed education, including professional development support below) that survey non-response may bias the results. for teachers. School closures in countries around the As a complement to the teacher surveys, we conducted a world meant that much of this training shifted online. detailed mapping of the seven largest providers of online The pandemic simultaneously increased the demand for training in Indonesia and coded the training along multiple training in technology specifically, to enable teachers to dimensions such as program content, delivery mechanisms, deliver their own classes online. Few empirical studies and duration, building on the provider mapping survey to date have looked at the challenges of online teacher developed by Arancibia et al (2018). This activity is part of training in middle-income countries such as Indonesia. the preparation process for an impact evaluation of specific in-service teacher training programs in Indonesia, financed The World Bank team was preparing to conduct an with the support of the Australian Government through the impact evaluation study of online teacher training Australia-World Bank Indonesia Partnership (ABIP). 1 Hybrid Learning for Indonesian Teachers (HYLITE) program was introduced as part of MoECRT program under responsibility of Directorate General of Quality Improvement of Teachers and Education Personnel (DGQITEP) and Directorate General of Higher Education (DGHE) and was facilitated by SEAMOLEC in 2007. More information is available here: https://seamolec.org/projects/3 04 The digital future of teacher training in Indonesia: What’s Next? Indonesia is a large lower-middle income country independently how to implement it, while 51 percent with approximately 4.8 million teachers and indicated that they would need additional support to education personnel . Schools closed in March 2020 2 implement it and three percent said they had never in an attempt to mitigate infection from COVID-91, and attempted to implement it. were partially reopened based on the situation in each district starting July 2020 following the issuance of Joint Online teacher training greatly expanded during Ministerial Decree of Ministry of Education, Research, school closures, but the quality and impact of this and Technology (MoECRT), the Ministry of Religious training is unclear. Pandemic-related restrictions shut Affairs (MoRA), the Ministry of Health (MoH), and the down almost all in-person teacher training in Indonesia. Ministry of Home Affairs (MoHA) in June and the gradual The closures served as a “gateway” to online training, re-opening for in-person learning in most parts of the as many teachers were forced to overcome existing country as of September 2021. By November 2021, 51% psychological and technical hurdles to participate of schools were open for limited face-to-face learning (Bhardwaj, Yarrow and Cali, 2020). This process of across the country3. learning and participation could lead to increased opportunities for future use of online training. However, We find that participation in online training this opportunity should be weighed against the current surged during pandemic-related school closures. quality and focus of online training. While it was According to our survey, 44 percent of primary and successfully delivered under difficult circumstances, junior secondary teachers in Indonesia participated in some of its characteristics mean that it is unlikely to online learning during the pandemic. Three quarters of improve teaching practice. More needs to be done to these teachers had never participated in online training develop effective online training, meaning training that previously. Most of the online training was provided will change teacher practices in ways that will improve by government agencies free of charge, and the focus student learning outcomes. The ability of teacher was predominantly on the use of technology. These training to improve student learning is a concern for training sessions were typically short, one to three teacher training in general, for teacher training in days, and in a synchronous, online teaching format Indonesia, and for online teacher training during the (webinar). They rarely included components that pandemic, which is the focus of this analysis. would support teachers to apply these lessons such as providing opportunities for practice, nor did they This paper proceeds as follows. Section II discusses provide coaching or mentoring through ongoing follow- relevant literature; Section III provides background to up or support. Most of the teachers in our sample (80 the Indonesian context, including results of a teacher percent) completed the training, but this fraction was training provider mapping conducted in 2020 and early substantially lower for rural teachers, and higher when 2021. Section IV discusses the methodology, section V the training was linked to a promotion or certification. presents results, and section VI provides broad policy recommendations. '' In terms of teachers’ perceptions, they responded According to our survey, favorably to the training and most (88 percent) would like to continue receiving training online after the pandemic ends. Despite these positive 44 percent of primary and junior secondary teachers in reports, many training participants also reported challenges in implementing what they learned from the online training. 22 percent indicated that they could Indonesia participated in online learning during the pandemic. '' implement what they were taught without difficulty and another 24 percent indicated that they could figure out 2 Education personnel include school principals, supervisors, administrative staff, librarian, tutors, and other staff working at schools. 3 Limited face-to-face learning (Pertemuan Tatap Muka Terbatas) facilitates students to attend school between 1-3 days per week with certain restrictions including shorter duration of class and requirement to wear masks all the time. Data as of November 16 is retrieved from here: https://sekolah.data.kemdikbud.go.id/ kesiapanbelajar/dashboard#pbm The digital future of teacher training in Indonesia: What’s Next? 05 2 Literature review: Effective Online Teacher Professional Development This section provides a brief review of studies that examine the effectiveness of virtual teacher professional development programs. There is limited high-quality evidence of the impact of online teacher professional development programs (TPDs), and most of the evidence comes from developed countries. Nonetheless, one can draw conclusions on how to design effective programs, building on both broad theoretical principles and evidence of in-person teacher professional development. Figure 1 - Theory of change linking teacher training programs and student learning. Participate in Improved knowledge/ Attempt new TPD program understanding techniques Greater student Behavior Evaluate and learning change feedback Most teacher professional development programs reached a point where (i) they see the benefit of these hope to not only impart knowledge but facilitate techniques in the classroom and (ii) applying these behavioral change. As outlined by Guskey (2002), techniques becomes easy and automatic, will they knowledge transfer and understanding are only the apply them in the classroom. Teachers therefore need first steps towards facilitating a change in teaching to receive ongoing support and feedback. An expert practice (see Figure 1 above). Teachers then need that observes their teaching and provides targeted to be provided with an opportunity to try the new feedback in the classroom can help teachers hone techniques, and then the opportunity for them to their skills. This will not only motivate teachers to evaluate their own performance. During this process try the new techniques, but it will also speed up the they need to receive targeted feedback to improve acquisition of a new skill set. on their techniques. Only when teachers have 06 The digital future of teacher training in Indonesia: What’s Next? In fact, the most successful teacher professional used coaches and mentors as part of its approach to development programs apply these principles. supporting teachers in schools and working groups Popova et al (2108) evaluated 33 professional after the program ends (INOVASI, 2020). However, development programs that have been implemented it is difficult to replicate all of these components in a in developing countries. After a review of evidence and virtual setting. A trainer/coach cannot provide targeted conversations with organizations that implemented feedback if she cannot observe teaching; and teachers successful TPD programs, they concluded that the might be less motivated to attempt the new teaching following features are associated with better outcomes: techniques if they are not being monitored. Indeed, the evidence of the efficacy of online TPD in developing 1. Opportunities for practice. In the most countries is mixed (Bruns et al, 2017; Cilliers et al, 2021). successful programs, teachers have a lot of time to practice the teaching techniques Successful online TPD programs we are aware of and work with the material they receive (e.g. Allen et al. 2011; Powel, et al 2010) all share the during the training. following components: they started the program 2. with an in-person workshop and relied on targeted Follow-up visits to the classroom feedback by coaches, based on videos of teaching and frequent monitoring. In the most submitted by the teachers. Two successful online TPD successful programs, teachers receive programs in the United States, one at a pre-primary follow-up support at least 2-3 times. and one at a secondary level, included an in-person workshop-based training at the start of the year, and 3. Link professional development to also required that the participating teachers upload career incentives. Teachers will be videos of themselves teaching twice a month (Allen more motivated to participate in teacher et al., 2011; Powel et al, 2010). A program in Brazil professional development programs, conducted classroom observations once at the start and use it as an opportunity to improve of the program, provided targeted feedback on the themselves, if it is linked to their career quality of teaching in the school, and then gave ongoing progression. virtual support to the school leadership on how to 4. improve the teaching practices in their school. The Subject specific focus. The research study found modest improvements in learning. Another found that programs with a specific subject study in South Africa experimentally compared in- focus result in higher learning gains person with virtual coaching and found that after three than programs with general knowledge. years the virtual coaching program did not improve Moreover, training that is not focused on student learning, whereas the in-person program did a given academic subject (i.e., counseling) (Cilliers et al, 2021). Both the in-person and virtual is associated with a 0.2 lower standard coaching programs included an in-person workshop deviation in student learning. at the beginning of the year, but in the virtual program teachers were not required to upload video recordings, One modality of providing TPD that exemplifies the so they did not receive targeted feedback. '' above principles is the use of coaches, who visit schools on a regular basis and provide targeted feedback. Rigorous evidence from randomized The research found that control trials have shown that this model can work in programs with a specific subject focus result in higher different contexts in both developed (Kraft et al, 2018) and developing countries (e.g., Piper, 2016; Eble et al, 2020; Kerwin and Thornton, 2020; Cilliers et al, 2020). learning gains than programs with general knowledge. '' The Australian-funded Innovation for Indonesia’s School Children program, (INOVASI), for example, has The digital future of teacher training in Indonesia: What’s Next? 07 3 Indonesian context The Indonesian education system is characterized professional education program (PPG) in order to by low levels of student learning, similar to many receive certification, which makes them eligible for middle-income countries. A key constraint to a higher salary. However, under the Teacher Law improving student learning outcomes is the quality 14/2005, this is not applicable for contract and other of teachers. According to multiple sources, many non-permanent teachers. There are some differences Indonesian teachers lack the subject knowledge and in the financing and management of schools under pedagogic capacity to support student learning at MoRA and MoECRT. MoECRT schools are managed the highest level. In a recent survey of 350 schools by the local government (district and provincial), and across 12 districts in Indonesia (Yarrow et al., 2020), they receive an operational school grant transferred only three out of 10 Grade Four teachers were able directly to schools’ bank accounts. Meanwhile to reach a minimum score of 80 in a set of Math religious education is centralized and managed by questions used for Grade Four students. Classroom central MoRA through local MoRA office in each observations reveal that teachers also rarely pose province/district. strategic questions that require complex responses from students (Ragatz, 2015). With the imposition of Through MoECRT and MoRA, the Government of distance learning during the pandemic, teacher’s roles Indonesia has started to adopt online training as a are still central, now mediated and supported through part of its teacher upgrading strategy. The pandemic technology. has made an online approach the only one that might be used in the near- to medium-term. Between March There are more than 3.8 million teachers in 2020 to March 2021, MoECRT and MoRA offered at Indonesia. Two separate government entities are 4 least ten online training courses for teachers. These responsible for public education service delivery. included the launch of Guru Belajar platform and The Ministry of Education, Culture, Research and webinar series, initially designed to support teachers Technology (MoECRT) oversees the civil servant, during the pandemic, to pre-service and in-service permanent, and non-permanent teachers from teacher training or the teacher’s professional private and public schools, and the Ministry of education (PPG) program which is now fully online. Religious Affairs (MoRA) is responsible for teaching in In addition to government programs, some private formal and informal religious schools following the companies are also providing online training covering same national curriculum as MoECRT. There were 2.9 topics from subject-specific pedagogy to knowledge million MoECRT and 929,511 MoRA teachers in the of technology in classroom learning and pedagogy. country in 2021. MoECRT and MoRA teachers are 5 The strategic challenge for teachers and school subject to different contracts and exposed to different and system leaders is that despite the existence of types of training. For example, civil servants and various online options, little is known about their permanent teachers need to complete the teacher’s effectiveness in increasing student learning. 4 As of academic year 2020/2021 5 https://referensi.data.kemdikbud.go.id/dashboardgtk/ptk_dash2.php?id=20 and http://emispendis.kemenag.go.id/dashboard/?smt=20201, accessed 25 August, 2021 08 The digital future of teacher training in Indonesia: What’s Next? 3.1 Teacher training in Indonesia Indonesia’s National Education Standards6 '' require all teachers to undergo teacher training Indonesia’s National to improve their competency. MoECRT regulation No. 16/2007 on the National Standard of Teacher’s Academic Qualification and Competence covers Education Standards require all teachers to pedagogy, personality, social, and professional attributes. Online training programs follow this regulatory framework. Multiple entities offer undergo teacher training to '' training for teachers, including central government (directly managed/organized by MoECRT and improve their competency. MoRA), local government (through district education offices and MoRA regional offices), teacher associations, teacher working groups, NGOs and development partners, as well as private entities. Some MoECRT and MoRA trainings are implemented with partner universities, such as MoECRT’s PPG program, but the design and structure of the PPG program is determined at the central level. The program is offered for all teachers nationwide and has specific requirements and a long waiting list. Meanwhile, some local governments allocate funds for teacher development programs and organize training for teachers at their own initiative. Training program designs vary across these providers and training types. Private providers can legally offer training for teachers if they obtain a license from the ministry as non-formal education institutions and training institutes. Once teachers complete the training from these providers and receive the training certificates, they can register the training to obtain credit towards their professional development (conversion of hours of training to credit are regulated in Minister of State Apparatus Empowerment and Bureaucratic Reform regulation No. 16/2009). 6 Indonesia Law No. 20/2003 on National Education System, Government Regulation No. 57/2021 (second revision of Government Regulation No. 19/2005) on National Education Standards, and Regulation of Minister of State Apparatus Empowerment and Bureaucratic Reform No. 16/2009. The digital future of teacher training in Indonesia: What’s Next? 09 3.2 Online Training Provider Mapping Between September 2020 and January 2021, we development programs in 14 countries using a similar conducted a detailed mapping of the largest instrument to ours and find that these programs providers of online teacher training in Indonesia. “differ sharply from those ...that evidence suggests are This includes data from four public and four private effective”. providers, including 25 training programs that cover a total of 1,466 hours of content (see Annex 1) 7. These While many of the shorter programs do not programs have delivered training for 273,142 teachers incorporate many of the promising practices discussed as of February 8, 2021, including some teachers who above, longer programs, such as those offered by may have completed more than one training course. Ruang Guru and Websis, incorporate pre-assessment The team reviewed these programs using indicators for teacher competencies, teaching practices, from In-service Teacher Training Survey Instrument individual and group projects, and coaching for (Popova et al, 2018) 8 , and then conducted follow-up teachers. Several other providers plan to incorporate discussions to confirm the results and, in some cases, these elements in their training programs. obtained access to the training platforms/webs/apps to review the modules. We find that programs vary substantially in scope, and that there are no specific guidelines from either MoECRT or MoRA on the content, duration or '' The majority focus on teachers’ digital literacy skills (20/25 programs) other aspects of the training. One program provides 306 hours of training, spread over a nine-month period, but the majority of programs are very short: 19 of the 25 programs reviewed last no more than a and remote learning month, and 10 of these are a week-long or shorter. (21/25), while fewer The training programs cover a range of different include subject-specific '' topics, including student assessment, classroom management, counseling, and use of technology pedagogy and distant learning. The majority focus on teachers’ digital literacy skills (20/25 programs) and remote learning (21/25), while fewer include subject-specific pedagogy (see Annex 1). In terms of activities, almost all consisted of lectures, but fewer included individual coaching (12/27), and none provide opportunities for personalized learning.9 This is consistent with what is found in other developing countries: Popova et al (2019) categorize at-scale teacher professional 7 There are more online teacher training programs available offered by other government institutions, development partners (INOVASI, Tanoto Foundation, UNICEF, etc.), and private entities. Selection of these four private entities and the government training programs are made solely for the purpose of identification of potential partner for a follow up impact evaluation study. The Directorate of Teachers under the Directorate General of Islamic Education of MoRA and MoRA Research, Development, Education and Training Agency (Balitbang Diklat) were also consulted during the study, but their training programs are not included in the mapping due to lack of available detail at the time of data collection. 8 The World Bank developed an In-service Teacher Training Survey Instrument which asks 70 questions about (1) overarching program aspects, (2) program content, (3) delivery mechanisms, and (4) perceptions of program effectiveness by program implementers. 9 One helpful definition of “personalized learning” is instruction that is varied in pace, method, objectives, and content for each learner (teacher in this case) and is tailored to the student’s interests and preferences (Center on Innovations in Learning, 2013). 10 The digital future of teacher training in Indonesia: What’s Next? 4 Methodology This section discusses our sampling strategy, the construction of survey weights, and a description of the survey. 4.1 4.2 Sample Sampling strategy We created a nationally representative We conducted two stages of random sampling: first sampling sample of 435 MoECRT and MoRA 30 sub-districts and then 385 teachers in these sub-districts. primary and junior secondary First, we randomly selected one sub-district in each of the 30 teachers, spread across 30 provinces main provinces in the country (from all sub-districts in Indonesia, in Indonesia. The sample covered all based on MoHA regulation No. 72/2019).10 We then asked MoRA primary and junior secondary teachers and MoECRT to share the names of all of the primary and lower – including those with experience in secondary teachers in those sub-districts, using their own database attending online training programs of teachers (the DAPODIK 2020 and SIMPATIKA 2020 databases and those without any experience. This respectively). Second, using these lists we drew a random sample was done to enable examination of the of 310 MoECRT and 75 MoRA teachers in these sub-districts, level of teacher participation in online stratifying by province. The breakdown of MoRA versus MoECRT training programs, as there is very little primary and junior secondary teachers in our sample (19.5 percent published information on questions of MoRA) is roughly equal to the breakdown of the country as a teacher participation in training during whole (20 percent MoRA).11 We also sampled a list of potential the pandemic in Indonesia. replacement teachers, randomizing the order in which these teachers should be surveyed. All the teachers shared by MoRA and MoECRT were included as potential replacement teachers. The second stage of the sampling was slightly complicated by the fact that we did not receive a complete list of all the MoECRT teachers in these sub-districts due to privacy concerns: they shared a list of a (randomly selected) sub-set of teachers in these sub-districts. It is for this reason that we draw the two samples separately: one sample from the full list of MoRA teachers in the selected sub-districts; and one sample of randomly selected MoECRT teachers in our selected sub-districts. Nonetheless, due to random sampling this sample is still representative at a national level, with application of appropriate weighting. 10 https://kemendagri.go.id/page/read/48/peraturan-menteri-dalam-negeri-no72-tahun-2019 11 As of 2020, there were 2,099,317 MoECRT and 525,035 MoRA primary and lower secondary teachers. The digital future of teacher training in Indonesia: What’s Next? 11 Initially, a total of 385 teachers were Table 1 - Summary statistics selected through a randomization process under the assumption that we would see a high proportion of teachers Variable Mean Std. Dev. with online training experience, given the COVID-19 pandemic moratorium on Urban 0.453 0.498 face-to-face training. However, after four Female 0.66 0.474 days of data collection, the team identified that the proportion of teachers without any Year experience 10.083 7.384 experience in online training was close to 50 percent. Based on this information, the team Primary (MoECRT) 0.538 0.499 randomly selected an additional 35 MoECRT and 15 MoRA teachers from the lists of Primary (MoRA) 0.094 0.293 replacement teachers developed previously. When adding the additional 50 teachers, we Junior High (MoECRT) 0.255 0.436 over-sampled two provinces (Jawa Tengah and Sulawesi) where the research team Junior High (MoRA) 0.113 0.317 intended to conduct an impact evaluation. In our analysis we apply appropriate weights Age to make our results representative of the population of teachers as a whole. The <26 years 0.039 0.194 team had completed a total of 435 unique interviews by the end of the survey. 26-30 0.156 0.364 Table 1 provides summary statistics of 31-35 0.198 0.399 the sample. The majority of the teachers surveyed are female (66 percent) and come 36-40 0.172 0.378 from rural areas (55 percent). This is close to the national average according to MoECRT 41-45 0.117 0.322 statistics 2020/2021 and MoRA’s Education Management Information System 2019/2020 46-50 0.106 0.308 (female 64 percent). The distribution of teacher types (MoRA versus MoECRT and >50 0.211 0.409 primary versus junior secondary) is also Classroom teacher 0.48 0.5 roughly proportional to the national average. Fewer than half the teachers currently have a Teach Mathematics 0.085 0.279 teaching certificate. '' Teach Indonesian 0.032 0.177 The majority of the teachers Teach Religion 0.048 0.215 surveyed are female (66 Certified percent) and come from 0.416 0.493 rural areas (55 percent) '' Notes: N=435 12 The digital future of teacher training in Indonesia: What’s Next? 4.3 4.4 Survey protocol and Survey survey response rates weights The team followed a pre-defined protocol in conducting In order to ensure that our results are representative the phone survey, including replacement procedures at a national level, in our analysis we weigh each for teachers who could not be contacted or refused to observation by the inverse probability of being sampled. be interviewed. A total of 718 teachers were called, with This probability is calculated as the product of: (i) the a success rate of 60.6 percent. The main reasons for probability that a sub-district was selected in a given sample replacement were: province, and (ii) the probability that a teacher —either a MoRA or a MoECRT teacher— was selected in a given 1. Registered phone number no longer active (31 sub-district. This is merely the fraction of sub-districts percent) that we selected in a province, times by the fraction of 2. Teacher was not the target teacher (24 percent) teachers that we surveyed in that sub-district. Formally: 3. The phone is active, but no-one picked up (13 1 nt,d percent) X 4. Teacher refused to be interviewed (9 percent) The response rates were higher for MoRA teachers (72 percent), compared to MoECRT teachers (58 percent). Np Nt,d The lower response rate from MoECRT teachers was likely linked to using an outdated list of teachers where Np is the total number of sub-districts in and phone numbers. Response rates also varied province, p;Nt, d is the total number of teachers of type, substantially by provinces: provinces with higher t ∈ (MoRA,MoECRT) in sub-district d; and n t, d is the total participation rates in online training also had higher number of teachers of type t that we actually surveyed response rates. in that sub-district. 4.5 Description of the survey The survey was divided into three sections. In the first section the team validated teacher information received from MoECRT and MoRA and collected more information for teacher and school profiling. In the second section, the team assessed whether teachers have any experience in participating in online training programs. The interview for those without any experience stopped here. In the third part, we assessed the teachers’ opinions of the online training program based on the training they had completed in the previous four years. Lastly, we asked how teachers accessed the online training program. A complete list of the questions is available in the annex for reference. The digital future of teacher training in Indonesia: What’s Next? 13 5 Results Participation in online training during the Young teachers in urban areas are more likely to pandemic was very high (Figure 2). 46 percent of participate than older teachers and those in rural teachers had participated in online training at areas (Figure 3 a,b). For example, over three quarters the time of the survey. Of all teachers surveyed, of teachers aged 26 or below participated in online 44 percent participated in online training during the training, compared to only 37 percent of teachers aged pandemic. Of teachers who participated in online 50 or above. Similarly, 64 percent of teachers in urban training during the pandemic, 75 percent participated areas participated, compared to 40 percent in rural in online training for the first time during the pandemic. areas. MoECRT and MoRA both offered online training According to the national databases of teachers, there to all teachers in the period (March 2020-March 2021).13 are roughly 2.62 million primary and junior secondary The variation in uptake by age and location is thus teachers in Indonesia. Our findings suggest that 1.15 likely a reflection of comfort and access to technology million of these teachers received some kind of online and internet connectivity, discussed below. Table A.1 training during the pandemic. 12 performs statistical significance tests for comparing the mean characteristics between teachers who participated in online training with those that did not. Figure 2 - Teacher participation in online There are three statistically significant determinants for training participation: age, experience and rural versus urban status. None of the other variables, such as teacher gender, type, and level of certification, are different at statistically significant level. 46% 44% Percentage of Teachers 33% Yes, past four years Yes, during pandemic Yes, first time during pandemic Notes: N=435. Sampling weights applied. 12 The current number is likely higher, since our data were collected in February-March 2021, and both the pandemic, and online training, have continued since then. 13 The impact of COVID-19 on Indonesian education: Responses, requirements, and policy needs for returning to school (World Bank, forthcoming) 14 The digital future of teacher training in Indonesia: What’s Next? Figure 3 - The proportion of teachers who Figure 4 - Perceptions of teachers who did not participated in online training, by age and participate in online training location. (a) Main hindrance (a) by age Access (internet) Not chosen Access (technology) Other Did not receive information (b) by location (b) Support Required Notes: N=239. Sampling weights are applied. Access to the internet and technology were the largest constraints to participation (Figure 4a). Of those that did not participate, 33 percent indicated that access to technology was the largest constraint, and 30 percent indicated that they did not know about the availability of online training. In terms of support that they needed the most, 44 percent asked for technical assistance, in terms of devices and internet quota, 34 percent asked for more information about the training, More transparent information Bigger training quota and nine percent asked for more expanded availability Technical assistance, in the form of gadgets or quota subsidies to allow more teachers to participate in online training Other incentives such as those for offline training programs. Those who reported that their main hindrance NONE was access to technology, or the internet were also far Other more likely to require technical assistance, relative to those that did not (68 versus 25 percent, respectively). Notes: N=194. Sampling weights applied. The digital future of teacher training in Indonesia: What’s Next? 15 Online training sessions were usually provided by 5b. Who paid? government agencies and were most commonly free (Figure 5). Of the teachers who had participated in online training at least once the past four years, 73 percent of MoECRT teachers had received training organized by MoECRT, and 66 percent of MoRA teachers had received training organized by MoRA. MoECRT teachers were far more likely to have participated in training organized by their local government (department of education) at 47 percent for MoECRT teachers versus 13 percent for MoRA teachers. This is likely related to the governance of teachers in Indonesia mentioned previously, as local governments do not oversee MoRA schools. In the last four years, 90 percent of teachers had participated in a free training program, while 16 percent of teachers had paid for training themselves.14 MoECRT teachers are more likely to have paid for training, compared to MoRA teachers (see Table A.2 for tests of statistical significance). Figure 5 - Who organized and who paid for training, by teacher type. 5a. Who organized? MoECRT Teacher MoRA Teacher RT Notes: N=239. Sampling weights applied. Training sessions focused mostly on coping with distance learning (Figure 6). 91 percent of the teachers who participated in training in the last four years had received online training on “use of media and technology”, whereas only 63 percent had received training covering “subject-specific pedagogy”. This breakdown of topic coverage seems to correspond with teacher demand; many teachers wanted and needed to understand how to effectively use technologies to support learning at a distance during the pandemic. It is therefore not surprising that the topics teachers reported most wanting to see in future online training sessions were “learning and technology” (70 percent), “pedagogical knowledge” (32 percent), “lesson plans” MoECRT Teacher MoRA Teacher (eight percent), and “curriculum” (seven percent). 14 These numbers are not exclusive and so do not add to 100 percent; 16 percent of these teachers paid for the training, but they might also have participated in a separate, different free training, since the scope of the question is ‘out of the training they participated in over the last four years.’ 16 The digital future of teacher training in Indonesia: What’s Next? Figure 6 - Topics covered during training Figure 7 - Content of online training Online seminars Downloadable materials Quizzes and assessment Coaching/mentoring Practice teaching Use of media and technology Notes: N=239. Sampling weights are applied. Classroom management Most teachers (80 percent) completed training, Assessment and Evaluation but this fraction was substantially lower for MoRA Learning innovation (STEM) teachers (56 percent) and rural teachers (74 percent) Subject-specific pedagogy (Figure 8). Note that for the MoECRT teachers, training Notes: N=239. Sampling weights are applied. Answers was often tied to certification and promotion, so they are not mutually exclusive since teachers could have participated in multiple training programs. might have had larger incentives to complete it. The most common reason given for not completing the Training sessions tended to be short, most often training was lack of access to the internet. In addition, the in the format of an online seminar, and very completion rate also depends on teachers’ motivation few involved coaching/mentoring or provided for doing the online training. The completion rate is opportunities for teachers to practice what they much higher for those who did it for certification (92.1 learned in class (Figure 7). The median lengths of the percent) or promotion (97.9 percent), compared to those shortest and longest training were one and three days, who reported doing it because it was requested by the respectively. For those teachers who had participated principal (79.6 percent). Table 2 shows results from a in online training in the previous four years, 97 percent multivariate regression, where the dependent variable of teachers had received online training that used an is a dummy variable equal to one if a teacher completed online seminar, but only 47 percent of teachers had the training. Teachers were more likely to complete the training that encouraged teachers to apply what they training if they were MoECRT teachers, living in urban learned, and only 48 percent of teachers had training areas, or if they were required to participate in training for that included a coaching or mentoring component. promotion. There are no differences by teacher gender. The digital future of teacher training in Indonesia: What’s Next? 17 Figure 8 - Profile of teachers who Most teachers faced challenges in implementing completed training what was learned during the training (Figure 9). Only 22 percent indicated that they could implement 8a. MoECRT versus MoRA everything that they learned, 23 of teachers said they had problems but can solve the problem themselves, and another 51 percent will need help to implement what was taught. This means that the impact of the training on teacher practices and student learning is likely to be less than expected because teachers might not be able to implement what they learned and did not receive the support they reported needing as a follow up. 8b. Rural versus urban Figure 9 - Problems in Implementing what was learned during the training Table 2 - Predictors for completing the training Variable Completed training? MoRA Teacher -0.134** (0.0518 Urban Teacher 0.131*** (0.0408) Reasons for participation Enrich knowledge 0.663*** (0.0686) Yes, but can solve independently Yes, and need help Requested by principal 0.0553 (0.0762) No problems Never implemented Required for promotion 0.215*** (0.0762) Despite the implementation challenges reported Required for certification 0.1000 (0.0755) by participants, teachers mostly found the online training sessions useful, and would like Observations 435 to continue receiving online training (Figure 10). R-squared 0.712 Of those who participated in an online training, 89 percent would like to continue receiving it, 83 percent Notes: Results from OLS multivariate regression. Dependent variable is a found it useful to improve their knowledge. A large dummy variable equal to one if the teacher completed the training. Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. Each percentage (68 percent) found it useful to improve observation is weighted by the inverse of the probability of being sampled. their skills. 18 The digital future of teacher training in Indonesia: What’s Next? Figure 10. Perceptions of online training 5.1 Study Limitations A shortcoming of this study is the potential for selection bias due to survey non-response; “Given this assumption however online participation remains high and applying the same survey weights, we even if one assumes that none of the non- respondents participated in online training. conclude that at least We were only able to reach 60.6 percent of the teachers we called. As noted above, outdated 31 percent of primary teacher lists (24 percent) and phone numbers (3 percent) led to non-responses, and it is unclear whether these teachers are less likely to have participated in online training. But it is a possibility and lower secondary that the 15 percent of non-respondents who never picked up the phone, or whose phone was teachers in Indonesia always off, are also less likely to have participated in online training (if they live in remote locations, – roughly 825,000 for example). We can place a lower bound by teachers – received online training during conservatively assuming that none of the non- respondents (i.e. every single individual who we the pandemic.” could not reach) participated in online training during the pandemic. Given this assumption and applying the same survey weights, we conclude that at least 31 percent of primary and lower secondary teachers in Indonesia – roughly 825,000 teachers – received online training during the pandemic. The digital future of teacher training in Indonesia: What’s Next? 19 6 Conclusion and Policy Recommendations We believe there is good These results show that there is great demand from teachers to participate in online training, and there is much to be done news and bad news from to enable this. Between 825,000 to 1.15 million primary and lower the results of this survey. secondary teachers participated in online training in the last four years across Indonesia, and most of them attended the training during the pandemic. A lot of the demand by teachers is for support and instruction in how to navigate in the new world of providing online instruction to students. However, it is clear that more can be done to provide teachers with sufficient support and motivation to complete training and apply what was learned to their teaching. Completion rates were high overall but were lower in rural areas and when training completion was not tied to certification or promotion. Access to technology is a barrier, and this can be improved upon as well. More can also be done to keep teachers motivated and engaged to complete the programs. Unfortunately, based on current evidence, only a few of the programs examined by this study had included elements necessary for enabling teachers to improve their teaching skills. As we saw in Section 2, the majority of training captured by our survey of the last four years was short in duration, and mostly delivered through online seminars or downloadable materials. Few training programs were designed to encourage teachers to practice the new techniques or provide coaching support and feedback to improve on their techniques. A short webinar might be effective at imparting basic knowledge on how to use online platforms such as Zoom, for example, but is not likely improve teaching skills and practices or promote the achievement of outcomes such as improved student learning. Some training programs serve the narrow purpose of simply imparting knowledge (e.g. how to use a new technology), but most have greater objectives of improving teacher practices (e.g. more child-centered teaching techniques). We provide policy recommendations at both a programmatic and at a systems level. 20 The digital future of teacher training in Indonesia: What’s Next? 1 Program level Based on available global evidence, training programs that aim to improve teaching skills are likely to be more effective if they integrate teaching practices in each training program and provide follow- up support for teachers. The challenge is that this all needs to take place virtually, particularly during a pandemic. A program could institute regular virtual follow-up meetings between the trainers and teachers and require teachers to record their teaching and share this with the trainer prior to meeting. This could be done in collaboration with private Education Technology (EdTech) firms to make affordable and easy to use content and technology widely available to improve the quality of digital learning. There is also a need to provide a greater variety of training topics. While the current technology- focused training is relevant to the pandemic context to help teachers learn how to manage distance learning, we expect that teachers will seek a variety of topics to improve their skills and knowledge specially to help students recover post-pandemic. This includes topics such as subject-specific pedagogy, counselling, and teaching at the-right level (for a detailed discussion, see The Struggle Against COVID-19 in Indonesian Education: Responses, Requirements, and Policy Needs for Learning Recovery, World Bank 2021). Other options include trainings to reduce gender bias, improve retention and performance of male students, inclusive approaches for students with disabilities, integrate climate change across the curriculum and other areas of potential interest and need (Afkar et al., 2020). There is also potential to rely more on online training even after the schools have opened, since many teachers have overcome initial barriers to participating in online training. The available evidence suggests (see section 2) that it is best to follow a blended model, which also allows some face-to-face contact especially at the start of the program, and also in-person classroom observations.15 One possible option is to integrate the school supervisor's role or another external coach in the training program. Teachers who participated in government training programs such as PPG, for example, can be selected to be observed and provided with feedback a few months after the training to support implementation of targeted skills. This can also be supported through the existing teacher working group model, which functions as peer support for teachers (INOVASI, 2017). Some existing training programs such as the PembaTIK (ICT-based learning) program by MoECRT, provide teachers with the opportunity to be ‘Teacher Ambassadors’ and share what they learned during training. Teachers can discuss real problems they faced in implementing the training among themselves and might find more relevant solutions through this exchange (Agus Rachman, Policy Analyst, MoECRT, personal communication, January 14, 2021). 15 Tanoto foundation has promoted this approach with its E-Pintar (E-smart) program https://youtu.be/_hKoJFU5tok Existing community-based professional development approaches and peer learning through Teacher Working Groups (Kelompok Kerja Guru) can also be supported. The digital future of teacher training in Indonesia: What’s Next? 21 2 Systems level Building on earlier recommendations, there are several key policy areas and actions the Government of Indonesia can take to support online teacher training and student learning improvement. 1. Collect and distribute information about online programs. After the start of the COVID-19 pandemic, there are increasing numbers of online training programs for teachers, but this study found that some teachers still have issues in accessing information about available programs. The government could start collecting information about content, length, modalities, and cost of all programs, perhaps by requiring periodic information submission from all training providers, both government and also private entities. This information then could be distributed through ministry, local government, as well as school leader and teacher association communication channels. This information can help identify gaps in supply and help teachers access and select the training programs most appropriate to their needs. 2. Set standards and regulate for quality. Currently, private providers are required to have an operating license, but this is more of an administrative process rather than accreditation of online training (Bhardwaj et al., 2020). It is important not to stifle the growing online teacher training industry while guiding it towards high quality offerings that meet teachers’ needs. One way of doing this is by providing information on different modalities of the training, the related cost and effectiveness based on available evidence. Some of these roles can be played by government, by civil society groups, academic institutions or consortia of these actors, depending on capacity and in partnership with the private sector. The most basic function of government regulation both of its own products and private sector initiatives is consumer protection, which necessarily includes a strong focus on data security, discussed below. At a more advanced level, government could encourage customization, one of the promises of EdTech, and support individuation and contextualization of learning for teacher skill levels, cultural context and the student challenges they seek to address. 3. Measure the impact and cost-effectiveness of online teacher training programs. Rigorous reviews of online teacher training have found that many programs offered in other countries have no measurable effect on teacher practices in the classroom or student learning outcomes (Loyalka et al., 2019). Though challenging, it is therefore essential to accurately measure the impact of training on teacher practices and student learning outcomes to justify the expense of these programs and to guide teachers and providers to the most effective approaches. The EdTech community, including EdTech firms, NGOs, academia and government, can work together to establish clear, functional standards for performance against which online teacher training products can be transparently rated and compared. These actors, including philanthropic capital and international partners, can invest in transparently and rigorously evaluating teacher training programs. 22 The digital future of teacher training in Indonesia: What’s Next? 4. Indonesia’s digital infrastructure and connectivity need to be upgraded, for schools, teachers and students. The difficulties both children and teachers faced when they had to access the internet for home-based learning during the school closures of the pandemic demonstrates the need for comprehensive approaches to expanding connectivity and connection speeds. The government can expand its current efforts to include a broader range of partners, including those from the private sector, to provide internet access and technology devices in underdeveloped areas and for underserved communities. This solution could also be applied on the island of Java itself, where connectivity and quality of digital infrastructure are uneven, despite being stronger on average than in other locations. There are plans to connect schools beyond the reach of fiber-optic cables to the new satellite provided by BAKTI16, however, this is expected to only offer basic connectivity levels. 5. Ensure data security and make sure teachers understand it. Korea’s ICT Skills Standards for Teachers (ISST) establishes clear guidelines for managing personal data, and teachers are required by the Korea Education and Research Information Service to undergo regular training in cybersecurity (Yarrow et al., forthcoming). In Indonesia, the government could ensure clear and consistent regulations to improve data security, data use and data privacy for the education sector. Currently, the Ministry of Communication and Informatics (MoCIT), MoRA and MoECRT are involved in regulating the sector, and authority for cybersecurity is dispersed.17 However, the regulations on what student data can be collected, how they can be used, and how they must be safeguarded are not clear or consistent across the various authorities. A first step would be for relevant regulators to meet with private sector actors, teachers and other stakeholders to create a regulatory road-map for regulation that addresses issues of data use and security. Example sources for well-informed regulation include the US-based Consortium for School Networking (CoSN), among others. Overall, we find that EdTech has a role in supporting teacher and student learning18, but support and planning are required to assure success. At this point, the publicly released drafts of the RPJMN 2020–2024 do not contain a clear road map for integrating EdTech into Indonesia’s education system, though Making Indonesia 4.0 has some links to the education sector19. Other countries with more developed EdTech ecosystems have a clear vision and strategy for integrating EdTech, which can make the often private-sector led expansion more equitable20. These plans can mandate, for example, the inclusion of EdTech services and devices that are usable by students and teachers with disabilities (World Bank, 2022), and can require that public sector investments preferentially target lower-performing districts and provinces. While the changing world of work and education may be challenging for middle-income countries such as Indonesia to navigate, the next industrial revolution is already underway. If Indonesia can develop a clear, detailed and implementable plan for integrating EdTech into its education system, this can help insulate it from the negative effects of the fourth industrial revolution and make the distribution of the potential gains from EdTech more equitable across the country.   16 BAKTI is a non-echelon organizational unit within Ministry of Communication and Informatics. BAKTI serves as a provider of ICT infrastructure and ecosystem for the public, funded by Universal Service Obligation (USO) contribution from telco providers, in order to overcome the digital divide in Indonesia 17 The Ministry of Communications and Information has some authority over cybersecurity and provides a Cybersecurity Essentials training (https://digitalent.kominfo. go.id/pelatihan/detail/1048). The State Cyber and Coding Agency (Badan Siber dan Sandi Negara - BSSN) is responsible for the Cyber Security Strategy of Indonesia (https://bssn.go.id/strategi-keamanan-siber-nasional/) as per Presidential Regulation No. 53/2017 and 133/2017. BSSN also provides training including tips for teachers on how to deal with cyber bullying and cyber security. MoECRT’s Center for Communication and Information Technology (PUSTEKKOM) could support the inclusion of basic cyber security training in the required curriculum for teacher training (both in-service and pre-service). 18, 19 Online learning and ICT is mentioned on page 112 of the RPJMN, but who will do what to achieve which specific goals is not clear. The Making Indonesia 4.0 initiative was launched on April 4th, 2018 as the government’s response to the global phenomenon of the industrial revolution 4.0 and is a major project under RPJMN 2020-2024 (Presidential Decree No. 18/2020) 20 China’s current ten-year plan calls for 10-Mb broadband in all rural schools and 100-Mb broadband in all urban schools; the U.S. National Education Technology Plan was first published in 1996 and is updated every year to provide strategic guidance to state governments and school districts (Bhardwaj, et. Al, 2020) The digital future of teacher training in Indonesia: What’s Next? 23 References Afkar, R., Yarrow, N., Surbakti, S., and Cooper, R. (2020). Inclusion in Indonesia’s Education Sector: A Subnational Review of Gender Gaps and Children with Disabilities. World Bank Policy Research Working Paper. Allen, J., Pianta, R., Gregory, A., Mikami, A., and Lun, J. (2011). An interaction-based approach to enhancing secondary school instruction and student achievement. Science. 333(6045), pp. 1034–1037. Bhardwaj, R., Yarrow, N., Cali, M. (2020). EdTech in Indonesia: Ready for Take-off? World Bank Report. Bruns, Barbara, Costa, L., and Cunha, N. (2017). Through the looking glass: can classroom observation and coaching improve teacher performance in Brazil? The World Bank Working Papers. Butcher,N., Khairina, N., Nurlaili Kumala, C., and Loots, S. (2021). The Struggle Against COVID-19 in Indonesian Education : Responses, Requirements, and Policy Needs for Learning Recovery (English). Washington, D.C. : World Bank Group. Cilliers, J., Fleisch, B., Kotze, J., Mohohlwane, N., Taylor, S., and Thulare, T. (2020). Can virtual replace in-person coaching? Experimental evidence on teacher professional development and student learning in South Africa. RISE Working Paper Series. 20 (050). doi.org/10.35489/BSG-RISE-WP_2020/050 Cilliers, J., Fleisch, B., Prinsloo, C., and Taylor, S. (2019). How to improve teaching practice? An experimental comparison of centralized training and in-classroom coaching. Journal of Human Resources. pp. 0618–9538R1. Eble, A., Frost, C., Camara, A., Bouy, B., Bah, M., Sivaraman, M., Hsieh, P., Jayanty, C., Brady, T., Gawron, P., Vansteelandt, S., Boone, P., Elbourne, D. (2020). How much can we remedy very low learning levels in rural parts of low-income countries? Impact and generalizability of a multi-pronged para-teacher intervention from a cluster-randomized trial in The Gambia. Journal of Development Economics, 148. doi.org/10.1016/j.jdeveco.2020.102539 Guskey, T. R. (2002). Professional development and teacher change. Teachers and teaching, 8(3), pp. 381-391. INOVASI (2017). A Case Study of A Teachers’ Cluster Working Group in Moyo Hulu, Sumbawa. INOVASI Report. Retrieved from https://www.inovasi.or.id/wp-content/uploads/2018/01/KKG-Study-Report-05012018_final.pdf INOVASI (2020). INOVASI Phase 1 Achievements and Analysis: What Works to Improve Learning Outcomes in Indonesia Schools?, Retrieved from https://www.dfat.gov.au/sites/default/files/inovasi-phase-i-2016-2020-completion-report-june-2020.docx. Kerwin, J., and Thornton, R. (2020). Making the grade: The sensitivity of education program effectiveness to input choices and outcome measures. Review of Economics and Statistics, pp. 1–45. doi.org/10.1162/rest_a_00911 Kraft, M., Blazar, D., and Hogan, D. (2018). The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence. Review of educational research, 88(4), pp. 547–588. Loyalka, P., Popova, A., Li, G., and Shi, Z. (2019). Does teacher training actually work? Evidence from a large-scale randomized evaluation of a national teacher training program. American Economic Journal: Applied Economics 11(3). pp. 128-54. 24 The digital future of teacher training in Indonesia: What’s Next? Ministry of Education, Culture, Research and Technology. (2021). Priority Programs of Directorate General for Teachers and Education Personnel (GTK). Presentation for Basic Education Working Group Meeting: February 4th, 2021. Piper, B., Zuilkowski, S., Kwayumba, D., and Strigel, C. (2016). Does technology improve reading outcomes? Comparing the effectiveness and cost-effectiveness of ICT interventions for early grade reading in Kenya. International Journal of Educational Development, 49, pp. 204–214. Popova, A., Evans, D., and Arancibia, V. (2016). Training teachers on the job: What works and how to measure it. The World Bank Working Paper, 6. Powell, D., Diamond, K., Burchinal, M., and Koehler, M. (2010). Effects of an early literacy professional development intervention on head start teachers and children. Journal of educational psychology, 102(2), pp. 299. OECD. (2009). Chapter 3. The Professional Development of Teachers” in Creating Effective Teaching and Learning Environments: First result from TALIS. UNICEF (2021). Situational Analysis on Digital Learning in Indonesia. UNICEF Report World Bank. (2020). The Promise of Education in Indonesia. World Bank. World Bank. (2022). A Landscape Review of ICT for Disability-inclusive Education (English). World Bank Group. Yarrow, N., Afkar, R., Masood, E., and Gauthier, B. (2020). Measuring the Quality of MoRA’s Education Services. World Bank, Jakarta. The digital future of teacher training in Indonesia: What’s Next? 25 Annex 1: Provider Mapping Table A.1: Topics covered in public provider training programs at the time of data collection (November 2020) General Subject- Public Program Classroom Technology/ Distance Subject Assessment Counselling Pedagogy Specific Provider Name Management Software Learning Content Theory Pedagogy PGRI Webinar Series X X X X X X X Asynchronous Series X X X X X X X Program Id-Teachers MOOC X X X X X X English for Indonesia X X X X Teachers WAVES X X X X X X X ToT with Microsoft X X X Indonesia Digital X X X Learning PPGTK PPG X X X X X X X X Guru Belajar X X X X X X GPK X X X X Guru Penggerak (PGP) X X X X X X ECED teacher training X X X PPPPTK ETT X X X Matematika E-Pensa X X X Google Classroom X X X X X Training ICT in Mathematics X X X X X X X Webinar Pusdatin PembaTIK X X Balitbang N/A Table A.2: Topics covered by online private provider training at the time of data collection (November 2020) General Subject- Public Program Classroom Technology/ Distance Subject Assessment Counselling Pedagogy Specific Provider Name Management Software Learning Content Theory Pedagogy Sekolah.mu Sekolah.mu Courses X X X X X X X Wardah Inspiring X X X X X X Teacher Ruangguru Indonesia Teaching X X X X X X Fellowship Ruang Belajar X X X X X X Zenius Zenius untuk Guru X X X X X X Websis for Websis Academy X X X X X X Edu Kelas Websis X X X X X X 26 The digital future of teacher training in Indonesia: What’s Next? Table A.3: Training content at the time of data collection (November 2020) Public Provider Program Name Content PGRI Webinar Series Remote learning related topics Asynchronous Series Program classroom management, digital literacy, 21st century skills for teachers, and technology in classroom. Id-Teachers Massive Online Open Course Asynchronous courses, cover content knowledge topics (i.e. (MOOC) science, social science, mathematics) English for Indonesia Teachers English learning pedagogy WAVES Focus on the use of technology in classroom ToT with Microsoft Focus on the use of technology in classroom Indonesia Digital Learning Focus on the use of technology in classroom PPPGTK PPG General pedagogy, content knowledge, classroom management, teaching practice, mentoring program, and subject-specific pedagogy for all teaching subjects in all grades. Guru Belajar 21st century skills, technology in classroom, student assessment (AKM) GPK Special education Guru Penggerak (PGP) General pedagogy, social emotional learning, leadership, personal coaching, school development, community engagement, ambassador preparation, etc. ECED teacher training Stunting prevention, student assessment, parent engagement, and report writing PPPPTK ETT Internet for math education, GeoGebra, PowerPoint, Pre-test and Matematika Post-test E-Pensa Spatial reasoning introduction, math curriculum, spatial reasoning activities G-Suite for Education The use of google product for math education ICT in Mathematics MS Word, Google Classroom, e-learning, internet for math, remote learning, GeoGebra Webinar General pedagogy, remote learning Pusdatin PembaTIK The use of technology in classroom Balitbang N/A N/A Sekolah.mu Sekolah.mu Courses General pedagogy, subject-specific pedagogy (e.g. active learning in numeracy and literacy), subject content (e.g. Maths, Geography, Sociology), classroom management (e.g. student behavior, engaging remote learning), assessment and evaluation, integrating technology in classroom, and engaging community in learning. Wardah Inspiring Teacher Technology in classroom, remote learning, similar to PembaTIK Ruangguru Indonesia Teaching Fellowship Learning strategies, curriculum planning, education management (including assessment), digital learning, knowledge of diversity in the classroom, inclusive learning, and teacher's social competence. Free access to subject-content learning resources Zenius Zenius untuk Guru Technology in classroom, assessment and evaluation, pedagogical learning theory and distance learning. Websis for Edu Websis Academy Technology in classroom, general pedagogy and personalized training in year 2 Kelas Websis Technology skill for teachers and principals The digital future of teacher training in Indonesia: What’s Next? 27 Annex 2: Summary Statistics Table A.1: Teacher Characteristics by Exposure to Online Training (1) (2) (3) T-test Variable No training Received training Total Difference Mean/SE Mean/SE Mean/SE (1)-(2) Urban 0.153 0.324 0.231 -0.171*** [0.032] [0.048] [0.029] Female 0.588 0.636 0.610 -0.048 [0.051] [0.052] [0.037] Age: <26 years 0.010 0.038 0.023 -0.028* [0.006] [0.015] [0.007] Yrs experience 11.975 8.734 10.494 3.241*** [0.880] [0.659] [0.592] Primary (MoECRT) 0.572 0.574 0.573 -0.002 [0.050] [0.052] [0.036] Primary (MoRA) 0.105 0.127 0.115 -0.022 [0.029] [0.031] [0.021] Junior High 0.214 0.168 0.193 0.046 (MoECRT) [0.041] [0.032] [0.027] Junior High (MoRA) 0.108 0.131 0.119 -0.023 [0.030] [0.039] [0.024] Classroom teacher 0.498 0.555 0.524 -0.057 [0.051] [0.053] [0.037] Teach mathematics 0.049 0.079 0.063 -0.030 [0.016] [0.023] [0.014] Teach Indonesian 0.032 0.023 0.028 0.009 [0.022] [0.009] [0.012] Teach Religion 0.062 0.047 0.055 0.015 [0.026] [0.025] [0.018] Certified 0.347 0.418 0.379 -0.071 [0.047] [0.053] [0.036] N 194 241 435 Notes: the values displayed for t-tests are the differences in the means across the groups. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level. Standard errors are in square brackets. Each observation is weighted by the inverse probability of being sampled. 28 The digital future of teacher training in Indonesia: What’s Next? Table A.2: Reasons for attending, and who provided the training, who paid for the training, and who completed the training, by teacher type (1) (2) (3) T-test Variable MoECRT MoRA Total Difference Mean/SE Mean/SE Mean/SE (1)-(2) Panel A. Reasons for attending the training. 0.973 1.000 0.980 -0.027** Enrich knowledge [0.012] [0.000] [0.009] 0.509 0.724 0.565 -0.215** Requested by principal [0.064] [0.079] [0.053] 0.345 0.059 0.271 0.287*** Required for promotion [0.067] [0.030] [0.054] 0.209 0.158 0.196 0.051 Required for certification [0.054] [0.069] [0.044] 0.295 0.212 0.273 0.083 Knowledge (no requirements) [0.056] [0.070] [0.045] Panel B. Who organized the training? 0.727 0.171 0.583 0.556*** MoECRT [0.052] [0.083] [0.052] 0.040 0.663 0.201 -0.623*** MoRA [0.014] [0.107] [0.038] 0.490 0.133 0.398 0.357*** Local government [0.064] [0.062] [0.052] Working groups and teaching 0.391 0.249 0.355 0.142 professional institutions [0.063] [0.091] [0.052] Panel C. Who paid for the training? 0.188 0.069 0.158 0.119** Personal money [0.044] [0.035] [0.034] 0.884 0.948 0.901 -0.064* Free [0.028] [0.025] [0.021] 0.079 0.124 0.091 -0.045 School [0.035] [0.078] [0.033] Completed training? 0.890 0.562 0.806 0.328*** Completed training? [0.035] [0.108] [0.045] N 189 52 241 Notes: The value displayed for t-tests are the differences in the means across the groups. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.. Each observation is weighted by the inverse probability of being sampled The digital future of teacher training in Indonesia: What’s Next? 29 Annex 3: Survey Instruments 30 The digital future of teacher training in Indonesia: What’s Next? The digital future of teacher training in Indonesia: What’s Next? 31 32 The digital future of teacher training in Indonesia: What’s Next? The digital future of teacher training in Indonesia: What’s Next? 33 34 The digital future of teacher training in Indonesia: What’s Next?