Photo: MBSSE SIERRA LEONE’S RADIO TEACHING PROGRAMME: CHALLENGES AND OPPORTUNITIES July 2021 ©2021 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington, DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the information included in this work. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 4.0 IGO license (CC BY 4.0 IGO) https://creativecommons.org/ licenses/by/4.0/, with the following mandatory and binding addition: Any and all disputes arising under this License that cannot be settled amicably shall be submitted to mediation in accordance with the WIPO Mediation Rules in effect at the time the work was published. If the request for mediation is not resolved within forty-five (45) days of the request, either You or the Licensor may, pursuant to a notice of arbitration communicated by reasonable means to the other party refer the dispute to final and binding arbitration to be conducted in accordance with UNCITRAL Arbitration Rules as then in force. The arbitral tribunal shall consist of a sole arbitrator and the language of the proceedings shall be English unless otherwise agreed. The place of arbitration shall be where the Licensor has its headquarters. The arbitral proceedings shall be conducted remotely (e.g., via telephone conference or written submissions) whenever practicable, or held at the World Bank headquarters in Washington, DC. Attribution – Please cite the work as follows: Gutierrez M., and Wurie, M. (2021). Sierra Leone’s Radio Teaching Programme: Challenges and Opportunities. Washington, DC: The World Bank. Translations – If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations – If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Views and opinions expressed in the adaptation are the sole responsibility of the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content – The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to reuse a component of the work, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. All queries on rights and licenses should be addressed to Marcela Gutierrez, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; e-mail: mgutierrezb@worldbankgroup.org. Acknowledgements: This note was prepared by Marcela Gutierrez Bernal, Mariama Wurie and Mari Shojo. Overall guidance for the development was provided by Halil Dundar, Practice Manager for Education in West and Southern Africa at the World Bank. The note borrows heavily from the Education Radio Knowledge Pack produced by the Education Technology team of the World Bank, the EdTech Hub’s 2020 Radio: Rapid Evidence Review, and Ho and Hetal’s Tuned in to student success: Assessing the impact of interactive radio instruction for the hardest-to-reach. It is not meant to be exhaustive; instead, it is intended to provide a brief overview of the situation of the Radio Teaching Program and the existing literature and to propose recommendations to the Teaching Service Commission and the Ministry of Basic and Senior Secondary Education. Contents 1. The Radio Teaching Program (RTP) in Sierra Leone ............................................. 1 RTP as an emergency response ................................................................................. 1 RTP after schools reopen ............................................................................................ 3 Emerging evidence of the RTP during school closures ......................................... 5 Engagement and Use................................................................................................ 5 Quality ......................................................................................................................... 7 2. Use of radio for learning: International experiences ........................................9 Impact ....................................................................................................................... 16 Key drivers of effectiveness .................................................................................. 18 3. Recommendations.................................................................................................. 21 Radio as an after-school remedial education tool ........................................... 21 IRI or IAI to maximize the quality of teaching during school hours ............ 23 Radio for continuous professional development of teachers ....................... 24 Transversal areas: community ownership, data, and consistency............... 25 References and Bibliography ....................................................................................... 27 Figures Figure 1: RTP Coverage Heat Map Based on the Transmitters Available as of 2020 .................................................................................................................................... 4 Figure 2: Percent of children engaged in any educational activities since school closures .............................................................................................................................. 5 Figure 3: Percentage of children who listened to educational programs on the radio since school closures.............................................................................................6 Figure 4: Percentage of parents responding positively to questions about their children’s engagement with the RTP during the COVID-19 school closures ....... 8 Figure 5: Uses of education radio ................................................................................ 10 Figure 6: Percentage of observed teachers in the FIER program adopting IRI instructional methods................................................................................................... 18 Figure 7: Recommended uses of RTP for Sierra Leone in the short- and medium-term ................................................................................................................. 21 Boxes Box 1: Interactive Audio Instruction ........................................................................... 11 Box 2: Education Development Center (EDC)’s IRI in Zanzibar (2006) - Radio Instruction to Strengthen Education (RISE) ............................................................ 13 Box 3:Training Teachers with the Somali Interactive Radio Instruction Program (SIRIP) 2005 – 2011 ...................................................................................... 15 Acronyms CIMS Covid-19 Impact Monitoring Survey COVID-19 Coronavirus Disease CPD Continuous Professional Development EDC Education Development Center EVD Ebola Virus Disease GRID3 Geo-Referenced Infrastructure and Demographic Data for Development IAI Interactive Audio Instruction IRI Interactive Radio Instruction MBSSE Ministry of Basic and Senior Secondary Education RECOVR Research for Effective Covid-19 Response Survey RISE Radio Instruction to Strengthen Education RTP Radio Teaching Program SIEF Strategic Impact Evaluation Fund SIRIP Somali Interactive Radio Instruction Program STEM Science, Technology, Engineering, and Mathematics TPD Teacher Professional Development TSC Teaching Service Commission UNFPA United Nations Population Fund USD United States Dollar 1. The Radio Teaching Program (RTP) in Sierra Leone RTP as an emergency response When the Ebola virus disease (EVD) epidemic hit Sierra Leone in 2014, the government implemented the Radio Teaching Program (RTP) as a mechanism to continue learning during school closures. A country of only 6.2 million people, Sierra Leone had more than 14,000 cases and lost close to 4,000 lives to this deadly disease. Schools were closed for eight months. Given Sierra Leone’s low access to computers (2 percent) and televisions (14 percent) at home1, the government decided to respond be establish the RTP, which reached more than 1 million children. The RTP broadcast lessons five days a week, in 30-minute segments, based on the primary and secondary school curricula in core academic subjects. To increase interactivity, students had the option to call in with questions at the end of each lesson. Although a lack of cellphone signal and devices posed challenges to interactivity, the RTP maintained a link to learning for children during the crisis. The RTP was again used in 2020, as the coronavirus disease (COVID-19) pandemic closed school doors one more time. When COVID-19 hit, the Ministry of Basic and Senior Secondary Education (MBSSE) instructed that all schools should be shut down by March 31, 2020, a week before they were scheduled to close for the Easter holiday break. Just one week after schools closed, the ministry revived the RTP as a distance education alternative. The Teaching Service Commission (TSC) managed the pedagogical functions of the RTP, while the MBSSE Communications Unit managed its technical and logistical functions. Schools remained closed until early October 2020. To establish the pedagogy for the RTP during the COVID-19 pandemic, the TSC drew on lessons from the RTP under the EVD epidemic. The TSC recruited some of the most qualified and successful teachers who taught the RTP during the EVD epidemic to participate in the design and implementation of the incumbent RTP program. In a two- day long pedagogy development workshop, these teachers contributed to developing the curricula and lessons for the incumbent RTP: they planned sample lessons and developed scripts that were then reviewed and edited with TSC´s feedback. This process took place the month before the government aimed to roll out school closures, ensuring the RTP was ready to launch six days after schools were officially closed. The teachers recording the RTP were provided with transportation and government passes to go to 1 Sierra Leone Demographic and Health Survey 2013. 1 the MBSSE Communications Unit studio at various hours during the nationwide lockdown and curfew. They were also paid a stipend to teach the programs. The program included a wide variety of subjects. The experts in the workshop recommended that the RTP focus on the core subjects for basic and senior secondary education students, namely: mathematics, applied sciences, and English language arts, history and literature. However, the TSC decided to include other non-core subjects, such as music and French. To improve the quality of the RTP, the TSC held a refresher course for teachers who were delivering the RTP lessons. This training included a Life Skills Program specifically designed to decrease the negative effects of the pandemic over girls’ education. The training was organized with the support of development partners including Plan International, World Vision, and Save the Children. It included training in lesson delivery, script performance and gendered education. The gender-related lessons were contained in what was called the Life Skills Program and were developed by the United Nations Populations Fund (UNFPA), Plan International, and a government gendered-education specialist. The Life Skills Program was accompanied by a strong communications campaign targeting male members of society, households, and schools. The communications campaign encouraged males to allow school-age girls in their households, and themselves, to listen to the Life Skills Program. To increase reach, the RTP lessons were aired on the MBSSE Education Radio Station and 10 community radio stations. Using partnerships, the government also delivered 25,000 solar- and wind-powered radios to remote communities and vulnerable children, particularly girls. The RTP learning programs were aired daily from 10:00 AM to 10:00 PM on the MBSSE Education Radio Station, which has different frequencies per district. The education radio station uses nine government-owned transmitters across the country which, despite providing important coverage, do not reach all areas of the larger districts, such as Kailahun, Bo, Moyamba, and Kenema. To achieve greater coverage, the RTP partnered with 10 community radio stations to air various lessons for at least nine hours per day. This was done at a discounted cost to MBSSE of 250,000SLL (approx. US$25) down from 500,000 SLL (approx. US$50) per hour. Throughout the lockdown, this cost was paid by the Government with the support of various NGOs (including Leh Wi Lan and CSR). The radio coverage thus achieved was nevertheless lower than during the EVD school closure, when funds were allocated to contract with more than 40 community radio stations through the Independent Radio Network. To make classes interactive, RTP lessons included time for questions from listeners and from students in the recording studio. The RTP lessons were 1 hour each— 45 minutes of teaching and 15 minutes of questions and answers to facilitate interactivity and gain feedback from students and parents on how effective the lessons were. Listeners could also call RTP teachers on their personal phone numbers to ask 2 questions. For some subjects, including French, the lessons were designed to simulate a classroom experience by having two or three students in the recording studio while the teacher presented the lesson. At the end of the 45-minute lesson, the students in the studio would then ask questions that were built into the lesson, and the teacher would answer accordingly. In addition to using the RTP for students’ distance education, the TSC also used the RTP to prepare primary-school teachers for school reopening. The remote teacher- training program reached approximately 4,000 teachers who taught in impoverished schools. The program included radio lessons on how to teach numeracy and literacy, and manuals that teachers used to follow along with the RTP broadcasts. Since many of the target teachers did not have radios in their homes, some formed groups with other teachers so they could listen to a single radio together while each of them followed along in his or her own manual. The teacher training program had two dedicated phone lines in the studio, to which teachers could call in with queries about the lessons. As in the RTP for students, listeners also had the opportunity to call the personal phone numbers of RTP instructors to ask questions. RTP after schools reopen The government of Sierra Leone plans to continue broadcasting the RTP now that schools have reopened. This will give children the chance to learn both at school and at home and will make the system more resilient to future crises. Further, it will allow teachers to continue strengthening their knowledge and practices throughout the school year, regardless of location. The program also has the potential to strengthen community outlook and values around education and learning and to empower community stakeholders to contribute to the learning of children in their neighborhoods. However, to maximize the effectiveness of the continued RTP, the country needs to invest in transmitters, electricity, human resources, and pedagogy. • Transmitters: The MBSSE and the TSC plan to purchase and strategically position transmitters so that all the territory is covered through government or community radio broadcast. To do so, the government has partnered with Geo- Referenced Infrastructure and Demographic Data for Development (GRID3). GRID3 works with countries to generate, validate and use geospatial data on population, settlements, infrastructure, and boundaries. Together, MBSSE and GRID3, analyzed the proportion of school-aged population by school level within the transmission range of the RTP (see Figure 1). Seven of the nine government- owned transmitters operate at 500 watts. If the 500-watt transmitters were to be replaced with 1,000-watt transmitters, the nine transmitters could 3 potentially cover the majority of the nation (Kambia, Bonthe and Pujehun would still be excluded unless additional transmitters were placed in these three districts).2 • Electricity: The RTP faces challenges regarding electric power connectivity. The government-owned transmitters benefitted from power connectivity during the lockdown, which was supplied by the Sierratel Telecommunication Company as a result of an unwritten memorandum of understanding. However, as Sierratel shutdown a number of their service grids across the country due to internal cutbacks, the MBSSE RTP transmitters were among many to lose access to electricity. The MBSSE is currently researching solar power options for the transmitters. • Human resources: The RTP administrative team at the MBSSE Communications Unit is composed of stipend-compensated volunteers. The team is centralized in the capital, Freetown, and funding, training, resources, and district support is limited. Training, hiring and equipping the existing team with the necessary tools to execute the program would provide quality assurance of the program. Maintenance of the team also needs to be assured through funding as a recurring expenditure. Staff is needed to facilitate, maintain, and monitor the radio equipment and program transmission in rural districts outside of the Urban Western Area. • Pedagogy: The RTP was designed Figure 1: RTP Coverage Heat Map Based on as an emergency response program the Transmitters Available as of 2020 to reach children and teachers during school closures. With schools functioning in parallel, it is important to rethink the pedagogy of the RTP to identify how to best complement in-person classes to maximize student and teacher learning. This will be the focus of this document: to analyze potential changes in design and pedagogy that could increase the impact of the RTP when schools are open. Source: Werne and Ecker, 2020 2 Werne and Ecker, 2020 4 Emerging evidence of the RTP during school closures Learners’ access to Sierra Leone’s RTP and its impact on engagement with schooling, prevention of dropouts, and continuity of learning are yet to be fully understood. However, as with similar programs worldwide, the sparse emerging evidence reveals a somber picture. Engagement and Use Remote learning programs, and radio in particular, had modest levels of voluntary participation during the pandemic. In most countries in Africa the percentage of children engaging in any education activities (including both government- and nongovernment-led ones) was low (see Figure 2). The percentage of children that listened to educational programs on radio was even lower (see Figure 3). Studies evaluating a TV-based learning program implemented in Bangladesh during COVID-19 school closures found that although 86 percent of students were aware of the program, only half of those with access chose to participate in them (Biswas et al., 2020). Figure 2: Percent of children engaged in any educational activities since school closures Note: This map illustrates the percentages out of all households with school-age children who had attended school before the pandemic. Source: World Bank, 2020. 5 Figure 3: Percentage of children who listened to educational programs on the radio since school closures Note: This map illustrates the percentages out of all households with school-age children who had attended school before the pandemic. Source: World Bank, 2020. Although there is limited research on the RTP during the COVID-19 school closures in Sierra Leone, two surveys shed light on its access and use. The Sierra Leone COVID-19 Impact Monitoring Survey (CIMS) (2020-2021), led by the Statistics Sierra Leone and the World Bank, in partnership with UNICEF, covers a large and significantly representative sample, which considers urban-rural, male-female and basic to senior secondary school representation. Innovations for Poverty Action also carried out a smaller, more urban survey, known as the Research for Effective COVID-19 Response (RECOVR) survey. Although the RTP aimed to reach all basic and senior secondary school students during the COVID-19 pandemic school closures, less than half of the children used radio for educational purposes. According to the CIMS and the RECOVR surveys, most households thought that it was important that students engage in education during the COVID-19 school closures. However, the most popular means to do so was to have children review their schoolbooks, whereas radio use was low. According to CIMS, just over half of the children in surveyed households were engaged in any educational activities during school closures, and 41 percent listened to radio lessons. Children that were engaged in learning spent just a few hours a week on learning activities, an average of four hours per week on radio lessons, and seven hours per week on private lessons. The RECOVR survey found that children in 75-80 percent of surveyed households were spending time on education at home. However, out of those that were 6 engaged in distance learning, only 18 percent of primary students and 20 percent of secondary students used the radio and 85 percent used their own schoolbooks to learn. Furthermore, most respondents (75 percent) supported their children’s education primarily by telling them to review their books and talking to them about school. Lack of access to a radio does not seem to be the reason behind the low usage of the RTP in urban areas.3 When stating the barriers children faced when attempting to spend time on education at home, the three most reported barriers cited by respondents to the RECOVR survey were lack of access to television (25 percent for both primary and secondary students), a lack of motivation (22 percent for primary and 24 percent for secondary), and competing priorities (22 percent for primary and 19 percent for secondary). Only 5 percent of respondents stated that “no access to radio� was a barrier to spending time on education at home— one of the lowest reported barriers. The lack of engagement with the RTP in urban areas could be a result of limited knowledge about the RTP or a limited belief in radio as an effective tool for learning . In Freetown and its outskirts, TV and WhatsApp are anecdotally the most common sources of news, current affairs, and informal education/knowledge sharing for both adults and students. Cultural beliefs in traditional education methods (books and television programs) might be more widely accepted and easily accessed as the most practical methods for learning in these areas, depressing parents’ desire to make use of radio as a substitute for education for their children. Perhaps, if the radio broadcast reach were to cover a larger rural area it would yield better results, since a majority of rural children would be encouraged by parents to use it and would in fact use it, lacking TV access. The moderate use of radio found in the CIMS (41 percent) and the low use found in the RECOVR survey (18-21 percent) could thus be a result of the split between urban and rural attitudes toward using radio for education. Other reasons could include a lack of accompanying learning materials and assessments to accompany the RTP, discouraging the will to access and/or use the program. Quality To our knowledge, there are few publications that rigorously analyze the impact of the RTP implemented during COVID-19 school closures in Sierra Leone over student engagement with learning, student learning or teaching practices. The EdTech Hub is supporting a study of the impact of interactive audio content on student learning and teacher knowledge based on students learning through Rising Academies (a private school operator in Sierra Leone). Additionally, the Strategic Impact Evaluation Fund (SIEF) supported an evaluation of the effectiveness of sending SMS reminders to 3 61% of respondents to the RECOVR survey live in Freetown, 45% in main Freetown and 16% in the outskirts of Freetown. 7 parents to encourage their children to listen to the radio programs, as well as having teachers call children to go over what they learned on the radio. The SIEF evaluation found that although teacher phone calls increased student activity and parent engagement, they did not include time spent on learning (reported retrospectively) or test scores (Crawfurd, 2021). Parents surveyed under the CIMS believed that the radio lessons were clear and at the right level, but insufficient for learning (Figure 4). Specifically, over 80 percent of parent respondents stated that their children could understand the lessons. Just over 60 percent stated their children were listening to the lessons at the right level, and just over 20 percent stated that the lessons were sufficient for their child to learn. Figure 4: Percentage of parents responding positively to questions about their children’s engagement with the RTP during the COVID-19 school closures Percentage of parents reporting a particular observation of their child/children’s interaction with the RTP Source: World Bank Sierra Leone CIMS 2020 Radio Teaching Program Questionnaire. Conversely, the former head of the TSC, Staneala Beckley, received feedback that the lessons were too fast for many students and that students often did not listen to the correct lesson for their level. In a World Bank EduTech podcast on Leveraging Educational Radio for Continued Teaching and Learning during COVID-19: a Conversation with Sierra Leone Teaching Service Commission, Beckley shared that one of the most common types of feedback received by the RTP team was that the pace of the lessons was too fast for many students. The RTP team also observed that students who would be taking exams in a year’s time were listening to the lessons for those exams, although 8 these lessons were intended for the grade level above them.4 Beckley also stated that sometimes students were not listening to the correct lesson for their class-level and/or their learning level. Additionally, she remarked that under-trained teachers from impoverished areas also used the RTP lessons to learn for themselves—to both learn the lessons and learn how to teach their classes. 2. Use of radio for learning: International experiences There are two main types of education radio: classical radio instruction and interactive radio instruction (IRI). Classical radio instruction is typically a lecture- styled lesson, delivered by radio. It does not emphasize interactive learning and is best suited for older students and the adult population. This type of instruction has been used in Sierra Leone under the RTP.5 On the other hand, IRI programs combine radio broadcasts with activities, so as to maximize learning and improve the quality of teaching. IRI lessons encourage teachers and students to react verbally and physically (e.g., sing, play, move, dance, answer questions, demonstrate skills to classmates) to exercises posed within radio lessons and to participate in group work, experiments, and other activities. As documented by the Education Radio Knowledge Pack (World Bank, 2021), education radio can be used in a variety of ways (Figure 5). First, it has been used to deliver distance education during school closures in states of emergency (e.g., conflict & fragile settings, the EVD epidemic, the COVID-19 pandemic). This is the case for Liberia, which used IRI during the EVD epidemic by broadcasting lessons twice a day on 10 community radio stations. The radio program was complemented by English and math texts for Level 2 and 3. Mali also used IRI during armed conflict, targeting grades one through three, and aiming to support the delivery of the new curriculum and the coaching of teachers in reformed reading instruction. Many other countries, including Nepal, delivered new content via radio during the COVID-19 school closures (FCDO, 2020). As mentioned in the first section of this report, Sierra Leone also used radio to maintain the continuity of learning during the EVD and the COVID-19 pandemics. 4 Sierra Leone has a 6-year primary cycle, a 3-year Junior Secondary School cycle and a 3-year Senior Secondary School cycle. Each cycle ends in a national exam that allows students to move to the next educational level (i.e. NPSE, BECE and WASSCE respectively). The RTP designed the lesson plans for the class in the final year of each cycle to be centered on preparing students for the class’s respective exam. TSC observations found that students in classes that are one-level-down from the exam taking class were also listening to the lessons from the exam-taking class, or the class one-level-up from their class, in order to be extra prepared for these exams when the time arriveed (in one year). 5 Please refer to the following hyperlink to access sample lessons: https://n.pr/2UQ6dVo 9 Figure 5: Uses of education radio Delivering distance education in emergency settings Deploying public safety Facilitating teacher training and announcements and certification sharing info on government programs during crisis Supporting teacher shortages, including Providing distance multigrade classrooms & education for out-of- local language school children Supporting teachers inside Delivering new content or revising classrooms content after school during normal schooling cycle Source: Modified from the Educational Radio Knowledge Pack, World Bank 2020 During times of crisis, education radio has also been used to support health-based education and share information on government programs. Educational content over radio is usually complemented by public safety announcements and data on available relief programs during emergencies. For instance, in Liberia radio was used to share health-based best practices and to support caregivers and comm unities during the EVD crisis. Ebola safety content was also included in Liberia’s radio teacher training lessons (Christina and Richmond, 2020). Educational radio has also been instrumental in providing distance education for out- of-school children. When children cannot attend school due to poverty, distance to school, or legal restrictions, education radio programs have been used to provide instruction in the home or in informal learning centers. For example, when pregnant teens were not allowed to attend schools in Sierra Leone, NGOs funded an informal after-school learning program for pregnant girls. Radio programs were also created for them to support what they learned in the Community Learning Centers (Mason, 2016). In Chikuni, Zambia, 80 percent of children attend school by gathering around a radio and learning from radio programs led by a teacher. Although the program is run by an NGO in partnership with the ministry, it is not considered formal school but rather radio school. Given that the closest formal school is 50 kilometers away, these students cannot afford to go to school and must balance listening to guided radio sessions with 10 farm work on a daily basis. However, the radio lessons are sufficient for them to take national exams, in which they often score above-average (Kinder not hilfe). Similarly, IAI has been used to help isolated rural children access early childhood education in Malawi; to integrate child laborers into formal schools in Tanzania; and to support education in Qur’anic schools in Nigeria (World Bank Group and Education Development Center, 2015). Potentially, an after-school radio program could be used to complement formal schooling if it focuses on delivering new content or on revising content covered during class. To our knowledge, this has not been systematically tested, but it is a great opportunity for countries that have already made the investment in the infrastructure for radio transmittal of educational lessons. Further, it may be possible for after-school programs to use IRI to guarantee quality instruction for remedial education. For instance, volunteers or teachers with low levels of training could be set up to oversee students, reinforce participation and observe instructional modeling of the lessons taking place. Skilled teachers could work alongside lesson broadcasts to provide instant feedback, encouragement, and further support of lesson concepts to students during or after the lesson. Interactive Radio Instruction (IRI) has been used to support teachers to improve the delivery of education inside classrooms during formal school hours . With IRI, the teacher facilitates students’ interaction and engagement guided by an interactive radio Box 1: Interactive Audio Instruction When audio instruction is recorded and disseminated not over the radio but through CD players, MP3 players, cell phone recordings, or other broadcast audio machines it is called Interactive Audio Instruction (IAI). IAI can be distributed to homes during times of school closures but is most effective when used in a classroom setting. In contrast to IRI (which is often broadcast live, giving listeners little flexibility over the time of the day in which the lesson is played or the speed at which it is played), IAI allows teachers to pause, rewind and play lessons as needed. This enables them to control the pace, repetition, length and facilitation of lesson sections to match the appropriate pace for the classroom or the teacher’s development program, ensuring that learning meets the learner at his/her level. For instance, they can adapt the time of pauses and breaks to answer student questions on lessons, enable peer-to-peer games and assignments, and use other such activities that involve high engagement. Beyond engagement, IAI also has wider reach, as recordings can be played in areas where there is no radio coverage. IAI is not without its limitations. There is almost no way to influence the way IAI is used in the classroom: the instructor can control whether the IAI programs are played in the correct order, whether they are used every day as intended or used in learning “binges� of three or more programs played back-to-back on a single day, whether the lesson is used in its entirety or 10 minutes at a time with pauses in between, or if some segments or entire programs are skipped altogether. 11 lesson that is played during school hours. IRI was first implemented in Nicaragua in 1974. It included a series of in-class radio lessons, each followed by 20 minutes of teacher-directed activities. Instead of a textbook, a worksheet was followed during the broadcast and oral and physical responses were given. Songs and games were included in the programs for a change of pace, but unembellished mathematical work also kept the children's attention (Dock and Helwig, 1999). The effectiveness of IRI in improving the quality of education has led to its being the most popular form of education radio outside of emergency contexts. For example, IRI was implemented nationwide in Guinea to teach language, math and science for grades 1-6 in formal schools (Bosch, Rhodes, and Kariuki, 2002).6 In Zambia, although the Taonga Market IRI program was designed for out-of-school orphan populations (delivered through radio learning centers staffed by community volunteers), it was adopted by the ministry in government schools as well given its high impact (EDC, 2010). IRI has also been used to improve quality when facing teacher shortages in particular subjects (e.g., in foreign or local languages; science, technology, engineering, and mathematics [STEM] subjects) or in particular regions (including multigrade classrooms). For instance, in India (Delhi and Rajasthan), 120 30-minute programs composed of 3- to 4-minute activities help children learn English and math following the national curriculum. The program models effective pedagogical techniques and behaviors for teachers and focuses on the hard-to-teach topics (Thukral, 2016)7. Radio to assist teachers handling multigrade classrooms (where one teacher is in charge of several grades) was used in Costa Rica for grade 1-6 students for the topic of environmental education. Under that program, teachers were taught how to adapt activities and discussions so that students at different levels had different tasks (e.g. more advanced students helped others). IRI was being implemented in India (Karnataka) for grade 4 and 5 math and science, focusing on hard-t0-teach topics (Anzalone and Bosh, 2005). It was also evaluated in a small pilot in Greece, which tested its impact on small and isolated islands; the pilot found significant positive impacts (Giannakos and Vlamos, 2012). Finally, radio has also been used to facilitate teacher training and teacher certification. Many teacher trainings have been conducted entirely through education radio programs while some complement in-service or pre-service teacher training (Christina and Louge, 2015). For instance, in Nepal the government created a “dual audience� IRI intended to help teachers learn active teaching methods for mathematics and English. Thus, the IRI explained why certain activities could improve learning and 6 Please refer to the following hyperlinks to access sample materials: (i) From Guinea https://www.edu- links.org/resources/interactive-radio-and-audio-instruction-resources-guinea; (ii) from Liberia and Kenya https://www.educationinnovations.org/post/interactive-radio-and-audio-instruction-resources-liberia-and-kenya 7 Please refer to the following hyperlink to hear a sample lesson: https://vimeo.com/29620689 12 guided teachers on how to adapt them in settings without a radio. Since 1998, Guinea’s IRI has also incorporated a reflective teacher-training series focusing on incorporating student-centered instructional practices into teaching. South Africa also has a professional development support system for its English in Action program, which includes workshops, manuals, and regional coaches to support and advise teachers (Anzalone and Bosch, 2005). As mentioned in the introductory section, this already started in Sierra Leone with the provision of teacher training with printed materials during COVID-19 school closures. In all types of education radio, teachers play a crucial role in the development, testing and delivery/recording of education radio lessons. Teachers have the most experience with teaching lessons to students. They also best understand the school curricula and class lessons. Thus, in education radio programs across the globe, teachers develop the lesson structures and the lessons/scripts and ensure that lessons are aligned to curriculum. Subject specialists and the most qualified teachers are often also responsible for testing and examining the lessons and their scripts. Once lessons are read, teachers usually present the lessons so they can be recorded and broadcast either at home (in times of emergencies) or in classrooms (to complement lessons). Box 2: Education Development Center (EDC)’s IRI in Zanzibar (2006) - Radio Instruction to Strengthen Education (RISE) The EDC employed IRI in public schools in Zanzibar through the Radio Instruction to Strengthen Education (RISE) series: 78-99 preschool, standard 1, and standard 2 radio programs carefully developed by Zanzibari educators for Zanzibari students. Based on inclusive, playful, active, and engaging child-friendly pedagogy, each 30-minute broadcast integrated the formal Zanzibar curriculum with games, songs, stories, and activities that encourage problem solving and self- directed exploration. Each program covered four subject areas: Kiswahili language, math, English, and basic life skills. A comprehensive impact evaluation found that beneficiaries outperformed control students by 7.5 points out of 75 total points (or 10%). The greatest subject gains were made in Kiswahili. Student learning gains among girls were greater than among boys. Source : USIAD, SMZ, EDC (2009). https://bit.ly/3dIaYar 13 When education radio programs are employed in classrooms in the form of IRI or IAI, teachers are also critical in making the lessons interactive. Usually, education radio lesson characters introduce and provide direction for games, exercises, and group work. Teachers guide students through questions and answers, organize educational activities, serve as role models through these lessons, and provide just-in-time support to students answering questions or clarifying concepts. In IAI, they are also needed to play, pause, and rewind lessons. For best results, such lessons are often paired with teacher guides. Teachers’ interventions, which generate interactivity, are what make IRI and IAI most effective compared with other education radio programs. When education radio programs are used for teachers’ continuous professional development (CPD), the role of teachers is twofold: some teachers are charged with developing, testing and delivering lessons for less trained and qualified peers. Others are tasked with receiving the lesson, participating actively and incorporating practices into their daily jobs. To increase efficiency and reach, education radio programs can be complemented by a variety of resources. These include • Print materials to be referenced in the lessons, such as student workbooks, textbooks, posters, teacher guides; • Manipulable resources to use in the lessons, such as sticks or rocks for counting; • Community loudspeakers to broadcast lessons, build awareness, share radio broadcast schedules and remind users to connect; • Other forms of mass messaging (SMS messaging or short codes, WhatsApp), which while not ideal for lesson broadcasting can help build awareness, share schedules, remind and encourage use, receive feedback, and conduct formative assessments; and • Call-center help desks to collect questions from students regarding lessons, and provide technical, pedagogical, and socioemotional support to students and those supporting them in their learning. As mentioned in Box 1, to make education radio asynchronous or help it reach areas where broadcast is not available, MP3 players, memory cards, USBs, CDs and online media can be used to store the lesson and allow for its delivery whenever the student or the class is ready. 14 Box 3:Training Teachers with the Somali Interactive Radio Instruction Program (SIRIP) 2005 – 2011 The Somali Interactive Radio Instruction Program (SIRIP) provided high-quality interactive audio programs to children attending formal, nongovernmental, Quranic, and community schools — as well as to teachers from impoverished backgrounds in the formal government school system and in refugee camps. SIRIP trained about 9,500 teachers in universal teaching competencies, which were reinforced by the IRI lessons in the classroom. The most important and lasting teacher training was embedded in the daily IRI lessons that children received and continue to receive to date. Each 30- minute program was made up of a series of activities, songs, poems, dramas, and interviews that address the day’s learning objectives. The same radio characters appear each day, including a radio teacher and students who model the activity and skills for those listening. For example, a reading program might use a song to address one letter, its sound and what it looks like. The radio teacher might then guide the teacher to lead an activity about that letter. The activity is designed under the assumption that most teachers have had no teacher training and may not even be completely literate. Thus, for example, the radio teacher could say, “Now, teacher and students, let’s play a game with the word BOOK. Remember, students, BOOK is spelled B-O-O-K� so that poorly educated teachers who are not sure how a word is spelled in Somali are confident that the spelling is correct. The radio teacher then could give step-by-step instructions to the teacher on how to lead the activity. The teachers’ guide and the radio program instruct the teacher on what to do after the program to carry out what has been modeled in the lesson. Source: EDC Employees at Radio Rural Community of Forecariah broadcast information daily about Ebola outbreak and how families can protect themselves. Source and credit: Dominic Chavez/ World Bank 15 3. Evidence of effectiveness Impact Rigorous studies have found that IRI is a highly cost-effective way to increase learning outcomes in conventional classroom settings. Several authors find that IRI improved learning outcomes between 10 and 20 percent when compared with classrooms relying on the regular teaching techniques (Bosch, 1997; Dock and Helwig, 1999; Helwig, Bosch, and Dock 1999; Potter and Naidoo, 2006; Christina and Richmond, 2020). For instance, in Ethiopia, IRI was used to teach English in the primary grades supporting the official curriculum on a day-to-day basis. The program, which invited learners to be active participants (by speaking back, reading information on the board, singing songs, etc.) led to beneficiaries scoring 13 percentage points higher than students without IRI, reducing gaps between urban and rural and particularly benefiting girls (Chapman and Mahlck, 2004). Importantly, student achievement seems to increase the more lessons they are exposed to. For instance, pupils that received less than 33 English in Action lessons in South Africa improved by close to 7 percent, those that received between 33 and 66 lessons improved by 14 percent, and those that received more than 66 lessons improved by 24 percent, all seeming to show greater achievement increases over time (Leigh, 1995). In more difficult settings (for children who are out of school, affected by conflict, or in extremely poor or remote settings), beneficiaries also demonstrated improvements in student achievement, particularly in grade 1, in English, math and literacy (Ho and Thukral, 2009). Grades 2 and 3 also experienced positive impacts, with social studies also showing significant improvements in grade 3.8 Results in grade 4 are less consistent. In the Dominican Republic, RADECO has been providing IRI for out-of- school students for over 12 years, offering one hour of IRI and 30 minutes of follow-up activities. With just 1.5 hours of daily schooling, beneficiaries had comparable results in reading, writing, and language (and better results in math) than students in regular formal schooling (Haddad and Draxler, 2002). Interestingly, positive impacts were also found in early childhood education. In one group of studies, children receiving IRI in formal settings outperformed those not receiving IRI in all development categories assessed. In another study, students receiving IRI in informal centers with untrained 8 Although early uses of IRI focused on mathematics, the scope of IRI has increased over time to include language- related subjects, which are now dominant (Potter, 2007; Odera, 2011; MacKinnon and MacKinnon, 2010). Wathen and colleagues (2010) found that students from a predominantly African American, low-income, urban high school benefited from a science education radio program. 16 caregivers kept pace with students not receiving IRI in formal settings with trained caregivers (Ho and Thukral, 2009). Importantly, classroom-based IRI has had particular success in closing education gaps. These include access and achievement gaps between rural and urban populations, and between boys and girls, and for certain special educational needs populations (Damani and Mitchell, 2020). IRI has also been found to be an effective tool for in-service professional development. IRI improves teacher understanding of pedagogical concepts and, importantly, utilization of active learning and student-centered techniques in both lessons facilitated by radio and traditional lessons not based on radio broadcasts (Ho and Thukral, 2009). Furthermore, in low- and middle-income countries (LMICs) teachers benefit from taking part in of the IRI design process (the lesson design, recording, broadcasting, and assisting teaching) (Damani and Mitchel, 2020). For instance, in Mali the Formation Interactive des Enseignants par la Radio (FIER) project, an education radio program, had positive significant impacts on teachers’ knowledge and use of child-centered interactive teaching practices. The program was based on in-class programming that included active learning, brainstorming, Total Physical Response (a participatory language learning method requiring students to respond physically to verbal directives), and cooperative learning (e.g., group work, role plays, and games). Teachers who were trained using this program were surveyed three times throughout the project to assess their levels of familiarity with each of the instructional strategies above. Teachers were also observed during non-IRI lessons to assess their adoption of these techniques independent of radio guidance. As can be seen in Figure 6, teacher’s familiarity with active learning, brainstorming, Total Physical Response, and cooperative learning significantly increased over time. Most 17 Student of the RTP program during COVID-19. Source and credit: MBSSE importantly, their instructional practice outside of radio-directed lessons also consistently improved over the course of the project (Ho and Thukral, 2009). Figure 6: Percentage of observed teachers in the FIER program adopting IRI instructional methods Photo: MBSSE Source: Reproduced from Ho and Thukral, 2009. Key drivers of effectiveness In terms of design, a rapid evidence review carried out by the EdTech Hub in 2020 (which analyzed over 66 studies on radio teaching in LMICs) found that the most effective radio education programs are those that are most interactive. Mechanisms to increase interactivity vary, and include using parents, volunteers or caregivers as facilitators; incorporating a variety of exercises such as active listening, role-playing, jumping, singing and dancing; including short and regular pauses to allow teachers and students to interact, participate and respond (Bosch, 2004)9; and including ample time (almost half of the lesson) for listeners to call in or build in social media communication options to enable genuine interactivity (Bates 2005; Gavaza and Pearse, 2019). Broadcasters can ask questions for the audience to call and offer answers, they can invite the audience to call for clarifications, or they can even ask listeners to call-in for 9 For an example from Cambodia of pauses to increase interaction, see the following: https://www.youtube.com/watch?v=CR33BMo_R5U&list=PL-QzptyepknX68uYL1Y_jUF3r--v1lfKE&index=3 18 discussion (World Bank, 2020). To increase equity, its desirable to make these lines toll-free. As with all learning processes, increased engagement with radio programming (from both teachers and students) is linked to increased learning. To increase engagement, programs can use narrative and story-based pedagogy (Chatterjee et al., 2019) or employ humor (McKenna, 1993). They can also use stimuli such as playing certain sounds around certain lessons (Rodero, 2012; Şendağ et al., 2018; and Elekaei, 2019). In fact, enhancing the auditory experience through the inclusion of jingles, catchy songs, or sound effects might lead to students being more interested and engaged.10 Supplementing IRI with visual aids such as posters, textbooks, lesson books, learning toys and other learning and teaching materials has also been associated with increased learning (Hapeshi and Jones, 1992; Edwards et al., 2019; Chatterjee et al., 2019). Supplementary materials for IRI are particularly important when using IRI for mathematics, especially in the older grades. Levine and Franzel (2015) found that the lack of visual imagery on radio encourages students to think about telling stories using only words. This is a useful tool for classroom-based language-related learning but may prove problematic for learning math, particularly the more complex topics applicable to older grades (Mangal and Mangal, 2009). In fact, secondary school students in Ghana used educational programming on television more than they did on radio, in part because radio did not include the visual aids needed for teaching complex calculations (Yelkpieri et al., 2011). Finally, the most effective radio programs are contextually relevant. Apart from aligning the content of the RTP to the country’s reality and curriculum, characters should be similar to the children in the country, including both girls and boys. Using problem-solving in scenarios that are close to real life can also help the programs contextualize and enable collaborative problem-solving and knowledge construction (Rodero, 2012). For teacher training programs, the activities, examples and topics must be aligned to the realities in the classrooms. In terms of implementation, the World Bank Toolkit on radio instruction identifies that the most effective IRI programs include continuous improvement and quality control. Piloting programs in unwelcoming environments might be useful, as it leads to designs that are normally easier to scale. Where programs are large and successful, quality has been controlled at the design, production, and broadcast stages. Further, continued attention must be paid to keeping them dynamic and lively. Given the importance of teachers for IRI, teachers’ interest and momentum must be maintained by revising the programs from time to time, regularly retraining teachers and 10 As example: (i) use the following hyperlink to listen a song with the days of the week in Zambia: http://idd.edc.org/resources/audio/days-week; ii) listen to an animal song in Somalia: http://idd.edc.org/resources/audio/heesta-xuquuqda-animals; 19 reinvigorating their enthusiasm, and training and supporting new teachers in the use of radio lessons. The diverse components of radio education must be well integrated into the administrative, budgetary, and professional structures of educational and broadcasting institutions. Changes in these structures (for example, because of decentralization of education systems) may jeopardize the continuity of the program. An important part of making IRI projects sustainable involves ensuring that IRI is introduced into an environment that is responsive to change and that there is a plan for shared ownership and shared costs between broadcasting authorities, ministries, and other agencies. Sierra Leonean teachers recording RTP lessons. Source and credit: GPE/Ludovica Pellicioli 20 3. Recommendations Drawing on the country’s experience with the RTP, this section presents a set of short- and medium-term recommendations to maximize the impact of the RTP in Sierra Leone over access, student learning, teacher training, and resiliency of the education system (summarized in Figure 7). Given that the MBSSE and the TSC are already making significant investments in infrastructure and personnel for education radio, the section presents several options to take full advantage of these investments. Figure 7: Recommended uses of RTP for Sierra Leone in the short- and medium-term Immediate term Medium & Long-term • Focus on remedial education • Train teachers on identifying children that • Make it interactive and engaging are falling behind As a complement to • Develop and provide complementary • Direct lagging students toward the program normal schooling materials, especially for math and older • Train parents/community volunteers to (after-school) children accompany after-school IRI and scale-up • If possible, engage community at system level volunteers to facilitate lessons • Identify schools that have a radio and/or procure radios • Scale up IRI at a national level During school hours • Develop IRI materials for in-classroom • Procure devices for all schools instruction • Train teachers at a national level • Train teachers on IRI and pilot it • Evaluate current program. Increase interactivity, revise guides For teacher training • Ensure complementarity between • Scale up nationally student-focused RTP and continuous professional development Community ownership Data for continuous improvement Consistency between programs and with other initiatives Radio as an after-school remedial education tool The first potential use of the RTP is as an after-school program that complements normal schooling. To offset the effects of Covid-19 on student learning, the RTP could focus on remedial education, prioritizing key subjects such as reading and mathematics. Linking the broadcast to the curriculum and to the lesson of the day (e.g., 21 so that if students learn right triangles in class, the radio also talks about right triangles), as planned by the government, will maximize the reinforcement of topics and deepen their understanding. However, it is important that the radio lesson complements (not repeats) classroom instruction, and that it is highly interactive. The radio lesson should not simply repeat the same lesson covered in the classroom; it should rather guide the listener through alternative ways to approach or understand the concepts. Equally important is the need to make the radio lessons interactive, involving songs, games, problem-solving and others so that children are encouraged to participate and stay engaged. The existing phone line (where students can call-in to ask questions) is a good start, but the recordings themselves must involve interactivity. In the immediate term: 1. The TSC and MBBSE could start by revising the existing recordings to increase interactivity and overlap with the classroom curriculum. Employing effective pedagogy in radio lessons is among the most critical aspects of such programs. Effective pedagogical design should include: teaching at the right level, making lessons highly interactive and engaging (e.g. songs, games, talking back to the radio/teacher, answering questions/problems/puzzles with complimentary learning materials), and increasing learner responsiveness with pauses and phases in lessons. 2. Complementary materials (guidebooks, posters) should be developed for mathematics, especially for children in older grades. 3. It would be ideal to explore the possibility of recruiting community volunteers willing to accompany students in after-school radio programs, as these could play a key role to increase interactivity. If available, volunteers could receive a quick training in how to accompany children listening to radio and could support children any time of the day during vacation period and in after-school hours once school starts again. If the volunteer-facilitated after-school program takes place, it would be critical to see if there are radios available to use either at the school or lent by the community. If not, radios could be purchased for use at the school level either by the TSC/MBSSE or by adding it as a recommended eligible expenditure of the Performance Based Financing (PBF) grants.11 In the medium and long term, the volunteer-led program could take a more targeted approach so that the number of volunteers required is smaller. Remedial learning facilitated by volunteers could thus center on children that lag behind, while the rest of 11 Under the PBF grants program, schools receive grants three times per year. The amount of the grant received by the school is based on the schools’ performance on a range of indicators including student and teacher attendance; the percentage of students assessed regularly by teachers; the percentage of teachers found teaching (during school visits); and the percentage of students achieving grade level competency. 22 the students can be tasked to follow the radio program at home. To identify lagging students (and those at risk of dropping out), teachers would need to be trained in formative classroom assessment techniques. Parents, caregivers and other community members and volunteers would also need a fast training on how to facilitate interaction of children listening to the remedial education program be it after school or at home. IRI or IAI to maximize the quality of teaching during school hours Additionally, the RTP could be transformed into an IRI/IAI to maximize the quality of teaching of key subjects during school hours. As shown in the evidence section, IRI has been identified as an effective tool to increase student learning and improve teaching practices (making them more interactive and student-centered) in a very short amount of time when used in classroom contexts. IRI is especially pertinent when teacher training or knowledge is low, on geographies/subject-matter areas where there is a scarcity of teachers, and for hard-to-teach topics. The content of the IRI/IAI should fit the needs of Sierra Leone’s existing education levels, responding to the existing learning environments in-classroom (such as the average number of students per classroom, existing teacher skills, possibility of group-work, additional learning materials), as well as to the available radio infrastructure for recording, disseminating and reception/listening. Given the lack of radios at the school level (and the significant investments that distributing radios to schools could entail), it is recommended that this approach first be piloted in a small group of schools to test teachers’ acceptance and to improve quality of the programming before extending it nationally. With this in mind, the country could use the student vacation time to design a pilot of in-school IRI/IAI and its potential scale-up. Specifically, in the short term: • The TSC and the MBSSE could identify schools that have a radio or procure radios for a small subset of schools. The subset of schools could be those with the lowest performance on national examinations, schools that are hard to reach, or schools that are being targeted under the sandbox pilot for continuous professional development of teachers (as this would ease the implementation of the pilot). • The TSC and MBSSE could work with teachers and experts to develop the scripts and the complementary materials for the initial target grades and subjects. As mentioned earlier, it is critical that these lessons employ effective pedagogy. Given the existing evidence on the subject, tackling languages and early grades could be an interesting option. Another option could be to focus on STEM subjects for which there are teacher shortages. It would be desirable to analyze 23 the results of national examinations to identify the hard-to-teach topics that scripts should emphasize. • Teacher training for IRI/IAI should be prepared and rolled out at the pilot-level. • Ideally, progress should be evaluated through a fast baseline and end-line, and through a process evaluation. This evaluation should measure the extent to which the program leads to improvement in student learning and teacher practices, as well as the extent to which teacher training worked (if teachers understand how to use it in their classrooms and are willing to regularly use it); the adequacy of the scope, sequence, and speeds of recordings; the subject- matter topics and grades that are most impactful; and the adequacy of IRI vis-a- vis IAI (broadcasted vs. played from MP3/CD/etc.), taking into account broadband and teachers’ behaviors; among others. This process should also capture cost, so that cost-effectiveness can be measured. In the medium to long term, it would make sense to gradually scale up the methodology to all schools, gradually increasing coverage in terms of grades and subject matters (consistently guided by evaluations of impact), as well as procurement of radios or other listening devices and escalating teacher training for all schools nationwide. Radio for continuous professional development of teachers The third important use of radio education for Sierra Leone is continuous professional development of teachers. As mentioned earlier, this is an area where the country already made significant progress during Covid-19. However, work needs to be done to: • Gather feedback from users on the effectiveness of the current program and on potential areas of improvement. An increase in the interactivity of the program is likely desirable. The existing phone and WhatsApp communication channels are great ways to keep teachers engaged, but the recordings themselves must involve interactivity and the principles of adult learning. Existing manuals and complementary materials should likely be revised to increase the depth of learning. • In the short term, it would be desirable to ensure that it complements the student-focused RTP (either the after school and/or school-level IRI/IAI programs described above). Further, it is critical that the program aligns with the continuous professional development strategies designed by the TSC, including those tested through the sandbox. Once revised, the program should be gradually scaled-up nationally, integrated into the career framework for teachers. 24 Transversal areas: community ownership, data, and consistency To increase the effectiveness and sustain a successful education radio program that complements the basic education sector, community-ownership of the program is critical. Active communications with government officials, teachers, parents and caregivers, students, and the community in general should be planned. In fact, the Sierra Leone basic education radio programs should include a community-ownership campaign that ensures that the different stakeholders understand the program, appreciate its benefits, and support its implementation. These campaigns could publish information about the importance, relevance, and success of the programs, using data on improved test scores and/or other learning outcomes. This will be particularly important if the government decides to pursue a volunteer-led remediation program, or if radios are financed using school-level PBF grants (as the school management committee would have to choose to use the assigned funds to purchase a radio). The success of the program would also rely on its continuous testing and improvement, which in turns requires data and feedback loops. The design of the programs should include robust and continuous evaluation of both processes and impact. Attention should be paid to the pedagogical design of the program; it would be desirable to build in feedback sessions where teachers or volunteers could give feedback to those designing and recording the session on the speed, sequence, and adequacy of contents. Mirror sessions should take place for nonpedagogical and logistical aspects. There should also be question-and-answer sessions to adjust the training for those facilitating the program and become aware of concerns around the non-pedagogical aspects as well. It is recommended to first start with process evaluations, and then continue with impact evaluations. For instance, for after-school radio programs, data should first focus on understanding the success of strategies to make beneficiaries aware of the existence of the program, whether the broadcast is working and airing consistently, whether the complementary materials are reaching homes, and whether volunteers are being recruited and are actively attending, among others. Once this is working, data- collection on access (at home and in facilitated sessions), use, and performance of the program (in terms of remediation of learning) should take place. For school based IRI/ARI programs, the pilot should start by focusing on processes evaluations, capturing whether teachers understand the lessons, whether they are at the right level and adapting the package. Once a minimum viable product is designed with a small group of teachers and students, the evaluation should capture the learning outcomes among students receiving the program alongside those among students without the program (as well as teachers’ practices for those in and outside the program). The same applies for radio-based teacher training: process evaluations should precede (or 25 overlap) with impact evaluations. The indicators evaluating success should align with teacher career progression indicators. Even after national scale-up, processes and impact should be tracked, maintaining the continuous feedback loops. The satisfaction and feedback of users (parents, caregivers, volunteers, teachers, students, trainers, supervisors) should always be collected, analyzed, and integrated into improvements in the materials and processes. Teams should make it a habit to define and diagnose problems, design options for improvement, implement these options, evaluate success, adapt the program, and start again by redefining and re-diagnosing the problem, re-designing options, implementing, evaluating, and adapting (Filmer et al., 2018). Consistency between the radio programs and other school- and home-based programs should be ensured. Throughout the process, it would be critical to ensure that the three different radio-programs complement each other, reinforcing messages, contents and teaching practices. Furthermore, they should complement the broader programs implemented by the TSC and MBBSE by, for instance, using complementary materials already in the classroom, aligning teacher practices being incentivized with the teacher competencies developed by the TSC, and aligning all contents to the curriculum. 26 References and Bibliography Anzalone, Stephen J., and Andrea Bosch. Improving educational quality through interactive radio instruction: A toolkit for policymakers and planners. Development Research Group, The World Bank, 2005. Bates, A.W.(2005). Technology, e-learning and Distance Education (2nd ed.). Routledge. Biswas, Kumar, T.M. Asaduzzaman, David K. Evans, Sebastian Fehrler, Deepika Ramachandran, and Shwetlena Sabarwal. 2020. TV-Based Learning in Bangladesh: Is it Reaching Students? Washington, DC: World Bank, https://openknowledge.worldbank.org/handle/10986/34138. Bosch, Andrea. 1997. “Interactive radio instruction: Twenty-three years of improving education quality.� Education and Technology Notes 1 (1). Bosch, Andrea. 2004. “Sustainability and interactive radio instruction: Why some projects last.� In Adapting Technology for School Improvement: A Global Perspective (2004), p. 149. Bosch, Andrea, Rebecca Rhodes, and Sera Kariuki. "Interactive radio instruction: An update from the field." In Technologies for Education: Potentials, Parameters, And Prospects (2002): 134-143. Chapman, David W., and Lars O. Mahlck, eds. 2004. Adapting Technology for School Improvement: A Global Perspective. Paris: International Institute for Educational Planning (IIEP), UNESCO. Chatterjee, Papiya et. Al 2019. “Digital Story-Telling: A Methodology of Web Based Learning of Teaching of Folklore Studies�. 2019 21st International Conference on Advanced Communication Technology (ICACT) (2019). https://ieeexplore.ieee.org/abstract/document/8702047/references#references. Christina, Rachel, and Simon Richmond. 2020. Learning at Home in Times of Crisis Using Radio. Education Development Center, Inc. https://www.edc.org/sites/default/files/WB-IAI.pdf. Crawfurd, Lee, David K. Evans, Susannah Hares, and Justin Sandefur. "Teaching and Testing by Phone in a Pandemic." (2021). Damani, Kalifa, and Joel Mitchell. 2020. Radio: A Rapid Evidence Review. Rapid Evidence Review No. 4. EdTech Hub. https://doi.org/10.5281/zenodo.4556932. Dock, Alan, and John Helwig. 1999. Interactive Radio Instruction: Impact, Sustainability, and Future Directions. Washington, DC: The World Bank. Education Development Center (EDC). 2020. Learning at Home in Times of Crisis Using Radio: Interactive Audio Instruction Repurposing Toolkit. Education Development Center. (EDC). 2021. Interactive Audio Instruction Makes a Comeback. Education Development Center, Inc. 2021. https://www.edc.org/interactive-audio-instruction-makes- comeback. Education Development Center (EDC). 2021. Interactive Radio and Audio Instruction Resources: Guinea. https://www.edu-links.org/resources/interactive-radio-and-audio-instruction-resources- guinea. Education Development Center (EDC). 2021. Radio Instruction in India. http://dot- edu.edc.org/projects/RadioInstructionIndia.html. Education Development Center (EDC). 2010. Zambia-QUESTT-IRI Fact Sheet. Accessed July 30, 2021. https://www.edc.org/zambia-questt-iri. 27 Education Development Center (EDC). Undated. The Somali Interactive Radio Instruction Program. https://www.edc.org/sites/default/files/EDC-Education-Fragility-Series-Somalia-SIRIP.pdf. Edwards, Elizabeth (2019). Radio and audio in 2018. Journal of Radio and Audio Media, 26(2), 351 –359. Scopus. https://doi.org/10/ggsrm8. Elekaei, Atefeh, and Hossein Heidari TABRIZI. 2019. “Distance Education and Vocabulary Podcasting Tasks: Attitude in Focus.� Turkish Online Journal of Distance Education, April, 105 –20. https://doi.org/10.17718/tojde.557852. FCDO. “Radio Revision: How Lockdown Exam Preparation in Nepal Took to the Airwaves.� 2020. Girls ’ Education Challenge blog, October 28. https://girlseducationchallenge.org/blogs/blog- article/radio-revision-how-lockdown-exam-preparation-in-nepal-took-to-the-airwaves/. Filmer, Deon, Margarita Langthaler, Robert Stehrer, and Thomas Vogel. Learning to Realize Education ’s Promise. World Development Report. The World Bank (2018). Gavaza, Mudiwa A., and Noel J. Pearse. 2019. “Assessing the Social Media Maturity of a Community Radio Station: The Case of Rhodes Music Radio in South Africa.� The African Journal of Information and Communication, no. 24 (December): 1–21. https://doi.org/10.23962/10539/28661. Giannakos, Michail, and Panayiotis Vlamos. 2012. “Using Educational Webcasts in Small Multigrade Schools of Isolated Islands." International Journal of Education and Development Using ICT 8 (2). GRID3 Sierra Leone (2021). Radio Transmitter Coverage in Sierra Leone Haddad, Wadi., and A. Draxler. 2002. “Technologies for Education: Potentials, Parameters and Prospects; Challenges and Possibilities of ICTs for Education. UNESCO and the Academy for Educational Development.� Hapeshi, Kevin, and Dylan Jones. 1992. “Interactive Multimedia for Instruction: A Cognitive Analysis of the Role of Audition and Vision.� International Journal of Human-Computer Interaction 4 (1): 79–99. https://doi.org/10.1080/10447319209526029. Hernes, Gudmund, and UNESCO. Emerging Trends in ICT and Challenges to Educational Planning. UNESCO. http://Lst-Iiep.Iiep-Unesco.Org/Cgi- Bin/Wwwi32.Exe/[In=epidoc1.in]/?T2000=015516/(100). Ho, Jennifer, and Hetal Thukral. 2009. “Tuned in to Student Success: Assessing the Impact of Interactive Radio Instruction for the Hardest-to-Reach.� Journal of Education for International Development 4 (2): 34-51. Kindernothilfe, E.V. 2021. “Zambia: Radio Schools—Kindernothilfe.� http://www.kindernothilfe.org/worldwide+Programming/Projects+and+Countries/Africa/Zambia/ Project+Example_+Radio+schools-p-652.html. Leigh, S. 1995. Changing Times in South Africa: Remodeling Interactive Learning. Washington, D.C. Learning Technologies For Basic Education. Levine, Sarah., & Franzel, Johana. (2015). Teaching Writing with Radio. English Journal, 104. https://srlevine.people.stanford.edu/sites/g/files/sbiybj4286/f/ncte_2015_levine.pdf. Mangal, S.K and Mangal, Uma (2009). Essential of Educational Technology. New Delhi: Learning Private limited. MacKinnon, Gregory, and Paula MacKinnon. 2010. “Technology Integration in the Schools of Guyana: A Case Study.� Computers in the Schools 27 (3-4): 221–46. https://doi.org/10.1080/07380569.2010.523884. 28 McKenna, Linda M. 1993. “The Relationship between Attributes of a Children’s Radio Program and Its Appeal to Listeners.� Educational Technology Research and Development 41 (1): 17–28. https://doi.org/10.1007/bf02297089. Mason, Harriet. 2016. “A second chance at schooling for pregnant teenagers in Ebola-affected Sierra Leone: New initiative supports pregnant teenage girls who refuse to give up on their education.� UNICEF Stories. UNICEF. Odera, Florence Y. 2011. Learning English Language by Radio in Primary Schools in Kenya. ERIC. https://eric.ed.gov/?id=ED529920. Poon, Linda. 2015. “Now This Is an Example of Truly Educational Radio.� NPR.Org. https://www.npr.org/sections/goatsandsoda/2015/02/18/387027766/now-this-is-an-example- of-truly-educational-radio. Potter, Charles. (2007). South Africa: Interactive Radio for Supporting Teachers of English as a Second Language. In H. Perraton (Ed.), International case studies of teacher education at a distance (pp. 231–252). BIS-Verl. der Carl-von-Ossietzky-Univ. Potter, Charles, and Gordon Naidoo. 2006. “Using Interactive Radio to Enhance Classroom Learning and Reach Schools, Classrooms, Teachers, and Learners.� Distance Education 27(1): 63-86. Rodero, Emma. 2010. “See It on a Radio Story.� Communication Research 39 (4): 458–79. https://doi.org/10.1177/0093650210386947. Şendağ, Serkan, Nuray Gedik, and Sacip Toker. 2018. “Impact of Repetitive Listening, Listening-Aid and Podcast Length on EFL Podcast Listening.� Computers & Education 125 (October): 273–83. https://doi.org/10.1016/j.compedu.2018.06.019. Statistics Sierra Leone (SSL) and ICF International. 2014. Sierra Leone Demographic and Health Survey 2013. Freetown. Thukral, Hetal. 2016. Review of Examining the Impact of Teacher Practice on Student Learning in Interactive Radio Instruction (Iri) Classrooms: Lessons from New Delhi and Rajasthan (India). University of Maryland. https://drum.lib.umd.edu/bitstream/handle/1903/18176/Thukral_umd_0117E_16905.pdf. Thukral, Hetal. “Examining the Impact of Teacher Practice on Student Learning in Interactive Radio Instruction (IRI) Classrooms: Lessons from New Delhi and Rajasthan (India).� Doctoral Dissertation, University of Maryland, College Park, July 30, 2021. https://www.proquest.com/openview/396a429f55d2cbb18bf8cd30491e50f8/1?pq- origsite=gscholar&cbl=18750. USAID, SMZ, and EDC. 2009. Radio Instruction to Strengthen Education (RISE) in Zanzibar — Learning Gains Assessment: More than Child’s Play. https://bit.ly/3dIaYar. Wathen, Adam., Llewellyn, Donna., Ludovice, Peter., Hunt, William., & Usselman, Marion. (2010). A Study of Inquiry Based Informal Science Education in an Urban High School Physics Class. Paper presented at 2010 Annual Conference & Exposition, Louisville, Kentucky. https://peer.asee.org/a-study-of- inquiry-based-informal-science-educatio n-in-an-urban-high-school-physics-class. Werne, Annia, and Jaymia Ecker. 2020. Education Coverage in Sierra Leone. GRID3 Sierra Leone. https://www.grid3.org/content/uploads/2020/12/GRID3SLEEducationReport_Final1.pdf. 29 World Bank. 2020. COVID-19 High-Frequency Monitoring Dashboard. https://www.worldbank.org/en/data/interactive/2020/11/11/covid-19-high-frequency- monitoring-dashboard. World Bank. 2020. Review of Sierra Leone: Covid-19 Impact Monitoring Survey (CIMS). Freetown, Sierra Leone: World Bank Group. World Bank Group; Education Development Center. 2015. Expanding Access to Early Childhood Development: Using Interactive Audio Instruction. World Bank Group, Washington, DC. © World Bank. https://openknowledge.worldbank.org/handle/10986/21623 License: CC BY 3.0 IGO. World Bank Group, USAID, and Education Development Center. 1999. Review of Interactive Radio Instruction: Impact, Sustainability, and Future Directions. Edited by Alan Dock, John Helwig, and Andrea Bosch. World Bank Group and USAID https://documents1.worldbank.org/curated/en/198821468779129452/pdf/multi-page.pdf. Yelkpieri, Daniel; Esia-Donkoh Kweku, Wilson; and Kwesi (2011) "Patronage of Educational Broadcasts and its Effects on Academic Growth of Students of Winneba and Apam Senior High Schools in the Central Region of Ghana.,� Academic Leadership: The Online Journal: Vol. 9 : Iss. 1 , Article 40. Available at: https://scholars.fhsu.edu/alj/vol9/iss1/40. Zacharia, Sharon, World Bank, and EdTech Hub. 2020. Review of Education Radio Knowledge Pack with a Focus on Low-Resource Settings. World Bank Group. https://pubdocs.worldbank.org/en/351561596545287034/EduRadio-KnowledgePack- WorldBank.pdf. 30 1