GUIDANCE NOTE Phone-Based Formative Assessment © 2022 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington, DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the information included in this work. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 4.0 International license (CC BY 4.0) https://creativecommons.org/licenses/by/4.0/, with the following mandatory and binding addition: Any and all disputes arising under this License that cannot be settled amicably shall be submitted to mediation in accordance with the WIPO Mediation Rules in effect at the time the work was published. If the request for mediation is not resolved within forty-five (45) days of the request, either You or the Licensor may, pursuant to a notice of arbitration communicated by reasonable means to the other party refer the dispute to final and binding arbitration to be conducted in accordance with UNCITRAL Arbitration Rules as then in force. The arbitral tribunal shall consist of a sole arbitrator and the language of the proceedings shall be English unless otherwise agreed. The place of arbitration shall be where the Licensor has its headquarters. The arbitral proceedings shall be conducted remotely (e.g., via telephone conference or written submissions) whenever practicable, or held at the World Bank headquarters in Washington, DC. Attribution – Please cite the work as follows: Khurana, Aishwarya; Lucy Kruske; Victoria Levin; Julia Liberman; and Diego Luna-Bazaldua. 2022. Guidance note on phone-based formative assessment. Washington, DC: The World Bank. License: Creative Commons Attribution CC BY 4.0 IGO. Translations – If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations – If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Views and opinions expressed in the adaptation are the sole responsibility of the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content: The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to reuse a component of the work, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. PHOTO BY: © 2021 STIVEN BEITIA/SHUTTERSTOCK. GUIDANCE NOTE PHONE-BASED FORMATIVE ASSESSMENT GUIDANCE FOR EVALUATING IMPLEMENTATION FEASIBILITY OF PHONE-BASED FORMATIVE ASSESSMENTS IN LOW-RESOURCE SETTINGS AISHWARYA KHURANA LUCY KRUSKE VICTORIA LEVIN JULIA LIBERMAN DIEGO LUNA BAZALDUA SUPPORTED WITH FUNDING FROM THE GLOBAL PARTNERSHIP FOR EDUCATION PHONE-BASED FORMATIVE ASSESSMENT Contents Contents Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   5 Executive Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   7 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   9 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   11 What is formative assessment?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   11 The case for phone-based formative assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   12 Three phone-based solutions for formative assessment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   13 Short Message Service (SMS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   14 Interactive Voice Response (IVR) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   16 Direct phone calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   16 2. Considerations for implementing phone-based formative assessments . . . . . . . . . .   25 Education system capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   26 Learning content and its alignment with assessment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   26 Human resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   26 Technological and logistical considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   27 Demographics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   27 Technology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   28 Service providers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   29 Financial considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   29 Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   30 Local implementer capacity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   30 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   33 Additional resources for consideration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   35 Annex 1. Examples of phone-based formative assessments using SMS, IVR and direct phone calls. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   36 4 PHONE-BASED FORMATIVE ASSESSMENT Acknowledgments  Acknowledgments  This guidance note was prepared by Aishwarya Khurana, Lucy Kruske, Victoria Levin, Julia Liberman, and Diego Luna Bazaldua of the World Bank’s Learning Assessment Platform (LeAP) team. This work was sponsored by the Global Partnership for Education’s (GPE’s) grant to support continuity of learning during the pandemic. The team worked under the overall guidance of Omar Arias (Practice Manager, Global Engagement and Knowledge Unit, Education Global Practice). Valuable feedback was obtained from Maria Barron, Robert Hawkins, and Inaki Sanchez Ciarrusta. Colleagues who contributed to this document with reviews and feedback include Rabia Ali, Marguerite Clarke, Martin Elias de Simone, Koen Geven, James Gresham, and Mari Shojo. Contributions were also provided by Mark Zelman and CREA Consultores. The team was supported by the Russia Education Aid for Development (READ) Trust Fund program. This note is part of a broader set of global knowledge products on phone-based learning assessments, including a note on psychometric considerations for phone-based assessments, a landscape review of existing phone-based assessment interventions and their key features, and a checklist template to assess implementation prerequisites and enabling conditions for phone-based formative assessment solutions. 5 PHOTO BY: © 2021 TAWANROONG/SHUTTERSTOCK. PHONE-BASED FORMATIVE ASSESSMENT Executive Summary Executive Summary ■ Phone-Based Formative Assessment refers to the use of mobile phone technology to engage students outside the classroom, assess their learning in real time, and provide timely, constructive feedback. Such assessment is particularly critical to promote learning continuity in the context of COVID-19-related school closures, which have led to the expansion and availability of remote teaching and learning resources. ■ This guidance note explores considerations for the use of Short Message Service (SMS), Interactive Voice Response (IVR), and direct phone calls to conduct formative learning assessments. It focuses on factors that may affect the feasibility of implementing such solutions in low-resource contexts and discusses conditions that may be needed for successful implementation. ■ A combination of phone-based assessment tools can be helpful to target different student groups, particularly those most disadvantaged, and provide tailored support for the learning process outside the physical classroom. ■ While some countries are considering solutions for remote formative assessment to support learning continuity as a response to COVID-19, low-cost mobile technologies offer a long-term opportunity to improve the quality of education overall by supporting learning outside the classroom and contributing to education systems’ resilience to future shocks, including natural disasters and unrest in fragile contexts. ■ Phone-based learning assessment activities can further be used to mitigate the digital divide, with different applications of mobile technologies having their own advantages and disadvantages. ■ Considerations for implementing phone-based formative assessments tend to fall under two categories: (1) the capacity of an education system and (2) technological and logistical considerations. Concerning the education system’s capacity, it is important to align the assessment with existing learning content and education policy, as well as with critical human resource capacity. Critical technological and logistical factors to consider when determining the appropriateness and implementation feasibility of such assessment modality include a country’s demographic makeup, access to and availability of mobile technologies, and the capacity of service providers. Additional logistical considerations pertain to financial resources and time constraints, as well as to the capacity of local implementing organizations with relevant experience 7 PHOTO BY: © 2020 AAK FILM FOOTAGE/SHUTTERSTOCK. PHONE-BASED FORMATIVE ASSESSMENT Introduction Introduction Since the beginning of 2020, the COVID-19 pandemic forced countries around the world to close schools and introduce or expland remote learning. While many countries were able to quickly roll out remote learning initiatives using various web-based applications, not all children were able to access such resources, partic- ularly when they required having internet connectivity and digital devices (such as computers, tablets, and smartphones) at home. This situation reflects some of the challenges faced in low-resource1 contexts, which are defined as rural or urban settings where human, economic, time, or environmental resources are con- strained. In general, low-resource contexts in low- and middle-income countries present limited infrastructure or cannot offer basic services, including education resources and services to support remote learning. As an alternative to reach students in low-resource contexts, countries such as Bhutan, Cambodia, Colom- bia, Costa Rica, Czech Republic, Ethiopia, Fiji, and many others delivered educational content through other means, such as television, radio, and hard-copy learning packets (World Bank 2020). While the initial focus during school closures was on ensuring the continued delivery of educational content, limited information was available on whether students accessed this content, whether they were learning, and what additional learning materials and resources they needed to progress along their learning trajectory. Prior to the COVID-19 pandemic, education was delivered in a traditional classroom setting. In the classroom, information about what students know, understand, and can do, and where students need further support and practice, was obtained by teachers through daily interaction with students. This process is known as formative assessment. While the concept of formative assessment includes peer-to-peer and self-assessment, formative assessment comprises activities that allow teachers to know students’ learning status, determine students’ progress towards curriculum learning goals, and provide timely feedback to students to support their learning process. With schools closed during the COVID-19 pandemic, the typical classroom-based mode of formative assess- ment and the ability of teachers to provide immediate feedback to students could not continue, and educa- tion policymakers, organizations, and practitioners sought innovative solutions to ensure that teachers could continue supporting student learning outside school doors. With a global penetration rate of 104 percent, feature (basic) phones are a promising option for supporting learning continuity through formative assessment, especially in contexts with limited access to more advanced digital technology and internet connectivity. Besides their immediate relevance in the COVID-19 context, investment in phone-based assessment solutions can strengthen crisis preparedness and make education systems more resilient to future shocks. The purpose of this note is to provide information on how basic phones, specifically through Short Message Service (SMS), Interactive Voice Response (IVR), and direct phone calls, can support formative assessment activities and con- tinuation of learning outside the classroom even in low-resource contexts. This document also provides guidance on how to assess the feasibility of implementing phone-based formative assessment activities in a given context. 1 For the purpose of this note,‘low-resource’ refers to areas where network connectivity for feature (basic) mobile phones and smartphones is limited or nonexistent. 9 PHOTO BY: © 2020 SONIA DHANKHAR/SHUTTERSTOCK. PHONE-BASED FORMATIVE ASSESSMENT 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond What is formative assessment? activities that inform teachers about what students know and can do. Formative assessment differs As the COVID-19 pandemic forced schools to close, from summative assessment, which tends to hap- many education systems shifted gears and made pen at the end of a curriculum unit or at the end available learning resources to students so they of the school year to establish whether students could learn remotely. With students learning outside achieved the learning goals specified in the cur- of the classroom, the usual practice of classroom as- riculum. Formative assessment aims to generate sessment was difficult to implement given the some- timely information that teachers can use to shape times limited, and often nonexistent, communication instruction, help students understand concepts between students and teachers. However, assessment and reinforce learning content, and inform subse- of student learning is a fundamental feedback mech- quent coursework. Moreover, formative assessment anism in education, allowing the stakeholders of the is an adaptive process centered on individualized teaching and learning process – teachers, principals, support and constructive feedback. As such, for- students, parents, policymakers, among others – to mative assessments are low-stakes, do not take the understand what is being learned, where further form of a particular task, and may or may not be support is needed, and what additional learning used in formal grading. With formative assessment, resources need to be provided to learners to ensure the focus is on supporting the students’ progress to- they can continue with their learning trajectory. 2 ward achieving learning goals, and that is why it is sometimes referred to as “assessment for learning.” Formative assessment consists of ongoing feedback between teachers and students to monitor stu- During sudden shocks to education systems that keep dents’ progress towards curriculum learning goals. students out of school for a long time, such as pan- Traditional methods of formative assessment rely demics, social unrest, natural disasters, conflicts, or on in-person interaction, such as oral questioning, other emergencies, information about what students alongside other formal approaches such as quizzes, are actually learning is at a premium, as students are classroom projects, and homework (Perry 2013). In not directly observed by teachers in the classroom. the context of in-school instruction, formative as- Conducting formative assessments remotely through sessment is typically administered as part of daily basic phones can provide such timely information teaching in the form of checks for understanding, to teachers, parents, and students, and thus support group classroom activities, homework, and other learning continuity in times of crisis and beyond. 2 For more information on learning assessment considerations and planning during the school reopening process, please retrieve and re- view our Learning Assessment Platform team guidance note: https://blogs.worldbank.org/education/assessing-outside-classroom-box-while- schools-are-closed-potential-phone-based-formative 11 PHONE-BASED FORMATIVE ASSESSMENT 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond Figure 1-1. Mobile Cellular Subscriptions (per 100 people) Mobile Cellular Subscriptions (Per 100 people) 140 130 122 2000 2019 120 109 100 94 80 60 60 48 40 20 12 8 1 0 0 World High Income Upper Middle Income Lower Middle Income Low Middle Income Source: World Bank (n.d.) The case for phone-based available modalities, such as basic phones,3 can fa- formative assessment cilitate learning continuity. As shown in Figure 1-1, even in low-resource contexts, more than half of the The COVID-19 crisis forced most countries around population in low-income countries has a mobile the world to partially or fully close schools to phone subscription (World Bank, n.d.). For instance, contain the spread of the coronavirus (that is, in Sub-Saharan Africa, mobile cellular subscription COVID-19), at the height of school closures affect- (per 100 people)4 increased from 2 in 2000 to 87 in ing 90 percent of the world’s learners (UNESCO 2019 (World Bank, n.d.). Although this represents a 2021). Without the ability to provide in-person rapidly increasing rate, when one looks at unique instruction, countries such as Bhutan, Cambodia, mobile subscribers5 the penetration rate remains Colombia, Ethiopia, Fiji, Ghana, Kenya, and many low in lower-middle and low-income countries. For others have turned to delivering learning content example, in Sub-Saharan Africa, unique mobile sub- to students and parents through various modali- scribers stand at 46 percent of the population or 495 ties, including online, TV/radio, and via delivery of million, and only 28 percent of the population (or hard-copy learning packets (World Bank 2020). 303 million) have access to internet on their mobile phone (Global System for Mobile Communications As countries’ experiences have shown, especially Association [GSMA] 2021). Therefore, conducting at the time of mass school closures, many children formative assessment remotely using basic phones and youth did not have access to digital re- sources, can improve access to learning support outside of due to poor connectivity or limited access to digital the school and can enable more students to contin- devices. In this context, more low-tech and readily ue to move along their learning trajectory. 3 Basic phones are the most basic forms of mobile phones that use second generation (2G) networks. They can be identified by their small screen, standard numeric keypad, long battery life, and absence of high-end features such as access to the internet. Basic phones primarily perform two main functions: sending and receiving voice calls and text messages. These phones rely on mobile networks, such as Vodafone, for communication. One common example is the Nokia 1100 phone. “Feature phones” and “feature smart phones” are largely similar in ex- ternal appearance to basic phone but also come with minimum multimedia and internet capabilities. Such types of phones were among the first to use third generation (3G) networks.. 4 Mobile cellular telephone subscriptions are subscriptions to a public mobile telephone service that provide access to the Public Switched Telephone Network (PSTN) using cellular technology, that helps in making connection between two telephones lasting as long as the dura- tion of the call. Mobile cellular telephone subscriptions (the indicator) includes (and is split into) the number of postpaid subscriptions, and the number of active prepaid accounts (i.e., that have been used during the last three months). The indicator applies to all mobile cellular subscriptions that offer voice communications. It excludes subscriptions via data cards or USB modems, subscriptions to public mobile data services, private trunked mobile radio, telepoint, radio paging and telemetry services. 5 A unique mobile subscriber is defined as an individual person who can own multiple mobile connections (i.e., SIM cards). 12 PHONE-BASED FORMATIVE ASSESSMENT 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond PHOTO BY: © 2021 FARID TAJUDDIN/SHUTTERSTOCK. Phone-based assessment may also improve the attendance to school, and reaching children who take-up and effective use of remote learning re- may be at risk of not returning to school. sources by providing students, their parents, and their teachers with the necessary information to Moreover, the usefulness of phone-based forma- support the learning process, albeit from a distance. tive assessment is not limited to the context of While mobile phone technologies include feature COVID-19. Remote learning and remote assessment phones, smart feature phones, and smartphones, may be sustained even after return to in-person students in low-resource and low-connectivity con- instruction if it proves to create value to stakehold- texts may typically only have access to basic phones ers and proves to be sustainable over time. In the with basic functionalities, such as text messaging long term, phone-based assessment can become an (SMS) and phone calls. To address their needs and essential ingredient for building the resilience of to support learning continuity for those who are education systems and expanding access to learning often behind the digital divide, this guidance note opportunities for out-of-school children and youth. In will focus exclusively on basic phones. fragile contexts, emergencies due to natural disasters, en shocks to in-person schooling, assessments or sudd­ Phone-based assessment can play a critical role administered over the phone can be part of formative in fostering learning continuity and learning re- strategies for the continuity of learning. Likewise, covery. In the short term, phone-based formative phone-based assessment can support learning of the assessment is critical to support learning continuity future, where students, even those in low-resource during COVID-19 school closures. Low-tech modes contexts, can learn anytime and anywhere. of remote content delivery have expanded through the use of TV and radio, and these can be comple- mented with the multimodal use of phone-based Three phone-based solutions formative assessment. In the medium term, some of for formative assessment the methods of phone-based formative assessment can support critical learning recovery by providing Basic phones enable users to send and receive SMS/ opportunities for complementing and reinforcing text messages, automated phone calls, and voice learning outside of the classroom, monitoring messages, and to connect one or more parties in 13 PHONE-BASED FORMATIVE ASSESSMENT 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond Box 1-1. Mobile-based Post Literacy Programme, Pakistan About the program: Implemented in the Punjab Province of Pakistan, this program aimed at pro- viding literacy teaching in Urdu to young and adult women ages 15-30 with limited or no literacy skills. The program ran from 2009-2012 in three phases: pilot (first) phase in 2009, second phase from April-September 2010, and third phase from March-August 2012. Over three years, the program provid- ed free mobile phones, SIM cards and SMS services for four months to over 2,500 participants. Content delivery: The program spanned more than six months and was divided into two stages: the first stage comprised in-person teaching that lasted two months and the second stage focused on mo- bile-based literacy and lasted for four months. During the first stage (two months), the participants met six times per week for two to three hours to learn and write the alphabet and to read, with an emphasis on phonics. During the second stage (four months), participants received SMS on their phones 6-8 times a day. After receiving the SMS, the participants were required to read the texts provided in the SMS, to write the text in their notebooks and to answer the questions that were asked. Additionally, basic math was taught to participants using the calculator function on their phones. Assessment procedure: To monitor the progress and participation, a web-based system was used to send SMS messages to the participants. Assessments mainly involved responding to questions/multi- ple-choice-questions (MCQs). Additionally, a monthly exam was conducted at learning centers to track the development of participants’ literacy skills and rates of retention in the program. Results/outcome: During the pilot phase with 250 learners, in the Sialkot district, test results showed that by the end of four months of mobile-based literacy instruction, only 14% of the participants fell into the low (0-50%) score range, as compared to 90% during the first month. There was also a notable increase in participants falling into the high (70-100%) score range, rising from 0% to 39%. Additionally, participants also reported that they developed the ability to read an Urdu newspaper, signboards, and simple Urdu books, and learned how to read and solve simple money problems using calculators. Source: U. Hanemann (Ed.), Mobile-Based Post Literacy Programme (Pakistan: UNESCO Institute for Lifelong Learning, 2013). Re- trieved from: https://uil.unesco.org/case-study/effective-practices-database-litbase-0/mobile-based-post-literacy-programme-pakistan a live phone conversation. This section discusses feature included across all types of mobile phones. applications of SMS/text messages, interactive voice Formative assessment through text messaging is response (IVR), and direct phone calls using basic an asynchronous process, during which students phones to conduct formative assessment remotely. respond to questions messaged based on previ- ously provided content. The learning content de- livery and assessment can occur either at different Short Message Service (SMS) moments in time or presented together to collect information about students’ learning immediately. SMS is by far the most widely used phone-based technology due to its low cost, relative ease to de- The delivery of assessment content through SMS velop and deploy in mobile applications, and com- typically requires the development of a tailored patibility with a wide variety of existing platforms software application that sends out SMS messages and mobile phones. Text messaging is a basic and receives responses, which can be done through 14 PHONE-BASED FORMATIVE ASSESSMENT 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond Box 1-2. Mobile Literacy Programme, Afghanistan About the program: The program, implemented by Afghan Institute of Learning in 2011, aimed at imparting literacy in Dari to 50 female students ages 14-32 living in two villages in rural Afghanistan. The program combined classroom teaching and mobile phone technology by providing basic mobile phones to all participants alongside a notebook to practice their reading and writing skills. The mobile phones were provided free of cost and with enough credit balance for students to complete literacy assignments sent by the teacher. The program also encouraged the personal use of the mobile phones, assuming that such use would strengthen literacy skills. Content delivery: Over the period of four months, students met six times per week for one hour. The in-person instruction in Dari comprised traditional methods, such as reading aloud, writing in the notebook, and memorizing words. Assessment procedure: The in-person classes during which students received literacy instruction were complemented with formative assessment that involved direct interaction between students and a teacher through SMS. After the in-person classroom instruction, the teacher sent daily text messages to students with a reading exercise, prompting them to respond by text messaging. Primarily, the text message comprised three types of questions: 1. Fill-in-the-blank sentences, which students had to rewrite with the word filled in; 2. Open-ended questions to facilitate critical thinking and writing skills to which students were re- quired to compose a response; and 3. Correctly reordering a jumbled sentence to practice comprehension and grammar. Results/outcomes: At the end of the program, some of the key results included: ■ The rate at which students covered content was four times faster relative to the regular speed of education in a classroom environment (4 months for an 18-month program). ■ 83% of the students were able to pass all the literacy assessments within four months. ■ Students sent approximately 1,750 messages per month (including messages to teachers and for other). ■ All students remained in the program during the assessment process (4 months). ■ The project addressed the issue of lower phone ownership by women, and some students left the course with the ability to read magazines and newspapers. Source: U. Hanemann (Ed.), Mobile Literacy Programme in Afghanistan (Afghanistan: UNESCO Institute for Lifelong Learning, 2013). Retrieved from: https://uil.unesco.org/case-study/effective-practices-database-litbase-0/mobile-literacy-programme-afghanistan-af- ghanistan 15 PHONE-BASED FORMATIVE ASSESSMENT 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond Box 1-3. Viamo, in Zimbabwe and across Africa About the program: In response to school closures as a result of the COVID-19 pandemic, the program was designed and implemented to offer prerecorded lessons, using the IVR platform. In Zimbabwe, the pilot phase included 10,000 students, all age 5. In other countries, the scale of the work varied; for example, Tanzania had 200,000 callers in a 12-month period. Content delivery: The program primarily focused on the delivery of literacy and numeracy lessons or modules in the Ndau language at the early childhood level. Each module was set to 10 minutes a week. Assessment procedure: Mini assessments were incorporated into the modules. These assessments were short and simple. To check if the child has listened to the audio and/or if they are still present, the system would first play a nursery rhyme and then prompt a question, such as “What sounds does this animal make?” and provide options for children to respond to. Source: M. Steinmetz, Mobile Distance and Hybrid Education Solutions: A Knowledge Pack (Washington, D.C.: World Bank, 2020). Retrieved from: https://pubdocs.worldbank.org/en/685691598013656403/pdf/WorldBank-EdTech-Team-Knowledge-Pack-MobileDis- tance-HybridEducationSolutions-version2.pdf a variety of mobile aggregator6 platforms, such IVR also allows users to leave a recorded voice mes- as Twilio and Tatango. See boxes 1-1 and 1-2 for sage with a detailed response or query. In addition to examples on the use of text messages for learning enabling access to content for individuals with limit- assessment. (For more examples on the use of SMS, ed literacy, IVR can be used to reach individuals with please see Annex 1.) visual impairments, and it facilitates delivering the same content in multiple languages, including those that have no written form (Diedhiou et al. 2015). All Interactive Voice Response (IVR) these features offered by IVR solutions may increase the engagement of students during the learning as- Interactive Voice Response (IVR) is a tool that can be sessment process. See boxes 1-3 and 1-4 for examples utilized to reach populations with limited literacy of the use of IVR in learning assessments. (Please see because it relies on speech rather than text (World Annex 1 for more examples on the use of IVR to as- Bank 2012). It can be used on any type of landline sess student learning.) or mobile phone to communicate information via a voice message system in the form of audio recording (Diedhiou et al. 2015). IVR works by inviting users to Direct phone calls call a number, usually toll-free. An automated sys- tem delivers pre-recorded audio messages that may Direct phone calls between students and teachers contain on-demand educational content or voice- can ensure continuity of the learning process. In based quizzes to which users respond by pressing a the context of COVID-19, the primary purpose of number key or saying the number that corresponds direct calls is to ensure that students can continue to their answer choice (i.e., “say yes or press 1, say no learning through guidance and constructive feed- or press 2”). back from teachers or instructors outside of the classroom (e.g., when doing homework, reviewing 6 Mobile aggregators link different carrier networks with SMS software / platform providers. They simplify the mass distribution of SMS to many carrier networks. Mobile aggregators have agreements with carrier networks to send and receive SMS through these networks. 16 PHONE-BASED FORMATIVE ASSESSMENT 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond Box 1-4. Allo Alphabet, Côte d’Ivoire About the program: The study was first piloted in a rural village in Adzopé Department of Côte d’Ivoire. The study included 38 students in grade 5 and spanned five weeks from October-December 2018. Later, from 2019 to 2020, the program was scaled up to reach 1,200 children (800 in the treatment group and 400 in the control) in remote rural communities of Côte d’Ivoire. The literacy curriculums in Attié and French were designed and implemented on an interactive voice response (IVR) system named Allo Alphabet. Content delivery: The content delivered focused on overall literacy skills such as phonological aware- ness and print-sound mapping. Over time, the difficulty of the content was increased from simple pho- neme and syllable awareness to mapping between letters, words, and sounds. For example, the learner performed tasks such as matching words/syllables sharing similar sounds/combination of sounds and choosing the word or sound that did not belong to the given set. The learner was required to call a specified number. Once the call was placed, it was automatically disconnected and the system was programmed to call back the learner to avoid any charges to them. At the start of each call, the system played a welcome message, recorded by an Ivorian researcher, that updated the learner of their progress and based on the mastery of concepts that the learner exhibited, selected a lesson to be played. Each lesson followed a structure and would start with an explanation of content and guidance on how to respond to the questions. Assessment procedure: To assess the learners, the system played a prerecorded audio message con- taining a question and two or three response options. After selecting a response, the learner received immediate feedback from the system. Following correct answers, the system would prompt the next question. For incorrect answers, the system first provided a hint (that might include focusing learners’ attention on a specific part of a word or syllable) and then gave the learner another opportunity to an- swer. After one or two wrong attempts, the system provided the correct answer with a brief explanation. Results/Outcomes: The analysis showed that, on average, learners called the system 14.2 days out of 32 days and initiated 81.4 calls over five weeks spending 6.2 minutes on calls. Source: M.A. Madaio et al., “’You Give a Little of Yourself’: Family Support for Children’s Use of an IVR Literacy System,” in ACM SIG- CAS Conference on Computing and Sustainable Societies (COMPASS) (COMPASS ’19), July 3–5, 2019 (Accra, Ghana, and New York, NY, 2019). https://doi.org/10.1145/3314344.3332504 new learning content or preparing for exams). which are particularly important when students Feedback provision can also come from parents or and teachers cannot be together physically in the older siblings acting in coordination with teachers. classroom (Lainchaur 2020, p. 16). Phone calls can Direct calls also allow teachers to assess students’ also help teachers encourage parents to support engagement with remote learning resources, un- learning at home and coordinate with them in derstand their absorption of learning content, and the delivery of learning content to students. In respond to students’ questions. addition to teachers calling students to check in on their learning, another way of connecting stu- Moreover, direct calls serve as an opportunity to dents and teachers remotely is by establishing a provide emotional support and encouragement, teacher hotline or call center, which students can 17 PHONE-BASED FORMATIVE ASSESSMENT 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond Box 1-5. Young 1ove, Botswana About the program: Between February and March 2020, before many countries closed schools due to the COVID-19 pandemic, a phone-based assessment of basic numeracy skills was administered to over 4,500 students in grades 3 to 5. The phone-based assessment was conducted by over 70 assessors who called the households directly. The assessors (mostly former teacher aides) were trained remotely using voice notes and sharing resources via WhatsApp. The program was implemented in nine out of ten regions of Botswana. Content delivery: The sample of 4,500 students was randomly divided into three sub-groups: the first subgroup received weekly text messages followed by a phone call; the second group received weekly text messages only; and the third was a control group. The students in treatment groups received targeted instruction in the form of tailored messages based on their learning level, as determined from the results of a midline survey conducted at the end of week four. For example, students who knew addition received subtraction problems, and students who knew multiplication received divi- sion problems. Assessment procedure: The Annual Status of Education Report (ASER) survey instrument was adapted for phone-based administration. The assessment entailed the following tasks conducted by assessors and students: a) Number operations task: This task involved the assessor reading out loud the numeracy questions in ascending order of difficulty: addition, subtraction, multiplication, and division. It also included fractions and place value questions. b) Timed word problem task: For this, word problems were texted to a student, who was then asked to read them out loud and to solve them. c) Explanation of the solution: Students explained their work during discussions with the assessors to check for understanding. On average, the phone calls with students lasted between 15 and 20 minutes and provided facilitators an opportunity to walk the students through the learning activities sent via SMS and to check for students’ understanding. Results/outcomes: Overall, the study suggested that text messages combined with phone calls can result in substantial learning gains; ‘text message followed by phone call’ led to a 31% increase in the numeracy score. It was also found that ‘text message followed by phone call’ had larger and more cost-effective effects on household engagement in learning, as compared to ‘text message only.’ Source: N. Angrist, P. Bergman, & M. Matsheng, School’s Out: Experimental Evidence on Limiting Learning Loss Using “Low-Tech” in a Pandemic, Working paper no. 28205 (Cambridge, MA: National Bureau of Economic Research, 2021). 18 PHONE-BASED FORMATIVE ASSESSMENT 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond Box 1-6. Pratham, India About the program: A month into the COVID-19-related lockdown, beginning April 8, 2020, Pratham began sending daily activities through SMS to keep children engaged in the learning process. These messages were followed by a phone call to check if the message was accurately understood by the parent and/or child and to provide feedback. The messages were sent to children in more than 12,000 communities across India in 11 regional languages and English. The content was focused on different levels of education (early childhood, primary, and upper primary). Content delivery: Pratham developed an SMS and activity-based curriculum that was aligned with the National Council of Educational Research and Training (NCERT) curriculum for grades K-10. For each level of education, Pratham prioritized essential topics or areas from the curriculum. The list of topics identified is contained in the table below. Level of Education Focus/Subject areas/Skills Targeted Pre-primary Six developmental areas: physical/motor skills, cognitive skills, creativity, socio- emotional skills, early reading, and early math. Primary level Math: number recognition and place value, number operations, measurement, time and calendar, shape (geometry), puzzles/riddles/rhythm. Language: comprehension, creation, and exploration. Upper primary level Science: Nature, heat, matter, sound, light Math: ■ - number & place values ■ - basic operations (addition with or without regrouping, subtraction with or without regrouping, division, multiplication; questions on basic opera- tions are further divided into simple sums and word problems) ■ - advanced math, including fractions, decimals, geometry Language: reading, writing, comprehension, grammar To maintain consistency, the text messages were sent at the same time six days a week. Each day of the week focused on a specific activity. To adhere to the character limit of 67 characters for Indian languages, the messages were divided into three parts (based on difficulty) and were sent one after the other to the child on the same day. Having different levels of questions/activities allowed children to attempt one or two questions/parts based on their level. Over time, the content also moved from simple and generic problems to more challenging application-based questions. Assessment procedure: Use of activity-based messages each day—such as measuring the height of all family members and water consumption—helped to easily assess children. To check for children’s understanding, two approaches were implemented. One involved receiving pictures and videos from the families. This was more common for early childhood and primary grades. 19 PHONE-BASED FORMATIVE ASSESSMENT 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond Second, for upper primary, a list of feedback questions to be asked over the phone was prepared that aimed to assess if the child was able to understand the question, navigate the text for information, and explain how he or she reached the solution. Examples of the feedback questions included asking the child if he can see a pattern in the numbers and asking the child to explain how he was able to solve 713 + 182. Pratham’s approach consisted of active involvement of parents in facilitating learning, so that after sending the SMSes, Pratham staff would call the parents to check if they (1) received the message, (2) understood the message, and (3) could help their child complete the activity contained in the message. If the parents did not receive the SMSes or had trouble understanding them, Pratham field staff and volunteers would help resolve the issues over phone. Before this program, Pratham also partnered with Education Above All to pilot the Digital School Program (DSP) in 2019 with 2,000 learners in 150 villages in Uttar Pradesh. The program aimed to provide access to quality content for English, Math and Language (Hindi). Due to the COVID-19 pan- demic, the focus shifted to the use of phone calls, SMS, radio, TV, and IVR to deliver the content. For the first three weeks of lockdown, facilitators called a student or a group of students (2-4) to deliver lessons. Each facilitator placed 9 calls per day to their group of children assigned to them and the calls lasted about 20 minutes. Starting in the fourth week, facilitators sent out curated activity-based messages and made individual calls to students twice a week to explain the activities, check if they completed the activity, and to clarify any doubts. It was found that over 100 students were in touch with facilitators twice a week and 70% of the students completed the activities sent to them each week. (To learn more about the program, please visit this link.) Source: Pratham, How to Create Learning Content in 160 Characters (SMS) (2021). Retrieved from https://drive.google.com/ file/d/1KAPEfOLASE6qxwsIL-7oTlpL7mTv0J6Q/view Note: For more information, see M. Steinmetz, Mobile Distance & Hybrid Education Solutions (Washington, DC: World Bank, 2021). Retrieved from https://pubdocs.worldbank.org/en/685691598013656403/pdf/WorldBank-EdTech-Team-Knowledge-Pack-MobileDis- tance-HybridEducationSolutions-version2.pdf call to speak with a teacher to ask questions about calls – and their use for the delivery of formative remote learning content or to obtain support assessments. The information in Table 1-1 covers with homework assignments (see boxes 1-5 and aspects related to phone-based assessment imple- 1-6). (See Annex 1 for more examples of learning mentation, requirements and considerations for assessments using SMS, IVR, and direct phone implementation, as well as the advantages and calls.) limitations of each technology. This information is a product of lessons learned from the existing Table 1-1 summarizes the three phone-based tech- experience of using these solutions particularly in nologies discussed above – SMS, IVR, and direct development projects. 20 PHONE-BASED FORMATIVE ASSESSMENT 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond Table 1-1. Functionality of phone-based solutions for formative assessment SMS IVR Direct Calls Assessment Short text-based quizzes Users call a phone number Teachers or instructors procedure are sent to students. to listen to learning content contact students by phone With two-way SMS, students and respond to a series of to go over learning content, can respond via SMS (as with voice-based multiple-choice assess students’ knowledge, the learning assessment, questions. guide students to appropriate learning content can also Answers are selected either learning content, and be delivered through SMS by pressing corresponding encourage their use of the either before or after the keys or by saying the answer learning content. Feedback assessment). out loud. can be provided during the The IVR system processes phone call. responses and may provide voice-based feedback based on the result. Assessment Received regularly— i.e., Accessed at students’ Made sporadically or frequency once a week to several convenience. regularly —can also be messages per day. requested by parents if needed. Solution Cost: The most cost-effective Accessibility: Supports Accessibility: Supports advantages of the three tools. students with visual students with visual Time flexibility: Users can disabilities as well as disabilities as well as respond at their convenience, students/parents with limited students/parents with limited given the method’s literacy. literacy. asynchronous nature. Higher response rate: As a Synchronicity: Synchronous Reminders: Additional result of more inclusive sample, interaction with students messages can be delivered IVR leads to higher response allows for a more to remind students to take the rate compared to SMS. personalized approach, assessment. Language flexibility: Content promotes active engagement can be delivered in multiple of students in assessment languages, including those and learning process, builds with no written form. rapport with the assessor. Infrastructure Access: Works on a basic Access: Works on any type Access: Works on any type requirements mobile phone of phone. of phone. Free SMS: Need SMS Free calls: Need to offer Free calls: Need to offer credits or a free-of-charge phone call credits or a toll- phone credits to teachers for alternative. free number. making calls. Mobile aggregators are Voice actors are needed Digital infrastructure is needed to link different to record the content (in needed to manage students’ carrier networks with SMS different languages). data and feedback. software / platform providers. Software: needed to Switchboard system: Software: needed to administer content and needed if hotlines are used. administer content and record students’ answers. record students’ answers. 21 PHONE-BASED FORMATIVE ASSESSMENT 1. The role of phone-based formative assessment during the COVID-19 pandemic and beyond SMS IVR Direct Calls Community Parent authorization: needed User familiarity: necessary Parent availability: may be considerations to send SMS and for reception for receiving and engaging needed for monitoring the of SMS outside of school with IVR. call between the child and the hours. enumerator. Solution limitations 160-character restriction Possible difficulties due Possible additional burden Non-Latin characters might lack of access to a personal on teachers to make the calls not display properly. phone and longer duration of (sometimes even outside the Phones with small screens engagement. school hours). may create usability issues. Students must be attentive Calls need to be scheduled Hard to track possible and complete assessments and rescheduled to ensure parental interference (e.g. in one sitting—saving isn’t an children’s availability and parents telling children the option. access to phone at the time correct answers). of call. Cannot be used with landline phones. Source: Aker & Ksoll (2015); Lau, Sanders, & Lombaard (2019); and Hortinela (2017). 22 PHOTO BY: © 2016 PAUL PRESCOTT/SHUTTERSTOCK. PHONE-BASED FORMATIVE ASSESSMENT 2. Considerations for implementing phone-based formative assessments Given the potential of the phone-based solutions Given that the use of mobile phones (and partic- described above to support learning continuity ularly basic phones) for formative assessment is in the context of the COVID-19 pandemic, and currently rather limited, it is difficult to refer to a their potential to strengthen the resilience of ed- “gold standard” for the successful implementation ucation systems to future shocks, many countries of these assessments. Therefore, this note uses may be interested in introducing these solutions, examples of international development initiatives particularly to expand access to learning outside that have used the same technologies – SMS, IVR of schools in low-resource contexts. This section and direct calls – to draw key considerations for focuses on factors that may affect the feasibility of implementing phone-based formative assessment. implementing such solutions in certain contexts. Specifically, the section examines eight dimensions, specifying the conditions needed for successful Table 2-1. Considerations for implementation. implementing phone-based formative assessments Each dimension includes several indicators Education system capacity based on their relevance to different phone- based solutions, as well as a set of other factors ■ Learning content and its alignment with that need to be considered in assessing the assessment country’s readiness for integrating phone-based ■ Human resources formative assessment as part of remote learn- ing. To ensure that implementation modalities Technological and logistical considerations for phone-based assessments can be practically evaluated, the factors presented in this section ■ Demographics assume a “snapshot” at which time the feasibility ■ Technology is assessed, recognizing the limitations of data and that efforts and initiatives may be ongoing ■ Service providers that would affect these factors over time. These ■ Financial considerations factors can be revisited and evaluated as addi- ■ Time tional information and opportunities for sup- ■ Local implementer capacity port become available. 25 PHONE-BASED FORMATIVE ASSESSMENT 2. Considerations for implementing phone-based formative assessments Education system capacity phone-based assessment. If the learning content is considered to be of low quality or not aligned Learning content and its alignment with to the learning needs of students, the recipients’ assessment motivation to participate in assessment can be negatively affected. Perhaps the most important factor in the success of phone-based formative assessment relates to the learning content that is being assessed. Given the Human resources formative nature of phone-based assessment solu- tions, assessment items need to be aligned very The critical human resources aspects that need to be closely to the learning resources that are being de- considered for the implementation of phone-based livered in person or remotely to the student. Thus, assessments relate mainly to teachers’ ability to in the context of school closures, existing distance participate in or use the results from the assessment. learning initiatives in the country, as well as the Teachers are at the center of the instruction process, type of technology used in distance learning, are and ideally they should be familiar with the common critical determinants of the feasibility and success misunderstandings students may have regarding of phone-based assessment. the learning content. Therefore, teachers can play a critical role in ensuring learning progress if they Countries that focus on teacher-guided home learn- can understand what students know and provide ing modalities, where teachers have remained in constructive feedback and support. It is essential, contact with students, can have higher take-up and therefore, to engage teachers either in direct imple- engagement in phone-based formative assessment mentation of phone-based assessments or in using than countries where the learning modalities rely the data obtained from these assessments to support on self-paced learning or assisted learning with pa- learning. Likewise, it is important that teachers rental involvement at home, such as TV and/or ra- coordinate with parents, caregivers or older siblings dio learning lessons. Distance learning approaches to deliver feedback indirectly to students and ensure that use any of the three technologies at the center continuity of learning at home. of this guidance note (i.e., SMS, IVR, or direct phone calls) as teacher-guided modalities to reach and in- In order to participate in the implementation of teract with students could increase the likelihood these assessments or use the resulting assessment of a successful remote learning intervention. data, teachers need to have access to smartphones or access to the internet via another device (e.g., a In selecting the learning content to align with the computer or a tablet). Internet access is necessary phone-based formative assessment, it is important to facilitate the efficient remote dissemination of to consider the following factors: learning and assessment content to teachers through applications. In contexts where teachers do not have ■ Awareness and coverage of the learning content. access to smartphones or other internet-connected Any information, quantitative or qualitative, devices, such devices can be provided to them, but in about the accessibility of this learning content these situations special emphasis needs to be placed can be useful. Of course, phone-based formative on configuring and testing procured phones and assessment activities can increase the aware- on training teachers in how to use these devices for ness about and even provide a nudge to using phone-based assessment. the learning content being provided, but if the content is not reaching a critical mass of its tar- Other human resources factors to consider as part of get population, engagement with phone-based the feasibility analysis are: assessment is likely to be minimal. ■ General perception regarding the quality of the ■ Teacher motivation and assessment competencies learning content. Perceptions around quality to engage. Conducting phone-based formative can influence take-up and engagement with assessment by direct calls can take up significant 26 PHONE-BASED FORMATIVE ASSESSMENT 2. Considerations for implementing phone-based formative assessments time in a teacher’s day. The use of incentives may While data on digital literacy may not be available be necessary to motivate teachers to participate. in most countries, adult literacy rates can serve as a In addition, teachers must acquire competencies good proxy for more advanced skills. The percent- in the use of formative learning assessments to age of the population of a given age group that can inform remote instruction strategies and pro- read and write can provide an estimate for digital vide constructive feedback to students. literacy and readiness to use digital devices as part ■ Assessment specialist capacity. Developing, main- of the phone-based assessment process. taining, and scaling up phone-based applications according to local needs is likely to require Other demographic factors to consider: dedicated local technical experts, including specialists familiar with content design, learning ■ Urban population share. In many countries, there assessment design, administration, and analysis, is a sharp divide in access, resources, and oppor- who can ensure compliance with important tunities between urban and rural populations. considerations for validity and reliability of Thus, the share of the population living in urban phone-based assessment results (Luna Bazaldua, areas can provide information about both access Liberman, & and Levin 2021). (For more infor- to technological solutions and digital literacy of mation on reliability and validity consideration, the targeted student population. review our note on psychometric considerations ■ Number of spoken languages. Multilingual for phone-based assessments, which is also part settings can increase the complexity of admin- of the knowledge products developed to support istering phone-based assessments. Language the implementation of phone-based assessment considerations are important for access, costs, activities.) and time. In terms of access, take-up of the as- ■ IT staff capacity. A dedicated IT team is likely to sessment may increase when an initial engage- be needed to support setting up and maintain- ment with caregivers and children occurs in the ing phone-based assessment activities. language they speak at home and assessments can be delivered in that home language. Voice Additionally, for large-scale adoption of phone- options are recommended in a multilingual based formative assessments, it is critical for the setting. IVR solutions are particularly well-suited Ministry of Education to partner with various to such settings, because routing to the appropri- actors, such as mobile service providers and assess- ate language can be easily set up and does not ment experts, to build their capacity and develop require synchronous matching of respondents’ sufficient fluency in the software used to collect home language to enumerators with fluency in and analyze assessment data. that language. Cultural and behavioral aspects can affect the ef- Technological and logistical fective implementation of phone-based formative considerations assessment. These aspects include students’ access to mobile phones, parents’ views on phone-based remote learning, parents’ attitudes towards phone Demographics sharing for learning and assessment at home, gender gaps in phone usage, and parental attitudes One critical constraint related to access and usage towards mobile phone use by boys and girls. of mobile phones can be the skills and competen- cies required to use these devices for the purposes Gender roles. In many countries, due to religious of the assessment. Parental involvement is essential and cultural constraints, girls or women are not for remote learning and the formative assessment allowed to own a mobile phone. This may affect process, particularly for younger children. Digital their participation in various remote learning literacy of users and caregivers impacts the take-up and assessment initiatives. In such cases, working and effectiveness of phone-based assessments. through female operators, encouraging the use of 27 PHONE-BASED FORMATIVE ASSESSMENT 2. Considerations for implementing phone-based formative assessments basic phones, making an effort to inform parents While the scope of this guidance note is on the use of about how this will help their children’s education, feature phone solutions, the technology component and building trust of the community through im- may also consider the use of more advanced phones. plementing partners by establishing transparent For instance, the penetration rate of smart feature communication is critical to success. phones and smartphones is steadily growing around the world, in part due to the increase of low-cost smartphone options in developing countries. Smart Technology feature phones and smartphones open doors to more efficient and scalable phone-based administration of The two most important factors for evaluating tech- learning assessments with the use of messaging and nological capacity are a country’s mobile phone learning management applications. Some of these penetration and network coverage. Implementing applications may also facilitate asynchronous and phone-based assessments requires students to have synchronous communication and learning-content access to mobile phones at home. Mobile phone pen- sharing between a teacher and multiple students etration can be measured as mobile cellular subscrip- working together remotely. tions (per 100 people), which means subscriptions to a public mobile telephone service that provide access In contexts where mobile phone penetration is low, to the Public Switched Telephone Network (PSTN) education authorities can try to reach agreements using cellular technology, based on data collected by with mobile providers to provide free or subsidized the International Telecommunication Union (ITU) as mobile phones for students (IICBA 2020), as in the part of the World Telecommunication/ICT Indicators case of China, where the Ministry of Education pro- Database.7 vided families with free SIM cards.8 Other countries, like Pakistan and Afghanistan (see Boxes 1-1 and 1-2 Given the prevalence of sharing mobile phones and in Section 1), followed a similar approach to ensure connections, particularly in low-resource contexts, it remote learning continuity. While it would be most is also important to consider indicators of household advantageous for students to possess their own ownership of phones (i.e., the proportion of house- device, previous experiences indicate that personal holds with mobile-cellular telephones) and usage of device ownership is not essential for effective remote mobile phones by individuals (i.e., the proportion assessment delivery: it is relatively easy for students of individuals using a mobile phone), as collected without mobile phones to share devices or SIM cards by ITU. The mobile network coverage indicator, also in order to access learning experiences (GSMA 2012). collected by ITU, is the percentage of inhabitants However, sharing devices with others can introduce that are within range of a mobile cellular signal, irre- disparities in access as well as difficulties in reach- spective of whether or not they are subscribers. This ing the right students at regular times. Moreover, indicator is sometimes broken down by network type phone-sharing solutions require extra planning and (2G/3G/4G/5G, as is done by GSMA Mobile Connec- flexibility in the delivery of the assessment in order tivity Index). This is an important indicator for the to reach the target student. When multiple members number of students that can potentially be reached in a single household use the same device, unique through mobile phone technologies regardless of identifiers are often necessary to attribute responses device ownership. to a particular student. 7 Readers are encouraged to also review mobile phone penetration information from national statistical agencies to validate penetration rates by country and disaggregated information on phone access by key demographic factors. 8 https://worldbankgroup.sharepoint.com/sites/gsg/Innovation/Documents/Education_continuty_stories_under_COVID_lockdown_compiled_ by_region.pdf 28 PHONE-BASED FORMATIVE ASSESSMENT 2. Considerations for implementing phone-based formative assessments Other technological factors to consider are: important to reach disadvantaged populations that cannot afford to pay for phone calls and SMS. ■ Access to electricity. The proportion of the pop- When toll-free numbers cannot be easily set up, a ulation with access to electricity can indicate missed-call solution may be an option for IVR and whether the use of phones is viable for the direct phone calls, wherein a call is placed without targeted intervention, as phones require a rela- a connection and the caller receives an immediate tively stable supply of electricity to be charged call-back at no charge (Ozonetel 2020). and used. ■ Mobile cellular subscriptions and mobile connec- Other service provider factors to consider: tions, including those that are prepaid. The num- ber of mobile cellular subscriptions and mobile ■ Frequency of internet outages/shutdowns in the connections that are prepaid can indicate actual country. Frequent internet shutdowns due to utilization of mobile phones as well as the af- infrastructure issues or government decisions fordability of services. can affect the service providers’ ability to deliver ■ Mobile accounts for financial transactions. The an effective service and may compromise the percent of adults who used mobile phones to intervention. conduct a financial transaction shows utiliza- tion of mobile phone services and can provide insights into the likelihood of take-up of phone- Financial considerations based development solutions. Financial considerations will impact the feasibility and long-term sustainability of a phone-based as- Service providers sessment activity. Service providers and software platforms are also In terms of technology-related costs, selection of a key component to successful delivery of educa- the most appropriate phone-based solution may tional applications and should be evaluated in the depend on the relative costs of different phone- context of usability, accessibility, and affordability. based technologies. In particular, the following costs could be considered in these decisions: At a minimum, the country must have at least one operational mobile aggregator that has ■ Average cost per SMS. It is important to consider capabilities to deliver at least one of the tar- the potential costs of both sending and receiving geted solutions (i.e., SMS, IVR, direct phone call SMS, and to make sure that students’ families are services). To assess the capacity of a mobile compensated for any costs involved in two-way aggregator, it is necessary to review its capabil- SMS communications. ities, as certain features, such as two-way SMS, ■ Average per-minute cost of different types of mo- may not be available in mobile aggregators in bile phone calls. The cost of mobile phone calls is all countries. Other considerations may include most relevant for direct calls and IVR solutions, how easy it is for users to purchase SIM cards in and includes the cost of 1-minute call time both the country, including access and identification in-network as well as out-of-network. requirements. Other financial factors to consider: Another key factor to consider is whether toll-free numbers and short codes9 are available or can be ■ Average cost of mobile data for 1GB. Affordability set up easily through agreements with telecom of mobile data can be important for teachers’ or providers in the country. Toll-free numbers are implementing partner’s use of software tools for 9 Short codes are short digit numbers (significantly shorter than telephone numbers) used to send SMS and abbreviate dialing. Short codes facilitate reading and remembering telephone numbers. 29 PHONE-BASED FORMATIVE ASSESSMENT 2. Considerations for implementing phone-based formative assessments managing SMS or calls. Countries like India and One logistical factor that can significantly impact Israel have a very low cost of data due to the con- the time frame for implementation is access to siderable market competition in each country. a database of caregivers’ or students’ phone num- On the other hand, countries like Malawi, Benin, bers. In an ideal scenario, such a phone number and Chad present the highest mobile data costs database already exists and is managed by the and very small market competition for telecom Ministry of Education, either centrally or at the companies that offer mobile data services. To in- school level. Currently, this is not the case for crease access to online resources for teachers or many countries, which means phone numbers implementing partners, governments can either need to be collected or requested from other min- create or facilitate access to online educational istries (e.g., Ministry of Information, Communica- resources by partnering with mobile service tion and Technology), international development providers to not charge individual users for organizations, or NGOs where available. Prior to data usage on specific websites or platforms. For any future school closures, phone numbers could instance, the Ministry of Education in Colombia be collected by students’ teachers either as part developed Aprender Digital, an online learning of teacher-parent interactions or as a form sent platform to access educational resources. In col- home with students. In addition to the existence of laboration with the Ministry of Technology and phone number data, the accuracy of such data can Communications, a decree was published for be another limiting factor, particularly in contexts mobile operators to provide zero-rating access 10 where people change their phone numbers often to the Aprender Digital portal. Also, to reach areas or use shared phones. with low or no connectivity, digital devices with preloaded educational materials were distribut- Other time-related factors to consider: ed to households by the government. A similar initiative was implemented in Pakistan (OECD ■ Implementation timeline. Depending on existing et al. 2021) technical capacity and the countries’ previous experience with the use of phone-based tech- nologies, each phone-based assessment solution Time (i.e., SMS, IVR, direct phone calls) would have dif- ferent startup timelines. It is important to take Phone-based assessment activities can be essential these into account when designing solutions for to support learning continuity during school clo- coping with crisis situations. Still, it is important sures. In such situations, time is of the essence. The to note that once set up, phone-based formative main question for implementation feasibility of assessments can support learning outside of the phone-based assessments is the expected duration classroom even during normal times and/or can of school closures and the delivery of distance learn- be ready to be deployed during the next school ing. Notably, gradual reopening using hybrid models closure, thereby raising crisis preparedness of of instruction can justify continuation of distance the education system. learning delivery and thus continuing phone-based formative assessments. A full reopening of schools may require rethinking the design of the phone- Local implementer capacity based formative assessment, for example switching to using it as a complementary intervention to sup- Much of the success of the project implementation port learning recovery and targeting it to students will depend on selecting the right implementing with observed learning losses. partner (e.g., NGO or development agency). In the process of identifying implementing partners, it is critical to consider whether the candidates have 10 Zero-rating is a practice of providing cost-free access to certain websites or services, i.e. mobile service provider does not charge the user for data usage on specific websites. It is sometimes referred to as ‘toll-free data’ or ‘sponsored data’) 30 PHONE-BASED FORMATIVE ASSESSMENT 2. Considerations for implementing phone-based formative assessments PHOTO BY: © 2020 PHOTOADVENTURE STUDIO/SHUTTERSTOCK. previous experience with implementing phone- such expertise can accelerate the startup phase based interventions or assessments. Other valu- of the implementation. Likewise, understanding able attributes include current role in delivering candidates’ ICT-related capacity can provide valu- distance learning interventions. It is particularly able insight into their ability to set up and test important to identify implementers’ previous ex- the system, configure phones, or carry out any perience working with mobile network operators additional ICT related activities for the duration and mobile service providers (e.g., aggregators), as of the project. 31 PHOTO BY: © 2020 KK ZUMBUL/SHUTTERSTOCK. PHONE-BASED FORMATIVE ASSESSMENT Conclusion Conclusion The assessment of student learning is a fundamental feedback mechanism in education, providing information on learning progress, areas of constraint, and uncovering what additional learning resources need to be provided. Remote formative assessment can support learning continuity as a response to prolonged school closures due to COVID-19 and improve the quality of education over- all. Low-cost mobile technologies also offer a long-term opportunity to support learning outside of the classroom and to contribute to education systems’ resilience to future shocks, such as natural disasters and unrest in fragile contexts. This guidance note analyzes the use of Short Message Service (SMS), Interactive Voice Response (IVR), and direct phone calls to students as three such solutions. It explores factors that may affect the feasibility of implementing such solutions in a low-resource context, and specifies conditions that may be needed for successful implementation of phone-based formative assessment solutions. Given the formative nature of the assessment, close alignment of learning resources to the as- sessment intervention is important. In the context of school closures, existing distance learning initiatives in the country as well as the type of technology used in distance learning are critical determinants of the feasibility and success of phone-based assessment. An additional critical con- sideration pertaining to the capacity of the education system is the teachers’ ability to participate in or use the results from the assessment. There are also technological and logistical considerations in assessing the feasibility of implement- ing phone-based assessments. It is essential to evaluate the technological capacity of a country, mainly mobile phone penetration and network coverage. In addition, a critical constraint is the access and usage of mobile phones, which includes a county’s mobile phone penetration rate as well as the skills and competencies required to use these devices for the purposes of the assess- ment. Service providers and software platforms are also a key component to successful delivery of educational applications. At a minimum, a country must have at least one operational mobile aggregator that has the capabilities to deliver at least one of the targeted solutions (i.e., SMS, IVR, direct phone call services). Financial and time-related considerations will also impact the feasibility and long-term sustain- ability of a phone-based assessment activity. Finally, much of the success of the project implemen- tation will depend on selecting the right implementing partner, one that has relevant previous experience with these interventions. 33 PHONE-BASED FORMATIVE ASSESSMENT References References Aker, J. C., & Ksoll, C. (2015). Call me educated: Evidence from a mobile monitoring experiment in Niger. Center for Global Development Working Paper no. 406. Angrist, N., Bergman, P., Brewster, C., & Matsheng, M. (2020). Stemming Learning Loss During the Pandemic: A Rapid Randomized Trial of a Low-Tech Intervention in Botswana. Available at SSRN 3663098. Cable.co. (2021). Worldwide Mobile Data Pricing 2021. https://www.cable.co.uk/mobiles/worldwide-data-pricing/ Clarke, M. (2012). What Matters Most for Student Assessment Systems: A Framework Paper. Washington, DC: World Bank. Clarke, M., & Luna-Bazaldua, D. (2021). Primer on Large-Scale Assessments of Educational Achievement. Washington, D.C.: World Bank. Diedhiou, A., Gilroy, K. E., Cox, C. M., Duncan, L., Koumtingue, D., Pacqué-Margolis, S., Fort, A., Settle, D., & Bailey, R. (2015). Successful mLearning pilot in senegal: Delivering family planning refresher training using inter- active voice response and SMS. Global Health: Science and Practice 3(2), 305–321. https://doi.org/10.9745/ GHSP-D-14-00220 Global System for Mobile Communications Association [GSMA]. (2012). Shaping the Future—Realising the Potential of Informal Learning Through Mobile. MasterCard Foundation. https://www.gsma.com/mobilefordevelopment/ wp-content/uploads/2012/05/mLearning_Report_230512_V2.pdf Global System for Mobile Communications Association [GSMA]. (2021). The Mobile Economy Sub-Saharan Africa 2021. GSM Association. https://www.gsma.com/mobileeconomy/wp-content/uploads/2021/09/GSMA_ME_ SSA_2021_English_Web_Singles.pdf Hortinela, C. (2017). Comparing the Benefits of Automated Phone Calls (IVR) versus SMS Campaigns. EngageSpark. https://www.engagespark.com/blog/blogcomparing-benefits-automated-phone-calls-ivr-versus-sms-cam- paigns/ IICBA. (2020). Conducting Learning Assessment in Africa During School Closures: Meeting Report (IIC/2020/ED/2). UNE- SCO International Institute for Capacity-Building in Africa. Lainchaur, L. M. (2020). Guidance on Distance Learning Modalities to Reach All Children and Youth During School Closures: Focusing on Low-and No-Tech Modalities to Reach the Most Marginalized. UNICEF Regional Office for South Asia. https://www.unicef.org/rosa/sites/unicef.org.rosa/files/2020-06/Guidance%20Continuity%20of%20 Learning%20during%20COVID-19%20-%20Reaching%20All%20Children%20ROSA.pdf Lau, C. Q., Cronberg, A., Marks, L., & Amaya, A. (2019). In search of the optimal mode for mobile phone surveys in developing countries—a comparison of IVR, SMS, and CATI in Nigeria. Survey Research Methods, 305-318. https:// doi.org/10.18148/SRM/2019.V13I3.7375 Lau, C. Q., Sanders, H., & Lombaard, A. (2019). Questionnaire design in short message service (SMS) surveys. Field Methods 31(3), 214–229. 34 PHONE-BASED FORMATIVE ASSESSMENT References Luna-Bazaldua, D., Liberman, J., and Levin, V. (2021). Assessing outside of the “classroom box” while schools are closed: The potential of phone-based formative assessments to support learning continuity. World Bank Blogs. Retrieved from: https://blogs.worldbank.org/education/assessing-outside-classroom-box-while-schools-are- closed-potential-phone-based-formative Liberman, J., Levin, V., Luna-Bazaldua, D., & Harnisch, M. (2020, April 1). High-stakes school exams during COVID-19 (Coronavirus): What is the best approach? World Bank Blogs. https://blogs.worldbank.org/education/high-stakes- school-exams-during-covid-19-coronavirus-what-best-approach OECD, World Bank, Global Education Innovation Initiative, and HundrED. (2021). Education Continuity Stories – Worldwide Initiatives to Ensure Education During the COVID-19 Lockdown: A Compilation of Experiences Across Regions. Retrieved from: https://worldbankgroup.sharepoint.com/sites/gsg/Innovation/Documents/Educa- tion_continuty_stories_under_COVID_lockdown_compiled_by_region.pdf Ozonetel. (2020). Pratham Books used Ozonetel’s “Missed call” feature along with a radio integration to provide story- telling to over 58528 children in 2 languages across 4 states of India. Retrieved from https://ozonetel.com/blog/ kb/pratham-books/ Perry, L. (2013). Review of Formative Assessment Use and Training in Africa. International Journal of School & Edu- cational Psychology 1(2), 94–101. https://doi.org/10.1080/21683603.2013.789809 UNESCO. (2021). Education: From Disruption to Recovery. Paris: UNESCO. Retrieved from https://en.unesco.org/ covid19/educationresponse#schoolclosures World Bank. (2012). 2012 Information and Communications for Development: Maximizing Mobile. https://openknowl- edge.worldbank.org/handle/10986/5996 World Bank. (2020). How Countries Are Using Edtech (Including Online Learning, Radio, Television, Texting) to Support Access to Remote Learning During the COVID-19 Pandemic. Retrieved from https://www.worldbank.org/en/topic/ edutech/brief/how-countries-are-using-edtech-to-support-remote-learning-during-the-covid-19-pandemic World Bank. (n.d.) Mobile cellular subscriptions (per 100 people) - Low income. Retrieved from https://data.world- bank.org/indicator/IT.CEL.SETS.P2?end=2018&locations=XM&name_desc=false&start=1992&view=chart Additional resources for consideration World Bank. Mobile Distance & Hybrid Education Solutions. Washington, DC: World Bank, 2020. DC. Retrieved from: https://pubdocs.worldbank.org/en/685691598013656403/WorldBank-EdTech-Team-Knowledge-Pack-MobileDis- tance-HybridEducationSolutions-version2.pdf World Bank. Knowledge Pack (KP): Devices for Education. Washington, DC: World Bank, 2021. Retrieved from: https://worldbankgroup.sharepoint.com/:p:/r/sites/gsg/Innovation/_layouts/15/Doc.aspx?sourcedoc=%7B- C1B59B23-5C9B-4DFB-A9F6-C60DE2ED516A%7D&file=WorldBank_Education_Devices_Knowledge_Packv07_ APRIL2021.pptx&action=edit&mobileredirect=true 35 PHONE-BASED FORMATIVE ASSESSMENT Annex Annex 1. Examples of phone-based formative assessments using SMS, IVR and direct phone calls Country Region Project Name Implementor/ Year of Short Interactive Phone Grade/Level Skills assessed Assessment procedure Link/Source Donor implementation Message Voice Calls of education Service Response (SMS) (IVR) Afghanistan SAR Mobile Literacy Afghan Institute 2011 Yes No No Adult educa- Literacy Each student received a handset (that ran on the https://uil.unesco.org/ Project of Learning (AIL), tion standard 2G system), a phonecard, and a notebook. case-study/effective-prac- UNESCO, U.S. Teachers sent daily texts to the students, who read the tices-database-litbase-0/ Afghan Women’s incoming message and responded via return text mes- mobile-literacy-programme-af- Council, Creating sage-demonstrating reading comprehension and writ- ghanistan-afghanistan Hope International ing skills. Students attended twice-monthly evaluation sessions to monitor progress and receive assistance, in addition to attending classes. Primarily, the messages comprised three types of questions: 1) fill in the blank type sentences which students had to rewrite with the word filled in; 2) open-ended questions to facilitate critical thinking and writing skills; and 3) reordering the jumbled sentence into its correct structure to practice comprehension and grammar. Africa (Rwanda, AFR Eneza Educa- No information 2013-present Yes No No Primary and All subjects Rwanda: quizzes were aligned to the Rwandan cur- https://enezaeducation.com/#im- Kenya, Ghana, Ivo- tion Secondary riculum. pact ry Coast) Kenya: Mock papers were designed to be as much like the real exam as possible (KCSE and KCPE exams). It is comprehensive, meaning that it tests all that has been taught in the previous classes. Africa (Ugan- AFR Educate! No information No information Yes No No Secondary ed- Socioemotion- Learning assessments are provided through SMS https://www.experienceeducate. da, Rwanda, and ucation al skills (such as org/covid-19-response Kenya) communication, teamwork, creativ- ity, and grit) and enterprenuership training 36 PHONE-BASED FORMATIVE ASSESSMENT Annex Country Region Project Name Implementor/ Year of Short Interactive Phone Grade/Level Skills assessed Assessment procedure Link/Source Donor implementation Message Voice Calls of education Service Response (SMS) (IVR) Bangladesh SAR Bangladesh Vir- Catholic Relief No information Yes No Yes Grade 9-High- Literacy SMS was used to answer questions during lectures by http://citeseerx.ist.psu. tual Interactive Services/Niger, er Education students. They also were able to provide interaction edu/viewdoc/download?- Classrooms (CRS/Niger), Tufts through voice calls. doi=10.1.1.625.5535&rep=rep1&- (BVIC) University, the type=pdf University of Ox- ford, and Systéme d’Information sur les, Marchés Ag- ricoles (SIMA), Örebro Universi- ty (Sweden) and Bangladesh Open University, with, support from the Swedish Program for ICT in Develop- ing Regions (Spi- der) at Stockholm University Bangladesh SAR BBC Janala BBC in partner- 2008-2013 No Yes No Adult educa- English (Literacy) No information https://www.gsma.com/mobil- ship with the UK tion efordevelopment/resources/ Department for In- bbc-janala/ ternational, Devel- opment (DFID) Botswana AFR No information Center for Glob- 2020-present Yes No Yes Grade 5 Numeracy Student learning outcomes were collected by directly https://www.csae.ox.ac.uk/mate- al Development, assessing the child or children over the phone: rials/papers/csae-wps-2020-13. University of Ox- pdf 1) Number operations task: Reading out loud the nu- ford, University meracy questions by assessor in ascending order of of Colombia, RTI difficulty: addition, subtraction, multiplication and di- International and vision. Young 1ove 2) Timed word problem task: Texting word problems to the student and asking them to read it out loud and solve them. 3) Explanation of the solution: the students explaining their work to check for understanding. 37 PHONE-BASED FORMATIVE ASSESSMENT Annex Country Region Project Name Implementor/ Year of Short Interactive Phone Grade/Level Skills assessed Assessment procedure Link/Source Donor implementation Message Voice Calls of education Service Response (SMS) (IVR) Côte d’Ivoire AFR Allô Alphabet Jacobs Founda- Feb-May 2019 No Yes No Grade 5 Literacy: Target- IVR plays a pre-recorded audio message with the https://dl.acm.org/doi/ tion Fellowship, ed phonological question and response options. Questions have ei- pdf/10.1145/3314344.3332504 Dec 2019-Feb Institute of Educa- awareness and ther two or three responses, depending on the type of 2020 tion Sciences, U.S. print-sound map- question, with most questions having three options. Department of ping, gradually Students receive feedback upon responding; correct Education, Eneza increasing in com- responses prompt new questions, while incorrect Education plexity and diffi- ones provide a hint and an opportunity to respond culty, from simple again to the question, focusing their attention on a phoneme and syl- particular part of the word or syllable. If incorrect, they lable awareness, to receive the same question again, with a hint message mappings between explaining the concept or prompting the student to letters, words, and focus their attention on a particular part of the word sounds. In this or syllable. After one or two wrong attempts (depend- study, users only ing on the question type), the answer is provided, with experienced the a brief explanation. Questions are given to students first 2 of 8 units, based on their mastery of the concepts. which ask learners to match words or syllables that share a sound or a com- bination of sounds, to choose the word or sound that “does not belong”, and others. Ghana AFR Viamo Viamo No information Yes No Yes No informa- English (Literacy) Tested children’s speaking skills through a series of https://viamo.io/case-studies/ tion audio quizzes. classroom-palm-hand-lessons- via-mobile/ India SAR No information Pratham 2020 Yes No No All literacy, numeracy, Conducted through activities sent to children via SMS. https://drive.google.com/ and life-skills Each SMS targeted a competency. file/d/1KAPEfOLASE6qxw- sIL-7oTlpL7mTv0J6Q/view India SAR No information Central Square 2020 No No Yes Ages 2-11 early grade numera- After asking the parents a few background questions, https://www.centralsquare- Foundation (CSF) cy skills parents were requested to pass on the phone to the foundation.org/wp-content/up- and Saarthi Edu- child for administering the phone-based assessment. loads/2021/06/ASSESSMENTS- cation Enumerators were trained to and were tasked with IN-THE-TIMES-OF-COVID-19.pdf calling the parents, seeking their consent for the sur- vey and for assessment of the child, and it was ex- plained to them that these assessments would not ad- versely affect the child in any way. 38 PHONE-BASED FORMATIVE ASSESSMENT Annex Country Region Project Name Implementor/ Year of Short Interactive Phone Grade/Level Skills assessed Assessment procedure Link/Source Donor implementation Message Voice Calls of education Service Response (SMS) (IVR) India SAR Digital School Pratham and Ed- 2019-present Yes No Yes Secondary Language (English, Remote assessments were conducted for students https://resources.educa- Program ucation Above All education Hindi), Mathe- who could only read words. The assessment tool was tionaboveall.org/sites/de- (EAA) (Grade 8 and matics sent to students as an image via WhatsApp, and facili- fault/files/ngo/attach- 10) tators assessed the students over a phone call. ments/2020-08/09.%20Pratham. pdf Kenya AFR M-Shule No information No information Yes No No Grades 4-8 No information Student progress is tracked through continuous as- https://resources.educa- sessment. Students interact with the platform in a tionaboveall.org/sites/de- series of responses to questions that determine how fault/files/ngo/attach- the learner is progressing through a learning path- ments/2020-08/07.%20 way. M-Shule uses a notification system to encourage M-Schule.pdf learners to return to the application. Kenya AFR No information Center for Rein- October-November No No Yes Grade 3, 5, Numeracy/mathe- A phone-based assessment took place in December https://www.edworkingpapers. venting Public 2020 (7 weeks) and 6 matics 2020 with 2,552 students. It was conducted by hired com/sites/default/files/ai21- Education (CRPE), enumerators and consisted of 14 questions, covering 432.pdf Innovations for two predetermined sections on (1) core numeracy, Poverty Action and (2) curriculum-aligned standards based on what (IPA), NewGlobe students would have been learning had schools been Education, Brown open and what they were supposed to be learning as University part of the phone-based interventions. The curricu- lum-aligned questions varied across grades while the core numeracy section and survey questions were the same across grades. In addition, in-person assessments were conducted in February and March 2021. Kenya AFR ElimuLeo Precision Devel- 2020 (two weeks) Yes No No Primary edu- Numeracy (Number The two-week pilot sent trial math exercises to chil- https://2uy7xawu7lg- opment (PxD), cation operations tasks) dren to assess their skill level and, thereafter, sent 2zqdax41x9oc1-wpengine. Young 1ove, IPA, them math exercises pitched to their abilities. As netdna-ssl.com/wp-content/ and the Kenya In- children master exercises of a certain difficulty, they uploads/2020/11/2020_Q3_ stitute for Curricu- receive more challenging exercises. Conversely, if a PAD-report.pdf lum Development child is struggling with their current set of exercises, (KICD) they receive easier exercises. 39 PHONE-BASED FORMATIVE ASSESSMENT Annex Country Region Project Name Implementor/ Year of Short Interactive Phone Grade/Level Skills assessed Assessment procedure Link/Source Donor implementation Message Voice Calls of education Service Response (SMS) (IVR) Nepal SAR Low-tech In- Ministry of Educa- November Yes No Yes Grades 3-5 Numeracy As a part of the phone survey (Baseline assessment), https://openknowledge. tervention for tion, Science and 2020-present a short Math test was administered (5 questions). This worldbank.org/bitstream/ Foundation- Technology, Local test is based on the ASER test and was adapted for handle/10986/35384/ al Education Govt, Teach for phone delivery (Angrist et. al 2020). The test includes Learning-in-the-Time-of-COVID- (LIFE) Nepal place value, addition, subtraction, multiplication and 19-Insights-from-Nepal.pdf?se- division questions that Grade 2 students are expect- quence=1&isAllowed=y ed to answer. As per Angrist et al. (2020), student learning outcomes were collected by directly assessing the child or chil- dren over the phone: 1) Number operations task: Reading out loud the nu- meracy questions by assessor in ascending order of difficulty: place value, addition, subtraction, multiplica- tion and division. 2) Timed word problem task: Texting word problems to the student and asking them to read it out loud and solve them. 3) Explanation of the solution: the students explaining their work to check for understanding. Niger AFR Alphabétisa- Catholic Relief 2008-2010 Yes No No Adult educa- Literacy and nu- No information on assessment procedure but the https://uil.unesco.org/ tion de Base par Services, Tufts tion meracy students were tested (1) in the beginning (January of case-study/effective-prac- Cellulaire (ABC): University, Oxford each year), (2) end of the course, and (3) the following tices-database-litbase-0/ Mobiles 4 Lit- University, UC-Da- January (sought to determine whether the acquired lit- alphabetisation-base-cellu- eracy vis Hitachi Fonda- eracy and numeracy skills had endured over time) laire-abc-mobiles-4 tion CITRIS Nigeria AFR EdoBest Pro- Ministry of Edu- 2018-present Yes No No Primary and No information Text-based assessments are distributed via SMS or https://blogs.worldbank.org/ed- gram cation, Universal Junior Sec- WhatsApp. The state is developing interactive quizzes ucation/learning-despite-crisis- Basic Education ondary delivered to parents’ mobile phones. These quizzes case-edo-state-nigeria Board (SUBEB), are aligned with the state curriculum, and they inter- LGEAs, World relate with other resources such as digital storybooks Bank and lesson guides delivered to parents. 40 PHONE-BASED FORMATIVE ASSESSMENT Annex Country Region Project Name Implementor/ Year of Short Interactive Phone Grade/Level Skills assessed Assessment procedure Link/Source Donor implementation Message Voice Calls of education Service Response (SMS) (IVR) Nigeria AFR Northern Edu- Creative Associ- 2015-present No Yes Yes Grades 1-3 literacy, numeracy, No information Hale, J., & Udeh, A. (2020, June cation Initiative ates International, and life-skills 8). NEI Plus COVID-19 Remote Plus (NEI+) USAID, Education Learning Response. BEC EdTech Development Cen- Webinar. ter, Florida State University, Over- seas Strategic Consulting, Value Minds, Civil Soci- ety Action Coali- tion on Education for All, Reading Association of Ni- geria, Federation of Muslim Wom- en’s Associations in Nigeria. Nigeria AFR Addressing Edu- FHI 360, USAID, 2018-present Yes Yes Yes Out of school No information SMS and IVR-based practice modules aimed at par- Koester, E. (2020, June 8). Ad- cation in North- Save the Children, children ents alongside their children dressing Education in North- east Nigeria Viamo, Nigerian east Nigeria (AENN): COVID Re- (AENN) Government, Lo- sponse. BEC EdTech Webinar. cal NGOs Nigeria AFR EdoBest No information 2018-present Yes No No K-12 Literacy and nu- Formative quizzes sent via SMS or WhatsApp. http://pubdocs.worldbank.org/ meracy en/685691598013656403/ WorldBank-EdTech-Team-Knowl- edge-Pack-MobileDistance-Hy- bridEducationSolutions-version2 Pakistan SAR Broad Class: Lis- The Communi- 2014-present No Yes No Kindergarten, Literacy and nu- During short pauses built into radio scripts, teachers https://www.globalinnovationex- ten to Learn cators (Pvt.) Lim- Grade 1 meracy and students participate in program, reacting verbally change.org/innovation/broad- ited, POWER99 and physically to questions and exercises posed by class-listen-to-learn Foundation, Mary- radio characters. mount University 41 PHONE-BASED FORMATIVE ASSESSMENT Annex Country Region Project Name Implementor/ Year of Short Interactive Phone Grade/Level Skills assessed Assessment procedure Link/Source Donor implementation Message Voice Calls of education Service Response (SMS) (IVR) Pakistan SAR Bunyad: Mo- Punjab Depart- 2009-present Yes No No adolescent Literacy Monitoring the learners’ participation in the mo- https://uil.unesco.org/ bile-Based Post ment of Literacy girls bile-based programme is done by the web-based sys- case-study/effective-prac- Literacy Pro- and Non-Formal tem which is used to send text messages to the learn- tices-database-litbase-0/ gramme Basic Education, ers. In addition, newly literates respond to questions/ mobile-based-post-literacy-pro- Lahore; BUNYAD multiple-choice-questions (MCQs) or tests sent by gramme-pakistan Foundation, La- SMS, and the results of these tests are summarised hore; Dhaka Ah- and recorded in the web-based system. Learners also sania Mission Pa- report to literacy centres on regular bases. A monthly kistan, Islamabad; exam is given to learners at learning centres to track Mobilink Pakistan; their retention rate and the development of their liter- Nokia Pakistan acy skills. Papua New Guin- EAP SMS Story Australian Govern- 2013 Yes No No Grades 1-2 Literacy Project included a baseline reading assessment, mid- https://blogs.worldbank.org/ ea ment, Voluntary point lesson and classroom observations and an end- edutech/promoting-litera- Services Over- point reading assessment. cy-mobile-phones-rural-pap- seas, Papua New ua-new-guinea Guinea Depart- ment of Education Sierra Leone AFR Rising Academy Rising Academies May 2020-August Yes No Yes Primary Numeracy- count- Learning assessments are provided through SMS for https://www.cgdev.org/pub- Network 2020 ing and simple a small subsample. For the rest, it’s an in-person as- lication/teaching-and-test- arithmetic sessment. ing-phone-pandemic Literacy- vocabu- lary, spelling, and aural comprehen- sion. United States NA Mobile Learning Cell-Ed, Centro 2014-present Yes Yes No Adult educa- Literacy, Job-skills SMS or IVR quizzes are prompted to participants. A https://edtech.worlded.org/ for All (Cell-Ed) Latino in Los An- tion correct response to the question triggers the begin- wp-content/uploads/2018/12/ geles ning of the next micro-module, whereas an incorrect Cell-Ed-Report-final-012319-2-1. response leads to a repetition of the same micro-mod- pdf ule until the user succeeds. Content is delivered at the learner’s pace and is able to track their responses and scores. Zambia AFR Makhalidwe USAID, Creative 2015-2016 Yes Yes Yes Grades 2-3 Literacy Assessment was done by phone through a question- https://pdf.usaid.gov/pdf_docs/ Athu Associates naire that tracked how many days did parents and chil- PA00SZJS.pdf dren use the tools to read together. 42 PHONE-BASED FORMATIVE ASSESSMENT Annex Country Region Project Name Implementor/ Year of Short Interactive Phone Grade/Level Skills assessed Assessment procedure Link/Source Donor implementation Message Voice Calls of education Service Response (SMS) (IVR) Zimbabwe AFR Viamo World Vision, Ak- No information No Yes No ECCE Literacy and nu- Using simple IVR formative assessments & retention https://pubdocs.worldbank.org/ tion Deutschland meracy surveys. For example, a nursery rhyme is told about en/685691598013656403/pdf/ Hilft, Save The what sounds animals make, and then the student is WorldBank-EdTech-Team-Knowl- Children, ECHO asked, “What sounds does this animal make?” so they edge-Pack-MobileDistance-Hy- & MoE check via these mini assessments if students have lis- bridEducationSolutions-ver- tened and are still present. sion2.pdf https://www.wvi.org/stories/zim- babwe/viamo-ensures-educa- tion-continues-midst-crises 43 SUPPORTED WITH FUNDING FROM THE GLOBAL PARTNERSHIP FOR EDUCATION