Publication: Psychometric Considerations for Implementing Phone-Based Learning Assessments
Loading...
Published
2022
ISSN
Date
2022-06-13
Author(s)
Editor(s)
Abstract
The Coronavirus disease 2019 (COVID-19) pandemic has had a significant impact on education worldwide, affecting whether and how students learn. World Bank estimates and initial analysis at the country level indicate that school closures and interruptions to in-person education are likely to negatively impact student learning. Moreover, these estimates predict that students in low- and middle-income countries will suffer from more significant learning losses than students in high-income countries, worsening the existing global learning crisis. As part of their education response to the pandemic, many countries have provided learning content to students at home through various means, including through the internet via devices such as computers, tablets, and smartphones, as well as through television, radio, printed materials, and feature (basic) phones. With schools closed, teachers have also had to adapt to using these different media to continue their teaching. So, phone-based assessment solutions can be used for at least two purposes. The first purpose is to conduct formative assessment, for example to gauge how well students have absorbed the learning content, identify any misconceptions in understanding, provide constructive feedback to students or caregivers, and offer additional learning resources and activities to support learning. The second purpose is to conduct impact evaluations, for example analyzing the effect on student learning outcomes of interventions introduced in response to the pandemic.
Link to Data Set
Citation
“Liberman, Julia; Luna-Bazaldua, Diego; Levin, Victoria; Khurana, Aishwarya. 2022. Psychometric Considerations for Implementing Phone-Based Learning Assessments. © World Bank. http://hdl.handle.net/10986/37533 License: CC BY 3.0 IGO.”
Digital Object Identifier
Associated URLs
Associated content
Other publications in this report series
Publication Greening Digital in Korea(Washington, DC, 2022-02)Digital technologies are making a significant impact on societies, economies, and the physical world, presenting both opportunities and challenges for the green agenda. Applications of these technologies in sectors such as energy, urban, transport, and agriculture are creating new possibilities for climate change mitigation strategies. However, the rapid expansion of digital technologies increases energy usage too, and is therefore also increasing greenhouse gas (GHG) emissions. In seeking to address these challenges, the World Bank’s Digital Development Global Practice (DD) will publish a flagship report on Digital Development Opportunities for Climate Change, which will assess opportunities for greening with information communication technology (ICT), as well as opportunities for greening the ICT sector itself. To inspire and inform this flagship report, DD studied Korea’s experience in greening its ICT sector, with support from the Korean Green Growth Trust Fund. The Republic of Korea was selected for the case study due to its experience in both the digital and green sectors, and its status as a globally recognized ICT powerhouse. The country was also an early adopter of a green policy agenda, and is integrating DNA (data, network, and AI) into these policies. The government announced a national policy vision of “Low Carbon, Green Growth” in 2008 and has taken concrete steps to build a solid foundation for the green transition, through legislation, standardization, information-based instruments, economic instruments, research and development (R&D), and green procurement. More recently, the country has been aligning its green ICT strategy with the broader national GHG reduction target. Korea's experience can offer meaningful lessons to other countries looking to reduce the ICT sector’s climate impact. It shows that public policies have an important impact on the ICT market. The policy tools that can spur decarbonization of the ICT sector include green government procurement, information-based instruments, economic instruments, and provision of guidelines on green business practices. Keys to success in applying such tools include strong and early political commitment; long-term planning and comprehensive policies; prioritization; research and development (R&D) and investment; and a governance structure that allows a whole-of-government approach. Additionally, Korea’s experience shows that renewable energy will play an increasingly important role in reducing GHG emissions from the energy-intensive ICT industry. Korea’s experience also underscores the fact that more evidence and analysis are needed to measure and determine the effectiveness of policy and regulatory pathways for greening the ICT sector.Publication Environmental Implications of a Central Bank Digital Currency (CBDC)(Washington, DC : World Bank, 2022-07)Two-thirds of central banks in the East Asia and Pacific (EAP) region have started researching or testing the implementation of a Central Bank Digital Currency (CBDC). At the same time, the region accounts for one-third of world CO2 emissions and is vulnerable to climate risks. As the Group of 7 (G7), European Central Bank (ECB), and Bank of England (BoE) have stated in their public statements, it is increasingly important to consider environmental impact when designing CBDC. However, only a few brief studies have been done on this subject, which will be crucial for the region. This Note explores the environmental implications of CBDC by comparing technical mechanisms and energy consumption within its distributed structure. It also illustrates differences in ecological footprint between CBDC and other payment methods (cryptocurrency, cash, and card networks). As the legitimacy of CBDC is backed by the trust of central banks, CBDC does not need to prove its legitimacy through its technological structure. Therefore, CBDC does not require the energy-intensive consensus or mining mechanisms used by a cryptocurrency, so its energy consumption is lower (comparable to that of a credit card system). CBDC can be designed to use various systems, such as Real Time Gross Settlement (RTGS), Distributed Ledger Technology (DLT), or a mixture of both. Careful deliberation to meet the objectives and implications will be important as CBDC can be a catalyst for financial innovation.Publication Assessing Incentives to Increase Digital Payment Acceptance and Usage(World Bank, Washington, DC, 2022-01-18)An important step to achieve greater financial inclusion is to increase the acceptance and usage of digital payments. Although consumer adoption of digital payments has improved dramatically globally, the acceptance and usage of digital payments for micro, small, and medium-sized retailers (MSMRs) remain challenging. Using random forest estimation, The authors identify 14 key predictors out of 190 variables with the largest predictive power for MSMR adoption and usage of digital payments. Using conditional inference trees, they study the importance of sequencing and interactions of various factors such as public policy initiatives, technological advancements, and private sector incentives. The authors find that in countries with low point of sale (POS) terminal adoption, killer applications such as mobile phone payment apps increase the likelihood of P2B digital transactions. They also find the likelihood of digital P2B payments at MSMRs increases when MSMRs pay their employees and suppliers digitally. The level of ownership of basic financial accounts by consumers and the size of the shadow economy are also important predictors of greater adoption and usage of digital payments. Using causal forest estimation, they find a positive and economically significant marginal effect for merchant and consumer fiscal incentives on POS terminal adoption on average. When countries implement financial inclusion initiatives, POS terminal adoption increases significantly and MSMRs’ share of person-to-business (P2B) digital payments also increases. Merchant and consumer fiscal incentives also increase MSMRs’ share of P2B electronic payments.Publication The Behavioral Professional(Washington, DC : World Bank, 2022)Over the past decade, governments, multilateral organizations, and think tanks have been increasingly using behavioral science as an additional tool to understand and tackle complex policy challenges in several sectors. Yet despite this increase in the use of behavioral science for policy design, little attention has been given so far to those individuals responsible for designing and implementing public policies and programs: policy professionals. This note aims to achieve three objectives. first, it highlights recent examples building on work done by the eMBeD team and the World Bank at large on how behavioral bottlenecks can hinder key development goals, from ensuring inclusive and equitable education for all (SDG4) to ensuring good health and well-being (SDG3), among others. Second, the note presents a behavioral framework highlighting the individual, group and institutional contexts that affect policy professionals. Finally, it showcases the relevance of the behavioral approach to a broad range of areas - including public service design, corruption and accountability, service design, access and delivery, civil servants’ performance - by pinpointing common bottlenecks faced, and potential solutions to overcome them.Publication World - Climate Services Operational Pathways(Washington, DC, 2021-12-22)This study was commissioned by climate investment funds (CIF’s) E and L initiative to distill lessons from CIF’s pilot program on climate resilience (PPCR) support identifying, designing, and implementing hydrometeorological and climate services investments. It seeks to generate learning and strategic insight into the different operational pathways that can be taken by national hydrological and meteorological agencies to develop, deliver, and strengthen hydrometeorological and climate services. The outputs from the study comprise of one synthesis report and three country studies for Jamaica, Mozambique, and Nepal. These three countries have been selected for the study due to their different institutional frameworks, hydrometeorological systems, and socio-economic context. They provide diverse in-depth insights in hydrometeorological and climate service development, delivery, and use. In this respect, the PPCR-supported Building Resilience to Climate-Related Hazards (BRCH) Project was selected as a case study project for Nepal to elucidate lessons learned on the process for modernizing hydrometeorological systems and developing climate services to users. It offers insight into challenges and opportunities for climate services development, delivery, and use in the South Asian developing countries. Qualitative methods, including structured interviews and literature review, were used to identify promising pathways to transform weather, water, and climate services in the three case study countries. In Nepal, the analysis of the data collected revealed five themes regarding critical pathways to continue to transform weather, water, and climate services in the country. These are: improving data resources, improving service design and delivery, improving hydrometeorological services through feedback and evaluation, fostering human resources capacity, and strengthening institutional arrangements. The report summarizes key findings and recommendations.
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication A Synthesis Report on Piloting of Remote Phone-Based Formative Assessment Solutions in Ghana, Nepal, and Pakistan(World Bank, Washington, DC, 2022)School closures related to Coronavirus disease 2019 (COVID-19) not only disrupted education but also impacted teachers’ ability to know whether and what their students were learning. This information gap was most challenging in contexts with limited internet connectivity and low access to smart digital devices. In such circumstances, conducting formative assessments remotely using basic phones was seen as a means to provide timely information to teachers, parents, and students and support learning continuity outside the classroom. As students return to school, the same solutions can be used to complement in-person instruction to accelerate learning recovery, expand the use of formative assessment in hard-to-reach schools, and improve the resilience of education systems when confronted by future shocks. This report describes the three pilot studies that aimed to test the logistical and technological feasibility of using short message service (SMS), interactive voice recording (IVR), and live phone calls (LPC) to conduct formative assessment of foundational math and literacy skills for primary-grade students. These pilot studies were conducted in Ghana, Nepal, and Pakistan between October 2020 and January 2022.Publication Guidance Note on Phone-based Formative Assessment(World Bank, Washington, DC, 2022)Phone-Based Formative Assessment refers to the use of mobile phone technology to engage students outside the classroom, assess their learning in real time, and provide timely, constructive feedback. Such assessment is particularly critical to promote learning continuity in the context of COVID-19-related school closures, which have led to the expansion and availability of remote teaching and learning resources.Publication Classroom Assessment to Support Foundational Literacy(Washington, DC: World Bank, 2025-03-21)This document focuses primarily on how classroom assessment activities can measure students’ literacy skills as they progress along a learning trajectory towards reading fluently and with comprehension by the end of primary school grades. The document addresses considerations regarding the design and implementation of early grade reading classroom assessment, provides examples of assessment activities from a variety of countries and contexts, and discusses the importance of incorporating classroom assessment practices into teacher training and professional development opportunities for teachers. The structure of the document is as follows. The first section presents definitions and addresses basic questions on classroom assessment. Section 2 covers the intersection between assessment and early grade reading by discussing how learning assessment can measure early grade reading skills following the reading learning trajectory. Section 3 compares some of the most common early grade literacy assessment tools with respect to the early grade reading skills and developmental phases. Section 4 of the document addresses teacher training considerations in developing, scoring, and using early grade reading assessment. Additional issues in assessing reading skills in the classroom and using assessment results to improve teaching and learning are reviewed in section 5. Throughout the document, country cases are presented to demonstrate how assessment activities can be implemented in the classroom in different contexts.Publication Guidance Note on Using Learning Assessment in the Process of School Reopening(World Bank, Washington, DC, 2020-11-21)As countries consider how to reopen schools safely in the context of COVID-19 (coronavirus), one key question is how to assess students' learning to support learning recovery. The expected magnitude of learning losses, particularly among students with the highest needs, makes it essential for key stakeholders in the education process — policymakers, teachers, school principals, students, and their parents — to determine where students are in their learning trajectory relative to what had been expected prior to the pandemic, so they can adjust instruction and allocate resources accordingly. To collect this information, stakeholders can rely on student learning assessment, which is an essential feedback mechanism in the education system. This note provides key steps that countries with different availability of resources should consider in developing their plans for learning assessment activities to support learning recovery in the context of school reopening. Throughout this note, assessment of student learning is defined as gathering and evaluating information on what students know, understand, and can do to make informed decisions about the next steps in the educational process. In addition, some considerations and country examples for the implementation of high-stakes examinations are discussed. This note concludes with examples of learning assessment activities that countries around the world are planning or implementing during the COVID-19 pandemic. Likewise, this note highlights important lessons that can support resilience to future emergencies and crises.Publication Primer on Large-Scale Assessments of Educational Achievement(Washington, DC: World Bank, 2021-04-26)To improve their education systems, countries around the world have increasingly initiated national largescale assessment programs or participated in international or regional large-scale assessment studies for the first time. Well-constructed large-scale assessments can provide credible information on student achievement levels, which, in turn, can promote better resource allocation to schools, stronger education service delivery, and improved learning outcomes. The World Bank developed this Primer on Large-Scale Assessments of Educational Achievement as a firststop resource for those wanting to understand how to design, administer, analyze, and use the results from these assessments of student achievement. The book addresses frequently asked questions from people working on large-scale assessment projects and those interested in making informed decisions about them. Each chapter introduces a stage in the assessment process and offers advice, guidelines, and country examples. This book also reports on emerging trends in large-scale assessment and provides updated information on regional and international large-scale assessment programs.
Users also downloaded
Showing related downloaded files
Publication Firm Foundations of Growth(Washington, DC: World Bank, 2025-05-22)In an era of rapid global technological change, productivity growth in the East Asia and Pacific (EAP) region has decelerated. The most-productive firms in EAP, the “national frontier,” are lagging behind the world’s leading firms, the “global frontier,” particularly in the digital-intensive sectors driving innovation. This widening gap is critical, as these national frontier firms are pivotal to output, employment, and the dissemination of advanced technologies to other domestic enterprises. Detailed firm-level analysis reveals that barriers to competition are stifling incentives to innovate and are inhibiting the movement of workers and capital to more-productive firms. At the same time, inadequate skills and infrastructure are constraining the region’s capacity to innovate. "Firm Foundations of Growth: Productivity and Technology in East Asia and Pacific" argues that enhancing competition, bolstering digital infrastructure, and developing relevant skills can reignite productivity growth across the region, particularly for frontier firms. This book will be of interest to researchers, businesses, and policy makers dedicated to understanding and addressing the productivity challenges in the EAP region.Publication Services Unbound(Washington, DC: World Bank, 2024-12-09)Services are a new force for innovation, trade, and growth in East Asia and Pacific. The dramatic diffusion of digital technologies and partial policy reforms in services--from finance, communication, and transport to retail, health, and education--is transforming these economies. The result is higher productivity and changing jobs in the services sector, as well as in the manufacturing sectors that use these services. A region that has thrived through openness to trade and investment in manufacturing still maintains innovation-inhibiting barriers to entry and competition in key services sectors. 'Services Unbound: Digital Technologies and Policy Reform in East Asia and Pacific' makes the case for deeper domestic reforms and greater international cooperation to unleash a virtuous cycle of increased economic opportunity and enhanced human capacity that would power development in the region.Publication The Journey Ahead(Washington, DC: World Bank, 2024-10-31)The Journey Ahead: Supporting Successful Migration in Europe and Central Asia provides an in-depth analysis of international migration in Europe and Central Asia (ECA) and the implications for policy making. By identifying challenges and opportunities associated with migration in the region, it aims to inform a more nuanced, evidencebased debate on the costs and benefits of cross-border mobility. Using data-driven insights and new analysis, the report shows that migration has been an engine of prosperity and has helped address some of ECA’s demographic and socioeconomic disparities. Yet, migration’s full economic potential remains untapped. The report identifies multiple barriers keeping migration from achieving its full potential. Crucially, it argues that policies in both origin and destination countries can help maximize the development impacts of migration and effectively manage the economic, social, and political costs. Drawing from a wide range of literature, country experiences, and novel analysis, The Journey Ahead presents actionable policy options to enhance the benefits of migration for destination and origin countries and migrants themselves. Some measures can be taken unilaterally by countries, whereas others require close bilateral or regional coordination. The recommendations are tailored to different types of migration— forced displacement as well as high-skilled and low-skilled economic migration—and from the perspectives of both sending and receiving countries. This report serves as a comprehensive resource for governments, development partners, and other stakeholders throughout Europe and Central Asia, where the richness and diversity of migration experiences provide valuable insights for policy makers in other regions of the world.Publication Digital Africa(Washington, DC: World Bank, 2023-03-13)All African countries need better and more jobs for their growing populations. "Digital Africa: Technological Transformation for Jobs" shows that broader use of productivity-enhancing, digital technologies by enterprises and households is imperative to generate such jobs, including for lower-skilled people. At the same time, it can support not only countries’ short-term objective of postpandemic economic recovery but also their vision of economic transformation with more inclusive growth. These outcomes are not automatic, however. Mobile internet availability has increased throughout the continent in recent years, but Africa’s uptake gap is the highest in the world. Areas with at least 3G mobile internet service now cover 84 percent of Africa’s population, but only 22 percent uses such services. And the average African business lags in the use of smartphones and computers as well as more sophisticated digital technologies that catalyze further productivity gains. Two issues explain the usage gap: affordability of these new technologies and willingness to use them. For the 40 percent of Africans below the extreme poverty line, mobile data plans alone would cost one-third of their incomes—in addition to the price of access devices, apps, and electricity. Data plans for small- and medium-size businesses are also more expensive than in other regions. Moreover, shortcomings in the quality of internet services—and in the supply of attractive, skills-appropriate apps that promote entrepreneurship and raise earnings—dampen people’s willingness to use them. For those countries already using these technologies, the development payoffs are significant. New empirical studies for this report add to the rapidly growing evidence that mobile internet availability directly raises enterprise productivity, increases jobs, and reduces poverty throughout Africa. To realize these and other benefits more widely, Africa’s countries must implement complementary and mutually reinforcing policies to strengthen both consumers’ ability to pay and willingness to use digital technologies. These interventions must prioritize productive use to generate large numbers of inclusive jobs in a region poised to benefit from a massive, youthful workforce—one projected to become the world’s largest by the end of this century.Publication Classroom Assessment to Support Foundational Literacy(Washington, DC: World Bank, 2025-03-21)This document focuses primarily on how classroom assessment activities can measure students’ literacy skills as they progress along a learning trajectory towards reading fluently and with comprehension by the end of primary school grades. The document addresses considerations regarding the design and implementation of early grade reading classroom assessment, provides examples of assessment activities from a variety of countries and contexts, and discusses the importance of incorporating classroom assessment practices into teacher training and professional development opportunities for teachers. The structure of the document is as follows. The first section presents definitions and addresses basic questions on classroom assessment. Section 2 covers the intersection between assessment and early grade reading by discussing how learning assessment can measure early grade reading skills following the reading learning trajectory. Section 3 compares some of the most common early grade literacy assessment tools with respect to the early grade reading skills and developmental phases. Section 4 of the document addresses teacher training considerations in developing, scoring, and using early grade reading assessment. Additional issues in assessing reading skills in the classroom and using assessment results to improve teaching and learning are reviewed in section 5. Throughout the document, country cases are presented to demonstrate how assessment activities can be implemented in the classroom in different contexts.