Publication: Psychometric Considerations for Implementing Phone-Based Learning Assessments
Loading...
Published
2022
ISSN
Date
2022-06-13
Editor(s)
Abstract
The Coronavirus disease 2019 (COVID-19) pandemic has had a significant impact on education worldwide, affecting whether and how students learn. World Bank estimates and initial analysis at the country level indicate that school closures and interruptions to in-person education are likely to negatively impact student learning. Moreover, these estimates predict that students in low- and middle-income countries will suffer from more significant learning losses than students in high-income countries, worsening the existing global learning crisis. As part of their education response to the pandemic, many countries have provided learning content to students at home through various means, including through the internet via devices such as computers, tablets, and smartphones, as well as through television, radio, printed materials, and feature (basic) phones. With schools closed, teachers have also had to adapt to using these different media to continue their teaching. So, phone-based assessment solutions can be used for at least two purposes. The first purpose is to conduct formative assessment, for example to gauge how well students have absorbed the learning content, identify any misconceptions in understanding, provide constructive feedback to students or caregivers, and offer additional learning resources and activities to support learning. The second purpose is to conduct impact evaluations, for example analyzing the effect on student learning outcomes of interventions introduced in response to the pandemic.
Link to Data Set
Citation
“Liberman, Julia; Luna-Bazaldua, Diego; Levin, Victoria; Khurana, Aishwarya. 2022. Psychometric Considerations for Implementing Phone-Based Learning Assessments. © World Bank. http://hdl.handle.net/10986/37533 License: CC BY 3.0 IGO.”
Associated URLs
Associated content
Other publications in this report series
Publication Environmental Implications of a Central Bank Digital Currency (CBDC)(Washington, DC : World Bank, 2022-07)Two-thirds of central banks in the East Asia and Pacific (EAP) region have started researching or testing the implementation of a Central Bank Digital Currency (CBDC). At the same time, the region accounts for one-third of world CO2 emissions and is vulnerable to climate risks. As the Group of 7 (G7), European Central Bank (ECB), and Bank of England (BoE) have stated in their public statements, it is increasingly important to consider environmental impact when designing CBDC. However, only a few brief studies have been done on this subject, which will be crucial for the region. This Note explores the environmental implications of CBDC by comparing technical mechanisms and energy consumption within its distributed structure. It also illustrates differences in ecological footprint between CBDC and other payment methods (cryptocurrency, cash, and card networks). As the legitimacy of CBDC is backed by the trust of central banks, CBDC does not need to prove its legitimacy through its technological structure. Therefore, CBDC does not require the energy-intensive consensus or mining mechanisms used by a cryptocurrency, so its energy consumption is lower (comparable to that of a credit card system). CBDC can be designed to use various systems, such as Real Time Gross Settlement (RTGS), Distributed Ledger Technology (DLT), or a mixture of both. Careful deliberation to meet the objectives and implications will be important as CBDC can be a catalyst for financial innovation.Publication Assessing Incentives to Increase Digital Payment Acceptance and Usage(World Bank, Washington, DC, 2022-01-18)An important step to achieve greater financial inclusion is to increase the acceptance and usage of digital payments. Although consumer adoption of digital payments has improved dramatically globally, the acceptance and usage of digital payments for micro, small, and medium-sized retailers (MSMRs) remain challenging. Using random forest estimation, The authors identify 14 key predictors out of 190 variables with the largest predictive power for MSMR adoption and usage of digital payments. Using conditional inference trees, they study the importance of sequencing and interactions of various factors such as public policy initiatives, technological advancements, and private sector incentives. The authors find that in countries with low point of sale (POS) terminal adoption, killer applications such as mobile phone payment apps increase the likelihood of P2B digital transactions. They also find the likelihood of digital P2B payments at MSMRs increases when MSMRs pay their employees and suppliers digitally. The level of ownership of basic financial accounts by consumers and the size of the shadow economy are also important predictors of greater adoption and usage of digital payments. Using causal forest estimation, they find a positive and economically significant marginal effect for merchant and consumer fiscal incentives on POS terminal adoption on average. When countries implement financial inclusion initiatives, POS terminal adoption increases significantly and MSMRs’ share of person-to-business (P2B) digital payments also increases. Merchant and consumer fiscal incentives also increase MSMRs’ share of P2B electronic payments.Publication Assessment of the Labor Market Information System (LMIS) in the Lao People’s Democratic Republic PDR(Washington, DC, 2022)The Lao People’s Democratic Republic (PDR) has made substantial progress in poverty reduction even though its resource-based development pattern has historically limited the impact of growth on poverty reduction. The objective of the assessment is to support the Ministry of Labor and Social Welfare (MOLSW) in their planned reforms of labor market institutions and systems in Lao; in particular by supporting investments in a comprehensive Labor Market Information System (LMIS) for improved jobseeker data collection and analytics, jobseeker profiling, and assignment to various employment support programs such as vocational trainings and job search assistance. The target audience of this report is the technical staff and management at the department levels as well as the policy makers at the MOLSW, Ministry of Planning and Investment (MPI) as well as the Ministry of Finance (MOF).Publication Greening Digital in Korea(Washington, DC, 2022-02)Digital technologies are making a significant impact on societies, economies, and the physical world, presenting both opportunities and challenges for the green agenda. Applications of these technologies in sectors such as energy, urban, transport, and agriculture are creating new possibilities for climate change mitigation strategies. However, the rapid expansion of digital technologies increases energy usage too, and is therefore also increasing greenhouse gas (GHG) emissions. In seeking to address these challenges, the World Bank’s Digital Development Global Practice (DD) will publish a flagship report on Digital Development Opportunities for Climate Change, which will assess opportunities for greening with information communication technology (ICT), as well as opportunities for greening the ICT sector itself. To inspire and inform this flagship report, DD studied Korea’s experience in greening its ICT sector, with support from the Korean Green Growth Trust Fund. The Republic of Korea was selected for the case study due to its experience in both the digital and green sectors, and its status as a globally recognized ICT powerhouse. The country was also an early adopter of a green policy agenda, and is integrating DNA (data, network, and AI) into these policies. The government announced a national policy vision of “Low Carbon, Green Growth” in 2008 and has taken concrete steps to build a solid foundation for the green transition, through legislation, standardization, information-based instruments, economic instruments, research and development (R&D), and green procurement. More recently, the country has been aligning its green ICT strategy with the broader national GHG reduction target. Korea's experience can offer meaningful lessons to other countries looking to reduce the ICT sector’s climate impact. It shows that public policies have an important impact on the ICT market. The policy tools that can spur decarbonization of the ICT sector include green government procurement, information-based instruments, economic instruments, and provision of guidelines on green business practices. Keys to success in applying such tools include strong and early political commitment; long-term planning and comprehensive policies; prioritization; research and development (R&D) and investment; and a governance structure that allows a whole-of-government approach. Additionally, Korea’s experience shows that renewable energy will play an increasingly important role in reducing GHG emissions from the energy-intensive ICT industry. Korea’s experience also underscores the fact that more evidence and analysis are needed to measure and determine the effectiveness of policy and regulatory pathways for greening the ICT sector.Publication The Behavioral Professional(Washington, DC : World Bank, 2022)Over the past decade, governments, multilateral organizations, and think tanks have been increasingly using behavioral science as an additional tool to understand and tackle complex policy challenges in several sectors. Yet despite this increase in the use of behavioral science for policy design, little attention has been given so far to those individuals responsible for designing and implementing public policies and programs: policy professionals. This note aims to achieve three objectives. first, it highlights recent examples building on work done by the eMBeD team and the World Bank at large on how behavioral bottlenecks can hinder key development goals, from ensuring inclusive and equitable education for all (SDG4) to ensuring good health and well-being (SDG3), among others. Second, the note presents a behavioral framework highlighting the individual, group and institutional contexts that affect policy professionals. Finally, it showcases the relevance of the behavioral approach to a broad range of areas - including public service design, corruption and accountability, service design, access and delivery, civil servants’ performance - by pinpointing common bottlenecks faced, and potential solutions to overcome them.
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication A Synthesis Report on Piloting of Remote Phone-Based Formative Assessment Solutions in Ghana, Nepal, and Pakistan(World Bank, Washington, DC, 2022)School closures related to Coronavirus disease 2019 (COVID-19) not only disrupted education but also impacted teachers’ ability to know whether and what their students were learning. This information gap was most challenging in contexts with limited internet connectivity and low access to smart digital devices. In such circumstances, conducting formative assessments remotely using basic phones was seen as a means to provide timely information to teachers, parents, and students and support learning continuity outside the classroom. As students return to school, the same solutions can be used to complement in-person instruction to accelerate learning recovery, expand the use of formative assessment in hard-to-reach schools, and improve the resilience of education systems when confronted by future shocks. This report describes the three pilot studies that aimed to test the logistical and technological feasibility of using short message service (SMS), interactive voice recording (IVR), and live phone calls (LPC) to conduct formative assessment of foundational math and literacy skills for primary-grade students. These pilot studies were conducted in Ghana, Nepal, and Pakistan between October 2020 and January 2022.Publication Guidance Note on Phone-based Formative Assessment(World Bank, Washington, DC, 2022)Phone-Based Formative Assessment refers to the use of mobile phone technology to engage students outside the classroom, assess their learning in real time, and provide timely, constructive feedback. Such assessment is particularly critical to promote learning continuity in the context of COVID-19-related school closures, which have led to the expansion and availability of remote teaching and learning resources.Publication Classroom Assessment to Support Foundational Literacy(Washington, DC: World Bank, 2025-03-21)This document focuses primarily on how classroom assessment activities can measure students’ literacy skills as they progress along a learning trajectory towards reading fluently and with comprehension by the end of primary school grades. The document addresses considerations regarding the design and implementation of early grade reading classroom assessment, provides examples of assessment activities from a variety of countries and contexts, and discusses the importance of incorporating classroom assessment practices into teacher training and professional development opportunities for teachers. The structure of the document is as follows. The first section presents definitions and addresses basic questions on classroom assessment. Section 2 covers the intersection between assessment and early grade reading by discussing how learning assessment can measure early grade reading skills following the reading learning trajectory. Section 3 compares some of the most common early grade literacy assessment tools with respect to the early grade reading skills and developmental phases. Section 4 of the document addresses teacher training considerations in developing, scoring, and using early grade reading assessment. Additional issues in assessing reading skills in the classroom and using assessment results to improve teaching and learning are reviewed in section 5. Throughout the document, country cases are presented to demonstrate how assessment activities can be implemented in the classroom in different contexts.Publication Guidance Note on Using Learning Assessment in the Process of School Reopening(World Bank, Washington, DC, 2020-11-21)As countries consider how to reopen schools safely in the context of COVID-19 (coronavirus), one key question is how to assess students' learning to support learning recovery. The expected magnitude of learning losses, particularly among students with the highest needs, makes it essential for key stakeholders in the education process — policymakers, teachers, school principals, students, and their parents — to determine where students are in their learning trajectory relative to what had been expected prior to the pandemic, so they can adjust instruction and allocate resources accordingly. To collect this information, stakeholders can rely on student learning assessment, which is an essential feedback mechanism in the education system. This note provides key steps that countries with different availability of resources should consider in developing their plans for learning assessment activities to support learning recovery in the context of school reopening. Throughout this note, assessment of student learning is defined as gathering and evaluating information on what students know, understand, and can do to make informed decisions about the next steps in the educational process. In addition, some considerations and country examples for the implementation of high-stakes examinations are discussed. This note concludes with examples of learning assessment activities that countries around the world are planning or implementing during the COVID-19 pandemic. Likewise, this note highlights important lessons that can support resilience to future emergencies and crises.Publication Primer on Large-Scale Assessments of Educational Achievement(Washington, DC: World Bank, 2021-04-26)To improve their education systems, countries around the world have increasingly initiated national largescale assessment programs or participated in international or regional large-scale assessment studies for the first time. Well-constructed large-scale assessments can provide credible information on student achievement levels, which, in turn, can promote better resource allocation to schools, stronger education service delivery, and improved learning outcomes. The World Bank developed this Primer on Large-Scale Assessments of Educational Achievement as a firststop resource for those wanting to understand how to design, administer, analyze, and use the results from these assessments of student achievement. The book addresses frequently asked questions from people working on large-scale assessment projects and those interested in making informed decisions about them. Each chapter introduces a stage in the assessment process and offers advice, guidelines, and country examples. This book also reports on emerging trends in large-scale assessment and provides updated information on regional and international large-scale assessment programs.
Users also downloaded
Showing related downloaded files
Publication Applying the Degree of Urbanisation(European Union/FAO/UN-Habitat/OECD/The World Bank, 2025-01-31)This manual develops a harmonised methodology to facilitate international statistical comparisons and to classify the entire territory of a country along an urban-rural continuum. The degree of urbanisation classification defines cities, towns and semi-dense areas, and rural areas. This first level of the classification may be complemented by a range of more detailed concepts, such as: metropolitan areas, commuting zones, dense towns, semi-dense towns, suburban or peri-urban areas, villages, dispersed rural areas and mostly uninhabited areas. The manual is intended to complement and not replace the definitions used by national statistical offices (NSOs) and ministries. It has been designed principally as a guide for data producers, suppliers and statisticians so that they have the necessary information to implement the methodology and ensure coherency within their data collections. It may also be of interest to users of subnational statistics so they may better understand, interpret and use official subnational statistics for taking informed decisions and policymaking. This report has been produced in close collaboration by six organisations, the European Commission, the Food and Agriculture Organization of the United Nations (FAO), the United Nations Human Settlements Programme (UNHabitat), the International Labour Organization (ILO), the Organisation for Economic Co-operation and Development (OECD) and The World Bank.Publication Services Unbound(Washington, DC: World Bank, 2024-12-09)Services are a new force for innovation, trade, and growth in East Asia and Pacific. The dramatic diffusion of digital technologies and partial policy reforms in services--from finance, communication, and transport to retail, health, and education--is transforming these economies. The result is higher productivity and changing jobs in the services sector, as well as in the manufacturing sectors that use these services. A region that has thrived through openness to trade and investment in manufacturing still maintains innovation-inhibiting barriers to entry and competition in key services sectors. 'Services Unbound: Digital Technologies and Policy Reform in East Asia and Pacific' makes the case for deeper domestic reforms and greater international cooperation to unleash a virtuous cycle of increased economic opportunity and enhanced human capacity that would power development in the region.Publication Greater Heights(Washington, DC: World Bank, 2025-03-12)Twenty-seven countries have reached high-income status since 1990. Ten of these are in the Europe and Central Asia region and have joined the European Union. Another 20 in the region have become more prosperous since the 1990s. However, their transition to high-income status has been delayed. These middle-income countries have found that the prospects for growth to high-income status have become even more difficult since the 2007–09 global financial crisis. This reflects partly a slowdown in structural reforms at home and partly the challenges associated with a deterioration in the global environment. The concern has emerged that many countries in the region may be caught in the middle-income trap, a phase in development characterized by a recurring deceleration in growth and by per capita incomes that are systematically below the high-income threshold. To ensure that these countries overcome the obstacles to growth and achieve progress toward high-income status, policy makers need to make the transition from a strategy driven largely by investment to a strategy that is supported by the importation and diffusion of global capital, knowledge, and technology and then to a strategy that complements these with innovation. The report Greater Heights: Growing to High Income in Europe and Central Asia relies on the 3i strategy described in World Development Report 2024—investment, infusion, and innovation—to propose policy options to assist middle-income countries in Europe and Central Asia in the effort to reach high-income status. Drawing on comprehensive empirical analysis, the report offers actionable recommendations that will enable policy makers to advance stronger economic growth across the region. Such a transition will require continued and sustained foundational reform to maximize the drivers of economic growth while pivoting to new transformative reforms to promote the development of more complex economic structures and institutions. These involve the need to discipline incumbents, boost the role of the private sector, strengthen the competitive environment, and reward merit. The emphasis on a strategy driven by innovation is also critically important for those countries that have already attained high-income status.Publication Fixing the Foundation(Washington, DC: World Bank, 2023-09-20)Countries in middle-income East Asia and the Pacific were already experiencing serious learning deficits prior to the COVID-19 pandemic. COVID-related school disruptions have only made things worse. Learning poverty -- defined as the percentage of 10-year-olds who cannot read and understand an age-appropriate text -- is as high as 90 percent in several countries. Several large Southeast Asian countries consistently perform well below expectations on adolescent learning assessments. This report examines key factors affecting student learning in the region, with emphasis on the central role of teachers and teaching quality. It also analyzes the role education technologies, which came into widespread use during the pandemic, and examines the political economy of education reform. The report presents recommendations on how countries can strengthen teaching to improve learning and, in doing so, can enhance productivity, growth, and future development in the region.Publication The Journey Ahead(Washington, DC: World Bank, 2024-10-31)The Journey Ahead: Supporting Successful Migration in Europe and Central Asia provides an in-depth analysis of international migration in Europe and Central Asia (ECA) and the implications for policy making. By identifying challenges and opportunities associated with migration in the region, it aims to inform a more nuanced, evidencebased debate on the costs and benefits of cross-border mobility. Using data-driven insights and new analysis, the report shows that migration has been an engine of prosperity and has helped address some of ECA’s demographic and socioeconomic disparities. Yet, migration’s full economic potential remains untapped. The report identifies multiple barriers keeping migration from achieving its full potential. Crucially, it argues that policies in both origin and destination countries can help maximize the development impacts of migration and effectively manage the economic, social, and political costs. Drawing from a wide range of literature, country experiences, and novel analysis, The Journey Ahead presents actionable policy options to enhance the benefits of migration for destination and origin countries and migrants themselves. Some measures can be taken unilaterally by countries, whereas others require close bilateral or regional coordination. The recommendations are tailored to different types of migration— forced displacement as well as high-skilled and low-skilled economic migration—and from the perspectives of both sending and receiving countries. This report serves as a comprehensive resource for governments, development partners, and other stakeholders throughout Europe and Central Asia, where the richness and diversity of migration experiences provide valuable insights for policy makers in other regions of the world.