Approach Paper Confronting the Learning Crisis: Lessons from World Bank Support for Basic Education, 2012–22 May 24, 2022 1. Introduction 1.1 The achievement of learning outcomes has been a long-standing challenge for education systems across the developing world and has significant consequences for economic development. To realize the development aims of education investments, students need to learn, but too many have not, especially in low-income countries. In 2017, 60 percent of children and adolescents were not achieving minimum proficiency levels in reading and mathematics. That is 617 million children, including more than 387 million children of primary school age. Moreover, the children being failed by their education systems are predominantly those who need a good education the most to succeed in life: those already disadvantaged by poverty, location, ethnicity, gender, or disability. In low-income countries, 94 percent of children of primary school age will not achieve minimum reading proficiency, and 87 percent will not achieve minimum math proficiency (UIS 2017). The cost to economic development associated with such poor outcomes from basic education alone has been estimated at $129 billion per year, equivalent to 10 percent of global spending on primary education (UNESCO 2014). 1.2 The World Bank has sought to address this learning crisis for more than a decade through the pursuit of quality education that enhances learning outcomes, especially since publication of its 2011 education strategy Learning for All: Investing in People’s Knowledge and Skills to Promote Development. The issues involved are multilayered, demanding a diversified approach. The complex issues involved in improving quality are social, structural, logistical, and institutional. They include ensuring that children are prepared to learn, teachers are well trained and motivated, learning inputs are available and culturally and grade-level appropriate, and management and governance have the capacity and authority to pull the various factors together. 1.3 The various issues involved create obstacles to orienting and aligning education systems toward learning. Stakeholders in the system often have divergent goals. For example, even when countries want to prioritize learning, they typically lack the reliable, timely metrics needed to answer pertinent questions, such as, is the new teacher training program improving teacher effectiveness? The many parts of an education system also need to be well aligned. For example, a new curriculum with increased emphasis on active learning and creative thinking requires appropriately trained teachers and relevant learning materials. 1.4 The proposed evaluation by Independent Evaluation Group (IEG) will assess the extent to which the World Bank’s Education Global Practice (GP) and its predecessor, the education sector unit, have supported efforts to improve learning outcomes over the past decade (fiscal years [FY]12–22). Based on that experience, the evaluation will assess the effectiveness, relevance, and adequacy of World Bank support to address the learning crisis. It will identify lessons and recommendations to inform the next education sector strategy and further development of the World Bank’s approach to this persistent development challenge and the exacerbation of learning deficits during the coronavirus (COVID-19) pandemic. 2. Background and Context 2.1 Recent success in ensuring access to education for all children was not matched by the provision of quality education. For example, the second Millennium Development Goal was to achieve universal primary education, with a focus on enrollments. By 2015, the primary school net enrollment rate in developing regions reached 91 percent, up from 83 percent in 2000; the number of out-of-school children of primary school age worldwide fell by almost half to an estimated 57 million in 2015, down from 100 million in 2000 (UN 2015). Yet, as already noted, the United Nations Educational, Scientific, and Cultural Organization’s (UNESCO) 2014 Global Monitoring Report, Teaching and Learning: Achieving Quality for All, found that learning was not keeping pace, with grave consequences for economic development and, by extension, for poverty reduction. At the time, the United Nations secretary-general’s special adviser on post-2015 development planning said, “We should really be angry about where we are in education 13 years later after Dakar…[so] many students have passed through education, but education has not passed through them” (Soliván and Winthrop 2014). 1 2.2 The World Development Report (WDR) 2018: Learning to Realize Education’s Promise highlighted the learning crisis, attributing it to schools failing learners and systems failing schools. Learning outcomes can improve, the report noted, if countries decide to make learning matter. The drivers of learning shortfalls are associated with immediate causes such as poor service delivery that amplifies the effects of poverty, but shortcomings also arise from deeper system-level challenges (technical and political) that allow low-quality schooling to persist. 2.3 In recent years, the World Bank’s Education GP introduced the concept of learning poverty (the inability to read and understand a simple text by age 10), noting that in almost all countries for which data are available, girls have on average 6 percentage points lower rates of learning poverty than boys do (World Bank 2019a). Recent data indicate that 53 percent of all children in low- and middle-income countries 2 suffer from learning poverty and, at the current rate of improvement, about 43 percent of children will still be learning-poor in 2030 (World Bank 2019a). Yet the average masks major differences in learning poverty across the developing world. • The learning poverty rate in upper-middle-income countries averages 29 percent compared with 55 percent in lower-middle-income countries, and 90 percent in low-income countries. • The rate of learning poverty is particularly high in Sub-Saharan Africa (about 87 percent), almost seven times as high as for World Bank clients in Europe and Central Asia (13 percent). 2.4 As part of recent efforts to address the learning crisis, the World Bank has also played a convening role, together with UNESCO Institute for Statistics and other partners, in setting “learning” as the global priority in education, with specific reference to measuring learning (and learning poverty) and, in turn, concretizing the learning agenda and focusing political awareness and attention on the challenge. 2.5 The COVID-19 pandemic has exacerbated the issues with learning outcomes, especially for poor people. The World Bank recently reported that “more than 130 countries sought to mitigate learning losses through remote learning initiatives using digital and nondigital education technology solutions, but capacity, logistical, and financial impediments often limited their effectiveness” (World Bank 2020b, 12). Citing research by Azevedo (2020), the World Bank estimates that COVID-19-related school closures could increase the learning poverty rate in low- and middle-income countries by 10 percentage points, putting 72 million more children of primary school age at risk of falling into learning poverty. 2.6 The pandemic has exposed fundamental weaknesses in education systems and presents an opportunity for collective action because stakeholders appreciate the cost of inaction more fully. The crisis has underscored the crucial role of schools for children and families beyond learning, such as socialization, nutrition, and social-emotional well- being. Recent research from the first wave of school closures in Organisation for Economic Co-operation and Development countries has identified a substantial achievement gap among lower-income children and mental health concerns (Thorn and Vincent-Lancrin 2021). France’s experience shows that the achievement gap created during its two-month school closure can be addressed successfully for young children with appropriate interventions (Thorn and Vincent-Lancrin 2021). Thus, the pandemic has highlighted the need for reorientating education systems, particularly at the level of basic education, to ensure that children have the foundational skills needed and that the achievement gaps created during school closures do not persist. 3 World Bank Approach to Addressing Learning 2.7 Schooling access typically received more attention in the World Bank approach to education in the 1990s and the first years of the new millennium, but there were signs that learning was a concern. A 1990 policy paper on primary education noted that “higher priority should be given to measures intended to increase children’s learning and primary school completion” (Haddad et al. 1990). Again in 1999, the World Bank Education Sector Strategy noted that support has failed to place enough emphasis on the quality of teaching and learning outcomes, focusing too narrowly on a single subsector in isolation from the rest of the education system or on expanding physical infrastructure without enough concern for the activities and policies that determine learning outcomes (World Bank 1999). Still, perhaps because of the World Bank’s commitment to meeting the Millennium Development Goals, the focus remained squarely on access to education and school completion rates. 2 2.8 By 2006, an IEG evaluation of support to primary education found that the World Bank had contributed significantly to improved access through construction of schools and reductions in barriers to access. However, the report also asserted, “Basic knowledge and skills—not educational attainment—are key to reducing poverty. Raising enrollments and completing primary schooling are necessary—but not sufficient—to ensure basic literacy and numeracy.” The report recommended a focus on improving learning outcomes (World Bank 2006). Such a refocusing was evident in the sector’s next and current strategy, World Bank (2011). That strategy also commits the World Bank to a focus on the education system, moving beyond the provision of inputs to ensure more effective use of those inputs. The approach also encourages investment in systems analysis, knowledge, and data support that will allow the World Bank and government policy makers to “analyze globally and act locally” (World Bank 2011). Also launched by the World Bank in 2011, the Systems Approach for Better Education Results was designed to help identify education policies and programs most likely to create quality learning environments and improve student performance, especially among the disadvantaged. Systems Approach for Better Education Results sought to produce comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems and the ultimate goal of promoting Learning for All. 3 2.9 Subsequently, WDR 2018, Learning to Realize Education’s Promise highlighted the learning crisis and emphasized the need for context-specific solutions, especially those developed by the country client. Although this means that countries must want to improve learning, WDR 2018 (3) argues that education systems often conspire against a learning-focused approach through, for example, goal misalignment (where learning may not be the central goal of the various components of or actors in the system) and a 4 lack of coherence (where the various components of the system fail to reinforce each other toward achieving learning). Support is needed, the WDR suggested to correct poor service delivery and address system-level technical and political challenges that allow low-quality schooling to persist (World Bank 2018). 2.10 The World Bank described its most recent approach to the learning crisis and its consequence, learning poverty, in Ending Learning Poverty: What Will it Take (World Bank 2019a). The response was elaborated further in Realizing the Future of Learning: From Learning Poverty to Learning for Everyone, Everywhere, taking on the additional challenges posed by the global pandemic (World Bank 2020b). In these formal statements of intent, the World Bank set out to strengthen its efforts to confront learning poverty. It also set out to influence the focus on learning poverty at the global level by launching a new operational global learning target to cut the learning poverty rate by at least half before 2030 and by introducing three key pillars of work: a literacy policy package, a refreshed education approach to strengthen entire education systems, and an ambitious measurement and research agenda. These pillars are intended to support countries to improve the human capital outcomes of their people. 2.11 In line with the Human Capital Project’s goal to accelerate more and better investments in people for greater equity and economic growth, the World Bank also recognized that education initiatives alone are not enough to tackle this critical development challenge and that a multisectoral approach (water and sanitation, health and nutrition, social protection, civil service reforms, and strengthened management and financing of public services) is needed. As asserted in WDR 2018, this requires a whole-of-government approach to ensure better learning outcomes and renewed attention to the role of families and communities in building the demand for education, creating the right environment for learning, and supporting the right education reforms. The World Bank envisions an approach in which countries can chart their own paths with a political commitment to carry out investments and reforms across five pillars that typify a well-functioning education system to ensure that: • Learners are prepared and motivated to learn, with a stronger emphasis on whole-child development and support to learning continuity beyond the school (foundational skills, and bolstering the role of and supporting families and their communities). • Teachers are effective and valued and ready to take on an increasingly complex role as facilitators of learning at and beyond the school with use of education technology (among other things, ensure that teaching is socially valued and that teachers have the tools and support they need to be effective). 5 • Learning resources, including curricula, are diverse and high-quality to support good pedagogical practices and personalized learning. • Schools are safe and inclusive spaces, with a whole-and-beyond-the-school approach to prevent and address violence and leave no child behind. • Education systems are well managed, with school leaders who spur more effective pedagogy and a competent educational bureaucracy adept at using technology, data, and evidence (strengthen and professionalize school leadership and development of strong bureaucracies in education systems to manage extremely complex service delivery systems). 2.12 To realize the global learning target, the World Bank proposes to contribute with country-level actions that are consistent with and contribute to each of the pillars. The World Bank asserts that interventions focused on literacy can accelerate progress toward the global learning target and raise overall education quality. The policy package in support of helping children learn to read has four components that bring focus to and facilitate what countries need to do: • Ensure political and technical commitment to clear goals, means, and measures for literacy; • Ensure effective teaching for literacy, noting that the evidence shows that when students are taught in the right way (content, sequence, and amount of instruction), nearly all of them learn to read; • Ensure timely access to more and better age- and skill-appropriate texts—the availability of quality, age-appropriate reading materials is a significant predictor of strong early literacy; and • Ensure that children are first taught in the language they speak and understand. 2.13 The World Bank proposes to differentiate the application of the literacy policy package across diverse country conditions using the Accelerator Program, through which it identifies critical factors related to the readiness of individual countries to meet the challenge, including the institutional capacity of the national education system.4 Depending on where the country is situated along these dimensions, different policies and interventions are implemented that reflect country capacities and circumstances. In that regard, the World Bank identifies four types of country context: (i) fragility, conflict, and violence (FCV) contexts, where innovative methods are often required to support education delivery; (ii) countries with low institutional capacity, where interventions require a focus on clear guidance and support to teachers, including structured lessons and coaching; (iii) countries with moderate levels of institutional capacity, where 6 textbooks may be available but there may be extreme absenteeism or chronic teacher shortages and outdated classroom practices; and (iv) countries with high institutional capacity, which may have more nuanced constraints, such as a lack of screening for disabilities or assessment data not necessarily informing policies and teacher practices. The World Bank also recognizes the importance of measurement and assessment at both the student and system levels that can provide information that can drive decisions. 2.14 Because the World Bank’s focus is on enhanced literacy, it is also working with partners to develop instruments assessing socioemotional skills that, once developed, would be made available for use by policy makers, researchers, and organizations interested in generating performance metrics of the education system with a whole-child approach. This work will also feed into the World Bank’s existing work on measuring and improving teaching practices related to developing students’ socioemotional skills in the classroom (through the Teach tool). The World Bank has also developed instruments to help countries assess and improve their learning assessment systems and to support assessment reforms. World Bank Financial and Analytical Support to Improve Learning in Basic Education 2.15 The World Bank’s support to improve basic education consists of 251 lending projects and 562 advisory services and analytics (ASA) projects in 125 countries. 5 This figure includes 96 operations funded by the Global Partnership for Education and implemented by the World Bank (box 2.1). Among lending projects, the International Development Association accounts the largest share (113), followed by recipient- executed trust funds (95), and the International Bank for Reconstruction and Development (38). By product line, the majority are investment project financing (202). Box 2.1. Select World Bank Partnerships in Education The World Bank hosts the Global Partnership for Education (GPE) secretariat and acts as a board member, trustee, and grant implementor for the majority of GPE grants. From partnership inception through 2020, the World Bank has implemented $5 billion in grants. The GPE has also funded research led by the World Bank Group (for example, Economic Costs of Child Marriage and Disability Gaps in Educational Attainment and Literacy). For the response to COVID-19, GPE approved $25 million in grants to fund a joint initiative by the United Nations Educational, Scientific, and Cultural Organization (UNESCO), the United Nations Children’s Fund, and the World Bank to ensure regional and global efficiencies and knowledge sharing related to education. GPE also joined the World Bank and other partners to produce the report Pivoting to Inclusion: Leveraging Lessons from the COVID-19 Crisis for Learners with Disabilities. 7 The Russia Education Aid for Development program, a partnership established in 2008, aims to strengthen the capacity of countries to assess student learning and use the information from those assessments to improve teaching and learning outcomes.a Another partnership, the Strategic Impact Investment Fund supported by the United Kingdom Department for International Development (now the Foreign, Commonwealth, and Development Office), launched its first call for proposals in 2012 to support research that measures the impact of programs and policies to improve education, health, access to quality water and sanitation, and early childhood development in low- and middle-income countries. The Results in Education for All Children Program, established in 2015, seeks to help countries strengthen their education services by focusing programs and initiatives on results, with the goal of boosting learning outcomes especially among the most vulnerable populations. The Results in Education for All Children Program is funded by the governments of Germany, Norway, and the United States of America. More recently, in 2019, the World Bank and UNESCO announced a collaborative effort to help countries strengthen their learning assessment systems, better monitor what students are learning in internationally comparable ways, and improve the breadth and quality of global education data.b In this instance, the UNESCO Institute for Statistics is leading global efforts to expand internationally comparable data on learning outcomes, leveraging countries’ national measurement efforts. The World Bank is developing the Global Education Policy Dashboard to enable countries to monitor how well their education systems are oriented toward improving learning and educational attainment for all children. Source: Global Partnership for Education Consolidated Financial Report 2020. Global Partnership for Education Results Report 2021. https://www.worldbank.org/en/programs/reach#12 Note: a. At the global level, the Russia Education Aid for Development program focuses on generating and sharing knowledge and good practices (tools, analytical reports, case studies, technical advice, and guidance), and at the country level, implementation of a set of program-supported activities that address gaps in the country’s existing learning assessment system. b. The partnership is framed within the overarching 2018 Strategic Partnership Framework signed between the United Nations and the World Bank Group to consolidate their joint commitment to help countries implement the 2030 Agenda for Sustainable Development. 2.16 The regional breakdown of lending and analytical support (figure 2.1) indicates that Western and Central Africa, Eastern and Southern Africa, South Asia, and Latin America and the Caribbean take the largest share of lending projects. Eight countries account for almost one-quarter of lending: Bangladesh, Ethiopia, Haiti, India, Nicaragua, Nigeria, Pakistan, and The Gambia. Latin America and the Caribbean accounted for the largest number of analytical projects, followed by Europe and Central Asia and Eastern and Southern Africa. The share of projects in FCV contexts is 25 percent for lending and 14 percent for ASA. In addition, the evaluation will include pandemic emergency response operations that consist of another 120 projects (80 lending projects and 40 ASA projects). 8 Figure 2.1. Financial and Analytical Support by Region, number of projects, FY12–22 Source: World Bank Enterprise Data Catalog. Note: ASA = advisory services and analytics; FY = fiscal year. 3. Objectives and Audience 3.1 The evaluation’s objective is to provide insights to inform a new World Bank education strategy, with particular reference to addressing education quality at the basic education level and for the ongoing operationalization of the World Bank’s support to enhancing learning outcomes. 6 It pays particular attention to the extent to which the World Bank has adopted a systems-level approach to its support for basic education, that is, deep policy-level engagement, eschewing a narrow focus on a single subsector in isolation from the rest of the education system, and moving beyond the provision of inputs to ensure more effective use of those inputs. The evaluation also responds to the increased urgency of the learning crisis (a priority for the Board of Executive Directors), which the COVID-19 pandemic has exacerbated. 3.2 In addition to fulfilling IEG’s accountability function for the World Bank Committee on Development Effectiveness, the evaluation aims to provide information to four groups of stakeholders: • The World Bank Education GP, to assist the team in the ongoing design of its approach to support for education quality and responding to the exacerbation of learning poverty associated with the ongoing pandemic; • Bank Group country teams in various country types, to assist them in the identification and implementation of priorities regarding their interaction with 9 country clients toward supporting education quality and addressing learning poverty; • Bank Group management, the Board, and other internal stakeholders, such as the Human Capital Project team, to inform them about progress in the implementation of this critical agenda; and • External stakeholders and partners, to include other multilateral development banks, United Nations agencies, bilateral donors, international nongovernmental organizations and foundations, governments, and national civil society organizations (CSOs), to inform them on progress being made and work to be done by the World Bank in support of education quality and enhanced learning outcomes and the World Bank’s role with partners toward achieving common objectives. 4. Evaluation Questions and Scope 4.1 The evaluation questions and scope were designed to provide insights for the new education strategy. To ensure coherence with current practice, a consultative process was used to inform the evaluation’s focus and scope. To identify evaluation questions and define scope, the team conducted semistructured interviews and other engagement with key staff in the Education GP to better understand how the World Bank’s support has been executed and what priorities have been addressed. IEG also met with Education GP management to ensure that the evaluation questions and design were likely to produce learning and evidence that would be useful in ongoing efforts to improve learning outcomes. Evaluation Questions 4.2 The evaluation aims to assess the extent to which the World Bank’s Education GP and Education sector unit have supported efforts to improve learning outcomes over the past decade (FY12–22). The evaluation is timed to inform the next education sector strategy through the provision of lessons and recommendations related to the World Bank’s past support to tackle the long-standing need to improve learning outcomes, and to more recent support to address the exacerbation of the learning crisis and student learning loss associated with the global pandemic’s effects on education systems. 4.3 The overarching evaluation question is, How has World Bank support for basic education contributed to the achievement of enhanced learning outcomes since the Learning for All strategy, and what can be learned from those efforts to inform support to the learning recovery from the COVID-19 pandemic? 10 4.4 To respond to this question, the evaluation will answer the following evaluation questions: • EQ1: How effective has World Bank support for basic education (FY12–22) been in addressing the binding constraints that hinder the achievement of enhanced learning outcomes in client countries? • EQ2: To what extent and how effectively has the World Bank: o Collaborated with country and global partners to support education quality and enhanced learning outcomes? o Used feedback from evidence and experience to inform its work to support improved education quality and learning outcomes for all? • EQ3: How well prepared is the World Bank to address additional challenges to education systems that have arisen because of the impact of the COVID-19 pandemic? 4.5 To address the first evaluation question, the team will explore the World Bank’s contribution, taking into account global knowledge (for example, data, programs, and initiatives), the project portfolio, and country-level engagement (for example, partnerships, policy dialogue, tailored advice, and analysis). This will involve an assessment of how well the World Bank responded to the learning crisis, to include a focus on key global issues (for example, through impact evaluations, global data, national student assessment systems, and foundational learning) and to identify and address binding constraints at the country level. The evaluation will also assess, in context, the extent to which World Bank projects targeted poor people and other potentially marginalized groups (for example, gender, children with disabilities, children from ethnic minorities, and those out of school). 4.6 The team will address the second evaluation question through an assessment of World Bank engagement in collaboration with other actors (for example, Global Partnership for Education, UNESCO, bilateral donors, governments, and CSOs) and World Bank adaptation to learning (for example, through its work with partners, through its own knowledge work, and through the evaluation of its portfolio in basic education). The assessment of the first aspect will involve a document review, interviews with key global partners, and interviews with partners at the country level and will be supported by a background paper on how other development institutions have approached the challenge of improving learning outcomes. The portfolio analysis will also assess the level and type of interaction between the World Bank and other entities at the project level. 11 4.7 For the third evaluation question, the team will examine the robustness of World Bank policies and support to address the exacerbation of the learning crisis by learning loss associated with the COVID-19 pandemic’s effects on school systems and students. This will involve a targeted assessment of more recent operations, policy dialogue and knowledge work, and assessment of how the World Bank has engaged at the country level (for example, how policy dialogue or financing changed to meet the new challenges, how existing partnerships evolved, and what new partnerships were initiated and developed), and how World Bank support has changed in response to COVID-19 compounding the crisis. Evaluation Scope 4.8 The scope of this evaluation was defined along four dimensions: global knowledge, country coverage, subject focus, and reference period. • Global knowledge: The evaluation will focus on a purposeful sample of strategies, initiatives, programs, and research supported by the Education GP over the 10-year period evaluated, such as data; impact evaluations; foundational learning and teachers; education system improvement via Systems Approach for Better Education Results; and student learning assessments at regional, national, and subnational levels. The evaluation will assess how well these efforts are generating knowledge and complementing the efforts of other partners and building awareness about how to improve quality and learning in basic education and strengthen education systems. • Country coverage: The evaluation will select countries of focus from among distinct country types—FCV, low institutional capacity, and moderate institutional capacity—identified in the World Bank’s response to WDR 2018 (see World Bank [2019a]). That document provides a useful framework for how the World Bank intends to engage with countries and their education systems on a differentiated basis. Further explanation of the criteria to select case studies is in the Evaluation Matrix and Design section. • Subject focus: The evaluation will focus on World Bank support for education quality and enhanced learning outcomes in basic education with an additional focus on support provided to address critical challenges to education delivery and the exacerbation of learning loss associated with the COVID-19 pandemic. • Reference period: The reference period for this evaluation is FY12–22. The evaluation will cover projects approved and a sample of knowledge products published during this period. 12 5. Evaluation Design 5.1 The evaluation will be implemented using an iterative, exploratory approach through which the available data will be tested and refined in three phases. The team is adopting this approach because of the evaluation’s breadth, effectively covering the entire life of the current World Bank education strategy, and the diversity of modalities adopted during implementation of that strategy in pursuit of learning outcomes and strengthening education systems to support basic education. This will help ensure that the evaluation is as nuanced as possible, given the importance and complexity of the development challenge in question. 5.2 Phase 1 of the evaluation will deploy a range of methods: secondary data analysis, interviews, virtual workshop with task team leaders (TTLs), and structured reviews relating to education quality and learning outcomes of: (i) the academic literature on education quality; (ii) a purposively selected sample of ASA at the global and regional levels; (iii) literature on education quality and learning outcomes produced by think tanks, CSOs, and other actors; and (iv) academic literature on various aspects of the pandemic experience (learning loss, mitigation strategies, long-term impacts, and nonlearning impacts like socioemotional effects). Background papers will examine: (i) approaches adopted by other development organizations to realize education quality and enhanced learning outcomes and strengthen education systems; (ii) how the pandemic is affecting the learning crisis (crisis on top of a crisis), including a summary review of the literature on various aspects of the pandemic experience; and (iii) what kinds of lessons can be learned from a large-organization approach to addressing low learning levels and identification of any pandemic-inspired innovation that can be applied postpandemic to help address the learning crisis. 5.3 Phase 2 will deploy the knowledge generated from the previous phase to update the conceptual framework (figure 5.1) and develop protocols for the data collection and analysis of case studies and portfolio review. Case studies of education systems in World Bank–supported countries represent the core context-specific data gathering activity to respond to all evaluation questions. This will involve a uniform approach based on a detailed protocol that will encompass a comprehensive review of strategic, operational, and ASA documentation relevant to the education sector; analysis of available secondary data on education quality and learning outcomes (to include coverage of the compounding effects of COVID-19); interviews with relevant World Bank personnel and other stakeholders, including government counterparts, academics, think tanks, and CSOs; and a review of national strategies, policies, and other key documents relevant to basic education and the pursuit of enhanced learning outcomes. 13 5.4 Phase 3 will validate and refine findings from the case studies and portfolio via interview or workshop with TTLs. The team will test the extent to which findings can be generalized with reference to the theory of change that will be developed from data analysis. The theory of change will describe aspects within education systems through which learning outcomes can be positively affected and will identify entry points for World Bank support. Conceptual Framework for World Bank Support 5.5 At this preliminary stage, the team has identified key concepts and developed a conceptual framework (figure 5.1) to illustrate the World Bank approach to supporting learning outcomes in basic education at the global and country levels. These concepts will guide aspects of initial data gathering during Phase 1 and ultimately will be replaced with the development of a theory of change that will be informed by research carried out during Phase 1 (for example, background paper on approaches by other actors, review of comprehensive approaches to tackling learning poverty, review of World Bank strategy and policies, review of literature on systems approaches to enhancing basic education) and engagement with TTLs from the Education GP. Figure 5.1. Evaluation Framework Source: Independent Evaluation Group. Note: FCV = fragility, conflict, and violence; Lv. = level. 5.6 The conceptual framework depicts the World Bank’s role in supporting education quality and learning outcomes through the education system in line with the 14 current sector strategy (World Bank 2011) and recent policy documents (World Bank 2019a, 2020b). Broadly, the education strategy is to generate global knowledge that can be applied and contextualized at the country level to support education system reform and to coordinate with partners to generate and manage data that describes both the challenges and improvements in education quality and learning outcomes at the global level and individual country levels. The framework provides a simple illustration of the links between the World Bank’s focus on the education system (at the level of basic education) and its support for education quality and learning outcomes. This evaluation also examines the degree to which the World Bank and its clients are learning and adapting, because the global pandemic may require different adaptations, indicated by the arrows among World Bank inputs and measurement and assessment and country- level and the contextual factors and additional consideration with the pandemic. 5.7 This evaluation will examine the full package of World Bank (that is, Education Global Practice and sector) support and inputs for education quality and learning outcomes, including financial support, knowledge, policy dialogue, and strategic partnerships. Regarding financial support, the evaluation will cover all operations as detailed in the portfolio review and analysis (appendix B). For case studies, it will also include relevant projects within the Country Management Unit related to the Education GP and other GPs directly supporting basic education (for example, cash transfer projects under the Social Protection and Labor GP [now the Social Protection GP] or development policy operations from noneducation GPs that incentivize education policy reforms or actions). This will require alignment across sections within the World Bank and among clients. The evaluation will also take account of World Bank knowledge input in the form of global knowledge and initiatives (such as teachers or data) and regional and country-level knowledge (the latter via country cases). The evaluation will also examine policy dialogue and the extent to which the World Bank engages with clients on learning outcomes aiming to maximize positive change and strategically leverage its points of influence. Support for data measurement and its analysis can be critical in policy dialogue and leveraging political and administrative support in favor of reform and may require interconnectedness across sectors. This level of engagement is essential to ensure that a premium is placed on the importance and value of learning for development, growth, human capital, and poverty alleviation (“impact” in the conceptual framework), and the value clients place on equitable learning outcomes. 5.8 An education system includes formal and nonformal programs, plus the full range of program beneficiaries and stakeholders (teachers, trainers, administrators, employees, and students and their families); the rules, policies, and accountability mechanisms that bind an education system together; and the resources and financing mechanisms that sustain the system and provide equitable outcomes for all learners. The 15 evaluation will explore the extent to which the World Bank’s country-level engagement has embraced the complexity of the education system and been inclusive in its efforts that have typically focused on enhancing the learning environment (such as investment in infrastructure; see Barrett et al. 2019); the quality of teachers and teaching (investment in training, materials, and tools such as the Teach and Coach initiatives); and management capacity in education, including at the administrative level and in schools on the basis that weak management capacity at all levels hinders the successful rollout of complex reforms and delivery of quality education services (see, for example, Abdul- Hamid, Saraogi, and Mintz [2017] and Adelman and Lemos [2021]). 5.9 The framework envisions learning outcomes at two levels. The first-level outcome within the education system will be enhanced quality, as shown in figure 2, that can be demonstrated through multiple indicators that improve the learning environment, such as the qualifications of teachers, quality of teacher training, and quality of student-teacher interactions. These are a necessary foundation to support enhanced learning outcomes as envisaged in figure 2 (level 2) with learning outcomes. Enhanced learning outcomes take time to deliver, establish, and verify. This will, in turn, require significant support for assessment—at both the national and international levels—and associated data generation and analysis. Tracking of the World Bank’s learning poverty indicator will be important in that regard. Enhanced human capital is further down the line (level 3). 7 5.10 The pandemic has almost certainly had a negative impact on student learning levels in countries that were already facing a learning crisis before the pandemic. The degree of impact is not fixed among or within countries and has its own set of drivers (for example, length of pandemic school closures, degree of mitigation, and other factors) that may themselves be varying as many countries still face pandemic-related complications (as of February 2022). There are likely other child outcomes that are directly (dropout) or indirectly (socioemotional) related to student learning that have worsened. In other words, the pandemic impact on learning is likely to be substantial and multifaceted, thus it is important to assess the adequacy of the World Bank’s support, given the changing landscape in its clients. Evaluation Matrix and Design 5.11 The evaluation matrix in appendix A details the activities planned to answer each evaluation question and identifies the sources of information. Table 5.1 relates the evaluation’s multiple levels with respective subquestions. 16 Table 5.1. Levels of Coverage for Evaluation Questions Country Global Evaluation Questions Portfolio Level Level EQ1: How effective has World Bank support for basic education (FY12–22) been in X X addressing the binding constraints that hinder the achievement of enhanced learning outcomes in client countries? EQ2: To what extent and how effectively has the World Bank: collaborated with X X X country and global partners to support education quality and enhanced learning outcomes; and used feedback from evidence and experience to inform its work to support improved education quality and learning outcomes for all? EQ3: How well prepared is the World Bank to address additional challenges to X X X education systems that have arisen because of the impact of the [coronavirus] COVID-19 pandemic? Source: Independent Evaluation Group. 5.12 The activities include the following: • Secondary data analysis will serve as a diagnostic tool, working with various data sets—UNESCO Institute for Statistics, World Inequality Database on Education, Service Delivery Indicators Database, World Development Indicators Database, and EdStats—to detail learning outcomes and identify the extent of the learning crisis over time in World Bank country clients. The secondary data analysis will reconstruct and update learning crisis data and figures from WDR 2018, reflecting the passage of time and the compounding of the crisis because of COVID-19 and associated learning loss. The analysis will also provide in-depth educational outcome data in case study countries. For example, it will merge the harmonized learning outcome and Learning-Adjusted Years of Schooling country-level data (using country codes) with UNESCO Institute for Statistics indicators and other data sources, such as COVID-19 response surveys and pandemic school closings data. This will make it possible to prepare a range of contextual summaries for countries to establish, for example, how much they are spending on education, and to what extent the pandemic affected education systems (for example, analysis of the effects of the pandemic school closings), systemic responses (such as distance learning), and analysis of available outcome-related indicators (enrollment trends and dropout) and learning outcome data. • Structured literature reviews will ensure that the evaluation is informed about the evidence and best practice in the pursuit of education quality and enhanced learning outcomes, and understanding important contextual factors. This will feed into all levels and aspects of the evaluation (for example, development of the theory of change, protocols for case studies, and refined portfolio review). Reviews will be conducted within (i) existing systematic literature reviews of 17 impact evaluations and qualitative studies associated with the review; (ii) a purposively selected sample of World Bank regional and global ASA; and (iii) literature on education quality and learning outcomes produced by think tanks, CSOs, and others. A literature review will also seek to discern how the pandemic may be exacerbating an already challenging situation (for example, with reference to learning loss, mitigation strategies, long-term impacts, and nonlearning impacts like socioemotional effects) not only in very vulnerable countries but also in countries that may have been performing relatively well before the pandemic. An additional literature review will seek to identify lessons that can be learned from comprehensive approaches to addressing low learning levels with structured pedagogy (such as Tusome in Kenya) and Teaching at the Right Level (such as Pratham-ACER in India). • Portfolio review and analysis: The team will conduct a refined portfolio review and analysis based on key concepts emerging from the knowledge generated in Phase 1 of the evaluation. Because the portfolio includes Global Partnership for Education operations implemented by the World Bank, and the policies and institutional arrangements of the two differ, separate analyses will be done for each set of operations. See appendix B for a description of the selection process to identify lending and ASA for the basic education portfolio and breakdown. • Case studies of World Bank support for enhanced quality in basic education systems in country clients will be based on multiple sources of evidence. For example, the team will review strategic documents relevant to the education sector—country strategy documents, country-level knowledge products, evaluations, and data—to learn how and to what extent the World Bank has supported the achievement of education quality and learning outcomes. The team will interview relevant national stakeholders and key development partners. To inform the case studies, the team will undertake online semistructured interviews with key informants (national stakeholders, World Bank staff, and key development partners) and use the interviews to collect information and documentation that will be used for triangulation of findings from other sources. • Remote workshops with TTLs will be undertaken both before and after country case studies. A workshop preceding the case studies will be designed to ensure that the team is fully apprised of how TTLs have engaged with strengthening education systems and supporting education quality and learning outcomes throughout the evaluation period, and how the World Bank has responded to the challenges posed by COVID-19. The team will use knowledge generated during Phase 1 and tacit knowledge of TTLs to develop indicators to measure education 18 system improvement. Workshops after the analysis of case study findings will be designed to validate and nuance case study findings. Case Study Selection Criteria 5.13 IEG will undertake nine case studies. Box 5.1 details the criteria used to select the cases. IEG will gather feedback from the Education GP regarding the specific cases to select but will ensure that the cases are selected from among diverse country types, permitting comparison of findings. The criteria ensure selection of both successful and less successful cases. Appendix C elaborates on the approach and selection criteria. Box 5.1. Case Study Selection Criteria Span and type of current engagement • Independent Evaluation Group will select eight cases in countries in which the World Bank has supported at least two financial operations and at least two advisory services and analytics products during the period evaluated. • The cases will include at least one country in which the World Bank has supported at least three advisory services and analytics but no financial operations during the period evaluated. This will ensure that the evaluation covers instances where the World Bank may be using evidence to support policy and system change and reform in the absence of direct investment. Country typology: varying capacity and varying system efficiency • Independent Evaluation Group grouped countries meeting the engagement criteria into country capacity types: low institutional capacity, moderate institutional capacity, and high institutional capacity. Fragile and conflict-affected situation countries are present among each of the capacity types. The secondary data analysis grouped these countries further by levels of system efficiency: low, average, and high, based in relation to spending per pupil and results attained. See appendix C for more details and a list of countries. Source: Independent Evaluation Group. Limitations 5.14 The evaluation has several limitations. The evaluation team recognizes that case study findings may not be generalizable because findings may be contextual. The team will mitigate this challenge by triangulating findings with global knowledge and evidence. The team will also apply robust case selection criteria to create a typology that identifies comparable cases and avoids selecting unique cases. To ensure the collection of comparable data, the team will develop and implement case studies with a common 19 protocol. The TTL workshops and interviews will examine the question of generalizability. 5.15 Limitations posed by data quality and availability. The team is also expecting to encounter low quality or limited availability of data in some countries (for example, in FCV countries or those with low institutional capacity), which may limit the specificity and precision of the analysis. Mitigation strategies include ensuring that data collection is context driven, collaboration with experienced local consultants to facilitate data collection (to include help with identification of key nongovernmental stakeholders), working closely with the Country Management Units and leveraging the support of the Education GP in that regard, engaging with as many relevant stakeholders as possible to ensure as broad a perspective as possible, and working with and analyzing existing (secondary) data sets to ensure robust coverage of any quantitative data available. 5.16 Limitations posed by remote qualitative data collection. IEG will consider mission travel on a case-by-case basis, thus may result in a mix of mission and remote data collection from case studies. Travel restrictions imposed because of the pandemic may prevent the Washington, DC–based team from conducting country visits, at least in some instances. Where this is the case, it will inhibit face-to-face engagement with stakeholders and may limit the quality of qualitative data collected. To mitigate this risk, the team will work closely with the Education GP and the Country Management Unit in each of the countries selected to engage an experienced local consultant to ensure that the team gets to interact, though remotely, with key stakeholders. The team also proposes to explore the possibility of using different methods of consultation tailored to each context, including the use of instant messages or virtual chatbots, mobile surveys, or digital elicitation. Table 5.2 summarizes how these instruments can vary depending on the level of synchronicity, number of individuals, type or means of communication, and virtual platform. The team will use this framework to adjust the methodology according to country conditions and to the profile and preferences of stakeholder groups. Table 5.2. Variations in Methods for Remote Data Collection Remote Number of Type of data Synchronicity individuals communication Communication plaforms collection Asynchro Individ Writin Facebo Whats methods Real time nous ual Group g Audio Video Phone ok App Zoom Audiovisual x x x x x X X x X interviews Chatbot x x x x x x x X x interviews/ instant messaging Mobile surveys x x x X 20 Remote Number of Type of data Synchronicity individuals communication Communication plaforms collection Asynchro Individ Writin Facebo Whats methods Real time nous ual Group g Audio Video Phone ok App Zoom Digital x x x x x x x X x elicitation/ journaling/ Photovoice Source: Independent Evaluation Group. 5.17 Limitations due to structural choices. Some structural evaluation choices have a bearing on the nature of this evaluation. First, COVID-19 has disrupted operations and in-person learning. Schools remain closed or partially open in some countries. Although this may appear as a limiting factor for the evaluation, IEG interprets it as an opportunity. The evaluation will provide learning that can help foster adaptability and resilience in the World Bank’s approach. Because the evaluation focuses on systems, it will not seek to engage with students who are the beneficiaries of education, and it will engage with teachers only through representative organizations such as trade unions. This approach allows for a focused effort designed to provide systems learning that can benefit World Bank engagement to improve outcomes for beneficiaries. Second, the evaluation scope does not include early childhood development, which is intrinsically needed for the learning process. IEG recognizes, as per management comments on the draft Approach Paper and its own evaluation of early childhood development (2015), that disparities in student readiness to learn once they reach primary education is an important dimension of learning poverty that requires early childhood education, nutrition, early stimulation, and parenting support. The World Bank and its partners are engaged in a continuum of interrelated support at this critical stage of life that the evaluation will not examine. However, focusing the evaluation’s scope on basic education (primary and lower secondary) is necessary for the evaluation to examine deeply the basic education systems in various contexts to ensure robust findings. Given the breadth of the evaluation coverage (the entire effective period of the current World Bank education strategy and the many modalities adopted to implement that strategy), the depth of findings would be jeopardized by also assessing systems supporting early nutrition, early education, and parenting and stimulation, which cut across multiple sectors. All of these limitations will be acknowledged in the evaluation report. 6. Quality Assurance Process 6.1 The evaluation will be subject to a rigorous quality assurance process. This Approach Paper will undergo IEG’s standard internal review process and be peer reviewed by three international experts: Barbara Bruns: adjunct professor, Global Human Development, Georgetown University, and research fellow at Center for Global Development; Michelle Kaffenberger: research fellow, Research on Improving Systems 21 of Education Program; and Paula Malan: senior adviser, Development Policy (Education), Ministry of Foreign Affairs, Finland. 6.2 The detailed methodology will be assessed further and finalized in consultation with IEG’s evaluation methods team. The evaluation team will also maintain close contact with IEG’s methods team regarding ongoing progress and any issues that may arise. 7. Engagement and Dissemination 7.1 The evaluation team has and will continue to pursue close engagement with the Education GP and other key stakeholders throughout the evaluation process. IEG met with senior management and key staff in the Education GP to ensure this evaluation’s relevance and utility. IEG will maintain that close contact throughout the evaluation. During Phase 1, this will involve virtual workshops with TTLs who will contribute to the development of the protocol for country case studies. During Phase 3, IEG will implement additional workshops with TTLs to validate findings for the triangulation and generalization of findings. 7.2 The collaboration with the Education GP will support the selection of case studies and the identification of key informants, government representatives, CSOs, and main stakeholders, and it will provide an opportunity to exchange data and information. Finally, the evaluation team anticipates strong interest from external stakeholders—in particular, ministries of education, CSOs, other development partners, academics, and other researchers—because the evaluation will provide insights into system-level issues that need to be addressed to improve learning outcomes. 7.3 This evaluation’s main output will be a report that presents relevant findings and lessons. IEG will design an evaluation outreach strategy for both internal and external audiences. In addition to the final report, the team will develop an outreach plan in collaboration with the IEG communications team. 8. Resources 8.1 The team members for the evaluation offer expertise in education, methods, and secondary data analysis, and experience with issues related to disability, education systems, and gender. The team consists of Susan Ann Caceres (task team leader), Mariana Branco, Jeffrey Marshall, Xiaoxiao Peng, Anthony Martin Tyrrell, Denise Vaillancourt, and Disha Zaidi. Other members will be added to the team, including local consultants to undertake country case studies. The budget for the evaluation is $800,000. The evaluation report will be finalized in the fourth quarter of FY23. 22 8.2 Estelle Raimondo will provide methodological advice and guidance. Yezena Yimer will provide administrative support. The work will be conducted under the guidance of Galina Sotirova (manager, Corporate and Human Development), Oscar Calvo-Gonzalez (director, Human Development and Economic Management), and Alison Evans (Director-General, Evaluation). 1Soliván and Winthrop (2014) also note that UNESCO (2014) rightly identifies teachers as the most important schooling factor; however, they comment that the four recommendations regarding teachers—attracting good quality teachers, improving teacher education, ensuring that the most disadvantaged students have the best teachers, and retaining good teachers with incentives—are not new ideas, and that the persistent issue has been failure to implement such recommendations at scale. The reference to Dakar is to the World Education Forum 2000 held there, which reaffirmed that all children have the human right to benefit from an education that will meet their basic learning needs in the best and fullest sense of the term. 2Specifically, Millennium Development Goal 2, Target 2a states: Ensure that, by 2015, children everywhere, boys and girls alike, will be able to complete a full course of primary schooling. For more information, visit https://www.un.org/millenniumgoals/education.shtml. 3Systems Approach for Better Education Results (SABER) initially focused on assessing how well a country’s education policies and institutions aligned with its education goals, and benchmarked these policies against global evidence of what works to improve learning. Later SABER began developing a framework for measuring and analyzing service delivery at the school level. SABER 2.0 moved on to measuring learning and its drivers with a view to meeting growing demand for comprehensive, streamlined, and cost-effective instruments that build on the existing SABER tools to measure the drivers of learning, and that can be scaled up to all countries to better identify binding constraints to improving learning, guide policy decisions, and monitor progress on policy efforts to address them. 4 The Accelerator Program recognizes and supports cohorts of governments that exhibit the crucial ingredients needed to fight learning poverty. The program was launched in late 2020 by the World Bank and UNICEF, in partnership with the Bill & Melinda Gates Foundation, UK’s Foreign, Commonwealth & Development Office (FCDO), UNESCO’s Institute of Statistics, and USAID. The initial cohort of Accelerators includes Brazil (state of Ceará), Ecuador, Kenya, Morocco, Mozambique, Niger, Nigeria (Edo State), Pakistan, Rwanda, and Sierra Leone. 5 This includes 206 parent projects and 45 additional financing projects. 6Basic education consists of educational activities at both primary (United Nations Educational, Scientific, and Cultural Organization Institute for Statistics level 1) and lower secondary education (level 2) that are designed to meet basic learning needs as defined in the World Declaration on Education for All (adopted by the World Conference on Education for All in Jomtien, Thailand, in March 1990). 23 7Within this paradigm, the need to improve the quality of learning assessment systems and the availability of reliable learning assessment data to signal or monitor changes in the learning crisis and learning outcomes is clearly of high importance. 24 Bibliography Abdul-Hamid, Husein, Namrata Saraogi, and Sarah Mintz. 2017. Lessons Learned from World Bank Education Management Information System Operations: Portfolio Review, 1998–2014. Washington, DC: World Bank. Adelman, Melissa, and Renata Lemos. 2021. Managing for Learning: Measuring and Strengthening Education Management in Latin America and the Caribbean. Washington, DC: World Bank. Azevedo, Joao Pedro. 2020. “Learning Poverty: Measures and Simulations.” Policy Research Working Paper 9446, World Bank, Washington, DC. Barrett, Peter, Alberto Treves, Tigran Shmis, Diego Ambasz, and Maria Ustinova. 2019. The Impact of School Infrastructure on Learning: A Synthesis of the Evidence. Washington, DC: World Bank. Datta, Lois-ellin. 1997. “Multimethod Evaluations: Using Case Studies Together with Other Methods.” In Evaluation for the 21st Century: A Handbook, edited by Eleanor Chelimsky and William R. Shadish, 344–59). Thousand Oaks, CA: Sage Publications, Inc. Haddad, Wadi D., Martin Carnoy, Rosemary Rinaldi, and Omporn Regel. 1990. Education and Development: Evidence for New Priorities. Washington, DC: World Bank. Hesse-Biber, Sharlene, and R. Burke Johnson, eds. 2015. The Oxford Handbook of Multimethod and Mixed Methods Research Inquiry. New York: Oxford University Press. Patton, Michael Quinn. 2015. Qualitative Research and Evaluation Methods, 4th ed. Thousand Oaks, CA: Sage Publications, Inc. Soliván, Maribel, and Rebecca Winthrop. 2014. “Stealing from the Future: The Urgency of the Global Learning Crisis.” Education Plus Development (blog), February 14, 2014. https://www.brookings.edu/blog/education-plus-development/2014/02/14/stealing-from- the-future-the-urgency-of-the-global-learning-crisis/. Thorn, William, and Stéphan Vincent-Lancrin. 2021. Schooling during a Pandemic: The Experience and Outcomes of Schoolchildren during the First Round of COVID-19 Lockdowns. Paris: OECD Publishing. United Nations Educational, Scientific, and Cultural Organization (UNESCO). 2014. Education for All Global Monitoring Report: Teaching and Learning: Achieving Quality for All. Paris: UNESCO. UNESCO Institute for Statistics (UIS). 2012. International Standard Classification of Education: ISCED 2011. Montreal: UNESCO Institute for Statistics. UNESCO Institute for Statistics (UIS). 2017. “More Than One-Half of Children and Adolescents Are Not Learning Worldwide.” Fact Sheet 46, UNESCO Institute for Statistics, Montreal. 25 UNESCO Institute for Statistics. 2019. “New Methodology Shows that 258 Million Children, Adolescents, and Youth Are Out of School.” Fact Sheet 56, UNESCO Institute for Statistics, Montreal. World Bank. 1999. World Bank Education Sector Strategy. Washington, DC: World Bank. World Bank. 2006. From Schooling Access to Learning Outcomes: An Unfinished Agenda. An Evaluation of World Bank Support to Primary Education. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/reports/primary_education_evalu ation.pdf. World Bank. 2011. Learning for All: Investing in People’s Knowledge and Skills to Promote Development. Washington, DC: World Bank. World Bank. 2013. Responding to Global Public Bads: Learning from Evaluation of the World Bank Experience with Avian Influenza 2006–13. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/reports/avian_flu1.pdf. World Bank. 2017. Crisis Response and Resilience to Systemic Shocks: Lessons from IEG Evaluations. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/reports/building-resilience.pdf. World Bank. 2018. World Development Report 2018: Learning to Realize Education’s Promise. Washington, DC: World Bank. World Bank. 2019a. Ending Learning Poverty: What Will It Take? Washington, DC: World Bank. World Bank. 2019b. Selected Drivers of Education Quality: Pre-and In-Service Teacher Training. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/Drivers_of_Educ ation_Quality.pdf. World Bank. 2020a. The Human Capital Index 2020 Update: Human Capital in the Time of COVID-19. Washington, DC: World Bank. World Bank. 2020b. Realizing the Future of Learning: From Learning Poverty to Learning for Everyone, Everywhere. Washington, DC: World Bank. Yardley, Lucy. 2008. “Demonstrating Validity in Qualitative Psychology.” In Qualitative Psychology: A Practical Guide to Research Methods, 2nd Ed edited by Jonathan A. Smith, 235– 51. London: Sage Publications Ltd. 26 Appendix A. Evaluation Methodology Table A.1. Evaluation Methodology Evaluation Questions Methods Sources of Information Overarching evaluation question: How To answer the overarching question, the team will compare and align findings from the multiple sources of evidence that will be has World Bank support for basic generated in responding to the subquestions with the concepts within the theory of change that will be developed during Phase 1 education contributed to the of the evaluation. The planned data collection and analysis for every source of evidence will be driven by common evaluation achievement of enhanced learning questions and protocols. The team will implement data analysis in a cohesive manner that triangulates the sources of evidence, outcomes since the Learning for All rather than independent inquiries of each source. In this manner, the evaluation can arrive at general findings and conclusions strategy, and what can be learned regarding the issues posed by the evaluation questions or contained within the key aspects of the theory of change. from those efforts to inform support to the learning recovery from the [coronavirus] COVID-19 pandemic? EQ1. How effective has World Bank (i) Review of World Bank operational planning, (i) World Bank operational planning and knowledge products retrieved from support for basic education (FY12–22) knowledge products, and knowledge initiatives the Open Knowledge Repository, ImageBank, and other World Bank been in addressing the binding related to, for example, education quality, learning repositories and supported websites. constraints that hinder the outcomes, the learning crisis/learning poverty. achievement of enhanced learning outcomes in client countries? (ii) Review of knowledge products and initiatives on (ii) Background papers will be produced based on broad internet searches education quality, learning outcomes, and the with keywords/terms (for example, education quality, learning outcomes, learning crisis/learning poverty produced by key learning crisis, and so on) and targeted searches via websites of key actors other than the World Bank to reflect on pros organizations to include other multilateral actors (United Nations Educational, and cons of how the World Bank has sought to Scientific, and Cultural Organization [UNESCO], United States Agency for address the crisis in learning. International Development, OECD, the European Union, other MDBs), and global nongovernmental organizations (NGOs) and global think tanks. (iii) Structured synthesis of existing literature (iii) Documents retrieved from JSTOR, International Initiative for Impact review(s) of the academic and impact evaluation Evaluation, EconLit, GPE, Education Resources Information Center, British literature on what works (or not) to enhance learning Education Index, and other sources. outcomes—what types and combinations of interventions work to enhance learning outcomes, what lessons can be learned in specific context, and so on. (iv) Secondary data analysis to identify, for example, (iv) Data retrieved from various sources to include UNESCO Institute for the scale of the crisis in learning and relative Statistics, World Inequality Database on Education, Service Delivery Indicators targeting of World Bank resources and attention. database, World Development Indicators database, EdStats database. 27 Evaluation Questions Methods Sources of Information (v) Semistructured interviews to explore the rationale (v) Interviews with senior World Bank personnel in the Education Global for the approach taken by the World Bank and views Practice and representatives of key institutions (for example, UNESCO and on the overall coherence of the approach. OECD), other MDBs, NGOs, think tanks, academics. (vi) Taking perspectives from education TTLs on the (vi) Virtual workshops with TTLs. World Bank’s response to the learning crisis/learning poverty and how this has influenced their approach to operations, ASA, and engagement overall. (vii) Case studies. (vii) Case studies (as described under EQ2) will be conducted to review broader World Bank support for basic education FY12–22. (viii) Portfolio review and analysis: The selection (viii) Enterprise Data Catalog data sets: All Projects, Project Sector, Project criteria and process for the portfolio review began Theme. with the identification of active or closed lending and ASA projects in the Education Global Practice approved in or after 2012. Then the portfolio excluded projects that focus on early childhood, tertiary, vocational, or adult education, based on project names, development objectives, and sector coding. Additional financing projects without parent projects identified in the same portfolio were also excluded. A total of 813 relevant operations were identified, consisting of 251 lending projects (206 parent projects and 45 additional financing) and 562 ASA projects. Excluding regional projects, the portfolio covers 125 countries, 59 percent of which have both lending and ASA projects. Within the portfolio, 120 projects are identified as relevant to COVID-19 response, including 80 lending and 40 ASA projects. Projects focused primarily on basic education were also flagged, including 169 basic education projects (94 lending projects and 75 ASA projects). A protocol for the initial expanded portfolio review and analysis for the entire lending portfolio will be developed on knowledge derived during Phase 1. EQ2: To what extent and how (i) Review of World Bank operational planning, (i) World Bank operational planning and knowledge products retrieved from effectively has the World Bank: knowledge products, and knowledge initiatives the Open Knowledge Repository, ImageBank, and other World Bank collaborated with country and global repositories and supported websites. 28 Evaluation Questions Methods Sources of Information partners to support education quality related to, for example, education quality and and enhanced learning outcomes; and learning outcomes. used feedback from evidence and experience to inform its work to support improved education quality and learning outcomes for all? (ii) Review of knowledge products and initiatives on (ii) Background papers will be produced based on broad internet searches education quality, learning outcomes, and the with keywords/terms (for example, education quality, learning outcomes, learning crisis/learning poverty produced by key learning crisis, and so on) and targeted search via websites of key actors other than the World Bank to reflect on organizations to include, in the first instance, other multilateral actors cutting-edge thinking and approaches to addressing (UNESCO, United States Agency for International Development, OECD, the the crisis in learning and the extent to which such European Union, other MDBs) and second, global NGOs and global think thinking is present in how the World Bank has tanks. sought to address that crisis. (iii) Structured synthesis of existing literature (iii) Documents retrieved from JSTOR, International Initiative for Impact review(s) of the academic and impact evaluation Evaluation, EconLit, GPE, Education Resources Information Center, British literature on what works (or not) to enhance learning Education Index, and other sources. outcomes—what types and combinations of interventions works to enhance learning outcomes, what lessons can be learned, and so on. (iv) Semistructured interviews to explore the (iv) Interviews with senior World Bank personnel in the Education Global rationale for the approach taken by the World Bank Practice and representatives of key institutions (for example, UNESCO and and views on the overall coherence of the approach. OECD), other MDBs, NGOs, think tanks, academics. (v) Taking perspectives from Education TTLs on the (v) Virtual workshops with TTLs. World Bank’s response to improve learning and address the learning crisis/learning poverty and how this has influenced their approach to operations, ASA, and engagement overall. (vi) Portfolio review and analysis: A protocol for the (vi) Enterprise Data Catalog data sets: All Projects, Project Sector, Project portfolio review and analysis for the entire lending Theme. portfolio will be developed based on knowledge generated in Phase 1. (vii) Case studies. (vii) Case studies will be conducted to consider World Bank support for basic education FY12–22. Selection of cases is detailed in section 5 (box 1) of the Approach Paper. Each of the nine case studies will be undertaken as follows. 29 Evaluation Questions Methods Sources of Information (a) Secondary data analysis to provide up-to-date (a) The secondary data analysis will draw on various international databases to picture (data permitting) of current situation provide an up-to-date picture of learning poverty for each of the country regarding the crisis in learning at the country level cases, to include detail on the effects of COVID-19 on the crisis in learning and available educational indicators and outcome and its broader impact on learning loss. data. (b) Review of country strategy documentation to (b) National education/development strategies; World Bank country program ascertain the relative strategic priority given to the units, Systematic Country Diagnostics, and other strategic documentation. learning crisis/learning poverty. (c) Review of ASA for each country relating to the (c) ASA retrieved from the Open Knowledge Repository, ImageBank, and as education sector. referred by Country Management Unit personnel. (d) Review of operations for each country relating to (d) In-depth structured country portfolio review (Project Appraisal Documents, the education sector. In addition, to support the Implementation Completion and Results Reports, Project Performance generalizability of case study findings, additional Assessment Reports, Implementation Completion and Results Report Reviews, portfolio analysis (to include projects and ASA) will impact and process-related evaluations, and any other relevant documents) of be undertaken to determine the extent to which World Bank education projects. findings about World Bank implementation in case study countries also applies (or not) for country types like those selected for study (for example, in countries affected by fragility, conflict, and violence; and in low, moderate, and high institutional capacity countries). (e) Peer reviewed literature (in English) on basic (e) As available and relevant for case studies. education, literacy, and the crisis in education at national level. (f) Review of other relevant literature produced by (f) Referral by interviewees and internet search with keywords (for example, development partners, NGOs, and think tanks. education, literacy, numeracy, learning crisis) associated with country name. (g) Review of relevant IEG evaluations. (g) IEG education-specific evaluations on, for example, education portfolio review, drivers of quality education, and other IEG evaluations such as the evaluation on World Bank convening power. (h) Semistructured interviews and virtual workshops (h) Senior government officials responsible for the education sector; senior with key government, World Bank, and other management at the Country Management Unit, TTLs working in human informants. development sectors, and TTLs in other sectors where the country portfolio review suggests a link with basic education; interviews with representatives of other development partners, NGOs, academics, and think tanks. (i) Taking perspectives from Education TTLs on the (i) Virtual workshop with TTLs. World Bank’s response to strengthening education 30 Evaluation Questions Methods Sources of Information systems and addressing the learning crisis/learning poverty and how this has influenced approaches to operations, ASA, and engagement. EQ3. How well prepared is the World (i) All of the methods referenced in relation to EQ2 (i) As under EQ2. Bank to address additional challenges are also relevant here, noting the requirement for a to education systems that have arisen particular and separate focus (that is, in interviews, because of the impact of the COVID- workshops, reviews of World Bank strategies and 19 pandemic? policies, case studies, and so on) on the period since 2019. (ii) Literature review on how the pandemic is (ii) Academic literature on pandemic effects on education systems and exacerbating the learning crisis. learning outcomes. (iii) Secondary data analysis to establish the scale of (iii) Data retrieved from various sources to include UNESCO Institute for the effects of the global pandemic (particularly with Statistics, World Inequality Database on Education, Service Delivery Indicators reference to case countries). database, World Development Indicators database, EdStats database. Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; GPE = Global Partnership for Education; IEG = Independent Evaluation Group; MDB = multilateral development bank; OECD = Organisation for Economic Co-operation and Development; TTL = Task team leader. 31 Appendix B. Portfolio Review The selection criteria and process for the portfolio review began by identifying active or closed lending and advisory services and analytics (ASA) projects in the Education Global Practice approved since fiscal year 2012. Then projects were rejected if they focused exclusively on early childhood, tertiary, vocational, or adult education. The exclusion was based on three variables: project name, project development objective, and project sector coding, with data pulled from several World Bank data sets and the Global Partnership for Education grant portfolio data set. 1 The following are the exclusion criteria. • All projects with names containing words related to early childhood, upper secondary, tertiary, vocational, adult education, and job skills. The search applies to project short name, display name, and legal name. Words searched include the following: “early childhood,” “preschool,” “pre-school,” “preprimary,” “pre-primary,” “upper secondary,” “senior secondary,” “post- basic,” “post basic,” “higher ed,” “tertiary,” “university,” “universities,” “college,” “vocation,” “labor,” “job,” “employ,” “entrepreneur,” “business,” “enterprise,” “continuing education,” “student loan,” “adult,” “ECD,” “ECEC,” “ECCD,” “ECED,” “ECE,” “ECCE,” “HE,” and “TVET.” In addition, the words “workforce” and “profession” were searched when not accompanied by the word “teach.” • Lending projects with (i) project development objective containing words related to early childhood, upper secondary, tertiary, vocational, adult education, and job skills; and (ii) without sector coding of primary education or secondary education or both. The rule applies only to lending projects because missing sector data among ASA projects is significant. The same list of words detailed in the previous bullet was used, and the sector condition was added to avoid false-negative exclusion. • Exclusion based on sector coding. o For all projects, the exclusion applies if the sum or count of nontargeted sectors (sectors excluding primary education, secondary education, public administration, and other education) equals the total sum or count of all sectors. o To exclude projects addressing the education sector but without any coverage for primary and secondary education, the combination of sectors was reviewed. Lending projects were excluded if sector coding (i) included public administration—education and/or other education, and (ii) included 32 early childhood education and/or tertiary education and/or workforce development and vocational education and/or adult, basic, and continuing education, but (iii) did not include primary education and secondary education. ASA projects with the targeted sector combination were identified and reviewed manually to determine exclusion, given the lower reliability of sector data for ASA projects. To ensure that the parent projects are included or excluded along with their additional financing projects, all additional financing projects tagged to exclude were reviewed manually and the exclusion status adjusted as needed. After dropping projects to exclude, additional financing projects without parent projects in the data set were also removed. Overall Portfolio A total of 813 operations were identified, consisting of 251 lending projects (206 parent projects and 45 additional financing projects) and 562 analytical projects (figure B.1). Figure B.1. Portfolio Composition Source: World Bank Enterprise Data Catalog. Note: AF = additional financing. Among lending projects, the International Development Association has the largest share both in number (45 percent) and volume (66 percent; figure B.2). Ninety-six lending projects (38 percent) received funding from the Global Partnership for Education (figure B.3). Regarding product line, the majority of projects are investment project financing (80 percent; figure B.4). 33 Figure B.2. Lending Projects by Agreement Type a. Number of projects b. Volume of projects (US$, millions) Source: World Bank Enterprise Data Catalog. Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association. Figure B.3. Lending Projects That Received Global Partnership for Education Funding Source: World Bank Enterprise Data Catalog. Note: GPE = Global Partnership for Education. Figure B.4. Lending Projects by Lending Instrument Source: World Bank Enterprise Data Catalog. 34 The regional breakdown of lending and analytical support (figure B.5) indicates that Western and Central Africa, Eastern and Southern Africa, South Asia, and Latin America and the Caribbean have the largest share of lending projects. The largest number of analytical projects was in Latin America and the Caribbean, followed by Europe and Central Asia and Eastern and Southern Africa. There is a notable share of projects in contexts of fragility, conflict, and violence—25 percent lending and 14 percent ASA (figure B.6). Excluding regional projects, the portfolio covers 124 countries, 60 percent (74) of which have both lending and ASA projects. Eight countries account for almost one-quarter of the lending projects: Bangladesh, Ethiopia, Haiti, India, Nicaragua, Nigeria, Pakistan, and The Gambia. 35 Figure B.5. Projects by Region and Product Line Source: World Bank Enterprise Data Catalog. Note: ASA = advisory services and analytics. Figure B.6. Projects by Fragile and Conflict-Affected Situation Status at Project Approval and Product Line Source: World Bank Enterprise Data Catalog. World Bank Group Classification of Fragile and Conflict-Affected Situations. Note: ASA = advisory services and analytics; FCS = fragile and conflict-affected situation. Coronavirus Portfolio Projects in the portfolio are flagged as coronavirus (COVID-19) response if they meet any of the following criteria: • The project is tagged with the COVID-19 emergency response code. • The project is tagged with one or more COVID-19 crisis response types. • The project had the pandemic response theme code and was approved after 2018. • The project name, project development objective, or components contained the word “COVID.” 36 The COVID-19 flagging led to the identification of 120 projects (80 lending projects and 40 ASA projects). The distribution of COVID-19-relevant projects across regions resembles the overall portfolio with one exception: South Asia has 25 percent of lending projects (figure B.7) compared with 15 percent in the overall portfolio (figure B.5). Figure B.7. COVID-19 Projects by Region and Product Line Source: World Bank Enterprise Data Catalog. Note: ASA = advisory services and analytics. Basic Education Portfolio According to the UNESCO International Standard Classification of Education standard, basic education consists of primary education (first stage of basic education) and lower secondary education (second stage). 2 Identifying basic education projects in the portfolio must therefore separate lower secondary from upper secondary education, which is difficult. Two approaches were attempted, though both had issues. Moreover, the tagging of ASA projects should be treated with additional caution, given the notable absence of sector coding values for ASA projects. A review of project documents is needed to arrive at accurate tagging of basic education projects. The details of the two identification approaches are as follows: • Under both approaches, projects with names containing words related to primary education and/or lower secondary education are tagged. The words searched include the following: “early grade,” “basic education,” “primary education,” and “lower secondary.” 37 • Under the first approach, projects were tagged further with the primary education sector coding at 50 percent or above. The limitation is the possible omission of projects addressing lower secondary education (figure B.8, panel a). • Under the second approach, projects with the sum of primary education and secondary education sector coding at 50 percent or above are tagged. The issue with this approach is the possible inclusion of upper secondary education projects (figure B.8, panel b). Figure B.8. Basic Education Projects by Product Line a. Approach 1 b. Approach 2 Source: World Bank Enterprise Data Catalog. Note: ASA = advisory services and analytics. The total number of identified basic education projects ranges from 169 (approach 1) to 270 (approach 2), but the real number should fall somewhere within this range. A significant share of lending projects are identified mainly because of the missing sector data among ASA projects. Among lending projects, basic education projects are within the 37 percent to 62 percent range. 1The World Bank data sets from the Enterprise Data Catalog include “All Projects,” “Project Sector,” “Project Development Objective,” “Project Components,” “Project Loan Summary,” “Project Crisis Response,” and “ASA Activity Details.” 2The definition of basic education can be found on the website of UNESCO Institute for Statistics http://uis.unesco.org/en/glossary-term/basic-education. 38 Appendix C. Case Study Selection Criteria Adopting a systematic approach to addressing this deep and long-standing issue suggests that case study selection should prioritize countries where the World Bank has had active projects during most of the evaluation period. Reflecting this, the evaluation team will select from among those countries in which the World Bank had at least two lending projects, a minimum of two advisory services and analytics projects, and one or more project (lending or advisory services and analytics) both before and since fiscal year 2017 during the evaluation period. This reduces the pool of choices from 125 to 41 countries distributed across the regions as follows: Western and Central Africa (12), Eastern and Southern Africa (9), Europe and Central Asia (5), South Asia (5), Latin America and the Caribbean (5), East Asia and Pacific (4), and Middle East and North Africa (1). Cases will be selected from among those 41 countries with reference to relative country capacity and education efficiency. This process will also identify country types such that learning from the cases can be cross-examined through testing differences and similarities been the types (through intensive desk-based investigation), and to ensure that learning can be validated and articulated in a manner that will provide World Bank management with information that will enhance organizational learning. The team measured capacity using percentiles among Country Policy and Institutional Assessment (CPIA) data from all countries (n = 138) with CPIA data available. The CPIA was highly correlated with other capacity measures, thus the overall CPIA score was used, as shown in table C.1. Table C.1. Correlation between Country Policy and Institutional Assessment Overall Score and Other Indicators Covariance Significance Data set—all countries with CPIA data CPIA Quality of Public Administration 0.8257 0.000 WGI Government Effectiveness Rank 0.8232 0.000 WGI Control of Corruption Rank 0.5959 0.000 Data set—41 selected countries CPIA Quality of Public Administration 0.8625 0.000 WGI Government Effectiveness Rank 0.8443 0.000 WGI Control of Corruption Rank 0.6146 0.000 Source: Enterprise Data Catalog; World Bank CPIA data; Worldwide Governance Indicators. Note: CPIA = Country Policy and Institutional Assessment; WGI = Worldwide Governance Indicators. A key consideration for this evaluation is the World Bank’s approach (through, for example, policy dialogue, convening power, lending, advisory services and analytics, 39 and partnership) to improving learning outcomes in diverse country contexts in which the World Bank will encounter a wide range of country capacity and management efficiency in education systems. Efficiency was examined to see whether some countries were performing high, average, or low on indicators of participation and student achievement, given their level of spending. The level of spending was measured in two ways. First, the evaluation team examined the level of relative education spending effort, captured by indicators such as education spending as a percentage of gross domestic product. The second set of indicators focused on real spending per student, measured in constant US dollars per pupil. Hence, the core criteria applied to identify case types will be as follows: • Capacity: Cases will be selected with reference to the country types identified in recent policy documents (World Bank 2019a, World Bank 2020b). Three country types—low capacity, medium capacity, and high capacity—will be decided with reference to CPIA scores that are available for all countries. Fragility, conflict, and violence countries, which have varying levels of capacity, will also be selected among these three types. • Efficiency: Cases will also be selected with reference to the relative efficiency of their education sector—results attained compared with spending per pupil (high, average, or low). The Independent Evaluation Group conducted analysis of education system performance based on the average (2010–19) of three sets of indicators: (i) net enrollment rates in primary and secondary education; (ii) the harmonized learning outcome measure, created by the World Bank, which puts countries on a single, comparable scale based on overlapping regional and international assessments; and (iii) the Learning-Adjusted Years of Schooling measure that combines the first two indicators. For countries for which data are available (31 of the 41 in the sample), preliminary analysis provided the relative level of education sector efficiency based on simple regressions where the two net enrollment rate measures and the Learning-Adjusted Years of Schooling measure were regressed onto the measure of average primary-secondary spending per pupil. For the 10 countries without a full set of data, Independent Evaluation Group regressed the available Learning-Adjusted Years of Schooling measure onto gross domestic product per capita to develop a proxy measure of efficiency to permit further examination of the full sample (n = 41). Table C.2 details the distribution of the sample case countries with reference to level of country capacity and level of efficiency in the education sector. 40 Table C.2. Sample for Case Selection by Capacity and Efficiency Country Capacity Low Efficiency Average Efficiency High Efficiency Low Niger a Cameroon a, e Tajikistan Chad a, e Myanmar a Mali a, e Timor-Lestec Ethiopiaa, e Sierra Leonee Gambia, Thee Zambiae Sudand, e Haitia Angola Lebanond Liberiae Bangladeshe Medium Tanzaniae Cambodiae Nepale Senegal Pakistane Sri Lanka Guinea Ghana Kyrgyz Republic Nigeria a, e Malawi e Madagascare High Indiae Vietnam Kenyae Georgia Uganda e Mexicoe Benine Perue Uruguaye Moldovae Armeniab Brazile Source: Enterprise Data Catalog; World Bank Country Policy and Institutional Assessment data; Worldwide Governance Indicators; Independent Evaluation Group analysis. Note: FCV = fragility, conflict, and violence. a. Medium-intensity conflict FCV category. b. High-intensity international conflict FCV category. c. Small state FCV category. d. High institutional and social fragility FCV category. e. Emergency response operations with the education theme code (coronavirus projects). Given that learning loss recovery will vary based on the extent of school closures and mitigation measures taken in each context, the sample will ensure that World Bank emergency operations with the education theme code are present among a portion of the sample (as noted in the table). Finally, Independent Evaluation Group will consult with the World Bank’s Education Global Practice to gather views on the final sample selected. 41